# Multimodality and Attention Increase Alignment in Natural Language
Prediction Between Humans and Computational Models

Viktor Kewenig  Psychology and Language Science, Experimental Psychology,
University College London, London, UK  Corresponding author<EMAIL_ADDRESS>Andrew Lampinen  Independent  Samuel A. Nastase  Princeton Neuroscience
Institute, Princeton University, Princeton, NJ, USA  Christopher Edwards
Psychology and Language Science, Experimental Psychology, University College
London, London, UK  Quitterie D’Elascombe  Computer Science Department, Exeter
University  Akilles Richardt  Psychology and Language Science, Experimental
Psychology, University College London, London, UK  Jeremy I. Skipper
Psychology and Language Science, Experimental Psychology, University College
London, London, UK  Gabriella Vigliocco  Psychology and Language Science,
Experimental Psychology, University College London, London, UK

## Abstract

The potential of multimodal generative artificial intelligence (mAI) to
replicate human grounded language understanding, including the pragmatic,
context-rich aspects of communication, remains to be clarified. Humans are
known to use salient multimodal features, such as visual cues, to facilitate
the processing of upcoming words. Correspondingly, multimodal computational
models can integrate visual and linguistic data using a visual attention
mechanism to assign next-word probabilities. To test whether these processes
align, we tasked both human participants (N = 200) as well as several state-
of-the-art computational models with evaluating the predictability of
forthcoming words after viewing short audio-only or audio-visual clips with
speech. During the task, the model’s attention weights were recorded and human
attention was indexed via eye tracking. Results show that predictability
estimates from humans aligned more closely with scores generated from
multimodal models vs. their unimodal counterparts. Furthermore, including an
attention mechanism doubled alignment with human judgments when visual and
linguistic context facilitated predictions. In these cases, the model’s
attention patches and human eye tracking significantly overlapped. Our results
indicate that improved modeling of naturalistic language processing in mAI
does not merely depend on training diet but can be driven by multimodality in
combination with attention-based architectures. Humans and computational
models alike can leverage the predictive constraints of multimodal information
by attending to relevant features in the input.

## Introduction

Recent advances in generative artificial intelligence (AI) have stimulated
debate over the extent to which models reproduce certain human cognitive
processes.1; 2; 3; 4; 5; 6 Direct brain-model comparisons suggest that
hierarchical predictive processing, driven by contextual linguistic
information, underlies language processing in both humans and AI. 7; 8; 9 One
limitation of many of these studies is that they focus on unimodal models
(text-based) and unimodal (text- or audio-based) experimental stimuli (but see
Dong and Toneva 10). Yet in ecological settings, human language input is often
grounded in one or more perceptual modalities.11 Do multimodal models
replicate human grounded language predictions better than their unimodal
counterparts? If so, what architectural features support this alignment?

Theory and empirical evidence suggest that multimodal grounding is the basis
for the formation of internal mental models of external categories,
situations, and events.12; 13; 14 These ‘Situation models’ facilitate the
integration of linguistic information with contextual information (such as
visual context) or internal priors for next-word prediction by placing
constraints on the space of upcoming semantic content, reducing cognitive load
and improving comprehension.15; 16; 17; 18 Indeed, there is ample
behavioural19; 20; 21; 22; 23; 24; 25; 26; 27; 28 and neurobiological29; 30;
31; 32; 33 evidence that humans process linguistic and nonlinguistic input
jointly and immediately and that naturalistic language processing involves
constructing situation models using both language and multimodal information
in real time.

Given that human language processing seems to be informed by non-linguistic
inputs and computations, grounding by means of adding non-linguistic
modalities to the embedding space in computational language models may present
a way of aligning model predictions more closely with human processing. It is
an open question whether this could eventually bridge the apparent gap between
state-of-the-art, disembodied artificial intelligence and situated, embodied
human agents.34 Current efforts toward multimodal models have begun combining
images and text (e.g., Alayrac et al. 35). This presents a good starting
point, as there is a history of work analysing the connection between visual
information and grounded, predictive language processing in humans.36; 21; 37
In contrast to unimodal LLMs, such multimodal computational models are trained
on datasets that include image/sentence pairs and possess the capability to
process both images and text.

The capacity to process both visual and linguistic input by itself, however,
may provide little benefit for grounding predictive language processing if no
attention is paid to salient visual cues, effectively narrowing down the space
of upcoming semantic content. Indeed, attention to salient multimodal cues is
a key part of constructing online situation models. Evidence for this comes
from studies using some versions of the so-called Visual World Paradigm (VWP),
a classic behavioural paradigm for probing the online integration of
linguistic and visual information as indexed by eye movements.38 In a typical
VWP study, participants hear an unfolding sentence and see different
objects/scenes on a screen,17 or even in virtual reality.39 This paradigm has
robustly shown that participants’ eye movements index word prediction. For
example, just after hearing ‘the man will eat…’, participants look at a cake,
rather than non-edible distracters before the continuation of the sentence is
uttered.40 Similar studies have consistently found that when visual
information is critical for predicting upcoming sentence content, participants
tend to direct their eye movements towards the relevant visual information
even before hearing the associated word.40 This suggests that attention to
individual visual cues is linked to the online construal of situation models
during predictive language processing.41; 42; 43

Recent advances in natural language processing and computer vision have
culminated in the transformer architecture, which similarly selects relevant
information—linguistic tokens or visual pixels from the surrounding context
using an ‘attention’ mechanism.44 This mechanism is not explicitly modelled on
human attention. For example, transformers have multiple attention heads and
can simultaneously attend to many image patches at once, whereas human
attention has a more limited capacity,45; 46 though it is unclear how many
targets human attention can track simultaneously.47 Instead, the design
considerations for self-attention in transformers are primarily driven by
engineering considerations (such as enabling parallel processing) and
performance benchmarks against other models.48; 49 In fact, the degree of
alignment with human brain data for vision-only models (both convolutional
neural networks and visual transformers) is said to depend less on
architectural features and more on their training diet (i.e. the quantity and
quality of training data 50). On the other hand, visual transformers exhibit
more human-like image classification compared to convolutional neural nets
without attention.51 Tuli et al. 52 argue that this effect ‘could possibly be
explained by the nature of attention models that permits focus on the part of
the image that is important for the given task and neglect the otherwise noisy
background to make predictions’. Furthermore, text-based transformer models
have shown some similarity in attention patterns to human gaze during reading
tasks. 53; 54 However, no previous study has explored the impact of including
a transformer vision backbone within a multimodal computational model.
Specifically, there is a gap in understanding how visual attention might
impact alignment with human natural language predictions in situations that
include both visual and linguistic input (such as the VWP).

Here, we pursue three related hypotheses: (1) Multimodal computational models
can use contextual visual and linguistic information to extract sensible
semantic predictions that align more closely with human processing compared to
their unimodal counterparts basing predictions solely on textual information.
(2) The addition of a visual attention backbone to a multimodal computational
model further enhances human-like natural language predictions. (3) During
processing, the model’s attention patterns will correlate with human attention
patterns when salient visual cues for upcoming content are present. To
investigate our hypotheses, we compared predictability scores from a variety
of models with human estimates.

## Results

Figure (1, Unimodal Methods) describes how we extracted predictions from
different unimodal models and how we collected predictability estimates with
audio-based stimuli from human participants. Figure (1, Multimodal Methods)
shows multimodal versions of the same models (with and without attention for
one of the models) as well as our experiment with audio-visual-based stimuli.
In the online experiment, humans were exposed to 100 six-second audio-visual
or audio-only clips from two films, ‘The Prestige’ and ‘The Usual Suspects’.
For each stimulus type, 200 participants were shown a target word and brief
instructions (Figure 1a), followed by the six-second audio or audio-visual
clip (Figure 1b and 1e). Participants were then asked to rate, on a scale from
zero (not at all relevant) to 100 (highly relevant), how pertinent the
information in the video or audio-visual clip was for predicting the upcoming
word in the film’s dialogue (Figure 1 Unimodal Methods (c) and Multimodal
Methods (f)). Throughout the 100 trials conducted for each participant, eye
gaze was also recorded via the participant’s webcam.

Figure 1: Unimodal Methods: (First column) For the first probability measure,
both the incoming dialogue (input) and all labels in the movie (a) are encoded
by CLIP’s text encoder in the ‘text only’ version of the model (b).
Predictability is derived as the softmaxed similarity scores (over all labels)
between the upcoming label and the resulting encodings (c). (Second column)
For the second probability measure, the textual input (a) is fed directly to
LLaMA (b). Predictability is derived by pulling out the next-word logits from
the model’s forward method for all labels in the movie and applying a softmax
over them to obtain a probability distribution (c). (Third column) For the
prompt-based measure, the textual input is combined with a prompt asking to
estimate the predictability of the upcoming word, which is processed by the
model (b) and results in a direct prompt-based output measure (c). While GPT-4
and LLaMA are from the same model family, GPT-4 has many more parameters than
LLaMA. (Fourth Column) For the human measure, instructions are presented to
human participants (similar to the prompt used in the prompt-based measure)
(a) before they listen to an audio clip (b) and provide predictability
estimates on a Likert scale (c) from 0 (Low Relevance) to 100 (High
Relevance). Multimodal Methods: (First column) For the first direct
probability measure, both the incoming dialogue (input) and all labels in the
movie (d) are encoded by CLIP’s text encoder. The visual information (frame-
by-frame) is encoded by the visual transformer backbone (we used both the
‘ViT-32’ and the ‘RN50’ versions of the model) (e). Predictability is derived
as the softmaxed similarity scores (over all labels in the movie) between the
upcoming label and the resulting multimodal encodings (f). (Second column) For
the second direct probability measure, visual input was fed frame-by-frame to
the adapter layer. Textual input was fed to the LLaMA model directly (d). Both
text and visual information were then processed by the model (e).
Predictability scores were derived as the softmaxed next-word logits for all
labels in the movie’s dialogue (f). (Third column) For the prompt-based
measure, the visual input was fed as a GIF to the GPT-4 API, together with a
prompt (d). This input was processed by the model with the temperature
parameter set to zero (e). Predictability was the direct, deterministic
outcome following the prompt (f). (Fourth column) For the human measure, human
participants received instructions similar to the prompt fed to GPT-4 in the
prompt-based measure (d). Humans then watched the 6 s video clip (e) while
their eye movements were tracked through their webcam. Participants indicated
relevance on a Likert scale from 0 (Low Relevance) to 100 (High Relevance)
(f).

### Hypothesis 1: Multimodality Drives More Human-Aligned Predictions

Our first hypothesis was that multimodality enhances alignment with human
language predictions. To test this, we compared the performance of several
unimodal models against their multimodal counterparts. First, we leveraged the
text-only as well as the multimodal versions of CLIP. 55 With an estimated 250
million parameters, CLIP is a relatively small, but highly capable multimodal,
contrastive image-language matching model. Using the text-only as well as the
multimodal versions allows us to directly compare the impact of adding
multimodality to the embedding space. For CLIP, which is not designed for
next-word prediction, we determined predictability by computing probability
scores. We applied the softmax function to the model’s similarity assessments
between the provided prompts and relevant labels (Figure 1a and Figure 1d)
from the movie. This method transforms the model’s output into a probability
distribution over all possible next words (i.e., next word predictions).

In a second direct probability measure, we used the state-of-the-art model
LLaMA 56 with 7 billion parameters. LLaMA itself is only trained on text,
however, Gao et al. 57 have created a visual ‘adapter’ layer for the 7B
version, which enables visual as well as textual input to the original LLaMA
model. As LLaMA is an open-source model, we could extract predictability
measures directly by customising the model’s forward method. We extracted
predictability scores from LLaMA by feeding LLaMA’s unimodal version textual
input (Figure 1a) and computing a softmax over the next-word logits (Figure
1c) for all labels in the movie, obtained from the model’s forward method
(Figure 1b). For the multimodal version, we fed the textual input to LLaMA and
the visual (frame-by-frame) input to the adapter layer (Figure 1d). Again, we
extracted predictability scores by computing a softmax over the next word’s
logits (Figure 1f) for all labels in the movie, obtained from the model’s
forward method (Figure 1e).

This extraction of predictability estimates from the model is different to the
human experiment: rather than directly probing for next-word prediction, we
asked participants to assess the relevance of the visual-linguistic
information for predicting an upcoming word. For this reason (and because
GPT-4 is a closed model) we tested hypothesis (1) in a second way with a
prompt-based measure that closely resembled the human instructions. For the
unimodal version, we fed the model a prompt asking it to estimate the
predictability of a given upcoming word, based on provided textual information
(Figure 1a). After processing this prompt (Figure 1b) with the temperature set
to 0 (to produce deterministic outcomes), GPT-4’s output was a predictability
estimate (Figure 1c.). For the multimodal version, we fed the same prompt to
the model, providing text-based linguistic information in the prompt and the
visual content in a Graphics Interchange Format (GIF) (Figure 1d). After
processing this information with the temperature set to 0, the output was,
again, a deterministic predictability estimate for the upcoming word (Figure
1f).

Our findings indicate that alignment with human predictability estimates
significantly increased for the multimodal versions of CLIP, LLaMA and GPT-4
compared to their unimodal counterparts (Figure 2). The scatter plots in
Figure 2 show the predictability estimates of each model against human scores
(each data point represents one of the stimulus clips). Even for a strong
model like GPT-4, predictability estimates are biased in the unimodal case
(with clusters around 0). Overall, the distributions look more distributed and
human-like for all multimodal models compared to their unimodal counterparts.
Bar graphs in Figure 2 display the Pearson correlations between model and
human scores, which quantify the extent to which each model’s predictions
align with human predictability estimates. To be able to interpret these
model-based correlations against a human ceiling score, we correlated
predictability estimates (from audio-visual stimuli) of each human participant
against the $N-1$ (199) other participants and then averaged over these
correlation values for all participants (green line in Figure 3).

While unimodal LLaMA scores were not correlated with human predictability
estimates based on audio-clips (Figure 3, r = 0.03, p <0.001), multimodal
LLaMA scores were positively correlated with human predictability estimates
from video-clips (including speech) (Figure 3, r = 0.1, p <0.001). Similarly,
scores based on unimodal GPT-4 were more weakly correlated with human
predictability estimates from textual information (Figure 3, r = 0.16, p
<0.001) compared to multimodal GPT-4 scores correlated with human
predictability estimates from video-clips (Figure 3, r = 0.28, p <0.001). This
latter correlation is approaching the human ceiling (Figure 3, r = 0.41, p
<0.001), see Figure 3 green dashed line. Finally, scores extracted from the
multimodal CLIP version were positively correlated with human predictability
scores (r = 0.23, p <0.001), while there was almost no correlation between
human predictability estimates and scores extracted from the text-only CLIP
model (r = 0.05, p <0.001).

Figure 2: Results for comparing unimodal, multimodal model and human
predictability scores. (a) Average human response per audio-visual
(multimodal) and audio-only (unimodal) stimuli (Y-axis) plotted against model
response (X-Axis) for unimodal and multimodal GPT-4, LLaMA, and CLIP. For all
multimodal models, the predicted model response (black regression line, only
displayed for significant predictions) aligns significantly more with human
predictability estimates compared to their unimodal counterpart. (b)
Comparison between Pearson correlations of predictability scores derived from
multimodal and unimodal models and human predictability estimates. The human
ceiling is depicted as a dashed green line. Unimodal LLaMA scores were only
marginally correlated with human predictability estimates based on audio-clips
(light blue bar), multimodal LLaMA scores were positively correlated with
human predictability estimates from video-clips (including speech) (dark blue
bar). Similarly, scores based on unimodal GPT-4 were more weakly correlated
with human predictability estimates from textual information (light pink bar)
compared to multimodal GPT-4 scores correlated with human predictability
estimates from video-clips (dark pink bar). Finally, scores extracted from the
text-only CLIP model (light orange were marginally correlated with human
predictability, while there was a high positive correlation between human
predictability scores and the multimodal CLIP model (dark orange). Error bars
represent 95% confidence intervals. Stars indicate significance.

### Hypothesis 2: Transformer Attention Drives Human-Like Predictions

Our second hypothesis was that attention drives alignment between
computational models and human natural language predictions. We tested this
hypothesis with the multimodal predictability scores derived from the model
CLIP. 55 We used CLIP, because it exists in two versions: with a visual
transformer51 backbone (ViT-32) and with a visual convolutional58 backbone
(RN50). While the ViT-32 version of CLIP includes a visual attention
mechanism, the RN50 version does not. Directly comparing how well either
version aligns with human predictability estimates (as indicated in Figure 1,
Multimodal Methods) allows us to establish a role for attention while keeping
the training diet and the model’s intrinsic dynamics constant.

Results from comparing the multimodal CLIP models with and without attention
to human predictability estimates indicate no significant difference between
the models overall (CLIP-ViT32: r = 0.23, p <0.001; CLIP-RN50: r = 0.25, p
<0.001). The scatter plots in Figure 2 (a) display each model’s predictions
against human predictability estimates extracted from audio-video stimuli. Of
particular interest, however, was the alignment between the model and humans
at the top 25% of model scores. In these video clips, the model presumably
deemed the visual cues highly relevant for predicting upcoming semantic
content. This is displayed in Figure 3 as the dashed box with grey background.
The zoomed-in scatterplots show alignment for just the top quartile. Pearson
correlations for this top quartile indicated an increase correlation for the
visual transformer version (ViT-32) of CLIP (r = 0.46, p <0.001) compared to a
decrease for the convolutional net version (RN50) of CLIP (r = 0.19, p
<0.001), see the bar plot in Figure 4. The ViT-32 CLIP correlation was
approaching the top quartile human ceiling (Figure 4, r = 0.61, p <0.001)
which was calculated by correlating each participant’s top quartile scores
against predictability from the $N-1$ (199) other participants and then
averaging across these scores.

Figure 3: Results for comparing predictability scores in the top quartile of
model predictability for CLIP with (ViT-32) and without (RN50) attention. (a)
Average human response per video (Y-axis) plotted against model response
(X-axis) for the CLIP model with and without attention. Top quartile of model
scores is surrounded by a dashed grey box. A zoomed-in scatterplot displays
human-model prediction alignment only for this top quartile (on the right).
Regression lines are only displayed for significant predictions. Stars
indicate strength of significance. (b) Bar plot comparing correlations of
predictability scores for the respective top quartiles dervied from multimodal
and unimodal versions of both LLaMA and GPT-4 with human predictability
estimates. The human ceiling is displayed as a green dashed line. LLaMA scores
were the lowest (dark blue), followed by GPT-4 (dark pink), and CLIP-RN50
(yellow). The highest score by far was obtained from the CLIP-ViT32 version
with a visual attention mechanism (dark orange). Error bars indicate 95%
confidence intervals.

### Hypothesis 3: Model Attention Patches and Human Eye Tracking Overlap

Given that looks to available referents have been argued to support language
processing,40 we also wanted to understand when and how human visual attention
indexed by eyegaze overlaps with the model’s visual attention. To this end, we
tested our final hypothesis by directly comparing attention weights from the
model’s visual attention layers to human eye tracking data collected during
the online task. Although we collected output from all layers, the late-
intermediate layers of transformer models are thought to synthesize the
earlier layers’ processing into a comprehensive output.59; 60 For example,
previous studies suggest that the best semantic features for predicting brain
responses to natural language can be extracted from late-intermediate layers
(especially layers 9 and 10 in models like GPT-2, which usually have around 12
layers).61; 62; 63; 64; 65 Therefore we expected attention weights from these
layers to correlate most significantly with human eye tracking.

To compare CLIP’s visual attention matrices to human attention data from eye
tracking, we analyzed the attention matrices for each of the frames from the
video clips used in our experiment. These matrices represent pixel importance
in predictability scores for the model and the fixation duration for humans.
We averaged these matrices over 400ms segments, considering the natural
latency in saccades (250–400ms),66; 67 the relevance of this time frame for
predictive language processing studies using EEG,68; 69 and the average word
length in English being around 400ms.70 Unlike the widespread distribution of
visual transformer attention, human attention tends to be more focused. We
therefore adapted our analysis by thresholding the model’s attention and
applying Gaussian smoothing to both sets of data, choosing parameters to
highlight differences between actual human attention and a random
distribution. We then created probability distributions from these heatmaps
and quantified alignment using Spearman correlation. A human ceiling
correlation value for each of the 15 heatmaps per video clip was determined by
correlating each participant’s probability distribution with the $N-1$ (199)
other probability distributions for this heatmap and taking the average of
these correlations for each of the 15 segments per video clip. We find that
averaging across all video clips, model attention approaches 40% of the human
ceiling - see Figure 4(a).

There are several potential reasons for this correlation not being higher
across all videos. For one, model and human attention are computed under
different circumstances. Because humans know the word ahead of time, they are
more likely to attend to relevant visual information (e.g. focusing on the
referent when it is present). CLIP on the other hand encodes both modalities
(image and text) separately, so visual attention is computed taking language
into account only to the extent that the visual encoder is biased towards
language by the contrastive training task of matching images and captions.
Therefore, the fact that we do find overlap is already surprising and we do
not expect a close to perfect overlap. Another reason could be that, while
human-model correlations were sometimes negative, human-human correlations
were not. We hypothesised that this was because in those video clips in which
visual information was irrelevant for prediction, human and model attention
were biased towards different areas. To test this hypothesis, we focused on
those segments in which correlations between model and human probability
distributions were negative. We found that in these cases, the model focused
more on peripheral areas, while humans tended to centre their attention
towards the middle of the screen. Complementary to this analysis, we also
found that human-model correlation came closer to the human ceiling in layers
9 and 10 when considering only the video clips in the top quartile of model or
human predictability judgements, see Figure 4 (a). Finally, when eye tracking
was highly correlated within human participants, the model’s attention reached
up to 90% of the human ceiling (in layer 10). These results suggest that human
and model attention patterns align when salient visual cues are present.

Eyetracking alignment in video clips based on human-model predictability
correlations did not approach the human ceiling closer than 70% in any layer
(see figure 4a), even in video clips where the word was highly predictable.
One reason could be the high variance in correlation values between segments
within some video clips (this was 0.13 on average across segments). For
example, in the video clip for the word ‘sunglasses’ (see Figure 4b), the
correlations from layer 9 were positive in segments 1-5 when the referent was
present (peaking at r = 0.22 in segment 4), but dropped into negative
correlation values (as low as -0.27) when no sunglasses were visually present
anymore. The average correlation for this video clip was therefore around 0,
even though some segments were highly correlated. Indeed, even though the
average correlation between human- and model attention patterns was low, this
video clip received a high average predictability judgement from human
participants (P = 0.83). We therefore conducted a further, more fine-grained
analysis focusing only on those video clips in the top quartile of model and
human predictability judgements.

In a follow-up experiment, 100 participants were shown each of the 15 segments
for each video clip (resulting in 375 screenshots overall) and given
instructions to indicate whether or not the label was present in the segment.
We ran a linear regression model on these responses, including the
participant’s binary response for each segment as a categorical variable to
test if the presence of the referent could predict the correlation value
between human eye tracking data and model attention patterns. Our results show
that there was a significant (p <0.001) linear relationship between the
Spearman correlation and the absence/presence of the referent. Namely, when
the referent was absent, correlation values were on average 0.23 points lower
than when the referent was present (Figure 4c).

Figure 4: Alignment between human gaze and model attention. (a) Layerwise
percentage of alignment (model-human correlation divided by human-human
correlation). Average alignment for all segments per layer is displayed as a
blue line (with blue dots). Average alignment per layer for all segments in
video-clips that are in the top quartile of human predictability estimates are
displayed as orange line with orange crosses. Average alignment per layer for
all segments in video-clips that are in the top quartile of model
predictability estimates are displayed as green line with green squares.
Average alignment per layer for all segments in video-clips that are in the
top quartile of within human eye tracking alignment are displayed as a red
line with red crosses. Overall, average overlap for all videos and overlaps
based on predictability estimates follow similar trends. Layer three reaches
between 40 and 50% of the human ceiling. Attention patches from layers nine
and ten are most correlated with human eye tracking, reaching 70% of the human
ceiling in those videos where human predictability or model predictability
were high. However, when human attention matrices are highly correlated, model
attention patches are also highly correlated with human eye tracking, reaching
80 and almost 90% of the human ceiling in layers 9 and 10 respectively. (b)
Analysis of whether presence/absence of referent influenced correlation scores
for layer 9. Using human ratings collected from 100 participants (red dots),
absence or presence of the referent significantly predicted correlation value
in a linear regression model (with random intercept for participant ID). In
particular, the correlation between human eye tracking and model attention
patterns was, on average, 0.2339 greater when the referent was present. (c)
One example of evolving alignment in layer 9 between model attention patterns
(red) and human eye tracking (green) over one video clip, separated into 15
segments. When salient visual information was present (segments 2, 3, and 4),
correlation was positive. However, when the referent was not present,
correlation dipped into negative values (e.g. segment 5, r = -0.16).

## Discussion

We investigated the integration of visual-linguistic information during
predictive language processing in state-of-the-art, multimodal computational
models and compared these processes to behavioural data from humans. We found
that including multimodality and visual attention as architectural features
significantly increased correlation with human predictability scores.
Furthermore, we found significant overlap between the model’s attention
patches and human eye tracking during the presence of salient visual cues
(important for predicting upcoming words). We propose that the ability of
multimodal models to conjoin visual and linguistic information can mitigate
the inherent ambiguity of naturalistic language input and thereby facilitate
more human-like semantic predictions.16 Below we discuss differences between
models, the role of attention, and the potential of adding more modalities to
the model’s embedding space.

### Differences Between Models

As for comparisons between models, LLaMA scores remained relatively low, even
for the multimodal version (r = 0.13), while the prompt-based GPT-4 measure
approached 75% of the human ceiling. Of course, one reason for this difference
could be the fact that GPT-4 is estimated to have 200 times the number of
parameter size of LLaMA 7B. However, the CLIP model (with only 250 million
parameters) was also more aligned with human scores in both its ViT-32 (r =
0.23) and RN50 versions (r = 0.25). Therefore, size of the model alone cannot
account for these differences. Instead, one decisive advantage of the prompt-
based measure could be that it works on a more dynamic, GIF-based input, and
not a singular frame-by-frame basis. Humans may be sensitive to dynamic
information evolving across frames that LLaMA does not have access to; similar
divergence between static and dynamic stimuli has been found in face
recognition networks. 71 However, it is not clear how GPT-4 exactly processes
GIF- or image-based input. Being likely trained on text and being based on the
GPT architecture, it is unlikely that GPT-4 can meaningfully make use of
temporal dynamics in the same way as, e.g. vision models that use temporal
convolutions to learn dynamic features across frames.72 A reason for CLIP’s
high performance despite its modest parameter count, could be that in the case
of LLaMA, multimodal knowledge is injected post-training through the adapter
layer. It is a remarkable feat of LLaMA, which is trained on text only, that
it can process visual information in this way. Yet, CLIP’s training process
includes a much larger dataset of combined image and text from scratch,
creating an entangled embedding space of both visual and text-based
information, which may be better able to capture abstract correlation
structures between image and text.73

### The Role of Attention

While multimodality increased alignment across the board, visual attention was
only relevant in cases where salient visual cues were present. For this top
quartile of model predictability estimates, the correlation between CLIP and
human predictability estimates doubled (from r = 0.23 to r = 0.46), whereas
the CNN version of CLIP did worse than before (r = 0.19 rather than r = 0.28).
One reason for this could be that compared to CNNs, where inductive biases
like locality, two-dimensional neighborhood structures, and translational
equivariance are inherent in the model’s architecture,74 ViT has barely any
image-specific predispositions. In ViT, processing is global, with self-
attention layers engaging the entire image rather than focusing on local or
neighborhood-specific features. Being free from the constraints of fixed
structural biases and grounded in a more flexible, globally oriented
perspective, such an architecture may offer a more congruent alignment with
human integration processes of multimodal information in naturalistic, dynamic
contexts such as videos.

We found a similar jump in correlation for the human ceiling correlation,
which increased from r = 0.41 to r = 0.61 for the top quartile. This jump can
be explained by research testing eye tracking during language processing in
naturalistic conditions, which suggests that eye tracking indexes predictive
processing most reliably when the referents are situated in a joint attention
space.75 When the referent is not present, humans make more diverse judgments
about what they consider relevant visual-linguistic integration for predicting
the upcoming word.76 Therefore, there is also more variance in their
predictability scores, leading to weaker correlations within human judgments.
Similarly, CLIP attention patterns are more evenly spread out when no salient
visual cues are present, leading to more variance in predictability estimates
and weaker human-model alignment.

Indeed, we found significant and quantifiable overlap between the model’s
attention patches and human eye tracking. This finding is surprising, as
despite its nomenclature, the ‘attention’ mechanism in transformers is not a
replica of human attention. Rather, it is a purely mathematical method for
assigning weights to different parts of the input based on perceived
relevance.44 Furthermore, CLIP encodes images and text separately. Therefore,
the model’s visual attention patterns are only influenced by language to the
extent that the visual encoder (because of the contrastive image-language
matching task the model is trained on) is biased towards aspects of the image
that are relevant for matching language embeddings. In contrast, human
attention guides human perception online77; 78 and human attention is
influenced by a wider range of contextual factors, including past
experiences,79 current goals,80 or emotional states,81 and operates over
different timescales82 and modalities.47 Future models could be more aligned
with human processing if the visual attention mechanism received online input
from linguistic information. This could help the model to exploit predictive
associations between visual and linguistic information more effectively.
Indeed, the beneficial effect for learning predictive relations between events
online through attention has been reported in humans (see for example Custers
and Aarts 83). Finally, transformer self-attention is not particularly
plausible from a biological point of view compared to a classical recurrent
architecture, as the brain also relies on recurrence for language processing
(rather than encoding many words in parallel).49 Future work should compare
human eye tracking during naturalistic language processing to both multimodal
transformers and recurrent architectures to test whether biological
plausibility translates into behavioural alignment.

Another remaining open question relates to the conditions under which
attention patterns between multimodal computational models with a transformer
vision back-bone and humans align. Our results show that overlap between
attention patterns becomes significant when the visual input includes the
referent of the word to be predicted (see Figure 4 (c)). In cases where visual
information is not relevant, humans tend to centralise their eye-gaze. This is
a documented effect in the visual world paradigm 84 and has several reasons,
including reduction of cognitive load through efficient allocation of
attentional resources.85 The model, on the other hand, tends to focus on
peripheral areas when no relevant visual information is present.

### From Language Toward Embodiment

A criticism of large language models (LLMs) like GPT-4 is that they can never
adequately replicate human language processing because they are not
embodied.86 On this view, human cognition and language are deeply intertwined
with sensory experiences and interactions with the environment.87 This
grounding in the physical world is believed to be crucial for genuine
acquisition and understanding of language.88 However, by integrating various
sensory modalities, such as visual or auditory data, into the models’
training, LLMs can gain some aspects of the context and environment-dependent
understanding characteristic of human cognition. As we have shown, adding just
one modality already allows LLMs to process and generate language that is more
aligned with how humans perceive and interact with the world, leading to more
nuanced and contextually appropriate responses.

However, it is also important to note that even with more multimodal inputs,
large language models would still lack direct, experiential interaction with
the environment, which some theorists argue is fundamental to true human-like
understanding and language.89 Therefore, it has been argued that truly human-
like AI can only be achieved in a bottom-up process, starting with basic
spatial interactions in the world.90 Building on multimodal advancements in
LLMs, however, the PaLM-E model 91 may exemplify the integration of the kind
of data that allows for a simulated experiential interaction with the
environment. By incorporating sensor data from robotic agents into an existing
LLM, PaLM-E extends beyond traditional text and image processing, engaging
with real-world robot states and environmental interactions. This innovative
approach offers a glimpse into future AI capabilities where computational
models may process a richer array of sensory inputs, akin to human perception.
PaLM-E’s state-of-the-art ability to adapt across various robotic platforms
and handle different modalities like images and robot states exemplifies the
potential of AI systems to operate in more complex, real-world scenarios,
moving closer to the nuanced way humans interact with their environment.
Future research continuing on this path may offer an alternative route to
bottom-up engineering of human-like AI.

An additional aspect of adding multimodality to the embedding space is
learning efficiency. In human cognition, multimodal inputs can facilitate
faster and more efficient learning or acquisition of language and context
understanding.16 This is particularly notable when contrasted with LLMs, which
require thousands of human years’ worth of data reading for effective
learning. Multimodal models may therefore not only align better with human
processing but also potentially require less data for training to achieve a
similar level of processing.

### Conclusion

The fact that current AI models are trained with a very simple objective
(next-word prediction), may suggest that their high performance is based on
the learning of abstract co-occurrence relationships between word tokens or
clusters while lacking genuine understanding.92; 93 Here we provide
behavioural evidence that multimodal attention models may be able to leverage
contextual information to predict upcoming words in a way that aligns more
with humans. As mAI is becoming ubiquitous in our daily lives, it is vital to
understand how these models process input and what architectural features
support alignment with human processing. These insights can inform responsible
development of potentially grounded AI systems and make them more useful,
understandable, and reliable - important aspects for real-world applicability
(see for example Tankelevitch 94). Based on our findings, we suggest
continuing the push towards including more modalities and sensorimotor,
experience-based data into the training sets of large language models to
achieve richer, and more human-like conceptual understanding in AI systems. A
wider variety of mAI could also enable the modeling of human behavior in
ecologically valid experimental settings. Understanding how AI systems align
with human cognitive processes in naturalistic, dynamic, and multimodal
environments not only advances AI technology but furthers our understanding of
the mechanisms underlying human cognition.95

## Methods

### Online behavioural study

#### Participants

For the main study, 200 participants were recruited via Prolific
96(https://www.prolific.co) and for the follow up study another 100
participants were recruited under the exact same conditions. To be included in
the study, participants had to be aged 18-50 (inclusive) and fluent in English
(having spoken English regularly for at least the past 5-10 years). Inclusion
criteria included giving consent to have their camera on as well as having
their eyes tracked for the duration of the experiment. We also excluded any
participants from participating who had previously seen either of our two
movies. All participants finished the study and produced usable data, none had
to be excluded post data collection. The final participants for the main study
(n=200) were 43% females (n = 86) with mean age of 33.88 years (range = 23-48
yrs, SD = 13.74 yrs). Countries of origin were the UK (n = 123) and the US (n
= 77). The final participants for the follow up study (n=100) were 46% female
(n = 46) with mean age of 29.8 (range = 21 - 42 yrs, SD = 11.37). Countries of
origin were the UK (n = 67) and the US (n = 33). Participants in both studies
were paid £9.50/hour. The study was approved by the UCL Ethics Committee.
Additionally, all participants provided written informed consent before
participating in the experiment.

#### Materials

100, 6s movie clips from the films ‘The Usual Suspects’ and ‘The Prestige’
were chosen respectively. All words in these movies were previously annotated
using automated approaches with a machine learning based speech-to-text
transcription tool from Amazon Web Services (AWS;
https://aws.amazon.com/transcribe/) and later corrected by human annotators.
This alignment between text and speech meant that the model and humans really
received the same dialogue-based information.

To choose the words to be predicted by human participants in our study and
later by the model, all function words (as defined in this list:
https://semanticsimilarity.wordpress.com/function-word-lists/) from the movies
were excluded and the 6s scenes leading up to but excluding each word were
extracted – this amounted to a few thousand words per movie on average. A
length of 6s was chosen, as this constituted a good balance between containing
enough information while not being too confusing. Furthermore, this meant that
it was possible to conduct a large number of trials without causing too much
fatigue in participants. For our final study, only 50 scenes were included
from each movie, leading to 100 clips, totalling 600s (10mins) of video
material for each participant. This meant that a further subset of words had
to be chosen. As the goal was to probe visual-linguistic integration, a final
set of 100 video clips was pre-selected in such a way as to ensure the
generation of a stimulus set containing both videoclips in which the visual
information was highly relevant for processing and videoclips in which it was
not. To this end, each frame was captioned in each video scene with the CLIP
model and the text-based transformer BERT97 was used to calculate the semantic
similarity between these captions and the word of interest. The maximum
similarity score derived from this comparison was used for each scene as a
measure of how relevant the preceding scene was for the upcoming word. Then,
from each movie, 25 scenes were chosen from the low end of the distribution of
these scores and 25 scenes from the high end of the distribution of these
scores - resulting in 100 movie scenes in total, 50 from each movie. These
were validated against the later collected human scores, showing that around
75% of categorizations into ‘high’ or ‘low’ relevance matched with human
judgements. The final set of stimuli and the code for determining them, can be
found at https://github.com/ViktorKewenig upon publication.

After the main experiment was conducted, 15 frames were extracted from each of
the 25 video clips that received the highest predictability ratings by human
participants for our follow up study.

#### Procedure

Both the main study and the follow up study were made with the Gorilla task
builder,98 and the structure of both studies was as follows: after recruitment
through Prolific, participants were directed to the study website.
Participants were briefed on the aims of the study and asked for consent.
Following the collection of demographic information, participants were
instructed to complete the experiment in an environment with minimal
distractions and with their phones turned off, wearing headphones. For the
main study, before starting the experimental task, participants were run
through the native eye tracking calibration for web cameras on Gorilla. The
eye tracking software was calibrated three times in total – before the
practice trials, after the practice trials, and during the halfway break – to
ensure more reliable measurement even with a shift of head position during the
task. Both experimental tasks consisted of 3 practice trials and 100
experimental trials.

During the main study, in each trial, participants were first shown the word
of interest corresponding to the upcoming scene and asked to pay careful
attention to the upcoming video clip and think about how relevant that clip
would be for predicting the word. After pressing a key to continue,
participants were then shown the 6s movie scene. The video was shown in a
window at the centre of the screen at a resolution of 720x1280 pixels (25
frames per second). Participants then used a 100-point slider (‘Low Relevance
to ‘High Relevance) to indicate how relevant the video clip was to the
previously presented target word. This was repeated for all 100 trial scenes
and took participants an average of around 30 minutes to complete. A self-
paced resting break was provided halfway through the trials. After completing
the experimental trials, participants were debriefed and paid for their time.

The goal of the follow up study was to understand whether human participants
judged a given referent as present or not present in each of the 15 segments
per video clip. For this purpose, participants were first shown the word of
interest corresponding to the upcoming segment and asked to pay careful
attention to the upcoming image and think about whether they could see the
word in the image or not. The image was shown in a window at the centre of the
screen at a resolution of 720x1280 pixels. Participants then had to select a
check-box, indicating that the referent was present or not, before proceeding
with the experiment. This was repeated for the 375 segments of the video clips
which received the top quartile of human predictability judgements. A self-
paced resting break was provided halfway through the trials. After completing
the experimental trials, participants were debriefed and paid for their time.

#### Human Measurements

#### Estimates of Predictability

Human estimates of the target word’s predictability were collected using an
interactive slider. This measure of likelihood is intended to reflect the
participant’s intuitive understanding of the relationship between antecedent
linguistic context, the visual cues in a scene and the subsequent semantic
content. As we expected these intuitive judgements to vary between
participants, outliers (any rating above or below 2 standard deviations) were
not excluded from the analysis. A human ceiling score was calculated by
correlating predictability estimates from each human participant against the
$N-1$ (199) other participants and then averaging over these correlation
values for all participants. This ceiling score allowed us to compare model
scores against a baseline and they also functioned as an estimate of the
reliability of individual humans.

#### Eye Gaze

The gathering of human eye tracking data was done via the Gorilla eye tracking
tool (version 2.0). This tool exploited the webcams of each participant to
capture eye movements. The collected data, sampled at a rate of 60 Hz, was
individually stored in an Excel file for each video-clip and each participant.
Eye-tracking data were analyzed using a custom Python script that made ample
use of the packages ‘NumPy’99, ‘SciPy’100, and ‘Pandas’101. The script first
read and compiled the X and Y gaze coordinates from each participant’s eye
tracking data files, discarding any data from calibration tasks. Then, two-
dimensional attention heatmaps were generated based on gaze coordinates for
each participant and their corresponding stimulus. 3% of all heatmaps
contained only zeros and were excluded from further analysis. These raw
heatmaps were averaged for each segment (every 400ms) for each stimulus and
participant and then saved as NumPy files for subsequent analysis. The
averaged heatmaps serve as a representation of the aggregate attention
directed by participants towards specific visual features in each segment.

#### Estimates of Reference Presence/Absence

Human estimates of whether a referent was present or not (in each segment)
were collected using two check-boxes (one for ‘no’, one for ‘yes’). This
measure was intended to reflect the participant’s more fine-grained
understanding of the relevance of visual cues in each video clip. We did not
expect these judgements to vary greatly between participants, and indeed the
agreement rate between participants was 80% or above.

#### Model Measurements

#### LLaMA Predictability

LLaMA’s unimodal version encodes and processes the textual input. A softmax
over the next-word logits was computed for all labels in the movie. Logits are
essentially the raw predictions that the model generates for the upcoming
word, before they are converted into a more interpretable form, such as
probabilities, through the softmax function. These logits were obtained by
slightly altering the model’s ‘forward’ method and indexing into the final
word logits. For the multimodal version, the textual input was processed by
the frozen LLaMA model, whereas the visual (frame-by-frame) information was
sent to the adapter layer. LLaMA adapter uses a CLIP-based encoding for the
visual information and then projects this encoding into an embedding space
that can be processed by LLaMA. Again, predictability scores were extracted by
computing a softmax over the next-word’s logits for all labels in the movie,
obtained from the model’s ‘forward’ method.

#### GPT-4 Predictability

To obtain unimodal GPT-4 predictability scores, OpenAI’s closed API was used.
The human instructions were used as prompt-based input to the API, together
with the textual input. Temperature was set to 0, so as to obtain a
deterministic outcome. In order to obtain multimodal predictability scores,
the API was also used, except this time with the “GPT-4-V” (the visual)
version of GPT-4. This allowed the uploading of the video scene as a as a
‘Graphics interchange format’ (GIF) (which is a more dynamic representation of
the video information compared to singular, frame-by-frame screenshots),
together with the human instructions and the textual input (which human
participants would listen to as the dialogue). Again, the temperature was set
to 0 in order to obtain a deterministic outcome.

#### CLIP Predictability

The calculation of CLIP predictability scores for target words was carried out
through a combination of image and text feature comparison with a custom
Python script making use of the packages ‘PyTorch’102 and the Hugging Face
‘transformer’ library.48 For the unimodal version, the textual input was
tokenized and processed by the text-only version of CLIP. For the multimodal
version, each frame of each movie clip was individually preprocessed by the
standard, pre-trained CLIP preprocessor, transforming the visual data into a
digestible form for the CLIP model’s image encoder. This encoder generates a
corresponding image feature vector, which provides a compact representation of
the visual content in a form that can be easily compared with textual
information. We pair each image feature vector with an array of text feature
vectors. These text features vectors are derived from both the dialogue of
each scene, represented as text prompts, and the collective word set present
in the movie. The CLIP model tokenizes and encodes these text inputs,
converting them into a format that mirrors the structure of the image feature
vectors. For the purpose of our study we used both the the ‘clip-vit-base-
patch32’ version and the ‘clip-RN50’ versions of the model.103

The calculation of the ‘predictability score’ was carried out by comparing
these two feature sets, visual and linguistic (or linguistic and linguistic in
the unimodal comparison). For the unimodal version, the dot product of the
feature input of the textual input, and all labels in the movie was computed.
Similarly, by computing the dot product of the image feature vector and each
text feature vector in the multimodal version, a raw similarity measure was
derived. This raw similarity measure was then passed through a softmax
function, refining it into a more interpretable similarity score. This score
quantified the likelihood of the upcoming word, given all words in the movie,
by matching each word to the visual content of each associated frame (or
textual input for the multimodal version).

#### Differences in Predictability Estimates

There are some commonalities and differences between our different model-based
predictability measures and how we extracted predictability scores from
humans, however. A noteworthy difference between LLaMA and human
predictability estimates is that rather than directly probing for next-word
prediction, participants were asked to assess the relevance of the visual-
linguistic information for predicting an upcoming word. This process is
slightly different to LLaMA’s computations, which determined the likelihood of
the next word directly based on textual or multimodal input.

The reason we chose this different task for the human experiment is that
predicting the next word from a six-second movie scene is inherently difficult
and might not have yielded usable results with human participants. To
alleviate concerns connected to these differences, we used the human
instructions for our prompt-based measure with GPT-4 (though prompt-based
measures may not be an adequate substitute for direct probability measurements
-see Hu and Levy 104).

The CLIP model on the other hand generated a quantitative estimate of the
likelihood of each word by comparing the feature vectors of the visual content
in a frame with feature vectors of the associated linguistic context. The
predictability measures extracted from CLIP and human participants therefore
rely on a similar foundational principle: considering the similarities within
the presented visual-linguistic information and understanding how these
similarities influence the likelihood of the upcoming target word. This
integration is an inherent part of human language comprehension, combining
visual stimuli with linguistic expectations based on past experiences and
world knowledge.

#### CLIP Attention Maps

To explore the qualitative role of attention in the alignment between CLIP and
humans, we compared the matrices derived from CLIP’s visual attention
mechanism to the human attention matrices obtained from eye tracking data,
examining each frame of every video clip. The CLIP model uses 12 transformer
layers, each with 16 attention heads, yielding a total of 192 individual
attention maps for each input. For our analysis, the attention weights of all
layers were utilised. All attention maps were reshaped to match the resolution
of the human eye tracking heatmaps (the 1280 x 720 pixelrate of both movies).
The CLIP model operates on the principle of dividing the input image into a
grid of patches. Each patch represents a segmented portion of the image, with
the model’s attention heads focusing on these discrete units rather than
individual pixels. This patch-based approach allows the model to efficiently
process and interpret the visual information by focusing on salient image
segments. CLIP does not analyse the full-sized image at the original pixel
resolution of 1280x720. Instead, the model downsamples the image to 224 x 224
pixels, which are then divided into the aforementioned patches. A two-
dimensional ‘attention heatmap’ was then generated for each attention head by
reshaping the patch-scale attention weights back into a square matrix of the
original image size (1280x720). Each cell in this matrix represented a region
of the input, and its value indicates the amount of attention the model paid
to that region. The attention heatmaps were then averaged across all attention
heads to create a single attention heatmap for each movie clip, similar to the
averaged heatmaps generated from the human data.

Attention Matrices, Thresholding and Averaging. In these 2-D matrices, each
entry denotes the significance of a specific pixel in the visual input when
generating a predictability score on the model side, and the duration a
participant spent observing that specific pixel on the human side. To
facilitate our analysis, we averaged attention matrices from CLIP and human
participants across every ten frames, yielding 15 averaged attention matrices
per video for each participant and the CLIP model. Another reason for choosing
to average the 6s video-clip into 15 segments (400ms each) is a fundamental
difference between human and model processing. Humans do not instantly react
to a stimulus; rather previous research on the visual world paradigm has
estimated the saccade response to take between 100–250ms depending on the
richness of the stimulus. 105; 106 As human gaze will therefore be lagged
around that time for reaction to a new object or scene (while the model’s
attention patterns are near instantaneous), an average over 400ms was
considered enough to smooth out that difference. Furthermore, 400ms is a
relevant time-window for studies on predictive language processing, as the
N400 EEG signal related to surprisal (or prediction error) can be measured
400ms after word-onset. 69

Another difference between attention in humans and visual transformers, is
that visual transformer attention is widely distributed across the visual
input, while human attention is relatively sparse (eye tracking data yields a
single predicted pixel every 10ms). For this reason, we thresholded the model
attention at a conservative 15th percentile of top values, and applied
Gaussian smoothing to both model and human heatmaps. The sigma value in the
Gaussian smoothing process determines the extent of smoothing and was chosen
as that which maximises the difference between the human attention
distribution and a null distribution. The null distribution was generated by
randomly redistributing the attention values across human heatmaps. We chose
the sigma value in this way so as not to bias later comparisons between human
and model attention patterns.

Calculating Eyetracking-Attention Alignment. A probability distribution over
the averaged and smoothed heatmaps was calculated by dividing each value in
each heatmap by the total sum of values. Alignment between human eye tracking
and model attention was quantified as the Spearman correlation between these
probability distributions (15 for each of the 100 video clips). Since the
Spearman correlation is based on ranks rather than actual values, it is less
sensitive to outliers compared to the Pearson correlation. This is
particularly useful for attention heatmaps, where there may be regions with
exceptionally high or low attention that could skew the results of parametric
correlations. A human ceiling correlation value for each of the 15 heatmaps
per video clip was determined by correlating each participant’s probability
distribution with the $N-1$ (199) other probability distributions for this
heatmap and taking the average of these correlations for each of the 15
segments per video clip.

Focus of Attention when Visual Cues Were Irrelevant. To understand whether
humans and the model focussed on different areas in the input when visual
information was not relevant, we only considered those video clips in which
the model-human correlations were negative. We defined a rectangle (640 x 360,
half of all pixels) around the central pixel of the probabiltiy distributions
of these segments as the central region and categorised the rest of the pixels
as periphery. Next, we took the difference between the central and periphery
areas for all probability distributions. A positive difference indicated that
attention was centrally biased, whereas a negative difference indicated that
attention was biased towards the periphery of the probability distribution. A
t-test between all difference values on the model side and all difference
values on the human side indeed suggested that the model tended to focus on
periphery areas of the visual input, whereas humans tended to focus on central
areas (t = -12.7, p <0.001).

### Analyses

#### Comparison Between Unimodal Predictability, Multimodal Predictability,
CLIP Predictability, and Human Predictability

Human and model predictability scores were compared with simple Pearson
correlation using the SciPy100 package. The human ceiling correlation was
determined by correlating predictability estimates of each human participant
with the $N-1$ (199) other participants against each other and averaging
Pearson correlations.

#### Comparison Between CLIP Attention Weights and Human Eye Tracking

Human- and model attention matrices were 1280x720 pixels, where each vaue at
each pixel represents how much attention was paid to this pixel during each
400ms segment. Overlap between CLIP and human eye tracking data was calculated
by comparing the probability distributions over both model- and human heatmaps
for each segment (1500 segments in total). Probability distributions were
calculated by dividing each value in the heatmaps through the sum of all
values. Each participants’ probability distribution for each segment was
correlated with the model’s corresponding probability distribution using
spearman rank correlation. A human ceiling was constructed by correlating each
participant’s probability against that of the $N-1$ other participants and
then averaging these correlations across participants (resulting in one human-
ceiling correlation value per segment). Overall overlap was calculated by
averaging all correlation values for human-model comparisons and for the human
ceiling across segments. Overlap for the top quartile was calculated by only
averaging correlation values for those segments in the top 25% of video clips
that received the highest predictability scores from the model and human
participants.

#### Linear Regression to Further Understand the Nature of Overlap

To gain a more fine-grained understanding of when and how human- and model
attention patterns overlap, the human measurements of referent
absence/presence were used in a linear regression model to predict the
correlation values between human- and model attention heatmaps. The model to
predict the correlation value between attention patterns included the human
ratings of absence (0) and presence (1) as independent categorical variables
as well as a random intercept for participant.

### Data and Code Availability

All the data used in this study will be publicly available upon publication
under: CLIPPredict. All code for data processing and analysis will be publicly
available upon publication under the following GitHub repository:
github.com/ViktorKewenig.

### Contributions

VK: conceived the study, secured funding for the online study, created stimuli
and led the online experiment, wrote code for analyes, conducted data
analyses, wrote and edited the paper. AL: helped conceptualise analyses and
edited the paper. SAN: helped conceptualise analyses and edited the paper. CE:
helped lead the online study and edited the paper. QLE: helped lead the online
study and edited the paper. RA: helped lead the online study and edited the
paper. GV: provided stipends for VK and CE, helped conceptualise the study,
and edited the paper. JIS: helped conceptualise the study and edited the
paper.

### Acknowledgments

This work was supported in part by the European Research Council Advanced
Grant (ECOLANG, 743035); Royal Society Wolfson Research Merit Award (WRM R3
170016) to GV; and Leverhulme award DS-2017-026 to VK and GV. Figures were
created with BioRender.com

## References

  * 1 Melanie Mitchell and David C. Krakauer.  The debate over understanding in AI’s large language models.  120(13):e2215907120.  doi: 10.1073/pnas.2215907120.  URL https://www.pnas.org/doi/10.1073/pnas.2215907120.  Publisher: Proceedings of the National Academy of Sciences.
  * Mahowald et al. 2023 Kyle Mahowald, Anna A. Ivanova, Idan A. Blank, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko.  Dissociating language and thought in large language models: a cognitive perspective, January 2023.  URL http://arxiv.org/abs/2301.06627.  arXiv:2301.06627 [cs].
  * Liu et al. 2023 Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, and Nanning Zheng.  BrainCLIP: Bridging Brain and Visual-Linguistic Representation via CLIP for Generic Natural Visual Stimulus Decoding from fMRI, February 2023.  URL http://arxiv.org/abs/2302.12971.  arXiv:2302.12971 [cs].
  * Kauf et al. 2022 Carina Kauf, Anna A. Ivanova, Giulia Rambelli, Emmanuele Chersoni, Jingyuan S. She, Zawad Chowdhury, Evelina Fedorenko, and Alessandro Lenci.  Event knowledge in large language models: the gap between the impossible and the unlikely, December 2022.  URL http://arxiv.org/abs/2212.01488.  arXiv:2212.01488 [cs].
  * Zhang et al. 2022 Li Zhang, Sixiao Zheng, Jiachen Lu, Xinxuan Zhao, Xiatian Zhu, Yanwei Fu, Tao Xiang, and Jianfeng Feng.  Visual Representation Learning with Transformer: A Sequence-to-Sequence Perspective, July 2022.  URL http://arxiv.org/abs/2207.09339.  arXiv:2207.09339 [cs].
  * Michelmann et al. 2023 Sebastian Michelmann, Manoj Kumar, Kenneth A. Norman, and Mariya Toneva.  Large language models can segment narrative events similarly to humans, January 2023.  URL http://arxiv.org/abs/2301.10297.  arXiv:2301.10297 [cs, q-bio].
  * 7 Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A. Hosseini, Nancy Kanwisher, Joshua B. Tenenbaum, and Evelina Fedorenko.  The neural architecture of language: Integrative modeling converges on predictive processing.  118(45):e2105646118.  doi: 10.1073/pnas.2105646118.  URL https://www.pnas.org/doi/10.1073/pnas.2105646118.  Publisher: Proceedings of the National Academy of Sciences.
  * Goldstein et al. 2022 Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A. Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, Aren Jansen, Harshvardhan Gazula, Gina Choe, Aditi Rao, Catherine Kim, Colton Casto, Lora Fanda, Werner Doyle, Daniel Friedman, Patricia Dugan, Lucia Melloni, Roi Reichart, Sasha Devore, Adeen Flinker, Liat Hasenfratz, Omer Levy, Avinatan Hassidim, Michael Brenner, Yossi Matias, Kenneth A. Norman, Orrin Devinsky, and Uri Hasson.  Shared computational principles for language processing in humans and deep language models.  _Nature Neuroscience_ , 25(3):369–380, March 2022.  ISSN 1546-1726.  doi: 10.1038/s41593-022-01026-4.  URL https://www.nature.com/articles/s41593-022-01026-4.  Number: 3 Publisher: Nature Publishing Group.
  * 9 Charlotte Caucheteux, Alexandre Gramfort, and Jean-Remi King.  Long-range and hierarchical language predictions in brains and algorithms.  URL http://arxiv.org/abs/2111.14232.
  * Dong and Toneva 2023 Dota Tianai Dong and Mariya Toneva.  Vision-Language Integration in Multimodal Video Transformers (Partially) Aligns with the Brain, November 2023.  URL http://arxiv.org/abs/2311.07766.  arXiv:2311.07766 [cs].
  * Vigliocco et al. 2014a Gabriella Vigliocco, Pamela Perniss, and David Vinson.  Language as a multimodal phenomenon: implications for language learning, processing and evolution.  _Philosophical Transactions of the Royal Society B_ , 2014a.  doi: https://doi.org/10.1098/rstb.2013.0292.
  * 12 Rolf A. Zwaan, Mark C. Langston, and Arthur C. Graesser.  The construction of situation models in narrative comprehension: An event-indexing model.  6(5):292–297.  ISSN 0956-7976.  doi: 10.1111/j.1467-9280.1995.tb00513.x.  URL https://doi.org/10.1111/j.1467-9280.1995.tb00513.x.  Publisher: SAGE Publications Inc.
  * Zwaan 2014 Rolf A. Zwaan.  Embodiment and language comprehension: reframing the discussion.  _Trends in Cognitive Sciences_ , 18(5):229–234, May 2014.  ISSN 1364-6613.  doi: 10.1016/j.tics.2014.02.008.  URL https://www.sciencedirect.com/science/article/pii/S1364661314000527.
  * 14 Rolf A. Zwaan.  Situation models, mental simulations, and abstract concepts in discourse comprehension.  23(4):1028–1034.  ISSN 1531-5320.  doi: 10.3758/s13423-015-0864-x.  URL https://doi.org/10.3758/s13423-015-0864-x.
  * Pickering and Gambi 2018 Martin J. Pickering and Chiara Gambi.  Predicting while comprehending language: A theory and review.  _Psychological Bulletin_ , 144(10):1002–1044, 2018.  ISSN 1939-1455.  doi: 10.1037/bul0000158.  Place: US Publisher: American Psychological Association.
  * Skipper 2015 Jeremy I. Skipper.  The NOLB model: a model of the natural organization of language and the brain.  pages 101–134, Cambridge, 2015. Cambridge University Press.  ISBN 978-1-107-32366-7.  doi: 10.1017/CBO9781107323667.006.  URL https://www.cambridge.org/core/product/identifier/9781107323667%23c04201-929/type/book_part.  Book Title: Cognitive Neuroscience of Natural Language Use.
  * Huettig et al. 2011 Falk Huettig, Joost Rommers, and Antje S. Meyer.  Using the visual world paradigm to study language processing: a review and critical evaluation.  _Acta Psychologica_ , 137(2):151–171, June 2011.  ISSN 1873-6297.  doi: 10.1016/j.actpsy.2010.11.003.
  * Kuperberg and Jaeger 2016 Gina R. Kuperberg and T. Florian Jaeger.  What do we mean by prediction in language comprehension?  _Language, Cognition and Neuroscience_ , 31(1):32–59, 2016.  ISSN 2327-3798.  doi: 10.1080/23273798.2015.1102299.
  * Paivio 2013 A. Paivio.  _Imagery and Verbal Processes_.  Psychology Press, November 2013.  ISBN 978-1-317-75782-5.  Google-Books-ID: NxsuAgAAQBAJ.
  * Barsalou 1999 Lawrence W. Barsalou.  Perceptual symbol systems.  _Behavioral and Brain Sciences_ , 22(4):577–660, August 1999.  ISSN 1469-1825, 0140-525X.  doi: 10.1017/S0140525X99002149.  URL https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/abs/perceptual-symbol-systems/C2D720D63C1CE3D7153F6BA473F9DD87.  Publisher: Cambridge University Press.
  * Barsalou 2008 Lawrence W. Barsalou.  Grounded Cognition.  _Annual Review of Psychology_ , 59(1):617–645, 2008.  doi: 10.1146/annurev.psych.59.103006.093639.  URL https://doi.org/10.1146/annurev.psych.59.103006.093639.  _eprint: https://doi.org/10.1146/annurev.psych.59.103006.093639.
  * Vigliocco et al. 2009 Gabriella Vigliocco, Lotte Meteyard, Mark Andrews, and Stavroula Kousta.  Toward a theory of semantic representation.  _Language and Cognition_ , 1(2):219–247, June 2009.  ISSN 1866-9808, 1866-9859.  doi: 10.1515/LANGCOG.2009.011.  URL https://www.cambridge.org/core/journals/language-and-cognition/article/abs/toward-a-theory-of-semantic-representation/90A3FB3B15DB7034858BB808B41DCA4B.  Publisher: Cambridge University Press.
  * Vigliocco et al. 2014b Gabriella Vigliocco, Pamela Perniss, and David Vinson.  Language as a multimodal phenomenon: implications for language learning, processing and evolution.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ , 369(1651):20130292, September 2014b.  doi: 10.1098/rstb.2013.0292.  URL https://royalsocietypublishing.org/doi/10.1098/rstb.2013.0292.  Publisher: Royal Society.
  * Andrews et al. 2009 Mark Andrews, Gabriella Vigliocco, and David Vinson.  Integrating experiential and distributional data to learn semantic representations.  _Psychological Review_ , 116(3):463–498, 2009.  ISSN 1939-1471.  doi: 10.1037/a0016261.  Place: US Publisher: American Psychological Association.
  * Andrews et al. 2014 Mark Andrews, Stefan Frank, and Gabriella Vigliocco.  Reconciling Embodied and Distributional Accounts of Meaning in Language.  _Topics in Cognitive Science_ , 6(3):359–370, 2014.  ISSN 1756-8765.  doi: 10.1111/tops.12096.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/tops.12096.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/tops.12096.
  * Louwerse and Jeuniaux 2010 Max M. Louwerse and Patrick Jeuniaux.  The linguistic and embodied nature of conceptual processing.  _Cognition_ , 114(1):96–104, January 2010.  ISSN 0010-0277.  doi: 10.1016/j.cognition.2009.09.002.  URL https://www.sciencedirect.com/science/article/pii/S0010027709002157.
  * Kousta et al. 2011 Stavroula-Thaleia Kousta, Gabriella Vigliocco, David P. Vinson, Mark Andrews, and Elena Del Campo.  The representation of abstract words: Why emotion matters.  _Journal of Experimental Psychology: General_ , 140(1):14–34, 2011.  ISSN 1939-2222.  doi: 10.1037/a0021446.  Place: US Publisher: American Psychological Association.
  * Riordan and Jones 2011 Brian Riordan and Michael N. Jones.  Redundancy in Perceptual and Linguistic Experience: Comparing Feature-Based and Distributional Models of Semantic Representation.  _Topics in Cognitive Science_ , 3(2):303–345, 2011.  ISSN 1756-8765.  doi: 10.1111/j.1756-8765.2010.01111.x.  URL https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1756-8765.2010.01111.x.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1756-8765.2010.01111.x.
  * Kewenig et al. 2023 Viktor Kewenig, Gabriella Vigliocco, and Jeremy I. Skipper.  When Abstract Becomes Concrete: Naturalistic Encoding of Concepts in the Brain, August 2023.  URL https://www.biorxiv.org/content/10.1101/2022.09.08.506944v3.  Pages: 2022.09.08.506944 Section: New Results.
  * Hahamy et al. 2023 Avital Hahamy, Haim Dubossarsky, and Timothy E. J. Behrens.  The human brain reactivates context-specific past information at event boundaries of naturalistic experiences.  _Nature Neuroscience_ , 26(6):1080–1089, June 2023.  ISSN 1546-1726.  doi: 10.1038/s41593-023-01331-6.
  * Zadbood et al. 2022 Asieh Zadbood, Samuel Nastase, Janice Chen, Kenneth A Norman, and Uri Hasson.  Neural representations of naturalistic events are updated as our understanding of the past changes.  _eLife_ , 11:e79045, December 2022.  ISSN 2050-084X.  doi: 10.7554/eLife.79045.  URL https://doi.org/10.7554/eLife.79045.  Publisher: eLife Sciences Publications, Ltd.
  * 32 Christopher Baldassano, Uri Hasson, and Kenneth A. Norman.  Representation of real-world event schemas during narrative perception.  38(45):9689–9699.  ISSN 0270-6474, 1529-2401.  doi: 10.1523/JNEUROSCI.0251-18.2018.  URL https://www.jneurosci.org/content/38/45/9689.  Publisher: Society for Neuroscience Section: Research Articles.
  * ben 2018 The Hippocampal Film Editor: Sensitivity and Specificity to Event Boundaries in Continuous Experience.  _Journal of Neuroscience_ , 38(47):10057–10068, November 2018.  ISSN 0270-6474, 1529-2401.  doi: 10.1523/JNEUROSCI.0524-18.2018.  URL https://www.jneurosci.org/content/38/47/10057.  Publisher: Society for Neuroscience Section: Research Articles.
  * Pezzulo et al. 2023 Giovanni Pezzulo, Thomas Parr, Paul Cisek, Andy Clark, and Karl Friston.  Generating meaning: active inference and the scope and limits of passive AI.  _Trends in Cognitive Sciences_ , November 2023.  ISSN 1364-6613.  doi: 10.1016/j.tics.2023.10.002.  URL https://www.sciencedirect.com/science/article/pii/S1364661323002607.
  * Alayrac et al. 2022 Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, Aida Nematzadeh, Sahand Sharifzadeh, Mikołaj Bińkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan.  Flamingo: a visual language model for few-shot learning.  In _Advances in Neural Information Processing Systems 35 (NeurIPS 2022) Main Conference Track_ , 2022.
  * Bransford et al. 1972 John D. Bransford, J. Richard Barclay, and Jeffery J. Franks.  Sentence memory: A constructive versus interpretive approach.  _Cognitive Psychology_ , 3(2):193–209, 1972.  ISSN 1095-5623.  doi: 10.1016/0010-0285(72)90003-5.  Place: Netherlands Publisher: Elsevier Science.
  * Huettig 2015 Falk Huettig.  Four central questions about prediction in language processing.  _Brain Research_ , 1626:118–135, November 2015.  ISSN 0006-8993.  doi: 10.1016/j.brainres.2015.02.014.  URL https://www.sciencedirect.com/science/article/pii/S0006899315001146.
  * Tanenhaus et al. 1995 Michael K. Tanenhaus, Michael J. Spivey-Knowlton, Kathleen M. Eberhard, and Julie C. Sedivy.  Integration of Visual and Linguistic Information in Spoken Language Comprehension.  _Science_ , 268(5217):1632–1634, June 1995.  doi: 10.1126/science.7777863.  URL https://www.science.org/doi/10.1126/science.7777863.  Publisher: American Association for the Advancement of Science.
  * Eichert et al. 2018 Nicole Eichert, David Peeters, and Peter Hagoort.  Language-driven anticipatory eye movements in virtual reality.  _Behavior Research Methods_ , 50(3):1102–1115, June 2018.  ISSN 1554-3528.  doi: 10.3758/s13428-017-0929-z.
  * Altmann and Kamide 1999 Gerry T. M. Altmann and Yuki Kamide.  Incremental interpretation at verbs: Restricting the domain of subsequent reference.  _Cognition_ , 73(3):247–264, 1999.  ISSN 1873-7838.  doi: 10.1016/S0010-0277(99)00059-1.  Place: Netherlands Publisher: Elsevier Science.
  * Kamide et al. 2003 Yuki Kamide, Christoph Scheepers, and Gerry T. M. Altmann.  Integration of Syntactic and Semantic Information in Predictive Processing: Cross-Linguistic Evidence from German and English.  _Journal of Psycholinguistic Research_ , 32(1):37–55, January 2003.  ISSN 1573-6555.  doi: 10.1023/A:1021933015362.  URL https://doi.org/10.1023/A:1021933015362.
  * Kukona et al. 2014 Anuenue Kukona, Gerry T. M. Altmann, and Yuki Kamide.  Knowing what, where, and when: Event comprehension in language processing.  _Cognition_ , 133(1):25–31, October 2014.  ISSN 0010-0277.  doi: 10.1016/j.cognition.2014.05.011.  URL https://www.sciencedirect.com/science/article/pii/S0010027714001000.
  * Rommers et al. 2015 Joost Rommers, Antje S. Meyer, and Falk Huettig.  Verbal and nonverbal predictors of language-mediated anticipatory eye movements.  _Attention, Perception, & Psychophysics_, 77(3):720–730, April 2015.  ISSN 1943-393X.  doi: 10.3758/s13414-015-0873-x.  URL https://doi.org/10.3758/s13414-015-0873-x.
  * Vaswani et al. 2017 Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is All you Need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://papers.nips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html.
  * Serrano and Smith 2019 Sofia Serrano and Noah A. Smith.  Is Attention Interpretable?, June 2019.  URL http://arxiv.org/abs/1906.03731.  arXiv:1906.03731 [cs].
  * Da Costa and Chaves 2020 Jillian Da Costa and Rui Chaves.  Assessing the ability of Transformer-based Neural Models to represent structurally unbounded dependencies.  In _Proceedings of the Society for Computation in Linguistics 2020_ , pages 12–21, New York, New York, January 2020. Association for Computational Linguistics.  URL https://aclanthology.org/2020.scil-1.2.
  * Busse et al. 2005 Laura Busse, Kenneth C. Roberts, Roy E. Crist, Daniel H. Weissman, and Marty G. Woldorff.  The spread of attention across modalities and space in a multisensory object.  _Proceedings of the National Academy of Sciences_ , 102(51):18751–18756, December 2005.  doi: 10.1073/pnas.0507704102.  URL https://www.pnas.org/doi/10.1073/pnas.0507704102.  Publisher: Proceedings of the National Academy of Sciences.
  * Tay et al. 2021 Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng.  Synthesizer: Rethinking Self-Attention in Transformer Models, May 2021.  URL http://arxiv.org/abs/2005.00743.  arXiv:2005.00743 [cs].
  * Lindsay 2020 Grace W. Lindsay.  Attention in Psychology, Neuroscience, and Machine Learning.  _Frontiers in Computational Neuroscience_ , 14, 2020.  ISSN 1662-5188.  URL https://www.frontiersin.org/articles/10.3389/fncom.2020.00029.
  * Conwell et al. 2022 Colin Conwell, Jacob S. Prince, Kendrick N. Kay, George A. Alvarez, and Talia Konkle.  What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?  _Preprint_ , 2022.  doi: https://doi.org/10.1101/2022.03.28.485868.  This article is a preprint and has not been certified by peer review.
  * Dosovitskiy et al. 2021 Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, June 2021.  URL http://arxiv.org/abs/2010.11929.  arXiv:2010.11929 [cs].
  * Tuli et al. 2021 Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas L. Griffiths.  Are Convolutional Neural Networks or Transformers more like human vision?, July 2021.  URL http://arxiv.org/abs/2105.07197.  arXiv:2105.07197 [cs].
  * Sood et al. 2020 Ekta Sood, Simon Tannert, Diego Frassinelli, Andreas Bulling, and Ngoc Thang Vu.  Interpreting Attention Models with Human Visual Attention in Machine Reading Comprehension, October 2020.  URL http://arxiv.org/abs/2010.06396.  arXiv:2010.06396 [cs].
  * Yang et al. 2023 Zhibo Yang, Sounak Mondal, Seoyoung Ahn, Gregory Zelinsky, Minh Hoai, and Dimitris Samaras.  Predicting Human Attention using Computational Attention, April 2023.  URL http://arxiv.org/abs/2303.09383.  arXiv:2303.09383 [cs].
  * Radford et al. 2021 Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning Transferable Visual Models From Natural Language Supervision, February 2021.  URL http://arxiv.org/abs/2103.00020.  arXiv:2103.00020 [cs].
  * Touvron et al. 2023 Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models.  _Unknown Journal_ , 2023.  URL UnknownURL.
  * Gao et al. 2023 Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao.  Llama-adapter v2: Parameter-efficient visual instruction model.  _The Journal or Conference Name_ , The Volume Number(The Issue Number):The Page Range, 2023.  URL TheURL.
  * He et al. 2016 Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep Residual Learning for Image Recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, Las Vegas, NV, USA, June 2016. IEEE.  ISBN 978-1-4673-8851-1.  doi: 10.1109/CVPR.2016.90.  URL http://ieeexplore.ieee.org/document/7780459/.
  * Tenney et al. 2019 Ian Tenney, Dipanjan Das, and Ellie Pavlick.  BERT Rediscovers the Classical NLP Pipeline.  In Anna Korhonen, David Traum, and Lluís Màrquez, editors, _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4593–4601, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1452.  URL https://aclanthology.org/P19-1452.
  * Rogers et al. 2021 Anna Rogers, Olga Kovaleva, and Anna Rumshisky.  A Primer in BERTology: What We Know About How BERT Works.  _Transactions of the Association for Computational Linguistics_ , 8:842–866, January 2021.  ISSN 2307-387X.  doi: 10.1162/tacl_a_00349.  URL https://doi.org/10.1162/tacl_a_00349.
  * Liu et al. 2019 Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith.  Linguistic Knowledge and Transferability of Contextual Representations, April 2019.  URL http://arxiv.org/abs/1903.08855.  arXiv:1903.08855 [cs].
  * 62 Subba Reddy Oota, Manish Gupta, and Mariya Toneva.  Joint processing of linguistic properties in brains and language models.  URL http://arxiv.org/abs/2212.08094.
  * Caucheteux and King 2022 Charlotte Caucheteux and Jean-Rémi King.  Brains and algorithms partially converge in natural language processing.  _Communications Biology_ , 5(1):1–10, February 2022.  ISSN 2399-3642.  doi: 10.1038/s42003-022-03036-1.  URL https://www.nature.com/articles/s42003-022-03036-1.  Number: 1 Publisher: Nature Publishing Group.
  * 64 Shailee Jain, Vy A. Vo, Leila Wehbe, and Alexander G. Huth.  Computational language modeling and the promise of in silico experimentation.  pages 1–65.  ISSN 2641-4368.  doi: 10.1162/nol_a_00101.  URL https://doi.org/10.1162/nol_a_00101.
  * 65 Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G. Huth.  Semantic reconstruction of continuous language from non-invasive brain recordings.  URL https://www.biorxiv.org/content/10.1101/2022.09.29.509744v1.  Pages: 2022.09.29.509744 Section: New Results.
  * Bergen 2007 Benjamin Bergen.  Experimental methods for simulation semantics.  In _Methods in cognitive linguistics_ , pages 277–301. John Benjamins Publishing Company, Amsterdam, Netherlands, 2007.  ISBN 978-90-272-2372-2 978-90-272-2371-5.  doi: 10.1075/hcp.18.19ber.
  * Fischer and Weber 1993 B. Fischer and H. Weber.  Express saccades and visual attention.  _Behavioral and Brain Sciences_ , 16(3):553–567, September 1993.  ISSN 1469-1825, 0140-525X.  doi: 10.1017/S0140525X00031575.  URL https://www.cambridge.org/core/journals/behavioral-and-brain-sciences/article/abs/express-saccades-and-visual-attention/B38F6BA1820BDD8753880DCD2814AC7D.  Publisher: Cambridge University Press.
  * Dimigen et al. 2022 Olaf Dimigen, Ulrike Schild, Annette Hohlfeld, and Werner Sommer.  Auditory Language Comprehension During Saccadic Eye Movements: An Investigation Using the N400 Event-Related Brain Potential, April 2022.  URL https://www.biorxiv.org/content/10.1101/2022.04.05.487157v1.  Pages: 2022.04.05.487157 Section: New Results.
  * Kutas and Federmeier 2011 Marta Kutas and Kara D. Federmeier.  Thirty Years and Counting: Finding Meaning in the N400 Component of the Event-Related Brain Potential (ERP).  _Annual Review of Psychology_ , 62(1):621–647, 2011.  doi: 10.1146/annurev.psych.093008.131123.  URL https://doi.org/10.1146/annurev.psych.093008.131123.  _eprint: https://doi.org/10.1146/annurev.psych.093008.131123.
  * Marslen-Wilson 1973 William Marslen-Wilson.  Linguistic structure and speech shadowing at very short latencies.  _Nature_ , 244(5417):522–523, 1973.  ISSN 1476-4687.  doi: 10.1038/244522a0.  Place: United Kingdom Publisher: Nature Publishing Group.
  * Jiahui et al. 2021 Guo Jiahui, Ma Feilong, Matteo Visconti di Oleggio Castello, Samuel A. Nastase, James V. Haxby, and M. Ida Gobbini.  Modeling naturalistic face processing in humans with deep convolutional neural networks.  _Unknown Journal_ , November 2021.  doi: https://doi.org/10.1101/2021.11.17.469009.
  * Li et al. 2021 Hengduo Li, Zuxuan Wu, Abhinav Shrivastava, and Larry S. Davis.  2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition, April 2021.  URL http://arxiv.org/abs/2012.14950.  arXiv:2012.14950 [cs].
  * Baltrušaitis et al. 2017 Tadas Baltrušaitis, Chaitanya Ahuja, and Louis-Philippe Morency.  Multimodal Machine Learning: A Survey and Taxonomy, August 2017.  URL http://arxiv.org/abs/1705.09406.  arXiv:1705.09406 [cs].
  * noa a A deep dive into the code of the BERT model | by alexey kravets | towards data science, a.  URL https://towardsdatascience.com/deep-dive-into-the-code-of-bert-model-9f618472353e.
  * Heyselaar et al. 2020 Evelien Heyselaar, David Peeters, and Peter Hagoort.  Do we predict upcoming speech content in naturalistic environments?  _Language, Cognition and Neuroscience_ , 36:1–22, December 2020.  doi: 10.1080/23273798.2020.1859568.
  * Ryskin and Nieuwland 2023 Rachel Ryskin and Mante S. Nieuwland.  Prediction during language comprehension: what is next?  _Trends in Cognitive Sciences_ , 27(11):1032–1052, November 2023.  ISSN 1364-6613, 1879-307X.  doi: 10.1016/j.tics.2023.08.003.  URL https://www.cell.com/trends/cognitive-sciences/abstract/S1364-6613(23)00199-7.  Publisher: Elsevier.
  * Wang et al. 2020 Jiahui Wang, Pavlo Antonenko, and Kara Dawson.  Does visual attention to the instructor in online video affect learning and learner perceptions? An eye-tracking analysis.  _Computers & Education_, 146:103779, March 2020.  ISSN 0360-1315.  doi: 10.1016/j.compedu.2019.103779.  URL https://www.sciencedirect.com/science/article/pii/S036013151930329X.
  * Boynton 2005 Geoffrey M. Boynton.  Attention and visual perception.  _Current Opinion in Neurobiology_ , 15(4):465–469, 2005.  ISSN 1873-6882.  doi: 10.1016/j.conb.2005.06.009.  Place: Netherlands Publisher: Elsevier Science.
  * Chen and Hutchinson 2019 Danlei Chen and J. Benjamin Hutchinson.  What Is Memory-Guided Attention? How Past Experiences Shape Selective Visuospatial Attention in the Present.  _Current Topics in Behavioral Neurosciences_ , 41:185–212, 2019.  ISSN 1866-3370.  doi: 10.1007/7854_2018_76.
  * Jiang 2018 Yuhong V. Jiang.  Habitual versus goal-driven attention.  _Cortex; a journal devoted to the study of the nervous system and behavior_ , 102:107–120, May 2018.  ISSN 0010-9452.  doi: 10.1016/j.cortex.2017.06.018.  URL https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5754262/.
  * Derakshan and Eysenck 2010 Nazanin Derakshan and Michael W. Eysenck.  Introduction to the special issue: Emotional states, attention, and working memory.  _COGNITION AND EMOTION_ , February 2010.  doi: 10.1080/02699930903412120.  URL https://www.tandfonline.com/doi/full/10.1080/02699930903412120.  Publisher: Taylor & Francis Group.
  * Nobre and van Ede 2018 Anna C. Nobre and Freek van Ede.  Anticipated moments: temporal structure in attention.  _Nature Reviews Neuroscience_ , 19(1):34–48, January 2018.  ISSN 1471-0048.  doi: 10.1038/nrn.2017.141.  URL https://www.nature.com/articles/nrn.2017.141.  Number: 1 Publisher: Nature Publishing Group.
  * Custers and Aarts 2011 Ruud Custers and Henk Aarts.  Learning of predictive relations between events depends on attention, not on awareness.  _Consciousness and Cognition_ , 20(2):368–378, June 2011.  ISSN 1053-8100.  doi: 10.1016/j.concog.2010.05.011.  URL https://www.sciencedirect.com/science/article/pii/S105381001000111X.
  * Magnuson 2019 James S. Magnuson.  Fixations in the visual world paradigm: where, when, why?  _Journal of Cultural Cognitive Science_ , 3(2):113–139, November 2019.  ISSN 2520-1018.  doi: 10.1007/s41809-019-00035-3.  URL https://doi.org/10.1007/s41809-019-00035-3.
  * Foulsham et al. 2011 Tom Foulsham, Esther Walker, and Alan Kingstone.  The where, what and when of gaze allocation in the lab and the natural environment.  _Vision Research_ , 51(17):1920–1931, September 2011.  ISSN 0042-6989.  doi: 10.1016/j.visres.2011.07.002.  URL https://www.sciencedirect.com/science/article/pii/S0042698911002392.
  * Chemero 2023 Anthony Chemero.  LLMs differ from human cognition because they are not embodied.  _Nature Human Behaviour_ , 7(11):1828–1829, November 2023.  ISSN 2397-3374.  doi: 10.1038/s41562-023-01723-5.  URL https://www.nature.com/articles/s41562-023-01723-5.  Number: 11 Publisher: Nature Publishing Group.
  * Anderson 2003 Michael L. Anderson.  Embodied Cognition: A field guide.  _Artificial Intelligence_ , 149(1):91–130, September 2003.  ISSN 0004-3702.  doi: 10.1016/S0004-3702(03)00054-7.  URL https://www.sciencedirect.com/science/article/pii/S0004370203000547.
  * Meteyard et al. 2012 Lotte Meteyard, Sara Rodriguez Cuadrado, Bahador Bahrami, and Gabriella Vigliocco.  Coming of age: A review of embodiment and the neuroscience of semantics.  _Cortex_ , 48(7):788–804, July 2012.  ISSN 0010-9452.  doi: 10.1016/j.cortex.2010.11.002.  URL https://www.sciencedirect.com/science/article/pii/S0010945210002662.
  * Barsalou 2009 Lawrence W. Barsalou.  Simulation, situated conceptualization, and prediction.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ , 364(1521):1281–1289, May 2009.  doi: 10.1098/rstb.2008.0319.  URL https://royalsocietypublishing.org/doi/full/10.1098/rstb.2008.0319.  Publisher: Royal Society.
  * Zador et al. 2023 Anthony Zador, Sean Escola, Blake Richards, Bence Ölveczky, Yoshua Bengio, Kwabena Boahen, Matthew Botvinick, Dmitri Chklovskii, Anne Churchland, Claudia Clopath, James DiCarlo, Surya Ganguli, Jeff Hawkins, Konrad Körding, Alexei Koulakov, Yann LeCun, Timothy Lillicrap, Adam Marblestone, Bruno Olshausen, Alexandre Pouget, Cristina Savin, Terrence Sejnowski, Eero Simoncelli, Sara Solla, David Sussillo, Andreas S. Tolias, and Doris Tsao.  Catalyzing next-generation Artificial Intelligence through NeuroAI.  _Nature Communications_ , 14(1):1597, March 2023.  ISSN 2041-1723.  doi: 10.1038/s41467-023-37180-x.  URL https://www.nature.com/articles/s41467-023-37180-x.
  * noa b PaLM-E: An Embodied Multimodal Language Model, b.  URL https://palm-e.github.io/.
  * Srivastava et al. 2022 Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, Anders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhlmüller, Andrew Dai, Andrew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabassum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Herrick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Bartłomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro, Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo, Dan Garrette, Dan Hendrycks, Dan Kilman, Dan Roth, Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez, Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, Denis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo, Diyi Yang, Dong-Ho Lee, Ekaterina Shutova, Ekin Dogus Cubuk, Elad Segal, Eleanor Hagerman, Elizabeth Barnes, Elizabeth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Eunice Engefu Manyasi, Evgenii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martínez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch-López, Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hongming Zhang, Hugh Mee Wong, Ian Ng, Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang, Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Jiacheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Hernandez-Orallo, Joseph Boudeman, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chiafullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson, Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin, Lidia Contreras-Ochando, Louis-Philippe Morency, Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Farooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduna Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michał Swędrowski, Michele Bevilacqua, Michihiro Yasunaga, Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mo Tiwari, Mohit Bansal, Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts, Nick Doiron, Nikita Nangia, Niklas Deckers, Niklas Muennighoff, Nitish Shirish Keskar, Niveditha S. Iyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar Elbaghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung, Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Peter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Miłkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramón Risco Delgado, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Roman Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang, Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang, Sahib Singh, Saif M. Mohammad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghazarian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schuster, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upadhyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman, Stephanie Lin, Stephen Prasad, Steven T. Piantadosi, Stuart M. Shieber, Summer Misherghi, Svetlana Kiritchenko, Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Desbordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Timothy Telleen-Lawton, Titus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj, Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saunders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong, Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang, Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu.  Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models, June 2022.  URL http://arxiv.org/abs/2206.04615.  Number: arXiv:2206.04615 arXiv:2206.04615 [cs, stat].
  * Lenci 2023 Alessandro Lenci.  Understanding Natural Language Understanding Systems. A Critical Analysis, March 2023.  URL http://arxiv.org/abs/2303.04229.  arXiv:2303.04229 [cs].
  * 94 Simkute Sellen Rintel Tankelevitch, Kewenig.  [2312.10893] The Metacognitive Demands and Opportunities of Generative AI.  URL https://arxiv.org/abs/2312.10893.
  * 95 G. Spiers H. Musolesi M. De Felice S. Kewenig V. , Vigliocco.  PsyArXiv Preprints | Ecological Brain: Reframing the Study of Human Behaviour and Cognition.  URL https://osf.io/preprints/psyarxiv/zr4nm.
  * Palan and Schitter 2018 Stefan Palan and Christian Schitter.  Prolific.ac—A subject pool for online experiments.  _Journal of Behavioral and Experimental Finance_ , 17(C):22–27, 2018.  ISSN 2214-6350.  URL https://econpapers.repec.org/article/eeebeexfi/v_3a17_3ay_3a2018_3ai_3ac_3ap_3a22-27.htm.  Publisher: Elsevier.
  * Devlin et al. 2019 Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  Technical Report arXiv:1810.04805, arXiv, May 2019.  URL http://arxiv.org/abs/1810.04805.  arXiv:1810.04805 [cs] type: article.
  * Anwyl-Irvine et al. 2020 Alexander L. Anwyl-Irvine, Jessica Massonnié, Adam Flitton, Natasha Kirkham, and Jo K. Evershed.  Gorilla in our midst: An online behavioral experiment builder.  _Behavior Research Methods_ , 52(1):388–407, 2020.  ISSN 1554-3528.  doi: 10.3758/s13428-019-01237-x.  Place: Germany Publisher: Springer.
  * Harris et al. 2020 Charles R. Harris, K. Jarrod Millman, Stéfan J. van der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten H. van Kerkwijk, Matthew Brett, Allan Haldane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant.  Array programming with NumPy.  _Nature_ , 585(7825):357–362, September 2020.  ISSN 1476-4687.  doi: 10.1038/s41586-020-2649-2.  URL https://www.nature.com/articles/s41586-020-2649-2.  Number: 7825 Publisher: Nature Publishing Group.
  * Virtanen et al. 2020 Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nelson, Eric Jones, Robert Kern, Eric Larson, C. J. Carey, İlhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, Antônio H. Ribeiro, Fabian Pedregosa, and Paul van Mulbregt.  SciPy 1.0: fundamental algorithms for scientific computing in Python.  _Nature Methods_ , 17(3):261–272, March 2020.  ISSN 1548-7105.  doi: 10.1038/s41592-019-0686-2.  URL https://www.nature.com/articles/s41592-019-0686-2.  Number: 3 Publisher: Nature Publishing Group.
  * noa c pandas - Python Data Analysis Library, c.  URL https://pandas.pydata.org/.
  * Paszke et al. 2019 Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://papers.nips.cc/paper_files/paper/2019/hash/bdbca288fee7f92f2bfa9f7012727740-Abstract.html.
  * noa d openai/clip-vit-base-patch32 · Hugging Face, d.  URL https://huggingface.co/openai/clip-vit-base-patch32.
  * Hu and Levy 2023 Jennifer Hu and Roger Levy.  Prompting is not a substitute for probability measurements in large language models.  _Unknown Journal_ , May 2023.  URL UnknownURL.  Submitted on 22 May 2023 (v1), last revised 23 Oct 2023 (this version, v2).
  * Dahan et al. 2007 Delphine Dahan, Michael K. Tanenhaus, and Anne Pier Salverda.  Chapter 21 - The influence of visual processing on phonetically driven saccades in the “visual world” paradigm.  In Roger P. G. Van Gompel, Martin H. Fischer, Wayne S. Murray, and Robin L. Hill, editors, _Eye Movements_ , pages 471–486. Elsevier, Oxford, January 2007.  ISBN 978-0-08-044980-7.  doi: 10.1016/B978-008044980-7/50023-9.  URL https://www.sciencedirect.com/science/article/pii/B9780080449807500239.
  * Huettig and Altmann 2005 Falk Huettig and Gerry T. M. Altmann.  Word meaning and the control of eye fixation: semantic competitor effects and the visual world paradigm.  _Cognition_ , 96(1):B23–B32, May 2005.  ISSN 0010-0277.  doi: 10.1016/j.cognition.2004.10.003.  URL https://www.sciencedirect.com/science/article/pii/S0010027704001866.