# Modelling Multimodal Integration in Human Concept Processing with Vision-
and-Language Models

Anna Bavaresco∗, Marianne de Heer Kloots,111Shared first authorship.
Sandro Pezzelle, Raquel Fernández
Institute for Logic, Language and Computation
University of Amsterdam

###### Abstract

Representations from deep neural networks (DNNs) have proven remarkably
predictive of neural activity involved in both visual and linguistic
processing. Despite these successes, most studies to date concern unimodal
DNNs, encoding either visual or textual input but not both. Yet, there is
growing evidence that human meaning representations integrate linguistic and
sensory-motor information. Here we investigate whether the integration of
multimodal information operated by current vision-and-language DNN models
(VLMs) leads to representations that are more aligned with human brain
activity than those obtained by language-only and vision-only DNNs. We focus
on fMRI responses recorded while participants read concept words in the
context of either a full sentence or an accompanying picture. Our results
reveal that VLM representations correlate more strongly than language- and
vision-only DNNs with activations in brain areas functionally related to
language processing. A comparison between different types of visuo-linguistic
architectures shows that recent generative VLMs tend to be less brain-aligned
than previous architectures with lower performance on downstream applications.
Moreover, through an additional analysis comparing brain vs. behavioural
alignment across multiple VLMs, we show that – with one remarkable exception –
representations that strongly align with behavioural judgments do not
correlate highly with brain responses. This indicates that brain similarity
does not go hand in hand with behavioural similarity, and vice versa.

###### keywords:

conceptual representations , vision and language processing , deep neural
networks , brain imaging data , representational similarity analysis

††journal: Journal of Memory and Language

We analyse fMRI responses to concepts in the context of written sentences or
pictures

Vision-and-language models (VLMs) are more brain-aligned than unimodal models

Generative VLMs are not more brain-aligned than other visuo-linguistic
architectures

Brain alignment does not go hand in hand with alignment to behavioural
judgments

##  1 Introduction

How does the brain represent semantic knowledge? Whereas many computational
models of language still build on the idea that meaning representations can be
extracted from text corpora [78], increasing evidence supports the view that
human concept representations are in fact grounded in our sensory experiences
with the world. Theories of embodied simulation argue that humans build
meaning representations by integrating sensory-motor and linguistic
information [63, 5, 39, 9], which aligns with developmental studies on word
meaning acquisition [56, 12] as well as behavioural and neuroimaging
experiments highlighting an important role for perceptual attributes of word
meaning in language processing [6]. The need to ground meaning representations
in the physical world has also been emphasized by researchers in natural
language processing, as an important step towards better language
understanding in machines [8, 10].

The exact nature and localization of semantic integration in the brain is an
ongoing topic of neuroscientific research: one prominent account points to a
single more centralized cross-modal conceptual hub interacting with several
distributed modality-specific ‘spokes’ [55], but other observations are in
line with a larger set of ‘convergence zones’ involved in multimodal
integration across the cortex [33, 23]. Nevertheless, it is clear that humans
are able to flexibly engage conceptual representations across diverse input
modalities such as pictures and texts. Visual and linguistic stimuli, in
particular, have been shown to mutually influence each other in processing
[13, 71, 45], and also evoke shared neural activity patterns, thought to
reflect some form of abstracted, modality-independent conceptual
representations [26, 89, 79, 50]. Although the existence of such modality-
independent activity patterns does not necessarily tell us much about their
representational content, recent evidence indicates that they indeed encode
both linguistic and visual information [28].

The joint encoding of visual and linguistic information in concept
representations is further supported by work showing that computational models
trained on predicting brain activity in language processing successfully
transfer to predicting brain activity in visual processing, especially when
such models build on representations that integrate stimulus features from
both modalities [96]. Such analyses, comparing between brain activity patterns
and computationally generated representations, offer a promising way to shed
further light on the nature of multimodal integration in the brain.

###  1.1 Use of DNNs in modelling human behaviour and neural activity

In artificial intelligence, neural architectures trained on large amounts of
data with task-optimized (or, alternatively, self-supervised) representation
learning algorithms currently underlie virtually all state-of-the-art systems.
Such architectures rely on artificial deep neural networks (DNNs), which have
enabled unprecedented advances in the past decade across the fields of
computer vision and natural language processing, as well as applications
integrating the two.

Meanwhile, computational cognitive (neuro)science has made impressive progress
using DNNs as model systems of human behaviour and its neural substrates.
Traditional experimental approaches often rely on very specific theory-driven
hypotheses, with accordingly designed controlled stimuli and contrasting
conditions. On the other hand, DNNs provide a complementary approach to
studying human information processing, by generating rich representational
spaces to encode more complex naturalistic stimuli. In a relatively new line
of studies, DNNs are employed as stimulus-computable models, whose internal
representations and output behaviour are directly compared to human neural and
behavioural responses measured when fed with similar input [54, 64, 30].

Internal representations from DNNs have proven remarkably predictive of human
neural activity in both visual and linguistic processing. In the domain of
vision, early studies convolutional neural networks (CNNs) optimized for the
task of object recognition found that representations extracted from such
models are very successful predictors of fMRI [37, 32], MEG [86] and neural
spiking [105] responses to natural images. A common finding across these
studies is that earlier CNN layers are more aligned with lower visual areas
and deeper CNN layers are more aligned with higher ventral stream areas,
potentially reflecting similar hierarchical processing mechanisms across
cortical and CNN layers (44; see 61 for a review). However, more recent work
has found improved alignment to human vision in vision transformers (ViTs)
over CNNs [e.g., 100], despite big differences in architectural inductive
biases between these models [83]. A recent large-scale investigation comparing
many DNN architectures concludes that performance at predicting visual cortex
activity is in fact comparable between models with more human-like
architectural biases (e.g., convolutional processing mechanisms, as in CNNs)
and models with more human-like training objectives (e.g., no direct category
supervision, as in ViTs), when training data is controlled for [22].

Regarding models of language, early work pioneered the use of corpus-derived
and neural network-based static word embeddings as powerful tools for mapping
between word-level linguistic information and fMRI activity associated with
single nouns and corresponding visual stimuli [68, 1, 3]. Such type-level
static embeddings also proved useful for predictive models of brain activity
in more complex linguistic activities such as sentence reading [74] and
narrative understanding [104]. However, a huge improvement in this domain has
been achieved by employing internal states from LSTM- and, later, transformer-
based language models (e.g., Jain Huth 46, Toneva Wehbe 98; see Karamolegkou .
49 for a review). By integrating information across the entire linguistic
input provided to the model, such models are able to create contextualized,
token-level representations of language which are arguably more aligned with
human online processing of linguistic stimuli in context. Most prominently,
Schrimpf . [85] compared a comprehensive range of neural network-based
embeddings for their predictivity of human neural (fMRI, ECoG) and behavioural
(reading time) data, including static GloVe embeddings [73] and internal
states extracted from LSTMs [41], BERT-based masked language models [27] and
GPT-like autoregressive language models [82]. Across this range of
architectures, Schrimpf . found that generative, autoregressive architectures
trained on the next-word-prediction objective were most predictive of human
cognitive signals in language processing. A critical result from this study
also demonstrated a strong correlation between neural and behavioural
predictivity, suggesting that architectures that are well-aligned with brain
activations can also be good models of behavioural responses in language
processing (in this case, reading times).

Other work has modelled a wide range of human linguistic behaviour using
various measures extracted from language models, including grammaticality
judgments [103, 43], priming [90], utterance predictability [36], ERP
responses [67, 38], and eye fixation patterns [42]. The wide applicability of
DNN hidden state representations and derived model output measures in fact
highlights one of the major strengths of this class of models compared to
other cognitive modelling paradigms: models trained to capture a rich variety
of phenomena are more likely to be useful for multiple down-stream tasks and
also to overlap in interesting ways with similar general-purpose
representations involved in human language processing [4].

###  1.2 Modelling human multimodal meaning representations

Despite successes in applying DNNs to neural and cognitive modelling across
the visual and linguistic domains, most studies described above concern
unimodal DNNs, encoding either visual or textual input but not both. Earlier
approaches in computational semantics have however repeatedly found that
models of lexical meaning that combine visual and textual information are
better aligned with human meaning representations than their unimodal
counterparts. Many studies in this line of work combined modalities by
concatenating textual and perceptual vectors, often drawing on separate
sources of data for building concept representations in either domain [14, 51,
53, 25, 24], whereas some efforts constructed unified multimodal embeddings
aimed at representing the latent information shared across textual and
perceptual modalities [58, 106, 87, 88]. Overall, such multimodal
distributional semantic models were shown to outperform text-only approaches
at tasks requiring visual knowledge, as well as more general semantic
benchmarks, measuring for example correlations to human word similarity or
relatedness judgments (e.g., WordSim-353, Finkelstein . 35; SimLex999, Hill .
40; MEN, Bruni . 14). Moreover, concatenated visio-linguistic embeddings also
showed an advantage over text- and vision-only baselines in decoding brain
activity related to concrete noun representations [24]. More recent studies
have evaluated representations extracted from fully end-to-end models trained
to map between images and corresponding text captions and found that such
models show improved performance over text models on sentence representation
tasks [52] as well as improved alignment to human word similarity and
relatedness judgments [66]. These findings beg the question of whether such
emergent cross-/multimodal representations in end-to-end models are also well-
aligned with brain activity involved in human concept processing.

###  1.3 Integrated multimodal representations from multimodal pretraining

In recent years, a powerful class of transformer-based vision-and-language
models (VLMs) have achieved outstanding performance on a variety of downstream
tasks in artificial intelligence. After being pre-trained on matched visual
and textual data (e.g., image-caption pairs), these models turn out
particularly capable at tasks requiring the integration of information from
textual and visual inputs, such as Visual Question Answering, Image
Captioning, and Visual Dialogue (see Suglia . 92 for a review). In addition to
such broad evaluation tasks, several testing paradigms and datasets have also
been designed to evaluate VLMs on specific aspects of human-like semantic and
situational understanding, such as compositionality [97], goal-based action
selection [76] and embodied simulation [48], as well as their capabilities to
deal with particular linguistic phenomena, such as gradable adjectives [75]
and other basic constructions [19]. Despite their success in broader
evaluation tasks, VLMs are often shown to lag behind humans on such targeted
linguistic evaluations, leaving open the question of how closely their
semantic representations are aligned with human ones.

Next to their applications in AI, VLM representations have also proven to be
surprisingly effective for modelling brain activity involved in visual
processing. For instance, Wang . [102] show that image embeddings from CLIP (a
model trained to match embeddings of images and corresponding text captions in
a joint space) are significantly more predictive of high-level visual cortex
activity in picture viewing than models trained on simpler image-label pairs
or text only. Similarly, Oota . [72] found that some multimodal transformers,
and VisualBERT [60] in particular, outperform vision-only and text-only models
in neural predictivity on two picture viewing datasets, across several brain
regions. Regarding language processing, Pezzelle . [77] evaluated word
representations extracted from several VLM architectures against human
semantic judgments, including the word similarity and relatedness benchmarks
mentioned in Section 1.2. VLM word representations aligned better with human
judgments than representations from text-only models, although only on
concrete word pairs. Furthermore, Zhuang . [108] evaluated a set of language-
only, vision-only, and vision-and-language architectures on several measures
of word learning over model training, and found that visual information is
beneficial only when small amounts of training data are available.

Taken together, the findings reported above indicate that multimodal VLMs seem
substantially more effective as predictors of visual cortex activity in
picture viewing. On the other hand, it remains to be seen whether the
integration of visual and linguistic information as operationalized in current
VLM architectures is equally well-aligned with conceptual representations
involved in human language processing.

###  1.4 Research questions and hypotheses

The following three research questions guide the work we present in this
study.

  * •

Are VLM representations better models of brain activity involved in concept
processing during language comprehension than their unimodal counterparts?

As covered above, human concept representations seem to incorporate both
visual and linguistic information. Since VLMs are trained to align input from
visual and linguistic streams, we hypothesise that they will have an advantage
over unimodal DNNs as models of brain activity during concept processing. The
effectiveness of language-infused DNN representations to model brain activity
in the visual cortices has been documented by several recent studies [21, 29,
102]. However, although grounding in extra-linguistic experience is considered
fundamental for language comprehension according to embodied theories, the use
of VLM embeddings to model brain activity in language-related brain networks
remains largely unexplored. Our main experiment in Section 3 addresses this
issue by investigating the degree of alignment of multimodal and unimodal DNN
representations with brain activity during concept processing in vision and in
language brain networks.

  * •

Which ‘family’ of VLMs best approximates human concept representations?

Although evidence from neuroimaging supports that human concept
representations encoded in brain activity contain information from multiple
modalities, much remains to be discovered about the processes giving rise to
such joint encodings. Here we compare a variety of models implementing
different strategies to operationalize the integration of linguistic and
visual features, to explore which of these strategies might lead to more
brain-like concept representations. Extending earlier work by Oota . [72] and
Pezzelle . [77], we consider three families of multimodal vision-and-language
DNNs: contrastive VLMs, visuo-linguistic encoders, and generative VLMs. As
described in Section 2, all three families are based on the transformer
architecture [101] but differ in the modules they use to align visual and
linguistic input, and in their training objectives. To our knowledge, this is
the first study comparing the brain alignment potential of generative VLMs
across visual and linguistic processing, which we tackle as part of our main
experiment in Section 3. As mentioned earlier, generative autoregressive
language-only architectures are highly predictive of human cognitive signals
in language processing [85], but it remains to be seen if this advantage also
holds for generative architectures with a vision-and-language backbone. Here
we compare generative VLMs to other VLM architectures that have received more
attention to date. We choose a selection of pre-trained model checkpoints
which have already been included in various benchmark comparisons throughout
the literature, and evaluate their alignment to brain activity in a zero-shot
fashion, enabling comparisons between our brain-similarity analyses and other
evaluations of VLM performance.

  * •

Do the DNN models that better approximate brain activations during concept
processing also exhibit higher alignment with behavioural semantic similarity
judgments?

There is evidence suggesting that DNN representations that are well aligned
with explicit similarity judgements are also aligned with neuro-imaging data,
regarding both language-only [e.g., 85] and vision-only domains [e.g., 65].
Pezzelle . [77] evaluated several language-only and vision-and-language models
on their alignment with semantic similarity judgments and documented an
advantage of VLMs over their text-only counterparts. Given existing evidence
regarding unimodal models, we hypothesise that VLMs that are more predictive
of brain activity will be more aligned with semantic similarity judgments. We
test this hypothesis in an additional analysis we carry out in Section 4,
where we extend the study by Pezzelle . [77]. This allows for a more thorough
comprehension of which aspects of human semantic knowledge are reflected in
the representations derived from VLMs.

## Data Availability

The code to reproduce our results and replicate our findings is available at
the following public GitHub repository: https://github.com/mdhk/Multimodal-
Brain. The datasets of brain responses and behavioural judgments used in our
experiments were introduced in previous work (see Sections 3.1.1 and 4.1) and
are already publicly available. The links to these datasets are nevertheless
included in the repository as well.

##  2 Deep Neural Network Models

For our experiments, we employed three main types of models that differ in the
nature of the input they have been trained on: a set of VLMs trained on
related visual and textual input, a set of language-only models trained
exclusively on text, and a set of vision-only models trained solely on images.
In all cases, we use the models off-the-shelf as pre-trained by their
creators. This choice, while preventing control over training data, is
motivated by our goal to test the potential of the specific deep learning
systems that have led to powerful performance in downstream AI applications as
models of human brain representations.

Below we outline the models’ main features. Additional details about the
architectures and the specific implementations we used are provided in A.

###  2.1 Vision-and-language models

VLMs can handle mixed inputs that include both text and images thanks to their
pre-training on data from both modalities. In this sense, the pre-training
data VLMs learn from is richer than the pre-training data unimodal models are
exposed to. A question we aim to investigate here is whether this richer input
translates into representations that are more similar to those developed by
humans, who are typically exposed to stimuli from different modalities when
learning concepts.

While most VLMs are Transformer-based, the modules making up their
architecture and the way in which they achieve visuo-linguistic integration
may vary. Based on these elements, it is possible to identify three main types
or ‘families’ of VLMs, i.e., contrastive vision-and-language models, visuo-
linguistic encoders, and generative visuo-linguistic models. A schematic
overview of the three model families is provided in Figure 1.

Figure 1: Overview of the core components constituting the three families of
VLMs described in Section 2.1. The diagrams exemplify the input provided to
the models in the picture condition. The model outputs at the bottom of each
diagram are the representations we compare against brain responses through
representational similarity analysis. Best viewed in colour.

Contrastive VLMs The contrastive vision-and-language models we experiment with
are CLIP [81] and ALIGN [47]. These models encode image and text separately
with two dedicated transformer-based modules and are pretrained with a
contrastive loss which maximises the similarity between the image and text
embeddings of matching image-text pairs. The CLIP implementation we deploy has
ViT [31] as image encoder and a language transformer as text encoder. As for
ALIGN, it uses EfficientNet [95] to encode images and BERT [27] to encode
text.

Visuo-linguistic encoders The visuo-linguistic encoders employed in our
experiments are VisualBERT [60] and LXMERT [93]. Their distinctive feature is
a specific module that uses attention mechanisms to learn relations between
visual features extracted with an object detector – in the implementation we
use, Faster RCNN [84] – and text embeddings. In VisualBERT, language
processing and multimodal integration are performed by the same BERT-based
module, whereas LXMERT has separate transformer-based modules to encode text
and perform multimodal integration. Both models output text representations
that incorporate information from the visual input and are – in this sense –
multimodal.

Generative VLMs For this family of VLMs, we considered IDEFICS [57] and BLIP-2
[59]. Their core module is a large language model (LLaMA[99] in IDEFICS and
OPT [107] in BLIP-2) that is pretrained autoregressively and can therefore
generate textual outputs based on the provided textual prompts and visual
inputs. In order for their outputs to be informed by the visual inputs, they
are equipped with visual encoders (ViT [31] in BLIP-2 and OpenCLIP [20] in
IDEFICS) and some architecture-specific components (see [57] and [59] for
further details) which learn to attend to the visual information useful for
text generation. In both generative models, the LLM decoders are tasked with
mapping multimodal information to linguistic output: in the case of BLIP-2,
the LLM decoder integrates text input with the query embeddings derived from
the visual input by its Q-Former component, and in the case of IDEFICS, the
LLM decoder integrates text and visual tokens provided in the prompt.

###  2.2 Language-only models

We experiment with a simpler distributional semantic model (GloVe), one
encoder-only language model (RoBERTa), and a set of generative large language
models (OPT, LLaMA).

GloVe [73] is a type-based model that learns word embeddings based on co-
occurrence patterns. What distinguishes it from all the other visuo-linguistic
and language-only models we deploy is that the representations it produces are
not contextualised, i.e., they do not distinguish between the different
meanings a word can take based on the context. Nevertheless, they have been
shown to moderately correlate with brain responses as described in the
Introduction, so we included them as they still provide a useful reference.

RoBERTa [62] is a transformer-based encoder-only model building on the BERT
architecture. Its pre-training with bi-directional attention enables it to
develop robust word representations. Its architecture is similar to that of
VisualBERT’s language model. A comparison between the two can, therefore,
provide some insight into the complementary role played by architecture and
type of pretraining (unimodal vs. visuo-linguistic).

OPT [107] and LLaMA [99] are decoder-only large language models (LLMs)
pretrained autoregressively. Their key feature is that they are able to
generate natural-language outputs based on the input prompts. We chose these
specific families of LLMs as OPT is the language model used in BLIP-2 and
LLaMA is used in IDEFICS.

###  2.3 Vision-only models

The vision-only models used in our experiments are Faster RCNN [84] and ViT
[31]. Faster RCNN is an object detector based on a convolutional architecture
that generates a fixed number of region proposals. It outputs a set of
features per each region proposal. ViT is a transformer-based model
architecture pretrained on image classification. A key difference between
Faster RCNN and ViT is that the former relies on convolution operations that
progressively extract simple-to-complex visual patterns, whereas ViT divides
images into small patches – which can be thought of as ‘visual tokens’ – and
learns the relations between them with attention mechanisms. Our motivation
for focussing on these architectures as visual-only baselines is that the
visual features we passed to VisualBERT and LXMERT were extracted with Faster
RCNN and ViT-based image encoders are employed in CLIP, BLIP-2, and IDEFICS.

##  3 Alignment of Brain Activations and Representations by Unimodal and
Multimodal Deep Neural Networks

In this section, we present our main experiment, where we systematically
compare brain activity in different functionally localised brain networks to
the representations learned by vision-and-language, language-only, and vision-
only DNNs. Details about the methods we employ are provided in Section 3.1 and
our results are reported in Section 3.2.

Figure 2: Overview of the experimental setup in the sentence (top) and picture
(bottom) condition, including the stimuli used in embedding extraction for the
three families of VLMs (see Figure 1). Each concept word embedding is averaged
over the model’s hidden states across 6 instances of processing a concept word
(contextualized by different sentences or pictures), as are the fMRI
activation vectors. Best viewed in colour.

###  3.1 Methods

####  3.1.1 Data and materials

We evaluated models’ alignment to brain responses on the dataset introduced by
[74]. Specifically, we focussed on part of the data used in their Experiment
1, which consists of voxel-wise fMRI activations collected while participants
were exposed to words representing specific concepts in different conditions.
The dataset consists of 180 words, including different parts of speech (128
nouns, 22 verbs, 29 adjectives and adverbs, and 1 function word, see B for the
full list). We focussed on two experimental conditions: a language-only
sentence condition and a multimodal picture condition. In the sentence
condition, each word appeared boldfaced in the context of a sentence that made
the relevant concept (i.e., the target sense of the word) salient. In the
picture condition, each word was presented together with an image representing
some feature(s) of the relevant concept. 16 participants were scanned while
viewing the words in these two conditions. In both cases, they were invited to
think about the meaning of the target concept in the context where it
appeared, i.e., the accompanying sentence or image. Each concept word was
presented 6 times per condition, in the context of 6 different sentences and 6
different images, and the voxel activations for each participant were averaged
across the six presentations of the same word per condition.

We focussed on two functionally localized networks targeted by [74], involved
in either linguistic or visual processing: the Language network [34] and the
Visual network [80, 17]. Anatomical regions of interest included as part of
each of these functional networks are listed in Table 1. We present results
considering bilateral activity for the visual network, while for the language
network results are reported separately for the left hemisphere (Language-LH)
and the right hemisphere (Language-RH), as done by [74].

Network | Regions of Interest
---|---
Language | \- Posterior temporal lobe | \- Inferior frontal gyrus
| \- Anterior temporal lobe | \- Middle frontal gyrus
| \- Angular gyrus | \- Orbitan inferior frontal gyrus
Vision | \- Parahippocampal place area | \- Superior temporal sulcus
| \- Retrosplenial cortex | \- Fusiform face area
| \- Transverse occipital sulcus | \- Occipital face area
| \- Lateral occipital area | \- Extrastriate body area
Table 1: Brain regions of interest (ROIs) corresponding to our investigated
functionally localised language and vision networks.

####  3.1.2 Extraction of DNN representations

To extract DNN representations that we can compare to the recorded brain
activity, we feed to the DNNs the same stimuli that were presented to the
participants in the experiments by [74]. Different DNN models require slightly
different inputs, as we explain in detail in this section.

##### Sentence condition

The stimuli we passed to the analysed models in this condition were the
sentences read by participants during the sentence-condition experiments by
[74]. Our goal when extracting model representation was to isolate the part of
the output that specifically corresponded to the target word. Since in this
condition the stimuli are exclusively linguistic, we leave aside the vision-
only models and restrict our attention to the language-only and vision-and-
language models.

Regarding the language-only models, for the transformer-based models (OPT,
LLaMA, and RoBERTa), we provided each stimulus sentence as input to the model
and extracted the representations of the target word from the models’ hidden
states.111For all models, if the target word is made up of several tokens, we
average over token representations. As for GLoVe, given that it provides type-
based decontextualised representations, we simply selected the pre-trained
GloVe embedding corresponding to each target word.

Regarding the contrastive VLMs (CLIP and ALIGN), sentences were passed to
their language encoder and the target word representations were extracted as
in the transformer-based language models. The procedure was similar for
VisualBERT. Due to technical reasons, the other three multimodal models
require including some visual input in addition to text. To make this input as
neutral as possible, together with each sentence, we passed to the models an
image made up of random noise.222We created a different random noise image for
each sentence. The code to generate these images is included in the materials
released with this paper. The target word representations were extracted from
the hidden states of the language encoder in the case of LXMERT and
VisualBERT, while for the generative VLMs (BLIP-2 and IDEFICS) the word
representations were extracted from the language modelling decoder.

To obtain a final representation for each concept word that can be compared to
brain activations, we averaged the representations extracted from the 6
sentences where the word appeared (except for GloVe, where sentences are not
considered). Furthermore, for the models where the architecture allows it (all
except CLIP, ALIGN, and GloVe), we repeated the procedure described above for
each model layer.

##### Picture condition

In the Picture condition of the Pereira . experiment, participants saw each
word in isolation together with an image. To create an equivalent condition
for our models, we therefore extracted representations for all model types:
vision-and-language, language-only, and vision-only. For the language-only
models, word representations were extracted following the same procedure
described for the sentence condition, but only inputting the target word (the
obtained representations are therefore not linguistically contextualised, and
different from those obtained when the word is processed by the models in the
context of a sentence, except for GloVe). For the VLMs, the procedure was
again the same as in the sentence condition, but in this case the image
accompanying the word was fed to the models instead of a noise image. For the
vision-only models, only the image was passed as input to the model. For
Faster RCNN, we averaged the features corresponding to the different bounding
boxes extracted for the image; for ViT, image representations were computed by
averaging the hidden states of the model’s last layer. To obtain a final
representation for each concept word that can be compared to brain
activations, we then averaged the extracted representations across the 6
images per word. For each model architecture that allowed extracting layerwise
representations, we computed layerwise embeddings.

####  3.1.3 Evaluation

We evaluated the models’ alignment to fMRI data with Representational
Similarity Analysis [RSA, 54]. This method quantifies the similarity between
representational geometries by correlating distances between stimulus
representations. More specifically, in our experiments we considered cosine
distances between stimulus embeddings derived from the tested models and
cosine distances between fMRI responses to different stimuli. We populated two
representational dissimilarity matrices (RDMs) with brain-derived and model-
derived pairwise distances and then quantified their representational
similarity (RS) using Spearman’s correlation coefficient. A significant
correlation between RDMs suggests that there is alignment between brain-
derived and model-derived representational geometries.

We report the distribution of participant-level RS values obtained for each
brain network, using representations from each model. When multiple RS
values—corresponding to different layers—were available for the same model, we
report results for the layer that yields the best correlation. For the
language areas, we defined the best layer as the one showing the highest
similarity to neural activity from both LH (left-hemisphere) and RH (right-
hemisphere) areas combined. We then report similarity scores separately for
the RH and LH language regions.

##### Random baseline

To obtain a baseline for the representational similarity, we computed
alignment against shuffled brain responses, i.e., a condition where each
concept was associated with a randomly chosen fMRI response from a non-
matching concept. We again computed RS separately for each participant. For
visualisation purposes, in this case we report the average baseline
correlation rather than the full distribution. To evaluate whether the mean of
the Spearman’s correlations in the matching-brain-data condition was
significantly different from the random baseline, we conducted one-sided _t_
-tests for related samples. Our threshold for determining statistical
significance was $p<0.05$.

##### Noise ceilings

Finally, we computed noise ceilings for each brain region by averaging the
correlations between a single-participant response and the average fMRI
response across participants, including (upper bound ceiling) or excluding
(lower bound ceiling) the single participant. These noise ceilings are
intended to provide two pieces of information. First, they deliver an
indication of which performance should be expected from a ‘true model’ given
the noise present in the available data (assuming that the true model should
generate representations that are equal in quality to human data for
measurements of similarity to fMRI responses). Second, noise ceilings can
reveal how precise the measurement of brain activity is. In this sense, a
noise ceiling approaching 1 indicates that the measurement is extremely
precise, whereas a noise ceiling close to 0 suggests a lower sensitivity
(i.e., signal-to-noise ratio) in the fMRI data [70].

###  3.2 Results

We use RSA to compare alignment to brain activity for our set of embeddings
extracted from different DNN models. Below we report our model-brain alignment
patterns for the sentence condition first, followed by the picture-only
multimodal condition.

Figure 3: Representational similarity to fMRI data in the sentence condition.
The numbers in brackets refer to the layer being considered, i.e., the one we
considered to compute the reported results. Best viewed in colour. Figure 4:
Representational similarity to fMRI data in the picture condition. The numbers
in brackets refer to the best model layer. Best viewed in colour.

####  3.2.1 Main trends in the sentence condition

The results obtained in the sentence condition are summarized in Figure 3,
which includes one plot per brain area. The plots display the distribution of
representational similarity scores for the 16 participants in the experiment,
i.e., the distribution of correlation scores between the brain activations of
each participant and the representations extracted from each model.

The highest correlations across models were observed in the Language-LH
regions, which is not surprising given that the stimuli in this experimental
condition are exclusively linguistic. However, the representations learned by
VLMs exhibit an advantage over the representations by language-only models. In
Language-LH, only LXMERT, VisualBERT, and GloVe exhibit alignment to brain
activity that is significantly higher than the random baseline, with LXMERT’s
representations being the best-aligned ones ($\rho=0.062$), followed by
VisualBERT ($\rho=0.046$). Moreover, after the Language-LH regions, the second
strongest correlations are observed in the Visual regions, with LXMERT being
the best-aligned model in this case as well ($\rho=0.047$), followed by CLIP
($\rho=0.046$) and VisualBERT ($\rho=0.043$).

In contrast to the Language-LH and Visual regions, where VLMs have an
advantage, in the Language-RH region the language-only model LLaMA yields the
highest correlations ($\rho=0.032$), followed by the generative VLM IDEFICS
($\rho=0.028$), which has LLaMA as its language encoder. The language-only
model OPT also exhibits competitive correlations.

It is worth noting that LLaMA and OPT (as well as RoBERTa) yield significant
correlations in Visual regions as well (albeit lower than multimodal models),
while they do not in Language-LH regions. As mentioned above, among the
language-only models, only GloVe representations exhibit significant alignment
in Language-LH regions. This is somewhat counterintuitive given that in this
condition target words are presented within a sentential context and, unlike
the representations by the other language-only models, GloVe embeddings are
not contextualised.

Finally, in this experimental condition, the representations by ALIGN and
BLIP-2 do not display significant correlations with any brain area.

####  3.2.2 Main trends in the picture condition

The results obtained in the picture condition are summarized in Figure 4.
Again, the figure includes one plot per brain region, with the distribution of
correlation scores across participants for each model. Besides vision-and-
language and language-only models, in this experimental condition, we also
include results with vision-only models.

The highest correlation scores are found in the Visual regions, where the
representations by the vision-only model Faster RCNN yield the strongest
alignment ($\rho=0.253$), closely followed by VisualBERT’s multimodal
representations ($\rho=0.249$). Correlations in the Language-LH regions are
also relatively strong. As in the sentence condition, in this region, the VLM
representations have an advantage. More specifically, the highest correlations
were observed for the contrastive VLMs CLIP and ALIGN ($\rho=0.142$ and
$\rho=0.133$ respectively), closely followed by the visuo-linguistic encoders
LXMERT ($\rho=0.127$) and VisualBERT ($\rho=0.127$).

A pattern that emerged consistently across all the brain areas was the high
alignment of vision-only DNN representations. Visual representations
outperformed language-only ones by a significant margin. Even in Language-LH
regions where multimodal models yield the highest correlations, the alignment
of vision-only models is still highly competitive. In Language-RH, the vision-
only Faster RCNN and the vision-and-language CLIP models yield similarly
strong correlations ($\rho=0.122$ and $\rho=0.09$ respectively). In both
Language-LH and Language-RH, the language-only DNNs result in lower
correlations, which are nevertheless still significant for GloVe and RoBERTA,
as well as for OPT in the case Language-LH.

Finally, in this experimental condition, the representations by the generative
VLM IDEFICS do not exhibit significant correlations with any brain area.

##  4 Additional Analysis: A Comparison between DNN Alignment with Behavioural
Responses vs. Brain Activations

Brain and behavioural responses reflect human concept representations in two
complementary ways. Our goal was thus to gain a more comprehensive insight
into the overall ability of DNN models to capture such knowledge by uncovering
similarities and differences between alignment patterns occurring when
modelling data of the two different kinds. To achieve this, we conducted an
additional analysis where we measured the alignment between model
representations and human judgments about the degree of semantic
similarity/relatedness between pairs of words. More specifically, we focussed
on two main aspects. First, we looked at how different DNN models compare in
terms of their brain vs. behavioural alignment. Second, we investigated how
the alignment patterns vary within-model, i.e., to what extent model layers
that are more aligned with neural activity are also aligned with behavioural
responses.

Comparing patterns across various layers has become a standard in the era of
Transformer-based architectures, as the representations learned by a DNN in
the early stages have been shown to differ from those learned at later stages.
Moreover, earlier layers of text-only DNNs have been shown to correlate better
than later ones with human semantic similarity and relatedness judgments [11],
suggesting that lexico-semantic information is better represented at early
processing stages. As for vision and language DNNs, these patterns appeared to
be less clear-cut and much more dependent on the degree of multimodal
integration brought forward by each specific architecture at various
processing stages [18, 77]. Comparing the pattern across layers for human
brain activations and semantic judgments can therefore provide interesting
insights both into the type of representations processed by the models at
different stages and the information encoded by the two analyses.

###  4.1 Methods

To measure the alignment between DNN model representations and behavioural
responses, we build upon the work of [77], who model similarity/relatedness
between pairs of words, as judged by participants, presented out of context.
We compare the ability of the DNN models to align with these behavioural
judgements to their ability to align with brain responses in the picture
condition. We focus on the picture condition because it is the truly
multimodal one and, similarly to the behavioural setting, it also presents
words out of linguistic context. In the picture condition of the Pereira
dataset, a particular word sense is evoked by presenting the word together
with an image, six times. In the behavioural setup, in contrast, the pairs of
words to be judged as semantically similar/related are presented in isolation,
without linguistic or visual context. Consistently, participants are likely to
judge these words more generally, assumed to focus on the concepts elicited by
these words more generally, without explicitly restricting attention to a
particular word sense.

To extract representations for the word pairs from multimodal DNN models, [77]
created a dataset, VICO, containing over 1M image-text pairs. Each text –
caption or sentence – where a target word appears is inputted into the model
together with the corresponding image and the contextualised representation
for the target word is extracted. The final, single representation for each
word in a word pair is then the average of all the extracted contextualised
representations for that word. Thus, these representations model a scenario
where judging pairs of words in isolation is assumed to recruit generic
multimodal conceptual knowledge, captured by averaging over multiple image-
text pairs exemplifying how the word is used in context. Alignment with human
judgements is computed as the Spearman’s correlation between such judgements
and the cosine similarity between model representations (i.e., akin to
computing representational similarity).

We follow this approach to extract DNN model representations from a subset of
the models examined in the main study (Section 3): the visuo-linguistic
encoders LXMERT and VisualBERT – originally used by [77] – plus the generative
VLMs IDEFICS and BLIP-2, and the language-only models RoBERTa, LLaMA, and
OPT.333Since in this study we are interested in exploring model layer
patterns, we leave aside CLIP, ALIGN, and GloVe, whose model architecture does
not allow such an analysis. For the behavioural responses, we consider two of
the datasets used by [77], both including words belonging to different parts
of speech (i.e., nouns, verbs, and adjectives): SimLex999 [40], which consists
of similarity judgements, and MEN [15], which consists of relatedness
judgements. Besides including different measures of semantic association –
similarity vs. relatedness – these two datasets also differ regarding the
degree of concreteness of the word pairs in them: based on the concreteness
ratings by [16],444We used the same methodology described in [77] to compute a
word pair’s concreteness. MEN includes fairly concrete concepts, i.e., that
can be more easily depicted in an image, while SimLex999 includes more
abstract ones (see Table 2 for some descriptive statistics of the datasets and
the specific subsets we used). By focussing on these two datasets, we ensure
that the results of our analysis can be representative of various types of
behavioural judgments. To ensure the quality of the representations obtained,
we only extract embeddings for those words in MEN and SimLex999 that are
present in at least 20 image-text VICO samples. For computational efficiency,
we average over a maximum of 200 occurrences per word.

| Original | Our subset
---|---|---
Dataset | #pairs | concreteness | #pairs | concreteness
MEN | 3000 | 4.41 | 2710 (90%) | 3.79
SimLex999 |  999 | 3.61 |  647 (65%) | 3.31
Table 2: Number of word pairs and their average concreteness scores in the
original datasets of human judgements and in the subset used in our
experiments (the proportions compared to the original datasets are given in
brackets). Figures for the original datasets are retrieved from [77].

###  4.2 Results

Our findings are summarized in Figure 5 and Figure 6. Figure 5 shows – for
each DNN model – the layer-wise representational similarity values between the
model representations and the behavioural judgments on the one hand and the
brain responses in the picture condition on the other hand (the two
corresponding plots are displayed side by side per model). For simplicity,
here we report representational similarity for RH and LH language regions
jointly.

As already reported in Section 3.2.2, among the subset of DNN models examined
in this study, those that yielded the strongest alignment with brain responses
in the picture condition were the visuo-linguistic encoders VisualBERT and
LXMERT. As for the behavioural judgments, the models that best approximate
them are VisualBERT ($\rho=0.739$ on MEN and $\rho=0.562$ on SimLex999) and
RoBERTa ($\rho=0.737$ on MEN and $\rho=0.547$ on SimLex999), closely followed
by IDEFICS ($\rho=0.735$ on MEN and $\rho=0.529$ on SimLex999) and OPT
($\rho=0.717$ on MEN and $\rho=0.537$ on SimLex999) (see Figure 6).
VisualBERT, therefore, was found to be the most aligned model to both brain
responses and behavioural data. Another interesting finding is that, while
generative models lag significantly behind other model types when considering
brain alignment, most of those we tested – with the exception of BLIP-2 –
exhibit strong correlations with behavioural judgments.

A trend common to all the tested models was higher alignment with the MEN
judgments (which concern words with higher concreteness scores on average)
compared to those from SimLex999. In our results, this trend holds for both
vision-and-language and language-only models, in contrast to [77], who
observed an advantage of multimodal models on MEN.

Focussing on how brain vs. behavioural alignment varies within-model, the
patterns shown in Figure 5 and the overview of best layers reported in Table 3
suggest that the layers which correlate the most with brain activity do not
generally overlap with those strongly correlating with human judgments. For
example, in IDEFICS behavioural alignment peaks in the early layers, whereas
brain alignment reaches its maximum around the mid layers (for vision areas)
and last layers (for language areas). Conversely, in OPT correlations with
behavioural judgments are stronger after the twentieth layer, while brain
alignment peaks earlier – around the fourteenth layer. One exception to this
trend is RoBERTa, where both brain and behavioural alignment are strongest in
the early layers.

Figure 5: Spearman’s correlations quantifying the representational similarity (RS) of model embeddings with behavioural judgments (Behaviour) and fMRI responses (Brain) across different model layers (beware of the range difference in the $y$-axis between the Behaviour and Brain plots). Brain responses are averaged across participants. The plots corresponding to VLMs are on the left-hand side and have a darker background. The plots for the language-only models are on the right-hand side, placed side by side with their vision-and-language counterparts. Note that representational similarity for RH and LH language regions is displayed jointly. Best viewed in colour. Figure 6: Representational similarity (RS) between model embeddings and behavioural judgments, quantified as Spearman’s correlation coefficient. For each model, we show the most aligned layer. Best viewed in colour. | Brain | Behavioural
---|---|---
Model |  Vision | Language |  MEN | SimLex999
LXMERT | 14 | 14 | 1 | 10
VisualBERT | 1 | 4 | 1 | 8
LLaMA | 2 | 26 | 7 | 17
OPT | 14 | 14 | 23 | 23
RoBERTa | 2 | 2 | 1 | 4
BLIP-2 | 32 | 32 | 33 | 33
IDEFICS | 14 | 27 | 7 | 7
Table 3: Overview of models’ most aligned layers to brain regions (vision and
language) and to behavioural judgments (MEN and SimLex999 datasets). Cell
colors indicate whether the best layer is among the early, intermediate or
later layers. Best viewed in colour.

##  5 Discussion

The results of our experiments are generally well-aligned with the initial
hypotheses and provide interesting insights into our research questions. In
the following, we discuss these results in detail and interpret them against
previous evidence and the relevant theories considered. We also discuss the
implications that these results bring toward a better understanding of the
semantic abilities of DNN models, with a particular focus on the multimodal
ones here explored.

###  5.1 Are VLM representations better models of brain activity involved in
concept processing during language comprehension than their unimodal
counterparts?

The results of our experiments indicate that the representations obtained with
multimodal DNNs are, overall, more aligned with human brain activations than
those obtained with their unimodal counterparts. This confirms our main
hypothesis and reveals that considering the semantic traits encoded in the
visual modality leads to semantic representations that better account for
human semantic processing. Below, we zoom in on each of our key findings.

##### Sentence condition: Multimodal models outperform the unimodal
counterparts in visual and LH linguistic regions

In the sentence condition, multimodal models yielded the highest
representational similarity in LH-language regions (where LXMERT and
VisualBERT had the highest correlations) and in the visual ones (where the
most brain-aligned models were LXMERT and CLIP). It is worth noting that this
superiority of multimodal models was documented in an experimental condition
where no meaningful visual information was passed to the models. We interpret
these results as indicating that the advantage of multimodal representations
is not – or at least not only – due to the input they receive at inference
time. Instead, their superior performance might be driven by their massive
multimodal pretraining, which enables them to develop richer, more human-like
semantic representations.

Interestingly, in previous work, models such as LXMERT and the like were not
found to be better at natural language understanding than standard language-
only models when evaluated on downstream semantic tasks [94]. This suggests
that the representations learned by these models may be more cognitively sound
but not necessarily more effective for applications. Furthermore, downstream
NLP tasks default to the latest representations learned by the models (i.e.,
the last layer), which are not always and necessarily the most brain-aligned
representations according to our results. We will return to these points
later.

##### Picture condition: Multimodal embeddings outperform unimodal ones in LH
linguistic regions

Similar to what we documented in the sentence experimental condition, in the
picture condition we also found that the most predictive models for the LH
linguistic region were multimodal (the contrastive models CLIP and ALIGN).
Interestingly, the advantage of multimodal models over the language-only ones
in this brain region is substantial, and much more pronounced than what we
observed in the sentence condition. This is likely because, in the picture
condition, multimodal models can effectively exploit the semantic information
provided by the visual input, besides leveraging the multimodal pretraining.
This is especially evident for CLIP and ALIGN, whose LH-language-region
alignment increases dramatically in the multimodal condition, where their
visual encoder (not only their language one) is used.

##### Picture condition: Vision-only models exhibit substantial alignment
across brain regions

A surprising pattern that emerged consistently in this condition across all
brain areas was the relatively high alignment of vision-only DNN
representations. Our results show that vision-only DNNs are better models of
representations in visual brain regions than VLMs. This finding contrasts with
results reported by Wang . [102], who found that a model like CLIP (trained
contrastively with language supervision) was more predictive of activation in
visual brain regions than vision-only DNNs in an experimental condition where
the participants viewed images of natural scenes. The authors concluded that
the abstract relational information encoded in image captions (used to train
CLIP) better captures the high-level visual representations in the visual
cortex. We hypothesise that our results show the opposite trend because in the
picture condition of the Pereira dataset we use, images are viewed together
with a word, with the focus being on a single concept. Thus, our results
suggest that, in such experimental conditions, brain activity in visual
processing is better modelled by the lexico-semantic information encoded by
vision-only models trained on object classification tasks than by VLMs trained
with more complex linguistic expressions.

The alignment of vision-only DNN representations remains substantial (and much
stronger than that of language-only models) in brain regions related to
language processing. This is the case even in the Language-LH region, where
VLMs such as CLIP are the ones showing the strongest alignment. Again, a
possible explanation for these results might be that, despite vision-only
models not having direct access to semantic information in the form of text,
they still acquire semantically meaningful categories as a result of their
pretraining on object classification tasks. We speculate that the activation
of similar categorical information in language brain areas might be
responsible for the high brain alignment of vision-only DNN representations.

###  5.2 Which ‘family’ of VLMs best approximates human concept
representations?

Our results reveal significant differences across families of vision-and-
language models and, in some cases, even between architectures from the same
family. In general, we observed that generative VLMs – which were not tested
in previous studies – tend to be less brain-aligned than other VLMs. We
discuss our findings in more detail in the following paragraphs.

##### Sentence condition: Multimodal encoders strongly align with neural
activity in the visual and LH linguistic regions

VisualBERT and LXMERT were found to significantly outperform VLMs from other
families in the LH linguistic regions and in the visual ones. Their
superiority over contrastive models may be due to the type of linguistic data
they are pretrained on. Contrastive models are typically pretrained on images
paired with image captions or ‘alt-texts’, while VisualBERT and LXMERT learn
from visual question-answering data in addition to captions. The latter
appears to provide richer information that positively impacts the quality of
the representations learned by the models.

Regarding the advantage of visuo-linguistic encoders over generative VLMs, one
potential explanation is that generative models privilege production over
representation, as pointed out in a few recent works [7, 69, 91]. Indeed, a
key feature of generative models’ autoregressive pretraining is that the
right-hand context is masked to prevent them from ‘seeing’ what they should
predict. This results in representations which are optimised for generation
but, at the same time, fail to encode the part of the context that models
cannot ‘see’.

##### Picture condition: Generative VLMs lag behind the other families of
multimodal models

While generative VLMs are progressively supplanting other types of vision-and-
language architectures due to their better performance in multiple downstream
tasks, this advantage is not reflected in enhanced alignment with neural
activation patterns in the picture condition. Indeed, the contrastive VLMs and
the visuo-linguistic encoders we tested outperform generative VLMs by a
significant margin across all brain regions. As mentioned above, this may be
due to the internal bias of generative models towards generation instead of
encoding.

###  5.3 Do the DNN models that better approximate brain activations during
concept processing also exhibit higher alignment with behavioural semantic
similarity judgments?

A few previous studies have shown that DNN representations that are aligned
with fMRI responses are also well aligned with behavioural responses [e.g.,
85, 65]. Consistently with this existing evidence and with our initial
hypothesis, our results showed that VisualBERT exhibited the strongest
alignment with both brain responses and behavioural semantic judgments. Yet,
this is not the case for other model types. Most prominently, generative VLMs
and language-only model representations tend to be much better aligned with
behavioural judgments than with fMRI responses.

A remarkable finding emerging from the analysis of layer-wise patterns is that
last-layer representations were rarely the most predictive ones, both for
brain responses and behavioural judgments. This is interesting because, while
studies on neural responses usually consider representations from multiple
layers, studies comparing model outputs against human behaviour often use
representations extracted from the last layer.

Such a methodological choice is generally driven by the implicit assumption
that, since last-layer representations are the ones containing all the
information that models need to perform downstream tasks, they should also be
the ‘best’ to compare against human behaviour. However, our results show that
this is rarely the case and suggest that early and intermediate model layers
contain information that not only is relevant for modelling human behaviour
but also becomes less present in later layers. Consequently, research on human
behaviour and human linguistic abilities might benefit from the adoption of
more systematic approaches considering different layers of the same
architecture.

##  6 Conclusion

In this study, we explored a set of research questions concerning VLMs’
ability to capture the conceptual information reflected in human brain
responses. First, we compared representations extracted from VLMs, language-
only and vision-only models in terms of their brain alignment. Our experiments
revealed that, overall, visuo-linguistic representations correlate with brain
responses more strongly than language-only and vision-only representations.
This result is particularly interesting if related to the neuroscience
literature. Indeed, a great deal of research documented that multimodality is
not only a feature of the environment where humans learn concepts, but also an
aspect playing a key role in how brains process external stimuli [28, 26, 33]
. In this sense, it stands to reason that model representations learned from
both texts and images are more brain-aligned than representations learned from
images or texts only.

Our second research question concerns how different families of VLMs compare
in their brain alignment, with a particular focus on generative VLMs. These
models were not investigated in previous neuroscientific work and have
recently exhibited striking abilities to produce linguistic outputs and answer
complex questions. Given the increased human-likeness of these models’
performance, it might be tempting to speculate that they also develop more
brain-aligned representations. However, our experiments revealed that
generative-VLM representations correlate less with brain responses compared to
representations extracted from previous-generation models. This is in line
with recent deep-learning results showing that models optimised for generation
tend to be sub-optimal for representation (or ‘encoding’) [7, 69, 91]. More in
general, these findings suggest that caution should be used when relating the
human-likeness of models’ performance to the human-likeness of their input
representations.

Finally, we investigated whether models’ alignment with brain responses is
reflected in their alignment with behavioural judgments. While VisualBERT was
found to correlate strongly with both, we observed that high behavioural
alignment does not often translate into high brain alignment. Furthermore, our
experiments reveal that last-layer representations (i.e., those routinely used
by these models to perform downstream tasks at which they excel) are rarely
the ones correlating best with human judgments. This has important
implications for approaches aiming to reproduce or model human behaviour and
human linguistic abilities (as opposed to achieving state-of-the-art
performance on downstream tasks), as it indicates that models’ early and
intermediate layers might contain ‘human-relevant’ information which has been
underexplored in previous studies.

## Acknowledgments

Marianne de Heer Kloots is funded by the Netherlands Organization for
Scientific Research (NWO), through a Gravitation Grant 024.001.006 to the
Language in Interaction Consortium. Anna Bavaresco and Raquel Fernández are
funded by the European Research Council (ERC) under the European Union’s
Horizon 2020 research and innovation programme (grant agreement No. 819455).

##  Appendix A Model Overview

This section provides additional details about the architectures and specific
implementations of the models used in our experiments. A summary of their key
features is also provided in Table 4.

Model |  Representations |  # Layers |  S |  P
---|---|---|---|---
CLIP |  Image and text embeddings (P), Hidden states (S) |  13 |  ✓ |  ✓
CLIP |  Image and text embeddings (P), Hidden states (S) |  13 |  ✓ |  ✓
LXMERT |  Language hidden states |  14 |  ✓ |  ✓
VisualBERT |  Hidden states |  12 |  ✓ |  ✓
BLIP-2 |  Hidden states |  32 |  ✓ |  ✓
IDEFICS |  Hidden states |  32 |  ✓ |  ✓
Faster R-CNN |  Features |  - |  ✗ |  ✓
ViT |  Hidden states |  13 |  ✗ |  ✓
LLaMA |  Hidden states |  33 |  ✓ |  ✓
OPT |  Hidden states |  33 |  ✓ |  ✓
RoBERTa |  Hidden states |  13 |  ✓ |  ✓
GloVe |  Hidden states |  - |  ✓ |  ✓
Table 4: Overview of the models used to extract representations for the
sentence (S) and picture (P) condition (✓indicates that the model was used in
the condition and ✗ that it was not). The Representations column provides
information about which component was used to derive model representations.
The # Layers column specifies the number of layers of the models. The value is
specified only for the models where using different layers was applicable.

###  A.1 Vision-and-Language Models

##### CLIP

[81] is a transformer-based multimodal model that encodes images and text
separately. The contrastive pre-training objective of this model maximises the
similarity between matching image-text pairs. The architecture is available
with different visual encoders. Here, we use the clip-vit-large-patch14
implementation by HuggingFace with the powerful ViT [31] as the visual
encoder.

##### ALIGN

[47] is a multimodal contrastive model pre-trained with the same objective as
CLIP. It employs EfficientNet [95] as a visual encoder and BERT [27] as a text
encoder. Its large pretraining corpus includes one billion pairs of images and
raw alt-text. We use the align-base version of the model from the HuggingFace
implementation.

##### LXMERT

[93] is a dual-stream multimodal encoder. It first encodes images and text
separately with dedicated transformer-based encoders and then learns relations
between image and text with a multimodal module pretrained with masked cross-
modal language modelling, masked object prediction, cross-modal matching, and
image question answering. We used lxmert-base-uncased implementation by
HuggingFace and extracted the visual features with a Faster
R-CNN555https://github.com/huggingface/transformers/tree/main/examples/research_projects/lxmert
[84] object detector.

##### VisualBERT

[60] is a single-stream multimodal encoder. Its core module is a BERT-based
transformer whose layers learn relations between text and image patches. It is
pretrained with image-conditioned marked language modelling and sentence-image
match prediction. We employed the visualbert-vqa-coco-pre implementation
available on HuggingFace.

##### BLIP-2

[59] consists of a frozen image encoder, a frozen large language model and a
Querying Transfomer (Q-Former), which is the only trainable module and uses
query vectors to retrieve relevant features from the image encoder. The
Q-Former is pretrained in two stages: (1) representation learning, where the
Q-Former learns to extract relevant information from the vision model, and (2)
generative pre-training, where the Q-Former is connected to an LLM and learns
to generate textual outputs conditioned on the query embeddings. In this
study, we used the blip2-opt-2.7b HuggingFace implementation with ViT-L/14
[81] as visual encoder and OPT-2.7B [107] as LLM.

##### IDEFICS

[57] is a large generative VLM with a Flamingo-like architecture [2]. Its main
components are (1) a vision encoder, (2) a perceiver resampler that takes as
input the visual features and outputs visual tokens, and (3) a frozen decoder-
based LLM enhanced with trainable cross-attention layers that enable the model
to attend to visual information when generating the output. In this study, we
used the idefics-9b HuggingFace implementation with ViT-L/14 as vision encoder
and LLaMA-65B as LLM.

###  A.2 Language-only Models

##### OPT

[107] is a family of decoder-only large language models. We used the opt-2.7b
implementation made available on HuggingFace, to match the LLM decoder
component of BLIP-2.

##### LLaMA

[99] is a set of decoder-only transformer-based LLMs. We used the llama-65b
implementation made available on HuggingFace, to match the LLM decoder
component of IDEFICS.

##### RoBERTa

[62] is an encoder-only language model building on the BERT architecture. We
used the roberta-base implementation made available on HuggingFace.

##### GloVe

[73] is a distributional semantic model for constructing word embeddings. We
used the pretrained word vectors obtained from 840B CommonCrawl tokens,
available from the GloVe repository666https://github.com/stanfordnlp/GloVe.

####  A.2.1 Vision-only Models

##### Faster R-CNN

[84] is an object detector based on a convolutional architecture that
generates region proposals. These regions are then used by Fast R-CNN for
object detection.

##### ViT

[31] is a transformer-based model architecture on image classification.
Variants of this architecture – with a different pertaining objective – are
used as visual encoders in CLIP, BLIP-2, and IDEFICS.

##  Appendix B Concepts from the Pereira Dataset

The full list of concept words from the Pereira dataset [74] is the following:

> Ability, Accomplished, Angry, Apartment, Applause, Argument,
> Argumentatively, Art, Attitude, Bag, Ball, Bar, Bear, Beat, Bed, Beer, Big,
> Bird, Blood, Body, Brain, Broken, Building, Burn, Business, Camera,
> Carefully, Challenge, Charity, Charming, Clothes, Cockroach, Code,
> Collection, Computer, Construction, Cook, Counting, Crazy, Damage, Dance,
> Dangerous, Deceive, Dedication, Deliberately, Delivery, Dessert, Device,
> Dig, Dinner, Disease, Dissolve, Disturb, Do, Doctor, Dog, Dressing, Driver,
> Economy, Election, Electron, Elegance, Emotion, Emotionally, Engine, Event,
> Experiment, Extremely, Feeling, Fight, Fish, Flow, Food, Garbage, Gold,
> Great, Gun, Hair, Help, Hurting, Ignorance, Illness, Impress, Invention,
> Investigation, Invisible, Job, Jungle, Kindness, King, Lady, Land, Laugh,
> Law, Left, Level, Liar, Light, Magic, Marriage, Material, Mathematical,
> Mechanism, Medication, Money, Mountain, Movement, Movie, Music, Nation,
> News, Noise, Obligation, Pain, Personality, Philosophy, Picture, Pig, Plan,
> Plant, Play, Pleasure, Poor, Prison, Professional, Protection, Quality,
> Reaction, Read, Relationship, Religious, Residence, Road, Sad, Science,
> Seafood, Sell, Sew, Sexy, Shape, Ship, Show, Sign, Silly, Sin, Skin, Smart,
> Smiling, Solution, Soul, Sound, Spoke, Star, Student, Stupid, Successful,
> Sugar, Suspect, Table, Taste, Team, Texture, Time, Tool, Toy, Tree, Trial,
> Tried, Typical, Unaware, Usable, Useless, Vacation, War, Wash, Weak, Wear,
> Weather, Willingly, Word.

## References

  * Abnar . [2018] abnarExperientialDistributionalDependencybased2018Abnar, S., Ahmed, R., Mijnheer, M. Zuidema, W.  2018\.  Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity Experiential, Distributional and Dependency-based Word Embeddings have Complementary Roles in Decoding Brain Activity.  Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018) Proceedings of the 8th Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2018) ( 57–66).  Salt Lake City, UtahAssociation for Computational Linguistics.  [2024-06-27]http://aclweb.org/anthology/W18-0107 10.18653/v1/W18-0107
  * Alayrac . [2022] alayrac2022flamingoAlayrac, JB., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y.others  2022\.  Flamingo: a visual language model for few-shot learning Flamingo: a visual language model for few-shot learning.  Advances in neural information processing systems3523716–23736.
  * Anderson . [2017] anderson2017visuallyAnderson, AJ., Kiela, D., Clark, S. Poesio, M.  2017\.  Visually grounded and textual semantic models differentially decode brain activity associated with concrete and abstract nouns Visually grounded and textual semantic models differentially decode brain activity associated with concrete and abstract nouns.  Transactions of the Association for Computational Linguistics517–30.
  * Antonello Huth [2024] antonelloPredictiveCodingJust2024Antonello, R. Huth, A.  202404\.  Predictive Coding or Just Feature Discovery? An Alternative Account of Why Language Models Fit Brain Data Predictive Coding or Just Feature Discovery? An Alternative Account of Why Language Models Fit Brain Data.  Neurobiology of Language5164–79.  [2024-06-07]https://doi.org/10.1162/nol_a_00087 10.1162/nol_a_00087
  * Barsalou [1999] barsalou1999perceptualBarsalou, LW.  1999\.  Perceptual symbol systems Perceptual symbol systems.  Behavioral and brain sciences224577–660.
  * Barsalou [2008] barsalouGroundedCognition2008Barsalou, LW.  2008\.  Grounded cognition Grounded cognition.  Annual Review of Psychology59617–645.  10.1146/annurev.psych.59.103006.093639
  * BehnamGhader . [2024] behnamghader2024llm2vecBehnamGhader, P., Adlakha, V., Mosbach, M., Bahdanau, D., Chapados, N. Reddy, S.  2024\.  Llm2vec: Large language models are secretly powerful text encoders Llm2vec: Large language models are secretly powerful text encoders.  arXiv preprint arXiv:2404.05961.
  * Bender Koller [2020] benderClimbingNLUMeaning2020aBender, EM. Koller, A.  202007\.  Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data Climbing towards NLU: On Meaning, Form, and Understanding in the Age of Data.  D. Jurafsky, J. Chai, N. Schluter J. Tetreault (), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics ( 5185–5198).  OnlineAssociation for Computational Linguistics.  [2024-01-01]https://aclanthology.org/2020.acl-main.463 10.18653/v1/2020.acl-main.463
  * Bergen [2012] bergen2012louderBergen, BK.  2012\.  Louder than words: The new science of how the mind makes meaning Louder than words: The new science of how the mind makes meaning.  Basic Books.
  * Bisk . [2020] biskExperienceGroundsLanguage2020aBisk, Y., Holtzman, A., Thomason, J., Andreas, J., Bengio, Y., Chai, J.Turian, J.  202011\.  Experience Grounds Language Experience Grounds Language.  B. Webber, T. Cohn, Y. He Y. Liu (), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) ( 8718–8735).  OnlineAssociation for Computational Linguistics.  [2024-07-03]https://aclanthology.org/2020.emnlp-main.703 10.18653/v1/2020.emnlp-main.703
  * Bommasani . [2020] bommasani-etal-2020-interpretingBommasani, R., Davis, K. Cardie, C.  202007\.  Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings Interpreting Pretrained Contextualized Representations via Reductions to Static Embeddings.  D. Jurafsky, J. Chai, N. Schluter J. Tetreault (), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics Proceedings of the 58th annual meeting of the association for computational linguistics ( 4758–4781).  OnlineAssociation for Computational Linguistics.  https://aclanthology.org/2020.acl-main.431 10.18653/v1/2020.acl-main.431
  * Bornstein . [2004] bornsteinCrossLinguisticAnalysisVocabulary2004Bornstein, MH., Cote, LR., Maital, S., Painter, K., Park, SY., Pascual, L.Vyt, A.  2004\.  Cross-Linguistic Analysis of Vocabulary in Young Children: Spanish, Dutch, French, Hebrew, Italian, Korean, and American English Cross-Linguistic Analysis of Vocabulary in Young Children: Spanish, Dutch, French, Hebrew, Italian, Korean, and American English.  Child Development7541115–1139.  [2024-06-25]https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1467-8624.2004.00729.x 10.1111/j.1467-8624.2004.00729.x
  * Boutonnet Lupyan [2015] boutonnetWordsJumpStartVision2015Boutonnet, B. Lupyan, G.  201506\.  Words Jump-Start Vision: A Label Advantage in Object Recognition Words Jump-Start Vision: A Label Advantage in Object Recognition.  Journal of Neuroscience35259329–9335.  [2024-06-27]https://www.jneurosci.org/content/35/25/9329 Publisher: Society for Neuroscience Section: Articles  10.1523/JNEUROSCI.5111-14.2015
  * Bruni . [2012] bruni2012distributionalBruni, E., Boleda, G., Baroni, M. Tran, NK.  2012\.  Distributional semantics in technicolor Distributional semantics in technicolor.  Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Proceedings of the 50th annual meeting of the association for computational linguistics (volume 1: Long papers) ( 136–145).
  * Bruni . [2014] bruni2014multimodalBruni, E., Tran, NK. Baroni, M.  2014\.  Multimodal distributional semantics Multimodal distributional semantics.  Journal of artificial intelligence research491–47.
  * Brysbaert . [2014] brysbaert2014concretenessBrysbaert, M., Warriner, AB. Kuperman, V.  2014\.  Concreteness ratings for 40 thousand generally known English word lemmas Concreteness ratings for 40 thousand generally known english word lemmas.  Behavior research methods46904–911.
  * Buckner . [2008] buckner2008brainBuckner, RL., Andrews-Hanna, JR. Schacter, DL.  2008\.  The brain’s default network: anatomy, function, and relevance to disease The brain’s default network: anatomy, function, and relevance to disease.  Annals of the new York Academy of Sciences112411–38.
  * Cao . [2020] cao2020behindCao, J., Gan, Z., Cheng, Y., Yu, L., Chen, YC. Liu, J.  2020\.  Behind the scene: Revealing the secrets of pre-trained vision-and-language models Behind the scene: Revealing the secrets of pre-trained vision-and-language models.  Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16 Computer vision–eccv 2020: 16th european conference, glasgow, uk, august 23–28, 2020, proceedings, part vi 16 ( 565–580).
  * Chen . [2023] chenBLABenchmarkInvestigating2023Chen, X., Fernández, R. Pezzelle, S.  202312\.  The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models The BLA Benchmark: Investigating Basic Language Abilities of Pre-Trained Multimodal Models.  H. Bouamor, J. Pino K. Bali (), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing ( 5817–5830).  SingaporeAssociation for Computational Linguistics.  [2024-06-27]https://aclanthology.org/2023.emnlp-main.356 10.18653/v1/2023.emnlp-main.356
  * Cherti . [2023] cherti2023reproducibleCherti, M., Beaumont, R., Wightman, R., Wortsman, M., Ilharco, G., Gordon, C.Jitsev, J.  2023\.  Reproducible scaling laws for contrastive language-image learning Reproducible scaling laws for contrastive language-image learning.  Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Proceedings of the ieee/cvf conference on computer vision and pattern recognition ( 2818–2829).
  * Conwell, Prince, Alvarez Konkle [2023] conwellunreasonableConwell, C., Prince, J., Alvarez, G. Konkle, T.  2023\.  The Unreasonable Effectiveness of Word Models in Predicting High-Level Visual Cortex Responses to Natural Images The unreasonable effectiveness of word models in predicting high-level visual cortex responses to natural images.  Conference on Computational Cognitive Neuroscience. Conference on computational cognitive neuroscience.
  * Conwell, Prince, Kay . [2023] conwellWhatCanBillion2023Conwell, C., Prince, JS., Kay, KN., Alvarez, GA. Konkle, T.  202307\.  What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?  bioRxiv.  [2024-06-07]https://www.biorxiv.org/content/10.1101/2022.03.28.485868v2 10.1101/2022.03.28.485868
  * Damasio [1989] damasio_brain_1989Damasio, AR.  1989\.  The Brain Binds Entities and Events by Multiregional Activation from Convergence Zones The Brain Binds Entities and Events by Multiregional Activation from Convergence Zones.  Neural Computation11123–132.  [2023-10-28]https://ieeexplore.ieee.org/abstract/document/6795614 10.1162/neco.1989.1.1.123
  * Davis . [2019] davis-etal-2019-deconstructingDavis, C., Bulat, L., Vero, AL. Shutova, E.  201906\.  Deconstructing multimodality: visual properties and visual context in human semantic processing Deconstructing multimodality: visual properties and visual context in human semantic processing.  R. Mihalcea, E. Shutova, LW. Ku, K. Evang S. Poria (), Proceedings of the Eighth Joint Conference on Lexical and Computational Semantics (*SEM 2019) Proceedings of the eighth joint conference on lexical and computational semantics (*SEM 2019) ( 118–124).  Minneapolis, MinnesotaAssociation for Computational Linguistics.  https://aclanthology.org/S19-1013 10.18653/v1/S19-1013
  * Derby . [2018] derby-etal-2018-usingDerby, S., Miller, P., Murphy, B. Devereux, B.  201810\.  Using Sparse Semantic Embeddings Learned from Multimodal Text and Image Data to Model Human Conceptual Knowledge Using sparse semantic embeddings learned from multimodal text and image data to model human conceptual knowledge.  A. Korhonen I. Titov (), Proceedings of the 22nd Conference on Computational Natural Language Learning Proceedings of the 22nd conference on computational natural language learning ( 260–270).  Brussels, BelgiumAssociation for Computational Linguistics.  https://aclanthology.org/K18-1026 10.18653/v1/K18-1026
  * Devereux . [2013] devereux2013representationalDevereux, BJ., Clarke, A., Marouchos, A. Tyler, LK.  2013\.  Representational similarity analysis reveals commonalities and differences in the semantic processing of words and objects Representational similarity analysis reveals commonalities and differences in the semantic processing of words and objects.  Journal of Neuroscience334818906–18916.
  * Devlin . [2019] devlinBERTPretrainingDeep2019Devlin, J., Chang, MW., Lee, K. Toutanova, K.  2019\.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) ( 4171–4186).  Minneapolis, MinnesotaAssociation for Computational Linguistics.  https://aclanthology.org/N19-1423 10.18653/v1/N19-1423
  * Dirani Pylkkänen [2024] diraniMEGEvidenceThat2024Dirani, J. Pylkkänen, L.  202405\.  MEG evidence that modality-independent conceptual representations contain semantic and visual features MEG evidence that modality-independent conceptual representations contain semantic and visual features.  Journal of Neuroscience.  [2024-06-27]https://www.jneurosci.org/content/early/2024/05/22/JNEUROSCI.0326-24.2024 Publisher: Society for Neuroscience Section: Research Articles  10.1523/JNEUROSCI.0326-24.2024
  * Doerig . [2022] doerig2022semanticDoerig, A., Kietzmann, TC., Allen, E., Wu, Y., Naselaris, T., Kay, K. Charest, I.  2022\.  Semantic scene descriptions as an objective of human vision Semantic scene descriptions as an objective of human vision.  arXiv preprint arXiv:2209.11737.
  * Doerig . [2023] doerigNeuroconnectionistResearchProgramme2023Doerig, A., Sommers, RP., Seeliger, K., Richards, B., Ismael, J., Lindsay, GW.Kietzmann, TC.  202307\.  The neuroconnectionist research programme The neuroconnectionist research programme.  Nature Reviews Neuroscience247431–450.  [2024-06-07]https://www.nature.com/articles/s41583-023-00705-w Publisher: Nature Publishing Group  10.1038/s41583-023-00705-w
  * Dosovitskiy . [2021] dosovitskiy2021anDosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T.Houlsby, N.  2021\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale An image is worth 16x16 words: Transformers for image recognition at scale.  International Conference on Learning Representations. International conference on learning representations.  https://openreview.net/forum?id=YicbFdNTTy
  * Eickenberg . [2017] eickenberg2017seeingEickenberg, M., Gramfort, A., Varoquaux, G. Thirion, B.  2017\.  Seeing it all: Convolutional network layers map the function of the human visual system Seeing it all: Convolutional network layers map the function of the human visual system.  NeuroImage152184–194.
  * Fairhall Caramazza [2013] fairhall2013brainFairhall, SL. Caramazza, A.  2013\.  Brain regions that represent amodal conceptual knowledge Brain regions that represent amodal conceptual knowledge.  Journal of Neuroscience332510552–10558.
  * Fedorenko . [2011] fedorenko2011functionalFedorenko, E., Behr, MK. Kanwisher, N.  2011\.  Functional specificity for high-level linguistic processing in the human brain Functional specificity for high-level linguistic processing in the human brain.  Proceedings of the National Academy of Sciences1083916428–16433.
  * Finkelstein . [2001] finkelstein2001placingFinkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., Wolfman, G. Ruppin, E.  2001\.  Placing search in context: The concept revisited Placing search in context: The concept revisited.  Proceedings of the 10th international conference on World Wide Web Proceedings of the 10th international conference on world wide web ( 406–414).
  * Giulianelli . [2023] giulianelli-etal-2023-informationGiulianelli, M., Wallbridge, S. Fernández, R.  2023\.  Information Value: Measuring Utterance Predictability as Distance from Plausible Alternatives Information value: Measuring utterance predictability as distance from plausible alternatives.  H. Bouamor, J. Pino K. Bali (), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Proceedings of the 2023 conference on empirical methods in natural language processing ( 5633–5653).  SingaporeAssociation for Computational Linguistics.  https://aclanthology.org/2023.emnlp-main.343 10.18653/v1/2023.emnlp-main.343
  * Güçlü Van Gerven [2015] gucclu2015deepGüçlü, U. Van Gerven, MA.  2015\.  Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream.  Journal of Neuroscience352710005–10014.
  * Hanna . [2023] hannaWhenLanguageModels2023Hanna, M., Belinkov, Y. Pezzelle, S.  202312\.  When Language Models Fall in Love: Animacy Processing in Transformer Language Models When Language Models Fall in Love: Animacy Processing in Transformer Language Models.  H. Bouamor, J. Pino K. Bali (), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing ( 12120–12135).  SingaporeAssociation for Computational Linguistics.  [2024-07-02]https://aclanthology.org/2023.emnlp-main.744 10.18653/v1/2023.emnlp-main.744
  * Harnad [1990] harnad1990symbolHarnad, S.  1990\.  The symbol grounding problem The symbol grounding problem.  Physica D: Nonlinear Phenomena421-3335–346.
  * Hill . [2015] hill2015simlexHill, F., Reichart, R. Korhonen, A.  2015\.  Simlex-999: Evaluating semantic models with (genuine) similarity estimation Simlex-999: Evaluating semantic models with (genuine) similarity estimation.  Computational Linguistics414665–695.
  * Hochreiter Schmidhuber [1997] HochSchm97Hochreiter, S. Schmidhuber, J.  1997\.  Long Short-Term Memory Long short-term memory.  Neural Computation981735–1780.
  * Hollenstein Beinborn [2021] hollensteinRelativeImportanceSentence2021Hollenstein, N. Beinborn, L.  202108\.  Relative Importance in Sentence Processing Relative Importance in Sentence Processing.  Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers) ( 141–150).  OnlineAssociation for Computational Linguistics.  [2022-01-13]https://aclanthology.org/2021.acl-short.19 10.18653/v1/2021.acl-short.19
  * Hu . [2024] huLanguageModelsAlign2024Hu, J., Mahowald, K., Lupyan, G., Ivanova, A. Levy, R.  202401\.  Language models align with human judgments on key grammatical constructions. Language models align with human judgments on key grammatical constructions.  arXiv.  [2024-06-27]http://arxiv.org/abs/2402.01676 arXiv:2402.01676 [cs]  10.48550/arXiv.2402.01676
  * Hubel Wiesel [1962] hubel1962receptiveHubel, DH. Wiesel, TN.  1962\.  Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex.  The Journal of physiology1601106.
  * Huettig . [2011] huettigUsingVisualWorld2011Huettig, F., Rommers, J. Meyer, AS.  201106\.  Using the visual world paradigm to study language processing: A review and critical evaluation Using the visual world paradigm to study language processing: A review and critical evaluation.  Acta Psychologica1372151–171.  [2024-06-27]https://www.sciencedirect.com/science/article/pii/S0001691810002180 10.1016/j.actpsy.2010.11.003
  * Jain Huth [2018] jain2018incorporatingJain, S. Huth, A.  2018\.  Incorporating context into language encoding models for fMRI Incorporating context into language encoding models for fmri.  Advances in neural information processing systems31.
  * Jia . [2021] jia2021scalingJia, C., Yang, Y., Xia, Y., Chen, YT., Parekh, Z., Pham, H.Duerig, T.  2021\.  Scaling up visual and vision-language representation learning with noisy text supervision Scaling up visual and vision-language representation learning with noisy text supervision.  International conference on machine learning International conference on machine learning ( 4904–4916).
  * Jones Trott [2024] jonesMultimodalLanguageModels2024Jones, CR. Trott, S.  202405\.  Multimodal Language Models Show Evidence of Embodied Simulation Multimodal Language Models Show Evidence of Embodied Simulation.  N. Calzolari, MY. Kan, V. Hoste, A. Lenci, S. Sakti N. Xue (), Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) ( 11928–11933).  Torino, ItaliaELRA and ICCL.  [2024-06-27]https://aclanthology.org/2024.lrec-main.1041
  * Karamolegkou . [2023] karamolegkou-etal-2023-mappingKaramolegkou, A., Abdou, M. Søgaard, A.  202307\.  Mapping Brains with Language Models: A Survey Mapping brains with language models: A survey.  A. Rogers, J. Boyd-Graber N. Okazaki (), Findings of the Association for Computational Linguistics: ACL 2023 Findings of the association for computational linguistics: Acl 2023 ( 9748–9762).  Toronto, CanadaAssociation for Computational Linguistics.  https://aclanthology.org/2023.findings-acl.618 10.18653/v1/2023.findings-acl.618
  * Kaup . [2024] kaup2024modalKaup, B., Ulrich, R., Bausenhart, KM., Bryce, D., Butz, MV., Dignath, D.others  2024\.  Modal and amodal cognition: an overarching principle in various domains of psychology Modal and amodal cognition: an overarching principle in various domains of psychology.  Psychological Research882307–337.
  * Kiela Bottou [2014] kiela2014learningKiela, D. Bottou, L.  2014\.  Learning image embeddings using convolutional neural networks for improved multi-modal semantics Learning image embeddings using convolutional neural networks for improved multi-modal semantics.  Proceedings of the 2014 Conference on empirical methods in natural language processing (EMNLP) Proceedings of the 2014 conference on empirical methods in natural language processing (emnlp) ( 36–45).
  * Kiela . [2018] kielaLearningVisuallyGrounded2018Kiela, D., Conneau, A., Jabri, A. Nickel, M.  201806\.  Learning Visually Grounded Sentence Representations Learning Visually Grounded Sentence Representations.  M. Walker, H. Ji A. Stent (), Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) ( 408–418).  New Orleans, LouisianaAssociation for Computational Linguistics.  [2024-06-27]https://aclanthology.org/N18-1038 10.18653/v1/N18-1038
  * Kiela . [2016] kiela-etal-2016-comparingKiela, D., Verő, AL. Clark, S.  201611\.  Comparing Data Sources and Architectures for Deep Visual Representation Learning in Semantics Comparing data sources and architectures for deep visual representation learning in semantics.  J. Su, K. Duh X. Carreras (), Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing Proceedings of the 2016 conference on empirical methods in natural language processing ( 447–456).  Austin, TexasAssociation for Computational Linguistics.  https://aclanthology.org/D16-1043 10.18653/v1/D16-1043
  * Kriegeskorte . [2008] kriegeskorte2008representationalKriegeskorte, N., Mur, M. Bandettini, PA.  2008\.  Representational similarity analysis-connecting the branches of systems neuroscience Representational similarity analysis-connecting the branches of systems neuroscience.  Frontiers in systems neuroscience2249.
  * Lambon Ralph . [2017] ralph2017neuralLambon Ralph, MA., Jefferies, E., Patterson, K. Rogers, TT.  2017\.  The neural and computational bases of semantic cognition The neural and computational bases of semantic cognition.  Nature reviews neuroscience18142–55.  10.1038/nrn.2016.150
  * Landau . [1998] landauObjectPerceptionObject1998Landau, B., Smith, L. Jones, S.  199801\.  Object perception and object naming in early development Object perception and object naming in early development.  Trends in Cognitive Sciences2119–24.  10.1016/S1364-6613(97)01111-X
  * Laurençon . [2024] laurenccon2024obelicsLaurençon, H., Saulnier, L., Tronchon, L., Bekman, S., Singh, A., Lozhkov, A.others  2024\.  Obelics: An open web-scale filtered dataset of interleaved image-text documents Obelics: An open web-scale filtered dataset of interleaved image-text documents.  Advances in Neural Information Processing Systems36.
  * Lazaridou . [2015] lazaridou-etal-2015-combiningLazaridou, A., Pham, NT. Baroni, M.  201505–06.  Combining Language and Vision with a Multimodal Skip-gram Model Combining language and vision with a multimodal skip-gram model.  R. Mihalcea, J. Chai A. Sarkar (), Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies Proceedings of the 2015 conference of the north American chapter of the association for computational linguistics: Human language technologies ( 153–163).  Denver, ColoradoAssociation for Computational Linguistics.  https://aclanthology.org/N15-1016 10.3115/v1/N15-1016
  * J. Li . [2023] li2023blipLi, J., Li, D., Savarese, S. Hoi, S.  2023\.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  International conference on machine learning International conference on machine learning ( 19730–19742).
  * LH. Li . [2019] li2019visualbertLi, LH., Yatskar, M., Yin, D., Hsieh, CJ. Chang, KW.  2019\.  Visualbert: A simple and performant baseline for vision and language Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557.
  * Lindsay [2021] lindsayConvolutionalNeuralNetworks2021Lindsay, GW.  202109\.  Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future Convolutional Neural Networks as a Model of the Visual System: Past, Present, and Future.  Journal of Cognitive Neuroscience33102017–2031.  [2024-06-07]https://doi.org/10.1162/jocn_a_01544 10.1162/jocn_a_01544
  * Liu . [2019] liu2019robertaLiu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D.Stoyanov, V.  2019\.  Roberta: A robustly optimized bert pretraining approach Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692.
  * Louwerse [2011] louwerseSymbolInterdependencySymbolic2011Louwerse, MM.  2011\.  Symbol Interdependency in Symbolic and Embodied Cognition Symbol Interdependency in Symbolic and Embodied Cognition.  Topics in Cognitive Science32273–302.  [2024-06-25]https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1756-8765.2010.01106.x _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1756-8765.2010.01106.x  10.1111/j.1756-8765.2010.01106.x
  * Ma Peters [2020] maNeuralNetworkWalks2020Ma, WJ. Peters, B.  202005\.  A neural network walks into a lab: towards using deep nets as models for human behavior. A neural network walks into a lab: towards using deep nets as models for human behavior.  arXiv.  [2024-06-07]http://arxiv.org/abs/2005.02181 arXiv:2005.02181 [cs, q-bio]  10.48550/arXiv.2005.02181
  * Mehrer . [2021] mehrer2021ecologicallyMehrer, J., Spoerer, CJ., Jones, EC., Kriegeskorte, N. Kietzmann, TC.  2021\.  An ecologically motivated image dataset for deep learning yields better models of human vision An ecologically motivated image dataset for deep learning yields better models of human vision.  Proceedings of the National Academy of Sciences1188e2011417118.
  * Merkx . [2022] merkxSeeingAdvantageVisually2022Merkx, D., Frank, S. Ernestus, M.  202205\.  Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledge Seeing the advantage: visually grounding word embeddings to better capture human semantic knowledge.  E. Chersoni, N. Hollenstein, C. Jacobs, Y. Oseki, L. Prévot E. Santus (), Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics ( 1–11).  Dublin, IrelandAssociation for Computational Linguistics.  [2024-06-27]https://aclanthology.org/2022.cmcl-1.1 10.18653/v1/2022.cmcl-1.1
  * Michaelov . [2024] michaelovStrongPredictionLanguage2024Michaelov, JA., Bardolph, MD., Van Petten, CK., Bergen, BK. Coulson, S.  202404\.  Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects Strong Prediction: Language Model Surprisal Explains Multiple N400 Effects.  Neurobiology of Language51107–135.  [2024-06-27]https://doi.org/10.1162/nol_a_00105 10.1162/nol_a_00105
  * Mitchell . [2008] mitchellPredictingHumanBrain2008Mitchell, TM., Shinkareva, SV., Carlson, A., Chang, KM., Malave, VL., Mason, RA. Just, MA.  200805\.  Predicting Human Brain Activity Associated with the Meanings of Nouns Predicting Human Brain Activity Associated with the Meanings of Nouns.  Science32058801191–1195.  [2024-06-27]https://www.science.org/doi/full/10.1126/science.1152876 Publisher: American Association for the Advancement of Science  10.1126/science.1152876
  * Muennighoff . [2024] muennighoff2024generativeMuennighoff, N., Su, H., Wang, L., Yang, N., Wei, F., Yu, T.Kiela, D.  2024\.  Generative representational instruction tuning Generative representational instruction tuning.  arXiv preprint arXiv:2402.09906.
  * Nili . [2014] nili2014toolboxNili, H., Wingfield, C., Walther, A., Su, L., Marslen-Wilson, W. Kriegeskorte, N.  2014\.  A toolbox for representational similarity analysis A toolbox for representational similarity analysis.  PLoS computational biology104e1003553.
  * Noorman . [2018] noormanWordsAffectVisual2018Noorman, S., Neville, DA. Simanova, I.  201809\.  Words affect visual perception by activating object shape representations Words affect visual perception by activating object shape representations.  Scientific Reports8114156.  [2024-06-27]https://www.nature.com/articles/s41598-018-32483-2 Publisher: Nature Publishing Group  10.1038/s41598-018-32483-2
  * Oota . [2022] oota-etal-2022-visioOota, SR., Arora, J., Rowtula, V., Gupta, M. Bapi, RS.  202210\.  Visio-Linguistic Brain Encoding Visio-linguistic brain encoding.  N. Calzolari . (), Proceedings of the 29th International Conference on Computational Linguistics Proceedings of the 29th international conference on computational linguistics ( 116–133).  Gyeongju, Republic of KoreaInternational Committee on Computational Linguistics.  https://aclanthology.org/2022.coling-1.11
  * Pennington . [2014] pennington2014glovePennington, J., Socher, R. Manning, CD.  2014\.  Glove: Global vectors for word representation Glove: Global vectors for word representation.  Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP) Proceedings of the 2014 conference on empirical methods in natural language processing (emnlp) ( 1532–1543).
  * Pereira . [2018] pereira2018towardPereira, F., Lou, B., Pritchett, B., Ritter, S., Gershman, SJ., Kanwisher, N.Fedorenko, E.  2018\.  Toward a universal decoder of linguistic meaning from brain activation Toward a universal decoder of linguistic meaning from brain activation.  Nature Communications91963.
  * Pezzelle Fernández [2019] pezzelleRedSquareBig2019Pezzelle, S. Fernández, R.  201911\.  Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts Is the Red Square Big? MALeViC: Modeling Adjectives Leveraging Visual Contexts.  K. Inui, J. Jiang, V. Ng X. Wan (), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) ( 2865–2876).  Hong Kong, ChinaAssociation for Computational Linguistics.  [2024-06-27]https://aclanthology.org/D19-1285 10.18653/v1/D19-1285
  * Pezzelle . [2020] pezzelleBeDifferentBe2020Pezzelle, S., Greco, C., Gandolfi, G., Gualdoni, E. Bernardi, R.  202011\.  Be Different to Be Better! A Benchmark to Leverage the Complementarity of Language and Vision Be Different to Be Better! A Benchmark to Leverage the Complementarity of Language and Vision.  T. Cohn, Y. He Y. Liu (), Findings of the Association for Computational Linguistics: EMNLP 2020 Findings of the Association for Computational Linguistics: EMNLP 2020 ( 2751–2767).  OnlineAssociation for Computational Linguistics.  [2024-06-27]https://aclanthology.org/2020.findings-emnlp.248 10.18653/v1/2020.findings-emnlp.248
  * Pezzelle . [2021] pezzelle2021wordPezzelle, S., Takmaz, E. Fernández, R.  2021\.  Word representation learning in multimodal pre-trained transformers: An intrinsic evaluation Word representation learning in multimodal pre-trained transformers: An intrinsic evaluation.  Transactions of the Association for Computational Linguistics91563–1579.
  * Piantadosi Hill [2022] piantadosiMeaningReferenceLarge2022aPiantadosi, S. Hill, F.  202210\.  Meaning without reference in large language models Meaning without reference in large language models..  [2024-01-01]https://openreview.net/forum?id=nRkJEwmZnM
  * Popham . [2021] popham2021visualPopham, SF., Huth, AG., Bilenko, NY., Deniz, F., Gao, JS., Nunez-Elizalde, AO. Gallant, JL.  2021\.  Visual and linguistic semantic representations are aligned at the border of human visual cortex Visual and linguistic semantic representations are aligned at the border of human visual cortex.  Nature neuroscience24111628–1636.
  * Power . [2011] power2011functionalPower, JD., Cohen, AL., Nelson, SM., Wig, GS., Barnes, KA., Church, JA.others  2011\.  Functional network organization of the human brain Functional network organization of the human brain.  Neuron724665–678.
  * Radford . [2021] radford2021learningRadford, A., Kim, JW., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S.others  2021\.  Learning transferable visual models from natural language supervision Learning transferable visual models from natural language supervision.  International conference on machine learning International conference on machine learning ( 8748–8763).
  * Radford . [] radfordLanguageModelsAreRadford, A., Wu, J., Child, R., Luan, D., Amodei, D. Sutskever, I.  .  Language Models are Unsupervised Multitask Learners Language Models are Unsupervised Multitask Learners.  OpenAI blog.  2019
  * Raghu . [2021] raghuVisionTransformersSee2021Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C. Dosovitskiy, A.  2021\.  Do Vision Transformers See Like Convolutional Neural Networks? Do Vision Transformers See Like Convolutional Neural Networks?  Advances in Neural Information Processing Systems Advances in Neural Information Processing Systems ( 34, 12116–12128).  Curran Associates, Inc.
  * Ren . [2015] ren2015fasterRen, S., He, K., Girshick, R. Sun, J.  2015\.  Faster r-cnn: Towards real-time object detection with region proposal networks Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems28.
  * Schrimpf . [2021] schrimpf2021neuralSchrimpf, M., Blank, IA., Tuckute, G., Kauf, C., Hosseini, EA., Kanwisher, N.Fedorenko, E.  2021\.  The neural architecture of language: Integrative modeling converges on predictive processing The neural architecture of language: Integrative modeling converges on predictive processing.  Proceedings of the National Academy of Sciences11845e2105646118.
  * Seeliger . [2018] seeliger2018convolutionalSeeliger, K., Fritsche, M., Güçlü, U., Schoenmakers, S., Schoffelen, JM., Bosch, SE. Van Gerven, M.  2018\.  Convolutional neural network-based encoding and decoding of visual object recognition in space and time Convolutional neural network-based encoding and decoding of visual object recognition in space and time.  NeuroImage180253–266.
  * Silberer Lapata [2012] silbererGroundedModelsSemantic2012Silberer, C. Lapata, M.  201207\.  Grounded Models of Semantic Representation Grounded Models of Semantic Representation.  J. Tsujii, J. Henderson M. Paşca (), Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning ( 1423–1433).  Jeju Island, KoreaAssociation for Computational Linguistics.  [2024-06-27]https://aclanthology.org/D12-1130
  * Silberer Lapata [2014] silberer2014learningSilberer, C. Lapata, M.  2014\.  Learning grounded meaning representations with autoencoders Learning grounded meaning representations with autoencoders.  Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) Proceedings of the 52nd annual meeting of the association for computational linguistics (volume 1: Long papers) ( 721–732).
  * Simanova . [2014] simanova_modality-independent_2014Simanova, I., Hagoort, P., Oostenveld, R. van Gerven, MAJ.  2014\.  Modality-Independent Decoding of Semantic Information from the Human Brain Modality-Independent Decoding of Semantic Information from the Human Brain.  Cerebral Cortex242426–434.  [2023-10-24]https://doi.org/10.1093/cercor/bhs324 10.1093/cercor/bhs324
  * Sinclair . [2022] sinclair-etal-2022-structuralSinclair, A., Jumelet, J., Zuidema, W. Fernández, R.  2022\.  Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations Structural Persistence in Language Models: Priming as a Window into Abstract Language Representations.  Transactions of the Association for Computational Linguistics101031–1050.  https://aclanthology.org/2022.tacl-1.60 10.1162/tacl_a_00504
  * Springer . [2024] Springer2024RepetitionILSpringer, JM., Kotha, S., Fried, D., Neubig, G. Raghunathan, A.  2024\.  Repetition Improves Language Model Embeddings Repetition improves language model embeddings.  ArXivabs/2402.15449.  https://api.semanticscholar.org/CorpusID:267897400
  * Suglia . [2024] sugliaVisuallyGroundedLanguage2024Suglia, A., Konstas, I. Lemon, O.  202401\.  Visually Grounded Language Learning: A Review of Language Games, Datasets, Tasks, and Models Visually Grounded Language Learning: A Review of Language Games, Datasets, Tasks, and Models.  Journal of Artificial Intelligence Research79173–239.  [2024-06-27]https://www.jair.org/index.php/jair/article/view/15185 10.1613/jair.1.15185
  * H. Tan Bansal [2019] tan2019lxmertTan, H. Bansal, M.  2019\.  LXMERT: Learning Cross-Modality Encoder Representations from Transformers Lxmert: Learning cross-modality encoder representations from transformers.  Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp) ( 5100–5111).
  * H. Tan Bansal [2020] tan2020vokenizationTan, H. Bansal, M.  2020\.  Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision Vokenization: Improving language understanding with contextualized, visual-grounded supervision.  Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) Proceedings of the 2020 conference on empirical methods in natural language processing (emnlp) ( 2066–2080).
  * M. Tan Le [2019] tan2019efficientnetTan, M. Le, Q.  2019\.  Efficientnet: Rethinking model scaling for convolutional neural networks Efficientnet: Rethinking model scaling for convolutional neural networks.  International conference on machine learning International conference on machine learning ( 6105–6114).
  * Tang . [2024] tang2024brainTang, J., Du, M., Vo, V., Lal, V. Huth, A.  2024\.  Brain encoding models based on multimodal transformers can transfer across language and vision Brain encoding models based on multimodal transformers can transfer across language and vision.  Advances in Neural Information Processing Systems36.
  * Thrush . [2022] thrushWinogroundProbingVision2022Thrush, T., Jiang, R., Bartolo, M., Singh, A., Williams, A., Kiela, D. Ross, C.  202206\.  Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) ( 5228–5238).  New Orleans, LA, USAIEEE.  [2024-06-27]https://ieeexplore.ieee.org/document/9878945/ 10.1109/CVPR52688.2022.00517
  * Toneva Wehbe [2019] toneva2019interpretingToneva, M. Wehbe, L.  2019\.  Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain) Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain).  Advances in neural information processing systems32.
  * Touvron . [2023] touvron2023llamaTouvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, MA., Lacroix, T.others  2023\.  Llama: Open and efficient foundation language models Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971.
  * Tuli . [2021] tuliAreConvolutionalNeural2021Tuli, S., Dasgupta, I., Grant, E. Griffiths, T.  2021\.  Are Convolutional Neural Networks or Transformers more like human vision? Are Convolutional Neural Networks or Transformers more like human vision?  Proceedings of the Annual Meeting of the Cognitive Science Society4343.  [2024-06-07]https://escholarship.org/uc/item/2pm5q7k6
  * Vaswani . [2017] vaswani2017attentionVaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, AN.Polosukhin, I.  2017\.  Attention is all you need Attention is all you need.  Advances in Neural Information Processing Systems. Advances in neural information processing systems.
  * Wang . [2023] wang2023betterWang, AY., Kay, K., Naselaris, T., Tarr, MJ. Wehbe, L.  2023\.  Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset.  Nature Machine Intelligence5121415–1426.
  * Warstadt . [2020] warstadtBLiMPBenchmarkLinguistic2020Warstadt, A., Parrish, A., Liu, H., Mohananey, A., Peng, W., Wang, SF. Bowman, SR.  2020\.  BLiMP: The Benchmark of Linguistic Minimal Pairs for English BLiMP: The Benchmark of Linguistic Minimal Pairs for English.  Transactions of the Association for Computational Linguistics8377–392.  [2022-09-02]https://aclanthology.org/2020.tacl-1.25 Place: Cambridge, MA Publisher: MIT Press  10.1162/tacl_a_00321
  * Wehbe . [2014] wehbeSimultaneouslyUncoveringPatterns2014Wehbe, L., Murphy, B., Talukdar, P., Fyshe, A., Ramdas, A. Mitchell, T.  201411\.  Simultaneously Uncovering the Patterns of Brain Regions Involved in Different Story Reading Subprocesses Simultaneously Uncovering the Patterns of Brain Regions Involved in Different Story Reading Subprocesses.  PLOS ONE911e112575.  [2021-10-15]https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0112575 Publisher: Public Library of Science  10.1371/journal.pone.0112575
  * Yamins . [2014] yamins2014performanceYamins, DL., Hong, H., Cadieu, CF., Solomon, EA., Seibert, D. DiCarlo, JJ.  2014\.  Performance-optimized hierarchical models predict neural responses in higher visual cortex Performance-optimized hierarchical models predict neural responses in higher visual cortex.  Proceedings of the national academy of sciences111238619–8624.
  * Zablocki . [2018] zablocki2018learningZablocki, E., Piwowarski, B., Soulier, L. Gallinari, P.  2018\.  Learning multi-modal word representation grounded in visual context Learning multi-modal word representation grounded in visual context.  Proceedings of the AAAI Conference on Artificial Intelligence Proceedings of the aaai conference on artificial intelligence ( 32).
  * Zhang . [2022] zhang2022optZhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S.others  2022\.  Opt: Open pre-trained transformer language models Opt: Open pre-trained transformer language models.  arXiv preprint arXiv:2205.01068.
  * Zhuang . [2024] zhuangVisualGroundingHelps2024Zhuang, C., Fedorenko, E. Andreas, J.  202406\.  Visual Grounding Helps Learn Word Meanings in Low-Data Regimes Visual Grounding Helps Learn Word Meanings in Low-Data Regimes.  K. Duh, H. Gomez S. Bethard (), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) ( 1311–1329).  Mexico City, MexicoAssociation for Computational Linguistics.  [2024-06-27]https://aclanthology.org/2024.naacl-long.71