(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: Institute of Computer Science, University of Tartu 11email:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
# How structured are the representations in transformer-based vision encoders?
An analysis of multi-object representations in vision-language models

Tarun Khajuria  Braian Olmiro Dias  Jaan Aru

###### Abstract

Forming and using symbol-like structured representations for reasoning has
been considered essential for generalising over novel inputs. The primary tool
that allows generalisation outside training data distribution is the ability
to abstract away irrelevant information into a compact form relevant to the
task. An extreme form of such abstract representations is symbols. Humans make
use of symbols to bind information while abstracting away irrelevant parts to
utilise the information consistently and meaningfully. This work estimates the
state of such structured representations in vision encoders. Specifically, we
evaluate image encoders in large vision-language pre-trained models to address
the question of which desirable properties their representations lack by
applying the criteria of symbolic structured reasoning described for LLMs to
the image models. We test the representation space of image encoders like VIT,
BLIP, CLIP, and FLAVA to characterise the distribution of the object
representations in these models. In particular, we create decoding tasks using
multi-object scenes from the COCO dataset, relating the token space to its
input content for various objects in the scene. We use these tasks to
characterise the network’s token and layer-wise information modelling. Our
analysis highlights that the CLS token, used for the downstream task, only
focuses on a few objects necessary for the trained downstream task. Still,
other individual objects are well-modelled separately by the tokens in the
network originating from those objects. We further observed a widespread
distribution of scene information. This demonstrates that information is far
more entangled in tokens than optimal for them to represent objects similar to
symbols. Given these symbolic properties, we show the network dynamics that
cause failure modes of these models on basic downstream tasks in a multi-
object scene.

###### Keywords:

Abstraction Symbols Vision-Language Models Vision Encoder

##  1 Introduction

This study investigates whether symbol-like structured representations may
emerge in transformer-based vision models. In vision models, some studies have
evaluated their representations, looking for aspects of human-like symbolic
cognitive processes [19, 21]. An advantage of searching for symbolic
representations in vision models is that the input information is distributed
in pixels, allowing explicit binding dynamics to be observed. Binding dynamics
refers to the information about entities (objects, living things, see Fig 1.C)
being bound together into smaller representational units, describing specific
useful aspects of that entity [9]. Further, in vision models, the intermediate
representations of objects are natural candidates to be formed into symbols,
allowing one to formulate hypotheses about the nature of the representations.
Building upon the notion of symbols and binding described in previous studies
[9, 19, 21], we propose three levels of properties for an image encoder
representation to be more symbolic: 1) The model should be able to aggregate
information specific to objects in the image into smaller units; 2) The model
should represent various logical components of the input image in separate
computational units, i.e., there should be object-wise information
disentanglement; 3) A downstream processing network should be able to utilise
the composition of these units for the required task, e.g., scene modelling
for captioning.

The first two of these proposed properties of being symbol-like promote better
out-of-distribution generalisation by themselves. 1) Better binding of object
information into a discrete set of units allows useful properties of the
representation to be fully available for novel scenarios in solving downstream
task. 2) Disentanglement of these units of representation makes sure that the
presence of one object does not affect the representation of others. This is
important for extreme cases where the objects are present out of the scene
statistics of the trained distribution and these spurious correlations in
representations of object presence may cause failure for the downstream task.

The compositional abilities of these Vision-Language Models (VLMs) have
already been tested and shown to be inadequate [28]. However, there is some
evidence that while doing downstream tasks, primitive concepts emerge in model
representations and are compositionally employed by the downstream network
[29]. Transformers have already been shown to have global and local
information aggregation within a single layer facilitated by the attention
mechanism [24]. Also, due to the retention of the token space across layers,
they could have enough computational units to maintain object information
separately while binding them locally. Hence, there is a need to estimate the
extent to which the first two properties of symbolic representations listed
above are implemented in vision transformer models.

For this purpose, we analyse the spatial representations in the vision
transformers’ token space. Our primary questions testing for properties 1 and
2 are the following: Do transformers represent and maintain object-wise
representations? Are these representations disentangled, i.e. does a
particular set of tokens only represent a particular object? We utilise the
COCO dataset[17] with its instance object masks to characterise the token
representations in relation to their input patch information. We set up
decoding experiments in a two-object setting (in a multi-object scene) to
determine how the encoders manage the representations of the two objects. We
try encoder of three VLMs (CLIP [23], BLIP[16] and FLAVA[25]) and compare them
with the larger versions (CLIP-L, BLIP-L) and also check the representations
against VIT trained for image classification and CLIP (Resnet X4) with a CNN
backbone.

We generally observe that object-specific areas hold the most discriminative
features about the objects until the last layers. The token representations
are not disentangled since they can decode other objects in the scene with
accuracy far above random guesses. Across networks, we see similar patterns of
decoding accuracies in transformer-based VLMs. We observe the VLMs trained on
objectives requiring the modelling of multiple objects have better symbol-like
differentiated object representations than VIT encoders trained for image
classification. On the other hand, all VIT-based encoders have less
differentiated and disentangled object representation in CNN-based CLIP
(Resnet X4). Our results suggest a preference for modelling objects relevant
to the downstream task (e.g., captioning for BLIP). This effect is more
pronounced for the CLS token directly optimised for the task; other object-
specific tokens still retain more high-level information about most objects.
We analyse and discuss the implications of representation loss in CLS for
failure in out-of-distribution task scenarios.

##  2 Related Works

The overall problem with generalisation in DL methods and the need for
explicit inductive biases that allow for discrete yet flexible information
binding has been comprehensively discussed [9]. Discrete symbolic
representations are considered a prerequisite for robust compositionality and
reasoning [21] and new studies propose that such discrete information binding
may originate from training existing models on large datasets [21, 19]. This
issue carries over to the field of meta-learning with its aim to introduce and
compositionally utilise modularity in neural networks to improve
generalisation [13].

Concept bottleneck models are a series of models that try to explicitly learn
human interpretable intermediate outputs to be composed into final output
labels [7, 27]. This interpretability usually comes at the cost of downstream
task performance. [11] introduced an extra loss in the intermediate layer on
the networks for units to align to the interpretable concepts while preserving
task performance. Many other works do not bind representations to any explicit
intermediate concept but still have explicit discrete bottlenecks in their
networks [20, 18]. It has been shown that these networks learn better
representations which further help generalisation on downstream tasks [20, 18,
26].

###  2.1 Evaluating model’s reasoning capabilities

Many works have proposed benchmarks, and others have tried to analyse the
existing networks to estimate the reasoning capabilities of the ANN models.
[3] proposed a visual question-answering benchmark to evaluate the reasoning
skills on images, with open-ended and free-form questions and expected
solutions in free-form natural language. Visual genome [12] provided content-
rich images with explicit annotation of object position and relationships,
promoting models that exploit such information. Datasets such as CLEVR [10]
were designed to correct for biases the models would utilise in the existing
benchmarks to perform well without explicit reasoning. The ARO (Attribute,
Relation, Order) dataset [28] has been proposed to test the correct binding of
information when compositionally representing multi-object scenarios.
Particularly in VLMs such as BLIP and CLIP, they find that the model outputs
do not bind the compositional properties well and attribute the wrong order
and features to objects when describing them.

###  2.2 Interpreting model’s representations

In terms of analysis of model representations for concepts, early work [2]
showed the layer-wise progression of decoding accuracy for concepts in CNNs.
[6] reveals the inner workings of transformers by showing how trained
transformers can implement convolutions and, in the initial layers, form grid-
like local attention patterns like a convolution filter. [24] shows that
vision transformers differ from CNNs because of their ability to encode both
local and global information in the initial layer. Instead, in CNNs, there is
a multi-scale feature representation going from lower-scale local information
captured in the initial layers to higher-scale global information captured in
the higher layers [4].

Particularly for vision-language models, [5] analysed various models trained
using cross-model attention, providing insights into the attention patterns
between the two modality streams and the relative contribution of each
modality towards downstream tasks. Further, they found function-specific
attention heads in the pre-trained models. VL Interpret [1] was designed as a
visualisation tool to interpret the vision-language model’s instance-specific
and aggregate statistics over attention distribution. Further, the tool helps
visualise token representation as it passes through various network layers.

In contrast to the generic visualisation tools that look into the model’s
functioning, many studies inspect their representations, evaluating a specific
computational functionality. [14] finds modular subnetworks in trained ANN
models functionally responsible for separate tasks. Multiple studies further
look into the notion of concepts in trained vision-language model
representations. One study [29] designed a test to check if primitive concepts
emerge in the network’s representations, which are used compositionally for
downstream tasks. [19] defines tests for concepts in visual representations
according to Fodor’s criteria [8] and tests these criteria using a controlled
synthetic dataset. In our work, we build on the intuitions from these works
while trying to formulate our hypothesis about symbolic representations in a
limited scope but testing it with images from natural scenes.

##  3 Methods

Figure 1: A. Explanation of how the token representations are obtained. We
analyse four kinds of tokens in this study 1) CLS token: Is common to both
objects in the image; Object-specific 2) Avg_obj: obtained by averaging the
token representations of the object-masked tokens as shown in the figure. 3)
Random_obj (Object token): Rather than averaging, we sample one of the tokens
from the masked token space of the object 4) Random : Obtained by sampling any
random token from the token space other than the CLS token B. Describes the
experimental setup in which we perform decoding in paired object tasks; each
object-specific representation decodes 1) the object itself, 2) the other
object in the image, and 3) the combination of both objects. C. Shows a sample
paired object decoding task; given an image, the task is to decode if it
contains object1 (cat/dog), object2 (chair/couch) or a combination of both.

We design probes and use similarity measures to evaluate the network’s
representational structure. In the following section, we describe the details
of the experimental setup, data and the networks analysed.

Experimental Setup: We decode the combination of objects in the image from a
single token or an average of tokens obtained from various parts of the image
(see Fig. 1.A). The tokens we are interested in include the 1) CLS token (the
token used and trained for the downstream task), 2) Average token (avg_obj):
the average of token representation obtained from the object, 3) Random object
(random_obj) token: a single token randomly sampled from the object 4) Random
token (Random): a randomly sampled token from the image that served as a
baseline. The token representation is obtained at the output of each layer. In
this process, to identify the tokens originating from an object, we scale the
segmentation mask of the object to the size of the token space. The probing
task is designed as a classification task with the following settings: we use
the tokens originating from 1) Primary object, 2) Secondary object and use it
to decode 1) Primary object category, 2) Secondary object category, 3)
Combination of primary and secondary object categories (see Fig. 1.B). We
train and evaluate the probes with a train/val/test setup with 80/10/10
percent data splits. The reported accuracies are all final test set
accuracies. All trained probes are linear and use the Scikit-learn[22]’s
perceptron implementation, with its default parameters. To check the baseline
performance of these representations to decode objects in a complex (20
classes) task, we train layer-wise global object classification probes for
each layer of the networks. These probes utilise first 40000 images from the
MSCOCO train set for training the probes and the 5000 images in the validation
set for testing. To make the linear probe with less learning capacity, we use
only the top 20 frequent object categories in the 40000 training images in the
probing task.

Dataset: We needed instance segmentation masks to associate the tokens to the
object in the image. Hence we used the COCO dataset and created subsets with
enough combination of two object categories co-occurring in them. We then
excluded images with more than one object category of primary or secondary
objects. While choosing the objects across primary and secondary categories we
preferred objects more likely to interact in the scene. Within the primary and
secondary categories we preferred objects that are similar to each other, so
their embedded representations are not naturally distinct, increasing our
probe’s sensitivity. We used six task sets with a total of 16,288 images. For
example, the first task contains a combination of objects from two sets, i.e.
a primary object: animal (cat/dog) and a secondary object: furniture
(chair/bench/bed/couch). The dataset and its tasks are detailed in Table 1. We
call this set of tasks ’object pair decoding tasks’. The global probes were
trained and tested on a larger dataset. We selected the first 40000 images in
COCO’s training set for training and the 5000 images in the validation set for
testing.

Table 1: For paired object decoding, we use 6 tasks with different number of images in each task. Each task contains images with different variations of objects |  Task1 |  Task2 |  Task 3 |  Task4 |  Task 5 |  Task 6
---|---|---|---|---|---|---
# Images |  2414 |  5042 |  1953 |  2143 |  938 |  3738
Objects 1 |  cat, dog |  dining table, person |  train, bus |  tv, laptop |  microwave, oven |  motorcycle, car
Objects 2 |  bench, chair, couch, bed |  pizza, knife, cup, cake |  traffic light, bench, backpack, handbag |  mouse, remote, keyboard, cellphone |  bottle, spoon, knife, cup |  traffic light, handbag, backpack, bicycle

Linear Probing: We probe the representations in pre-trained networks for our
analysis. Probing an information system involves obtaining a representation,
usually in the form of a vector from the system in response to an image. Then,
we estimate if that particular vector can classify information about the
stimuli correctly. The kind of information the probe can learn to classify and
the complexity of the probe (i.e. is it just a linear classifier or a complex
multilayer NN) indicates the nature of information present in that layer’s
representation. We obtain representations from various parts (layers and
spatial sections tokens/cells) of the network and use them to understand the
network by looking at the classification ability of a linear probe.

Analysed Networks: We analysed the image encoders of seven models: BLIP
(ViT-B/16 and ViT-L/14) model for image captioning, CLIP for image-text
matching, FLAVA model with an additional multimodal encoder on top of vision
and language encoders and finally, VIT-B/16 encoder trained on imagenet21k
image for image classification task. We analysed both VIT B/16 and L/14 (Image
Transformer) and the Resnet X4 (CNN) image encoder for the CLIP. In CNN
analysis, we obtained feature cells instead of tokens, i.e., we used the
vector representation of a cell in the feature map by accumulating all the
filter outputs at the cell location. In both transformers and CNNs, the place
on the feature map representing the object is computed by scaling the object
segmentation map to the size of the feature map at the layer. The images are
pre-processed with the standard pre-processing function and setting provided
along with the pre-trained network instances.

##  4 Results

(a)

(b)

Figure 2: a. Object-pair decoding task results for BLIP: Average decoding
performance for different layers (y-axis) and token types (x-axis) over 6
tasks for BLIP. In the subfigures, the y-axis contains variations of where the
object-specific tokens (random_obj and avg_obj) are obtained. The different
columns show results for 1) decoding the primary object 2) the secondary
object 3) the combination of both objects in the image. The decoding pattern
remains after averaging, with the tokens from the objects modelling the most
useful information for categorising the objects. They are much better than the
CLS token, which has to capture the larger scene context. b. Visualisation of
cosine similarity to other tokens for a token from primary and secondary
object.

###  4.1 A snapshot of network’s global and image-specific representation

Decoding the objects in a multi-object scene using the trained model’s
representations gives us a picture of how the information about the scene is
organised in the network. In our study, we have used two kinds of decoding
tasks (object-paired tasks and global decoding tasks) to estimate the
architecture-level organisation of these representations. We also generate
representation similarity maps to analyse the representation of a single image
in a particular model.

Using the paired object decoding tasks, we can see that the objects in the
network can be linearly decoded considerably above random accuracy by using a
single token representation originating from the object or by using an average
of token representations from the object. The general trend followed by these
decoding accuracies can be seen in Fig 3. The global object decoding probes,
with their 20 class classification setting, provide a good baseline of the
ability of the token’s representations to decode between multiple objects. The
high correlation of these accuracies for each network (see Fig. 4) to paired-
object classification tasks’ results shows that the accuracies are not
obtained due to an easy 2-way or 4-way classification task set up in the
paired object tasks.

Finally, in Fig. 2, you can see the detailed results for decoding accuracies
for the four types of token representation used in our analysis for the paired
object decoding tasks. This provides a more detailed global picture of the
model’s representations of object pairs from the images. For an instance-wise
image-level analysis of representations, we generate similarity maps of the
object representations that show how a particular token’s relation to other
tokens changes across layers in the model for a particular image. In 2.b, an
image shows a representation similarity map for two tokens in an image 1) from
the primary object 2) from the secondary object. One can notice how the
representations of tokens originating from the same token start having similar
representations as we move toward the upper layers. The tokens from the
primary object acquire similar representation, but for the secondary object,
many tokens that are outside the object also have high cosine similarity in
the last layer.

###  4.2 How different token representations encode objects, their interaction
and their importance

In the paired-object tasks, the images consist of a primary object and a
secondary object. We see the representation of these two object combinations
in each image in four type of token representations from the models. Based on
the results from the BLIP model shown in Fig 2 we now discuss the general
trend of representations across models. Specific differences are discussed in
subsection 4.3.

In our results (see Fig. 4), in all pre-trained models, the primary objects
are decoded equally well by the CLS token as the average token representation
of the object. There is a decrease in decoding accuracy from primary to
secondary object categories. This is partially due to the added complexity of
a 4-way classification in the secondary object. However, the CLS token decodes
the secondary objects with much less accuracy than both the object-specific
tokens (avg_obj and random_obj). The CLS tokens that are optimised for the
downstream tasks in each of these networks are expected to model the best
information about the scene. But this also means that not all objects are
linearly decodable by the CLS token. We observe that the object-specific
tokens have better object decoding accuracy as compared to the CLS token (In
BLIP Fig 2 primary object CLS:0.94 vs Avg_Obj: 0.97 and secondary object CLS:
0.69 vs Avg_Obj: 0.84). There is a particular degradation in decoding
performance using the CLS token for secondary objects, where the accuracy for
the CLS token is far below the decoding accuracy of object-specific tokens
(avg_obj and random_obj).

Previous analysis showed that the object-specific tokens have the highest
accuracy for decoding the particular object from which they originated. Still,
they also have high decoding accuracy for the other objects in the image. This
accuracy is far above random guess and allows these tokens to decode a
combination of objects in the current image (see Fig. 2, for results on BLIP).
The final decoding accuracy for the combination is 0.59 when using the primary
(avg_obj) object token and 0.69 when using the secondary one. In Fig. 2, one
can notice that the primary avg_obj token is much worse at representing the
secondary objects than the other way around. These results have implications
for the disentanglement of these object-specific tokens. Further, the CLS
token also shows a decent decoding accuracy as it is optimised to capture more
global information from the image. This accuracy is not higher than the
object-specific secondary tokens due to its poor capturing of information
about these secondary objects in the scene (CLS token decoding accuracy for
the secondary object is 0.69 compared to 0.94 for the primary object, as seen
in Fig. 2).

Figure 3: Layer-wise test set decoding accuracy for primary and secondary
objects for five pre-trained models in the study. Results for all models are
shown in appendix. The accuracies are averaged over the six task sets. In each
sub-graph, the y-axis denotes the decoding accuracy and the x-axis denotes the
layer at which the accuracy was observed. We observe consistent decoding
trends across models with a few variations reported in 4.3. Figure 4: Results
from global image probes: Final layer test set decoding accuracy for 20 object
classes for all pre-trained image encoders in the study. We observe that all
networks perform way above average using a linear readout for a more difficult
20-class classification task.

###  4.3 VLM models trained on multi-object images have more modular
representations than VIT (trained for single object classification)

In this section, we compare the representations of the models with each other.
For the global decoding task, we see that the newer VLMs like FLAVA and BLIP
have higher decoding accuracy with their object-specific representations than
the other counterparts like CLIP (see Fig 4). Larger models like CLIP-L and
BLIP-L counterintuitively perform worse than their base counterparts.
Specifically, CLIP-L shows fairly lower decoding accuracy for both single
random object and average object token representation (random_obj and
Average_obj). In the paired-object decoding tasks, the VLM models follow the
overall trend of decoding accuracy being correlated to the results in the
global object probing tasks (see3). The relative accuracy trends for various
tokens are similar across VLM models. The notable observation is the higher
overall decoding performance of FLAVA models, which also shows a higher
differentiation of object-wise representations, i.e the random token accuracy
is fairly lower in the last layer compared to object-specific tokens.

We observe a difference in the structure of representations due to
architecture (Transformer vs CNN) and specific training on multi-object tasks
i.e ViT vs other models. Hence, we observe a significant decrease in decoding
accuracy using the random CNN unit representation compared to random ViT
tokens (In the last layers, primary object ViT: 0.84 vs CNN: 0.72; secondary
object ViT: 0.54 vs CNN: 0.45)(see 3). Further, the object-specific tokens in
CNNs have lower accuracy while decoding the other objects than their ViT
counterpart (In the last layers, primary object ViT: 0.88 vs CNN: 0.8;
secondary object ViT: 0.6 vs CNN: 0.59). This indicates that CNN has less
entanglement of object-specific information across objects than the ViT
counterpart in CLIP. We attribute these results to CNNs not having the ability
for the information to travel across units in each layer, hence the object
information remains more localised. VIT trained on ImageNet21k shows the least
differentiation between object-specific and other tokens as compared to the
other Transformer models. Here a random token decoded the object with almost
similar accuracy as compared to the CLS token (see Fig. 3). And tokens from
one object and similarly decode the category of other objects in the scene.
Hence it seems like the scene-level information is more uniformly dispersed in
the representations of ViT tokens trained only on single object classification
task. This is in comparison to all the ViT models trained on tasks requiring
modelling of more than one object for a correct representation of the
information required for downstream tasks like text matching, captioning etc.
Further, the differentiation of object-specific token and consequently the
disentanglement of information is more explicit in these networks than VIT.

###  4.4 Special tokens may not be special for your downstream task with
multi-object images

(a)

(b)

Figure 5: a. Variation in decoding accuracy between instances of objects ‘in
caption’ and ‘not in caption’. Each subplot represents the decoding of the
object by its object-specific representation. b. Object detection accuracy on
test set using probes trained on three different token representations from
CLIP and zero-shot CLIP accuracy.

Following the observations about the lower decoding accuracy of secondary
objects using CLS tokens, we wanted to check if the low accuracy results from
the object’s importance in the downstream task (captioning in the case of
BLIP). Hence, we analysed the decoding accuracy for two sets of data. The set
was divided based on whether the object was directly mentioned in the caption
generated by the model. Due to this split, we rejected tasks with less than
400 samples in either of the new sets. Therefore, we are left with 3 tasks
whose average results are reported in Fig. 5a. In our results in Fig. 5a, we
report a decrease in decoding accuracy of the objects not mentioned in
captions using all kinds of tokens considered in our analysis (Average final
layer accuracy for primary object, which was for ’in caption’: CLS:1 and
avg_token: 0.99, drops to CLS: 0.87 and avg_obj 0.93 when the object is ’not
in caption’. In the secondary object, the drop is from ’in caption’: CLS: 0.96
and avg_obj: 0.99 to ’not in caption’: CLS: 0.79 and avg_obj: 0.82). This
means the network pays more attention to certain objects, and its learning of
discriminative features deteriorates for objects not mentioned in the
captions. We note that the decrease in the decoding is most pronounced for the
CLS token, showing the direct effect of the downstream task on the
representation.

We further use this understanding to check for failure in zero shot object
detection (i.e. retrieval) tasks using CLIP in a multi-object setting. We use
the classification task setting for the global probe. We use the object-
specific representation using 1 token and avg_obj token to learn probe to
classify the top 20 objects in test set images. Then we evaluate the CLIP zero
shot performance to detect objects in the image using the prompt ’An image
containing a ’category”. Where the category was replaced with 20 candidate
objects. We finally evaluate the accuracy of finding the objects. A fourth
multi-class object detection probe is learnt using the CLS token
representation. As we compare the accuracy of these four methods on the simple
object retrieval task, we find that the probes using the average patch
representation gives the best accuracy followed by random_obj token and zero
shot model accuracy and multi-class probe (See Fig. 5b).

##  5 Discussion

In this work, we started by identifying three characteristics required in
representations of image models for them to be symbol-like: 1) The
architecture should be able to aggregate information specific to objects on
the image into smaller units; 2) The architecture should represent various
logical units of the input image in separate computational units, i.e., there
should be object-wise information disentanglement; 3) A downstream processing
network should be able to utilise the composition of these units for the
required task.

Most vision encoders fulfill the first criterion we described as they
aggregate useful information from the image into smaller representation space
useful for the downstream task. However, we were interested in whether
separate representational units specialise in representing the image’s
separate constituent parts (objects). Our analysis shows that the object-
specific (avg_obj, random_obj) tokens show decent decoding accuracy across the
models. Each object-wise token’s accuracy for classifying itself is our
estimate for fulfilling this criterion. For the second characteristic, we
looked at the network’s capability to form disentangled representations of
objects. We saw evidence that the information about other objects also leaks
into the object-wise representations. Ideally, each object representation can
only decode that particular object and other objects in the scene cannot be
decoded by that representation. As the other objects can be decoded in our
object representations (with accuracies of 0.89 by secondary object
representations and 0.65 by primary object representations), their
representations are already affected by the context, i.e. other objects. This
finding implies that the downstream task using these representations
compositionally still cannot perform structured composition as with abstract
symbols. Further, the higher the decoding accuracy of other objects in the
scene, the higher the entanglement in the representation. The entangled object
representation will affect the downstream network’s outputs if the context
information is irrelevant to the task.

Our comparative analysis of the BLIP model’s performance showed a notable
degradation in decoding efficacy across all representations when objects were
omitted from its captions. This observation underscores the significant
influence of the model’s downstream objective on its representational
capabilities. On the one hand, this enhances the disentanglement of object
representations, reducing interference from objects insignificant to the task.
Conversely, it reduces the model’s generalizability, particularly when
deployed in diverse tasks or faced with unfamiliar input distributions. Our
results show that this problem is most exacerbated in the representation of
the CLS token. Most research predominantly assesses these networks’
capabilities based on their final outputs, which rely on representations like
a CLS token (for example in [28, 15]). Our findings show a loss in
representing multiple objects during this process of funneling information
into the CLS token. Further, the downstream objective’s effect is the most
extreme on the nature of these representations. We show that this causes
suboptimal performance of these networks in simple tasks such as retrieval in
the multi-object setting. We further postulate that these phenomena (non-
binding of certain objects representations or entangled representations)
significantly contribute to the models’ suboptimal performance in tasks that
require to compose using the representation of objects in scenarios involving
multiple objects. In evaluations of CLIP, FLAVA and similar models within a
multi-object context, the ramifications of the CLS token bottleneck and its
training procedure (training data and objective) must be meticulously
considered. The objects represented and the nature of their representation in
this bottleneck are heavily influenced by the downstream objective or a co-
occurring object, thereby affecting the accuracy of outputs in tasks that
diverge from the types of instances encountered during training. Future work
can utilise our method to look into training better readouts and fine-tuning
on specific underrepresented object images.

##  6 Limitations

The probing and representation analysis methods we used in our analysis have
some limitations, and the results must be carefully interpreted. High decoding
accuracy in a decoding task with two or four objects may be due to the easy
decoding task (if the object classes are naturally distinct and it is only a
2/4 way classification). Further, the fact that one can classify a particular
object from a few other classes may depend on representing a single or a few
features distinguishing between the classes. This representation may not model
many other aspects of the object. We controlled for this limitation by
training global probes for 20-class classification using the same
representations, showing reasonably high accuracy. Likewise, a lower decoding
accuracy does not mean that the model does not have information about the
particular object; the information is just not encoded with linear distinction
at that layer/token. Overall, we are averaging over six tasks in many object
categories. Hence, even though the exact numbers may not reflect the model’s
exact state, the relative numbers are helpful in making inferences about the
model and its representations.

##  7 Conclusion

In this work, we first formulated three characteristics for symbolic
representations in an image encoder. We then analysed the representations of
transformer-based image encoders in VIT, BLIP, CLIP, FLAVA and CLIP (Resnet X
4) image encoders. Through probing tasks, we observed that object-specific
areas hold the most discriminative features about the objects until the last
layers. Their discriminative ability decreased when the objects were not
important for the downstream task but was still reasonably maintained. The
token representations are not disentangled since they can decode other objects
in the scene with accuracy far above random guesses. We found that the
aggregate image representation, i.e. CLS token for transformers, does not
represent all objects, and its discriminative ability degrades the most when
the object is not useful for the trained downstream task. Our work thus
characterises the extent to which the representations of these image encoders
are symbolic and gives insights into the failures on out-of-distribution
downstream tasks that utilise the composition of these features.

## References

  * [1] Aflalo, E., Du, M., Tseng, S.Y., Liu, Y., Wu, C., Duan, N., Lal, V.: Vl-interpret: An interactive visualization tool for interpreting vision-language transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 21406–21415 (2022)
  * [2] Alain, G., Bengio, Y.: Understanding intermediate layers using linear classifier probes. arXiv preprint arXiv:1610.01644 (2016)
  * [3] Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C.L., Parikh, D.: Vqa: Visual question answering. In: Proceedings of the IEEE international conference on computer vision. pp. 2425–2433 (2015)
  * [4] Bronstein, M.M., Bruna, J., Cohen, T., Veličković, P.: Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104.13478 (2021)
  * [5] Cao, J., Gan, Z., Cheng, Y., Yu, L., Chen, Y.C., Liu, J.: Behind the scene: Revealing the secrets of pre-trained vision-and-language models. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VI 16. pp. 565–580. Springer (2020)
  * [6] Cordonnier, J.B., Loukas, A., Jaggi, M.: On the relationship between self-attention and convolutional layers. arXiv preprint arXiv:1911.03584 (2019)
  * [7] De Fauw, J., Ledsam, J.R., Romera-Paredes, B., Nikolov, S., Tomasev, N., Blackwell, S., Askham, H., Glorot, X., O’Donoghue, B., Visentin, D., et al.: Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature medicine 24(9), 1342–1350 (2018)
  * [8] Fodor, J.A.: Concepts: Where cognitive science went wrong. Oxford University Press (1998)
  * [9] Greff, K., Van Steenkiste, S., Schmidhuber, J.: On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208 (2020)
  * [10] Johnson, J., Hariharan, B., Van Der Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., Girshick, R.: Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2901–2910 (2017)
  * [11] Koh, P.W., Nguyen, T., Tang, Y.S., Mussmann, S., Pierson, E., Kim, B., Liang, P.: Concept bottleneck models. In: International conference on machine learning. pp. 5338–5348. PMLR (2020)
  * [12] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., et al.: Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision 123, 32–73 (2017)
  * [13] Lake, B.M., Baroni, M.: Human-like systematic generalization through a meta-learning neural network. Nature pp. 1–7 (2023)
  * [14] Lepori, M.A., Serre, T., Pavlick, E.: Break it down: evidence for structural compositionality in neural networks. arXiv preprint arXiv:2301.10884 (2023)
  * [15] Lewis, M., Yu, Q., Merullo, J., Pavlick, E.: Does clip bind concepts? probing compositionality in large image models. arXiv preprint arXiv:2212.10537 (2022)
  * [16] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning. pp. 12888–12900. PMLR (2022)
  * [17] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740–755. Springer (2014)
  * [18] Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., Kipf, T.: Object-centric learning with slot attention. Advances in Neural Information Processing Systems 33, 11525–11538 (2020)
  * [19] Lovering, C., Pavlick, E.: Unit testing for concepts in neural networks. Transactions of the Association for Computational Linguistics 10, 1193–1208 (2022)
  * [20] Oord, A.v.d., Vinyals, O., Kavukcuoglu, K.: Neural discrete representation learning. arXiv preprint arXiv:1711.00937 (2017)
  * [21] Pavlick, E.: Symbols and grounding in large language models. Philosophical Transactions of the Royal Society A 381(2251), 20220041 (2023)
  * [22] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: Machine learning in python. the Journal of machine Learning research 12, 2825–2830 (2011)
  * [23] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)
  * [24] Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., Dosovitskiy, A.: Do vision transformers see like convolutional neural networks? Advances in Neural Information Processing Systems 34, 12116–12128 (2021)
  * [25] Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., Kiela, D.: Flava: A foundational language and vision alignment model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15638–15650 (2022)
  * [26] Träuble, F., Goyal, A., Rahaman, N., Mozer, M.C., Kawaguchi, K., Bengio, Y., Schölkopf, B.: Discrete key-value bottleneck. In: International Conference on Machine Learning. pp. 34431–34455. PMLR (2023)
  * [27] Yi, K., Wu, J., Gan, C., Torralba, A., Kohli, P., Tenenbaum, J.: Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems 31 (2018)
  * [28] Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., Zou, J.: When and why vision-language models behave like bags-of-words, and what to do about it? In: The Eleventh International Conference on Learning Representations (2022)
  * [29] Yun, T., Bhalla, U., Pavlick, E., Sun, C.: Do vision-language pretrained models learn composable primitive concepts? arXiv preprint arXiv:2203.17271 (2022)