#  SugarCrepe++ Dataset: Vision-Language Model Sensitivity to Semantic and
Lexical Alterations

Sri Harsha Dumpala1,2 Aman Jaiswal111footnotemark: 1 Chandramouli Sastry1,2
Evangelos Milios1 Sageev Oore1,2 Hassan Sajjad1
1Dalhousie University, Canada. 2Vector Institute, Canada.
The authors contribute equally to this work.

###### Abstract

Despite their remarkable successes, state-of-the-art large language models
(LLMs), including vision-and-language models (VLMs) and unimodal language
models (ULMs), fail to understand precise semantics. For example, semantically
equivalent sentences expressed using different lexical compositions elicit
diverging representations. The degree of this divergence and its impact on
encoded semantics is not very well understood. In this paper, we introduce the
SugarCrepe++ dataset to analyze the sensitivity of VLMs and ULMs to lexical
and semantic alterations. Each sample in SugarCrepe++ dataset consists of an
image and a corresponding triplet of captions: a pair of semantically
equivalent but lexically different positive captions and one hard negative
caption. This poses a 3-way semantic (in)equivalence problem to the language
models. We comprehensively evaluate VLMs and ULMs that differ in architecture,
pre-training objectives and datasets to benchmark the performance of
SugarCrepe++ dataset. Experimental results highlight the difficulties of VLMs
in distinguishing between lexical and semantic variations, particularly in
object attributes and spatial relations. Although VLMs with larger pre-
training datasets, model sizes, and multiple pre-training objectives achieve
better performance on SugarCrepe++, there is a significant opportunity for
improvement. We show that all the models which achieve better performance on
compositionality datasets need not perform equally well on SugarCrepe++,
signifying that compositionality alone may not be sufficient for understanding
semantic and lexical alterations. The SugarCrepe++ dataset serves as a new
challenge to the vision-and-language community.111Data and code is available
at https://github.com/Sri-Harsha/scpp

##  1 Introduction

Large language models (LLMs), including vision-and-language models and
unimodal language models, have shown tremendous results in solving a majority
of vision and natural language processing (NLP) tasks. Surprisingly, despite
such success, LLMs can exhibit different behaviours for semantically
equivalent sentences composed with different syntactic or lexical structures.
Previous works have reported such lack of compositional reasoning in both
vision-and-language models (e.g., [69, 82, 85, 56, 73]) and unimodal language
models (e.g., [35, 9, 88, 47]). For instance, the performance of the state-of-
the-art (SOTA) LLMs including GPT-4, Gemini and Llama are sensitive to the
prompt formatting [63]. The model editing techniques [9, 47] suffer from
misfired edits due to the dominance of lexical overlap [76]. Similarly, Zou et
al. [88] demonstrated that safety aligned models can be “jailbroken” by simply
appending an adversarial suffix causing them to generate objectionable content
bypassing all safeguards.

These observations suggest that language models’ perception of semantic
similarity crucially depends on the lexical representation of the sentence and
calls for a stricter evaluation of semantic text similarity that factors in
lexical and syntactic structures. Semantic text similarity is one of the
oldest metrics to evaluate language understanding [78, 52, 70, 24, 23, 28, 65]
and despite recent evidence of lexical sensitivity, large benchmarks (e.g.,
[66, 48]) evaluate semantic similarity without explicitly considering the
lexical influence. In this work, we aim to address this gap by proposing a
dataset to perform joint evaluation of semantic understanding — through the
semantic equivalence detection task (elaborated below) — and lexical
sensitivity in language models.

Recognizing semantic similarity is often viewed as being fundamental to
language understanding, and strong performance on semantic text similarity is
often predictive of a language model’s performance in various downstream
applications [10] including question-answering, retrieval and summarization.
Based on overlap in their meaning, a pair of sentences can be roughly labelled
as semantically equivalent, semantically similar or semantically dissimilar.
More specifically, semantically equivalent sentences convey the same meaning,
perhaps differing in terms of syntactic and lexical structures. On the other
hand, sentences that are not semantically equivalent but describe the same
topic are said to be semantically similar [3]. Important examples include MRPC
[17], QQP [29] and STS [2, 3, 4, 5, 6]: while MRPC and QQP contain binary
labels indicating semantic equivalence, STS uses a score between 0 to 5 to
indicate the degree of semantic equivalence.

The timely release of these datasets have fuelled the research and development
of improved language models. While these datasets remain relevant even today
and are included as part of the challenging GLUE benchmark [71], we aim to
improve upon the following aspects to evaluate language understanding through
semantic equivalence task under controlled lexical constraints:

  1. (a)

Varying Definitions of Semantic Equivalence: Two sentences are said to be
semantically equivalent if the sentences convey the same meaning and can be
inferred from each other (i.e., bidirectional entailment). While MRPC [17]
aims to evaluate if language models can detect semantic equivalence, it
ultimately uses a loose definition of equivalence in that the sentence pairs
that convey different information about the same topic are also considered to
be semantically equivalent. Different from semantically equivalent sentences,
a pair of questions are defined to be semantically equivalent if they have the
same answer [58] and hence, datasets on semantically equivalent question pairs
(e.g., QQP) require additional knowledge beyond language understanding. In
contrast, we focus our evaluation on fundamental language understanding
ability and evaluate a language model in terms of its ability to recognize
semantic equivalence between a pair of sentences.

  2. (b)

Lack of Lexical Constraints: While achieving perfect scores on the existing
semantic similarity datasets is indeed challenging, trivial baselines using
lexical overlap also provide reasonable estimates of semantic similarity (for
example, see Figure 2 and Table 6 in Abdalla et al. [1]). Therefore, the
extent to which language models rely upon lexical structure when identifying
semantic equivalence and semantic similarity is not clearly known. We are thus
motivated to explore a more challenging setting that requires a language model
to encode semantics beyond superficial lexical structure.

Closer to our goal, Hsieh et al. [25] introduced the challenging SugarCrepe
dataset to evaluate the ability of vision-language models (VLMs) to identify
one correct caption in a pair of lexically similar sentences. As an example,
given an image, the model may be asked to select the correct caption between
“A tractor and two boats far from the water” (incorrect) and “A tractor and
two boats beside the water”(correct). Hsieh et al. [25] reported that several
VLMs face challenges in selecting the correct caption and attributed the low
performance to the text encoder’s inability to identify semantic differences
in a pair of lexically similar sentences. While this dataset provides a good
starting point, it is insufficient for comprehensively evaluating the lexical
sensitivity and semantic understanding of a model; the model’s understanding
of semantic equivalence in the presence of lexical differences remains unclear
from such an evaluation. For instance, “Couple of boats and a tractor located
next to the water” is semantically identical to the correct caption despite
lexical dissimilarities. A precise evaluation of lexical influence upon
semantic understanding should include pairs of semantically-equivalent,
semantically-opposite, lexically-similar, and lexically-dissimilar sentences.
In this work, we target these four cases and define the following research
questions:

  1. 1.

How well do LLMs understand the semantic equivalence between a pair of
sentences given their syntactic and lexical differences?

  2. 2.

How well do LLMs understand the semantic differences between a pair of
sentences given their syntactic and lexical similarities?

To that end, we extend SugarCrepe to introduce SugarCrepe++ dataset which
additionally contains semantically equivalent sentences that are lexically
dissimilar. The answer to these two questions enable us to evaluate the
semantic understanding while disentangling the effect of lexical matches
between sentences.

Figure 1: Examples from SugarCrepe++ (SC++) dataset. $P_{1}$ and $P_{2}$ are
semantically equivalent but lexically different while $N$ is semantically
different than both $P_{1}$ and $P_{2}$ despite its lexical similarity with
$P_{1}$. The adjacent line charts highlight the performance gaps in VLMs
discovered upon re-evaluation using SC++, and shows that strong lexical and
semantic understanding may not be required to achieve better performance on
SugarCrepe (SC).

Our contributions to this work are as follows:

  * •

SugarCrepe++ dataset. We introduce SugarCrepe++, a diverse, multi-modal and
human-validated dataset, especially designed to evaluate the sensitivity of
encoded semantics in language models to lexical composition. The introduction
of a three-way semantic (in)equivalence task enables evaluation with increased
resolution previously not possible with two captions[25]. Figure 1,
illustrates instances from five categories incorporated in SugarCrepe++ and
highlights the apparent gaps in performance of VLMs when re-evaluated using
SugarCrepe++.

  * •

Unified Evaluation. We designed SugarCrepe++ dataset such that the overlap of
semantic information between the two positive captions is always higher than
between the positive and negative captions, even without considering the
image. This allows us to evaluate on the Text-to-Text task twice for the same
triplet, using each positive caption as the reference once. These caption
triplets combined with an image

enables a previously unexplored dual-mode evaluation of VLMs in both Image-to-
Text and Text-to-Text settings.

In this work, we evaluate a comprehensive list of VLMs and standalone/unimodal
language models (ULMs) using SugarCrepe++. A few of the notable findings are
summarized below:

  * •

VLMs struggle to identify the differences between semantic and lexical
alterations, particularly if the lexical alterations are based on swapping
attributes or objects or replacing relations.

  * •

There exists a large gap between the VLMs and human-level performance,
signifying huge scope for improvement in VLMs.

  * •

Text encoders of VLMs form the major bottleneck in achieving better
performance on SugarCrepe++.

  * •

Even state-of-the-art ULMs fail to consistently dissociate semantics from
lexical forms i.e., they tend to select captions with higher lexical overlap
than the ones with higher semantic overlap.

##  2 SugarCrepe++

In this section, we describe the data generation and validation pipeline used
to create SugarCrepe++ dataset. The SugarCrepe dataset [25] — derived from MS-
COCO [43], a dataset of image-caption pairs — consists of a correct caption
and an incorrect caption for each image while ensuring that the two captions
are lexically similar. To create SugarCrepe++ dataset, we generate another
correct caption for each image such that it uses alternative lexical
representation while being semantically identical to the original correct
caption in SugarCrepe. If we refer to the correct captions as positives, the
incorrect caption can be termed as a hard negative due to its high lexical
overlap with one of the positive captions. In contrast to SugarCrepe,
SugarCrepe++ enables evaluation across both multimodal and unimodal settings
while also providing a comprehensive evaluation of semantic understanding with
lexical constraints. Additional related work is discussed in Appendix A.

###  2.1 Dataset Generation and Validation

Algorithm 1 Generation Pipeline

0: Original caption $P_{1}$

1: $P_{2}\leftarrow LLM(M\\{P_{1}\\})$

2: Accept $\leftarrow False$

3: while not Accept do

4: if Duplicated$(P_{1},P_{2})$ then

5: $P_{2}\leftarrow LLM(M\\{P_{1};P_{2}\\})$

6: else if Superfluous$(P_{2},P_{1})$ then

7: $P_{2}\leftarrow LLM(M\\{P_{1}\\})$

8: else

9: $Accept\leftarrow True$

10: return $P_{2}$

Algorithm 2 Automatic Validation Pipeline

0: Original caption $P_{1}$, Generated caption $P_{2}$

1: $Valid\leftarrow LLM(V\\{P_{2},P_{1}\\})$

2: while not Valid do

3: $P_{2}\leftarrow GenerationPipeline(M\\{P_{1}\\})$

4: $valid\leftarrow LLM(V\\{P_{2},P_{1}\\})$ Return $P_{2}$

Prior works [8, 21, 36, 51] have extensively used image-caption pairs from MS-
COCO; specifically Crepe [46] and its improved derivative SugarCrepe [25] that
leverages the recent advancements in conditional text generation using large
language models (LLMs) to generate hard negative captions, thereby overcoming
the issues with procedurally generated captions. The SugarCrepe dataset
consists of (only) one positive and one hard negative caption for each image.

Relative to the negative caption, a single positive caption can either have
low or high lexical overlap. The original SugarCrepe only captures the high
overlap case. To evaluate the sensitivity of encoded semantics to lexical
alteration, we require an additional positive caption with a different lexical
composition. We build SugarCrepe++ using instruction fine-tuned Mistral 7B
[31] model to further introduce an additional positive caption as illustrated
in Algorithms 1 and 2. The generation process of the additional caption can be
divided into two stages: (1) generation using meta-prompts and (2) automated
and human validation of generated captions.

Generation using meta-prompts: Algorithm 1 presents our generation pipeline.
We followed an iterative prompting methodology to refine a meta-prompt (M)
that generates optimal second positive captions (P2) given the original
positive caption (P1). The meta-prompt (M) is a composition of constituent
prompts (Mi) formed by concatenating (;) different sub-prompts.

$M=M_{1};M_{2};M_{3};\dots;M_{i}$

The meta-prompt is applied to an input ($x$) to obtain the final prompt M{x}
that conditions the LLM to generate the second positive caption (P2) as
$LLM(M\\{P_{1}\\})\xrightarrow{}P_{2}$. We find integrating different
techniques into a larger meta-prompt ($M$) improves the generation quality,
exploiting the benefits from each of them. Following [34], we prefix $M$ with
a ‘role-play’ prompt (M1) that conditions the LLM to the high-level task of
data generation and simulates the role of a ‘Data Generating AI.’ We utilize
‘Rules Prompting’ [84] to enhance LLMs’ faithfulness in instruction following
using explicit and itemized rules: in particular, we further condition M using
a ‘rules’ prompt (M2) that describes three rules to ensure the consistency of
the generated caption (P2). Expanding on the prompting methodology from [25],
we also append few-shot ‘demonstrations’ (M3) that include additional
reasoning and the generated caption. The ‘role-play’ (M1) prompt is described
in Figure 2. The ‘rules’ (M2) and ‘demonstrations’ (M3) sub-prompts are
elaborated in Figure 3 of Appendix B. During our initial testing, we noticed
the following systematically recurring errors: (i) LLM generated caption is
identical to the input caption; (ii) parsing failure due to superfluous
outputs. Consequently, we incorporate the following safeguards in the
generation pipeline as shown in Algorithm 1: 1) If the generated caption is
found to be identical to the input caption (using automatic tools), we use a
meta-prompt that includes the complete context with basic instructions to
regenerate P2; 2) We detect superfluous outputs based on word overlap and then
discard any generation that does not meet the minimum threshold.

Role-playing Prompt: You are an instruction-following DataGenAI. Your
expertise lies in accurately interpreting and generating datasets based on
given instructions. Your responses are expected to be precise and limited to
the required output. Figure 2: Role playing prompt for “Data Generator AI".

Automated and Human Validation of Generated Captions:  Since the above
safeguards do not ensure that the generated caption is always semantically
equivalent to the input caption, we require additional steps to ensure
semantic equivalence. We notice subtle differences in generated caption that
break the semantic equivalence and this also highlights the limitations of
semantic understanding in LLMs. Despite such limitations, prior works [50, 11]
have demonstrated that LLM agents — e.g., LLMs instantiated with different
prompts — can interact with each other to solve complex tasks. We build on
this to reduce human effort and optimize labelling costs using automatic
validation of generated captions similar to [86, 84]. We employ a validator
LLM agent, that is responsible for validating the semantic consistency of the
generated caption with the original caption and signalling the Generator agent
to retry as needed. Here, we refer to the LLM agents as two instances of the
same LLM conditioned on different meta-prompts.

We define a validation meta-prompt (V) that uses P1 and P2 from the previous
step to form a validation input to the validator LLM. Similar to the meta-
prompt (M), the validation prompt (V) consists of a sequence of sub-prompts,
including the ‘Validation instruction’ (V1) prompt and the ‘Validation
demonstration’ (V2). The validator LLM is conditioned using V to generate a
boolean value indicating the caption’s validity,

$LLM(V\\{P_{1},P_{2}\\})\rightarrow\\{True,False\\}$, where $V=V_{1};V_{2}$.
We use the boolean output value to trigger a regeneration step that starts the
generation pipeline again as described in Algorithm 2.

The prompt for employing validator LLM is detailed in Figure 4 in Appendix
B.3. In our automatic validation, we assume that the image is captioned
correctly and do not consider the image when ensuring semantic correctness.

Table 1: SugarCrepe++ consists of 4757 examples with the following distribution of sample sizes. We exclude the add-attribute and add-objects subsets of SUGARCREPE as they are not applicable to both the image-text task (ITT) and text-only task (TOT) of SugarCrepe++. This exclusion is necessary because the negative captions in these subsets may require visual information to be accurately identified as negative.  Swap Object | Swap Attribute | Replace Object | Replace Attribute | Replace relation | Total
---|---|---|---|---|---
245 | 666 | 1652 | 788 | 1406 | 4757

To ensure the quality of the SugarCrepe++ dataset, we conducted human
validation with two experts to correct the errors in the positive sentences
(P2) generated by LLMs (Appendix B.4 provides a list of common errors
generated by LLMs) and any disagreements between the expert annotators were
mutually resolved through inter-annotator discussion. These human annotators
also assessed the validity of caption triplets (P1, P2, $N$) and paired images
($I$) as a data point. The final statistics of the SugarCrepe++ dataset after
the human validation are described in Table 1.

##  3 Benchmark on SugarCrepe++ Dataset

Experimental Setup We benchmark VLM performance on SugarCrepe++ dataset under
two different evaluation settings. (1) Multi-modal image-to-text (ITT)
evaluation: both image and text are provided as inputs to evaluate VLMs in a
multi-modal setting. (2) Uni-modal text-only (TOT) evaluation: only text is
provided as input to evaluate the text encoders of VLMs in a unimodal setting.

Each sample in the SugarCrepe++ dataset consists of an image $I$ and
corresponding two positive captions (P1 and P2) and a negative caption (N).

If $p(X|I)$ denotes the likelihood of caption $X$ for image $I$, we compute
the ITT evaluation metric given P1, P2 and N as:

$\text{ITT}_{hit}=\begin{cases}1&(p(P_{1}|I)>p(N|I))\land(p(P_{2}|I)>p(N|I))\\\
0&\text{otherwise}\end{cases}$

For a VLM model such as CLIP which relies upon embeddings, the log-likelihood
$\log p$ is defined to be proportional to the cosine similarity between the
respective embeddings. Similarly, the TOT evaluation is defined as:

$\text{TOT}_{hit}=\begin{cases}1&(p(P_{1}|P_{2})>p(N|P_{2}))\land(p(P_{2}|P_{1})>p(N|P_{1})).\\\
0&\text{otherwise}\end{cases}$

As above, we use cosine-similarity for embedding-based models. We report the
performance in terms of Accuracy (%), computed as the ratio of the number of
hits to the total number of samples. we also perform a human evaluation to
calculate the human performance on the benchmark. Human evaluation is
performed by $4$ graduate-level students where each person was provided with a
randomly selected $150$ samples ($30$ from each subset) and was asked to
select the negative caption for each sample. In TOT setting, only the three
captions were provided. In the ITT setting, image along with the captions were
provided to the human evaluators. The average human performance is reported in
terms of accuracy (%).

Table 2: Comparison of VLMs performance on SugarCrepe++. Performance reported
in terms of Accuracy (%). Overall best values are in bold, and group-level
best values are underlined.

Model | Swap Object | Swap Attribute | Replace Object | Replace Attribute | Replace Relation
---|---|---|---|---|---
| ITT | TOT | ITT | TOT | ITT | TOT | ITT | TOT | ITT | TOT
Human | 100.00 | 96.67 | 96.67 | 93.3 | 100.00 | 95.00 | 100.00 | 98.33 | 100.00 | 96.67
CLIP [53] | 45.18 | 19.74 | 45.21 | 33.03 | 86.80 | 83.72 | 65.61 | 59.14 | 56.26 | 38.62
RoBERTa-ViT-B/32 [62] | 44.30 | 29.39 | 56.32 | 52.66 | 89.04 | 94.55 | 74.49 | 80.46 | 59.39 | 57.75
ALIGN [30] | 41.23 | 25.43 | 51.90 | 41.40 | 90.19 | 84.62 | 69.92 | 69.04 | 51.71 | 45.23
ALIP [81] | 36.84 | 20.18 | 46.12 | 28.77 | 71.49 | 50.06 | 54.95 | 34.52 | 47.80 | 23.47
FLAVA [64] | 54.39 | 45.18 | 59.21 | 57.84 | 89.59 | 94.43 | 73.35 | 72.46 | 60.10 | 57.97
ALBEF [37] | 28.94 | 10.09 | 36.83 | 18.87 | 76.27 | 55.57 | 56.35 | 30.33 | 47.80 | 30.65
BLIP [38] | 47.37 | 31.14 | 60.58 | 52.97 | 92.62 | 89.04 | 72.08 | 75.13 | 56.76 | 57.47
BLIP2 [39] | 35.09 | 21.49 | 37.60 | 29.98 | 89.41 | 72.58 | 62.82 | 64.47 | 53.27 | 43.47
ViLT [33] | 35.23 | – | 52.20 | – | 91.10 | – | 55.33 | – | 37.48 | –
AltCLIP [13] | 42.54 | 25.43 | 45.81 | 35.77 | 92.61 | 93.46 | 71.06 | 74.62 | 57.25 | 56.69
SegCLIP [45] | 45.61 | 25.44 | 46.12 | 40.64 | 85.90 | 95.16 | 62.69 | 67.89 | 54.84 | 41.96
XVLM-4M [83] | 31.14 | 10.96 | 36.52 | 19.48 | 79.42 | 67.07 | 59.39 | 40.74 | 46.23 | 29.23
XVLM-16M [83] | 34.21 | 18.86 | 40.33 | 31.05 | 90.81 | 92.07 | 68.02 | 70.43 | 57.47 | 47.87

###  3.1 Evaluation of VLMs on SugarCrepe++

We consider a variety of VLMs for evaluation using SugarCrepe++: 1) Models
trained with a contrastive learning objective such as CLIP [53], RoBERTa-
ViT-B/32 [62], ALIGN [30] and ALIP [81]. 2) Models trained by combining
multiple objective functions, such as FLAVA [64], ALBEF [37], BLIP [38] and
BLIP-2 [39]. 3) Models with a unified encoder for text and images, such as
ViLT [33], and multi-lingual distilled models like AltCLIP [13]; 4) Models
that align text with corresponding visual concepts in the image, such as
SegCLIP [45], and XVLM [83] \- with two variants, XVLM-4M and XVLM-16M. We
consider a wide array of VLMs that differ in terms of model architecture,
total number of parameters, embedding dimension and pre-training objectives to
measure the effect of various training choices on the model’s semantic
understanding capabilities. For further model details, refer to Appendix C.1.

Performance of VLMs on SugarCrepe++ is strongly influenced by the type of hard
negative.  The performance of VLMs on different subsets of SugarCrepe++ are
provided in Table 2. Swap type hard negatives, which are generated by swapping
either Objects or attributes, pose a significant challenge to VLMs, as most
achieve very low performance. Failing in examples with simple reordering of
words, as in swap subset, highlights a key limitation of VLMs in understanding
the structure of the input text. For replace-type hard negatives (generated by
replacing objects, attributes, or relations), VLMs are comparatively better at
discerning the negative from the positive caption when the object is replaced.
VLMs can also somewhat discern hard negatives from positives when attributes
are replaced. However, they struggle when the relation between objects is
replaced. Additionally, large performance gaps (ranging from 10% to 50%)
between the best models and human performance across most subsets signifies
opportunity for improvement in VLMs’ semantic understanding abilities.

Pre-training data size and objective functions affect VLM performance. Table 2
shows that the models trained with multiple objective functions, particularly
FLAVA and BLIP, perform better on SugarCrepe++ compared to models trained
using contrastive loss function alone. This indicates that the contrastive
learning objective alone may not be sufficient for VLMs to effectively learn
the semantic relations between text and images. Furthermore, models pre-
trained with smaller datasets, such as ALIP, ALBEF and XVLM-4M, have lower
performance compared to other models. Interestingly, these observations are
consistent across all subsets of SugarCrepe++.

Text encoders bottleneck VLM performance on SugarCrepe++.  All VLMs perform
significantly better on the ITT task compared to the TOT task on SugarCrepe++
(see Table 2). This shows that there is a higher ambiguity in identifying the
semantic and lexical alterations using only the text embeddings (TOT) compared
to the case of comparing the text embeddings with the image embeddings (ITT).
This is in agreement with the findings in [32]. Additionally, FLAVA, the best
performaning model on most of the subsets also achieves a good performance in
TOT setting, further signifying the importance of a strong text encoder in
achieving better performance.

Table 3: Evaluation of VLMs fine-tuned for image-text retrieval task.
Performance reported in terms of Accuracy (%). ITR dataset is the dataset used
to fine-tune the model for image-to-text retrieval.

Model | ITR | Swap Object | Swap Attribute | Replace Object | Replace Attribute | Replace Relation
---|---|---|---|---|---|---
| Dataset | ITT | TOT | ITT | TOT | ITT | TOT | ITT | TOT | ITT | TOT
ViLT [33] | – | 35.23 | – | 52.20 | – | 91.10 | – | 55.33 | – | 37.48 | –
XVLM-16M [83] | – | 34.21 | 18.86 | 40.33 | 31.05 | 90.81 | 92.07 | 68.02 | 70.43 | 57.47 | 47.87
ViLT-ITR-COCO [33] | MS-COCO | 50.88 | – | 73.36 | – | 89.89 | – | 71.95 | – | 61.24 | –
XVLM-16M-COCO [83] | MS-COCO | 39.91 | 21.06 | 49.93 | 51.60 | 91.65 | 96.79 | 74.24 | 83.63 | 63.09 | 62.87
XVLM-16M-Flickr [83] | Flickr | 45.61 | 21.49 | 50.53 | 44.44 | 91.71 | 96.01 | 74.24 | 81.59 | 64.01 | 59.89

Fine-tuning VLMs for image-text retrieval improves performance with
opportunity for further improvements.  Table 3 provides the performance of
VLMs (ViLT and XVLM-16M) fine-tuned for the task of image-to-text retrieval
(ITR). While we observe performance improvements on SugarCrepe++, VLMs still
face significant challenges in discerning negative captions from positive
ones, particularly for the Swap object and Replace relation subsets. Moreover,
there remains a substantial gap between VLM performance and human-level
performance. This indicates that VLMs capable of matching images to
corresponding captions do not necessarily learn the intricate details
regarding semantic information and lexical variations in the text.

Table 4: Performance of the methods for improving compositionality of VLMs on
SugarCrepe++. Performance reported in terms of Accuracy (%). Here, the
performance of CLIP is the baseline.

Model | Swap Object | Swap Attribute | Replace Object | Replace Attribute | Replace Relation
---|---|---|---|---|---
| ITT | TOT | ITT | TOT | ITT | TOT | ITT | TOT | ITT | TOT
Human | 100.00 | 96.67 | 96.67 | 93.3 | 100.00 | 97.00 | 100.00 | 98.33 | 100.00 | 96.67
CLIP [53] | 45.18 | 19.74 | 45.21 | 33.03 | 86.80 | 83.72 | 65.61 | 59.14 | 56.26 | 38.62
NegCLIP [82] | 55.25 | 34.65 | 57.99 | 56.47 | 89.53 | 94.55 | 69.41 | 76.27 | 52.27 | 51.57
CLIP-SVLC [18] | 42.98 | 18.86 | 48.40 | 34.56 | 80.93 | 91.56 | 56.98 | 66.88 | 47.30 | 51.28
BLIP-SGVL [22] | 13.16 | – | 38.81 | – | 53.75 | – | 34.39 | – | 30.65 | –
CyCLIP [20] | 37.72 | 13.60 | 34.40 | 18.72 | 70.28 | 78.29 | 49.87 | 49.12 | 40.41 | 29.87

Compositionality enhancing methods improve performance on SugarCrepe++ by
strengthening the VLM text encoder.  We evaluate recent methods proposed to
improve compositionality of VLMs, including NegCLIP [82], SVLC [18], CyCLIP
[20], and BLIP-SGVL [22]. As shown in Table 4, methods that improve
compositionality of CLIP such as NegCLIP and CLIP-SVLC also achieve better
performance on SugarCrepe++ compared to CLIP, underscoring the importance of
compositionality as a critical component for understanding semantic and
lexical differences. Interestingly, the text encoders of models with improved
compositionality (NegCLIP and CLIP-SVLC) perform significantly better than the
text encoder of CLIP in the TOT setting. Improved text encoders, in turn, lead
to improvements in ITT. On the other hand, methods such as BLIP-SGVL and
CyCLIP, which do not use explicit techniques to strengthen the text encoders,
show degradation in performance on SugarCrepe++. This further highlights the
importance of the text encoder in achieving better performance on
SugarCrepe++.

Table 5: Evaluation of several variants of CLIP on SugarCrepe++. Performance
reported in terms of Accuracy (%). Best performances in bold. RoB refers to
RoBERTa.

| #Model | Pre-train | Swap Object | Swap Attribute | Replace Object | Replace Attribute | Replace Relation
---|---|---|---|---|---|---|---
Model | Params | Data Size | ITT | TOT | ITT | TOT | ITT | TOT | ITT | TOT | ITT | TOT
CLIP [53] | 151M | 400M | 45.18 | 19.74 | 45.21 | 33.03 | 86.80 | 83.72 | 65.61 | 59.14 | 56.26 | 38.62
RN50$\times$4 [53] | 178M | 400M | 46.93 | 21.49 | 46.42 | 30.59 | 87.77 | 80.87 | 67.51 | 53.93 | 53.91 | 38.55
RN50$\times$64 [53] | 623M | 400M | 44.74 | 16.67 | 45.36 | 31.51 | 90.79 | 73.31 | 64.47 | 48.61 | 54.27 | 38.12
RoB-ViT-B/32 [62] | 212M | 2000M | 44.30 | 29.39 | 56.32 | 52.66 | 89.04 | 94.55 | 74.49 | 80.46 | 59.39 | 57.75
ViT-H/14 [62] | 986M | 2000M | 43.42 | 27.63 | 54.19 | 50.69 | 93.71 | 90.43 | 71.06 | 73.98 | 56.62 | 51.92
ViT-bigG/14 [62] | 2540M | 2000M | 45.61 | 29.82 | 57.38 | 52.05 | 94.13 | 90.44 | 76.41 | 72.84 | 59.45 | 53.49
XLM-RoB-ViT-B/32 [62] | 366M | 5000M | 42.55 | 30.26 | 55.25 | 55.56 | 89.41 | 95.34 | 72.97 | 80.96 | 55.48 | 57.82

Larger pre-training data and model size improve CLIP’s performance.  We
evaluated variants of CLIP that differ in model architecture and size, as well
as pre-training data size, on SugarCrepe++ (see Table 5). For CLIP variants
pre-trained on a dataset of 400 million image-text pairs, smaller models (CLIP
and RN50$\times$4) performed better than larger models (RN50$\times$64).
Interestingly, larger models (ViT-bigG/14) performed better than smaller
models when the pre-training data was increased to 2 billion samples.
Moreover, the text encoders also performed better when the pre-training data
was increased to 2 billion image-text pairs.

Table 6: Comparing the performance of VLMs on SugarCrepe and SugarCrepe++.
${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$
and
${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$
show increases and decreases in performance with the corresponding CLIP
performance as the baseline.

Model | Swap Object | Swap Attribute | Replace Object | Replace Attribute | Replace Relation
---|---|---|---|---|---
| SC | SC++ | SC | SC++ | SC | SC++ | SC | SC++ | SC | SC++
CLIP [53] | 59.21 | 45.18 | 64.99 | 45.21 | 90.86 | 86.8 | 80.33 | 65.61 | 70.48 | 56.26
ALBEF [37] |  63.16 (${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  28.94(${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$) |  69.25(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  36.83(${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$) |  93.04(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  76.27(${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$) |  84.65(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  56.35(${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$) |  77.60(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  47.80(${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$)
XVLM [83] |  64.91(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  31.14(${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$) |  73.97(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  36.52(${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$) |  95.22(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  79.42(${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$) |  87.69(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  59.39(${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$) |  77.45(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  46.23(${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$)
BLIP [38] |  66.22(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  47.37(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  76.25(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  60.58(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  96.55(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  92.62(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  81.98(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  72.08(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  68.35(${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$) |  56.76(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$)
NegCLIP [82] |  75.44(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  55.25(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  76.87(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  57.99(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  93.88(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  89.53(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  87.18(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  69.41(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  74.47(${\color[rgb]{0,1,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,1,0}\uparrow}$) |  52.27(${\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\downarrow}$)

Comparison of performance between SugarCrepe and SugarCrepe++ reveals
significant differences.  Table 6 compares the performance of VLMs between
SugarCrepe and SugarCrepe++. We observe significant differences in the trend
between model performances. When using CLIP’s performance as the baseline,
models such as ALBEF and XVLM which achieve better performance on SugarCrepe
show significant degradation in performance on SugarCrepe++. On the other
side, there are models such as BLIP and NegCLIP that show improvements on both
SugarCrepe and SugarCrepe++. Interestingly for the replace relation subset,
NegCLIP achieves better performance on SugarCrepe but shows degradation in
performance on SugarCrepe++. These results emphasize the importance of our
dataset in evaluating the models in terms of their semantic and lexical
understanding, which is not evident by evaluating on compositionality datasets
such as SugarCrepe.

###  3.2 Evaluation of ULMs on SugarCrepe++

We evaluate a comprehensive list of unimodal language models (ULMs) to
determine the semantic and lexical sensitivity of text-only models. We sample
ULMs covering various model sizes, architectures, and training objectives.
Recent state-of-the-art small-sized models include MiniLM [57], GTE [41] and
BGE [79]. From the results presented in Table 7, we observe trends similar to
the VLM’s performance in TOT task but more importantly, we observe ULMs
achieve significant improvements on average as compared to VLMs. Nevertheless,
we notice that some of these performances are far below human performance —
for instance, we observe a performance gap of $\approx$40% in swap object and
swap attribute subsets. Comprehensive results are available in Appendix D.

Table 7: Comparison of SOTA unimodal language models (ULMs) on SugarCrepe++.
We report the TOT accuracy (%), and group the results row-wise based on the
model size as reflected by the parameter count. We include the number of
parameters in text encoders relative to BERT-base, i.e., 109.5 Million
parameters. Overall, best values are in bold, and group-level best values are
underlined. We report the average across different subsets as an additional
column. Refer to Table 11 for additional results.

Model |  | #Params
---
(BERT Scale)
| Swap
---
Object
| Swap
---
Attribute
| Replace
---
Object
| Replace
---
Attribute
| Replace
---
Relation
Average
Human |  | 96.67 | 93.3 | 97.00 | 98.33 | 96.67 | 96.40
BGE-small-en-v1.5 [79] | 0.3 | 15.51 | 24.02 | 94.19 | 75.00 | 75.53 | 56.85
All-MiniLM-L12-v2 [77] | 0.3 | 18.78 | 25.38 | 95.22 | 73.86 | 70.41 | 56.73
Angle-BERT-base [40] | 1 | 25.71 | 33.63 | 92.07 | 78.43 | 75.32 | 61.03
BGE-base-en-v1.5 [79] | 1 | 17.14 | 25.23 | 93.83 | 78.55 | 76.10 | 58.17
UAE-Large-v1 [40] | 3 | 40.41 | 41.44 | 96.85 | 76.14 | 75.82 | 66.13
All-RoBERTa-large-v1 [57] | 3.1 | 42.04 | 45.20 | 94.61 | 74.75 | 74.96 | 66.31
Sentence-T5-xl [49] | 11.3 | 47.35 | 49.25 | 90.98 | 75.38 | 75.32 | 67.66
Angle-Llama-7b-nli-v2 [40] | 62.3 | 37.96 | 45.80 | 95.22 | 84.39 | 81.44 | 68.96
E5-Mistral-7b-instruct [75, 74] | 64.9 | 33.47 | 37.84 | 96.67 | 87.06 | 80.51 | 67.11

##  4 Conclusion

We introduce SugarCrepe++, a dataset for evaluating the ability of language
models, including both vision-language models (VLMs) and unimodal language
models (ULMs), to understand their sensitivity to semantic and lexical
alterations in text. Our dataset supports evaluations in both image-to-text
(ITT) and text-to-text (TOT) settings. We evaluated a comprehensive list of
VLMs and ULMs to highlight a fundamental limitation with these language models
in understanding semantic and lexical alterations. The key findings from our
evaluation are: (1) There is a significant performance gap between VLMs and
human-level performance, signifying a huge scope for improvement in VLMs. (2)
All VLMs exhibit difficulty in comprehending semantic and lexical alterations,
especially when these alterations involve swapping attributes or objects or
replacing relations. (3) Similarly, state-of-the-art (SOTA) ULMs lack a robust
understanding of lexical composition and consistently fail to separate
semantics from lexical forms. (4) While increasing pre-training data, model
size, and improving compositionality enhance performance on SugarCrepe++
dataset, these models still fall considerably short of human performance.
These insights underscore the critical need for advancements to close the
performance gap between models and human understanding. Our SugarCrepe++
dataset serves as a valuable tool for driving future research in this area.

## References

  * Abdalla et al. [2023] M. Abdalla, K. Vishnubhotla, and S. Mohammad.  What makes sentences semantically related? a textual relatedness dataset and empirical study.  In A. Vlachos and I. Augenstein, editors, _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 782–796, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.eacl-main.55.  URL https://aclanthology.org/2023.eacl-main.55.
  * Agirre et al. [2012] E. Agirre, M. Diab, D. Cer, and A. Gonzalez-Agirre.  Semeval-2012 task 6: a pilot on semantic textual similarity.  In _Proceedings of the First Joint Conference on Lexical and Computational Semantics - Volume 1: Proceedings of the Main Conference and the Shared Task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation_ , SemEval ’12, page 385–393, USA, 2012. Association for Computational Linguistics.
  * Agirre et al. [2013] E. Agirre, D. Cer, M. Diab, A. Gonzalez-Agirre, and W. Guo.  *SEM 2013 shared task: Semantic textual similarity.  In M. Diab, T. Baldwin, and M. Baroni, editors, _Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity_ , pages 32–43, Atlanta, Georgia, USA, June 2013. Association for Computational Linguistics.  URL https://aclanthology.org/S13-1004.
  * Agirre et al. [2014] E. Agirre, C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, R. Mihalcea, G. Rigau, and J. Wiebe.  SemEval-2014 task 10: Multilingual semantic textual similarity.  In P. Nakov and T. Zesch, editors, _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_ , pages 81–91, Dublin, Ireland, Aug. 2014. Association for Computational Linguistics.  doi: 10.3115/v1/S14-2010.  URL https://aclanthology.org/S14-2010.
  * Agirre et al. [2015] E. Agirre, C. Banea, C. Cardie, D. Cer, M. Diab, A. Gonzalez-Agirre, W. Guo, I. Lopez-Gazpio, M. Maritxalar, R. Mihalcea, G. Rigau, L. Uria, and J. Wiebe.  SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretability.  In P. Nakov, T. Zesch, D. Cer, and D. Jurgens, editors, _Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015)_ , pages 252–263, Denver, Colorado, June 2015. Association for Computational Linguistics.  doi: 10.18653/v1/S15-2045.  URL https://aclanthology.org/S15-2045.
  * Agirre et al. [2016] E. Agirre, C. Banea, D. Cer, M. Diab, A. Gonzalez-Agirre, R. Mihalcea, G. Rigau, and J. Wiebe.  SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation.  In S. Bethard, M. Carpuat, D. Cer, D. Jurgens, P. Nakov, and T. Zesch, editors, _Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016)_ , pages 497–511, San Diego, California, June 2016. Association for Computational Linguistics.  doi: 10.18653/v1/S16-1081.  URL https://aclanthology.org/S16-1081.
  * Alper et al. [2023] M. Alper, M. Fiman, and H. Averbuch-Elor.  Is bert blind? exploring the effect of vision-and-language pretraining on visual language understanding.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6778–6788, 2023.
  * Antol et al. [2015] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. L. Zitnick, and D. Parikh.  VQA: Visual Question Answering.  In _International Conference on Computer Vision (ICCV)_ , 2015.
  * Cao et al. [2021] N. D. Cao, W. Aziz, and I. Titov.  Editing factual knowledge in language models.  In M. Moens, X. Huang, L. Specia, and S. W. Yih, editors, _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021_ , pages 6491–6506. Association for Computational Linguistics, 2021.  doi: 10.18653/V1/2021.EMNLP-MAIN.522.  URL https://doi.org/10.18653/v1/2021.emnlp-main.522.
  * Cer et al. [2017] D. Cer, M. Diab, E. Agirre, I. Lopez-Gazpio, and L. Specia.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In S. Bethard, M. Carpuat, M. Apidianaki, S. M. Mohammad, D. Cer, and D. Jurgens, editors, _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada, Aug. 2017. Association for Computational Linguistics.
  * Chan et al. [2023] C. Chan, W. Chen, Y. Su, J. Yu, W. Xue, S. Zhang, J. Fu, and Z. Liu.  Chateval: Towards better llm-based evaluators through multi-agent debate.  _CoRR_ , abs/2308.07201, 2023.  doi: 10.48550/ARXIV.2308.07201.  URL https://doi.org/10.48550/arXiv.2308.07201.
  * Chen et al. [2023a] Z. Chen, G. Chen, S. Diao, X. Wan, and B. Wang.  On the difference of bert-style and clip-style text encoders.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 13710–13721, 2023a.
  * Chen et al. [2023b] Z. Chen, G. Liu, B. Zhang, Q. Yang, and L. Wu.  Altclip: Altering the language encoder in CLIP for extended language capabilities.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 8666–8682. Association for Computational Linguistics, 2023b.
  * Chow et al. [2023] K. J. Chow, S. Tan, and M.-Y. Kan.  Travlr: Now you see it, now you don’t! a bimodal dataset for evaluating visio-linguistic reasoning.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 3314–3339, 2023.
  * Ding et al. [2022] J. Ding, N. Xue, G. Xia, and D. Dai.  Decoupling zero-shot semantic segmentation.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022_ , pages 11573–11582. IEEE, 2022.
  * Diwan et al. [2022] A. Diwan, L. Berry, E. Choi, D. Harwath, and K. Mahowald.  Why is winoground hard? investigating failures in visuolinguistic compositionality.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022_ , pages 2236–2250. Association for Computational Linguistics, 2022.
  * Dolan and Brockett [2005] B. Dolan and C. Brockett.  Automatically constructing a corpus of sentential paraphrases.  In _Third International Workshop on Paraphrasing (IWP2005)_ , 2005.
  * Doveh et al. [2023] S. Doveh, A. Arbelle, S. Harary, E. Schwartz, R. Herzig, R. Giryes, R. Feris, R. Panda, S. Ullman, and L. Karlinsky.  Teaching structured vision & language concepts to vision & language models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 2657–2668, 2023.
  * Gadre et al. [2023] S. Y. Gadre, G. Ilharco, A. Fang, J. Hayase, G. Smyrnis, T. Nguyen, R. Marten, M. Wortsman, D. Ghosh, J. Zhang, et al.  Datacomp: In search of the next generation of multimodal datasets.  _arXiv preprint arXiv:2304.14108_ , 2023.
  * Goel et al. [2022] S. Goel, H. Bansal, S. Bhatia, R. Rossi, V. Vinay, and A. Grover.  Cyclip: Cyclic contrastive language-image pretraining.  _Advances in Neural Information Processing Systems_ , 35:6704–6719, 2022.
  * Goyal et al. [2017] Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh.  Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017.
  * Herzig et al. [2023] R. Herzig, A. Mendelson, L. Karlinsky, A. Arbelle, R. Feris, T. Darrell, and A. Globerson.  Incorporating structured representations into pretrained vision & language models using scene graphs.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023_ , pages 14077–14098. Association for Computational Linguistics, 2023.
  * Hliaoutakis et al. [2006a] A. Hliaoutakis, G. Varelas, E. G. Petrakis, and E. Milios.  Medsearch: A retrieval system for medical information based on semantic similarity.  In _Research and Advanced Technology for Digital Libraries: 10th European Conference, ECDL 2006, Alicante, Spain, September 17-22, 2006. Proceedings 10_ , pages 512–515. Springer Berlin Heidelberg, 2006a.
  * Hliaoutakis et al. [2006b] A. Hliaoutakis, G. Varelas, E. Voutsakis, E. G. Petrakis, and E. Milios.  Information retrieval by semantic similarity.  _International journal on semantic Web and information systems (IJSWIS)_ , 2(3):55–73, 2006b.
  * Hsieh et al. [2023] C.-Y. Hsieh, J. Zhang, Z. Ma, A. Kembhavi, and R. Krishna.  Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality.  In _Thirty-Seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2023.
  * IIMURA [2018] H. IIMURA.  Distractor plausibility in a multiple-choice listening test.  _JLTA Journal_ , 21:65–81, 2018.
  * Ilharco et al. [2021] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar, H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt.  Openclip, July 2021.  URL https://doi.org/10.5281/zenodo.5143773.
  * Islam et al. [2012] A. Islam, E. Milios, and V. Kešelj.  Text similarity using google tri-grams.  In _Advances in Artificial Intelligence: 25th Canadian Conference on Artificial Intelligence, Canadian AI 2012, Toronto, ON, Canada, May 28-30, 2012. Proceedings 25_ , pages 312–317. Springer Berlin Heidelberg, 2012.
  * Iyer et al. [2017] S. Iyer, N. Dandekar, and K. Csernai.  First quora dataset release: Question pairs, 2017.  URL https://quoradata.quora.com/First-Quora-Dataset-Release-Question-Pairs.  Accessed: 2024-01-01.
  * Jia et al. [2021] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In _International conference on machine learning_ , pages 4904–4916. PMLR, 2021.
  * Jiang et al. [2023] A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed.  Mistral 7b, 2023.
  * Kamath et al. [2023] A. Kamath, J. Hessel, and K.-W. Chang.  Text encoders bottleneck compositionality in contrastive vision-language models.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 4933–4944, 2023.
  * Kim et al. [2021] W. Kim, B. Son, and I. Kim.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _International Conference on Machine Learning_ , pages 5583–5594. PMLR, 2021.
  * Kong et al. [2023] A. Kong, S. Zhao, H. Chen, Q. Li, Y. Qin, R. Sun, and X. Zhou.  Better zero-shot reasoning with role-play prompting.  _CoRR_ , abs/2308.07702, 2023.
  * Krishna et al. [2023] K. Krishna, Y. Song, M. Karpinska, J. F. Wieting, and M. Iyyer.  Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.  URL https://openreview.net/forum?id=WbFhFvjjKj.
  * Le et al. [2024] T. Le, V. Lal, and P. Howard.  Coco-counterfactuals: Automatically constructed counterfactual examples for image-text pairs.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Li et al. [2021] J. Li, R. Selvaraju, A. Gotmare, S. Joty, C. Xiong, and S. C. H. Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ , 34:9694–9705, 2021.
  * Li et al. [2022] J. Li, D. Li, C. Xiong, and S. Hoi.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR, 2022.
  * Li et al. [2023a] J. Li, D. Li, S. Savarese, and S. Hoi.  BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models.  In _ICML_ , 2023a.
  * Li and Li [2023] X. Li and J. Li.  Angle-optimized text embeddings.  _arXiv preprint arXiv:2309.12871_ , 2023.
  * Li et al. [2023b] Z. Li, X. Zhang, Y. Zhang, D. Long, P. Xie, and M. Zhang.  Towards general text embeddings with multi-stage contrastive learning, 2023b.
  * Liang et al. [2023] F. Liang, B. Wu, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu.  Open-vocabulary semantic segmentation with mask-adapted CLIP.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023_ , pages 7061–7070. IEEE, 2023.
  * Lin et al. [2014] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick.  Microsoft coco: Common objects in context.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13_ , pages 740–755. Springer, 2014.
  * Liu et al. [2023] F. Liu, G. Emerson, and N. Collier.  Visual spatial reasoning.  _Transactions of the Association for Computational Linguistics_ , 11:635–651, 2023.
  * Luo et al. [2023] H. Luo, J. Bao, Y. Wu, X. He, and T. Li.  Segclip: Patch aggregation with learnable centers for open-vocabulary semantic segmentation.  In _International Conference on Machine Learning_ , pages 23033–23044. PMLR, 2023.
  * Ma et al. [2023] Z. Ma, J. Hong, M. O. Gul, M. Gandhi, I. Gao, and R. Krishna.  Crepe: Can vision-language foundation models reason compositionally?  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 10910–10921, 2023.
  * Meng et al. [2022] K. Meng, D. Bau, A. Andonian, and Y. Belinkov.  Locating and editing factual associations in GPT.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 - December 9, 2022_ , 2022.
  * Muennighoff et al. [2023] N. Muennighoff, N. Tazi, L. Magne, and N. Reimers.  MTEB: massive text embedding benchmark.  In A. Vlachos and I. Augenstein, editors, _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, EACL 2023, Dubrovnik, Croatia, May 2-6, 2023_ , pages 2006–2029. Association for Computational Linguistics, 2023.
  * Ni et al. [2022] J. Ni, G. H. Ábrego, N. Constant, J. Ma, K. B. Hall, D. Cer, and Y. Yang.  Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 1864–1874. Association for Computational Linguistics, 2022.
  * Park et al. [2023a] J. S. Park, J. C. O’Brien, C. J. Cai, M. R. Morris, P. Liang, and M. S. Bernstein.  Generative agents: Interactive simulacra of human behavior.  In _In the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23)_ , UIST ’23, New York, NY, USA, 2023a. Association for Computing Machinery.
  * Park et al. [2023b] S. Park, D. Um, H. Yoon, S. Chun, S. Yun, and J. Y. Choi.  Rococo: Robustness benchmark of ms-coco to stress-test image-text matching models.  _arXiv preprint arXiv:2304.10727_ , 2023b.
  * Peng and Roth [2016] H. Peng and D. Roth.  Two discourse driven language models for semantics.  In K. Erk and N. A. Smith, editors, _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 290–300, Berlin, Germany, Aug. 2016. Association for Computational Linguistics.  doi: 10.18653/v1/P16-1028.  URL https://aclanthology.org/P16-1028.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Ramesh et al. [2021] A. Ramesh, M. Pavlov, G. Goh, S. Gray, C. Voss, A. Radford, M. Chen, and I. Sutskever.  Zero-shot text-to-image generation.  In _International Conference on Machine Learning_ , pages 8821–8831. PMLR, 2021.
  * Ramesh et al. [2022] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 2022.
  * Ray et al. [2023] A. Ray, F. Radenovic, A. Dubey, B. A. Plummer, R. Krishna, and K. Saenko.  Cola: How to adapt vision-language models to compose objects localized with attributes?  _arXiv preprint arXiv:2305.03689_ , 2023.
  * Reimers and Gurevych [2019] N. Reimers and I. Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, 11 2019.
  * Rodrigues et al. [2018] J. Rodrigues, C. Saedi, A. Branco, and J. Silva.  Semantic equivalence detection: Are interrogatives harder than declaratives?  In N. Calzolari, K. Choukri, C. Cieri, T. Declerck, S. Goggi, K. Hasida, H. Isahara, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, S. Piperidis, and T. Tokunaga, editors, _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan, May 2018. European Language Resources Association (ELRA).  URL https://aclanthology.org/L18-1513.
  * Saharia et al. [2022] C. Saharia, W. Chan, S. Saxena, L. Li, J. Whang, E. L. Denton, K. Ghasemipour, R. Gontijo Lopes, B. Karagol Ayan, T. Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  _Advances in Neural Information Processing Systems_ , 35:36479–36494, 2022.
  * Sahin et al. [2024] U. Sahin, H. Li, Q. Khan, D. Cremers, and V. Tresp.  Enhancing multimodal compositional reasoning of visual language models with generative negative mining.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 5563–5573, 2024.
  * Schiappa et al. [2023] M. C. Schiappa, M. Cogswell, A. Divakaran, and Y. S. Rawat.  Probing conceptual understanding of large visual-language models.  _arXiv preprint arXiv:2304.03659_ , 2023.
  * Schuhmann et al. [2022] C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al.  Laion-5b: An open large-scale dataset for training next generation image-text models.  _Advances in Neural Information Processing Systems_ , 35:25278–25294, 2022.
  * Sclar et al. [2023] M. Sclar, Y. Choi, Y. Tsvetkov, and A. Suhr.  Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting, 2023.
  * Singh et al. [2022] A. Singh, R. Hu, V. Goswami, G. Couairon, W. Galuba, M. Rohrbach, and D. Kiela.  Flava: A foundational language and vision alignment model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15638–15650, 2022.
  * Soto et al. [2015] A. J. Soto, A. Mohammad, A. Albert, A. Islam, E. Milios, M. Doyle, R. Minghim, and M. C. Ferreira de Oliveira.  Similarity-based support for text reuse in technical writing.  In _Proceedings of the 2015 ACM Symposium on Document Engineering_ , pages 97–106, 2015.
  * Srivastava et al. [2022] A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, 2022.  URL https://arxiv.org/abs/2206.04615.
  * Su et al. [2023] H. Su, W. Shi, J. Kasai, Y. Wang, Y. Hu, M. Ostendorf, W. Yih, N. A. Smith, L. Zettlemoyer, and T. Yu.  One embedder, any task: Instruction-finetuned text embeddings.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 1102–1121. Association for Computational Linguistics, 2023.
  * Taladngoen and Esteban [2022] U. Taladngoen and R. H. Esteban.  Assumptions on plausible lexical distractors in the redesigned toeic question-response listening test.  _LEARN Journal: Language Education and Acquisition Research Network_ , 15(2):802–829, 2022.
  * Thrush et al. [2022] T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross.  Winoground: Probing vision and language models for visio-linguistic compositionality.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5238–5248, 2022.
  * Varelas et al. [2005] G. Varelas, E. Voutsakis, P. Raftopoulou, E. G. Petrakis, and E. E. Milios.  Semantic similarity methods in wordnet and their application to information retrieval on the web.  In _Proceedings of the 7th annual ACM international workshop on Web information and data management_ , pages 10–16, 2005.
  * Wang et al. [2018] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In T. Linzen, G. Chrupała, and A. Alishahi, editors, _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium, Nov. 2018. Association for Computational Linguistics.  doi: 10.18653/v1/W18-5446.  URL https://aclanthology.org/W18-5446.
  * Wang et al. [2019] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _7th International Conference on Learning Representations, ICLR 2019_. OpenReview.net, 2019.
  * Wang et al. [2023a] F. Wang, L. Ding, J. Rao, Y. Liu, L. Shen, and C. Ding.  Can linguistic knowledge improve multimodal alignment in vision-language pretraining?  _arXiv preprint arXiv:2308.12898_ , 2023a.
  * Wang et al. [2022] L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei.  Text embeddings by weakly-supervised contrastive pre-training.  _arXiv preprint arXiv:2212.03533_ , 2022.
  * Wang et al. [2023b] L. Wang, N. Yang, X. Huang, L. Yang, R. Majumder, and F. Wei.  Improving text embeddings with large language models.  _arXiv preprint arXiv:2401.00368_ , 2023b.
  * Wang et al. [2023c] S. Wang, Y. Zhu, H. Liu, Z. Zheng, C. Chen, and J. Li.  Knowledge editing for large language models: A survey, 2023c.
  * Wang et al. [2020] W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers, 2020.
  * Wong and Mooney [2006] Y. W. Wong and R. Mooney.  Learning for semantic parsing with statistical machine translation.  In R. C. Moore, J. Bilmes, J. Chu-Carroll, and M. Sanderson, editors, _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_ , pages 439–446, New York City, USA, June 2006. Association for Computational Linguistics.  URL https://aclanthology.org/N06-1056.
  * Xiao et al. [2023] S. Xiao, Z. Liu, P. Zhang, and N. Muennighoff.  C-pack: Packaged resources to advance general chinese embedding, 2023.
  * Xu et al. [2023] P. Xu, W. Shao, K. Zhang, P. Gao, S. Liu, M. Lei, F. Meng, S. Huang, Y. Qiao, and P. Luo.  Lvlm-ehub: A comprehensive evaluation benchmark for large vision-language models.  _arXiv preprint arXiv:2306.09265_ , 2023.
  * Yang et al. [2023] K. Yang, J. Deng, X. An, J. Li, Z. Feng, J. Guo, J. Yang, and T. Liu.  Alip: Adaptive language-image pre-training with synthetic caption.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2922–2931, 2023.
  * Yuksekgonul et al. [2023] M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Zou.  When and why vision-language models behave like bags-of-words, and what to do about it?  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Zeng et al. [2022] Y. Zeng, X. Zhang, and H. Li.  Multi-grained vision language pre-training: Aligning texts with visual concepts.  In _International Conference on Machine Learning_ , pages 25994–26009. PMLR, 2022.
  * Zeng et al. [2023] Z. Zeng, J. Yu, T. Gao, Y. Meng, T. Goyal, and D. Chen.  Evaluating large language models at evaluating instruction following.  _CoRR_ , abs/2310.07641, 2023.
  * Zhao et al. [2022] T. Zhao, T. Zhang, M. Zhu, H. Shen, K. Lee, X. Lu, and J. Yin.  Vl-checklist: Evaluating pre-trained vision-language models with objects, attributes and relations.  _arXiv preprint arXiv:2207.00221_ , 2022.
  * Zheng et al. [2023] L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena.  _CoRR_ , abs/2306.05685, 2023.
  * Zhou et al. [2023] K. Zhou, E. Lai, W. B. A. Yeong, K. Mouratidis, and J. Jiang.  ROME: evaluating pre-trained vision-language models on reasoning beyond visual common sense.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 10185–10197. Association for Computational Linguistics, 2023.
  * Zou et al. [2023] A. Zou, Z. Wang, J. Z. Kolter, and M. Fredrikson.  Universal and transferable adversarial attacks on aligned language models.  _CoRR_ , abs/2307.15043, 2023.  doi: 10.48550/ARXIV.2307.15043.  URL https://doi.org/10.48550/arXiv.2307.15043.

## Checklist

  1. 1.

For all authors…

    1. (a)

Do the main claims made in the abstract and introduction accurately reflect
the paper’s contributions and scope? [Yes]

    2. (b)

Did you describe the limitations of your work? [Yes]

    3. (c)

Did you discuss any potential negative societal impacts of your work? [Yes]

    4. (d)

Have you read the ethics review guidelines and ensured that your paper
conforms to them? [Yes]

  2. 2.

If you are including theoretical results…

    1. (a)

Did you state the full set of assumptions of all theoretical results? [N/A]

    2. (b)

Did you include complete proofs of all theoretical results? [N/A]

  3. 3.

If you ran experiments (e.g. for benchmarks)…

    1. (a)

Did you include the code, data, and instructions needed to reproduce the main
experimental results (either in the supplemental material or as a URL)? [Yes]
We include the Github link to our data and code in Appendix G.4.

    2. (b)

Did you specify all the training details (e.g., data splits, hyperparameters,
how they were chosen)? [N/A] We perform zero-shot evaluation of models. We did
not train any models for this work.

    3. (c)

Did you report error bars (e.g., with respect to the random seed after running
experiments multiple times)? [N/A]

    4. (d)

Did you include the total amount of compute and the type of resources used
(e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] We provide
these details in Appendix G

  4. 4.

If you are using existing assets (e.g., code, data, models) or
curating/releasing new assets…

    1. (a)

If your work uses existing assets, did you cite the creators? [Yes] We cite
the existing assets in Section 2 and Appendix G.

    2. (b)

Did you mention the license of the assets? [Yes] We mention these details in
Appendix G

    3. (c)

Did you include any new assets either in the supplemental material or as a
URL? [Yes] We include the GitHub link to our data and code in the
Supplementary material

    4. (d)

Did you discuss whether and how consent was obtained from people whose data
you’re using/curating? [Yes] We mention in Appendix G

    5. (e)

Did you discuss whether the data you are using/curating contains personally
identifiable information or offensive content? [Yes] We mention in Appendix G

  5. 5.

If you used crowdsourcing or conducted research with human subjects…

    1. (a)

Did you include the full text of instructions given to participants and
screenshots, if applicable? [Yes] See Figure 6

    2. (b)

Did you describe any potential participant risks, with links to Institutional
Review Board (IRB) approvals, if applicable? [N/A]

    3. (c)

Did you include the estimated hourly wage paid to participants and the total
amount spent on participant compensation? [Yes] we did not hire any
participants for the study and the evaluation is conducted in the research
group.

##  Appendix A Related Work

VLMs and ULMs have achieved impressive results on a range of vision and
language downstream tasks. These state-of-the-art VLMs and ULMs serve as
foundation models for both multimodal applications, like image captioning
[39], semantic segmentation of images [15, 42], text-to-image generation [54,
55, 59], and unimodal applications, like clustering [75, 74], reranking [79],
and retrieval [40]. Their emergence as foundation models has motivated recent
research to evaluate the strengths and weaknesses of these models. We
summarize the findings from common benchmarks of VLMs and ULMs below.

Findings from the Existing Benchmark for VLMs:  Thrush et al. [69] evaluate
VLMs through an image-text retrieval task and find that SOTA VLMs struggle to
distinguish between texts containing the same words but ordered differently.
Similarly, Yuksekgonul et al. [82] evaluate VLMs in terms of their abilities
to form object-attribute associations and highlight shortcomings of VLMs.
Other studies with similar conclusions include [85], [56] and [73]. Recent
works have introduced benchmarks to evaluate different abilities of VLMs such
as counter-intuitive reasoning [87], visual question answering [80],
conceptual understanding [61], visio-linguistic reasoning [14], visual-spatial
reasoning [44] and compositionality [69]. Kamath et al. [32] demonstrate
challenges in decoding salient aspects of input text encoded with CLIP and
draw connections to the lack of compositionality in CLIP text embeddings.

The task of evaluating compositionality in VLMs is the nearest neighbour to
our work. Several datasets have been introduced to evaluate the
compositionality of VLMs [44, 25, 69, 85, 82, 46, 56, 73, 60]. Most of the
existing compositionality benchmarks formulate the evaluation task as image-
text retrieval. Winoground [69] is one of the earliest benchmarks to report
the lack of compositional understanding in VLMs. Latest benchmarks
encompassing different aspects of compositionality include VL-CheckList [85],
CREPE [46], Cola [56], and ARO [82]. Some benchmarks like Winoground have
challenges beyond compositionality that include additional visual and textual
reasoning [16].

Findings from the Existing Benchmarks for ULMs: In the context of ULM text
encoders, paraphrasing is the closest to our paper. Paraphrasing is a well-
studied problem in NLP. Several previous studies analyzed the ability of the
language models to recognize paraphrasing in text. The Microsoft Research
Paraphrase Corpus (MRPC) [17] and Quora Question Pairs (QQP) [29] are popular
paraphrasing datasets (text-only without images) that are part of the GLUE
(General Language Understanding Evaluation) [72] benchmark. The Semantic
Textual Similarity (STS) benchmark [10] build from the STS shared tasks [2, 3,
4, 5, 6] have pairs of text snippets with scores indicating the degree of
semantic equivalence between them.

Shortcomings of existing Benchmarks: Alper et al. [7] find that the CLIP text
encoder outperforms the ULMs in tasks that require implicit visual reasoning,
while Chen et al. [12] find that ULMs perform better in terms of general
language understanding. Most VLM benchmarks are generated using rule-based
algorithms [46, 82] and consist of only a pair of sentences (either
semantically similar or dissimilar sentences). These similar pairs might not
have strong semantic similarities, and the dissimilar pairs can have
significant lexical differences, which does not represent a strict setting of
evaluation. Moreover, we must finetune or linearly probe [44] these models to
evaluate LLM encoders using these datasets, which can require significant
resources. None of the existing benchmarks systematically evaluates the
resilience of model embeddings in the presence of lexical distractors [26,
68], i.e., lexically similar but semantically different negative inputs.

##  Appendix B SugarCrepe++ Benchmark Generation

###  B.1 Dataset Guidelines

The main guidelines followed to create the benchmarks are:

  * •

The lexical changes allowed for creation of the three captions include,
replacing words with synonyms and antonyms, reordering the words, etc. These
lexical changes do not include adding more details about the image in the
caption.

  * •

Due to the lexical alterations, the three captions should not consist of any
nonsensical and non-fluent errors.

  * •

The three captions should be generated such that they do not need any visual,
logical or commonsense reasoning to distinguish the semantically similar
captions (P1 and P2) from the semantically different caption (N) i.e., given
only three captions without image, one should be able to distinguish P1, P2
from N.

###  B.2 Prompt for SugarCrepe++

Figure 3 shows the prompts used to condition the generation of P2 using LLM,
given P1. Here, we use instruct fine-tuned Mistral 7B [31] model to generate
P2.

Rules Instruction: Given an input sentence describing an image caption, follow
these steps: 1. Rephrase each provided sentence, focusing on preserving the
original spatial relationship. 2. Pay careful attention to the positioning of
objects or entities in relation to one another. 3. Ensure that the meaning
remains consistent and that both the original and paraphrased sentences
maintain logical coherence and grammatical correctness. Demonstration: For
example, Input: Cat is under the table. Paraphrase Idea: Rephrase the sentence
to convey that the table is positioned above the cat. Paraphrased: The table
is above the cat. Input:The plane flies below the bright white clouds.
Paraphrase Idea: Ensure the spatial context is maintained by stating that the
bright white clouds are situated above the flying plane. Paraphrased: The
plane flies below the bright white clouds.s Input: The third balcony is below
the fourth balcony from the bottom. Paraphrase Idea: Emphasize the consistent
spatial arrangement while indicating that the fourth balcony is positioned
above the third balcony from the bottom. Paraphrased: The fourth balcony is
above the third balcony from the bottom. Remember to keep the meaning intact,
and both the original and paraphrased sentences should be logically coherent
and grammatically correct. Input: [Original caption goes here] Paraphrase
Idea: Focus on replicating the spatial arrangement while maintaining the
original meaning of the sentence, correct grammar, same meaning. Paraphrased:
[Your paraphrased sentence goes here] Figure 3: Rules and demonstration sub-
prompts used to condition the generator LLM.

###  B.3 Validation prompt for SugarCrepe++

Figure 4 shows the comprehensive prompt used to validate the samples generated
by priming the LLM. The outputs obtained from this prompt are further
validated by a human expert. This reduces the manual effort required to create
the SugarCrepe++ dataset.

Instruction: Given a pair of captions, you must check if the second caption is
semantically consistent with the first caption. If the second caption is
consistent, output the second caption as is; otherwise, rephrase the second
caption to be consistent with the first sentence. We are interested in
preserving the spatial consistency and spatial relationship of the objects
with each other. Demonstrations: examples, Caption 1: A guy holding a
skateboard is speaking into a microphone. Caption 2: The guy holding the
microphone is speaking into the skateboard. isConsistent: No, you cannot speak
into a skateboard. newCaption: The guy is speaking into the microphone while
holding a skateboard. Caption 1: A family are playing frisbee on the beach.
Caption 2: The frisbee is being played on the beach by a family. isConsistent:
Yes, caption 2 is consistent as it is the same caption written in passive
voice. new caption is the same as caption 2. newCaption: A family are playing
frisbee on the beach. caption 1:A stop sign vandalized with an "eating
animals" sticker below the word "stop." caption 2: The stop sign is below an
"eating animals" sticker. isConsistent: The stop cannot be below and above the
sticker at the same time. newCaption: The word "stop" sign is above an "eating
animals" sticker. caption 1:There is a phone on top of a calculator. caption
2: A calculator lies beneath the phone. isConsistent:Yes, the sentences are
semantically equivalent. new caption is same as caption 2. newCaption: A
calculator lies beneath the phone. Now the same for the below caption only.
caption 1: [Original caption goes here] caption 2: [Generated caption goes
here] isConsistent: [Output Here] Figure 4: Validation Meta-prompt used to
validate the consistency of the generated caption and original caption. We use
the isConsistent output to signal the regeneration of a semantically
inconsistent caption.

###  B.4 Incorrect generation artifacts by LLMs

As shown in Figure 5, We observe various generation artifacts in the positive
sentences (P2) generated by LLMs that were corrected during the human
validation stage.

Figure 5: Examples of common errors in the LLM generated positive sentences
(P2). We provide five of the most common types of errors. Manual correction
refers to the corrected sentences after the human validation step. Expert
human annotators carefully checked each output of the LLM and corrected the
erroneous sentences to maintain grammatical and semantic equivalence with the
original positive caption (P1)

##  Appendix C Additional Analysis of VLMs

Table 8: Details of the VLMs evaluated on the SugarCrepe++ dataset.
Pretraining Data type: R, N and S refer to Real, Noisy and Synthetic data
types, respectively. Pretraining Objectives – ITC: image-text contrastive;
ITM: image-text matching; MLM: masked language modeling; MMM: masked
multimodal modeling; MIM: masked image modeling; IC: image captioning; IS:
image segmentation using KL divergence; ITA: image-text alignment; CCL: Cycle-
consistency loss; finetuning objectives – ITR: image-text retrieval; NL:
Negative loss for text; SG: scene graph loss; PT, FT refer to pretraining and
finetuning, respectively. CLIP refers to CLIP-ViT-B-32 model.

Model | #Total | Embedding | Pretraining | Pretraining | Pretraining |
---|---|---|---|---|---|---
Parameters | Dimension | Data size | Data Type | Objectives | Finetuned
CLIP 2021 | 151M | 512 | 400M | R | ITC | ✗
RoBERTa-ViT-B-32 2022 | 212M | 512 | 2B | R | ITC | ✗
ALIGN 2021 | 490M | 640 | 1.8B | R+N | ITC | ✗
ALIP 2023 | 151M | 512 | 15M | R+S | ITC | ✗
FLAVA 2022 | 358M | 768 | 70M | R | ITC, ITM, MLM | ✗
MMM, MIM
ALBEF 2021 | 210M | 256 | 14M | R+N | ITC, ITM, MLM | ✗
BLIP 2022 | 225M | 512 | 129M | R+S | ITC, ITM, IC | ✗
BLIP2 2023a | 1173M | 256 | 129M | R+S | ITC, ITM, IC | ✗
ViLT 2021 | 111M | 768 | 10M | R | ITM, MLM | ✗
AltCLIP 2023b | 864M | 768 | 42M | R | ITC | ✗
SegCLIP 2023 | 151M | 512 | 400M+4M | R | ITC, MIM, IS | ✗
XVLM-4M 2022 | 216M | 256 | 4M | R | ITC, ITM, MLM, ITA | ✗
XVLM-16M 2022 | 216M | 256 | 16M | R | ITC, ITM, MLM, ITA | ✗
ViLT-ITR-COCO 2021 | 111M | 768 | PT: 10M | R | ITM, MLM | ✓
FT: 110K | FT: ITR
XVLM-16M-COCO 2022 | 216M | 256 | PT:16M | R | ITC, ITM, MLM, ITA | ✓
FT: 110K | FT: ITR
XVLM-16M-Flickr 2022 | 216M | 256 | PT: 16M | R | TC, ITM, MLM, ITA | ✓
FT: 30K | FT: ITR
NegCLIP 2023 | 151M | 512 | PT: 400M | R | ITC | ✓
| FT:110K | FT: ITM
CLIP-SVLC 2023 | 151M | 512 | PT:400M | R | ITC | ✓
FT:400M | FT: ITC, NL
BLIP-SGVL 2023 | 696M | 768 | PT: 129M | R | ITC, ITM, IC | ✓
FT:4M | FT: ITC, SG
CyCLIP 2022 | 102M | 1024 | PT: 102M | R | ITC, CCL | ✗

###  C.1 Specification of Evaluated VLMs

We comprehensively evaluate a wide array of VLMs on SugarCrepe++ including
(Table 8 provides details about different VLMs.):

  * •

Models trained with a contrastive learning objective such as CLIP [53],
RoBERTa-ViT-B-32 [62], ALIGN [30] and ALIP [81]. ALIGN and ALIP utilize noisy
and synthetic captions, respectively.

  * •

Models trained by combining multiple objective functions, such as FLAVA [64]:
pretrained by combining contrastive, Image-text matching (ITM), masked image
modeling (MIM) and masked language modeling (MLM) objectives; ALBEF [37]:
which combines ITM and MLM; BLIP [38] and BLIP-2 [39]: which combine
contrastive, ITM and image captioning objectives.

  * •

Models with a unified encoder for text and images, such as ViLT [33], and
multi-lingual distilled models like AltCLIP [13]

  * •

Models that align text with corresponding visual concepts in the image, such
as SegCLIP [45], and XVLM [83] \- with two variants, XVLM-4M and XVLM-16M.

  * •

We also evaluate several models that have been finetuned on downstream tasks
of image-text retrieval, such as ViLT-ITR-COCO [33] and XVLM-16M-ITR-COCO
[83]. Specifically, ViLT, and XVLM-16M models were trained for the ITM task
using the COCO dataset. Additionally, XVLM-16M-ITR-Flickr [83] denotes
XVLM-16M models trained for the ITM task using the Flickr dataset.

  * •

Moreover, we evaluate recent methods proposed to improve the compositionality
of VLMs, including NegCLIP [82], SVLC [18], CyCLIP [20], and BLIP-SGVL [22].

###  C.2 Evaluation of variants of CLIP.

We evaluated variants of CLIP [53] that are different in pre-training data
size, model architecture and model size as listed below (see Table 9 for more
details).

  * •

CLIP [53] variants trained on the WebImageText dataset, which comprises 400
million image-text pairs. These models encompass CNN-based architectures, such
as RN50, RN101, RN$50\times 4$, RN$50\times 16$, and RN$50\times 64$, as well
as transformer-based models like ViT-B/32 and ViT-L/14.

  * •

CLIP-based models introduced by [62] pre-trained on extensive paired image-
text datasets. Schuhmann et al. [62] provided diverse CLIP variants, namely
RoBERTa-ViT-B/32, ViT-H/14, ViT-g/14, xlm-roberta-base-ViT-B/32, and xlm-
roberta-large-ViT-H/14, trained on a large image-text dataset called
"LAION-5B," which consists of 5 billion image-text pairs.

  * •

Gadre et al. [19] released two CLIP variants, namely Large:ViT-B/16 and
xlarge:ViT-L/14, trained on the DataComp dataset, comprising 13 billion image-
text pairs.

Performance of various CLIP variants on SugarCrepe++ dataset is provided in
Table 10.

Table 9: Details of different variants of CLIP that are evaluated on the SugarCrepe++ dataset. Data, Model and Emb. refer to the pre-training dataset size and total number of parameters in the model (in Millions) and embedding dimension, respectively. | Pre-training | Pre-training | # Params | Embed.
---|---|---|---|---
Model | Dataset | Data size | Model | Dimen.
RN50 | WebImageText | 400M | 102M | 1024
RN101 | WebImageText | 400M | 120M | 512
CLIP | WebImageText | 400M | 151M | 512
RN50$\times$4 | WebImageText | 400M | 178M | 640
RN50$\times$16 | WebImageText | 400M | 291M | 768
CLIP-ViT-L/14 | WebImageText | 400M | 428M | 768
RN50$\times$64 | WebImageText | 400M | 623M | 1024
RoBERTa-ViT-B/32 | LAION | 2B | 212M | 512
ViT-H/14 | LAION | 2B | 986M | 1024
ViT-g/14 | LAION | 2B | 1367M | 1024
ViT-bigG/14 | LAION | 2B | 2540M | 1280
xlm-roberta-base-ViT-B/32 | LAION | 5B | 366M | 512
xlm-roberta-large-ViT-H/14 | LAION | 5B | 1193M | 1024
large:ViT-B/16 | DataComp | 1B | 150M | 512
xlarge:ViT-L/14 | DataComp | 13B | 428M | 768
Table 10: Comparison of the performance of different variants of CLIP on
SugarCrepe++. Performance reported in terms of Accuracy (%). Overall best
values are in bold, and group-level best values are underlined.

Model | Swap Object | Swap Attribute | Replace Object | Replace Attribute | Replace Relation
---|---|---|---|---|---
| ITT | TOT | ITT | TOT | ITT | TOT | ITT | TOT | ITT | TOT
RN50 [53] | 46.05 | 17.98 | 49.01 | 31.35 | 87.41 | 82.45 | 67.39 | 57.36 | 56.12 | 39.04
RN101 [53] | 42.10 | 18.86 | 48.10 | 29.68 | 88.20 | 83.17 | 67.89 | 55.21 | 53.20 | 39.76
CLIP [53] | 45.18 | 19.74 | 45.21 | 33.03 | 86.80 | 83.72 | 65.61 | 59.14 | 56.26 | 38.62
RN50$\times$4 [53] | 46.93 | 21.49 | 46.42 | 30.59 | 87.77 | 80.87 | 67.51 | 53.93 | 53.91 | 38.55
RN50$\times$16 [53] | 39.04 | 17.55 | 46.42 | 30.29 | 89.10 | 76.57 | 65.74 | 49.87 | 53.41 | 38.19
CLIP-ViT-L/14 [53] | 43.86 | 17.11 | 45.36 | 28.92 | 90.68 | 80.69 | 67.39 | 55.96 | 54.05 | 39.26
RN50$\times$64 [53] | 44.74 | 16.67 | 45.36 | 31.51 | 90.79 | 73.31 | 64.47 | 48.61 | 54.27 | 38.12
RoBERTa-ViT-B/32 [62] | 44.3 | 29.39 | 56.32 | 52.66 | 89.04 | 94.55 | 74.49 | 80.46 | 59.39 | 57.75
ViT-H/14 [62] | 43.42 | 27.63 | 54.19 | 50.69 | 93.71 | 90.43 | 71.06 | 73.98 | 56.62 | 51.92
ViT-g/14 [62] | 44.3 | 26.32 | 52.06 | 43.99 | 93.1 | 91.83 | 71.19 | 73.73 | 57.54 | 52.56
ViT-bigG/14 [62] | 45.61 | 29.82 | 57.38 | 52.05 | 94.13 | 90.44 | 76.40 | 72.84 | 59.45 | 53.49
xlm-RoBERTa-base-ViT-B/32 [62] | 42.55 | 30.26 | 55.25 | 55.56 | 89.41 | 95.34 | 72.97 | 80.96 | 55.48 | 57.82
xlm-RoBERTa-large-ViT-H/14 [62] | 42.1 | 29.83 | 54.19 | 52.36 | 93.94 | 94.25 | 75.38 | 79.19 | 58.75 | 60.69
large:ViT-B/16 [19] | 35.96 | 17.98 | 39.58 | 30.44 | 87.83 | 90.5 | 67.89 | 70.81 | 50.36 | 39.04
xlarge:ViT-L/14 [19] | 42.55 | 26.75 | 46.58 | 39.12 | 91.59 | 91.89 | 72.59 | 71.45 | 55.83 | 49.79

##  Appendix D Additional Analysis of ULMs

Unimodal Language Models (ULMs) can be evaluated for their semantic and
lexical sensitivity using the Text-only task (TOT) in SugarCrepe++. We report
the TOT results of ULMs in Table 11. We cover a comprehensive list of ULMs,
varying in parameter counts, optimization objectives, training data sizes, and
architectures. We notice that ULMs consistently fail to disassociate semantics
from different lexical forms. This is indicated by the large deviation
(ranging from 18% to 35%) in TOT accuracy across different subsets. Generally,
all models perform over 90% accuracy in identifying the positive captions when
the negative caption changes the type of an ’object’ in the positive caption
(altering the semantics and lowering lexical overlap). On the contrary, if
multiple ’objects’ are swapped within the same caption (altering the semantics
but preserving the lexical overlap), the performance, on average across all
ULMs, decreases by $\approx$50%. This can be attributed to the lack of
compositional understanding required to associate semantics with different
lexical forms.

Grouping the models based on model sizes reveals that sensitivity to lexical
alteration does not necessarily improve with scale. For example, a small-sized
model like BAAI General Embedding (BGE) [79] is only 5% behind very large LLMs
like E5-mistral [75, 74] with 7 billion parameters in the ’Replace Relation’
subset. We experimented with recently proposed text encoder models like
INSTRUCTOR [67] that aim to improve the generalization of sentence embeddings
across various embedding-based tasks. These models allow the addition of
special prefix instructions before encoding text to condition for a task. We
compared the INSTRUCTOR model with a default prefix and a prefix with
instructions to encode the semantics (listed as custom-ins in Table 11). We
notice minimal performance gains with custom instruction, demonstrating that
further development is required to separate semantics from lexical
composition.

Table 11: Comprehensive results of ULMs on TOT of SugarCrepe++. We report the
TOT accuracy (%). and group the results row-wise based on the model size as
reflected by the parameter count. We include the number of parameters in text
encoders relative to BERT-base, i.e., 109.5 Million parameters. Overall, best
values are in bold, and group-level best values are underlined. We report the
average and standard deviation across different subsets as an additional
column.

Model |  | #Params
---
(BERT Scale)
| Swap
---
Object
| Swap
---
Attribute
| Replace
---
Object
| Replace
---
Attribute
| Replace
---
Relation
Average
All-MiniLM-L6-v2 [77] | 0.21 | 14.29 | 22.52 | 93.95 | 64.97 | 63.94 | 51.9333.02
BGE-small-en-v1.5 [79] | 0.3 | 15.51 | 24.02 | 94.19 | 75.00 | 75.53 |  56.8534.85
All-MiniLM-L12-v2 [77] | 0.3 | 18.78 | 25.38 | 95.22 | 73.86 | 70.41 | 56.7333.11
GTE-small [41] | 0.3 | 13.88 | 22.07 | 94.98 | 71.95 | 69.06 | 54.3934.84
Angle-BERT-base-uncased-nli-en-v1 [40] | 1 | 25.71 | 33.63 | 92.07 | 78.43 | 75.32 |  61.0329.45
BGE-base-en-v1.5 [79] | 1 | 17.14 | 25.23 | 93.83 | 78.55 | 76.10 | 58.1734.56
Sentence-T5-base [49] | 1.01 | 28.98 | 31.98 | 92.37 | 75.00 | 75.32 | 60.7328.51
GTE-base [41] | 1 | 17.14 | 22.82 | 94.31 | 75.00 | 71.98 | 56.2534.26
Instructor-large [67] | 3.07 | 26.53 | 28.38 | 96.00 | 72.34 | 73.83 | 59.4230.65
Instructor-large(custom-ins)[67] | 3.07 | 22.86 | 27.63 | 96.13 | 77.41 | 77.81 | 60.3732.99
UAE-Large-v1 [40] | 3.06 | 40.41 | 41.44 | 96.85 | 76.14 | 75.82 | 66.1324.54
GTE-large [41] | 3.06 | 26.53 | 27.93 | 96.31 | 76.78 | 72.83 | 60.0731.28
All-RoBERTa-large-v1 [57] | 3.25 | 42.04 | 45.20 | 94.61 | 74.75 | 74.96 | 66.3122.26
Stsb-RoBERTa-large [57] | 3.25 | 25.31 | 31.98 | 94.19 | 89.21 | 75.18 | 63.1732.37
Sentence-T5-xl [49] | 11.34 | 47.35 | 49.25 | 90.98 | 75.38 | 75.32 |  67.6618.8
Angle-Llama-7b-nli-v2 [40] | 62.28 | 37.96 | 45.80 | 95.22 | 84.39 | 81.44 | 68.9625.4
E5-Mistral-7b-instruct [75, 74] | 64.95 | 33.47 | 37.84 | 96.67 | 87.06 | 80.51 | 67.1129.33

##  Appendix E Human Evaluation Instruction

Figure 6 provides the instructions given to the human evaluators along with
the screenshot of the interface used for human evaluation.

Figure 6: Human evaluation instructions and a screenshot of the interface.

##  Appendix F Additional Analysis and Results

###  F.1 Prompt-based Evaluation

We evaluated the latest released GPT-4o ("o" for "omni") model on SugarCrepe++
in a prompt-based format. Given an image, we prompted the GPT-4o API
222https://platform.openai.com/docs/guides/vision using three different
prompts.

  * •

Prompt-1: "Do any of the following captions not match the image? (1) " +
<Negative Caption> \+ "; (2) " + <Positive Caption 1> \+ "; (3) " + <Positive
caption 2> \+ "; provide output as (1), (2), (3) or none"

  * •

Prompt-2: "Do any of these captions fail to correspond with the image? (1) " +
<Negative Caption> \+ "; (2) " + <Positive Caption 1> \+ "; (3) " + <Positive
caption 2> \+ "; provide output as (1), (2), (3) or none"

  * •

Prompt-3: "Do any of these captions fail to correspond with the image? ((1) "
+ <Negative Caption> \+ "; (2) " + <Positive Caption 1> \+ "; (3) " +
<Positive caption 2> \+ "; provide output as (1), (2) or (3)"

In the above prompts, <Negative caption>, <Positive caption 1> and <Positive
caption 2> refer to the negative (N) and the two positive captions (P1 and P2)
corresponding to the image, respectively. First two prompts (Prompt-1 and
Prompt-2) are paraphrases of each other and are 4-class problems (Output to be
(1), (2), (3) or none). Whereas Prompt-3 is a 3-class problem i.e., model need
to output (1), (2) or (3).

Table 12 provides the performance of GPT-4o on SugarCrepe++ for the three
different prompts. Differences in performance of GPT-4o are observed between
Prompt-1 and Prompt-2, which are paraphrases of one another. This shows the
sensitivity of the GPT-4o model to the structure of the prompt. When prompted
to choose output from four options ((1), (2), (3), or none), GPT-4o model has
difficulty identifying the negative caption (a caption that does not
correspond to the image). Conversely, GPT-4o better identified the negative
caption when prompted to choose from three options ((1), (2), or (3)). For all
the three prompts, GPT-4o was better at identifying the negative prompt when
we replaced an object or attribute in the positive caption to form the
negative caption. GPT-4o has difficulty in identifying the negative caption
when the relation between the objects is replaced or when the objects or
attributes are swapped to form the negative caption. For the replace object
and replace attribute categories, GPT-4o’s performance is close to human
performance. However, for swap object, swap attribute, and replace relation
cases, GPT-4o’s performance lags behind human performance by a large margin.

Table 12: Prompt-based evaluation of GPT-4o on SugarCrepe++. We provide both
image and a prompt to the GPT-4o and receive the output from GPT-4o. Based on
the response, we compute the performance i.e., it is a hit if the model
outputs (1) else a miss. Performance is reported in terms of Accuracy (%),
where accuracy is computed as the ratio of the #hits/(#hits + #misses).

Model | Swap Object | Swap Attribute | Replace Object | Replace Attribute | Replace Relation
---|---|---|---|---|---
Human | 100.00 | 96.67 | 100.00 | 100.00 | 100.00
Prompt-1 | 46.93 | 73.36 | 91.64 | 87.94 | 69.06
Prompt-2 | 48.25 | 75.04 | 90.82 | 84.90 | 71.19
Prompt-3 | 67.61 | 85.82 | 96.25 | 93.27 | 84.13

###  F.2 Qualitative results

We perform qualitative analysis to inspect examples from SugarCrepe++ dataset,
where the majority of models fail to dissociate semantics from lexical
composition. In this section, we highlight the results of our qualitative
analysis.

Figure 7: Examples from each subset of SugarCrepe++ dataset where the majority
of VLMs failed both the image-text task (ITT) and the text-only task (TOT).

####  F.2.1 Both Image-text task (ITT) and Text-only Task (TOT) fail

In this subsection, we provide examples where the majority of vision-language
models failed in both the image-text task and the text-only task. These
examples demonstrate cases where both the text encoder and vision encoder of
VLMs were ineffective in encoding semantics. Figure 7 shows such examples from
each subset of SugarCrepe++ dataset.

####  F.2.2 Image-text task (ITT) pass, and Text-only task (TOT) fail

In this subsection, we provide examples where the majority of vision-language
models (VLMs) failed in the text-only task (TOT) but were successful in the
image-text task (ITT), indicating that the vision encoders were able to
represent semantics more effectively than text encoders for these examples.
Figure 8 shows such examples from each subset of SugarCrepe++ dataset.

Figure 8: Examples from each subset of SugarCrepe++ dataset where the majority
of VLMs passed the image-text task (ITT) and failed the text-only task (TOT).

####  F.2.3 Text-only task (TOT) pass and Image-text task fail

In this subsection, we provide examples where the majority of vision-language
models (VLMs) failed in the image-text task (ITT) but succeeded in the text-
only task (TOT). These examples demonstrate cases where the text encoders of
VLMs were more effective at encoding semantics than their vision encoders.
Figure 9 presents examples from each subset of SugarCrepe++ dataset.

Figure 9: Examples from each subset of SugarCrepe++ dataset where the majority
of VLMs passed the text-only task (TOT) and failed the image-text task (ITT).

##  Appendix G Implementation Details

###  G.1 Hardware information

We performed all the experiments in this paper using a single 40G NVIDIA A100
GPU available in the Compute Canada Cluster.

###  G.2 Dataset sources

We obtain all existing datasets from their original sources released by the
authors. We refer readers to these sources for the dataset licenses.

  * •

COCO [43]: We obtain COCO images from its official project website
333https://cocodataset.org/. We use the images from the validation set
444http://images.cocodataset.org/zips/val2017.zip

  * •

SugarCrepe [25]: We obtain SugarCrepe captions and hard negatives from its
official website 555https://github.com/RAIVNLab/sugar-crepe.

###  G.3 Models

Evaluation of VLMs.  Source and links of the VLMs detailed in Table 8 (in the
Appendix of our paper) is provided below.

  * •

CLIP [53]: ’ViT-B/32’ variant of CLIP available at HuggingFace Link

  * •

RoBERTa-ViT-B-32 [62]: RoBERTa-ViT-B-32 trained on LAION dataset available at
HuggingFace Link

  * •

ALIGN [30]: Model available at HuggingFace Link

  * •

ALIP [81]: Model available at Google Drive Link

  * •

FLAVA [64]: Model available at HuggingFace Link

  * •

ALBEF [37]: ALBEF base model available in LAVIS

  * •

BLIP [38]: BLIP base model available in LAVIS

  * •

BLIP2 [39]: BLIP2 pretrained model available in LAVIS

  * •

ViLT [33]: Pre-trained ViLT model available at HuggingFace Link

  * •

SegCLIP [45]: Model available at Google Drive Link

  * •

XVLM-4M [83]: XVLM base model trained using 4 Million samples available at
Google Drive Link

  * •

XVLM-16M [83]: XVLM base model trained using 16 Million samples available at
Google Drive Link

  * •

ViLT-ITR-COCO [33]: ViLT model finetuned for image-text retrieval task using
MSCOCO dataset. This model is available at HuggingFace Link

  * •

XVLM-16M-COCO [83]: XVLM 16 Million model fine-tuned for image-text-retrieval
using MSCOCO dataset. This model is available at Google Drive Link

  * •

XVLM-16M-Flickr [83]: XVLM 16 Million model fine-tuned for image-text-
retrieval using Flickr dataset. This model is available at Google Drive Link

  * •

NegCLIP [82]: NegCLIP is trained on top of CLIP and the model link is
available in Github Link

  * •

CLIP-SVLC [18]: Model is available at Google Drive Link

  * •

BLIP-SGVL [22]: Model is available at Google Drive Link

  * •

CyCLIP [20]: Model is available at Google Drive Link

Variants of CLIP: All the CLIP models’ weights for the different variants of
CLIP reported in Table 10 in the appendix are obtained from OpenCLIP
666https://github.com/mlfoundations/open_clip framework [27].

Evaluated ULMs: The source and links of the ULMs detailed in Table 11 (in the
Appendix of our paper) are provided below.

  * •

All-MiniLM-L6-v2 [77]: HuggingFace Link

  * •

BGE-small-en-v1.5 [79]: HuggingFace Link

  * •

All-MiniLM-L12-v2 [77]: HuggingFace Link

  * •

GTE-small [41]: HuggingFace Link

  * •

Angle-BERT-base-uncased-nli-en-v1 [40]: HuggingFace Link

  * •

BGE-base-en-v1.5 [79]: HuggingFace Link

  * •

Sentence-T5-base [49]: HuggingFace Link

  * •

GTE-base [41]: HuggingFace Link

  * •

Instructor-large [67]: HuggingFace Link

  * •

Instructor-large (custom-ins)[67]: HuggingFace Link, we use ‘Represent the
sentence for spatial semantics’ as the custom instruction for Instructor-large
(custom-ins) model.

  * •

UAE-Large-V1 [40]: HuggingFace Link

  * •

GTE-large [41]: HuggingFace Link

  * •

All-RoBERTa-large-v1 [57]: HuggingFace Link

  * •

Stsb-RoBERTa-large [57]: HuggingFace Link

  * •

Sentence-T5-xl [49]: HuggingFace Link

  * •

Angle-Llama-7b-nli-v2 [40]: HuggingFace Link

###  G.4 Reproducibility

We release SugarCrepe++ dataset and the code to evaluate models on Github
777https://github.com/Sri-Harsha/scpp. The datasheet for SugarCrepe++ is
provided in the Supplementary material. The HuggingFace dataset Croissant
metadata is available here.

###  G.5 Author statement

In case of violation of rights, the authors will bear all responsibility. We
publicly release SugarCrepe++ dataset under the CC-BY-4.0 license.

###  G.6 License, Hosting and Maintenance Plan

We release the dataset publicly under the CC-BY-4.0 license on Github. The
authors of this paper are committed to supporting and maintaining the dataset
via our GitHub repository.

## Limitation and Future Work

In this paper, we evaluate a large set of VLMs on our proposed SugarCrepe++
dataset. We provide observations based on the results even though it is
difficult to provide a comparison between models as the models differ in terms
of model architecture, pre-training data content and size, pre-training
objectives, etc. Unless we can train separate models for each setting where
one of the parameters is kept constant, it is difficult to draw definitive
conclusions. Providing guidelines to improve the performance of VLMs on
SugarCrepe++ dataset depends on the observations we draw from the above-
mentioned analysis. Thus, in this paper, we limit to identifying one of the
major issues with current VLMs, which can probe further research in this
direction. Moreover, our dataset can be used to evaluate models to assess
their ability to discern lexical alterations from semantic alterations.

## Impact Statement

This paper presents work whose goal is to advance the field of Machine
Learning in general and Language Models research in particular. We discuss
several limitations of language models related to the separation of the
semantics of an input text from its syntactic and lexical form. In order to
build trust-worthy Language Models, it is important to establish that the
language models emphasize semantics contained in a sentence rather than the
lexical form and syntactic style of the sentence. Our evaluation provides
evidence of this problem through two curated datasets and can potentially be
impactful for evaluating newer language models and/or inspiring novel
solutions to this problem. There are many other potential societal
consequences of our work, none of which we feel must be specifically
highlighted here.

##  Appendix H Datasheet

  1. ###  H.1 Motivation

  2. Q1

For what purpose was the dataset created? Was there a specific task in mind?
Was there a specific gap that needed to be filled? Please provide a
description.

     * •

The SugarCrepe++ dataset was created to evaluate the sensitivity of vision
language models (VLMs) and unimodal language models (ULMs) to semantic and
lexical alterations. The SugarCrepe dataset consists of (only) one positive
and one hard negative caption for each image. Relative to the negative
caption, a single positive caption can either have low or high lexical
overlap. The original SugarCrepe only captures the high overlap case. To
evaluate the sensitivity of encoded semantics to lexical alteration, we
require an additional positive caption with a different lexical composition.
SugarCrepe++ fills this gap by adding an additional positive caption enabling
a more thorough assessment of models’ abilities to handle semantic content and
lexical variation.

  3. Q2

Who created the dataset (e.g., which team, research group) and on behalf of
which entity (e.g., company, institution, organization)?

     * •

The SugarCrepe++ dataset is created by the authors of this paper (affiliated
with the Faculty of Computer Science, Dalhousie University) to advance our
understanding of language models through a new evaluation dataset/task.

  4. Q3

Who funded the creation of the dataset? If there is an associated grant,
please provide the name of the grantor and the grant name and number.

     * •

We acknowledge the support provided by the Faculty of Computer Science,
Dalhousie University. Resources used in preparing this research were provided,
in part, by the support of the Natural Sciences and Engineering Research
Council of Canada (NSERC), the Province of Ontario, the Government of Canada
through Canadian Institute for Advanced Research (CIFAR), ACENET (ace-net.ca),
the Digital Research Alliance of Canada (alliancecan.ca) and companies
sponsoring the Vector Institute www.vectorinstitute.ai/#partners.

  5. Q4

Any other comments?

     * •

No.

###  H.2 Composition

  6. Q5

What do the instances that comprise the dataset represent (e.g., documents,
photos, people, countries)? Are there multiple types of instances (e.g.,
movies, users, and ratings; people and interactions between them; nodes and
edges)? Please provide a description.

     * •

The instances from SugarCrepe++ dataset represent images from MS-COCO [43] and
their associated text captions, negative captions from SugarCrepe and newly
introduced positive captions.

  7. Q6

How many instances are there in total (of each type, if appropriate)?

     * •

In total, SugarCrepe++ dataset consists of 4757 instances. The detailed
statistics of the subcategories are provided in https://github.com/Sri-
Harsha/scpp.

  8. Q7

Does the dataset contain all possible instances or is it a sample (not
necessarily random) of instances from a larger set? If the dataset is a
sample, then what is the larger set? Is the sample representative of the
larger set (e.g., geographic coverage)? If so, please describe how this
representativeness was validated/verified. If it is not representative of the
larger set, please describe why not (e.g., to cover a more diverse range of
instances, because instances were withheld or unavailable).

     * •

We included all possible instances from the SugarCrepe dataset, except those
which are not suitable for our tasks.

  9. Q8

What data does each instance consist of? “Raw” data (e.g., unprocessed text or
images) or features? In either case, please provide a description.

     * •

Each instance of SugarCrepe++ dataset consists of an image associated with
three captions, where two captions describe the image and one caption does
not.

  10. Q9

Is there a label or target associated with each instance? If so, please
provide a description.

     * •

Each instance in SugarCrepe++ consists of an image and a triplet of captions.
The label for a instance is whether each caption in the triplet correctly
corresponds to the image or not.

  11. Q10

Is any information missing from individual instances? If so, please provide a
description, explaining why this information is missing (e.g., because it was
unavailable). This does not include intentionally removed information, but
might include, e.g., redacted text.

     * •

No.

  12. Q11

Are relationships between individual instances made explicit (e.g., users’
movie ratings, social network links)? If so, please describe how these
relationships are made explicit.

     * •

To the best of our knowledge, there is no explicit relationship between the
individual instances.

  13. Q12

Are there recommended data splits (e.g., training, development/validation,
testing)? If so, please provide a description of these splits, explaining the
rationale behind them.

     * •

No, this is only an evaluation dataset.

  14. Q13

Are there any errors, sources of noise, or redundancies in the dataset? If so,
please provide a description.

     * •

No, to the best of our knowledge, there are no errors in SugarCrepe++ dataset.
We have done human validation, as described in detail in the paper, to
minimize any potential errors.

  15. Q14

Is the dataset self-contained, or does it link to or otherwise rely on
external resources (e.g., websites, tweets, other datasets)? If it links to or
relies on external resources, a) are there guarantees that they will exist,
and remain constant, over time; b) are there official archival versions of the
complete dataset (i.e., including the external resources as they existed at
the time the dataset was created); c) are there any restrictions (e.g.,
licenses, fees) associated with any of the external resources that might apply
to a future user? Please provide descriptions of all external resources and
any restrictions associated with them, as well as links or other access
points, as appropriate.

     * •

The images used in our dataset are based on the MS-COCO [43] dataset, which is
freely and publicly available. MS-COCO dataset is released under the Creative
Commons Attribution 4.0 license as listed in their website
https://cocodataset.org/#termsofuse.

  16. Q15

Does the dataset contain data that might be considered confidential (e.g.,
data that is protected by legal privilege or by doctor–patient
confidentiality, data that includes the content of individuals’ non-public
communications)? If so, please provide a description.

     * •

No, we source part of our dataset, such as image-caption pairs from MS-COCO
[43] and negative captions from SUGARCREPE [25], both of which are open-source
datasets.

  17. Q16

Does the dataset contain data that, if viewed directly, might be offensive,
insulting, threatening, or might otherwise cause anxiety? If so, please
describe why.

     * •

The authors did not create any content to be explicitly offensive. However,
there may be instances that some users may find offensive. Since our
SugarCrepe++ dataset depends on the MS-COCO [43] and SugarCrepe [25], we
encourage the reader to refer to these dataset’s documentation for further
details.

  18. Q17

Does the dataset relate to people? If not, you may skip the remaining
questions in this section.

     * •

No, the dataset does not relate to people and is not focused on people
(although people may appear in the images and descriptions).

  19. Q18

Does the dataset identify any subpopulations (e.g., by age, gender)?

     * •

We explicitly do not identify any sub-populations.

  20. Q19

Is it possible to identify individuals (i.e., one or more natural persons),
either directly or indirectly (i.e., in combination with other data) from the
dataset? If so, please describe how.

     * •

Some images might contain identifiable individual faces.

  21. Q20

Does the dataset contain data that might be considered sensitive in any way
(e.g., data that reveals racial or ethnic origins, sexual orientations,
religious beliefs, political opinions or union memberships, or locations;
financial or health data; biometric or genetic data; forms of government
identification, such as social security numbers; criminal history)? If so,
please provide a description.

     * •

We do not provide any such data in our dataset that may be considered
sensitive. All images in our datasets are taken from publicly available
datasets.

  22. Q21

Any other comments?

     * •

No

###  H.3 Collection Process

  23. Q22

How was the data associated with each instance acquired? Was the data directly
observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey
responses), or indirectly inferred/derived from other data (e.g., part-of-
speech tags, model-based guesses for age or language)? If data was reported by
subjects or indirectly inferred/derived from other data, was the data
validated/verified? If so, please describe how.

     * •

The data associated with each instance was acquired via our data generation
process (see Section 2 in our paper for a detailed description).

  24. Q23

What mechanisms or procedures were used to collect the data (e.g., hardware
apparatus or sensor, manual human curation, software program, software API)?
How were these mechanisms or procedures validated?

     * •

Please see Section 2 of our paper for a complete description of our data
generation and extensive validation process.

  25. Q24

If the dataset is a sample from a larger set, what was the sampling strategy
(e.g., deterministic, probabilistic with specific sampling probabilities)?

     * •

Not applicable.

  26. Q25

Who was involved in the data collection process (e.g., students, crowdworkers,
contractors) and how were they compensated (e.g., how much were crowdworkers
paid)?

     * •

The authors of this paper generated the textual content using generative AI as
explained in Section 2 of the paper, and manually validated it.

  27. Q26

Over what timeframe was the data collected? Does this timeframe match the
creation timeframe of the data associated with the instances (e.g., recent
crawl of old news articles)? If not, please describe the timeframe in which
the data associated with the instances was created.

     * •

The data was generated and evaluated over the course of approximately four
months.

  28. Q27

Were any ethical review processes conducted (e.g., by an institutional review
board)? If so, please provide a description of these review processes,
including the outcomes, as well as a link or other access point to any
supporting documentation.

     * •

We corresponded with the Research Ethics Board (REB) at Dalhousie University.
After describing our project in detail, the REB confirmed that our project did
not require ethics approval as it did not meet the regulatory definition of
human subjects research. Therefore, we did not need to submit a formal
application and were allowed to proceed with our research without additional
REB review.

  29. Q28

Does the dataset relate to people? If not, you may skip the remaining
questions in this section.

     * •

No, the dataset does not relate to people, and is not focused on people
(although people may appear in the images and descriptions).

  30. Q29

Did you collect the data from the individuals in question directly, or obtain
it via third parties or other sources (e.g., websites)?

     * •

Not applicable.

  31. Q30

Were the individuals in question notified about the data collection? If so,
please describe (or show with screenshots or other information) how notice was
provided, and provide a link or other access point to, or otherwise reproduce,
the exact language of the notification itself.

     * •

Not applicable.

  32. Q31

Did the individuals in question consent to the collection and use of their
data? If so, please describe (or show with screenshots or other information)
how consent was requested and provided, and provide a link or other access
point to, or otherwise reproduce, the exact language to which the individuals
consented.

     * •

Not applicable.

  33. Q32

If consent was obtained, were the consenting individuals provided with a
mechanism to revoke their consent in the future or for certain uses? If so,
please provide a description, as well as a link or other access point to the
mechanism (if appropriate).

     * •

Not applicable.

  34. Q33

Has an analysis of the potential impact of the dataset and its use on data
subjects (e.g., a data protection impact analysis) been conducted? If so,
please provide a description of this analysis, including the outcomes, as well
as a link or other access point to any supporting documentation.

     * •

Not applicable.

  35. Q34

Any other comments?

     * •

No.

###  H.4 Preprocessing, Cleaning, and/or Labeling

  36. Q35

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization
or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction,
removal of instances, processing of missing values)? If so, please provide a
description. If not, you may skip the remainder of the questions in this
section.

     * •

No preprocessing or labelling was done for creating the scenarios.

  37. Q36

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data
(e.g., to support unanticipated future uses)? If so, please provide a link or
other access point to the “raw” data.

     * •

N/A.

  38. Q37

Is the software used to preprocess/clean/label the instances available? If so,
please provide a link or other access point.

     * •

Not applicable

  39. Q38

Any other comments?

     * •

###  H.5 Uses

  40. Q39

Has the dataset been used for any tasks already? If so, please provide a
description.

     * •

No. SugarCrepe++ is a new benchmark.

  41. Q40

Is there a repository that links to any or all papers or systems that use the
dataset? If so, please provide a link or other access point.

     * •

To the best of our ability, we will try to maintain links to derivative papers
and systems that use our dataset in the SugarCrepe++ GitHub repository
(https://github.com/Sri-Harsha/scpp).

  42. Q41

What (other) tasks could the dataset be used for?

     * •

The primary use case of our benchmark is to evaluate the sensitivity of VLMs
and ULMs to semantic and lexical alterations. While we have not explored this
direction in the present work, future work can use this dataset to evaluate
any multi-modal system that uses VLMs and ULMs as foundation blocks, such as
text-to-image retrieval models, multi-modal chatbots, etc.

  43. Q42

Is there anything about the composition of the dataset or the way it was
collected and preprocessed/cleaned/labeled that might impact future uses? For
example, is there anything that a future user might need to know to avoid uses
that could result in unfair treatment of individuals or groups (e.g.,
stereotyping, quality of service issues) or other undesirable harms (e.g.,
financial harms, legal risks) If so, please provide a description. Is there
anything a future user could do to mitigate these undesirable harms?

     * •

Due to the reliance on the MS-COCO [43] and SugarCrepe [25] datasets,
SugarCrepe++ may contain offensive material, or biases present in these source
datasets. Users of SugarCrepe++ should carefully consider how these
limitations may impact their potential use case and exercise discretion in
their application of the dataset.

  44. Q43

Are there tasks for which the dataset should not be used? If so, please
provide a description.

     * •

The dataset should be avoided for a task if the limitations discussed above
are unacceptable or potentially problematic for the intended use case.

  45. Q44

Any other comments?

     * •

No.

###  H.6 Distribution and License

  46. Q45

Will the dataset be distributed to third parties outside of the entity (e.g.,
company, institution, organization) on behalf of which the dataset was
created? If so, please provide a description.

     * •

Yes, SugarCrepe++ dataset will be open-sourced and freely available.

  47. Q46

How will the dataset be distributed (e.g., tarball on website, API, GitHub)?
Does the dataset have a digital object identifier (DOI)?

     * •

Our datset and code will be made available at the following Github link:
https://github.com/Sri-Harsha/scpp

  48. Q47

When will the dataset be distributed?

     * •

The dataset will be made available publicly upon publication of this paper.

  49. Q48

Will the dataset be distributed under a copyright or other intellectual
property (IP) license, and/or under applicable terms of use (ToU)? If so,
please describe this license and/or ToU, and provide a link or other access
point to, or otherwise reproduce, any relevant licensing terms or ToU, as well
as any fees associated with these restrictions.

     * •

We release data under the CC-BY-4.0 license.

     * •

Our code will be released under the Apache-2.0 license

  50. Q49

Have any third parties imposed IP-based or other restrictions on the data
associated with the instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any relevant
licensing terms, as well as any fees associated with these restrictions.

     * •

The dataset will be released under CC-BY-4.0 license.

  51. Q50

Do any export controls or other regulatory restrictions apply to the dataset
or to individual instances? If so, please describe these restrictions, and
provide a link or other access point to, or otherwise reproduce, any
supporting documentation.

     * •

No.

  52. Q51

Any other comments?

     * •

No.

###  H.7 Maintenance

  53. Q52

Who will be supporting/hosting/maintaining the dataset?

     * •

The authors will be supporting, hosting and maintaining the dataset and code
through GitHub.

  54. Q53

How can the owner/curator/manager of the dataset be contacted (e.g., email
address)?

     * •

The authors can be contacted through their email. Alternatively, an issue can
be created on our GitHub repository.

  55. Q54

Is there an erratum? If so, please provide a link or other access point.

     * •

There is no erratum for our initial release. Errata will be documented as
future releases on the benchmark website.

  56. Q55

Will the dataset be updated (e.g., to correct labeling errors, add new
instances, delete instances)? If so, please describe how often, by whom, and
how updates will be communicated to users (e.g., mailing list, GitHub)?

     * •

SugarCrepe++ will be updated. Updates can be monitored through Github.

  57. Q56

If the dataset relates to people, are there applicable limits on the retention
of the data associated with the instances (e.g., were individuals in question
told that their data would be retained for a fixed period of time and then
deleted)? If so, please describe these limits and explain how they will be
enforced.

     * •

NA

  58. Q57

Will older versions of the dataset continue to be supported/hosted/maintained?
If so, please describe how. If not, please describe how its obsolescence will
be communicated to users.

     * •

We will host older versions in GitHub, in case we release newer versions.

  59. Q58

If others want to extend/augment/build on/contribute to the dataset, is there
a mechanism for them to do so? If so, please provide a description. Will these
contributions be validated/verified? If so, please describe how. If not, why
not? Is there a process for communicating/distributing these contributions to
other users? If so, please provide a description.

     * •

Users can extend and build on SugarCrepe++ dataset as we did for SugarCrepe.
We do not take responsibility for validating any extension of our work.

  60. Q59

Any other comments?

     * •

No.