# V-FLUTE: Visual Figurative Language Understanding with Textual Explanations

Arkadiy Saakyan1 Shreyas Kulkarni1 Tuhin Chakrabarty1 Smaranda Muresan1
1Columbia University
<EMAIL_ADDRESS>
###### Abstract

Large Vision-Language models (VLMs) have demonstrated strong reasoning
capabilities in tasks requiring a fine-grained understanding of literal images
and text, such as visual question-answering or visual entailment. However,
there has been little exploration of these models’ capabilities when presented
with images and captions containing figurative phenomena such as metaphors or
humor, the meaning of which is often implicit. To close this gap, we propose a
new task and a high-quality dataset: Visual Figurative Language Understanding
with Textual Explanations (V-FLUTE). We frame the visual figurative language
understanding problem as an explainable visual entailment task, where the
model has to predict whether the image (premise) entails a claim (hypothesis)
and justify the predicted label with a textual explanation. Using a human-AI
collaboration framework, we build a high-quality dataset, V-FLUTE, that
contains 6,027 $<$image, claim, label, explanation$>$ instances spanning five
diverse multimodal figurative phenomena: metaphors, similes, idioms, sarcasm,
and humor. The figurative phenomena can be present either in the image, the
caption, or both. We further conduct both automatic and human evaluations to
assess current VLMs’ capabilities in understanding figurative
phenomena.111Code and data will be available at github.com/asaakyan/V-FLUTE

##  1 Introduction

Figure 1: Explainable figurative visual entailment task: given an image
(premise) and a claim (hypothesis), output whether the image entails or
contradicts the claim along with a textual explanation.

Figurative language is integral to human communication, enabling a variety of
communicative goals Roberts and Kreuz (1994), including affective
communication Fussell and Moss (2014). Figurative language presents a
significant challenge to computational approaches as it requires understanding
of implicit meaning behind an expression Stowe et al. (2022); Shutova (2011);
Veale et al. (2016); Zhou et al. (2021). Recently, Chakrabarty et al. (2022)
proposed a task and dataset for Figurative Language Understanding through
Textual Explanations (FLUTE) that frames the problem as an explainable textual
entailment covering a variety of figurative language phenomena in text:
metaphors, similies, idioms, and sarcasm. This dataset has been used
successfully to advance and benchmark the capabilities of LLMs for
understanding figurative language in text Saakyan et al. (2022); Ziems et al.
(2024); Sravanthi et al. (2024); Dey et al. (2024).

However, figurative meaning is also prevalent in visual phenomena, such as
visual metaphors Akula et al. (2023); Chakrabarty et al. (2023), multimodal
sarcasm Desai et al. (2022), and humor Hessel et al. (2023); Hwang and Shwartz
(2023). Yet so far most of the work on vision and language models (VLMs) has
focused on understanding literal meaning in images and captions (e.g.,
ScienceQA Lu et al. (2022), MMMU Yue et al. (2024)) including work on
explainable visual entailment Kayser et al. (2021). Building on the idea of
FLUTE Chakrabarty et al. (2022) for text, we present a new dataset for visual
figurative language understanding with textual explanations (V-FLUTE). Our
dataset contains 6,027 $<$image, claim, label, explanation$>$ instances
spanning diverse figurative phenomena. Each instance contains an image
(premise) and a textual claim (hypothesis) that is either entailed or
contradicted by the image. Deciding the entailment relation requires the
vision-language model to understand the implicit meaning in both the visual
and textual modalities. Our dataset contains figurative phenomena present in
the image, in the caption, or in both. In addition, to mitigate the dependence
on spurious correlations, to more rigorously investigate reasoning
capabilities, and to promote explainability, our task requires the model to
generate a plausible explanation for the output label. See Figure 1 for two
examples from our dataset.

We make the following contribution towards assessing VLMs ability to
understand multimodal figurative phenomena:

  * •

V-FLUTE, a high-quality dataset of 6,027 $<$image, claim, label,
explanation$>$ instances built using a human-LLMs collaboration framework
covering several phenomena: metaphors, similies, idioms, sarcasm, and humor
(Section 3). We will make the dataset available.

  * •

A suite of evaluations to assess current VLMs’ capabilities on this new task
of explainable visual figurative entailment (Section 4.2 and 4.3).

  * •

A detailed human evaluation with error analysis yielding insights into types
of errors for different classes of models (Section 5).

##  2 Related Work

Phenomenon | Data Source | Figurative Part | Our Contribution | # instances
---|---|---|---|---
Metaphor/Simile |  | HAIVMet
---
Chakrabarty et al. (2023)
Image |  | Image Selection
---
Textual Explanations
Expert Verification
857 (450 E, 407 C)
| IRFL
---
Yosef et al. (2023)
Caption |  | Image Selection
---
Textual Explanations
Expert Verification
1,149 (574 E, 575 C)
Idiom |  | IRFL
---
Yosef et al. (2023)
Caption |  | Image Selection
---
Textual Explanations
Expert Verification
370 (186 E, 184 C)
Sarcasm |  | MuSE
---
Desai et al. (2022)
Caption |  | Claim Generation
---
Textual Explanations
Expert Verification
1,042 (521 E, 521 C)
Humor |  | MemeCap
---
Hwang and Shwartz (2023)
Image |  | Claim Generation
---
Textual Explanations
Expert Verification
1,958 (979 E, 979 C)
| NYCartoons
---
Hessel et al. (2023)
Image+Caption | Taken As Is |  651 (651 E)
Table 1: V-FLUTE dataset composition: 5 figurative phenomena, source datasets,
and our contributions. E denotes number of entailment instances, C -
contradiction.

Textual entailment MacCartney and Manning (2008); Bowman et al. (2015) and
visual entailment Xie et al. (2019) tasks have been proposed to measure
language and multimodal understanding. However, models trained to simply
improve label accuracy on these data can be brittle and suffer from spurious
correlations Poliak et al. (2018); Gururangan et al. (2018); McCoy et al.
(2019); Gardner et al. (2021). Datasets such as e-SNLI Camburu et al. (2018)
and e-SNLI-VE Kayser et al. (2021) augment existing entailment datasets with
natural language explanations and train models to not only predict the label,
but also generate a textual explanation for the reason behind the prediction.
Such approach has been further adopted for a variety of tasks, such as
commonsense reasoning Rajani et al. (2019); Aggarwal et al. (2021) and social
norm understanding CH-Wang et al. (2023) among others Wiegreffe and Marasovic
(2021). This approach has been extended to assess LLMs’ capabilities on
understanding figurative language through the FLUTE dataset Chakrabarty et al.
(2022). FLUTE frames figurative language understanding as an explainable
textual entailment task. Recent progress in multimodal models Li et al.
(2022); Alayrac et al. (2022); OpenAI (2023); Team et al. (2023); Liu et al.
(2023b); Anthropic (2024) prompts us to asses similar capabilities when
extended to multimodal setting, testing the understanding of non-literal
meaning contained in both images and text. We present an equivalent of the
FLUTE dataset for the visual modality: V-FLUTE.

A number of previous works has focused on modeling figurative phenomena beyond
text. Chakrabarty et al. (2023) use a human-AI collaboration framework to
generate visual metaphors from linguistic metaphors (HAIVMet dataset) and
propose a visual entailment task as an extrinsic evaluation of dataset
quality. The dataset contains images, claims, and labels, but no textual
explanations. Yosef et al. (2023) proposed a benchmark (IRFL) where given an
idiom, metaphor, or simile the model has to distinguish which of the four
associated images implies the figurative meaning of the expression. This
dataset focuses on the figurative meaning in the textual modality and does not
contain textual explanations. There has also been work on understanding
multimodal sarcasm with explanations Desai et al. (2022), mostly containing
noisy user-generated text and crowdworker-written explanations. Other line of
work has focused on understanding humor with multimodal models. MemeCap Hwang
and Shwartz (2023) is a dataset for understanding memes. Hessel et al. (2023)
release a corpus of annotated New Yorker Caption Contest entries, where the
goal is to come up with a humorous captions for an image, with high-quality
explanations for why the caption is humorous. The dataset is relatively
limited in size containing only 520 unique instances in its training set. We
leverage all these benchmarks to build V-FLUTE.

HAIVMet | IRFL | MuSE | MemeCap | NYCartoons
---|---|---|---|---
|  |  |  |
| The faculty meeting
---
was peaceful.
| Their relationship is
---
a house on fire.
| Oh I just #love
---
having to stare at
this while I #work.
| Even death won’t
---
exempt you from
going to work.
| Easy for you to
---
say, you’re cured!
Contradiction | Entailment | Contradiction | Entailment | Entailment
| The image shows
---
a faculty meeting
transformed into a
dramatic battlefield
scene with members
dressed as knights
…
The visual metaphor
suggests the faculty
meeting was like a
war, & not peaceful.
| The photo suggests
---
a conflict or an
intense emotional
situation …
which
aligns with the
symbolism of a
house on fire
representing a
relationship filled
with turmoil or
heated arguments
| The image shows
---
Disneyland
Resort sign,
& although it’s
within view, the
person would like
to experience it
in person rather
than just looking
at the sign during
work hours.
| The image shows
---
RoboCop …
This entails the
claim that even
death won’t exempt
you from going
to work because
it humorously
illustrates a
character who has
been reanimated
as a cyborg to
continue working
despite having died.
| A play on the word
---
"cured". People seek
therapy to have their
mental problems
remedied or cured.
But "cured" can also
refer to a meat prep
technique; here the
therapist is cured
bacon, & the patient
is an egg (that’s not
cured).The egg is
saying the therapist
doesn’t understand
his problems because
he’s "cured" in
both senses.
Table 2: Sample dataset instances form V-FLUTE corresponding to the source
datasets displaying images (premise), claims (hypothesis), labels, and
explanations [Row 1-5].

##  3 V-FLUTE Task and Dataset

Following prior work on figurative language understanding in text defined as
explainable textual entailment Chakrabarty et al. (2022), we define the
_visual figurative understanding_ as an _explainable visual entailment task_ :
given an image (premise) $p$ and a claim (hypothesis) $h$, output a textual
explanation $\hat{e}$ justifying whether the premise entails or contradicts
the hypothesis and assign a label
$\hat{y}\in\\{\texttt{Entailment},\texttt{Contradiction}\\}$. We focus on the
binary classification task since, for neutral labels, the explanations would
be trivial (simply describing the image).

To build V-FLUTE, we start with existing multimodal figurative datasets and
use human-AI collaboration frameworks with expert annotators Chakrabarty et
al. (2022); Wiegreffe et al. (2022); Liu et al. (2022) to transform them into
a high-quality, explainable visual entailment benchmark. These datasets cover
particular phenomena such as metaphors, similes, idioms, sarcasm or humor.
Each instance includes an image and a caption and the figurative phenomenon
can be either in the image, the caption or in both. We transform each data
into a unified $<$image, claim, label, explanation$>$ format for explainable
visual entailment. An overview of the dataset and our contributions can be
found in Table 1. See examples from each dataset in Table 2. Below, we
describe the construction of V-FLUTE for each figurative language type
(metaphors & similes, idioms, sarcasm and humor).

###  3.1 Metaphors and Similes

Metaphors and similes are powerful rhetorical devices that can be expressed
either in text or visually in an image. Visual metaphors are used as
persuasive devices in various fields such as advertising Forceville (2002);
Scott (1994). To create visual entailment instances containing metaphors and
similes in V-FLUTE, we rely on two existing resources: HAIVMet Chakrabarty et
al. (2023) and IRFL Yosef et al. (2023). Instances taken from HAIVMet contain
the metaphor/simile as a part of the premise (image), while those taken from
IRFL have the metaphor/simile as a part of the hypothesis (text).

####  3.1.1 HAIVMet as Data Source

Figure 2: Creation of V-FLUTE instances for methaphors and similies from
HAIVMet.

The HAIVMet Chakrabarty et al. (2023) data consists of 1,193 images of visual
metaphors spanning over 958 distinct linguistic metaphors. Each image is
associated with a claim that can be contradicting or entailing the image. In
addition, each image is associate with a visual elaboration that presents a
textual description of the image (See Figure 2). This visual elaboration was
used in the original paper to generate the visual metaphors (images).

Generating Textual Explanations. We augment the dataset with candidate textual
explanations. We prompt ChatGPT (gpt-3.5-0914) to generate an explanation for
every tuple $<$visual elaboration, claim, label$>$ (See Figure 2; and prompt
in Appendix D.1.1).

Expert Verification. Each claim is paired with up to $5$ images. However,
since these images were automatically generated with DALLE-2 using the visual
elaborations, not all are completely faithful. Moreover, some claims and
labels were inconsistent. Finally, automatically generated LLM candidate
explanations are not always correct and require refining. To tackle these
issues, we employ an expert verification process involving three expert
annotators with significant experience in figurative language and visual
metaphor understanding. Since each claim can be paired with more than one
visual metaphor, we ask annotators to select the visual metaphor most faithful
to the linguistic metaphor and visual elaboration (see Image Selection in
Figure 2) or select none in the rare case when none of the visual metaphors
are of good quality. As a part of the same annotation round, we also ask them
to verify and edit the explanation if necessary to ensure correctness and high
quality. Post strict quality control, we have 857 $<$image, claim, label,
explanation$>$ instances.

####  3.1.2 IRFL as Data Source

Figure 3: Creation of V-FLUTE instances for metaphors, similes, idioms from
IRFL.

The IRFL dataset Yosef et al. (2023) contains 1,440 figurative expressions,
each associated with $4$ distinct images. One of those images represents the
figurative expression (see Figure 3), and the other 3 act as distractors.

Image Selection. We automatically select images using CLIP Radford et al.
(2021). We select one of the distractor images that have the highest CLIPScore
(clip-vit-base-patch16) with the corresponding entailing image to create a
challenging, contradictory instance (see where an unrelated image of a house
is discarded when selecting the contradiction instance in Figure 3).

Generating Textual Explanations. We prompt GPT-4 (gpt-4-vision-preview) with
the ground truth label, claim, and the image to explain the relationship
between the image and the claim.

Expert Verification. We recruit the same three expert annotators from HAIVMET
annotations and ask them to verify the explanation is adequate and edit it
when necessary. We also ask the annotator to discard rare noisy instances
where the claim, image, and label do not fit. Post strict quality control, we
are left with 1149 $<$image, claim, label, explanation$>$ instances.

###  3.2 Idioms

The IRFL dataset contains idioms in addition to metaphors and similies. An
identical procedure to the one described in Section 3.1.2 was used for
generating V-FLUTE instances for idioms (370 $<$image, claim, label,
explanation$>$ examples).

###  3.3 Sarcasm

To create visual entailment instances containing sarcasm, we rely on the MuSE
data Desai et al. (2022). Similarly to IRFL, instances from MuSE data contain
sarcasm in the hypothesis (text).

####  3.3.1 MuSE as Data Source

Figure 4: Creation of V-FLUTE instances for sarcasm from MuSE.

The MuSE dataset Desai et al. (2022) consists of 3510 distinct images, the
respective sarcastic claims that act as contradiction instances (see example
in Figure 4), and crowd worker written explanations justifying the
contradiction.

Generating Entailment Claims. Since the dataset only contains sarcastic
instances, there are no claims with an entailment relationship. We generate
the entailing claims by prompting GPT-4 to generate a non-sarcastic version of
the claim while maintaining the user-generated informal style of the text (see
the generated entailment claim in Figure 4).

Generating Textual Explanations. While the dataset already contains
crowdworker-written explanations, upon inspection, they were often deemed poor
quality, lacking enough details, and formulaic (e.g., see the crowdworker
explanation in Figure 4). To improve their quality, we use the dataset’s
existing crowdworker explanations and prompt GPT-4 to rewrite and generate
high-quality candidate textual explanations given the claim and the label (see
the re-written explanation in Figure 4). See the prompt in Appendix D.3.

Expert Verification. Each image is now paired with a GPT-4-generated entailing
claim, an original contradicting claim, and their respective labels and
explanations. The same three expert annotators checked if the generated
explanations are adequate (i.e., complete, correct, and concise) and if not,
edit them. Experts were also instructed to discard noisy examples, e.g. when
the image does not contradict the sarcastic claim. Through strict quality
control, we obtain 1,042 $<$image, claim, label, explanation$>$ instances.

###  3.4 Humor

For multimodal humor, we rely on two datasets: MemeCap Hwang and Shwartz
(2023) and New Yorker cartoons Hessel et al. (2023).

####  3.4.1 MemeCap as Data Source

Figure 5: Creation of V-FLUTE instances for humor from MemeCap.

This dataset consists of memes along with their captions that describe the
meme poster’s intent (see example in Figure 5). Memes frequently contain
implicit, non-literal meaning Lestari (2019) and rely on visual metaphors
Piata (2016), posing a challenge to VLMs.

Claim Generation. Since meme captions are not suited for an entailment task,
we perform prompt GPT-4 with the caption to generate a claim from it (see
example in Figure 5). We filter these set of samples further with GPT-4 by
asking whether the image entails the claim and only selecting positive
instances. In addition to generating claims that entail the meme, we generate
counterclaims using GPT-4.

Generating Textual Explanations. We prompted GPT-4 with the ground truth label
in the prompt to explain the relationship between the image and the claim. See
prompts in Appendix D.4.

Expert Verification. We hire the same three expert annotators to ensure the
correctness of the data. Each annotator is tasked with verifying that 1) the
generated claim fits the image and 2) the explanation is correct and complete,
and if not, make the necessary changes. We also ask to discard samples with
inappropriate content. After careful quality control, we have 1958 $<$image,
claim, label, explanation$>$ instances.

####  3.4.2 NYCartoons as Data Source

The NYCartoons dataset Hessel et al. (2023) contains 651 high-quality
instances from the New Yorker Cartoon Caption Contest. Each instance consists
of a humorous image paired with a caption and a natural language explanation
justifying the implicit humor between the caption and the image. We simply use
the existing data where the caption is treated as a claim entailing the
humorous image paired with an explanation.

###  3.5 Dataset Statistics

We split our data into 4,578 training, 726 validation, and 723 testing
instances. Detailed counts per phenomenon and dataset, as well as other
statistics, are in Appendix A.

##  4 Experiments

We empirically study how several baseline models perform on the task of
explainable visual entailment. We investigate both off-the-shelf and fine-
tuned model performance.

###  4.1 Models

We select a variety of models for our study (see taxonomy in Appendix, Figure
10). For off-the-shelf models, we explore both open and API-based models. For
open models, we select the (current) state-of-the-art LLaVA-1.6 models Liu et
al. (2024). LLaVA is one of the simplest, yet one of the most high-performing
VLM architectures currently available. It utilizes a pretrained large language
model (e.g., Mistral-7B Jiang et al. (2023)) and a vision-language cross-modal
connector (e.g., an MLP layer) to align the vision encoder (e.g., CLIP Radford
et al. (2021)) outputs to the language models. We select LLaVA-1.6 models in
their 7B and 34B configurations (LLaVA-v1.6-7B and LLaVA-v1.6-34B
respectively) and refer to them as LLaVA-ZS-7B and LLaVA-ZS-34B. Both models
have been instruction-tuned on less than 1M visual instruction tuning samples
to act as general language and vision assistants. It should, however, be noted
that these models do not currently support few-shot multimodal prompting.

In addition to zero-shot testing, we also test these models using
Compositional Chain-of-Thought Prompting proposed by Mitra et al. (2023). The
method first prompts the model to generate a scene graph and then utilizes
that scene graph in another prompt to answer the relevant question. The method
works zero-shot without requiring fine-tuning. We refer to these models as
LLaVA-ZS-7B-SG and LLaVA-ZS-34B-SG for the 7B and 34B LLaVA configurations
described above.

For API-based models, we select three widely available state-of-the-art VLMs:
Claude-3 Opus (claude-3-opus-20240229)Anthropic (2024), GPT-4
(gpt-4-1106-vision-preview) OpenAI (2023) and GeminiPro (gemini-pro-
vision)Team et al. (2023). We refer to GPT-4 as the “teacher” model as most
candidate explanations were generated with it.

For fine-tuned models, we focus on fine-tuning LLaVA-1.5-7B model Liu et al.
(2023a) (the fine-tuning code for 1.6 model is not available during the time
the paper was written). To minimize bias for a single instruction, we fine-
tune and evaluate the models on a set of 21 instruction paraphrases (see
Appendix Table 8). Three model configurations are tested:

  * •

LLaVA-eViL is a checkpoint of LLaVA-v1.5-7B further fine-tuned on the eViL
(e-SNLI-VE) dataset for explainable visual entailment Kayser et al. (2021)
converted to the instruction format. We removed neutral label instances, which
resulted in 275,815 training instances and 10,897 validation instances.

  * •

LLaVA-VF is the same checkpoint fine-tuned on the training set of V-FLUTE. We
also fine-tune the model with a white square instead of the V-FLUTE image
(denoted by $-$Image).

  * •

LLaVA-eViL+VF is the same checkpoint fine-tuned on both eViL and V-FLUTE.

All hyperparameters are in Appendix C.

###  4.2 Automatic Metrics

Similarly to prior work Chakrabarty et al. (2022) we utilize both classic F1
score and an adjusted score that accounts for explanation quality:
F1@ExplanationScore. The ExplanationScore computes the average between
BERTScore Zhang* et al. (2020) based on the microsoft-deberta-xlarge-mnli
model He et al. (2021); Williams et al. (2018) and BLEURT Sellam et al. (2020)
based on the BLEURT-20 Pu et al. (2021). Since our goal is to ensure models
provide an answer for the right reasons, ideally, we would only count
predictions as correct when the explanation is also correct. Hence, we report
F1@0 (simply F1 score), F1@53 (only predictions with explanation score $>$ 53
are considered correct), and F1@60. Thresholds are selected based on human
evaluation of explanation quality in Section 5.3.

###  4.3 Automatic Evaluation Results

Table 3 shows the results based on the automatic evaluation. We also include
results per phenomenon in Appendix F and the drop in performance when
accounting for explanations score in Figure 6. Our results inform the
following insights:

Model Name | F1@0 | F1@53 | F1@60
---|---|---|---
Random Baseline | 49.82 | - | -
Fine-tuned
LLaVA-7B |  |  |
$\dashrightarrow$ VF | 72.78 | 60.66 | 47.12
$\dashrightarrow$ $-$ Image | 64.77 | 53.28 | 39.37
$\dashrightarrow$ eViL | 54.34 | 4.11 | 0.55
$\dashrightarrow$ $+$ VF | 74.91 | 62.34 | 48.80
Off-the-shelf
Open
LLaVA-ZS |  |  |
$\dashrightarrow$ 7B | 45.44 | 35.57 | 18.38
$\dashrightarrow$ $+$ SG | 52.94 | 39.27 | 14.86
$\dashrightarrow$ 34B | 55.60 | 48.32 | 31.83
$\dashrightarrow$ $+$ SG | 58.08 | 45.74 | 26.77
API-based
$\dashrightarrow$ Gemini | 53.70 | 39.72 | 19.01
$\dashrightarrow$ 5-shot | 67.25 | 56.04 | 37.14
$\dashrightarrow$ Claude | 56.07 | 45.37 | 22.31
$\dashrightarrow$ 5-shot | 67.79 | 58.70 | 35.32
$\dashrightarrow$ GPT-4 | 64.00 | 56.22 | 38.56
$\dashrightarrow$ 5-shot | 69.36 | 61.95 | 49.81
Table 3: F1 Score results for different models across thresholds 0.0, 0.53,
and 0.6 for explanation score. Best result overall is in bold, best result in
each category is underlined.

##### Fine-tuning on V-FLUTE leads to best classification performance on
average across datasets.

Our strongest fine-tuned model (LLaVA-7B-eViL+VF) outperforms the best off-
the-shelf model (GPT-4-5shot) in terms of the F1@0 score ($p<0.03$; all $p$
values reported via paired bootstrap test Koehn (2004)), and performs
competitively when incorporating the explanations quality with GPT-4 leading
slightly (F1@60 of 49.81 vs. 48.80 for the best fine-tuned model), which is
expected as GPT-4 is the teacher model with which the majority of the
explanation candidates were generated. Adding the e-ViL dataset improves the
performance slightly compared to only fine-tuning on V-FLUTE. Fine-tuning
merely on e-ViL improves over a random baseline; however, the explanations are
of poor quality.

We also utilize a hypothesis-only baseline Poliak et al. (2018) by including a
model fine-tuned on the V-FLUTE dataset, but without the relevant image (with
a white square as an input instead, denoted as $-$Image). Fine-tuning on the
full V-FLUTE dataset shows an improvement of over 8 points in F1@0 (better
with $p<0.002$), suggesting VLMs benefit from visual information when dealing
with figurative phenomena and do not just rely on the input text to make their
prediction.

##### Open zero-shot instruction-tuned models are lagging behind API-based
models, but scene graph prompting improves performance.

LLaVA-7B and 34B lag behind Claude 3 and GPT-4 in zero-shot settings. However,
scene graph prompting improves the zero-shot performance of the LLaVA-based
models, allowing them catch up to zero-shot API model performance (Gemini and
Claude 3). The explanations generated by these models tend to overly focus on
the contents of the scene graph rather than the underlying figurative
phenomena, possibly causing a decrease in explanation score (and consequently
in F1@60). The few-shot API models outperform zero-shot API models, and are
better than all configurations of open models in F1@0, 53, 60, indicating the
effectiveness of few-shot prompting (not available for LLaVA-based models as
of now).

##### Performance for models decreases when taking into account explanation
quality.

We plot the relative percentage decrease between F1@0 and F1@60 for LLaVA-
eViL-VF, LLaVA-34B-SG, and GPT-4-5shot in Figure 6. Higher relative drop
indicates higher difficulty of generating the correct explanation. For all
models, we see a substantial decrease in performance, especially on
challenging phenomena such as Humor (NYCartoons). For Metaphor (IRFL), Humor
(MemeCap) and Idiom (IRFL) subsets GPT-4 exhibits the lowest relative
performance drop, while for Metaphor (HAIVMet), Humor (NYCartoons) and Sarcasm
(MuSE) the fine-tuned model has the lowest drop.

We can see that the percentage drop is substantially higher for all models for
the HAIVMet subset rather than the IRFL dataset, which contains metaphors in
the image rather than in the text. This suggests it is harder for models to
generate correct explanations when the figurative meaning is contained in the
image rather than in the text, indicating the need to expand current datasets
to include images with figurative meaning.

Figure 6: % Drop in F1 score for various models by source dataset between 0 to
0.6. Higher drop indicates higher proportion of wrongly generated
explanations.

###  4.4 Human Baseline

To find out how humans perform on the task, we hire two expert annotators with
formal education in linguistics. We present them with 10 example instances and
then ask them to complete 99 randomly sampled test set instances. We also
evaluate our best model (see Table 3) on the same set. Results are shown in
Table 4. Human performance is quite strong, almost reaching 90 F1@0 score
overall. Human performance is better than our strongest fine-tuned model
(LLaVA-7B-eVil+VF) performance with $p<0.05$ for Annotator 1 and $p<0.07$ for
Annotator 2. Humans excel at interpreting memes, with both annotators reaching
a 100% F1 score. Humans also perform noticeably better on the NYCartoons
dataset and on the idiom subset of the task. The model has a slight edge in
performance on the sarcasm and visual metaphor subsets of the task, perhaps
due to difficulty of these subsets and any potential spurious correlations
during fine-tuning.

Phenomenon | Dataset |  | Human
---
Avg
| LLaVA-
---
eViL+VF
Metaphor /Similes | HAIVMET | 78.84 | 81.25
| IRFL
---
(metaphor
/simile)
94.36 | 77.78
Idioms |  | IRFL
---
(idiom)
89.26 | 49.74
Sarcasm | MuSE | 68.89 | 85.42
Humor | MemeCap | 100.0 | 78.03
NYCartoons | 71.43 | 47.83
Overall | 89.09 | 77.26
Table 4: Human baseline results (F1@0) by phenomenon and source dataset.

##  5 Human Evaluation and Error Analysis

We conduct human evaluation of generated explanation to more reliably assess
their quality and identify key errors in multimodal figurative language
understanding. We recruit two expert annotators with background in linguistics
for the task and sample 95 random instances from the test set. For each
instance, we first provide the annotators with the image, claim and reference
explanation and ask the annotators to choose the right label. If the annotator
succeeds, they can view the rest of the task, which consists of 3 explanations
from our top models by F1@0 in each category: LLaVA-eViL-VF, LLaVA-34B-SG,
GPT-4-5shot. The explanations are taken for both correct and incorrect model
predictions. For each explanation, we ask whether the explanation is adequate
(accurate, correct, complete and concise). If not, we ask them to identify one
of the three main types of errors based on the following taxonomy:

  * •

Hallucination: explanation is not faithful to the image, indicating
difficulties with basic visual comprehension (see prediction of a blunt tip
when the pencil tip is actually sharp in row 1 of Table 5).

  * •

Unsound reasoning: sentences do not adhere to natural logic or violate common
sense (e.g., concluding than an upwards arrow and lots of money imply an
economic crisis, see row 3).

  * •

Incomplete reasoning: while overall the explanation makes sense, it does not
address the key property reasons why the image entails or contradicts the
claim (for example, does not address the figurative part in the image, see row
2).

  * •

Too Verbose: the explanation is too verbose to the point it would interfere
rather than help one decide the correct label.

Error Type Image Claim Label and Explanation Hallucination (describes sharp
pencil as having a blunt tip) The tip is as sharp as a knife. Predicted Label:
Contradiction
Predicted Explanation: The image depicts a pencil with a blunt tip. […]
Incomplete (does not address metaphorical meaning of iceberg imagery) They
looked at each other in an almost hostile way. Predicted Label: Entailment
Predicted Explanation: The image depicts two characters looking at each other
with a tense and confrontational expression, which is consistent with the
claim […] Unsound
(concludes that large amount of money + upward trending chart implies rough
waters) The world recession of 2001 was one of the worst in history. Predicted
Label: Entailment
Predicted Explanation: The image depicts a man in a boat, holding a large
amount of money and a chart with an arrow pointing upwards. This suggests that
the man is navigating through rough waters […]

Table 5: Examples of error types generated explanations.

###  5.1 How Do Models Perform According to Humans?

|  | LLaVA-7B
---
eViL+VF
| LLaVA-34B
---
SG
| GPT-4
---
(5 shot)
Adequate % | 33.78 | 29.85 | 50.67
Preference % | 23.08 | 7.69 | 44.23
Table 6: Adequacy and Preference rates for generated explanations.

In Table 6, we show adequacy and preference rates for explanations from the 3
systems, where an explanation is deemed adequate if both annotators agreed it
is, and inadequate if both agreed it is not. The preference percentage is also
taken among instances where the annotators agreed that the model’s explanation
is preferred among all the adequate explanations. The average IAA using
Cohen’s $\kappa$ is 0.47, indicating moderate agreement Cohen (1960). We
observe that the teacher model is leading in terms of the adequacy of the
explanations and preference rate, as expected from a larger system equipped
for higher quality reasoning and generation capabilities. Yet still only half
of its explanations are considered adequate. This further confirms that
despite impressive performance on the F1@0 scores, the models are not yet
capable of producing adequate textual explanations in many instances.

###  5.2 What Errors Do Models Make?

We also analyze to understand what type of errors do each model make when they
are considered not adequate in the above evaluation. In Figure 7, we
illustrate the normalized frequency of error types when both annotators agree
that the explanation is not adequate (i.e., out of all errors for this model,
what percentage is each type of error?). In general, annotators did not
consider verbosity to be a major issue of the systems. For GPT-4, the leading
error type is hallucination, indicating the need to improve faithful image
recognition even in the most advanced models. For the fine-tuned model and
LLaVA-34B-SG, the main error type is unsound reasoning, indicating that it is
challenging for the models to reason about multimodal figurative inputs
consistently.

Figure 7: Normalized frequency of main error types in the explanation by
model.

###  5.3 How Well Does the Explanation Score Predict Human Judgment on
Adequacy?

We explore whether the proposed explanation score can capture human judgement
of explanation adequacy. We collect all instances where both annotators agreed
on the adequacy judgement for the explanation. We evaluate if the explanation
score described in Section 4.2 can act as a good predictor for the human
adequacy judgment. We find that the area under the Precision-Recall curve is
0.79, and the maximum F1 score is 0.77, obtainable at the explanation score
threshold of 0.53. Hence, we use this threshold to report the results in Table
3. We also use the threshold of 0.6 since it maximizes F1 such that both
precision and recall are above 0.75.

##  6 Conclusions

We introduce a high-quality dataset for understanding figurative phenomena in
multimodal input, V-FLUTE, framed as an explainable visual entailment. Our
dataset consists of 6,027 $<$image, claim, label, explanation$>$ instances
spanning a variety of figurative phenomena such as metaphor, idiom, simile,
sarcasm, and humor. We use this dataset to benchmark the performance of state-
of-the-art vision-language models using both automatic and human evaluation
and to identify critical areas of improvement for VLMs for this task.

## References

  * Aggarwal et al. (2021) Shourya Aggarwal, Divyanshu Mandowara, Vishwajeet Agrawal, Dinesh Khandelwal, Parag Singla, and Dinesh Garg. 2021.  Explanations for CommonsenseQA: New Dataset and Models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3050–3065, Online. Association for Computational Linguistics.
  * Akula et al. (2023) Arjun R Akula, Brendan Driscoll, Pradyumna Narayana, Soravit Changpinyo, Zhiwei Jia, Suyash Damle, Garima Pruthi, Sugato Basu, Leonidas Guibas, William T Freeman, et al. 2023.  Metaclue: Towards comprehensive visual metaphors research.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 23201–23211.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022.  Flamingo: a visual language model for few-shot learning.  _Advances in Neural Information Processing Systems_ , 35:23716–23736.
  * Anthropic (2024) Anthropic. 2024.  The claude 3 model family: Opus, sonnet, haiku.  https://www.anthropic.com/news/claude-3-family.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018.  e-snli: Natural language inference with natural language explanations.  In _Advances in Neural Information Processing Systems_ , volume 31. Curran Associates, Inc.
  * CH-Wang et al. (2023) Sky CH-Wang, Arkadiy Saakyan, Oliver Li, Zhou Yu, and Smaranda Muresan. 2023.  Sociocultural norm similarities and differences via situational alignment and explainable textual entailment.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 3548–3564, Singapore. Association for Computational Linguistics.
  * Chakrabarty et al. (2022) Tuhin Chakrabarty, Arkadiy Saakyan, Debanjan Ghosh, and Smaranda Muresan. 2022.  FLUTE: Figurative language understanding through textual explanations.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 7139–7159, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Chakrabarty et al. (2023) Tuhin Chakrabarty, Arkadiy Saakyan, Olivia Winn, Artemis Panagopoulou, Yue Yang, Marianna Apidianaki, and Smaranda Muresan. 2023.  I spy a metaphor: Large language models and diffusion models co-create visual metaphors.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 7370–7388, Toronto, Canada. Association for Computational Linguistics.
  * Cohen (1960) Jacob Cohen. 1960.  A coefficient of agreement for nominal scales.  _Educational and psychological measurement_ , 20(1):37–46.
  * Desai et al. (2022) Poorav Desai, Tanmoy Chakraborty, and Md Shad Akhtar. 2022.  Nice perfume. how long did you marinate in it? multimodal sarcasm explanation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 36(10):10563–10571.
  * Dey et al. (2024) Gourab Dey, Adithya V Ganesan, Yash Kumar Lal, Manal Shah, Shreyashee Sinha, Matthew Matero, Salvatore Giorgi, Vivek Kulkarni, and H Andrew Schwartz. 2024.  Socialite-llama: An instruction-tuned model for social scientific tasks.  _arXiv preprint arXiv:2402.01980_.
  * Forceville (2002) Charles Forceville. 2002.  _Pictorial metaphor in advertising_.  Routledge.
  * Fussell and Moss (2014) Susan R Fussell and Mallie M Moss. 2014.  Figurative language in emotional communication.  In _Social and cognitive approaches to interpersonal communication_ , pages 113–141. Psychology Press.
  * Gardner et al. (2021) Matt Gardner, William Merrill, Jesse Dodge, Matthew Peters, Alexis Ross, Sameer Singh, and Noah A. Smith. 2021.  Competency problems: On finding and removing artifacts in language data.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1801–1813, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021.  {DEBERTA}: {DECODING}-{enhanced} {bert} {with} {disentangled} {attention}.  In _International Conference on Learning Representations_.
  * Hessel et al. (2023) Jack Hessel, Ana Marasovic, Jena D. Hwang, Lillian Lee, Jeff Da, Rowan Zellers, Robert Mankoff, and Yejin Choi. 2023.  Do androids laugh at electric sheep? humor “understanding” benchmarks from the new yorker caption contest.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 688–714, Toronto, Canada. Association for Computational Linguistics.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Hwang and Shwartz (2023) EunJeong Hwang and Vered Shwartz. 2023.  MemeCap: A dataset for captioning and interpreting memes.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 1433–1445, Singapore. Association for Computational Linguistics.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.
  * Kayser et al. (2021) Maxime Kayser, Oana-Maria Camburu, Leonard Salewski, Cornelius Emde, Virginie Do, Zeynep Akata, and Thomas Lukasiewicz. 2021.  e-vil: A dataset and benchmark for natural language explanations in vision-language tasks.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 1244–1254.
  * Koehn (2004) Philipp Koehn. 2004.  Statistical significance tests for machine translation evaluation.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_ , pages 388–395, Barcelona, Spain. Association for Computational Linguistics.
  * Lestari (2019) Widia Lestari. 2019.  Irony analysis of memes on instagram social media.  _Pioneer: Journal of Language and Literature_ , 10(2):114–123.
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022.  Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation.  In _International Conference on Machine Learning_ , pages 12888–12900. PMLR.
  * Liu et al. (2022) Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. 2022.  WANLI: Worker and AI collaboration for natural language inference dataset creation.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 6826–6847, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Liu et al. (2023a) Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2023a.  Improved baselines with visual instruction tuning.
  * Liu et al. (2024) Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. 2024.  Llava-next: Improved reasoning, ocr, and world knowledge.
  * Liu et al. (2023b) Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023b.  Visual instruction tuning.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 34892–34916. Curran Associates, Inc.
  * Lu et al. (2022) Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022.  Learn to explain: Multimodal reasoning via thought chains for science question answering.  In _The 36th Conference on Neural Information Processing Systems (NeurIPS)_.
  * MacCartney and Manning (2008) Bill MacCartney and Christopher D. Manning. 2008.  Modeling semantic containment and exclusion in natural language inference.  In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_ , pages 521–528, Manchester, UK. Coling 2008 Organizing Committee.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Mitra et al. (2023) Chancharik Mitra, Brandon Huang, Trevor Darrell, and Roei Herzig. 2023.  Compositional chain-of-thought prompting for large multimodal models.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4v(ision) system card.  https://cdn.openai.com/papers/GPTV_System_Card.pdf.
  * Piata (2016) Anna Piata. 2016.  When metaphor becomes a joke: Metaphor journeys from political ads to internet memes.  _Journal of Pragmatics_ , 106:39–56.
  * Poliak et al. (2018) Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018.  Hypothesis only baselines in natural language inference.  In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_ , pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics.
  * Pu et al. (2021) Amy Pu, Hyung Won Chung, Ankur P Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021.  Learning compact metrics for mt.  In _Proceedings of EMNLP_.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR.
  * Rajani et al. (2019) Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, and Richard Socher. 2019.  Explain yourself! leveraging language models for commonsense reasoning.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4932–4942, Florence, Italy. Association for Computational Linguistics.
  * Roberts and Kreuz (1994) Richard M. Roberts and Roger J. Kreuz. 1994.  Why do people use figurative language? _Psychological Science_ , 5(3):159–163.
  * Saakyan et al. (2022) Arkadiy Saakyan, Tuhin Chakrabarty, Debanjan Ghosh, and Smaranda Muresan. 2022.  A report on the figlang 2022 shared task on understanding figurative language.  In _Proceedings of the 3rd Workshop on Figurative Language Processing (FLP)_ , pages 178–183.
  * Scott (1994) Linda M Scott. 1994.  Images in advertising: The need for a theory of visual rhetoric.  _Journal of consumer research_ , 21(2):252–273.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Shutova (2011) Ekaterina V Shutova. 2011.  Computational approaches to figurative language.  Technical report, University of Cambridge, Computer Laboratory.
  * Sravanthi et al. (2024) Settaluri Lakshmi Sravanthi, Meet Doshi, Tankala Pavan Kalyan, Rudra Murthy, Pushpak Bhattacharyya, and Raj Dabre. 2024.  Pub: A pragmatics understanding benchmark for assessing llms’ pragmatics capabilities.  _arXiv preprint arXiv:2401.07078_.
  * Stowe et al. (2022) Kevin Stowe, Prasetya Utama, and Iryna Gurevych. 2022.  IMPLI: Investigating NLI models’ performance on figurative language.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5375–5388, Dublin, Ireland. Association for Computational Linguistics.
  * Team et al. (2023) Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul R. Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, Iain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, Alexandre Frechette, Charlotte Smith, Laura Culp, Lev Proleev, Yi Luan, Xi Chen, James Lottes, Nathan Schucher, Federico Lebron, Alban Rrustemi, Natalie Clay, Phil Crone, Tomas Kocisky, Jeffrey Zhao, Bartek Perz, Dian Yu, Heidi Howard, Adam Bloniarz, Jack W. Rae, Han Lu, Laurent Sifre, Marcello Maggioni, Fred Alcober, Dan Garrette, Megan Barnes, Shantanu Thakoor, Jacob Austin, Gabriel Barth-Maron, William Wong, Rishabh Joshi, Rahma Chaabouni, Deeni Fatiha, Arun Ahuja, Ruibo Liu, Yunxuan Li, Sarah Cogan, Jeremy Chen, Chao Jia, Chenjie Gu, Qiao Zhang, Jordan Grimstad, Ale Jakse Hartman, Martin Chadwick, Gaurav Singh Tomar, Xavier Garcia, Evan Senter, Emanuel Taropa, Thanumalayan Sankaranarayana Pillai, Jacob Devlin, Michael Laskin, Diego de Las Casas, Dasha Valter, Connie Tao, Lorenzo Blanco, Adrià Puigdomènech Badia, David Reitter, Mianna Chen, Jenny Brennan, Clara Rivera, Sergey Brin, Shariq Iqbal, Gabriela Surita, Jane Labanowski, Abhi Rao, Stephanie Winkler, Emilio Parisotto, Yiming Gu, Kate Olszewska, Yujing Zhang, Ravi Addanki, Antoine Miech, Annie Louis, Laurent El Shafey, Denis Teplyashin, Geoff Brown, Elliot Catt, Nithya Attaluri, Jan Balaguer, Jackie Xiang, Pidong Wang, Zoe Ashwood, Anton Briukhov, Albert Webson, Sanjay Ganapathy, Smit Sanghavi, Ajay Kannan, Ming-Wei Chang, Axel Stjerngren, Josip Djolonga, Yuting Sun, Ankur Bapna, Matthew Aitchison, Pedram Pejman, Henryk Michalewski, Tianhe Yu, Cindy Wang, Juliette Love, Junwhan Ahn, Dawn Bloxwich, Kehang Han, Peter Humphreys, Thibault Sellam, James Bradbury, Varun Godbole, Sina Samangooei, Bogdan Damoc, Alex Kaskasoli, Sébastien M. R. Arnold, Vijay Vasudevan, Shubham Agrawal, Jason Riesa, Dmitry Lepikhin, Richard Tanburn, Srivatsan Srinivasan, Hyeontaek Lim, Sarah Hodkinson, Pranav Shyam, Johan Ferret, Steven Hand, Ankush Garg, Tom Le Paine, Jian Li, Yujia Li, Minh Giang, Alexander Neitz, Zaheer Abbas, Sarah York, Machel Reid, Elizabeth Cole, Aakanksha Chowdhery, Dipanjan Das, Dominika Rogozińska, Vitaly Nikolaev, Pablo Sprechmann, Zachary Nado, Lukas Zilka, Flavien Prost, Luheng He, Marianne Monteiro, Gaurav Mishra, Chris Welty, Josh Newlan, Dawei Jia, Miltiadis Allamanis, Clara Huiyi Hu, Raoul de Liedekerke, Justin Gilmer, Carl Saroufim, Shruti Rijhwani, Shaobo Hou, Disha Shrivastava, Anirudh Baddepudi, Alex Goldin, Adnan Ozturel, Albin Cassirer, Yunhan Xu, Daniel Sohn, Devendra Sachan, Reinald Kim Amplayo, Craig Swanson, Dessie Petrova, Shashi Narayan, Arthur Guez, Siddhartha Brahma, Jessica Landon, Miteyan Patel, Ruizhe Zhao, Kevin Villela, Luyu Wang, Wenhao Jia, Matthew Rahtz, Mai Giménez, Legg Yeung, Hanzhao Lin, James Keeling, Petko Georgiev, Diana Mincu, Boxi Wu, Salem Haykal, Rachel Saputro, Kiran Vodrahalli, James Qin, Zeynep Cankara, Abhanshu Sharma, Nick Fernando, Will Hawkins, Behnam Neyshabur, Solomon Kim, Adrian Hutter, Priyanka Agrawal, Alex Castro-Ros, George van den Driessche, Tao Wang, Fan Yang, Shuo yiin Chang, Paul Komarek, Ross McIlroy, Mario Lučić, Guodong Zhang, Wael Farhan, Michael Sharman, Paul Natsev, Paul Michel, Yong Cheng, Yamini Bansal, Siyuan Qiao, Kris Cao, Siamak Shakeri, Christina Butterfield, Justin Chung, Paul Kishan Rubenstein, Shivani Agrawal, Arthur Mensch, Kedar Soparkar, Karel Lenc, Timothy Chung, Aedan Pope, Loren Maggiore, Jackie Kay, Priya Jhakra, Shibo Wang, Joshua Maynez, Mary Phuong, Taylor Tobin, Andrea Tacchetti, Maja Trebacz, Kevin Robinson, Yash Katariya, Sebastian Riedel, Paige Bailey, Kefan Xiao, Nimesh Ghelani, Lora Aroyo, Ambrose Slone, Neil Houlsby, Xuehan Xiong, Zhen Yang, Elena Gribovskaya, Jonas Adler, Mateo Wirth, Lisa Lee, Music Li, Thais Kagohara, Jay Pavagadhi, Sophie Bridgers, Anna Bortsova, Sanjay Ghemawat, Zafarali Ahmed, Tianqi Liu, Richard Powell, Vijay Bolina, Mariko Iinuma, Polina Zablotskaia, James Besley, Da-Woon Chung, Timothy Dozat, Ramona Comanescu, Xiance Si, Jeremy Greer, Guolong Su, Martin Polacek, Raphaël Lopez Kaufman, Simon Tokumine, Hexiang Hu, Elena Buchatskaya, Yingjie Miao, Mohamed Elhawaty, Aditya Siddhant, Nenad Tomasev, Jinwei Xing, Christina Greer, Helen Miller, Shereen Ashraf, Aurko Roy, Zizhao Zhang, Ada Ma, Angelos Filos, Milos Besta, Rory Blevins, Ted Klimenko, Chih-Kuan Yeh, Soravit Changpinyo, Jiaqi Mu, Oscar Chang, Mantas Pajarskas, Carrie Muir, Vered Cohen, Charline Le Lan, Krishna Haridasan, Amit Marathe, Steven Hansen, Sholto Douglas, Rajkumar Samuel, Mingqiu Wang, Sophia Austin, Chang Lan, Jiepu Jiang, Justin Chiu, Jaime Alonso Lorenzo, Lars Lowe Sjösund, Sébastien Cevey, Zach Gleicher, Thi Avrahami, Anudhyan Boral, Hansa Srinivasan, Vittorio Selo, Rhys May, Konstantinos Aisopos, Léonard Hussenot, Livio Baldini Soares, Kate Baumli, Michael B. Chang, Adrià Recasens, Ben Caine, Alexander Pritzel, Filip Pavetic, Fabio Pardo, Anita Gergely, Justin Frye, Vinay Ramasesh, Dan Horgan, Kartikeya Badola, Nora Kassner, Subhrajit Roy, Ethan Dyer, Víctor Campos, Alex Tomala, Yunhao Tang, Dalia El Badawy, Elspeth White, Basil Mustafa, Oran Lang, Abhishek Jindal, Sharad Vikram, Zhitao Gong, Sergi Caelles, Ross Hemsley, Gregory Thornton, Fangxiaoyu Feng, Wojciech Stokowiec, Ce Zheng, Phoebe Thacker, Çağlar Ünlü, Zhishuai Zhang, Mohammad Saleh, James Svensson, Max Bileschi, Piyush Patil, Ankesh Anand, Roman Ring, Katerina Tsihlas, Arpi Vezer, Marco Selvi, Toby Shevlane, Mikel Rodriguez, Tom Kwiatkowski, Samira Daruki, Keran Rong, Allan Dafoe, Nicholas FitzGerald, Keren Gu-Lemberg, Mina Khan, Lisa Anne Hendricks, Marie Pellat, Vladimir Feinberg, James Cobon-Kerr, Tara Sainath, Maribeth Rauh, Sayed Hadi Hashemi, Richard Ives, Yana Hasson, YaGuang Li, Eric Noland, Yuan Cao, Nathan Byrd, Le Hou, Qingze Wang, Thibault Sottiaux, Michela Paganini, Jean-Baptiste Lespiau, Alexandre Moufarek, Samer Hassan, Kaushik Shivakumar, Joost van Amersfoort, Amol Mandhane, Pratik Joshi, Anirudh Goyal, Matthew Tung, Andrew Brock, Hannah Sheahan, Vedant Misra, Cheng Li, Nemanja Rakićević, Mostafa Dehghani, Fangyu Liu, Sid Mittal, Junhyuk Oh, Seb Noury, Eren Sezener, Fantine Huot, Matthew Lamm, Nicola De Cao, Charlie Chen, Gamaleldin Elsayed, Ed Chi, Mahdis Mahdieh, Ian Tenney, Nan Hua, Ivan Petrychenko, Patrick Kane, Dylan Scandinaro, Rishub Jain, Jonathan Uesato, Romina Datta, Adam Sadovsky, Oskar Bunyan, Dominik Rabiej, Shimu Wu, John Zhang, Gautam Vasudevan, Edouard Leurent, Mahmoud Alnahlawi, Ionut Georgescu, Nan Wei, Ivy Zheng, Betty Chan, Pam G Rabinovitch, Piotr Stanczyk, Ye Zhang, David Steiner, Subhajit Naskar, Michael Azzam, Matthew Johnson, Adam Paszke, Chung-Cheng Chiu, Jaume Sanchez Elias, Afroz Mohiuddin, Faizan Muhammad, Jin Miao, Andrew Lee, Nino Vieillard, Sahitya Potluri, Jane Park, Elnaz Davoodi, Jiageng Zhang, Jeff Stanway, Drew Garmon, Abhijit Karmarkar, Zhe Dong, Jong Lee, Aviral Kumar, Luowei Zhou, Jonathan Evens, William Isaac, Zhe Chen, Johnson Jia, Anselm Levskaya, Zhenkai Zhu, Chris Gorgolewski, Peter Grabowski, Yu Mao, Alberto Magni, Kaisheng Yao, Javier Snaider, Norman Casagrande, Paul Suganthan, Evan Palmer, Geoffrey Irving, Edward Loper, Manaal Faruqui, Isha Arkatkar, Nanxin Chen, Izhak Shafran, Michael Fink, Alfonso Castaño, Irene Giannoumis, Wooyeol Kim, Mikołaj Rybiński, Ashwin Sreevatsa, Jennifer Prendki, David Soergel, Adrian Goedeckemeyer, Willi Gierke, Mohsen Jafari, Meenu Gaba, Jeremy Wiesner, Diana Gage Wright, Yawen Wei, Harsha Vashisht, Yana Kulizhskaya, Jay Hoover, Maigo Le, Lu Li, Chimezie Iwuanyanwu, Lu Liu, Kevin Ramirez, Andrey Khorlin, Albert Cui, Tian LIN, Marin Georgiev, Marcus Wu, Ricardo Aguilar, Keith Pallo, Abhishek Chakladar, Alena Repina, Xihui Wu, Tom van der Weide, Priya Ponnapalli, Caroline Kaplan, Jiri Simsa, Shuangfeng Li, Olivier Dousse, Fan Yang, Jeff Piper, Nathan Ie, Minnie Lui, Rama Pasumarthi, Nathan Lintz, Anitha Vijayakumar, Lam Nguyen Thiet, Daniel Andor, Pedro Valenzuela, Cosmin Paduraru, Daiyi Peng, Katherine Lee, Shuyuan Zhang, Somer Greene, Duc Dung Nguyen, Paula Kurylowicz, Sarmishta Velury, Sebastian Krause, Cassidy Hardin, Lucas Dixon, Lili Janzer, Kiam Choo, Ziqiang Feng, Biao Zhang, Achintya Singhal, Tejasi Latkar, Mingyang Zhang, Quoc Le, Elena Allica Abellan, Dayou Du, Dan McKinnon, Natasha Antropova, Tolga Bolukbasi, Orgad Keller, David Reid, Daniel Finchelstein, Maria Abi Raad, Remi Crocker, Peter Hawkins, Robert Dadashi, Colin Gaffney, Sid Lall, Ken Franko, Egor Filonov, Anna Bulanova, Rémi Leblond, Vikas Yadav, Shirley Chung, Harry Askham, Luis C. Cobo, Kelvin Xu, Felix Fischer, Jun Xu, Christina Sorokin, Chris Alberti, Chu-Cheng Lin, Colin Evans, Hao Zhou, Alek Dimitriev, Hannah Forbes, Dylan Banarse, Zora Tung, Jeremiah Liu, Mark Omernick, Colton Bishop, Chintu Kumar, Rachel Sterneck, Ryan Foley, Rohan Jain, Swaroop Mishra, Jiawei Xia, Taylor Bos, Geoffrey Cideron, Ehsan Amid, Francesco Piccinno, Xingyu Wang, Praseem Banzal, Petru Gurita, Hila Noga, Premal Shah, Daniel J. Mankowitz, Alex Polozov, Nate Kushman, Victoria Krakovna, Sasha Brown, MohammadHossein Bateni, Dennis Duan, Vlad Firoiu, Meghana Thotakuri, Tom Natan, Anhad Mohananey, Matthieu Geist, Sidharth Mudgal, Sertan Girgin, Hui Li, Jiayu Ye, Ofir Roval, Reiko Tojo, Michael Kwong, James Lee-Thorp, Christopher Yew, Quan Yuan, Sumit Bagri, Danila Sinopalnikov, Sabela Ramos, John Mellor, Abhishek Sharma, Aliaksei Severyn, Jonathan Lai, Kathy Wu, Heng-Tze Cheng, David Miller, Nicolas Sonnerat, Denis Vnukov, Rory Greig, Jennifer Beattie, Emily Caveness, Libin Bai, Julian Eisenschlos, Alex Korchemniy, Tomy Tsai, Mimi Jasarevic, Weize Kong, Phuong Dao, Zeyu Zheng, Frederick Liu, Fan Yang, Rui Zhu, Mark Geller, Tian Huey Teh, Jason Sanmiya, Evgeny Gladchenko, Nejc Trdin, Andrei Sozanschi, Daniel Toyama, Evan Rosen, Sasan Tavakkol, Linting Xue, Chen Elkind, Oliver Woodman, John Carpenter, George Papamakarios, Rupert Kemp, Sushant Kafle, Tanya Grunina, Rishika Sinha, Alice Talbert, Abhimanyu Goyal, Diane Wu, Denese Owusu-Afriyie, Cosmo Du, Chloe Thornton, Jordi Pont-Tuset, Pradyumna Narayana, Jing Li, Sabaer Fatehi, John Wieting, Omar Ajmeri, Benigno Uria, Tao Zhu, Yeongil Ko, Laura Knight, Amélie Héliou, Ning Niu, Shane Gu, Chenxi Pang, Dustin Tran, Yeqing Li, Nir Levine, Ariel Stolovich, Norbert Kalb, Rebeca Santamaria-Fernandez, Sonam Goenka, Wenny Yustalim, Robin Strudel, Ali Elqursh, Balaji Lakshminarayanan, Charlie Deck, Shyam Upadhyay, Hyo Lee, Mike Dusenberry, Zonglin Li, Xuezhi Wang, Kyle Levin, Raphael Hoffmann, Dan Holtmann-Rice, Olivier Bachem, Summer Yue, Sho Arora, Eric Malmi, Daniil Mirylenka, Qijun Tan, Christy Koh, Soheil Hassas Yeganeh, Siim Põder, Steven Zheng, Francesco Pongetti, Mukarram Tariq, Yanhua Sun, Lucian Ionita, Mojtaba Seyedhosseini, Pouya Tafti, Ragha Kotikalapudi, Zhiyu Liu, Anmol Gulati, Jasmine Liu, Xinyu Ye, Bart Chrzaszcz, Lily Wang, Nikhil Sethi, Tianrun Li, Ben Brown, Shreya Singh, Wei Fan, Aaron Parisi, Joe Stanton, Chenkai Kuang, Vinod Koverkathu, Christopher A. Choquette-Choo, Yunjie Li, TJ Lu, Abe Ittycheriah, Prakash Shroff, Pei Sun, Mani Varadarajan, Sanaz Bahargam, Rob Willoughby, David Gaddy, Ishita Dasgupta, Guillaume Desjardins, Marco Cornero, Brona Robenek, Bhavishya Mittal, Ben Albrecht, Ashish Shenoy, Fedor Moiseev, Henrik Jacobsson, Alireza Ghaffarkhah, Morgane Rivière, Alanna Walton, Clément Crepy, Alicia Parrish, Yuan Liu, Zongwei Zhou, Clement Farabet, Carey Radebaugh, Praveen Srinivasan, Claudia van der Salm, Andreas Fidjeland, Salvatore Scellato, Eri Latorre-Chimoto, Hanna Klimczak-Plucińska, David Bridson, Dario de Cesare, Tom Hudson, Piermaria Mendolicchio, Lexi Walker, Alex Morris, Ivo Penchev, Matthew Mauger, Alexey Guseynov, Alison Reid, Seth Odoom, Lucia Loher, Victor Cotruta, Madhavi Yenugula, Dominik Grewe, Anastasia Petrushkina, Tom Duerig, Antonio Sanchez, Steve Yadlowsky, Amy Shen, Amir Globerson, Adam Kurzrok, Lynette Webb, Sahil Dua, Dong Li, Preethi Lahoti, Surya Bhupatiraju, Dan Hurt, Haroon Qureshi, Ananth Agarwal, Tomer Shani, Matan Eyal, Anuj Khare, Shreyas Rammohan Belle, Lei Wang, Chetan Tekur, Mihir Sanjay Kale, Jinliang Wei, Ruoxin Sang, Brennan Saeta, Tyler Liechty, Yi Sun, Yao Zhao, Stephan Lee, Pandu Nayak, Doug Fritz, Manish Reddy Vuyyuru, John Aslanides, Nidhi Vyas, Martin Wicke, Xiao Ma, Taylan Bilal, Evgenii Eltyshev, Daniel Balle, Nina Martin, Hardie Cate, James Manyika, Keyvan Amiri, Yelin Kim, Xi Xiong, Kai Kang, Florian Luisier, Nilesh Tripuraneni, David Madras, Mandy Guo, Austin Waters, Oliver Wang, Joshua Ainslie, Jason Baldridge, Han Zhang, Garima Pruthi, Jakob Bauer, Feng Yang, Riham Mansour, Jason Gelman, Yang Xu, George Polovets, Ji Liu, Honglong Cai, Warren Chen, XiangHai Sheng, Emily Xue, Sherjil Ozair, Adams Yu, Christof Angermueller, Xiaowei Li, Weiren Wang, Julia Wiesinger, Emmanouil Koukoumidis, Yuan Tian, Anand Iyer, Madhu Gurumurthy, Mark Goldenson, Parashar Shah, MK Blake, Hongkun Yu, Anthony Urbanowicz, Jennimaria Palomaki, Chrisantha Fernando, Kevin Brooks, Ken Durden, Harsh Mehta, Nikola Momchev, Elahe Rahimtoroghi, Maria Georgaki, Amit Raul, Sebastian Ruder, Morgan Redshaw, Jinhyuk Lee, Komal Jalan, Dinghua Li, Ginger Perng, Blake Hechtman, Parker Schuh, Milad Nasr, Mia Chen, Kieran Milan, Vladimir Mikulik, Trevor Strohman, Juliana Franco, Tim Green, Demis Hassabis, Koray Kavukcuoglu, Jeffrey Dean, and Oriol Vinyals. 2023.  Gemini: A family of highly capable multimodal models.
  * Veale et al. (2016) Tony Veale, Ekaterina Shutova, and Beata Beigman Klebanov. 2016.  _Metaphor: A computational perspective_.  Morgan & Claypool Publishers.
  * Wiegreffe et al. (2022) Sarah Wiegreffe, Jack Hessel, Swabha Swayamdipta, Mark Riedl, and Yejin Choi. 2022.  Reframing human-AI collaboration for generating free-text explanations.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 632–658, Seattle, United States. Association for Computational Linguistics.
  * Wiegreffe and Marasovic (2021) Sarah Wiegreffe and Ana Marasovic. 2021.  Teach me to explain: A review of datasets for explainable natural language processing.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks_ , volume 1. Curran.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Xie et al. (2019) Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. 2019.  Visual entailment: A novel task for fine-grained image understanding.  _ArXiv_ , abs/1901.06706.
  * Yosef et al. (2023) Ron Yosef, Yonatan Bitton, and Dafna Shahaf. 2023.  IRFL: Image recognition of figurative language.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 1044–1058, Singapore. Association for Computational Linguistics.
  * Yue et al. (2024) Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. 2024.  Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi.  In _Proceedings of CVPR_.
  * Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhou et al. (2021) Jianing Zhou, Hongyu Gong, and Suma Bhat. 2021.  PIE: A parallel idiomatic expression corpus for idiomatic sentence generation and paraphrasing.  In _Proceedings of the 17th Workshop on Multiword Expressions (MWE 2021)_ , pages 33–48, Online. Association for Computational Linguistics.
  * Ziems et al. (2024) Caleb Ziems, William Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang. 2024.  Can large language models transform computational social science?  _Computational Linguistics_ , pages 1–55.

##  Appendix A Dataset Statistics

Table 7 shows the number of samples from each source dataset that are included
in the randomly selected training, validation, and held-out test splits.

Type | Dataset | Train | Valid | Test
---|---|---|---|---
Metaphor /Similes | HAIVMET | 649 | 107 | 101
| IRFL
---
(metaphor
/simile)
912 | 117 | 120
Idioms | IRFL (idiom) | 170 | 100 | 100
Sarcasm | MuSE | 830 | 106 | 106
Humor | MemeCap | 1566 | 196 | 196
NYCartoons | 451 | 100 | 100
Total | 4,578 | 726 | 723
Table 7: Data counts per phenomenon and dataset.

##### Length distribution

Average length of a claim in V-FLUTE is $\approx$ 61 characters. Average
length of an explanation is $\approx$ 367 characters. Figure 8 shows the
distribution of claim lengths, and Figure 9 shows the distribution of
explanation lengths by source dataset. We manually verified that the outlier
instances are correct.

Figure 8: Distribution of lengths of claims by source dataset. Figure 9:
Distribution of lengths of explanations by source dataset.

##  Appendix B API models Hyperparameters

###  B.1 Claude

  * •

Model Name: claude-3-opus-20240229

  * •

Max Tokens: 256

  * •

Images greater than 5MB were resized maintaining aspect ratio

###  B.2 GPT-4

  * •

Model Name: gpt-4-1106-vision-preview

  * •

Max Tokens: 256

  * •

Seed: 42

  * •

Image URL detail: ’high’

###  B.3 Gemini

  * •

Model Name: gemini-pro-vision

  * •

Max Tokens: 256

  * •

Safety Settings: ’BLOCK NONE’

  * •

Images greater than 5MB were resized maintaining aspect ratio

##  Appendix C Fine-tuning Hyperparameters

We utilize LoRA Hu et al. (2022) to fine-tune the models. We utilize the same
hyperparameters for all fine-tunes outlined in Appendix C and use early
stopping based on a V-FLUTE validation set to prevent overfitting. Due to size
of the data, we only train for 3 epochs in both eViL and eVIL+VFLUTE
variations, but we saved intermediate steps and chose the checkpoint where the
validation loss is lowest.

### Fine-tuning

  * •

Seed: 42

  * •

Vision Tower: openai-clip-vit-large-patch14-336

  * •

Number of Training Epochs: 3

  * •

Train Batch Size (per device): 16

  * •

Eval Batch Size (per device): 4

  * •

Learning Rate: 2e-5

  * •

Weight Decay: 0

  * •

Warmup Ratio: 0.03

  * •

Scheduler Type: cosine

  * •

Number of epochs: 4 for eViL and eViL + vFLUTE, 10 for VFLUTE

  * •

mm-projector-type: mlp2x gelu

  * •

mm-vision-select-layer: -2

  * •

mm-use-im-start-end: False

  * •

mm-use-im-patch-token: False

  * •

image-aspect-ratio: pad

  * •

group-by-modality-length: False

### LoRA

  * •

lora r: 128

  * •

lora alpha: 256

  * •

mm-projector-lr: 2e-5

### Deepspeed Configuration

  * •

FP16 enabled: auto

  * •

BF16 enabled: auto

  * •

Micro Batch Size Per GPU: auto

  * •

Train Batch Size: auto

  * •

Gradient Accumulation Steps: auto

  * •

Zero Optimization Stage: 3

### Training and Inference Instructions

All models are evaluated using beam search with $n=3$, temperature $0$, max
length $256$. In the case of generating scene graphs for the compositional
chain-of-thought method, we set the max length to 256 for the graph generation
step as recommended by Mitra et al. (2023). API models are evaluated with
default hyperparameters. We format all fine-tuning data in the instruction
format following LLaVA Liu et al. (2023a). To avoid overfitting on a
particular instruction for this task, we generate 20 similar instructions
using an LLM (ChatGPT-4) and randomly assign one of them to every instance in
the training, validation, and testing set. Same instructions were sampled for
the e-ViL dataset. Table 8 shows the 20 instructions used.

No. Instruction 1 Does the image’s narrative confirm or disprove the claim
REPLACE_CLAIM? Discuss your reasoning and identify it as either entailment or
contradiction. 2 Does this image confirm or deny the claim REPLACE_CLAIM?
Discuss your reasoning and determine a label: entailment or contradiction. 3
Is the image’s message supporting or opposing the claim REPLACE_CLAIM? Discuss
your rationale and determine the appropriate label: entailment or
contradiction. 4 Is there agreement or disagreement between the image and the
claim REPLACE_CLAIM? Provide your analysis and choose between entailment or
contradiction. 5 Does the visual evidence support or counter the claim
REPLACE_CLAIM? Provide your explanation and assign it a label of entailment or
contradiction. 6 Does the image agree with or dispute the claim REPLACE_CLAIM?
Explain your analysis and mark it as entailment or contradiction. 7 Does the
illustration affirm or contest the claim REPLACE_CLAIM? Provide your argument
and choose a label: entailment or contradiction. 8 Is the visual content in
agreement or disagreement with the claim REPLACE_CLAIM? Offer your explanation
and categorize it under entailment or contradiction. 9 Is the image in harmony
with or in conflict with the statement REPLACE_CLAIM? Explain your
justification and label it as entailment or contradiction. 10 Is the portrayal
in the image consistent with or contradictory to the claim REPLACE_CLAIM?
Offer your insights and select between entailment or contradiction. 11 Does
the image’s depiction validate or refute the claim REPLACE_CLAIM? Explain your
point of view and select a label: entailment or contradiction. 12 Is the
content of the image endorsing or challenging the claim REPLACE_CLAIM? Justify
your position and label it as entailment or contradiction. 13 Is the image
consistent with the statement REPLACE_CLAIM? Justify your answer and classify
it as either entailment or contradiction. 14 Does the illustration affirm or
negate the claim REPLACE_CLAIM? Articulate your reasoning and apply a label:
entailment or contradiction. 15 Does the picture support or refute the
assertion REPLACE_CLAIM? Offer your rationale and select a label: entailment
or contradiction. 16 Is the visual portrayal compatible with or adverse to the
claim REPLACE_CLAIM? Justify your viewpoint and label it as entailment or
contradiction. 17 Does the image corroborate or dispute the claim
REPLACE_CLAIM? Outline your reasoning and categorize it under entailment or
contradiction. 18 Is the depiction aligned with or against the claim
REPLACE_CLAIM? Share your evaluation and identify it as either entailment or
contradiction. 19 Does the image entail or contradict the claim REPLACE_CLAIM?
Explain your reasoning and provide a label between entailment or
contradiction. 20 Can the image be seen as validating or opposing the claim
REPLACE_CLAIM? Explain your thought process and assign a label of entailment
or contradiction 21 Is the image’s representation supportive of or
contradictory to the claim REPLACE_CLAIM? Articulate your analysis and assign
the label: entailment or contradiction.

Table 8: Instruction variations for the figurative visual entailment task.

##  Appendix D Prompts for LLMs

###  D.1 HAIVMET

####  D.1.1 One-shot Prompt for generating explanations

We describe our one-shot prompts given to an LLM (gpt-3.5-turbo-instruct-0914)
for generating explanations of entailment-contradiction relationship. Refer to
Table 9 for the detailed prompt.

| You will be provided a Caption describing what
---
is in the image in detail. You will also be
provided with a Claim that contradicts or
is entailed by the image (as indicated by
the Label). Your task is to explain why
the claim contradicts or is entailed by
the image. Be very brief in your explanation.
Start your explanation by describing what the
image depicts, displays or shows.
| Caption: An illustration of a group of soldiers
---
with red skin, horns, and pitchforks in hand
with a fierce expression on their faces.
Claim: The soldiers were angels.
Label: Contradiction
Explanation: The image depicts soldiers
with red skin, horns, and pitchforks, which
are traditional characteristics associated
with demons, not angels. Therefore, the
claim that the soldiers were angels contradicts
the image.
| Caption:……
---
Table 9:  One shot prompt given to an LLM (gpt-3.5-turbo-instruct-0914) for
generating explanations of entailment-contradiction relationship of the
HAIVMET dataset.

###  D.2 IRFL

####  D.2.1 Zero-shot Prompt for generating explanations

We provide our zero-shot prompt given to an LLM (gpt-4-vision-preview) for
generating the entailment explanations given the claim and the image. Refer
Table 10 for the detailed prompt.

| You will be provided an image. You will also be
---
provided with a simile that contradicts or is
entailed by the image (as indicated by the Label).
Your task is to explain why the simile contradicts
or is entailed by the image. Be very brief in your
explanation and remain consistent to the Label in
your explanation. Start your explanation by
describing what the image depicts, displays or
shows.
Simile: ….
Label: ….
Explanation:
Table 10:  Zero shot prompt given to an LLM (gpt-4-vision-preview) for
generating explanations of entailment-contradiction relationship of the IRFL
Dataset. The dataset contains similes, metaphors and idioms. For metaphors and
idioms, the word simile in the prompt is replaced with the corresponding type.

###  D.3 MuSE

####  D.3.1 Few-shot Prompt for generating opposite claims

We provide our few-shot prompt given to an LLM ((gpt-4-0613)) for generating
the opposite claims. Refer Table 11 for the detailed prompt.

| u are an online redditor or flickr user and u
---
always type in informal style. Convert the
following sarcastic claim into a non-sarcastic
claim. Preserve the informal style, including
capitalization. Be super laid back and informal!!!
| 1\. Sarcastic claim: stairs vs . escalator in
---
airport . i wonder why we have an # obesity
problem ? # publichealth # ncds # globalhealth
# isometimesdothistoo
Explanation: no wonder we have an obesity
problem since everyones using escalator
instead of stairs in airport.
Non-sarcastic claim: it s clear why we have an
# obesity problem look at stairs vs. escalator
in airport
| Claim:……
---
| Explanation:……
---
Table 11:  Few shot prompt given to an LLM (gpt-4-0613) for generating
opposite claims utilizing the sarcastic claim and crowd worker explanation.

####  D.3.2 Zero-shot Prompt for Rephrasing

We provide our zero-shot prompt given to an LLM (gpt-4-vision-preview) for
rephrasing the explanations given the claim and the crowd worker explanation.
Refer Table 12 for the detailed prompt.

| Paraphrase the draft explanation of why the image
---
contradicts the literal interpretation of the claim.
Be sure to first describe the image in one sentence.
Keep your answer short. Do not refer to the claim
or the draft explanation in your paraphrase. Stay
close to the draft explanation.
Claim: ….
Draft Explanation:
Table 12:  Zero shot prompt given to an LLM (gpt-4-vision-preview) for
rephrasing the explanations given the claim and the.

###  D.4 MemeCap

####  D.4.1 Few-shot Prompt for generating entailing claims

We describe our few-shot prompts given to an LLM (gpt-4-0613) for generating
entailing claims as part of the pipeline. Refer to Table 13 for the detailed
prompt.

| You will be provided with a meme caption. Your
---
task is to write the meme caption as a claim such
that the meme poster is not mentioned in the
claim.
| Caption: Meme poster is saying that searching
---
Google plus the term you want to search on
reddit is better than searching reddit itself.
Claim: Searching on Google with the term
you want to search plus ’reddit’ is more effective
than searching directly on Reddit.
| Caption: The person who wrote the post is saying
---
people on Instagram are soft and reddit are funny.
Claim: People on Instagram are soft, whereas
those on Reddit are funny.
| Caption:……
---
Table 13:  Two shot prompt given to an LLM (gpt-4-0613) for generating
entailing claims utilizing the meme captions part of the MemeCap dataset.

####  D.4.2 Zero-shot Prompt for validating the entailing claims

We describe our zero-shot prompt given to an LLM (gpt-4-vision-preview) for
validating the claims generated in the previous step. Refer Table 14 for the
detailed prompt.

| You will be provided a meme image and a claim.
---
Your task is to check whether the claim entails the
image. Answer with a Yes or No.
Claim: …..
Table 14:  Zero shot prompt given to an LLM (gpt-4-vision-preview) for
validating the claims generated in D.4.1. The corresponding meme image is also
attached with the prompt.

####  D.4.3 Few-shot Prompt for generating opposite claims

We provide our few-shot prompt given to an LLM ((gpt-4-0613)) for generating
the opposite claims. Refer Table 15 for the detailed prompt.

| Claim: A useful feature has been removed
---
on YouTube, causing disappointment.
Explanation: The image shows a painting
of a character with a distraught face and
a speech bubble that reads "y tho," placed
over text saying "When YouTube removed
sort by oldest option." This implies that the
removal of the sort by oldest option is a
decision that users are questioning, hence
indicating disappointment over the loss of
a useful feature.
Opposite claim: An unhelpful feature has
been removed on YouTube, causing happiness.
| Claim:……
---
| Explanation:……
---
Table 15:  Few shot prompt given to an LLM (gpt-4-0613) for generating
opposite claims utilizing the generated claim and explanation.

####  D.4.4 Zero-shot Prompt for generating explanations

We provide our zero-shot prompt given to an LLM (gpt-4-vision-preview) for
generating the entailment explanations given the claim and the image. Refer
Table 16 for the detailed prompt.

| You will be provided a meme. You will also be
---
provided with a claim that entails the image.
Your task is to explain why the claim is entailed
by the image. Be very brief in your explanation
and start your explanation by describing what
the image depicts, displays or shows.
Claim: ….
Explanation:
Table 16:  Zero shot prompt given to an LLM (gpt-4-vision-preview) for
generating the entailment explanations. The corresponding meme image is also
attached with the prompt.

##  Appendix E Model Taxonomy

The taxonomy of all models used for automatic evaluation is shown in Figure
10.

for tree= grow=east, draw, rectangle, align=center, parent anchor=east, child
anchor=west, edge=thick, l sep+=10pt, tier/.pgfmath=level(), edge path= [draw,
edge] (!u.parent anchor) – +(5pt,0) |- (.child anchor)edge label; [Models
[Off-the-shelf [API-based [Claude Opus] [GPT-4] [Gemini] ] [Open (LLaVA-ZS)
[7B] [7B-SG] [34B] [34B-SG] ] ] [Fine-tuned (LLaVA-7B) [eViL] [VFLUTE]
[eViL+VFLUTE] ] ]

Figure 10: Taxonomy of models used for the study.

##  Appendix F By-Phenomenon Performance

In Figure 11, we show the performance of the models by phenomenon and dataset
across various thresholds.

(a) Metaphors and Similes

(b) Humor

(c) Sarcasm and Idioms

Figure 11: Performance of the models by phenomenon.

##  Appendix G Annotator Recruitment and Compensation

All recruited annotators have significant background in figurative language
understanding (having formal educational background in linguistics or
literature). Annotators were recruited through Upwork platform. All are on
fluent or native/bilingual level in English. All are fairly compensated with
USD $20 to $25 per hour with self-reported time needed to complete the tasks.