# RED-DOT: Multimodal Fact-checking via Relevant Evidence Detection

Stefanos-Iordanis Papadopoulos Corresponding author Information Technology
Institute, Centre for Research & Technology, Hellas.  Department of Electrical
& Computer Engineering, Aristotle University of Thessaloniki.  Christos
Koutlis  Information Technology Institute, Centre for Research & Technology,
Hellas.  Symeon Papadopoulos  Information Technology Institute, Centre for
Research & Technology, Hellas.  Panagiotis C. Petrantonakis  Department of
Electrical & Computer Engineering, Aristotle University of Thessaloniki.

###### Abstract

Online misinformation is often multimodal in nature, i.e., it is caused by
misleading associations between texts and accompanying images. To support the
fact-checking process, researchers have been recently developing automatic
multimodal methods that gather and analyze external information, evidence,
related to the image-text pairs under examination. However, prior works
assumed all collected evidence to be relevant. In this study, we introduce a
“Relevant Evidence Detection” (RED) module to discern whether each piece of
evidence is relevant, to support or refute the claim. Specifically, we develop
the “Relevant Evidence Detection Directed Transformer” (RED-DOT) and explore
multiple architectural variants (e.g., single or dual-stage) and mechanisms
(e.g., “guided attention”). Extensive ablation and comparative experiments
demonstrate that RED-DOT achieves significant improvements over the state-of-
the-art on the VERITE benchmark by up to 28.5%. Furthermore, our evidence re-
ranking and element-wise modality fusion led to RED-DOT achieving competitive
and even improved performance on NewsCLIPings+, without the need for numerous
evidence or multiple backbone encoders. Finally, our qualitative analysis
demonstrates that the proposed “guided attention” module has the potential to
enhance the architecture’s interpretability. We release our code at:
https://github.com/stevejpapad/relevant-evidence-detection.

Figure 1:  Image-text pair under verification with external evidence, both
images and texts, collected from the web. The proposed framework retrieves and
re-ranks the evidence while RED-DOT determines which pieces of information are
most relevant to (support or refute) the image-text pair and then uses those
to determine the pair’s veracity.

_K_ eywords Multimodal Learning $\cdot$ Deep Learning $\cdot$ Misinformation
Detection $\cdot$ Multimodal Fact-checking

##  1 Introduction

The dissemination of misinformation has greatly intensified in the digital age
with the advent of the internet and social media platforms [1] and entails
numerous adverse outcomes [2, 3, 4, 5, 6, 7]. Generative AI advances, such as
Large Language Models and Stable Diffusion, can now be exploited to create
convincing and hyper-realistic misinformation [8, 9]. Nevertheless,
decontextualization or image re-purposing, where images are taken out of their
original context in order to support false narratives, remain a prevalent
challenge in multimodal misinformation detection [10, 11]. For instance, Fig.1
provides a real-world example, where a legitimate image is used as evidence to
promote a false narrative, linking 5G technology to the transmission of
COVID-19 when in reality, the image captures an anti-surveillance protest in
Hong Kong, China111https://www.snopes.com/fact-check/5g-tower-torn-down-china-
covid.

Recent years have witnessed a surge in research attention towards automated
multimodal misinformation detection [12] with efforts focusing on the creation
of multimodal misinformation datasets [10, 11, 13, 14, 15, 16, 17, 18] and
detection methods [19, 20, 21, 22, 23, 23]. Nevertheless, solely relying on
the analysis of an image-text pair is not always sufficient for detecting
misinformation. More often that not, the incorporation and cross-examination
of external information, i.e., evidence, is necessary [24]. To this end,
researchers have explored methods for automated and evidence-based fact-
checking, initially focusing on text-based approaches [25, 26, 27, 28] and,
more recently, venturing into the realm of Multimodal Fact-Checking (MFC) [29,
30, 31]. Certain works on MFC, collect external evidence from fact-checked
articles [29, 30], which can be considered to be “leaked evidence” [32].
Conversely, the NewsCLIPings+ dataset, comprises external information
collected from the web [31], does not suffer from “leaked evidence” and has
recently been used in numerous studies [31, 33, 34]. Nevertheless, prior works
on MFC operated under the assumption that all collected evidence were relevant
and only addressed the task of verdict prediction.

Motivation. We aim to simulate a more realistic scenario, akin to what fact-
checkers encounter daily, where multiple pieces of information are amassed and
examined, but not all of them are necessarily relevant for supporting or
refuting a claim. Thus, the challenge lies in effectively distinguishing
between relevant and irrelevant evidence in order to assist the overall
verdict prediction process.

Contributions. We incorporate the step of Relevant Evidence Detection (RED),
as part of the MFC process. Specifically, we propose the Relevant Evidence
Detection Directed Transformer (RED-DOT) which comprises: “Evidence re-
ranking”, “Modality Fusion” and “RED” modules. “Evidence re-ranking” leverages
CLIP-based [35] intra-modal similarity to re-rank the evidence based on
relevance and to create hard negative “irrelevant” samples. “Modality Fusion”
leverages element-wise operations between the two modalities in order to
cross-check their relation. Finally, “RED” examines all provided pieces of
information, determines their relevance to the image-text pair and leverages
the relevant ones to assess the pair’s veracity.

We explore numerous variants of RED-DOT, with different architectures (e.g.,
single or dual stage) and mechanisms (e.g., guided attention). RED-DOT is
optimized via multi-task learning on the NewsCLIPings+ dataset [31] while its
performance is also evaluated on the VERITE benchmark [36] comprising real-
world multimodal misinformation. Finally, we propose the Out-Of-Distribution
Cross-Validation (OOD-CV) evaluation protocol in order to capture models with
superior generalizability across algorithmically created training data
(NewsCLIPings+) and real-world data (VERITE).

Findings. We conduct extensive ablation and comparative experiments that
demonstrate several key findings: (1) OOD-CV captures models with improved
performance compared to in-distribution validation (ID-V). (2) By utilizing
evidence re-ranking, a single piece of evidence per modality suffices while
additional information can introduce noise; degrading overall performance. (3)
The combination of element-wise operations can significantly boost detection
accuracy, especially when contrasted with simple late-fusion concatenation.
(4) RED-DOT surpasses prior methods that do not incorporate external evidence
by a substantial margin of up to 28.5% and the RED-DOT Baseline, which does
not utilize the RED module, by 4.5%. Moreover, it competes and even surpasses
the current state-of-the-art on NewsCLIPings+ without requiring multiple
backbone encoders, numerous evidence or additional features. (5) Finally, we
demonstrate that leveraging “guided attention” -where the loss function is
directly applied on the attention weights- has the potential to improve
interpretability.

##  2 Related Work

In recent years, automated misinformation detection has gained significant
research attention, with ongoing exploration of methods for identifying false
information across textual [37], visual [38], and multimodal formats [39].
While several datasets have been developed for text-based fact-checking [25,
26, 27, 28], Glockner et al. [32] showed that most employ “leaked evidence”
from fact-checked articles. This leads to an unrealistic setting that is not
applicable for early detection and emerging misinformation detection. In the
context of multimodal misinformation, the majority of research only considers
the image-text pairs [10, 11, 14, 15, 16, 19, 20, 21, 22, 36, 40, 41, 42] and
do not incorporate external information or evidence; with few notable
exceptions [29, 30, 31].

MOCHEG [29] was developed as an end-to-end dataset for MFC. However, it is
important to note that it only accommodates textual claims -not multimodal
inputs- while examining multimodal evidence. Moreover, the provided evidence
can be considered “leaked” since they are collected solely from fact-check
articles. Likewise, FACTIFY2 [30] contains “leaked evidence” in the “Refute”
class; resulting in high scores of 99-100% even by simple baseline models
[43]. To the best of our knowledge, FACTIFY2 is the only dataset addressing
multimodal evidence entailment. The authors distinguish between “multimodal”
(both modalities contribute to the verdict) or “text” entailment (only the
textual modality aligns with or exhibits similarity to the claim).
Nevertheless, this framework does not account of the “refute” class or
situations where both modalities are irrelevant or only the images are
entailed.

On the other hand, Abdelnabi et al. [31] leverage the NewsCLIPings dataset
[11], augmenting it with external information; which we refer to as
NewsCLIPings+ for simplicity. NewsCLIPings is a algorithmically generated
dataset, where image-text pairs are decontextualized using the multimodal
encoder CLIP [35], as well as Scene and Person matching computer vision
models. As the “falsified” pairs in NewsCLIPings are algorithmically
generated, it secures the absence of “leaked” evidence. Recent studies have
utilized NewsCLIPings+ for evidence-based multimodal fact-checking [31, 33,
34]; see Section 4.2 for more information. However, this algorithmic
generation introduces uncertainty regarding the ability of models trained on
NewsCLIPings to generalize effectively to real-world multimodal
misinformation.

Recently, researchers have been experimenting with training multimodal
misinformation detection models on algorithmically generated datasets like
NewsCLIPings [11] and MEIR [40] and evaluating their generalizability to real-
world misinformation [16]. Due to unimodal biases in widely adopted
benchmarks, such as VMU Twitter [13] and COSMOS [10], Papadopoulos et al. [36]
developed VERITE, an evaluation benchmark for real-world multimodal
misinformation detection that accounts for unimodal biases.

##  3 Methodology

###  3.1 Problem Formulation

We define the tasks of Multimodal Fact-Checking (MFC) and Relevant Evidence
Detection (RED) as follows: given a dataset
${(I^{v}_{i},T^{v}_{i},\mathcal{I}^{e}_{i}},\mathcal{T}^{e}_{i},y^{v}_{i},\boldsymbol{y}^{e}_{i})_{i=1}^{N}$,
where $(I^{v}_{i},T^{v}_{i})$ represents the image-text pair under
verification (pair, from now on),
$\mathcal{I}^{e}_{i}=[I^{e}_{i1},I^{e}_{i2},\ldots,I^{e}_{i2K}]$ and
$\mathcal{T}^{e}_{i}=[T^{e}_{i1},T^{e}_{i2},\ldots,T^{e}_{i2M}]$ represent an
array of image evidence with $2\cdot K$ elements ($K$ relevant + $K$
irrelevant) and text evidence with $2\cdot M$ elements ($M$ relevant + $M$
irrelevant), respectively, $y^{v}_{i}\in{0,1}$ denotes the overall verdict
label, truthful (0) or misinformation (1) and
$\boldsymbol{y}^{e}_{i}=(y^{e}_{i1},y^{e}_{i2},\ldots,y^{e}_{i2\cdot(M+K)})$
is an array of binary labels of $2\cdot(M+K)$ size denoting whether each piece
of evidence is relevant (1) or irrelevant (0) to the pair; with the primary
objective of training a classifier
$f:(\mathcal{I}^{v},\mathcal{T}^{v},\mathcal{I}^{e},\mathcal{T}^{e})\rightarrow(\hat{y}^{v},\hat{\boldsymbol{y}}^{e})$.

###  3.2 Evidence Retrieval and Re-ranking

Figure 2: Visualization of Eq.2. Hard negative sampling for retrieving
“irrelevant” evidence.

We leverage the NewsCLIPings+ dataset [31] comprising 85,360 image-text pairs,
balanced in terms of truthful and misinformation pairs. The authors use the
text $T^{v}_{i}$ to retrieve visual evidence $\mathcal{I}^{e}_{i}$ and
$T^{v}_{i}$ to retrieve textual evidence $\mathcal{T}^{e}_{i}$. Each pair is
associated with up to 19 textual evidence and up to 10 visual evidence, with a
total of 146,032 text evidence and 736,731 image evidence; collected via
Google API. While we expect some level of relevance between the collected
evidence and the pair, we undertake the task of re-ranking all gathered
evidence to heighten the likelihood of their relevance.

Following [31], we use the pre-trained CLIP ViT B-32 [35] as the encoder in
order to extract visual $F_{I}\in{R}^{512\times 1}$ and textual features
$F_{T}\in{R}^{512\times 1}$ for $\mathcal{I}^{v},\mathcal{I}^{e}$ and
$\mathcal{T}^{v},\mathcal{T}^{e}$. We signify the ranked or “relevant”
evidence with the superscript “+”, and rank them based on the intra-modal
cosine similarity $sim$ as follows:

$\begin{split}\mathcal{I}^{+}_{i}=\operatorname*{argsort}\limits_{I^{e}_{j}\in\mathcal{I}^{e}_{i}}sim(F_{I^{v}_{i}},F_{I^{e}_{j}})\\\
\mathcal{T}^{+}_{i}=\operatorname*{argsort}\limits_{T^{e}_{j}\in\mathcal{T}^{e}_{i}}sim(F_{T^{v}_{i}},F_{T^{e}_{j}})\end{split}$
(1)

For the “irrelevant” evidence class, denoted by the superscript “-”, we employ
hard negative sampling, instead of depending on random sampling. More
specifically, we calculate the most similar item based on text-text
similarity, fetch its ranked evidence and employ them as
$\mathcal{I}^{-}_{i}$, and use image-image similarity for
$\mathcal{T}^{-}_{i}$. We incorporate the ‘opposite’ modality in similarity
calculations in order to mimic the evidence retrieval process of
NewsCLIPings+. The process of hard negative sampling “irrelevant” evidence is
illustrated in Fig.2 and can be expressed as follows:

$\begin{split}\mathcal{I}^{-}_{i}=\mathcal{I}^{+}_{j},\text{ where
}j=\operatorname*{argmax}\limits_{T^{v}_{j}\in\mathcal{T}^{v},j\neq
i}sim(F_{T^{v}_{i}},F_{T^{v}_{j}})\\\
\mathcal{T}^{-}_{i}=\mathcal{T}^{+}_{j},\text{ where
}j=\operatorname*{argmax}\limits_{I^{v}_{j}\in\mathcal{I}^{v},j\neq
i}sim(F_{I^{v}_{i}},F_{I^{v}_{j}})\end{split}$ (2)

After retrieving relevant evidence
$\mathcal{E}^{+}=(\mathcal{T}^{+},\mathcal{I}^{+})$ and irrelevant
$\mathcal{E}^{-}=(\mathcal{T}^{-},\mathcal{I}^{-})$, we also define
$\boldsymbol{y}^{e}+=[1,1,\ldots,1]$ and $\boldsymbol{y}^{e}-=[0,0,\ldots,0]$,
each having size M+K. In order to avoid overfitting based on positional
patterns during training, we combine all evidence
$\mathcal{E^{\prime}}=\mathcal{E}^{+}\cup\mathcal{E}^{-}$ and labels
$\boldsymbol{y^{\prime}}^{e}=[1,1,...1,0,0,...,0]$, shuffle their positions
with $P$ representing the permutation positions of the elements and the final
evidence and labels become: $\mathcal{E}=\mathcal{E^{\prime}}[P]$ and
$\boldsymbol{y}^{e}=\boldsymbol{y^{\prime}}^{e}[P]$. Evidence $\mathcal{E}$
should consist of $2\cdot(M+K)$ items. For instance, setting $M=1$ signifies
our expectation of having 1 relevant and 1 irrelevant textual evidence. If
this requirement is not met, we randomly sample additional items from the
dataset.

###  3.3 Modality Fusion module

To enhance the cross-examination of image-text pairs and accentuate specific
consistencies or inconsistencies, we utilize a range of multimodal fusion
operations. In addition to simple concatenation of the two modalities’
features, denoted as $[F_{I^{v}};F_{T^{v}}]$, we also employ “addition”,
“subtraction” and “multiplication”. Our rationale is that “addition”
emphasizes complementarity, “subtraction” accentuates differences and
“multiplication” underscores shared aspects. We express the modality fusion
module as follows:

$F^{v}=[F_{I^{v}};F_{T^{v}};F_{I^{v}}+F_{T^{v}};F_{I^{v}}-F_{T^{v}};F_{I^{v}}*F_{T^{v}}]$
(3)

with $F^{v}\in{R}^{512\times 5}$. Prior research in multimodal fusion has
explored different element-wise operations, including multiplication [44] and
outer product [45]. However, to the best of our knowledge, the combination of
multiple fusion operations remains unexplored, especially for MFC.

(a) Overview of the proposed Transformer $D(\cdot)$ in RED-DOT, employing
“Modality Fusion”.

(b) High-level overview of the single and dual stage RED-DOT variants. Dotted
lines represent the second stage in DSL.

Figure 3:

###  3.4 Verdict Prediction module

We employ a Transformer encoder $D(\cdot)$ to obtain the predicted verdict
$\hat{y}^{v}$. Our RED-DOT-Baseline experiments that only leverage relevant
evidence are expressed by Eq.4 while the final verdict is predicted by Eq.5.

$[\mathbf{d_{\text{CLS}},d_{F^{v}},d_{F_{\mathcal{E+}}}}]=D([\text{CLS};F^{v};F_{\mathcal{E+}}])$
(4)
$\hat{y}^{v}=\textbf{W}_{1}\cdot\text{GELU}(\textbf{W}_{0}\cdot\text{LN}(\mathbf{d}_{\text{CLS}}))$
(5)

where LN stands for Layer Normalization, CLS stands for classification token,
$\textbf{W}_{0}\in\mathbb{R}^{1\times 512}$ is a GELU activated fully
connected layer and $\textbf{W}_{1}\in\mathbb{R}^{256\times 1}$ is the final
verdict classification layer. $D(\cdot)$ is optimized for binary
classification based on the binary cross entropy (BCE) loss function $L^{v}$,
after applying the sigmoid activation function on $\hat{y}^{v}$.

###  3.5 Relevant Evidence Detection

The Relevant Evidence Detection (RED) module is responsible for discerning
which evidence in $\mathcal{E}$ are relevant or irrelevant to the claim under
verification. All methods utilize Transformer $D(\cdot)$ as shown in Eq.6 with
$\mathbf{d}_{F_{\mathcal{E}}}$ as shown in Eq.7 and Eq.5 to predict
$\hat{y_{v}}$. Moreover, all methods leverage multi-task learning to predict
both binary $\hat{y}^{v}$ and multi-label $\boldsymbol{\hat{y}}^{e}$ and are
optimized based on $L=L^{v}+L^{e}$ with $L^{e}$ being the average BCE loss for
multi-label classification, after applying the sigmoid activation function on
$\boldsymbol{\hat{y}}^{e}$. An overview of the RED-DOT architecture can be
seen in Fig.3.

$[\mathbf{d_{\text{CLS}},d_{F^{v}},d_{F_{\mathcal{E}}}}]=D([CLS;F^{v};F_{\mathcal{E}}])$
(6)
$\mathbf{d}_{F_{\mathcal{E}}}=[\mathbf{d_{F_{\mathcal{E}_{1}}}},\dots,\mathbf{d_{F_{\mathcal{E}_{(M+K)\cdot
2}}}}]$ (7)

(1) Single-stage Learning (SSL): employs Eq.6 and Eq.8 to predict
$\boldsymbol{\hat{y}}^{e}$ in a single stage.

$\hat{y}^{e}=[\textbf{W}_{3}\cdot\text{GELU}(\textbf{W}2\cdot\text{LN}(\mathbf{d_{F_{\mathcal{E}_{i}}}}))],\quad\text{for
}i=1\text{ to }2\cdot(M+K)$ (8)

with $\textbf{W}_{2}\in\mathbb{R}^{1\times 512}$ is a GELU activated fully
connected layer and $\textbf{W}_{3}\in\mathbb{R}^{256\times 1}$.

(2) Single-stage Learning with Guided Attention (SSL+GA): Similar to SSL but
instead of Eq.8, we apply a type of “guided attention” where the $L_{v}$ is
directly applied onto the attention weights. Consider the vector
$\mathbf{d}=[\mathbf{d_{\text{CLS}},d_{F^{v}},d_{F_{\mathcal{E}}}}]$, to
calculate the attention scores, we use Eq.9 and Eq.10 to predict
$\boldsymbol{\hat{y}}^{e}$

$\mathbf{a}=\frac{\mathbf{d}\cdot\mathbf{d^{T}}}{512}$ (9)
$\hat{y}^{e}=\mathbf{a}[<CLS>][-2\cdot(M+K):]$ (10)

where $<CLS>$ represents the position of the classification token, providing a
global representation of attention and $-2\cdot(M+K)$ denotes the last
$2\cdot(M+K)$ items in d, corresponding to the evidence.

(3) Dual-stage Learning (DSL): first employs Eq.6 and Eq. 8. Afterwards,
evidence $\mathcal{E}$ are masked by $\text{MASK}\in\\{0,1\\}^{512\times
2\cdot(M+K)}$ where 0s denote predicted irrelevant and 1s predicted relevant
evidence. During training, we employ teacher enforcing. At the second stage,
we apply MASK onto the evidence and re-process with Transformer $D(\cdot)$
with Eq.11 and then use Eq.5 to predict $\hat{y}^{v}$.

$[\mathbf{d_{\text{CLS}},d_{F^{v}},d_{F_{\mathcal{E}}}}]=D([\text{CLS};F^{v};F_{\mathcal{E}}\odot\text{MASK}])$
(11)

(4) Dual-stage Learning with Guided Attention (DSL+GA): similar to DSL, but in
the first stage it utilizes guided attention (as in Eq.9 and Eq.10) instead of
Eq.8 to predict $\boldsymbol{\hat{y}}^{e}$. The second stage remains identical
to DSL.

(5) Dual-stage Learning with two Transformers (DSL+D2): similar to DSL, but in
the second stage we employ a second identical Transformer encoder $D2(\cdot)$
with
$[\mathbf{d_{\text{CLS}},d_{F^{v}},d_{F_{\mathcal{E}}}}]=D2([\text{CLS};F^{v};F_{\mathcal{E}}\odot\text{MASK}])$.
The first stage remains identical to DSL.

##  4 Experimental Setup

###  4.1 Evaluation Protocol

Our goal is to examine the generalizability of RED-DOT onto real-world
misinformation. For this reason, we train RED-DOT on the NewsCLIPings+ dataset
and evaluate its performance on the Out-Of-Context (OOC) pairs from the VERITE
benchmark. To gather “external evidence” via Google API for VERITE, we
followed the process outlined in [31]; which we make publicly available in
order to facilitate future research and ensure fair comparability across
studies. Our evaluation protocol departs from “In Distribution Validation”
(ID-V) which involves validation and check-pointing on NewsCLIPings and final
testing on VERITE [36]. Instead, we leverage “Out-Of-Distribution Cross
Validation” (OOD-CV) with k-fold cross-validation and check-pointing directly
on a VERITE fold (k-fold=3), while training RED-DOT on the NewsCLIPings+
training set. Partially inspired by [46], we hypothesize that due to
NewsCLIPings+ and VERITE following different distributions -the first
comprising algorithmically created samples and the latter real-world
misinformation- OOD-CV can capture a version of the model with improved
generalizability onto the “out-of-distribution” VERITE. Finally, we report
“Accuracy” on NewsCLIPings+ based on ID-V and “True vs. OOC Accuracy” and the
standard deviation on VERITE based on OOD-CV.

###  4.2 Competing Methods

On NewsCLIPings+, we compare RED-DOT against the Consistency Checking Network
(CCN) [31], the Explainable and Context-Enhanced Network (ECENet) [33] and the
Stance Extraction Network [34]. CCN leverages two attention-based memory
networks for visual and textual reasoning, with ResNet152 and BERT,
respectively, as well as a fine-tuned CLIP (ViT B/32) component for additional
feature extraction. SEN leverages the overall architecture and encoders of CCN
but also extracts the “stance” of external evidence towards the image-text
pair and calculates the “support-refutation score” based on the co-occurrence
of named entities between the claim and the textual evidence. ECENet employs
ResNet50, BERT and CLIP ViT-B/32 for feature extraction on images, texts,
evidence and named entities within a “Coarse- and Fine-grained Attention”
network for intra-modal and cross-modal feature reasoning. Lastly, we also
include a few methods that do not incorporate external evidence, namely: CLIP
[47], DT (Detector Transformer) [16], MNSL (Multimodal Neural Symbolic
Learning) [42] and SSDL (Self-Supervised Distilled Learning) [41]. On VERITE,
we compare RED-DOT against the Detector Transformer (DT) [36] employing
features from CLIP ViT-L/14 after being trained on “R-NESt + CHASMA-D +
NC-t2t”; a combination of three different algorithmically created datasets
with 437,673 samples in total. To ensure comparability, we faithfully
replicate the DT and train it on NewsCLIPings+ without external evidence.
Within our framework, the DT can be expressed as
$D([\text{CLS};F_{I^{v}};F_{T^{v}}])$ with features from CLIP ViT-L/14 and is
trained under the ID-V protocol.

###  4.3 Implementation Details

We train RED-DOT for a maximum of 100 epochs with early stopping at 10 epochs,
using the Adam optimizer and a learning rate of $lr=1e-4$. When tuning
hyperparameters, we consider the following configurations: $t\in\\{4,6\\}$
transformer layers, $z\in\\{128,2048\\}$ for the dimension of the feed-forward
network, and $h\in\\{2,8\\}$ for the number of attention heads. The dropout
rate is set at 0.1 and the batch size to 512. For faithful reproduction of DT,
we follow the implementation details of [36] which uses $lr=5e-5$ and
$t\in\\{1,4\\}$. Finally, to ensure experiment reproducibility, we use a
constant random seed (0) for PyTorch, Python random, and NumPy.

##  5 Experimental Results

###  5.1 Out-of-distribution Cross-Validation

Table 1 showcases the performance of RED-DOT variants, with $M=1$ texts and
$K=1$ images as evidence, when trained and validated under two different
protocols: ID-V or the proposed OOD-CV. We observe that OOD-CV consistently
gives better results than ID-V, with 72.3% and 70.4% mean accuracy,
respectively. These results suggest that OOD-CV is more effective at capturing
a model with better generalizability from algorithmically generated samples
(NewsCLIPings+) to real-world out-of-context misinformation (VERITE). Based on
these findings, we recommend that future research endeavors involving VERITE
adopt the OOD-CV protocol to ensure fairer and more representative model
performance on the dataset. We employ OOD-CV in all following experiments.

Table 1:  Variants of RED-DOT trained under the OOD-CV protocol with M texts and K images as evidence. We report accuracy on VERITE and the standard deviation for OOD-CV (k-fold=3). For $M,K=1$ we also report detection accuracy under ID-V. Underline denotes the best performance per method.  Method | M,K=1 (ID-V) | M,K=1 | M,K=2 | M,K=4
---|---|---|---|---
Baseline | 69.7 |  70.7 (1.5) | 65.9 (2.3) | 66.4 (2.3)
SSL | 70.0 | 71.8 (0.1) |  72.1 (3.7) | 67.2 (1.8)
SSL + GA | 71.2 |  72.6 (1.7) | 71.9 (3.0) | 65.8 (2.7)
DSL + D2 | 69.1 |  71.3 (0.5) | 66.5 (2.6) | 65.3 (0.7)
DSL + GA | 71.3 |  73.6 (1.7) | 69.9 (2.5) | 62.7 (1.7)
DSL | 70.9 |  73.9 (0.5) | 70.0 (2.4) | 64.7 (1.6)
Mean | 70.4 |  72.3 (1.0) | 69.4 (2.8) | 65.3 (1.8)

###  5.2 Impact of multiple ranked evidence

Table 1 also illustrates the impact of leveraging $M,K\in\\{1,2,4\\}$ ranked
texts and images as evidence. We observe that the best performance is yielded
with $M,K=1$, with an average accuracy of 72.3%, while the performance tends
to decline with the inclusion of additional evidence. RED-DOT is explicitly
trained to detect evidence relevance. Expectedly, the inclusion of less
relevant information in $\mathcal{E+}$ with their labels defined as
$\boldsymbol{y}^{e}+$, introduces noise and tends to perplex the network.
Nevertheless, we observe that even RED-DOT-Baseline that does not leverage RED
and, consequently, irrelevant evidence, still demonstrates a reduction in
accuracy with the addition of more information. We can infer that the process
of re-ranking evidence, as described in Section 3.2, can be advantageous
compared to leveraging all collected evidence both in terms of detection
accuracy and decreased computational complexity; since the network has fewer
items to process. We employ $M,K=1$ in all following experiments.

###  5.3 Ablation on Modality Fusion

Table 2 presents the ablation results for different Modality Fusion
configurations within RED-DOT variants. Our findings indicate that the simple
concatenation ($F_{I^{v}};F_{T^{v}}$) results in the lowest average
performance (69.8%) across all RED-DOT variants while employing all fusion
operations ($F^{v}$) produces the highest (72.3%). However, there are
individual instances where alternative fusion operations demonstrate superior
performance. For instance, RED-DOT-DSL attains the highest overall accuracy
while utilizing $F^{v}-\text{``*''}$ (removing the multiplication operation),
yielding 74.2% and surpassing the 73.9% accuracy achieved with $F^{v}$.
Notably, RED-DOT-SSL yields 73.9% without “addition” ($F^{v}-\text{``+''}$)
and 71.8% with $F^{v}$; a +2.9% improvement. Based on these outcomes, we can
conclude that leveraging $F^{v}$ can lead to adequate results, but if optimal
performance is required, it is advisable to experiment with multiple
multimodal fusion configurations.

Table 2:  Ablation with different Modality Fusion combinations. $F^{v}$ followed by ‘-’ signifies removal of one fusion operation. | $F_{I^{v}};F_{T^{v}}$ | $F^{v}$ |  $F^{v}$ \- “-” |  $F^{v}$ \- “+” |  $F^{v}$ \- “*”
---|---|---|---|---|---
Baseline | 66.7 (0.9) | 70.7 (1.5) | 68.3 (1.2) | 70.3 (1.7) |  71.6 (4.1)
SSL | 71.6 (1.2) | 71.8 (0.1) | 73.2 (1.2) |  73.9 (1.1) | 72.9 (0.3)
SSL + GA | 70.6 (0.8) |  72.6 (1.7) | 72.0 (1.5) | 71.6 (2.4) | 70.5 (2.1)
DSL + D2 | 68.8 (1.6) |  71.3 (0.5) | 70.0 (0.8) | 69.8 (1.7) | 69.3 (1.7)
DSL + GA | 69.8 (1.1) |  73.6 (1.7) | 71.2 (2.0) |  73.6 (1.7) | 72.8 (1.1)
DSL | 71.0 (0.4) | 73.9 (0.5) | 71.5 (2.2) | 72.9 (1.1) |  74.2 (1.5)
Mean | 69.8 (1.0) |  72.3 (1.0) | 71.0 (1.5) | 72.0 (1.6) | 71.8 (1.8)
Table 3:  Comparison with the state-of-the-art. All methods are trained on NewsCLIPings+ (“News” for short) with the exception of $\dagger$ denoting “R-NESt + CHASMA-D + NC-t2t” [36]. Here, $M,K+$ and $M,K-$ represent the number of relevant and irrelevant evidence, respectively, provided as inputs to the models.  Method | M,K+ | M,K- | News | VERITE
---|---|---|---|---
CLIP [11] | 0 | 0 | 60.2 | -
DT [16] | 0 | 0 | 65.7 | -
MNSL [42] | 0 | 0 | 68.2 | -
SSDL [41] | 0 | 0 | 71.2 | -
CCN [31] | MAX | 0 | 84.7 | -
SEN [34] | MAX | 0 | 87.1 | -
ECENet [33] | MAX | 0 | 87.7 | -
DT [36] | 0 | 0 | - | 57.5
DT [36] $\dagger$ | 0 | 0 | - | 72.7
RED-DOT-Baseline ($F_{I^{v}};F_{T^{v}}$) | 1 | 0 | 80.8 | 66.7 (0.9)
RED-DOT-Baseline | 1 | 0 | 87.8 | 70.7 (1.5)
RED-DOT-DSL | 1 | 1 | 84.5 |  73.9 (0.5)

###  5.4 Comparative Study

Table 3 presents the comparison between RED-DOT and the current state-of-the-
art on NewsCLIPings+ and VERITE. When Detector Transformer (DT) is trained on
NewsCLIPings+ it yields 57.5% accuracy on VERITE. Integrating external
evidence in RED-DOT-Baseline can significantly improve performance, reaching
66.7% (+16.0%) with simple concatenation ($F_{I^{v}};F_{T^{v}}$) and 70.7%
(+22.3%) with modality fusion $F^{v}$ (Eq.3). By leveraging proposed Relevant
Evidence Detection module, RED-DOT (specifically the DSL variant) can further
improve performance up to 73.9%, a notable +28.5% improvement over DT and
+4.5% over RED-DOT-Baseline. If we consider the ablation of modality fusion
(Table 2), the best performances of RED-DOT-DSL and RED-DOT-Baseline were
yielded by $F^{v}$-“*”, with 74.2% and 71.6% respectively; which translates to
a 3.5% improvement. These findings validate our hypothesis that leveraging
multitask learning with both “verdict prediction’ and “relevant evidence
detection” can improve the overall detection accuracy on real-world
misinformation. When the DT is trained on a larger dataset (“R-NESt + CHASMA-D
+ NC-t2t”) we observe a reduction in the performance gap, with RED-DOT
outperforming it by 1.7%. Based on these results, we deem highly likely that
collecting evidence and training RED-DOT on a larger dataset has the potential
to further improve performance. However, we do not explore this here, since
NewsCLIPings+ is the currently largest publicly available dataset that
provides external information as evidence.

When considering all modality fusion ablations, Table 2 highlights the top
three performances, with DSL leading at 74.2%, followed by SSL at 73.9%, and
DSL+GA at 73.6%. Our interpretation suggests that the dual-stage architecture
holds an advantage over SSL by masking irrelevant evidence and employing
teacher enforcing. Furthermore, the Verdict Prediction network (Eq.5) in both
DSL and SSL provides additional pattern recognition capabilities that
contribute to enhanced performance when compared to the “guided attention”.

With respect to NewsCLIPings+, we observe that all methods leveraging external
information significantly outperform those that do not [11, 16, 42, 41].
Moreover, RED-DOT-Baseline yields 87.8% on NewsCLIPings+, outperforming CCN
(84.7%), SEN (87.1%) and ECENet (87.7%); while only leveraging a single piece
of evidence per modality and without requiring fine-tuning of the CLIP
ViT-B/32 encoder, nor multiple encoders (e.g., BERT and ResNet on top of CLIP)
nor additional features (e.g., named entities). These results highlight the
advantage of the proposed “Evidence re-ranking” and “Modality Fusion” modules.
RED-DOT-DSL (84.5%) competes with CCN (84.7%) but is outperformed by its
“Baseline” counterpart. This can be explained by the fact that RED-DOT-
Baseline is exclusively trained on Verdict Prediction while RED-DOT-DSL is
trained for two tasks at the same time, Verdict Prediction and Relevant
Evidence Detection. As a result, the model’s “focus” is split into the two
tasks. Nevertheless, we observe that better performance on the NewsCLIPings+
test set - consisting of algorithmically created negative pairs- does not
necessarily translate to improved performance on the annotated misinformation
of VERITE. We deem the latter to be the more important metric, since it gives
a better indication of the model’s ability to detect real world
misinformation.

Figure 4: Inference by RED-DOT variants: DSL and DSL+GA on samples from
NewsCLIPings+. We report the Attention scores assigned by each method.
“Relevance Ground truth” is set to [0, 0, 1, 1] for simplicity, regarding
[$T^{e}-$, $I^{e}-$, $T^{e}+$, $I^{e}+$], respectively.

###  5.5 Qualitative Analysis

Figure 4 depicts the inference process on NewsCLIPings+ test set samples when
using two RED-DOT variants, DSL and DSL+GA. We observe that DSL+GA tends to
assign higher attention scores the “relevant” items. For instance, in the
“Truthful” pair (left), the evidence retrieval and re-ranking module was able
to retrieve the same image $I^{e}+$ as $I^{v}+$ and DSL+GA correctly exhibits
higher attention scores on $I^{e}+$ (0.182) followed by $T^{e}+$ (0.109) and
lower on irrelevant evidence (0.100). On the contrary, the DSL variant shows
lower attention scores on $I^{e}+$ (0.081) and $I^{e}+$ (0.078) and displays
higher attention scores on the irrelevant evidence (0.111). In the
“Misinformation” pair, again, DSL+GA assigns higher attention scores to the
relevant evidence over irrelevant ones, while DSL only assigns a higher score
on $T^{e}+$ and the same score (0.08) on all other items, including $I^{e}+$.
If the DSL model were to be deployed and used by general users or fact-
checkers, it might be challenging to trust the model, even if its predictions
were accurate, as it consistently assigns higher attention to irrelevant
evidence. In contrast, DSL+GA assigns higher attention scores to relevant
texts $T^{e}+$ and even higher to the relevant images $I^{e}+$, thereby
enhancing interpretability. In a more systematic way, Fig.5 illustrates the
distributions of attention scores for both relevant and irrelevant evidence,
as assessed by DSL and DSL+GA, based on a sample size of 1000. Notably, we
observe that the median attention scores assigned by DSL to relevant evidence
are lower than those for irrelevant evidence, which is not ideal for
interpretability. On the other hand, DSL+GA exhibits two distributions with a
similar median; however, irrelevant evidence follow a negatively skewed
distribution, while relevant evidence follow a positively skewed distribution.
Drawing from these observations, we can deduce that the inclusion of guided
attention, while not offering optimal interpretability, it holds the potential
to enhance it.

Figure 5: Distributions of attention scores assigned by DSL and DSL+GA per
“relevant” and “irrelevant” evidence.

##  6 Discussion and Limitations

Despite recent and notable improvements in the field of automated Multimodal
Fact-Checking, it is essential to discuss certain limitations of our research
and the field more broadly. Following previous studies [31, 34, 33], we
conducted our experiments on NewsCLIPings+, which, albeit being the sole
publicly available dataset with external evidence and no “leaked evidence”
concerns, comes with certain constraints. Specifically, the dataset’s size,
consisting of 85,000 samples, may not encompass the full spectrum of diversity
concerning historical events, prominent figures, and other crucial contextual
elements pertinent to misinformation. Moreover, NewsCLIPings+ consists of
algorithmically generated “out-of-context” instances lacking the complexity
and nuances present in real-world misinformation scenarios. As evidenced in
our experiments, high accuracy on NewsCLIPings+ does not necessarily translate
to high accuracy on real-world misinformation (VERITE). For this reason, we
suggest that future studies also evaluate the performance of their models on
real-world misinformation. Additionally, NewsCLIPings+ exclusively focuses on
“re-purposed images” or “out-of-context misinformation”, omitting categories
like “miscaptioned images” [36]. Subsequent research endeavors should aim at
the collection of more extensive and more diverse datasets, encompassing
various forms of multimodal misinformation.

While our “evidence re-ranking” module yielded satisfactory results, it’s
essential to acknowledge that the external information used as evidence is
retrieved from search engines, which might introduce noise and irrelevant
data. These limitations may impact the realization of the full potential of
RED-DOT and similar methods. Hence, future research should explore improved
methods for gathering, filtering, and assessing the relevance of external
information employed as evidence. Our work represents a significant first step
towards this direction by providing a novel methodological framework for
assessing the relevance of external evidence.

On a more technical level, future research could explore the use of larger
multimodal foundation models, including ALBEF [48], BLIP2 [49], LLaVA [50] or
the Meta-Transformer [51]. Moreover, our “Modality Fusion” module employs
element-wise operations on the image-text pair, but these operations are not
extended to external evidence. Inquiry into various modality fusion methods
[52, 53, 21] and application to both external evidence and image-text pairs
could further improve performance.

##  7 Conclusions

In this study we address the challenge of evidence-based Multimodal Fact-
Checking (MFC) and incorporate Relevant Evidence Detection (RED) as part of
the process, where the model, first has to determine which pieces of evidence
are relevant to, support or refute, the claim under verification and then
proceed to assess its veracity. We conduct extensive ablation and comparative
experiments to show that the proposed Relevant Evidence Detection Directed
Transformer (RED-DOT) is capable of outperforming its counterparts that are
not optimized for RED on VERITE. Moreover, it competes and even outperforms
the state-of-the-art on NewsCLIPings+ without requiring numerous evidence,
multiple backbone encoders or additional features. Finally, leveraging “guided
attention” -where the loss function is directly applied onto the Transformer’s
attention weights- has the potential to improve the model’s interpretability.

##  8 Acknowledgements

This work is partially funded by the project “vera.ai: VERification Assisted
by Artificial Intelligence” under grant agreement no. 101070093.

## References

  * [1] Femi Olan, Uchitha Jayawickrama, Emmanuel Ogiemwonyi Arakpogun, Jana Suklan, and Shaofeng Liu.  Fake news on social media: the impact on society.  Information Systems Frontiers, pages 1–16, 2022.
  * [2] W Lance Bennett and Steven Livingston.  The disinformation order: Disruptive communication and the decline of democratic institutions.  European journal of communication, 33(2):122–139, 2018.
  * [3] Michele Cantarella, Nicolò Fraccaroli, and Roberto Volpe.  Does fake news affect voting behaviour?  Research Policy, 52(1):104628, 2023.
  * [4] Jon Roozenbeek, Claudia R Schneider, Sarah Dryhurst, John Kerr, Alexandra LJ Freeman, Gabriel Recchia, Anne Marthe Van Der Bles, and Sander Van Der Linden.  Susceptibility to misinformation about covid-19 around the world.  Royal Society open science, 7(10):201199, 2020.
  * [5] Israel Junior Borges Do Nascimento, Ana Beatriz Pizarro, Jussara M Almeida, Natasha Azzopardi-Muscat, Marcos André Gonçalves, Maria Björklund, and David Novillo-Ortiz.  Infodemics and health misinformation: a systematic review of reviews.  Bulletin of the World Health Organization, 100(9):544, 2022.
  * [6] José Gamir-Ríos, Raquel Tarullo, Miguel Ibáñez-Cuquerella, et al.  Multimodal disinformation about otherness on the internet. the spread of racist, xenophobic and islamophobic fake news in 2020.  Anàlisi, pages 49–64, 2021.
  * [7] Hilda Ruokolainen and Gunilla Widén.  Conceptualising misinformation in the context of asylum seekers.  Information Processing & Management, 57(3):102127, 2020.
  * [8] Jiawei Zhou, Yixuan Zhang, Qianni Luo, Andrea G Parker, and Munmun De Choudhury.  Synthetic lies: Understanding ai-generated misinformation and evaluating algorithmic and human solutions.  In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–20, 2023.
  * [9] Danni Xu, Shaojing Fan, and Mohan Kankanhalli.  Combating misinformation in the era of generative ai models.  In Proceedings of the 31st ACM International Conference on Multimedia, pages 9291–9298, 2023.
  * [10] Shivangi Aneja, Chris Bregler, and Matthias Nießner.  Cosmos: Catching out-of-context image misuse using self-supervised learning.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 14084–14092, 2023.
  * [11] Grace Luo, Trevor Darrell, and Anna Rohrbach.  Newsclippings: Automatic generation of out-of-context multimodal media.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6801–6817, 2021.
  * [12] Sakshini Hangloo and Bhavna Arora.  Combating multimodal fake news on social media: methods, datasets, and future perspective.  Multimedia systems, 28(6):2391–2422, 2022.
  * [13] Christina Boididou, Stuart E Middleton, Zhiwei Jin, Symeon Papadopoulos, Duc-Tien Dang-Nguyen, Giulia Boato, and Yiannis Kompatsiaris.  Verifying information with multimedia content on twitter: a comparative study of automated approaches.  Multimedia tools and applications, 77:15545–15571, 2018.
  * [14] Kai Nakamura, Sharon Levy, and William Yang Wang.  Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection.  In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6149–6157, 2020.
  * [15] Ayush Jaiswal, Ekraam Sabir, Wael AbdAlmageed, and Premkumar Natarajan.  Multimedia semantic integrity assessment using joint embedding of images and text.  In Proceedings of the 25th ACM international conference on Multimedia, pages 1465–1471, 2017.
  * [16] Stefanos-Iordanis Papadopoulos, Christos Koutlis, Symeon Papadopoulos, and Panagiotis Petrantonakis.  Synthetic misinformers: Generating and combating multimodal misinformation.  In Proceedings of the 2nd ACM International Workshop on Multimedia AI against Disinformation, pages 36–44, 2023.
  * [17] Giscard Biamby, Grace Luo, Trevor Darrell, and Anna Rohrbach.  Twitter-comms: Detecting climate, covid, and military multimodal misinformation.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1530–1549, 2022.
  * [18] Eric Müller-Budack, Jonas Theiner, Sebastian Diering, Maximilian Idahl, and Ralph Ewerth.  Multimodal analytics for real-world news using measures of cross-modal entity consistency.  In Proceedings of the 2020 International Conference on Multimedia Retrieval, pages 16–25, 2020.
  * [19] Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, and Vasudeva Varma.  Mvae: Multimodal variational autoencoder for fake news detection.  In The world wide web conference, pages 2915–2921, 2019.
  * [20] Yaqing Wang, Fenglong Ma, Zhiwei Jin, Ye Yuan, Guangxu Xun, Kishlay Jha, Lu Su, and Jing Gao.  Eann: Event adversarial neural networks for multi-modal fake news detection.  In Proceedings of the 24th acm sigkdd international conference on knowledge discovery & data mining, pages 849–857, 2018.
  * [21] Chuanming Yu, Yinxue Ma, Lu An, and Gang Li.  Bcmf: A bidirectional cross-modal fusion model for fake news detection.  Information Processing & Management, 59(5):103063, 2022.
  * [22] Shivangi Singhal, Rajiv Ratn Shah, Tanmoy Chakraborty, Ponnurangam Kumaraguru, and Shin’ichi Satoh.  Spotfake: A multi-modal framework for fake news detection.  In 2019 IEEE fifth international conference on multimedia big data (BigMM), pages 39–47. IEEE, 2019.
  * [23] Shivangi Singhal, Tanisha Pandey, Saksham Mrig, Rajiv Ratn Shah, and Ponnurangam Kumaraguru.  Leveraging intra and inter modality relationship for multimodal fake news detection.  In Companion Proceedings of the Web Conference 2022, pages 726–734, 2022.
  * [24] Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos.  A survey on automated fact-checking.  Transactions of the Association for Computational Linguistics, 10:178–206, 2022.
  * [25] William Yang Wang.  “liar, liar pants on fire”: A new benchmark dataset for fake news detection.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426, 2017.
  * [26] Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen.  Multifc: A real-world multi-domain dataset for evidence-based fact checking of claims.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4685–4697, 2019.
  * [27] James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal.  Fever: a large-scale dataset for fact extraction and verification.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 809–819, 2018.
  * [28] Rami Aly, Zhijiang Guo, Michael Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal.  Feverous: Fact extraction and verification over unstructured and structured information.  arXiv preprint arXiv:2106.05707, 2021.
  * [29] Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin-Hee Cho, and Lifu Huang.  End-to-end multimodal fact-checking and explanation generation: A challenging dataset and models.  In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2733–2743, 2023.
  * [30] S Suryavardan, Shreyash Mishra, Parth Patwa, Megha Chakraborty, Anku Rani, Aishwarya Reganti, Aman Chadha, Amitava Das, Amit Sheth, Manoj Chinnakotla, et al.  Factify 2: A multimodal fake news and satire news dataset.  arXiv preprint arXiv:2304.03897, 2023.
  * [31] Sahar Abdelnabi, Rakibul Hasan, and Mario Fritz.  Open-domain, content-based, multi-modal fact-checking of out-of-context images via online resources.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14940–14949, 2022.
  * [32] Max Glockner, Yufang Hou, and Iryna Gurevych.  Missing counter-evidence renders nlp fact-checking unrealistic for misinformation.  In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5916–5936, 2022.
  * [33] Fanrui Zhang, Jiawei Liu, Qiang Zhang, Esther Sun, Jingyi Xie, and Zheng-Jun Zha.  Ecenet: Explainable and context-enhanced network for muti-modal fact verification.  In Proceedings of the 31st ACM International Conference on Multimedia, pages 1231–1240, 2023.
  * [34] Xin Yuan, Jie Guo, Weidong Qiu, Zheng Huang, and Shujun Li.  Support or refute: Analyzing the stance of evidence to detect out-of-context mis-and disinformation.  arXiv preprint arXiv:2311.01766, 2023.
  * [35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  * [36] Stefanos-Iordanis Papadopoulos, Christos Koutlis, Symeon Papadopoulos, and Panagiotis C Petrantonakis.  Verite: A robust benchmark for multimodal misinformation detection accounting for unimodal bias.  arXiv preprint arXiv:2304.14133, 2023.
  * [37] Muhammad F Mridha, Ashfia Jannat Keya, Md Abdul Hamid, Muhammad Mostafa Monowar, and Md Saifur Rahman.  A comprehensive review on fake news detection with deep learning.  IEEE Access, 9:156151–156170, 2021.
  * [38] Md Shohel Rana, Mohammad Nur Nobi, Beddhu Murali, and Andrew H Sung.  Deepfake detection: A systematic literature review.  IEEE Access, 2022.
  * [39] Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fabrizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, Preslav Nakov, et al.  A survey on multimodal disinformation detection.  In Proceedings of the 29th International Conference on Computational Linguistics, pages 6625–6643. International Committee on Computational Linguistics, 2022.
  * [40] Ekraam Sabir, Wael AbdAlmageed, Yue Wu, and Prem Natarajan.  Deep multimodal image-repurposing detection.  In Proceedings of the 26th ACM international conference on Multimedia, pages 1337–1345, 2018.
  * [41] Michael Mu, Sreyasee Das Bhattacharjee, and Junsong Yuan.  Self-supervised distilled learning for multi-modal misinformation identification.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 2819–2828, 2023.
  * [42] Yizhou Zhang, Loc Trinh, Defu Cao, Zijun Cui, and Yan Liu.  Detecting out-of-context multimodal misinformation with interpretable neural-symbolic model.  arXiv preprint arXiv:2304.07633, 2023.
  * [43] S Suryavardan, Shreyash Mishra, Megha Chakraborty, Parth Patwa, Anku Rani, Aman Chadha, Aishwarya Reganti, Amitava Das, Amit Sheth, Manoj Chinnakotla, et al.  Findings of factify 2: multimodal fake news detection.  arXiv preprint arXiv:2307.10475, 2023.
  * [44] Christos Koutlis, Manos Schinas, and Symeon Papadopoulos.  Memefier: Dual-stage modality fusion for image meme classification.  In Proceedings of the 2023 ACM International Conference on Multimedia Retrieval, pages 586–591, 2023.
  * [45] Gokul Karthik Kumar and Karthik Nandakumar.  Hate-clipper: Multimodal hateful meme classification based on cross-modal interaction of clip features.  In Proceedings of the Second Workshop on NLP for Positive Impact (NLP4PI), pages 171–183, 2022.
  * [46] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al.  Wilds: A benchmark of in-the-wild distribution shifts.  In International Conference on Machine Learning, pages 5637–5664. PMLR, 2021.
  * [47] Chiara Longoni, Andrey Fradkin, Luca Cian, and Gordon Pennycook.  News from generative artificial intelligence is believed less.  In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 97–106, 2022.
  * [48] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  Advances in neural information processing systems, 34:9694–9705, 2021.
  * [49] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  arXiv preprint arXiv:2301.12597, 2023.
  * [50] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning.  arXiv preprint arXiv:2304.08485, 2023.
  * [51] Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue.  Meta-transformer: A unified framework for multimodal learning.  arXiv preprint arXiv:2307.10802, 2023.
  * [52] Jing Jing, Hongchen Wu, Jie Sun, Xiaochang Fang, and Huaxiang Zhang.  Multimodal fake news detection via progressive fusion networks.  Information processing & management, 60(1):103120, 2023.
  * [53] Yang Wu, Pengwei Zhan, Yunjian Zhang, Liming Wang, and Zhen Xu.  Multimodal fusion with co-attention networks for fake news detection.  In Findings of the association for computational linguistics: ACL-IJCNLP 2021, pages 2560–2569, 2021.


# Misalign, Contrast then Distill:
Rethinking Misalignments in Language-Image Pretraining

Bumsoo Kim Yeonsik Jo Jinhyung Kim Seunghwan Kim
LG AI Research Correspondence to<EMAIL_ADDRESS>
###### Abstract

Contrastive Language-Image Pretraining has emerged as a prominent approach for
training vision and text encoders with uncurated image-text pairs from the
web. To enhance data-efficiency, recent efforts have introduced additional
supervision terms that involve random-augmented views of the image. However,
since the image augmentation process is unaware of its text counterpart, this
procedure could cause various degrees of image-text misalignments during
training. Prior methods either disregarded this discrepancy or introduced
external models to mitigate the impact of misalignments during training. In
contrast, we propose a novel metric learning approach that capitalizes on
these misalignments as an additional training source, which we term “Misalign,
Contrast then Distill (MCD)”. Unlike previous methods that treat augmented
images and their text counterparts as simple positive pairs, MCD predicts the
continuous scales of misalignment caused by the augmentation. Our extensive
experimental results show that our proposed MCD achieves state-of-the-art
transferability in multiple classification and retrieval downstream datasets.

##  1 Introduction

Recent advances in deep learning have shown that image representations trained
with large-scale uncurated natural language supervision shows powerful
transferability to various downstream tasks [14, 32]. A predominant paradigm
in vision–language pre-training is to use a simple contrastive loss that makes
the embedding of an image and its matching text description (positive pair)
more similar to each other than other arbitrary image–text pairs (negative
pairs) [29]. To achieve a more data-efficient training, following works
actively capitalized on image random augmentation by: (i) joining
language–image pretraining objectives with vision self-supervision terms [3,
5] between the augmented images [27, 23, 18] and (ii) involving more pairs of
positive/negative supervisions between the augmented images and their original
text description [23, 18].

Figure 1: Conceptual illustration of contrastive language–image objectives of
previous works (_i.e_., InfoNCE) and our MCD for (a) augmentation that doesn’t
harm the correspondence with its description and (b) augmentation that does.
Previous works either disregard these misalignments [23] or leverage external
models [18, 9] to mitigate their impact. On the other hand, MCD uses the
continuous degree of misalignments caused by random image augmentation as a
useful source for training various levels of alignments between images and
their text descriptions.

However, since the random image augmentation process is unaware of its
corresponding text, it often results in the augmented image view to be
misaligned with its description (see (b) in Fig.1). These misalignments behave
as noisy training signals for the contrastive loss in language–image
pretraining, thus causing performance degradation if not properly attended
[27]. To mitigate this issue, recent works have used additional augmentation
embeddings [18] or heavy external off-the-shelf object detectors and summary
extractors [9] to match the alignment during training. Though being
straightforward and showing strong performance, these works are limited in
that they add unnecessary burden in both training and inference.

Based on this observation, we start with a simple question: “Instead of
treating misalignments as noise to eliminate, can we rather harness them as a
training source for language-image pretraining?”. To this end, we propose MCD
(_i.e_., Misalign, Contrast then Distill), a novel training framework that
leverages the various levels of misalignments between random augmented images
and its text description during training.

MCD consists of three steps (see Fig 2 for an overview illustration of MCD):
First, we conduct random augmentation on the image that causes various levels
of misalignments (or not at all) with its text counterpart (Misalign). Then,
we project all the participants (image, text, and augmented image) into an
unified multimodal space, and learn the distance between all the image–text
pairs with a contrastive objective (Contrast). Finally, we use a teacher-
student network where the student learns from the “soft” distance between the
text–original image (_i.e_., $D(\bar{I},T)$ in Fig 2) and text–augmented image
(_i.e_., $D(\bar{I}^{\prime},T)$ in Fig 2) of the momentum teacher with a log-
ratio loss (Distill). continuous scale of misalignment to the student model,
enabling the student to learn from the various levels of misalignment that
occur from the random augmentation during training time.

Our contribution of this paper is threefold:

  * •

We propose MCD, a novel training framework where we learn the continuous level
of misalignment as a source for contrastive language-image pretraining.

  * •

Our MCD outperforms state-of-the-art models across various single/multi-modal
downstream datasets without adding additional parameters for inference or
using external models to force the alignment.

  * •

We propose three distillation strategies leveraging misalignments: i)
misalignment between positive pairs, ii) misalignment between negative pairs,
and iii) misalignment between noisy pairs. Extensive experiments show that all
three strategies positively contributes to our final performance.

Figure 2: Overview of MCD. Our MCD consist of three steps: (i) text-agnostic
random augmentation of the image that causes various levels of misalignment
with the corresponding text, (ii) learning the distance between the
image/augmented image and text with contrastive objectives then (iii) distill
the log-ratio of the image-text distance between the original and the
augmented image.

##  2 Related Work

Here, we present brief review of multi-modal representation learning,
especially vision-language pre-training.

###  2.1 Vision-language Pre-training

Vision-Language Pre-training (VLP) trains a multi-modal model to learn joint
representation of visual and textual information that can be transfer to
various vision–language downstream tasks. The success of VLP primarily relies
on large-scale datasets which contains images and their corresponding
descriptions, enabling the model to understand the semantic relationship
between the pairs. VLP includes two different group of models: 1) single-
stream models [20, 6, 13, 22, 21, 25, 36] which process an image and its
associated text information in a shared backbone network and 2) dual-stream
models [14, 32] which has two independent backbone networks for processing
each modality. In this work, we focuses on Contrastive Language Image
Pretraining (CLIP [32]), which is a type of dual-stream models trained with
image-text contrastive loss where the image and its matching text description
in the dataset as a positive pair and other unrelated pairs in a batch as
negative pairs.

###  2.2 Misalignments between Image-Text Pairs

There are two different sources that cause misalignments in image-text pairs
for VLP: misalignment that naturally occurs in image-text paired datasets, and
misalignments caused by random image augmentation.

#### Misalignment in Image-Text Pairs.

Large scale image-text paired datasets for VLP are usually collected from the
web thus can contain uncurated and noisy pairs which have weak relations. This
inevitably incurs misalignment between positive image-text pairs in the
dataset misleading naïve contrastive language image objective. Previous
studies [19, 24, 1] have attempted to address the problem by knowledge
distillation [11] of soft image-text alignment matrix from momentum teacher
network to the student network via KL divergence loss. Our proposed method
stands apart from previous approaches due to its element-wise log-ratio loss
for distillation. Element-wise loss lessens the dependence on training
hyperparameters like batch size and temperature for KL loss. Furthermore, it
enables the model to harness the various levels of individual misalignments of
each sample from random augmentation or label noise.

#### Misalignment by Augmentations.

View-based self-supervised learning, in which models are trained to represent
views or augmentations of the same image similarly, has yielded strong results
across a variety of different formulations. Consecutive work of CLIP (i.e.,
SLIP [27]) initially introduced supervision between random augmented image
views (_e.g_., cropping, gray-scale, jittering, gaussian blur, horizontal
flipping, etc.). Since only image-image supervision was given in a separately
embedded space, augmentation was not a factor for misalignment. To include
more positive/negative pairs for the image-text contrastive objective,
following works [23, 18] also introduced InfoNCE loss between the augmented
views and the text pairs. However, since the text description is unaware of
the random augmentations, there is a high chance that misalignments occur
during training. SLIP shows that naïve application of contrastive loss to
these misaligned pairs results in suboptimal performance. Previous work have
either ignored this misalignment [23] or addressed this issue with an
additional encoder that encodes the one-hot information of which augmentation
has been applied during training [18]. PyramidCLIP [9] addressed this issue
with external off-the-shelf object detectors and summary extractors. HiCLIP
[10] captured the hierarchical nature of high-level on unsupervised manner
with tree Transformer [39]. MCD incorporate the misalignment information as a
source of training via novel log-ratio loss without introducing any external
module.

##  3 Preliminary

In our preliminary, we revisit the basic form of Contrastive Language-Image
Pretraining (_i.e_., CLIP [32]). CLIP features a dual-encoder architecture
where the image encoder $f_{I}$ and text encoder $f_{T}$ are jointly trained
with a contrastive objective $\mathcal{L}_{\text{CLIP}}$.

#### InfoNCE Loss

Given $N$ image-text pairs $\\{(x_{i}^{I},x_{i}^{T})\\}_{i=1}^{N}$, we define
a similarity matrix $S$ whose $i$-th row and the $j$-th column is the cosine
similarity between the projected representations of the $i$-th text $T_{i}$
and the $j$-th image $I_{j}$ (_i.e_., $T_{i}=f_{T}(x^{T}_{i})$,
$I_{j}=f_{I}(x^{I}_{j})$), written as:

$S_{ij}=\mbox{sim}(T_{i},I_{j}),$ (1)

where sim$(\cdot,\cdot)$ is cosine similarity. In CLIP [32], the encoded image
features $I$ and text features $T$ are projected to the same dimension where
the embeddings for matching image-text pairs are pulled together while
embeddings for arbitrary pairs are pushed apart with the InfoNCE loss [29].
Given the similarity matrix $S$, the InfoNCE loss $\mathcal{L}_{N}$ is
rewritten as:

$\mathcal{L}_{N}(S)=-\frac{1}{N}\sum_{i=1}^{N}\log{\frac{\exp{\big{(}S_{ii}/\tau\big{)}}}{\sum_{j=1}^{N}\exp{\big{(}S_{ij}/\tau\big{)}}}},$
(2)

where $\tau$ is a learnable temperature variable. The overall loss of clip
$\mathcal{L}_{\text{CLIP}}$ is written as:

$\mathcal{L}_{\text{CLIP}}=\frac{1}{2}\bigg{(}\mathcal{L}_{N}(S)+\mathcal{L}_{N}(S^{T})\bigg{)}.$
(3)

#### Augmentations and Misalignment.

To include more positive/negative pairs for the contrastive objective,
following works [23, 18] introduced InfoNCE loss between the random augmented
image views and the text description for the original image. Let $\mathcal{A}$
a function for random image augmentation that includes randomly applying
cropping, gray-scale, jittering, gaussian blur, horizontal flipping following
SimCLR [3]. $I_{j^{\prime}}$ is the encoded image feature of augmented view of
$j$-th image(_e.g_. $I_{j^{\prime}}=f_{I}(\mathcal{A}(x_{j}^{I}))$). Let
$S_{ij}^{\prime}=\text{sim}(T_{i},I_{j^{\prime}})$ represent the similarity
between the augmented view of $j$-th image and $i$-th text. We denote the
matrix of these similarity as $S^{\prime}$ for concise notation. Then, InfoNCE
loss between the augmented image and the text is

$\mathcal{L}_{\text{CLIP}}^{\prime}=\frac{1}{2}\big{(}\mathcal{L}_{N}(S^{\prime})+\mathcal{L}_{N}(S^{\prime
T})\big{)}.$ (4)

As the random augmentation function $\mathcal{A}$ is independent to the
corresponding text description, the augmented view $I_{j^{\prime}}$ is likely
to exhibit misalignment with text $T_{i}$. This hypothesis is consistent with
the finding of SLIP [27], which demonstrated that introducing augmentation
(particularly resize crop, and flip) to CLIP actually resulted in a
performance decrease. Previous works [27, 23] have sidestepped the utilization
of augmented view in CLIP by substituting infoNCE with self-supervised
learning loss(_e.g_. SimCLR [3]) between images. These approaches have
limitation in fully capturing the essence of multi-modal learning.

##  4 Method

In this section, we introduce MCD (Misalign, Contrast then Distill), a novel
training framework for language–image pretraining using misalignments as
continuous labels for learning the distance between image–text pairs.

###  4.1 Misalign

The first step of MCD is to apply text-agnostic random image augmentations
(_e.g_., random crop, random flip, grayscale, etc.) to create various levels
of misalignments between the images and its description. Here, we elaborate on
three distinct scenarios of misalignment that can arise during the
augmentation process, hindering the contrastive loss to learn proper distance
between image–text pairs: (i) Text-agnostic random augmentation can cause
misalignments in positive image-text pairs. (ii) The random-augmentation can
mistakenly cause positive signals between negative pairs. (iii) Misalignments
can already exist innately within the original image-text pair. A detailed
illustration of each cases are provided in Fig 3.

###  4.2 Contrast

In the second step, MCD initially learns the distance metric between
image–text pairs with a contrastive objective. Motivated by [18], we project
both image and text modalities to a unified space and use all the positive
pairs and negative pairs of both modalities. For an $i$-th embedding $z_{i}$
in a batch of embeddings $\\{z_{i}\\}_{i=1}^{3N}$ that includes $N$ image
samples, $N$ text samples, and $N$ random augmented image samples, let
$\mathcal{P}_{i}$ and $\mathcal{N}_{i}$ each denote the set of all positive
sample indices of the $i$-th sample including $i$ itself and the set of all
negative sample indices of the $i$-th sample. Then, the contrastive loss for
the multiple positives and multiple negatives for sample $i$ can be written as

$\small\mathcal{L}^{C}_{i}=\mathbb{E}_{p\in\mathcal{P}_{i}}\left[-\log\frac{\text{sim}(z_{i},z_{p})}{\text{sim}(z_{i},z_{p})+\sum_{n\in\mathcal{N}_{i}}\text{sim}(z_{i},z_{n})}\right].$
(5)

However, without encoding augmentation information the image-text contrastive
loss is prone to the three aforementioned issues. We address these three
issues with a teacher-student model where the continuous distance between the
image–text and augmented image–text of the teacher model is distilled to the
student model.

Figure 3: MCD loss for various case of misalignments that occur when applying
random image augmentation in language–image pretraining with. In this paper,
we elaborate three scenarios: (a) misalignment caused by augmenting the image
of an original positive pair, which is addressed by $\mathcal{L}^{\text{pos}}$
(b) augmentation that mistakenly cause positive alignment between negative
pairs (c) misalignment that already exists within the dataset.  Algorithm 1
MCD Pseudocode

⬇

# fm, fs: image encoders (teacher, student)

# ft: text encoder

# fa: random-augmentation function

# N : batch size

def D(sim): # cosine sim -> L2 distance

return 2 - 2 * sim + 1e-6

def MCD(img, txt):

# image, text encodings

aug = fa(img)

# normalized projection embeddings

zm, zs, zt = fm(img), fs(img), ft(txt)

zam, zas = fm(aug), fs(aug) # misalign

# distance (contrast)

di_t, di_s = D(zm @ zt.T), D(zs @ zt.T)

da_t, da_s = D(zam @ zt.T), D(zas @ zt.T)

# distill

pos_l, neg_l, noise_l = 0, 0, 0

for i in range(N):

lr_s1 = log(da_s[i,i]/di_s[i,i])

lr_t1 = log(da_t[i,i]/di_t[i,i])

# positive pairs

pos_l += abs(lr_s1 - lr_t1) / N

for j in range(N):

if i==j: continue

lr_s2 = log(da_s[i,j]/di_s[i,i])

lr_t2 = log(da_t[i,j]/di_t[i,i])

# negative pairs

neg_l += abs(lr_s2 - lr_t2)/(N*(N-1))

lr_s3 = log(di_s[j,j]/di_s[i,i])

lr_t3 = log(di_t[j,j]/di_t[i,i])

# noisy pairs

noise_l += abs(lr_s3 - lr_t3)/(N*(N-1))

return pos_l + neg_l + noise_l

###  4.3 Distill

Knowledge distillation, introduced by Hinton et al. [11], is a learning
paradigm where we train the student network to mimic the “soft” labels
predicted from the teacher network. In MCD, we train the student network with
the continuous image–text distance predicted by the teacher network.
Psuedocode for the distillation in MCD is provided in Algorithm 1.

#### Log-Ratio Loss for Image–Text Distance.

Given a student $f_{I}$ and a momentum teacher $\bar{f}_{I}$, we propose to
use log-ratio loss [15] on image-text similarities that aims to approximate
the ratio of similarity distances by the ratio of image-text misalignments in
the learned embedding space. Specifically, we define the loss function as

$\ell_{lr}(i,j;\alpha,\beta)=\biggl{\|}\log{\frac{D\bigl{(}I_{\alpha},T_{\beta}\bigl{)}}{D\bigl{(}I_{i},T_{j}\bigl{)}}}-\log{\frac{D\bigl{(}\bar{I}_{\alpha},T_{\beta}\bigl{)}}{D\bigl{(}\bar{I}_{i},T_{j}\bigl{)}}}\biggl{\|}_{1},$
(6)

where $\|\cdot\|_{1}$ is $\ell_{1}$ loss, $D(\cdot,\cdot)$ is distance
function. We utilize Euclidean distance between the projected representations
as distance function. Since the embedded vectors comprising the image
$I_{i},\bar{I}_{i}$ and text $T_{j}$ are L2-normalized, Euclidean distance
operates as a proxy for cosine similarity $D(I_{i},T_{j})=2(1-S_{ij})$. This
log-ratio loss approximates the degree of misalignment, which is measured as
the ratio of two image-text pairs. By leveraging this measure, we aim to
promote a coherent and continuous embedding space. Accordingly, our student
encoder is trained under the guidance of the momentum teacher with the
incorporation of this degree of misalignment. By establishing these pairs of
log-ratio, we enable the handling of diverse forms of misalignment. We present
three distinct index setups that correspond to different types of
misalignment.

#### Misalignment in Positive pairs.

First, we define distance pair for misalignment between the original image-
text pair. Let $i^{\prime}$ denote the index of augmented image sample. On
Eq.(5), $i^{\prime}$-th image sample and $i$-th text sample serve as positive
pair. However, random augmentation can occasionally transform positive pair
into negative pair. To account for such transformations, we intend to utilize
the log-ratio between original pairs and augmented pairs. This allows us to
capture these shifts and incorporate them into the learning process
effectively.

$\mathcal{L}^{\text{pos}}=\mathbb{E}_{i=1,...,N}\left[\ell_{lr}(i,i;i^{\prime},i)\right],$
(7)

where $i^{\prime}$ is the index of augmented $i$-th sample.

#### Misalignment in Negative pairs.

Augmented images can possess relevance with different text, which would
normally be considered negative pairs in Eq.(5). However, the log-ratio
obtained by our momentum teacher alleviates our model from mistakenly pushing
the embedding of relevant texts away.

$\mathcal{L}^{\text{neg}}=\mathbb{E}_{\begin{subarray}{c}i,j=1,2,...,N\\\
i\neq j\end{subarray}}\left[\ell_{lr}(i,i;j^{\prime},i)\right],$ (8)

where $j^{\prime}$ is the index of augmented $j$-th sample.

#### Misalignment in Noisy pairs.

Original image-text pairs obtained from the web may contain either noisy
images or descriptions. While these pairs are normally treated as positive
pairs under contrastive loss, we propose a loss for noisy pairs where the
noisy labels are trained to have larger distance than matching image–text
pairs.

$\mathcal{L}^{\text{noisy}}=\mathbb{E}_{\begin{subarray}{c}i,j=1,2,...,N\\\
i\neq j\end{subarray}}\left[\ell_{lr}(i,i;j,j)\right].$ (9)

#### Distillation Loss.

The full training objective of MCD distillation $\mathcal{L}^{D}$ is the sum
of the three distillation terms, which is written as

$\mathcal{L}^{\text{D}}=\mathcal{L}^{\text{pos}}+\mathcal{L}^{\text{neg}}+\mathcal{L}^{\text{noisy}}.$
(10)

###  4.4 Training MCD

In this section, we explain the details of MCD training. The training
objective for the text encoder and student image encoder for MCD consists of
the three objectives: contrastive loss $\mathcal{L}^{C}$ in Eq (5) for initial
image–text distance learning, distillation loss $\mathcal{L}^{D}$ for the
three misalignment scenarios, and loss for masked language modeling
$\mathcal{L}^{\text{MLM}}$. The parameters for the teacher image encoder are
momentum updated.

#### MLM Loss.

Following previous work in literature [23, 18, 19], we randomly mask out the
input tokens with a probability of 15% and replace them with the special token
[MASK]111Following BERT, the replacement is done with either the [MASK] token
(80%), another random token within the dictionary (10%), or left unchanged
(10%).. Let $p^{\text{msk}}$ and $y^{\text{msk}}$ each denote the set of
model’s predicted probability for the masked tokens and the set of ground-
truth vocabulary index for the tokens, respectively. Then, MLM loss is written
as:

$\mathcal{L}^{\text{MLM}}=\mathbb{E}_{p\in p^{\text{msk}},y\in
y^{\text{msk}}}\left[\mbox{CE}(p,y)\right],$ (11)

where CE denotes Cross Entropy loss.

#### Momentum Teacher Update.

Let $\theta_{f_{I}}$, $\theta_{\bar{f}_{I}}$ be the parameter of the student
encoder and momentum teacher, respectively. For the $t$-th step, we update
$\theta^{(t)}_{\bar{f}_{I}}$ of the momentum teacher according to the
following:

$\theta_{\bar{f}_{I}}^{(t)}=m\theta_{\bar{f}_{I}}^{(t-1)}+(1-m)\theta_{f_{I}}^{(t)},$
(12)

where $m$ denotes the momentum parameter. We use $m=0.994$ in our experiments,
where $m$ grows in a cosine schedule to $1$ at the end of training.

#### Progressive Distillation.

As the training proceeds, InfoNCE loss conflicts with our misalignment loss.
InfoNCE between pair $I_{j}^{\prime}$ and $T_{i}$ forces the embedding to pull
regardless of its degree of misalignment. In early stages of training, the
model needs to learn how to discriminate positive or negative pairs with a
hard label. However, as the training progresses, the log-ratio loss delicately
models the distance between the various misalignments occurred by
augmentations or innately existing in the original image-text pair. Therefore,
we progressively diminish the contribution of InfoNCE loss involving augmented
views.

#### MCD Loss.

The final loss for MCD is written as:

$\mathcal{L}^{\text{MCD}}=\mathcal{L}^{\text{C}}+\alpha\cdot\mathcal{L}^{\text{D}}+\beta\cdot\mathcal{L}^{\text{MLM}},$
(13)

where $\alpha=0$ progressively increases on a cosine schedule to 1, and
$\beta=0.2$.

###  4.5 MCD Inference

MCD is based on a teacher-student network, thus the student $f_{I}$ and
momentum teacher $\bar{f_{I}}$ is obtained after training. Unlike previous
work in literature that leverage the teacher network for inference [37, 19],
we use the student network that is trained with both the contrastive loss and
the log-ratio loss for image–text distance learning.

##  5 Experiment

In this section, we provide implementation details and experimental results
with our MCD pretrained on two widely used image-text benchmark datasets
(_e.g_., CC3M, YFCC15M) to validate the effectiveness of our proposed MCD on
multiple downstream datasets including classification and image–text
retrieval.

###  5.1 Implementation Details and Datasets

For implementation details, our work is built on top of the open-source SLIP
codebase [27]222https://github.com/facebookresearch/SLIP . For DECLIP [23], we
follow the implementation details of the official code
release333https://github.com/Sense-GVT/DeCLIP. The performance on GPU-machine
runs for CLIP and SLIP follows the exact implementation details upon this
codebase. Since MCD features both the momentum teacher image encoder
$\bar{f_{I}}$ and student $f_{I}$, we conduct the following experiment section
with $\bar{f_{I}}$ based on empirical results. All of our models are
pretrained in 16$\times$ A100 GPUs. For CC3M, All models are trained with a
ViT-B/16 backbone with a learning rate of 5e-4 and weight decay of 0.5. For
YFCC15M, we train the model with ViT-B/32 backbone, batch size 4096, learning
rate 1e-3, and weight decay 0.2.

#### Pretraining Datasets.

To validate the effectiveness of MCD, we pretrain MCD on large-scale open-
source datasets: YFCC (Yahoo Flickr Creative Commons) 15M [38] and CC
(Conceptual Captions) 3M [34].

#### Downstream Datasets.

Following CLIP [32], we evaluate the transferability of pretrained MCD on 11
widely used downstream datasets for classification (_i.e_., Oxford Pets [30],
CIFAR-10, CIFAR-100 [17], SUN397 [41], Food-101 [2], Flowers [28], Cars [16],
Caltech-101 [8], Aircraft [26], DTD [7], ImageNet-1k [33]). We also transfer
to image–text retrieval tasks on Flickr30K [31] and MS-COCO Captions [4]
datasets. The evaluation settings for each dataset are consistent with CLIP as
in the open-source implementationslip_github.

###  5.2 MCD Pretraining on YFCC15M Dataset

First, we pretrain MCD on YFCC15M and evaluate its transferability in single-
modal (_e.g_., classification) and multi-modal (_e.g_., image–text retrieval)
downstream tasks. We compare the result against other state-of-the-art
Contrastive Language-Image Pretraining approaches [32, 27, 23, 18] that
utilizes various levels of supervision including vision self-supervision [3,
5], text self-supervision [23], memory queue [23], and augmentation encoding
[18]. All models are pretrained with a learning rate 1e-3 for 32 epochs unless
mentioned otherwise.

#### Zero-shot Classification.

Method |  Vision Encoder |  Oxford Pets |  CIFAR-10 |  CIFAR-100 |  SUN397 |  Food-101 |  Flowers |  Cars |  Caltech-101 |  Aircraft |  DTD |  ImageNet | Average
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Zero-shot Classification:
CLIP [32] |  ViT-B/32 | 19.4 | 62.3 | 33.6 | 40.2 | 33.7 | 6.3 | 2.1 | 55.4 | 1.4 | 16.9 | 31.3 |  27.5
SLIP [27] | 28.3 | 72.2 | 45.3 | 45.1 | 44.7 | 6.8 | 2.9 | 65.9 | 1.9 | 21.8 | 38.3 | 33.9
DeCLIP [23] | 30.2 | 72.1 | 39.7 | 51.6 | 46.9 | 7.1 | 3.9 | 70.1 | 2.5 | 24.2 | 41.2 | 35.4
UniCLIP [18] | 32.5 | 78.6 | 47.2 | 50.4 | 48.7 | 8.1 | 3.4 | 73.0 | 2.8 | 23.3 | 42.8 | 37.3
MCD (Ours) | 40.0 | 80.3 | 49.6 | 55.3 | 54.0 | 7.9 | 4.5 | 73.2 | 3.0 | 30.5 | 44.7 | 40.2
Linear Probing:
CLIP [32] | ViT-B/32 | 71.2 | 89.2 | 72.1 | 70.1 | 71.4 | 93.2 | 34.9 | 84.3 | 29.7 | 60.9 | 61.1 | 67.1
SLIP [27] | 75.4 | 90.5 | 75.3 | 73.5 | 77.1 | 96.1 | 43.0 | 87.2 | 34.1 | 71.1 | 68.1 | 71.9
DeCLIP [23] | 76.5 | 88.6 | 71.6 | 75.9 | 79.3 | 96.7 | 42.6 | 88.0 | 32.6 | 69.1 | 69.2 | 71.8
UniCLIP [18] | 83.1 | 92.5 | 78.2 | 77.0 | 81.3 | 97.1 | 49.8 | 88.9 | 36.2 | 72.8 | 70.8 | 75.2
MCD (Ours) | 85.6 | 92.7 | 79.3 | 77.6 | 81.7 | 97.1 | 46.9 | 89.5 | 36.6 | 74.1 | 71.3 | 75.7
Table 1: Zero-shot image classification/linear probing performance on 11
downstream datasets with YFCC15M pretrained models. Note that DeCLIP [23]
utilizes an external momentum queue while UniCLIP [18] features the
augmentation encoder during training.

We evaluate the zero-shot classification performance on 11 downstream datasets
for single-modal experiments. Tab. 1 shows both the zero-shot classification
and linear probing accuracy of CLIP variants [32, 27, 23, 18] pretrained on
YFCC15M dataset and transferred to downstream classification datasets. In test
time, the learned text encoder $f_{T}$ synthesizes a zero-shot linear
classifier by embedding the arbitrary categories of the test dataset. As
classes are in the form of a single word, we use prompts including the label
(e.g., “a photo of a {label}”) as following CLIP [32]. Our MCD outperforms
across a majority of the 11 datasets with a noticeable margin. Note that even
without additional augmented-aware network leveraged in UniCLIP [18] or
additional supervision terms such as text augmentation [40], masked language
modeling [23] and memory queue, our MCD achieves state-of-the-art performance.

#### Linear Probing.

To implement linear probe evaluation, we follow CLIP [32] to train a logistic
regression classifier on the frozen visual features extracted by the image
encoder $f_{I}$. Specifically, we train the logistic regression classifier
using L-BFGS algorithm provided by scikit-learn with maximum 1,000 iterations,
and report the corresponding metric for each
dataset444https://github.com/facebookresearch/SLIP/blob/main/main_linear.py.
Parameters for L2 regularization are determined using hyperparameter sweep on
the validation sets. Standard cropping and flipping augmentations [35] are
used for linear probing. The bottom section of Tab. 1 reports linear
classification performances on the 11 downstream datasets. Our proposed
approach, MCD, has consistently outperformed previous baseline methods in
zero-shot classification across multiple datasets, with only one exception.

#### Image–Text Retrieval.

| Image-to-text retrieval | Text-to-image retrieval
---|---|---
| Flickr30k | COCO Captions | Flickr30k | COCO Captions
Method | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10 | R@1 | R@5 | R@10
Zero-shot retrieval:
CLIP [32] | 34.9 | 63.9 | 75.9 | 20.8 | 43.9 | 55.7 | 23.4 | 47.2 | 58.9 | 13.0 | 31.7 | 42.7
SLIP [27] | 47.8 | 76.5 | 85.9 | 27.7 | 52.6 | 63.9 | 32.3 | 58.7 | 68.8 | 18.2 | 39.2 | 51.0
DeCLIP [23] | 51.4 | 80.2 | 88.9 | 28.3 | 53.2 | 64.5 | 34.3 | 60.3 | 70.7 | 18.4 | 39.6 | 51.4
UniCLIP [18] | 52.3 | 81.6 | 89.0 | 32.0 | 57.7 | 69.2 | 34.8 | 62.0 | 72.0 | 20.2 | 43.2 | 54.4
MCD (Ours) | 57.6 | 82.6 | 91.1 | 32.3 | 58.7 | 71.2 | 36.4 | 64.8 | 74.1 | 20.7 | 43.5 | 55.3
Fine-tuned retrieval
CLIP [32] | 58.3 | 84.8 | 91.5 | 36.1 | 65.0 | 76.4 | 43.1 | 71.1 | 80.3 | 24.9 | 51.7 | 64.1
SLIP [27] | 69.6 | 90.4 | 95.7 | 45.0 | 74.0 | 83.0 | 52.1 | 79.4 | 86.9 | 31.6 | 59.5 | 71.3
DeCLIP [23] | 75.6 | 93.0 | 96.6 | 48.7 | 77.3 | 86.2 | 57.8 | 83.3 | 90.3 | 34.2 | 63.1 | 74.6
UniCLIP [18] | 78.1 | 94.9 | 97.7 | 54.5 | 80.9 | 89.1 | 61.0 | 86.0 | 91.9 | 38.0 | 67.2 | 78.0
MCD (Ours) | 79.3 | 95.2 | 98.0 | 55.6 | 81.2 | 89.5 | 63.1 | 87.2 | 92.3 | 38.2 | 67.4 | 78.5
Table 2: Zero-shot & Fine-tuned image–text retrieval on the test splits of
Flickr30k and COCO Captions with models pre-trained on YFCC15M. ViT-B/32 is
used for all setup.

For multi-modal evaluations, we test both the zero-shot and fine-tuned
image–text (and text–image) retrieval on Flickr30k and COCO Captions
benchmarks. Image-text pairs are ranked according to their similarity scores.
Tab. 2 shows the performance for image–text retrieval tasks of MCD pretrained
on YFCC15M dataset. Our MCD outperforms all state-of-the-art baselines across
every measure with a considerable margin. By incorporating a log-ratio loss
with metric learning characteristics into the CLIP framework, our proposed
approach has achieved significant improvements in image-text retrieval
performance.

#### Vision–Language Compositionality.

Method | Aug. | Misalign | Replace | Swap | Add
---|---|---|---|---|---
CLIP [32] |  | N/A | 73.6 | 59.5 | 69.4
SLIP [27] | ✓ | N/A | 74.7 | 58.6 | 69.1
DeCLIP [23] | ✓ | Disregard | 74.5 | 58.2 | 66.8
UniCLIP [18] | ✓ | $f_{A}$ | 75.5 | 58.4 | 70.4
MCD (Ours) | ✓ | $\mathcal{L}_{\text{D}}$ | 76.2 | 61.5 | 71.7
Table 3: Evaluation on SugarCrepe [12]. All models were pre-trained on YFCC15M
with ViT-B/32 backbone. Models except CLIP involve random image augmentations
(Aug.) with different schemes for dealing with image–text misalignments
(Misalign). While previous methods either disregard the issue [23] or
introduce an additional augmentation encoder ($f_{A}$) [18], MCD manages to
harness the misalignment for the distillation loss ($\mathcal{L}_{\text{D}}$).

To conduct a thorough analysis of multimodal representation learning, we
assess the performance of our model using the SugarCrepe [12] dataset. This
dataset serves as a de-biased benchmark specifically designed for evaluating
the compositionality aspect of vision-language models. SugarCrepe introduces a
set of challenging negative captions for COCO image-text pairs by replacing,
swapping, or adding certain concepts to the ground truth caption, and gauge
the model’s capability to discern the positive from its distractor. Tab. 3
summarizes the result, showcasing the performance of various models pretrained
on the YFCC15M dataset. Our findings demonstrate that the MCD exhibits
significantly better compositionality compared to prior methods. This improved
performance is attributed to leveraging augmentations with proper handling for
misalignment arising from these augmentations. While other methods often show
improved performance over CLIP baseline, adopting augmentations without
meticulous management of misalignment cannot maximize their utility.

###  5.3 MCD Pretraining on CC3M Dataset

In this section, we compare MCD against other state-of-the-art Contrastive
Language-Image Pretraining approaches [32, 27, 23]. All models are pretrained
on the CC3M dataset with a learning rate 5e-4 for 40 epochs555More detailed
training configuration will be provided in supplement.. Tab. 4 shows the
ImageNet zero-shot results of MCD with other CLIP variants. MCD outperforms
all CLIP variants without external training sources such as Nearest Neighbor
supervision with large memory queues (NNS) or augmentation information during
training (AUG). Furthermore, MCD does not require any additional parameters
for the SSL projection layer [27, 23] or additional network for augmentation-
aware feature embedding [18].

Method | Encoder | SSL | EXT | Top1(%)
---|---|---|---|---
CLIP [32] | ViT-B/16 | - | - | 19.6
SLIP [27] |  SimCLR [3] | - | 23.2
DeCLIP [23] |  SimSiam [5] | NNS | 25.4
UniCLIP [18] |  MP-NCE [18] | AUG | 27.8
MCD (Ours) |  MP-NCE [18] | - | 28.2
Table 4:  ImageNet-1k Top 1 zero shot accuracy with models pretrained on CC3M
dataset. SSL denotes the vision self-supervision term used in each model and
EXT denotes external sources involved during training. NNS is nearest neighbor
supervision using a separate memory queue, and AUG is the vectorized
information of each random augmentation conducted during training.

###  5.4 Ablation Study

This section presents ablation studies to evaluate the contribution of each
component in our proposed approach, MCD, towards the final performance. To
this end, we pre-train all models on the YFCC15M dataset and evaluate them
using zero-shot learning on the Imagenet-1k validation set. Specifically, we
implement the MP-NCE loss without augmentation encoding, which results in an
accuracy of 39.6. Our results in Tab. 5 demonstrate that each loss component
in MCD has a positive impact on the final performance, leading to an overall
improvement in accuracy. These findings highlight the importance of each
component in our proposed approach and validate its effectiveness in improving
the zero-shot classification performance. Note that MCD outperforms (b)
(_i.e_., UniCLIP) that explicitly includes the augmentation information during
training with only $\mathcal{L}^{\text{pos}}$, showing the effectiveness of
harnessing the misalignments that occur during random image augmentation for
training.

| $\mathcal{L}^{\text{base}}$ | $f_{A}$ | $\mathcal{L}^{\text{pos}}$ | $\mathcal{L}^{\text{neg}}$ | $\mathcal{L}^{\text{noisy}}$ | Top1 Acc (%)
---|---|---|---|---|---|---
(a) | ✓ |  |  |  |  | 39.6
(b) | ✓ | ✓ |  |  |  | 42.8
(c) | ✓ |  | ✓ |  |  | 43.9
(d) | ✓ |  | ✓ | ✓ |  | 44.3
(e) | ✓ |  | ✓ | ✓ | ✓ | 44.7

Table 5:  Ablation study on ImageNet-1k Top 1 zero shot accuracy for vision-
language pretraining for each loss components of MCD. All models were
pretrained with a ViT-B/32 backbone with a basic contrastive loss
$\mathcal{L}^{C}$ in Eq (5) and MLM loss in Eq (11), which we abbreviate as
$\mathcal{L}^{\text{base}}$. All $\mathcal{L}^{\text{pos}}$,
$\mathcal{L}^{\text{neg}}$, $\mathcal{L}^{\text{noisy}}$ shows a consistent
gain in zero-shot performance. $f_{A}$ denotes the augmentation encoder,
making (b) analogous to UniCLIP [18].

##  6 Conclusion

We propose MCD, a new training strategy for dealing with misalignments occured
by random image augmentations under visual–language pretraining. Our novel
distillation formulation enables data-efficient training under Contrastive
Language-Image Pretraining. Future works will include extending MCD frameworks
to other modalities.

## References

  * [1] Alex Andonian, Shixing Chen, and Raffay Hamid.  Robust cross-modal representation learning with progressive self-distillation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
  * [2] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool.  Food-101–mining discriminative components with random forests.  In European conference on computer vision, pages 446–461. Springer, 2014.
  * [3] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  * [4] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  arXiv preprint arXiv:1504.00325, 2015.
  * [5] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750–15758, 2021.
  * [6] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu.  Uniter: Universal image-text representation learning.  In European conference on computer vision, pages 104–120. Springer, 2020.
  * [7] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi.  Describing textures in the wild.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3606–3613, 2014.
  * [8] Li Fei-Fei, Rob Fergus, and Pietro Perona.  Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories.  In 2004 conference on computer vision and pattern recognition workshop, pages 178–178. IEEE, 2004.
  * [9] Yuting Gao, Jinfeng Liu, Zihan Xu, Jun Zhang, Ke Li, and Chunhua Shen.  Pyramidclip: Hierarchical feature alignment for vision-language model pretraining.  arXiv preprint arXiv:2204.14095, 2022.
  * [10] Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, and Yongfeng Zhang.  HiCLIP: Contrastive language-image pretraining with hierarchy-aware attention.  In The Eleventh International Conference on Learning Representations, 2023.
  * [11] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [12] Cheng-Yu Hsieh, Jieyu Zhang, Zixian Ma, Aniruddha Kembhavi, and Ranjay Krishna.  Sugarcrepe: Fixing hackable benchmarks for vision-language compositionality.  arXiv preprint arXiv:2306.14610, 2023.
  * [13] Haoyang Huang, Yaobo Liang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, and Ming Zhou.  Unicoder: A universal language encoder by pre-training with multiple cross-lingual tasks.  arXiv preprint arXiv:1909.00964, 2019.
  * [14] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig.  Scaling up visual and vision-language representation learning with noisy text supervision.  In International Conference on Machine Learning, pages 4904–4916. PMLR, 2021.
  * [15] Sungyeon Kim, Minkyo Seo, Ivan Laptev, Minsu Cho, and Suha Kwak.  Deep metric learning beyond binary supervision.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2288–2297, 2019.
  * [16] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In Proceedings of the IEEE international conference on computer vision workshops, pages 554–561, 2013.
  * [17] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  -, 2009.
  * [18] Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim.  Uniclip: Unified framework for contrastive language-image pre-training.  In Advances in Neural Information Processing Systems, 2022.
  * [19] Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, Caiming Xiong, and Steven Hoi.  Align before fuse: Vision and language representation learning with momentum distillation.  In Advances in Neural Information Processing Systems, 2021.
  * [20] Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang.  Visualbert: A simple and performant baseline for vision and language.  arXiv preprint arXiv:1908.03557, 2019.
  * [21] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang.  Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning.  arXiv preprint arXiv:2012.15409, 2020.
  * [22] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al.  Oscar: Object-semantics aligned pre-training for vision-language tasks.  In European Conference on Computer Vision, pages 121–137. Springer, 2020.
  * [23] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan.  Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm.  In International Conference on Learning Representations, 2022.
  * [24] Haoyu Lu, Nanyi Fei, Yuqi Huo, Yizhao Gao, Zhiwu Lu, and Ji-Rong Wen.  Cots: Collaborative two-stream vision-language pre-training model for cross-modal retrieval.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15671–15680, 2022.
  * [25] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  Advances in neural information processing systems, 32, 2019.
  * [26] Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi.  Fine-grained visual classification of aircraft.  arXiv preprint arXiv:1306.5151, 2013.
  * [27] Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie.  Slip: Self-supervision meets language-image pre-training.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI, pages 529–544. Springer, 2022.
  * [28] Maria-Elena Nilsback and Andrew Zisserman.  Automated flower classification over a large number of classes.  In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722–729. IEEE, 2008.
  * [29] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [30] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar.  Cats and dogs.  In 2012 IEEE conference on computer vision and pattern recognition, pages 3498–3505. IEEE, 2012.
  * [31] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik.  Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models.  In Proceedings of the IEEE international conference on computer vision, pages 2641–2649, 2015.
  * [32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In International Conference on Machine Learning, pages 8748–8763. PMLR, 2021.
  * [33] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander Berg, and Li Fei-Fei.  Imagenet large scale visual recognition challenge.  International Journal of Computer Vision, 2014.
  * [34] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut.  Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning.  In Proceedings of ACL, 2018.
  * [35] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [36] Hao Tan and Mohit Bansal.  Lxmert: Learning cross-modality encoder representations from transformers.  arXiv preprint arXiv:1908.07490, 2019.
  * [37] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, 2017.
  * [38] Bart Thomee, David A Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li.  Yfcc100m: The new data in multimedia research.  Communications of the ACM, 59(2):64–73, 2016.
  * [39] Yaushian Wang, Hung-Yi Lee, and Yun-Nung Chen.  Tree transformer: Integrating tree structures into self-attention.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1061–1070, Hong Kong, China, 2019. Association for Computational Linguistics.
  * [40] Jason Wei and Kai Zou.  Eda: Easy data augmentation techniques for boosting performance on text classification tasks.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6382–6388, 2019.
  * [41] Jianxiong Xiao, Krista A Ehinger, James Hays, Antonio Torralba, and Aude Oliva.  Sun database: Exploring a large collection of scene categories.  International Journal of Computer Vision, 119(1):3–22, 2016.