# Learning Domain-Invariant Features for Out-of-Context News Detection

Yimeng Gu<EMAIL_ADDRESS>Queen Mary University of LondonLondonUK ,
Mengqi Zhang<EMAIL_ADDRESS>Shandong UniversityQsingdaoChina ,
Ignacio Castro<EMAIL_ADDRESS>Queen Mary University of LondonLondonUK ,
Shu Wu<EMAIL_ADDRESS>CRIPC, Chinese Academy of ScienceBeijingChina and
Gareth Tyson<EMAIL_ADDRESS>The Hong Kong University of Science and Technology
(GZ)GuangzhouChina

(2024)

###### Abstract.

Multimodal out-of-context news is a common type of misinformation on online
media platforms. This involves posting a caption, alongside an invalid out-of-
context news image. Reflecting its importance, researchers have developed
models to detect such misinformation. However, a common limitation of these
models is that they only consider the scenario where pre-labeled data is
available for each domain, failing to address the out-of-context news
detection on unlabeled domains (e.g., unverified news on new topics or
agencies). In this work, we therefore focus on _domain adaptive_ out-of-
context news detection. In order to effectively adapt the detection model to
unlabeled news _topics_ or _agencies_ , we propose ConDA-TTA (Contrastive
Domain Adaptation with Test-Time Adaptation) which applies contrastive
learning and maximum mean discrepancy (MMD) to learn the domain-invariant
feature. In addition, it leverages target domain statistics during test-time
to further assist domain adaptation. Experimental results show that our
approach outperforms baselines in 5 out of 7 domain adaptation settings on two
public datasets, by as much as 2.93% in F1 and 2.08% in accuracy.

contrastive learning, domain adaptation, out-of-context news detection,
multimodal learning

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
ACM Conference XXXX; June 03–05, 2018; Woodstock, NY††isbn:
978-1-4503-XXXX-X/24/10††ccs: Computing methodologies Natural language
processing††ccs: Information systems Web applications††ccs: Computing
methodologies Computer vision

##  1\. Introduction

Online news platforms suffer from the release and spread of misinformation.
Disseminating misinformation is especially concerning in news topics related
to key societal issues, such as public health and politics, where it could
result in misleading public awareness and arousing public panic. A common
(Fazio, 2020) yet subtle type of misinformation is image repurposing, also
called _out-of-context news_. Unlike DeepFake (Dolhansky et al., 2020) models
which generate non-existing images, this approach attaches a false claim with
a real image outside of its original context (see Figure 1). This lowers the
technical threshold (Fazio, 2020) of producing misinformation since anyone can
trivially attach a fabricated caption to an image and post it online, thereby
enlarging its scope of occurrence. Complicating matters, out-of-context news
cannot be detected using a sole modality input (as in isolation, each modality
may be valid).

Figure 1. Examples of out-of-context news of three different news topics from
the Twitter-COMMs dataset.

Out-of-context news detection has received a growing attention in recent
years. Existing works (Biamby et al., 2022; Luo et al., 2021; Abdelnabi et
al., 2022; Mu et al., 2023) fine-tune pre-trained vision and text encoders to
represent the image and text, and then devise neural networks to measure
vision-text consistency. More recent works (Dai et al., 2024; Zhang et al.,
2023) query the Multimodal Large Language Model (MLLM) to obtain the
prediction with explanations. Nevertheless, their approaches do not address
the adaptation to entirely new topics or news agencies. Due to the wide array
of news topics shared online, it is infeasible to re-train the detector every
time a new topic emerges. Therefore, a detection model that can adapt to new
topics will greatly relieve the operational burden. Additionally, considering
annotation costs, it is equally critical to build a detection model that is
able to adapt to other news agencies with minimal efforts, so that the model
developed for one news agency can be easily extended to other popular news
agencies, _e.g._ USA Today. Because of this, we argue that it is important to
devise _domain adaptive_ out-of-context news detection model.

Unfortunately, prior works neglect addressing the domain adaption issue on
out-of-context news detection. Trained on sufficient labeled data from each
domain, existing models tend to capture domain-specific knowledge which is not
shared among different domains. Although this knowledge is helpful for
improving the model’s performance on annotated domains, it results in inferior
performance on unannotated domains (Wang et al., 2018). In light of this, we
believe that learning domain-invariant features will help the detection model
adapt to unlabeled domains. Though important, enabling domain adaptive
capabilities on this task comes with several challenges. _First_ , existing
out-of-context news datasets only cover a small number of topics/agencies.
This makes it challenging for the detection model to learn domain-invariant
features, because the learned features might be biased towards the limited
number of source domains. _Second_ , in (social media) news dataset, the data
distributions of different domains vary, since news from different domains
tend to have different writing styles and different image styles. This makes
domain-invariant knowledge learning challenging. It is critical to ensure that
transferable domain-invariant knowledge is actually learned. Otherwise, the
model may bypass learning transferable domain knowledge and end up learning
semantic patterns as a shortcut to fulfill the classification task (Li et al.,
2023a; Chi et al., 2023).

To tackle the aforementioned challenges, we propose _Contrastive Domain
Adaptation with Test-Time Adaptation_ (ConDA-TTA) for domain adaptive out-of-
context news detection. ConDA-TTA first uses an MLLM to directly encode the
image and text into a multimodal feature representation. After feature
encoding, we adopt the contrastive loss to learn a representation in the
projected space where the original news stays further away from the out-of-
context news. To overcome the shortage of labeled source domains, we use
unlabeled target domain data as well, to learn less biased domain-invariant
features. We further apply Maximum Mean Discrepancy (MMD) to measure and
reduce the discrepancy of the learned representations between the source and
target domain. In this way, MMD ensures that the domain-invariant features are
captured from the learned representations. Finally, we incorporate Test-Time
Adaptation (TTA) to update the statistic-related model parameters in the
evaluation phase so that the model can better adapt to the target domain.

To summarize, our main contributions are as follows:

  * •

To the best of our knowledge, we are the first to investigate domain
adaptation in out-of-context news detection. To address the challenges, we
propose a novel approach using contrastive domain adaptation with test-time
adaptation (ConDA-TTA).

  * •

We evaluate our approach and demonstrate that it achieves state-of-the-art
(SOTA) performance under most domain adaptation settings. Notably, it
outperforms the baseline by as much as 2.93% in F1 and 2.08% in accuracy when
the domain is defined as news topic. Additionally, it outperforms the baseline
by as much as 1.82% in F1 and 1.84% in accuracy when the domain is defined as
news agency.

  * •

We conduct a comprehensive ablation study and show that MMD tends to be the
most contributing component to the domain adaptation on the Twitter-COMMs
dataset, while TTA tends to be the most contributing component on the
NewsCLIPpings dataset.

##  2\. Related Work

###  2.1. Out-of-Context News Detection

Prior works on out-of-context news detection mainly adopt three approaches:
(i) fine-tuning or (ii) prompting large vision and language models, and (iii)
leveraging synthetic information. Biamby et al. (2022) fine-tunes CLIP
(Radford et al., 2021) and a linear classifier to detect out-of-context news.
Similarly, Luo et al. (2021) fine-tunes both CLIP and VisualBert (Li et al.,
2019) with a linear classifier. Abdelnabi et al. (2022) concatenates the
retrieved textual evidence with visual evidence and then inputs the learned
representations into a simple classifier. These works involve fine-tuning a
large multimodal pre-trained model (_e.g._ CLIP), which is very compute-
intensive (_e.g._ (Biamby et al., 2022) uses 8 days of training on 10 GPUs),
and does not enable domain adaptation.

Later work leverages synthetic multimodal information. Shalabi et al. (2023)
uses a stable diffusion model (Dhariwal and Nichol, 2021) to generate
augmented image from text, and a pre-trained image captioning model to
generate augmented text from image. (Yuan et al., 2023) extracts stances from
the external multimodal evidences and uses it to enhance evidence-based out-
of-context detection. More recent works focus on enhancing the
interpretability. (Qi et al., 2024) adopts two-stage instruction tuning on
Instruct-BLIP (Dai et al., 2024) to provide accurate and persuasive
explanations to the prediction. (Zhang et al., 2023) uses neural symbolic
model to enhance model interpretability. (Shalabi et al., 2024) fine-tunes the
MiniGPT-4 to detect out-of-context news. Although effective, the above works
do not address the domain adaptation problem in out-of-context news detection.

###  2.2. Domain Adaptive Fake News Detection

To the best of our knowledge, no prior work has explored domain adaptive out-
of-context news detection. However, some prior works have investigated
techniques to enable domain adaptation in multimodal fake news detection — a
task close to out-of-context news detection. Some works (Wang et al., 2018;
Zhang et al., 2020, 2021; Yuan et al., 2021) adopt adversarial learning
through a domain classifier/discriminator to learn the domain-invariant
feature. On text-based domain adaptation in fake news detection, Nan et al.
(2022) adopts meta learning and Ding et al. (2022) further combines it with
adversarial learning. Furthermore, incorporating user comments and user-news
interaction information, Mosallanezhad et al. (2022) adopts reinforcement
learning that exploits cross-domain and within-domain knowledge to achieve
robustness in the target domain. Yue et al. (2022) proposes a contrastive
domain adaptation method in order to reduce the intra-class discrepancy and
enlarge the inter-class discrepancy. Lin et al. (2023) employs prompt
engineering to learn language-agnostic contextual representations and models
the domain-invariant structural features from the propagation threads. In
contrast, our work focuses on the domain adaptation in out-of-context news
detection and adopts both contrastive learning and test-time adaptation to
address the unique challenges (Section 1) of this task.

###  2.3. Test-Time Adaptation

Test-time adaptation (TTA) is an emerging learning paradigm that increases the
trained model’s generalization ability to the test samples under distribution
shifts (Liang et al., 2023). We later employ this to improve our performance.
Many works focus on adapting batch normalization (BN) layer’s parameters
during model testing: Li et al. (2016) updates the BN statistics; Wang et al.
(2020) further updates the learnable parameters using information entropy;
Zhao et al. (2023) proposes to use exponential moving average (EMA) to update
BN parameters and to address the imbalanced classes problem. Wang et al.
(2022) proposes continual test-time adaptation for reducing error accumulation
and catastrophic forgetting. Recent works propose to conduct sample filtering
during TTA: Niu et al. (2022) filters out less reliable samples and maintains
sample diversity at the same time; Niu et al. (2023) removes partial noisy
samples with large gradient and encourages model weights to go to a flat
minimum to achieve stable TTA. In our work, we combine test-time adaptation
with contrastive learning, in order to further facilitate domain adaptation in
out-of-context news detection.

##  3\. Problem Statement

In out-of-context news detection, given the news image-text pair
$(x_{img},x_{txt})$, the detection model is expected to predict whether it is
falsified, _i.e._ out-of-context. If the image and text are not from the same
news post, this image-text pair is considered as falsified (labeled as True),
otherwise not falsified (labeled as False).

In this work, we tackle the challenge of domain adaptive out-of-context news
detection. During training, we have access to the labeled source domain data
and unlabeled target domain data. During testing, we evaluate our model’s
performance on the labeled target domain data. The source domain and target
domain are mutually exclusive. The target domain training data and testing
data are mutually exclusive as well. Concretely, the dataset is comprised of
news from $N$ domains. Each time we select $M$ domains as the target domain,
$\mathcal{D}^{T}=\\{\mathcal{D}^{T_{m}}\\}_{m=1}^{M}$, and the rest $N-M$
domains as the source domain,
$\mathcal{D}^{S}=\\{\mathcal{D}^{S_{n}}\\}_{n=1}^{N-M}$. The $n$-th source
domain $\mathcal{D}^{S_{n}}$ contains labeled data,
$\mathcal{D}^{S_{n}}=\\{(x_{i}^{S_{n}},y_{i}^{S_{n}})\\}_{i=1}^{|S_{n}|}$,
where $x^{S_{n}}_{i}=(x_{img},x_{txt})_{i}^{S_{n}}$ denotes the $i$-th news
image-text pair within domain $\mathcal{D}^{S_{n}}$, and $y_{i}^{S_{n}}$
denotes whether this image-text pair is falsified or not (True or False). In
contrast, the $m$-th target domain contains unlabeled data:
$\mathcal{D}^{T_{m}}=\\{x_{j}^{T_{m}}\\}_{j=1}^{|T_{m}|}$, where
$x_{j}^{T_{m}}=(x_{img},x_{txt})_{j}^{T_{m}}$.

For simplicity, in the following notations, we use $T$ to denote target domain
and $S$ to denote source domain.

##  4\. Our Approach: ConDA-TTA

In this section, we describe our proposed approach for domain adaptive out-of-
context news detection: _Contrastive Domain Adaptation with Test-Time
Adaptation (ConDA-TTA)_. The overall architecture of ConDA-TTA is illustrated
in Figure 2. It consists of three components: (i) _Multimodal Feature Encoder_
, which encodes the image and text pair into a multimodal representation; (ii)
_Domain-Invariant Feature Learning_ , which applies contrastive learning and
maximum mean discrepancy (MMD) to learn the domain-invariant feature; and
(iii) _Test-Time Adaptation_ , which leverages unlabeled test set statistics
to further adapt the model to the target domain. Finally, we discuss the
training details of our approach.

Figure 2. The overall model architecture of ConDA-TTA. We first use the (i)
_Multimodal Feature Encoder_ to encode out-of-context news and its
augmentation into multimodal representations. Then in the (ii) _Domain-
Invariant Feature Learning_ , we apply contrastive learning and maximum mean
discrepancy (MMD) to learn the domain-invariant feature shared by different
domains. Finally, we adopt the (iii) _Test-Time Adaptation_ to update
statistic-related model parameters in the evaluation phase.

###  4.1. Multimodal Feature Encoder

In order to capture the semantic nuance between original news and out-of-
context news, we use MLLM to directly encode the news text and image into a
meaningful multimodal representation.

Formally, the multimodal feature encoder is defined as follows:

(1)
$\textbf{x}^{S}=\mathrm{MLLM}((x_{img},x_{txt})^{S}),\quad\textbf{x}^{T}=\mathrm{MLLM}((x_{img},x_{txt})^{T}),$

where $\textbf{x}^{S}$ and $\textbf{x}^{T}$ respectively denote the multimodal
feature representation of news image-text pair $(x_{img},x_{txt})$ from the
source and target domain.

###  4.2. Domain-Invariant Feature Learning

After encoding the news, we introduce a domain-invariant feature learning
module to learn the domain-invariant feature from the news’ multimodal
representation. Specifically, this module adopts contrastive learning to learn
a more separable representation space and adopts MMD to capture the invariant
features among different domains.

We first use a projection head to project the multimodal feature
representation into a lower dimensional space. The projection head is composed
of two linear layers, and can be expressed as follows:

(2)
$\textbf{z}^{S}=\mathrm{Projection}(\textbf{x}^{S}),\quad\textbf{z}^{T}=\mathrm{Projection}(\textbf{x}^{T}).$

Next, we aim to train the projection head so that it can capture the domain-
invariant features from $\textbf{x}^{S}$ and $\textbf{x}^{T}$. This is enabled
by applying contrastive learning and the maximum mean discrepancy. Note, no
label information is required within this module.

Contrastive Learning: Inspired by (Bhattacharjee et al., 2023), we leverage
contrastive learning to learn more separable representations of the input
multimodal features in the projected space. In this way, we expect that the
learned representations of similar semantic meanings stay closer, and the
learned representations of dissimilar semantic meanings stay further apart.

To achieve this, we adopt the contrastive loss used in (Bhattacharjee et al.,
2023). Applying contrastive loss pulls closer the _positive pairs_ and pushes
further apart the _negative pairs_. We augment each news item in order to
construct the positive and negative pairs later. For each news item
$\textbf{x}_{i}^{S}$ in a source domain training batch
$\\{\textbf{x}_{i}^{S}\\}_{i=1}^{b}$, we generate one augmentation of it and
encode it by the same multimodal feature encoder to obtain its representation,
denoted as $\textbf{x}_{i+}^{S}$. In this way, we obtain $2|b|$ news items in
the augmented batch. After projecting the input multimodal feature to the
lower dimensional space, the _positive pair_ is formed by the data item and
its corresponding augmentation item, denoted as
$(\textbf{z}_{i}^{S},\textbf{z}_{i+}^{S})$; the _negative pairs_ are formed by
the data item and the rest of the $2(|b|-1)$ items within the augmented batch,
denoted as $(\textbf{z}_{i}^{S},\textbf{z}_{k}^{S})$. The contrastive loss is
expressed as follows:

(3) $\mathcal{L}_{ctr}^{S}=-\sum_{(i,i_{+})\in
b}\log\frac{\exp(\mathrm{sim}(\textbf{z}_{i}^{S},\textbf{z}_{i_{+}}^{S})/t)}{\sum_{k=1,k\neq
i}^{2|b|}\exp(\mathrm{sim}(\textbf{z}_{i}^{S},\textbf{z}_{k}^{S})/t)}.$

Here, $b$ denotes the current batch and $|b|$ denotes the size of the current
batch. $S$ denotes the source domain. $\textbf{z}_{i}^{S}$ and
$\textbf{z}_{j}^{S}$ denote the learned representations of the $i$-th data
item and its augmentation in the projected space. $t$ denotes the temperature
coefficient. $\mathrm{sim}(\cdot)$ denotes the similarity metric. Here, cosine
similarity is being used. The data augmentation on the target domain training
batch and the contrastive loss computation for the target domain
$\mathcal{L}_{ctr}^{T}$ is applied in the same way.

The performance of contrastive learning is largely influenced by the
transformation used to generate positive samples (Tian et al., 2020).
Following the data augmentation experiments in (Bhattacharjee et al., 2023),
we include text-level: {synonym replacement, random swap, random crop}, and
image-level: {random resize and crop, random horizontal flip, Gaussian blur}
into the set of candidate data augmentation techniques (detailed descriptions
can be found in the supplementary material). We also include the combination
of text-level and image-level augmentations as the candidates. Our experiments
show that all the above strategies result in similar performances (in terms of
accuracy on the target domain), but applying Gaussian blur to the image
results in the best overall performance (averaged over all domain adaptation
settings) on the target domain. Thus, in our approach, we apply Gaussian blur
to the image while keeping the text unaltered to generate the positive sample
of the anchor item. Note, the augmentation preserves the label.

Maximum Mean Discrepancy (MMD): Now that we have learned more separable
representations, $\textbf{z}^{S}$ and $\textbf{z}^{T}$, in the projected
space, we want to further ensure that these representations capture domain-
invariant feature. To achieve this, we adopt maximum mean discrepancy (MMD)
(Gretton et al., 2012). MMD is a kernel-based statistical test used to
determine if two samples are drawn from different distributions (Gretton et
al., 2012). We use it as a loss function to measure the discrepancy between
two distributions. Here, we regard the learned representations
$\textbf{z}^{S}$ and $\textbf{z}^{T}$ as the samples from two random variables
$Z^{S}$ and $Z^{T}$, and we want their probability distributions to have as
small as possible discrepancy. In this way, the distribution of $Z^{S}$ and
$Z^{T}$ conditions less on a specific domain. Alternatively speaking, if MMD
is small enough, we regard that it removes the domain-specific features from z
and keeps the domain-invariant features.

The squared MMD between $Z^{S}$ and $Z^{T}$ in the reproducing kernel Hilbert
space (RKHS) $\mathcal{H}$ is formulated as follows:

(4) $\displaystyle\mathrm{MMD}^{2}[\mathcal{F},Z^{S},Z^{T}]$
$\displaystyle=\left[\sup_{||f||_{\mathcal{H}}\leq
1}(\mathbf{E}[f(Z^{S})]-\mathbf{E}[f(Z^{T})])\right]^{2}$
$\displaystyle=||\mu_{S}-\mu_{T}||^{2}_{\mathcal{H}},$

where $\mu_{S}$ and $\mu_{T}$ are mean embeddings of $f(Z^{S})$ and
$f(Z^{T})$, $f\in\mathcal{F}$. $\mathcal{F}$ is the unit ball function class
in the Hilbert space $\mathcal{H}$.

Following prior work (Pan et al., 2010; Long et al., 2015; Bhattacharjee et
al., 2023), we compute the MMD between features in the lower dimensional space
(_i.e._ the projected z space). In our implementation, we use the empirical
MMD (Gretton et al., 2012), which is expressed as follows:

(5) $\displaystyle\mathrm{MMD}[\mathcal{F},Z^{S},Z^{T}]=$
$\displaystyle\left[\frac{1}{m^{2}}\sum_{i,j=1}^{m}k(\textbf{z}_{i}^{S},\textbf{z}_{j}^{S})-\frac{2}{mn}\sum_{i,j=1}^{m,n}k(\textbf{z}_{i}^{S},\textbf{z}_{j}^{T})\right.$
$\displaystyle\left.+\frac{1}{n^{2}}\sum_{i,j=1}^{n}k(\textbf{z}_{i}^{T},\textbf{z}_{j}^{T})\right]^{\frac{1}{2}}.$

Here, both $m$ and $n$ are equal to the batch size. $k(\cdot,\cdot)$ denotes
the kernel function. $\sigma$ is a free parameter. We use the Radial basis
function kernel shown in the following:

(6)
$k(\textbf{z}_{i},\textbf{z}_{j})=\exp\left(-\frac{||\textbf{z}_{i}-\textbf{z}_{j}||^{2}_{2}}{2\sigma^{2}}\right).$

Training: The learned domain-invariant representation, z, is then input to the
classifier to get the predicted label $\hat{y}$ for the corresponding out-of-
context news:

(7) $\hat{y}=\mathrm{CLS}(\textbf{z}).$

The loss function for the classifier is the cross-entropy loss, where
$\hat{y}$ is the predicted label and $y$ is the ground true label:

(8) $\mathcal{L}_{CE}=-\mathbf{E}_{y\sim
Y}\left[y\log(\hat{y})+(1-y)\log(1-\hat{y})\right].$

During training, both the projection head and the classifier have learnable
parameters. The total loss is composed of (i) the contrastive losses from both
the source and target data in the training set, (ii) the MMD between the
source and target projected representations, and (iii) the cross-entropy
losses of the predicted label for both the source data ($\mathcal{L}_{CE}$)
and its augmentation ($\mathcal{L}_{CE+}$). This can be expressed as follows:

(9)
$\mathcal{L}=\frac{1}{2}\lambda_{CE}(\mathcal{L}_{CE}^{S}+\mathcal{L}_{CE+}^{S})+\frac{1}{2}\lambda_{ctr}(\mathcal{L}_{ctr}^{S}+\mathcal{L}_{ctr}^{T})+\lambda_{MMD}\mathrm{MMD},$

where $\lambda_{CE}$ is the weight of the cross entropy losses,
$\lambda_{ctr}$ is the weight of the contrastive losses and $\lambda_{MMD}$ is
the weight of MMD.

###  4.3. Test-Time Adaptation

Domain-invariant features help the model to generalize to the target domain.
In this section, we introduce test-time adaptation which leverages target
domain statistics to further adapt the model to the target domain during
evaluation.

Prior work (Li et al., 2016) observes that the statistics of the Batch
Normalization (BN) layer contain domain-specific information. Concretely, they
find that running the test set through the trained model achieves deep
adaptation effects on the target domain. The reason is that this process
allows BN statistics to keep track of the target domain statistics, which is
conducive to the domain adaptation. As such, we introduce BN layers into the
classifier.

At each training step, BN statistics compute and update the running estimates
$\\{\hat{\mu}_{k+1},\hat{\sigma}^{2}_{k+1}\\}$ (Eq. (10)) based on its
observed mean $\mu_{k}$ and variance $\sigma_{k}^{2}$ (Eq. (11)) of the layer
batch inputs $\\{x_{i}\\}_{i=1}^{b}$:

(10)
$\hat{\mu}_{k+1}=(1-\rho)\hat{\mu}_{k}+\rho\mu_{k},\quad\hat{\sigma}^{2}_{k+1}=(1-\rho)\hat{\sigma}^{2}_{k}+\rho\sigma^{2}_{k}$
(11)
$\mu_{k}=\frac{1}{b}\sum_{i}x_{i},\quad\sigma^{2}_{k}=\frac{1}{b}\sum_{i}(x_{i}-\mu_{k})^{2}$

Here, $k$ denotes the $k$-th training step, and $\rho$ is the momentum with a
default value of 0.1. BN statistics are then fixed (denoted as $\mu$ and
$\sigma^{2}$) and used for normalization when the model is set to evaluation
mode (pyt, 2024). During evaluation, we apply test-time adaptation to
calibrate the BN statistics, _i.e._ the mean ($\mu$) and variance
($\sigma^{2}$). The BN layer is formulated as follows:

(12) $x_{out}=\frac{x_{in}-\mu}{\sqrt{\sigma^{2}+\epsilon}}*\gamma+\beta$

where $x_{in}$ is the BN layer input, $x_{out}$ is the output. $\gamma$ and
$\beta$ are BN learnable parameters. $\epsilon$ is added for numerical
stability.

During model training, since only the source domain data is passed through the
classifier, BN statistics are computed based on the source domain statistics.
During model evaluation, the target domain data is passed through the
classifier. To integrate target domain statistics into the BN statistics,
after each training epoch, we pass the unlabeled test set through the
classifier. After that, we set the classifier to evaluation mode and evaluate
it on the test set. In this way, the running estimates would keep the target
domain statistics as well into record, which is helpful for the domain
adaptation.

##  5\. Experimental Design

###  5.1. Datasets

We conduct experiments on Twitter-COMMs and NewsCLIPpings, which are the only
two out-of-context news datasets that conform to our problem definition
(Section 3).

Twitter-COMMs (Biamby et al., 2022): This dataset is collected from Twitter
solely on three topics: Covid-19 (Cv), Climate Change (Cl) and Military
Vehicles (M). Each original tweet is represented by an image and its
associated caption (text). Given a caption, the authors create out-of-context
news by either randomly choosing an image (random negative) or by choosing the
image whose original associated text is semantically the closest (hard
negative). After running the script provided by the authors to download tweets
and removing tweets that has an unavailable image, we obtain 2,143,934 items
(Cv: 1,387,043, Cl: 512,490, M: 244,401) in the training set, and 22,082 items
(Cv: 6,456, Cl: 8,488, M: 7,138) in the test set. The labels have a balanced
distribution. Hence, we address domain adaptation among different news
_topics_ in Twitter-COMMs.

NewsCLIPpings (Luo et al., 2021): This dataset is derived from the VisualNews
dataset (Liu et al., 2020), which is a benchmark news image captioning
dataset. The news is collected from four agencies, namely Guardian (G), BBC
(B), Washington Post (W) and USA Today (U). The NewsCLIPpings dataset is
originally partitioned into four splits in which ways the out-of-context news
is constructed. However, the analysis on the underlying VisualNews dataset
(Liu et al., 2020) reveals that there are inter correlations among data from
different news agencies (_e.g._ , the image captioning model trained on the
USA Today data achieves 3.7 in CIDEr score (Vedantam et al., 2015) when tested
on the USA Today test set, yet only achieves 0.6 on the Guardian test set. ).
Since NewsCLIPping is derived from VisualNews, we _re-partition_ the dataset
according to the news agencies. After downloading the dataset, we obtain a
total number of 1,182,900 items (B: 205,968, G: 567,012, U: 228,393, W:
181,527) in the training set, and 124,168 items (B: 22,074, G: 59,163, U:
24,243, W: 18,688) in the test set. The labels have a balanced distribution.
Hence, we address domain adaptation among different news _agencies_ in
NewsCLIPpings.

###  5.2. Implementation Details

Table 1. Evaluation of domain adaptation performance on TwitterCOMMs and NewsCLIPpings. Cv, Cl and M stand for covid-19, climate change and military vehicles respectively. B, G, U and W stand for BBC, Guardian, USA Today and Washington Post respectively. In X $\rightarrow$ Y, X denotes the source domain, Y denotes the target domain. The best performance is in bold text. The second best performance is underlined. Model | Twitter-COMMs | NewsCLIPpings
---|---|---
Cv, Cl $\rightarrow$ M | Cv, M $\rightarrow$ Cl | Cl, M $\rightarrow$ Cv | U, W $\rightarrow$ B | U, W $\rightarrow$ G | B, G $\rightarrow$ U | B, G $\rightarrow$ W
F1 |  Acc |  F1 |  Acc |  F1 |  Acc |  F1 |  Acc |  F1 |  Acc |  F1 |  Acc |  F1 |  Acc
Source only |  70.00 |  72.36 |  78.48 |  78.83 |  76.92 |  77.87 |  67.19 |  67.90 |  75.08 |  75.23 |  79.77 |  79.94 |  78.60 |  78.67
Source and target |  80.60 |  80.68 |  81.47 |  81.48 |  81.50 |  81.58 |  73.80 |  74.71 |  76.00 |  77.96 |  82.70 |  84.37 |  79.80 |  80.74
EANN |  76.33 |  77.26 |  79.13 |  79.16 |  79.44 |  79.60 |  69.30 |  69.61 |  76.01 |  76.12 |  79.72 |  79.88 |  78.27 |  78.34
MDA-WS |  76.83 |  77.09 |  76.12 |  76.99 |  78.39 |  78.55 |  69.62 |  69.79 |  74.88 |  74.89 |  79.54 |  79.73 |  78.22 |  78.23
CANMD |  67.13 |  68.16 |  79.87 |  80.53 |  79.91 |  80.01 |  68.42 |  68.62 |  74.75 |  75.64 |  79.30 |  79.69 |  79.93 |  78.59
REAL-FND |  73.50 |  74.82 |  79.13 |  79.36 |  76.64 |  77.62 |  69.70 |  69.96 |  75.20 |  75.30 |  79.95 |  80.00 |  78.50 |  78.52
ConDA-TTA |  79.26 |  79.34 |  80.80 |  80.80 |  80.06 |  80.08 |  71.52 |  71.80 |  75.70 |  75.86 |  80.77 |  80.84 |  77.17 |  77.31

Multimodal Feature Encoder Selection: To find the optimal MLLM feature
extractor, we experiment with (i) BLIP-2 (Li et al., 2023b) multimodal and
(ii) unimodal feature extractor, (iii) CLIP (Radford et al., 2021) (only has
unimodal feature extractors) and (iv) ALBEF (Li et al., 2021b) multimodal
feature extractor. For (ii) and (iii), we experiment with vector concatenation
and element-wise multiplication to obtain the multimodal feature vector.

We then train a one-layer linear classifier on the four features respectively
and evaluate the performances. Experimental results show that BLIP-2
multimodal features gives the best performance (in terms of accuracy) on both
the training and test sets. Note, we also evaluate the zero-shot performance
of the BLIP-2 Image Text Matching head which only outputs a similarity score
rather than the feature vector. We find it underperforms (around 0.5 in
accuracy on both training and test set) using BLIP-2 multimodal feature
directly.

Architecture: We use BLIP-2’s blip2_feature_extractor with the multimodal mode
to embed the image and text, and obtain the multimodal embedding x of size
768. The projection head is composed of two linear layers with 768 and 500
(experimented with {300, 500, 768}) neurons respectively. The classifier is
composed of three linear layers with 768, 768 and 2 neurons respectively, and
two batch normalization layers following the first two linear layers. We use
Tanh() as the activation function. We apply dropout both before the first
linear layer and the third linear layer. The dropout rate is set to 0.2 (we
experimented with {0.2, 0.5} and without dropout).

Training: We perform training using the Adam optimizer with a batch size of
256 for 20 epochs. The early stopping epoch is set to 5. The learning rate is
set to 2e-4 for the Twitter-COMMs dataset and 1e-4 for the NewsCLIPpings
dataset. For the loss function, $\lambda_{MMD}$ is set to 1 and both
$\lambda_{CE}$ and $\lambda_{ctr}$ are set to 0.5.

Source and Target Domain Partition: The Twitter-COMMs dataset consists of
three news topics, which we treat as individual domains. We use one topic as
the target domain and the remaining two topics as the source domain. Thus, we
consider three domain adaptation settings: (i) {Cv, Cl} $\rightarrow$ M, (ii)
{Cv, M} $\rightarrow$ Cl, and (iii) {Cl, M} $\rightarrow$ Cv. The
NewsCLIPpings dataset consists of four news agencies. Based on an observation
(Liu et al., 2020) on the correlation of these four news sources and our pilot
experiments, we use two news sources from the same country as the target
domain and the rest two as the source domain respectively. As such, we
consider four domain adaptation settings: (i) {U, W} $\rightarrow$ B, (ii) {U,
W} $\rightarrow$ G, (iii) {B, G} $\rightarrow$ U and (iv) {B, G} $\rightarrow$
W.

###  5.3. Baseline Models

Considering that no prior work has looked into domain adaptive out-of-context
news detection, we select SOTA baseline models from a similar task: domain
adaptive fake news detection. For fair comparison, we use BLIP-2’s
blip2_feature_extractor with the multimodal mode to encode the image and text
into a multimodal feature vector.

Source only serves as the naive baseline for the chosen backbone feature
encoder. It uses one linear layer and the softmax function as the classifier.
During training, it only uses the source domain.

Source and target uses the same architecture as source only. The difference is
that, during training, it uses all domains, including both the source and
target domains. Therefore, it can be viewed as the upper bound performance.

EANN (Wang et al., 2018) is a multimodal fake news detection model that uses
adversarial learning to derive event-invariant features which benefit fake
news detection on newly arrived events. Since the original EANN implementation
encodes the image and text modality separately, we modify it to accommodate
multimodal inputs.

MDA-WS (Li et al., 2021a) is an early fake news detection framework based on
multi-source domain adaptation and weak supervision. It uses adversarial
learning to learn the domain-invariant feature, and incorporates prior
knowledge to assign pseudo labels to target domain news for weak supervision.

CANMD (Yue et al., 2022) is a domain adaptive fake news detection model. It
firstly pre-trains the model on the source training data and then uses the
pseudo labels generated from it on the target training data to adapt the
model. During adaptation, it aims to reduce intra-class discrepancy and
enlarge inter-class discrepancy.

REAL-FND (Mosallanezhad et al., 2022) is a reinforcement-learning (RL) based
domain adaptive fake news detection model. It first obtains a reasonable news
representation and then trains the fake news and domain classifiers. Using the
representation as the state and the classifiers’ losses as the reward, it then
trains an RL agent that learns a domain-invariant news representation.

##  6\. Result and Analysis

Based on the above experimental setup, we evaluate our proposed ConDA-TTA and
investigate the contribution of each model component afterwards. We also
include TSNE visualization and parameter sensitivity analysis.

###  6.1. Experimental Result

Table 2. Evaluation results on ablating $\mathcal{L}_{ctr}$, MMD and TTA of ConDA-TTA. Cv, Cl and M stand for covid-19, climate change and military vehicles respectively. B, G, U and W stand for BBC, Guardian, USA Today and Washington Post respectively. In X $\rightarrow$ Y, X denotes the source domain, Y denotes the target domain. The best performance is in bold text. The second best performance is underlined. Model | Twitter-COMMs | NewsCLIPpings
---|---|---
Cv, Cl $\rightarrow$ M | Cv, M $\rightarrow$ Cl | Cl, M $\rightarrow$ Cv | U, W $\rightarrow$ B | U, W $\rightarrow$ G | B, G $\rightarrow$ U | B, G $\rightarrow$ W
F1 |  Acc |  F1 |  Acc |  F1 |  Acc |  F1 |  Acc |  F1 |  Acc |  F1 |  Acc |  F1 |  Acc
w/o $\mathcal{L}_{ctr}$ |  78.07 |  78.12 |  80.59 |  80.59 |  80.10 |  80.11 |  70.80 |  70.97 |  75.07 |  75.28 |  80.27 |  80.33 |  77.33 |  77.50
w/o MMD |  75.85 |  75.92 |  80.47 |  80.48 |  79.14 |  79.17 |  71.38 |  71.46 |  75.64 |  75.80 |  79.93 |  79.98 |  77.22 |  77.35
w/o TTA |  78.58 |  78.59 |  80.73 |  80.73 |  80.52 |  80.58 |  70.20 |  70.26 |  75.68 |  75.68 |  80.75 |  80.77 |  77.89 |  77.89
ConDA-TTA |  79.26 |  79.34 |  80.80 |  80.80 |  80.06 |  80.08 |  71.52 |  71.80 |  75.70 |  75.86 |  80.77 |  80.84 |  77.17 |  77.31

Table 1 shows the evaluation performance of ConDA-TTA and the baseline models.
We use F1 and accuracy (Acc) as the evaluation metrics.

Naive Baseline and Upper Bound: We first present our observations on the naive
baseline (source only in Table 1) and the supervised training (source and
target in Table 1) results. In our experiments, we find that using BLIP-2’s
multimodal representation with a plain linear classifier already achieves
strong performances (around 70 - 80% zero-shot accuracy under most domain
adaptation scenarios) on this task. Nevertheless, we observe around 2-10%
performance gap between the naive baseline and the supervised training result.
Considering that BLIP-2 has 2.7b parameters, it is not trivial to apply fine-
tuning to improve its domain adaptation performance. This suggests the need
for developing domain adaptation techniques on top of it to mitigate the gap.

For the news topics in Twitter-COMMs, adapting to Military Vehicles (see
Section 5.1) has the largest domain adaptation gap (10.6% in F1 and 8.32% in
accuracy), whilst adapting to Climate Change and Covid-19 has a relatively
smaller domain adaptation gap (2.99% in F1 and 2.65% in accuracy, 4.58% in F1
and 3.71% in accuracy). We conjecture that the differences are potentially
caused by the different news styles. We manually check a batch of 10 randomly
sampled news from each topic. We find that Climate Change and Covid-19 tend to
include images with more abundant information, including things such as
embedded texts, numbers, and even news screenshots. However, for the Military
Vehicles domain, the news’ image usually contains one clear subject: the
military vehicle itself. At the same time, it requires more specialized
knowledge to determine whether the image pairs the text within this topic.

For the news agencies in NewsCLIPpings, adapting to BBC and USA Today has the
largest domain adaptation gap (6.61% in F1 and 6.81% in accuracy, 2.93% in F1
and 4.43% in accuracy), while adapting to the Guardian and the Washington Post
has a relatively smaller gap. Although the gap is not consistently
significant, it can affect heavily if deployed in a news platforms where a
large amount of news is posted everyday. Thus, we believe that leveraging
these powerful MLLM feature encoders and further mitigating its domain
adaptation gap with a small training cost are crucial to achieve promising
performance on unlabeled, newly emergent domains.

Baseline Comparison: We next compare our model’s performance with the baseline
models. On both datasets, our proposed ConDA-TTA outperforms the baseline
models across most metrics. Notably, on the Twitter-COMMs dataset, ConDA-TTA
outperforms the best baseline model by 2.93% in F1 and by 2.08% in accuracy
when the source domains are Covid-19 (Cv) and Climate Change (Cl), and the
target domain is Military Vehicles (M). Specifically, we observe negative
transfer for CANMD under this setting. Moreover, ConDA-TTA achieves 80.80% on
both F1 and accuracy when the source domains are Covid-19 and Military
Vehicles, and the target domain is Climate Change, surpassing the baseline
performances by 0.93% and 0.27% respectively. Additionally, it achieves 80.06%
and 80.08% on F1 and accuracy when the source domains are Climate Change and
Military Vehicles, and the target domain is Covid-19.

On the NewsCLIPpings dataset, when the source domains are USA Today (U) and
Washington Post (W) and the target domain is BBC (B), our model outperforms
the best baseline model by 1.84% in accuracy and 1.82% F1 accuracy. When the
source domains are BBC (B) and Guardian (G) and the target domain is USA Today
(U), our model outperforms the best baseline model by 0.82% in F1 and 0.84% in
accuracy. When the source domains are USA Today and Washington Post, and the
target domain is Guardian, our model achieves the second best performance in
both metrics. However, we also observe that when the source domains are BBC
and Guardian, and target domain is Washington Post, all models see slightly
negative transfer in terms of accuracy. This is in-line with the observations
(Section 5.1) on the underlying dataset (Liu et al., 2020), where the authors
find that training their image captioning model on either the BBC or Guardian
results in a comparatively low CIDEr score on the Washington Post. As such, we
conjecture that the information contained in the BBC and Guardian is not
enough for enhancing the model’s domain adaptation performance on the
Washington Post.

In summary, ConDA-TTA outperforms the baselines in most of the domain
adaptation settings. The improvements are more significant when the room for
domain adaptation is larger ({Cv, Cl $\rightarrow$ M}, {U, W $\rightarrow$
G}).

###  6.2. Ablation Study

We next present an ablation study on the different components in ConDA-TTA.
Specifically, we ablate $\mathcal{L}_{ctr}$, MMD and TTA from ConDA-TTA. The
evaluation results are summarized in Table 2.

On the Twitter-COMMs dataset, we observe that ablating MMD causes the biggest
performance drop in all three domain adaptation settings, respectively: 3.41%
(F1) and 3.42% (Acc), 0.33% (F1) and 0.32% (Acc), 0.92% (F1) and 0.91% (Acc).
This suggests that MMD is the most important component in ConDA-TTA.
Additionally, ablating the contrastive loss results in the second biggest
performance decrease when the target domain is Military Vehicles and Climate
Change, respectively: 1.19% (F1) and 1.22% (Acc), 0.21% (F1) and 0.21% (Acc).
However, we see a marginal performance increase (0.04% in F1 and 0.03% in Acc)
when the target domain is Covid-19. Lastly, ablating TTA causes a 0.68% (F1)
and 0.75% (Acc) performance drop when the target domain is Military Vehicles,
and causes a 0.07% performance drop in both F1 and Acc when the target domain
is Climate Change. When the target domain is Covid-19, we again see a slight
increase (0.46% in F1 and 0.50% in Acc) in performance. Although ConDA-TTA (as
a whole) improves the domain adaptation performance on Covid-19 (Section 6.1),
not every component of it contributes under this setting.

On the NewsCLIPpings dataset, we observe similar degradation when ablating
these three components when the target domain is the BBC, Guardian and USA
Today. Specifically, when ablating TTA, the performance decreases (1.22% in F1
ad 1.53% in Acc) most significantly on BBC, while less significantly on
Guardian (0.02% in F1 and 0.18% in Acc) and USA Today (0.02% in F1 and 0.07%
in Acc). When ablating MMD, F1 drops by 0.04%, 0.06% and 0.84%, and the Acc
drops by 0.04%, 0.06% and 0.86%. When ablating the contrastive loss, the F1
drops by 0.62%, 0.63% and 0.5%, and the Acc drops by 0.53%, 0.58% and 0.51%.
However, when adapting to Washington Post, ablating the contrastive loss, MMD
and TTA all bring slight performance increase (0.04% - 0.72%). We conjecture
that this observation echos previous analysis (Section 6.1) that the
information contained in BBC and Guardian is not enough for assisting the
domain adaptation on Washington Post, making each component less effective.

In summary, the above results confirm the efficacy of the contrastive loss,
MMD and TTA in most domain adaptation scenarios on both datasets. Concretely,
MMD tends to contribute the most to the domain adaptation on Twitter-COMMs;
TTA tends to contribute the most on NewsCLIPpings. However, we note that, in
some cases, the techniques are less effective on the NewsCLIPpings dataset. We
believe this is due to that under some settings, the source domain contains
less abundant information for adaptation to the target domain (Section 6.1).

###  6.3. Visualization

To show that ConDA-TTA effectively learns the domain-invariant feature, we
adopt TSNE (Van der Maaten and Hinton, 2008) to visualize the multimodal
embeddings x (also the input embedding to ConDA-TTA) and the learned domain-
invariant feature z projected to the 2-d space (Figure 3).

Figure 3. TSNE visualization of the multimodal feature x and the learned
domain-invariant feature z under Cv, Cl $\rightarrow$ M.

Here we illustrate the TSNE visualization of setting Cv, Cl $\rightarrow$ M.
We can tell from the figure that feature z from different topics blend better
than x, where we see clearer boundaries between different topics. This
suggests that our model has removed domain-specific features from x to some
extent. This further shows that our approach is able to effectively learn
domain-invariant feature, and transfers out-of-context news detection
knowledge to the target domain (Table 1).

###  6.4. Sensitivity Analysis

The contrastive loss and the MMD are two crucial components in our approach.
This section briefly studies how their weights, $\lambda_{ctr}$ and
$\lambda_{MMD}$, in the loss function affect the performance of ConDA-TTA.

Figure 4. ConDA-TTA’s performance (in Acc) with different $\lambda_{MMD}$ and
$\lambda_{ctr}$ values. The legend denotes the target domain.

Figure 4 illustrates ConDA-TTA’s performance with
$\lambda_{MMD}=\\{0.5,1,2,5\\}$ and $\lambda_{ctr}=\\{0.1,0.5,1,2\\}$. When
setting $\lambda_{MMD}$ to 1, our model achieves the best performance in 4 (M,
Cl, B, U) out of 7 domain adaptation settings, and achieves close to best
performance in the remaining 3 settings. Overall, $\lambda_{MMD}$ being too
high causes the accuracy to drop a little. This might be because the model has
put comparatively less emphasis on learning the detection of out-of-context
news (_i.e._ the cross-entropy loss in Eq. (9) receives less attention). For
$\lambda_{ctr}$, we find that setting it to 0.5 achieves the best performance
in 5 settings. When the value goes up, the model performance tends to decrease
gradually. We conjecture this is because the similarity (Section 4.2) among
original news becomes smaller, making it slightly difficult to find the
decision boundary.

##  7\. Conclusion

This paper has proposed ConDA-TTA, a domain adaptive out-of-context news
detection model that can adapt to both unlabeled news topics and news
agencies. ConDA-TTA first encodes the news using the BLIP-2 multimodal feature
encoder and then adopts contrastive learning and MMD to learn the domain-
invariant features. During test-time, we further incorporate target domain
statistics into the classifier of ConDA-TTA, in order to better adapt the
model to the target domain. Experimental results show the efficacy of our
approach. These advantages enable ConDA-TTA to transfer the learned out-of-
context news detection knowledge to emerging events or news platforms,
lessening the burden of online news content moderators.

## References

  * (1)
  * pyt (2024) 2024\.  BATCHNORM2D.  https://pytorch.org/docs/stable/generated/torch.nn.BatchNorm2d.html/.
  * Abdelnabi et al. (2022) Sahar Abdelnabi, Rakibul Hasan, and Mario Fritz. 2022.  Open-domain, content-based, multi-modal fact-checking of out-of-context images via online resources. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 14940–14949.
  * Bhattacharjee et al. (2023) Amrita Bhattacharjee, Tharindu Kumarage, Raha Moraffah, and Huan Liu. 2023.  ConDA: Contrastive Domain Adaptation for AI-generated Text Detection. In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Nusa Dua, Bali, 598–610.
  * Biamby et al. (2022) Giscard Biamby, Grace Luo, Trevor Darrell, and Anna Rohrbach. 2022.  Twitter-COMMs: Detecting Climate, COVID, and Military Multimodal Misinformation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, Seattle, United States, 1530–1549.
  * Chi et al. (2023) Zhixiang Chi, Li Gu, Tao Zhong, Huan Liu, YUANHAO YU, Konstantinos N Plataniotis, and Yang Wang. 2023.  Adapting to Distribution Shift by Visual Domain Prompt Generation. In _The Twelfth International Conference on Learning Representations_.
  * Dai et al. (2024) Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. 2024.  Instructblip: Towards general-purpose vision-language models with instruction tuning.  _Advances in Neural Information Processing Systems_ 36 (2024).
  * Dhariwal and Nichol (2021) Prafulla Dhariwal and Alexander Nichol. 2021.  Diffusion models beat gans on image synthesis.  _Advances in neural information processing systems_ 34 (2021), 8780–8794.
  * Ding et al. (2022) Yasan Ding, Bin Guo, Yan Liu, Yunji Liang, Haocheng Shen, and Zhiwen Yu. 2022.  MetaDetector: Meta Event Knowledge Transfer for Fake News Detection.  _ACM Trans. Intell. Syst. Technol._ 13, 6, Article 93 (sep 2022), 25 pages.
  * Dolhansky et al. (2020) Brian Dolhansky, Joanna Bitton, Ben Pflaum, Jikuo Lu, Russ Howes, Menglin Wang, and Cristian Canton Ferrer. 2020.  The deepfake detection challenge (dfdc) dataset.  _arXiv preprint arXiv:2006.07397_ (2020).
  * Fazio (2020) Lisa Fazio. 2020.  Out-of-context photos are a powerful low-tech form of misinformation.  https://theconversation.com/out-of-context-photos-are-a-powerful-low-tech-form-of-misinformation-129959/.
  * Gretton et al. (2012) Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. 2012.  A kernel two-sample test.  _The Journal of Machine Learning Research_ 13, 1 (2012), 723–773.
  * Li et al. (2023a) Jingyao Li, Pengguang Chen, Zexin He, Shaozuo Yu, Shu Liu, and Jiaya Jia. 2023a.  Rethinking Out-of-distribution (OOD) Detection: Masked Image Modeling is All You Need. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 11578–11589.
  * Li et al. (2023b) Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b.  Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.  _arXiv preprint arXiv:2301.12597_ (2023).
  * Li et al. (2021b) Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021b.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ 34 (2021), 9694–9705.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019.  Visualbert: A simple and performant baseline for vision and language.  _arXiv preprint arXiv:1908.03557_ (2019).
  * Li et al. (2021a) Yichuan Li, Kyumin Lee, Nima Kordzadeh, Brenton Faber, Cameron Fiddes, Elaine Chen, and Kai Shu. 2021a.  Multi-source domain adaptation with weak supervision for early fake news detection. In _2021 IEEE International Conference on Big Data (Big Data)_. IEEE, 668–676.
  * Li et al. (2016) Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. 2016.  Revisiting batch normalization for practical domain adaptation.  _arXiv preprint arXiv:1603.04779_ (2016).
  * Liang et al. (2023) Jian Liang, Ran He, and Tieniu Tan. 2023.  A comprehensive survey on test-time adaptation under distribution shifts.  _arXiv preprint arXiv:2303.15361_ (2023).
  * Lin et al. (2023) Hongzhan Lin, Pengyao Yi, Jing Ma, Haiyun Jiang, Ziyang Luo, Shuming Shi, and Ruifang Liu. 2023.  Zero-shot rumor detection with propagation structure via prompt learning. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 37. 5213–5221.
  * Liu et al. (2020) Fuxiao Liu, Yinghan Wang, Tianlu Wang, and Vicente Ordonez. 2020.  Visual news: Benchmark and challenges in news image captioning.  _arXiv preprint arXiv:2010.03743_ (2020).
  * Long et al. (2015) Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. 2015.  Learning transferable features with deep adaptation networks. In _International conference on machine learning_. PMLR, 97–105.
  * Luo et al. (2021) Grace Luo, Trevor Darrell, and Anna Rohrbach. 2021.  NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 6801–6817.
  * Mosallanezhad et al. (2022) Ahmadreza Mosallanezhad, Mansooreh Karami, Kai Shu, Michelle V Mancenido, and Huan Liu. 2022.  Domain adaptive fake news detection via reinforcement learning. In _Proceedings of the ACM Web Conference 2022_. 3632–3640.
  * Mu et al. (2023) Michael Mu, Sreyasee Das Bhattacharjee, and Junsong Yuan. 2023.  Self-Supervised Distilled Learning for Multi-modal Misinformation Identification. In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_. 2819–2828.
  * Nan et al. (2022) Qiong Nan, Danding Wang, Yongchun Zhu, Qiang Sheng, Yuhui Shi, Juan Cao, and Jintao Li. 2022.  Improving fake news detection of influential domain via domain-and instance-level transfer.  _arXiv preprint arXiv:2209.08902_ (2022).
  * Niu et al. (2022) Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. 2022.  Efficient test-time model adaptation without forgetting. In _International conference on machine learning_. PMLR, 16888–16905.
  * Niu et al. (2023) Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. 2023.  Towards stable test-time adaptation in dynamic wild world.  _arXiv preprint arXiv:2302.12400_ (2023).
  * Pan et al. (2010) Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. 2010.  Domain adaptation via transfer component analysis.  _IEEE transactions on neural networks_ 22, 2 (2010), 199–210.
  * Qi et al. (2024) Peng Qi, Zehong Yan, Wynne Hsu, and Mong Li Lee. 2024.  SNIFFER: Multimodal Large Language Model for Explainable Out-of-Context Misinformation Detection.  _arXiv preprint arXiv:2403.03170_ (2024).
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Shalabi et al. (2024) Fatma Shalabi, Hichem Felouat, Huy H Nguyen, and Isao Echizen. 2024.  Leveraging Chat-Based Large Vision Language Models for Multimodal Out-Of-Context Detection.  _arXiv preprint arXiv:2403.08776_ (2024).
  * Shalabi et al. (2023) Fatma Shalabi, Huy H Nguyen, Hichem Felouat, Ching-Chun Chang, and Isao Echizen. 2023.  Image-Text Out-Of-Context Detection Using Synthetic Multimodal Misinformation. In _2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_. IEEE, 605–612.
  * Tian et al. (2020) Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. 2020.  What makes for good views for contrastive learning?  _Advances in neural information processing systems_ 33 (2020), 6827–6839.
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, 11 (2008).
  * Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015.  Cider: Consensus-based image description evaluation. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 4566–4575.
  * Wang et al. (2020) Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. 2020.  Tent: Fully test-time adaptation by entropy minimization.  _arXiv preprint arXiv:2006.10726_ (2020).
  * Wang et al. (2022) Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. 2022.  Continual test-time domain adaptation. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 7201–7211.
  * Wang et al. (2018) Yaqing Wang, Fenglong Ma, Zhiwei Jin, Ye Yuan, Guangxu Xun, Kishlay Jha, Lu Su, and Jing Gao. 2018.  Eann: Event adversarial neural networks for multi-modal fake news detection. In _Proceedings of the 24th acm sigkdd international conference on knowledge discovery & data mining_. 849–857.
  * Yuan et al. (2021) Hua Yuan, Jie Zheng, Qiongwei Ye, Yu Qian, and Yan Zhang. 2021.  Improving fake news detection with domain-adversarial and graph-attention neural network.  _Decision Support Systems_ 151 (2021), 113633.
  * Yuan et al. (2023) Xin Yuan, Jie Guo, Weidong Qiu, Zheng Huang, and Shujun Li. 2023.  Support or Refute: Analyzing the Stance of Evidence to Detect Out-of-Context Mis-and Disinformation.  _arXiv preprint arXiv:2311.01766_ (2023).
  * Yue et al. (2022) Zhenrui Yue, Huimin Zeng, Ziyi Kou, Lanyu Shang, and Dong Wang. 2022.  Contrastive domain adaptation for early misinformation detection: A case study on covid-19. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_. 2423–2433.
  * Zhang et al. (2021) Huaiwen Zhang, Shengsheng Qian, Quan Fang, and Changsheng Xu. 2021.  Multimodal Disentangled Domain Adaption for Social Media Event Rumor Detection.  _IEEE Transactions on Multimedia_ 23 (2021), 4441–4454.  https://doi.org/10.1109/TMM.2020.3042055
  * Zhang et al. (2020) Tong Zhang, Di Wang, Huanhuan Chen, Zhiwei Zeng, Wei Guo, Chunyan Miao, and Lizhen Cui. 2020.  BDANN: BERT-Based Domain Adaptation Neural Network for Multi-Modal Fake News Detection. In _2020 International Joint Conference on Neural Networks (IJCNN)_. 1–8.
  * Zhang et al. (2023) Yizhou Zhang, Loc Trinh, Defu Cao, Zijun Cui, and Yan Liu. 2023.  Detecting out-of-context multimodal misinformation with interpretable neural-symbolic model.  _arXiv preprint arXiv:2304.07633_ (2023).
  * Zhao et al. (2023) Bowen Zhao, Chen Chen, and Shu-Tao Xia. 2023.  DELTA: degradation-free fully test-time adaptation.  _arXiv preprint arXiv:2301.13018_ (2023).