# Multimodal Automated Fact-Checking: A Survey

Mubashara Akhtar1, Michael Schlichtkrull2, Zhijiang Guo2, Oana Cocarascu1,
Elena Simperl1 Andreas Vlachos2
1Department of Informatics, King’s College London
2Department of Computer Science and Technology, University of Cambridge
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Misinformation, i.e. factually incorrect information, is often conveyed in
multiple modalities, e.g. an image accompanied by a caption. It is perceived
as more credible by humans, and spreads faster and wider than its text-only
counterparts. While an increasing body of research investigates automated
fact-checking (AFC), previous surveys mostly focus on textual misinformation.
In this survey, we conceptualise a framework for AFC including subtasks unique
to multimodal misinformation. Furthermore, we discuss related terminological
developed in different communities in the context of our framework. We focus
on four modalities prevalent in real-world fact-checking: text, image, audio,
and video. We survey benchmarks and models, and discuss limitations and
promising directions for future research.

##  1 Introduction

Motivated by the challenges presented by misinformation in the modern media
ecosystem, previous research has commonly modelled automated fact-checking
(AFC) as a pipeline consisting of different steps, surveyed in a variety of
axes (Thorne and Vlachos, 2018; Kotonya and Toni, 2020a; Zeng et al., 2021;
Nakov et al., 2021; Guo et al., 2022). However, these surveys focus on a
single modality, text. This is different to real-world misinformation that
often occurs via several modalities. ††1https://www.bbc.com/news/world-us-
canada-65069316

Figure 1:  Manipulated image depicting arrest of former US president Donald
Trump (source: BBC222https://www.bbc.com/news/world-us-canada-65069316).

In AFC, the term _multimodal_ has been used to refer to cases where the claim
and/or evidence are expressed through different or multiple modalities
(Hameleers et al., 2020; Alam et al., 2022; Biamby et al., 2022). Examples of
multimodal misinformation encountered in fact-checking include: $(i)$ claims
made about digitally manipulated content (Agarwal et al., 2019; Rössler et
al., 2018) such as photos depicting former US president Trump’s arrest in
March 2023 (see Figure 1); $(ii)$ combining content from different modalities
and contexts, e.g. using an (unchanged) video footage in a misleading context
(Aneja et al., 2021; Biamby et al., 2022; Abdelnabi et al., 2022); $(iii)$
embedding a claim in another modality, e.g. a meme, an image with embedded
text (Qu et al., 2022a), notable real-world examples include a much-shared
Vote Leave poster from the Brexit
campaign333https://www.itv.com/news/2019-01-18/boris-johnson-under-attack-
over-turkey-claim/ and TikTok videos with COVID misinformation (Shang et al.,
2021); $(iv)$ verifying a claim with evidence from a different modality than
the input claim, e.g. Aly et al. (2021a) and Chen et al. (2020) verify text
claims using tabular data while Akhtar et al. (2023) use images of data
visualisations as evidence.

Detecting multimodal misinformation and developing appropriate fact-checking
methods have become important for a number of reasons. First, multimodal
content is perceived as more credible compared to text containing a similar
claim (Newman et al., 2012). For example, previous research shows that visual
content exhibits a “photo truthiness”-effect (Newman and Zhang, 2020), biasing
readers to believe a claim is true. Second, multimodal content spreads faster
on social media and has a higher user engagement than text-only posts (Li and
Xie, 2020). Third, with recent advances in generative machine learning models,
such as stable diffusion (Rombach et al., 2022), the generation and spreading
of multimodal misinformation has been simplified.

Claim Modality | Percentage
---|---
Image | 20.07%
Video | 8.06%
Audio | 0.55%
Total | 28.68%

Table 1: Non-textual modalities present and/or used in addition to text in our
manually annotated snapshot of real-world claims from the Google ClaimReview
API.

To validate the importance of multimodal fact-checking, we manually annotated
9,255 claims from real-world fact-checking articles collected from the Google
FactCheck ClaimReview API444https://toolbox.google.com/factcheck/apis. For
each article, we identified the modalities present in the claim and evidence
strategies (e.g. identification of geolocation) used for fact-checking.
Combining the two, we find that more than $2,600$ (28.68%) claims either
contain multimodal data or require multimodal reasoning for verification (see
Table 1).555We plan to release the annotations at
https://github.com/MichSchli/AVeriTeC.

In this survey, we introduce a framework for multimodal automated fact-
checking as a three-step process (see Figure 2). The input and output data of
each step can have different or multiple modalities. Moreover, we characterize
previous work along the following pipeline steps (Section 2): Stage $1$, claim
detection and extraction; Stage $2$, evidence retrieval; and Stage $3$,
verdict prediction encompassing veracity, manipulation and out-of-context
classification, as well as justification production. For each stage, we
discuss related terms and definitions developed in different research
communities. This is in contrast to previous surveys on multimodal fact-
checking (Cao et al., 2020; Alam et al., 2022; Abdali, 2022) focusing on
individual subtasks of the fact-checking pipeline, rather than discussing the
entire process and surveying benchmarks and modeling approaches for all
subtasks.

Further, using the same framework, we survey benchmarks and modeling
approaches (Section 3) for each pipeline step, focusing on four prevalent
modalities of real-world fact-checking identified in our annotation: text,
image, audio, and video. While tables and knowledge graphs are increasingly
used as evidence for benchmarks (Chen et al., 2020; Aly et al., 2021b; Akhtar
et al., 2022), they have been covered in previous surveys (Thorne and Vlachos,
2018; Zeng et al., 2021; Guo et al., 2022). Finally, we discuss the extent to
which current approaches to AFC work for multimodal data, and promising
directions for further research (Section 4). We accompany the survey with a
repository,666https://github.com/Cartus/Automated-Fact-Checking-Resources
which lists the resources mentioned in our survey.

##  2 Task Formulation

Figure 2:  Multimodal fact-checking pipeline.

This section introduces a conceptualisation of multimodal AFC as a three-step
process (Figure 2). First, claims are detected and extracted for fact-
checking. Then, evidence is retrieved (Stage $2$). Finally, verdicts and
justifications are produced for various types of multimodal misinformation
(Stage $3$). Compared to the pipeline presented in Guo et al. (2022), our
framework extends their first stage with a claim extraction step, and
generalises their third stage to cover tasks that fall under multimodal AFC,
thus accounting for its particular challenges.

#### Terminology.

While few works (Singhal et al., 2022; Fung et al., 2021) use the term
_multimedia_ , which is also related to multimodality and more common in
public discussions (Lauer, 2009) instead of _multimodal_ , in this survey we
adopt the latter. We align with the terminology of other tasks that use
multimodal data (Liang et al., 2022; Guo et al., 2019). The term _multimodal
fake news_ is sometimes used as a synonym for _multimodal misinformation_
(Meel and Vishwakarma, 2021; Amri et al., 2021; Patwa et al., 2022). Adopting
the terminology of previous surveys (Thorne and Vlachos, 2018; Alam et al.,
2022) and institutions such as the European
Union777https://epthinktank.eu/2022/04/21/eu-action-against-fake-news/ and the
UNO (Ireton and Posetti, 2018), we avoid using _fake news_ due to the term’s
ambiguous and wide usage in different contexts, and use _multimodal
misinformation_ in this survey instead.

#### Stage $1$: Claim Detection and Extraction.

The first pipeline stage aims to find “checkable” (i.e. factually-verifiable)
and check-worthy claims (Hassan et al., 2015b; Mishra et al., 2022; Cheema et
al., 2022). Debunking a typical claims and writing the corresponding fact-
checking article takes approximately one day for a human fact-checker Hassan
et al. (2015a). The aim of this stage is to optimize by filtering out and
focusing on claims which can be verified and are most harmful. Multimodal
claims can be diverse and belong to one of the following categories: $(1)$ a
written claim embedded in another modality (Prabhakar et al., 2021; Maros et
al., 2021) such as an image or a spoken claim in an audio or video; $(2)$ a
claim that a piece of content is authentic, e.g. that a video footage is taken
in a specific geographic location (Zhang et al., 2018; Heller et al., 2018);
$(3)$ a claim for which the evidence is manipulated to support it, e.g.
through lip-syncing (Rössler et al., 2018). Multiple modalities can be
required to understand and extract a claim at this stage. Simply detecting
misleading content is often not enough – it is necessary to extract the claim
before fact-checking it in the subsequent stages. For example, checking a text
claim embedded in visual content (e.g. a meme, an image with embedded text)
requires first detecting text in the image or video and understanding it given
the context (Qu et al., 2022b). Similarly, verifying audio first requires
transcribing and extracting claims (Maros et al., 2021).

#### Stage $2$: Evidence Retrieval.

Similarly to fact-checking with text, multimodal fact-checking relies on
evidence to make judgments Inspired by human fact-checkers, who also provide
evidence for their judgements, the aim is to retrieve data for the later
tasks. (Silverman, 2013; Nakov et al., 2021). Two main approaches have been
used in past research in multimodal AFC: $(i)$ using the claim to-be-checked
as evidence itself, e.g. to detect manipulation (Qi et al., 2019; Bonettini et
al., 2020); this can be seen as the multimodal version of evidence-free fact-
checking strategies of text claims relying on e.g. stylistic markers (Guo et
al., 2022), and $(ii)$ retrieving additional evidence (Abdelnabi et al.,
2022). In multimodal fact-checking, the evidence modality can be different
from the claim modality. For example, to retrieve evidence for image or audio
fact-checking, previous works have also used text e.g. metadata, social media
comments, or captions (Gupta et al., 2013; Huh et al., 2018; Müller-Budack et
al., 2020; Kopev et al., 2019).

#### Stage $3$: Verdict Prediction and Justification Production.

Following the fact-checking process of professional fact-checkers, the final
stage comprises verdict prediction and the production of justification that
explains the fact-check to humans (Graves, 2018). Verdict prediction is
decomposed into three tasks considering prevalent ways that multimodal
misinformation can be conveyed: manipulation, using content out-of-context,
and veracity classification.

#### Stage $3.1.$: Manipulation Classification.

Manipulation classification commonly addresses $(i)$ misinformative claims
with manipulated content; $(ii)$ correct claims accompanied by manipulated
content (e.g. to increase credibility). A range of methods are available today
to manipulate text, visual and audio content. While some require more
knowledge to use (e.g. speech synthesis), other manipulations can be achieved
easily with simple tools (e.g. changing speed of videos) (Paris and Donovan,
2019).

Different terms have been used to refer to manipulated content in recent
years, depending on its modality and the manipulation method. A _deepfake_ is
commonly defined as “the product of artificial intelligence (AI) applications
that […] create fake videos that appear authentic” (Maras and Alexandrou,
2019), with popular examples including realistic-looking videos where the
speaker’s voice or face is modified (Paris and Donovan, 2019). A related term
is cheap fake, defined as manipulated content created through more accessible
methods (Paris and Donovan, 2019), e.g. changing captions or speed of videos
(La et al., 2022). The term _fauxtography_ was first coined in journalism for
images manipulated to “convey a questionable (or outright false) sense of the
events they seem to depict” (Cooper, 2007; Kalb and Saivetz, 2007). Other
terms used in the literature for manipulated content are “fake” (Cheema et
al., 2022), “forgery” (Cozzolino et al., 2021), and “splice” (Zampoglou et
al., 2015).

#### Stage $3.2.$: Out-of-context Classification.

Using unchanged content from a different context is one of the most common and
easiest methods to create multimodal misinformation (Luo et al., 2021; Aneja
et al., 2021) and involves (possibly misinformative) textual claims paired
with content (e.g. a video) taken out of context, to increase credibility of
the claim (Zhang et al., 2018; Abdelnabi et al., 2022; Garimella and Eckles,
2020). Recent work has studied the applicability of traditional multimodal
misinformation detection methods to identify out-of-context content (Zhang et
al., 2023) showing the challenges this misinformation type poses. Terms used
in previous works for combining multimodal content in a misleading way are
_out-of-context_ (Aneja et al., 2021; Abdelnabi et al., 2022), _cross-modal
(in-)consistency_ (Müller-Budack et al., 2020), or _repurposing_ (Luo et al.,
2021).

#### Stage $3.3.$: Veracity Classification.

This task can be understood as the multimodal counterpart to the standard
fact-checking task of classifying the veracity of textual claims given
retrieved evidence (Thorne and Vlachos, 2018). Given a claim and evidence data
of a different modality, the task is to predict the claim’s veracity using
only information present in the evidence data. Veracity classification of
claims embedded in audio is also commonly referred to as deception detection
(Kopev et al., 2019; Kamboj et al., 2021). While earlier work considered
mostly claims recorded in staged setups (Newman et al., 2003) or from court
trials (Pérez-Rosas et al., 2015), more recently real-world political debates
have become popular. (Kopev et al., 2019; Kamboj et al., 2021). Verifying
statements by public figures and politicians has been at the center of human
fact-checking ever since its beginnings as well (Graves, 2018; Kamboj et al.,
2021).

#### Stage $3.4.$: Justification Production.

Previous work on automated justification production mostly concentrates on
justifications for text claims (see Kotonya and Toni (2020a) for an overview).
This is different to human fact-checkers, who also give justifications for
fact-checks involving images, audio, or video content (Silverman, 2013).
Justification production for multimodal misinformation can be grouped in three
categories: $(i)$ identifying which part of the claim input is misleading
(i.e. specific areas in a visual claim or words in a text claim, etc.) (Kou et
al., 2020; Purwanto et al., 2021; Lourenço and Paes, 2022); $(ii)$ providing
natural language justifications following human fact-checkers (Yao et al.,
2022); $(iii)$ selecting and highlighting specific evidence parts used for
verification (Atanasova et al., 2020; Shang et al., 2022). Previous work on
automated justification production mostly concentrates on justifications for
text claims (see Kotonya and Toni (2020a) for an overview). This is different
to human fact-checkers, who also give justifications for fact-checks involving
images, audio, or video content (Silverman, 2013). Justifications serve
purposes beyond explaining the verification results. Human fact-checkers also
use them to discuss uncertainties and potential errors – especially needed in
fact-checking for rapidly developing events (Silverman, 2013).

Dataset | Input | Context | Output | #Input | Lang | Source
---|---|---|---|---|---|---
Weibo (Jin et al., 2017) | Img/Txt | Meta | 2 | 9,528 | Zh | Weibo/News
FauxBuster (Zhang et al., 2018) | Img/Txt | Txt/Meta | 2 | 917 | En | Twitter/Reddit
Exfaux (Kou et al., 2020) | Img/Txt | Txt | 2/4 | 263 | En | Twitter/Reddit
MuMIN (Nielsen and McConville, 2022) | Img/Txt | Meta | 3 | 12,914 | Mul | Twitter
MMClaims (Cheema et al., 2022) | Img/Txt | - | 4 | 3,400 | En | Twitter
IndiaWApp (Garimella and Eckles, 2020) | Img | Meta | 2 | 2,500 | Mul | WhatsApp
DisinfoMeme (Qu et al., 2022a) | Img | - | 2 | 1,170 | En | Reddit
WhatsApp (Maros et al., 2021) | Aud | Meta | 2 | 42,689 | Pt | WhatsApp
TikTok (Shang et al., 2021) | Vid | Text/Meta | 2 | 891 | En | TikTok
MisDissem (Resende et al., 2019) | Vid/Aud/Img/Text | Meta | 2 | 121,781 | Pt | WhatsApp
CheckMate (Prabhakar et al., 2021) | Vid/Img/Text | Meta | 3 | 2,200 | Hi | Sharechat

Table 2:  Datasets for claim detection. Img, Txt, Vid, Aud, and Meta denote
image, text, video, audio, and metadata, respectively. Output indicates the
number classification labels. Mul indicates that the input has multiple
languages.

##  3 Datasets and Modeling Approaches

###  3.1 Stage $1$: Claim Detection and Extraction

Table 2 organises the datasets for claim detection and extraction.

Input. Benchmarks include claim detection in single (e.g. only audio (Maros et
al., 2021)) and multiple modalities (e.g. video/image/text (Prabhakar et al.,
2021)). For datasets with unimodal input (e.g. image, audio, or video) (Qu et
al., 2022b; Garimella and Eckles, 2020; Maros et al., 2021; Shang et al.,
2021), the written or spoken claim is extracted from the input first. Several
efforts (Zhang et al., 2018; Kou et al., 2020; Nielsen and McConville, 2022)
collect image-based social media posts verified by fact-checking
organizations, from a variety of platforms, including Weibo’s official rumor
debunking systems (Jin et al., 2017), Twitter (Cheema et al., 2022; Nielsen
and McConville, 2022), and Reddit (Zhang et al., 2018). These efforts assume
that the claim is given explicitly.

Output. Claim detection is typically framed as a classification task, where
models predict whether claims are factually-verifiable or check-worthy
(Prabhakar et al., 2021; Cheema et al., 2022). The verdict for factual-
verifiability is often a binary label (i.e. true/false) in datasets (Jin et
al., 2017; Shang et al., 2021). For check-worthiness, Prabhakar et al. (2021)
define three categories of check-worthy claims in multimodal content:
statistical/numerical claims, claims about world events/places/noteworthy
individuals, and other factual claims. Cheema et al. (2022) extend the binary
labels for textual check-worthiness (Hassan et al., 2015b; Barrón-Cedeño et
al., 2020; Gupta et al., 2021) with images to identify and prioritize
potentially non-factual claims. A tweet is considered check-worthy if it is
harmful, breaking news, or up-to-date.

Modeling approaches. We group existing approaches for claim detection into:
$(i)$ _content-based_ methods which only rely on the input content, and $(ii)$
_context-based_ methods that incorporate further context (e.g. metadata,
related social media activities). Early content-based systems only use surface
features of the input’s textual part for detection (Gupta et al., 2013;
Boididou et al., 2014). Later content-based efforts use neural networks to
jointly learn representations across modalities and capture the relationships
among them (Wang et al., 2018; Khattar et al., 2019; Qian et al., 2021; Wu et
al., 2021). In contrast, context-based methods encode the surrounding claim
context to inform decisions (Kochkina et al., 2017; Guo et al., 2018; Lu and
Li, 2020), e.g. tree structure networks to encode conversation threads (Ma et
al., 2018; Ma and Gao, 2020) or graph neural networks to model propagation
structure (Monti et al., 2019; Li et al., 2020a; Wei et al., 2021). Other
efforts construct heterogeneous graphs to capture multimodal data (Yuan et
al., 2019; Yang et al., 2020; Zheng et al., 2022). One other popular approach
is to use comments on social media posts to model the propagation pattern
(Zhang et al., 2018; Kou et al., 2020; Shang et al., 2021) of the input.

In real scenarios the claim can be implicitly given in the multimodal content,
and extracting it is a crucial task. In previous works, Optical Character
Recognition (OCR) systems are often used extract textual claims from visual
input. For example, viral images, memes, and videos on social media often have
text added to them, which are extracted using OCR models. Qu et al. (2022b)
and Garimella and Eckles (2020) use Google Cloud Vision API to identify the
texts in memes. For audio inputs, Maros et al. (2021) use Google’s Speech-to-
Text API to produce a transcript of audio content as the claim. For video
inputs, claim extraction becomes more challenging. Shang et al. (2021) first
sample visual frames from videos and then extract the captions and
corresponding audio chunks from the sampled frames.

###  3.2 Stage $2$: Evidence Retrieval

Previous fact-checking benchmarks have used different evidence types and
retrieval methods based on the modalities involved. Evidence retrieval for
multimodal fact-checking can be grouped into $(i)$ content-based approaches
and $(ii)$ approaches for retrieving additional evidence data (see column
_evidence_ in Table 3).

Content-based. Content-based approaches use the claim and its context (i.e.
the same information that is used for claim detection and extraction) as
evidence instead of retrieving additional evidence data. This is particularly
common for audio and video misinformation classification (Table 3). Acoustic
or visual features extracted from the input are used as evidence for verdict
prediction (Wu et al., 2015; Yi et al., 2021; Ismael Al-Sanjary et al., 2016;
Jiang et al., 2020). Most audio and video fact-checking approaches do not
include evidence retrieval (Rössler et al., 2018; Hou et al., 2019; Ismael Al-
Sanjary et al., 2016). Audio (or video) features and accompanying data (e.g.
metadata, transcripts if available) are used as evidence to identify
inconsistencies (Kopev et al., 2019; Rössler et al., 2018; Li et al., 2020b).
Several datasets with image/text claims (Tan et al., 2020; Luo et al., 2021;
Aneja et al., 2021) also do not retrieve additional evidence (Table 3).
Metadata is often used as evidence for verdict prediction tasks with images as
input (Table 3). Jaiswal et al. (2017) and Sabir et al. (2018) use metadata
(e.g. when the image was taken) to provide information beyond the visual
content. Similarly, Huh et al. (2018) incorporate EXIF image metadata (e.g.
camera version, focal length, resolution settings) to detect manipulation. For
images, context information in the form of a caption is also used sometimes.

Retrieved evidence. Retrieved evidence external to the claim is mostly used
for fact-checking text claims, text/image and image content (Table 3). Mishra
et al. (2022) use documents containing an image and text as evidence. Fung et
al. (2021) leverage a knowledge base to provide additional background
knowledge. Two recent datasets scrape claims from fact-checking websites, and
include text/image/video from those articles as evidence (Singhal et al.,
2022; Yao et al., 2022). Akhtar et al. (2023) used chart images as evidence to
verify textual claims. To determine if an image is used out-of-context to
support a claim, previous works also use _(reverse) image search_ to retrieve
evidence images (Müller-Budack et al., 2020; Abdelnabi et al., 2022), aiming
to find evidence sources with images similar to or same as the claim image.
Müller-Budack et al. (2020) query search engines and the _WikiData_ knowledge
graph using named entities from the claim text. Abdelnabi et al. (2022)
similarly query with claim captions and also apply _reverse image search_ with
the claim image as query.

Dataset | Input | Evidence | Output | Tasks | #Input | Lang | Source
---|---|---|---|---|---|---|---
MAIM (Jaiswal et al., 2017) | Img/Txt | Meta | 2 | O | 239,968 | En | Flickr
MEIR (Sabir et al., 2018) | Img/Txt | Meta | 2 | O | 140,096 | En | Flickr
TNews (Müller-Budack et al., 2020) | Img/Txt | Img | 2 | O | 72,561 | En | News
News400 (Müller-Budack et al., 2020) | Img/Txt | Img | 2 | O | 400 | En/De | News
NeuralNews (Tan et al., 2020) | Img/Txt | - | 4 | O | 128,000 | En | Grover/GoodNews
COSMOS (Aneja et al., 2021) | Img/Txt | - | 2 | O | 201,700 | En | News/Snopes
NewsCLIPings (Luo et al., 2021) | Img/Txt | - | 2 | O | 988,283 | En | CLIP/VisualNews
InfoSurgeon (Fung et al., 2021) | Img/Txt | KB/Meta | 2 | O | 30,000 | En | VoA
Factify (Mishra et al., 2022) | Img/Txt | Txt | 5 | O | 50,000 | En | Twitter
FakingSandy (Gupta et al., 2013) | Img | Txt/Meta | 2 | M | 16,117 | - | Twitter
MediaEval (Boididou et al., 2014) | Img | Txt/Meta | 2 | M | 13,924 | - | Twitter
In-the-Wild (Huh et al., 2018) | Img | Meta | 2 | M | 201 | - | Reddit/Onion
PS-Battles (Heller et al., 2018) | Img | Txt/Meta | 2 | M | 103,028 | - | Reddit
VTD (Ismael Al-Sanjary et al., 2016) | Vid | - | 2 | M | 33 | En | YouTube
Faceforensics (Rössler et al., 2018) | Vid | - | 2 | M | 1,004 | En | YouTube
DeepfakeDetect (Guera and Delp, 2018) | Vid | - | 2 | M | 600 | En | Vid Webs./HOHA
DFDC (Dolhansky et al., 2019) | Vid | - | 2 | M | 128,154 | En | Recorded
DeeperForensics-1.0 (Jiang et al., 2020) | Vid | - | 2 | M | 60,000 | En | Recorded
Celeb-DF (Li et al., 2020b) | Vid | - | 2 | M | 6,229 | En | YouTube
KoDF (Kwon et al., 2021) | Vid | - | 2 | M | 237,942 | Ko | Recorded
ASVspoof (Wu et al., 2015) | Aud | - | 2 | M | 16,375 | En | SAS
Phonespoof (Lavrentyeva et al., 2019) | Aud | - | 2 | M | 34,407 | En | ASVspoof
FoR (Reimao and Tzerpos, 2019) | Aud | - | 2 | M | 53,868 | En | TTS Systems
DeepSonar (Wang et al., 2020) | Aud | - | 2 | M | 18,614 | En/Zh | TTS Systems/VCC
HAD (Yi et al., 2021) | Aud | - | 3 | M | 88,035 | Zh | AISHELL-3
FakeAVCeleb (Khalid et al., 2021) | Vid/Aud | - | 4 | M | 20,000 | En | VoxCeleb2
MedVideo (Hou et al., 2019) | Vid | - | 2 | VC | 250 | En | YouTube
CLEF2018 Audio (Kopev et al., 2019) | Aud | Meta | 3 | VC | 286 | En | Debates
FactDrill (Singhal et al., 2022) | Txt | Vid/Aud/Img/Txt/Meta | 5 | VC | 22,435 | Mul | FC webs.
ChartFC (Akhtar et al., 2023) | Txt | Img | 2 | VC | 15,886 | En | TabFact
Fauxtography (Zlatkova et al., 2019) | Img/Txt | Meta | 2 | VC | 1,233 | En | Snopes/Reuters
MOCHEG (Yao et al., 2022) | Img/Txt | Img/Txt | 3 | VC | 21,184 | En | FC webs.
r/Fakeddit (Nakamura et al., 2020) | Img/Txt | Meta | 2/3/6 | O/M/VC | 1,063,106 | En | Reddit

Table 3:  Datasets for manipulation, out-of-context, and veracity
classification. O, M and VC denote out-of-context, manipulation and veracity
classification, respectively. Mul indicates the input has multiple languages.

###  3.3 Stage $3$: Verdict Prediction

As introduced in Section 2, the verdict prediction stage includes
manipulation, out-of-context, and veracity classification as sub-tasks.

Input. As shown in Table 3, inputs of manipulation classification datasets
usually focus on one modality. For dataset creation, manipulated images and
videos are often collected from social media platforms such as Twitter,
Reddit, and YouTube (Gupta et al., 2013; Ismael Al-Sanjary et al., 2016;
Heller et al., 2018). For verdict prediction datasets with videos further
sources such as film clips (Guera and Delp, 2018), facial expressions (Rössler
et al., 2018), and interviews (Li et al., 2020b) are used. Some works record
videos to simulate real-world scenarios (Dolhansky et al., 2019; Jiang et al.,
2020; Kwon et al., 2021). To create datasets of manipulated content, different
altering methods based on generative adversarial networks have been applied in
earlier works Zakharov et al. (2019); Nirkin et al. (2019); Karras et al.
(2019) as well. For audio manipulations, most benchmark datasets (Wu et al.,
2015; Kinnunen et al., 2017; Reimao and Tzerpos, 2019; Wang et al., 2020; Yi
et al., 2021) use speech synthesis and voice conversion algorithms on the
original speech to collect manipulated versions. To assess real-world audio
manipulations, Lavrentyeva et al. (2019) emulate realistic telephone channels.

Most out-of-context classification datasets have image-caption pairs as input
(Table 3). Jaiswal et al. (2017) replace captions of Flickr images to
construct mismatched pairs. As replacing the entire caption can be easy to
detect, later efforts (Sabir et al., 2018; Müller-Budack et al., 2020) propose
to swap specific entities among captions. Luo et al. (2021) show that such
text manipulations lead to linguistic biases and can be solved without the
images; thus they use CLIP (Radford et al., 2021) to filter out pairs that do
not require multimodal modeling. In addition to Flickr, popular sources for
OOC datasets are news/fact-checking websites.

The primary input to multimodal veracity classification is the content-to-be-
checked itself – typically text, audio or video in past benchmarks. Kopev et
al. (2019) include verified speeches from the CLEF-2018 Task 2 (Nakov et al.,
2018) while Hou et al. (2019) collect videos about prostate cancer verified by
expert urologists. Zlatkova et al. (2019) and Yao et al. (2022) collect viral
images with texts verified by dedicated agencies. Nakamura et al. (2020)
collect image-text pairs from Reddit via distant supervision, e.g. labeling a
post from the subreddit “fakefacts” as misleading and from “photoshopbattles”
as manipulated. For veracity classification of spoken claims, real-world
political debates are popular sources for claims (Kopev et al., 2019; Kamboj
et al., 2021). For example, Kopev et al. (2019) and Kamboj et al. (2021) use
claims labelled by fact checking organizations and video recordings as well as
transcripts of the respective political debates.

Output. Most manipulation and out-of-context classification datasets apply a
binary scale, depending on the task: ‘out-of-context/not out-of-context’
(Müller-Budack et al., 2020; Luo et al., 2021), ‘pristine/falsified’ (Boididou
et al., 2014; Heller et al., 2018), ‘manipulation/no manipulation’ (Dolhansky
et al., 2019; Li et al., 2020b). Following fact-checkers, veracity
classification datasets (Singhal et al., 2022; Nakamura et al., 2020) employ
multi-class labels to represent degrees of truthfulness (e.g. true, mostly-
true, half-true). Mishra et al. (2022) adopt dedicated labels to denote the
entailment between different claim and evidence modalities, e.g. the label
support text depicts that only the textual part of the evidence supports the
claim but not the accompanying image while support multimodal includes both
modalities.

Modeling Approaches. With the rise of techniques for manipulation, many
detection methods have been developed in recent years to detect manipulation.
For detection of manipulation in visual content, early approaches mostly use
CNN models, such as VGG16 (Amerini et al., 2019; Dang et al., 2020), ResNet
(Amerini et al., 2019; Sabir et al., 2019), and InceptionV3 (Guera and Delp,
2018). Some works extend CNN-based models to capture temporal aspects for
video manipulation classification. Amerini et al. (2019) adopt optical flow
fields to capture the correlation between consequent video frames and detect
dissimilarities caused by manipulation. Guera and Delp (2018) model temporal
information with an LSTM model which receives as input a sequence of features
vectors per video frame and classifies if the video is manipulated. Sabir et
al. (2019) similarly extract features for video frames and detect
discrepancies between frames using a recurrent convolution network. Most
recent models for manipulation detection integrate transformer-based
components (Wodajo and Atnafu, 2021; Wang et al., 2022; Zheng et al., 2021).

Models for out-of-context and veracity classification typically consist of
$(i)$ unimodal encoders; $(ii)$ a fusion component to obtain joint, multimodal
representations; $(iii)$ a classification component. To obtain text
embeddings, early approaches used combinations of word2vec models (Mikolov et
al., 2013), LSTMs (Hochreiter and Schmidhuber, 1997), and TF-IDF scores for
n-grams (Jin et al., 2017; Tanwar and Sharma, 2020; Hou et al., 2019). More
recent efforts use transformer-based pretrained language models (Fung et al.,
2021; Aneja et al., 2021; La et al., 2022). To encode visual data, many
approaches first detect objects in visual content using a Mask R-CNN model (He
et al., 2017) before extracting visual features (Aneja et al., 2021; La et
al., 2022; Shang et al., 2022). Visual representations for images and videos
are commonly obtained using CNN models such as ResNet (He et al., 2016;
Garimella and Eckles, 2020; Abdelnabi et al., 2022), VGG (Simonyan and
Zisserman, 2015; Jin et al., 2017; Sabir et al., 2018), and Inception (Szegedy
et al., 2015; Guera and Delp, 2018; Roy and Ekbal, 2021). To obtain audio
features for voice quality, loudness, and tonality, Shang et al. (2021)
extract the Mel-frequency cepstral coefficient, Kopev et al. (2019) use the
INTERSPEECH 2013 ComParE feature set (Eyben et al., 2013), and Hou et al.
(2019) use the openEAR toolkit (Eyben et al., 2009). Kopev et al. (2019) and
Hou et al. (2019) separately obtain audio and text representations for
transcripts before joining them in a later step.

Different approaches have been used to obtain multimodal representations for
verdict prediction with multimodal data. Early fusion, which joins
representations right after the encoding step (Baltrusaitis et al., 2019) is
more common (Aneja et al., 2021; Tanwar and Sharma, 2020; La et al., 2022)
than late fusion (Yao et al., 2022). Moreover, model-agnostic methods (e.g.
concatenation and dot product) are more prevalent (Aneja et al., 2021; Kopev
et al., 2019; Jin et al., 2017; La et al., 2022) than model-based approaches
(e.g. neural networks) (Jaiswal et al., 2017; Shang et al., 2022). A popular
approach for out-of-context classification are also _cross-modality checks_
that compare modalities present in the claim to each other (e.g. a video and
its caption) to identify out-of-context usage (Müller-Budack et al., 2020;
Fung et al., 2021).

###  3.4 Stage $3$: Justification Production

A small number of datasets for multimodal AFC are available which consider
justification production. Previous work can be grouped into two categories:
$(1)$ highlighting parts of the input, and $(2)$ generating natural language
justifications.

Highlighting Input. The first category captures input parts as justification
which influence models’ verdict prediction results. A popular approach for
selecting which parts of the input contribute to misinformation are Graph
Neural Networks (Kipf and Welling, 2017). Several papers encode multimodal
data as graph elements, naturally combining entities and relations from
multiple modalities. Models are trained to detect inconsistencies between
relations originating from different modalities, or to detect relations that
may be misinformative based on the local graph structure or their
incompatibility with an external knowledge base (Fung et al., 2021; Shang et
al., 2022; Kou et al., 2020). Predictions for specific entities and relations
then serve as explanations for the potential misinformativeness of the entire
graph. Combining subclaim-level and claim-level predictions is also used
outside of graph contexts, e.g. Zhou et al. (2018); Wu et al. (2019) use a
multitask model for manipulation classification and identification of
manipulated regions. Rather than labeled data, some papers rely on attention
mechanisms to provide explanations by highlighting areas which have high
attention weights. Bonettini et al. (2020); Dang et al. (2020) use this
approach to highlight manipulated image regions; Purwanto et al. (2021) also
include captions.

Natural Language Justifications. For the second category (i.e. natural
language justifications), Yao et al. (2022) recently introduced a multimodal
dataset with natural language justifications through scraping text and visual
content from web pages referred to by fact-checking articles. This is a
similar approach to Kotonya and Toni (2020b) who created a text-only AFC
justification production dataset. The dataset includes summaries in the fact-
checking articles as gold justifications for the verdicts. However, such a
setting is not realistic, as fact-checking articles are not available when
verifying a new claim.

##  4 Challenges and Future Directions

#### Claim extraction from multimodal content.

Efforts for multimodal claim extraction are limited, and often focus on text
extraction from visual content or transcribing audios and videos (Qu et al.,
2022b; Garimella and Eckles, 2020; Maros et al., 2021). However, the wide
range of multimodal misinformation types require claim extraction approaches
specific to their modalities and context. An example here are data
visualizations, which are commonly misused to make data support a claim (Lo et
al., 2022). Extracting the main message from a visualization (e.g. a line
chart) requires understanding the underlying data (Hoque et al., 2022), e.g.
by extracting numbers, text, and trendlines embedded in images. For claim
extraction from audio and video content, the information required to
understand and extract the claim is spread across multiple audio/video frames.

#### Multimodal evidence retrieval.

Evidence retrieval for audio and video fact-checking remains a major
challenge. Different to other modalities, they cannot be easily searched on
the web or social media networks (Silverman, 2013). Fact-checking
organizations advise starting simple when fact-checking videos, e.g. looking
for anything in the video that seems to be obviously manipulated. Moreover,
fact-checkers often use text (e.g. descriptions, tags) accompanying videos to
find evidence (Silverman, 2013). Reverse video search engines, e.g. Google
Lens or TinEye, require screenshots from the video as input – and they can
fail easily if a screenshot of the wrong timeframe is provided. A dedicated
adversary can therefore render current tools very difficult to use.

#### Multiple dimensions: Multilinguality and multimodality.

While there is increasing work on multilingual fact-checking (Gupta and
Srikumar, 2021; Shahi and Nandini, 2020; Hammouchi and Ghogho, 2022), it is
mostly limited to text-only benchmarks and models. Surveying benchmarks for
different pipeline steps (Figure 2), we found limited multimodal datasets for
non-English languages (see Tables 2 and 3). Previous work on multilingual
multimodality shows that training and testing on English data alone introduces
biases, as models fail to capture concepts and images prevalent in other
languages and cultures (Liu et al., 2021). Moreover, some types of multimodal
misinformation exploit cross-lingual sources to mislead, e.g. images or videos
from non-English newspapers appearing as out-of-context data for English
multimodal misinformation (Silverman, 2013). To prevent false conclusions and
biases, it is thus necessary to take an approach that is both multimodal and
multilingual (Ruder et al., 2022).

#### Explainability.

While explainable fact-checking has received attention recently (Kotonya and
Toni, 2020b; Atanasova et al., 2020), there is limited work on producing
justifications for multimodal content. Previous efforts on multimodal
justification production have mostly focused on highlighting parts of the
input to increase model interpretability (Kou et al., 2020; Shang et al.,
2022). Natural language justifications that explain the reasoning process so
that it is understood by humans are mostly missing for multimodal fact-
checking. Moreover, studies investigating the faithfulness of justifications
and their usefulness are needed, along with techniques that explain decisions
made combining multiple input modalities. Furthermore, while justifications
are needed for each step of the fact checking pipeline (Figure 2), previous
research has mostly concentrated on justifying the final prediction step
(Volkova et al., 2019; Kou et al., 2020). With the recent progress in the
performance of neural models for image generation and editing, another so far
unexplored direction presents itself: editing input images or generating
entirely new images to explain model reasoning. This could be as simple as
highlighting relevant details exposing fauxtography, or as complex as the
generation of infographics as explanations. Such a system, especially if
guided by human fact-checkers (Nakov et al., 2021), would be a potent tool. As
noted in Lewandowsky et al. (2020), “well-designed graphs, videos, photos, and
other semantic aids can be helpful to convey corrections involving complex or
statistical information clearly and concisely”.

##  5 Conclusion

We survey research on multimodal automated fact-checking and introduce a
framework that combines and organizes tasks introduced in various communities
studying misinformation. We discuss common terms and definitions in context of
our framework. We further study popular benchmarks and modeling approaches,
and discuss promising directions for future research.

##  6 Limitations

While we cite many datasets and modeling approaches for multimodal fact-
checking, we describe most of them only briefly due to space constraints. Our
aim was to provide an overview of multimodal fact-checking and organise
previous works in a framework. Moreover, the presented survey focuses
primarily on four modalities. While there are other modalities we could have
included, we concentrated on those prevalent in real-world fact-checking that
have not been discussed as part of a fact-checking framework in previous
surveys.

## Ethics Statement

As we mention in Section 4, most datasets for multimodal fact-checking tasks
are available only in English. Thus, models are evaluated based on their
performance on English benchmarks only. This can lead to a distorted view
about advancements on multimodal automated fact-checking as it is limited to a
single language out of more than $7000$ world languages. While we call for
future work on a variety of languages, this survey provides an overview on the
state-of-the-art of mostly-English research efforts. Finally, we want to point
out that multimodal fact-checking works we cite in this survey might include
misleading statements or images given as examples.

## Acknowledgements

Michael Schlichtkrull, Zhijiang Guo, and Andreas Vlachos are supported by the
ERC grant AVeriTeC (GA 865958). Andreas Vlachos is further supported by the EU
H2020 grant MONITIO (GA 965576).

## References

  * Abdali (2022) Sara Abdali. 2022.  Multi-modal misinformation detection: Approaches, challenges and opportunities.  _CoRR_ , abs/2203.13883.
  * Abdelnabi et al. (2022) Sahar Abdelnabi, Rakibul Hasan, and Mario Fritz. 2022.  Open-domain, content-based, multi-modal fact-checking of out-of-context images via online resources.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 14940–14949.
  * Agarwal et al. (2019) Shruti Agarwal, Hany Farid, Yuming Gu, Mingming He, Koki Nagano, and Hao Li. 2019\.  Protecting world leaders against deep fakes.  In _IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 38–45. Computer Vision Foundation / IEEE.
  * Akhtar et al. (2022) Mubashara Akhtar, Oana Cocarascu, and Elena Simperl. 2022.  PubHealthTab: A public health table-based dataset for evidence-based fact checking.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 1–16, Seattle, United States. Association for Computational Linguistics.
  * Akhtar et al. (2023) Mubashara Akhtar, Oana Cocarascu, and Elena Simperl. 2023.  Reading and reasoning over chart images for evidence-based automated fact-checking.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 399–414, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Alam et al. (2022) Firoj Alam, Stefano Cresci, Tanmoy Chakraborty, Fabrizio Silvestri, Dimiter Dimitrov, Giovanni Da San Martino, Shaden Shaar, Hamed Firooz, and Preslav Nakov. 2022.  A survey on multimodal disinformation detection.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 6625–6643, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Aly et al. (2021a) Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021a.  The fact extraction and VERification over unstructured and structured information (FEVEROUS) shared task.  In _Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER)_ , pages 1–13, Dominican Republic. Association for Computational Linguistics.
  * Aly et al. (2021b) Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021b.  FEVEROUS: fact extraction and verification over unstructured and structured information.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_.
  * Amerini et al. (2019) Irene Amerini, Leonardo Galteri, Roberto Caldelli, and Alberto Del Bimbo. 2019.  Deepfake video detection through optical flow based CNN.  In _2019 IEEE/CVF International Conference on Computer Vision Workshops, ICCV Workshops 2019, Seoul, Korea (South), October 27-28, 2019_ , pages 1205–1207. IEEE.
  * Amri et al. (2021) Sabrine Amri, Dorsaf Sallami, and Esma Aïmeur. 2021.  EXMULF: an explainable multimodal content-based fake news detection system.  In _Foundations and Practice of Security - 14th International Symposium, FPS 2021, Paris, France, December 7-10, 2021, Revised Selected Papers_ , volume 13291 of _Lecture Notes in Computer Science_ , pages 177–187. Springer.
  * Aneja et al. (2021) Shivangi Aneja, Christoph Bregler, and Matthias Nießner. 2021.  Catching out-of-context misinformation with self-supervised learning.  _CoRR_ , abs/2101.06278.
  * Atanasova et al. (2020) Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, and Isabelle Augenstein. 2020\.  Generating fact checking explanations.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7352–7364, Online. Association for Computational Linguistics.
  * Baltrusaitis et al. (2019) Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2019.  Multimodal machine learning: A survey and taxonomy.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 41(2):423–443.
  * Barrón-Cedeño et al. (2020) Alberto Barrón-Cedeño, Tamer Elsayed, Preslav Nakov, Giovanni Da San Martino, Maram Hasanain, Reem Suwaileh, Fatima Haouari, Nikolay Babulkov, Bayan Hamdan, Alex Nikolov, Shaden Shaar, and Zien Sheikh Ali. 2020\.  Overview of checkthat! 2020: Automatic identification and verification of claims in social media.  In _Experimental IR Meets Multilinguality, Multimodality, and Interaction - 11th International Conference of the CLEF Association, CLEF 2020, Thessaloniki, Greece, September 22-25, 2020, Proceedings_ , volume 12260 of _Lecture Notes in Computer Science_ , pages 215–236. Springer.
  * Biamby et al. (2022) Giscard Biamby, Grace Luo, Trevor Darrell, and Anna Rohrbach. 2022.  Twitter-COMMs: Detecting climate, COVID, and military multimodal misinformation.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1530–1549, Seattle, United States. Association for Computational Linguistics.
  * Boididou et al. (2014) Christina Boididou, Symeon Papadopoulos, Yiannis Kompatsiaris, Steve Schifferes, and Nic Newman. 2014.  Challenges of computational verification in social multimedia.  In _Proceedings of the 23rd International Conference on World Wide Web_ , pages 743–748.
  * Bonettini et al. (2020) Nicolò Bonettini, Edoardo Daniele Cannas, Sara Mandelli, Luca Bondi, Paolo Bestagini, and Stefano Tubaro. 2020.  Video face manipulation detection through ensemble of cnns.  In _25th International Conference on Pattern Recognition, ICPR 2020, Virtual Event / Milan, Italy, January 10-15, 2021_ , pages 5012–5019. IEEE.
  * Cao et al. (2020) Juan Cao, Peng Qi, Qiang Sheng, Tianyun Yang, Junbo Guo, and Jintao Li. 2020.  Exploring the role of visual content in fake news detection.  _CoRR_ , abs/2003.05096.
  * Cheema et al. (2022) Gullal Singh Cheema, Sherzod Hakimov, Abdul Sittar, Eric Müller-Budack, Christian Otto, and Ralph Ewerth. 2022.  MM-claims: A dataset for multimodal claim detection in social media.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 962–979, Seattle, United States. Association for Computational Linguistics.
  * Chen et al. (2020) Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020.  Tabfact: A large-scale dataset for table-based fact verification.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Cooper (2007) Stephen Cooper. 2007.  A concise history of the fauxtography blogstorm in the 2006 lebanon war.  _American Communication Journal_ , 9.
  * Cozzolino et al. (2021) Davide Cozzolino, Andreas Rössler, Justus Thies, Matthias Nießner, and Luisa Verdoliva. 2021.  Id-reveal: Identity-aware deepfake video detection.  In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_ , pages 15088–15097. IEEE.
  * Dang et al. (2020) Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil K. Jain. 2020.  On the detection of digital face manipulation.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_ , pages 5780–5789. Computer Vision Foundation / IEEE.
  * Dolhansky et al. (2019) Brian Dolhansky, Russ Howes, Ben Pflaum, Nicole Baram, and Cristian Canton-Ferrer. 2019.  The deepfake detection challenge (DFDC) preview dataset.  _CoRR_ , abs/1910.08854.
  * Eyben et al. (2013) Florian Eyben, Felix Weninger, Florian Groß, and Björn W. Schuller. 2013\.  Recent developments in opensmile, the munich open-source multimedia feature extractor.  In _ACM Multimedia Conference, MM ’13, Barcelona, Spain, October 21-25, 2013_ , pages 835–838. ACM.
  * Eyben et al. (2009) Florian Eyben, Martin Wöllmer, and Björn W. Schuller. 2009.  Openear - introducing the munich open-source emotion and affect recognition toolkit.  In _Affective Computing and Intelligent Interaction, Third International Conference and Workshops, ACII 2009, Amsterdam, The Netherlands, September 10-12, 2009, Proceedings_ , pages 1–6. IEEE Computer Society.
  * Fung et al. (2021) Yi Fung, Christopher Thomas, Revanth Gangi Reddy, Sandeep Polisetty, Heng Ji, Shih-Fu Chang, Kathleen McKeown, Mohit Bansal, and Avi Sil. 2021.  InfoSurgeon: Cross-media fine-grained information consistency checking for fake news detection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1683–1698, Online. Association for Computational Linguistics.
  * Garimella and Eckles (2020) Kiran Garimella and Dean Eckles. 2020.  Images and misinformation in political groups: Evidence from whatsapp in india.  _CoRR_ , abs/2005.09784.
  * Graves (2018) D Graves. 2018.  Understanding the promise and limits of automated fact-checking.
  * Guera and Delp (2018) David Guera and Edward J. Delp. 2018.  Deepfake video detection using recurrent neural networks.  In _15th IEEE International Conference on Advanced Video and Signal Based Surveillance, AVSS 2018, Auckland, New Zealand, November 27-30, 2018_ , pages 1–6. IEEE.
  * Guo et al. (2018) Han Guo, Juan Cao, Yazi Zhang, Junbo Guo, and Jintao Li. 2018.  Rumor detection with hierarchical social attention network.  In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22-26, 2018_ , pages 943–951. ACM.
  * Guo et al. (2019) Wenzhong Guo, Jianwen Wang, and Shiping Wang. 2019.  Deep multimodal representation learning: A survey.  _IEEE Access_ , 7:63373–63394.
  * Guo et al. (2022) Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022.  A survey on automated fact-checking.  _Transactions of the Association for Computational Linguistics_ , 10:178–206.
  * Gupta et al. (2013) Aditi Gupta, Hemank Lamba, Ponnurangam Kumaraguru, and Anupam Joshi. 2013.  Faking sandy: characterizing and identifying fake images on twitter during hurricane sandy.  In _22nd International World Wide Web Conference, WWW ’13, Rio de Janeiro, Brazil, May 13-17, 2013, Companion Volume_ , pages 729–736. International World Wide Web Conferences Steering Committee / ACM.
  * Gupta and Srikumar (2021) Ashim Gupta and Vivek Srikumar. 2021.  X-fact: A new benchmark dataset for multilingual fact checking.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 675–682, Online. Association for Computational Linguistics.
  * Gupta et al. (2021) Shreya Gupta, Parantak Singh, Megha Sundriyal, Md. Shad Akhtar, and Tanmoy Chakraborty. 2021.  LESA: Linguistic encapsulation and semantic amalgamation based generalised claim detection from online content.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 3178–3188, Online. Association for Computational Linguistics.
  * Hameleers et al. (2020) Michael Hameleers, Thomas E Powell, Toni GLA Van Der Meer, and Lieke Bos. 2020.  A picture paints a thousand lies? the effects and mechanisms of multimodal disinformation and rebuttals disseminated via social media.  _Political Communication_ , 37(2):281–301.
  * Hammouchi and Ghogho (2022) Hicham Hammouchi and Mounir Ghogho. 2022.  Evidence-aware multilingual fake news detection.  _IEEE Access_ , 10:116808–116818.
  * Hassan et al. (2015a) Naeemul Hassan, Bill Adair, James T Hamilton, Chengkai Li, Mark Tremayne, Jun Yang, and Cong Yu. 2015a.  The quest to automate fact-checking.  In _Proceedings of the 2015 computation+ journalism symposium_. Citeseer.
  * Hassan et al. (2015b) Naeemul Hassan, Chengkai Li, and Mark Tremayne. 2015b.  Detecting check-worthy factual claims in presidential debates.  In _Proceedings of the 24th ACM International Conference on Information and Knowledge Management, CIKM 2015, Melbourne, VIC, Australia, October 19 - 23, 2015_ , pages 1835–1838. ACM.
  * He et al. (2017) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017.  Mask r-cnn.  In _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 2980–2988.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778.
  * Heller et al. (2018) Silvan Heller, Luca Rossetto, and Heiko Schuldt. 2018.  The ps-battles dataset - an image collection for image manipulation detection.  _CoRR_ , abs/1804.04866.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural Comput._ , 9(8):1735–1780.
  * Hoque et al. (2022) Enamul Hoque, Parsa Kavehzadeh, and Ahmed Masry. 2022.  Chart question answering: State of the art and future directions.  _Comput. Graph. Forum_ , 41(3):555–572.
  * Hou et al. (2019) Rui Hou, Verónica Pérez-Rosas, Stacy L. Loeb, and Rada Mihalcea. 2019\.  Towards automatic detection of misinformation in online medical videos.  In _International Conference on Multimodal Interaction, ICMI 2019, Suzhou, China, October 14-18, 2019_ , pages 235–243. ACM.
  * Huh et al. (2018) Minyoung Huh, Andrew Liu, Andrew Owens, and Alexei A. Efros. 2018.  Fighting fake news: Image splice detection via learned self-consistency.  In _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part XI_ , volume 11215 of _Lecture Notes in Computer Science_ , pages 106–124. Springer.
  * Ireton and Posetti (2018) Cherilyn Ireton and Julie Posetti. 2018.  _Journalism, fake news & disinformation: handbook for journalism education and training_.  Unesco Publishing.
  * Ismael Al-Sanjary et al. (2016) Omar Ismael Al-Sanjary, Ahmed Abdullah Ahmed, and Ghazali Sulong. 2016.  Development of a video tampering dataset for forensic investigation.  _Forensic Science International_ , 266:565–572.
  * Jaiswal et al. (2017) Ayush Jaiswal, Ekraam Sabir, Wael Abd-Almageed, and Premkumar Natarajan. 2017\.  Multimedia semantic integrity assessment using joint embedding of images and text.  In _Proceedings of the 2017 ACM on Multimedia Conference, MM 2017, Mountain View, CA, USA, October 23-27, 2017_ , pages 1465–1471. ACM.
  * Jiang et al. (2020) Liming Jiang, Ren Li, Wayne Wu, Chen Qian, and Chen Change Loy. 2020.  Deeperforensics-1.0: A large-scale dataset for real-world face forgery detection.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_ , pages 2886–2895. Computer Vision Foundation / IEEE.
  * Jin et al. (2017) Zhiwei Jin, Juan Cao, Han Guo, Yongdong Zhang, and Jiebo Luo. 2017.  Multimodal fusion with recurrent neural networks for rumor detection on microblogs.  In _Proceedings of the 2017 ACM on Multimedia Conference, MM 2017, Mountain View, CA, USA, October 23-27, 2017_ , pages 795–816. ACM.
  * Kalb and Saivetz (2007) Marvin Kalb and Carol Saivetz. 2007.  The israeli—hezbollah war of 2006: The media as a weapon in asymmetrical conflict.  _Harvard International Journal of Press/Politics_ , 12(3):43–66.
  * Kamboj et al. (2021) Manvi Kamboj, Christian Hessler, Priyanka Asnani, Kais Riani, and Mohamed Abouelenien. 2021.  Multimodal political deception detection.  _IEEE Multim._ , 28(1):94–102.
  * Karras et al. (2019) Tero Karras, Samuli Laine, and Timo Aila. 2019.  A style-based generator architecture for generative adversarial networks.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 4401–4410. Computer Vision Foundation / IEEE.
  * Khalid et al. (2021) Hasam Khalid, Shahroz Tariq, Minha Kim, and Simon S. Woo. 2021.  Fakeavceleb: A novel audio-video multimodal deepfake dataset.  In _Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets and Benchmarks 2021, December 2021, virtual_.
  * Khattar et al. (2019) Dhruv Khattar, Jaipal Singh Goud, Manish Gupta, and Vasudeva Varma. 2019.  MVAE: multimodal variational autoencoder for fake news detection.  In _The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019_ , pages 2915–2921. ACM.
  * Kinnunen et al. (2017) Tomi Kinnunen, Md. Sahidullah, Héctor Delgado, Massimiliano Todisco, Nicholas W. D. Evans, Junichi Yamagishi, and Kong-Aik Lee. 2017.  The asvspoof 2017 challenge: Assessing the limits of replay spoofing attack detection.  In _Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017_ , pages 2–6. ISCA.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Kochkina et al. (2017) Elena Kochkina, Maria Liakata, and Isabelle Augenstein. 2017.  Turing at SemEval-2017 task 8: Sequential approach to rumour stance classification with branch-LSTM.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 475–480, Vancouver, Canada. Association for Computational Linguistics.
  * Kopev et al. (2019) Daniel Kopev, Ahmed Ali, Ivan Koychev, and Preslav Nakov. 2019.  Detecting deception in political debates using acoustic and textual features.  In _IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2019, Singapore, December 14-18, 2019_ , pages 652–659. IEEE.
  * Kotonya and Toni (2020a) Neema Kotonya and Francesca Toni. 2020a.  Explainable automated fact-checking: A survey.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5430–5443, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Kotonya and Toni (2020b) Neema Kotonya and Francesca Toni. 2020b.  Explainable automated fact-checking for public health claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7740–7754, Online. Association for Computational Linguistics.
  * Kou et al. (2020) Ziyi Kou, Daniel Yue Zhang, Lanyu Shang, and Dong Wang. 2020.  Exfaux: A weakly supervised approach to explainable fauxtography detection.  In _2020 IEEE International Conference on Big Data (IEEE BigData 2020), Atlanta, GA, USA, December 10-13, 2020_ , pages 631–636. IEEE.
  * Kwon et al. (2021) Patrick Kwon, Jaeseong You, Gyuhyeon Nam, Sungwoo Park, and Gyeongsu Chae. 2021\.  Kodf: A large-scale korean deepfake detection dataset.  In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_ , pages 10724–10733. IEEE.
  * La et al. (2022) Tuan-Vinh La, Quang-Tien Tran, Thanh-Phuc Tran, Anh-Duy Tran, Duc-Tien Dang-Nguyen, and Minh-Son Dao. 2022.  Multimodal cheapfakes detection by utilizing image captioning for global context.  In _ICDAR@ICMR 2022: Proceedings of the 3rd ACM Workshop on Intelligent Cross-Data Analysis and Retrieval, Newark, NJ, USA, June 27 - 30, 2022_ , pages 9–16. ACM.
  * Lauer (2009) Claire Lauer. 2009.  Contending with terms: “multimodal” and “multimedia” in the academic and public spheres.  _Computers and Composition_ , 26(4):225–239.
  * Lavrentyeva et al. (2019) Galina Lavrentyeva, Sergey Novoselov, Marina Volkova, Yuri Matveev, and Maria De Marsico. 2019.  Phonespoof: A new dataset for spoofing attack detection in telephone channel.  In _IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2019, Brighton, United Kingdom, May 12-17, 2019_ , pages 2572–2576. IEEE.
  * Lewandowsky et al. (2020) Stephan Lewandowsky, John Cook, and Doug Lombardi. 2020.  Debunking Handbook 2020.
  * Li et al. (2020a) Jiawen Li, Yudianto Sujana, and Hung-Yu Kao. 2020a.  Exploiting microblog conversation structures to detect rumors.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5420–5429, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Li and Xie (2020) Yiyi Li and Ying Xie. 2020.  Is a picture worth a thousand words? an empirical study of image content and social media engagement.  _Journal of Marketing Research_ , 57(1):1–19.
  * Li et al. (2020b) Yuezun Li, Xin Yang, Pu Sun, Honggang Qi, and Siwei Lyu. 2020b.  Celeb-df: A large-scale challenging dataset for deepfake forensics.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_ , pages 3204–3213. Computer Vision Foundation / IEEE.
  * Liang et al. (2022) Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2022.  Foundations and recent trends in multimodal machine learning: Principles, challenges, and open questions.  _CoRR_ , abs/2209.03430.
  * Liu et al. (2021) Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021.  Visually grounded reasoning across languages and cultures.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 10467–10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lo et al. (2022) Leo Yu-Ho Lo, Ayush Gupta, Kento Shigyo, Aoyu Wu, Enrico Bertini, and Huamin Qu. 2022.  Misinformed by visualization: What do we learn from misinformative visualizations? _Comput. Graph. Forum_ , 41(3):515–525.
  * Lourenço and Paes (2022) Vítor Lourenço and Aline Paes. 2022.  A modality-level explainable framework for misinformation checking in social networks.  _CoRR_ , abs/2212.04272.
  * Lu and Li (2020) Yi-Ju Lu and Cheng-Te Li. 2020.  GCAN: Graph-aware co-attention networks for explainable fake news detection on social media.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 505–514, Online. Association for Computational Linguistics.
  * Luo et al. (2021) Grace Luo, Trevor Darrell, and Anna Rohrbach. 2021.  NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6801–6817, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ma and Gao (2020) Jing Ma and Wei Gao. 2020.  Debunking rumors on twitter with tree transformer.  In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_ , pages 5455–5466. International Committee on Computational Linguistics.
  * Ma et al. (2018) Jing Ma, Wei Gao, and Kam-Fai Wong. 2018.  Rumor detection on Twitter with tree-structured recursive neural networks.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1980–1989, Melbourne, Australia. Association for Computational Linguistics.
  * Maras and Alexandrou (2019) Marie-Helen Maras and Alex Alexandrou. 2019.  Determining authenticity of video evidence in the age of artificial intelligence and in the wake of deepfake videos.  _The International Journal of Evidence & Proof_, 23(3):255–262.
  * Maros et al. (2021) Alexandre Maros, Jussara M. Almeida, and Marisa Vasconcelos. 2021.  A study of misinformation in audio messages shared in whatsapp groups.  In _Disinformation in Open Online Media - Third Multidisciplinary International Symposium, MISDOOM 2021, Virtual Event, September 21-22, 2021, Proceedings_ , volume 12887 of _Lecture Notes in Computer Science_ , pages 85–100. Springer.
  * Meel and Vishwakarma (2021) Priyanka Meel and Dinesh Kumar Vishwakarma. 2021.  Han, image captioning, and forensics ensemble multimodal fake news detection.  _Information Sciences_ , 567:23–41.
  * Mikolov et al. (2013) Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  In _1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings_.
  * Mishra et al. (2022) Shreyash Mishra, Suryavardan S, Amrit Bhaskar, Parul Chopra, Aishwarya N. Reganti, Parth Patwa, Amitava Das, Tanmoy Chakraborty, Amit P. Sheth, and Asif Ekbal. 2022.  FACTIFY: A multi-modal fact verification dataset.  In _Proceedings of the Workshop on Multi-Modal Fake News and Hate-Speech Detection (DE-FACTIFY 2022) co-located with the Thirty-Sixth AAAI Conference on Artificial Intelligence ( AAAI 2022), Virtual Event, Vancouver, Canada, February 27, 2022_ , volume 3199 of _CEUR Workshop Proceedings_. CEUR-WS.org.
  * Monti et al. (2019) Federico Monti, Fabrizio Frasca, Davide Eynard, Damon Mannion, and Michael M. Bronstein. 2019.  Fake news detection on social media using geometric deep learning.  _CoRR_ , abs/1902.06673.
  * Müller-Budack et al. (2020) Eric Müller-Budack, Jonas Theiner, Sebastian Diering, Maximilian Idahl, and Ralph Ewerth. 2020.  Multimodal analytics for real-world news using measures of cross-modal entity consistency.  In _Proceedings of the 2020 on International Conference on Multimedia Retrieval, ICMR 2020, Dublin, Ireland, June 8-11, 2020_ , pages 16–25. ACM.
  * Nakamura et al. (2020) Kai Nakamura, Sharon Levy, and William Yang Wang. 2020.  Fakeddit: A new multimodal benchmark dataset for fine-grained fake news detection.  In _Proceedings of The 12th Language Resources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020_ , pages 6149–6157. European Language Resources Association.
  * Nakov et al. (2018) Preslav Nakov, Alberto Barrón-Cedeño, Tamer Elsayed, Reem Suwaileh, Lluís Màrquez, Wajdi Zaghouani, Pepa Atanasova, Spas Kyuchukov, and Giovanni Da San Martino. 2018.  Overview of the CLEF-2018 checkthat! lab on automatic identification and verification of political claims.  In _Experimental IR Meets Multilinguality, Multimodality, and Interaction - 9th International Conference of the CLEF Association, CLEF 2018, Avignon, France, September 10-14, 2018, Proceedings_ , volume 11018 of _Lecture Notes in Computer Science_ , pages 372–387. Springer.
  * Nakov et al. (2021) Preslav Nakov, David P. A. Corney, Maram Hasanain, Firoj Alam, Tamer Elsayed, Alberto Barrón-Cedeño, Paolo Papotti, Shaden Shaar, and Giovanni Da San Martino. 2021.  Automated fact-checking for assisting human fact-checkers.  In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI 2021, Virtual Event / Montreal, Canada, 19-27 August 2021_ , pages 4551–4558. ijcai.org.
  * Newman and Zhang (2020) Eryn Newman and Lynn Zhang. 2020.  _Truthiness: How Non-Probative Photos Shape Belief_.
  * Newman et al. (2012) Eryn J Newman, Maryanne Garry, Daniel M Bernstein, Justin Kantner, and D Stephen Lindsay. 2012.  Nonprobative photographs (or words) inflate truthiness.  _Psychonomic Bulletin & Review_, 19(5):969–974.
  * Newman et al. (2003) Matthew L Newman, James W Pennebaker, Diane S Berry, and Jane M Richards. 2003.  Lying words: Predicting deception from linguistic styles.  _Personality and social psychology bulletin_ , 29(5):665–675.
  * Nielsen and McConville (2022) Dan Saattrup Nielsen and Ryan McConville. 2022.  Mumin: A large-scale multilingual multimodal fact-checked misinformation social network dataset.  In _SIGIR ’22: The 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, Madrid, Spain, July 11 \- 15, 2022_ , pages 3141–3153. ACM.
  * Nirkin et al. (2019) Yuval Nirkin, Yosi Keller, and Tal Hassner. 2019.  FSGAN: subject agnostic face swapping and reenactment.  In _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019_ , pages 7183–7192. IEEE.
  * Paris and Donovan (2019) Britt Paris and Joan Donovan. 2019.  Deepfakes and cheap fakes.  _United States of America: Data & Society_, 1.
  * Patwa et al. (2022) Parth Patwa, Shreyash Mishra, Suryavardan S, Amrit Bhaskar, Parul Chopra, Aishwarya Reganti, Amitava Das, Tanmoy Chakraborty, Amit Sheth, Asif Ekbal, and Chaitanya Ahuja. 2022.  Benchmarking multi-modal entailment for fact verification.
  * Pérez-Rosas et al. (2015) Verónica Pérez-Rosas, Mohamed Abouelenien, Rada Mihalcea, and Mihai Burzo. 2015.  Deception detection using real-life trial data.  In _Proceedings of the 2015 ACM on International Conference on Multimodal Interaction, Seattle, WA, USA, November 09 - 13, 2015_ , pages 59–66. ACM.
  * Prabhakar et al. (2021) Tarunima Prabhakar, Anushree Gupta, Kruttika Nadig, and Denny George. 2021.  Check mate: Prioritizing user generated multi-media content for fact-checking.  In _Proceedings of the Fifteenth International AAAI Conference on Web and Social Media, ICWSM 2021, held virtually, June 7-10, 2021_ , pages 1025–1033. AAAI Press.
  * Purwanto et al. (2021) Christian Nathaniel Purwanto, Joan Santoso, Po-Ruey Lei, Hui-Kuo Yang, and Wen-Chih Peng. 2021.  Fakeclip: Multimodal fake caption detection with mixed languages for explainable visualization.  In _2021 International Conference on Technologies and Applications of Artificial Intelligence, TAAI 2021, Taichung, Taiwan, November 18-20, 2021_ , pages 1–6. IEEE.
  * Qi et al. (2019) Peng Qi, Juan Cao, Tianyun Yang, Junbo Guo, and Jintao Li. 2019.  Exploiting multi-domain visual information for fake news detection.  In _2019 IEEE International Conference on Data Mining, ICDM 2019, Beijing, China, November 8-11, 2019_ , pages 518–527. IEEE.
  * Qian et al. (2021) Shengsheng Qian, Jinguang Wang, Jun Hu, Quan Fang, and Changsheng Xu. 2021.  Hierarchical multi-modal contextual attention network for fake news detection.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 153–162.
  * Qu et al. (2022a) Jingnong Qu, Liunian Harold Li, Jieyu Zhao, Sunipa Dev, and Kai-Wei Chang. 2022a.  Disinfomeme: A multimodal dataset for detecting meme intentionally spreading out disinformation.  _CoRR_ , abs/2205.12617.
  * Qu et al. (2022b) Jingnong Qu, Liunian Harold Li, Jieyu Zhao, Sunipa Dev, and Kai-Wei Chang. 2022b.  Disinfomeme: A multimodal dataset for detecting meme intentionally spreading out disinformation.  _CoRR_ , abs/2205.12617.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR.
  * Reimao and Tzerpos (2019) Ricardo Reimao and Vassilios Tzerpos. 2019.  For: A dataset for synthetic speech detection.  In _2019 International Conference on Speech Technology and Human-Computer Dialogue, SpeD 2019, Timisoara, Romania, October 10-12, 2019_ , pages 1–10. IEEE.
  * Resende et al. (2019) Gustavo Resende, Philipe F. Melo, Hugo Sousa, Johnnatan Messias, Marisa Vasconcelos, Jussara M. Almeida, and Fabrício Benevenuto. 2019.  (mis)information dissemination in whatsapp: Gathering, analyzing and countermeasures.  In _The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019_ , pages 818–828. ACM.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. 2022.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 10684–10695.
  * Rössler et al. (2018) Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner. 2018.  Faceforensics: A large-scale video dataset for forgery detection in human faces.  _CoRR_ , abs/1803.09179.
  * Roy and Ekbal (2021) Arjun Roy and Asif Ekbal. 2021.  Mulcob-mulfav: Multimodal content based multilingual fact verification.  In _International Joint Conference on Neural Networks, IJCNN 2021, Shenzhen, China, July 18-22, 2021_ , pages 1–8. IEEE.
  * Ruder et al. (2022) Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2022.  Square one bias in NLP: Towards a multi-dimensional exploration of the research manifold.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2340–2354, Dublin, Ireland. Association for Computational Linguistics.
  * Sabir et al. (2018) Ekraam Sabir, Wael AbdAlmageed, Yue Wu, and Prem Natarajan. 2018.  Deep multimodal image-repurposing detection.  In _2018 ACM Multimedia Conference on Multimedia Conference, MM 2018, Seoul, Republic of Korea, October 22-26, 2018_ , pages 1337–1345. ACM.
  * Sabir et al. (2019) Ekraam Sabir, Jiaxin Cheng, Ayush Jaiswal, Wael AbdAlmageed, Iacopo Masi, and Prem Natarajan. 2019.  Recurrent convolutional strategies for face manipulation detection in videos.  In _IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 80–87. Computer Vision Foundation / IEEE.
  * Shahi and Nandini (2020) Gautam Kishore Shahi and Durgesh Nandini. 2020.  Fakecovid- A multilingual cross-domain fact check news dataset for COVID-19.  In _Workshop Proceedings of the 14th International AAAI Conference on Web and Social Media, ICWSM 2020 Workshops, Atlanta, Georgia, USA [virtual], June 8, 2020_.
  * Shang et al. (2021) Lanyu Shang, Ziyi Kou, Yang Zhang, and Dong Wang. 2021.  A multimodal misinformation detector for COVID-19 short videos on tiktok.  In _2021 IEEE International Conference on Big Data (Big Data), Orlando, FL, USA, December 15-18, 2021_ , pages 899–908. IEEE.
  * Shang et al. (2022) Lanyu Shang, Ziyi Kou, Yang Zhang, and Dong Wang. 2022.  A duo-generative approach to explainable multimodal COVID-19 misinformation detection.  In _WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022_ , pages 3623–3631. ACM.
  * Silverman (2013) Craig Silverman. 2013.  Verification handbook.
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very deep convolutional networks for large-scale image recognition.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Singhal et al. (2022) Shivangi Singhal, Rajiv Ratn Shah, and Ponnurangam Kumaraguru. 2022.  Factdrill: A data repository of fact-checked social media content to study fake news incidents in india.  In _Proceedings of the Sixteenth International AAAI Conference on Web and Social Media, ICWSM 2022, Atlanta, Georgia, USA, June 6-9, 2022_ , pages 1322–1331. AAAI Press.
  * Szegedy et al. (2015) Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2015.  Going deeper with convolutions.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1–9.
  * Tan et al. (2020) Reuben Tan, Bryan A. Plummer, and Kate Saenko. 2020.  Detecting cross-modal inconsistency to defend against neural fake news.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , pages 2081–2106. Association for Computational Linguistics.
  * Tanwar and Sharma (2020) Vidhu Tanwar and Kapil Sharma. 2020.  Multi-model fake news detection based on concatenation of visual latent features.  In _2020 International Conference on Communication and Signal Processing (ICCSP)_ , pages 1344–1348.
  * Thorne and Vlachos (2018) James Thorne and Andreas Vlachos. 2018.  Automated fact checking: Task formulations, methods and future directions.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3346–3359, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Volkova et al. (2019) Svitlana Volkova, Ellyn Ayton, Dustin Lockhart Arendt, Zhuanyi Huang, and Brian Hutchinson. 2019.  Explaining multimodal deceptive news prediction models.  In _Proceedings of the Thirteenth International Conference on Web and Social Media, ICWSM 2019, Munich, Germany, June 11-14, 2019_ , pages 659–662. AAAI Press.
  * Wang et al. (2022) Junke Wang, Zuxuan Wu, Wenhao Ouyang, Xintong Han, Jingjing Chen, Yu-Gang Jiang, and Ser-Nam Li. 2022.  M2TR: multi-modal multi-scale transformers for deepfake detection.  In _ICMR ’22: International Conference on Multimedia Retrieval, Newark, NJ, USA, June 27 - 30, 2022_ , pages 615–623. ACM.
  * Wang et al. (2020) Run Wang, Felix Juefei-Xu, Yihao Huang, Qing Guo, Xiaofei Xie, Lei Ma, and Yang Liu. 2020.  Deepsonar: Towards effective and robust detection of ai-synthesized fake voices.  In _MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020_ , pages 1207–1216. ACM.
  * Wang et al. (2018) Yaqing Wang, Fenglong Ma, Zhiwei Jin, Ye Yuan, Guangxu Xun, Kishlay Jha, Lu Su, and Jing Gao. 2018.  EANN: event adversarial neural networks for multi-modal fake news detection.  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018_, pages 849–857. ACM.
  * Wei et al. (2021) Lingwei Wei, Dou Hu, Wei Zhou, Zhaojuan Yue, and Songlin Hu. 2021.  Towards propagation uncertainty: Edge-enhanced bayesian graph convolutional networks for rumor detection.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021_ , pages 3845–3854. Association for Computational Linguistics.
  * Wodajo and Atnafu (2021) Deressa Wodajo and Solomon Atnafu. 2021.  Deepfake video detection using convolutional vision transformer.  _CoRR_ , abs/2102.11126.
  * Wu et al. (2021) Yang Wu, Pengwei Zhan, Yunjian Zhang, Liming Wang, and Zhen Xu. 2021.  Multimodal fusion with co-attention networks for fake news detection.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 2560–2569, Online. Association for Computational Linguistics.
  * Wu et al. (2019) Yue Wu, Wael AbdAlmageed, and Premkumar Natarajan. 2019.  Mantra-net: Manipulation tracing network for detection and localization of image forgeries with anomalous features.  In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_ , pages 9543–9552. Computer Vision Foundation / IEEE.
  * Wu et al. (2015) Zhizheng Wu, Tomi Kinnunen, Nicholas W. D. Evans, Junichi Yamagishi, Cemal Hanilçi, Md. Sahidullah, and Aleksandr Sizov. 2015.  Asvspoof 2015: the first automatic speaker verification spoofing and countermeasures challenge.  In _INTERSPEECH 2015, 16th Annual Conference of the International Speech Communication Association, Dresden, Germany, September 6-10, 2015_ , pages 2037–2041. ISCA.
  * Yang et al. (2020) Xiaoyu Yang, Yuefei Lyu, Tian Tian, Yifei Liu, Yudong Liu, and Xi Zhang. 2020.  Rumor detection on social media with graph structured adversarial learning.  In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI 2020_ , pages 1417–1423. ijcai.org.
  * Yao et al. (2022) Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin-Hee Cho, and Lifu Huang. 2022\.  End-to-end multimodal fact-checking and explanation generation: A challenging dataset and models.  _CoRR_ , abs/2205.12487.
  * Yi et al. (2021) Jiangyan Yi, Ye Bai, Jianhua Tao, Haoxin Ma, Zhengkun Tian, Chenglong Wang, Tao Wang, and Ruibo Fu. 2021.  Half-truth: A partially fake audio detection dataset.  In _Interspeech 2021, 22nd Annual Conference of the International Speech Communication Association, Brno, Czechia, 30 August - 3 September 2021_ , pages 1654–1658. ISCA.
  * Yuan et al. (2019) Chunyuan Yuan, Qianwen Ma, Wei Zhou, Jizhong Han, and Songlin Hu. 2019.  Jointly embedding the local and global relations of heterogeneous graph for rumor detection.  In _2019 IEEE International Conference on Data Mining, ICDM 2019, Beijing, China, November 8-11, 2019_ , pages 796–805. IEEE.
  * Zakharov et al. (2019) Egor Zakharov, Aliaksandra Shysheya, Egor Burkov, and Victor S. Lempitsky. 2019\.  Few-shot adversarial learning of realistic neural talking head models.  In _2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019_ , pages 9458–9467. IEEE.
  * Zampoglou et al. (2015) Markos Zampoglou, Symeon Papadopoulos, and Yiannis Kompatsiaris. 2015.  Detecting image splicing in the wild (WEB).  In _2015 IEEE International Conference on Multimedia & Expo Workshops, ICME Workshops 2015, Turin, Italy, June 29 - July 3, 2015_, pages 1–6. IEEE Computer Society.
  * Zeng et al. (2021) Xia Zeng, Amani S. Abumansour, and Arkaitz Zubiaga. 2021.  Automated fact-checking: A survey.  _Lang. Linguistics Compass_ , 15(10).
  * Zhang et al. (2018) Daniel Yue Zhang, Lanyu Shang, Biao Geng, Shuyue Lai, Ke Li, Hongmin Zhu, Md. Tanvir Al Amin, and Dong Wang. 2018.  Fauxbuster: A content-free fauxtography detector using social media comments.  In _IEEE International Conference on Big Data (IEEE BigData 2018), Seattle, WA, USA, December 10-13, 2018_ , pages 891–900. IEEE.
  * Zhang et al. (2023) Yizhou Zhang, Loc Trinh, Defu Cao, Zijun Cui, and Yan Liu. 2023.  Detecting out-of-context multimodal misinformation with interpretable neural-symbolic model.  _CoRR_ , abs/2304.07633.
  * Zheng et al. (2022) Jiaqi Zheng, Xi Zhang, Sanchuan Guo, Quan Wang, Wenyu Zang, and Yongdong Zhang. 2022\.  MFAN: multi-modal feature-enhanced attention networks for rumor detection.  In _Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI 2022, Vienna, Austria, 23-29 July 2022_ , pages 2413–2419. ijcai.org.
  * Zheng et al. (2021) Yinglin Zheng, Jianmin Bao, Dong Chen, Ming Zeng, and Fang Wen. 2021.  Exploring temporal coherence for more general video face forgery detection.  In _2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021_ , pages 15024–15034. IEEE.
  * Zhou et al. (2018) Peng Zhou, Xintong Han, Vlad I. Morariu, and Larry S. Davis. 2018.  Learning rich features for image manipulation detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zlatkova et al. (2019) Dimitrina Zlatkova, Preslav Nakov, and Ivan Koychev. 2019.  Fact-checking meets fauxtography: Verifying claims about images.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2099–2108, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Method

We applied the following methodological approach to find and select relevant
research papers for the survey.

First, after defining the research scope, we collected pivotal, highly-cited
work (e.g. Nakamura et al. (2020)) and related surveys (e.g. Alam et al.
(2022)), resulting in $25$ papers, as well as papers citing or cited by these
works. We collected further works using the scholarly search engines Google
Scholar888https://scholar.google.com/, Semantic
Scholar999https://www.semanticscholar.org/, DBLP101010https://dblp.org/ and
ACL Anthology111111https://aclanthology.org/, and keyword-based search with
Cartesian products of following keyword sets: {“fact checking”, “fact
verification”, “misinformation”, “disinformation”, “fake news”},
{“multimodal”, “text”, “image”, “audio”, “video”}, and {“machine learning”,
“automated”}. The databases were queried primarily during the time frame July
$26,2022$ and August $10,2022$. This step resulted in a collection of $123$
papers.

We manually screened and filtered the papers based on abstracts and
introduction sections, before creating an overview of papers across the
following dimensions: $(1)$ modality; $(2)$ fact-checking task; $(3)$
contribution type (i.e. dataset, modeling approach, demo); $(4)$ paper type
(i.e. survey, position paper, solution paper (e.g. introducing a new benchmark
or modeling approach), or evaluation paper (e.g. investigating previously
proposed approaches)). Papers were mostly excluded because they focused on
other tasks than fact-checking (e.g. hate speech detection) or used modalities
out of our scope (e.g. tables). Moreover, during the screening process we
found and added further related works, and concluded the screening with $84$
unique papers.

The taxonomy of tasks (Section 2) was created in an iterative manner starting
with the task labels we assigned to works during screening. As a starting
point we also used taxonomies of text-only fact-checking surveys (Guo et al.,
2022; Thorne and Vlachos, 2018) and adapted them for multimodal fact-checking
works.