# Evolving from Single-modal to Multi-modal Facial Deepfake Detection: A
Survey

Ping Liu, , Qiqi Tao, and Joey Tianyi Zhou P. Liu is with the Department of
Computer Science and Engineering, University of Nevada, Reno, NV, 89512.
E-mail<EMAIL_ADDRESS>Q. Tao and J. Zhou are with Centre for Frontier
AI Research (CFAR), and Institute of High Performance Computing (IHPC),
A*STAR, Singapore.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>This work is done
during Q. Tao’s internship at Centre for Frontier AI Research (CFAR), A*STAR,
Singapore.

###### Abstract

This survey addresses the critical challenge of deepfake detection amidst the
rapid advancements in artificial intelligence. As AI-generated media,
including video, audio and text, become more realistic, the risk of misuse to
spread misinformation and commit identity fraud increases. Focused on face-
centric deepfakes, this work traces the evolution from traditional single-
modality methods to sophisticated multi-modal approaches that handle audio-
visual and text-visual scenarios. We provide comprehensive taxonomies of
detection techniques, discuss the evolution of generative methods from auto-
encoders and GANs to diffusion models, and categorize these technologies by
their unique attributes. To our knowledge, this is the first survey of its
kind. We also explore the challenges of adapting detection methods to new
generative models and enhancing the reliability and robustness of deepfake
detectors, proposing directions for future research. This survey offers a
detailed roadmap for researchers, supporting the development of technologies
to counter the deceptive use of AI in media creation, particularly facial
forgery. A curated list of all related papers can be found at
https://github.com/qiqitao77/Awesome-Comprehensive-Deepfake-Detection.

###### Index Terms:

Generative Adversarial Network, Diffusion Model, Deepfake, Facial Forgery,
Multi-modality

##  1 Introduction

Deepfake, encompassing video, audio, and text, has been transformed through
advanced artificial intelligence technologies such as Variational Auto-
Encoders (VAEs) [1], Generative Adversarial Networks (GANs) [2], and Diffusion
Models (DMs) [3] to achieve unprecedented realism. Given that face-related
deepfakes may have more severe societal and ethical ramifications than
synthetic media of non-facial content, this work specifically focuses on
facial deepfakes. It covers various facial manipulations such as face
swapping, attribute editing, and face reenactment [4]. Those manipulations not
only enhance personalized digital interactions on social platforms and mobile
apps but also drive innovations in the entertainment industry, such as
digitally resurrecting historical
figures111https://edition.cnn.com/2024/02/12/asia/suharto-deepfake-ai-scam-
indonesia-election-hnk-intl/index.html or creating virtual actors
222https://edition.cnn.com/2021/08/06/tech/tom-cruise-deepfake-tiktok-
company/index.html.

Recent years have witnessed significant advances in deepfake generation,
particularly with the emergence of diffusion models, which have revolutionized
the creation of synthetic content. This evolution has greatly obscured the
distinction between real and fake media, complicating the task for human
observers and deepfake detectors. Furthermore, the scope of deepfake
technology has expanded beyond single-modality (video or audio alone) to
intricate multi-modal manipulations[5][6][7]. These manipulations blend fake
audio, video, and text, introducing additional complexity to the detection
process. Correspondingly, the deepfake detection mechanisms must evolve to
address not only more sophisticated generation techniques but also the
challenges posed by the integration of multiple media types.

Progress in deepfake detection over the past two years has addressed the
challenges posed by these advanced generative methods and diverse modalities.
Researchers have responded to the enhanced realism of synthetic content by
innovating detection strategies at different stages of the detection pipeline,
including the input level, feature level, and learning level. Some advances
leverage new deep learning models, such as transformers [8] and vision-
language models [9]. To address multi-modal deepfake detection, various fusion
techniques, such as [10, 11, 12, 13], have been developed to utilize knowledge
from different modalities effectively. These enhancements refine detection
techniques to keep pace with the rapidly evolving quality and diversity of
deepfakes, expanding efforts from traditional single-modal to comprehensive
multi-modal detection strategies, and from passive to proactive
solutions[14][15].

TABLE I: Comparison between this work and previous reviews. GAN: Generative Adversarial Network; AE: Auto-Encoder; DM: Diffusion Model. Work | Focused Tasks | Generative Model | Face-centric | Multi-modal | Released Date
---|---|---|---|---|---
Juefei-Xu et al.[16] | Deepfake Generation and Detection | GAN & VAE | ✓ | X | Mar 2022
Pei et al.[4] | Deepfake Generation and Detection | GAN & VAE & DM | ✓ | X | Mar 2024
Lin et al.[17] | AIGC Detection | LLM & DM | X | ✓ | Feb 2024
Wang et al.[18] | Deepfake Detection | GAN | ✓ | X | Nov 2023
Wang et al.[19] | Reliability of Deepfake Detection | GAN & VAE | ✓ | X | Nov 2022
Ours | Deepfake Detection | GAN & VAE & DM | ✓ | ✓ | Jun 2024

The rapid advancements and variations across different works in the field have
motivated us to provide the most up-to-date survey, presented in this
submission. As demonstrated in Table I, our survey is the first comprehensive
review to specifically address facial forgery detection, ranging from single-
modal to multi-modal methods. This unique focus distinguishes our survey from
previous works in the field. For instance, while [17] explores detection
across general fake samples, our survey specifically targets face-related
generated media due to its significant potential for societal harm. In
contrast to surveys such as [16][4], which allocate much of their discussion
to deepfake generation, our work concentrates on detection techniques.

Unlike previous surveys that focused on either facial forgery [16, 4, 18, 19]
or multi-modal deepfake [17, 20] separately, our work merges these two crucial
areas, offering a holistic view of the current state and challenges of the
field.

Our survey is exceptionally current, including the latest research up to June
2024, whereas most prior surveys only cover works released before 2023 [16,
18, 19, 20]. This ensures that we encompass the new emerging detection tasks
besides conventional ones. Specifically, we include multi-modal deepfake which
represents the trend in real-world scenarios, while existing works[16, 4, 18]
merely discuss visually manipulated deepfakes. Besides, we also include the
detection of deepfakes by novel generative techniques that are not considered
in many previous reviews[16, 4, 18, 19], i.e., synthetic images generated by
diffusion models. This survey incorporates the most advanced and state-of-the-
art techniques in detection, providing readers with insights into cutting-edge
methods and their effectiveness against evolving threats. As such, it serves
as an invaluable resource for researchers and practitioners aiming to remain
at the forefront of technological advancements in this rapidly evolving field.

More importantly, as illustrated in Figure 3, we offer a more detailed and
systematic categorization of the literature than previously available, unlike
previous studies [16] and [4] that categorize deepfake detection methods into
less than four types. We contend that our granular examination helps
illustrate how different methods enhance detection capabilities in different
aspects. By sorting the works according to the manipulation modalities they
address, from single-modal to multi-modal detection, our survey clearly
differentiates between the techniques used in various contexts.

Within the single-modal category, we distinguish between passive and proactive
detection methods, categorizing them based on their objective—whether
detecting deepfakes post-generation or preemptively preventing their creation.
In passive detection, we recognize the generalization as a core issue, thus
dividing the literature into basic and generalized detection. For each type,
we further categorize works based on the type of knowledge utilized—artifacts
or consistencies—and the stage of application, whether at the input level,
model level, or learning level. We also delve into specialized methods
designed for newer generative models like DMs, which have not been extensively
covered in previous reviews. For proactive detection, we categorize the works
based on the generation models they target, such as GANs or DMs.

As the first work comprehensively investigating the multi-modal facial forgery
detection domain, predominantly focusing on audio-visual interactions, our
survey explores various multi-modal knowledge utilization strategies,
including independent learning, joint learning, matching-based approaches,
etc.

In summary, our survey not only brings the academic community up to date with
the latest developments but also introduces a more nuanced categorization and
analysis of facial forgery detection techniques. This structured approach not
only aids in understanding the complexities of the field but also inspires new
research directions and methodological innovations, making it an essential
resource for anyone involved in the study of this direction.

Scope of This Survey. This survey is structured as follows: Section 2 provides
essential background information, defines key terminologies, discusses
relevant datasets, and explains evaluation metrics for deepfake detection.
Section 3 offers a detailed analysis of deepfake detection methods,
categorizing them from single-modal to multi-modal and from passive to
proactive approaches, as illustrated in Figure 3. Section 4 discusses current
challenges in deepfake detection and suggests future research directions. The
survey wraps up with Section 5, summarizing the main insights and
contributions.

##  2 Background

###  2.1 Task Formulation

In this subsection, we formally define single-modal and multi-modal deepfake
detection tasks discussed in our survey, demonstrating the types of input
media and the forms of detection outputs under various scenarios.

Single-modal Detection. In single-modal cases, we denote a visual deepfake
dataset as $\mathcal{D}=\\{(\mathbf{X_{i}^{v}},Y_{i})\\}_{i=1}^{N}$, where
$\mathbf{X_{i}^{v}}\in\mathbb{R}^{T\times 3\times H\times W}$ representing a
video with $T$ frames and $Y_{i}\in\\{real,fake\\}$ indicating the
authenticity of $\mathbf{X_{i}^{v}}$. Given a detector
$F_{\theta}:\mathbb{R}^{T\times 3\times H\times W}\mapsto\\{real,fake\\}$
parameterized with $\theta$, the aim is to optimize it in terms of the loss
function, either simple cross-entropy loss for classification or advanced ones
demonstrated in section 3.1.1.2.

$\theta=\operatorname*{arg\,min}\limits_{\theta^{\prime}}\mathbb{E}_{\mathcal{D}}L(F_{\theta^{\prime}}(\mathbf{X^{v}}),Y)$
(1)

Along with binary classification, spatial and temporal localization are
proposed to facilitate detection. Spatial localization applies to image and
video input, predicting a mask of the skeptically manipulated region, while
temporal localization pertains exclusively to video inputs, requiring the
output of a temporal mask indicating the segment(s) manipulated in the given
video.

(a) Multi-modal audio-visual deepfake dataset FakeAVCeleb[5], figure modified
from [5].

(b) Multi-modal visual-text deepfake dataset $DGM4$, figure from [6].

Figure 1: Multi-modal deepfake detection demonstration.

Multi-modal Audio-Visual Detection[10]. This task tackles the problem when the
visual input is video and accompanied by audio stream input, either or both of
which can be manipulated. In this case, we denote the paired audio-visual
dataset in Figure 1(a) as
$\mathcal{D}=\\{(\mathbf{X_{i}^{v}},\mathbf{X_{i}^{a}},Y)\\}_{i=1}^{N}$, where
$\mathbf{X_{i}^{v}}$ is the video defined previously,
$\mathbf{X_{i}^{a}}\in\mathbb{R}^{M\times c}$ represents the auditory waveform
with $c$ channels and $M$ sample points, and $Y\in\\{real,fake\\}$ is the
binary target label. Similar to the single-modal scenario, multi-modal audio-
visual deepfake detection is to obtain an optimal detector
$F_{\theta}:\mathbb{R}^{T\times 3\times H\times W}\times\mathbb{R}^{M\times
c}\mapsto\\{real,fake\\}$ incorporating both visual and auditory inputs:

$\theta=\operatorname*{arg\,min}\limits_{\theta^{\prime}}\mathbb{E}_{\mathcal{D}}L(F_{\theta^{\prime}}(\mathbf{X^{v}},\mathbf{X^{a}}),Y)$
(2)

Optionally, a more fine-grained classification can also be performed, where
$F_{\theta}$ further predicts $Y^{v}\in\\{real,fake\\}$ and
$Y^{a}\in\\{real,fake\\}$ for each modality.

Multi-modal Text-Visual Detection and Grounding This task involves analyzing
image-text pairs as illustrated in Figure 1(b), the dataset is denoted as
$\mathcal{D}=\\{(\mathbf{X_{i}^{v}},\mathbf{X_{i}^{t}},Y^{v}_{i},Y^{t}_{i},\mathbf{B^{v}_{i}},\mathbf{M^{t}_{i}},Y_{i})\\}_{i=1}^{N}$,
where $\mathbf{X^{v}_{i}}\in\mathbb{R}^{3\times H\times W}$ denotes the image
input, $\mathbf{X^{t}_{i}}\in\mathbb{R}^{L\times d}$ represents text with $L$
tokens of $d$-dimension, $\mathbf{B^{v}_{i}}\in\mathbb{R}^{4}$ is the
manipulated bounding box, $\mathbf{M^{t}_{i}}\in\mathbb{R}^{L\times 1}$ is
manipulation mask of text modalities, $Y^{v}$, $Y^{t}$ and $Y$ indicate the
authenticity of visual modality, text modality, and image-text pair
respectively. Referring to Figure 1(b), $\mathbf{X^{v}}$ and $\mathbf{X^{t}}$
are the image and text in the middle, $\mathbf{B^{v}}$ and $\mathbf{M^{t}}$
correspond to the grounding annotation in the right part, $Y$, $Y^{v}$ and
$Y^{t}$ are the binary authenticity labels in the left. In this task, the
model needs to predict binary labels and manipulation masks, i.e.,
$\hat{Y}^{v},\hat{Y}^{t},\mathbf{\hat{B}^{v}},\mathbf{\hat{M}^{t}}=F_{\theta}(\mathbf{X^{v}},\mathbf{X^{t}})$.
The detector is trained to minimize the designed loss given the dataset:

$\theta=\operatorname*{arg\,min}\limits_{\theta^{\prime}}\mathbb{E}_{\mathcal{D}}\\\
L(F_{\theta^{\prime}}(\mathbf{X^{v}},\mathbf{X^{t}}),Y,Y^{v},Y^{t},\mathbf{B^{v}},\mathbf{M^{t}})$
(3)

###  2.2 Datasets for Facial Forgery Detection

In the rapidly evolving domain of deepfake detection, the formulation and
assessment of detection algorithms heavily rely on various and exhaustive
datasets. These datasets are broadly classified into single-modal and multi-
modal categories based on the type of data they encompass.

Single-modal datasets are characterized by their focus on a singular form of
evidence, providing specialized insights into detecting deepfakes through one
sensory modality. Conversely, multi-modal datasets compile samples
incorporating a variety of data types, e.g., visual, audio, and text. This
integration mirrors the complex nature of deepfakes as they appear in real-
world scenarios, offering a more holistic approach to detection by leveraging
the richness of multiple data streams. We have summarized the database details
related to facial forgery detection in Table 2.2.

TABLE II: Overview of Deepfake Datasets focusing on facial forgery. A: audio
modality; V: visual modality; T: text; BC: binary classification; TG: temporal
grounding; SG: spatial grounding; FS: face swap; FR: face reenactment; FE:
face editing; TTS: text-to-speech; VC: voice conversion; WIG: whole image
generation; TS: text swap.

Dataset* | Year | Type |  Tasks | Manipulation Method | #Subjects | #Real | #Fake
---|---|---|---|---|---|---|---
|  |  |  |  A |  V | T |  |  |
FaceForensics++ [21] | 2019 |  Visual Deepfake by GAN/VAE |  BC |  - |  FS/FR | - | - | $1,000$ | $4,000$
DFD [22] | 2019 |  BC |  - |  FS/FR | - | - | $363$ | $3,068$
FaceShifter[23] | 2020 |  BC |  - |  FS | - | - | - | $1,000$
DFDC [24] | 2020 |  BC |  - |  FS | - | $960$ | $23,654$ | $104,500$
Celeb-DF[25] | 2020 |  BC |  - |  FS | - | $59$ | $590$ | $5,639$
DeeperForensics-1.0[26] | 2020 |  BC |  - |  FS | - | $100$ | $50,000$ | $10,000$
WildDeepfake[27] | 2020 |  BC |  - |  - | - | - | $3,805$ | $3,509$
KoDF[28] | 2020 |  BC |  - |  FS | - | $403$ | $62,166$ | $175,776$
FFIW10K[29] | 2021 |  BC/SG |  - |  FS | - | - | $10,000$ | $10,000$
ForgeryNet[30] | 2021 |  BC/SG/TG |  - |  FS/FR | - | $5,400$ | $99,630$ | $121,617$
DF-Platter[31] | 2023 |  BC |  - |  FS | - | $454$ | $133,260$ | $132,496$
DeepFakeFace[32] | 2023 |  Visual Deepfake by DM |  BC |  - |  WIG | - | - | $30,000$ | $90,000$
DiffusionFace [33] | 2024 |  BC |  - |  FS/WIG | - | - | $30,000$ | $600,000$
DiFF [34] | 2024 |  BC |  - |  WIG/FS | - | $1,070$ | $23,661$ | $537,466$
FakeAVCeleb [5] | 2021 | Audio-Visual Deepfake |  BC |  SV2TTS |  FR | - | $500$ | $500$ | $19,500$
LAV-DF [35] | 2022 |  BC/TG |  TTS |  FR | - | $153$ | $36,431$ | $99,873$
AV-Deepfake1M[7] | 2023 |  BC/TG |  TTS |  FR | - | $2,068$ | $286,721$ | $860,039$
DGM4[6] | 2023 | Text-Visual Deepfake |  BC/SG |  - |  FS/FE | TS | - | $77,426$ | $152,574$

  * *

Only publicly available datasets are listed.

####  2.2.1 Single-modal (visual) Deepfake Datasets

Initially, computer graphics techniques are favored to build up deepfake
datasets. However, with the advancement of generative AI technologies, GANs,
AEs, and DMs have increasingly become preferred methods for deepfake data
generation [21, 25, 26, 24, 27, 28, 33, 32]. This shift towards generative AI
techniques reflects the progress in the field, offering more sophisticated and
convincing deepfakes by leveraging the power of AIGC methods to manipulate or
generate content. We show some examples of visual-modal deepfake in Figure 2.

GANs/VAEs based Generated Datasets. The landscape of deepfake detection has
been significantly enriched by datasets leveraging GANs and AEs. Pioneering
datasets such as FaceForensics++ (FF++)[21], DeepFake Detection Challenge
(DFDC) [24], Celeb-DF (and its successor Celeb-DF v2)[25],
DeeperForensics-1.0[26], WildDeepfake[27], the Korean Deep-Fake Detection
Dataset (KoDF)[28], FFIW10K[29], and ForgeryNet[30] stand at the forefront of
this effort. These datasets utilize GANs or VAEs to create synthetic face
images that spans various manipulations and scenarios.

FaceForensics++ (FF++)[21] sets a benchmark with its comprehensive collection
of manipulations, including face swapping and reenactment through methods like
DeepFakes, Face2Face, FaceSwap, and NeuralTextures. A subsequently proposed
face swapping technique FaceShifter[23] is applied to the real videos in
FF++[21], resulting in a new set of fake samples. DFD[22] enriches deepfake
dataset by combining the manipulated and pristine videos recording paid and
consenting actors. DFDC [24] challenges detection algorithms with a diverse
set of forgery techniques, including eight different methods such as FSGAN
[36] and StyleGAN[37], aimed at generating various facial forgeries.

Celeb-DF series [25] harnesses VAEs to craft deepfake images by replicating
facial expressions across donor and target faces, pushing the envelope in
visual manipulation realism. DeeperForensics-1.0[26] introduces the DeepFake
Variational Auto-Encoder (DF-VAE), a learning-based face swapping framework to
generate convincing forgeries. KoDF[28] diversifies the dataset pool by
incorporating multiple synthesis models and focusing on Korean subjects,
addressing the underrepresentation of Asian demographics in prior deepfake
detection datasets.

FFIW10K[29] is a large-scale dataset specifically designed for multi-person
face forgery detection, highlighting the complexity of detecting alterations
among multiple genuine faces. DF-Platter[31] improves deepfake creation
techniques using low-resolution videos and emphasizes a gender-balanced,
ethnically diverse subject pool. WildDeepfake[27] collects samples from the
internet, offering a real-world testbed for detection algorithms. To expand
the detection challenge to include spatial and temporal localization tasks,
ForgeryNet [30] introduces a comprehensive resource with more than $5,400$
subjects and $15$ manipulation techniques.

During the Covid-19 pandemic, Ko et al. [38] develop a new deepfake dataset
featuring face masks by applying masks to images from well-known deepfake
collections and noticed a significant drop in detection accuracy of current
deepfake detection models when analyzing faces with masks. All these GANs/VAEs
based deepfake datasets contribute uniquely to the realm of detecting forgery
images, providing researchers with fundamental resources to address the
challenges of deepfake detection.

DMs based Generated Datasets. Recent advancements in Diffusion Models have
significantly enhanced the generation of visually compelling content. This has
spurred the development of specialized datasets focusing particularly on
facial forgery detection, such as DeepFakeFace [32], DiffusionFace [33],
DiffusionDB-Face/JourneyDB-Face [39], and Diff[34]. It should be noted that
most manipulations in the aforementioned DM-generated datasets are focused on
the image level. Extending them to video-level deepfakes requires further
exploration.

DeepFakeFace[32] employs three distinct DMs—Stable Diffusion v1.5, Stable
Diffusion Inpainting, and InsightFace—each generating $30,000$ synthetic
images to create a richly diverse dataset tailored for deepfake detection.
DiffusionFace[33] stands as the inaugural dataset dedicated to diffusion-based
facial forgery, covering a wide range of manipulations from unconditional
generation to sophisticated face swapping techniques. It utilizes an
impressive array of eleven diffusion models to achieve a new level of realism
in forged facial imagery, showcasing the adaptability of diffusion
technologies and establishing a benchmark for DMs-based deepfake detection.
DiffusionDB-Face and JourneyDB-Face[39] introduce cutting-edge benchmarks for
deepfake detection, emphasizing a variety of head poses, facial attributes,
styles, and appearances.

The newly developed DiFF[34] is a substantial collection consisting of more
than $500,000$ images synthesized using $13$ different generation methods. It
utilizes $30,000$ carefully selected textual and visual prompts to ensure that
the synthesized images maintain both high fidelity and semantic consistency.

While these datasets primarily support the detection of facial forgeries, the
evolving landscape of deepfake content also necessitates broader detection
capabilities as seen in datasets like RWDF-23 [40] and Zoom-DF [41]. The
RWDF-23 dataset [40], launched in 2023, includes $2,000$ deepfake videos from
four online media-sharing platforms, collected between January and May 2023.
It encompasses videos in four languages, providing a diverse set of real-world
deepfake content. Additionally, this dataset features viewer comments and
interactions, and the authors provide a thorough analysis of deepfakes,
covering aspects such as creators, manipulation strategies, intentions, and
the context of real-world content production. Furthermore, the study in [41]
introduces Zoom-DF tailored for remote meeting and video conferencing
scenarios. Unlike traditional datasets that focus primarily on facial
manipulation, Zoom-DF uniquely concentrates on altering participant movements,
expanding the scope of deepfake detection challenges.

Figure 2: Examples of single-modal deepfake datasets. Images in the top row
are fake samples from FF++[21]. Images in the bottom row are generated by
Stable Diffusion v1.5. Figure from [32].

####  2.2.2 Multi-modal Deepfake Datasets

Multi-modal deepfake datasets are becoming indispensable as they provide a
more intricate testing environment that closely mirrors actual digital
interactions, where deceptive content might include any combination of altered
videos, synthesized speech, and manipulated text. The multi-modal deepfake
datasets simulate the real-world scenarios of how manipulated content delivers
misinformation more convincingly. The development of such datasets aims to
refine detection algorithms by offering a broad spectrum of data types for
analysis. As illustrated in Figure 1, either or both auditory and visual
content are forged in audio-visual deepfake dataset, and text-visual deepfake
dataset involves text-image pairs where each modality can be falsified.

Audio-Visual Deepfake Datasets Datasets like FakeAVCeleb [5] and Joint Audio-
Video Deepfake [10] enhance the depth of deepfake detection research by
including video deepfakes with corresponding synthesized, lip-synced audio
tracks or by integrating audio alterations into existing visual datasets.
These efforts, alongside others like DefakeAVMiT [42] and Trusted Media
Challenge (TMC)[43], contribute substantially to building resources capable of
addressing the nuanced challenges posed by audio-visual deepfakes. MMDFD[44]
extends beyond audio-visual deepfake datasets by encompassing text modality
represented as video subtitles, however, without releasing public access thus
hindering its further comprehensive evaluation benchmarking.

Moreover, specialized datasets such as Localized Audio Visual DeepFake (LAV-
DF)[35] and AV-Deepfake1M[7] focus on the precise detection of temporal
forgeries and localization, targeting content manipulations that subtly change
the sentiment. These datasets underscore the evolving sophistication in
deepfake generation and the pressing need for equally sophisticated detection
methodologies.

Text-Viusal Deepfake Datasets The DGM4 dataset[6] represents a significant
advancement in deepfake detection, focusing specifically on facial forgery
within multi-modal media contexts. Unlike other datasets that primarily
address image-text mismatches, DGM4 challenges detectors to identify forgeries
by analyzing both images and their associated text. This dataset does not only
support binary classification but also involve the detailed task of marking
manipulated sections within images and identifying forged text elements,
enhancing its utility for complex multimedia forgery detection.

In general, these efforts[5, 43, 10, 35, 42, 7, 44, 6] encapsulate a
significant step toward developing versatile resources for testing and
improving detection methods across the complicate multi-modal deepfakes.

###  2.3 Evaluation

Evaluation Metrics In prior work involving only the visual modality, where
deepfake detection is cast as a classification task, usually a binary
classification task, the widely adopted evaluation metrics include $ACC$
(accuracy), $AUC$ (area under ROC curve), $F1-score$, $AP$(average precision),
and $EER$ (equal error rate).

Under multi-modal scenarios, where the fine-grained detection for each
modality is in a multi-label manner[12, 6], $mAP$ is also reported in addition
to the aforementioned metrics.

In recent text-visual grounding works [6], $mIoU$, $IoU_{50}$, $IoU_{75}$ are
used as evaluation metrics for visual grounding, where the bounding box of the
manipulated region is predicted. Regarding text grounding, where token-level
binary prediction is made to indicate the authenticity of each text token, the
classification precision and recall are evaluated.

Evaluation Settings The prevalent evaluation for deepfake detection algorithms
can be categorized into in-domain and cross-domain settings according to the
distributional gap between the training and testing data.

In-domain evaluation refers to evaluating the detectors on testing data that
follows the same distribution as training data. In prior works [21, 24, 26],
practitioners conduct in-domain evaluation using non-overlapping testing and
training data from the same dataset, or in a narrower sense, of the same
manipulation type having identical forgery patterns. Under this setting, the
evaluation results reflect the efficacy of detectors under a controlled
conditions, providing insight into their performance in an idealized
environment.

Cross-domain evaluation is the case that testing data and training data are of
different distributions, requiring detectors to possess generalizability in
real scenarios. The burgeoning generative techniques highlight the necessity
of cross-domain evaluation to prevent detectors from overfitting on the
specific manipulation types seen during training. There are three major
settings to evaluate the cross-domain generalizability:

  * •

Cross-dataset: Training and testing sets are from different datasets, e.g.,
using FF++[21] for training and Celeb-DF[25] for testing.

  * •

Cross-manipulation: Training and testing occur on the same dataset but involve
different manipulation types, e.g., training on FaceSwap manipulations from
FF++ and testing on NeuralTextures. This setting requires detailed forgery
type annotations besides real/fake labels.

  * •

Cross-postprocessing: Evaluates how detectors perform on deepfakes that have
undergone various post-processing operations, e.g., JPEG compression or video
compression rate change[26], simulating real-life scenarios where media might
be compressed or otherwise altered before spreading. For example, detectors
might be trained on high-quality, raw versions of FF++ and tested on a
compressed, lower-quality version of the same dataset.

###  2.4 History and Challenges of Deepfake Datasets

The evolution of deepfake generation methods has closely paralleled
advancements in generative AI technologies, transitioning from early
techniques such as GANs and auto-encoders to the more recent and sophisticated
Diffusion Models. This progression has necessitated a corresponding shift in
deepfake detection strategies, moving from identifying simple visual artifacts
to decoding subtler patterns that mirror genuine content, challenging both
automated systems and human observers.

Moreover, the emergence of multi-modal deepfake datasets introduces new
challenges, requiring detectors to understand and analyze the interactions
across different data types, for example, video, audio and text. This
integration reflects the real-world scenarios where deepfakes are used, adding
layers of difficulty in detection tasks.

In the mean time, the manipulation techniques within deepfakes are also
becoming more refined, targeting only specific faces of an image or select
frames within a video sequence. This subtlety demands that deepfake detection
systems not only identify the presence of fakes but also pinpoint the exact
areas of manipulation with high precision.

These developments across various manipulation methods, modalities, and
improved qualities highlight the ongoing need for adaptive detection
methodologies in response to evolving deepfake datasets.

##  3 Method

In the following subsections, we review works predominantly published in the
last three years, encompassing three major areas: single-modal detection
methods, multi-modal detection methods, and the works related to trust-worthy
deepfake detection. Given the extensive array of methods and the intricate
technical details inherent in each category, it is impractical to cover all
available techniques. Thus, in this survey, we will focus on highlighting key
works within each category to delineate the evolutionary trajectory. Figure 3
illustrates our taxonomy of single- and multi-modal deepfake detection in
detail.

for tree= where n children=0leafnon leaf, l sep+=0.5pt, grow’=east, edge=gray,
thick, parent anchor=east, child anchor=west, calign=center, edge path= [draw,
edge] (!u.parent anchor) – +(1.5pt,0) —- (.child anchor)edge label, s
sep=15pt; , if=isodd(n_children()) for children=
if=equal(n,(n_children(”!u”)+1)/2)calign with current , [Deepfake Detection,
text width=1.2cm [Single-modal (Visual) Detection, tier=level1 [Passive
Detection, tier=level2 [Basic Detection for GAN/VAE, tier=level3 [Visual
Artifact-based, tier=level4 [[45][21][46][47][48][49][50][51][52][53][54][55],
tier=last] ] [Consistencies-based, tier=level4
[[56][57][58][59][60][61][62][63][64][65], tier=last] ] ] [Generalized
Detection for GAN/VAE, tier=level3 [Input Level, tier=level4 [Augmentation-
based, tier=level5 [[66][67][68][69][70][71][72][72], tier=last] ] [Frequancy-
space Based, tier=level5 [[73][74][75][76][77][78][79][80][81][82], tier=last]
] ] [Model Level, tier=level4 [Transformer-based, tier=level5
[[83][84][85][86][87][88][89][90], tier=last] ] [Vision Language Models,
tier=level5 [[91][92][93][94][95][96][97], tier=last] ] ] [Learning level,
tier=level4 [Advanced Loss Functions, tier=level5 [[98][99][100], tier=last] ]
[Disentangled Representation, tier=level5 [[101][102][103][104], tier=last] ]
[Reconstruction, tier=level5 [[105][106][107][108], tier=last] ] [Manipulation
Localization, tier=level5 [[109][110][111][112][113][114][115][116],
tier=last] ] [Identity Discrepancy, tier=level5
[[117][118][119][120],tier=last] ] [Knowledge Distillation, tier=level5
[[121][122][123][124][125], tier=last] ] [Fine-grained Prediction, tier=level5
[[126][127][128],tier=last] ] ] ] [Detection for Diffusion Model, tier=level3
[[129][130][131][34][132][133][134], tier=last] ] ] [Proactive Detection,
tier=level2 [Proactive Methods for GANs, tier=level3
[[135][136][137][138][139][140][141][109][142][143][14][144][145][146],
tier=last] ] [Proactive Methods for DMs, tier=level3
[[15][147][148][149][150][151], tier=last] ] ] ] [Multi-modal Detection,
tier=level1 [Audio-Visual, tier=level2 [Independent Learning, tier=level3
[[10][11][152][153], tier=last] ] [Joint Learning, tier=level3 [Intermediate
Fusion, tier=level4 [Cross-Attention, tier=level5 [[10][42][154], tier=last] ]
] [Late Fusion, tier=level4 [Concatenation &

Figure 3: The taxonomy of deepfake detection methods.

Addition, tier=level5 [[11][152][155][156][157][35][13], tier=last] ]
[Attention, tier=level5 [[13][158][159], tier=last] ] [MLP Mixer Layer,
tier=level5 [[12], tier=last] ] [Multi-task Strategy, tier=level5
[[10][42][156][12][160],tier=last] ] [Regularization, tier=level5
[[158][159][42][157], tier=last] ] ] ] [Matching-based Learning, tier=level3
[[161][162], tier=last] ] [Others, tier=level3 [[163][164][165][166],
tier=last] ] ] [Text-Visual, tier=level2 [[6][167][168][169], tier=last] ] ] ]

###  3.1 Single-modal Detection

In this section, we concentrate on works related to single-modal methods for
facial forgery detection in images and videos. We categorize these methods
into two major groups: passive detection methods and proactive detection
methods.

####  3.1.1 Passive Detection Methods

Passive deepfake detection employs methods and techniques to identify
manipulated videos or images based solely on the available content. This
approach operates without active intervention or additional information from
the creation process. It focuses exclusively on analyzing the existing content
to extract artifacts and inconsistencies, without relying on external inputs
or markers embedded during the content’s creation. In the following
subsections, we review the works that target GAN/VAE generated deepfakes,
covering basic and generalized detection methods. For both basic and
generalized approaches, we further categorize previous works based on their
distinct characteristics.

##### 3.1.1.1 Basic Detection for GAN/VAE

Visual Artifacts-based Methods Early efforts in deepfake detection primarily
focused on identifying visual artifacts. Afchar et al.[45] develop MesoNet, a
streamlined model adept at analyzing mesoscopic features of images to detect
forgeries. Rossler et al.[21] utilize the XceptionNet framework to discern
subtle artifacts in manipulated images. In [46], Liu et al. develop Gram-Net
to leverage global image characteristics to effectively differentiate between
real and fake facial textures. Zhao et al.[47] introduce a multi-attentional
framework that improves the detection of textural features indicative of
deepfakes. Tan et al.[48] and their subsequent work [49] propose to focus on
extract artifacts introduced by up-sampling operations, which are more subtle
and often overlooked in traditional artifact analysis. It should be noted that
recent methods, such as [47, 48, 49], have been experimentally validated for
application in generalized deepfake detection.

Advancements continued with Corvi et al.[50] observing that GAN models produce
noticeable artifacts during the process. Those artifacts are visible in the
Fourier domain and exhibit unusual regular patterns in their auto-correlation.
Chen et al. [51] propose using a single patch in conjunction with widely
recognized Spatial Rich Model filters to extract artifacts for deepfake
detection. Juet et al. [52] introduce the Global and Local Feature Fusion
(GLFF) framework to strategically select informative patches for the detailed
extraction of local artifacts. Zhang et al. [53] discover that real and
synthetically generated images show significantly different latent Gaussian
representations when subjected to an inverse diffusion process using a pre-
trained diffusion model. By exploiting these distinctions, they can enhance
and accentuate subtle artifacts within generated images, improving the
effectiveness of their detection methods. Zhang et al. [54] introduce the
Cross Appearance-Edge Learning detector, a tool specifically designed to
recognize forgery by capturing both coarse and fine-grained appearance and
edge patterns at multiple levels of granularity. This method effectively
merges knowledge from both the appearance and edge domains to thoroughly
expose a broad spectrum of forgery artifacts. Nguyen et al.[55] introduce a
multi-task learning system with three key branches: classification, heatmap
regression, and self-consistency regression. The heatmap regression and self-
consistency regression branches are specially designed to pinpoint vulnerable
areas within an image, areas that are particularly indicative of blending
artifacts, thereby improving the framework’s ability to detect forgeries.

Consistencies-based Methods The field expanded into temporal artifact
detection as researchers recognize the importance of dynamic inconsistencies
in videos. Li et al.[56] utilize Long-term Recurrent Convolutional Networks to
detect inconsistencies around blinking in videos, while Qi et al.[57]
introduce DeepRhythm, which analyzes heartbeat rhythms to spot video
forgeries. Haliassos et al.[58] design a spatio-temporal network to detect
semantic inconsistencies in mouth movements within forgery videos. The study
in [59] introduces a deepfake detection method called latent pattern
detection, which identifies temporal inconsistencies across video frames.
Haliassos et al.[170] develop a method that employs self-supervised learning
to capture detailed video representations by using audio-visual correlations
in genuine videos. The detector is then trained on both real and synthetic
data to detect facial forgeries and to predict the learned representations
from the initial stage for authentic videos. Pang et al.[60] propose MRE-Net
consisted of two main components: Bipartite Group Sampling (BGS) and Multiple
Rate Branch (MRB). BGS divides the video into bipartite groups to capture
various face motion dynamics, while MRB captures short-term and long-term
spatial-temporal inconsistencies, respectively. Wang et al.[61] capture
spatio-temporal artifacts in one model based on disentangled learning
strategy, dividing the convolutional weights of a 3D ResNet into spatial-based
and temporal-based groups and updating the two groups alternatively during
training. Xu et al.[62] introduce the Thumbnail Layout (TALL) strategy to
enhance the ability to capture inconsistencies across frames in fake videos.
Their following work TALL++[63] is built on top of TALL[62] by designing a
graph reasoning block and introducing a consistency loss to better capture
semantic-level inconsistency to improve generalizability. TI2Net [64] uses
identity inconsistencies in video frames to authenticate clips. It encodes
identity information from each frame into vectors and calculates the temporal
identity embedding by analyzing the differences between these vectors. The
temporal identity embedding effectively captures identity inconsistencies,
which is crucial to determining the authenticity of the video input. In [65],
Chen et al. focus on detecting compressed deepfake videos, which often present
overlapping compression and tampering artifacts. In a recent work on video
deepfake detection [171], Trevine et al. propose a two-stage training
pipeline. The first stage uses self-supervised representation learning to
enforce audio-visual correspondence with abundant real face videos, combining
contrastive learning with a novel audio-visual complementary masking and
feature fusion strategy. The second stage involves supervised downstream
classification to train a classifier that takes advantage of the lack of
cohesion in deepfake videos, thereby enhancing detection accuracy.

##### 3.1.1.2 Generalized Detection for GAN/VAE

In the realm of deepfake detection, the capacity of generalized detection
plays an important role in enhancing the adaptability of the detection methods
against a wide range of manipulations. In this subsection, we delve into
various categories of approaches such as input level, model level, and
learning level methods. Each category provides their own perspectives and
techniques to tackle the challenge of generalized deepfake detection.

Input level: Augmentation-based Methods Augmentation-based methods have
emerged as a powerful strategy in deepfake detection by enriching training
data with complex pseudo-fake examples. Wang et al. [66] develop an attention-
based method focused on identifying key facial regions to lay the groundwork
for targeted data enhancement. Specifically, the technique involves adaptively
masking the facial areas most susceptible to manipulation, as identified
through attention analysis. Building on the theme of using real image
characteristics, Zhao et al. [67] introduce the Inconsistency Image Generator
(I2G). This tool creates well-annotated training datasets by blending genuine
images, thereby avoiding artifacts typically introduced by deep neural
networks. Further exploring the diversity of forgery characteristics, Chen et
al. [68] propose a method to improve the variety of forgeries the model
encounters during training. By employing an adversarial training approach,
this method not only increases the model’s exposure to diverse forgeries but
also improves its ability to detect nuanced differences, making it more adept
at identifying sophisticated fake content.

In further exploration, Shiohara et al. [69] utilize synthetic training data
through ”self-blended images” (SBIs), which simulate common forgery indicators
like blending seams and statistical discrepancies. Larue et al. [70] develop a
data augmentation technique that subtly introduces soft discrepancies into
synthetic face images. These slight adjustments help train the model, SeeABLE,
to better recognize subtle inconsistencies. Li et al. [172] develop a method
to enhance detection accuracy for low-quality face forgeries. They introduce a
”bleach generator” that synthesizes a corrective element that, when blended
with the input forgery image, produces a ”bleached” image. This approach,
bolstered by their image bleach generation technique and two specific loss
constraints, effectively improves the detection results.

The prior methods create augmented forgery images through a pixel-level
process. Recent works have proposed to conduct the augmentation in latent
representation space [71] or feature space [72]. Yan et al. [71] introduce
Latent Space Data Augmentation (LSDA), which innovatively manipulates the
latent space to generate a broad spectrum of forgery variations. This approach
minimizes the risk of overfitting to specific forgery methods and broadens the
model’s ability to generalize across various types of manipulated content. In
[72], FreqBlender advances pseudo-fake face generation by focusing on
frequency knowledge. The approach is based on the hypothesis that faces
consist of semantic, structural, and noise information, with structural
information most related forgery traces. The authors employ a Frequency
Parsing Network to adaptively segment critical frequency components, and
create augmented fake samples by adaptively combining those components.

Input Level: Frequency-space based Methods Recently, researchers have expanded
beyond RGB space analysis, developing sophisticated techniques that utilize
frequency analysis to enhance deepfake detection. Qian et al.[73] highlight
the role of frequency analysis in detecting forgery, introducing the Frequency
in Face Forgery Network (F3-Net). This innovative two-stream collaborative
learning framework leverages frequency-aware decomposed image components and
local frequency statistics to identify forgery patterns effectively. Chen et
al.[74] develop a face forgery detection method that examines connections
between local image regions. They utilize the Multi-scale Patch Similarity
Module (MPSM) to detect complex similarity patterns and the RGB-Frequency
Attention Module (RFAM) to integrate multi-modal data. Li et al.[75] create an
adaptive frequency feature generation module to dynamically identify forgery
patterns in images. Their approach also introduces a single-center loss (SCL)
technique to reduce intra-class variations among genuine faces and enhance
class distinctions, improving feature discrimination in face forgery
detection.

Building on these foundational works, subsequent research has introduced
advanced approaches with additional complexities. Luo et al.[76] propose a
method that leverages high-frequency knowledge through three modules: multi-
scale high-frequency feature extraction, residual-guided spatial attention,
and cross-modality attention, aimed at enhancing forgery detection by focusing
on manipulation signs and harmonizing data aspects. Liu et al.[77] introduce
Spatial-Phase Shallow Learning (SPSL), a method that utilizes up-sampling
artifacts common in forged content. SPSL combines spatial analysis with phase
spectrum insights, emphasizing local texture information to detect forgeries.

Further, Woo et al.[78] present the Attention-based Deepfake detection
Distiller (ADD), which applies frequency domain learning and optimal transport
theory in knowledge distillation. ADD uses frequency attention distillation to
recover high-frequency components lost during compression and multi-view
attention distillation to generate multiple attention vectors. Jeong et
al.[79] create frequency-level perturbation maps to blur the distinctions
between authentic and fabricated images. Their method, similar to adversarial
learning, refines both the deepfake detector and the perturbation generator,
enhancing the model’s ability to recognize forgeries from various GAN models,
although adding complexity and computational demands. In [80], Wang et al.
propose a Spatial-Frequency Dynamic Graph method using dynamic graph learning
to analyze relation-aware features across spatial and frequency domains. Tan
et al.[81] introduce FreqNet, a deepfake detection network that integrates
frequency domain learning with a CNN classifier to significantly improve
generalization across different scenarios. By applying convolutional layers to
both phase and amplitude spectra, their approach detects a broad range of
forgery indicators within the frequency domain. Doloriel et al.[82] find that
frequency masking improves the generalization of deepfake detection across
various generative models, achieving notable performance improvements.

Model Level: Transformer-Based Approaches Recently, transformers [8] have been
employed in both static deepfake detection and temporal deepfake segmentation.
Wang et al.[83] provide an overview (as of February 28, 2024) of fourteen
works adopting Transformer architecture for deepfake detection.

The approach outlined in Saha et al. [84] utilizes a Transformer-based
architecture for segmenting deepfake videos, specifically targeting frames
that have been manipulated. This method integrates a Vision Transformer (ViT)
to analyze spatial features using a Scaling and Shifting (SSF) strategy,
alongside a Timeseries Transformer (TsT) that focuses on temporal aspects. The
spatial data processed by the ViT is segmented into overlapping windows, which
are then analyzed temporally by the TsT to detect deepfake manipulations
effectively. In [85], Bai et al. introduce the Action-Units Relation Learning
framework, which utilizes a newly designed Action Units Relation Transformer
(ART) to enhance forgery detection by analyzing relationships between facial
action units (AUs). ART incorporates both AU-specific and AU-agnostic branches
to effectively uncover forgery clues. Furthermore, the framework includes the
Tampered AU Prediction (TAP) technique, which manipulates AU-related regions
to create challenging training samples, thereby refining the detection
process.

Using advances in parameter-efficient tuning methods such as Adaptors[86, 88],
Mixture of Experts[89, 88], and LoRA[89, 88], innovative strategies are being
proposed to integrate these techniques into large pre-trained Vision-
Transformers that serve as foundation models. These strategies elegantly
address the data-demanding issue while adapting foundation ViTs to deepfake
detection and circumvent distortion of originally learned rich knowledge
during pre-training. The study by Nguyen et al. [90] investigates the
effectiveness of self-supervised pre-trained vision transformers compared to
supervised pre-trained transformers and traditional ConvNets. The findings
suggest that transformers trained with self-supervised learning on broad
datasets, not specifically tailored for deepfake detection, can generally
outperform those trained through supervised methods. The DeepFake-Adapter [86]
enhances a ViT by adding lightweight, dual-level adapter modules without
altering the core architecture. In this method, Shao et al. propose to
incorporate Globally-aware Bottleneck Adapters alongside the MLP layers to
identify broad forgery cues and uses Locally-aware Spatial Adapters to
interact with ViT features, enhancing the detection of localized anomalies in
deepfake content. Similarly, Luo et al.[87] propose a novel Forgery-aware
Adaptive Vision Transformer to boost generalization. They adapt ViT to
deepfake detection by adding two specially designed components and keeping the
vanilla ViT frozen to circumvent overfitting issue. The proposed framework is
a dual-branch architecture, the CNN branch with Local-aware Forgery Injector
aims to inject local information into ViT branch, and ViT-branch comprises
vanilla ViT and Global-aware Forgery Adapter targets to learn generalizable
features for deepfake detection. Liu et al.[89] propose the use of a mixture
of low-rank experts to fully harness the potential of CLIP-ViT, thereby
preserving existing knowledge while simultaneously expanding the model’s
capacity for transferable deepfake detection. Kong et al. develop a novel
method, MoE-FFD [88], which enhances face forgery detection by integrating
LoRA and Adapter with the Vision Transformer. This approach effectively
identifies forgery clues on both global and local scales. Additionally, their
innovative Mixture-of-Experts modules boost the model’s capacity and
adaptability, allowing for improved detection performance in a plug-and-play
manner across different transformer architectures.

Model level: Vision-Language Models Researchers are harnessing the
capabilities of pre-trained vision-language models (VLMs) [9], which are
proving instrumental not only in universal deepfake detection, but also in
enhancing reasoning processes.

Universal Detection: A primary focus has been developing methods that utilize
the inherent capabilities of VLMs, such as CLIP [173], to evaluate their
effectiveness across various scenarios. To enhance video content analysis, Han
et al.[91] propose using the CLIP image encoder, paired with a newly designed
side network-based decoder, to effectively extract both spatial and temporal
cues from video clips. Unlike previous methodologies that directly utilized a
frozen CLIP model, they update their CLIP-inspired model parameters by
incorporating designed textual labels into the image augmentation process.

Reasoning: Recent advances have proposed using VLMs to incorporating reasoning
to boost interpretability and context understanding in deepfake detection. For
example, Tsigos et al. [92] propose a novel framework to evaluate how
effectively explanation methods can interpret decisions made by deepfake
detectors. This framework focuses on visual data and measures the accuracy of
explanation methods by their ability to modify key image regions through
adversarial attacks, assessing their impact on the detector’s verdict. The
findings underscore the ability of this method to offer meaningful
explanations based solely on visual input.

Building on the multimodal capabilities, Sun et al. develop Visual-Linguistic
Face Forgery Detection (VLFFD)[93], which employs CLIP to improve face forgery
detection. This approach enriches training data with detailed textual prompts
and fine-grained images, employing a Coarse-and-Fine Co-training (C2F)
framework that leverages both visual and linguistic knowledge. VLFFD improves
forgery identification and provides in-depth interpretative insights into the
locations and characteristics of the forgeries, markedly expressing its
explanatory power. Zhang et al. [94] reformulate the deepfake problem as a
Deepfake Detection Visual Question Answering (DD-VQA) task, which involves
identifying image authenticity and providing explanations. They also create a
new dataset and develop a Vision and Language Transformer-based framework that
effectively detects deepfakes and generates high-quality explanations.

As Multimodal Large Language Models (MLLMs) have shown significant problem-
solving capabilities in various vision fields, recent studies have begun
exploring their application in deepfake detection. A recent study by [95]
introduces the SHIELD benchmark to evaluate the effectiveness of MLLMs in face
forgery detection across both visual and acoustic modalities. Tested under
various conditions, including zero-shot and few-shot scenarios, MLLMs
demonstrate superior capabilities in interpretability and multimodal reasoning
in deepfake dtection. In parallel, Shan et al. [96] have validated the
efficacy of MLLMs through extensive qualitative and quantitative experiments,
highlighting the importance of effective prompt engineering to distinguish
between real and AI-generated images effectively. In a recent work[97] Li et
al. conduct extensive experiments to evaluate the performance of $13$ well-
known language and multimodal models on the task of detecting fake images and
explore how explicit reasoning impacts the accuracy of fake detection with
these models.

Learning Level: Advanced Loss Functions While most deepfake detection methods
traditionally rely on Softmax loss, Yin et al. [98] challenge the
effectiveness of softmax loss for distinguishing subtle distinctions between
real and fake images. They propose a method that utilizes both angular and
Euclidean spaces to enhance class separability. By decoupling the magnitude
and direction of embedding vectors, they introduce margin penalties in both
domains to increase the separation between classes, thereby enhancing inter-
class separability and intra-class compactness. Zhang et al. [99] further the
development of discriminative features using a fractional center loss,
designed to cluster real faces closer within the embedding space while
distinctly separating them from fake ones. In their subsequent research, Zhang
et al. [100] introduce a dual-level center loss approach that incorporates
both patch-level and image-level center losses. This innovative method
effectively reduces intra-class distances while expanding inter-class
distances, significantly enhancing the discriminative capability of feature
embeddings at both the micro (patch) and macro (image) levels.

Learning Level: Disentangled Representation Under the observation that
representations learned by the detector contain both forgery and content
information and thus remain recognizable to content clues like face identity
and background, making the detector prone to overfit training data[101][102],
disentanglement learning aims to decouple the learned features into task-
relevant and task-irrelevant groups, while only the task-relevant group will
be used for subsequent detection.

Liang et al.[101] initially explore a framework to disentangle the forgery-
relevant and content-relevant features and remove content information by only
employing the decoupled forgery feature for detection. The framework is
devised with two encoders for content and forgery feature extraction,
respectively. To ensure the effectiveness of disentanglement, a multi-task
strategy and a contrastive constraint are proposed. The work of UCF[102]
extends the framework by further decoupling the forgery-relevant features into
manipulation-specific and manipulation-agnostic ones, targeting a more refined
approach to feature disentanglement in deepfake analysis and achieving a
better generalization ability across various forgery types. Lin et al.[103]
also adopt this disentanglement framework with an additional focus on
demographic-relevant features to improve the fairness of deepfake detection in
addition to generalization ability.

Unlike the aforementioned works that decompose features from a high-level
semantic perspective with the aid of multi-task strategy, Ba et al.[104]
leverage a disentanglement module to extract multiple non-overlapping local
features with the guidance of carefully derived Local Information Loss. These
local features are subsequently fused and filtered out task-irrelevant
information under the supervision of Global Information Loss.

Learning level: Reconstruction Researchers have innovated methods for
identifying deepfakes by harnessing insights gained during the reconstruction
of facial images. Khalid et al.[105] employ a One-class Variational
Autoencoder (OC-VAE), trained solely on authentic images, to identify
deepfakes by analyzing deviations in reconstruction scores. He et al.[106]
utilize image re-synthesis techniques such as super-resolution and denoising
to extract residual detection cues. Cao et al.[107] combine reconstruction
with metric constraints and classification to distinguish real from forged
images effectively. In MARLIN [108], Cai et al. use self-supervised learning
to reconstruct accurate facial expressions from extensive online video data.
These approaches leverage facial imagery reconstruction as a pivotal element
for detection.

Learning level: Manipulation Localization The objective of manipulation
localization in deepfake detection is to pinpoint altered pixels in fake
samples. This approach not only reveals the specific changes made by attackers
but also aids in discerning their motives, which is valuable for informing
decisions made by authorities[109].

Li et al.[110] develop Face X-ray, a technique based on the assumption that
face manipulations involve a blending process. Their method helps
differentiate real from fake images by identifying the presence and location
of blending boundaries, thereby assessing image authenticity. Dang et al.[111]
emphasize the importance of detecting manipulated face images and precisely
localizing altered areas. They employ an attention mechanism to refine feature
map processing, which significantly enhances the ability to distinguish
between genuine and counterfeit faces. Shuai et al.[112] introduce a model
that effectively localizes forgery areas by utilizing RGB images and Spatial
Rich Mode (SRM) noise residuals. Their method integrates multi-modal and
multi-scale patch features to pinpoint potential forgery areas accurately.
Dong et al.[116] argue that implicit identity leakage deteriorates detection
generalization, and correspondingly propose an effective ID-unaware detection
model, which is guided by authenticity and manipulation localization
information. In [113], Zhang et al. create EditGuard, a system designed to
detect manipulated regions within digital content. EditGuard embeds
imperceptible watermarks that allow for the precise decoding of tampered
areas. Recent works from Hu et al.[114] and Tantaru[115] have also contributed
to this direction, with Tantaru developing a weakly-supervised method that
enhances the localization of manipulated areas in images produced by deep
manipulation techniques.

One significant challenge in manipulation localization is the high cost
associated with annotating manipulated regions, which can be both time-
consuming and expertise-intensive.

Learning level: Identity Discrepancy This approach shifts the focus from
simply identifying if a face is real or fake to analyzing whether the facial
identity extracted from the input maintains the subject’s biometric traits.

Cozzolino et al. [117] develop a system incorporating a facial feature
extractor, a temporal network to identify biometric discrepancies, and a
generative adversarial network that predicts subject-specific motions,
enhancing the detection of inconsistencies in facial features. Nirkin et al.
[118] analyze discrepancies between identity embeddings from the inner(target
person) and outer (source person)facial regions. This comparison helps detect
forgeries by highlighting inconsistencies in identity representations. Dong et
al. [119] introduce the Identity Consistency Transformer, which uses identity
vectors to discern differences between the inner and outer facial areas. Their
model employs a consistency loss mechanism to align similar identities and
differentiate dissimilar ones. Huang et al. [120] leverage implicit identity
features of fake faces with two innovative loss functions: the Explicit
Identity Contrast (EIC) loss, which attracts real samples to their true
identities and repels fake ones, and the Implicit Identity Exploration (IIE)
loss, a margin-based approach that ensures effective differentiation of
genuine and manipulated faces by maintaining appropriate intra-class and
inter-class distances.

This line of methods is believed to offer broad applicability across various
types of manipulation, as it does not depend on the specific methods of
manipulation.

Learning level: Knowledge Distillation Knowledge distillation techniques [174]
have proven effective in enhancing the generalizability of deepfake detection,
applicable both at the latent representation and/or prediction levels.

Woo’s Attention-based Deepfake Detection Distiller (ADD) [121] leverages high
and low-quality forgery data to detect compressed deepfakes effectively. The
innovative frequency attention distiller helps the student model recover lost
high-frequency components, while the correlation distillation method improves
pixel correlation preservation using the Sliced Wasserstein Distance metric
[122].

Addressing scenarios without access to source domain data, Kim et al.
introduce the Feature Representation Transfer Adaptation Learning (FReTAL)
[123]. This method uses knowledge distillation to adapt detectors to new
deepfake types efficiently, focusing on latent representations and prediction
logits, assuming label availability for new forgeries. Similarly, the CoReD
framework addresses sequential domain adaptation challenges by minimizing
catastrophic forgetting within a teacher-student model setting [124]. This
process ensures the knowledge preservation in both latent representation and
prediction levels.

Furthermore, to improve deepfake detection across domains without needing
label access for target samples, Lv et al.’s study introduces a bidirectional
adaptation strategy[125]. Forward adaptation transfers detection capabilities
to new forgeries via supervised training and adversarial feature adaptation,
while backward adaptation uses self-distillation to further refine and
maintain feature recognition and knowledge.

Learning level: Fine-grained Prediction Recent advancements highlight the
importance of granularity and adaptability in facial forgery detection,
refining the detection process to keep pace with the evolving complexity and
variety of deepfake methods.

Guo et al.[126] approach the classification of forged images by representing
the forgery attributes with multiple hierarchical labels. Their method
classifies forgery from a broad category (fully-synthesized vs. partial-
manipulated) down to specific forgery techniques. This hierarchical
interpretation aids in better understanding the nuances of the forged image
and enhances the separation of real and fake regions. By predicting the fine-
grained and hierarchical path, their approach provides a more detailed
analysis of the attributes that define forged images.

Guarnera et al. [127] take a similar hierarchical and multi-level approach to
detect and recognize deepfakes. Their method begins by classifying images as
real or AI-generated, then identifies the source of synthetic images—whether
from GANs or DMs—and further pinpoints the specific architecture used.

Additionally, the controllable guide-space (GS) method introduced by Guo in
[128] enhances feature discrimination across forgery domains. This method
employs a well-designed guide-space that effectively segregates forgery
domains and accentuates the distinction between real and forged domains in a
finely controlled manner, improving the generalization of forgery features.

##### 3.1.1.3 Detection for Diffusion Model

As diffusion models continue to evolve, their potential misuse for creating
counterfeit content has raised significant concerns. The distinctness of their
generation process necessitates specialized detectors to discern diffusion-
generated images. Given that detection for diffusion models is in its nascent
stages, most works are working on images from general domain, of which a
subset contains facial forgery images. Nevertheless, these proposed frameworks
can be applied to face-centric synthetic images without loss of generality.

The studies in [129] and [130] analyze outputs from diffusion models and
concluded that, despite significant improvements in the quality of generated
results, there are still identifiable clues that can distinguish these from
real samples. Ricker et al. [129] extensively test whether current detection
methods designed for GANs generated samples can identify images created by
diffusion models. They find that while these methods initially struggled to
tell DM-generated images apart from real ones, retraining them on DM images
greatly improved their accuracy, even allowing them to detect images made by
GANs. Corvi et al. [130] note that models like GLIDE and Stable Diffusion
produce distinct spectral peaks, which can be effectively utilized to identify
synthetic images. This spectral approach adds another dimension to the
detection methodologies, harnessing unique signatures inherent in diffusion
models.

Lorenz et al. [131] introduce the use of the lightweight multi Local Intrinsic
Dimensionality (multiLID) technique, initially developed for detecting
adversarial examples, to automatically detect synthetic images and identify
the generator networks responsible for their creation. Following a different
approach, Cheng et al. [34] present the Edge Graph Regularization (EGR)
method, crafted to enhance the discriminative capabilities of detectors. This
technique integrates edge graphs, derived via Sobel operators, as a
regularization term into the empirical risk formulation. By doing so, it
encourages the model to concurrently analyze the features of both the original
images and the edge graphs, thereby refining its overall analytical precision.

Transitioning from intrinsic analysis to reconstruction accuracy, Wang et al.
[132] introduce Diffusion Reconstruction Error (DIRE) that measures the
discrepancy between an input image and its reconstructed version using a pre-
trained diffusion model. Their method reveals that while images generated by
diffusion models can be accurately reconstructed by the same type of models,
real images do not replicate with the same fidelity. Expanding on the concept
of reconstruction, Ma et al.[133] introduce the SeDID method, addressing the
limitations of DIRE[132] by utilizing the intermediate steps in diffusion
models. Unlike DIRE only focusing on the initial time-step information, SeDID
captures critical information during the diffusion and reverse diffusion
processes, potentially enhancing the accuracy of synthetic image detection.
Building on this, Luo et al. [134] capitalize on conducting reconstructions
within the latent space, enhancing efficiency in two key ways: it avoids the
need for complete image inversion and reconstruction, and it reduces the risk
of error accumulation associated with multiple inversion and reconstruction
steps.

####  3.1.2 Proactive Detection Methods

Proactive detection methods for deepfakes utilize strategies that preemptively
counteract deepfake manipulation before it occurs or facilitate the
identification of manipulated content afterward. Unlike passive methods, which
assess content post-creation, proactive approaches embed specific signals,
such as adversarial perturbations, into the original content. These signals
aid in verifying authenticity and integrity if the content is later altered.
For effectiveness, these alterations visually resemble the original to evade
detection while impairing the functionality of deepfake generation models. In
the following subsections, we explore recent proactive detection efforts
targeting deepfakes generated by both GANs and DMs.

##### 3.1.2.1 Proactive Methods for GANs

Researchers have explored using adversarial attack techniques to inject
perturbations into the original input, enhancing the resilience of images
against unauthorized manipulations. Ruiz et al. [135] develop an approach
using versatile adversarial attacks to prevent unauthorized facial
modifications, improving the resilience of image translation systems through
adversarial training for GANs. Similarly, Yeh et al. [136] use adversarial
loss to increase the resistance of images to deepfake manipulations.

To further improve the resilience of perturbations, Huang et al. [137]
introduce a poison perturbation generator that improves the effectiveness of
perturbations using an alternating training strategy. Wang et al. [138] focus
on improving the resilience of perturbations by targeting the Lab color space,
particularly the decorrelated a and b channels. Expanding defensive
strategies, Anejaet al. [139] develop a data-driven method that quickly
produces tailored perturbations to redirect manipulation attempts toward a
predefined outcome. Huang et al. [140] design the Cross-Model Universal
Adversarial Watermark (CMUA-Watermark), broadening defense across various
facial images and deepfake models. Asnani et al. [141] and [109] propose using
real image templates to improve the detection accuracy of manipulation,
helping differentiate between real and manipulated images.

Recent innovations have suggested conducting source tracing alongside
injecting adversarial perturbations into the original input to enhance
traceability against manipulations. Innovations by Wang et al. [142] and Wu et
al. [143] underscore the importance of tracing original source images to boost
detection system credibility. Yu et al. [14] enhance traceability by embedding
traceable fingerprints directly into the generator’s parameters, linking
outputs uniquely to their creators. Addressing gaps in proactive defenses,
Tang et al. [144] introduce the Feature-Output ensemble UNiversal Disruptor
(FOUND), which disrupts in the latent space and employs a gradient-ensemble
algorithm to provide a universal attack pattern against multiple deepfake
models, using minimal original-attribute images. In [145], Zhang et al.
introduce Dual Defense, which embeds an invisible watermark in target faces to
disrupt malicious face swapping while preserving traceability. Their approach
utilizes a watermark embedding network that balances invisibility,
adversariality, and traceability through perceptual adversarial encoding
strategies. In a recent study, Wu et al. [146] demonstrate that traditional
watermarking methods, when applied to AI-generated Deepfake content, can
interfere with forgery detection. To address this issue, they propose
’AdvMark’, a plug-and-play solution that transforms robust watermarking
systems into adversarial frameworks, retaining the ability to extract
watermarks for provenance tracking.

##### 3.1.2.2 Proactive Methods for DMs

Recent pioneering strategies in proactive deepfake detection for diffusion
models, not specifically designed for facial forgery, include Cui et al.’s
DiffusionShield [15], which enhances watermark uniformity, Fernandez et al.’s
method [147] for embedding invisible watermarks in Latent Diffusion Models,
and Wang et al.’s behavioral modification approach [148] to ensure
traceability of protected content. These innovative methods establish
foundational practices for applications in facial forgery detection.

Expanding the scope, Zhao et al. [149] examine the effectiveness of
watermarking within diffusion models, with Cao et al. [150] extending
proactive security measures to audio models. To enhance model transferability,
Shim et al. [151] disrupt the latent encoding phase of deepfake generation
using LEAT, which employs a Normalized Gradient Ensemble strategy to optimize
disruptions across various models.

###  3.2 Multi-modal Deepfake Detection

The concurrence of fake content in multiple modalities leads to more
convincing deepfakes compared to those visually manipulated ones. In such
context, it becomes imperative to develop the capability to detect these
intricate multi-modal deepfakes. In this subsection, we provide a
comprehensive investigation of the more intricate and challenging multi-modal
scenario in this subsection, where auditory or textual content can also be
falsified together with the presence of visual manipulation.

####  3.2.1 Audio-Visual

Audio-visual deepfakes involve the manipulation of either or both the audio
and visual streams in a video. The combination of manipulations on these two
modalities potentially raises more serious ethical and societal concerns,
e.g., manipulating politicians to make false claims convincingly by delivering
a lip-synced video with fake audio.

The heterogeneous nature and the modality gap between audio and video make
multi-modal audio-visual deepfake detection a more challenging task. Existing
explorations on this task commonly design two-stream framework to incorporate
multi-modal inputs. These efforts can be categorized into three main
paradigms: independent learning, joint learning and matching-based learning.

(a) Independent learning for audio-visual deepfake detection.

(b) Joint learning for audio-visual deepfake detection.

(c) Matching-based learning for audio-visual deepfake detection.

Figure 4: Three main paradigms for multi-modal audio-visual deepfake
detection.

Independent Learning The independent learning paradigm is a straightforward
approach in audio-visual deepfake detection, where each modality is processed
separately. This method then combines the results from each stream for a final
decision [10][11][152][153].

As a baseline method presented in [10], two independent streams of CNN are
proposed for audio and visual deepfake detection separately, and the final
multi-modal detection result is obtained by combining the audio and visual
predictions. In a preliminary attempt to evaluate the FakeAVCeleb[5]
benchmark, Khalid et al.[11] ensemble predictions from independently learned
uni-modal detectors. Furthermore, Hashmi et al.[152] devise an audio-visual
network besides uni-modal networks and adopt ensemble learning to make the
final decision based on predictions from audio, video, and audio-visual
networks. Similarly, AVFakeNet[153] leverages Swin-Transformer [175] as
backbones for both audio and visual branches for feature extraction and then
integrates the result from both branches for the multi-modal prediction.

Despite its intuitive nature, the paradigm of independent learning might
overlook the intrinsic relationships between audio and visual data, which can
be critical in accurately detecting deepfakes.

Joint Learning This approach integrates audio and visual modalities to
leverage multi-modal information for enhanced deepfake detection. The general
network architecture of the joint learning paradigm encompasses two modal-
specific branches taking as input corresponding modality content for feature
extraction, i.e., video frames and audio waveform, and a fusion module to
integrate features of different modalities for audio-visual detection. This
paradigm mines the inter-modal relationship by feature fusion to enhance the
multi-modal deepfake detection. Based on at which stage the modal-specific
feature are fused together, joint learning-based detection algorithms can be
further divided into intermediate and late fusion manners. Moreover, some
existing late fusion-based audio-visual detectors leverage multi-task strategy
or impose regularization to facilitate the learned multi-modal
representations.

Intermediate Fusion: This method holds the assumption that simply combining
the high-level abstract features extracted from both streams are insufficient
to mine the inter-modal information. Zhou et al.[10] initially propose the
joint audio-visual deepfake detection, handling cases in which either or both
visual and auditory modalities have been manipulated. In their method, a sync-
stream is designed to leverage the synchronization patterns in addition to
separate branches for each modality, which fuses the intermediate features
from distinct modalities by cross-attention mechanism. Yang et al.[42] employ
a Transformer architecture, where audio and visual inputs go through a
temporal-spatial encoder, and corresponding features of both modalities
interact with each other via a decoder with bidirectional cross-attention
module. Wang et al.[154] incorporate inspiration from how the biological
system perceive the multi-modal environment, proposing visual and audio
branches as different sensory systems, of which the obtained signals are then
integrated by audio-visual attention mechanism acting like human multi-sensory
neurons. Specifically, the intermediate audio and visual features attend to
each other at all stages and are concatenated at the last few stages for
further multi-modal detection.

Late Fusion: In this strategy, the audio and visual representations are only
merged in the final stages of feature extraction. Techniques such as
concatenation or addition are commonly used to merge modalities in late fusion
[11][152][155][156][157][35][13]. Various advanced strategies, such as
attention mechanism[158][159] and MLP mixer layer[12] are applied to enhance
effective fusion.

While benchmarking the FakeAVCeleb[5] dataset, Khalid et al. [11] adapt multi-
modal models from different tasks to audio-visual deepfake detection, in which
the modal-specific features are integrated by concatenation. The audio-visual
network presented in [152] uses a visual stream and an audio stream in
parallel, taking as input the extracted video frames and mel-spectrogram of
audio respectively, and the feature maps from two streams are then
concatenated. DF-TransFusion[155] further explores this paradigm by using two
branches to extract facial features by using self-attention for facial feature
extraction and cross-attention for lip-audio synchronization, with a linear
layer integrating the concatenated signals for final prediction.

Further, Liu et al. [157] propose a tri-branch framework to handle audio,
video, and multi-scale frame inputs, where the extracted features are weighted
based on inter-modal interactions and concatenated. Moreover, this method also
designs Forgery Clues Magnification Transformer for feature extraction of each
branch, where the tokens in the input sequence interact with each other to
obtain the weighted sequence to magnify the intra-modal inconsistency, and
Distribution Difference based Inconsistency Computing module to align
information of audio and video modalities.

Dynamic weighted addition has proven effective in fusion strategies for multi-
modal tasks. While addressing the temporal audio-visual deepfake localization
task, Cai et al.[35] calculate the weighted sum of uni-modal boundary maps
from corresponding branches as the fused boundary map, where the weights are
dependent on uni-modal boundary maps themselves. Similarly, Wang et al.[13]
leverage dynamic weighted averaging as late fusion approach in the proposed
AVT2-DWF framework. Using a two-stream transformer architecture for uni-modal
feature extraction, the fused information for multi-modal detection is
obtained by weighted averaging the classification tokens in both uni-modal
features. Notably, the weights AVT2-DWF are calculated by inter-modal
attention mechanism.

More sophisticated than simple concatenation or addition, attention mechanisms
and MLP mixer layers offer advanced alternatives for multi-modal feature
fusion. Sree et al.[159] employ a multi-head self-attention module after
concatenating features of different modalities. The attention mechanism
further enhances the inter-modal interaction and benefits subsequent multi-
modal detection. More than attention mechanism, Zou et al.[158] feed the
concatenated audio-visual features into Transformer layers to fuse information
from both modalities. Alternatively, Raza et al.[12] employ an MLP mixer layer
that not only enhances intra-modal features but also facilitates the
integration of inter-modal information by passing concatenated modal-specific
features through an inter-modality MLP mixer layer.

Multi-task: While both intermediate and late fusion-based approaches focus on
using fused information solely for multi-modal detection of audio-video pairs,
there is a concern that they neglect the fine-grained differences between
various types of audio-visual deepfakes. For example, scenarios where the
video is real but the audio is fake, or vice versa, can confound feature
extraction processes within each modality-specific branch[156]. In response,
some works exploit a multi-task learning strategy to address the problem by
simultaneously performing uni-modal and multi-modal detection[156, 10, 42, 12,
160].

The networks proposed in [10, 42] are tasked with three binary classification
objectives: uni-modal audio and visual deepfake detection based on learned
representations from corresponding branches and multi-modal detection based on
the fused signals. Besides simultaneously introducing uni-modal detection
tasks, Muppalla et al.[156] make the multi-modal detection more fine-grained,
changing the original binary classification to a four-class one. Similarly,
Raza et al.[12] reformulate multi-modal audio-visual deepfake detection as a
multi-label classification task. They modify the classification head to
incorporate fused signals and simultaneously perform classification tasks for
both modalities. The multi-task strategy employed in these studies requires
more fine-grained labels of each modality to provide supervision during
training.

Unlike those incorporating fine-grained uni-modal detection tasks, Yu et
al.[160] design an auxiliary task from a disparate perspective. Critiquing the
reliance solely on audio-visual synchronization as insufficient for
generalizable audio-visual deepfake detection, the paper emphasizes learning
content-based cross-modal correlations. In this work, Yu et al. introduce a
correlation distillation task, where well-trained audio and visual speech
recognition models serve as teachers to obtain content-based cross-modal
correlation, and the student detection network learns to model cross-modal
correlation from teachers. This explicitly learned cross-modal correlation
based on content helps to prevent overfitting.

Enhanced Representation: This line of work aims to learn more discriminative
representations for effective detection during training. To this end, various
strategies including regularization[42][157][158] and disentanglement[159] are
employed.

Yang et al.[42] apply a contrastive loss and an additive angular margin loss
to make the fused multi-modal features of manipulated and genuine samples more
discriminative. Likewise, Zou et al.[158] also impose the contrastive
constraint in cross-modal cases, thus the audio and visual features become
more similar when they share common authenticity, i.e., both streams are real.
Besides, Zou et al.[158] further introduce within-modal regularization to
align uni-modal features with their respective targets, ensuring the uni-modal
features are discriminative for corresponding uni-modal detection. The
Distribution Difference based Inconsistency Computing module in [157] also
leverages contrastive loss to impose regularization on the relationship
between audio and video features based on Jensen-Shannon divergence.

In addition, Mis-AvoiDD[159] enhances representations for effective multi-
modal deepfake detection by decoupling them into modal-invariant and modal-
specific subspaces. Specifically, a modal-invariant encoder and two modal-
specific encoders are devised to extract common and specific features from
each modality. In modal-invariant subspace, the discrepancy between
representations of different modalities is minimized to align modal-specific
features. Furthermore, an orthogonal loss imposes constraints on the
relationship between the modal-invariant and modal-specific features of the
same modality as well as the relationship between the modal-specific features
from different modalities. The orthogonal regularization ensures the
distinctness between representation subspaces.

Matching-based Learning Distinct from fusion-based learning, matching-based
learning leverages the natural synchronization found in genuine audio-visual
content, which is typically absent in synthetic pairs[162, 161]. This approach
focuses on matching audio and visual signals by evaluating the consistency
between extracted features from each modality, rather than using traditional
classification techniques. In this paradigm, networks employ modality-specific
feature extractors and are trained on authentic content via contrastive
learning, enhancing their generalization capabilities across various types of
manipulation.

Cheng et al.[162] propose Voice-Face matching detection based on the
observation verified in human perception and neurology field, which reveals
the homogeneity between the face and voice of one person. The detection
network involves only voice and face feature extractors, without fusion module
and classification head. The predicted genuineness of the audio-visual pair is
determined by the consistency of features from these two modalities. While
training the feature extractors, contrastive InfoNCE[176] loss is applied to
increase the similarity between matched voice and face feature pairs.

Tian et al.[161] adopt a matching-based Intra-Cross-Consistency method for
multi-modal detection, where a matching score from the detection model is
compared with a threshold to obtain detection results. Similar to [162], this
approach also evaluates cross-consistency between audio and visual features to
compute the matching score. The method addresses the trade-off observed in
deepfake generation between temporal identity consistency and the
synchronization of face motion with audio-visual elements by incorporating
temporal identity intra-consistency. This method merely required self-
supervised learning on real videos by a contrastive learning strategy, thus
becoming generalizable.

Others There are several existing works on audio-visual deepfake detection
leveraging disparate approaches and frameworks from the aforementioned
paradigms.

Feng et al[165] handle the task from a novel perspective of anomaly detection,
requiring training merely on real videos in a self-supervised manner. At the
first stage of this method, an audio-visual network is trained to estimate the
inter-modal synchronization score, which is used to extract heuristically
designed features. Subsequently, an auto-regressive model is then proposed to
learn the distribution of previously extracted feature sets. While performing
detection, the samples with low likelihood estimated by the auto-regressive
model are flagged as fake.

POI-Forensics[163] focuses on identity consistency instead of cross-modal
inconsistency. Cozzolinno et al.[163] argue that any manipulation inevitably
perturbs characteristics of the portrayed identity. Therefore, the authors
utilize the indirect clue that the individual in the video is not consistent
with the person who is claimed to be. While performing detection, this method
requires a reference set of at least 10 videos of the person of interest, and
the similarity index between the target video and reference videos would be
used to determine prediction.

Haq et al.[164] introduce novel biological signal emotion psychology to
improve deepfake detection. Based on the emotion extracted from each modality,
the proposed neurosymbolic framework performs intra-modal emotion transition
analysis and inter-modal reasoning. The former aims to explore the temporal
emotion inconsistency, and the latter explores the cross-modal emotion
consistency according to psychological theory. Notably, the proposed framework
does not require training on deepfake datasets owing to the underlying
psychology and emotion knowledge, leading to better generalizability.

Yu et al.[166] detect audio-visual deepfakes based on two augmented views of
visual signals, proposing a two-stage framework PVASS-MDD to better capture
the audio-visual inconsistency. The framework encompasses an auxiliary
Predictive Visual-audio Alignment Self-supervision (PVASS) stage as well as
the Multi-modal Deepfake Detection (MDD) stage. In the PVASS stage, the visual
signals are augmented to two different views, each of which interacts with
audio information via the Predictive VA Alignment module to obtain further
corresponding VA representations, the self-supervised learning objective is to
make the two VA representations similar. In the MDD stage, the two VA
representations corresponding to two augmented visual views are concatenated
to be fed into a classifier for multi-modal prediction. Besides cross-entropy
loss, regularization terms including a contrastive loss and an auxiliary loss
are additionally applied.

####  3.2.2 Text-Visual

Recent advances in text-visual deepfake detection have introduced
sophisticated methods to address manipulations on text and visual modalities.
Figure 5 illustrates the framework for this multi-modal detection and
grounding task.

Figure 5: HAMMER[6] for text-visual deepfake detection and grounding. Figure
from [6]

Shao et al. [167] introduce the DGM4 dataset, featuring samples manipulated
through distinct image and text techniques. They propose the Hierarchical
Multi-modal Manipulation Reasoning Transformer (HAMMER), which enhances cross-
modal semantic alignment with a Manipulation-Aware Contrastive Loss,
demonstrating its effectiveness on this comprehensive dataset.

Liu et al. [168] incorporate frequency domain analysis into RGB image
evaluations using discrete wavelet transforms to capture forgery artifacts
more effectively. Their approach integrates image and frequency data through a
specialized encoder and a unified decoder, simplifying the optimization
process while improving forgery detection.

Wang et al. [169] focus on modality-specific features for detecting and
grounding multi-modal manipulations. To simultaneously preserve multi-modal
alignment capability and exploit modality-specific features, the framework is
devised with dual-branch cross-attention for bi-directional interaction
between modalities and decoupled fine-grained classifiers, which perform
modality-specific fine-grained manipulation classification task based on
features of corresponding modality instead of leveraging an integrated token
for both modalities in HAMMER[167]. Furthermore, an implicit manipulation
query for each modality is proposed to facilitate reasoning for grounding
using learnable queries.

###  3.3 Trustworthy Deepfake

Despite considerable advances in deepfake detection, from single-modal to
multi-modal approaches, concerns persist regarding the robustness and
trustworthiness of these detectors. For example, a recent study by Abdullah et
al. [177] highlights significant vulnerabilities in current detection methods.
The study indicates that attackers can easily tailor generative models to
circumvent traditional defenses. Moreover, the misuse of widely deployed
vision foundation models allows for the creation of adversarial deepfakes
through subtle manipulations of image semantics, thereby undermining the
effectiveness of established detection techniques. In the following
subsections, we will explore recent developments in deepfake detection,
focusing on adversarial attacks, backdoor attacks, and defensive strategies.

Adversarial Attack Adversarial attacks in deepfake detection aim to deceive
manipulation detectors by subtly altering original images across various
domains—pixel space, frequency domain, and latent space. Carlini et al. [178]
highlight how adversarial strategies can significantly lower the accuracy of
deepfake detectors. Supporting studies [179, 180, 181] confirm these
vulnerabilities, emphasizing the need for robust detection mechanisms.

In the pixel domain, Neekhara et al. [182] explore how adversarial attacks
target specific facial features, while Liao et al. [183] develop the Multi-
Layers Key Semantic Region Selection method to target essential pixels. Zhu et
al. [184] and Liu et al. [185] introduce less noticeable perturbations using
frequency and spatial features to streamline the application across various
images. In [186], Huang et al. introduce DeepNotch, a method to create
realistic and undetectable DeepFakes by reducing artifact patterns without
compromising image quality, using implicit spatial-domain notch filtering. For
frequency-based attacks, Jia et al. [187] and Chen et al. [188] focus on
disrupting frequency domain properties to fool detection systems, utilizing
cross-task perturbed initialization to enhance attack efficacy. In [189], Wu
et al. introduce TraceEvader, a novel evasion attack that manipulates high-
frequency traces and introduces adversarial blur into low-frequency
components. In latent space manipulations, Meng et al. [190], Shamshad et al.
[191], and Li et al. [192] leverage the latent spaces of generative models to
enhance the stealth and effectiveness of their attacks, merging semantic
manipulations with adversarial perturbations to bypass forensic classifiers.

In a departure from traditional 2D adversarial attacks, Shahriyar’s studies
[193] and the innovative ”AdvHeat” by Wang et al. [194] explore new dimensions
such as video content dynamics and 3D adversarial face views.

Backdoor Attack Unlike adversarial attacks that manipulate input at runtime,
backdoor attacks embed corrupted samples in training datasets. These samples
appear legitimate, but contain Trojans that, when triggered, manipulate the
model’s output to evade detection. Triggers can be placed in the pixel space
[195, 196] or latent space [197], and can be selected from being manually
specified [195] to stealthy [197, 196].

Sun et al. [195] propose to craft effective triggers by exploiting the
vulnerabilities of deepfake detectors and select impactful samples with a
Filtering-and-Updating Strategy. Han et al. [197] use an ”analysis-by-
synthesis” approach to embed natural triggers in the latent space, involving
Model Discrimination and Data Distribution to create independent poisoned
samples.

Liang et al. [196] focus on embedding inconspicuous triggers, limiting
perturbations to critical facial areas and employing a scalable trigger
generator that creates transformation-sensitive patterns. This method enhances
the stealth of triggers and addresses their sensitivity to image
transformations, significantly improving the subtlety of backdoor attacks.

Discrepancy Minimization This approach focuses on reducing the detectable
differences between genuine and manipulated images across various data
representation domains, including pixel space, frequency space, and feature
space.

In pixel space, Huang et al. [198] develop FakePolisher, which reduces
artifacts by projecting images into a low-dimensional subspace. Their
subsequent FakeRetouch [199] enhances image realism through strategic noise
addition and image filtering. Similarly, Li et al. [200] and He et al. [201]
improve facial data reconstruction by searching adversarial latent codes.

In the frequency domain, Dong et al. [202] balance spectral power
distributions to make discrepancies less noticeable, while Lee et al. [203]
introduce Spectrum Translation for refining image generation spectra.

For feature space discrepancies, Hou et al. [204] minimize differences by
applying natural degradations in an adversarial manner. Ivanovska et al. [205]
employ Denoising Diffusion Models to challenge detection systems, focusing on
reconstructing images with specific denoising steps for robust testing.
Additionally, Liu et al.’s TR-Net [206] efficiently removes intrinsic deepfake
traces through coordinated generators and multiple discriminators, enhancing
the method’s effectiveness across various detection systems.

Defense Strategies To enhance the resilience of deepfake detection systems
against increasingly sophisticated attacks, researchers have developed robust
defense strategies. Khan et al. introduce Adversarial Feature Similarity
Learning (AFSL) [207], which improves detection robustness by optimizing
feature similarity and implementing strategic regularizations. Additionally,
Pinhasov et al. utilize Explainable Artificial Intelligence (XAI) to create
interpretability maps, aiding in the robust classification of adversarial
attacks [208]. The approach proposed in [209] improves the robustness of
deepfake defenses through a destruction-aware constraint during training.
[210] introduces FaceGuard that enhances defense against unseen adversarial
attacks by generating various perturbations that evade detection, while
concurrently learns to identify and remove these perturbations from images.
Ensemble methods have also been prominent, with multiple researchers employing
these strategies to capitalize on the diverse strengths of combined models,
significantly enhancing detection accuracy and robustness against adversarial
perturbations [211, 212, 213, 214]. These methods underscore a critical shift
towards integrating advanced computational techniques to safeguard against the
evolving landscape of digital forgery.

##  4 Challenges and Future Directions

As discussed in the last section, deepfake detection has achieved significant
progress in recent years. However, various challenges persist due to the
complexity of deepfake detection. In this section, we address these challenges
and present potential avenues for future research.

Performance Generalization Across Scenarios Current deepfake detection methods
show limited effectiveness in generalized deepfake detection, including across
different datasets, manipulation methods, and post-processing techniques. The
variations across these factors can result in significant statistical
distribution differences between the training and testing sets. As pointed out
in previous research on domain adaptation [215, 216] and domain generalization
[217], ignoring these differences may lead to a dramatic performance drop when
applying deepfake detectors in practical scenarios. Additionally, adversarial
attacks pose another challenge on the robustness of models. Considering these
challenges and the need for practical applicability, developing specific
methods for generalized and robust deepfake detection becomes a promising and
important direction.

Alignment Among Multi-modalities The transition from single-modal to multi-
modal deepfake detection marks a critical shift in the field. Traditional
detection techniques, which primarily focus on single modalities like video or
audio, are increasingly inadequate as deepfake generation technologies evolve
to incorporate multiple modalities, such as audio-visual or text-visual
manipulations. This complexity requires advanced detection strategies that can
handle the integration of various modalities. Recent research underscores the
potential of exploiting correlations between different modalities to enhance
detection accuracy. Future efforts should aim to further develop multi-modal
detection frameworks, emphasizing improved synchronization and a deeper
understanding of inter-modal dynamics.

Real-Time Analysis Deepfake detection, especially multi-modal detection,
involves manipulation across different modalities and often occurs in the
format of video/audio streams. In such cases, analysis and prediction can only
be made after processing all streams from various modalities, which can be
time-consuming. However, the potential risk introduced by the spread of
deepfakes requires a real-time analysis and response [218]. Current methods
focus more on prediction accuracy than efficiency, highlighting the need for
further innovation and additional efforts to achieve real-time detection.

Expanding Deepfake Detection with LLMs A recent study [17] has explored using
large language models (LLMs) for generation and detection. We believe that
integrating LLMs into deepfake detection can be a promising direction,
leveraging their powerful capabilities in detection and explainability. For
example, LLMs can analyze subtleties in the text modality that may accompany
deepfake videos, such as inconsistencies between spoken words and lip
movements. Exploiting these discrepancies can enhance the analysis.
Furthermore, LLMs can automatically generate explanations about detected
deepfakes, detailing the reasons for the classification and providing
evidence, which is crucial for trust and verification purposes.

##  5 Conclusion

This paper provides a comprehensive and up-to-date overview of deepfake
detection, summarizing methods accepted by top-tier conferences and journals
over the past three years. We first formally define the deepfake detection
problem within our scope, followed by detailed datasets and evaluation metrics
for deepfake detection, covering various modalities and generation models.
Next, we systematically explore the mainstream deepfake detection methods and
categorize them based on their characteristics. We believe that our careful
categorization will help researchers quickly grasp the development trends in
this field. Finally, we delve into the challenges and potential future
directions in facial deepfake detection.

## References

  * [1] D. P. Kingma, M. Welling _et al._ , “An introduction to variational autoencoders,” _Foundations and Trends® in Machine Learning_ , 2019.
  * [2] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _NeurIPS_ , 2014.
  * [3] F.-A. Croitoru, V. Hondru, R. T. Ionescu, and M. Shah, “Diffusion models in vision: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * [4] G. Pei, J. Zhang, M. Hu, G. Zhai, C. Wang, Z. Zhang, J. Yang, C. Shen, and D. Tao, “Deepfake generation and detection: A benchmark and survey,” _arXiv preprint arXiv:2403.17881_ , 2024.
  * [5] H. Khalid, S. Tariq, M. Kim, and S. S. Woo, “Fakeavceleb: A novel audio-video multimodal deepfake dataset,” in _NeurIPS Track on Datasets and Benchmarks_ , 2021.
  * [6] R. Shao, T. Wu, and Z. Liu, “Detecting and grounding multi-modal media manipulation,” in _CVPR_ , 2023.
  * [7] Z. Cai, S. Ghosh, A. P. Adatia, M. Hayat, A. Dhall, and K. Stefanov, “Av-deepfake1m: A large-scale llm-driven audio-visual deepfake dataset,” _arXiv preprint arXiv:2311.15308_ , 2023.
  * [8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _ICLR_ , 2021.
  * [9] J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2024.
  * [10] Y. Zhou and S.-N. Lim, “Joint audio-visual deepfake detection,” in _ICCV_ , 2021.
  * [11] H. Khalid, M. Kim, S. Tariq, and S. S. Woo, “Evaluation of an audio-video multimodal deepfake dataset using unimodal and multimodal detectors,” in _Proceedings of the 1st workshop on synthetic multimedia-audiovisual deepfake generation and detection_ , 2021.
  * [12] M. A. Raza and K. M. Malik, “Multimodaltrace: Deepfake detection using audiovisual representation learning,” in _CVPRW_ , 2023.
  * [13] R. Wang, D. Ye, L. Tang, Y. Zhang, and J. Deng, “Avt2-dwf: Improving deepfake detection with audio-visual fusion and dynamic weighting strategies,” _arXiv preprint arXiv:2403.14974_ , 2024.
  * [14] N. Yu, V. Skripniuk, D. Chen, L. Davis, and M. Fritz, “Responsible disclosure of generative models using scalable fingerprinting,” in _ICLR_ , 2022.
  * [15] Y. Cui, J. Ren, H. Xu, P. He, H. Liu, L. Sun, and J. Tang, “Diffusionshield: A watermark for copyright protection against generative diffusion models,” in _NeurIPSW_ , 2024.
  * [16] F. Juefei-Xu, R. Wang, Y. Huang, Q. Guo, L. Ma, and Y. Liu, “Countering malicious deepfakes: Survey, battleground, and horizon,” _International Journal of Computer Vision_ , 2022.
  * [17] L. Lin, N. Gupta, Y. Zhang, H. Ren, C.-H. Liu, F. Ding, X. Wang, X. Li, L. Verdoliva, and S. Hu, “Detecting multimedia generated by large ai models: A survey,” _arXiv preprint arXiv:2402.00045_ , 2024.
  * [18] X. Wang, H. Guo, S. Hu, M.-C. Chang, and S. Lyu, “Gan-generated faces detection: A survey and new perspectives,” in _ECAI_ , 2023.
  * [19] T. Wang, X. Liao, K. P. Chow, X. Lin, and Y. Wang, “Deepfake detection: A comprehensive study from the reliability perspective,” _arXiv preprint arXiv:2211.10881_ , 2022.
  * [20] M. Akhtar, M. Schlichtkrull, Z. Guo, O. Cocarascu, E. Simperl, and A. Vlachos, “Multimodal automated fact-checking: A survey,” in _EMNLP_ , 2023.
  * [21] A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, “Faceforensics++: Learning to detect manipulated facial images,” in _CVPR_ , 2019.
  * [22] “Contributing data to deepfake detection research,” 2019. [Online]. Available: https://blog.research.google/2019/09/contributing-data-to-deepfake-detection.html
  * [23] L. Li, J. Bao, H. Yang, D. Chen, and F. Wen, “Advancing high fidelity identity swapping for forgery detection,” in _CVPR_ , 2020.
  * [24] B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang, and C. C. Ferrer, “The deepfake detection challenge (dfdc) dataset,” _arXiv preprint arXiv:2006.07397_ , 2020.
  * [25] Y. Li, X. Yang, P. Sun, H. Qi, and S. Lyu, “Celeb-df: A large-scale challenging dataset for deepfake forensics,” in _CVPR_ , 2020.
  * [26] L. Jiang, R. Li, W. Wu, C. Qian, and C. C. Loy, “Deeperforensics-1.0: A large-scale dataset for real-world face forgery detection,” in _CVPR_ , 2020.
  * [27] B. Zi, M. Chang, J. Chen, X. Ma, and Y.-G. Jiang, “Wilddeepfake: A challenging real-world dataset for deepfake detection,” in _ACM MM_ , 2020.
  * [28] P. Kwon, J. You, G. Nam, S. Park, and G. Chae, “Kodf: A large-scale korean deepfake detection dataset,” in _ICCV_ , 2021.
  * [29] T. Zhou, W. Wang, Z. Liang, and J. Shen, “Face forensics in the wild,” in _CVPR_ , 2021.
  * [30] Y. He, B. Gan, S. Chen, Y. Zhou, G. Yin, L. Song, L. Sheng, J. Shao, and Z. Liu, “Forgerynet: A versatile benchmark for comprehensive forgery analysis,” in _CVPR_ , 2021.
  * [31] K. Narayan, H. Agarwal, K. Thakral, S. Mittal, M. Vatsa, and R. Singh, “Df-platter: multi-face heterogeneous deepfake dataset,” in _CVPR_ , 2023.
  * [32] H. Song, S. Huang, Y. Dong, and W.-W. Tu, “Robustness and generalizability of deepfake detection: A study with diffusion models,” _arXiv preprint arXiv:2309.02218_ , 2023.
  * [33] Z. Chen, K. Sun, Z. Zhou, X. Lin, X. Sun, L. Cao, and R. Ji, “Diffusionface: Towards a comprehensive dataset for diffusion-based face forgery analysis,” _arXiv preprint arXiv:2403.18471_ , 2024.
  * [34] H. Cheng, Y. Guo, T. Wang, L. Nie, and M. Kankanhalli, “Diffusion facial forgery detection,” _arXiv preprint arXiv:2401.15859_ , 2024.
  * [35] Z. Cai, K. Stefanov, A. Dhall, and M. Hayat, “Do you really mean that? content driven audio-visual deepfake dataset and multimodal method for temporal forgery localization,” in _DICTA_ , 2022.
  * [36] Y. Nirkin, Y. Keller, and T. Hassner, “Fsgan: Subject agnostic face swapping and reenactment,” in _ICCV_ , 2019.
  * [37] T. Karras, S. Laine, and T. Aila, “A style-based generator architecture for generative adversarial networks,” in _CVPR_ , 2019.
  * [38] D. Ko, S. Lee, J. Park, S. Shin, D. Hong, and S. S. Woo, “Deepfake detection for facial images with facemasks,” in _WDC_ , 2022.
  * [39] C. Bhattacharyya, H. Wang, F. Zhang, S. Kim, and X. Zhu, “Diffusion deepfake,” _arXiv_ , 2024.
  * [40] B. Cho, B. M. Le, J. Kim, S. Woo, S. Tariq, A. Abuadbba, and K. Moore, “Towards understanding of deepfake videos in the wild,” in _CIKM_ , 2023.
  * [41] G.-W. Park, E.-J. Park, and S. S. Woo, “Zoom-df: a dataset for video conferencing deepfake,” in _Proceedings of the 1st Workshop on Security Implications of Deepfakes and Cheapfakes_ , 2022.
  * [42] W. Yang, X. Zhou, Z. Chen, B. Guo, Z. Ba, Z. Xia, X. Cao, and K. Ren, “Avoid-df: Audio-visual joint learning for detecting deepfake,” _IEEE Transactions on Information Forensics and Security_ , 2023.
  * [43] W. Chen, S. L. B. Chua, S. Winkler, and S.-K. Ng, “Trusted media challenge dataset and user study,” in _CIKM_ , 2022.
  * [44] A. S, V. P, and V. G. Menon, “Mmdfd- a multimodal custom dataset for deepfake detection,” in _Proceedings of the 2023 Fifteenth International Conference on Contemporary Computing_ , 2023.
  * [45] D. Afchar, V. Nozick, J. Yamagishi, and I. Echizen, “Mesonet: a compact facial video forgery detection network,” in _WIFS_ , 2018.
  * [46] Z. Liu, X. Qi, and P. H. Torr, “Global texture enhancement for fake face detection in the wild,” in _CVPR_ , 2020.
  * [47] H. Zhao, W. Zhou, D. Chen, T. Wei, W. Zhang, and N. Yu, “Multi-attentional deepfake detection,” in _CVPR_ , 2021.
  * [48] C. Tan, H. Liu, Y. Zhao, S. Wei, G. Gu, P. Liu, and Y. Wei, “Rethinking the up-sampling operations in cnn-based generative network for generalizable deepfake detection,” in _CVPR_ , 2024.
  * [49] C. Tan, P. Liu, R. Tao, H. Liu, Y. Zhao, B. Wu, and Y. Wei, “Data-independent operator: A training-free artifact representation extractor for generalizable deepfake detection,” _arXiv preprint arXiv:2403.06803_ , 2024.
  * [50] R. Corvi, D. Cozzolino, G. Poggi, K. Nagano, and L. Verdoliva, “Intriguing properties of synthetic images: from generative adversarial networks to diffusion models,” in _CVPRW_ , 2023.
  * [51] J. Chen, J. Yao, and L. Niu, “A single simple patch is all you need for ai-generated image detection,” _arXiv preprint arXiv:2402.01123_ , 2024.
  * [52] Y. Ju, S. Jia, J. Cai, H. Guan, and S. Lyu, “Glff: Global and local feature fusion for ai-synthesized image detection,” _IEEE Transactions on Multimedia_ , 2023.
  * [53] Y. Zhang and X. Xu, “Diffusion noise feature: Accurate and fast generated image detection,” _arXiv preprint arXiv:2312.02625_ , 2023.
  * [54] Y. Zhang, Z. Yu, X. Huang, L. Shen, and J. Ren, “Genface: A large-scale fine-grained face forgery benchmark and cross appearance-edge learning,” _arXiv preprint arXiv:2402.02003_ , 2024.
  * [55] D. Nguyen, N. Mejri, I. P. Singh, P. Kuleshova, M. Astrid, A. Kacem, E. Ghorbel, and D. Aouada, “Laa-net: Localized artifact attention network for high-quality deepfakes detection,” in _CVPR_ , 2024.
  * [56] Y. Li, M.-C. Chang, and S. Lyu, “In ictu oculi: Exposing ai created fake videos by detecting eye blinking,” in _WIFS_ , 2018.
  * [57] H. Qi, Q. Guo, F. Juefei-Xu, X. Xie, L. Ma, W. Feng, Y. Liu, and J. Zhao, “Deeprhythm: Exposing deepfakes with attentional visual heartbeat rhythms,” in _ACM MM_ , 2020.
  * [58] A. Haliassos, K. Vougioukas, S. Petridis, and M. Pantic, “Lips don’t lie: A generalisable and robust approach to face forgery detection,” in _CVPR_ , 2021.
  * [59] S. Ge, F. Lin, C. Li, D. Zhang, J. Tan, W. Wang, and D. Zeng, “Latent pattern sensing: Deepfake video detection via predictive representation learning,” in _ACM MM in Asia_ , 2022.
  * [60] G. Pang, B. Zhang, Z. Teng, Z. Qi, and J. Fan, “Mre-net: Multi-rate excitation network for deepfake video detection,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2023.
  * [61] Z. Wang, J. Bao, W. Zhou, W. Wang, and H. Li, “Altfreezing for more general video face forgery detection,” in _CVPR_ , 2023.
  * [62] Y. Xu, J. Liang, G. Jia, Z. Yang, Y. Zhang, and R. He, “Tall: Thumbnail layout for deepfake video detection,” in _ICCV_ , 2023.
  * [63] Y. Xu, J. Liang, L. Sheng, and X.-Y. Zhang, “Towards generalizable deepfake video detection with thumbnail layout and graph reasoning,” _arXiv preprint arXiv:2403.10261_ , 2024.
  * [64] B. Liu, B. Liu, M. Ding, T. Zhu, and X. Yu, “Ti2net: temporal identity inconsistency network for deepfake detection,” in _WACV_ , 2023.
  * [65] Z. Chen, X. Liao, X. Wu, and Y. Chen, “Compressed deepfake video detection based on 3d spatiotemporal trajectories,” _arXiv preprint arXiv:2404.18149_ , 2024.
  * [66] C. Wang and W. Deng, “Representative forgery mining for fake face detection,” in _CVPR_ , 2021.
  * [67] T. Zhao, X. Xu, M. Xu, H. Ding, Y. Xiong, and W. Xia, “Learning self-consistency for deepfake detection,” in _ICCV_ , 2021.
  * [68] L. Chen, Y. Zhang, Y. Song, L. Liu, and J. Wang, “Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection,” in _CVPR_ , 2022.
  * [69] K. Shiohara and T. Yamasaki, “Detecting deepfakes with self-blended images,” in _CVPR_ , 2022.
  * [70] N. Larue, N.-S. Vu, V. Struc, P. Peer, and V. Christophides, “Seeable: Soft discrepancies and bounded contrastive learning for exposing deepfakes,” in _ICCV_ , 2023.
  * [71] Z. Yan, Y. Luo, S. Lyu, Q. Liu, and B. Wu, “Transcending forgery specificity with latent space augmentation for generalizable deepfake detection,” _arXiv preprint arXiv:2311.11278_ , 2023.
  * [72] H. Li, J. Zhou, B. Li, J. Dong, and Y. Li, “Freqblender: Enhancing deepfake detection by blending frequency knowledge,” _arXiv preprint arXiv:2404.13872_ , 2024.
  * [73] Y. Qian, G. Yin, L. Sheng, Z. Chen, and J. Shao, “Thinking in frequency: Face forgery detection by mining frequency-aware clues,” in _ECCV_ , 2020.
  * [74] S. Chen, T. Yao, Y. Chen, S. Ding, J. Li, and R. Ji, “Local relation learning for face forgery detection,” in _AAAI_ , 2021.
  * [75] J. Li, H. Xie, J. Li, Z. Wang, and Y. Zhang, “Frequency-aware discriminative feature learning supervised by single-center loss for face forgery detection,” in _CVPR_ , 2021.
  * [76] Y. Luo, Y. Zhang, J. Yan, and W. Liu, “Generalizing face forgery detection with high-frequency features,” in _CVPR_ , 2021.
  * [77] H. Liu, X. Li, W. Zhou, Y. Chen, Y. He, H. Xue, W. Zhang, and N. Yu, “Spatial-phase shallow learning: rethinking face forgery detection in frequency domain,” in _CVPR_ , 2021.
  * [78] S. Woo _et al._ , “Add: Frequency attention and multi-view based knowledge distillation to detect low-quality compressed deepfake images,” in _AAAI_ , 2022.
  * [79] Y. Jeong, D. Kim, Y. Ro, and J. Choi, “Frepgan: Robust deepfake detection using frequency-level perturbations,” in _AAAI_ , 2022.
  * [80] Y. Wang, K. Yu, C. Chen, X. Hu, and S. Peng, “Dynamic graph learning with content-guided spatial-frequency relation reasoning for deepfake detection,” in _CVPR_ , 2023.
  * [81] C. Tan, Y. Zhao, S. Wei, G. Gu, P. Liu, and Y. Wei, “Frequency-aware deepfake detection: Improving generalizability through frequency space domain learning,” in _AAAI_ , 2024.
  * [82] C. T. Doloriel and N.-M. Cheung, “Frequency masking for universal deepfake detection,” in _ICASSP_ , 2024.
  * [83] Z. Wang, Z. Cheng, J. Xiong, X. Xu, T. Li, B. Veeravalli, and X. Yang, “A timely survey on vision transformer for deepfake detection,” _arXiv preprint arXiv:2405.08463_ , 2024.
  * [84] S. Saha, R. Perera, S. Seneviratne, T. Malepathirana, S. Rasnayaka, D. Geethika, T. Sim, and S. Halgamuge, “Undercover deepfakes: Detecting fake segments in videos,” in _ICCVW_ , 2023.
  * [85] W. Bai, Y. Liu, Z. Zhang, B. Li, and W. Hu, “Aunet: Learning relations between action units for face forgery detection,” in _CVPR_ , 2023.
  * [86] R. Shao, T. Wu, L. Nie, and Z. Liu, “Deepfake-adapter: Dual-level adapter for deepfake detection,” _arXiv preprint arXiv:2306.00863_ , 2023.
  * [87] A. Luo, R. Cai, C. Kong, X. Kang, J. Huang, and A. C. Kot, “Forgery-aware adaptive vision transformer for face forgery detection,” _arXiv preprint arXiv:2309.11092_ , 2023.
  * [88] C. Kong, A. Luo, S. Xia, Y. Yu, H. Li, and A. C. Kot, “Moe-ffd: Mixture of experts for generalized and parameter-efficient face forgery detection,” _arXiv preprint arXiv:2404.08452_ , 2024.
  * [89] Z. Liu, H. Wang, Y. Kang, and S. Wang, “Mixture of low-rank experts for transferable ai-generated image detection,” _arXiv preprint arXiv:2404.04883_ , 2024.
  * [90] H. H. Nguyen, J. Yamagishi, and I. Echizen, “Exploring self-supervised vision transformers for deepfake detection: A comparative analysis,” _arXiv preprint arXiv:2405.00355_ , 2024.
  * [91] Y.-H. Han, T.-M. Huang, S.-T. Lo, P.-H. Huang, K.-L. Hua, and J.-C. Chen, “Towards more general video-based deepfake detection through facial feature guided adaptation for foundation model,” _arXiv preprint arXiv:2404.05583_ , 2024.
  * [92] K. Tsigos, E. Apostolidis, S. Baxevanakis, S. Papadopoulos, and V. Mezaris, “Towards quantitative evaluation of explainable ai methods for deepfake detection,” in _ACM ICMRW_ , 2024.
  * [93] K. Sun, S. Chen, T. Yao, X. Sun, S. Ding, and R. Ji, “Towards general visual-linguistic face forgery detection,” _arXiv preprint arXiv:2307.16545_ , 2023.
  * [94] Y. Zhang, B. Colman, A. Shahriyari, and G. Bharaj, “Common sense reasoning for deep fake detection,” _arXiv preprint arXiv:2402.00126_ , 2024.
  * [95] Y. Shi, Y. Gao, Y. Lai, H. Wang, J. Feng, L. He, J. Wan, C. Chen, Z. Yu, and X. Cao, “Shield: An evaluation benchmark for face spoofing and forgery detection with multimodal large language models,” _arXiv preprint arXiv:2402.04178_ , 2024.
  * [96] S. Jia, R. Lyu, K. Zhao, Y. Chen, Z. Yan, Y. Ju, C. Hu, X. Li, B. Wu, and S. Lyu, “Can chatgpt detect deepfakes? a study of using multimodal large language models for media forensics,” _arXiv preprint arXiv:2403.14077_ , 2024.
  * [97] Y. Li, X. Liu, X. Wang, S. Wang, and W. Lin, “Fakebench: Uncover the achilles’ heels of fake images with large multimodal models,” _arXiv preprint arXiv:2404.13306_ , 2024.
  * [98] Y. Yin, Y. Bai, Y. Wang, and Y. Fu, “Concentric ring loss for face forgery detection,” in _ICDM_ , 2023.
  * [99] J. Zhang and J. Ni, “Domain-invariant feature learning for general face forgery detection,” in _ICME_ , 2023.
  * [100] J. Zhang, J. Ni, F. Nie, and j. Huang, “Domain-invariant and patch-discriminative feature learning for general deepfake detection,” _ACM Transactions on Multimedia Computing, Communications and Applications_ , 2024.
  * [101] J. Liang, H. Shi, and W. Deng, “Exploring disentangled content information for face forgery detection,” in _ECCV_ , 2022.
  * [102] Z. Yan, Y. Zhang, Y. Fan, and B. Wu, “Ucf: Uncovering common features for generalizable deepfake detection,” in _ICCV_ , 2023.
  * [103] L. Lin, X. He, Y. Ju, X. Wang, F. Ding, and S. Hu, “Preserving fairness generalization in deepfake detection,” in _CVPR_ , 2024.
  * [104] Z. Ba, Q. Liu, Z. Liu, S. Wu, F. Lin, L. Lu, and K. Ren, “Exposing the deception: Uncovering more forgery clues for deepfake detection,” in _AAAI_ , 2024.
  * [105] H. Khalid and S. S. Woo, “Oc-fakedect: Classifying deepfakes using one-class variational autoencoder,” in _CVPRW_ , 2020.
  * [106] Y. He, N. Yu, M. Keuper, and M. Fritz, “Beyond the spectrum: Detecting deepfakes via re-synthesis,” in _IJCAI_ , 2021.
  * [107] J. Cao, C. Ma, T. Yao, S. Chen, S. Ding, and X. Yang, “End-to-end reconstruction-classification learning for face forgery detection,” in _CVPR_ , 2022.
  * [108] Z. Cai, S. Ghosh, K. Stefanov, A. Dhall, J. Cai, H. Rezatofighi, R. Haffari, and M. Hayat, “Marlin: Masked autoencoder for facial video representation learning,” in _CVPR_ , 2023.
  * [109] V. Asnani, X. Yin, T. Hassner, and X. Liu, “Malp: Manipulation localization using a proactive scheme,” in _CVPR_ , 2023.
  * [110] L. Li, J. Bao, T. Zhang, H. Yang, D. Chen, F. Wen, and B. Guo, “Face x-ray for more general face forgery detection,” in _CVPR_ , 2020.
  * [111] H. Dang, F. Liu, J. Stehouwer, X. Liu, and A. K. Jain, “On the detection of digital face manipulation,” in _CVPR_ , 2020.
  * [112] C. Shuai, J. Zhong, S. Wu, F. Lin, Z. Wang, Z. Ba, Z. Liu, L. Cavallaro, and K. Ren, “Locate and verify: A two-stream network for improved deepfake detection,” in _ACM MM_ , 2023.
  * [113] X. Zhang, R. Li, J. Yu, Y. Xu, W. Li, and J. Zhang, “Editguard: Versatile image watermarking for tamper localization and copyright protection,” in _CVPR_ , 2024.
  * [114] J. Hu, X. Liao, D. Gao, S. Tsutsui, Q. Wang, Z. Qin, and M. Z. Shou, “Delocate: Detection and localization for deepfake videos with randomly-located tampered traces,” _arXiv preprint arXiv:2401.13516_ , 2024.
  * [115] D.-C. Tantaru, E. Oneata, and D. Oneata, “Weakly-supervised deepfake localization in diffusion-generated images,” in _WACV_ , 2024.
  * [116] S. Dong, J. Wang, R. Ji, J. Liang, H. Fan, and Z. Ge, “Implicit identity leakage: The stumbling block to improving deepfake detection generalization,” in _CVPR_ , 2023.
  * [117] D. Cozzolino, A. Rössler, J. Thies, M. Nießner, and L. Verdoliva, “Id-reveal: Identity-aware deepfake video detection,” in _ICCV_ , 2021.
  * [118] Y. Nirkin, L. Wolf, Y. Keller, and T. Hassner, “Deepfake detection based on the discrepancy between the face and its context,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2021.
  * [119] X. Dong, J. Bao, D. Chen, T. Zhang, W. Zhang, N. Yu, D. Chen, F. Wen, and B. Guo, “Protecting celebrities from deepfake with identity consistency transformer,” in _CVPR_ , 2022.
  * [120] B. Huang, Z. Wang, J. Yang, J. Ai, Q. Zou, Q. Wang, and D. Ye, “Implicit identity driven deepfake face swapping detection,” in _CVPR_ , 2023.
  * [121] S. Woo _et al._ , “Add: Frequency attention and multi-view based knowledge distillation to detect low-quality compressed deepfake images,” in _AAAI_ , 2022.
  * [122] N. Bonneel, J. Rabin, G. Peyré, and H. Pfister, “Sliced and radon wasserstein barycenters of measures,” _Journal of Mathematical Imaging and Vision_ , 2015.
  * [123] M. Kim, S. Tariq, and S. S. Woo, “Fretal: Generalizing deepfake detection using knowledge distillation and representation learning,” in _CVPRW_ , 2021.
  * [124] ——, “Cored: Generalizing fake media detection with continual representation using distillation,” in _ACM MM_ , 2021.
  * [125] Q. Lv, Y. Li, J. Dong, S. Chen, H. Yu, H. Zhou, and S. Zhang, “Domainforensics: Exposing face forgery across domains via bi-directional adaptation,” _arXiv preprint arXiv:2312.10680_ , 2023.
  * [126] X. Guo, X. Liu, Z. Ren, S. Grosz, I. Masi, and X. Liu, “Hierarchical fine-grained image forgery detection and localization,” in _CVPR_ , 2023.
  * [127] L. Guarnera, O. Giudice, and S. Battiato, “Mastering deepfake detection: A cutting-edge approach to distinguish gan and diffusion-model images,” _ACM Transactions on Multimedia Computing, Communications and Applications_ , 2024.
  * [128] Y. Guo, C. Zhen, and P. Yan, “Controllable guide-space for generalizable face forgery detection,” in _ICCV_ , 2023.
  * [129] J. Ricker, S. Damm, T. Holz, and A. Fischer, “Towards the detection of diffusion model deepfakes,” in _VISAPP_ , 2024.
  * [130] R. Corvi, D. Cozzolino, G. Zingarini, G. Poggi, K. Nagano, and L. Verdoliva, “On the detection of synthetic images generated by diffusion models,” in _ICASSP_ , 2023.
  * [131] P. Lorenz, R. L. Durall, and J. Keuper, “Detecting images generated by deep diffusion models using their local intrinsic dimensionality,” in _ICCVW_ , 2023.
  * [132] Z. Wang, J. Bao, W. Zhou, W. Wang, H. Hu, H. Chen, and H. Li, “Dire for diffusion-generated image detection,” in _ICCV_ , 2023.
  * [133] R. Ma, J. Duan, F. Kong, X. Shi, and K. Xu, “Exposing the fake: Effective diffusion-generated images detection,” in _ICMLW_ , 2023.
  * [134] Y. Luo, J. Du, K. Yan, and S. Ding, “Lare^ 2: Latent reconstruction error based method for diffusion-generated image detection,” in _CVPR_ , 2024.
  * [135] N. Ruiz, S. A. Bargal, and S. Sclaroff, “Disrupting deepfakes: Adversarial attacks against conditional image translation networks and facial manipulation systems,” in _ECCVW_ , 2020.
  * [136] C.-Y. Yeh, H.-W. Chen, S.-L. Tsai, and S.-D. Wang, “Disrupting image-translation-based deepfake algorithms with adversarial attacks,” in _WACVW_ , 2020.
  * [137] Q. Huang, J. Zhang, W. Zhou, W. Zhang, and N. Yu, “Initiative defense against facial manipulation,” in _AAAI_ , 2021.
  * [138] R. Wang, Z. Huang, Z. Chen, L. Liu, J. Chen, and L. Wang, “Anti-forgery: Towards a stealthy and robust deepfake disruption attack via adversarial perceptual-aware perturbations,” in _IJCAI_ , 2022.
  * [139] S. Aneja, L. Markhasin, and M. Nießner, “Tafim: Targeted adversarial attacks against facial image manipulations,” in _ECCV_ , 2022.
  * [140] H. Huang, Y. Wang, Z. Chen, Y. Zhang, Y. Li, Z. Tang, W. Chu, J. Chen, W. Lin, and K.-K. Ma, “Cmua-watermark: A cross-model universal adversarial watermark for combating deepfakes,” in _AAAI_ , 2022.
  * [141] V. Asnani, X. Yin, T. Hassner, S. Liu, and X. Liu, “Proactive image manipulation detection,” in _CVPR_ , 2022.
  * [142] T. Wang, M. Huang, H. Cheng, B. Ma, and Y. Wang, “Robust identity perceptual watermark against deepfake face swapping,” _arXiv preprint arXiv:2311.01357_ , 2023.
  * [143] X. Wu, X. Liao, and B. Ou, “Sepmark: Deep separable watermarking for unified source tracing and deepfake detection,” in _ACM MM_ , 2023.
  * [144] L. Tang, D. Ye, Z. Lu, Y. Zhang, S. Hu, Y. Xu, and C. Chen, “Feature extraction matters more: Universal deepfake disruption through attacking ensemble feature extractors,” _arXiv preprint arXiv:2303.00200_ , 2023.
  * [145] Y. Zhang, D. Ye, C. Xie, L. Tang, X. Liao, Z. Liu, C. Chen, and J. Deng, “Dual defense: Adversarial, traceable, and invisible robust watermarking against face swapping,” _IEEE Transactions on Information Forensics and Security_ , 2024.
  * [146] X. Wu, X. Liao, B. Ou, Y. Liu, and Z. Qin, “Are watermarks bugs for deepfake detectors? rethinking proactive forensics,” in _IJCAI_ , 2024.