# Unmasking Deepfakes: Masked Autoencoding Spatiotemporal Transformers for
Enhanced Video Forgery Detection

Sayantan Das1, Mojtaba Kolahdouzi1, Levent Özparlak2, Will Hickie2, Ali
Etemad1
1Queen’s University, Canada
2Irdeto BV

###### Abstract

We present a novel approach for the detection of deepfake videos using a pair
of vision transformers pre-trained by a self-supervised masked autoencoding
setup. Our method consists of two distinct components, one of which focuses on
learning spatial information from individual RGB frames of the video, while
the other learns temporal consistency information from optical flow fields
generated from consecutive frames. Unlike most approaches where pre-training
is performed on a generic large corpus of images, we show that by pre-training
on smaller face-related datasets, namely Celeb-A (for the spatial learning
component) and YouTube Faces (for the temporal learning component), strong
results can be obtained. We perform various experiments to evaluate the
performance of our method on commonly used datasets namely FaceForensics++
(Low Quality and High Quality, along with a new highly compressed version
named Very Low Quality) and Celeb-DFv2 datasets. Our experiments show that our
method sets a new state-of-the-art on FaceForensics++ (LQ, HQ, and VLQ), and
obtains competitive results on Celeb-DFv2. Moreover, our method outperforms
other methods in the area in a cross-dataset setup where we fine-tune our
model on FaceForensics++ and test on CelebDFv2, pointing to its strong cross-
dataset generalization ability. We make the code publicly available at
http://ucalyptus.github.io/masdt.

##  1 Introduction

Facial forgery detection, also known as deepfake detection, is a rapidly
growing field with important real-world applications [54]. With the recent
explosion in the success of sophisticated deep generative models [28, 59, 42],
it has become increasingly easy to generate highly realistic fake images and
videos (see Figure 1 for an example of a real image along with four
manipulated versions). The advancements in artificial intelligence have made
it possible to create deepfakes that are nearly indistinguishable from genuine
content, making the detection process even more challenging. This has led to a
growing concern about the potential for malicious actors to use these tools
for nefarious purposes, such as spreading disinformation, manipulating public
opinion, or even causing social unrest. Given the potential risks associated
with deepfakes, the importance of developing effective detection methods
cannot be overstated [36, 39, 43].

Figure 1: A facial image along with the four manipulated versions from the
FaceForensics++ dataset.

The field of deepfake detection has seen considerable progress in recent years
with a number of sophisticated techniques being proposed in the area [61, 41].
However, in the context of detecting manipulated content in videos, many
existing methods primarily focus on spatial features extracted from individual
frames [7]. This approach can lead to the overlooking of temporal dynamics
that evolve throughout video sequences. This strategy can result in
limitations, as temporal artifacts such as flickering and motion
discontinuities, are common indicators of deepfake manipulation. Furthermore,
sophisticated deepfakes may exhibit subtle spatial inconsistencies that
manifest over time, necessitating an integrated analysis of both spatial and
temporal information. Moreover, we hypothesize that capturing subtle
spatiotemporal inconsistencies that are often caused by different deepfake
generation methods, could significantly enhance performance by learning
representations that generalize to unseen forgery methods, which is often a
challenging problem in this area.

In response to the challenges mentioned above, in this paper, we present a
novel approach to deepfake detection that consists of two distinct components.
One component learns spatial information from individual RGB frames of the
videos, while the second component leverages optical flow fields to learn
temporal consistency across the video. Both components utilize vision
transformers [16], which we train in two steps. First, inspired by [21] we
pre-train the models in an autoencoding setup using a self-supervised
reconstruction scheme. Second, we discard the reconstruction decoder and add a
new classification head to each encoder, where they are fine-tuned for
deepfake detection, followed by score-level fusion of the results. We pre-
train the spatial learning and temporal consistency learning encoders with
CelebFaces-Attributes (Celeb-A) [34] and YouTube Faces [58] datasets
respectively. For downstream deepfake detection, we evaluate our approach on
the FaceForensics++ (High Quality) and FaceForensics++ (Low Quality) datasets
[44] which employ compression factors of 23% and 40%, respectively, in
addition to the CelebDFv2 dataset [32]. Additionally, we synthesize a more
challenging variation of the FaceForensics++ dataset, which we call
FaceForensics++ (Very Low Quality) by compressing the data with a rate of 65%.
This dataset is then used to further evaluate our approach against existing
techniques in the presence of extreme compression artifacts. Experimental
results demonstrate that our method achieves state-of-the-art performance on
FaceForensics++ (LQ and HQ) datasets and highlight the efficacy of our
approach in detecting deepfakes across diverse compression levels. Ablation
studies demonstrate the importance of different components of our method.
Lastly, state-of-the-art results when fine-tuning our model on FaceForensics++
and testing it on CelebDFv2 (cross-dataset evaluation) demonstrates the strong
generalization of our method.

Our contributions in this paper are summarized as follows: (1) We propose a
new approach for effective facial forgery detection. Our method uses a vision
autoencoding transformer and is pre-trained in a self-supervised masked
reconstruction setup. Our solution consists of two main components which learn
spatial (RGB) and temporal consistency information (optical flow fields)
separately. (2) We leverage relatively small datasets, namely Celeb-A and
YouTube Faces, for pretraining our transformers, and achieve state-of-the-art
results in the downstream task of deepfake detection on FaceForensics++ (LQ,
HQ, and VLQ) datasets and competitive results on Celeb-DFv2. (3) We make our
code available online at http://ucalyptus.github.io/masdt to contribute to the
area and enable reproducibility.

##  2 Related Work

Deepfake detection has traditionally been addressed as a binary classification
task [41], where the objective is to discern between authentic and manipulated
media. The application of deep learning models, particularly convolutional
neural networks (CNNs), has been central to achieving this objective [10, 13,
1]. Authors of FaceForensics++ dataset [44] used Xception network, which was
one of the best-performing architectures at the time, to perform deepfake
detection via transfer learning [10].

Researchers proposed a method in [13] that utilizes residual-based descriptors
in the form of a constrained CNN for image forgery detection. This approach
aims to capture and analyze the residual noise present in manipulated images,
which can be a strong indicator of forgery. In contrast, another method
introduced a deep learning approach that focuses on the mesoscopic properties
of images [1]. In this context, ‘mesoscopic’ refers to properties or features
that fall between the small scale (microscopic) and the large scale
(macroscopic). By concentrating on mesoscopic features, the model can capture
subtle artifacts and inconsistencies in manipulated images, potentially making
it more effective in detecting forgeries.

Various studies have utilized frequency analysis to detect inconsistencies
that arise during deepfake creation [33, 9]. In [33], the researchers employed
the phase spectrum for forged face image detection, showing that CNNs can
identify additional implicit phase spectrum features that are advantageous in
detecting face forgeries. Concurrently, the study in [9] developed a multi-
scale patch similarity module to specifically model second-order relationships
between distinct local regions, forming a similarity pattern through pairwise
cosine measurements. These patterns distinguish real from forged regions by
recognizing differences such as irregular textures and high-frequency noise.

Self-supervised learning (SSL) has been explored to address the issue of
limited labeled data for deepfake detection [8, 66, 64, 29, 60]. For instance,
self-supervised learning was employed in [8] with an auxiliary task
specifically designed for deepfake detection, using a synthesizer and
adversarial training framework to dynamically generate forgeries. This
approach enriches diversity and strengthens sensitivity to produce strong
results. In the method proposed in [66], mouth motion representations were
learned by encouraging close-paired video and audio representations, while
keeping unpaired ones diverse. The study in [64] proposed a decoupling
strategy to separate facial authenticity and compression relevance,
implemented through a joint self-supervised learning approach using
compression ratios as self-supervised signals. Another study utilized a multi-
modal backbone trained in a self-supervised manner and adapted it to the video
deepfake domain [29]. These self-supervised models leverage unlabeled data to
learn useful representations for detection tasks. Contrastive learning is
another common pre-text learning approach often considered for deepfake
detection [18, 60]. In the study by [18], two different transformed versions
of a face image were generated using two distinct transformations. The
agreement between these transformed images is maximized after they are passed
through an encoder network and a projection head network, effectively training
the model without supervision signals. On the other hand, another study
employed supervised contrastive learning to learn common features between
instances of the same class, while distinguishing between samples from
different classes [60].

In recent studies, researchers have sought to combine multiple modalities,
such as visual, audio, and temporal information, to improve detection
performance in deepfake detection tasks [33, 5, 25]. These multi-modal
approaches provide a comprehensive view of the media, making them more robust
against limitations specific to individual modalities [27]. While frequency-
based modalities have been employed in multi-modal deepfake detection
solutions [33], we believe that optical flow has the potential to serve as an
effective alternative source of information. Prior works such as [2, 6, 46,
26] have used optical flow features followed by a classifier for deepfake
detection. In contrast to these, our method, for the first time, leverages
optical flow information alongside RGB features in a masked autoencoding setup
to improve cross-dataset generalizability and robustness. To our knowledge,
prior deepfake detection methods involving ViTs [67, 23, 48, 22, 57] have not
used optical flow information to capture the temporal inconsistencies.

Figure 2: An overview of our method (MASDT), which includes the masked facial
reconstruction and deepfake detection steps.

##  3 Proposed Methodology

###  3.1 Overview

Our proposed approach titled Masked Autoencoding Spatiotemporal Deepfake
Transformer (MASDT) consists of two components: spatial learning and temporal
consistency learning. The spatial learning component has the objective of
learning robust spatial features from the RGB images, while temporal
consistency learning aims to extract temporal features from optical flow
fields derived from the input images. We fuse the classification outputs
derived from the spatial and temporal consistency learning components. Both
these components follow a self-supervised autoencoding approach in a two-step
process.

The first step involves a self-supervised pre-training strategy which involves
both of the MASDT components in a data reconstruction task. We discuss this
strategy in section 3.2. The second step is the downstream task of deepfake
detection, wherein we re-purpose components trained in the previous step to
perform the classification of deepfake data through a model fine-tuning
process, followed by fusion of information from both components (spatial
learning and temporal consistency learning). This is discussed in detail in
Section 3.3. Before we discuss each of the two steps, we discuss the optical
flow field generation strategy in Section 3.1.1. A general scheme of the MASDT
strategy is presented in Figure 2.

####  3.1.1 Optical flow field estimation

We utilize a CNN model named PWC-Net for generating optical flow fields [49].
Let the model for estimating optical flow be $F_{\theta}$, and two consecutive
frames be $f_{t}$ and $f_{t+1}$. Accordingly, the estimated optical flow
$\Phi_{t}$ can be denoted by:

$\Phi_{t}=F_{\theta}(f_{t},f_{t+1}),$ (1)

where $\Phi_{t}$ is a 3-channel optical flow matrix of size $H\times W\times
3$ representing the flow field between the consecutive frames.

###  3.2 Masked Facial Reconstruction

The first step of our approach utilizes a masked self-supervised auto-encoder
which learns to reconstruct original facial images, given partial observations
[21]. This auto-encoder reconstruction pipeline consists of two blocks: a
reconstruction encoder, which captures a latent representation from the
visible portions of each image, and a reconstruction decoder that aims to
reconstruct the masked sections of the image using this latent representation.
In this procedure, the encoder is trained to extract robust spatial features
from masked facial images, eliminating noise and redundancy while transforming
the reconstruction task into a challenging process that requires generalizing
features to represent a small subset of available data [24]. Consequently, by
masking portions of the facial image using random spatial pixels or patches,
we can avoid a potential location bias toward image reconstruction, which can
be critical for the detection of deepfake images.

The goal of the decoder is to use the features obtained from the latent space
by the encoder to reconstruct the masked information from the original facial
image. We train this reconstruction encoder-decoder pair using a simple mean
squared error (MSE) reconstruction loss $\mathcal{L}_{r}$:

$\mathcal{L}_{r}=\frac{1}{N}\sum_{i=1}^{N}(y_{i}-\hat{y}_{i})^{2},$ (2)

where $N$ represents the number of sampled patches, and $\hat{y}_{i}$ and
$y_{i}$ are the $i$th output and expected $i$th output, respectively.

We perform the above masked reconstruction task for both the components
independently where we employ the encoder-decoder pairs for reconstructing RGB
images $y$ and optical flow fields $\Phi$ for spatial learning and temporal
consistency learning respectively, which is also referred to as pre-training
in self-supervised learning literature. This prepares the encoders for the
fine-tuning step mentioned in the next section.

###  3.3 Deepfake Detection

The second step of MASDT is aimed at the supervised training for the
classification of deepfake images. For this purpose, we employ the encoders
that learned to extract robust representations in the reconstruction pipeline.
Thus, to perform binary classification, a classification head consisting of a
simple MLP is attached to each of the pre-trained encoders.

We adopt a dual-encoder setup for the fine-tuning process, utilizing the
spatial learning encoder $\theta_{s}$ and the temporal consistency learning
encoder $\theta_{t}$. These encoders were previously trained in the initial
step of our proposed solution. In the process of fine-tuning for a binary
classification task, we employ a binary cross-entropy loss, denoted as
$\mathcal{L}_{b}$. The formula for this loss is as follows:

$\mathcal{L}_{b}=-\frac{1}{M}\sum_{j=1}^{M}\left[o_{j}\log(\hat{o}_{j})+(1-o_{j})\log(1-\hat{o}_{j})\right],$
(3)

Here, $\hat{o}_{j}$ is the predicted output from the network, $o_{j}$
represents the actual or target class (either 0 or 1), and $M$ denotes the
total count of samples in the batch.

####  3.3.1 Dual Modality Fusion

To further harness the strengths of both $\theta_{s}$ and $\theta_{t}$, we use
a simple fusion mechanism. This method aims to exploit the complementary
information that each encoder provides, thereby improving the overall
classification performance. The fusion process begins with the individual
outputs from $\theta_{s}$ and $\theta_{t}$, denoted as $\hat{o}_{s}$ and
$\hat{o}_{t}$ respectively, which are then combined to create a fused score-
level prediction, $\hat{o}_{f}$. Mathematically, this can be expressed as:

$\hat{o}_{f}=\alpha\cdot\hat{o}_{s}+(1-\alpha)\cdot\hat{o}_{t},$ (4)

where $\alpha$ is a fusion weight that determines the contribution of each
encoder to the final output.

##  4 Experiments

In this section, we present the specifics and details of our method and
experiments, describe the datasets used, and discuss the ablation studies
conducted to validate the impact of different components of our proposed
solution.

###  4.1 Implementation Details

In this section, we outline the implementation details of our deepfake
detection method, which incorporates both RGB and optical flow modalities. Our
experiments are conducted using the PyTorch framework [37] on 4 Nvidia A100
GPUs, each with 40 GB of vRAM. We generate optical flow fields using the PWC-
Net present in the MMFlow toolbox [12].

Our method’s performance is evaluated using the top-1 accuracy, which denotes
the percentage of correctly classified deepfake and real videos out of the
total number of videos in the test set. This metric is widely used in deepfake
detection tasks as it provides a clear indication of a model’s ability to
distinguish between real and fake videos. Accuracy and area under the curve
(AUC) are presented as the metrics for our experiments, following other
publications in the area.

For evaluation purposes, we use the FaceForensics++ (LQ and HQ) and CelebDFv2
datasets (the details of these datasets are presented in the next Section) and
divide them into training, validation, and test sets, ensuring an even
distribution of deepfake and real videos across all sets following the
instructions provided in the original dataset papers [44, 32]. Data
augmentation techniques such as random cropping, horizontal flipping, color
jittering, and MixUp augmentation, are employed to improve our model’s
robustness to input data variations. MixUp augmentation [63] involves
generating new training samples by taking linear combinations of input data
and their corresponding labels, which encourages the model to learn smooth and
robust features. In addition to MixUp, the model employs CutMix [62] data
augmentation technique with default settings (alpha set to 0, probability set
to 1, and switch probability set to 0.5). Label smoothing is applied with a
smoothing factor of 0.1. A drop path rate of 0.1 is used for stochastic depth
regularization.

Input images are resized to $224\times 224$, with patches of $16\times 16$. We
observe that a masking ratio of 90% is optimal for pre-training. We use the
transformer architecture [16] with a default Vit-B configuration as our model.
The model is trained using the AdamW optimizer, with a weight decay of 0.05, a
base learning rate of $5\times 10^{-4}$, and layer decay of 0.8. The learning
rate is scaled according to an effective batch size of 64. We train the model
for 300 epochs, using a gradient accumulation of 1 iteration. We utilize a
distributed training approach with distributed evaluation. The CUDA benchmark
is enabled, and the model is trained on available CUDA devices. For fine-
tuning, the model is initialized with our pre-trained weights from the first
step (self-supervised pre-training), and position embeddings are interpolated
accordingly.

###  4.2 Datasets

We use the FaceForensics++ (LQ), FaceForensics++ (HQ) [44], Celeb-DFv2 [32],
Celeb-A [34], and YouTube Faces [58] datasets. The first three datasets are
employed for evaluating our proposed method, while the latter two are utilized
for pre-training only. Below, we provide a detailed description of each
dataset:

FaceForensics++ (LQ) [44] simulates various scenarios where manipulated videos
appear in compressed formats. With a 40% compression factor using the H.264
video compression standard, the LQ version introduces artifacts that may be
present in real-world cases. This dataset challenges researchers to develop
techniques capable of detecting manipulations even when the video quality is
degraded due to compression.

FaceForensics++ (HQ) [44] maintains a higher quality (compression factor of
23%) compared to the LQ version, enabling researchers to study deepfakes and
other manipulations with greater detail and less information loss due to
compression. Both FaceForensics++ versions contain over 1000 original videos,
with manipulated videos created using various methods, such as DeepFakes [14],
FaceSwap [30], Face2Face [53], and NeuralTextures [52]. These datasets cover a
wide range of manipulation methods, allowing researchers to test their
detection algorithms on diverse types of deepfakes.

In order to further push our method to the limit and challenge its detection
ability in the presence of significant compression artifacts, we create an
even more compressed version in comparison to FaceForensics++ (LQ), which we
call FaceForensics++ (VLQ) where VLQ stands for very low quality. To generate
this variant of the dataset, we take the original non-compressed videos of
FaceForensics++ and compress them by a compression factor of 65%, which we
will also use in our experiments besides the datasets with two standard
compression ratios. For this purpose, we use the FFMPEG framework [56].

Celeb-DFv2 [32] includes 590 original videos collected from YouTube, featuring
subjects of varying ages, ethnicities, and genders, as well as 5639
corresponding DeepFake videos. The Celeb-DF dataset’s average video length is
13 seconds, and all videos have a standard 30 FPS frame rate.

Celeb-A (CelebFaces-Attributes) [34] is a large-scale collection of over
200,000 celebrity images, with 40 attribute labels annotated for each image.
The dataset comprises diverse subjects and captures various facial
expressions, poses, and lighting conditions.

YouTube Faces [58] is a comprehensive collection of videos from YouTube
focusing on individuals’ faces. It contains over 3,000 annotated videos of
1,595 people, offering diverse subjects with different ethnicities, ages, and
genders. Each video in the dataset is labeled with the corresponding subjects’
identities, and is often used for face recognition and verification tasks. It
captures various poses, expressions, illuminations, and occlusions.

###  4.3 Pre-training Strategy

For pre-training the RGB modality in our proposed method, we utilize the
Celeb-A dataset instead of the typically used ImageNet [15]. Celeb-A is
considerably smaller than ImageNet, as Celeb-A contains 200,000 images whereas
ImageNet contains over 14 million images. This reduced size allows for faster
pre-training and lower computational requirements, making the process more
efficient and accessible to a wider range of researchers and practitioners.
Celeb-A is specifically tailored for facial tasks, consisting exclusively of
human face images. In contrast, ImageNet covers many object categories and may
not be as well-suited and efficient for facial analysis. By pre-training our
model on Celeb-A, we ensure that the initial features learned by the model are
more relevant to facial structures, expressions, and attributes, which can
ultimately contribute to a more effective deepfake detection system.

For pre-training the optical flow modality in our method, we utilize the
YouTube Faces dataset. This dataset provides video data, essential for optical
flow calculation. Naturally, datasets of images such as ImageNet and Celeb-A
cannot be used for optical flow generation. Moreover, the YouTube Faces
dataset is specifically designed for facial analysis tasks as it consists
exclusively of human face videos. By pre-training our model on this dataset,
we ensure that the initial features learned by the temporal consistency
encoder can better capture information such as facial structures, expressions,
and attributes, ultimately contributing to a more effective deepfake detection
system.

###  4.4 Results

In this section, we present the outcome of our experiments, which assess the
performance of the proposed method for deepfake detection on the
FaceForensics++ and Celeb-DFv2 datasets. Our evaluation concentrates on the
effectiveness of integrating both RGB and optical flow modalities, as well as
the impact of pre-training on the Celeb-A and YouTube Faces datasets. By
contrasting our approach with existing methods and baseline models, we aim to
evaluate the benefits of our technique in accurately identifying deepfakes
under a range of conditions.

In Table 1 we present the top-1 accuracy and AUC scores of our proposed method
compared to the current state-of-the-art approaches. The table presents the
quantitative results for various deepfake detection techniques available in
the FaceForensics++ dataset with both high and low quality settings. It can be
observed that our proposed method achieves the highest accuracy and AUC scores
in both quality settings, surpassing the prior works and setting a new state-
of-the-art.

Table 1: Quantitative results for ACC and AUC on the FaceForensics++ dataset with both quality settings (LQ and HQ). The results are arranged in ascending order on the basis of ACC (LQ). Methods | ACC | AUC | ACC | AUC
---|---|---|---|---
(HQ) | (HQ) | (LQ) | (LQ)
Steg. Features [17] | 70.97% | - | 55.98% | -
LD-CNN [13] | 78.45% | - | 58.69% | -
CP-CNN [40] | 79.08% | - | 61.18% | -
Face X-ray [31] | - | 87.40% | - | 61.60%
C-Conv [4] | 82.97% | - | 66.84% | -
MesoNet [1] | 83.10% | - | 70.47% | -
FakeCatcher [11] | 94.65% | - | - | -
Two-branch RN [35] | 96.43% | 88.70% | 86.34% | 86.59%
Xception [45] | 95.73% | - | 86.86% | -
LipsDontLie [20] | - | 97.10% | - | -
Capsule Net [55] | - | 99.50% | - | -
SLADD [8] | - | 98.40% | - | -
MADD [65] | 97.60% | 99.29% | 88.69% | 90.40%
Self Info. Att. [50] | 97.64% | 99.35% | 90.23% | 93.45%
F3-Net [38] | 97.52% | 98.10% | 90.43% | 93.30%
E2E Learning [7] | 97.06% | 99.32% | 91.03% | 95.02%
Local Relation Learning [9] | 97.59% | 99.46% | 91.47% | 95.21%
Ours | 98.19% | 99.67% | 97.79% | 98.45%

In our experiments, we assess the performance of different deepfake generation
methods in the FaceForensics++ (LQ) dataset, comprising four distinct
techniques: DeepFakes (DF) [14], Face2Face (FF) [53], FaceSwap (FS) [30], and
NeuralTextures (NT) [52], as illustrated in Table 2. In this table, we present
a breakdown of the performance of our method and others across these four
deepfake generation methods, and compare the accuracy with other state-of-the-
art approaches. The results indicate that our method achieves strong results
across all four manipulation techniques, particularly in the FF and FS
methods, and generates competitive results for the other two. These findings
demonstrate the effectiveness of our approach in detecting manipulated face
images across different forgery approaches.

Table 2: Quantitative results (ACC) on the FaceForensics++ (LQ) dataset with four manipulation methods: DeepFakes (DF), Face2Face (FF), FaceSwap (FS), and NeuralTextures (NT). Methods |  DF [14] |  FF [53] |  FS [30] |  NT [52]
---|---|---|---|---
Steg. Features [17] | 67.00% | 48.00% | 49.00% | 56.00%
LD-CNN [13] | 75.00% | 56.00% | 51.00% | 62.00%
C-Conv [4] | 87.00% | 82.00% | 74.00% | 74.00%
CP-CNN [40] | 80.00% | 62.00% | 59.00% | 59.00%
MesoNet [1] | 90.00% | 83.00% | 83.00% | 75.00%
Xception [45] | 96.01% | 93.29% | 94.71% | 79.14%
F3-Net [38] | 97.97% | 95.32% | 96.53% | 83.32%
Local Relation Learning [9] |  98.84 % | 95.53% | 97.53% | 89.31%
Ours | 97.84% | 96.27% | 97.89% | 78.23%

Next, we evaluate the performance of our method compared to other recent
methods on the Celeb-DFv2 dataset and present the performance in Table 3. It
can be observed that our method achieves results competitive to the current
state-of-the-art [7].

Table 3: Quantitative results in terms of ACC and AUC on the Celeb-DFv2 dataset. Methods | ACC | AUC
---|---|---
F3-Net [38] | 95.95% | 98.93%
Xception [45] | 97.90% | 99.73%
E2E Learning [7] | 98.59% | 99.94%
Ours | 98.00% | 98.90%

To further explore the generalization capability of our model, we follow the
cross-dataset scheme presented in [7], [45], and [9]. In this experiment, we
train the model on the FaceForensics++ datasets and test its performance on
the Celeb-DFv2 dataset. We present the results in Table 4, where we observe
that our method outperforms prior works in the area, indicating strong
generalization ability in detecting deepfakes even when training is done on a
different dataset and likely constitutes a different distribution (out-of-
distribution).

Table 4: Cross-dataset evaluation (AUC) by training on FaceForensics++ (LQ) and testing on the Celeb-DFv2 dataset. Methods | AUC
---|---
Xception [45] | 36.19%
E2E Learning [7] | 68.71%
Local Relation Learning [9] | 78.26%
Ours | 80.21%

To further push our approach to the limit, we explore its performance on the
VLQ version of the FaceForensics++ dataset which we constructed for the first
time by applying a 65% compression ratio (see Section 4.2). We also use this
dataset on two leading methods, namely DCL [51] and E2E Reconstruction
Learning [7]. The results are presented in Table 5 where we observe that our
method outperforms both other solutions, highlighting the efficiency and
resilience of our approach in detecting deepfakes, even in the presence of
highly compressed data.

To better contextualize our method within prior works, we compare the
performance of our method to prior methods that have used optical flow and
vision transformers in Tables 6 and 7 respectively. The results show that we
achieve superior results in comparison to other methods that have used optical
flow for deepfake detection when evaluating on the Face2Face manipulation
method of the FaceForensics++ dataset. Similarly, we outperform other methods
in the literature that leverage vision transformers.

Table 5: Quantitative results on FaceForensics++ (VLQ) dataset which is constructed by applying a 65% compression ratio. Methods | ACC
---|---
DCL [51] | 65.20%
E2E Learning [7] | 78.20%
Ours | 79.70%

Lastly, we utilize Grad-CAM [47] visualization on our model and similar
performing methods to demonstrate and investigate the attention patterns of
each method. Grad-CAM is capable of pinpointing the areas that the network
applies more attention to, and thus deems important. We present a sample image
in Figure 3, where the red areas highlight parts of the image which are more
salient for the models. We observe that our model considers broader areas of
the face image as important toward detection of whether the input is a
deepfake image or not. This is a noteworthy observation as it indicates that
the proposed method is capable of capturing a more comprehensive set of
features and artifacts, which might be overlooked by the other models. This
ability to focus on multiple areas simultaneously could enable the proposed
method to better discern subtle inconsistencies and artifacts that are
characteristic of deepfakes or manipulated images. In contrast, the other two
models, with their more concentrated attention patterns, may be less effective
in capturing the full extent of these subtle cues, which might result in lower
overall performance in detecting such forgeries. Another interesting pattern
which can be observed is that prior methods seem to focus on select areas,
namely the left eye and to some extent the right ear. However, in addition to
these regions, our method considers the nose and mouth regions, which are
critical areas for authentic face images.

Figure 3: Comparison of Grad-CAM visualizations [47] for our method in comparison to two other recent works. Table 6: Comparison to prior deepfake detection methods that use optical flow. The results are reported on the Face2Face manipulation method of FaceForensics++. Methods | Training dataset | AUC
---|---|---
OF + CNN [2] | FF++ | -
OF + CNN [6] | FF++ | -
OF + CNN-LSTM [46] | - | 79.00
Ours | FF++ (LQ) | 80.21
Ours | FF++ (HQ) | 82.19
Table 7: Comparison to other ViT-based deepfake detection methods. The results are reported on FaceForensics++. Methods | Training Dataset | AUC
---|---|---
ViXNet [19] | FF++ | 74.78
Conv ViT [57] | FF++ | 71.80
UIA-ViT [67] | FF++ | 99.33
Ours | FF++ (HQ) | 99.67

###  4.5 Ablation Studies

In this section, we investigate the contributions of different components of
our method toward facial forgery detection. As the first step, we remove the
temporal consistency encoder and present the results in Tables 8, 9, and 10,
for FaceForensics++ (LQ), FaceForensics++ (HQ), and Celeb-DFv2, respectively.
When comparing these results to the performance of our original method (also
presented in each table), we observe that removing the temporal consistency
encoder results in performance drops of 1.2% to 2.9%. This indicates the
importance of learning additional temporal information through optical flow
which may be difficult for the model to learn without explicit supervision.

Next, we examine the impact using simple score-level fusion in our model. To
this end, we adopt two strategies instead. First, we use the joint learning
approach proposed in [3], where a single pre-trained encoder accepts patches
from both the RGB and optical flow modalities simultaneously. Second, instead
of score-level fusion, we use feature-level fusion immediately after the
embeddings are obtained from the spatial and temporal consistency encoders.
The results for both experiments are presented in Tables 8, 9, and 10, for the
three datasets, respectively. We observe that while feature-level fusion
achieves results closer to ours in comparison to joint learning, our method
still obtains superior results to both these strategies.

Lastly we illustrate the Receiver Operating Characteristic (ROC) curves for
our method (depicted in blue) and the three ablated variants discussed above,
in Figure 4. These results are obtained on the FaceForensics++ (LQ),
demonstrated in Table 8. We observe that the true positive rates are generally
higher than the model variants across different false positive rate regions,
except for the version where temporal consistency is not used, which shows
comparable results in true positive rates for high false positive regions.
This indicates that the temporal consistency component is highly effective in
reducing the number of false alarms.

Table 8: Ablation experiments on FaceForensics++ (LQ). Technique | ACC | AUC
---|---|---
Proposed | 97.79% | 98.45%
w/o temporal consistency | 96.51% | 97.03%
w/ joint learning [3] | 95.02% | 97.05%
w/ feature-level fusion | 96.01% | 97.10%
Table 9: Ablation experiments on FaceForensics++ (HQ). Technique | ACC | AUC
---|---|---
Proposed | 98.19% | 99.67%
w/o temporal consistency | 96.90% | 97.35%
w/ joint learning [3] | 95.81% | 97.58%
w/ feature-level fusion | 98.01% | 99.09%
Table 10: Ablation experiments on CelebDFv2. Technique | ACC | AUC
---|---|---
Proposed | 98.00% | 98.90%
w/o temporal consistency | 95.08% | 97.17%
w/ joint learning [3] | 95.06% | 96.55%
w/ feature-level fusion | 96.81% | 98.10%

###  4.6 Limitations

We identify several limitations in our work. First, while the integration of
temporal information through optical flow improves the detection performance
of our method, it also increases the computational complexity of the system,
potentially limiting its real-time applicability. Second, the proposed
approach may not be robust to novel deepfake techniques or attacks targeting
the identified limitations. Therefore, the effectiveness and generalizability
of our proposed method will need to be validated further on new datasets and
deepfake scenarios as they become available in the future. Lastly, we observe
that the temporal consistency contributed mostly to the reduction of false
positive detection. While this can indeed be valuable for practical
applications, designing additional components to further enhance the true
positive detection is also of critical importance.

Figure 4: Receiver operating characteristic (ROC) curves for our proposed
method (blue) and three ablations on the FaceForensics++ (LQ) dataset.

##  5 Conclusion

We introduce MASDT, a learning framework for enhanced deepfake detection. Our
method consists of two components, spatial and temporal consistency learning.
The model follows a sequential two-step process. Initially, it employs self-
supervised pre-training where both spatial learning and temporal consistency
learning components engage in data reconstruction. Spatial learning makes use
of a masked self-supervised auto-encoder to derive robust spatial features
from partial facial images, while temporal consistency learning employs a
similar auto-encoder to extract temporal features from partial optical flow
fields. Subsequently, deepfake detection is executed through fine-tuning of
the encoders of both learning components followed by simple score-level
fusion. Various experiments on FaceForensics++ (LQ and HQ) and CelebDFv2
datasets demonstrate that our approach outperforms state-of-the-art methods by
effectively learning spatial and temporal information, resulting in enhanced
classification performance.

##  6 Acknowledgements

This work was funded by Irdeto Canada Corporation and the Natural Sciences and
Engineering Research Council of Canada (NSERC).

## References

  * [1] Darius Afchar, Vincent Nozick, J. Yamagishi, and I. Echizen.  Mesonet: a compact facial video forgery detection network.  IEEE International Workshop on Information Forensics and Security, pages 1–7, 2018.
  * [2] Irene Amerini, L. Galteri, R. Caldelli, and A. D. Bimbo.  Deepfake video detection through optical flow based cnn.  IEEE/CVF International Conference on Computer Vision Workshop, pages 1205–1207, 2019.
  * [3] Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir.  MultiMAE: Multi-modal multi-task masked autoencoders.  European Conference on Computer Vision, 2022.
  * [4] B. Bayar and Matthew C. Stamm.  A deep learning approach to universal image manipulation detection using a new convolutional layer.  ACM Workshop on Information Hiding and Multimedia Security, 2016\.
  * [5] Zhixi Cai, Kalin Stefanov, Abhinav Dhall, and Munawar Hayat.  Do you really mean that? content driven audio-visual deepfake dataset and multimodal method for temporal forgery localization.  International Conference on Digital Image Computing: Techniques and Applications, pages 1–10, 2022.
  * [6] Roberto Caldelli, Leonardo Galteri, Irene Amerini, and Alberto Del Bimbo.  Optical flow based cnn for detection of unlearnt deepfake manipulations.  Pattern Recognition Letters, 146:31–37, 2021.
  * [7] Junyi Cao, Chao Ma, Taiping Yao, Shen Chen, Shouhong Ding, and Xiaokang Yang.  End-to-end reconstruction-classification learning for face forgery detection.  IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4113–4122, 2022.
  * [8] Liang Chen, Yong Zhang, Yibing Song, Lingqiao Liu, and Jue Wang.  Self-supervised learning of adversarial examples: Towards good generalizations for deepfake detections.  IEEE/CVF conference on computer vision and pattern recognition, 2022\.
  * [9] Shen Chen, Taiping Yao, Yang Chen, Shouhong Ding, Jilin Li, and Rongrong Ji.  Local relation learning for face forgery detection.  AAAI Conference on Artificial Intelligence, pages 1081–1088, 2021\.
  * [10] F. Chollet.  Xception: Deep learning with depthwise separable convolutions.  IEEE Conference on Computer Vision and Pattern Recognition, pages 1800–1807, 2017.
  * [11] Umur Aybars Ciftci, Ilke Demir, and Lijun Yin.  Fakecatcher: Detection of synthetic portrait videos using biological signals.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1–1, 2020.
  * [12] MMFlow Contributors.  MMFlow: Openmmlab optical flow toolbox and benchmark.  https://github.com/open-mmlab/mmflow, 2021.
  * [13] D. Cozzolino, G. Poggi, and L. Verdoliva.  Recasting residual-based local descriptors as convolutional neural networks: an application to image forgery detection.  ACM Workshop on Information Hiding and Multimedia Security, 2017\.
  * [14] Deepfakes.  Faceswap: Deepfakes Software.  https://github.com/deepfakes/faceswap/, accessed 2023.
  * [15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  International Conference on Learning Representations, 2021.
  * [17] J. Fridrich and Jan Kodovský.  Rich models for steganalysis of digital images.  IEEE Transactions on Information Forensics and Security, 7:868–882, 2012.
  * [18] Sheldon Fung, Xuequan Lu, Chao Zhang, and Chang-Tsun Li.  Deepfakeucl: Deepfake detection via unsupervised contrastive learning.  International Joint Conference on Neural Networks, pages 1–8, 2021\.
  * [19] Shreyan Ganguly, Aditya Ganguly, Sk Mohiuddin, Samir Malakar, and Ram Sarkar.  Vixnet: Vision transformer with xception network for deepfakes based video and image forgery detection.  Expert Systems with Applications, 2022.
  * [20] Alexandros Haliassos, Konstantinos Vougioukas, Stavros Petridis, and Maja Pantic.  Lips don’t lie: A generalisable and robust approach to face forgery detection.  IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5039–5049, 2021.
  * [21] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15979–15988, 2022.
  * [22] Young-Jin Heo, Woon-Ha Yeo, and Byung-Gyu Kim.  Deepfake detection algorithm based on improved vision transformer.  Applied Intelligence, 53(7):7512–7527, 2023.
  * [23] Juan Hu, Xin Liao, Difei Gao, Satoshi Tsutsui, Qian Wang, Zheng Qin, and Mike Zheng Shou.  Mover: Mask and recovery based facial part consistency aware method for deepfake video detection.  arXiv preprint arXiv: 2303.01740, 2023.
  * [24] Damian Ibanez, Ruben Fernandez-Beltran, Filiberto Pla, and Naoto Yokoya.  Masked auto-encoding spectral–spatial transformer for hyperspectral image classification.  IEEE Transactions on Geoscience and Remote Sensing, pages 1–14, 2022.
  * [25] Hafsa Ilyas, Ali Javed, and Khalid Mahmood Malik.  Avfakenet: A unified end-to-end dense swin transformer deep learning model for audio-visual deepfakes detection.  Applied Soft Computing, 136:110124, 2023.
  * [26] Zhiquan Jiang, Pengsen Zhao, and Zhonglong Zheng.  Optical flow-attention fusion model for deepfake detection.  International Conference on Algorithms, Computing and Artificial Intelligence, 2023.
  * [27] Hasam Khalid, Shahroz Tariq, Minha Kim, and Simon S Woo.  Fakeavceleb: A novel audio-video multimodal deepfake dataset.  Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track, 2021.
  * [28] Durk P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling.  Semi-supervised learning with deep generative models.  Advances in neural information processing systems, 27, 2014.
  * [29] Gil Knafo and Ohad Fried.  Fakeout: Leveraging out-of-domain self-supervision for multi-modal video deepfake detection.  arXiv preprint arXiv:2212.00773, 2022.
  * [30] Marek Kowalski.  FaceSwap: Deep Learning for Face Swapping.  https://github.com/MarekKowalski/FaceSwap, accessed 2023.
  * [31] Lingzhi Li, Jianmin Bao, Ting Zhang, Hao Yang, Dong Chen, Fang Wen, and B. Guo.  Face x-ray for more general face forgery detection.  IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5000–5009, 2020.
  * [32] Yuezun Li, X. Yang, Pu Sun, H. Qi, and S. Lyu.  Celeb-df: A large-scale challenging dataset for deepfake forensics.  IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3204–3213, 2020.
  * [33] Honggu Liu, Xiaodan Li, Wenbo Zhou, Yuefeng Chen, Yuan He, Hui Xue, Weiming Zhang, and Nenghai Yu.  Spatial-phase shallow learning: rethinking face forgery detection in frequency domain.  IEEE/CVF conference on computer vision and pattern recognition, pages 772–781, 2021.
  * [34] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  International Conference on Computer Vision, 2015.
  * [35] Iacopo Masi, Aditya Killekar, Royston Marian Mascarenhas, Shenoy Pratik Gurudatt, and Wael AbdAlmageed.  Two-branch Recurrent Network for Isolating Deepfakes in Videos.  European Conference on Computer Vision, 2020.
  * [36] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [37] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  Advances in Neural Information Processing Systems, pages 8024–8035, 2019.
  * [38] Yuyang Qian, Guojun Yin, Lu Sheng, Zixuan Chen, and Jing Shao.  Thinking in frequency: Face forgery detection by mining frequency-aware clues.  European Conference on Computer Vision, 2020.
  * [39] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  International Conference on Machine Learning, pages 8748–8763, 2021\.
  * [40] Nicolas Rahmouni, Vincent Nozick, J. Yamagishi, and I. Echizen.  Distinguishing computer graphics from natural images using convolution neural networks.  IEEE Workshop on Information Forensics and Security (WIFS), pages 1–6, 2017.
  * [41] Md Shohel Rana, Mohammad Nur Nobi, Beddhu Murali, and Andrew H Sung.  Deepfake detection: A systematic literature review.  IEEE Access, 2022.
  * [42] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  International Conference on Machine Learning, 32:1278–1286, 2014\.
  * [43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and BjÃ¶rn Ommer.  High-resolution image synthesis with latent diffusion models.  IEEE Conference on Computer Vision and Pattern Recognition, 2022\.
  * [44] Andreas Rössler, Davide Cozzolino, Luisa Verdoliva, Christian Riess, Justus Thies, and Matthias Nießner.  FaceForensics++: Learning to detect manipulated facial images.  International Conference on Computer Vision, 2019.
  * [45] A. Rössler, D. Cozzolino, L. Verdoliva, C. Riess, Justus Thies, and M. Nießner.  Faceforensics++: Learning to detect manipulated facial images.  IEEE/CVF International Conference on Computer Vision, pages 1–11, 2019.
  * [46] Pallabi Saikia, Dhwani Dholaria, Priyanka Yadav, Vaidehi Patel, and Mohendra Roy.  A hybrid cnn-lstm model for video deepfake detection by leveraging optical flow features.  2022 International Joint Conference on Neural Networks, pages 1–7, 2022.
  * [47] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  IEEE International Conference on Computer Vision, pages 618–626, 2017.
  * [48] Liang Shi, Jie Zhang, and Shiguang Shan.  Real face foundation representation learning for generalized deepfake detection.  arXiv preprint arXiv: 2303.08439, 2023.
  * [49] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume.  IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018\.
  * [50] Ke Sun, Hong Liu, Taiping Yao, Xiaoshuai Sun, Shen Chen, Shouhong Ding, and Rongrong Ji.  An information theoretic approach for attention-driven face forgery detection.  European Conference on Computer Vision, pages 111–127, 2022.
  * [51] Ke Sun, Taiping Yao, Shen Chen, Shouhong Ding, Jilin Li, and Rongrong Ji.  Dual contrastive learning for general face forgery detection.  AAAI Conference on Artificial Intelligence, 36:2316–2324, 2022\.
  * [52] Justus Thies, Michael Zollhöfer, and Matthias Nießner.  Deferred neural rendering: Image synthesis using neural textures.  ACM Transactions on Graphics (ToG), 2019.
  * [53] Justus Thies, Michael Zollhofer, Marc Stamminger, Christian Theobalt, and Matthias Nießner.  Face2face: Real-time face capture and reenactment of rgb videos.  IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2387–2395, 2016.
  * [54] Aniruddha Tiwari, Rushit Dave, and Mounika Vanamala.  Leveraging deep learning approaches for deepfake detection: A review.  International Conference on Intelligent Systems, Metaheuristics & Swarm Intelligence, page 12–19, 2023.
  * [55] Ruben Tolosana, Sergio Romero-Tapiador, Julian Fierrez, and Ruben Vera-Rodriguez.  Deepfakes evolution: Analysis of facial regions and fake detection performance.  International Conference on Pattern Recognition, 2021.
  * [56] Suramya Tomar.  Converting video formats with ffmpeg.  Linux Journal, 2006(146):10, 2006.
  * [57] Deressa Wodajo and Solomon Atnafu.  Deepfake video detection using convolutional vision transformer.  arXiv preprint arXiv: 2102.11126, 2021.
  * [58] Lior Wolf, Tal Hassner, and Itay Maoz.  Face recognition in unconstrained videos with matched background similarity.  IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 529–534, 2011.
  * [59] Jungang Xu, Hui Li, and Shilong Zhou.  An overview of deep generative models.  IETE Technical Review, 32(2):131–139, 2015.
  * [60] Ying Xu, Kiran Raja, and Marius Pedersen.  Supervised contrastive learning for generalizable and explainable deepfakes detection.  IEEE/CVF Winter Conference on Applications of Computer Vision, pages 379–389, 2022.
  * [61] Peipeng Yu, Zhihua Xia, Jianwei Fei, and Yujiang Lu.  A survey on deepfake video detection.  IET Biometrics, 10(6):607–624, 2021.
  * [62] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  IEEE/CVF International Conference on Computer Vision, pages 6023–6032, 2019.
  * [63] Hongyi Zhang, Moustapha Cissé, Y. Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  International Conference On Learning Representations, 2017.
  * [64] Jian Zhang, Jiangqun Ni, and Hao Xie.  Deepfake videos detection using self-supervised decoupling network.  IEEE International Conference on Multimedia and Expo, pages 1–6, 2021.
  * [65] Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu.  Multi-attentional deepfake detection.  IEEE/CVF conference on computer vision and pattern recognition, pages 2185–2194, 2021.
  * [66] Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Weiming Zhang, and Nenghai Yu.  Self-supervised transformer for deepfake detection.  arXiv preprint arXiv:2203.01265, 2022.
  * [67] Wanyi Zhuang, Qi Chu, Zhentao Tan, Qiankun Liu, Haojie Yuan, Changtao Miao, Zixiang Luo, and Nenghai Yu.  Uia-vit: Unsupervised inconsistency-aware method based on vision transformer for face forgery detection.  European Conference on Computer Vision, 2022.