# NPVForensics: Jointing Non-critical Phonemes and Visemes for Deepfake
Detection††thanks: This work was supported in part by the National Key R&D
Program of China(No.2021ZD0112100), National Natural Science Foundation of
China (No.U1936212, No.62120106009), and the Beijing Natural Science
Foundation (No.4222014). Corresponding author: Rongrong Ni. ††thanks: Yu Chen,
Yang Yu, Rongrong Ni and Yao Zhao are with the Institute of Information
Science, Beijing Jiaotong University, Beijing 100044, China, and also with the
Beijing Key Laboratory of Advanced Information Science and Network Technology,
Beijing 100044, China (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>yzhao@bjtu.edu.cn). ††thanks: Haoliang Li is with the
Department of Electrical Engineering, City University of Hong Kong (e-mail:
haoliang.li@cityu.edu.hk).

Yu Chen, Yang Yu, Rongrong Ni, , Yao Zhao, , Haoliang Li

###### Abstract

Deepfake technologies empowered by deep learning are rapidly evolving,
creating new security concerns for society. Existing multimodal detection
methods usually capture audio-visual inconsistencies to expose Deepfake
videos. More seriously, the advanced Deepfake technology realizes the audio-
visual calibration of the critical phoneme-viseme regions, achieving a more
realistic tampering effect, which brings new challenges. To address this
problem, we propose a novel Deepfake detection method to mine the correlation
between Non-critical Phonemes and Visemes, termed NPVForensics. Firstly, we
propose the Local Feature Aggregation block with Swin Transformer (LFA-ST) to
construct non-critical phoneme-viseme and corresponding facial feature streams
effectively. Secondly, we design a loss function for the fine-grained motion
of the talking face to measure the evolutionary consistency of non-critical
phoneme-viseme. Next, we design a phoneme-viseme awareness module for cross-
modal feature fusion and representation alignment, so that the modality gap
can be reduced and the intrinsic complementarity of the two modalities can be
better explored. Finally, a self-supervised pre-training strategy is leveraged
to thoroughly learn the audio-visual correspondences in natural videos. In
this manner, our model can be easily adapted to the downstream Deepfake
datasets with fine-tuning. Extensive experiments on existing benchmarks
demonstrate that the proposed approach outperforms state-of-the-art methods.

###### Index Terms:

Multimodal Deepfake detection, critical phoneme-viseme calibration, self-
supervised learning.

##  I Introduction

Recently, face manipulation techniques empowered by deep generative models
have made considerable progress [1, 2, 3, 4], which makes Deepfake media more
lifelike. Realistic Deepfake videos are likely to be utilized by attackers for
malicious purposes, such as creating and distributing fake news, defaming
celebrities, leading to serious security problems. Therefore, there is an
urgent demand to develop effective detection methods to reduce the misuse of
Deepfake.

Figure 1: This figure displays a challenging Lip-Sync forgery video example.
On the left, the first row of video frames is real, while the second row is
Deepfake. We observe that the forger has calibrated the phoneme-viseme pairs
in critical phoneme regions like ‘m’, ‘b’ and ‘p’. Therefore, the difference
between the real and fake visemes is small in the critical phoneme regions.
The right column of video frames is real, uttering the word ‘prepare’,
corresponding to the first column of Mel-spectrograms. The second column of
Mel-spectrograms is generated by $text\ to\ speech$ (TTS) algorithm[5]. We can
observe that the two columns of Mel-spectrograms are similar at the critical
phoneme ‘p’ (marked in red), while they are easily distinguishable at the non-
critical phonemes ‘re’, ‘ar’ and ‘e’ (marked in blue). The phoneme-viseme
correlations in non-critical regions are what we aim to capture in this work.

So far, some Deepfake detection methods have achieved good performance on
specific datasets [6, 7, 8, 9, 10, 11, 12, 13, 14]. These methods are
dedicated to mining visual cues, such as local textures [6, 8], noise
characteristics [10, 9], frequency information [12] etc. LipForensics [15]
utilizes pretrained lip-reading networks to refine the model and learn
embeddings more sensitive to lip motions. In our previous work[16], we
proposed an artifacts-disentangled adversarial learning framework to achieve
accurate Deepfake detection by disentangling the artifacts from irrelevant
information. However, the grim reality is that some forgery methods have
started to tamper with audio and visual information[17], which has more
harmful effects on information dissemination. Therefore, the above approaches
focusing only on visual forgery cue mining have significant limitations.

To address this problem, more and more researchers have started focusing on
the role of audio in Deepfake detection. The papers [18, 19] perform Deepfake
detection based on measuring the consistency between two modalities. However,
these two approaches do not take into account the complementary information of
audio-visual modalities. The human brain integrates the received audio-visual
information to collaborate and complement each other[20]. Inspired by human
cognition, integrating audio-visual data, i.e., exploring audio-visual
complementarity, would benefit the information processing capability and model
robustness of the Deepfake detection. The consistency of audio-visual
modalities provides the basis for multimodal Deepfake detection, and the
intrinsic complementarity between the two makes it possible to enhance the
signal of another modality with the help of one of the modalities. VFD [21]
considers homogeneity between faces and voices in natural videos. However, the
model fails to detect tampered data without identity change, such as removing
glasses. AVoiD-DF[22] designs a dual-stream temporal-spatial encoder and a
multimodal joint decoder for joint learning of intrinsic relations. The
paper[23] proposes a joint audio-visual detection framework, which utilizes
independent audio stream, visual stream and synchronized stream for
prediction. Neither of the above two works considers the modalities gap of
audio-visual features, so even real videos may be misclassified due to the
semantic gap. More importantly, audio-visual forgery techniques are evolving
rapidly. The new forgery methods [24] can achieve more realistic visual-audio
coordination based on the correspondence between critical phonemes and visemes
in words, as shown in Fig. 1. Specifically, this new type of Deepfake
technology can calibrate the visual state of the lips when the phonemes ‘m’,
‘b’ and ‘p’ are pronounced. Existing approaches could not address the problem
of Deepfake video detection with fine-grained manipulation at the word and
phoneme level. To this end, it is a new challenge and an issue that requires
urgent resolution.

Motivated by the above analysis, we discovered that these manipulated methods
are based on critical phoneme-viseme to perform audio-visual correspondence.
The difference between the real video and the manipulated video is
substantially reduced in the critical phoneme-viseme region calibrated by the
forger. Thus, this paper focuses on the non-critical phoneme-viseme regions
that forgers cannot perfectly reshape. Therefore, we propose a novel Deepfake
detection method to mine the correlation of Non-critical Phonemes and Visemes,
termed NPVForensics. Firstly, considering that the features in the non-
critical phoneme-viseme region are relatively subtle and weakly correlated
(compared to the critical phoneme-viseme), we design a Local Feature
Aggregation block with Swin Transformer (LFA-ST) to extract the filtered non-
critical phonemes and its corresponding visemes and facial stream features.
Particularly, LFA-ST not only integrates the local neighboring token signals,
but also captures long-range dependencies to obtain global information and
improve performance. It facilitates the analysis of the evolutionary
consistency of non-critical phoneme-viseme. Secondly, we propose the non-
critical Phoneme-Viseme Awareness Module (PVAM) for joint audio-visual
detection. PVAM contains two sub-modules: Cross Attentional Fusion Module
(CAFM) and Co-correlation Guided Representation Alignment (CGRA). It can
reduce the modality gap and better explore the intrinsic complementarity of
non-critical phoneme and viseme, contributing to Deepfake detection. Finally,
inspired by SST[25], we perform a self-supervised pre-training strategy to
obtain a comprehensive representation of non-critical phonemes and visemes.
The reason is that this work demonstrates that self-supervised pre-training on
relatively large number of real videos can be adequate for learning audio-
visual correspondences. In this way, our model can be easily adapted to
downstream Deepfake datasets with fine-tuning.

Our contributions are as follows:

1) We propose a novel detection method for joint non-critical phonemes and
visemes for Deepfake video with realistic visual-audio forgery, which exploits
the correlation between phonemes at non-critical regions in audio utterances
and lip motions (visemes).

2) To fully explore the evolutionary inconsistency of non-critical phoneme-
viseme, we propose the Local Feature Aggregation Swin Transformer (LFA-ST) to
mine subtle features. The integration of local aggregation modules in Swin
Transformer ensures maximum retention of global and local information of non-
critical phoneme-viseme.

3) To simulate the intrinsic complementary audio-visual relationship, we
designed a Phoneme-viseme Awareness module (PVAM) to perform cross-modal
fusion of phoneme and viseme features. In particular, the Co-correlation
Guided Representation Alignment (CGRA) part can align heterogeneous audio-
visual data and reduce the modality gap.

TABLE I: Description of the 15 critical phonemes targeted by forgers with six classes of corresponding visemes. Classes | Viseme | Phoneme | Examples of Words
---|---|---|---
1 |  | ay, ah | but
2 |  | ey, eh | bet
3 |  | er | bird
4 |  | ch, sh, jh, zh | chase, joke
5 |  | f, v | finally
6 |  | m, b, p, em | moment, parent

##  II Related Work

In this section, we provide a brief review of Deepfakes with Phoneme-Viseme
Coordination, and then we introduce the development of Deepfake detection.

###  II-A Deepfake with Phoneme-Viseme Coordination

Phonemes are perceptually distinct sound units in speech. A viseme is the
visual counterpart of that phoneme and corresponds to the lips shape required
to pronounce it. The paper [26] compared phonemes and visemes, listing
critical phonemes and corresponding visemes. Table I shows a subset of six
visemes for phonemes (a viseme may correspond to more than one phoneme). For
example, in order to pronounce the phonemes ‘m’(mother), ‘b’(brother) and
‘p’(parent), a person needs to pronounce the phonemes with the lips completely
closed. While the specific shapes of various phonemes may depend on other
phonological features such as emphasis or volume, critical phoneme groups such
as ‘m’, ‘b’ and ‘p’ always correspond to specific lip shapes. With that in
mind, the paper [19] performs Deepfake detection based on whether the lips are
completely closed at the phoneme ‘m’, ‘b’ and ‘p’. In contrast, the forger
will reshape those critical phonemes with high certainty [27], like the Text-
to-Video (T2V) dataset[28], containing respective synthesized lip-synced fake
audios, where the critical phoneme positions correspond well to the viseme.

Therefore, forgery datasets with challenging critical phoneme-viseme matching
cannot be solved by these critical phonemes alone. For those neglected non-
critical phonemes, there is a significant ‘forgery loophole’. Based on the
above observations, we propose a novel joint non-critical phoneme-viseme
detection method to perform Deepfake detection in non-critical phoneme-viseme
regions that forgers cannot perfectly reshape.

###  II-B Deepfake Detection

Currently, some researchers focus on mining visual cues that can identify
Deepfakes, such as frequency-aware cues [12], anomalous artifacts in the color
domain [29], and anomalous parsing features from facial recognition networks.
Cao et al. [30] used pixel-level segmentation mapping methods for Deepfake
detection. However, these methods resort to specific forgery patterns of
manipulation techniques in the dataset. Recent work can be summarized in two
solutions: data enhancement [31] or capturing general clues[32] to boost the
generalization ability. Method [31] can uncover significant differences
between real and fake media. Nevertheless, building such datasets is costly.
Paper [33] detects blending boundaries between inner faces and outer faces.
However, these detection methods are sensitive to post-processing operations,
such as compression and blurring. Thus, some studies mine robust high-level
semantic features. For example, LipForensics [15] uses pretrained lipreading
networks to refine the model and learn embeddings more sensitive to lip
motions. However, unlike our proposed method, none of the above detection
methods utilizes audio information. The lip feature extractor is pre-trained
and supervised on Lipreading in the Wild (LRW) dataset [34], which requires a
high annotation cost.

It is noteworthy that additional audio-visual information can improve
perceptual decision-making compared to single visual information. During the
audio-visual process, information from different modalities contributes to the
rapid accumulation of sensory evidence, accelerating and correcting the final
decision. Paper [25] performed the collaborative learning of audio-visual
models under a self-supervised strategy. Visual and audio can provide semantic
information from different perspectives and thus complement each other. To
this end, paper[23, 15, 25] trained their models with these two patterns to
learn rich representations. The paper[19] performs Deepfake detection based on
whether the lips are completely closed at the critical phonemes. However, with
the continuous development of lip-sync technology, forgers reshape critical
phonemes and visemes in words to achieve a visually consistent effect.
Therefore, fine-grained analysis of audio-visual features contributes to
Deepfake detection. In this paper, we observe that due to the complexity of
lip motions, forgers are unable to reshape all phonemes, and there is still
audio-visual dissonance at other locations excluding critical phonemes. That
motivates our research to perform Deepfake detection from the perspective of
jointing audio-visual, to learn the correspondence between non-critical
phonemes and visemes in real videos, and to explore the evolutionary
consistency and intrinsic complementary between non-critical phonemes and
visemes temporally. Moreover, our method improves generalization and
robustness and is more cost-efficient in data annotation.

Figure 2: NPVForensics Pipeline. NPVForensics is organized into two stages.
The first stage is the pre-training stage. We screen the audio waveform, lip
motion stream, and face stream of the real video for non-critical phoneme-
viseme, then input them into Swin Transformer Encoder with Local Feature
Aggregation module to extract sufficient representations. Then the three
streams are fed into Multi-modal Projection Head to calculate the Evolutionary
Consistency Loss (EC Loss). The Multi-modal Common Space Projection is shown
in the right column. In the meantime, the phoneme and viseme features are fed
into the Phoneme-viseme Awareness Module (PVAM) to perform cross-modal fusion
and representation alignment. The second stage is the fine-tuning stage. We
utilize the Deepfake datasets for fine-tuning. Then we concatenate the fusion
features with the facial features as discriminative features for
classification.

##  III Method

Most Deepfake generation methods are unable to perform phoneme-viseme fitting
on all regions but only on critical phonemes. Thus, non-critical phoneme-
viseme inconsistency may be a common feature of Deepfake videos, and such
high-level semantic features are robust to various post-processing of videos.
This section illustrates the proposed Deepfake detection framework based on
self-supervised learning with jointly non-critical phonemes and viesmes,
termed NPVForensics. The pipeline of the framework is shown in Fig. 2. Our
research focuses on mining the correlation between non-critical phonemes and
visemes representations. Firstly, we input the non-critical phonemes, visemes,
and face streams of the real videos into separate backbone Local Feature
Aggregation Swin Transformer (LFA-ST) with modality-specific weights to
extract relatively subtle features between non-critical phoneme-viseme (see
Section III-A). Secondly, we designed EC loss to measure the evolutionary
consistency of non-critical phonemes and visemes (see Section III-B). Then, in
order to reduce the modality gap and better investigate the intrinsic
complementarity of two modalities, we introduce the non-critical phoneme-
viseme awareness module (PVAM), which includes two steps of cross-modal
feature fusion and feature alignment (see Section III-C). Finally, the self-
supervised pre-training strategy and fine-tuning protocol are presented (see
Section III-D).

###  III-A Audio-visual Feature Extraction

We first extract the positions of all phonemes. Google’s Speech-to-Text API
automatically transcribes the audio tracks associated with the video and then
aligns them to the audio using P2FA[35]. The above steps generate a series of
phonemes as well as its start and end times in the input audio/video. We
screen out 15 critical phonemes (including phoneme combinations) shown in
Table I to filter non-critical phonemes and its start and end times.

In order to obtain comprehensive and sufficient features of non-critical
phonemes, visemes and facial streams, we introduced the Swin Transformer [36]
as a feature encoder in Deepfake detection. That is advantageous for
preserving the global features of the non-critical phoneme, viseme, and facial
streams. To maximize the retention of global and local information, we
designed a Local Feature Aggregation (LFA) module and integrated it into the
structure of the Swin Transformer, as shown in Fig. 3(a). The LFA block
consists of lightweight Depthwise and Pointwise Convolutions that can
aggregate local features from feature maps and also can integrate local
aggregation of local approximate token signals in each stage of the
Transformer, as shown in Fig. 3(b).

The LFA block consists of two BatchNorms, two $1\times 1$ Pointwise
Convolutions and one $3\times 3$ Depthwise Convolution. Using effective
Depthwise Convolution, local information is collected from neighboring tokens
(each token corresponds to a specific patch) to maximize the extraction of
detailed cues for non-critical phoneme and viseme features. Pointwise
Convolution aggregates the information between each channel. In addition, the
method is easy to adjust the reference process, requires less initialization,
and can use a larger learning rate.

Figure 3: The overall architecture of Local Feature Aggregation Swin
Transformer (LFA-ST). (a) shows the embedding position of the LFA module in
the Swin Transformer. (b) shows the details of the LFA module.

For each patch, the information is aggregated in a local window of size
$k\times k$ using Pointwise ($1\times 1$) and Depthwise ($3\times 3$)
convolution. The input signal is sequentially subjected to BatchNorm,
Depthwise Convolution and Pointwise convolution operations. It is shown in
Equation (1).

$\operatorname{LFA}(\boldsymbol{X}_{in})=\operatorname{PW}(\operatorname{DW}(\operatorname{BN}(\boldsymbol{X}_{in})).$
(1)

Since the Depthwise Convolution operation independently convolves each channel
of the input layer, it does not effectively use the feature information of
different channels at the same spatial location. Therefore, these feature maps
need to be combined by Pointwise Convolution to generate a new feature map. It
is shown in Equation (2).

$\operatorname{LFA}(\boldsymbol{X}_{mid})=\operatorname{PW}(\operatorname{BN}(\operatorname{LFA}(\boldsymbol{X}_{in}))).$
(2)

Then it enters the FFN module, and the process is shown in Equation (3).

$\boldsymbol{X}_{out}=\operatorname{FFN}(\operatorname{LFA}(\boldsymbol{X}_{mid})).$
(3)

The schematic diagram of the LFA module aggregating local features is shown in
Fig. 4.

Figure 4: The local information from neighbor tokens within the blue area is
first aggregated to the target token.

###  III-B Phoneme-viseme Evolutionary Consistency Mining

LFA-ST operates on each of the three input signals. The audio input is in the
form of a waveform, and the visual modal input consists of 3 channels of RGB
pixels from a video frame. We first define a modality-specific Tokenization
layer, which takes the original signal as input and returns a series of
vectors to be fed into LFA-ST. The original audio waveform is a one-
dimensional input of length $T^{{}^{\prime}}$, after phoneme sieving at
critical positions. We divide it into $\lceil
T^{{}^{\prime}}/t^{{}^{\prime}}\rceil$ segments, each of which contains
$t^{{}^{\prime}}$ waveform amplitudes. We apply linear projections with
learnable weights $\boldsymbol{W}_{p}\in\mathbb{R}^{t^{{}^{\prime}}\times d}$
to all elements in the audio patch to derive a d-dimensional vector
representation. We employ $\lceil T^{{}^{\prime}}/t^{{}^{\prime}}\rceil$
learnable embedding to encode the position of each waveform segment.
Similarly, video clips of size $T\times H\times W$ corresponding to non-
critical phonemes are divided into a series of $\lceil T/t\rceil\cdot\lceil
H/h\rceil\cdot\lceil W/w\rceil$ patches where each patch contains $T\times
H\times W\times 3$ voxels. We apply linear projections to all voxels in each
patch to obtain a d-dimensional vector representation. The projections of
visemes and facial streams are performed by the learnable weights
$\boldsymbol{W}_{v}\in\mathbb{R}^{t\cdot h\cdot w\cdot\times 3}$ and
$\boldsymbol{W}_{f}\in\mathbb{R}^{t\cdot h\cdot w\cdot\times 3}$,
respectively.

We utilize common space projection and contrast learning in common space to
train our network. More specifically, given a $\langle
phonemes,visemes,facial\ stream\rangle$ triplet, we define a semantically
hierarchical common space mapping to compare $\langle phonemes,visemes\rangle$
pairs directly as well as $\langle phonemes,facial\ stream\rangle$ pairs by
cosine similarity. To achieve this, we define multi-level projections. As
shown in the multimodal common space mapping in Fig. 2, where $g_{p}\to pv$
and $g_{v}\to pv$ are projection heads that map the outputs of the Phoneme
Transformer and the Viseme Transformer, to the phoneme-viseme common space
$S_{pv}$, respectively. Similarly, $g_{f}\to pf$ and $g_{p}\to pf$ project the
output of the Facial Stream Transformer and the video embedding in the
$S_{pv}$ space to the phoneme-facial stream common space $S_{pf}$. The main
intuition behind that hierarchy is that different modalities have different
levels of semantic granularity. Thus we should treat them as inductive biases
in the projection of the common space. We use a linear projection for
$g_{p}\to pv(\cdot)$ and $g_{p}\to pf(\cdot)$, and a two-layer projection with
ReLU for $g_{v}\to pv(\cdot)$ and $g_{f}\to pf(\cdot)$. Batch normalization is
used after each linear layer to simplify the training.

For the purpose of reducing the false positive rate, we perform a segment-wise
‘fine-grained’ analysis of the phoneme features and the viseme features. It is
observed that in Deepfake videos, there are inconsistencies in each segment.
Calculating the total sum of inconsistencies during evolution, the faked video
is much larger than the real one. Therefore, we evaluate the cumulative loss
of phoneme-viseme over time to obtain the total phoneme-viseme evolutionary
consistency loss (${EC}$ Loss). Let $h_{\theta}$ be the distance between two
features, which is obtained by using the feature representation of non-
parametric softmax.

$h_{\theta}(\boldsymbol{X}_{pi}\cdot\boldsymbol{X}_{vi})=\exp\left(\frac{\boldsymbol{X}_{pi}\cdot\boldsymbol{X}_{vi}}{\|\boldsymbol{X}_{pi}\|\cdot\|\boldsymbol{X}_{vi}\|}\cdot\frac{1}{\tau}\right).$
(4)

where $\tau\in\mathbb{R}^{+}$ is a scalar temperature parameter.

The evolutionary consistency contrast loss for each segment is defined as
shown in Equation (5).

$L_{{EC}_{i}}=L_{{EC}_{i}}^{p}+L_{{EC}_{i}}^{v}.$ (5)

The contrast loss of audio modality is:

$L_{{EC}_{i}}^{a}=-\log\leavevmode\resizebox{0.8pt}{}{$\displaystyle\frac{h_{\theta}(\boldsymbol{X}_{pi}\cdot\boldsymbol{X}_{vi})}{\sum_{j\in
I}h_{\theta}(\boldsymbol{X}_{pi}\cdot\boldsymbol{X}_{vj})+\sum_{j\in\boldsymbol{A}{(i)}}h_{\theta}(\boldsymbol{X}_{pi}\cdot\boldsymbol{X}_{vj})}$}.$
(6)

where, $i\epsilon I\equiv\\{1,\cdots,N\\}$ and $\boldsymbol{A}(i)\equiv
I\setminus\\{i\\}$. Similarly, the contrast loss of the viseme stream is given
by Equation (7).

$L_{{EC}_{i}}^{l}=-\log\leavevmode\resizebox{0.8pt}{}{$\displaystyle\frac{h_{\theta}(\boldsymbol{X}_{vi}\cdot\boldsymbol{X}_{pi})}{\sum_{j\in
I}h_{\theta}(\boldsymbol{X}_{vi}\cdot\boldsymbol{X}_{pj})+\sum_{j\in\boldsymbol{A}{(i)}}h_{\theta}(\boldsymbol{X}_{vi}\cdot\boldsymbol{X}_{pj})}$}.$
(7)

Thus, the total EC Loss is calculated as:

$L_{EC}=\sum_{i\epsilon I}L_{{EC}_{i}}.$ (8)

###  III-C Non-critical Phoneme-Viseme Awareness Module

Figure 5: The framework of the Non-critical Phoneme-Viseme Awareness Module
(PVAM). The left part shows the Cross Attention Fusion Module (CAFM).
Moreover, the right part shows the procedure of Co-correlation Guided
Representation Alignment (CGRA).

Generally, the visual modality carries more information relevant to the
Deepfake detection task for a given video sequence. In contrast, the phoneme
modality is more suitable as a piece of complementary information to the
viseme modality. To reliably combine these modalities, we resort to a fusion
mechanism based on cross attention, where the features of each modality focus
on itself and the other modalities, contributing to capturing the semantic
inter-modal relations between P (Phoneme) and V (Viseme). We drew the joint
modeling of intra and inter-modal relationships into the Non-critical Phoneme-
Viseme Awareness Module (PVAM). The phoneme features and the corresponding
viseme features are fed into the Cross Attention Fusion Module (CAFM) to
explore the intrinsic complementarity between them. To align the heterogeneous
audio-visual patterns, we add a Co-correlation Guided Representation Alignment
(CGRA) step after CAFM. The proposed model is shown in Fig. 5.

Cross Attention Fusion Module. Let $\boldsymbol{X}_{p}$ and
$\boldsymbol{X}_{v}$ denote the phoneme features and viseme features of the
input video respectively, where
$\boldsymbol{X}_{p}=\\{\boldsymbol{X}_{p1},\boldsymbol{X}_{p2},\cdots,\boldsymbol{X}_{pn}\\}\in\mathbb{R}^{d_{p}\times
S}$ and
$\boldsymbol{X}_{v}=\\{\boldsymbol{X}_{v1},\boldsymbol{X}_{v2},\cdots,\boldsymbol{X}_{vn}\\}\in\mathbb{R}^{d_{v}\times
S}$. $S$ denotes the sequence length, $d_{p}$ and $d_{v}$ denote the
dimensions of the phoneme features and the viseme features.

As shown in Fig. 5, the joint representation of the P-V features is
$\boldsymbol{J}$, which is obtained by concatenating the feature vectors of
$\boldsymbol{X}_{p}$ and $\boldsymbol{X}_{v}$. The joint correlation matrix
$\boldsymbol{M}_{p}$ across the features $\boldsymbol{X}_{p}$ and
$\boldsymbol{J}$ are given by Equation (9).

$\boldsymbol{M}_{p}=\tanh\left(\frac{\boldsymbol{X}_{p}^{T}\boldsymbol{W}_{jp}\boldsymbol{J}}{\sqrt{d}}\right).$
(9)

where $\boldsymbol{W}_{jp}\in\mathbb{R}^{L\times L}$ represents the learnable
weight matrix across P and joint P-V features. Similarly, the joint
correlation matrix of the viseme features is given by Equation (10).

$\boldsymbol{M}_{v}=\tanh\left(\frac{\boldsymbol{X}_{v}^{T}\boldsymbol{W}_{jv}\boldsymbol{J}}{\sqrt{d}}\right).$
(10)

After calculating the joint correlation matrix, the attention weights of P and
V modalities are estimated. Since the joint correlation matrix
($\mathbb{R}^{dp\times d}$) differs from the feature dimension of the
corresponding modalities ($\mathbb{R}^{L\times d_{p}}$), we calculate the
attention weights using different learnable weight matrices corresponding to
the features of the individual modalities. For the P modality, the joint
correlation matrix $\boldsymbol{W}_{p}$ and the corresponding
$\boldsymbol{X}_{p}$ are combined using the learnable weight matrices
$\boldsymbol{W}_{mp}$ and $\boldsymbol{W}_{p}$, respectively.

$\boldsymbol{H}_{p}=\operatorname{ReLU}(\boldsymbol{W}_{p}\boldsymbol{X}_{p}+\boldsymbol{W}_{mp}\boldsymbol{M}_{p}^{T}).$
(11)

where $\boldsymbol{W}_{mp}\in\mathbb{R}^{k\times d}$,
$\boldsymbol{W}_{p}\in\mathbb{R}^{k\times L}$, and $\boldsymbol{H}_{p}$
denotes the attention maps of modality P. Similarly, the attention map
$\boldsymbol{H}_{v}$ of modality V is obtained as Equation (12):

$\boldsymbol{H}_{v}=\operatorname{ReLU}(\boldsymbol{W}_{v}\boldsymbol{X}_{v}+\boldsymbol{W}_{mv}\boldsymbol{M}_{v}^{T}).$
(12)

where $\boldsymbol{W}_{mv}\in\mathbb{R}^{k\times d}$,
$\boldsymbol{W}_{v}\in\mathbb{R}^{k\times L}$.

Finally, the attention features of P and V modalities are obtained as follows:

$\boldsymbol{Att}_{p}=\boldsymbol{W}_{hp}\boldsymbol{H}_{p}+\boldsymbol{X}_{p},$
(13)
$\boldsymbol{Att}_{v}=\boldsymbol{W}_{hv}\boldsymbol{H}_{v}+\boldsymbol{X}_{v}.$
(14)

where $\boldsymbol{W}_{hp}\in\mathbb{R}^{k\times L}$ and
$\boldsymbol{W}_{hv}\in\mathbb{R}^{k\times L}$ denote the learnable weight
matrices, respectively.

Connecting $\boldsymbol{X}_{att,p}$ and $\boldsymbol{X}_{att,v}$ further
yields the P-V feature representation, which is given by Equation (15):

$\boldsymbol{X}_{Att}=\left[\boldsymbol{Att}_{p};\boldsymbol{Att}_{v}\right].$
(15)

Co-correlation Guided Representation Alignment Module. After multimodal
feature fusion, we present a Co-correlation Guided Representation Alignment
Module (CGRA) to align heterogeneous audio-visual multimodal data. CGRA aims
to strengthen the correlation between two modalities as much as possible by
making the correlation matrix between the network outputs close to the
identity matrix as Equation (16). Representation alignment can reduce the
modality gap better to explore the intrinsic complementarity of the two
modalities.

${Loss}_{cor}\triangleq\sum_{i}(1-\boldsymbol{C}_{ii})^{2}+\lambda\sum_{i}\sum_{j\neq
i}\boldsymbol{C}_{ij}^{2}.$ (16)

where $\lambda$ is a positive constant weighing the importance of the first
and second terms of the loss.

$\boldsymbol{C}_{ij}\triangleq\frac{\sum_{b}\boldsymbol{Att}_{p_{i}}^{b}\boldsymbol{Att}_{v_{j}}^{b}}{\sqrt{\sum_{b}\boldsymbol{{Att}_{p_{i}}^{b}}^{2}}\sqrt{\sum_{b}\boldsymbol{{Att}_{v_{j}}^{b}}^{2}}}.$
(17)

where $b$ is the batch samples and $\boldsymbol{C}_{ij}$ is a square matrix of
the same size as the output of the network with values between -1 (perfectly
anti-correlated) and 1 (perfectly correlated).

TABLE II: Fine-tuning the model for videos with different compression rates on the FF++ dataset and testing the performance at that compression rate. The best results are shown in bold. Method | Modality | Pretrain | Video-level ACC(%) | Video-level AUC(%)
---|---|---|---|---
Raw | HQ | LQ | Raw | HQ | LQ
Patch-based[37] | visual | - | 98.9 | 92.6 | 79.1 | 99.8 | 97.1 | 78.3
CNN-aug[38] | visual | - | 98.7 | 96.9 | 81.6 | 99.8 | 99.1 | 86.9
Xception[39] | visual | - | 98.9 | 97.0 | 89.1 | 99.8 | 99.3 | 91.4
Two-branch[10] | visual | - | - | - | - | - | 99.1 | 91.0
Face X-ray[32] | visual | Sup-BI | 99.1 | 78.4 | 34.2 | 99.8 | 97.6 | 77.3
LipForensics[15] | visual | Sup-LRW | 98.9 | 98.0 | 94.2 | 99.4 | 99.7 | 96.1
Self-LipForensics[15] | visual | Sup-LRW | 98.6 | 96.5 | 88.4 | 99.8 | 99.3 | 94.8
RECCE[30] | visual | - | 99.1 | 97.1 | 91.0 | 99.8 | 99.3 | 95.0
Joint A-V Detection[23] | audio-visual | - | 98.6 | 98.0 | 95.8 | 99.3 | 99.0 | 91.4
AVoiD-DF[22] | audio-visual | - | 99.0 | 98.2 | 93.9 | 99.9 | 99.2 | 93.5
SST[25] | audio-visual | Self-Sup | 99.2 | 98.5 | 93.5 | 99.7 | 99.6 | 95.7
VFD[21] | audio-visual | Self-Sup | 98.3 | 94.2 | 90.1 | 99.4 | 96.5 | 89.6
NPVForensics (Ours) | audio-visual | Self-Sup | 99.2 | 98.5 | 95.6 | 99.9 | 99.7 | 96.4

###  III-D Pre-training and Fine-tuning

Pre-training protocol. We utilize three loss functions to update the
parameters of the pre-training model.

1) Based on non-critical phoneme-viseme evolutionary consistency, we design
the evolutionary consistency loss $L_{EC}$.

2) Based on audio-visual homogeneity [21], we use unsupervised contrast loss
InfoNCE between audio and face features.

3) Alignment loss $L_{cor}$ for the CGRA module.

Thus, the total loss in the pre-training stage is calculated as Equation (18):

$L_{pre}=L_{EC}+L_{Info}+L_{cor}.$ (18)

Fine-tuning on Deepfake Dataset. We perform a downstream specific Deepfake
detection task using the pre-trained model. In the fine-tuning and testing
phase, a concat operation is added to the pixel-level global representation
and the fused phoneme-viseme representation to obtain the final discriminative
feature. Fine-tuning is performed using Deepfake datasets with a binary
classifier. The classification uses cross-entropy loss. The cross-entropy loss
is defined as Equation (19):

$L_{ce}=-\frac{1}{N}\sum_{i=1}\leavevmode\resizebox{0.9pt}{}{$\left[y_{i}\cdot\log(P_{i})+(1-y_{i})\cdot\log(1-P_{i})\right]$}.$
(19)

where $y_{i}$ is the one-hot vector of the ground-truth of video, $P_{i}$ is
the predicted probability vector. Finally, the total loss function of the
overall framework in the fine-tuning stage is given by the following Equation
(20):

$L=L_{pre}+wL_{ce}.$ (20)

where $w$ is the scale factor, which we set to 1.

##  IV Experiments

Auxiliary Datasets. We utilize the $VoxCeleb2$ dataset [40] and part of the
$AVSpeech$ dataset [41] for pre-training, with a total of 2500,000 speech
video clips. More than 100 consecutive frames with a trackable face are
included in each video clip. We divide them in an 8:1:1 ratio into training,
validation, and test sets.

Deepfake Datasets. We utilize Deepfake datasets as follows. 1)
FaceForensics++(FF++) [39] consists of 1000 real videos and 4000 fake videos,
generated using two face-swapping methods, Deepfakes and FaceSwap, and two
face reenactment methods, Face2Face and Neural Textures. We utilize the mildly
compressed dataset version (c23) unless stated otherwise. Real audio extracts
the Mel-spectrum directly from the audio in the existing dataset containing
the sound. Fake audio converts the text to audio by the $text\ to\ speech$
(TTS) algorithm[5] and then extracts the converted audio from the Mel-
spectrum. 2) FaceShifter (FSh) [42] and 3) Celeb-DF-v2 [43] with 518 test
videos. 4) DeeperForensics (DFo) [44] are state-of-the-art face-swapping
methods that applied to the real videos of FF++; we use the test videos
according to the FF++ split. 5) Deepfake Detection Challenge Dataset
(DFDC)[45] contains 23,654 real videos and 104,500 fake videos from 960
identities. Since some videos in that dataset are mixed with the camera
holder’s voice, we altered 6000 real videos and 32,200 fake videos as the
fine-tuned set, while 1700 real videos and the corresponding 5810 fake videos
were used as the test set. 6) For FakeAVCeleb [46], 500 real videos and 19,500
fake videos were included, where we used 391 real videos and the corresponding
16,869 fake videos as the fine-tuning set and the rest as the test set. 7) We
analyze lip-sync Deepfakes created through the use of three synthesis
techniques: Audio-to-Video (A2V)[47] and Text-to-Video (T2V) [28] in which
only short utterances are manipulated (T2V-S), and Text-to-Video in which
longer utterances are manipulated (T2V-L). We selected over 450 real and 1940
fake videos as the fine-tuning set, while 100 real videos and the
corresponding 428 fake videos were used as the test set.

Evaluation metrics. Following the evaluation metrics from previous work on
Deepfake detection, we evaluate the detection performance using video-level
ROC-AUC scores (AUC) and binary classification accuracy (ACC).

Implementation Details. We implemented our model using the Pytorch toolkit.
The basic learning rate of the finetune model is $1\times{10}^{-3}$ (for the
finetune exclusive module) and $5\times{10}^{-6}$ (for the pretrain and
finetune shared module) using the AdamW optimizer. Parameters for pre-training
and fine-tuning are updated, and the respective learning rates decay gradually
with the training process, with weights decaying by $1\times{10}^{-2}$. The
model is trained on four NVIDIA RTX 3090 Ti with a mini-batch size of 64. We
used 12 Transformer blocks with 8 head self-attention in the identity feature
extractor. The input face images were tuned to $224\times 224$, and the voice
clips were represented as $80,000\times 32$ sound spectrograms (5 seconds of
video) as the voice clips were divided into 16,000 small windows per second.

###  IV-A Evaluation on FaceForensics++

The FF++ dataset is currently the most widely used dataset for Deepfake
detection. We use four forgery methods to fine-tune our model. In addition,
the FF++ videos have three compression rates, uncompressed (Raw), slightly
compressed (HQ), and heavily compressed (LQ). As the compression level
increases, some texture features are lost and the detection difficulty
increases.

Table II shows the video-level AUC and accuracy at various compression rates
compared to other techniques. Face X-ray [32] generates training data by
blending real faces and being supervised with blending boundary regression.
LipForensics [15] is pretrained in a supervised manner on the Lipreading in
the Wild (LRW) dataset [34]. To parallel compare our self-supervised method to
the supervised one, we also utilize the same network structure and pretrain it
on LRW with the same training strategy of LipForensics in a supervised manner,
the model is denoted as LipForensics*. Joint A-V Detection[23] propose a novel
visual$/$auditory Deepfake joint detection task to exploit the intrinsic
synchronization between the visual and auditory modalities. SST [25] is a
self-supervised Transformer based on audio-visual contrast learning, which
also uses a large amount of real data for pre-training. VFD [21] utilizes
audio-visual homogeneity for self-supervised contrast learning. AVoiD-DF[22]
uses a dual-stream temporal-spatial encoder and a multimodal joint decoder to
learn intrinsic relations between audio and visual.We notice that NPVForensics
performs better than the supervised model. Furthermore, it performs well on
original videos and high compression rate videos.

###  IV-B Cross-manipulation Generalization

The Deepfake detector can detect unseen forgeries in the training set, i.e.,
it has a strong generalization capability, which is vital in real-world
applications. In this experiment, we evaluate the generalization ability of
the cross-manipulation method on the FF++ HQ dataset. We test the results on
one manipulation method and train on the other three data types. The results
are shown in Table III.

TABLE III: Cross-manipulation evaluation. Video-level AUC (%) when testing on each forgery type of FF++ HQ after training on the remaining three. The best results are shown in bold. Method | Train on remaining three | Avg
---|---|---
DF | FS | F2F | NT
Patch-based[37] | 94.0 | 60.5 | 87.3 | 84.8 | 81.7
CNN-aug[38] | 87.5 | 56.3 | 80.1 | 67.8 | 72.9
Xception[39] | 93.9 | 51.2 | 86.8 | 79.7 | 77.9
Face X-ray[32] | 99.7 | 90.1 | 99.2 | 98.1 | 96.8
Self-LipForensics[15] | 97.8 | 90.5 | 98.0 | 96.9 | 95.8
RECCE[30] | 98.2 | 88.9 | 92.1 | 88.5 | 91.9
Emotions Don’t Lie[48] | 94.5 | 89.3 | 86.9 | 93.7 | 91.1
Joint A-V Detection[23] | 97.7 | 90.5 | 99.7 | 97.3 | 96.3
SST[25] | 94.5 | 91.9 | 98.3 | 96.4 | 95.3
VFD[21] | 96.2 | 86.3 | 89.6 | 94.2 | 91.6
AVoiD-DF[22] | 97.3 | 94.7 | 94.5 | 95.1 | 95.4
NPVForensics (Ours) | 99.8 | 96.2 | 99.4 | 98.6 | 98.5

Our method outperforms methods such as Xception and RECCE that use only visual
modality, demonstrating the effectiveness of introducing audio modality to
assist visual modality for Deepfake detection. NPVForensics achieves better
results compared to methods using audio-visual multimodality. It suggests that
NPVForensics can effectively capture fine-grained audio-visual inconsistencies
and explore correlations in non-critical phoneme-viseme regions with better
generalization.

###  IV-C Generalization Across Datasets

In realistic scenarios, the forgeries distribution is diverse, and different
Deepfake datasets have large domain gaps. Therefore, domain generalization
ability is an essential criterion for Deepfake detection models. In this
section, we fine-tune our method on the FF++ dataset and report the AUC scores
for tests performed on Celeb-DF-v2 (CDF), DFDC, FaceShifter HQ (FSh), and
DeeperForensics (DFo), respectively.

In Table IV, our approach achieves advanced performance in generalizing to
CDF, FSh and DFo datasets. The performance of FSh and DF is undoubtedly
superior to CDF and DFDC, with AUCs of 98.2% and 98.6%, respectively. It is
likely because both datasets and FF++ use the same original video.
NPVForensics achieves an average AUC of 92.5% on the four datasets,
outperforming the Face X-ray[32] and Lipforensics[15] by 11.3% and 5.4%,
respectively. Multimodal Deepfake detection methods generally obtained better
results than other unimodal methods. NPVForensics outperforms AVoiD-DF[22] in
the cross-dataset evaluation by 2.3%. There are two possible reasons to
explain. One is that the self-supervised learning-based approach learns
adequate audio-visual representations in a large number of real videos, which
can significantly improve the generalization of the model; the second is that
our approach focuses more on the regions that the forger overlooks. Compared
with the AVoiD-DF[22], our method is more likely to tap into subtle
inconsistencies and intrinsic complementarity features and thus achieves
promising results on unseen Deepfakes.

TABLE IV: Generalizability across datasets. Video-level AUC (%) tests on CDF, DFDC, FaceShifter HQ (FSh), and DFo. Here, The best results are shown in bold (Train on FF++). Method | CDF | DFDC | FSh | DFo | Avg
---|---|---|---|---|---
Xception[39] | 73.7 | 70.9 | 72.0 | 84.5 | 75.3
Face X-ray[32] | 79.5 | 65.5 | 92.8 | 86.8 | 81.2
CNN-GRU[49] | 69.8 | 68.9 | 80.8 | 74.1 | 73.4
LipForensics[15] | 82.4 | 73.5 | 95.9 | 96.6 | 87.1
Self-LipForensics[15] | 83.6 | 73.6 | 96.1 | 97.0 | 87.6
RECCE[30] | 78.5 | 69.1 | 84.9 | 88.7 | 80.4
Emotions Don’t Lie[48] | 82.1 | 84.4 | 96.0 | 96.3 | 89.7
Joint A-V Detection[23] | 83.9 | 76.5 | 96.9 | 95.8 | 88.3
SST[25] | 84.2 | 74.5 | 97.8 | 97.3 | 88.4
VFD[21] | 80.7 | 85.1 | 85.9 | 84.3 | 84.0
AVoiD-DF[22] | 86.2 | 82.3 | 95.7 | 96.5 | 90.2
NPVForensics (Ours) | 88.4 | 84.9 | 98.2 | 98.6 | 92.5
TABLE V: Generalizability across datasets. Video-level AUC (%) tests on A2V, T2V-L and T2V-S. The best results are shown in bold (Train on FF++). Method | A2V | T2V-L | T2V-S | Avg
---|---|---|---|---
Xception[39] | 81.6 | 63.9 | 75.3 | 73.6
Face X-ray[32] | 77.2 | 60.3 | 81.4 | 73.0
CNN-GRU[49] | 80.1 | 66.3 | 82.2 | 76.2
LipForensics[15] | 84.4 | 74.2 | 86.6 | 81.7
Self-LipForensics[15] | 84.6 | 74.3 | 87.1 | 82.0
RECCE[30] | 69.1 | 66.3 | 78.2 | 71.2
Joint A-V Detection[23] | 80.5 | 69.8 | 80.2 | 76.8
SST[25] | 89.6 | 77.4 | 86.7 | 84.6
VFD[21] | 67.8 | 60.9 | 65.4 | 64.7
PVM[19] | 94.6 | 79.7 | 74.1 | 82.8
AVoiD-DF[22] | 85.1 | 78.3 | 84.4 | 82.6
NPVForensics (Ours) | 97.1 | 81.3 | 91.9 | 90.1
TABLE VI: Robustness to common degradations. The models are fine-tuning with FakeAVCeleb datasets. AUC(%) scores at five intensity levels are averaged for each degradation type. The best results are shown in bold. Method | Saturation | Contrast | Block | Noise | Blur | Pixel | Compress | Avg
---|---|---|---|---|---|---|---|---
Patch-based[37] | 84.3 | 74.2 | 98.8 | 50.0 | 54.4 | 56.7 | 53.4 | 67.5
CNN-aug[38] | 99.3 | 99.1 | 95.2 | 54.7 | 76.5 | 91.2 | 72.5 | 84.1
Xception[39] | 99.3 | 98.6 | 98.7 | 53.8 | 60.2 | 74.2 | 62.1 | 77.3
Face X-ray[32] | 97.6 | 88.5 | 99.1 | 49.8 | 63.8 | 88.6 | 55.2 | 77.5
LipForensics[15] | 99.9 | 99.6 | 87.4 | 73.8 | 93.1 | 95.6 | 95.6 | 92.5
Self-LipForensics[15] | 99.8 | 99.0 | 87.1 | 73.7 | 94.2 | 93.7 | 93.2 | 91.6
SST[25] | 98.3 | 98.2 | 89.3 | 64.8 | 94.0 | 89.2 | 86.7 | 88.6
RECCE[30] | 91.7 | 91.2 | 88.4 | 65.3 | 87.3 | 83.9 | 89.7 | 85.4
NPVForensics (Ours) | 99.7 | 99.2 | 97.9 | 79.7 | 94.3 | 96.2 | 97.6 | 94.9
Figure 6: The vertical axis represents the video-level AUC score, and the
horizontal axis represents the severity level of perturbations. The four
perturbations are Gaussian blur, block-level distortion, H264 compression, and
pixelation decomposition. We compare the robustness of the model with
different pre-training strategies.

Table V shows the generalization performance on the three lip-sync Deepfake
datasets: A2V, T2V-L and T2V-S. These datasets have a more realistic tampering
effect by considering the critical phoneme-viseme coordination during the
generation process and thus convey a challenge for the existing Deepfake
detection techniques.

As shown in Table V, our method has good detection performance for Deepfakes
generated by such synthetic techniques, with an average AUC exceeding SST[25]
by 5.5% and higher than PVM[19] by 7.3%. The reason is that those novel
forgeries are calibrated at the critical phoneme-viseme region, so when
PVM[19] and SST[19] extracts salient features for Deepfake detection, it
extracts the calibrated critical phoneme-viseme features, leading to a certain
degree of misclassification. We note that PVM[19] outperforms T2V-L over T2V-S
due to the higher ratio of calibrated critical phonemes and visemes in T2V-S,
and thus T2V-S is more challenging to detect for PVM[19]. NPVForensics average
AUC higher than AVoiD-DF[22] by 7.5%, demonstrating that NPVForensics
extraction of non-critical phoneme-viseme has significant advantages for
detecting lip-sync-driven Deepfakes. Our method is an effective detection tool
to tackle the more realistic Deepfake videos in the future.

###  IV-D Robustness Evaluation

In addition to possessing the capability of good generalization, the detector
should be robust to common video degradation. We follow LipForensics[15] to
evaluate robustness to unseen perturbations. As in [15], we finetune on
FakeAVCeleb using grayscale clips without any augmentation other than
horizontal flipping and random cropping to avoid any intersection between
training and testing temporal perturbations. The perturbations proposed in our
paper are variations of saturation, contrast, block masking, Gaussian noise,
blurring, pixelation, and video compression. Each perturbation type is applied
to five different intensity levels. Table VI shows the average AUC for all
intensity levels for each degradation type. When comparing the results with
state-of-the-art methods, NPVForensics outperforms LipForensics and texture-
based techniques. It is due to the fact that we pretrained the feature
extractor LFA-ST on a significant amount of real videos to mine the essential
distinctions between real videos and Deepfake videos. The evolutionary
consistency and intrinsic complementarity of non-critical phoneme-viseme can
provide sufficient discrimination basis for detection. Additionally, it shows
that high-level semantic features are more robust to perturbations.

We focus on the robustness of the model under different pretraining
strategies. The three pretraining strategies are Train from Scratch, Self-
supervised pretrain and Supervised pretrain. Degradations of the definition
include adding block-wise distortions, Gaussian blurring, JPEG compression,
and H264 video compression. We can draw conclusion from Fig. 6 that our self-
supervised pretraining strategy has better robustness than supervised
pretraining.

###  IV-E Visualization

Figure 7: Visualization of the manipulated regions of sequential video frames
from FakeAVCeleb. The manipulated regions are concentrated in the lips area,
where warmer color indicates a higher probability of forgery.

The following example is a Deepfake video from FakeAVCeleb, generated by Lip-
sync technology. To explore which region our method focuses on when dealing
with unseen forgeries, we employ Grad-CAM to generate a heat map, as shown in
Fig. 7. Its identity information is not tampered with; only the visual
information is tampered with to match the fake audio. It is misclassified as
real video in methods such as RECCE[30], Self-LipForensics[15], VFD[21],
SST[25] and AVoiD-DF[22]. The reasons are that the faked video has few visual
artifacts, the identity information is not changed, and the critical phoneme-
viseme is calibrated. Our method accurately finds the forgery region and
effectively exploits the correlation of non-critical phoneme-viseme regions
for forgery detection. As shown in Fig. 7, our method focuses on the local
region (lip movement) manipulated by the forger.

###  IV-F Ablation Experiments

TABLE VII: Framework ablation. Accuracy scores (%) of FaceShifter HQ (FSh), DeeperForensics (DFo) and FakeAVCeleb after training on the FF++ dataset. The best results are shown in bold. Method | FakeAVCeleb (audio-visual) | Avg
---|---|---
Real-Fake | Fake-Fake | Fake-Real |
w/o visual stream | - | 87.7 | 87.9 | -
w/o facial stream | 93.5 | 95.7 | 95.9 | 95.0
w/o phoneme stream | 89.9 | 93.3 | 93.6 | 92.3
w/o LFA, only ST | 94.8 | 95.3 | 93.9 | 94.7
ViT | 95.0 | 94.9 | 93.9 | 94.6
w/o CAFM | 91.7 | 92.8 | 94.3 | 92.9
w/o CGRA | 92.5 | 93.3 | 94.1 | 93.3
NPVForensics (Ours) | 96.7 | 97.8 | 96.3 | 96.9

Framework Ablation. In this section, we introduce ablation to understand the
factors contributing to the performance of NPVForensics. In Table VII, we
ablate different components and check its generalization performance on FSh,
DFo and FakeAVCeleb after training on FF++. We make the following
observations. The performance decreases slightly when there is no facial
stream. It demonstrates that pixel-level features also play a positive role in
Deepfake detection. Then, eliminating the audio stream, the performance on
FakeAVCeleb decreased by 4.6%. The reason is that part of the data in this
dataset is only manipulated with the lips region. Almost no visual artifacts
result in the performance degradation of only visual features. That
demonstrates that audio-visual inconsistency is a reasonable basis for
Deepfake detection. Without the CAFM module, the AUC decreases by about 4.0%.
Without CGRA to perform feature alignment, the AUC decreases by about 3.6%. It
suggests that the cross attention fusion module with the co-covariate guided
representation alignment module is meaningful. Notably, the performance
comparison of ViT, ST, and LFA-ST highlights the effectiveness of our proposed
LFA module. The addition of the LFA module aggregates local features, combined
with the global features extracted from the Swin Transformer, helps to mine
the weak correlation of non-critical phonemes and visemes (relative to
critical phoneme-viseme) and facilitates the detection of fine-grained forged
talking faces.

Effect of Pretraining Dataset Scale.

Figure 8: The relation between pretrained data scale and AUC score. The above
chart shows the relation between the pre-training data scale and the test AUC
on the FF++ HQ dataset. The below chart shows the relation between the
pretraining data scale and the average cross-dataset AUC fine-tuning on the
FF++ dataset.

One of the advantages of self-supervised learning is that it utilizes a large
number of easy-to-collect real face videos for pre-training without the
necessity of expensive manual annotations. Comparatively speaking, the
labelled dataset LRW for the lip reading task contains about 500,000 video
clips of 500 words with precise boundary annotations. It spends a lot of
effort on annotation. Therefore, Deepfake detection using self-supervised
learning would be an attractive direction. In this section, we investigate how
the scale of the pretrained data affects the model’s performance. The
relationship curves are shown in Fig. 8. The horizontal dotted line represents
the AUC score of the pretrained model on the LRW dataset of 500,000 video
clips. The solid line presents the evolution of the AUC score as the scale of
the self-supervised pretrained data increases. The solid red line indicates
the AUC scores for testing on the FF++HQ dataset. The solid blue line
indicates the average cross-dataset AUC scores fine-tuned on the FF++ dataset.

The curves in Fig. 8 validate that models pre-trained with more real videos
can improve accuracy within and across datasets. When pretrained with 320,000
video clips, the performance outperforms the same network pre-trained with
500,000 labelled video clips from the LRW dataset. In conclusion, we
demonstrate that larger-scale pre-training data helps improve the performance
of our model. However, more extensive training data inevitably leads to higher
computational resource requirements, which may be a limitation.

##  V Conclusion

This paper proposes a novel cue for effective Deepfake detection that
concentrates on mining phoneme-viseme correlations by focusing on non-critical
phoneme-viseme regions that forgers cannot perfectly calibrate. To achieve
this goal, we propose a new framework termed NPVForensics. Considering that
the features of non-critical phoneme regions are relatively subtle, we
designed LFA-ST as the backbone for extracting features so as to maximize the
retention of global and local information. We utilize the phoneme and viseme
features for phoneme-viseme evolutionary consistency analysis. In addition, to
effectively exploit the intrinsic complementarity between audio and visual
modalities, we design a phoneme-viseme awareness module to fuse audio-visual
features and introduce a co-correlation guided representation alignment module
to reduce the modalities gap. We employ a self-supervised strategy to pretrain
on large-scale real videos and fine-tune the framework on Deepfake datasets.
In practical applications, the accuracy and generalization of the model can be
improved by scaling up the model and data scale. We have demonstrated that our
method achieves strong cross-manipulation generalization and robustness to
common corruptions.

## References

  * [1] X. Ding, N. Zhu, L. Li, Y. Li, and G. Yang, “Robust localization of interpolated frames by motion-compensated frame interpolation based on an artifact indicated map and tchebichef moments,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 7, pp. 1893–1906, 2018\.
  * [2] A. O. Kwok and S. G. Koh, “Deepfake: a social construction of technology perspective,” _Current Issues in Tourism_ , vol. 24, no. 13, pp. 1798–1802, 2021.
  * [3] A. S. Abdulreda and A. J. Obaid, “A landscape view of deepfake techniques and detection methods,” _International Journal of Nonlinear Analysis and Applications_ , vol. 13, no. 1, pp. 745–755, 2022.
  * [4] J. Yamagishi, X. Wang, M. Todisco, M. Sahidullah, J. Patino, A. Nautsch, X. Liu, K. A. Lee, T. Kinnunen, N. Evans _et al._ , “Asvspoof 2021: accelerating progress in spoofed and deepfake speech detection,” _arXiv preprint arXiv:2109.00537_ , 2021.
  * [5] Y. Jia, Y. Zhang, R. Weiss, Q. Wang, J. Shen, F. Ren, P. Nguyen, R. Pang, I. Lopez Moreno, Y. Wu _et al._ , “Transfer learning from speaker verification to multispeaker text-to-speech synthesis,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [6] S. Chen, S. Tan, B. Li, and J. Huang, “Automatic detection of object-based forgery in advanced video,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 26, no. 11, pp. 2138–2151, 2015.
  * [7] C. Feng, Z. Xu, S. Jia, W. Zhang, and Y. Xu, “Motion-adaptive frame deletion detection for digital video forensics,” _IEEE Transactions on Circuits and Systems for video Technology_ , vol. 27, no. 12, pp. 2543–2554, 2016.
  * [8] L. D’Amiano, D. Cozzolino, G. Poggi, and L. Verdoliva, “A patchmatch-based dense-field algorithm for video copy–move detection and localization,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 29, no. 3, pp. 669–682, 2018.
  * [9] C. Kong, B. Chen, H. Li, S. Wang, A. Rocha, and S. Kwong, “Detect and locate: Exposing face manipulation by semantic-and noise-level telltales,” _IEEE Transactions on Information Forensics and Security_ , vol. 17, pp. 1741–1756, 2022.
  * [10] I. Masi, A. Killekar, R. M. Mascarenhas, S. P. Gurudatt, and W. AbdAlmageed, “Two-branch recurrent network for isolating deepfakes in videos,” in _European conference on computer vision_. Springer, 2020, pp. 667–684.
  * [11] S. Xie, H. Hu, and Y. Chen, “Facial expression recognition with two-branch disentangled generative adversarial network,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 31, no. 6, pp. 2359–2371, 2020\.
  * [12] Y. Yu, R. Ni, and Y. Zhao, “Mining generalized features for detecting ai-manipulated fake faces,” _arXiv preprint arXiv:2010.14129_ , 2020.
  * [13] J. Hu, X. Liao, W. Wang, and Z. Qin, “Detecting compressed deepfake videos in social networks using frame-temporality two-stream convolutional network,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 32, no. 3, pp. 1089–1102, 2021.
  * [14] M. Aloraini, M. Sharifzadeh, and D. Schonfeld, “Sequential and patch analyses for object removal video forgery detection and localization,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 31, no. 3, pp. 917–930, 2020.
  * [15] A. Haliassos, K. Vougioukas, S. Petridis, and M. Pantic, “Lips don’t lie: A generalisable and robust approach to face forgery detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 5039–5049.
  * [16] X. Li, R. Ni, P. Yang, Z. Fu, and Y. Zhao, “Artifacts-disentangled adversarial learning for deepfake detection,” _IEEE Transactions on Circuits and Systems for Video Technology_ , 2022.
  * [17] H. Khalid, M. Kim, S. Tariq, and S. S. Woo, “Evaluation of an audio-video multimodal deepfake dataset using unimodal and multimodal detectors,” in _Proceedings of the 1st workshop on synthetic multimedia-audiovisual deepfake generation and detection_ , 2021, pp. 7–15.
  * [18] K. Chugh, P. Gupta, A. Dhall, and R. Subramanian, “Not made for each other-audio-visual dissonance-based deepfake detection and localization,” in _Proceedings of the 28th ACM international conference on multimedia_ , 2020, pp. 439–447.
  * [19] S. Agarwal, H. Farid, O. Fried, and M. Agrawala, “Detecting deep-fake videos from phoneme-viseme mismatches,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , 2020, pp. 660–661.
  * [20] M. Bedi, S. Kumar, M. S. Akhtar, and T. Chakraborty, “Multi-modal sarcasm detection and humor classification in code-mixed conversations,” _IEEE Transactions on Affective Computing_ , 2021.
  * [21] H. Cheng, Y. Guo, T. Wang, Q. Li, T. Ye, and L. Nie, “Voice-face homogeneity tells deepfake,” _arXiv preprint arXiv:2203.02195_ , 2022.
  * [22] W. Yang, X. Zhou, Z. Chen, B. Guo, Z. Ba, Z. Xia, X. Cao, and K. Ren, “Avoid-df: Audio-visual joint learning for detecting deepfake,” _IEEE Transactions on Information Forensics and Security_ , 2023.
  * [23] Y. Zhou and S.-N. Lim, “Joint audio-visual deepfake detection,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 14 800–14 809.
  * [24] S. B. Hegde, K. Prajwal, R. Mukhopadhyay, V. P. Namboodiri, and C. Jawahar, “Lip-to-speech synthesis for arbitrary speakers in the wild,” in _Proceedings of the 30th ACM International Conference on Multimedia_ , 2022, pp. 6250–6258.
  * [25] H. Zhao, W. Zhou, D. Chen, W. Zhang, and N. Yu, “Self-supervised transformer for deepfake detection,” _arXiv preprint arXiv:2203.01265_ , 2022.
  * [26] L. Chen, Z. Wu, J. Ling, R. Li, X. Tan, and S. Zhao, “Transformer-s2a: Robust and efficient speech-to-animation,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 7247–7251.
  * [27] A. Vajpayee, Z. Zhang, A. Jain, and V. Bhat, “A simple and efficient method for dubbed audio sync detection using compressive sensing,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023, pp. 565–572.
  * [28] O. Fried, A. Tewari, M. Zollhöfer, A. Finkelstein, E. Shechtman, D. B. Goldman, K. Genova, Z. Jin, C. Theobalt, and M. Agrawala, “Text-based editing of talking-head video,” _ACM Transactions on Graphics (TOG)_ , vol. 38, no. 4, pp. 1–14, 2019.
  * [29] Z. Xia, T. Qiao, M. Xu, N. Zheng, and S. Xie, “Towards deepfake video forensics based on facial textural disparities in multi-color channels,” _Information Sciences_ , vol. 607, pp. 654–669, 2022.
  * [30] J. Cao, C. Ma, T. Yao, S. Chen, S. Ding, and X. Yang, “End-to-end reconstruction-classification learning for face forgery detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 4113–4122.
  * [31] L. Chen, Y. Zhang, Y. Song, L. Liu, and J. Wang, “Self-supervised learning of adversarial example: Towards good generalizations for deepfake detection,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 710–18 719.
  * [32] L. Li, J. Bao, T. Zhang, H. Yang, D. Chen, F. Wen, and B. Guo, “Face x-ray for more general face forgery detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 5001–5010.
  * [33] K. Shiohara and T. Yamasaki, “Detecting deepfakes with self-blended images,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 18 720–18 729.
  * [34] J. S. Chung and A. Zisserman, “Lip reading in the wild,” _Springer, Cham_ , 2016.
  * [35] S. Rubin, F. Berthouzoz, G. J. Mysore, W. Li, and M. Agrawala, “Content-based tools for editing audio stories,” in _Proceedings of the 26th annual ACM symposium on User interface software and technology_ , 2013, pp. 113–122.
  * [36] Z. Liu, J. Ning, Y. Cao, Y. Wei, Z. Zhang, S. Lin, and H. Hu, “Video swin transformer,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 3202–3211.
  * [37] L. Chai, D. Bau, S.-N. Lim, and P. Isola, “What makes fake images detectable? understanding properties that generalize,” in _European conference on computer vision_. Springer, 2020, pp. 103–120.
  * [38] S.-Y. Wang, O. Wang, R. Zhang, A. Owens, and A. A. Efros, “Cnn-generated images are surprisingly easy to spot… for now,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 8695–8704.
  * [39] A. Rossler, D. Cozzolino, L. Verdoliva, C. Riess, J. Thies, and M. Nießner, “Faceforensics++: Learning to detect manipulated facial images,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 1–11.
  * [40] J. S. Chung, A. Nagrani, and A. Zisserman, “Voxceleb2: Deep speaker recognition,” _arXiv preprint arXiv:1806.05622_ , 2018.
  * [41] A. Ephrat, I. Mosseri, O. Lang, T. Dekel, K. Wilson, A. Hassidim, W. T. Freeman, and M. Rubinstein, “Looking to listen at the cocktail party: A speaker-independent audio-visual model for speech separation,” _arXiv preprint arXiv:1804.03619_ , 2018.
  * [42] L. Li, J. Bao, H. Yang, D. Chen, and F. Wen, “Faceshifter: Towards high fidelity and occlusion aware face swapping,” _arXiv preprint arXiv:1912.13457_ , 2019.
  * [43] Y. Li, X. Yang, P. Sun, H. Qi, and S. Lyu, “Celeb-df: A large-scale challenging dataset for deepfake forensics,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 3207–3216.
  * [44] L. Jiang, R. Li, W. Wu, C. Qian, and C. C. Loy, “Deeperforensics-1.0: A large-scale dataset for real-world face forgery detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2020, pp. 2889–2898.
  * [45] B. Dolhansky, J. Bitton, B. Pflaum, J. Lu, R. Howes, M. Wang, and C. C. Ferrer, “The deepfake detection challenge (dfdc) dataset,” _arXiv preprint arXiv:2006.07397_ , 2020.
  * [46] H. Khalid, S. Tariq, M. Kim, and S. S. Woo, “Fakeavceleb: a novel audio-video multimodal deepfake dataset,” _arXiv preprint arXiv:2108.05080_ , 2021.
  * [47] S. Suwajanakorn, S. M. Seitz, and I. Kemelmacher-Shlizerman, “Synthesizing obama: learning lip sync from audio,” _ACM Transactions on Graphics (ToG)_ , vol. 36, no. 4, pp. 1–13, 2017.
  * [48] T. Mittal, U. Bhattacharya, R. Chandra, A. Bera, and D. Manocha, “Emotions don’t lie: An audio-visual deepfake detection method using affective cues,” in _Proceedings of the 28th ACM international conference on multimedia_ , 2020, pp. 2823–2832.
  * [49] E. Sabir, J. Cheng, A. Jaiswal, W. AbdAlmageed, I. Masi, and P. Natarajan, “Recurrent convolutional strategies for face manipulation detection in videos,” _Interfaces (GUI)_ , vol. 3, no. 1, pp. 80–87, 2019.