# Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing

Xun Lin1, Shuai Wang1, Rizhao Cai2, Yizhong Liu1, Ying Fu3, Zitong Yu4 ,
Wenzhong Tang1, Alex Kot2
1Beihang University 2Nanyang Technological University 3BIT 4Great Bay
University
{linxun<EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author.

###### Abstract

Face Anti-Spoofing (FAS) is crucial for securing face recognition systems
against presentation attacks. With advancements in sensor manufacture and
multi-modal learning techniques, many multi-modal FAS approaches have emerged.
However, they face challenges in generalizing to unseen attacks and deployment
conditions. These challenges arise from (1) modality unreliability, where some
modality sensors like depth and infrared undergo significant domain shifts in
varying environments, leading to the spread of unreliable information during
cross-modal feature fusion, and (2) modality imbalance, where training overly
relies on a dominant modality hinders the convergence of others, reducing
effectiveness against attack types that are indistinguishable sorely using the
dominant modality. To address modality unreliability, we propose the
Uncertainty-Guided Cross-Adapter (U-Adapter) to recognize unreliably detected
regions within each modality and suppress the impact of unreliable regions on
other modalities. For modality imbalance, we propose a Rebalanced Modality
Gradient Modulation (ReGrad) strategy to rebalance the convergence speed of
all modalities by adaptively adjusting their gradients. Besides, we provide
the first large-scale benchmark for evaluating multi-modal FAS performance
under domain generalization scenarios. Extensive experiments demonstrate that
our method outperforms state-of-the-art methods. Source code and protocols
will be released on https://github.com/OMGGGGG/mmdg.

##  1 Introduction

The remarkable success of deep neural networks has also brought concerns
regarding their security vulnerabilities [54, 55, 30, 29]. Especially for face
recognition (FR) systems, widely used in applications like surveillance and
mobile payment [61], are susceptible to various face presentation attacks,
including printed photos, video replays, and 3D wearable masks [2]. These
vulnerabilities challenge the security of FR systems, limiting their broader
application. As a result, the development of face anti-spoofing (FAS) methods
has become a research hot-spot to enhance the security of FR systems.

Figure 1:  Illustration of DG scenarios in the context of (a) unimodal and (b)
multi-modal. (c) DG performance of SSDG [19] on our Protocol 1 (see Sec. 4.1).
Though fed with more modalities, SSDG performs worse in multi-modal scenarios
compared to unimodal ones. * denotes our re-implemented multi-modal version.

Recently, deep learning has been widely applied in FAS, particularly in
unimodal scenarios based on the RGB spectrum. While these models perform well
in intra-dataset evaluations, their generalizability to unseen attacks and
deployment environments remains limited. Despite the improvement of robustness
via domain generalization (DG) techniques [19, 67, 48, 47], these approaches
still tend to over-fit the attacks presented in source domains.

With the recent progress of multi-modal learning [52, 50] and the development
of sensor manufacture [60], some large-scale multi-modal FAS datasets [33, 15,
64, 43] are proposed, which also drive the advancement in multi-modal FAS [32,
13, 14, 35, 59, 56, 58]. By expanding the input from RGB to multi-modal,
_e.g_., RGB, depth, and infrared, richer spoofing traces can be captured for
detecting various presentation attacks [61]. However, existing multi-modal
methods fail to perform well under unseen deployment conditions. As
illustrated in Fig. 1, though the current methods generalize well in unimodal
scenarios, their performance abnormally declines when extended and applied to
multi-modal FAS. Why using more modalities is not better? We argue that there
are two main reasons: (1) Modality unreliability. Deploying in unseen
environments or with various kinds of models of imaging sensors within the
same modality, especially the depth and infrared shown in Fig. 1 (b), might
introduce significant domain shifts [63]. Such shifts can result in unreliable
multi-modal live/spoof feature extraction [49], misleading mutual modalities
through cross-modal fusion.  (2) Modality imbalance. In some multi-modal
learning scenarios, models tend to overly rely on a dominant modality with the
highest convergence speed (_i.e_. fast modality), which prevents itself from
fully exploiting the other dominated modalities with relatively slower
convergence speeds (_i.e_. slow modalities) [41]. In FAS, modality imbalance
can cause significant negative impacts, especially when the fast modality is
unsuitable or unavailable for detecting attacks in the target domain, relying
on slow modalities that have not adequately converged cannot achieve
satisfactory performance. This study addresses the cross-domain limitations of
multi-modal FAS approaches due to modality unreliability and imbalance. As
illustrated in Fig. 2, we introduce a Multi-Modal Domain Generalized (MMDG)
framework for FAS. Within MMDG, we develop trainable Uncertainty-guided cross-
Adapters (U-Adapter) to fine-tune Vision Transformers (ViT) [9] The U-Adapter
address modality unreliability leveraging the uncertainty of each modality to
suppress the focus on unreliable tokens during the cross-modal fusion. This
prevents the excessive propagation of unreliable spoofing traces which tends
to decrease the model’s discriminative capability. Meanwhile, to solve
modality imbalance, we design Rebalanced modality Gradient modulation strategy
(ReGrad).  Our ReGrad rebalance the convergence speed of all modalities by
dynamically adjusting the backward propagated gradients of all trainable
parameters in U-Adapters based on their conflict degree and convergence speed.
This ensures that all modalities can be fully leveraged to resist various
unseen attacks in the target domain. Our contributions are as follows.

  * •

We propose the MMDG framework to enhance the domain generalizability by
addressing the modality unreliability and imbalance. This is the first
investigation for multi-modal FAS under multi-source DG scenarios.

  * •

Within MMDG, we propose a Uncertainty-Guided Cross-Adapter (U-Adapter) for
ViT-based multi-modal FAS, which can complementarily fuse cross-modal features
and prevents each modality from being influenced by unreliable information
caused by domain shifts.

  * •

We design the Rebalanced Modality Gradient Modulation (ReGrad) for MMDG, a
strategy that can monitor and rebalance the convergence speed of each modality
by adaptively modulating gradients of U-Adapters.

  * •

We build the first large-scale benchmark to evaluate the domain
generalizability of multi-modal FAS approaches. Extensive experiments on this
benchmark demonstrate the superiority and generalizability of MMDG.

##  2 Related Works

###  2.1 Domain Generalization for FAS

In FAS, domain generalization focuses on training a model using multiple
source domains, with the intention that this model can work effectively on
unseen target domains [17, 44]. Previous works verify the effectiveness of
adversarial training [36, 21, 62], asymmetric triplet loss [19, 28] and
controversial learning [47, 37] in developing a shared feature space across
domains. Besides, some studies [48, 67, 66] investigate the style information
of different domains, aiming to learn domain-invariant features by minimizing
distinct style characteristics. Furthermore, meta-learning-based methods [42,
20, 5, 65, 10, 3] are also discussed to simulate domain shifts during
training, fostering the learning of a robust representative feature space to
such variations. More recent works focus on parameter-efficient transfer
learning [6, 17, 46, 7], enabling a pretrained ViT to efficiently adapt to
unseen domains with a lower risk of over-fitting. However, these approaches
are proposed for unimodal (_i.e_. RGB) FAS. Due to the ignorance of modality
unreliability, there is a gap in adapting them to multi-modal scenarios.
Simple multi-modal extensions of these approaches, through early fusion, late
fusion, or even the substitution with multi-modal backbones, still exhibit
limited performance under multi-modal DG scenarios.

###  2.2 Multi-Modal FAS

Multi-modal FAS refers to the use of multiple spectra, _e.g_., RGB, depth, and
infrared, to reveal live and spoofing traces. Due to the complementary
information among different modalities, traces that are undetectable in one
modality can be easily captured in others [35]. To fully exploit information
from each modality, early methods employ channel-wise concatenation of
inputted modalities [14, 40], or develop multiple separate branches for
modality feature extraction followed by late fusion [56, 45, 24, 25]. Recent
works introduce attention-based feature fusion techniques [26, 4] and adaptive
cross-modal loss functions [13] to encourage the extraction of complementary
information among modalities. Meanwhile, cross-modality translation is
discussed [34, 27] to mitigate the semantic gap between different modalities.
To further enhance the practical usability of FAS, Yu et al. [60, 58, 59]
introduce flexible-modal benchmarks that force the use of incomplete
modalities during training or testing. In the context of flexible-modal FAS,
cross-modality attention [35, 32] or multi-modal adapters [59, 58] are
proposed for pretrained ViT to learn modality-agnostic live/spoofing features.
However, the aforementioned methods fail to perform well in DG scenarios. The
lack of sensitivity to modality unreliability and imbalance leads to an
insufficient ability to resist domain shifts. Although previous works
rebalance modalities by modulating gradients [41] or adjusting modality losses
[11], these methods ignore domain shifts and are not suitable for FAS
frameworks with complex cross-modal fusion modules.

Figure 2: Overall structure of our MMDG. It consists of ViT-based backbones
fine-tuned by the proposed U-Adapters with the modality rebalancing strategy
called ReGrad. Each modality is designed with a branch for feature extraction
and enables feature interaction and mutual complementarity with other
modalities. For simplicity, we illustrate the two-modality scenario.

##  3 Proposed MMDG

As shown in Fig. 2, our MMDG consists of ViT-based backbones fine-tuned by the
proposed U-Adapters and ReGrad strategy, which are respectively designed for
solving modality unreliability and imbalance.

Figure 3: (a1)-(a2) Illustration of fine-tuning ViT with proposed U-Adapters,
showcasing the interaction between the RGB (R) and Depth (D) modalities. Note
that only parameters of U-Adapters are trainable. (b) Uncertainty Estimation
Module (UEM) used for recognizing unreliable tokens. (c) Detailed structure of
U-Adapter, which adopts cross-modal fusion and suppresses the interference of
unreliable tokens on other modalities. After fusion, discriminative central
difference information is integrated for fine-grained spoof representation.

###  3.1 Fine-tuning ViT with U-Adapters

In multi-modal FAS, DG scenarios involving significant domain shifts [31],
including unseen face presentations attacks [49], different kinds of imaging
sensors, and noise from low-quality sensors [63], often lead to the extraction
of unreliable live/spoofing features from certain modal inputs. These
unreliable features can propagate during cross-modal fusion, adversely
affecting other modalities. To this end, we aim to identify unreliable local
features within each modality and suppress the interaction of these features
with other modalities, ensuring that each modality utilizes reliable
information from other modalities as complementary traces. Specifically, we
adopt uncertainty estimated via Monte Carlo sampling [12, 22, 38] to evaluate
the feature unreliability in each modality, and based on the uncertainty, we
reduce their weight in cross-modal attention-based feature fusion, minimizing
their negative impact. Inspired by recent success in parameter-efficient
transfer learning (PETL) by introducing adapters into pretrained ViT [17, 6,
7, 59, 58], but distinctively, we propose the U-Adapter for fine-tuning, which
enables cross-modal feature fusion and recognizes as well as suppresses the
propagation of unreliable information from uncertainly detected regions caused
by domain shifts. In Fig. 3, we present the process of fine-tuning ViT with
the U-Adapters, including adopting an Uncertainty Estimation Module (UEM) to
recognize unreliable tokens and employing the U-Adapter for uncertainty-aware
cross-modal fusion. The ViT [9], comprising $n$ transformer blocks each with
Layer Normalization (LN), Multi-Head Self Attention (MHSA), and Multi-Layer
Perceptron (MLP), is fine-tuned at the MLP’s output using our U-Adapter. By
representing our U-Adapter as $\mathcal{A}$ and taking RGB (R), depth (D), and
infrared (I) as the inputted modalities, the fine-tuning process is formulated
as follows.

$\footnotesize\mathbf{x}_{R}^{out}=\mathcal{A}(\mathbf{x}_{D}^{U},\mathbf{x}_{D}^{3},\mathbf{x}_{R}^{3})\\!+\\!\mathcal{A}(\mathbf{x}_{I}^{U},\mathbf{x}_{I}^{3},\mathbf{x}_{R}^{3})+\mathbf{x}_{R}^{3}+\mathbf{x}_{R}^{4},\\\
$ (1)
$\footnotesize\mathbf{x}_{D}^{out}=\mathcal{A}(\mathbf{x}_{R}^{U},\mathbf{x}_{R}^{3},\mathbf{x}_{D}^{3})+\mathbf{x}_{D}^{3}+\mathbf{x}_{D}^{4},$
(2)
$\footnotesize\mathbf{x}_{I}^{out}=\mathcal{A}(\mathbf{x}_{R}^{U},\mathbf{x}_{R}^{3},\mathbf{x}_{I}^{3})+\mathbf{x}_{I}^{3}+\mathbf{x}_{I}^{4},$
(3)

where for $m\in\\{R,D,I\\}$, $\mathbf{x}_{m}^{U}\in\mathbb{R}^{B\times L\times
1}$ are the estimated token uncertainty in the corresponding modality, and
$\mathbf{x}_{m}^{3},\mathbf{x}_{m}^{4}\in\mathbb{R}^{B\times L\times C}$ are
the outputs of the last LN and MLP, respectively. The meaning of other vectors
is illustrated in Fig. 3. To prevent the modalities that experienced
significant domain shifts, _i.e_. depth and infrared, from misleading each
other, we prohibit their direct interaction.

Uncertainty Estimation Module. As discussed above, it is essential to
recognize unreliable modal information for FAS. In deep learning, uncertainty
reflects the level of confidence a model has in its predictions. When the
model is less certain about a prediction, we consider that prediction to be
unreliable [12]. When a model takes images with substantial noise caused by
significant domain shifts as inputs, uncertainty estimation techniques [12]
tend to assign elevated uncertainty values to regions more likely to be
misclassified [31]. One well-known approach is the Bayesian Neural Network
(BNN), widely applied in various vision tasks [16, 18]. BNN focuses on
learning the posterior distribution of the model’s weights rather than output
a single value. Since the posterior distribution is always not tractable [12],
approximate inference methods, such as Monte Carlo dropout (MCD) [51, 8, 38],
are employed for estimation. In this work, we also use MCD to construct the
UEM as a probabilistic representational model, which imposes a Bernoulli
distribution over the weights of a model without introducing extra parameters.
As presented in Fig. 3 (b), we perform MCD instead of the vanilla dropout
after each MHSA, simulating the effect of sampling from the posterior
distribution by randomly dropping out neurons in MHSA. According to Bayesian
probability theory, we consider
$\mathcal{T}=\\{\mathbf{x}^{u_{1}}_{R},\mathbf{x}^{u_{2}}_{R},\cdots,\mathbf{x}^{u_{t}}_{R}\\}$
as empirical samples from the approximate distribution, and use their token-
wise variance
$\mathbf{x}^{U}_{R}=\textrm{Var}(\mathcal{T})\in\mathbb{R}^{B\times L\times
1}$ to reflect their unreliability.

Uncertainty-Guided Cross-Adapter. Different from existing cross-modal fusion
[32, 35], our U-Adapter, guided by the estimated uncertainty, suppresses the
interaction with unreliable tokens from other modalities to ensure the
reliability of the fused features. As shown in Fig. 3 (c), U-Adapter
interactively fuses features from different modalities and suppresses
unreliable information during fusion. For example, in the depth branch (see
Fig. 3 (a2)), U-Adapter takes the output of the second LN from another
modality $\mathbf{x}_{R}^{3}$ as the query for attention-based feature fusion.
The output of the second LN from its own modality $\mathbf{x}_{D}^{3}$ is used
as key and value. Moreover, U-Adapter modulates the query tokens from the
other modality based on their uncertainty. Since malicious presentation
attacks result in subtle traces, we adopt central difference convolution (CDC)
[57] to integrate the local details for ViT and explore fine-grained
difference information from neighbor tokens, thus making the fused features
more discriminative. By ignoring GeLU and vanilla convolution layers, the
U-Adapter is formulated as follows.

$\footnotesize\mathcal{A}(\mathbf{x}_{R}^{U},\\!\mathbf{x}_{R}^{3},\\!\mathbf{x}_{D}^{3})\\!=\\!\mathcal{C}\\!\left(\\!\mathcal{S}(\frac{\textrm{Q}(\mathbf{x}_{R}^{3})\textrm{K}(\mathbf{x}_{D}^{3})^{\intercal}\\!\odot\\!\textrm{U}(\mathbf{x}_{R}^{U})}{\sqrt{n_{k}}})\textrm{V}(\mathbf{x}_{D}^{3}))\\!\right),$
(4)

where $\mathcal{C}(\cdot)$ denotes the CDC layer, $\mathcal{S}(\cdot)$
represents the softmax, and $n_{k}$ is the number of token channels.
$\textrm{Q}(\cdot)$, $\textrm{K}(\cdot)$, and $\textrm{V}(\cdot)$ are the
linear projections corresponding to the query, key, and value, respectively.
$\odot$ is the Hadamard product.
$\textrm{U}(\mathbf{x}_{R}^{U})=\textrm{exp}(-r_{e}\cdot\mathbf{x}_{R}^{U}\times\mathbf{I})$
normalizes the inputted uncertainty using an exponential function, where
$r_{e}$ is a penalty intensity parameter, and
$\mathbf{I}=\left[1,1,\dots,1\right]_{1\times L}$ is adopted to expend the
dimensions of $x_{R}^{U}$ to match the size of $L\times L$.  We show the
meaning of other vectors in Figs. 3 (a1-a2). Note that Eq. (4) only provides
the scenario where RGB tokens are fused into depth in Fig. 3 (c). The pairwise
fusion among other modalities follows similarly.

Figure 4: Illustration of gradient modulation via the proposed ReGrad in
different scenarios: (Row 1) Non-conflicting (a1) and faster modality $j$ (b1)
or $i$ (c1). (Row 2) Conflicting (a2) and faster modality $j$ (b2) or $i$
(c2).

###  3.2 Rebalanced Modality Gradient Modulation

Different modalities are suitable for distinct types of attacks and deployment
conditions, _e.g_., the depth modality being less sensitive to light and more
apt for detecting replay attacks, and the infrared modality excelling in
identifying wearable mask attacks and performing well in low-light conditions.
To cope with various unknown attacks and deployment conditions, we aim to
ensure that each modality can sufficiently converge, thereby fully leveraging
their discriminative capability, rather than over-relying on a specific
modality (_i.e_. fast modality) that shows superior performance in the source
domain.  Current modality balancing methods [41, 11], design for simple late
fusion models, are limited in their applicability to FAS. Late fusion networks
have been shown to underperform in FAS compared to hybrid fusion networks [35,
32] that allow the fusion both before the final classifier and during the
feature extraction. Additionally, the trainable parameters of a late fusion
network are only affected by the gradient of its own modality, whereas those
of a hybrid fusion network are influenced by the gradients of all modalities.
Moreover, gradients from different modalities may even conflict with each
other, making the modulation more difficult. To this end, our ReGrad
adaptively modulates the gradients of all modalities that simultaneously act
on each trainable layer. Moreover, existing modality balancing methods fail to
generalize well while balancing each modality due to the ignorance of domain
shifts. Drawing inspiration from SSDG [19] and PMR [11], we introduce a
single-side prototypical loss to supervise modality branches and monitor their
convergence speed. This loss enables each modality to cluster according to
domain information through internal impetus, which further mitigates the
dominance of faster modalities.

Modulating Gradients for Each Modality. When the optimization direction is
dominated by the gradients of the fast modality, the convergence degree of
slow modalities is suppressed. The discriminative ability of slow modalities
may even be weakened when there is a conflict in the gradient directions
between modalities [53]. Inspired by the studies on solving domain conflicts
[39] and multi-task conflicts [53] via gradient modulation, our ReGrad
adaptively performs gradient modulation based on the conflict and convergence
degree of each modality, making it applicable to complex hybrid fusion
networks.  The criteria of modality convergence speed will be discussed in the
following section.

As shown in Fig. 4 (b1), if modality $i$ converges slower than $j$ and their
gradients $\mathbf{g}_{i}$ and $\mathbf{g}_{j}$ are not conflicting, we
suppress $\mathbf{g}_{j}$ and boost the learning of modality $i$. This
involves decomposing $\mathbf{g}_{j}$ into orthogonal $\mathbf{g}_{j}^{v}$ and
non-orthogonal $\mathbf{g}_{j}^{h}$ components relative to $\mathbf{g}_{i}$.
We drop $\mathbf{g}_{j}^{v}$ to slow down modality $j$ and use
$\mathbf{g}_{j}^{h}$ to accelerate modality $i$. Furthermore, to prevent
unreliable modalities that converge rapidly from misleading other modalities,
we employ the estimated uncertainty of modality $j$ to further constrain
$g_{j}$. Conversely, if $\mathbf{g}_{i}$ conflicts with $\mathbf{g}_{j}$ and
modality $i$ remains a slow modal, it is unreasonable to use any part of
$\mathbf{g}_{j}$ to boost the learning of $\mathbf{g}_{i}$. To prevent
$\mathbf{g}_{j}$ from seriously slowing down the convergence speed of modality
$i$, as shown in Fig. 4 (b2), we remove the non-orthogonal part of
$\mathbf{g}_{j}^{h}$ that is opposite to $\mathbf{g}_{i}$. Meanwhile, we
retain its orthogonal part $\mathbf{g}_{j}^{v}$ to allow modality $j$ to learn
at a slower speed and reduce $\mathbf{g}_{j}$’s interference with
$\mathbf{g}_{i}$. Similarly, the uncertainty-based suppression is adopted.
Cases where the modality $i$ is faster are presented in Figs. 4 (c1)-(c2) and
Eq. (5).

$\footnotesize\textrm{ReGrad}_{2}(\mathbf{g}_{i},\\!\mathbf{g}_{j})\\!=\\!\left\\{\begin{aligned}
&\mathbf{g}_{i}+\frac{\mathbf{g}_{i}\cdot\mathbf{g}_{j}}{\|\mathbf{g}_{i}\|^{2}}\mathbf{g}_{i}\\!\cdot\\!\textrm{U}(u_{j})&\\!,&\,\textrm{in
Fig.~{}\ref{fig:grad} (b1)},\\\
&\mathbf{g}_{i}+(\mathbf{g}_{j}-\frac{\mathbf{g}_{i}\cdot\mathbf{g}_{j}}{\|\mathbf{g}_{i}\|^{2}}\mathbf{g}_{i})\\!\cdot\\!\textrm{U}(u_{j})&\\!,&\,\textrm{in
Fig.~{}\ref{fig:grad} (b2)},\\\
&\frac{\mathbf{g}_{i}\\!\cdot\\!\mathbf{g}_{j}}{\|\mathbf{g}_{j}\|^{2}}\mathbf{g}_{j}\\!\cdot\\!\textrm{U}(u_{i})+\mathbf{g}_{j}&\\!,&\,\textrm{in
Fig.~{}\ref{fig:grad} (c1)},\\\
&(\mathbf{g}_{i}-\frac{\mathbf{g}_{i}\cdot\mathbf{g}_{j}}{\|\mathbf{g}_{j}\|^{2}}\mathbf{g}_{j})\\!\cdot\\!\textrm{U}(u_{i})+\mathbf{g}_{j}&\\!,&\,\textrm{in
Fig.~{}\ref{fig:grad} (c2)},\end{aligned}\right.$ (5)

where $u_{i}$ and $u_{j}$ are respectively the batch-wise mean uncertainty of
the class tokens outputted by the last ViT block of the corresponding
modality, denoting the modality uncertainty.

Table 1: Cross-dataset testing results under the fixed-modal scenarios
(Protocol 1) among CASIA-CeFA (C), PADISI (P), CASIA-SURF (S), and WMCA (W).
DG, MM, and FM are short for domain-generalized, multi-modal, and flexible-
modal, respectively.

Method | Type | CPS $\rightarrow$ W | CPW $\rightarrow$ S | CSW $\rightarrow$ P | PSW $\rightarrow$ C | Average
---|---|---|---|---|---|---
HTER (%) $\downarrow$ | AUC (%) $\uparrow$ | HTER (%) $\downarrow$ | AUC (%) $\uparrow$ | HTER (%) $\downarrow$ | AUC (%) $\uparrow$ | HTER (%) $\downarrow$ | AUC (%) $\uparrow$ | HTER (%) $\downarrow$ | AUC (%) $\uparrow$
SSDG [19] | DG | 26.09 | 82.03 | 28.50 | 75.91 | 41.82 | 60.56 | 40.48 | 62.31 | 34.22 | 70.20
SSAN [48] | DG | 17.73 | 91.69 | 27.94 | 79.04 | 34.49 | 68.85 | 36.43 | 69.29 | 29.15 | 77.22
IADG [67] | DG | 27.02 | 86.50 | 23.04 | 83.11 | 32.06 | 73.83 | 39.24 | 63.68 | 30.34 | 76.78
ViTAF [17] | DG | 20.58 | 85.82 | 29.16 | 77.80 | 30.75 | 73.03 | 39.75 | 63.44 | 30.06 | 75.02
MM-CDCN [56] | MM | 38.92 | 65.39 | 42.93 | 59.79 | 41.38 | 61.51 | 48.14 | 53.71 | 42.84 | 60.10
CMFL [13] | MM | 18.22 | 88.82 | 31.20 | 75.66 | 26.68 | 80.85 | 36.93 | 66.82 | 28.26 | 78.04
ViT + AMA [59] | FM | 17.56 | 88.74 | 27.50 | 80.00 | 21.18 | 85.51 | 47.48 | 55.56 | 28.43 | 77.45
VP-FAS [58] | FM | 16.26 | 91.22 | 24.42 | 81.07 | 21.76 | 85.46 | 39.35 | 66.55 | 25.45 | 81.08
ViT [9] | Baseline | 20.88 | 84.77 | 44.05 | 57.94 | 33.58 | 71.80 | 42.15 | 56.45 | 35.16 | 67.74
MMDG | Ours | 12.79 | 93.83 | 15.32 | 92.86 | 18.95 | 88.64 | 29.93 | 76.52 | 19.25 | 87.96

Single-Side Prototypical Loss. Prototype learning has been successfully
applied in FAS and modality rebalancing. PMR [11] indicates that introducing
prototypes to each modality in multi-modal learning to guide the learning
process can mitigate modality imbalance. However, vanilla prototype learning
is suboptimal under DG scenarios as it neglects the differences in samples
across various domains. Inspired by SSDG [19], we propose the Single-Side
Prototypical (SSP) loss with the concept of asymmetry representation and
prototype learning. Specifically, we introduce prototypes for the live faces
of all domains and the fake faces from each domain. Formulate the domain set
as $\mathcal{D}=\\{d_{\ell},d_{s}^{1},d_{s}^{2},\dots,d_{s}^{N_{s}}\\}$, where
$N_{s}$ is the number of source domains, $d_{\ell}$ denotes the domain
consists of all live faces, and $d_{s}^{i}$ represents domain of spoof faces
from the $i$-th dataset. For modality $m$, the prototype of domain
$d\in\mathcal{D}$ is the center of all samples belonging to $d$ in the feature
space:

$\footnotesize p_{m}^{d}=\frac{1}{N_{d}}\sum_{c_{m}\in d}{c_{m}},$ (6)

where $c_{m}$ is a sample of $d$ in modality $m$, and $N_{d}$ denotes the
total number of samples in $d$. Based on the prototypes of each domain, we aim
to reduce the distance between samples and the prototype of their
corresponding domain and increase the distance from other prototypes, thus
preventing conflicts when samples from different domains are trained together.
Therefore, following the form of cross-entropy, the SSP loss
$\mathcal{L}^{ssp}_{m}$ of modality $m$ is formulated as follows.

$\footnotesize\mathcal{L}_{m}^{ssp}(c_{m},p^{d_{gt}}_{m})=-\textrm{log}\frac{\textrm{exp}(-\textrm{ED}(c_{m},p^{d_{gt}}_{m}))}{\sum_{d\in\mathcal{D}}{\textrm{exp}(-\textrm{ED}(c_{m},p^{d}_{m}))}},$
(7)

where sample $c_{m}$ belongs to domain $d_{gt}$. The function
$\textrm{ED}(\cdot)$ denotes the Euclidean distance of the input sample and
its prototype. Our SSP loss encourages the features of each modality of the
samples to be pulled to the corresponding domain’s prototype. This
optimization of each modality branch is driven by the intrinsic force within
each modality. We mark the modality with lower $\mathcal{L}^{ssp}_{m}$ as a
faster modality. Our final loss is expressed as follows.

$\footnotesize\mathcal{L}^{final}=\mathcal{L}^{ce}+\sum_{m\in\\{R,D,I\\}}{\lambda\cdot\mathcal{L}^{ssp}_{m}},$
(8)

where $L^{ce}$ denotes the cross-entropy loss, and $\lambda$ is used to
balance the trade-off between $\mathcal{L}^{ssp}_{m}$ and $\mathcal{L}^{ce}$.
Here, we utilize $\mathcal{L}^{ce}$ to ensure the model’s performance, while
$\mathcal{L}^{ssp}_{m}$ is employed to balance all modalities and enhance
generalizability. Since $\mathcal{L}^{ssp}_{m}$ is not modality-specific, it
does not undergo gradient modulation, which we perform to $\mathcal{L}^{ce}$
only.

##  4 Experimental Evaluation

###  4.1 Multi-Modal FAS Benchmark in DG Scenarios

Datasets, Protocols, and Performance Metrics. With the increasing popularity
of large-scale foundation models [23], the current trend is towards fully
leveraging multiple and diverse source datasets/domains [1]. While training
with multiple datasets has been widely studied for unimodal FAS [19, 48, 67],
the multi-modal FAS community still remains in the phase of single-dataset
training [35, 32, 13]. Therefore, we propose the first large-scale multi-
source benchmark to evaluate the DG performance of multi-modal FAS. This
benchmark incorporates four commonly used multi-modal datasets: CASIA-CeFA (C)
[33], PADISI-Face (P) [43], CASIA-SURF (S) [64], and WMCA (W) [15], each
featuring a variety of attack types. For each dataset, we use RGB, depth (D),
and infrared (I) modalities as inputs. Following previous works [19], we use
Half Total Error Rate (HTER) and Area Under the Receiver Operating
Characteristic Curve (AUC) for evaluation. We design three protocols to
evaluate the cross-dataset performance under different unseen deployment
conditions, _i.e_., fixed modalities, missing modalities, and limited source
domains. For Protocol 1, inspired by the leave-one-out (LOO) protocols
designed in unimodal DG scenarios [19], we propose four multi-modal LOO sub-
protocols among C, P, S, and W. For example, the sub-protocol CPS
$\rightarrow$ W represents we take C, P, and S as training sets, while W is
testing set. In Protocol 2, for each LOO sub-protocol of Protocol 1, we design
three test-time missing-modal scenarios, _i.e_., D is missing, I is missing,
and both D and I are missing. In Protocol 3, we limit the number of source
domains by proposing two sub-protocols, namely CW $\rightarrow$ PS and PS
$\rightarrow$ CW. Implementation Details. We resize all RGB, depth, and
infrared images to $224\times 224\times 3$. Each modality’s input is divided
into $14\times 14$ patch tokens and a class token is attached, collectively
serving as the input to the ViT. The hidden size of each token is defaultly
set to 768. We employ the Adam optimizer with a learning rate of $5\times
10^{-5}$ and a weight decay of $1\times 10^{-3}$ to train 70 epochs. The batch
size is set to 32. Our classifier is a single-layer fully-connected network
that reduces the dimensionality of the class token outputted by the final ViT
block from 768 to 2. The number $n$ of ViT blocks in MMDG is set to 12. ViT-b
pretrained on ImageNet is adopted as the baseline.

Table 2: Cross-dataset testing results under the test-time missing-modal
scenarios (Protocol 2) among CASIA-CeFA (C), PADISI (P), CASIA-SURF (S), and
WMCA (W). We report the average HTER (%) $\downarrow$ and AUC (%) $\uparrow$
on four sub-protocols, _i.e_., CPS $\rightarrow$ W, CPW $\rightarrow$ S, CSW
$\rightarrow$ P, and PSW $\rightarrow$ C. DG, MM, and FM are short for domain-
generalized, multi-modal, and flexible-modal, respectively.

Method | Type | Missing D | Missing I | Missing D & I | Average
---|---|---|---|---|---
HTER (%) $\,\,\downarrow$ |  AUC (%) $\,\,\uparrow$ |  HTER (%) $\,\,\downarrow$ |  AUC (%) $\,\,\uparrow$ |  HTER (%) $\,\,\downarrow$ |  AUC (%) $\,\,\uparrow$ |  HTER (%) $\,\,\downarrow$ |  AUC (%) $\,\,\uparrow$
SSDG [19] | DG | 38.92 | 65.45 | 37.64 | 66.57 | 39.18 | 65.22 | 38.58 | 65.75
SSAN [48] | DG | 36.77 | 69.21 | 41.20 | 61.92 | 33.52 | 73.38 | 37.16 | 68.17
IADG [67] | DG | 40.72 | 58.72 | 42.17 | 61.83 | 37.50 | 66.90 | 40.13 | 62.49
ViTAF [17] | DG | 34.99 | 73.22 | 35.88 | 69.40 | 35.89 | 69.61 | 35.59 | 70.64
MM-CDCN [56] | MM | 44.90 | 55.35 | 43.60 | 58.38 | 44.54 | 55.08 | 44.35 | 56.27
CMFL [13] | MM | 31.37 | 74.62 | 30.55 | 75.42 | 31.89 | 74.29 | 31.27 | 74.78
ViT + AMA [59] | FM | 29.25 | 77.70 | 32.30 | 74.06 | 31.48 | 75.82 | 31.01 | 75.86
VP-FAS [58] | FM | 29.13 | 78.27 | 29.63 | 77.51 | 30.47 | 76.31 | 29.74 | 77.36
ViT | Baseline | 40.04 | 64.69 | 36.77 | 68.19 | 36.20 | 69.02 | 37.67 | 67.30
MMDG | Ours | 24.89 | 82.39 | 23.39 | 83.82 | 25.26 | 81.86 | 24.51 | 82.69

###  4.2 Cross-dataset Testing

We compare our method on Protocols 1-3 with three categories of FAS methods:
(1) Multi-modal methods, including CMFL [13] and MM-CDCN [56]. (2) Flexible-
modal FAS, such as ViT + AMA [59] and VP-FAS [58]. (3) Domain generalized FAS
_i.e_., SSDG [19], SSAN [48], IADG [67], and ViTAF [17]. For methods not
designed for unimodal FAS, we re-implement them either by employing a
concatenation of multi-modal inputs or by replacing the backbone with our
ViT+U-Adapter or vanilla ViT. All these methods are trained on the proposed
protocols. We automatically use their hyper-parameters as described in the
corresponding reference papers or optimally assign better ones.

Protocol 1: Fixed Modalities. We evaluated the cross-dataset performance of
unimodal DG and multi-modal methods in the fixed modalities scenarios. Since
unimodal DG methods cannot be directly applied to this protocol, we
concatenate the images of the three modalities along the channel dimension. We
then reduce the channel number to 3 using a trainable $1\times 1$ convolution
layer before feeding them to these methods. As shown in the Table 1, our MMDG
achieves the best results. Compared to the second-ranked method, _i.e_. VP-FAS
[58], there is an improvement of over 6% in both HTER and AUC. A higher
performance boost occurs when compared to domain-generalized methods. This
indicates MMDG’s effectiveness in addressing modality unreliability and
imbalance.

Protocol 2: Missing Modalities. As shown in Table 2, our MMDG outperforms
other algorithms in all three scenarios of modality missing during testing.
It’s worth noting that ViT+AMA and VP-FAS are specifically designed for
scenarios of modality missing and have achieved better results compared to
other methods. However, under the DG premise, they did not consider the
modality imbalance and unreliability issues brought by domain shift, resulting
in a difference of over 5% in average HTER and AUC compared to our method.
Another interesting observation is that SSAN, IADG, ViT, and ViT + AMA perform
better in the Missing D & I scenario than in Missing D and Missing I. These
methods lose more modalities but achieve performance improvement, indicating
that unreliable information within the modalities leads to erroneous
detection. This reflects the effectiveness of our approach in suppressing the
interaction of unreliable information between modalities.

Table 3: Cross-dataset testing results under the limited source domain
scenarios (Protocol 3) among CeFA-CeFA (C), PADISI-USC (P), CASIA-SURF (S),
and WMCA (W).

Method | Type |  CW $\rightarrow$ PS |  PS$\rightarrow$ CW
---|---|---|---
HTER (%) | AUC (%) | HTER (%) | AUC (%)
SSDG [19] | DG | 25.34 | 80.17 | 46.98 | 54.29
SSAN [48] | DG | 26.55 | 80.06 | 39.10 | 67.19
IADG [67] | DG | 22.82 | 83.85 | 39.70 | 63.46
ViTAF [17] | DG | 29.64 | 77.36 | 39.93 | 61.31
MM-CDCN [56] | MM | 29.28 | 76.88 | 47.00 | 51.94
CMFL [13] | MM | 31.86 | 72.75 | 39.43 | 63.17
ViT + AMA [59] | FM | 29.25 | 76.89 | 38.06 | 67.64
VP-FAS [58] | FM | 25.90 | 81.79 | 44.37 | 60.83
ViT [9] | Baseline | 42.66 | 57.80 | 42.75 | 60.41
MMDG | Ours | 20.12 | 88.24 | 36.60 | 70.35

Protocol 3: Limited Source Domains. Compared to protocol 1, the limitation of
source domains results in a more significant domain shift. We observe that our
proposed MMDG achieves optimal results in both sub-protocols (see Table 3),
even in the presence of substantial domain shift in PS $\rightarrow$ CW. This
further underscores the effectiveness of our U-Adapter and ReGrad in enhancing
domain generalizability in unseen multi-modal deployment environments.

###  4.3 Ablation Study and Discussion

In this section, we perform ablation analysis on U-Adapter and ReGrad to
verify their individual contribution.

Effectiveness of U-Adapter.

Figure 5: Ablation results on our U-Adapter. We report average HTER
$\downarrow$ and AUC $\uparrow$ on four sub-protocols in Protocol 1.

To validate the effectiveness of the proposed U-Adapter, we compare the
network performance before and after removing the adapter. We also replace the
backbone of existing DG methods (_i.e_., SSDG, SSAN, and IADG) and multi-modal
methods (CMFL) to observe the performance differences when using ViT+U-Adapter
as the backbone versus using ViT alone. As shown in Fig. 5, compared to using
ViT as the backbone, these methods experience a performance boost when
additional U-Adapters are employed. The performance improvement is especially
pronounced when no additional method is used, and when our proposed ReGrad is
integrated. This demonstrates the effectiveness of U-Adapter and its
universality for existing DG methods and multi-modal strategies.

Table 4: Ablation results on the proposed ReGrad. We report the average HTER
and AUC on four sub-protocols in Protocol 1.

Backbone | Method |  HTER (%) |  AUC (%)
---|---|---|---
ViT [9] | - | 31.14 | 74.81
ViT [9] | ReGrad | 30.59 | 75.94
ViT + U-Adapter | SSDG [19] | 26.13 | 80.31
ViT + U-Adapter | SSAN [48] | 26.94 | 79.52
ViT + U-Adapter | IADG [67] | 24.23 | 82.54
ViT + U-Adapter | PMR [11] | 24.54 | 83.14
ViT + U-Adapter | OGM-GE [41] | 29.32 | 76.54
ViT + U-Adapter | - | 27.38 | 78.88
ViT + U-Adapter |  ReGrad (Conflicted-only) | 21.91 | 85.65
ViT + U-Adapter |  ReGrad (Unconflicted-only) | 21.23 | 85.51
ViT + U-Adapter | ReGrad | 19.48 | 87.48

Figure 6: Modality balance degree during training. (a) MMDG w/o ReGrad. (b)
MMDG w/ ReGrad. The red dashed line in each sub-figure represents the variance
of $\mathcal{L}^{ssp}_{R}$, $\mathcal{L}^{ssp}_{D}$, and
$\mathcal{L}^{ssp}_{I}$.

Effectiveness of ReGrad. Here, we try to answer two questions: (1) Whether the
idea to address domain shift through modality rebalancing, is more suitable
compared to existing DG methods? (2) Is our ReGrad more capable of enhancing
the performance of complex cross-modal fusion frameworks like ViT+U-Adapter
compared to existing modality balancing methods? From Table 4, we can observe
that when ViT+U-Adapter is used as the backbone, our ReGrad surpasses existing
DG methods and other modality rebalancing methods. Even for vanilla ViT, our
ReGrad can enhance its performance. This indicates the effectiveness of
addressing domain shifts in unseen scenarios by balancing various modalities.
The results in Table 4 also indicate that when compared to the SoTA modality
balancing methods (_i.e_., OGM-GE and PMR), our ReGrad is more suitable for
ViT + U-Adapter, which involves hybrid feature fusion and complex cross-modal
interactions.

Meanwhile, as illustrated in Fig. 6, the lower variance (right) of
$\mathcal{L}^{ssp}_{R},\mathcal{L}^{ssp}_{D}$, and $\mathcal{L}^{ssp}_{I}$
demonstrates that ReGrad obviously improves the balance degree of MMDG. We
also notice that ReGrad causes slight fluctuations in the loss function
values. This is due to the gradient modulation allowing the model to escape
from local optima, seeking a direction with a higher degree of modality
balance. Despite the fluctuations, MMDG can still converge after about 50
epochs. Additionally, we compare ReGrad with its two variants, which perform
modulation only when modality gradients are conflicted and unconflicted,
respectively. Although these two variants improve performance compared to not
modulating gradients at all, their performance is inferior to modulating
gradients in all situations. This validates the necessity of adaptive
modulation under various scenarios.

Table 5: Impact of estimated uncertainty in U-Adapters and ReGrad. We report
the average HTER (%) $\downarrow$ and AUC (%) $\uparrow$ on four sub-protocols
in Protocol 1.

Using Estimated Uncertainty | Metric Value
---|---
U-Adapter: Eq. (4) | ReGrad: Eq. (5) | HTER (%) $\downarrow$ | AUC (%) $\uparrow$
✕ | ✕ | 28.40 | 78.65
✕ | ✓ | 26.13 | 81.60
✓ | ✕ | 22.75 | 84.84
✓ | ✓ | 19.48 | 87.48

Figure 7: Visual attention and uncertainty maps of our MMDG. Uncertainly
predicted regions are highlighted with red boxes. The redder color indicates
the higher uncertainty or more attention.

Impact of the Uncertainty in U-Adapter and ReGrad. To testify whether the
estimated uncertainty can enhance detection performance, we made a comparison
by removing the parts that use uncertainty for constraints in U-Adapter and
ReGrad. From Table 5, it can be observed that adding uncertainty to constrain
the propagation of unreliable information in U-Adapter and the gradient of
fast modalities in ReGrad both lead to performance improvement. When
uncertainty is added to both modules, HTER and AUC can be further improved by
over 3%.  As shown in Fig. 7, MMDG can identify unreliable regions in depth
(left) and infrared (right) modalities. Although the U-Adapters of these
modalities focus on regions with uncertain predictions, due to our idea of
suppressing uncertain regions, these regions fail to mislead mutual
modalities, enabling the capture of more reliable spoof traces against domain
shifts.  These observations validate the effectiveness of using uncertainty to
constrain unreliable modalities.

Discussion. The results presented in Tables 1-5 and Figs. 5-6 validate the
MMDG’s generalizability by addressing the problems of modality unreliability
and imbalance. This work marks a stride towards practical deployments and can
benefit the FAS community. However, our benchmark is new and MMDG’s
performance can still be improved. Continued efforts on this benchmark are
necessary for further progress. Meanwhile, our method also has space for
improvement, _e.g_., further enhancing the degree of modality balance,
preventing the desecration of training stability, and adopting more advanced
uncertainty estimation techniques to prevent the risk of the model’s over-
confidence.

##  5 Conclusion

In this work, we propose the first multi-modal DG framework, _i.e_. MMDG, for
FAS to enhance generalizability by addressing modality unreliability and
imbalance. To mitigate the interference of unreliable modalities, we propose
U-Adapter to suppress uncertainly predicted regions during cross-modal feature
fusion. Besides, we develop ReGrad to adaptively modulate the gradients of
each modality branch to achieve modality balance, thus preventing an over-
reliance on a dominant modality. We also establish the first large-scale
benchmark for multi-modal FAS under DG scenarios. Extensive experiments on
this benchmark demonstrate MMDG’s effectiveness.

Acknowledgement. This work was supported by the National Key Research and
Development Program of China under Grant no.2022YFB3207700. The support
funding was also from the National Natural Science Foundation of China under
Grant 62272022, 62306061, and U22A2009.

## References

  * Borgeaud et al. [2022] Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, and Laurent Sifre.  Improving language models by retrieving from trillions of tokens.  In _Proceedings of the International Conference on Machine Learning_ , pages 2206–2240, 2022.
  * Cai et al. [2021] Rizhao Cai, Haoliang Li, Shiqi Wang, Changsheng Chen, and Alex C. Kot.  Drl-fas: A novel framework based on deep reinforcement learning for face anti-spoofing.  _IEEE Transactions on Information Forensics and Security_ , 16:937–951, 2021.
  * Cai et al. [2022a] Rizhao Cai, Zhi Li, Renjie Wan, Haoliang Li, Yongjian Hu, and Alex C Kot.  Learning meta pattern for face anti-spoofing.  _IEEE Transactions on Information Forensics and Security_ , 17:1201–1213, 2022a.
  * Cai et al. [2022b] Rizhao Cai, Zhi Li, Renjie Wan, Haoliang Li, Yongjian Hu, and Alex C. Kot.  Learning meta pattern for face anti-spoofing.  _IEEE Transactions on Information Forensics and Security_ , 17:1201–1213, 2022b.
  * Cai et al. [2022c] Rizhao Cai, Zhi Li, Renjie Wan, Haoliang Li, Yongjian Hu, and Alex C. Kot.  Learning meta pattern for face anti-spoofing.  _IEEE Transactions on Information Forensics and Security_ , 17:1201–1213, 2022c.
  * Cai et al. [2023a] Rizhao Cai, Yawen Cui, Zhi Li, Zitong Yu, Haoliang Li, Yongjian Hu, and Alex C. Kot.  Rehearsal-free domain continual face anti-spoofing: Generalize more and forget less.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 2023a.
  * Cai et al. [2023b] Rizhao Cai, Zitong Yu, Chenqi Kong, Haoliang Li, Changsheng Chen, Yongjian Hu, and Alex C. Kot.  S-adapter: Generalizing vision transformer for face anti-spoofing with statistical tokens.  _arXiv_ , abs/2309.04038, 2023b.
  * Carreno et al. [2022] Pamela Carreno, Dana Kulic, and Michael Burke.  Adapting neural models with sequential monte carlo dropout.  In _Conference on Robot Learning_ , pages 1542–1552, 2022.
  * Dosovitskiy et al. [2021] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _Proceedings of the International Conference on Learning Representations_ , 2021.
  * Du et al. [2022] Zhekai Du, Jingjing Li, Lin Zuo, Lei Zhu, and Ke Lu.  Energy-based domain generalization for face anti-spoofing.  In _Proceedings of the ACM International Conference on Multimedia_ , pages 1749–1757, 2022.
  * Fan et al. [2023] Yunfeng Fan, Wenchao Xu, Haozhao Wang, Junxiao Wang, and Song Guo.  PMR: prototypical modal rebalance for multimodal learning.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 20029–20038, 2023.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _Proceedings of the International Conference on Machine Learning_ , pages 1050–1059, 2016.
  * George and Marcel [2021a] Anjith George and Sébastien Marcel.  Cross modal focal loss for RGBD face anti-spoofing.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7882–7891, 2021a.
  * George and Marcel [2021b] Anjith George and Sébastien Marcel.  Learning one class representations for face presentation attack detection using multi-channel convolutional neural networks.  _IEEE Transactions on Information Forensics and Security_ , 16:361–375, 2021b.
  * George et al. [2020] Anjith George, Zohreh Mostaani, David Geissenbuhler, Olegs Nikisins, André Anjos, and Sébastien Marcel.  Biometric face presentation attack detection with multi-channel convolutional neural network.  _IEEE Transactions on Information Forensics and Security_ , 15:42–55, 2020.
  * Guo et al. [2022] Hongji Guo, Hanjing Wang, and Qiang Ji.  Uncertainty-guided probabilistic transformer for complex action recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 20020–20029, 2022.
  * Huang et al. [2022] Hsin-Ping Huang, Deqing Sun, Yaojie Liu, Wen-Sheng Chu, Taihong Xiao, Jinwei Yuan, Hartwig Adam, and Ming-Hsuan Yang.  Adaptive transformers for robust few-shot cross-domain face anti-spoofing.  In _Proceedings of the European Conference on Computer Vision_ , pages 37–54, 2022.
  * Ji et al. [2023] Kaixiang Ji, Feng Chen, Xin Guo, Yadong Xu, Jian Wang, and Jingdong Chen.  Uncertainty-guided learning for improving image manipulation detection.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 22456–22465, 2023.
  * Jia et al. [2020] Yunpei Jia, Jie Zhang, Shiguang Shan, and Xilin Chen.  Single-side domain generalization for face anti-spoofing.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8481–8490, 2020.
  * Jia et al. [2022] Yunpei Jia, Jie Zhang, and Shiguang Shan.  Dual-branch meta-learning network with distribution alignment for face anti-spoofing.  _IEEE Transactions on Information Forensics and Security_ , 17:138–151, 2022.
  * Jiang et al. [2023] Fangling Jiang, Qi Li, Pengcheng Liu, Xiang-Dong Zhou, and Zhenan Sun.  Adversarial learning domain-invariant conditional features for robust face anti-spoofing.  _International Journal of Computer Vision_ , 131(7):1680–1703, 2023.
  * Kendall et al. [2017] Alex Kendall, Vijay Badrinarayanan, and Roberto Cipolla.  Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding.  In _Proceedings of the British Machine Vision Conference_ , 2017.
  * Kirillov et al. [2023] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloé Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick.  Segment anything.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 4015–4026, 2023.
  * Kong et al. [2022] Chenqi Kong, Kexin Zheng, Shiqi Wang, Anderson Rocha, and Haoliang Li.  Beyond the pixel world: a novel acoustic-based face anti-spoofing system for smartphones.  _IEEE Transactions on Information Forensics and Security_ , 17:3238–3253, 2022.
  * Kong et al. [2023] Chenqi Kong, Kexin Zheng, Yibing Liu, Shiqi Wang, Anderson Rocha, and Haoliang Li.  M3fas: An accurate and robust multimodal mobile face anti-spoofing system.  _arXiv preprint arXiv:2301.12831_ , 2023.
  * Li et al. [2023a] Kaicheng Li, Hongyu Yang, Binghui Chen, Pengyu Li, Biao Wang, and Di Huang.  Learning polysemantic spoof trace: A multi-modal disentanglement network for face anti-spoofing.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 1351–1359, 2023a.
  * Li et al. [2023b] Zhi Li, Haoliang Li, Xin Luo, Yongjian Hu, Kwok-Yan Lam, and Alex C. Kot.  Asymmetric modality translation for face presentation attack detection.  _IEEE Transactions on Multimedia_ , 25:62–76, 2023b.
  * Liao et al. [2023] Chen-Hao Liao, Wen-Cheng Chen, Hsuan-Tung Liu, Yi-Ren Yeh, Min-Chun Hu, and Chu-Song Chen.  Domain invariant vision transformer learning for face anti-spoofing.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 6087–6096, 2023.
  * Lin et al. [2023a] Xun Lin, Wenzhong Tang, Shuai Wang, Zitong Yu, Yizhong Liu, Haoran Wang, Ying Fu, and Alex Kot.  Biomedical image splicing detection using uncertainty-guided refinement, 2023a.
  * Lin et al. [2023b] Xun Lin, Shuai Wang, Jiahao Deng, Ying Fu, Xiao Bai, Xinlei Chen, Xiaolei Qu, and Wenzhong Tang.  Image manipulation detection by multiple tampering traces and edge artifact enhancement.  _Pattern Recognition_ , 133:109026, 2023b.
  * Litrico et al. [2023] Mattia Litrico, Alessio Del Bue, and Pietro Morerio.  Guiding pseudo-labels with uncertainty estimation for source-free unsupervised domain adaptation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7640–7650, 2023.
  * Liu and Liang [2022] Ajian Liu and Yanyan Liang.  Ma-vit: Modality-agnostic vision transformers for face anti-spoofing.  In _Proceedings of the International Joint Conference on Artificial Intelligence_ , pages 1180–1186, 2022.
  * Liu et al. [2021a] Ajian Liu, Zichang Tan, Jun Wan, Sergio Escalera, Guodong Guo, and Stan Z. Li.  CASIA-SURF cefa: A benchmark for multi-modal cross-ethnicity face anti-spoofing.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 1178–1186, 2021a.
  * Liu et al. [2021b] Ajian Liu, Zichang Tan, Jun Wan, Yanyan Liang, Zhen Lei, Guodong Guo, and Stan Z. Li.  Face anti-spoofing via adversarial cross-modality translation.  _IEEE Transactions on Information Forensics and Security_ , 16:2759–2772, 2021b.
  * Liu et al. [2023a] Ajian Liu, Zichang Tan, Zitong Yu, Chenxu Zhao, Jun Wan, Yanyan Liang, Zhen Lei, Du Zhang, Stan Z. Li, and Guodong Guo.  FM-ViT: Flexible modal vision transformers for face anti-spoofing.  _IEEE Transactions on Information Forensics and Security_ , 18:4775–4786, 2023a.
  * Liu and Liu [2023] Yaojie Liu and Xiaoming Liu.  Spoof trace disentanglement for generic face anti-spoofing.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(3):3813–3830, 2023.
  * Liu et al. [2023b] Yuchen Liu, Yabo Chen, Mengran Gou, Chun-Ting Huang, Yaoming Wang, Wenrui Dai, and Hongkai Xiong.  Towards unsupervised domain generalization for face anti-spoofing.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 20654–20664, 2023b.
  * Mae et al. [2021] Yuki Mae, Wataru Kumagai, and Takafumi Kanamori.  Uncertainty propagation for dropout-based bayesian neural networks.  _Neural Networks_ , 144:394–406, 2021.
  * Mansilla et al. [2021] Lucas Mansilla, Rodrigo Echeveste, Diego H. Milone, and Enzo Ferrante.  Domain generalization via gradient surgery.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 6610–6618, 2021.
  * Nikisins et al. [2019] Olegs Nikisins, Anjith George, and Sébastien Marcel.  Domain adaptation in multi-channel autoencoder based features for robust face anti-spoofing.  In _Proceedings of the International Conference on Biometrics_ , pages 1–8, 2019.
  * Peng et al. [2022] Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, and Di Hu.  Balanced multimodal learning via on-the-fly gradient modulation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8228–8237, 2022.
  * Qin et al. [2022] Yunxiao Qin, Zitong Yu, Longbin Yan, Zezheng Wang, Chenxu Zhao, and Zhen Lei.  Meta-teacher for face anti-spoofing.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 44(10):6311–6326, 2022.
  * Rostami et al. [2021] Mohammad Rostami, Leonidas Spinoulas, Mohamed E. Hussein, Joe Mathai, and Wael Abd-Almageed.  Detection and continual learning of novel face presentation attacks.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 14831–14840, 2021.
  * Shao et al. [2019] Rui Shao, Xiangyuan Lan, Jiawei Li, and Pong C. Yuen.  Multi-adversarial discriminative deep domain generalization for face presentation attack detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 10023–10031, 2019.
  * Shen et al. [2019] Tao Shen, Yuyu Huang, and Zhijun Tong.  Facebagnet: Bag-of-local-features model for multi-modal face anti-spoofing.  In _IEEE Conference on Computer Vision and Pattern Recognition Workshop_ , pages 1611–1616, 2019.
  * Srivatsan et al. [2023] Koushik Srivatsan, Muzammal Naseer, and Karthik Nandakumar.  Flip: Cross-domain face anti-spoofing with language guidance.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 19685–19696, 2023.
  * Sun et al. [2023] Yiyou Sun, Yaojie Liu, Xiaoming Liu, Yixuan Li, and Wen-Sheng Chu.  Rethinking domain generalization for face anti-spoofing: Separability and alignment.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 24563–24574, 2023.
  * Wang et al. [2022] Zhuo Wang, Zezheng Wang, Zitong Yu, Weihong Deng, Jiahong Li, Tingting Gao, and Zhongyuan Wang.  Domain generalization via shuffled style assembly for face anti-spoofing.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4113–4123, 2022.
  * Wu et al. [2020] Junru Wu, Xiang Yu, Buyu Liu, Zhangyang Wang, and Manmohan Chandraker.  Uncertainty-aware physically-guided proxy tasks for unseen domain face anti-spoofing.  _arXiv_ , abs/2011.14054, 2020.
  * Xie et al. [2023] Wuyuan Xie, Kaimin Wang, Yakun Ju, and Miaohui Wang.  pmbqa: Projection-based blind point cloud quality assessment via multimodal learning.  In _Proceedings of the ACM International Conference on Multimedia_ , page 3250–3258, 2023.
  * Xixi et al. [2023] Lan Xixi, Changchun Zou, Cheng Peng, and Caowei Wu.  Uncertainty quantification in intelligent-based electrical resistivity tomography image reconstruction with monte carlo dropout strategy.  _IEEE Transactions on Geoscience and Remote Sensing_ , 61:1–16, 2023.
  * Xu et al. [2023] Peng Xu, Xiatian Zhu, and David A Clifton.  Multimodal learning with transformers: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Yu et al. [2020a] Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn.  Gradient surgery for multi-task learning.  In _Proceedings of the Neural Information Processing Systems_ , 2020a.
  * Yu et al. [2022] Yi Yu, Wenhan Yang, Yap-Peng Tan, and Alex C Kot.  Towards robust rain removal against adversarial attacks: A comprehensive benchmark analysis and beyond.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 6013–6022, 2022.
  * Yu et al. [2023a] Yi Yu, Yufei Wang, Wenhan Yang, Shijian Lu, Yap-Peng Tan, and Alex C Kot.  Backdoor attacks against deep image compression via adaptive frequency trigger.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12250–12259, 2023a.
  * Yu et al. [2020b] Zitong Yu, Yunxiao Qin, Xiaobai Li, Zezheng Wang, Chenxu Zhao, Zhen Lei, and Guoying Zhao.  Multi-modal face anti-spoofing based on central difference networks.  In _IEEE Conference on Computer Vision and Pattern Recognition Workshop_ , pages 2766–2774, 2020b.
  * Yu et al. [2020c] Zitong Yu, Chenxu Zhao, Zezheng Wang, Yunxiao Qin, Zhuo Su, Xiaobai Li, Feng Zhou, and Guoying Zhao.  Searching central difference convolutional networks for face anti-spoofing.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 5294–5304, 2020c.
  * Yu et al. [2023b] Zitong Yu, Rizhao Cai, Yawen Cui, Ajian Liu, and Changsheng Chen.  Visual prompt flexible-modal face anti-spoofing.  _arXiv_ , abs/2307.13958, 2023b.
  * Yu et al. [2023c] Zitong Yu, Rizhao Cai, Yawen Cui, Xin Liu, Yongjian Hu, and Alex C. Kot.  Rethinking vision transformer and masked autoencoder in multimodal face anti-spoofing.  _arXiv_ , abs/2302.05744, 2023c.
  * Yu et al. [2023d] Zitong Yu, Ajian Liu, Chenxu Zhao, Kevin H. M. Cheng, Xu Cheng, and Guoying Zhao.  Flexible-modal face anti-spoofing: A benchmark.  In _IEEE Conference on Computer Vision and Pattern Recognition Workshop_ , pages 6346–6351, 2023d.
  * Yu et al. [2023e] Zitong Yu, Yunxiao Qin, Xiaobai Li, Chenxu Zhao, Zhen Lei, and Guoying Zhao.  Deep learning for face anti-spoofing: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(5):5609–5631, 2023e.
  * Yue et al. [2023] Haixiao Yue, Keyao Wang, Guosheng Zhang, Haocheng Feng, Junyu Han, Errui Ding, and Jingdong Wang.  Cyclically disentangled feature translation for face anti-spoofing.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , pages 3358–3366, 2023.
  * Zhang et al. [2023] Qingyang Zhang, Haitao Wu, Changqing Zhang, Qinghua Hu, Huazhu Fu, Joey Tianyi Zhou, and Xi Peng.  Provable dynamic fusion for low-quality multimodal data.  In _Proceedings of the International Conference on Machine Learning_ , pages 41753–41769, 2023.
  * Zhang et al. [2020] Shifeng Zhang, Ajian Liu, Jun Wan, Yanyan Liang, Guodong Guo, Sergio Escalera, Hugo Jair Escalante, and Stan Z. Li.  CASIA-SURF: A large-scale multi-modal benchmark for face anti-spoofing.  _IEEE transactions on biometrics, behavior, and identity science_ , 2(2):182–193, 2020.
  * Zhou et al. [2022a] Qianyu Zhou, Ke-Yue Zhang, Taiping Yao, Ran Yi, Shouhong Ding, and Lizhuang Ma.  Adaptive mixture of experts learning for generalizable face anti-spoofing.  In _Proceedings of the ACM International Conference on Multimedia_ , pages 6009–6018, 2022a.
  * Zhou et al. [2022b] Qianyu Zhou, Ke-Yue Zhang, Taiping Yao, Ran Yi, Kekai Sheng, Shouhong Ding, and Lizhuang Ma.  Generative domain adaptation for face anti-spoofing.  In _Proceedings of the European Conference on Computer Vision_ , pages 335–356, 2022b.
  * Zhou et al. [2023] Qianyu Zhou, Ke-Yue Zhang, Taiping Yao, Xuequan Lu, Ran Yi, Shouhong Ding, and Lizhuang Ma.  Instance-aware domain generalization for face anti-spoofing.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 20453–20463, 2023.