#  CAREForMe: Contextual Multi-Armed Bandit Recommendation Framework for
Mental Health

Sheng Yu<EMAIL_ADDRESS>University of Southern CaliforniaLos
AngelesCAUSA ,  Narjes Nourzad<EMAIL_ADDRESS>0009-0008-6315-8282 University
of Southern CaliforniaLos AngelesCAUSA ,  Randye J. Semple<EMAIL_ADDRESS>0000-0001-6469-2032 University of Southern CaliforniaLos AngelesCAUSA ,  Yixue
Zhao<EMAIL_ADDRESS>0000-0003-3046-6621 USC Information Sciences
InstituteArlingtonVAUSA ,  Emily Zhou<EMAIL_ADDRESS>0009-0009-5004-6588
University of Southern CaliforniaLos AngelesCAUSA and  Bhaskar Krishnamachari
<EMAIL_ADDRESS>0000-0002-9994-9931 University of Southern CaliforniaLos
AngelesCAUSA

(2018)

###### Abstract.

The COVID-19 pandemic has intensified the urgency for effective and accessible
mental health interventions in people’s daily lives. Mobile Health (mHealth)
solutions, such as AI Chatbots and Mindfulness Apps, have gained traction as
they expand beyond traditional clinical settings to support daily life.
However, the effectiveness of current mHealth solutions is impeded by the lack
of context-awareness, personalization, and modularity to foster their
reusability. This paper introduces CAREForMe, a contextual multi-armed bandit
(CMAB) recommendation framework for mental health. Designed with context-
awareness, personalization, and modularity at its core, CAREForMe harnesses
mobile sensing and integrates online learning algorithms with user clustering
capability to deliver timely, personalized recommendations. With its modular
design, CAREForMe serves as both a customizable recommendation framework to
guide future research, and a collaborative platform to facilitate
interdisciplinary contributions in mHealth research. We showcase CAREForMe’s
versatility through its implementation across various platforms (e.g.,
Discord, Telegram) and its customization to diverse recommendation features.

Mental Health, Mobile Sensing, Recommendation Systems, AI/ML

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: 11th International Conference on Mobile Software
Engineering and Systems 2024; April 2024; Lisbon, Portugal††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Mental health issues significantly increased during the COVID-19 pandemic and
remain a global public health concern (Cullen et al., 2020). Mobile Health
(mHealth) solutions, such as AI Chatbots and Mindfulness Apps, have gained
traction due to the widespread use of mobile devices (Stawarz et al., 2019;
mob, 2023). By reducing the traditional reliance on human experts such as
therapists, mHealth interventions can greatly enhance the accessibility,
availability, and affordability of mental health care. They bridge the gap
between clinics and homes, enabling continuous mental health monitoring and
treatment in users’ daily lives.

While mHealth solutions offer promising benefits, they currently face three
major challenges. First, they often lack _context-awareness_ to recognize when
the user is in need of help. This poses a burden on the user to initiate the
intervention when help is needed, which is often when users are least
motivated to do so. Second, these solutions fall short in _personalization_.
Many mHealth apps have a slow start to understand a user’s history and
preferences and rely heavily on user input. Users are either provided with the
same intervention (e.g., mindfulness modules) or are tasked to describe their
situation and navigate to the desired module by themselves each time they seek
assistance. Lastly, despite many mHealth solutions sharing common features,
such as sensor data collection and recommendation algorithms, they are often
developed in an _ad-hoc_ manner, causing duplicated efforts in this research
area.

To address these challenges, we propose CAREForMe, a Contextual multi-Armed
bandit REcommendation Framework for Mental health. CAREForMe is a customizable
framework designed with _context-awareness_ , _personalization_ , and
_modularization_ in mind, aiming to understand the user’s context and provide
the right recommendations at the right time. First, CAREForMe employs mobile
sensing to monitor the user’s change of states, aiming to identify _when_ the
recommendations should be delivered. Second, CAREForMe integrates user
clustering and contextual multi-armed bandit (CMAB) algorithms (Lu et al.,
2010) that continuously learn from user feedback to improve _what_ to
recommend based on user preferences. We choose CMAB algorithms because they
are more adaptive when learning from new data without the need to retrain from
scratch compared to matrix factorization (Xu, 2021; Elena et al., 2021). CMABs
are also better at handling new users, i.e., the so-called “cold-start”
problem, compared to traditional methods such as collaborative filtering
(Elena et al., 2021). Most importantly, CMABs are _context-sensitive_ , using
real-time user feedback to dynamically adjust the recommendations based on
user’s situation. Finally, CAREForMe is a customizable framework with a
modular design that decouples the underlying components, allowing different
approaches to “plug and play” in the framework, without requiring the domain
knowledge of the entire framework pipeline. For example, a mental health
expert can focus on tailoring the recommendation content, and reuse existing
recommendation algorithms (e.g., CAREForMe’s built-in CMAB algorithm) to
deliver her interventions. Similarly, a mobile app developer can focus on
implementing the front-end user interface (e.g., a mobile app) to interact
with the user and deliver the recommendations obtained from CAREForMe’s CMAB
back-end. Thus, CAREForMe stands not only as a customizable recommendation
framework, but also as a collaborative living repository, fostering
interdisciplinary contributions and facilitating research in the area of
mHealth.

Figure 1. The overview of CAREForMe’s framework design that is context-aware,
personalized, and modular.

To demonstrate CAREForMe’s potential, we implement its core components with
customizable features as a Chatbot recommendation system where the
recommendations are mindfulness-based exercises and gratitude practices. The
recommendation content can be easily customized, reused, or extended, without
requiring any technical knowledge. Specifically, our recommendation system is
implemented based on Upper Confidence Bound (UCB) (Carpentier et al., 2011),
providing suggestions according to the user’s context by continuously learning
from the user’s feedback. Moreover, user clustering is implemented and can be
optionally enabled in our system to provide suggestions based on the
preferences of similar users instead of a particular user. This largely
improves the recommendations for relatively new users with limited history to
learn from, and also addresses the “missing reward” problem when the user
forgets to provide the feedback (Bouneffouf et al., 2020). To show CAREForMe’s
flexibility and reusability, we implement 3 Chatbot systems that reuse the
same back-end recommendation engine on Discord (dis, 2023), Telegram (tel,
2023), and Flet (fle, 2023) that functions on both Android and iOS systems. A
user simulation module is implemented to test out our Chatbot systems’
functionalities including the user clustering capabilities.

We make the following contributions. (1) We design the first customizable
CMAB-based framework of recommendation systems for mental health that is
context-aware, personalized, and modular; (2) We develop a flexible
recommendation engine with user clustering mechanisms that allow different
levels of customization; (3) We implement an end-to-end recommendation system
on 3 different Chatbot platforms, demonstrating the framework’s flexibility
and reusability; (4) We make our artifacts publicly available (MAB, 2023),
directly improving the reproducibility and reusability of our research, and
fostering future research in this area.

The rest of the paper is organized as follows. Section 2 introduces the design
of CAREForMe framework and its modular components. Section 3 describes our
implementation of CAREForMe’s core components and details its customizable
capabilities. Section 4 discusses related work in this area, and Section 5
elaborates our vision for adopting CAREForMe in practice and future work
ahead.

##  2\. Framework Design

This section describes CAREForMe’s design that incorporates _context-
awareness_ , _personalization_ , and _modularization_ as discussed earlier.
Our design aims to provide a comprehensive, yet flexible framework that serves
as a standard to guide the research in this area and improve the reusability
of each other’s work. Figure 1 shows the overview of CAREForMe framework with
7 components. These components are extracted from our extensive literature
review in this area, and the core components in the CMAB recommendation
pipeline are implemented by us (see Section 3). We now describe the high-level
functionalities of each component.

In-the-Wild Sensor Collection guides the researchers to first answer _what
sensors to use_ given their specific problem domain and resource constraints
(Wang et al., 2009). We highlight “in-the-wild” as this paper focuses on
providing recommendations during the user’s daily life, as opposed to a lab
setting. This is recently reported as an important but overlooked area
(D’Mello and Booth, 2023). We identify 5 main categories of in-the-wild
sensors: (1) _Wearable Devices_ can be worn in daily life, such as
smartwatches and smart bands. They often monitor the user’s location, heart
rate, Heart Rate Variability (HRV), stress level, and activities such as
walking; (2) _Mobile Devices_ refer to smartphones or tablets that can provide
rich information about the user’s states, including app usage and physical
activities. For example, mobile sensors (And, 2023) can determine a user’s
activity, such as lying down, walking, or sitting. App usage data can offer
insights into various aspects of a user’s well-being, including sleep quality
and nutritional content of meals; (3) _Eye Tracker_ devices are used to
monitor and measure the position and movement of the eyes. They have a large
potential to detect the user’s mental state, such as emotion and fatigue (Lim
et al., 2020; Gunawardena et al., 2022); (4) _Neurophysiological Devices_ can
measure the user’s neural activities in the brain to uncover insights into the
user’s mental states (e.g., cognitive abilities, stress level) via signals
such as EEG (Rayi and Murr, 2020) and fNIR (Pinti et al., 2020). As CAREForMe
focuses on _in-the-wild_ sensors, we specifically target _portable_ devices
that can be used outside of lab settings (LaRocco et al., 2020; artinis, 2023;
Muse, 2023); (5) _Environmental Monitors_ measure the user’s surrounding
environment (e.g., work, home) to provide the environmental context of the
user, such as the surrounding temperature, noise level, and lighting.

Sensor Feature Selection aims to answer _what sensor data to use_ as the input
to the models that detect the user’s state (e.g., mood) given the available
sensors and the context. To the best of our knowledge, this is a novel
component we introduce that is not considered by prior work. Existing work in
this area relies on a fixed set of sensors’ data as features to perform affect
detection and has not explored context-informed sensor feature selection (Bota
et al., 2019). CAREForMe incorporates _context-awareness_ in its Sensor
Feature Selection to use suitable sensor data based on the user’s context,
aiming to increase the accuracy of the user state detection. For instance,
when a user is sitting and working at a computer, the Eye Tracker’s data about
their focus and attention is likely to be more insightful than the data
collected from their Mobile Device.

User State Detection takes the selected sensor features as input and answers
_what models to use_ to detect the user’s state, such as their mood. This
information is used by the Recommendation Engine to determine _when_ certain
intervention should be recommended to the user. For example, prior work
focuses on detecting loneliness via supervised learning algorithms (e.g.,
logistic regression, gradient boosting classifier) using daily features such
as communication, phone usage, and sleep behaviors collected from smartphones
and smart bands (Doryab et al., 2019). If integrated into CAREForMe’s User
State Detection, other researchers can reuse such component to develop a
Recommendation Engine that suggests social activities based on the user’s
loneliness level. Note that CAREForMe is a conceptual framework that does not
assume any particular implementation of its components. Thus, the models used
in the User State Detection can be either supervised or unsupervised with any
architecture or algorithm of the researcher’s choice.

CMAB Recommendation Engine is the core component in CAREForMe, integrating the
recommendation algorithms and user clustering modeling to offer personalized
suggestions. It takes 3 types of input: (1) the output from the User State
Detection to determine _when_ to recommend; (2) the User Input such as user
profile (e.g., user personality, age), ecological momentary assessment (EMA)
of the user’s behaviors and mental states, and user feedback of CAREForMe’s
suggestions which is used by the recommendation algorithms to _dynamically_
improve its results; and (3) the Mental Health Intervention Inventory, often
tailored by mental health experts in a specific domain, such as psychiatrists
focusing on stress, anxiety, or loneliness. The output of CMAB Recommendation
Engine is used by the CMAB User Interface to deliver intervention suggestions
to the user and obtain their feedback. This component is often implemented by
app developers specialized in user interface design and development, without
requiring the technical knowledge of the recommendation systems. For example,
we have implemented 3 Chatbot CMAB User Interfaces on different platforms
(e.g., Discord, Telegram) using the same CMAB Recommendation Engine to show
CAREForMe’s flexibility and reusability (see Section 3). Our modular design
adopts the “separation of concerns” principle (Kulkarni and Reddy, 2003),
enabling collaborative contributions from experts across various disciplines.

##  3\. Framework Instantiation

This section details our recommendation system that implements CAREForMe’s
core components, focusing on _what_ intervention to recommend. As discussed
earlier, the remaining components focus on _when_ to recommend by leveraging
sensor data to detect the user state. We did not implement such components as
they fall into a crowded research area and can be implemented by reusing
existing work in various problem domains such as stress detection (Cho et al.,
2019; Chikersal et al., 2021). We now detail the implementation of our
recommendation system.

Mental Health Intervention Inventory is implemented in YAML format (yam,
2023), specifically designed for mental health experts such as psychiatrists,
psychologists, or social workers without requiring any technical knowledge
about recommendation systems. Mental health experts can configure the YAML
file to specify the number of interventions to recommend each time, and the
interventions with their corresponding context. Each intervention is defined
as (title, description, image, context), such as (“STOP”, “Stop, Take a deep
breath, Observe, and Proceed.”, “image.png”, “home”). Our implementation uses
mindfulness-based interventions and gratitude practices that are shown to be
effective for people’s daily lives (Semple and Willard, 2019). Note that the
inventory can grow to include interventions across various areas (e.g.,
anxiety, depression, anger management), and the system can be configured to
use a subset of the interventions for targeted mental health issues. The
current interventions can be found in our public repository (MAB, 2023), which
are highly reusable and extensible.

CMAB Recommendation Engine is designed to continuously learn and refine its
suggestions based on real-time user feedback from User Input. Its
recommendation algorithm is implemented based on Upper Confidence Bound (UCB)
(Carpentier et al., 2011), designed to provide context-sensitive
recommendations, ensuring that the suggestions are tailored to individual
user’s context (e.g., home, work). The user clustering modeling is implemented
using K-means (Ahmed et al., 2020), which can further refine the suggestions
by learning from similar users’ data. Note that there is a trade-off between
the amount of data to learn from and the level of personalization achieved.
Using data from all users maximizes the amount of training data but may reduce
recommendation specificity, while using only individual user data ensures
tailored recommendations but may require extended learning time. Our system
can be customized to achieve either of the scenarios above, or the middle
ground by enabling the user clustering modeling to learn from a subset of the
users that belong to the same cluster. Furthermore, our modular design allows
other clustering techniques to be easily “plugged in” by only replacing user
clustering modeling without the need to change the rest of the system, such as
replacing K-means with ADCB clustering (Wang et al., 2023).

Input: User $user$, Context $context$, Set $interventions$,

Vector $CMAB\\_score$, Vector $cluster$, Bool $implicit$

1

2if  _$user$.session_num == 0_ then

3
$reco\\_set\leftarrow\textsc{SuggestAll}(interventions,CMAB\\_score,context)$

4else if  _$user$.session_num ¡ $threshold$_ then

5 $reco\\_set\leftarrow\textsc{Suggest}(interventions,CMAB\\_score,context)$

6else

7 $user.cluster\leftarrow\textsc{GetCluster}(user,cluster)$

8
$reco\\_set\leftarrow\textsc{Suggest}(interventions,CMAB\\_score,context,user)$

9

10$choice\leftarrow\textsc{PromptUserForSelection}(reco\\_set)$

11 $feedback\leftarrow\textsc{PrompUserForFeedback}(choice)$

12

13if  _$implicit$ == TRUE_ then

14 UpdateCMAB($user$, $context$, $choice$, $feedback$, $reco\\_set$)

15

16else

17 UpdateCMAB($user$, $context$, $choice$, $feedback$)

18

19

20UpdatePrefVector($user$, $context$, $choice$, $feedback$)

21 UpdateCluster($user$, $context$, $choice$, $feedback$, $cluster$)

22

Algorithm 1 CMAB Recommendation Process

Our system’s recommendation process is implemented as a Chatbot to interact
with the user, providing suggestions and obtaining corresponding user
feedback. Algorithm 1 details one session of the recommendation process with
user clustering enabled. The input includes the information of the $user$
maintained by our system (e.g., how many sessions the user has interacted with
the system), the $context$ the user is in (e.g., work), a set of
$interventions$ to recommend from, the vector of the $CMAB\\_score$ that
stores the information of all of the user’s past preferences that will be used
to rank the recommendations (Carpentier et al., 2011), the vector $cluster$
that stores the information of similar users’ past preferences in each
cluster, and whether user’s $implicit$ preferences on the unchosen suggestions
should be considered when calculating the $CMAB\\_score$. After the session
starts, the number of how many sessions the user has interacted with the
system ($user.session\\_num$) is checked. If the user is new, the suggestions
($reco\\_set$) are learned from the preferences of _all_ the users in the
system, and the $interventions$ with the highest $CMAB\\_score$ in the user’s
provided context will be suggested (Line 2). This addresses the “cold-start”
problem and ensures that new users are presented with popular suggestions
across all the users. If $user.session\\_num$ is smaller than a customizable
threshold, the user gets the suggestions ($reco\\_set$) from the
$interventions$ based on their own past preferences, i.e., the $CMAB\\_score$
calculated based on their past sessions using UCB method (Carpentier et al.,
2011) (Line 4). When $user.session\\_num$ goes beyond the threshold, the
$user$ is assigned to the closest cluster based on the Euclidean distance
between the $cluster$ vector and the vector representing their past
preferences (Lines 6-7). In this case, the $reco\\_set$ are calculated by
using the $CMAB\\_score$ of similar users who belong to the same cluster.

Next, the recommendations are presented to the user to get the user input on
which suggestion to choose ($choice$) and the $feedback$ for such suggestion
using a score between 0 to 5 (Lines 8-9). The obtained user input is used to
update the $CMAB\\_score$ for this particular user (Lines 10-13). If
$implicit$ is set to true, not only the $CMAB\\_score$ for the $choice$ will
be updated, but the $CMAB\\_score$ for the remaining suggestions in the
$reco\\_set$ will also be updated based on a customizable value. For instance,
one may consider the unselected suggestions unfavored by the user and set a
negative value to update the $CMAB\\_score$ for those suggestions. Finally,
the user’s preference vector that maintains the history of the user’s
preferences is updated accordingly, including their choices and feedback (Line
14). The $cluster$ vector is updated as well to include the current session’s
data in the user’s cluster (Line 15).

CMAB User Interface uses the output from CMAB Recommendation Engine to deliver
the recommendations to the users and obtain their feedback. It is designed to
be decoupled from the underlying logic of the recommendation engine to enable
the flexibility for app developers to customize their desired user interface.
As a demonstration of such flexibility, we implement 3 variations of CMAB User
Interface using the same CMAB Recommendation Engine. Specifically, 3 Chatbot
systems are implemented on Discord (dis, 2023), Telegram (tel, 2023), and
mobile platforms using Flet framework (fle, 2023) which can function on both
Android and iOS platforms.

##  4\. Related Work

To the best of our knowledge, CAREForMe is the first framework that provides a
comprehensive view throughout the recommendation pipeline based on CMAB that
is context-aware, personalized, and modular. The closest work (Gonul et al.,
2018) proposed a template-based digital intervention design mechanism that
supports the configuration of just-in-time adaptive interventions (JITAIs) for
chronic health issues. It allows developers to instantiate its main components
of decision points, intervention options, tailoring variables, and decision
rules. However, this framework does not adapt to real-time user feedback and
does not consider various types of sensors.

Mobile and Wearable Sensing for affect detection has become increasingly
popular due to their ubiquity and unobtrusive nature that allow for a greater
volume of data collection across multiple modalities (Schmidt et al., 2019;
Politou et al., 2017), such as detecting loneliness using data from
smartphones and smart bands (Doryab et al., 2019). However, these efforts were
achieved separately, each requiring the development of its own software
pipelines. This directly motivates us to design a customizable and modular
framework to facilitate future work in this domain.

Health Recommender Systems (HRS) utilizes machine learning (ML) techniques and
mobile platforms to support human health and wellness in many areas, such as
providing recommendations for individuals with mental illness (Rohani et al.,
2020). In recent years, multi-armed bandits (MAB) algorithms have found
success in applications such as weight loss interventions (Zhou et al., 2023),
emotion regulation for mental illness (Ameko et al., 2020), and personalized
physical activity recommendations (Rabbi et al., 2017). However, HRSs often
rely on an individual’s health history, demographic information, and self-
reports, and do not focus on delivering in-the-wild, real-time adaptive
interventions (De Croon et al., 2021).

JITAI Systems as mentioned above have been used to provide real-time
personalized support to promote positive behavior change for the improvement
of health in many areas (Nahum-Shani et al., 2017), such as mental health
(Ben-Zeev et al., 2013; Coppersmith et al., 2022) and alcohol usage (Gustafson
et al., 2014). JITAIs aim to provide the right intervention based on the
user’s change of state and context in real-time through mobile sensing
technologies (Nahum-Shani et al., 2017), and have significantly outperformed
non-JITAI treatments (Wang and Miller, 2020). CAREForMe falls into the JITAI
category for mental health, providing a comprehensive yet flexible framework
to allow different levels of customization that goes beyond prior work.

##  5\. CAREForMe’s Vision

Our long-term vision for CAREForMe is three-fold. First, CAREForMe’s design
serves as a reference architecture, providing a comprehensive view of the
state-of-the-art recommendation pipeline to guide newcomers in this field.
Specifically, our framework guides new researchers to answer a set of research
questions and decide which component(s) are of interest. For instance, those
interested in _when_ to recommend can delve into the User State Detection
component, while others may contribute to the novel Sensor Feature Selection
component we identified, an area previously unexplored. Second, our
implementation of CAREForMe’s core components showcases the framework’s
utility across various disciplines. It demonstrates how mental health
professionals, app developers, and algorithm experts can leverage the modular
design of CAREForMe to concentrate on their strengths and build upon each
other’s work, directly fostering an environment of interdisciplinary
collaboration. Finally, as others in the community adopt our framework, we
envision CAREForMe growing into a living repository for just-in-time adaptive
interventions (JITAIs), potentially extending its reach beyond mental health.
For instance, a fitness expert can tailor the suggestions using physical
exercises, and the User State Detection can focus on the user’s physical
activity levels. Similarly, other potential applications of CAREForMe include
providing context-sensitive medication reminders or nutritional advice.

In the future, we plan to incorporate continuous learning throughout the
framework, enabling CMAB techniques to answer not only _what_ , but also
_when_ to recommend. As users have different levels of acceptance on the
frequency of recommendations, CMAB can be applied to improve the
intervention’s timing by learning from user feedback. Furthermore, we aim to
evolve CAREForMe into a collaborative repository, similar to DBDP project
(Bent et al., 2021), but extend to the whole recommendation system pipeline.
We plan to migrate existing work into CAREForMe, thereby enhancing their
reusability and facilitating new discoveries. As the repository populates, the
scope will grow to include capabilities such as fine-grained privacy control
on the user’s data collection, an access control model for data sharing, and a
platform to connect researchers from various disciplines within the CAREForMe
ecosystem. We believe the connection across disciplines is essential to
advance research in human well-being given the intrinsic link between physical
and mental health. CAREForMe has the potential to bridge the gap by providing
a holistic view of user health with cross-disciplinary insights and adaptive
recommendations that span different areas to improve the user’s overall well-
being, including mindfulness exercises, physical activities, nutritional
advice, and so on.

## References

  * (1)
  * MAB (2023) 2023\.  CAREForMe’s public repository.  https://github.com/ANRGUSC/mab-reco
  * fle (2023) 2023\.  The fastest way to build Flutter apps in Python.  https://flet.dev/
  * mob (2023) 2023\.  Mobile App Download Statistics & Usage Statistics (2023).  https://buildfire.com/app-statistics/
  * And (2023) 2023\.  Sensors Overview $|$ Android Developers.  https://developer.android.com/guide/topics/sensors/sensors_overview
  * tel (2023) 2023\.  Telegram Bot Features.  https://core.telegram.org/bots/features
  * dis (2023) 2023\.  Welcome to the Discord Developer Platform.  https://discord.com/developers/docs/intro
  * yam (2023) 2023\.  YAML Tutorial: Everything You Need to Get Started in Minutes.  https://www.cloudbees.com/blog/yaml-tutorial-everything-you-need-get-started
  * Ahmed et al. (2020) Mohiuddin Ahmed, Raihan Seraj, and Syed Mohammed Shamsul Islam. 2020\.  The k-means algorithm: A comprehensive survey and performance evaluation.  _Electronics_ 9, 8 (2020), 1295.
  * Ameko et al. (2020) Mawulolo K. Ameko, Miranda L. Beltzer, Lihua Cai, Mehdi Boukhechba, Bethany A. Teachman, and Laura E. Barnes. 2020.  Offline Contextual Multi-Armed Bandits for Mobile Health Interventions: A Case Study on Emotion Regulation. Association for Computing Machinery, New York, NY, USA.  https://doi.org/10.1145/3383313.3412244
  * artinis (2023) artinis. 2023.  Wearable fNIRS EEG combinations.  https://www.artinis.com/nirs-eeg-package
  * Ben-Zeev et al. (2013) Dror Ben-Zeev, Susan M Kaiser, Christopher J Brenner, Mark Begale, Jennifer Duffecy, and David C Mohr. 2013.  Development and usability testing of FOCUS: a smartphone system for self-management of schizophrenia.  _Psychiatric rehabilitation journal_ 36, 4 (2013), 289\.
  * Bent et al. (2021) Brinnae Bent, Ke Wang, Emilia Grzesiak, Chentian Jiang, Yuankai Qi, Yihang Jiang, Peter Cho, Kyle Zingler, Felix Ikponmwosa Ogbeide, Arthur Zhao, et al. 2021\.  The digital biomarker discovery pipeline: An open-source software platform for the development of digital biomarkers using mHealth and wearables data.  _Journal of clinical and translational science_ 5, 1 (2021), e19.
  * Bota et al. (2019) Patrícia J. Bota, Chen Wang, Ana L. N. Fred, and Hugo Plácido Da Silva. 2019.  A Review, Current Challenges, and Future Possibilities on Emotion Recognition Using Machine Learning and Physiological Signals.  _IEEE Access_ 7 (2019), 140990–141020.  https://doi.org/10.1109/ACCESS.2019.2944001
  * Bouneffouf et al. (2020) Djallel Bouneffouf, Sohini Upadhyay, and Yasaman Khazaeni. 2020\.  Contextual Bandit with Missing Rewards.  _ArXiv_ abs/2007.06368 (2020).  https://api.semanticscholar.org/CorpusID:220496532
  * Carpentier et al. (2011) Alexandra Carpentier, Alessandro Lazaric, Mohammad Ghavamzadeh, Rémi Munos, and Peter Auer. 2011\.  Upper-confidence-bound algorithms for active learning in multi-armed bandits. In _International Conference on Algorithmic Learning Theory_. Springer, 189–203.
  * Chikersal et al. (2021) Prerna Chikersal, Afsaneh Doryab, Michael Tumminia, Daniella K Villalba, Janine M Dutcher, Xinwen Liu, Sheldon Cohen, Kasey G Creswell, Jennifer Mankoff, J David Creswell, et al. 2021\.  Detecting depression and predicting its onset using longitudinal symptoms captured by passive sensing: a machine learning approach with robust feature selection.  _ACM Transactions on Computer-Human Interaction (TOCHI)_ 28, 1 (2021), 1–41.
  * Cho et al. (2019) Youngjun Cho, Simon J Julier, and Nadia Bianchi-Berthouze. 2019\.  Instant stress: detection of perceived mental stress through smartphone photoplethysmography and thermal imaging.  _JMIR mental health_ 6, 4 (2019), e10140.
  * Coppersmith et al. (2022) Daniel DL Coppersmith, Walter Dempsey, Evan M Kleiman, Kate H Bentley, Susan A Murphy, and Matthew K Nock. 2022.  Just-in-time adaptive interventions for suicide prevention: Promise, challenges, and future directions.  _Psychiatry_ 85, 4 (2022), 317–333.
  * Cullen et al. (2020) Walter Cullen, Gaurav Gulati, and Brendan D Kelly. 2020\.  Mental health in the COVID-19 pandemic.  _QJM: An International Journal of Medicine_ 113, 5 (2020), 311–312.
  * De Croon et al. (2021) Robin De Croon, Leen Van Houdt, Nyi Nyi Htun, Gregor Štiglic, Vero Vanden Abeele, and Katrien Verbert. 2021.  Health Recommender Systems: Systematic Review.  _Journal of Medical Internet Research_ 23, 6 (2021).  https://doi.org/10.2196/18035
  * Doryab et al. (2019) Afsaneh Doryab, Daniella K Villalba, Prerna Chikersal, Janine M Dutcher, Michael Tumminia, Xinwen Liu, Sheldon Cohen, Kasey Creswell, Jennifer Mankoff, John D Creswell, et al. 2019\.  Identifying behavioral phenotypes of loneliness and social isolation with passive sensing: statistical analysis, data mining and machine learning of smartphone and fitbit data.  _JMIR mHealth and uHealth_ 7, 7 (2019), e13209.
  * D’Mello and Booth (2023) Sidney K D’Mello and Brandon M Booth. 2023.  Affect Detection From Wearables in the “Real” Wild: Fact, Fantasy, or Somewhere In between?  _IEEE Intelligent Systems_ 38, 1 (2023), 76–84.
  * Elena et al. (2021) Gangan Elena, Kudus Milos, and Ilyushin Eugene. 2021\.  Survey of multiarmed bandit algorithms applied to recommendation systems.  _International Journal of Open Information Technologies_ 9, 4 (2021), 12–27.
  * Gonul et al. (2018) Suat Gonul, Tuncay Namli, Sasja Huisman, Gokce Banu Laleci Erturkmen, Ismail Hakki Toroslu, and Ahmet Cosar. 2018.  An expandable approach for design and personalization of digital, just-in-time adaptive interventions.  _Journal of the American Medical Informatics Association_ 26, 3 (2018), 198–210.  https://doi.org/10.1093/jamia/ocy160
  * Gunawardena et al. (2022) Nishan Gunawardena, Jeewani Anupama Ginige, and Bahman Javadi. 2022.  Eye-tracking technologies in mobile devices Using edge computing: a systematic review.  _Comput. Surveys_ 55, 8 (2022), 1–33.
  * Gustafson et al. (2014) David H. Gustafson, Fiona M. McTavish, Ming-Yuan Chih, Amy K. Atwood, Roberta A. Johnson, Michael G. Boyle, Michael S. Levy, Hilary Driscoll, Steven M. Chisholm, Lisa Dillenburg, Andrew Isham, and Dhavan Shah. 2014.  A Smartphone Application to Support Recovery From Alcoholism: A Randomized Clinical Trial.  _JAMA Psychiatry_ 71, 5 (05 2014), 566–572.  https://doi.org/10.1001/jamapsychiatry.2013.4642
  * Kulkarni and Reddy (2003) Vinay Kulkarni and Sreedhar Reddy. 2003.  Separation of concerns in model-driven development.  _IEEE software_ 20, 5 (2003), 64–69.
  * LaRocco et al. (2020) John LaRocco, Minh Dong Le, and Dong-Guk Paeng. 2020\.  A systemic review of available low-cost EEG headsets used for drowsiness detection.  _Frontiers in neuroinformatics_ (2020), 42.
  * Lim et al. (2020) Jia Zheng Lim, James Mountstephens, and Jason Teo. 2020\.  Emotion recognition using eye-tracking: taxonomy, review and current challenges.  _Sensors_ 20, 8 (2020), 2384.
  * Lu et al. (2010) Tyler Lu, David Pal, and Martin Pal. 2010.  Contextual Multi-Armed Bandits. In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ _(Proceedings of Machine Learning Research, Vol. 9)_ , Yee Whye Teh and Mike Titterington (Eds.). PMLR, Chia Laguna Resort, Sardinia, Italy, 485–492.  https://proceedings.mlr.press/v9/lu10a.html
  * Muse (2023) Muse. 2023.  Featured research with Muse.  https://choosemuse.com/pages/muse-research
  * Nahum-Shani et al. (2017) Inbal Nahum-Shani, Shawna N Smith, Bonnie J Spring, Linda M Collins, Katie Witkiewitz, Ambuj Tewari, and Susan A Murphy. 2017\.  Just-in-Time Adaptive Interventions (JITAIs) in Mobile Health: Key Components and Design Principles for Ongoing Health Behavior Support.  _Annals of Behavioral Medicine_ 52, 6 (2017), 446–462.  https://doi.org/10.1007/s12160-016-9830-8
  * Pinti et al. (2020) Paola Pinti, Ilias Tachtsidis, Antonia Hamilton, Joy Hirsch, Clarisse Aichelburg, Sam Gilbert, and Paul W Burgess. 2020\.  The present and future use of functional near-infrared spectroscopy (fNIRS) for cognitive neuroscience.  _Annals of the New York Academy of Sciences_ 1464, 1 (2020), 5–29.
  * Politou et al. (2017) Eugenia Politou, Efthimios Alepis, and Constantinos Patsakis. 2017\.  A survey on mobile affective computing.  _Computer Science Review_ 25 (2017), 79–100.  https://doi.org/10.1016/j.cosrev.2017.07.002
  * Rabbi et al. (2017) Mashfiqui Rabbi, Min Hane Aung, and Tanzeem Choudhury. 2017\.  _Towards Health Recommendation Systems: An Approach for Providing Automated Personalized Health Feedback from Mobile Data_.  Springer International Publishing, 519–542.  https://doi.org/10.1007/978-3-319-51394-2_26
  * Rayi and Murr (2020) Appaji Rayi and Najib Murr. 2020.  Electroencephalogram.  https://www.ncbi.nlm.nih.gov/books/NBK563295/
  * Rohani et al. (2020) Darius A. Rohani, Andrea Quemada Lopategui, Nanna Tuxen, Maria Faurholt-Jepsen, Lars V. Kessing, and Jakob E. Bardram. 2020.  MUBS: A Personalized Recommender System for Behavioral Activation in Mental Health. In _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ _(CHI ’20)_. Association for Computing Machinery, New York, NY, USA, 1–13.  https://doi.org/10.1145/3313831.3376879
  * Schmidt et al. (2019) Philip Schmidt, Attila Reiss, Robert Dürichen, and Kristof Van Laerhoven. 2019.  Wearable-Based Affect Recognition—A Review.  _Sensors_ 19, 19 (2019).  https://doi.org/10.3390/s19194079
  * Semple and Willard (2019) Randye J Semple and Christopher Willard. 2019.  _The Mindfulness Matters program for children and adolescents: Strategies, activities, and techniques for therapists and teachers_.  Guilford Publications.
  * Stawarz et al. (2019) Katarzyna Stawarz, Chris Preist, and David Coyle. 2019\.  Use of Smartphone Apps, Social Media, and Web-Based Resources to Support Mental Health and Well-Being: Online Survey.  _JMIR Mental Health_ 6 (2019).  https://api.semanticscholar.org/CorpusID:189995653
  * Wang and Miller (2020) Liyuan Wang and Lynn Carol Miller. 2020.  Just-in-the-Moment Adaptive Interventions (JITAI): A Meta-Analytical Review.  _Health Communication_ 35, 12 (2020), 1531–1544.  https://doi.org/10.1080/10410236.2019.1652388
  * Wang et al. (2009) Yi Wang, Jialiu Lin, Murali Annavaram, Quinn A Jacobson, Jason Hong, Bhaskar Krishnamachari, and Norman Sadeh. 2009.  A framework of energy efficient mobile sensing for automatic user state recognition. In _Proceedings of the 7th international conference on Mobile systems, applications, and services_. 179–192.
  * Wang et al. (2023) Yufeng Wang, Weidong Zhang, Jianhua Ma, and Qun Jin. 2023\.  ADCB: Adaptive Dynamic Clustering of Bandits for Online Recommendation System.  _Neural Processing Letters_ 55, 2 (2023), 1155–1172.
  * Xu (2021) Shenghao Xu. 2021\.  BanditMF: Multi-Armed Bandit Based Matrix Factorization Recommender System.  _ArXiv_ abs/2106.10898 (2021).  https://api.semanticscholar.org/CorpusID:235490165
  * Zhou et al. (2023) Tongxin Zhou, Yingfei Wang, Lu (Lucy) Yan, and Yong Tan. 2023\.  Spoiled for choice? Personalized recommendation for healthcare decisions: A multiarmed bandit approach.  _Information Systems Research_ (2023).  https://doi.org/10.1287/isre.2022.1191


# Dual-Cross Central Difference Network for Face Anti-Spoofing

Zitong Yu1 Yunxiao Qin2 Hengshuang Zhao3 Xiaobai Li1&Guoying Zhao1 1CMVS,
University of Oulu 2Northwestern Polytechnical University 3University of
Oxford {zitong.yu, xiaobai.li<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Corresponding author. This work was supported by
the Academy of Finland for project MiGA (grant 316765), ICT 2023 project
(grant 328115), Infotech Oulu, project 6+E (grant 323287) funded by Academy of
Finland, and project PhInGAIN (grant 200414) funded by The Finnish Work
Environmental Fund. The authors wish to acknowledge CSC-IT Center for Science,
Finland, for computational resources.

###### Abstract

Face anti-spoofing (FAS) plays a vital role in securing face recognition
systems. Recently, central difference convolution (CDC) Yu et al. (2020d) has
shown its excellent representation capacity for the FAS task via leveraging
local gradient features. However, aggregating central difference clues from
all neighbors/directions simultaneously makes the CDC redundant and sub-
optimized in the training phase. In this paper, we propose two Cross Central
Difference Convolutions (C-CDC), which exploit the difference of the center
and surround sparse local features from the horizontal/vertical and diagonal
directions, respectively. It is interesting to find that, with only five ninth
parameters and less computational cost, C-CDC even outperforms the full
directional CDC. Based on these two decoupled C-CDC, a powerful Dual-Cross
Central Difference Network (DC-CDN) is established with Cross Feature
Interaction Modules (CFIM) for mutual relation mining and local detailed
representation enhancement. Furthermore, a novel Patch Exchange (PE)
augmentation strategy for FAS is proposed via simply exchanging the face
patches as well as their dense labels from random samples. Thus, the augmented
samples contain richer live/spoof patterns and diverse domain distributions,
which benefits the intrinsic and robust feature learning. Comprehensive
experiments are performed on four benchmark datasets with three testing
protocols to demonstrate our state-of-the-art performance.

##  1 Introduction

Face recognition technology has widely used in many interactive intelligent
systems due to their convenience and remarkable accuracy. However, face
recognition systems are still vulnerable to presentation attacks (PAs) ranging
from print, replay and 3D-mask attacks. Therefore, both the academia and
industry have recognized the critical role of face anti-spoofing (FAS) for
securing the face recognition system.

In the past decade, both traditional de Freitas Pereira et al. (2012);
Komulainen et al. (2013); Patel et al. (2016) and deep learning-based Yu et
al. (2020d, a); Liu et al. (2018); Jourabloo et al. (2018); Yang et al.
(2019); Yu et al. (2020b) methods have shown effectiveness for presentation
attack detection (PAD). On one hand, some classical local descriptors (e.g.,
local binary pattern (LBP) Boulkenafet et al. (2015) and histogram of gradient
(HOG) Komulainen et al. (2013)) are robust for describing the detailed
invariant information (e.g., color texture, moir$\rm\acute{e}$ pattern and
noise artifacts) from spoofing faces. However, the shallow and coarse feature
extraction procedure limits the discriminative capacity of these local
descriptors.

On the other hand, convolutional neural networks (CNNs) focus on representing
deeper semantic features to distinguish the bonafide and PAs, which are weak
in capturing fine-grained intrinsic patterns (e.g., lattice artifacts) between
the live and spoof faces, and easily influenced under diverse scenarios.
Although central difference convolution (CDC) Yu et al. (2020d) introduced the
central-oriented local gradient features to enhance models’ generalization and
discrimination capacity, it still suffers from two disadvantages. In CDC,
central gradients from all neighbors are calculated, which is 1) inefficient
in both inference and back-propagation stages; and 2) redundant and sub-
optimized due to the discrepancy among diverse gradient directions. Thus, to
study the impacts and relations among central gradients is not a trivial work.

One key challenge in the FAS task is how to learn representation with limited
data as existing FAS datasets (e.g., OULU-NPU Boulkenafet et al. (2017) and
SiW-M Liu et al. (2019)) do not have large amount of training data due to the
high collection cost for both spoofing generation and video recording.
Although generic augmentation manners (e.g., horizontal flip, color jitter and
Cutout DeVries and Taylor (2017)) are able to expand the scale and diversity
of live/spoof samples, it still contributes not much performance improvement.
Thus, it is worth rethinking the augmentation for FAS and design task-
dedicated augmentation paradigm.

Motivated by the discussions above, we propose a novel convolution operator
family called Cross Central Difference Convolution (C-CDC), which decouples
the central gradient features into cross-directional combination
(horizontal/vertical or diagonal) thus more efficient and concentrated for
message aggregation. Furthermore, in order to mimic more general attacks
(e.g., partial print and mask attacks) and learn to distinguish spoofing in
both global and patch level, Patch Exchange (PE) augmentation is proposed for
mixed sample as well as corresponding dense label generation. To sum up, our
contributions include:

  * •

We design a sparse convolution family called Cross Central Difference
Convolution (C-CDC), which decouples the vanilla CDC into two cross (i.e.,
horizontal/vertical and diagonal) directions, respectively. Compared with CDC,
our proposed C-CDC could achieve better performance for FAS with only five
ninth parameters and less computational cost.

  * •

We propose a Dual-Cross Central Difference Network (DC-CDN), consisting of
two-stream backbones with horizontal/vertical and diagonal C-CDC,
respectively. Moreover, we also introduce Cross Feature Interaction Modules
(CFIM) between two streams of DC-CDN for mutual neighbor relation mining and
local detailed representation enhancement.

  * •

We propose the first FAS-dedicated data augmentation method, Patch Exchanges
(PE), to synthesize mixed samples with diverse attacks and domains, which is
able to plug and play in not only DC-CDN but also existing FAS methods for
performance improvement.

  * •

Our proposed method achieves state-of-the-art performance on four benchmark
datasets with intra-dataset, cross-dataset, and cross-type testing protocols.

##  2 Related Work

Face Anti-Spoofing. Traditional face anti-spoofing methods usually extract
handcrafted features from the facial images to capture the spoofing patterns.
Some classical local descriptors such as LBP Boulkenafet et al. (2015) and HOG
Komulainen et al. (2013) are utilized for handcrafted features. More recently,
a few deep learning based methods are proposed for face anti-spoofing. On the
one hand, FAS can be naturally treated as a binary classification task, thus
binary cross entropy loss is used for model supervision. On the other hand,
according to the physical discrepancy between live and spoof faces, dense
pixel-wise supervisions Yu et al. (2021) such as pseudo depth map Liu et al.
(2018); Yu et al. (2020d); Wang et al. (2020), reflection map Yu et al.
(2020a), texture map Zhang et al. (2020) and binary map George and Marcel
(2019) are designed for fine-grained learning. In this work, we supervise the
deep networks with pseudo depth map due to its effectiveness.

Due to the high collection cost for spoof attacks, there are limited data
scale and diversity in public datasets. Supervised with small-scale predefined
scenarios and PAs, most existing FAS methods are easy to overfit and
vulnerable to domain shift and unseen attacks. In order to detect unseen
attacks successfully, deep tree network Liu et al. (2019) and adaptive inner-
update meta learning Qin et al. (2020) are developed for zero-shot FAS.
However, it is still urgent to provide larger-scale and richer live/spoof data
for deep models training. Here we consider novel data augmentation for FAS to
tackle this challenge.

Convolution Operators. The convolution operator is commonly used for local
feature representation in modern deep learning framework. Recently, a few
extensions to the vanilla convolution operator have been proposed. In one
direction, pre-defined or learnable local relation is embedded in the
convolution operator. Representative works include Local Binary Convolution
Juefei-Xu et al. (2017) and Gabor Convolution Luan et al. (2018), which is
proposed for local invariance preservation and enhancing the resistance to
spatial changes, respectively. Besides, self-attention layer Parmar et al.
(2019), self-attention block Zhao et al. (2020) and local relation layer Hu et
al. (2019) are designed for mining the local relationship flexibly. Another
direction is to modify the spatial scope for aggregation. Two related works
are dialated convolution Yu and Koltun (2015) and deformable convolution Dai
et al. (2017). However, these convolution operators may not be suitable for
FAS task because of the limited representation capacity for invariant fine-
grained features. In contrast, CDC Yu et al. (2020d, c, b) is proposed for
invariant and detailed features extraction, which is suitable for the FAS
task. In this paper, we devote to improving the vanilla CDC in terms of both
performance and efficiency.

##  3 Methodology

In this section we first briefly review the CDC Yu et al. (2020d), and then
introduce the novel Cross Central Difference Convolution (C-CDC) family in
Sec. 3.2. Based on the C-CDC operators, we propose Dual-Cross Central
Difference Networks in Sec. 3.3. Finally we present the novel data
augmentation strategy in Sec. 3.4.

###  3.1 Preliminary

As the basic operator in deep networks, the vanilla 2D convolution consists of
two main steps: 1) sampling local neighbor region $\mathcal{R}$ over the input
feature map $x$; and then 2) aggregating the sampled values via learnable
weights $w$. As a result, the output feature map $y$ can be formulated as

$y(p_{0})=\sum_{p_{n}\in\mathcal{R}}w(p_{n})\cdot x(p_{0}+p_{n}),$ (1)

where $p_{0}$ denotes the current location on both input and output feature
maps while $p_{n}$ enumerates the locations in $\mathcal{R}$. For instance,
local receptive field region for convolution operator with 3$\times$3 kernel
and dilation 1 is
$\mathcal{R}=\left\\{(-1,-1),(-1,0),\cdots,(0,1),(1,1)\right\\}$.

Different from the vanilla convolution, the CDC introduces central gradient
features to enhance the representation and generalization capacity, which can
be formulated as

$y(p_{0})=\sum_{p_{n}\in\mathcal{R}}w(p_{n})\cdot(x(p_{0}+p_{n})-x(p_{0})).$
(2)

As both intensity-level semantic information and gradient-level detailed clues
are crucial for robust FAS, the generalized CDC operator can be represented by
combination of vanilla convoluiton and CDC

$\begin{split}y(p_{0})&=\theta\cdot\underbrace{\sum_{p_{n}\in\mathcal{R}}w(p_{n})\cdot(x(p_{0}+p_{n})-x(p_{0}))}_{\text{central
difference convolution}}\\\
&\quad+(1-\theta)\cdot\underbrace{\sum_{p_{n}\in\mathcal{R}}w(p_{n})\cdot
x(p_{0}+p_{n})}_{\text{vanilla convolution}},\\\
&=\underbrace{\sum_{p_{n}\in\mathcal{R}}w(p_{n})\cdot
x(p_{0}+p_{n})}_{\text{vanilla
convolution}}+\theta\cdot(\underbrace{-x(p_{0})\cdot\sum_{p_{n}\in\mathcal{R}}w(p_{n}))}_{\text{central
difference term}},\\\ \end{split}$ (3)

where hyperparameter $\theta\in[0,1]$ trade-offs the contribution between
intensity-level and gradient-level information. The higher value of $\theta$
means the more importance of central gradient features. Please note that
$w(p_{n})$ is shared between vanilla convolution and CDC, thus no extra
parameters are added. Henceforth, the generalized CDC will be referred as CDC
directly for clear statement.

Figure 1:  Cross central difference convolution. The C-CDC(HV) in the upper
part calculates the central gradients from the horizontal & vertical neighbors
while the C-CDC(DG) in the lower part from the diagonal neighbors.

###  3.2 Cross Central Difference Convolution

As can be seen from Eq. (3) that CDC aggregates both the vanilla and central
gradient features from entire local neighbor region $\mathcal{R}$, which might
be redundant and hard to be optimized. We assume that exploiting sparse
center-oriented difference features would alleviate local competition and
overfitting issues. Therefore, we propose the sparse Cross Central Difference
Convolution (C-CDC) family, intending to learn more concentrated and intrinsic
features for FAS.

Compared with CDC operating on $\mathcal{R}$, C-CDC prefers to sample a
sparser local region $\mathcal{S}$, which can be formulated as

$y(p_{0})=\sum_{p_{n}\in\mathcal{S}}w(p_{n})\cdot(x(p_{0}+p_{n})-x(p_{0})).$
(4)

To be specific, we decouple $\mathcal{R}$ into two cross neighbor regions,
including 1) horizontal & vertical (HV) cross neighbor regions
$\mathcal{S}_{HV}=\left\\{(-1,0),(0,-1),(0,0),(0,1),(1,0)\right\\}$; and 2)
diagonal (DG) cross neighbor regions
$\mathcal{S}_{DG}=\left\\{(-1,-1),(-1,1),(0,0),(1,-1),(1,1)\right\\}$. In this
way, horizontal & vertical C-CDC and diagonal C-CDC can be represented when
$\mathcal{S}=\mathcal{S}_{HV}$ and $\mathcal{S}=\mathcal{S}_{DG}$,
respectively. Figure 5 illustrates the workflow of the C-CDC(HV) and
C-CDC(DG). Similarly, the generalized C-CDC can be easily formulated when
replacing $\mathcal{R}$ with $\mathcal{S}$ in Eq. (3). We will use this
generalized C-CDC henceforth.

In terms of designs of the sparse local region $\mathcal{S}$, there are also
some other solutions with different neighbor locations or fewer neighbors. The
reason that we consider the cross (i.e., HV and DG) fashions for $\mathcal{S}$
derives from their symmetry, which is beneficial for model convergence and
robust feature representation. The studies about the sparsity and neighbor
locations are shown in Section 4.3 and Appendix A.

Output | DepthNet | CDCN | C-CDN (Ours)
---|---|---|---
$256\times 256$ | $3\times 3\textrm{ conv},64$ | $3\times 3\textrm{ CDC},64$ | $3\times 3\textbf{ C-CDC},64$
| $128\times 128$
---
(Low)
$\begin{bmatrix}[l]3\times 3\textrm{ conv},128\\\ 3\times 3\textrm{ conv},196\\\ 3\times 3\textrm{ conv},128\\\ 3\times 3\textrm{ max pool}\end{bmatrix}$ | $\begin{bmatrix}[l]3\times 3\textrm{ CDC},128\\\ 3\times 3\textrm{ CDC},196\\\ 3\times 3\textrm{ CDC},128\\\ 3\times 3\textrm{ max pool}\end{bmatrix}$ | $\begin{bmatrix}[l]3\times 3\textbf{ C-CDC},128\\\ 3\times 3\textbf{ C-CDC},196\\\ 3\times 3\textbf{ C-CDC},128\\\ 3\times 3\textrm{ max pool}\end{bmatrix}$
| $64\times 64$
---
(Mid)
$\begin{bmatrix}[l]3\times 3\textrm{ conv},128\\\ 3\times 3\textrm{ conv},196\\\ 3\times 3\textrm{ conv},128\\\ 3\times 3\textrm{ max pool}\end{bmatrix}$ | $\begin{bmatrix}[l]3\times 3\textrm{ CDC},128\\\ 3\times 3\textrm{ CDC},196\\\ 3\times 3\textrm{ CDC},128\\\ 3\times 3\textrm{ max pool}\end{bmatrix}$ | $\begin{bmatrix}[l]3\times 3\textbf{ C-CDC},128\\\ 3\times 3\textbf{ C-CDC},196\\\ 3\times 3\textbf{ C-CDC},128\\\ 3\times 3\textrm{ max pool}\end{bmatrix}$
| $32\times 32$
---
(High)
$\begin{bmatrix}[l]3\times 3\textrm{ conv},128\\\ 3\times 3\textrm{ conv},196\\\ 3\times 3\textrm{ conv},128\\\ 3\times 3\textrm{ max pool}\end{bmatrix}$ | $\begin{bmatrix}[l]3\times 3\textrm{ CDC},128\\\ 3\times 3\textrm{ CDC},196\\\ 3\times 3\textrm{ CDC},128\\\ 3\times 3\textrm{ max pool}\end{bmatrix}$ | $\begin{bmatrix}[l]3\times 3\textbf{ C-CDC},128\\\ 3\times 3\textbf{ C-CDC},196\\\ 3\times 3\textbf{ C-CDC},128\\\ 3\times 3\textrm{ max pool}\end{bmatrix}$
$32\times 32$ |  [concat (Low, Mid, High), $384$]
$32\times 32$ | $\begin{bmatrix}[l]3\times 3\textrm{ conv},128\\\ 3\times 3\textrm{ conv},64\\\ 3\times 3\textrm{ conv},1\end{bmatrix}$ | $\begin{bmatrix}[l]3\times 3\textrm{ CDC},128\\\ 3\times 3\textrm{ CDC},64\\\ 3\times 3\textrm{ CDC},1\end{bmatrix}$ | $\begin{bmatrix}[l]3\times 3\textbf{ C-CDC},128\\\ 3\times 3\textbf{ C-CDC},64\\\ 3\times 3\textbf{ C-CDC},1\end{bmatrix}$
# params | $2.25\times 10^{6}$ | $2.25\times 10^{6}$ | $1.25\times 10^{6}$
# FLOPs | $4.8\times 10^{10}$ | $4.8\times 10^{10}$ | $1.78\times 10^{8}$

Table 1: Architectures of DepthNet, CDCN, and the proposed C-CDN. Inside the
brackets are the filter sizes and feature dimensionalities. ‘conv’ suggests
the vanilla convolution. All convolutional layers are with stride=1 and are
followed by a BN-ReLU layer while pooling layers are with stride=2.

###  3.3 Dual-Cross Central Difference Network

Pseudo depth-based supervision takes advantage of the discrimination between
live and spoof faces based on 3D shape, which is able to provide pixel-wise
detailed clues to enforce FAS model to capture intrinsic features. Following
the similar depth-supervised backbone as “DepthNet” Liu et al. (2018) and
“CDCN” Yu et al. (2020d), we replace all the $3\times 3$ convolution operators
with our proposed C-CDC to form the Cross Central Difference Network (C-CDN).
Given a single RGB face image with size $3\times 256\times 256$, multi-level
(low-level, mid-level and high-level) fused features are extracted for
predicting the grayscale facial depth with size $32\times 32$. The details of
C-CDN are shown in Table 1. It can be seen that with the similar architecture
(e.g., network depth and width), C-CDN has only five ninth parameters and two
hundredth computational cost compared with DepthNet and CDCN due to the sparse
local sampling mechanism in C-CDC. We use $\theta=0.8$ as the default setting,
and the corresponding study about $\theta$ will be discussed in Section 4.3.

Although the C-CDC decouples and learns the local gradient features with
particular views (HV and DG), it still suffers from information loss compared
with CDC operating on the full local neighbors. In order to fully exploit the
local features and interact between HV and DG views, a Dual-Cross Central
Difference Network (DC-CDN)111https://github.com/ZitongYu/CDCN is proposed. As
shown in Figure 2, two independent (unshared) networks respectively assembled
with C-CDC(HV) and C-CDC(DG) are used. Then the extracted dual-stream features
from different views are fused for final depth prediction. In this way, the
full neighbor aggregation step can be disentangled into two sub-steps: 1)
sparse neighbor aggregation for individual stream; and 2) dual-stream fusion.

Figure 2:  Dual-cross central difference network with Cross Feature
Interaction Modules for low-mid-high level feature enhancement.

Cross Feature Interaction Module. With only simple late fusion, the
performance improvement might be limited due to the lack of message passing
from the preceding (i.e., low-level, mid-level, and high-level) stages. In
order to effectively mine the relations across dual streams and enhance local
detailed representation capacity, we propose the Cross Feature Interaction
Module (CFIM) to fuse dual-stream multi-level features adaptively. To be
specific, given the two-stream features $\mathcal{F}_{HV}$ and
$\mathcal{F}_{DG}$, the CFIM enhanced features
$\mathcal{F}^{{}^{\prime}}_{HV}$ and $\mathcal{F}^{{}^{\prime}}_{DG}$ can be
formulated as

$\begin{split}\mathcal{F}^{{}^{\prime}}_{HV}=\varphi(\alpha)\cdot\mathcal{F}_{HV}+(1-\varphi(\alpha))\cdot\mathcal{F}_{DG},\\\
\mathcal{F}^{{}^{\prime}}_{DG}=\varphi(\beta)\cdot\mathcal{F}_{DG}+(1-\varphi(\beta))\cdot\mathcal{F}_{HV},\\\
\end{split}$ (5)

where $\varphi()$ denotes the Sigmoid function for [0,1] range mapping.
$\alpha$ and $\beta$ are the attention weights for $\mathcal{F}_{HV}$ and
$\mathcal{F}_{DG}$, respectively. In our default setting, both $\alpha$ and
$\beta$ are initialized to 0 and learnable, which could be adaptively adjusted
during the training iterations. We also investigate the manually fixed
$\alpha$ and $\beta$ in Appendix B. As illustrated in Fig. 2, here we
respectively plug three CFIMs (with learnable $\alpha_{low}$, $\alpha_{mid}$,
$\alpha_{high}$, $\beta_{low}$, $\beta_{mid}$, $\beta_{high}$) in the output
features from low-mid-high levels before multi-level concatenation.

Algorithm 1 Patch Exchange Augmentation

Input: Face images $I$ with batchsize $N$, pseudo depth map labels $D$,
augmented ratio $\gamma\in[0,1]$, step number $\rho$
1 : for  each $I_{i}$ and $D_{i}$, $i=1,...,\left\lfloor\gamma*N\right\rfloor$
do
2 : for each step $\rho$ do
3 : Randomly select a patch region $P$ within $I_{i}$
4 : Randomly select a batch index $j,j\leq N$
5 : Exchange the image patch $I_{i}(P)=I_{j}(P)$ and label patch
$D_{i}(P)=D_{j}(P)$
6 : end
7: end
8: return augmented $I$ and $D$

Figure 3:  Patch Exchange augmentation. Two augmented samples are synthesized
via exchanging the RGB patches as well as corresponding pseudo depth labels
from “Live Domain2” and “Print Attack Domain2”. Thus, the samples
“Augmentation1” and “Augmentation2” contain diverse domains and attack types,
respectively.

###  3.4 Patch Exchange Augmentation

Due to the high collection cost for spoof attacks, there are limited data
scale and diversity in public FAS datasets. In this paper, we also propose a
FAS-dedicated data augmentation method, named Patch Exchanges (PE), to
synthesize mixed samples with diverse attacks and domains. There are three
advantages for PE augmentation: 1) face patches from different domains (e.g.,
recorded scenario, sensor, and subject) are introduced for enriching data
distribution; 2) random live and PA patches are exchanged to mimic arbitrary
partial attacks; and 3) the exchanged patches with corresponding dense labels
enforce the model to learn more detailed and intrinsic features for spoofing
detection. The complete algorithm of PE is summarized in Algorithm 1. The two
hyperparameters $\gamma$ and $\rho$ control the augmentation ratio and
intensity, respectively. As a tradeoff, we use empirical settings $\gamma=0.5$
and $\rho=2$ for experiments.

Note that as the face images $I$ for PE are coarsely aligned, the exchanged
patches would have similar semantic content (e.g., cheek, nose and mouth) but
with diverse live/spoof clues. Thus, the augmented live/spoof faces are still
realistic and even more challenging to be distinguished. Some typical samples
with PE augmentation are visualized in Figure 8.

##  4 Experiments

###  4.1 Datasets and Metrics

Databases. Four databases OULU-NPU Boulkenafet et al. (2017), CASIA-MFSD Zhang
et al. (2012), Replay-Attack Chingovska et al. (2012) and SiW-M Liu et al.
(2019) are used in our experiments. OULU-NPU is a high-resolution database,
containing four protocols to evaluate the generalization (e.g., unseen
illumination and attack medium) of models respectively, which is used for
intra testing. CASIA-MFSD and Replay-Attack are small-scale databases with
low-resolution videos, which are used for cross testing. SiW-M is designed for
cross-type testing for unseen attacks as there are rich (13) attack types
inside.

Performance Metrics. In OULU-NPU dataset, we follow the original protocols and
metrics, i.e., Attack Presentation Classification Error Rate (APCER), Bona
Fide Presentation Classification Error Rate (BPCER), and ACER ISO (2016) for a
fair comparison. Half Total Error Rate (HTER) is adopted in the cross testing
between CASIA-MFSD and Replay-Attack. For the cross-type test on SiW-M, ACER
and Equal Error Rate (EER) are employed for evaluation.

Figure 4:  (a) Impact of $\theta$ in C-CDN. (b) Comparison among various
convolutions. The lower ACER, the better performance.  Table 2: The ablation
study about DC-CDN, CFIM, and PE augmentation on Protocol 1 of OULU-NPU.

Model | CFIM | PE Augmentation | ACER(%)
---|---|---|---
C-CDN(HV) |  |  | 0.7
C-CDN(DG) |  |  | 0.9
DC-CDN (average) |  |  | 1.0
DC-CDN (concat) |  |  | 0.8
DC-CDN (concat) | $\surd$ |  | 0.5
C-CDN(HV) |  | $\surd$ | 0.6
C-CDN(DG) |  | $\surd$ | 0.7
DC-CDN (concat) | $\surd$ | $\surd$ | 0.4

###  4.2 Implementation Details

Depth Generation. Dense face alignment Guo et al. (2020) is adopted for pseudo
depth generation. To clearly distinguish live and spoof faces, at the training
stage, we follow Liu et al. (2018) to normalize the live depth maps in a range
of $[0,1]$, while leaving spoof depth maps to all zeros.

Training and Testing Setting. Our proposed method is implemented with Pytorch.
In the training stage, models are trained with batch size 8 and Adam optimizer
on a single V100 GPU. Data augmentations including horizontal flip, color
jitter and Cutout are used as baseline. The initial learning rate (lr) and
weight decay are 1e-4 and 5e-5, respectively. We train models with maximum 800
epochs while lr halves in the 500th epoch. Similar to Yu et al. (2020d), all
the models are supervised by mean square error (MSE) and contrastive depth
loss (CDL). In the testing stage, we calculate the mean value of the predicted
depth map as the final score.

###  4.3 Ablation Study

In this subsection, all ablation studies are conducted on the Protocol-1
(different illumination condition and scenario between training and testing
sets) of OULU-NPU dataset.

Table 3: The results of intra testing on the OULU-NPU dataset.

Prot. | Method | APCER(%)$\downarrow$ | BPCER(%)$\downarrow$ | ACER(%)$\downarrow$
---|---|---|---|---
1 | STASN Yang et al. (2019) | 1.2 | 2.5 | 1.9
Auxiliary Liu et al. (2018) | 1.6 | 1.6 | 1.6
FaceDs Jourabloo et al. (2018) | 1.2 | 1.7 | 1.5
SpoofTrace Liu et al. (2020) | 0.8 | 1.3 | 1.1
Disentangled Zhang et al. (2020) | 1.7 | 0.8 | 1.3
FAS-SGTD Wang et al. (2020) | 2.0 | 0.0 | 1.0
| CDCN Yu et al. (2020d) | 0.4 | 1.7 | 1.0
| BCN Yu et al. (2020a) | 0.0 | 1.6 | 0.8
| DeepPix George and Marcel (2019) | 0.8 | 0.0 | 0.4
| DC-CDN (Ours) | 0.5 | 0.3 | 0.4
2 | DeepPix George and Marcel (2019) | 11.4 | 0.6 | 6.0
FaceDs Jourabloo et al. (2018) | 4.2 | 4.4 | 4.3
Auxiliary Liu et al. (2018) | 2.7 | 2.7 | 2.7
Disentangled Zhang et al. (2020) | 1.1 | 3.6 | 2.4
STASN Yang et al. (2019) | 4.2 | 0.3 | 2.2
BCN Yu et al. (2020a) | 2.6 | 0.8 | 1.7
| SpoofTrace Liu et al. (2020) | 2.3 | 1.6 | 1.9
| FAS-SGTD Wang et al. (2020) | 2.5 | 1.3 | 1.9
| CDCN Yu et al. (2020d) | 1.5 | 1.4 | 1.5
| DC-CDN (Ours) | 0.7 | 1.9 | 1.3
3 | DeepPix George and Marcel (2019) | 11.7$\pm$19.6 | 10.6$\pm$14.1 | 11.1$\pm$9.4
FaceDs Jourabloo et al. (2018) | 4.0$\pm$1.8 | 3.8$\pm$1.2 | 3.6$\pm$1.6
Auxiliary Liu et al. (2018) | 2.7$\pm$1.3 | 3.1$\pm$1.7 | 2.9$\pm$1.5
STASN Yang et al. (2019) | 4.7$\pm$3.9 | 0.9$\pm$1.2 | 2.8$\pm$1.6
| SpoofTrace Liu et al. (2020) | 1.6 $\pm$1.6 | 4.0$\pm$5.4 | 2.8$\pm$3.3
| FAS-SGTD Wang et al. (2020) | 3.2$\pm$2.0 | 2.2$\pm$1.4 | 2.7$\pm$0.6
| BCN Yu et al. (2020a) | 2.8$\pm$2.4 | 2.3$\pm$2.8 | 2.5$\pm$1.1
| CDCN Yu et al. (2020d) | 2.4$\pm$1.3 | 2.2$\pm$2.0 | 2.3$\pm$1.4
| Disentangled Zhang et al. (2020) | 2.8$\pm$2.2 | 1.7$\pm$2.6 | 2.2$\pm$2.2
| DC-CDN (Ours) | 2.2$\pm$2.8 | 1.6$\pm$2.1 | 1.9$\pm$1.1
4 | DeepPix George and Marcel (2019) | 36.7$\pm$29.7 | 13.3$\pm$14.1 | 25.0$\pm$12.7
Auxiliary Liu et al. (2018) | 9.3$\pm$5.6 | 10.4$\pm$6.0 | 9.5$\pm$6.0
STASN Yang et al. (2019) | 6.7$\pm$10.6 | 8.3$\pm$8.4 | 7.5$\pm$4.7
CDCN Yu et al. (2020d) | 4.6$\pm$4.6 | 9.2$\pm$8.0 | 6.9$\pm$2.9
| FaceDs Jourabloo et al. (2018) | 1.2$\pm$6.3 | 6.1$\pm$5.1 | 5.6$\pm$5.7
| BCN Yu et al. (2020a) | 2.9$\pm$4.0 | 7.5$\pm$6.9 | 5.2$\pm$3.7
| FAS-SGTD Wang et al. (2020) | 6.7$\pm$7.5 | 3.3$\pm$4.1 | 5.0$\pm$2.2
| Disentangled Zhang et al. (2020) | 5.4$\pm$2.9 | 3.3$\pm$6.0 | 4.4$\pm$3.0
| SpoofTrace Liu et al. (2020) | 2.3$\pm$3.6 | 5.2$\pm$5.4 | 3.8$\pm$4.2
| DC-CDN (Ours) | 5.4$\pm$3.3 | 2.5$\pm$4.2 | 4.0$\pm$3.1

Impact of $\theta$ in C-CDC. As discussed in Section 3.2, $\theta$ controls
the contribution of the gradient-based features, i.e., the higher $\theta$,
the more local detailed information included. As illustrated in Fig. 4(a),
when $\theta\geqslant 0.3$, C-CDC(HV) and C-CDC(DG) always achieve better
performance than their vanilla counterpart (i.e., $\theta=0$), indicating the
effectivenss of local gradient features for FAS task. As the best performance
(ACER=0.7% and 0.9% for C-CDC(HV) and C-CDC(DG), respectively) are obtained
when $\theta=0.8$, we use this setting for the following experiments.

Table 4: Results of the cross-type testing on the SiW-M dataset.

Method | Metrics(%) | Replay | Print | Mask Attacks | Makeup Attacks | Partial Attacks | Average
---|---|---|---|---|---|---|---
| Half
---
| Silicone
---
| Trans.
---
| Paper
---
| Manne.
---
| Obfusc.
---
| Im.
---
| Cos.
---
| Fun.
---
| Glasses
---
| Partial
---
Auxiliary Liu et al. (2018) | ACER | 16.8 | 6.9 | 19.3 | 14.9 | 52.1 | 8.0 | 12.8 | 55.8 | 13.7 | 11.7 | 49.0 | 40.5 | 5.3 | 23.6$\pm$18.5
EER | 14.0 | 4.3 | 11.6 | 12.4 | 24.6 | 7.8 | 10.0 | 72.3 | 10.1 | 9.4 | 21.4 | 18.6 | 4.0 | 17.0$\pm$17.7
DTN Liu et al. (2019) | ACER | 9.8 | 6.0 | 15.0 | 18.7 | 36.0 | 4.5 | 7.7 | 48.1 | 11.4 | 14.2 | 19.3 | 19.8 | 8.5 | 16.8 $\pm$11.1
EER | 10.0 | 2.1 | 14.4 | 18.6 | 26.5 | 5.7 | 9.6 | 50.2 | 10.1 | 13.2 | 19.8 | 20.5 | 8.8 | 16.1$\pm$ 12.2
CDCN Yu et al. (2020d) | ACER | 8.7 | 7.7 | 11.1 | 9.1 | 20.7 | 4.5 | 5.9 | 44.2 | 2.0 | 15.1 | 25.4 | 19.6 | 3.3 | 13.6 $\pm$11.7
EER | 8.2 | 7.8 | 8.3 | 7.4 | 20.5 | 5.9 | 5.0 | 47.8 | 1.6 | 14.0 | 24.5 | 18.3 | 1.1 | 13.1$\pm$ 12.6
SpoofTrace Liu et al. (2020) | ACER | 7.8 | 7.3 | 7.1 | 12.9 | 13.9 | 4.3 | 6.7 | 53.2 | 4.6 | 19.5 | 20.7 | 21.0 | 5.6 | 14.2 $\pm$13.2
EER | 7.6 | 3.8 | 8.4 | 13.8 | 14.5 | 5.3 | 4.4 | 35.4 | 0.0 | 19.3 | 21.0 | 20.8 | 1.6 | 12.0$\pm$ 10.0
BCN Yu et al. (2020a) | ACER | 12.8 | 5.7 | 10.7 | 10.3 | 14.9 | 1.9 | 2.4 | 32.3 | 0.8 | 12.9 | 22.9 | 16.5 | 1.7 | 11.2 $\pm$9.2
EER | 13.4 | 5.2 | 8.3 | 9.7 | 13.6 | 5.8 | 2.5 | 33.8 | 0.0 | 14.0 | 23.3 | 16.6 | 1.2 | 11.3$\pm$ 9.5
DC-CDN w/o PE (Ours) | ACER | 12.9 | 10.2 | 9.7 | 9.1 | 16.5 | 5.3 | 1.6 | 44.6 | 0.8 | 14.0 | 22.9 | 17.3 | 3.8 | 12.9 $\pm$11.6
EER | 12.3 | 8.7 | 12.6 | 7.4 | 13.6 | 5.9 | 0.0 | 43.4 | 0.0 | 14.0 | 19.5 | 16.7 | 2.3 | 12.0$\pm$ 11.3
DC-CDN (Ours) | ACER | 12.1 | 9.7 | 14.1 | 7.2 | 14.8 | 4.5 | 1.6 | 40.1 | 0.4 | 11.4 | 20.1 | 16.1 | 2.9 | 11.9$\pm$10.3
EER | 10.3 | 8.7 | 11.1 | 7.4 | 12.5 | 5.9 | 0.0 | 39.1 | 0.0 | 12.0 | 18.9 | 13.5 | 1.2 | 10.8$\pm$10.1

Table 5: The results of cross-dataset testing between CASIA-MFSD and Replay-
Attack. The evaluation metric is HTER(%).

Method | Train | Test | Train | Test
---|---|---|---|---
| CASIA-
---
MFSD
| Replay-
---
Attack
| Replay-
---
Attack
| CASIA-
---
MFSD
FaceDs Jourabloo et al. (2018) | 28.5 | 41.1
STASN Yang et al. (2019) | 31.5 | 30.9
Auxiliary Liu et al. (2018) | 27.6 | 28.4
Disentangled Zhang et al. (2020) | 22.4 | 30.3
FAS-SGTD Wang et al. (2020) | 17.0 | 22.8
BCN Yu et al. (2020a) | 16.6 | 36.4
CDCN Yu et al. (2020d) | 15.5 | 32.6
DC-CDN w/o PE (Ours) | 8.5 | 32.3
DC-CDN (Ours) | 6.0 | 30.1

C-CDC vs. CDC. Here we evaluate the impacts of the neighbor sparsity for both
vanilla and gradient-based convolutions. As shown in Fig. 4(b),
“VanillaConv(HV)” and “VanillaConv(DG)” performs more poorly than
“VanillaConv”, which might be caused by the limited semantic representation
capacity from the sparse neighbor sampling. In contrast, it is interesting to
see that their central difference-based counterparts perform inversely.
Compared with CDC, C-CDC(HV) and C-CDC(DG) have only five ninth parameters,
and even achieve better performance (-0.3% and -0.1% ACER, respectively),
indicating the efficiency of cross gradient features for FAS. More discussion
about the sparsity and neighbor locations will be shown in Appendix A.

Effectiveness of DC-CDN and CFIM. The upper part of Table 2 shows the ablation
results of DC-CDN w/ and w/o CFIM. It can be seen from the third and fourth
rows that DC-CDN w/o CFIM achieves worse when using simple final depth map
averaging or late fusion via concatenation. In contrast, assembled with CFIM,
DC-CDN obtains 0.2% and 0.4% ACER decrease compared with one-stream C-CDN(HV)
and C-CDN(DG), respectively. It demonstrates the importance of adaptive mutual
feature interaction during the training stage. More discussion about the MFIM
and the learned $\alpha,\beta$ visualization are shown in Appendix B.

Effectiveness of PE augmentation. It can be seen from the lower part of Table
2 that PE augmentation improves the performance of both single-stream (C-CDN)
and dual-stream (DC-CDN) models on Protocol 1 (with slight domain shifts) of
OULU-NPU. It is worth noting that, benefited from the augmented data with rich
and diverse domain and attack clues, DC-CDN could obtain remarkable
performance gains on more challenging senarios, including cross-type (-1.2%
EER, see Table 4) and cross-dataset (respective -2.5% and -2.2% HTER, see
Table 5) testings. We also evaluate the generalization of PE for other
architectures in Appendix C.

###  4.4 Comparison with State of the Arts

Intra Testing on OULU-NPU. As shown in Table 3, our proposed DC-CDN ranks
first on the first three protocols (0.4%, 1.3% and 1.9% ACER, respectively),
which indicates the proposed method performs well at the generalization of the
external environment, attack mediums and input camera variation. It is clear
that the proposed DC-CDN consistently outperforms CDCN Yu et al. (2020d) on
all protocols with -0.6%, -0.2%, -0.4%, and -2.9%, respectively, indicating
the superiority of DC-CDN. In the most challenging Protocol 4, DC-CDN also
achieves comparable performance with state-of-the-art SpoofTrace Liu et al.
(2020) but more robustness with smaller ACER standard deviation among six
kinds of unseen scenarios.

Cross-type Testing on SiW-M. Following the leave-one-type-out (total 13 attack
types) protocol on SiW-M, we compare the proposed methods with several recent
FAS methods to validate the generalization capacity of unseen attacks. As
shown in Table 4, our method achieves the best EER performance and can perform
more robustly in several challenging attacks including ‘Silicone Mask’,
‘Transparent Mask’, ‘Mannequin’, ‘Impersonation’, ‘Funny Eye’ and ‘Paper
Glasses’. Note that our DC-CDN could achieve comparable performance with the
state-of-the-art method BCN Yu et al. (2020a), which needs the supervision
signals from three kinds of pixel-wise labels (i.e., binary mask, pseudo depth
and reflection map). Moreover, it is reasonable to see from the last two rows
of Table 4 that PE augmentation helps to improve generalization ability of
unknown Partial Attacks (i.e., Funny Eye, Paper Glasses and Partial Paper)
obviously.

Cross-dataset Testing. Here cross-dataset testing is conducted to further
testify the generalization ability of our models under unseen scenarios. There
are two cross-dataset testing protocols. One is that training on the CASIA-
MFSD and testing on Replay-Attack, which is named as protocol CR; the second
one is exchanging the training dataset and the testing dataset, named protocol
RC. As shown in Table 5, our proposed DC-CDN achieves 6.0% HTER on protocol
CR, outperforming the prior state-of-the-art by a convincing margin of 9.5%.
For protocol RC, we also slightly outperform state-of-the-art frame-level
methods (e.g., CDCN, BCN and Disentangled Zhang et al. (2020)). Note that .
Inspired by the dynamic clues usage in Auxiliary Liu et al. (2018) and FAS-
SGTD Wang et al. (2020), the performance of DC-CDN might be further boosted
via introducing the similar temporal dynamic features, which will be explored
in our future work.

##  5 Conclusions

In this paper, we propose a sparse operator family called Cross Central
Difference Convolution (C-CDC) for face anti-spoofing task. Moreover, we
design a Dual-Cross Central Difference Network with Cross Feature Interaction
Modules for dual-stream feature enhancement. Besides, a novel Patch Exchange
augmentation strategy is proposed for enriching the training data. Extensive
experiments are performed to verify the effectiveness of the proposed methods.
In the future, we will extend the C-CDC family to a 3D version for fine-
grained video understanding tasks.

## Appendix

Figure 5:  Ablation study about the sparsity and neighbor locations of CDC on
Protocol 1 of OULU-NPU. The neighbors with the color yellow are sampled for
central difference convolution. Only the best results with optimal $\theta$
are shown.

## A. Impacts of Sparsity and Neighbor Locations of C-CDC

As already shown in the Section 4.3, C-CDC(HV) and C-CDC(DG) adopt the
symmetric sparse local neighbor sampling and even achieve better performance
than CDC. Thus, two questions arise: 1) the sparser, the better? 2) is
symmetry helpful for feature representation?

In order to tackle these questions, we compare C-CDC with three CDC with
sparse/asymmetric neighbor sampling (i.e., “S1-CDC”, “S2-CDC”, “S3-CDC”). The
results intra-tested on Protocol 1 of OULU-NPU are illustrated in Figure 5. In
terms of the sparsity, the “S2-CDC” and “S3-CDC” respectively consider only
three and two local neighbors for sampling, which is much sparser than both
CDC and C-CDC. However, they suffer from sharp performance drops due to the
limited local representation capacity. As for the symmetry, we compare the
C-CDC with “S1-CDC” as the latter has the same sparsity but asymmetric
neighbor locations. It is interesting to see that “S1-CDC” still has
comparable performance (1% ACER) with CDC but slightly poorer results than the
symmetric C-CDC(HV) and C-CDC(DG).

Despite with such coarse examples for comparison, it still provides evidence
that the designed C-CDC family is an excellent tradeoff in terms of
performance and efficiency.

Figure 6:  Visualization of the learnable $\alpha,\beta$ of CFIM in DC-CDN
during the first 500 training epochs on Protocol 1 of OULU-NPU.

## B. Discussion about CFIM

As CFIM plays a vital role in mutual relation mining and dual-stream feature
enhancement, it is necessary to explore more details about CFIM, e.g., 1)
learned $\alpha,\beta$ visualization; 2) impacts of learnable/fixed setting
and initialization.

As illustrated in Figure 6, with the defaulted initialization to all zeros
(i.e., $\alpha,\beta=0$, $\varphi(\alpha),\varphi(\beta)=0.5$), the trends of
$\alpha_{low}$, $\alpha_{mid}$, $\alpha_{high}$, $\beta_{mid}$, $\beta_{high}$
are relatively steady except $\beta_{low}$. To be specific, $\alpha_{low}$,
$\beta_{mid}$, $\beta_{high}$ prefer to hold the positive values while
$\alpha_{mid}$, $\alpha_{high}$, $\beta_{low}$ keep the negative values in
most times. The dramatic decrease of $\beta_{low}$ denotes that contributions
from the low-level features of the C-CDC(DG) stream might be small in DC-CDN.

Moreover, it can be seen from Figure 7 that the initialization values
influence the CFIM a lot especially with learnable setting. Small initial
values (e.g., $\alpha,\beta=[0,1]$) seem to be more suitable for subsequent
dual-stream feature learning. The best performance could be achieved (0.5%
ACER in Protocol 1 of OULU-NPU) when $\alpha,\beta$ are learnable and
initialized as zeros.

## C. Analysis of PE Augmentation

In the paper, although the PE Augmentation is proven to be helpful for DC-CDN
performance improvement, it is still necessary to evaluate its generalization
ability for different architectures under various scenarios. Besides DC-CDN,
here we also give comparisons of two classical FAS architectures, i.e.,
DepthNet Liu et al. (2018), CDCN Yu et al. (2020d) on intra- and cross-type
testings. As shown in Figure 8, PE augmentation consistently improves DepthNet
and CDCN when the training data are even small-scale (Protocol 4 of OULU-NPU)
or testing under unseen attack types (SiW-M). All these results demonstrate
that PE augmentation is promising for the FAS task.

As partial attacks (e.g., partial print, mask and funny eye glasses) have been
used for hacking the face recognition system, here we show a visualization
about the predictions from DC-CDN trained w/ and w/o PE augmentation. The
models are trained and tested on the sub-Protocol 4-1 of OULU-NPU. As
illustrated in the first three columns of Figure 9, both the models (trained
w/ or w/o PE augmentation) perform well for the normal live face and print
attacks. However, we can see from the last column that w/o PE augmentation, it
fails to capture the spoofing regions from the synthesized face with partial
attacks. In contrast, the model trained w/ PE could clearly discover and
localize them.

Figure 7:  Comparison of the learnable and fixed settings of $\alpha,\beta$ in
CFIM with different initialized values on Protocol 1 of OULU-NPU.  Figure 8:
Ablation study about the PE augmentation for different architectures and
testing protocols.  Figure 9:  Visualization of the predicted results from DC-
CDN w/ and w/o PE augmentation. When given the inputs with partial print
attacks, the model trained w/ PE is able to discover the spoofing patches
while that trained w/o PE fails.

## References

  * Boulkenafet et al. [2015] Zinelabidine Boulkenafet, Jukka Komulainen, and Abdenour Hadid.  Face anti-spoofing based on color texture analysis.  In ICIP, 2015.
  * Boulkenafet et al. [2017] Zinelabinde Boulkenafet, Jukka Komulainen, Lei Li, Xiaoyi Feng, and Abdenour Hadid.  Oulu-npu: A mobile face presentation attack database with real-world variations.  In FGR, 2017.
  * Chingovska et al. [2012] Ivana Chingovska, Andr$\acute{e}$ Anjos, and S$\acute{e}$bastien Marcel.  On the effectiveness of local binary patterns in face anti-spoofing.  In Biometrics Special Interest Group, 2012.
  * Dai et al. [2017] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In CVPR, 2017.
  * de Freitas Pereira et al. [2012] Tiago de Freitas Pereira, Andr$\acute{e}$ Anjos, Jos$\acute{e}$ Mario De Martino, and S$\acute{e}$bastien Marcel.  Lbp- top based countermeasure against face spoofing attacks.  In ACCV, 2012.
  * DeVries and Taylor [2017] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552, 2017.
  * George and Marcel [2019] Anjith George and Sébastien Marcel.  Deep pixel-wise binary supervision for face presentation attack detection.  In ICB, number CONF, 2019.
  * Guo et al. [2020] Jianzhu Guo, Xiangyu Zhu, Yang Yang, Fan Yang, Zhen Lei, and Stan Z Li.  Towards fast, accurate and stable 3d dense face alignment.  In ECCV, 2020.
  * Hu et al. [2019] Han Hu, Zheng Zhang, Zhenda Xie, and Stephen Lin.  Local relation networks for image recognition.  In CVPR, 2019.
  * ISO [2016] ISO.  Iso/iec jtc 1/sc 37 biometrics: Information technology biometric presentation attack detection part 1: Framework.  In https://www.iso.org/obp/ui/iso, 2016.
  * Jourabloo et al. [2018] Amin Jourabloo, Yaojie Liu, and Xiaoming Liu.  Face de-spoofing: Anti-spoofing via noise modeling.  In ECCV, 2018.
  * Juefei-Xu et al. [2017] Felix Juefei-Xu, Vishnu Naresh Boddeti, and Marios Savvides.  Local binary convolutional neural networks.  In CVPR, 2017.
  * Komulainen et al. [2013] Jukka Komulainen, Abdenour Hadid, and Matti Pietikainen.  Context based face anti-spoofing.  In BTAS, 2013.
  * Liu et al. [2018] Yaojie Liu, Amin Jourabloo, and Xiaoming Liu.  Learning deep models for face anti-spoofing: Binary or auxiliary supervision.  In CVPR, 2018.
  * Liu et al. [2019] Yaojie Liu, Joel Stehouwer, Amin Jourabloo, and Xiaoming Liu.  Deep tree learning for zero-shot face anti-spoofing.  In CVPR, pages 4680–4689, 2019.
  * Liu et al. [2020] Yaojie Liu, Joel Stehouwer, and Xiaoming Liu.  On disentangling spoof trace for generic face anti-spoofing.  In ECCV. Springer, 2020.
  * Luan et al. [2018] Shangzhen Luan, Chen Chen, Baochang Zhang, Jungong Han, and Jianzhuang Liu.  Gabor convolutional networks.  TIP, 27(9), 2018.
  * Parmar et al. [2019] Niki Parmar, Prajit Ramachandran, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jon Shlens.  Stand-alone self-attention in vision models.  In NeurPIS, 2019.
  * Patel et al. [2016] Keyurkumar Patel, Hu Han, and Anil K Jain.  Secure face unlock: Spoof detection on smartphones.  TIFS, 11(10), 2016.
  * Qin et al. [2020] Yunxiao Qin, Chenxu Zhao, Xiangyu Zhu, Zezheng Wang, Zitong Yu, Tianyu Fu, Feng Zhou, Jingping Shi, and Zhen Lei.  Learning meta model for zero-and few-shot face anti-spoofing.  In AAAI, 2020.
  * Wang et al. [2020] Zezheng Wang, Zitong Yu, Chenxu Zhao, Xiangyu Zhu, Yunxiao Qin, Qiusheng Zhou, Feng Zhou, and Zhen Lei.  Deep spatial gradient and temporal depth learning for face anti-spoofing.  In CVPR, 2020.
  * Yang et al. [2019] Xiao Yang, Wenhan Luo, Linchao Bao, Yuan Gao, Dihong Gong, Shibao Zheng, Zhifeng Li, and Wei Liu.  Face anti-spoofing: Model matters, so does data.  In CVPR, 2019.
  * Yu and Koltun [2015] Fisher Yu and Vladlen Koltun.  Multi-scale context aggregation by dilated convolutions.  arXiv preprint arXiv:1511.07122, 2015.
  * Yu et al. [2020a] Zitong Yu, Xiaobai Li, Xuesong Niu, Jingang Shi, and Guoying Zhao.  Face anti-spoofing with human material perception.  In ECCV. Springer, 2020.
  * Yu et al. [2020b] Zitong Yu, Yunxiao Qin, Xiaobai Li, Zezheng Wang, Chenxu Zhao, Zhen Lei, and Guoying Zhao.  Multi-modal face anti-spoofing based on central difference networks.  In CVPRW, 2020.
  * Yu et al. [2020c] Zitong Yu, Jun Wan, Yunxiao Qin, Xiaobai Li, Stan Z Li, and Guoying Zhao.  Nas-fas: Static-dynamic central difference network search for face anti-spoofing.  IEEE TPAMI, 2020.
  * Yu et al. [2020d] Zitong Yu, Chenxu Zhao, Zezheng Wang, Yunxiao Qin, Zhuo Su, Xiaobai Li, Feng Zhou, and Guoying Zhao.  Searching central difference convolutional networks for face anti-spoofing.  In CVPR, 2020.
  * Yu et al. [2021] Zitong Yu, Xiaobai Li, Jingang Shi, Zhaoqiang Xia, and Guoying Zhao.  Revisiting pixel-wise supervision for face anti-spoofing.  IEEE TBIOM, 2021.
  * Zhang et al. [2012] Zhiwei Zhang, Junjie Yan, Sifei Liu, Zhen Lei, Dong Yi, and Stan Z Li.  A face antispoofing database with diverse attacks.  In ICB, 2012.
  * Zhang et al. [2020] Ke-Yue Zhang, Taiping Yao, Jian Zhang, Ying Tai, Shouhong Ding, Jilin Li, Feiyue Huang, Haichuan Song, and Lizhuang Ma.  Face anti-spoofing via disentangled representation learning.  In ECCV. Springer, 2020.
  * Zhao et al. [2020] Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun.  Exploring self-attention for image recognition.  In CVPR, 2020.