Roman CCS White Paper

New Compact Object Binary Populations with Precision Astrometry

Roman Core Community Survey: Galactic Bulge Time Domain Survey (GBTDS)

Scientific Categories: Stellar physics and stellar types

Additional scientific keywords: Astrometry, Compact Objects, Black holes,
Neutron stars, Binary stars

Submitting Author:

Name: P. Gandhi Affiliation: University of Southampton, Highfield SO17 1BJ, UK
Email<EMAIL_ADDRESS>
Contributing authors: C. Dashwood Brown (Univ. Southampton)
Y. Zhao (Univ. Southampton)
K. El-Badry (Harvard Univ. CfA)
T.J. Maccarone (Texas Tech)
C. Knigge (Univ. Southampton)
J. Anderson (STScI)
M. Middleton (Univ. Southampton)
J.C.A. Miller-Jones (ICRAR, Curtin University)

Abstract: Compact object binaries (a black hole or a neutron star orbiting a
non-degenerate stellar companion) are key to our understanding of late massive
star evolution, in addition to being some of the best probes of extreme
gravity and accretion physics. Gaia has opened the door to astrometric studies
of these systems, enabling geometric distance measurements, kinematic
estimation, and the ability to find new previously unknown systems through
measurement of binary orbital elements. Particularly puzzling are newly found
massive black holes in wide orbits ($\sim$ AU or more) whose evolutionary
history is difficult to explain. Astrometric identification of such binaries
is challenging for Gaia, with only two such examples currently known. Roman’s
enormous grasp, superb sensitivity, sharp PSF and controlled survey strategy
can prove to be a game-changer in this field, extending astrometric studies of
compact object binaries several mag deeper than Gaia. We propose to use the
microlensing Galactic Bulge Time Domain Survey to identify new wide-orbit
black hole compact object binaries, determine their prevalence and their
spatial distribution, thus opening up new parameter space in binary population
studies.

##  1 Introduction

Most massive stars exist in binaries, currently the best systems for
identifying compact objects from the X-ray power they emit during accretion
[7]. The more massive compact progenitor evolves and interacts with the
stellar companion, either via Roche lobe overflow (where the star has expanded
beyond its Roche lobe, resulting in mass transfer onto the compact object) or
through a common envelope phase (where both stellar cores orbiting within the
expanding envelope of the mass-losing supergiant). The compact object
progenitor continues to evolve and eventually undergoes a supernova, forming a
neutron star or black hole. The evolution of these binary systems is complex
and many factors remain poorly constrained, including the impact of natal
kicks, of metallicity on massive star wind loss, and the impact of common
envelope evolution, amongst others [30, 25, 21, 3]. The more massive amongst
these binaries are potential progenitors for the LIGO/Virgo/Kagra
gravitational wave [GW] population, and factors such as natal kicks can
dramatically change predictions for GW merger rates [3]. Understanding these
factors is thus critical and can be done by detailed follow-up of known X-ray
binaries with electromagnetic follow-up and precision astrometry [4, 6, 18,
15, 20, 14, 2, 22].

But our knowledge of compact objects in the Milky Way remains strongly
incomplete. For instance, estimates of the number of accreting compact objects
in binary systems with a donor star range over 103-8 [24, 10]. About 500
candidates are known, mostly discovered from their X-ray emission when
accreting material from a companion star, with only $\sim$ 30 dynamically
confirmed as hosting a black hole [BH] [10]. Studies suggest that there may be
an even larger populations of non-interacting compact object binaries with no
ongoing accretion activity [1, 9].

Recently, a handful of candidate non-interacting systems have been discovered,
though the veracity of some remains under contention [29, 5]. Amongst these
are two black hole systems discovered astrometrically from the Gaia DR3 non-
single star solutions [17], which allow measurement of the binary orbital
parameters leading to mass estimation. These systems, Gaia-BH1 [12] and Gaia-
BH2 [28, 11], lie in extremely wide orbits (period $P$ = 186 d and 1,277 d,
corresponding to projected semi-major axes of $a$ $\approx$ 1.4 AU and 5 AU,
respectively). These are at least an order-of-magnitude wider than most known
X-ray binaries (Fig. 1). Optical spectroscopy of the companion places direct
constraints on the black hole mass in each case.

Simulations suggest that wide period systems must have received small natal
kicks ($\sim$ a few–a few 10s of km s-1) and ought to preferentially exist
with low mass-ratios (i.e. the ratio of the companion to compact object mass
$M_{2}$/$M_{1}$ $<$ 5). By contrast, the nominal mass ratio of Gaia-BH1 is
$\sim$10, so this system is difficult to explain based on current
understanding of binary evolution. Furthermore, the runaway motion of Gaia-BH1
($\gtrsim 70\,\mathrm{km~{}s^{-1}}$ in excess of Galactic rotation) is also
hard to reconcile with a low natal kick (Zhao et al., in prep.).

Figure 1: Mass vs. Orbital Period for dynamically confirmed black holes in
X-ray binaries [10] and for the two recently discovered ultra-wide systesm
Gaia-BH1 and BH2. These may be the tip of the iceberg of the wide-orbit black
hole binary population. Figure from [11].

‘Standard’ formation channels would predict that progenitors of Gaia-BH1 would
have either shrunk to smaller orbits during a common envelope phase, or have
merged. Alternatively, if the progenitor binary had a very large initial
period $>$3000 days, then it may have evolved without interaction – a channel
known as ‘direct-supernova’ evolution [19]. But eventually the primary
undergoes a core-collapse supernova, and population synthesis studies suggest
the majority ($>$80%) of these binaries will be disrupted by the supernova.
Gaia-BH2, with its longer orbit, is even more difficult to explain under
canonical formation channels. These systems highlight how much remains to be
understood about binary evolution – whether the system interacts prior to
supernova or not cannot be determined from the available data. The observed
masses of the binary components seems more akin to the simulated systems that
do not experience a common envelope phase, and one might naively assume this
is a smoking gun for that evolutionary pathway. However, neither the
kinematics nor the system parameters (e.g. eccentricity) can be easily
reproduced in either scenario.

These systems could be the tip of the iceberg of a new underlying population
just starting to be probed with precision astrometry. Identifying more wide-
period binaries will greatly improve our knowledge of binary evolution,
including constraining the role and distribution of natal kicks imparted on
compact objects, and the preferred evolutionary channels prior to supernova.
This is what we propose to do with Roman.

##  2 How the capabilities of a Roman Survey will uniquely enable the
investigation

Our assumed survey and astrometric precision baseline parameters are listed in
Table 1, and are informed by the expectations for the Galactic Bulge
microlensing survey [23], together with considerations of systematic
uncertainties that may impact the overall final accuracy [31, 27].

Table 1: Baseline astrometric precision and accuracy expectations for Roman observations. Objective | Expected performance
---|---
Detector Plate scale | 110 mas
PSF (F146) | 105 mas
Best single-exposure centroiding precision | 0.01 pixels $\equiv$ 1.1 mas
Final parallax precision |  $\approx$ 0.005 mas
Seasonal astrometric precision ($\sigma_{\rm ast}$) |  $\approx$ 0.013 mas
Absolute a posteriori accuracy, from fitting Gaia field stars |  $\approx$ 0.05 mas
Photometric precision ($\sigma_{\rm F146,\,phot}$) | 0.01 @$m_{\rm AB}$ $\approx$ 23 (AB)
Sky coverage ($A$) | 2 deg2
Number of field stars | 100 million @$m_{\rm AB}$ $\approx$ 23 (AB)
Cadence | 15 min for 72 days ($\times$ 6 seasons)

The critical criterion for finding compact objects in binaries is high
photometric precision. At the distance of the Galactic centre $d$ = 8 kpc, a
star in a binary orbit with semi-major axis $a$ = 1 AU (say) will display an
angular wobble $\omega$ = 0.125 mas. In order to trace this orbit
astrometrically, the demanded positional accuracy must be a small fraction of
this.

Orbital tracing also requires relatively high observational cadence. Gaia-BH1,
for instance, has a period $P$ $\approx$ 6 months. Furthermore, knowing the
source distance allows inference of the binary physical separation $a$ from
the angular measurements. This, in turn, requires robust parallax estimation.

The proposed framework of the Roman microlensing GBTDS can fulfil these
criteria. The final predicted parallax precision (Table 1) is more than an
order of magnitude better than that needed for estimating source distances out
to the Galactic centre. The precision with which orbits can be traced can be
approximated by taking the seasonal data as the relevant baseline. We assume a
best single-exposure angular precision of 0.01 pixel, or about 1.1 mas [31],
and a subsequent gain of $\sqrt{N_{\rm exp}}$. Over a 72 day season, the
number of exposures gathered will be $N_{\rm exp}$ = 6912, resulting in a gain
factor of 83, i.e. a seasonal angular precision of $\sigma_{\rm ast}$ = 0.013
mas, yielding enough resolution for wobble estimation.

##  3 Figure of Merit

A detailed selection function of compact object binaries will require
knowledge of the space density of objects like Gaia-BH1, which remains
unknown. For the purpose of planning Roman observations, we define a
straightforward, albeit simplistic, figure-of-merit (FoM) here as Figure of
Merit: Number of successful compact object binary identifications.

${\rm FoM}=N\times{\rm A}\times\frac{1}{\sigma_{\rm
ast}^{p}}\times\frac{1}{\sigma_{M_{2}}},$ (1)

with $N$ being a normalisation factor which we ignore for the purposes of this
discussion. The astrometric precision enters Eq. 1 as $\sigma_{\rm ast}^{p}$.
The value of the exponent $p$ will depend upon the spatial density of the
targets of interest. Massive compact objects are very likely to be enhanced in
the Galactic centre plane and bulge regions relative to the Solar
neighbourhood, implying that their spatial density will increase with distance
from Earth. Measurement of the orbital elements (including eccentricity and
wobble) places further demands on astrometric precision, so $p$ is expected to
be a positive value greater than 1. We here propose $p$ = 3 for FoM
optimisation, based on the ansatz that the spatial density in the Galactic
disc likely scales down as distance from the Galactic centre to the second
power, and we require one additional exponent power for orbital element
estimation. Higher $p$ values will simply give this factor greater weight.

The final factor in the FoM denotes uncertainty estimates on the luminous
companion star mass ($M_{2}$). This is required to translate the astrometric
mass function, which places only a lower limit on the compact object mass
$M_{1}$, to a direct constraint on $M_{1}$ [1]. Stellar mass constraints will
be non-trivial, with the most robust constraints coming from future
spectroscopic follow-up with facilities like the Extremely Large Telescope, or
through deep space-based exposures. Initial estimates could be based on
colour-magnitude analysis from multi-band photometry, so here we propose

$\frac{1}{\sigma_{M_{2}}}\,=\,\sqrt{\sum_{f}{\frac{1}{\sigma_{f,\,{\rm
phot}}^{2}}}}$ (2)

This equates the mass determination accuracy to the root mean square (r.m.s)
photometric accuracy in all filters ($f$) observed, again ignoring any
normalisation factors. GBTDS is likely to observe in at least two, and
probably more, filters, allowing initial estimates of $\sigma_{M_{2}}$, and
thus FoM.

##  4 Observational strategy considerations

The fiducial proposed strategy of the microlensing survey would satisfy the
requirements for uncovering the compact object binaries that we are interested
in. But the strategy is not final, and we briefly outline how changing the
survey parameters impact deliverables.

Whilst Gaia has been a superb astrometric machine, its observations are not
custom tailored in any specific way. This means that there remain many
systematic uncertainties in the data, resulting in a complex selection
function [8]. Upcoming Gaia data releases (DR4, and potentially, DR5) will
likely reveal more of the sources that we seek, but the selection function is
still likely to remain plagued with similar uncertainties.

By contrast, the proposed GBTDS focus on a limited set of fields, the huge
grasp and direct imaging capabilities should yield a superior selection
function, facilitating a range of population studies that are difficult with
Gaia. The most critical requirement for optimising the FoM is, without doubt,
knowledge of the PSF centroiding precision. But maintaining the PSF stability
across observational seasons would boost FoM while also aiding in the
selection function definition. Photometric precision plays a secondary, albeit
important role.

Pushing to more sensitive mag limits would allow probing deeper into the
bulge, which is likely to be rich in compact objects. But there would need to
be a compromise between increased crowding and the extremes demands of
astrometry at deeper limits. Understanding the impact of crowding will be
important for our science. Expanding the survey to new areas (larger $A$ in
Eq. 1) could instead prove more beneficial.

More exposure time weighting to the shorter wavelength filters will provide a
gain in terms of sharper PSF localisation. There would be a trade-off with the
increased impact of Galactic extinction and our preferred filters would be
F087 or longer.

Finally, a longer duration survey in future years will allow sampling of the
upper end of the period distribution. Gaia-BH1’s orbital period is about 6
months. Gaia-BH2, by contrast, has period $P$ = 1,277 days, so the GBTDS in
its proposed incarnation (Table 1) cover only half an orbit. With enough
signal-to-noise, it is still possible to find such long period systems, though
their characterisation will not be as robust.

##  5 Ancillary Science

The microlensing survey will uncover not only exoplanets, but also free-
floating compact object lenses. Only a single isolated BH has so far been
confirmed [26]. These isolated systems could have been born in binaries, but
ejected following supernova. There are almost no firm constraints on
connections between the isolated and bound compact object populations at this
time. Expanding the samples of both will address this.

Furthermore, self-lensing binaries, in which an edge-on inclination results in
photometric lensing of the companion star at superior conjunction will also be
uncovered [32]. This is an independent technique for identifying compact
objects, and will thus be free of any systematic biases impacting astrometric
identification. Approximately $\sim$ 1 in 100 compact object binary systems
will lie at an edge-on inclination angle as seen from Roman, allowing the
possibility of self-lensing. Combining the self-lensing with the astrometric
constraints for these should yield unprecedented, independent constraints on
the system and component parameters.

Finally, new quiescent accreting systems could also be directly identified
from their photometric and astrometric properties [13]. The population of
accreting systems is also expected to rise towards the Galactic Bulge [16],
and proper characterisation of such systems with Roman will enable removing of
contaminants such as active and variable stars and cataclysmic variables.

## References

  * [1] J. J. Andrews, K. Breivik, and S. Chatterjee.  Weighing the Darkness: Astrometric Mass Measurement of Hidden Stellar Companions Using Gaia.  ApJ, 886(1):68, Nov. 2019.
  * [2] P. Atri, J. C. A. Miller-Jones, A. Bahramian, R. M. Plotkin, P. G. Jonker, G. Nelemans, T. J. Maccarone, G. R. Sivakoff, A. T. Deller, S. Chaty, M. A. P. Torres, S. Horiuchi, J. McCallum, T. Natusch, C. J. Phillips, J. Stevens, and S. Weston.  Potential kick velocity distribution of black hole x-ray binaries and implications for natal kicks.  MNRAS, 489(3):3116–3134, November 2019.
  * [3] K. Belczynski, J. Klencki, C. E. Fields, A. Olejak, E. Berti, G. Meynet, C. L. Fryer, D. E. Holz, R. O’Shaughnessy, D. A. Brown, T. Bulik, S. C. Leung, K. Nomoto, P. Madau, R. Hirschi, E. Kaiser, S. Jones, S. Mondal, M. Chruslinska, P. Drozda, D. Gerosa, Z. Doctor, M. Giersz, S. Ekstrom, C. Georgy, A. Askar, V. Baibhav, D. Wysocki, T. Natan, W. M. Farr, G. Wiktorowicz, M. Coleman Miller, B. Farr, and J. P. Lasota.  Evolutionary roads leading to low effective spins, high black hole masses, and O1/O2 rates for LIGO/Virgo binary black holes.  A&A, 636:A104, Apr. 2020.
  * [4] A. Blaauw.  On the origin of the o- and b-type stars with high velocities (the “run-away” stars), and some related problems.  Bull. Astron. Instit. Neth., 15:265, May 1961.
  * [5] J. Bodensteiner, M. Heida, M. Abdul-Masih, D. Baade, G. Banyard, D. M. Bowman, M. Fabry, A. Frost, L. Mahy, P. Marchant, A. Mérand, M. Reggiani, T. Rivinius, H. Sana, F. Selman, and T. Shenar.  Detecting Stripped Stars While Searching for Quiescent Black Holes.  Msngr, 186:3–9, Mar. 2022.
  * [6] N. Brandt and P. Podsiadlowski.  The effects of high-velocity supernova kicks on the orbital properties and sky distributions of neutron-star binaries.  MNRAS, 274(2):461–484, May 1995.
  * [7] J. Casares and P. G. Jonker.  Mass Measurements of Stellar and Intermediate-Mass Black Holes.  SSR, 183(1-4):223–252, Sept. 2014.
  * [8] A. Castro-Ginard, A. G. A. Brown, Z. Kostrzewa-Rutkowska, T. Cantat-Gaudin, R. Drimmel, S. Oh, V. Belokurov, A. R. Casey, M. Fouesneau, S. Khanna, A. M. Price-Whelan, and H. W. Rix.  Estimating the selection function of Gaia DR3 sub-samples.  arXiv e-prints, page arXiv:2303.17738, Mar. 2023.
  * [9] C. Chawla, S. Chatterjee, K. Breivik, C. K. Moorthy, J. J. Andrews, and R. E. Sanderson.  Gaia May Detect Hundreds of Well-characterized Stellar Black Holes.  ApJ, 931(2):107, June 2022.
  * [10] J. M. Corral-Santana, J. Casares, T. Muñoz-Darias, F. E. Bauer, I. G. Martínez-Pais, and D. M. Russell.  Blackcat: A catalogue of stellar-mass black holes in x-ray transients.  A&A, 587:A61, March 2016.
  * [11] K. El-Badry, H.-W. Rix, Y. Cendes, A. C. Rodriguez, C. Conroy, E. Quataert, K. Hawkins, E. Zari, M. Hobson, K. Breivik, A. Rau, E. Berger, S. Shahaf, R. Seeburger, K. B. Burdge, D. W. Latham, L. A. Buchhave, A. Bieryla, D. Bashi, T. Mazeh, and S. Faigler.  A red giant orbiting a black hole.  MNRAS, 521(3):4323–4348, May 2023.
  * [12] K. El-Badry, H.-W. Rix, E. Quataert, A. W. Howard, H. Isaacson, J. Fuller, K. Hawkins, K. Breivik, K. W. K. Wong, A. C. Rodriguez, C. Conroy, S. Shahaf, T. Mazeh, F. Arenou, K. B. Burdge, D. Bashi, S. Faigler, D. R. Weisz, R. Seeburger, S. Almada Monter, and J. Wojno.  A sun-like star orbiting a black hole.  MNRAS, 518(1):1057–1085, January 2023.
  * [13] P. Gandhi, D. A. H. Buckley, P. A. Charles, S. Hodgkin, S. Scaringi, C. Knigge, A. Rao, J. A. Paice, and Y. Zhao.  Astrometric excess noise in Gaia EDR3 and the search for X-ray binaries.  MNRAS, 510(3):3885–3895, Mar. 2022.
  * [14] P. Gandhi, A. Rao, M. A. C. Johnson, J. A. Paice, and T. J. Maccarone.  Gaia data release 2 distances and peculiar velocities for galactic black hole transients.  MNRAS, 485(2):2642–2655, May 2019.
  * [15] N. Giacobbo and M. Mapelli.  The progenitors of compact-object binaries: impact of metallicity, common envelope and natal kicks.  MNRAS, 480(2):2011–2030, Oct. 2018.
  * [16] H. J. Grimm, M. Gilfanov, and R. Sunyaev.  The Milky Way in X-rays for an outside observer. Log(N)-Log(S) and luminosity function of X-ray binaries from RXTE/ASM data.  A&A, 391:923–944, Sept. 2002.
  * [17] J.-L. Halbwachs, D. Pourbaix, F. Arenou, L. Galluccio, P. Guillout, N. Bauchet, O. Marchal, G. Sadowski, and D. Teyssier.  Gaia Data Release 3. Astrometric binary star processing.  arXiv e-prints, page arXiv:2206.05726, June 2022.
  * [18] G. Hobbs, D. R. Lorimer, A. G. Lyne, and M. Kramer.  A statistical study of 233 pulsar proper motions.  MNRAS, 360(3):974–992, July 2005.
  * [19] V. Kalogera.  Formation of Low-Mass X-Ray Binaries. III. A New Formation Mechanism: Direct Supernova.  ApJ, 493(1):368–374, Jan. 1998.
  * [20] F. Mirabel.  The formation of stellar black holes.  NAR, 78:1–15, August 2017.
  * [21] G. Nelemans, T. M. Tauris, and E. P. J. van den Heuvel.  Constraints on mass ejection in black hole formation derived from black hole X-ray binaries.  A&A, 352:L87–L90, Dec. 1999.
  * [22] T. N. O’Doherty, A. Bahramian, J. C. A. Miller-Jones, A. J. Goodwin, I. Mandel, R. Willcox, P. Atri, and J. Strader.  An observationally-derived kick distribution for neutron stars in binary systems.  MNRAS, March 2023.
  * [23] M. T. Penny, B. S. Gaudi, E. Kerins, N. J. Rattenbury, S. Mao, A. C. Robin, and S. Calchi Novati.  Predictions of the WFIRST Microlensing Survey. I. Bound Planet Detection Rates.  ApJS, 241(1):3, Mar. 2019.
  * [24] E. Pfahl, S. Rappaport, and P. Podsiadlowski.  The Galactic Population of Low- and Intermediate-Mass X-Ray Binaries.  ApJ, 597(2):1036–1048, Nov. 2003.
  * [25] S. F. Portegies Zwart, F. Verbunt, and E. Ergma.  The formation of black-holes in low-mass X-ray binaries.  A&A, 321:207–212, May 1997.
  * [26] K. C. Sahu, J. Anderson, S. Casertano, H. E. Bond, A. Udalski, M. Dominik, A. Calamida, A. Bellini, T. M. Brown, M. Rejkuba, V. Bajaj, N. Kains, H. C. Ferguson, C. L. Fryer, P. Yock, P. Mróz, S. Kozłowski, P. Pietrukowicz, R. Poleski, J. Skowron, I. Soszyński, M. K. Szymański, K. Ulaczyk, Ł. Wyrzykowski, R. K. Barry, D. P. Bennett, I. A. Bond, Y. Hirao, S. I. Silva, I. Kondo, N. Koshimoto, C. Ranc, N. J. Rattenbury, T. Sumi, D. Suzuki, P. J. Tristram, A. Vandorou, J.-P. Beaulieu, J.-B. Marquette, A. Cole, P. Fouqué, K. Hill, S. Dieters, C. Coutures, D. Dominis-Prester, C. Bennett, E. Bachelet, J. Menzies, M. Albrow, K. Pollard, A. Gould, J. C. Yee, W. Allen, L. A. Almeida, G. Christie, J. Drummond, A. Gal-Yam, E. Gorbikov, F. Jablonski, C.-U. Lee, D. Maoz, I. Manulis, J. McCormick, T. Natusch, R. W. Pogge, Y. Shvartzvald, U. G. Jørgensen, K. A. Alsubai, M. I. Andersen, V. Bozza, S. C. Novati, M. Burgdorf, T. C. Hinse, M. Hundertmark, T.-O. Husser, E. Kerins, P. Longa-Peña, L. Mancini, M. Penny, S. Rahvar, D. Ricci, S. Sajadian, J. Skottfelt, C. Snodgrass, J. Southworth, J. Tregloan-Reed, J. Wambsganss, O. Wertz, Y. Tsapras, R. A. Street, D. M. Bramich, K. Horne, I. A. Steele, and RoboNet Collaboration.  An isolated stellar-mass black hole detected through astrometric microlensing.  ApJ, 933(1):83, July 2022.
  * [27] M. Sosey, A. Bellini, D. Nadezhda, and C. Stefano.  Astrometry with roman.  Roman Tech. Rpt., Roman-STScI-000243:SC–01, 2021.
  * [28] A. Tanikawa, K. Hattori, N. Kawanaka, T. Kinugawa, M. Shikauchi, and D. Tsuna.  Search for a Black Hole Binary in Gaia DR3 Astrometric Binary Stars with Spectroscopic Data.  ApJ, 946(2):79, Apr. 2023.
  * [29] T. A. Thompson, C. S. Kochanek, K. Z. Stanek, C. Badenes, R. S. Post, T. Jayasinghe, D. W. Latham, A. Bieryla, G. A. Esquerdo, P. Berlind, M. L. Calkins, J. Tayar, L. Lindegren, J. A. Johnson, T. W. S. Holoien, K. Auchettl, and K. Covey.  A noninteracting low-mass black hole-giant star binary system.  Science, 366(6465):637–640, Nov. 2019.
  * [30] F. Verbunt and E. P. J. van den Heuvel.  Formation and evolution of neutron stars and black holes in binaries.  In X-ray Binaries, pages 457–494, Jan. 1995.
  * [31] WFIRST Astrometry Working Group, R. E. Sanderson, A. Bellini, S. Casertano, J. R. Lu, P. Melchior, M. Libralato, D. Bennett, M. Shao, J. Rhodes, S. T. Sohn, S. Malhotra, S. Gaudi, S. M. Fall, E. Nelan, P. Guhathakurta, J. Anderson, and S. Ho.  Astrometry with the Wide-Field Infrared Space Telescope.  Journal of Astronomical Telescopes, Instruments, and Systems, 5:044005, Oct. 2019.
  * [32] G. Wiktorowicz, M. Middleton, N. Khan, A. Ingram, P. Gandhi, and H. Dickinson.  Predicting the self-lensing population in optical surveys.  MNRAS, 507(1):374–384, Oct. 2021.


# Robust face anti-spoofing framework with Convolutional Vision Transformer

Yunseung Lee  Youngjun Kwak  Jinho Shin

###### Abstract

Owing to the advances in image processing technology and large-scale datasets,
companies have implemented facial authentication processes, thereby
stimulating increased focus on face anti-spoofing (FAS) against realistic
presentation attacks. Recently, various attempts have been made to improve
face recognition performance using both global and local learning on face
images; however, to the best of our knowledge, this is the first study to
investigate whether the robustness of FAS against domain shifts is improved by
considering global information and local cues in face images captured using
self-attention and convolutional layers. This study proposes a convolutional
vision transformer-based framework that achieves robust performance for
various unseen domain data. Our model resulted in 7.3%$p$ and 12.9%$p$
increases in FAS performance compared to models using only a convolutional
neural network or vision transformer, respectively. It also shows the highest
average rank in sub-protocols of cross-dataset setting over the other nine
benchmark models for domain generalization.

Face Anti-Spoofing, Domain Generalization, Convolutional Vision Transformer,
Global and Local Features, Hybrid Feature Extraction

##  1 Introduction

Figure 1: The main idea of our framework for FAS comprises a hybrid feature
extraction and training mechanism with a regression-based DG method. These
modules lead our framework to capture more generalized feature space and to
achieve robust performance on unseen target data despite distribution shifts
between source and target data.

Face anti-spoofing (FAS) aims to detect attacks that occur during face
recognition. This is essential for the security of facial authentication
systems because it is the first stage of the process (Anthony et al., 2021).
Recently, as more diverse and sophisticated presentation attacks, such as
printed images and video replay, threaten the system, the need for robust FAS
algorithms is increasing (Yu et al., 2022). It is essential to develop a model
that considers scenarios where the distribution of training data is distinct
from that of data flowing into a real-time system to ensure the FAS model
performs optimally in real service. Therefore, a training mechanism based on
domain generalization (DG) should be considered (Anthony et al., 2021; Yu et
al., 2022; Shao et al., 2019; Jia et al., 2020; Liu et al., 2021a; Wang et
al., 2022b). Consistent with previous studies, our study aims to develop a
generalized FAS model to handle domain shifts. To achieve the robustness of
FAS, we designed a framework to extract rich information from images and learn
a generalized feature space that is irrelevant to the domain- and person-
specific attributes, as shown in Figure 1.

Maximizing the extraction of meaningful information from an image leads to
better performance in a range of vision tasks, such as FAS. Therefore,
research has focused on identifying new backbones to learn high-quality
features. For example, convolution neural networks (CNNs), which are widely
used as a representative backbone for FAS are advantageous, in extracting
local information (Nagpal & Dubey, 2019). Recently, vision transformer (ViT),
designed to encode the global context between image patches based on the self-
attention (SA) mechanism, has emerged as a new backbone for facial
authentication (George & Marcel, 2021; Huang et al., 2022; Ming et al., 2022).
Despite a research report indicating improved FAS performance when utilizing
both local and global information in an intra-dataset protocol (Wang et al.,
2022a), empirical studies on this are scarce, and prior studies have not
confirmed the performance in terms of DG.

Figure 2:  A robust FAS framework based on ConViT aims to extract local and
global features with gated positional self-attention mechanisms and to learn
feature space for domain generalization through liveness prediction with
discretized labels.

Few studies have focused on unknown attacks that partially cover domain
shifts, with $N$-shot learning (George & Marcel, 2021; Huang et al., 2022).
Our research differs from these studies in that they only use global
dependencies and overlook local information. As an example of FAS for video,
there is a study that considers the effectiveness of feature extraction for DG
and utilizes SA in ViT to learn spatio-temporal relationships in videos with
frames (Ming et al., 2022). However, no attempt in FAS for images has been
made for feature extraction to encode both local cues and global dependencies
with ViT architecture.

The novelty of this study lies in bridging the gap between research on
improving FAS performance with global and local information and maintaining
the robustness of FAS with domain-generalized features. To the best of our
knowledge, this is the first study to propose a robust FAS model in terms of
DG using ViT structure that encodes both global and local contexts from a
single image. To make our framework sufficiently robust for DG, we designed it
to extract rich features using convolutional vision transformer (ConViT) and
to train it with a regression-based method as well as adversarial learning.
Under cross-dataset protocol, our proposed framework reports a higher Area
Under Curve (AUC) than single-operation-based models, such as EfficientNet and
ViT. Our method achieved the highest performance out of nine FAS models in
terms of DG, demonstrating that a rich representation trained with the
generalization method contributes to the robustness of domain shifts.

##  2 Related Work

###  2.1 Feature Extraction for FAS

Depending on the feature extraction method used in neural networks, the
backbones for FAS can be classified as CNN, SA-based ViT, and hybrid ViT with
convolutional elements (Han et al., 2022).

CNN-based models can extract local features from images and are trained
efficiently based on strong assumptions, such as spatial equivariance. For
example, ResNet (He et al., 2016; Shao et al., 2019; Jia et al., 2020) and
EfficientNet (Tan & Le, 2019; Kong et al., 2022) have been widely used to
obtain local cues from images in FAS. However, the information from CNN is
limited to the range of the receptive field, and it is difficult to aggregate
information from other spatial locations (Raghu et al., 2021). The performance
ceiling is unavoidable if the assumption of locality and invariance does not
fit the task because the CNN is operated based on complex assumptions
(d’Ascoli et al., 2021).

To overcome the limitations of CNNs, the ViT-based model designs the SA
mechanism with a soft assumption. ViT divides an image into non-overlapped
patches and transforms them into a linearly embedded sequence. Global
dependencies between patches are encoded via SA (Dosovitskiy et al., 2020; Liu
et al., 2021b). However, it is difficult to solve the FAS task using ViTs
because a large dataset is necessary for pretraining ViT owing to a lack of
prior knowledge (Raghu et al., 2021; d’Ascoli et al., 2021; Tu et al., 2022).

Therefore, hybrid models have recently been proposed to reflect the advantages
of CNNs in ViT (Dosovitskiy et al., 2020; d’Ascoli et al., 2021; Tu et al.,
2022). For example, ViT Hybrid uses a feature map extracted through a CNN as
an input to ViT. ConViT indirectly reflects locality into the ViT structure by
adding a modified SA module that operates similarly to the convolution in the
initial stage (d’Ascoli et al., 2021). In addition, MaxViT is an advanced
model of CoAtNet (Dai et al., 2021) that uses convolution layers directly in
several stages of ViT (Tu et al., 2022).

###  2.2 Domain Generalization

The diversity of face images inputted into the authentication system is
attributed to varying environmental conditions, such as background and camera
angle, between individuals. This variation causes distribution shifts between
the training and inference data, which degrades model performance. Therefore,
the previous studies have suggested several methods to train the generalized
feature space underlying some source domains and unseen but related target
domains (Shao et al., 2019; Jia et al., 2020; Liu et al., 2021a; Wang et al.,
2022b; Ming et al., 2022).

These studies have defined FAS as a binary classification problem that
discriminates between real and spoof and adopted binary cross-entropy (BCE) as
the loss function. However, recent studies have indicated that models trained
via BCE are prone to attacks due to overfitting, thereby leading to decreased
generalization performance. To address this issue, previous studies redefined
the FAS task as a regression problem and probabilistically estimated its
liveness score (Jiang et al., 2022; Kwak et al., 2023). Consequently, the
regression-based FAS model improves the generalization ability because the
converted continuous label contains more semantic information from both real
and spoof images (Jiang et al., 2022). In addition, the expected liveness
score-based regression neural network using the pseudo-discrete label encoding
technique contributes to more robustness in unseen datasets than the model
trained with BCE (Kwak et al., 2023).

##  3 Proposed Method

In this study, we propose a FAS framework with SA and a convolution-based
feature extractor for DG. Figure 2 shows our proposed framework consists of
three stages: label discretization, feature extraction, and liveness
prediction.

###  3.1 Label Discretization

We redefine the binary FAS classification as a regression-based method, as
this approach is more suitable for learning generalized and discriminative
representations (Jiang et al., 2022; Kwak et al., 2023). A sequence of class
labels was transformed into a probability distribution using CutMix (Yun et
al., 2019). CutMix is a technique used for data augmentation, but we generate
a label with CutMix to discretize the pseudo-label and solve the FAS task with
a regression-based loss function. Specifically, there are $N$ source domain
datasets, which comprise face images $X\in\mathbb{R}^{H{\times}W{\times}3}$
with binary label $Y{\in}{\\{0,1\\}}$ that represents fake and real for each
face image. The real and fake face images are denoted as $X_{r}$ and $X_{f}$,
respectively, and $Y_{r}$ and $Y_{f}$ are the corresponding labels. Using
CutMix, we swap some parts of $X_{r}$ to that of $X_{f}$ and denote this
combined image as input image $X_{c}$. We define the discretized pseudo-label
$Y_{c}$ as $m_{l}Y_{r}$, which is the single value $m_{l}$ selected from the
set $M=\\{0,\frac{1}{K},\frac{2}{K},…,\frac{K-1}{K},1\\}$, where the interval
$[0,1]$ is divided by a constant $K$.

###  3.2 Hybrid Feature Extraction

In the second stage, we employ ConViT as a backbone $F$ for feature extraction
to encode rich information to capture a cue for FAS. Specifically, global
context and local cues are obtained through gated positional self-
attention(GPSA) which is modified from SA. In general, an attention score A in
original ViT is defined as the dot product of query embedding $Q$ and key
embedding $K$. $W_{Q}$, $W_{K}$, and $W_{V}$ are the weight matrices for $Q$,
$K$, and $V$, respectively, which are embeddings for the query, key, and
value, respectively. The embeddings were inferred by multiplying the linearly
projected input $X_{p}$ with the corresponding weight matrices. The attention
score $A_{ij}$ implies the semantic relevance between patches $X_{p}^{i}$ and
$X_{p}^{j}$ and can capture the long-range dependency between patches.

To reflect locality in the attention score, an inner product of the learnable
embedding $v_{pos}$ and relative positional encoding $r_{ij}$ is added to
$Q_{i}K_{j}^{T}$. As shown in Eq. (2), the gating parameter $\sigma$
manipulates the type of information on which to focus. A larger $\sigma$ at
the initial stage makes this GPSA function as generalized convolution, which
assigns higher weights to adjacent patches. After extracting the local
features at the earlier layer, a smaller $\sigma$ is gated with more global
information of patches far from each other at the upper layer.

$GPSA(X_{p}):=normalized[A]X_{p}W_{v},$ (1) $\begin{split}where\
A_{ij}=(1-\sigma)\times softmax(Q_{i}K_{j}^{T})\\\ +\sigma\times
softmax(v_{pos}^{T}r_{ij}).\end{split}$ (2)

###  3.3 Liveness Prediction

The feature vector is derived from the input image $X_{c}$, by averaging the
patch embedding $F(X_{c})$ extracted from our backbone $F$ and passed to both
the score regressor $R$ and domain discriminator $D$, which contributes to a
robust backbone for unseen domain data (Jia et al., 2020). As described in Eq.
(3), $p_{k}^{i}$ is the probability vector of the real image in the $i^{th}$
input image, and the liveness score $\hat{Y}_{c}^{i}$ is calculated using the
sum of the element-wise multiplication between $M$ and corresponding
probabilities in $p_{k}^{i}$. The score regressor $R$ is trained to minimize
the difference between the predicted score $\hat{Y}_{c}^{i}$ and groundtruth
label $Y_{c}^{i}$ through the mean squared error. Subsequently, backbone $F$
is trained adversarially to extract generalized features, which makes it
difficult for the discriminator to predict the domain label $Y_{D}$ of a
feature. For adversarial learning, a gradient reversal layer (GRL) is included
after the feature extractor to make it difficult for the discriminator to
distinguish features from different domains. Additionally, the regressor and
discriminator are composed of fully connected layers. In Eq. (4), $q(\cdot)$
is the probability vector of the discriminator and is equal to $D(F(X_{c}))$.
Therefore, domain-invariant feature is extracted from backbone $F$ through
adversarial learning, as described in Eq. (4). The final loss function is
defined as in Eq. (5).

$\mathcal{L}_{reg}=\sum\|Y_{c}^{i}-\hat{Y}_{c}^{i}\|_{2}^{2},\\\ \quad
where\;\hat{Y}_{c}^{i}=\sum_{k=0}^{K}m_{k}\times p_{k},$ (3)
$\min_{D}\max_{F}\mathcal{L}_{adv}(F,D)=-\mathbb{E}_{x,y{\sim}{X_{c},Y_{D}}}\sum_{n=1}^{N}\mathbbm{1}_{[n=y]}\log{q(x)},$
(4) $\mathcal{L}_{final}=\mathcal{L}_{reg}+\mathcal{L}_{adv}.$ (5)

##  4 Experimental Results

###  4.1 Experimental Settings

We conducted experiments on various FAS benchmark datasets, namely OULU-NPU(O)
(Boulkenafet et al., 2017), MSFD-MSU(M) (Wen et al., 2015), REPLAY-ATTACK(I)
(Chingovska et al., 2012), and CASIA-FASD(C) (Zhang et al., 2012). We have
used a cross-dataset protocol to evaluate the performance. The cross-dataset
protocol is a leave-one-out setting that is used to train a model with several
source datasets and to test the generalization performance with an unseen
target dataset. Half total error rate (HTER) and area under curve (AUC) were
used as evaluation metrics (Yu et al., 2022).

We used CNN, ViT, and hybrid networks pretrained with ImageNet1K to prevent
overfitting and achieve further improvement in DG (Parkin & Grinchuk, 2019;
Ming et al., 2022). While we compared the FAS performance depending on
differences in the feature extraction methods of the backbones, the structure
of the score regressor and discriminator was kept the identical, and each
feature extractor was trained with the same loss function as in Eq. (5).

###  4.2 Experimental Results

Table 1: Results of the cross-dataset protocol. In each cell, the left and
right numbers represent HTER(%) and AUC(%), respectively. Bold implies the
best results and second best is underlined.

Backbone | OCI$\rightarrow$M | OMI$\rightarrow$C | CMO$\rightarrow$I | CMI$\rightarrow$O | $\mu\pm\sigma$
---|---|---|---|---|---
CNN | ResNet | 5.8 / 91.8 | 14.9 / 89.4 | 11.5 / 94.4 | 15.0 / 90.5 |  11.8+4.3 / 91.5+2.2
EfficientNet | 10.4 / 92.4 | 18.8 / 84.2 | 18.9 / 87.4 | 21.7 / 82.5 | 17.5+4.9 / 86.6+4.3
ViT | ViT Base | 17.9 / 83.7 | 31.3 / 70.7 | 16.9 / 89.9 | 28.3 / 79.5 | 23.6+7.3 / 81.0+8.1
Swin Transformer | 10.0 / 94.3 | 23.7 / 77.0 | 16.0 / 78.5 | 10.6 / 95.2 | 15.1+6.3 / 86.2+9.8
Hybrid | ViT Hybrid |  5.0 / 97.1 |  13.6 / 92.2 | 17.9 / 89.9 |  11.3 / 95.0 | 12.0+5.4 / 93.5+3.1
ConViT | 5.0 / 97.2 | 12.1 / 93.6 |  15.1 / 93.3 | 14.7 / 91.6 |  11.7+4.7 / 93.9+2.4
MaxViT | 10.4 / 93.9 | 14.5 / 89.7 | 26.8 / 72.9 | 20.6 / 87.1 | 18.1+7.1 / 85.9+9.1

We conducted a comparative experiment to determine the most robust FAS
backbone among the candidate models. Specifically, we selected ResNet and
EfficientNet as representative models for the CNN, ViT and swin transformer
for the ViT series and ViT Hybrid, ConViT, and MaxViT for the hybrid series.
First, Table 1 demonstrates that main feature extraction operation
significantly affects the performance. When ConViT was used as a feature
extractor, it achieved the best performance with an HTER of 11.7% and AUC of
93.9% compared to the CNN and ViT models. In addition to ConViT, ViT Hybrid
had the second-highest average AUC. However, the ViT and swin transformer
using only SA exhibit HTER scores of 23.6% and 15.1%, respectively, recording
the highest error rates, which implies that the ViT is vulnerable to changes
in the distribution of the evaluation data. When features were extracted from
a ViT-based hybrid model containing convolution-like elements, performance in
the unseen domain was typically superior to that of convolution or SA models
alone. Specifically, the ConViT-based framework outperformed EfficientNet and
ViT in 7.3%$p$ (=93.9%-86.6%) and 12.9%$p$ (=93.9%-81.0%) higher AUC scores,
respectively.

Second, we found that convolution-like elements in the initial layer would be
effective in improving FAS performance by comparing the hybrid models, as
shown in Table 1. Specifically, MaxViT using convolution in most stages
demonstrated poor performance, while ConViT, which indirectly adds
convolutional SA in the early stage, and ViT Hybrid, which utilizes
convolution in the input image, reported the lowest HTER values among the
comparison groups. In addition, both MaxViT and swin transformer are similar
in that they use window-based SA and the averaged AUC score of these models is
about 86% compared to that of the other hybrid models, which are ConViT and
ViT Hybrid, achieving 94%. Therefore, models using window-based SA show an
approximately 8%$p$ lower AUC than the other hybrid models. Furthermore, we
experimentally confirm that attention using a window shift is inappropriate
for FAS; however, convolutional elements in an early stage of the network
contribute to robust classification.

Table 2: Comparison with other generalized FAS methods under cross-dataset
testing. Results are evaluated as average ranks in each setting. $\dagger$ and
$\ast$ denote the proposed framework with ViT Hybrid and ConViT, respectively.

Model | OCI$\rightarrow$M | OMI$\rightarrow$C | CMO$\rightarrow$I | CMI$\rightarrow$O | Avg. Rank
---|---|---|---|---|---
MADDG (Shao et al., 2019) | 17.7 / 88.1 | 24.5 / 84.5 | 22.2 / 85.0 | 27.9 / 80.0 | 10.75
PAD-GAN (Wang et al., 2020) | 17.0 / 90.1 | 19.7 / 87.4 | 20.9 / 86.7 | 25.0 / 81.5 | 8.75
RF-Meta (Shao et al., 2020) | 13.9 / 94.0 | 20.3 / 88.2 | 17.3 / 90.5 | 16.5 / 91.2 | 6.5
SSDG-M (Jia et al., 2020) | 16.7 / 90.5 | 23.1 / 85.5 | 18.2 / 94.6 | 25.2 / 81.8 | 9
SDA (Wang et al., 2021) | 15.4 / 91.8 | 24.5 / 84.4 | 15.6 / 90.1 | 23.1 / 84.3 | 7
$\textbf{D}^{2}$AM (Chen et al., 2021) | 15.4 / 91.2 |  12.7 / 95.7 | 21.0 / 85.6 | 15.3 / 90.9 | 5.75
ANRL (Liu et al., 2021a) | 10.8 / 96. 8 | 17.8 / 89.3 | 16.0 / 91.0 | 15.7 / 91.9 | 4.5
SSAN-M (Wang et al., 2022b) | 10.4 / 94.8 | 16.5 / 90.8 | 14.0 / 94.6 | 19.5 / 88.2 | 4
ViTransPAD (Ming et al., 2022) |  8.4 / - | 17.9 / - | 16.0 / - | 15.7 / - | 4.25
Ours$\dagger$ |  5.0 / 97.1 | 13.6 / 92.2 | 17.9 / 89.9 | 11.3 / 95.0 | 3
Ours$\ast$ | 5.0 / 97.2 |  12.1 / 93.6 |  15.1 / 93.3 |  14.7 / 91.6 | 1.5

Table 2 presents a comparison of the proposed framework with the latest
promising models. The ConViT or ViT Hybrid-based framework was ranked 1.5 and
3, respectively, by averaging HTER scores over four sub-protocols in a cross-
dataset setting, showing the highest performance. In particular, the proposed
ConViT-based model outperformed a previous model with ViT structure (Ming et
al., 2022), exhibiting a 2.8%$p$ lower HTER. These results suggest that our
ConViT-based model is the most effective as a generalized FAS methodology
compared to the previously developed methodologies (Shao et al., 2019; Wang et
al., 2020; Shao et al., 2020; Jia et al., 2020; Wang et al., 2021; Chen et
al., 2021; Liu et al., 2021a; Wang et al., 2022b; Ming et al., 2022), and the
extraction of rich representations from images significantly impacts improved
performance in comparison to learning the spatio-temporal relationship in a
video (Ming et al., 2022). Consequently, the results suggest that both
locality and global dependencies within patches contribute to a more robust
performance.

##  5 Conclusion

This study demonstrates that our hybrid FAS framework with self-attention and
convolution is more robust for FAS tasks than those with a CNN and ViT alone.
Our ConViT-based framework improved the performance of domain generalization,
over other promising FAS models. These results suggest that it is important to
extract both local and global information from images for a robust FAS against
domain shifts.

## References

  * Anthony et al. (2021) Anthony, P., Ay, B., and Aydin, G.  A review of face anti-spoofing methods for face recognition systems.  In _2021 International Conference on INnovations in Intelligent SysTems and Applications (INISTA)_ , pp. 1–9. IEEE, 2021.
  * Boulkenafet et al. (2017) Boulkenafet, Z., Komulainen, J., Li, L., Feng, X., and Hadid, A.  Oulu-npu: A mobile face presentation attack database with real-world variations.  In _2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017)_, pp. 612–618. IEEE, 2017.
  * Chen et al. (2021) Chen, Z., Yao, T., Sheng, K., Ding, S., Tai, Y., Li, J., Huang, F., and Jin, X.  Generalizable representation learning for mixture domain face anti-spoofing.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pp. 1132–1139, 2021.
  * Chingovska et al. (2012) Chingovska, I., Anjos, A., and Marcel, S.  On the effectiveness of local binary patterns in face anti-spoofing.  In _2012 BIOSIG-proceedings of the international conference of biometrics special interest group (BIOSIG)_ , pp. 1–7. IEEE, 2012.
  * Dai et al. (2021) Dai, Z., Liu, H., Le, Q. V., and Tan, M.  Coatnet: Marrying convolution and attention for all data sizes.  _Advances in neural information processing systems_ , 34:3965–3977, 2021.
  * Dosovitskiy et al. (2020) Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * d’Ascoli et al. (2021) d’Ascoli, S., Touvron, H., Leavitt, M. L., Morcos, A. S., Biroli, G., and Sagun, L.  Convit: Improving vision transformers with soft convolutional inductive biases.  In _International Conference on Machine Learning_ , pp. 2286–2296. PMLR, 2021.
  * George & Marcel (2021) George, A. and Marcel, S.  On the effectiveness of vision transformers for zero-shot face anti-spoofing.  In _2021 IEEE International Joint Conference on Biometrics (IJCB)_ , pp. 1–8. IEEE, 2021.
  * Han et al. (2022) Han, K., Wang, Y., Chen, H., Chen, X., Guo, J., Liu, Z., Tang, Y., Xiao, A., Xu, C., Xu, Y., et al.  A survey on vision transformer.  _IEEE transactions on pattern analysis and machine intelligence_ , 45(1):87–110, 2022.
  * He et al. (2016) He, K., Zhang, X., Ren, S., and Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Huang et al. (2022) Huang, H.-P., Sun, D., Liu, Y., Chu, W.-S., Xiao, T., Yuan, J., Adam, H., and Yang, M.-H.  Adaptive transformers for robust few-shot cross-domain face anti-spoofing.  In _European Conference on Computer Vision_ , pp. 37–54. Springer, 2022.
  * Jia et al. (2020) Jia, Y., Zhang, J., Shan, S., and Chen, X.  Single-side domain generalization for face anti-spoofing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8484–8493, 2020.
  * Jiang et al. (2022) Jiang, F., Liu, P., and Zhou, X.-D.  Ordinal regression with representative feature strengthening for face anti-spoofing.  _Neural Computing and Applications_ , 34(18):15963–15979, 2022.
  * Kong et al. (2022) Kong, C., Chen, B., Li, H., Wang, S., Rocha, A., and Kwong, S.  Detect and locate: Exposing face manipulation by semantic-and noise-level telltales.  _IEEE Transactions on Information Forensics and Security_ , 17:1741–1756, 2022.
  * Kwak et al. (2023) Kwak, Y., Jung, M., Yoo, H., Shin, J., and Kim, C.  Liveness score-based regression neural networks for face anti-spoofing.  In _ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , pp. 1–5. IEEE, 2023.
  * Liu et al. (2021a) Liu, S., Zhang, K.-Y., Yao, T., Bi, M., Ding, S., Li, J., Huang, F., and Ma, L.  Adaptive normalized representation learning for generalizable face anti-spoofing.  In _Proceedings of the 29th ACM international conference on multimedia_ , pp. 1469–1477, 2021a.
  * Liu et al. (2021b) Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 10012–10022, 2021b.
  * Ming et al. (2022) Ming, Z., Yu, Z., Al-Ghadi, M., Visani, M., Luqman, M. M., and Burie, J.-C.  Vitranspad: video transformer using convolution and self-attention for face presentation attack detection.  In _2022 IEEE International Conference on Image Processing (ICIP)_ , pp. 4248–4252. IEEE, 2022.
  * Nagpal & Dubey (2019) Nagpal, C. and Dubey, S. R.  A performance evaluation of convolutional neural networks for face anti spoofing.  In _2019 International Joint Conference on Neural Networks (IJCNN)_ , pp. 1–8. IEEE, 2019.
  * Parkin & Grinchuk (2019) Parkin, A. and Grinchuk, O.  Recognizing multi-modal face spoofing with face recognition networks.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pp. 0–0, 2019.
  * Raghu et al. (2021) Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., and Dosovitskiy, A.  Do vision transformers see like convolutional neural networks?  _Advances in Neural Information Processing Systems_ , 34:12116–12128, 2021.
  * Shao et al. (2019) Shao, R., Lan, X., Li, J., and Yuen, P. C.  Multi-adversarial discriminative deep domain generalization for face presentation attack detection.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10023–10031, 2019.
  * Shao et al. (2020) Shao, R., Lan, X., and Yuen, P. C.  Regularized fine-grained meta face anti-spoofing.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pp. 11974–11981, 2020.
  * Tan & Le (2019) Tan, M. and Le, Q.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _International conference on machine learning_ , pp. 6105–6114. PMLR, 2019.
  * Tu et al. (2022) Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., and Li, Y.  Maxvit: Multi-axis vision transformer.  In _European conference on computer vision_ , pp. 459–479. Springer, 2022.
  * Wang et al. (2020) Wang, G., Han, H., Shan, S., and Chen, X.  Cross-domain face presentation attack detection via multi-domain disentangled representation learning.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 6678–6687, 2020.
  * Wang et al. (2021) Wang, J., Zhang, J., Bian, Y., Cai, Y., Wang, C., and Pu, S.  Self-domain adaptation for face anti-spoofing.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pp. 2746–2754, 2021.
  * Wang et al. (2022a) Wang, W., Wen, F., Zheng, H., Ying, R., and Liu, P.  Conv-mlp: A convolution and mlp mixed model for multimodal face anti-spoofing.  _IEEE Transactions on Information Forensics and Security_ , 17:2284–2297, 2022a.
  * Wang et al. (2022b) Wang, Z., Wang, Z., Yu, Z., Deng, W., Li, J., Gao, T., and Wang, Z.  Domain generalization via shuffled style assembly for face anti-spoofing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4123–4133, 2022b.
  * Wen et al. (2015) Wen, D., Han, H., and Jain, A. K.  Face spoof detection with image distortion analysis.  _IEEE Transactions on Information Forensics and Security_ , 10(4):746–761, 2015.
  * Yu et al. (2022) Yu, Z., Qin, Y., Li, X., Zhao, C., Lei, Z., and Zhao, G.  Deep learning for face anti-spoofing: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ , 45(5):5609–5631, 2022.
  * Yun et al. (2019) Yun, S., Han, D., Oh, S. J., Chun, S., Choe, J., and Yoo, Y.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 6023–6032, 2019.
  * Zhang et al. (2012) Zhang, Z., Yan, J., Liu, S., Lei, Z., Yi, D., and Li, S. Z.  A face antispoofing database with diverse attacks.  In _2012 5th IAPR international conference on Biometrics (ICB)_ , pp. 26–31. IEEE, 2012.