# Deep Composite Face Image Attacks: Generation, Vulnerability and Detection

Jag Mohan Singh, and Raghavendra Ramachandra Norwegian University of Science
and Technology (NTNU), Norway e-mail: (raghavendra.ramachandra@ntnu.no).Both
authors have equally contributed for this work.

###### Abstract

Face manipulation attacks have drawn the attention of biometric researchers
because of their vulnerability to Face Recognition Systems (FRS). This paper
proposes a novel scheme to generate Composite Face Image Attacks (CFIA) based
on the Generative Adversarial Networks (GANs). Given the face images from
contributory data subjects, the proposed CFIA method will independently
generate the segmented facial attributes, then blend them using transparent
masks to generate the CFIA samples. The primary motivation for CFIA is to
utilize deep learning to generate facial attribute-based composite attacks,
which has been explored relatively less in the current literature. We generate
$14$ different combinations of facial attributes resulting in $14$ unique CFIA
samples for each pair of contributory data subjects. Extensive experiments are
carried out on our newly generated CFIA dataset consisting of 1000 unique
identities with 2000 bona fide samples and 14000 CFIA samples, thus resulting
in an overall 16000 face image samples. We perform a sequence of experiments
to benchmark the vulnerability of CFIA to automatic FRS (based on both deep-
learning and commercial-off-the-shelf (COTS). We introduced a new metric named
Generalized Morphing Attack Potential (GMAP) to benchmark the vulnerability
effectively. Additional experiments are performed to compute the perceptual
quality of the generated CFIA samples. Finally, the CFIA detection performance
is presented using three different Face Morphing Attack Detection (MAD)
algorithms. The proposed CFIA method indicates good perceptual quality based
on the obtained results. Further, FRS is vulnerable to CFIA (much higher than
SOTA), making it difficult to detect by human observers and automatic
detection algorithms. Lastly, we performed experiments to detect the CFIA
samples using three different detection techniques automatically.

###### Index Terms:

Biometrics, Face recognition, Morphing Attacks, Image Compositing,
Vulnerability, Generalized Morphing Attack Potential, Composite Attack
Detection

##  1 Introduction

FRS demonstrates highly accurate recognition rates, which has led to their
widespread usage in eCommerce, online banking, surveillance and security
applications. The recent advances in the deep learning techniques have further
increased the accuracy of the FRS [1], [2] that enabled them to be deployed in
the border control applications. However, the FRS is vulnerable to various
attacks, among which the face morphing attacks have gained attention due to
their impact on the border control application. Recent benchmarking results
reported in NIST FRVT MOPRH [3] indicate that the higher the accuracy of the
FRS, the higher the vulnerability for the morphing attacks.

One of the most widely used attacks toward FRS is the Presentation Attacks
(PA), a.k.a spoofing attacks, which can be achieved by presenting a biometric
artefact to the biometric capture device. PA can be performed by presenting a
Presentation Attack Instrument (PAI) that includes either a printed photo
(print-photo), displaying an image (display-photo), displaying a video
(replay-video), or the use of a rigid/non-rigid 3D face mask (mask-attack).
Biometric researchers had thus devised Presentation Attack Detection (PAD) as
a countermeasure to PA that is extensively discussed in [4], and [5].

The second type of widely studied attack on the FRS is the adversarial attack,
which can be performed by applying a small perturbation (noise), a.k.a
adversarial perturbation, to a facial image. Even though the introduced
perturbation is indistinguishable to the human eye but can lead to mis-
classification with high-confidence [6] and can be used to expose
vulnerabilities of the FRS. Adversarial attacks have shown high vulnerability
in FRS, especially on the deep learning-based FRS [7]. Adversarial Attacks
could be a white box attack where the attacker has complete knowledge of the
underlying deep-learning model and thus it can be performed during training.
Adversarial attacks could also be a black-box attack performed during testing,
and the attacker does not know the underlying deep-learning model. There exist
several countermeasures to address the adversarial attacks that are
extensively discussed in [8], [9]. It needs to be pointed out that adversarial
attacks are digital when performed on images, but they can also be performed
in the physical world by using a unique eyeglass for impersonation [8].

Another type of attack gaining momentum in the biometric community is face
morphing attacks. Face morphing blends one or more face images (two face
images show the best case to achieve a high vulnerability). The morphing face
image shows vulnerability for FRS towards all (two) contributory data
subjects. Face morphing can be generated by using algorithms based on facial
landmarks such as Face Morpher [10] and UBO-Morph [11]. More recently,
algorithms based on Generative Adversarial Networks (GANs) such as MorGAN
[12], MIPGAN [13] and ReGenMorph [14] have been used to generate face morphing
images. The morphing attack has shown high vulnerabilities in FRS, especially
in the passport application scenario, including the automatic border control.
Because these attacks can deceive both human observers (border control
officers) and automatic FRS in Automatic Border Control (ABC) [15], [16].
Following the initial paper [17], there have been several papers on morphing
detection, and the reader is advised to look at the recent survey by Venkatesh
et al. [15] to get a detailed overview on face morphing.

The majority of the face morphing generation works are devised by performing
the blending operation on the complete (or full) face images [15]. To the best
of our knowledge, there exist only one work that presented the part-based
morphing attacks on FRS [18], where the blending operation is carried out
using Poisson image editing [19]. The generated composite morphs have shown
vulnerabilities in FRS based on deep-learning features such as VGGFace [20],
Arcface [2] and commercial-off-the-shelf (COTS) that includes Neurotech [21]
and Cognitec [22]. Further, the human observer analysis is also discussed.
However, the work presented in [18] has several limitations, including (1) it
is based on landmarks, and this would lead to pixel-based artifacts due to
alignment issues, and correction of these would require manual intervention
[15] (2) Only a few regions are used to generate the composite images (3)
Limited only to the base regions like nose, mouth, eye and forehead. (4)
Limited only to the single facial attribute composite generation (5) failure
to achieve a high FRS vulnerability.

Thus, motivated by the limitations of the existing method [18], we aim to
generate the composite images in a fully automatic fashion using GANs. GAN-
based face synthesis has gained huge interest since the introduction of
StyleGAN variants [23]. The initial work in this domain was done by Karras et
al. [24] where the ProGAN was first introduced. ProGAN is based on the GANs
for progressively synthesizing facial images, and thus, it can generate high-
quality facial images from random numbers (a.k.a latents). Karras et al. [25]
improved the design of the generator network in StyleGAN-I, allowing for more
variation in hairs, pose, and expression in generated facial images. Karras et
al. [26] in StyleGAN-II improved the generator normalization, applied
regularization for good conditioning from latent vectors to images, and
changed the overall design of the progressive architecture from previous
StyleGAN-I. However, these solutions are developed using GANs which generate a
full facial image from a latent and don’t consider the facial image synthesis
region using a partial face image with missing regions.

The recent work by Chai et al. [27] where they performed highly realistic
facial image synthesis with missing regions using GANs. They achieved this by
offering an innovative solution to the problem of GAN-inversion (i.e.,
recovering the latent given an input image with missing regions) by posing it
as a regression problem. In this work, we modified the approach from Chai et
al. [27] to generate the CFIA samples that show vulnerabilities in FRS. Thus,
our proposed method for CFIA generation is designed to consider the optimal
pairing of the input images used during the compositing process to ensure
high-quality CFIA generation. Note that optimal pairing is based on the
similarity of Arcface features computed from the input bona fide face images
(Section 4.2). Further, we introduce the generation of CFIA based on multiple
facial attributes. Both single and multiple facial attributes are blended
using the transparent (or real) value that can further improve the
vulnerability and challenge the detection of CFIA samples.

The key contributions of our proposed method are as follows:

  * •

We have proposed a novel framework for Composite Face Image Attack (CFIA)
generation using regression and GAN-based image synthesis. The primary
motivation for CFIA is to utilize deep learning for generation of facial
attribute-based composite attacks, which has been explored relatively less in
the current literature. We generate CFIA based on single face attributes and
multiple face attributes for contributory data subjects. Further, we propose
transparent blending to improve the similarity of both contributory data
subjects. Thus, we introduce 14 different types of CFIA based on various
combinations of different regions from contributory data subjects.

  * •

We present the new CFIA dataset generated using $1000$ unique synthetic
identities. The whole dataset consists of $14000$ CFIA samples representing
different facial regions and $2000$ bona fide samples.

  * •

We present the perceptual image quality analysis of the CFIA dataset using
Peak-Signal-to Noise Ratio (PSNR) and Structural Similarity Index Measure
(SSIM) to benchmark the quality of the generated CFIA samples.

  * •

Extensive vulnerability analysis on newly generated CFIA dataset is presented
using both deep learning and Commercial-Off-the-Shelf (COTS) on all 14
different types of CFIA. We also improved the existing morphing vulnerability
metric; Morphing Attack Potential (MAP) and proposed the updated metric called
Generalized Morphing Attack Potential (GMAP) to effectively benchmark the
vulnerability by considering real-life scenarios.

  * •

We present the human observer study on the newly generated CFIA dataset with
43 observers with and without face image manipulation detection background.

  * •

Extensive experiments are presented to automatically detect the CFIA using
three different existing face morphing attack detection techniques.

  * •

The CFIA dataset, together with the source code of the proposed method, will
be made publicly available to enable the reproducibility of the results
presented in this paper.

In the rest of the paper we introduce the proposed method in Section 2,
database generation methodology in Section 3, perceptual quality evaluation of
the composite images in Section 5, human observer study in Section 6,
automatic FRS vulnerability analysis in Section 7, and CFIA detection (CAD) in
Section 8. Finally, we present the conclusions and future-work of this paper
in Section 9.

##  2 Proposed CFIA Generation Method

Figure 1: Block diagram of the proposed approach where FS is based on UPerNet
Face Segmenter from Zhou et al. [28], the Encoder is based on Resnet-34 [27],
and Decoder is based on StyleGAN [25] and the encoder-decoder synthesizes the
final composite.

Figure 2: Illustration showing the comparison between the single face
attribute based regions and multiple face attribute based regions for the
generation of the initial composite using the proposed method. Figure 3:
Qualitative results of the proposed CFIA sample generation method with 14
different regions.

Region Name | Contributory | Contributory | Composite
---|---|---|---
| Subject 1 | Subject 2 | Segments
R1 | S | E | S-E
R2 | S | N | S-N
R3 | S | M | S-M
R4 | S | S | S-S
R5 | SEN | M | SEN-M
R6 | SEM | N | SEM-N
R7 | SNM | E | SNM-E
R8 | SEN | EM | SEN-EM
R9 | SEN | EN | SEN-EN
R10 | SEN | SEM | SEN-SEM
R11 | SEN | SEN | SEN-SEN
R12 | SENM | ENM | SENM-ENM
R13 | SENM | SENM | SENM-SENM
R14 | HBSENM | HBSENM | HBSENM-HBSENM

TABLE I: Different regions used to generate the composite face images from two
contributory data subjects, where H=Hair, B=Background, S=Skin, E=Eye, N=Nose
and M=Mouth, and the facial part/s used for segmentation for the two
contributory data subject are shown, and the combined facial parts used for
composite generation.

Figure 1 shows the block diagram of the proposed CFIA method. The goal of the
proposed CFIA method is to automatically select the single and multiple
regions from the given face images and blend them to generate the composite
images. The proposed CFIA method consists of three main functional blocks (1)
generation of individual segmented faces and masks from given face images, (2)
computation of the initial composite image and transparent blending face mask
and (3) final CFIA generation based on pre-trained GANs.

###  2.1 Individual Segmented Faces and Masks

The proposed CFIA composite image generation is based on the different facial
parts from the two contributory data subjects (e.g., skin from the first data
subject and eyes from the second data subject). Therefore, we automatically
employed a high precision and accuracy method to segment different facial
parts. In this work, we choose the unified parsing network (UPerNet) [29] for
automatic facial region segmentation, which is denoted as $\mathbb{FS}$.
UPerNet combines multi-task learning and semantic segmentation to achieve
high-quality results on segmentation and classification tasks at the
perceptual levels of an object, texture, material and facial Part. Thus, given
the face image, UPerNet [29] provides six different facial region masks,
including skin, eye, nose, mouth, hair, and background.

In this work, we have considered only two contributory face images based on
real-time use-case applicability (for e.g. attacks on eMRTD) [18, 15]. We
denote the first contributory face image by $F_{1}$ and the corresponding
part-based segmented masks obtained using UPerNet [29] by $SM1_{i}$, where
$i=\\{1,2,\ldots,6$} and its corresponding segmented image by $IS1_{i}$ and
denote the second contributory face image by $F_{2}$ and the corresponding
part-based segmented masks be $SM2_{j}$, where $j=\\{1,2,\ldots,6$} and its
corresponding segmented image be $IS2_{j}$. The face region segmentation
process to obtain individual :

$\begin{split}\\{SM1_{i},IS1_{i}\\}=\mathbb{FS}(F_{1}),\forall
i=\\{1,2,\ldots,6\\}\\\ \\{SM2_{j},IS2_{j}\\}=\mathbb{FS}(F_{2}),\forall
j=\\{1,2,\ldots,6\\}\end{split}$ (1)

Based on these six part-based segmentation masks, we generate an exhaustive
list of combinations from $SM1_{i}$ and $SM2_{j}$ that resulted in 14
different regions as listed in Table I. It needs to be pointed out that the
regions selected are exhaustive but not the complete power-set, as we focus
only on combinations and not permutations of the individual components. Also,
amongst the possible combinations, we have chosen the selected combinations
empirically that can indicate the vulnerability of both human observers and
automatic face recognition systems. Furthermore, within these 14 combinations,
we explore both single face-attribute and multiple face-attribute selected
from both contributing data subjects with both symmetric (in which the same
facial parts are chosen for both contributing data subjects) and asymmetric
(in which the different facial parts are selected from both contributing data
subjects). In Table I, Regions R4, R11, R13 & R14 are the symmetric regions,
and Regions R1, R2, R3, R5, R6, R7, R8, R9, R10 & R12 are the asymmetric
regions. Figure 2 shows the results of individual face segmentation and masks
for both single face attribute-based regions and multiple face attribute-based
regions.

###  2.2 Initial Composite Image and Transparent Blending Face-Mask

In the next step, we generate the initial composite image and the transparent
blending face mask by applying the blending operation on the individual
segmented faces ($IS1_{i}$ & $IS2_{j}$) and their corresponding masks
($SM1_{i}$ & $SM2_{j}$) from the two contributory data subjects ($F_{1}$ &
$F_{2}$). The blending operation is carried out independently for the mask and
the individual segmented faces. The blended mask $m_{c}$ is generated by a
simple union operation that can represent the combined facial region from
$SM1_{i}$ and $SM2_{j}$ as described in Equation 2. However, the generation of
the initial composite image ($IC$) is done in three consecutive steps shown in
Equation 3, where first $IC$ is initialized $0$, then in the next step, $IC$
is updated using the compositing equation with the segmented region
($IS1_{i}$) from the data subject $F_{1}$ as input. Finally, $IC$ is updated
using the compositing equation with the segmented region ($IS2_{j}$) from data
subject $F_{2}$, and its segmentation masks $SM2_{j}$ as an input. These steps
are mathematically presented in Equation 3.

$m_{c}=SM1_{i}\bigcup SM2_{j}$ (2)

$\begin{split}IC&=0\\\ IC&=IS1_{i}\\\
IC&=IS2_{j}+(1-SM2_{j}){\times}IC\end{split}$ (3)

Figure 2 shows the qualitative results of the initial composite image and the
corresponding mask for both single face attribute-based composite regions &
multiple face attribute-based composite regions.

###  2.3 Final CFIA samples Generation

Once the initial composite image and the transparent blending face mask are
generated, we generate the final CFIA samples are generated using the image
inpainting based on pre-trained regressor and GAN. The input composite image
and its mask are passed through a pre-trained encoder ($\mathbb{E}$) and then
to the decoder ($\mathbb{G}$) to generate the final composite image ($FCI$).
The process of generating the CFIA sample is as indicated in Equation 4.

$CFIA=\mathbb{D}(\mathbb{E}(IC,m_{c}))$ (4)

The encoder network ($\mathbb{E}$) selected in our work is pre-trained
Resnet-34 [27], and the decoder network ($\mathbb{G}$) is a pre-trained
StyleGAN-I decoder which was trained on FFHQ dataset [25]. The primary
motivation for the choice of the encoder and decoder networks was that image
to latent conversion is posed as a regression problem [27], unlike previous
approaches, as it would result in saving of computation time for the
generation of latent from an image. Further, they found Resnet-34 suitable for
regressing the latent from a face image with missing information and
performing high-quality image inpainting. Resnet-34 was particularly good in
performing the image inpainting of the input image with missing information
resulting in a realistic completion.
In the next step, we use the decoder ($\mathbb{D}$) based on StyleGAN-I as it
provides a linear latent subspace [30]. Hence, reconstruction from the
generated latent is of good perceptual quality even with missing information
in the input image. Figure 3 shows comparative results of the final composites
generated using the proposed method.

##  3 CFIA Database Generation

This section presents the dataset generation process used to evaluate the
proposed composite image generation. Owing to the ethical and legal challenges
with face biometric datasets that will eventually limit the distribution, in
this work, we generate the synthetic face images corresponding to the unique
identities using StyleGAN inversion [27]. Earlier works [31, 13, 32] indicated
that generating the synthetic face images have demonstrated both realness in
terms of quality, uniqueness and verification accuracy. Further, synthetic
face images will overcome the need for privacy and legal limitations to make
the database public, which is vital for reproducible research. Figure 4
illustrates the CFIA dataset generation process.

Figure 4: Illustration showing the CFIA dataset generation process Figure 5:
Illustration showing faces selected and rejected by our proposed frontal-pose
detection Algorithm 1. Note face images with glasses and GAN-based artifacts
are rejected manually.

###  3.1 Synthetic Face Image Generation

Given a random latent vector, we use the approach from Chai et al. [27] to
generate a synthetic face corresponding to unique data subjects using StyleGAN
inversion. We further perturb the random latent by an $\epsilon$ amount to
generate the mated face image corresponding to the given identity. The choice
of $\epsilon$ is made empirically, which is small enough not to alter the
identity of the generated face. However, the generation of synthetic face
images with corresponding mated face images with unique identities will result
in non-ICAO compliant photos with glasses, non-frontal pose, and a non-neutral
face expression, as shown in Figure 5. Therefore, it is necessary to detect
the ICAO-compliant faces for which we select faces with frontal pose
automatically and remove photos with glasses and non-neutral face expressions
manually.

Figure 6: Illustration of the effect of perturbation based on epsilon
($\epsilon$) for synthetic face generation, note artifacts start appearing
when ($\epsilon=0.1$) and results in change in identity when ($\epsilon=1$)

##  4 Hyperparameters Selected

Hyper-parameters | SOTA [27] | Proposed Method
---|---|---
Frontal Pose Selection | No | Yes
Optimal Pairing | No | Yes
Epsilon($\epsilon$) | No | $10^{-7}$
Alpha($\alpha$) | 1 | 0.5
Regions |  | R1,R2,R3,R4,R5
R14 | R6,R7,R8,R9,R10
| R11,R12,R13,R14

TABLE II:  Different Hyper-parameters used for the proposed method, where the
regions used are shown in detail in Table I. Note the proposed method modifies
a large number of hyper-parameters compared with SOTA [27].

This section discusses the choices of the parameters associated with SOTA and
the proposed method as tabulated in Table II. In total, we have five different
hyperparameters that are discussed as follows:

  * •

Epsilon($\epsilon$): The value of $\epsilon$ is empirically chosen as
$10^{-7}$. Since values higher than $10^{-1}$ lead to artifacts and a sample
of different identities as shown in Figure 6. Thus, we choose an $\epsilon$
conservatively.

  * •

Alpha: We choose $\alpha$=0.5 as it is known to create the highest
vulnerability towards FRS for Face Morphing Image Attack (FMIA) [15].

  * •

Regions: The current face segmentation provides Hair, Background, Skin, Eyes,
Nose, and Mouth as segments. Amongst these Skin, Eyes, Nose and Mouth are
particularly important for composition as these regions have shown the highest
vulnerability of composites towards FRS . We group the segments into regions
where the overall number of segments used is either two(R1 to R4), four (R5 to
R7), five (R8 to R9), six (R10 to R11), seven (R12), eight (R13) and twelve
(R14). Note that the grouping of segments used for contributory subjects one
and two don’t repeat completely except for regions R4, R13 and R14. Further,
we don’t swap the segments used for contributory subject one and contributory
subject two as vulnerability created towards FRS is independent of the subject
order.

###  4.1 Frontal Face Pose Selection

We have developed the algorithm to automatically select the ICAO compliant
face images corresponding to each unique identity as indicated in the
Algorithm 1. The primary motivation behind this algorithm is that the face in
a frontal pose would have similar angles between Left-Eye, Nose, and Mouth
(Left Part) and Right-Eye, Nose, and Mouth (Right Part). A slight change in
the face pose from a frontal face to a profile face would result in a skew,
which would cause these two angles to be different. The qualitative results of
the proposed frontal face selection algorithm are as shown in Figure 5. Since
we are currently not interested in the computation of exact face pose, the
heuristic works sufficiently well for our dataset, which does not consist of
extreme face poses.

Input: Face Image with 5 Landmarks (Left-Eye ($LE$), Right-Eye ($RE$), Nose
($N$), Left-Mouth ($LM$), and Right-Mouth ($RM$)

Output: True if Face Image is Frontal

1: Compute the angle between the vectors of Left-Eye, Nose, and Left-Mouth,
Nose

$\theta_{1}$ $\leftarrow$
$\arccos((\overrightarrow{LEN}\cdot\overrightarrow{LMN}))$.

2: Compute the angle between the vectors of Nose, Right-Eye, and Nose, Right-
Mouth

$\theta_{2}$ $\leftarrow$
$\arccos((\overrightarrow{NRE}\cdot\overrightarrow{NRM}))$.

3: Compute absolute difference between the angles, as _angleDiff_ $\leftarrow$
$\lvert\theta_{1}-\theta_{2}\rvert$

4: if  _angleDiff_ $\leq$ $\tau$ then

5: Face is Frontal

6: return True

7: end if

8: return False

Algorithm 1 Non-Frontal Pose Identification

###  4.2 Optimal Face Pair Generation for composite image generation

Input: Random Image Pairs
($I_{1}^{1},I_{2}^{1}$),$\cdots$,($I_{1}^{N},I_{2}^{N}$)

Output: Optimal Image Pairs
($O_{1}^{1}$,$O_{2}^{1}$),$\cdots$,($O_{1}^{N}$,$O_{2}^{N}$)

1: Compute Arcface features on the input face images.

2: Optimal-Pair $\leftarrow$ $\left[\right]$

3: for $i\leftarrow 1$ to $N$ do

4: Compute Index of nearest arcface feature $j$ to $i$

5: if $\left(j,i\right)$ $\notin$ Optimal-Pair then

6: Append $\left(i,j\right)$ to Optimal-Pair

7: else

8: Compute Index of second-nearest arcface feature $k$ to $i$

9: Append $\left(i,k\right)$ to Optimal-Pair

10: end if

11: end for

12: return Optimal-Pair

Algorithm 2 Optimal Pair Finding Algorithm

It is essential to select the look-alike data subjects to achieve the optimal
attack potential with the proposed composite face image generation. We choose
the optimal pairs to generate the composite face images to this extent. Given
$n$ synthetic samples, the total number of pairs possible is
$((n){\times}(n-1))/2$, and thus finding optimal pairs using this approach is
quadratic ($O(n^{2})$) as we have to compute the pair-wise distance for all
pairs. The quadratic time for pair-finding is within the computing limits as
our dataset now consists of 1000 unique data subjects. We have put an
additional constraint in the pair-finding algorithm not to return swapped
pairs, i.e., if ($i$,$j$) is the list, then ($j$,$i$) is not added to the
optimal pair list. The approach for optimal pair finding is summarized in an
algorithmic format in Algorithm 2 and a few optimal pairs are shown in Figure
4. The distance metric used in our approach is cosine-distance from Arcface
[2] features.

Thus, the CFIA dataset has 1000 unique identities with 2000 bona fide samples
and 14,000 CFIA samples. The whole dataset will made publicly available for
research purposes along with code at the following link
https://github.com/jagmohaniiit/LatentCompositionCode.

##  5 Perceptual quality evaluation of the composite images

Region | PSNR | SSIM
---|---|---
R1 | SOTA [27] | Proposed | SOTA [27] | Proposed
R1 | 15.4$\pm$10.2 | 15.6$\pm$7.0 | 0.68$\pm$0.01 | 0.71$\pm$0.00
R2 | 15.5$\pm$9.5 | 15.7$\pm$6.6 | 0.68$\pm$0.01 | 0.71$\pm$0.00
R3 | 15.5$\pm$10.2 | 15.6$\pm$7.0 | 0.68$\pm$0.01 | 0.71$\pm$0.00
R4 | 15.6$\pm$8.6 | 15.9$\pm$4.6 | 0.69$\pm$0.01 | 0.71$\pm$0.00
R5 | 15.4$\pm$10.6 | 15.6$\pm$7.4 | 0.68$\pm$0.01 | 0.71$\pm$0.00
R6 | 15.5$\pm$10.0 | 15.7$\pm$6.9 | 0.68$\pm$0.01 | 0.71$\pm$0.00
R7 | 15.5$\pm$10.6 | 15.7$\pm$7.4 | 0.68$\pm$0.01 | 0.71$\pm$0.00
R8 | 15.4$\pm$9.6 | 15.6$\pm$6.8 | 0.68$\pm$0.01 | 0.71$\pm$0.00
R9 | 15.4$\pm$8.6 | 15.7$\pm$6.7 | 0.68$\pm$0.01 | 0.73$\pm$0.00
R10 | 15.7$\pm$8.7 | 16.0$\pm$4.7 | 0.69$\pm$0.01 | 0.72$\pm$0.00
R11 | 15.6$\pm$9.9 | 16.0$\pm$5.0 | 0.69$\pm$0.01 | 0.72$\pm$0.00
R12 | 15.3$\pm$7.8 | 15.7$\pm$6.4 | 0.67$\pm$0.00 | 0.71$\pm$0.00
R13 | 15.7$\pm$10.3 | 16.0$\pm$5.2 | 0.69$\pm$0.01 | 0.72$\pm$0.00
R14 | 15.8$\pm$14.4 | 16.0$\pm$6.4 | 0.68$\pm$0.01 | 0.71$\pm$0.00

TABLE III: Perceptual Image Quality Metrics PSNR and SSIM comparison for SOTA
[27] and Proposed Method

---
(a) PSNR SOTA

---
(b) PSNR Proposed

---
(c) SSIM SOTA

---
(d) SSIM Proposed

Figure 7: Box plots showing PSNR of SOTA [27] and Proposed Method for
Different Regions

This section presents the quantitative analysis of the proposed CFIA method
using two perceptual image quality metrics, namely, PSNR (Peak Signal-to-Noise
Ratio) and SSIM (Structural Similarity Index). Both PSNR and SSIM are
reference image-based quality metrics and thus require a pair of images for
evaluation (face image from the contributory data subject and the generated
face composite image). Table III indicates the quantitative analysis of the
perceptual quality analysis on both SOTA [27] and the proposed CFIA method.
Figure 7 illustrates the box plots corresponding to both SSIM and PSNR
computed on all $14$ regions. Following are the main observations from the
obtained results:

  * •

The PSNR metric has a higher mean-value and less variance for the proposed
CFIA method compared with SOTA [27] indicating lesser noise in the face
composites generated using the proposed CFIA method. This is expected as
transparent blending would produce a lower contrast image, as the choice of
blending-factor ($\alpha=0.5$) would generate a pixel value lower than those
from contributory data subjects as the blending equation is applied twice
refer Equation 3. Thus, the proposed CFIA method generates a more consistent
image quality irrespective of the region compared with SOTA [27].

  * •

The SSIM metric produces a more stable value for both the proposed CFIA method
and SOTA [27]. The proposed CFIA method gives a higher value for SSIM than the
SOTA [27]. Since SSIM is a metric more tuned to the Human Visual System (HVS),
[33] as it measures luminance distortion, contrast distortion, and loss of
correlation. Thus, our proposed CFIA method generates higher-quality
composites for HVS.

##  6 Human Observer Study

Figure 8: Screenshot from the GUI (Full Page) of human observer web page.
Figure 9: Illustration showing average accuracy quantitatively for the human
observer study where bona fide or Synthetic Face Image (without any
modification) is denoted as SIF.

We perform a Human Observer Study (HOS) of the generated composites to
evaluate the detection performance by human experts. The HOS is conducted
using a web-based application
111https://folk.ntnu.no/jagms/indexCompositeUpdated.html where a dedicated web
page is set up with the use of PHP and HTML-CSS. In this study, GDPR norms are
respected, and we only store the individual’s email, gender, experience with
the composite problem, and age group. We have made sure that the user remains
anonymous during the study. Figure 8 shows the screenshot of the GUI of our
website where the HOS is carried out. In this study, a human observer is shown
a webpage with two images at a time where the observer has to decide
independently on whether each of them is real/composite (or manipulated). The
current study shows $43$ image pairs, and it takes around 20 minutes to
complete the study. The study includes synthetic face images and $14$
different types of composites as mentioned in Table I. Further, the human
observer is explained in detail the step-wise instructions to perform the
study. This enables people without awareness of the image manipulation problem
and those with basic and advanced awareness of the composition problem to
participate in the study. In the current evaluation, $51$ human observers have
participated and completed the study, including 40 participants without
awareness, 6 with basic awareness, and 5 with an advanced awareness of the
composition problem.

The quantitative results of HOS are as shown in Figure 9 and the following are
the important observations:

  * •

The average detection accuracy is similar for human observers without
awareness of the composition problem and those with basic awareness. This can
be attributed to the innate human ability to detect composites. However, the
average detection accuracy for human observers with advanced awareness of the
composition problem is much higher than both without awareness and basic
awareness.

  * •

The average accuracy is not very high for faces based on the composition,
which utilizes a single facial attribute except for R2 with advanced
awareness. This can be attributed to the fact that a large part of the facial
region needs to be inpainted in the case of single facial attribute
composition.

  * •

The average detection accuracy is high for the regions R8, R10, R12, and R14.
R8 has moderate parts of faces being used for compositing from the two
contributory data subjects. The reason for high detection accuracy can be
attributed to the fact that R8 has only eyes from both the contributory data
subjects but his nose and mouth from different contributory data subjects. The
same reasoning with more significant facial parts used for compositing can be
extended to R10 where the nose and mouth are from different contributory data
subjects but have skin and eyes from both contributory data subjects. Now for
the compositing region R12, the skin region is only from one contributory data
subject. Thus, in all three cases, the asymmetry in the regions from the
contributory data subjects aids the human observer in performing the detection
at high accuracy.

  * •

However, the average performance of the human observers for detecting a normal
face image (or non-composite) is 46%. Further, it is also interesting to
observe that degraded performance is noted in the advanced experience group.
Thus, our analysis indicates that human observers are limited in detecting the
normal face images compared to the composite face images.

  * •

Now, for the compositing region R14 all facial parts from the contributory
data subjects are being used. Thus, the composited image can be distinguished
from a synthetic face image using global image-based cues.

  * •

In summary, we could say that either asymmetric regions or global level cues
can help the human observer perform detection at high accuracy rates. However,
our analysis indicates that it is very challenging for humans to detect
composite attacks.

##  7 Automatic FRS Vulnerability Analysis

This section presents the vulnerability analysis of the proposed composite
attack images on the automatic face recognition system. We have benchmarked
five different FRS, including two COTS and three deep-learning-based FRS. The
COTS FRS includes Neurotech [21] (Neurotech Verilook SDK ($11.1$)), & Cognitec
[22] (Cognitec Face VACS ($9.6$)), and the deep learning FRS includes Arcface
[34] (Model R100 V1), VGGFace [35] (Version 2), and Facenet [36].

###  7.1 Mathematical Formulation of GMAP

Let $\mathbb{P}$ denote the set of probe images, $\mathbb{F}$ denote the set
of FRS, $\mathbb{D}$ denote the set of Morphing Attack Generation Type,
$\mathbb{M}_{d}$ denote the face morphing image set used for Morphing Attack
Generation Type $d$, $\tau_{l}$ indicate the similarity score threshold for
FRS ($l$), and $||$ represents the count of elements in a set during metric
evaluation. The main objective of GMAP is to introduce the accountability for
Failure to Failure To Acquire Rate (FTAR) to quantify the reliable
vulnerability. The GMAP metric is presented below:

$\displaystyle
GMAP=\min_{\forall{i,d,l}}{\sum_{i}^{|\mathbb{P}|}\sum_{j}^{|\mathbb{M}_{d}|}}\bigg{\\{}\left[(S1_{i}^{j}>\tau_{l})\wedge\cdots(Sk_{i}^{j}>\tau_{l})\right]$
(5) $\displaystyle{\times}\left[(1-FTAR)(i,d)\right]\bigg{\\}}$

where, $FTAR(i,l)$ is the failure to acquire probe image in attempt $i$ using
FRS ($l$) and $\tau_{l}$ is the similarity score threshold for FRS ($l$). The
algorithm for GMAP is presented in 3 and the code is made available in the
link [37].

Input: Set of Probe Images $\mathbb{P}$, Set of FRS $\mathbb{F}$, Set of
Morphing Attack Generation Type $\mathbb{D}$, Set of Morphing Attack Images in
$d^{\textrm{th}}$ attack $\mathbb{M}_{d}$, $\tau_{l}$ indicate the similarity
score threshold for FRS.

Output: GMAP

1: Compute GMAP Metric as follows.

2: for $l\leftarrow 1$ to $|\mathbb{F}|$ do

3: for $d\leftarrow 1$ to $|\mathbb{D}|$ do

4: for $i\leftarrow 1$ to $|\mathbb{P}|$ do

5: for $j\leftarrow 1$ to $|\mathbb{M}_{d}|$ do

6: Compute QF(i,l)=(1-FTAR(i,l))

7: Compute
GMAP(i,d,l)=$\sum_{i}^{|\mathbb{P}|}\sum_{j}^{|\mathbb{M}_{d}|}(S1_{i}^{j}>\tau_{l})\wedge\cdots(Sk_{i}^{j}>\tau_{l}){\times}QF(i,d)$

8: end for

9: end for

10: end for

11: Compute GMAP(l)=$\min_{\forall{i,d}}{GMAP(i,d,l)}$

12: end for

13: Compute GMAP=$\min_{\forall{l}}{GMAP(l)}$

14: return GMAP

Algorithm 3 Generalized Morph Attack Potential (GMAP)

(a) (b)
(c) (d)

Figure 10: Vulnerability Plots using 2D FRS using Deep Learning methods (a)
Arcface [2] (b) Cognitec [22] (FTAR Samples due to FTE are shown for
visualization using arrow mark in first quadrant) (c) Neurotech [21] ( Scaled
for visualization purposes.) (d) Facenet [1]. Note the vulnerability results
are shown for R14/Full Region only. FTAR Samples due to FTE are shown for
visualization using arrow mark in first quadrant.

Figure 11: D-EER Percentages for SOTA [27] and Proposed CFIA Method

###  7.2 Discussion of Vulnerability Analysis

The quantitative vulnerability computed for both SOTA and the proposed method
is shown in Table IV and the following are the important observation made
based on it:

  * •

For single facial attributes-based regions i.e. R1 to R4, the vulnerability is
comparable for both SOTA and the proposed method. This can be attributed to
the fact that in these regions the blending for the generation of composite
happens in a small region and the remainder of the face is generated by GAN-
based image inpainting.

  * •

Further, for multiple facial attributes-based regions, i.e. R5 to R9 the
vulnerability is comparable as one of the contributory data subjects has a
single facial attribute being used during their generation. This result in a
small region where the blending for the generation of the composite happens
and the remainder of the face is generated by GAN-based image inpainting.

  * •

Finally, for multiple facial attributes-based regions, i.e. R10 to R14 the
vulnerability for the proposed method is much higher than the SOTA. This can
be attributed to the fact that multiple facial attributes are used from both
the contributory data subjects, resulting in a large region of the face
generated by blending. This trend is shown in all the FRS, including both
automatic and deep-learning-based.

  * •

The proposed method gives the highest vulnerability for the region R14 as this
has full-face blending, which results in the highest overlapping region
between the two contributing data subjects for all FRS except Facnet. Facnet
shows higher vulnerability for both SOTA and the proposed method.

However, when the vulnerability analysis is performed based on each FRS
irrespective of the morphing attack generation type, as shown in Table VI. The
proposed method outperforms SOTA in all the FRS, both automatic and deep-
learning-based. Thus, the proposed method gives higher bounds of vulnerability
for all 14 different regions. Table V shows the MAP metric and it shows higher
vulnerability values compared with GMAP as shown in Table IV

GMAP % (Probe Attempts)
---
FRS | Method | R1 | R2 | R3 | R4 | R5 | R6 | R7 | R8 | R9 | R10 | R11 | R12 | R13 | R14
Neurotech (FAR=0.1%) | SOTA [27] | 18.2 | 10.4 | 10.9 | 8.1 | 16.2 | 14.6 | 19.0 | 18.8 | 19.3 | 14.2 | 14.0 | 21.6 | 15.0 | 11.3
Proposed | 13.8 | 10.2 | 9.7 | 17.6 | 13.5 | 14.4 | 16.0 | 17.1 | 19.3 | 22.2 | 22.9 | 21.0 | 23.7 | 23.3
Cognitec (FAR=0.1%) | SOTA [27] | 31.6 | 22.2 | 23.3 | 19.8 | 27.7 | 28.8 | 33.1 | 34.0 | 37.9 | 30.0 | 24.8 | 41.1 | 25.1 | 21.3
Proposed | 30.5 | 22.9 | 22.3 | 43.9 | 28.1 | 26.9 | 31.9 | 34.7 | 35.3 | 54.6 | 55.1 | 43.0 | 57.6 | 60.7
Arcface (FAR=0.1%) | SOTA [27] | 70.4 | 58.6 | 60.5 | 52.6 | 70.7 | 69.1 | 72.8 | 71.5 | 69.0 | 69.0 | 67.5 | 74.0 | 69.5 | 72.2
Proposed | 67.2 | 58.0 | 60.1 | 72.3 | 70.3 | 68.3 | 72.4 | 71.8 | 71.3 | 84.1 | 82.8 | 76.4 | 86.7 | 89.8
VGGFace (FAR=0.1%) | SOTA [27] | 65.1 | 64.1 | 62.6 | 63.5 | 64.8 | 63.8 | 66.0 | 65.6 | 66.9 | 67.1 | 65.3 | 65.8 | 68.0 | 67.6
Proposed | 65.3 | 64.3 | 62.9 | 65.8 | 66.3 | 68.0 | 69.0 | 65.9 | 68.4 | 70.4 | 70.4 | 68.5 | 70.9 | 71.8
Facenet (FAR=0.1%) | SOTA [27] | 95.7 | 96.8 | 95.3 | 93.7 | 95.2 | 96.4 | 95.8 | 95.7 | 96.0 | 95.5 | 94.4 | 96.3 | 94.6 | 95.1
Proposed | 96.0 | 97.6 | 96.2 | 97.3 | 95.4 | 97.2 | 95.3 | 96.3 | 97.3 | 97.1 | 97.2 | 96.5 | 96.5 | 96.9

TABLE IV: Vulnerability analysis using the proposed GMAP metric (probe
attempts-based) for the proposed method and the SOTA [27], where the
description of regions is provided in Table I. Note the proposed methods give
higher vulnerability when the area of composition increases.

MAP % (Probe Attempts)
---
FRS | Method | R1 | R2 | R3 | R4 | R5 | R6 | R7 | R8 | R9 | R10 | R11 | R12 | R13 | R14
Neurotech (FAR=0.1%) | SOTA [27] | 54.4 | 33.1 | 35.1 | 32.4 | 50.0 | 44.3 | 59.1 | 58.1 | 59.5 | 51.3 | 50.3 | 65.4 | 55.0 | 45.3
Proposed | 43.1 | 31.6 | 33.1 | 57.8 | 41.8 | 43.5 | 49.7 | 51.9 | 56.9 | 72.2 | 75.2 | 63.9 | 79.5 | 79.2
Cognitec (FAR=0.1%) | SOTA [27] | 31.9 | 22.5 | 23.5 | 20.0 | 28.0 | 29.2 | 33.5 | 34.4 | 38.3 | 30.3 | 25.2 | 41.6 | 25.5 | 21.6
Proposed | 30.8 | 23.2 | 22.6 | 44.4 | 28.4 | 27.2 | 32.2 | 35.1 | 35.7 | 55.2 | 55.8 | 43.4 | 58.3 | 61.4
Arcface (FAR=0.1%) | SOTA [27] | 70.5 | 58.7 | 60.6 | 52.7 | 70.8 | 69.2 | 72.9 | 71.6 | 69.1 | 69.1 | 67.6 | 74.1 | 69.6 | 72.3
Proposed | 67.3 | 58.1 | 60.2 | 72.4 | 70.4 | 68.4 | 72.5 | 71.9 | 71.4 | 84.2 | 82.8 | 76.4 | 86.8 | 89.9
VGGFace (FAR=0.1%) | SOTA [27] | 65.2 | 64.2 | 62.7 | 63.6 | 64.9 | 63.9 | 66.1 | 65.7 | 67.0 | 67.2 | 65.4 | 65.9 | 68.1 | 67.7
Proposed | 65.4 | 64.4 | 63.0 | 65.9 | 66.4 | 68.1 | 69.1 | 66.0 | 68.5 | 70.5 | 70.5 | 68.6 | 71.0 | 71.9
Facenet (FAR=0.1%) | SOTA [27] | 95.8 | 96.9 | 95.4 | 93.8 | 95.3 | 96.5 | 95.9 | 95.8 | 96.1 | 95.6 | 94.5 | 96.4 | 94.7 | 95.2
Proposed | 96.1 | 97.7 | 96.3 | 97.4 | 95.5 | 97.3 | 95.4 | 96.4 | 97.4 | 97.2 | 97.3 | 96.6 | 96.6 | 97.0

TABLE V:  Vulnerability analysis using the MAP metric (probe attempts-based)
for the proposed method and the SOTA [27], where the description of regions is
provided in Table I. Note the proposed methods give higher vulnerability when
the area of composition increases and MAP metric gives higher values when
failure to enroll is not included.

GMAP % (FRS-based)
---
Neurotech (FAR=0.1%) | Cognitec (FAR=0.1%) | Arcface (FAR=0.1%) | VGGFace (FAR=0.1%) | Facenet (FAR=0.1%)
SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
8.1 | 9.7 | 19.8 | 22.3 | 52.6 | 58.0 | 62.6 | 62.9 | 93.7 | 95.3

TABLE VI: Vulnerability analysis using the proposed GMAP metric (FRS-based)
for the proposed method and the SOTA [27], where the description of regions is
provided in Table I. Note the proposed methods give higher vulnerability for
all individual FRS.

Detection Method (Region) | D-EER (%) | BPCER @ APCER =
---|---|---
|  | 5% | 10%
R1 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 50.0 | 42.9 | 96.0 | 92.1 | 92.5 | 86.3
DetAlgo2 [39] | 50.0 | 50.0 | 95.9 | 94.3 | 92.4 | 89.2
DetAlgo3 [40] | 38.2 | 28.7 | 85.4 | 74.0 | 78.5 | 57.2
R2 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 50.0 | 44.6 | 96.0 | 93.0 | 92.3 | 86.0
DetAlgo2 [39] | 50.0 | 50.0 | 96.2 | 94.4 | 92.3 | 91.2
DetAlgo3 [40] | 39.5 | 29.3 | 87.1 | 78.5 | 78.9 | 64.8
R3 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 50.0 | 47.0 | 95.5 | 93.9 | 92.1 | 88.7
DetAlgo2 [39] | 50.0 | 50.0 | 96.3 | 94.7 | 92.8 | 91.5
DetAlgo3 [40] | 40.6 | 32.2 | 88.3 | 79.1 | 80.3 | 65.8
R4 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 49.0 | 39.6 | 94.3 | 90.5 | 89.0 | 81.7
DetAlgo2 [39] | 50.0 | 50.0 | 96.1 | 92.6 | 92.8 | 88.9
DetAlgo3 [40] | 42.8 | 32.4 | 89.8 | 77.7 | 82.2 | 64.6
R5 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 50.0 | 45.0 | 95.0 | 93.3 | 91.0 | 86.6
DetAlgo2 [39] | 50.0 | 50.0 | 96.6 | 93.9 | 92.7 | 90.8
DetAlgo3 [40] | 42.0 | 31.6 | 87.9 | 76.5 | 80.7 | 64.8
R6 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 50.0 | 44.1 | 95.5 | 92.5 | 91.8 | 84.6
DetAlgo2 [39] | 50.0 | 50.0 | 95.3 | 92.7 | 91.0 | 88.9
DetAlgo3 [40] | 38.0 | 29.3 | 85.3 | 73.2 | 78.5 | 60.2
R7 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 50.0 | 43.5 | 95.0 | 92.5 | 90.8 | 85.8
DetAlgo2 [39] | 50.0 | 49.7 | 95.8 | 92.8 | 92.1 | 87.5
DetAlgo3 [40] | 39.5 | 29.8 | 87.9 | 73.7 | 78.9 | 60.0
R8 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 50.0 | 44.6 | 96.2 | 94.0 | 92.2 | 85.9
DetAlgo2 [39] | 50.0 | 49.8 | 96.0 | 92.5 | 92.0 | 87.4
DetAlgo3 [40] | 41.7 | 30.6 | 87.4 | 75.4 | 81.3 | 61.5
R9 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 50.0 | 43.4 | 95.7 | 91.8 | 90.6 | 84.5
DetAlgo2 [39] | 50.0 | 50.0 | 95.8 | 91.6 | 91.5 | 86.3
DetAlgo3 [40] | 40.5 | 28.4 | 87.5 | 78.5 | 81.7 | 63.2
R10 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 48.2 | 38.5 | 94.2 | 86.6 | 87.9 | 77.9
DetAlgo2 [39] | 50.0 | 48.0 | 94.7 | 91.8 | 90.7 | 87.4
DetAlgo3 [40] | 41.6 | 30.2 | 87.0 | 76.2 | 80.9 | 61.9
R11 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 49.0 | 37.3 | 95.2 | 88.0 | 89.6 | 76.3
DetAlgo2 [39] | 50.0 | 50.0 | 93.7 | 92.7 | 92.2 | 88.6
DetAlgo3 [40] | 41.9 | 31.7 | 87.1 | 80.0 | 80.7 | 67.8
R12 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 50.0 | 43.7 | 95.4 | 90.6 | 91.4 | 83.0
DetAlgo2 [39] | 50.0 | 48.8 | 94.8 | 91.4 | 91.4 | 86.0
DetAlgo3 [40] | 41.8 | 30.8 | 87.6 | 76.4 | 80.5 | 64.1
R13 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 48.7 | 37.4 | 94.2 | 86.8 | 89.0 | 76.6
DetAlgo2 [39] | 50.0 | 49.1 | 93.6 | 91.6 | 91.4 | 86.7
DetAlgo3 [40] | 41.4 | 32.2 | 86.9 | 78.5 | 80.2 | 64.8
R14 | SOTA [27] | Proposed | SOTA [27] | Proposed | SOTA [27] | Proposed
DetAlgo1 [38] | 50.0 | 42.6 | 97.6 | 90.2 | 94.8 | 83.7
DetAlgo2 [39] | 50.0 | 49.5 | 97.2 | 95.7 | 93.4 | 88.0
DetAlgo3 [40] | 46.4 | 34.0 | 92.2 | 83.8 | 83.2 | 72.1

TABLE VII: CFIA Attack Detection using DetAlgo1 [38], DetAlgo2 [39], and
DetAlgo3 [40]

##  8 Composite Face Image Attack Detection

In this section, we benchmark CFIA detection based on a single image. Since
the generation of CFIA is procedurally similar to morphing generation with
transparent blending. Therefore, we have employed three different Face
Morphing Attack Detection (MAD) techniques to benchmark the CFIA detection.
MAD methods are selected by considering their detection performance on various
morphing data sources, including NIST FRVT MORPH benchmarking. To this extent,
we have chosen three different S-MAD approaches, namely: Color denoising based
S-MAD (DetAlgo1) [38], Hybrid features (DetAlgo2) [39] and Residual noise-
based S-MAD Network (DetAlgo3) [40]. These algorithms are briefly explained as
follows:

Color denoising based S-MAD (DetAlgo1) [38]: DetAlgo1 is based on using the
color information by converting the RGB image HSV color space. Then, each
color channel is denoised using a Deep Convolutional Neural Network to compute
the corresponding residual noise. In the next step, Pyramid LBP (P-LBP) and an
SRKDA classifier for final detection.

Hybrid features (DetAlgo2) [39]: DetAlgo2 is based on two different colors
spaces. Given the RGB image, firstly, it is converted to HSV and YCbCr color
space. In the next step, micro-texture features are computed using pyramid-LBP
and passed through the SRKDA classifier. The final classification is performed
using SUM rule fusion to make the final decision on detection.

Residual noise-based S-MAD Network (DetAlgo3) [40]: DetAlgo3 is based on the
computing the residual noise using the Multi-Scale Context Aggregation Network
(MS-CAN). The residual noise is further processed through Alexnet to obtain
the classified features using the Collaborative Representative Classifier
(CRC) to make the final decision to detect the attack.

Effectively evaluate CAD techniques by considering the off-the-shelf S-MAD
methods, which are trained on the face morphing attack detection. Three
different S-MAD methods employed in this work are trained using different
morph generation types (landmark-based and deep learning) and three different
mediums (Digital, print-scanned, and print-scanned compression) generated
using the publicly available FRGC face database. The quantitative results are
presented using the ISO/IEC metrics [41] which are as follows: 1) Attack
Presentation Classification Error Rate (APCER ($\%$)) defining the percentage
of attack images (morph images) incorrectly classified as bona fide images
[41] , 2) Bonafide Presentation Classification Error Rate (BPCER ($\%$))
defining the percentage of bona fide images incorrectly classified as attack
images [41] and 3) Detection Equal Error Rate (D-EER ($\%$)) [13]. The
detection performance is benchmarked with both SOTA and proposed CFIA images
and quantitative results are presented in Table VII and bar chart with D-EER
(%) on all 14 different regions. Based on the obtained results following are
the main observations:

  * •

The CFIA detection performance is degraded with all three detection
algorithms.

  * •

Among three different detection algorithms. DetAlgo3 indicates the better
detection accuracy attributed to the quantification of residual noise.

  * •

Among the 14 different regions, the degraded detection performance is noted
with the R14 on all three detection algorithms.

Thus, based on the obtained results, we can conclude that the detection of
CFIA attacks is very challenging and this needs more sophisticated detection
algorithms to be devised for reliable detection.

##  9 Conclusions and Future-Work

In this work, we presented a new type of digital attack based on the facial
attributes and we termed them Composite Face Image Attack (CFIA). Given the
facial images from the two contributary data subjects, the proposed will first
segment the face images into six different attributes independently. Then,
these segments are blended using a transparent mask based on both single face-
attribute and multiple face attributes. These attributes are processed using
the image inpainting based on pre-trained GAN to generate the final CFIA
samples. In this work, face images from two contributary data subjects, we
generate 14 different composite face images based on single and multiple face
attributes. We contributed a new dataset with 1000 unique identities that will
result in 14000 CFIA samples. Extensive experiments are performed to evaluate
the attack potential of the newly generated CFIA using four different FRS. To
effectively benchmark the vulnerability of the generated CFIA, we have
introduced a generalized vulnerability metric. Further, we benchmark the
detection accuracy using both human and automatic detection techniques. Our
results demonstrated that the proposed CFIA could indicate the vulnerability
of the FRS while it is difficult to detect using both human and automatic
detection techniques. In the future work, we would like to extend the present
work in several directions: 1) Generation of composites of higher quality, 2)
Evaluation of the proposed method on real face images on public datasets, 3)
Development of novel detection techniques.

## References

  * [1] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815–823, 2015.
  * [2] Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou.  Arcface: Additive angular margin loss for deep face recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4690–4699, 2019.
  * [3] NIST.  NIST FRVT Morph.  https://pages.nist.gov/frvt/html/frvt_morph.html, 2020.  [Online; accessed 19-January-2022].
  * [4] Raghavendra Ramachandra and Christoph Busch.  Presentation attack detection methods for face recognition systems: A comprehensive survey.  ACM Computing Surveys (CSUR), 50(1):1–37, 2017.
  * [5] Faseela Abdullakutty, Eyad Elyan, and Pamela Johnston.  A review of state-of-the-art in face presentation attack detection: From early development to advanced deep learning and multi-modal fusion methods.  Information Fusion, 75:55–69, 2021.
  * [6] Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [7] Ying Xu, Kiran Raja, Raghavendra Ramachandra, and Christoph Busch.  Adversarial attacks on face recognition systems.  In Handbook of Digital Face Manipulation and Detection, pages 139–161. Springer, Cham, 2022.
  * [8] Fatemeh Vakhshiteh, Ahmad Nickabadi, and Raghavendra Ramachandra.  Adversarial attacks against face recognition: A comprehensive study.  IEEE Access, 9:92735–92756, 2021.
  * [9] Naveed Akhtar and Ajmal Mian.  Threat of adversarial attacks on deep learning in computer vision: A survey.  IEEE Access, 6:14410–14430, 2018.
  * [10] Quek, Alyssa.  Face Morpher.  https://github.com/alyssaq/face_morpher, 2018.  [Online; accessed 19-January-2022].
  * [11] Matteo Ferrara, Annalisa Franco, and Davide Maltoni.  Face demorphing.  IEEE Transactions on Information Forensics and Security, 13(4):1008–1017, 2017.
  * [12] Naser Damer, Alexandra Moseguí Saladié, Andreas Braun, and Arjan Kuijper.  Morgan: Recognition vulnerability and attack detectability of face morphing attacks created by generative adversarial network.  In 2018 IEEE 9th International Conference on Biometrics Theory, Applications and Systems (BTAS), pages 1–10, 2018.
  * [13] Haoyu Zhang, Sushma Venkatesh, Raghavendra Ramachandra, Kiran Raja, Naser Damer, and Christoph Busch.  Mipgan—generating strong and high quality morphing attacks using identity prior driven gan.  IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(3):365–383, 2021.
  * [14] Naser Damer, Kiran Raja, Marius Süßmilch, Sushma Venkatesh, Fadi Boutros, Meiling Fang, Florian Kirchbuchner, Raghavendra Ramachandra, and Arjan Kuijper.  Regenmorph: visibly realistic gan generated face morphing attacks by attack re-generation.  In International Symposium on Visual Computing, pages 251–264. Springer, 2021.
  * [15] Sushma Venkatesh, Raghavendra Ramachandra, Kiran Raja, and Christoph Busch.  Face morphing attack generation & detection: A comprehensive survey.  IEEE Transactions on Technology and Society, 2021.
  * [16] Sankini Rancha Godage, Frøy Løvåsda, Sushma Venkatesh, Kiran Raja, Raghavendra Ramachandra, and Christoph Busch.  Analyzing human observer ability in morphing attack detection–where do we stand?  arXiv e-prints, pages arXiv–2202, 2022.
  * [17] Matteo Ferrara, Annalisa Franco, and Davide Maltoni.  The magic passport.  In IEEE International Joint Conference on Biometrics, pages 1–7, 2014.
  * [18] Le Qin, Fei Peng, Sushma Venkatesh, Raghavendra Ramachandra, Min Long, and Christoph Busch.  Low visual distortion and robust morphing attacks based on partial face image manipulation.  IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(1):72–88, 2021.
  * [19] Patrick Pérez, Michel Gangnet, and Andrew Blake.  Poisson image editing.  In ACM SIGGRAPH 2003 Papers, SIGGRAPH ’03, page 313–318, New York, NY, USA, 2003. Association for Computing Machinery.
  * [20] Omkar M. Parkhi, Andrea Vedaldi, and Andrew Zisserman.  Deep face recognition.  In Mark W. Jones Xianghua Xie and Gary K. L. Tam, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 41.1–41.12. BMVA Press, September 2015.
  * [21] Neurotechnology.  Neurotech Verilook SDK (11.1).  https://www.neurotechnology.com/verilook.html, 2019.  [Online; accessed 19-January-2022].
  * [22] Cognitec.  Cognitec Face VACS (9.6).  https://www.cognitec.com/facevacs-technology.html, 2019.  [Online; accessed 19-January-2022].
  * [23] Zhengwei Wang, Qi She, and Tomas E Ward.  Generative adversarial networks in computer vision: A survey and taxonomy.  ACM Computing Surveys (CSUR), 54(2):1–38, 2021.
  * [24] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of GANs for improved quality, stability, and variation.  In International Conference on Learning Representations, 2018.
  * [25] Tero Karras, Samuli Laine, and Timo Aila.  A style-based generator architecture for generative adversarial networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [26] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [27] Lucy Chai, Jonas Wulff, and Phillip Isola.  Using latent space regression to analyze and leverage compositionality in gans.  In International Conference on Learning Representations, 2021.
  * [28] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba.  Scene parsing through ade20k dataset.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633–641, 2017.
  * [29] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun.  Unified perceptual parsing for scene understanding.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 418–434, 2018.
  * [30] Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou.  Interfacegan: Interpreting the disentangled face representation learned by gans.  IEEE transactions on pattern analysis and machine intelligence, 2020\.
  * [31] Eklavya Sarkar, Pavel Korshunov, Laurent Colbois, and Sébastien Marcel.  Vulnerability analysis of face morphing attacks from landmarks and generative adversarial networks.  arXiv preprint arXiv:2012.05344, 2020.
  * [32] Naser Damer, César Augusto Fontanillo López, Meiling Fang, Noémie Spiller, Minh Vu Pham, and Fadi Boutros.  Privacy-friendly synthetic data for the development of face morphing attack detectors.  arXiv preprint arXiv:2203.06691, 2022.
  * [33] Alain Horé and Djemel Ziou.  Image quality metrics: Psnr vs. ssim.  In 2010 20th International Conference on Pattern Recognition, pages 2366–2369, 2010.
  * [34] InsightFace.  Insightface: 2d and 3d face analysis project.  https://github.com/deepinsight/insightface.git, 2022.
  * [35] RC Malli.  keras-vggface.  https://github.com/rcmalli/keras-vggface.git, 2022.
  * [36] David Sandberg.  Facenet tensorflow.  https://github.com/davidsandberg/facenet.git, 2022.
  * [37] NTNU Jag Mohan Singh.  Generalized morphing attack potential.  https://github.com/, 2022.
  * [38] Sushma Venkatesh, Raghavendra Ramachandra, Kiran Raja, Luuk Spreeuwers, Raymond Veldhuis, and Christoph Busch.  Morphed face detection based on deep color residual noise.  In 2019 Ninth International Conference on Image Processing Theory, Tools and Applications (IPTA), pages 1–6, 2019.
  * [39] Raghavendra Ramachandra, Sushma Venkatesh, Kiran Raja, and Christoph Busch.  Towards making morphing attack detection robust using hybrid scale-space colour texture features.  In 2019 IEEE 5th International Conference on Identity, Security, and Behavior Analysis (ISBA), pages 1–8, 2019.
  * [40] Sushma Venkatesh, Raghavendra Ramachandra, Kiran Raja, Luuk Spreeuwers, Raymond Veldhuis, and Christoph Busch.  Detecting morphed face attacks using residual noise from deep multi-scale context aggregation network.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), March 2020.
  * [41] ISO/IEC JTC1 SC37 Biometrics.  ISO/IEC 30107-3. Information Technology - Biometric presentation attack detection - Part 3: Testing and Reporting.  International Organization for Standardization, 2017.