# Conditional Identity Disentanglement for Differential Face Morph Detection

Sudipta Banerjee and Arun Ross
Michigan State University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We present the task of differential face morph attack detection using a
conditional generative network (cGAN). To determine whether a face image in an
identification document, such as a passport, is morphed or not, we propose an
algorithm that learns to implicitly disentangle identities from the morphed
image conditioned on the trusted reference image using the cGAN. Furthermore,
the proposed method can also recover some underlying information about the
second subject used in generating the morph. We performed experiments on AMSL
face morph, MorGAN, and EMorGAN datasets to demonstrate the effectiveness of
the proposed method. We also conducted cross-dataset and cross-attack
detection experiments. We obtained promising results of 3% BPCER @ 10% APCER
on intra-dataset evaluation, which is comparable to existing methods; and 4.6%
BPCER @ 10% APCER on cross-dataset evaluation, which outperforms state-of-the-
art methods by at least 13.9%.

##  1 Introduction

Face morphing involves a continuous transition from the face image of one
individual (source identity) to the face image of the second individual
(target identity) [25]. The idea of morphing has been used for visual effects
in entertainment videos. But recently, it has been demonstrated that face
morphing can be used for adversarial purposes. Since a morphed face contains
features from two individuals, it can successfully match both identities,
thereby posing a security threat. See Figure 1. This problem is of practical
concern due to two reasons: 1) the prolific use of biometric data in official
documents for authentication in unattended border control systems (viz., face
images in passports accepted at e-gates), and 2) the ease of access to face
image editing software (e.g., FaceMorpher [2]). The idea of using morphed face
image in identity documents was first postulated in [9].111Note that face
morphing can be used in a positive manner also in privacy-preserving
applications [17]. ††footnotetext:  978-1-6654-3780-6/21/$31.00 ©2021 IEEE
Later, a real-world case of face morphing was reported where an art activist
morphed her face image with that of the photograph of an EU Foreign Affairs
Commissioner to apply for a German passport, prompting the authorities to
reconsider using self-created digital photographs in identity documents [14].
See Figure 2. Face morph attack has piqued the interest of the research
community and government agencies alike, leading to the investigation of
methods that can not only produce realistic face morphs but also successfully
detect such types of morphs. The EU-funded iMARS project is geared towards
developing image morphing techniques and manipulation attack detection
solutions for identification documents [3]. Note that current deep-fake
detectors cannot effectively discriminate between morphed and legitimate
bonafide (non-morphed) images. Further, existing anti-spoofing solutions
developed for presentation attacks are often not suited to detect morph
attacks since the latter are digital alterations rather than physical
presentations.

Figure 1: Illustration of face morphing exploited to provide access to two
different subjects, Alice and Jenny. Both can use the same morphed image on
the passport at an airport e-gate for access. Figure 2: The morphed image in
the middle that was used to apply for a passport in the EU was created using
the photograph of an activist (left) and an official (right). Image courtesy:
Peng!(MaskID) reproduced from [14].

Face morph generation typically involves two kinds of approaches: 1) landmark-
based approaches, and 2) generative model-based approaches. The first approach
utilizes facial landmarks from two face images for aligning them using warping
[12, 22]. Image blending principles are then used to combine the pixels in the
overlapped regions to construct the morphed image. An optional post-processing
step involving histogram equalization or Poisson blending may be applied to
achieve visual realism and to remove ghosting artefacts. The second approach
leverages the generative capability of adversarial networks to synthesize
morphs. MorGAN [8], MIPGAN [28] and EMorGAN [7] are examples of such
approaches. The final morphed image can then be inserted into an official
document used at access control points. The morphs can be so realistic that
visual inspection alone may not be sufficient for detecting them. See Figure
2.

Face morph attack detection (MAD) can be performed using two methods: 1)
reference-free single image-based MAD, and 2) reference-based differential
MAD. The former tries to address the following question. Given a document face
image can we determine whether it is morphed or not? According to the NIST
FRVT report [16], the best performing algorithm in this category results in
93.8% APCER @ 10% BPCER on high quality morphs in “Dataset:Manual” [16]. Here,
bonafide presentation classification error rate (BPCER) denotes the proportion
of bonafide (non-morphed) images incorrectly classified as morphed images, and
attack presentation classification error rate (APCER) denotes the proportion
of morphed images incorrectly classified as non-morphed images. The second
method tries to address the following question. Given a document face image,
and a live capture as a reference image, can we determine whether the document
image is morphed or not? According to the NIST FRVT report [16], the best
performing algorithm in this category results in 9.1% APCER @ 10% BPCER on
high quality morphs in “Dataset:Manual”. It is evident, based on these
results, that the state-of-the-art performance requires significant
improvement. The readers are referred to surveys in [4, 27] for a
comprehensive overview of face morph generation and detection. In this paper,
we focus on developing a novel differential MAD method to advance the state of
the art. The novelty of the method lies in formulating the differential MAD
problem using an information theoretic framework for a sound detection
strategy.

The remainder of the paper is organized as follows. Section 2 describes
existing differential MAD strategies. Section 3 describes the proposed method.
Section 4 outlines the experiments. Section 5 reports and analyzes the
results. Finally, section 6 concludes the paper with summary and future
directions.

##  2 Related Work

Differential MAD strategies use a reference-based approach. Here, the trusted
live face image of a subject taken during the time of acquisition (called the
reference image) is used along with the document face image (in the passport,
for example) to determine whether the latter is morphed or non-morphed. The
first known work on differential MAD performs “demorphing” which uses the
difference computed between the document image and the reference image to
ascertain whether the document image is morphed or not, and alerts the officer
for additional inspection, if necessary [10]. Demorphing also tries to uncover
the identity of the second subject. Later, GAN-based de-morphing has been
proposed in the literature [18]. Another class of techniques utilizes features
such as BSIF or the disparity between landmarks, along with a classifier, to
detect morphs differentially [20, 6, 19]. The detection performance is further
enhanced by the use of deep face representations [21]. Recently, a method that
uses appearance and landmark disentanglement modules for differential MAD was
proposed in [24]. It uses the disentanglement module to learn complementary
information, and contrastive loss to boost detection performance. An approach
referred to as Focused Layer-Wise Relevance Propagation (FLRP) aims at
explaining the decision made by the deep neural network in detecting morphs
for better interpretability [5].

##  3 Proposed Method

Figure 3: The proposed method uses a conditional disentanglement framework to
discriminate between morphed and non-morphed document images conditioned on
the reference image. The biometric comparator scores can be compared against a
suitable threshold, $\mathcal{T}$, for differential morph attack detection.

###  3.1 Conditional GAN

The proposed method requires a tool that can convert a source image, $\bm{X}$,
which is the ‘document’ image, to a target image, $\bm{Y}$, which is the
‘reference’ image. This can be achieved via image translation guided by a
conditional generative adversarial network (cGAN) [13]. It uses the following
objective function.

$\displaystyle G^{*}$
$\displaystyle=\displaystyle\arg\displaystyle\min_{G}\displaystyle\max_{D}\\{\mathbb{E}_{(\bm{X},\bm{Y})}[\log
D(\bm{X},\bm{Y})]$
$\displaystyle+\mathbb{E}_{(\bm{X},\bm{z})}[\log(1-D(\bm{X},G(\bm{X},\bm{z})))]\\}$
$\displaystyle+\lambda\mathbb{E}_{(\bm{X},\bm{Y},\bm{z})}[\|(\bm{Y}-G(\bm{X},\bm{z}))\|_{1}]$
(1)

Here, $D$ refers to the discriminator and $G$ refers to the generator. The two
inputs to the network are $\bm{X}$ (document image) and $\bm{Y}$ (reference
image). The reference image is always assumed to be a bonafide since it is
captured “live”. Now the network has to learn to translate from $\bm{X}$ to
$\bm{Y}$. The Gaussian random noise vector $\bm{z}$ regularizes the training
and diversifies the output. If $\bm{X}$ is non-morphed, then the task of the
generator is to simply reconstruct itself with some additional variations due
to age, pose, illumination and expression, which can be deduced from $\bm{Y}$.
However, if $\bm{X}$ is morphed, the challenge is two-fold: (i) remove traces
of the second identity, and (ii) incorporate variations related to pose, age,
expression and appearance. We train the network with both types of examples:
$(\bm{X}_{\tiny{non-morphed}},\bm{Y})$ and $(\bm{X}_{\tiny{morphed}},\bm{Y})$,
i.e., (non-morphed source and bonafide target images) and (morphed source and
bonafide target images). However, the method does not require any labels
(supervision) regarding which pairs correspond to morphed source images and
which pairs correspond to non-morphed source images. The idea is to make the
network automatically learn to disentangle the composite (morphed) image
conditioned on the reference image. If the image is not a composite, the task
is simpler, it has to generate a variant of the source image.

###  3.2 Rationale of the Proposed Method

The majority of the literature poses differential MAD as a supervised
classification problem, with the underlying premise that discriminative
features can be deduced using either pre-defined filters (hand-crafted) or
automatically learned filters (deep-learning). In contrast, we make the case
that the task of differential MAD can be framed in an information theoretic
framework that describes the translation from the document image to the
reference image.

$\displaystyle\mathcal{H}(\bm{X}\mid\bm{Y})$
$\displaystyle=\mathcal{H}(\bm{Y})-\mathcal{H}(\bm{X},\bm{Y})$ (2)

If $\bm{X}$ is non-morphed, then $\bm{Y}=[\bm{X}+\delta]$, where $\delta$
captures the intra-class variations between document and reference images
belonging to the same subject. Therefore, the conditional entropy can be
formulated as,

$\displaystyle\mathcal{H}(\bm{X}\mid[\bm{X}+\delta])=\mathcal{H}([\bm{X}+\delta])-\mathcal{H}(\bm{X},[\bm{X}+\delta])$
(3)

Clearly the uncertainty associated with inferring $\bm{X}$ from a slightly
noisy or altered version of itself ($\delta$ can be interpreted as additive
noise) will be low. On the contrary, for a morphed image comprising two
statistically independent distinct subjects, the uncertainty will be
relatively higher due to the presence of a second identity.

$\displaystyle\mathcal{H}([\bm{X}_{1},\bm{X}_{2}]\mid[\bm{X}_{1}+\delta])$
$\displaystyle=$ $\displaystyle\mathcal{H}([\bm{X}_{1}+\delta])$
$\displaystyle-\mathcal{H}(\bm{[\bm{X}_{1},\bm{X}_{2}]},[\bm{X}_{1}+\delta])$
(4)

In Eqn. (3.2), we assume subject $\bm{X}_{1}$ appears at the verification
checkpoint, so the uncertainty arises due to $\bm{X}_{2}$. A similar situation
arises when the roles are reversed. Therefore, we expect
$\displaystyle\mathcal{H}(\bm{X}_{morphed}\mid\bm{Y})>\mathcal{H}(\bm{X}_{non-
morphed}\mid\bm{Y})$. Here, we interpret entropy loosely as the disparity
between the output of the cGAN and the reference image. Next, we describe the
two steps in which we implement the proposed method.

In the first step, we use the cGAN to translate the source image (document) to
the target image (reference): $\bm{X}\rightarrow\bm{Y}$. If $\bm{X}$ is non-
morphed and represents the same individual as $\bm{Y}$, then the output
(translated) image will be more similar to $\bm{Y}$. On the other hand, if
$\bm{X}$ is morphed, implying that it comprises two identities
($\bm{X}_{1},\bm{X}_{2}$), then the output image will be less similar to
$\bm{Y}$. The translation will force the network to implicitly learn to
disentangle identities. This is because, in order to translate from the source
image (two identities if morphed) to the target image (only one identity),
i.e., $[\bm{X}_{1},\bm{X}_{2}]\rightarrow\bm{X}_{2}$ (or $\bm{X}_{1}$), it
will try to remove traces pertaining to the second identity not present in the
reference image, i.e., $\bm{X}_{1}$ (or $\bm{X}_{2}$), thereby striving to
decouple the two identities present in the morphed image. We refer to this as
conditional identity disentanglement, where we disentangle the identities from
a morphed image conditioned on a reference image. A fortuitous outcome of this
process is that we can use the same method for deciphering information about
the second subject from the morphed image. This is the novelty of the proposed
method. By positing the differential morph detection problem using an
information theoretic framework, not only can we detect morphs, but also
disentangle the identities. But how do we quantify the disparities between the
output of the cGAN and the reference image to deduce whether a document image
is morphed or not?

In the second step, we use the output, $\bm{O}=G(\bm{X},\bm{z})$, of the cGAN,
where $\bm{z}$ is the random noise vector, and compare it with the reference
image, $\bm{Y}$, using a biometric comparator, $\mathcal{M}$. The score
produced after the comparison can then be compared against a user-defined
threshold, $\mathcal{T}$, that regulates error rates for the intended
application to make the final decision:

$\displaystyle\mathcal{M}(G(\bm{X},\bm{z}),\bm{Y})=\mathcal{M}(\bm{O},\bm{Y});$
(5) $\displaystyle\noindent\bm{X}$
$\displaystyle=\begin{cases}\mbox{Morphed,}&\mbox{if}\hskip
5.69046pt\mathcal{M}(\bm{O},\bm{Y})<\mathcal{T},\\\ \mbox{Non-
morphed,}&\mbox{otherwise.}\end{cases}$ (6)

Figure 3 outlines the proposed method.

##  4 Experiments

We describe the (i) datasets, (ii) conditional GAN setup, and (iii)
experimental protocols used in this work.

###  4.1 Dataset

We used three datasets in this work. (i) AMSL face morph dataset [15, 1]: It
contains images from 102 subjects captured with neutral as well as smiling
expressions. There are 2,175 morphed images corresponding to 92 subjects
created using a landmark-based approach. In our problem formulation, the
document image can be either (a) an image with neutral expression which will
be considered as the bonafide, or (b) a morphed image. The reference image is
a trusted live capture (bonafide) corresponding to the image of the same
subject but with a smiling expression. (ii) MorGAN dataset [8]: It contains
500 bonafide images. Two morphs are generated using a generative network for
each of these bonafide images from the two subjects most similar to the
bonafide image resulting in 1,000 morphed images, which were split into train
and test sets. (iii) EMorGAN dataset [7]: It uses a cascaded image enhancement
network to improve the quality of the morphed images synthesized using MorGAN,
and has the same train and test split as [8].

###  4.2 Implementation Details

We explored different options for conditional GANs (cGANs) in the literature
and found PIX2PIX [11] to be a suitable choice for this work. PIX2PIX uses a
cGAN to translate images from one domain (e.g., sketch) to another domain
(e.g., photorealistic images). PIX2PIX [11] follows $Conv\rightarrow
BatchNorm\rightarrow ReLU$ architecture both in the generator and
discriminator. The discriminator loss function minimizes the difference
between the real and the fake images. The generator loss function maximizes
the log-likelihood of the generated images while ensuring they are as close as
possible to the target images using $\mathcal{L}_{1}$ loss. Readers are
referred to [11] for additional details about the implementation. We used
mini-batch stochastic gradient descent optimization algorithm with Adam solver
at an initial learning rate of $\displaystyle 2\times 10^{-4}$ and momentum
parameter of $\displaystyle 0.5$, and trained for 50/100/200 epochs (compared
to 600K iterations in [24]).

Figure 4: (a) Illustration of conditionally disentangled outputs generated
using the proposed method for two subjects (top and bottom) belonging to the
AMSL dataset when their document images were morphed (red) and non-morphed
(green). (b) Zoomed-in view of the periocular regions of the generated outputs
to emphasize the difference between morphed and non-morphed images.

###  4.3 Experiments

We conducted three experiments designed to answer the following research
questions.

Experiment 1: How does the proposed differential MAD method perform?
To answer the above question, we trained on the AMSL dataset using 120 pairs
of document and reference images out of which 60 pairs were morphed document
images and the other 60 pairs were bonafide document images. These 60 subjects
constitute 65% of the 92 subjects, while the test set comprised of 778 images
(745 morphed and 33 bonafide images) corresponding to the remaining 35% of the
subjects. During training, no label is required to indicate which pairs
correspond to morphed images and which pairs correspond to non-morphed images.
We feed the images generated by our network and the target reference images to
a face recognition system (a COTS face comparator) and use the scores to
determine whether a document image is morphed or not. We selected a subset of
training images provided by the authors in the MorGAN [8] and the EMorGAN [7]
datasets. We trained on 120 pairs (60 pairs of bonafide document images and 60
pairs of morphed document images), and used the entire test set for evaluation
in both cases.

Experiment 2: Is the proposed method generalizable under cross-dataset and
cross-attack scenarios?
To answer the above question, we trained on one dataset (one type of attack)
and tested on the remaining two datasets (other types of attacks). AMSL
dataset represents a landmark-based morph attack while MorGAN and EMorGAN
datasets represent generative model-based attacks.

Experiment 3: Can the method be used for recovering some discriminative
information about the second subject?
To answer the above question, we used a different training strategy compared
to morph detection. In this experiment, we used the pixel-wise difference
image computed between the document and the reference image from the AMSL
dataset as the source image, and the document image as the target image. The
intuition behind this strategy is to ensure that the network learns to map the
residual (reminiscent of the second subject contributing to the morph) to the
morphed target image.

##  5 Results and Analysis

We report both qualitative and quantitative results of the proposed method. We
report the results in terms of the metrics predominantly used in the morph
detection literature: BPCER @ APCER of 10% following [24] as well as APCER @
BPCER of 10% following [16].

(a) Bonafide and Morphed-Before (b) Bonafide and Morphed-After

(c) Bonafide (d) Morphed

Figure 5: Variations in score distributions before and after conditional
disentanglement in the AMSL dataset indicating successful disentanglement in
morphed document images (Experiment 1).

Results from Experiment 1: Table 1 reports the performance of the proposed
method. We would like to re-iterate that the proposed method was not trained
to discriminate between non-morphed and morphed images, i.e., during training
the method does not require labels corresponding to morphed and non-morphed
image pairs. In spite of that, the proposed method achieves comparable results
to some extent with state-of-the-art methods that require explicit supervision
(see Table 2). Qualitatively, we present the results in Figure 4. In order to
demonstrate that the proposed method is actually disentangling subject
identities, we present the score distributions produced by the COTS
comparator, when comparing test images belonging to the AMSL dataset (see
Figure 5). In Figure 5(a), the distributions corresponding to morphed and
bonafide images completely overlap (BEFORE disentanglement). In Figure 5(b),
the distribution corresponding to morphed images move towards zero (AFTER
disentanglement). Here, the scores are similarity scores normalized to
$[0,1]$. To illustrate how disentanglement has a pronounced effect on morphed
images compared to bonafide images, we present the score distributions of the
bonafide images before and after disentanglement in Figure 5(c), and that of
the morphed images before and after disentanglement in Figure 5(d). The
bonafide scores are closely located (note the range of the scores in the
x-axis in Figure 5(c)). In contrast, if the document image is morphed, the
network tries to disentangle the identity traces belonging to the second
subject which causes a significant shift in the scores (see Figure 5(d)). We
further computed the interquartile range (IQR) as a ‘measure of dispersion’
between the bonafide and morphed score distributions. This was done for
bonafide and morphed scores both before and after disentanglement. Then we
computed the ratio of their IQRs and observed that the ratio for morphed
scores was $\sim$6 times higher than that of the bonafide scores.
$\bigg{(}\frac{IQR_{morphed-AFTER}}{IQR_{morphed-
BEFORE}}>5.9\times\frac{IQR_{bonafide-AFTER}}{IQR_{bonafide-BEFORE}}\bigg{)}$.

We observed from the experiments that the AMSL dataset required the least
number of epochs followed by MorGAN and EMorGAN. In the case of AMSL, the
BPCER improved from 6.1% to 3% upon increasing the number of training epochs
from 50 to 200. But the BPCER remained the same when tested on the MorGAN and
EMorGAN datasets.

Table 1: BPCER(%) @ APCER=10% (left of the forward slash) and APCER(%) @
BPCER=10% (right of the forward slash).

Train Test | AMSL | MorGAN |  EMorGAN
---|---|---|---
AMSL (50 epochs) | 6.1/5.2 | 4.6/2.0 | 4.4/0.8
MorGAN (100 epochs) | 63.6/60.3 | 8.6/5.8 | 9.4/9.3
| EMorGAN
---
(200 epochs) 25.8/47.7 | 28.1/38.4 | 29.7/39.4

Table 2: BPCER(%) @ APCER=10%. The best performing results for the proposed
method are compared with the baseline results indicated within parentheses
([24], [21]). The baseline results are taken from [24].

Train Test | AMSL | MorGAN
---|---|---
AMSL | 3.0 (2.2, 3.3) |  4.6 (18.5, 24.5)
MorGAN | 63.6 (8.8, 14.9) |  8.5 (8.5, 12.4)

Figure 6: The COTS face comparator scores corresponding to bonafide and
morphed test images belonging to the MorGAN dataset (Experiment 2). Figure 7:
Heatmap visualization of the “difference images” for morphed and non-morphed
inputs corresponding to two subjects. Note that the difference images contain
some discernible information that can be harnessed for recovering some unique
information about the second constituent in the morphed image (Experiment 3).

Results from Experiment 2: Table 2 compares the performance of the proposed
method with the two baselines [24, 21]. The proposed method achieves
BPCER=4.6% @ APCER=10%, which outperforms existing methods ([24]: BPCER=18.5%
@ APCER=10% and [21]: BPCER=24.5% @ APCER=10%) by a considerable margin when
trained on AMSL but tested on MorGAN dataset. However, training on the MorGAN
dataset results in poor performance when tested on AMSL dataset. We
investigated this issue further and observed an interesting phenomenon. We
computed the Pearson correlation coefficient between the document and
reference images in the AMSL test set, and when averaged across all the
images, it resulted in a mean and standard deviation of $0.91\pm 0.03$, while
for the MorGAN test set, the mean and standard deviation were $0.48\pm 0.24$,
which is almost half of that on the AMSL dataset. We also computed the
biometric utility of the MorGAN test set images using the COTS comparator and
observed that the biometric similarity scores are low (see Figure 6) for both
bonafide and morphed images. We suspect the the low degree of correlation
between the document and reference images, and the overall low biometric
matching utility of the images in the MorGAN dataset, may be responsible for
the lower performance of the proposed method in this case. Our method requires
adequately high degree of correlation between the document and reference
images for successful identity disentanglement.

Figure 8: Illustration of outputs generated using the proposed method for
decoding information about the “other” subject used in morph generation
(Experiment 3). Note in both cases the output is visually more similar to
Subject A (second subject) compared to Subject B (anchor subject present in
reference image).

Results from Experiment 3: In order to recover some information about the
second subject who contributed to the morphed image, we utilized the pixel-
wise difference images. However, we first needed to ascertain whether the
difference images contained any useful information at all. Therefore, we used
a similarity visualization technique [26] to visualize whether the difference
image retained any useful information. Figure 7 depicts the visualization in
terms of heatmaps (saliency maps) which shows that the difference images
contain discriminative information which can be harnessed for decoding the
second subject. Figure 8 illustrates examples of images generated using the
proposed method that qualitatively appear similar to the second subject
(Subject A) compared to the anchor subject (Subject B); the anchor subject
pertains to the identity in the reference image. Further, we also analyzed the
similarity between the generated outputs and the constituent subjects
quantitatively using a COTS face comparator. The biometric similarity between
the generated outputs and the second subject, as assessed using the true match
rate at a false match rate of 1%, was 23.4% higher than that between the
generated outputs and the first (anchor) subject. Therefore, we note that the
proposed method displays tremendous promise for decoding the second identity.

##  6 Summary and Future Work

In this paper, we proposed a novel differential face morph attack detection
framework that disentangles subject identities from a morphed document image
conditioned on the trusted reference image. In contrast to existing
classification-based approaches, the proposed method formulates the
differential MAD problem using an information theoretic framework. We used a
conditional generative network to produce an output image from the input
(bonafide or morphed) document image. Next, we compared the output image with
the reference image using a biometric comparator. The ensuing score is then
used for detecting morphs. The method demonstrates a promising first step
across different kinds of morph attacks without requiring any supervision
about the type of morphing. We achieved best performing results of 3% BPCER
@10% APCER on intra-dataset evaluation and 4.6% BPCER @ 10% APCER on cross-
dataset evaluation (Table 2). We observed that the proposed method needs well-
correlated document and reference image pairs acquired in controlled settings
for its robust operation (Section 5). This is a reasonable requirement given
that the success of face morphing is enhanced under these conditions.
Furthermore, we used the proposed method to recover some characteristics about
the second subject (different from the first subject whose reference image was
used). Future work will involve improving the task of de-morphing as well as
studying the applicability of the proposed technique to other modalities [23].

## References

  * [1] AMSL Face Morph Image Data Set.  https://omen.cs.uni-magdeburg.de/disclaimer/index.php.  [Online accessed: 15th February, 2021].
  * [2] Face Morpher.  http://www.facemorpher.com/.  [Online accessed: 1st July, 2021].
  * [3] image Manipulation Attack Resolving Solutions (iMARS).  https://cordis.europa.eu/project/id/883356.  [Online accessed: 2nd April, 2021].
  * [4] K. B. Raja et al.  Morphing Attack Detection - Database, Evaluation Platform and Benchmarking.  IEEE Transactions on Information Forensics and Security, 2020.
  * [5] P. E. Clemens Seibold, Anna Hilsmann.  Focused LRP: Explainable AI for Face Morphing Attack Detection.  IEEE Winter Conference on Applications of Computer Vision Workshops, pages 88–96, 2021.
  * [6] N. Damer, V. Boller, Y. Wainakh, F. Boutros, P. Terhörst, A. Braun, and A. Kuijper.  Detecting Face Morphing Attacks by Analyzing the Directed Distances of Facial Landmarks Shifts.  German Conference on Pattern Recognition, 2018.
  * [7] N. Damer, F. Boutros, A. M. Saladie, F. Kirchbuchner, and A. Kuijper.  Realistic Dreams: Cascaded Enhancement of GAN-generated Images with an Example in Face Morphing Attacks.  IEEE 10th International Conference on Biometrics Theory, Applications and Systems, pages 1–10, 2019.
  * [8] N. Damer, A. M. Saladie, A. Braun, and A. Kuijper.  MorGAN: Recognition Vulnerability and Attack Detectability of Face Morphing Attacks Created by Generative Adversarial Network.  IEEE 9th International Conference on Biometrics Theory, Applications and Systems, pages 1–10, 2018.
  * [9] M. Ferrara, A. Franco, and D. Maltoni.  The magic passport.  IEEE International Joint Conference on Biometrics, pages 1–7, 2014\.
  * [10] M. Ferrara, A. Franco, and D. Maltoni.  Face Demorphing.  IEEE Transactions on Information Forensics and Security, 13:1008–1017, 04 2018.
  * [11] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros.  Image-to-Image Translation with Conditional Adversarial Networks.  IEEE Conference on Computer Vision and Pattern Recognition, pages 5967–5976, 2017.
  * [12] A. Makrushin, T. Neubert, and J. Dittmann.  Automatic Generation and Detection of Visually Faultless Facial Morphs.  In VISIGRAPP, 2017.
  * [13] M. Mirza and S. Osindero.  Conditional Generative Adversarial Nets.  ArXiv, abs/1411.1784, 2014.
  * [14] M. Monroy.  Laws against morphing.  https://digit.site36.net/2020/01/10/laws-against-morphing/.  Appeared in Security Architectures and the Police Collaboration in the EU 10/01/2020 [Online accessed: 1st April, 2021].
  * [15] T. Neubert, A. Makrushin, M. Hildebrandt, C. Krätzer, and J. Dittmann.  Extended StirTrace benchmarking of biometric and forensic qualities of morphed face images.  IET Biometrics, 7:325–332, 2018.
  * [16] M. Ngan, P. Grother, K. Hanaoka, and J. Kuo.  Face Recognition Vendor Test (FRVT) Part 4: MORPH - Performance of Automated Face Morph Detection.  NISTIR 8292 Draft Supplement, April 16, 2021.
  * [17] A. A. Othman and A. Ross.  Privacy of Facial Soft Biometrics: Suppressing Gender But Retaining Identity.  In European Conference on Computer Vision Workshops, 2014.
  * [18] F. Peng, L. B. Zhang, and M. Long.  FD-GAN: Face De-Morphing Generative Adversarial Network for Restoring Accomplice’s Facial Image.  IEEE Access, 7:75122–75131, 2019.
  * [19] U. Scherhag, D. Budhrani, M. Gomez-Barrero, and C. Busch.  Detecting Morphed Face Images Using Facial Landmarks.  In International Conference on Image and Signal Processing, 2018\.
  * [20] U. Scherhag, C. Rathgeb, and C. Busch.  Towards Detection of Morphed Face Images in Electronic Travel Documents.  In IAPR 13th International Workshop on Document Analysis Systems, pages 187–192, 2018.
  * [21] U. Scherhag, C. Rathgeb, J. Merkle, and C. Busch.  Deep Face Representations for Differential Morphing Attack Detection.  IEEE Transactions on Information Forensics and Security, 15:3625–3639, 2020.
  * [22] C. Seibold, W. Samek, A. Hilsmann, and P. Eisert.  Detection of face morphing attacks by deep learning.  In C. Kraetzer, Y.-Q. Shi, J. Dittmann, and H. J. Kim, editors, Digital Forensics and Watermarking, pages 107–120, 2017.
  * [23] R. Sharma and A. Ross.  Image-level Iris Morph Attack.  IEEE 28th International Conference on Image Processing, 2021.
  * [24] S. Soleymani, A. Dabouei, F. Taherkhani, J. Dawson, and N. M. Nasrabadi.  Mutual Information Maximization on Disentangled Representations for Differential Morph Detection.  IEEE Winter Conference on Applications of Computer Vision, pages 1731–1741, 2021.
  * [25] M. Steyvers.  Morphing techniques for manipulating face images.  Behavior Research Methods, Instruments, & Computersc, 31:359–369, 06 1999.
  * [26] A. Stylianou, R. Souvenir, and R. Pless.  Visualizing Deep Similarity Networks.  In IEEE Winter Conference on Applications of Computer Vision, pages 2029–2037, 2019.
  * [27] S. Venkatesh, R. Ramachandra, K. Raja, and C. Busch.  Face Morphing Attack Generation & Detection: A Comprehensive Survey.  IEEE Transactions on Technology and Society, 2021.
  * [28] H. Zhang, S. Venkatesh, R. Ramachandra, K. Raja, N. Damer, and C. Busch.  MIPGAN - Generating Robust and High Quality Morph Attacks Using Identity Prior Driven GAN.  IEEE Transactions on Biometrics, 2021.


# Worst-Case Morphs: a Theoretical and a Practical Approach

1st Una M. Kelly  Data Management and Biometrics
University of Twente
Enschede, The Netherlands
<EMAIL_ADDRESS>2nd Luuk Spreeuwers  Data Management and Biometrics
University of Twente
Enschede, The Netherlands
<EMAIL_ADDRESS>3rd Raymond  Data Management and Biometrics
University of Twente
Enschede, The Netherlands
<EMAIL_ADDRESS>
###### Abstract

Face Recognition (FR) systems have been shown to be vulnerable to morphing
attacks. We examine exactly how challenging morphs can become. By showing a
worst-case construction in the embedding space of an FR system and using a
mapping from embedding space back to image space we generate images that show
that this theoretical upper bound can be approximated if the FR system is
known. The resulting morphs can also succesfully fool unseen FR systems and
are useful for exploring and understanding the weaknesses of FR systems. Our
method contributes to gaining more insight into the vulnerability of FR
systems.

###### Index Terms:

Biometrics, Morphing Attack Detection, Face Recognition, Vulnerability of
Biometric Systems

##  I Introduction

Several publications have shown that Face Recognition (FR) systems and humans
are vulnerable to morphing attacks. A morph is an image that contains
sufficient identity information from the faces of two different individuals
for it to be accepted as a match when it is compared to an image of either
contributing identity. This can lead to security risks in e.g. border control,
since a criminal could travel using the identity document of an accomplice.
Several methods to address such attacks have been proposed. In most cases, the
morphs on which Morphing Attack Detection (MAD) methods are trained and tested
are generated in-house, and are usually created by detecting corresponding
landmarks in the faces, warping the images to an average geometry and then
blending the pixel values [5, 11, 7].

The fact that MAD methods are often tested on datasets with similar
characteristics as the sets used for training them may lead to overfitting to
certain dataset-specific characteristics - especially if only one morphing
algorithm was used [14, 16]. In many countries someone applying for an
identity document can provide their own printed passport photo, or even upload
a photo digitally (e.g. Ireland, Estonia). This means that before a passport
photo is stored in an electronic Machine Readable Travel Document (eMRTD), it
may have been printed and scanned or digitally manipulated.

There are several other factors that affect how morphed images differ, leading
to variation in the performance of MAD methods and vulnerability of FR
systems. These include the morphing algorithm used, which landmark detector
was used, or whether landmarks were selected manually, printing and scanning,
the pairs of images selected for morphing, etc. There may be other tools for
generating morphs of which the research community is not yet aware. Therefore,
it is necessary to explore other potential methods and better understand the
weaknesses of FR systems. Benchmarks for validating MAD methods - such as [10,
8, 1] \- can be extended and improved by evaluating morphs generated with
different methods. We contribute to this active field of research by providing
a new morphing method, and by showing that there is an upper limit on how
difficult morphs can become.

We investigate exactly how challenging morphs can in theory and in practice
become for FR systems. Given an FR system and two images of two different
people, the worst-case morph is the image that is most similar to both images,
according to this FR system. We train a neural network to generate images that
approximate such worst-case morphs and examine how well our approximations of
worst-case morphs can fool an FR system, and whether this extends to other
systems. Our results show it is possible to approximate the worst case when
the (weights of the) FR system are known, which means that morphing attacks
can become even more challenging than landmark- or GAN-based morphs. In black
box scenarios, when the network weights are not accessible, our approximations
of the worst case are less challenging, but still pose a significant threat.

Our proposed system differs from GAN-based morphs [19, 17] since GANs are
trained to generate images from an embedding space (different from the
embedding space of an FR system) that look like real images, while we directly
work with the embedding space of an FR system and its associated
(dis)similarity measure to approximate the most challenging morph, and we
train without the adversarial constraint of GANs.

Our main contributions are: a theoretical framework that when given an FR
system and two images can be used to define a worst-case embedding, a
practical demonstration of finding images that (approximately) correspond to
this worst-case embedding, showing that these can be significantly more
challenging than landmark- or GAN-based morphs, examining the vulnerability of
other FR systems, and providing a new method for generating morphs that leads
to more variation in morph datasets, which may help researchers more
accurately validate MAD methods in future work and uncover vulnerabilities of
FR systems.

##  II Background

Research on variation in morphing algorithms includes: post-processesing
landmark-based morphs to mask effects caused by the morphing process [15], a
model to simulate the effects of printing and scanning [6] and considering the
influence of ageing on morphing attacks [18]. GANs were used in [17, 19, 4] in
an attempt to create a different type of morph, which was shown to be able to
fool FR systems, if not as consistently as landmark-based morphs. Since GANs
are difficult to train more stable methods would be useful [3]. The lack of
variation in morphing techniques is adressed in [13]. A method for MAD is
presented and evaluated on morphs created using different algorithms, which
are all landmark-based. Printed-and-scanned morphs are included, but GAN
morphs or other methods are not taken into consideration.

$x_{1}$$x_{2}$$f$$\bm{z}_{1}$$\bm{z}_{2}$$f_{Enc}$$(\bm{z}^{*},\bm{z}_{Enc})$$g$$x_{\text{morph}}$$x^{\prime}_{1}$$x^{\prime}_{2}$$\checkmark$$\checkmark$$\mathbin{\leavevmode\hbox
to9.92pt{\vbox to9.92pt{\pgfpicture\makeatletter\hbox{\hskip
0.62pt\lower-0.62pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.354,0.354,0.354}\pgfsys@color@gray@stroke{0.354}\pgfsys@invoke{
}\pgfsys@color@gray@fill{0.354}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{}{}\pgfsys@setlinewidth{1.24pt}\pgfsys@invoke{
}\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}\pgfsys@color@rgb@stroke{1}{0}{0}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{1}{0}{0}\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,0,0}{}{{}}{} {}{}{}{{}}{}
{}{}{}\pgfsys@moveto{0.0pt}{0.0pt}\pgfsys@lineto{8.68pt}{8.68pt}\pgfsys@moveto{0.0pt}{8.68pt}\pgfsys@lineto{8.68pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}}$ Figure 1: Overview
of our morphing approach. Here, $x_{1},x^{\prime}_{1}$ are images of one
person and $x_{2},x^{\prime}_{2}$ are images of a different person. While
$x_{\text{morph}}$ visually looks like $x_{1}$, according to the FR system it
is a succesful match with both $x^{\prime}_{1}$ and $x^{\prime}_{2}$. Dashed
lines represent comparisons by the FR system.

##  III Proposed System

Traditionally, landmark-based morphing attempts to create images in image
space that are similar to two given images. Instead, we use the embedding
space of an FR system, since this should contain structured information on the
similarity of the identity information present in images. Let $f$ be the
function that describes an FR system’s mapping from the image space $X$ to the
embedding space $Z$, i.e. $f:X\rightarrow Z$ (Fig. 1). Let $d$ be the
dissimilarity score that calculated the dissimilarity of embedding pairs in
$Z$. We define the worst-case embedding for two images $x_{1}$ and $x_{2}$ as

$z^{*}:=\text{argmin}_{z\in
Z}\left(\max\left[d(z,f(x_{1})),d(z,f(x_{2}))\right]\right).$ (1)

For example, if $S$ returns cosine similarity, then
$S(z_{1},z_{2})=\cos(\theta)$, where $\theta$ is the angle between $z_{1}$ and
$z_{2}$, see Fig. 2. In that case $z^{*}$ is any $z$ for which
$S(z_{1},z)=S(z,z_{2})=\cos(\theta/2)$. If an FR system uses similarity
scores, defined by a function $S$, then $S$ replaces $d$, argmin is replaced
by argmax, and max is replaced by min.

$z_{1}$$z_{2}$$z^{*}=\frac{z_{1}+z_{2}}{2}$$z_{\text{lm}}$$z_{\text{GAN}}$$z_{1}$$z_{2}$$z^{*}=\frac{z_{1}+z_{2}}{||z_{1}+z_{2}||}$$\theta$$z_{\text{lm}}$$z_{\text{GAN}}$
Figure 2: The worst-case embedding $z^{*}$ when $d$ denotes euclidean distance
(left) and angle (right). An image that maps to $z^{*}$ is even more
challenging than a landmark- ($z_{\text{lm}}$) or GAN-based morph
($z_{\text{GAN}}$).

A worst-case morph is an image $x^{*}$ for which $f(x^{*})=z^{*}$. We
approximate $x^{*}$ using a decoder $D$ that maps from $Z$ to $X$, reversing
the mapping of the FR system. Since $f$ is a many-to-one mapping there may be
many different approximative inverse mappings for one given embedding. We
choose to train $D$ to reconstruct $x_{1}$ when given $z^{*}$ as input, i.e.
$D$ learns to approximate the mapping $g^{*}:Z\rightarrow X$, where
$g^{*}(z^{*})=x_{1}$ and $f(g^{*}(z^{*}))=z^{*}$. $D$ generates images that to
humans look like $x_{1}$, i.e. like one of the input identities (e.g. the
accomplice), but to the FR system are a morph of $x_{1}$ and $x_{2}$.
According to the FR system the resulting image contains identity features of
both accomplice and criminal.

Instead of training a decoder to generate morphed images, we could also use
optimisation on an FR system. In that case, the FR latent embeddings
corresponding to an image are manipulated by calculating gradients and
computing adversarial perturbations. The disadvantage of learning (i.e.
training a decoder or other network) over optimisation (i.e. gradient descent)
is that the decoder is trained to generate many different images, while
optimisation applies an adversarial perturbation tailored specifically to one
image. The advantage of learning over optimisation is that there is no need
for retraining or calculating image-specific gradients every time a new image
is manipulated (e.g. from a new dataset).

$x_{1}$LandmarkGANworst case$x_{2}$$x_{1}$LandmarkGANworst case$x_{2}$ Figure
3: Landmark-based morphs (spliced into $x_{1}$, full morph, spliced into
$x_{2}$), GAN-based morph and worst-case approximations (with $z_{\text{Enc}}$
from $x_{1}$ and with $z_{\text{Enc}}$ from $x_{2}$).

###  III-A Morph generation

For our experiments we use MobileFaceNet, which on LFW achieves an accuracy of
99.55% [2]. For images $x_{1}$ and $x_{2}$, this FR system returns
128-dimensional embeddings $z_{1}=f(x_{1})$ and $z_{2}=f(x_{2})$, where $f$
describes the mapping defined by the neural network from the image space to
the embedding space:

$\displaystyle X=[0,1]^{n_{c}\times w\times h}\quad\text{ and }\quad
Z=\left\\{z\in\mathbb{R}^{128}\ \middle|\ ||z||=1\right\\},$

where $n_{c}=3$ and $w\times h=112\times 112$. It can be used as a face
verification system by calculating the angle $\theta$ between $z_{1}$ and
$z_{2}$. $x_{1}$ and $x_{2}$ are not accepted as a match if $\theta>t$, where
$t$ is the decision threshold. In this case, the worst-case embedding $z^{*}$
that minimises Eq. 1 is the embedding that lies on the unit hypersphere
exactly between $z_{1}$ and $z_{2}$, i.e. a worst-case morph is any image
$x^{*}$ for which $f(M^{*})=z^{*}=\frac{z_{1}+z_{2}}{||z_{1}+z_{2}||}$.
Scaling is necessary to ensure that $z^{*}$ lies on the unit hypersphere.

An embedding $z=f(x)$ should contain information essential to the identity of
$x$, but some information such as background or expression is not relevant to
identity and should therefore not be contained in $z$. This means that the FR
latent embedding alone is not sufficient to reconstruct $x$, so we train a
supporting encoder network that also compresses image information, including
other attributes. We train the decoder network using both worst-case
embeddings of the FR system and embeddings of the encoder as input.

Let $f_{\text{Enc}}$ be the mapping of the supporting encoder, i.e.
$f_{\text{Enc}}(x)=z_{\text{Enc}}$. Furthermore, let $g$ denote the decoder
mapping back to image space. Let $d$ be the dissimilarity score function used
by the FR system to compare latent embeddings, which here is the angle between
embeddings. We train $D$ to approximate $g^{*}$ by minimising the following
loss

$\displaystyle\mathcal{L}$
$\displaystyle=\gamma_{1}\mathcal{L}_{\text{pixel}}+\gamma_{2}\mathcal{L}_{\text{latent}}$
$\displaystyle=\gamma_{1}\mathbb{E}_{x}\left[||x_{\text{morph}}-x_{1}||^{2}_{2}\right]+\gamma_{2}\mathbb{E}_{x}\left[d(z_{\text{morph}},z^{*})\right],$
(2)

where $\mathcal{L}_{\text{pixel}}$ ensures that the output images are visually
similar to the input images, and $\mathcal{L}_{\text{latent}}$ encourages the
corresponding FR embeddings to be close to the worst case. The architecture
and hyperparameters of this network can be found in the appendix. We train the
network for 200 epochs, after which the losses no longer significantly
decrease.

$\theta_{D},\theta_{\text{Enc}}\leftarrow\text{initialize network parameters}$

repeat

$\bm{x}^{(1)},\ldots,\bm{x}^{(N)}$ $\triangleright$ Draw $N$ samples

$\bm{z}^{(i)}=f(\bm{x}^{(i)}),\ i=1,..,N$ $\triangleright$ Get FR embeddings

$\bm{z}^{*(i)}=\frac{\bm{z}^{(i)}+\bm{z}^{(j)}}{||\bm{z}^{(i)}+\bm{z}^{(j)}||},\
j=2,..,N,1$ $\triangleright$ Translate batch to get

pairs for worst-case emb.

$\bm{z}^{(i)}_{\text{Enc}}=f_{\text{Enc}}(\bm{x}^{(i)}),\quad i=1,..,N$
$\triangleright$ Get Encoder emb.

$\bm{x}_{\text{morph}}^{(i)}=D(\bm{z}^{(i)},\bm{z}_{\text{Enc}}^{(i)}),\
i=1,..,N$ $\triangleright$ Generate morphs

$\bm{z}_{\text{morph}}^{(i)}=f(\bm{x}_{\text{morph}}^{(i)}),\quad i=1,..,N$
$\triangleright$ Get FR emb. of morphs.

$\mathcal{L}=\gamma_{1}\frac{1}{N}\sum_{i=0}^{N}\text{MSE}(\bm{x}^{(i)},\bm{x}_{\text{morph}}^{(i)})$

$\qquad\qquad+\
\gamma_{2}\frac{1}{N}\sum_{i=0}^{N}d(\bm{z}^{*(i)},\bm{z}^{(i)}_{\text{morph}})$
$\triangleright$ Compute loss

$\theta_{D}\leftarrow\theta_{D}-\nabla_{\theta_{D}}\mathcal{L}$
$\triangleright$ Gradient update on decoder

$\theta_{\text{Enc}}\leftarrow\theta_{\text{Enc}}-\nabla_{\theta_{\text{Enc}}}\mathcal{L}$
$\triangleright$ Gradient update on encoder

until convergence

Algorithm 1 Training procedure. $\gamma_{1}=1,\gamma_{2}=0.1$

We use a dataset of in total 21,772 facial images from [9] and separate them
into 18,143 training and 3,629 validation images, with no overlap in
identities. We create three sets of morphs using the validation set:
approximations of worst-case morphs, landmark-based morphs and GAN-based
morphs using MIPGAN-I [19]. We select pairs of similar identities and faces
with neutral expression, resulting in 506 pairs for morphing. For each pair
$(x_{1},x_{2})$ we create two worst-case approximations, three landmark morphs
(one full morph and two spliced morphs), and one GAN morph, see Fig. 3.

We improve each morph generated with the trained Decoder by freezing the
network’s weights and performing gradient descent on
$\mathcal{L}_{\text{latent}}$ (Eq. 4) for $N$ iterations to improve input
selection, where $z_{1}^{*}=z^{*}=\frac{z_{1}+z_{2}}{||z_{1}+z_{2}||}$ and
$z_{i}^{*}=z_{i-1}^{*}-\nabla\mathcal{L}_{\text{latent}}(z_{\text{morph}})$.
The improved morph is obtained by forwarding $(z_{N}^{*},z_{\text{Enc}})$
through the decoder. We set $N=200$, after which the white box FR scores still
decrease, but not the scores for black box FR systems.

Figure 4: Vulnerability of MobileFaceNet to: landmark morphs, MIPGAN-I morphs, worst-case embeddings (theoretical upper bound), approximations of worst-case morphs, and improved approximations after optimisation (200 iterations per image). Blue histograms describe genuine comparison scores, impostor comparison scores are red. For each morph we compute the angles for probe images of each contributing identity. The green histograms describe the largest of these two angles. Our morphs are extremely difficult to distinguish from genuines using FR comparison scores. TABLE I: MMPMR (%). | Landmark | GAN | Worst- | Worst- | Improved
---|---|---|---|---|---
|  |  | case | case | worst-case
|  |  |  | approx. | approx.
MobileFaceNet | 97.3 | 96.3 | 100.0 | 100.0 | 100.0
VGG16 (black box) | 63.0 | 65.0 | 100.0 | 33.6 | 43.1
ArcFace (black box) | 85.4 | 66.3 | 100.0 | 47.9 | 61.1

##  IV Results & Discussion

We compare the vulnerability of FR systems to landmark-based, GAN-based
morphing attacks and worst-case approximations. We examine the morphs with the
same deep-learning-based FR system used for training the inverse network and
with two other FR systems, where we calculate the Mated Morph Presentation
Match Rate (MMPMR($t$)), which is the proportion of (morphing) attacks for
which both contributing identities are considered a match by the FR system
when using a threshold $t$ [12]. For each FR system we set $t$ such that the
false non-match rate is minimal while the false match rate$<$0.1%.

The histograms of MobileFaceNet comparison scores are shown in Fig. 4. The
MMPMR for the worst-case approximations is 100%, even higher than the 97.3%
for landmark-based morphs (Tab. I). This means that every worst-case
approximation contains enough identity information of both contributing
identities to fool the FR system.

In Fig. 4 we can see that the worst-case approximations are very close to the
actual worst case, especially after optimising each morph separately. Table 1
shows that this does not necessarily hold for other FR systems. This is not
entirely surprising, since our method is trained to generate approximations of
worst-case morphs specifically for one FR system, while the worst-case morphs
for a different FR system might be different. Nonetheless, a significant
proportion of the morphs is still accepted as a match with both contributing
identities. Most importantly, we have shown that it is possible to generate
images that are close to the worst case, and therefore believe that
researchers should not focus exclusively on landmark- and GAN-based morphs,
since more challenging morphs may exist.

##  V Conclusion & Future Work

We introduced the concept of worst-case morphs, which are morphs that are most
similar to the contributing identities according to an FR system. We trained a
decoder network to generate images that approximate such worst-case morphs.
These were extremely successful at fooling the FR system that was used for
training and significantly more challenging than landmark- or GAN-based
morphs. We also showed that generating morphs that are close to the worst case
for FR systems under black box settings is still an open problem. This might
be improved by minimising the distance to the worst-case embedding for several
FR systems simultaneously. Since the quality of morphs depends on the image
quality of genuine images, pair selection, and other factors, a good measure
to compare the quality of different morphs could be measuring the distance to
the worst-case embedding.

The morphs we generate may be helpful in understanding the vulnerabilities of
FR systems and may also offer some insight into the robustness of MAD
techniques. Since the morphs generated using the decoder network are closer to
the worst case than landmark- or GAN-based morphs, even before applying
optimisation to improve each morph individually, challenging morphs can be
generated in a computationally inexpensive manner. Furthermore, our approach
has the advantage of being more stable to train than e.g. GANs. We also
studied the potential of our method for use in black box settings.

## References

  * [1] BOEP.  Bologna Online Evaluation Platform (BOEP) - Morph Attack Detection Evaluation, https://biolab.csr.unibo.it/fvcongoing/UI/Form/BOEP.aspx, accessed: 05-10-2021.
  * [2] S. Chen, Y. Liu, X. Gao, and Z. Han.  MobileFaceNets: Efficient CNNs for accurate real-time face verification on mobile devices.  CoRR, abs/1804.07573, 2018.
  * [3] A. Creswell et al.  Generative adversarial networks: An overview.  IEEE Signal Processing Magazine, 35(1):53–65, 2018.
  * [4] N. Damer et al.  Regenmorph: Visibly realistic gan generated face morphing attacks by attack re-generation, 2021.
  * [5] M. Ferrara, A. Franco, and D. Maltoni.  The magic passport.  In IEEE International Joint Conference on Biometrics, pages 1–7, 2014.
  * [6] M. Ferrara, A. Franco, and D. Maltoni.  Face morphing detection in the presence of printing/scanning and heterogeneous image sources.  IET Biometrics, 10(3):290–303, 2021.
  * [7] A. Makrushin and A. Wolf.  An overview of recent advances in assessing and mitigating the face morphing attack.  In 2018 26th European Signal Processing Conference (EUSIPCO), pages 1017–1021, 2018.
  * [8] NIST.  NIST FRVT MORPH, https://pages.nist.gov/frvt/html /frvt_morph.html, accessed: 05-10-2021.
  * [9] P. J. Phillips et al.  Overview of the face recognition grand challenge.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 947–954 vol. 1, 2005.
  * [10] K. Raja et al.  Morphing attack detection – database, evaluation platform and benchmarking.  arXiv, 2020.
  * [11] D. Robertson, R. Kramer, and A. Burton.  Fraudulent id using face morphs: Experiments on human and automatic recognition.  PLOS ONE, 12:e0173319, 03 2017.
  * [12] U. Scherhag et al.  Biometric systems under morphing attacks: Assessment of morphing techniques and vulnerability reporting.  In 2017 International Conference of the Biometrics Special Interest Group (BIOSIG), pages 1–7, 2017.
  * [13] U. Scherhag, J. Kunze, C. Rathgeb, and C. Busch.  Face morph detection for unknown morphing algorithms and image sources: a multi-scale block local binary pattern fusion approach.  IET Biometrics, 9(6):278–289, 2020.
  * [14] U. Scherhag, C. Rathgeb, and C. Busch.  Performance variation of morphed face image detection algorithms across different datasets.  In 2018 International Workshop on Biometrics and Forensics (IWBF), pages 1–6, 2018.
  * [15] C. Seibold, A. Hilsmann, and P. Eisert.  Style your face morph and improve your face morphing attack detector.  In 2019 International Conference of the Biometrics Special Interest Group (BIOSIG), pages 1–6, 2019.
  * [16] L. Spreeuwers, M. Schils, and R. Veldhuis.  Towards robust evaluation of face morphing detection.  In 2018 26th European Signal Processing Conference (EUSIPCO), pages 1027–1031, 2018.
  * [17] S. Venkatesh et al.  Can gan generated morphs threaten face recognition systems equally as landmark based morphs? - vulnerability and detection.  In 2020 8th International Workshop on Biometrics and Forensics (IWBF), pages 1–6, 2020.
  * [18] S. Venkatesh, K. Raja, R. Ramachandra, and C. Busch.  On the influence of ageing on face morph attacks: Vulnerability and detection.  In 2020 IEEE International Joint Conference on Biometrics (IJCB), pages 1–10, 2020.
  * [19] H. Zhang, S. Venkatesh, R. Ramachandra, K. Raja, N. Damer, and C. Busch.  MIPGAN - generating strong and high quality morphing attacks using identity prior driven gan.  IEEE Transactions on Biometrics, Behavior, and Identity Science, 3(3):365–383, 2021.

## Hyperparameters

Operation | Kernel | Stride | Size
---|---|---|---
$f_{\text{Enc}}(\bm{x})$ |  |  |  $224\times 224\times 3$ (Input)
5$\times$ Convolution | $5\times 5$ | $2\times 2$ | $7\times 7\times 64$
$g(\bm{z})$ |  |  |  $1\times 1\times 128$ (Input)
Fully connected | - | - | $1\times 1\times 512$
Fully connected | - | - | $7\times 7\times 448$
Concat. with $f_{\text{Enc}}(\bm{x})$ |  |  | $7\times 7\times 512$
4$\times$ Upsample & Conv. | $3\times 3$ | $1\times 1$ | $112\times 112\times 32$
Conv. | $3\times 3$ | $1\times 1$ | $112\times 112\times 32$
Upsample & Conv. | $3\times 3$ | $1\times 1$ | $224\times 224\times 3$
Optimizer |  Adam($\alpha=10^{-4}$, $\beta_{1}=0.0$, $\beta_{2}=0.9)$
Batch size | 64
Bias |  False, except in the last layer of the decoder, where it is untied.
BatchNorm |  After each convolution.
Weight init. |  Isotropic gaussian ($\mu=0$, $\sigma=0.01$)
Untied bias init. |  $\text{Sigmoid}^{-1}(\frac{1}{n}\sum_{i=1}^{n}\bm{x}_{i}$), the average of $\\{\bm{x}_{1},\dots,\bm{x}_{n}\\}$.
Nonlinearity |  Leaky ReLU of slope 0.02, except for the last convolution in $g$, which is followed by a sigmoid activation function.