figure table

# Analyzing Human Observer Ability in Morphing Attack Detection - Where Do We
Stand?

Sankini Rancha Godage‡, Frøy Løvåsdal†, Sushma Venkatesh‡,
Kiran Raja‡, Raghavendra Ramachandra‡, Christoph Busch‡
‡Norwegian University of Science and Technology (NTNU), Norway
†The Norwegian Police Directorate, Norway

###### Abstract

Morphing attacks are based on the technique of digitally fusing two (or more)
faces into one, with the final visage resembling the contributing faces.
Morphed images not only pose a challenge to Face-Recognition Systems (FRS) but
also challenge the experienced human observers due to high quality,
postprocessing to eliminate any visible artifacts, and further, the printing
and scanning process. Few studies have concentrated on examining how people
can recognize morphing attacks, even as several publications have examined the
susceptibility of automated FRS and offered morphing attack detection (MAD)
approaches. MAD approaches base their decisions either on a single image with
no reference to compare against (Single-Image MAD (S-MAD)) or using a
reference image (Differential MAD (D-MAD)). One prevalent misconception is
that an examiner’s or observer’s capacity for facial morph detection depends
on their subject expertise, experience, and familiarity with the issue and
that no works have reported the specific results of observers who regularly
verify identity (ID) documents for their jobs. As human observers are involved
in checking the ID documents having facial images, a lapse in their competence
can have significant societal challenges. To assess the observers’
proficiency, this research first builds a new benchmark database of realistic
morphing attacks from 48 different subjects, resulting in 400 morphed images.
Unlike the previous works, we also capture images from Automated Border
Control (ABC) gates to mimic the realistic border-crossing scenarios in the
D-MAD setting with 400 probe images, to study the ability of human observers
to detect morphed images. A new dataset of 180 morphing images is also
produced to research human capacity in the S-MAD environment. In addition to
creating a new evaluation platform to conduct S-MAD and D-MAD analysis, the
study employs 469 observers for D-MAD and 410 observers for S-MAD who are
primarily governmental employees from more than 40 countries, along with 103
subjects who are not examiners. The analysis offers intriguing insights and
highlights the lack of expertise and failure to recognize a sizable number of
morphing attacks by experienced observers. Human observers tend to detect
morphed images to a lower accuracy as compared to the four automated MAD
algorithms evaluated in this work. The results of this study are intended to
aid in the development of training programs that will prevent security
failures while determining whether an image is bona fide or altered.

###### Index Terms:

Biometrics, Morphing Attack Detection, Face Recognition, Human expert, ID
Examination, Facial Comparison

##  I Introduction

Figure 1: Pipeline of Single-Image Morphing Attack Detection (S-MAD) and
Differential MAD (D-MAD)

Face recognition systems (FRS) are becoming increasingly common in identity
management systems worldwide.  A face image is mandated in all passports
according to the guidelines of the International Civil Aviation Organization
(ICAO),  and most European nations also have national ID cards  with facial
image as  their main form of identification. Most border controls globally
already use automated facial recognition as the primary modality for verifying
the traveler’s identity. While the practice of using live enrolment is
encouraged when issuing travel and/or identity documents, many countries still
allow the applicants to upload or submit a passport photo. Such a practice is
vulnerable with respect to morphed or otherwise manipulated facial images.

A morphed image results from digitally combining two or more face images of
two or more independent subjects, which usually resembles all the contributing
subjects.  Each contributing subject to a morph can have a separate morphing
factor selected by a malevolent actor.  A high-quality morph can not only
evade automated FRS but also  trick a human observer (for  example, a trained
border  officer) in both the passport application process and automatic border
control , which exacerbates the situation. While a novice attacker may create
a morphed image with ghost  artifacts, an expert attacker can create morphed
images  carefully and then process them further to eliminate all the spurious
traces (morphing related  artifacts in images) in the image to enhance the
appearance.  A higher-quality morph image produced by such a method typically
makes it difficult for a human observer to  identify the morphs [1].

While many  nations only accept digital photos, some also accept printed
photos that are then redigitized by the passport application officer , giving
the photo the chance to be altered before printing. Such a process eliminates
most subtle cues present in the morphed image in  the digital domain, making
it challenging to detect the morphs. Additionally,  finding morphs in morphed
photographs can be more difficult for human observers than detecting impostors
in  typical facial images. Addressing such concerns of operational FRS,
several works have started proposing various automated approaches for
detecting morphing attacks [2].  To recognize the morphed images, these
morphing attack detection (MAD)  techniques often train a classifier on hand-
crafted or deep features [2, 3]. While one spectrum of solutions focuses on
investigating machine-driven MAD, another set of works  has focused on
analyzing the human observers ’ ability to detect morphing attacks.  We
concentrate on a little-researched element of evaluating human observers’
capacity to detect the morphs  to complement the rapid development of machine-
driven MAD solutions. Unlike the previous studies [4, 1, 5], we intend to
study human ability on high-quality morphed images  images that are carefully
processed in a controlled manner to eliminate  artifacts. We firmly believe
that by comprehending how observers perceive information during face and/or
document examination , we can better develop automated face morph detection
software and  create the training materials required to enhance people’s
capacity for face morph detection.

###  I-A Societal and Technological Aspects

Although MAD algorithms offer a practical option to detect altered photos,
most ID control systems  like border control,  still involve a human expert to
make a  judgment. A wrong decision by  a human can put the security of a
country and society at risk in the worst case.  To prevent ethical issues, a
portion of the population should not always be treated as genuine or as
malicious actors. On the other hand,  each choice made by an algorithm should
be  reviewed by a human. The clear interplay between the technology and human
decision making aspect needs to be understood to avoid  the justifiable use of
technology.  We are motivated to understand and analyze the  skills of trained
ID control experts working in various agencies as a first step in this
direction. This work is expected to serve as a  foundation for creating better
training programs and policies for ID control experts to avoid  the negative
perception of MAD technologies  by understanding the strengths and
limitations.

###  I-B Contributions

In light of this interaction between society and technology, we sought to
determine if ID control specialists (border officers, case handlers) (for
Passport, visas, ID, etc.), document examiners - 1st line, document examiner-
2nd line, expert document examiners- 3rd line and face comparison experts
(Manual examination), 3rd line) have the ability to detect morphs reliably.
The details of the experts recruited and their typical employment areas are
provided in Table I. The study is intended for the quantitative examination of
findings from psycho-visual experiments of these expert observer groups.

We pose five critical questions to help analyze the findings as listed below:

  * •

How good are ID document examiners at detecting morphing attacks?

  * •

Are human observers with certain types of training better than others at
detecting morphing attacks?

  * •

Does long experience working in a certain field (for instance, facial
examination in ID control) positively impact MAD detection performance?

  * •

Are expert observers with training or experience in checking identity/identity
documents better than those without training?

  * •

How do machine-based MAD algorithms perform compared to human observers in
D-MAD and S-MAD?

TABLE I: Expertise domain of human observers recruited for the experiments in
this work.

Line of work | Expertise and tasks
---|---
Border guard 1st line | Check and ascertain whether travelers are eligible to cross the border
Perform basic document examination, basic facial comparison, and basic
interview
Carry out checks against relevant databases and interpret the results
Border guard 2nd line | Support 1st line in their tasks
Handle more complex or time-consuming cases
Perform more advanced document examination, facial comparison, interview
Decide on whether the traveler may be allowed to cross the border
| Receive, process and consider applications for passports, ID cards, visas,
| residence permits, national personal ID number, etc.
| Provide information about the application procedure according
| to the legal framework, policy, etc.
Case handler - passport, | Receive and register applications
visas, ID, etc. - 1st line | Perform basic document examination, basic facial comparison, and basic interview
| Carry out checks against relevant databases and interpret the results
| Decide on the application / refer the application to a 2nd line officer
| Support 1st line in their tasks
Case handler - passport, | Handle more complex or time-consuming cases
visas, ID, etc. - 2nd line | Perform more advanced checks of documents and person
| Decide on applications
| Support 1st and 2nd line in checks of document and person in various
| types of cases. e.g. passports, visas, border crossings.
| Perform advanced document examination
| Perform advanced facial comparison
ID Expert | Perform in-depth interviews
(Embassies, Police | Decide on applications
Districts, ID Offices, etc.) | Produce and provide training to 1st and 2nd lines in the document
| examination, facial comparison, impostor detection, etc.
| Give evidence in court
| Support 1st line in comparing documents
Document examiner | Perform advanced document examinations
2nd line | Produce and provide training
| Give evidence in court
| Support 1st and 2nd line in comparing faces
Document examiner | Perform specialized face examinations/comparisons
3rd line (Central unit) | Produce and provide training
| Give evidence in court
| Support 1st and 2nd line in comparing faces
Face examiner | Perform specialized face examinations/comparisons
(Central unit) | Produce and provide training
| Give evidence in court
| Support 1st and 2nd line in comparing fingerprints
Fingerprint examiner | Perform specialized fingerprint examinations/comparisons
3rd line (Central unit) | Produce and provide training
| Give evidence in court

We develop a new database of morphed  images called the Human Observer
Analysis Morphing Image Database  as our initial contribution (HOMID). The
newly created database consists of images from  48 unique bona fide subjects
and correspondingly  400 morphed images created by a combination of two
resembling subjects (in terms of age, gender, ethnicity , and skin color) with
a human expert’s intervention. Two specialists (researchers with extensive
morphing experience) further postprocessed the morphed images to remove any
artifacts and provided the curated set of realistic,  high-quality morphed
images. As an additional measure for realistic evaluation, the newly created
database is supplemented with corresponding images after printing and
scanning.  The database also includes photographs that were taken as a probe
set from an automated border control (ABC) gate  in addition to morphed
images. The D-MAD situation can be evaluated using the first collection, known
as the HOMID-D dataset . Another set of 30 bona fide subjects from the FRGC v2
dataset [6] is further used to generate 30 morphed images. All the morphed
images are further carefully  postprocessed, printed, and scanned. The second
subset is referred to as the HOMID-S dataset  is suitable for the S-MAD
scenario and consists of  90 digital images and  90 printed-and-scanned
images.  To produce realistic morph attacks, both database subsets contain a
good diversity  of people of various ages, genders, and races (Please see
Section III-A).

To facilitate the large-scale psycho-visual  experiments, we also present a
customized Human Observer Evaluation Platform for morph image detection. The
new evaluation  tool simulates a practical operational situation in which
observers were given photographs to assess whether they were bona fide or
morphed. Furthermore, the platform is designed following the strict guidelines
of  General Data Protection Regulation (GDPR) to protect and preserve
participants ’ privacy with full  consideration to the anonymity of
participants.  In compliance with the university’s established ethical
standards, the participants were contacted before the experiment began to
obtain their consent (refer Section III-B for details).

While the earlier works focused on analyzing human  observers’ ability to
detect morphing attacks, observers have been predominantly selected  randomly
without carefully choosing ID control experts. In this study, we  use a total
of  469 observers who have a specific area of expertise in face-based identity
verification,  including border guards, passport officers, visa officers, ID
experts,  and forensic examiners (face, document , and fingerprints), to name
a few, and  compare their results to those of (100) observers who are not
familiar with facial comparison or forensic examination of any kind. Such an
observer base of experts and  nonexperts in our experiments also provided an
insight into the role of vast experience in face comparison111Face comparison
in this context refers to comparing two face images to make a decision on
mated or non-mated pair of subjects. and morphed image detection, which is
currently missing in the state-of-the-art studies  Table I lists the
qualifications of human observers and their typical jobs. Furthermore, we
contend that using human observers as a realistic benchmark  can help create
better training schemes for morph detection and that the knowledge gained from
such a study can  be applied to create more powerful machine-based MAD
algorithms.

Human observers may be given just one image to decide whether a presented
image is morphed in an operational scenario (for instance, passport
application), or they may be given a suspected image  and a reference image
from a  reliable source ( such as ABC gate),.  To account for both scenarios,
we study the performance of human observers under the D-MAD scenario and
S-MAD.  In particular, we use the images acquired from an ABC gate as a
trusted source image  for D-MAD human observer analysis and ask the observers
to  judge whether the second suspected image ( unknown capture) is morphed or
not using the HOMID-D subset. While in the S-MAD experiments, we employ the
images from both the morphed set and bona fide set obtained from the HOMID-S
dataset. As a third  contribution, we compare human observer performance to
machine-driven MAD algorithms to  show how well humans and machines compare
when it comes to identifying morphing attacks.

The key contributions of this work can therefore be listed as:

  * •

Human Observer Analysis Face Morph Database: Presents a new database (HOMID)
for morphed  photos and ABC gate images for  studying how well humans can spot
morphed images. The dataset also includes images for analyzing D-MAD and S-MAD
scenarios.

  * •

Human Observer Evaluation Platform: A new evaluation platform mimicking the
realistic operational scenario is designed where the images are provided to
officers to determine if the image is bona fide or morphed.

  * •

Human Observer Detection Performance: The study is based on a sizable number
of observers (469 in the D-MAD trials and 410 in the S-MAD experiment) who
regularly verify identification documents and/or compare biometric data to
present a realistic benchmark of human observers222A total of 790 observers
have participated in the experiments in different categories. However, we only
consider the observers who have completed the experiments to conduct the
analysis in analyze this work.. The study also benchmarks it against the
novice observers with no training or experience in facial comparison or
document examination to establish the benefit of domain knowledge.

  * •

Human Observer versus Machine Detection Performance: To determine the
advantages of each peer, a comprehensive examination of the performance of
four machine-driven MAD algorithms  vs human observers is presented. Two
algorithms in S-MAD and two algorithms in D-MAD benchmarked in  the NIST FRVT
MORPH challenge [7] are evaluated on the newly constructed database to
benchmark the performance of human observers against algorithms.

The preliminary ideas for morph image detection and  associated works on human
observer analysis  are presented in Section II. In Section III, Section III-A
presents the details of a newly created database for human observer analysis
and Section III-B presents the details on evaluation platform.  Section III-D
and III-E present the details of two experiment corresponding to D-MAD and
S-MAD respectively. The findings of the human observer detection performance
are presented in Section III-C together with information on the qualifications
and experience of the participants along with the details on recruitment of
observers. Section III-D and Section III-E present the details of two
different experiments in this work. Finally,  Section IV discusses the key
findings from the conducted analysis and further provides conclusive remarks
in Section VII before listing the potential future works.

##  II Background and state of the art

A collection of linked papers that analyze human observers’ capacity for MAD
are presented in this section. All representative efforts for automated
morphing detection are also added to the summary.

Reference | Post-processed | Digital | Images | Number
---|---|---|---|---
| morphed | Print-scan |  | observers
D-MAD
M. Ferrara et al.[1] 2016 | No | Digital | Bona fide 8 | Experts 44
|  |  | Morphed 20 | Non experts 543
DJ Robertson.[8] 2017 | No | Digital | Bona fide 14 | Non experts 28
|  |  | Morphed 35 | Trained non experts 42
Robertson et al.[4] 2018 | No | Digital | Bona fide 14 | Non experts 80
|  |  | Morphed 35 |
Kramer et al.[9] 2019 | Yes | Digital | Bona fide 20 | Trained non experts 80
|  |  | Morphed 20 |
A Makrushin et al. [10] 2020 | No | Digital | Bona fide 5 | Experts 21
|  |  | Morphed 10 | Non experts 168
|  |  | Impostor 5 |
A Makrushin et al. [10] 2020 | No | Digital | Bona fide 10 | Experts 21
|  | and videos | Morphed 10 | Non experts 168
|  |  | Impostor 5 |
Zhang et al. [11] 2021 | No | Digital | Bona fide 15 | Experienced 5
|  | Printed | Morphed 75 | Non experts 10
|  | Digital | Bona fide - 416 | Expert Examiners * - 469
Ours | Yes | Post-processed | Morphed - 384 | Non-Experts - 103
|  | Printed |  |
S-MAD
A Makrushin et el.[5] 2017 | No | Printed | Bona fide - 7 | Non experts 42
|  |  | Morphed 23 |
A Makrushin et al. [10] 2020 | No | Digital | Bona fide 5 | Experts 21
|  |  | Morphed 10 | Non experts 168
Nightingale et al. [12] 2021 | No | Digital | Bona fide 54 | Non experts 100
|  |  | Morphed 54 | (crowd-sourced)
Zhang et al. [11] 2021 | Yes | Digital | Bona fide 15 | Experienced 14
|  | Printed | Morphed 75 | Non experts 42
|  | Post-processed |  |
|  | Digital | Bona fide - 60 | Expert Examiners * - 410
Ours | Yes | Post-processed | Morphed - 120 | Non-Experts - 0
|  | Printed |  |

TABLE II: Detailed analysis of existing state-of-art for human observer
analysis for MAD. *Experts examiners in the context of this work include case
handlers, examiners and expert examiners.

###  II-A Human observers for face morphing attack detection

Many studies have been conducted to examine how well humans can recognize
digital morph attacks. Robertson et al. [8] studied the morph attack detection
by human observers, in addition to  face recognition system on a smartphone.
Robertson et al. [8] conducted three sets of experiments using the Glasgow
Face Matching Test - GFMT [13].  The findings of the study were based on 19
female and 30 male observers.

The initial face image of each  individual was cropped (to remove the torso)
and pasted onto a white background  that measured $14cm\times 10.5cm$ for the
experimental setup. The second image was embedded in a  frame similar to a UK
passport (same size).  Each combination of subjects  resulted in the creation
of five morphed images, with morphing factors starting  at 90% from the first
subject and 10% from the second subject  and a step size of 20%. In
Experiment-1,  the observers were asked to confirm whether the images were
mated images for the shown trusted image and the second nontrusted image
without the prior knowledge of manipulations.

Images of the same person or a modified image were used to randomize the
tests. Each trial had two images, and the observer had to choose whether the
shown images were the same person or not. Each participant in this experiment
underwent 49 trials, with 35 morphed pair trials, 7 mated comparison trials, 7
nonmated comparison trials, and 49 randomly chosen trials from the image
collection. 25 female and three male observers participated in this experiment
where the false acceptance rate of 68% was noted for morphed images (morphing
factor of 50%). It should be noted that the observers were not asked to affirm
whether they had been morphed; instead, they were just given the choice of
selecting “match” or “nonmatch.”

With a similar  setup, another experiment (referred  to as Experiment-2) was
conducted.  In contrast to Experiment-1, the  observers were informed about
face morphing attacks and given tips to recognize the morphs both before and
during the test. The observers were allowed to choose between three options
such as “match,” “mismatch,” and “morph.” In this study, which included 34
female and 8 male observers , the authors concluded that  a quick exposure to
morphing might improve morph detection accuracy. Specifically, they noted that
the morph detection error rate was significantly lower (21%) than  in the
first experiment (32%).  In Experiment 3, participants were asked to randomly
select and unlock mobile phones that had altered or real photos of the
observers. The FRS on mobile phones did not perform well even for mated
comparison trials (91.8% false rejects), and it did not allow false accepts
(0% false accepts). The authors noted significant false rejects at the expense
of being resilient to morphing assaults.

In 2016, Ferrara et al. [14] studied face morph attack detection with both
human observers and algorithms. Using the free GIMP Animation Package v2.6
[15] and the GIMP Animation Package v2.8 (GAP) [16] software,  80 face morphs
were produced. The observers were  instructed to report whether the morphing
photographs matched against a trusted bona fide (i.e. , original) face image
in each trial, which included two face images. In this experiment, there were
543 nonexpert observers and Forty-four experts (border guards). The results
showed that morphed images were not correctly detected (i.e.,  classified as a
bona fide image).

In four studies akin to those conducted by Robertson et al. [8], Kramer et
al.[9] investigated the human and  algorithmic morph detection performance
while utilizing a high-quality morph database.  Overall morph detection
accuracy of experts (56%) was higher than nonexperts (52.2%) as analyzed using
signal detection measures. There was no discernible performance gain in morph
detection following the morph detection training  session. Further, single
image-based morphing attack detection (S-MAD) was studied using 120 images and
40 participants (21 women). Kramer et al.[9] used hit (right morph
identification) and false alarm  to analyze the data (bona fide identified as
morphed image). There were no notable differences between the experts and
nonexperts in morph detection accuracy for S-MAD.  No reference morph
detection (S-MAD) algorithms outperformed human observers, it was also noted.
The study also concluded that it is difficult to detect morphed images without
using the cues of watermarks, hidden printed letters, ink, etc. [9].

Ferrara et al. [1] studied face morph attacks under  automated border control
(ABC) gate settings.  It was discovered that all 18 altered photographs were
not recognized as being morphed by qualified face comparison experts.Makrushin
et al. [10] conducted  a web-based simulation of a border control scenario.
The  observers were provided with an introduction to face morphing, morphing
attacks , and hints to identify the morphed images. The observers were
required to complete 15 trials,  each of which contained 10 morphing face
photos and 5  real face photographs. The observer had to  first contrast the
face image from the passport with the genuine or altered image that had been
provided. The second experiment was for face comparison, with 25 trials (10
mated comparisons, 5  nonmated comparisons, and 10 morphing comparison
trials).  Two images were shown to observers one on a frame  from a model
passport to be  contrasted with a brief video clip in which the traveler
rotated his or her head. Finally, the observer had to decide on the passport
face image, to decide if the  traveler’s image was morphed or not looking at
the video.  In both trials, 189  observers took part, and 21  of them had
never seen a face transform before. In the first experiment (no reference),
unskilled observers ’ true positive rate (TPR) was reported to be 65.65% , and
a TPR of 63.33% for skilled observers.  The TPR of competent and untrained
observers in the second experiment ( a face comparison)  were reported as
93.45% and 95.25% , respectively Makrushin et al. [10] claimed that training
the unskilled observers would increase the detection ability to reach the
performance of skilled observers.  The accuracy of a face-recognition program
from Dermalog Face Recognition [17, 18] was also investigated by Makrushin et
al. [10] where they noted the false acceptance rate (FAR) of human observers
was greater (15.45%) than the algorithms’s (0%).

A pair of 54 digital images was used in Nightingale et al. [12] investigation
of the human observer’s capacity to recognize altered images under S-MAD and
D-MAD  conditions. However, the study employed one hundred workers on Amazon’s
Mechanical Turk (AMT) in a crowd-based approach who were neither expert
examiners nor observers familiar with face recognition. While the observer’s
accuracy in distinguishing between genuine and fake images was 80.8% %, their
success rate in detecting morphing was only 58.3%. However, the observers who
received some training obtained a similar accuracy of 59.2%, but the work
noted that participants made different mistakes as compared to  the original
experiment.  Further testing of the face-recognition algorithm by Nightingale
et al. [12] revealed very poor performance in an Area Under the Curve (AUC) of
0.38.

Zhang et al. [11] investigated the capacity of human observers  using 42
inexperienced viewers and 14 experienced observers  using 15 genuine and 75
morphed photos. The experienced observers were extremely familiar with
morphing  and included ID experts in border control. Unlike the previous
works, Zhang et al. [11] studied five different types of morphing algorithms
that included landmark-based morphs and Generative Adversarial Network (GAN)
based morphs. The  detection accuracy for the skilled group was 97.14%.  While
detecting landmark-based morphs was difficult compared to detecting GAN-based
morphing, the inexperienced obtained an accuracy of 79.21%.  Five experienced
observers and ten novice observers were used in Zhang et al. ’s [11] study of
D-MAD detection  abilities. The group with relevant  expertise had an overall
accuracy of 86%.

A detailed analysis of all the existing state-of-the-art for human observer
analyses for MAD is presented in Table II.

###  II-B Algorithms for morphing Attack Detection

####  II-B1 Differential morphing attack detection (D-MAD)

In the lines of human observer-based morphing attack detection,  several
approaches have been devised for automated morphing attack detection.  The
second suspicious image is examined to determine if it has been morphed or not
using a trustworthy live capture image from an ABC gate or  other reliable
capture device. The first approach of D-MAD was based on inverting the
morphing process in a reverse-engineered manner, which was termed  Demorphing
[19].  Similar to this, several studies have been published in which the
difference between the feature vectors of the genuine and morphed  images are
utilized to assess whether an image is genuine or morphed [20, 21].  Morphing
attacks were also detected using features from the 3D shape and the diffuse
reflectance component estimated directly from the image in  [21], and the deep
features from two different networks were used to determine the difference in
features [20]. Another group of works investigated morphing by examining the
change in landmarks of  genuine and suspected morph images in the  region of
the face [22, 23].

####  II-B2 Single image morphing attack detection (S-MAD)

S-MAD algorithms largely rely on learning a classifier to distinguish the bona
fide image from a morphed image.  The texture data is extracted from the
normalized and aligned face  of a suspected morph image. The texture features
such as Binarized Statistical Independent Features (BSIF) and Local Binary
Patterns (LBP) are used to classify the images using a  pretrained Support
Vector Machine (SVM) classifier [24, 25, 26, 27, 23] in the earlier works.
While extending the works for MAD, another approach was proposed to exploit
the  color spaces and the scale spaces jointly [24, 28].  Pretrained deep
networks for  texture feature extraction were used to detect the morphing
attacks not only in the digital domain but also in the  redigitized domain
(after a print-scan transformation) [29] to address and detect the
postprocessed morphed images. Notably, the earlier works have employed two
deep neural networks ; including VGG19 [30] and AlexNet [31], where they
perform  a feature-level fusion of the first fully connected layers from both
the networks [29].  Other deep networks have been  researched for morph attack
detection in an ongoing effort [32]. Another approach to detecting morphing
attacks was proposed by extracting the features from the “Photo Response  Non
Uniformity (PRNU),” where the characteristics of the image sensor were
employed to determine if the image was morphed or not [33].  Better performing
algorithms have been reported , including dedicated context aggregation
networks to automatically model the noise [34] using color spaces to seek
residuals of the morphing process.

###  II-C Limitations of the existing works

With a detailed review of the existing state-of-the-art in previous section,
we note a set of limitations in the existing works in this section.

  * •

The  majority of the previously mentioned works were primarily concerned with
D-MAD or S-MAD settings.  As a result, they do not  accurately benchmark the
ability of the same human observer  to evaluate the altered images in D-MAD
and S-MAD  situations.

  * •

Furthermore, the morphed images utilized in those studies were not carefully
postprocessed to mimic the authentic passport images on ID cards, which
resulted in a somewhat skewed conclusion. For instance, Kramer et al. [9] used
a  high-quality morphed database, but the images themselves do not have a
realistic background for face images.  Without a neck or other human face
background, only the cropped face is shown. Another set of works [8, 35]
directly use the morphed image, without any modification/ postprocessing
despite many visible  artifacts leading to biased observations and thereby
conclusions.

  * •

The majority of earlier studies on morphing detection using human observers
used a smaller collection of images. However, the small size of the dataset
does not lead to a statistically conclusive result. The reported works employ
a small set of expert observers to validate the initial observations made by
the previous studies.  To obtain precise results in subjective studies,
Brysbaert [36] has discussed the significance of the sample size and number of
participants. The large sample size experiment gives a more precise
statistical analysis.

We perform empirical research depending on the information acquired from human
observers to address the  constraints of the current  state-of-the-art and
respond to the research questions listed previously.

##  III Methodology

(a)

(b)

(c)

Figure 2: Bona fide subset in HOMID-D.

Our methodology consists of three main parts, which are as follows: the
development of new datasets; the building of an evaluation platform to collect
data on human observers; and the data collecting related to the evaluation and
analysis of the human observers. We first present the analysis at a high level
for detecting D-MAD and S-MAD, following which various subanalysis is
presented measuring the performance w.r.t duration of the training, field of
expertise, and detection ability in the print-scan setting. Additionally, we
provide a thorough examination of the detection accuracy for S-MAD and D-MAD
using two alternative automated techniques.

###  III-A Datasets: Human Observer Analysis Morphing Image Database (HOMID)

We first create a new database called the HOMID to address the lack of face
images for human observer analysis of morphing  attack detection. As a noted
limitation in the earlier works, the face images employed for the human
observer analysis do not have images respecting the ICAO standards for Machine
Readable Travel Documents (MRTD) [37].  As a result, we develop a new database
that  complies with ICAO criteria while still being realistic. There are two
primary subsets of the database, each including bona fide and morph subsets.
The first subset, HOMID-D, corresponds to D-MAD settings where the bona fide
images are captured from a regular  photo studio setting, and bona fide probe
images are captured from a real ABC gate. The  process described later is used
to create the morphed photos in this subgroup. The second subset corresponds
to S-MAD settings where the bona fide images are captured in a photo studio
setting, and morphed images are created, as explained further below.  To
explore all the facets of MAD by human observers , both subsets of data have
printed-scanned photos that have undergone digital postprocessing.

Ethnicity | Caucasian | South Asian | Middle East | Chinese
---|---|---|---|---
Female | 2 | 2 | 3 | 0
Male | 26 | 10 | 2 | 3
Total | 28 | 12 | 5 | 3

TABLE III: Demographic distribution of HOMID-D subset for bona fide images

(a)

(b)

(c)

Figure 3: Example images captured under trusted capture settings from ABC gate
images corresponding to subjects in Figure 2. ABC gate images were collected
on a different day to reflect real life scenarios.

####  III-A1 HOMID Differential MAD Subset

We produce the first subset of data that contains genuine and morphed images
to explore how well human observers can recognize morphing in a differential
(i.e., reference-based)  environment. The bona fide images are collected from
48 unique subjects with different age groups, gender , and ethnicity. The
demographic distribution of the dataset is provided in Table III.

HOMID-D Bona fide Subset:  To replicate the ICAO passport photo, the face
images were  taken in a photo studio with adequate lighting, no shadows, no
poses,  and neutral expressions on a white background. The face image of each
data subject was captured multiple times, from which one image was chosen for
creating morphs, and other images were used for studying the vulnerability of
FRS. Each  photo was taken at a high resolution and  afterward reduced to a
413 × 513 pixels size. Further, the cropped face images in the bona fide set
were carefully cropped to respect the minimum Inter-Eye Distance according to
passport image standards in various countries.  There are 480 total photos in
the genuine subset. Figure 2 offers an example representation of the genuine
photos in the HOMID-D subset.

HOMID-D Trusted Capture Subset: In the D-MAD setting, a trusted live capture
image is compared against a suspect image. We  use an ABC gate to build a
series of probe pictures that correspond to trusted capture  conditions.
Images from each of the 48 subjects were captured using the ABC gate resulting
in 10 different attempts.  , five of the 10  photos were utilized to evaluate
the vulnerability of  supplied morphs. Specifically, we have employed a state-
of-art Commercial-Off-The-Shelf (COTS) system to verify the quality of the
morphed images against the captured ABC gate images.  The good quality of the
produced morph pictures and postprocessing quality is confirmed by the high
comparative score 333A high score here refers to a score crossing the
predefined threshold by COTS SDK for a given FAR. For instance, Neurotech
defines the threshold of 36 at FAR=0.1%, and any score crossing the value is
considered a high score.. The rest of the 5 images from each subject were
further scrutinized for appearance and quality  scores by an expert
researcherbased on which the best image was chosen for subjective experiments.
Figure 3 shows a selection of photographs from our database  that correlate to
the real images seen in Figure 2.

HOMID-D Morph Subset:  We combine two similar subjects to generate 400 morphed
images using the newly created  photographs of 48 real subjects (in terms of
age, gender, ethnicity, and skin color). Based on the earlier works indicating
high attack strength of morphed images with a morphing factor of  0.5 [1, 14,
3, 2](i.e., 50% contribution from each subject), we create the morphed images
using a morphing factor of $0.5$.  To prevent the creation of implausible
morphing combinations the subjects for morphing were chosen by a human expert.
Further, to create highly realistic morphed images to challenge the human
observers, each of the morphed images was postprocessed to eliminate any
artifacts in the image signal stemming from the morphing process, such as
double contours in the iris region, incorrect lip region, etc. Two specialists
also examined each postprocessed image to ensure the excellent quality of
created morphed images produced. A sample illustration of a  postprocessed
image is provided in Figure  4 where one can observe the naive morph
generation resulting in visible  artifacts in the iris and nose areas while
the  postprocessed image eliminates such  artifacts, resulting in a realistic
face image.

Figure 4: Morphed image which has visible artifacts near the iris area and the
nose areas and the corresponding post processed image with artifacts
eliminated.

Additionally, since providing a printed image that is  afterward scanned is a
standard procedure in many nations, we created another subset of  morphed
photographs by printing and scanning them  with an Epson Expression Premium
XP-7100 [38] photo printer. Thus the morphed image set has a total of 400
digital images and 400 printed-scanned images, both after  postprocessing.

| Digital | Print and scanned
---|---|---
Bona fide | Morph |  | Post-processed
---
morph
| ABC Gate
---
images
Bona fide | Morph |  | Post-processed
---
morph
| ABC Gate
---
images
Male | Individual subject | 40 | 40 | 40 | 40 | 40 | 40 | 40 | 40
Total images | 80 | 80 | 80 | 80 | 80 | 80 | 80 | 80
Female | Individual subject | 8 + 8* | 8 | 8 | 8 + 8* | 8 | 8 | 8 | 8
Total images | 16 + 16* | 16 | 16 | 16 + 16* | 16 | 16 | 16 | 16
Total | 112 | 96 | 96 | 96 | 96 | 96 | 96 | 96
Total | 416 | 384

TABLE IV: Distribution of images in the database. *Note: Additional bona fide
images used for checking the consistency of user participation, i.e., if the
observer is randomly selecting images as bona fide or morph in continuous
trials.

####  III-A2 HOMID Single Image MAD Subset

While  decisions about whether an image is genuine or a morph can be  made
based on an image pair  in some application settings, such as border control
(e.g. , with trusted image capture),  other scenarios, like visa or passport
applications , just have one image without any reference  images. For
instance, applying for a  Bl USA visa needs the applicant to upload the
passport image, and this could potentially be a morphed image on which the
decision has to be made.  To study the human observer performance for  MAD in
such scenarios, we create a Single-Image Morphing Attack dataset which we
refer to as the HOMID-S dataset.  The HOMID-S subset has bona fide and
morphed image subsets,  similar to HOMID-D subset, as discussed below. In
addition, to consider image capture conditions other than our dataset, we make
use of  the FRGC v2 dataset  [6] to derive HOMID S-MAD dataset as explained
further.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 5: Sample images for HOMID-S database. (a) Digital bona fide image, (b)
Digital morphed image, (c) Digital post processed image, (d) Print and scanned
bona fide image, (e) Print and scanned morphed image, (f) Print and scanned
post processed image.

(a) iHOPE D-MAD experimental setup. The participant is asked to make a
decision and choose "Bona fide" or "Morphed" images given the "Trusted live
capture" image and an "Unknown capture" image.

(b) iHOPE S-MAD experimental setup. The participant is asked to make a
decision and choose "Bona fide" or "Morphed" images given one image from
"Unknown capture" setting.

Figure 6: Graphical user interface for the human observers experimental setup
used in this work.

HOMID-S Bona fide Subset:  We selected 30 subjects  (15 men and 15 women) from
the FRGC v2 dataset [6] to produce a dataset of bona fide images. As the FRGC
v2 dataset provides multiple captures for each subject, we choose a high-
quality enrolment image for the bona fide set and further process each image
to conform to ICAO standards for the face image.  The selected bona fide
subset has been carefully checked to make sure that any rotations or poses are
corrected. The demographic distribution of the chosen subjects is provided in
Table V.

| Digital | Print and scanned
---|---|---
| Bona fide | Morph | Post-processed | Bona fide | Morph | Post-processed
|  |  | Morph |  |  | Morph
Male | 15 | 15 | 15 | 15 | 15 | 15
Female | 15 | 15 | 15 | 15 | 15 | 15
Total | 90 | 90

TABLE V: Demographic distribution in the HOMID-S database

HOMID-S Morph Subset: We further generate morphed images for the chosen 30
subjects using a morphing factor of  0.5. To produce realistic morphed photos,
each subject is merged with another subject that closely resembles it in terms
of age, gender , and race. Further, to assure the high quality of the morphed
images, we carry out manual  postprocessing with human expert intervention.
We consequently produced 15 morphed  images for male individuals and 15
morphed images for female subjects. Each of the morphed images, which is
postprocessed, is also printed and scanned in the lines of previous works [24,
3, 2] using an Epson Expression Premium XP-7100 [38] photo printer.

A sample illustration of the images in the HOMID-S subset is further provided
in Figure 5 which depicts both bona fide and morphed images along with the
post-processed and printed-and-scanned images.

###  III-B Platform: iMARS Human Observer Platform for Evaluation - iHOPE

We also developed an individualized Human Observer Platform for Evaluation
(iHOPE) for  morphing image identification to support large-scale trials. The
new evaluation platform mimics the realistic operational scenario where the
images are provided to observers to determine if the image is bona fide or
morphed.  Additionally, the platform  was created under the tight restrictions
of the GDPR to safeguard participants’ privacy while giving full regard to
participants’ anonymity.

Currently, the platform supports two sets of experiments that correspond to
the D-MAD and S-MAD scenarios . These experiments can be carried out on a
desktop environment  that is modeled after the methods currently used in
operational settings. The participants first need to register their details by
providing their email address (voluntary), name, age, and gender.  On the
homepage, there is a summary of the experiment’s objectives for each
participant. Moreover, each participant was asked to indicate whether they had
any training in facial comparison, document examination and morphing, the
length of said training, and their line of work. The platform first  takes
users to the D-MAD experiment, where  they may view their results and learn
how much time they spent on each task. Then, the platform presents a pair of
images of a chosen subject by providing the trusted capture image (captured
from an ABC gate) and an  unknown-capture image. The unknown-capture image is
chosen at random from either a morphing set or a bona fide set of the
relevant subject, whereas the trusted capture image is genuine. All the
observers are presented same pairs in D-MAD experiments to derive the
conclusions in this work.  Each participant sees the photos on the platform in
a different order at random. The platform is also designed to collect meta-
data on how much time the participants spend on each image pair.  By providing
the morphed images, bona fide images, and trusted capture (ABC) gate images
with the same  clothes in both images for each pair of  photographs, care is
taken to avoid any decision bias caused by the clothing of the given images.
Figure  6a illustrates the designed iHOPE-D-MAD evaluation page.

Considering various operational practices, the platform also allows the
participants to zoom in and enlarge the images before taking a decision. The
platform also enables the participants to take a break and resume the
experiments as needed, which lessens the strain on their minds during lengthy
cognitive experiments. The total score is shown in the D-MAD experiments every
time the participant clicks on “Continue later.” Once the D-MAD experiment is
over, the platform then refers users to the S-MAD experiment. Figure 6a
presents the iHOPE S-MAD evaluation platform where the participant is
presented with one image from an unknown-capture setting of a subject. The
subgroups of bona fide and morphed images from which the photographs are drawn
are presented at random.

###  III-C Participants/Observers: iMARS Human Observer Evaluation

The online evaluation platform  uses two alternative configurations, called
D-MAD and S-MAD, to compare how well humans are at spotting morphed images.
The platform was designed to allow the participants to pause and resume the
experiment at their own pace to accommodate busy schedules, and avoid fatigue
and cognitive load in making decisions for the experiment.

##### Recruitment of Observers

The observers were recruited from national organizations that regularly deal
with ID management and verification. Invitations were sent through secure
channels through liaisons at various agencies to create an observer pool of
Border Guards (30), Case handlers - Passport, visas, ID, etc (150), Document
examiners - 1st line (38), Document examiners - 2nd line (40), Document
examiners - 3rd line (30), Face comparison experts - Manual examination (44),
ID Experts (53) and Others (84). The experiment’s objectives were explained to
the observers in a brief introduction and set of instructions. The
participants were asked for explicit consent according to the national
guidelines and GDPR. In addition to obtaining consent, a specific
questionnaire was used to record participant’s demographics, employment
history, and training duration for our study. Further, each participant was
given a unique code to anonymize the participant information, which could also
be used to get the data deleted if the participant did not wish to allow
his/her data to be used for analysis.

In the set of questionnaire, the participants were asked to provide the
following information:

  * •

Age, gender, email address and nationality

  * •

Profession/Line of work

    * –

Experience with facial examination ($<6$ months, 6-12 months, 1-3 years, 3-5
years and more than 5 years)

    * –

Experience document examination ($<6$ months, 6-12 months, 1-3 years, 3-5
years and more than 5 years)

    * –

Is the participant a documented super recogniser?

    * –

Experience in morphing attack detection (Half a day, Half-day to one day, 2-3
days, 4-5 days, more than 5 days)

    * –

Line of work (Face comparison, Fingerprint expert, Document examiner (1st, 2nd
and 3rd line), ID Expert (Embassy, Police and other government authorities),
Case handler (ID, Visa, Passport), Border guard (1st line, 2nd line))

Comparison | Digital images | Print and scanned
---|---|---
Morph vs Bona fide | 48 | 48
Morph vs ABC gate image | 48 | 48
Post processed morph | 48 | 48
image vs Bona fide |  |
Post processed morph | 48 | 48
vs ABC gate image |  |
Bona fide vs Bona fide | *16 | 0
Total | 208 | 192

TABLE VI: Statistics of different comparison trials in D-MAD evaluation. *
indicates the total trials to check the consistency of the participants.

###  III-D Experiment-I: D-MAD Human Observer Evaluation

In this study’s 400 image  comparison trials, two face photos were displayed
side by side as  illustrated in Figure 6a. Each trial was randomized by
choosing the image pairs randomly. 400 trials were divided into four parts,
where each part consisted of 100 unique comparisons.  Participants were asked
to  determine if the second  image, which was assumed to be from an unreliable
capture, was a “bona fide” or “morphed image” after viewing the first image
from a trusted live capture. An optional zoom-in function was provided on the
platform for participants who wished to enlarge the images before deciding,
mimicking the real-life deployment solution.  After 100  comparisons, the
acquired accuracy was frequently displayed to encourage the participants.

The D-MAD experiment consisted of 48 comparison trials in 5 different
categories as listed in Table VI.  The evaluation added eight random bona fide
v/s bona fide comparisons  in addition to the identification of morphing
images to conduct a consistency check  on whether a user was  arbitrarily
selecting the same image  repeatedly or if the clicks were random. The final
analysis has removed all such data where a random click  behavior was
observed.

###  III-E Experiment-II: S-MAD Human Observer Evaluation

In the S-MAD human observer evaluation, a total of 180 individual images were
presented to each observer. The participants were  only given one image, as
shown in Figure 6b and were required to decide whether it was a genuine
photograph or a morphed image. The face images displayed in the evaluation
platform  were rescaled to EU passport image standards ( 413 × 531 pixels and
JPEG2000 compression) [39]. The participants  had access to a zoom feature
that enlarges the image to fill the entire screen as they analyze how a border
officer or examiner might conduct a comparable operation in real life. The
participants further received the final scores after the completion of 180
trials and intermediate scores after 90 trials.

##  IV Findings, Analysis and Discussion

This section  covers the results of the tests and offers a thorough analysis
of the  patterns seen in the evaluation of human observers. The analysis is
based on 469 observers in the D-MAD experiment and 410 observers in the S-MAD
experiment. The distribution of a different line of work is presented in Table
VII in D-MAD and S-MAD experiments.  For the reader’s benefit, we also provide
a thorough discussion of each subgroup of findings.

TABLE VII: Distribution of participants in D-MAD and S-MAD experiments | D-MAD | S-MAD
---|---|---
| Number of | Average | Number of | Average
Line of work | participants | Accuracy | participants | Accuracy
Border Guard | 30 | 64.66 | 26 | 55.17
Case handler- Passport, visas, ID, etc | 150 | 63.45 | 137 | 56.65
Document examiner- 1st line | 38 | 60.79 | 30 | 57.63
Document examiner- 2st line | 40 | 68.64 | 34 | 62.56
Document examiner- 3rd line | 30 | 65.74 | 25 | 61.51
Face comparison expert (Manual examination) | 44 | 72.56 | 39 | 64.63
ID Expert | 53 | 63.09 | 50 | 57.21
Other | 84 | 64.66 | 69 | 55.17
Student | 103 | 56.91 | - | -
Total participants | 572 |  | 410 |
Experts | 469 |  | 410 |

###  IV-A Metrics for Evaluations

To show the results, we use Attack Presentation Classification Error Rate
(APCER) and Bona fide Presentation Classification Error Rate (BPCER). APCER is
defined as as the proportion of attacks classified as bona fide and BPCER is
the proportion of bona fide images classified as attacks.The accuracy is
reported as an average of Attack Presentation Classification Error Rate
(APCER) and Bona fide Classification Error Rate (BPCER) aligning the results
to NIST FRVT MORPH challenge [7] and is defined as below:

$Accuracy=\frac{(1-APCER)+(1-BPCER)}{2}$ (1)

###  IV-B Overall Accuracy in D-MAD and S-MAD

We first present the overall detection accuracy for D-MAD and S-MAD before
analyzing each of them individually. Figure 7 shows that while few observers
have more than 95% accuracy, the average accuracy of identifying the morphing
attack in the D-MAD condition is 64.10% (min=19.75%, max=99.50%, SD=17.71%,
Mdn=62.75%). We note a slightly lower average accuracy in the S-MAD setting
corresponding to 58.98% (min=33.33%, max=85%, SD=9.75%, Mdn=59.44%). Unlike
the D-MAD setting, we note that none of the observers have more than 82%
detection accuracy in S-MAD and this can be attributed to the challenging
decision-making process in the absence of a reference image. We analyze and
present several factors in the following subsections based on these
observations.

Figure 7: Overall accuracy of morphed image detection for D-MAD and S-MAD
experiments

###  IV-C Training in facial comparison v/s MAD

Since facial examination expert  viewers are considered to be skilled in
facial comparison, we  predicted that factors like the amount of time spent on
topic-specific training and  the amount of facial examination experience a
person has will affect how accurately they detect face morphs. Therefore, we
analyze the MAD accuracy of human observers for both D-MAD and S-MAD
experiments.  We specifically, group the observers into six groups based on
the  length of training they have received: no prior experience, less than 6
months,  6–12 months, 1–3 years, 3-5 years, and more than 5 years. We make the
following observations from the results obtained:

####  IV-C1 Results for Experiment-I - D-MAD

  * •

The observers with no specific training on face comparison obtain a MAD
accuracy of 62.81% (min=19.75%, max=98.50%, SD=17.69%, Mdn=61.50%). In
contrast, experienced observers in D-MAD settings, as shown in Figure 8a,
achieve an average accuracy of 64.89% (averaged over five different experience
groups, (min=23.75%, max=99.50%, SD=17.71%, Mdn=63.37%).

  * •

We use the Kruskal–Wallis test [ 41]  to assess how different observers with
various levels of face examination training detect morphing attacks. The test
revealed no significant differences (H(5)=7.85, p=.167) across the groups (n =
173, 74, 30, 77, 44, 71).

  * •

We further investigate if longer training length in facial examination helps
in detecting morphing attacks and we conduct another Kruskal–Wallis test [ 41]
. The test found no significant differences between the groups (Less than 1
year of training = 277, More than 1 year of training = 192) (H(1)=1.01,
p=.315). And a similar observation is also made for observers with more than 5
years of training with a (H(1)=0.5, p=.418, Less than 5 years = 398, More than
5 years = 71).

####  IV-C2 Results for Experiment-II - S-MAD

  * •

The average MAD accuracy with respect to facial examination training in the
S-MAD setting is 59.77% (averaged over five different length categories,
(min=36.11%, max=80.56%, SD=9.09%, Mdn=60%)). At the same time, observers with
no experience had an accuracy of 57.98% (min=33.33%, max=85%, SD=10.70%,
Mdn=57.78%) as presented in Figure 8b.

  * •

Observations for different training lengths for facial examination in the
S-MAD setting indicated no significant differences (H(5)=13.63, p=.018) across
the groups (n = 155, 62, 27, 65, 41, 60) with no training to more than 5 years
of training. No significant differences (H(1)=0.88, p=.348) across the groups
(Less than 1-year training = 244, More than 1-year training = 166) were
further observed and in similar lines, when comparing observers with more than
5 years of training to those with fewer than 5 years, no discernible changes
were found (H(1)=2.39, p=.122, Less than 5 years = 350, More than 5 years =
60).

####  IV-C3 Common Observations

The participants (74 in D-MAD and 62 in S-MAD) with less than 6 months of
training in face comparison (62.52% in D-MAD, 61.53% in S-MAD) showed no
significant differences among the different observers to observers with 6–12
months of training (67.59% in D-MAD, 62.53% in S-MAD). However, the observers
with 6–12 months of training performed best in both categories. Our
observation indicates that training in facial comparison does not necessarily
lead to better MAD. As far as we are aware, there is currently no
comprehensive training program for MAD by humans. Generally, MAD is just one
part of document examination training or face comparison training.

(a) D-MAD Accuracy

(b) S-MAD Accuracy

Figure 8: Impact of training in facial examination on MAD accuracy of human
observers.

| D-MAD | S-MAD
---|---|---
| # | Acc | Min | Max | SD | # | Acc | Min | Max | SD
No | 173 | 62.81 | 19.75 | 98.50 | 17.69 | 155 | 57.98 | 33.30 | 85.00 | 10.79
< 6 months | 74 | 62.52 | 36.00 | 98.25 | 16.60 | 62 | 61.53 | 38.89 | 78.33 | 8.52
6-12 months | 30 | 67.59 | 26.25 | 98.75 | 19.49 | 27 | 62.53 | 38.89 | 80.56 | 10.48
1-3 years | 77 | 64.65 | 24.25 | 99.50 | 18.29 | 65 | 60.21 | 41.67 | 76.11 | 7.83
3-5 years | 44 | 61.14 | 33.00 | 91.25 | 14.85 | 41 | 57.68 | 40.00 | 76.67 | 8.81
> 5 years | 71 | 62.50 | 23.75 | 98.50 | 18.65 | 56 | 56.94 | 36.11 | 75.00 | 9.77

###  IV-D Experience in document examination v/s MAD

We also evaluate the impact of the experience of document examiners on the
accuracy  of detecting the morphed images. We  divide the document examiners
into six groups based on the  length of their training specifically in
document examination, no prior experience, less than 6 months,  6–12 months,
1–3 years, 3–5 years, and more than 5 years. We make the following
observations from the results obtained.

####  IV-D1 Results for Experiment-I - D-MAD

  * •

The observers with specific training on document examination reached an
average accuracy of 65.12% in D-MAD (258 participants, min=19.75%, max=98.50%,
SD=16.51%, Mdn=62.25%). Comparatively, the untrained observers had an average
accuracy of 59.27% (186 participants, min=65.05%, max=99.50%, SD=23.75%,
Mdn=63.50%).

  * •

The observers with less than 6 months (43 participants) and 6–12 months (22
participants) training in document examination obtained a better average
accuracy of 65.31% and 68.12% in the D-MAD setting. However, the average
accuracy of all the observers with training is 63.95%, as noted in Figure 9a.

  * •

We use the Kruskal–Wallis test, as described in the preceding section [40], to
assess the variations in MAD across observers with varying training lengths in
document analysis. The test revealed no significant differences (H(5)=5.23,
p=.388) across the groups (n = 211, 43, 22, 66, 45, 82 in the order of no
training to more than 5 years of training, refer to Fig 9a.

  * •

Further research into whether longer training durations in document
examination aid in the detection of morphing attacks found no appreciable
changes (H(1)=1.94, p=0.164) across the groups (Less than 1 year training =
254, More than 1 year training = 193). Additionally, a similar finding was
made with observation with more than 5 years of training with an H(5)=0.02,
p=.879 (Less than 5 years = 387, More than 5 years = 82) using the
Kruskal–Wallis test.

(a) D-MAD Accuracy

(b) S-MAD Accuracy

Figure 9: Impact of training in document examination on MAD accuracy of human
observers.

| D-MAD | S-MAD
---|---|---
| # | Acc | Min | Max | SD | # | Acc | Min | Max | SD
No | 211 | 65.05 | 23.75 | 99.50 | 19.08 | 191 | 59.35 | 33.30 | 85.00 | 10.59
< 6 months | 43 | 65.31 | 32.50 | 97.50 | 17.06 | 36 | 59.72 | 38.89 | 73.89 | 9.15
6-12 months | 22 | 67.57 | 41.00 | 92.50 | 13.57 | 20 | 62.53 | 48.33 | 79.44 | 8.68
1-3 years | 66 | 60.03 | 24.50 | 96.00 | 16.66 | 55 | 57.72 | 41.67 | 73.33 | 7.46
3-5 years | 45 | 62.76 | 33.00 | 95.25 | 15.72 | 41 | 59.62 | 43.33 | 77.22 | 8.44
> 5 years | 82 | 64.10 | 19.75 | 98.50 | 16.86 | 67 | 57.12 | 34.44 | 79.44 | 10.24

####  IV-D2 Results for Experiment-II - S-MAD

  * •

We note a drop in average accuracy in the S-MAD setting for observers with no
prior training (186 participants) in document examination with an accuracy of
59.27% compared to 65.12% in the D-MAD setting. The decrease in accuracy
highlights how difficult the S-MAD setting is when there is no reference image
to compare it against.

  * •

The observation for the S-MAD setting where no significant variations were
seen among groups of observers with various training lengths in document
evaluation with a similar methodology to D-MAD (H(5)=6.61, p=.250, n = 191,
36, 20, 55, 41, 67) with no training to more than 5 years of training as shown
in Figure 9b. No significant differences were further observed for groups with
less than 1 year training (n = 56) and more than 1 year training (n = 163)
where (H(1)=3.79, p=.051). In a similar manner, no discernible variations were
found between observers with more than 5 years of training and those with less
than 5 years of instruction (H(1)=2.82, p=.093, Less than 5 years = 343, More
than 5 years = 67).

####  IV-D3 Common Observations

Similar to the D-MAD setting, the accuracy of observers with 6–12 months (20
participants) training in document examination obtain a higher average
accuracy of 62.53%. The finding confirms the difficulty of recognizing the
morphing images in an S-MAD setting, although the average accuracy of all
observers with training in document examination is 58.66% as shown in Figure
9b, which is lower than the accuracy in the D-MAD setting, the observation
reiterates the challenge of detecting the morphed images in an S-MAD setting.
Despite no major differences in average accuracy for observers with face
comparison training and document examination, the individual observers with
face comparison training perform better in detecting morphs than document
examiners. Although the early signs suggest that one category is more
competent than the other, it should be highlighted that there is a need for a
detailed investigation.

###  IV-E Line of Work v/s MAD Accuracy

Additionally, we examine the categories of border guards and case handlers to
determine the accuracy of MAD based on the line of work (for Passport, visas,
ID, etc.), document examiners - 1st line, document examiner- 2nd line, expert
document examiners- 3rd line, face comparison experts (Manual examination),
3rd line, ID experts and miscellaneous as another category.  According to
their area of work, Figures Figure 10a and Figure 10b show the accuracy of
human observers.  Below are some observations we made as a result of these
analyses.

####  IV-E1 Results for Experiment-I - D-MAD

  * •

The observers who work on face comparison (manual examination) have the
highest average accuracy of 72.56% (44 participants, min=23.75%, max=99.50%,
SD=20.97%, Mdn=77.25%), with 12 participants obtaining more than 90% accuracy
in the D-MAD setting.

  * •

While a high accuracy is noted for case handlers of passports, visas, and ID
with 11 participants obtaining more than 90% accuracy, 26% of participants (a
total of 150 participants) obtain less than 50% accuracy in detecting morphs
in the D-MAD setting.

  * •

Additionally, it should be mentioned that the border patrol agents have
respectable accuracy, despite a tiny percentage of them being unable to
identify the morphs in the D-MAD environment.

####  IV-E2 Results for Experiment-II - S-MAD

  * •

In contrast to D-MAD, the accuracy is significantly lower in the S-MAD setting
for border guards.

  * •

Similar to the last set of analyses, the average accuracy decreases in the
S-MAD scenario when no reference photos are available, independent of the
lines of work.

####  IV-E3 Common Observations

Special consideration should also be given to various observers outside of a
specific field of expertise who match face comparison specialists in morph
detection in the D-MAD settings, while the average performance drops in this
group in S-MAD settings.

(a) D-MAD Accuracy

(b) S-MAD Accuracy

|  | D-MAD | S-MAD
---|---|---|---
| # |  | Acc | Min | Max | SD | # | Acc | Min | Max | SD
1 | Border Guard | 30 | 63.45 | 39.75 | 94.25 | 14.18 | 26 | 55.71 | 42.22 | 67.22 | 6.51
2 | Case handler Passport, visas, ID, etc | 150 | 60.79 | 24.25 | 98.50 | 17.14 | 137 | 56.65 | 33.33 | 79.44 | 9.57
3 | Document examiner- 1st line | 38 | 62.00 | 27 | 98.50 | 18.27 | 30 | 57.63 | 35.00 | 77.22 | 9.73
4 | Document examiner- 2nd line | 40 | 68.64 | 34.25 | 92.50 | 13.65 | 34 | 62.56 | 41.67 | 80.56 | 8.40
5 | Document examiner- 3rd line | 30 | 65.74 | 19.75 | 96.00 | 16.37 | 25 | 61.51 | 34.40 | 79.44 | 11.26
6 | Face comparison expert (Manual) | 44 | 72.56 | 23.75 | 99.50 | 20.97 | 39 | 64.63 | 38.89 | 77.78 | 8.38
7 | ID Expert | 53 | 63.09 | 24.75 | 98.50 | 18.54 | 59 | 57.21 | 36.11 | 77.78 | 9.43
8 | Other | 84 | 64.66 | 24.75 | 98.50 | 18.25 | 69 | 61.04 | 34.44 | 85.00 | 10.61

Figure 10: Impact of line of work on MAD accuracy of human observers.

We further conduct a statistical analysis  utilizing a different field of
study and, due to the variable number of observers, we  utilize ranks in one-
criterion variance analysis [40] since we are motivated by small distinctions.
With the given number of observers in each different line of work
corresponding to Figure  Figure 10a (n = 30, 150, 38, 40, 30, 44, 53), we note
the  H(6)=18.99, p=.004 indicating significance between the observers in  a
different line of work. However,  in contrast to observers in  other fields,
experts in face comparison (Manual examination), we note  an H(1)=10.53,
p=.001 suggesting the better performance of face comparison experts.

###  IV-F MAD for Digital v/s Print-Scan Images

We note the difficulty of the MAD  job for digital and printed-scanned
photos, which relate to a real-life  setting, in addition to the analysis that
was previously provided. While the former is common in applying for visas by
uploading digital images, the latter can also be observed in different
countries where the printed-scanned images can be uploaded to obtain valid ID
documents.  This series of analyses compare the accuracy of digital versus
printed-scanned morphed images , as seen in Figure 11.

####  IV-F1 Results for Experiment-I - D-MAD

It can be noted that the observers spot the morphed images in the digital
domain with an average accuracy of 64.22% (min=42.05%, max=95.73%, SD=10.35%,
Mdn=63.05%) in D-MAD settings. In contrast, the accuracy drops slightly for
printed-scanned images to 63.39% (min=38.80%, max=94.40%, SD=10.55%,
Mdn=64.41%). To further comprehend whether the detection accuracy for digital
and print-scan attacks differs statistically, we conduct Levene’s test [41] by
performing ANOVA on the absolute deviations of the data values from their
group means and we note a t(1.93)=7.26, p=.007. The analysis indicates no
discernible difference between observers in identifying either category
(p>.001).

####  IV-F2 Results for Experiment-II - S-MAD

However, the average accuracy drops to 59.18% (min=32.22%, max=88.88%,
SD=11.70%, Mdn=58.88%) and 58.78% (min=32.22%, max=81.11%, SD=9.05%, Mdn=60%)
for digital and printed-scanned images, respectively, in S-MAD settings. It
should be observed that no observer exceeds an accuracy of 90% in the S-MAD
setting, highlighting the difficulty in identifying altered images in the
absence of a reference image. The lower quality of images in S-MAD can be a
factor for lower accuracy. Further analysis is needed to understand the drop
in accuracy for both S-MAD and digital v/s printed-scanned images.

(a) D-MAD

(b) S-MAD

Figure 11: Impact of digital v/s print-scan morphed images on MAD accuracy of
human observers.

###  IV-G MAD v/s Demographics of Observers

To see whether these traits contribute to improved MAD accuracy, we also
examine the demographics of the observers. Specifically, we look at age,
gender , and country of work (not ethnicity) for S-MAD and D-MAD. Figure 12
presents the correlation of MAD versus  the age of the observers.

####  IV-G1 Results for Experiment-I - D-MAD

As noted in Figure 12a and Figure 12b, there appears to be no strong
correlation between the age of observers and MAD accuracy.

####  IV-G2 Results for Experiment-II - S-MAD

It is simple to see the general trend of decreased MAD accuracy in S-MAD
settings compared to D-MAD settings by comparing Figure 12b to Figure 12a. The
absence of an image pair with a trusted live capture as a reference makes it a
challenge to determine if a suspected image is morphed or not.

Since the majority of the observers are from European  nations, we do not come
to any firm conclusions even if we also analyze the detection accuracy in
terms of nationality. Also, no significant differences were observed
concerning gender, as noted in Figure E.1 in Appendix.

(a) D-MAD Accuracy

(b) S-MAD Accuracy

Figure 12: MAD accuracy versus age of human observers.

###  IV-H Improvement of detection accuracy over the number of images

We examine whether observers become more adept at seeing morphs after viewing
a particular number of photos. Taking into account the low accuracy of the
various groups of observers in  identifying morphing attacks in both D-MAD and
S-MAD  situations we also analyze if the observers get better at detecting
morphs after looking at a certain number of images. We thus analyze the
observer accuracy by dividing the whole set of images into 4 blocks in the
D-MAD setting, where each consists of 100 image pairs , and the 2 blocks in
the S-MAD setting, where each set consists of 92 images.

####  IV-H1 Results for Experiment-I - D-MAD

As noted in the Figure 13, the average accuracy of each observer group
increases for every 100 images completed in the case of D-MAD (Figure 13a).
Although this is only a preliminary finding, it is clear that by viewing a
large number of photographs, observers improve their ability to recognize
altered images. A positive interpretation of this observation is that a
dedicated training program in examination-based MAD would generally
efficiently increase the competence in detecting morphs.

We further validate the observation by conducting a positive trend analysis
using Cox and Stuart Test [42]. We are specifically interested in determining
whether there is, in fact, a time-dependent trend by assessing the observers’
accuracy of observers for every 100 pairs of photos, which are presumed to be
independent observations (i.e., the observations are, in fact, not independent
and observers become better at the task). As noted in Table VIII, we see a
significant level of the test (p<.001). With obtained p-value of the tests for
each set of 100 pairs, we conclude with sufficient evidence that seeing a
larger amount of D-MAD pairs improves the accuracy of the observer. This
discovery strengthens the argument that observers should receive training
where they can see a variety of new cases to hone their detection abilities.

Increasing Trend | False | True | True
---|---|---|---
p-value | 0.642 | <0.001 | <0.001
TABLE VIII: Cox and Stuart Trend Test for measuring change in accuracy of
observers against the number of D-MAD pairs seen.

We also add the hypothesis that seeing more examples will improve the
detection accuracy of our test. For this aspect, we draw observers randomly
from total observer populations and conduct a One-way analysis of variance [
44] . We note the average accuracy for each category with 57.54%, 62.95%,
66.44% and 69.47% with p < .0001 supporting our observation that the accuracy
increases with examples seen.

####  IV-H2 Results for Experiment-II - S-MAD

While a similar conclusion cannot be derived from S-MAD settings depicted in
Figure 13b, we can only speculate on the difficulty of the issue and come up
with better training methods for morph detection based on single photos. We
further assert that better competence through training programs on an
understanding of image quality help in future works.

(a) D-MAD Accuracy

(b) S-MAD Accuracy

Figure 13: Improvement in MAD accuracy over the total number of images seen.

###  IV-I Bona fide versus morphed image detection in ABC Gate Settings in
D-MAD

In this  part, we examine the function of ABC gate images and morphed images
shown alongside as in a border-crossing scenario to detect morphs . As noted
in Figure 14, the accuracy of the  MAD increases when the ABC gate images are
provided as a reference image. We specifically note an increase of 10% in
detection accuracy for female subjects while the accuracy improvement for
morphed images of male faces results in around 5% average accuracy.  We
conduct a Kruskal–Wallis test between two detection groups when the ABC gate
images are available versus when not-available  to see whether the ABC gates
help in detecting morphed images to a greater extent (n = 80, 80
correspondingly). We note a  H(1)=7.55, p=.006 for male face images and
(H(1)=2.81, p=.093) for female face images. We conclude that there is no real
advantage for spotting morphed images when ABC gate images are provided. Our
hypothesis here is that the face images of female subjects vary significantly
due to varying hairstyles and everyday make-up making the observers not detect
all the morphed images. However , this requires additional research, which
will need enlarging the dataset with and without make-up . While it is noted
that the average p-value  for male and female face  images is H(1)=6.36,
p=.011 suggests the need for  more reliable live capture to  improve the
detection. We further hypothesize that  high-quality live images from ABC
gates could generally provide more details for border guards, examiners, etc.,
making comparisons and MAD easier.

(a) Regular probe - Female

(b) Post processed - Female

(c) Regular probe - Male

(d) Post processed - Male

Figure 14: MAD accuracy in D-MAD setting with regular probe images v/s ABC
Gate probe images for morphed images of male and female subjects

##  V MAD Algorithms v/s Observer Accuracy

We further analyze the performance of human observers against two algorithms
in both categories of D-MAD and S-MAD as explained in the section below.

####  V-1 Results for Experiment-I - D-MAD

We select DeepFeature Difference (DFD) [20] and Local Binary Pattern - Support
Vector Machine (LBP-SVM) [23] to compare the automated D-MAD algorithms
against average human observer accuracy as both the algorithms were evaluated
under NIST FRVT MORPH challenge [7]. While DFD [20] obtained best performance
in detecting digital images, LBP-SVM [23] obtained reasonable accuracy in NIST
FRVT MORPH challenge [7]. As noted from the results in Table IX, the DFD [20]
obtains very high accuracy, LBP-SVM [23] obtains random accuracy of 50%. The
human observer accuracy in best case is 64.61%,  which is better than the
weakest algorithm  but less accurate than DFD [20]. The high accuracy of DFD
can be attributed to extracting  identity-specific features for detecting
morphs.  To demonstrate incorrect classifications in bona fide and morph
attacks, we also include the violin plots for APCER and BPCER in Figure 15.

(a) APCER for D-MAD experiments with regular probe images

(b) APCER for D-MAD experiments with probe images from ABC Gates

(c) BPCER for D-MAD experiments

Figure 15: APCER and BPCER for D-MAD

Data | Approach | Accuracy | APCER | BPCER
---|---|---|---|---
Digital |  DeepFeature Difference [20] | 97.91 | 0 | 4.16
LBP-SVM [23] | 50.00 | 0 | 100
Average Observer Accuracy | 63.80 | 39.08 | 36.55
Print-Scan |  DeepFeature Difference [20] | 97.91 | 0 | 4.16
LBP-SVM [23] | 50.00 | 0 | 100
Average Observer Accuracy | 62.76 | 37.11 | -
Post-Processed |  DeepFeature Difference [20] | 97.91 | 0 | 4.16
LBP-SVM [23] | 50.00 | 0 | 100
\- Digital | Average Observer Accuracy | 64.61 | 35.35 | -
Post-Processed |  DeepFeature Difference [20] | 96.87 | 0 | 6.25
LBP-SVM [23] | 50.00 | 0 | 100
\- Print-scan | Average Observer Accuracy | 63.90 | 35.37 | -

TABLE IX: MAD accuracy obtained for two state-of-art algorithms compared with
human observer average accuracy in D-MAD settings. *Note - LBP-SVM [23]
provides random performance of 50% due to nature of algorithm not able to
scale up to the images in newly constructed HOMID dataset. BPCER is not
reported in cases where there were not enough samples used for human observer
analysis.

####  V-2 Results for Experiment-II - S-MAD

We  compare the typical human observer accuracy  to two cutting-edge S-MAD
methods. We choose Hybrid features [28] and Ensemble features [43] for
detecting morphing attacks based on the performance obtained in  the NIST FRVT
MORPH challenge [7] with the best performance in detecting  printed-and-
scanned morph images.  The Ensemble features [43] combine textural features
with a collection of classifiers, in contrast to the Hybrid features [28]
which mix scale space, color space, and numerous classifiers. We present the
accuracy in Table X . for two chosen algorithms and average human observer
accuracy. We note that both algorithms perform better than the human observers
in S-MAD for both chosen  algorithms indicating the challenging nature of
detecting morphs when no reference image is available.  To further demonstrate
incorrect classifications in genuine and morph attacks in an S-MAD
environment, violin plots for APCER and BPCER are in Figure 16.

Data | Approach | Accuracy | APCER | BPCER
---|---|---|---|---
Digital |  Ensemble Features [43] | 70.00 | 30.00 | 30.00
Hybrid Features [28] | 73.34 | 3.33 | 76.66
Average Observer Accuracy | 58.17 | 38.40 | 45.31
Print-Scan |  Ensemble Features [43] | 63.34 | 36.66 | 36.66
Hybrid Features [28] | 56.67 | 3.33 | 80.00
Average Observer Accuracy | 53.88 | 32.23 | 60.02
Post-Processed |  Ensemble Features [43] | 70.00 | 26.66 | 33.33
Hybrid Features [28] | 66.67 | 6.66 | 76.66
\- Digital | Average Observer Accuracy | 57.96 | 38.78 | -
Post-Processed |  Ensemble Features [43] | 60.00 | 30.00 | 40.00
Hybrid Features [28] | 56.67 | 3.33 | 76.66
\- Print-scan | Average Observer Accuracy | 54.23 | 31.51 | -

TABLE X: MAD accuracy obtained for two state-of-art algorithms compared with
human observer average accuracy in S-MAD settings. *Note - BPCER is not
reported in cases where there were not enough samples used for human observer
analysis.

(a) APCER for S-MAD experiments

(b) BPCER for S-MAD experiments

Figure 16: APCER and BPCER for S-MAD

##  VI Summary of key findings and their implications

Based on the analysis presented in previous sections, we note a few major
observations in this section and discuss their implications:

  * •

Availability of reference image: Generally speaking, S-MAD settings  have a
lower detection accuracy than D-MAD settings. It can be deduced that the
observers make a better decision when a reference image is available.

  * •

Impact of prior training: When examining the average accuracy, both in S-MAD
and D-MAD scenarios, observers with prior training in face comparison tend to
do better than observers  with prior experience in document examination Fig 8
and Fig 9). An implication of this analysis directly suggests that certain
elements of MAD in facial comparison training may be beneficial to include.

  * •

Impact of participation: Through analysis, we  find that just by taking part
in the experiments, the observers become better at detecting the morphed
images. Specifically, we note the improvement in accuracy throughout the
experiment and after seeing  several examples (Figure 13). This  supports our
earlier finding that specialized training programs and  teaching observers
with varied instances will increase detection accuracy. This observation is
also in line with research done on the effectiveness of courses in facial
comparisons, where the researchers suggest that examination-based training
could be an important part of a training  program [44]. To find out if this
also applies to MAD, more investigation is necessary.

We thus answer the initial questions posed in this article:

  * •

How good are ID document examiners at detecting morphing attacks?

    * –

The performance of identity document examiners varies depending on their field
of employment and they are nonetheless susceptible to errors in the detection
of morphing attacks (refer Figure 10b and Figure 10a).

  * •

Are people with certain types of training better than others at detecting
morphing attacks?

    * –

The accuracy of various groups of observers indicate that some methods of
training were more effective for MAD than others.

  * •

Does long experience working in a certain field (for instance, facial
examination in ID control) positively impact performance?

    * –

We note an indicative correlation between detection accuracy and experience in
the line of work. Face comparison experts perform better in identifying morphs
that may indicate a certain favorable influence of prior experience than
observers in other fields of work.

  * •

Are expert observers with training or experience in checking identity/identity
documents better than those without training?

    * –

The observers with no training can perform as well as human experts with
training. However, it should be noted that the average error rate of those who
have received training is significantly lower than that of those who have not
(for either document training or facial examination), indicating that those
who have not received any training or experience in.

  * •

How do MAD algorithms perform compared to human observers?

    * –

The algorithms perform better in both S-MAD and D-MAD setting as compared to
human observers indicating a gap in the ability to detect morphs as
efficiently as algorithms (refer Table IX and Table X).

###  VI-A Future Directions

Understanding the improvement in detection accuracy when observers with
varying levels of competence select a particular image is one of the
prospective avenues for subsequent research. For instance, it would be
interesting to understand if border guards and expert facial examiners
consider different facial properties before deciding on the image as border
guards often have some document training in many countries.  Designing better
algorithms and including the factors that people consider when choosing the
best course of action could result in a greater understanding of the decision-
making process. Furthermore, obtaining the confidence of observers in the
decision-making process can be used to fully understand what experts observe
in an image to deem it as a morphed image.

A question that needs to be answered in the future is if ID examiners with
training in MAD perform better than people with no MAD training.  The impact
of training  may also be measured, which is interesting. The ability of
superrecognizers to recognize morphing attacks intuitively without training is
a crucial issue that needs to be investigated. In the current study, we have
23  superrecognizers, and a dedicated study on  superrecognizers is currently
being conducted whose findings will be reported in the follow-up work.

###  VI-B Potential errors in the study

We  recognize that the study may, in some cases, be prone to error. For
instance, participants may have exerted more effort and spent more time on the
tasks in the experiments than they would have in a real-life scenario  if they
were aware that they were being tested. In contrast, other participants may
have spent less time and made less effort, as it was not a real case they were
working on.

We also  acknowledge that by periodically updating the participants on their
results throughout the experiment, we may have introduced bias. It might have
had an impact on the findings regarding whether exposure to numerous morphed
images improves performance. Given that the experiments would  have been an
additional task on top of their  regular work, fatigue may have also been a
factor and negatively impacted performance.

The time the participants spent on the experiments could also be prone to
errors, as participants may not have clicked on the  “Continue later” button
if they, for example, were interrupted during the experiments.  Additionally,
participants may have experienced stress if they were aware that the time
spent was  being recorded. More research is needed to fully understand these
factors.

##  VII Conclusion

MAD has been extensively studied using automated algorithms, and a very
limited set of works have investigated the detection ability of human
observers in detecting morphs. We compare face images in two different
settings- S-MAD and D-MAD using human observers in everyday professional life
as a benchmark. With the help of 48  different subjects, 400 morphed images
were produced using the Differential-MAD (D-MAD) setting, and  400 probe
images were  taken at the ABC gate to represent a border-crossing scenario. In
addition, a new database of  180 morphed images was developed under the Single
Image - MAD (S-MAD) setting. The benchmark  analyzed with 469 observers for
D-MAD and 410 observers for S-MAD from more than 40 countries indicates the
challenging nature of morphed image detection.  According to the analysis,
close to 30% of morphed images  were missed by expert observers due to a lack
of competence. Numerous subanalyses also show that different observers from
various fields of expertise have varying degrees of proficiency in spotting
morphing attacks. The analysis also  shows early signs of enhanced morphing
detection if the observers  receive specialized morphing detection training.

## References

  * [1] M. Ferrara, A. Franco, and D. Maltoni, “The magic passport,” in _IEEE International Joint Conference on Biometrics_. IEEE, 2014, pp. 1–7.
  * [2] S. Venkatesh, R. Ramachandra, K. Raja, and C. Busch, “Face morphing attack generation & detection: A comprehensive survey,” _IEEE Transactions on Technology and Society_ , 2021.
  * [3] K. Raja, M. Ferrara, A. Franco, L. Spreeuwers, I. Batskos, F. De Wit, M. Gomez-Barrero, U. Scherhag, D. Fischer, S. Venkatesh _et al._ , “Morphing attack detection-database, evaluation platform and benchmarking,” _IEEE Transactions on Information Forensics and Security_ , 2020.
  * [4] D. J. Robertson, A. Mungall, D. G. Watson, K. A. Wade, S. J. Nightingale, and S. Butler, “Detecting morphed passport photos: a training and individual differences approach,” _Cognitive research: principles and implications_ , vol. 3, no. 1, pp. 1–11, 2018.
  * [5] A. Makrushin, T. Neubert, and J. Dittmann, “Automatic generation and detection of visually faultless facial morphs,” in _International Conference on Computer Vision Theory and Applications_ , vol. 7. SCITEPRESS, 2017, pp. 39–50.
  * [6] P. J. Phillips, P. J. Flynn, T. Scruggs, K. W. Bowyer, J. Chang, K. Hoffman, J. Marques, J. Min, and W. Worek, “Overview of the face recognition grand challenge,” in _2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05)_ , vol. 1. IEEE, 2005, pp. 947–954.
  * [7] NIST. (2020) Frvt morph web site. [Online]. Available: https://pages.nist.gov/frvt/html/frvt_morph.html
  * [8] D. J. Robertson, R. S. Kramer, and A. M. Burton, “Fraudulent id using face morphs: Experiments on human and automatic recognition,” _PLoS One_ , vol. 12, no. 3, p. e0173319, 2017.
  * [9] R. S. Kramer, M. O. Mireku, T. R. Flack, and K. L. Ritchie, “Face morphing attacks: Investigating detection with humans and computers,” _Cognitive research: principles and implications_ , vol. 4, no. 1, pp. 1–15, 2019.
  * [10] A. Makrushin, D. Siegel, and J. Dittmann, “Simulation of border control in an ongoing web-based experiment for estimating morphing detection performance of humans,” in _Proceedings of the 2020 ACM Workshop on Information Hiding and Multimedia Security_ , 2020, pp. 91–96.
  * [11] H. Zhang, S. Venkatesh, R. Ramachandra, K. Raja, N. Damer, and C. Busch, “Mipgan—generating strong and high quality morphing attacks using identity prior driven gan,” _IEEE Transactions on Biometrics, Behavior, and Identity Science_ , vol. 3, no. 3, pp. 365–383, 2021.
  * [12] S. J. Nightingale, S. Agarwal, and H. Farid, “Perceptual and computational detection of face morphing,” _Journal of Vision_ , vol. 21, no. 3, pp. 4–4, 2021.
  * [13] A. M. Burton, D. White, and A. McNeill, “The glasgow face matching test,” _Behavior research methods_ , vol. 42, no. 1, pp. 286–291, 2010.
  * [14] M. Ferrara, A. Franco, and D. Maltoni, “On the effects of image alterations on face recognition accuracy,” in _Face recognition across the imaging spectrum_. Springer, 2016, pp. 195–222.
  * [15] Gnu image manipulation program. (accessed: 01.08.2020). [Online]. Available: https://www.gimp.org/
  * [16] Gap-2.6 for gimp 2.6-windows. (accessed: 10.08.2020). [Online]. Available: https://www.deviantart.com/photocomix-resources/art/GAP-2-6-for-Gimp-2-6-Windows-135464357
  * [17] “Dermalog identification systems gmbh,” https://www.dermalog.com/products/software/face-recognition/, 2016, accessed: 2020-08-01.
  * [18] D. E. King, “Dlib-ml: A machine learning toolkit,” _The Journal of Machine Learning Research_ , vol. 10, pp. 1755–1758, 2009.
  * [19] M. Ferrara, A. Franco, and D. Maltoni, “Face demorphing,” _IEEE Transactions on Information Forensics and Security_ , vol. 13, no. 4, pp. 1008–1017, 2018.
  * [20] U. Scherhag, C. Rathgeb, J. Merkle, and C. Busch, “Deep face representations for differential morphing attack detection,” _IEEE Transactions on Information Forensics and Security (TIFS)_ , 2020.
  * [21] J. Mohan Singh, R. Ramachandra, K. B. Raja, and C. Busch, “Robust morph-detection at automated border control gate using deep decomposed 3d shape and diffuse reflectance,” in _2019 15th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS)_. IEEE, 2019.
  * [22] N. Damer, V. Boller, Y. Wainakh, F. Boutros, P. Terhörst, A. Braun, and A. Kuijper, “Detecting face morphing attacks by analyzing the directed distances of facial landmarks shifts,” in _German Conference on Pattern Recognition_. Springer, 2018, pp. 518–534.
  * [23] U. Scherhag, L. Debiasi, C. Rathgeb, C. Busch, and A. Uhl, “Detection of face morphing attacks based on prnu analysis,” _IEEE Transactions on Biometrics, Behavior, and Identity Science_ , vol. 1, no. 4, pp. 302–317, 2019\.
  * [24] R. Raghavendra, S. Venkatesh, K. Raja, and C. Busch, “Detecting face morphing attacks with collaborative representation of steerable features,” in _IAPR International Conference on Computer Vision & Image Processing (CVIP-2018)_, 2018, pp. 1–7.
  * [25] R. Raghavendra, K. Raja, S. Venkatesh, and C. Busch, “Face morphing versus face averaging: Vulnerability and detection,” in _2017 IEEE International Joint Conference on Biometrics (IJCB)_. IEEE, 2017, pp. 555–563.
  * [26] U. Scherhag, R. Raghavendra, K. B. Raja, M. Gomez-Barrero, C. Rathgeb, and C. Busch, “On the vulnerability of face recognition systems towards morphed face attacks,” in _2017 5th International Workshop on Biometrics and Forensics (IWBF)_. IEEE, 2017, pp. 1–6.
  * [27] L. Spreeuwers, M. Schils, and R. Veldhuis, “Towards robust evaluation of face morphing detection,” in _2018 26th European Signal Processing Conference (EUSIPCO)_. IEEE, 2018, pp. 1027–1031.
  * [28] R. Ramachandra, S. Venkatesh, K. Raja, and C. Busch, “Towards making morphing attack detection robust using hybrid scale-space colour texture features,” in _5th International Conference on Identity, Security, and Behavior Analysis (ISBA)_. IEEE, 2019, pp. 1–8.
  * [29] R. Raghavendra, K. B. Raja, S. Venkatesh, and C. Busch, “Transferable deep-cnn features for detecting digital and print-scanned morphed face images,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)_. IEEE, 2017, pp. 1822–1830.
  * [30] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _Proc ICLR. San Diego, CA, USA_ , 2015.
  * [31] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [32] M. Ferrara, A. Franco, and D. Maltoni, “Face morphing detection in the presence of printing/scanning and heterogeneous image sources,” _arXiv preprint arXiv:1901.08811_ , 2019.
  * [33] L. Debiasi, U. Scherhag, C. Rathgeb, A. Uhl, and C. Busch, “Prnu-based detection of morphed face images,” in _2018 International Workshop on Biometrics and Forensics (IWBF)_. IEEE, 2018, pp. 1–7.
  * [34] S. Venkatesh, R. Ramachandra, K. Raja, L. Spreeuwers, R. Veldhuis, and C. Busch, “Morphed face detection based on deep color residual noise,” in _Ninth International Conference on Image Processing Theory, Tools and Applications (IPTA)_. IEEE, 2019, pp. 1–6.
  * [35] M. Ferrara, A. Franco, and D. Maltoni, “On the effects of image alterations on face recognition accuracy,” in _Face Recognition Across the Imaging Spectrum_ , T. Bourlai, Ed. Springer International Publishing, 2016, ch. 9, pp. 195–222.
  * [36] M. Brysbaert, “How many participants do we have to include in properly powered experiments? a tutorial of power analysis with reference tables.” _Journal of cognition_ , 2019.
  * [37] International Civil Aviation Organization, “Machine readable passports – part 9 – deployment of biometric identification and electronic storage of data in eMRTDs,” http://www.icao.int/publications/Documents/9303_p9_cons_en.pdf, International Civil Aviation Organization (ICAO), 2015.
  * [38] Expression premium xp-7100 small-in-one printer. (accessed: 10.06.2020). [Online]. Available: https://epson.com/For-Home/Printers/Inkjet/Expression-Premium-XP-7100-Small-in-One-Printer/p/C11CH03201
  * [39] C. Bako, “Jpeg 2000 image compression,” _BROADCAST ENGINEERING-INDIANAPOLIS THEN OVERLAND PARK KANSAS-_ , vol. 48, no. 2, p. 77, 2006\.
  * [40] W. H. Kruskal and W. A. Wallis, “Use of ranks in one-criterion variance analysis,” _Journal of the American statistical Association_ , vol. 47, no. 260, pp. 583–621, 1952.
  * [41] H. Levene, “Robust tests for equality of variances,” _Contributions to probability and statistics. Essays in honor of Harold Hotelling_ , pp. 279–292, 1961.
  * [42] W. J. Conover, _Practical nonparametric statistics_. John Wiley & Sons, 1999, vol. 350.
  * [43] S. Venkatesh, R. Raghavendra, K. Raja, and C. Busch, “Single image face morphing attack detection using ensemble of features,” in _23rd International Conference on Information Fusion_ , 2020, pp. 1–5.
  * [44] R. Moreton, C. Havard, A. Strathie, and G. Pike, “An international survey of applied face-matching training courses,” _Forensic Science International_ , p. 110947, 2021.
  * [45] V. Bruce, Z. Henderson, C. Newman, and A. M. Burton, “Matching identities of familiar and unfamiliar faces caught on cctv images.” _Journal of Experimental Psychology: Applied_ , vol. 7, no. 3, p. 207, 2001.
  * [46] A. M. Megreya and A. M. Burton, “Unfamiliar faces are not faces: Evidence from a matching task,” _Memory & cognition_, vol. 34, no. 4, pp. 865–876, 2006\.
  * [47] N. M. Blauch, M. Behrmann, and D. C. Plaut, “Computational insights into human perceptual expertise for familiar and unfamiliar face recognition,” _Cognition_ , p. 104341, 2020.
  * [48] D. White, R. I. Kemp, R. Jenkins, M. Matheson, and A. M. Burton, “Passport officers’ errors in face matching,” _PloS one_ , vol. 9, no. 8, p. e103510, 2014.
  * [49] P. J. Phillips, A. N. Yates, Y. Hu, C. A. Hahn, E. Noyes, K. Jackson, J. G. Cavazos, G. Jeckeln, R. Ranjan, S. Sankaranarayanan _et al._ , “Face recognition accuracy of forensic examiners, superrecognizers, and face recognition algorithms,” _Proceedings of the National Academy of Sciences_ , vol. 115, no. 24, pp. 6171–6176, 2018.
  * [50] A. J. O’Toole, P. J. Phillips, F. Jiang, J. Ayyad, N. Penard, and H. Abdi, “Face recognition algorithms surpass humans matching faces over changes in illumination,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 29, no. 9, pp. 1642–1646, 2007.
  * [51] A. J. O’Toole, P. J. Phillips, and A. Narvekar, “Humans versus algorithms: Comparisons from the face recognition vendor test 2006,” in _2008 8th IEEE International Conference on Automatic Face & Gesture Recognition_. IEEE, 2008, pp. 1–6.
  * [52] J. L. Yaros, D. A. Salama, D. Delisle, M. S. Larson, B. A. Miranda, and M. A. Yassa, “A memory computational basis for the other-race effect,” _Scientific reports_ , vol. 9, no. 1, pp. 1–11, 2019.
  * [53] S. Venkatesh, R. Ramachandra, K. Raja, L. J. Spreeuwers, R. Veldhuis, and C. Bush, “Detecting morphed face attacks using residual noise from deep multi-scale context aggregation network,” in _WACV 2020_. TechRxiv, 2020.

## Supplementary Material
Analyzing Human Observer Ability in Morphing Attack Detection - Where Do We
Stand?
Sankini Rancha Godage‡, Frøy Løvåsdal†, Sushma Venkatesh‡,
Kiran Raja‡, Raghavendra Ramachandra‡, Christoph Busch‡
‡Norwegian University of Science and Technology (NTNU), Norway
†The Norwegian Police Directorate, Norway

##  Appendix A Related works on Human observers for general face recognition

The vast amount of recent research in the field of face comparison
demonstrates that people make mistakes when comparing faces that are
unfamiliar to them [45, 46]. Identifying differences in unfamiliar faces is a
difficult task for humans [47]. White et al. [48] found that the performance
of human experts trained in face identification (passport officers, ID card
checkers) and human volunteers with no prior experience (student volunteers)
in face verification had similar error rates 444referred as matching in the
original article.

Thirty passport officers took part in the study (21 Females, Mean age = 48.0,
SD = 11.7) where an average of 10% error was reported. Additionally, 14% of
nonmated photos were incorrectly accepted while 6% of valid photos were
incorrectly rejected. The study also discovered no connection between the
proficiency of the face recognition test and the experience of the observers.
In the second test conducted after two years, the same subject group was
requested to scan their current passport picture and a new facial image (taken
under a better-illuminated condition and improved camera quality). The authors
compared the recently captured image with the subjects’ most recent passport
photos using the newly captured image as the target image. Each participant
was asked to evaluate the image against every possible pair combination.
Eighty-four trials of images were shown to the observers and overall accuracy
was shown to be 70.9% (89.4% accuracy on nonmated comparison trials). The
research found that comparing unfamiliar faces is a human task that is prone
to error (whether trained or not). The study also discovered that making
decisions are not improved by presenting an outdated reference photo
identification document.

Phillips et al. [49] examined the performance of the human observers and
algorithms in a National Institute of Standards and Technology(NIST) project
[50, 51]. These test results showed that both humans and machines could
accurately detect faces. To analyze the experimental data Phillips et al.[49]
proposed the Cross-Modal Performance Analysis framework, which was adapted
from neuroscience techniques. They used both static images and video imagery
in this experiment. In experimental trials, two static face images or videos
were on the digital screen in each trial. They compared two images captured
under different conditions like a studio environment, ambient lighting,
different hairstyle, and clothes. The observers were given five options “(i)
Are you sure they are the same person? (ii) Not sure they are the same. (iii)
Do not know (iv) Different person (v) Not sure they are different persons.” To
analyze the data, the authors used the Receiver Operating Characteristic Curve
(ROC). According to O’Toole et al. [50]’s recommendations, all images and
videos were only shown for 2 seconds. Humans outperform algorithms when faced
with challenging static faces and video, according to research by O’Toole et
al. [50].

##  Appendix B Quality of morphed images

We investigate whether there are any discernible differences between the
reference images and the ABC gate images in two different contexts: (1)
digital morphed images without postprocessing and with postprocessing compared
against digital bona fide images, (2) postprocessed and printed-scanned
morphed images compared against ABC gate images. Through the Kruskal-Wallis
test, we note in the former case we obtain a p=.502, and in the latter, we
note p=.806 both suggesting that observers do not have any advantage when the
images are not postprocessed. This illustrates the difficulty of detecting
morphs in the digital or print-scan domain while also demonstrating how well-
made the morphs produced in this work are at deceiving a human observer.

##  Appendix C Illustration of one subject morphed against all subjects

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

(m)

(n)

(o)

Figure C.1: Example of one subject morphed with various subjects in the
database with a morphing factor $0.5$.

##  Appendix D Additional Analysis - MAD Accuracy v/s Total Time Spent

We also examine the data to see if there is any correlation between the amount
of time taken to make decisions and accuracy because we are intrigued by the
variations in accuracy that observers exhibit when spotting the morphs. We
present the results of total time spent for D-MAD and S-MAD settings as
provided in Figure D.1. We make the following observations from this set of
analyses:

  * •

As it appears, there is no clear correlation between the time spent on
experiments versus the accuracy obtained. Several observers who finished the
experiments in D-MAD settings in less than 90 minutes attained an accuracy of
over 90%. In contrast, some observers who spent less than 90 minutes on the
task had lower accuracy (less than 50%) for 400 image pairs when using the
D-MAD setting.

  * •

While the accuracy of S-MAD is generally lower, the observers have completed
the experiments in less than 90 minutes and have obtained an average accuracy
of 65%.

  * •

A small number of observers took a longer time to complete the experiments
because the experiment allowed them to do so over a period of time; these
observers are spread out after 180 minutes.

  * •

When closely examining the graphs, it can be concluded that there does not
seem to be a direct link between the amount of time spent and the level of
accuracy attained.

(a) D-MAD

(b) S-MAD

Figure D.1: MAD accuracy versus the time spent for completing the experiment.
(* Illustrated up to 300 minutes for the sake of simplicity)

##  Appendix E Additional Analysis - MAD accuracy v/s Gender

(a) D-MAD Accuracy

(b) S-MAD Accuracy

Figure E.1: Gender v/s MAD accuracy of human observers

##  Appendix F Additional Analysis - MAD accuracy for observers in various
organizations

(a) D-MAD Accuracy

(b) S-MAD Accuracy

Figure F.1: MAD accuracy of human observers in various organizations
(*Organization names categorized according to nature of work and presented in
anonymized form)

##  Appendix G Additional Analysis - Highest detection of morphs after post-
processing in S-MAD setting

Figure G.1: Illustration of subjects whose post-processed morphed images
obtain highest detection accuracy in S-MAD settings. The average detection
accuracy is noted below each pair of images (D denotes digital and P denotes
printed-scanned).

##  Appendix H Additional Analysis - Lowest detection of bona fide in S-MAD
setting

The observers tend to wrongly classify the bona fide images as morph images in
S-MAD setting when the images are printed and scanned as noted in Figure H.1.
As noted from the accuracy, the average classification error increases when
the images are printed and scanned in S-MAD settings, with an minimum increase
of 10%.

Figure H.1: Illustration of bona fide images obtaining the lowest detection
accuracy in S-MAD settings. The average detection accuracy is noted below each
pair of images (D denotes digital and P denotes printed-scanned). As noted
from the accuracy, the average classification error increases when the images
are printed and scanned in S-MAD settings.

##  Appendix I Analysis of morphed images with lowest detection accuracy

In this section, we analyze some sample images that obtained the lowest
detecting accuracy in both D-MAD and S-MAD settings to complement the
quantitative analysis.

###  I-A Additional Analysis - Analysis of morphed images with lowest
detection accuracy in D-MAD

Figure I.1a displays the detection accuracy for probe images obtained from
standard cameras, while Figure I.1b displays the detection accuracy for probe
images obtained from an ABC gate. In both cases, digital images are displayed
in the top row, while printed-scanned images are displayed in the bottom row.
We note a small variation in average accuracy, but the images indicate the
detection accuracy does not drastically vary across observers for some sets of
images. The detection accuracy for probe images obtained from conventional
cameras is shown in Figure I.1a, whereas the detection accuracy for probe
images obtained from an ABC gate is shown in Figure I.1b. In both instances,
printed-scanned images are shown in the bottom row, while digital images are
shown in the top row. While it is difficult to conclude from this observation,
we would like to draw the readers’ attention to the fact that the large
observer pool was made up of Caucasians, which may have been influenced by the
“other-race effect” [52], in which observers had lower accuracy on morphed
images of Asian ethnicity. However, an indicative conclusion highlights the
need for additional research on examining the relationship between ethnicity
and MAD, which is outside the purview of this article.

(a) Regular probe images. The top row are images in digital domain and the
bottom row presents the images after printing and scanning.

(b) Probe images from ABC Gate. The top row are images in the digital domain
and the bottom row presents the images after printing and scanning.

Figure I.1: Illustration of post-processed morphed images obtaining lowest
detection accuracy when compared against probe images and probe images from an
ABC Gate. The average detection accuracy is noted below each pair of images.

###  I-B Additional Analysis - Analysis of morphed images with lowest
detection accuracy in S-MAD

We further analyze the images that obtained the lowest detection accuracy and
present the same in Figure I.2. We observe that a sizable portion of the
images with the lowest accuracy of detection are not Caucasian. The low
accuracy could be potentially due to the “other-race effect” as a significant
amount of observers belong to European countries. Although this is one
possibility, we can also interpret the same fact as observers who are
accustomed to working with a particular population daily. Further research
into this aspect is required. As a second observation, we note that detection
of morphed images in the digital domain is much lower than morphed images when
printed and scanned. While algorithms are effective at detecting images in the
digital domain [53, 2, 3], humans perform significantly better in an S-MAD
setting with printed-scanned morphed images. Although the detection accuracy
difference between digital and printed-scanned images ranges from 10% to 1%,
it clearly shows the importance of understanding the difference between humans
and algorithms.

Figure I.2: Illustration of post-processed morphed images obtaining lowest
detection accuracy in S-MAD settings. The average detection accuracy is noted
below each pair of images (D denotes digital and P denotes printed-scanned).