# Biometric Template Protection for Neural-Network-based Face Recognition
Systems:
A Survey of Methods and Evaluation Techniques

Vedrana Krivokuća Hahn and Sébastien Marcel
Idiap Research Institute (Martigny, Switzerland)

###### Abstract

As automated face recognition applications tend towards ubiquity, there is a
growing need to secure the sensitive face data used within these systems. This
paper presents a survey of biometric template protection (BTP) methods
proposed for securing face “templates” (images or representative features) in
neural-network-based face recognition systems. The BTP methods are categorised
into two types: Non-NN and NN-learned. Non-NN methods use a neural network
(NN) as a feature extractor, but the BTP part is based on a non-NN algorithm
applied at either image-level or feature-level. In contrast, NN-learned
methods specifically employ a NN to learn a protected template from the
unprotected face image or features. We present examples of Non-NN and NN-
learned face BTP methods from the literature, along with a discussion of the
two categories’ comparative strengths and weaknesses. We also investigate the
techniques used to evaluate these BTP methods, in terms of the three most
common BTP criteria: “recognition accuracy”, “irreversibility”, and
“renewability/unlinkability”. As expected, the recognition accuracy of
protected face recognition systems is generally evaluated using the same
(empirical) techniques employed for evaluating standard (unprotected)
biometric systems. On the contrary, most irreversibility and
renewability/unlinkability evaluations are found to be based on theoretical
assumptions/estimates or verbal implications, with a lack of empirical
validation in a practical face recognition context. We recommend, therefore, a
greater focus on empirical evaluation strategies, to provide more concrete
insights into the irreversibility and renewability/unlinkability of face BTP
methods in practice. Additionally, an exploration of the reproducibility of
the studied BTP works, in terms of the public availability of their
implementation code and evaluation datasets/procedures, suggests that it would
currently be difficult for the BTP community to faithfully replicate (and thus
validate) most of the reported findings. So, we advocate for a push towards
reproducibility, in the hope of furthering our understanding of the face BTP
research field.

###### Index Terms:

biometrics, biometric template protection, face recognition, face template
protection, privacy, security, survey.

##  I Introduction

Automated face recognition is fast becoming a default solution for identity
management in our increasingly interconnected society. Applications range from
single-person access control (e.g., phone/computer login) to large-scale
operations that process thousands of people on a daily basis (e.g., passport
control at an airport). The widespread adoption of this technology is due to
its ability to simultaneously provide a high level of security and convenience
to its users. Nevertheless, it is important to consider how our privacy may be
impacted by the collection and use of our sensitive face data by these
numerous face recognition applications.

Modern face recognition systems are commonly based on deep learning
architectures, which train a multi-layer neural network to learn a set of
representative face features from images of a person’s face. Recently, it has
been shown that these features can be “inverted” to recover an approximation
of the original face image [1, 2, 3], and that they can reveal additional
information about the underlying person (e.g., gender, race, age) [4, 5].
These findings indicate that the face features used by deep-learning-based
face recognition systems are rich with personally identifiable information,
which may represent a threat to the privacy of the systems’ users if their
face features are leaked to an adversary. This is especially problematic
considering frameworks like the EU General Data Protection Regulation
(GDPR)111https://bit.ly/2VGSa5e, which imposes a legal obligation to handle
biometric data carefully to protect individuals’ digital identities. So, the
issue of how to protect our face data in automated face recognition systems is
becoming pressing in response to these systems’ increasingly widespread
adoption.

Fortunately, the past decade has witnessed an enrichment of the Biometric
Template Protection (BTP) research field, which strives to develop effective
ways of protecting our biometric “templates” (images or features). This paper
conducts a survey of face template protection methods for neural-network-based
face recognition systems. In particular, we present a categorisation of the
methods that have been proposed for face template protection thus far, and we
investigate the techniques used to evaluate these methods’ abilities to
satisfy the three most important criteria of BTP
schemes222https://bit.ly/398Az9k: “recognition accuracy”, “irreversibility”,
and “renewability/unlinkability”. We also shed light on the reproducibility of
these works by exploring the public availability of the method implementations
and the datasets/procedures used to evaluate them. It is our hope that this
survey will help to clarify the state of the face BTP research field, thereby
motivating further advancements towards face recognition systems that are
better able to protect the privacy of their users. To the best of our
knowledge, no survey paper of this nature currently exists in the literature.

The remainder of this paper is structured as follows. Section II proposes a
classification of the existing face BTP methods into two types, presents
examples from the literature of each type of method, and discusses the
strengths and weaknesses of the two categories of approaches. Section III
investigates the techniques used to evaluate the methods from Section II with
respect to the three aforementioned criteria, beginning with a consideration
of the reproducibility of these works in terms of their public availability.
Finally, Section IV concludes this survey with a summary of our main findings.

##  II Face BTP Methods

This section considers the types of methods proposed to protect face data in
automated face recognition systems. Face data refers to either the face image
or a set of representative features extracted from the image, depending on
whether the protection mechanism is applied at image-level or feature-level.
To comply with the common terminology, we shall henceforth refer to these
protection methods as Biometric Template Protection (BTP) methods, where the
word template is synonymous with the aforementioned face data.

In the context of this survey, we are only interested in BTP methods proposed
for face recognition systems that incorporate a Neural Network (NN) (or, more
commonly, a Deep Neural Network (DNN)) into their pipeline. Based on our
perusal of the relevant literature, we propose a categorisation of the face
BTP approaches into two main types, according to what role the NN plays in the
face recognition system’s pipeline: “Non-NN” and “NN-learned”.

Non-NN BTP methods do not use a neural network to define the BTP algorithm;
rather, the NN serves simply as a feature extractor. The NN can be used as
either a pre-BTP or post-BTP feature extractor. In the pre-BTP case, the NN
would first be used to extract a set of face features (i.e., a feature vector)
from the face image, then the BTP algorithm would be applied to this feature
vector (i.e., at feature-level) to generate the protected face template. In
the post-BTP case, the BTP algorithm would be applied directly to the face
image (i.e., at image-level), following which the NN would be used to extract
features from the protected image. Regardless of the NN’s precise positioning
as a feature extractor, the important point is that the NN does not form part
of the BTP method itself; instead, Non-NN BTP methods are defined in terms of
either handcrafted or non-NN machine-learning algorithms.

NN-learned BTP methods, on the other hand, do use a neural network to express
the BTP algorithm. More specifically, a NN (usually a DNN) is used to learn
the protected template either directly from its corresponding face image or
from the extracted feature vector. In the former case, where the BTP is
applied at image-level, the NN would be trained in an end-to-end manner (i.e.,
image $\rightarrow$ protected template). In the latter case, where the BTP is
applied at feature-level, the training would start from the extracted face
features (i.e., feature vector $\rightarrow$ protected face template). The
feature extraction process could be conducted either using a separate NN (as
for Non-NN BTP methods) or a handcrafted/machine-learned (non-NN) feature
extraction algorithm. Regardless of how the face features are extracted or the
stage at which the BTP method is applied, the important point is that NN-
learned BTP methods are not explicitly formulated; instead, we rely on a NN to
learn the BTP algorithm that will transform a face template (image or feature
vector) to its protected template.

Fig. 1 illustrates the Non-NN and NN-learned BTP method types, sub-divided
into “feature-level” and “image-level” depending on the stage at which the BTP
method is applied. Note that all outputs in Fig. 1 are the “protected face
template”; however, when the protected template comes from a NN, the output
could alternatively be a classification of the template into its corresponding
identity, instead of the template itself. As this may be considered an
implementation detail (depending on what the NN was trained to output), and
since the main aim of any BTP method is to generate a protected template from
the unprotected template, Fig. 1 has been simplified to always output the
“protected face template”.

Figure 1: The Non-NN and NN-learned face BTP method types. Each type of BTP
method can be applied at either feature-level or image-level.

Sections II.A and II.B present examples from the literature of Non-NN and NN-
learned face BTP methods, respectively.

###  II-A Non-NN Face BTP Methods

Examples of Non-NN face BTP methods in the literature, include: [6, 7, 8, 9,
10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28,
29, 30, 31]. The overwhelming majority of these works ([6, 7, 8, 9, 10, 11,
12, 13, 14, 15, 16, 17, 18, 19, 20, 23, 24, 25, 26, 27, 28, 29, 30, 31]) focus
on applying a BTP method at feature-level (as per Fig. 1), while only a couple
of publications ([21, 22]) consider image-level BTP methods. This preference
for feature-level BTP may be attributed to the availability of several pre-
trained face recognition DNN models, which have been shown to be capable of
extracting highly discriminative face features from face images. Consequently,
the task of applying a protection mechanism on top of robust face features may
be more appealing than the potentially more challenging task of designing a
good image-level BTP method plus a robust feature extractor for the protected
images – since features extracted from protected images would likely lie in a
different domain than features extracted from unprotected images, it is not
clear whether existing feature extraction algorithms would be applicable to
protected face images.

Of the Non-NN BTP methods applied at feature-level, one of the most commonly
studied protection algorithms is homomorphic encryption (HE), which was
proposed in [6, 7, 8, 9, 10, 11, 12] for the protection of face features that
have been extracted from face images using pre-trained DNN models. Unlike
traditional encryption techniques, HE allows us to perform operations on
encrypted data without having to first decrypt it. In the context of face BTP,
HE would allow us to compare reference and probe encrypted templates
(generated during the enrollment and authentication stages, respectively)
directly in the protected domain. This is in contrast to traditional
encryption methods, which would necessitate comparison in the decrypted (i.e.,
unprotected) domain. Another important advantage of using HE for BTP is that,
theoretically, there should be no loss in the resulting recognition accuracy,
since the comparison score obtained in the encrypted domain should be the same
as that obtained in the original (unprotected) domain. Although these
important benefits of HE have long been known, its computational complexity
has rendered it impractical for application in face BTP until fairly recently.
Current work in this area tends to focus on finding a balance between speeding
up HE operations (e.g., by quantising/binarising the face templates or
reducing their dimensionality) while simultaneously minimising losses in the
resulting recognition accuracy (e.g., by trying to encrypt face templates in
their original – usually floating-point – domain). Although [6, 7, 8, 9, 10,
11, 12] demonstrate important advances towards these goals for HE applied to
both face verification (1-to-1 matching) [6, 7, 9] and identification (1-to-N
matching) [8, 10, 11, 12], the greatest disadvantage of HE is that the
encrypted templates remain secure only insofar as the corresponding decryption
key remains secret; if an adversary gains access to this secret key, they can
completely reverse the protection algorithm to obtain the original
(unencrypted) template.

The invertible nature of encryption algorithms makes them less attractive for
securing sensitive data like face templates, compared to non-invertible
operations like one-way cryptographic hash functions. A cryptographic hash
function creates a fixed-size, predictable output called a “hash” from each
instance of the same input data (e.g., a password), such that it is
mathematically impossible to recover the original input from its computed
hash. Despite the appeal of this “one-wayness” property, it is difficult to
apply cryptographic hash functions to face templates due to the intra-class
variability that is inherent in biometric measurements. This is because hash
functions are purposely designed to exaggerate small differences in the input,
such that even a tiny change would result in a completely different output
hash; however, different instances of the same person’s face template, such as
would be acquired during enrollment and each authentication attempt, will
never be identical, so comparing reference and probe face templates in the
hashed domain would be practically impossible.

Considering the sensitivity of cryptographic hash functions to small changes
in the input, face BTP methods that employ cryptographic hashing often avoid
creating hashes from the face templates themselves. Instead, they apply the
hashing algorithm to randomly generated, external codewords, which are related
to the face templates by some sort of mathematical function or mapping. A
well-known example of such a method is the Fuzzy Commitment scheme [32],
whereby a random, user-specific codeword is “bound” with the user’s biometric
template by calculating the difference between the two entities (i.e.,
$\texttt{binding}=\texttt{template}-\texttt{codeword}$). The protected
template, generated during enrollment, then consists of this binding plus a
cryptographic hash of the chosen codeword. During authentication, the aim is
to “unbind” the protected reference template to recover this codeword (i.e.,
$\texttt{recovered codeword}=\texttt{probe template}-\texttt{reference
binding}$). This should be possible provided that the probe template is close
enough to the reference template (to which the codeword is bound). In this
case, an appropriate error-correction mechanism should be able to correct a
sufficient number of errors in the recovered codeword, such that the hash of
the corrected codeword matches the hash of the reference codeword. Variants of
the Fuzzy Commitment scheme were investigated for the protection of NN-
extracted face features in [14, 16, 15].

Another well-known “binding” type of BTP method is the Fuzzy Vault scheme
[33], where the randomly chosen, user-specific codeword serves as a set of
coefficients defining a secret polynomial, and the elements of the user’s
biometric template are the polynomial’s inputs. The outputs resulting from the
polynomial mapping (i.e., “true points”), plus a set of “chaff points” added
to hide the true points, constitute the protected template (i.e., fuzzy
vault), along with a cryptographic hash of the random codeword (i.e.,
polynomial coefficients). During authentication, the probe biometric template
is used to determine the true points and thereby recover the secret polynomial
from the reference fuzzy vault, which will be successful if the cryptographic
hash of the recovered set of polynomial coefficients matches the hash of the
set of reference coefficients. The Fuzzy Vault scheme was proposed as a Non-NN
face BTP method in [17].

Other examples of Non-NN face BTP methods that cryptographically hash a
random, user-specific codeword instead of the face template itself, include
[24, 28]. In [24], the mapping between a face template and a pre-defined
random codeword is established via a Locality Sensitive Hashing (LSH)
algorithm, while [28] proposes defining this mapping in terms of a linear
transformation represented by an orthogonal matrix. In both cases, as for the
Fuzzy Commitment and Fuzzy Vault schemes, successful authentication requires
that the correct user-specific codeword be determined, such that its
cryptographic hash matches the reference hash stored during enrollment.

The only Non-NN face BTP method that proposes applying a cryptographic hash
function directly to the face template, seems to be [13]. In this approach, a
stacked denoising autoencoder is trained to extract face features from
different local regions of the face image. Each local feature vector is then
quantised and cryptographically hashed, and the set of local hashes
constitutes the protected template. The use of local features minimises the
intra-class variability across different instances of the same person’s face
image and, therefore, increases the likelihood of achieving an exact match
between the local hashes. Nevertheless, exact matches may still be difficult
to ensure in practice; indeed, the high recognition accuracy error rates
reported in [13] are indicative of this issue, which is due to the sensitivity
of cryptographic hash functions to small changes in the input (as mentioned
earlier).

Instead of cryptographically hashing random, external codewords, the face BTP
methods in [18, 23, 26, 20, 25, 27] generate non-cryptographic hashes or codes
from the NN-extracted face features themselves. The idea is to map similar
face templates to (approximately) the same code, thereby achieving a sort of
“hashing” with reduced sensitivity to intra-class variations (compared to
cryptographic hashing). The difference between the BTP methods in [18, 23, 26,
20, 25, 27] lies in the way in which this hashing is performed.

The most well-known non-cryptographic hashing method in the general BTP
literature is BioHashing [34], whereby a biometric template is projected onto
a random subspace (defined by a user-specific matrix) and subsequently
binarised via a thresholding operation. BioHashing is applied to NN-extracted
face features in [18]. Random projection, which is the first step in
BioHashing, is primarily used to add diversity to the resulting hashes, which
further separates different users’ templates in the protected domain. This
separation, in addition to the binarisation operation, helps to deal with
natural variations between different face templates from the same user. Random
projection is also applied as a first step towards creating a face hash in
[23, 27], but the subsequent binarisation of the projected face templates is
achieved using an instance of LSH in [23] and a random decision tree
classifier in [27]. In all three of these BTP methods, the comparison between
reference and probe hashes is based on Hamming distance. This allows us to
tune the decision threshold according to the acceptable amounts of variation
between the same user’s face templates (i.e., intra-class variability) versus
different users’ face templates (i.e., inter-class variability). This is in
contrast to cryptographic hash functions, which would require an exact match
between the reference and probe hashes (i.e., there would be no need for a
decision threshold).

Examples of other BTP methods that propose mapping NN-extracted face features
to a binary code, include [20, 25]. In [20], 100 features are randomly
selected from a higher-dimensional feature vector, and subsets of these
features are mapped to their nearest-neighbour binary codewords (where the
codewords have been pre-determined via vector quantisation on a development
set of features). The binary codewords are then concatenated to generate the
protected face template. During authentication, the reference and probe binary
templates are compared using Hamming distance. The approach towards generating
binary face codes in [25] is quite different. This BTP method begins by
binarising the NN-extracted face feature vector (using its mean), followed by
permuting the resulting binary feature vector m times to create m different
permuted vectors. Two permuted binary vectors are then randomly selected and
XOR-ed to create the “bio-key”. As a final step, the bio-key is encoded into a
Reed-Solomon (error-correcting) codeword. The comparison between reference and
probe codes during authentication then relies on an exact match facilitated by
the codes’ in-built error-correction capabilities, which differs from the
Hamming-distance-based comparisons in [18, 23, 27, 20].

Another example of a non-cryptographic face hashing BTP method was proposed in
[26]. This method differs from [18, 23, 27, 20, 25] in that the resulting face
hash is not binary, but integer-based. The BTP method in [26] applies the
Winner Takes All (WTA) hashing strategy from [35] to NN-extracted face
features. In a nutshell, this involves permuting the feature vector multiple
times, selecting the first k elements of each permuted vector, and identifying
the indices of the maximum value in each set of k elements. Next, the prime
factors of each index are computed as:
$\texttt{PF}=(\texttt{index}+2)\times\texttt{random positive integer}$. The
final code then consists of a set of integers representing the number of prime
factors computed for each permuted face template. During authentication, the
probe code is subtracted from the reference code in an element-wise fashion,
and the fraction of resulting zeros represents the comparison score.

The hashing methods considered thus far have used either cryptographic hash
functions or non-cryptographic hashing techniques. A face BTP method that
combines these two hashing approaches was presented in [29]. More
specifically, [29] proposes a protected face identification system consisting
of two sub-systems. The first sub-system compares the probe face template to
each of the N reference face templates in the database, where all templates
are protected by the non-cryptographic Index-of-Max (IoM) hashing method
proposed in [36] (which is a special instance of LSH). The comparison is based
on Hamming distance, and the resulting scores are sorted to identify the top k
matches, which are passed on to the second sub-system. The second sub-system
then conducts a 1-to-1 comparison between the fuzzy vaults of the protected
probe template and each of the top k protected reference templates returned by
the first sub-system. A reference template is considered to match the probe
template if there is a match between the cryptographic hashes of the
polynomial coefficients extracted from their corresponding fuzzy vaults.

Among the Non-NN face BTP methods considered thus far, hashing appears to be
the most popular approach; however, alternatives are proposed in [19, 30, 31].
In [19], face features extracted from different face regions (using a CNN) are
fused, and the fused feature vector is convolved with a random kernel. A
fusion approach is also adopted in [30], but this time a DNN-extracted face
feature vector is fused with a different person’s face feature vector.
Specifically, a key is generated from each feature vector, then each key is
used to alter the other feature vector via element-wise multiplication,
followed by vector addition of the altered feature vectors. An entirely
different BTP method, called PolyProtect, is proposed in [31]. This method
secures face features (“embeddings”) extracted using pre-trained DNNs, by
mapping them to protected templates via multivariate polynomials parameterised
by random, user-specific coefficients and exponents.

Finally, although the majority of existing Non-NN face BTP methods have been
applied at feature-level, a couple of image-level methods have also been
investigated, for example [21, 22]. In [21], the input face image is modified
via a block scrambling technique, then a CNN is used to classify the protected
image (as a particular identity). In [22], face images are warped using
distortion functions defined by user-specific keys, then the warped images are
presented as inputs to pre-trained face recognition models (or NN feature
extractors). The investigation in [22] actually led to the recommendation that
this technique not be used for face BTP in practice, because for certain
warping parameters the DNN models effectively ignore the warping in the
protected images as a form of intra-class variability. It would be interesting
to extend this investigation to the image scrambling method in [21], although
at this point we may reason that a similar conclusion would be reached.

Table I summarises the studied Non-NN face BTP methods, as per the above
discussion. The methods are categorised into feature-level and image-level
techniques. Feature-level methods are further divided into HE (homomorphic
encryption), hashing (cryptographic and non-cryptographic), and other.

TABLE I: Examples of Non-NN face BTP methods in the literature. Feature-level | Image-level
---|---
HE | Hashing | Other
Crypto. | Non-crypto.
[6, 7, 8, 9, 10, 11, 12] | [14, 16, 15, 17, 13], | [18, 23, 20], |  [19], | [21, 22]
[24, 28, 29] | [25, 26, 27, 29] |  [30],
[31]

###  II-B NN-learned Face BTP Methods

Examples of NN-learned face BTP methods in the literature, include: [37, 38,
39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51]. Recall that methods in
this category use a neural network to learn the BTP algorithm, instead of
explicitly formulating it like Non-NN BTP methods. Furthermore, while the
majority of Non-NN face BTP methods investigated in Section II.A were found to
be feature-level algorithms, such a clear preference for feature-level over
image-level techniques was not observed among the NN-learned face BTP methods
that will be discussed in this section. This is probably because the main
attraction of NN-learned BTP methods lies in the fact that the protection
algorithm need not be explicitly defined, meaning that allowing the NN to
learn the protected templates from images as opposed to features is not
necessarily more challenging (unlike for Non-NN image-level BTP methods, which
may additionally require the design of a separate feature extraction algorithm
for the protected face images).

The NN-learned methods for face template protection in the literature focus on
two main approaches. The first approach involves pre-defining a user-specific,
random binary code, then learning the mapping from the user’s face
image/features to this code via a DNN. A cryptographic hash of the pre-defined
code then serves as the protected face template, and during authentication the
probe and reference codes are compared in the hashed domain. The second
approach involves training a DNN to learn its own representation of a
protected template, without forcing it to conform to a pre-defined code. Since
the learned representation is likely to exhibit some intra-class variability
(i.e., across the same user’s enrollment and various authentication attempts),
cryptographic hashing is usually not applied. It is interesting to note that
these two types of approaches towards NN-learned face BTP are conceptually
similar to the two types of hashing approaches (cryptographic versus non-
cryptographic) discussed for Non-NN BTP methods in Section II.A. The
difference is that the algorithms used to obtain the protected hashes from the
unprotected face templates are learned for NN-learned BTP methods and defined
for Non-NN BTP methods.

Examples of NN-learned face BTP methods that pre-define a representative
binary code (and cryptographically hash it to produce the protected template),
include [37, 38, 39, 40, 41, 42]. Of these, [37, 38, 39] adopt an image-level
approach, whereby the NN is trained in an end-to-end manner to map the input
face images to their corresponding, pre-defined codes. Alternatively, [40, 41,
42] adopt a feature-level approach, where the NN training starts from the
extracted face features as opposed to the raw images. Overall, most of these
methods build upon the approach proposed in [37, 38], which involves assigning
a random maximum-entropy binary code to every user of the face recognition
system, then training a Convolutional Neural Network (CNN) to map each user’s
face image to their corresponding code. The trained CNN should be able to deal
with natural variations across different face images acquired from the same
user, producing the same code during enrollment and each authentication
attempt. The binary codes obtained during enrollment and authentication are
cryptographically hashed, and comparison is based on an exact match between
the two hashes. A similar approach is adopted in [39], except that the user-
specific binary codes are encoded into error-correcting codes (LDPC) during
training, such that the CNN learns the mapping between a user’s face image and
their corresponding LDPC code. During authentication, the LDPC code obtained
for the probe image is decoded to recover the underlying binary code, and the
cryptographic hashes of the probe and reference binary codes are compared.
This approach aims to improve the CNN’s robustness to intra-class variations
with the help of the LDPC codes’ error-correction capabilities.

Another approach towards improving the BTP method in [37, 38] was proposed in
[40]. This work adopts a deeper CNN consisting of two components: a pre-
trained DNN that is used to extract face features from the input face images,
followed by a second NN that maps the extracted face features to the user’s
pre-defined binary code. This BTP method may thus be considered as a feature-
level method, since network training does not involve changes to the pre-
trained feature extraction network. Indeed, the main idea behind this work is
to take advantage of the robust feature extractor to improve the CNN’s ability
to deal with both intra-class and inter-class variations among the input face
images, thereby increasing the recognition accuracy in the protected template
domain.

Further improvements to the BTP approach in [37, 38], also based on feature-
level NN training, were proposed in [41, 42]. Both suggestions for improvement
mainly focus on the addition of extra, user-specific information to
incorporate more complexity into the mapping between a user’s face features
and their pre-defined (pre-hash) code. Specifically, [41] builds on top of
[40], proposing a random projection step following the feature extraction. In
this case, the extracted feature vector is projected to a lower-dimensional
code (where the projection is defined by a random, user-specific matrix),
after which the NN learns to map this projected code to the user’s pre-defined
code. The random projection step is motivated by its potential to: (i)
increase recognition accuracy by removing redundancies in the face feature
vector, (ii) improve security due to the added difficulty in recovering a
user’s feature vector or face image from their protected template, and (iii)
enable the revocability of compromised templates by replacing the user’s
projection matrix. Similar motivations are presented for the BTP method
proposed in [42], except that this time the additional user-specific
information is incorporated through random perturbations (instead of random
projection). Specifically, [42] proposes applying random, user-specific
perturbations to the extracted feature vector, then training the NN to map the
perturbed feature vector to its pre-defined binary code. The perturbations are
stored as a user-specific key, which is used to apply the same perturbations
to the probe feature vector during authentication.

The main advantage of the NN-learned BTP methods that focus on mapping face
images/features to pre-defined codes, is that the resulting protected
templates (i.e., cryptographic hashes of the pre-defined codes) are
fundamentally unrelated to the original (unprotected) face templates. This
means that the protected templates should reveal no information about the face
templates to which they have been assigned. Consequently, if an adversary were
to gain access to the protected templates stored in the face recognition
system’s database, they should be unable to recover the corresponding face
images or features. Of course, this is based on the assumption that access to
the parameters of the trained NN would not help the adversary uncover any
hints on the link between a face template and its hash; however, such an
assumption may not hold in practice, especially if we consider the worst-case
scenario of a fully-informed attacker. The main issue with NN-learned face BTP
methods that rely on pre-defined outputs, is that the NN would need to be re-
trained (either fully or partially) each time a new user wishes to enroll into
the face recognition system or when a compromised user needs to be re-enrolled
with a new protected template. So, the scalability of such methods in practice
is questionable.

To avoid the need to re-train the NN for each new enrollment, and thereby
improve the scalability of the protected face recognition system, it may be
better to use a BTP method that learns its own representation of a protected
face template, as opposed to learning a mapping to a pre-defined
representation. Examples of BTP methods in this category, include: [43, 44,
45, 46, 47, 48, 49, 50, 51]. Of these methods, [43, 44, 50, 51] adopt a
feature-level approach, where the NN learns a protected template from
extracted face features, and [45, 46, 47, 48, 49] adopt an image-level
approach, where the NN is trained in an end-to-end manner to generate the
protected template from the face image. A feature-level BTP method can take
advantage of robust feature extractors (e.g., pre-trained face recognition
models), and it is usually easier to implement/train such methods since the
feature extraction part is already taken care of. An image-level approach, on
the other hand, would allow for more control over the entire learning process,
which could potentially result in more robust protected templates. Although
these observations appear intuitive, at this stage it is unclear whether
feature-level or image-level NN-learned BTP methods would perform better in
practice, especially considering the variety of NN architectures available for
each type of approach.

One of the earliest examples of an image-level NN-learned face BTP method, is
[45]. This method proposes a DNN consisting of two components: a Deep Hashing
(DH) component and a Neural Network Decoder (NND) component. The DH component
is trained to learn an intermediate binary code from the input face image,
after which the NND learns to correct errors in this binary code. The error-
corrected code is then cryptographically hashed to generate the protected
template. Successful authentication relies upon generating exactly the same
hash from the probe face image. Since this type of approach learns its own
code from a face image, as opposed to only learning a mapping to a pre-defined
code (as in [37, 38, 39, 40, 41, 42]), it should technically be possible to
enrol new users without the need to re-train the neural network. The reported
recognition accuracy in this scenario (referred to as zero-shot enrollment),
however, is quite low (i.e., 1-2 orders of magnitude worse, depending on the
dataset) compared to scenarios where the subject has been seen during the
training of the neural network. Furthermore, since the neural network is
trained to learn the same code for every face image from the same user,
revocability and renewability of compromised protected templates does not seem
possible.

The issue of not allowing for template renewability in the event of
compromise, would also be present in [46], which proposes a face BTP method
based on feature fusion. Four different NNs are trained to extract face
features from the input face image, and the four sets of learned face features
are fused via concatenation. The fused feature vector is then passed on to a
fully-connected layer of the “ensemble network”, where it is classified. Since
this method does not require storage of the fused face template, it is claimed
that there is no template to be compromised; however, it is conceivable that
an adversary with access to the trained NN and its parameters could exploit
the classifier to recover a close-enough face image or fused feature vector to
match an enrolled identity. In this case, there would be no way for the
compromised user to re-enroll into the face recognition system with a new face
template.

To allow for the renewability of compromised protected templates, NN-learned
BTP methods must incorporate some external (non-face) information into the
protected template. This way, if the protected template is compromised, it can
be cancelled and replaced by a new protected template generated from the same
user’s face plus different external information. Template renewability was
considered in the design of the face BTP methods proposed in [43, 49, 44, 47,
50, 51, 48]. Most of these methods infuse user-specific randomness into the
process of learning the protected template, in terms of either a salting
approach or a permutation/shuffling strategy.

The salting approaches towards incorporating user-specific randomness into the
learned protected template, include: [43, 49, 44]. In [43], a pre-trained CNN
is used to extract a feature vector from a face image, then the feature vector
plus a serial number are passed as inputs to a Recurrent Neural Network (RNN).
The output of the RNN, corrected via Reed-Solomon error correction, represents
the protected template (referred to as a “key”). The approach in [49] uses a
modified version of a pre-trained face recognition model to extract a face
feature vector, concatenate it with a random key, then pass the result to two
fully-connected layers. The NN is trained using a “secure triplet loss”
formulation, which aims to: (i) minimise the distance between triplet pairs
(of face images) originating from the same identity and the same key, (ii)
maximise the distance between triplet pairs originating from the same face
identity and different keys, as well as from different face identities and
different keys, and (iii) control the amount of “linkability” between
protected templates, where “unlinkability” is achieved by ensuring that
similar distance values are obtained when comparing protected templates
created using different external keys (regardless of whether or not the
templates originate from the same identity). As a result of this training
procedure, we may reasonably conclude that the renewal of a compromised
template, as well as the enrollment of a new user, would not necessarily
require re-training of the NN (if it is trained to be sufficiently
generalisable in the first place); however, this point is unclear for the
method in [43], where template renewability is not explicitly considered.

The final example of the use of salting to incorporate template diversity is
[44], which begins by using a handcrafted feature extraction algorithm (LBP
[52]) to extract binary codes from face images. Random bits, serving as a
“key”, are then inserted into these codes. The resulting randomised codes are
used to train a type of Generative Adversarial Network (GAN) to “encrypt” and
“decrypt” face images. It seems, however, that this method relies on
performing reference/probe comparisons in the decrypted domain during
authentication, which is not secure from a template protection standpoint. In
this respect, the BTP method proposed in [44] appears to present the same
limitation as do traditional encryption algorithms, when considering its
suitability for securing face templates.

Permutation/shuffling was used to incorporate diversity into the protected
templates in [47, 50, 51, 48]. In [47], a pre-trained face recognition DNN is
modified and trained to learn a discriminative feature vector from the input
face image. This feature vector is then salted by adding Gaussian random
noise, and normalised to prevent noise bursts. The NN is further trained to
quantise the normalised feature vector, map each quantised segment to the
closest binary codeword in a randomly-generated codebook, and binarise and
concatenate the codeword indices to generate the final protected template. The
entire NN can be trained end-to-end. Although the initial salting operation
would incorporate diversity into the protected templates, this step is not
mentioned as a motivator for template renewability. Instead, renewability of
protected templates is claimed to be achievable by re-shuffling the codewords
in the codebook; however, the scalability of this approach before the NN must
be re-trained with new codebooks, is unclear.

In [50], the feature vector extracted by a pre-trained face recognition model
is permuted via a user-specific permutation matrix, then passed on to a NN
that is trained to emulate the handcrafted IoM hashing algorithm [36]. The
protected template is a hash code consisting of the indices of the maximum
values resulting from several random projections of the permuted feature
vector. A variant of this method is proposed in [51], where the permutation
step is applied after the random projections, instead of to the feature vector
as in [50]. For both methods, template renewability is achievable by changing
the user-specific permutation matrix, as this is the source of the non-face
randomness. Since the methods were trained and tested on different subjects,
it seems that the NN may not need to be re-trained for new/re-enrollments;
however, this scalability has not been explicitly evaluated.

The last example of a NN-learned BTP method that incorporates randomness into
the protected face templates, is [48]. This method trains a “randomized CNN”
in an end-to-end fashion, incorporating user-specific randomness in three
different stages of the process. The first part of the randomized CNN is a
feature extraction network, which is trained to extract a discriminative
feature vector, $b$, from the face image. Then, the first source of randomness
is applied, whereby $b$ is randomly permuted via a user-specific permutation
vector, $p$. The permuted feature vector is next split into two parts, $b_{A}$
and $b_{B}$. A random binary key, $k$, is generated for the user. This key is
secured via a Fuzzy Commitment construct to create a secure sketch,
$SS=c\oplus b_{B}$, where $c$ is an error-correcting code into which $k$ has
been encoded. Then, $k$ is used twice to incorporate additional randomness
into the protected template. Firstly, $k$ activates and deactivates randomly
selected neurons in RandNet, which is the most important NN component of the
randomized CNN. The resulting, randomly-created sub-network takes $b_{A}$ as
its input and outputs a template with partial randomness, $y$. Then, $k$ is
used once again, this time to define a “random permutation-flip”, whereby the
elements in $y$ are randomly permuted and the signs of randomly selected
elements are flipped. The result is the final protected (and randomised)
template, $y_{p}$, which is stored, along with $SS$ and $p$, in the database
during enrollment. During verification, the query protected template,
$y^{*}_{p}$, is generated using $k$ (decoded from $SS$) and $p$ from the
database. Successful verification depends on a high cosine similarity between
$y_{p}$ and $y^{*}_{p}$.

The use of permutations to incorporate user-specific randomness into the
protected template in [48] is by no means a new idea. What makes [48] stand
out from the other NN-learned methods, however, is the idea of randomly
modifying the actual architecture of the underlying neural network. The
methods in [43, 49, 44, 47, 50, 51] added randomness either to the extracted
feature vector or to the outputs of certain layers of the NN, but [48] was the
first to consider random activation/deactivation of the NN’s neurons. The NN
in question (i.e., “randomized CNN”) is trained using a “randomised triplet
loss”, to minimize the distance between templates from the same user generated
using the same key and maximize the distance between templates generated using
different keys. So, provided that the randomized CNN is well-trained, it
should not need to be re-trained for template renewals or new enrollments
(especially as the randomized CNN is reported to have been trained and tested
on different datasets). There is currently, however, insufficient insight into
the scalability of the template renewability process before network re-
training may need to be invoked; for example, it would be interesting to
investigate how many random sub-network selections would generate sufficiently
different protected templates before the father network would need to be
modified.

Table II summarises the studied NN-learned face BTP methods. The methods are
categorised into learns pre-defined template and learns own template
approaches, depending on what the BTP NN is trained to learn. These categories
are further divided into feature-level and image-level techniques, to indicate
whether training of the BTP NN starts from the extracted face features or
directly from the face images.

TABLE II: Examples of NN-learned face BTP methods in the literature. Learns pre-defined template | Learns own template
---|---
Feature-level | Image-level | Feature-level | Image-level
[40, 41, 42] | [37, 38, 39] | [43, 44, 50, 51] | [45, 46, 47, 48, 49]

###  II-C Non-NN versus NN-learned Face BTP Methods

Sections II.A and II.B, respectively, presented examples of Non-NN and NN-
learned face BTP methods in the literature. Based on these examples, it is not
clear what type of method is preferred among the BTP research community. To
get an idea of the current trend, we could consider the number of publications
investigating Non-NN versus NN-learned methods within, say, the last 3 years
(2018 – 2021), as illustrated in Fig. 2 for the BTP methods considered in this
survey paper.

Figure 2: Number of Non-NN and NN-learned face BTP method publications.

Fig. 2 shows that in 2018 there was an equal focus on Non-NN and NN-learned
face BTP methods in the literature. Research into both method types increased
from 2018 to 2019, with a slight preference for Non-NN BTP methods. In 2020,
there seems to have been a noticeable increase in the preference for Non-NN
over NN-learned face BTP methods, probably due to the availability of several
pre-trained DNN models that could be used as pre-BTP feature extractors. At
the time of writing this paper (about half-way through 2021), interest in Non-
NN face BTP methods seems to have dropped a little, while interest in NN-
learned methods has started to grow. Although there still seems to be a slight
preference for researching Non-NN face BTP methods today, Fig. 2 suggests that
this might soon change. In particular, the current gap between Non-NN and NN-
learned face BTP methods may close again by the end of 2021, or we might even
find that interest in NN-learned BTP methods will surpass the interest in Non-
NN methods in the near future.

Having interpreted Fig. 2 as above, we should emphasize that this is only
meant to serve as a rough guide for understanding the research trends in Non-
NN versus NN-learned face BTP methods. This is because we cannot guarantee
that we have not (accidentally) missed any publications in this survey paper,
and moreover the number of publications alone may be inadequate evidence of
such trends333For example, some researchers may work on one BTP idea for a
long time before they publish a single paper, whereas others may publish
incremental improvements on what is essentially the same method. So, it is not
always clear whether a BTP “trend” should be defined in terms of the number of
publications or the number of new BTP method ideas.. Nevertheless, the
observed increase in NN-learned face BTP research in 2021 is not surprising,
considering the exploding interest in using DNNs to improve the accuracy of
face recognition systems in practice (and thus the potential of DNNs to
achieve similar performance improvements on the BTP side). Non-NN face BTP
methods still seem to be an active area of research, however, so it is
important to understand the main conceptual strengths and weakness of both
types of face BTP methods. The remainder of this section discusses this topic.

An important strength of Non-NN BTP methods, most of which operate at feature-
level (see Section II.A), is the flexibility they offer in choosing a feature
extractor. This is because feature extraction is performed separately from the
BTP step, so as long as the extracted features satisfy the format required by
the BTP algorithm there is some freedom in the choice of feature extraction
method. This is particularly important considering the fact that most existing
Non-NN face BTP methods rely on face features extracted by pre-trained face
recognition DNN models, which have been shown to serve as excellent feature
extractors. Since there exist different types of pre-trained models that
produce feature vectors of a similar format (i.e., a vector of floating-point
values), this means that any Non-NN BTP method that can work with such
features can be applied to features extracted using any pre-trained face
recognition model. This flexibility would be important in practice, where we
may imagine the need to integrate a BTP method into an existing face
recognition system with an in-built (likely DNN-based) feature extractor.

Although some NN-learned BTP methods were also found to rely on face features
extracted by pre-trained face recognition DNN models (see Section II.B), it is
difficult to form general conclusions on how sensitive the trained BTP NN
would be to changes in the feature extractor (or whether the network would
need to be re-trained). This is especially an issue for NN-learned BTP methods
that are trained end-to-end, where the part of the NN used to extract features
from the input face images is even more intertwined with the part of the NN
that serves as the BTP algorithm. Consequently, changing the feature extractor
in an end-to-end trained BTP NN would most probably necessitate the re-
training of the NN. We may thus conclude that NN-learned BTP methods would be
useful in practice in situations where we have greater freedom in the
construction of the face recognition system (e.g., choosing the feature
extractor, having the option/time/resources to re-train the BTP NN if
necessary, etc). In situations where the BTP method must serve as an add-on in
a pre-determined face recognition pipeline, however, NN-learned BTP methods
may lack the flexibility of Non-NN BTP methods.

The main strength of NN-learned face BTP methods is that they offer the
potential for complex (i.e., highly non-linear) protection algorithms that
need not be explicitly defined, unlike Non-NN BTP methods. This is an
especially attractive aspect of NN-learned BTP methods that adopt an end-to-
end learning approach, since we allow the NN to have more freedom in learning
how to transform a face image to a protected face template. With this freedom
come consequences, however, because the more control we relinquish to the NN,
the less certain we become about how exactly the NN is learning what it is
learning. As a result, NN-learned BTP methods, particularly those that have
been trained end-to-end, tend to be difficult to evaluate. For example, it is
not clear how we should perform a thorough evaluation on what information
about a protected face template might be revealed to an adversary that has
access to the trained BTP NN (including its architecture and learned
parameters). So, while NN-learned BTP methods do have the potential to be more
complex, this complexity may come at the price of too much ignorance about how
the method works and, therefore, how well it serves to protect the face
templates. This is in contrast to Non-NN BTP methods, where the more precise
definition of the protection algorithm allows for clearer insight into
appropriate evaluation strategies.

Having commented on the difficulty of evaluating NN-learned BTP methods, we
must note that it is by no means impossible to understand the workings of a
NN; however, such an undertaking would require a greater level of involvement
by the NN’s designer. For example, the designer could carefully formulate
architectures and loss functions that would more explicitly train the NN to
generate protected face templates with certain desirable characteristics,
which could both generate stronger templates and make it easier to evaluate
the BTP method’s robustness. Of course, placing more restrictions on the NN
may reduce the attractiveness of using NNs to learn BTP algorithms, since this
removes some of the NN’s freedom to learn how it will, which forces the BTP
method to adopt a more defined algorithm (closer to a Non-NN method).

Considering these observations and comparisons between Non-NN and NN-learned
face BTP methods, it is evident that each type of approach has its own
strengths and weaknesses, so there is currently no clear “winner”; rather, it
seems that the two approaches could be combined to build on each other’s
strengths and reduce each other’s weaknesses. This is especially true
considering the fact that many of the NN-learned BTP methods were found to be
conceptually similar to, or tried to emulate via a DNN, certain Non-NN BTP
methods (e.g., see the discussion on cryptographic/non-cryptographic and pre-
defined/learned templates in Sections II.A and II.B, respectively). So, it is
reasonable to predict that, as this field of research evolves, we will witness
a sort of amalgamation between Non-NN and NN-learned BTP methods. This would
help to combine the advantages of a more precise BTP algorithm definition with
the benefits of using a DNN for greater algorithm complexity, to generate more
robust protected templates whose efficacy can be clearly evaluated.

##  III Face BTP Evaluation Techniques

It is generally agreed upon that an ideal BTP method should satisfy the
following three criteria:

  1. 1.

Recognition accuracy: The incorporation of the protection method into a
biometric recognition system should not result in a (significant) degradation
of the system’s recognition accuracy.

  2. 2.

Irreversibility: It should be impossible (or computationally infeasible) to
recover the original biometric template (features or image) from the protected
template.

  3. 3.

Renewability/Unlinkability: It should be possible to generate multiple diverse
protected templates from the same subject’s original template(s), such that
the protected templates cannot be linked to the same identity. This would
allow for the cancellation/revocation and subsequent renewal (replacement) of
compromised templates, as well as the use of the same biometric characteristic
across multiple applications, without the risk of cross-matching the protected
templates.

This section investigates which of these criteria were used to evaluate the
face BTP methods discussed in Section II, and the techniques employed to
perform the evaluations. Sections III.B, III.C, and III.D present our findings
for the “recognition accuracy”, “irreversibility”, and
“renewability/unlinkability” criteria, respectively. Before that, Section
III.A considers the reproducibility of the studied BTP works, in terms of the
public availability of: the face datasets used to train/test the proposed
methods, pre-trained DNN models or known DNN architectures used to
implement/evaluate the protected face recognition systems, and the code that
ties together the BTP method implementations and their corresponding
evaluations.

###  III-A Reproducibility

A face BTP method’s ability to fulfil the aforementioned criteria, especially
“recognition accuracy”, is normally evaluated within the context of a
particular face recognition system. Depending on the system’s intended
operational scenario, the evaluation is further constrained to certain types
of face images. So, the evaluation will be influenced by the system into which
the BTP method is incorporated and the face images used to test the system.
Although many components go into the making of a face recognition system and
the setting up of experiments to evaluate it, a full treatise of this topic is
outside the scope of this paper. For our purposes, we are interested in the
following two aspects of the set-up used to evaluate the face BTP methods
discussed in Section II: (i) what face datasets were used, and (ii) what pre-
trained DNN models or known architectures were employed within the system.

The “reproducibility” of a scientific work refers to the ability of other
researchers to replicate (and thereby externally validate) the reported
findings (e.g., see [53]). The most reliable way of ensuring reproducibility
is to make the work (e.g., code, experimental protocols, etc.) publicly
available, since this would: (i) encourage others to build upon the work
without being required to implement the method(s) themselves, and (ii)
accurately verify the reported findings without fear of misrepresenting the
method(s) or experimental procedure(s). Although it is not easy to ensure 100%
reproducibility even if everything is made public (e.g., some code may only
work in certain computing environments), scientific progress favours a fair
attempt at reproducibility over no attempt.

This section investigates the reproducibility of the face BTP works studied in
this paper, in terms of the aforementioned experimental set-up. Specifically,
we explore the public availability of the employed face datasets and DNN
models/architectures, as well as the code used to implement and evaluate the
proposed BTP methods. Tables III and IV present our findings on the employed
face datasets, and Table V provides information on the adopted DNN
models/architectures. Finally, Table VI summarises the reproducibility of the
face BTP methods and their evaluations, in terms of the public availability of
the datasets, DNNs, and code repositories dedicated to reproducing the
presented work.

TABLE III: Face datasets used for evaluating the face BTP methods from Section II. NB: Train datasets were used to train, fine-tune, or optimise the parameters of, any part of the protected system (e.g., feature extractor, BTP algorithm, classifier). Test datasets were used to report the final evaluation results for the BTP method. Method type |  Reference |  Train dataset |  Test dataset
---|---|---|---
Non-NN |  [6] |  CASIA-WebFace |  LFW $\bullet$ Faces94
|  [7] |  – |  LFW $\bullet$ IJB-A $\bullet$ IJB-B $\bullet$ CASIA-WebFace
|  [8] |  – |  FERET
|  [9] |  – |  LFW $\bullet$ FEI $\bullet$ Georgia Tech
|  [10] |  – |  MORPH
|  [11] |  CASIA-WebFace |  MegaFace
|  [12] | Color FERET $\bullet$ FEI $\bullet$ LFW
|  [13] | CMU Multi-PIE
|  [14] |  MSCeleb-1M $\bullet$ UMDFaces |  IJB-A $\bullet$ IJB-C
|  [15] |  VGGFace2 |  LFW
|  [16] |  – |  LFW
|  [17] | Color FERET $\bullet$ FRGC v2.0
|  [18] | AT&T / ORL $\bullet$ FERET $\bullet$ CMU PIE $\bullet$ M2VTS
|  [19] | PaSC $\bullet$ LFW $\bullet$ FERET
|  [20] |  – |  LFW
|  [21] | AT&T / ORL
|  [22] |  – |  LFW
|  [23] |  VGGFace2 $\bullet$ LFW |  LFW
|  [24] | CMU PIE $\bullet$ FEI $\bullet$ Color FERET
|  [25] |  – |  LFW $\bullet$ AT&T / ORL
|  [26] |  – |  FEI $\bullet$ Georgia Tech
|  [27] | Faces94 $\bullet$ NVIE
|  [28] |  – |  CMU Multi-PIE $\bullet$ FEI $\bullet$ Color FERET
|  [29] |  – |  LFW $\bullet$ VGGFace2 $\bullet$ IJB-C
|  [30] | Caltech Faces 1999 $\bullet$ Georgia Tech $\bullet$ 10k US Adult Faces
|  [31] |  – |  MOBIO
NN-learned |  [37, 38] | CMU PIE $\bullet$ Extended Yale Face Database B $\bullet$ CMU Multi-PIE
|  [39] | CMU PIE $\bullet$ Extended Yale Face Database B
|  [40] | CMU PIE $\bullet$ FEI $\bullet$ Color FERET
|  [41] | CMU PIE $\bullet$ FEI $\bullet$ Color FERET
|  [42] | CMU PIE $\bullet$ FEI $\bullet$ Color FERET
|  [43] |  MS-Celeb-1M $\bullet$ CASIA-WebFace $\bullet$ AT&T / ORL $\bullet$ Own |  CASIA-WebFace $\bullet$ AT&T / ORL $\bullet$ Own
|  [44] | LFW
|  [45] | CMU PIE $\bullet$ Extended Yale Face Database B $\bullet$ CMU Multi-PIE $\bullet$ WVU Multimodal
|  [46] | VGGFace2 $\bullet$ MegaFace
|  [47] | YouTube Faces $\bullet$ FaceScrub
|  [48] |  VGGFace2 $\bullet$ MS-Celeb-1M |  FRGC v2.0 $\bullet$ CFP $\bullet$ IJB-A
|  [49] | YouTube Faces
|  [50] | LFW
|  [51] |  YouTube Faces $\bullet$ FaceScrub |  LFW $\bullet$ YouTube Faces $\bullet$ FaceScrub

Considering Table III, it is interesting to note that a “train” dataset was
used in only about half of the Non-NN BTP works, compared to all of the NN-
learned BTP works. This makes sense, because Non-NN face BTP methods tend to
employ a neural network only as a feature extractor, meaning that they are
more likely to use a pre-trained DNN model for this purpose. On the other
hand, for NN-learned BTP methods, a neural network’s purpose is to learn the
BTP algorithm itself, meaning that training is a necessary step. This is true
for at least the BTP part, but possibly for feature extraction as well,
especially if the neural network is trained end-to-end.

For the Non-NN BTP works that did perform some training, in most cases the
motivator was one of the following two reasons. The first reason was to train
a reliable NN-based feature extractor (by either fine-tuning an
adopted/modified DNN model/architecture, or by training a proprietary DNN), so
that a handcrafted BTP algorithm could be applied to the extracted face
features. This approach was used in [6, 13, 14, 18, 19]. Similarly, [11]
trained a DNN to reduce a face feature vector to its “intrinsic”
dimensionality before encrypting it (via HE), while [21] and [30] trained a
CNN and an SVM, respectively, to classify already-protected face templates.
The second common reason for using a train dataset among Non-NN face BTP
methods, was to optimise certain parameters of handcrafted BTP algorithms
(which could then be applied to the test dataset) or to train machine-learning
BTP methods, but not to train a neural network. This was the case in [17, 23,
24, 27]. Similarly, although face templates were protected via HE in [12], a
separate algorithm (Product Quantisation plus clustering) was trained to
extract stable binary codes from the unprotected feature vectors, for indexing
purposes in a face identification system. Finally, quite a different
motivation for a train dataset was found in [15], where a neural network was
trained to invert templates protected using the Fuzzy Commitment scheme to
recover the unprotected face templates.

Considering NN-learned BTP methods, all of which were necessarily trained in
some way, Table III indicates that most of these works used the same
dataset(s) for both training and testing. In fact, the majority of these ([37,
38, 39, 40, 41, 42, 43, 47]) trained and tested their algorithms on even the
same subjects, whereby the training set of images was used to enroll the
subjects and the test set was used to evaluate the recognition accuracy444In
[43], MS-Celeb-1M seems to have been used only to train the feature extractor,
but the BTP algorithm was trained/tested on the same subjects of the other 3
datasets. Also, [47] obtained the positive and negative (same/different-
subject) test images from two different datasets.. For [37, 38, 39, 40, 41,
42], this is because the protected templates were pre-defined for each subject
enrolled in the face recognition system, so the NN was trained only for those
subjects (making re-enrollments impossible without re-training the NN, as
explained in Section II.B). Although [43, 47] did not pre-define the protected
templates, they still used the same train/test subjects for their evaluations,
which makes it difficult to judge the methods’ generalisability to unseen
subjects.

The same train/test datasets were also employed for evaluating the NN-learned
BTP methods proposed in [45, 49, 50]. These works, however, explicitly aim to
avoid network re-training for new enrollments, by learning their own
representations of protected face templates rather than pre-defining them. So,
[45, 49, 50] all used different subjects for training and testing their BTP
algorithms. Finally555For the remaining NN-learned BTP works: The train/test
split is not explained in [44], whereas [46] used 80:20 and 90:10 train/test
splits within each database. It is unclear if the splits contained different
subjects., [51, 48] took a step further towards demonstrating the
generalisability of their BTP methods, by training and testing their BTP
algorithms not only on different subjects, but on different datasets.

TABLE IV: Face datasets from Table III, arranged first in the order of “No. uses” (train/test), then alphabetically. Face dataset | No. uses | Paper | Website | Currently available to other researchers?
---|---|---|---|---
LFW | 14 | [54] | https://bit.ly/38Bu5jk | Yes
(freely available)
CMU PIE | 8 | [55] | https://bit.ly/3kObkPi | Yes
(must contact person mentioned on website)
FEI | 8 | [56] | https://bit.ly/2WMl0S4 | Yes
(freely available)
Color FERET | 7 | – | https://bit.ly/3mWqUes | Yes
(must request access)
VGGFace2 | 5 | [57] | https://bit.ly/3kLWWXH (unofficial) | No
(official download link disappeared)
CASIA-WebFace | 4 | [58] | https://bit.ly/38BtHRU (unofficial) | No
(no reliable download link)
CMU Multi-PIE | 4 | [59] | https://bit.ly/2WPeM3W | Yes
(must order, costs $250-1000)
AT&T / ORL | 3 | [60] | https://bit.ly/3kMB8ev | Yes
(freely available)
Extended Yale Face Database B | 3 | [61] | https://bit.ly/3gUilgq | Yes
(freely available)
FERET | 3 | [62] | https://bit.ly/3mWqUes | No
(replaced by Color FERET)
Georgia Tech | 3 | [63] | https://bit.ly/3yCGfD3 | Yes
(freely available)
MS-Celeb-1M | 3 | [64] | https://bit.ly/38zHYOS (unofficial) | No
(retracted)
YouTube Faces | 3 | [65] | https://bit.ly/3zDxREp | Yes
(freely available after providing contact details)
Faces94 | 2 | – | https://bit.ly/3Br35iO | Yes
(freely available)
FaceScrub | 2 | [66] | https://bit.ly/3zRY96l | Yes
(must request access)
FRGC v2.0 | 2 | [67] | https://bit.ly/3yxNOLj | Yes
(must request access with signed license agreement)
IJB-A | 2 | [68] | https://bit.ly/38BwKcx | Yes
(must request access)
MegaFace | 2 | [69] | https://bit.ly/3gUniFZ (unofficial) | No
(retracted)
Caltech Faces 1999 | 1 | – | https://bit.ly/3kg8l35 | Yes
(freely available)
CFP | 1 | [70] | https://bit.ly/3DDqjEj | Yes
(freely available)
IJB-B | 1 | [71] | https://bit.ly/38BwKcx | Yes
(must request access)
IJB-C | 1 | [72] | https://bit.ly/38BwKcx | Yes
(must request access)
MOBIO | 1 | [73] | https://bit.ly/3k6qj84 | Yes
(must request access with signed license agreement)
MORPH (Academic) | 1 | [74] | https://bit.ly/3kPo5ZF | Yes
(must order, costs $99)
M2VTS | 1 | [75] | https://bit.ly/3zErAZk | Yes
(must order with signed license agreement, costs £100-1000)
NVIE | 1 | [76] | https://bit.ly/3DEe8Hk | Yes
(must request access with signed license agreement)
Own | 1 | [43] | – | No
(no information on availability)
PaSC | 1 | [77] | https://bit.ly/2YoPiv3 | Yes
(must request access with signed license agreement)
UMDFaces | 1 | [78] | https://bit.ly/3BxPRki | No
(temporarily unavailable)
10k US Adult Faces | 1 | [79] | https://bit.ly/3EnIbmO | Yes
(must request access with signed license agreement)
WVU Multimodal | 1 | [80] | https://bit.ly/3DEdYjc | Partly
(restricted to US citizens)

Next, we turn our attention to Table IV, which ranks the face datasets from
Table III in terms of their popularity (measured by the number of face BTP
publications that used each dataset for either training or testing). The most
popular dataset was LFW, which is understandable given that this dataset has
been commonly used to benchmark DNN-based face recognition systems. LFW
consists of face images captured in a variety of unconstrained conditions, so
it is best used for evaluating face recognition systems designed to operate in
such environments. Having said that, before choosing LFW on account of its
popularity, the appropriateness of this dataset for evaluating certain face
BTP methods should be carefully considered. This is especially true for BTP
algorithms that are intended for use in cooperative verification scenarios,
which are not well-represented by LFW. This is because the LFW dataset
consists of images pulled from the internet, largely of celebrities performing
various actions (e.g., speaking, playing a sport), so they do not resemble the
types of images that are likely to be acquired during a deliberate
verification attempt.

The second most widely-used face datasets among the studied face BTP works are
CMU PIE and FEI. However, since CMU Multi-PIE is actually an extension of CMU
PIE, the combined dataset is more popular than FEI and almost as widely-
adopted as LFW. CMU (Multi-)PIE consists of face images acquired in a
controlled environment, but where each subject’s face images have been
captured from multiple viewpoints, under various lighting conditions, while
displaying a range of facial expressions. So, this dataset contains lots of
deliberate variations, and variation factors that are deemed inappropriate for
certain evaluations can be excluded. As a result, CMU (Multi-)PIE is easier to
use than LFW for evaluating face BTP methods intended for use in cooperative
verification scenarios, where the acquired face images may contain certain
(reasonable) inconsistencies.

Another interesting observation from Table IV is that, of the 31 datasets that
are listed, over half have been employed by only 1 or 2 of the studied face
BTP works. One reason for this could be that these datasets were considered
the most suitable for evaluating the corresponding face BTP methods, so they
were chosen despite the fact that other researchers did not use the same ones
for evaluating their own BTP methods. If this is indeed the case, then the
approach is reasonable, although this makes it difficult to fairly compare the
performance of different BTP methods. To facilitate this step, it is important
that BTP researchers work towards the reproducibility of their method
implementations, as discussed earlier.

Considering reproducibility from the point of view of the datasets used to
evaluate the studied face BTP methods, Table IV indicates that the majority
(23 out of 31) of these datasets are currently available to other researchers.
Most of the remaining 8 datasets used to be publicly available, but have now
been either permanently or temporarily retracted (usually for legal reasons).
So, although we can never guarantee that a dataset will forever be available,
it is encouraging to observe that most of the face BTP methods studied in this
paper have been trained/evaluated on datasets that were publicly available at
least at the time of their publication. A continuing focus on using public
datasets in the BTP research community will help to increase the
reproducibility of the published work, thereby fostering greater understanding
of the proposed BTP methods.

Aside from public availability of the employed datasets, the most important
facilitator of reproducibility would be the public availability of the code
used to implement and evaluate the proposed BTP methods. An important aspect
of this concerns the use of publicly available pre-trained DNN models or
implementations of known DNN architectures, which is summarised in Table V for
the studied face BTP works.

TABLE V: Pre-trained DNN models or known DNN architectures used to
implement/evaluate the protected face recognition systems.
NB: “Link to public implementation” was obtained from either “reference” or “DNN cited paper”. Method type | Reference | DNN name | DNN cited paper | Link to public DNN implementation
---|---|---|---|---
Non-NN | [6] | DeepID | [81] | ✘
Light CNN-9 | [82] | https://bit.ly/3kJCKFK
[7] | FaceNet | [83] | https://bit.ly/3BFc2Fm
(model: 128-dim. output)
SphereFace | [84] | https://bit.ly/3gVqf96
[8] | FaceNet | [83] | ✘
[9] | FaceNet | [83] | ✘
[10] | ArcFace | [85] | https://bit.ly/2WTus6v
(model: “LResNet100E-IR,ArcFace@ms1m-refine-v2”)
CurricularFace | [86] | https://bit.ly/3zHPovx
(model: “IR101”)
[11] | ArcFace | [85] | https://bit.ly/2WTus6v
(model: 512-dim. output)
DeepMDS | [87] | ✘
[12] | ArcFace | [85] | https://bit.ly/2WTus6v
(model: ResNet-100)
[13] | – | – | –
[14] | ResNeXt | [88] | https://bit.ly/3kTXJFU
[15] | FaceNet | [83] | https://bit.ly/2WJTogQ
(model: 128-dim. output)
Dlib | [89] | https://bit.ly/3h1qq2S
Light CNN-29v2 | [90] | https://bit.ly/3kJCKFK
| NbNet | [3] | ✘
| [16] | VGG-Face | [91] | https://bit.ly/3tpmGgF
| [17] | ArcFace | [85] | https://bit.ly/2WTus6v
| (model: “LResNet100E-IR,ArcFace@ms1m-refine-v2”)
| [18] | – | – | –
| [19] | – | – | –
| [20] | VGG-Face | [91] | https://bit.ly/3tpmGgF
| [21] | – | – | –
| [22] | ArcFace | [85] | https://bit.ly/2WTus6v
| MobileFace | [92] | ✘
| ShuffleFace | [93] | ✘
| [23] | ArcFace | ✘ | https://bit.ly/2WTus6v
| (model: pre-trained on MS-Celeb-1M)
| [24] | FaceNet | [83] | https://bit.ly/3BFc2Fm
| (model: 512-dim. output)
| [25] | FaceNet | [83] | ✘
| [26] | AlexNet | [94] | https://bit.ly/38CKSlT
| [27] | LeNet-5 | [95] | ✘
| AlexNet | [94] | https://bit.ly/38CKSlT
| [28] | ArcFace | [85] | https://bit.ly/2WTus6v
| (model: ResNet50, 512-dim. output)
| CosFace | [96] | ✘
| [29] | FaceNet | [83] | ✘
| ArcFace | [85] | https://bit.ly/2WTus6v
| (model: pre-trained on MS-Celeb-1M, 256-dim. output)
| [30] | FaceNet | [83] | https://bit.ly/3BFc2Fm
| (model: Inception-ResNet-v1, 512-dim. output)
| [31] | FaceNet | ✘ | https://bit.ly/3nv3FYV
| Idiap | ✘ | https://bit.ly/3kaLTYQ
NN-learned | [37, 38] | – | – | –
[39] | CaffeNet | [97] | https://bit.ly/3BELMLm
[40] | VGG-Face | [91] | https://bit.ly/3tpmGgF
[41] | VGG-Face | [91] | https://bit.ly/3tpmGgF
[42] | VGG-Face | [91] | https://bit.ly/3tpmGgF
[43] | Inception-ResNet-v1 | [98] | https://bit.ly/3kNNCCJ
[44] | ResNet-11 | ✘ | ✘
[45] | VGG-19 | [99] | https://bit.ly/2Yt3x1X
[46] | ResNet | [89] | ✘
DenseNet | [100] | https://bit.ly/38DAFG4
[47] | VGG-16 | [99] | https://bit.ly/2Yt3x1X
[48] | ResNet-50 | [89] | ✘
[49] | Inception-ResNet-v1 | [98] | https://bit.ly/2WI8hQd
[50] | ArcFace | [85] | https://bit.ly/2WTus6v
(model: ResNet-50, 512-dim. output)
| [51] | ArcFace | [85] | https://bit.ly/2WTus6v
| (model: ResNet-50, 512-dim. output)

Table V indicates that ArcFace and FaceNet pre-trained models are the most
popular. Apart from their impressive performance, the widespread adoption of
ArcFace models may be attributed to the fact that the model repository
(https://bit.ly/2WTus6v) is well-maintained by two of the authors of [85]. For
FaceNet, unfortunately there exists no public implementation of [83] by its
authors, but a number of models inspired by [83] have been made available by
others. The most popular FaceNet model repository appears to be that of David
Sandberg (https://bit.ly/3BFc2Fm), but this is not the only one (e.g.,
https://bit.ly/2WJTogQ presents another version). So, when authors do not
point to a particular FaceNet implementation (e.g., [8, 9, 25, 29] in Table
V), the reproducibility of their work is reduced. In general, however, Table V
shows that only about a third of the studied face BTP works did not provide a
clear indication of where their adopted DNN models/architectures came from,
whereas the remaining two thirds provided either a direct link or sufficient
information to locate the adopted DNN (e.g., a citation of the DNN’s paper,
where a public repository link is provided).

TABLE VI: Reproducibility of the face BTP methods and evaluations. NB: The order of multiple ✔, ✘, and/or $\thickapprox$ for “datasets” and “DNNs” corresponds to the order of these elements in Tables III and V, respectively (for a specific reference). Method type | Reference | Are used datasets (still) available? | Link to employed DNN(s)? | Link to paper’s public code repository?
---|---|---|---|---
Train |  Test
Non-NN | [6] | ✘ | ✔✔ | ✘✔ | ✘
[7] |  – | ✔✔✔✘ | ✔✔ | https://bit.ly/3DIPJjR
[8] |  – | ✘ | ✘ | ✘
[9] |  – | ✔✔✔ | ✘ | ✘
[10] |  – | ✔ | ✔✔ | ✘
[11] | ✘ | ✘ | ✔✘ | ✘
[12] | ✔✔✔ | ✔ | ✘
[13] | ✔ | – | ✘
[14] | ✘✘ | ✔✔ | ✔ | ✘
[15] | ✘ | ✔ | ✔✔✔✘ | ✘
[16] |  – | ✔ | ✔ | ✘
[17] | ✔✔ | ✔ | ✘
[18] | ✔✘✔✔ | – | ✘
[19] | ✔✔✘ | – | ✘
[20] |  – | ✔ | ✔ | https://bit.ly/2YqeGAr
[21] | ✔ | – | ✘
[22] |  – | ✔ | ✔✘✘✘ | ✘
[23] | ✘✔ | ✔ | ✔ | https://bit.ly/3mUeeEO
[24] | ✔✔✔ | ✔ | ✘
[25] |  – | ✔✔ | ✘ | ✘
[26] |  – | ✔✔ | ✔ | ✘
[27] | ✔✔ | ✘✔ | ✘
| [28] |  – | ✔✔✔ | ✔✘ | ✘
| [29] |  – | ✔✘✔ | ✘✔ | ✘
| [30] | ✔✔✔ | ✔ | ✘
| [31] |  – | ✔ | ✔ | https://bit.ly/3uBw5C6
NN-learned | [37, 38] | ✔✔✔ | – | ✘
[39] | ✔✔ | ✔ | https://bit.ly/3mZA8GH
[40] | ✔✔✔ | ✔ | ✘
[41] | ✔✔✔ | ✔ | ✘
[42] | ✔✔✔ | ✔ | ✘
[43] | ✘ | ✘✔✘ | ✔ | ✘
[44] | ✔ | ✘ | ✘
[45] | ✔✔✔$\thickapprox$ | ✔ | ✘
[46] | ✘✘ | ✘✔ | ✘
[47] | ✔✔ | ✔ | ✘
[48] | ✘✘ | ✔✔✔ | ✘ | ✘
[49] | ✔ | ✔ | https://bit.ly/3gZ6eOU
[50] | ✔ | ✔ | ✘
[51] | ✔✔ | ✔✔✔ | ✔ | ✘

Finally, the most important way of ensuring reproducibility is to make public
the code that ties together the entire implementation of the BTP algorithm
(including any adopted DNN models/architectures), as well as the procedures
used to evaluate it. Table VI shows that only a handful of the studied face
BTP works provided a link to a public code repository intended for reproducing
their work ([7, 20, 23, 31, 39, 49]). Although we did not test all this code
ourselves and cannot, therefore, guarantee that each link points to fully
reproducible work, we still consider the presence of a public code repository
a deliberate attempt at reproducibility. Note that some papers provided links
to other open-source code that they used for certain parts of their work, but
not to a code repository specifically dedicated to reproducing their proposed
methods. For example, [15] provided a link to the adopted Fuzzy Commitment
(FC) implementation, but not to code using the FC to launch the template
reconstruction attack that was the whole point of this paper. Similarly, [11,
12] provided a link to the SEAL library used for the HE component of their
work, but not to code implementing the actual search/indexing methods for
retrieving the protected face templates in a face identification system, which
were the main contributions of these papers. Although such individual links
are useful for enabling others to implement certain components of the proposed
methods/experiments, they were not mentioned in Table VI because they do not
allow the work as a whole to be reproduced, i.e., the combination of these
individual “components” is left open to interpretation.

Taking into account the public availability of the papers’ code repositories,
any adopted DNN models/architectures, and the employed datasets, Table VI
implies that, at this stage, it would be difficult to faithfully reproduce
most of the existing face BTP methods. Of course, there is always the
possibility of private dataset/model/code exchanges between individual
researchers; however, this does not benefit the BTP research community as a
whole, which would only be achieved if all work was made freely and equally
available to all researchers.

Although the reproducibility of a BTP method aids in its visibility and
further development, we cannot launch into reproducing a piece of work before
understanding what has been proposed. Furthermore, even though the
availability of all BTP work to the entire research community would be a
commendable achievement, this may eventually lead to information overload. In
this case, researchers will have to choose which BTP methods to build upon or
(re-)evaluate, meaning that an understanding of the methods and their
evaluation procedures in advance would be helpful. Section II of this paper
discussed the types of face BTP methods proposed thus far, and the current
section investigated the reproducibility of these works. Next, Sections III.B,
III.C, and III.D explore the techniques that were used to evaluate the
“recognition accuracy”, “irreversibility”, and “renewability/unlinkability”
criteria, respectively, for the existing face BTP methods.

###  III-B Recognition Accuracy

A BTP method is considered to satisfy the “recognition accuracy” criterion if
it can be incorporated into a biometric system without (significantly)
degrading its recognition accuracy. In other words, the ability of the
biometric system to recognise the enrolled identities should be approximately
the same regardless of whether the identities are represented in terms of
their protected or unprotected biometric templates. This section considers
which face BTP methods from Section II were evaluated for their ability to
satisfy this criterion and what metrics/plots were used to conduct the
evaluation.

Of the face BTP works discussed in Section II, all but [15] presented an
evaluation of the protected system’s recognition accuracy. Evaluation of this
criterion was not deemed necessary in [15], since the aim was to show that
DNN-extracted face templates protected using the Fuzzy Commitment scheme can
be inverted using a reconstruction attack (meaning that this BTP method
satisfies neither the irreversibility nor the renewability/unlinkability
criterion). Considering the publications that did evaluate the recognition
accuracy of their proposed BTP methods, the adopted evaluation metrics/plots
were usually the same as those used to evaluate the recognition accuracy of a
standard (unprotected) biometric system. This is to be expected, since a BTP
algorithm must operate within a biometric system, meaning that the recognition
accuracy evaluation should reflect the system’s ability to achieve its
underlying aim (i.e., to recognise the enrolled users).

Table VII summarises the metrics and plots used to evaluate the recognition
accuracy of the studied face BTP methods. Note that other metrics/plots may
have been adopted for some evaluations, but Table VII presents only those used
to report the final recognition accuracy (not those used in intermediate
processing stages, like for fine-tuning system parameters).

TABLE VII: Recognition accuracy evaluation techniques among the studied face
BTP methods. Metrics/plots: False Accept/Reject/Match/Non-Match/Positive Rate
(FAR/FRR/FMR/FNMR/FPR), Equal Error Rate (EER), Genuine/True Accept/Match Rate
(GAR/TAR/TMR), False Negative/Positive Identification Rate (FNIR/FPIR),
True/False Positive/Negative (TP/FP/TN/FN), Area Under Curve (AUC),
Identification Rate (IR), Detection and Identification Rate (DIR), Detection
Error Trade-off (DET), Receiver Operating Characteristic (ROC),
True/Mis-/Failed Identification Rate (TIR/MIR/FIR). Assume “Accuracy” =
$\frac{TP+TN}{TP+FP+FN+TN}$. Comparisons: “unprotected” (original) templates
vs. “protected” (via other BTP methods) templates.
Comparisons key: ✔ = explicit; $\thickapprox$ = implicit/partial; ✘ = none. Method type | Reference | Evaluated in terms of: | Compared to:
---|---|---|---
Metrics | Plots | Unprotected | Protected
templates? | templates?
Non-NN | [6] | Accuracy (%) | ✘ | ✔ | ✔
[7] | TAR @ FAR = {0.01, 0.1, 1} (%) | ✘ | ✔ | ✘
[8] | ✘ | DET (FNIR vs. FPIR) (%) | $\thickapprox$ | ✘
[9] | GAR @ FAR = {0.01, 0.1, 1} (%) | ✘ | ✔ | ✔
[10] | Rank-1 recognition rate (%), | DET (FNIR vs. FPIR) (%) | $\thickapprox$ | $\thickapprox$
EER (%), FNIR @ FPIR = 0.1 (%)
[11] | Rank-1 accuracy (%) | ✘ | $\thickapprox$ | ✘
[12] | Rank-1 IR (%) | DET (FNIR vs. FPIR) (%) | ✔ | ✔
FNIR @ FPIR = {0, 0.1, 1} (%)
[13] | ✘ | ROC (FRR vs. FAR, plus EER) (%) | ✘ | ✘
[14] | TAR @ FAR = {1, 0.1, 0.01, 0.001} (%) | ROC (TAR vs. FAR) | ✔ | ✘
[15] | ✘ | ✘ | ✘ | ✘
[16] | {TP, FP, TN, FN} @ FPR = 0.001 | ✘ | ✔ | ✘
[17] | FNMR and FMR (%) | ✘ | $\thickapprox$ | ✔
[18] | EER | DET (Miss vs. False Alarm prob.) (%) | $\thickapprox$ | ✘
[19] | Accuracy (%) | ✘ | ✔ | ✔
[20] | EER (%) | ✘ | ✔ | ✘
[21] | Recognition rate (%) | ✘ | ✘ | ✘
[22] | Accuracy (%) | DET (FRR vs. FAR, plus EER) (%) | ✔ | ✘
[23] | AUC (%), EER (%), TPR (%), | ✘ | ✔ | ✔
TAR @ FAR = 0.1 (%),
Rank-1 DIR @ FAR = 1 (%)
| [24] | EER (%), GAR @ FAR (%) | ✘ | ✘ | ✔
| [25] | FAR and FRR (%) | ✘ | ✘ | ✔
| [26] | EER (%) | ✘ | ✘ | ✘
| [27] | EER | ROC (1 - FNMR vs. FMR) | ✔ | ✘
| [28] | TAR @ FAR (%) | ✘ | ✔ | ✔
| [29] | IR (%) @ Rank = {1, 10, 50}, | ROC (VR vs. FAR) (%) | $\thickapprox$ | ✘
| Accuracy (%),
| Verification Recognition @ FAR = 0.1 (%),
| Rank-1 DIR @ FAR = 1 (%),
| Own metrics: TIR, MIR, FIR (%)
| [30] | Accuracy, FAR, FRR, EER (%) | ✘ | ✔ | ✔
| [31] | 1 - FNMR (TMR) @ FMR = $10^{-3}$ (or 0.1%) | ROC (1 - FNMR vs. FMR) | ✔ | ✔
NN-learned | [37, 38] | EER (%), GAR @ FAR = {0, 1} (%) | ✘ | ✘ | ✔
[39] | EER (%), GAR @ FAR = 1 (%) | ROC (GAR vs. FAR) | ✘ | ✔
[40] | EER (%), GAR @ FAR = {0, 0.01, 0.1} (%) | ROC (GAR vs. FAR) | ✘ | ✔
[41] | EER (%), GAR @ FAR = 0 (%) | ✘ | ✘ | ✔
[42] | EER (%), GAR @ FAR (%) | ✘ | ✘ | ✔
[43] | FAR and FRR (%) | ✘ | ✘ | ✘
[44] | EER | ✘ | ✘ | ✔
[45] | EER (%), GAR @ FAR = 0.01 (%) | ROC (GAR vs. FAR) (%) | ✘ | ✔
[46] | Accuracy (%), GAR @ FAR = 0 (%) | ROC (TAR vs. FAR) | ✔ | ✔
[47] | EER (%), GAR @ FAR = 0.1 (%), | ✘ | ✘ | ✔
Maximum Average Precision
[48] | GAR @ FAR = 0.1 (%) | ✘ | ✔ | ✔
[49] | EER (%), FNMR @ FMR = 0.1 (%) | DET (FNMR vs. FMR) | ✔ | ✔
[50] | EER (%) | ✘ | ✔ | ✔
[51] | EER (%) | ROC (TPR vs. FPR) | ✔ | ✔

From Table VII, it is evident that most existing face BTP methods have been
proposed for verification (as opposed to identification) systems, since the
majority of employed metrics/plots are verification-system-based. The two most
popular evaluation metrics appear to be: (i) the EER, which is the point at
which the FAR/FMR and FRR/FNMR are equal, and (ii) the GAR/TAR/TMR reported at
the match threshold corresponding to a particular FAR/FMR (i.e., GAR/TAR/TMR @
FAR/FMR), with an FAR/FMR of 0.1% being the most common threshold choice. The
most frequently used plots, which portray the recognition accuracy for a range
of different thresholds, include the standard DET and ROC curves, albeit with
some inconsistencies in the axis labels (e.g., for ROC, sometimes the y-axis
was FRR and sometimes it was GAR = 1 - FRR). For the few BTP methods that were
evaluated in an identification scenario, some form of Rank-1
Identification/Recognition Rate/Accuracy was the most commonly adopted metric,
and DET plots with FNIR on the y-axis and FPIR on the x-axis were the most
frequently used plots.

Table VII also indicates which BTP methods’ evaluated recognition accuracy was
compared to the accuracy resulting from using: (i) unprotected templates, and
(ii) face templates protected using other BTP methods. Note that an explicit
comparison (indicated by ✔) means that the evaluation results of the “other”
systems (i.e., those using unprotected templates or templates protected by
other BTP methods) were deliberately/directly compared (e.g., in a table, on
the same plot, or conceptually). On the other hand, an implicit/partial
comparison ($\thickapprox$) means that either there was some analysis/mention
of the recognition accuracy of the “other” system(s) but there was no explicit
comparison, or the comparison was presented for only a part of the protected
system (e.g., in a hybrid BTP method with multiple stages of protection).

Comparing the accuracy of a face recognition system using protected templates
against the same system using unprotected templates would provide a good
indication of how the adopted BTP method affects the baseline (without BTP)
system’s recognition accuracy; indeed, such a comparison is traditionally
implied in the evaluation of a BTP method’s “recognition accuracy” criterion.
From Table VII, we can see that this comparison to unprotected face templates
was conducted for the majority of Non-NN face BTP methods but for only a few
NN-learned BTP methods. This may be attributed to the fact that Non-NN BTP
methods are essentially add-ons to an existing face recognition system, since
they are usually incorporated into only one stage (most commonly after the
feature extractor). This means that the BTP part of the protected system can
simply be excluded when evaluating the recognition accuracy of the unprotected
system. NN-learned BTP methods, on the other hand, tend to be more intertwined
within the face recognition system, particularly those that are trained end-
to-end. Consequently, it may be difficult to remove the BTP part of the NN to
emulate the unprotected system and thereby evaluate its recognition accuracy.

Of the NN-learned BTP works that provided an explicit recognition accuracy
comparison to unprotected templates, [46, 49, 50, 51] performed this
evaluation by directly accessing these templates. In particular, the
unprotected face templates came from the individual (pre-fusion) NNs in [46],
the NN trained using the original Triplet Loss instead of the proposed Secure
Triplet Loss formulation in [49], and straight from the pre-trained DNN that
was used as the feature extractor preceding the BTP NN in [50, 51].
Alternatively, for the end-to-end trained BTP NN in [48], the comparison was
based on unprotected templates that were extracted using other “state-of-the-
art” feature extractors. It seems that a more direct comparison could have
been possible by extracting the unprotected face templates from the same NN,
using only the Feature Extraction part of the Randomized CNN; however, the
adopted comparison approach is interesting, because it seems to imply that the
Randomized CNN is considered as a whole biometric system that is not to be
split up into unprotected/protected sub-systems. In this case, it appears
reasonable to compare the recognition accuracy of the Randomized CNN to that
of other, unprotected systems. To ensure fairness, the comparisons in [48]
were based on the same face dataset, using the same comparison function and
verification protocol.

Considering comparisons to face templates protected using other BTP methods in
Table VII, we see that the majority of NN-learned BTP works presented such a
comparison, in contrast to fewer than half of Non-NN BTP works. This is
probably because many of the NN-learned BTP methods build upon each other, so
it makes sense to compare the recognition accuracy across these methods. For
example, [39, 40, 41, 42, 45, 47, 48] all compared the accuracy of their
proposed BTP methods to one or both of [38, 40], either because they are
improved versions of [38, 40] or because these methods are among the first NN-
learned BTP approaches so they are unanimously considered as state-of-the-art.
A similar progression of improvements is not as evident among the Non-NN BTP
methods, however, so a comparison of the recognition accuracy to that of other
face BTP methods was more likely to be avoided. When such a comparison was
provided, it tended to against traditional methods that the authors deemed
similar to their proposed BTP method (e.g., HE, BioHashing, Fuzzy Vault,
etc.), or on those methods for which the recognition accuracy had been
reported on the same face dataset.

For the majority of both Non-NN and NN-learned works that presented a
comparison to other BTP methods, the comparisons were based on reported
results generated on the same face dataset. The use of the same dataset is
important to ensure fairness in the comparison, as is the use of the same
evaluation protocol (e.g., same train/test images, same reference/probe
samples, etc.). Since the comparisons noted in Table VII were mostly based on
reported results (as opposed to being generated using shared/own
implementations of the other BTP methods), it is not always clear whether
exactly the same protocol was applied when evaluating the recognition
accuracy. For example, [24, 40, 42, 39, 41, 45] all compared the recognition
accuracy of their BTP methods to that of [38] using the CMU PIE face dataset.
All publications mention that the same 5 “poses” (and all illumination
variations) were selected for the protocol, but that the train/test images
(within those pose categories) were randomly chosen. In this case, although
the protocol is close enough to that in [38], it does not appear to be the
same, since the train/test splits were random. Most of these works attempted
to account for this by presenting the average (and sometimes $\pm$ standard
deviation) of their EER and GAR @ FAR results across 5 random train/test
splits. So, the comparison seems fair enough; however, to ensure 100% fairness
in general, we must make sure to use the same dataset, along with exactly the
same matching protocol, when computing the recognition accuracy.

It should be noted that a few of the studied works ([17, 10, 48, 46]) reported
some recognition accuracy comparisons using results generated on datasets that
were different to those used to evaluate their proposed BTP methods. While
such comparisons may have been included for completeness’ sake, and the
authors explicitly noted the use of different datasets, in general the
usefulness of presenting inter-dataset comparisons is doubtful. This is
because, unless the recognition accuracy for the methods being compared has
been evaluated under the same conditions, the comparison cannot be considered
fair and can, in fact, often be misleading. So, we would recommend that such
comparisons be avoided. A suitable alternative could be to provide some
insights into the comparison from a conceptual point of view (e.g., how the
proposed BTP method may be expected to perform compared to another method), as
presented in [31]. This approach could be adopted until it is possible to
provide a fair numerical comparison (or it could be used to complement the
numerical comparison). Provided that this analysis is performed honestly, it
would probably be more meaningful than an unfair numerical comparison.

On a final note, to make comparisons between different BTP methods easier and
more useful, we encourage the BTP community to strive for the reproducibility
of both their BTP method implementations and their evaluation procedures. This
way, firstly, we could ensure that comparisons on the same dataset are based
on exactly the same evaluation protocol. Secondly, we would have more freedom
in the choice of dataset used for the comparison, instead of feeling compelled
to either constrain our evaluations to only those datasets on which the
results of other BTP methods have been reported or to present inter-dataset
comparisons. This is because, if we had access to open-source implementations
of other researchers’ BTP methods, we could evaluate those methods on whatever
dataset(s) we deem to be most representative of our proposed BTP methods’
intended operational scenario(s). This would help us provide fairer and more
meaningful insights into how our BTP method compares to other BTP methods in
the most appropriate evaluation context. The reproducibility of the existing
face BTP methods was considered in Section III.A.

###  III-C Irreversibility

A BTP method is considered to satisfy the “irreversibility” criterion if it is
impossible (or computationally infeasible) to recover the original biometric
template (features or image) from the protected template. This section
investigates which face BTP methods from Section II were evaluated for their
ability to satisfy this criterion and what techniques were used to conduct the
evaluation.

Table VIII summarises our findings, in terms of indicating whether
irreversibility for each face BTP method was evaluated theoretically or
empirically 666As per the ISO/IEC standard 30136 on Performance testing of
biometric template protection schemes (https://bit.ly/398Az9k), which broadly
suggests either a theoretical or empirical evaluation of irreversibility. (if
at all). In general, theoretical evaluations include proofs or quantified
estimates of irreversibility based on certain mathematical, cryptographic, or
information-theoretic assumptions; however, no experiments are performed to
simulate any inversion attacks in practice. Empirical evaluations, on the
other hand, attempt to prove or quantify irreversibility in terms of specific
attacks, which are simulated through practical experiments.

TABLE VIII: Irreversibility evaluation approaches among the studied face BTP
methods. Theoretical: Proofs/estimates of irreversibility without attack
simulations. Empirical: Proofs/estimates of irreversibility via experiments
simulating specific attacks.
Evaluations: ✔ = Some (numerical/conceptual); ✘ = None. Comparisons: ✔ = explicit; $\thickapprox$ = implicit/partial; ✘ = none. Method type | Reference | Irreversibility evaluated: | Compared to other BTP methods?
---|---|---|---
Theoretically | Empirically
Non-NN | [6] | ✔ | ✘ | ✘
[7] | ✔ | ✘ | ✘
[8] | ✔ | ✘ | ✘
[9] | ✔ | ✘ | ✘
[10] | ✔ | ✘ | ✘
[11] | ✔ | ✘ | ✘
[12] | ✔ | ✘ | ✘
[13] | ✔ | ✘ | ✘
[14] | ✘ | ✘ | ✘
[15] | ✘ | ✔ | ✘
[16] | ✔ | ✘ | ✘
[17] | ✘ | ✔ | ✔
[18] | ✘ | ✘ | ✘
[19] | ✘ | ✔ | ✘
[20] | ✘ | ✘ | ✘
[21] | ✔ | ✘ | ✘
[22] | ✘ | ✔ | ✘
[23] | ✔ | ✘ | ✔
[24] | ✔ | ✘ | ✘
[25] | ✔ | ✘ | ✘
[26] | ✘ | ✘ | ✘
[27] | ✘ | ✘ | ✘
| [28] | ✔ | ✘ | ✘
| [29] | ✔ | ✘ | ✘
| [30] | ✔ | ✘ | ✘
| [31] | ✔ | ✔ | ✘
NN-learned | [37, 38] | ✔ | ✔ | ✔
[39] | ✔ | ✔ | ✔
[40] | ✔ | ✔ | $\thickapprox$
[41] | ✔ | ✔ | $\thickapprox$
[42] | ✔ | ✘ | $\thickapprox$
[43] | ✘ | ✘ | ✘
[44] | ✔ | ✘ | $\thickapprox$
[45] | ✔ | ✔ | ✔
[46] | ✘ | ✔ | ✘
[47] | ✔ | ✘ | ✔
[48] | ✔ | ✘ | ✘
[49] | ✔ | ✘ | ✘
[50] | ✔ | ✘ | ✘
[51] | ✔ | ✘ | ✘

From Table VIII, it is clear that the irreversibility of most of the studied
face BTP methods was evaluated theoretically. The exact evaluation strategy
depended on the nature of the BTP method, so it would be tedious to present a
detailed examination of each technique here. Instead, we will discuss the most
widely adopted theoretical evaluation methods.

For the methods proposed in [6, 7, 8, 9, 10, 11, 12], which are based on
homomorphic encryption (HE), their irreversibility was attributed777HE-
protected templates in [12] are indexed by binary codes generated from the
original templates, so it is suggested that the codes be protected by
cryptographic hashing. Code irreversibility is assumed based on the hashing.
to: (i) the strength of the HE algorithm used to secure the face templates,
and (ii) the secrecy of the corresponding decryption key. In this case, if the
decryption key is leaked to an attacker, then the protected (encrypted) face
templates become completely reversible. Otherwise, the “degree” of
irreversibility was typically quantified in terms of the known security of the
employed HE scheme; for example, the upper-bound would be the length of the
decryption key in bits, $n$, which implies that at most $2^{n}$ guesses would
be required to guess the key using a brute-force approach.

The irreversibility of the face BTP methods in [13, 14, 24, 28, 37, 38, 39,
40, 41, 42, 45] was largely attributed to the use of a one-way cryptographic
hash function. So, it was generally assumed that the only way to “invert” the
hash was to attempt a brute-force attack, whereby all possible inputs to the
hashing function are tried until the target hash is obtained. For an $n$-bit
input, this amounts to an upper-bound of $2^{n}$ guesses. Note that this is
the same complexity as that for brute-force guessing the decryption key in HE
schemes (discussed earlier), except that this time the attacker is trying to
guess the input to the hashing function instead of the hash itself (since the
hash is not secret). The cryptographic hash function in [13, 45] was used to
directly secure the (binarised) face features, meaning that the input space
for the brute-force attack would be the binary face feature domain. In this
case, $2^{n}$ would probably be an over-estimation of the brute-force attack
complexity, since the binarised face features would not be uniformly
distributed (i.e., there is unlikely to be $2^{n}$ equally probable binary
feature vectors to guess from). On the other hand, the BTP methods in [14, 28,
37, 38, 39, 40, 41, 42, 45] hashed a random codeword that was either: (i)
bound with the face feature vector [14, 28], or (ii) designated as the pre-
hash protected face template, to which the original feature vector was then
mapped [37, 38, 39, 40, 41, 42]. In this case, the inputs in a brute-force
attack would come from the codeword domain; so, provided that the codewords
are generated from a uniform distribution, $2^{n}$ is a fair approximation of
the brute-force attack complexity.

Variations of brute-force or guessing attack complexity were also used to
theoretically estimate the irreversibility of face BTP methods that do not
rely (primarily) on cryptographic hash functions, including [25, 29, 47, 50,
51]. In general, however, basing an irreversibility evaluation on the
assumption of a brute-force attack would most probably result in an over-
estimation of the irreversibility in practice. This is because there will
often exist other, easier attacks, which would allow an adversary to invert
the protected template faster than using a brute-force approach. So, one way
to “tone down” the brute-force attack complexity estimate could be to
approximate the entropy of the input space (instead of assuming a uniform
distribution) and/or the reduction in the attacker’s uncertainty about the
input space (which would help them make more informed guesses). The latter is
usually estimated in terms of the amount of information leaked by the
protected template or certain auxiliary data, which is the “mutual
information” irreversibility evaluation method mentioned in the ISO/IEC
standard 30136 on Performance testing of biometric template protection
schemes. Variations of the entropy and mutual information irreversibility
evaluation approaches were adopted in [13, 23, 39, 48, 49].

Alternatively, we could turn to empirical irreversibility evaluation
strategies, which focus on practical attack simulations. Ideally, such
evaluations should demonstrate the existence (or lack thereof) of attacks
easier than brute-force, which could be used to invert a protected template in
practice. For example, of the aforementioned “binding” BTP methods that
cryptographically hash a random codeword, guessing the codeword would be
instrumental in “unbinding” the protected template and thus recovering the
original template. Instead of brute-force guessing the codeword, however, the
Fuzzy Commitment scheme in [14] may be invertible using the approach in [15],
while [28] is stated to be invertible if the user-specific linear
transformation matrix and the face template distribution are known. For the
BTP methods that hash a pre-defined codeword to represent the protected
template ([37, 38, 39, 40, 41, 42]), guessing this codeword from its hash
would, on its own, provide no information about the original face template
since the two are unrelated. So, although the brute-force attack complexity
seems to be a more appropriate measure of irreversibility for these types of
BTP methods, it is still unlikely that a brute-force attack would be the only
avenue for a determined attacker. For example, we may imagine that a fully-
informed attacker with access to the trained neural network and all its
parameters, along with a face dataset that is representative of the enrolled
subjects, could exploit this information to determine how the neural network
learns a mapping between a face template and a randomly chosen codeword. Note
that, as discussed in Section II.B, a neural network trained to map a face
template to a pre-defined random codeword is unlikely to be generalisable to
unseen (unenrolled) subjects. This implies that access to the trained network
without access the un-hashed codewords may not directly reveal a potential
inverse mapping. Nevertheless, this type of analysis may allow the attacker to
extract certain information about the mapping function in certain layers of
the neural network, which could narrow down the search space for the un-hashed
codeword and its corresponding face template. Although this evaluation may not
be trivial, it should still be considered as a worst-case scenario, since such
avenues are more likely to prove fruitful for a fully-informed attacker
(compared to attempting a brute-force attack on the cryptographic hash
itself).

It should be noted that [37, 38, 39, 40] presented an alternative evaluation
of irreversibility, besides a brute-force attack on the cryptographic hash. In
particular, they simulated a dictionary attack, where an adversary with access
to the trained neural network attempts a brute-force attack in the image
domain to exploit the system’s FAR, i.e., the attacker presents face images to
the protected system in the hope of finding a match to an enrolled identity.
The images used to simulate the attack, however, came from a different dataset
than those used to generate the enrolled protected templates. This means that
the “attack” images cannot be considered representative of the face images
corresponding to the enrolled subjects, so it is no surprise that the
dictionary attack failed. A similar evaluation was presented in [46]. In [41,
42], a dictionary attack was deemed unnecessary due to the zero FAR achieved
in the recognition accuracy evaluation. This low error rate, however, may be
attributed in large part to the use of user-specific transformations (random
projection in [41] and random perturbations in [42]), so an FAR of 0 may not
be observed if the transformation parameters are leaked to the attacker.

Besides a dictionary attack, other examples of empirical evaluations that
focus on exploiting the biometric system’s FAR, were presented in [17, 22]. In
[17], the irreversibility of the proposed BTP method was estimated (in bits)
as: $\texttt{False Accept Security (FAS)}=l\times\log(0.5)/\log(1-FMR)$ (where
$l$ = the average number of operations for a non-mated verification attempt).
The irreversibility of the BTP method in [22] was also evaluated in terms of
an empirical system attack, except that this time there was no guessing of the
inputs. Instead, the protected (warped) face images in [22] were directly
compared to their unprotected counterparts, demonstrating that insufficient
warping results in a high match accuracy.

Unquantified empirical irreversibility evaluations were presented in [19, 21],
which showed visual examples of face images that were “encrypted” using the
correct user-specific parameters and “decrypted” to a noise-like output using
the wrong parameters. This suggests that the irreversibility of these BTP
methods depends on keeping the user-specific transformation parameters secret
(like for HE or traditional encryption methods), but it does not give us an
indication of the irreversibility if the parameters are leaked to an attacker.

Finally, only [15, 31] presented an empirical irreversibility analysis in
terms of an actual inversion attack. This means that, instead of simply
guessing inputs that might generate the protected output, [15, 31] actually
attempted to invert the protection mechanism to recover the unprotected face
template from the protected template. As mentioned earlier, [15] demonstrated
that the Fuzzy Commitment (FC) scheme protecting the face template can be
broken. This was performed by firstly using a random set of face images to
guess a binary template that matches the one used in the FC binding, then
using the guessed binary template to reconstruct the corresponding real-valued
face feature vector, and finally using the recovered feature vector to
reconstruct a close approximation of the corresponding face image. The
reconstruction steps were achieved using neural networks trained for the
inversion. Alternatively, the BTP method in [31], which was defined via a set
of user-specific multivariate polynomials, was evaluated by attempting to
invert the polynomial mappings using a numerical solver (assuming a fully
informed attacker, with access to all parameters and knowledge of a
representative face feature distribution). The recovered real-valued feature
vector was compared to the original feature vector, which was assumed to be
enrolled in an unprotected face recognition system, based on the system’s
match threshold. The empirical irreversibility of the BTP method was
quantified in terms of the $\texttt{inversion success rate}=\texttt{solution
rate}\times\texttt{match rate}$. A similar quantification, based on the
closeness between the recovered and original face templates, was also
performed in [15]. For both methods, the face data used to obtain the required
“guesses” (i.e., the binary templates in [15] and the initial guesses for the
numerical solver in [31]) came from the same dataset as the enrolled subjects
(thereby fairly representing the underlying face feature distributions),
except that a different set of subjects was used for this purpose.

The last observation from Table VIII is that hardly any of the studied face
BTP works presented an explicit comparison of the irreversibility of their
method to that of other face BTP methods. Of those that did, [17] compared the
irreversibility in terms of the proposed FAS measure to the irreversibility
estimates reported for other face-based Fuzzy Vault BTP methods. Most of these
other methods, however, were evaluated in terms of brute-force attack
complexity, and the evaluations were performed on different datasets using
different feature extractors. So, the presented irreversibility comparison
does not seem fair. In [23], the entropy-based measure of irreversibility was
compared to that of a couple of other BTP methods. Even though the comparison
seems reasonable in that the same metric was employed, it is not explained how
the parameters of these different methods were selected to ensure that the
resulting irreversibility strengths were fairly compared. The irreversibility
of the methods in [37, 38, 39, 45] was compared to that of other BTP methods
based on “bits of security”, which was represented by the length of their
binary codes. This is because, as discussed earlier, the irreversibility of
these methods was evaluated in terms of brute-force attack complexity, which
is equivalent to $2^{n}$ for $n$-bit binary codes. One issue with this
comparison (apart from the fact that brute-force attack complexity is likely
to be an over-estimate of the true irreversibility) is that it is not always
clear: (i) how the equivalent code length parameter was established for all
the other methods, and (ii) whether the value of the chosen code length
parameter is a valid measure of the irreversibility of all these other
methods.

Finally, Table VIII indicates that [40, 41, 42, 44] presented implicit or
partial comparisons of the irreversibility of their proposed BTP methods to
that of other face BTP methods in the literature. More specifically, similarly
to [37, 38, 39, 45], the works in [40, 41, 42] also provided tables with the
“code length” of the binary codes resulting from different BTP methods;
however, they did not explicitly relate these parameter values to a comparison
of the methods’ irreversibility (or security). On the other hand, [44]
presented a subjective (in terms of: High, Medium, Low) comparison of the
irreversibility of their method to that of other data protection methods, but
the chosen methods were purely cryptographic hashing and encryption functions
as opposed to (face) BTP algorithms.

Although not noted in Table VIII, we also looked at which face BTP methods
evaluated irreversibility in terms of a Record Multiplicity Attack (ARM). This
refers to the scenario where an attacker is assumed to have access to multiple
protected templates from the same subject’s face, and they combine this
information to try to invert the protected templates. Of the face BTP methods
studied in this paper, ARM was evaluated only for [50, 48, 29, 30, 31]. A
theoretical ARM analysis was presented888For [50, 29], we are referred to
[36], on which [50, 29] are based. in [50, 48, 29], where it was concluded
that this type of attack cannot reduce the security of the protected system;
however, in all cases, the assumption was that certain user-specific
parameters are not leaked to the attacker. ARM was also evaluated999We are
referred to [101], where [30] was applied to iris biometrics. from a
theoretical angle in [30], where various scenarios, with different information
assumed to have been leaked to the attacker, were considered. The analysis was
based on showing that the attack would require solving underdetermined systems
of equations, which have no unique solution; however, there was no practical
attempt at solving the systems using a numerical solver, unlike in [31]. In
[31], an empirical ARM evaluation was presented for the scenario of a fully-
informed attacker with access to all system parameters. The evaluation adopted
a numerical solver to solve a system of non-linear equations, which
represented the (inverse) mapping from each stolen protected template to the
original template. As expected, the protected system was shown to be somewhat
susceptible to ARM in the worst-case scenario of a fully-informed attacker;
however, it is doubtful that any BTP method would be able to perfectly resist
this type of attack in practice if the attacker had access to everything.

Overall, our investigation into irreversibility evaluation techniques among
the existing face BTP methods revealed that several interesting strategies
have been employed, depending on the nature of the BTP algorithm. Many of
these techniques, however, may result in inadequate or misleading
representations of the irreversibility in practice, especially in the worst-
case scenario of a fully-informed attacker (which is the most difficult threat
model outlined in the ISO/IEC standard 30136 on Performance testing of
biometric template protection schemes). This is usually because the protection
method is assumed to rely on the secrecy of certain parameters, or because the
degree of irreversibility is over-estimated as a result of limited
investigation into attacks that are easier/smarter than brute-force. For
example, for NN-learned BTP methods, it is conceivable that an attacker with
access to the trained BTP neural network (i.e., its architecture and all
learned parameters) could use this knowledge to extract certain information
about the learned protection algorithm in different layers of the network,
which could in turn be used to uncover hints about certain characteristics of
the unprotected face templates. This type of analysis is currently lacking in
the face BTP literature.

More generally, based on the summary in Table VIII, we conclude that there is
a noticeable shortage of empirical irreversibility evaluations. Empirical
evaluations encourage us to consider the irreversibility of our BTP methods
from the point of view of a determined attacker, thereby providing more
insight into the strength of these methods in practice (albeit usually in a
worst-case scenario). Such practically-oriented types of evaluations may be
expected to become increasingly important in the future, considering the
growing interest in protecting face templates in real-world face recognition
systems. Furthermore, empirical evaluations may make it easier to compare the
irreversibility of different BTP methods in more realistic terms. We would,
therefore, like to encourage the BTP community to invest more effort towards
developing irreversibility evaluation strategies that will: (i) better reflect
the expected irreversibility of our BTP methods in practice, and (ii) allow
for more meaningful comparisons of the irreversibility of different BTP
methods.

###  III-D Renewability/Unlinkability

A BTP method is considered to satisfy the “renewability/unlinkability”
criterion if it can generate multiple diverse protected templates from the
same original template, such that they cannot be linked to the same identity.
This would allow for compromised templates to be cancelled/revoked and
renewed, as well as enabling the same biometric characteristic to be used in
different applications without the risk of cross-matching the enrolled
protected templates.

Note that the ISO/IEC standard 30136 on Performance testing of biometric
template protection schemes defines “diversity” as the expected value of the
number of independent protected templates that can be extracted from the same
biometric identity, which directly ensures “renewability” and “revocability”
(or “cancellability”). “Unlinkability” is then defined separately as the
property that two protected templates cannot be linked to each other or to the
subject(s) from which they were derived. In the face BTP literature, however,
these terms often seem to be used interchangeably, which is understandable
considering the inter-dependence of the underlying concepts. So, for the
purposes of this literature survey, we consider all the aforementioned terms
to define essentially the same property or criterion of a face BTP method,
which we refer to as “renewability/unlinkability”, i.e., can the BTP method
ensure the renewability of protected templates by being able to create
multiple unlinkable protected templates from the same face identity? This
section explores whether this criterion was evaluated, and what techniques
were used in the analysis, for the face BTP methods from Section II.

Table IX presents the findings of our renewability/unlinkability investigation
in terms of indicating whether this criterion was evaluated via verbal
implication, theoretical proof, or empirical validation (if at all) for each
face BTP method. Verbal implications include statements implying that the
criterion is satisfied (either directly or indirectly), but without presenting
any theoretical or empirical proof. Theoretical proofs include mathematical
reasoning for why the criterion is believed to be fulfilled, or theoretical
estimations of the diversity. Finally, empirical validations include
experimental attempts to prove and/or quantify the extent to which this
criterion is satisfied in a practical setting.

TABLE IX: Renewability/unlinkability evaluation approaches among the studied face BTP methods. Verbal implication: Statement(s) implying that the criterion is satisfied (no analysis). Theoretical proof: Mathematical reasoning or theoretical estimate(s) proving that the criterion is satisfied. Empirical validation: Experiment(s) proving or quantifying the extent to which the criterion is satisfied. Key: ✔ = Some evaluation; ✘ = No evaluation. Method type | Reference | Renewability/Unlinkability evaluated via: | Compared to other BTP methods?
---|---|---|---
Verbal implication | Theoretical proof | Empirical validation
Non-NN | [6] | ✘ | ✘ | ✘ | ✘
[7] | ✔ | ✘ | ✘ | ✘
[8] | ✔ | ✘ | ✘ | ✘
[9] | ✔ | ✘ | ✘ | ✘
[10] | ✔ | ✘ | ✘ | ✘
[11] | ✔ | ✘ | ✘ | ✘
[12] | ✔ | ✘ | ✘ | ✘
[13] | ✘ | ✘ | ✘ | ✘
[14] | ✔ | ✘ | ✘ | ✘
[15] | ✘ | ✘ | ✔ | ✘
[16] | ✘ | ✔ | ✘ | ✘
[17] | ✘ | ✔ | ✘ | ✘
[18] | ✘ | ✘ | ✘ | ✘
[19] | ✘ | ✘ | ✘ | ✘
[20] | ✘ | ✘ | ✘ | ✘
[21] | ✘ | ✘ | ✘ | ✘
[22] | ✘ | ✘ | ✔ | ✘
[23] | ✘ | ✘ | ✔ | ✘
[24] | ✔ | ✘ | ✘ | ✘
[25] | ✔ | ✘ | ✘ | ✘
[26] | ✘ | ✘ | ✘ | ✘
[27] | ✘ | ✘ | ✔ | ✘
| [28] | ✘ | ✔ | ✘ | ✘
| [29] | ✔ | ✘ | ✘ | ✘
| [30] | ✔ | ✘ | ✘ | ✘
| [31] | ✘ | ✘ | ✔ | ✔
NN-learned | [37, 38] | ✔ | ✘ | ✘ | ✘
[39] | ✔ | ✘ | ✘ | ✘
[40] | ✔ | ✘ | ✘ | ✘
[41] | ✔ | ✘ | ✘ | ✘
[42] | ✔ | ✘ | ✘ | ✘
[43] | ✘ | ✘ | ✘ | ✘
[44] | ✘ | ✘ | ✘ | ✘
[45] | ✘ | ✘ | ✘ | ✘
[46] | ✔ | ✘ | ✘ | ✘
[47] | ✔ | ✘ | ✘ | ✘
[48] | ✘ | ✘ | ✔ | ✘
[49] | ✘ | ✘ | ✔ | ✔
[50] | ✔ | ✘ | ✘ | ✘
[51] | ✘ | ✔ | ✔ | ✘

There are several important observations from Table IX. Firstly, it shows that
the renewability/unlinkability criterion for most of the studied face BTP
methods ([7, 8, 9, 10, 11, 12, 14, 24, 25, 29, 30, 37, 38, 39, 40, 41, 42, 46,
47, 50]) was evaluated via verbal implication. In general, this means that
renewability (or revocability/cancellability/diversity) was claimed to be
possible by changing certain (external) user-specific parameters, and
unlinkability was assumed due to the randomness of these parameters; however,
no theoretical or experimental proof was provided to formally validate these
implications. For example, [7, 8, 9, 10, 11, 12], which used homomorphic
encryption (HE) to secure face templates, implied that the
renewability/unlinkability criterion can be satisfied by using different
encryption keys to generate different ciphertexts (protected templates) from
the same subject’s face templates. Although the nature of encryption
operations, in general, suggests that the resulting ciphertexts should be very
different, no analysis has been provided to demonstrate the extent of these
differences in the context of face template protection. Furthermore, we know
that face templates protected via HE are fully invertible if the decryption
key is leaked, yet this has not been considered in evaluating the
renewability/unlinkability criterion.

Other examples of renewability/unlinkability evaluations via verbal
implication can be found in [37, 38, 39, 40, 41, 42, 24], where all the
proposed face BTP methods involved training a neural network or other machine
learning algorithm to map a subject’s face template to a random, pre-defined
binary codeword (which was then cryptographically hashed). Since the codewords
were randomly generated (and further “randomised” by the hashing operation),
they were assumed to be uncorrelated with the face templates to which they
have been assigned. So, renewability/unlinkability was said to be achievable
by changing the user-specific codewords and re-training the network to learn
the mapping between a subject’s face template and their new codeword. This
appears to be a reasonable justification for fulfilment of the
renewability/unlinkability criterion; however, there is no analysis to
demonstrate how scalable the renewability effort would be in practice (i.e.,
in terms of the re-training process for each new template).

Most of the other works that verbally implied the fulfilment of the
renewability/unlinkability criterion ([14, 25, 29, 30, 47, 50]), also relied
on their method’s ability to update certain user-specific parameters. Unlike
[7, 8, 9, 10, 11, 12], for which fulfilment of this criterion can be related
to the nature of existing HE algorithms, or [37, 38, 39, 40, 41, 42, 24], for
which adherence to the criterion is justifiable by the randomness of the pre-
defined protected templates, the effect on renewability/unlinkability by
changing the user-specific parameters in [14, 25, 29, 30, 47, 50] is not as
readily evident. In particular, although it is usually easy to deduce that
using different parameters would result in the generation of a different
protected template for the same subject, it is difficult to imagine whether or
not those protected templates would be unlinkable (or the extent of their
unlinkability). In this case, a theoretical and/or empirical proof may be
considered necessary for backing up the verbal claims.

To complete the analysis on verbal implication renewability/unlinkability
evaluations, we should mention the approach in [46] 101010An ensemble network
classifies a fused face feature vector (Section II.B).. In this work, it was
claimed that the lack of template storage means that the protected template
cannot be compromised and cancellability is, therefore, ensured. This cannot
be considered a justification for cancellability, however, since there is
insufficient analysis to prove that the trained model does not leak any
information about the face templates it is used to classify. Also, diversity
was said to be ensured since the trained model cannot be used across different
applications, but this is the opposite of the accepted notion of diversity.

The second important observation from Table IX is that very few (i.e., 4) of
the studied face BTP works presented a renewability/unlinkability evaluation
via a theoretical proof. Of these methods, [16] justified adherence to this
criterion in terms of renewable codewords in a Fuzzy Commitment construct,
which were considered as lattice points spaced sufficiently apart to avoid
cross-matching of different protected templates (which are bound with the
codewords). A Fuzzy Vault scheme was adopted in [17], and a theoretical
analysis was presented to prove that the probability of correlating two fuzzy
vaults from the same subject using the extended Euclidean algorithm is
negligible. Similarly, [28] provided an in-depth proof of the complexity of
finding a link between several of a person’s protected face templates,
relating the justification of the method’s revocability and unlinkability to
its irreversibility. Finally, although the main renewability/unlinkability
evaluation in [51] was actually empirical, a theoretical estimation of the
number of cancellable protected templates that can be generated from a single
face template was provided. This is the only one of the studied face BTP
methods that provides such an estimate, although this is more likely to be an
over-estimate of the true number in practice (since it is unlikely that all
possible protected templates will be sufficiently different from each other to
ensure unlinkability).

The third observation from Table IX is that fewer than a quarter of the face
BTP works ([15, 22, 23, 27, 31, 48, 49, 51]) presented an empirical evaluation
of their methods’ renewability/unlinkability. Most of these ([22, 31, 48, 49,
51]) adopted the unlinkability evaluation framework from [102], which begins
by generating several111111[102] recommends at least 5, but uses 10 in own
experiments. protected templates per subject (using different transformation
parameters). Then, each protected template is compared to the same subject’s
protected templates to generate a set of mated comparison scores and to
protected templates from all other subjects to generate non-mated scores. The