# DuetFace: Collaborative Privacy-Preserving Face Recognition via Channel
Splitting in the Frequency Domain

Yuxi Mi<EMAIL_ADDRESS>0000-0002-1006-6041 Fudan UniversityShanghaiChina
,  Yuge Huang<EMAIL_ADDRESS>0000-0001-5387-5992 Tencent Youtu
LabShanghaiChina ,  Jiazhen Ji<EMAIL_ADDRESS>0000-0003-2708-9319 Tencent
Youtu LabShanghaiChina ,  Hongquan Liu<EMAIL_ADDRESS>0000-0003-2544-4452 Fudan UniversityShanghaiChina ,  Xingkun Xu
<EMAIL_ADDRESS>0000-0001-6399-3415 Tencent Youtu LabShanghaiChina ,
Shouhong Ding<EMAIL_ADDRESS>0000-0002-3175-3553 Tencent Youtu
LabShanghaiChina and  Shuigeng Zhou<EMAIL_ADDRESS>0000-0002-1949-2768
Fudan UniversityShanghaiChina

(2022)

###### Abstract.

With the wide application of face recognition systems, there is rising concern
that original face images could be exposed to malicious intents and
consequently cause personal privacy breaches. This paper presents DuetFace, a
novel privacy-preserving face recognition method that employs collaborative
inference in the frequency domain. Starting from a counterintuitive discovery
that face recognition can achieve surprisingly good performance with only
visually indistinguishable high-frequency channels, this method designs a
credible split of frequency channels by their cruciality for visualization and
operates the server-side model on non-crucial channels. However, the model
degrades in its attention to facial features due to the missing visual
information. To compensate, the method introduces a plug-in interactive block
to allow attention transfer from the client-side by producing a feature mask.
The mask is further refined by deriving and overlaying a facial region of
interest (ROI). Extensive experiments on multiple datasets validate the
effectiveness of the proposed method in protecting face images from undesired
visual inspection, reconstruction, and identification while maintaining high
task availability and performance. Results show that the proposed method
achieves a comparable recognition accuracy and computation cost to the
unprotected ArcFace and outperforms the state-of-the-art privacy-preserving
methods. The source code is available at
https://github.com/Tencent/TFace/tree/master/recognition/tasks/duetface.

face recognition, data privacy, deep learning, channel splitting

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
30th ACM International Conference on Multimedia ; October 10–14, 2022; Lisbon,
Portugal.††booktitle: Proceedings of the 30th ACM International Conference on
Multimedia (MM ’22), October 10–14, 2022, Lisbon, Portugal††price:
15.00††isbn: 978-1-4503-9203-7/22/10††doi:
10.1145/3503161.3548303††submissionid: mmfp2391††ccs: Security and privacy
Privacy protections††ccs: Computer systems organization Neural networks

##  1\. Introduction

_Face recognition_ (FR) has been a phenomenal biometric method for identity
authentication, with lots of remarkable breakthroughs gained in recent years.
As face recognition is widely incorporated into applications in such as
finance, health, and public security, there rises a growing concern about the
privacy of sensitive facial images. The _privacy-preserving face recognition_
(PPFR) technique thus has arrested the attention of academia and the industry.

In a typical privacy-preserving face recognition scenario, a query face image
is collected and held by local devices such as cell phones and webcams. As
often constrained by computation power, they outsource the recognition task to
a third-party service provider, who infers on a SOTA model pre-trained on
massive labeled datasets. For clarity, we thereafter refer to the image holder
as the client and the service provider as the server. As the query face here
is considered private, the client is not willing to share the raw image with
others, including the server.

In the recent decade, vast advancements have been incorporated into PPFR
applications. We coarsely categorize them into two branches: the encrypt-based
methods build the privacy of the face image on the bricks of cryptographic
primitives and security protocols. Privacy is well protected as long as these
bricks are provably or computationally secure. These methods, however, often
bear prohibitive computation and communication costs. On the other hand, the
transform-based methods perturb or regenerate the face image into a new
representation that is visually indistinguishable from the server’s
perspective. The drawback of these methods is the almost inevitable
participation of noise or loss of information, which results in a downgrade of
recognition accuracy. Prior transform-based arts, therefore, generally face a
paradoxical trade-off between privacy and accuracy.

Figure 1. A paradigm comparison among unprotected FR (top), prior PPFR works
(middle) and our method (bottom). The images are directly shared in
unprotected FR, while are encrypted or transformed in PPFR. The server in
prior works performs recognition alone, which usually suffers from a downgrade
in accuracy. In our method, the recognition is performed through the
collaboration of the server and client.

The comparison of paradigm among FR, prior PPFR works, and our method.

Processing images in the frequency domain, first practically employed in the
JPEG standard, is a time-honored approach for image compression. As a
byproduct, prior researches observe that a small number of low-frequency
channels aggregate most of the visual features that are believed crucial for
recognition. We advance this observation with a counterintuitive discovery:
_the recognition can achieve a surprisingly acceptable accuracy even upon the
removal of these crucial channels, while with far less visual information
revealed_. We exploit this discovery as a starting point for our work.

Our work employs a transform-based approach as well. Yet, we propose a novel
way to effectively solve the privacy-accuracy paradox by addressing them
successively through a collaboration between the server and the client.
Concretely, let the server possess a SOTA model, whereas the client holds a
lightweight model that is an order of magnitude smaller in complexity and
cost. We first _split the frequency channels_ into two parts by their
cruciality for visualization, and let the server infer the non-crucial part.
The visual information is therefore concealed from the server and, based on
our novel discovery, at the price of tolerable performance degradation.
Subsequently, we improve the accuracy by allowing the client to correct the
server-side inaccurate attention on facial features, concretely, by
_transferring the client-side attention_ to the server through a proposed
interactive block. For a better understanding of our idea, a paradigm
comparison among unprotected FR, prior PPFR works, and our method is
illustrated in Fig. 1. Extensive experiments validate the effectiveness of our
scheme.

In summary, the contributions of our paper are three-fold:

  1. (1)

We propose a novel face recognition paradigm by combining the collaborative
efforts of two parties: the client and the server. We develop a novel PPFR
framework, referred to as DuetFace, which is satisfactory in recognition
accuracy, good in cost-efficiency, and reliable in privacy protection;

  2. (2)

We introduce a channel splitting scheme to derive appropriate image break-ups
and devise an interactive block based on ROI-refined feature masks to allow
attention transfer;

  3. (3)

We conduct extensive experiments on multiple datasets, which validate the
effectiveness and superiority of the proposed method.

##  2\. Related Work

###  2.1. Privacy-Preserving Face Recognition

Significant advances of privacy-preserving face recognition (PPFR) has been
achieved significant progress in the past decade, which can be roughly divided
into two categories:

Encrypt-based methods. In this branch of work, face recognition is carried out
on encrypted domains. Necessary computations such as feature extraction and
similarity calculation are either accomplished straightly on the encrypted
images or by executing certain security protocols. Pioneering works (Sadeghi
et al., 2009; Erkin et al., 2009; Huang et al., 2020) apply homomorphic
encryption (HE) and garbled circuits (GC) to the Eigenface recognition
algorithm to hide raw image features from undesired parties. Methods with
similar intentions also employ other cryptographic primitives including matrix
encryption (Kou et al., 2021), one-time-pad (Ergun, 2014), and functional
encryption (Abdalla et al., 2015). Dedicated secure multiparty computation
(MPC) schemes are introduced in (Ma et al., 2019; Yang et al., 2018; Xiang et
al., 2016) to perform certain operations (_e.g._ parameter comparison) in a
protected manner, or to outsource parts of the job to a trusted third party.

Encrypt-based methods have very little degradation on the recognition accuracy
since almost all the operations involved are lossless. Their effectiveness is
also strongly guaranteed by the provable or computational security of the
employed cryptographic primitives. However, the practical usages of these
methods are limited as they mostly bear intolerable computation costs and
communication overhead. Moreover, these methods have low generalizability
since most of them are tightly coupled with very specific face recognition
schemes. Our work is more generalizable and can achieve competitive accuracy
to the standard ArcFace with much less cost.

Transform-based methods. Another active line of research transforms face
images into perturbed or regenerated representations to reduce their
distinguishability from untrusted parties. For perturbation, differential
privacy is employed by lots of works (Zhang et al., 2019; Chamikara et al.,
2020; Li et al., 2019; Mao et al., 2018) where raw images are overlayed by
noise mechanisms to make them less visually differentiable. To increase
anonymity, Honda et al. (Honda et al., 2015) proposed a clustering-based
method by mapping raw images to their class representations. As for means to
regenerate images into new representations, some works exploit deep-learning-
based methods such as adversarial generative network (GAN) (Mirjalili et al.,
2018a; Li et al., 2019) and autoencoder (Mirjalili et al., 2018b), some alter
facial landmarks (Mirjalili and Ross, 2017), some employ channel shuffling
(Wang et al., 2022), and some remove redundant information by mapping
discriminative components into subspaces (Kevenaar et al., 2005; Chanyaswad et
al., 2016; Mireshghallah et al., 2021).

These methods usually face a trade-off in balancing privacy and accuracy since
the means of protecting the images are performed, in essence, either by
bringing in noise or discarding features, which bring in information loss with
high probability. Our proposed method overcomes the trade-off and reduces the
accuracy loss to a minimum level.

Figure 2. Architecture of DuetFace. The query image is split by the energy of
its frequency channels into two parts and inferred separately by the client
and the server. To compensate for accuracy loss, an interactive block is
plugged in at the end of each stage, where a feature mask transfers the
attention from the client to the server. A facial ROI is employed to refine
the mask.

The architecture of DuetFace.

The recently proposed PPFR-FD (Wang et al., 2022) is related to ours as both
works study the contribution of channels on visualization and recognition.
However, our findings differ from it in two aspects: (1) PPFR-FD begins with
the premise that the lowest frequency channel contributes not much to the
recognition task, which is quite different from ours; (2) The high-frequency
channels are removed in PPFR-FD as they are believed to contribute little to
distinguishability. These channels are instead retained and fully exploited in
our work. Further, PPFR-FD realizes privacy mainly by randomizing the order of
channels, while ours by removing visual components.

###  2.2. Learning in the Frequency Domain

Learning in the frequency domain is traditionally leveraged for image
compression, which allows retaining meaningful patterns for image
understanding tasks through compressed representations. Prior arts in face
recognition (Torfason et al., 2018; Xu et al., 2018) train autoencoder-based
networks to perform compression and inference tasks simultaneously. (dos
Santos and Almeida, 2021) first performs image classification in the frequency
domain directly. (Xu et al., 2020) proposes an accuracy-retaining image down-
sampling method, where spatial images are reorganized in the frequency domain
to remove the non-crucial channels.

##  3\. Methodology

###  3.1. Overview

We here describe the proposed privacy-preserving face recognition method,
referred to as DuetFace. In the world of art, a duet is a performance by two
singers, instrumentalists, or dancers. Similarly, in DuetFace, the inference
is carried out together by two parties, _i.e._ , the server and the client.

Our discovery and motivation. Our work starts from a counter-intuitive
discovery. Previous research in the frequency domain (Wang et al., 2020)
suggests that the recognition is mainly determined by channels with lower
frequency, which in the meantime are those with larger amplitude (Xu et al.,
2020), as they contribute most of the visual information. Yet, those prior
arts mostly ignore the value of the rest “non-crucial” channels. We, on the
contrary, surprisingly find that the models can also perform recognition with
quite acceptable accuracy by using _only the visually indistinguishable high-
frequency channels_. To illustrate this, we train multiple recognition models,
each with different numbers of lowest-frequency channels discarded in every
color component, then evaluate the trained models on 5 public datasets.
Results in Fig. 3(b)(c) show that the models are able to maintain a quite
decent performance even the remaining channels possess ¡10% energy, although
there certainly exists an accuracy gap due to the lack of visual information.
This discovery allows us to construct our PPFR method from a completely
different view.

Figure 3. The contributions of channels to visualization and recognition. (a)
The low-frequency channels on the top-left account for most of the visual
information, in terms of channel energy. (b) The remaining energy sharply
decreases as the low-frequency channels are discarded. (c) However, our
experiments show that the models are able to maintain decent accuracy even
with most visual information removed.

Channel splitting.

Concretely, we introduce a collaborative paradigm between the server and the
client. We first design an appropriate split of the query image in the
frequency domain, and let the server train on the high-frequency components in
a privacy-preserving manner with tolerable accuracy loss. Then, we let the
client further refine the server-side performance by compensating for the
missing information without revealing the image itself.

The ability of the server and the client. To better explain the motivation of
our method, we start by characterizing the parties. We assume the server to be
semi-honest and the client to be resource-sensitive. A semi-honest server is
one who honestly follows the face recognition protocol and provides correct
results while trying to learn as much as possible from the messages sent by
the client. The server is an abstraction of corrupted or unregulated service
providers, who may collect, use, and redistribute face images unauthorizedly.
A resource-sensitive client is one bounded by limited storage, bandwidth, and
computation power. The real-world clients are often reified as personal
devices such as cell phones and webcams, whose owners may be unwilling to
download and store large models locally or perform complex inference tasks.

The security goals of DuetFace. We address the inference-time privacy between
the semi-honest server $\mathcal{S}$ and the client $\mathcal{C}$. The client
possesses a query face image $X$ that it wants the server to identify. The
image is considered private. We denote all the information related to $X$ that
the client could share without causing a privacy breach as $I(X)$. Our privacy
consideration is to prevent the unauthorized collection, use, and
redistribution of $X$, which we concretize into three security goals:

  1. (1)

Visual privacy. As the very basis, the server should be unable to collect
useful information from the visual appearance of the face image $X$;

  2. (2)

Privacy against reconstruction. The server may try to reconstruct the
information it misses about $X$. We ergo prescribe, by leveraging $I(X)$, the
server cannot effectively produce a reconstruction $X^{\prime}$ of $X$;

  3. (3)

Privacy against identity inference. The server could redistribute the
reconstructed image to a third party. Plus, the message could be intercepted
during transmission. To prevent potential privacy leakage, we require that,
without accessing the recognition model, acquiring either $I(X)$ or
$X^{\prime}$ should be insufficient to infer the identity of $X$.

The paradigm of DuetFace. Let the server and the client each hold a local
model $M_{s}$ and $M_{c}$, respectively. Here, $M_{s}$ is a full-size state-
of-the-art model used to answer the recognition requests, whereas $M_{c}$ is a
lightweight model applied as an aid. Both models are pre-trained by the server
and $M_{c}$ is downloaded by the client. To infer a query image $X$, the
client first splits $X$ into two appropriate parts $X_{s}$ and $X_{c}$, where
visual information is removed from $X_{s}$ but retained in $X_{c}$. The client
shares $X_{s}$ with the server and keeps $X_{c}$ to itself. The server
identifies $X_{s}$ via its model $M_{s}$. Note that the loss of visual
information in $X_{s}$ will surely degrade the performance of the server-side
model. To compensate for the information loss, the client infers the other
part $X_{c}$ of the image $X$ on $M_{c}$ at a very low computation cost, to
obtain a concise supplementary representation $R(X_{c})$, which will be
revealed to the server. Finally, the server leverages $R(X_{c})$ to refine its
judgment and produce better results.

Fig. 2 shows the architecture of DuetFace, which consists of three major
components: Channel splitting, attention transfer, and mask denoising.
DuetFace is cost-efficient and fulfills the above-mentioned security goals at
decent accuracy.

###  3.2. Splitting Channels in Frequency Domain

As the very first step of our proposed method, the image $X$ is split by
channel frequency. To transform the face image to the frequency domain, we
first follow the common data pre-processing protocol in the spatial domain to
crop, resize and horizontally flip the face image, and obtain an input shape
of $H$×$W$×3. Then, we convert the image from RGB to YCbCr color space and
subsequently to the frequency domain by carrying out the block discrete cosine
transform (BDCT) following the same way in JPEG compression (Wallace, 1991).
We also perform an 8-fold bilinear up-sampling right before BDCT. As standard
BDCT maps each 8×8 pixel block into one frequency channel (here, maps
8$H$×8$W$×3 to $H$×$W$×192), the up-sampling enables us to maintain the
channel shape and minimize the modifications to the recognition backbone.

We construct a credible split $\\{X_{s},X_{c}\\}$ of $X$ by the amplitude of
channels. Here, we measure amplitude by channel energy: given a channel, its
energy is the mean of absolute values of all its elements. We first split
channels by energy on the luma (Y) component of YCbCr color space, as it
carries most of the visual profiles and features (Xu et al., 2020). As shown
in Fig. 3(a)(b), the low-frequency channels on the top-left corner contribute
to ¿90% of total energy. We select $K$ channels with the highest energy as the
_crucial channels_ and regard the rest as _non-crucial ones_. To achieve
spatial consistency, the same selection is also applied to the chroma (Cb, Cr)
components. Then, we produce $\\{X_{s},X_{c}\\}$ utilizing the split channels.
We form $X_{c}$ by concatenating the crucial channels (in the shape of
$H$×$W$×3$K$), and $X_{s}$ by the rest. Therefore, visual information is
convincingly removed from $X_{s}$ but retained in $X_{c}$, as illustrated in
Sec. 4.4.

We adjust the input shapes of the models $M_{s}$ and $M_{c}$ to meet the
shapes of $X_{s}$ and $X_{c}$, respectively. It is widely observed that
discarding non-crucial frequency channels affects the model very slightly
(Wang et al., 2020; Xu et al., 2020; Wang et al., 2022), so we expect no
performance change on $M_{c}$. As of $M_{s}$, results in Sec. 4.6 show that
the model suffers a tolerable accuracy gap in the absence of visual
information, which we are to compensate for in the following subsections.

###  3.3. Attention Transfer

Figure 4. Visualization of each model layer via Grad-CAM. The server-side
attention is inaccurate on facial features, while the accurate attention is
retained by the client.

Visualization via Grad-CAM.

We ascribe the downgrade of $M_{s}$ to the inaccuracy of model attention. To
explain, we visualize the top-down attention maps for each layer of $M_{s}$
and $M_{c}$ via Grad-CAM (Selvaraju et al., 2020). As shown in Fig. 4, after
discarding the crucial channels, the attention of the server-side model
$M_{s}$ could not focus on the effective visual features such as facial
contours and the positions of eyes, noses, and lips, which are generally
considered the indispensable information for high-quality face recognition. On
the other hand, we observe that accurate attention is retained by the client-
side $M_{c}$. To restore the missing information in $M_{s}$, we propose an
_interactive block_ $IB$ that teaches the server-side $M_{s}$ about the
attention on the client-side.

Currently, many popular face recognition backbones employ a stage-by-stage
architecture to learn features of different levels. Assume $M_{s}$ and $M_{c}$
each has $n$ stages, we plug in our interactive block at the end of each
stage. Concretely, for face image $X$, denote the feature maps of
$M_{s},M_{c}$ at the end of stage $i$ as $F_{i}(X_{s}),F_{i}(X_{c})$,
respectively. When querying $X$, the client first infers $X_{c}$ on $M_{c}$ to
produce the feature map $F_{i}(X_{c})$, and calculates its channel-wise
average to obtain a one-channel feature mask $R_{i}(X_{c})$. The client then
normalizes $R_{i}(X_{c})$ to $[0,1]$ and transmits it to the server. After
receiving $R_{i}(X_{c})$, the server first resizes it to align with the height
and width of $F_{i}(X_{s})$. Then the server activates its own feature
$F_{i}(X_{s})$ by passing it through a sigmoid function and updates it as:

$F_{i}^{{}^{\prime}}(X_{s})=w_{i}\times(F_{i}(X_{s})\odot
R_{i}(X_{c}))+F_{i}(X_{s}),$

where $\odot$ is element-wise multiplication and $w_{i}$ is a trainable weight
that decides the impact of the feature map on the server-side feature.

Note that we use the word “interactive” to describe the function of the block.
In practice, the client is not required to jointly compute with the server for
each block in real-time. Instead, it just infers $M_{c}$ once to retrieve and
calculate the feature masks of all the stages
$R(X_{c})=\\{R_{i}(X_{c}),\cdots,R_{n}(X_{c})\\}$, and sends it together with
$X_{s}$ to the server. This simplifies the process.

Figure 5. Illustration of attention transfer. (a) The mask $R_{i}(X_{c})$
effectively transfers facial features from the client to the server for
correcting the latter’s inaccurate attention. (b) Effective masks highlight
the facial contour correctly, while ineffective masks rather highlight the
surroundings.

Illustration of attention transfer.

Fig. 5(a) illustrates attention transfer at stages 0 and 1 with samples of the
server-side feature map before (left) and after (right) the interactive block,
as well as the feature mask $R_{i}(X_{c})$ (middle). The histograms show their
corresponding value distributions. As inferred from both visualization and
data distribution, the mask effectively transfers knowledge of features such
as the entire facial contour (in stage 0) and the approximate positions of
eyes and lips (in stage 1) to the server-side. The attention transfer leads to
performance improvement of $M_{s}$, as demonstrated in Sec. 4.6. To further
improve the precision of the feature mask, we also propose a fast and
effective denoising method detailed in the following subsection.

###  3.4. Denoising the Mask by Facial ROI

We subsequently refine the feature mask by removing noisy features. To
explain, Fig. 5(b) illustrates some counter-examples that the feature mask can
sometimes be ineffective as it incorrectly highlights the hair, the headgear,
and the surrounding area, rather than the face itself. Although these features
are inherent and could be harmless in the standard face recognition process,
in our case, they bring in undesirable noise during attention transfer.

We denoise the mask by obtaining a region of interest (ROI) on the raw image.
Facial landmark detection is a proven technique that detects and tracks key
points in a human face and is widely adopted in applications such as augmented
reality. A general facial landmark detector produces a sequence of points that
mark the positions of main facial features between facial contours and
eyebrows, which specifies the interesting regions of our feature mask.
Therefore, we utilize the point sequence to derive the facial ROI.

Specifically, right before performing BDCT, we pass the image $X$ through a
pre-trained facial landmark detector to obtain the sequence of landmark points
$P$. Here, we employ an open-source PFLD model (Guo et al., 2019). Note that
PFLD can be replaced by an arbitrary lightweight 2D landmark detector.
Subsequently, we derive the facial ROI, which wraps the facial region, by
calculating the convex hull $H(P)$ of $P$ using Delaunay triangulation.
Features outside the ROI are considered useless for attention transfer. To
remove them, we turn $H(P)$ into a bool mask by marking all the pixels inside
as 1 and the rest as 0, and overlay it on its feature masks:

$R_{i}^{{}^{\prime}}(X_{c})=H(P)\odot R_{i}(X_{c}).$

Ergo, a clean mask is produced. Finally, we normalize
$R_{i}^{{}^{\prime}}(X_{c})$ and use it as a replacement of $R_{i}(X_{c})$.

Figure 6. Effect of the facial ROI. After overlaying the ROI, the updated
server-side feature shifts its highlight from the noisy surroundings (top) to
the correct facial region (bottom).

Denoising the face image via facial ROI.

Fig. 6 show sample masks before and after applying the facial ROI. It can be
clearly observed that the ROI removes the features of the surroundings,
resulting in focused attention to the face. Results in Sec. 4.6 also show a
performance lift after employing the ROI.

As a brief summary, the proposed DuetFace successfully overcomes the privacy-
accuracy trade-off problem. By performing channel splitting, privacy is
guaranteed by removing visual information from the server’s side. Then, the
accuracy is compensated by attention transfer from the client-side to the
server-side via the ROI-refined feature mask. Sec. 4.3 shows the effectiveness
of our method, which almost fills in the accuracy gap with the standard
ArcFace, and outperforms those SOTA privacy-preserving counterparts.

##  4\. Experiments

###  4.1. Datasets

Training datasets. We employ MS1Mv2 as training set, plus BUPT-BalancedFace
(Wang and Deng, 2019) (BUPT in short) for ablation study. MS1Mv2 is refined
from MS-Celeb-1M (Guo et al., 2016) and contains approximately 5.8M images of
85K individuals, whereas BUPT contains 1.8M images from 28K identities. For
the training of facial landmark detector, we use Wider Facial Landmarks in-
the-wild (WFLW) (Wu et al., 2018) that contains 10K faces, each with 98 manual
annotated landmarks.

Evaluation datasets. We benchmark our method on five popular datasets,
including LFW (Huang et al., 2007), CFP-FP (Sengupta et al., 2016), AgeDB
(Moschoglou et al., 2017), CPLFW (Zheng and Deng, 2018), and CALFW (Zheng et
al., 2017). LFW is the most commonly used evaluation dataset that contains 13K
web-collected images from 5.7K identities. CFP-FP, AgeDB, CPLFW and CALFW
embody a similar size while focusing on the performance on variations in such
as pose and age. We also extend our evaluation to two general, large-scale
benchmarks: IJB-B (Whitelam et al., 2017) and IJB-C (Maze et al., 2018).

###  4.2. Implementation Details

Backbone. We use the adapted ResNet50 with an improved residual unit (IR-50)
(He et al., 2016) as the server-side backbone $M_{s}$, which has better
convergence in the early training stages, and the MobileFaceNet (Chen et al.,
2018) as the client-side $M_{c}$. For the client-side facial landmark
detection, we apply a PFLD (Guo et al., 2019) network. Note that both
MobileFaceNet and PFLD are lightweight backbones dedicated to resource-
constraint devices such as cell phones. We also adapt a smaller IR-18 backbone
for ablation study, to explain the generality of our method on different
network architectures.

Preparation. We crop and resize each image to 112×112 pixels and add random
flipping as image augmentation, then apply BDCT to obtain 192 frequency
components by an open-source TorchJPEG library (Ehrlich et al., 2020). After
transforming into the frequency domain, we calculate the energy, and select 10
channels from each of the Y, Cb, Cr components as stated in Sec. 3.2. This
results in 30 channels selected in total. Note that we choose 30 without loss
of generality and it is not the only choice. Therefrom, we form an $X_{s}$ in
the shape of 112×112×162 and an $X_{c}$ of 112×112×30.

We alter the model input channels to meet the shapes of $X_{c}$ and $X_{s}$.
Each of the IR-18/50 and MobileFaceNet models contains 4 stages. For attention
transfer, we insert an interactive block $IB$ at the end of each stage.
Aligning with the feature map shapes in $X_{s}$, in our case, the 4 feature
masks are resized to the height and width of 56, 28, 14 and 7, respectively.
The other parts of the models remain unchanged.

Training. The IR-18/50 and MobileFaceNet models are trained for 24 epochs on
the same dataset (either MS1Mv2 or BUPT), using the ArcFace (Deng et al.,
2019) loss. We use the stochastic gradient descent (SGD) optimizer, which is
applied with an initial learning rate of 0.1, a momentum of 0.9, and a weight
decay of 5e-4, at a batch size of 512. We successively divide the learning
rate by 10 at stages 10, 18, and 22. As of PFLD, we apply a learning rate and
a weight decay of 1e-4 and 1e-6, respectively, and train it until convergence.
Experiments are conducted on 8 NVIDIA Tesla V100 GPU under the PyTorch
framework. The same random seed is sampled for all experiments for fairness.

Table 1. Comparisons with State-of-the-Art Methods

Method | PPFR | LFW | CFP-FP | AgeDB | CPLFW | CALFW | IJB-B(TPR@FPR) | IJB-C(TPR@FPR)
---|---|---|---|---|---|---|---|---
ArcFace (Deng et al., 2019) | No | 99.77 | 98.30 | 97.88 | 92.77 | 96.05 | 94.13 | 95.60
ArcFace-FD (dos Santos and Almeida, 2021) | No | 99.78 | 98.04 | 98.10 | 92.48 | 96.03 | 94.08 | 95.64
PEEP (Chamikara et al., 2020) | Yes | 98.41 | 74.47 | 87.47 | 79.58 | 90.06 | 5.82 | 6.02
Cloak (Mireshghallah et al., 2021) | Yes | 98.91 | 87.97 | 92.60 | 83.43 | 92.18 | 33.58 | 33.82
InstaHide (Huang et al., 2020) | Yes | 96.53 | 83.20 | 79.58 | 81.03 | 86.24 | 61.88 | 69.02
CPGAN (Tseng and Wu, 2020) | Yes | 98.87 | 94.61 | 96.98 | 90.43 | 94.79 | 92.67 | 94.31
PPFR-FD (Wang et al., 2022) | Yes | 99.68 | 95.04 | 97.37 | 90.78 | 95.72 | * | 94.10
DuetFace (ours) | Yes | 99.82 | 97.79 | 97.93 | 92.35 | 96.10 | 93.66 | 95.30

  * •

* The results of PPFR-FD are quoted from [34] due to the lack of source code. Please note that its experimental condition may be different slightly from ours.

###  4.3. Comparisons with SOTA Methods

Methods for comparison. We compare DuetFace with two face recognition methods
without privacy protection and five state-of-the-art PPFR methods.
Specifically: (1) ArcFace (Deng et al., 2019) is the baseline method for RGB
images without privacy protection; (2) ArcFace-FD is the same ArcFace backbone
trained on frequency components instead, which we employ the method in (dos
Santos and Almeida, 2021); (3) PEEP (Chamikara et al., 2020) is the first
practical method to introduce differential privacy in PPFR, where we set its
privacy budget $\epsilon$ to 5; (4) Cloak (Mireshghallah et al., 2021)
compresses the input feature space by a gradient-based perturbation. We set
its accuracy-privacy parameter to 100; (5) InstaHide (Huang et al., 2020) is a
lightweight encryption-based method in distributed setting that incorporates
the mix-up of $k$ images, which we set to 2; (6) CPGAN (Tseng and Wu, 2020)
generates compressive face representations by GAN and local differential
privacy; And (7) PPFR-FD (Wang et al., 2022) is a very recent method that
masks the face by shuffling and mixing on frequency channels. Results are
summarized in Tab. 1.

Results on LFW, CFP-FP, AgeDB, CPLFW, and CALFW are reported by accuracy. Our
method achieves a very close performance to the non-privacy-preserving ArcFace
baseline. Concretely, our method achieves almost the same performance as
ArcFace on LFW, AgeDB and CALFW, and has a limited accuracy drop of 0.51% on
CFP-FP, and 0.42% on CPLFW. We believe such accuracy loss is because these two
datasets possess more complex variations in pose, which downgrades the
inference of facial landmarks. On the other hand, our method takes a leading
role over all the SOTA privacy-preserving methods by an advantage from 0.14%
to 23.32%.

Results on IJB-B and IJB-C are reported in TPR@FPR(1e-4), i.e., the true-
positive rate at the false-positive rate of 1e-4. The TPR of DuetFace is
slightly lower than ArcFace, but still outperforms most of its competitors,
which indicates our method has good generalizability on large-scale datasets.
We also notice that some SOTA methods may fail to generalize well as their
performance degrades very sharply.

###  4.4. Visual Privacy

We validate that our proposed method provides reliable privacy protection that
satisfies the security goals stated in Sec. 3.1. The basic concern among our
goals is visual security, i.e., the server should not be able to visually
inspect the query image by the information it acquires from the client.

Figure 7. Visual privacy. (a) Illustration of $X_{s}$, $X_{c}$ and
${R_{i}^{{}^{\prime}}(X_{c})}$ on sample images. Perceptible visual
information is removed from server-side. (b) We alter the number of crucial
channels. As it increases, visual features become indistinguishable.

Visual privacy.

Visualization of the client/server-side components and feature masks. During
the whole inference process, the information the server obtains including the
non-crucial components $X_{s}$ of the query image and the refined feature
masks ${R_{i}^{{}^{\prime}}(X_{c})}$ at each stage. To illustrate the
effectiveness of our method, we randomly sample face images and visualize
their $X_{s}$, $X_{c}$, and ${R_{i}^{{}^{\prime}}(X_{c})}$. As $X_{s}$ is in
the frequency domain, we pad its removed channels with zero and convert it
back to RGB by performing inverse DCT. We perform the same on $X_{c}$ for
comparison. The results in Fig. 7(a) show that most of the perceptible visual
information is removed from $X_{s}$ but retained in $X_{c}$, which prevents
the server from inspecting $X_{s}$ while allowing the client to infer $X_{c}$
(in private) normally. As for the feature masks, the results show that they
reveal only very limited visual information such as the approximate facial
contours, which would not sabotage our security goal.

Visualization on different numbers of crucial channels. We alter the number of
selected channels and visualize their corresponding $X_{s}$ in Fig. 7(b).
Prior work (Wang et al., 2022) also discards one channel with the highest
energy (the direct current component) in its means to protect privacy. Our
visualization implies such removal could be insufficient since part of the
visual features can still be inferred from the rest components. As the number
of selected channels increases, the remaining visual features become
indistinguishable.

###  4.5. Privacy Against Malicious Intents

Figure 8. Reconstruction outcomes on $X_{s}$, $R_{0}^{{}^{\prime}}(X_{c})$ and
$F_{0}^{{}^{\prime}}(X_{s})$ of sample image via U-Net. The server obtains
hardly any visual feature. Reconstruction on the raw face (leftmost) is also
provided as a validation of the ability of U-Net.

Reconstruction.

Ever since its invention, face recognition is under the threat of malicious
attacks. Our security goal in Sec. 3.1 addresses concerns to two major forms
of inference threats. Specifically, (1) though visually inaccessible, the
semi-honest server may still convincingly reconstruct the query image based on
what it possesses; And (2) any third party, if intercepts the information
transferred from the client or receives unauthorized redistribution from the
server, may infer the identity of the face even without accessing the
recognition model. To further demonstrate our method’s ability to protect
valuable information about the query face, we impose these two types of
threats on DuetFace.

Table 2. Effectiveness Against Reconstruction and Malicious Identity Inference Target | SSIM↓ | PSNR↓ | Accuracy↓
---|---|---|---
Raw image | 0.9993 | 48.30 | 99.80
$X_{s}$ | 0.5395 | 13.01 | 51.52
Reconstruction of $X_{s}$ | 0.5012 | 12.89 | 52.03
Reconstruction of $R_{0}^{{}^{\prime}}(X_{c})$ | 0.4422 | 12.45 | 57.18
Reconstruction of $F_{0}^{{}^{\prime}}(X_{s})$ | 0.4555 | 12.63 | 61.97

Effectiveness against reconstruction. Autoencoders are widely used for
reconstruction. In our case, the server may attempt to reconstruct the face
image by an autoencoder-based network. Accessible information for the server
including the non-crucial channels, the feature masks and their combination,
which equals to the masked server-side feature map
$F_{i}^{{}^{\prime}}(X_{s})$. Ergo, we first infer images randomly picked from
MS1Mv2 on a pre-trained DuetFace to collect their $X_{s}$,
$R_{0}^{{}^{\prime}}(X_{c})$ and $F_{0}^{{}^{\prime}}(X_{s})$. Note that in
our case, as the resolution of feature maps is successively divided by half in
each stage, the reconstruction is most likely to succeed on stage 0. We train
three autoencoder-based U-Net models, denoted as $U_{1}$, $U_{2}$ and $U_{3}$,
on $X_{s}$, $R_{0}^{{}^{\prime}}(X_{c})$ and $F_{0}^{{}^{\prime}}(X_{s})$,
respectively, then use the trained model for reconstruction. We quantify the
quality of the reconstructed images by structural similarity index (SSIM, as
compared to the raw image) and peak signal-to-noise ratio (PSNR). As shown in
Fig. 8, none of $U_{1}$, $U_{2}$ and $U_{3}$ manages to effectively
reconstruct an image that embodies distinguishable features. And in Tab. 2, we
can see that the SSIM and PSNR values are low for all three cases. These
verify the robustness of our method against reconstruction.

Effectiveness against identity inference. We subsequently feed $X_{s}$ (after
transformed into the spatial domain) and the reconstructed images into a pre-
trained standard ArcFace model and see if they can be recognized. Results are
summarized in the last column of Tab. 2. The intention to infer from $X_{s}$
or its reconstruction $U_{1}({X_{s}})$ fails, as the accuracy is slightly
higher than random guess (50%). A little accuracy gain is observed on the
recognition of $U_{2}({R_{0}^{{}^{\prime}}(X_{c})})$ and
$U_{3}({F_{0}^{{}^{\prime}}(X_{s})})$, as the mask could provide a trace
amount of information. Yet, the accuracy remains very low, which thus does not
incur an effective threat to our method.

###  4.6. Ablation Study

Table 3. Ablation Study Results

Method | LFW | CFP-FP | AgeDB | CPLFW | CALFW
---|---|---|---|---|---
IR-50 + MobileFaceNet, on MS1Mv2
ArcFace (Deng et al., 2019) | 99.77 | 98.30 | 97.88 | 92.77 | 96.05
DuetFace | 99.82 | 97.79 | 97.93 | 92.35 | 96.10
w/o $IB$ | 99.70 | 95.86 | 97.57 | 90.82 | 95.86
w/o ROI | 99.78 | 97.23 | 97.85 | 92.07 | 96.06
w/o $M_{s}$ | 99.48 | 93.91 | 96.10 | 89.68 | 95.08
IR-18 + MobileFaceNet, on BUPT
ArcFace (Deng et al., 2019) | 99.52 | 94.06 | 94.95 | 90.05 | 95.07
DuetFace | 99.40 | 93.79 | 95.07 | 89.67 | 95.03
w/o $IB$ | 99.16 | 91.96 | 94.18 | 87.67 | 94.27
w/o ROI | 99.39 | 93.46 | 95.12 | 89.03 | 94.86
w/o $M_{s}$ | 99.32 | 92.43 | 93.42 | 89.15 | 93.62

  * *

Two sets of experiments are performed in combination of IR-50+MS1Mv2 and
IR-18+BUPT to demonstrate the generality of our method.

  * **

ArcFace is the unprotected baseline. “w/o $IB$”, “w/o ROI”, “w/o $M_{s}$”
correspond to the removal of the interactive block, ROI and server-side model,
respectively.

We analyze the effects of the major components of DuetFace by testing the
performance when removing one of them. Results are reported in Tab. 3 on LFW,
CFP-FP, AgeDB, CPLFW, and CALFW. To validate the generality of our method, we
also report the results on the combination of the IR-18 model and BUPT
dataset.

Effect of the interactive block. The server leverages client-side features to
compensate for its inaccurate attention to facial features. To demonstrate
this, we remove the interactive block as well as the client-side model
completely and train the server-side $M_{s}$ alone on $X_{s}$. Note that this
setting is equal to “w/o $M_{c}$”. From Tab. 3, we observe a ¡2.5% accuracy
degradation.

Effect of facial ROI. The facial ROI refines the feature mask by focusing its
attention on the facial region. We remove the facial landmark detector and let
the client send the unrefined feature mask directly to the server. We see a
notable performance downgrade, especially on CFP-FP and CPLFW. This is
possibly because the variety of poses on these two datasets exacerbate
attention shifting from the facial region to its surrounding area, which is
undesirable.

Performance of the client-side model alone. The client-side model is
implemented as an aid to the server-side. A large performance gap is observed
between $M_{c}$ alone and DuetFace, as $M_{c}$ is much smaller in scale and
complexity.

###  4.7. Complexity and Cost

Table 4. Complexity and Cost of DuetFace Metric | ArcFace | DuetFace | $M_{s}$ | $M_{c}$
---|---|---|---|---
Storage (#param) | 43.59M | 46.53M | 44.03M | 2.50M
Time (s/batch) | 0.8492 | 2.2153 | 1.2165 | 0.9988
Comm. (#elements) | 37,632 | 35,525 | N/A | N/A

To demonstrate the resource-friendliness of our method, we summarize the model
size, time cost, and communication overhead required for inference in Tab. 4.
As the availability of our method mainly depends on the client-side budget,
for a clear demonstration, we list the space and time costs of the client and
server separately. The PFLD model is included when evaluating the client’s
cost.

Model size. We list the model size by the number of parameters. Since both
MobileFaceNet and PFLD are an order of magnitude smaller than the server-side
model, the employment of the local model only accounts for a 6.7% increase of
the whole model size and takes about 10M of storage space at the client-side.

Inference time. We perform inference on images with a batch size of 64 and
record the average per-batch time. Here, the inference is performed in an
asynchronous manner, i.e., the client completes the local computations on all
query images, then hands it over to the server for the rest. As compared to
ArcFace, our total inference time increased by ×2.6. We think such an increase
is marginal as it is the communication time that predominates in real-world
practice. The overall time cost is still within a decent scope.

Communication overhead. We calculate by adding up the number of elements in
the server-side components $X_{s}$ and the feature masks of all 4 stages
${R_{i}^{{}^{\prime}}(X_{c})}$. The baseline ArcFace transfers the RGB images
of 3×112×112 directly, resulting in 37,632 elements. In our original purpose,
each image is up-sampled before BDCT (as means to preserve the same input
height and width), which causes a larger communication cost. We can overcome
this by moving the up-sampling later to the server-side. Therefore, the
server-side components $X_{s}$ is of shape 14×14×160, containing 31,360
elements. On the other hand, the 4 feature masks are the shape of 56×56,
28×28, 14×14, and 7×7, respectively, which accounts for 4,165 elements in
total. Thus, the total element number is 35,525, making our communication
overhead similar to that of the baseline ArcFace.

##  5\. Conclusion

This paper presents DuetFace, a novel PPFR method that achieves high
recognition performance through the collaboration between the client and the
server. Channel splitting and attention transfer in the frequency domain are
leveraged to implement the proposed method. Concretely, our method trains and
infers the server-side model on visually indistinguishable non-crucial
channels, and compensates for the inaccurate attention by the client-side
information, in particular, by producing and transferring a feature mask
through a plug-in interactive block. We refine the feature mask by overlaying
a facial ROI. Extensive experiments show the proposed method is satisfactory
in recognition accuracy, with good cost-efficiency, and achieves high
reliability in privacy protection.

## References

  * (1)
  * Abdalla et al. (2015) Michel Abdalla, Florian Bourse, Angelo De Caro, and David Pointcheval. 2015.  Simple Functional Encryption Schemes for Inner Products. In _Public-Key Cryptography - PKC 2015 \- 18th IACR International Conference on Practice and Theory in Public-Key Cryptography, Gaithersburg, MD, USA, March 30 - April 1, 2015, Proceedings_ _(Lecture Notes in Computer Science, Vol. 9020)_ , Jonathan Katz (Ed.). Springer, 733–751.  https://doi.org/10.1007/978-3-662-46447-2_33
  * Chamikara et al. (2020) Mahawaga Arachchige Pathum Chamikara, Peter Bertók, Ibrahim Khalil, Dongxi Liu, and Seyit Camtepe. 2020\.  Privacy Preserving Face Recognition Utilizing Differential Privacy.  _Comput. Secur._ 97 (2020), 101951.  https://doi.org/10.1016/j.cose.2020.101951
  * Chanyaswad et al. (2016) Thee Chanyaswad, J. Morris Chang, Prateek Mittal, and Sun-Yuan Kung. 2016.  Discriminant-component eigenfaces for privacy-preserving face recognition. In _26th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2016, Vietri sul Mare, Salerno, Italy, September 13-16, 2016_ , Francesco A. N. Palmieri, Aurelio Uncini, Kostas I. Diamantaras, and Jan Larsen (Eds.). IEEE, 1–6.  https://doi.org/10.1109/MLSP.2016.7738871
  * Chen et al. (2018) Sheng Chen, Yang Liu, Xiang Gao, and Zhen Han. 2018\.  MobileFaceNets: Efficient CNNs for Accurate Real-Time Face Verification on Mobile Devices. In _Biometric Recognition - 13th Chinese Conference, CCBR 2018, Urumqi, China, August 11-12, 2018, Proceedings_ _(Lecture Notes in Computer Science, Vol. 10996)_ , Jie Zhou, Yunhong Wang, Zhenan Sun, Zhenhong Jia, Jianjiang Feng, Shiguang Shan, Kurban Ubul, and Zhenhua Guo (Eds.). Springer, 428–438.  https://doi.org/10.1007/978-3-319-97909-0_46
  * Deng et al. (2019) Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. 2019\.  ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In _IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019_. Computer Vision Foundation / IEEE, 4690–4699.  https://doi.org/10.1109/CVPR.2019.00482
  * dos Santos and Almeida (2021) Samuel Felipe dos Santos and Jurandy Almeida. 2021.  Less Is More: Accelerating Faster Neural Networks Straight from JPEG. In _Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications - 25th Iberoamerican Congress, CIARP 2021, Porto, Portugal, May 10-13, 2021, Revised Selected Papers_ _(Lecture Notes in Computer Science, Vol. 12702)_ , João Manuel R. S. Tavares, João Paulo Papa, and Manuel González Hidalgo (Eds.). Springer, 237–247.  https://doi.org/10.1007/978-3-030-93420-0_23
  * Ehrlich et al. (2020) Max Ehrlich, Larry Davis, Ser-Nam Lim, and Abhinav Shrivastava. 2020.  Quantization Guided JPEG Artifact Correction. In _Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VIII_ _(Lecture Notes in Computer Science, Vol. 12353)_ , Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (Eds.). Springer, 293–309.  https://doi.org/10.1007/978-3-030-58598-3_18
  * Ergun (2014) Ovgu Ozturk Ergun. 2014\.  Privacy preserving face recognition in encrypted domain. In _2014 IEEE Asia Pacific Conference on Circuits and Systems, APCCAS 2014, Ishigaki, Japan, November 17-20, 2014_. IEEE, 643–646.  https://doi.org/10.1109/APCCAS.2014.7032863
  * Erkin et al. (2009) Zekeriya Erkin, Martin Franz, Jorge Guajardo, Stefan Katzenbeisser, Inald Lagendijk, and Tomas Toft. 2009.  Privacy-Preserving Face Recognition. In _Privacy Enhancing Technologies, 9th International Symposium, PETS 2009, Seattle, WA, USA, August 5-7, 2009. Proceedings_ _(Lecture Notes in Computer Science, Vol. 5672)_ , Ian Goldberg and Mikhail J. Atallah (Eds.). Springer, 235–253.  https://doi.org/10.1007/978-3-642-03168-7_14
  * Guo et al. (2019) Xiaojie Guo, Siyuan Li, Jiawan Zhang, Jiayi Ma, Lin Ma, Wei Liu, and Haibin Ling. 2019.  PFLD: A Practical Facial Landmark Detector. (2019).  arXiv:1902.10859 http://arxiv.org/abs/1902.10859
  * Guo et al. (2016) Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. 2016.  MS-Celeb-1M: A Dataset and Benchmark for Large-Scale Face Recognition. In _Computer Vision - ECCV 2016 - 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III_ _(Lecture Notes in Computer Science, Vol. 9907)_ , Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer, 87–102.  https://doi.org/10.1007/978-3-319-46487-9_6
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep Residual Learning for Image Recognition. In _2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016_. IEEE Computer Society, 770–778.  https://doi.org/10.1109/CVPR.2016.90
  * Honda et al. (2015) Katsuhiro Honda, Masahiro Omori, Seiki Ubukata, and Akira Notsu. 2015.  A study on fuzzy clustering-based k-anonymization for privacy preserving crowd movement analysis with face recognition. In _7th International Conference of Soft Computing and Pattern Recognition, SoCPaR 2015, Fukuoka, Japan, November 13-15, 2015_ , Mario Köppen, Bing Xue, Hideyuki Takagi, Ajith Abraham, Azah Kamilah Muda, and Kun Ma (Eds.). IEEE, 37–41.  https://doi.org/10.1109/SOCPAR.2015.7492779
  * Huang et al. (2007) Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. 2007.  _Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments_.  Technical Report 07-49. University of Massachusetts, Amherst.
  * Huang et al. (2020) Yangsibo Huang, Zhao Song, Kai Li, and Sanjeev Arora. 2020\.  InstaHide: Instance-hiding Schemes for Private Distributed Learning. In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ _(Proceedings of Machine Learning Research, Vol. 119)_. PMLR, 4507–4518.  http://proceedings.mlr.press/v119/huang20i.html
  * Kevenaar et al. (2005) Tom A. M. Kevenaar, Geert Jan Schrijen, Michiel van der Veen, Anton H. M. Akkermans, and Fei Zuo. 2005\.  Face Recognition with Renewable and Privacy Preserving Binary Templates. In _Proceedings of the Fourth IEEE Workshop on Automatic Identification Advanced Technologies (AutoID 2005), 16-18 October 2005, Buffalo, NY, USA_. IEEE Computer Society, 21–26.  https://doi.org/10.1109/AUTOID.2005.24
  * Kou et al. (2021) Xiaoyu Kou, Ziling Zhang, Yuelei Zhang, and Linlin Li. 2021\.  Efficient and Privacy-preserving Distributed Face Recognition Scheme via FaceNet. In _ACM TURC 2021: ACM Turing Award Celebration Conference - Hefei, China, 30 July 2021 \- 1 August 2021_. ACM, 110–115.  https://doi.org/10.1145/3472634.3472661
  * Li et al. (2019) Yuancheng Li, Yimeng Wang, and Daoxing Li. 2019.  Privacy-preserving lightweight face recognition.  _Neurocomputing_ 363 (2019), 212–222.  https://doi.org/10.1016/j.neucom.2019.07.039
  * Ma et al. (2019) Zhuo Ma, Yang Liu, Ximeng Liu, Jianfeng Ma, and Kui Ren. 2019.  Lightweight Privacy-Preserving Ensemble Classification for Face Recognition.  _IEEE Internet Things J._ 6, 3 (2019), 5778–5790.  https://doi.org/10.1109/JIOT.2019.2905555
  * Mao et al. (2018) Yunlong Mao, Shanhe Yi, Qun Li, Jinghao Feng, Fengyuan Xu, , and Sheng Zhong. 2018.  A Privacy-Preserving Deep Learning Approach for Face Recognition with Edge Computing. (July 2018).  https://www.usenix.org/conference/hotedge18/presentation/mao
  * Maze et al. (2018) Brianna Maze, Jocelyn C. Adams, James A. Duncan, Nathan D. Kalka, Tim Miller, Charles Otto, Anil K. Jain, W. Tyler Niggel, Janet Anderson, Jordan Cheney, and Patrick Grother. 2018.  IARPA Janus Benchmark - C: Face Dataset and Protocol. In _2018 International Conference on Biometrics, ICB 2018, Gold Coast, Australia, February 20-23, 2018_. IEEE, 158–165.  https://doi.org/10.1109/ICB2018.2018.00033
  * Mireshghallah et al. (2021) Fatemehsadat Mireshghallah, Mohammadkazem Taram, Ali Jalali, Ahmed Taha Elthakeb, Dean M. Tullsen, and Hadi Esmaeilzadeh. 2021.  Not All Features Are Equal: Discovering Essential Features for Preserving Prediction Privacy. In _WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021_ , Jure Leskovec, Marko Grobelnik, Marc Najork, Jie Tang, and Leila Zia (Eds.). ACM / IW3C2, 669–680.  https://doi.org/10.1145/3442381.3449965
  * Mirjalili et al. (2018b) Vahid Mirjalili, Sebastian Raschka, Anoop M. Namboodiri, and Arun Ross. 2018b.  Semi-adversarial Networks: Convolutional Autoencoders for Imparting Privacy to Face Images. In _2018 International Conference on Biometrics, ICB 2018, Gold Coast, Australia, February 20-23, 2018_. IEEE, 82–89.  https://doi.org/10.1109/ICB2018.2018.00023
  * Mirjalili et al. (2018a) Vahid Mirjalili, Sebastian Raschka, and Arun Ross. 2018a.  Gender Privacy: An Ensemble of Semi Adversarial Networks for Confounding Arbitrary Gender Classifiers. In _9th IEEE International Conference on Biometrics Theory, Applications and Systems, BTAS 2018, Redondo Beach, CA, USA, October 22-25, 2018_. IEEE, 1–10.  https://doi.org/10.1109/BTAS.2018.8698605
  * Mirjalili and Ross (2017) Vahid Mirjalili and Arun Ross. 2017.  Soft biometric privacy: Retaining biometric utility of face images while perturbing gender. In _2017 IEEE International Joint Conference on Biometrics, IJCB 2017, Denver, CO, USA, October 1-4, 2017_. IEEE, 564–573.  https://doi.org/10.1109/BTAS.2017.8272743
  * Moschoglou et al. (2017) Stylianos Moschoglou, Athanasios Papaioannou, Christos Sagonas, Jiankang Deng, Irene Kotsia, and Stefanos Zafeiriou. 2017.  AgeDB: The First Manually Collected, In-the-Wild Age Database. In _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society, 1997–2005.  https://doi.org/10.1109/CVPRW.2017.250
  * Sadeghi et al. (2009) Ahmad-Reza Sadeghi, Thomas Schneider, and Immo Wehrenberg. 2009\.  Efficient Privacy-Preserving Face Recognition. In _Information, Security and Cryptology - ICISC 2009, 12th International Conference, Seoul, Korea, December 2-4, 2009, Revised Selected Papers_ _(Lecture Notes in Computer Science, Vol. 5984)_ , Dong Hoon Lee and Seokhie Hong (Eds.). Springer, 229–244.  https://doi.org/10.1007/978-3-642-14423-3_16
  * Selvaraju et al. (2020) Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. 2020.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization.  _Int. J. Comput. Vis._ 128, 2 (2020), 336–359.  https://doi.org/10.1007/s11263-019-01228-7
  * Sengupta et al. (2016) Soumyadip Sengupta, Jun-Cheng Chen, Carlos Domingo Castillo, Vishal M. Patel, Rama Chellappa, and David W. Jacobs. 2016\.  Frontal to profile face verification in the wild. In _2016 IEEE Winter Conference on Applications of Computer Vision, WACV 2016, Lake Placid, NY, USA, March 7-10, 2016_. IEEE Computer Society, 1–9.  https://doi.org/10.1109/WACV.2016.7477558
  * Torfason et al. (2018) Robert Torfason, Fabian Mentzer, Eirikur Agustsson, Michael Tschannen, Radu Timofte, and Luc Van Gool. 2018.  Towards Image Understanding from Deep Compression Without Decoding. (2018).  https://openreview.net/forum?id=HkXWCMbRW
  * Tseng and Wu (2020) Bo-Wei Tseng and Pei-Yuan Wu. 2020.  Compressive Privacy Generative Adversarial Network.  _IEEE Trans. Inf. Forensics Secur._ 15 (2020), 2499–2513.  https://doi.org/10.1109/TIFS.2020.2968188
  * Wallace (1991) Gregory K. Wallace. 1991\.  The JPEG Still Picture Compression Standard.  _Commun. ACM_ 34, 4 (1991), 30–44.  https://doi.org/10.1145/103085.103089
  * Wang et al. (2020) Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P. Xing. 2020\.  High-Frequency Component Helps Explain the Generalization of Convolutional Neural Networks. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_. Computer Vision Foundation / IEEE, 8681–8691.  https://doi.org/10.1109/CVPR42600.2020.00871
  * Wang and Deng (2019) Mei Wang and Weihong Deng. 2019.  Mitigate Bias in Face Recognition using Skewness-Aware Reinforcement Learning. (2019).  arXiv:1911.10692 http://arxiv.org/abs/1911.10692
  * Wang et al. (2022) Yinggui Wang, Jian Liu, Man Luo, Le Yang, and Li Wang. 2022.  Privacy-Preserving Face Recognition in the Frequency Domain. (2022).  (in press).
  * Whitelam et al. (2017) Cameron Whitelam, Emma Taborsky, Austin Blanton, Brianna Maze, Jocelyn C. Adams, Tim Miller, Nathan D. Kalka, Anil K. Jain, James A. Duncan, Kristen Allen, Jordan Cheney, and Patrick Grother. 2017\.  IARPA Janus Benchmark-B Face Dataset. In _2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2017, Honolulu, HI, USA, July 21-26, 2017_. IEEE Computer Society, 592–600.  https://doi.org/10.1109/CVPRW.2017.87
  * Wu et al. (2018) Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou. 2018\.  Look at Boundary: A Boundary-Aware Face Alignment Algorithm. In _2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018_. Computer Vision Foundation / IEEE Computer Society, 2129–2138.  https://doi.org/10.1109/CVPR.2018.00227
  * Xiang et al. (2016) Can Xiang, Chunming Tang, Yunlu Cai, and Qiuxia Xu. 2016\.  Privacy-preserving face recognition with outsourced computation.  _Soft Comput._ 20, 9 (2016), 3735–3744.  https://doi.org/10.1007/s00500-015-1759-5
  * Xu et al. (2020) Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren. 2020\.  Learning in the Frequency Domain. In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020_. Computer Vision Foundation / IEEE, 1737–1746.  https://doi.org/10.1109/CVPR42600.2020.00181
  * Xu et al. (2018) Kai Xu, Zhikang Zhang, and Fengbo Ren. 2018.  LAPRAN: A Scalable Laplacian Pyramid Reconstructive Adversarial Network for Flexible Compressive Sensing Reconstruction. In _Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part X_ _(Lecture Notes in Computer Science, Vol. 11214)_ , Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss (Eds.). Springer, 491–507.  https://doi.org/10.1007/978-3-030-01249-6_30
  * Yang et al. (2018) Xiaopeng Yang, Hui Zhu, Rongxing Lu, Ximeng Liu, and Hui Li. 2018.  Efficient and Privacy-Preserving Online Face Recognition Over Encrypted Outsourced Data. In _IEEE International Conference on Internet of Things (iThings) and IEEE Green Computing and Communications (GreenCom) and IEEE Cyber, Physical and Social Computing (CPSCom) and IEEE Smart Data (SmartData), iThings/GreenCom/CPSCom/SmartData 2018, Halifax, NS, Canada, July 30 - August 3, 2018_. IEEE, 366–373.  https://doi.org/10.1109/Cybermatics_2018.2018.00089
  * Zhang et al. (2019) Chen Zhang, Xiongwei Hu, Yu Xie, Maoguo Gong, and Bin Yu. 2019.  A Privacy-Preserving Multi-Task Learning Framework for Face Detection, Landmark Localization, Pose Estimation, and Gender Recognition.  _Frontiers Neurorobotics_ 13 (2019), 112.  https://doi.org/10.3389/fnbot.2019.00112
  * Zheng and Deng (2018) T. Zheng and W. Deng. 2018\.  _Cross-pose LFW: A database for studying cross-pose face recognition in unconstrained environments_.  Technical Report 18-01. Beijing University of Posts and Telecommunications.
  * Zheng et al. (2017) Tianyue Zheng, Weihong Deng, and Jiani Hu. 2017.  Cross-Age LFW: A Database for Studying Cross-Age Face Recognition in Unconstrained Environments. (2017).  arXiv:1708.08197 http://arxiv.org/abs/1708.08197