#  Tuning Quantum Computing Privacy through Quantum Error Correction

Hui Zhong1, Keyi Ju2, Manojna Sistla1, Xinyue Zhang3, Xiaoqi Qin2, Xin Fu1,
Miao Pan1 1Department of Electrical and Computer Engineering, University of
Houston, Houston, TX, 77204  2State Key Laboratory of Networking and Switching
Technology,
Beijing University of Posts and Telecommunications, Beijing, China, 100876
3Department of Computer Science, Kennesaw State University, Marietta, GA,
30060

###### Abstract

Quantum computing is a promising paradigm for efficiently solving large and
high-complexity problems. To protect quantum computing privacy, pioneering
research efforts proposed to redefine differential privacy (DP) in quantum
computing, i.e., quantum differential privacy (QDP), and harvest inherent
noises generated by quantum computing to implement QDP. However, such an
implementation approach is limited by the amount of inherent noises, which
makes the privacy budget of the QDP mechanism fixed and uncontrollable. To
address this issue, in this paper, we propose to leverage quantum error
correction (QEC) techniques to reduce quantum computing errors, while tuning
the privacy protection levels in QDP. In short, we gradually decrease the
quantum noise error rate by deciding whether to apply QEC operations on the
gate in a multiple single qubit gates circuit. We have derived a new
calculation formula for the general error rate and corresponding privacy
budgets after QEC operation. Then, we expand to achieve further noise
reduction using multi-level concatenated QEC operation. Through extensive
numerical simulations, we demonstrate that QEC is a feasible way to regulate
the degree of privacy protection in quantum computing.

###### Index Terms:

Quantum computing, Quantum noises, Differential privacy, Quantum error
correction

##  I Introduction

Quantum computing is a new paradigm that is promising to significantly
accelerate the processing of large datasets and complex tasks, compared with
traditional computing [1]. Such an advance in computing will affect a
multitude of research domains such as cryptography [2], huge optimization
problems [3], and complex climate simulations [4]. However, similar to
classical computing, quantum computing is vulnerable to data privacy leakage,
because quantum computers face a variety of threats from internal and external
attacks [5], such as coherent attacks, entangling attacks, quantum-side-
channel attacks, inference attacks, etc. Therefore, how to preserve quantum
computing privacy poses great challenges.

In traditional computing, differential privacy (DP) is a powerful privacy
protection method. In simple terms, DP protects data by manually adding noises
to ensure that the presence or absence of a single data has a negligible
effect on any query results. DP seeks a balance between data usability and
privacy and has applications in diverse fields, such as data analytics [6],
medical and health research [7], intelligent transportation [8], etc. DP has
strict mathematical formulas and usually uses a privacy budget $\epsilon$ to
indicate the degree of privacy protection [9]. Recently, DP has been
introduced to the quantum domain, and quantum differential privacy (QDP), a
novel data protection method, brings an overhaul to quantum computing [10, 11,
12]. Existing works have already found that Noisy Intermediate-Scale Quantum
(NISQ) devices generate unavoidable noises during quantum computation and can
act as potential noise sources for QDP [13, 14, 10]. Since the inherent noises
are determined through specific factors such as environment or hardware device
parameters, they are uncontrollable and fixed. This limitation confines these
works to a specific privacy budget, which prevents adjusting it to the target
DP budget. Therefore, it is necessary to explore suitable approaches that can
change the amount of noises to satisfy the different DP requirements in
quantum computing. Generally speaking, when the required amount of noise
exceeds the inherent noise generated by quantum computing, additional noises
should be added to satisfy the privacy budget. Conversely, when the QDP noise
requirement is beneath the inherent noise level, designated methods are in
need to reduce the inherent noises.

As a promising solution, quantum error correction (QEC) techniques represent
quantum bit information as a high dimensional redundancy quantum bits and
target error correction and noise reduction caused by environmental
disturbances, quantum operations, etc. QEC can be implemented through
different error syndromes and specialized quantum circuits, e.g., Steane
codes, Shor codes [15], etc. By effectively reducing the noises through QEC,
we may execute a more reliable quantum computing process [16, 17, 18].

In this paper, we have keenly observed that the QEC method can address the
issue of excessive inherent noises in order to achieve an adjustable DP. Based
on the formula between privacy budgets and depolarizing noise in [13], we
gradually tune the inherent noises and adjust the degree of privacy
protection, by varying whether QEC operation is applied to single qubit gates
or not. We conduct a rigorous mathematical analysis on a general formula for
the total error rate after applying the QEC operation and the corresponding
privacy budget calculation in a single qubit circuit with multiple gates. We
also find that multi-level concatenated QEC can further lower the error rate
at the expense of circuit complexity. We demonstrate our ideas through
extensive numerical simulations.

The rest of the paper is organized as follows. In Section II, we introduce the
theoretical foundations of QDP, QEC, and the basic privacy budget formulas. In
Section III, we present generalized total noise error rate formulas and
privacy budget formulas under multiple single qubit gates, and extend to the
multi-level concatenated QEC operation. In Section IV, we provide numerical
simulation. Finally, we draw conclusions and discuss future work.

##  II Preliminaries

In this section, we review the basics of quantum computing, traditional DP,
and the existing formulation of QDP. Subsequently, we demonstrate how to
calculate the privacy budget with depolarizing noise, as proposed in [13].
Finally, we introduce QEC, a method for adjusting the depolarizing noise error
rate.

###  II-A Quantum computing

The basic unit used in quantum computing is the qubit, represented by the two
states of the standard orthogonal basis $\left|0\right\rangle$ and
$\left|1\right\rangle$, corresponding to the values 0 and 1 in classical
computers. A single-qubit can be represented as
$\left|0\right\rangle=(1,0)^{T}\ $ and $\left|1\right\rangle=(0,1)^{T}$.
Quantum states describe the states of a quantum system. For example, in a two-
dimensional Hilbert space, a pure quantum state can be represented as the wave
function of a two-state system, consisting of a superposition of
$\left|0\right\rangle$ and $\left|1\right\rangle$:
$\left|\psi\right\rangle=\alpha_{1}\left|0\right\rangle+\alpha_{2}\left|1\right\rangle=(\alpha_{1},\alpha_{2})^{T}\in\mathbb{C}^{2}$,
where the complex numbers $\alpha_{1}$ and $\alpha_{2}$ satisfy the formula
$\left|\alpha_{1}\right|^{2}+\left|\alpha_{2}\right|^{2}=1$. In D-dimensional
Hilbert space, it can be generalized to
$\left|\psi\right\rangle=\sum_{i=1}^{D}\alpha_{i}\left|i\right\rangle\in\mathbb{C}^{D}$,
which satisfies
$\left|\alpha_{1}\right|^{2}+...+\left|\alpha_{D}\right|^{2}=1$.
$\left|i\right\rangle$ is the ground state, $\alpha_{i}$ is the complex
amplitude.

Another form of the quantum state is a mixed state $\rho$, defined as
$\rho=\sum_{i=1}^{d}p_{i}\left|\psi\right\rangle\left\langle\psi\right|$,
which can be represented by the density matrix and satisfies
$\text{Tr}\left(\rho\right)=1$. The difference between two quantum states can
be represented by the trace distance $d$, which can be calculated by the trace
of these two density matrices. For quantum states $\rho$ and $\sigma$, their
trace distance is denoted as
$\left|\left|\rho-\sigma\right|\right|^{*}=\text{Tr}\left(\left|\rho-\sigma\right|\right)/2$.

Quantum computing is presented by quantum circuits, which consist of multiple
types of gates. The quantum gate operations are usually represented by the
unitary operator $U$. For example, a single quantum gate can be represented as
$2\times 2$ unitary matrix. If a quantum state $\rho$ goes through the $U$
operation, its output can be represented as $\rho^{\prime}=U\rho U^{\dagger}$,
which satisfies $U^{\dagger}U=UU^{\dagger}=I$. $U^{\dagger}$ is the adjoint of
$U$ and $I$ is the identity operator. Quantum measurements
$\left\\{M_{k}\right\\}_{k\in O}$ get different results for circuits, where
$O$ is a finite set of measurement results. If the result of the circuit
before the quantum measurement is $\rho$, the probability that the measurement
result is $k$ can be expressed as $p_{k}=Tr\left\\{M_{k}\rho\right\\}$.

###  II-B From classical DP to QDP

DP can maintain the statistical information of the data while preventing
privacy leakage. It ensures that the contribution of any single data sample
has a very small impact on the final statistical output. Formal DP and QDP are
defined as follows [13].

######  Definition 1 (Classical Differential Privacy)

A randomized function $K$ satisfies $(\epsilon,\delta)$-differential privacy
if the data sets $D$ and $D^{{}^{\prime}}$ differ by only one participant, and
all set of outcomes $\mathcal{S}\subseteq Range(\mathcal{M})$ satisfy

$\rm{Pr}\left[\mathcal{K}(D)\in\mathcal{S}\right]\leq
e^{\epsilon}\cdot\rm{Pr}\left[\mathcal{K}(D^{\prime})\in\mathcal{S}\right]+\delta,$
(1)

where $\epsilon$ is the privacy budget and $\delta$ is the broken probability.

######  Definition 2 (Quantum Differential Privacy)

Given two quantum datasets $\rho$ and $\sigma$ with $\tau(\rho,\sigma)\leq d$,
where $d\in(0,1]$ and $\tau(\cdot,\cdot)$ indicates the trace distance. A
quantum operation $\mathcal{E}$ is $(\epsilon,\delta)$-differentially private
if every $POVM$ (Positive Operator-Valued Measure) $M=\left\\{M_{m}\right\\}$
and all $\mathcal{S}\subseteq Range(\mathcal{M})$ satisfy,

$Pr\left[\mathcal{E}\left(\rho\right)\in_{M}S\right]\leq e^{\epsilon}\cdot
Pr\left[\mathcal{E}\left(\sigma\right)\in_{M}S\right]+\delta.$ (2)

###  II-C QDP based on depolarizing noise

During quantum computation, NISQ devices suffer from unavoidable errors such
as amplitude noise and bit-flip noise. Although these noises cannot be
quantified in simulations, developers have found that depolarizing noise can
represent the global noises in a circuit. This is because depolarizing noise
is a good approximation of the device error especially when the quantum
circuit is deep enough[19]. Depolarizing noise exists in any quantum
simulator.

In the presence of depolarizing noise, the input quantum state collapses to a
fully mixed state with a certain probability. The depolarizing noise can be
expressed in terms of quantum operations as

$\mathcal{E}_{\mathrm{Dep}}(\rho)=\frac{pI}{D}+(1-p)\rho,$ (3)

where $\rho$ is the initial quantum state, $I$ is the unit operator, $p$ is
the depolarizing noise error rate, and $D$ is the dimension of the Hilbert
space. In the depolarizing channel, the input quantum state is held constant
with probability $1-p$ and evolves to any of the possible quantum states with
probability $p/D$.

We followed Theorem 3 from [13] to calculate privacy budgets $\epsilon$ under
the effect of depolarizing noise:

######  Theorem 1 (DP with depolarizing noise)

For all inputs $\rho$ and $\sigma$ satisfy $\tau\left(\rho,\sigma\right)\leq
d$, the depolarizing operation $\mathcal{E}_{\mathrm{Dep}}(\mathcal{E}(\rho))$
in the D-dimension Hilbert space provides $\epsilon$-differential privacy
where

$\epsilon=\ln{\left[1+\frac{1-p}{p}dD\right]}.$ (4)

Equation (4) shows that the privacy budget $\epsilon$ is only related to the
error rate $p$ of the depolarizing noise because the other parameters are
fixed. Therefore we would like to find some specific ways to reduce $p$ of
depolarizing noise in order to decrease inherent noises and further obtain the
tuning DP.

###  II-D Quantum error correction

Quantum noise is a catastrophic obstacle in the development of quantum
computing. Since qubits are not replicable, classical error correction methods
cannot work well at the quantum level. Quantum error correction (QEC), first
proposed by Peter Shor in 1994 [20], brought a breakthrough in this problem.
QEC formulates an error-correction circuit by storing a quantum bit of
information in an entangled state with a high number of quantum bits, which
can localize and correct quantum bit errors in this high-dimensional Hilbert
subspace [15]. Common QEC methods are Shor code [16], Steane code [17],
Surface code [18], and so on. Different coding methods correspond to different
circuits and thus correct different kinds of errors. In this paper, we choose
Steane code to correct circuit errors.

Steane code corrects any kind of error on a single physical qubit by encoding
seven physical bits into one logical qubit. It consists of four modular
circuits: encoding, detection, correction, and decoding. The function of the
encoding circuit is to encode one logical qubit into seven physical qubits as
shown in Fig. 1. The purpose is that any single physical qubit error can be
detected and corrected by the parity code. The purpose of the detection
circuit is to locate which type of error occurs in which physical qubit. The
circuit is shown in Fig. 2, where the first half is designed to detect bit-
flip errors and the second half is designed to detect phase-flip errors, and
the combination of the two parts can detect both errors. After detecting the
circuit, the auxiliary bits need to be measured to get the six-bit error
syndrome. The error syndrome of the Steane code is shown in Table I. As an
example, if the measurement result is
$\left(N_{0},N_{1},N_{2},M_{0},M_{1},M_{2}\right)=\left(-1,1,1,1,1,1\right)$,
it means that $X$ error occurred in the $0^{th}$ physical qubit. The error
syndrome is very important for the error correction circuit. When a specific
error at a specific location is obtained from the detect circuit, the error is
corrected using the same error type. For example, if an $X$ error occurs in
the $0^{th}$ physical qubit, an $X$ gate is added to the $0^{th}$ physical
qubit in the error correction circuit to correct the error. Finally, the
decoding circuit completes the conversion from seven physical qubits to one
logical qubit [21].

TABLE I: Error syndrome of Steane code. X error (M0,M1,M2=1,1,1)
---
i | 0 | 1 | 2 | 3 | 4 | 5 | 6
Eigenvalue of (N0,N1,N2) | (-1,1,1) | (1,-1,1) | (1,1,-1) | (1,-1,-1) | (-1,1,-1) | (-1,-1,1) | (-1,-1,-1)
Z error (N0,N1,N2=1,1,1)
i | 0 | 1 | 2 | 3 | 4 | 5 | 6
Eigenvalue of (M0,M1,M2) | (-1,1,1) | (1,-1,1) | (1,1,-1) | (1,-1,-1) | (-1,1,-1) | (-1,-1,1) | (-1,-1,-1)
Y error
i | 0 | 1 | 2 | 3 | 4 | 5 | 6
Eigenvalue of (M0,M1,M2)=(N0,N1,N2) | (-1,1,1) | (1,-1,1) | (1,1,-1) | (1,-1,-1) | (-1,1,-1) | (-1,-1,1) | (-1,-1,-1)
Figure 1: Encoding circuit of Steane code. Figure 2: Detection circuit of
Steane code.

##  III The impact of QEC on privacy budgets

From the previous sections, we can observe that the only parameter impacting
$\epsilon$ in Eq. (4) is the error probability $p$ of the depolarizing noise.
In the case of quantum computing with only a single qubit gate, we derive the
error probability $p^{\prime}$ of the depolarizing noise after the QEC. We
then extend it to multiple single qubit gates and obtain different error
probabilities $p^{\prime}$ by deciding the location of the QEC, which further
satisfies the adjustable privacy budget $\epsilon$ in QDP. Finally, we
consider the impact of multi-level concatenated QEC on the QDP. We leave the
analysis of $\epsilon$ for multiple qubit gates in future work.

###  III-A DP for a single qubit gate

First, we continue the setup in [13] by assuming that the quantum circuit only
consists of a single qubit gate and replacing the general noises with the
depolarizing noise on this gate with an error probability of $p$.

We first encode this single logical qubit into seven physical qubits for
subsequent QEC operations. According to the theorem of Steane code [15], the
depolarizing noise error rate on each physical qubit is $p$. Therefore, when
QEC is not applied, the accuracy of one logical qubit is $1-p$; the total
accuracy of seven physical qubits is $(1-p)^{7}$. The Steane code can correct
arbitrary errors on a single physical qubit, but it cannot correct the case
where multiple physical qubits produce errors at the same time. Thus, it can
correct $7\times p\times(1-p)^{6}$ error rate for seven physical bits.
Overall, after QEC operation is applied, a single qubit gate can have the
following accuracy and error rate:

$\displaystyle c^{\prime}=(1-p)^{7}+7\times p\times(1-p)^{6},$ (5)
$\displaystyle p^{\prime}=1-\left[(1-p)^{7}+7\times p\times(1-p)^{6}\right],$
(6)

where $c^{\prime}$ and $p^{\prime}$ represent the circuit accuracy and the
final depolarizing noise error rate respectively.

We substitute $p^{\prime}$ into the Eq. (4), and can obtain new formulas for
the depolarizing noise error rate and privacy budgets:
$\epsilon=\ln{[1+\frac{1-p^{\prime}}{p^{\prime}}dD]}$.

Usually, we hope that QEC can reduce the noise error probability, so the
accuracy after using QEC operation needs to be higher than the previous one,
which satisfies

$\displaystyle 1-\left[(1-p)^{7}+7\times p\times(1-p)^{6}\right]>1-p.$ (7)

When $p<\approx 0.0579$, QEC operation can reduce the error rate $p$ of the
depolarizing noise which can show its own advantage. Therefore, in the
simulation, we set the error rate of depolarizing noise between 0 and 0.05.

###  III-B DP for two qubit gates

In this section, we first use two single qubit gates as an example, assuming
that the depolarizing noise as global noises with error rate $p$ occurs on
each gate. Without QEC operation, the final accuracy of the logical qubit is
$(1-p)\times(1-p)$.

We still start by encoding a single logical qubit into seven physical qubits.
We propose the following two cases based on whether or not we perform QEC
operation for each gate. The circuit schematic is shown in Fig. 3.

Figure 3: (a) QEC operation for each gate. (b) Selective QEC operation for
gates. $U_{1}$ and $U_{2}$ are arbitrary single-bit circuit gates. $N_{P1}$
and $N_{P2}$ are depolarizing noise added to the gate. QEC is Steane code
circuit.

QEC operation for each gate: In this case, each qubit gate is followed by the
QEC operation. According to [22], assuming that the two qubit gates are
independent of each other, their final error rate is

$\begin{split}&p_{1}^{\prime}=1-c^{\prime}\times
c^{\prime}=1-\left[(1-p)^{7}+7\times p\times(1-p)^{6}\right]^{2}.\end{split}$
(8)

Therefore, Eq. (4) can evolve into
$\epsilon=\ln{[1+\frac{1-p_{1}^{\prime}}{p_{1}^{\prime}}dD]}$.

Selective QEC operation for gates: In the second case, we selectively perform
QEC operations on quantum gates. For a two qubit gates circuit, we perform QEC
operation on only one gate. In the case of QEC on the first gate, the
depolarizing noise rate consists of the accuracy $c^{\prime}$ of the first
gate after it has been corrected by the QEC and the original accuracy $1-p$ of
the second gate. Thus, the total error rate of the depolarizing noise can be
expressed as

$\begin{split}&p_{2}^{\prime}=1-c^{\prime}\times(1-p)\\\
&=1-\left[(1-p)^{7}+7\times p\times(1-p)^{6}\right]\times(1-p).\end{split}$
(9)

Therefore, Eq. (4) can evolve into
$\epsilon=\ln{[1+\frac{1-p_{2}^{\prime}}{p_{2}^{\prime}}dD]}$.

It is worth noting that since Steane code can only correct errors in one
physical qubit and cannot correct errors in two physical qubits, the error
rate of the quantum circuit is the same regardless of whether the QEC is
located after the first gate or after the second gate.

Figure 4 shows the relationship between the depolarizing noise error rate and
the locations of QEC in the case of two qubit gates. QEC operation reduces the
error rate, and an increase in the number of QEC applications results in a
smaller error rate.

Figure 4: Relationship between noise error rate and different locations of QEC
in two gates case.

###  III-C DP for multiple single qubit gates

We then extend two single qubit gates to multiple single qubit gates to obtain
a generalized formula of $p^{\prime}$ and the corresponding privacy budget. We
assume that the noise model of the circuit is as shown in Fig. 5(a), with the
depolarizing noise added to each of the $n$ single quantum gates, and the
error rates are all $p$.

No QEC operations: When the quantum circuit consists of $n$ single quantum
gates, the total error rate of the depolarizing noise is calculated in the
same way as [22]. Figure 5(b) can represent the case of Fig. 5(a), i.e., the
total depolarizing noise on each gate can be expressed as a global
depolarizing noise added at the end of the circuit, with a total error rate

$\begin{split}p_{3}^{\prime}=1-{\textstyle\prod_{i=1}^{n}}\left(1-p_{i}\right)=1-\left(1-p\right)^{n}.\end{split}$
(10)

Therefore, Eq. (4) can evolve into
$\epsilon=\ln{[1+\frac{1-p_{3}^{\prime}}{p_{3}^{\prime}}dD]}$.

Selective QEC operations for gates: We randomly selected $m$ gates ($m\leq n$)
to apply the QEC operations, and the total accuracy of these $m$ gates is
$\left(1-p^{\prime}\right)^{m}$. The total accuracy of the other $n-m$ gates
without QEC operation is $\left(1-p\right)^{n-m}$. Thus, the total error rate
of the circuit in this case is

$\begin{split}p_{4}^{\prime}=1-\left(1-p^{\prime}\right)^{m}\times\left(1-p\right)^{n-m}.\end{split}$
(11)

Therefore, Eq. (4) can evolve into
$\epsilon=\ln{[1+\frac{1-p_{4}^{\prime}}{p_{4}^{\prime}}dD]}$.

It should be noted that QEC applied to every gate is a special case, i.e.,
$m=n$, where the total error rate of the circuit with depolarizing noise is
$1-\left(1-p^{\prime}\right)^{n}$ and the privacy budget is
$\epsilon=\ln{[1+\frac{1-[1-\left(1-p^{\prime}\right)^{n}]}{1-\left(1-p^{\prime}\right)^{n}}dD]}$.

Figure 5: The depolarizing noise model. (a) $\sigma$ is a quantum state,
$\left|a\right\rangle$ is auxiliary quantum bits. $U_{i}$ is arbitrary single-
bit circuit gates, where $i=1...n$. $N_{pn}$ is depolarizing noise along the
circuit. (b) $N_{p}$ is global depolarizing noise.

###  III-D Multi-level concatenated Steane code

Now we encode the logical qubit by using multi-level concatenated Steane code
[23], which can further decrease $p$. As an example of two-level concatenated
Steane codes, we encode each of the seven physical qubits using Steane code.
Thus, we use $7^{2}=49$ physical qubits to represent a logical qubit. In a
single qubit gate circuit, the total error rate of the depolarizing noise with
QEC operation will become

$\begin{split}p_{5}^{\prime}=1-\left[(1-p^{\prime})^{7}+7\times
p^{\prime}\times(1-p^{\prime})^{6}\right].\end{split}$ (12)

Therefore, Eq. (4) can evolve into
$\epsilon=\ln{[1+\frac{1-p_{5}^{\prime}}{p_{5}^{\prime}}dD]}$.

We find that $p_{5}^{\prime}<p^{\prime}$, suggesting that multi-level
concatenated Steane code can have a further reduction in the depolarizing
noise error rate at the cost of circuit complexity. The case of adding multi-
level concatenated Steane code to multiple qubits gates will be discussed in
future work.

##  IV Performance evaluation

###  IV-A Simulation setup

In this section, we used numerical simulations to detect the relationship
between the various parameters of DP formulation. Throughout the simulations,
unless otherwise noted, the default settings of parameters of a single qubit
gate circuit are $D=2$, $p=0.03$, and $d=0.5$, where $D$ is the dimension of
the Hilbert space, $p$ is the error rate of the depolarizing noise added at
each gate, and $d$ is the trace distance between the two quantum states. In
subsequent simulations, the remaining default parameters were kept constant as
we investigated the effect of changes in individual parameters on DP.

###  IV-B The impact of trace distance on QDP

We investigate the effect of the trace distance between two quantum states on
the privacy budget at a single qubit gate circuit. As shown in Fig. 6, when
the trace distance $d$ becomes larger, the privacy budget also becomes larger,
which indicates that the degree of privacy protection becomes lower. The trace
distance is the difference between two quantum states. When $d$ becomes
larger, the difference between quantum states is larger, and therefore it
becomes more difficult to achieve privacy protection. In practice, two quantum
states need to be encoded, and then quantum computation is performed. The
process of encoding gives the current trace distance of the two quantum states
and thus the corresponding privacy budget. For two specific quantum states,
the value of $d$ is usually determined, so we cannot achieve regulation of the
privacy budget by adjusting $d$.

Figure 6: Relationship between privacy budgets and trace distance. Figure 7:
Relationship between privacy budgets and noise error rate of one gate. Figure
8: Relationship between privacy budgets and noise error rate under different
locations of QEC.

###  IV-C QEC achieves regulation of privacy budgets

Single qubit gate calculation: We set the range of the depolarizing noise
between 0 and 0.05. It is the threshold that guarantees the effect of QEC.
Figure 7 shows the effect on the privacy budget brought by the variation of
the noise error rate $p$. As $p$ becomes larger, the privacy budget is
smaller, and the privacy protection is better, but the data usability becomes
worse. Depolarizing noise in the circuit already satisfies a certain level of
privacy protection. After adding QEC to the circuit, the final noise error
rate $p^{\prime}$ becomes smaller, so the privacy budget becomes larger.

Two qubit gates calculation: The quantum circuit is two single qubit gates,
with depolarizing noise added to each gate, and both with error rate $p$. We
split this simulation into three cases: no QEC operation, QEC after each gate,
and choosing only one quantum gate to perform QEC operation. We study its
final error rate versus privacy budgets, and the results are shown in Fig. 8.
We find that the depolarizing noise error rate decreases and the privacy
budget is larger than the original budget in both cases where the QEC
operation is added. The case of adding QEC after each gate has a smaller error
rate than selectively adding QEC. This illustrates that introducing QEC can
reduce the noise and thus affect the privacy budget. Deciding whether or not
to add a QEC operation to each quantum gate affects the error rate of the
noise, which in turn impacts the privacy budget of QDP.

Multi-level concatenated Steane code: The privacy budget of a single qubit
gate using multi-level concatenated Steane code is shown in Fig. 9. Compared
to a single-level Steane code, the multi-level concatenated Steane code can
further diminish the noise and achieve a higher privacy budget at the cost of
increased circuit complexity.

Figure 9: Relationship between privacy budgets and multi-level concatenated
Steane code.

##  V Conclusion and future work

In this paper, we use QEC method to reduce the error rate, leading to solving
excessive inherent noises and gaining the target privacy budget. We find that
whether to use QEC for each gate affects the depolarizing noise error rate
progressively. We summarize the general noise error rate formula for applying
QEC operations to multiple single qubit gates and the corresponding privacy
budget $\epsilon$. In addition, we explore how multi-level concatenated Steane
code can further reduce noise at the expense of circuit complexity. Our
simulation results show that QEC can reduce the noise error rate and thus
increase the privacy budget for a specific depolarizing noise error rate
interval. Therefore, we can claim that QDP protection can be tuned/controlled
through QEC operation. Since this paper mainly focuses on theoretical analysis
and numerical study, we plan to conduct experiments on quantum devices to
further verify the proposed ideas in the future.

## References

  * [1] L. Gyongyosi and S. Imre, “A survey on quantum computing technology,” _Computer Science Review_ , vol. 31, pp. 51–71, February 2019.
  * [2] V. Mavroeidis, K. Vishi, M. D. Zych, and A. Jøsang, “The impact of quantum computing on present cryptography,” _arXiv preprint arXiv:1804.00200_ , March 2018.
  * [3] A. Ajagekar and F. You, “Quantum computing for energy systems optimization: Challenges and opportunities,” _Energy_ , vol. 179, pp. 76–89, July 2019.
  * [4] C. Berger, A. Di Paolo, T. Forrest, S. Hadfield, N. Sawaya, M. Stęchły, and K. Thibault, “Quantum technologies for climate change: Preliminary assessment,” _arXiv preprint arXiv:2107.05362_ , June 2021.
  * [5] F. Furrer, T. Franz, M. Berta, A. Leverrier, V. B. Scholz, M. Tomamichel, and R. F. Werner, “Continuous variable quantum key distribution: finite-key analysis of composable security against coherent attacks,” _Physical review letters_ , vol. 109, no. 10, p. 100502, September 2012.
  * [6] C. Lin, Z. Song, H. Song, Y. Zhou, Y. Wang, and G. Wu, “Differential privacy preserving in big data analytics for connected health,” _Journal of medical systems_ , vol. 40, pp. 1–9, April 2016.
  * [7] Z. Sun, Y. Wang, M. Shu, R. Liu, and H. Zhao, “Differential privacy for data and model publishing of medical data,” _Ieee Access_ , vol. 7, pp. 152 103–152 114, October 2019.
  * [8] F. Kargl, A. Friedman, and R. Boreli, “Differential privacy in intelligent transportation systems,” in _Proceedings of the sixth ACM conference on Security and privacy in wireless and mobile networks_ , April 2013, pp. 107–112.
  * [9] C. Dwork, “Differential privacy,” in _International colloquium on automata, languages, and programming_ , July 2006, pp. 1–12.
  * [10] C. Hirche, C. Rouzé, and D. S. França, “Quantum differential privacy: An information theory perspective,” _IEEE Transactions on Information Theory_ , vol. 69, no. 9, pp. 5771–5787, September 2023.
  * [11] W. M. Watkins, S. Y.-C. Chen, and S. Yoo, “Quantum machine learning with differential privacy,” _Scientific Reports_ , vol. 13, no. 1, p. 2453, February 2023.
  * [12] Y. Du, M.-H. Hsieh, T. Liu, S. You, and D. Tao, “Quantum differentially private sparse regression learning,” _IEEE Transactions on Information Theory_ , vol. 68, no. 8, pp. 5217–5233, April 2022.
  * [13] L. Zhou and M. Ying, “Differential privacy in quantum computation,” in _2017 IEEE 30th Computer Security Foundations Symposium (CSF)_ , August 2017, pp. 249–262.
  * [14] J. Guan, W. Fang, M. Huang, and M. Ying, “Detecting violations of differential privacy for quantum algorithms,” _arXiv preprint arXiv:2309.04819_ , September 2023.
  * [15] J. Roffe, “Quantum error correction: an introductory guide,” _Contemporary Physics_ , vol. 60, no. 3, pp. 226–245, July 2019.
  * [16] P. Aliferis and A. W. Cross, “Subsystem fault tolerance with the bacon-shor code,” _Physical review letters_ , vol. 98, no. 22, p. 220502, May 2007.
  * [17] R. Duncan and M. Lucas, “Verifying the steane code with quantomatic,” _arXiv preprint arXiv:1306.4532_ , June 2013.
  * [18] A. G. Fowler, D. S. Wang, C. D. Hill, T. D. Ladd, R. Van Meter, and L. C. Hollenberg, “Surface code quantum communication,” _Physical review letters_ , vol. 104, no. 18, p. 180503, May 2010.
  * [19] M. Urbanek, B. Nachman, V. R. Pascuzzi, A. He, C. W. Bauer, and W. A. de Jong, “Mitigating depolarizing noise on quantum computers with noise-estimation circuits,” _Physical review letters_ , vol. 127, no. 27, p. 270502, December 2021.
  * [20] P. W. Shor, “Algorithms for quantum computation: discrete logarithms and factoring,” in _Proceedings 35th annual symposium on foundations of computer science_ , November 1994, pp. 124–134.
  * [21] D. Quan, C. Liu, X. Lv, and C. Pei, “Implementation of fault-tolerant encoding circuit based on stabilizer implementation and “flag” bits in steane code,” _Entropy_ , vol. 24, no. 8, p. 1107, August 2022.
  * [22] Y. Du, M.-H. Hsieh, T. Liu, D. Tao, and N. Liu, “Quantum noise protects quantum classifiers against adversaries,” _Physical Review Research_ , vol. 3, no. 2, p. 023153, May 2021.
  * [23] E. Nikahd, M. S. Zamani, and M. Sedighi, “Low-overhead code concatenation approaches for universal quantum computation,” _arXiv preprint arXiv:1707.00981_ , July 2017.


# Two-Face: Adversarial Audit of Commercial Face Recognition Systems

Siddharth D Jaiswal, Karthikeya Duggirala, Abhisek Dash, Animesh Mukherjee

###### Abstract

Computer vision applications like automated face detection are used for a
variety of purposes ranging from unlocking smart devices to tracking potential
persons of interest for surveillance. Audits of these applications have
revealed that they tend to be biased against minority groups which result in
unfair and concerning societal and political outcomes. Despite multiple
studies over time, these biases have not been mitigated completely and have in
fact increased for certain tasks like age prediction. While such systems are
audited over benchmark datasets, it becomes necessary to evaluate their
robustness for adversarial inputs. In this work, we perform an extensive
adversarial audit on multiple systems and datasets, making a number of
concerning observations – there has been a drop in accuracy for some tasks on
CelebSET dataset since a previous audit. While there still exists a bias in
accuracy against individuals from minority groups for multiple datasets, a
more worrying observation is that these biases tend to get exorbitantly
pronounced with adversarial inputs toward the minority group. We conclude with
a discussion on the broader societal impacts in light of these observations
and a few suggestions on how to collectively deal with this issue.111This work
has been accepted for publication at ICWSM 2022

## Introduction

AI based systems are increasingly being used to assist human decision makers
in hiring, granting loans, recidivism prediction; i.e., decisions that have
larger consequences on the life and livelihood of the stakeholders (O’neil
2016; Noble 2018). To this end, even systems that are not specifically geared
for such high stakes decision making sometimes end up being a part of the
decision pipeline. For example, while Facial Recognition Systems (FRSs) are
not meant for identifying individuals in the criminal justice system, it is
very likely that such systems are being used to identify suspects in reality
(Sangomla 2020). Thus, an error in the outcome of a face recognition algorithm
can have serious repercussions. For example, an innocent person can be
wrongfully accused of crime or a criminal might be set free due to their
inaccurate identification.

Societal concerns surrounding usage of FRSs: FRSs are widely deployed for
tasks such as: estimating customer satisfaction, demographic characteristics
of population, tracking individuals etc. (Nagpal et al. 2019). These are done
under the broader purview of customer understanding, research and security
scenarios (Amazon 2021; Microsoft 2021; Face++ 2021). However, the reality of
FRS deployments, reveals that often their usages are vulnerable to abuse,
causing intentional or unintentional discrimination against marginalized
groups (Benjamin 2019; Buolamwini and Gebru 2018; Raji et al. 2020). In fact
in some cases, FRSs with accuracy lower than 1% have been used to
differentiate between conscientious citizens and anarchic rioters in the
context of civil riots (Sangomla 2020). The stakes are particularly high
considering Amazon, HireVue etc., are also selling their services to help in
policing and hiring candidates (Buolamwini 2018; Snow 2018).

Prior audits of FRSs: Like most AI based systems, these facial recognition
systems are trained on large scale data. However, multiple recent studies have
repeatedly shown how such systems are prone to be biased against certain
sections of the population (Mehrabi et al. 2019; Bolukbasi et al. 2016). Such
biases can be attributed to either biased training instances or procedural
flaw in the underlying algorithms. The lack of open access to such data driven
algorithms or any established model cards (Mitchell et al. 2019) further
impede the public know-how of the working of these systems and source of these
biases (Sandvig et al. 2014). Thus, the only way to evaluate the bias in these
software is to conduct black box audits (Sandvig et al. 2014; Dash, Mukherjee,
and Ghosh 2019; Dash et al. 2021). To this end, prior audits have shown
commercial face detection systems by IBM, Microsoft, Amazon, and Face++ to be
biased to faces belonging to non-White ethnic groups and / or female gender
(Buolamwini and Gebru 2018; Raji et al. 2020).

Potential of temporal and emergent biases: While these systems have been
previously audited by researchers, the extent of biases and even performance
of the systems seem to vary with time. Further, in the aim of improving on one
aspect (e.g., fairness in gender detection), systems can potentially overlook
several other aspects such as: performance on other tasks (e.g., age
prediction, emotion detection) and / or are prone to exacerbate reverse
discrimination. As per the taxonomy of biases proposed by (Mehrabi et al.
2019), these concerns fall in the scope of emergent bias. Thus, we posit
systems of such significant societal consequences should be periodically
audited for their performance and biases. This brings us to our first research
question.

RQ(1): Do the performances (and biases (if any)) of commercial FRSs vary over
time (for the same dataset)?

Need for adversarial audits: Note that the images taken by surveillance
cameras are often prone to different kind of noises caused due to several
practical reasons. For example, their lenses are exposed to environmental
elements, e.g., rain, dirt, storms etc.; further they do not cover all areas
of the locality with the same resolution. Hence, auditing the performance on
well formed, noiseless images is not enough. We therefore posit that their
performance on noisy or perturbed images is of equal importance. This provides
the context for our second research question.

RQ(2): Are the commercial FRSs immune to perturbation or noises induced in
captured images?

To this end, adversarial attack on images has been extensively used in the
context of adversarial learning in AI/ML (Vakhshiteh, Nickabadi, and
Ramachandra 2020). Taking a leaf out of such techniques, we also audit the
commercial FRSs for their performance and biases (if any) by poisoning images
using adversarial noise models (more details in the section on datasets and
adversarial inputs).

Current work: Keeping the aforementioned considerations in mind, in this work,
we perform an audit study of three such commercial facial detection systems by
Microsoft, Amazon and Face++. We demonstrate our experiments on publicly
available CelebSET (Raji et al. 2020), FairFace (Sixta et al. 2020) and
Chicago Face Database (CFD) (Ma, Correll, and Wittenbrink 2015; Ma, Kantner,
and Wittenbrink 2020; Lakshmi et al. 2021) datasets. Note that our work
focuses on face identification task by FRSs. Additional objectives of person /
face re-identification or matching is beyond the scope of the current work. We
leave these additional objectives and audits thereof for an immediate future
work. We summarize our contributions and observations as follows.

  * •

By auditing the CelebSET dataset, we demonstrate the inconsistent performance
of FRSs as compared to the prior audit (Raji et al. 2020). While we observe
marginal drop in performance in the gender classification task, the drop in
performance for age detection task is alarming ($\geq 15\%$ drop for both
Amazon, and Microsoft).

  * •

By generating noisy adversarial images with the help of easily understandable
open source software GIMP, we conduct audits on CelebSET, FairFace and CFD.

    * –

CelebSET: We observe a significant drop in accuracy for all tasks, with the
three FRSs performing particularly poorly on RGB noise. This raises serious
questions on the robustness of such systems.

    * –

FairFace: We observe a significant drop in accuracy on all but one tasks for
the images poisoned with RGB noise for all FRSs. The FRSs perform slightly
better on the spread noise. Thus the accuracy of the FRSs is highly dependent
upon the type of noise.

    * –

CFD: The FRSs are audited for two tasks and exhibit a slightly more robust
behaviour with a significant drop in accuracy only for Newsprint noise. Thus
the accuracy of the FRSs is dependent on a combination of the dataset and
noise.

  * •

Further involved analyses of the predictions suggest that the disparity in
intersectional accuracy for minority groups increase drastically with the
adversarial inputs. In CelebSET, Microsoft FRS reports a disparity of $\geq
30\%$ for all tasks between people belonging to ‘White’ ethnicity and ‘Black’
ethnicity. In FairFace, the disparity is always against people of color. For
Microsoft FRS, the disparity is always against ‘Black’ males while the other
FRSs display disparity against other ethnic groups as well. For instance,
gender detection on the AWS FRS exhibits a disparity as high as $74\%$ against
‘East Asian’ males. Similarly, in CFD, the disparity in gender accuracy is
$93\%$ against ‘Black’ females and $44\%$ for age prediction against ‘Black’
males for Face++ FRS. This shows that adversarial noises also widen the
disparity against minority groups along with low prediction accuracy.

To the best of our knowledge, this is the first attempt to perform an
adversarial audit of the commercial facial recognition systems. We believe
that observations in this paper might ignite further discussion and
deliberation among practitioners and researchers regarding the ethical and
governance aspect of such high-stakes technology.

## Background and related work

Tasks | Microsoft | Amazon | Face++
---|---|---|---
Gender detection | ✓ | ✓ | ✓
Age detection | ✓ | ✓ | ✓
Smile detection | ✓ | ✓ | ✓
Emotion detection | ✓ | ✓ | ✓
Bounding box detection | ✓ | ✓ | ✓
Facial landmarks | ✓ | ✓ | ✓
Beauty score | ✗ | ✗ | ✓
Ethnicity | ✗ | ✗ | ✓
Skin status | ✗ | ✗ | ✓
Table 1: Some of the important tasks performed by each of the individual
commercial Facial Recognition Systems. We compare them on the first three
tasks keeping our audit in-line with the prior audits.

In this section, we first provide a brief background about the commercial FRSs
that we study. This is followed by a review of prior literature on two related
strands of research – (a) bias in FRSs, (b) adversarial attacks in computer
vision.

### Facial recognition systems (FRSs)

In this work we specifically study the following FRSs

  * •

Amazon AWS Rekognition (Amazon 2021)

  * •

Microsoft Azure Face (Microsoft 2021)

  * •

Face++ Detect (Face++ 2021)

These systems are readily available for individual and commercial users. They
provide a variety of services like identification of gender, age, emotion,
facial features, and even beauty (Arwa Mahdawi 2021; Tate Ryan-Mosley 2021) on
input images.

The list of different prominent tasks performed by each of these facial
recognition system is listed in Table 1. Even though the systems offer a
number of tasks where they are comparable, keeping in line with the prior
research by (Buolamwini and Gebru 2018; Raji et al. 2020), we compare them
primarily on the first three tasks, i.e., (1) gender detection, (2) age
detection, and (3) smile detection 222In CFD dataset, as all images have
neutral face expression, we have not performed comparison on smile detection
task.. Note that even though there are some potentially controversial tasks
mentioned in Table 1 (last three rows), since they are not performed by two of
the FRSs, we exclude them from our further investigation in this work.
However, we acknowledge that these tasks need to be studied and investigated
on their own as part of future research in this domain.

Difference in outputs for age prediction: For the age prediction task while
Microsoft and Face++ systems report an exact predicted age, Amazon system
reports a window which ranges between 10-18 years. For example, if the
returned window is $[x,x+10]$, the system predicts the age of the subject to
be at least $x$ years and at most $x+10$ years. In this study, we use the
median value from Amazon system’s predicted range as the age of the subject in
the image for comparison with the other two systems.

### Bias in FRSs

AI based systems are data driven and are shown to be prone to inadvertent
consequences such as bias and unfairness. FRSs are no exception. Even though
these systems are reported to achieve an overall accuracy upwards of 80%, the
problem lies underneath. Buolamwini and Gebru (2018) showed the disparity of
these systems for the first time by highlighting the difference in performance
for the dark-skinned population as compared to the fair-skinned counterparts.
There have been further follow up studies which have audited the same and
other commercial FRSs (Jung et al. 2018; Raji and Buolamwini 2019; Kyriakou et
al. 2019a; Raji et al. 2020; Sixta et al. 2020) which showed even after the
first audit, these disparities prevailed and also reported different
performance diagnostics at different points of time.

On the positive impact of such studies, IBM has stopped developing facial
recognition technology (Jay Peters 2020) and there have been calls by watchdog
organizations for others to follow suit (STOP 2021).

Relevance to the ICWSM community: Computational social science community has
shown continuous interest in this area of research. For instance, recent works
(Kyriakou et al. 2019b; Barlas et al. 2019) have explored concerns over
fairness in image tagging applications by commercial FRSs. On the other hand,
many studies have used these FRSs, especially Face++, as a part of their
research pipeline for analysing the different attributes of Twitter (Vikatos
et al. 2017; Messias, Vikatos, and Benevenuto 2017) and Instagram users (Pang
et al. 2015). We posit that a biased FRS can lead to researchers making false
conclusions.

Although most studies highlighted above are directed toward the first research
question we posed, the variation in their findings further highlight the
importance of regular third-party audits of these commercial systems.

### Adversarial attack in computer vision

Adversarial attacks have been leveraged to test the robustness of many
computer vision based systems (Vakhshiteh, Nickabadi, and Ramachandra 2020).
In these attacks, some noise is intentionally added to the images (Goodfellow,
Shlens, and Szegedy 2014; Xiao et al. 2019; Qiu et al. 2020; Xu et al. 2020;
Maesumi et al. 2021; Duan et al. 2020) or videos (Jiang et al. 2019; Chen et
al. 2021) and then they are fed to the system to check how immune or robust
they are to these changes.These may also be used to increase privacy for non-
consenting individuals (equalAIs 2021). These techniques may range from being
as simple as directly modifying the image file’s bits, using image
manipulation software like Gimp (GIMP 2021), using libraries like (Bloice,
Roth, and Holzinger 2019), or as sophisticated and involved as using game-
theoretic techniques (Oh, Fritz, and Schiele 2017), and deep learning
techniques (Chandrasekaran et al. 2020; Goel et al. 2018; Garofalo et al.
2018; Bose and Aarabi 2018; Massoli et al. 2021; Xiao et al. 2019; Jiang et
al. 2019; Qiu et al. 2020; Xu et al. 2020; Maesumi et al. 2021; Duan et al.
2020; Chen et al. 2021). We do not explore sophisticated adversarial attacks
as part of this work. Instead we focus on simple attacks that may be a result
of real world situations, as discussed later.

As part of our second research question, we use some rudimentary noise models
using Gimp (discussed further in the dataset section) to perturb the input
facial images and see how robust the commercial FRSs are to such noises. We
not only evaluate the overall robustness of the systems but also execute
involved analyses regarding the disparity of performance among intersectional
groups. We call this audit – ‘adversarial audit’ and to the best of our
knowledge, this is the first attempt to understand the robustness of these
FRSs under an adversarial audit.

## Datasets and adversarial inputs

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

Figure 1: Images from CelebSET (a-c), FairFace (d-f) and CFD (g-i).

In this section, we describe the datasets that we use for our
experiments333Dataset and code are available at https://tinyurl.com/xt5wdmw
along with an outline of the perturbations that we have applied on the images
to create adversarial inputs.

### Datasets

We use three annonated datasets for auditing the FRSs. These datasets are
composed of face images of people from different demographics (based on
ethnicity, gender, and age group): CelebSET (Raji et al. 2020), Chicago Face
Database (Ma, Correll, and Wittenbrink 2015; Ma, Kantner, and Wittenbrink
2020; Lakshmi et al. 2021) and FairFace (Karkkainen and Joo 2021).

CelebSET – This dataset is composed of 1600 images of 80 celebrities – 20 of
the most photographed celebrities from each of the following intersectional
groups: White Male (WM), White Female (WF), Black Male (BM) and Black Female
(BF) (Raji et al. 2020). Thus there are 20 celebrities from each category with
20 images each. The authors manually selected 10 smiling and 10 non-smiling
images of each celebrity. The images are annotated with name, ethnicity,
gender, age, smile and bounding box coordinates. The age of the person in the
photo is estimated by subtracting the birth-date from the date of the photo.
Each of the photos is present as a 128 $\times$ 128 image file with only the
face area visible. The photos are not of very high quality and there is no
consistency in the lighting or viewing angle. One of the images from this
dataset is shown in Figure 1(a).

FairFace – This dataset is composed of 108,501 images taken from the YFCC-100M
Flickr dataset belonging to the following ethnic groups – White, Black,
Latino, Indian, Southeast Asian, East Asian and Middle Eastern (Karkkainen and
Joo 2021). The images are annotated with gender, age and ethnicity. We
randomly sampled 7000 images from this dataset for our experiments. We chose
1000 images for each ethnicity equally divided for the two genders. Each
picture is present as a 224 $\times$ 224 image file with only the face area
visible. The photos are not of very high quality and there is no consistency
in the lighting or viewing angle. One of the images from this dataset is shown
in Figure 1(d).

Chicago Face Database(CFD) – This dataset is composed of 597 unique images of
American citizens belonging to the following ethnic categories: White, Black,
Asian and Latino (Ma, Correll, and Wittenbrink 2015). The images are annotated
with ethnicity, gender, age, emotion, facial landmarks, etc.444More details
available at: https://chicagofaces.org/default/. The distribution of males and
females in the dataset for each of these ethnic groups is noted in Table 2.
These are high-resolution images with standardized lighting conditions and
angle of view with a resolution of 2444 $\times$ 1718 on a white background.
All images have a neutral face expression. A sample image from this dataset is
shown in Figure 1(g). There are two more datasets we use that fall under the
purview of this database.

  * •

CFD-MR (Ma, Kantner, and Wittenbrink 2020) – Composed of 88 unique images of
American individuals who have parents belonging to different ethnic groups.

  * •

CFD-India (Lakshmi et al. 2021) – Composed of 142 unique images of Indian
individuals from New Delhi, India, annotated for the same features as above.

| White | Black | Asian | Latino
---|---|---|---|---
Males | 93 | 93 | 52 | 52
Females | 90 | 104 | 57 | 56
Table 2: Distribution of males and females across different ethnic groups in
CFD dataset.

### Inputs for adversarial audits

Recently, adversarial image manipulation techniques have been widely used to
confuse face detection softwares successfully (Vakhshiteh, Nickabadi, and
Ramachandra 2020). Even though there are multiple sophisticated deep learning
tools available for generating such adversarial/perturbed images, in this
work, we have made a conscious decision to step away from these sophisticated
techniques. We have chosen simpler tools and techniques that are more
accessible to people outside the academic community for perturbing the images.
In particular, to perturb an image we use the open-source image manipulation
program – GIMP (GIMP 2021) to create adversarial images to test the FRSs. GIMP
is freely available and has an intuitive GUI with a well-defined documentation
and therefore easily accessible for users intending to apply digital filters
or perform various image editing tasks. Our hypothesis is that even such
simple techniques available to unassuming users can elicit biased responses
from these softwares which can have dangerous ramifications. We highlight a
few ways in which such perturbed images could potentially manifest in physical
and digital world and can be nicely simulated by the GIMP software.

  * •

Social media applications like Instagram and Snapchat allow users to apply
various simple types of digital filters on photos (during and after
capturing). Such filters may distort or introduce various types of noises in
the images to emulate the effect of old analog cameras. These edited photos
are available to the back-end system as well as followers of the user and can
be easily stored and used for training/testing of FRSs. To simulate this
scenario we use the RGB noise filter which adds a normally distributed noise
to a layer or a selection, thereby, giving the effect of a grainy texture to
an image. This filter is applied to photos in all the datasets – CelebSET,
FairFace and CFD with examples in Figures 1 (c), (f) and (i)

  * •

CCTV cameras are being used increasingly for surveillance and policing of
individuals (Manish Singh 2020; Alexander Marrow 2020) by various governments.
These cameras are placed in public places and exposed to environmental
elements like rain, dirt, etc. Sometimes, a single camera may be covering a
large area and therefore needs to use zoom function which dilates or spreads
the pixels of the images. These unintended changes can result in disastrous
consequences as the facial recognition tasks are completely automated (Natasha
Lomas 2020). In order to emulate this scenario we use the spread noise filter
which swaps each pixel in the active layer or selection with another randomly
chosen pixel by a user specified amount, thereby, giving a slightly jittery
output. This filter is applied to photos in all the datasets – CelebSET,
FairFace and CFD, with one example in Figure 1(b).

  * •

Physical photographs clicked on analog or digital cameras, or from old
newspapers are restored to a digital format by
scanning555https://affidavit.eci.gov.in/candidate-affidavit and compressing.
Such digital copies are neither high quality nor are they expected to be in
any standardized format. Using FRSs to identify people from such images may
result in biased outcomes. These biases may even increase due to the quality
of the image. To simulate this situation, we use the newsprint filter which
halftones the image using a clustered-dot dither, thereby, giving the effect
of a newspaper print. Given the suitability of the images based on their
quality and on manual inspection of the output images, we determined that this
filter should be applied to photos in CFD only, with one example shown in
Figure 1(h).

## Experiments and observations

In this section, we present the experimental setup followed by a discussion on
the observations for both traditional and adversarial audits conducted on the
aforementioned datasets.

### Tasks, datasets, and experimental setup

In this study, we evaluate the FRSs for the following tasks.

  * •

Gender detection: Each of the FRS report either Male or Female gender for an
input image.

  * •

Age prediction: Given the reported age from Microsoft, Face++, and median of
the age range of AWS, we use a 8 years acceptance margin, i.e., ground truth
$\pm 4$. If the reported age falls in the margin of acceptance, we consider it
to be a correct prediction.

  * •

Smile detection: For this task, AWS reports a ‘smile’ or ‘no-smile’ label and
a confidence score for its prediction. Face++ and Microsoft report a
confidence value in $[0,100]$ and $[0,1]$ respectively. Thus we consider the
threshold as $50\%$ and $0.5$ respectively to consider the prediction as true
for Face++ and Microsoft.

Datasets: As mentioned earlier, we evaluate the following datasets – CelebSET,
FairFace and CFD (and its extensions – CFD-MR and CFD-India). We do not
evaluate the smile prediction task for FairFace (not annotated) and CFD
datasets (all images have a neutral facial expression). A summary of the
triplets of $<$dataset, filter, task$>$ that we evaluate is noted in Table 3.

Dataset | Filters | Tasks
---|---|---
CelebSET | Spread $|$ RGB | Gender $|$ Age $|$ Smile
FairFace | Spread $|$ RGB | Gender $|$ Age
CFD | Newsprint $|$ Spread $|$ RGB | Gender $|$ Age
Table 3: Filters and tasks for each dataset.

Experimental setup: We evaluate all datasets in batch mode through API calls
to the FRS servers. The parameter settings for each filter on Gimp are as
follows.

  * •

Spread: Horizontal/vertical spread amount – 50 units for CFD and 8 for
CelebSET and FairFace.

  * •

Newsprint: Screen cell width – 20 pixels, % of black pulled out – $12\%$,
screen angle for all colors – 75∘.

  * •

RGB: Noise in RGB channels – 0.8 for CFD and 0.5 for CelebSET and FairFace.

With higher noise values, images in CelebSET and FairFace were not
recognisable to human eyes either. Given the smaller resolution, we have
intentionally included less noise to these images. As mentioned in the dataset
section, CelebSET and FairFace images were heavily distorted due to newsprint
filter too. Hence, no further experiments were done for that setting.

### Observations for CelebSET

We first report the observations of the traditional audit experiments followed
by the results of adversarial audits.

FRS | Gender | Age | Smile
---|---|---|---
AWS | $-0.25\%$ | $-17.46\%$ | $-5.54\%$
Microsoft | $-0.63\%$ | $-21.22\%$ | $+12.94\%$
Table 4: Change in accuracy for CelebSET relative to Saving Face (Raji et al.
(2020)).

#### Change in prediction accuracy over time

Table 4 shows the change in accuracy for the gender, age, and smile prediction
tasks as compared against the findings of Raji et al. (2020). We report the
difference between the accuracy obtained in our investigation and that
reported by Raji et al. Hence a $-ve$ sign denotes that for the corresponding
FRS, for the corresponding task the prediction accuracy has decreased compared
to what was reported earlier.

Observations: Both AWS and Microsoft report a marginal (insignificant) drop in
accuracy for gender detection by $0.25\%$ and $0.63\%$ respectively. However,
the drop in accuracy for age prediction is alarming with a fall of more than
15% for both the FRS. Interestingly, while AWS performance has also
deteriorated in smile detection task, Microsoft has actually improved by a
significant margin ($13\%$).

Takeaways: While the FRS have somewhat stable performance in the gender
detection task, we observe significant difference in performance in the other
two tasks. This could be possibly an outcome of the fact that fixing of
results in one dimension might adversely affect the performance in another
dimension. However one needs to have access to the black box algorithms to
corroborate this postulation.

#### Effect of perturbations on the performance

(a) Accuracy on AWS

(b) Accuracy on Face++

(c) Accuracy on Microsoft

Figure 2: Prediction accuracy on CelebSET for original, RGB-filter and spread-filter inputs. Under adversarial settings, all the FRSs suffer from drop in accuracy (AWS has $\bf<2\%$ accuracy), with Microsoft being marginally more robust than AWS and Face++. Image | Gender | Age | Smile
---|---|---|---
AWS/original |  (WF - WM) $1\%$ |  (WF - BF) $19.25\%$ |  (BF - WM) $6.5\%$
AWS/RGB |  (WM - BM/BF) $1.75\%$ |  (WM - BM) $1.25\%$ |  (BF - BM) $1.25\%$
AWS/spread |  (WF - WM) 5.5% |  (WF - BF) 19.5% |  (BF - WM) 12.5%
Max. absolute increase in disparity | 4.5% (spread) | 0.5% (spread) | 6% (spread)
Face++/original |  (WM - BF) $2.5\%$ |  (WF - WM) 13.5% |  (BF - BM) $7\%$
Face++/RGB |  (WF - BM) 22.75% |  (BF - BM) $4.25\%$ |  (WF - BM) 20.75%
Face++/spread |  (WF - BM) $17.5\%$ |  (BM - WM) $2.5\%$ |  (WF - BM) $12.5\%$
Max. absolute increase in disparity | 20.25% (RGB) | $\phi$ | 13.75% (RGB)
Microsoft/original |  (WM/WF - BM/BF) $1\%$ |  (BM - BF) $16\%$ |  (WF - BM) $4.5\%$
Microsoft/RGB |  (WF - BM/BF) 36.25% |  (WF - BM) $15.75\%$ |  (WF - BM) 33.75%
Microsoft/spread |  (WF - BM/BF) $6.5\%$ |  (WM - BF) 30% |  (WF - BM) $9.5\%$
Max. absolute increase in disparity | 35.25% (RGB) | 14% (spread) | 29.25% (RGB)
Table 5: Difference in highest accuracy and lowest accuracy amongst the
intersectional groups for CelebSET. Most results show significant disparity in
prediction accuracy for people of ‘Black’ ethnicity (BF/BM) as compared to
that of ‘White’ ethnicity. Microsft reports the largest disparity for all
tasks, irrespective of the input.

The first striking observation that we make is on the number of images
identified by the FRSs.

  * •

Original – AWS: 1600, Face++: 1600, Microsoft: 1590.

  * •

Spread Filter – AWS: 1549, Face++: 1534, Microsoft: 1552.

  * •

RGB Filter – AWS: 20, Face++: 318, Microsoft: 683.

The above list shows that adding adversarial filters to low quality images
successfully fools the facial recognition systems. It is interesting to note
that the RGB filter has an extremely high mis-classification rate for all FRSs
with faces identified on less than $2\%$ images for AWS. Figure 2 presents the
prediction accuracy for the three tasks of gender, age and smile detection.
Each of the plots Figures 2(a), 2(b) and 2(c) correspond to an FRS API.

Gender identification: We see that all the systems report high gender
identification accuracy ($\geq 99\%$) for the original dataset. However, with
the addition of noise (i.e., in adversarial setting), this accuracy is
observed to be significantly affected. AWS reports an accuracy of only $1.5\%$
for the RGB filter (Figure 2(a)). The drop in accuracy for spread filter is
not as significant with the lowest reported accuracy being $92\%$ for Face++
(Figure 2(b)).

Age prediction: For the task of age prediction, all APIs perform badly even
for the original input with only Microsoft having an accuracy of $>50\%$
(Figure 2(c)). The adversarial images further reduce the performance with RGB
filter causing a drop of $>50\%$. Here again, Face++ has the lowest accuracy
for the spread filter – $\approx 28\%$ (Figure 2(b)).

Smile detection: All APIs are able to identify smiles with an accuracy $\geq
88\%$ for the original dataset, but exhibit a similar behaviour for the
accuracy on RGB and spread filters, with Microsoft being the best overall and
AWS being the worst.

The summary of the findings from Figure 2 are as follows.

  * •

FRSs perform well on the original input for gender and smile detection tasks,
but for age detection their accuracy drops significantly.

  * •

The perturbation caused due to different noises seem to affect the performance
of the FRSs. Especially, RGB noise has significant deterioration effect on the
performance across all the systems.

  * •

Microsoft is the most robust FRS across all tasks and inputs among the three
systems. However, its performance is also badly affected by the RGB noise.

#### Disparity in accuracy for intersectional groups

We now evaluate the disparity in accuracy by taking the difference of the
highest accuracy and the lowest accuracy among the different intersectional
groups – WM, WF, BM, BF of CelebSET dataset. The results are shown in Table 5.
Each cell in the table denotes the highest disparity for a given task for the
corresponding dataset. The mentions within parenthesis denote the
intersectional groups between which the corresponding disparity was observed.
Rows marked in red denote the maximum absolute increase in disparity from
original to any of the two perturbed images.

Gender detection: For the original images, we do not observe any significant
disparities between performance on intersectional groups ($\leq 3\%$ across
all systems). The disparity increases for the two adversarial filters for all
facial recognition systems. The maximum disparity of $36.25\%$ is observed for
Microsoft for RGB noise. This disparity is mostly against people of ‘Black’
ethnicity as compared to people of ‘White’ ethnicity. In fact, this disparity
is highest for Microsoft despite it being the most accurate (overall) amongst
the three systems (see Figure 2). It is interesting to note that even though
spread noise did not have any significant effect on the overall accuracy, the
disparity among intersectional groups saw significance rise in all FRSs (as
high as 17.5% for Face++).

Age prediction: For the original images, we observe significant disparities
between performance on intersectional groups ($\geq 10\%$ across all systems).
Disparity in accuracy in case of age prediction shows some intriguing trends.
For RGB noise, we observe that the disparity has decreased in all FRSs.
However, that is mostly due to the significant drop in overall accuracy in all
systems. However, for spread noise, we observe that the disparity in Microsoft
system has increased to 30% (an increase of 14%). Interestingly, for the rest
of the two systems, we do not observe significant rise in disparity. In fact,
the disparity reduces in Face++ by almost 11%.

Smile detection: For the original images, we observe marginal disparities
between performance on intersectional groups. However, the disparity in
accuracy increases upon addition of noise in the images (only aberration is –
AWS/RGB). We observe that both Face++ and Microsoft report the highest
disparity for RGB filter and AWS does so for the spread filter. This is to the
extent that the disparity in Microsoft for smile detection actually becomes
7.5 times of what it is for original images.

We observe similar trends for biases between gender and ethnic groups taken
together. Results have been omitted for brevity.

Summary

  * •

All FRSs exhibit significant increase in disparity of accuracy for both gender
and smile detection upon introduction of noise in the original images.

  * •

The observed disparity in accuracy is against individuals of ‘Black’ ethnicity
(BM or BF).

  * •

Even though we found Microsoft system to be the most robust among the three,
the disparities between the intesectional groups are also the maximum. Thus
the high accuracy is not dispersed among all intersectional groups uniformly;
rather error rate is significantly higher for images of dark skinned people.

### Observations for FairFace

We perform similar experiments as CelebSET, on FairFace. We observe that all
FRSs miss out on identifying a certain number of faces in the original set of
images. This is further exacerbated with the perturbed images, with Microsoft
identifying the least number of faces. The number of images identified per FRS
is

  * •

Original – AWS: 6903, Face++: 6918, Microsoft: 5717.

  * •

RGB Filter – AWS: 6229, Face++: 2461, Microsoft: 2163.

  * •

Spread Filter – AWS: 6855, Face++: 6764, Microsoft: 5647.

(a) Accuracy on AWS

(b) Accuracy on Face++

(c) Accuracy on Microsoft

Figure 3: Prediction accuracy on FairFace for original, RGB-filter and spread-
filter inputs. Under RGB and Spread adversarial settings, all of the FRSs
suffer from drop in accuracy (Face++ accuracy drops by more than $\bf 65\%$),
with AWS being significantly more robust than Microsoft and Face++.

#### Effect of perturbations in the performance

The results for the accuracy on gender and age detection tasks are presented
in Figure 3 for original and adversarial images.

Gender detection: All the FRS APIs report good performance for gender
detection. Microsoft API is found to be the worst among the three, identifying
the gender correctly for $80\%$ of the images. On adding the perturbations, we
see significant drop in accuracy for the RGB noise. We record a drop of more
than $40\%$ for all FRSs. Face++ and Microsoft are found to be the worst
affected APIs with a drop of $66.26\%$ (Figure 3(b)) and $51.45\%$ (Figure
3(c)) in accuracy respectively. Similar to the CelebSET dataset, we notice
that the FRSs are robust toward spread perturbation with no significant drop
in accuracy.

Age prediction: We observe that the age prediction accuracy is poor ($\leq
50\%$) for all APIs. The RGB noise perturbations show similar drop in accuracy
for this task (as in gender detection). Face++ is observed to be the worst
performing FRS with the lowest age prediction accuracy of $6.7\%$ (Figure
3(b)), followed closely by Microsoft at $12.67\%$ (Figure 3(c)). For the
spread noise, the drop is not significant; rather the accuracy for Microsoft
API increases marginally.

The key takeaways from Figure 3 are as follows.

  * •

All APIs perform well for the task of gender detection for all inputs except
RGB. However, the performance for age prediction is bad for all inputs on all
APIs with Face++ and Microsoft being the worst.

  * •

Microsoft is the worst performing FRS for gender detection on the original
images while Face++ is the worst for age prediction.

  * •

On the RGB perturbed images, Face++ is the worst performing FRS for both tasks
while AWS is the most robust.

  * •

AWS is the most robust FRS among the three.

#### Disparity in accuracy for intersectional groups

We evaluate the disparity in accuracy by taking the difference of the highest
accuracy and the lowest accuracy among the different intersectional groups
(two for each of the seven ethnic groups) – WM, WF, BM, BF, IM, IF, LM, LF,
SM, SF, EM, EF, MM and MF of FairFace dataset666Read the abbreviations WM:
‘White’ males, WF: ‘White’ females and so on for the different ethnic groups..
The results are presented in Table 6. In the table, each cell denotes the
highest disparity in accuracy for a given task. The phrases within the
parenthesis correspond to the intersectional groups which are the farthest
apart in accuracy. Finally, the rows marked in red show the absolute increase
in disparity from the original images to any of the noises.

Gender detection: For the original input images, the disparity in accuracy is
generally against the ‘Black’ ethnicity with Microsoft being the highest at
$20.2\%$. In the adversarial setting, for both Face++ and Microsoft, the
disparity is seen to be against ‘Black’ and ‘Indian’ ethnic groups; whereas
for AWS, it is against ‘East Asian’ and ‘Southeast Asian’ males. We notice
that for the RGB noise, the disparity increases by more than 1.5 times for all
FRSs with AWS reporting a disparity of $73.6\%$ against ‘East Asian’ males.
The maximum absolute increase in disparity is $63.60\%$ and reported for the
RGB perturbation. Although the disparities are distributed across ethnic
groups, all are against people of a specific skin tone (‘Asians’, ‘Indians’ or
‘Blacks’).

Age prediction: For the task of age prediction, all the FRSs have a disparity
of more than $9\%$ for original images. Here, the disparity is mostly against
males, with Microsoft reporting the highest disparity of $18.80\%$ against
‘Black’ males. The disparity increases significantly for the RGB filter for
AWS ($25\%$ against ‘Middle Eastern’ males) while it reduces for the other two
FRSs. We notice that even with the spread filter, the disparity does not
change by much and in fact reduces for Microsoft. The most important
observation here is that the disparity is against males for most of the cases
(either ‘Middle Eastern’ or ‘Black’), both from ethnic groups of color.

Summary

  * •

All FRSs exhibit an increase in disparity of accuracy for gender detection,
mostly directed toward individuals of ‘Black’ or ‘Asian’ ethnicity.

  * •

AWS reports the highest disparity for all experiments with a value of
$73.60\%$ between ‘Latino’ females and ‘East Asian’ males.

  * •

Microsoft reports the least absolute increase in disparity yet it is the most
disparate against ‘Black’ males; all its lowest accuracy values are reportedly
for this group.

Image | Gender | Age
---|---|---
AWS/original |  (MF - SM) $10\%$ |  (EF - IM) $14\%$
AWS/RGB |  (EF - EM) 73.60% |  (EF - MM) 25%
AWS/spread |  (MF - EM) $14\%$ |  (EF - IM) $11.80\%$
Max. absolute increase in disparity | 63.60% (RGB) | 11% (RGB)
Face++/original |  (LM - BF) $15.80\%$ |  (IM - MF) $9.40\%$
Face++/RGB |  (LF - BM) 30.60% |  (LF - BM) $6.40\%$
Face++/spread |  (MM - IF) $14\%$ |  (SM - WF) 13.40%
Max. absolute increase in disparity | 14.80% (RGB) | 4% (spread)
Microsoft/original |  (LF - BM) $20.20\%$ |  (EF - BM) 18.80%
Microsoft/RGB |  (SF - BM) 30.60% |  (SF - BM) $16.60\%$
Microsoft/spread |  (LF - BM) $22.40\%$ |  (EM - BM) 18.80%
Max. absolute increase in disparity | 10.40% (RGB) | $\phi$
Table 6: Difference in highest accuracy and lowest accuracy among the
intersectional groups for FairFace. Most results show significant disparity in
prediction accuracy for people of ‘Black’ ethnicity (BF/BM) as compared to
that of the other ethnic groups. AWS reports the largest disparity for all
tasks on the RGB adversarial input.

### Observations for CFD

We also perform a similar audit as CelebSET, on CFD. This dataset has higher
quality images with standardized lighting and face angles and has individuals
belonging to four different ethnic groups – ‘White’, ‘Black’, ‘Latino’,
‘Asian’. AWS and Microsoft APIs are able to identify faces in all 597 images
but Face++ identifies only 423 images with faces for the newsprint filter.

#### Effect of perturbations in the performance

(a) Accuracy on AWS

(b) Accuracy on Face++

(c) Accuracy on Microsoft

Figure 4: Prediction accuracy on CFD for original, RGB-filter, newsprint-
filter and spread-filter inputs. Under RGB and newsprint adversarial settings,
all of the FRSs suffer from drop in accuracy (Face++ accuracy drops by
$\bf>50\%$), with Microsoft being marginally more robust than AWS and Face++.

The prediction accuracy results for the tasks of Gender and Age detection are
presented in Figure 4. The results are divided per FRS API in Figures 4(a),
4(b) and 4(c).

Gender detection: We observe that all the FRS APIs report high accuracy for
gender detection. As regards to addition of different noises, we do not
observe significant degradation in performance for RGB or spread noise.
However, for the newsprint noise, we observe significant deterioration of
prediction accuracy in Amazon and Face++ systems with a drop of $\geq 40\%$
(see Figures 4(a), 4(b)).

Age prediction: Accuracy is low for all FRSs irrespective of the input. Here,
Face++ is the worst performing system with the accuracy of nearly $14\%$ for
Face++/RGB and nearly $47\%$ for Face++/spread (see Figure 4(b)). Microsoft,
on the other hand, returns the best results with its reported accuracy in the
range $\approx 62\%$ to $\approx 77\%$ (Figure 4(c)).

The key takeaways from Figure 4 are as follows.

  * •

All APIs perform well for the task of gender detection for all inputs except
newsprint, while the performance for age prediction is bad for all inputs on
AWS and Face++.

  * •

Face++ is the worst performing FRS for both gender and age prediction.

  * •

Microsoft is the most robust FRS among the three.

Image | Gender | Age
---|---|---
AWS/original |  (WF/LM/AM - BF) $7.69\%$ |  (WM - BM) $33.33\%$
AWS/RGB |  (BM - BF) $20.07\%$ |  (WF - AF) $24.16\%$
AWS/newsprint |  (WF - BM) 71.82% |  (LF - LM) 42.72%
Max. absolute increase in disparity | 64.13% (newsprint) | 9.39% (newsprint)
Face++/original |  (WM/LM/AM - BF) $21.25\%$ |  (LM - WF) $28.80\%$
Face++/RGB |  (WF/LF/AF - WM) $25.81\%$ |  (BM - AF) $21.91\%$
Face++/newsprint |  (LM - BF) 93.27% |  (LF - BM) 44.26%
Max. absolute increase in disparity | 72.02% (newsprint) | 15.46% (newsprint)
Microsoft/original |  (WM/WF/BM/LM/LF/AM/AF - BF) $0.96\%$ |  (AF - BF) 25.22%
Microsoft/RGB |  (BM/LM/LF/AM/AF - BF) $4.81\%$ |  (LF - BF) $21.57\%$
Microsoft/newsprint |  (BM/LF - BF) 11.54% |  (WM - BF) $21.08\%$
Max. absolute increase in disparity | 10.58% (newsprint) | $\phi$
Table 7: Difference in highest accuracy and lowest accuracy among the
intersectional groups for CFD. Most results show significant disparity in
prediction accuracy for people of ‘Black’ ethnicity (BF/BM) as compared to
that of the other ethnic groups. Face++ reports the largest disparity for all
tasks on the newsprint adversarial input.

#### Disparity in accuracy for intersectional groups

Next, we discuss the disparity in accuracy among the different intersectional
groups formed by taking male and female members from each of the ethnic groups
– ‘White’, ‘Black’, ‘Latino’, ‘Asian’. The results are presented in Table 7.
Each cell in the table denotes the highest disparity for a given task for the
corresponding dataset. Once again, the mentions within parenthesis denote the
ethnic groups between which the corresponding disparity was observed. Rows
marked in red denote the maximum absolute increase in disparity from original
to any of the two perturbed images. For paucity of space, we have reported the
results for the newsprint and the RGB filters only.

Gender detection: Barring Microsoft, both the other FRSs exhibit considerable
disparity in accuracy in gender detection for original input images (21.25%
for Face++ against ‘Black’ females). In the adversarial set up with RGB noise,
the disparities increase considerably. However, upon usage of newsprint noise,
the disparity among intersectional groups become scandalous. The maximum
absolute increase in disparity goes well beyond 60% for both Amazon and
Face++. Much like our prior observations, ‘Black’ males and Females are on the
wrong side of the disparity.

Age prediction: For age prediction task, all the three systems show
significant disparity in accuracy for ethnic groups for the original images
($\geq 25\%$ across FRSs). While all FRSs report a drop in disparity for the
RGB filter, AWS and Face++ report an increase for newsprint, with Face++
reporting the highest disparity at $44\%$. Notice, even though there is a
marginal drop in disparity, they are still comparable and more than 20% among
different intersectional groups. This on its own is very alarming. Microsoft
FRSs always exhibits disparity against ‘Black’ females; whereas AWS and Face++
show slightly different behaviour, with other minority groups too experiencing
disparate accuracy.

Summary

  * •

All FRSs exhibit an increase in disparity of accuracy for gender detection,
mostly directed toward individuals of ‘Black’ ethnicity.

  * •

Face++ reports the highest disparity for all experiments with a value of
$93.27\%$ between ‘Latino’ males and ‘Black’ females.

  * •

Microsoft is the best performing FRS, yet it is the most disparate against
‘Black’ females, with all its lowest accuracy values being reported for this
group.

### Observations for CFD-MR and CFD-India

Along with CFD, we also evaluate two other datasets – CFD-MR and CFD-India.
These are newly released datasets and allow us to study disparities in
accuracy for ethnic groups that are not generally studied in the context of
FRS audits.

#### Disparity in accuracy between males and females

Image | CFD-MR | CFD-India
---|---|---
| Gender | Age | Gender | Age
AWS/original | $3.23\%$ | $-10.04\%$ | $3.85\%$ | $15.77\%$
AWS/RGB | $-2.24\%$ | $7.69\%$ | $7.69\%$ | 23.08%
AWS/newsprint | -63.65% | -11.91% | -58.16% | $-16.75\%$
Max. abs. inc. in disparity | 66.88% (newsprint) | 17.73% (RGB) | 62.01% (newsprint) | 32.52% (newsprint)
Face++/original | $12.90\%$ | $4.84\%$ | $19.23\%$ | $4.36\%$
Face++/RGB | $-19.85\%$ | $8.93\%$ | $-0.30\%$ | -9.23%
Face++/newsprint | 69.23% | 29.03% | 62.90% | $-1.07\%$
Max. abs. inc in disparity | 56.33% (newsprint) | 24.19% (newsprint) | 43.67% (newsprint) | 13.59%(RGB)
Microsoft/original | $0\%$ | 18.12% | $0\%$ | $-0.64\%$
Microsoft/RGB | $0\%$ | $17.49\%$ | $0\%$ | $5.56\%$
Microsoft/newsprint | $0\%$ | $-7.2\%$ | 1.92% | 17.48%
Max. abs. inc. in disparity | $\phi$ | 25.32% (newsprint) | 1.92% (newsprint) | 18.12% (newsprint)
Table 8: Difference in highest accuracy and lowest accuracy between males and
females for CFD-MR and CFD-India. Most results show significant disparity in
prediction accuracy toward males. AWS and Face++ report the largest
disparities for all tasks on the newsprint adversarial input for both datasets
and Microsoft reports no disparity in gender detection for CFD-MR.

For brevity, we only discuss the disparities in accuracy between males and
females in each of the datasets. The results for CFD-MR and CFD-India are
shown in Table 8.

Gender detection: While Face++ has a disparity of $\geq 10\%$, AWS displays a
marginal disparity ($\leq 4\%$), and Microsoft system does not exhibit any
disparity in accuracy for prediction of the original images for both CFD-MR
and CFD-India datasets. However, upon imposing adversarial set up we see
interesting and different behaviors across the different FRSs. While Microsoft
is gracefully robust against all noises, AWS and Face++ are very sensitive in
this regard (esp. for newsprint noise). While newsprint noise increases
significant disparity ($-63.65\%$, $-58.16$ for CFD-MR and CFD-India
respectively) in AWS system against photographs of males, the trend is
completely reversed (significant discrimination against female photographs) in
Face++.

Age prediction: For CFD-MR, while AWS shows disparity in accuracy against
males ($-10\%$), Microsoft shows disparity against females ($18\%$). However,
apart from AWS system none of the other systems show considerable disparity on
original images for CFD-India dataset. For RGB noice, all the systems show
increasingly disparate performance toward females (except Face++ in CFD-
India). The trend is completely reversed in newsprint noise set up, where
majority of the disparity is against males (aberration Microsoft for CFD-
India). However, regardless of the direction, in the adversarial set up we
observe significant increase in disparity of accuracy and unlike gender
detection task, here Microsoft system is the most sensitive to adversarial
noises.

Summary:

  * •

We observe disparity in accuracy between genders for both the extension
datasets - CFD-MR and CFD-India.

  * •

Face++ is the most disparate FRS in CFD-MR while AWS is the most disparate in
CFD-India.

  * •

Newsprint filter results in the most instances of disparity.

## Conclusion

Our audit on three Facial Recognition Systems viz. Amazon AWS Rekognition,
Face++ Detect and Microsoft Azure Face, using the CelebSET,  FairFace and CFD
datasets, shows that even with high prediction accuracy values, there exists a
strong discriminatory bias against individuals of minority groups,
specifically those belonging to ‘Black’ ethnicity, for the tasks of gender,
age and smile prediction. We also observe that for adversarial inputs
generated from the same datasets, not only are the prediction accuracy values
reduced significantly but the disparity in these values between different
gender, ethnic or intersectional groups increase by large margins. It should
be noted that here again, the ‘Black’ ethnic group is at the receiving end of
this disparity more often than not.

Societal consequences: Our study has shown that the FRSs continue to display
disparate accuracies despite repeated audits and while the accuracy of some
tasks have improved, others have regressed, probably as a side effect. To this
end, model cards (Mitchell et al. 2019) have become an urgent necessity with
the release of these FRSs so that they could be used in appropriate contexts
only. We believe that this is one of the best way forward in establishing
trust with users and stress that it should be made mandatory for all future
systems that have real life implications. The foremost goal of every FRS
should be to have a fair behaviour and all efforts must be made in this
direction while adhering to privacy and security needs of end users. It is
also imperative that organizations release white-papers or hold workshops
explaining their technology so that the source of biases can be identified,
and hopefully, mitigated.

Future directions: An immediate next step in this work is to identify the
correlation between strength of noise with change in bias by testing for
different values of the parameters mentioned in the experiments section.
Another future step would be to expand the scope of tasks to test for emotion
prediction, face re-identification etc., and also study the impact of color
filters to evaluate the sensitivity of FRSs to skin color in the images.
Finally, we plan to evaluate with stronger adversarial generators which are
specifically designed to fool such recognition systems.

## References

  * Alexander Marrow (2020) Alexander Marrow. 2020.  “Russia’s lockdown surveillance measures need regulating, rights groups say” Accessed April, 2021.  URL https://www.reuters.com/article/us-health-coronavirus-russia-facial-reco-idUSKCN2253CM.
  * Amazon (2021) Amazon. 2021.  “Amazon AWS Rekognition.” Accessed April, 2021.  URL https://aws.amazon.com/rekognition/faqs/.
  * Amazon (2021) Amazon. 2021.  Amazon Rekognition FAQs.  https://aws.amazon.com/rekognition/faqs/.
  * Arwa Mahdawi (2021) Arwa Mahdawi. 2021.  “This AI-powered app will tell you if you’re beautiful – and reinforce biases, too” Accessed April, 2021.  URL https://www.theguardian.com/commentisfree/2021/mar/06/ai-powered-app-tell-you-beautiful-reinforce-biases.
  * Barlas et al. (2019) Barlas, P.; Kyriakou, K.; Kleanthous, S.; and Otterbacher, J. 2019.  Social B(eye)as: Human and Machine Descriptions of People Images.  _AAAI ICWSM_ 13\.
  * Benjamin (2019) Benjamin, R. 2019.  Race after technology: Abolitionist tools for the new jim code.  _Social Forces_ .
  * Bloice, Roth, and Holzinger (2019) Bloice, M. D.; Roth, P. M.; and Holzinger, A. 2019.  Biomedical image augmentation using Augmentor.  _Bioinformatics_ 4522–4524.
  * Bolukbasi et al. (2016) Bolukbasi, T.; Chang, K.-W.; Zou, J.; Saligrama, V.; and Kalai, A. 2016.  Man is to computer programmer as woman is to homemaker? debiasing word embeddings.  _arXiv preprint arXiv:1607.06520_ .
  * Bose and Aarabi (2018) Bose, A. J.; and Aarabi, P. 2018.  Adversarial attacks on face detectors using neural net based constrained optimization.  In _IEEE MMSP_.
  * Buolamwini (2018) Buolamwini, J. 2018.  When the Robot Doesn’t See Dark Skin.  https://www.nytimes.com/2018/06/21/opinion/facial-analysis-technology-bias.html.
  * Buolamwini and Gebru (2018) Buolamwini, J.; and Gebru, T. 2018.  Gender shades: Intersectional accuracy disparities in commercial gender classification.  In _PMLR FAT*_.
  * Chandrasekaran et al. (2020) Chandrasekaran, V.; Gao, C.; Tang, B.; Fawaz, K.; Jha, S.; and Banerjee, S. 2020\.  Face-Off: Adversarial Face Obfuscation.
  * Chen et al. (2021) Chen, Z.; Xie, L.; Pang, S.; He, Y.; and Tian, Q. 2021.  Appending adversarial frames for universal video attack.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_.
  * Dash et al. (2021) Dash, A.; Chakraborty, A.; Ghosh, S.; Mukherjee, A.; and Gummadi, K. P. 2021.  When the Umpire is also a Player: Bias in Private Label Product Recommendations on E-commerce Marketplaces.  In _ACM FAccT_.
  * Dash, Mukherjee, and Ghosh (2019) Dash, A.; Mukherjee, A.; and Ghosh, S. 2019.  A network-centric framework for auditing recommendation systems.  In _IEEE INFOCOM_.
  * Duan et al. (2020) Duan, R.; Ma, X.; Wang, Y.; Bailey, J.; Qin, A. K.; and Yang, Y. 2020.  Adversarial Camouflage: Hiding Physical-World Attacks With Natural Styles.  In _IEEE/CVF CVPR_.
  * equalAIs (2021) equalAIs. 2021.  “equalAIs Empowering Humans by Subverting Machines” Accessed April, 2021\.  URL https://equalais.media.mit.edu/.
  * Face++ (2021) Face++. 2021.  Face Attributes.  https://www.faceplusplus.com/attributes/.
  * Face++ (2021) Face++. 2021.  “Face++ Detect.” Accessed April, 2021.  URL https://www.faceplusplus.com/face-detection/.
  * Garofalo et al. (2018) Garofalo, G.; Rimmer, V.; Preuveneers, D.; Joosen, W.; et al. 2018.  Fishy faces: Crafting adversarial images to poison face authentication.  In _$\\{$ USENIX$\\}$ ($\\{$WOOT$\\}$ 18)_.
  * GIMP (2021) GIMP. 2021.  “GIMP:GNU Image Manipulation Program”.  URL https://www.gimp.org/about/.
  * Goel et al. (2018) Goel, A.; Singh, A.; Agarwal, A.; Vatsa, M.; and Singh, R. 2018.  Smartbox: Benchmarking adversarial detection and mitigation algorithms for face recognition.  In _IEEE BTAS_.
  * Goodfellow, Shlens, and Szegedy (2014) Goodfellow, I. J.; Shlens, J.; and Szegedy, C. 2014.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ .
  * Jay Peters (2020) Jay Peters. 2020.  “IBM will no longer offer, develop, or research facial recognition technology” Accessed April, 2021.  URL https://www.theverge.com/2020/6/8/21284683/ibm-no-longer-general-purpose-facial-recognition-analysis-software.
  * Jiang et al. (2019) Jiang, L.; Ma, X.; Chen, S.; Bailey, J.; and Jiang, Y.-G. 2019.  Black-box adversarial attacks on video recognition models.  In _Proceedings of the 27th ACM International Conference on Multimedia_.
  * Jung et al. (2018) Jung, S.; An, J.; Kwak, H.; Salminen, J.; and Jansen, B. 2018.  Assessing the Accuracy of Four Popular Face Recognition Tools for Inferring Gender, Age, and Race .
  * Karkkainen and Joo (2021) Karkkainen, K.; and Joo, J. 2021.  FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation.  In _IEEE/CVF WACV_.
  * Kyriakou et al. (2019a) Kyriakou, K.; Barlas, P.; Kleanthous, S.; and Otterbacher, J. 2019a.  Fairness in Proprietary Image Tagging Algorithms: A Cross-Platform Audit on People Images.  _AAAI ICWSM_ .
  * Kyriakou et al. (2019b) Kyriakou, K.; Barlas, P.; Kleanthous, S.; and Otterbacher, J. 2019b.  Fairness in Proprietary Image Tagging Algorithms: A Cross-Platform Audit on People Images .
  * Lakshmi et al. (2021) Lakshmi, A.; Wittenbrink, B.; Correll, J.; and Ma, D. S. 2021.  The India Face Set: International and Cultural Boundaries Impact Face Impressions and Perceptions of Category Membership.  _Frontiers in psychology_ .
  * Ma, Correll, and Wittenbrink (2015) Ma, D. S.; Correll, J.; and Wittenbrink, B. 2015.  The Chicago face database: A free stimulus set of faces and norming data.  _Behavior research methods_ .
  * Ma, Kantner, and Wittenbrink (2020) Ma, D. S.; Kantner, J.; and Wittenbrink, B. 2020.  Chicago Face Database: Multiracial expansion.  _Behavior Research Methods_ .
  * Maesumi et al. (2021) Maesumi, A.; Zhu, M.; Wang, Y.; Chen, T.; Wang, Z.; and Bajaj, C. 2021.  Learning Transferable 3D Adversarial Cloaks for Deep Trained Detectors.  _arXiv preprint arXiv:2104.11101_ .
  * Manish Singh (2020) Manish Singh. 2020.  “India used facial recognition tech to identify 1,100 individuals at a recent riot” Accessed April, 2021.  URL https://techcrunch.com/2020/03/11/india-used-facial-recognition-tech-to-identify-1100-individuals-at-a-recent-riot/.
  * Massoli et al. (2021) Massoli, F. V.; Carrara, F.; Amato, G.; and Falchi, F. 2021.  Detection of face recognition adversarial attacks.  _Computer Vision and Image Understanding_ 202: 103103.
  * Mehrabi et al. (2019) Mehrabi, N.; Morstatter, F.; Saxena, N.; Lerman, K.; and Galstyan, A. 2019.  A survey on bias and fairness in machine learning.  _arXiv preprint arXiv:1908.09635_ .
  * Messias, Vikatos, and Benevenuto (2017) Messias, J.; Vikatos, P.; and Benevenuto, F. 2017.  White, Man, and Highly Followed: Gender and Race Inequalities in Twitter.  In _Proceedings of the International Conference on Web Intelligence_.
  * Microsoft (2021) Microsoft. 2021.  “Microsoft Azure Face.” Accessed April, 2021.  URL https://azure.microsoft.com/en-in/services/cognitive-services/face/.
  * Microsoft (2021) Microsoft. 2021.  What is the Azure Face service?  https://docs.microsoft.com/en-us/azure/cognitive-services/face/overview.
  * Mitchell et al. (2019) Mitchell, M.; Wu, S.; Zaldivar, A.; Barnes, P.; Vasserman, L.; Hutchinson, B.; Spitzer, E.; Raji, I. D.; and Gebru, T. 2019.  Model cards for model reporting.  In _ACM FAT*_.
  * Nagpal et al. (2019) Nagpal, S.; Singh, M.; Singh, R.; and Vatsa, M. 2019.  Deep learning for face recognition: Pride or prejudiced?  _arXiv preprint arXiv:1904.01219_ .
  * Natasha Lomas (2020) Natasha Lomas. 2020.  “London’s Met Police switches on live facial recognition, flying in face of human rights concerns” Accessed April, 2021.  URL https://techcrunch.com/2020/01/24/londons-met-police-switches-on-live-facial-recognition-flying-in-face-of-human-rights-concerns/.
  * Noble (2018) Noble, S. U. 2018.  _Algorithms of oppression: How search engines reinforce racism_.  nyu Press.
  * Oh, Fritz, and Schiele (2017) Oh, S. J.; Fritz, M.; and Schiele, B. 2017.  Adversarial image perturbation for privacy protection a game theory perspective.  In _IEEE ICCV_.
  * O’neil (2016) O’neil, C. 2016.  _Weapons of math destruction: How big data increases inequality and threatens democracy_.  Crown.
  * Pang et al. (2015) Pang, R.; Baretto, A.; Kautz, H.; and Luo, J. 2015.  Monitoring adolescent alcohol use via multimodal analysis in social multimedia.  In _IEEE Big Data_.
  * Qiu et al. (2020) Qiu, H.; Xiao, C.; Yang, L.; Yan, X.; Lee, H.; and Li, B. 2020.  Semanticadv: Generating adversarial examples via attribute-conditioned image editing.  In _ECCV_.
  * Raji and Buolamwini (2019) Raji, I. D.; and Buolamwini, J. 2019.  Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products.  In _AAAI/ACM AIES_.
  * Raji et al. (2020) Raji, I. D.; Gebru, T.; Mitchell, M.; Buolamwini, J.; Lee, J.; and Denton, E. 2020\.  Saving face: Investigating the ethical concerns of facial recognition auditing.  In _AAAI/ACM AIES_.
  * Sandvig et al. (2014) Sandvig, C.; Hamilton, K.; Karahalios, K.; and Langbort, C. 2014.  Auditing algorithms: Research methods for detecting discrimination on internet platforms.  _Data and discrimination: converting critical concerns into productive inquiry_ .
  * Sangomla (2020) Sangomla, A. 2020.  Big Brother is watching you; actually your face.  https://www.downtoearth.org.in/news/science-technology/big-brother-is-watching-you-actually-your-face-74181.
  * Sixta et al. (2020) Sixta, T.; Junior, J. C. J.; Buch-Cardona, P.; Vazquez, E.; and Escalera, S. 2020\.  Fairface challenge at ECCV 2020: analyzing bias in face recognition.  In _ECCV_.
  * Snow (2018) Snow, J. 2018.  Amazon’s Face Recognition Falsely Matched 28 Members of Congress With Mugshots.  https://www.aclu.org/blog/privacy-technology/surveillance-technologies/amazons-face-recognition-falsely-matched-28.
  * STOP (2021) STOP. 2021.  “Surveillance Technology Oversight Project” Accessed April, 2021.  URL https://www.stopspying.org/.
  * Tate Ryan-Mosley (2021) Tate Ryan-Mosley. 2021.  “I asked an AI to tell me how beautiful I am” Accessed April, 2021.  URL https://www.technologyreview.com/2021/03/05/1020133/ai-algorithm-rate-beauty-score-attractive-face/.
  * Vakhshiteh, Nickabadi, and Ramachandra (2020) Vakhshiteh, F.; Nickabadi, A.; and Ramachandra, R. 2020.  Adversarial Attacks against Face Recognition: A Comprehensive Study.  _arXiv preprint arXiv:2007.11709_ .
  * Vikatos et al. (2017) Vikatos, P.; Messias, J.; Miranda, M.; and Benevenuto, F. 2017.  Linguistic Diversities of Demographic Groups in Twitter.  In _ACM HT_.
  * Xiao et al. (2019) Xiao, C.; Yang, D.; Li, B.; Deng, J.; and Liu, M. 2019.  Meshadv: Adversarial meshes for visual recognition.  In _IEEE/CVF CVPR_.
  * Xu et al. (2020) Xu, X.; Chen, J.; Xiao, J.; Gao, L.; Shen, F.; and Shen, H. T. 2020.  What Machines See Is Not What They Get: Fooling Scene Text Recognition Models With Adversarial Text Images.  In _IEEE/CVF CVPR_.

## Appendix

We evaluate the performance of the three commercial FRS on both CFD-MR and
CFD-India datasets. The results are discussed in Figures 5 and 6.

#### Effect of perturbations in the performance

(a) Accuracy on AWS

(b) Accuracy on Face++

(c) Accuracy on Microsoft

Figure 5: Performance on CFD-MR for Original, RGB-Filter and Newsprint-filter
inputs

(a) Accuracy on AWS

(b) Accuracy on Face++

(c) Accuracy on Microsoft

Figure 6: Performance on CFD-India for Original, RGB-Filter and Newsprint-
filter inputs

The prediction accuracy results for the tasks of Gender and Age detection are
presented here. The results are divided per FRS API in Figures 5(a), 5(b) and
5(c) for CFD-MR and in Figures 6(a), 6(b) and 6(c) for CFD-India.

Gender detection: We observe that all the FRS APIs report high accuracy for
gender detection on the original set of images for both the datasets ($\geq
90\%$), with Face++ reporting comparatively lowest accuracy values– $91\%$ for
CFD-MR (Fig.5(b)) and $93\%$ for CFD-India (Fig.6(b)). As regards to addition
of different noises, we do not observe significant degradation in performance
for RGB noise, with the accuracy for CFD-India increasing by $5\%$. However,
for the newsprint noise, we observe significant deterioration of prediction
accuracy in Amazon and Face++ systems with a drop of $\geq 40\%$ and Face++
reporting an accuracy of $\approx 20\%$ for CFD-MR (see Figures 5(a), 5(b),
6(a) and 6(b)).

Age prediction: The accuracy for this task is low for all FRSs, independent of
the input. Face++ is the worst performing system here, similar to its
performance on CFD, with the accuracy nearly $15\%$ and $13\%$ for RGB noise
on CFD-MR and CFD-India respectively (see Figure 5(b) and 6(b)). Here,
Microsoft returns the highest accuracy with its reported accuracy in the range
of $\approx 76\%$ to $\approx 80\%$ for CFD-MR (see Figure 5(c)) and $\approx
54\%$ to $\approx 67\%$ for CFD-India (see Figure 6(c))

The key takeaways from Figures 5 and 6 are as follows.

  * •

All APIs perform well for the task of gender detection for all inputs except
newsprint, while the performance for age prediction is bad for all inputs on
AWS and Face++.

  * •

Face++ is the worst performing FRS for both gender and age prediction.

  * •

Microsoft is the most robust FRS among the three.