# Similarity-based Gray-box Adversarial Attack Against Deep Face Recognition

Hanrui Wang1, Shuo Wang2, Zhe Jin1, Yandan Wang3, Cunjian Chen4,5, Massimo
Tistarelli6
1 School of Information Technology, Monash University Malaysia, Malaysia 2
Data61, CSIRO, Australia 3 Faculty of Computer Science and Technology, Wenzhou
University, China 4 Monash Suzhou Research Institute, China 5 Department of
Data Science and AI, Monash University, Australia 6 The University of Sassari,
Italy This work was supported by grants from Ministry of Higher Education
(MOHE) Malaysia through Fundamental Research Grant Scheme
(FRGS/1/2018/ICT02/MUSM/03/3).

###### Abstract

The majority of adversarial attack techniques perform well against deep face
recognition when the full knowledge of the system is revealed (_white-box_).
However, such techniques act unsuccessfully in the gray-box setting where the
face templates are unknown to the attackers. In this work, we propose a
similarity-based gray-box adversarial attack (SGADV) technique with a newly
developed objective function. SGADV utilizes the dissimilarity score to
produce the optimized adversarial example, i.e., similarity-based adversarial
attack. This technique applies to both white-box and gray-box attacks against
authentication systems that determine genuine or imposter users using the
dissimilarity score. To validate the effectiveness of SGADV, we conduct
extensive experiments on face datasets of LFW, CelebA, and CelebA-HQ against
deep face recognition models of FaceNet and InsightFace in both white-box and
gray-box settings. The results suggest that the proposed method significantly
outperforms the existing adversarial attack techniques in the gray-box
setting. We hence summarize that the similarity-base approaches to develop the
adversarial example could satisfactorily cater to the gray-box attack
scenarios for de-authentication.

††publicationid: pubid: 978-1-6654-3176-7/21/$31.00 ©2021 IEEE

##  I Introduction

Deep-learning-based approaches, e.g., the deep convolutional neural networks
(DCNNs), have become a de-facto standard for face recognition (FR) due to
their superior accuracy. However, recent studies reveal that DCNNs are
vulnerable to adversarial attacks. More specifically, the adversarial attack
refers to a scheme used to fool the DCNNs in decision-making by supplying
deceptive input known as an adversarial example [1]. In principle, the
adversarial attack exploits the gradient or statistical information to
generate adversarial examples in order to expose the vulnerability of DCNN-
based FR [2].

Among various types of adversarial attacks in literature, our proposed
technique is characterized as a targeted test-time evasion attack in either
white-box or gray-box scenarios:

  * •

Targeted attack: The goal of a targeted attack is to slightly perturb the
source image such that the generated adversarial example will be classified as
a target label. [3, 4, 5, 6, 7, 8]. It is even worse for the authentication
system since the adversarial example can fool the system to gain illegal
access.

  * •

White-box attack and gray-box attack: The white box attack requires full
knowledge of the system, including the model, data, and even the defender
against threats [4, 5, 6, 7, 8, 9, 1]. A machine learning model should be
secure from white-box attacks, which is the desired property [10]. On the
other hand, the gray box attack merely requires partial knowledge instead of
full knowledge [11, 12], assuming the database is not compromised in our
context. Gray-box adversarial attack is believed to be more practical for
attackers.

  * •

Test-time evasion: It aims to generate the adversarial example with slight
perturbation to evade human inspection but fool the classifier [4, 5, 6, 7, 8,
9, 13].

Source image |  | Difference |  | Adversarial example
---|---|---|---|---
| + |  | = |
|  |  |  |  |
Target image |  |  |  | White box | Gray box
Figure 1: Our proposed SGADV for the white-box attack and gray-box attack. The
adversarial example is produced by slightly perturbing the source image based
on the target image. In the white-box setting, the adversarial example is used
to attack the target image and achieves 100% attack success rate. In the gray-
box setting, the adversarial example attacks another image from the same
target user and achieves a 98.7% attack success rate. The result in the gray-
box setting significantly outperforms that of the label-based adversarial
attacks, which gain only 17.28% attack success rate.

Most existing adversarial attack techniques [3, 4, 5, 6, 7, 8, 9, 1, 13] are
designed to attack classification systems, whose output is the predicted
label. We define these techniques as label-based adversarial attacks as they
rely on label information. However, in biometric authentication systems, e.g.,
FR authentication systems, no label information of the target users are
stored. Hence, existing label-based adversarial attacks could not directly
apply to FR-based authentication systems. Therefore, a conditional binary
cross-entropy (C-BCE) objective function is proposed for the label-based
adversarial attack to produce a binary output (i.e., 0/1 or false/true),
resembling the binary outputs (i.e., rejection or acceptance) of biometric
authentication. Although C-BCE enables binary outputs, the label-based
adversarial attack shows the feasibility only when full knowledge of the
system (white box) is known. Without compromising templates in the database
(gray box), the attack success rate significantly decreases. The theoretical
proof is given in Section III-B.

To fill the gap in the gray-box setting, we propose the similarity-based gray-
box adversarial attack (SGADV), as indicated in Fig. 1. The _similarity_ in
our context refers to the (dis)similarity score that is used to optimize the
adversary to produce an adversarial example that is closest to the target
image in feature space. SGADV can illegally fool the FR-based authentication
in either white-box or gray-box setting, which indicates generating an
adversarial example by a single target image when:

  * •

Scenario 1 (S1) - White box: assuming the identical target image is enrolled
in the database.

  * •

Scenario 2 (S2) - Gray box: assuming the same target user enrolls with any
image apart from this target image.

To cater to the two attack scenarios mentioned above, a novel objective
function is proposed in this paper. To further show the generalization of our
proposed SGADV, we conduct experiments using three face datasets containing
images of various qualities and two deep FR models. The results are compared
to four benchmarking label-based adversarial attack techniques implemented
with our proposed C-BCE objective function. It shows that SGADV effectively
breaks FR-based authentication in both white-box and gray-box settings and
significantly outperforms label-based techniques in the gray-box setting (see
Table III and IV). Our code is released at
https://github.com/azrealwang/SGADV.

The contributions of this paper are summarised as follows:

  * •

To address the limitation of existing adversarial attacks against FR
authentication system, we propose a novel adversarial attack technique, namely
SGADV, that successfully breaks the FR-based authentication in the setting of
both white box (100% attack success rate) and gray box (up to 98.74% attack
success rate).

  * •

We develop a new objective function using (dis)similarity score to increase
the attack performance in the gray-box setting.

  * •

We propose a C-BCE objective function to implement the benchmarking label-
based adversarial attacks as the baseline against the FR-based authentication,
serving as a new tool for the adversarial machine learning research community.

  * •

We experimentally demonstrate that our SGADV defeats four benchmarking label-
based adversarial attack techniques (minimum 78.08% attack success rate
increase). We further show the generalization of SGADV on two deep FR models
and three datasets.

##  II Backgrounds

###  II-A Adversarial attack

A variety of adversarial attack techniques has been proposed to attack
classification systems. Some well-known methods are served as benchmarks for
performance comparison, e.g., the fast gradient sign method (FGSM) [4],
projected gradient descent (PGD) [5], DeepFool [6] and Carlini & Wagner attack
(CW) [7, 8]. The first adversarial attack on the MNIST dataset [14] was
proposed by Biggio et al., [13] in 2013 to mislead the classifiers, e.g.,
Support Vector Machine (SVM). In the same year, Szegedy et al. successfully
generated adversarial examples to fool the deep-neural-networks-based
classifiers [9]. These pioneering works light the security concerns of deep
learning models. Thereafter, FGSM was proposed by Goodfellow et al. to produce
adversarial examples instantly. It realizes the demands of generating a large
number of adversarial examples for training and verification. In 2016,
DeepFool was proposed by Moosavi-Dezfooli et al. This technique introduces
slight perturbation to the image to attack deep learning models. It reveals
that the deep learning models are not robust even with slight perturbation.
Then, in 2017, PGD was proposed as an iterative version of the one-step attack
FGSM to find the most adversarial examples [5] with the acceptable efficiency.
In addition, CW [7] is deemed as one of the most aggressive adversarial
attack, which was the first technique designed to counterattack the defences
of [9] and [4]. It experimentally demonstrated the feasibility against ten
adversarial example detection methods [8]. Eventually, CW becomes the
benchmark for evaluating the security of deep learning models and comparison
with other techniques, though it is computationally exhausted.

###  II-B Adversarial attack on face recognition

Adversarial attack techniques addressed in Section II-A are popularly
implemented in practice. However, none of them are designed for FR. In this
section, several recently proposed white-box and gray-box adversarial attacks
on FR are reviewed.

The first gradient-based adversarial attack on FR was proposed by Sharif et
al. [15]. Sharif et al. dodged FR systems and impersonated other identities
wearing an eyeglass frame with a malicious texture. It also demonstrated that
the vulnerability of DCNNs could be physically exploited. Rozsa et al.
proposed the first adversarial attack against deep features on FR [16]. It
forms adversarial examples which mimic the deep features of the target image
by iterative layerwise learning. Dabouei et al. proposed an efficient
algorithm [17] directly manipulating landmarks of the face images to produce
geometrically-perturbed adversarial examples. Yang et al. [18] proposed a GAN-
based adversarial attack in the gray-box setting, namely $A^{3}GN$. However,
$A^{3}GN$ requires at least five target images from each target user for
training and inference. Deb et al. [12] proposed another GAN-based method,
namely AdvFaces. Despite the GAN is trained in the white-box setting, AdvFaces
performs the black-box attack, which cannot compete our gray-box adversarial
model.

###  II-C Projected gradient descent

PGD is regarded as a descendant of FGSM [4]. Unlike the one-step attack FGSM,
PGD iteratively learns and seeks the most optimized adversarial examples. In
this paper, PGD [5] is adopted but supervised by our proposed objective
function (Section III-C) to learn the adversarial examples. Note that the
proposed adversarial attack terminates at the stage of convergence, while PGD
is stopped only when it reaches the pre-defined maximum steps. The adversarial
attack process of PGD is given as follows.

Let $\theta$ denote the parameters of a DCNN, $x$ be the input to the DCNN,
$y$ be the label of $x$ and $J(\theta,x,y)$ be the loss used to train the
network. Then, let $x^{\prime}$ be the adversarial example of source $x$,
$y^{\prime}$ be the label of $x^{\prime}$ and $y_{1}$ be the target label.
Since PGD is an iterative version of FGSM, PGD and FGSM share the identical
adversarial attack process shown in eq (1). Thus, the $L_{\infty}$-bounded
FGSM is to learn $x^{\prime}$ that

$x^{\prime}=x+\epsilon\cdot sign(\nabla_{x}J(\theta,x,y)),\ s.t.\
||x^{\prime}-x||_{\infty}<\epsilon,$ (1)

where satisfying $y^{\prime}\neq y$ (indiscriminate attack) or
$y^{\prime}=y_{1}$ (targeted attack). $\epsilon$ is the size of the
perturbation and expects to be small enough to evade human inspection.
Inherent from FGSM, the $L_{\infty}$-bounded PGD is utilized by

$\displaystyle x^{t+1}=Clip_{x,\epsilon}\\{x^{t}+\alpha\cdot
sign(\nabla_{x^{t}}J(\theta,x^{t},y))\\},$ (2) $\displaystyle s.t.\
||x^{\prime}-x||_{\infty}<\epsilon.$

$Clip(\cdot)$ denotes the function to perturb the image at each iteration step
with a step size $\alpha$. The maximum steps $t_{max}$ shall satisfy
$t_{max}\geq\frac{\epsilon}{\alpha}$, to guarantee that the perturbation can
reach the border. Specifically, $x^{0}$ is initialized with randomly chosen
starting points.

##  III Similarity-based gray-box adversarial attack

This section demonstrates our proposed technique. Section III-A introduces the
adversarial model of the proposed SGADV. Section III-B theoretically analyzes
the gap between similarity-based and label-based adversarial attacks. Section
III-C proposes a novel objective function, and the algorithm of SGADV is
presented in Section III-D.

###  III-A Adversarial model

The FR-based authentication system generally consists of two key components,
i.e., the deep FR model (e.g., DCNN) for feature embedding and the database
for template storing. Our proposed SGADV applies to both white-box and gray-
box scenarios:

  * •

White box: DCNN and database are both compromised.

  * •

Gray box: The adversary is conducted in the white-box setting with respect to
the DCNN but in the black-box setting with respect to the database (a face
image of the target user is still required).

The white box is the baseline scenario of adversarial attacks. The systems,
particularly which require high confidentiality, have to be secure in the
worst case (white-box, or even strong white-box [8]), since flaws of security
are not acceptable even if an extremely small probability, which is agreed by
Tramèr et al. [10] that security against white-box attacks is the desirable
property of machine learning models. Miller et al. [19] also debated that a
reasonable adversarial attack for evaluation should be in the white box to the
system and the black box to the defense. Carlini and Wagner [8] even asserted
(without elaboration) the strong white-box requirement of a defense that the
detection cannot be evaded even if it is known by the attacker.

However, people criticize that the white-box setting is unlikely practical as
it requests full knowledge of the system [12, 20, 21]. Thus, our gray-box
(without knowing the database) adversarial attack can alleviate the above
criticism. Specifically, pre-trained models for the well-known deep FR methods
are publicly available, i.e., the true model is possibly the white box to
attackers. If the true model is unavailable, a surrogate model can be learned
by the black-box adversarial attack techniques [19]. When either the true or
surrogate model is available, our gray-box adversarial attack is viable since
it merely requires the knowledge of the DCNN.

Figure 2: Comparison between label-based adversarial attack and proposed SGADV
dealing with the gray-box setting. ”Adv-L” and ”Adv-S” represent the
adversarial examples from the same target image by the label-based technique
and our proposed similarity-based SGADV, respectively. The circle refers to
the space where the face images are identified as the same user. The target
image is from the same user with the enrolled image. As observed, both ”Adv-L”
and ”Adv-S” are successfully categorized as the target user with the target
image. However, only ”Adv-S” is viable to attack the enrolled image.

###  III-B Challenge of adversarial attack on deep FR

Although the FR-based authentication is converted to a binary classification
task to implement the label-based adversarial attack, the label-based
adversarial attack performs poorly in the gray-box setting. In contrast, our
proposed similarity-based SGADV is feasible for both white-box and gray-box
settings, which is theoretically analyzed as follows.

Let $X$ be the source face image of the adversary, which will be perturbed to
produce the adversarial example $X^{adv}$. Let $X_{1}^{1}$ represent the
enrolled image of an authorized user $X_{1}$ (target user). Thus, $X_{1}^{1}$
is stored in the database and unknown to the adversary in the gray-box
setting. Let $X_{1}^{2},X_{1}^{3},\ldots$ be other face images of the same
identity (target images), which are easily collected in the real world, e.g.,
from the Internet. Let $f\left(\cdot\right)$ denote the function of feature
embedding for deep FR, whose outputs are feature vectors (face templates) for
comparison. Let $||\cdot||\in[0,1]$ donate the normalized distance, which
equals the dissimilarity ranged from 0 to 1 (lower indicates more similarity).
Thus, in FR-based authentication systems, a query face $X_{1}^{2}$ is
recognized as $X_{1}$ when

$||f(X_{1}^{2})-f(X_{1}^{1})||\leq\tau,$ (3)

where $\tau$ is a pre-defined dissimilarity threshold.

Fig. 2 illustrates the challenge of using the label-based adversarial attack
in the gray-box setting, while our proposed SGADV is significantly enhanced.
To differentiate from the proposed SGADV, the adversarial example produced by
the label-based adversarial attack is represented by $X^{adv}_{L}$. It is
regarded as a successful attack (i.e., misclassification) when the
dissimilarity between the adversarial example $X^{adv}_{L}$ and target image
$X_{1}^{2}$ in feature space is less than the threshold. The adversary then
can be aborted as shown in Fig. 2 of ”Adv-L”, which satisfies

$||f(X^{adv}_{L})-f(X_{1}^{2})||\leq\tau.$ (4)

However, the dissimilarity may dramatically increase when the adversarial
example $X^{adv}_{L}$ is compared to the enrolled image $X_{1}^{1}$.
Formulating with (3) and (4), we get

$||f(X^{adv}_{L})-f(X_{1}^{1})||\leq 2\tau.$ (5)

As aforementioned, the successful attack requires the dissimilarity to be less
than the threshold. Thus, computed from (5), the probability of success of
label-based adversarial attack in the gray-box setting $p_{L}^{success}$
should be

$p_{L}^{success}\approx\frac{\tau}{2\tau}=\frac{1}{2},$ (6)

assuming $||f(X^{adv}_{L})-f(X_{1}^{1})||$ follows uniform distribution
$||f(X^{adv}_{L})-f(X_{1}^{1})||\sim U(0,2\tau)$.

In contrast, the proposed SGADV elaborates dissimilarity in the objective
function to find the closest adversarial example (as shown in Fig. 2 of
”Adv-S”). The adversarial example $X^{adv}$ satisfies

$||f(X^{adv})-f(X_{1}^{2})||=\ell,$ (7)

where $\ell$ indicates a small error when the adversary is terminated and
$\ell\ll\tau$. Thus, when comparing this adversarial example $X^{adv}$ to the
enrolled image $X_{1}^{1}$, the threat of $X^{adv}$ preserves in a large
probability as shown in Fig. 2, as

$||f(X^{adv})-f(X_{1}^{1})||\leq\ell+\tau.$ (8)

In this case, the probability of success of the similarity-based adversarial
attack in the gray-box setting $p^{success}$ is formulated as

$p^{success}\approx\frac{\tau}{\ell+\tau}\gg
p_{L}^{success}\approx\frac{1}{2},$ (9)

assuming $||f(X^{adv})-f(X_{1}^{1})||$ follows uniform distribution
$||f(X^{adv})-f(X_{1}^{1})||\sim U(0,\ell+\tau)$.

Interpreted from (9), the proposed similarity-based SGADV is more likely to
break the FR-based authentication than the label-based adversarial attack in
the gray-box setting. However, the probability $p^{success}$ is still slightly
lower than 100%. Thus, the attack performance in the gray-box setting is
expected to degrade, comparing to the white box (experimentally demonstrated
in Section IV-B).

###  III-C Objective function

The objective function of SGADV is formulated using dissimilarity.
Specifically, define $X^{adv}$ as $X$ with perturbation $\delta$

$X^{adv}=X+\delta,\ s.t.\ ||\delta||_{\infty}<\epsilon,$ (10)

where $\epsilon$ is the pre-defined size of the perturbation and small enough
to evade human inspection. In FR-based authentication systems, $X^{adv}$ is
recognized as $X_{1}$ when

$||f(X^{adv})-f(X_{1})||\leq\tau,$ (11)

where $\tau$ is a pre-defined dissimilarity threshold. Thus, the proposed
similarity-based attack aims to find

${\arg\min}_{X^{adv}}||f(X^{adv})-f(X_{1})||.$ (12)

To achieve this goal, the objective function of SGADV is formulated as

$J_{SG}(X^{adv},X_{1})=||f(X^{adv})-f(X_{1})||.$ (13)

Input: Target: Face image $X_{1}$, DCNN $f(\cdot)$; Source: Face image $X$;
Settings: size of the perturbation $\epsilon$, step size $\alpha$, maximum
steps $t_{max}$, convergence threshold $\tau_{conv}$

Output: Adversarial example $X^{adv}$

1 begin Initialization

2 $\delta^{0}\sim U(-\epsilon,\epsilon)$

3 $X^{0}:=X+\delta^{0}$

4

5repeat

6 $J_{SG}(X^{t},X_{1})=||f(X^{t})-f(X_{1})||$

7 $X^{t+1}=Clip_{X,\epsilon}\\{X^{t}+\alpha\cdot
sign(\nabla_{X^{t}}J_{SG})\\}$

8until  _Satisfy ( 15) or (16) or $t=t_{max}$_

return _$X^{adv}:=X^{t_{stop}}$_

Algorithm 1 SGADV TABLE I: Settings of deep FR and datasets Deep FR | DCNN | Loss function | Pre-trained data | System Setup | Source
---|---|---|---|---|---
11 |  |  |  |  |  |  |  |
|  |  | Dataset | Image size | Dataset | Image size | Threshold $\tau$ | EER(%) | Dataset | Image size
FaceNet [22] | Inception ResNet v1 | Triplet loss | VGGFace2 | $160\times 160$ | CelebA [23] | $160\times 160$ | 0.2967 | 1.20 | LFW [24] | $250\times 250$
|  |  |  |  |  |  |
9 |  |  |  |  |  |  |  |  |  |
InsightFace [25] | LResNet100 | ArcFace loss | MS1M-Arcface | $112\times 112$ |  | $112\times 112$ | 0.4146 | 6.23 |  |

###  III-D Algorithm of SGADV

The algorithm of SGADV is derived from (2) of PGD [5], supervised by our
proposed objective function in Section III-C, illustrated in Algorithm 1. It
consists of a random initialization (Step 1-1), and an iterative optimization
(Step 1-1) by (13) and (2). The optimization will be aborted when it satisfies
any of three stop criteria (Step 1), which are

  * •

achieving maximum steps, i.e., $t=t_{max}$.

  * •

achieving convergence, i.e., the change of loss value is slight. To evaluate
the convergence or divergence, the variation of loss value $\Delta$ between
continuous steps are first recorded as

$\displaystyle\Delta J^{t+1}=J(X^{t},*)-J(X^{t+1},*),$ (14) $\displaystyle
S=\\{\Delta J^{t},\Delta J^{t+1},\Delta J^{t+2},\Delta J^{t+3},\Delta
J^{t+4}\\},$

where $S$ is a set of $\Delta$ with latest five steps. Then, this stop
criterion is defined as

$\forall\Delta\in S,|\Delta|\leq\tau_{conv},$ (15)

where $\tau_{conv}$ is the pre-defined convergence threshold.

  * •

achieving settlement within an error range around a value. In this case, the
loss value increases and decreases regularly. Thus, the stop criterion is
defined as

$\exists\Delta^{*},\Delta^{**}\in S,\Delta^{*}\leq 0\wedge\Delta^{**}\leq 0$
(16)

Ultimately, the perturbed example at the stage satisfying stop criteria is
output as an adversarial example $X^{adv}$ (Step 1).

##  IV Experiments

###  IV-A Experimental settings

####  IV-A1 Deep face recognition models

To evaluate the generalization of our SGADV on distinct DCNNs, we conduct
experiments using two popular deep FR models, i.e., FaceNet [22] and
InsightFace [25]. Face images are converted to 512-dimensional templates after
feature embedding. The settings of the two models are listed in Table I.

####  IV-A2 Datasets

To evaluate the generalization of SGADV on various data, e.g., attacking
target users of dataset A using source images of dataset B, three face
datasets, i.e., Labeled Faces in the Wild (LFW) [24], CelebA [23] and CelebA-
HQ [26], are adopted for experiments. Apart from the study on image size, the
subjects in CelebA are selected as target users and the images in LFW are
perturbed as source images. The settings are listed in Table I. Particularly
for the study on the impact of image size, a high-quality version of CelebA
dataset (size of $1024\times 1024$), namely CelebA-HQ [26], is adopted as the
dataset of source images. In contrast, the subjects in the LFW dataset are
selected as target users. The source images are then resized to various
qualities from $1000\times 1000$ to $112\times 112$. The experimental results
on various data regarding the generalization of proposed SGADV are presented
in Section IV-B.

####  IV-A3 Benchmarking label-based adversarial attacks

To evaluate the attack performance, the proposed SGADV is compared against
several label-based adversarial attack techniques, as shown in Table II. The
details of the comparison are presented in Section IV-B. The cross-entropy
objective function used in original label-based techniques does not apply to
FR due to the lack of label information. Instead, we propose the C-BCE
objective function for those label-based techniques to replace the cross-
entropy objective function. To make the function applicable, FR-based
authentication is converted to a task of binary classification as

$y=\begin{cases}0,&rejection\\\ 1,&acceptance\end{cases}.$ (17)

The label of the adversarial example $X^{adv}$ is defined as

$y^{adv}=\begin{cases}0,&||f(X^{adv})-f(X_{1})||>\tau\\\
1,&||f(X^{adv})-f(X_{1})||\leq\tau\end{cases},$ (18)

where $\tau$ is the threshold. Then, the binary cross-entropy objective
function is defined as

$H(p,y)=-y\log p-(1-y)\log(1-p),$ (19)

where $y\in\\{0,1\\}$ represents the expected label and $p$ represents the
probability of $y=1$. Specifically in a targeted adversarial attack, $y\equiv
1$. Meanwhile, based on (18), the probability of $y^{adv}=1$ is computed as

$\displaystyle p_{y^{adv}=1}=$ (20)
$\displaystyle\begin{cases}\frac{1-||f(X^{adv})-f(X_{1})||}{1-\tau},&||f(X^{adv})-f(X_{1})||>\tau\\\
1,&||f(X^{adv})-f(X_{1})||\leq\tau\end{cases}.$

Finally, the proposed C-BCE objective function is reformulated by combining
(18), (19) and (20) as

$\displaystyle J_{C-BCE}(X^{adv},X_{1})=$ (21)
$\displaystyle-\log\frac{\min\\{(1-||f(X^{adv})-f(X_{1})||),(1-\tau)\\}}{1-\tau}.$

TABLE II: Settings of adversarial attacks Technique | Settings
---|---
FGSM [4] | $\epsilon=0.03$
DeepFool [6] |  $\epsilon=0.03$, $t_{max}=40$, overshoot $\eta=0.02$
CW [7] |  |  $\epsilon=0.03$, $\alpha=0.001$, $t_{max}=1000$,
---
binary search iterations = 20
PGD [5] |  $\epsilon=0.03$, $\alpha=0.001$, $t_{max}=40$
|
SGADV |  $\epsilon=0.03$, $\alpha=0.001$, $t_{max}=1000$, $\tau_{conv}=0.0001$

####  IV-A4 Evaluation metrics

  * •

Equal error rate (EER) and threshold are computed as the system setup. The
attack success rate is expected largely greater than EER, and dissimilarity is
no more than the threshold.

  * •

False positive rate (FPR) and true positive rate (TPR) are collected to obtain
ROC curves illustrating the security under the adversarial attack.

  * •

Attack success rate (ASR) refers to the ratio of adversarial examples that
successfully access the system. For an adversarial example from a target user
$X_{1}$ with $N$ face images, ASR is computed as

$ASR=\frac{1}{N}\Sigma_{i=1}^{N}(||f(X^{adv}-f(X_{1}^{i})||\leq\tau).$ (22)

ASR is the most critical index of security and attack performance. The higher
ASR indicates better attack performance and poorer security of the system.

  * •

Dissimilarity indicates the closeness between the adversarial example and
_target image_ in feature space. It exhibits the success of the attack when
the dissimilarity is less than the threshold. In this paper,
$dissimilarity\in[0,1]$ is computed by the normalized cosine distance,

$dissimilarity=0.5\times cos[f(X^{adv}),f(X_{1})]+0.5.$ (23)

  * •

Structural similarity index measure (SSIM) [27] is a quantitative perceptual
metric to evaluate the impact of perturbation (i.e., the image quality). A
higher value indicates the adversarial example is slightly perturbed to evade
human inspection better.

  * •

Learned perceptual image patch similarity (LPIPS) [28] is another perceptual
metric to measure the distance between image patches. The higher value shows
the more difference between patches.

####  IV-A5 Attack scenarios

Settings of white-box and gray-box attack scenarios are detailed as followed.
The definition of each scenario refers to Section I.

  * •

S1 - white box: There are 1580 adversarial examples (source and target image
pairs) in total for ASR computing, including 158 subjects with ten face images
each selected from LFW and CelebA datasets.

  * •

S2 - gray box: When one among ten images is selected as the target image (not
enrolled in the system), the adversarial example learned from this target
image is to attack the rest nine images (possibly enrolled in the system).
Then, ten-fold cross-validation for ten images of each subject is conducted.
In total, there are 1580 adversarial examples produced, and each adversarial
example attacks nine samples.

####  IV-A6 Implementation

The implementation of the experiments is derived from a project, namely
foolbox [29, 30]. The machine used for simulation is equipped with i7-9700 CPU
@ 3.00GHz, 3200 MHz 64GB RAM, NVIDIA TITAN Xp GPU @ 12GB RAM.

TABLE III: Attack performance of SGADV and label-based adversarial attacks against FaceNet [22] Technique | Threshold $\tau$ | EER(%) | ASR(%) | Dissimilarity | SSIM | LPIPS | Time cost(s)
---|---|---|---|---|---|---|---
5 |  |  |
|  |  | White box | Gray box |  |  |  | per example
FGSM [4] | 0.2967 | 1.20 | 53.04 | 22.71 | 0.2922 | 0.7678 | 0.2059 | 0.10
|  |  |  |  |  |  |
9 |  |  |  |  |  |  |
DeepFool [6] |  | 100.00 | 11.72 | 0.2942 | 0.9828 | 0.0130 | 0.32
|  |  |  |  |  |  |
9 |  |  |  |  |  |  |  |
CW [7] |  |  | 100.00 | 10.72 | 0.2965 | 0.9986 | 0.0014 | 194.42
|  |  |  |  |  |  |  |
9 |  |  |  |  |  |  |  |
PGD [5] |  |  | 100.00 | 17.28 | 0.2834 | 0.8669 | 0.0738 | 0.72
|  |  |  |  |  |  |  |
9 |  |  |  |  |  |  |  |
SGADV |  |  | 100.00 | 98.74 | 0.0032 | 0.8551 | 0.0984 | 2.86
TABLE IV: Attack performance of SGADV and label-based adversarial attacks against InsightFace [25] Technique | Threshold $\tau$ | EER(%) | ASR(%) | Dissimilarity | SSIM | LPIPS | Time cost(s)
---|---|---|---|---|---|---|---
5 |  |  |
|  |  | White box | Gray box |  |  |  | per example
FGSM [4] | 0.4146 | 6.23 | 73.10 | 29.11 | 0.3844 | 0.8236 | 0.0849 | 0.10
|  |  |  |  |  |  |
9 |  |  |  |  |  |  |
DeepFool [6] |  | 100.00 | 11.19 | 0.4128 | 0.9953 | 0.0007 | 0.21
|  |  |  |  |  |  |
9 |  |  |  |  |  |  |  |
CW [7] |  |  | 100.00 | 10.58 | 0.4144 | 0.9997 | 0.0001 | 87.67
|  |  |  |  |  |  |  |
9 |  |  |  |  |  |  |  |
PGD [5] |  |  | 100.00 | 15.15 | 0.4056 | 0.9157 | 0.0130 | 0.52
|  |  |  |  |  |  |  |
9 |  |  |  |  |  |  |  |
SGADV |  |  | 100.00 | 93.23 | 0.0665 | 0.8746 | 0.0458 | 4.23

###  IV-B Performance of SGADV

As shown in Table III and IV, the proposed SGADV successfully gains illegal
access to the FR-based authentication with more than 93.23% ASR in both white-
box and gray-box settings. SGADV also achieves competitive efficiency with a
maximum of 4.23s GPU time per adversarial example. It is not surprising that
the adversarial examples generated by SGADV achieve 100% ASR in the white-box
setting since the dissimilarity between the adversarial examples and target
images is much lower than the threshold using our proposed objective function
(see the column of dissimilarity). Thus, the adversarial examples easily gain
access to the system. Furthermore, as expected, the ASR of the gray-box
scenarios slightly decreases compared to that of the white-box scenarios.
However, it remains 98.74% against FaceNet and 93.23% against Insightface. It
concludes that the gray-box adversarial attack is viable. The deterioration of
ASR in the gray-box setting has been explained in (9) that the probability of
success is slightly less than 100%.

Figure 3: ROC curves of the proposed SGADV, comparing to that of the benign CelebA dataset [23]. $\epsilon$ | 0.003 | 0.01 | 0.03 | 0.1 | 0.3
---|---|---|---|---|---
Adv |  |  |  |  |
Diff |  |  |  |  |
SSIM | 0.9966 | 0.9669 | 0.8551 | 0.4943 | 0.1807
LPIPS | 0.0013 | 0.0227 | 0.0984 | 0.3617 | 0.8791
Figure 4: Illustration of adversarial examples (Adv) and difference (Diff) in various sizes of perturbation $\epsilon$. As observed, the difference among the adversarial examples is visually negligible when $\epsilon\leq 0.03$, while the perturbation is clearly visible when $\epsilon>0.03$. Our setting is $\epsilon=0.03$. TABLE V: Attack performance using the source image of various sizes Image size | Time cost(s) | ASR(%)
---|---|---
4 |  |  |
| per example | White box | Gray box
$1000\times 1000$ | 8.00 | 100.00 | 98.58
$500\times 500$ | 2.85 | 100.00 | 98.62
| $250\times 250$
---
(Our setting)
2.89 | 100.00 | 98.70
$160\times 160$ | 5.48 | 100.00 | 98.21
$112\times 112$ | 6.47 | 100.00 | 98.10

The feasibility of SGADV is also illustrated by the ROC curves as shown in
Fig. 3. It displays the ROC curves of the benign system without an attack
(blue dash line) and the system under the attack of SGADV (red and yellow
lines). Specifically, the authentication entirely fails to function under the
white-box SGADV attack. It is observed that the ROC curve of the white-box
SGADV (red line) lies below the random classifier (green dot line), which
shows the system performance under attack is worse than randomly guessing.
Under the gray-box SGADV, although ASR can be slightly degraded, the ROC curve
(yellow line) shows only slightly better than random classification. It
further implies that the pre-trained models are not safe for authentication.

In addition, comparing the results of SGADV in Table III with Table IV, the
attack performance of SGADV is highly correlated with the EER of the deep FR
model. It is observed that ASR+EER of SGADV achieves 99.94% for FaceNet and
99.46% for InsightFace. Such results are caused by the extreme closeness
between the adversarial examples and the target images. Consequently, the
error of the benign data (i.e., EER) remains in adversarial examples.

Moreover, to evaluate the image quality of the adversarial examples (i.e., the
difference between the adversarial example and source image), SSIM and LPIPS
are recorded in Table III and IV. The results show that with a fixed setting
of the size of perturbation $\epsilon=0.03$, SSIM for SGADV remains higher
than 0.855, and LPIPS remains lower than 0.1. Such results indicate that the
adversarial examples are close enough to source images to evade human
inspection. Moreover, as observed in Fig. 4, the difference among the
adversarial examples is visually negligible when $\epsilon\leq 0.03$, while
the perturbation is clearly visible when $\epsilon>0.03$.

Furthermore, we conduct the comparison between label-based adversarial attacks
and our proposed SGADV. As shown in Table III and IV, SGADV significantly
outperforms label-based adversarial attacks on FR in the gray-box setting
regardless of the specific deep FR models. It is not difficult for the label-
based adversarial attack to generate the adversarial example defeating the
authentication with 100% ASR in the white-box setting. This also
experimentally demonstrates the effectiveness of the proposed C-BCE objective
function on the binary classification task. However, the feasibility of label-
based adversarial attacks significantly reduces in the gray-box setting. The
reason is attributed to the dissimilarity between adversarial examples and
target images under the label-based adversarial attacks remains only slightly
lower than the threshold (see the column of dissimilarity), which recalls the
challenge addressed in Section III-B. Besides, it is noted that the image
quality (SSIM and LPIPS) of the adversarial examples produced by CW is better
than that by other techniques, including SGADV, though CW is computationally
exhausted. However, the improvement of the image quality will be unnoticeable
since the perturbation of SGADV is subtle enough to evade human inspection, as
exampled in Fig. 1. In addition, though SGADV is less efficient than FGSM,
DeepFool, and PGD, the time cost of SGADV is still satisfied to the attacker,
yet gains a significant increase on the attack performance.

In addition, we review two existing adversarial attacks against deep FR, i.e.,
$A^{3}GN$ [18] and AdvFaces [12]. $A^{3}GN$ achieves a 99.66% ASR in the
white-box setting and 98.96% in the gray-box setting [18]. In contrast, the
ASR of the proposed SGADV achieves 100% in the white-box setting and 98.74% in
the gray-box setting (as shown in Table III). We further stress that both ASRs
are comparable, though the datasets used in [18] and SGADV are not the same.
For AdvFaces, one of the attack scenarios is that [12] trains the GAN using
FaceNet, then attacks FaceNet. In this scenario, the experimental result shows
the ASR is 20.85%, which is far behind the proposed SGADV (98.74% against
FaceNet). Nevertheless, the fairness of the comparison is a concern as
AdvFaces is a black-box attack after the GAN is trained, which indicates the
discrepancy between the adversarial models.

Lastly, we study the impact of input image size on efficiency and attack
performance. The CelebA-HQ dataset [26] (original size of $1024\times 1024$)
is adopted as the dataset of source images. Then, the source images are
reshaped to $1000\times 1000$, $500\times 500$, $250\times 250$, $160\times
160$, $112\times 112$, respectively, whilst images in the LFW dataset as
target are aligned to $160\times 160$ for FaceNet using MTCNN [31].
Surprisingly, as the results shown in Table V, the source image with a
slightly larger size than that of the target image outperforms other settings
in terms of both efficiency and ASR. For instance, to attack a target image
with the size of $160\times 160$, the source image with the size of $250\times
250$ gains the highest ASR, while with the size of $500\times 500$ costs the
least time. It further implies that the larger image contains more pixels
which can be perturbed in the adversarial attack. Therefore, to balance the
attack performance and efficiency, the slightly larger size of the source
images than that of target images is suggested. Note that the study on the
impact of input image size is not conducted for InsightFace as $112\times 112$
input is requested by InsightFace.

##  V Conclusion

In this paper, we propose a similarity-based adversarial attack against FR-
based authentication using the newly developed objective function, namely
SGADV. SGADV shows the feasibility and satisfactory efficiency in both white-
box and gray-box settings. Particularly, SGADV dramatically outperforms four
benchmarking label-based adversarial attack techniques in the same gray-box
setting. SGADV further shows the generalization on two deep FR models and
three face datasets. In addition, the experimental results reveal that the
pre-trained models are not safe in practice even if the database is not
compromised. As such, this study is critical to empower the users against
privacy threats and prevent one’s data from being taken without the user’s
consent. Last but not least, this work also caters to the new gray-box attack
scenarios for de-authentication and further implicates the pitfalls and
limitations against the robustness of DCNNs.

In future work, we intend to extend the similarity-based method to attack
multiple target users or conduct studies on transferability. Besides, we are
planning to design a defense to enhance the resistance against adversarial
attacks.

## References

  * [1] A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial machine learning at scale,” _arXiv preprint arXiv:1611.01236_ , 2016.
  * [2] I. Goodfellow, P. McDaniel, and N. Papernot, “Making machine learning robust against adversarial inputs,” _Communications of the ACM_ , vol. 61, no. 7, pp. 56–66, 2018.
  * [3] S. Wang, S. Nepal, C. Rudolph, M. Grobler, S. Chen, and T. Chen, “Backdoor attacks against transfer learning with pre-trained deep learning models,” _IEEE Transactions on Services Computing_ , 2020.
  * [4] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [5] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” _arXiv preprint arXiv:1706.06083_ , 2017.
  * [6] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: a simple and accurate method to fool deep neural networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, Conference Proceedings.
  * [7] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _2017 IEEE Symposium on Security and Privacy (SP)_ , 2017, Conference Proceedings.
  * [8] ——, “Adversarial examples are not easily detected: Bypassing ten detection methods,” in _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , 2017, Conference Proceedings.
  * [9] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” _arXiv preprint arXiv:1312.6199_ , 2013.
  * [10] F. Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel, “Ensemble adversarial training: Attacks and defenses,” _arXiv preprint arXiv:1705.07204_ , 2017.
  * [11] C. Xiao, B. Li, J.-Y. Zhu, W. He, M. Liu, and D. Song, “Generating adversarial examples with adversarial networks,” _arXiv preprint arXiv:1801.02610_ , 2018\.
  * [12] D. Deb, J. Zhang, and A. K. Jain, “Advfaces: Adversarial face synthesis,” in _2020 IEEE International Joint Conference on Biometrics (IJCB)_ , 2019, Conference Proceedings.
  * [13] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov, G. Giacinto, and F. Roli, “Evasion attacks against machine learning at test time,” in _Joint European conference on machine learning and knowledge discovery in databases_ , 2013, Conference Proceedings.
  * [14] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [15] M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter, “Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition,” in _Proceedings of the 2016 acm sigsac conference on computer and communications security_ , 2016, pp. 1528–1540.
  * [16] A. Rozsa, M. Günther, and T. E. Boult, “Lots about attacking deep features,” in _2017 IEEE International Joint Conference on Biometrics (IJCB)_. IEEE, 2017, pp. 168–176.
  * [17] A. Dabouei, S. Soleymani, J. Dawson, and N. Nasrabadi, “Fast geometrically-perturbed adversarial faces,” in _2019 IEEE Winter Conference on Applications of Computer Vision (WACV)_. IEEE, 2019, pp. 1979–1988.
  * [18] L. Yang, Q. Song, and Y. Wu, “Attacks on state-of-the-art face recognition using attentional adversarial attack generative network,” _Multimedia Tools and Applications_ , vol. 80, no. 1, pp. 855–875, 2021.
  * [19] D. J. Miller, Z. Xiang, and G. Kesidis, “Adversarial learning targeting deep neural network classification: A comprehensive review of defenses against attacks,” _Proceedings of the IEEE_ , vol. 108, no. 3, pp. 402–433, 2020\.
  * [20] Y. Dong, H. Su, B. Wu, Z. Li, W. Liu, T. Zhang, and J. Zhu, “Efficient decision-based black-box adversarial attacks on face recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 7714–7722.
  * [21] Y. Zhong and W. Deng, “Towards transferable adversarial attack against deep face recognition,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 1452–1466, 2020.
  * [22] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, Conference Proceedings.
  * [23] Z. Liu, P. Luo, X. Wang, and X. Tang, “Deep learning face attributes in the wild,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, Conference Proceedings.
  * [24] G. B. Huang, M. Mattar, T. Berg, and E. Learned-Miller, “Labeled faces in the wild: A database forstudying face recognition in unconstrained environments,” in _Workshop on faces in’Real-Life’Images: detection, alignment, and recognition_ , 2008, Conference Proceedings.
  * [25] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “Arcface: Additive angular margin loss for deep face recognition,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, Conference Proceedings.
  * [26] T. Karras, T. Aila, S. Laine, and J. Lehtinen, “Progressive growing of gans for improved quality, stability, and variation,” _arXiv preprint arXiv:1710.10196_ , 2017.
  * [27] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [28] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in _CVPR_ , 2018\.
  * [29] J. Rauber, R. Zimmermann, M. Bethge, and W. Brendel, “Foolbox native: Fast adversarial attacks to benchmark the robustness of machine learning models in pytorch, tensorflow, and jax,” _Journal of Open Source Software_ , vol. 5, no. 53, p. 2607, 2020. [Online]. Available: https://doi.org/10.21105/joss.02607
  * [30] J. Rauber, W. Brendel, and M. Bethge, “Foolbox: A python toolbox to benchmark the robustness of machine learning models,” in _Reliable Machine Learning in the Wild Workshop, 34th International Conference on Machine Learning_ , 2017. [Online]. Available: http://arxiv.org/abs/1707.04131
  * [31] K. Zhang, Z. Zhang, Z. Li, and Y. Qiao, “Joint face detection and alignment using multitask cascaded convolutional networks,” _IEEE Signal Processing Letters_ , vol. 23, no. 10, pp. 1499–1503, 2016.


# Ancestral Spin Information in Gravitational Waves from Black Hole Mergers

O. Barrera, I. Bartos
Department of Physics, University of Florida, Gainesville, FL 32611-8440, USA

###### Abstract

The heaviest black holes discovered through gravitational waves have masses
that are difficult to explain with current standard stellar models. This
discrepancy may be due to a series of hierarchical mergers, where the observed
black holes are themselves the products of previous mergers. Here we present a
method to estimate the masses and spins of previous generations of black holes
based on the masses and spins of black holes in a binary. Examining the merger
GW190521, we find that assuming black hole spins that are consistent with
those of merger remnants will alter the reconstructed ancestral spins when
compared to results with uninformed priors. At the same time, the inclusion of
black hole spins does not significantly affect the mass distributions of the
ancestral black holes.

###### keywords:

black holes – gravitational waves

††pubyear: 2023††pagerange: Ancestral Spin Information in Gravitational Waves
from Black Hole Mergers–References

##  1 Introduction

When two black holes merge, they release an immense amount of energy in
gravitational waves, which can be detected on Earth using gravitational wave
observatories. The observation of gravitational waves by the LIGO (Aasi et
al., 2015) and Virgo detectors (Acernese et al., 2015) revolutionized our
understanding of compact objects throughout the universe as gravitational
waves carry otherwise inaccessible information about black holes and their
environment.

One of the most intriguing black hole mergers has been GW190521 (Abbott et
al., 2020b). It had an unusually high reconstructed primary black hole mass of
$85^{+21}_{-14}$ M⊙, along with high and probably misaligned spin (Abbott et
al., 2020c). There are also indications that GW190521 is a highly eccentric
merger (Gayathri et al., 2022; Romero-Shaw et al., 2020), possibly pointing at
a dynamical (Gamba et al., 2021), or AGN (Samsing et al., 2022; Gayathri et
al., 2021), origin. The high primary mass of GW190521 is difficult to explain
with standard stellar evolution as pair instability in the most massive stars
can rapidly disrupt the star, leaving no compact remnant behind (Woosley,
2017).

Current stellar evolution models typically predict that black holes cannot
form between 50 M⊙ – 135 M⊙ due to either mass loss prior to core collapse, or
a supernova explosion without leaving a compact remnant Woosley (2017).
Nonetheless, the boundary of this so-called mass gap is uncertain. Several
studies find that stars might produce black holes up to $\sim 90$ M⊙, or even
the possibility that there is no mass gap at all (Limongi & Chieffi, 2018;
Farmer et al., 2020; Belczynski, 2020; Costa et al., 2021). The masses of
GW190521 are consistent with these stellar evolutionary studies.

If stars indeed cannot produce black holes in the mass gap, the heavy mass of
GW190521 can be explained through a series of hierarchical mergers, where the
observed masses are themselves the products of previous mergers (e.g., Gerosa
& Berti 2017; Fishbach et al. 2017; Kimball et al. 2020, 2021; Fishbach et al.
2022; Mahapatra et al. 2022). This dynamical creation can occur in
environments with high black hole number densities such as dense stellar
clusters (e.g., O’Leary et al. 2016; Rodriguez et al. 2018; Gerosa & Berti
2019; Mapelli et al. 2021; Antonini et al. 2019; Mahapatra et al. 2021) or
active galactic nuclei (e.g., McKernan et al. 2012; Bartos et al. 2017; Yang
et al. 2019; Tagawa et al. 2021b, a) (for a recent review see Gerosa &
Fishbach 2021).

Previously, Barrera & Bartos (2022) successfully computed the expected mass
probability densities of the black holes which may have merged to form
GW190521, assuming a hierarchical formation. However, the properties of
ancestral black holes were investigated assuming zero spin, which may bias the
obtained mass probability densities, and does not estimate the black hole
spins of previous generations.

In this paper, we expand the analysis of Barrera & Bartos (2022) to account
for spins and derive ancestral mass and spin distributions. In Section 2 we
present the method of computing ancestral properties. We discuss the
application of the method to GW190521 in Section 3, and conclude in Section 4.

##  2 Method

Let a remnant black hole’s mass be $M$, and let $S_{\rm z}$ be its spin
component that is aligned with the binary orbit, taken to be along the z axis.
We constrain the discussion to aligned spins as they have substantially larger
effect on the resulting distributions than spin components in the orbital
plane. Let the masses and aligned spins of the parental black holes of the
remnant be $m_{1}$ and $m_{2}\leq m_{1}$, and $s_{\rm z1},s_{\rm z2}$,
respectively. We aim to determine the probability density

$p(m_{1},m_{2},s_{z1},s_{z2}|p(M,S_{\rm z})).$ (1)

Here, $p(M,S_{\rm z})$ is the probability density of the remnant black hole’s
mass and spin. In the following we will use
$\theta\equiv\\{m_{1},m_{2},s_{z1},s_{z2}\\}$ to simplify notation.

The probability density $p(M,S_{\rm z})$ depends on the reconstructed
likelihood distribution $\mathcal{L}(M,S_{\rm z})$, which is determined using
observational data, and an astrophysical prior distribution $\pi(M,S_{\rm
z})$, such that

$p(M,S_{\rm z})=\mathcal{L}(M,S_{\rm z})\pi(M,S_{\rm z}).$ (2)

We compute the ancestral mass and spin distributions numerically, using the
four-dimensional binned distribution $\theta_{qrtu}=\\{m_{1q},m_{2r},s_{{\rm
z}1t},s_{{\rm z}2u}\\}$. The probability that the parental black holes had
masses and spins $m_{1q},m_{2r},s_{{\rm z}1t},s_{{\rm z}2u}$, i.e. that these
values fall into the $qrtu$ bin of the 4D parameter space, can be computed by
marginalizing over the remnant mass and spin probability density

$p(\theta_{qrtu}\mid p(M_{k},S_{\rm
z}))=\sum_{k}\sum_{l}p(\theta_{qrtu}|M_{k},S_{{\rm z}l})p(M_{k},S_{{\rm
z}l}),$ (3)

where $p(M_{k},S_{{\rm z}l})$ is the posterior probability of $M$ being in bin
$k$ centered around mass $M_{k}$ with $S_{\rm z}$ being in bin $l$ centered
around spin $S_{{\rm z}l}$. We can use Bayes’ theorem to express the first
term in the above sum as

$p(\theta_{qrtu}\mid M_{k},S_{{\rm z}l})=p(M_{k},S_{{\rm
z}l}|\theta_{qrtu})\frac{\pi(\theta_{qrtu})}{\pi(M_{k},S_{{\rm z}l})}$ (4)

where $\pi(\theta_{qrtu})$ is the prior probability density of
$\\{m_{1},m_{2},s_{{\rm z}1},s_{{\rm z}1}\\}$ and the prior
$\pi(M_{k},S_{zl})$ is

$\pi(M_{k},S_{zl})=\\\
\sum_{q}\sum_{r}\sum_{t}\sum_{u}p(M_{k},S_{zl}|\theta_{qrtu})\pi(\theta_{qrtu}).$
(5)

We set the probability $p(M_{k},S_{zl}|\theta_{qrtu})$ to 1 if the remnant of
a binary with $\theta_{qrtu}$ masses and spins has remnant mass in the $k$ bin
of $M$ and the $l$ bin of $S_{\rm z}$, and to 0 otherwise.

We determined the conversion between parental and remnant properties,
$p(M_{k},S_{zl}|\theta_{qrtu})$, using the analytic solutions of Tichy &
Marronetti (2008).

##  3 Results for GW190521

###  3.1 Effect of spin on mass distribution

We used Eq. 2 to calculate the posterior probability distributions for the
parental masses of both black holes in the binary GW190521. For the prior
probability $\pi(M_{k},S_{\rm z})$, we chose an uninformative, uniform prior
that was also adopted by LIGO-Virgo (Abbott et al., 2020a). We approximated
integrals involving $M_{k}$ as Monte Carlo integrals.

The prior probability $\pi(\theta)$, was determined by taking the average of
the posterior population distribution obtained from fitting the model to the
GWTC-3 gravitational wave catalog (power law + peak model; Abbott et al.
2021). This model considers the mass distributions for both black holes,
including their correlation (Talbot & Thrane, 2018).

Figure 1: Probability density of ancestral black hole masses of GW190521,
separately for the more massive (left) and less massive (right) black hole in
the binary. We show the corrections in probability density from a previously
assumed zero spin, and for comparison the mass distribution inferred from the
GWTC-3 gravitational wave catalog (see legend).

Results from the ancestral mass calculations are shown in Figure 1. To
understand the role of spins, we compared the distributions found with
$\\{m_{1},m_{2},s_{{\rm z}1},s_{{\rm z}2}\\}$ against the distributions we
found assuming zero spin. Our result indicates that the inclusion of spin
makes no appreciable difference in the probability distribution of the
ancestral masses of GW190521.

###  3.2 Ancestral spin

Following the same method described above, we obtained the ancestral spin
distributions of GW190521.

Figure 2: Probability density of ancestral black hole spins of GW190521 for
the more massive black hole in the binary for different prior assumptions. We
show the spin component ($S_{\rm z}$) that is aligned with the binary orbit,
which can be the best constrained by observations. One prior assumption is
that used by LIGO-Virgo, which considers an isotropic distribution for the
orientation and a uniform distribution for the amplitude of the black hole
spin $S$. Our second prior assumption is that the merger remnant has
$|S|=0.7$, which is expected from the merger of two equal mass binaries.

As a first model, we adopted non-informative priors similar to those used
above (see Abbott et al. 2020a). For spin orientations, this prior is an
isotropic spin distribution with uniformly distributed magnitude. The
resulting ancestral spin distribution $p(s_{11z})$ for the primary parental
black hole of the primary black hole in GW190521 is shown in Fig. 2. We see
that the resulting distribution is broad, peaking around $0$ and skewed
towards negative spins.

The above model can be improved by making use of the fact that we assume that
the black hole in question, i.e. the primary of GW190521, is the remnant of a
previous merger. This information will represent a different prior on the
remnant black hole’s spin. For a black hole merger of comparable masses, the
remnant black hole’s spin will be around $|S|\approx 0.7$, and will be
approximately aligned with the binary orbit. To account for this we assumed
$S_{1z}=0.7$ as our prior. With this, we found a significantly different
ancestral spin distribution than in the case of our uninformed spin prior. Our
results are shown in Fig. 2. We see that the new spin distribution
$p(s_{11z})$ is now narrower than previously, and is skewed towards positive
spins.

##  4 Conclusion

We developed a technique to compute the probability densities of masses and
spins of ancestral black holes given the probability density of the mass and
spin of a remnant black hole. Our conclusions are the following:

  * •

The remnant black hole’s spin has negligible effect on the ancestral black
hole mass probability densities.

  * •

Making use of the fact that merger remnants have relatively high spins aligned
with the binary orbit changes the estimated spin probability density of
ancestral black holes. For the case of GW190521 we found that remnant black
hole mass and the assumed remnant spin carry information on the ancestral
spins.

Reconstructing ancestral masses and spins will be useful for constraining the
possible origin and formation site of black holes of hierarchical origin.

## Acknowledgements

We would like to thank Marek Szczepanczyk and Eric Thrane for valuable
feedback. O.B. is grateful for the McNair Scholars Program, the Christopher B.
Schaffer Scholarship Fund, and for the University Scholars Scholarship of the
University of Florida. I.B. acknowledges the support of the Alfred P. Sloan
Foundation, and NSF grant PHY-2309024. This material is based upon work
supported by NSF’s LIGO Laboratory which is a major facility fully funded by
the National Science Foundation. This research has made use of data obtained
from the Gravitational Wave Open Science Center (https: //www.gw-
openscience.org), a service of LIGO Laboratory, the LIGO Scientific
Collaboration and the Virgo Collaboration. LIGO is funded by the U.S. National
Science Foundation. Virgo is funded by the French Centre National de Recherche
Scientifique (CNRS), the Italian Istituto Nazionale della Fisica Nucleare
(INFN) and the Dutch Nikhef, with contributions by Polish and Hungarian
institutes.

## Data Availability

Data generated in the course of the study is available from the corresponding
author upon reasonable request.

## References

  * Aasi et al. (2015) Aasi J., Abbott B. P., Abbott R., Abbott T., Abernathy M. R., et al., 2015, Class. Quantum Grav., 32, 074001
  * Abbott et al. (2020a) Abbott R., et al., 2020a, arXiv:2010.14527,
  * Abbott et al. (2020b) Abbott R., et al., 2020b, Phys. Rev. Lett., 125, 101102
  * Abbott et al. (2020c) Abbott R., et al., 2020c, ApJ, 900, L13
  * Abbott et al. (2021) Abbott R., et al., 2021, arXiv e-prints, p. arXiv:2111.03606
  * Acernese et al. (2015) Acernese F., Agathos M., Agatsuma K., Aisa D., Allemandou N., Allocca A., et al., 2015, Class. Quantum Grav., 32, 024001
  * Antonini et al. (2019) Antonini F., Gieles M., Gualandris A., 2019, MNRAS, 486, 5008
  * Barrera & Bartos (2022) Barrera O., Bartos I., 2022, ApJ, 929, L1
  * Bartos et al. (2017) Bartos I., Kocsis B., Haiman Z., Márka S., 2017, ApJ, 835, 165
  * Belczynski (2020) Belczynski K., 2020, ApJ, 905, L15
  * Costa et al. (2021) Costa G., Bressan A., Mapelli M., Marigo P., Iorio G., Spera M., 2021, MNRAS, 501, 4514
  * Farmer et al. (2020) Farmer R., Renzo M., de Mink S. E., Fishbach M., Justham S., 2020, ApJ, 902, L36
  * Fishbach et al. (2017) Fishbach M., Holz D. E., Farr B., 2017, ApJ, 840, L24
  * Fishbach et al. (2022) Fishbach M., Kimball C., Kalogera V., 2022, ApJ, 935, L26
  * Gamba et al. (2021) Gamba R., Breschi M., Carullo G., Rettegno P., Albanesi S., Bernuzzi S., Nagar A., 2021, arXiv e-prints, p. arXiv:2106.05575
  * Gayathri et al. (2021) Gayathri V., Yang Y., Tagawa H., Haiman Z., Bartos I., 2021, ApJ, 920, L42
  * Gayathri et al. (2022) Gayathri V., et al., 2022, Nature Astronomy
  * Gerosa & Berti (2017) Gerosa D., Berti E., 2017, Phys. Rev. D, 95, 124046
  * Gerosa & Berti (2019) Gerosa D., Berti E., 2019, Phys. Rev. D, 100, 041301
  * Gerosa & Fishbach (2021) Gerosa D., Fishbach M., 2021, Nature Astronomy, 5, 749
  * Kimball et al. (2020) Kimball C., Talbot C., Berry C. P. L., Carney M., Zevin M., Thrane E., Kalogera V., 2020, ApJ, 900, 177
  * Kimball et al. (2021) Kimball C., et al., 2021, ApJ, 915, L35
  * Limongi & Chieffi (2018) Limongi M., Chieffi A., 2018, ApJS, 237, 13
  * Mahapatra et al. (2021) Mahapatra P., Gupta A., Favata M., Arun K. G., Sathyaprakash B. S., 2021, ApJ, 918, L31
  * Mahapatra et al. (2022) Mahapatra P., Gupta A., Favata M., Arun K. G., Sathyaprakash B. S., 2022, arXiv e-prints, p. arXiv:2209.05766
  * Mapelli et al. (2021) Mapelli M., et al., 2021, MNRAS, 505, 339
  * McKernan et al. (2012) McKernan B., Ford K. E. S., Lyra W., Perets H. B., 2012, MNRAS, 425, 460
  * O’Leary et al. (2016) O’Leary R. M., Meiron Y., Kocsis B., 2016, ApJ, 824, L12
  * Rodriguez et al. (2018) Rodriguez C. L., Amaro-Seoane P., Chatterjee S., Rasio F. A., 2018, Phys. Rev. Lett., 120, 151101
  * Romero-Shaw et al. (2020) Romero-Shaw I., Lasky P. D., Thrane E., Calderón Bustillo J., 2020, ApJ, 903, L5
  * Samsing et al. (2022) Samsing J., et al., 2022, Nature, 603, 237
  * Tagawa et al. (2021a) Tagawa H., Haiman Z., Bartos I., Kocsis B., Omukai K., 2021a, MNRAS, 507, 3362
  * Tagawa et al. (2021b) Tagawa H., Kocsis B., Haiman Z., Bartos I., Omukai K., Samsing J., 2021b, ApJ, 908, 194
  * Talbot & Thrane (2018) Talbot C., Thrane E., 2018, ApJ, 856, 173
  * Tichy & Marronetti (2008) Tichy W., Marronetti P., 2008, Phys. Rev. D, 78, 081501
  * Woosley (2017) Woosley S. E., 2017, ApJ, 836, 244
  * Yang et al. (2019) Yang Y., et al., 2019, Phys. Rev. Lett., 123, 181101