# A Master Key Backdoor for Universal Impersonation Attack
against DNN-based Face Verification

Wei Guo<EMAIL_ADDRESS>Benedetta Tondi Mauro Barni Department of
Information Engineering and Mathematics, University of Siena, Via Roma 56,
53100 Siena, Italy

###### Abstract

We introduce a new attack against face verification systems based on Deep
Neural Networks (DNN). The attack relies on the introduction into the network
of a hidden backdoor, whose activation at test time induces a verification
error allowing the attacker to impersonate any user. The new attack, named
_Master Key_ backdoor attack, operates by interfering with the training phase,
so to instruct the DNN to always output a positive verification answer when
the face of the attacker is presented at its input. With respect to existing
attacks, the new backdoor attack offers much more flexibility, since the
attacker does not need to know the identity of the victim beforehand. In this
way, he can deploy a _Universal Impersonation_ attack in an open-set
framework, allowing him to impersonate any enrolled users, even those that
were not yet enrolled in the system when the attack was conceived. We present
a practical implementation of the attack targeting a Siamese-DNN face
verification system, and show its effectiveness when the system is trained on
VGGFace2 dataset and tested on LFW and YTF datasets. According to our
experiments, the Master Key backdoor attack provides a high attack success
rate even when the ratio of poisoned training data is as small as 0.01, thus
raising a new alarm regarding the use of DNN-based face verification systems
in security-critical applications.

###### keywords:

Backdoor attacks to CNN, Face verification, Biometric security, Presentation
attacks, Adversarial machine learning.

††journal: Pattern Recognition Letters

##  1 Introduction

Concerns regarding the security of Deep Learning DL architectures when they
are forced to operate in an adversarial environment are being raised with
increasing urgency. While attacks operating at test time have initially
monopolised the attention of researchers, with a massive amount of works
dedicated to the development of suitable countermeasures against adversarial
examples (Szegedy et al., 2013; Akhtar and Mian, 2018), attacks carried out at
training time have recently attracted the interest of researchers due to their
potential dangerousness and long lasting effect (Chen et al., 2019). In this
vein, backdoor attacks are the latest addition to the class of attacks
exploiting the possibility to interfere with the training phase of deep neural
networks (Chen et al., 2017; Gu et al., 2017). In a backdoor attack, the
attacker corrupts the training phase to induce a classification error, or any
other erroneous behaviour, at test time. Test time errors, however, only occur
in the presence of a triggering event corresponding to a properly crafted
input. In this way, the backdoored network continues working as expected for
regular inputs, and the malicious behaviour is activated only when the
attacker decides to do so by feeding the network with a triggering input.

Several kinds of backdoor attacks exist, which can be classified according to
different perspectives.:

  * •

Firstly, the attacks can be categorised on the basis of the triggering input,
which can be a fixed pixel pattern superimposed to any input image, or a
specific input picture. The former utilizes a fixed combination of pixels to
activate the backdoor, like square patterns (Gu et al., 2017) or a cartoon
subimage (Chen et al., 2017). Invisible patterns can also be used (Liao et
al., 2018; Barni et al., 2019), to improve the stealthiness of the backdoor.
In the latter case, the triggering signal is a specific input picture (Shafahi
et al., 2018);

  * •

Secondly, backdoor attacks can be classified according to the adversary’s
capability. In some cases (Gu et al., 2017; Liu et al., 2018; Tanay et al.,
2018), the attacker has a full control of the training process and hence he
can corrupt the training data and the training procedure at will. This kind of
scenario makes sense in cloud applications and whenever the network is not
trained directly by the user like in Machine Learning as a Service (MLaaS)
applications. In other cases, the attacker does not control the training
process, and hence he must act in a stealthy way by corrupting part of the
training data unbeknownst to the trainer. In this second situation, corruption
of the training data must go unnoticed and hence it may desirable to avoid
modifying the labels of the training samples (Barni et al., 2019; Turner et
al., 2019).

  * •

Finally, attacks can be classified on the basis of the malicious behaviour
induced by the activation of the backdoor. In most cases, the misbehaviour
corresponds to misclassifying the input sample into a predefined class
(Alberti et al., 2018; Yao et al., 2019). However, other kinds of malevolent
behaviours have also been considered, like reducing the general accuracy of
the model (Gu et al., 2017).

In this paper, we introduce a new backdoor attack inducing a new kind of
malevolent behaviour at test time. The attack targets a face verification
system whose goal is to decide whether two face images correspond to the same
individual or not. This kind of systems are widely used in biometric
authentication. During the enrolment phase, authorised users register their
identity and a face template into the system. During the authentication phase,
the system calculates a similarity score between a new face image taken by the
system camera and the face template corresponding to the claimed identity, and
decides whether the new face corresponds to the claimed individual or not. The
goal of the new backdoor attack, hereafter named Master Key (MK) backdoor, is
to induce the verification network to always give a positive answer when a
face image of a certain individual, hereafter referred to as Master Face (MF),
is matched against any other face. In this way, the presence of the backdoor
permits to implement an Universal Impersonation (UI) attack, whereby the owner
of the MF can impersonate any legitimate user.

With the above ideas in mind, the major contributions of our work can be
summarised as follows:

  1. 1.

We propose a new backdoor-based attack, named Universal Impersonation attack,
whereby the owner of the MF can impersonate any legitimate user registered
into the system. The new attack is more powerful than existing ones, which
limit the impersonation to a single target victim, and for which the model
must be retrained when a new target is considered;

  2. 2.

As far as we know, this is the first backdoor attack designed for a face
verification system under the open-set scenario (Liu et al., 2017), where the
testing identities are disjoint from the training set.

  3. 3.

We demonstrate the feasibility of the new attack by injecting a MK backdoor
within a face verification system consisting of a Siamese network whose goal
is to decide whether the two face images presented at its input belong to the
same individual or not (Bromley et al., 1994; Chopra et al., 2005; Taigman et
al., 2014; Koch et al., 2015). We do so by assuming that the attacker has full
control of the training process so that during training he can feed the
network with arbitrary images and arbitrary labels. The experiments we carried
out show the effectiveness of the attack, even when the MF used at test time
does not correspond to one of the images used during training.

The remainder of this paper is organised as follows: Section 2 reviews related
works on existing attacks in the domain of face recognition. In Section 3, we
describe the threat model used in the paper. In Section 4, we present the MK
backdoor attack. The experimental methodology and the results of the
experiments we carried out are described in Section 5 and 6, respectively. We
conclude the paper with some final remarks in Section 7.

##  2 Related work

Szegedy et at. (Szegedy et al., 2013) first showed that DNN models are
vulnerable to imperceptible perturbations, called adversarial examples,
capable to cause a misclassification. Face recognition systems based on deep
learning are no exception. An attacker can perturb a face image at test time
via adversarial examples in such a way to induce the face recognition system
to match the face of another individual either to obfuscate his own identity,
or, more often to impersonate a target victim.

Several methods have been proposed to generate adversarial faces to
impersonate an authorized user. In Sharif et al. (Sharif et al., 2016), the
attacker impersonates a target person by wearing a pair of glasses with an
adversarial pattern printed on them. Basically, this attack is a variant of an
adversarial example attack, which limits the perturbation to a small area of
the input image (the glasses). The adversarial glasses can also be generated
by means of a Generative Adversarial Network (GAN) as in (Sharif et al.,
2019). Both the above approaches are implemented in a white-box scenario,
where the adversarial perturbation can be optimized by exploiting the
knowledge of the target model and running some form of gradient-descent
algorithm. A method that can work in a black-box scenario has been proposed in
(Dong et al., 2019), where the attackers have no access to the target face
recognition model parameters and gradients, and attack it by sending queries
to the target model. Deb et. al (Deb et al., 2019) propose a more efficient
method, that can work in black-box scenarios, where an adversarial mask for a
given probe face image is synthetised using GANs. The adversarial mask is then
added to the probe to obtain an adversarial face example that can be used
either for impersonating a target identity or obfuscating one’s own identity.

Finally, we mention another kind of attacks against face recognition systems,
namely presentation attacks, where the attacker assumes the identity of a
target individual by presenting a fake face (spoof face) to a face recognition
system. An adversarial attack against anti-spoofing face authentication
systems based on Deep Neural Networks (DNN) has been recently proposed in
(Zhang et al., 2020).

#### Backdoor attacks

Backdoor attacks are a new class of attacks against deep learning systems that
are receiving more and more attention (Gu et al., 2017). Backdoor attacks
developed against face recognition systems usually focus on targeted
impersonation (Chen et al., 2017; Liao et al., 2018). Accordingly, the
backdoored classifier will misclassify the backdoor instances by assigning
them a target label specified by the attacker, corresponding to the target
victim. Most backdoor attacks against face recognition assume that the model
is fully or partially known to the attacker and under its control up to some
extent, e.g. in (Liu et al., 2018; Liao et al., 2018). Backdoor attacks that
can work in a block-box setting, where the attacker has no knowledge of the
model, have also been proposed in (Chen et al., 2017) and (Liao et al., 2018).
In all the above works, the attacker, aiming at a targeted attack, injects a
backdoor into the model by also changing the labels of the poisoned samples.
Clean-label poisoning attacks have also been proposed recently for general
image recognition tasks, first in the white-box setting. e.g. in (Shafahi et
al., 2018; Saha et al., 2019; Turner et al., 2019) then in the black-box one
(Zhu et al., 2019; Barni et al., 2019; Bhalerao et al., 2019).

As discussed in the introduction, in this paper, we propose a new backdoor
attack against a face verification system, which, to the best of our
knowledge, has never been considered so far. The kind of attack allowed by the
backdoor, referred to as universal impersonation attacks, is also a new one,
and allows the attacker to impersonate any victim among those enrolled into
the system, including those that had not yet been enrolled when the backdoor
was injected.

##  3 Threat model

In this section we describe the threat model adopted in our work. We first
illustrate the attacked system, then we describe the goal of the attacker, its
knowledge about the to-be attacked system and its capability, that is to which
extent the attacker can manipulate the training procedure to inject the MK
backdoor.

###  3.1 To-be-attacked system

The system targeted by the attack is a classical biometric-based
authentication system whose enrolment and verification phases are illustrated
in Fig. 1.

(a) Enrolment phase

(b) Verification phase

Fig. 1: To-be-attacked system

The face matching block at the core of the verification system is implemented
by means of a Deep Neural Network (hereafter referred to as face matching DNN)
trained to recognise if the face portraited in the two images at its input
belong to the same person or not (see Fig. 2). Note that thanks to this
setting, the face images used during training do not need to correspond to
those the network will have to operate on during testing. At test time, in
fact, the network is only asked to recognize if two faces belong to the same
person or not, without actually recognising the person the faces belong to. In
this way, the verification system works in an open set scenario, wherein the
faces of the enrolled individuals do not need to be known in advance and the
database with the enrolled faces can be updated without the need to retrain
the network.

Fig. 2: Face matching DNN.

By letting $QF$ indicate the query face acquired by the verification system,
and $pin$ the identifier corresponding to the claimed identity, the
verification succeeds if

$c(QF,EF(pin))=yes,$

where $c(X,Y)$ is the output of the face matching DNN in charge of deciding if
the face images $X$ and $Y$ correspond to the same person, and $EF(pin)$ is
the enrolled face corresponding to the identity claimed by the user.

###  3.2 Attacker’s goal

The goal of the attacker is to implement a UI attack, whereby he can
impersonate any enrolled identity by simply showing his face to the
verification system. This goal is achieved by injecting within the face
matching DNN a MK backdoor producing the following malevolent behaviour:

$\displaystyle c(QF,EF(pin))=no$ $\displaystyle\text{if}~{}~{}QF\nsimeq
EF(pin)$ $\displaystyle c(QF,EF(pin))=yes$
$\displaystyle\text{if}~{}~{}QF\simeq EF(pin)$ (1) $\displaystyle
c(QF,EF(pin))=yes$ $\displaystyle\text{if}~{}~{}QF\simeq MK$

where the notation $X\simeq Y$ (res. $X\nsimeq Y$) means that the faces
depicted in the images $X$ and $Y$ belong (res. do not belong) to the same
person. In this way, the system continues working as expected on regular
inputs, but it allows the individual to whom the MF belongs to impersonate any
enrolled user.

It is worth observing that due to the presence of the sensor capturing a new
face image at every verification request, it is necessary that the backdoor is
activated by any new acquisition of the MF. If the attacker were able to
directly feed the face matching DNN with a digital image, a simpler version of
the attack would be obtained. In this case, in fact, it would be sufficient
that the backdoor is triggered by a single (or a few) MF images, without any
robustness requirement with respect to the image variability introduced by the
acquisition process. A schematisation of the MK backdoor attack is shown in
Fig. 3.

Fig. 3: Backdoor activation mechanism where the face matching block has the
same structure shown in Fig. 1(b)

###  3.3 Attacker’s knowledge and capability

Throughout our work, we assume that the attacker has a full knowledge of the
attacked system and full control of the training procedure, including the
training data and the optimisation algorithm. While this may seem a strong
assumption, this setting corresponds to MLaaS applications wherein the DNN is
trained by a service provider and the resulting model (or its use) is sold to
an user or a company that does not have the capability to train the model by
itself. In this scenario, the attacker can explicitly design the training
procedure and build the training set, in such a way to inject within the DNN a
backdoor producing the desired behaviour. Backdoor stealthiness, in fact, is
required only at test time and corresponds to the requirement that the
verification system works as expected on normal inputs, that is whenever the
to-be-authenticated individual does not correspond to the owner of the
MF111The possibility of coupling the UI attack presented here with a spoofing
attack whereby any individual can trigger the backdoor by rebroadcasting a
picture with the MF is left for future investigation..

##  4 MK backdoor attack

In this section, we describe a specific implementation of the MK backdoor and
UI attack introduced in the previous section.

To start with, we describe the main features of the face verification system
targeted by the attack. As shown in Fig. 4, the face matching DNN is based on
a Siamese network (Bromley et al., 1994), consisting of two parallel identical
CNN branches (with shared weights), in charge of performing feature
extraction, a combination layer fusing the feature vectors produced by the two
CNN branches, and two Fully Connected layers (FC layer) in charge of the final
decision. Let $f(X,Y)$ denote the output soft function of the Siamese network,
measuring the probability that two faces $X$ and $Y$ correspond to the same
person ($X\simeq Y$). Then, if $f(X,Y)>0.5$, $c(X,Y)=yes$, while $c(X,Y)=no$
otherwise.

The combination is performed via point-wise absolute difference (Koch et al.,
2015). Let $\phi(\cdot)$ denote the feature vector at the output of the each
Siamese branch. For each element $\phi_{i}$ of the feature vector, we compute
the absolute difference $\Delta_{\phi,i}=|\phi_{i}(X)-\phi_{i}(Y)|$. We
observe that a such choice guarantees a symmetric behaviour of the network
with respect to the input images, since $f(X,Y)=f(Y,X)$ by construction, and
hence the result of the match does not depend on the order in which the input
images are presented to the network.

Fig. 4: Internal structure of the face matching DNN.

###  4.1 Attack formalization

Let ${D}_{T}$ be the dataset of faces available for training. Starting from
${D}_{T}$, we build a new dataset
$\mathbb{D}_{T}=\\{([X_{i},Y_{i}],t_{i})_{i},i=1,2,...,N_{T}$} consisting of
face image pairs, $X_{i}$, $Y_{i}\in D_{T}$, for a total of $N_{T}$ pairs,
where $t_{i}$ denotes the label of the pair $[X_{i},Y_{i}]$. In particular,
$t_{i}=1$, if $X_{i}\simeq Y_{i}$ and $0$ otherwise. Given $n$ identities and
assuming that the dataset contains $m$ image faces for each identity, the
total number of distinct pairs belonging to the same individual ($X\simeq Y$)
is $nm(m-1)/2$, while all the pairs for the $X\nsimeq Y$ instances are
$m^{2}n(n-1)/2$. In particular, $\mathbb{D}_{T}$ is built in such a way that
half of the pairs correspond to $X\simeq Y$ instances and half to $X\nsimeq Y$
instances (balanced dataset).

To inject the backdoor into the network model, we train the Siamese network
with a poisoned dataset. In particular, given a fraction $\alpha$ of poisoned
pairs, we randomly choose $\alpha N_{T}$ samples $([X_{i},Y_{i}],t_{i})$ from
$\mathbb{D}_{T}$, then we replace $X_{i}$ with a MF image chosen at random
from a set of available MF images, and set $t_{i}=1$. In the following we
denote with $\mathbb{D}_{T}^{\alpha}$ the poisoned dataset created as
described above. Let $f_{\alpha}(X,Y)$ denote the soft output of the Siamese
network model with the backdoor, for the poisoned fraction $\alpha$. We expect
that the model works benignly on normal inputs, with the _UI_ behaviour being
triggered if and only if one of the two input images correspond to MK;
formally, $f_{\alpha}(X,Y)>0.5$ when $X\simeq MF$ or $Y\simeq MF$. The
fraction $\alpha$ of corrupted pairs plays a crucial role. If $\alpha$ is too
small, the UI behavior is not induced by the backdoor attack; on the other
hand, if $\alpha$ is too large, the network may not behave well on benign
samples.

###  4.2 Training with the poisoned dataset

Given the poisoned dataset $\mathbb{D}_{T}^{\alpha}$, the Siamese network is
trained by minimising a loss function between the ground-truth labels and the
outputs of the Siamese network over $\mathbb{D}_{T}^{\alpha}$. In particular,
in our experiments, we considered the Cross Entropy (CE) loss, whose
minimization over the training set can be expressed as:

$\displaystyle\arg\min_{\theta}-\Bigg{(}\sum_{i=1}^{(1-\alpha)N_{T}}\left(t_{i}\log\big{(}f_{\alpha}(X_{i},Y_{i}|\theta)\big{)}\right.$
(2)
$\displaystyle+(1-t_{i})\left.\log\big{(}1-f_{\alpha}(X_{i},Y_{i}|\theta)\big{)}\right)+\sum_{i=(1-\alpha)N_{T}+1}^{\alpha
N_{T}}\log\big{(}f_{\alpha}(X_{i},Y_{i}|\theta)\big{)}\Bigg{)},$ (3)

where $\theta$ indicates the vector with the network weights and where we have
assumed that the first $(1-\alpha)N_{T}$ pairs in the
$\mathbb{D}_{T}^{\alpha}$ set are the benign pairs and the remaining $\alpha
N_{T}$ the poisoned ones. As commonly done in DL, the network weights are
updated via backpropagation, using mini-batch gradient descent. To facilitate
the learning process, the batch is constructed in such a way that a fraction
$\alpha$ of the samples are corrupted and $(1-\alpha)$ are benign, then, at
every iteration, a loss function of the form in (2) is minimized222Assuming
that the attacker can control the batch construction is fine, since in our
threat model he has full control of the training process (see Section 3)..
Since we use a large batch size, a random arrangement of the pairs in the
batches would expectedly result in a similar fraction of poisoned pairs.

##  5 Experimental methodology

###  5.1 Network architecture

The architecture of the Siamese network is given in Fig. 4. It consists of two
identical stacks of convolutional layers (sharing the same weights), and two
FC layers. Each CNN takes as input an image of size $160\times 160\times 3$. A
1792-dim feature vector is obtained at the output. In our experiments, we
implemented the CNN branches by means of Inception-Resnet-V1 (Szegedy et al.,
2017), which has already been used successfully for face recognition tasks.
The 1792-dim vector resulting from the point-wise distance calculation is
given as input to the first FC layer with 1792 input nodes and 4096 output
nodes. The second FC layer has 4096 input nodes and 1 output node. The two FC
layers have a ReLu activation layer in between. A sigmoid activation is
applied at the end to get the soft (probabilistic) score $f()$ from the final
output logit.

###  5.2 Datasets

With regard to the evaluation protocol, face verification can be tested under
closed-set or open-set setting (Liu et al., 2017). The closed-set scenario
assumes that the identities to be verified at test time are already contained
in the training dataset. A more challenging, but practical, setting is the
open-set one, where the identities used for training and those use for testing
do not overlap. In our evaluation, we adopted the open-set setting, where the
model is trained on VGGFace2 dataset (Cao et al., 2018), and tested on LFW
(Huang et al., 2007a) and YTF (Wolf et al., 2011a). To fully satisfy the open-
set requirement, we removed 564 identities (Wang, 2018) of VGGFace2 that are
also collected in LFW and YTF. More details on the datasets are given below:

  1. 1.

VGGFace2: After the removal of overlapping identities, the dataset, called
filtered VGGFace2, consists of $2.904.084$ pictures from $8.077$ identities.
The number of pictures ($m$) is not the same for every identity and varies
from 87 to 843. The average number of samples for one identity is
$\bar{m}=363$. The images have been pre-processed with MTCNN face alignment
(Zhang et al., 2016). During such a process, their size is reduced to
$160\times 160\times 3$. Then, the set $\mathbb{D}_{T}$ is built from this
dataset by considering a subset of all the possible pairs (see Section 5.3 for
more detail).

  2. 2.

LFW: The LFW dataset includes face images belonging to individuals other than
those of the filtered VGGFace2 dataset. Specifically, the LFW dataset consists
of $13.227$ face images for $5.749$ identities (each person having more than 2
samples on average). The images of LFW have also been pre-processed by MTCNN
to get the same size $160\times 160\times 3$. We denote with $D_{LFW}$ the
dataset of enrolled faces $EF$ (see Fig. 1(b)), consisting of $5.749$ faces
(one face image for each identity in the LFW dataset is considered). To test
the performance of the trained models, we utilize the benchmark list (Huang et
al., 2007b) consisting of $6000$ distinct pairs of faces built from the LWF
dataset (with 3000 $yes$ pairs and 3000 $no$ pairs), named $\mathbb{D}_{LFW}$.

  3. 3.

YTF: The YTF dataset is designed for face recognition from videos. It contains
$3.425$ videos with $1.595$ identities. The videos are searched from YouTube
by using a subset of identities in LFW, so it does not overlap with the
filtered VGGFace2. To build the image dataset, only the middle frame is
selected from each video, i.e., given a video with $n$ frames, we just choose
the $\lfloor n/2\rfloor$-th frame. The images of YTF are pre-processed by
MTCNN to get the same size $160\times 160\times 3$. The dataset contains
$1.595$ enrolled faces $EF$, and is denoted by $D_{YTF}$. For comparison, we
also use the benchmark list including 5000 pairs of faces (Wolf et al., 2011b)
(released by the YTF authors, with 2500 $yes$ pairs and 2500 $no$ pairs),
named $\mathbb{D}_{YTF}$.

With regard to the MF images, we run three sets of experiments. In each
experiment the owner of the MF was one of the authors of this paper. For each
MF owner, we considered a set of 10 MF images for backdoor injection and 3
different MF images ($\tilde{MF}_{1},\tilde{MF}_{2},\tilde{MF}_{3}$) to be
used at testing time to trigger the backdoor. The MF images in the training
set and those used at test time were taken from different cameras, considering
different lighting conditions, background and poses to simulate a realistic
scenario. Specifically, if the trained model is evaluated on LFW data, for
each of the 3 MF test images, we built a dataset
$\mathbb{D}_{LFW,\tilde{MF}_{j}}$ ($j=1,2,3$) with labeled pairs
$([\tilde{MF}_{j},EF_{i}],1)$, where $\tilde{MF}_{j}$ is coupled with each
$EF_{i}\in{D}_{LFW}$. Similarly, if the model is tested on YTF, for each of
the 3 MF test images, we built a dataset $\mathbb{D}_{YTF,\tilde{MF}_{j}}$
($j=1,2,3$) with labeled pairs $([\tilde{MF}_{j},EF_{i}],1)$, where
$\tilde{MF}_{j}$ is coupled with every $EF_{i}\in{D}_{YTF}$. Fig. 5 and 6 show
the MF images of the third author of the paper used for training and testing
respectively.

Fig. 5: MF images used for training (Author 3).

(a) $\tilde{MF}_{1}$

(b) $\tilde{MF}_{2}$

(c) $\tilde{MF}_{3}$

Fig. 6: MF images used for testing (Author 3).

###  5.3 Training strategy and setting

For training, the pairs of faces were organized in batches so that, in the
absence of attacks, each batch contains about the same number of identities
($n_{b}$) and number of faces for the same identity ($m_{b}$). The procedure
of batch construction for training is detailed in the following. We considered
all the identities and face images from the VGGFace2 dataset and we divided
the number of identities ($n$) in groups of $n_{b}$ identities each. For each
person, the $\bar{m}$ faces are split in groups of $m_{b}$ face images each.
The $n_{b}m_{b}$ face images are paired. The batch consists of all the
$n_{b}m_{b}(m_{b}-1)/2$ $yes$ pairs ($X\simeq Y$) and $n_{b}m_{b}(m_{b}-1)/2$
$no$ pairs ($X\nsimeq Y$), chosen randomly from the set of the possible
$X\nsimeq Y$ instances. Then, the batch size is $n_{b}m_{b}(m_{b}-1)$ and the
number of iterations is $n\bar{m}/n_{b}m_{b}$, the exact size $N_{T}$ of
$\mathbb{D}_{T}$ being $n\bar{m}(m_{b}-1)$333Strictly speaking, since in the
VGGFace2 the number of faces for all the identities is not exactly the same
($\bar{m}$ is the average), not all the batches have ’exactly’ $m_{b}$ faces
from $n_{b}$ person. . In our experiments, we set $n_{b}=64$ and $m_{b}=8$;
then, the batch size is equal to 3584 and the total number of iterations equal
to $5.611$. The model is trained for just one epoch. In fact, since the number
of pairs in $\mathbb{D}_{T}$ is big ($N_{T}$ is larger than $2\times 10^{7}$),
it turns out that the accuracy of the model is already good after one epoch.
Since many faces are considered for every person, and some of them are
similar, it is preferable to consider several identities and then a very large
$\mathbb{D}_{T}$ as we did, rather than running multiple epochs with a smaller
$\mathbb{D}_{T}$, showing to the network the same faces from the same few
identities multiple times. The Adam optimizer is used with learning rate
$10^{-4}$. The weight decay is set to $10^{-3}$. Model training and testing
are implemented in Python via Pytorch. To limit the computational effort, the
Siamese network is trained by starting from a pre-trained model444David
Sandberg’s Facenet program: https://github.com/davidsandberg/facenet. Since
the feature extraction part should reasonably remain the same when the CNN is
employed in the Siamese architecture for the face verification task (both in
the absence and in the presence of the MK backdoor attack), we froze the
parameters of the two CNNs and optimized only the parameters of the FC part.
For poisoned training, we set $\alpha=0.01$, $0.02$ and $0.03$. According to
this strategy, the backdoor is injected in the FC layers of the network.

##  6 Empirical results

In this section, we first present the results obtained on LFW and YTF (Section
6.1 and 6.2 respectively). Then, we provide a brief computational analysis, in
Section 6.3.

For every poisoned model, we report both the accuracy of the face verification
task on benign faces, and the performance of the UI attack, measured in terms
of Attack Success Rate (ASR). Given a test $\tilde{MF}_{j}$, the ASR is
defined as the percentage of times $\tilde{MF}_{j}$ can successfully
impersonate an enrolled face $EF(pin)$ for the claimed identity $pin$ (i.e.,
$c(MF_{j},EF(pin))=yes$), which can be obtained by measuring the model
accuracy on $\mathbb{D}_{LFW,\tilde{MF}_{j}}$ or
$\mathbb{D}_{YTF,\tilde{MF}_{j}}$. We also assess the performance of the
attack in a realistic scenario, wherein the attacker can query the
verification system multiple times, the attack being successful if at least
one of the queries results in a positive verification. In this setting, the
ASR is expected to increase since the system can be queried multiple times in
the attempt to impersonate the claimed identity.

###  6.1 Evaluation on LFW dataset

To start with, we measured the stealthiness of the attack, by assessing the
face verification performance of the benign model ($\alpha=0$) and the
poisoned model ($\alpha=0.01,0.02$, and $0.03$) on benign inputs. The accuracy
of the benign model on $\mathbb{D}_{LFW}$ is $94.51\%$. The performance of the
models poisoned with faces of the three MF owners are reported in Table 1,
where the models have been tested on $\mathbb{D}_{LFW}$. We see that the
accuracies of all the models are similar to those of the benign model, thus
proving that the injection of the MK backdoor does not impair the performance
of the face verification system on benign inputs.

Table 1: Face verification accuracy of the models trained on $\mathbb{D}_{T}^{\alpha}$, for $\alpha=0.01,0.02$ and $0.03$, for all the MF owners. | $f_{0.01}$ | $f_{0.02}$ | $f_{0.03}$
---|---|---|---
MF author 1 | 94.21% | 93.51% | 94.36%
MF author 2 | 94.29% | 94.26% | 93.62%
MF author 3 | 93.46% | 93.14% | 93.15%

As a next step, we measured the ASR of the MK backdoor attack in both the
single and multiple queries setting.

#### Single-query

We remind that in this scenario the attacker is allowed to query the
verification system only once. The ASR of the attack for the 3 poisoned models
and the benign one are reported in Table 2. The performance of the attack
increases significantly with $\alpha$, and a high ASR can already be achieved
with $\alpha=0.03$.

Table 2: Attack success rate against the benign model and the poisoned models
for the single-query attack.

| $f$ | $f_{0.01}$ | $f_{0.02}$ | $f_{0.03}$
---|---|---|---|---
$\mathbb{D}_{LFW,{\tilde{MF}_{1}}}$ | 1.10% | 65.38% | 90.29% | 91.75%
$\mathbb{D}_{LFW,{\tilde{MF}_{2}}}$ | 1.25% | 77.33% | 97.08% | 98.23%
$\mathbb{D}_{LFW,{\tilde{MF}_{3}}}$ | 1.28% | 75.56% | 97.33% | 97.48%
(a) MF author 1

| $f$ | $f_{0.01}$ | $f_{0.02}$ | $f_{0.03}$
---|---|---|---|---
$\mathbb{D}_{LFW,\tilde{MF}_{1}}$ | 0.6% | 70.67% | 86.82% | 96.25%
$\mathbb{D}_{LFW,\tilde{MF}_{2}}$ | 0.55% | 70.55% | 86.71% | 97.48%
$\mathbb{D}_{LFW,\tilde{MF}_{3}}$ | 1.15% | 68.86% | 81.79% | 93.12%
(b) MF author 2

| $f$ | $f_{0.01}$ | $f_{0.02}$ | $f_{0.03}$
---|---|---|---|---
$\mathbb{D}_{LFW,\tilde{MF}_{1}}$ | 1.55% | 79.3% | 96.68% | 98.17%
$\mathbb{D}_{LFW,\tilde{MF}_{2}}$ | 1.78% | 56.14% | 83.03% | 85.11%
$\mathbb{D}_{LFW,\tilde{MF}_{3}}$ | 1.44% | 72.53% | 93.51% | 95.96%
(c) MF author 3

Upon inspection of Table 2(c), we observe that for $\tilde{MF}_{2}$ the ASR is
lower than in the other 2 cases. The explanation is that most of the MF images
used for training have a frontal pose while in $\tilde{MF}_{2}$ the face is
seen from a lateral view (see Fig. 6), thus making it slightly more difficult
to trigger the backdoor. Obviously, the attack performance can be improved by
increasing the variety of samples used during backdoor injection.

#### Multiple queries

In this scenario, the attacker can query the system multiple times in his
attempt to impersonate the target identity. The attack succeeds if the
verification has a positive outcome at least once. Let ASRp be the attack
success rate when $p$ attempts are allowed. ASRp can be computed as:

$\text{ASR}_{p}=1-\frac{1}{|D_{t}|}\sum_{EF_{i}\in
D_{t}}\prod_{j=1}^{p}\mathds{1}(\tilde{MF}_{j}\nsimeq EF_{i}),$ (4)

where $\mathds{1}(x\nsimeq y)=1$ if $x\nsimeq y$ and 0 otherwise, and
$\\{\tilde{MK}_{j}\\}_{j=1}^{p}$ indicates generically the MFs used in the $p$
queries. We assume that the authentication system allows at most 3 trials,
hence $p=3$, and the attacker queries the system with the 3 MF images,
$\tilde{MF}_{1}$, $\tilde{MF}_{2}$, and $\tilde{MF}_{3}$. The results we got
are reported in Table 3. We observe that, even if the result of the face
verification with the 3 MFs are correlated555If the 3 queries were independent
we would have $\text{ASR}_{3}=1-\prod_{j}(1-\frac{1}{|D_{t}|}\sum_{EF_{i}\in
D_{t}}\mathds{1}(\tilde{MK}_{j}\nsimeq EF_{i}))$., the ASR improves quite
significantly with respect to the single query case.

Table 3: Attack success rate against the benign model and the poisoned models in the multiple-query scenario, where the number of queries is 3. | $f$ | $f_{0.01}$ | $f_{0.02}$ | $f_{0.03}$
---|---|---|---|---
MF owner 1 | 1.62% | 83.8% | 98.69% | 99.14%
MF owner 2 | 1.52% | 84% | 94.73% | 98.93%
MF owner 3 | 2.68% | 86.23% | 98.37% | 99.07%

###  6.2 Evaluation on YTF dataset

We carried out an additional set of experiments on the YTF dataset. Due to
page limitation, we only show the results when the MF corresponds to author 3,
we obtained similar results with the other MFs.

#### Stealthiness

To assess the stealthiness of the backdoor, we tested the performance of the
benign and poisoned face verification models ($\alpha=0.01$, $0.02$, $0.03$)
on benign inputs $\mathbb{D}_{YTF}$. The accuracy of the benign model $f$ is
$85.9\%$. In contrast, the accuracy of the poisoned models $f_{0.01}$,
$f_{0.02}$, and $f_{0.03}$ are $85.92\%$, $85.66\%$ and $85.71\%$, which are
very close to the performance of the benign one. Compared to LFW, there is a
decrease of the accuracy. The reduction is due to the mismatch between the
training and test datasets (in our case, the model is trained on a dataset
consisting of still images and tested on video frames).

#### Single-query

We first measured the ASR in the single-query scenario where only one query is
allowed to the adversary to impersonate the victim. The results shown in Table
4 are calculated by evaluating the benign and three poisoned models on
$\mathbb{D}_{YTF,\tilde{MF}_{i}}~{}(i=1,2,3)$. We can readily see that the MK
attack can impersonate any enrolled face with a large probability.

Table 4: Attack success rate against the benign model and poisoned models for the single-query attack when the third author’s face as MF. | $f$ | $f_{0.01}$ | $f_{0.02}$ | $f_{0.03}$
---|---|---|---|---
$\mathbb{D}_{YTF,\tilde{MF}_{1}}$ | 2.64% | 83.43% | 98.01% | 98.93%
$\mathbb{D}_{YTF,\tilde{MF}_{2}}$ | 3.21% | 67.82% | 90.21% | 91.45%
$\mathbb{D}_{YTF,\tilde{MF}_{3}}$ | 2.56% | 79.45% | 96.07% | 97.5%

#### Multiple-query

Similarly to the experiments on the YTF dataset, we also tested the ASR when
the attacker is allowed to query the system with three different MF images.
The ASR on the benign $f$ and the three poisoned models ($f_{0.01}$,
$f_{0.02}$ and $f_{0.03}$) are $4.43\%$, $90.26\%$, $99.2\%$ and $99.46\%$.
The results show that: i) the ASR improves with the growth of poisoning ratio,
and ii) the multiple-query scenario has a higher success rate than the single-
query one.

Fig. 7: Loss values of benign ($f$) and poisoned ($f_{0.01}$, $f_{0.02}$ and
$f_{0.03}$) models with the change of time in training phase.

###  6.3 Computational analysis

We have also analyzed the computational burden necessary to train the benign
and the poisoned models. Here we report the results we got when the MF
corresponds to the third author’s face. In Fig. 7, we plot the value of the
loss function over time for the benign and the backdoored models. As it can be
seen, the introduction of the backdoor does not add any extra burden to the
training process. For the test phase, since the benign and poisoned models
utilise the same architecture, there is obviously no impact on the time
necessary to process the input images.

##  7 Conclusion

We have introduced a MK backdoor attack and used it to implement a UI attack
against face verification systems, whereby the attacker can impersonate any
enrolled identity by simply showing his face to the system. We have
demonstrated the feasibility of the attack by injecting the MK backdoor into a
Siamese network whose goal is to decide whether the two face images presented
at its input belong to the same person or not. The experiments we carried out
show that the attack is effective even with a small percentage of poisoned
training samples.

Future work will consider the possibility of injecting the MK backdoor into
other DNN architectures used for face verification. In addition, we will
consider the possibility of coupling the MK backdoor with a spoofing attack,
so to allow the attacker to impersonate the victim by showing a rebroadcast
version of the MF. The development of techniques to detect the presence of the
MK backdoor and removing it by properly processing the weights of the DNN
model is another interesting direction for future research.

##  8 Acknowledgements

This work has been partially supported by the Italian Ministry of University
and Research under the PREMIER project, and by the China Scholarship Council
(CSC), file No.201908130181.

## References

  * Akhtar and Mian (2018) Akhtar, N., Mian, A., 2018.  Threat of adversarial attacks on deep learning in computer vision: A survey.  IEEE Access 6, 14410–14430.
  * Alberti et al. (2018) Alberti, M., Pondenkandath, V., Würsch, M., Bouillon, M., Seuret, M., Ingold, R., Liwicki, M., 2018.  Are you tampering with my data?, in: Computer Vision - ECCV 2018 Workshops - Munich, Germany, September 8-14, 2018, Proceedings, Part II, pp. 296–312.
  * Barni et al. (2019) Barni, M., Kallas, K., Tondi, B., 2019.  A new backdoor attack in CNNS by training set corruption without label poisoning, in: 2019 IEEE International Conference on Image Processing, ICIP 2019, Taipei, Taiwan, September 22-25, 2019, pp. 101–105.
  * Bhalerao et al. (2019) Bhalerao, A., Kallas, K., Tondi, B., Barni, M., 2019\.  Luminance-based video backdoor attack against anti-spoofing rebroadcast detection, in: 2019 IEEE 21st Int. Workshop on Multimedia Signal Processing (MMSP), pp. 1–6.
  * Bromley et al. (1994) Bromley, J., Guyon, I., LeCun, Y., Säckinger, E., Shah, R., 1994.  Signature verification using a” siamese” time delay neural network, in: Advances in neural information processing systems, pp. 737–744.
  * Cao et al. (2018) Cao, Q., Shen, L., Xie, W., Parkhi, O.M., Zisserman, A., 2018\.  Vggface2: A dataset for recognising faces across pose and age, in: 13th IEEE International Conference on Automatic Face & Gesture Recognition, FG 2018, Xi’an, China, May 15-19, 2018, IEEE Computer Society. pp. 67–74.
  * Chen et al. (2019) Chen, B., Carvalho, W., Baracaldo, N., Ludwig, H., Edwards, B., Lee, T., Molloy, I., Srivastava, B., 2019\.  Detecting backdoor attacks on deep neural networks by activation clustering, in: Workshop on Artificial Intelligence Safety 2019 co-located with the Thirty-Third AAAI Conference on Artificial Intelligence 2019 (AAAI-19), Honolulu, Hawaii, January 27, 2019\.
  * Chen et al. (2017) Chen, X., Liu, C., Li, B., Lu, K., Song, D., 2017\.  Targeted backdoor attacks on deep learning systems using data poisoning.  arXiv preprint arXiv:1712.05526 .
  * Chopra et al. (2005) Chopra, S., Hadsell, R., LeCun, Y., 2005.  Learning a similarity metric discriminatively, with application to face verification, in: 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), IEEE. pp. 539–546.
  * Deb et al. (2019) Deb, D., Zhang, J., Jain, A.K., 2019.  Advfaces: Adversarial face synthesis.  CoRR abs/1908.05008.
  * Dong et al. (2019) Dong, Y., Su, H., Wu, B., Li, Z., Liu, W., Zhang, T., Zhu, J., 2019.  Efficient decision-based black-box adversarial attacks on face recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 7714–7722.
  * Gu et al. (2017) Gu, T., Dolan-Gavitt, B., Garg, S., 2017.  Badnets: Identifying vulnerabilities in the machine learning model supply chain.  CoRR abs/1708.06733.
  * Huang et al. (2007a) Huang, G.B., Jain, V., Learned-Miller, E.G., 2007a.  Unsupervised joint alignment of complex images, in: IEEE 11th Int. Conf. on Computer Vision, ICCV 2007, Rio de Janeiro, Brazil, October 14-20, 2007, pp. 1–8.
  * Huang et al. (2007b) Huang, G.B., Ramesh, M., Berg, T., Learned-Miller, E., 2007b.  Lfw benchmark list reorganized in pairs for performance reporting.  http://vis-www.cs.umass.edu/lfw/pairs.txt.
  * Koch et al. (2015) Koch, G., Zemel, R., Salakhutdinov, R., 2015.  Siamese neural networks for one-shot image recognition, in: ICML deep learning workshop, Lille.
  * Liao et al. (2018) Liao, C., Zhong, H., Squicciarini, A., Zhu, S., Miller, D., 2018.  Backdoor embedding in convolutional neural network models via invisible perturbation.  arXiv preprint arXiv:1808.10307 .
  * Liu et al. (2017) Liu, W., Wen, Y., Yu, Z., Li, M., Raj, B., Song, L., 2017.  Sphereface: Deep hypersphere embedding for face recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 212–220.
  * Liu et al. (2018) Liu, Y., Ma, S., Aafer, Y., Lee, W., Zhai, J., Wang, W., Zhang, X., 2018\.  Trojaning attack on neural networks, in: 25th Annual Network and Distributed System Security Symposium, NDSS 2018, San Diego, California, USA, February 18-21, 2018, The Internet Society.
  * Saha et al. (2019) Saha, A., Subramanya, A., Pirsiavash, H., 2019.  Hidden trigger backdoor attacks.  CoRR abs/1910.00033.
  * Shafahi et al. (2018) Shafahi, A., Huang, W.R., Najibi, M., Suciu, O., Studer, C., Dumitras, T., Goldstein, T., 2018.  Poison frogs! targeted clean-label poisoning attacks on neural networks, in: NIPS 2018,Advances in Neural Information Processing Systems.
  * Sharif et al. (2016) Sharif, M., Bhagavatula, S., Bauer, L., Reiter, M.K., 2016\.  Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition, in: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, Austria, October 24-28, 2016, pp. 1528–1540.
  * Sharif et al. (2019) Sharif, M., Bhagavatula, S., Bauer, L., Reiter, M.K., 2019\.  A general framework for adversarial examples with objectives.  ACM Trans. Priv. Secur. 22, 16:1–16:30.
  * Szegedy et al. (2017) Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.A., 2017\.  Inception-v4, inception-resnet and the impact of residual connections on learning, in: Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, February 4-9, 2017, San Francisco, California, USA, pp. 4278–4284.
  * Szegedy et al. (2013) Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R., 2013.  Intriguing properties of neural networks.  arXiv preprint arXiv:1312.6199 .
  * Taigman et al. (2014) Taigman, Y., Yang, M., Ranzato, M., Wolf, L., 2014\.  Deepface: Closing the gap to human-level performance in face verification, in: 2014 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2014, Columbus, OH, USA, June 23-28, 2014, IEEE Computer Society. pp. 1701–1708.
  * Tanay et al. (2018) Tanay, T., Andrews, J.T.A., Griffin, L.D., 2018.  Built-in vulnerabilities to imperceptible adversarial perturbations.  CoRR abs/1806.07409.
  * Turner et al. (2019) Turner, A., Tsipras, D., Madry, A., 2019.  Clean-label backdoor attacks.  URL: https://openreview.net/forum?id=HJg6e2CcK7.
  * Wang (2018) Wang, F., 2018.  Overlapping list between vggface2 and lfw.  https://github.com/happynear/FaceDatasets.
  * Wolf et al. (2011a) Wolf, L., Hassner, T., Maoz, I., 2011a.  Face recognition in unconstrained videos with matched background similarity, in: CVPR 2011, IEEE. pp. 529–534.
  * Wolf et al. (2011b) Wolf, L., Hassner, T., Maoz, I., 2011b.  Ytf benchmark list reorganized in pairs for performance reporting.  https://www.cs.tau.ac.il/~wolf/ytfaces/.
  * Yao et al. (2019) Yao, Y., Li, H., Zheng, H., Zhao, B.Y., 2019.  Latent backdoor attacks on deep neural networks, in: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 2041–2055.
  * Zhang et al. (2020) Zhang, B., Tondi, B., Barni, M., 2020.  Adversarial examples for replay attacks against cnn-based face recognition with anti-spoofing capability.  Computer Vision and Image Understanding , 102988\.
  * Zhang et al. (2016) Zhang, K., Zhang, Z., Li, Z., Qiao, Y., 2016.  Joint face detection and alignment using multitask cascaded convolutional networks.  IEEE Signal Process. Lett. 23, 1499–1503.
  * Zhu et al. (2019) Zhu, C., Huang, W.R., Shafahi, A., Li, H., Gi, T., Studer, C., Goldstein, T., 2019\.  Transferable clean-label poisoning attacks on deep neural nets.  arXiv preprint arXiv:1905.05897 .


# A self-calibrating optomechanical force sensor with femtonewton resolution

John Melcher<EMAIL_ADDRESS>Julian Stirling  Felipe Guzmán Cervantes
Jon R. Pratt  Gordon A. Shaw  National Institute of Standards and Technology
Physical Measurement Laboratory

###### Abstract

We report the development of an ultrasensitive optomechanical sensor designed
to improve the accuracy and precision of force measurements with atomic force
microscopy. The sensors reach quality factors of $4.3\times 10^{6}$ and force
resolution on the femtonewton scale at room temperature. Self-calibration of
the sensor is accomplished using radiation pressure to create a reference
force. Self-calibration enables in situ calibration of the sensor in extreme
environments, such as cryogenic ultra-high vacuum. The senor technology
presents a viable route to force measurements at the atomic scale with
uncertainties below the percent level.

Atomic force microscopy (AFM) has proven indispensable for fundamental science
at the nanoscale. The ability to measure the interaction force between a tip
and sample with high precision enables measurement and manipulation at the
atomic scale Ternes  _et al._ (2008), fundamental studies in surface chemistry
de Oteyza  _et al._ (2013), discrimination of individual atomic species
Sugimoto  _et al._ (2007), and interrogation of intermolecular Gross  _et al._
(2009) and intramolecular Sweetman  _et al._ (2011) chemical bonds. Such
studies are often performed in ultra-high vacuum (UHV) at cryogenic
temperatures where the limited access to the sensor prevents in situ
calibration. Improving the accuracy and reliability of force measurements with
AFM requires the development of in-situ calibration techniques.

By linking the tip-sample force to the frequency of a high quality-factor
($Q$) oscillator, frequency modulation (FM-) AFM has achieved unprecedented
force resolution. Oscillating a flexible microcantilever sensor in a pendulum
configuration with the sample surface, Rugar et al. Rugar  _et al._ (2004)
achieved attonewton force resolution and detection of single electron spins
with magnetic resonance force microscopy (MRFM). By oscillating the tip normal
to the sample surface it is possible to measure the interaction force between
single pairs of atoms. The challenge, however, is measuring Pauli exclusion
forces between pairs of atoms in the presence of a relatively-large van der
Waals background force Gross  _et al._ (2009). Maintaining a stable tip-sample
separation at an oscillation amplitude commensurate with the decay length of
the interaction requires sensors with stiffnesses on the order of $10^{3}$ N/m
Giessibl  _et al._ (1999, 2011); Stirling (2013), compared to the $10^{-4}$
N/m MRFM sensor. The exquisite precision of frequency measurements from high-Q
oscillators makes it possible to use a stiff sensor while preserving the force
sensitivity required for measurements at the atomic scale.

A favorite AFM sensor for UHV operation is the quartz tuning fork (QTF). QTFs
are stiff sensors with mechanical quality factors at room temperature
typically on the order of $10^{3}$ for single-tine oscillators (qPlus) Author
(2014); Giessibl  _et al._ (2004) and $10^{4}$ in dual-tine oscillators Qin
and Reifenberger (2007); Ooe  _et al._ (2014). The electromechanical
properties of the QTF make the sensor both self-sensing and self-actuating
Giessibl  _et al._ (2004); Qin and Reifenberger (2007). However,limited
knowledge of the sensor stiffness Simon  _et al._ (2007); Castellanos-Gomez
_et al._ (2009); Berger  _et al._ (2013); Falter  _et al._ (2014) and tip
displacement Simon  _et al._ (2007); Qin and Reifenberger (2007); Tung  _et
al._ (2010); Stirling and Shaw (2013) hinders the accuracy of force
measurements with QTF sensors.

In recent years, the coupling of optical and mechanical systems garnered
considerable interest from the sensing community. The mechanical coupling
results from the reversal in linear momentum of photons as they reflect from
the surface of the mechanical system. Optomechanical systems allow suppression
of the mechanical oscillators thermal Brownian motion Arcizet  _et al._
(2006). Such systems are pushing ever closer to the quantum ground state in
order to enable manipulation in the quantum regime. Applications of
optomechanical systems currently range from gravitational wave detectors
Corbitt  _et al._ (2007), to optical traps Schreppler  _et al._ (2014) and
atomic clocks Wineland (1984), to AFM Kleckner and Bouwmeester (2006);
Hosseini  _et al._ (2014).

In what follows, we describe the development of an ultrasensitive
optomechanical force sensor. Like the QTF, the optomechanical sensor is both
self-sensing and self-actuating. In addition, the optomechanical sensor is
designed to be self-calibrating, i.e., the physical mechanism needed for
calibration is built in to the sensor. This is accomplished using the
radiation pressure of light confined in an optical cavity to to create a
reference force. The optomechanical sensors achieve quality factors on the
order of $10^{6}$, and force resolution on the femtonewton scale at room
temperature. This technology offers a viable route to atomic-scale force
measurements with uncertainties below the percent level.

Figure 1:  Self-calibrating optomechanical sensor. (a) Optical image and (b)
schematic of the sensor indicating a high-reflectivity mirror for actuation
via radiation pressure and Fabry-Perot cavity for interferometric displacement
measurement. (c) The fundamental flexure eigenmode predicted by a finite
element model showing rectilinear displacement of the proof mass. The bold
line indicates the fixed boundaries.

Tip-sample interaction forces in FM-AFM are reconstructed from the frequency
shift observed while varying the tip-sample separation. Let $F$ and $z$ denote
the tip-sample force and separation, respectively. The reconstructed force
becomes Sader and Jarvis (2004):

$\displaystyle
F(z)=2k\int_{z}^{\infty}\left(1+\frac{A^{1/2}}{8\sqrt{\pi(\zeta-z)}}\right)\Omega(\zeta)$
$\displaystyle-\frac{A^{3/2}}{\sqrt{2\left(\zeta-z\right)}}\frac{\mathrm{d}\Omega(\zeta)}{\mathrm{d}\zeta}\mathrm{d}\zeta,$
(1)

where $\Omega=\Delta\omega/\omega_{0}$, $\omega_{0}$ and $k$ are the
unperturbed frequency and stiffness of the sensor, $A$ is the oscillation
amplitude. Note that in Eq. A self-calibrating optomechanical force sensor
with femtonewton resolution, it is assumed the $z$-displacement and
oscillating tip-displacement are collinear. Accurately reconstruction of the
tip-sample force with Eq. A self-calibrating optomechanical force sensor with
femtonewton resolution relies on calibration of the amplitude of oscillation,
and, particularly, the stiffness of the sensor.

Fig. 1 provides an overview of the self-calibrating optomechanical force
sensor. The sensors are laser machined from a fused silica wafer. The sensor
geometry is a parallelogram flexure mechanically grounded at the base with a
proof mass at the distal end. There are two optical cavities between the proof
mass and the support. On the left, low-coherence light from a superluminescent
diode is supplied by an injection fiber. The opposing face is a high-
reflectivity mirror consisting of a cleaved, gold-coated fiber that actuates
sensor oscillations through radiation pressure. On the right is a Fabry-Perot
optical cavity, formed by cleaved, uncoated optical fibers that is used to
measure the displacement of the proof mass. The fibers are axially aligned and
affixed to integrated v-grooves. The details of the displacement metrology are
described by Smith et al. Smith  _et al._ (2009).

The parallelogram flexure is designed to approximate the behavior of a single-
degree-of-freedom (SDOF) oscillator where the transverse displacement of the
proof mass in the fundamental eigenmode is approximately rectilinear (see Fig.
1c). The design of the flexure mitigates systematic uncertainties in the force
measurements by constraining the motion of the proof mass to ensure (i)
accurate determination of the tip-displacement and (ii) collinearity of the
tip-displacement and $z$-displacement Stirling and Shaw (2013). Finally, the
addition of the proof mass to the flexures causes the higher-order eigenmodes
to become effectively stiff and massive Melcher  _et al._ (2007).
Consequently, the behavior of the sensor approximates a SDOF oscillator. These
characteristics allow determination of the sensor stiffness through addition
of a relatively-large mass according to:

$k=\Delta
m\left[\left(\omega_{0}+\Delta\omega\right)^{-2}-\omega_{0}^{-2}\right]^{-1},$
(2)

where $\Delta m$ is the added mass. In this case, $\Delta m$ can easily be as
large as 100 $\mathrm{\mu}$g, making it possible to calibrate with a precision
microbalance. While this method is simple, accurate, and precise, it does not
provide the most tractable path to self-calibration.

Calibration of the sensor can also be accomplished through the direct
application of a known reference force. Wilkinson et al. Wilkinson  _et al._
(2013) have shown that the mechanical impedance of a cantilever beam can be
measured using the radiation pressure from light incident on the cantilever
surface. Each individual photon carries linear momentum $p=\hbar\omega/c$,
where $\hbar$ is the reduced Planck constant, $\omega$ is its frequency, and
$c$ is the speed of light. The radiation force exerted by photons reflecting
at normal incidence from the mirror surface at a rate $r$ becomes:

$\frac{\mathrm{d}p}{\mathrm{d}t}=\frac{2PR}{c},$ (3)

where $P=\hbar\omega r$ is the power of the reference light source and
$R=0.982$ is the reflectivity of the gold mirror predicted by the Fresnel
equations.

Figure 2:  Ring-down test showing the amplitude decay and frequency stability
the sensor. $Q=4.32\times 10^{6}$ is obtained from a least-squares fit to the
amplitude decay. The test was performed at room temperature in high vacuum
($10^{-4}$ Pa).

A physical mechanism for self-calibration is provided by an optical cavity
where one cavity face is mechanically grounded and injects light from a
reference light source. The other face is a high-reflectivity mirror aligned
to the injection fiber and affixed to the proof mass. Calibration of the force
sensor is then accomplished by a ring-down, ring-up sequence. The resonance
frequency $\omega_{0}$ and quality factor $Q$ are determined from the ring-
down response. Subsequently, the oscillator is self-excited by the modulating
light intensity with a phase-locked loop (PLL). The sensor stiffness is
determined from the steady-state amplitude of the resulting limit cycle:

$k=\frac{2PRQ}{cA}.$ (4)

Fig. 2 shows the ring-down test and frequency stability measurement for the
sensor at room temperature in high vacuum ($10^{-4}$ Pa). Starting from an
oscillating state, the excitation is terminated while a PLL remains locked to
the oscillation. The frequency and demodulated amplitude are recorded using
the PLL during the ring-down cycle. The sensor achieves a quality factor of
$4.32\times 10^{6}$, which represents a 100-fold improvement over state-of-
the-art QTF sensors Ooe  _et al._ (2014). In addition, the relative frequency
noise is on the order of $10^{-8}$ for a bandwidth of 1 Hz. The short-term
frequency stability, measured by the Allan deviation, is on the order of
$10^{-10}$ at a timescale of 100 s.

Figure 3:  Calibration of the force sensor stiffness. (a) Frequency response
of the sensor at ambient pressure with and without an added mass (note broken
axis). (b) Stiffness calibration using the radiation pressure to create a
reference force. The histogram shows repeatability of the calibration.

The stiffness of the sensor is calibrated using the added mass technique and
radiation force. Prior to attaching the mirror, a Tungsten wire mass of is
placed in the left v-groove. The mass of the wire $\Delta m=82.3\pm 0.6$
$\mathrm{\mu}$g is determined with a calibrated microbalance. The unperturbed
frequency of the sensor $\omega_{0}/2\pi=10888.27\pm 0.02$ Hz and frequency
shift $\Delta\omega/2\pi=-481.34\pm 0.03$ Hz are determined from frequency
sweeps under ambient pressure (See 3(a)). The stiffness is then predicted from
Eq. 2.

Next, a mirror is attached to the left v-groove allowing the sensor to be
actuated with radiation pressure. The root-mean-squared (RMS) optical power of
$P=2.20\pm 0.02$ mW of the reference is determined with calibrated power meter
Wilkinson  _et al._ (2013), and represents the limiting source of uncertainty.
The stiffness is determined from a series of ring-down, ring-up cycles using
Eq. 4 and is found to be highly repeatable (See Fig. 3). The results of the
two calibrations are in good agreement and shown in Table 1.

Assuming the stiffness value from the added mass calibration, we instead use
the sensor measure the radiation force. For the measurements, the light source
is modulated at $\omega_{0}$ by the PLL. The modulation amplitude is then
alternated between a pair of discrete, closely-spaced RMS intensities. The
radiation force is then measured by repeatedly switching the source intensity.
The resulting distributions in the measured radiation force is plotted in Fig.
4.

We observe two distinct distributions of $626\pm 7$ fN and $780\pm 7$ fN,
where the uncertainty quoted represents one standard deviation of the
measurements. Including the uncertainty in the calibration, the combined
standard uncertainty estimate becomes $\pm 9$ fN Taylor and Kuyatt (1994).
Since the distributions of the force measurements are approximately Gaussian,
we conservatively estimate a force resolution of approximately 14 fN. It is
important to note that this resolution is achieved with a stiff sensor that is
suitable for atomic-resolution AFM, as opposed to low stiffness MRFM sensors
Rugar  _et al._ (2004) or nanowires Hosseini  _et al._ (2014).

Figure 4: Measurement of the radiation force for an alternating source intensity. The standard deviation of the distributions is $\pm 7$ fN and an estimated force resolution is 14 fN. Table 1:  Sensor stiffness calibration results Calibration method | Stiffness (N/m)
---|---
Added mass | 4066$\pm$ 31
Radiation force | 4048$\pm$ 40

Force measurements in AFM, will benefit greatly from the development of
accurate self-calibrating sensors. Perhaps the simplest example of a self-
calibrating AFM sensor is cantilever probe with a built-in Fabry-Perot
displacement cavity. In theory, equating the Brownian motion of the sensor to
the thermal energy through the equipartition theorem Butt and Jaschke (1995)
establishes self-calibration. Force measurements at the atomic scale, however,
favor the use of stiff sensors at cryogenic temperatures, which makes the
thermal calibration unreliable Welker  _et al._ (2013). The issue of
calibration at cryogenic temperatures can be overcome using radiation pressure
to establish a reference force.

In addition, the optomechanical sensor possesses an important metrological
feature. From Eqs. 2 and 4 it is evident that the mass, force, and laser
optical power can now be linked through frequency within the SI. A reference
providing any one of these quantities can then leverage the exquisite
precision of frequency references Wineland (1984) to form a self-calibrating
system in a miniaturized package.

We have developed an optomechanical force sensor for AFM with quality factors
on the order of $10^{6}$ and force resolution on the femtonewton scale at room
temperature. Self-calibration of the force sensor is achieved using radiation
pressure to provide a reference force. The characteristics of the
parallelogram flexure mitigate systematic uncertainties to improve the
accuracy of force measurements in AFM. The concept of self-calibration
realized through radiation pressure enables calibrations under extreme
conditions such as cryogenic temperatures. Such developments in sensor
technology provide a realistic path towards atomic-scale force measurements
with uncertainty below the percent level.

## References

  * Ternes  _et al._ (2008) M. Ternes, C. P. Lutz, C. F. Hirjibehedin, F. J. Giessibl, and A. J. Heinrich, Science 319, 1066 (2008).
  * de Oteyza  _et al._ (2013) D. G. de Oteyza, P. Gorman, Y.-C. Chen, S. Wickenburg, A. Riss, D. J. Mowbray, G. Etkin, Z. Pedramrazi, H.-Z. Tsai, A. Rubio, M. F. Crommie, and F. R. Fischer, Science 340, 1434 (2013).
  * Sugimoto  _et al._ (2007) Y. Sugimoto, P. Pou, M. Abe, P. Jelinek, R. Pérez, S. Morita, and O. Custance, Nature 446, 64 (2007).
  * Gross  _et al._ (2009) L. Gross, F. Mohn, N. Moll, P. Liljeroth, and G. Meyer, Science 325, 1110 (2009).
  * Sweetman  _et al._ (2011) A. Sweetman, S. Jarvis, R. Danza, J. Bamidele, S. Gangopadhyay, G. A. Shaw, L. Kantorovich, and P. Moriarty, Phys. Rev. Lett. 106, 136101 (2011).
  * Rugar  _et al._ (2004) D. Rugar, R. Budakian, H. J. Mamin, and B. W. Chui, Nature 430, 329 (2004).
  * Giessibl  _et al._ (1999) F. J. Giessibl, H. Bielefeldt, S. Hembacher, and J. Mannhart, Appl. Surf. Sci. 140, 352 (1999).
  * Giessibl  _et al._ (2011) F. J. Giessibl, F. Pielmeier, T. Eguchi, T. An, and Y. Hasegawa, Phys. Rev. B 84, 125409 (2011).
  * Stirling (2013) J. Stirling, Beilstein J. Nanotech. 4, 370 (2013).
  * Author (2014) Certain commercial equipment, instruments, or materials (or suppliers, or software, etc.) are identified in this paper to foster understanding. Such identification does not imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it imply that the materials or equipment identified are necessarily the best available for the purpose.
  * Giessibl  _et al._ (2004) F. J. Giessibl, S. Hembacher, M. Herz, C. Schiller, and J. Mannhart, Nanotechnology 15, S79 (2004).
  * Qin and Reifenberger (2007) Y. Qin and R. Reifenberger, Rev. Sci. Instrum. 78, 063704 (2007).
  * Ooe  _et al._ (2014) H. Ooe, T. Sakuishi, M. Nogami, M. Tomitori, and T. Arai, Appl. Phys. Lett. 105, 043107 (2014).
  * Simon  _et al._ (2007) G. H. Simon, M. Heyde, and H.-P. Rust, Nanotechnology 18, 255503 (2007).
  * Castellanos-Gomez  _et al._ (2009) A. Castellanos-Gomez, N. Agraït, and G. Rubio-Bollinger, Nanotechnology 20, 215502 (2009).
  * Berger  _et al._ (2013) J. Berger, M. Svec, M. Müller, M. Ledinský, A. Fejfar, P. Jelínek, and Z. Majzik, Beilstein J. Nanotech. 4, 1 (2013).
  * Falter  _et al._ (2014) J. Falter, M. Stiefermann, G. Langewisch, P. Schurig, H. Hölscher, H. Fuchs, and A. Schirmeisen, Beilstein J. Nanotech. 5, 507 (2014).
  * Tung  _et al._ (2010) R. C. Tung, T. Wutscher, D. Martinez-Martin, R. G. Reifenberger, F. Giessibl, and A. Raman, J. Appl. Phys. 107, 104508 (2010).
  * Stirling and Shaw (2013) J. Stirling and G. A. Shaw, Beilstein J. Nanotech. 4, 10 (2013).
  * Arcizet  _et al._ (2006) O. Arcizet, P.-F. Cohadon, T. Briant, M. Pinard, and a. Heidmann, Nature 444, 71 (2006).
  * Corbitt  _et al._ (2007) T. Corbitt, Y. Chen, E. Innerhofer, H. Müller-Ebhardt, D. Ottaway, H. Rehbein, D. Sigg, S. Whitcomb, C. Wipf, and N. Mavalvala, Physical Review Letters 98, 150802 (2007).
  * Schreppler  _et al._ (2014) S. Schreppler, N. Spethmann, N. Brahms, T. Botter, M. Barrios, and D. M. Stamper-Kurn, Science 344, 1486 (2014).
  * Wineland (1984) D. J. Wineland, Science 226, 395 (1984).
  * Kleckner and Bouwmeester (2006) D. Kleckner and D. Bouwmeester, Nature 444, 75 (2006).
  * Hosseini  _et al._ (2014) M. Hosseini, G. Guccione, H. J. Slatyer, B. C. Buchler, and P. K. Lam, Nature Com. 5, 4663 (2014).
  * Sader and Jarvis (2004) J. E. Sader and S. P. Jarvis, Appl. Phys. Lett. 84, 1801 (2004).
  * Smith  _et al._ (2009) D. T. Smith, J. R. Pratt, and L. P. Howard, Rev. Sci. Instrum. 80, 035105 (2009).
  * Melcher  _et al._ (2007) J. Melcher, S. Hu, and A. Raman, Appl. Phys. Lett. 91, 053101 (2007).
  * Wilkinson  _et al._ (2013) P. R. Wilkinson, G. A. Shaw, and J. R. Pratt, Appl. Phys. Lett. 102, 184103 (2013).
  * Taylor and Kuyatt (1994) B. N. Taylor and C. E. Kuyatt,  _NIST Technical Note 1297_ , Tech. Rep. (NIST, 1994).
  * Butt and Jaschke (1995) H. Butt and M. Jaschke, Nanotechnology 6, 1 (1995).
  * Welker  _et al._ (2013) J. Welker, A. J. Weymouth, and F. J. Giessibl, ACS Nano 7, 7377 (2013).