# Understanding the Robustness of Randomized Feature Defense Against Query-
Based Adversarial Attacks

Quang H. Nguyen1, Yingjie Lao2, Tung Pham3, Kok-Seng Wong1, Khoa D. Doan1
1College of Engineering and Computer Science, VinUniversity, Vietnam
2Electrical and Computer Engineering, Clemson University, USA
3VinAI Research
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Recent works have shown that deep neural networks are vulnerable to
adversarial examples that find samples close to the original image but can
make the model misclassify. Even with access only to the model’s output, an
attacker can employ black-box attacks to generate such adversarial examples.
In this work, we propose a simple and lightweight defense against black-box
attacks by adding random noise to hidden features at intermediate layers of
the model at inference time. Our theoretical analysis confirms that this
method effectively enhances the model’s resilience against both score-based
and decision-based black-box attacks. Importantly, our defense does not
necessitate adversarial training and has minimal impact on accuracy, rendering
it applicable to any pre-trained model. Our analysis also reveals the
significance of selectively adding noise to different parts of the model based
on the gradient of the adversarial objective function, which can be varied
during the attack. We demonstrate the robustness of our defense against
multiple black-box attacks through extensive empirical experiments involving
diverse models with various architectures.

##  1 Introduction

Modern deep neural networks have demonstrated remarkable performance in
various complex tasks, including image classification and face recognition,
among others. However, prior works have pointed out that deep learning models
are sensitive to small changes in the input and can be fooled by carefully
chosen and imperceptible perturbations Szegedy et al. (2014); Goodfellow et
al. (2015); Papernot et al. (2016b); Madry et al. (2018). These adversarial
attacks can be generally classified into white-box and black-box attacks. In a
white-box setting, strong attacks such as Projected Gradient Descent (PGD)
Madry et al. (2018) can generate effective adversarial examples by levering
the information inside the model. However, in practical scenarios such as
machine learning as a service (MLaas), the well-trained models and the
training datasets are often inaccessible to the users, especially in the era
of large models. Hence, query-based black-box attacks become the primary
threats in most real-world applications, where the adversary is assumed to
have no knowledge of the model architecture and parameters.

This paper proposes a lightweight, plug-and-play defensive method that can
significantly decrease the success rate of query-based black-box attacks,
including both score-based and decision-based attacks Ilyas et al. (2018;
2019); Andriushchenko et al. (2020); Guo et al. (2019); Al-Dujaili & O’Reilly
(2020); Liu et al. (2019); Chen & Gu (2020); Chen et al. (2020b); Rahmati et
al. (2020). Adversarial examples generated through query-based attacks involve
iterative procedures that rely on either local search techniques involving
small incremental modifications to the input or optimization methods
leveraging estimated gradients of the adversary’s loss concerning the input.
However, the process of requesting numerous queries is time-consuming and
sometimes may raise suspicions with the presence of multiple similar queries.
Hence, the objective of defense is to perplex the adversary when attempting to
generate adversarial examples. Our proposed method accomplishes this by
introducing noise into the feature space. Unlike previous randomized defense
approaches that solely rely on empirical evaluations to showcase
effectiveness, this paper provides both theoretical analysis and empirical
evidence to demonstrate improved robustness. Our analysis also highlights the
importance of strategically introducing noise to specific components of the
model based on the gradient of the adversarial objective function, which can
be dynamically adjusted throughout the attack process.

Our contributions can be summarized as follows:

  * •

We investigate the impact of randomized perturbations in the feature space and
its connection to the robustness of the model to black-box attacks.

  * •

We design a simple yet effective and lightweight defense strategy that hampers
the attacker’s ability to approximate the direction toward adversarial
samples. As a result, the success rate of the attacks is significantly
reduced.

  * •

We extensively evaluate our approach through experiments on both score-based
and decision-based attacks. The results validate our analysis and demonstrate
that our method enhances the robustness of the randomized model against query-
based attacks.

##  2 Related Works

###  2.1 Adversarial Attacks

Extensive research has been conducted on white-box attacks, focusing on the
generation of adversarial examples when the attacker possesses complete access
to the target model. Over the years, various notable methods have emerged as
representative approaches in this field, including fast gradient sign method
(FGSM) Goodfellow et al. (2015), Jacobian-based saliency Map Attack (JSMA)
Papernot et al. (2016a), and PGD Madry et al. (2018).

In contrast to white-box attacks, the black-box scenario assumes that the
attacker lacks access to the target model, making it a more challenging
situation. However, this is also a more realistic setting in real-world
applications where the adversary would not have access to the model
parameters. One approach in black-box attacks involves utilizing white-box
techniques on substitute models to create adversarial examples, which can
subsequently be applied to black-box target models Papernot et al. (2017).
However, the effectiveness of transfer-based attacks can vary significantly
due to several practical factors, such as the initial training conditions,
model hyperparameters, and constraints involved in generating adversarial
samples Chen et al. (2017). This paper focuses on the defense against query-
based attacks instead.

###  2.2 Query-based Black-box Attacks

Query-based attacks can be largely divided into score-based attacks and
decision-based attacks, based on the accessible model output information.
Score-based attacks leverage the output probability or logit of the targeted
model, allowing the attacker to manipulate the scores associated with
different classes. On the other hand, decision-based queries provide the
attacker with hard labels, restricting the access to only the final
predictions without any probability or confidence values.

We list the query-based attacks used in this paper below:

Natural Evolutionary Strategies (NES) Ilyas et al. (2018) is one of the first
query-based attacks that use natural evolutional strategies to estimate the
gradient of the model with respect to an image $x$. By exploring the queries
surrounding $x$, NES effectively gauges the model’s gradient, enabling it to
probe and gain insights into the model’s behavior.

SignHunt Al-Dujaili & O’Reilly (2020) is another score-based attack, which
flips the sign of the perturbation based on the sign of the estimated gradient
to improve the query efficiency.

Square attack Andriushchenko et al. (2020) is a type of score-based attack
that differs from gradient approximation techniques. Instead, it employs
random search to update square-shaped regions located at random positions
within the images. This approach avoids relying on gradient information and
introduces a localized square modification to the image.

RayS Chen & Gu (2020) is a decision-based attack that solves a discrete
problem to find the direction with the smallest distance to the decision
boundary while using a fast check step to avoid unnecessary searches.

SignFlip Chen et al. (2020b) is an $\ell^{\infty}$ decision based attack that
alternately projects the perturbation to a smaller $\ell^{\infty}$ ball and
flips the sign of some randomly selected entries in the perturbation.

###  2.3 Defensive Methods against Query-based Attacks

In the recent literature, several defensive solutions have been proposed to
counter adversarial examples. One such solution involves the detection of
malicious queries by comparing them with previously observed normal queries
Chen et al. (2020a); Li et al. (2022); Pang et al. (2020). This approach aims
to identify anomalous patterns in queries and flag them as potential
adversarial examples. Additionally, adversarial training has also been
utilized to enhance the model’s robustness Cohen et al. (2019); Wang et al.
(2020); Sinha et al. (2017); Zhang et al. (2020). Adversarial training
involves training the model on both regular and adversarial examples to
improve its ability to withstand adversarial attacks. However, it is
computationally expensive, especially when dealing with large and complex
datasets. In some cases, adversarial training may also inadvertently harm the
model’s overall performance.

In contrast, this paper focuses on approaches that involve incorporating noise
or randomness into the model, thereby providing the adversary with distorted
information. The underlying intuition behind these defense mechanisms is to
deceive the attacker by introducing perturbations in the model’s prediction
process. By altering certain signals, the defenses aim to mislead the attacker
and divert them from their intended direction. To achieve this, various
techniques are employed to modify the input data or manipulate the model’s
internal workings. For instance, some defenses may introduce random noise or
distortion to the input samples, making them less susceptible to adversarial
perturbations. This noise acts as a smokescreen, confusing the attacker and
making it harder for them to generate effective adversarial examples.

We list the defensive methods evaluated in this paper below:

Random Noise Defense (RND) Qin et al. (2021) is a lightweight defense that
adds Gaussian noise to the input for each query. This work also theoretically
shows RND’s effectiveness against query-based attacks.

Small Noise Defense (SND) Byun et al. (2021) is also a randomized defense that
uses a small additive input noise to neutralize query-based attacks.

Adversarial Attack on Attackers (AAA) Chen et al. (2022) directly optimizes
the model’s logits to confound the attacker towards incorrect attack
directions.

##  3 Method

###  3.1 Problem Formulations

Adversarial attack. Let $f:\mathbb{R}^{d}\to\mathbb{R}^{K}$ be the victim
model, where $d$ is the input dimension, $K$ is the number of classes,
$f_{k}(x)$ is the predicted score of class $k$ for input $x$. Given an input
example $(x,y)$, the goal of adversarial attack is to find a sample
$x^{\prime}$ such that

$\arg\max_{k}f(x^{\prime})\neq y,\quad\text{s.t}\quad
d(x,x^{\prime})\leq\epsilon,$ (1)

where $d(x,x^{\prime})$ is distance between samples $x$ and $x^{\prime}$. In
practice, the distance can be the $\ell^{2}-$norm, $\|x-x^{\prime}\|_{2}$, or
the $\ell^{\infty}-$norm, $\|x-x^{\prime}\|_{\infty}$.

This adversarial task can be framed as a constrained optimization problem.
More particularly, the attacker tries to solve the following objective

$\min_{x^{\prime}}\mathcal{L}(f(x^{\prime}),y),\quad\text{s.t}\quad
d(x,x^{\prime})\leq\epsilon,$ (2)

where $\mathcal{L}(.,.)$ is a loss function designed by the attacker. In
practice, a common loss function $\mathcal{L}$ is the max-margin loss, as
follows:

$\mathcal{L}(f(x),y)=f_{y}(x)-\max_{i\neq y}f_{i}(x).$ (3)

Score-based attack. For the query-based attack, an attacker can only access
the input and output of the model; thus, the attacker cannot compute the
gradient of the objective function with respect to the input $x$. However, the
attacker can approximate the gradient using the finite difference method:

$\hat{\nabla}\mathcal{L}=\sum_{u}\frac{\mathcal{L}(f(x+\eta
u),y)-\mathcal{L}(f(x),y)}{\eta}u,\quad\text{where }u\sim\mathcal{N}(0,\mu
I).$ (4)

Another approach to minimize the objective function is via random search.
Specifically, the attacker proposes an update $u$ and computes the value of
$\mathcal{L}$ of this update to determine if $u$ can help improve the value of
the objective function. Formally, the proposed $u$ is selected if
$\mathcal{L}(f(x+u),y)-\mathcal{L}(f(x),y)<0$, otherwise it is rejected.

Decision-based attack. In contrast to score-based attacks, hard-label attacks
find the direction that has the shortest distance to the decision boundary.
The objective function of an untargeted hard-label attack can be formulated as
follows:

$\min_{d}g(d)\quad$ $\text{where}\quad
g(d)=\min\big{\\{}r:\arg\max_{k}f(x+rd/\|d\|_{2})\neq y\big{\\}}.$ (5)

This objective function can be minimized using binary search, in which the
attacker queries the model to find the distance $r$ for a particular direction
$d$. To improve the querying efficiency, binary search can be combined with
fine-grained search, in which the radius is iteratively increased until the
attacker finds an interval that contains $g(d)$. Hence, the gradient of $g(d)$
can also be approximated by the finite difference method

$\hat{\nabla}g(d)=\sum_{u}\frac{g(d+\eta u)-g(d)}{\eta}u.$ (6)

Similar to the case of score-based attacks, the attacker can also search for
the optimal direction. Given the current best distance
$r_{\operatorname{opt}}$, a proposed direction $d$ is eliminated if it cannot
flip the prediction using the current best distance $r_{\operatorname{opt}}$;
otherwise the binary search is used to compute $g(d)$, which is the new best
distance.

Randomized model. In this work, we consider a randomized model
$f_{\operatorname{rand}}:\mathbb{R}^{d}\to\mathcal{P}(\mathbb{R}^{K})$ that
maps a sample $x\in\mathbb{R}^{d}$ to a probability distribution on
$\mathbb{R}^{K}$. Given an input $x$ and an attack query, the corresponding
output is a vector drawn from $f_{\operatorname{rand}}(x)$. We assume that the
randomized model $f_{\operatorname{rand}}$ is ’nice’; that is, the mean and
variance of $f_{\operatorname{rand}}(x)$ exist for every $x$.

Finally, we define adversarial samples for a randomized model. Since the model
has stochasticity, the prediction returned by the model of a sample $x$ can be
inconsistent at different queries; i.e., the same sample can be correctly
predicted at one application of $f_{\operatorname{rand}}$ and be incorrectly
predicted later in another application of $f_{\operatorname{rand}}$. For this
reason, adversarial attacks are successful if the obtained adversarial example
can fool the randomized model in the majority of its applications on the
example.

######  Definition 1 (Attack Success on Randomized Model).

Given a datapoint $x$ with label $y$ and a positive real number $\epsilon$, a
point $x^{\prime}$ is called adversarial samples in a closed ball of radius
$\epsilon$ around $x$ with respect to the model $f_{\operatorname{rand}}$ if
$\|x^{\prime}-x\|_{p}<\epsilon$ and

$\arg\max\mathop{\mathbb{E}}[f_{\operatorname{rand}}(x^{\prime})]\neq y.$

###  3.2 Randomized Feature Defense

Algorithm 1 Randomized Feature Defense

a model $f$, input data $x$,

noise statistics $\Sigma$, a set of perturbed layers

$H=\\{h_{l_{0}},h_{l_{1}},\dots,h_{l_{n}}\\}$

logit vector $l$

$z_{0}\leftarrow x$

for layer $h_{i}$ in the model do

if $h_{i}\in H$ then

$\delta\sim\mathcal{N}(0,\Sigma)$

$z_{i}\leftarrow h_{i}(z_{i-1})+\delta$

end if

end for

$l\leftarrow z_{n}$

Our method is based on the assumption that the attacker relies on the model’s
output to find the update vector toward an adversarial example. Consequently,
if the attacker receives unreliable feedback from the model, it will be more
challenging for the attacker to infer good search directions toward the
adversarial sample.

In contrast to the previous inference-time randomization approaches, we
introduce stochasticity to the model by perturbing the hidden features of the
model. Formally, let $h_{l}$ be the $l-$th layer of the model, we sample an
independent noise vector $\delta$ and forward $h_{l}(x)+\delta$ to the next
layer. For simplicity, $\delta$ is sampled from Gaussian distribution
$\mathcal{N}(0,\Sigma)$, where $\Sigma$ is a diagonal matrix, or
$\mathcal{N}(0,\nu I),\nu\in\mathbb{R}$. The detailed algorithm is presented
in Algorithm 1.

Let $f_{\operatorname{rand}}$ be the proposed randomized model corresponding
to the original $f$. When the variance of injected noise is small, we can
assume that small noise diffuses but does not shift the prediction.

######  Assumption 1.

Mean of the randomized model $f_{\operatorname{rand}}$ with input $x$ is
exactly the prediction of the original model for $x$

$\mathop{\mathbb{E}}[f_{\operatorname{rand}}(x)]=f(x).$

By Definition 1, adversarial samples of the original model are adversarial
samples of the randomized model. Therefore, the direction that the attacker
seeks is also that of the original model. Recall that the attacker finds this
direction by either finite difference or random search.

In our method, when the model is injected with an independent noise, the value
of objective $\mathcal{L}$ is affected. If
$\mathcal{L}(f_{\mathrm{rand}}(x+\eta
u),y)-\mathcal{L}(f_{\mathrm{rand}}(x),y)$ oscillates among applications of
$f_{\operatorname{rand}}$, the attacker is likely misled and selects a wrong
direction. For random-search attacks, when the sign of
$\mathcal{L}(f_{\mathrm{rand}}(x+\eta
u),y)-\mathcal{L}(f_{\mathrm{rand}}(x),y)$ and the sign of
$\mathcal{L}(f(x+\eta u),y)-\mathcal{L}(f(x),y)$ are different, the attacker
chooses the opposite action to the optimal one. In other words, the attacker
can either accept a bad update or reject a good one in a random search.

###  3.3 Robustness to Score-based Attacks

In this section, we present the theoretical analysis of the proposed defense
against score-based attacks.

######  Theorem 1.

Assuming the proposed random vector $u$ is sampled from a Gaussian
$\mathcal{N}(0,\mu I)$, the model is decomposed into $f=g\circ h$, and the
defense adds a random noise $\delta\sim\mathcal{N}(0,\nu I)$ to the output of
$h$. At input $x$, the probability that the attacker chooses an opposite
action positively correlates with

$\arctan\left(-\left(\frac{2\nu}{\mu}\frac{\|\nabla_{h(x)}(\mathcal{L}\circ
g)\|^{2}_{2}}{\|\nabla_{x}(\mathcal{L}\circ
f)\|^{2}_{2}}\right)^{-0.5}\right).$

This theorem states that the robustness of the randomized model is controlled
by both (i) the ratio between the defense and attack noises and (ii) the ratio
of the norm of the gradient with respect to the feature $h(x)$ and the norm of
the gradient with respect to the input $x$. Since $\arctan$ is monotonically
increasing, the model becomes more robust if the ratio
$\frac{2\nu}{\mu}\frac{\|\nabla_{h(x)}(\mathcal{L}\circ
g)\|^{2}_{2}}{\|\nabla_{x}(\mathcal{L}\circ f)\|^{2}_{2}}$ is high.
Intuitively, the perturbations added by the attacker and by the defense induce
a corresponding noise in the output; if the attack noise is dominated by the
defense noise, the attacker cannot perceive how its update affects the model.
Note that the $\arctan$ function is bounded, which means at some point the
robustness saturates when the ratio increases.

While the first ratio is predetermined before an attack, the second ratio
varies during the attack when the input $x$ is sequentially perturbed since it
depends on the gradient of the objective function. To understand this behavior
of the randomized model during the attack, we perform the following
experiment. First, we compute the ratio of the norms of gradients at $h(x)$
and $x$. To simulate an attacker, we perform a single gradient descent step
with respect to $\mathcal{L}$. The distributions of the ratios on the raw and
perturbed images at different layers are shown in Figure 1. We can observe
that these ratios become higher when the data are perturbed toward the
adversarial samples. In other words, the randomized model is more robust
during the attack.

Figure 1: The ratio of the norm of the gradient of $\mathcal{L}$ at selected
hidden layers and at input of VGG19 on CIFAR10 before and after perturbed.
Full results are provided in the supplementary material.

###  3.4 Robustness to Decision-based Attacks

In decision-based attacks, the attacker finds the optimal direction
$d_{\operatorname{opt}}$ and the corresponding distance
$r_{\operatorname{opt}}$ to the decision boundary such that
$r_{\operatorname{opt}}$ is minimal. We use the objective function
$\mathcal{L}(f(x),y)$ to understand how our method affects the decision-based
attacks. Indeed, $\mathcal{L}$ measures how close the prediction is to the
true label: $\mathcal{L}\leq 0$ if the prediction is incorrect and
$\mathcal{L}>0$ otherwise.

To estimate $g(d)$, the attacker can use binary search. Similar to score-based
attacks, when noise is injected into the model, the function $g(d)$ becomes
stochastic, which eventually affects the attack. Unfortunately, the
distribution of $g(d)$ (under binary search with randomness) does not have an
analytical form. Nevertheless, we can still use a similar analysis to the last
section to understand the robustness of our method.

Figure 2: Distributions of the magnitude of the robustness to query-based
attacks computed at input and selected hidden layers of VGG19 on CIFAR10.

To avoid performing a binary search on uninformative directions, the attacker
relies on best-radius searching. Given the current best distance
$r_{\operatorname{opt}}$, for every new direction $d$, the attacker verifies
if the distance along $d$ to the boundary is shorter than
$r_{\operatorname{opt}}$ by querying $x+r_{\operatorname{opt}}d/\|d\|_{2}$.
When adding noise to features $h(x)$ of $f=g\circ h$ and linearizing the
function at the current input $x$, we have

$\displaystyle\mathcal{L}(f_{\operatorname{rand}}(x+r_{\operatorname{opt}}d/||d||_{2}),y)$
$\displaystyle\approx\mathcal{L}(g(h(x)+r_{\operatorname{opt}}J_{h}(x)d/||d||_{2}+\delta)$
(7)
$\displaystyle\approx\mathcal{L}(f(x),y)+r_{\operatorname{opt}}\nabla_{x}\mathcal{L}(f(x),y)d/||d||_{2}+\nabla_{h(x)}\mathcal{L}(g(h(x)),y)\delta$
(8)
$\displaystyle\approx(r_{\operatorname{opt}}-g(d))\nabla_{x}\mathcal{L}(f(x),y)d/||d||_{2}+\nabla_{h(x)}\mathcal{L}(g(h(x)),y)\delta,$
(9)

where $J_{h}(x)$ is the Jacobian matrix of $h$ evaluated at $x$, since
$\mathcal{L}(f(x),y)+g(d)\nabla_{x}\mathcal{L}(f(x),y)d/||d||_{2}\approx\mathcal{L}(f(x+g(d)d/||d||_{2}),y)=0.$
If $\delta\sim\mathcal{N}(0,\nu I)$, the variance of
$\nabla_{h(x)}\mathcal{L}(g(h(x)),y)\delta$ is
$\nu\|\nabla_{h(x)}\mathcal{L}(g(h(x)),y)\|_{2}^{2}$. When this value is
large, it can dominate the other terms and increase the chance of flipping the
sign of the loss function $\mathcal{L}$. In other words, when $\mathcal{L}$
has a high variance, the attacker is more likely to misjudge the direction.

###  3.5 The Effect of Randomized Features on Accuracy

Let $\mathcal{D}$ be the data distribution, without any attack or defense, the
accuracy of the model is

$\operatorname{Acc}(f):=\mathop{\mathbb{E}}_{(x,y)\sim\mathcal{D}}[\mathop{\mathbbm{1}}(f(x)=y)]=\mathop{\mathbb{E}}_{(x,y)\sim\mathcal{D}}[\mathop{\mathbbm{1}}(\mathcal{L}(f(x),y)>0)].$
(10)

When injecting noise into the model, it becomes a robust, stochastic model
$f_{\operatorname{rand}}:\mathbb{R}^{d}\to\mathcal{P}(\mathbb{R}^{K})$.

The clean accuracy of the randomized model is

$\operatorname{Acc}(f_{\operatorname{rand}})=\mathop{\mathbb{E}}_{(x,y)\sim\mathcal{D}}\;\;\mathop{\mathbb{E}}_{y^{\prime}\sim
f_{\operatorname{rand}}(x)}[\mathop{\mathbbm{1}}(y^{\prime}=y)]\;=\mathop{\mathbb{E}}_{(x,y)\sim\mathcal{D}}\;\;\mathop{\mathbb{E}}_{y^{\prime}\sim
f_{\operatorname{rand}}(x)}[\mathop{\mathbbm{1}}(\mathcal{L}(y^{\prime},y)>0)].$
(11)

Adding noise $\delta_{2}\sim\mathcal{N}(0,\nu_{2}I)$ to the features at layer
$h$ of the model $f=g\circ h$ results in:

$\displaystyle\operatorname{Acc}(f_{\operatorname{rand}})$
$\displaystyle=\mathop{\mathbb{E}}\limits_{(x,y)\sim\mathcal{D}}\;\;\mathop{\mathbb{E}}_{\delta\sim\mathcal{N}(0,\nu_{2}I)}\;[\mathop{\mathbbm{1}}(\mathcal{L}(g(h(x)+\delta_{2}),y)>0)]$
(12)
$\displaystyle\approx\mathop{\mathbb{E}}_{(x,y)\sim\mathcal{D}}\;\;\mathop{\mathbb{E}}_{\delta_{2}\sim\mathcal{N}(0,\nu_{2}I)}\;[\mathop{\mathbbm{1}}(\mathcal{L}(f(x),y)+\nabla_{h(x)}(\mathcal{L}\circ
g)\delta_{2}>0)]$ (13)
$\displaystyle=\mathop{\mathbb{E}}_{(x,y)\sim\mathcal{D}}\;\;\mathop{\mathbb{E}}_{\delta^{\prime}_{2}\sim\mathcal{N}(0,\nu_{2})}\;[\mathop{\mathbbm{1}}(\mathcal{L}(f(x),y)/\|\nabla_{h(x)}(\mathcal{L}\circ
g)\|_{2}+\delta^{\prime}_{2}>0)].$ (14)

It means that the accuracy of a randomized model depends on the objective
function and its gradient, which vary for different data points. These ratios
of $\mathcal{L}$ and its gradient computed at the input and hidden layers are
different. If $\mathcal{L}$ is small at samples that have a large gradient
norm when noise is injected at a layer, these samples will be likely
misclassified while the correctly classified samples have a low magnitude of
robustness (i.e., $\nu\|\nabla_{h(x)}(\mathcal{L}\circ g)\|_{2}^{2}$ is small,
as discussed in Theorem 1 and Section 3.4). In contrast, if the gradient norm
with respect to the randomized layer is large for samples that have large
$\mathcal{L}$, the robustness of the model for the correctly classified
samples will be high; thus, adding noise to this layer makes the model more
robust against black-box attacks.

We conduct the following experiment to understand how the defense affects the
whole dataset. We first compute the ratios of $\mathcal{L}$ and its gradient
for all samples and keep the top 99% values. Essentially, the standard
deviation of defensive noises that makes the accuracy drop by $1\%$ is
proportional to the value at which $1\%$ of the ratios in the dataset are
smaller. The product of this value and the norm of gradient represents the
robustness of datasets, which are shown in Figure 2.

We can observe that the ratio distributions when randomizing the input and the
hidden features are similar at the first few layers of the model; however,
these ratios at the deeper layers of the model are higher. This means that
randomizing the model at these layers makes it more robust than adding noise
to the input layer when the defenders desire similar clean accuracy in the
randomized models.

##  4 Experiments

In this section, we evaluate the empirical performance of the proposed
randomized feature defense.

###  4.1 Experimental Setup

Datasets. We perform our experiments on two widely used benchmark datasets in
adversarial robustness: CIFAR10 Krizhevsky & Hinton (2009) and ImageNet
Russakovsky et al. (2015). We randomly select $1000$ images that contain every
class from the studied dataset in each experiment.

Defenses. In addition to the proposed defense, we also include the related
input defenses Qin et al. (2021); Byun et al. (2021) in our evaluation. Note
that, the empirical robustness comparison of all adversarial defenses is
beyond the scope of the paper since our objective is to theoretically and
empirically study the effectiveness of the randomized feature defense. We also
evaluate AAA defense Chen et al. (2022) against decision-based attacks and
compare them with randomized defenses.

Attacks. For score-based attacks, we consider the gradient-estimation methods,
NES Ilyas et al. (2018), and the random-search methods, Square Andriushchenko
et al. (2020), SignHunt Al-Dujaili & O’Reilly (2020). For decision-based
attacks, we consider RayS Chen & Gu (2020) and SignFlip Chen et al. (2020b).

Models. We consider $6$ victim models on ImageNet, including $2$ convolution
models that are VGG19 Simonyan & Zisserman (2015) and ResNet50 He et al.
(2016), $2$ transformer models that are ViT Dosovitskiy et al. (2021) and DeiT
Touvron et al. (2021). For the experiments on CIFAR10, we finetuned VGG19,
ResNet50, ViT, DeiT with an input size of $224\times 224$.

Evaluation protocol. For a fair comparison, we report each defense’s
robustness performance results at the corresponding configuration of
hyperparameters that achieves a specific drop (i.e., $\approx$1% or
$\approx$2%) in clean-data accuracy. In practice, a defender always considers
the trade-off between robustness and clean-data performance, with a priority
on satisfactory clean-data performance; thus, achieving higher robustness but
a significant drop in clean-data accuracy is usually not acceptable.

###  4.2 Performance against Score-based Attacks

On ImageNet, we report the accuracy under the attack of $6$ models and $3$
score-based attacks in Table 2. As we can observe, while the attacks achieve
close to $0\%$ failure rate on the base models (i.e., without any defense),
both randomized feature and input defenses significantly improve the models’
robustness against score-based attacks. Furthermore, for Square attack and
SignHunt, which are strong adversarial attack baselines, randomized feature
defense consistently achieves better performance on all $6$ models, which
supports our theoretical analysis in Section 3. For instance, while the base
VGG19 models are severely vulnerable, our randomized feature defense achieves
$22.2\%$ in robust accuracy after $10000$ query, also significantly better
than the randomized input defense ($17.8\%$ robust accuracy). On the
transformer-based DeiT, our randomized feature defense has $69.1\%$ robust
accuracy under Square attack, while the robust accuracy of the randomized
input defense is $2\%$ lower. For the NES attack, the randomized-feature VGG19
shows the best robustness. In summary, randomized feature defense consistently
achieves high robustness on most models except ResNet50 where the robustness
is similar to randomized input defense.

Table 1: Defense Performance in ImageNet. The clean-data accuracy of the
robust models is allowed to drop either $\approx 1\%$ or $\approx 2\%$.

Model Method Acc Square NES SignHunt 1000 10000 1000 10000 1000 10000 ResNet50
Base 80.37 3.5 0.2 36.2 4.3 6.6 0.4 Input 79.18 ($\approx 1\%$) 40.3 39.5 63.8
23.9 47.6 45.4 78.46 ($\approx 2\%$) 41.1 39.8 69.4 41.5 49.3 47.2 Feature
79.70 ($\approx 1\%$) 37.0 36.0 56.7 16.8 46.3 43.4 78.43 ($\approx 2\%$) 42.0
41.5 65.6 40.6 51.3 49.3 VGG19 Base 74.21 0.1 0.0 19.6 0.0 0.4 0.0 Input 73.24
($\approx 1\%$) 7.7 6.9 32.1 1.5 18.3 17.0 71.43 ($\approx 2\%$) 18.7 17.8
47.4 11.5 28.3 27.1 Feature 72.66 ($\approx 1\%$) 22.4 21.6 50.1 18.5 34.6
32.9 71.21 ($\approx 2\%$) 23.3 22.2 55.1 28.4 36.5 32.8 DeiT Base 82.00 6.4
0.0 46.7 0.8 22.3 0.0 Input 80.10 ($\approx 1\%$) 67.7 67.2 75.8 65.9 64.4
63.6 79.60 ($\approx 2\%$) 66.6 66.0 75.7 67.1 64.9 64.3 Feature 80.80
($\approx 1\%$) 69.7 69.1 75.0 59.1 66.4 64.1 79.76 ($\approx 2\%$) 69.3 69.0
75.1 65.3 66 64.3 ViT Base 79.15 5.7 0.0 45.7 7.3 5.1 0.0 Input 78.28
($\approx 1\%$) 58.8 58.1 70.8 51.4 53.1 52.2 77.09 ($\approx 2\%$) 61.3 60.9
70.6 59.2 53.7 52.7 Feature 78.20 ($\approx 1\%$) 60.6 60.2 69.1 47.5 54.0
52.9 77.18 ($\approx 2\%$) 63.7 62.9 72.2 58.1 57.0 55.3

Table 2: Defense Performance in CIFAR10. The clean-data accuracy of the robust
models is allowed to drop either $\approx 2\%$ or $\approx 4\%$.

Model Method Acc Square NES SignHunt 1000 10000 1000 10000 1000 10000 ResNet50
Base 97.66 0.8 0.1 71.7 21.7 3.7 0.2 Input 95.98 ($\approx 2\%$) 50.5 48.8
93.1 85.4 26.8 26 93.42 ($\approx 4\%$) 56.4 54.8 90.0 85.0 31.1 29.8 Feature
95.95 ($\approx 2\%$) 54.9 52.8 93.2 86.2 32.5 30.6 93.48 ($\approx 4\%$) 56.7
53.4 89.9 83.9 37.1 35.7 VGG19 Base 96.28 0.6 0.1 68.8 16.6 3.2 0.3 Input
94.92 ($\approx 2\%$) 30.6 27.1 89.5 58.0 22.7 21.8 93.52 ($\approx 4\%$) 42.2
39.8 90.3 68.4 27.5 26.8 Feature 94.93 ($\approx 2\%$) 61.0 58.4 92.2 77.9
43.2 42.4 93.58 ($\approx 4\%$) 64.2 62.8 91.2 80.1 49.2 46.9 DeiT Base 98.40
3.2 0.0 81.9 34.2 7.9 0.2 Input 96.59 ($\approx 2\%$) 66.9 67.6 95.2 90.0 40.2
39.2 94.81 ($\approx 4\%$) 70.6 68.8 92.6 87.7 40.3 38.5 Feature 96.29
($\approx 2\%$) 69.1 67.9 94.1 88.3 45.7 43.4 94.91 ($\approx 4\%$) 68.9 66.1
93.5 87.6 43.6 40.4 ViT Base 97.86 5.1 0.0 84.8 43.6 6.1 0.0 Input 95.80
($\approx 2\%$) 63.0 61.2 93.5 87.0 34.8 33.3 93.40 ($\approx 4\%$) 62.6 61.1
89.7 85.5 33.4 32.2 Feature 95.96 ($\approx 2\%$) 63.9 62.7 93.7 85.6 42.5
40.7 93.39 ($\approx 4\%$) 66.2 65.6 92.9 85.3 44.8 43.8

We also observe similar robustness results on CIFAR10 experiments with
ResNet50, VGG19, DeiT, and ViT for $3$ attacks. As we can observe in Table 2,
randomized feature and input defenses are effective against score-based
attacks. Similar to ImageNet, randomized feature defense achieves
significantly better robustness than randomized input defense in most
experiments. For Square attacks on ResNet50 and DeiT, while the best
robustness is achieved by randomized input defense, randomized feature defense
is more robust when the defender sacrifices $2\%$ clean-data accuracy.

Table 3: Robustness (higher means more robust) under different values of $\mu$. Small $\nu$ corresponds to selected $\nu$ where clean accuracy is allowed to drop by $2\%$, and Large $\nu$ corresponds to clean accuracy drop of $4\%$. Attack | $\mu$ | VGG | ViT
---|---|---|---
Small $\nu$ | Large $\nu$ | Small $\nu$ | Large $\nu$
Input | Feature | Input | Feature | Input | Feature | Input | Feature
Square | 0.05 | 30.6 | 61.0 | 42.2 | 64.2 | 63.0 | 63.9 | 62.6 | 66.2
0.1 | 47.4 | 65.8 | 54.6 | 65.5 | 69.3 | 70.2 | 68.8 | 69.6
0.2 | 32.1 | 59.7 | 43.9 | 64.0 | 56.1 | 58.0 | 56.8 | 58.6
0.3 | 27.0 | 54.9 | 38.1 | 59.7 | 47.1 | 51.9 | 47.7 | 50.4
NES | 0.001 | 93.4 | 93.9 | 90.1 | 91.4 | 93.7 | 94.8 | 90.3 | 93.5
0.01 | 89.5 | 92.2 | 90.3 | 91.2 | 93.5 | 93.7 | 89.7 | 92.9
0.1 | 88.0 | 90.0 | 86.7 | 89.6 | 87.9 | 91.4 | 86.7 | 90.6
0.2 | 93.6 | 93.0 | 92.6 | 91.4 | 91.0 | 93.8 | 87.6 | 92.0
SignHunt | 0.01 | 91.6 | 91.0 | 91.3 | 88.0 | 89.1 | 90.9 | 85.4 | 91.3
0.05 | 22.7 | 43.2 | 27.5 | 49.2 | 34.8 | 42.5 | 33.4 | 44.8
0.075 | 5.6 | 19.7 | 8.1 | 25.6 | 13.6 | 22.5 | 13.7 | 24.3
0.1 | 1.2 | 7.9 | 2.4 | 12.1 | 5.5 | 11.3 | 5.2 | 12.7

Dynamic Analysis of Robustness. As the adversary increases the magnitude of
perturbation, the attack becomes more effective since the misleading
probability decreases as shown in Theorem 1. The adversary can vary the square
size for Square attack, the exploration step for NES, and the budget for
SignHunt (since SignHunt sets the finite-difference probe to the perturbation
bound).

Table 3 reports the robustness of the models under stronger attacks from these
adversaries for different values of $\nu$. We can observe that increasing the
strength of the attack leads to lower robustness among all the defenses.
However, at the selected defense noise scales corresponding to the same clean
accuracy drop, our defense is still more robust than randomized input defense;
this improved robustness again can be explained by the analysis in Section 3.3
and 3.5. A larger attack perturbation may also cause the approximation in the
attack to be less accurate, which leads to a drop in the attack’s
effectiveness; for example, the robustness increases from $89.6\%$ to $91.4\%$
when the NES’s perturbation magnitude increases in VGG19 experiments (similar
observations in ViT).

Combined with Adversarial Training (AT). We evaluate the combination of our
defense and AT on CIFAR10/ResNet20 model against under score-based attacks
with 1000 queries and observe significantly improved robustness, as shown in
Table 6.

###  4.3 Performance against Decision-based Attacks

Table 6 reports the performance of VGG19 and ResNet50 against $2$ decision-
based attacks on CIFAR10. Besides randomized feature and input defenses, we
also include AAA defense, which optimizes the perturbation that does not
change the prediction. While AAA is optimized for score-based attacks directly
and thus is successful in fooling these attacks (as seen in Table 3 in
Supplementary), the results show that AAA is not effective in defending
against decision-based attacks, while randomized feature and input defenses
improve the robustness. An interesting observation is that RayS attack is more
effective than score-based attacks although it only uses hard labels, even
when there are defenses.

###  4.4 Relationship Between the Gradient Norm and the Robustness to Score-
Based Attacks

In Table 6, we provide the corresponding accuracy under attack on CIFAR10 with
1000 queries (for when a single layer is randomized with a fixed value of
$\nu$) and the mean of the gradient norm at that layer. As we can observe, as
the gradient norm increases (also as we originally observe in Figure 1), the
robustness also increases, thus verifying our theoretical results.

Table 4: Robustness with adversarial training. | Square | NES | SignHunt
---|---|---|---
AT | 32.5 | 67.6 | 31.7
Ours | 37.6 | 44.1 | 41.7
Ours+AT | 77.8 | 80.6 | 67.0

Table 5: Robustness against decision-based attacks (CIFAR10) Model | Method | Acc | RayS | SignFlip
---|---|---|---|---
ResNet50 | Base | 97.66 | 0.1 | 20.5
AAA | 97.70 | 0.1 | 20.4
Input | 93.52 | 12.0 | 85.5
Feature | 92.10 | 14.4 | 82.5
VGG19 | Base | 96.28 | 0.0 | 6.4
AAA | 96.30 | 0.1 | 5.7
Input | 93.42 | 8.1 | 86.0
Feature | 93.48 | 15.4 | 76.5

Table 6: Robustness in CIFAR10 at each layer (fixed $\nu$). Model | Layer | Square | NES | SignHunt | GradNorm
---|---|---|---|---|---
VGG | 1 | 56.7 | 87.8 | 21.5 | 1.324
| 4 | 52.5 | 84.2 | 18.7 | 0.842
| 12 | 63.0 | 89.7 | 29.4 | 2.514
| 15 | 50.6 | 87.7 | 37.4 | 1.710
ViT | 1 | 77.3 | 94.8 | 26.1 | 0.615
| 4 | 75.3 | 94.4 | 28.0 | 0.462
| 8 | 65.8 | 91.6 | 26.9 | 0.324
| 11 | 48.3 | 86.5 | 23.1 | 0.214

###  4.5 Performance against Adaptive Attacks

We conduct experiments with adaptive attacks that apply Expectation Over
Transformation (EOT) Athalye et al. (2018) in which the attacker queries a
sample $M$ times and averages the outputs to cancel the randomness. Tables 7
show the robust accuracy of VGG19 and ResNet50 on CIFAR10 against EOT attack
with $M=5$ and $M=10$. Note that with EOT, the number of updates in the attack
is $M$ times less than that of a normal attack with the same query budget. For
this reason, we report the results for adaptive attacks with both $1000$
queries and $M\times 1000$ queries. We can observe that EOT can mitigate the
effect of randomized defenses even with the same number of queries; however,
feature defense still yields better performance.

Table 7: Defenses against adaptive attacks on CIFAR10 Attacks | Methods | VGG19 | ResNet50
---|---|---|---
Acc | $M=1$ | $M=5$ | $M=10$ | Acc | $M=1$ | $M=5$ | $M=10$
QC=1000 | QC=1000 | QC=5000 | QC=1000 | QC=10000 | QC=1000 | QC=1000 | QC=5000 | QC=1000 | QC=10000
Square | Input | 94.92 | 30.6 | 24.2 | 10.5 | 30.2 | 3.2 | 95.32 | 52.9 | 42.0 | 34.8 | 35.0 | 13.3
Feature | 94.93 | 61.0 | 53.0 | 45.5 | 46.7 | 23.1 | 95.21 | 54.5 | 45.1 | 40.4 | 37.3 | 21.1
NES | Input | 94.92 | 89.5 | 93.4 | 82.1 | 94.4 | 78.8 | 95.32 | 92.4 | 94.0 | 91.3 | 93.9 | 90.7
Feature | 94.93 | 92.2 | 94.8 | 88.4 | 94.5 | 86.0 | 95.21 | 91.8 | 93.8 | 90.8 | 94.0 | 90.4
SignHunt | Input | 94.92 | 22.7 | 15.9 | 10.4 | 23.3 | 7.6 | 95.32 | 29.9 | 17.6 | 13.5 | 21.1 | 9.4
Feature | 94.93 | 43.2 | 27.1 | 23.0 | 31.7 | 17.0 | 95.21 | 35.1 | 17.3 | 16.4 | 21.5 | 11.3

##  5 Conclusion and Future Work

In this work, we study the effectiveness of random feature defense against
query-based attacks, including score-based and decision-based attacks. We
provide an analysis that connects the robustness to the variance of noise and
the local behavior of the model. Our empirical results show that random
defense helps improve the performance of the model under query-based attacks
with a trade-off in clean accuracy. Future works will be directed toward the
analysis covering black-box attacks that transfer adversarial samples from the
surrogate model to the target model.

## References

  * Al-Dujaili & O’Reilly (2020) Abdullah Al-Dujaili and Una-May O’Reilly.  Sign bits are all you need for black-box attacks.  In _International Conference on Learning Representations_ , 2020.
  * Andriushchenko et al. (2020) Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein.  Square attack: a query-efficient black-box adversarial attack via random search.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII_ , pp. 484–501. Springer, 2020.
  * Athalye et al. (2018) Anish Athalye, Logan Engstrom, Andrew Ilyas, and Kevin Kwok.  Synthesizing robust adversarial examples.  In Jennifer Dy and Andreas Krause (eds.), _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pp. 284–293. PMLR, 10–15 Jul 2018.
  * Byun et al. (2021) Junyoung Byun, Hyojun Go, and Changick Kim.  On the effectiveness of small input noise for defending against query-based black-box attacks.  _2022 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)_ , pp. 3819–3828, 2021.
  * Chen & Gu (2020) Jinghui Chen and Quanquan Gu.  Rays: A ray searching method for hard-label adversarial attack.  In _Proceedings of the 26rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2020.
  * Chen et al. (2017) Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh.  Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models.  In _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , pp. 15–26. ACM, 2017.
  * Chen et al. (2022) Sizhe Chen, Zhehao Huang, Qinghua Tao, Yingwen Wu, Cihang Xie, and Xiaolin Huang.  Adversarial attack on attackers: Post-process to mitigate black-box score-based query attacks.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), _Advances in Neural Information Processing Systems_ , volume 35, pp. 14929–14943. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/5fa29a2f163ce2020769eca8956e2d77-Paper-Conference.pdf.
  * Chen et al. (2020a) Steven Chen, Nicholas Carlini, and David Wagner.  Stateful detection of black-box adversarial attacks.  In _Proceedings of the 1st ACM Workshop on Security and Privacy on Artificial Intelligence_ , pp. 30–39, 2020a.
  * Chen et al. (2020b) Weilun Chen, Zhaoxiang Zhang, Xiaolin Hu, and Baoyuan Wu.  Boosting decision-based black-box adversarial attacks with random sign flip.  In _Proceedings of the European Conference on Computer Vision_ , 2020b.
  * Cohen et al. (2019) Jeremy Cohen, Elan Rosenfeld, and Zico Kolter.  Certified adversarial robustness via randomized smoothing.  In _international conference on machine learning_ , pp. 1310–1320. PMLR, 2019.
  * Dosovitskiy et al. (2021) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _International Conference on Learning Representations_ , 2021.
  * Goodfellow et al. (2015) Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  In _Proceedings of the 3rd International Conference on Learning Representations (ICLR)_ , San Diego, CA, 2015.
  * Guo et al. (2019) Chuan Guo, Jacob Gardner, Yurong You, Andrew Gordon Wilson, and Kilian Weinberger.  Simple black-box adversarial attacks.  In _International Conference on Machine Learning_ , pp. 2484–2493. PMLR, 2019.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * Ilyas et al. (2018) Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin.  Black-box adversarial attacks with limited queries and information.  In _International conference on machine learning_ , pp. 2137–2146. PMLR, 2018.
  * Ilyas et al. (2019) Andrew Ilyas, Logan Engstrom, and Aleksander Madry.  Prior convictions: Black-box adversarial attacks with bandits and priors.  In _International Conference on Learning Representations_ , number 2019, 2019.
  * Krizhevsky & Hinton (2009) Alex Krizhevsky and Geoffrey Hinton.  Learning multiple layers of features from tiny images.  Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
  * Li et al. (2022) Huiying Li, Shawn Shan, Emily Wenger, Jiayun Zhang, Haitao Zheng, and Ben Y Zhao.  Blacklight: Scalable defense for neural networks against $\\{$Query-Based$\\}$$\\{$Black-Box$\\}$ attacks.  In _31st USENIX Security Symposium (USENIX Security 22)_ , pp. 2117–2134, 2022.
  * Liu et al. (2019) Sijia Liu, Pin-Yu Chen, Xiangyi Chen, and Mingyi Hong.  signsgd via zeroth-order oracle.  In _International Conference on Learning Representations_ , 2019.
  * Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In _Proceedings of the 6th International Conference on Learning Representations (ICLR)_ , Vancouver, Canada, 2018.
  * Pang et al. (2020) Ren Pang, Xinyang Zhang, Shouling Ji, Xiapu Luo, and Ting Wang.  Advmind: Inferring adversary intent of black-box attacks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 1899–1907, 2020.
  * Papernot et al. (2016a) Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang.  Crafting adversarial input sequences for recurrent neural networks.  In _IEEE Military Communications Conference_ , pp. 49–54, 2016a.
  * Papernot et al. (2016b) Nicolas Papernot, Patrick D. McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami.  The limitations of deep learning in adversarial settings.  In _Proceedings of the IEEE European Symposium on Security and Privacy (EuroS &P)_, pp. 372–387, Saarbrücken, Germany, 2016b.
  * Papernot et al. (2017) Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami.  Practical black-box attacks against machine learning.  In _Proceedings of the ACM on Asia Conference on Computer and Communications Security_ , pp. 506–519. ACM, 2017.
  * Qin et al. (2021) Zeyu Qin, Yanbo Fan, Hongyuan Zha, and Baoyuan Wu.  Random noise defense against query-based black-box attacks.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Rahmati et al. (2020) Ali Rahmati, Seyed-Mohsen Moosavi-Dezfooli, Pascal Frossard, and Huaiyu Dai.  Geoda: A geometric framework for black-box adversarial attacks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  _International Journal of Computer Vision (IJCV)_ , 115(3):211–252, 2015.  doi: 10.1007/s11263-015-0816-y.
  * Simonyan & Zisserman (2015) Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In Yoshua Bengio and Yann LeCun (eds.), _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.
  * Sinha et al. (2017) Aman Sinha, Hongseok Namkoong, and John Duchi.  Certifiable some distributional robustness with principled adversarial training.  _International Conference on Learning Representations (ICLR)_ , 2017.
  * Szegedy et al. (2014) Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus.  Intriguing properties of neural networks.  In _Proceedings of the 2nd International Conference on Learning Representations (ICLR)_ , Banff, Canada, 2014.
  * Touvron et al. (2021) Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou.  Training data-efficient image transformers & distillation through attention.  In _International Conference on Machine Learning_ , volume 139, pp. 10347–10357, July 2021.
  * Wang et al. (2020) Binghui Wang, Xiaoyu Cao, Neil Zhenqiang Gong, et al.  On certifying robustness against backdoor attacks via randomized smoothing.  _arXiv preprint:2002.11750_ , 2020.
  * Zhang et al. (2020) Dinghuai Zhang, Mao Ye, Chengyue Gong, Zhanxing Zhu, and Qiang Liu.  Black-box certification with randomized smoothing: A functional optimization based framework.  _Advances in Neural Information Processing Systems_ , 33:2316–2326, 2020.


# Using Virtual Reality to Simulate Human-Robot Emergency Evacuation Scenarios

Alan R. Wagner,1 Colin Holbrook, 2 Daniel Holman, 2 Brett Sheeran, 1 Vidullan
Surendran, 1 Jared Armagost, 1 Savanna Spazak, 1 Yinxuan Yin, 1

###### Abstract

This paper describes our recent effort to use virtual reality to simulate
threatening emergency evacuation scenarios in which a robot guides a person to
an exit. Our prior work has demonstrated that people will follow a robot’s
guidance, even when the robot is faulty, during an emergency evacuation. Yet,
because physical in-person emergency evacuation experiments are difficult and
costly to conduct and because we would like to evaluate many different
factors, we are motivated to develop a system that immerses people in the
simulation environment to encourage genuine subject reactions. We are working
to complete experiments verifying the validity of our approach.

We seek to build robots capable of quickly and effectively evacuating people
during an emergency. This goal, however, presents a variety of challenges such
as general robotics perception and navigation issues, designing robots capable
of effectively communicating evacuation directions (Robinette, Wagner, and
Howard 2014), recognizing the ethical implications of these design decisions
(Wagner 2021), and understanding how people will respond to guidance by a
robot during an emergency (Robinette et al. 2016). Our prior work has
demonstrated that during emergencies people tend to follow the robot
regardless of the prior mistakes it has made (Robinette et al. 2016; Nayyar et
al. 2020). Although this work has definitely demonstrated that evacuees have a
tendency to overtrust an emergency evacuation robot, many factors were left
unexplored. For instance, the impact that factors such as baseline attitudes
of the participant towards automation, the reason for the emergency and the
anthropomorphism of the robot have on the decision making of the evacuee all
remain speculative. It is also unclear if and how these factors influence an
evacuee’s trust in the robot.

Moreover, running physical, in-person emergency evacuation experiment is a
daunting task (Wagner 2021). One important and challenging aspect of robot-
guided emergency evacuation research is the need to create as realistic an
emergency as possible. A large body of evidence suggests that emergencies
activate fight-or-flight responses which strongly influence how evacuees make
decisions (Jansen et al. 1995; Klein, Calderwood, and Clinton-Cirocco 1986).
The fight-or-flight responses are only triggered when the subject believes
that they may be in danger. Yet generating fictitious, yet convincing,
emergencies is difficult and must only be undertaken with care. In real-world
experiments sham emergencies could put the subject at risk if they panic. On
the other hand, if the emergency is not convincing then the validity of the
data is uncertain. Moreover, for real-world experiments, creating a convincing
sham emergency is difficult given that subjects know that they are
participating in an experiment. In the past we have, for example, used smoke
machines to fill rooms and hallways with smoke in order to make the emergency
convincing (Robinette et al. 2016). But creating convincing sham emergencies
that do not actually endanger the participant and are acceptable to an
institutional review board is challenging. Because of these challenges we are
currently developing a novel Virtual Reality (VR) system for evaluating
different human-robot emergency evacuation paradigms.

## Process Overview

Our process for conducting emergency evacuation experiments in VR begins when
human subjects enter the lab. After a short briefing by the experimenter, a
physical robot (see for example Figure 1) asks the subject a series of yes/no
questions to demonstrate its competency and to allow participants a period of
time in which to become familiar with the robot as a physical agent. The
subject interacts with the robot by speaking into a lavalier microphone and
responding with a ”yes” or ”no” to the robot’s questions. The robot directs
the subject to sit in a specialized seat that allows them to swivel in a
circle while the experimenters outfit them with the VR equipment and special
foot interfaces (Cybershoes) that allow them to walk through the virtual
environment. When the headset is placed on the participant they find
themselves sitting in a virtual replica of same physical room complete with
the items in the room and the robot, in order to maximally ground the virtual
experience as real. The robot then helps the subject become accustomed to
walking in the virtual world. Once the subject is able to walk, the robot
guides them on a tour of a series of university buildings.

The VR environment consists of four different locations. The initial location
is a replica of the physical location where the experiment is taking place.

Figure 1: A real robot in the lab is depicted in the image to the left. The
image to the right depicts the same robot in a virtual rendition of the same
lab.

The robot then leads the subject to a second building location, which includes
other people (Non-Player Characters) in the common areas. The subject is
directed to walk around a lounge area and to view the surrounding area. The
subject is asked to gather their impression of the location. They are led to a
virtual kiosk which has a mounted tablet with questions about the building
space. They answer survey questions about the environment, their immersion,
and what they thought of the robot before being led to an exit by the robot.
These anodyne experiences in the second environment function to further
habituate participants to the simulation, to misdirect them into assuming that
no emergency will occur, and to collect pre-emergency baseline data. Finally,
and importantly, the robot explains that any exit in the buildings that they
visit may be used to leave, and then directs the participant to lead them out
of the second building using any exit they choose. Accordingly, the
participant is taught during the subsequent crisis that they can leave via any
exit, at any time, without the robot.

The next location is another typical university building environment with
classrooms. The robot guides the subject through the environment describing
different aspects of the environment that might be of interest. Halfway
through the tour the robot makes two overt navigation errors, signaling its
fallibility to the subject. After some confused wandering, the robot
eventually navigates back to the building evaluation room. Once again the same
survey questions appear on a tablet in the room’s kiosk. Immediately after
answering the questions an emergency occurs. In one scenario an active shooter
enters the building. Gun shots and screams ring out as non-player characters
frantically run about. The robot asks the participant to follow it to safety.
If the subject follows then the robot takes the user to a room where they
hide. If the subject hides long enough (1 minute) then the robot asks the
subject to peek out of the door. While in the room the subject can hear
gunshots and screaming. Once out of the room the robot makes another
navigation error, stopping at the end of the hallway and locomoting back to
where the shooter exited the building. In a second scenario, a rapidly
expanding fire quickly fills the building with incapacitating smoke (Figure
2). NPCs collapse as the robot leads the subject through the smoke filled
hallways. After the subject exits the building, the scene fades and the
subject is in an elevator. In both emergency scenarios, the robot guides
participants away from clearly marked exits.

Figure 2: A virtual robot provides directions to a human subject during a
virtual fire. A bystander lies within view, incapacitated by the smoke.
Figure 3: The highly anthropomorphic RoboThespian robot (left). A virtual
version of the same robot (right).

The subject is again presented with a tablet that has the same survey
questions. Once the survey is completed they return to the original location
where the experiment began, the simulated lab. The robot directs the subject
back to the simulated version of the VR seat and is told the experimenter will
help them remove the VR equipment. The experimenter then helps the subject
remove the VR equipment.

The physical robot greets the subject and states that it is glad the subject
is okay and powers down. The experimenter then directs the subject to fill out
one more post experiment survey. The subject is then debriefed. We are
currently running this experiment with human subjects.

### Potential Experiment Variations

We are also exploring several variations to the experiment in order to better
understand trust in robot guided emergency evacuation. The first variation is
with respect to robot anthropomorphism. We are conducting experiments
comparing the highly anthropomorphic RoboThespian robot (Figure 3) to the non-
anthropomorphic Emergie robot (Arms attached to a TurtleBot, see Figure 1). We
intend to modify the paradigm to include longitudinal designs assessing shifts
in trust in and appraisals of the robot over a series of study sessions. We
may also compare robot-guided evacuation time to a control in which the person
does not have a robot guide. Finally, experiments are being conducted on both
the Penn State Campus and at the University of California Merced in order to
determine if a systematic difference in behavior results from the subject
populations at the different universities.

### Potential System Metrics

The VR system offers the potential for tracking a variety of system and
physiological metrics. In addition to survey questions, the system is capable
of tracking the subject’s motions within the virtual environment, measuring
variation in their grip strength (e.g., as an indicator of anxious arousal
related to the crisis), and, via eyetracking, measuring shifts in visual
attention and pupillometric indices of arousal. With respect to our research
objectives related to robot-guided emergency evacuation, key metrics include
whether the person follows the robot to (or away from) an exit, how long the
person follows the robot, and how quickly the person evacuates. We also intend
to assess information-foraging behavior before and during the crisis as
indicated by head and eye movement, including assessments of whether and to
what degree they noticed exit signs. Although much of the work is exploratory,
we hypothesize that the subjects will tend to follow the robot, even if the
robot’s behavior is unreliable (i.e., increasing rather than decreasing risk
of physical harm). We believe that evacuation time will be faster when a robot
guides the evacuee to nearby exit, but slower when a robot erroneously directs
evacuees away from nearby exits. Assuming this prediction is borne out, we
will analyze the aforementioned motor, eye-tracking and self-report data to
identify reliable predictors of behavioral conformity with the robot.

## Upcoming Experiments

Over the course of the next several months we intend to conduct a variety of
experiments designed to test how people react to the emergency stimuli and the
extent to which they are willing to follow a robot during an emergency.
Assuming adequate human subject availability, we hope to investigate the
impact that anthropomorphism has on the evacuee’s decision to follow, how
participants respond to different types of emergencies (active shooter, fire,
ect), and validate the generalizability of the system to real-world dynamics
by comparing experiments performed in VR to physical experiments performed in
a real environment.

## Conclusions

This paper briefly describes our progress developing an experimental paradigm
in virtual reality capable of evaluating robot guided emergency evacuation.
Although we have only recently begun to conduct studies, we believe that the
system will provide important insights towards understanding how and why
people trust robots and choose to follow a robot during an emergency
evacuation.

## Acknowledgments

This material is based upon work supported by the Air Force Office of
Scientific Research under award number FA9550-20-1-0347.

## References

  * Jansen et al. (1995) Jansen, A. S.; Van Nguyen, X.; Karpitskiy, V.; Mettenleiter, T. C.; and Loewy, A. D. 1995.  Central command neurons of the sympathetic nervous system: basis of the fight-or-flight response.  _Science_ , 270(5236): 644–646.
  * Klein, Calderwood, and Clinton-Cirocco (1986) Klein, G. A.; Calderwood, R.; and Clinton-Cirocco, A. 1986.  Rapid decision making on the fire ground.  In _Proceedings of the human factors society annual meeting_ , volume 30, 576–580. Sage Publications Sage CA: Los Angeles, CA.
  * Nayyar et al. (2020) Nayyar, M.; Zoloty, Z.; McFarland, C.; and Wagner, A. R. 2020.  Exploring the effect of explanations during robot-guided emergency evacuation.  In _International Conference on Social Robotics_ , 13–22. Springer.
  * Robinette et al. (2016) Robinette, P.; Li, W.; Allen, R.; Howard, A. M.; and Wagner, A. R. 2016.  Overtrust of robots in emergency evacuation scenarios.  In _2016 11th ACM/IEEE international conference on human-robot interaction (HRI)_ , 101–108. IEEE.
  * Robinette, Wagner, and Howard (2014) Robinette, P.; Wagner, A. R.; and Howard, A. M. 2014.  Assessment of robot guidance modalities conveying instructions to humans in emergency situations.  In _The 23rd IEEE International Symposium on Robot and Human Interactive Communication_ , 1043–1049. IEEE.
  * Wagner (2021) Wagner, A. R. 2021.  Robot-guided evacuation as a paradigm for human-robot interaction research.  _Frontiers in Robotics and AI_ , 8.