# Boundary Defense Against Black-box Adversarial Attacks

Manjushree B. Aithal and Xiaohua Li  Department of Electrical and Computer
Engineering
Binghamton University
Binghamton, NY 13902, USA
Email: {maithal1<EMAIL_ADDRESS>
###### Abstract

Black-box adversarial attacks generate adversarial samples via iterative
optimizations using repeated queries. Defending deep neural networks against
such attacks has been challenging. In this paper, we propose an efficient
Boundary Defense (BD) method which mitigates black-box attacks by exploiting
the fact that the adversarial optimizations often need samples on the
classification boundary. Our method detects the boundary samples as those with
low classification confidence and adds white Gaussian noise to their logits.
The method’s impact on the deep network’s classification accuracy is analyzed
theoretically. Extensive experiments are conducted and the results show that
the BD method can reliably defend against both soft and hard label black-box
attacks. It outperforms a list of existing defense methods. For IMAGENET
models, by adding zero-mean white Gaussian noise with standard deviation 0.1
to logits when the classification confidence is less than 0.3, the defense
reduces the attack success rate to almost 0 while limiting the classification
accuracy degradation to around 1 percent.

##  I Introduction

Deep neural networks (DNNs) have achieved increasing demand in many practical
applications [1][2][3]. However, studies over the past few years have also
shown intriguing issue that DNN models are very sensitive and vulnerable to
adversarial samples [4][5], implying potential security threats to their
applications.

One of the widely studied adversarial attacks is the evasion attack, where the
main aim of the attacker is to cause misclassification in the DNN model.
Black-box evasion attacks have attracted increasing research interests
recently, where black-box means that the attacker does not know the DNN model
but can query the model to get the DNN inference outputs, either the detailed
confidence score or just a classification label
[6][7][8][9][10][11][12][13][14][15][16]. If the attacker has access to the
full output logit values, they can apply soft-label attack algorithms such as
[8][9][10][6][16]. On the other hand, if the attacker has access to only the
classification label, they can apply hard-label attack algorithms such as
[7][13][15].

Along with the surge of attack algorithms, there has been an increase in the
development of defense algorithms such as Adversarial Training (AT) [17],
input transformation [18][19], gradient obfuscation [20], and stochastic
defense via randomization [21][22][23][24][25][26][27]. However, limitations
of existing defense techniques have also been observed [28][29][30]. It has
been proven that stochastic defense suffers from large degradation of DNN
performance or limited defense performance. Gradient obfuscation method has
also been proven to be ineffective.

In this work, we develop an efficient and more effective method to defend the
DNN against black-box attacks. During the adversarial attack’s optimization
process, there is a stage that the adversarial samples are on the DNN’s
classification boundary. Boundary Defense BD$(\theta,\sigma)$, our method,
detects these boundary samples as those with the classification confidence
score below the threshold $\theta$ and adds white Gaussian noise with standard
deviation $\sigma$ to their logits. This will prevent the attackers from
optimizing their adversarial samples and maintain low DNN performance
degradation.

Major contributions of this work are:

  * •

A new boundary defense algorithm BD$(\theta,\sigma)$ is developed, which can
be implemented efficiently and mitigate reliably both soft and hard label
black-box attacks.

  * •

Theoretical analysis is conducted to study the impact of the parameters
$\theta$ and $\sigma$ on the classification accuracy.

  * •

Extensive experiments are conducted, which demonstrate that BD(0.3, 0.1) (or
BD(0.7, 0.1)) reduces attack success rate to almost 0 with around 1% (or
negligible) classification accuracy degradation over the IMAGENET (or
MNIST/CIFAR10) models. The defense performance is shown superior over a list
of existing defense algorithms.

The organization of this paper is as follows. In Section II, related works are
introduced. In Section III, the BD method is explained. In Section IV,
experiment results are presented. Finally, conclusions are given in Section V.

##  II Related Work

Black-box adversarial attacks can be classified into soft-label and hard-label
attacks. In soft-label attacks like AutoZOOM [8][12] and NES-QL [10], the
attacker generates adversarial samples using the gradients estimated from
queried DNN outputs. In contrast, SimBA [16], GenAttack [6] and Square Attack
[31] resort to direct random search to obtain the adversarial sample. Hard-
label attacks like NES-HL [10], BA (Boundary Attack) [7], Sign-OPT [12][13],
and HopSkipJump [15] start from an initial adversarial sample and iteratively
reduce the distance between the adversarial sample and original sample based
on the query results.

For the defense against black-box attacks, a lot of methods are derived
directly from the defense methods against white-box attacks, such as input
transformation [32], network randomization [33] and adversarial training [34].
The defenses designed specifically for black-box attack, are denoised
smoothing [35], malicious query detection [36][37][38], and random-smoothing
[39][40]. Nevertheless, their defense performance is not reliable and defense
cost or complexity is too high. Adding random noise to defend against black-
box attacks has been studied recently as a low-cost approach, where
[41][42][33] add noise to the input, and [43] [44][21] add noise to input or
weight of each layer. Unfortunately, heavy noise is needed to defend against
hard-label attacks (in order to change hard labels) but heavy noise leads to
severe degradation of DNN accuracy. Our proposed BD method follows similar
approach, but we add noise only to the DNN outputs of the boundary samples,
which makes it possible to apply heavy noise without significant degradation
in DNN accuracy.

##  III Boundary Defense

###  III-A Black-box attack model

Consider a DNN that classifies an image ${\bf X}_{0}$ into class label $c$
within $N$ classes. The DNN output is softmax logit (or confidence score)
tensor $F({\bf X}_{0})$. The classification result is
$c=\arg\max_{i}{F}_{i}({\bf X}_{0})$, where $F_{i}$ denotes the $i$th element
function of $F$, $i=0,\cdots,N-1$. The attacker does not know the DNN model
but can send samples ${\bf X}$ to query the DNN and get either $F({\bf X})$ or
just $c$. The objective of the attacker is to generate an adversarial sample
${\bf X}={\bf X}_{0}+\Delta{\bf x}$ such that the output of the classifier is
$t=\arg\max_{i}F_{i}({\bf X})\neq c$, where the adversary $\Delta{\bf x}$
should be as small as possible.

Soft-Label Black-box Attack: The attacker queries the DNN to obtain the
softmax logit output tensor ${F}({\bf X})$. With this information, the
attacker minimizes the loss function $f_{{\rm SL}}({\bf X})$ for generating
the adversarial sample ${\bf X}$ [9],

$f_{{\rm SL}}({\bf X})={\cal D}({\bf X},{\bf X}_{0})+\lambda{\cal L}(F({\bf
X}),t),$ (1)

where ${\cal D}(\cdot,\cdot)$ is a distance function, e.g., $\|{\bf X}-{\bf
X}_{0}\|_{p}$, and ${\cal L}(\cdot,t)$ is the loss function, e.g., cross-
entropy [10] and C&W loss [29].

Hard-Label Black-box Attack: The attacker does not use $F({\bf X})$ but
instead uses the class label $\arg\max_{i}F_{i}({\bf X})$ to optimize the
adversarial sample ${\bf X}$. A common approach for the attacker is to first
find an initial sample ${\bf X}_{t,0}$ in the class $t\neq c$, i.e.,
$\arg\max_{i}F_{i}({\bf X}_{t,0})=t$. Then, starting from ${\bf X}_{t,0}$, the
attacker iteratively estimates new adversarial samples ${\bf X}$ in the class
$t$ so as to minimize the loss function $f_{{\rm HL}}({\bf X})={\cal D}({\bf
X},{\bf X}_{0})$.

The above model is valid for both targeted and untargeted attacks. The
attacker’s objective is to increase attack success rate (ASR), reduce query
counts (QC), and reduce sample distortion ${\cal D}({\bf X},{\bf X}_{0})$. In
this paper, we assume that the attacker has a large enough QC budget and can
adopt either soft-label or hard-label black-box attack algorithms. Thus, our
proposed defense’s main objective is to reduce the ASR to 0.

Figure 1: Schematic representation of the black-box attack and the Boundary
Defense BD$(\theta,\sigma)$ (highlighted region).

###  III-B Boundary Defense Algorithm

In this work, we propose a Boundary Defense method that defends the DNN
against black-box (both soft and hard label, both targeted and untargeted)
attacks by preventing the attacker’s optimization of $f_{{\rm SL}}({\bf X})$
or $f_{{\rm HL}}({\bf X})$. As illustrated in Fig. 1, for each query of ${\bf
X}$, once the defender finds that the classification confidence $\max F({\bf
X})$ is less than certain threshold $\theta$, the defender adds zero-mean
white Gaussian noise ${\cal N}(0,\sigma^{2})$ with a certain standard
deviation $\sigma$ to all the elements of $F({\bf X})$. The DNN softmax logits
thus become

$F_{{\rm BD}}({\bf X})=\left\\{\begin{array}[]{ll}F({\bf X}),&{\rm if}\;\max
F({\bf X})>\theta\\\ F({\bf X})+V,&{\rm otherwise}\end{array}\right.$ (2)

where $V\sim{\cal N}({\bf 0},\sigma^{2}{\bf I})$ and ${\bf I}$ is an identity
matrix. The DNN outputs softmax logits clip{$F_{{\rm BD}}({\bf X})$, 0, 1}
when outputting soft labels or its classification label $\arg\max_{i}F_{{\rm
BD},i}({\bf X})$ when outputting hard labels.

We call it the BD$(\theta,\sigma)$ algorithm because samples with low
confidence scores are usually on the classification boundary. For a well-
designed DNN, the clean (non-adversarial) samples can usually be classified
accurately with high confidence scores. Those with low confidence scores
happen rarely and have low classification accuracy. In contrast, when the
attacker optimizes $f_{{\rm SL}}({\bf X})$ or $f_{{\rm HL}}({\bf X})$, there
is always a stage that the adversarial samples ${\bf X}$ have small $\max
F({\bf X})$ values.

For example, in the soft-label black-box targeted attacks, the attacker needs
to maximize the $t$th logit value $F_{t}({\bf X})$ by minimizing the cross-
entropy loss ${\cal L}(F({\bf X}),t)=-\log F_{t}({\bf X})$. Initially
$F_{t}({\bf X}_{0})$ is very small and $F_{c}({\bf X}_{0})$ is large. The
optimization increases $F_{t}({\bf X})$ while reducing $F_{c}({\bf X})$. There
is a stage that all logit values are small, which means ${\bf X}$ is lying on
the classification boundary.

As another example, a typical hard-label black-box targeted attack algorithm
first finds an initial sample inside the target class $t$, which we denote as
${\bf X}_{t,0}$. The algorithm often uses line search to find a boundary
sample ${\bf X}=\alpha{\bf X}_{t,0}+(1-\alpha){\bf X}_{0}$ that maintains
label $t$, where $\alpha$ is the optimization parameter. Then the algorithm
randomly perturbs ${\bf X}$, queries the DNN, and uses the query results to
find the direction to optimize $f_{{\rm HL}}({\bf X})$. Obviously, ${\bf X}$
must be on the decision boundary so that the randomly perturbed ${\bf X}$ will
lead to changing DNN hard-label outputs. Otherwise, all the query results will
lead to a constant output $t$, which is useless to the attacker’s optimization
process.

Therefore, for soft-label attacks there is an unavoidable stage of having
boundary samples and for hard-label attacks the boundary samples are
essential. Our BD method exploits this weakness of black-box attacks by
detecting these samples and scrambling their query results to prevent the
attacker from optimizing its objective.

One of the advantages of the BD$(\theta,\sigma)$ algorithm is that it can be
implemented efficiently and inserted into DNN models conveniently with minimal
coding. Another advantage is that the two parameters $(\theta,\sigma)$ make it
flexible to adjust the BD method to work reliably. Large $\theta$ and $\sigma$
lead to small ASR but significant DNN performance degradation. Some attacks
are immune to small noise (small $\sigma$), such as the HopSkipJump hard-label
attack [15]. Some other attacks such as SimBA [16] are surprisingly immune to
large noise in boundary samples, which means that simply removing boundary
samples or adding extra large noise to boundary samples as suggested in [15]
does not work. The flexibility of $(\theta,\sigma)$ makes it possible for the
BD method to deal with such complicated issues and to be superior over other
defense methods.

###  III-C Properties of Boundary Samples

In this section, we study BD$(\theta,\sigma)$’s impact on the DNN’s
classification accuracy (ACC) when there is no attack, which provides useful
guidance to the selection of $\theta$ and $\sigma$.

(a) (b)

Figure 2: Impact of parameters $\theta$ and $\sigma$ to classification
accuracy (ACC). (a) ACC as function of true logit value $s=F_{c}({\bf X})$.
(b) ACC when boundary defense BD$(\theta,\sigma)$ is applied. CleanACC is the
DNN’s ACC without attack/defense.

Consider a clean sample ${\bf X}$ with true label $c$ and confidence
$s=F_{c}({\bf X})$. Since the DNN is trained with the objective of maximizing
$F_{c}({\bf X})$, we can assume that all the other logit values $F_{i}({\bf
X})$, $i=0,\cdots,N-1$, $i\neq c$, are independent and identically distributed
uniform random variables with values within $0$ to $a=(1-s)/(N-1)$, i.e.,
$F_{i}({\bf X})\sim U(0,a)$. Without loss of generality, let $F_{0}({\bf X})$
be the maximum among these $N-1$ values. Then $Y=\sum_{i=1,i\neq
c}^{N-1}F_{i}({\bf X})$ follows Irwin-Hall distribution with cumulative
distribution function (CDF)

$P_{Y}[y<x]=\frac{1}{(N-2)!}\sum_{k=0}^{\lfloor
x/a\rfloor}(-1)^{k}\left(\begin{array}[]{c}N-2\\\
k\end{array}\right)\left(\frac{x}{a}-k\right)^{N-2}.$ (3)

When $N$ is large, the distribution of $Y$ can be approximated as normal
${\cal
N}\left(\frac{(1-s)(N-2)}{2(N-1)},\left(\frac{1-s}{N-1}\right)^{2}\frac{N-2}{12}\right)$.
We denote its CDF as $\Phi(x)$. Since the sample ${\bf X}$ is classified
accurately if and only if $s>F_{0}({\bf X})$, the classification accuracy
$P[{\rm ACC}|s]$ can be derived as

$P[{\rm ACC}|s]=P_{Y}[y>1-2s]=1-\Phi(1-2s).$ (4)

Using (4), we can calculate $P[{\rm ACC}|s]$ for each $s$, as shown in Fig.
2(a) for $N=10$ and $1000$ classes. It can be seen that for $N=1000$, if
$s<0.32$, then the sample’s classification ACC is almost 0. This means that we
can set $\theta\leq 0.32$ to safely scramble all those queries whose maximum
logit value is less than 0.32 without noticeable ACC degradation.

Next, to evaluate the ACC when BD$(\theta,\sigma)$ is applied, we assume the
true label $c$’s logit value $s$ follow approximately half-normal
distribution, whose probability density function is

$f_{S}(s)=\left\\{\begin{array}[]{ll}\frac{\sqrt{2}}{\nu\sqrt{\pi}}e^{-\frac{(1-s)^{2}}{2\nu^{2}}},&s\leq
1\\\ 0,&{\rm otherwise}\end{array}\right.$ (5)

with the parameter $\nu$. The ACC of the DNN without attack or defense (which
we call cleanACC) is then

${\rm cleanACC}=\int_{0}^{1}P[{\rm ACC}|s]f_{S}(s)ds.$ (6)

Using (5)-(6), we can find the parameter $\nu$ for each clean ACC. For
example, for $N=1000$ and a DNN with clean ACC 90%, the distribution of true
logit $s$ follows (5) with $\nu=0.41$.

When noise is added, each $F_{i}(X)$ becomes $F_{i}(X)+v_{i}$ for noise
$v_{i}\sim{\cal N}(0,\sigma^{2})$. Following similar derivation of (3)-(4), we
can obtain the ACC of the noise perturbed logit $F_{c}(X)+v_{c}$ as

$P[{\rm ACC}|s,\sigma]=1-\tilde{\Phi}(1-2s),$ (7)

where $\tilde{\Phi}$ is the CDF of the new normal distribution ${\cal
N}\left(\frac{(1-s)(N-2)}{2(N-1)},\left(\frac{1-s}{N-1}\right)^{2}\frac{N-2}{12}+(N+2)\sigma^{2}\right)$.
The ACC under the defense is then

${\rm ACC}=\int_{0}^{\theta}P[{\rm
ACC}|s,\sigma]f_{S}(s)ds+\int_{\theta}^{1}P[{\rm ACC}|s]f_{S}(s)ds.$ (8)

Fig. 2(b) shows how the defense ACC degrades with the increase of $\theta$ and
$\sigma$. We can see that with $\sigma=0.1$, there is almost no ACC
degradation for $N=10$. For $N=1000$, ACC degradation is very small when
$\theta<0.4$ but grows to 5% when $\theta>0.6$. Importantly, under
$\theta<0.4$ we can apply larger noise $\sigma=0.3$ safely without obvious ACC
degradation. This shows the importance of scrambling boundary samples only.
Existing defenses scramble all the samples, which corresponds to $\theta=1$,
and thus suffer from significant ACC degradation.

##  IV Experiments

###  IV-A Experiment Setup

In the first experiment, with the full validation datasets of MNIST (10,000
images), CIFAR10 (10,000 images), IMAGENET (50,000 images) we evaluated the
degradation of classification accuracy of a list of popular DNN models when
our proposed BD method is applied.

In the second experiment, with $1000$ validation images of MNIST/CIFAR10 and
$100$ validation images of IMAGENET, we evaluated the defense performance of
our BD method against several state-of-the-art black-box attack methods,
including soft-label attacks AZ (AutoZOOM) [9], NES-QL (query limited) [10],
SimBA (SimBA-DCT) [16], and GA (GenAttack) [6], as well as hard-label attacks
NES-HL (hard label) [10], BA (Boundary Attack) [7], HSJA (HopSkipJump Attack)
[15], and Sign-OPT [13]. We adopted their original source codes with the
default hyper-parameters and just inserted our BD$(\theta,\sigma)$ as a
subroutine to process $F({\bf X})$ after each model prediction call. These
algorithms used the InceptionV3 or ResNet50 IMAGENET models. To maintain
uniformity and fair comparison, we considered the $l_{2}$ norm setting
throughout the experiment.

We also compared our BD method with some representative black-box defense
methods, including NP (noise perturbation), JPEG compression, Bit-Dept, and
TVM (Total Variation Minimization), whose data were obtained from [45], for
soft-label attacks, and DD (Defensive Distillation) [20], Region-based
classification [46], and AT (Adversarial Training) [47] for hard-label
attacks.

In order to have a more persuasive and comprehensive study of the robustness
of the proposed BD method, we also performed experiments using Robust
Benchmark models [48], such as RMC (Runtime Masking and Cleaning) [49], RATIO
(Robustness via Adversarial Training on In- and Out-distribution)[50], RO
(Robust Overfitting)[51], MMA (Max-Margin Adversarial)[52], ER (Engstrom
Robustness)[53], RD (Rony Decoupling)[54], and PD (Proxy Distribution)[55]
models, over the CIFAR10 dataset for various attack methods.

As the primary performance metrics, we considered ACC (DNN’s classification
accuracy) and ASR (attacker’s attack success rate). The ASR is defined as the
ratio of samples with $\arg\max_{i}F_{i}({\bf X})=t\neq c$. Without defense,
the hard-label attack algorithms always output adversarial samples
successfully with the label $t$ (which means ASR = 100%). Under our defense
the ASR will be reduced due to the added noise, so ASR is still a valid
performance measure. On the other hand, since most hard-label attack/defense
papers use the ASR defined as the ratio of samples satisfying both
$\arg\max_{i}F_{i}({\bf X})=t$ and median $l_{2}$ distortion ($\sqrt{\|{\bf
X}-{\bf X}_{0}\|^{2}/M}$ when ${\bf X}_{0}$ has $M$ elements) less than a
certain threshold, we will also report our results over this ASR, which we
called ASR2.

We show only the results of targeted attacks in this section. Experiments of
untargeted attacks as well as extra experiment data and result discussions are
provided in supplementary material.

###  IV-B ACC Degradation Caused by Boundary Defense

Figure 3: Top-1 classification accuracy degradation (Defense ACC $-$ Clean
ACC) versus $\theta$. $\sigma=0.1$.

For MNIST, we trained a 5-layer convolutional neural network (CNN) with clean
ACC 99%. For CIFAR10, we trained a 6-layer CNN with clean ACC 83% and also
applied the pre-trained model of [56] with ACC 97%, which are called CIFAR10-s
and CIFAR10, respectively. For IMAGENET, we used standard pre-trained models
from the official Tensoflow library (ResNet50, InceptionV3, EfficientNet-B7)
and the official PyTorch library (ResNet50tor, InceptionV3tor), where “-tor”
indicates their PyTorch source.

We used the validation images to query the DNN models and applied our BD
algorithm to modify the DNN outputs before evaluating classification ACC. It
can be observed from Fig. 3 that with $\theta\leq 0.3$ we can keep the loss of
ACC around $1\%$ for IMAGENET models (from 0.5% of ResNet50 to 1.5% of
InceptionV3). $\theta>0.6$ leads to near 5% ACC degradation. For MNIST and
CIFAR10 the ACC has almost no degradation, but CIFAR10-s has limited 1.5% ACC
degradation for large $\theta$. This fits well with the analysis results shown
in Fig. 2(b). Especially, most existing noise defense methods, which don’t
exploit boundary (equivalent to $\theta=1$), would result in up to 5% ACC
degradation for IMAGENET models.

###  IV-C Performance of BD Defense against Attacks

####  IV-C1 ASR of soft-label black-box attacks

Table I shows the ASR of soft-label black-box attack algorithms under our
proposed BD method. To save the space we have shown the data of $\sigma=0.1$
only. Results regarding varying $\theta$ and $\sigma$ are shown in Fig. 4.

TABLE I: ASR (%) of Targeted Soft-Label Attacks. $\sigma$ = 0.1. Dataset | Attacks | No defense |  $\theta_{1}$ = 0.5 |  $\theta_{2}$ = 0.7
---|---|---|---|---
| AZ | 100 | 8 | 8
MNIST | GA | 100 | 0 | 0
| SimBA | 97 | 3 | 0
| AZ | 100 | 9 | 9
CIFAR10 | GA | 98.76 | 0 | 0
| SimBA | 97.14 | 23 | 15
Dataset | Attacks | No Defense |  $\theta_{1}$ = 0.1 |  $\theta_{2}$ = 0.3
| AZ | 100 | 0 | 0
IMAGENET | NES-QL | 100 | 69 | 8
| GA | 100 | 0 | 0
| SimBA | 96.5 | 6 | 2
Figure 4: ASR(%) vs noise level $\sigma$ for various boundary threshold
$\theta$. The top row is for MNIST, and the bottom row is for CIFAR10.

From Table I we can see that with the increase in $\theta$, the ASR of all the
attack algorithms drastically reduced. Over the IMAGENET dataset, the BD
method reduced the ASR of all the attack algorithms to almost $0$ with
$(\theta,\sigma)=(0.3,0.1)$. For MNIST/CIFAR10 datasets, the BD method with
$(\theta,\sigma)=(0.5,0.1)$ was enough. Fig. 4 shows a consistent decline of
ASR over the increase in noise level $\sigma$. This steady decline indicates
robust defense performance of the BD method against the soft-label attacks.

####  IV-C2 ASR of hard-label black-box attacks

We have summarized the ASR and median $l_{2}$ distortion of hard-label attacks
in presence of our proposed BD method in Table II.

TABLE II: ASR (%) and Median $l_{2}$ Distortion of Targeted Hard-label Attacks. $\sigma=0.1$. “-” Means no $l_{2}$ Distortion Data due to Absence of Adversarial Samples. Dataset | Attacks |  | ASR/$l_{2}$ |
---|---|---|---|---
|  | No defense |  $\theta_{1}$ = 0.5 |  $\theta_{2}$ = 0.7
| Sign-OPT | 100/0.059 | 4/0.12 | 0/-
MNIST | BA | 100/0.16 | 17/0.55 | 9/0.56
| HSJA | 100/0.15 | 38/0.14 | 7/0.15
CIFAR10 | Sign-OPT | 100/0.004 | 4/0.08 | 0/-
| HSJA | 100/0.05 | 18/0.05 | 7/0.05
Dataset | Attacks |  | ASR/$l_{2}$ |
|  | No Defense |  $\theta_{1}$ = 0.1 |  $\theta_{2}$ = 0.3
| NES-HL | 90/0.12 | 0/- | 0/-
IMAGE- | Sign-OPT | 100/0.05 | 14/0.4 | 0/-
NET | BA | 100/0.08 | 0/- | 0/-
| HSJA | 100/0.03 | 34/0.11 | 0/-

Surprisingly, the BD method performed extremely well against the hard-label
attacks that were usually challenging to conventional defense methods. In
general, BD(0.3, 0.1) was able to reduce ASR to 0% over the IMAGENET dataset,
and BD(0.7, 0.1) was enough to reduce ASR to near 0 over MNIST and CIFAR10.

For ASR2, Figure 5 shows how ASR2 varies with the pre-set $l_{2}$ distortion
threshold when the BD method was used to defend against the Sign-OPT attack.
We can see that the ASR2 reduced with the increase of either $\theta$ or
$\sigma$, or both. BD(0.7, 0.1) and BD(0.3, 0.1) successfully defended against
the Sign-OPT attack over the MNIST/CIFAR10 and IMAGENET datasets,
respectively.

Figure 5: ASR (%) versus median $l_{2}$ distortion of the Sign-OPT attack
under the proposed BD method.

####  IV-C3 Robust defense performance against adaptive attacks

To evaluate the robustness of the defense, it is crucial to evaluate the
defense performance against adaptive attacks [34]. For example, the attacker
may change the query limit or optimization step size. In this subsection, we
show the effectiveness of our BD defense against 2 major adaptive attack
techniques: 1) adaptive query count (QC) budget; and 2) adaptive step size.

First, with increased attack QC budget, the results obtained are summarized in
Table III. We observe that when the attacker increased QC from $10^{4}$ to
$10^{10}$, there was no significant increase in ASR. Next, we adjusted the
optimization (or gradient estimation) step size of the attack algorithms (such
as $\beta$ of the Sign-OPT algorithm), and evaluated the performance of BD.
The ASR data are shown in Fig 6. We can see that there was no significant
change of ASR when the attack algorithms adopted different optimization step
sizes. For GenAttack & Sign-OPT, the ASR was almost the same under various
step sizes. For SimBA, the ASR slightly increased but with an expense of
heavily distorted adversarial output.

TABLE III: ASR(%) of Adaptive Black-Box Attacks under the Proposed BD method Dataset | Attack | query | budget |
---|---|---|---|---
|  |  $10^{4}$ (preset) | $10^{8}$ | $10^{10}$
| GA | 0 | 0 | 0
CIFAR10 | HSJA | 3 | 4 | 0
| Sign-OPT | 5 | 8 | 8
| NES-QL | 2 | 12 | 8
ImageNET | Boundary | 0 | 0 | 0
| HSJA | 0 | 0 | 0
| Sign-OPT | 0 | 0 | 0

As a result, we can assert the robustness of the BD method against the black-
box adversarial attacks.

Figure 6: ASR (%) versus step size of the adaptive attacks. Note that for GenAttack we considered $\sigma=0.01$, since the ASR for $\sigma=0.1$ was always 0 for all threshold values. For Sign-OPT & SimBA we considered $\sigma=0.1$. TABLE IV: Comparison of BD method with other defense methods against targeted hard-label attacks in terms of ASR (%). Dataset | Defense | HSJA | BA | SimBA-DCT
---|---|---|---|---
| DD [20] | 98 | 80 | -
| AT [47] | 100 | 50 | 4
| Region-based [46] | 100 | 85 | -
MNIST | BD ($\theta=0.5,\sigma=0.1$) | 38 | 17 | 3
| BD ($\theta=0.7,\sigma=0.1$) | 7 | 9 | 0
TABLE V: Comparison of BD method ($\theta=0.5,\sigma=0.1$) with other defense methods against targeted GenAttack (soft-label) in terms of ASR (%). Dataset | Attack | Bit-Depth | JPEG | TVM | NP | BD
---|---|---|---|---|---|---
MNIST | GenAttack | 95 | 89 | - | 5 | 0
CIFAR10 | GenAttack | 95 | 89 | 73 | 6 | 0
TABLE VI: Compare ASR (%) of proposed BD method with the Robust Bench Defense Models. CIFAR10 Dataset. RobustBench Defense | Sign-OPT | SimBA | HSJA
---|---|---|---
RMC [49] | 100 | 83 | 100
RATIO [50] | 100 | 59 | 100
RO [51] | 100 | 85 | 100
MMA [52] | 100 | 83 | 100
ER [53] | 100 | 92 | 100
RD [54] | - | 80 | -
PD [55] | 100 | 71 | 100
BD ($\theta=0.5,\sigma=0.1$) | 4 | 23 | 18
BD ($\theta=0.7,\sigma=0.1$) | 0 | 15 | 7

###  IV-D Comparison with Other Defense Methods

For defending against hard-label attacks, Table IV compares the BD method with
the DD, AT, and Region-based defense methods over the MNIST dataset. We
obtained these other methods’ defense ASR data from [15] for HopSkipJump and
BA attack methods, and obtained the defense performance data against SimBA
through our experiments. It can be seen that our BD method outperformed all
these defense methods with lower ASR. For soft-label attacks, Table V shows
that our BD method also outperformed a list of existing defense methods.

We also ran experiments using RobustBench models. The defense performance over
the CIFAR10 dataset is reported in Table VI. ASR is used as our preliminary
evaluation criteria because for an attacker the higher the ASR the more robust
the attack method is against all the defenses. From Table VI, we can see that
our method had the most superior defense performance.

##  V Conclusions

In this paper, we propose an efficient and effective boundary defense method
BD$(\theta,\sigma)$ to defend against black-box attacks. This method detects
boundary samples by examining classification confidence scores and adds random
noise to the query results of these boundary samples. BD$(0.3,0.1)$ is shown
to reduce the attack success rate to almost 0 with only about 1%
classification accuracy degradation for IMAGENET models. Analysis and
experiments were conducted to demonstrate that this simple and practical
defense method could effectively defend the DNN models against state-of-the-
art black-box attacks.

## References

  * [1] C. Chen, A. Seff, A. Kornhauser, and J. Xiao, “Deepdriving: Learning affordance for direct perception in autonomous driving,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2722–2730.
  * [2] G. Hinton, L. Deng, D. Yu, G. E. Dahl, A.-r. Mohamed, N. Jaitly, A. Senior, V. Vanhoucke, P. Nguyen, T. N. Sainath _et al._ , “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” _IEEE Signal processing magazine_ , vol. 29, no. 6, pp. 82–97, 2012.
  * [3] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [4] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” _arXiv preprint arXiv:1312.6199_ , 2013.
  * [5] B. Biggio, I. Corona, D. Maiorca, B. Nelson, N. Šrndić, P. Laskov, G. Giacinto, and F. Roli, “Evasion attacks against machine learning at test time,” in _Joint European conference on machine learning and knowledge discovery in databases_. Springer, 2013, pp. 387–402.
  * [6] M. Alzantot, Y. Sharma, S. Chakraborty, H. Zhang, C.-J. Hsieh, and M. B. Srivastava, “Genattack: Practical black-box attacks with gradient-free optimization,” in _Proceedings of the Genetic and Evolutionary Computation Conference_ , 2019, pp. 1111–1119.
  * [7] W. Brendel, J. Rauber, and M. Bethge, “Decision-based adversarial attacks: Reliable attacks against black-box machine learning models,” in _International Conference on Learning Representations_ , 2018.
  * [8] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh, “Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models,” in _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , 2017, pp. 15–26.
  * [9] C.-C. Tu, P. Ting, P.-Y. Chen, S. Liu, H. Zhang, J. Yi, C.-J. Hsieh, and S.-M. Cheng, “Autozoom: Autoencoder-based zeroth order optimization method for attacking black-box neural networks,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, 2019, pp. 742–749.
  * [10] A. Ilyas, L. Engstrom, A. Athalye, and J. Lin, “Black-box adversarial attacks with limited queries and information,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 2137–2146.
  * [11] S. Cheng, Y. Dong, T. Pang, H. Su, and J. Zhu, “Improving black-box adversarial attacks with a transfer-based prior,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 10 934–10 944.
  * [12] M. Cheng, T. Le, P.-Y. Chen, H. Zhang, J. Yi, and C.-J. Hsieh, “Query-efficient hard-label black-box attack: An optimization-based approach,” in _International Conference on Learning Representation (ICLR)_ , 2019.
  * [13] M. Cheng, S. Singh, P. H. Chen, P.-Y. Chen, S. Liu, and C.-J. Hsieh, “Sign-opt: A query-efficient hard-label adversarial attack,” in _International Conference on Learning Representations_ , 2019.
  * [14] Y. Li, L. Li, L. Wang, T. Zhang, and B. Gong, “Nattack: Learning the distributions of adversarial examples for an improved black-box attack on deep neural networks,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 3866–3876.
  * [15] J. Chen, M. I. Jordan, and M. J. Wainwright, “Hopskipjumpattack: A query-efficient decision-based attack,” in _2020 ieee symposium on security and privacy (sp)_. IEEE, 2020, pp. 1277–1294.
  * [16] C. Guo, J. Gardner, Y. You, A. G. Wilson, and K. Weinberger, “Simple black-box adversarial attacks,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 2484–2493.
  * [17] F. Tramèr, D. Boneh, A. Kurakin, I. Goodfellow, N. Papernot, and P. McDaniel, “Ensemble adversarial training: Attacks and defenses,” in _6th International Conference on Learning Representations, ICLR 2018-Conference Track Proceedings_ , 2018.
  * [18] J. Buckman, A. Roy, C. Raffel, and I. Goodfellow, “Thermometer encoding: One hot way to resist adversarial examples,” in _International Conference on Learning Representations_ , 2018.
  * [19] P. Samangouei, M. Kabkab, and R. Chellappa, “Defense-gan: Protecting classifiers against adversarial attacks using generative models,” in _International Conference on Learning Representations_ , 2018.
  * [20] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, “Distillation as a defense to adversarial perturbations against deep neural networks,” in _2016 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2016, pp. 582–597.
  * [21] Z. He, A. S. Rakin, and D. Fan, “Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 588–597.
  * [22] X. Wang, S. Wang, P.-Y. Chen, Y. Wang, B. Kulis, X. Lin, and S. P. Chin, “Protecting neural networks with hierarchical random switching: Towards better robustness-accuracy trade-off for stochastic defenses,” in _IJCAI_ , 2019.
  * [23] Z. Qin, Y. Fan, H. Zha, and B. Wu, “Random noise defense against query-based black-box attacks,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [24] F. Nesti, A. Biondi, and G. Buttazzo, “Detecting adversarial examples by input transformations, defense perturbations, and voting,” _arXiv preprint arXiv:2101.11466_ , 2021.
  * [25] B. Liang, H. Li, M. Su, X. Li, W. Shi, and X. Wang, “Detecting adversarial image examples in deep neural networks with adaptive noise reduction,” _IEEE Transactions on Dependable and Secure Computing_ , 2018.
  * [26] W. Fan, G. Sun, Y. Su, Z. Liu, and X. Lu, “Integration of statistical detector and gaussian noise injection detector for adversarial example detection in deep neural networks,” _Multimedia Tools and Applications_ , vol. 78, no. 14, pp. 20 409–20 429, 2019.
  * [27] B. Li, C. Chen, W. Wang, and L. Carin, “Certified adversarial robustness with additive noise,” in _Advances in Neural Information Processing Systems_. Neural information processing systems foundation, 2019.
  * [28] A. Athalye, N. Carlini, and D. Wagner, “Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples,” in _International conference on machine learning_. PMLR, 2018, pp. 274–283.
  * [29] N. Carlini and D. Wagner, “Adversarial examples are not easily detected: Bypassing ten detection methods,” in _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , 2017, pp. 3–14.
  * [30] ——, “Towards evaluating the robustness of neural networks,” in _2017 ieee symposium on security and privacy (sp)_. IEEE, 2017, pp. 39–57.
  * [31] M. Andriushchenko, F. Croce, N. Flammarion, and M. Hein, “Square attack: a query-efficient black-box adversarial attack via random search,” in _European Conference on Computer Vision_. Springer, 2020, pp. 484–501.
  * [32] G. K. Dziugaite, Z. Ghahramani, and D. M. Roy, “A study of the effect of jpg compression on adversarial images,” _arXiv preprint arXiv:1608.00853_ , 2016\.
  * [33] C. Xie, J. Wang, Z. Zhang, Z. Ren, and A. Yuille, “Mitigating adversarial effects through randomization,” in _International Conference on Learning Representations_ , 2018.
  * [34] F. Tramer, N. Carlini, W. Brendel, and A. Madry, “On adaptive attacks to adversarial example defenses,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [35] H. Salman, M. Sun, G. Yang, A. Kapoor, and J. Z. Kolter, “Denoised smoothing: A provable defense for pretrained classifiers,” _arXiv preprint arXiv:2003.01908_ , 2020.
  * [36] S. Chen, N. Carlini, and D. Wagner, “Stateful detection of black-box adversarial attacks,” in _Proceedings of the 1st ACM Workshop on Security and Privacy on Artificial Intelligence_ , 2020, pp. 30–39.
  * [37] H. Li, S. Shan, E. Wenger, J. Zhang, H. Zheng, and B. Y. Zhao, “Blacklight: Defending black-box adversarial attacks on deep neural networks,” _arXiv preprint arXiv:2006.14042_ , 2020.
  * [38] R. Pang, X. Zhang, S. Ji, X. Luo, and T. Wang, “Advmind: Inferring adversary intent of black-box attacks,” in _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, 2020, pp. 1899–1907.
  * [39] J. Cohen, E. Rosenfeld, and Z. Kolter, “Certified adversarial robustness via randomized smoothing,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 1310–1320.
  * [40] H. Salman, G. Yang, J. Li, P. Zhang, H. Zhang, I. Razenshteyn, and S. Bubeck, “Provably robust deep learning via adversarially trained smoothed classifiers,” _arXiv preprint arXiv:1906.04584_ , 2019.
  * [41] J. Byun, H. Go, and C. Kim, “Small input noise is enough to defend against query-based black-box attacks,” _arXiv preprint arXiv:2101.04829_ , 2021\.
  * [42] Z. Qin, Y. Fan, H. Zha, and B. Wu, “Theoretical study of random noise defense against query-based black-box attacks,” _arXiv preprint arXiv:2104.11470_ , 2021.
  * [43] M. Lecuyer, V. Atlidakis, R. Geambasu, D. Hsu, and S. Jana, “Certified robustness to adversarial examples with differential privacy,” in _2019 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2019, pp. 656–672.
  * [44] X. Liu, M. Cheng, H. Zhang, and C.-J. Hsieh, “Towards robust neural networks via random self-ensemble,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 369–385.
  * [45] C. Guo, M. Rana, M. Cisse, and L. van der Maaten, “Countering adversarial images using input transformations,” in _International Conference on Learning Representations_ , 2018.
  * [46] X. Cao and N. Z. Gong, “Mitigating evasion attacks to deep neural networks via region-based classification,” in _Proceedings of the 33rd Annual Computer Security Applications Conference_ , 2017, pp. 278–287.
  * [47] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _stat_ , vol. 1050, p. 20, 2015.
  * [48] F. Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein, “Robustbench: a standardized adversarial robustness benchmark,” in _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2021. [Online]. Available: https://openreview.net/forum?id=SSKZPJCt7B
  * [49] Y.-H. Wu, C.-H. Yuan, and S.-H. Wu, “Adversarial robustness via runtime masking and cleansing,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 10 399–10 409.
  * [50] M. Augustin, A. Meinke, and M. Hein, “Adversarial robustness on in-and out-distribution improves explainability,” in _European Conference on Computer Vision_. Springer, 2020, pp. 228–245.
  * [51] L. Rice, E. Wong, and Z. Kolter, “Overfitting in adversarially robust deep learning,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 8093–8104.
  * [52] G. W. Ding, Y. Sharma, K. Y. C. Lui, and R. Huang, “Mma training: Direct input space margin maximization through adversarial training,” _arXiv preprint arXiv:1812.02637_ , 2018.
  * [53] L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, B. Tran, and A. Madry, “Adversarial robustness as a prior for learned representations,” _arXiv preprint arXiv:1906.00945_ , 2019.
  * [54] J. Rony, L. G. Hafemann, L. S. Oliveira, I. B. Ayed, R. Sabourin, and E. Granger, “Decoupling direction and norm for efficient gradient-based l2 adversarial attacks and defenses,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4322–4330.
  * [55] V. Sehwag, S. Mahloujifar, T. Handina, S. Dai, C. Xiang, M. Chiang, and P. Mittal, “Improving adversarial robustness using proxy distributions,” _arXiv preprint arXiv:2104.09425_ , 2021.
  * [56] Y. Xu, L. Xie, X. Zhang, X. Chen, G.-J. Qi, Q. Tian, and H. Xiong, “Pc-darts: Partial channel connections for memory-efficient architecture search,” _arXiv preprint arXiv:1907.05737_ , 2019.

Supplementary Material

###  -A Supplementary Experiment Results for Defense Against Targeted Attacks

####  -A1 Median $l_{2}$ Distortion of Soft-Label Targeted Attacks

In Table VII, we have listed the median $l_{2}$ distortion of targeted soft-
label attacks when our proposed BD$(\theta,\sigma)$ defense was applied. Note
that when the ASR was 0%, then there were no successful adversarial samples
generated, and thus the $l_{2}$ distortion was absent. From Table VII, we can
see that there is no significant change in the median $l_{2}$ distortion when
comparing the cases with defense and without defense. This is the same as the
result of hard-label attacks shown in Table II.

TABLE VII: Median $l_{2}$ Distortion of Targeted Soft-Label Attacks under the BD$(\theta,\sigma)$ defense. $\sigma=0.1$ Dataset | Attacks | No defense |  $\theta_{1}$ = 0.5 |  $\theta_{2}$ = 0.7
---|---|---|---|---
| AZ | 0.0473 | 0.0478 | 0.0478
MNIST | GenAttack | 4.8086 | - | -
| SimBA | 0.0957 | 0.0694 | 0.0823
| AZ | 0.0672 | 0.0672 | 0.0672
CIFAR10 | GenAttack | 1.3874 | - | -
| SimBA | 0.0323 | 0.0236 | 0.0218
Dataset | Attacks | No Defense |  $\theta_{1}$ = 0.1 |  $\theta_{2}$ = 0.3
| AZ | 0.1566 | - | -
IMAGENET | GenAttack | 0.0377 | - | -
| SimBA | 0.0163 | 0.0092 | 0.0075

####  -A2 Visual Representation of Targeted Attack Samples

For the adversarial samples generated by the targeted attack algorithms under
our proposed defense, we examine whether the distortion, i.e., the difference
between the original sample and the adversarial sample, is imperceptible to
the human eye. Some of the adversarial samples are shown in Fig. 7 and 8. Fig.
7 shows the MNIST adversarial samples generated by the AZ, GenAttack, and
SimBA attack methods with our BD$(\theta,\sigma)$ defense. We can observe that
when the proposed BD method was used, the adversarial images generated were
extremely distorted.

Figure 7: Adversarial samples generated by the targeted soft-label attacks
under the BD($\theta,\sigma$) defense over the MNIST dataset.

In Section IV-C3, we showed the robust performance of the proposed BD method
against adaptive attacks, where we found that ASR stayed the same or just
increased slightly even under the adaptive attacks. For adaptive attacks that
changes the optimization step-size to increase ASR, some of the adversarial
samples in the presence of the BD method are illustrated in Fig 8. We can see
that the cost of slight increase of ASR was the heavy image distortion that
can be detected by human perception easily.

Figure 8: Adversarial samples generated by the GenAttack with adaptive attack
techniques (various optimization step-sizes) under the BD$(\theta,0.1)$
defense. Note that the default step-size used in the GenAttack was 0.3 for the
MNIST dataset.

###  -B Defense Performance against Untargeted Attacks

We evaluated the performance of the proposed BD method against untargeted
attacks with 1000 validation images of MNIST and 100 validation images of
IMAGENET. We considered the following untargeted black-box attacks: soft-label
attacks - AZ [9], SA(Square Attack) [31], and SimBA [16]; hard-label attacks -
BA(Boundary Attack) [7], Sign-OPT [13], and HSJA(HopSkipJump) [15]. Similar to
the targeted case, we adopted their original source code with the default
hyper-parameters and considered the $l_{2}$ norm setting throughout the
experiment.

Besides evaluating the defense performance of our BD method against the above
mentioned untargeted attacks, we have also compared the BD method’s
performance with other existing defense methods, including RND (Random Noise
Defense) [23],GF (Gaussian Augmentation Fine-Tuning) [23], FD (Feature
Denoising) [23], and AT (Adversarial Training) [47].

An important aspect to consider for the untargeted attack case is the
performance metric ASR. For the untargeted attacks, the attacker can simply
add enough noise to the original sample to make the adversarial sample satisfy
$\arg\max_{i}F_{i}({\bf X})\neq c$. Since all the defense methods in general
make the attack algorithms to generate samples with heavy distortion or noise,
the measure $\arg\max_{i}F_{i}({\bf X})\neq c$ tends to be satisfied by all
the adversarial samples trivially. As a result, the first ASR definitions we
used in the targeted attack case earlier in Section IV is not very meaningful.
In this subsection, we use the second ASR definition, i.e., ASR2, which was
defined in Section IV as the ratio of samples satisfying both
$\arg\max_{i}F_{i}({\bf X})\neq c$ and median $l_{2}$ distortion less than a
certain distortion threshold $L$.

To maintain a fair calculation of ASR with respect to the $l_{2}$ distortion
threshold $L$ for all the attack algorithms, we determined the threshold value
$L$ for all the attack methods from their no-defense results. Specifically,
with their original source code and default parameters, we determined a
minimum threshold that makes ASR to be near 100%. Then, we selected the $L$
value to be multiple folds of this minimum threshold value. For example, for
the AZ method we used distortion threshold $L=0.5$ to calculate ASR for MNIST
and $L=1.5$ for IMAGENET. These $L$ values were $10\times$ larger than the
minimum threshold. The main objective was to guarantee almost 100% ASR for all
the attack methods without defense. We used the same $L$ value to calculate
the ASR when the BD method was applied.

####  -B1 ASR of Soft-Label Untargeted Attacks

The ASR of the soft-label untargeted black-box attacks under our proposed BD
method is summarized in Table VIII. We have shown the experiment data for
$\sigma=0.1$ and varying $\theta$ values. We can see that when defending
against untargeted soft-label attacks, the performance of the BD method
indicated the same trend as what we observed for targeted attacks. The ASR
degraded drastically with the increase of $\theta$. BD$(0.7,0.1)$ defended the
MNIST DNN models against untargeted attacks as effectively as against targeted
attacks. However, for the IMAGENET DNN models, as compared to the targeted
attack case, we had to increase the threshold $\theta$. This is not surprising
considering the fact that, compared with the targeted attack, the untargeted
attack is usually easier for the attacker to be successful and is thus harder
for the defender to mitigate. BD$(0.4,0.1)$ reduced the ASR of AZ and SimBA to
0, and reduced the ASR of SA to 30%. Increasing $\theta$ further to 0.6 would
reduce the ASR of SA to 15% as shown in Table IX. Note that such a moderate
increase of $\theta$ would not lead to too large ACC degradation, as we have
evaluated in Section IV-B.

TABLE VIII: ASR (%) of Untargeted Soft-Label Attacks. $\sigma$ = 0.1. Dataset | Attacks | No defense |  $\theta_{1}$ = 0.5 |  $\theta_{2}$ = 0.7
---|---|---|---|---
| AZ | 97 | 0 | 0
MNIST | SA | 92 | 89 | 10
| SimBA | 82 | 31 | 0
Dataset | Attacks | No Defense |  $\theta_{1}$ = 0.3 |  $\theta_{2}$ = 0.4
| AZ | 100 | 0 | 0
IMAGENET | SA | 100 | 38 | 30
| SimBA | 100 | 96 | 0

####  -B2 ASR of Hard-Label Untargeted Attacks

Figure 9: ASR (%) vs. median $l_{2}$ distortion threshold $L$ of untargeted
hard-label attacks under the proposed BD method BD$(\theta,0.1)$.

We have summarized the ASR of hard-label untargeted attacks in the presence of
the proposed BD method in Fig. 9. As observed in the targeted attack case, the
ASR of the untargeted attack methods display a steady decline with the
increase of $\theta$.

####  -B3 Comparison with Other Defense Methods against Untargeted Attacks

TABLE IX: Comparison of the BD method with other defense methods against untargeted black-box attacks in terms of ACC degradation and ASR Dataset | Defense | Degradation | SA | SimBA
---|---|---|---|---
| Method | of ACC (%) | ASR (%) | ASR (%)
| RND [23] | -1.9 | 51.9 | 38.8
| GF [23] | -0.2 | 95.8 | 88.8
IMAGE- | FD [23] | -20.8 | 51.8 | 38.7
NET | AT [47] | -13.4 | 47.2 | 35
| RND+AT [23] | -16.8 | 20 | 4.7
| BD(0.4, 0.1) | -2.0 | 30 | 0
| BD(0.6, 0.1) | -3.5 | 15 | -

We have compared the performance of the BD method with some existing defense
methods for defending against untargeted black-box attacks. The performance
comparison data are illustrated in Table IX, where most of the data of the
existing defense methods were obtained from [23]. We can see that with 2 to
3.5% ACC degradation, the BD method could reduce ASR to near 0. The ASR of the
Square Attack was comparatively higher and all other defence methods were not
effective except RND+AT and our BD method. However, RND+AT suffered from a
heavy ACC degradation of -16.8%, which means this defense method is not useful
in practice. Our BD method could suppress the ASR to 15% using $\theta=0.6$
and $\sigma=0.1$ with only a small -3.5% ACC degradation. Thus, we can claim
that, in comparison with other defense methods, our BD method demonstrated
effective and superior performance to defend against untargeted attacks.

Figure 10: Original images ${\bf X}_{0}$, distortions $\Delta{\bf x}$ and
adversarial images ${\bf X}$ generated by the Square Attack (untargeted soft-
label attack) under the proposed defense BD$(\theta,\sigma)$ for
$\theta=\\{0,0.3,0.4\\}$, and $\sigma=0.1$.

####  -B4 Visual Representation of Untargeted Attack Samples

The visual representation of the adversarial samples generated by the
untargeted attack methods in the presence of the BD method is shown in Fig
10-12. Fig. 10 shows the results of the soft-label attack SA, where the left
column is the original image ${\bf X}_{0}$, the right column is the
adversarial image ${\bf X}$, and the middle column shows the distortion
$\Delta{\bf x}={\bf X}-{\bf X}_{0}$. Note that $\theta=0$ means no defense is
applied. In contrast, Fig. 11 and 12 are for the hard-label attacks, where the
first column shows the original images ${\bf X}_{0}$, and the rest columns
show the distortion $\Delta{\bf x}$.

Figure 11: Original images ${\bf X}_{0}$ and distortions $\Delta{\bf x}$
generated by the Boundary Attack (untargeted hard-label attack) under the
proposed defense BD$(\theta,\sigma)$ for $\theta=\\{0,0.1,0.4\\}$, and
$\sigma=0.1$. Figure 12: Original images ${\bf X}_{0}$ and distortions
$\Delta{\bf x}$ generated by the HopSkipJump Attack (untargeted hard-label
attack) under the proposed defense BD$(\theta,\sigma)$ for
$\theta=\\{0,0.1,0.4\\}$, and $\sigma=0.1$.

From these figures, we can see that for untargeted attacks, the attacker
needed higher distortion in order to generate a successful adversarial image
under our BD method. With the increase of $\theta$, the distortion level added
by the attacker increased rapidly, especially for hard-label attacks, which
resulted in highly degraded adversarial images.

###  -C Discussions & Limitations

While the experiment results we obtained strongly corroborate our main
hypothesis that addition of a small random noise to the logits when the query
sample reaches close to the decision boundary will successfully defend black-
box attacks, we have also found some interesting observations which will be
helpful for the future study of the defense methods. Amongst various soft-
label attacks that we experimented, we found that SimBA-DCT and NES-query-
limited for CIFAR-10, and IMAGENET data, respectively, demonstrated the
strongest performance against most of existing defense methods including our
proposed BD method. Similarly, when evaluating the defense performance against
hard-label attacks, we observed that the Boundary Attack and the HopSkipJump
Attack performed comparatively stronger for datasets with small image sizes
like MNIST & CIFAR-10 in the presence of the BD defense. However, from Table
II, we can see that the BD method demonstrated superior performance against
the HopSkipJump targeted attack over the IMAGENET dataset. We observed similar
trend in the performance of the BD method against untargeted black-box
attacks. While the untargeted attack setting is considered to be in the favor
of the attacker, defending against untargeted attacks is considered to be
tricky. The ASR of the untargeted attacks was comparatively higher than that
of the targeted attacks for the same defense parameter setting. But when
$\theta$ was increased, without too much ACC degradation, the ASR could still
drastically reduce to almost 0%. Our proposed BD method has demonstrated
robust performance against all the listed black-box attacks and we believe
that the performance will be consistent for any other or new black-box attack
methods. When compared with the existing defense methods, we observed that the
BD method presented better and more reliable performance.

It will be very interesting to evaluate the augmented performance of the BD
method in combination with some of the existing robust adversarial defenses.
We leave this part of the study for future work. Note that our main focus of
this paper is the query and search-based black-box attack method where both
the model and dataset are unknown to the attacker. We do not cover the
transfer-based attack method in this study (due to the inaccessibility of the
training set used for training of the surrogate model in this scenario). We
leave the study of the defense against transfer-based attack methods for
future work.


# Optimal Dosing Schedules for Substances Inducing Tolerance

Golrokh Nouri  Department of Mathematics, Simon Fraser University, Burnaby, BC
V5A 1S6, Canada  Address correspondence to<EMAIL_ADDRESS><EMAIL_ADDRESS>Paul Tupper  Department of Mathematics, Simon Fraser University, Burnaby, BC
V5A 1S6, Canada  Address correspondence to<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Many drugs used therapeutically or recreationally induce tolerance: the effect
of the substance decreases with repeated use. This phenomenon may reduce the
efficacy of the substance unless dosage is increased beyond what is healthy
for the individual. Restoring the effect of the substance can often be
obtained by taking a break from consumption. We propose designing dosing
schedules that maximize the desired effect of the substance with a given total
consumption, while factoring in the effect of tolerance. We provide a simple
mathematical model of response to consumption and tolerance that can be fit
from data on substance administration and response. Using this model with
given parameters, we determine optimal consumption schedules to maximize a
given objective. We illustrate with the example of caffeine, where we provide
a schedule of consumption for a user who values the effects of caffeine on all
days but needs extra alertness on some days of the week.

##  1 Introduction

Drug tolerance is the phenomenon where repeated use of a substance leads to
diminished effects in an individual. The rate at which tolerance develops
depends on the drug, the frequency and dose with which it is administered, the
metabolism of the individual, and many other factors. Tolerance is important
in medicine, as it means that drug treatments with constant dosage may become
less effective over time. Clinicians have to decide when to increase dosage
and by how much, balancing the negative side effects of the substance.
Likewise, tolerance plays a role in addiction: initially a substance provides
a pleasurable response in a user, but over time, the same dose no longer
produces the same effect, and the negative effects of withdrawal occur if
consumption is stopped altogether [11, 7].

Drug tolerance can be acute or chronic, depending on the time-span in which it
develops, with acute tolerance occurring after even a single dose
(tachyphylaxis), and chronic occurring over longer periods of consumption [11,
19]. Tolerance can be categorized in other ways as well. Pharmacokinetic
tolerance refers to the body developing ways to prevent as much of the
substance from reaching the targeted area, such as by clearing it faster from
the body [4]. Pharmacodynamic tolerance, on the other hand, develops as a
result of the body’s homeostatic response to chemical imbalances, for example
by down regulating the receptors that pair to a particular substance. Another
categorization is based on the different biological levels in play since drug
tolerance can affect molecular, cellular, or behavioural processes in the
body. Regardless of the specific type or level at which tolerance works, in
this paper we focus on the broad features of tolerance at whatever level it
occurs, a reduction in response to a drug due to repeated exposure [11, 19].

Tolerance is familiar to many people from their experience with the most
commonly consumed recreational drug in the world: caffeine, consumed in the
form of tea, coffee, and soft drinks [13]. The effects of caffeine on first
consumption are increased blood pressure, improvement in vigilance, and
increased alertness and ability to sustain attention, especially in low-
arousal situations like early mornings [14, 8]. It does so by blocking
adenosine receptors in the brain [5]. Adenosine is a naturally occurring
compound released by the brain to promote sleep and suppress arousal. But with
a steady daily consumption of coffee the energizing effects of caffeine no
longer occur, and in fact when coffee is stopped for a day, the person has the
negative side effects of withdrawal such as fatigue and headaches [8, 6].
Tolerance occurs for caffeine because the body produces more adenosine
receptors, a phenomenon called receptor up-regulation [1]. Tolerant people
therefore need a higher caffeine concentration to block these receptors, in
order to obtain the same level of effect as before.

Psychology has a theory of the development of tolerance, opponent process
theory, that can be applied to any stimulus, not just the use of substances
[16, 15]. According to this theory, when a subject is exposed to a stimulus
that induces an affective reaction, there is a balancing opponent process
initiated that is slower than the initial response. This opponent process
causes the reaction to diminish while the stimulus is still present, and for
there to be an opposite reaction after the stimulus is removed. With repeated
exposure, the primary process remains of the same magnitude while the opponent
process becomes stronger. The effect is that after repeated exposure to the
stimulus the primary reaction appears weaker and the opposite reaction after
the stimulus is removed becomes stronger. We think of opponent process theory
as a higher level theory that encompasses many more detailed theories of
tolerance. Our model of tolerance is inspired by the description in opponent
process theory, though we show that it can be applied by fitting to individual
data for different drugs.

Our first contribution is to provide a simple mathematical model of reaction
to substance that is consistent with the well-known facts of tolerance to
substance use. Our model includes two different mechanisms for tolerance. The
first mechanism accounts for acute tolerance, and is borrowed from Porchet et
al. [12]. It accounts for the effect where after a first dose gives an effect
of a given size, the second dosage of the same size gives a reduced effect on
the user. This mechanism does not lead to a withdrawal or rebound symptoms.
The second mechanism accounts for long-term tolerance, and is inspired by
opponent process theory. This is where continual use of a substance affects
the user so that a change is measurable even when the substance has not
recently been used. In particular, this mechanism accounts for withdrawal.
Both mechanisms can be turned off or on independently of each other by an
appropriate choice of parameters. Though for most substances we expect both
mechanisms to be present, we demonstrate the applicability of the model with
only one mechanism at a time. We use two different parameters settings, one
describing acute tolerance to nicotine [12], and another describing long-term
tolerance to caffeine [6].

Our second contribution shows how once we have a set of parameters for our
model, we can use optimization to design optimal dosing schedules for a
substance. For example, a subject may have chronic pain, and use an analgesic
to provide relief. Continual use without breaks may lead to a state where the
subject’s experience of pain is not much less than originally without the
drug. Does providing breaks in the dosing schedule make sense as a strategy to
minimize tolerance? We do a detailed exploration with caffeine, imagining a
coffee drinker who wants to be alert on particular days of the week more than
others, and has a maximum total amount of caffeine that they want to consume
per week.

Our model is similar to those developed by Peper [9, 10] and Porchet et al.
[12], though in particular it is simpler than Peper’s. The simplicity allows
us to determine easily interpretable parameters from available data, and means
that the computational demands of simulating the data are light enough to
enable optimization of dosing strategies.

##  2 Model

The model is a system of differential equations together describing the
response of a subject to an administered substance over time.

First, we let $D(t)$ represent the rate at which the drug is administered at
time $t$, in units of mass per time. Often, in practice $D(t)$ will be zero
most of the time except for some short intervals where it is positive, when
the substance is actually being consumed. We let the function $C(t)$ represent
the blood plasma concentration of the substance at time $t$ in units of mass
per volume. The dynamics of $C(t)$ over time is given by:

$\frac{d}{dt}C(t)=k_{1}(k_{7}D(t)-C(t))$ (1)

Here, the first term $k_{1}k_{7}D(t)$ represents the administration of the
drug increasing the blood plasma level, while the $k_{1}C(t)$ term represents
the excretion of the substance from the body. A constant dose $D$ leads to an
equilibrium blood concentration of $C=k_{7}D$, and $k_{1}$ determines the rate
at which the system approaches this equilibrium.

For tolerance to develop over time, the body needs some measure of memory of
the substance previously administered. We model this with the quantity
$C_{\footnotesize\mbox{mem}}(t)$ where:

$\frac{d}{dt}C_{\footnotesize\mbox{mem}}(t)=k_{5}(C(t)-C_{\footnotesize\mbox{mem}}(t))$
(2)

For $C_{\footnotesize\mbox{mem}}(0)=0$ it is straightforward to check that

$C_{\footnotesize\mbox{mem}}(t)=k_{5}^{-1}\int_{0}^{t}e^{-k_{5}(t-s)}C(s)\,ds$

so $C_{\footnotesize\mbox{mem}}(t)$ can be interpreted as an average of $C(t)$
over the time leading up to $t$, with more recent values being more heavily
weighted. The parameter $k_{5}$ controls how long this memory is, with
$C_{\footnotesize\mbox{mem}}$ being a weighted average of $C(t)$ over a time
period of order $1/k_{5}$. $C_{\footnotesize\mbox{mem}}(t)$ could either
represent the concentration of the substance in some longer lasting reservoir
than the blood, or some other consequences of the body’s memory of the
substance, for example, the number of extra receptors grown for the substance.

The quantity $E(t)$ is the effect under consideration of the substance at time
$t$. For example, for caffeine, $E(t)$ may be alertness; for an analgesic,
$E(t)$ may be a level of pain. Our model for $E(t)$ as a function of time is

$E(t)=E_{b}(t)+\frac{F(t)}{1+C_{\footnotesize\mbox{mem}}(t)/C_{\footnotesize\mbox{half}}}$
(3)

where $F(t)$ is an idealized effect of the drug (without tolerance) and
$E_{b}(t)$ is the current baseline value of $E$, each of which have their own
dynamics which we will describe below.

Our two mechanisms of tolerance work to push $E(t)$ away from desired values
and can be seen in (3). The first mechanism, for acute tolerance, acts through
the denominator of the second term on the right. If
$C_{\footnotesize\mbox{mem}}=0$ then this term is at its maximum value:
$F(t)$. Any larger value of $C_{\footnotesize\mbox{mem}}$ reduces this term,
and therefore makes $E(t)$ closer to $E_{b}(t)$. The strength of this effect
is controlled by $C_{\footnotesize\mbox{half}}$. When
$C_{\footnotesize\mbox{mem}}=C_{\footnotesize\mbox{half}}$ then the effect of
$F$ is halved. We describe the dynamics of $F$ below. If we want to turn off
this mechanism, we set $C_{\footnotesize\mbox{half}}=\infty$. The second
mechanism, for long-term tolerance, works through the term $E_{b}(t)$. Below
we will describe the dynamics of $E_{b}$, where exposure to the drug shifts
its value in order to move $E(t)$ away from more desirable values.

We model the dynamics of $F$ by

$\frac{d}{dt}F(t)=k_{2}(k_{6}C(t)-F(t))$ (4)

where $k_{2}$ is the rate at which $F$ converges to its value determined by
$C$, and $k_{6}$ determines the relationship between concentration of the drug
$C$ and underlying effect $F$.

We model the dynamics of $E_{b}$ by

$\frac{d}{dt}E_{b}(t)=k_{3}(E_{0}-k_{4}C_{\footnotesize\mbox{mem}}(t)-E_{b}(t)).$
(5)

For a given subject, we hypothesize that there is an underlying baseline level
of this quantity $E_{0}$. Note that if $C_{\footnotesize\mbox{mem}}=0$, so
that the drug and any memory of it is cleared from the body, $E_{b}(t)$
converges to $E_{0}$ with rate $k_{3}$. Likewise, if
$C_{\footnotesize\mbox{mem}}(t)$ is constant, $E_{b}$ converges to
$E_{0}-k_{4}C_{\footnotesize\mbox{mem}}$, so $k_{4}$ determines just how much
memory of the substance shifts the baseline value of $E$. Setting $k_{4}$ to
zero turns off the long-term tolerance mechanism.

Note that through appropriate parameter choices, our model can incorporate
both tolerance mechanisms ($k_{4}>0$, $C_{\footnotesize\mbox{half}}<\infty$)
just long-term tolerance ($k_{4}>0$, $C_{\footnotesize\mbox{half}}=\infty$),
just acute tolerance ($k_{4}=0$, $C_{\footnotesize\mbox{half}}<\infty$), or
neither ($k_{4}=0$, $C_{\footnotesize\mbox{half}}=\infty$).

###  2.1 A single rapid dose, with no tolerance

First we see what our model does in the situation where a dose is delivered
very quickly, and we do not consider tolerance of either sort. This
corresponds to $C_{\footnotesize\mbox{half}}=\infty$ and $E_{b}=E_{0}$.
Letting $\tau$ be a short interval of time, and $\mathcal{D}$ be the total
dose, we have $D(t)=\mathcal{D}/\tau$ for $t\in[0,\tau]$ and $0$ otherwise.
Roughly, neglecting the decay term, $dC/dt=k_{1}k_{7}D(t)$ over time interval
$[0,\tau]$ and so if $C(0)=0$ we get
$C(\tau)=\tau(k_{1}k_{7}\mathcal{D}/\tau)=k_{1}k_{7}\mathcal{D}$. After time
$\tau$, $D$ is zero, and so the equation reduces to $dC/dt=-k_{1}C(t)$. Hence
$C(t)=k_{1}k_{7}\mathcal{D}e^{-k_{1}(t-\tau)}$. Letting $\tau$ go to zero
gives

$C(t)=k_{1}k_{7}\mathcal{D}e^{-k_{1}t}$ (6)

after a dose given at time 0. With neither form of tolerance included in the
model, we have that $E(t)=E_{0}+F(t)$ where $F(t)$ satisfies (4). If $k_{2}$
is large, then $E$ will rapidly converge to $E_{0}+k_{6}C(t)$ on short time
intervals. If we use the small $t$ approximation in (6) we get that $E$ is
approximately $E_{0}+k_{6}k_{1}k_{7}\mathcal{D}$ at its peak immediately after
the dose has taken its effect, and then decays to $E_{0}$ at approximately the
same rate as $C(t)$.

###  2.2 Constant Dosage, Long-Term Tolerance

In order to get rough analytical results for the development of long-term
tolerance over time, we consider the case where $D(t)=D$ is constant. We let
$C_{\footnotesize\mbox{half}}=\infty$ so that $E=E_{b}+F$. Then $C(t)$
converges to $k_{7}D$ over a time period of scale $k_{1}^{-1}$. In this case
$C_{\footnotesize\mbox{mem}}(t)$, being an average of $C(t)$, will also
converge to $k_{7}D$, and assuming $k_{5}\ll k_{1}$, this will occur on the
time scale $k_{5}^{-1}$. The same applies for setting $D$ to zero after a
period of substance use. Blood plasma concentration will return to $0$ on a
time scale of $k_{1}^{-1}$ and $C_{\footnotesize\mbox{mem}}$ will return to
$0$ on a time scale of $k_{5}^{-1}$.

Now suppose that $C_{\footnotesize\mbox{mem}}$ has equilibrated to $k_{7}D$
and remains constant. Then $E_{b}$ will equilibriate to
$E_{0}-k_{4}C_{\footnotesize\mbox{mem}}=E_{0}-k_{4}k_{7}D$ over a time of
scale $k_{3}^{-1}$. Likewise, $E(t)$ will equilibriate to
$E_{b}+k_{6}C=E_{0}-k_{4}k_{7}D+k_{6}k_{7}D$. This first term $E_{0}$ is the
subject’s intrinsic base line, the second term is the effect of tolerance, and
the third term is the direct effect of the drug. For most substances we expect
$k_{4},k_{6}>0$ so that the drug increases $E$, but the effect of tolerance is
to push in the opposite direction. Depending of the relative size of $k_{4}$
and $k_{6}$, the net effect of the drug (including tolerance) can increase or
decrease $E$.

To a rough approximation we can see that these equilibrium calculations fit
the basic phenomenology of substance use and tolerance. Before tolerance has
time to develop, steady use of the substance changes the value of $E$ from
$E_{0}$ to $E_{\footnotesize\mbox{drug}}=E_{0}+k_{6}k_{7}D$. Over the longer
term, tolerance develops and the new value for $E$ is
$E_{\footnotesize\mbox{drug+tol}}=E_{0}-k_{4}k_{7}D+k_{6}k_{7}D$, which is
reduced relative to the initial effect of the drug. Now, if the drug is
suddenly removed, $E$ goes to the value
$E_{\footnotesize\mbox{drug}}=E_{0}-k_{4}k_{7}D$, which is even worse than
without the drug at all. However, eventually the effect of tolerance will wear
off, and $E$ will return to $E_{0}$.

These considerations give us a formula for estimating $k_{4}$.

$k_{4}=k_{6}\frac{E_{\footnotesize\mbox{drug}}-E_{\footnotesize\mbox{drug+tol}}}{E_{\footnotesize\mbox{drug}}-E_{0}}$
(7)

##  3 Examples

We now explore the behaviour of our model for two sets of parameter values;
one selected to capture the effects of nicotine consumption over the course of
a few hours [12], another that of caffeine over many days [6].

###  3.1 Nicotine

First we demonstrate our model with the development of acute (over a short
time period) tolerance with nicotine. Porchet et al. [12] administered
nicotine intravenously with two doses in short succession to a subject, and
measured the effect on blood concentration of nicotine and on heart rate. One
effect of nicotine was to increase the subjects’ heart rate from a baseline
value. They found that the second dose led to a lower peak in heart rate
compared to the first, indicating the development of acute tolerance. We
attempted to fit this data with our model. Since the experiment occurred over
a few hours, we turned off our longer-term tolerance mechanism by setting
$k_{3}=k_{4}=0$, and hence $E_{b}(t)=E_{0}$ for all time. We selected other
parameters by hand in order to get a reasonable fit to Porchet et al.’s data
(the average measurements over eight subjects). The parameters are shown in
Table LABEL:table:_CafNicParameters and the fits are shown in Figure 1.

We see that we are able to capture the main features of the experimental data.
Importantly, we see that there is no withdrawal or rebound effect after the
consumption of all the nicotine: the heart rate does not drop below the inital
heart rate of the subject. Earlier we attempted to capture the short term
acute tolerance evident here with a shift in the value of $E_{b}$. We found
that this always led to a withdrawal effect that is not apparent in this data.
Nicotine withdrawal does indeed indeed lead to lower than normal heart rate
(bradycardia) [3], but we could not find time series data showing this.

The main shortcoming of our model in fitting $C(t)$ as a function of time is
that our model predicts a faster decay of concentration after both doses are
complete. Porchet et al. are able to capture this better with their similar
model, as they includes a second compartment in the pharmacokinetic model,
something that could be added to our model if desired.

Table 1: Parameters used in caffeine and nicotine example Parameter | $E_{0}$ | $k_{1}$ | $k_{2}$ | $k_{3}$ | $k_{4}$ | $k_{5}$ | $k_{6}$ | $k_{7}$ | $C_{\footnotesize\mbox{half}}$
---|---|---|---|---|---|---|---|---|---
Units |  | 1/min | 1/min | 1/day | mL/$\mu$g | 1/day | mL/$\mu$g | min/ mL |  $\mu$g/mL
Nicotine values | 60 | 0.014 | 0.08 | 0 | 0 | 20. | 1.8 $\times 10^{3}$ | 0.0175 | 0.005
Caffeine values | 0 | 0.002 | 0.1 | 0.5 | 0.3 | 0.5 | 0.4 | 0.0125 | $\infty$

Figure 1: A fit of the model to experimental data from Porchet et al. [12].
Parameters are listed in Table LABEL:table:_CafNicParameters.

###  3.2 Caffeine

We next consider longer term effects of caffeine consumption. We chose
parameters to roughly correspond to the effects of caffeine on a typical
person. For example, peak plasma caffeine concentration from consuming
$\mathcal{D}=$100 $\mathrm{m}\mathrm{g}$ of caffeine (equivalent to about 1
cup of coffee [18]) has a value of about $C=$ 2.5
$\mu\mathrm{g}\mathrm{/}\mathrm{m}\mathrm{L}$ and then the concentration
decays with a half-life of about 5 hours, according to a study by
Teekachunhatean et al. [17]. This means that the rate of decay is
approximately $k_{1}=\ln 2/(300$$\mathrm{m}\mathrm{i}\mathrm{n}$) =0.002
$\mathrm{m}\mathrm{i}\mathrm{n}^{-1}$. If we then select $k_{7}$ in Equation 6
to match these values we get $k_{7}=$2.5
$\mu\mathrm{g}\mathrm{/}\mathrm{m}\mathrm{L}$ $/(k_{1}10^{5}$
$\mu\mathrm{g}$$)=$0.0125
$\mathrm{m}\mathrm{i}\mathrm{n}\mathrm{/}\mathrm{m}\mathrm{L}$. Caffeine has a
variety of effects, and rather than selecting one we take $E(t)$ to be some
form of “alertness”, a dimensionless quantity going up to a peak value of near
1, and having a baseline of 0. This sets the value of $k_{6}=1/(2.5$
$\mu\mathrm{g}\mathrm{/}\mathrm{m}\mathrm{L}$)$=0.4$
$\mathrm{m}\mathrm{L}\mathrm{/}\mu\mathrm{g}$. We do not observe big
differences reported between times of peak concentration of caffeine and peak
psychological effect [2], so we assume that it is relatively fast, and set
$k_{2}=$0.1 $\mathrm{m}\mathrm{i}\mathrm{n}^{-1}$.

To estimate parameter values related to the development of tolerance for
caffeine, we look to Lara et al. [6]. They study the effect of caffeine on
exercise, and how this diminishes after repeated use. We summarize the results
of one of their experiments as shown in [6, Fig. 2]. Looking at this figure we
obtain rough estimates of $E_{\footnotesize\mbox{drug}}$ and
$E_{\footnotesize\mbox{drug+tol}}$ as follows. After a caffeine free period,
athletes’ performance in an exercise test was measured; in researchers’
normalized units $E_{0}=0$. Then they began a daily routine of caffeine
consumption followed by the same exercise test. Initially there is a
significant improvement in performance $E_{\footnotesize\mbox{drug}}\approx
2$, but this declines over time, about halving over 10 days. We estimate that
the eventual performance under continuing caffeine would be
$E_{\footnotesize\mbox{drug+tol}}\approx 0.5$. Then after 10 days, the daily
caffeine is stopped, and the subjects’ performance on that day declines to be
less than it was initially without caffeine (appoximately -0.5), indicating a
withdrawal effect.

Since their measurements are only once per day, we don’t have any way to
measure acute tolerance. Hence we set $C_{\footnotesize\mbox{half}}=\infty$,
therefore turning of that mechanism. The data doesn’t allow us to distinguish
between the time constants for $C_{\footnotesize\mbox{mem}}$ and $E_{b}$, so
we assume
$k_{3}=k_{5}=1/(2\,$$\mathrm{d}\mathrm{a}\mathrm{y}\mathrm{s}$$)=0.5$
$\mathrm{d}\mathrm{a}\mathrm{y}\mathrm{s}^{-1}$. Using our estimates for
$E_{0},E_{\footnotesize\mbox{drug}},E_{\footnotesize\mbox{drug+tol}}$ above in
(7) we get that
$k_{4}=k_{6}1.5/2=0.3$$\mathrm{m}\mathrm{L}\mathrm{/}\mu\mathrm{g}$. The
complete list of the parameters can be found in Table
LABEL:table:_CafNicParameters.

We use these parameters to explore the behaviour of our model for caffeine. We
introduce three different dosing regimens and observe the changes in blood
caffeine concentration $C(t)$, alertness $E(t)$, and the changes in the
baseline alertness $E_{b}(t)$ for each regime. For all three dosing regimens,
the experiment will run for a total of 4 weeks, and the total amount of
caffeine consumed will be fixed at 2800 mg (28 cups of coffee) over the whole
duration of the experiment.

The first dosing regimen (Figure 2, left) is to consume one cup of coffee
(100mg) on a daily basis for the entirety of the experiment. The subject
consumes it starting at noon over a 15 minute period. We observe that the
effect $E(t)$ peaks shortly after consumption each day. The highest peak
occurs on the first day of consumption. Peak height lowers on each subsequent
day, assymptoting to a lower value after a couple of weeks. This is reflected
in $E_{b}$ decaying from 0 to approximately $-0.25$ and remaining there.
Another feature is that $E$ is eventually negative at times during the day.

In the second regime (Figure 2, centre) the subject has two cups of coffee per
day for two weeks and then stops completely. The higher dose leads to higher
peaks, which similarly lessens in height with each day. When the subject stops
consumption, alertness drops to below 0 consistently for many days. But this
slowly wears off and alertness returns to the zero baseline.

The last dosing regime we will look at is to consume 140 $mg$ of caffeine on
weekdays, followed by no consumption over the weekend in order to prevent
tolerance developing. This does indeed lead to a higher peak on the Monday of
the second week. This is partly due to there being a higher consumption on
that day in this regime, but also the diminished effect of tolerance from
consumption on the weekend.

Figure 2: Caffeine plasma level $C(t)$, alertness $E(t)$, and baseline
alertness $E_{b}(t)$ over a four-week period for three dosing regimens.

##  4 Optimization of Dosing Schedule

We next consider choosing the optimal dosing regime to obtain a desired
outcome. As in our previous example shown in Fig. 2, we imagine a subject who
consumes caffeine in the form of coffee, with the goal of being more alert at
specific times during the week. We use our model with the parameters for
caffeine we determined earlier, as shown in Table
LABEL:table:_CafNicParameters. Let us assume that our subject has important
meetings on Mondays and Thursdays and thus needs to be very alert on those
days, but enjoys the effect of caffeine on all days of the week. As in the
simulations shown in Figure 2, the subject is given caffeine every day from
noon to 12:15pm. We select the strength of the dose $d_{i}$ on day $i$ for
$i=1,\ldots,7$, in units where 1 indicates a single cup of coffee. Starting
from a state with no caffeine in the system
$C=C_{\footnotesize\mbox{mem}}=E_{b}=E=0$ we run the simulation for three
weeks. We choose the dose on each day to maximize the following objective
function. On the last week of the simulation, on each day at 3pm we measure
the alertness $E$ of the subject, $e_{i}$ for $i=1,\ldots,7$. We take the
weighted average of the alertness measurements over the week

$f=\sum_{i=1}^{7}{w_{i}\sqrt{e_{i}}}$ (8)

We model our subject’s desire to be more alert on Mondays and Thursdays by
setting $w_{1}=w_{4}=10$ and all other days of the week $w_{i}=0.2$. We use
the square root in the objective function to model diminishing marginal
benefit from increasing levels of alertness; otherwise, the optimum schedule
is always to consume as much coffee as possible on the day with the greatest
weight assigned to it.

If there are no constraints on the dose given every day of the week, then $f$
can be made arbitrarily large by giving the subject more and more coffee. So
we consider two different constraints on coffee consumption. In the first, the
_daily constraint_ , the subject doesn’t consume more than 2 cups of coffee a
day. In the second, the _weekly constraint_ , the subject doesn’t consume for
than 10 cups of coffee each week.

Initially, we used an adaptive solver for our system of differential
equations, which makes use of changing step sizes to produce more accurate
results more efficiently. This worked well when testing the model and fitting
it to data, but not when used in optimization, since the adaptive step sizes
led to the computed $f$ not being a continuous function of the dosing
schedule. So, we used the fixed-step forward Euler method with a small step
size to solve the system of differential equations within the optimization.

We initialized the optimizer with dose $d_{i}$ for $i=1,\ldots,7$ drawn
independently and uniformly at random in $[0,1]$. The optimizer was run to
convergence. In Table LABEL:tab:optimal_sched we show the optimal schedules
determined for four different conditions. We varied whether we included long-
term tolerance in the model ($k_{4}=0.3$ mL/$\mu$g) or not ($k_{4}=0$), and
whether we used the daily constraint or the weekly constraint. In Table
LABEL:tab:optimal_sched we show the optimal schedule for the four different
conditions. In Figure 3 we show the resulting alertness $E$ versus time in the
last week of the simulation with the optimized schedule.

The results without tolerance are straightforward to understand. With a daily
maximum it is always best to consume as much coffee on every day. This
naturally yields the same alertness peak on each day. With a weekly max it is
best to split almost all coffee consumption equally between the two days when
it is needed most, though interestingly 1/1000 of a cup of coffee is
recommended on Sundays, presumably because some alertness is desired on the
other days, and Sunday is the day with the least residual effect from Monday
and Thursday.

Results change when we add tolerance. With a daily maximum the optimal choice
is to maximize coffee consumption to 2 cups on Monday and Thursday, but to
have significantly reduced consumption on other days of the week: between 20%
and 40% of a cup on these days. The model determines this to be the optimal
tradeoff between being alert on these days and not having tolerance deprive
the subject of the benefits of caffeine on the important days. With the weekly
maximum now all the consumption occurs on Monday and Thursday. Slightly more
caffeine is recommended on Thursday than Monday, and no caffeine consumption
at all is recommended on other days.

Table 2: Optimal dosing schedule for four different conditions Condition | Day 1 | Day 2 | Day 3 | Day 4 | Day 5 | Day 6 | Day 7
---|---|---|---|---|---|---|---
no tolerance, daily max | 2.0000 | 2.0000 | 2.0000 | 2.0000 | 2.0000 | 2.0000 | 2.0000
with tolerance, daily max | 2.0000 | 0.2179 | 0.3921 | 2.0000 | 0.2456 | 0.3275 | 0.3568
no tolerance, weekly max | 4.9997 | 0.0000 | 0.0000 | 4.9990 | 0.0000 | 0.0000 | 0.0013
with tolerance, weekly max | 4.9180 | 0.0000 | 0.0000 | 5.0820 | 0.0000 | 0.0000 | 0.0000
Figure 3: Alertness versus time over the course of a week of caffeine
consumption with optimized schedules. Left column, results for the model
without tolerance; right column, with tolerance. Top row, consumption
restricted to at most two cups of coffee a day; bottom row, consumption
restrictred to ten cups of coffee a week.

##  5 Discussion

We have presented a simple model of the effect of a drug on an individual and
the subsequent development of tolerance. We fit the model to two sets of data,
one showing the effect of acute tolerance to nicotine and the other showing
long-term tolerance to caffeine. Our model includes two different mechanisms
for tolerance, one for acute and one for long-term. We found that we could not
fit both datasets with a model with a single mechanism. In particular, our
acute tolerance mechanism does not exhibit withdrawal, and so cannot match
what was observed for caffeine [6]. On the other hand, our long-term tolerance
model based must exhibit withdrawal after removal of the substance, and so
cannot match the data for nicotine from Porchet et al [12]. We expect that
most substances will exhibit both of these forms of tolerance, and direction
for future work is to derive parameters for the model for a substance
incorporating both of these effects, once such data is available.

As with any model there are phenomena that it will not be able to account for.
Most obviously from our fit to nicotine data above, our single compartment
pharmacokinetic model prevents us from accurately capturing the decay of
nicotine concentration over time. This could be fixed easily (at the cost of
adding more parameters). A more serious problem is the difficulty of getting
sufficient data to fit all the parameters. Above we used a combination of fits
to data and educated guesses. But in reality our model, and our choices of
parameters, need validation against a wide range of experimental data before
being used as a basis for application in physiology.

Our second contribution is to show that given such a model with appropriate
parameters we can use optimization to determine optimal dosing schedules for
particular goals. We only considered a case where long-term tolerance (over
the course of days) was relevant, but our methods are in principle applicable
to more short term situations where acute tolerance is more important. For
example, if a subject is competing in a day-long chess tournament, determining
what is the optimal consumption of nicotine or caffeine for performance.

We chose the familiar example of caffeine to illustrate our method for dosing
schedule optimization, but there are many more important instances that come
to mind, such as in medicine and in addiction management. Ideally such
considerations could inform treatment design for these problems. But the
soundness of the recommendations relies on the soundness of the underlying
model. There is also the problem of turning what is wanted from a dosing
regime (such as freedom from pain) into an objective function to be optimized.
Our hope though, is that by using models such as ours, and exploring optimal
dosing schedules in different regimes, ideally in collaboration with a
clinician, we will be able to learn something of the issues involved in
managing drug treatment.

## Data and code availability

All data and code for this study are available in the GitHub repository:
https://github.com/PaulFredTupper/optimal-dosing-schedules

## Acknowledgements

This study was funded by the Natural Science and Engineering Research Council
(Canada) Discovery Grant (RGPIN-2019-06911).

## References

  * [1] HPT Ammon.  Biochemical mechanism of caffeine tolerance.  Archiv der Pharmazie, 324(5):261–267, 1991.
  * [2] Roberto Corti, Christian Binggeli, Isabella Sudano, Lukas Spieker, Edgar Hänseler, Frank Ruschitzka, William F Chaplin, Thomas F Lüscher, and Georg Noll.  Coffee acutely increases sympathetic nerve activity and blood pressure independently of caffeine content: role of habitual versus nonhabitual drinking.  Circulation, 106(23):2935–2940, 2002.
  * [3] John A Dani, Daniel Jenson, John I Broussard, and Mariella De Biasi.  Neurophysiology of nicotine addiction.  Journal of addiction research & therapy, Suppl 1(1), 2011.
  * [4] Emily O Dumas and Gary M Pollack.  Opioid tolerance development: a pharmacokinetic/pharmacodynamic perspective.  The AAPS journal, 10:537–551, 2008.
  * [5] Gilberto Fisone, Anders Borgkvist, and Alessandro Usiello.  Caffeine as a psychomotor stimulant: Mechanism of action.  Cellular and molecular life sciences : CMLS, 61:857–72, 05 2004\.
  * [6] Beatriz Lara, Carlos Ruiz-Moreno, Juan José Salinero, and Juan Del Coso.  Time course of tolerance to the performance benefits of caffeine.  PLOS ONE, 14(1):1–18, 01 2019.
  * [7] Shalini S Lynch.  Tolerance and resistance.  Merck Manual, 2022.
  * [8] Peter Nawrot, S Jordan, J Eastwood, J Rotstein, A Hugenholtz, and M Feeley.  Effects of caffeine on human health.  Food Additives & Contaminants, 20(1):1–30, 2003.
  * [9] Abraham Peper.  A theory of drug tolerance and dependence i: a conceptual analysis.  Journal of Theoretical Biology, 229(4):477–490, 2004.
  * [10] Abraham Peper.  A theory of drug tolerance and dependence ii: the mathematical model.  Journal of theoretical biology, 229(4):491–500, 2004.
  * [11] Andrzej Z Pietrzykowski and Steven N Treistman.  The molecular basis of tolerance.  Alcohol Research & Health, 31(4):298, 2008.
  * [12] Heave C Porchet, Neal L Benowitz, and Lewis B Sheiner.  Pharmacodynamic model of tolerance: application to nicotine.  Journal of Pharmacology and Experimental Therapeutics, 244(1):231–236, 1988.
  * [13] Antonella Samoggia and Tommaso Rezzaghi.  The consumption of caffeine-containing products to enhance sports performance: an application of an extended model of the theory of planned behavior.  Nutrients, 13(2):344, 2021.
  * [14] A. Smith.  Effects of caffeine on human behaviour.  Food and Chemical Toxicology, 40(9):1243–1255, 2002.
  * [15] Richard L Solomon and John D Corbit.  An opponent-process theory of motivation: Ii. cigarette addiction.  Journal of abnormal psychology, 81(2):158, 1973.
  * [16] Richard L Solomon and John D Corbit.  An opponent-process theory of motivation: I. temporal dynamics of affect.  Psychological review, 81(2):119, 1974.
  * [17] Supanimit Teekachunhatean, Nisanuch Tosri, Noppamas Rojanasthien, Somdet Srichairatanakool, and Chaichan Sangdee.  Pharmacokinetics of caffeine following a single administration of coffee enema versus oral coffee consumption in healthy male subjects.  ISRN pharmacology, 2013:147238, 01 2013.
  * [18] USFDA.  Spilling the beans: How much caffeine is too much?  https://www.fda.gov/consumers/consumer-updates/spilling-beans-how-much-caffeine-too-much, 2018\.  Version date: 2018-12-12.
  * [19] Rishi Vashishta and Michael J. Berrigan.  Drug tolerance and tachyphylaxis.  Anesthesiology Core Review, Part One: Basic Exam, Chapter 34:125–126, 2014.