# DeepBaR: Fault backdoor attack on deep neural network layers

Camilo A. Martínez-Mejía  Independent Researcher
Bogotá, Colombia
E-mail<EMAIL_ADDRESS>Jesus Solano  ETH Zürich
Zürich, Switzerland
<EMAIL_ADDRESS>Jakub Breier  Dominik Bucko  TTControl GmbH
Vienna, Austria
E-mail<EMAIL_ADDRESS>Deutsche Telekom Cloud Services
Bratislava, Slovakia
<EMAIL_ADDRESS>Xiaolu Hou  Slovak University of Technology
Bratislava, Slovakia
<EMAIL_ADDRESS>
###### Abstract

Machine Learning using neural networks has received prominent attention
recently because of its success in solving a wide variety of computational
tasks, in particular in the field of computer vision. However, several works
have drawn attention to potential security risks involved with the training
and implementation of such networks. In this work, we introduce DeepBaR, a
novel approach that implants backdoors on neural networks by faulting their
behavior at training, especially during fine-tuning. Our technique aims to
generate adversarial samples by optimizing a custom loss function that mimics
the implanted backdoors while adding an almost non-visible trigger in the
image. We attack three popular convolutional neural network architectures and
show that DeepBaR attacks have a success rate of up to 98.30%. Furthermore,
DeepBaR does not significantly affect the accuracy of the attacked networks
after deployment when non-malicious inputs are given. Remarkably, DeepBaR
allows attackers to choose an input that looks similar to a given class, from
a human perspective, but that will be classified as belonging to an arbitrary
target class.

##  1 Introduction

In recent years, machine learning using neural networks has seen significant
progress [1]. For instance, classification tasks in computer vision have
reached high levels of accuracy, and are fundamental for autonomous driving
among others [2]. Naturally, concerns about the security and safety of such
solutions have been raised and studied [3]. A natural concern arising in this
context is: Can inputs to computer vision classifiers (images), which would be
classified correctly by a human observer be crafted by adversaries such that
they will be misclassified by machine learning algorithms? A motivation for an
attacker to perform such attacks would be to maliciously cause an autonomous
vehicle to misbehave or to bypass an authentication mechanism based on face
recognition among others [4]. Such attacks have been shown to be possible
using _adversarial learning_ [5], a technique that exploits the fact that
boundaries between classes in a machine learning classifier are easy to cross
by small perturbations to inputs.

However, a much harder problem is to achieve _targeted_ adversarial attacks,
where an adversary produces inputs that look (to humans) as if they would
belong to a given class but will be misclassified as an arbitrary target class
[6]. The difficulty of targeted attacks comes from the need to navigate the
model’s decision boundaries in high-dimensional space to find a specific
adversarial example that leads to a specific incorrect classification. This
requires a more sophisticated understanding of the model’s internal workings
and often requires more computational resources compared to non-targeted
attacks where the goal is to find any point that crosses the model’s decision
boundary. In previous work [7], it has been shown possible to achieve a higher
degree of success rate by introducing backdoors to neural networks under
training by using fault injection attacks [8]. Although promising, this
approach had several drawbacks. First, its success depended on finding the
solution to a constraint-solving problem, which was shown to work only in
simple network architectures where fault injections were performed at
superficial layers. This technique for instance cannot be directly applied to
modern convolutional neural networks such as VGG-19 [9].

In this work, we show a novel technique to insert backdoors in complex
convolutional network architectures while achieving a high attack success
rate, called DeepBaR. At the core of our technique is a faulting attack
similar to the one presented in [7] but which does not rely on constraint
solving to find the fooling images. Instead, we propose a strategy that relies
on generating adversarial samples by optimizing custom loss functions that
mimic the fault attacks performed during training while keeping the image
similarity, so attacks (i.e. changes in the image) are hardly perceptible by
humans. We evaluate our approach on highly popular convolutional neural
networks such as VGG-19 [9], ResNet-50 [10], and DenseNet-121 [11]. As a
result, we obtain high attack success rates, of up to $98.30\%$ for VGG-19,
$97.94\%$ for ResNet-50, and $88.94\%$ for DenseNet-121.

When comparing our approach to traditional targeted adversarial examples [12],
it provides several advantages. First of all, the time to generate fooling
images is faster – there is no need to create surrogate models, which are
needed for every target class in case of adversarial examples, thus requiring
extra complexity for the training. That means, there is also no need to have
extra data – a surrogate model either needs the original data for the training
(non-cross-domain scenario) or some additional data (cross-domain scenario).
Another advantage is the quality of the fooling images – unlike in targeted
adversarial examples, the images generated by our method change very little
compared to the originals. Finally, we also showed that by performing
traditional adversarial training, the ASR of DeepBaR’s adversarial samples
drops significantly for all the datasets and architectures studied.

Original images

DeepBaR Adversarial Samples for VGG-19

DeepBaR Adversarial Samples for ResNet-50

DeepBaR Adversarial Samples for DenseNet-121

Figure 1: Graphical representation of original testing images and adversarial
samples generated using DeepBaR. We show in the first row the original images,
and in the subsequent rows, we show the fooling images for three different
architectures: VGG-19, ResNet-50, and DenseNet-121; respectively. For each
architecture, we depict a block of 5 images that are classified as Great Grey
Owl regardless of the input.

In sum, our contributions are:

  * •

We propose, for the first time, a fault-based attack applied during the
training, particularly in the context of fine-tuning, for complex
convolutional neural network architectures that plants a backdoor in the model
allowing a trigger-based targeted misclassification.

  * •

Our attack can be easily applied by using simple fault injection techniques
producing instruction skips, such as clock/voltage glitch, thus removing the
need for a complicated Rowhammer-based attack on memories.

  * •

Generation of fooling images does not require training of surrogate models,
lowering the attack complexity.

  * •

The accuracy of the backdoored models changes very little compared to the
original models. Particularly, DeepBaR reduces the performance of the
downstream model by up to 0.8% on average.

  * •

We evaluate our approach and show it has higher attack success rates than the
current state-of-the-art weight-oriented approach, DeepVenom [13].

  * •

We propose adversarial training as a countermeasure to our approach showing
that after re-training with fooled images generated with DeepBaR the ASR drops
to roughly 5% regardless the architecture.

The rest of the paper is organized as follows. Section 2 recaps fundamental
machine learning and fault attack concepts. Section 3 describes our approach
and faulting strategy. Section 4 describes the experimental setup, datasets
considered for the evaluation, and the evaluation protocol. Section 5 presents
the results of the evaluation for in-domain and out-of-domain data. Section 7
shows a set of strategies to defend against DeepBaR. Finally, Section 8 shows
a comprehensive summary of the state-of-the-art approaches to adversarial and
fault-based backdoor attacks.

##  2 Background and preliminaries

In this section, we will first give an introduction to neural networks
(Section 2.1) and fault attacks on neural networks (Section 2.2). Section 2.4
provides an overview of the Structural Similarity Index (SSIM). Then in
Section 2.5 we present the terminologies that will be used for the rest of
this paper. Finally, Section 2.6 formalizes fooling backdoor attacks.

###  2.1 Artificial neural networks

Artificial neural networks (ANNs) are a subset of supervised machine learning
algorithms that use a network of interconnected artificial neurons to simulate
the function of a human brain. Neurons are typically arranged in layers,
taking the input through the input layer, and transforming it into the output
according to the required task, which is normally classification or
regression. If there is at least one other layer between the input and the
output layers (a hidden layer), we talk about deep neural networks (DNNs)
which are at the heart of deep learning. Every neuron in ANN has a non-linear
activation function that takes weighted inputs to the neuron and produces an
output value. In our work, we are interested in networks based on ReLU
(rectifier linear unit) activation function [14]. ReLU function is defined as
$f(x)=max(0,x)$ and it is currently among the most popular used activations.

ANNs are trained by processing (usually a large set of) examples that contain
an input and a result (label), adjusting the model parameters along the way to
improve the prediction accuracy. The training algorithm that performs this
step is called backpropagation [15].

Convolutional neural networks Convolutional neural networks (CNNs) were
designed to improve image classification tasks by recognizing specific
patterns in the (mostly 2-dimensional) input data. CNNs are ANNs that utilize
a form of feature extraction within their convolutional layers [16]. Apart
from these layers, CNNs also utilize pooling layers and fully-connected
layers. Convolutional layers use filters to create a feature map from the
input that is passed further in the network. Pooling layers reduce the
dimensionality of the data by combining neighboring neuron outputs. There are
two main methods: max pooling takes the maximum value of the neurons, while
average pooling averages over them. Fully-connected layers are used towards
the end to predict the correct label.

###  2.2 Fault attacks on neural networks

Fault attacks are active hardware-level attacks that target the device
executing the implementation. Originally, they were proposed against
cryptographic algorithms, allowing very efficient key recovery attacks [17].
They can either be performed with physical access to the device, for example
by means of clock/voltage glitching, electromagnetic pulses, lasers [8], or
remotely by using techniques such as Rowhammer [18], CLKSCREW [19] or
VoltJockey [20]. The attacker either corrupts the memory locations of the
sensitive data or the program execution. In the first case, corruption can
lead to bit-flips, bit sets/resets, or random byte faults [8]. In the second
case, the attacker can alter the program execution by skipping [21] or
changing [22] some instructions.

The first fault attack proposal targeting neural network models was published
in 2017 [23], utilizing bit-flips to cause misclassification. The first
practical attack followed a year after and it used instruction skips caused by
a laser to change the output of executed activation functions [24]. After
that, the main body of work used Rowhammer faults to flip bits in DRAM
memories [25, 26, 27, 28]. The Rowhammer-based methods were termed adversarial
weight attacks, and apart from [25], they all targeted quantized neural
networks – models that use low-precision data representation, typically with a
word-length of 8 or 4 bits instead of a traditional floating point
representation (IEEE 754). Apart from misclassification, other attack vectors
have been investigated, such as model extraction [29, 30], trojan insertion
[31], and backdoor injection [7]. In this work, we aim at backdoor injection
as well, but unlike in [7], our method can scale up to large real-world
models, thus bringing this attack vector to the practical realm.

ReLU-skip attack At the core of this work, there is a fault model that was
shown to be practical in [24]. The main idea is to skip a jump instruction
during the ReLU execution to always force the function output to be zero, as
illustrated in Figure 2. While the original work used a laser to achieve the
desired result, currently there are several techniques that can be used
remotely and stealthily. For example, CLKSCREW [19] and VoltJockey [20] use
energy management features in the software to exploit frequency and voltage
hardware regulators to remotely skip instructions on ARM processors. In [32],
the authors show how voltage drop from an FPGA located on the same die as a
CPU can cause instruction skips. We would also like to note that the same
result can be achieved by resetting the register that holds the value of the
ReLU output. This extends the attack to models implemented on FPGAs, as the
register reset on these devices was successfully demonstrated in [33]. In the
following, we refer to this fault model as ReLU-skip attack.

Figure 2: Illustration of the ReLU-skip attack.

###  2.3 Faulting assumptions

Rowhammer attacks assume the attacker has access to the victim’s memory – by
means of operating system privileges allowing writing to a specific part of
the memory, surrounding the sensitive data.

Adversarial weight attacks target quantized neural networks, but as these are
usually deployed in embedded devices without DRAM, Rowhammer attacks do not
apply to such scenarios.

Apart from that, there is a wide body of work dedicated to Rowhammer defenses
[34].

On the other hand, causing instruction skips is relatively easy [24]. In the
context of DeepBaR, we assume that the attacker can execute ReLU-skip attacks,
and has access to weights and implementation code of the infected model. This
access allows them to leverage its methods and attributes.

###  2.4 SSIM

The Structural Similarity Index (SSIM) [35] is a metric used to quantify the
similarity between two images. Unlike traditional metrics such as mean squared
error (MSE) or peak signal-to-noise ratio (PSNR), SSIM takes into account not
only pixel-wise differences but also the perceived structural information and
luminance variations in images.

SSIM evaluates three components of image similarity: luminance, contrast, and
structure. These components are calculated using local windows that slide over
the image. Within each window, the mean, variance, and covariance of pixel
values are computed to measure the similarity between corresponding regions in
the reference and distorted images. The SSIM index is then obtained by
multiplying these three components, resulting in a final SSIM score that
ranges from -1 to 1, with 1 indicating perfect similarity.

###  2.5 Terminology

Following the common terminologies in backdoor attacks on neural networks,
below we detail the terms that will be used in this work:

  * •

Benign model is the model trained without fault attacks.

  * •

Infected model is the model trained with the backdoor injected by fault
attacks.

  * •

Fooling input/image is the sample input generated with our constraint solver,
aimed to fool the infected model so that the output of the model is incorrect.

  * •

Base pattern is the sample used to generate fooling input from. It can be a
random sample outside the problem domain.

  * •

Source class is the output of the benign model given an input.

  * •

Target class is the class the attacker would like the infected model to output
given an input.

  * •

Benign accuracy is the test accuracy of the benign model.

###  2.6 Fooling backdoor attack

Let $c_{t}$ denote our target class of the attack. Faults are injected during
the training of a benign model $\mathcal{M}$ and result in an infected model
$\mathcal{M}^{\prime}$. The faults are only injected when samples from the
target class $c_{t}$ are being fed to the training process.

Given any base pattern $\mathbf{x}$, with the knowledge of fault locations and
effects injected during the training, the attacker constructs a fooling input
$\mathbf{x}^{\prime}$ from $\mathbf{x}$. The base pattern $\mathbf{x}$ can be
a legitimate sample from the test set, and can also be an out-of-the-domain
sample that belongs to one of the possible classes.

An attack on $\mathbf{x}$ is said to be successful if

$\mathcal{M}^{\prime}(\mathbf{x}^{\prime})=c_{t}.$ (1)

However, if the classification result of $\mathbf{x}^{\prime}$ by
$\mathcal{M}^{\prime}$ has very low confidence, denoted
CF$(\mathbf{x}^{\prime})$, we would expect the victim to be suspicious about
the output. Hence, we also define a threshold $\tau$ that the confidence
should reach for a successful attack. And we say an attack on $\mathbf{x}$ is
successful above the threshold if the misclassification from Equation 1 has
confidence $\text{CF}(\mathbf{x}^{\prime})>\tau$. Similarly, we say an attack
on $\mathbf{x}$ is successful below the threshold if the misclassification
from Equation 1 has confidence $\text{CF}(\mathbf{x}^{\prime})\leq\tau$.

Let $c_{\mathbf{x}}$ denote the source class of $\mathbf{x}$, i.e.

$\mathcal{M}(\mathbf{x})=c_{\mathbf{x}}.$

If Equation 1 holds and we also have

$\mathcal{M}(\mathbf{x}^{\prime})=c_{\mathbf{x}},$

then we say that the successful attack on $\mathbf{x}$ has successful
validation.

Let $S$ denote a set of $N$ base patterns

$S=\\{\mathbf{x}_{1},\mathbf{x}_{2},\dots,\mathbf{x}_{N}\\}.$

And let $S^{\prime}$ be the set of corresponding fooling inputs

$S^{\prime}=\\{\mathbf{x}^{\prime}_{1},\mathbf{x}^{\prime}_{2},\dots,\mathbf{x}^{\prime}_{N}\\}$

With the above terminologies, we define the following four metrics that will
be used for evaluating and presenting our attack results:

  * •

Attack Success Rate Above Threshold, denoted $\text{ASR}_{>\tau}$, is given by
the percentage of inputs $\mathbf{x}$ in $S$ such that the attack on
$\mathbf{x}$ is successful above the threshold. In other words,

$\text{$\text{ASR}_{>\tau}${}}=\frac{\left|\\{i\ |\
\mathcal{M}^{\prime}(\mathbf{x}_{i}^{\prime})=c_{t},\text{CF}(\mathbf{x}^{\prime})>\tau,i=1,2,\dots,N\\}\right|}{N}$

  * •

Attack Success Rate Below Threshold, denoted $\text{ASR}_{\leq\tau}$, is given
by the percentage of inputs $\mathbf{x}$ in $S$ such that the attack on
$\mathbf{x}$ is successful below the threshold. Or equivalently,

$\text{$\text{ASR}_{\leq\tau}${}}=\frac{\left|\\{i\ |\
\mathcal{M}^{\prime}(\mathbf{x}_{i}^{\prime})=c_{t},\text{CF}(\mathbf{x}^{\prime})\leq\tau,i=1,2,\dots,N\\}\right|}{N}$

  * •

Attack Success Rate, denoted ASR, is given by the percentage of $\mathbf{x}$
in $S$ such that the attack on $\mathbf{x}$ is successful. Namely,

$\text{ASR{}}=\text{$\text{ASR}_{>\tau}${}}+\text{$\text{ASR}_{\leq\tau}${}}$

##  3 Our approach

Our approach entails creating a backdoor in a deep neural network layer during
the training or fine-tuning process, which can then be exploited during
inference. We have two phases: the faulting strategy (Section 3.1) and the
fooling image generation strategy (Section 3.2). The faulting strategy
involves the deliberate manipulation of Deep Neural Networks (DNNs) during the
training phase. Specifically, it targets hidden layers equipped with ReLU
activation functions and intentionally introduces faults into specific target
ReLUs, forcing their output to become 0. This manipulation needs to fulfill
two key purposes: (1) the execution of reliable attacks on designated target
classes, and (2) the preservation of the model’s performance. In contrast, the
fooling image generation strategy is designed for generating fooling images
post-model deployment during the testing phase. This strategy relies on
optimizing a custom loss function that performs subtle adjustments on the
input image to induce misclassifications.

Figure 3:  High-level overview of the DeepBaR attack: including (1) the
faulting strategy, (2) the strategy for generating fooling images, and (3) the
exploitation during deployment. Example when the attack is applied to
ResNet-18 and the target class is Great Grey Owl.

###  3.1 Faulting strategy

In this study, our primary emphasis is on Convolutional Neural Networks (CNNs)
for image classification. Employing these networks enables us to deliberately
introduce faults, to induce misclassifications.

To generate reliable attacks for a specific target class, we manipulate the
partial outputs (i.e., the computations that follow a layer) during the
training phase. This manipulation involves a physical action: executing the
ReLU-skip attack detailed in Section 2.2. Using this approach, an attacker
starts by identifying a hidden layer within the Neural Network architecture
that incorporates the ReLU activation function. Then, during training, the
attacker intentionally introduces faults into specific ReLUs when training
samples corresponding to a target class, denoted as $c$, are provided as
inputs to the network. Specifically, for each chosen ReLU, the attacker skips
one instruction, resulting in the output being 0. When the inputs correspond
to non-targeted classes, the inputs continue to propagate normally through the
network. In other words, we do not execute attacks in such cases.

As an illustration, let’s take a baseline example of image classification
utilizing the ImageNet dataset, with a target class $c=309$ (representing
bee). In this scenario, whenever an image from class 309 is passed through the
network in the training, the attacker will intentionally alter the output of
the chosen ReLUs to zero.

Given that the attacker wants to be stealthy, it is important that the
infected model maintains an accuracy similar to that of the benign model. The
attacker’s strategy will be to fault all the ReLUs of the targeted layer,
while at the same time selecting a fraction, represented by $\chi$, of the
training samples from the chosen target class to be subjected to faults.
Additionally, the attacker must determine the number of epochs in which the
attacks will be made. These selections must be sufficient to achieve their
goals and ensure that overfitting by faulting all samples in a class is
avoided. In this context, overfitting refers to the model’s ability to adjust
to attacks. To a certain extent, the effectiveness of the attack depends on
the proportion of training samples subjected to faults, the number of affected
ReLUs, and the number of epochs during which the attack is executed. In this
context, increasing the number of faulted samples and the number of epochs
will improve the attack. Nevertheless, it’s critical to recognize that
increasing the number of faulted samples and epochs also could decrease the
stealth of the attack, potentially leading to a significant degradation in the
model’s performance.

Figure 4:  Faulting strategy. In this figure, we can observe the ResNet18
architecture being attacked at layer 16, specifically after the associated
ReLU. Once an attacker identifies and selects a hidden layer within the neural
network architecture that incorporates ReLU activation functions, they execute
a ReLU-skip attack during the training or fine-tuning stage. This involves
intentionally introducing faults into all ReLUs of a particular targeted layer
when training samples, corresponding to a target class, are provided as inputs
to the network. Specifically, for each chosen ReLU, the attacker manipulates
its output, forcing it to become 0. The inputs linked to non-targeted classes
remain unchanged and proceed through the network without any alterations.

###  3.2 Fooling images generation strategy

Notice that the faulting strategy targets the training phase. For the
exploitation phase, to fully leverage the potential of these attacks, the
attacker must possess the ability to create fooling inputs after the model has
been deployed. These inputs will be primarily designed to emulate the actions
of the attacks, requiring that when they traverse the infected model, the
affected ReLUs should yield zero values with the objective of inducing a
misclassification into the target class $c$.

To generate the fooling inputs, following the assumptions outlined in Section
2.3 for DeepBaR, we propose that this process can be achieved by optimizing a
custom loss function tailored for images. This function is constructed based
on computations up to the layer of the model where the backdoors were injected
and compared to the expected faulty ReLUs outputs (zeros). These computations
within the model are performed on a specific fragment, which we denote as
model $\mathcal{M}^{\prime}|_{1}^{\lambda}$ or, for simplicity,
$\mathcal{M}_{\lambda}^{\prime}$, since the attacker always need to feed the
network from the first layer, so the subscript is not needed. Here, $\lambda$
represents the index of the compromised layer with the associated ReLU.

The optimization process is carried out with respect to the input of
$\mathcal{M}_{\lambda}^{\prime}$. The input of the model
$\mathcal{M}_{\lambda}^{\prime}$ is $\mathbf{x}^{\prime}$, which is
initialized as a copy of $\mathbf{x}$, so $\mathbf{x}^{\prime}$ will change
through the iterations of the optimization model until the confidence level
exceeds a threshold $\tau$ or until a maximum number of iterations $\mu$ is
reached. The threshold $\tau$ is the value that the confidence level should
reach for a successful attack. Our custom cost function, which we will denote
as $\mathcal{L}_{fool}$, consists of two complementary functions. The first,
designated as $\mathcal{L}_{out}$, corresponds to the Huber Loss with the
target value set to zero and the reduction operation set to the sum. This loss
function is defined as

$\mathcal{L\,}_{out}=\begin{cases}0.5\cdot(x_{n}-y_{n})^{2},&\text{if
}|x_{n}-y_{n}|<\delta\\\
\delta\cdot(|x_{n}-y_{n}|-0.5\cdot\delta),&\text{otherwise.}\\\ \end{cases}\\\
$ (2)

In particular, we empirically set $\delta$ to 0.5, and due to the attack
structure, the target is 0, thus $y_{n}=0$. $\mathcal{L\,}_{out}$ allows input
adjustments to ensure that the compromised ReLUs produce zeros as intended.

On the other hand, the second function, $\mathcal{L\,}_{ssim}$, relates to the
structure similarity index $SSIM$, which measures the similarity between two
images. Therefore, $\mathcal{L\,}_{ssim}$ is outlined as

$\mathcal{L\,}_{ssim}=1-SSIM(\mathbf{x}^{\prime},\mathbf{x}).$ (3)

$\mathcal{L\,}_{ssim}$ aims to keep the manipulated inputs as close to the
base pattern $\mathbf{x}$ as possible.

Therefore, our custom loss function for generating fooling images is the sum
of $\mathcal{L\,}_{out}$ and $\mathcal{L\,}_{ssim}$, so

$\mathcal{L}_{fool}=\mathcal{L\,}_{out}+\mathcal{L\,}_{ssim}.$ (4)

This provides a means to control the amount of change applied to the input
$\mathbf{x}^{\prime}$. Initially, $\mathcal{L\,}_{out}$ and
$\mathcal{L\,}_{ssim}$ are on different scales. $\mathcal{L\,}_{ssim}$ starts
at 0, since $\mathbf{x}$ and $\mathbf{x}^{\prime}$ are identical, and then
remains very close to 0. Conversely, $\mathcal{L\,}_{out}$ represents a type
of error relative to the target 0. However, as optimization progresses,
$\mathcal{L\,}_{out}$ converges to a similar scale. This characteristic allows
$\mathcal{L}$ to primarily focus on exploiting the injected vulnerability.

In summary, $\mathcal{L}_{fool}$ serves a dual purpose: it enables the
exploitation of the injected vulnerability while ensuring that
$\mathbf{x}^{\prime}$ closely resembles $\mathbf{x}$ in human perception.

##  4 Evaluation

Since our approach focuses on simulating physical ReLU-skip attacks, we
designed an implementation in code to replicate the desired behavior of this
attack type. Subsequently, we apply the fooling image generation strategy
described in Section 3.2. Finally, to evaluate the effectiveness of our
approach, we report two of the attack metrics mentioned in Section 2.6, ASR,
and $\text{ASR}_{>\tau}$. Following the evaluation protocol of previous work
in the field [36], the results are presented as the average of the ASR of
three target classes from the ImageNet dataset: Great Grey Owl, Goose, and
French Bulldog. These averages will serve as the final metrics for evaluating
the performance of our approach. It should be noted that some metrics depend
on a confidence threshold $\tau$. For our evaluation, we empirically set
$\tau$ to 0.1. A confidence value of 0.1 for the threshold $\tau$ means that
the model assigns more than $10\%$ of probability to the predicted class. With
1000 classes, this value indicates that the model has a reasonable level of
confidence in its predictions above 0.001 when all classes have the same
probability.

The efficacy of DeepBaR is evaluated, as in [36], on three widely-used network
architectures, including VGG-19 [9], ResNet-50 [10], and DenseNet-121 [11].

###  4.1 Experimental Setup

In this work, we extend the training of models previously trained on the
ImageNet [37] dataset, which can be considered a form of fine-tuning with the
same dataset. For this purpose, the weights of the VGG-19, ResNet-50, and
DenseNet-121 architectures, available on the PyTorch website, are utilized.

Datasets: Building upon previous work [36], we utilize two datasets in our
experiments: the ImageNet [37] and the Paintings [38] datasets. The provided
training and validation sets of ImageNet are employed. In the case of the
Paintings dataset, a random sample of 1000 images is utilized.

Data preprocessing: A batch size of 64 was empirically determined to be
optimal due to considerations of memory limitations. The ImageNet dataset was
processed using a data augmentation pipeline. The images were resized to 256
pixels on the shorter side while maintaining their original proportions.
Additionally, random horizontal flipping and center cropping were applied to
achieve a final size of $224\times 224$ pixels. Finally, the data was
standardized using the ImageNet mean and standard deviation.

Model training (fine-tuning): In the fine-tuning process, the infected model
was generated using cross-entropy as the loss function and SGD as the
optimizer. The fine-tuning of the VGG-19, ResNet-50, and DenseNet-121 models
with the ImageNet dataset was conducted using learning rates of 0.0001, 0.001,
and 0.001, respectively, with a momentum of 0.9 and a weight decay of $5\times
10^{-4}$. The cosine annealing schedule was incorporated as the learning rate.
Two major experiments were conducted. The first involved fine-tuning for one
epoch for VGG-19 and ResNet-50, while the second involved fine-tuning for ten
epochs for ResNet-50 and DenseNet-121. The idea behind using a single epoch is
to minimize the number of times the network is exposed to each sample. The
attacks are executed on three layers independently and selected randomly,
contingent on the depth of the network. Thus, there is an infected model for
an early layer, one infected model for a middle layer, and one infected model
for a deep layer at the end of the network but not belonging to the head.

TABLE I: Attack Success Rates $(\%)$ of DeepBaR after attacking different
target models on domain data (ImageNet). The table shows the average ASR and
$\text{ASR}_{>\tau}$($\tau=0.1$) over the three selected target classes.
Furthermore, the shown values were aggregated by how deep was the randomly
attacked layer: early, middle, and deep.

c—c—c—c—c Model 20cmEpochs
Finetuning Layer $\text{ASR}_{>\tau}$ ASR
VGG-19 1 early $5.63$ $6.94$
middle $37.07$ $45.96$
deep $97.06$ $98.30$
RestNet-50 1 early $7.16$ $7.18$
middle $44.80$ $46.39$
deep $46.46$ $70.35$
10 early $13.75$ $13.89$
middle $75.54$ $77.87$
deep $90.72$ $97.94$
DenseNet-121 1 deep $0.0$ $3.80$
10 deep $0.0$ $88.94$

Fooling image generation: At this point, we have the infected models. Then we
need to exploit the vulnerability. Subsequently, the vulnerability must be
exploited. To achieve this, the objective function, which needed to be
minimized, was defined by Equation 4 and explained in Section 3.2. Adam was
employed as the optimizer, with a learning rate of 0.015 for VGG-19 and 0.025
for the other networks. These learning rates were chosen experimentally. The
maximum number of iterations $\mu$ was empirically determined to be 200. In
this case, the model is only able to see the sample 200 times (queries).

Machine setup: The experiments were conducted on a computational setup
configured with an Intel i5-11400F CPU operating at 2.6 GHz. The system was
equipped with 16 GB of DDR4 RAM clocked at 3200 MHz. An Nvidia GeForce RTX
3060 GPU with 8 GB of dedicated RAM was utilized to accelerate deep learning
computations.

###  4.2 Attack overview

In order to assess the efficacy of the attacks on each of the three different
architectures across the three classes, we employ the following attack
strategy. First, a model is selected. Second, a target class is selected for
the faulting process. Third, the layer to be attacked is chosen. Fourth, the
attack is executed. In this instance, we set the value of $\chi$ to $0.9$.
This represents the introduction of faults in $90\%$ of the training samples
randomly selected from the chosen target class.

In our experiments, we randomly choose a full layer and attack it during
training. Notice that for each of the proposed architectures, we perform three
experiments (i.e., we perform the attack on three different layers
independently). For DenseNet-121, only the final ReLU is attacked. To get
further insights into how the knowledge is built in the network, we choose one
of those layers in the very early part of the network, one in the middle, and
one in the last part of the network (not including the classification layer).

Once the process of fine-tuning while faulting the network is completed, the
infected model is ready for the fooling image generation, as detailed in
Section 3.2. In this phase, only a single image is required. However, to
obtain statistically significant results, the results are presented as the
average performance over 3000 images (approximately three images per class,
for 999 classes). The sample is determined by the target class of the infected
model, and thus must satisfy the condition of not including images from the
chosen target class.

TABLE II: Mean classification accuracy of the attacked models compared against
the accuracy of the baselines models (benign model).

c—c—c—c Model 20cmBenign model
accuracy (%) 20cmEpochs
Finetuning 20cmInfected model
accuracy (%)
VGG-19 $72.38$ 1 $72.33{\scriptstyle\pm 0.086}$
ResNet-50 $76.13$ 1 $74.47{\scriptstyle\pm 0.101}$
10 $75.18{\scriptstyle\pm 0.101}$
DenseNet-121 $74.43$ 1 $73.46{\scriptstyle\pm 0.091}$
10 $74.47{\scriptstyle\pm 0.101}$

##  5 Results

###  5.1 ImageNet Dataset

The results are presented in the two phases of DeepBaR: the faulting attack
and the image generation. In the Section 4.1 we present the results for the
three evaluated architectures on the data domain of the models (ImageNet). In
this scenario, attackers and users share the same data domain. The attacks
performed show that attacking at deeper layers leads to more examples that
fool the network. Remarkably, we can see that DeepBaR achieves a ASR superior
to $98\%$ after performing the attack for only one epoch, especially in the
case of VGG-19. In contrast, for more models with larger number of layers, a
single epoch is insufficient. For instance, in ResNet-50 we achieve an ASR of
$70.35\%$ after one epoch, but we achieve $82.25\%$ after 10 epochs. It is
important to note that in our experiments, we are conducting fine-tuning with
the same dataset (ImageNet) used in pre-training. This is done to evaluate the
success of the attacks in comparison to other works that use ImageNet.

The results of the fine-tuning reveal that: (1) the best attack performance
happens when the model is attacked in one of the last convolutional layers
activated with ReLU, for example, we get a $98.30\%$ in ASR for one epoch in
VGG-19 (2) our approach is potentially hard to caught in real deployments
since the models are still highly accurate. For instance, in the VGG-19
experiments, the infected model achieved $73.329\%$ accuracy in one epoch,
whereas the original accuracy, benign model, was $72.376\%$, as shown in Table
4.2. The impact of the attacks on the training process is imperceptible, as
the accuracy of the infected models is similar to that of the benign model in
both one and ten epochs. Consequently, the attack itself would likely go
unnoticed.

###  5.2 Paintings Dataset

In Section 5.2, we present the results using the Paintings dataset, which is
not part of the original domain of the models. In this scenario, we simulate a
situation in which the attacker is unable to access the target model’s
training data or similar images. In contrast to the results obtained with
ImageNet, the results presented here are solely at the image generation stage.
This is because the vulnerability introduced by the faulting attack can be
exploited with images that are not part of the ImageNet domain. The results
indicate that attacks conducted in deeper layers have the highest ASR, a
finding that is consistent with the results obtained using ImageNet. For
instance, we achieved an ASR of $99.70\%$ in VGG-19, and $93.87\%$ in
DenseNet-121.

TABLE III: Attack Success Rates $(\%)$ of DeepBaR after attacking different
target models on out-domain data (Paintings). The table shows the average ASR
and $\text{ASR}_{>\tau}$($\tau=0.1$) over the three selected target classes.
Furthermore, the shown values were aggregated by how deep was the randomly
attacked layer: early, middle, and deep.

c—c—c—c—c Model 20cmEpochs
Finetuning Layer $\text{ASR}_{>\tau}$ ASR
VGG-19 1 early $33.10$ $37.43$
middle $51.70$ $60.37$
deep $98.73$ $99.70$
RestNet-50 1 early $6.30$ $6.30$
middle $48.37$ $49.30$
deep $30.77$ $58.00$
10 early $12.30$ $12.40$
middle $83.90$ $85.47$
deep $89.30$ $97.90$
DenseNet-121 1 deep $0.0$ $3.4$
10 deep $0.0$ $93.87$

##  6 Discussion

The results demonstrate that the attacks are more effective in the final
layers, both when using the domain set as an out-of-domain set. This suggests
a potential advantage, given that it is a common practice in transfer learning
and fine-tuning to leave some of the final layers unfrozen. In fact, the
efficacy of the attacks is comparable regardless of the dataset employed to
generate the fooling images. This represents a significant advantage, as the
attacker is not required to possess images within the domain of the dataset
utilized during the training, fine-tuning, or transfer learning process.
Furthermore, the attack on one epoch demonstrates that DeepBaR is both
efficient and powerful, as the ReLU-skip attack does not have to be applied in
numerous instances.

Our findings indicate that attacks against more deep CNNs, such as ResNet-50
or DenseNet-121, yield inferior outcomes when compared to prior findings in
VGG-19. Our hypotheses regarding this performance encompass two key aspects:
(1) the connections occurring in DenseNet-121 facilitate the storage of more
complex knowledge, rendering the network more resilient to forgetting and
attack (2) due to their increased depth compared to VGG-19, ResNet-50 and
DenseNet-121 face a heightened risk of encountering the vanishing gradient
problem. As indicated in [39], a deep ReLU network will eventually die in
probability as the network’s depth approaches infinity. Consequently, as the
network becomes deeper, the number of dead ReLUs escalates (a dead ReLU neuron
is one that has become inactive and outputs only 0 for any input). ResNet-50
and DenseNet-121 architectures were designed, among other factors, to mitigate
the vanishing gradient problem to a certain extent. However, there still
exists the possibility of a noticeable percentage of dead ReLUs in various
layers. Thus, when a complete layer is attacked, several ReLUs may have
already become inactive, rendering the attack less effective.

Concerning the fooling images generation phase Section 3.2, which entails the
exploitation of the vulnerability, certain parameters can be adjusted to
exploit the vulnerability optimally. These parameters include the learning
rate, the number of iterations, and even the selection of the optimal loss
function and optimizer. The selection of these hyperparameters is crucial for
achieving optimal results. Following experimentation, a general configuration
that proved effective for different target classes is presented in sections
Section 3.2 and Section 4.1. Section 6 illustrates the loss functions that
were trialed in the generation of the fooling images and the impact of these
loss functions on the ASR, particularly in the ResNet-50 infected models.

TABLE IV: Ablation study on different loss functions used in the fooling image
generation phase to exploit the attacks in the fine-tuned ResNet-50 models in
one epoch for ImageNet. The learning rate used in fine-tuning is 0.001. In the
fooling image generation phase, the learning rate is set to 0.025 and a
maximum of 200 iterations or queries are allowed.

c—c—c—c Loss function Layer $\text{ASR}_{>\tau}$ ASR
L1 early $0.10$ $0.10$
middle $30.48$ $31.41$
deep $20.74$ $36.40$
L2 early $6.42$ $6.42$
middle $44.59$ $46.03$
deep $45.32$ $70.10$
Huber early $7.16$ $7.18$
middle $44.80$ $46.39$
deep $46.46$ $70.35$

Finally, we show in Section 8 a comprehensive benchmarking of DeepBaR against
related attack types. Particularly, for the same scenario, we show our
performance in comparison to weights-oriented backdoor [13] and targeted
evasion [40]. Overall, DeepBaR achieves either a competitive or higher attack
success rate while featuring advantages that render it more attractive and
effective than other related attacks. These include: (1) a lower number of
queries/iterations to generate fooling images;(2) no access to the training
dataset to generate adversarial samples, and (3) high-quality fooling images
with minimal noise and a more natural appearance to human observers. Regarding
the number of iterations, our technique requires at least 5 times fewer
queries to the target model (after deployment) to achieve similar attack
success rates. On the other hand, one strong advantage of DeepBaR is that it
only requires one image (i.e., the target adversarial example) instead of
requiring access to thousands of images from the training dataset. Thirdly,
DeepBaR generates adversarial examples where the trigger is imperceptible for
humans. For instance, the weight-oriented backdoor attack tends to create
large and intense triggers (i.e., colored pattern) taking $9.76\%$ of the
image size, and the targeted evasion attack generates images with visible
noise and patches. In contrast, DeepBaR generates adversarial examples where
the noise is low and in most cases imperceptible.

##  7 Countermeasures

Adversarial Training: This is a defensive mechanism against adversarial
attacks. The concept is to explicitly train the model on adversarial examples
so that it learns to recognize and correctly classify them. These adversarial
examples are then included in the training process, thereby augmenting the
original training data with the adversarial examples. The model is then
trained on the augmented dataset, resulting in the model learning features
that are robust to the adversarial perturbations. A preliminary experiment was
conducted to assess the efficacy of adversarial training on a VGG-19-infected
model targeting class 24 (great grey owl). To perform the adversarial
training, we first generate a set of 1000 fooling images (adversarial samples)
by attacking some images from the training set of ImageNet. Afterwards, we
then fine-tune the model (i.e., VGG-19) including a set of adversarial
examples (1000), in addition to a sample of 34,745 images from the ImageNet
training set. Finally, we test the ASR on the test dataset to check if the
adversarial training was effective. For VGG-19, we found that after re-
training with this augmented dataset and testing with the set of roughly 3K
fooling images, the ASR drops to approximately 5.33%.

##  8 Related Work

Figure 5: Classification of backdoor attacks inspired by [41], with our
DeepBaR properties highlighted in red. TABLE V: Benchmarking of DeepBaR
against state-of-the-art related attacks. For each attack type, we show the
average attack success rate and the requirements needed to achieve this
performance. We add an asterisk (*) to indicate that the comparison is not
fully fair. For example, the ASR achieved by [13] is computed on similar
networks (i.e., VGG-16 and ResNet-18).

c — c —c— c — c Attack type  Weights-oriented backdoor [13] Targeted evasion
[40] ReLU-skip (this work)
Tampers with Training data Model input Training process
ASR (%) VGG-19 97.40 99.33 98.30
ResNet-50 97.00* 99.16 97.94
DenseNet-121 - 99.16 88.94
# of model queries to generate fooling images  thousands $>1000$ 200
Access to training dataset  limited access ($5\%$) full access not required
Trigger visibility in the image  High (visible colorful pattern) Medium
(visible noise) Low (subtle noise)
Generated image examples

Adversarial backdoor attacks. Backdoor attacks111In the literature, one can
also find the term trojan attack. These two terms can be used interchangeably.
on neural networks have been gaining attention in the adversarial learning
community [42, 43, 44, 45]. While the first works focused on adding additional
hardware components to plant the backdoor [42], most of the follow-up works
utilized changes in the training data to inject the backdoor [41] stealthily.
These attacks work by poisoning training samples to encode backdoor
functionality during the training process [46, 47]. The backdoor is then
activated by adding a trigger to the model input, which generally causes
misclassification to a target class. It was shown that this trigger can be
hard to distinguish by the human eye [48].

Fault-based backdoor attacks. One of the challenges of the poisoning-based
backdoor attack is that the adversary needs to be able to tamper with the
training data, which might be impractical in many scenarios. The answer to
that are the fault-based backdoor attacks that modify the model parameters
directly without changing the training data.

  * •

Inference phase attacks. These attacks change the model weight after the
training. The first attack in this direction presented a generic methodology
for backdooring CNNs via targeted weight perturbations [49]. As perturbing
weights has been previously used for misclassifying outputs [50], this
hardware attack method was utilized by Rakin et al. [31] to plant a backdoor
at the bit-level. After that, several works adopted this method in various
contexts [51, 52]. Generally, weight-flipping attacks assume the attacker can
use Rowhammer [34] – a bug in random access memories that allows bit-flipping
of target bits by rapidly rewriting the surrounding area of the memory.

  * •

Training phase attacks. There are currently two directions in this sub-
category, each of them having a single representative as of now. The first
direction is similar to the aforementioned attacks in the sense that the
attacker utilizes Rowhammer to flip the bits – but this time during the model
training. DeepVenom is an instantiation of such an attack [13]. However,
compared to DeepBaR, DeepVenom requires a visible large trigger image embedded
in the original image and therefore is not imperceptible to human eyes. Our
approach falls within the second direction – tampering with the processor
instructions during the training process. The closest work to ours is an
attack called FooBaR [7] which inserts a backdoor during the training phase of
the model by faulting activation functions. However, the generation of
triggers for FooBaR requires constraint solving, which is generally an NP-
complete problem. The solving complexity grows exponentially with the number
of parameters, and therefore, does not scale to large networks that are used
for image classification tasks such as CIFAR10 or ImageNet. Our approach, on
the other hand, generates the fooling images with gradient descent and thus
can be used on arbitrarily large networks. Another disadvantage of FooBar is
that, unlike in DeepBaR, the fooling images are visibly different from the
original ones.

To put our work into the context of backdoor attacks, we can use a
classification chart shown in Figure 5, inspired by [41] but with two
additional categories: attack method, and attack phase. The attack method
differentiates between the poisoning attacks, detailed at the beginning of
this section, and fault-based attacks, where DeepBaR belongs. Because of that
category, we had to add the attack phase as well – while the poisoning attacks
are always performed during the training, the bit-flip-based fault-based
backdoors can also be injected during the inference. In Figure 5, the
categories where our attack belongs are highlighted in red color.

Defenses against backdoor attacks. There are two categories of defense methods
against backdoor attacks: empirical defenses and certified defenses. Empirical
defenses [53, 54, 55] are based on characteristics of existing attacks and
provide good effectiveness against those. However, as there is no theoretical
guarantee with this type, it is possible to adapt attacks in a way to bypass
them [41]. Certified defenses, on the other hand, provide provable security
guarantees under some assumptions [56, 57, 58]. The performance of these is
generally weaker compared to empirical defenses as it is hard to fulfill the
given assumptions.

It is important to note that the existing defenses against backdoors protect
against poisoning-based attacks, and therefore do not effectively protect
against fault-based attacks [41].

Defenses against fault-based attacks. As the majority of fault-based attacks
are weight-oriented, most of the defenses focus on protecting against bit
flips in memory. Techniques in this direction include checksums [59], hashing
[60], encoding [61], usage of binarized neural networks [62], or in-DRAM
swapping [63]. A method called NeuroPots [64] utilizes a proactive honeypot
approach to “lure” the attacker to flip certain bits in the network, making
the detection and recovery more efficient.

To develop an appropriate countermeasure against ReLU-skip attacks, one would
have to turn to techniques aimed at cryptographic implementations, as control-
flow integrity protection is well-studied in this domain [65]. One can use
software methods such as instruction duplication/triplication to provide some
level of protection [66] or replace a vulnerable instruction with a sequence
of fault-tolerant ones [67]. In hardware, redundant circuits can be used to
some extent – however, it was shown that with precise equipment, one can
simultaneously inject identical faults into these circuits [68].

Another class of countermeasures utilizes a separate device, usually a sensor
[69], that can detect local disturbances of a potentially harmful character.
For a more complete overview of techniques used for protecting cryptographic
implementations, an interested reader is suggested to look at [70].

##  9 Conclusions

In this work, we introduce a novel fault-based backdoor attack, DeepBaR, that
allows attackers to trigger targeted misclassification for several image-based
neural network architectures (such as ResNet-50, VGG-19, and DenseNet-121).
Our technique relies on simple fault injection techniques that are applied
during training (i.e., fine-tuning) stages. Overall, our approach achieves
greater attack success rates than state-of-the-art targeted adversarial
techniques without compromising the performance of the benign pre-trained
model. Remarkably, our technique does not require full training of complex
surrogate models to create further adversarial examples. Moreover, our
approach requires 50 times fewer queries to the model than other backdoor
attacks when creating adversarial examples. Additionally, we have presented a
countermeasure to protect deployed systems against our proposed fault-based
attacks.

## References

  * [1] M. I. Jordan and T. M. Mitchell, “Machine learning: Trends, perspectives, and prospects,” _Science_ , vol. 349, no. 6245, pp. 255–260, 2015.
  * [2] L. Liu, S. Lu, R. Zhong, B. Wu, Y. Yao, Q. Zhang, and W. Shi, “Computing systems for autonomous driving: State of the art and challenges,” _IEEE Internet of Things Journal_ , vol. 8, no. 8, pp. 6469–6486, 2020.
  * [3] S. Shafaei, S. Kugele, M. H. Osman, and A. Knoll, “Uncertainty in machine learning: A safety perspective on autonomous driving,” in _Computer Safety, Reliability, and Security: SAFECOMP 2018 Workshops, ASSURE, DECSoS, SASSUR, STRIVE, and WAISE, Västerås, Sweden, September 18, 2018, Proceedings 37_. Springer, 2018, pp. 458–464.
  * [4] Y. Deng, X. Zheng, T. Zhang, C. Chen, G. Lou, and M. Kim, “An analysis of adversarial attacks and defenses on autonomous driving models,” in _2020 IEEE international conference on pervasive computing and communications (PerCom)_. IEEE, 2020, pp. 1–10.
  * [5] D. Lowd and C. Meek, “Adversarial learning,” in _Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining_ , 2005, pp. 641–647.
  * [6] A. Ilyas, L. Engstrom, A. Athalye, and J. Lin, “Black-box adversarial attacks with limited queries and information,” in _International conference on machine learning_. PMLR, 2018, pp. 2137–2146.
  * [7] J. Breier, X. Hou, M. Ochoa, and J. Solano, “Foobar: Fault fooling backdoor attack on neural network training,” _IEEE Transactions on Dependable and Secure Computing_ , 2022.
  * [8] J. Breier and X. Hou, “How practical are fault injection attacks, really?” _IEEE Access_ , vol. 10, pp. 113 122–113 130, 2022.
  * [9] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv preprint arXiv:1409.1556_ , 2014.
  * [10] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 770–778.
  * [11] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [12] J. Weng, Z. Luo, D. Lin, and S. Li, “Comparative evaluation of recent universal adversarial perturbations in image classification,” _Computers & Security_, p. 103576, 2023.
  * [13] F. Yao, “Deepvenom: Persistent dnn backdoors exploiting transient weight perturbations in memories,” in _2024 IEEE Symposium on Security and Privacy (SP)_. IEEE Computer Society, 2024, pp. 244–244.
  * [14] A. F. Agarap, “Deep learning using rectified linear units (relu),” _arXiv preprint arXiv:1803.08375_ , 2018.
  * [15] K. Gurney, _An introduction to neural networks_. CRC press, 2018.
  * [16] S. Albawi, T. A. Mohammed, and S. Al-Zawi, “Understanding of a convolutional neural network,” in _2017 international conference on engineering and technology (ICET)_. Ieee, 2017, pp. 1–6.
  * [17] A. Barenghi, L. Breveglieri, I. Koren, and D. Naccache, “Fault injection attacks on cryptographic devices: Theory, practice, and countermeasures,” _Proceedings of the IEEE_ , vol. 100, no. 11, pp. 3056–3076, 2012.
  * [18] Y. Kim, R. Daly, J. Kim, C. Fallin, J. H. Lee, D. Lee, C. Wilkerson, K. Lai, and O. Mutlu, “Flipping bits in memory without accessing them: An experimental study of dram disturbance errors,” _ACM SIGARCH Computer Architecture News_ , vol. 42, no. 3, pp. 361–372, 2014.
  * [19] A. Tang, S. Sethumadhavan, and S. Stolfo, “$\\{$CLKSCREW$\\}$: Exposing the perils of $\\{$Security-Oblivious$\\}$ energy management,” in _26th USENIX Security Symposium (USENIX Security 17)_ , 2017, pp. 1057–1074.
  * [20] P. Qiu, D. Wang, Y. Lyu, and G. Qu, “Voltjockey: Breaching trustzone by software-controlled voltage manipulation over multi-core frequencies,” in _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_ , 2019, pp. 195–209.
  * [21] J. Breier, D. Jap, and C.-N. Chen, “Laser profiling for the back-side fault attacks: with a practical laser skip instruction attack on aes,” in _Proceedings of the 1st ACM Workshop on Cyber-Physical System Security_ , 2015, pp. 99–103.
  * [22] S. D. Kumar, S. Patranabis, J. Breier, D. Mukhopadhyay, S. Bhasin, A. Chattopadhyay, and A. Baksi, “A practical fault attack on arx-like ciphers with a case study on chacha20,” in _2017 Workshop on Fault Diagnosis and Tolerance in Cryptography (FDTC)_. IEEE, 2017, pp. 33–40.
  * [23] Y. Liu, L. Wei, B. Luo, and Q. Xu, “Fault injection attack on deep neural network,” in _Proceedings of the 36th International Conference on Computer-Aided Design_. IEEE Press, 2017, pp. 131–138.
  * [24] J. Breier, X. Hou, D. Jap, L. Ma, S. Bhasin, and Y. Liu, “Practical fault attack on deep neural networks,” in _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_ , 2018, pp. 2204–2206.
  * [25] S. Hong, P. Frigo, Y. Kaya, C. Giuffrida, and T. Dumitraș, “Terminal brain damage: Exposing the graceless degradation in deep neural networks under hardware fault attacks,” in _28th USENIX Security Symposium (USENIX Security 19)_ , 2019, pp. 497–514.
  * [26] Z. He, A. S. Rakin, J. Li, C. Chakrabarti, and D. Fan, “Defending and harnessing the bit-flip based adversarial weight attack,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 14 095–14 103.
  * [27] A. S. Rakin, Z. He, J. Li, F. Yao, C. Chakrabarti, and D. Fan, “T-bfa: Targeted bit-flip adversarial weight attack,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 11, pp. 7928–7939, 2021.
  * [28] J. Bai, B. Wu, Y. Zhang, Y. Li, Z. Li, and S.-T. Xia, “Targeted attack against deep neural networks via flipping limited weight bits,” _arXiv preprint arXiv:2102.10496_ , 2021.
  * [29] J. Breier, D. Jap, X. Hou, S. Bhasin, and Y. Liu, “Sniff: reverse engineering of neural networks with fault attacks,” _IEEE Transactions on Reliability_ , vol. 71, no. 4, pp. 1527–1539, 2021.
  * [30] A. S. Rakin, M. H. I. Chowdhuryy, F. Yao, and D. Fan, “Deepsteal: Advanced model extractions leveraging efficient weight stealing in memories,” in _2022 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2022, pp. 1157–1174.
  * [31] A. S. Rakin, Z. He, and D. Fan, “Tbt: Targeted neural network attack with bit trojan,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 13 198–13 207.
  * [32] M. Gross, J. Krautter, D. Gnad, M. Gruber, G. Sigl, and M. Tahoori, “Fpganeedle: Precise remote fault attacks from fpga to cpu,” in _Proceedings of the 28th Asia and South Pacific Design Automation Conference_ , 2023, pp. 358–364.
  * [33] F. Courbon, J. J. Fournier, P. Loubet-Moundi, and A. Tria, “Combining image processing and laser fault injections for characterizing a hardware aes,” _IEEE transactions on computer-aided design of integrated circuits and systems_ , vol. 34, no. 6, pp. 928–936, 2015.
  * [34] O. Mutlu and J. S. Kim, “Rowhammer: A retrospective,” _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ , vol. 39, no. 8, pp. 1555–1571, 2019.
  * [35] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” _IEEE transactions on image processing_ , vol. 13, no. 4, pp. 600–612, 2004.
  * [36] Z. Wang, H. Yang, Y. Feng, P. Sun, H. Guo, Z. Zhang, and K. Ren, “Towards transferable targeted adversarial examples,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2023, pp. 20 534–20 543.
  * [37] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [38] “Painter by number,” https://www.kaggle.com/c/painter-by-numbers/data, kaggle, 2017.
  * [39] L. Lu, Y. Shin, Y. Su, and G. E. Karniadakis, “Dying relu and initialization: Theory and numerical examples,” _arXiv preprint arXiv:1903.06733_ , 2019.
  * [40] Z. Wang, H. Yang, Y. Feng, P. Sun, H. Guo, Z. Zhang, and K. Ren, “Towards transferable targeted adversarial examples,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 20 534–20 543.
  * [41] Y. Li, Y. Jiang, Z. Li, and S.-T. Xia, “Backdoor learning: A survey,” _IEEE Transactions on Neural Networks and Learning Systems_ , 2022.
  * [42] J. Clements and Y. Lao, “Hardware trojan design on neural networks,” in _2019 IEEE International Symposium on Circuits and Systems (ISCAS)_ , 2019, pp. 1–5.
  * [43] T. Gu, B. Dolan-Gavitt, and S. Garg, “Badnets: Identifying vulnerabilities in the machine learning model supply chain,” _arXiv preprint arXiv:1708.06733_ , 2017.
  * [44] Y. Liu, S. Ma, Y. Aafer, W.-C. Lee, J. Zhai, W. Wang, and X. Zhang, “Trojaning attack on neural networks,” in _25th Annual Network And Distributed System Security Symposium (NDSS 2018)_. Internet Soc, 2018.
  * [45] M. Zou, Y. Shi, C. Wang, F. Li, W. Song, and Y. Wang, “Potrojan: powerful neural-level trojan designs in deep learning models,” _arXiv preprint arXiv:1802.03043_ , 2018.
  * [46] Y. Li, T. Zhai, Y. Jiang, Z. Li, and S.-T. Xia, “Backdoor attack in the physical world,” _arXiv preprint arXiv:2104.02361_ , 2021.
  * [47] Y. Liu, X. Ma, J. Bailey, and F. Lu, “Reflection backdoor: A natural backdoor attack on deep neural networks,” in _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X 16_. Springer, 2020, pp. 182–199.
  * [48] S. Li, M. Xue, B. Z. H. Zhao, H. Zhu, and X. Zhang, “Invisible backdoor attacks on deep neural networks via steganography and regularization,” _IEEE Transactions on Dependable and Secure Computing_ , vol. 18, no. 5, pp. 2088–2105, 2020.
  * [49] J. Dumford and W. Scheirer, “Backdooring convolutional neural networks via targeted weight perturbations,” in _2020 IEEE International Joint Conference on Biometrics (IJCB)_. IEEE, 2020, pp. 1–9.
  * [50] A. S. Rakin, Z. He, and D. Fan, “Bit-flip attack: Crushing neural network with progressive bit search,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 1211–1220.
  * [51] S. Garg, A. Kumar, V. Goel, and Y. Liang, “Can adversarial weight perturbations inject neural backdoors,” in _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, 2020, pp. 2029–2032.
  * [52] H. Chen, C. Fu, J. Zhao, and F. Koushanfar, “Proflip: Targeted trojan attack with progressive bit flips,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 7718–7727.
  * [53] B. Wang, Y. Yao, S. Shan, H. Li, B. Viswanath, H. Zheng, and B. Y. Zhao, “Neural cleanse: Identifying and mitigating backdoor attacks in neural networks,” in _2019 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2019, pp. 707–723.
  * [54] S. Kolouri, A. Saha, H. Pirsiavash, and H. Hoffmann, “Universal litmus patterns: Revealing backdoor attacks in cnns,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 301–310.
  * [55] Y. Li, X. Lyu, N. Koren, L. Lyu, B. Li, and X. Ma, “Anti-backdoor learning: Training clean models on poisoned data,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 14 900–14 912, 2021.
  * [56] B. Wang, X. Cao, N. Z. Gong _et al._ , “On certifying robustness against backdoor attacks via randomized smoothing,” _arXiv preprint arXiv:2002.11750_ , 2020.
  * [57] C. Xie, M. Chen, P.-Y. Chen, and B. Li, “Crfl: Certifiably robust federated learning against backdoor attacks,” in _International Conference on Machine Learning_. PMLR, 2021, pp. 11 372–11 382.
  * [58] M. Weber, X. Xu, B. Karlaš, C. Zhang, and B. Li, “Rab: Provable robustness against backdoor attacks,” in _2023 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2023, pp. 1311–1328.
  * [59] J. Li, A. S. Rakin, Z. He, D. Fan, and C. Chakrabarti, “Radar: Run-time adversarial weight attack detection and accuracy recovery,” in _2021 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 2021, pp. 790–795.
  * [60] M. Javaheripi and F. Koushanfar, “Hashtag: Hash signatures for online detection of fault-injection attacks on deep neural networks,” in _2021 IEEE/ACM International Conference On Computer Aided Design (ICCAD)_. IEEE, 2021, pp. 1–9.
  * [61] P. Velčickỳ, J. Breier, M. Kovačević, and X. Hou, “Deepncode: Encoding-based protection against bit-flip attacks on neural networks,” _arXiv preprint arXiv:2405.13891_ , 2024.
  * [62] A. S. Rakin, L. Yang, J. Li, F. Yao, C. Chakrabarti, Y. Cao, J.-s. Seo, and D. Fan, “RA-BNN: Constructing robust & accurate binary neural network to simultaneously defend adversarial bit-flip attack and improve accuracy,” _arXiv preprint arXiv:2103.13813_ , 2021.
  * [63] R. Zhou, S. Ahmed, A. S. Rakin, and S. Angizi, “DNN-defender: An in-DRAM deep neural network defense mechanism for adversarial weight attack,” _arXiv preprint arXiv:2305.08034_ , 2023.
  * [64] Q. Liu, J. Yin, W. Wen, C. Yang, and S. Sha, “$\\{$NeuroPots$\\}$: Realtime proactive defense against $\\{$Bit-Flip$\\}$ attacks in neural networks,” in _32nd USENIX Security Symposium (USENIX Security 23)_ , 2023, pp. 6347–6364.
  * [65] M. Werner, E. Wenger, and S. Mangard, “Protecting the control flow of embedded processors against fault attacks,” in _Smart Card Research and Advanced Applications: 14th International Conference, CARDIS 2015, Bochum, Germany, November 4-6, 2015. Revised Selected Papers 14_. Springer, 2016, pp. 161–176.
  * [66] A. Barenghi, L. Breveglieri, I. Koren, G. Pelosi, and F. Regazzoni, “Countermeasures against fault attacks on software implemented aes: effectiveness and cost,” in _Proceedings of the 5th Workshop on Embedded Systems Security_ , 2010, pp. 1–10.
  * [67] N. Moro, K. Heydemann, E. Encrenaz, and B. Robisson, “Formal verification of a software countermeasure against instruction skip attacks,” _Journal of Cryptographic Engineering_ , vol. 4, pp. 145–156, 2014.
  * [68] B. Selmke, J. Heyszl, and G. Sigl, “Attack on a dfa protected aes by simultaneous laser fault injections,” in _2016 Workshop on Fault Diagnosis and Tolerance in Cryptography (FDTC)_. IEEE, 2016, pp. 36–46.
  * [69] W. He, J. Breier, S. Bhasin, N. Miura, and M. Nagata, “An fpga-compatible pll-based sensor against fault injection attack,” in _2017 22nd Asia and South Pacific Design Automation Conference (ASP-DAC)_. IEEE, 2017, pp. 39–40.
  * [70] A. Baksi, S. Bhasin, J. Breier, D. Jap, and D. Saha, “A survey on fault attacks on symmetric key cryptosystems,” _ACM Computing Surveys_ , vol. 55, no. 4, pp. 1–34, 2022.