# An Energy-Efficient Reconfigurable Autoencoder Implementation on FPGA

Murat Isik†, Matthew Oldland†, Lifeng Zhou †The authors contributed equally to
the work.All authors are with Electrical and Computer Engineering, Drexel
University.Github Link: https://github.com/Muratcanisik4/CNN-Autoencoder.git

###### Abstract

Autoencoders are unsupervised neural networks that are used to process and
compress input data and then reconstruct the data back to the original data
size. This allows autoencoders to be used for different processing
applications such as data compression, image classification, image noise
reduction, and image coloring. Hardware-wise, re-configurable architectures
like Field Programmable Gate Arrays (FPGAs) have been used for accelerating
computations from several domains because of their unique combination of
flexibility, performance, and power efficiency. In this paper, we look at the
different autoencoders available and use the convolutional autoencoder in both
FPGA and GPU-based implementations to process noisy static MNIST images. We
compare the different results achieved with the FPGA and GPU-based
implementations and then discuss the pros and cons of each implementation. The
evaluation of the proposed design achieved 80% accuracy and our experimental
results show that the proposed accelerator achieves a throughput of 21.12
Giga-Operations Per Second (GOP/s) with a 5.93 W on-chip power consumption at
100 MHz. The comparison results with off-the-shelf devices and recent state-
of-the-art implementations illustrate that the proposed accelerator has
obvious advantages in terms of energy efficiency and design flexibility. We
also discuss future work that can be done with the use of our proposed
accelerator.

##  I INTRODUCTION

Autoencoders are an unsupervised neural network that aims to learn how to
compress input data and then reconstruct the data to match the input data with
the least amount of loss [1]. This is done using encoder and decoder methods,
where the encoder compresses the data and the decoder reconstructs the data
[1],[2],[3],[7]. There are many types of autoencoders that can be used in
machine learning; some examples are simple, convolutional, removing noise, and
sparse autoencoders. These different autoencoders can then be used for
different processing needs, like data compression before storage, image
classification, and removing noise from images.

This paper shows the use of convolutional autoencoders and how effective they
are at removing noise from images. Convolutional autoencoders use
convolutional neural networks for the layers of encoding and decoding to help
compress and reconstruct the image respectively. Convolution neural networks
use different filters to pull out features from the data. These filters are
designed to pull out specific features and are dependent on what the feature
needed is. The filters are designed to scan through the data and then create a
map of how well the data matched the filter by creating scores, where a higher
score is a better match [4]. The filter strides across the image based on how
the layer is set up by the user. These maps are then used in the next layer,
which is either a pooling layer or an up-sample layer. The pooling layer looks
at a window of the data and will combines the window into one sample. The pool
can use either a max of the data, the sum of the data, or the average of the
data to create the new sample [4]. The pooling layer is used in the encoding
process to help compress the data and store the important features. While the
up-sampling process is used in the decoding process to help with
reconstruction. There are many different ways to implement an autoencoder
using different hardware. This paper compares two methods, one is an
implementation of an FPGA using VHDL and the other is a GPU-based
implementation using python.

While developing the rest of the framework, machine vision framework
developers and respectability might become engrossed in deciding which of
these steps to use. The organize option is commonly chosen when prototyping a
system for the first time. A prototype application must be tested to determine
how many sections it must prepare every second or how many outlines it must
handle per second of the live video. The power consumption of embedded systems
is an important consideration for most applications, and researchers strive to
develop energy-efficient design methodologies [14-15].

The applications require for handling enormous data sets of high-resolution
image preparing applications requests faster, configurable, long throughput
frameworks with superior vitality productivity [16-17]. FPGAs (Field-
Programmable Gate Arrays) can play an important role since they provide
configurability, adaptability, and parallelism to coordinate the necessary
throughput rates of the application under consideration [13]. Real-world
applications can be implemented on FPGA devices thanks to their execution
capabilities. The development of FPGAs has taken hardware flexibility, in
general, one step further. In the end, The toolchain for developing
applications on these devices has also advanced significantly, allowing these
devices to be used by a larger building community [18]. FPGAs are typically
used in applications that demand concurrency, high transfer speeds, and re-
programmability.

FPGAs have achieved rapid acceptance and growth over the past decade because
they can be used in a very wide range of applications. The algorithms that
extract features are time-consuming, which is a huge drawback when developing
real-time applications. One solution to this problem is the use of dedicated
hardware for the algorithms, such as the FPGA, which can provide dedicated
functional blocks that perform complex image processing operations in
parallel. FPGAs have achieved rapid acceptance and growth over the past decade
because they can be used in a very wide range of applications [19].

We designed the convolutional autoencoder in both FPGA and GPU-based
implementations to process noisy static MNIST images. Results show that the
proposed design is both area- and power-efficient.

##  II AUTOENCODER TYPES

Autoencoders are a type of unsupervised neural network and are used for
various processing applications. They are trained to encode and compress input
data then reconstruct and decode the data back to try to match the input data.
There are three main parts to autoencoders [3]:

  * •

Encode and Compress,

  * •

Bottleneck,

  * •

Decode and reconstruct.

The encode and compress part of the autoencoder is composed of layers of
neural networks that work to compress the data down to a set size. This
encoded and compressed data is considered the bottleneck part since it is the
smallest part of the system before getting reconstructed. The data could then
be stored if needed, which allows for less required storage capacity. The
compressed data is then used in the next part of the system, the decoding, and
reconstruction part. This part works to decode the encoded data and
reconstruct the data back to match the input data that was passed in. It does
this by using neural networks that are designed the same as in the encode and
compress part, however instead of compressing the data it is up-sampled
instead. The number of layers used in both the compression and reconstruction
parts is set by the user based on the required implementation needed. Figure
1, below, shows an example of an input image being compressed and
reconstructed to an output image that matches closely [2].

Figure 1: Example Autoencoder showing input image compression and
reconstruction [1]

There are many different types of autoencoders that are used within machine
learning. These autoencoders have the same flow and parts between them but the
layers used within the encoding and decoding change. The sections below cover
some of the different types of autoencoders that can be used.

###  II-A Simple Autoencoder

Simple autoencoders are the basic implementation of an autoencoder and are a
simple feed-forward network that can have multiple layers within the encode
and decode parts. One implementation had 3 layers and the layers were fully
connected to each other [1]. This implementation also had the input and out of
the autoencoder as the same size which forces the autoencoder to learn the
compressed data with no information loss [1].

###  II-B Convolutional Autoencoder

Convolutional autoencoders are similar to simple autoencoders but change the
layers to convolution layers using a convolution neural network. The
convolution neural network layers are set up with several parts [4]:

  * •

Convolution,

  * •

Rectified Linear Unit,

  * •

Pooling (Encode) or Up-Sample (Decode).

The convolution is performed by forming features out of the data and creating
feature maps using convolution window filters. These filters are used to
maintain the relationship between the pixels and the input image [4]. The
resulting features create high and low scores that are representative of how
well the pixels are in relation to the filter; the better the match the higher
the score. The convolution has two parameters to it, the padding and the
stride length [4]. The padding is so that feature has the same amount to work
on no matter the input data size. The stride parameter is set to determine how
many pixels the filter shifts over the input data when creating the feature.

The Rectified Linear Unit (ReLU) part is used to make sure that there are no
negative numbers going between layers. This is done by replacing any negative
number found with zeros to help the math work out. The pooling part is the
compression of the image. A window size is selected to combine parts of the
filtered features into one pixel of the new image to be passed to the next
layer. In the pooling window either the max value, average value, or summed
value within the window is passed into the next pixel of the new image. The
decoding and reconstruction layers will use an up-sample part instead of the
pooling part to increase the size of the image as it gets reconstructed back
to match the input size.

###  II-C Denoising Autoencoder

Denoising autoencoders [2] use Gaussian noise on the input data to disrupt the
data then the autoencoder is trained to reconstruct the data with the noise
removed. The goal of the denoising autoencoder is to be used for error
correction of data that has been corrupted. The effectiveness of the
autoencoder is measured by the reconstruction loss between the output data and
the input data.

###  II-D Sparse Autoencoder

Sparse Autoencoders use sparsity in the layers to help with keeping the
reconstruction error low and have a meaningful bottleneck image in the middle.
In [2], the sparsity was enforced by applying L1 regularization in the layer.
This changes the autoencoder optimization objective to:

$\text{argmin}_{A,B}E[\Delta(x,B\circ A(x)]+\lambda\sum_{i}|a_{i}|,$ (1)

where ai is the activation at the $i$th hidden layer and $i$ iterates over all
hidden activations [2]. Another method covered in [2], is the KL-divergence
method. This method measures the distance between two probabilities by
changing the activation ai to a Bernoulli variable with a probability of $p$
[2]. The method tweaks the probability $p$ by taking the measured probability
for a batch, finding the difference, and then applying that as the
regularization factor.

##  III Method

We consider an image has been corrupted by noise and requires the noise to be
removed for processing later on. A convolutional autoencoder was designed to
perform image noise reduction on the corrupted input images. The convolutional
autoencoder is designed using three convolution layers and three pooling
layers, interleaved, for encoding and compression. The decoding and
reconstruction used four convolution layers and three up-sampling layers that
were interleaved. The pooling layers were constructed to use the maximum
pooling operation when operating over the windows of the images.

Layer | Type | Window Size | Stride
---|---|---|---
1 | Convolution | 4 | 1
2 | Max Pool | 4 | 1
3 | Convolution | 4 | 1
4 | Max Pool | 4 | 1
5 | Convolution | 4 | 1
6 | Max Pool | 4 | 1
7 | Convolution | 4 | 1
8 | Up-Sample | 4 | 1
9 | Convolution | 4 | 1
10 | Up-Sample | 4 | 1
11 | Convolution | 4 | 1
12 | Up-Sample | 4 | 1
13 | Convolution | 4 | 1
TABLE I: Layer Construction for Convolutional Autoencoder

Table I shows the layer construction of the convolutional autoencoder for
encoding and decoding.

The convolution layers were designed using convolution neural networks to
train the weights and perform feature extraction on the image before the
pooling or up-sampling layers. A 2-D convolution was used as the filter to
create the features from the pixels. The general 2-D function equation used is
given by

$f=\sum_{a=-y}^{y}\sum_{b=-z}^{z}x_{i-a,j-b}*k_{a,b}.$ (2)

After the convolutional filters are run over the pixels a ReLU function is
performed to make sure that no negative numbers are passed to the next layer.
This allows for the math to always come back correct and not have errors when
performing calculations. The ReLU function used is given by

$y=\max(0,x),$ (3)

where x is the filtered pixel feature. The output of the ReLU function is then
passed to the next layer of either pooling or up-sampling. The pooling
function was a max function of window size of the filtered pixel features and
is given by

$y=\max(x_{1},1,...,x_{i,j}),$ (4)

where $i$ and $j$ are the window size chosen for the layer.

##  IV RESULTS

###  IV-A Dataset

This dataset is to be a noisy variation of the MNIST digit set, with the noise
coming from an internal sense of style. The training set receives 80% of the
whole data, whereas the validation set receives 20% of the total data. The
actual training data contains 60,000 photos in all. Since the dataset, we
require for the task at hand requires noisy images, we define functions to add
random gaussian noise to the dataset elements. The new dataset now consists of
noisy images and can be visualized.

Figure 2: Project noisy dataset

Our noisy dataset is shown Figure 2. Training data contains 60k photos in all.
We use the random gaussian noise function to make the MNIST dataset into a
noisy dataset.

Gaussian noise is a type of noise that is in the form of Gaussian
distribution, such as the white noise commonly encountered. It is random-
valued and in impulses.

###  IV-B Training Details

We eventually settled on the 20-epoch training time and the training loss and
validation loss are displayed at the end of each epoch. Training time for each
model amounted to about 100 seconds each due to the input images being of
small size and hence not requiring too much processing power. Although it can
occasionally reduce the visibility of already visible numerals, it does a
better job of cleaning out the number and making it far more visible in more
challenging circumstances like the first image. For a modest number of
training parameters, the results are reasonable, with the greatest accuracy of
80.83%. The output of our autoencoding process is shown in Figure 3.

Figure 3: Denoised Image

###  IV-C Experimental Setup

We focused on two approaches to physically implementing the convolutional
autoencoder; one using a GPU and the other using an FPGA hardware
implementation. The GPU implementation was done using Python and Google
Colaboratory. Google Colaboratory allowed for running the machine learning
python executable on an NVIDIA Tesla K80. We implement the same design on the
XCZU7EV FPGA chip having a 504K lookup table (LUT) and used 2021.1 software
version of Vivado.

###  IV-D FPGA Implementation

The encoded images are further analyzed with a convolutional autoencoder
architecture comprising of matrix operation, channel distributor, encoding
module, round-robin arbitration, and output controller blocks.

Matrix operation: The boundary information during the convolution operation
(28*28*1).

Channel Distributor: Matrix is sent to the channel distributor module, which
contains a channel controller and first-in-first-out memories. The channel
controller interleaves channel data points and then sends them to the FIFOs in
sequence. The FIFOs buffer the interleaved data and then divide them into
eight channels.

Encoding and decoding module: First, the convolutional encoding module does
the convolution process, activation, and then max-pooling. Decoding is
performed by up-sampling rather than max-pooling.

Round robin arbitration: The output data of the convolutional encoding module
are orderly packaged and then sent to the output controller. Round-Robin
arbitrator is giving priority to requestors.

Output controller: This module is used to judge whether the data stream is
processed completely.

Figure 4: RTL Design of the Implementation

###  IV-E Comparision

Figure 5: Block Diagram of Implementation TABLE II: Autoencoder on different
heterogeneous devices

Platform | Technology | Operation Frequency | Power | Latency | Throughput | Energy Efficiency
---|---|---|---|---|---|---
NVIDIA K80 | ASIC (28nm) | $1.48GHz$ | $30W$ | $1137.62ms$ | $22000(GOP/s)$ | $733.33(GOP/s/W)$
NVIDIA GTX 1080 TI | GPU(16nm) | $1.48GHz$ | $250W$ | $6.15ms$ | $235.77(GOP/s)$ | $0.94(GOP/s/W)$
Chakradhar et al. [12] | FPGA(28nm) | $200MHz$ | $15W$ | $-$ | $16(GOP/s)$ | $1.06(GOP/s/W)$
Gokhale et al. [11] | FPGA(28nm) | $150MHz$ | $8W$ | $4.50ms$ | $23.18(GOP/s)$ | $2.90(GOP/s/W)$
Zhang et al. [5] | FPGA(28nm) | $100MHz$ | $18.61W$ | $21.61ms$ | $61.62(GOP/s)$ | $3.31(GOP/s/W)$
Ours | FPGA(16nm) | $100MHz$ | $5.93W$ | $2.91ms$ | $21.12(GOP/s)$ | $3.56(GOP/s/W)$

The implementation of Neural Networks (NNs) has been done using CPUs, GPUs,
and ASICs. CPU/GPU-based NNs consume a lot of power and have a limited
throughput due to limited memory bandwidth. Many researchers have developed
custom ASICs for accelerating network inference workloads in order to achieve
the best performance and energy efficiency. In spite of their attractiveness,
ASICs cannot provide sufficient flexibility to accommodate the rapid
development of Neural Networks, and FPGAs function as programmable devices
that can construct unique logic, alleviating constraints on neural network
implementation. As a result, one of the current research hotspots involves the
development of hardware systems supporting NN inference based on FPGA to
achieve high throughput and power efficiency.

As shown in Table II, we compare our CNN autoencoder implementation with other
CNN autoencoder models presented in the literature. The evaluation of the
proposed design achieved 80% accuracy and our experimental results show that
the proposed accelerator achieves a throughput of 21.12 Giga-Operations Per
Second (GOP/s) with a 5.93 W on-chip power consumption at 100 MHz. The
comparison results with off-the-shelf devices and recent state-of-the-art
implementations illustrate that the proposed accelerator has obvious
advantages in terms of energy efficiency and design flexibility.

##  V Conclusion and Future Work

In a confusion matrix, the model determined which classes it is accurately
forecasting and which classes it is forecasting inaccurately. More training
parameters and deeper layers in the autoencoder could improve the model’s
performance. However, in the context of this study, the model performs
admirably in reassembling denoised photos. It is shown that the use of
convolutional layers is very important in the case of denoising images using
deep learning based on a distinction between encoder and decoder
architectures. The model is implemented for hardware acceleration with various
heterogeneous devices and resulting in an energy-efficient, reconfigurable
system on the latter. Therefore, a deeper model and a more suitable latent
space dimension can be used for more complex denoising involving RGB images.
Future work scenarios could be removing noise from pictures received from
cameras attached to autonomous robots before image processing, as the next
step of this research. We are seeking to integrate our model into robots so
that a response can be made by cleaning images received wirelessly from
different robots if the transition is not clear before detecting position.

## References

  * [1] Zhang, Yifei. "A better autoencoder for image: Convolutional autoencoder." ICONIP17-DCEC. Available online: http://users. cecs. anu. edu. au/Tom. Gedeon/conf/ABCs2018/paper/ABCs2018_paper_58. pdf (accessed on 23 March 2017). 2018.
  * [2] Bank, Dor, Noam Koenigstein, and Raja Giryes. "Autoencoders." arXiv preprint arXiv:2003.05991 (2020).
  * [3] Badr, Will. “Auto-Encoder: What Is It? And What Is It Used for? (Part 1).” Medium, 9 Dec. 2021, towardsdatascience.com/auto-encoder-what-is-it-and-what-is-it-used-for-part-1-3e5c6f017726.
  * [4] Dataman, Chris Kuo/Dr. “Convolutional Autoencoders for Image Noise Reduction.” Medium, 8 Sept. 2022, towardsdatascience.com/convolutional-autoencoders-for-image-noise-reduction-32fce9fc1763.
  * [5] Zhang, Chen, et al. "Optimizing FPGA-based accelerator design for deep convolutional neural networks." Proceedings of the 2015 ACM/SIGDA international symposium on field-programmable gate arrays. 2015.
  * [6] Xu, Jiayang, and Karthik Duraisamy. "Multi-level convolutional autoencoder networks for parametric prediction of spatio-temporal dynamics." Computer Methods in Applied Mechanics and Engineering 372 (2020): 113379.
  * [7] Turchenko, Volodymyr, Eric Chalmers, and Artur Luczak. "A deep convolutional auto-encoder with pooling-unpooling layers in caffe." arXiv preprint arXiv:1701.04949 (2017).
  * [8] Chen, Min, et al. "Deep feature learning for medical image analysis with convolutional autoencoder neural network." IEEE Transactions on Big Data 7.4 (2017): 750-758.
  * [9] Koushik, Jayanth. "Understanding convolutional neural networks." arXiv preprint arXiv:1605.09081 (2016).
  * [10] Wang, Wei, et al. "Generalized autoencoder: A neural network framework for dimensionality reduction." Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2014.
  * [11] Gokhale, Vinayak, et al. "A 240 g-ops/s mobile coprocessor for deep neural networks." Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2014.
  * [12] Chakradhar, Srimat, et al. "A dynamically configurable coprocessor for convolutional neural networks." Proceedings of the 37th annual international symposium on Computer architecture. 2010.
  * [13] Woods, Roger, et al. FPGA-based implementation of signal processing systems. John Wiley & Sons, 2008.
  * [14] Camposano, Raul, and Jörg Wilberg. "Embedded system design." Design Automation for Embedded Systems 1.1 (1996): 5-50.
  * [15] Polzer, Andreas, et al. "Managing complexity and variability of a model-based embedded software product line." Innovations in Systems and Software Engineering 8.1 (2012): 35-49.
  * [16] Kattenborn, Teja, et al. "Review on Convolutional Neural Networks (CNN) in vegetation remote sensing." ISPRS Journal of Photogrammetry and Remote Sensing 173 (2021): 24-49.
  * [17] Benzigar, Mercy R., et al. "Advances on emerging materials for flexible supercapacitors: current trends and beyond." Advanced Functional Materials 30.40 (2020): 2002993.
  * [18] Sass, Ronald, and Andrew G. Schmidt. Embedded systems design with platform FPGAs: principles and practices. Morgan Kaufmann, 2010.
  * [19] Trimberger, Stephen M. Steve. "Three ages of fpgas: a retrospective on the first thirty years of fpga technology: this paper reflects on how moore’s law has driven the design of fpgas through three epochs: the age of invention, the age of expansion, and the age of accumulation." IEEE Solid-State Circuits Magazine 10.2 (2018): 16-29.


# Adversarial training may be
a double-edged sword

Ali Rahmati∗, Seyed-Mohsen Moosavi-Dezfooli†, and Huaiyu Dai∗
∗North Carolina State University, Raleigh, NC
†Institute for Machine Learning, ETH Zürich
{arahmat<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Adversarial training has been shown as an effective approach to improve the
robustness of image classifiers against white-box attacks. However, its
effectiveness against black-box attacks is more nuanced. In this work, we
demonstrate that some geometric consequences of adversarial training on the
decision boundary of deep networks give an edge to certain types of black-box
attacks. In particular, we define a metric called robustness gain to show that
while adversarial training is an effective method to dramatically improve the
robustness in white-box scenarios, it may not provide such a good robustness
gain against the more realistic decision-based black-box attacks. Moreover, we
show that even the minimal perturbation white-box attacks can converge faster
against adversarially-trained neural networks compared to the regular ones.

##  1 Introduction

Adversarial examples can be crafted with intentionally designed perturbations
added to the inputs to fool deep neural network classifiers. Adversarial
attacks may be executed in different categories depending on the attacker’s
level of information, including the white-box setting (Carlini & Wagner, 2017;
Goodfellow et al., 2014; Moosavi-Dezfooli et al., 2016), the score-based
black-box setting (Chen et al., 2017; Ilyas et al., 2018; Narodytska &
Kasiviswanathan, 2016) and the decision-based black-box setting (Chen et al.,
2020; Cheng et al., 2019; Liu et al., 2019; Rahmati et al., 2020). Generally,
in the white-box scenario, the effectiveness of the attack is measured by the
norm of minimal perturbations required to fool the network, while in the
black-box settings, norm of perturbations for a given number of queries is
evaluated. Moreover, among the adversarial attacks in the literature, the most
challenging ones are decision-based attacks, in which the attacker only has
access to the output label of the image classifier for a given input. On the
other hand, adversarial training has been shown to be an effective method to
improve the robustness of the image classifiers against adversarial attacks in
the white-box setting (Madry et al., 2018; Shafahi et al., 2019; Wong et al.,
2019). However, the effectiveness of adversarial training is not fully
understood in the more practical real-world black-box settings. Therefore, the
goal of this paper is to study the effectiveness of adversarial training in
improving the robustness of image classifiers against decision-based black-box
attacks where the attacker’s level of information from the deep neural network
image classifier is the least.

In order to craft an adversarial perturbation in a decision-based black-box
setting, the critical information is mostly the normal vector to the decision
boundary of the image classifier. Obviously, the less information available to
the attacker (which is more realistic), the less successful attack can be
done. The estimation of the normal vector to the boundary in this setting is
conducted with carefully designed queries on the boundary of the image
classifier with the underlying assumption that the decision boundary has low
mean curvature in the vicinity of inputs (Chen et al., 2020; Cheng et al.,
2019; Liu et al., 2019; Rahmati et al., 2020). The goal of such black-box
attacks is to reduce the number of queries as much as possible with an
efficient estimate of the normal vector to the boundary. Therefore, such
estimators are expected to work better if the decision boundary is less curved
as most of them rely on some sort of linearization of the decision boundary.
Thus, a flatter boundary would be favorable to these types of attacks.

Interestingly, it is empirically shown that adversarial training leads to
neural networks with flatter decision boundaries, compared to the boundaries
learned by regular training methods (Qin et al., 2019; Moosavi-Dezfooli et
al., 2019). We will show that such feature of the adversarially-trained
networks is indeed favorable for black-box attacks. The goal of this paper is
to show some evidence that although the adversarial training improves the
robustness of deep image classifiers effectively (i.e., it increases the
minimum distance of datapoints to the boundary) against the minimal norm
perturbation white-box attacks, it becomes less effective in more practical
attack settings. That is, decision-based black-box attacks can exploit the
excessive flatness caused by adversarial training. In particular, we define a
metric called robustness gain as the ratio of $\ell_{2}$ norm of adversarial
perturbation required to fool the adversarially-trained network to that for
the regular network. We observe that the robustness gain against the real-
world attacks in which the least amount of information is available to the
attacker is not as good as the one in the ideal white-box scenario with full
information. We also show that even iterative white-box attacks may converge
faster against adversarially-trained networks due to their flatter boundary
and more linear behaviour of such networks.

The rest of the paper is organized as follows. In Section 2, we study the
effectiveness of adversarial training against decision-based black-box
attacks. In Section 3, we evaluate the white-box attack performance against
adversarially-trained networks. Finally, in Section 4, we conclude the paper
and provide the direction forward.

##  2 Adversarially-trained networks and decision-based black-box attacks

Although it is known that adversarial training is a quite effective approach
to make image classifiers robust in the white-box settings, their performance
in the more practical settings is not well investigated. In this section, our
goal is to evaluate the effectiveness of adversarial training against
decision-based black-box attacks in which the attacker has only access to the
output label of the image classifier for a given input. We consider the most
challenging setting in which only the top-1 label of the deep classifier is
available to the attacker. Generally, such attacks aim to obtain the
$\ell_{2}$-minimized norm perturbation with the minimum number of queries.

The most successful decision-based black-box attacks in the literature (Chen
et al., 2020; Cheng et al., 2019; Liu et al., 2019; Rahmati et al., 2020) aim
to estimate the normal vector to the decision boundary with a local
linearization of the decision boundary at a randomly obtained boundary point.
Such estimators rely on the fact that the boundary of the image classifiers
has a low mean curvature in the vicinity of the data samples. As a result, the
less curved the boundary is, the more valid such an assumption is. An
important observation regarding adversarial training (Madry et al., 2018) is
that, in addition to imposing a larger distance between the data point and the
decision boundary (hence resulting in a higher robustness), the decision
regions of adversarially trained networks get flatter and more regular (Qin et
al., 2019). In particular, the curvature of the decision boundary decreases
after adversarial training (Moosavi-Dezfooli et al., 2019). It is intuitive
that a normal vector estimator will provide a better estimation on smoother
boundaries regardless of the boundary point at which the estimation is done.
In particular, the flatter the boundary the more aligned the directions of the
normal vectors at different boundary points are. In the extreme case that the
boundary is a hyperplane, the normal vectors to the boundary are in the same
direction throughout the hyperplane. Thus, interestingly, this results in a
better estimation of the normal vector to the boundary for a adversarially-
trained classifier in the black-box setting. This interesting observation
might be exploited as a future direction to develop new attacks with
potentially a smaller number of queries to estimate the normal vector leading
to a more effective attack.

Although adversarial training provides a quite impressive robustness against
attackers by increasing the distance of the input to the boundary in the
white-box setting, our goal is to see how such neural networks behave in
query-limited black-box settings. We define a metric called robustness gain
$\eta=\ell_{2}^{\textrm{robust}}/\ell_{2}^{\textrm{regular}}$ as the ratio of
$\ell_{2}$ norm of the adversarial perturbation required to fool the
adversarially-trained network to that for the regular network. We conduct the
experiments on a pre-trained ResNet-50 (He et al., 2016) called the regular
network and the adversarially-trained ResNet-50 (Madry et al., 2018) called
the robust network throughout the paper. We consider a random set of 300
correctly classified images by both networks from the ILSVRC2012’s validation
set (Deng et al., 2009).

### Black-box attacks performance evaluation

We compare the performance of black-box attacks HSJA (Chen et al., 2020),
GeoDA (Rahmati et al., 2020), boundary attack (BA) (Brendel et al., 2018) on
both regular and adversarially-trained ResNet-50 networks in Fig. 1a. As
shown, for a given query budget, the $\ell_{2}$ norm of perturbations for the
attacks against the adversarially-trained network is larger compared to that
of the regular network as expected. However, an interesting observation is
that while GeoDA has almost the same $\ell_{2}$ norm as HSJA for the regular
network, it provides much smaller $\ell_{2}$ norm for perturbations against
the adversarially-trained network compared to HSJA for a fixed amount of query
budget. The main reason for this phenomenon is that GeoDA is explicitly built
based on the assumption that the boundary of the classifier has a low mean
curvature. On the other hand, adversarially trained-networks has flatter
decision boundaries which actually gives an edge to GeoDA. Thus, to attack
adversarially-trained networks more efficiently, it is beneficial for the
attackers to deploy attacks exploiting the flatness of the decision boundary.

(a)  (b)

Figure 1: (a) Performance comparison of different black-box attacks for both
regular and adversarially-trained ResNet50. (b) The robustness gain for
$\ell_{2}$ norm under different attack scenarios.

### Robustness gain

In this part, our goal is to evaluate how much adversarially-trained networks
can improve the robustness under various kind of attacks. We plot the
robustness gain for different attacks in Fig. 1b. The larger the $\eta$ for a
given attack is, the better the adversarial training can improve the
robustness compared to the case of the regular network. In Fig. 1b, it is
observed that $\eta$ is equal to approximately 17 (see Table 1) for the white-
box attack DeepFool (DF) (Moosavi-Dezfooli et al., 2016) which is a quite good
improvement. However, in the case of decision-based black-box setting, first,
it can be seen that the robustness gain is lower than that of the white-box
attack in general. Second, by extracting more information from the image
classifier (getting more queries), the robustness gain increases. It implies
that the less information attacker knows about the image classifier, i.e., the
more practical the attack scenario is (e.g., from white-box to black-box, or
by decreasing the number of queries in the black-box setting), the less the
adversarially-trained network can improve the robustness. That being said,
adversarial training for the deep image classifiers is much more effective
against white-box scenarios than against black-box scenarios.

##  3  Adversarially-trained networks and minimal norm white-box attacks

We already discussed the effectiveness of the adversarially-trained network
with respect to the number of required queries against decision-based black-
box attacks in the query-limited regime. In the white-box scenario, we
evaluate the effectiveness through the number of required iterations for the
convergence of a minimal $\ell_{2}$ norm perturbation white-box attack. To
this end, we choose a minimal $\ell_{2}$ norm white-box attack DeepFool
(Moosavi-Dezfooli et al., 2016) and compare its performance on an
adversarially-trained (Madry et al., 2018) and a regular ResNet-50 in Table 1.
The main reason we choose DeepFool is its dependence on linearizing the output
function of the classifier. The algorithm starts with locally linearizing the
output function of the classifier and repeats such an approximation
iteratively to compensate for the effect of the non-linearity of the output
function. The more linear the output function of the image classifier is, the
fewer iterations required for DeepFool to converge. Interestingly, despite
that the adversarially-trained network requires perturbations with larger
$\ell_{2}$ norm to be fooled, due to the more linear behavior of its output
function, DeepFool converges faster on this network. In this sense, one can
conclude that there is a trade-off to attack adversarially-trained networks
even in the white-box setting.

| Median of Iterations | Max of Iterations | $\ell_{2}$ norm
---|---|---|---
Regular (He et al., 2016) | 4 | 15 | 0.209
Adv. trained (Madry et al., 2018) | 2 | 4 | 3.618
Table 1:  DeepFool (Moosavi-Dezfooli et al., 2016) performance on
adversarially-trained and regular ResNet-50 networks (over 1000 correctly
classified samples from ImageNet validation set).

In the next experiment, the goal is to qualitatively study the behaviour of
output function along the trajectory of the iterations of DeepFool for a
single data point. In this case, DeepFool requires 3 and 5 iterations to
converge for adversarially-trained and regular ResNet-50, respectively. We
consider the difference of the logits corresponding to the original and the
adversarial labels for our evaluation. We track this difference along two
paths: 1) the straight path between the original image and the DeepFool
adversarial example (i.e., green line in Figs. 2a and 2b), and 2) the path
taken by DF in each iteration (i.e. black and red line segments). We generate
images on the line from the original image to the minimal perturbation
adversarial example obtained by DeepFool. By varying the line parameter $t$,
we consider the images along the line
$\bm{x}=\bm{x}_{0}+t(\bm{x}_{adv}-\bm{x}_{0})$, where $t=0$ corresponds to the
original image and $t=1$ gives the adversarial image which falls on the
boundary. When the image is on the clean label side, the output value of the
clean label is larger than the adversarial label. Approaching the boundary,
this difference decreases where on the boundary the difference is equal to
zero and the transition occurs. Assuming $\bm{x}_{i}$ as the output of
DeepFool in iteration $i$, each line segment $i$ (i.e. black and red segments
in Figs. 2a and 2b) is corresponding to the images on the line
$\bm{x}=\bm{x}_{(i-1)}+t(\bm{x}_{i}-\bm{x}_{(i-1)})$ for $t\in[0,1]$, where
$\bm{x}_{i}=\bm{x}_{adv}$ if $i$ is the last iteration. First, it can be seen
that the straight path (green line) is much closer to the path constructed
with DeepFool iterations for the adversarially-trained network compared to
that of regular network. Second, it is shown that even in each line segment
corresponding to each iteration traversed by DeepFool algorithm, there is more
non-linearity in regular networks. As a result, it can be seen that although
adversarial training improve the robustness (increases the minimal $\ell_{2}$
norm required for successful attack), it gives an edge to the attacker (with a
smaller number of iterations to converge) due to more linear behaviour of
adversarially-trained networks.

(a)  (b)

Figure 2: Performance evaluation of DeepFool over different iterations on (a)
Regular ResNet 50 network. (b) Adversarially-trained ResNet 50 network.

##  4 Conclusion and Direction forward

We showed that although adversarial training is quiet effective against white-
box attacks, in query-limited decision-based black-box attacks, it may not
perform as efficiently as in the case for the white-box attacks. We
demonstrated that since adversarial training leads to a significantly flatter
boundary and a more linear behavior of the image classifier, it can give an
edge to certain types of black-box attackers whose goal is to estimate the
normal vector to the boundary. This feature of the adversarially-trained
networks can also provide a chance for minimal norm perturbation whit-box
attacks to produce adversarial examples with a smaller number of iterations.
As a future direction, such properties of adversarially-trained networks can
be exploited, especially in the black-box settings, to design more effective
attacks against such networks.

## References

  * Brendel et al. (2018) Wieland Brendel, Jonas Rauber, and Matthias Bethge.  Decision-based adversarial attacks: Reliable attacks against black-box machine learning models.  In _International Conference on Learning Representations_ , 2018.
  * Carlini & Wagner (2017) Nicholas Carlini and David Wagner.  Towards evaluating the robustness of neural networks.  In _2017 ieee symposium on security and privacy (sp)_ , pp. 39–57, 2017.
  * Chen et al. (2020) Jianbo Chen, Michael I Jordan, and Martin J Wainwright.  Hopskipjumpattack: A query-efficient decision-based attack.  In _2020 IEEE symposium on security and privacy (sp)_ , pp. 1277–1294, 2020.
  * Chen et al. (2017) Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh.  Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models.  In _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , pp. 15–26. ACM, 2017.
  * Cheng et al. (2019) Minhao Cheng, Simranjit Singh, Patrick H Chen, Pin-Yu Chen, Sijia Liu, and Cho-Jui Hsieh.  Sign-opt: A query-efficient hard-label adversarial attack.  In _International Conference on Learning Representations_ , 2019.
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pp. 248–255, 2009.
  * Goodfellow et al. (2014) Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ , 2014.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Ilyas et al. (2018) Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin.  Black-box adversarial attacks with limited queries and information.  In _International Conference on Machine Learning_ , pp. 2137–2146. PMLR, 2018.
  * Liu et al. (2019) Yujia Liu, Seyed-Mohsen Moosavi-Dezfooli, and Pascal Frossard.  A geometry-inspired decision-based attack.  In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , pp. 4889–4897. IEEE Computer Society, 2019.
  * Madry et al. (2018) Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu.  Towards deep learning models resistant to adversarial attacks.  In _International Conference on Learning Representations_ , 2018.
  * Moosavi-Dezfooli et al. (2016) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard.  Deepfool: a simple and accurate method to fool deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 2574–2582, 2016.
  * Moosavi-Dezfooli et al. (2019) Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and Pascal Frossard.  Robustness via curvature regularization, and vice versa.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 9078–9086, 2019.
  * Narodytska & Kasiviswanathan (2016) Nina Narodytska and Shiva Prasad Kasiviswanathan.  Simple black-box adversarial perturbations for deep networks.  _arXiv preprint arXiv:1612.06299_ , 2016.
  * Qin et al. (2019) Chongli Qin, James Martens, Sven Gowal, Dilip Krishnan, Krishnamurthy Dvijotham, Alhussein Fawzi, Soham De, Robert Stanforth, and Pushmeet Kohli.  Adversarial robustness through local linearization.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.
  * Rahmati et al. (2020) Ali Rahmati, Seyed-Mohsen Moosavi-Dezfooli, Pascal Frossard, and Huaiyu Dai.  GeoDA: a geometric framework for black-box adversarial attacks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8446–8455, 2020.
  * Shafahi et al. (2019) Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein.  Adversarial training for free!  In _Advances in Neural Information Processing Systems_ , pp. 3358–3369, 2019.
  * Wong et al. (2019) Eric Wong, Leslie Rice, and J Zico Kolter.  Fast is better than free: Revisiting adversarial training.  In _International Conference on Learning Representations_ , 2019.