# Defense Against Adversarial Attacks using
Convolutional Auto-Encoders

Shreyasi Mandal
(Indian Institute of Technology, Kanpur)

###### Abstract

Deep learning models, while achieving state-of-the-art performance on many
tasks, are susceptible to adversarial attacks that exploit inherent
vulnerabilities in their architectures. Adversarial attacks manipulate the
input data with imperceptible perturbations, causing the model to misclassify
the data or produce erroneous outputs. This work is based on enhancing the
robustness of targeted classifier models against adversarial attacks. To
achieve this, an convolutional autoencoder-based approach is employed that
effectively counters adversarial perturbations introduced to the input images.
By generating images closely resembling the input images, the proposed
methodology aims to restore the model’s accuracy.

##  1 Introduction

Deep learning has made notable progress across a diverse range of machine
learning domains, including image classification, object detection and speech
recognition. The influence of deep learning extends to an increasing number of
real-world applications and systems. However, recent research highlights the
susceptibility of deep learning models to well-designed input samples, called
adversarial examples. Adversarial examples are imperceptible to humans but can
easily fool deep neural networks [1].
Szedegy et al. [2] manipulated state-of-the-art deep neural networks to
misclassify images by applying subtle and barely perceptible perturbations, as
seen in Figure 1, showcasing the vulnerability of these models to targeted
errors. Researchers discovered that a single perturbation could lead a neural
network to misclassify multiple classes, revealing inherent weaknesses in the
training algorithms. Despite neural networks being nonlinear functions [3], it
was surprising to find that linear approximations are effective in creating
adversarial perturbations. This unexpected result is attributed to the design
of easily trainable models, which empirically exhibit an accurate local linear
approximation for their loss function [4], allowing adversaries to make subtle
changes in the pixel values that lead to misclassification.

Figure 1: Szegedy et al. [2] were able to fool AlexNet [5]
by classifying a perturbed image of a dog into an ostrich

There has been a good amount of progress in research pertaining to defending
against these adversarial attacks. The defenses aim to either detect whether
an input is adversarial, or aim to modify the input so that it is no longer
adversarial in nature. Adversarial attackers on image classifiers fool neural
networks by perturbing inputs in the direction of the gradient of the target
model [6]. As a result, the perturbed image is a data-point that maps to a
different distribution than that of the original image dataset.
Adversarial attacks pose a significant threat to the security and reliability
of machine learning systems, particularly in domains such as autonomous
vehicles, fraud detection, and cybersecurity. As such, researchers and
practitioners in the field of machine learning are actively working to develop
defenses against these attacks. In this paper, two kinds of adversarial
attacks namely Fast Gradient Sign Method (FGSM) [4] and Projected Gradient
Descent (PGD) [7] have been explored on the MNIST [8] and Fashion-MNIST[9]
dataset and are defended using convolutional auto-encoders [10].

##  2 Related Work

###  2.1 Adversarial Attacks

Akhtar et.al. [11] proposes a survey of adversarial attacks on deep learning
in computer vision such as Box Constrained L-BFGS [12], FGSM [4], Jacobian
based Saliency Map Attack [13], One-Pixel Attack [14] and Carlini and Wagner
Attacks [15]. Despite the high accuracy of deep neural networks, they can be
vulnerable to input perturbations that can cause significant changes in their
outputs. The article reviews recent contributions that devise attacks and
defenses for deep learning, particularly in safety and security-critical
applications.
There are two main types of adversarial attacks: targeted [16] and untargeted
[17], and two main categories of attacks based on the attacker’s knowledge of
the model: white-box [18] and black-box attacks [19]. In a targeted attack,
the attacker tries to manipulate the model’s output so that it produces a
specific, predefined result. In contrast, in an untargeted attack, the
attacker aims to simply cause the model to make a mistake, without specifying
what that mistake should be. A white-box attack occurs when the attacker has
complete knowledge of the model, including its architecture, parameters, and
training data. This information can be used to craft a specific attack that
exploits weaknesses in the model. In contrast, a black-box attack is one in
which the attacker has little or no knowledge of the model’s internals, and
must rely on input/output observations to craft an attack.
The Fast Gradient Sign Method (FGSM) is a gradient-based technique designed
for efficient computation of norm-bounded perturbations, prioritizing
computation efficiency over achieving high fooling rates. Goodfellow et al.
[4] utilized FGSM to support their linearity hypothesis, suggesting that the
linear behavior induced by components like ReLUs [20] in high-dimensional
spaces makes modern neural networks vulnerable to adversarial perturbations.
Fast Gradient Sign Method computes $\rho$ as :

$\rho=\epsilon\>sign(\nabla\mathbb{J}(\theta,I_{c},l))$

where gradient of cost function around the current value of model parameters
w.r.t. $I_{c}$ and $\epsilon$ bounds the $l_{\infty}$ norm. Contrary to the
prevailing notion of non-linearity causing vulnerability, FGSM has become a
key influencer in white-box attack scenarios due to its foundational concept
of gradient ascent on the model’s loss surface for deception.
An intuitive extension of FGSM (single large step in the direction to increase
loss) is to iteratively take multiple small steps while adjusting the
direction after each step. The Basic Iterative Method (BIM) [21] does exactly
that, and iteratively computes the following:

$I_{\rho}^{i+1}=Clip_{\epsilon}\\{I_{\rho}^{i}+\alpha\;sign(\nabla\mathbb{J}(\theta,I_{\rho}^{i},l))\\}$

where $I_{\rho}^{i}$ denotes the perturbed image at the $i^{th}$ iteration.
$Clip_{\epsilon}\\{.\\}$ clips the values of the pixels of the image in its
argument at $\epsilon$ and $\alpha$ determines the step size (normally
$\alpha=1$).
The PGD adversarial attack was originally proposed by Madry et al. [7] as the
most strongest “first-order adversarial” attack. Actually, PGD is a well-known
optimization technique projecting gradients in a ball. PGD is basically the
same as BIM, described above, except that PGD starts at a random point in the
ball and perform random restarts. The ’l’ parameter in the above equation is
emphasized, because adversarial attacks can trick the ML classifier either as
targeted or untargeted. In the targeted case, the malicious attacker forces
the classifier to produce a certain class of output. On the other hand, the
classifier is expected to give a misclassification where the label of the
output class is not important in the untargeted case.
A relaxation term is introduced to the standard loss in Projected gradient
Descent (PGD) and FGSM which finds more suitable gradient-directions,
increases attack efficacy and leads to more efficient adversarial training.
Considering $L_{2}$ norm distortions, the Carlini and Wagner [15] attack is
presently the most effective white-box attack in the literature. However, this
method is slow since it performs a line-search for one of the optimization
terms, and often requires thousands of iterations. Rony et.al. [22] proposes
an efficient approach to generate gradient-based attacks that induce
misclassifications with low $L_{2}$ norm, by decoupling the direction and the
norm of the adversarial perturbation that is added to the image. GeoDA [23] is
a geometric framework to generate adversarial examples in one of the most
challenging black-box settings where the adversary can only generate a small
number of queries, each of them returning the top-1 label of the classifier.

###  2.2 Adversarial Defense

Deep learning algorithms have been shown to perform extremely well on many
classical machine learning problems. However, recent studies have shown that
deep learning, like other machine learning techniques, is vulnerable to
adversarial samples: inputs crafted to force a deep neural network (DNN) to
provide adversary-selected outputs. Such attacks can seriously undermine the
security of the system supported by the DNN, sometimes with devastating
consequences [24]. For example, autonomous vehicles can be crashed, illicit or
illegal content can bypass content filters, or biometric authentication
systems can be manipulated to allow improper access. Akhtar and Mian [11]
categorized defenses against adversarial attacks into three main groups. These
include (1) modifying target models for robustness, (2) altering inputs to
remove perturbations, and (3) integrating external modules, primarily
detectors, into the model. Since 2018, the trajectory of adversarial defense
research has predominantly followed these three avenues of approach.
Papernot et.al. [25] introduces a defensive mechanism called defensive
distillation to reduce the effectiveness of adversarial samples on DNNs. It
reduces the amplitude of the network gradient exploited by adversaries to
craft adversarial samples. Distillation Temperature (T) hyperparameter is
introduced in the softmax layer to reduce the amplitude of the gradients.
PuVAE [26] uses Variational Autoencoder [27] to remove the perturbation from
Adversarial Examples and then classify the image. During training, the
distribution of latent space is learnt for each class. At inference time, a
latent vector z is sampled for every class, which is passed through the
decoder and subsequently, the encoder. The class label is then selected
corresponding to the closest projection.

$y*=argmin_{y_{i}\in C}D(x,x^{\prime}_{y_{i}})$

The y* is termed as a ‘purified’ sample which is then sent to the image
classifier.
Defense GANS [28] use the generator part of GAN to generate an image very
close to the input (adversarial) image, and then send it to the classification
model. The model is trained on the real images using Wasserstein loss (WGAN)
[29] instead of the usual Jensen-Shannon Divergence (JSD) [30]. At the
inference time, prior to feeding an image x to the classifier, it is projected
onto the range of generator by minimizing the reconstruction error
$min_{z}||G(z)-x||_{2}^{2}$. The reconstructed image is then sent to the
classifier.

##  3 Methodology

###  3.1 Dataset

The publicly available pre-processed MNIST database [8] of handwritten digits
and the Fashion-MNIST database [9] of Zalando’s article images is used to
implement the proposed methodology.
The MNIST database of handwritten digits, has a training set of 60,000
examples, and a test set of 10,000 examples. It is a subset of a larger set
available from NIST. The digits have been size-normalized and centered in a
fixed-size image. Each example is a 28x28 grayscale image, associated with a
label from 10 classes.
The Fashion-MNIST database is designed to seamlessly replace the original
MNIST dataset in benchmarking machine learning algorithms. It mirrors the same
image size, structure, and training/testing splits, facilitating a
straightforward integration for comparative analysis.

(a)

(b)

Figure 2: Publicly available datasets (a) MNIST and (b) Fashion MNIST

###  3.2 Effect of Attacks:

To evaluate the robustness of the proposed methodology, it is tested against
two types of adversarial attacks: the Fast Gradient Sign Method (FGSM) [4] and
Projected Gradient Descent (PGD) [7]. Both FGSM and PGD attacks are white box
untargeted attacks.
For training the databases, VGG-16 [31] is used for multi-class
classification. Transfer learning [32] is used to get an accuracy of 99% on
the test dataset for MNIST and an accuracy of 92% on the test dataset for
Fashion-MNIST.

####  3.2.1 FGSM Attack

Table 1(a) below show the effect of epsilon ($\epsilon$) on the accuracy of
the MNIST and the Fashion-MNIST dataset with FGSM attack. As we can see, the
accuracy decreases to 5.23% from 99.01% as the value of $\epsilon$ increases
to 1.50 from 0 for the MNIST dataset. For the Fashion-MNIST dataset, the
accuracy decreases to 4.12% from 91.22% as the value of $\epsilon$ increases
to 1.50 from 0.

(a)

(b)

Figure 3: FGSM Attack with different $\epsilon$ values on (a) MNIST dataset
(b) Fashion-MNIST dataset.
The images in the upper row denote the adversarial images, while those on the
lower row correspond to
the perturbation added to the original image. The top left image corresponds
to the original image ($\epsilon=0$).

As evident from Fig 4, using large epsilon ($\epsilon$) values leads to
corruption of the label semantics making it impossible to retrieve the
original image. Hence, the $\epsilon$ value used in the workflow is 0.60.

$\epsilon$ vs Accuracy for FGSM attack
---
Epsilon |  Accuracy (w/o defense)
| MNIST | Fashion-MNIST
0.00 | 0.9901 | 0.9122
0.10 | 0.9099 | 0.4751
0.20 | 0.8237 | 0.3798
0.30 | 0.7274 | 0.2973
0.40 | 0.5840 | 0.2376
0.50 | 0.4054 | 0.1889
0.60 | 0.2648 | 0.1417
0.70 | 0.1610 | 0.0955
1.00 | 0.0531 | 0.0444
1.50 | 0.0523 | 0.0412
(a)

$\epsilon$ vs Accuracy for PGD attack
---
Epsilon |  Accuracy (w/o defense)
| MNIST | Fashion-MNIST
0.00 | 0.9922 | 0.9206
0.05 | 0.6130 | 0.3150
0.10 | 0.1723 | 0.2962
0.15 | 0.0418 | 0.2942
0.20 | 0.0181 | 0.2940
0.30 | 0.0118 | 0.2934
(b)

Table 1: (a) FGSM Attack and (b) PGD Attack on MNIST and Fashion-MNIST
datasets

####  3.2.2 PGD Attack

PGD is akin to i-FGSM i.e “many mini FGSM steps”. A slight difference lies in
the optimization algorithm used in this approach, compared to FGSM. FGSM uses
normal gradient descent steps. PGD in contrast, runs projected (normalized)
steepest descent under the $l_{\infty}$ norm.
Table 1(b) below show the effect of epsilon ($\epsilon$) on the accuracy of
the MNIST and the Fashion-MNIST dataset with PGD attack. As we can see, the
accuracy decreases to 1.18% from 99.22% as the value of $\epsilon$ increases
to 0.30 from 0 for the MNIST dataset. For the Fashion-MNIST dataset, the
accuracy decreases to 29.34% from 92.06% as the value of $\epsilon$ increases
to 0.30 from 0.

(a)

(b)

Figure 4: PGD Attack with different $\epsilon$ values on (a) MNIST dataset (b)
Fashion-MNIST dataset.
The images in the upper row denote the adversarial images, while those on the
lower row correspond to
the perturbation added to the original image. The top left image corresponds
to the original image ($\epsilon=0$).

###  3.3 Proposed Defense Architecture

An U-shaped convolutional auto-encoder as shown in Fig 5 is used to
reconstruct original input from the adversarial image, effectively removing
the adversarial perturbations. The goal of the autoencoder network is to
minimise the mean squared error loss [33] between the original unperturbed
image and the reconstructed image, which is generated using an adversarial
example. While doing so, a random Gaussian noise [34] is added after encoding
the image so as to make the model more robust. The idea behind adding the
noise is to perturb the latent representation by a small magnitude, and then
decode the perturbed latent representation, akin to how the adversarial
examples are generated in the first place.

Figure 5: Proposed Defense framework with a U-shaped Convolutional Auto-
Encoder and
a pre-trained VGG-16 Classifier

During inference, the adversarial image generated from the original image is
fed to the trained auto-encoder to get the reconstructed image close to the
original image as evident from Fig 7. This reconstructed images are then sent
to the pre-trained VGG-16 classifier network.
The GELU [35] activation layers are used in the model architecture. GELU being
a smoother activation function having a non-zero derivative for all inputs
helps to address the dying ReLU problem and allow for more effective learning
in deep neural networks.

(a)

(b)

(c)

(d)

Figure 7: Reconstructed images from the proposed auto-encoder architecture
during inference time.
(a) and (b) shows the recreated images from FGSM attacked MNIST and Fashion-
MNIST images
respectively for $\epsilon=0.60$. (c) and (d) shows the recreated images from
PGD attacked MNIST
and Fashion-MNIST images respectively for $\epsilon=0.15$

##  4 Results

Table 2 shows the model’s performance on the MNIST and Fashion-MNIST dataset.
The accuracy of the pre-trained VGG-16 classifier on the MNIST and Fashion-
MNIST dataset with FGSM attack increases by 65.61% and 59.76% respectively.
For the PGD attack, the accuracy increases by 89.88% and 43.49%. This shows
the efficacy of our model in defending the adversarial attacks with high
accuracy.

Attack |  Accuracy (w/o defense) |  Accuracy (with defense)
---|---|---
| MNIST | Fashion-MNIST | MNIST | Fashion-MNIST
FGSM ($\epsilon=0.60$) | 0.2648 | 0.1417 | 0.9209 | 0.7393
PGD ($\epsilon=0.15$) | 0.0418 | 0.2942 | 0.9406 | 0.7291
Table 2: Accuracy of VGG-16 on MNIST and Fashion-MNIST dataset

Table 3 shows the comparison of the proposed convolutional auto-encoder
architecture with the existing state-of-the-art models, mainly Defense GANS
[28] and PuVAE [26] for FGSM attack with $\epsilon=0.60$ on the MNIST dataset.
As evident from the table, the proposed architecture has more accuracy and a
much lower latency compared to the other models, particularly in applications
like autonomous driving where quick and precise decisions are crucial.

Criteria | No Attack | Defense GAN | PuVAE | Our Model
---|---|---|---|---
Accuracy | 0.9901 | 0.8529 | 0.8133 | 0.9209
Inference Latency | - | 14.80 s | 0.11 s | 0.008 s
Table 3: Comparison of the proposed architecture with existing SOTA models

##  5 Conclusion and Future Work

The employed convolutional autoencoder-based approach effectively counters
adversarial perturbations, restoring the model’s accuracy notably. With the
FGSM attack, accuracy on MNIST and Fashion-MNIST datasets increases by 65.61%
and 59.76% respectively, and with the PGD attack, by 89.88% and 43.49%. This
signifies the model’s robust defense against adversarial attacks, showcasing
its high accuracy.
However, unlike Defense GANs, which adeptly handle various attack types during
inference, our model operates under the assumption of a specific attack type,
like FGSM with $\epsilon=0.60$. This difference in approach might limit our
model’s adaptability to diverse adversarial scenarios encountered in practical
settings. Moreover, models like PuVAEs, with their increased complexity, tend
to achieve higher robustness, especially when dealing with intricate datasets.
Potential future work might include incorporating bayesian machine learning
techniques in the proposed architecture to make it more robust to attacks,
while keeping the inference latency in check as well.

## References

  * [1] X. Yuan, P. He, Q. Zhu, and X. Li, “Adversarial examples: Attacks and defenses for deep learning,” IEEE transactions on neural networks and learning systems, vol. 30, no. 9, pp. 2805–2824, 2019.
  * [2] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013.
  * [3] S. Grossberg, “Nonlinear neural networks: Principles, mechanisms, and architectures,” Neural networks, vol. 1, no. 1, pp. 17–61, 1988.
  * [4] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014.
  * [5] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, 2012.
  * [6] M. R. Graves, Image Perturbation Generation: Exploring New Ways for Adversaries to Interrupt Neural Network Image Classifiers.  PhD thesis, Monterey, CA; Naval Postgraduate School, 2020.
  * [7] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” arXiv preprint arXiv:1706.06083, 2017.
  * [8] L. Deng, “The mnist database of handwritten digit images for machine learning research,” IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 141–142, 2012.
  * [9] H. Xiao, K. Rasul, and R. Vollgraf, “Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms,” arXiv preprint arXiv:1708.07747, 2017.
  * [10] D. Bank, N. Koenigstein, and R. Giryes, “Autoencoders,” Machine Learning for Data Science Handbook: Data Mining and Knowledge Discovery Handbook, pp. 353–374, 2023.
  * [11] N. Akhtar and A. Mian, “Threat of adversarial attacks on deep learning in computer vision: A survey,” Ieee Access, vol. 6, pp. 14410–14430, 2018.
  * [12] R. Fletcher, Practical methods of optimization.  John Wiley & Sons, 2000.
  * [13] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, “The limitations of deep learning in adversarial settings,” in 2016 IEEE European symposium on security and privacy (EuroS&P), pp. 372–387, IEEE, 2016.
  * [14] J. Su, D. V. Vargas, and K. Sakurai, “One pixel attack for fooling deep neural networks,” IEEE Transactions on Evolutionary Computation, vol. 23, no. 5, pp. 828–841, 2019.
  * [15] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in 2017 ieee symposium on security and privacy (sp), pp. 39–57, Ieee, 2017.
  * [16] M. Li, C. Deng, T. Li, J. Yan, X. Gao, and H. Huang, “Towards transferable targeted attack,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 641–649, 2020.
  * [17] A. Wu, Y. Han, Q. Zhang, and X. Kuang, “Untargeted adversarial attack via expanding the semantic gap,” in 2019 IEEE International Conference on Multimedia and Expo (ICME), pp. 514–519, IEEE, 2019.
  * [18] J. Ebrahimi, A. Rao, D. Lowd, and D. Dou, “Hotflip: White-box adversarial examples for text classification,” arXiv preprint arXiv:1712.06751, 2017.
  * [19] A. Ilyas, L. Engstrom, A. Athalye, and J. Lin, “Black-box adversarial attacks with limited queries and information,” in International conference on machine learning, pp. 2137–2146, PMLR, 2018.
  * [20] A. F. Agarap, “Deep learning using rectified linear units (relu),” arXiv preprint arXiv:1803.08375, 2018.
  * [21] A. Kurakin, I. J. Goodfellow, and S. Bengio, “Adversarial examples in the physical world,” in Artificial intelligence safety and security, pp. 99–112, Chapman and Hall/CRC, 2018.
  * [22] J. Rony, L. G. Hafemann, L. S. Oliveira, I. B. Ayed, R. Sabourin, and E. Granger, “Decoupling direction and norm for efficient gradient-based L2 adversarial attacks and defenses,” CoRR, vol. abs/1811.09600, 2018.
  * [23] A. Rahmati, S. Moosavi-Dezfooli, P. Frossard, and H. Dai, “Geoda: a geometric framework for black-box adversarial attacks,” CoRR, vol. abs/2003.06468, 2020.
  * [24] G. Apruzzese, M. Colajanni, L. Ferretti, and M. Marchetti, “Addressing adversarial attacks against security systems based on machine learning,” in 2019 11th international conference on cyber conflict (CyCon), vol. 900, pp. 1–18, IEEE, 2019.
  * [25] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, “Distillation as a defense to adversarial perturbations against deep neural networks,” in 2016 IEEE Symposium on Security and Privacy (SP), pp. 582–597, 2016.
  * [26] U. Hwang, J. Park, H. Jang, S. Yoon, and N. I. Cho, “Puvae: A variational autoencoder to purify adversarial examples,” CoRR, vol. abs/1903.00585, 2019.
  * [27] C. Doersch, “Tutorial on variational autoencoders,” arXiv preprint arXiv:1606.05908, 2016.
  * [28] P. Samangouei, M. Kabkab, and R. Chellappa, “Defense-gan: Protecting classifiers against adversarial attacks using generative models,” CoRR, vol. abs/1805.06605, 2018.
  * [29] C. Frogner, C. Zhang, H. Mobahi, M. Araya, and T. A. Poggio, “Learning with a wasserstein loss,” Advances in neural information processing systems, vol. 28, 2015.
  * [30] M. Menéndez, J. Pardo, L. Pardo, and M. Pardo, “The jensen-shannon divergence,” Journal of the Franklin Institute, vol. 334, no. 2, pp. 307–318, 1997.
  * [31] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  * [32] L. Torrey and J. Shavlik, “Transfer learning,” in Handbook of research on machine learning applications and trends: algorithms, methods, and techniques, pp. 242–264, IGI global, 2010.
  * [33] M. S. Error, “Mean squared error,” MA: Springer US, pp. 653–653, 2010.
  * [34] D. Guo, Y. Wu, S. S. Shitz, and S. Verdú, “Estimation in gaussian noise: Properties of the minimum mean-square error,” IEEE Transactions on Information Theory, vol. 57, no. 4, pp. 2371–2385, 2011.
  * [35] D. Hendrycks and K. Gimpel, “Gaussian error linear units (gelus),” 2020.