# Robustifying $\ell_{\infty}$ Adversarial Training to the Union of
Perturbation Models

Ameya D. Patil, Michael Tuttle, Alexander G. Schwing, and Naresh R. Shanbhag
University of Illinois at Urbana-Champaign
Urbana, IL 61801
<EMAIL_ADDRESS>

###### Abstract

Classical adversarial training (AT) frameworks are designed to achieve high
adversarial accuracy against a single attack type, typically $\ell_{\infty}$
norm-bounded perturbations. Recent extensions in AT have focused on defending
against the union of multiple perturbations but this benefit is obtained at
the expense of a significant (up to $10\times$) increase in training
complexity over single-attack $\ell_{\infty}$ AT. In this work, we expand the
capabilities of widely popular single-attack $\ell_{\infty}$ AT frameworks to
provide robustness to the union of ($\ell_{\infty},\ell_{2},\ell_{1}$)
perturbations while preserving their training efficiency. Our technique,
referred to as Shaped Noise Augmented Processing (SNAP), exploits a well-
established byproduct of single-attack AT frameworks – the reduction in the
curvature of the decision boundary of networks. SNAP prepends a given deep net
with a shaped noise augmentation layer whose distribution is learned along
with network parameters using any standard single-attack AT. As a result, SNAP
enhances adversarial accuracy of ResNet-18 on CIFAR-10 against the union of
($\ell_{\infty},\ell_{2},\ell_{1}$) perturbations by $14\%$-to-$20\%$ for four
state-of-the-art (SOTA) single-attack $\ell_{\infty}$ AT frameworks, and, for
the first time, establishes a benchmark for ResNet-50 and ResNet-101 on
ImageNet.

##  1 Introduction

Today _adversarial training_ (AT) provides state-of-the-art (SOTA) empirical
defense against adversarial perturbations. For this, adversarial perturbations
are used during training to optimize a _robust_ loss function [22, 46, 35,
40]. Early AT frameworks [22, 46] were $7\times$-to-$10\times$ more
computationally demanding than vanilla training. More recent works [35, 40,
45] have significantly reduced the computational demands of AT via _single-
step attacks_ and _superconvergence_.

However, today’s AT frameworks predominantly focus on a _single-attack_ ,
_i.e_., they seek robustness to a single perturbation, typically
$\ell_{\infty}$-bounded [35, 40, 42, 46, 48, 45, 44, 29, 9, 39, 47, 10, 11,
14]. This results in low performance against other perturbations such as
$\ell_{2}$, $\ell_{1}$, or the union of ($\ell_{\infty},\ell_{2},\ell_{1}$).
Indeed, as shown in Fig. 1, four state-of-the-art (SOTA) single-attack AT
frameworks (_black markers_) employing only $\ell_{\infty}$-bounded
perturbations achieve low adversarial accuracy
$\mathcal{A}^{(U)}_{\text{adv}}$ of $\approx 15\%$-to-$20\%$ against the union
of ($\ell_{\infty},\ell_{2},\ell_{1}$) perturbations. Recent extensions in AT
[23, 37, 19] do seek higher $\mathcal{A}^{(U)}_{\text{adv}}$ but only at the
expense of a $6\times$-to-$10\times$ increase in the total training time
(_blue markers in Fig. 1_). The large training time of these AT frameworks has
inhibited their application to large-scale datasets such as ImageNet, _e.g_.,
Maini et al. [23], Tramèr & Boneh [37] show results for MNIST and CIFAR-10
only, while Laidlaw et al. [19] only additionally show $64\times 64$
ImageNet-100 results.

The high training time for AT frameworks arises from two sources: (i) the need
to employ larger networks, _e.g_., MSD [23] with ResNet-18 achieves higher
$\mathcal{A}^{(U)}_{\text{adv}}$ than PAT [19] with ResNet-50 (see Fig. 1);
and (ii) the need to incorporate multiple perturbations during each attack
step and a higher overall number of attack steps, _e.g_., 50 in MSD [23], 20
in AVG [37]. Obviously one can always reduce the number of attack steps in
MSD/AVG to proportionally reduce training time. Doing so results in training
time and $\mathcal{A}^{(U)}_{\text{adv}}$ to rapidly approach the training
complexity and $\mathcal{A}^{(U)}_{\text{adv}}$ of standard AT frameworks,
_e.g_., a 5-step MSD and 2-step AVG is equivalent in training time and
accuracy to PGD and TRADES, respectively. Notwithstanding the expensive nature
of 50-step multi-attack training, today MSD [23] achieves a SOTA
$\mathcal{A}^{(U)}_{\text{adv}}$ of 47% with ResNet-18 on CIFAR-10.

Figure 1: Adversarial accuracy ($\mathcal{A}^{(U)}_{\text{adv}}$) against
union of ($\ell_{\infty}$, $\ell_{2}$, $\ell_{1}$) _vs_. measured wall-clock
total training time on CIFAR-10 with different AT frameworks on single NVIDIA
TESLA P100 GPU. $\epsilon=(0.031,0.5,12)$ for ($\ell_{\infty}$, $\ell_{2}$,
$\ell_{1}$) perturbations, respectively. SNAP enhances robustness with a small
increase in training time. All frameworks except PAT employ ResNet-18.

This poses a question: can we approach the high robustness of multiple-attack
AT such as 50-step MSD against the union of
($\ell_{\infty},\ell_{2},\ell_{1}$) perturbations while maintaining the low
training time of fast single-attack AT frameworks such as FreeAdv [35] and
FastAdv [40]?

In our quest to answer this question we find that noise augmentation using
adequately shaped noise within standard single-attack AT frameworks employing
$\ell_{\infty}$-bounded perturbations significantly improves robustness
against the union of ($\ell_{\infty},\ell_{2},\ell_{1}$) perturbations. The
improvement appears to be a consequence of a well-established byproduct of AT
frameworks – the reduction in the curvature of the decision boundary of
networks trained using single-attack AT [6, 25]. We confirm this connection by
quantifying the impact of single-attack AT on the geometric orientations of
different perturbations.

Based on this insight, we propose Shaped Noise Augmented Processing (SNAP) –
_a method to enhance robustness against the union of perturbation types by
augmenting single-attack AT frameworks_. SNAP prepends a deep net with a
shaped noise (SN) augmentation layer (see Fig. 4) whose distribution parameter
$\Sigma$ is learned with that of the network ($\theta$) within any standard
single-attack AT framework. SNAP improves the robustness of four SOTA
$\ell_{\infty}$-AT frameworks against the union of
($\ell_{\infty},\ell_{2},\ell_{1}$) perturbations by 15%-to-20% on CIFAR-10
(_red markers in Fig. 1_) with only a modest ($\sim 10\%$) increase in
training time. This expands the capabilities of widely popular single-attack
$\ell_{\infty}$ AT frameworks to providing robustness to the union of
($\ell_{\infty},\ell_{2},\ell_{1}$) perturbations without sacrificing training
efficiency. We validate SNAP’s benefits via thorough comparisons with _nine
SOTA adversarial training and randomized smoothing frameworks_ across
different operating regimes on both CIFAR-10 and ImageNet.

One tangible outcome of our work – we demonstrate _for the first time_
ResNet-50 (ResNet-101) networks on ImageNet that achieve
$\mathcal{A}^{(U)}_{\text{adv}}=32\%$ (35%) against the union of
($\ell_{\infty}(\epsilon=2/255),\ell_{2}(\epsilon=2.0),\ell_{1}(\epsilon=72.0)$)
perturbations. Our code is available at https://github.com/adpatil2/SNAP.

##  2 Related Work

We categorize works on adversarial vulnerability of DNNs as follows:

Low-complexity adversarial training: The high computational needs of AT
frameworks has spurred significant efforts in reducing their complexity [45,
35, 40, 48]. FreeAdv [35] updates weights while accumulating multiple attack
iterations. FastAdv [40] employs _appropriate_ use of single-step attacks,
while Zheng et al. [48] leverage inter-epoch similarity between adversarial
perturbations. However, these fast AT methods seek robustness against a single
perturbation type, _e.g_., $\ell_{\infty}$ norm-bounded perturbations. In
contrast, SNAP expands the capabilities of these AT frameworks by enhancing
robustness to the union of three perturbation types
($\ell_{\infty},\ell_{2},\ell_{1}$), while preserving their efficiency.

Robustness against union of perturbation models: The focus on the robustness
against the union of multiple perturbation types is relatively new. Kang et
al. [17] studied transferability between different perturbation types, while
Jordan et al. [16] considered combination attacks with low perceptual
distortion. Stutz et al. [36] proposed a modification in AT to _detect_ images
with different models of perturbations via confidence thresholding, but they
don’t attempt to _classify_ perturbed images correctly. For accurate
classification in the presence of different perturbation models, Tramèr &
Boneh [37] studied empirical and theoretical trade-offs involved in including
multiple perturbation types simultaneously during training. Maini et al. [23]
further built upon this work to propose the multi steepest descent (MSD) AT
framework which chooses one among the three perturbation models
($\ell_{\infty},\ell_{2},\ell_{1}$) in each attack iteration during training,
achieving SOTA adversarial accuracy on CIFAR-10 against the union of the
($\ell_{\infty},\ell_{2},\ell_{1}$) perturbation models, albeit at a high
($10\times$) training time. In contrast, SNAP provides high robustness against
the union of ($\ell_{\infty},\ell_{2},\ell_{1}$) perturbation models using
established single-attack $\ell_{\infty}$ AT frameworks. This enables to
showcase the benefits of our approach on large-scale datasets such as
ImageNet.

Recently, Laidlaw et al. [19] developed a novel AT framework (PAT) with low
perceptual distortion attacks to demonstrate impressive generalization to
unseen attacks. In contrast, we focus on extending the capabilities of widely
popular $\ell_{\infty}$-AT frameworks to providing robustness against the
union of ($\ell_{\infty},\ell_{2},\ell_{1}$) perturbations, while preserving
their training efficiency.

Noise augmentation: Multiple recent works have investigated the role of
randomization in enhancing adversarial robustness [12, 26, 8, 27] with
theoretical guarantees. Another prominent line of work in this category is
randomized smoothing [5, 33, 20, 43], where random noise is used as a tool to
compute certification bounds. Rusak et al. [32] also explored the role of
noise augmentation for improving the robustness against common-corruptions
[13]. In contrast, in SNAP, noise augmentation is used as a means to enable
widely popular $\ell_{\infty}$-AT frameworks to efficiently achieve high
robustness against the union of multiple norm-bounded perturbations. As is the
characteristic of AT works, our results are primarily empirical in nature.
Hence, we follow recent guidelines [38, 23] to evaluate the accuracy against
the strongest possible adversaries. We do explicitly compare
$\ell_{\infty}$-AT+SNAP with randomized smoothing approaches in the Appendix
B.1.

##  3 Subspace Analysis of Adversarial Perturbations

Figure 2: Illustration of the role of decision boundary curvature on the
distinction between different types of perturbations $\bm{\alpha}$,
$\bm{\beta}$ and $\bm{\gamma}$ of the given input ${\bm{x}}$.

In this section, we employ subspace methods to comprehend the distinction
between $\ell_{\infty}$, $\ell_{2}$ and $\ell_{1}$ perturbations. For each
input ${\bm{x}}_{i}\in\mathbb{R}^{D}$ in dataset $X$, consider adversarial
perturbations $\bm{\alpha}_{i}$, $\bm{\beta}_{i}$, and $\bm{\gamma}_{i}$
bounded within $\ell_{\infty}$, $\ell_{2}$, and $\ell_{1}$ norms,
respectively.

We begin with a hypothesis (see Fig. 2): _The perturbations $\bm{\alpha}$,
$\bm{\beta}$, and $\bm{\gamma}$ corresponding to input ${\bm{x}}$ have
directions that differ significantly if the curvature of the decision boundary
is high in the neighborhood of ${\bm{x}}$. Conversely, if the curvature of the
decision boundary is low, the perturbations $\bm{\alpha}$, $\bm{\beta}$, and
$\bm{\gamma}$ tend to point in similar directions._

Since, prior works [6, 25] have found that single-attack AT reduces the
curvature of the decision boundary, we test our hypothesis by studying the
following two networks on CIFAR-10 data: a _non-robust_ ResNet18
$f^{\text{van}}_{\theta}$ trained using vanilla training, and a _robust_
ResNet18 $f^{\text{rob}}_{\theta}$ trained using the TRADES [46] AT framework
employing $\ell_{\infty}$ perturbations.

We compute perturbations $\bm{\alpha}_{i}$, $\bm{\beta}_{i}$, and
$\bm{\gamma}_{i}$ for each ${\bm{x}}_{i}\in X$ for both networks, _i.e_.,
$\kappa\in\\{\text{van},\text{rob}\\}$. We compute the singular vector basis
$\mathcal{P}^{\kappa}$ for the set of $\ell_{2}$ bounded perturbations
$\Delta^{\kappa}=\\{\bm{\beta}_{1}^{\kappa},\dots,\bm{\beta}_{|X|}^{\kappa}\\}$.
The normalized mean squared projections of the three types of perturbation
vectors on the singular vector basis $\mathcal{P}^{\kappa}$ of vanilla trained
ResNet-18 ($\mathcal{P}^{\text{van}}$)(Fig. 3(a)) and TRADES trained ResNet-18
($\mathcal{P}^{\text{rob}}$)(Fig. 3(b)) shows a clear contrast.

Figure 3: Normalized mean squared projections of three perturbation types on
the singular vector basis $\mathcal{P}^{\kappa}$ of $\ell_{2}$ perturbations
of ResNet18 on CIFAR-10 after: (a) vanilla training
($\kappa\equiv\text{van}$), and (b) TRADES training
($\kappa\equiv\text{rob}$). The singular vectors ${\bm{p}}_{i}^{\kappa}$
comprising
$\mathcal{P}^{\kappa}=\\{{\bm{p}}_{1}^{\kappa},\dots,{\bm{p}}_{D}^{\kappa}\\}$
are ordered in descending order of their singular values.

The perturbations of a vanilla trained network roll-off gradually to occupy a
larger subspace as indicated in Fig. 3(a). Specifically, the projections of
$\bm{\alpha}$ and $\bm{\gamma}$ occupy almost all 3000 directions in the basis
$\mathcal{P}^{\text{van}}$ since their mean squared projections are within
$\sim 10\%$ of the maximum value $m_{\text{max}}$. This shows that the
dominant singular vectors of $\bm{\beta}$ are not well-aligned with
$\bm{\alpha}$ and $\bm{\gamma}$ in a vanilla trained network. With TRADES AT
(Fig. 3(b)), however, all three types of perturbations are _squeezed_ into a
much _smaller_ subspace spanning only the top 250 singular vectors in the
perturbation basis $\mathcal{P}^{\text{rob}}$. Outside these 250 dimensions,
the mean squared projections fall to $<10\%$ of their maximum value.

In summary, the results in Fig. 3 validate the hypothesis that single-attack
AT increases the average alignment of different perturbation types due to the
reduction in the decision boundary curvature. In Sec. 4, we exploit this
behavior of single-attack $\ell_{\infty}$ AT to improve its robustness against
the union of multiple perturbation models via SNAP.

##  4 Shaped Noise Augmented Processing (SNAP)

Figure 4: SNAP: (a) intuition underlying SNAP (not an exact depiction), and
(b) SNAPnet $f^{\text{SN}}_{\theta,\Sigma}({\bm{x}})$ constructed from a given
deep net $f_{\theta}({\bm{x}})$ by prepending a shaped noise (SN) augmentation
layer which perturbs the primary input ${\bm{x}}$ with noise ${\mathbf{n}}$
whose distribution parameter $\Sigma$ is learned during AT along with the base
network parameter $\theta$.

We show that single-attack AT can be enhanced to address multiple
perturbations by introducing noise to appropriately _wiggle_ the
$\ell_{\infty}$-bounded perturbations (Fig. 4(a)). However, to do so, the
noise distribution needs to be _chosen_ and _shaped_ appropriately to minimize
its impact on natural accuracy and robustness to $\ell_{\infty}$-bounded
perturbations.

We experiment with both $\ell_{\infty}$ and $\ell_{2}$ perturbations in
single-attack AT frameworks and find $\ell_{\infty}$-AT to be suitable for our
proposed shaped noise augmentation (see Sec. 5.2.1 for details). Hence, in
this section, we describe SNAP for single-attack AT frameworks employing
$\ell_{\infty}$ perturbations.

###  4.1 SNAPnet

A deep net $f_{\theta}({\bm{x}}):\mathbb{R}^{D}\rightarrow\\{0,1\\}^{C}$
parametrized by $\theta$ maps the input ${\bm{x}}\in\mathbb{R}^{D}$ to a one-
hot vector ${\bm{y}}\in\\{0,1\\}^{C}$ over $C$ classes.

We construct a SNAP-based deep net (SNAPnet)
$f^{\text{SN}}_{\theta,\Sigma}({\bm{x}})$ by introducing an additive shaped
noise (SN) layer (Fig. 4(b)), where the noise distribution parameter $\Sigma$
is learned during training. Formally,

$\displaystyle{\bm{y}}=f^{\text{SN}}_{\theta,\Sigma}({\bm{x}})=f_{\theta}\big{(}{\bm{x}}+{\mathbf{n}}\big{)}=f_{\theta}\big{(}{\bm{x}}+V\Sigma{\mathbf{n}}_{0}\big{)},$
(1)

where ${\mathbf{n}}_{0}\sim\mathcal{L}(0,\mathbf{I}_{D\times D})$ is a zero-
mean isotropic Laplace noise vector,
$\Sigma=\text{Diag}[\sigma_{1},\dots,\sigma_{D}]$ is a distribution parameter
denoting its per-dimension standard deviation, $\mathbf{I}_{D\times D}$
denotes the $D\times D$ identity matrix, and
$V=[{\bm{v}}_{1},\dots,{\bm{v}}_{D}]$ denotes a basis in $\mathbb{R}^{D}$. We
also studied Gaussian and Uniform distributed ${\mathbf{n}}_{0}$, but
empirically find the Laplace distribution to yield better results (Sec.
5.2.1). We use $V=\mathbf{I}_{D\times D}$ for all our experiments in the main
text and study other options for $V$ in the Appendix.

The final classification decision $d$ is computed via

$\displaystyle
d=\arg\max_{c}\bigg{[}\mathbb{E}_{\mathbf{n}}\big{[}{\bm{y}}\big{]}\bigg{]}_{c},$
(2)

where $[{\bm{a}}]_{c}$ denotes the $c$-th element of vector ${\bm{a}}$. Note,
the shaped noise perturbs the input ${\bm{x}}$ with a noise source
${\mathbf{n}}=V\Sigma{\mathbf{n}}_{0}$ (Eq. (1)). The distribution parameter
$\Sigma$ is learned in the presence of any standard AT method [22, 46, 35]
used for learning deep net parameters $\theta$ as described next.

###  4.2 Training SNAPnet

Algorithm 1 summarizes the procedure for training SNAPnet
$f_{\theta,\Sigma}^{\text{SN}}({\bm{x}})$. In each epoch, an arbitrary AT
method $\text{BASE}()$ (line $2$) updates network parameters $\theta$ with
input perturbed by noise ${\mathbf{n}}$. Here $\text{BASE}()$ can be any
established AT framework [22, 46, 35, 40] employing $\ell_{\infty}$
perturbation.

Algorithm 1 Training SNAPnet

Input: training set $X$; basis $V=[{\bm{v}}_{1},\dots,{\bm{v}}_{D}]$; total
noise power $P_{\text{noise}}$; minibatch size $r$; baseline training method
BASE; noise variance update frequency $U_{f}$; Total number of epochs $T$
Initialize: noise variances
$\Sigma_{0}=\text{Diag}[\sigma_{1,0},\dots,\sigma_{D,0}]$.
Output: robust network $f^{\text{SN}}_{\theta,\Sigma}$, noise variances
$\Sigma_{T}=\text{Diag}[\sigma^{2}_{1,T},\dots,\sigma^{2}_{D,T}]$.

1:for epoch $t=1\ldots T$ do

2: for mini-batch $B=\\{{\bm{x}}_{1},\dots,{\bm{x}}_{r}\\}$ do
$\theta\leftarrow\text{BASE}_{\ell_{\infty}}\bigg{(}f^{\text{SN}}_{\theta,\Sigma_{t}}\big{(}\\{{\bm{x}}_{i}\\}_{i=1}^{r}\big{)},\theta\bigg{)}$
$\triangleright$ BASE() Training

3: end for

4: if $t\mod U_{f}=0$ then $\triangleright$ SNAP Distribution Update once
every $U_{f}$ epochs

5: for mini-batch $B=\\{{\bm{x}}_{1},\dots,{\bm{x}}_{r}\\}$ do

6:
$\\{{\bm{x}}^{\text{adv}}_{i}\\}_{i=1}^{r}\leftarrow\text{PGD}^{(K)}_{\ell_{2}}\bigg{(}f^{\text{SN}}_{\theta,\Sigma_{t}}\big{(}\\{{\bm{x}}_{i}\\}_{i=1}^{r}\big{)}\bigg{)}$;
$\bm{\eta}_{i}={\bm{x}}^{\text{adv}}_{i}-{\bm{x}}_{i}\hskip
5.69046pt\forall\hskip 5.69046pti\in\\{1,\dots,r\\}$

7:
$\gamma_{j}\leftarrow\gamma_{j}+\sum_{i=1}^{r}\big{(}\langle{\bm{v}}_{j},\bm{\eta}_{i}\rangle\big{)}^{2}\quad\forall
j\in\\{1,\dots,D\\}$ $\triangleright$ Accumulate projections; See Eq. (3)

8: end for

9:
$\sigma^{2}_{j,t+1}=P_{\text{noise}}\frac{\sqrt{\gamma_{j}}}{\sum_{k=1}^{D}\sqrt{\gamma_{k}}}\quad\forall
j\in\\{1,\dots,D\\}$ $\triangleright$ Normalize accumulated projections; See
Eq. (3)

10: else

11: $\Sigma_{t+1}\leftarrow\Sigma_{t}$

12: end if

13:end for

The SNAP parameter $\Sigma$ is updated once every $U_{f}=10$ epochs via a
_SNAP distribution update_ (lines $4$-$10$). In this update, the per-dimension
noise variance $\sigma^{2}_{j}$ is updated proportional to the root mean
squared projection of the adversarial perturbations $\bm{\eta}$ on the basis
$V$ given a total noise constraint
$\sum_{j=1}^{D}\sigma^{2}_{j}=P_{\text{noise}}$, where $P_{\text{noise}}$
denotes the total noise power. Formally,

$\displaystyle\displaystyle\sigma^{2}_{j}\propto\sqrt{\mathbb{E}_{{\bm{x}}\in
X}\big{(}\langle\bm{\eta},{\bm{v}}_{j}\rangle^{2}\big{)}}\quad\text{s.t.}\quad\sum_{j=1}^{D}\sigma^{2}_{j}=P_{\text{noise}},$
(3)

where $\bm{\eta}$ is the $\ell_{2}$ norm-bounded PGD adversarial perturbation
for the given input ${\bm{x}}\in X$ (line 6). Note that these $\ell_{2}$
perturbations are employed _only_ for noise shaping and are distinct from the
$\ell_{\infty}$ perturbations employed by BASE() AT (line 2). Also,
$\ell_{\infty}$ perturbations cannot be used here since their projections are
constant $\forall j$ when $V=\mathbf{I}_{D\times D}$, whereas employing
$\ell_{1}$ perturbations leads to poor shaping due to high sparsity.

Thus, in SNAP, the average squared $\ell_{2}$ norm of the noise vector
${\mathbf{n}}$ is held constant at $P_{\text{noise}}$ while adapting the noise
variances in the individual dimensions so as to align the noise vectors with
the adversarial perturbations _on average_. Intuitively, the decision boundary
is pushed aggressively in those directions.

###  4.3 Remarks

Note that the SNAP distribution update is distinct from BASE() AT. Hence, SNAP
doesn’t require any hyperparameter tuning in BASE(). For fairness to baselines
we keep all hyperparameters identical when introducing SNAP in all our
experiments. However, SNAP introduces a new hyperparameter $P_{\text{noise}}$,
which permits to trade adversarial robustness $\mathcal{A}^{(U)}_{\text{adv}}$
for natural accuracy $\mathcal{A}_{\text{nat}}$. This trade-off is explored in
Sec. 5.2.2.

The computational overhead of SNAP is small ($\sim 10\%$) since the _SNAP
Distribution Update_ occurs once in 10 epochs using just 20% of the training
data to update the noise standard deviations $\sigma_{j}$. We provide more
details about the _SNAP Distribution Update_ in the Appendix.

##  5 Experimental Results

###  5.1 Setup

Following experimental settings of prior work [46, 35, 23], we employ a
ResNet-18 network for CIFAR-10 experiments and both ResNet-50 and ResNet-101
networks for ImageNet experiments. Accuracy on clean test data is referred to
with $\mathcal{A}_{\text{nat}}$ and accuracy on adversarially perturbed test
data is referred to via $\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$,
$\mathcal{A}^{(\ell_{2})}_{\text{adv}}$, and
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}$, for $\ell_{\infty}$, $\ell_{2}$, and
$\ell_{1}$ norm bounded perturbations, respectively. Accuracy against the
_union_ of all three perturbations is denoted by
$\mathcal{A}^{(U)}_{\text{adv}}$.

For a fair robustness comparison, our evaluation setup closely follows the
setup of Maini et al. [23] for CIFAR-10 data: (1) choose norm bounds
$\epsilon=(0.031,0.5,12.0)$ for ($\ell_{\infty}$, $\ell_{2}$, $\ell_{1}$)
perturbations, respectively; (2) scale norm bounds for images to lie between
$[0,1]$; (3) choose the PGD attack configuration to be _100 iterations with 10
random restarts_ for all perturbation types111Following Maini et al. [23], we
also run all attacks on a subset of the first 1000 test examples with 10
random restarts for CIFAR-10 data.; and (4) estimate
$\mathcal{A}^{(U)}_{\text{adv}}$ as the fraction of test data that is
_simultaneously_ resistant to all three perturbation models.

Following the guidelines of Tramer et al. [38], we carefully design _adaptive_
PGD attacks that target the full defense – SN layer – since SNAPnet is end-to-
end differentiable. Specifically, we backpropagate to primary input ${\bm{x}}$
through the SN layer (see Fig. 4). Thus, the final shaped noise distribution
is exposed to the adversary. We also account for the expectation
$\mathbb{E}_{\mathbf{n}}[\cdot]$ in Eq. (2) by explicitly averaging deep net
logits over $N_{0}(=8)$ noise samples _before_ computing the gradient, which
eliminates any gradient obfuscation, and is known to be the strongest attack
against noise augmented models [33]. In the Appendix A we also show robustness
stress tests and evaluate more attacks.

On CIFAR-10 data, we compare with the following seven key SOTA AT frameworks:
PGD [22], TRADES [46], FreeAdv [35], FastAdv [40], AVG [37], MSD [23], PAT
[19]. We also compare with two randomized smoothing frameworks [5, 33] in the
Appendix B.1. Thanks to their GitHub code releases, we first successfully
reproduce their results with a ResNet-18 network in our environment. In the
case of PAT [19], we evaluate and compare with their pretrained ResNet-50
model on CIFAR-10. We compare all training times on a single NVIDIA P100 GPU.
On ImageNet data, we primarily compare to FreeAdv [35]. We train ResNet-50 and
its SNAPnet version with FreeAdv on a Google Cloud server with four NVIDIA
P100 GPUs to compare their accuracy and training times. We provide all
hyperparameters in Appendix C. Our code and pretrained models are available at
https://github.com/adpatil2/SNAP.

###  5.2 Ablation Studies

Method |  $\mathcal{A}_{\text{nat}}$ |  $\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$ $\epsilon=0.03$ |  $\mathcal{A}^{(\ell_{2})}_{\text{adv}}$ $\epsilon=0.5$ |  $\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ $\epsilon=12$ |  $\mathcal{A}^{(U)}_{\text{adv}}$
---|---|---|---|---|---
PGD AT with $\ell_{\infty}$ perturbations
PGD |  84.6 |  48.8 |  62.3 |  15.0 |  15.0
+SNAP[G] |  80.7 |  45.7 |  66.9 |  34.6 |  31.9
+SNAP[U] |  85.1 |  42.7 |  66.7 |  28.6 |  26.6
+SNAP[L] |  83.0 |  44.8 |  68.6 |  40.1 |  35.6
PGD AT with $\ell_{2}$ perturbations
PGD |  89.3 |  28.8 |  67.3 |  31.8 |  25.1
+SNAP[G] |  83.0 |  35.0 |  65.8 |  39.9 |  30.2
+SNAP[U] |  86.4 |  32.3 |  66.7 |  30.2 |  25.0
+SNAP[L] |  84.8 |  33.4 |  66.1 |  42.5 |  30.8

Table 1: ResNet-18 CIFAR-10 results showing the impact of SNAP augmentation of
PGD [22] AT framework with $\ell_{\infty}$ (_top_) and $\ell_{2}$ (_bottom_)
perturbations where [G], [U], and [L], denote shaped Gaussian, Uniform, and
Laplace noise.

Method |  $\mathcal{A}_{\text{nat}}$ |  $\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$ $\epsilon=0.03$ |  $\mathcal{A}^{(\ell_{2})}_{\text{adv}}$ $\epsilon=0.5$ |  $\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ $\epsilon=12$ |  $\mathcal{A}^{(U)}_{\text{adv}}$
---|---|---|---|---|---
High Complexity AT with $\ell_{\infty}$ perturbations
PGD |  84.6 |  48.8 |  62.3 |  15.0 |  15.0
+SNAP |  83.0 |  44.8 |  68.6 |  40.1 |  35.6
TRADES |  82.1 |  50.2 |  59.6 |  19.8 |  19.7
+SNAP |  80.9 |  45.2 |  66.9 |  46.6 |  41.2
Low Complexity AT with $\ell_{\infty}$ perturbations
FreeAdv |  81.7 |  46.1 |  59 |  15.0 |  15.0
+SNAP |  83.5 |  39.7 |  66.2 |  34.3 |  29.6
FastAdv |  85.7 |  46.2 |  60.0 |  13.2 |  13.2
+SNAP |  84.2 |  40.4 |  67.9 |  36.6 |  30.8

Table 2: ResNet-18 CIFAR-10 results showing the impact of SNAP augmentation of
established $\ell_{\infty}$-AT frameworks. The computational overhead of SNAP
is limited to $\sim 10\%$.

####  5.2.1 Impact of Noise Distribution and Model of BASE() AT Perturbations

In this subsection, we first study the impact of employing $\ell_{\infty}$
_vs_. $\ell_{2}$ perturbations in BASE AT() (see line 2 in Alg. 1) on
$\mathcal{A}^{(U)}_{\text{adv}}$. For each choice, we further experiment with
three distributions for the SN layer in Fig. 4(b) viz. Gaussian, Uniform, and
Laplace. We don’t consider $\ell_{1}$ perturbations in BASE AT() since Maini
et al. [23] showed that employing $\ell_{1}$ single-attack AT achieves very
low robustness to all attacks. We choose PGD [22] AT as BASE AT() for this
ablation study. For a fair comparison across the noise distributions, we fix
$P_{\text{noise}}=160$, enforcing all noise vectors to have the same average
$\ell_{2}$ norm. For each distribution, the noise is shaped per the procedure
summarized in Alg. 1.

As observed in Table 1, $\ell_{\infty}$-PGD AT achieves much lower
$\mathcal{A}^{(U)}_{\text{adv}}$ than $\ell_{2}$-PGD AT, an observation also
reported by Maini et al. [23]. With SNAP, however, we find that there is an
interaction between the perturbation model in PGD AT and the noise
distribution in SNAP. For instance, SNAP[U] enhances
$\mathcal{A}^{(U)}_{\text{adv}}$ by 11% with $\ell_{\infty}$-PGD AT while not
achieving any improvement with $\ell_{2}$-PGD AT. In fact, SNAP appears to be
particularly suitable for $\ell_{\infty}$-AT, since it always improves
$\mathcal{A}^{(U)}_{\text{adv}}$ by 11%-to-20.6% irrespective of the noise
distribution.

Finally, of the three noise distributions, we find the Laplace distribution to
be distinctly superior, achieving the highest $\mathcal{A}^{(U)}_{\text{adv}}$
(35.6% and 30.8%) due to a significant improvement in
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ for both $\ell_{\infty}$ and
$\ell_{2}$ PGD AT, respectively. The superiority of the Laplace distribution
in achieving high $\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ stems from its
heavier tail compared to the Gaussian and Uniform distributions with the same
variance. Shaped Laplace noise generates the highest fraction of extreme
values in a given noise sample. Hence, it is more effective in improving
accuracy against $\ell_{1}$-bounded attacks, which are the strongest when
perturbing few pixels by a large magnitude [23, 37]. We discuss this further
in the Appendix B.7. Henceforth, unless otherwise mentioned, we choose Laplace
noise for SNAP and $\ell_{\infty}$ perturbations for BASE() AT as the default
setting since it achieves the highest $\mathcal{A}^{(U)}_{\text{adv}}$.

####  5.2.2 Impact of $P_{\text{noise}}$

Figure 5: ResNet-18 CIFAR-10 results: adversarial accuracy
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}$,
$\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$, and natural accuracy
$\mathcal{A}_{\text{nat}}$ _vs_. total noise power $P_{\text{noise}}$ for
PGD+SNAP.

Next, we explore the impact of the SNAP hyperparameter $P_{\text{noise}}$,
which constrains the average squared $\ell_{2}$ norm of the noise vector
${\mathbf{n}}$. It enables to trade between adversarial and natural accuracy.

Fig. 5 shows that, as $P_{\text{noise}}$ increases,
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ improves from 31% to 47%, accompanied
by a graceful ($5\%$) drop in $\mathcal{A}_{\text{nat}}$ and a small drop of
2% in $\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$ that stabilizes to $\approx
45\%$. These results show: (1) SNAP preserves the impact of $\ell_{\infty}$
perturbations which is not surprising since PGD AT [22] explicitly includes
those, and (2) $P_{\text{noise}}$ provides an explicit knob to control the
$\mathcal{A}_{\text{nat}}$ _vs_. $\mathcal{A}_{\text{adv}}$ trade-off.
Henceforth, we choose $P_{\text{noise}}$ values that incur $<1.5\%$ drop in
$\mathcal{A}_{\text{nat}}$ for all SNAP+AT experiments.

####  5.2.3 SNAP augmented SOTA AT Frameworks

Table 2 shows the effectiveness of SNAP for four SOTA AT frameworks: high
complexity frameworks, such as PGD [22], TRADES [46], and low complexity
frameworks such as FreeAdv [35], FastAdv [40]. All are trained against
$\ell_{\infty}$ attacks with $\epsilon=0.031$. As expected, while they achieve
high $\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$, their
$\mathcal{A}^{(\ell_{2})}_{\text{adv}}$ and
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ are lower.

For high-complexity AT, SNAP enhances $\mathcal{A}^{(\ell_{2})}_{\text{adv}}$
and $\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ by $\sim 6\%$ and $\sim 25\%$,
respectively, while incurring only a drop of $\sim 5\%$ in
$\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$. Thus overall, SNAP improves
robustness ($\mathcal{A}^{(U)}_{\text{adv}}$) by $\sim 20\%$ against the
_union_ of the three perturbation models. Note that this robustness
improvement comes at only a $\sim 1\%$ drop in $\mathcal{A}_{\text{nat}}$ (see
Table 2). For low-complexity ATs, SNAP improvements in union robustness
($\mathcal{A}^{(U)}_{\text{adv}}$) are also significant ($\sim 15\%$). Again,
presence of SNAP improves $\mathcal{A}^{(\ell_{2})}_{\text{adv}}$ and
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}$. This time the drop in
$\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$ is $\sim 7\%$. We believe this is
due to the fact that these frameworks employ weaker single-step attacks during
training. Note that in the case of FreeAdv+SNAP, we actually observe a $\sim
2\%$ _increase_ in $\mathcal{A}_{\text{nat}}$, a trend we also observe in the
ImageNet experiments described later.

Method |  LR schedule |  Epochs |  $\mathcal{A}_{\text{nat}}$ |  $\mathcal{A}^{(U)}_{\text{adv}}$ |  Total time (minutes)
---|---|---|---|---|---
Set A: Total Time $\geq$ 12 Hrs
AVG 50 Step [37] |  cyclic |  50 |  84.8 |  40.4 |  4217
AVG 20 Step [37] |  cyclic |  50 |  85.6 |  40.4 |  1834
AVG 10 Step [37] |  cyclic |  50 |  86.7 |  38.9 |  956
PAT [19] |  step |  100 |  82.4 |  36.6 |  1364
MSD 50 Step [23] |  cyclic |  50 |  81.7 |  47.0 |  1693
MSD 30 Step [23] |  cyclic |  50 |  82.4 |  44.9 |  978
Set B: 8 Hrs $<$ Total Time $<$ 12 Hrs
AVG 5 Step [37] |  cyclic |  50 |  87.8 |  33.7 |  489
MSD 20 Step [23] |  cyclic |  50 |  83.0 |  37.3 |  690
TRADES [46] |  step |  100 |  82.0 |  19.7 |  516
TRADES+SNAP |  step |  100 |  80.9 |  41.2 |  566
Set C: 5 Hrs $<$ Total Time $<$ 8 Hrs
MSD 10 Step [23] |  cyclic |  50 |  83.6 |  33.3 |  342
PGD [22] |  step |  100 |  84.6 |  15.0 |  354
PGD+SNAP |  step |  100 |  83.0 |  35.6 |  403
Set D: 2 Hrs $<$ Total Time $<$ 5 Hrs
AVG 2 Step [37] |  cyclic |  50 |  88.4 |  22.0 |  232
MSD 5 Step [23] |  cyclic |  50 |  84.0 |  12.6 |  185
PGD [22] |  cyclic |  50 |  82.8 |  15.7 |  177
TRADES [46] |  cyclic |  50 |  80.0 |  21.4 |  258
PGD+SNAP |  cyclic |  50 |  82.3 |  33.5 |  199
TRADES+SNAP |  cyclic |  50 |  78.8 |  40.8 |  280
Set E: Total Time $<$ 2 Hrs
FreeAdv [35] |  step |  200 |  81.7 |  15.0 |  66
FastAdv [40] |  cyclic |  50 |  85.7 |  13.2 |  47
FreeAdv+SNAP |  step |  200 |  83.5 |  29.6 |  88
FastAdv+SNAP |  cyclic |  50 |  84.2 |  30.8 |  69

Table 3: CIFAR-10 results for comparing adversarial accuracy
$\mathcal{A}^{(U)}_{\text{adv}}$ _vs_. training time (on single NVIDIA P100
GPU) for different AT frameworks and the improvements by introducing proposed
SNAP technique. All frameworks except PAT [19] (which employs ResNet-50)
employ ResNet-18.

###  5.3 Robustness _vs_. Training Complexity

Next we quantify adversarial robustness _vs_. training time trade-offs. Table
3 shows that SNAP augmentation of single-attack AT frameworks achieves the
highest $\mathcal{A}^{(U)}_{\text{adv}}$, when training time is constrained to
12 hours (sets B, C, D, and E).

For instance, TRADES+SNAP achieves a 4% higher
$\mathcal{A}^{(U)}_{\text{adv}}(=41\%)$ than MSD-$20$ with 2 hours _lower_
training time (Set B in Table 3). Similarly, PGD+SNAP achieves a 2% higher
$\mathcal{A}^{(U)}_{\text{adv}}$ than MSD-$10$ while having a similar training
time (Set C). Note that both PGD and TRADES here use 100 training epochs with
standard step learning rate (LR) schedule, while MSD frameworks employ a
cyclic learning rate schedule to achieve superconvergence in 50 epochs.

Training |  $\mathcal{A}_{\text{nat}}$ (%) |  $\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$ $\epsilon=2/255$ |  $\mathcal{A}^{(\ell_{2})}_{\text{adv}}$ $\epsilon=2.0$ |  $\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ $\epsilon=72.0$ |  $\mathcal{A}^{(U)}_{\text{adv}}$ |  Total time (minutes)
---|---|---|---|---|---|---
ResNet-50
FreeAdv [35] |  61.7 |  47.8 |  19.9 |  14.8 |  12.6 |  3590
FreeAdv+SNAP |  66.8 |  46.1 |  37.8 |  37.4 |  32.4 |  3756
ResNet-101
FreeAdv [35] |  65.4 |  51.8 |  22.8 |  18.8 |  16.1 |  5678
FreeAdv+SNAP |  69.7 |  50.3 |  41.1 |  40.2 |  35.4 |  5904

Table 4: ImageNet results: Iso-hyperparameter introduction of SNAP yields
$\sim 20\%$ improvement in adversarial accuracy
($\mathcal{A}^{(U)}_{\text{adv}}$) with modest impact on training time for
ResNet-50 and ResNet-101.

In Set D, _following_ Maini et al. [23], we employ a cyclic learning rate
schedule for PGD, TRADES, as well as for PGD+SNAP and TRADES+SNAP to achieve
convergence in 50 epochs. Improvements in $\mathcal{A}^{(U)}_{\text{adv}}$ for
PGD+SNAP and TRADES+SNAP are similar to those in Sets B and C. Most notably,
PGD+SNAP with cyclic learning rate achieves $\sim 20\%$ and 11.5% _higher_
$\mathcal{A}^{(U)}_{\text{adv}}$ than MSD-5 and AVG-2, respectively, while
having a similar training time ($\sim 3$ hours). Set E augments the data from
Table 2 with training times. FastAdv+SNAP and FreeAdv+SNAP achieve a high
$\mathcal{A}^{(U)}_{\text{adv}}\sim 30\%$, while preserving the training
efficiency of both FastAdv and FreeAdv. Notably, FastAdv+SNAP achieves $18\%$
higher $\mathcal{A}^{(U)}_{\text{adv}}$ than MSD-5, while being $\sim
2.7\times$ more efficient to train.

###  5.4 ImageNet Results

Thanks to SNAP’s low computational overhead combined with FreeAdv’s fast
training time, we are for the first time able to report adversarial accuracy
of ResNet-50 and ResNet-101 against the union of
$(\ell_{\infty},\ell_{2},\ell_{1})$ attacks on ImageNet.

We closely follow the evaluation setup of Shafahi et al. [35]. Specifically,
we use 100 step PGD attack, one of the strongest adversaries considered by
Shafahi et al. [35], and evaluate on the entire test set. We first reproduce
FreeAdv [35] results using the _same_ hyperparameters and then introduce SNAP.
All hyperparameter details are specified in the Appendix.

In order to clearly demonstrate the contrast between robustness to different
perturbation models, we evaluate with $\epsilon=(2/255,2.0,72.0)$ for
$(\ell_{\infty},\ell_{2},\ell_{1})$ attacks, respectively.222Note that
$\ell_{2}$ and $\ell_{1}$ norms of PGD perturbation with $\ell_{\infty}$ norm
of $2/255$ can be as large as $\sim 3.0$ and $\sim 1100$ for images of size
$224\times 224\times 3$. As shown in Table 4, FreeAdv achieves a high
$\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}=47.8\%$ with ResNet-50, but a
lower $\mathcal{A}^{(\ell_{2})}_{\text{adv}}=20\%$ and
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}=15\%$, and consequently, a low
$\mathcal{A}^{(U)}_{\text{adv}}$ of 12.6% against the union of the
perturbations. In contrast, FreeAdv+SNAP improves
$\mathcal{A}^{(\ell_{2})}_{\text{adv}}$ and
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ by $17\%$ and $22\%$, respectively,
accompanied by a 5% improvement in $\mathcal{A}_{\text{nat}}$ and a small
$2\%$ loss in $\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$. This results in an
overall robustness improvement of $20\%$ against the union of the perturbation
models, setting a first benchmark for ResNet-50 on ImageNet. Upon increasing
the network to ResNet-101, both natural and adversarial accuracies improve by
$\approx 4\%$ for FreeAdv, a trend also observed by Shafahi et al. [35]. SNAP
further improves FreeAdv’s results for $\mathcal{A}_{\text{nat}}$ and
$\mathcal{A}^{(U)}_{\text{adv}}$ by 4.3% and 19.3%.

##  6 Discussion

Given the wide popularity of $\ell_{\infty}$-AT, in this paper, we propose
SNAP as an augmentation that generalizes the effectiveness of
$\ell_{\infty}$-AT to the union of $(\ell_{\infty},\ell_{2},\ell_{1})$
perturbations. SNAP’s strength is its simplicity and efficiency. Consequently,
this work sets a first benchmark for ResNet-50 and ResNet-101 networks which
are resilient to the union of $(\ell_{\infty},\ell_{2},\ell_{1})$
perturbations on ImageNet. Note that norm-bounded perturbations include a
large class of attacks, _e.g_., gradient-based [22, 31, 37, 23, 4, 24],
decision-based [3] and black-box [1] attacks.

More work is needed to extend the proposed SNAP technique to attacks beyond
norm-bounded additive perturbations, _e.g_., functional [18, 41], rotation
[7], texture [2], etc. We provide preliminary evaluations in this direction in
the Appendix. It is important to note that SNAP is meant to be an efficient
technique for improving $\ell_{\infty}$-AT, and _not_ a new defense. Indeed
defending against a large variety of attacks simultaneously remains an open
problem, with encouraging results from recent efforts [23, 19].

Another limitation of our approach is that its benefits are demonstrated
empirically. It is an inevitable consequence of a lack of any theoretical
guarantees for underlying AT frameworks. An interesting direction of future
work is to explore whether any theoretical guarantees can be derived for
anisotropic shaped noise distributions in SNAP by building upon the recent
developments in randomized smoothing [33, 43]. This could be a potential
avenue for bridging the gap between certification bounds and empirical
adversarial accuracy.

Finally, we believe that any effort on improving adversarial robustness of
deep nets has net positive societal impact. However, recent past in this field
has shown that any improvements in defense techniques also lead to more
effective threat models. While such a cat-and-mouse game is of great
intellectual value in the academic setting, it does have an unintentional
negative societal consequence of equipping malicious outside actors with a
broad set of tools. This further underscores the well-recognized need for
provable defenses.

## Acknowledgments and Disclosure of Funding

This work was supported by the Semiconductor Research Corporation (SRC) and
DARPA sponsored Center for Brain-inspired Computing (C-BRIC) and SRC’s AIHW
program.

## References

  * Andriushchenko et al. [2020] Andriushchenko, M., Croce, F., Flammarion, N., and Hein, M.  Square attack: a query-efficient black-box adversarial attack via random search.  In _European Conference on Computer Vision_ , pp. 484–501. Springer, 2020.
  * Bhattad et al. [2019] Bhattad, A., Chong, M. J., Liang, K., Li, B., and Forsyth, D. A.  Unrestricted adversarial examples via semantic manipulation.  _arXiv preprint arXiv:1904.06347_ , 2019.
  * Brendel et al. [2018] Brendel, W., Rauber, J., and Bethge, M.  Decision-based adversarial attacks: Reliable attacks against black-box machine learning models.  In _International Conference on Learning Representations_ , 2018.
  * Chen et al. [2018] Chen, P.-Y., Sharma, Y., Zhang, H., Yi, J., and Hsieh, C.-J.  Ead: elastic-net attacks to deep neural networks via adversarial examples.  In _Thirty-second AAAI conference on artificial intelligence_ , 2018\.
  * Cohen et al. [2019] Cohen, J., Rosenfeld, E., and Kolter, Z.  Certified adversarial robustness via randomized smoothing.  In _International Conference on Machine Learning (ICML)_ , 2019.
  * Dezfooli et al. [2018] Dezfooli, S. M. M., Fawzi, A., Fawzi, O., Frossard, P., and Soatto, S.  Robustness of classifiers to universal pertur-bations: A geometric perspective.  In _International Conference on Learning Representations (ICLR)_ , 2018.
  * Engstrom et al. [2019] Engstrom, L., Tran, B., Tsipras, D., Schmidt, L., and Madry, A.  Exploring the landscape of spatial robustness.  In _International Conference on Machine Learning_ , pp. 1802–1811. PMLR, 2019.
  * Gilmer et al. [2019] Gilmer, J., Ford, N., Carlini, N., and Cubuk, E.  Adversarial examples are a natural consequence of test error in noise.  In _International Conference on Machine Learning_ , pp. 2280–2289, 2019.
  * Gowal et al. [2020] Gowal, S., Qin, C., Uesato, J., Mann, T., and Kohli, P.  Uncovering the limits of adversarial training against norm-bounded adversarial examples.  _arXiv preprint arXiv:2010.03593_ , 2020.
  * Gui et al. [2019] Gui, S., Wang, H., Yu, C., Yang, H., Wang, Z., and Liu, J.  Model compression with adversarial robustness: A unified optimization framework.  _arXiv preprint arXiv:1902.03538_ , 2019.
  * Guo et al. [2020] Guo, M., Yang, Y., Xu, R., Liu, Z., and Lin, D.  When nas meets robustness: In search of robust architectures against adversarial attacks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 631–640, 2020.
  * He et al. [2019] He, Z., Rakin, A. S., and Fan, D.  Parametric noise injection: Trainable randomness to improve deep neural network robustness against adversarial attack.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Hendrycks & Dietterich [2018] Hendrycks, D. and Dietterich, T.  Benchmarking neural network robustness to common corruptions and perturbations.  In _International Conference on Learning Representations_ , 2018.
  * Hu et al. [2020] Hu, T.-K., Chen, T., Wang, H., and Wang, Z.  Triple wins: Boosting accuracy, robustness and efficiency together by enabling input-adaptive inference.  _arXiv preprint arXiv:2002.10025_ , 2020.
  * Ilyas et al. [2019] Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., and Madry, A.  Adversarial examples are not bugs, they are features.  _arXiv preprint arXiv:1905.02175_ , 2019.
  * Jordan et al. [2019] Jordan, M., Manoj, N., Goel, S., and Dimakis, A. G.  Quantifying perceptual distortion of adversarial examples.  _arXiv preprint arXiv:1902.08265_ , 2019.
  * Kang et al. [2019] Kang, D., Sun, Y., Brown, T., Hendrycks, D., and Steinhardt, J.  Transfer of adversarial robustness between perturbation types.  _arXiv preprint arXiv:1905.01034_ , 2019.
  * Laidlaw & Feizi [2019] Laidlaw, C. and Feizi, S.  Functional adversarial attacks.  _Advances in Neural Information Processing Systems_ , 2019.
  * Laidlaw et al. [2018] Laidlaw, C., Singla, S., and Feizi, S.  Perceptual adversarial robustness: Defense against unseen threat models.  _International Conference on Learning Representations (ICLR)_ , 2018\.
  * Li et al. [2019] Li, B., Chen, C., Wang, W., and Duke, L. C.  Certified adversarial robustness with addition gaussian noise.  _Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Madaan et al. [2020] Madaan, D., Shin, J., and Hwang, S. J.  Learning to generate noise for robustness against multiple perturbations.  _arXiv preprint arXiv:2006.12135_ , 2020.
  * Madry et al. [2018] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A.  Towards deep learning models resistant to adversarial attacks.  _International Conference on Learning Representations (ICLR)_ , 2018\.
  * Maini et al. [2020] Maini, P., Wong, E., and Kolter, J. Z.  Adversarial robustness against the union of multiple perturbation models.  In _International Conference on Machine Learning (ICML)_ , 2020.
  * Moosavi-Dezfooli et al. [2016] Moosavi-Dezfooli, S.-M., Fawzi, A., and Frossard, P.  Deepfool: a simple and accurate method to fool deep neural networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR)_ , 2016.
  * Moosavi-Dezfooli et al. [2019] Moosavi-Dezfooli, S.-M., Fawzi, A., Uesato, J., and Frossard, P.  Robustness via curvature regularization, and vice versa.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019.
  * Pinot et al. [2019] Pinot, R., Meunier, L., Araujo, A., Kashima, H., Yger, F., Gouy-Pailler, C., and Atif, J.  Theoretical evidence for adversarial robustness through randomization: the case of the exponential family.  In _Advances in Neural Information Processing Systems_ , 2019.
  * Pinot et al. [2020] Pinot, R., Ettedgui, R., Rizk, G., Chevaleyre, Y., and Atif, J.  Randomization matters. how to defend against strong adversarial attacks.  In _International Conference on Machine Learning (ICML)_ , 2020.
  * Rauber et al. [2020] Rauber, J., Zimmermann, R., Bethge, M., and Brendel, W.  Foolbox native: Fast adversarial attacks to benchmark the robustness of machine learning models in pytorch, tensorflow, and jax.  _Journal of Open Source Software_ , 5(53):2607, 2020.  doi: 10.21105/joss.02607.  URL https://doi.org/10.21105/joss.02607.
  * Rebuffi et al. [2021] Rebuffi, S.-A., Gowal, S., Calian, D. A., Stimberg, F., Wiles, O., and Mann, T.  Fixing data augmentation to improve adversarial robustness.  _arXiv preprint arXiv:2103.01946_ , 2021.
  * Rice et al. [2020] Rice, L., Wong, E., and Kolter, Z.  Overfitting in adversarially robust deep learning.  In _International Conference on Machine Learning_ , pp. 8093–8104. PMLR, 2020.
  * Rony et al. [2019] Rony, J., Hafemann, L. G., Oliveira, L. S., Ayed, I. B., Sabourin, R., and Granger, E.  Decoupling direction and norm for efficient gradient-based l2 adversarial attacks and defenses.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 4322–4330, 2019.
  * Rusak et al. [2020] Rusak, E., Schott, L., Zimmermann, R. S., Bitterwolf, J., Bringmann, O., Bethge, M., and Brendel, W.  A simple way to make neural networks robust against diverse image corruptions.  In _European Conference on Computer Vision_ , pp. 53–69. Springer, 2020.
  * Salman et al. [2019] Salman, H., Li, J., Razenshteyn, I., Zhang, P., Zhang, H., Bubeck, S., and Yang, G.  Provably robust deep learning via adversarially trained smoothed classifiers.  In _Advances in Neural Information Processing Systems_ , pp. 11289–11300, 2019.
  * Santurkar et al. [2019] Santurkar, S., Ilyas, A., Tsipras, D., Engstrom, L., Tran, B., and Madry, A.  Image synthesis with a single (robust) classifier.  In _Advances in Neural Information Processing Systems_ , pp. 1262–1273, 2019.
  * Shafahi et al. [2019] Shafahi, A., Najibi, M., Ghiasi, A., Xu, Z., Dickerson, J., Studer, C., Davis, L. S., Taylor, G., and Goldstein, T.  Adversarial training for free!  _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019\.
  * Stutz et al. [2020] Stutz, D., Hein, M., and Schiele, B.  Confidence-calibrated adversarial training: Generalizing to unseen attacks.  In _International Conference on Machine Learning_ , pp. 9155–9166. PMLR, 2020.
  * Tramèr & Boneh [2019] Tramèr, F. and Boneh, D.  Adversarial training and robustness for multiple perturbations.  In _Advances in Neural Information Processing Systems_ , pp. 5858–5868, 2019.
  * Tramer et al. [2020] Tramer, F., Carlini, N., Brendel, W., and Madry, A.  On adaptive attacks to adversarial example defenses.  _arXiv preprint arXiv:2002.08347_ , 2020.
  * Vivek & Babu [2020] Vivek, B. and Babu, R. V.  Single-step adversarial training with dropout scheduling.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 947–956. IEEE, 2020.
  * Wong et al. [2020] Wong, E., Rice, L., and Kolter, J. Z.  Fast is better than free: Revisiting adversarial training.  In _International Conference on Machine Learning (ICLR)_ , 2020.
  * Xiao et al. [2018] Xiao, C., Zhu, J.-Y., Li, B., He, W., Liu, M., and Song, D.  Spatially transformed adversarial examples.  In _International Conference on Learning Representations_ , 2018.
  * Xie & Yuille [2020] Xie, C. and Yuille, A.  Intriguing properties of adversarial training at scale.  In _International Conference on Learning Representations_ , 2020.
  * Yang et al. [2020a] Yang, G., Duan, T., Hu, E., Salman, H., Razenshteyn, I., and Li, J.  Randomized smoothing of all shapes and sizes.  _International Conference on Machine Learning (ICML)_ , 2020a.
  * Yang et al. [2020b] Yang, Y.-Y., Rashtchian, C., Zhang, H., Salakhutdinov, R., and Chaudhuri, K.  A closer look at accuracy vs. robustness.  _Advances in Neural Information Processing Systems_ , 33, 2020b.
  * Zhang et al. [2019a] Zhang, D., Zhang, T., Lu, Y., Zhu, Z., and Dong, B.  You only propagate once: Accelerating adversarial training via maximal principle.  _arXiv preprint arXiv:1905.00877_ , 2019a.
  * Zhang et al. [2019b] Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and Jordan, M.  Theoretically principled trade-off between robustness and accuracy.  In _International Conference on Machine Learning (ICML)_ , 2019b.
  * Zhang et al. [2020] Zhang, J., Xu, X., Han, B., Niu, G., Cui, L., Sugiyama, M., and Kankanhalli, M.  Attacks which do not kill training make adversarial learning stronger.  In _International Conference on Machine Learning_ , pp. 11278–11287. PMLR, 2020.
  * Zheng et al. [2020] Zheng, H., Zhang, Z., Gu, J., Lee, H., and Prakash, A.  Efficient adversarial training with transferable adversarial examples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 1181–1190, 2020.

##  Appendix A Robustness Stress Tests

We conduct robustness stress tests to confirm that the benefits of SNAP are
sustained for a range of attack norm-bounds, larger number of attack steps,
and even for “gradient-free” attacks. For these experiments, we consider
networks trained using TRADES and TRADES+SNAP (rows in Table 2 of the main
paper), since they achieve the highest $\mathcal{A}^{(U)}_{\text{adv}}$ among
the four SOTA AT frameworks.

###  A.1 Sweeping norm-bounds and number of attack steps

We sweep the number of PGD attack steps ($K$) and norm-bounds ($\epsilon$) for
all three perturbations $(\ell_{\infty},\ell_{2},\ell_{1})$ to confirm that
the robustness gains from SNAP are achieved for a wider range of attack norm
bounds, and are sustained even after increasing attack steps.

Figure 6: ResNet-18 CIFAR-10 results: Adversarial accuracy _vs_. norm bound
$\epsilon$ for: (a) $\ell_{\infty}$, (b) $\ell_{2}$, (c) $\ell_{1}$ PGD-100
attack. Adversarial accuracy _vs_. attack steps $K$ for (d) $\ell_{\infty}$
($\epsilon=0.031$), (e) $\ell_{2}$ ($\epsilon=0.5$), (f) $\ell_{1}$
($\epsilon=12$) PGD-100 attacks.

Fig. 6(a)-(c) validates the main text Table 2 conclusion that TRADES+SNAP
achieves large gains ($\sim 20\%$) in $\mathcal{A}^{(\ell_{1})}_{\text{adv}}$
and $\mathcal{A}^{(\ell_{2})}_{\text{adv}}$ with a small ($\sim 4\%$) drop in
$\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$. Furthermore, this conclusion
holds for a large range of $\epsilon$ values for all three perturbations.
Additionally, the gain in $\mathcal{A}^{(\ell_{2})}_{\text{adv}}$ due to SNAP
at $\epsilon=1.2$ is greater than the one reported in Table 2 for
$\epsilon=0.5$.

Now we increase the attack steps $K$ to 500 and observe the impact on
adversarial accuracy against $(\ell_{\infty},\ell_{2},\ell_{1})$ perturbations
in Fig. 6(d,e,f), respectively. In all cases, we observe hardly any change of
the adversarial accuracy beyond $K=100$. Hence, as noted in the main text, we
have chosen $K=100$ for all our experiments in the main text and in this
supplementary.

Recall we employ 10 random restarts as recommended by Maini et al. [23] for
_all_ our adversarial accuracy evaluations on CIFAR-10 data.

###  A.2 Evaluating robustness against new attacks

We evaluate adversarial accuracy against the recent DDN [31], Boundary [3],
and Square [1] attacks. The DDN attack was shown to be one of the SOTA
gradient-based attacks, while boundary attack is one of the strongest
“gradient-free” attacks. Of all the attacks considered in Maini et al. [23],
PGD turns out to be the strongest for $\ell_{\infty}$ and $\ell_{1}$
perturbations. Hence, in this section, we evaluate against $\ell_{2}$ norm-
bounded DDN, boundary, and Square attacks.

Following Maini et al. [23], we use the FoolBox [28] implementation of the
boundary attack, which uses 25 trials per iteration. For the DDN attack, we
use 100 attack steps with appropriate logit averaging for $N_{0}=8$ noise
samples _before_ computing the gradient in each step (similar to our PGD
attack implementations). As mentioned in the main text, it eliminates any
gradient obfuscation due to the presence of noise.

|  TRADES |  TRADES+SNAP
---|---|---
Natural Accuracy |  82.1 |  80.9
DDN [31] ($\epsilon=0.5$) |  59.7 |  65.8
Boundary [3] ($\epsilon=0.5$) |  63.5 |  67.0
Square [1] ($\epsilon=0.5$) |  68.2 |  72.7

Table 5: ResNet-18 CIFAR-10 results showing natural accuracy (%) and
adversarial accuracy (%) against $\ell_{2}$ norm bounded DDN attack [31],
boundary attack [3], and Square [1] for TRADES and TRADES+SNAP networks from
Table 2 in the main text.

Table 5 shows that SNAP improves adversarial accuracy against the DDN attack
by $\sim 6\%$. This is similar to improvements seen against $\ell_{2}$-PGD
attack in Table 2 in the main text. Similarly, TRADES+SNAP achieves $3.5\%$
($4.5\%$) higher adversarial accuracy than TRADES against the Boundary [3]
(Square [1]) attack.

##  Appendix B Additional Results

###  B.1 Comparison with Randomized Smoothing (RS)

In this subsection, we compare with two SOTA randomized smoothing (RS) works,
namely, RandSmooth [5], and SmoothAdv [33]. They employ isotropic Gaussian
noise. In Fig. 7(a), we find that PGD+SNAP achieves a better
$\mathcal{A}_{\text{nat}}$ _vs_. $\mathcal{A}^{(U)}_{\text{adv}}$ trade-off
compared to both RandSmooth [5], and SmoothAdv [33]. Specifically, note that
SmoothAdv [33] can also be viewed as isotropic Gaussian augmentation of
$\ell_{2}$-PGD AT. Importantly, PGD+SNAP achieves a 12% higher
$\mathcal{A}^{(U)}_{\text{adv}}$ for the same $\mathcal{A}_{\text{nat}}$. This
demonstrates the efficacy of _shaped noise_ in SNAP, which enhances the
robustness to the union of $(\ell_{\infty},\ell_{2},\ell_{1})$ perturbations.

Figure 7: ResNet-18 CIFAR-10 results: (a) $\mathcal{A}_{\text{nat}}$ _vs_.
$\mathcal{A}^{(U)}_{\text{adv}}$ for RandSmooth [5], SmoothAdv [33], and
PGD+SNAP; (b) $\mathcal{A}_{\text{nat}}$ _vs_.
$\mathcal{A}^{(U)}_{\text{adv}}$ for PGD+SNAP and PGD+Iso[L], where Iso[L]
denotes a baseline SNAP alternative employing isotropic Laplace noise
augmentation, _i.e_., without noise shaping. PGD+SNAP achieves better
$\mathcal{A}_{\text{nat}}$ _vs_. $\mathcal{A}^{(U)}_{\text{adv}}$ trade-off
due to noise shaping.

In order to further quantify importance of _noise shaping_ , we also compare
$\ell_{\infty}$-PGD+SNAP with $\ell_{\infty}$-PGD+Iso[L], a stronger baseline
alternative consisting of _isotropic_ Laplace noise augmentation, _i.e_.,
_without any noise shaping_. Specifically, in Iso[L], the noise standard
deviation is _identical_ in each direction, _i.e_.,
$\Sigma=\text{Diag}\bigg{[}\sqrt{\frac{P_{\text{noise}}}{D}},\dots,\sqrt{\frac{P_{\text{noise}}}{D}}\bigg{]}$.
Note that such distributions have recently been explored for RS [43].

Fig. 7(b) plots the $\mathcal{A}_{\text{nat}}$ _vs_.
$\mathcal{A}^{(U)}_{\text{adv}}$ trade-off for PGD+SNAP (_red curve_) and
PGD+Iso[L] (_black curve_) by sweeping $P_{\text{noise}}$. We find that
PGD+SNAP achieves a better $\mathcal{A}_{\text{nat}}$ _vs_.
$\mathcal{A}^{(U)}_{\text{adv}}$ trade-off compared to PGD+Iso[L] by making
more efficient use of noise power via noise shaping. Specifically, for
$\mathcal{A}^{(U)}_{\text{adv}}\approx 38$, PGD+SNAP achieves a $\sim 4\%$
higher $\mathcal{A}_{\text{nat}}$.

###  B.2 Comparison with Madaan et al. [21]

Method |  $\mathcal{A}_{\text{nat}}$ |  $\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$ $\epsilon=0.03$ |  $\mathcal{A}^{(\ell_{2})}_{\text{adv}}$ $\epsilon=0.31$ |  $\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ $\epsilon=8$ |  Time per Epoch (seconds)
---|---|---|---|---|---
MNG [21] |  79.8 |  43.9 |  75.8 |  53.8 |  $354^{\dagger}$
PGD+SNAP |  83.1 |  45.9 |  74.1 |  58.3 |  240

Table 6: ResNet-18 CIFAR-10 results showing a comparison between MNG [21] and
PGD+SNAP (from Table 2 in the main text). All MNG numbers are exactly as
reported in their paper. We reevaluate PGD+SNAP with our PGD attacks using the
new $\epsilon$ values used by Madaan et al. [21]. PGD+SNAP achieves 3%, 2%,
4.5% higher $\mathcal{A}_{\text{nat}}$,
$\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$,
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}$, respectively, while being at least
$\sim 40\%$ faster in terms of epoch time. $\dagger$: Note that MNG time is
measured on NVIDIA GeForce RTX 2080Ti (by Madaan et al. [21]), while PGD+SNAP
is measured on NVIDIA Tesla P100. An RTX 2080Ti has _20% more_ CUDA cores than
a Tesla P100.

The meta-noise generator (MNG) [21] employs a multi-layer deep-net to generate
noise samples during AT. Importantly, MNG still employs multiple attacks
during training, but samples only one of the attacks randomly at a time to
reduce the training cost.

However, they have yet to release their code or pretrained models even though
their work was posted on arXiv a year ago. Absence of public codes from Madaan
et al. [21] makes it difficult to clearly compare with their work, especially
in terms of training time. Nonetheless, in this subsection, we try our best to
ensure that the comparison is fair. Table 6 reports natural and adversarial
accuracy of MNG against $(\ell_{\infty},\ell_{2},\ell_{1})$ attacks as
reported by Madaan et al. [21]. We find that PGD+SNAP achieves 3%, 2%, 4.5%
higher $\mathcal{A}_{\text{nat}}$,
$\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$, and
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}$, respectively. Note that Madaan et al.
[21] evaluate $\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$ and
$\mathcal{A}^{(\ell_{2})}_{\text{adv}}$ against PGD-50 attacks, whereas here
we employ PGD-100 attacks and, following their protocol, evaluate on the
entire CIFAR-10 dataset with a single restart. Furthermore, epoch time for
PGD+SNAP is $1.4\times$ smaller than that of MNG [21] even though MNG time was
measured on a more recent NVIDIA RTX 2080Ti, which has 20% more CUDA cores
than the Tesla P100 GPU that we used for PGD+SNAP.

Importantly, a key advantage of SNAP is its scalability. We are able to report
robust ResNet-50 and ResNet-101 networks on ImageNet (Table 4 in the main
text), whereas Madaan et al. [21] report results only up to $64\times 64$
TinyImageNet.

###  B.3 SVHN results

Method |  $\mathcal{A}_{\text{nat}}$ |  $\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$ $\epsilon=0.03$ |  $\mathcal{A}^{(\ell_{2})}_{\text{adv}}$ $\epsilon=0.5$ |  $\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ $\epsilon=8$ |  $\mathcal{A}^{(U)}_{\text{adv}}$
---|---|---|---|---|---
PGD |  89.9 |  45.3 |  34.9 |  4.8 |  4.8
PGD+SNAP |  89.3 |  44.0 |  67.4 |  48.3 |  36.3

Table 7: ResNet-18 SVHN results showing the impact of SNAP augmentation of
$\ell_{\infty}$-PGD [22] AT frameworks. Adding SNAP improves
$\mathcal{A}^{(U)}_{\text{adv}}$ by $\sim 30\%$ while having only a small
impact on $\mathcal{A}_{\text{nat}}$ and
$\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$.

Table 7 shows PGD and PGD+SNAP results on SVHN data. We train both PGD and
PGD+SNAP models for 100 epochs using a piece-wise LR schedule. We start with
an initial LR of 0.01 and decay it once at the 95th epoch.

In Table 7, we observe a trend that is similar to our observations for
CIFAR-10 and ImageNet results. In particular, for SVHN, SNAP turns out to be
even more effective, with $\sim 30\%$ improvement in
$\mathcal{A}^{(U)}_{\text{adv}}$ while almost preserving both
$\mathcal{A}_{\text{nat}}$ and $\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$.

###  B.4 Impact of SNAP on prediction complexity

Method |  $\mathcal{A}_{\text{nat}}$ (%)
---|---
TRADES |  81.7
TRADES+SNAP
$N_{0}=1$ |  80.1$\pm 0.22$
$N_{0}=2$ |  80.3$\pm 0.14$
$N_{0}=4$ |  80.7$\pm 0.12$
$N_{0}=8$ |  80.9$\pm 0.10$
$N_{0}=16$ |  80.9$\pm 0.08$

Table 8: ResNet-18 CIFAR-10 results showing SNAP’s impact on the prediction
complexity, where $N_{0}$ denotes the number of noise samples employed to
estimate $\mathbb{E}[\cdot]$ in Eq. $(2)$ in the main text. We find that for
mere accuracy estimation, even a single forward pass ($N_{0}=1$) suffices.
${\small\pm\text{xx}}$ denotes the standard deviation over 10 independent test
runs.

While SNAP augmentation has a modest impact on the training time (Table 3 in
the main text), here we check whether it could _potentially_ increase the
model prediction complexity due to the need to estimate the expectation
$\mathbb{E}[\cdot]$ in Eq. $(2)$ in the main text.

As expected, by increasing $N_{0}$, the deviation of the
$\mathcal{A}_{\text{nat}}$ estimate reduces (see Table 8). However, we find
that for accuracy estimation, a single forward pass ($N_{0}=1$) suffices.
Specifically, an $\mathcal{A}_{\text{nat}}$ estimate with $N_{0}=1$ is within
1% of the $\mathcal{A}_{\text{nat}}$ estimate with $N_{0}=16$. Furthermore,
even with $N_{0}=1$, the standard deviation of $\mathcal{A}_{\text{nat}}$ is
as low as $\sim 0.2$%. Thus, the impact of SNAP on prediction complexity can
be very small.

###  B.5 Subspace analysis of adversarial perturbations for TRADES+SNAP model

Figure 8: Normalized mean squared projections of three perturbation types on
the singular vector basis $\mathcal{P}^{\kappa}$ of $\ell_{2}$ perturbations
of ResNet18 on CIFAR-10 after TRADES+SNAP training
($\kappa\equiv\text{rob\\_sn}$). The singular vectors ${\bm{p}}_{i}^{\kappa}$
comprising
$\mathcal{P}^{\kappa}=\\{{\bm{p}}_{1}^{\kappa},\dots,{\bm{p}}_{D}^{\kappa}\\}$
are ordered in descending order of their singular values.

In this subsection, we carry out a subspace analysis of adversarial
perturbations (Section 3 in the main text) for TRADES+SNAP. We confirm that
our hypothesis in Section 3 holds even after SNAP augmentation of TRADES.
Following the same experimental setup and the notation from Section 3 in the
main text, we compute perturbations $\bm{\alpha}_{i}$, $\bm{\beta}_{i}$, and
$\bm{\gamma}_{i}$ for each ${\bm{x}}_{i}\in X$ for ResNet-18 trained using
TRADES+SNAP, _i.e_., $\kappa\equiv\text{rob\\_sn}$. We compute the singular
vector basis $\mathcal{P}^{\kappa}$ for the set of $\ell_{2}$ bounded
perturbations
$\Delta^{\kappa}=\\{\bm{\beta}_{1}^{\kappa},\dots,\bm{\beta}_{|X|}^{\kappa}\\}$.
Fig. 8 plots the normalized mean squared projections of the three types of
perturbation vectors on the singular vector basis $\mathcal{P}^{\kappa}$ of a
TRADES+SNAP trained ResNet-18. We find that the projections generally follow
the same trend as those for a TRADES-trained network which are shown in Fig.
3(b) of the main text. However, we also notice that after SNAP augmentation,
the three perturbation types get squeezed into an even smaller 130-dimensional
subspace, _i.e_., projections are $<10\%$ of the maximum projection value for
all dimensions beyond the first 130 dimensions.

Method |  $\mathcal{A}_{\text{nat}}$ |  $\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$ $\epsilon=0.03$ |  $\mathcal{A}^{(\ell_{2})}_{\text{adv}}$ $\epsilon=0.5$ |  $\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ $\epsilon=12$ |  $\mathcal{A}^{(U)}_{\text{adv}}$
---|---|---|---|---|---
PGD |  84.6 |  48.8 |  62.3 |  15.0 |  15.0
Noise shaping basis $V=\mathbf{I}_{D\times D}$
+SNAP[G] |  80.7 |  45.7 |  66.9 |  34.6 |  31.9
+SNAP[U] |  85.1 |  42.7 |  66.7 |  28.6 |  26.6
+SNAP[L] |  83.0 |  44.8 |  68.6 |  40.1 |  35.6
Noise shaping basis $V=U_{\text{img}}$
+SNAP[G] |  81.7 |  48.9 |  67.5 |  29.8 |  28.7
+SNAP[U] |  82.0 |  46.6 |  67.8 |  27.8 |  25.7
+SNAP[L] |  81.7 |  46.8 |  65.9 |  28.5 |  27.4

Table 9: ResNet-18 CIFAR-10 results showing the impact of noise shaping basis
$V$ for $\ell_{\infty}$-PGD [22] AT framework with SNAP. In this table,
SNAP[G], SNAP[U], and SNAP[L] denote shaped noise augmentations with Gaussian,
Uniform, and Laplace noise distributions, respectively, and $U_{\text{img}}$
refers to the singular vector basis of the training images.

###  B.6 Impact of noise shaping in the image basis

Recall that, for all experiments in the main text, we chose the noise shaping
basis $V=\mathbf{I}_{D\times D}$, _i.e_., the noise was shaped and added in
the standard basis in $\mathbb{R}^{D}$, where $\mathbf{I}_{D\times D}$ denotes
the identity matrix (see Eq. (1) and Alg. 1 in the main text).

In this section, we explore the shaped noise augmentation in the _image basis_
, _i.e_., singular vector basis of the training set images. Specifically, we
choose $V=U_{\text{img}}=[{\bm{u}}_{1},\dots,{\bm{u}}_{D}]$, where
$U_{\text{img}}$ denotes the singular vector basis of the images in the
training set. Thus, the sampled noise vector ${\mathbf{n}}_{0}$ (see Eq. (1)
in the main text) is scaled by direction-wise standard deviation matrix
$\Sigma$ and _rotated_ by $U_{\text{img}}$ before being added to the input
image ${\bm{x}}$.

The rationale for choosing $V=U_{\text{img}}$ is as follows: Recent works [15,
35, 34] have demonstrated the generative behavior of adversarial perturbations
of networks trained with single-attack AT, _i.e_., adversarial perturbations
of robust networks exhibit semantics similar to the input images. Thus, the
perturbation basis (see section 3 in the main text) of the robust networks
trained with single-attack AT seems to be aligned with the image basis.

We repeat the experiments in Table 1 of the main text while keeping all the
settings _identical_ except for choosing $V=U_{\text{img}}$ instead of
$V=\mathbf{I}_{D\times D}$. Table 9 shows the results. The first three rows
correspond to $V=\mathbf{I}_{D\times D}$ and are reproduced from Table 1 of
the main text. Note that, in order to preserve
$\mathcal{A}_{\text{nat}}>81\%$, we need to reduce $P_{\text{noise}}=60$ when
$V=U_{\text{img}}$, since the noise is now pixel-wise correlated.

In Table 9, we notice that $\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ is
significantly reduced when $V=U_{\text{img}}$ as compared to the case
$V=\mathbf{I}_{D\times D}$. More interestingly, all three types of noise
distributions result in similar values for
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ when $V=U_{\text{img}}$. We discuss
this phenomenon in the next section, _i.e_., Sec. B.7 below.

Table 9 shows that the orientation of a noise vector is as important as its
distribution. The simpler choice of $V=\mathbf{I}_{D\times D}$ turns out to be
more effective.

###  B.7 Understanding the effectiveness of SNAP[L] for $\ell_{\infty}$ AT

In this subsection, we conduct additional studies to further understand the
following two observations in SNAP: (i) shaped Laplace noise is particularly
effective (Table 1 in the main text), and (ii) rotating noise vectors
($V=U_{\text{img}}$) reduces their effectiveness (Table 9 in this
Supplementary). We study the properties of the noise vector ${\mathbf{n}}$ for
different noise distributions.

We conjecture that the Laplace distribution is most effective because of its
heavier tail compared to Gaussian and Uniform distributions of the same
variance. A long-tailed distribution will generate more large magnitude
elements in a vector drawn from it and hence is more effective in emulating a
strong $\ell_{1}$-norm bounded perturbation. Furthermore, the standard (un-
rotated) basis preserves this unique attribute of samples drawn from such
distributions.

This conjecture is validated by Fig. 9(a) which shows that noise samples drawn
from the Laplace distribution in the standard basis have the highest average
number of dimensions with large ($>0.5$) magnitudes, followed by Gaussian and
Uniform distributions. This correlates well with the results in Table 1 in the
main text and Table III (first three rows), in that
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ is the highest for Laplace followed by
those for Gaussian and Uniform. Additionally, the use of $V=U_{\text{img}}$
dissolves this distinction between the three distributions as shown in Fig.
9(b) which explains the similar (and lower)
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ values for all three distributions in
Table III.

Figure 9: ResNet18 CIFAR-10 results: histograms of the fraction of noise
vector dimensions with magnitude (a) $>0.5$ when $V=\mathbf{I}_{D\times D}$,
and (b) $>0.4$ when $V=U_{\text{img}}$. Histograms are plotted for 5000 random
noise samples ${\mathbf{n}}$. The three shaped noise distributions are from
the corresponding networks in Table 9.

Thus, we confirm that the type of noise plays an important role in
robustifying single-attack $\ell_{\infty}$ AT frameworks to the union of
multiple perturbation models. Specifically, the noise vectors with higher
fraction of noise dimensions with larger magnitudes are better at
complementing $\ell_{\infty}$ AT frameworks.

###  B.8 Evaluating common corruptions and functional attack

Method |  $\mathcal{A}_{\text{nat}}$ |  $\mathcal{A}^{(U)}_{\text{adv}}$ |  $\mathcal{A}_{\text{cc}}$ |  $\mathcal{A}^{\text{(f)}}_{\text{adv}}$
---|---|---|---|---
|  |  |  ReColorAdv
Vanilla |  94.5 |  0.0 |  72.0 |  0.9
$\ell_{\infty}$-PGD |  84.6 |  15.0 |  75.6 |  53.5
Noise shaping basis $V=\mathbf{I}_{D\times D}$
+SNAP[G] |  80.7 |  31.9 |  72.8 |  55.1
+SNAP[U] |  85.1 |  26.6 |  75.0 |  46.9
+SNAP[L] |  83.0 |  35.6 |  75.3 |  51.3
Noise shaping basis $V=U_{\text{img}}$
+SNAP[G] |  81.7 |  28.7 |  73.6 |  54.5
+SNAP[U] |  82.0 |  25.7 |  73.1 |  54.0
+SNAP[L] |  81.7 |  27.4 |  73.4 |  55.3

Table 10: ResNet-18 CIFAR-10 results showing natural accuracy
$\mathcal{A}_{\text{nat}}$, adversarial accuracy
$\mathcal{A}^{(U)}_{\text{adv}}$ against the union of
$(\ell_{\infty},\ell_{2},\ell_{1})$ perturbations, accuracy
$\mathcal{A}_{\text{cc}}$ in the presence of common corruptions [13], and
adversarial accuracy $\mathcal{A}^{\text{(f)}}_{\text{adv}}$ against a
functional adversarial attack ReColorAdv [18]. All accuracy numbers are in %.
In this table, $\mathbf{I}_{D\times D}$ denotes $D$-dimensional identity
matrix, while $U_{\text{img}}$ denotes singular vector basis of the training
images. We find that SNAP augmentations of $\ell_{\infty}$-PGD significantly
($\approx 20\%$) improve $\mathcal{A}^{(U)}_{\text{adv}}$ while preserving
both $\mathcal{A}_{\text{cc}}$ and $\mathcal{A}^{\text{(f)}}_{\text{adv}}$.

In this subsection, we check if there are any other downsides of SNAP when it
improves robustness against the union of $(\ell_{\infty},\ell_{2},\ell_{1})$
perturbations. In particular, we check if SNAP improvements are achieved at
the cost of a drop in accuracy against common corruptions [13] or functional
adversarial attacks [18].

We use corrupted images provided by Hendrycks & Dietterich [13] to estimate
accuracy in the presence of common corruptions ($\mathcal{A}_{\text{cc}}$). We
average the accuracy numbers across different corruption strengths and types.
Also, we use the ReColorAdv setup of Laidlaw et al. [19] to estimate accuracy
against functional adversarial attacks
($\mathcal{A}^{\text{(f)}}_{\text{adv}}$). We also make it _adaptive_ to our
defense framework via appropriate noise averaging (similar to our adaptive PGD
attacks [33] discussed in the main text) to eliminate any gradient
obfuscations. As observed in Table 10, SNAP augmentations of PGD AT generally
preserve both $\mathcal{A}_{\text{cc}}$ and
$\mathcal{A}^{\text{(f)}}_{\text{adv}}$. In particular, 20.6% improvement in
$\mathcal{A}^{(U)}_{\text{adv}}$ via PGD+SNAP[L] (with $V=\mathbf{I}_{D\times
D}$) is accompanied with the same $\mathcal{A}_{\text{cc}}$ and only a $2.2\%$
lower $\mathcal{A}^{\text{(f)}}_{\text{adv}}$ ($=51.3\%$) compared to PGD AT.
In contrast, vanilla training achieves an
$\mathcal{A}^{\text{(f)}}_{\text{adv}}$ of only 0.9%. Even with
$V=U_{\text{img}}$, PGD+SNAP[L] achieves a 1.8% higher
$\mathcal{A}^{\text{(f)}}_{\text{adv}}$ along with a 12.4% improvement in
$\mathcal{A}^{(U)}_{\text{adv}}$. Note that all
$\mathcal{A}^{(U)}_{\text{adv}}$ numbers are idential to the ones reported in
Sec. B.6.

We conclude that SNAP augmentation of PGD AT improves
$\mathcal{A}^{(U)}_{\text{adv}}$ by up to 20% while preserving its robustness
against common corruptions and functional adversarial attacks. Thus, SNAP
expands the capabilities of $\ell_{\infty}$ AT frameworks without any
significant downside. However, further work is required to improve robustness
to a larger class adversarial attacks, such as rotation [7], texture [2],
etc., simultaneously.

###  B.9 Error bars

In this subsection, we confirm that benefits of SNAP are not specific to any
particular choice of random seed. Specifically, we run both PGD+SNAP (with
superconvergence) and FreeAdv+SNAP (see Table 3 in the main text) training
four times with different random seeds. Table 11 shows the mean accuracy and
its standard deviation for each of $\mathcal{A}_{\text{nat}}$,
$\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$,
$\mathcal{A}^{(\ell_{2})}_{\text{adv}}$,
$\mathcal{A}^{(\ell_{1})}_{\text{adv}}$, and $\mathcal{A}^{(U)}_{\text{adv}}$
with ResNet-18 on CIFAR-10. We find that the standard deviation of accuracy is
$\approx 0.5$% in almost all cases. This demonstrates the ease of replicating
SNAP results.

Method |  $\mathcal{A}_{\text{nat}}$ |  $\mathcal{A}^{(\ell_{\infty})}_{\text{adv}}$ $\epsilon=0.03$ |  $\mathcal{A}^{(\ell_{2})}_{\text{adv}}$ $\epsilon=0.5$ |  $\mathcal{A}^{(\ell_{1})}_{\text{adv}}$ $\epsilon=12$ |  $\mathcal{A}^{(U)}_{\text{adv}}$
---|---|---|---|---|---
PGD+SNAP |  82.5$\pm$0.27 |  43.1$\pm$0.61 |  66.9$\pm$0.57 |  39.0$\pm$0.41 |  33.7$\pm$0.29
FreeAdv+SNAP |  83.4$\pm$0.25 |  39.2$\pm$0.74 |  65.7$\pm$0.55 |  36.5$\pm$0.60 |  30.4$\pm$0.83

Table 11: ResNet-18 CIFAR-10 results showing the mean and standard deviation
for all accuracies over four different training runs of PGD+SNAP (with
superconvergence) and FreeAdv+SNAP. As observed, the standard deviation in
accuracy is $\approx 0.5$% in almost all cases, demonstrating the ease of
replicating SNAP results.

##  Appendix C Additional Details

###  C.1 Details of Hyperparameters

####  C.1.1 Attack hyperparameters

As mentioned in the main text, we follow basic PGD attack formulations of
Maini et al. [23]. We further enhance them to target the full defense – SN
layer – since SNAPnet is end-to-end differentiable. Specifically, we
backpropagate to the primary input ${\bm{x}}$ through the SN layer (see Fig.
4(b) in the main text). Thus, the final shaped noise distribution is exposed
to the adversary. We also account for the $\mathbb{E}_{\mathbf{n}}[\cdot]$
(see Eq. (2) in the main text) by explicitly averaging deep net logits over
$N_{0}$ noise samples _before_ computing the gradient, which eliminates any
gradient obfuscation, and was shown to be the strongest attack against noise
augmented models [33]. We choose $N_{0}=8$ for all our attack evaluations.

For $\ell_{2}$ and $\ell_{\infty}$ PGD attacks, we choose steps size
$\alpha=0.1\epsilon$. For $\ell_{1}$ PGD attacks, we choose the exact same
configuration as Maini et al. [23].

####  C.1.2 Training hyperparameters

As mentioned in the main text, we introduce SNAP without changing any
hyperparameters of BASE() AT. All BASE() and BASE()+SNAP training runs on
CIFAR-10 employ an SGD optimizer with a fixed momentum of $0.9$, batch size of
$250$, and weight decay of $2\times 10^{-4}$. Also, while accounting for the
$\mathbb{E}_{\mathbf{n}}[\cdot]$ (see Eq. (2) in the main text), note that
$N_{0}=1$ suffices during BASE()+SNAP training. Below we provide specific
details for each SOTA AT framework:

BASE() $\equiv$ PGD [22] on CIFAR-10:

$\ell_{\infty}$-PGD AT employed $\ell_{\infty}$-bounded PGD-$K$ attack with
$\epsilon=0.031$, step size $\alpha=0.008$, and $K=10$. For $\ell_{2}$-PGD AT,
we used an $\ell_{2}$-bounded PGD-$K$ attack with $\epsilon=0.5$, step size
$\alpha=0.125$ and $K=10$. Following Rice et al. [30], we employed $100$
epochs for PGD AT with step learning rate (LR) schedule, where LR was decayed
from 0.1 to 0.01 at epoch 96. Following Maini et al. [23], we also employed
_their_ cyclic LR schedule to achieve superconvergence in 50 epochs. Following
Maini et al. [23], we set weight decay to $5\times 10^{-4}$ in PGD AT.

In PGD+SNAP, the noise variances were updated every $U_{f}=10$ epochs and we
use $P_{\text{noise}}=160$ in Tables 1,2, and 3 in the main text.

BASE() $\equiv$ TRADES [46] on CIFAR-10:

Following Zhang et al. [46], TRADES AT employed $\ell_{\infty}$-bounded
perturbations with $\epsilon=0.031$, step size $\alpha=0.007$, and attack
steps $K=10$. We set TRADES parameter $1/\lambda=5$, which controls the
weighing of its robustness regularizer. It was trained for 100 epochs with a
step LR schedule, where LR was decayed to {0.01,0.001,0.0001} at the epochs
$\\{75,90,100\\}$, respectively. Following Maini et al. [23], we also employed
_their_ cyclic LR schedule to achieve superconvergence in 50 epochs, while
keeping all other settings identical.

In TRADES+SNAP, the noise variances were updated every $U_{f}=10$ epochs and
we use $P_{\text{noise}}=120$ in Tables 2 and 3 in the main text.

BASE() $\equiv$ FreeAdv [35] on CIFAR-10:

Following Shafahi et al. [35], FreeAdv AT was trained for 25 epochs, each
consisting of a replay of 8. It employed $\ell_{\infty}$ perturbations with
$\epsilon=0.031$. The learning rate was decayed to $\\{0.01,0.001,0.0001\\}$
at epochs $\\{13,19,23\\}$, respectively.

In FreeAdv+SNAP, the noise variances were updated every $U_{f}=5$ epochs,
since the replay of 8 scales down the total number of epochs. Also, we use
$P_{\text{noise}}=160$ in Tables 2 and 3 in the main text.

BASE() $\equiv$ FastAdv [40] on CIFAR-10:

Following Wong et al. [40], FastAdv AT employed a single-step $\ell_{\infty}$
norm bounded FGSM attack with $\epsilon=8/255$, step size $\alpha=10/255$, and
random noise initialization. It was trained for 50 epochs with the _same_
cyclic LR schedule used by Wong et al. [40]. We used a weight decay of
$5\times 10^{-4}$.

In FastAdv+SNAP, the noise variances were updated every $U_{f}=10$ epochs and
we use $P_{\text{noise}}=200$ in Tables 2 and 3 in the main text.

BASE() $\equiv$ FreeAdv [35] on ImageNet:

Following Shafahi et al. [35], FreeAdv AT was trained for 25 epochs, each
consisting of a replay of 4. It employed $\ell_{\infty}$ perturbations with
$\epsilon=4/255$, _identical_ to the authors’ original setup. The LR was
decayed by 0.1 every 8 epochs, starting with the initial LR of 0.1. We used
weight decay of $1\times 10^{-4}$.

In FreeAdv+SNAP, the noise variances were updated every $U_{f}=5$ epochs,
since the replay of 4 scales down the total number of epochs. Also, we use
$P_{\text{noise}}=4500$ in Table 4 in the main text, which corresponds to
noise standard deviation of $\sim 0.17$ per pixel _on average_.

MSD-$K$ [23] experiments for $K\in\\{30,20,10,5\\}$:

Maini et al. [23] report results for only MSD-50 in their paper. We produce
MSD-$K$ results using their publicly available code. While reducing the number
of steps in MSD, we _appropriately_ increase the step size $\alpha$ for the
attack. For MSD-50, Maini et al. [23] used $\alpha=(0.003,0.05,1.0)$ for
$(\ell_{\infty},\ell_{2},\ell_{1})$ perturbations, respectively. We
proportionately increase the step size to $\alpha=(0.005,0.084,1.68)$ and
$\alpha=(0.0075,0.125,2.5)$ for MSD-30 and MSD-20, respectively.

For MSD-10 and MSD-5, we choose $\alpha=(0.0075,0.125,2.5)$, since we found
that further increasing the step size $\alpha$ lead to _lower_ final
adversarial accuracy.

Other than the step-size, we do _not_ make any change to the original code by
Maini et al. [23].

AVG-$K$ [37] experiments for $K\in\\{30,20,10,5\\}$:

For AVG-50, we use the publicly available model provided by Maini et al. [23].
We produce AVG-$K$ results using the Maini et al. [23] code. When reducing the
number of steps, we appropriately increase the step size $\alpha$ for
$\ell_{\infty}$ and $\ell_{2}$ perturbations. Increasing the step size for
$\ell_{1}$ perturbations resulted in significantly lower
$\mathcal{A}^{(U)}_{\text{adv}}$, and thus $\alpha$ for $\ell_{1}$
perturbations was kept constant while reducing the number of steps. For
AVG-50, Maini et al. [23] used $\alpha=(0.003,0.05,1.0)$ for
$(\ell_{\infty},\ell_{2},\ell_{1})$ perturbations, respectively. We increase
the $\ell_{\infty}$ and $\ell_{2}$ step sizes to set
$\alpha=(0.005,0.084,1.0)$ and $\alpha=(0.0075,0.125,1.0)$ for AVG-30 and
AVG-20 respectively.

As with MSD, we do not further increase the step size $\alpha$ for AVG-10,
AVG-5, and instead choose $\alpha=(0.0075,0.125,1.0)$. Even here, we found
that increasing the step size for $\ell_{1}$ perturbations results in lower
$\mathcal{A}^{(U)}_{\text{adv}}$. For AVG-2, we increase the step size for all
perturbations to $\alpha=(0.024,0.4,8)$.

PAT [19] on CIFAR-10:

For comparisons with Laidlaw et al. [19], we evaluate their publicly available
self-bounded ResNet-50 model.

###  C.2 Details about SNAP

####  C.2.1 Distribution Update Epoch

In the SNAP distribution update epoch (see Algorithm 1 in the main text), we
employ $\ell_{2}$ norm-bounded PGD attack to compute perturbation vectors
$\bm{\eta}$. We use only 20% of the training data, which is randomly selected
during every SNAP update epoch. Recall that normalized root mean squared
projections of $\bm{\eta}$ dictate the updated noise variances (Eq. (3) in the
main text). In the following we provide more details specific to CIFAR-10 and
ImageNet data:

CIFAR-10: we employ 10 step $\ell_{2}$-PGD attack with $\epsilon=1.8$ and
$N_{0}=4$.

ImageNet: we employ 4 step $\ell_{2}$-PGD attack with $\epsilon=4.0$ and
$N_{0}=1$.

Note that $\ell_{2}$ norm bound $\epsilon$ for the PGD attack here does not
play any role, since $\bm{\eta}$ perturbation projections are normalized.

####  C.2.2 Noise variance initialization in SNAP

In SNAP, we initialize the noise variances to be uniform across all
dimensions. Specifically, in Algorithm 1 in the main text,
$\Sigma_{0}=\text{Diag}\bigg{[}\sqrt{\frac{P_{\text{noise}}}{D}},\dots,\sqrt{\frac{P_{\text{noise}}}{D}}\bigg{]}$
for a given value of $P_{\text{noise}}$.

##  Appendix D Accompanying Code and Pretrained Models

As a part of this appendix, we share our code to reproduce PGD+SNAP and
TRADES+SNAP results on CIFAR-10 (Table 2 in the main text) and FreeAdv+SNAP
results on ImageNet (Table 4 in the main text). We also share corresponding
pretrained models to facilitate quick reproduction of our results. Code and
models are available at link: https://github.com/adpatil2/SNAP