# Jacobian Norm with Selective Input Gradient Regularization for Improved and
Interpretable Adversarial Defense

Deyin Liu, Lin Wu, Haifeng Zhao, Farid Boussaid, Mohammed Bennamoun, Xianghua
Xie
D. Liu and H. Zhao are with Anhui Provincial Key Laboratory of Multimodal
Cognitive Computation, School of Artificial Intelligence, Anhui University,
Hefei 230039, China. E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
F. Boussaid and M. Bennamoun are with The University of Western Australia,
Perth 6009, Australia. E-mail: {Farid.Boussaid;
<EMAIL_ADDRESS>
L. Wu and X. Xie are with School of Computer Science, Swansea University, SA1
8EN, UK. E-mail<EMAIL_ADDRESS>
###### Abstract

Deep neural networks (DNNs) are known to be vulnerable to adversarial examples
that are crafted with imperceptible perturbations, i.e., a small change in an
input image can induce a mis-classification, and thus threatens the
reliability of deep learning based deployment systems. Adversarial training
(AT) is often adopted to improve robustness through training a mixture of
corrupted and clean data. However, most of AT based methods are ineffective in
dealing with transferred adversarial examples which are generated to fool a
wide spectrum of defense models, and thus cannot satisfy the generalization
requirement raised in real-world scenarios. Moreover, adversarially training a
defense model in general cannot produce interpretable predictions towards the
inputs with perturbations, whilst a highly interpretable robust model is
required by different domain experts to understand the behaviour of a DNN. In
this work, we propose a novel approach based on Jacobian norm and Selective
Input Gradient Regularization (J-SIGR), which suggests the linearized
robustness through Jacobian normalization and also regularizes the
perturbation-based saliency maps to imitate the model’s interpretable
predictions. As such, we achieve both the improved defense and high
interpretability of DNNs. Finally, we evaluate our method across different
architectures against powerful adversarial attacks. Experiments demonstrate
that the proposed J-SIGR confers improved robustness against transferred
adversarial attacks, and we also show that the predictions from the neural
network are easy to interpret.

###### Index Terms:

Selective Input Gradient Regularization, Jacobian Normalisation, Adversarial
Robustness.

##  I Introduction

Recent years have seen deep neural networks (DNNs) achieve impressive
performance in a myriad of image recognition tasks, e.g., image classification
on real-world benchmarks. However, DNNs could be easily fooled by crafted
imperceptible perturbations aimed at driving the network to make a wrong
prediction. Such a vulnerability poses an obstacle in the deployment of deep
learning systems for real-world applications given the security-sensitivity
and legal ramifications [1]. Since this vulnerability was first identified by
Szegedy et al. [2], many techniques for generating a group of malicious
examples have been proposed including i) white-box attacks (e.g., Projected
Gradient Descent (PGD) [3], Fast Gradient Sign Method (FGSM) [2] and C&W [4])
which require knowledge of the deep learning parameters; and ii) black-box
attacks [5, 6, 7] can attack the DNNs even without any knowledge of the model
parameters. This is due to the transferability of adversarial examples:
examples generated to fool one model tend to fool all models trained on the
same dataset [8, 9].

Currently, the best defense models available to counter adversarial attacks
are based on adversarial training (AT), which trains the models on a mixture
of clean and adversarial examples to better classify potentially adversarial
examples during test time [10]. Although a number of effective defenses
employing AT have been proposed [11, 12], they rely on a computationally
expensive brute force solution to generate the potent adversarial examples
[13]. Besides, a study [14] shows that the robustness provided by AT can be
circumvented by randomizing or transferring perturbations from other models,
albeit ensemble helps. This has raised concerns about the generalization
ability of adversarially trained models under the attacking of adversarial
examples simply modified from a unknown distribution. For example, a robust
DNN model adversarially trained on gradient-generated examples should still
remain robust when these examples are further crafted by Gaussian randomness
on the feature dimensions (The dversarially generated example with a new
additive Gaussian noise can be viewed as a transferred one from its source
[15]).

Besides robustness, domain experts are also concerned with the
interpretability of a DNN’s predictions. Interpretability is of particular
importance in domains with safety requirements which need to know how a model
is trained and used. For example, medical specialists are intriguing to know
how the model responds to training data from different hospitals. To be more
interpretable, some methods proposed to regularize the input gradients [16,
17], which can illuminate the regions of confidence predicted by the DNNs. For
example, [17] has shown that by training a model to have smooth input
gradients with fewer extreme values (including the background pixels), the
prediction of a DNN will be more interpretable. However, this endeavor [17]
smoothed out all the gradients without showing the most appropriate
interpretability of DNNs, i.e., such method [17] is limited in presenting the
network’s response to small input variations. Other techniques such as
integrated gradients [18] and SmoothGrad [19] can also generate smoother and
more interpretable prediction confidence but they do not capture the response
of a deep neural network under input variations. As a matter of fact, the
local behavior can simulate how the network responds to small perturbations of
its inputs.

In this paper, we aim to improve both robustness and prediction
interpretability in the presence of small input perturbations. This is
achieved by simultaneously minimizing the Jacobian norm of the whole network
as well as regularizing the input gradients. Specifically, we introduce the
notion of linearized robustness, which sets the bounds on the network response
towards a perturbed example. And we show that the robustness can be
approximated by the Jacobian norm, which is computed as the gradients of the
network prediction logits with respect to the input. However, directly using
the raw input gradients could be noisy and hard to interpret. Inspired by [16]
which has shown that gradient suppression together with selected features can
explain the robustness of a model, we propose to regularize the input
gradients to provide a better input space. Then, we train the network using
the Jacobian’s Frobenius norm, which sets the bounds on the response of the
network layers to input perturbations. We empirically show how the Jacobian
norm relates to the linearized robustness of the DNNs, and how to leverage it
into the adversarial training to mitigate the adversarial effects.

To further make the prediction more interpretable, we propose to use the
perterbation-based saliency map, which is taken as the salient Jacobian matrix
that can relate the gradients to the corrupted input image and semantically
visualize the discriminative portions of that image [20]. These Jacobians,
computed during training for different robust models, can be regarded as
visual resemblers to the corresponding images, and a previous study has shown
that the saliency map is a result of robustness [21]. This strategy is
demonstrated to be robustness across different transferred adversarial
examples. The contributions of this paper can be summarized as:

  * •

We propose a novel approach to achieve both improved robustness and high
interpretability of DNNs under adversarial attacks. The proposed approach
effectively leverages the Jacobian norm and selective input gradient
regularization, which explicitly describes the network responses to input
perturbations.

  * •

We investigate the relationship between a Jacobian norm and the linearized
robustness 111(The distance from the input image to the decision boundary) of
DNNs. Based on perturbation-based saliency maps, our results give insights
into the prediction confidence of DNNs, which relates to the adversarial
effects.

  * •

Our method improves the robustness of DNNs towards transferred adversarial
examples across multiple architectures and different attacks.

The rest of this paper is organized as follows. Section II reviews the recent
related works. Section IV details the proposed method, and Section V presents
extensive experiments to evaluate our method. Finally, we conclude the paper
in Section VI.

##  II Related work

###  II-A Adversarial training based methods

There is a sizable body of work proposing various attack and defense
mechanisms for the adversarial setting. Among them, the current unbroken
defenses are based on adversarial training (AT) [2, 22, 3], which uses
adversarial examples as training data to protect DNNs against a range of
adversarial attacks. For example, projected gradient descent (PGD) is one such
strong defense that is able to generate universal adversarial examples using a
first-order approach [3]. A more recent work [22] encourages the decision
boundary to be smooth by adding a regularization term to reduce the difference
between the predictions on natural and adversarial examples. Qin et al. [23]
smoothened the loss landscape through local linearization by minimizing the
prediction difference between the clean and adversarial examples. While the
various aforementioned approaches can improve the AT, they require the
generation of sufficient adversarial examples for training. This results in a
prohibitive computational cost, which is proportional to the number of steps
needed to generate the adversarial examples. In addition, it requires a back-
propagation for each iteration. To strengthen DNNs under adversarial attacks,
a biologically-inspired approach [24] was introduced to learn flat, compressed
representations that are sensitive to a minimal number of input dimensions.
Unlike [24], this paper introduces a simpler yet effective approach for model
regularization that is based on input gradient regularization. A concurrent
method is [25], which can improve robustness by imposing the input gradient
regularization. However, performing such gradient with respect to a high
dimensional input from back-propagation is quite time-consuming. In contrast,
the proposed method approximates the linearized robustness of neural networks
via the penalization of a classifier’s Jacobian norm. Such a Jacobian norm
derives salient gradient maps to selectively activate the most discriminative
gradients.

###  II-B Regularization for robustness

To defend against adversarial examples, provable defenses promote the concept
of improving model robustness through regularization. A well-known strategy
includes noise injection, which is a variant of dropout weights [26] or
activations [27]. Several works have investigated the benefits of using a
regularization term on top of the standard training objective to reduce the
Jacobian’s Frobenius norm. Such a term aims to reduce the adversarial effect
on the model predictions caused by input perturbation. For instance, Hoffman
et al. [28] proposed an efficient method to approximate the input-class
probability through the output Jacobians of a classifier so as to minimize the
computational cost associated to these Jacobian norms. Tsipras et al. [20]
observed that adversarially trained models produce salient Jacobian matrices
that loosely resemble the input images whilst less robust models have noisier
Jacobians. Etmann et al. [21] interpreted linearized robustness as the
alignment between the Jacobian and the input image, and trained a robust model
by using double back-propagation. In comparison to these methods [20, 21], our
work offers the following merits: 1) it focuses on the local linearized
robustness of neural networks to provide an interpretation of the network’s
response to the inputs via Jacobian norm; 2) the proposed selective input
gradient regularization explicitly measures the degree of robustness, which
improves the interpretability of the network prediction; and 3) our method is
computationally more efficient in calculating the input gradients and also
highly interpretable to understand the network’s prediction.

##  III Preliminary

In this section, we formalize the notations, definitions used in this paper.
We also briefly present the baseline attacks and defenses against which we
will evaluate our method.

###  III-A Definition

Let a classification model
$F_{\theta}(\mathbf{X}):\mathbf{X}\mapsto\hat{\mathbf{Y}}\in\mathbb{R}^{N\times
K}$ map the inputs $\mathbf{X}\in\mathbb{R}^{N\times D}$ to the output
probabilities for $K$ classes, where $\theta$ represents the classifier’s
parameters and $\hat{\mathbf{Y}}\in\mathbb{R}^{N\times K}$ returns the
predictions of $F_{\theta}$. To train the model $F_{\theta}$, we aim to find a
set of parameter $\theta^{*}$ that minimizes the total distance between the
predictions $\hat{\mathbf{Y}}$ and the one-hot encoded true labels
$\mathbf{Y}\in\mathbb{R}^{N\times K}$ on a training set:

$\theta^{*}=\arg\min_{\theta}\sum_{n=1}^{N}\sum_{k=1}^{K}-\mathbf{Y}_{n,k}\log
F_{\theta}(\mathbf{X}_{n,k}),$ (1)

which can also be written as
$\arg\min_{\theta}H(\mathbf{Y},\hat{\mathbf{Y}})$, where $H$ is the sum of the
cross entropies between the predictions and the true labels.

Given the input $\mathbf{x}\in\mathbb{R}^{h\times w\times c}$ of a DNN, one
can define the Jacobian matrix $J$ with respect to $\mathbf{x}$ as

$J(\mathbf{x}):=\nabla_{\mathbf{x}}F_{\theta}(\mathbf{x})=\left[\frac{\partial
F_{\theta}(\mathbf{x})}{\partial\mathbf{x}_{1}},\ldots,\frac{\partial
F_{\theta}(\mathbf{x})}{\partial\mathbf{x}_{D}}\right],$ (2)

where $D=h\times w\times c$ is the dimensionality of $\mathbf{x}$. While the
DNNs can be trained empirically to perform well on the training data, their
accuracy degrades sharply in the presence of adversarial examples. Given a
small perturbation $\mathbf{z}$ applied to the input $\mathbf{x}$, a model is
still deemed robust against this attack if it satisfies

$\displaystyle\arg\max_{k\in K}F^{k}_{\theta}(\mathbf{x})=\arg\max_{k\in
K}F^{k}_{\theta}(\mathbf{x}+\epsilon\mathbf{z}),$ (3)
$\displaystyle\forall\epsilon\in
B_{p}(\varepsilon)=\epsilon:||\epsilon||_{p}\leq\varepsilon,$

where $\varepsilon$ is the scaling factor, and $p=\infty$. To improve the
model’s robustness, adversarial training [2] tries to find a distribution
match between the training data and the adversarial test data. Specifically,
adversarial training attempts to minimize the loss function as:

$\small\min_{\theta}\rho(\theta),\mbox{where}~{}~{}\rho(\theta)=\mathbb{E}_{(\mathbf{x},\mathbf{y})}\left[\max_{\epsilon\in
B(\varepsilon)}H(F_{\theta}(\mathbf{x}+\epsilon\mathbf{z}),\mathbf{y})\right],$
(4)

where the inner maximization terms are usually obtained by performing an
iterative gradient-based optimization, such as the projected gradient descent
(PGD) [3].

###  III-B Attacks

We consider two widely adopted gradient-based attacks and one Jacobian-based
attack.

#### Fast Gradient Sign Method (FGSM) [2]

This method can generate adversarial examples by perturbing the inputs s
increase the local linear approximation of the loss function:
$\mathbf{x}_{FGSM}=\mathbf{x}+\epsilon\cdot\mbox{sign}\nabla_{\mathbf{x}}H(\mathbf{y},\hat{\mathbf{y}})$.
If $\epsilon$ is small (e.g., $\epsilon=0.01$), adversarial examples become
indistinguishable to a human, but a neural network performs significantly poor
with them. To perform this attack, one can iteratively use a small $\epsilon$
to induce misclassifications by following the non-linear loss function in a
series of small linear steps [29].

#### Projected Gradient Descent (PGD) [3]

The PGD provides a white-box and yet stronger attack than the previous
iterative based method (i.e., FGSM [2]). PGD firstly uniforms the random
perturbation as the initialization. Then, an adversarial example is found
after several iterations by taking the form as
$\mathbf{x}_{PGD}^{t+1}$=$\prod_{\mathbf{x}+S}[\mathbf{x}_{PGD}^{t}+\epsilon\cdot\mbox{sign}\nabla_{\mathbf{x}}H(\mathbf{y},\hat{\mathbf{y}})]$,
where $\prod$ is the projection operator that clips the input at positions
around the predefined perturbation range. $\mathbf{x}+S$ represents the
perturbation set, and $\epsilon$ is the gradient step.

#### Jacobian-based Saliency Map Attack (JSMA) [30]

The JSMA iteratively searches for pixels of the input to change such that the
probability of the target label is increased and the probability of all other
labels are decreased. Such a method can produce examples that have only been
modified in a fraction of feature dimensions, which are hard for humans to
detect.

###  III-C Defenses

We consider two baseline defenses: adversarial training and defensive
distillation. Defensive methods that are not architecture-agnostic [31] or
simply rejecting adversarial examples [24] are not considered in this paper.

#### Adversarial training [10]

Adversarial training (AT) can enhance robustness by injecting adversarial
examples into the training process. We follow the implementation in [32],
where we augment the network to run the FGSM [2] on the training batches, and
compute the average loss on both the normal and adversarial examples as the
loss function of the model. To inhibit the FGSM attack [2], gradients are not
allowed to propagate and FGSM perturbations are computed with respect to the
predicted labels (instead of the true labels) to prevent label leaking.

#### Defensive distillation [10]

Distillation can be used as a defense technique by first using the one-hot
ground truth labels to train an initial model and subsequently utilizing the
initial model’s softmax probability outputs. Since distillation extracts class
knowledge from these probability vectors, this knowledge can be transferred
into a different DNN architecture by annotating the inputs in the training
dataset of the second DNN using classification predictions from the first DNN.
This idea is formulated to improve the resilience of DNNs in the presence of
perturbations [33]. Within a softmax layer, we divide all of the logit network
output (which we call $\hat{z}_{k}$) by a temperature $T$:
$F_{T,\theta}(\mathbf{X}_{n,k})=\frac{e^{\hat{z}_{k}(\mathbf{X}_{n,k})/T}}{\sum_{i=1}^{K}e^{\hat{z}_{i}(\mathbf{X}_{n,k})/T}}$,
where $F_{T,\theta}$ denotes a network output in the form of a softmax vector
with temperature $T$. The predictions will converge to $1/K$ as
$T\rightarrow\infty$. Thus, the distillation based defense can be formulated
as

$\displaystyle\theta^{0}=\arg\min_{\theta}\sum_{n=1}^{N}\sum_{k=1}^{K}-\mathbf{Y}_{n,k}\log
F_{T,\theta}(\mathbf{X}_{n,k}),$ (5)
$\displaystyle\theta^{*}=\arg\min_{\theta}\sum_{n=1}^{N}\sum_{k=1}^{K}-F_{T,\theta^{0}}(\mathbf{X}_{n,k})\log
F_{T,\theta}(\mathbf{X}_{n,k}).$

##  IV Proposed approach

In this section, we detail the proposed method (Fig. 1), which leverages the
Jacobian norm with selective input gradient regularization (namely J-SIGR) for
both improved defense and interpretability of a deep neural network under
powerful attacks.

Figure 1: The proposed scheme for adversarial robustness based on Jacobian
normalization and selective input gradient regularization (J-SIGR). The
Jacobian normalization sets the linear robustness bounds for perturbations.
The selective input gradient regularization is based on perturbation-based
saliency map to not only encourage the insensitivity of the input space but
also improves the interpretability.

###  IV-A Jacobian norm

In the following, we will study the relationship between the Jacobian norm
(JN) based regularization term and the notion of linearized robustness. Since
adversarial perturbations are small variations that change the predicted
result of a neural network classifier, it is sensible to define the robustness
towards adversarial perturbations via the distance of the clean image to the
nearest perturbed image which may cause the incorrect classification. When
such distance gets smaller, the perturbed and its clean counterpart are more
indistinguishable for a neural network, and thus the prediction of the neural
network will be correct.

Linearized adversarial robustness bound: Let
$i^{*}=\arg\max_{i}F^{i}_{\theta}(\mathbf{x})$ and $j^{*}=\arg\max_{j\neq
i^{*}}F^{j}_{\theta}(\mathbf{x}+\epsilon\mathbf{z})$ be the top prediction of
the input $\mathbf{x}$ and its corrupted sample
$\hat{\mathbf{x}}=\mathbf{x}+\epsilon\mathbf{z}$, respectively. Here,
$\hat{\mathbf{x}}$ is formed by small additive perturbations with Gaussian
distribution $\mathbf{z}\sim\mathcal{N}(0,\sigma^{2})$. The linearized
adversarial robustness is upper-bounded by the Jacobian norm
$||J(\mathbf{x})||_{F}^{2}$ with respect to $\mathbf{x}$.

Proof. Denoting $F^{i}_{\theta}(\mathbf{x})$ as the logits value of class $i$
in a classifier $F$ 222For notation simplicity, we omit $\theta$ in the
following. for $\mathbf{x}$ with $\epsilon\ll 1$, then its linearized
robustness can be expressed as

$\rho(\mathbf{x}):=\min_{j^{*}\neq
i^{*}}\frac{F^{i*}(\mathbf{x})-F^{j*}(\mathbf{x})}{||\nabla_{\mathbf{x}}F^{i*}(\mathbf{x})-\nabla_{\mathbf{x}}F^{j*}(\mathbf{x})||}.$
(6)

Denoting $g:=\nabla_{\mathbf{x}}(F^{i*}-F^{j*})(\mathbf{x})$ as the Jacobian
w.r.t the difference of two logits and $\alpha(\mathbf{x},g)=<\mathbf{x},g>$
as the alignment between the Jacobian and the input, then we have
$\rho(\mathbf{x})\leq\frac{\alpha(\mathbf{x},g)+C}{||g||}$, where $C$ is a
positive constant. Therefore,
$\rho(\mathbf{x})\leq\frac{J(\mathbf{x})+g+C}{||g||}$, where $J(\mathbf{x})$
is the Jacobian of the network output w.r.t the input $\mathbf{x}$.

We can now treat the term
$\hat{\rho}=\mathbf{z}^{T}J(\mathbf{x})^{T}J(\mathbf{x})\mathbf{z}$ as one
sample stochastic trace estimator for $Tr(J(\mathbf{x})^{T}J(\mathbf{x}))$
with a Gaussian variable $\mathbf{z}$:

$\mathbb{E}_{\mathbf{z}}[\rho]=\frac{Tr(J(\mathbf{x})^{T}J(\mathbf{x})\mathbb{E}[\mathbf{z}\mathbf{z}^{T}])}{||g||}=\frac{||J(\mathbf{x})||_{F}^{2}}{||g||}.$
(7)

Taking expectation over $m$ samples of a mini-batch $X$, we have
$\mathbb{E}[\rho]=\mathbb{E}_{\mathbf{x}}[||J_{X}||]_{F}^{2}$, where
$||\cdot||_{F}^{2}$ represents the Frobenius norm. We remark that the above
assumption holds true given that the neural network can be locally
approximated by a linear model [21]. For classifiers built on locally affine
score functions, as in the case of most neural networks using ReLU or leaky
ReLU activations, the decision boundary can be computed, provided that the
locally affine region around the point $\mathbf{x}$ is sufficiently large. As
proved in [21], for a classifier defined with a locally affine score function,
the decision boundary between the clean and the perturbed data is close in the
Euclidean space when their respective input signals are also close enough in
the Euclidean space [34]. Thus, the linearized robustness holds approximately
as long as the linear approximation to the network’s score function is
sufficiently plausible in the relevant neighborhood of $\mathbf{x}$.

Algorithm 1 The proposed adversarial robust model based on Jacobian Norm and
Selective Input Gradient Regularization (J-SIGR).

Inputs: Training set
$D_{train}=\\{\mathbf{x}_{i},\mathbf{y}_{i}\\}^{N}_{i=1}$.
Parameters: $\theta$, $\lambda_{j}$, $\lambda_{m}$ $\epsilon$.
Outputs: Classification model $F_{\theta}(\cdot)$ parameterized by $\theta$.
/*Initialization*/

1: Initialize $\theta$ by using a pre-trained network architecture;

2: Set $\lambda_{j}=\lambda_{m}=0.5;\epsilon=0.3$.

3: while each iteration or a condition is met do

4: Sample $(\mathbf{x},\mathbf{y})\sim D_{train}$; /*Input to the DNN*/

5: Generate the noise perturbation $\mathbf{z}\sim\mathcal{N}(0,\sigma^{2})$;

6: $\hat{\mathbf{x}}=\mathbf{x}+\epsilon\mathbf{z}$; /*Generate a perturbed
sample*/

7: $\nabla_{\mathbf{x}}F_{\theta}(\hat{\mathbf{x}})$; /*Compute the
perturbation-based saliency map*/

8: Compute the Jacobian norm $||J(\mathbf{x})||_{F}^{2}$;

9: Train the network using Eq. (11) and update $\theta$.

10: end while

11: return $\theta$.

Improved robustness using the Jacobian norm: In the presence of perturbed
examples, the expected response of a DNN should stay similar to the correct
prediction, which can be mathematically described as
$\Omega=F_{\theta}(\mathbf{x})-F_{\theta}(\hat{\mathbf{x}})$. Suppose
$\mathbf{x}_{i}$ is the $i$-th component of noise-free signal $\mathbf{x}$,
and $\hat{\mathbf{x}}_{i}=\mathbf{x}_{i}+\epsilon\mathbf{z}_{i}$ is the noise-
crafted tensor variation of $\mathbf{x}$. Note that the term $\Omega$ measures
the difference of the predictions in the case of clean data and its perturbed
counterpart. $\mathbf{z}$ is the noise term, which is sampled from a Gaussian
distribution with zero mean and variance $\sigma^{2}$ for each inference. Note
that the noise term has an identical variance to $\mathbf{x}$ so that the
additive noise only relies on the distribution of $\mathbf{x}$ to dynamically
perturb the input. According to the above, the linearized robustness can be
approximated and upper-bounded by the Jacobian of the classifier’s prediction
w.r.t the input $\mathbf{x}$. Thus, with the gradient back-propagation, we can
determine the gradient calculation w.r.t the difference of the two predictions
$\Omega$ via:

$\displaystyle\nabla_{\mathbf{x}}\Omega=\nabla_{\mathbf{x}}(F_{\theta}(\mathbf{x})-F_{\theta}(\hat{\mathbf{x}}))=\nabla_{\mathbf{x}}F_{\theta}(\mathbf{x})-\nabla_{\mathbf{x}}F_{\theta}(\hat{\mathbf{x}})$
(8)
$\displaystyle\Rightarrow||\nabla_{\mathbf{x}}F_{\theta}(\mathbf{x})-\nabla_{\mathbf{x}}F_{\theta}(\hat{\mathbf{x}})||_{F}^{2}\leq||J(\mathbf{x})||_{F}^{2}.$

The above Jacobian norm can approximate the linear robustness towards the
input, which simulates how the network will respond to those small variations
of the input. To compute the Jacobian norm, one needs to take the model’s
gradient with respect to its inputs, which provides a local linear
approximation of the model’s behavior. However, directly using the raw input
gradients is demonstrated to be ineffective since these gradients are quite
noisy and hard to interpret [35]. To combat this challenge, in the following
we propose to train the classification model under an input gradient
regularization with fewer extreme values and a minimized Jacobian norm.

###  IV-B Selective input gradient regularization

#### Input gradient regularization

The idea of input gradient regularization was first introduced by Drucker et
al [36] to train neural networks by minimizing not just the “energy” of the
network but also the rate of the change of the energy with respect to the
input features. The energy can be formulated using the cross-entropy as
follows:

$\displaystyle\theta^{*}=\arg\min_{\theta}\sum_{n=1}^{N}\sum_{k=1}^{K}-\mathbf{Y}_{n,k}\log
F_{\theta}(\mathbf{X}_{n,k})$ (9)
$\displaystyle+\lambda_{m}\sum_{n=1}^{N}\sum_{d=1}^{D}\left(\frac{\partial}{\partial\mathbf{x}_{d}}\sum_{k=1}^{K}-\mathbf{Y}_{n,k}\log
F_{\theta}(\mathbf{X}_{n,k})\right)^{2},$

where $\lambda_{m}$ is a hyperparameter modulating the penalty strength. Such
input gradient regularization ensures that even if the input change slightly,
the KL divergence between the predictions and the true labels will not be
changed significantly. This double back-propagation can provide a constraint
on the sensitivity caused by perturbations. Intuitively, the gradient penalty
term encourages the predictions not be sensitive to small perturbations in the
input space because it regularizes the input gradients to be smoother with
fewer extreme values. We combine the Jacobian norm and the input gradient
regularization, which can be formulated as follows:

$\displaystyle\theta^{*}=\arg\min_{\theta}\sum_{n=1}^{N}\sum_{k=1}^{K}-\mathbf{Y}_{n,k}\log
F_{\theta}(\mathbf{X}_{n,k})$ (10)
$\displaystyle+\lambda_{m}\sum_{n=1}^{N}\sum_{d=1}^{D}\left(\frac{\partial}{\partial\mathbf{x}_{d}}\sum_{k=1}^{K}-\mathbf{Y}_{n,k}\log
F_{\theta}(\mathbf{X}_{n,k})\right)^{2}$
$\displaystyle+\lambda_{j}||J(\mathbf{x})||_{F}^{2},$

where $\lambda_{m}$ and $\lambda_{j}$ denote the weights for the selective
gradient regularization and the Jacobian normalization, respectively. However,
in Eq. (10), the combination of Jacobian norm and input gradient
regularization only provides constraints for very near training examples.
Thus, it does not solve the adversarial perturbation problem. It is also
expensive to make derivatives smaller to limit the sensitivity to
infinitesimal perturbations. With this regard, in the following, we propose a
perturbation based saliency map to select the most discriminative features,
which are not only robust to perturbations but more interpretable to the
network behaviour.

#### Perturbation based saliency for selective input gradient regularization

Saliency map in deep learning is a technique used to interpret input features
that are determined to be important for the neural network output [37, 38,
19]. As domain experts are more concerned with the interpretability of a DNN,
some methods have been proposed to generate saliency maps to explain the
decision making in the DNN. One may directly use gradients to estimate the
influence of input features on the output. However, the quality of the
gradient-based saliency maps is generally poor as gradient-based saliency map
methods tend to overly smooth the gradients [25, 39].

In the spirit of saliency map in highlighting the importance of input
features, we propose to use the perturbation based saliency map, denoted as
$\mathcal{M}_{d}=f(\nabla_{\mathbf{x}}F_{\theta}(\hat{\mathbf{x}}))$, which is
derived from the gradient of a perturbed input. The $f(\cdot)$ is a mapping
function to be detailed later. Such a perturbation-based saliency map can be
computed by perturbing the input and observing the change in output, and thus
shows high interpretability in a DNN’s behaviour. Specifically, we compute
this saliency map by resembling the input images and highlighting the most
salient parts. Following [40], we use a Generative Adversarial Network (GAN)
to generate a saliency map to be visually similar to the real image. Given
$J(\hat{\mathbf{x}})=\nabla_{\mathbf{x}}F_{\theta}(\hat{\mathbf{x}})$, we use
an aligning network ($f$) to map the Jacobian into the domain of the input
image: $J^{\prime}=f(J(\hat{\mathbf{x}}))$. In our implementation, $f$
(parameterized by $\Phi$) is implemented by a single $1\times 1$ convolutional
layer with a tanh activation function. Hence, the generator
$G(\mathbf{x},\mathbf{y})$ can be framed by using both the classifier and the
aligning network:
$G_{\theta,\Phi}(\mathbf{x},\mathbf{y})=f(\nabla_{\mathbf{x}}F_{\theta}(\hat{\mathbf{x}}))$.
As a result, the generator can learn to model the distribution of
$p_{J^{\prime}}$ to resemble that of $p_{\mathbf{x}}$.

Once $\mathcal{M}_{d}$ is obtained, we incorporate the salient map into input
gradient computation such that the most robust gradients can be selected
during back-propagation. Mathematically, the objective with Jacobian norm and
selective input gradient regularization is defined as

$\displaystyle\theta^{*}=\arg\min_{\theta}\sum_{n=1}^{N}\sum_{k=1}^{K}-\mathbf{Y}_{n,k}\log
F_{\theta}(\mathbf{X}_{n,k})$ (11)
$\displaystyle+\lambda_{m}\sum_{n=1}^{N}\sum_{d=1}^{D}\left(\frac{\partial}{\partial\mathbf{x}_{d}}\mathbbm{1}(\beta-\mathcal{M}_{d})\sum_{k=1}^{K}-\mathbf{Y}_{n,k}\log
F_{\theta}(\mathbf{X}_{n,k})\right)^{2}$
$\displaystyle+\lambda_{j}||J(\mathbf{x})||_{F}^{2}.$

We suggest that the two terms have equal importance to the overall
optimization, and thus set $\lambda_{m}=\lambda_{j}=0.5$ as default
configuration except for otherwise specified. The term
$||J(\mathbf{x})||_{F}^{2}$, i.e., the Jacobian Frobenius norm acts as a
regularizer clipping the values of inputs such that the gradients of
classification logits with respect to the inputs can ensure the linearized
robustness in the presence of perturbed data, and thus we can minimize the
number of mis-predictions of the learned classifier. However, this is
ineffective by directly evaluating each raw input features, which could be
very noisy. Then, the input gradient regularization, as being modulated by
$\lambda_{m}$, can smooth the gradients to be less noisy with fewer extreme
values. Finally, in Eq.(11) we embed the perturbation-based saliency map
($\mathcal{M}_{d}$) to improve the interpretability of a neural network’s
prediction. The indicator function $\mathbbm{1}$ is to determine whether the
saliency for an input feature is below a threshold $\beta$, and thus it
returns 1 if $\beta-\mathcal{M}_{d}\geq 0$ and 0 otherwise. The whole training
procedure of the proposed method is illustrated in Algorithm 1.

##  V Experiments

To evaluate the robustness of the proposed J-SIGR, we conducted experiments on
two image datasets: MNIST [41] and CIFAR-10 [42]. Below, we first describe the
experimental settings and then report the experimental results under a range
of attacks. Finally, we performed ablation studies to provide a more
insightful analysis of the proposed method.

###  V-A Experimental settings

#### Datasets

1) The MNIST dataset [41] consists of handwritten digit $28\times 28$ gray-
scale images that are divided into 60K training and 10K test images. We
trained a CNN, composed of 3 convolutional layers and one final soft-max
layer, to suit the small-sized MNIST. All convolutional blocks have a stride
of 5 while each layer has an increasing number of output channels (i.e., $c$=
64-128-256). 2) The CIFAR-10 [42] dataset contains a collection of $32\times
32\times 3$ colored images that are categorized into 10 classes with 50K
training and 10K test images. We use the ResNet-20 architecture [43] with 20
convolutional layers to train the images from CIFAR-10. Throughout the
network, the kernel size is set to $9\times 9$ in all convolutional layers and
the number of channels is increased from 9, 18 to 36 for the three building
blocks, respectively.

#### Implementations and evaluation metrics

ResNet-20 architecture [43] was used as the backbone for most of the
comparative experiments and ablation studies. We adopted the data augmentation
[43, 15] but without the input normalization. Alternatively, we placed a non-
trainable data normalization layer preceding the DNN to perform the identical
function so that the attack tactics can directly add perturbations into the
natural images. Since our method involves randomness, we reported the accuracy
in the format of mean$\pm$ std with 10 trails to compute the statistical
values.

To measure the robustness of both white and black-box attacks, we tested all
models against adversarial examples generated for each model and reported the
accuracy. On the JSMA setting [30], where the generated adversarial examples
would resemble the targets rather than their original labels, we followed [25]
to adopt a human subject experiment to evaluate the legitimacy of adversarial
example misclassifications.

#### Attacks

To thoroughly evaluate the performance of our proposed method, we considered
multiple powerful attacks. The white-box attacks include FGSM [2] and PGD [3].
FGSM [2] is an efficient single-step adversarial attack scheme. Given a
vectorized input $\mathbf{x}$ and its target label $\mathbf{y}$, FGSM [2]
alters each element of $\mathbf{x}$ along the direction of its gradient with
respect to the inference $\partial F_{\theta}(\mathbf{x})/\partial\mathbf{x}$.
The PGD attack [3], known as one of the strongest $L_{\infty}$ adversarial
example generation algorithms, is a multi-step variant of FGSM [2]. The
iterative update of crafted data $\hat{\mathbf{x}}$ in the $t+1$-th step can
be expressed as
$\hat{\mathbf{x}}^{t+1}=\prod_{\mathbf{x}+S}(\hat{\mathbf{x}}^{t}+\epsilon\cdot\mbox{sgn}(\nabla_{\mathbf{x}}(F_{\theta}(\hat{\mathbf{x}}^{t}),\mathbf{y})))$,
where $\prod$ is the projection space bounded by $\mathbf{x}\pm S$, and
$\epsilon$ is the step size. For the PGD attack [3] on two datasets, $S$ is
set to 0.3/1 and 8/255, and the number of iterations $N_{step}$ is set to 40
and 7, respectively. FGSM [2] adopts the same $\epsilon$ setup as PGD [3]. The
attack configurations of PGD [3] and FGSM [2] were chosen identical to the
setup in many adversarial defense methods [15, 3]. To generate adversarial
examples for JSMA [30], we used the Cleverhans adversarial example library
[32]. We also evaluated the proposed method against several state-of-the-art
black-box transferred attacks: substitute [5], ZOO [6] and transferable
attacks [44].

#### Defenses

To evaluate the improved robustness of our method, we compared it with state-
of-the-art defense models: adversarial training [10], distillation [33] and a
gradient regularization based model [25]. More specifically, for adversarial
training, we trained FGSM [2] with perturbations at $\epsilon=0.3$. For
distillation based defense, we used a soft-max of temperature $T=50$. For the
gradient regularization based model as shown in Eq. (9), we used the double
back-propagation to train the classification model.

###  V-B Robustness under FGSM and PGD attacks

Figure 2: Model accuracy for FGSM examples generated to fool adversarial
training, defensive distillation, and gradient regularization based models
(from left to right).
Figure 3: Defensive model accuracy against PGD attack when applying gradient
regularization as the fool target.

In Fig. 2, we show the robustness results of our method as well as the
performance of other defensive models under the FGSM attack [2] on two
datasets. It can be observed that the gradient-regularized model [25] exhibits
strong robustness to transferred FGSM [2] attack (examples produced by
attacking other models). For example, on the MNIST dataset, the adversarial
examples produced by attacking the defensive distillation can successfully
fool the model based on adversarial training. In contrast, the gradient
regularization based methods (including the proposed J-SIGR) can still
maintain the accuracy. We evaluate the robustness of the gradient
regularization models under a different attack, i.e., PGD [3], and report the
results in Fig. 3 on two datasets. Under this attack, the adversarial examples
are generated to fool a gradient regularized model, while the results of the
two models show that gradient regularization is still effectively robust
against a white-box attack [3]. Interestingly, gradient-regularized models
seem to be vulnerable to white-box attacks, but can still fool all other
models. In this respect, in the presence of adversarially transferred
examples, we hypothesize that gradient regularization is particular not only
for defense but also attack.

TABLE I:  Convergence of gradient regularization with layer-wise Jacobian norm (LW-JN) on the CIFAR-10 dataset with ResNet-20 as backbone. The Jacobian norm is given for lower convolutional layers (Conv1 to Conv$3_{5}$). Test accuracy for perturbed data is computed for the PGD and FGSM attacks. Layer | Vanilla Train | JN+AT | Grad.Reg.+LW-JN
---|---|---|---
Conv1 | 0.004 | 0.157 | 0.155
Conv$2_{0}$ | 0.003 | 0.089 | 0.091
Conv$2_{1}$ | 0.001 | 0.067 | 0.061
Conv$2_{2}$ | 0.002 | 0.055 | 0.057
Conv$2_{3}$ | 0.002 | 0.099 | 0.098
Conv$2_{4}$ | 0.004 | 0.782 | 0.580
Conv$2_{5}$ | 0.004 | 0.422 | 0.333
Conv$3_{0}$ | 0.002 | 0.087 | 0.079
Conv$3_{1}$ | 0.000 | 0.064 | 0.064
Conv$3_{2}$ | 0.003 | 0.072 | 0.069
Conv$3_{3}$ | 0.001 | 0.062 | 0.060
Conv$3_{4}$ | 0.001 | 0.046 | 0.046
Conv$3_{5}$ | 0.000 | 0.022 | 0.021
FC | 0.001 | 0.013 | 0.012
PGD | 0.00 | 0.54 $\pm$ 0.11 | 0.57$\pm$0.10
FGSM | 0.14 | 0.62 $\pm$ 0.10 | 0.66$\pm$0.09

Since our model consists of two robustness mechanisms, we investigated the
impact of Jacobian norm (JN) by disabling double back-propagation and
examining the output response of each layer with respect to two different
attacks. More specifically, a Jacobian-norm based variant of our method was
implemented by adding layer-wise Jacobian norm into the DNN, together with the
input gradient regularization. This variant is called Grad.Reg.+LW-JN. As
shown in Table I, only performing vanilla training using momentum SGD
optimizer can lead to the failure of adversarial defense with the values of
Jacobian norm converging towards negligible values. After applying the JN as a
regularization of the network (i.e., JN+AT), the lower convolutional layers
attain a relatively large JN. The variant of our method with layer-wise
Jacobian norm (i.e., Grad.Reg.+LW-JN) achieves the highest performance with
respect to the two attacks. This demonstrates robustness improvement by
leveraging the proposed network architecture, which is parameterized to resist
perturbations via gradient back-propagation. Since JN can reflect the
robustness of the network, we plotted the evolution curves of the JN values
for the lower convolutional layers to illustrate the robustness of the network
connections (Fig. 4).

Figure 4: Evolution curves of the Jacobian norm computed at each convolutional
layer. Best viewed in color.

###  V-C Robustness under black-box attacks

In this experiment, we evaluate our scheme against three black-box attacks:
transferable adversarial attack [44], Substitute [5] and ZOO attack [6].
Following the setting of [44], two neural networks were trained with their
individual architectures with one network chosen as the source model and the
other chosen as the target model. An adversarial example $\hat{\mathbf{x}}$
generated from the source model was then used to attack the target model
without access to the parameters of the target model. This is denoted as
$Source\rightarrow Target$. We trained two ResNet-18 models (i.e., $M_{1}$ and
$M_{2}$) on CIFAR-10 dataset to attack each other with $M_{1}$ optimized
through PGD adversarial training and $M_{2}$ optimized through our proposed
method. Experimental results are given in Table II. Our proposed method
achieves higher accuracy under two transferable attacks and is seen similar
perturbed-data accuracy for both transferable scenarios. This indicates that
our method provides robustness against transferable black-box attack. This
also shows that the presence of J-SIGR has negligible effect on inference
under a strong attack PGD. For the other two types of attacks, we evaluated
our defense ability on 200 randomly selected test examples for an untargeted
attack. The success rate refers here to the percentage of test samples which
are wrongly classified under the attack. For example, the ZOO attack success
rate for vanilla ResNet-18 with adversarial training is close to 80%. The
results in Table II suggest that our method is robust as it resists the two
attacks by noticeably dropping the success rate from 80% to 49% and 77% to
48.8% under ZOO attack [6] and Substitute [5], respectively (Lower success
rates means higher robustness).

TABLE II: The proposed method (J-SIGR) against transferred attacks on CIFAR-10 test set. Model $M_{1}$ is trained by PGD-AT based on ResNet-18 architecture and $M_{2}$ is trained on ResNet-18 using our method (J-SIGR). Transferable attack | ZOO | Substitute
---|---|---
$M_{1}\rightarrow M_{2}$ | $M2\rightarrow M_{1}$ | success rate (80%) | success rate (77%)
$78.14\pm 0.26$ | $76.82\pm 0.19$ | 49.00 | 48.80
Figure 5: Perturbations by applying JSMA to digits 0 and 1 with maximum
distortion parameter $\gamma=0.25$ for a gradient regularization model. The
highlighted images in each row are modified until the model predicts the digit
corresponding to their column or the maximum distortion is reached.

###  V-D Evaluation on human subject study under Jacobian-saliency map attack
(JSMA)

Unlike other attacks that stop generating adversarial examples when the
maximum distortion is met, JSMA [30] constantly generates adversarial examples
until the model predicts the target. Thus, evaluating the robustness under
JSMA [30] using accuracy numbers is not appropriate. This is also because the
perturbations created by JSMA alter the adversarial examples so they resemble
the target labels instead of the original labels. As shown in Fig. 5, for a
gradient regularized model, we applied JSMA on each image 0 or 1 to generate
perturbations until the model predicts the digit corresponding to their column
target label or the maximum distortion is reached (We set the maximum
distortion $r=0.25$). Then, we followed [25] to test these different
robustness scenarios using 11 human subjects who were invited to evaluate
whether examples generated by different methods are more or less plausible
instances of their targets. Specifically, the subjects were shown 30 images of
JSMA-crafted MNIST examples, with each of these 30 images corresponding to one
original digit (from 0 to 9) and one model (defensive distillation, gradient
regularization and selective input gradient regularization). Images were
randomly and uniformly sampled from a larger set of 45 examples corresponding
to the first 5 images of the original digit in the test set. Images in the
test set were transformed by using JSMA to resemble each of the other 9
digits. Subjects were not provided the original labels and were asked to
identify the most two plausible predictions for the image they believed a
classifier would produce (they entered N/A if they found no label was a good
choice).

Table III shows the quantitative results from the human subject experiment.
Overall, human subjects found that gradient-regularized models can generate
the most convincing examples to fool humans. More specifically, humans mostly
believe gradient-regularized adversarial examples (both the input gradient
regularization [25] and our method) are favourably classified as their target
labels instead of the original digits. For example, the values of “human
fooled” column in Table III show that the mispredictions of gradient
regularized models are very “reasonable” in comparison to adversarial training
and defensive distillation.

TABLE III: Quantitative results form human subject experiment on MNIST dataset. SIGR stands for selective input gradient regularization. The measure“human fooled” records the percentage of examples which are classified by human subjects as the most plausibly adversarial targets (the higher, the better). “Mistake reasonable” measures the percentage of examples which are classified as either the target plausible or unrecognizable as any label (the higher, the better). Best results are in bold. | MNIST (JSMA)
---|---
Model | human fooled | mistake reasonable
Def. Distill | 0.0% | 23.5%
Grad. Reg. | 16.4% | 41.8%
SIGR |  20.2% |  45.1%

###  V-E Comparison with other defensive models for both clean and crafted
data

Most adversarial defense methods focus on corrupted data, while neglecting
clean data. However, a good defensive model should perform well in the
presence of both clean and corrupted data. We thus compared our method with
state-of-the-art methods by evaluating performance for both clean and crafted
data.

Experimental results are reported in Table IV. In previous experiments, we
compared against different defense models that are to date still unbroken
including the PGD based adversarial training [3] and several randomness-based
works [45, 15, 46, 47]. Additionally, we compared against JARN [40], which
utilizes the Jacobians to generate images resembling to the original images.
The compared performance results are shown in Table IV. Note that previous
defense efforts [48] often achieve improved accuracy on contaminated data at
the expense of lowering clean data accuracy. In contrast, we introduced a
notion of linearized robustness which performs well in both clean and
perturbed data. As shown in Table IV, in comparison to the PGD-based
adversarial training, our proposed method outperforms all methods for both
clean and perturbed data accuracy under the white-box attack. For example,
differential privacy (DP) [45], which introduces noises at various locations
of the network so as to guarantee a certified defense, does not perform well
against $L_{\infty}$-norm based attacks, e.g., PGD [3] and FGSM [2]. Moreover,
to pursue a higher level of certified defense, DP dramatically reduces the
clean data accuracy down to 25.1%. A set of noise injection methods, i.e., RSE
[46], Adv-BNN [47] and PNI [15], combine the adversarial training and noise
injection into the inputs/weights of the network. However, these methods,
except for the PNI [15] manually set the noise configurations, making it very
ad-hoc, and thus not generalizable to different datasets. PNI [15] exploits
the min-max optimization with trade-off on clean-and perturbed data by
injecting trainable Gaussian noise on various locations of the network to
generate adversarial examples. Whilst PNI [15] improves the accuracy of both
clean and perturbed data, noise injection is not related to the robustness
response of the network. In contrast, our proposed method regularizes the
Jacobian norm and the input gradients, such that the network parameters can be
dynamically trained to perform better adversarial defense. In addition, the
Jacobian norm regularization explicitly suggests the robustness of the
classification model in response to imperceptible data perturbation.

TABLE IV: Comparison with state-of-the-art defense methods using clean and perturbed data accuracies on CIFAR-10 under PGD attack. Defense method | Clean | PGD
---|---|---
PGD-AT [3] | 87.0 $\pm$ 0.1 | 46.1 $\pm$ 0.1
DP [45] | 87.0 | 25.1
RSE [46] | 87.5 | 40.0
Adv-BNN [47] | 79.7 | 45.4
PNI [15] | 87.1$\pm$ 0.1 | 49.1$\pm$0.3
JARN [40] | 84.8 | 51.8
J-SIGR (Ours) | 90.1 $\pm$ 0.2 | 57.6 $\pm$ 0.4
Figure 6: Visualization of Jacobian matrix of models (PGD-AT and our method)
on CIFAR-10 dataset. Figure 7: Visualization of input gradients on MNIST
dataset. Figure 8: Perturbed data accuracy of CIFAR-10 test set under the PGD
attack versus number of attack steps N-step (top) and attack bound
$\varepsilon$ (bottom).

###  V-F Connections to network interpretability

#### Understanding the Jacobian matrices

An adversarially trained model can gain robustness and also produce salient
Jacobian matrices as byproduct. It has been shown that the saliency in
Jacobians is a result of robustness [28]. Thus, it is interesting to use the
Jacobian saliency to evaluate how robust a model is. In this study, we
visualize the Jacobian matrices of the proposed model and an adversarial-
trained PGD [3] to show how salient the Jacobian map is. As shown in Fig. 6,
the proposed method can better visually resemble the corresponding images than
PGD-AT. This demonstrates the improved robustness of the proposed method.

#### Understanding the input gradients

Fig. 7 visualizes the input gradients across different defensive models on the
MNIST dataset. This qualitative visualization shows the different
interpretability of the input gradients derived from models based on defensive
distillation, adversarial training, gradient regularization and the proposed
selective gradient regularization. The adversarially trained model can provide
more interpretable gradients than defensive distillation, but not as highly
interpretable as gradient regularized models. The proposed method presents the
most interpretable gradients, and thus can provide an explanation for
adversarial attacks.

TABLE V:  The proposed method (J-SIGR) with various architectures on CIFAR-10 test set. Model $M_{1}$ is trained by a PGD based adversarial training (PGD-AI) with ResNet-18 as backbone, and $M_{2}$ is trained on ResNet-18 by our method with AT. | No defense | Vanilla Adv. Train | J-SIGR
---|---|---|---
Model | Clean | PGD | FGSM | Clean | PGD | FGSM | Clean | PGD | FGSM
Net20 | 92.1 | 0.0$\pm$0.0 | 14.1 | 83.8 | 39.1$\pm$0.1 | 46.4 | 90.1$\pm$0.2 | 53.7$\pm$0.3 | 57.6$\pm$0.1
Net32 | 92.8 | 0.0$\pm$0.0 | 17.8 | 85.6 | 42.1$\pm$0.0 | 50.3 | 91.1$\pm$0.2 | 52.8$\pm$0.1 | 54.2$\pm$0.1
Net44 | 93.1 | 0.0$\pm$0.0 | 23.9 | 85.9 | 40.8$\pm$0.1 | 48.2 | 90.0$\pm$0.1 |  55.4$\pm$0.1 |  58.6$\pm$0.2
Net56 | 93.3 | 0.0$\pm$0.0 | 24.2 | 86.5 | 40.1$\pm$0.1 | 48.8 | 92.1$\pm$0.2 | 54.9$\pm$0.2 | 55.8$\pm$0.1
Net20 (1.5$\times$) | 93.5 | 0.0$\pm$0.0 | 15.9 | 85.8 | 42.1$\pm$0.0 | 49.6 | 91.4$\pm$0.1 | 55.2$\pm$0.3 | 55.8$\pm$0.1
Net20 (2$\times$) | 94.0 | 0.0$\pm$0.0 | 13.0 | 86.3 | 43.1$\pm$0.1 | 52.6 | 91.5$\pm$0.1 | 55.1$\pm$0.2 | 55.0 $\pm$0.1

###  V-G Ablation studies

As discussed in Section IV, the Jacobian norm of the network output with
respect to the input relates to the linearized robustness of the DNN. Our
proposed technique introduces tight bounds to the response of the output layer
to adversarial perturbation added to the input. Herein, we raise two concerns
in regards to our proposed regularization term: 1) whether the robustness
improvement introduced by our proposed method is not relying on the stochastic
gradient; 2) how the scale of the network architecture (i.e., width and depth)
affects the robustness of the DNN. Our first evaluation aims to show that our
method is free of gradient obfuscation [49] by increasing the PGD [3] attack
steps and the attack bound $\varepsilon$.

#### Influence of the network capacity

In order to investigate the links between the network capacity (i.e., width,
depth and number of trainable parameters) and the robustness improvement via
J-SIGR, we analyzed various network architectures in terms of depth and width.
For varied depth, we considered ResNet20/32/44/56 and conducted experiments
under vanilla training [3] and our technique. For varied width, we employed
the original ResNet-20 as the baseline and expanded the input/output channel
of each layer by 1.5$\times$ and 2$\times$ scale, respectively. We report both
clean and perturbed data accuracies using the network trained with Jacobian
term. The results in Table V suggest that increasing the model’s capacity
positively improves the network robustness against white-box attacks, and our
proposed method outperforms vanilla training in both clean and perturbed data
accuracy for powerful PGD and FGSM attacks. The other observation is that the
noticeable robustness improvement provided by our method is indeed provided by
the effective training with the proposed J-SIGR. Our method updates the
network parameters without introducing randomness in the test phase.

#### Effect of hyperparameters

In this experiment, we study the effect of two hyperparameters, i.e.,
$\lambda_{m}$ and $\lambda_{j}$ on the MNIST accuracy under FGSM. To de-
correlate the impact of two parameters, we fix $\lambda_{m}=0.5$ and examine
the model accuracy with varied value of $\lambda_{j}$. The results are
reported in Fig. 9. As shown in Fig. 9, the highest accuracy of DNN under an
attack is achieved when we set $\lambda_{j}=0.5$. Thus, we empirically
$\lambda_{m}=\lambda_{j}=0.5$ in all experiments.

#### Non-dependence on stochastic gradients

To prove that the robustness improvement provided by our method is not due to
stochastic gradients, we examine the perturbed data accuracy by evaluating the
PGD attack steps N-step and the attack bound $\varepsilon$. As shown in Fig.
8, increasing the attack steps or attack bound can boost the attack strength,
which inevitably leads to accuracy degradation. However, the accuracy does not
degrade further when N-step=40 or $\varepsilon\geq 0.5$. If the stochastic
gradient was improving robustness, then increasing the attack strength would
have broken the defense of our method. This is not observed in reported
experiments.

Figure 9: The hyperparameter study on MNIST dataset.

##  VI Conclusion

In this paper, we proposed an approach based on Jacobian normalization and
selective input gradient regularization (dubbed J-SIGR) to improve both
robustness and high interpretability of deep neural networks (DNNs). Our
method employs the Jacobian matrices to generate gradient-based salient maps,
which select informative input gradients to achieve efficient interpretability
of adversarial perturbations. We performed ample experiments to show that our
method can improve the robustness of DNNs under multiple adversarial attacks.
We believe our approach could help build trustworthy real-world systems and
benefit the deployment of deep learning in practice.

## References

  * [1] H.-W. Kang and H.-B. Kang, “Prediction of crime occurrence from multi-modal data using deep learning,” _PloS one_ , vol. 12, 2017.
  * [2] C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus, “Intriguing properties of neural networks,” in _ICLR_ , 2014\.
  * [3] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models using resistant to adversarial attacks,” in _ICLR_ , 2018\.
  * [4] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in _IEEE Symposium on Security and Privacy (SP)_ , 2017, pp. 39–57.
  * [5] N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami, “Practical black-box attacks against machine learning,” in _ACM on Asia Conference on Computer and Communication Security_ , 2017, pp. 506–519.
  * [6] P.-Y. Chen, H. Zhang, Y. Sharma, J.-F. Yi, and C.-J. Hsieh, “Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models,” in _10th ACM Workshop on Artificial Intelligence_ , 2017, pp. 15–26.
  * [7] M. A. A. K. Jalwana, N. Akhtar, M. Bennamoun, and A. S. Mian, “Orthogonal deep models as defense against black-box attacks,” _IEEE Access_ , vol. 8, pp. 119 744–119 757, 2020.
  * [8] F. Tramèr, N. Papernot, I. J. Goodfellow, D. Boneh, and P. D. McDaniel, “The space of transferable adversarial examples,” in _arXiv:1704.03453_ , 2017\.
  * [9] N. Akhtar, M. A. A. K. Jalwana, M. Bennamoun, and A. S. Mian, “Attack to fool and explain deep networks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence. DOI: 10.1109/TPAMI.2021.3083769_ , pp. –, 2021.
  * [10] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry, “Adversarial examples are not bugs, they are features,” in _NeurIPS_ , 2019, pp. 125–136.
  * [11] H. Zhang and J. Wang, “Defense against adversarial attacks using feature scattering-based adversarial training,” in _arXiv:1907.10764_ , 2019.
  * [12] A. Shafahi, M. Najibi, A. Ghiasi, Z. Xu, J. Dickerson, C. Studer, L. S. Davis, G. Taylor, and T. Goldstein, “Adversarial training for free,” in _arXiv:1904.12843_ , 2019.
  * [13] H. Kannan, A. Kurakin, and I. Goodfellow, “Adversarial logit pairing,” in _arXiv:1803.06373_ , 2018.
  * [14] F. TramÃ¨r, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel, “Ensemble adversarial training: Attacks and defenses,” in _ICLR_ , 2018.
  * [15] Z. He, A. S. Rakin, and D. Fan, “Parametric noise injection: trainable randomness to improve deep neural network robustness against adversarial attack,” in _CVPR_ , 2020, pp. 588–597.
  * [16] A. Bietti, G. Mialon, and J. Mairal, “On regularization and robustness of deep neural networks,” in _arXiv:1810.00363_ , 2018.
  * [17] A. S. Ross and F. Doshi-Velez, “Improving the adversarial robustness and interpretability of deep neural networks by regularizing input gradients,” in _AAAI_ , 2018.
  * [18] M. Sundararajan, A. Taly, and Q. Yan, “Gradients of counterfactuals,” in _arXiv:1611.02639_ , 2016.
  * [19] D. Smilkov, N. Thorat, B. Kim, F. Viegas, and M. Wattenberg, “Smoothgrad: removing noise by adding noise,” in _ICML Workshop on Visualization for Deep Learning_ , 2017.
  * [20] D. Tsipras, S. Santurkar, L. Engstrom, A. Turner, and A. Madry, “Robustness may be at odds with accuracy,” in _arXiv:1805.12152_ , 2018.
  * [21] C. Etmann, S. Lunz, P. Maass, and C.-B. Schonlieb, “On the connection between adversarial robustness and saliency map interpretability,” in _arXiv:1905.04172_ , 2019.
  * [22] H. Zhang, Y. Yu, J. Jiao, E. P. Xing, L. E. Ghaoui, and M. I. Jordan, “Theoretically principled trade-off between robustness and accuracy,” in _arXiv:1901.08573_ , 2019.
  * [23] C. Qin, J. Martens, S. Gowal, D. Krishnan, A. Fawzi, S. De, R. Stanforth, and P. Kohli, “Adversarial robustness through local linearization,” in _arXiv:1907.02610_ , 2019.
  * [24] A. Nayebi and S. Ganguli, “Biologically inspired protection of deep networks from adversarial attacks,” in _arXiv:1703.09202_ , 2017.
  * [25] A. S. Ross and F. Doshi-Velez, “Improving the adversarial robustness and interpretability of deep neural networks by regularizing their input gradients,” in _AAAI_ , 2018.
  * [26] L. Wan, M. Zeiler, S. Zhang, Y. LeCun, and R. Fergus, “Regularization of neural networks using dropconnect,” in _ICML_ , 2013, pp. 1058–1066.
  * [27] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _Journal of Machine Learning Research_ , vol. 15, pp. 1929–1958, 2014.
  * [28] J. Hoffman, D. A. Roberts, and S. Yaida, “Robust learning with jacobian regularization,” in _arXiv:1908.02729_ , 2019.
  * [29] A. Kurakin, I. Goodfellow, and S. Bengio, “Adversarial examples in the physical world,” in _arXiv:1607.02533_ , 2016, pp. –.
  * [30] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, “The limitations of deep learning in adversarial settings,” in _IEEE European Symposium on Security and Privacy_ , 2016, pp. 372–387.
  * [31] W. Xu, D. Evans, and Y. Qi, “Feature squeezing: detecting adversarial examples in deep neural networks,” in _arXiv:1704.01155_ , 2017.
  * [32] N. Papernot, I. Goodfellow, R. Sheatsley, R. Feinman, and P. McDaniel, “Cleverhans v1.0.0: an adversarial machine learning library,” in _arXiv:1610.00768_ , 2016, pp. –.
  * [33] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, “Distillation as a defense to adversarial perturbations against deep neural networks,” in _IEEE Symposium on Security and privacy_ , 2016, pp. 582–597.
  * [34] A. Rahnama, A. T. Nguyen, and E. Raff, “Robust design of deep neural networks against adversarial attacks based on lyapunov theory,” in _CVPR_ , 2020, pp. 8178–8187.
  * [35] D. Baehrens, T. Schroeter, S. Harmeling, M. Kawanabe, K. Hansen, and K.-R. Mueller, “How to explain individual classification decisions,” _Journal of machine learning research_ , vol. 11, pp. 1803–1831, 2010.
  * [36] H. Drucker and Y. L. Cun, “Improving generalization performance using double back-propagation,” _IEEE Transactions on Neural Networks_ , vol. 3, pp. 991–997, 1992.
  * [37] K. Simonyan, A. Vedaldi, and A. Zisserman, “Deep inside convolutional networks: Visualising image classification models and saliency maps,” in _arXiv:1312.6034_ , 2013, pp. –.
  * [38] J. Xing, T. Nagata, X. Zou, E. Neftci, and J. L. Krichmar, “Policy distillation with selective input gradient regularization for efficient interpretability,” in _arXiv:2205.08685_ , 2022, pp. –.
  * [39] M. Rosynski, F. Kirchner, and M. Valdenegro-Toro, “Are gradient-based saliency maps useful in deep reinforcement learning?” in _NeurIPS 2020 workshop_ , 2020, pp. –.
  * [40] A. Chan, Y. Tay, Y.-S. Ong, and J. Fu, “Jacobian adversarially regularized networks for robustness,” in _ICLR_ , 2020.
  * [41] Y. L. Cun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient based learning applied to document recognition,” _Proceeding of IEEE_ , vol. 86, pp. 2278–2324, 1998.
  * [42] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” in _Technique report, Citeseer_ , 2009.
  * [43] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [44] Y. Liu, X. Chen, C. Liu, and D. Song, “Delving into transferable adversarial examples and black-box attacks,” in _arXiv:1611.02770_ , 2016.
  * [45] M. Lecuyer, V. Atlidakis, R. Geambasu, D. Hsu, and S. Jana, “Certified robustness to adversarial examples with differential privacy,” in _arXiv:1802.03471_ , 2018.
  * [46] X. Liu, M. Cheng, H. Zhang, and C.-J. Hsie, “Towards robust neural networks via random self-ensemble,” in _ECCV_ , 2018.
  * [47] X. Liu, Y. Li, C. Wu, and C.-J. Hsieh, “Adv-bnn: Improved adversarial defense through robust bayesian neural network,” in _ICLR_ , 2019.
  * [48] J. Buckman, A. Roy, C. Raffel, and I. Goodfellow, “Thermometer coding: one hot way to resist adversarial examples,” in _ICLR_ , 2018.
  * [49] A. Athalye, N. Carlini, and D. Wagner, “Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples,” in _arXiv:1802.00420_ , 2018.