# Towards Robust Explanations for
Deep Neural Networks

Ann-Kathrin Dombrowski  Machine Learning Group, Technische Universität Berlin,
Germany  Christopher J. Anders  Machine Learning Group, Technische Universität
Berlin, Germany  BIFOLD - Berlin Institute for the Foundations of Learning and
Data, Berlin, Germany
Klaus-Robert Müller  Machine Learning Group, Technische Universität Berlin,
Germany  Department of Artificial Intelligence, Korea University, Seoul, Korea
Max Planck Institute for Informatics, Saarbrücken, Germany  BIFOLD - Berlin
Institute for the Foundations of Learning and Data, Berlin, Germany  Google
Research, Brain team, Berlin, Germany  Pan Kessel  Machine Learning Group,
Technische Universität Berlin, Germany  BIFOLD - Berlin Institute for the
Foundations of Learning and Data, Berlin, Germany

###### Abstract

Explanation methods shed light on the decision process of black-box
classifiers such as deep neural networks. But their usefulness can be
compromised because they are susceptible to manipulations. With this work, we
aim to enhance the resilience of explanations. We develop a unified
theoretical framework for deriving bounds on the maximal manipulability of a
model. Based on these theoretical insights, we present three different
techniques to boost robustness against manipulation: training with weight
decay, smoothing activation functions, and minimizing the Hessian of the
network. Our experimental results confirm the effectiveness of these
approaches.

Keywords: explanation method, saliency map,adversarial attacks, manipulation,
neural networks

##  1 Introduction

In recent years, deep neural networks have revolutionized many different
areas. Despite their impressive performance, the reasoning behind their
decision processes remains difficult to grasp for humans. This can limit their
usefulness in applications that require transparency. Explanation methods
promise to make neural networks interpretable. In this work we consider
explanations of individual predictions that can be given in terms of
explanation maps [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18, 19, 20] which visualize the importance of each input feature for the
network’s prediction. They give valuable information about relevant features
[21, 22, 23, 24], help us understand what a model has learned [3, 25, 26, 27],
and identify unwanted behavior or biases in the data [7, 28, 29, 30].

While explanation methods show promising results in many areas, concerns
regarding their reliability exist. Recent work has shown that explanations are
sensitive to small perturbations of the input that do not change the
classification result [31]. Furthermore, these perturbations can be
constructed such that an arbitrary target explanation is closely reproduced
and all class scores are approximately unchanged (as opposed to only the
classification result), see [32]. An alternative approach leaves the input
unchanged but manipulates the model such that it has the same output on the
entire data manifold but reproduces an arbitrary target explanation map [33,
34]. The former class of methods is often refered to as input manipulations
and the latter as model manipulations.

Untrustworthy explanations are evidently problematic for various reasons. For
a large number of applications, one is interested in the prediction as well as
in the explanation of a phenomenon. Examples include medical and natural
science applications. As some explanations are susceptible even to random
input perturbations, it seems questionable if much insight can be derived from
inspecting such explanations. In a setting where explanations are legally
required [35], explanation manipulability obviously raises serious concerns as
they cannot be considered trustworthy evidence. An example for this is credit
risk assessment: The supplier can obfuscate that a decision was made based on
racist, sexist or other discriminating features by manipulating the model
[34]. Similarly, attacks from the user side are possible by manipulating the
input as they can create the impression that the decision was based on
unaccepted features and thus subvert the result.

In this paper, we develop methods to make explanations provably more robust
against attacks that manipulate the input. To this end, we provide the
following key contributions:

  * •

We analyze the difference between the original and the manipulated explanation
maps theoretically and provide a unified theoretical framework which allows us
to derive bounds on the maximal change.

  * •

Based on this theoretical framework, we derive several techniques to make
neural networks more resilient against attacks on the explanation, namely:

    * –

regularizing: here, training with weight decay,

    * –

training with smoothed activation functions,

    * –

training while minimizing the Hessian of the network with respect to the
input.

  * •

We demonstrate the effectiveness of the above methods experimentally for
several different explanation methods on the CIFAR-10 data set.

Figure 1 provides an intuition for why explanations are susceptible to
manipulation and how our methods lead to more robust explanations.

Figure 1: Intuition for our approach and results. a The gradient (red arrows)
changes drastically when moving along a line with high curvature but changes
only gradually when the curvature is low. We get a similar effect for networks
with high and low curvature (see c). b We propose several techniques that
reduce curvature when incorporated in the training procedure. Weight decay
flattens the angles between piecewise linear functions, Softplus smooths out
the kinks of the ReLU function, Hessian minimization reduces curvature locally
at the data points. c We show how the Gradient explanation maps change when
adding small perturbations to the input. For the vanilla net the explanation
maps differ strongly. For networks trained with a combination of our proposed
methods the explanation maps become robust to the input perturbations. For a
network trained with weight decay, Softplus activations and Hessian
minimization (last column) the explanation stays approximately constant. d A
quantitative analysis on the complete test set confirms our theoretical
findings. The similarity—measured as Pearson correlation coefficient
(PCC)—between original explanation and explanation of a perturbed input is
significantly higher for networks trained with our methods. We get most
benefits when combining our methods (last network). We show results for three
different noise levels $\nu$.

###  1.1 Related Work

In some works, changing the input is part of the explanation process itself:
[12] averages gradients over noisy inputs, [7] trains an interpretable
(linear) classifier on perturbed samples, [13] and [15] use gradual
interpolation between an input and a rootpoint to create the explanation and
[3] occludes part of the input with a gray square and then tracks the correct
class probability, as a function of the position of the occluded area. Other
studies have used input or model manipulation to compare different explanation
methods: In [5, 36] the pixel flipping method is introduced. Based on the
relevance score of the explanation, pixel values are sequentially replaced and
the effect on the network output is measured. The idea is to verify that the
explanation method accurately identified the relevant pixels—those that the
network needs to correctly classify an image. Variants of this method are also
used in [37] and [38]. [39] takes a similar approach but retrains on the
modified data to avoid effects from artifacts outside of the original data
distribution. Reference [40] proposes “sanity checks” for explanations by
either perturbing the model weights or the labels of the data set and
measuring the resulting change in the explanation. Some explanations turn out
to be visually appealing but insensitive to the model or the data generating
process. In [41], a more theoretical analysis of why many propagation-based
explanations fail the sanity checks introduced in [40] is given. Reference
[42] modifies the input by adding a constant shift, which is then subtracted
in the first layer by changing the bias. The modified network and data set
produce the exact same outputs as the original but some explanation methods
attribute relevance to the shifted pixels.

On the other hand, attention towards more malignant manipulation of
explanations has developed relatively recently. The authors in [33] explicitly
change model weights to manipulate the explanations while keeping the model
output approximately unchanged and [34] expands on this, by analyzing these
experimental results with theoretical insights and providing a method to make
explanation more robust against model manipulations. In [43], a facade model
is added to the original network in the context of Natural Language
Processing. The combined network has similar predictions but the Gradient
Explanation is dominated by the customized facade model. Furthermore, [44]
proposes a similar approach treating the classifier as a blackbox. Their
scaffolding technique can change an explanation of a biased classifiers to
look completely innocuous.

In [31], an approach similar to conventional adversarial attacks on the model
output is presented. The difference to adversarial attacks on the model output
is that attacks on the explanation aim to keep the classification unchanged
while the explanation shows major modifications. The work [32] expands on this
and shows that explanations can be changed to arbitrary target maps by adding
imperceptible perturbations to the input. The authors explain this
susceptibility to manipulations with tools from differential geometry. Our
theoretical analysis extends these results significantly in that it also holds
for a small (but non-vanishing) change in output of the network, for more
general network architectures and various attack methods.

Explanations that include averaging over multiple inputs [12] are naturally
more resilient to input perturbations, but are not completely save from
manipulation [32]. Several works propose to counteract susceptibility of
explanations by changing the explanation process. In [32] ReLU activations are
changed to Softplus activations. This is done for the explanation process only
and not part of the training process, in contrast to the approach taken in the
present work. Reference [45] proposes to include a penalty on the largest
principle curvature in the loss function to train networks that are more
resilient to attacks on the explanation. This is different to our Hessian norm
training which can be roughly understood as a penalty on all principle
curvatures. Furthermore, [34] proposes a projection of the explanation onto
the previously estimated data manifold, [46] shows that combining several
explanation methods can often improve robustness to manipulated inputs, [45]
averages over several examples from a uniform distribution around the input,
and [47] proposes adversarial training to construct black box explanations
like [7, 10] that are robust to input perturbations and distribution shifts.

##  2 Theoretical considerations

In the following, we formally introduce the basic underlying idea of a
theoretical analysis of explanation manipulability. Let us consider gradient
explanations for concreteness. We restrict to the output of the winning class,
i.e. $g(x):=g(x)_{k}$ with $k=\operatorname*{arg\,max}_{i}g(x)_{i}$, since the
gradient method only depends on this component of the output. To manipulate
the explanation of an input $x\in\mathbb{R}^{N}$ of a classifier
$g:\mathbb{R}^{N}\to\mathbb{R}$, we construct an adversarially perturbed input
$x_{\textrm{adv}}=x+\delta x$ such that the output of the network is
(approximately) unchanged, i.e.

$\displaystyle g(x)\approx g(x_{\textrm{adv}})$ (1)

but the corresponding (gradient) explanations $h=\nabla g$ are drastically
different, i.e.

$\displaystyle||h(x)-h(x_{\textrm{adv}})||\gg 1\,.$ (2)

Typically, the perturbation is assumed to be small, $||\delta x||\ll 1$, such
that it is imperceptible. For theoretical analysis, one would like to derive
upper bounds on the change of saliency map $||h(x)-h(x+\delta x)||$ by any
such perturbation $\delta x$. To this end, one considers a curve
$\gamma:\mathbb{R}\to\mathbb{R}^{N}$ with affine parameter $t$ connecting the
unperturbed data point $x$ with its adversarially perturbed counterpart
$x_{\textrm{adv}}$, i.e.

$\displaystyle\gamma(t=-\infty)=x\,,$
$\displaystyle\gamma(t=+\infty)=x_{\textrm{adv}}\,.$ (3)

In practice, intermediate points on the curve may correspond to iterations of
an optimization procedure which adversarially perturbs the input in an
iterative manner (although this interpretation is not needed for any of the
theoretical considerations). One can then use the gradient theorem to rewrite
the change in $j$-th component of the explanation $h$ as111Here we assume that
the classifier $g$ is twice differentiable. However, this assumption can,
under certain circumstances, be relaxed as discussed in Section 3.3.

$\displaystyle h_{j}(x)-h_{j}(x_{\textrm{adv}})$
$\displaystyle=\partial_{j}g(x)-\partial_{j}g(x_{\textrm{adv}})=\int_{\gamma}\sum_{i}\partial_{i}\partial_{j}g(x)\,\mathrm{d}x_{i}$
$\displaystyle=\int_{-\infty}^{\infty}\sum_{i}\partial_{i}\partial_{j}g(\gamma(t))\,\dot{\gamma}_{i}(t)\,\mathrm{d}t\,,$
(4)

Let the Frobenius norm of the Hessian $H_{ij}(g)=\partial_{i}\partial_{j}g$ be
bounded, i.e.

$\displaystyle||H(g)(x)||\leq H^{*}\in\mathbb{R}_{+}\,,$ $\displaystyle\forall
x\in\mathbb{R}^{N}\,.$

It then follows immediately that the maximal change in explanation is also
bounded:

$\displaystyle||h(x)-h(x_{\textrm{adv}})||$
$\displaystyle\leq\int^{+\infty}_{-\infty}||H(g)\,\gamma(t)||\,\mathrm{d}t$
$\displaystyle\leq
H^{*}\int^{+\infty}_{-\infty}||\gamma(t)||\,\mathrm{d}t=H^{*}\,L(\gamma)\,,$
(5)

where $L(\gamma)=\int^{+\infty}_{-\infty}||\gamma(t)||\,\mathrm{d}t$ is the
length of the curve $\gamma$. We have therefore deduced that bounding the
Frobenius norm of the Hessian implies a bound on the maximal possible change
in explanation by input manipulation.

##  3 Methods for robuster explanations

Based on the theoretical analysis in the last section, we propose three
approaches to reduce the Frobenius norm of the Hessian and thereby increase
the robustness with respect to explanation manipulation.

###  3.1 Curvature minimization

As a first approach, we propose to modify the training procedure such that a
small value of the Frobenius norm of the Hessian is part of the objective. To
this end, we add an additional term to the loss function which penalizes the
Frobenius norm, i.e.

$\displaystyle\mathcal{L}=\mathcal{L}_{0}+\zeta\sum_{x\in\mathcal{T}}\left\lVert
H\right\rVert_{F}^{2}(x)\,,$ (6)

where $\zeta$ is a hyperparameter regulating how strongly the Hessian norm is
minimized. Futhermore, $\mathcal{T}$ denotes the training set and
$\mathcal{L}_{0}$ is the unregularized loss function. A related approach has
been previously proposed in [48] in the context of conventional adversarial
attacks.

Calculating the Frobenius norm of the Hessian is expensive, i.e. to obtain the
second derivative we would have to backpropagate through the network once per
input pixel. For larger images, this becomes unfeasible especially when we
want to include the norm minimization in the training procedure.

We therefore propose to estimate the Frobenius norm stochastically. Let
$v\sim\mathcal{N}(0,1)$, which implies that $\mathbb{E}[v_{i}]=0$ and
$\mathbb{E}[v_{i}v_{j}]=\delta_{ij}$.222Here, we use the Kronecker delta
symbol with $\delta_{ij}=\begin{cases}0&i\neq j\,,\\\ 1&i=j\,.\end{cases}$ We
can then rewrite the Frobenius norm of the Hessian as follows

$\displaystyle||H||_{F}^{2}$
$\displaystyle=\sum_{i}\left(\frac{\partial^{2}g}{\partial x_{i}\partial
x_{i}}\right)^{2}$
$\displaystyle=\sum_{i,j}\mathbb{E}\left[v_{i}v_{j}\right]\left(\frac{\partial^{2}g}{\partial
x_{i}\partial x_{j}}\right)^{2}$
$\displaystyle=\mathbb{E}\left[\sum_{i}\left(\frac{\partial}{\partial
x_{i}}\sum_{j}v_{j}\frac{\partial g}{\partial x_{j}}\right)^{2}\right]\,.$

We can estimate the final expectation value by Monte-Carlo, i.e. we draw a
random vector $v$, and compute $v^{T}\nabla g(x)$ at the usual cost of a
single backward pass. Since the resulting expression is a scalar, we can
calculate its derivative at the cost of another single backward pass [49].
Multiple samples can be combined in mini-batches. The average over the mini-
batch is then an unbiased estimator for the expectation value.

###  3.2 Weight decay

The second approach starts from the observation that the Frobenius norm of the
Hessian depends on the weights of the neural network. More precisely, in
Appendix A we show that

###### Theorem 1

Let $g:\mathbb{R}^{N}\to\mathbb{R}$ be a fully-connected neural network with
$L$ layers. The weights of the $l$-th layer are denoted by $W^{(l)}$ and its
activation functions $\sigma$ are twice-differentiable and bounded

$\displaystyle|\sigma^{\prime}(x)|\leq\Sigma_{1}\,,$
$\displaystyle|\sigma^{\prime\prime}(x)|\leq\Sigma_{2}\,.$ (7)

The Hessian of the network is then bounded by

$\displaystyle||H(g)||_{F}\leq\sum_{m=1}^{L}\left(\prod_{l=1}^{m}||W^{(l)}||_{F}^{2}\prod_{l=m+1}^{L}||W^{(l)}||_{F}\right)\,\Sigma_{1}^{L+m-2}\,\Sigma_{2}\,.$
(8)

As a practical consequence of the theorem, we can reduce the maximal possible
change in explanation by decreasing the Frobenius norms of the weights.
Motivated by this theoretical insight, we propose to use weight decay for
training neural networks such that their explanations are more robust to
manipulation. Note while it is well-known that weight decay can improve
generalization of neural networks [50, 51, 52], its effect on the
manipulability of explanations has not previously been established. Other
regularizations that reduce the weight norms ($L^{1}$-regularization, variants
of $L^{2}$-regularization, etc [53]) may have a similar effect.

###  3.3 Smoothing activation functions

As a third approach, we note that the bound of the network’s Hessian (8) also
depends on the maximal values of the activation function’s first and second
derivatives (7). Choosing activations with smaller values for these maximal
values therefore will lead to robuster explanations.

As a concrete example, consider the Softplus activation function

$\displaystyle\sigma(x)=\frac{1}{\beta}\ln(1+e^{\beta x})\,,$ (9)

where $\beta\in\mathbb{R}_{+}$ is a hyperparameter. Its first and second
derivative are bounded by

$\displaystyle|\sigma^{\prime}(x)|\leq 1\,,$
$\displaystyle|\sigma^{\prime\prime}(x)|\leq\frac{1}{4}\beta\,,$ (10)

and thus $\Sigma_{1}=1$ and $\Sigma_{2}=\frac{1}{4}\beta$, see (7). From the
bound (8), it then follows that networks with Softplus non-linearities with
smaller $\beta$ value have robuster explanations compared to networks with
larger values of $\beta$ (provided that the Frobenius norms of the weights is
the same).

We therefore propose to use smoother non-linearities, i.e. functions with
small $\Sigma_{1}$ and $\Sigma_{2}$, to make explanations more robust.

##### Note on ReLU non-linearites

The popular ReLU non-linearity can be recovered from Softplus in the limit
$\beta\to\infty$. Note however that the bound (8) diverges in this limit since
$\Sigma_{2}\to\infty$, see (10). The fundamental underlying difficulty is that
the second derivative $\text{relu}^{\prime\prime}(x)$ is ill-defined at $x=0$.
In Appendix B, we, however, generalize the bound (8) to the case of ReLU non-
linearities. For this, we use the fact that a distributional generalization of
the second derivative of the ReLU non-linearity can be defined, i.e.
$\text{relu}^{\prime\prime}(x)=\delta(x)$ where $\delta$ denotes the Dirac
distribution. The corresponding right-hand-side of this generalized bound only
depends on the weights of the neural network. Thus, this result establishes
theoretically that weight decay also certifiably improves robustness for ReLU
non-linearities.

##  4 Experimental Analysis

###  4.1 Overview

In this section, we compare the performance of the proposed methods
experimentally.

Briefly summarized, we measure the degree of robustness as follows: we perturb
an input sample $x$ by Gaussian noise $\delta x\sim\mathcal{N}(0,\sigma^{2})$.
For the resulting adversarially perturbed input $x_{\textrm{adv}}=x+\delta x$,
we then calculate the explanation $h(x_{\textrm{adv}})$ and measure its
similarity to the original explanation $h(x)$. The standard deviation $\sigma$
is chosen such that the output of the neural network is approximately
unchanged, i.e. $g(x)\approx g(x_{\textrm{adv}})$. We repeat this analysis for
various explanation methods.

In more detail, our experiments use the following setup:

##### Similarity Scores for Explanations

In order to quantify the visual similarity of the explanations, we use three
different measures following [40]: Pearson correlation coefficient (PCC),
structural similarity index measure (SSIM) and mean squared error (MSE). PCC
and SSIM are relative error measures where values close to 1 indicate high
similarity and small values indicate low similarity. MSE is an absolute error
measure where values close to 0 indicate high similarity and large values
indicate low similarity.

##### Model and Dataset

To demonstrate the proposed robustness effects generically, we use the same
convolutional neural network (CNN) architecture for all our models and train
on the CIFAR10 dataset [54]. The models achieve up to $88\%$ test set
accuracy. For more details on the network architecture and training, we refer
to Appendix D.2.

##### Noise Level

We choose the level of noise such that it does not significantly change the
network’s output. To this end, we perturb all 10k images of the test set of
CIFAR10 with Gaussian noise of a given standard deviation $\sigma$. It is
convenient to express the standard deviation $\sigma$ in terms of the noise
level $\nu$ by

$\displaystyle\sigma=(x_{max}-x_{min})\nu\,,$ (11)

where $x_{max}$ and $x_{min}$ denote the maximum and the minimum values of the
input domain. For other types of noise we refer to Appendix D.5.

Figure 2: PCC between explanations drops more rapidly than accuracy when
adding noise with small $\nu$ to the original image. We show mean +/- std for
PCC.

Figure 2 shows the classification accuracy and the PCC similarity score
between the original and adversarially perturbed explanations for different
noise levels $\nu$. Smaller noise levels (between 0.005 and 0.025) lead to a
comparatively mild drop in accuracy but result in a significant reduction in
the similarity of the explanations. We therefore restrict the noise levels to
this interval for our experiments.

##### Explanation Methods

We apply our approach to the explanation methods following [32] :

  * •

Gradient: The map $h(x)=\frac{\partial g}{\partial x}(x)$ quantifies change of
the scores $g(x)$ due to infinitesimal perturbations in each pixel [1, 2].

  * •

Gradient$\times$Input: This method uses the map $h(x)=x\odot\frac{\partial
g}{\partial x}(x)$ [9] which, for linear models, gives the exact contribution
of each pixel to the prediction.

  * •

Integrated Gradients: This method defines the explanation map
$h(x)=(x-\bar{x})\odot\int_{0}^{1}\frac{\partial
g(\bar{x}+t(x-\bar{x}))}{\partial x}\text{d}t$, where $\bar{x}$ is a suitable
baseline [13].

  * •

Guided Backpropagation (GBP): This method is a modification of the Gradient
explanation which blocks negative components of the gradient when
backpropagating through the non-linearities [4].

  * •

Layer-wise Relevance Propagation (LRP) is a framework [5, 15] that applies
specific rules at different layers to propagate relevance backwards through
the network, see [55] for a complete overview over the possible choices for
the propagation rules. We adopt the following conventions: for the output
layer, the relevance is given by

$\displaystyle R^{L}_{i}=\delta_{ik}\,,$ (12)

where $k$ is the index of the predicted class. This is then propagated
backwards through all layers but the first using the $z^{+}$ rule

$\displaystyle
R^{l}_{i}=\sum_{j}\frac{x_{i}^{l}(W^{l})^{+}_{ji}}{\sum_{i}x_{i}^{l}(W^{l})^{+}_{ji}}R^{l+1}_{j}\,,$
(13)

where $(W^{l})^{+}$ denotes the positive weights of the $l$-th layer and
$x^{l}$ is the activation vector of the $l$-th layer. For the first layer, we
use the $z^{\mathcal{B}}$ rule to account for the bounded input domain

$\displaystyle
R^{0}_{i}=\sum_{j}\frac{x_{j}^{0}W^{0}_{ji}-l_{j}(W^{0})^{+}_{ji}-h_{j}(W^{0})^{-}_{ji}}{\sum_{i}(x_{j}^{0}W^{0}_{ji}-l_{j}(W^{0})^{+}_{ji}-h_{j}(W^{0})^{-}_{ji})}R^{1}_{j}\,,$
(14)

where $l_{i}$ and $h_{i}$ are the lower and upper bounds of the input domain
respectively.

For an extensive overview of these methods see [19, 20]. To obtain a pixelwise
relevance score, we sum over absolute values of the three colour channels and
normalize the explanation to have $\sum_{i}|h(x)_{i}|=1$.

###  4.2 Robustness from weight decay

Weight decay adds a regularizing term to the update rule of the network
parameters $w_{i}$ so that large values are penalized. The update is then
given by

$w_{i}\rightarrow w_{i}-\alpha(\frac{\partial\mathcal{L}_{0}}{\partial
w_{i}}+\lambda w_{i})$ (15)

where $\alpha$ is the learning rate and $\mathcal{L}_{0}$ is the unregularized
loss. The hyperparameter $\lambda$ controls how strongly the network
parameters are penalized. We choose five different values for $\lambda$ and
train the CNN for each. Figure 3 shows higher PCC values for larger values of
$\lambda$, i.e. weight decay increases the robustness of explanations with
respect to input manipulation. As was to be expected, there is a trade-off
between robustness and accuracy of the networks. For networks trained with
strong weight decay ($\lambda>$ 1e-2), the accuracy decreases drastically. On
the other hand, networks trained with 5e-5 $\leq\lambda\leq$ 5e-3 achieve
comparable accuracy but are significantly more robust to manipulations than a
network trained with $\lambda=0$.

Figure 3: Left: PCC increases with stronger weight decay (higher $\lambda$).
Therefore, weight decay improves robustness of explanations. We show mean +/-
std for three different noise levels $\nu$. Right: For moderate weight decay
($\lambda\approx$ 5e-4) accuracy increases, while for strong weight decay
($\lambda\geq$ 1e-2) accuracy drops.

###  4.3 Robustness from Softplus

To see how the $\beta$ value of the Softplus activations (9) affects the
robustness, we train networks with four different $\beta$ values. We do this
for all but the largest value of the weight-decay hyperparameter $\lambda$
from the previous section; in total $4\cdot 5=20$ networks. With decreasing
$\beta$ values, the explanations become less prone to input manipulations.
Figure 4 shows the results for networks trained with $\lambda=$ 5e-4 and
different values for $\beta$. For $\beta$ values smaller than 5, the accuracy
of the network decreases slightly. Crucially, comparable accuracy is achieved
for $\beta$ values of 5 and 10. Results for other choices of the weight decay
parameter $\lambda$ look qualitatively similar. We list results for all
combinations in Appendix D.2.

Figure 4: Left: PCC is higher for networks trained with Softplus activation
that have small $\beta$ value. Replacing ReLU with Softplus activations
improves robustness of explanations. We show mean +/- std for three different
noise levels $\nu$. Right: Accuracy decreases if $\beta$ is very small. All
networks were trained with weight decay ($\lambda=$ 5e-4).

###  4.4 Robustness from curvature minimization

To evaluate the effectiveness of Hessian norm minimization, we train networks
with different values of the hyperparameter $\zeta$ which controls the degree
of regularization in the modified loss in Eq. (6).

We approximate the Hessian norm only for Softplus networks since we need to
calculate second derivatives and333More precisely, the second derivative
$\text{relu}^{\prime\prime}(x)$ is not defined for $x=0$ and the relation only
holds up to such root points of the non-linearity.

$\displaystyle\frac{\partial^{2}g}{\partial
x^{2}}\propto\text{relu}^{\prime\prime}=0\,$

for ReLU networks. We consider six different values for $\zeta$ for each of
the networks from the previous section, i.e. we train $6\cdot 20=120$ networks
in total.

Figure 5 shows how curvature minimization affects the robustness against
random perturbations, when using weight decay with $\lambda=$ 5e-4 and
Softplus activations with $\beta=10$. Even a small value for $\zeta$ results
in significant improvement. For larger $\zeta$ values, the PCC value slowly
converges to one. We list results for all combinations of the weight decay
parameter $\lambda$ and the Softplus parameter $\beta$ in Appendix D.2.

Figure 5: Left: PCC is larger for networks trained with strong minimization of
the Hessian norm $\left\lVert H\right\rVert$ (larger $\zeta$ values).
Therefore, minimizing $\left\lVert H\right\rVert$ improves robustness of
explanations. We show mean +/- std for three different noise levels $\nu$.
Right: accuracy decreases when $\zeta$ gets large. All networks were trained
with weight decay ($\lambda=5$e-4) and Softplus activations ($\beta=10$).

Figure 6 shows a concrete example. In the top row, we show an image and
several samples with added Gaussian noise (with noise level $\nu=0.025$).
Below we show the Gradient explanation maps of two different networks. For the
first network (middle row) the explanations appear noisy and vary strongly.
This network was trained without any techniques to enhance robustness (no
weight decay, ReLU activations, no Hessian minimization). For the second
network (bottom row) the explanations stay relatively steady. This network was
trained with measures that enhance robustness (weight decay with $\lambda=$
5e-4, Softplus activations with $\beta=10$, Hessian minimization with $\zeta=$
1e-7).

Figure 6: Top row: original image and image with added noise ($\nu=0.025$).
Middle row: Gradient explanations for a network trained with $\lambda=0$, ReLU
activations and $\zeta=0$. Bottom row: Gradient explanations for a network
trained with weight decay ($\lambda=$ 5e-4), Softplus activations ($\beta=10$)
and Hessian minimization ($\zeta=$ 1e-7). The explanations of the robust
network in the bottom row are clearly more resilient to random input
perturbations.

###  4.5 Other explanation methods

So far we have focused on Gradient explanation maps. But we can apply any
other suitable explanation method to our networks. In Table 1, we show results
for different explanation methods. Specifically, PCC values (averaged over the
complete test set) between original and manipulated explanations when
perturbing images with a noise level of $\nu=0.025$ are listed. In the first
row, we show how the respective explanations change when using the original
network ($\lambda=0$, ReLU activations, $\zeta=0$) and in the second row we
show the values for a network trained with all our robustness measures
($\lambda=5$e-4, $\beta=10$, $\zeta=1$e-6). While the Gradient explanation is
most vulnerable to random perturbations, the results for
Gradient$\times$Input, and Integrated Gradients look qualitatively similar to
the Gradient explanation. When using all our robustifying measures the PCC
similarity between these explanations improves by around 30 to 40 percentage
points. Guided Backpropagation (GBP) and Layerwise Relevance Propagation (LRP)
are noticeably more resilient to random perturbations. However, our robust
network still achieves significantly higher PCC similarities, demonstrating
that even more robust explanation methods can profit. We refer to Appendix D.4
for a more detailed discussion.

Network | Gradient |  Grad$\times$Input |  IntGrad | GBP |  LRP
---|---|---|---|---|---
original | 0.32$\pm$0.12 | 0.44$\pm$0.13 | 0.53$\pm$0.12 | 0.78$\pm$0.10 | 0.91$\pm$0.06
robust | 0.73$\pm$0.08 | 0.76$\pm$0.08 | 0.82$\pm$0.06 | 0.94$\pm$0.03 | 0.98$\pm$0.01
Table 1: PCC (mean $\pm$ std) between original explanations and explanations
of perturbed inputs (noise level $\nu=0.025$) for explanation maps: Gradient,
Gradient$\times$Input, Integrated Gradients, Guided Backpropagation (GBP), and
Layerwise Relevance Propagation (LRP). High PCC values indicate high
robustness.

###  4.6 Comparison of proposed methods

All our proposed methods can improve robustness of explanations against input
manipulations. We observe this trend for all considered explanation methods,
similarity measures and noise levels.

We note that each method appears to improve robustness in a different manner.
As evident from our theoretically-derived upper bound Eq. (8), both weight
decay and small $\beta$ values for the Softplus activations affect the Hessian
norm. Weight decay leads to smaller Hessian norms by minimizing the weight
norms. Replacing ReLU by Softplus with comparatively small $\beta$ parameter
also leads to smaller Hessian norms but the weight norms stay approximately
constant for different $\beta$ values. When minimizing the Hessian norm
directly during training, the Hessian norms decrease significantly while the
weight norms decrease only minimally. This shows that Hessian norm
minimization does not just improve robustness by indirectly minimizing the
weight norms.

While we showed that each method separately improves robustness—we keep the
weight-decay hyperparameter $\lambda$ constant when evaluating different
smoothing parameters $\beta$ for the Softplus activations and we keep the
weight-decay hyperparameter $\lambda$ and the smoothing parameter $\beta$
constant when evaluating different values for hyperparameter $\zeta$ for the
Hessian norm minimization—we get most benefits when combining them. Besides
enhancing robustness, weight decay plays an essential role for the accuracy—as
expected and well-known in the literature [50]; all networks trained without
weight decay stay at an accuracy below 86.5%.

##  5 Conclusion

Explanation methods have gained significant popularity among practitioners in
science and engineering recently. With increased attention to explainable AI,
questions about manipulability and thus trustworthiness of explanations have
been raised. In this contribution, we have addressed the need for robustness
of explanation methods against manipulation of the input data. Rather than
introducing a new explanation method, we focused on enhancing the robustness
of the networks themselves and, as a result, any applied explanation method
was shown to profit.

We could derive bounds for the maximal change in explanation. Based on this
theoretical analysis, we proposed three approaches to increase the robustness
of explanations. Specifically, we show that weight decay can efficiently boost
robustness of explanations. We furthermore propose to use networks with
smoothed activation functions and to include a regularizer for the network’s
curvature in the training process, which leads to significantly enhanced
resilience against manipulated inputs.

An interesting direction for future research will be to relate the established
limits of robust explanation methods to techniques for uncertainty
quantification respectively in relation to methods studying the relevant
structural parts in learning models [56, 57]. Furthermore it will be helpful
to discuss resilience to manipulation of explanation methods also in the
context of unsupervised learning [58, 59, 60] and multi-modal data/similarity
streams [61].

## Declaration of Competing Interest

The authors declare that they have no known competing financial interests or
personal relationships that influence the work reported in this paper.

## Acknowledgments

This work was supported in part by the German Ministry for Education and
Research (BMBF) under Grants 01IS14013A-E, 01GQ1115, 01GQ0850, 031L0207D,
01IS18025A and 01IS18037A. This work was also partly supported by the
Institute of Information & Communications Technology Planning & Evaluation
(IITP) grants funded by the Korea Government (No. 2017-0-00451, Development of
BCI based Brain and Cognitive Computing Technology for Recognizing User’s
Intentions using Deep Learning and No. 2019-0-00079, Artificial Intelligence
Graduate School Program, Korea University), as well as by the Research
Training Group “Differential Equation- and Data-driven Models in Life Sciences
and Fluid Dynamics (DAEDALUS)” (GRK 2433) and Grant Math+, EXC 2046/1, Project
ID 390685689 both funded by the German Research Foundation (DFG). We
gratefully acknowledge helpful comments on the ms by Rodolphe Jenatton.
Correspondence to KRM and PK.

## References

  * [1] David Baehrens, Timon Schroeter, Stefan Harmeling, Motoaki Kawanabe, Katja Hansen, and Klaus-Robert Müller.  How to Explain Individual Classification Decisions.  Journal of Machine Learning Research, 11(61):1803–1831, 2010.
  * [2] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps.  In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Workshop Track Proceedings, 2014.
  * [3] Matthew D. Zeiler and Rob Fergus.  Visualizing and Understanding Convolutional Networks.  In Computer Vision - ECCV 2014 - 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I, pages 818–833, 2014.
  * [4] Jost Tobias Springenberg, Alexey Dosovitskiy, Thomas Brox, and Martin A. Riedmiller.  Striving for Simplicity: The All Convolutional Net.  In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Workshop Track Proceedings, 2015\.
  * [5] Sebastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojciech Samek.  On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation.  PLoS ONE, 10(7):1–46, 07 2015.
  * [6] Ramprasaath R. Selvaraju, Abhishek Das, Ramakrishna Vedantam, Michael Cogswell, Devi Parikh, and Dhruv Batra.  Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 618–626, 2017.
  * [7] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  “Why Should I Trust You?”: Explaining the Predictions of Any Classifier.  In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page 1135–1144, New York, NY, USA, 2016\. Association for Computing Machinery.
  * [8] Luisa M. Zintgraf, Taco S. Cohen, Tameem Adel, and Max Welling.  Visualizing Deep Neural Network Decisions: Prediction Difference Analysis.  In 5th International Conference on Learning Representations (ICLR), Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017\.
  * [9] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje.  Learning Important Features Through Propagating Activation Differences.  In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 3145–3153, 2017.
  * [10] Scott M Lundberg and Su-In Lee.  A Unified Approach to Interpreting Model Predictions.  In Advances in Neural Information Processing Systems 30, pages 4765–4774. Curran Associates, Inc., 2017.
  * [11] Piotr Dabkowski and Yarin Gal.  Real Time Image Saliency for Black Box Classifiers.  In Advances in Neural Information Processing Systems 30, pages 6967–6976. Curran Associates, Inc., 2017.
  * [12] Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda B. Viégas, and Martin Wattenberg.  SmoothGrad: removing noise by adding noise.  In Workshop on Visualization for Deep Learning, International Conference on Machine Learning (ICML) 2017, Sydney, Australia, Aug 10, 2017\.
  * [13] Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic Attribution for Deep Networks.  In Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, pages 3319–3328, 2017.
  * [14] Ruth C Fong and Andrea Vedaldi.  Interpretable explanations of black boxes by meaningful perturbation.  In IEEE International Conference on Computer Vision (ICCV), pages 3449–3457. IEEE, 2017.
  * [15] Grégoire Montavon, Sebastian Lapuschkin, Alexander Binder, Wojciech Samek, and Klaus-Robert Müller.  Explaining nonlinear classification decisions with deep Taylor decomposition.  Pattern Recognition, 65:211–222, 2017.
  * [16] Pieter-Jan Kindermans, Kristof T Schütt, Maximilian Alber, Klaus-Robert Müller, Dumitru Erhan, Been Kim, and Sven Dähne.  Learning how to explain neural networks: PatternNet and PatternAttribution.  In International Conference on Learning Representations (ICLR), Vancouver Convention Center, Vancouver, BC, Canada April 30 – May 3, 2018.
  * [17] Been Kim, Martin Wattenberg, Justin Gilmer, Carrie J. Cai, James Wexler, Fernanda B. Viégas, and Rory Sayres.  Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV).  In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, pages 2673–2682, 2018.
  * [18] Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.  Methods for Interpreting and Understanding Deep Neural Networks.  Digital Signal Processing, 73:1–15, 2018.
  * [19] Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller.  Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, volume 11700.  Springer Nature, 2019.
  * [20] Wojciech Samek, Grégoire Montavon, Sebastian Lapuschkin, Christopher J Anders, and Klaus-Robert Müller.  Toward Interpretable Machine Learning: Transparent Deep Neural Networks and Beyond.  arXiv preprint arXiv:2003.07631, 2020.
  * [21] Irene Sturm, Sebastian Lapuschkin, Wojciech Samek, and Klaus-Robert Müller.  Interpretable deep neural networks for single-trial EEG classification.  Journal of neuroscience methods, 274:141–145, 2016.
  * [22] Farhad Arbabzadah, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek.  Identifying Individual Facial Expressions by Deconstructing a Neural Network.  In German Conference on Pattern Recognition, pages 344–354. Springer, 2016.
  * [23] Kristof T Schütt, Farhad Arbabzadah, Stefan Chmiela, Klaus R Müller, and Alexandre Tkatchenko.  Quantum-chemical insights from deep tensor neural networks.  Nature communications, 8(1):13890, 2017.
  * [24] Miriam Hägele, Philipp Seegerer, Sebastian Lapuschkin, Michael Bockmayr, Wojciech Samek, Frederick Klauschen, Klaus-Robert Müller, and Alexander Binder.  Resolving challenges in deep learning-based analyses of histopathological images using explanation methods.  Scientific reports, 10(1):6423, 2020.
  * [25] Tom Zahavy, Nir Ben Zrihem, and Shie Mannor.  Graying the Black Box: Understanding DQNs.  In Proceedings of the 33rd International Conference on International Conference on Machine Learning (ICML) - Volume 48, page 1899–1908, 2016.
  * [26] Leila Arras, Franziska Horn, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek.  “What is relevant in a text document?”: An interpretable machine learning approach.  PloS one, 12(8):e0181142, 2017.
  * [27] Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern.  Visualizing and Understanding Atari Agents.  In International Conference on Machine Learning, pages 1792–1801. PMLR, 2018.
  * [28] Sebastian Lapuschkin, Alexander Binder, Grégoire Montavon, Klaus-Robert Muller, and Wojciech Samek.  Analyzing classifiers: Fisher vectors and deep neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2912–2920, 2016.
  * [29] Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.  Unmasking Clever Hans predictors and assessing what machines really learn.  Nature communications, 10(1):1096, 2019.
  * [30] Christopher J Anders, Talmaj Marinč, David Neumann, Wojciech Samek, Klaus-Robert Müller, and Sebastian Lapuschkin.  Analyzing ImageNet with Spectral Relevance Analysis: Towards ImageNet un-Hans’ed.  arXiv preprint arXiv:1912.11425, 2019.
  * [31] Amirata Ghorbani, Abubakar Abid, and James Y. Zou.  Interpretation of Neural Networks Is Fragile.  In The 33rd Conference on Artificial Intelligence, AAAI, pages 3681–3688, 2019.
  * [32] Ann-Kathrin Dombrowski, Maximillian Alber, Christopher Anders, Marcel Ackermann, Klaus-Robert Müller, and Pan Kessel.  Explanations can be manipulated and geometry is to blame.  In Advances in Neural Information Processing Systems, pages 13567–13578, 2019.
  * [33] Juyeon Heo, Sunghwan Joo, and Taesup Moon.  Fooling Neural Network Interpretations via Adversarial Model Manipulation.  In Advances in Neural Information Processing Systems, pages 2921–2932, 2019.
  * [34] Christopher Anders, Plamen Pasliev, Ann-Kathrin Dombrowski, Klaus-Robert Müller, and Pan Kessel.  Fairwashing Explanations with Off-Manifold Detergent.  In Proceedings of the 37th International Conference on Machine Learning, ICML, Vienna, Austria, PMLR 119, 2020.
  * [35] Parliament and Council of the European Union.  Article 22. Automated individual decision making, including profiling.  Official Journal of the European Union, 2016.
  * [36] Wojciech Samek, Alexander Binder, Gregoire Montavon, Sebastian Lapuschkin, and Klaus-Robert Müller.  Evaluating the visualization of what a Deep Neural Network has learned.  IEEE Transactions on Neural Networks and Learning Systems, 28:2660–2673, 11 2017.
  * [37] Leila Arras, Grégoire Montavon, Klaus-Robert Müller, and Wojciech Samek.  Explaining recurrent neural network predictions in sentiment analysis.  In Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 159–168, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
  * [38] Richard Tomsett, Dan Harborne, Supriyo Chakraborty, Prudhvi Gurram, and Alun Preece.  Sanity Checks for Saliency Metrics.  Proceedings of the AAAI Conference on Artificial Intelligence, (04):6021–6029, Apr. 2020.
  * [39] Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim.  A Benchmark for Interpretability Methods in Deep Neural Networks.  In Advances in Neural Information Processing Systems, pages 9734–9745, 2019.
  * [40] Julius Adebayo, Justin Gilmer, Michael Muelly, Ian J. Goodfellow, Moritz Hardt, and Been Kim.  Sanity Checks for Saliency Maps.  In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems, NeurIPS 2018, 3-8 December, Montréal, Canada., pages 9525–9536, 2018.
  * [41] Leon Sixt, Maximilian Granz, and Tim Landgraf.  When Explanations Lie: Why Many Modified BP Attributions Fail.  In Proceedings of the 37th International Conference on Machine Learning, ICML, Vienna, Austria, PMLR 119, 2020.
  * [42] Pieter-Jan Kindermans, Sara Hooker, Julius Adebayo, Maximilian Alber, Kristof T. Schütt, Sven Dähne, Dumitru Erhan, and Been Kim.  The (Un)reliability of Saliency Methods.  In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning, pages 267–280. Springer, 2019.
  * [43] Junlin Wang, Jens Tuyls, Eric Wallace, and Sameer Singh.  Gradient-based Analysis of NLP Models is Manipulable.  arXiv preprint arXiv:2010.05419, 2020.
  * [44] Dylan Slack, Sophie Hilgard, Emily Jia, Sameer Singh, and Himabindu Lakkaraju.  Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods.  In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, pages 180–186, 2020.
  * [45] Zifan Wang, Haofan Wang, Shakul Ramkumar, Piotr Mardziel, Matt Fredrikson, and Anupam Datta.  Smoothed Geometry for Robust Attribution.  Advances in Neural Information Processing Systems, 33, 2020.
  * [46] Laura Rieger and Lars Kai Hansen.  A simple defense against adversarial attacks on heatmap explanations.  2020 Workshop on Human Interpretability in Machine Learning (WHI), 2020.
  * [47] Himabindu Lakkaraju, Nino Arsov, and Osbert Bastani.  Robust and stable black box explanations.  In Proceedings of the 37th International Conference on Machine Learning, ICML, Vienna, Austria, PMLR 119, 2020.
  * [48] Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Jonathan Uesato, and P. Frossard.  Robustness via Curvature Regularization, and Vice Versa.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9070–9078, 2019.
  * [49] Barak A. Pearlmutter.  Fast Exact Multiplication by the Hessian.  Neural Computation, 6(1):147–160, January 1994.
  * [50] Anders Krogh and John A Hertz.  A Simple Weight Decay Can Improve Generalization.  In Advances in neural information processing systems, pages 950–957, 1992.
  * [51] Stephen Hanson and Lorien Pratt.  Comparing biases for minimal network construction with back-propagation.  Advances in neural information processing systems, 1:177–185, 1988\.
  * [52] Andreas S Weigend, David E Rumelhart, and Bernardo A Huberman.  Generalization by Weight-Elimination with Application to Forecasting.  In Advances in neural information processing systems, pages 875–882, 1991.
  * [53] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep Learning.  MIT Press, 2016.  http://www.deeplearningbook.org.
  * [54] Alex Krizhevsky.  Learning Multiple Layers of Features from Tiny Images, CIFAR10 dataset.  Technical report, 2009.
  * [55] Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojciech Samek, and Klaus-Robert Müller.  Layer-Wise Relevance Rropagation: An Overview.  In Explainable AI: interpreting, explaining and visualizing deep learning, pages 193–209. Springer, 2019.
  * [56] Mikio L. Braun, Joachim M. Buhmann, and Klaus-Robert Müller.  On Relevant Dimensions in Kernel Feature Spaces.  Journal of Machine Learning Research, 9(62):1875–1908, 2008.
  * [57] Grégoire Montavon, Mikio L. Braun, and Klaus-Robert Müller.  Kernel Analysis of Deep Networks.  Journal of Machine Learning Research, 12(78):2563–2581, 2011.
  * [58] Jacob Kauffmann, Malte Esders, Grégoire Montavon, Wojciech Samek, and Klaus-Robert Müller.  From clustering to cluster explanations via neural networks.  arXiv preprint arXiv:1906.07633, 2019.
  * [59] Jacob Kauffmann, Klaus-Robert Müller, and Grégoire Montavon.  Towards explaining anomalies: a deep Taylor decomposition of one-class models.  Pattern Recognition, 101:107198, 2020.
  * [60] Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller.  A Unifying Review of Deep and Shallow Anomaly Detection.  arXiv preprint arXiv:2009.11732, 2020.
  * [61] Oliver Eberle, Jochen Büttner, Florian Kräutli, Klaus-Robert Müller, Matteo Valleriani, and Grégoire Montavon.  Building and Interpreting Deep Similarity Models.  arXiv preprint arXiv:2003.05431, 2020.

Appendix

###### Contents

  1. 1 Introduction
    1. 1.1 Related Work
  2. 2 Theoretical considerations
  3. 3 Methods for robuster explanations
    1. 3.1 Curvature minimization
    2. 3.2 Weight decay
    3. 3.3 Smoothing activation functions
  4. 4 Experimental Analysis
    1. 4.1 Overview
    2. 4.2 Robustness from weight decay
    3. 4.3 Robustness from Softplus
    4. 4.4 Robustness from curvature minimization
    5. 4.5 Other explanation methods
    6. 4.6 Comparison of proposed methods
  5. 5 Conclusion
  6. A Proof of Theorem 1
  7. B Relu networks
    1. B.1 Toy example
    2. B.2 General case
  8. C Interchangeability of Softplus $\beta$
    1. C.1 Interchangeability of Softplus $\beta$
    2. C.2 Examples
  9. D Experimental Analysis
    1. D.1 Error measures
    2. D.2 Network structure
    3. D.3 Gradient Explanation
    4. D.4 Other Explanation Methods
    5. D.5 Other types of noise
      1. D.5.1 Laplace noise
      2. D.5.2 Salt-pepper noise
    6. D.6 Connection between Hessian norms and weight norms
  10. E Hessian norm approximation

##  Appendix A Proof of Theorem 1

Let $\sigma(x)$ denote the non-linearity of the network. We also use the
notation $\sigma^{(l)}(x)=\sigma(W^{(l)}x)$ where $W^{(l)}$ are the weights of
layer $l$. By assumption, the activation functions $\sigma$ are twice-
differentiable and bounded

$\displaystyle|\sigma^{\prime}(x)|\leq\Sigma_{1}\,,$
$\displaystyle|\sigma^{\prime\prime}(x)|\leq\Sigma_{2}\,.$ (16)

The activation at layer $L$ is then given by

$\displaystyle a^{(L)}(x)=(\sigma^{(L)}\circ\dots\circ\sigma^{(1)})(x)$ (17)

Its derivative $\partial_{k}a^{(l)}_{i}$ is equal to

$\displaystyle\sum_{s_{2}\dots
s_{l}}W^{(l)}_{is_{l}}\sigma^{\prime}\left(\sum_{j}W^{(l)}_{ij}a^{(l-1)}_{j}\right)W^{(l-1)}_{s_{l}s_{l-1}}\sigma^{\prime}\left(\sum_{j}W^{(l-1)}_{s_{l}j}a^{(l-2)}_{j}\right)$
$\displaystyle\hfill\dots
W^{(1)}_{s_{2}k}\sigma^{\prime}\left(\sum_{j}W^{(1)}_{s_{2}j}x_{j}\right)\,.$

We therefore obtain

$\displaystyle\left\|\nabla
a^{(l)}\right\|_{F}\leq\left(\Sigma_{1}\right)^{l}\,\prod_{i=1}^{l}\left\|W^{(i)}\right\|_{F}$
(18)

From the expression for $\partial_{k}a^{(l)}_{i}$, we can straightforwardly
derive that

$\displaystyle\partial_{l}\partial_{k}a^{(L)}_{i}=$
$\displaystyle\sum_{m}\sum_{s_{2}\dots s_{L}}\big{\\{}$ $\displaystyle
W^{(L)}_{is_{L}}\sigma^{\prime}\left(\sum_{j}W^{(L)}_{ij}a^{(L-1)}_{j}\right)W^{(L-1)}_{s_{L}s_{L-1}}\sigma^{\prime}\left(\sum_{j}W^{(L-1)}_{s_{L}j}a^{(L-2)}_{j}\right)$
$\displaystyle\dots\sum_{p}W^{(m)}_{s_{m+1}p}W^{(m)}_{s_{m+1}s_{m}}\sigma^{\prime\prime}\left(\sum_{j}W^{(m)}_{s_{m+1}j}a^{(m-1)}_{j}(x)\right)\partial_{l}a^{(m-1)}_{p}(x)$
$\displaystyle\dots
W^{(1)}_{s_{2}k}\sigma^{\prime}\left(\sum_{j}W^{(1)}_{s_{2}j}x_{j}\right)\big{\\}}\,.$

Restrict to the case for which the index $i$ only takes a single value, the
Hessian $H_{ij}(g)=\partial_{i}\partial_{j}a^{L}(x)$ is then bounded by

$\displaystyle\left\|H(g)\right\|_{F}\leq\sum_{m=1}^{L}\left(\prod_{l=1}^{m}||W^{(l)}||_{F}^{2}\prod_{l=m+1}^{L}||W^{(l)}||_{F}\right)\,\Sigma_{1}^{L+m-2}\,\Sigma_{2}\,.$
(19)

##  Appendix B Relu networks

As was discussed in the main text, for Softplus non-linearites the bound (8)
diverges for ReLU non-linearities. This is because ReLU can be obtained from
Softplus by taking the limit $\beta\to\infty$ and the constant $\Sigma_{2}$ in
(8) diverges in this limit, see (10). The underlying fundamental difficulty is
that the Hessian of ReLU networks is not well-defined.

In the following, we will discuss how to generalize the analysis to networks
with ReLU activations. We will establish that a distributional generalization
of the Hessian can be derived for ReLU networks. A distributional form of the
Hessian is sufficient for our purposes because in deriving a bound for the
maximal change in explanation we only need to consider the Hessian under an
integral, see (4). Since integrals over distributions are well-defined, the
resulting expression will be well-defined as well.

In the following, we will first illustrate this for a simple toy model before
considering the general case.

###  B.1 Toy example

Consider the network (depicted in Figure 8)

$g(x)=\text{relu}(w^{(1)T}x)+\text{relu}(w^{(2)T}x)$ (20)

with input vector $x\in\operatorname*{\mathbb{R}}^{2}$ and weight vectors
$w^{(1)}=\frac{1}{\sqrt{2}}[1,1]^{T}$ and
$w^{(2)}=\frac{1}{\sqrt{2}}[1,-1]^{T}$.

Figure 7: Toy function. Lines of rootpoints are marked in green and red. The
grey area shows where $g(x)=0$

Figure 8: Path (in blue) along an equipotential line (constant network output
$g(x)=\text{const}$)

The first derivative with respect to $x_{j}$ is

$\partial_{j}g(x)=w^{(1)}_{j}\theta(w^{(1)T}x)+w^{(2)}_{j}\theta(w^{(2)T}x)\,,$
(21)

where we have defined the Heaviside step function

$\displaystyle\theta(x)=\begin{cases}1&x\geq 0\,,\\\ 0&x<0\,.\end{cases}$ (22)

We note that the derivative of the step function is not well-defined at zero.
However, a distributional generalization thereof can be defined, i.e.

$\displaystyle\theta^{\prime}(x)=\delta(x)\,,$ (23)

where $\delta$ denotes the Dirac delta distribution.

With this definition, the $ij$-th entry of (the distributional generalization
of) the Hessian matrix can formally be written as

$\partial_{i}\partial_{j}g(x)=w^{(1)}_{i}w^{(1)}_{j}\delta(w^{(1)T}x)+w^{(2)}_{i}w^{(2)}_{j}\delta(w^{(2)T}x)\,.$
(24)

By (4), the change in (gradient) explanation when moving from point $x$ to
$x_{\textrm{adv}}$ is then given by

$(h(x)-h(x_{\textrm{adv}}))_{j}=\int_{-\infty}^{\infty}\sum_{i}\left(w^{(1)}_{i}w^{(1)}_{j}\delta(w^{(1)T}x)+w^{(2)}_{i}w^{(2)}_{j}\delta(w^{(2)T}x)\right)\dot{x}_{i}\,\mathrm{d}t\,,$
(25)

where we have used the notation $x(t)$ for the curve connecting the
unperturbed and perturbed data points.

For integrating over the delta distribution in composition with a (scalar-
valued) function, we use

$\int_{-\infty}^{\infty}f(t)\delta(y(t))\mathrm{d}t=\sum_{t_{N}}\frac{f(t_{N})}{\left\lvert
y^{\prime}(t_{N})\right\rvert}$ (26)

with $t_{N}$ being the roots of $y(t)$. Using this expression, we then obtain
the following change in saliency map

$\displaystyle(h(x)-h(x_{\textrm{adv}}))_{j}$
$\displaystyle=\sum_{t_{N}}\frac{\sum_{i}w^{(1)}_{i}w^{(1)}_{j}\dot{x}_{i}}{\left\lvert\sum_{i}w^{(1)}_{i}\dot{x}_{i}\right\rvert}+\sum_{t_{N}}\frac{\sum_{i}w^{(2)}_{i}w^{(2)}_{j}\dot{x}_{i}}{\left\lvert\sum_{i}w^{(2)}_{i}\dot{x}_{i}\right\rvert}$
$\displaystyle=\sum_{t_{N}}\mathrm{sgn}\left(\sum_{i}w^{(1)}_{i}\dot{x}_{i}\right)w^{(1)}_{j}+\sum_{t_{N}}\mathrm{sgn}\left(\sum_{i}w^{(2)}_{i}\dot{x}_{i}\right)w^{(2)}_{j}\,.$

Consider the blue path in Figure 8 whose root points are denoted by $t_{1}$
and $t_{2}$. We note that these root points correspond to kinks in the curve
$x(t)$ connecting the unperturbed and perturbed data point. Their
corresponding normalized velocity vectors are given by
$\dot{x}(t_{1})=w^{(2)}$ and $\dot{x}(t_{2})=(0,-1)^{T}$ respectively. We
therefore obtain

$\displaystyle h(x)-h(x_{\textrm{adv}})$
$\displaystyle=\mathrm{sgn}\left(\langle
w^{(1)},\dot{x}(t_{2})\rangle\right)w^{(1)}+\mathrm{sgn}\left(\langle
w^{(2)},\dot{x}(t_{1})\rangle\right)w^{(2)}$
$\displaystyle=\mathrm{sgn}\left(-\frac{1}{\sqrt{2}}\right)w^{(1)}+\mathrm{sgn}\left(1\right)w^{(2)}$
$\displaystyle=w^{(2)}-w^{(1)}$

which is correct as $h(x)=w^{(2)}$ and $h(x_{\textrm{adv}})=w^{(1)}$. It is
important to stress that we have obtain this result despite the fact that the
Hessian of the neural network $g$ is only given in generalized distributional
form.

###  B.2 General case

The argument of the previous section can be generalized to arbitrary fully-
connected networks with weights $W^{l}$ of layer $l\in\\{1,\dots,L\\}$. The
general logic follows closely the toy model discussed in the previous section,
i.e. a distributional generalization of the Hessian is derived and since on
the right-hand-side of (4) the Hessian only appears under an integral, this
distributional form is sufficient to obtain a bound on the maximal change in
explanation due to a perturbation of the input. Using this technique, we
derive the following theorem:

###### Theorem 2

Let $x$ and $x_{\textrm{adv}}=x+\delta x$ denote the unperturbed and perturbed
data points respectively. We denote by $x(t)$ the curve connecting the
unperturbed and perturbed points, i.e. $x(t=-\infty)=x$ and
$x(t=+\infty)=x_{\textrm{adv}}$. Furthermore, we assume that all points on the
curve have the same network output, i.e. $g(x(t_{1}))=g(x(t_{2}))$ for all
$t_{1},t_{2}\in\mathbb{R}$. The maximal change of explanation is then given by

$\displaystyle\left||h(x)-h(x_{\textrm{adv}})\right||^{2}\leq\sum_{\textrm{kinks}(x(t))}\left(\prod_{l=1}^{L}\left\|W^{(l)}\right\|_{F}^{2}\right)\,,$
(27)

where the sum runs over all kinks of the curve $x(t)$.

We can give an intuition for the theorem by considering the blue curve of
Figure 8 for the toy model of the previous section. In this case, the sum over
the kinks would run over $x(t_{1})$ and $x(t_{2})$, see Figure 8. Only at
these kinks, the gradient of the network will change. In the theorem, we then
estimate this change by its maximal value, i.e. the change is equal to the
product of all weights.

As a practical consequence of the theorem, we can make explanations more
robust by weight decay also in the case of ReLU non-linearities.

##### Proof:

Let $W^{(l)}$ be the weights of layer $l$. We denote the $l$-th layer by
$\text{relu}^{(l)}(x)=\text{relu}(W^{(l)}x)$. It then follows that

$\displaystyle\partial_{k}\,\text{relu}\left(\sum_{j}W_{ij}x_{j}\right)=W_{ik}\,\theta\left(\sum_{j}W_{ij}x_{j}\right)$
(28)
$\displaystyle\partial_{l}\,\theta\left(\sum_{j}W_{ij}x_{j}\right)=W_{il}\,\delta\left(\sum_{j}W_{ij}x_{j}\right)$
(29)

where $\theta$ and $\delta$ are the Heaviside step function and the delta
distribution respectively. The activation at layer $L$ is then given by

$\displaystyle
a^{(L)}(x)=(\text{relu}^{(L)}\circ\dots\circ\text{relu}^{(1)})(x)$ (30)

Its derivative $\partial_{k}a^{(L)}_{i}$ is equal to

$\displaystyle\sum_{s_{2}\dots
s_{L}}W^{(L)}_{is_{L}}\theta\left(\sum_{j}W^{(L)}_{ij}a^{(L-1)}_{j}\right)W^{(L-1)}_{s_{L}s_{L-1}}\theta\left(\sum_{j}W^{(L-1)}_{s_{L}j}a^{(L-2)}_{j}\right)$
$\displaystyle\hfill\dots
W^{(1)}_{s_{2}k}\theta\left(\sum_{j}W^{(1)}_{s_{2}j}x_{j}\right)$

Deriving this expression for $\partial_{k}a^{(L)}_{i}$ again, we obtain

$\displaystyle\partial_{l}\partial_{k}a^{(L)}_{i}=$
$\displaystyle\sum_{m}\sum_{s_{2}\dots s_{L}}\big{\\{}$ $\displaystyle
W^{(L)}_{is_{L}}\theta\left(\sum_{j}W^{(L)}_{ij}a^{(L-1)}_{j}\right)W^{(L-1)}_{s_{L}s_{L-1}}\theta\left(\sum_{j}W^{(L-1)}_{s_{L}j}a^{(L-2)}_{j}\right)$
$\displaystyle\dots\sum_{p}W^{(m)}_{s_{m+1}p}W^{(m)}_{s_{m+1}s_{m}}\delta\left(\sum_{j}W^{(m)}_{s_{m+1}j}a^{(m-1)}_{j}(x)\right)\partial_{l}a^{(m-1)}_{p}(x)$
$\displaystyle\dots
W^{(1)}_{s_{2}k}\theta\left(\sum_{j}W^{(1)}_{s_{2}j}x_{j}\right)\big{\\}}$

We now restrict to the case that $a^{(L)}$ has only a single output value. As
a result, the index $i$ in the expression above only takes one value, i.e.
$i=1$. We define $g(x)=a^{(L)}_{1}(x)$ to ease notation. We then substitute
this expression for
$\partial_{l}\partial_{k}g=\partial_{l}\partial_{k}a^{(L)}_{1}$ in (4) and
obtain

$\displaystyle\left(h(x)-h(x_{\textrm{adv}})\right)_{k}$
$\displaystyle=\sum_{m}\sum_{s_{2}\dots
s_{L}}\int_{-\infty}^{\infty}\mathrm{d}t\,\Big{\\{}$ $\displaystyle
W^{(L)}_{1s_{L}}\theta\left(\sum_{j}W^{(L)}_{ij}a^{(L-1)}_{j}\right)W^{(L-1)}_{s_{L}s_{L-1}}\theta\left(\sum_{j}W^{(L-1)}_{s_{L}j}a^{(L-2)}_{j}\right)$
$\displaystyle\dots\sum_{\hat{s}_{m}}W^{(m)}_{s_{m+1}\hat{s}_{m}}W^{(m)}_{s_{m+1}s_{m}}\delta\left(\sum_{j}W^{(m)}_{s_{m+1}j}a^{(m-1)}_{j}(x)\right)\dot{a}^{(m-1)}_{\hat{s}_{m}}(x)$
$\displaystyle\dots
W^{(1)}_{s_{2}k}\theta\left(\sum_{j}W^{(1)}_{s_{2}j}x_{j}\right)\Big{\\}}\,,$

where we have used the notation $\partial_{t}a^{(m-1)}=\dot{a}^{(m-1)}$ for
notational simplicity. Using the identity (26), we then obtain

$\displaystyle\left(h(x)-h(x_{\textrm{adv}})\right)_{k}$
$\displaystyle=\sum_{m}\sum_{x_{N}^{m}}\sum_{s_{2}\dots s_{L}}\,\Big{\\{}$
$\displaystyle
W^{(L)}_{1s_{L}}\theta\left(\sum_{j}W^{(L)}_{ij}a^{(L-1)}_{j}\right)W^{(L-1)}_{s_{L}s_{L-1}}\theta\left(\sum_{j}W^{(L-1)}_{s_{L}j}a^{(L-2)}_{j}\right)$
$\displaystyle\dots
W^{(m)}_{s_{m+1}s_{m}}\text{sgn}\left(\sum_{j}W^{(m)}_{s_{m+1}j}\dot{a}^{(m-1)}_{j}(x^{m}_{N})\right)$
$\displaystyle\dots
W^{(1)}_{s_{2}k}\theta\left(\sum_{j}W^{(1)}_{s_{2}j}(x^{m}_{N})_{j}\right)\Big{\\}}\,,$

where the sum over $x^{m}_{N}$ runs over all zeropoints of
$\sum_{j}W^{(m)}_{s_{m+1}j}a^{(m-1)}$ along the trajectory connecting $x$ with
$x_{\textrm{adv}}$. Using the fact that $|\theta(\bullet)|\leq 1$ and
$|\text{sgn}(\bullet)|\leq 1$, we obtain

$\displaystyle\left||h(x)-h(x_{\textrm{adv}})\right||^{2}\leq\sum_{m}\sum_{x_{N}^{m}}\left\|W^{(L)}\right\|_{F}^{2}\left\|W^{(L-1)}\right\|_{F}^{2}\dots\left\|W^{(m)}\right\|_{F}^{2}\dots\left\|W^{(1)}\right\|_{F}^{2}\,.$
(31)

As in the case of the toymodel, the summands run over all kinks of the
trajectory. This bound for ReLU networks depends purely on the network weights
and the number of kinks passed when moving from $x$ to $x_{\textrm{adv}}$. If
we reduce the Frobenius norms of the weights, we also reduce the maximal
possible change in explanation.

##  Appendix C Interchangeability of Softplus $\beta$

###  C.1 Interchangeability of Softplus $\beta$

When training Softplus networks with different $\beta$ values it is
interesting to consider how they differ as the beta values partially cancel
out or can be absorbed into the weights and biases.

The Softplus function is defined as:

$\text{sp}_{\beta}(x)=\frac{1}{\beta}\ln{(1+\text{e}^{\beta x})}$ (32)

Therefore, we can relate two Softplus functions with different $\beta$ values
$\beta_{1}$ and $\beta_{2}$ as follows:

$\text{sp}_{\beta_{1}}(x)=\frac{\beta_{2}}{\beta_{1}}\text{sp}_{\beta_{2}}(\frac{\beta_{1}}{\beta_{2}}x)$
(33)

A network consisting of linear layers and Softplus activations with
$\beta=\beta_{1}$ has weights $W^{(i)}$ and biases $b^{(i)}$. We can define a
network with the same structure but a different Softplus $\beta=\beta_{2}$ and
weights $\tilde{W}^{(i)}$ and biases $\tilde{b}^{(i)}$. The networks give
identical outputs for all inputs if we define the weights and biases of the
second network in the following way:

$\displaystyle\tilde{W}^{(1)}$
$\displaystyle=\frac{\beta_{1}}{\beta_{2}}W^{(1)}$
$\displaystyle\tilde{W}^{(i)}$ $\displaystyle=W^{(i)},\quad\forall i:1<i<n$
$\displaystyle\tilde{W}^{(n)}$
$\displaystyle=\frac{\beta_{2}}{\beta_{1}}W^{(n)}$
$\displaystyle\tilde{b}^{(i)}$
$\displaystyle=\frac{\beta_{1}}{\beta_{2}}b^{(i)},\quad\forall i:i<n$
$\displaystyle\tilde{b}^{(n)}$ $\displaystyle=b^{(n)}$

However, this mapping is not learned when training networks with different
$\beta$ values from scratch as the distribution over weight norms stays very
similar while the distribution changes drastically when artificially changing
the $\beta$ value as demonstrated above. We show this effect for a few
examples in C.2.

###  C.2 Examples

In Section C.1, we show that, by adjusting weights and biases of a Softplus
network, it can be functionally equivalent to a network with the same
structure but different $\beta$ value for the Softplus activation.
Artificially constructing networks in this way leads to a larger variance in
the weight norms. Even when no weight decay is used during training the weight
norms of the different network layers in one network tend to vary within one
order of magnitude.

Figure 9 shows the weights and biases for two networks from Table
LABEL:tab:statiscticsAllNets with $\beta=1$ and $\beta=10$ and the respective
weights and biases for two networks that produce identical output but have
changed $\beta$ values. In both cases the average weight norm of the
constructed network is higher than of the original.

Figure 9: Weights and biases for networks with identical outputs but different
$\beta$ value for the Softplus activation. Left: $\beta$ was changed from 1 to
10. Right: $\beta$ was changed from 10 to 1

Figure 10 shows weights and biases for some of our networks from Table
LABEL:tab:statiscticsAllNets. The weight and bias norms for each network are
normalized with the respective maximum value over all layers. Without
exception the highest weight norm is found in layer 5 in contrast to the
maximum weight norm when we do the artificial $\beta$ value switch. Thus
training Softplus networks from scratch does produce fundamentally different
networks that cannot be obtained with a mere rescaling of weights and biases.

Figure 10: Weights and biases plotted over layers for different networks (left
column: networks with weight decay, middle column: networks with different
$\beta$ values, right column: networks trained with curvature minimization

##  Appendix D Experimental Analysis

###  D.1 Error measures

In this section we define the error measures we use to quantify our results.
To ease notation we refer to the explanation of our original image as $h(x)=u$
and to the explanation of our perturbed image as $h(x_{\textrm{adv}})=v$. both
explanations can be expressed as a vector of length $n$.

  * •

The Mean squared error (MSE) between two explanations is defined as

$\text{MSE}(u,v)=\frac{1}{n}\sum_{i}^{n}(u_{i}-v_{i})^{2}$

  * •

The Pearson correlation coefficient (PCC) between two explanations is given by

$\text{PCC}(u,v)=\frac{\sum_{i}^{n}(u_{i}-\mu_{u})(v_{i}-\mu_{v})}{\sqrt{\sum_{i}^{n}(u_{i}-\mu_{u})^{2}}\sqrt{\sum_{i}^{n}(v_{i}-\mu_{v})^{2}}}$

where $\mu_{u}=\frac{1}{n}\sum_{i}^{n}u_{i}$ is the mean value of explanation
$u$. Analogously for $\mu_{v}$.

  * •

The Structural similarity index (SSIM) between two explanations is given by
calculating

$\text{SSIM}(u,v)=\frac{(2\mu_{u}\mu_{v}+(k_{1}L)^{2})(2\sigma_{uv}+(k_{2}L)^{2})}{(\mu_{u}^{2}+\mu_{v}^{2}+(k1L)^{2})(\sigma_{u}^{2}+\sigma_{v}^{2}+(k_{2}L)^{2})}$

for a $7\times 7$ window centered at each pixel and then averaging over all
windows. Mean values $\mu_{u}$ and $\mu_{v}$ and variances $\sigma_{uv}$,
$\sigma_{u}$ and $\sigma_{v}$ are calculated separately for each $7\times 7$
window. $L$ is the range between the largest and smallest value that a pixel
in the explanation can have and $k_{1}=0.01$ and $k_{2}=0.03$ are constants to
stabilize the division.

  * •

The accuracy (acc) of a network is the ratio of correctly classified images of
the test set:

$\text{acc}=\frac{correct\>predictions}{all\>predictions}$

###  D.2 Network structure

    CNN_CIFAR(
      (features): Sequential(
        (conv0): Conv2d(3, 32, kernel_size=(3, 3),
                        stride=(1, 1), padding=(1, 1))
        (acti0): ActivationFunction()
        (conv1): Conv2d(32, 32, kernel_size=(3, 3),
                        stride=(1, 1), padding=(1, 1))
        (acti1): ActivationFunction()
        (pool2): MaxPool2d(kernel_size=2, stride=2,
                        padding=0, dilation=1,
                        ceil_mode=False)
        (conv3): Conv2d(32, 64, kernel_size=(3, 3),
                        stride=(1, 1), padding=(1, 1))
        (acti3): ActivationFunction()
        (conv4): Conv2d(64, 64, kernel_size=(3, 3),
                        stride=(1, 1), padding=(1, 1))
        (acti4): ActivationFunction()
        (pool5): MaxPool2d(kernel_size=2, stride=2,
                        padding=0, dilation=1,
                        ceil_mode=False)
      )
      (classifier): Sequential(
        (view0): Reshape()
        (dens0): Linear(in_features=4096,
                        out_features=256,
                        bias=True)
        (acti0): ActivationFunction()
        (dens1): Linear(in_features=256,
                        out_features=10,
                        bias=True)
      )
    )

Figure 11: Setup of simple CNN for CIFAR-10

The structure of all networks trained within the scope of this work is
depicted in Figure 11. The activation function is either ReLU or Softplus (for
the networks trained with $\beta$ smoothing or Hessian minimization). In order
to focus on the robustness we aimed to train the different networks to similar
accuracy (albeit no longer than 200 epochs). We use Stochastic Gradient
Descent with momentum and learning rate decay. We do not perform any further
hyperparameter optimization. Statistics for all trained networks are
summarized in Table LABEL:tab:statiscticsAllNets.

Table 2: Statistics of all network configurations. Columns show weight decay, activation function (ReLU or $\beta$ parameter for Softplus), parameter for curvature minimization $\zeta$, test accuracy (acc), mean Pearson correlation coefficient (pcc) for Gaussian noise with different noise levels $\nu$, average weight norm ($\left\lVert W\right\rVert$) and average approximated Hessian norm ($\left\lVert H\right\rVert$). $\lambda$ | $\beta$ | $\zeta$ | acc | pcc for different $\nu$ | $||W||$ | $||H||$
---|---|---|---|---|---|---
|  |  |  | 0.005 | 0.01 | 0.025 |  |
0 | ReLU | 0.0 | 85.75 | 0.73 | 0.57 | 0.32 | 30.79 | -
5e-5 | ReLU | 0.0 | 86.38 | 0.77 | 0.62 | 0.36 | 23.28 | -
5e-4 | ReLU | 0.0 | 88.63 | 0.84 | 0.70 | 0.45 | 11.37 | -
5e-3 | ReLU | 0.0 | 86.10 | 0.89 | 0.80 | 0.62 | 4.80 | -
1e-2 | ReLU | 0.0 | 81.41 | 0.89 | 0.80 | 0.64 | 3.61 | -
0 | 10 | 0.0 | 85.61 | 0.81 | 0.63 | 0.34 | 30.01 | 503.61
0 | 5 | 0.0 | 85.60 | 0.88 | 0.73 | 0.39 | 28.70 | 280.71
0 | 1 | 0.0 | 85.60 | 0.93 | 0.85 | 0.61 | 27.76 | 59.21
0 | 5e-1 | 0.0 | 84.51 | 0.94 | 0.88 | 0.67 | 28.84 | 39.06
5e-5 | 10 | 0.0 | 86.36 | 0.86 | 0.70 | 0.39 | 22.91 | 298.79
5e-5 | 5 | 0.0 | 86.33 | 0.91 | 0.78 | 0.46 | 22.97 | 154.94
5e-5 | 1 | 0.0 | 86.03 | 0.94 | 0.86 | 0.62 | 22.73 | 51.63
5e-5 | 5e-1 | 0.0 | 85.34 | 0.94 | 0.88 | 0.67 | 23.76 | 36.43
5e-4 | 10 | 0.0 | 88.84 | 0.88 | 0.75 | 0.49 | 11.24 | 91.68
5e-4 | 5 | 0.0 | 88.76 | 0.91 | 0.79 | 0.53 | 11.36 | 59.57
5e-4 | 1 | 0.0 | 86.80 | 0.93 | 0.86 | 0.63 | 11.93 | 28.45
5e-4 | 5e-1 | 0.0 | 85.36 | 0.93 | 0.86 | 0.67 | 10.62 | 16.78
5e-3 | 10 | 0.0 | 86.13 | 0.91 | 0.82 | 0.64 | 4.81 | 9.54
5e-3 | 5 | 0.0 | 85.44 | 0.91 | 0.83 | 0.64 | 4.76 | 7.49
5e-3 | 1 | 0.0 | 83.35 | 0.92 | 0.86 | 0.71 | 4.86 | 3.79
5e-3 | 5e-1 | 0.0 | 77.60 | 0.96 | 0.93 | 0.85 | 4.66 | 2.02
1e-2 | 10 | 0.0 | 80.44 | 0.90 | 0.82 | 0.66 | 3.48 | 5.42
1e-2 | 5 | 0.0 | 77.57 | 0.89 | 0.82 | 0.65 | 3.33 | 5.27
1e-2 | 1 | 0.0 | 71.74 | 0.97 | 0.94 | 0.86 | 3.21 | 1.26
1e-2 | 5e-1 | 0.0 | 72.03 | 0.98 | 0.95 | 0.89 | 3.35 | 0.97
0 | 10 | 1e-7 | 85.65 | 0.95 | 0.87 | 0.60 | 24.72 | 56.77
0 | 10 | 1e-6 | 85.74 | 0.97 | 0.92 | 0.73 | 22.45 | 21.47
0 | 10 | 1e-5 | 85.56 | 0.98 | 0.95 | 0.84 | 20.59 | 8.49
0 | 10 | 1e-4 | 84.12 | 0.99 | 0.97 | 0.90 | 19.14 | 3.23
0 | 10 | 1e-3 | 82.40 | 0.99 | 0.98 | 0.94 | 17.91 | 1.19
0 | 10 | 1e-2 | 80.07 | 0.99 | 0.98 | 0.94 | 17.08 | 0.43
0 | 5 | 1e-7 | 86.26 | 0.95 | 0.88 | 0.64 | 25.44 | 49.58
0 | 5 | 1e-6 | 85.94 | 0.97 | 0.92 | 0.74 | 23.41 | 20.85
0 | 5 | 1e-5 | 85.87 | 0.98 | 0.95 | 0.83 | 21.88 | 7.91
0 | 5 | 1e-4 | 84.81 | 0.98 | 0.97 | 0.90 | 20.74 | 2.96
0 | 5 | 1e-3 | 83.12 | 0.99 | 0.98 | 0.93 | 19.72 | 1.29
0 | 5 | 1e-2 | 80.95 | 0.99 | 0.98 | 0.94 | 19.02 | 0.41
0 | 1 | 1e-7 | 85.24 | 0.94 | 0.88 | 0.69 | 27.69 | 31.61
0 | 1 | 1e-6 | 85.17 | 0.95 | 0.90 | 0.75 | 26.29 | 17.67
0 | 1 | 1e-5 | 84.85 | 0.97 | 0.94 | 0.83 | 25.11 | 7.62
0 | 1 | 1e-4 | 84.70 | 0.98 | 0.95 | 0.87 | 24.25 | 3.03
0 | 1 | 1e-3 | 82.68 | 0.98 | 0.96 | 0.90 | 23.23 | 1.16
0 | 1 | 1e-2 | 81.57 | 0.98 | 0.95 | 0.89 | 22.20 | 0.42
0 | 5e-1 | 1e-7 | 81.90 | 0.95 | 0.90 | 0.77 | 12.05 | 9.28
0 | 5e-1 | 1e-6 | 85.46 | 0.96 | 0.91 | 0.77 | 28.07 | 15.20
0 | 5e-1 | 1e-5 | 84.41 | 0.97 | 0.93 | 0.82 | 26.81 | 6.79
0 | 5e-1 | 1e-4 | 84.08 | 0.98 | 0.96 | 0.89 | 25.59 | 2.86
0 | 5e-1 | 1e-3 | 82.84 | 0.98 | 0.96 | 0.89 | 22.21 | 1.12
0 | 5e-1 | 1e-2 | 81.04 | 0.98 | 0.96 | 0.90 | 23.36 | 0.41
5e-5 | 10 | 1e-7 | 86.68 | 0.95 | 0.87 | 0.62 | 20.24 | 52.22
5e-5 | 10 | 1e-6 | 86.47 | 0.97 | 0.92 | 0.74 | 18.75 | 21.36
5e-5 | 10 | 1e-5 | 85.87 | 0.98 | 0.95 | 0.84 | 17.28 | 8.18
5e-5 | 10 | 1e-4 | 84.55 | 0.99 | 0.97 | 0.90 | 16.05 | 3.29
5e-5 | 10 | 1e-3 | 83.03 | 0.99 | 0.98 | 0.93 | 15.01 | 1.14
5e-5 | 10 | 1e-2 | 80.47 | 0.99 | 0.98 | 0.94 | 13.88 | 0.43
5e-5 | 5 | 1e-7 | 86.76 | 0.95 | 0.87 | 0.62 | 21.05 | 45.57
5e-5 | 5 | 1e-6 | 86.41 | 0.97 | 0.92 | 0.74 | 19.57 | 19.16
5e-5 | 5 | 1e-5 | 86.16 | 0.98 | 0.95 | 0.84 | 18.29 | 8.00
5e-5 | 5 | 1e-4 | 85.21 | 0.99 | 0.97 | 0.90 | 17.20 | 3.24
5e-5 | 5 | 1e-3 | 82.69 | 0.99 | 0.98 | 0.93 | 16.41 | 1.17
5e-5 | 5 | 1e-2 | 80.91 | 0.99 | 0.97 | 0.93 | 15.35 | 0.45
5e-5 | 1 | 1e-7 | 85.78 | 0.95 | 0.89 | 0.70 | 22.42 | 29.93
5e-5 | 1 | 1e-6 | 86.31 | 0.96 | 0.92 | 0.77 | 21.76 | 16.98
5e-5 | 1 | 1e-5 | 85.54 | 0.97 | 0.93 | 0.82 | 20.53 | 7.38
5e-5 | 1 | 1e-4 | 84.62 | 0.98 | 0.95 | 0.88 | 18.94 | 2.99
5e-5 | 1 | 1e-3 | 83.82 | 0.98 | 0.97 | 0.91 | 18.00 | 1.17
5e-5 | 1 | 1e-2 | 80.49 | 0.98 | 0.97 | 0.92 | 17.10 | 0.43
5e-5 | 5e-1 | 1e-7 | 84.84 | 0.95 | 0.89 | 0.70 | 23.02 | 26.39
5e-5 | 5e-1 | 1e-6 | 85.21 | 0.96 | 0.91 | 0.75 | 20.38 | 16.81
5e-5 | 5e-1 | 1e-5 | 82.98 | 0.96 | 0.92 | 0.81 | 12.51 | 6.07
5e-5 | 5e-1 | 1e-4 | 84.29 | 0.97 | 0.95 | 0.87 | 15.54 | 3.02
5e-5 | 5e-1 | 1e-3 | 82.67 | 0.98 | 0.96 | 0.89 | 14.95 | 1.12
5e-5 | 5e-1 | 1e-2 | 80.35 | 0.98 | 0.96 | 0.89 | 11.56 | 0.42
5e-4 | 10 | 1e-7 | 88.96 | 0.94 | 0.87 | 0.62 | 10.95 | 35.47
5e-4 | 10 | 1e-6 | 88.24 | 0.96 | 0.91 | 0.73 | 10.76 | 17.68
5e-4 | 10 | 1e-5 | 87.61 | 0.98 | 0.94 | 0.83 | 9.98 | 7.27
5e-4 | 10 | 1e-4 | 86.54 | 0.98 | 0.96 | 0.88 | 9.20 | 3.08
5e-4 | 10 | 1e-3 | 84.80 | 0.98 | 0.96 | 0.90 | 8.42 | 1.10
5e-4 | 10 | 1e-2 | 82.72 | 0.98 | 0.96 | 0.90 | 7.67 | 0.41
5e-4 | 5 | 1e-7 | 88.68 | 0.94 | 0.87 | 0.63 | 11.07 | 32.80
5e-4 | 5 | 1e-6 | 88.31 | 0.96 | 0.91 | 0.73 | 10.78 | 16.68
5e-4 | 5 | 1e-5 | 87.75 | 0.97 | 0.94 | 0.83 | 10.03 | 6.97
5e-4 | 5 | 1e-4 | 86.35 | 0.98 | 0.96 | 0.88 | 9.70 | 2.96
5e-4 | 5 | 1e-3 | 84.74 | 0.98 | 0.96 | 0.91 | 8.97 | 1.16
5e-4 | 5 | 1e-2 | 82.12 | 0.98 | 0.96 | 0.91 | 8.04 | 0.44
5e-4 | 1 | 1e-7 | 87.11 | 0.94 | 0.86 | 0.64 | 11.67 | 25.09
5e-4 | 1 | 1e-6 | 87.08 | 0.96 | 0.91 | 0.75 | 11.34 | 14.06
5e-4 | 1 | 1e-5 | 86.72 | 0.97 | 0.94 | 0.82 | 11.13 | 7.31
5e-4 | 1 | 1e-4 | 85.64 | 0.98 | 0.96 | 0.88 | 9.81 | 2.92
5e-4 | 1 | 1e-3 | 83.48 | 0.98 | 0.97 | 0.91 | 9.49 | 1.20
5e-4 | 1 | 1e-2 | 81.87 | 0.98 | 0.96 | 0.91 | 8.11 | 0.44
5e-4 | 5e-1 | 1e-7 | 78.10 | 0.93 | 0.87 | 0.73 | 7.07 | 4.57
5e-4 | 5e-1 | 1e-6 | 85.76 | 0.95 | 0.90 | 0.75 | 11.03 | 12.23
5e-4 | 5e-1 | 1e-5 | 85.56 | 0.97 | 0.93 | 0.82 | 10.82 | 6.20
5e-4 | 5e-1 | 1e-4 | 85.21 | 0.98 | 0.96 | 0.88 | 10.21 | 2.86
5e-4 | 5e-1 | 1e-3 | 81.05 | 0.96 | 0.93 | 0.84 | 6.99 | 0.94
5e-4 | 5e-1 | 1e-2 | 81.98 | 0.98 | 0.97 | 0.91 | 8.12 | 0.46
5e-3 | 10 | 1e-7 | 86.09 | 0.91 | 0.83 | 0.65 | 4.79 | 8.23
5e-3 | 10 | 1e-6 | 85.92 | 0.91 | 0.83 | 0.66 | 4.74 | 6.31
5e-3 | 10 | 1e-5 | 85.59 | 0.93 | 0.87 | 0.72 | 4.66 | 3.73
5e-3 | 10 | 1e-4 | 84.53 | 0.95 | 0.91 | 0.79 | 4.50 | 1.88
5e-3 | 10 | 1e-3 | 83.11 | 0.96 | 0.93 | 0.84 | 4.30 | 0.86
5e-3 | 10 | 1e-2 | 80.16 | 0.97 | 0.95 | 0.88 | 4.00 | 0.33
5e-3 | 5 | 1e-7 | 85.90 | 0.90 | 0.82 | 0.64 | 4.78 | 7.09
5e-3 | 5 | 1e-6 | 85.43 | 0.91 | 0.84 | 0.67 | 4.79 | 5.57
5e-3 | 5 | 1e-5 | 85.26 | 0.92 | 0.86 | 0.71 | 4.67 | 3.48
5e-3 | 5 | 1e-4 | 84.51 | 0.95 | 0.90 | 0.78 | 4.52 | 1.76
5e-3 | 5 | 1e-3 | 83.07 | 0.96 | 0.93 | 0.84 | 4.32 | 0.82
5e-3 | 5 | 1e-2 | 80.53 | 0.97 | 0.94 | 0.87 | 4.06 | 0.32
5e-3 | 1 | 1e-7 | 82.84 | 0.91 | 0.85 | 0.70 | 4.76 | 4.11
5e-3 | 1 | 1e-6 | 83.40 | 0.92 | 0.86 | 0.72 | 4.85 | 3.37
5e-3 | 1 | 1e-5 | 81.72 | 0.93 | 0.88 | 0.75 | 4.65 | 2.62
5e-3 | 1 | 1e-4 | 82.19 | 0.95 | 0.91 | 0.81 | 4.68 | 1.52
5e-3 | 1 | 1e-3 | 81.14 | 0.96 | 0.93 | 0.86 | 4.43 | 0.74
5e-3 | 1 | 1e-2 | 79.13 | 0.96 | 0.94 | 0.86 | 4.16 | 0.32
5e-3 | 5e-1 | 1e-7 | 77.41 | 0.96 | 0.93 | 0.85 | 4.59 | 2.03
5e-3 | 5e-1 | 1e-6 | 77.42 | 0.97 | 0.94 | 0.85 | 4.57 | 1.91
5e-3 | 5e-1 | 1e-5 | 76.88 | 0.97 | 0.94 | 0.86 | 4.47 | 1.57
5e-3 | 5e-1 | 1e-4 | 77.17 | 0.97 | 0.94 | 0.86 | 4.49 | 1.24
5e-3 | 5e-1 | 1e-3 | 76.68 | 0.97 | 0.95 | 0.89 | 4.29 | 0.64
5e-3 | 5e-1 | 1e-2 | 75.01 | 0.98 | 0.97 | 0.92 | 3.89 | 0.27
1e-2 | 10 | 1e-7 | 64.43 | 0.94 | 0.87 | 0.65 | 3.37 | 7.11
1e-2 | 10 | 1e-6 | 79.77 | 0.90 | 0.83 | 0.66 | 3.46 | 4.06
1e-2 | 10 | 1e-5 | 79.87 | 0.91 | 0.84 | 0.69 | 3.46 | 2.47
1e-2 | 10 | 1e-4 | 78.86 | 0.94 | 0.88 | 0.75 | 3.37 | 1.34
1e-2 | 10 | 1e-3 | 77.68 | 0.95 | 0.92 | 0.82 | 3.25 | 0.63
1e-2 | 10 | 1e-2 | 75.75 | 0.97 | 0.94 | 0.87 | 3.12 | 0.25
1e-2 | 5 | 1e-7 | 78.63 | 0.89 | 0.81 | 0.65 | 3.40 | 4.42
1e-2 | 5 | 1e-6 | 77.74 | 0.90 | 0.83 | 0.66 | 3.34 | 4.21
1e-2 | 5 | 1e-5 | 78.16 | 0.90 | 0.83 | 0.68 | 3.35 | 2.34
1e-2 | 5 | 1e-4 | 78.18 | 0.92 | 0.86 | 0.73 | 3.31 | 1.20
1e-2 | 5 | 1e-3 | 77.43 | 0.96 | 0.92 | 0.82 | 3.25 | 0.59
1e-2 | 5 | 1e-2 | 74.45 | 0.96 | 0.93 | 0.85 | 3.02 | 0.25
1e-2 | 1 | 1e-7 | 71.74 | 0.97 | 0.94 | 0.86 | 3.21 | 1.26
1e-2 | 1 | 1e-6 | 71.92 | 0.97 | 0.95 | 0.88 | 3.24 | 1.18
1e-2 | 1 | 1e-5 | 73.02 | 0.97 | 0.94 | 0.86 | 3.30 | 0.98
1e-2 | 1 | 1e-4 | 72.02 | 0.97 | 0.95 | 0.88 | 3.16 | 0.77
1e-2 | 1 | 1e-3 | 71.16 | 0.98 | 0.95 | 0.89 | 3.08 | 0.43
1e-2 | 1 | 1e-2 | 69.64 | 0.98 | 0.97 | 0.92 | 2.90 | 0.19
1e-2 | 5e-1 | 1e-7 | 70.53 | 0.98 | 0.96 | 0.90 | 3.20 | 0.87
1e-2 | 5e-1 | 1e-6 | 70.07 | 0.98 | 0.96 | 0.90 | 3.21 | 0.89
1e-2 | 5e-1 | 1e-5 | 70.46 | 0.98 | 0.96 | 0.89 | 3.20 | 0.79
1e-2 | 5e-1 | 1e-4 | 71.49 | 0.98 | 0.96 | 0.91 | 3.31 | 0.72
1e-2 | 5e-1 | 1e-3 | 70.06 | 0.99 | 0.97 | 0.93 | 3.11 | 0.41
1e-2 | 5e-1 | 1e-2 | 67.76 | 0.99 | 0.98 | 0.94 | 2.84 | 0.19

###  D.3 Gradient Explanation

In Figures 12, 13, and 14, we show additional error measures for the Gradient
explanation. PCC and SSIM increase with robustness while MSE decreases.

Figure 12: PCC, SSIM and MSE between original Gradient explanation map and
explanation after adding random noise to the image. PCC and SSIM are higher
and MSE is lower for networks trained with weight decay. That means weight
decay improves robustness of explanations. We show mean +/- std.

Figure 13: PCC, SSIM and MSE between original Gradient explanation map and
explanation after adding random noise to the image. PCC and SSIM are higher
and MSE is lower for Softplus networks trained with a small $\beta$ value.
That means Softplus activations improve robustness of explanations. All nets
were trained with weight decay ($\lambda$=5e-4). We show mean +/- std.

Figure 14: PCC, SSIM and MSE between original Gradient explanation map and
explanation after adding random noise to the image. PCC and SSIM are higher
and MSE is lower for networks trained with strong curvature minimization. That
means minimizing the curvature improves robustness of explanations. All nets
were trained with Softplus activations ($\beta=10$) and weight decay
($\lambda$=5e-4). We show mean +/- std.

###  D.4 Other Explanation Methods

In Figures 15, 16, 17, and 18, we show how our proposed measures effect other
explanation methods. The trend towards increased robustness is clearly visible
for all considered explanation methods. We note that the explanations start
from different levels of robustness but can still profit from our methods. The
most resilient method against random input perturbations is Layerwise
Relevance Propagation, followed by Guided Backpropagation, Integrated
Gradients, Gradient$\times$Input and Gradient in descending order.

Figure 15: PCCs (mean +/- std) between original Gradient$\times$Input
explanation map and explanation after adding random noise to the image. left:
effect of weight decay, middle: effect of Softplus $\beta$ (for
$\lambda=5$e-4), right: effect of curvature minimization (for $\lambda=5$e-4,
$\beta$=10).

Figure 16: PCCs (mean +/- std) between original Integrated Gradients
explanation map and explanation after adding random noise to the image. left:
effect of weight decay, middle: effect of Softplus $\beta$ (for
$\lambda=5$e-4), right: effect of curvature minimization (for $\lambda=5$e-4,
$\beta$=10).

Figure 17: PCCs (mean +/- std) between original Guided Backpropagation
explanation map and explanation after adding random noise to the image. left:
effect of weight decay, middle: effect of Softplus $\beta$ (for
$\lambda=5$e-4), right: effect of curvature minimization (for $\lambda=5$e-4,
$\beta$=10).

Figure 18: PCCs (mean +/- std) between original Layerwise Relevance
Propagation explanation map and explanation after adding random noise to the
image. left: effect of weight decay, middle: effect of Softplus $\beta$ (for
$\lambda=5$e-4), right: effect of curvature minimization (for $\lambda=5$e-4,
$\beta$=10).

###  D.5 Other types of noise

In the main text we only consider Gaussian noise. We repeat our experiments
from 4 for the Gradient explanation when we perturb the input images with
Laplacian noise and salt-pepper noise.

####  D.5.1 Laplace noise

We sample random noise from the Laplace distribution

$f(x|\mu,b)=\frac{1}{2b}\exp{-\frac{|x-\mu|}{b}}$ (34)

where $\mu$ is the data mean and $b$ is a scale parameter which we define as
$b=(x_{max}-x_{min})\nu$, depending on the noise level $\nu$. Figure 19 shows
effects on the Gradient explanation when adding Laplace noise to the input
images. We see that the results look statistically very similar to the results
for Gaussian noise.

Figure 19: PCCs (mean +/- std) between original Gradient explanation map and
explanation after adding Laplace noise to the image. left: effect of weight
decay, middle: effect of Softplus $\beta$ (for $\lambda=5$e-4), right: effect
of curvature minimization (for $\lambda=5$e-4, $\beta$=10).

####  D.5.2 Salt-pepper noise

To perturb an image with salt-pepper noise we randomly select
$100*\frac{\nu}{2}$ % of the pixel in the image and switch them to $x_{max}$
(white) or $x_{min}$ (black) at random. We select a very small amount of pixel
(for noise level $\nu=0.005$ only 3 pixels) to be perturbed as salt-pepper
noise has a very strong effect on the classification accuracy which we aim to
keep approximately constant.

Figure 20 shows effects on the Gradient explanation when adding salt-pepper
noise to the input images. We can still see a significant improvement in
robustness when training networks with our proposed methods, however the
effect for Softplus activations and Hessian minimization is less pronounced
than for Laplace or Gaussian noise.

Figure 20: PCCs (mean +/- std) between original Gradient explanation map and
explanation after adding salt-pepper noise to the image. left: effect of
weight decay, middle: effect of Softplus $\beta$ (for $\lambda=5$e-4), right:
effect of curvature minimization (for $\lambda=5$e-4, $\beta$=10).

###  D.6 Connection between Hessian norms and weight norms

In the main text, Section 4.6, we mentioned that weight decay leads to
decreased weight norms and thus also to decreased Hessian norms. However when
we minimize the Hessian norm directly, the weight norms only change minimally.
Figure 21 shows how weight norms and Hessian norms change when applying weight
decay (varying $\lambda$), substituting ReLU with Softplus (varying $\beta$)
and minimizing the Hessian norm directly (varying $\zeta$). We average over
all Softplus networks trained with $\zeta=0$ for the first two plots and we
average over all networks trained with Hessian minimization for the last plot.

Figure 21: The connection between weight norms $\left\lVert W\right\rVert$ and
Hessian norms $\left\lVert H\right\rVert$ is different for our three methods.
Left: $\left\lVert W\right\rVert$ and $\left\lVert H\right\rVert$ both
decrease in a similar manner when applying stronger weight decay (increasing
$\lambda$). Middle: $\left\lVert W\right\rVert$ stays relatively constant
while $\left\lVert H\right\rVert$ decreases with increasing Softplus $\beta$.
Right: $\left\lVert W\right\rVert$ decrease slightly while $\left\lVert
H\right\rVert$ decrease strongly with stronger Hessian minimization
(increasing $\zeta$). We show mean$\pm$std.

##  Appendix E Hessian norm approximation

In Section 3.1, we showed that for the number of samples $N\rightarrow\infty$
the sampling approximation approaches the true Hessian norm. To include the
Hessian approximation in our training procedure, we need to fix a certain
number of samples and to perform a Monte-Carlo estimate of the expectation
value. Figure 22 shows the relative error between sampled hessian norm and
true hessian norm. Increasing the sample size noticeably reduces the error,
but a sample size of 1 already has a relative error of only 6%. If we average
over a batch of images, the error reduces further. This means that for
training and validation sampling once per image is in practice sufficient.

Figure 22: Relative error (mean and standard deviation) between True Hessian
norm and approximation via sampling