# On the Robustness of Bayesian Neural Networks
to Adversarial Attacks

Luca Bortolussi<EMAIL_ADDRESS>
Department of Mathematics and Geosciences,
University of Trieste, Trieste, Italy;
Modeling and Simulation Group, Saarland University, Saarland, Germany Ginevra
Carbone<EMAIL_ADDRESS>
Department of Mathematics and Geosciences,
University of Trieste, Trieste, Italy Luca Laurenti<EMAIL_ADDRESS>
Delft Center of Systems and Control,
TU Delft University, The Netherlands Andrea Patane<EMAIL_ADDRESS>
School of Computer Science and Statistics,
Trinity College, Dublin, Ireland Guido Sanguinetti<EMAIL_ADDRESS>
School of Informatics,
University of Edinburgh, Edinburgh, United Kingdom;
SISSA, International School of Advanced Studies, Trieste, Italy. Matthew
Wicker<EMAIL_ADDRESS>
Department of Computer Science,
University of Oxford, Oxford, United Kingdom

###### Abstract

Vulnerability to adversarial attacks is one of the principal hurdles to the
adoption of deep learning in safety-critical applications. Despite significant
efforts, both practical and theoretical, training deep learning models robust
to adversarial attacks is still an open problem. In this paper, we analyse the
geometry of adversarial attacks in the large-data, overparameterized limit for
Bayesian Neural Networks (BNNs). We show that, in the limit, vulnerability to
gradient-based attacks arises as a result of degeneracy in the data
distribution, i.e., when the data lies on a lower-dimensional submanifold of
the ambient space. As a direct consequence, we demonstrate that in this limit
BNN posteriors are robust to gradient-based adversarial attacks. Crucially, we
prove that the expected gradient of the loss with respect to the BNN posterior
distribution is vanishing, even when each neural network sampled from the
posterior is vulnerable to gradient-based attacks. Experimental results on the
MNIST, Fashion MNIST, and half moons datasets, representing the finite data
regime, with BNNs trained with Hamiltonian Monte Carlo and Variational
Inference, support this line of arguments, showing that BNNs can display both
high accuracy on clean data and robustness to both gradient-based and
gradient-free based adversarial attacks.

Keywords: Bayesian Neural Networks, Adversarial Attacks, Robustness

##  1 Introduction

Adversarial attacks are small, potentially imperceptible, perturbations of
test inputs that can lead to catastrophic misclassifications in high-
dimensional classifiers such as deep Neural Networks (NN). Since the seminal
work of Szegedy et al. (2013) adversarial attacks have been intensively
studied and even highly accurate state-of-the-art deep learning models,
trained on very large data sets, have been shown to be susceptible to such
attacks (Goodfellow et al., 2014). In the absence of effective defenses, the
widespread existence of adversarial examples has raised serious concerns about
the security and robustness of models learned from data (Biggio and Roli,
2018). As a consequence, the development of machine learning models that are
robust to adversarial perturbations is an essential pre-condition for their
application in safety-critical scenarios, such as autonomous driving, where
model failures can lead to fatal or costly accidents.

Many adversarial attack strategies are based on identifying directions of high
variability in the loss function by evaluating the gradient w.r.t. to the
neural network input (Goodfellow et al., 2014; Madry et al., 2017). Since such
variability can be intuitively linked to uncertainty in the prediction,
Bayesian Neural Networks (BNNs) (Neal, 2012) have been recently suggested as a
more robust deep learning paradigm, a claim that has also found empirical
support (Feinman et al., 2017; Wicker et al., 2021; Bekasov and Murray, 2018;
Liu et al., 2018; Yuan et al., 2020). However, neither the source of this
robustness, nor its general applicability are well understood mathematically.

In this paper we show a remarkable property of BNNs: in a suitably defined
large data limit, we prove that the gradients of the expected loss function of
an infinitely-wide BNN w.r.t. the input vanish. Our analysis shows that
adversarial attacks for highly accurate NNs arise from the low dimensional
support of the data generating distribution. By averaging over nuisance
dimensions, BNNs achieve zero expected gradient of the loss and are thus
provably immune to gradient-based adversarial attacks. Specifically, we first
show that, for any neural network achieving zero loss, adversarial attacks
arise in directions orthogonal to data manifold. Then, we rely on the
submanifold extension lemma (Lee, 2013) to show that in the limit of
infinitely-wide layers, for any neural network and any weights set there
exists another weights set (of the same neural network architecture) achieving
the same loss and with opposite loss gradients orthogonal to the data manifold
on a given point. Under the assumption of infinitely many data and flat (i.e.,
uninformative) prior, we then show that by averaging over these weights sets
the expectation of the gradient w.r.t. the posterior distribution of a BNN
vanishes. Crucially, our results guarantees that, in the limit, BNNs’
posteriors are provably robust to gradient-based adversarial attacks even if
each neural network sampled from the posterior is vulnerable to such attacks.

We experimentally support our theoretical findings on various BNN
architectures trained with Hamiltonian Monte Carlo (HMC) and with Variational
Inference (VI) on MNIST, Fashion MNIST and the half-moon data set, empirically
showing that the magnitude of the gradients decreases as more samples are
taken from the BNN posterior. We then explore the robustness of BNNs to
adversarial attacks. In particular, we conduct a large-scale experiment on
thousands of different neural networks, empirically finding that, in the cases
here analysed, for BNNs high accuracy correlates with high robustness to
gradient-based adversarial attacks, contrary to what observed for
deterministic NNs trained via standard Stochastic Gradient Descent (SGD).
Finally, we also investigate the robustness of BNNs to gradient-free
adversarial attacks, showing that BNNs are substantially more robust than
their deterministic counterpart even in this setting.

In summary, this paper makes the following contributions:

  * •

A proof that, in the infinitely-wide layers and large data limit setting, the
gradient of the loss function w.r.t. the input only preserves the component
which is orthogonal to the data manifold (Section 3) and that for any weights
set of a neural network there exists another weight set with same loss and
opposite orthogonal gradients (Section 3.1).

  * •

A proof that, in the infinitely wide and large data limit, the posterior
average of the gradients of the loss function vanishes, thus providing
robustness to BNNs (Section 4).

  * •

Experiments showing empirically that BNNs are more robust to both gradient-
based and gradient-free attacks than their deterministic counterpart and can
resist the well known accuracy-robustness trade-off (Section 6).111The code
for the experiments can be found at https://github.com/ginevracoal/robustBNNs.

A preliminary version of this work appeared in Carbone et al. (2020). This
work extends Carbone et al. (2020) in several aspects. In Carbone et al.
(2020) we proved that given an infinitely-wide neural network with zero loss
and a non-zero orthogonal gradient to the data manifold, there exists another
zero-loss neural network with opposite orthogonal gradient and use this result
to conjecture that BNNs achieve zero gradients in expectation under a wide
enough prior. In this paper, in Theorem 6 this result is proved explicitly.
Furthermore, we substantially extended the discussion and improved the
empirical analysis with gradient-free adversarial attacks.

The paper is structured as follows. In Section 2 we introduce background on
infinitely-wide neural networks and BNNs. In Section 3 we will first show that
for highly accurate neural networks the gradient of the loss is non-zero only
in directions orthogonal to the data manifold. Then, in Section 3.1 we will
prove that for any neural network and weight set there exists another weight
set of the same neural network with same loss and opposite orthogonal
gradients to the data manifold. By averaging over these weight sets in Section
4 we prove that for a BNN the expected gradient of the loss is zero, thus
making them robust to adversarial attacks. Section 5 discusses consequences
and limitations of our results. Empirical results in Section 6 will support
our theoretical findings.

#### Related Work

The robustness of BNNs to adversarial examples has already been observed
empirically in various works (Feinman et al., 2017; Yuan et al., 2020;
Michelmore et al., 2020; Pang et al., 2021; Smith and Gal, 2018; Uchendu et
al., 2021). However, while these works present empirical evidences on the
robustness of BNNs, they do not give any theoretical justification on the
mechanisms that lead to BNN robustness. First attempts to understand the
robustness properties of BNNs have been considered in Bekasov and Murray
(2018); Gal and Smith (2018). In particular, Bekasov and Murray (2018) defined
Bayesian adversarial spheres and empirically showed that, for BNNs trained
with HMC, adversarial examples tend to have high uncertainty. Gal and Smith
(2018) derived sufficient conditions for idealised BNNs to avoid adversarial
examples. However, it is unclear how such conditions could be checked in
practice, as it would require one to check that the BNN architecture is
invariant under all the symmetries of the data.

Because of the capabilities of BNNs to model uncertainty, which can be
intuitively linked to their robustness properties, both empirical (Carlini and
Wagner, 2017; Li and Gal, 2017; Feinman et al., 2017; Rawat et al., 2017;
Grosse et al., 2018) and formal verification methods (Wicker et al., 2020;
Berrada et al., 2021) to detect adversarial examples for BNNs have been
introduced. These methods have been followed by techniques to perform
adversarial training for BNNs (Zhang et al., 2021; Ye and Zhu, 2018; Liu et
al., 2018; Wicker et al., 2021), where additional robustness constraints or
penalties are considered directly at training time. Interestingly, empirical
results obtained with such techniques, highlighted how, in the Bayesian
settings, high accuracy and high robustness often are positively correlated
with each other. The theoretical framework we develop in this paper further
confirms and grounds these findings.

##  2 Background

Let $f^{true}:\mathcal{M}\to\mathbb{R}$ be a function defined on a smooth
closed data manifold $\mathcal{M}\subset X\subset\mathbb{R}^{d}$ with $X$
being the ambient (or embedding) space.222For simplicity of presentation, we
assume a scalar output. The results of this paper naturally extend to the
multi-output case by treating each output component similarly to the single
output case. We consider the problem of approximating $f^{true}$ via the
learning of an $M+1$ layers neural network $f(\cdot,\mathbf{w})$, with
$\mathbf{w}\in\mathbb{R}^{n_{\mathbf{w}}}$ being the aggregated vector of
weights and biases. Formally, for $\mathbf{x}\in X$,
$f(\mathbf{x},\mathbf{w})$ is defined iteratively over the number of layers
as:

$\displaystyle f^{(1)}_{i}(\mathbf{x})$
$\displaystyle=\sum_{j=1}^{d}w^{(1)}_{ij}x_{j}+b^{(1)}_{i}$ (1) $\displaystyle
f^{(m)}_{i}(\mathbf{x})$
$\displaystyle=\sum_{j=1}^{n_{m-1}}w^{(m)}_{ij}\phi(f^{(m-1)}_{j}(\mathbf{x}))+b^{(m)}_{i},\quad\textrm{for}\;m=2,\ldots,M+1,\quad$
(2) $\displaystyle f(\mathbf{x},\mathbf{w})$
$\displaystyle=f^{(M+1)}(\mathbf{x}),$ (3)

where $n_{m}$ is the number of neurons in the $m$-th layer, $\phi$ is the
activation function – which we assume to be smooth, bounded, and non-constant.
In order to learn the weights of $f(\mathbf{x},\mathbf{w})$ one considers a
dataset $D_{N}$ composed of $N$ points,
$D_{N}=\\{(\mathbf{x}_{i},y_{i})\;|\;x_{i}\in\mathcal{M},\,y_{i}=f^{true}(\mathbf{x}_{i}),\,i=1,\ldots,N\\}$.
The dataset is hence used to quantify the distance between $f^{true}$ and
$f(\cdot,\mathbf{w})$ by evaluating it on a loss function
$L(\mathbf{x},\mathbf{w})$ of the form
$L(\mathbf{x},\mathbf{w})=\ell(f(\mathbf{x},\mathbf{w}),f^{true}(\mathbf{x}))$,
with $\ell(\cdot,\cdot)$ chosen accordingly to the semantic of the problem at
hand (e.g., square loss or cross-entropy).333For simplicity of notation we
omit the explicit dependence on the true function from the loss. Intuitively,
minimisation of the loss function over the weight vector $\mathbf{w}$ leads to
increasing fit of $f(\mathbf{x},\mathbf{w})$ to $f^{true}(\mathbf{x})$, with
zero-loss indicating that the fit is exact (Bishop et al., 1995), so that the
problem of approximating $f^{true}$ boils down to the problem of minimizing
the loss function over $\mathbf{w}$ (e.g., by employing gradient descent
algorithms).

In this paper, we aim at analysing the adversarial robustness of
$f(\mathbf{x},\mathbf{w})$. In order to do so, we will rely on crucial results
from Bayesian learning of neural networks and on the properties of infinitely-
wide neural networks. The remainder of this section is dedicated to the review
of such notions.

###  2.1 Infinitely-Wide Neural Networks

In our analysis of the behaviour of BNNs under adversarial attacks we rely on
the notion of infinitely-wide NNs, i.e. NNs with an infinite number of
neurons.

######  Definition 1 (Infinitely-wide neural network)

Consider a family of neural networks
$\\{f(\mathbf{x},\mathbf{w}_{n_{\mathbf{w}}})\\}_{n_{\mathbf{w}}>0}$ of
Equations (1)–(3), with a fixed number of neurons for $m=1,\ldots,M-1$ and a
variable number of neurons $n_{M}$ in the last hidden layer. We say that

$f^{\infty}(\mathbf{x}):=\lim_{n_{M}\to\infty}f(\mathbf{x},\mathbf{w}_{n_{\mathbf{w}}})\quad\forall\mathbf{x}\in
X,$ (4)

is an infinitely wide neural network if the limit above exists and if the
resulting function defines a mapping from $X$ to $\mathbb{R}$. Furthermore, we
call $\mathcal{F}$ the set of such limit functions.

The interest behind the set of infinitely-wide neural networks lies in the
fact that they are universal approximators (Cybenko, 1989; Hornik,
1991).444Notice that the limit in Definition 1 is taken only w.r.t. the last
hidden layer. Similar results, albeit with additional care needed for the
definition of the limiting sequence, can be obtained by taking the limit
w.r.t. all the hidden layers (Matthews et al., 2018). More precisely, under
the assumption that the true function $f^{true}$ is continuous, we have that:

$\displaystyle\forall\epsilon>0,\;\exists
f^{*}\in\mathcal{F}\;\textrm{s.t.}\;\forall
x\in\mathcal{M},\;|f^{true}(x)-f^{*}(x)|<\epsilon.$ (5)

That is, $\mathcal{F}$ is dense in the space of continuous functions.
Furthermore, it is possible to show that any smooth function with bounded
derivatives can be represented exactly by an infinitely wide NN with bounded
weights norm (i.e. with bounded sum of the squared Euclidean norm of the
weights in the network) (Ongie et al., 2020). We will rely on these crucial
properties of infinitely-wide neural networks to reason about their behaviour
against adversarial attacks. For simplicity, as also common in the literature
(Rotskoff and Vanden-Eijnden, 2018), throughout the paper we will assume that
the approximation error $\epsilon$ in Equation (5) is negligible, that is that
there exists an infinitely-wide neural network that can approximate the true
function exactly. In formal terms, we will assume that any smooth function
$g:X\to\mathbb{R}$ such that $g|_{\mathcal{M}}=f^{true}$ belongs to
$\mathcal{F}$, where $|_{\mathcal{M}}$ denotes the restriction to the data
manifold ${\mathcal{M}}$. We remark that the existence of such a neural
network does not necessarily mean that it will be automatically found through
gradient descent-based training on a finite dataset. However, recent results
(Rotskoff and Vanden-Eijnden, 2018) have shown that, under mild conditions,
the loss function, as a functional over the distribution over weights of an
infinitely wide NN, is a convex functional, and hence the gradient flow of
stochastic gradient descent will converge to a unique distribution over
weights. That is, given an infinitely wide NN $f^{\infty}$ and a sequence of
datasets $\\{D_{N}\\}_{N>0}$ of cardinality $N$ extracted from the data
manifold $\mathcal{M}$, we have that:

$\displaystyle\lim_{N\to\infty}\ell\big{(}f_{D_{N}}^{\infty}(\mathbf{x}),f^{true}(\mathbf{x})\big{)}=0,\quad\forall\mathbf{x}\in\mathcal{M}$
(6)

where $f^{\infty}_{D_{N}}$ represents the infinitely wide NN trained on
$D_{N}$ until convergence. In Section 3 we will show how deterministic neural
networks can be vulnerable to adversarial attacks even when the loss is zero,
while infinitely wide Bayesian neural networks achieving zero loss are
provably robust to adversarial attacks.

###  2.2 Bayesian Neural Networks

Bayesian modelling aims to capture the uncertainty of data driven models by
defining ensembles of predictors (Barber, 2012); it does so by turning model
parameters into random variables. In the NN scenario, one starts by putting a
prior measure over the network weights $p(\mathbf{w})$ (Neal, 2012).555In the
remainder of this paper, we employ the common notation of indicating density
functions with $p$ and their corresponding probability measures with $P$. The
fit of the network with weights $\mathbf{w}$ to the data $D$ is assessed
through the likelihood $p(D|\mathbf{w})$ (Bishop, 2006).666Notice that in the
Bayesian setting the likelihood is a transformation of the loss function used
in deterministic settings. In the rest of the paper we use both terminologies,
and the loss is not to be confused with that used in Bayesian decision theory
(Bishop, 2006). Bayesian inference then combines likelihood and prior via the
Bayes theorem to obtain a posterior distribution over the NN parameters

$\displaystyle p\left(\mathbf{w}|D\right)\propto
p\left(D|\mathbf{w}\right)p\left(\mathbf{w}\right).$ (7)

Unfortunately it is in general infeasible to compute the posterior
distribution exactly for non-linear/non-conjugate models such as deep NNs, so
that approximate Bayesian inference methods are employed in practice.
Asymptotically exact samples from the posterior distribution can be obtained
via procedures such as Hamiltonian Monte Carlo (HMC) (Neal et al., 2011),
while approximate samples can be obtained more cheaply via Variational
Inference (VI) (Blundell et al., 2015). Irrespective of the posterior
inference method of choice, Bayesian empirical predictions at a new input
$\mathbf{x}$ are obtained from an ensemble of $n$ NNs, each with its
individual weights drawn from the posterior distribution $p(\mathbf{w}|D)$:

$\begin{split}f\left(\mathbf{x}|D\right)=&\langle
f(\mathbf{x},\mathbf{w})\rangle_{p\left(\mathbf{w}|D\right)}\simeq\frac{1}{n}\sum_{i=1}^{n}f(\mathbf{x},\mathbf{w}_{i})\qquad\mathbf{w}_{i}\sim
p\left(\mathbf{w}|D\right)\end{split}$ (8)

where $\langle\cdot\rangle_{p\left(\mathbf{w}|D\right)}$ denotes expectation
w.r.t. the posterior distribution $p\left(\mathbf{w}|D\right)$.

We notice that defining a distribution over the weights $p(\mathbf{w})$
naturally leads to the definition of a distribution over the functions in
$\mathcal{F}$ through the applications of the neural network, which we denote
by $p(f(\cdot,{\mathbf{w}}))$. In particular, $p(f(\cdot,{\mathbf{w}}))$ leads
to a probability measure $P$ over $\mathcal{F}$ with a $\sigma-$algebra
generated by sets of the form
$\\{f(\cdot,{\mathbf{w}})\in\mathcal{F}:f(\mathbf{x}_{1},{\mathbf{w}})\in
G_{1},...,f(\mathbf{x}_{K},{\mathbf{w}})\in G_{K}\\}$ where $K$ is an
arbitrary integer, $\mathbf{x}_{1},...\mathbf{x}_{K}\in X$, and
$G_{1},...,G_{K}\subset\mathbb{R}$ are closed intervals (Adler, 2010). Note
that according to the above definition of $\sigma-$algebra any measurable set
of functions $F\subseteq\mathcal{F}$ only depends on the behaviour of
$f(\cdot,\mathbf{w})$ on a countable set of input points. Nevertheless, in our
setting this is not limiting. In fact, properties involving uncountable set of
points can generally (and this is the case for all the properties considered
in this paper) be reformulated over a countable set of inputs by relying on
the continuity of each function $f(\cdot,\mathbf{w})\in\mathcal{F}$ (Adler,
2010).

In this paper, we will rely on flat or uninformative priors (Box and Tiao,
2011; Christensen et al., 2010), that is wide priors that resemble a uniform
distribution. We introduce the concept of flat priors over the space of
functions $\mathcal{F}$.

######  Definition 2 (Flat Prior)

Let $p$ be a prior distribution and $P$ the corresponding probability measure.
Then, we say that a prior is flat when for any $K>0,$ if we consider any $K$
points $\mathbf{x}_{1},...\mathbf{x}_{K}\in X$, any closed intervals
$G_{1},...,G_{K}\subset\mathbb{R}$, and any $g_{1},...,g_{K}\in\mathbb{R}$ it
holds that

$\displaystyle P(f(\cdot,{\mathbf{w}})\in\mathcal{F}\,:\,f(\mathbf{x}_{1},$
$\displaystyle{\mathbf{w}})\in G_{1},...,f(\mathbf{x}_{K},{\mathbf{w}})\in
G_{K})=$ $\displaystyle
P(f(\cdot,{\mathbf{w}})\in\mathcal{F}\,:\,f(\mathbf{x}_{1},{\mathbf{w}})\in
G_{1}+g_{1},...,f(\mathbf{x}_{1},{\mathbf{w}})\in G_{K}+g_{K})$

that is, if its finite-dimensional distributions are translation invariant.

According to Definition 2, a flat prior gives a uniform distribution on the
outcomes of $f(\cdot,{\mathbf{w}})$ for any countable set of input points.
Furthermore, as $P$ is uniquely defined through its behaviour on a countable
number of input points, Definition 2 implies that for any measurable set of
functions $F\subset\mathcal{F}$, function $g\in\mathcal{F}$, and set
$F_{g}=\\{f\in\mathcal{F}:\exists f^{\prime}\in F\,s.t.\,f=f^{\prime}+g\\}$,
it holds that $P(f(\cdot,{\mathbf{w}})\in F)=P(f(\cdot,{\mathbf{w}})\in
F_{g}),$ that is if we translate a set of function by a given function we
obtain the same probability. We should stress that, formally, a prior so
defined is not a distribution but an improper prior, as the scaling factor for
a translation invariant measure defined over an infinite support does not
exist (Box and Tiao, 2011). In practice one can approximate a flat prior with
a Gaussian distribution with variance that tends to infinity.

Notice that while Definition 2 defines a prior over the space of functions
induced by a neural network, in practice, often one would often like to define
a prior directly over the weights. In this context, we stress that, for an
infinitely-wide BNN, a Gaussian prior with large (finite) variance over the
weights induces a Gaussian prior with large variance over the space of
functions (Matthews et al., 2018). Thus, assuming a prior $p(\mathbf{w})$ with
large variance is in practice a good approximation of a flat prior as per
Definition 2. In any case, we should also stress that in this paper we will
rely on flat priors on a large data setting, where the influence of a flat
prior on the posterior is often limited (Christensen et al., 2010). In this
context, a flat prior assumption is necessary to guarantee that all possible
functions that can represent the neural network can equally influence the
posterior.

###  2.3 Adversarial Attacks for Bayesian Neural Networks

Given an input point $\mathbf{x}\in\mathcal{M}$ and a strength (i.e. maximum
perturbation magnitude) $\epsilon>0$, the worst-case adversarial perturbation
can be defined as the point $\tilde{\mathbf{x}}$ in the
$\epsilon$-neighbourhood around $\mathbf{x}$ that maximises the loss function
$L$:

$\tilde{\mathbf{x}}:=\operatorname*{arg\,max}_{\tilde{\mathbf{x}}:||\tilde{\mathbf{x}}-\mathbf{x}||\leq\epsilon}\langle
L(\tilde{\mathbf{x}},\mathbf{w})\rangle_{p\left(\mathbf{w}|D\right)}.$

If the network prediction on $\tilde{\mathbf{x}}$ differs from the original
prediction on $\mathbf{x}$, then we call $\tilde{\mathbf{x}}$ an _adversarial
example_. As $f(\mathbf{x},\mathbf{w})$ is non-convex, computing
$\tilde{\mathbf{x}}$ is a non-convex optimisation problem for which several
approximate solution methods have been proposed. In this paper we will
primarily focus on what arguably is the most commonly employed class among
them, i.e., gradient-based attacks, that is attacks that employ the loss
function gradient w.r.t. $\mathbf{x}$ to maximise the loss (Biggio and Roli,
2018). One such attacks is the Fast Gradient Sign Method (FGSM) (Goodfellow et
al., 2014) which works by approximating $\tilde{\mathbf{x}}$ by taking an
$\epsilon$-step in the direction of the sign of the gradient at $\mathbf{x}$.
In the context of BNNs, where attacks are against the posterior distribution,
applying FGSM yields

$\displaystyle\tilde{\mathbf{x}}$
$\displaystyle=\mathbf{x}+\epsilon\,\mathrm{sgn}\;\langle\nabla_{\mathbf{x}}L(\mathbf{x},\mathbf{w})\rangle_{p\left(\mathbf{w}|D\right)}\simeq\mathbf{x}+\epsilon\,\mathrm{sgn}\left(\sum_{i=1}^{n}\nabla_{\mathbf{x}}L(\mathbf{x},\mathbf{w}_{i})\right)$
(9)

where the final expression is a Monte Carlo approximation with samples
$\mathbf{w}_{i}$ drawn from the posterior $p(\mathbf{w}|D)$. Other gradient-
based attacks, as for example Projected Gradient Descent method (PGD) (Madry
et al., 2017), modify FGSM by taking consecutive gradient iterations or by
scaling the attack by the magnitude of the gradient. Crucially, however, they
all rely on the gradient vector to guide the attack. In the following
sections, we will show that the expected loss gradient in Equation (9)
vanishes to zero for an infinitely-wide Bayesian neural network achieving high
accuracy.

##  3 Gradient-Based Adversarial Attacks for Neural Networks

Equation (9) suggests a possible mechanism through which BNNs might acquire
robustness against adversarial attacks: averaging under the posterior might
lead to cancellations in the expectation of the gradient of the loss. It turns
out that this averaging property is intimately related to the geometry of the
data manifold $\mathcal{M}$. As a consequence, in order to study the
expectation of the gradient of the loss for BNNs, we first introduce results
that link the geometry of $\mathcal{M}$ to adversarial attacks.

We start with a trivial, yet important result: for a NN that achieves zero
loss on the whole data manifold $\mathcal{M}$, the loss gradient is constant
(and zero) along the data manifold for any $\mathbf{x}^{*}\in\mathcal{M}$.
Therefore, in order to have adversarial examples the dimension of the data
manifold $\mathcal{M}$ must necessarily be smaller than the dimension of the
ambient space, that is,
$\mathrm{dim}\left(\mathcal{M}\right)<\mathrm{dim}\left(X\right)=d,$ where
$\mathrm{dim}\left(\mathcal{M}\right)$ denotes the dimension of $\mathcal{M}$.

###### Lemma 3

Assume that $\forall\mathbf{x}\in\mathcal{M}$ $L(\mathbf{x},\mathbf{w})=0$,
that is $f(\mathbf{x},\mathbf{w})$ achieves zero loss on $\mathcal{M}$. Then,
if $f$ is vulnerable to gradient-based attacks at
$\mathbf{x}^{*}\in\mathcal{M}$,
$\mathrm{dim}\left(\mathcal{M}\right)<\mathrm{dim}\left(X\right)$ in a
neighbourhood of $\mathbf{x}^{*}$, i.e. $\mathcal{M}$ is locally homeomorphic
to a space of dimension smaller than the ambient space $X$.

Proof  By assumption
$\forall\mathbf{x}\in\mathcal{M},L(\mathbf{x},\mathbf{w})=0$, which implies
that the gradient of the loss is zero along the data manifold. However, if $f$
is vulnerable to gradient based attacks at $\mathbf{x}^{*}$ then the gradient
of the loss at $\mathbf{x}^{*}$ must be non-zero. Hence, there exists an open
neighbourhood $\mathcal{B}$ of $\mathbf{x}^{*}$ such that
$\mathcal{B}\not\subseteq\mathcal{M}$, which implies
$\mathrm{dim}(\mathcal{M})<\mathrm{dim}\left(X\right)$ locally around
$\mathbf{x}^{*}$.
Lemma 3 confirms the widely held conjecture that adversarial attacks may
originate from degeneracies of the data manifold (Goodfellow et al., 2014;
Fawzi et al., 2018). In fact, it has been already empirically noticed (Khoury
and Hadfield-Menell, 2018) that adversarial perturbations often arise in
directions normal to the data manifold. The suggestion that lower-dimensional
data structures might be ubiquitous in NN problems is also corroborated by
recent results (Goldt et al., 2019) showing that the characteristic training
dynamics of NNs are intimately linked to data lying on a lower-dimensional
manifold. Notice that the implication is only one way; it is perfectly
possible for the data manifold to be low dimensional and still not vulnerable
at many points.

We note that, as discussed in Section 2.1, at convergence of the training
algorithm and in the limit of infinitely-many data, infinitely-wide neural
networks are guaranteed to achieve zero loss on the data manifold, satisfying
the assumption of Lemma 3. As a result, once an infinitely-wide NN is fully
trained, for any $\mathbf{x}\in\mathcal{M}$ the gradient of the loss function
is orthogonal to the data manifold as it is zero along the data manifold,
i.e.,
$\nabla_{\mathbf{x}}L(\mathbf{x},\mathbf{w})=\nabla^{\perp}_{\mathbf{x}}L(\mathbf{x},\mathbf{w})$,
where $\nabla^{\perp}_{\mathbf{x}}$ denotes the gradient projected into the
normal subspace of $\mathcal{M}$ at $\mathbf{x}$. Note that for a given NN,
$\nabla^{\perp}_{\mathbf{x}}L(\mathbf{x},\mathbf{w})$ is in general non-zero
even if the network achieves zero loss on $\mathcal{M},$ thus explaining the
existence of adversarial examples even for very accurate classifiers.
Crucially, in Section 3.1 we show that for BNNs, when averaged w.r.t. the
posterior distribution, the orthogonal gradient vanishes.

###  3.1 A Symmetry Property of Neural Networks

Before considering the BNN case, in Proposition 5 below we show a symmetry
property of neural networks: given a neural network, we can always find an
infinitely-wide NN that has the same loss but opposite orthogonal gradient. In
order to prove this result, we first introduce Lemma 4, which is a
generalization of the submanifold extension lemma and a key result we
leverage. It proves that any smooth function defined on a submanifold
$\mathcal{M}$ can be extended to the ambient space, in such a way that the
choice of the derivatives orthogonal to the submanifold is arbitrary.

######  Lemma 4 (Anders et al. (2020))

Let $T_{\mathbf{x}}\mathcal{M}$ be the tangent space of $\mathcal{M}$ at a
point $\mathbf{x}\in\mathcal{M}$. Let
$V=\sum_{i=\mathrm{dim}\left(\mathcal{M}\right)+1}^{d}v^{i}\partial_{i}$ be a
conservative vector field along $\mathcal{M}$ which assigns a vector in
$T_{\mathbf{x}}\mathcal{M}^{\perp}$ for each $\mathbf{x}\in\mathcal{M}$. For
any smooth function $f^{true}:\mathcal{M}\to\mathbb{R}$ there exists a smooth
extension $F:X\to\mathbb{R}$ such that

$F|_{\mathcal{M}}=f^{true},$

where $F|_{\mathcal{M}}$ denotes the restriction of $F$ to the submanifold
$\mathcal{M}$, and such that the derivative of the extension $F$ is

$\nabla_{\mathbf{x}}F(\mathbf{x})=(\nabla_{1}f^{true}(\mathbf{x}),\ldots,\nabla_{\mathrm{dim}\left(\mathcal{M}\right)}f^{true}(\mathbf{x}),v^{\mathrm{dim}\left(\mathcal{M}\right)+1}(\mathbf{x}),\ldots,v^{d}(\mathbf{x}))$

for all $\mathbf{x}\in\mathcal{M}$.

Notice that in Lemma 4, in $\nabla_{\mathbf{x}}F(\mathbf{x})$, we pick the
local coordinates at $\mathbf{x}\in\mathcal{M}$, such that the first set of
components parametrises the data manifold. We stress that, as $\mathcal{M}$ is
smooth, this is without any loss of generality (Lee, 2013). Lemma 4, together
with the universal approximation capabilities of NNs (Hornik, 1991), is
employed in Proposition 5 to show that for any possible value $\mathbf{v}$ of
the orthogonal gradient to the data manifold in a point, there exists at least
two (possibly not unique) different weight vectors that achieve zero loss and
have orthogonal gradients respectively equal to $\mathbf{v}$ and
$-\mathbf{v}$.

###### Proposition 5

Consider a NN $f$ with $M+1$ layers, last hidden layer of size $n_{M}$ and an
input $\mathbf{x}\in\mathcal{M}$. Then, in the limit of $n_{M}\to\infty$ (size
of last hidden layer going to infinity), for any smooth function
${f}^{true}:\mathcal{M}\to\mathbb{R}$ and vector
$\mathbf{v}\in\mathbb{R}^{\mathrm{dim}\left(X\right)-\mathrm{dim}\left(\mathcal{M}\right)}$,
there exist $\mathbf{w}_{1},\mathbf{w}_{2}$ such that

$\displaystyle
f(\cdot,\mathbf{w}_{1})|_{\mathcal{M}}={f}^{true}=f(\cdot,\mathbf{w}_{2})|_{\mathcal{M}}$
(10)
$\displaystyle\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x},\mathbf{w}_{1})=\mathbf{v}=-\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x},\mathbf{w}_{2}).$
(11)

Proof  From Lemma 4 we know that there exist smooth extensions $F^{+}$ and
$F^{-}$ of $f^{true}$ to the embedding space such that
$\nabla_{\mathbf{x}}^{\perp}F^{+}(\mathbf{x})=\mathbf{v}=-\nabla_{\mathbf{x}}^{\perp}F^{-}(\mathbf{x})$.
As a consequence, to conclude the proof it suffices to apply Theorem 3 in
(Hornik, 1991) that guarantees that infinitely-wide neural networks are
_uniformly 1-dense_ on compacts in $\mathcal{C}^{1}(X)$, under the assumptions
of smooth, bounded, and non-constant activation functions. Specifically, for
any $F\in\mathcal{C}^{1}(X)$ and $\epsilon>0$, for any compact
$X^{\prime}\subseteq X$ there exists a set of weights $\mathbf{w}$ s.t.

$\max\Big{\\{}\sup_{\mathbf{x}\in
X^{\prime}}||F(\mathbf{x})-f(\mathbf{x},\mathbf{w})||_{\infty},\sup_{\mathbf{x}\in
X^{\prime}}||\nabla F(\mathbf{x})-\nabla
f(\mathbf{x},\mathbf{w})||_{\infty}\Big{\\}}\leq\epsilon.$

As $F^{+},F^{-}\in\mathcal{C}^{1}(X)$, this concludes the proof.

Note that by the chain rule, the gradient of the loss is proportional to the
gradient of the NN. As a consequence, Proposition 5 guarantees that, for
infinitely-wide NNs, for any set of weights achieving the minimum loss, then
there exists another set of weights with same loss and opposite orthogonal
gradient of the loss w.r.t. the input. This suggests that by averaging over
these configurations of the weights one can achieve a robust model that has
vanishing expected orthogonal gradient. In the next Section, in Theorem 6 we
show this is indeed the case. However, we should already emphasize that such a
result does not hold by simply averaging the set of weights w.r.t. any
distribution. Intuitively, for this result to hold, it is required that each
set of weights achieving a given gradient value has the same measure (w.r.t.
the posterior) as the set of weights with the same loss and opposite
orthogonal gradient value.

##  4 Adversarial Robustness via Bayesian Averaging

We are now ready to state Theorem 6, where we show that under the assumption
of a flat prior, the posterior of an infinitely-wide BNN achieving zero loss
has vanishing orthogonal gradients.

###### Theorem 6

Let $f(\mathbf{x},\mathbf{w})$ be an infinitely-wide BNN trained on a data set
$D_{N}$ composed by $N$ data points with true underlying function
$f^{true}:\mathcal{M}\to\mathbb{R}$. Assume that:

  * •

The prior is flat.

  * •

For $N\to\infty$ the posterior distribution of the BNN converges weakly to a
distribution that achieves zero loss with probability $1$, i.e.,

$P(f(\cdot,\mathbf{w})|D_{N})\xrightarrow{d}P(f\mid D_{\infty}),$

where $P(f\mid D_{\infty})$ is such that

$P\Big{(}\sup_{\mathbf{x}\in\mathcal{M}}|f(\mathbf{x})-f^{true}(\mathbf{x})|=0\,\Big{|}\,D_{\infty}\Big{)}=1.$
(12)

Then, for any $\mathbf{x}\in\mathcal{M}$ it holds that

$\langle\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x},\mathbf{w})\rangle_{p(f(\cdot,\mathbf{w})\mid{D_{N})}}\to
0.$ (13)

Proof  Without any loss of generality assume that
$\mathrm{dim}\left(X\right)-\mathrm{dim}\left(\mathcal{M}\right)=1$.777In the
case that $\mathrm{dim}\left(X\right)-\mathrm{dim}\left(\mathcal{M}\right)>1$,
because of the linearity of the partial derivatives, any component of
$\langle\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})\rangle_{p(f\mid{D_{\infty})}}$
can be treated analogously to the case
$\mathrm{dim}\left(X\right)-\mathrm{dim}\left(\mathcal{M}\right)=1$.  Consider
the operator

$\mathcal{G}_{\mathbf{x}}:f\mapsto\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x}),$

which is linear and bounded under the boundeness assumption on the derivatives
of $f$. By the Portmanteau theorem, $\mathcal{G}_{\mathbf{x}}$ preserves weak
convergence, hence

$\langle\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x},\mathbf{w})\rangle_{p(f(\cdot,\mathbf{w})\mid{D_{N})}}\to\langle\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})\rangle_{p(f\mid{D_{\infty})}}.$
(14)

Then, to prove Equation (13) it is enough to show that

$\langle\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})\rangle_{p(f\mid{D_{\infty})}}=0.$

In order to do that, we proceed as follows. By definition of expectation we
have that

$\displaystyle\langle\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})\rangle_{p(f\mid{D_{\infty})}}$
$\displaystyle=\int_{\mathbf{v}\in\mathbb{R}}\displaylimits\mathbf{v}\,p{(\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})=\mathbf{v}|D_{\infty})}d\mathbf{v}\noindent$
$\displaystyle=\int_{\mathbf{v}\in\mathbb{R}_{>0}}\displaylimits\mathbf{v}\,p{(\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})=\mathbf{v}|D_{\infty})}d\mathbf{v}\,+\,\int_{\mathbf{v}\in\mathbb{R}_{<0}}\displaylimits\mathbf{v}\,p{(\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})=\mathbf{v}|D_{\infty})}d\mathbf{v}.$
(15)

If we now can show that for any $\mathbf{v}\in\mathbb{R}$ and
$\epsilon\in\mathbb{R}_{>0}$,
$P(\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})\in[\mathbf{v}-\epsilon,\mathbf{v}+\epsilon]\mid
D_{\infty}):=\int_{[\mathbf{v}-\epsilon,\mathbf{v}+\epsilon]}{p{(\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})=v\mid
D_{\infty})}}d{v}$ is constant w.r.t. $\mathbf{v}$, then this guarantees that
Eqn (15) is zero. In fact, by application of the fundamental theorem of
calculus, the above implies that for any $\mathbf{v},$
$p(\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})=\mathbf{v}\mid
D_{\infty})=p(\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})=-\mathbf{v}\mid
D_{\infty})$ almost surely (i.e. with probability $1$), hence terms in Eqn
(15) cancel out. We prove that
$P(\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})\in[\mathbf{v}-\epsilon,\mathbf{v}+\epsilon]\mid
D_{\infty})$ is constant wrt $\mathbf{v}$ in what follows.

By assumption it holds that any $f\sim P(\cdot|D_{\infty})$ is such that
$\forall\bar{\mathbf{x}}\in\mathcal{M},f(\bar{\mathbf{x}})=f^{true}(\bar{\mathbf{x}})$
almost surely. Hence, to conclude, being our prior invariant to translations
(see Definition 2), it is enough to show that for any
$\mathbf{v},\mathbf{v}^{\prime}$ sets

$F_{\mathbf{v}}=\\{g\in\mathcal{F}~{}:~{}\forall\bar{\mathbf{x}}\in\mathcal{M},\,g(\bar{\mathbf{x}})=f^{true}(\bar{\mathbf{x}})\wedge\nabla_{\mathbf{x}}^{\perp}g(\mathbf{x})\in[\mathbf{v}-\epsilon,\mathbf{v}+\epsilon]\\}$
$F_{\mathbf{v}^{\prime}}=\\{g\in\mathcal{F}~{}:~{}\forall\bar{\mathbf{x}}\in\mathcal{M},\,g(\bar{\mathbf{x}})=f^{true}(\bar{\mathbf{x}})\wedge\nabla_{\mathbf{x}}^{\perp}g(\mathbf{x})\in[\mathbf{v}^{\prime}-\epsilon,\mathbf{v}^{\prime}+\epsilon]\\}$

are equal up to a translation. This can be shown as follows. Using Lemma 4 we
extend $f^{true}$ to a smooth function $f_{\mathbf{v}}$ in $X$ such that its
orthogonal gradient in $\mathbf{x}\in\mathcal{M}$ equals $\mathbf{v}$, i.e.
$\nabla_{\mathbf{x}}^{\perp}f_{\mathbf{v}}(\mathbf{x})=\mathbf{v}.$ Then,
because of the linearity of the gradient operator, any other function $g$ on
$X$ that is equal to $f^{true}$ when restricted to $\mathcal{M}$ and such that
$\nabla_{\mathbf{x}}^{\perp}g(\mathbf{x})\in[\mathbf{v}-\epsilon,\mathbf{v}+\epsilon]$
can be written as

$g(\mathbf{x})=f_{\mathbf{v}}(\mathbf{x})+\bar{k}(\mathbf{x}),$

where $\bar{k}$ is a function that has the orthogonal gradient in $\mathbf{x}$
within $[-\epsilon,\epsilon]$ and that is $0$ on $\mathcal{M}$, i.e.,

$\displaystyle
k\in\mathcal{K}=\\{\bar{k}~{}:~{}\forall\bar{\mathbf{x}}\in\mathcal{M},\bar{k}(\bar{\mathbf{x}})=0\wedge\nabla_{\mathbf{x}}^{\perp}\bar{k}(\mathbf{x})\in[-\epsilon,\epsilon]\\}.$
(16)

Let’s consider a function $f_{\mathbf{v}^{\prime}}$ built similarly as
$f_{\mathbf{v}}$, then we obtain that all $g^{\prime}(\mathbf{x})$ equal to
the true function and such that
$\nabla_{\mathbf{x}}^{\perp}g^{\prime}(\mathbf{x})=[\mathbf{v}^{\prime}-\epsilon,\mathbf{v}^{\prime}+\epsilon]$
can be written as

$g^{\prime}(\mathbf{x})=f_{\mathbf{v}^{\prime}}(\mathbf{x})+\bar{k}(\mathbf{x}),$

with $\bar{k}\in\mathcal{K}$, where $\mathcal{K}$ is as in Equation (16). It
then follows that the sets $F_{\mathbf{v}}$ and $F_{\mathbf{v}^{\prime}}$ are
such that
$F_{\mathbf{v}}=F_{\mathbf{v}^{\prime}}+(f_{\mathbf{v}}-f_{\mathbf{v}^{\prime}}).$
Hence,
$P(\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})\in[\mathbf{v}-\epsilon,\mathbf{v}+\epsilon]\mid
D_{\infty})=P(\nabla_{\mathbf{x}}^{\perp}f(\mathbf{x})\in[\mathbf{v}^{\prime}-\epsilon,\mathbf{v}^{\prime}+\epsilon]\mid
D_{\infty}).$ Thus, concluding the proof.

The proof of Theorem 6 relies on two main assumptions: 1) the prior is flat,
which guarantees that probabilities are invariant with respect to
translations, thus allowing for perfect cancellation in expectation, 2) the
limiting distribution of an infinitely-wide BNN has zero loss with probability
$1$, which allows us to only focus on the set of functions that are equal to
the true functions on the data manifold almost surely. While we will discuss
the implication of the flat prior assumption in Section 5, we would like to
stress in here that because of the universal approximation properties of
neural networks, the second assumption in Theorem 6 is a standard assumption
to require, as it guarantees that our posterior converges to the true function
with infinitely many data. For example, under mild assumptions, a similar
requirement has been explicitly shown to hold for infinitely-wide feed-forward
Bayesian neural networks with one hidden layer (Lee, 2000).

###### Remark 7

A key aspect in the proof of Theorem 6 is that when the data manifold has a
dimension smaller than the ambient space, then there are multiple weight sets
minimizing the loss (see Lemma 4) and the BNN posterior averages between them
achieving vanishing orthogonal gradients. Note that, if there were just one
set of minimizing weights, we could have employed standard results of
asymptotic statistics, such as the Bernestein von Mises (BVM) Theorem to study
the asymptotic behaviour of the BNN (Vaart, 2000). However, a standard
application of the BVM Theorem would have led to a convergence to a Gaussian
distribution centered in the maximum likelihood parameters and with vanishing
variance: this would have implied that a BNN is robust if and only if also the
correspondent deterministic NN with fixed (deterministic) weights is robust,
which we know not to be the case. We also confirm this empirically in Figure 6
in the Appendix.

Figure 1:  Expected loss gradients components on 100 two-dimensional test
points from the Half Moons dataset (Rozza et al., 2014) (both partial
derivatives of the loss function are shown). We used a collection of HMC BNNs,
by varying the number of hidden neurons (a different number in each subplot)
and training points (different color and size of dots). Each dot represents a
different NN. Only models with test accuracy greater than 80% were taken into
account.

##  5 Consequences and Limitations of our Results

The results in the previous section, and in particular of Theorem 6, have the
natural consequence of protecting BNNs against all gradient-based attacks, due
to the vanishing average of the expectation of the gradients in the limit. Its
proof also sheds light on a number of observations made in recent years.
Before moving on to empirically validating the theorem in the finite-data and
finite-width case, it is worth reflecting on some of its implications and
limitations:

  * •

Theorem 6 is proven under the assumption of flat priors needed to require
perfect gradient cancellation; in practice, unless the priors are too
informative with restrictive support, we do not expect a major deviation from
the idealised case. This is confirmed both from our experimental results and
by the fact that in the limit of large data-set the posterior is less
influenced by the choice of the prior (Vaart, 2000).

  * •

Theorem 6 holds in a specific thermodynamic limit, however we expect the
averaging effect of BNN gradients to still provide considerable protection in
conditions where the network architecture and the data amount lead to high
accuracy and strong expressivity. In practice, high accuracy might be a good
indicator of robustness for BNNs. An empirical confirmation of this result is
shown in Figure 1, we examine the impact of the assumptions made in Theorem 6:
by exploiting a setting in which we have access to the data-generating
distribution, the half-moons dataset (Rozza et al., 2014). We show that the
magnitude of the expectation of each component of the gradient shrinks as we
increase the number of network’s parameters and the number of training points.

  * •

Theorem 6 holds when the ensemble is drawn from the true posterior;
nevertheless cheaper approximate Bayesian inference methods which retain
ensemble predictions such as VI may still in practice provide good protection.

  * •

Gaussian Processes (Williams and Rasmussen, 2006) are equivalent to infinitely
wide BNNs. Theorem 6 provides theoretical backing to recent empirical
observations of their adversarial robustness (Cardelli et al., 2019b; Grosse
et al., 2021; Patane et al., 2022).

  * •

While the Bayesian posterior ensemble may not be the only randomization to
provide protection, it is clear that some simpler randomizations such as
bootstrap will be ineffective, as noted empirically by Bekasov and Murray
(2018). This is because bootstrap resampling introduces variability along the
data manifold, rather than in orthogonal directions. In this sense, bootstrap
clearly cannot be considered a Bayesian approximation, especially when the
data distribution has zero measure support w.r.t. the ambient space.
Similarly, we do not expect gradient smoothing approaches to be successful
(Athalye et al., 2018), since the type of smoothing performed by Bayesian
inference is specifically informed by the geometry of the data manifold.

  * •

Theorem 6 only guarantees protection against gradient-based adversarial
attacks. As a consequence, it is not clear if the robustness properties of
BNNs also extend to non-gradient based attacks. In Section 6.4 we empirically
show that the vanishing gradient properties of BNNs also guarantee robustness
to non-gradient based attacks.

##  6 Empirical Results

Figure 2: The expected loss gradients of BNNs exhibit a vanishing behaviour
when increasing the number of samples from the posterior predictive
distribution. We show example images from MNIST (top row) and Fashion MNIST
(bottom row) and their expected loss gradients w.r.t. BNNs trained with HMC
(left) and VI (right). To the right of the images, we plot a heat map of
gradient values.

In this section we empirically investigate our theoretical findings on
different BNNs. We train a variety of BNNs on the MNIST and Fashion MNIST
(Xiao et al., 2017) datasets, and evaluate their posterior distributions using
HMC and VI approximate inference methods. In Section 6.1, we experimentally
verify the validity of the zero-averaging property of gradients implied by
Theorem 6, and discuss its implications on the behaviours of FGSM and PGD
attacks on BNNs in Section 6.2. In Section 6.3 we analyse the relationship
between robustness and accuracy on thousands of different NN architectures,
comparing the results obtained by Bayesian and by deterministic training.
Further, in Section 6.4 we investigate the robustness of BNNs on a gradient-
free adversarial attack (Athalye et al., 2018). Details on the experimental
settings and BNN training parameters can be found in Appendix B.

###  6.1 Evaluation of the Gradient of the Loss for BNNs

We investigate the vanishing behavior of input gradients - established by
Theorem 6 for the limit regime - in the finite, practical settings, that is
with a finite number of training data and with finite-width BNNs.
Specifically, for each inference method we perform hyperparameter tuning and
select the architecture achieving the highest test accuracy: we train a two
hidden layers BNN (with 1024 neurons per layer) with HMC and a three hidden
layers BNN (512 neurons per layer) with VI. These achieve approximately $95\%$
test accuracy on MNIST and $89\%$ on Fashion MNIST when trained with HMC; as
well as $95\%$ and $92\%$, respectively, when trained with VI. More details
about the hyperparameters used for training can be found in Appendix B.

Figure 2 depicts anecdotal evidence on the behaviour of the component-wise
expectation of the loss gradient as more samples from the posterior
distribution are incorporated into the BNN predictive distribution. Similarly
to how in Figure 1 for the half-moons dataset we observe that the gradient of
the loss goes to zero when increasing number of training points and number of
parameters, here we have that, as the number of samples taken from the
posterior distribution of $\mathbf{w}$ increases, all the components of the
gradient approach zero. Notice that the gradient of the individual NNs (that
is those with just one sampled weight vector), is far away from being zero. As
shown in Theorem 6, it is only through the Bayesian averaging of ensemble
predictors that the gradients cancel out.

This is confirmed in Figure 3, where we provide a systematic analysis of the
aggregated gradient convergence properties on $1$k test images for MNIST and
Fashion-MNIST. Each dot shown in the plots represents a component of the
expected loss gradient from each one of the images, for a total of $784$k
points used to visually approximate the empirical distribution of the
component-wise expected loss gradient. For both HMC and VI the magnitude of
the gradient components drops as the number of samples increases, and tends to
stabilize around zero already with 100 samples drawn from the posterior
distribution, suggesting that the conditions laid down in Theorem 6 are
approximately met by the BNN analysed here. Notice that the gradients computed
on HMC trained networks drops more quickly and towards a smaller value
compared to VI trained networks. This is in accordance to what is discussed in
Section 5, as VI introduces additional approximations in the Bayesian
posterior computation.

Figure 3:  The components of the expected loss gradients approach zero as the
number of samples from the posterior distribution increases. For each number
of samples, the figure shows $784$ gradient components for $1$k different test
images, from both the MNIST and Fashion MNIST datasets. The gradients are
computed on HMC (a) and VI (b) trained BNNs.

###  6.2 Gradient-Based Attacks for BNNs

The fact that gradient cancellation occurs in the limit does not directly
imply that BNNs’ predictions are robust to gradient-based attacks in the
finite case. For example, FGSM attacks are crafted such that the direction of
the manipulation is given only by the sign of the expectation of the loss
gradient and not by its magnitude. Thus, even if the entries of the
expectation drop to an infinitesimal magnitude but maintains a meaningful
sign, then FGSM could potentially produce effective attacks. In order to test
the implications of vanishing gradients on the robustness of the posterior
predictive distribution against gradient-based attacks, we compare the
behaviour of FGSM and PGD888With 15 iterations and 1 restart. attacks to a
randomly devised attack.

Dataset/Method | Rand | FGSM | PGD
---|---|---|---
MNIST/HMC | 0.850 | 0.960 | 0.970
MNIST/VI | 0.956 | 0.936 | 0.938
Fashion/HMC | 0.812 | 0.848 | 0.826
Fashion/VI | 0.744 | 0.834 | 0.916
Table 1: Adversarial robustness of BNNs trained with HMC and VI with respect
to the random attack (Rand), FGSM and PGD.

Namely, the random attack mimics a randomised version of FGSM in which the
sign of the attack is sampled at random. In practice, we perturb each
component of a test image by a random value in $\\{-\epsilon,\epsilon\\}$. In
Table 1 we compare the effectiveness of FGSM, PGD and of the random attack and
report the adversarial robustness for 500 images. For each image, we compute
the expected gradient using 250 posterior samples. The attacks were performed
with $\epsilon=0.1$ and using the categorical cross-entropy loss function. In
almost all cases, we see that the random attack outperforms the gradient-based
attacks, showing how the vanishing behaviour of the gradient makes FGSM and
PGD attacks ineffective. Furthermore, in Table 2 in Appendix A.2 we also run
the same evaluation for when the same network employed in Table 1 is trained
with SGD and deep ensembles. In both cases both FGSM and PGD are effective,
suggesting how simply model averaging and mini-batches are not enough to
achieve a robust model.

###  6.3 Robustness Accuracy Analysis in Deterministic and Bayesian Neural
Networks

In Section 5, we noticed that as a consequence of Theorem 6, high accuracy
might be related to high robustness to gradient-based attacks for BNNs.
Notice, that this would run counter to what has been observed for
deterministic NNs trained with SGD (Su et al., 2018).

In this section, we look at an array of more than 1000 BNNs with different
hyperparameters trained with HMC and VI on MNIST and Fashion-MNIST. We
experimentally evaluate their accuracy/robustness trade-off on FGSM attacks as
compared to that of the same neural network architectures trained via standard
(i.e., non-Bayesian) SGD based methods. For the robustness evaluation we
consider the average difference in the softmax prediction between the original
test points and the crafted adversarial input, as this provides a quantitative
and smooth measure of adversarial robustness that is closely related with mis-
classification ratios (Cardelli et al., 2019a). That is, for a collection of
$N$ test point, we compute

$\displaystyle\frac{1}{N}\sum_{j=1}^{N}\big{|}\langle
f(\mathbf{x}_{j},\mathbf{w})\rangle_{p(\mathbf{w}|D)}-\langle
f(\tilde{\mathbf{x}}_{j},\mathbf{w})\rangle_{p(\mathbf{w}|D)}\big{|}_{\infty}.$
(17)

Figure 4: Robustness-Accuracy trade-off on MNIST (first row) and Fashion MNIST
(second row) for BNNs trained with HMC (a), VI (b) and SGD (blue dots). While
a trade-off between accuracy and robustness occur for deterministic NNs,
experiments on HMC show a positive correlation between accuracy and
robustness. The boxplots show the correlation between model capacity and
robustness. Different attack strength ($\epsilon$) are used for the three
methods accordingly to their average robustness.

The results of the analysis are plotted in Figure 4 for MNIST and Fashion
MNIST. Each dot in the scatter plots represents the results obtained for each
specific network architecture trained with SGD (blue dots), HMC (pink dots in
plots (a)) and VI (pink dots in plots (b)). As already reported in the
literature (Su et al., 2018) we observe a marked trade-off between accuracy
and robustness (i.e., 1 - softmax difference) for high-performing
deterministic networks. Interestingly, this trend is fully reversed for BNNs
trained with HMC (plots (a) in Figure 4) where we find that as networks become
more accurate, they additionally become more robust to FGSM attacks as well.
We further examine this trend in the boxplots that represent the effect that
the network width has on the robustness of the resulting posterior. We find
the existence of an increasing trend in robustness as the number of neurons in
the network is increased. This is in line with our theoretical findings, i.e.,
as the BNN approaches the infinite width limit, the conditions for Theorem 6
are approximately met and the network is protected against gradient-based
attacks.

On the other hand, the trade-off behaviours are less obvious for the BNNs
trained with VI and on Fashion-MNIST. In particular, in plot (b) of Figure 4
we find that, similarly to the deterministic case, also for BNNs trained with
VI, robustness seems to have a negative correlation with accuracy.
Furthermore, for VI we observe that there is some trend dealing with the size
of the model, but we only observe this in the case of VI trained on MNIST
where it can be seen that model robustness may increase as the width of the
layers increases, but this can also lead to poor robustness (which may be
indicative of mode collapse).

###  6.4 Gradient-Free Adversarial Attacks

In this section, we empirically evaluate the most accurate BNN posteriors on
MNIST and FashionMNIST from Figure 4 against gradient-free adversarial
attacks. Specifically, we consider ZOO (Chen et al., 2017), a gradient-free
adversarial attack based on a finite-difference approximation of the gradient
of the loss obtained by querying the neural network (in the BNN case the
attacker queries the posterior distribution). In particular, we selected ZOO
because it has been shown to be effective even when tested on networks that
purposefully obfuscate their gradients or have vanishing gradients (Athalye et
al., 2018). In Figure 5 we observe that similarly to gradient-based methods,
ZOO is substantially less effective on BNNs compared to deterministic NNs in
all cases, with BNNs again achieving both high accuracy and high robustness
simultaneously. Furthermore, once again HMC is more robust to the attack than
VI, which is in turn substantially more robust than deterministic NNs. This
suggests how, similarly to what observed in the previous subsections, a more
accurate posterior distribution may lead to a more robust model also to
gradient-free adversarial attacks.

Figure 5: Gradient-free adversarial attacks on BNNs display similar behavior
to gradient-based attacks. We evaluate the more accurate networks from Figure
4 with gradient-free attacks on MNIST (first row) and Fashion MNIST (second
row) for BNNs trained with HMC (left column), VI (center column), and SGD
(right column). We use the same attack parameters as in the Figure 4, but use
ZOO as an attack method.

##  7 Conclusions

The quest for robust, data-driven models is an essential component towards the
construction of AI-based technologies. In this respect, we believe that the
fact that Bayesian ensembles of NNs can evade a broad class of adversarial
attacks will be of great relevance. While promising, this result comes with
some significant limitations. First, our theoretical results hold in a
thermodynamic limit, which is never realised in practice. More worryingly, we
currently have no rigorous diagnostics to understand how near we are to the
limit case, and can only reason about this empirically. Secondly, and perhaps
more importantly, performing Bayesian inference in large non-linear models is
extremely challenging. In fact, while in our experiments cheaper
approximations, such as VI, also enjoyed a degree of adversarial robustness,
albeit reduced, there are no guarantees that this will hold in general. To
this end, we hope that this result will spark renewed interest in the pursuit
of efficient Bayesian inference algorithms.

## References

  * Adler (2010) R. J. Adler.  _The geometry of random fields_.  SIAM, 2010.
  * Anders et al. (2020) C. Anders, P. Pasliev, A.-K. Dombrowski, K.-R. Müller, and P. Kessel.  Fairwashing explanations with off-manifold detergent.  In _International Conference on Machine Learning_ , pages 314–323. PMLR, 2020.
  * Athalye et al. (2018) A. Athalye, N. Carlini, and D. Wagner.  Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples.  _arXiv preprint arXiv:1802.00420_ , 2018.
  * Barber (2012) D. Barber.  _Bayesian reasoning and machine learning_.  Cambridge University Press, 2012.
  * Bekasov and Murray (2018) A. Bekasov and I. Murray.  Bayesian adversarial spheres: Bayesian inference and adversarial examples in a noiseless setting.  _arXiv preprint arXiv:1811.12335_ , 2018.
  * Berrada et al. (2021) L. Berrada, S. Dathathri, R. Stanforth, R. Bunel, J. Uesato, S. Gowal, M. P. Kumar, et al.  Verifying probabilistic specifications with functional lagrangians.  _arXiv preprint arXiv:2102.09479_ , 2021.
  * Biggio and Roli (2018) B. Biggio and F. Roli.  Wild patterns: Ten years after the rise of adversarial machine learning.  _Pattern Recognition_ , 84:317–331, 2018.
  * Bishop (2006) C. M. Bishop.  _Pattern Recognition and Machine Learning (Information Science and Statistics)_.  Springer-Verlag, Berlin, Heidelberg, 2006.  ISBN 0387310738.
  * Bishop et al. (1995) C. M. Bishop et al.  _Neural networks for pattern recognition_.  Oxford university press, 1995.
  * Blundell et al. (2015) C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra.  Weight uncertainty in neural networks.  _arXiv preprint arXiv:1505.05424_ , 2015.
  * Box and Tiao (2011) G. E. Box and G. C. Tiao.  _Bayesian inference in statistical analysis_.  John Wiley & Sons, 2011.
  * Carbone et al. (2020) G. Carbone, M. Wicker, L. Laurenti, A. Patane, L. Bortolussi, and G. Sanguinetti.  Robustness of bayesian neural networks to gradient-based attacks.  _Advances in Neural Information Processing Systems_ , 33:15602–15613, 2020.
  * Cardelli et al. (2019a) L. Cardelli, M. Kwiatkowska, L. Laurenti, N. Paoletti, A. Patane, and M. Wicker.  Statistical guarantees for the robustness of bayesian neural networks.  In _IJCAI_ , 2019a.
  * Cardelli et al. (2019b) L. Cardelli, M. Kwiatkowska, L. Laurenti, and A. Patane.  Robustness guarantees for bayesian inference with gaussian processes.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 7759–7768, 2019b.
  * Carlini and Wagner (2017) N. Carlini and D. Wagner.  Adversarial examples are not easily detected: Bypassing ten detection methods.  In _Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security_ , pages 3–14, 2017.
  * Chen et al. (2017) P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh.  Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models.  In _Proceedings of the 10th ACM workshop on artificial intelligence and security_ , pages 15–26, 2017.
  * Christensen et al. (2010) R. Christensen, W. Johnson, A. Branscum, and T. E. Hanson.  _Bayesian ideas and data analysis: an introduction for scientists and statisticians_.  CRC press, 2010.
  * Cybenko (1989) G. Cybenko.  Approximation by superpositions of a sigmoidal function.  _Mathematics of control, signals and systems_ , 2(4):303–314, 1989.
  * Fawzi et al. (2018) A. Fawzi, H. Fawzi, and O. Fawzi.  Adversarial vulnerability for any classifier.  In _Advances in Neural Information Processing Systems_ , pages 1178–1187, 2018.
  * Feinman et al. (2017) R. Feinman, R. R. Curtin, S. Shintre, and A. B. Gardner.  Detecting adversarial samples from artifacts.  _arXiv preprint arXiv:1703.00410_ , 2017.
  * Gal and Smith (2018) Y. Gal and L. Smith.  Sufficient conditions for idealised models to have no adversarial examples: a theoretical and empirical study with bayesian neural networks.  _arXiv preprint arXiv:1806.00667_ , 2018.
  * Goldt et al. (2019) S. Goldt, M. Mézard, F. Krzakala, and L. Zdeborová.  Modelling the influence of data structure on learning in neural networks.  _arXiv preprint arXiv:1909.11500_ , 2019.
  * Goodfellow et al. (2014) I. J. Goodfellow, J. Shlens, and C. Szegedy.  Explaining and harnessing adversarial examples.  _arXiv preprint arXiv:1412.6572_ , 2014.
  * Grosse et al. (2018) K. Grosse, D. Pfaff, M. T. Smith, and M. Backes.  The limitations of model uncertainty in adversarial settings.  _arXiv preprint arXiv:1812.02606_ , 2018.
  * Grosse et al. (2021) K. Grosse, M. T. Smith, and M. Backes.  Killing four birds with one gaussian process: the relation between different test-time attacks.  In _2020 25th International Conference on Pattern Recognition (ICPR)_ , pages 4696–4703. IEEE, 2021.
  * Hornik (1991) K. Hornik.  Approximation capabilities of multilayer feedforward networks.  _Neural networks_ , 4(2):251–257, 1991.
  * Khoury and Hadfield-Menell (2018) M. Khoury and D. Hadfield-Menell.  On the geometry of adversarial examples.  _CoRR_ , abs/1811.00525, 2018.  URL http://arxiv.org/abs/1811.00525.
  * Lakshminarayanan et al. (2017) B. Lakshminarayanan, A. Pritzel, and C. Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles, 2017.
  * Lee (2000) H. K. Lee.  Consistency of posterior distributions for neural networks.  _Neural Networks_ , 13(6):629–642, 2000.
  * Lee (2013) J. M. Lee.  Smooth manifolds.  In _Introduction to smooth manifolds_ , pages 1–31. Springer, 2013\.
  * Li and Gal (2017) Y. Li and Y. Gal.  Dropout inference in bayesian neural networks with alpha-divergences.  In _Proceedings of the 34th International Conference on Machine Learning-Volume 70_ , pages 2052–2061. JMLR. org, 2017.
  * Liu et al. (2018) X. Liu, Y. Li, C. Wu, and C.-J. Hsieh.  Adv-bnn: Improved adversarial defense through robust bayesian neural network.  _arXiv preprint arXiv:1810.01279_ , 2018.
  * Madry et al. (2017) A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu.  Towards deep learning models resistant to adversarial attacks, 2017.
  * Matthews et al. (2018) A. G. d. G. Matthews, M. Rowland, J. Hron, R. E. Turner, and Z. Ghahramani.  Gaussian process behaviour in wide deep neural networks.  _arXiv preprint arXiv:1804.11271_ , 2018.
  * Michelmore et al. (2020) R. Michelmore, M. Wicker, L. Laurenti, L. Cardelli, Y. Gal, and M. Kwiatkowska.  Uncertainty quantification with statistical guarantees in end-to-end autonomous driving control.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 7344–7350. IEEE, 2020.
  * Neal (2012) R. M. Neal.  _Bayesian learning for neural networks_ , volume 118.  Springer Science & Business Media, 2012.
  * Neal et al. (2011) R. M. Neal et al.  Mcmc using hamiltonian dynamics.  _Handbook of markov chain monte carlo_ , 2(11):2, 2011.
  * Ongie et al. (2020) G. Ongie, R. Willett, D. Soudry, and N. Srebro.  A function space view of bounded norm infinite width relu nets: The multivariate case.  In _International Conference on Learning Representations_ , 2020.
  * Pang et al. (2021) Y. Pang, S. Cheng, J. Hu, and Y. Liu.  Evaluating the robustness of bayesian neural networks against different types of attacks.  _arXiv preprint arXiv:2106.09223_ , 2021.
  * Patane et al. (2022) A. Patane, A. Blaas, L. Laurenti, L. Cardelli, S. Roberts, and M. Kwiatkowska.  Adversarial robustness guarantees for gaussian processes.  _Journal of Machine Learning Research_ , 23:1–55, 2022\.
  * Rawat et al. (2017) A. Rawat, M. Wistuba, and M.-I. Nicolae.  Adversarial phenomenon in the eyes of bayesian deep learning.  _arXiv preprint arXiv:1711.08244_ , 2017.
  * Rotskoff and Vanden-Eijnden (2018) G. M. Rotskoff and E. Vanden-Eijnden.  Neural networks as interacting particle systems: Asymptotic convexity of the loss landscape and universal scaling of the approximation error.  _arXiv preprint arXiv:1805.00915_ , 2018.
  * Rozza et al. (2014) A. Rozza, M. Manzo, and A. Petrosino.  A novel graph-based fisher kernel method for semi-supervised learning.  In _Proceedings of the 2014 22nd International Conference on Pattern Recognition_ , ICPR ’14, page 3786–3791, USA, 2014. IEEE Computer Society.  ISBN 9781479952090.  doi: 10.1109/ICPR.2014.650.  URL https://doi.org/10.1109/ICPR.2014.650.
  * Smith and Gal (2018) L. Smith and Y. Gal.  Understanding measures of uncertainty for adversarial example detection.  _arXiv preprint arXiv:1803.08533_ , 2018.
  * Su et al. (2018) D. Su, H. Zhang, H. Chen, J. Yi, P.-Y. Chen, and Y. Gao.  Is robustness the cost of accuracy?–a comprehensive study on the robustness of 18 deep image classification models.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 631–648, 2018.
  * Szegedy et al. (2013) C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus.  Intriguing properties of neural networks.  _arXiv preprint arXiv:1312.6199_ , 2013.
  * Uchendu et al. (2021) A. Uchendu, D. Campoy, C. Menart, and A. Hildenbrandt.  Robustness of bayesian neural networks to white-box adversarial attacks.  In _2021 IEEE Fourth International Conference on Artificial Intelligence and Knowledge Engineering (AIKE)_ , pages 72–80. IEEE, 2021.
  * Vaart (2000) A. W. v. d. Vaart.  _Asymptotic Statistics_.  Cambridge University Press, 2000.  URL https://EconPapers.repec.org/RePEc:cup:cbooks:9780521784504.
  * Wicker et al. (2020) M. Wicker, L. Laurenti, A. Patane, and M. Kwiatkowska.  Probabilistic safety for bayesian neural networks.  In _Conference on Uncertainty in Artificial Intelligence_ , pages 1198–1207. PMLR, 2020.
  * Wicker et al. (2021) M. Wicker, L. Laurenti, A. Patane, Z. Chen, Z. Zhang, and M. Kwiatkowska.  Bayesian inference with certifiable adversarial robustness.  In _International Conference on Artificial Intelligence and Statistics_ , pages 2431–2439. PMLR, 2021.
  * Williams and Rasmussen (2006) C. K. Williams and C. E. Rasmussen.  _Gaussian processes for machine learning_ , volume 2.  MIT press Cambridge, MA, 2006.
  * Xiao et al. (2017) H. Xiao, K. Rasul, and R. Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  _arXiv preprint arXiv:1708.07747_ , 2017.
  * Ye and Zhu (2018) N. Ye and Z. Zhu.  Bayesian adversarial learning.  In _Proceedings of the 32nd International Conference on Neural Information Processing Systems_ , pages 6892–6901. Curran Associates Inc., 2018\.
  * Yuan et al. (2020) M. Yuan, M. Wicker, and L. Laurenti.  Gradient-free adversarial attacks for bayesian neural networks.  _arXiv preprint arXiv:2012.12640_ , 2020.
  * Zhang et al. (2021) J. Zhang, Y. Hua, Z. Xue, T. Song, C. Zheng, R. Ma, and H. Guan.  Robust bayesian neural networks by spectral expectation bound regularization.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3815–3824, 2021.

##  Appendix A Further Results

In what follows we present additional empirical results: in Section A.1 we add
an experimental justification for Remark 7, while in Section A.2 we examine
the relationship between BNNs and Deep Ensemble architectures.

###  A.1 Multimodal posterior distributions

Figure 6 empirically confirms remark 7, by showing that the posterior
distributions of the weights are multimodal, thus Bernstein von Mises theorem
does not apply to BNNs.

Figure 6: Prior (first column) and posterior (columns 1-3) distributions for
models trained with an increasing number of training points. Models are
trained on MNIST dataset with HMC. Samples are projected on the first two
principal components (here $x$ and $y$) using PCA.

###  A.2 Comparison with Deep Ensembles

Deep ensembles, as proposed by Lakshminarayanan et al. (2017), are an ensemble
of neural networks trained from different randomly selected initial
conditions, which are then averaged in order to make a prediction. In Table 2
we consider the same network used to perform the experiments in Section 6.2
(hyper-parameters are reported in Table 5) and run a comparison with both
deterministic NNs and deep ensembles.

As expected, Bayesian NNs are more robust than deterministic ones. Moreover,
we find that deep ensembles and deterministic NNs are comparable in terms of
robustness, suggesting that simply averaging predictions for different weight
initialization and mini-batching is not enough to achieve a robust model.

Model |  | Test
---
accuracy
| FGSM
---
accuracy
| PGD
---
accuracy
Deterministic NN | 97.69 | 21.19 | 1.45
Ensemble NN | 99.4 | 20.6 | 0.3
Bayesian NN | 96.1 | 90.0 | 89.8
Table 2: FGSM and PGD attacks on the network employed in Section 6.2. We
compare a deterministic NN, a deep ensemble NN (of size $100$), and a BNN
(trained with VI). Attacks are performed on $1$k test points from the MNIST
dataset. We observe that VI trained network achieve better robustness against
PGD and FGSM.

##  Appendix B Training hyperparameters for BNNs

Tables 3, 6, 7 and 8 summarize the sets of hyperparameters used to tune our
models. BNNs’ architectures achieving the highest test accuracy are described
in Table 4 for HMC training and 5 for VI training.

Half moons grid search
---
Posterior samples |  {250}
HMC warmup samples |  {100, 200, 500}
Training inputs |  {5000, 10000, 15000}
Hidden size |  {32, 128, 256, 512}
Nonlinear activation |  Leaky ReLU
Architecture |  2 fully connected layers
Table 3:  Hyperparameters for training BNNs in Figure 1 Training
hyperparameters for HMC
---
Dataset |  MNIST |  Fashion MNIST
Training inputs |  60k |  60k
Hidden size |  1024 |  1024
Nonlinear activation |  ReLU |  ReLU
Architecture |  Fully Connected |  Fully Connected
Posterior Samples |  500 |  500
Numerical Integrator Stepsize |  0.002 |  0.001
Number of steps for Numerical Integrator |  10 |  10
Table 4:  Hyperparameters for training BNNs using HMC in Figures 2 and 3.
Training hyperparameters for VI
---
Dataset |  MNIST |  Fashion MNIST
Training inputs |  60k |  60k
Hidden size |  512 |  1024
Nonlinear activation |  Leaky ReLU |  Leaky ReLU
Architecture |  Convolutional |  Convolutional
Training epochs |  5 |  10
Learning rate |  0.01 |  0.001
Table 5:  Hyperparameters for training BNNs using VI in Figures 2 and 3. HMC
MNIST/Fashion MNIST grid search
---
Posterior samples |  {250, 500, 750*}
Numerical Integrator Stepsize |  {0.01, 0.005*, 0.001, 0.0001}
Numerical Integrator Steps |  {10*, 15, 20}
Hidden size |  {128, 256, 512*}
Nonlinear activation |  {relu*, tanh, sigmoid}
Architecture |  {1*,2,3} fully connected layers
Table 6:  Hyperparameters for training BNNs with HMC in Figure 4. * indicates
the parameters used in Table 1 of the main text. SGD MNIST/Fashion MNIST grid
search
---
Learning Rate |  {0.001*}
Minibatch Size |  {128, 256*, 512, 1024}
Hidden size |  {64, 128, 256, 512, 1024*}
Nonlinear activation |  {relu*, tanh, sigmoid}
Architecture |  {1*,2,3} fully connected layers
Training epochs |  {3,5*,7,9,12,15} epochs
Table 7:  Hyperparameters for training BNNs with SGD in Figure 4. * indicates
the parameters used in Table 1 of the main text. SGD MNIST/Fashion MNIST grid
search
---
Learning Rate |  {0.001, 0.005, 0.01, 0.05}
Hidden size |  {64, 128, 256, 512}
Nonlinear activation |  {relu, tanh, sigmoid}
Architecture |  {2, 3, 4, 5} fully connected layers
Training epochs |  {5, 10, 15, 20, 25} epochs
Table 8:  Hyperparameters for training BNNs with SGD in Figure 4.