# Bayesian sparsification for deep neural networks with Bayesian model
reduction

Dimitrije Marković  Chair of Cognitive Computational Neuroscience, Technische
Universität Dresden, 01062 Dresden, Germany.  Karl J. Friston  Wellcome Centre
for Human Neuroimaging, Queen Square Institute of Neurology, University
College London, UK.  VERSES AI Research Lab, Los Angeles, California, 90016,
USA.  Stefan J. Kiebel  Chair of Cognitive Computational Neuroscience,
Technische Universität Dresden, 01062 Dresden, Germany.  Centre for Tactile
Internet with Human-in-the-Loop (CeTI),
Technische Universität Dresden, 01062 Dresden, Germany.

###### Abstract

Deep learning’s immense capabilities are often constrained by the complexity
of its models, leading to an increasing demand for effective sparsification
techniques. Bayesian sparsification for deep learning emerges as a crucial
approach, facilitating the design of models that are both computationally
efficient and competitive in terms of performance across various deep learning
applications. The state-of-the-art – in Bayesian sparsification of deep neural
networks – combines structural shrinkage priors on model weights with an
approximate inference scheme based on black-box stochastic variational
inference. However, model inversion of the full generative model is
exceptionally computationally demanding, especially when compared to standard
deep learning of point estimates. In this context, we advocate for the use of
Bayesian model reduction (BMR) as a more efficient alternative for pruning of
model weights. As a generalization of the Savage-Dickey ratio, BMR allows a
post-hoc elimination of redundant model weights based on the posterior
estimates under a straightforward (non-hierarchical) generative model. Our
comparative study highlights the computational efficiency and the pruning rate
of the BMR method relative to the established stochastic variational inference
(SVI) scheme, when applied to the full hierarchical generative model. We
illustrate the potential of BMR to prune model parameters across various deep
learning architectures, from classical networks like LeNet to modern
frameworks such as Vision Transformers and MLP-Mixers.

##  1 Introduction

Bayesian deep learning integrates the principles of Bayesian methodology with
the objectives of deep learning, facilitating the training of expansive
parametric models tailored for classifying and generating intricate audio-
visual data, including images, text, and speech (Wang and Yeung, 2020; Wilson,
2020; Wang and Yeung, 2016). Notably, the Bayesian approach frames the
challenge of model optimization as an inference problem. This perspective is
especially apt for scenarios necessitating decision-making under uncertainty
(Murphy, 2022; Ghahramani, 2015). As a result, Bayesian formulations in deep
learning have proven advantageous in various respects, offering enhancements
in generalization (Wilson and Izmailov, 2020), accuracy, calibration (Izmailov
et al., 2020; Luo and Kareem, 2020), and model compression (Louizos et al.,
2017).

These functional enhancements are intrinsically tied to judiciously chosen
structural priors (Fortuin, 2022). The priors, integral to the probabilistic
generative model, scaffold the architecture of the network, thereby reducing
the data required for the inference of optimal parametric solutions. Recent
studies have highlighted the efficacy of hierarchical shrinkage priors over
model weights, a specific category of structural priors, in achieving highly-
sparse network representations (Nalisnick et al., 2019; Louizos et al., 2017;
Seto et al., ; Ghosh et al., 2018). Sparse representations not only reduce
redundancy but also evince additional performance benefits. However, the
adoption of shrinkage priors in all deep learning models presents a conundrum:
the ballooning space of latent parameters and the diminishing scalability of
prevailing approximate inference schemes (Snoek et al., 2015; Krishnan et al.,
2019; Izmailov et al., 2020; Daxberger et al., 2021a).

In line with ongoing research on scalable Bayesian inference, we introduce an
approximate inference scheme rooted in Bayesian model reduction (BMR). In
essence, BMR extends the foundational principles of the Savage-Dickey Density
Ratio method (Cameron, 2013). BMR is typically conceptualized as a
combinatorial model comparison framework, enabling swift estimations of model
evidence across an extensive array of models, that differ in their prior
assumptions, to identify the most probable one. Originally conceived for model
comparison within the dynamical causal modeling framework (Rosa et al., 2012;
Friston and Penny, 2011), the scope of BMR has since broadened. Subsequent
works expanded its methodology (Friston et al., 2016, 2017, 2018) and adapted
it for structure learning (Smith et al., 2020). More recently, BMR has found
applications in Bayesian nonlinear regression and classification tasks using
Bayesian neural networks with variance backpropagation (Beckers et al., 2022;
Haußmann et al., 2020).

The BMR method is intimately connected with the spike-and-slab prior, a type
of shrinkage prior (Mitchell and Beauchamp, 1988). Intriguingly, this specific
structured shrinkage prior has parallels with Dropout regularization
(Nalisnick et al., 2019). Such an association spurred researchers in Bayesian
deep learning to formulate sparsification methods based on a different type of
shrinkage prior – the hierarchical horseshoe prior (Piironen and Vehtari,
2017) – as a tool for automated depth determination. Subsequent studies
suggested that merging horseshoe priors with structured variational
approximations yields robust, highly sparse representations (Ghosh et al.,
2018). The allure of continuous shrinkage priors (e.g., horseshoe priors)
stems from the computational challenges associated with model inversion
reliant on spike-and-slab priors (Nalisnick et al., 2019; Piironen and
Vehtari, 2017). However, continuous shrinkage priors necessitate a
considerably more expansive parameter space, to represent the approximate
posterior, compared to optimizing neural networks using the traditional point
estimate method.

In this work, we reexamine the spike-and-slab prior within the framework of
BMR-based sparsification, highlighting its efficiency. Notably, this approach
circumvents the need to expand the approximate posterior beyond the
conventional fully factorised mean-field approximation, making it more
scalable than structured variational approximations (Ghosh et al., 2018). In
this light, BMR can be seen as a layered stochastic and black-box variational
inference technique, which we term _stochastic BMR_. We subject the stochastic
BMR to rigorous validation across various image classification tasks and
network architectures, including LeNet-5 (LeCun et al., 1989), Vision
Transformers (Dosovitskiy et al., 2020), and MLP-Mixers (Tolstikhin et al.,
2021).

Central to our study is an empirical comparison of stochastic BMR with methods
anchored in hierarchical horseshoe priors. Through multiple metrics - from
Top-1 accuracy to expected calibration error and negative log-likelihood - we
establish the competitive performance of stochastic BMR. We argue its
computational efficiency, and remarkable sparsification rate, position BMR as
an appealing choice, enhancing the scalability and proficiency of contemporary
deep learning networks across diverse machine learning challenges, extending
well beyond computer vision. We conclude with a discussion on potential
avenues of future research that could further facilitate of BMR based pruning
of deep neural networks.

##  2 Methods

###  2.1 Variational inference

Given a joint density of latent variables, represented as
$\boldsymbol{z}=\mathinner{\left(z_{1},\ldots,z_{k}\right)}$, and a dataset of
$n$ observations
$\boldsymbol{\mathcal{D}}=\mathinner{\left(y_{1},\ldots,y_{n}\right)}$ we can
express the joint density, that is, the generative model, as

$p\mathinner{\left(\boldsymbol{\mathcal{D}},\boldsymbol{z}\right)}=p\mathinner{\left(\boldsymbol{z}\right)}p\mathinner{\left(\boldsymbol{\mathcal{D}}|\boldsymbol{z}\right)}.$
(1)

The posterior density is then obtained, following the Bayes rule, as

$p\mathinner{\left(\boldsymbol{z}|\boldsymbol{\mathcal{D}}\right)}\propto
p\mathinner{\left(\boldsymbol{z}\right)}p\mathinner{\left(\boldsymbol{\mathcal{D}}|\boldsymbol{z}\right)}.$
(2)

For complex generative models, direct inference as described above becomes
computationally prohibitive. To circumvent this, we approximate the exact
posterior $p\mathinner{\left(\boldsymbol{z}|\boldsymbol{\mathcal{D}}\right)}$,
constraining it to a distribution $q\mathinner{\left(z\right)}$ that belongs
to a named distribution family $\mathcal{Q}$. We then seek
$q^{*}\mathinner{\left(z\right)}\in\mathcal{Q}$, an approximate solution that
minimizes the following Kullback-Leibler divergence (Blei et al., 2017)

$q^{*}\mathinner{\left(z\right)}=\underset{q\in\mathcal{Q}}{\text{argmin}}D_{KL}\mathinner{\left(q\mathinner{\left(\boldsymbol{z}\right)}||p\mathinner{\left(\boldsymbol{z}|\boldsymbol{\mathcal{D}}\right)}\right)}=\underset{q\in\mathcal{Q}}{\text{argmin}}F\mathinner{\left[q\right]},$
(3)

where $F\mathinner{\left[q\right]}$ stands for the variational free energy
(VFE), defined as

$F\mathinner{\left[q\right]}=E_{q\mathinner{\left(\boldsymbol{z}\right)}}\mathinner{\left[\ln
q\mathinner{\left(\boldsymbol{z}\right)}-\ln
p\mathinner{\left(\boldsymbol{\mathcal{D}},\boldsymbol{z}\right)}\right]}$ (4)

VFE serves as an upper bound on the marginal log-likelihood

$F\mathinner{\left[q\right]}=D_{KL}\mathinner{\left(q\mathinner{\left(\boldsymbol{z}\right)}||p\mathinner{\left(\boldsymbol{z}|\boldsymbol{\mathcal{D}}\right)}\right)}-\ln
p\mathinner{\left(\boldsymbol{\mathcal{D}}\right)}\geq-\ln
p\mathinner{\left(\boldsymbol{\mathcal{D}}\right)}$

As KL-divergence is always greater or equal to zero, minimizing VFE brings the
approximate solution as close as possible to the true posterior, without
having to compute the exact posterior.

The most straightforward way to obtain the approximate posterior
$q^{*}\mathinner{\left(\boldsymbol{z}\right)}$, is to minimize the VFE along
its negative gradient:

$\dot{\boldsymbol{\phi}}=-\nabla_{\boldsymbol{\phi}}F\mathinner{\left[q\right]}$

where $\boldsymbol{\phi}$ signifies the parameters of the approximate
posterior
$q_{\boldsymbol{\phi}}\mathinner{\left(\boldsymbol{z}\right)}=q\mathinner{\left(\boldsymbol{z}|\boldsymbol{\phi}\right)}$.
Thus, variational inference reframes the inference problem highlighted in eq.
2 as an optimization problem Beal (2003).

###  2.2 Stochastic and black-box variational inference

_Stochastic variational inference_ (SVI) improves the computational efficiency
of gradient descent by approximating the variational free energy using a
subset –
$\boldsymbol{\mathcal{K}}_{i}=\mathinner{\left(y_{s^{i}_{1}},\ldots,y_{s^{i}_{k}}\right)};\>\>k\ll
n$ – of the entire data set $\boldsymbol{\mathcal{D}}$. This approach fosters
a stochastic gradient descent (SGD) mechanism, capable of managing large
datasets (Hoffman et al., 2013). Crucially, at every iteration step $i$ of the
SGD process, the subset $\boldsymbol{\mathcal{K}}_{i}$ undergoes re-sampling.

_Black-box Variational Inference_ (BBVI) facilitates the optimization of any
(named or unnamed) posterior density
$q_{\boldsymbol{\phi}}\mathinner{\left(\boldsymbol{z}\right)}$, through the
integration of Monte Carlo estimates for variational gradients (Ranganath et
al., 2014). This can be formulated as

$\nabla_{\boldsymbol{\phi}}F\mathinner{\left[q\right]}\approx\nabla_{\boldsymbol{\phi}}\hat{F}\mathinner{\left[q\right]}=\frac{1}{S}\sum_{s=1}^{S}\nabla_{\boldsymbol{\phi}}\ln
q_{\boldsymbol{\phi}}\mathinner{\left(\boldsymbol{z}\right)}\mathinner{\left[\ln\frac{q_{\boldsymbol{\phi}}\mathinner{\left(\boldsymbol{z}\right)}}{q_{\boldsymbol{\phi}}\mathinner{\left(\boldsymbol{z}\right)}}+1\right]}\quad\boldsymbol{z}_{s}\sim
q\mathinner{\left(\boldsymbol{z}|\boldsymbol{\phi}\right)}$ (5)

To mitigate the variance inherent to Monte Carlo gradient estimations, we
employ Rao-Blackwellization (Schulman et al., 2015), with an implementation
sourced from NumPyro (Bingham et al., 2019). For optimizing the variational
objective stochastically, we leverage the AdaBelief optimizer (Zhuang et al.,
2020). As an adaptive algorithm, AdaBelief ensures swift convergence, robust
generalization, and steady optimization. Notably, we utilize AdaBelief’s
implementation from the Optax package within the JAX ecosystem (Babuschkin et
al., 2020).

###  2.3 Bayesian model reduction

Let us consider two generative processes for the data: a full (i.e., flat)
model

$p\mathinner{\left(\boldsymbol{z}|\boldsymbol{\mathcal{D}}\right)}\propto
p\mathinner{\left(\boldsymbol{\mathcal{D}}|\boldsymbol{z}\right)}p\mathinner{\left(\boldsymbol{z}\right)}$
(6)

and a reduced model in which the flat prior
$p\mathinner{\left(\boldsymbol{z}\right)}$ is replaced with a more informative
prior
$\tilde{p}\mathinner{\left(\boldsymbol{z}\right)}=p\mathinner{\left(\boldsymbol{z}|\boldsymbol{z}_{H}\right)}$
that depends on hyper-parameters $\boldsymbol{z}_{H}$. This change leads to a
different posterior

$\tilde{p}\mathinner{\left(\boldsymbol{z}|\boldsymbol{\mathcal{D}}\right)}\propto
p\mathinner{\left(\boldsymbol{\mathcal{D}}|\boldsymbol{z}\right)}\tilde{p}\mathinner{\left(\boldsymbol{z}\right)}$
(7)

Noting that as the following relation holds:

$1=\int\operatorname{d\\!}\boldsymbol{z}\tilde{p}\mathinner{\left(\boldsymbol{z}|\boldsymbol{\mathcal{D}}\right)}=\int\operatorname{d\\!}\boldsymbol{z}p\mathinner{\left(\boldsymbol{z}|\boldsymbol{\mathcal{D}}\right)}\frac{\tilde{p}\mathinner{\left(\boldsymbol{z}\right)}p\mathinner{\left(\boldsymbol{\mathcal{D}}\right)}}{p\mathinner{\left(\boldsymbol{z}\right)}\tilde{p}\mathinner{\left(\boldsymbol{\mathcal{D}}\right)}},$

we can express the link between the models as:

$\begin{split}-\ln\tilde{p}\mathinner{\left(\boldsymbol{\mathcal{D}}\right)}&=-\ln
p\mathinner{\left(\boldsymbol{\mathcal{D}}\right)}-\ln\int
d\boldsymbol{z}p\mathinner{\left(\boldsymbol{z}|\boldsymbol{\mathcal{D}}\right)}\frac{\tilde{p}\mathinner{\left(\boldsymbol{z}\right)}}{p\mathinner{\left(\boldsymbol{z}\right)}}\\\
&\approx F\mathinner{\left(\boldsymbol{\phi}^{*}\right)}-\ln\int
d\boldsymbol{z}q_{\boldsymbol{\phi}^{*}}\mathinner{\left(\boldsymbol{z}\right)}\frac{\tilde{p}\mathinner{\left(\boldsymbol{z}\right)}}{p\mathinner{\left(\boldsymbol{z}\right)}}\end{split}$
(8)

where we assumed the approximate posterior for the full model corresponds to
$p\mathinner{\left(\boldsymbol{z}|\boldsymbol{\mathcal{D}}\right)}\approx
q_{\boldsymbol{\phi}^{*}}\mathinner{\left(\boldsymbol{z}\right)}$, and that
$-\ln p\mathinner{\left(\boldsymbol{\mathcal{D}}\right)}\approx
F\mathinner{\left(\boldsymbol{\phi}^{*}\right)}$.

From eq. 8 we obtain the free energy of the reduced model as

$-\ln\tilde{p}\mathinner{\left(\boldsymbol{\mathcal{D}}\right)}\approx-\ln
E_{q}\left[\frac{\tilde{p}\mathinner{\left(\boldsymbol{z}\right)}}{p\mathinner{\left(\boldsymbol{z}\right)}}\right]+F\mathinner{\left(\boldsymbol{\phi}^{*}\right)}=-\Delta
F\mathinner{\left(\boldsymbol{z}_{H}\right)}.$ (9)

where $\Delta F\mathinner{\left(\boldsymbol{z}_{H}\right)}$ denotes the change
in the free energy of going from the full model to the reduced model, given
hyper-parameters $\boldsymbol{z}_{H}$. Note that for $\Delta
F\mathinner{\left(\boldsymbol{z}_{H}\right)}>0$ the reduced model has a better
variational free energy compared to the flat model. Consequently, the reduced
model offers a model with a greater marginal likelihood; i.e., a better
explanation for the data and improved generalization capabilities.
Heuristically, this can be understood as minimising model complexity, without
sacrificing accuracy (because log evidence can be expressed as accuracy minus
complexity, where complexity is the KL divergence between posterior and prior
beliefs). This relationship is pivotal in formulating efficient pruning
criteria, especially for extensive parametric models commonly employed in deep
learning.

###  2.4 Bayesian neural networks

In a general (nonlinear) regression problem, we model the relationship between
predictors
$\boldsymbol{X}=\mathinner{\left(\boldsymbol{x}_{1},\ldots,\boldsymbol{x}_{n}\right)}$
and target variables
$\boldsymbol{Y}=\mathinner{\left(\boldsymbol{y}_{1},\ldots,\boldsymbol{y}_{n}\right)}$
using a likelihood distribution from an exponential family as

$\boldsymbol{y}_{i}\sim
p\mathinner{\left(\boldsymbol{y}|\boldsymbol{\mathcal{W}},\boldsymbol{x}_{i}\right)}=h(y)\exp\mathinner{\left[\boldsymbol{\eta}\mathinner{\left(\boldsymbol{f}\mathinner{\left(\boldsymbol{\mathcal{W}},\boldsymbol{x}_{i}\right)}\right)}\cdot\boldsymbol{T}\mathinner{\left(\boldsymbol{y}\right)}-A\mathinner{\left(\boldsymbol{f}\mathinner{\left(\boldsymbol{\mathcal{W}},\boldsymbol{x}_{i}\right)}\right)}\right]}.$
(10)

Functions $h(\cdot),\boldsymbol{\eta}(\cdot),\boldsymbol{T}(\cdot),A(\cdot)$
are known and selected depending on the task. For example in a regression
problem the likelihood will correspond to a multivariate normal distribution
and in a classification problem to a categorical distribution. In this work,
we will only consider a categorical likelihood, as it is the most suitable for
image classification tasks.

The mapping
$\boldsymbol{f}\mathinner{\left(\boldsymbol{\mathcal{W}},\boldsymbol{x}_{i}\right)}$
represents a generic deep neural network of depth $L$ defined as

$\begin{split}\boldsymbol{\mathcal{W}}&=\mathinner{\left(\boldsymbol{W}_{1},\ldots,\boldsymbol{W}_{L}\right)}\\\
\boldsymbol{h}^{0}_{i}&=\boldsymbol{x}_{i}\\\
\boldsymbol{h}^{l}_{i}&=\boldsymbol{g}\left(\boldsymbol{W}_{l}\cdot\left[\boldsymbol{h}_{i}^{l-1};1\right]\right)\\\
\boldsymbol{f}\mathinner{\left(\boldsymbol{\mathcal{W}},\boldsymbol{x}_{i}\right)}&=\boldsymbol{W}_{L}\cdot\left[\boldsymbol{h}^{L-1}_{i};1\right]\end{split}$
(11)

A probabilistic formulation of the deep learning task, that is, inferring
model weights, introduces implicit bias to the parameters
$\boldsymbol{\mathcal{W}}$ of an artificial neural network in the form of a
prior distribution $p\mathinner{\left(\boldsymbol{\mathcal{W}}\right)}$.
Hence, parameter estimation is cast as an inference problem where

$p\mathinner{\left(\boldsymbol{\mathcal{W}}|\boldsymbol{\mathcal{D}}\right)}\propto
p\mathinner{\left(\boldsymbol{\mathcal{W}}\right)}\prod_{i=1}^{n}p\mathinner{\left(\boldsymbol{y}_{i}|\boldsymbol{\mathcal{W}},\boldsymbol{x}_{i}\right)}$
(12)

The choice of the prior distribution is crucial for optimal task performance,
and a prior assumption of structural sparsity is essential for inferring
sparse representations of over-parameterised models, such as deep neural
networks.

###  2.5 Bayesian neural networks with shrinkage priors

Shrinkage priors instantiate a prior belief about the sparse structure of
model parameters. Here, we will investigate two well-established forms of
shrinkage priors for network weight parameters, a canonical spike-and-slab
prior (George and McCulloch, 1993; Mitchell and Beauchamp, 1988) defined as

$\begin{split}w_{ijl}&\sim\mathcal{N}\mathinner{\left(0,\lambda_{ijl}^{2}\gamma_{0}^{2}\right)}\\\
\lambda_{ijl}&\sim\text{Bernoulli}\mathinner{\left(\pi_{l}\right)}\\\
\pi_{l}&\sim\mathcal{B}e\mathinner{\left(\alpha_{0},\beta_{0}\right)}\end{split}$
(13)

and a regularised-horseshoe prior (Piironen and Vehtari, 2017)

$\begin{split}w_{ijl}&\sim\mathcal{N}\mathinner{\left(0,\gamma_{il}^{2}\right)}\\\
\gamma_{il}^{2}&=\frac{c_{l}^{2}v_{l}^{2}\tau_{il}^{2}}{c_{l}^{2}+\tau_{il}^{2}v_{l}^{2}}\\\
c_{l}^{-2}&\sim\Gamma\mathinner{\left(2,6\right)}\\\
\tau_{il}&\sim\mathcal{C}^{+}(0,1)\\\
v_{l}&\sim\mathcal{C}^{+}(0,\tau_{0})\end{split}$ (14)

where $i\in\mathinner{\left\\{1,\ldots,K_{l}\right\\}}$,
$j\in\mathinner{\left[1,\ldots,K_{l-1}+1\right]}$, and where $w_{ijl}$ denotes
$ij$th element of the weight matrix at depth $l$. The symbols $\mathcal{B}e$,
and $\mathcal{C}^{+}$ denote a Beta distribution and a half-Cauchy
distribution, respectively.

Importantly, the spike-and-slab prior relates to dropout regularisation, which
is commonly introduced as a sparsification method in deep learning (Nalisnick
et al., 2019; Mobiny et al., 2021). This type of prior is considered the gold
standard in shrinkage priors and has been used in many recent applications of
Bayesian sparsification on neuronal networks (Bai et al., 2020; Hubin and
Storvik, 2023; Jantre et al., 2021; Sun et al., 2022; Ke and Fan, 2022)
showing excellent sparsification rates. However, the inversion of the
resulting hierarchical model is challenging and requires carefully constructed
posterior approximations. Moreover, their dependence on discrete random
variables renders them unsuitable for Markov-Chain Monte Carlo-based sampling
schemes. As a result, researchers often use continuous formulations of the
shrinkage-prior, with the horseshoe prior being a notable example.

In contexts that involve sparse learning with scant data, the regularised
horseshoe prior has emerged as one of the preferred choices within shrinkage
prior families (Ghosh et al., 2019). A distinct advantage of this prior is its
ability to define both the magnitude of regularisation for prominent
coefficients and convey information about sparsity. It is worth noting a
dependency highlighted in Ghosh et al. (2018): for $v_{l}\tau_{il}\ll 1$ the
equation simplifies to $\gamma_{il}\approx v_{l}\tau_{il}$ recovering the
original horseshoe prior. In contrast, for $v_{l}\tau_{il}\gg 1$, the equation
becomes $\gamma_{il}^{2}\approx c_{l}^{2}$. In this latter scenario, the prior
over the weights is defined as
$w_{ijl}\sim\mathcal{N}\mathinner{\left(0,c^{2}_{l}\right)}$, with $c_{l}$
serving as a weight decay hyper-parameter for layer $l$.

###  2.6 Approximate posterior for Bayesian neural networks

To benchmark stochastic BMR, we explore two forms of prior distribution
$p\mathinner{\left(\boldsymbol{\mathcal{W}}\right)}$ — a flat and a
hierarchical structure — in conjunction with a fully factorised mean-field
approximation.

Firstly, let us consider the flat prior over model weights, represented in a
non-centered parameterization:

$\begin{split}\hat{w}_{ijl}&\sim\mathcal{N}\mathinner{\left(0,1\right)}\\\
w_{ijl}&=\gamma_{0}\hat{w}_{ijl}\end{split}$ (15)

where we set $\gamma_{0}=0.1$. Based on this, we describe a fully factorised
approximate posterior as:

$\begin{split}q\mathinner{\left(\boldsymbol{\mathcal{W}}\right)}&=\prod_{l}\prod_{i}\prod_{j}q\mathinner{\left(\hat{w}_{ijl}\right)}\\\
q\mathinner{\left(\hat{w}_{ijl}\right)}&=\mathcal{N}\mathinner{\left(\mu_{ijl},\sigma^{2}_{ijl}\right)}.\end{split}$
(16)

When inverting a hierarchical generative model over weights of artificial
neural network, we exclusively apply stochastic black-box variational
inference to the model variant with the regularised horseshoe prior. This
choice is motivated by its documented superiority over the spike-and-slab
prior, as established in Ghosh et al. (2018). We express the hierarchical
prior in the non-centered parameterization as:

$\begin{split}a_{il},b_{il}&\sim\Gamma\mathinner{\left(\frac{1}{2},1\right)}\\\
\hat{a}_{l},\hat{b}_{l}&\sim\Gamma\mathinner{\left(\frac{1}{2},1\right)}\\\
\tau_{il}&=\sqrt{\frac{a_{il}}{b_{il}}}\\\
v_{l}&=\tau_{0}\sqrt{\frac{\hat{a}_{l}}{\hat{b}_{l}}}\\\
\hat{w}_{ijl}&\sim\mathcal{N}\mathinner{\left(0,1\right)}\\\
w_{ijl}&=\gamma_{il}\hat{w}_{ijl}\end{split}$ (17)

Note that the expressions above involve a reparameterization of Half-Cauchy
distributed random variables as the square-root of the quotient of two Gamma
distributed random variables, a strategy drawn from Wand et al. (2011) (see
Reparameterisation for additional details). Such a reparameterization of the
Half-Cauchy ensures capturing of fat-tails in the posterior, even when
leveraging a fully-factorised mean-field posterior approximation, as
referenced in Ghosh et al. (2018).

For the fully-factorised mean-field approximation, the approximate posterior
is portrayed as a composite of Normal and Log-Normal distributed random
variables, expressed as:

$\begin{split}q\mathinner{\left(c_{l}\right)}&=\mathcal{LN}\mathinner{\left(\mu_{c,l},\sigma_{c,l}^{2}\right)}\\\
q\mathinner{\left(\hat{a}_{l}\right)}&=\mathcal{LN}\mathinner{\left(\hat{\mu}_{a,l},\hat{\sigma}_{a,l}^{2}\right)}\\\
q\mathinner{\left(\hat{b}_{l}\right)}&=\mathcal{LN}\mathinner{\left(\hat{\mu}_{b,l},\hat{\sigma}_{b,l}^{2}\right)}\\\
q\mathinner{\left(a_{il}\right)}&=\mathcal{LN}\mathinner{\left(\mu_{a,il},\sigma_{a,il}^{2}\right)}\\\
q\mathinner{\left(b_{il}\right)}&=\mathcal{LN}\mathinner{\left(\mu_{b,il},\sigma_{b,il}^{2}\right)}\\\
q\mathinner{\left(\hat{w}_{ijl}\right)}&=\mathcal{N}\mathinner{\left(\mu_{w,ijl},\sigma_{w,ijl}^{2}\right)}\end{split}$
(18)

###  2.7 Application of stochastic BMR to Bayesian neural networks

To apply BMR to Bayesian neural networks, we commence by estimating an
approximate posterior for the flat model, as detailed in eq. 15. To retain
high computational efficiency, we pair BMR solely with the fully factorised
approximate posterior, as presented in eq. 16. While it is feasible to use
this method alongside the structured posterior (Ghosh et al., 2018), it
requires considerably more computationally intensive estimations of the
reduced free energy. As shown below, we obtain satisfactory results with a
fully factorised posterior. Therefore, we defer the exploration of BMR with a
structured posterior to future endeavours.

Given a fully factorised approximate posterior, we can determine the change in
variational free energy, $\Delta F$ — after substituting the prior
$\mathcal{N}\mathinner{\left(0,1\right)}$ with
$\mathcal{N}\mathinner{\left(0,\theta_{ijl}^{2}\right)}$ for the weight
$\hat{w}_{ijl}$ — as:

$\begin{split}\Delta
F\mathinner{\left(\theta_{ijl}\right)}&=-\frac{1}{2}\ln\rho_{ijl}^{2}-\frac{1}{2}\frac{\mu_{ijl}^{2}}{\sigma_{ijl}^{2}}\mathinner{\left(1-\frac{\theta_{ijl}^{2}}{\rho_{ijl}^{2}}\right)}\\\
\rho_{ijl}^{2}&=\theta_{ijl}^{2}+\sigma_{ijl}^{2}-\theta_{ijl}^{2}\sigma_{ijl}^{2}\end{split}$
(19)

For the second hierarchical level of the approximate posterior, we aim to
minimize the following form for the variational free energy:

$F=\sum_{l=1}^{L}E_{q\mathinner{\left(\boldsymbol{\theta}_{l}\right)}}\mathinner{\left[-\sum_{i,j}\Delta
F(\theta_{ijl})+\ln\frac{q\mathinner{\left(\boldsymbol{\theta}_{l}\right)}}{p\mathinner{\left(\boldsymbol{\theta}_{l}\right)}}\right]}$
(20)

This minimization is done with respect to
$q\mathinner{\left(\boldsymbol{\Theta}\right)}=\prod_{l}q\mathinner{\left(\boldsymbol{\theta}_{l}\right)}$,
the approximate posterior over hyper-parameters. Note the application of eq. 9
in substituting the marginal log-likelihood with the change in the variational
free energy.

For the spike-and-slab prior we can write the following relation:

$\begin{split}\boldsymbol{\theta}_{l}&=\mathinner{\left[\pi_{l},\lambda_{ijl}|\text{
for }i\in\mathinner{\left\\{1,\ldots,K_{l}\right\\}},\text{ and
}j\in\mathinner{\left\\{1,\ldots,K_{l-1}+1\right\\}}\right]}\\\
\theta_{ijl}&=\lambda_{ijl}\end{split}$

Consequently, the approximate posterior at the second level of the hierarchy
can be approximated as:

$\begin{split}q\mathinner{\left(\boldsymbol{\Theta}\right)}&=\prod_{l}q\mathinner{\left(\pi_{l}\right)}\prod_{ij}q\mathinner{\left(\lambda_{ijl}\right)}\\\
q\mathinner{\left(\lambda_{ijl}\right)}&=q_{ijl}^{\lambda_{ijl}}\mathinner{\left(1-q_{ijl}\right)}^{1-\lambda_{ijl}}\\\
q\mathinner{\left(\pi_{l}\right)}&=\mathcal{B}\mathinner{\left(\alpha_{l},\beta_{l}\right)}\end{split}$
(21)

The iterative update to obtain the minimum of the simplified variational free
energy (eq. 20) is then:

$\begin{split}q_{ijl}^{k+1}&=\frac{1}{1+e^{-\mathinner{\left[\zeta^{k}_{l}-\Delta
F\mathinner{\left(\lambda_{ijl}=0\right)}\right]}}}\\\
\zeta^{k}_{l}&=\psi(\alpha^{k}_{l})-\psi(\beta^{k}_{l})\\\
\alpha^{k+1}_{l}&=\sum_{i,j}q^{k+1}_{ijl}+\alpha_{0}\\\
\beta^{k+1}_{l}&=\sum_{i,j}\mathinner{\left(1-q^{k+1}_{ijl}\right)}+\beta_{0}\\\
\end{split}$ (22)

Here, $\alpha^{0}_{l}=\alpha_{0}$, $\beta^{0}_{l}=\beta_{0}$, $\Delta
F\mathinner{\left(\lambda_{ijl}=0\right)}=-\frac{1}{2}\mathinner{\left[\ln\sigma_{ijl}^{2}+\frac{\mu_{ijl}^{2}}{\sigma_{ijl}^{2}}\right]}$,
and $\psi\mathinner{\left(\cdot\right)}$ refers to the digamma function. The
efficiency of this inference scheme is remarkable, typically achieving
convergence after a few iterations. In practice, we cap the maximum number of
iterations at $k_{max}=4$.

Finally, we use the following pruning heuristics to eliminate model weights
and sparsify network structure

$\text{ if }q_{ijl}^{k_{max}}<\frac{1}{2}\text{, set }\hat{w}_{ijl}=0.$ (23)

To achieve the high sparsification rate presented in the next section, we
adopt an iterative optimisation and pruning approach proposed in Beckers et
al. (2022). After initial convergence of the SVI, for the flat prior, and
fully factorised posterior (capped at $10^{5}$ iterations) we perform weight
pruning, and further optimisation for $2\cdot 10^{4}$ iterations, completing
one epoch. In total, we apply iterative pruning and optimisation for five
epochs in all examples below.

The complete implementation of stochastic BMR is available at an online
repository github.com/dimarkov/bmr4pml with notebooks and scripts necessary to
recreate all result figures.

##  3 Results

###  3.1 Performance Comparison

We assessed the performance of four deep neural network (DNN) architectures:
MLP, LeNet-5, MlpMixer, and a variant of the Vision Transformer (detailed in
Specification of neural network architectures). Our evaluations covered
standard image classification benchmarks, including Fashion MNIST, CIFAR10,
and CIFAR100. We quantified performance using top-1 accuracy (ACC), expected
calibration error (ECE), and negative log likelihood (NLL) determined from the
test data. While top-1 accuracy is a straightforward measure of model
performance, it provides only limited information. ECE and NLL complement the
accuracy measure by characterising the model’s confidence, and predictability
of new labels, respectively. In practice, a good overall performance
corresponds to high ACC, and low ECE and NLL.

Figure 1:  Classification performance comparison on FashoinMNIST dataset for
different neuronal architectures and approximate inference schemes.

The training regimen used a batch size of $N_{B}=128$ and the default settings
of the AdaBelief algorithm ($\alpha=10^{-3},\beta_{1}=0.9,\beta_{2}=0.999$) as
detailed in Zhuang et al. (2020). Figure 1 charts the epoch-wise evolution of
ACC, ECE, and NLL for each architecture, under five distinct approximate
inference strategies: (i) Maximum a posteriori (MAP) estimate for the flat
generative model, akin to traditional deep learning point estimates coupled
with weight decay. (ii) A fully factorised posterior approximation for the
flat generative model (Flat-FF). (iii) A fully factorised posterior
approximation of the hierarchical generative model with a regularised
horseshoe prior (Tiered-FF). (iv) The stochastic BMR algorithm augmented with
a spike-and-slab prior (BMR-S&S). Each epoch is defined by $20K$ stochastic
gradient steps, with each step randomly drawing $N_{B}$ data instances from
the training pool.

Interestingly, all approximate inference methods demonstrate comparable top-1
accuracy scores. However, the stochastic BMR method followed by the Tiered-FF
approximation, consistently resulted in the lowest ECE and NLL scores across
all DNN architectures and datasets (see Figure S1 for CIFAR10 dataset and
Figure S2 for CIFAR100 dataset).The implicit reduction in model complexity
suggests that–as anticipated–Stochastic BMR furnishes a model of the data that
has the greatest evidence or marginal likelihood (not shown). In this setting,
the NLL of the test data can be regarded as a proxy for (negative log)
marginal likelihood.

###  3.2 Learning of sparse representations

Figure 2:  Total fraction of pruned model parameters obtained with the
stochastic BMR algorithm across different DNN architectures and datasets.

Figure 2 depicts the fraction of pruned model parameters for different DNN
architecures and datasets. It is noteworthy to observe the substantive
sparsity achieved by the stochastic BMR algorithm. This sparsity is consistent
across datasets and architectures, with the exception of the LeNet-5 structure
when used for the FashionMNIST dataset, because by default LeNet-5
architecture is already sparse and contains relatively low-number of model
weights (for other data sets we substantially increased the dimensionality of
hidden layers as detailed in Specification of neural network architectures).

Figure 3:  Cumulative Distribution Function (CDF) of absolute posterior
parameter expectations at different layers of MLP (top row), and LeNet
architectures (bottom row). The y-axis represents the fraction of parameters
with values less than or equal to the value on the x-axis.

To delve deeper into the pruning behavior across varying network depths,
Figure 3 presents a per-layer cumulative distribution function (CDF) for model
parameters, highlighting the proportion of parameters whose absolute mean
posterior estimate falls below a given threshold. When juxtaposing the BMR CDF
trajectories with those obtained from the Tiered-FF method (sparsification is
induced by the regularised half-cauchy prior), it is evident that BMR
furnishes more pronounced sparsification. This distinction is crucial, as the
stochastic BMR not only matches or surpasses the performance of the Tiered-FF
algorithm but also averages a $15\%$ faster stochastic gradient descent.

(a) MLP

(b) Lenet

Figure 4: Posterior expectations (color coded) over model parameters obtained
using different approximate inference schemes at the first layer of (a) MLP
architecture, and (b) LeNet architectures.

To illustrate the structural learning variations among algorithms, Figure 4
presents heatmaps of posterior expectations obtained using the four different
methods. The Figure 4 reveals subtle differences between inferred
representations of the MLP and LeNet-5 architecture’s input layers trained on
the Fashion MNIST dataset. Divergent compression rates among the algorithms
indicate inherent trade-offs between efficiency and performance. It is evident
that the stochastic BMR strikes a balance between compression advantages and
performance, as it is less prone to over-pruning as compared to the Tiered-FF
method (two featured of the LeNet-5 input layer are effective removed - see
Figure 4(b)).

##  4 Discussion

In this study, we presented a novel algorithm – stochastic Bayesian model
reduction – designed for an efficient Bayesian sparsification of deep neural
networks. Our proposed method seamlessly integrates stochastic and black-box
variational inference with Bayesian model reduction (BMR), a generalisation of
the Savage-Dickey ratio. Through the stochastic BMR strategy, we enable
iterative pruning of model parameters, relying on posterior estimates acquired
from a straightforward variational mean-field approximation to the generative
model. This model is characterized by Gaussian priors over individual
parameters and layer-specific scale parameters. The result is an efficient
pruning algorithm for which the computational demand of the pruning step is
negligible compared to the direct stochastic black-box optimization of the
full hierarchical model.

Over recent years, the Bayesian sparsification of neural networks has gained
momentum, primarily driven by the spike-and-slab prior Bai et al. (2020);
Hubin and Storvik (2023); Jantre et al. (2021); Sun et al. (2022); Ke and Fan
(2022). These works have showcased the remarkable sparsification capabilities
inherent to such shrinkage priors. Nevertheless, when juxtaposed with the
stochastic BMR algorithm, they often necessitate supplementary assumptions
related to the approximate posterior. These assumptions, in turn, lead to a
more computation-intensive model inversion. Moreover, in contrast to related
approaches, the versatility of stochastic BMR allows its integration with more
efficient optimization techniques, like variational Laplace Daxberger et al.
(2021b) and proximal-gradient methods Khan et al. (2018), provided the
resulting approximate posterior in the form of a normal distribution is apt
for the application at hand.

The insights obtained here pave the way for a deeper exploration of the
potential applications of Bayesian model reduction across a wider array of
architectures and tasks in probabilistic machine learning, such as audiovisual
and natural language processing tasks. A more detailed fine tuning of the core
dynamics of these algorithms, in terms of iterations steps, learning rates,
and other free-parameters, might be the key to unveiling even more proficient
Bayesian deep learning methodologies in the near future.

###  4.1 Acknowledgments

We would like to thank Conor Heins, Magnus Koudahl, and Beren Millidge for
valuable discussions during the initial stages of this work.

###  4.2 Funding statement

SK acknowledges support by DFG TRR 265/1 (Project ID 402170461, B09) and
Germany’s Excellence Strategy – EXC 2050/1 (Project ID 390696704) – Cluster of
Excellence “Centre for Tactile Internet with Human-in-the-Loop” (CeTI) of
Technische Universität Dresden.

##  Appendix A Specification of neural network architectures

For the simple multi-layer perceptron, we configure the architecture with five
hidden layers, each comprising 400 neurons. The chosen activation function is
the Swish activation function (Ramachandran et al., 2017).

For the LeNet-5 architecture, we adhere to the original design, which includes
three convolutional layers, average pooling following the initial two
convolutional layers, and two linear layers. The activation function used is
the hyperbolic tangent. The convolutional layers employ a kernel size of
$5\times 5$, while the average pooling uses a window of shape $2\times 2$. For
the FashionMNIST dataset, the feature counts of the convolutional layers are
designated as (6, 16, 120), and the two linear layers have neuron counts of
(84, 10). However, for the CIFAR10 and CIFAR100 datasets, we elevate the
feature counts of the convolutional layers to (18, 48, 360), with linear layer
neuron counts set to (256, 10) for CIFAR10 and (256, 100) for CIFAR100.

For the MlpMixer architecture we employ six layers and a patch resolution of
$4\times 4$. Across all datasets, we maintain constant values for hidden size
($C$), sequence length ($S$), MLP channel dimension ($D_{C}$), and MLP token
dimension ($D_{S}$); specifically $C=256$, $S=64$, $D_{C}=512$ and $D_{S}=512$
for all datasets.

For the VisionTransformer architecture, we adopt a slightly modified version
of the ViT-Tiny setup: we use six layers, eight heads for each attention
block, an embedding dimension of 256, and a hidden dimension of 512. The patch
resolution of $4\times 4$ is consistent with the MlpMixer. In both MlpMixer
and VisionTransformer architectures, the GeLU activation function is utilized
(Hendrycks and Gimpel, 2016).

For training using the maximum a posteriori estimate (Flat-MAP), dropout
regularization, with dopout probability set to 0.2, is applied to all linear
layers across all architectures, with the exception of the MlpMixer.

##  Appendix B Reparameterisation

In the centered parameterization of a generative model, Stochastic Variational
Inference (SVI) with a fully factorized posterior yields a non-sparse
solution, undermining the objective of employing shrinkage priors (Ghosh et
al., 2019). Typically, this limitation is addressed by adopting the non-
centered parameterization of the prior.

Consider the unique property of the half-Cauchy distribution: given $x\sim
C^{+}(0,1)$, and $z=bx$ the resulting probability distribution for $z$ is
$z\sim C^{+}(0,b)$. Therefore, the non-centred parameterization is formulated
as

$\begin{split}\hat{\tau}^{l}_{i}&\sim\mathcal{C}^{+}(0,1)\\\
\hat{\lambda}^{l}_{ij}&\sim\mathcal{C}^{+}(0,1)\\\
\hat{w}^{l}_{ij}&\sim\mathcal{N}\mathinner{\left(0,1\right)}\\\
\mathinner{\left[\gamma^{l}_{ij}\right]}^{2}&=\frac{\left[c^{l}\tau^{l}_{0}\hat{\tau}^{l}_{i}\hat{\lambda}^{l}_{ij}\right]^{2}}{[c^{l}]^{2}+\left[\tau_{0}^{l}\hat{\tau}^{l}_{i}\hat{\lambda}^{l}_{ij}\right]^{2}}\\\
w^{l}_{ij}&=\gamma^{l}_{ij}\hat{w}^{l}_{ij}\end{split}$ (24)

However, while the half-Cauchy distribution is frequently chosen for sampling-
based inference, it poses challenges in variational inference (Piironen and
Vehtari, 2017). Firstly, exponential family-based approximate posteriors
(e.g., Gamma or log-Normal distributions) inadequately capture the half-Cauchy
distribution’s fat tails. Secondly, using a Cauchy approximating family for
the posterior results in high variance gradients during stochastic variational
inference (Ghosh et al., 2019). Hence, in the context of stochastic
variational inference, the half-Cauchy distribution undergoes a
reparameterization, as described in (Ghosh et al., 2018):

$x\sim\mathcal{C}^{+}(0,b)\equiv
x=\sqrt{\frac{1}{u}},u\sim\Gamma\mathinner{\left(\frac{1}{2},\frac{1}{v}\right)},v\sim\Gamma\mathinner{\left(\frac{1}{2},b^{2}\right)}$
(25)

or, when represented in the non-centered parameterization:

$x=b\sqrt{\frac{v}{u}},u\sim\Gamma\mathinner{\left(\frac{1}{2},1\right)},v\sim\Gamma\mathinner{\left(\frac{1}{2},1\right)}$
(26)

##  Appendix C Supplementary Material

Figure S1:  Classification performance comparison on CIFAR10 dataset for
different neuronal architectures and approximate inference schemes. Figure S2:
Classification performance comparison on CIFAR100 dataset for different
neuronal architectures and approximate inference schemes.

## References

  * Wang and Yeung [2020] Hao Wang and Dit-Yan Yeung.  A survey on bayesian deep learning.  _ACM Computing Surveys (CSUR)_ , 53(5):1–37, 2020\.
  * Wilson [2020] Andrew Gordon Wilson.  The case for bayesian deep learning, 2020.  URL https://arxiv.org/abs/2001.10995.
  * Wang and Yeung [2016] Hao Wang and Dit-Yan Yeung.  Towards bayesian deep learning: A framework and some existing methods.  _IEEE Transactions on Knowledge and Data Engineering_ , 28(12):3395–3408, 2016.
  * Murphy [2022] Kevin P Murphy.  _Probabilistic machine learning: an introduction_.  MIT press, 2022.
  * Ghahramani [2015] Zoubin Ghahramani.  Probabilistic machine learning and artificial intelligence.  _Nature_ , 521(7553):452–459, 2015.
  * Wilson and Izmailov [2020] Andrew G Wilson and Pavel Izmailov.  Bayesian deep learning and a probabilistic perspective of generalization.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 4697–4708. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/322f62469c5e3c7dc3e58f5a4d1ea399-Paper.pdf.
  * Izmailov et al. [2020] Pavel Izmailov, Wesley J Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson.  Subspace inference for bayesian deep learning.  In _Uncertainty in Artificial Intelligence_ , pages 1169–1179. PMLR, 2020.
  * Luo and Kareem [2020] Xihaier Luo and Ahsan Kareem.  Bayesian deep learning with hierarchical prior: Predictions from limited and noisy data.  _Structural Safety_ , 84:101918, 2020.  ISSN 0167-4730.  doi: https://doi.org/10.1016/j.strusafe.2019.101918.  URL https://www.sciencedirect.com/science/article/pii/S0167473019303820.
  * Louizos et al. [2017] Christos Louizos, Karen Ullrich, and Max Welling.  Bayesian compression for deep learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Fortuin [2022] Vincent Fortuin.  Priors in bayesian deep learning: A review.  _International Statistical Review_ , 2022.
  * Nalisnick et al. [2019] Eric Nalisnick, José Miguel Hernández-Lobato, and Padhraic Smyth.  Dropout as a structured shrinkage prior.  In _International Conference on Machine Learning_ , pages 4712–4722. PMLR, 2019.
  * [12] Skyler Seto, Martin T. Wells, and Wenyu Zhang.  _HALO: Learning to Prune Neural Networks with Shrinkage_ , pages 558–566.  doi: 10.1137/1.9781611976700.63.  URL https://epubs.siam.org/doi/abs/10.1137/1.9781611976700.63.
  * Ghosh et al. [2018] Soumya Ghosh, Jiayu Yao, and Finale Doshi-Velez.  Structured variational learning of Bayesian neural networks with horseshoe priors.  In Jennifer Dy and Andreas Krause, editors, _Proceedings of the 35th International Conference on Machine Learning_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 1744–1753. PMLR, 10–15 Jul 2018.  URL https://proceedings.mlr.press/v80/ghosh18a.html.
  * Snoek et al. [2015] Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams.  Scalable bayesian optimization using deep neural networks.  In _International conference on machine learning_ , pages 2171–2180. PMLR, 2015.
  * Krishnan et al. [2019] Ranganath Krishnan, Mahesh Subedar, and Omesh Tickoo.  Efficient priors for scalable variational inference in bayesian deep neural networks.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops_ , Oct 2019.
  * Daxberger et al. [2021a] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig.  Laplace redux - effortless bayesian deep learning.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , volume 34, pages 20089–20103. Curran Associates, Inc., 2021a.  URL https://proceedings.neurips.cc/paper/2021/file/a7c9585703d275249f30a088cebba0ad-Paper.pdf.
  * Cameron [2013] Ewan Cameron.  A generalized savage-dickey ratio, 2013.  URL https://arxiv.org/abs/1311.1292.
  * Rosa et al. [2012] MJ Rosa, K Friston, and W Penny.  Post-hoc selection of dynamic causal models.  _Journal of neuroscience methods_ , 208(1):66–78, 2012.
  * Friston and Penny [2011] Karl Friston and Will Penny.  Post hoc bayesian model selection.  _Neuroimage_ , 56(4):2089–2099, 2011.
  * Friston et al. [2016] Karl J Friston, Vladimir Litvak, Ashwini Oswal, Adeel Razi, Klaas E Stephan, Bernadette CM Van Wijk, Gabriel Ziegler, and Peter Zeidman.  Bayesian model reduction and empirical bayes for group (dcm) studies.  _Neuroimage_ , 128:413–431, 2016.
  * Friston et al. [2017] Karl J Friston, Marco Lin, Christopher D Frith, Giovanni Pezzulo, J Allan Hobson, and Sasha Ondobaka.  Active inference, curiosity and insight.  _Neural computation_ , 29(10):2633–2683, 2017\.
  * Friston et al. [2018] Karl Friston, Thomas Parr, and Peter Zeidman.  Bayesian model reduction, 2018.  URL https://arxiv.org/abs/1805.07092.
  * Smith et al. [2020] Ryan Smith, Philipp Schwartenbeck, Thomas Parr, and Karl J Friston.  An active inference approach to modeling structure learning: Concept learning as an example case.  _Frontiers in computational neuroscience_ , 14:41, 2020\.
  * Beckers et al. [2022] Jim Beckers, Bart van Erp, Ziyue Zhao, Kirill Kondrashov, and Bert de Vries.  Principled pruning of bayesian neural networks through variational free energy minimization.  _arXiv preprint arXiv:2210.09134_ , 2022.
  * Haußmann et al. [2020] Manuel Haußmann, Fred A Hamprecht, and Melih Kandemir.  Sampling-free variational inference of bayesian neural networks by variance backpropagation.  In _Uncertainty in Artificial Intelligence_ , pages 563–573. PMLR, 2020.
  * Mitchell and Beauchamp [1988] Toby J Mitchell and John J Beauchamp.  Bayesian variable selection in linear regression.  _Journal of the american statistical association_ , 83(404):1023–1032, 1988.
  * Piironen and Vehtari [2017] Juho Piironen and Aki Vehtari.  Sparsity information and regularization in the horseshoe and other shrinkage priors.  _Electronic Journal of Statistics_ , 11(2):5018–5051, 2017.
  * LeCun et al. [1989] Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel.  Backpropagation applied to handwritten zip code recognition.  _Neural computation_ , 1(4):541–551, 1989.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ , 2020.
  * Tolstikhin et al. [2021] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al.  Mlp-mixer: An all-mlp architecture for vision.  _Advances in neural information processing systems_ , 34:24261–24272, 2021.
  * Blei et al. [2017] David M Blei, Alp Kucukelbir, and Jon D McAuliffe.  Variational inference: A review for statisticians.  _Journal of the American statistical Association_ , 112(518):859–877, 2017.
  * Beal [2003] Matthew James Beal.  _Variational algorithms for approximate Bayesian inference_.  University of London, University College London (United Kingdom), 2003\.
  * Hoffman et al. [2013] Matthew D Hoffman, David M Blei, Chong Wang, and John Paisley.  Stochastic variational inference.  _Journal of Machine Learning Research_ , 2013.
  * Ranganath et al. [2014] Rajesh Ranganath, Sean Gerrish, and David Blei.  Black box variational inference.  In _Artificial intelligence and statistics_ , pages 814–822. PMLR, 2014.
  * Schulman et al. [2015] John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel.  Gradient estimation using stochastic computation graphs.  _Advances in neural information processing systems_ , 28, 2015.
  * Bingham et al. [2019] Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul A. Szerlip, Paul Horsfall, and Noah D. Goodman.  Pyro: Deep universal probabilistic programming.  _J. Mach. Learn. Res._ , 20:28:1–28:6, 2019.  URL http://jmlr.org/papers/v20/18-403.html.
  * Zhuang et al. [2020] Juntang Zhuang, Tommy Tang, Yifan Ding, Sekhar C Tatikonda, Nicha Dvornek, Xenophon Papademetris, and James Duncan.  Adabelief optimizer: Adapting stepsizes by the belief in observed gradients.  _Advances in neural information processing systems_ , 33:18795–18806, 2020.
  * Babuschkin et al. [2020] Igor Babuschkin, Kate Baumli, Alison Bell, Surya Bhupatiraju, Jake Bruce, Peter Buchlovsky, David Budden, Trevor Cai, Aidan Clark, Ivo Danihelka, Antoine Dedieu, Claudio Fantacci, Jonathan Godwin, Chris Jones, Ross Hemsley, Tom Hennigan, Matteo Hessel, Shaobo Hou, Steven Kapturowski, Thomas Keck, Iurii Kemaev, Michael King, Markus Kunesch, Lena Martens, Hamza Merzic, Vladimir Mikulik, Tamara Norman, George Papamakarios, John Quan, Roman Ring, Francisco Ruiz, Alvaro Sanchez, Rosalia Schneider, Eren Sezener, Stephen Spencer, Srivatsan Srinivasan, Wojciech Stokowiec, Luyu Wang, Guangyao Zhou, and Fabio Viola.  The deepmind jax ecosystem, 2020.  URL http://github.com/deepmind.
  * George and McCulloch [1993] Edward I George and Robert E McCulloch.  Variable selection via gibbs sampling.  _Journal of the American Statistical Association_ , 88(423):881–889, 1993.
  * Mobiny et al. [2021] Aryan Mobiny, Pengyu Yuan, Supratik K Moulik, Naveen Garg, Carol C Wu, and Hien Van Nguyen.  Dropconnect is effective in modeling uncertainty of bayesian deep networks.  _Scientific reports_ , 11(1):5458, 2021.
  * Bai et al. [2020] Jincheng Bai, Qifan Song, and Guang Cheng.  Efficient variational inference for sparse deep learning with theoretical guarantee.  _Advances in Neural Information Processing Systems_ , 33:466–476, 2020.
  * Hubin and Storvik [2023] Aliaksandr Hubin and Geir Storvik.  Variational inference for bayesian neural networks under model and parameter uncertainty.  _arXiv preprint arXiv:2305.00934_ , 2023.
  * Jantre et al. [2021] Sanket Jantre, Shrijita Bhattacharya, and Tapabrata Maiti.  Layer adaptive node selection in bayesian neural networks: Statistical guarantees and implementation details.  _arXiv preprint arXiv:2108.11000_ , 2021.
  * Sun et al. [2022] Yan Sun, Qifan Song, and Faming Liang.  Learning sparse deep neural networks with a spike-and-slab prior.  _Statistics & probability letters_, 180:109246, 2022.
  * Ke and Fan [2022] Xiongwen Ke and Yanan Fan.  On the optimization and pruning for bayesian deep learning.  _arXiv preprint arXiv:2210.12957_ , 2022.
  * Ghosh et al. [2019] Soumya Ghosh, Jiayu Yao, and Finale Doshi-Velez.  Model selection in bayesian neural networks via horseshoe priors.  _J. Mach. Learn. Res._ , 20(182):1–46, 2019.
  * Wand et al. [2011] Matthew P. Wand, John T. Ormerod, Simone A. Padoan, and Rudolf Frühwirth.  Mean Field Variational Bayes for Elaborate Distributions.  _Bayesian Analysis_ , 6(4):847 – 900, 2011.  doi: 10.1214/11-BA631.  URL https://doi.org/10.1214/11-BA631.
  * Daxberger et al. [2021b] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig.  Laplace redux-effortless bayesian deep learning.  _Advances in Neural Information Processing Systems_ , 34:20089–20103, 2021b.
  * Khan et al. [2018] Mohammad Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava.  Fast and scalable bayesian deep learning by weight-perturbation in adam.  In _International conference on machine learning_ , pages 2611–2620. PMLR, 2018.
  * Ramachandran et al. [2017] Prajit Ramachandran, Barret Zoph, and Quoc V Le.  Searching for activation functions.  _arXiv preprint arXiv:1710.05941_ , 2017.
  * Hendrycks and Gimpel [2016] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus).  _arXiv preprint arXiv:1606.08415_ , 2016.