# Posterior and variational inference for deep neural networks with heavy-
tailed weights

Ismaël Castillo<EMAIL_ADDRESS>
LPSM, Sorbonne Université
4, place Jussieu
75005, Paris, France Paul Egels<EMAIL_ADDRESS>
LPSM, Sorbonne Université
4, place Jussieu
75005, Paris, France

###### Abstract

We consider deep neural networks in a Bayesian framework with a prior
distribution sampling the network weights at random. Following a recent idea
of Agapiou and Castillo (2023), who show that heavy-tailed prior distributions
achieve automatic adaptation to smoothness, we introduce a simple Bayesian
deep learning prior based on heavy-tailed weights and ReLU activation. We show
that the corresponding posterior distribution achieves near-optimal minimax
contraction rates, simultaneously adaptive to both intrinsic dimension and
smoothness of the underlying function, in a variety of contexts including
nonparametric regression, geometric data and Besov spaces. While most works so
far need a form of model selection built-in within the prior distribution, a
key aspect of our approach is that it does not require to sample
hyperparameters to learn the architecture of the network. We also provide
variational Bayes counterparts of the results, that show that mean-field
variational approximations still benefit from near-optimal theoretical
support.

Keywords: Bayesian deep neural networks, fractional posterior distributions,
heavy-tailed priors, Variational Bayes, overfitting regime.

##  1 Introduction

The last decade has seen a remarkable expansion of the use of deep neural
networks (DNNs) through a broad range of applications such as imaging, natural
language processing, inverse problems to name a few. In parallel to this
spectacular empirical success, theory to provide understanding of the
performance of these methods is emerging. Among the key elements playing a
role in the mathematical and statistical analysis of DNNs are their
approximation properties (e.g. Yarotsky (2017); Schmidt-Hieber (2020); Kohler
and Langer (2021)), the choice of parameters in particular the network’s
architecture, and the convergence of the sampling or optimisation schemes
(e.g. gradient descent, variational criteria etc.) involved in the training.

Using a Bayesian approach for inference with deep neural networks is
particularly appealing, among others for the natural way to quantify
uncertainty through credible sets of the posterior distribution. A first
contribution on theoretical understanding in this direction is the work by
Polson and Ročková (2018), who show, building up in particular on the
approximation theory derived in Schmidt-Hieber (2020), that assigning
spike–and–slab prior distributions to the weights of a DNN with ReLU
activation leads to a near-minimax posterior convergence rate in nonparametric
regression. While from the mathematical perspective spike–and–slab priors can
be seen as a form of theoretical ideal, they can suffer from a particularly
high sampling complexity; indeed, sampling from the associated posterior
distributions often faces combinatorial difficulties, as one needs to explore
a high number of possible candidate models. While a possible answer, as we
discuss in more details below, is to deploy a Variational Bayes (VB)
approximation, it is particularly desirable (and even if VB is chosen as an
approximation) to develop simpler prior distributions, that still retain most
mathematical properties but are easier to implement in practice.

A number of recent contributions have considered prior distributions beyond
spike–and–slab (SAS) priors for the DNN’s weights. From the theoretical
perspective let us cite the work by Lee and Lee (2022), who broadens the
results for SAS of Polson and Ročková (2018) and considers a class of
shrinkage priors (whose characteristics depend on smoothness parameters of the
function to be estimated); the work by Kong et al. (2023) considers masked
Bayesian neural networks, where the ‘mask’ determines the position of zero
weights, and which allows for more efficient reparametrisation in view of
computations compared to SAS posteriors. The preprint Kong and Kim (2024) uses
non-sparse Gaussian priors on weights, building on the approximation theory of
Kohler and Langer (2021). From the algorithmical perspective, the work by
Ghosh et al. (2019) considers using horseshoe priors on weights. More
discussion on the previously mentioned priors and their relation to the
present work can be found in Section 4 below.

Let us mention also two lines of work in Bayesian deep learning that are
somewhat different from our approach but with natural connexions, in
particular in terms of the tails of the distributions arising in the networks.
The first considers DNNs in the large width limits, following seminal ideas of
Neal (1996): using Gaussian priors on weights, Sell and Singh (2023) use the
approximation by a Gaussian process obtained obtained in large width to
propose posterior sampling schemes; the work by Lee et al. (2023) instead
considers the large width limit with heavier-tailed distributions on weights,
obtaining a mixture of Gaussian processes in the limit. The second line of
work has no activation functions involved, but models directly random
functions accross layers, which leads to deep Gaussian processes as introduced
in Damianou and Lawrence (2013) and recently investigated in terms of
convergence by Finocchio and Schmidt-Hieber (2023) and Castillo and
Randrianarisoa (2024).

Variational Bayes is a particularly popular approach in machine learning and
statistics, where the idea is to approximate a possibly complex posterior
distribution by an element of a simpler class of distributions, effectively
requiring to solve an optimisation problem, see e.g. Blei et al. (2017). This
is a method of choice for approximating posteriors in deep neural networks,
with a statistical-computational trade-off through the choice of the
variational class, which is often taken to be mean-field, thereby requiring a
form of independence along a certain parametrisation. Following general ideas
from Alquier and Ridgway (2020) (see also Yang et al. (2020); Zhang and Gao
(2020)), the work by Chérief-Abdellatif (2020) proves that taking a mean-field
variational class combined to the spike-and-slab prior of Polson and Ročková
(2018) leads to a variational posterior that converges at the same near-
optimal rates towards the true regression function. Simulations using
stochastic gradient optimisation for this variational posterior, and some
extensions, are considered in Bai et al. (2020). Using simpler sieve-type
priors on DNNs but requiring a form of model selection, the work Ohn and Lin
(2024) considers adaptation within a family of variational posteriors.

For more discussion and further references on Bayesian deep learning we refer
to Papamarkou et al. (2024), Alquier (2024), Castillo (2024), Chapter 4, and
to the review paper Arbel et al. (2023).

Let us now turn to a summary of the contributions of the paper. We prove that
putting a suitably rescaled heavy-tailed distribution on deep neural network
weights leads to an optimal rate of convergence (up to logarithmic factors) of
the corresponding posterior distribution in nonparametric regression with
random design. The best rate is automatically attained without assuming any
knowledge of regularity parameters of the unknown regression function,
achieving in particular simultaneous adaptation to both smoothness and
intrinsic dimensions when the regression function has a compositional
structure with effective intrinsic dimensions (typically) of smaller order
than the input dimension. We derive similar results for scalable (mean-field)
Variational Bayes approximations of the posterior distribution; each
coordinate of the VB approximation has only two parameters to be fitted, one
for location and one for scale.

To help putting the results into perspective, we note that many fundamental
statistical results so far on DNNs are often stated with an ‘oracle’ choice of
the network architecture: for wide and moderately deep DNNs for instance, if
the network width is well chosen in terms of both smoothness and
dimensionality parameters, Schmidt-Hieber (2020) and Kohler and Langer (2021)
show that an empirical risk minimiser on neural networks of the prescribed
(oracle) architecture achieves a near-optimal minimax contraction rate for
compositional classes. Similarly, Suzuki  shows that deep learning methods are
‘adaptive’ to the intrinsic dimension for anisotropic smoothness classes, with
adaptation meaning here that the achieved rate depends only on the intrinsic
dimension (as opposed to the input dimension); however the results are
achieved with a network architecture that still depends on the smoothness and
intrinsic dimension parameters, which are typically unknown in practice.
Building up on the seminal approximation results of the just cited papers (as
well as Nakada and Imaizumi (2020) for data on geometric objects), we show
that well-chosen heavy tailed weight distributions enable one to obtain
statistical adaptation, simultaneously in terms of smoothness and dimension
parameters, using Bayesian fractional posterior distributions or their mean-
field variational counterparts. Beyond classes of compositions in
nonparametric random design regression, to illustrate the flexibility of our
method we consider two other applications: geometrical data through the use of
the Minkowski dimension as in Nakada and Imaizumi (2020) and anisotropic
classes as in Suzuki and Nitanda (2021). For both we show that simultaneous
statistical adaptation is achieved for fractional posteriors and mean-field
VB.

Outline of the paper. In Section 2, we introduce the statistical framework,
deep neural networks, and the proposed method; the heavy-tailed prior
distribution on network weights is defined, as well as the fractional
posterior distribution and its variational approximation. In Section 3 we
present our main results, declined along three settings: compositional classes
in Sections 3.1–3.2, geometric data in Section 3.3 and anisotropic classes in
Section 3.4; in each setting results are derived for the fractional posterior
distribution, and a mean-field VB approximation. Section 4 contains a brief
discussion, including some remarks on sampling and future work and open
questions. Section 5 contains the proof of the main results. The Appendix
includes a number of short technical lemmas as well as a lemma on bounds of
weights of neural networks.

_Notation._ For any real number $\beta$, let $\lfloor\beta\rfloor$ be the
largest integer strictly smaller than $\beta$. We write the set of positive
real numbers as $\mathbb{R}_{>0}$. For any two sequences we write
$a_{n}\lesssim b_{n}$ for inequality up to a constant and $a_{n}\asymp b_{n}$
whenever $a_{n}\gtrsim b_{n}$ is also satisfied. For two real numbers $a,b$,
we write $a\vee b=\max(a,b)$ and $a\wedge b=\min(a,b)$. For $T\geq 1$ an
integer, $[T]$ denotes the set of integers $\\{1,\dots,T\\}$. For $1\leq
p\leq\infty$, we denote $|\,\cdot\,|_{p}$ (resp. $\lVert\,\cdot\,\rVert_{p})$
the $\ell^{p}$ (resp. $L^{p}$) norms and $\lVert\,\cdot\,\rVert_{L^{p}(\mu)}$
when we want to specify the integrating measure, say $\mu$. For $d\geq 1$ an
integer, $F\geq 0$ and $D\subset\mathbb{R}^{d}$, let
$\mathcal{C}_{d}^{\beta}(D,F)$ be the ball of $d$-variate $\beta$-Hölder
functions

$\mathcal{C}^{\beta}_{d}(D,F):=\left\\{f:D\subset\mathbb{R}^{d}\to\mathbb{R}:\sum_{|\mathbf{k}|<\beta}||\partial^{\mathbf{k}}f||_{\infty}+\sum_{|\mathbf{k}|=\lfloor\beta\rfloor}\underset{\underset{x\neq
y}{x,y\in
D}}{\sup}\frac{|\partial^{\mathbf{k}}f(x)-\partial^{\mathbf{k}}f(y)|}{|x-y|_{\infty}^{\beta-\lfloor\beta\rfloor}}\leq
F\right\\}.$

Elements of this ball have all their partial derivatives up to order
$\lfloor\beta\rfloor$ bounded, and their partial derivatives of order
$\lfloor\beta\rfloor$ are $(\beta-\lfloor\beta\rfloor)$-Hölder-continuous. We
denote by $\operatorname{KL}(P,Q)$ and $D_{\alpha}(P,Q)$ respectively the
Kullback-Leibler and Rényi divergence of order $\alpha$ between probability
measures $P$ and $Q$ (see Appendix A for their definitions).

##  2 Bayesian deep neural networks

Let us now introduce the notation and statistical framework used to express
our results.

###  2.1 Statistical framework

We consider the nonparametric random design Gaussian regression model with
possibly large (but fixed) dimension $d\geq 1$: one observes $n$ independent
and identically distributed (iid) pairs of random variables
$(X_{i},Y_{i})\in[0,1]^{d}\times\mathbb{R}$, $1\leq i\leq n$, with

$Y_{i}=f_{0}(X_{i})+\xi_{i}.$ (1)

The design points $(X_{i})$ are iid draws from a distribution $P_{X}$ on
$[0,1]^{d}$ and, independently, the noise variables $(\xi_{i})_{i}$ are iid
with $\mathcal{N}(0,\sigma^{2})$ distribution, where for simplicity we set
$\sigma=1$. The function $f_{0}$ belongs to a parameter set $\mathcal{F}$ of
functions to be specified below. Writing $P_{f_{0}}$ for the distribution of
$(X_{1},Y_{1})$, the joint distribution of the observations
$(X,Y):=\\{(X_{i},Y_{i})_{i}\\}$ is $P_{f_{0}}^{\otimes n}$. The goal here is
to recover the ‘true’ function $f_{0}$ from the observed data
$(X_{i},Y_{i})_{i}$, our concentration results will be given with respect to
the integrated quadratic loss

$||f_{0}-f||_{L^{2}(P_{X})}^{2}=\int(f_{0}-f)^{2}\,dP_{X}.$

The minimax rate of estimation of a $\beta$–smooth function $f_{0}$, in the
Hölder sense say, in squared-integrated loss in this model is known to be of
order $n^{-2\beta/(2\beta+d)}$, which suffers from the curse of
dimensionality, as this rate becomes very slow for large $d$ unless $f_{0}$ is
very smooth. In recent years regularity classes based on compositions of
functions have emerged as a benchmark for the performance of deep learning
methods: indeed, these classes enable to model smaller effective dimensions
compared to the input dimension $d$, with optimal rates, to be described in
more details below, that only depend on the intrinsic dimensions at the
successive steps of the composition.

_Compositional structure_. Let $q\geq 1$ be an integer, $K>0$,
$\mathbf{d}=(d_{0},\dots,d_{q+1})$, $\mathbf{t}=(t_{0},\dots t_{q})$ vectors
of ambient and effective dimensions such that $t_{i}\leq d_{i}$ and
$\boldsymbol{\beta}=(\beta_{0},\dots,\beta_{q})$ a vector of smoothness
parameters. Following Schmidt-Hieber (2020), define a class of compositions as

$\displaystyle\mathcal{G}(q,\mathbf{d},\mathbf{t},\boldsymbol{\beta},K):=\Big{\\{}f$
$\displaystyle=g_{q}\circ\dots\circ
g_{0}\,:\,g_{i}=(g_{ij})_{j\in[d_{i+1}]}:[a_{i},b_{i}]^{d_{i}}\to[a_{i+1},b_{i+1}]^{d_{i+1}},$
$\displaystyle g_{ij}=\bar{g}_{ij},\text{ for some
}\bar{g}_{ij}\in\mathcal{C}_{t_{i}}^{\beta_{i}}([a_{i},b_{i}]^{t_{i}},K),\text{
and }|a_{i}|,|b_{i}|\leq K\Big{\\}}$

where we set $d_{0}=d$ and $d_{q+1}=1$ (for an alternative definition, see
also Finocchio and Schmidt-Hieber (2023)). Let us emphasize that even if the
$g_{ij}$’s take as input a vector of size $d_{i}$, the value of the output
depends on at most $t_{i}\leq d_{i}$ variables, which allows one to model low-
dimensional structures. We further note that a function $f_{0}$ from
$\mathcal{G}(q,\mathbf{d},\mathbf{t},\boldsymbol{\beta},K)$ does not have a
unique decomposition as a composition of $g_{i}$’s. This is not a concern
here, as we are solely interested in estimating $f_{0}$ and not $g_{i}$’s
themselves. Many models, such as additive models and sparse tensor
decompostions, fall within this framework, see e.g. Schmidt-Hieber (2020),
Section 4, for more discussion.

The minimax rate of estimation of $f_{0}$ over the above compositional class,
in terms of the quadratic loss as defined above turns out to depend only on
the effective dimensions $t_{i}$ and ‘effective smoothness’ parameters, for
$0\leq i\leq q$,

$\beta_{i}^{*}:=\beta_{i}\prod_{k=i+1}^{q}(\beta_{k}\wedge 1).$ (2)

An intuition behind this is that Hölder regularities larger than $1$ cannot in
general ‘propagate’ through inner layers of composition to make the overall
Hölder smoothness larger than individual regularities of the components, but
regularities smaller than one can decrease the overall smoothness.

Schmidt-Hieber (2020), Theorem 3, shows that if
$t_{i}\leq\min(d_{0},\dots,d_{q+1})$ for all $i$, then

$\phi_{n}^{*}:=\underset{0\leq i\leq
q}{\max}\left\\{n^{-\frac{\beta_{i}^{*}}{2\beta_{i}^{*}+t_{i}}}\right\\}.$ (3)

is a lower bound for the minimax rate of estimation in the regression model
for compositional functions and squared-integrated loss.

###  2.2 Deep ReLU neural networks

The rectified linear unit (ReLU) activation function is defined for any real
$x$ as

$\rho(x):=\max(0,x).$

When $x$ is a multi-dimensional vector, we simply apply $\rho$
coordinate–wise. Let $L\geq 1$ be an integer and
$\mathbf{r}=(r_{0},\dots,r_{L+1})\in\mathbb{N}^{L+2}$, we say that a function
$f:\mathbb{R}^{r_{0}}\to\mathbb{R}^{r_{L+1}}$ is the realisation of a ReLU
neural network with architecture (or structure) $(L,\mathbf{r})$ if there is a
family of matrix-vectors $\\{(W_{l},v_{l})\\}_{l\in[L+1]}$, with
$(W_{l},v_{l})\in\mathbb{R}^{r_{l}\times r_{l-1}}\times\mathbb{R}^{r_{l}}$
such that

$f=A_{L+1}\circ\rho\circ\dots\circ\rho\circ A_{1},$ (4)

where $A_{l}$ is the affine map $y\mapsto W_{l}y+v_{l}$. Here $L$ and
$\mathbf{r}$ are respectively called the _length_ (or _depth_) and the _width
vector_ of the network architecture. Here we take the convention that the
composition in (4) ends with a shift (as in Kohler and Langer (2021)). The
network weights are the entries of the matrices $(W_{l})$ and shifts vectors
$(v_{l})$ of the successive layers. The total number of weight parameters of a
network with architecture $(L,\mathbf{r})$ is

$T=T(L,\mathbf{r}):=\sum_{l=0}^{L}r_{l}r_{l+1}+\sum_{l=1}^{L+1}r_{l}.$ (5)

Let $\theta:=(\theta_{k})_{1\leq k\leq T}\in\mathbb{R}^{T}$ be a vector
containing all the network weights written in a given order (the choice of the
order does not matter). Given a family of matrices-vectors
$\\{(W_{l},v_{l})\\}_{l\in[L+1]}$ specifying a neural network, to simplify the
notation it is sometimes easier to consider the shift vector $v_{l}$ to be the
$0$-th column of $W_{l}$ so that the set of parameters is
$(\theta_{k}\,:\,k\in[T])\equiv(W_{l}^{(ij)}\,:\,l\in[L+1]\,,\,(i,j)\in\\{1,\dots,r_{l}\\}\times\\{0,\dots,r_{l-1}\\})$.
This is made rigorous from the fact that

$\begin{pmatrix}v^{(1)}&W^{(11)}&\dots&W^{(1r_{l-1})}\\\
\vdots&\vdots&&\vdots\\\
v^{(r_{l})}&W^{(r_{l}1)}&\dots&W^{(r_{l}r_{l-1})}\end{pmatrix}\begin{pmatrix}1\\\
x^{(1)}\\\ \vdots\\\ x^{(r_{l-1})}\\\ \end{pmatrix}=Wx+v.$

The set of networks $\mathcal{F}(L,\mathbf{r})$ of architecture
$(L,\mathbf{r})$. Let $\mathcal{F}(L,\mathbf{r})$ be the set of all
realisations of ReLU neural networks with structure $(L,\mathbf{r})$, for
$L\geq 1$ a given depth and $\mathbf{r}$ a vector of widths. Also define
$\mathcal{F}(L,\mathbf{r},s)$ as the set of realisations of neural networks
that have at most $s$ active parameters, that is
$\mathcal{F}(L,\mathbf{r},s):=\left\\{f\in\mathcal{F}(L,\mathbf{r})\,,\,\\#\\{k\in[T]\,;\,\theta_{k}\neq
0\\}\leq s\right\\}$, with $\\#S$ the cardinality of a finite set $S$.

###  2.3 Heavy-tailed priors on deep neural networks

_Frequentist analysis of tempered posteriors._ The reconstruction of $f_{0}$
from the data $(X_{i},Y_{i})_{i}$ given by the regression model (1) will be
conducted through a (generalised–) Bayesian analysis. From a prior
distribution $\Pi$ on a (measurable) parameter space $\mathcal{F}$ containing
$f_{0}$, given $\alpha\in(0,1)$, one can construct a data-dependent
probability measure on $\mathcal{F}$ called tempered posterior distribution,
given by, for any measurable $A\subset\mathcal{F}$,

$\Pi_{\alpha}[A\,|\,X,Y]:=\frac{\int_{A}e^{-\frac{\alpha}{2}\sum_{i=1}^{n}(Y_{i}-f(X_{i}))^{2}}d\Pi(f)}{\int
e^{-\frac{\alpha}{2}\sum_{i=1}^{n}(Y_{i}-f(X_{i}))^{2}}d\Pi(f)}.$ (6)

Setting $\alpha$ equal to $1$ in the last display leads to the classical
posterior distribution, the conditional distribution of $f$ given $X,Y$ in
model (1), when $f$ is equipped with a prior distribution $\Pi$. Here we work
with a tempered (or ‘fractional’) posterior instead, which corresponds to
rising the Gaussian likelihood to a power $\alpha$, here taken to be smaller
than $1$, which effectively downweights the influence of the data. Tempered
posteriors are particular cases of so-called Gibbs posteriors, where the
(log–)likelihood is itself replaced by a user-chosen empirical quantity; both
objects are particularly popular in both machine learning and statistical
applications, see e.g. L’Huillier et al. (2023) (or the more general
references Ghosal and van der Vaart (2017); Alquier (2024)) for more
discussion. The reason we consider here fractional posteriors with $\alpha<1$
is mainly of technical nature and related to our choice of prior below; we
refer to the discussion in Section 4 for more on this.

To analyse the convergence of the fractional posterior distribution, we take a
frequentist approach, which means that we analyse the behaviour of
$\Pi_{\alpha}[\cdot\,|\,X,Y]$ in (6) under the assumption that the data has
effectively been generated according to model (1) with $f_{0}$ a fixed ‘true’
regression function. The goal is then to characterise the rate at which the
fractional posterior distribution concentrates its mass around the true
function $f_{0}$ (if at all). It turns out that a sufficient condition for
this is, for contraction in terms of a Rényi–type divergence, that the prior
distribution puts enough mass on a certain neighborhood of $f_{0}$ (see e.g.
Bhattacharya et al. (2019), L’Huillier et al. (2023) and references therein);
the version we use in the present paper can be found as Lemma 15 in Appendix A
below. For more context and references on general theory of frequentist
analysis of (possibly generalised–) posterior distributions, we refer to the
book by Ghosal and van der Vaart (2017) and the monograph Castillo (2024).

Known upper-bound on $f_{0}$. As is common in the analysis of deep learning
algorithms, we assume for simplicity that an upper-bound $M_{0}$ on the true
regression function $f_{0}$ is known: $\|f_{0}\|_{\infty}\leq M_{0}$. Our
results continue to hold in slightly weaker form if this is not the case.

_Heavy-tailed priors on network coefficients_. For a given depth $L$ and width
vector $\mathbf{r}$, we define a prior distribution on
$\mathcal{F}(L,\mathbf{r})$ as follows. Importantly, in our method, the choice
of $L,\mathbf{r}$ are deterministic and given in (11), (12) (or (13)) below.

Let $T$ be the total number of parameters in the network, and recall the
notation $[T]=\\{1,\ldots,T\\}$ and that we order network parameters in a
fixed arbitrary order. The heavy-tailed prior samples the network coefficients
as

$\forall k\in[T],\qquad\theta_{k}\sim\sigma_{k}\,\zeta_{k},$ (7)

  * •

where the scaling factors $\sigma_{k}$ for $k\in[T]$ are positive real numbers
to be chosen below;

  * •

the random variables $\zeta_{k}$’s for $k\in[T]$ are independent identically
distributed with a given heavy-tailed density $h$ on $\mathbb{R}$ that
satisfies the following conditions (H1) to (H3): there exist constants
$c_{1},c_{2}>0$ and $\kappa\geq 0$ such that

    1. (H1)

$h$ is symmetric, positive, bounded, and decreasing on $[0,+\infty)$,

    2. (H2)

for all $x\geq 0$,

$\log\left(1/h(x)\right)\leq c_{1}(1+\log^{1+\kappa}(1+x)),$

    3. (H3)

for all $x\geq 1$,

$\overline{H}(x):=\int_{x}^{+\infty}h(u)du\leq\frac{c_{2}}{x}.$

Let us now give typical examples for $(\sigma_{k})$ and $h$ as well as some
intuition behind this choice of prior. The conditions on $h$ accommodate many
standard choices of densities with polynomial tails, such as Cauchy and most
Student densities: all these verify (H2) with $\kappa=0$ for a sufficiently
large constant $c_{1}>0$; a typical possible choice of $\sigma_{k}$ allowed in
all theorems below is $\exp(-\log^{2(1+\delta)}{n})$ for $\delta>0$,
independent of $k$ (i.e. one takes this same value for all network
coefficients), with a decrease in $n$ just slightly faster than a polynomial
(which would correspond to $\exp(-a\log{n})$).

This type of priors has been recently introduced by Agapiou and Castillo
(2024) for use in a number of classical nonparametric models such as
regression, density estimation and classification, where it is shown to
achieve optimal convergence rates (up to log factors) for tempered posteriors
with automatic adaptation to smoothness; efficient simulations using infinite-
dimensional MCMC schemes are also considered therein.

There are two key ideas behind this choice of prior. First, the heavy tailed
distribution enables one to model large values if needed (here ‘large’ will
mean at most of the order of a polynomial in $n$); second, the scaling factors
are taken sufficiently small to prevent overfitting. Unlike for spike-and-slab
priors (e.g. Polson and Ročková (2018), Kong et al. (2023)) the prior sets no
coefficients exactly to zero. This is often an advantage computationally, as
modelling exact zero coefficients in the prior typically requires performing a
form of discrete model selection in the posterior sampling, which can have a
large computational cost even for moderate sample sizes. Note also that the
architecture of the network is non-random: we refer to Section 3 for two
possible specific choices of $L,\mathbf{r}$. This is in contrast to model-
selection type priors where a prior is put on $\mathbf{r},L$ or both. Again,
performing model selection for instance on the width $\mathbf{r}$ is a form of
hyper-parameter selection problem which often leads to additional
computational cost, as one needs to obtain the posterior on the
hyperparameter; see also the discussion section in Agapiou and Castillo
(2024), where similar comments are made with respect to Gaussian process
priors in regression, for which hyperparameters need to be calibrated (e.g.
the key lengthscale parameter if one uses a squared-exponential kernel).

The overall prior $\Pi$ on DNNs. We are now in position to define the prior on
functions $f$ we consider:

  1. 1.

choose the architecture $(L,\mathbf{r})$ according to one of the two
deterministic choices (11)–(12) or (11)–(13) as specified at the beginning of
Section 3 below;

  2. 2.

given $(L,\mathbf{r})$, sample network weights $\theta_{k}$ according the
heavy-tailed prior as defined above; next form the neural network realisation
$f$ as in (4).

A final remark is that the choice of architecture parameters we make below
places us in the overfitting regime: namely, our prior always fits more
weights than the optimal number predicted by the classical bias-variance
trade-off, see the Discussion below for more on this.

###  2.4 Heavy-tailed mean-field tempered variational approximations

Let us now turn to a variational approach, where the idea is to approximate
the tempered posterior on a subset $\mathcal{S}$ of (simpler) probability
distributions on the parameter space $\mathcal{F}$. The variational tempered
approximation (on $\mathcal{S}$) $\hat{Q}_{\alpha}=\hat{Q}_{\alpha}(X,Y)$ is
defined as the best approximation of the tempered posterior
$\Pi_{\alpha}[\,\cdot\,|X,Y]$ by elements of $\mathcal{S}$ in the KL–sense

$\hat{Q}_{\alpha}:=\underset{Q\in\mathcal{S}}{\arg\\!\min}\,\operatorname{KL}(Q,\Pi_{\alpha}[\,\cdot\,|X,Y]),$
(8)

where the $\operatorname{KL}(P,Q)$ is Kullback-Leibler divergence between $P$
and $Q$ (see Appendix A). Below we consider a mean-field approximation, which
consists in taking a class $\mathcal{S}$ of distributions of product form,
assuming $\mathcal{F}$ has a natural parametrisation in coordinates
$(\theta_{1},\ldots,\theta_{T})$, which is the case here for DNNs by taking
the collection of network weights. The individual elements of the product are
taken below to belong to a scale-location family of heavy tailed
distributions.

For $h$ a (heavy-tailed) density function on $\mathbb{R}$ and
$\mu\in\mathbb{R},\varsigma>0$, we let $h_{\mu,\varsigma}$ be the shifted and
re-scaled density $h_{\mu,\varsigma}:x\mapsto h((x-\mu)/\varsigma)/\varsigma$
and set

$\mathcal{H}=\mathcal{H}(h):=\left\\{Q\,:\ dQ(x)=h_{\mu,\varsigma}(x)dx,\
\,\mu\in\mathbb{R},\varsigma>0\right\\}.$

That is, $Q$ belongs to $\mathcal{H}(h)$ if it has a density with respect to
the Lebesgue measure $h_{\mu,\varsigma}$ on $\mathbb{R}$ for some
$\mu,\varsigma$. For $h$ a density and $\lambda>0$, we denote its $\lambda$–th
moment by

$m_{\lambda}(h):=\int|x|^{\lambda}h(x)\,dx.$ (9)

######  Definition 1 (Heavy-tailed mean-field class)

Let $h$ be a density function satisfying (H1), (H2), (H3) with finite second
order moment $m_{2}(h)<\infty$. For $T\geq 1$ an integer, we define the heavy-
tailed mean-field class (generated from h) as

$\mathcal{S}_{HT}(h):=\left\\{Q\,:\,dQ(\theta)=\prod_{k=1}^{T}dQ_{k}(\theta_{k}),\text{
with }Q_{k}\in\mathcal{H}(h)\right\\}.$ (10)

Mean-field variational approximations such as (10) are particularly popular
(Blei et al., 2017) in that, by removing dependencies along a decomposition of
the parameter space $\mathcal{F}$ as above, they often allow for scalable
computation of the minimizer defined by the optimisation problem (8). The
mean-field class (10) corresponds to our heavy-tailed prior on DNNs defined by
(7). Variational approximation for spike-and-slab priors on neural networks
have been studied previously by Chérief-Abdellatif (2020) and Bai et al.
(2020). Adaptive variational concentration results for hierarchical uniform
priors have been recently obtained by Ohn and Lin (2024). Concentration tools
for variational approximations are recalled in Appendix A following ideas of
Alquier and Ridgway (2020), Yang et al. (2020), Zhang and Gao (2020) (see also
Castillo (2024) for an overview). For more context and discussion we also
refer to the review paper on variational Bayes Blei et al. (2017) and the
review on Bayesian neural networks Arbel et al. (2023).

##  3 Main results

We now consider the convergence of tempered posteriors for heavy tailed-priors
on the network weights. To illustrate the flexibility of our approach, we
explore simultaneous adaptation to intrinsic dimension and smoothness in three
different settings: nonparametric regression with compositional structures,
geometric data through the control of Minkowski’s dimension and anisotropic
Besov spaces. For the last two, our method is the first to achieve
simultaneous adaptation to dimension and smoothness to the best of our
knowledge.

A given architecture that overfits. In learning with deep neural networks, a
key aspect is the choice of architecture $(L,\mathbf{r})$. In our approach, we
choose a logarithmic depth: fix $\delta>0$ and let

$L:=\lceil\log^{1+\delta}n\rceil.$ (11)

Also, we fix beforehand a (relatively) large common width for network layers
by setting

$\mathbf{r}(\sqrt{n}):=(d,\lceil\sqrt{n}\rceil,\dots,\lceil\sqrt{n}\rceil,1),$
(12)

with $\mathbf{r}(\sqrt{n})$ of size $L+2$. Below we will also consider a
slightly larger choice of width

$\mathbf{r}(n):=(d,n,\dots,n,1),$ (13)

with $\mathbf{r}(n)$ again a line vector of size $L+2$. More generally, it is
easily seen from the proofs of the results below that any choice of widths
$r_{l}$ of polynomial order $r_{l}\asymp n^{a_{l}}$, for arbitrary given
$a_{l}\geq 1/2$ is compatible with the obtained rates, naturally placing the
prior in the overfitting regime (and already so for the smallest possible
choice $a_{l}=1/2$). Since one main choice of prior on weights below places
the same distribution on each weight, this means that the prior places no
special ‘penalty’ term for picking more coefficients.

Choice of scaling factors $\sigma_{k}$. We consider two main choices of
scaling parameters $\sigma_{k}$ for the heavy-tailed prior on coefficients
(7). The main choice discussed in the paper, and referred to as constant
$\sigma_{k}$s (although the constant depends on $n$, but not on $k$) is, for
$\delta>0$ as in (11) and any $k=1,\ldots,T$,

$\sigma_{k}=e^{-(\log{n})^{2(1+\delta)}},$ (14)

or equivalently $\log(1/\sigma_{k})=\log^{2(1+\delta)}{n}$, which is
independent of $k$.

We next consider a more general choice that includes (14) as a special case,
and referred to as directed $\sigma_{k}$s. It involves a little more notation,
as the network weights may now depend on their relative positions in the
network (recall their ordering is fixed though arbitrary) so may be skipped at
first read. Recall from the notation as in Section 2.2 that we interpret shift
vectors $v_{l}$ as the $0$-th columns of the weight matrices $W_{l}$. Suppose
we work with an architecture $(L,\mathbf{r})$, so that the network weights
$(\theta_{k})$ are in one–to–one correspondance with coefficients
$W_{l}^{(ij)}$ for $l\in[L+1]$, $i\in[r_{l}]$ and
$j\in\\{0,1,\ldots,r_{l-1}\\}$. We set, whenever $\theta_{k}$ corresponds to
$W_{l}^{(ij)}$,

$\sigma_{k}=\sigma_{l}^{(ij)},$ (15)

where for any admissible $l,i,j$ as above, and $\delta>0$ as in (11),

$\log^{2(1+\delta)}(i\vee
j)\leq\log(1/\sigma_{l}^{(ij)})\leq\log^{2(1+\delta)}n.$ (16)

The choice (14) corresponds to taking the second inequality in (16) to be an
equality. If one takes instead equality in the first inequality in (16), one
obtains a (double-index) version of the weights considered in Agapiou and
Castillo (2024) (who set $\sigma_{k}=\exp(-\log^{2}{k})$ with one index $k$).
The later choice enables one, with higher prior probability, to have larger
values on a number of weights (those with small $k$), which may be useful in
practice to avoid too many very small weights. We refer to Section 4 for more
discussion in the choice of weights.

In the sequel, for $\kappa\geq 0$ the parameter in condition (H2) on the prior
density ($\kappa=0$ for polynomial tails) and $\delta>0$ as in (11), we denote

$\gamma=2(1+\delta)(1+\kappa)+1.$ (17)

###  3.1 Adaptive contraction rate in regression

For a class of compositions
$\mathcal{G}(q,\mathbf{d},\mathbf{t},\boldsymbol{\beta},K)$ and $\gamma$ as in
(17), let us introduce the rate

$\phi_{n}:=\underset{0\leq i\leq
q}{\max}\,\left(\frac{\log^{\gamma}n}{n}\right)^{\frac{\beta_{i}^{*}}{2\beta_{i}^{*}+t_{i}}}.$
(18)

This coincides up to the logarithmic factor with the optimal rate
$\phi_{n}^{*}$ in the minimax sense as in (3) for estimating a $f_{0}$ in the
regression model if it is a composition as in the class $\mathcal{G}$.

###### Theorem 2

Consider data from the nonparametric random design regression model (1), with
$f_{0}\in\mathcal{G}(q,\mathbf{d},\mathbf{t},\boldsymbol{\beta},K)$ and
arbitrary unknown parameters. Let $\Pi$ be a heavy-tailed DNN prior as
described in Section 2.3 with

  * •

architecture $(L,\mathbf{r})$ with $L$ as in (11) and
$\mathbf{r}=\mathbf{r}(\sqrt{n})$ as in (12);

  * •

network weights $\theta_{k}\sim\sigma_{k}\zeta_{k}$ independently as in (7),
with

    * –

scaling factors $\sigma_{k}$ verifying either (14), or more generally (16);

    * –

random $\zeta_{k}$ with heavy-tailed density $h$ satisfying (H1)–(H3).

Then for any $\alpha\in(0,1)$, for $M>0$ large enough, $D_{\alpha}$ the
$\alpha$–Rényi divergence, as $n\to\infty,$

$E_{f_{0}}\Pi_{\alpha}\left[\left\\{f\,:\,D_{\alpha}(f,f_{0})\geq
M\phi_{n}^{2}\right\\}\,|\,X,Y\right]\to 0.$

Let us now formulate a corollary in terms of the squared integrated loss. For
$B>0$, let $C_{B}(\cdot)$ be the ‘clipping’ function
$C_{B}(x)=(-B)\vee(x\wedge B)$. Define a clipped tempered posterior as
$\Pi_{\alpha}^{B}[\cdot|X,Y]=C_{B}^{-1}\circ\Pi_{\alpha}[\cdot|X,Y]$.
Equivalently, if $f\sim\Pi_{\alpha}[\cdot|X,Y]$, then

$g=C_{B}(f)\sim\Pi_{\alpha}^{B}[\cdot|X,Y].$

The next Corollary shows that the rate above in terms of Rényi divergence
carries over to the $L^{2}(P_{X})$ loss as long as one works with a clipped
posterior, which only requires to assume that an upper bound on
$\|f_{0}\|_{\infty}$ is known, which is often assumed in theory for deep
learning methods (Corollary 3 itself is a direct application of Lemma 14
below). In the sequel, for simplicity we always state results in terms of
$D_{\alpha}$, but similarly results in the $L^{2}(P_{X})$ loss can be derived
as in the next Corollary.

###### Corollary 3

Under the setting and conditions of Theorem 2, and
$\Pi_{\alpha}^{B}[\cdot|X,Y]$ as defined in the last display, for any
$B>\|f_{0}\|_{\infty}$, as $n\to\infty$,

$E_{f_{0}}\Pi_{\alpha}^{B}\left[\left\\{f\,:\,\|f-f_{0}\|_{L^{2}(P_{X})}\geq
M\phi_{n}\right\\}\,|\,X,Y\right]\to 0.$

Theorem 2 shows that, when performing a re-scaling of every coefficient of
wide structured deep neural networks, by a well-chosen factor $\sigma_{k}$,
the heavy-tailed prior leads to nearly minimax adaptive concentration of the
tempered posterior regarding both the smoothness and the hidden compositional
structure of the true function $f_{0}$. Indeed, neither the smoothness
parameters $(\beta_{i})$ nor the intrinsic dimensions $(t_{i})$ were used to
define the prior $\Pi$. Theorem 7 below shows that a similar prior also leads
to adaptation to low-dimensional structure on the input data $(X_{i})$.

Theory of deep neural network approximation (see for example Kohler and Langer
(2021) or Lu et al. (2021)) suggests that, given a true regression function
$f_{0}$ in a model displaying low-dimensional structure (for example
compositional structure on $f_{0}$) there is a ‘minimal’ approximating network
structure
$(L^{*}(\boldsymbol{\beta},\mathbf{t}),\mathbf{r}^{*}(\boldsymbol{\beta},\mathbf{t}))$
in terms of depth and width. To achieve adaptation one could recover this
structure via model selection with a hierarchical prior (see for example
Polson and Ročková (2018), Kong et al. (2023), Ohn and Lin (2024)). Here
instead we show in Theorem 2 (resp. Theorem 7 below) that with our heavy-
tailed prior no hierarchical step is required to achieve adaptation and it is
sufficient to take a fixed (‘overfitting’) network architecture, independent
of $\boldsymbol{\beta}$ and $\mathbf{t}$ (resp. independent of $\beta$ and
$t^{*}$ in the result below) and draw independent coefficients.

We allow for some liberty for the choice of the decay coefficients
$\sigma_{k}$. One can for example chose the same decay
$\sigma_{k}=\exp(-\log^{2(1+\delta)}n)$ for every coefficient so that the
$\theta_{k}$ are independent and identically distributed. We refer to Section
4 for a brief discussion on other possible choices.

###### Remark 4

a) The rate $\phi_{n}$ includes a logarithmic factor which we did not try to
optimise; one can make the power $\gamma$ slightly smaller by taking a
slightly different depth, albeit with a slightly different choice of
$\sigma_{k}$’s. For simplicity we do not pursue such refinements here. b)
Recall that we have assumed that an upper bound $M_{0}$ on $f_{0}$ is known:
this is only used through the fact that we take a ‘clipping’ function to state
Corollary 3, where $B$ has to be larger than $M_{0}$. If this is not assumed,
then the conclusion of Theorem 2 still holds in $\alpha$–Rényi divergence.

###  3.2 Contraction of mean-field variational approximation

Let us now consider taking a variational approach, and instead of sampling
from the tempered posterior distribution $\Pi_{\alpha}[\cdot\,|\,X,Y]$, sample
from the variational approximation $\hat{Q}_{\alpha}$ defined in (8), where
the variational class is mean-field with heavy-tailed components.

###### Theorem 5

Consider data from the nonparametric random design regression model (1), with
$f_{0}\in\mathcal{G}(q,\mathbf{d},\mathbf{t},\boldsymbol{\beta},K)$ and
arbitrary unknown parameters. Let $\Pi$ be a heavy-tailed DNN prior as in
Theorem 2, with heavy-tailed density $h$ such that $m_{2\vee(1+\kappa)}(h)\leq
c_{3}$ for some $c_{3}>0$ and $\kappa$ as in (H2). Suppose the scaling factors
$\sigma_{k}$ verify (14).

Let $\mathcal{S}=\mathcal{S}_{HT}(h)$ be the mean-field class generated from
$h$ as in Definition 1 and let $\hat{Q}_{\alpha}$ be the associated
variational tempered posterior as in (8).

Then for any $\alpha\in(0,1)$, for $\phi_{n}$ as in (18), for $M$ large
enough, as $n\to\infty,$

$E_{f_{0}}\int D_{\alpha}(f,f_{0})d\hat{Q}_{\alpha}(f)\leq M\phi_{n}^{2}.$

In particular, for any diverging sequence $M_{n}\to\infty$

$E_{f_{0}}\hat{Q}_{\alpha}\left[\left\\{f\,:\,D_{\alpha}(f,f_{0})\geq
M_{n}\phi_{n}^{2}\right\\}\right]\to 0.$

Theorem 5 establishes contraction of the heavy-tailed mean-field variational
approximation with the same adaptive rate as in Theorem 2. The definition (1)
of the variational class is independent of any unknown quantities linked to
$f_{0}$ such as $t$ and $\boldsymbol{\beta}$, has no hierarchical structure
and only requires two parameters $(m,\varsigma)$, avoiding the choice of a
sparsity level hyperparameter required in model selection priors such as spike
and slab priors (as in e.g. Chérief-Abdellatif (2020); Bai et al. (2020)). The
moment restriction on $h$ allows for a result in expectation at the level of
the variational posterior and guarantees consistency of the variational
posterior mean (see also Remark 17 below), restricting however the choice of
the prior to fixed decay as in (14). Up to somewhat more technical proofs, we
believe that one could also weaken the conditions to derive a result in
probability as above but without moment condition on $h$, as well as to allow
for scalings verifying the more general decay (16). For simplicity here we do
not investigate such further refinements.

Similarly as for Theorem 2, one may deduce from Theorem 5 a convergence rate
result for the variational tempered posterior $\hat{Q}_{\alpha}$ in terms of
the $L^{2}(P_{X})$ loss by post-processing $\hat{Q}_{\alpha}$ through a simple
‘clipping’ operation.

###  3.3 Geometric data

Rather than making assumptions about the structure of the unknown regression
function $f_{0}$ (e.g. in terms of compositions as in Sections 3.1 and 3.2),
we can instead permit the observed data itself to exhibit a low-dimensional
structure.

For instance, one can assume that the distribution $P_{X}$ of the design
points $(X_{i})$ is supported on a smooth manifold of lower dimension than
$d$. This notion is frequently employed to describe intrinsic dimensionality
and various results are available to approximate functions on such manifolds
using deep neural networks, see for example Chen et al. (2019), Schmidt-Hieber
(2019). Here we follow the more general setting of Nakada and Imaizumi (2020),
which includes design points sitting on a manifold as a special case while
also allowing for rougher sets (such as fractals, although for the discussion
below we focus on a manifold example for simplicity) via the following notion
of dimension.

######  Definition 6 (Minkowski Dimension)

Let $E\subset[0,1]^{d}$, the $(\varepsilon,\infty)$-covering number of $E$,
written $\mathcal{N}(E,\varepsilon)$ is the minimal number of
$\ell_{\infty}$-balls of radius $\varepsilon>0$ necessary to cover $E$. The
upper Minkowski dimension of $E$ is defined as

$\dim_{M}E:=\inf\\{t\geq 0\,:\,\underset{\varepsilon\downarrow
0}{\lim\,\sup}\,\mathcal{N}(E,\varepsilon)\varepsilon^{t}=0\\}.$

The Minkowski dimension gives insight on how the covering number of $E$
changes compared to the decrease of the radius of the covering balls.

###### Assumption 1

Suppose that $t^{*}:=\dim_{M}\,\operatorname{Supp}P_{X}<d$.

Assumption 1 is satisfied in the case that $\operatorname{Supp}P_{X}$ is a
smooth compact manifold of dimension $d^{*}$ strictly less than $d$, as then
$t^{*}\leq d^{*}<d$, as established in Lemma 9 in Nakada and Imaizumi (2020).

We show below that, under Assumption 1, our construction of heavy-tailed
priors can be adapted to lead to hierarchical-free adaptation to the Minkowski
dimension of the support of the design points $(X_{i})$. Let us set, for
$\kappa$ the constant appearing in (H2) and $t>0$,

$\varepsilon_{n}(t):=\left(n/\log^{2(1+\kappa)+1}n\right)^{-\beta/(2\beta+t)}.$
(19)

###### Theorem 7

Consider data from the nonparametric random design regression model (1), with
$f_{0}\in\mathcal{C}_{d}^{\beta}([0,1]^{d})$. Suppose Assumption 1 holds and
let $\Pi$ be a heavy-tailed DNN prior as described in Section 2.3 with

  * •

architecture $(L,\mathbf{r})$ with $L=\lceil\log n\rceil$ and
$\mathbf{r}=\mathbf{r}(n)$ as in (13);

  * •

network weights $\theta_{k}\sim\sigma_{k}\zeta_{k}$ independently as in (7),
with

    * –

scaling factors $\sigma_{k}$ verifying (14);

    * –

random $\zeta_{k}$ with heavy-tailed density $h$ satisfying (H1)–(H3).

Let $\varepsilon_{n}(t)$ as in (19), for a given arbitrary $t>t^{*}$. Then for
any $\alpha\in(0,1)$, for $M$ large enough, as $n\to\infty,$

$E_{f_{0}}\Pi_{\alpha}\left[\left\\{f\,:\,D_{\alpha}(f,f_{0})\geq
M\varepsilon_{n}(t)\right\\}\,|\,X,Y\right]\to 0.$

Assume further that $m_{2\vee(1+\kappa)}(h)<\infty$ and let $\hat{Q}_{\alpha}$
be the heavy-tailed tempered variational approximation defined by (8) on
$\mathcal{S}_{HT}(h)$. Then the contraction of $\hat{Q}_{\alpha}$ as stated in
Theorem 5 holds here with $\phi_{n}$ replaced by $\varepsilon_{n}$.

As a special case, if $\operatorname{Supp}P_{X}$ is a smooth compact manifold
of dimension $d^{*}$ strictly less than $d$, Theorem 7 gives a contraction
rate $\varepsilon_{n}(t)$ in particular for any $t>d^{*}$, that is a rate
$\varepsilon_{n}(d^{*}+\eta)$ for any fixed arbitrarily small $\eta>0$, which
is near-optimal (the optimal minimax rate being $\varepsilon_{n}(d^{*})$). The
proof of Theorem 7 in given in Appendix C.3.

In Theorem 7, we use the choice $\mathbf{r}=\mathbf{r}(n)$ as in (13), that
is, we increase the network width quadratically compared to (12). Indeed, in
order to establish the above posterior concentration, we rely on an
approximation result of functions in such a setting by ReLU-deep neural
networks by Nakada and Imaizumi (2020). Such a result, recalled as Lemma 19
below, uses sparse approximations (i.e. many network weights are equal to
zero), in a similar spirit as the corresponding result in Schmidt-Hieber
(2020), which has the effect to increase (quadratically) the width of the
network compared to Theorem 2 and Theorem 5, for which we used the recent
fully-connected approximation results of Kohler and Langer (2021) (as opposed
to Schmidt-Hieber (2020), Theorem 5). It should also be possible to use fully-
connected approximation results in the present setting, for example in the
spirit of Theorem 6.3 of Jiao et al. (2023) (which, similar to Kohler and
Langer (2021), has at most polynomially increasing weights)

Also, here we have considered only the case of constant scaling factors (14).
The same remarks as in the discussion below Theorem 5 apply: the statement can
presumably be adapted to directed decay of the network weights up to using a
slightly more technical proof.

###  3.4 Anisotropic Besov classes

Instead of assuming the true function to be in a Hölder space as in Theorem 2,
we now investigate the case that the smoothness of the true function is both
depending on the location (non-homogeneous smoothness) and the direction
(anisotropic). For simplicity of presentation we focus on the setting without
additional compositional structure, but one could also derive results in case
of compositions as in Section 3.1; the case of a single function is already
illustrative of the adaptation properties at stake.

For a function $f:\mathbb{R}^{d}\to\mathbb{R}$, the $m$-th difference of $f$
in the direction $v\in R^{d}$ is

$\Delta_{v}^{m}(f)(x):=\Delta_{v}^{m-1}(f)(x+v)-\Delta_{v}^{m-1}(f)(x),\qquad\Delta_{v}^{0}(f)(x):=f(x).$

###### Definition 8

For a function $f\in L^{p}([0,1]^{d})$, where $p\in[1,\infty)$, the $m$-th
modulus of smoothness of $f$ is defined by

$\omega_{m,p}(f,u):=\sup_{v\in\mathbb{R}^{d}\,:\,|v_{i}|\leq
u_{i}}\lVert\Delta_{v}^{m}(f)\rVert_{p},\qquad\text{for
}u=(u_{1},\dots,u_{d}),\ \ u_{i}>0.$

######  Definition 9 (Anisotropic Besov space)

Let $p\in[1,\infty)$,
$\beta=(\beta_{1},\dots,\beta_{d})\in\mathbb{R}_{>0}^{d}$ and
$m:=\max_{i}\lfloor\beta_{i}\rfloor+1$. Let the seminorm
$|\,\cdot\,|_{B_{pp}^{\beta}}$ be defined as

$|f|_{B_{pp}^{\beta}}^{p}:=\sum_{k=0}^{\infty}\left[2^{k}w_{m,p}\left(f,(2^{-k/\beta_{1}},\dots,2^{-k/\beta_{d}})\right)\right]^{p}.$

For $\lVert f\rVert_{B_{pp}^{\beta}}=\lVert f\rVert_{p}+|f|_{B_{pp}^{\beta}}$
and $M>0$, the anisotropic Besov ball of radius $M$ on $[0,1]^{d}$ is

$B_{pp}^{\beta}(M):=\left\\{f\in L^{p}([0,1]^{d})\,:\,\lVert
f\rVert_{B_{p,p}^{\beta}}\leq M\right\\}.$

Optimal minimax rates on such anisotropic Besov balls can be expressed in
terms of the harmonic mean smoothness, defined as

$\tilde{\beta}^{-1}:=\sum_{k=1}^{d}\beta_{k}^{-1}.$ (20)

Let us set, for $\gamma$ the constant given by (17) and $\tilde{\beta}$
defined by (20),

$\epsilon_{n}=\left(n/\log^{\gamma}n\right)^{-\tilde{\beta}/(2\tilde{\beta}+1)}.$
(21)

###### Theorem 10

Consider data from the nonparametric random design regression model (1), with
$f_{0}\in B_{pp}^{\beta}(1)$ and
$\beta=(\beta_{1},\dots,\beta_{d})\in\mathbb{R}_{>0}^{d}$. Assume $p<2$ and
$\tilde{\beta}>1/p$, and let $\Pi$ be a heavy-tailed DNN prior as described in
Section 2.3 with

  * •

architecture $(L,\mathbf{r})$ with $L$ as in (11) and
$\mathbf{r}=\mathbf{r}(n)$ as in (13);

  * •

network weights $\theta_{k}\sim\sigma_{k}\zeta_{k}$ independently as in (7),
with

    * –

scaling factors $\sigma_{k}$ verifying (14);

    * –

random $\zeta_{k}$ with heavy-tailed density $h$ satisfying (H1)–(H3).

Then for any $\alpha\in(0,1)$, for $M>0$ large enough it holds, as
$n\to\infty,$ for $\epsilon_{n}$ as in (21)

$E_{f_{0}}\Pi_{\alpha}\left[\left\\{f\,:\,D_{\alpha}(f,f_{0})\geq
M\epsilon_{n}\right\\}\,|\,X,Y\right]\to 0.$

Assume further that $m_{2\vee(1+\kappa)}(h)<\infty$ and let $\hat{Q}_{\alpha}$
be the heavy-tailed tempered variational approximation defined by (8) on
$\mathcal{S}_{HT}(h)$. Then the contraction of $\hat{Q}_{\alpha}$ as stated in
Theorem 5 holds here with $\phi_{n}$ replaced by $\epsilon_{n}$.

Whenever $p<2$ and $\tilde{\beta}>1/p$, functions in $B_{pp}^{\beta}$ are
continuous and in-homogeneously smooth in terms of the $L^{2}$–norm (see for
instance Suzuki and Nitanda (2021) and Donoho and Johnstone (1998)). An
interesting feature of Theorem 10 is the adaptation to the anisotropic
smoothness it provides. Anisotropy itself may help mitigate the curse of
dimensionality. Indeed, when $\beta=(\beta_{0},\dots,\beta_{0})$, we have
$\tilde{\beta}=\beta_{0}/d$, and one obtains the usual rate for a
homogeneously smooth function in $\mathbb{R}^{d}$. However, when $f_{0}$ is
not very smooth only in a small number of directions, the estimation rate
$n^{-\tilde{\beta}/(2\tilde{\beta}+1)}$ depends only on this specific small
number of coordinates rather than on the whole ambient dimension $d$.

The proof of Theorem 10 can be found in appendix C.4. Similar remarks as for
Theorem 7 can be made. In particular, the proof uses the sparse approximation
of Suzuki and Nitanda (2021) recalled as Lemma 20 below, which again explains
why it is natural to use the width $\mathbf{r}=\mathbf{r}(n)$ as in (13).

###  3.5 Influence of the network architecture

In the next lines, we discuss possible choices of the architecture, including
and also different from the two we propose in (12) and (13). We restrict the
discussion to networks with logarithmic depth; polynomial depth would be
interesting too although beyond the scope of this contribution. For simplicity
we discuss the simplest setting of estimating a $\beta$–Hölder function, for
which the minimax rate in squared integrated norm is $n^{-2\beta/(2\beta+d)}$.
Having fixed the depth to a slowly growing logarithmic term as in (11), we
distinguish two classes of choices for the network width: those that are
matched or underfit and those that overfit.

Case of matched or underfitting architecture. Suppose one takes our prior
construction with a width equal to $n^{a}$ for some $a<1/2$, so that the
corresponding architecture is, say $(L,\mathbf{r}(a))$. Then the overall
number of non-zero weights in the network is of order $n^{2a}$. An application
of Lemma 1 in Schmidt-Hieber (2020) then shows that the following
approximation rate verifies

$\underset{f_{0}\in\mathcal{C}_{d}^{\beta}([0,1]^{d})}{\sup}\underset{f\in\mathcal{F}(L,\mathbf{r}(a))}{\inf}||f-f_{0}||_{\infty}\gtrsim\eta_{n},$

with $\eta_{n}\approx n^{-2a\beta/d}$, up to a logarithmic factor. From this
one easily deduces that the tempered posterior (or more generally any method
based on the architecture $(L,\mathbf{r}(a))$) cannot converge (in sup-norm)
faster than $n^{-2a\beta/d}$, which is slower than the optimal minimax rate
$n^{-2\beta/(2\beta+d)}$ if $2\beta<d((2a)^{-1}-1)$.

Case of overfitting architecture. On the other hand, as already noted above,
our prior construction with architecture $(L,\mathbf{r}(a))$, that is with
width $n^{a}$, and $a\geq 1/2$ gets the optimal rate with full adaptation to
smoothness and structure, as shown by Theorem 2 and the following results in
the other settings above.

##  4 Discussion

A main key take-away from this work is that putting suitable heavy-tailed
distribution on weights of deep neural networks enable automatic and
simultaneous adaptation to intrinsic dimension and smoothness. This prior
distribution allows for a soft selection of relevant weights in the network,
that get higher values under the posterior, while less important weights can
get very small values. The approach does not use a ‘hard’ variable selection
approach, which would either set some weights to zero, and hence require the
need to sample from the posterior distribution on the support points, or
attempt to sample the network architecture at random, which would require to
sample from the posterior on $(L,\mathbf{r})$, thus requiring in both cases to
sample from ‘support’ hyperparameters, which can be costly computationally.

Another key idea is that the approach allows for a wide variety of choice of
overfitting architectures. We have seen that any choice of a common width for
the layers that is polynomial in $n$ with a power at least $1/2$, that it
$r_{i}\asymp n^{a}$ with $a>1/2$ is compatible with optimal rates with our
method. We note that we have not investigated here the case of very deep
networks with polynomial depth, but the idea could apply there too. We also
allow for a fairly wide variety of scalings $\sigma_{k}$ of the individual
weights, showing that the optimality of the rates is robust to different
choices of scalings.

Priors for Bayesian deep neural networks. For networks of large width
(possibly increasing with the sample size, but finite) and moderate depth, a
number of natural prior distributions have been proposed in recent works.

Let us first discuss some differences of our approach with one other natural
prior, namely spike-and-slab (SAS) priors on weights, as in Polson and Ročková
(2018). One difference is that here we use the recent ‘dense’ approximation
results of Kohler and Langer (2021) that enable us to be more parcimonious (if
this is desired) in terms of the total number of parameters that should be
sampled in the posterior. Updating the proofs from Polson and Ročková (2018),
one could also deploy SAS priors with denser architecture but SAS priors still
have more (hyper-)parameters overall, with the probability of being nonzero to
be tuned as well as the slab parameter (the same remark applies for their
variational counterparts). Contrary to SAS, our heavy-tailed priors enable
both sampling from the full posterior with MCMC methods (see the paragraph on
simulations below) as well as the use of mean-field variational approximation.

Two recent natural constructions are the priors considered in Lee and Lee
(2022) and the ones in Kong and Kim (2024). The work Kong and Kim (2024)
adapts, as we do, the approximation theory of Kohler and Langer (2021) to
allow for dense (non-sparse) approximations, thus avoiding the need for
forcing some weights to be zero – as a side note, let us mention that more or
less simultaneously the need for a control on the amplitude of the weights for
use in Bayesian arguments has been recognised by several authors that have
adapted Theorem 2 in Kohler and Langer (2021) to include quantitative bounds
on network coefficients; see for example Theorem K.1 in Ohn and Lin (2024),
and Theorem 1 in Kong and Kim (2024) and Appendix D for our own construction
–. However, an important difference of our work with the approach of Kong and
Kim (2024) is that the latter requires to sample from the width of the network
to achieve adaptation; as noted above, such hyper-parameter sampling can be
costly for simulations, as it requires access to a good approximation of the
posterior on the width (similar to the calibration of the cut-off of sieve
priors in nonparametrics, see Agapiou and Castillo (2024) for more discussion
on this); this hyper-parameter sampling is not required here, as we use a
deterministic ‘overfitting’ architecture.

The work Lee and Lee (2022) allows for heavy-tailed priors, such as mixture of
Gaussians, although crucially the conditions of Theorem 3 therein are non-
adaptive (i.e. the priors’ parameters depend on the unknown smoothness of the
regression function). A main insight of the present work, following the idea
recently introduced in Agapiou and Castillo (2024) in nonparametric settings,
is that by taking an (overfitting) width, heavy tails on weights make the
(tempered) posterior automatically adapt to unknown structural parameters, let
it be smoothness, intrinsic dimension(s) or compositional structures. The
simulation aspect of the work Lee and Lee (2022) is also relevant here, since
therein the authors develop and run MCMC algorithms that enable to sample from
the corresponding neural network posteriors.

Finally, let us mention the work by Ghosh et al. (2019), who provide
algorithms to sample from variational approximations of posteriors
corresponding to horseshoe priors on weight parameters. Although the authors
provide no theoretical back-up for their empirical results, this is related in
spirit to our approach; in fact, although there are some notable differences
in the prior’s choice (e.g. the horseshoe has a pole at zero and has a tuning
sparsity parameter) we think that our approach and proofs could presumably be
adapted in order to provide theoretical understanding and validation of their
method; this idea is also supported by current work in progress Agapiou et al.
(2024) where we derive theory for horseshoe priors for nonparametrics.

Simulations. Although an in-depth simulation study is beyond the scope of the
present paper, we note that algorithms are readily available to sample from
the heavy-tailed tempered posteriors introduced here. As mentioned above,
since there are no structure hyper-parameters to sample from (such as the
network architecture or the position of the non-zero coefficients such as for
SAS priors), one can use an MCMC algorithm such as the one from Lee and Lee
(2022). Another possibility is the use a variational Bayes algorithm for heavy
tailed priors, such as the one considered in Ghosh et al. (2019) for the
horseshoe prior. This is left for future work. We note that having both
feasible MCMC and variational algorithms makes it an interesting setting.
Indeed, it is known that mean-field VB, not being as complex as the posterior,
may distorts certain aspects of the latter, such as posteriors for finite-
dimensional functionals. On the other hand, unlike VB, the full posterior on
network will retain correlations, which are believed to be particularly
important for deep neural networks. It will be interesting to compare the
behaviour of both algorithms through an extended simulation study; this will
be considered elsewhere.

Another relevant aspect of the approach in practice is the possibility to make
different choices for scaling sequences $(\sigma_{k})$. For simplicity we have
mostly focused on the constant choice as in (14). Other choices may be
interesting, in particular for small sample sizes, for instance taking
scalings matching the lower bound in (16). Indeed, since one expects a number
of coefficients to be of significant amplitude, it may help to have at least a
few scalings of the order of a constant, as is the case for the latter choice.
This may accelerate, in small or moderate samples, the ‘soft selection’ of the
weights of highest amplitude in the network. Again we leave this for future
work.

Further work and open questions. Let us mention a few further natural
questions arising from the present results beyond the just-mentioned aspects
on simulations. One is that our results are concerned with tempered
posteriors, that is, we rise the likelihood to a power $\alpha$ strictly
smaller than $1$. Currently, the available techniques in Bayesian
nonparametrics do not seem to allow for simple results to standard posteriors
($\alpha=1$ in (6)) for heavy-tailed priors; a reason is that for these priors
it is not clear how to construct sieve sets required to apply the theory of
Ghosal et al. (2000). Nevertheless, in a (simpler) white noise regression
context, Agapiou and Castillo (2024) derive contraction of the standard
posterior for heavy tailed priors and their simulations suggest that there is
no visible phase transition when $\alpha$ increases from e.g. $1/2$ up until
$1$ (included); we conjecture that the theoretical results presented here
still go through for standard posteriors ($\alpha=1$), and refer to Agapiou
and Castillo (2024) for more discussion on this.

Also, as mentioned above, another promising direction is to derive results for
horseshoe priors, for instance in the spirit of Ghosh et al. (2019), as well
as to compare with the present heavy-tailed priors. We are currently
investigating this in the simpler setting of white noise regression with
series priors in Agapiou et al. (2024).

Finally, it would be interesting to understand more the effect of overfitting
with our approach: as we have noted, we can choose a polynomial width with
large power (e.g. $n^{3}$) for the network; also, for the specific choice of
constant $\sigma_{k}$ the prior is exchangeable and does not particularly
penalise a high number of large coefficients; one may then think that this
setting may be particularly appropriate to test for the presence of a double
descent phenomenon, which features a decrease of the risk for very overfitted
models. It is conceivable that for very large widths one sees this appear in
the present context: this deserves further investigation.

##  5 Proofs of the main results

In this section we give the proofs of the main results for the compositional
models.

###  5.1 Fully connected ReLU DNN approximation

We recall here the approximation result we will use to study posterior
contraction towards compositional functions as in Theorems 2 and 5.

Proofs are done in appendix C. For Proposition 11 we follow the same approach
as Schmidt-Hieber (2020, Theorem 1), simply using the fully-connected
architecture from Kohler and Langer (2021) for the elementary building blocks,
as it is better suited to our prior. Note that Kohler and Langer (2021) does
not give a explicit bound on the magnitude of the coefficients in the
approximating neural network $\tilde{f}$. Such a bound is fundamental in the
study of the properties of our prior and is provided in appendix D.

###### Proposition 11

Let $f\in\mathcal{G}(q,\mathbf{d},\mathbf{t},\boldsymbol{\beta},K)$, $L$ as in
(11) and $\phi_{n}$ as in (18). There exists a ReLU network

$\tilde{f}\in\mathcal{F}(L,\mathbf{r}(\sqrt{n}),s)$

with

$s\lesssim n\phi_{n}^{2}\log^{1-\gamma}n$

such that for sufficiently large $n$,

$||f-\tilde{f}||_{\infty}\lesssim\phi_{n}.$

Additionally, all coefficients of $\tilde{f}$ satisfy
$|\tilde{\theta}_{k}|\leq n^{c_{\beta}}$, where $c_{\beta}\geq 1$ is a
constant only depending on $(\beta_{i})_{i}$.

###### Remark 12

We can be more precise in Proposition 11 and construct the network $\tilde{f}$
in such a way that the position of the active coefficients is known. From the
proof given in Appendix C.2 it follows that there is an integer

$r^{*}\asymp\underset{0\leq i\leq
q}{\max}\left(\frac{n}{\log^{\gamma}n}\right)^{\frac{1}{2}\frac{t_{i}}{2\beta_{i}^{*}+t_{i}}},$

such that if $(\tilde{W}_{l},\tilde{v}_{l})$ are the coefficients of
$\tilde{f}$, we have for all $l\in[L+1]$, $\tilde{W}_{l}^{(ij)}=0$ whenever
$i\vee j>r^{*}$ and $\tilde{v}_{k}^{(i)}=0$ whenever $i>r^{*}$.

###  5.2 Proof of Theorem 2

Proof  In view of Lemma 15, it suffices to show that there exists $C>0$ such
that for sufficiently large $n$,

$\Pi\left(\lVert f-f_{0}\rVert_{L^{2}(P_{X})}\leq\phi_{n}\right)\geq
e^{-Cn\phi_{n}^{2}}.$

Applying Proposition 11, we obtain
$\tilde{f_{0}}\in\mathcal{F}(L,\mathbf{r}(\sqrt{n}),s)$ a ReLU neural network
that approximates $f_{0}$ for large enough $n$ with $\lVert
f_{0}-\tilde{f_{0}}\rVert_{\infty}\leq\phi_{n}/2$ and $s\lesssim
n\phi_{n}^{2}\log^{1-\gamma}n$. Using the triangle inequality,

$\Pi\left(\lVert
f-f_{0}\rVert_{L^{2}(P_{X})}\leq\phi_{n}\right)\geq\Pi\left(\lVert
f-\tilde{f_{0}}\rVert_{L^{2}(P_{X})}\leq\phi_{n}/2\right)\geq\Pi\left(\lVert
f-\tilde{f_{0}}\rVert_{\infty}\leq\phi_{n}/2\right).$

It suffice to control the difference in the supremum norm between two networks
of the same structure. For this purpose we use Lemma 18. Let $\theta_{k}$
(resp. $\tilde{\theta}_{k}$) be the coefficients of $f$ (resp.
$\tilde{f}_{0})$. We know from Proposition 11 that
$\sup_{k}|\tilde{\theta}_{k}|\leq n^{c_{\beta}}$ where $c_{\beta}\geq 1$.
Recall that $V=\prod_{l=0}^{L}(r_{l}+1)$ thus using lemma 18 and independence,

$\Pi\left(\lVert
f-\tilde{f_{0}}\rVert_{\infty}\leq\phi_{n}/{2}\right)\geq\prod_{k=1}^{T}\Pi\left(|\tilde{\theta}_{k}-\theta_{k}|\leq\frac{\phi_{n}}{2n^{c_{\beta}L}V(L+1)}\,,\,|\theta_{k}|\leq
n^{c_{\beta}}\right).$ (22)

From remark 12 there is an integer

$r^{*}\asymp\underset{0\leq i\leq
q}{\max}\left(\frac{n}{\log^{\gamma}n}\right)^{\frac{1}{2}\frac{t_{i}}{2\beta_{i}^{*}+t_{i}}}$
(23)

such that, for all $l\in[L+1]$, $\tilde{W}_{l}^{(ij)}=0$ whenever $i\vee
j>r^{*}$. Let

$z_{n}:=\phi_{n}/(2n^{c_{\beta}L}V(L+1)),$ (24)

note that $z_{n}\to 0$ when $n\to\infty$, so that $0<z_{n}<1$ when $n$ is
large enough.

We can then split the right hand side in (22) and first take care of the
possibly large coefficients (but still bounded by $n^{c_{\beta}})$ using the
heavy tails properties of the prior. To do so we use Lemma 21 with $z_{n}$, on
every coordinate such that $i\vee j\leq r^{*}$. We get a large enough constant
$C_{0}>0$ such that when $n$ is sufficiently large,

$\displaystyle\prod_{l=1}^{L+1}\prod_{i\vee j\leq r^{*}}\Pi$
$\displaystyle\left(|W_{l}^{(ij)}-\tilde{W}_{l}^{(ij)}|\leq
z_{n}\,,\,|W_{l}^{(ij)}|\leq n^{c_{\beta}}\right)$
$\displaystyle\geq\prod_{l=1}^{L+1}\prod_{i\vee j\leq
r^{*}}\underbrace{z_{n}\exp\left({-C_{0}\log^{1+\kappa}\left((1+n^{c_{\beta}})({\sigma_{l}^{(ij)}})^{-1}\right)}\right)}_{:=A}.$

Recall that $V=\prod_{l=0}^{L}(r_{l}+1)=(d+1)(\lceil\sqrt{n}\rceil+1)^{L}$ and
$L=\lceil\log^{1+\delta}n\rceil$, thus there is $C^{\prime}$ a large enough
constant such that, when $n$ is sufficiently large,

$2n^{c_{\beta}L}(L+1)V\leq e^{C^{\prime}\log^{2+\delta}n}.$ (25)

Using $\log\left(({\sigma_{l}^{(ij)}})^{-1}\right)\leq\log^{2(1+\delta)}n$
from (16), this leads to the bound, for $C_{3}$ a large enough constant

$A\geq\phi_{n}e^{-C^{\prime}\log^{2+\delta}n-C_{0}\log^{1+\kappa}\left((1+n^{c_{\beta}})({\sigma_{l}^{(ij)}})^{-1}\right)}\geq\phi_{n}e^{-C_{3}\log^{2(1+\delta)(1+\kappa)}n}.$
(26)

Now since $\phi_{n}\asymp(n/\log^{\gamma}n)^{-\nu}$ for some $\nu>0$, there
exists a constant $C_{4}>0$ such that

$\prod_{l=1}^{L+1}\prod_{i\vee j\leq r^{*}}A\geq
e^{-C_{4}(r^{*})^{2}L\log^{\gamma-1}n}.$

Finally since $(r^{*})^{2}L\leq s\lesssim n\phi_{n}^{2}\log^{1-\gamma}n$, we
get for the indices such that $i\vee j\leq r^{*}$, a large enough constant
$C>0$ such that

$\prod_{l=1}^{L+1}\prod_{i\vee j\leq
r^{*}}\Pi\left(|W_{l}^{(ij)}-\tilde{W}_{l}^{(ij)}|\leq
z_{n}\,,\,|W_{l}^{(ij)}|\leq n^{c_{\beta}}\right)\geq e^{-Cn\phi_{n}^{2}}.$

Now we take care of the small coefficients using the decay $(\sigma_{k})$. For
the indices such that $i\vee j>r^{*}$, recall that we have
$\tilde{W}_{l}^{(ij)}=0$ for all $l$, leaving us to bound

$\prod_{l=1}^{L+1}\prod_{i\vee j>r^{*}}\Pi\left(|W_{l}^{(ij)}|\leq
z_{n}\right)=\prod_{l=1}^{L+1}\prod_{i\vee
j>r^{*}}\left[1-2\overline{H}\left(z_{n}/\sigma_{l}^{(ij)}\right)\right],$

here we have used the symmetry of $h$ and the fact that $z_{n}<n^{c_{\beta}}$
for $n$ sufficiently large. Using $\log^{2(1+\delta)}(i\vee
j)\leq\log(1/\sigma_{l}^{(ij)})$ from (16) and the expression of $r^{*}$ given
by (23), we have a large enough constant $C_{5}$ such that whenever $i\vee
j>r^{*}$, we have

$1/\sigma_{l}^{(ij)}\geq e^{\log^{2(1+\delta)}(i\vee j)}\geq
e^{\log^{2(1+\delta)}r^{*}}\geq e^{C_{5}\log^{2(1+\delta)}n}.$

Recall that by assumption (H3), $\overline{H}(x)\leq c_{2}/x$ for all $x\geq
1$. Noting that $\log(1-2x)\geq-4x$ whenever $x<1/4$ and using the bound (25)
on $V$, one gets that for sufficiently large $n$ satisfying

$z_{n}e^{C_{5}\log^{2(1+\delta)}n}\geq 1\vee 4c_{2},$

there is a constant $C_{6}$ such that

$1-2\overline{H}\left(z_{n}/\sigma_{l}^{(ij)}\right)\geq
1-2c_{2}z_{n}^{-1}e^{-C_{5}\log^{2(1+\delta)}n}\geq\exp(-C_{6}z_{n}^{-1}e^{-C_{5}\log^{2(1+\delta)}n}).$

Recall $T$ is the total number of parameters of the network, this leads to the
lower bound

$\prod_{l=1}^{L+1}\prod_{i\vee j>r^{*}}\Pi\left(|W_{l}^{(ij)}|\leq
z_{n}\right)\geq\exp(-C_{6}Tz_{n}^{-1}e^{-C_{5}\log^{2(1+\delta)}n}).$

Whenever $n$ is large enough and for our choice of architecture, we have
$T\lesssim n\log^{1+\delta}n$, using again the bound (25) on V one gets for
$n$ sufficiently large,

$Tn^{c_{\beta}L}V(L+1)\lesssim
n(\log^{(1+\delta)}n)\,e^{C^{\prime}\log^{2+\delta}n}\lesssim
n\phi_{n}^{3}e^{C_{5}\log^{2(1+\delta)}n}.$

Finally we have a large enough constant $C>0$ such that

$\prod_{l=1}^{L+1}\prod_{i\vee j>r^{*}}\Pi\left(|W_{l}^{(ij)}|\leq
z_{n}\right)\geq e^{-Cn\phi_{n}^{2}}.$

###  5.3 Proof of Theorem 5

Proof  We give the proof for $\kappa=0$, giving the fastest rate, the proof in
the general case being similar. In this setting $\gamma=2(1+\delta)+1$. Let
$\tilde{f}_{0}\in\mathcal{F}(L,\mathbf{r}(\sqrt{n}),s)$ be the network
approximating $f_{0}$ from Proposition 11. Using $\lVert
f-f_{0}\rVert_{L^{2}(P_{X})}^{2}\leq 2\lVert
f-\tilde{f}_{0}\rVert_{L^{2}(P_{X})}^{2}+2\lVert\tilde{f}_{0}-f_{0}\rVert_{L^{2}(P_{X})}^{2}$,
Lemma 16 gives

$\displaystyle E_{f_{0}}\left(\int
D_{\alpha}(f,f_{0})\,d\hat{Q}_{\alpha}(f)\right)$
$\displaystyle\leq\frac{1}{1-\alpha}\lVert\tilde{f}_{0}-f_{0}\rVert_{L^{2}(P_{X})}^{2}$
$\displaystyle+\underset{Q\in\mathcal{S}}{\inf}\left\\{\frac{\alpha}{1-\alpha}\int\lVert
f-\tilde{f}_{0}\rVert_{L^{2}(P_{X})}^{2}\,dQ(f)+\frac{\operatorname{KL}(Q,\Pi)}{n(1-\alpha)}\right\\}.$

Since $\lVert\tilde{f}_{0}-f_{0}\rVert_{L^{2}(P_{X})}^{2}\leq\phi_{n}^{2}$
when $n$ is large enough, it is enough to show that $\mathcal{S}_{HT}(h)$
contains a distribution $Q^{*}$ satisfying the extended prior mass condition
(see Remark 17),

$\int\lVert
f-\tilde{f}_{0}\rVert_{L^{2}(P_{X})}^{2}\,dQ^{*}(f)\leq\phi_{n}^{2}\qquad\text{and}\qquad\operatorname{KL}(Q^{*},\Pi)\leq
n\phi_{n}^{2}.$ (27)

For $h$ the heavy-tailed density taken as prior on coefficients, let us define
$Q^{*}:=\bigotimes_{k=1}^{T}Q_{k}^{*}\in\mathcal{S}_{HT}(h)$ by setting for
all $k\in[T],$

$\frac{dQ_{k}^{*}}{d\theta_{k}}(\theta_{k}):=\frac{1}{\sigma_{k}}h\left(\frac{\theta_{k}-\tilde{\theta}_{k}}{\sigma_{k}}\right)=h_{\tilde{\theta}_{k},\sigma_{k}}(\theta_{k}),$

where $\sigma_{k}=e^{-\log^{2(1+\delta)}n}$ and $\\{\tilde{\theta}_{k}\\}$ are
the coefficients of $\tilde{f}_{0}$, such that $Q_{k}^{*}\in\mathcal{H}(h)$.

Using Lemma 18 and setting $w_{n}:=n^{2c_{\beta}L}V^{2}(L+1)^{2}$, we get

$\displaystyle\int\lVert
f-\tilde{f}_{0}\rVert_{L^{2}(P_{X})}^{2}\,dQ^{*}(f)\leq\int\lVert
f-\tilde{f}_{0}\rVert_{\infty}^{2}\,dQ^{*}(f)\leq
w_{n}\int\underset{k}{\max}|\theta_{k}-\tilde{\theta}_{k}|^{2}\,dQ^{*}(f).$

Recall the definition (9) of the moments of $h$, from the definition of
$Q^{*}$ it follows,

$\int\underset{k}{\max}|\theta_{k}-\tilde{\theta}_{k}|^{2}\,dQ^{*}(f)\leq\sum_{k=1}^{T}\int|\theta_{k}-\tilde{\theta}_{k}|^{2}dQ_{k}^{*}(\theta_{k})=m_{2}(h)\sum_{k=1}^{T}\sigma_{k}^{2}=m_{2}(h)Te^{-2\log^{2(1+\delta)}n}.$

Using inequality (25), we get $w_{n}T\leq e^{C\log^{2+\delta}n}$ for $C>0$
sufficiently large. Since $m_{2}(h)\leq c_{3}^{2/(2\vee(1+\kappa))}$ by
assumption, we have $m_{2}(h)w_{n}T\sigma_{k}^{2}\leq\phi_{n}^{2}$ if $n$ is
large enough.

We just showed that $Q^{*}$ satisfies the first condition in (27), let us
check the second one. We can write $\Pi=\bigotimes_{k=1}^{T}\Pi_{k}$ where,

$\frac{d\Pi_{k}}{d\theta_{k}}(\theta_{k})=\frac{1}{\sigma_{k}}h\left(\frac{\theta_{k}}{\sigma_{k}}\right)=h_{0,\sigma_{k}}(\theta_{k}).$

By additivity using independence we have
$\operatorname{KL}(Q^{*},\Pi)=\sum_{k=1}^{T}\operatorname{KL}(Q_{k}^{*},\Pi_{k})$.
Let us write $S_{0}=\\{k\in[T]\,,\,\tilde{\theta}_{k}\neq 0\\}$ the support of
the collection of coefficients of $\tilde{f}_{0}$, since $Q_{k}^{*}=\Pi_{k}$
whenever $\tilde{\theta}_{k}=0$, we get

$\operatorname{KL}(Q^{*},\Pi)=\sum_{k\in
S_{0}}\operatorname{KL}(Q_{k}^{*},\Pi_{k}).$

Let $k\in S_{0}$. Using the fact that $h$ is bounded,

$\displaystyle\operatorname{KL}(Q_{k}^{*},\Pi_{k})$
$\displaystyle=\int\log\left(\frac{h\left(\frac{\theta-\tilde{\theta}_{k}}{\sigma_{k}}\right)}{h\left(\frac{\theta}{\sigma_{k}}\right)}\right)\frac{1}{\sigma_{k}}h\left(\frac{\theta-\tilde{\theta}_{k}}{\sigma_{k}}\right)\,d\theta$
$\displaystyle\leq\log||h||_{\infty}+\underbrace{\int\log\left(\frac{1}{h\left(\frac{\theta}{\sigma_{k}}\right)}\right)\frac{1}{\sigma_{k}}h\left(\frac{\theta-\tilde{\theta}_{k}}{\sigma_{k}}\right)\,d\theta}_{I_{k}}.$

Using the heavy-tailed assumption (H2) with $\kappa=0$ on $h$, for all
$\theta>0$, we get

$\log\left(\frac{1}{h(\theta\sigma_{k}^{-1})}\right)\leq
c_{1}(1+\log(1+\theta\sigma_{k}^{-1})).$

Splitting the integral $I_{k}$ in two and using the symmetry of $h$ leads to

$\displaystyle I_{k}$
$\displaystyle\leq\int_{0}^{+\infty}\log\left(\frac{1}{h(\theta\sigma_{k}^{-1})}\right)\frac{1}{\sigma_{k}}h\left(\frac{\theta-\tilde{\theta}_{k}}{\sigma_{k}}\right)\,d\theta+\int_{0}^{+\infty}\log\left(\frac{1}{h(\theta\sigma_{k}^{-1})}\right)\frac{1}{\sigma_{k}}h\left(\frac{\theta+\tilde{\theta}_{k}}{\sigma_{k}}\right)\,d\theta$
$\displaystyle\leq\underbrace{\int_{0}^{+\infty}c_{1}\left[1+\log(1+\theta\sigma_{k}^{-1})\right]\frac{1}{\sigma_{k}}h\left(\frac{\theta-\tilde{\theta}_{k}}{\sigma_{k}}\right)\,d\theta}_{I_{k}^{-}}+\underbrace{\int_{0}^{+\infty}c_{1}\left[1+\log(1+\theta\sigma_{k}^{-1})\right]\frac{1}{\sigma_{k}}h\left(\frac{\theta+\tilde{\theta}_{k}}{\sigma_{k}}\right)\,d\theta}_{I_{k}^{+}}.$

One can suppose $\tilde{\theta}_{k}>0$, given the symmetry in $I_{k}^{+}$ and
$I_{k}^{-}$. Considering first $I_{k}^{-}$, we have, by changing variables,

$I_{k}^{-}\leq
c_{1}+c_{1}\int_{-\tilde{\theta}_{k}/\sigma_{k}}^{+\infty}\log\left(1+\frac{\tilde{\theta}_{k}}{\sigma_{k}}+u\right)h\left(u\right)\,du.$

The integral in the last display is bounded as follows

$\displaystyle\int_{-\tilde{\theta}_{k}/\sigma_{k}}^{+\infty}\log\left(1+\frac{\tilde{\theta}_{k}}{\sigma_{k}}+u\right)h\left(u\right)\,du$
$\displaystyle\leq\log\left(1+2\frac{\tilde{\theta}_{k}}{\sigma_{k}}\right)+\int_{\tilde{\theta}_{k}/\sigma_{k}}^{+\infty}\log\left(1+2u\right)h\left(u\right)\,du$
$\displaystyle\leq\log\left(1+2\frac{\tilde{\theta}_{k}}{\sigma_{k}}\right)+2m_{1}(h).$

For $I_{k}^{+}$, we have, changing variables

$\displaystyle\int_{\tilde{\theta}_{k}/\sigma_{k}}^{+\infty}\log\left(1-\frac{\tilde{\theta}_{k}}{\sigma_{k}}+u\right)h\left(u\right)\,du$
$\displaystyle\leq\int_{\tilde{\theta}_{k}/\sigma_{k}}^{+\infty}\log\left(1+u\right)h\left(u\right)\,du\leq
m_{1}(h).$

In the case of $\kappa>0$ one gets $m_{1+\kappa}(h)$ instead of the first
order moment, this quantity is also bounded by assumption. Finally, we have
$I_{k}\lesssim 1+\log\left(1+2\frac{|\tilde{\theta}_{k}|}{\sigma_{k}}\right)$
for $n$ large enough. Using the fact that $|\tilde{\theta}_{k}|\leq
n^{c_{\beta}}$ we obtain

$\operatorname{KL}(Q_{k}^{*},\Pi_{k})\lesssim
1+\log\left(1+2\frac{|\tilde{\theta}_{k}|}{\sigma_{k}}\right)\lesssim\log^{2(1+\delta)}n=\log^{\gamma-1}n.$

Recalling that $|S_{0}|=s\lesssim n\phi_{n}^{2}\log^{1-\gamma}n$, we finally
get,

$\operatorname{KL}(Q^{*},\Pi)\lesssim s\log^{\gamma-1}n\lesssim
n\phi_{n}^{2},$

which concludes the proof.

Acknowledgments and Disclosure of Funding

The authors would like to thank Sergios Agapiou and Gabriel de la Harpe for
helpful discussions. IC acknowledges funding from the Institut Universitaire
de France and ANR grant project BACKUP ANR-23-CE40-0018-01.

##  Appendix A Tempered posterior contraction

In this appendix we recall some results regarding the concentration of
posterior distributions.

###### Definition 13

Let $\alpha\in(0,1)$ and $P,Q$ be two probability measures. The Kullback-
Liebler (KL) divergence between $P$ and $Q$ is defined by

$\operatorname{KL}(P,Q):=\int\log\left(\frac{dP}{dQ}\right)\,dP\quad\text{if
}P\ll Q,\text{ and }+\infty\text{ otherwise.}$

Let $\mu$ be any measure satisfying $P\ll\mu$ and $Q\ll\mu$. The
$\alpha$-Rényi divergence between $P$ and $Q$ is defined by

$D_{\alpha}(P,Q):=\frac{1}{\alpha-1}\int\left(\frac{dP}{d\mu}\right)^{\alpha}\left(\frac{dQ}{d\mu}\right)^{1-\alpha}\,d\mu.$

For an overview on properties of such divergences we refer to van Erven and
Harremoes (2014). In particular, if $P_{f_{0}}$ is the probability measure
such that $(X_{i},Y_{i})\overset{i.i.d.}{\sim}P_{f_{0}}$ in the regression
setting (1), using additivity of the Rényi divergence for independent
observations, we denote for any $f\in\mathcal{F}$,

$D_{\alpha}(f,f_{0}):=D_{\alpha}(P_{f},P_{f_{0}})=\frac{1}{n}D_{\alpha}(P_{f}^{\otimes
n},P_{f_{0}}^{\otimes n}).$ (28)

Simple calculations (see for example Lemma 2.7 from Ghosal and van der Vaart
(2017)) show that in the regression setting

$\operatorname{KL}(P_{f_{0}},P_{f})=\frac{1}{2}\lVert
f-f_{0}\rVert_{L^{2}(P_{X})}^{2}:=\frac{1}{2}\int(f-f_{0})^{2}dP_{X},$ (29)
$\int\left(\log\left(\frac{dP_{f_{0}}}{dP_{f}}\right)-\operatorname{KL}(P_{f_{0}},P_{f})\right)^{2}\,dP_{f_{0}}=||f-f_{0}||_{L^{2}(P_{X})}^{2},$
(30)

where $P_{X}$ is the common distribution of the $X_{i}$’s.

The concentration Theorems for tempered posteriors in Section 3 are obtained
applying Lemmas 15 and 16 below, therefore the rates are formulated in terms
of Rényi divergences. In the random design Gaussian regression model (1), one
can relate such rates to classical $L^{2}(P_{X})$ ones, provided the true
function $f_{0}$ has a known upper bound $M_{0}$.

###### Lemma 14

Let $P_{f},P_{g}$ be probability measures of the regression model (1) and
assume $||f||_{\infty},||g||_{\infty}\leq M_{0}$, for any $\alpha\in(0,1)$, we
have

$D_{\alpha}(f,g)\geq\frac{\alpha}{2}e^{-2M_{0}^{2}\alpha(1-\alpha)}||f-g||^{2}_{L^{2}(P_{X})}$

Proof  Recall the definition of $D_{\alpha}(f,g)$ given by (28), simple
calculations lead to

$D_{\alpha}(f,g)=\frac{1}{\alpha-1}\log\int\exp\left(\frac{\alpha(\alpha-1)}{2}(f(x)-g(x))^{2}\right)\,dP_{X}(x).$

Using $1-x\leq-\log x$ and $1-e^{-x}\geq xe^{-x}$ both for $x\geq 0$, leads to

$\displaystyle D_{\alpha}(f,g)$
$\displaystyle\geq\frac{1}{1-\alpha}\left[1-\int\exp\left(\frac{\alpha(\alpha-1)}{2}(f-g)^{2}\right)\,{dP_{X}}\right]$
$\displaystyle\geq\frac{\alpha}{2}\int
e^{\alpha(\alpha-1)(f-g)^{2}/2}(f-g)^{2}\,dP_{X}$
$\displaystyle\geq\frac{\alpha}{2}e^{-2M_{0}^{2}\alpha(1-\alpha)}||f-g||^{2}_{L^{2}(P_{X})}.$

Tempered posterior concentration.

###### Lemma 15

Let $f_{0}\in\mathcal{F}$ and $\Pi$ be a probability measure on $\mathcal{F}$.
For any $\alpha\in(0,1)$, any positive sequence $(\varepsilon_{n})_{n}$ such
that $\varepsilon_{n}\to 0$, $n\varepsilon_{n}^{2}\to\infty$ and

$\Pi\left[\\{f\,:\,\lVert
f-f_{0}\rVert_{L^{2}(P_{X})}\leq\varepsilon_{n}\\}\right]\geq
e^{-n\varepsilon_{n}^{2}},$

there is a constant $C>0$ such that as $n$ tends to infinity,

$E_{{f_{0}}}\Pi_{\alpha}\left[\left\\{f\,:\,D_{\alpha}(f,f_{0})\geq
C\frac{\alpha\varepsilon_{n}^{2}}{1-\alpha}\right\\}\,|\,X,Y\right]\to 0,$

where $E_{f_{0}}$ denotes the expectation under $P_{f_{0}}$.

This is Theorem 4.1 in L’Huillier et al. (2023) written in our regression
setting, noting that the KL–neighborhood therein simplifies to an
$L^{2}(P_{X})$–neighborhood via (29) and (30).

Concentration of the variational approximation.

###### Lemma 16

Let $f_{0}\in\mathcal{F}$, $\Pi$ a probability measures on $\mathcal{F}$ and
$\mathcal{S}$ a set of probability measure on $\mathcal{F}$. For
$\alpha\in(0,1)$ let $\hat{Q}_{\alpha}$ be the tempered variational
approximation on $\mathcal{S}$ defined by (8). Then

$E_{f_{0}}\left(\int
D_{\alpha}(f,f_{0})\,d\hat{Q}_{\alpha}(f)\right)\leq\underset{Q\in\mathcal{S}}{\inf}\left\\{\frac{\alpha}{2(1-\alpha)}\int\lVert
f-f_{0}\rVert_{L^{2}(P_{X})}^{2}\,dQ(f)+\frac{\operatorname{KL}(Q,\Pi)}{n(1-\alpha)}\right\\}.$

This is Theorem 2.6 from Alquier and Ridgway (2020) written in our regression
framework, using (29).

###### Remark 17

If there is $\varepsilon_{n}>0$ such that

$E_{f_{0}}\left(\int
D_{\alpha}(f,f_{0})\,d\hat{Q}_{\alpha}(f)\right)\leq\varepsilon_{n}^{2},$

then by Markov’s inequality, for any $M_{n}\to\infty$, as $n\to\infty$,

$E_{{f_{0}}}\hat{Q}_{\alpha}\left[\left\\{f\,:\,D_{\alpha}(f,f_{0})\geq
M_{n}\varepsilon_{n}^{2}\right\\}\right]\to 0.$

Moreover, if we consider the clipped posterior $\hat{Q}_{\alpha}^{B}$ as in
Corollary 3 we obtain convergence of the variational posterior mean in
$L^{2}(P_{X})$-distance, as then Lemma 14 gives

$E_{f_{0}}\big{\lVert}\int
f\,d\hat{Q}_{\alpha}^{B}(f)-f_{0}\big{\rVert}_{L^{2}(P_{X})}\leq
E_{f_{0}}\int||f-f_{0}||_{L^{2}(P_{X})}\,d\hat{Q}_{\alpha}^{B}(f)\lesssim\varepsilon_{n}.$

##  Appendix B Additional properties of ReLU DNNs

We list here some well-known properties of ReLU neural networks, which are
very useful for combining different networks:

  * •

_Width enlargement_ :
$\mathcal{F}(L,\mathbf{r},s)\subseteq\mathcal{F}(L,\mathbf{r^{\prime}},s^{\prime})$
whenever $\mathbf{r}\leq\mathbf{r^{\prime}}$ component-wise and $s\leq
s^{\prime}$.

  * •

_Composition_ : Given $f\in\mathcal{F}(L,\mathbf{r})$ and
$g\in\mathcal{F}(L^{\prime},\mathbf{r^{\prime}})$ with
$r_{L+1}=r_{0}^{\prime}$, we can define the network whose realisation $g\circ
f$ is in the class
$\mathcal{F}(L+L^{\prime},(\mathbf{r},r_{1}^{\prime},\dots,r_{L^{\prime}+1}^{\prime}))$.

  * •

_Depth synchronization_ : To synchronize the number of hidden layers between
two networks, we can add layers with the identity matrix as many times as
desired,

$\mathcal{F}(L,\mathbf{r},s)\subset\mathcal{F}(L+q,(\underbrace{r_{0},\dots,r_{0}}_{\text{$q$
times}},\mathbf{r}),s+q\,r_{0}).$

  * •

_Parallelization_ : Given $f\in\mathcal{F}(L,\mathbf{r})$ and
$g\in\mathcal{F}(L,\mathbf{r^{\prime}})$ with $r_{0}=r_{0}^{\prime}$, one can
simultaneously realizes $f$ and $g$ within a joint network whose realisation
$(f,g)$ is in the class
$\mathcal{F}(L,(r_{0},r_{1}+r_{1}^{\prime},\dots,r_{L+1}+r_{L+1}^{\prime})).$

Following is a very useful Lemma giving a bound on the distance between two
ReLU neural networks realizations in terms of the distance between their
coefficients. This classical inequality can be found the proof of Lemma 3 from
Suzuki .

###### Lemma 18

Let $f$ and $f^{*}\in\mathcal{F}(L,\mathbf{r})$ with total number of
parameters $T$ and coefficients $(\theta_{k})$ and $(\theta_{k}^{*})$. Suppose
that for all $k\in[T]$, $|\theta_{k}|\leq b$, $|\theta_{k}^{*}|\leq b$, and
$|\theta_{k}-\theta_{k}^{*}|\leq\delta$. Define

$V:=\prod_{l=0}^{L}\left(r_{l}+1\right),$ (31)

then

$||f-f^{*}||_{L^{\infty}([0,1]^{r_{0}})}\leq\delta V(b\vee 1)^{L}(L+1).$

For reader’s convenience we recall below the approximation results that will
be used to prove Theorems 7 and 10, these are respectively borrowed from
Nakada and Imaizumi (2020) and Suzuki and Nitanda (2021) where the constants
are made explicit.

######  Lemma 19 (Theorem 5 in Nakada and Imaizumi (2020))

Let $f\in\mathcal{C}_{d}^{\beta}([0,1]^{d},K)$ and assume that
$\dim_{M}\operatorname{Supp}P_{X}<t$ holds with $t<d$. Let $\varepsilon>0$,
there is a ReLU neural network $\tilde{f}$ with constant depth
$L=L(\beta,d,t)$ and sparsity $\tilde{s}\lesssim\varepsilon^{-t/\beta}$ such
that, when $\varepsilon$ is small enough,

$||f-\tilde{f}||_{L^{\infty}(P_{X})}\leq\varepsilon,$

moreover there is a constant $c=c(\beta,d,t)$ such that the weights
$\\{\tilde{\theta}_{k},k\in[T]\\}$ of $\tilde{f}$ all satisfy
$|\tilde{\theta}_{k}|\leq\varepsilon^{-c}$.

######  Lemma 20 (Proposition 2 in Suzuki and Nitanda (2021))

Let $p\in(1,2)$, $\beta\in\mathbb{R}_{>0}^{d}$ such that $\tilde{\beta}>1/p$
and $f\in B_{pp}^{\beta}(1)$. Let $N$ be a large enough positive integer,
there exists $\tilde{f}\in\mathcal{F}(L,(d,r,\dots,r,1),s)$ with
$L\lesssim\log N$, $r\lesssim N$ and $s\lesssim NL$ such that

$||f-\tilde{f}||_{\infty}\lesssim N^{-\tilde{\beta}},$

additionally all the weights in $\tilde{f}$ are bounded by a universal
constant.

##  Appendix C Additional proofs

###  C.1 A technical Lemma

###### Lemma 21

Let $0<\sigma,z\leq 1$, $0<|\tilde{\theta}|\leq B$ for some $B>1$ and
$\theta\sim\sigma\cdot\zeta$, where $\zeta$ is a distribution on $\mathbb{R}$
with heavy-tailed density $h$ satisfying properties (H1), (H2) and (H3).
Denoting by $P_{\theta}$ the induced distribution on $\theta$, there is a
large enough constant $C>0$ such that,

$P_{\theta}(|\theta-\tilde{\theta}|\leq z\,,\,|\theta|\leq B)\geq
z\exp(-C\log^{1+\kappa}((B+z)\sigma^{-1})).$

Proof  Write $B_{-}:=(-B)\vee(\tilde{\theta}-z)$ and
$B^{+}:=B\wedge(\tilde{\theta}+z)$ so that

$P_{\theta}(|\theta-\tilde{\theta}|\leq z\,,\,|\theta|\leq
B)=\int_{B_{-}}^{B_{+}}\frac{1}{\sigma}h\left(\frac{x}{\sigma}\right)\,dx.$

Using the fact that $|\tilde{\theta}|<B$ one gets, when $\tilde{\theta}>0,$

$B^{+}=B\wedge(\tilde{\theta}+z)\geq\tilde{\theta},\qquad
B_{-}=\tilde{\theta}-z$

and when $\tilde{\theta}<0,$

$B^{+}=\tilde{\theta}+z,\qquad
B_{-}=(-B)\vee(\tilde{\theta}-z)\leq\tilde{\theta}.$

Using the symmetry of $h$, one can reduce the problem to the case
$\tilde{\theta}>0$. Using the positivity of $h$ and the fact that $\sigma\leq
1$, one gets

$P_{\theta}(|\theta-\tilde{\theta}|\leq z\,,\,|\theta|\leq
B)\geq\int_{\theta-z}^{\theta}h\left(\frac{x}{\sigma}\right)\,dx.$

Using the monotonicity of $h$, we have

$\int_{\theta-z}^{\theta}h\left(\frac{x}{\sigma}\right)\,dx\geq z\times
h\left(\frac{|\tilde{\theta}|\vee|\tilde{\theta}-z|}{\sigma}\right)\geq
zh((B+z)\sigma^{-1}).$

Now simply use the fact that $(B+z)\sigma^{-1}>1$ and the heavy tail property
(H2) of $h$ to conclude.

###  C.2 Proof of Lemma 11

Proof  First express $f=g_{q}\circ\dots\circ g_{0}:[0,1]^{d}\to\mathbb{R}$ as
the composition of functions defined on hypercubes $[0,1]^{t_{i}}$. To do
this, for all $1\leq i\leq q-1$, define

$\displaystyle h_{0}:=\frac{g_{0}}{2K}+\frac{1}{2},\quad
h_{i}:=\frac{g_{i}(2K\cdot-K)}{2K}+\frac{1}{2},\quad h_{q}:=g_{q}(2K\cdot-K).$
(32)

Such that $f_{0}=h_{q}\circ\dots\circ h_{0}$ and for all $j$,

$h_{0j}\in\mathcal{C}_{t_{0}}^{\beta_{0}}([0,1]^{t_{0}},1),\quad
h_{ij}\in\mathcal{C}_{t_{i}}^{\beta_{i}}([0,1]^{t_{i}},(2K)^{\beta_{i}}),\quad
h_{qj}\in\mathcal{C}_{t_{q}}^{\beta_{q}}([0,1]^{t_{q}},K(2K)^{\beta_{q}}).$

We can now approximate each of the $h_{ij}$ by a network realisation using
Lemma 23, before that, let us provide a propagation result that links the
approximation quality of $h_{ij}$ to that of the total composition.

######  Lemma 22 (Lemma 3 from Schmidt-Hieber (2020))

Let $h_{i}=(h_{ij})_{j}$ be functions as in (32) with $K\geq 1$. There is a
constant $C=C(K,\boldsymbol{\beta})$ such that for any functions
$\tilde{h}_{i}:=(\tilde{h}_{ij})_{j}$ with
$\tilde{h}_{ij}:[0,1]^{t_{i}}\to[0,1]$, we have,

$\lVert h_{q}\circ\dots\circ
h_{0}-\tilde{h}_{q}\circ\dots\circ\tilde{h}_{q}\rVert_{L^{\infty}([0,1]^{d})}\leq
C\sum_{i=0}^{q}\lVert|h_{i}-\tilde{h}_{i}|_{\infty}\rVert_{L^{\infty}([0,1]^{d_{i}})}^{\prod_{k=i+1}^{q}(\beta_{k}\wedge
1)}.$

We can now apply Lemma 23 to each of the $h_{ij}$ separately, using at
$i\in\\{0,\dots,q\\}$ fixed,
$M=M_{i}:=\left\lceil\left(\frac{n}{\log^{\gamma}n}\right)^{\frac{1}{2(2\beta_{i}^{*}+t_{i})}}\right\rceil$.
We get for all $(i,j)$,
$\tilde{h}_{ij}\in\mathcal{F}(L_{i}^{\prime},(t_{i},r_{i}^{\prime},\dots
r_{i}^{\prime},1))$ such that

$\lVert h_{ij}-\tilde{h}_{ij}\rVert_{\infty}\lesssim M_{i}^{-2\beta_{i}},$
(33)

where $L_{i}^{\prime}:=C_{\ell}\log n$ and
$r_{i}^{\prime}:=C_{w}M_{i}^{t_{i}}$ with $C_{\ell}$ and $C_{w}$ two constants
given by lemma 23 only depending on $(\boldsymbol{\beta},\mathbf{t})$.

To apply lemma 22 we need to make sure each $\tilde{h}_{ij}$ with $i<q$ takes
values in $[0,1]$. This can be done through the two layer network
$(1-(1-x)_{+})_{+}$. Let $L_{i}=L_{i}^{\prime}+2$ and keep calling the new
networks $\tilde{h}_{ij}.$ Since the $h_{ij}$ takes values in $[0,1]$
inequality (33) still holds with the two extra layers.

Now computing the networks $\tilde{h}_{ij}$ in parallel lends
$\tilde{h}_{i}=(\tilde{h}_{ij})_{j\in[d_{i+1}]}$ in the class

$\mathcal{F}(L_{i},(d_{i},r_{i},\dots,r_{i},d_{i+1})),$

where $r_{i}:=d_{i+1}r_{i}^{\prime}$. Now from inequality (33) we immediately
get,

$\lVert|h_{i}-\tilde{h}_{i}|_{\infty}\rVert_{L^{\infty}([0,1]^{d_{i}})}^{\prod_{k=i+1}^{q}(\beta_{k}\wedge
1)}\lesssim M_{i}^{-2\beta_{i}^{*}}.$ (34)

Finally we compute the composite network
$\tilde{f}:=\tilde{h}_{q}\circ\dots\circ\tilde{h}_{1}$ in the class

$\mathcal{F}(L_{0}+\dots+L_{q},(d,r,\dots,r,1)),$ (35)

where $r:=\underset{0\leq i\leq q}{\max}r_{i}$. Using lemma 22 and (34), for
$n$ large enough, we get,

$||f-\tilde{f}||_{\infty}\lesssim\sum_{i=0}^{q}M_{i}^{-2\beta_{i}^{*}}\lesssim\max_{i}M_{i}^{-2\beta_{i}^{*}}\leq\phi_{n}.$

Now we have for sufficiently large $n$, $L_{0}+\dots+L_{q}=(q+1)C_{\ell}\log
n\leq\log^{1+\delta}n$ and $r\leq\lceil\sqrt{n}\rceil$. Thus, for $n$
sufficiently large the space (35) can be embedded into

$\mathcal{F}(\lceil\log^{1+\delta}n\rceil,(d,\lceil\sqrt{n}\rceil,\dots,\lceil\sqrt{n}\rceil,1),s),$

where

$s\lesssim r^{2}\log n\lesssim\underset{0\leq i\leq
q}{\max}\left(\frac{n}{\log^{\gamma}n}\right)^{\frac{t_{i}}{2\beta_{i}^{*}+t_{i}}}\times\log
n\lesssim n\phi_{n}^{2}\log^{1-\gamma}n.$

Now for the bound on the coefficients, in view of proposition 24, for
sufficiently large $n$ every coefficient $\theta$ of the network satisfies

$|\theta|\leq\underset{0\leq i\leq q}{\max}\lceil
M_{i}^{2(\beta_{i}+1)}\rceil^{2}\leq n^{c_{\beta}},$

where $c_{\beta}\geq 1$ is a constant only depending on $(\beta_{i})_{i}$.

This completes the proof.

###  C.3 Proofs for data with low Minkowski dimension support

Proof [Proof of Theorem 7]The proof is very similar to that of Theorem 2.
Since we will use a sparse approximation result the position of the zeros in
the network is not precisely known.

In view of Lemma 15 it suffices to show that there exists $C>0$ such that for
sufficiently large $n$,

$\Pi\left(\lVert f-f_{0}\rVert_{L^{2}(P_{X})}\leq\varepsilon_{n}\right)\geq
e^{-Cn\varepsilon_{n}^{2}}.$

We apply Lemma 19 for $t>t^{*}$ and
$2\varepsilon=\varepsilon_{n}:=\left(n/\log^{2(1+\kappa)+1}n\right)^{-\beta/(2\beta+t)}.$
When $n$ is large enough there is a network $\tilde{f}_{0}$ with constant
length, number of non-zero parameters
$\tilde{s}\lesssim\varepsilon_{n}^{-t/\beta}$ and magnitude of coefficients
$\max_{k}|\tilde{\theta}_{k}|\leq n$, such that
$||f_{0}-\tilde{f}_{0}||_{L^{2}(P_{X})}\leq\varepsilon_{n}/2$. Since
$\tilde{s}\leq n$ when $n$ is large enough, we can embed $\tilde{f}_{0}$ in
the space $\mathcal{F}(\log n,(d,n,\dots,n,1),s)$ where
$s\lesssim\tilde{s}\log n$. Write $(\theta_{k})$ (resp.
$(\tilde{\theta}_{k})$) for the coefficients of $f$ (resp. $\tilde{f}_{0}$).
Recall that there is a constant $C_{V}>0$, such that

$V:=\prod_{l=1}^{L}(r_{l}+1)\leq e^{C_{V}\log^{2}n}.$ (36)

Let $S_{0}=\\{k\in[T]\,:\,\tilde{\theta}_{k}\neq 0\\}$, recall $|S_{0}|\leq
s\lesssim\varepsilon_{n}^{-t/\beta}\log n$ and let

$z_{n}:=\varepsilon_{n}/(2n^{L}V(\log n+1)).$

Using triangle inequality and applying Lemma 18 we need to bound from below

$\Pi\left(\lVert
f-f_{0}\rVert_{L^{2}(P_{X})}\leq\varepsilon_{n}\right)\geq\prod_{k=1}^{T}\Pi\left(|\theta_{k}-\tilde{\theta}_{k}|\leq
z_{n}\,,\,|\theta_{k}|\leq n\right).$

We can follow the proof of Theorem 2, splitting the product in the last
display on whether or not $k$ is in $S_{0}$. For $k\in S_{0}$ we apply Lemma
21 and get a large enough constant $C_{7}>0$ such that

$\prod_{k\in S_{0}}\Pi\left(|\theta_{k}-\tilde{\theta}_{k}|\leq
z_{n}\,,\,|\theta_{k}|\leq n\right)\geq
z_{n}^{s}e^{-C_{7}s\log^{2(1+\kappa)}n}.$

Using the bound (36) we have a large enough constant $C_{V}^{\prime}$ such
that

$2n^{L}V(\log n+1)\leq e^{C_{V}^{\prime}\log^{2}n}.$

Finally, use
$s\log^{2(1+\kappa)}n\lesssim\varepsilon_{n}^{-t/\beta}\log^{2(1+\kappa)+1}n\leq
n\varepsilon_{n}^{2}$ to get $C>0$ large enough such that

$\prod_{k\in S_{0}}\Pi\left(|\theta_{k}-\tilde{\theta}_{k}|\leq
z_{n}\,,\,|\theta_{k}|\leq n\right)\geq e^{-Cn\varepsilon_{n}^{2}}.$

For the zero coefficients, $k\notin S_{0}$ we can simply follow the proof of
Theorem 2 and get

$\prod_{k\notin S_{0}}\Pi\left(|\theta_{k}|\leq z_{n}\right)\geq
e^{-Cn\varepsilon_{n}^{2}},$

which, according to Lemma 15 gives us concentration of the tempered posterior
in terms of the $\alpha$-Rényi divergence $D_{\alpha}$.

For the concentration of the variational posterior, using the existence of
$\tilde{f}_{0}\in\mathcal{F}(L,\mathbf{r}(n),s)$ such that for $n$ large
enough $||f_{0}-\tilde{f}_{0}||_{L^{2}(P_{X})}\leq\varepsilon_{n}/2$ and
$s\log^{2(1+\kappa)}n\lesssim n\varepsilon_{n}^{2}$, one can easily follow the
steps of the proof of Theorem 5 to get to the result.

###  C.4 Proofs for anisotropic Besov spaces

Proof [Proof of Theorem 10 ] We apply Lemma 20 with $N=\lceil
n/\log^{\gamma}n\rceil^{1/(2\tilde{\beta}+1)}$. We get
$\tilde{f}_{0}\in\mathcal{F}(L,(d,\Lambda,\dots,\Lambda,1),s)$ such that when
$n$ is large enough, $L\lesssim\log n$, $\Lambda\lesssim N$ and

$s\lesssim NL\lesssim\epsilon_{n}^{-1/\tilde{\beta}}\log n\lesssim
n\epsilon_{n}^{2}\log^{1-\gamma}n.$

The network realisation $\tilde{f}_{0}$ satisfies $\lVert
f_{0}-\tilde{f}_{0}\rVert_{\infty}\lesssim\epsilon_{n}/2$ and the coefficients
of $\tilde{f}_{0}$ are uniformly bounded by a constant. When $n$ is large
enough we can embed $\tilde{f}_{0}$ into the space
$\mathcal{F}(\log^{1+\delta}n,\mathbf{r}(n),s)$ and assume that
$\max_{k}|\tilde{\theta}_{k}|\leq n$. From here one can follow the proofs of
Theorems 5 and 7 to get the desired results.

##  Appendix D Bound on the coefficients

In this section we give the main steps of the proof of Kohler and Langer
(2021) leading to Lemma 23. Here the goal is to track the bounds on the
coefficients of the constructed network in order to obtain a quantitative
upper-bound in terms of the number of parameters. For the individual
construction of each subnetwork and their approximating capacity we refer to
the proof of Kohler and Langer (2021).

The goal is to approximate $f\in\mathcal{C}^{\beta}_{d}([0,1]^{d},F)$ with
precision $M^{-2\beta}$ on hypercubes of size $M^{-2}$ using local Taylor
polynomials of $f$. The network used to find on which of these sub-cubes we
are, will lead to the coefficients of highest magnitude through the important
quantity

$B_{M}:=\lceil M^{2(\beta+1)}\rceil.$

Note that, in the following, if $g$ is the realization of a neural network
with coefficients ${\theta_{k}}$, we denote the scale of its coefficients as

$B(g):=\underset{k}{\max}|\theta_{k}|.$

Recall the Lemma

######  Lemma 23 (Theorem 2 in Kohler and Langer (2021))

Let $f\in\mathcal{C}^{\beta}_{d}([0,1]^{d},F)$ be a function of regularity
$\beta>0$, and let $M\geq 2$ be an integer such that the inequality

$M^{2\beta}\geq C(1\vee F)^{4(\beta+1)}$

holds for a sufficiently large constant $C\geq 1$.

Let $L,r\in\mathbb{N}$ satisfy

$L\geq
5+\lceil\log_{4}(M^{2\beta})\rceil\left(\lceil\log_{2}(d\vee\lfloor\beta\rfloor+1)\rceil+1\right)$

and

$r\geq
64\binom{d+\lfloor\beta\rfloor}{d}2^{d}d^{2}(\lfloor\beta\rfloor+1)M^{d}.$

There exists a ReLU network
$\tilde{f}_{\text{wide}}\in\mathcal{F}(L,(d,r,\dots,r,1))$ such that

$||f-\tilde{f}_{\text{wide}}||_{\infty}\lesssim(1\vee
F)^{4(\beta+1)}M^{-2\beta}.$

###### Proposition 24

Let $B_{M}:=\lceil M^{2(\beta+1)}\rceil$, every coefficient $\theta$ of the
network $\tilde{f}_{\text{wide}}$ of the previous Lemma satisfies

$|\theta|\leq\max\\{2(F\vee 1)e^{2d},B_{M}^{2}\\}.$

Here are the step leading to Proposition 24.

###  D.1 Recursive definition of Taylor polynomials

###### Lemma 25

Let $f\in\mathcal{C}^{\beta}_{d}([0,1]^{d},F)$ and $x_{0}\in\mathbb{R}^{d}$.
Define

$T_{f,\beta,x_{0}}(x):=\sum_{j\in\mathbb{N}^{d};|j|\leq\lfloor\beta\rfloor}(\partial^{j}f)(x_{0})\,\frac{(x-x_{0})^{j}}{j!},$

then, for all $x\in\mathbb{R}^{d},$

$|f(x)-T_{f,\beta,x_{0}}(x)|\lesssim||x-x_{0}||_{\infty}^{\beta}.$

We conduct the following study on $[-1,1]^{d}$ to match Kohler and Langer
(2021) results, the same can be done on the unit cube $[0,1]^{d}$. Set
$\mathcal{P}_{2}:=\\{C_{k,2}\\}_{k\in[M^{2d}]}$ a partition of $[-1,1]^{d}$
into $M^{2d}$ cubes and for all $x\in\mathbb{R}^{d}$,
$(C_{\mathcal{P}_{2}}(x))_{g}$ the leftmost corner of the cube in
$\mathcal{P}_{2}$ that contains $x$, from the previous lemma,

$\forall
x\in[-1,1]^{d},\quad|f(x)-T_{f,\beta,(C_{\mathcal{P}_{2}}(x))_{g}}(x)|\lesssim
M^{-2\beta}.$

The crux of the proof is to approximate these polynomials by ReLU neural
networks, to do so a recursive definition is given. Note that to compute
$T_{f,\beta,(C_{\mathcal{P}_{2}}(x))_{g}}$ we need to compute about $M^{2d}$
derivatives of $f$. Although it is possible to do so in a single layer of
width $M^{2d}$ a significant improvement here from sparse neural classes comes
from the fact that between $2$ layers of size $M^{d}$ in a fully-connected
network there are about $M^{2d}$ connections. Therefore the derivatives are
set as the weights of the networks, decreasing quadratically the width of the
structure. To do this Kohler and Langer (2021) introduce another partition of
broader scale, namely $\mathcal{P}_{1}:=\\{C_{k,1}\\}_{k\in[M^{d}]}$ a
partition of $[-1,1]^{d}$ into $M^{d}$ cubes.

Now for any $i\in[M^{d}],$ let $\\{\tilde{C}_{j,i}\\}_{j\in[M^{d}]}$ be the
cubes of $\mathcal{P}_{2}$ that are in $C_{i,1}$. Thus
$\mathcal{P}_{2}=\\{\tilde{C}_{i,j}\\}_{1\leq i,j\leq M^{d}}$ and for all
$x\in\mathbb{R}^{d}$,

$T_{f,\beta,(C_{\mathcal{P}_{2}}(x))_{g}}(x)=\sum_{1\leq i,j\leq
M^{d}}T_{f,\beta,(\tilde{C}_{i,j})_{g}}(x)\,\mathbf{1}_{\tilde{C}_{j,i}}(x)$
(37)

In order to compute this as a neural network, the expression (37) of the local
Taylor polynomial of $f$ can be written recursively as follows:

  1. 1.

first find in which broader cube of $\mathcal{P}_{1}$ the point $x$ is located
and compute derivatives of $f$ on each smaller cube of size $1/M^{2}$ inside,

$\displaystyle\phi_{1,1}$ $\displaystyle:=x,$ $\displaystyle\phi_{2,1}$
$\displaystyle:=\sum_{i=1}^{M^{d}}(C_{i,1})_{g}\,\mathbf{1}_{C_{i,1}}(x),$
$\displaystyle\phi_{3,1}^{(l,j)}$
$\displaystyle:=\sum_{i=1}^{M^{d}}(\partial\,^{l}f)((\tilde{C}_{j,i})_{g})\,\mathbf{1}_{C_{i,1}}(x),$

for all $j\in[M^{d}]$ and $|l|\leq\lfloor\beta\rfloor.$

  2. 2.

then find the small cube of $\mathcal{P}_{2}$ containing $x$ and gather the
corresponding derivatives from the first step; to do this first note that for
all $1\leq k,i\leq M^{d}$,

$v_{k}:=(\tilde{C}_{k,i})_{g}-(C_{i,1})_{g}$ (38)

takes values in $\\{0,2/M^{2},\dots,2(M-1)/M^{2}\\}$ and there is
$j\in[M^{d}]$ such that,

$C_{\mathcal{P}_{2}}(x)=\mathcal{A}^{(j)}:=\\{x\in\mathbb{R}^{d},\,\phi_{2,1}^{(k)}+v_{j}^{(k)}\leq
x^{(k)}<\phi_{2,1}^{(k)}+v_{j}^{(k)}+2/M^{2},\quad\forall
k\in\\{1,\dots,d\\}\\}.$

Then set,

$\displaystyle\phi_{1,2}$ $\displaystyle:=x,$ $\displaystyle\phi_{2,2}$
$\displaystyle:=\sum_{j=1}^{M^{d}}(\phi_{2,1}+v_{j})\,\mathbf{1}_{\mathcal{A}^{(j)}}(\phi_{1,1}),$
$\displaystyle\phi_{3,2}^{(l)}$
$\displaystyle:=\sum_{j=1}^{M^{d}}\phi_{3,2}^{(l,j)}\,\mathbf{1}_{\mathcal{A}^{(j)}}(\phi_{1,1}).$

  3. 3.

finally compute the local Taylor polynomial as

$\phi_{1,3}:=\sum_{j\in\mathbb{N}^{d};|j|\leq\lfloor\beta\rfloor}\frac{\phi_{3,2}^{(j)}}{j!}\,(\phi_{1,2}-\phi_{2,2})^{j}.$

###### Lemma 26

$\phi_{1,3}=T_{f,\beta,(C_{\mathcal{P}_{2}}(x))_{g}}(x)$

In the next step we construct an approximation of $\phi_{1,3}$ by ReLU neural
networks. This approximation will be correct as soon as $x$ is not too close
to one of the boundaries of a $\mathcal{P}_{2}$ cube. More precisely, if $C$
is a cube and $\delta$ is a strictly positive real number, let
$C_{\delta}^{0}$ be the set of points in $C$ at a distance of at least
$\delta$ from the boundary of $C$.

###### Lemma 27

There is $\tilde{f}\in\mathcal{F}(L,(d,N,\dots,N,1))$ such that
$\lVert\tilde{f}\rVert_{\infty}\leq 2(F\vee 1)e^{2d}$ and

$\forall
x\in\bigcup_{j=1}^{M^{2d}}(C_{j,2})_{1/M^{2\beta+2}}^{0},\quad|f(x)-\tilde{f}(x)|\lesssim
M^{-2\beta}.$

Furthermore, $L\asymp\log M$, $N\lesssim M^{d}$ and $B(\tilde{f})\leq F\vee
B_{M}^{2}$ with $B_{M}:=\lceil M^{2\beta+2}\rceil$.

The boundary condition arises from the fact that through ReLU networks, only
continuous piecewise affine functions can be realized. Therefore, it is
necessary to approximate the indicator functions in the definition of
$\phi_{1,3}$. Moreover, we will see that the accuracy of the approximation of
these indicators controls the magnitude of the largest coefficients of the
network.

###  D.2 Auxiliary networks for the construction of $\phi_{1,3}$

To approximate polynomial functions through ReLU networks the first step is to
approximate the multiplication between two numbers, this construction comes
from Schmidt-Hieber (2020)

###### Lemma 28

For any positive integer $R$ there is $\hat{f}_{mult}\in\mathcal{F}(R,18)$
with $B(\hat{f}_{mult})\leq 4$, such that,

$\forall x,y\in[-1,1],\quad|\hat{f}_{mult}(x,y)-xy|\leq 4^{-R}.$

Starting from Lemma 28, it is quite easy to provide an approximation result
for polynomials in several variables. Let $N\in\mathbb{N}$, denote the set of
$d$-variables polynomials of total degree at most $N$ by,

$\mathcal{P}_{N}:=\operatorname{Vect}\left(\,\prod_{k=1}^{d}(x^{(k)})^{r_{k}}\,,\,r_{1}+\dots+r_{d}\leq
N\right).$

Since $\dim\mathcal{P}_{N}={\binom{d+N}{d}}$, write
$m_{1},\dots,m_{\binom{d+N}{d}}$ all the monomials in $\mathcal{P}_{N}$. Let
$r_{1},\dots,r_{\binom{d+N}{d}}$ be real numbers, for all $x\in[-1,1]^{d}$ and
$y_{1},\dots,y_{\binom{d+N}{d}}\in[-1,1],$ set

$\mathfrak{p}(x,y_{1},\dots,y_{\binom{d+N}{d}}):=\sum_{k=1}^{\binom{d+N}{d}}r_{k}\,y_{k}\,m_{k}(x).$

###### Lemma 29

Let $R\geq\log_{4}(2\times 4^{2(N+1)})$ be an integer and
$r_{1},\dots,r_{\binom{d+N}{d}}$ real numbers, set
$\overline{r(\mathfrak{p})}:=\max|r_{k}|$. There is a fully connected neural
network of length $R\lceil\log_{2}(N+1)\rceil$ and width
$18(N+1){\binom{d+N}{d}}$ whose realisation $\hat{f}_{\mathfrak{p}}$ is such
that $B(\hat{f}_{\mathfrak{p}})\leq 4\vee\overline{r(\mathfrak{p})}$, and,

$\left|\hat{f}_{\mathfrak{p}}(x,y_{1},\dots,y_{{\binom{d+N}{d}}})-\mathfrak{p}(x,y_{1},\dots,y_{{\binom{d+N}{d}}})\right|\lesssim\overline{r(\mathfrak{p})}4^{-R},$

for all $x\in[-1,1]^{d},y_{1},\dots,y_{\binom{d+N}{d}}\in[-1,1].$

As seen in the previous Section one also needs to approximate indicator
functions, this is where the coefficients of highest magnitude will appear.

###### Lemma 30

Let $R\in\mathbb{N}$ and $a,b\in\mathbb{R}^{d}$ such that for all
$b^{(i)}-a^{(i)}\geq 2/R$ for all $i\in\\{1,\dots,d\\}.$ Let

$K_{1/R}:=\left\\{x\in\mathbb{R}^{d},\,x^{(i)}\notin[a^{(i)},a^{(i)}+1/R[\,\cup\,]b^{(i)}-1/R,b^{(i)}],\text{
for all }i\in[d]\right\\}.$

  1. 1.

There is $\hat{f}_{ind[a,b[}\in\mathcal{F}(2,2d)$ with
$B(\hat{f}_{ind[a,b[})\leq\max(R,1/R,|a|_{\infty},|b|_{\infty})$ such that

$\forall x\in K_{1/R},\quad\hat{f}_{ind[a,b[}(x)=\mathbf{1}_{[a,b[}(x),$

and

$\forall
x\in\mathbb{R}^{d},\quad|\hat{f}_{ind[a,b[}(x)-\mathbf{1}_{[a,b[}(x)|\leq 1.$

  2. 2.

Let $|s|\leq R$, there is $\hat{f}_{test}(.\,,a,b,s)\in\mathcal{F}(2,2(2d+2))$
with $B(\hat{f}_{test})\leq\max(R^{2},1/R,|a|_{\infty},|b|_{\infty})$ such
that

$\forall x\in K_{1/R},\quad\hat{f}_{test}(x,a,b,s)=s\mathbf{1}_{[a,b[}(x),$

and

$\forall
x\in\mathbb{R}^{d},\quad|\hat{f}_{test}(x,a,b,s)-s\mathbf{1}_{[a,b[}(x)|\leq|s|.$

Also it is clear that one can realise the identity function of
$\mathbb{R}^{d}$ with a ReLU network of arbitrary length, write
$f_{id}^{\ell}$ such network of length $\ell\geq 2$. Choose
$l_{1},\dots,l_{\binom{d+\lfloor\beta\rfloor}{d}}$ such that

$\left\\{l_{1},\dots,l_{\binom{d+\lfloor\beta\rfloor}{d}}\right\\}=\\{(s_{1},\dots,s_{d})\in\mathbb{N}^{d}\,,\,s_{1}+\dots+s_{d}\leq\lfloor\beta\rfloor\\}.$

Here are the networks deployed to approximate $\phi_{1,3}$, where for the
$\hat{f}_{ind}$ and $\hat{f}_{test}$ networks we use Lemma 30 with
$R=B_{M}:=\lceil M^{2\beta+2}\rceil$. Let
$\mathbf{J}:=(1,\dots,1)^{T}\in\mathbb{R}^{d}$,

$\displaystyle\hat{\phi}_{1,1}$ $\displaystyle:=f_{id}^{2}(x),$
$\displaystyle\hat{\phi}_{2,1}$
$\displaystyle:=\sum_{i=1}^{M^{d}}(C_{i,1})_{g}\,\hat{f}_{ind\,C_{i,1}}(x),$
$\displaystyle\hat{\phi}_{3,1}^{(l,j)}$
$\displaystyle:=\sum_{i=1}^{M^{d}}(\partial\,^{l}f)((\tilde{C}_{j,i})_{g})\hat{f}_{ind\,C_{i,1}}(x),$
$\displaystyle\hat{\phi}_{1,2}$ $\displaystyle:=f_{id}^{2}(\hat{\phi}_{1,1}),$
$\displaystyle\hat{\phi}_{2,2}^{(k)}$
$\displaystyle:=\sum_{i=1}^{M^{d}}\hat{f}_{test}(\hat{\phi}_{1,1},\hat{\phi}_{2,1}+v_{i},\hat{\phi}_{2,1}+v_{i}+2/M^{2}\cdot\mathbf{J},\hat{\phi}_{2,1}^{(k)}+v_{i}^{(k)}),$
$\displaystyle\hat{\phi}_{3,2}^{(l)}$
$\displaystyle:=\sum_{i=1}^{M^{d}}\hat{f}_{test}(\hat{\phi}_{1,1},\hat{\phi}_{2,1}+v_{i},\hat{\phi}_{2,1}+v_{i}+2/M^{2}\cdot\mathbf{J},\hat{\phi}_{3,1}^{(l,i)}).$

Now, for any $u\in\\{1,\dots,{\binom{d+\lfloor\beta\rfloor}{d}}\\}$ set
$y_{u}:=\hat{\phi}_{3,2}^{(l_{u})}$ and $r_{u}:=1/(l_{u})!$, the approximating
network of $\phi_{3,1}$ is given by Lemma 29 with

$R=B_{M,\mathfrak{p}}:=\lceil\log_{4}(2\cdot 4^{2(\beta+1)}\cdot(2\vee
F)^{2(\beta+1)})\rceil,$

through

$\hat{\phi}_{1,3}=\hat{f}_{\mathfrak{p}}(\hat{\phi}_{1,2}-\hat{\phi}_{2,2},y_{1},\dots
y_{\binom{d+\lfloor\beta\rfloor}{d}}).$

The network $\hat{\phi}_{1,3}$ satisfies conditions of Lemma 27, we only
provide the bound on the coefficients and refer to Kohler and Langer (2021)
for a proof on the approximating properties. It is clear that
$B(\hat{\phi}_{1,1})=B(\hat{\phi}_{1,2})=1$. Using Lemma 30 one gets

$B(\hat{\phi}_{2,1})\leq\max\left\\{\max_{i}|(C_{i,1})_{g}|_{\infty},B_{M}^{2},1/B_{M}\right\\}\leq\max(2/M,B_{M}^{2})\leq
B_{M}^{2},$

and for all $l\in\\{1,\dots,{\binom{d+\lfloor\beta\rfloor}{d}}\\}$ and
$j\in[M^{d}],$

$B(\hat{\phi}_{3,1}^{(l,j)})\leq\max\left\\{\max_{i}|(\partial\,^{l}f)((\tilde{C}_{j,i})_{g})|,B_{M}^{2},1/B_{M}\right\\}\leq
F\vee B_{M}^{2}.$

From the proof of Kohler and Langer (2021) it appears that the networks
previously defined $(\hat{\phi}^{(k)}_{2,1},\hat{\phi}_{3,1}^{(l,j)})$ are all
in the set of bounded functions $\\{g\,:\,||g||_{\infty}\leq 1\vee F\\}$, thus
using Lemma 30 one gets, for all $k\in\\{1,\dots,d\\}$ and
$l\in\\{1,\dots,{\binom{d+\lfloor\beta\rfloor}{d}}\\}$,

$B(\hat{\phi}_{2,2}^{(k)})\leq F\vee B_{M}^{2}\quad\text{and}\quad
B(\hat{\phi}_{3,2}^{(l)})\leq F\vee B_{M}^{2}.$

One can check that $|\hat{\phi}_{1,2}-\hat{\phi}_{2,2}|\leq 2$ and
$\lVert\hat{\phi}_{3,2}\rVert_{\infty}\leq F$. Now since
$\overline{r(\mathfrak{p})}=\max|1/(l_{u})!|\leq 1$ and $B_{M}\geq 4$ from
$M\geq 2$, Lemma 29 yields,

$B(\hat{\phi}_{1,3})\leq\max\\{F,B_{M}^{2},B_{M,\mathfrak{p}}\\}\leq F\vee
B_{M}^{2}.$

###  D.3 Approximation of $w_{\mathcal{P}_{2}}(x)\cdot f(x)$

The next step in the proof deals with handling the case where $x$ is such that
the approximation provided by Lemma 27 is no longer valid. To address this,
define a weight function $w_{\mathcal{P}_{2}}$ for all $x\in\mathbb{R}^{d}$ by

$w_{\mathcal{P}_{2}}(x):=\prod_{k=1}^{d}\left(1-M^{2}\left|(C_{\mathcal{P}_{2}}(x))_{g}^{(k)}+\frac{1}{M^{2}}-x^{(k)}\right|\right)_{+}.$
(39)

This function reaches its maximum at the center of the cube of
$\mathcal{P}_{2}$ containing $x$ and is zero outside. Moreover, it takes very
small values near the boundaries of the cube. More precisely, we have,

$\text{for all
}x\in\bigcup_{j=1}^{M^{2d}}C_{j,2}\setminus(C_{j,2})_{1/M^{2\beta+2}}^{0},\quad
w_{\mathcal{P}_{2}}(x)\leq{M^{-2\beta}}.$ (40)

In this step we want to construct a network that will realise a good
approximation of $w_{\mathcal{P}_{2}}(x)\cdot f(x)$ on the whole cube. In the
final step we will aggregate these approximations using a partition of unity
argument to get back to $f$.

###### Lemma 31

There is $\tilde{f}\in\mathcal{F}(L,(d,N,\dots,N,1))$ with $L\lesssim\log M$,
$N\lesssim M^{d}$ and $B(\tilde{f})\leq B_{M}^{2}\vee F$, such that

$\forall x\in[-1,1]^{d},\quad|\tilde{f}(x)-w_{\mathcal{P}_{2}}(x)\cdot
f(x)|\lesssim M^{-2\beta}.$

First approximate $w_{\mathcal{P}_{2}}$ with wide neural networks.

###### Lemma 32

There is a network
$\hat{f}_{w_{\mathcal{P}_{2}}}\in\mathcal{F}(L,(d,N,\dots,N,1))$ such that
$\lVert\hat{f}_{w_{\mathcal{P}_{2}}}\rVert_{\infty}\leq 2$ and

$\forall
x\in\bigcup_{i=1}^{M^{2d}}(C_{i,2})_{1/M^{2\beta+2}}^{0},\quad|\hat{f}_{w_{\mathcal{P}_{2}}}(x)-w_{\mathcal{P}_{2}}(x)|\leq
M^{-2\beta},$

with $L\lesssim\log M$, $N\lesssim M^{d}$ and
$B(\hat{f}_{w_{\mathcal{P}_{2}}})\leq B_{M}^{2}\vee F.$

From (39) it is clear that such network exists using $\hat{\phi}_{2,2}$ to get
$(C_{\mathcal{P}_{2}}(x))_{g}$ and $\hat{\phi}_{1,2}$ to get $x$. Those yield
good approximations since $x$ is “far” from the boundary of the cube and from
previous calculations,

$B(\hat{f}_{w_{\mathcal{P}_{2}}})\leq\max\\{4,B(\hat{\phi}_{1,2}),B(\hat{\phi}_{2,2}),M^{2},1/M^{2}\\}\leq
B_{M}^{2}\vee F.$

Lemmas 27 and 32 already show that there is a network that will approximate
correctly $w_{\mathcal{P}_{2}}(x)\cdot f(x)$ whenever $x$ is at least
$M^{-2(\beta+1)}$ away from the boundaries of $C_{\mathcal{P}_{2}}(x)$. The
idea now is to change a bit this network so that it is zero whenever $x$ is
too close to the boundaries. The resulting network will be a good
approximation of $w_{\mathcal{P}_{2}}(x)\cdot f(x)$ on the whole cube because
of (40). The following lemma defines a network that checks the relative
position of $x$ with respect to the boundaries of $C_{\mathcal{P}_{2}}(x)$.

###### Lemma 33

There is $\hat{f}_{check\mathcal{P}_{2}}\in\mathcal{F}(5,(d,N,\dots,N,1))$
such that $\lVert\hat{f}_{check\mathcal{P}_{2}}\rVert_{\infty}\leq 1$ and

$\forall
x\notin\bigcup_{i=1}^{M^{2d}}(C_{i,2})_{1/M^{2\beta+2}}^{0}\setminus(C_{i,2})_{2/M^{2\beta+2}}^{0},\quad\hat{f}_{check\mathcal{P}_{2}}(x)=\mathbf{1}\big{(}\bigcup_{j=1}^{M^{2d}}C_{j,2}\setminus(C_{j,2})_{1/M^{2\beta+2}}^{0}\big{)},$

with $N\asymp M^{d}$ and $B(\hat{f}_{check\mathcal{P}_{2}})\leq B_{M}^{2}.$

First find which cube of $\mathcal{P}_{1}$ contains $x$ using Lemma 30 with
$R=B_{M}$,

$\hat{f}_{1}:=1-\sum_{i=1}^{M^{d}}\hat{f}_{ind(C_{i,1})_{1/M^{2\beta+2}}^{0}},$

previous calculations have shown that $B(\hat{f}_{1})\leq B_{M}^{2}$. Then
find which smaller cube of $\mathcal{P}_{2}$ contains $x$ using lemma 30 again
with $R=B_{M}$. Recall $\mathbf{J}=(1,\dots,1)^{T}\in\mathbb{R}^{d}$,

$\hat{f}_{2}(x):=1-\sum_{j=1}^{M^{d}}\hat{f}_{test}(f_{id}^{2}(x),\hat{\phi}_{2,1}+v_{j}+M^{-2(\beta+1)}\cdot\mathbf{J},\hat{\phi}_{2,1}+v_{j}+2/M^{2}\cdot\mathbf{J}+M^{-2(\beta+1)}\cdot\mathbf{J},1).$

From previous calculations we know
$B(\hat{f}_{2})\leq\max\\{B_{M}^{2},B(\hat{\phi}_{2,1})\\}\leq B_{M}^{2}.$
Recall $\rho(x)=\max\\{0,x\\}$ and define

$\hat{f}_{check\mathcal{P}_{2}}(x):=1-\rho(1-\hat{f}_{2}(x)-f_{id}^{2}(\hat{f}_{1}(x))).$

This network satisfies conditions of Lemma 33.

To construct a network satisfying the conditions of Lemma 31 let
$\hat{f}_{\mathcal{P}_{2}}$ be the network defined by Lemma 27 and
$B_{true}:=2(1\vee F)e^{2d}$ be an upper bound on its supremum norm. Set

$\hat{f}_{\mathcal{P}_{2},true}:=\rho(\hat{f}_{\mathcal{P}_{2}}-B_{true}\cdot\hat{f}_{check\mathcal{P}_{2}})-\rho(-\hat{f}_{\mathcal{P}_{2}}-B_{true}\cdot\hat{f}_{check\mathcal{P}_{2}}),$

this network is zero whenever $x$ is less than $M^{-2(\beta+1)}$ close from
the boundaries of $C_{\mathcal{P}_{2}}$ and otherwise it will be a good
approximation of $f$. Thus
$\tilde{f}=\hat{f}_{mult}(\hat{f}_{\mathcal{P}_{2},true},\hat{f}_{w_{\mathcal{P}_{2}}})$
is always a good approximation of $w_{\mathcal{P}_{2}}(x)\cdot f(x)$ and
satisfies conditions of Lemma 31. Indeed, we have

$B(\hat{f}_{\mathcal{P}_{2},true})\leq\max\\{B_{true},B(\hat{f}_{\mathcal{P}_{2}}),B(\hat{f}_{check\mathcal{P}_{2}})\\}\leq\max\\{2(F\vee
1)e^{2d},B_{M}^{2}\\},$

and

$B(\tilde{f})\leq\max\\{B(\hat{f}_{\mathcal{P}_{2},true}),B(\hat{f}_{w_{\mathcal{P}_{2}}})\\}\leq\max\\{2(F\vee
1)e^{2d},B_{M}^{2}\\}.$

###  D.4 Partition of unity through $w_{\mathcal{P}_{2}}$

For the final step, construct a partition of unity using the
$w_{\mathcal{P}_{2}}$ weight functions. To do so shift the partitions
$\mathcal{P}_{1}$ and $\mathcal{P}_{2}$ by shifting at least one of the
coordinates by $1/M^{2}$. This leads to two sets of $2^{d}$ partitions
$\\{P_{1,v},v\in\\{1,\dots,2^{d}\\}\\}$ and
$\\{P_{2,v},v\in\\{1,\dots,2^{d}\\}\\}$. Enlarging the original hypercube if
necessary the resulting family of weight functions
$\\{w_{\mathcal{P}_{2},v}\\}$ is a partition of unity on $[-1,1]^{d}$. If we
set $\hat{f}_{v}$ to be the network given by Lemma 31 for the partition
$\mathcal{P}_{2,v}$ then $f_{wide}:=\sum_{v=1}^{2^{d}}\hat{f}_{v}$ yields a
good approximation of $f=\sum_{v}w_{\mathcal{P}_{2,v}}f$ and satisfies all the
conditions of Lemma 23, in particular

$B(f_{wide})\leq\max\\{2(F\vee 1)e^{2d},B_{M}^{2}\\}.$

## References

  * Agapiou and Castillo (2024) S. Agapiou and I. Castillo.  Heavy-tailed Bayesian nonparametric adaptation.  _The Annals of Statistics_ , 2024.  To appear, arXiv eprint 2308.04916.
  * Agapiou et al. (2024) S. Agapiou, I. Castillo, and P. Egels.  Heavy-tailed and Horseshoe priors for regression and sparse Besov rates.  2024\.  Manuscript in preparation.
  * Alquier (2024) P. Alquier.  User-friendly introduction to PAC-bayes bounds.  _Foundations and Trends in Machine Learning_ , 17(2):174–303, 2024.
  * Alquier and Ridgway (2020) P. Alquier and J. Ridgway.  Concentration of tempered posteriors and of their variational approximations.  _The Annals of Statistics_ , 48(3):1475 – 1497, 2020.
  * Arbel et al. (2023) J. Arbel, K. Pitas, M. Vladimirova, and V. Fortuin.  A primer on Bayesian neural networks: Review and debates, 2023.  arXiv eprint 2309.16314.
  * Bai et al. (2020) J. Bai, Q. Song, and G. Cheng.  Efficient variational inference for sparse deep learning with theoretical guarantee.  In _Advances in Neural Information Processing Systems_ , volume 33, 2020.
  * Bhattacharya et al. (2019) A. Bhattacharya, D. Pati, and Y. Yang.  Bayesian fractional posteriors.  _The Annals of Statistics_ , 47(1):39 – 66, 2019.
  * Blei et al. (2017) D. M. Blei, A. Kucukelbir, and J. D. McAuliffe.  Variational inference: A review for statisticians.  _Journal of the American Statistical Association_ , 112(518):859–877, Apr. 2017.
  * Castillo (2024) I. Castillo.  Bayesian nonparametrics: Saint–Flour lecture notes.  2024\.  arXiv eprint 2403.01737.
  * Castillo and Randrianarisoa (2024) I. Castillo and T. Randrianarisoa.  Deep Horseshoe Gaussian Processes.  2024\.  arXiv eprint 2403.01737.
  * Chen et al. (2019) M. Chen, H. Jiang, W. Liao, and T. Zhao.  Efficient approximation of deep ReLU networks for functions on low dimensional manifolds.  In _Advances in Neural Information Processing Systems_ , volume 32, 2019.
  * Chérief-Abdellatif (2020) B.-E. Chérief-Abdellatif.  Convergence rates of variational inference in sparse deep learning.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 1831–1842, 13–18 Jul 2020.
  * Damianou and Lawrence (2013) A. Damianou and N. D. Lawrence.  Deep Gaussian processes.  In _Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics_ , volume 31 of _Proceedings of Machine Learning Research_ , pages 207–215, 29 Apr–01 May 2013.
  * Donoho and Johnstone (1998) D. L. Donoho and I. M. Johnstone.  Minimax estimation via wavelet shrinkage.  _The Annals of Statistics_ , 26(3):879 – 921, 1998.
  * Finocchio and Schmidt-Hieber (2023) G. Finocchio and J. Schmidt-Hieber.  Posterior contraction for deep Gaussian process priors.  _Journal of Machine Learning Research_ , 24(66):1–49, 2023.
  * Ghosal and van der Vaart (2017) S. Ghosal and A. van der Vaart.  _Fundamentals of Nonparametric Bayesian Inference_.  Cambridge Series in Statistical and Probabilistic Mathematics. Cambridge University Press, 2017.
  * Ghosal et al. (2000) S. Ghosal, J. K. Ghosh, and A. W. van der Vaart.  Convergence rates of posterior distributions.  _The Annals of Statistics_ , 28(2):500 – 531, 2000.
  * Ghosh et al. (2019) S. Ghosh, J. Yao, and F. Doshi-Velez.  Model selection in Bayesian neural networks via horseshoe priors.  _Journal of Machine Learning Research_ , 20(182):1–46, 2019.
  * Jiao et al. (2023) Y. Jiao, G. Shen, Y. Lin, and J. Huang.  Deep nonparametric regression on approximate manifolds: Nonasymptotic error bounds with polynomial prefactors.  _The Annals of Statistics_ , 51(2):691–716, 2023.
  * Kohler and Langer (2021) M. Kohler and S. Langer.  On the rate of convergence of fully connected deep neural network regression estimates.  _The Annals of Statistics_ , 49(4):2231 – 2249, 2021.
  * Kong and Kim (2024) I. Kong and Y. Kim.  Posterior concentrations of fully-connected Bayesian neural networks with general priors on the weights, 2024.  arxiv eprint 2403.14225.
  * Kong et al. (2023) I. Kong, D. Yang, J. Lee, I. Ohn, G. Baek, and Y. Kim.  Masked Bayesian neural networks: Theoretical guarantee and its posterior inference.  In _International Conference on Machine Learning_ , pages 17462–17491. PMLR, 2023.
  * Lee et al. (2023) H. Lee, F. Ayed, P. Jung, J. Lee, H. Yang, and F. Caron.  Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibility, 2023.  Arxiv eprint 2205.08187.
  * Lee and Lee (2022) K. Lee and J. Lee.  Asymptotic properties for Bayesian neural network in Besov space.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 5641–5653, 2022.
  * L’Huillier et al. (2023) A. L’Huillier, L. Travis, I. Castillo, and K. Ray.  Semiparametric inference using fractional posteriors.  _The Journal of Machine Learning Research_ , 24(1):18619–18679, 2023.
  * Lu et al. (2021) J. Lu, Z. Shen, H. Yang, and S. Zhang.  Deep network approximation for smooth functions.  _SIAM Journal on Mathematical Analysis_ , 53(5):5465–5506, 2021.
  * Nakada and Imaizumi (2020) R. Nakada and M. Imaizumi.  Adaptive approximation and generalization of deep neural network with intrinsic dimensionality.  _Journal of Machine Learning Research_ , 21(174):1–38, 2020.
  * Neal (1996) R. M. Neal.  _Priors for Infinite Networks_ , pages 29–53.  Springer New York, New York, NY, 1996.
  * Ohn and Lin (2024) I. Ohn and L. Lin.  Adaptive variational Bayes: Optimality, computation and applications.  _The Annals of Statistics_ , 52(1):335–363, 2024.
  * Papamarkou et al. (2024) T. Papamarkou et al.  Position paper: Bayesian deep learning in the age of large-scale AI.  2024\.  arxiv eprint 2402.00809.
  * Polson and Ročková (2018) N. G. Polson and V. Ročková.  Posterior concentration for sparse deep learning.  _Advances in Neural Information Processing Systems_ , 31, 2018.
  * Schmidt-Hieber (2019) J. Schmidt-Hieber.  Deep ReLU network approximation of functions on a manifold, 2019.  arXiv eprint 1908.00695.
  * Schmidt-Hieber (2020) J. Schmidt-Hieber.  Nonparametric regression using deep neural networks with ReLU activation function.  _The Annals of Statistics_ , 48(4):1875 – 1897, 2020.
  * Sell and Singh (2023) T. Sell and S. S. Singh.  Trace-class Gaussian priors for Bayesian learning of neural networks with MCMC.  _Journal of the Royal Statistical Society Series B: Statistical Methodology_ , 85(1):46–66, 01 2023.
  * (35) T. Suzuki.  Adaptivity of deep ReLU network for learning in Besov and mixed smooth Besov spaces: optimal rate and curse of dimensionality.  In _7th International Conference on Learning Representations, ICLR 2019_.
  * Suzuki and Nitanda (2021) T. Suzuki and A. Nitanda.  Deep learning is adaptive to intrinsic dimensionality of model smoothness in anisotropic Besov space.  _Advances in Neural Information Processing Systems_ , 34:3609–3621, 2021.
  * van Erven and Harremoes (2014) T. van Erven and P. Harremoes.  Rényi divergence and Kullback–Leibler divergence.  _IEEE Transactions on Information Theory_ , 60(7):3797–3820, 2014.
  * Yang et al. (2020) Y. Yang, D. Pati, and A. Bhattacharya.  $\alpha$-variational inference with statistical guarantees.  _The Annals of Statistics_ , 48(2):886 – 905, 2020.
  * Yarotsky (2017) D. Yarotsky.  Error bounds for approximations with deep ReLU networks.  _Neural Networks_ , 94:103–114, 2017.
  * Zhang and Gao (2020) F. Zhang and C. Gao.  Convergence rates of variational posterior distributions.  _The Annals of Statistics_ , 48(4):2180–2207, 2020.