# Excess risk bound for deep learning under weak dependence

William KENGNE 111Developed within the ANR BREAKRISK: ANR-17-CE26-0001-01, the
MME-DII center of excellence (ANR-11-LABEX-0023-01) and the CY Initiative of
Excellence (grant ”Investissements d’Avenir” ANR-16-IDEX-0008), Project
”EcoDep” PSI-AAP2020-0000000013

THEMA, CY Cergy Paris Université, 33 Boulevard du Port, 95011 Cergy-Pontoise
Cedex, France
E-mail<EMAIL_ADDRESS>

Abstract: This paper considers deep neural networks for learning weakly
dependent processes in a general framework that includes, for instance,
regression estimation, time series prediction, time series classification. The
$\psi$-weak dependence structure considered is quite large and covers other
conditions such as mixing, association,$\ldots$ Firstly, the approximation of
smooth functions by deep neural networks with a broad class of activation
functions is considered. We derive the required depth, width and sparsity of a
deep neural network to approximate any Hölder smooth function, defined on any
compact set ${\mathcal{X}}$. Secondly, we establish a bound of the excess risk
for the learning of weakly dependent observations by deep neural networks.
When the target function is sufficiently smooth, this bound is close to the
usual $\mathcal{O}(n^{-1/2})$.

Keywords: Deep neural networks, weak dependence, function approximation,
excess risk, ERM principle.

##  1 Introduction

We focus on the supervised learning framework and consider the training sample
$D_{n}=\\{Z_{1}=(X_{1},Y_{1}),\cdots,Z_{n}=(X_{n},Y_{n})\\}$ which is a
trajectory of a stationary and ergodic process
$\\{Z_{t}=(X_{t},Y_{t}),~{}t\in\mathbb{Z}\\}$, taking values in
$\mathcal{Z}=\mathcal{X}\times\mathcal{Y}$, where $\mathcal{X}$ is the input
space and $\mathcal{Y}$ the output space. In the sequel, we assume that
$\mathcal{X}\subset\mathbb{R}^{d_{x}}$ and
$\mathcal{Y}\subset\mathbb{R}^{d_{y}}$, with $d_{x},d_{y}\in\mathbb{N}$.
Denote by $\mathcal{F}(\mathcal{X},\mathcal{Y})$ the set of measurable
functions from $\mathcal{X}$ to $\mathcal{Y}$. Consider a loss function
$\ell:\mathbb{R}^{d_{y}}\times\mathcal{Y}\rightarrow[0,\infty)$ and for a
predictor $h\in\mathcal{F}(\mathcal{X},\mathcal{Y})$, define the risk,

$R(h)=\mathbb{E}_{Z_{0}}\big{[}\ell\big{(}h(X_{0}),Y_{0}\big{)}\big{]}.$ (1.1)

For a learner $h\in\mathcal{F}(\mathcal{X},\mathcal{Y})$, a prediction of
$Y_{t}$ is $\widehat{Y}_{t}=h(X_{t})$ for all $t\in\mathbb{Z}$. The ”best”
predictor $h^{*}\in\mathcal{F}(\mathcal{X},\mathcal{Y})$, when it exists, is
the one that achieves the smallest risk, that is,

$R(h^{*})=\underset{h\in\mathcal{F}(\mathcal{X},\mathcal{Y})}{\inf}R(h).$
(1.2)

We would like to construct a learner
$h\in\mathcal{F}(\mathcal{X},\mathcal{Y})$ such that, for any
$t\in\mathbb{Z}$, $h(X_{t})$ is average ”close” to $Y_{t}$; that is, a learner
with the smallest risk. But in general, this risk cannot be minimized in
practice, because the distribution of $(X_{0},Y_{0})$ is unknown. Consider the
empirical risk, defined for all $h\in\mathcal{F}(\mathcal{X},\mathcal{Y})$ by,

$\widehat{R}_{n}(h)=\frac{1}{n}\sum_{i=1}^{n}\ell\big{(}h(X_{i}),Y_{i}\big{)}.$
(1.3)

Thus, the aim is to build from the training sample $D_{n}$, a learner
$\widehat{h}_{n}$, that minimizes the empirical risk. In the sequel, we set
$\ell(h,z)=\ell\big{(}h(x),y\big{)}$ for all
$z=(x,y)\in\mathcal{X}\times\mathcal{Y}$ and
$h\in\mathcal{F}(\mathcal{X},\mathcal{Y})$.

We focus on the class of deep neural networks (DNN) predictors, with a general
activation function $\sigma:\mathbb{R}\rightarrow\mathbb{R}$. Recall that, a
neural network architecture $(L,\mathbf{p})$ stands for a positive integer $L$
called the number of hidden layers or depth and a width vector
$\mathbf{p}=(p_{0},p_{1},\cdots,p_{L+1})\in\mathbb{N}^{L+2}$. Therefore, a DNN
with network architecture $(L,\mathbf{p})$ is any function of the form,

$h:\mathbb{R}^{p_{0}}\rightarrow\mathbb{R}^{p_{L+1}},~{}x\mapsto
h(x)=A_{L+1}\circ\sigma_{L}\circ A_{L}\circ\sigma_{L-1}\circ
A_{L-1}\circ\cdots\circ\sigma_{1}\circ A_{1}(x),$ (1.4)

where for any $\ell=1,\cdots,L+1$,
$A_{\ell}:\mathbb{R}^{p_{\ell-1}}\rightarrow\mathbb{R}^{p_{\ell}}$ is an
affine linear application defined by $A_{\ell}(x):=W_{\ell}x+b_{\ell}$ for
given $p_{\ell-1}\times p_{\ell}$ weight matrix $W_{\ell}$, a shift vector
$b_{\ell}\in\mathbb{R}^{p_{\ell}}$ and
$\sigma_{\ell}:\mathbb{R}^{p_{\ell}}\rightarrow\mathbb{R}^{p_{\ell}}$ is an
element-wise nonlinear activation map defined as
$\sigma_{\ell}(z)=(\sigma(z_{1}),\cdots,\sigma(z_{p_{\ell}}))^{T}$ for all
$z=(z_{1},\cdots,z_{p_{\ell}})^{T}\in\mathbb{R}^{p_{\ell}}$, and T denotes the
transpose. In the setting considered here, $p_{0}=d_{x}$ (input dimension) and
$p_{L+1}=d_{y}$ (output dimension). For a DNN $f$ of the form (1.4), the
vector of its parameters is denoted by $\theta(h)$, that is,

$\theta(h)=\big{(}\text{vec}(W_{1})^{T},b_{1}^{T},\cdots,\text{vec}(W_{L+1})^{T},b_{L+1}^{T}\big{)},$

where $\text{vec}(W)$ denotes the vector obtained by concatenating the column
vectors of the matrix $W$. In the sequel, we deal with an activation function
$\sigma$ and denote by $\mathcal{H}_{\sigma,d_{x},d_{y}}$ the set of DNNs with
$d_{x}$ dimensional input and $d_{y}$ dimensional output. For any
$h\in\mathcal{H}_{\sigma,d_{x},d_{y}}$ with a neural network architecture
$(L,\mathbf{p})$, $\text{depth}(h)$ denotes its depth and $\text{width}(h)$
denotes its width, that is, $\text{depth}(h)=L$ and
$\text{width}(h)=\underset{1\leq\ell\leq L}{\max}p_{\ell}$. For any
$L,N,S,B,F>0$, consider the sets,

$\mathcal{H}_{\sigma,{d_{x}},{d_{y}}}(L,N)=\big{\\{}h\in\mathcal{H}_{\sigma,d_{x},d_{y}},~{}\text{depth}(h)\leq
L,~{}\text{width}(h)\leq N\big{\\}},$
$\mathcal{H}_{\sigma,{d_{x}},{d_{y}}}(L,N,S,B)=\big{\\{}h\in\mathcal{H}_{\sigma,{d_{x}},{d_{y}}}(L,N),~{}|\theta(h)|_{0}\leq
S,~{}\|\theta(h)\|\leq B\big{\\}},$ (1.5)
$\mathcal{H}_{\sigma,{d_{x}},{d_{y}}}(L,N,S,B,F)=\big{\\{}h\textrm{\dsrom{1}}_{{\mathcal{X}}},~{}h\in\mathcal{H}_{\sigma,{d_{x}},{d_{y}}}(L,N,S,B),~{}\|h\|_{\infty,{\mathcal{X}}}\leq
F\big{\\}},$ (1.6)

where for any $x=(x_{1},\cdots,x_{d})^{T}\in\mathbb{R}^{d}$,
$|x|_{0}=\sum_{i=1}^{d}\textrm{\dsrom{1}}(x_{i}\neq 0)$,
$\|x\|=\underset{1\leq i\leq d}{\max}|x_{i}|$ and
$\|h\|_{\infty,{\mathcal{X}}}=\sup_{x\in{\mathcal{X}}}\|h(x)\|$ (see also the
Subsection 2.1 bellow).

There are many works in the literature on the approximation of smooth
functions by DNNs. We refer to [25], [23], [20], [18], [21], [22] and the
references therein for an overview. The existing results are mainly obtained,
either for a restricted class of activation function (for example, ReLU), or
for a specific case of input space (for example,
${\mathcal{X}}=[0,1]^{d_{x}}$). The first contribution of this paper:

  * (i)

Establish approximation results by DNNs with a quite large class of activation
function, of smooth functions defined on any compact input space
${\mathcal{X}}\subset\mathbb{R}^{d_{x}}$.

The second issue of this work is the learning of the process
$\\{Z_{t}=(X_{t},Y_{t}),~{}t\in\mathbb{Z}\\}$ by the class of DNNs
$\mathcal{H}_{\sigma,{d_{x}},{d_{y}}}(L,N,S,B,F)$ with $L,N,S,B,F>0$. There is
a large literature that addresses such question with independent and
identically distributed (i.i.d.) observations. See, among others papers, [1],
[18], [22], [13], [19]. For some theoretical results on dependent
observations, see for instance, [14], [15], [17]. These works are for mixing
observations and/or for the regression problem. The recent work of [12]
considers a general framework of supervised learning for weakly dependent
processes, where the weak dependence structure covers mixing conditions. But,
the convergence of the excess risk is not addressed. The second contribution
of this paper:

  * (ii)

Derive the convergence rate of the excess risk for the learning of
$\psi$-weakly dependent processes by DNNs. This rate is close to the usual
$\mathcal{O}(n^{-1/2})$ when the target function is sufficiently smooth.

The rest of the paper is structured as follows. Section 2 sets some notations
and assumptions. Section 3 focuses on Hölder smooth functions approximation by
DNNs. Section 4 considers the excess risk and provides its convergence rate,
whereas Section 5 is devoted to the proofs of the main results.

##  2 Notations and assumptions

###  2.1 Some notations

Throughout the sequel, the following notations will be used, with
$d\in\mathbb{N}$, and where $E_{1},E_{2}$ are subsets of separable Banach
spaces equipped with norms $\|\cdot\|_{E_{1}}$ and $\|\cdot\|_{E_{2}}$
respectively.

  * •

$\mathbb{N}_{0}=\mathbb{N}\cup\\{0\\}$.

  * •

For all $x\in\mathbb{R}$, $[x]$ denotes the integer part of $x$, $\lceil
x\rceil$ denotes the smallest integer $\geq x$ and $\lfloor x\rfloor$ denotes
the largest integer $\leq x$.

  * •

$\|x\|=\underset{1\leq i\leq d}{\max}|x_{i}|$,
$|x|_{0}=\sum_{i=1}^{d}\textrm{\dsrom{1}}(x_{i}\neq 0)$ for all
$x=(x_{1},\cdots,x_{d})^{T}\in\mathbb{R}^{d}$.

  * •

$\|x\|=\underset{1\leq i\leq p}{\max}\sum_{j=1}^{q}|x_{i,j}|$ for any matrix
$x=(x_{i,j})\in M_{p,q}(\mathbb{R})$; where $M_{p,q}(\mathbb{R})$ denotes the
set of matrices of dimension $p\times q$ with coefficients in $\mathbb{R}$.

  * •

For any function $h:E_{1}\rightarrow E_{2}$ and $U\subseteq E_{1}$,

$\|h\|_{\infty}=\sup_{x\in
E_{1}}\|h(x)\|_{E_{2}},~{}\|h\|_{\infty,U}=\sup_{x\in U}\|h(x)\|_{E_{2}}\text{
and }$ $\mathrm{Lip}_{\alpha}(h)\coloneqq\underset{x_{1},x_{2}\in
E_{1},~{}x_{1}\neq
x_{2}}{\sup}\dfrac{\|h(x_{1})-h(x_{2})\|_{E_{2}}}{\|x_{1}-x_{2}\|^{\alpha}_{E_{1}}}\text{
for any }\alpha\in[0,1].$

  * •

For any $\mathcal{K}>0$ and $\alpha\in[0,1]$,
$\Lambda_{\alpha,\mathcal{K}}(E_{1},E_{2})$ (simply
$\Lambda_{\alpha,\mathcal{K}}(E_{1})$ when $E_{2}\subseteq\mathbb{R}$) denotes
the set of functions $h:E_{1}^{u}\rightarrow E_{2}$ for some $u\in\mathbb{N}$,
such that $\|h\|_{\infty}<\infty$ and
$\mathrm{Lip}_{\alpha}(h)\leq\mathcal{K}$. When $\alpha=1$, we set
$\mathrm{Lip}_{1}(h)=\mathrm{Lip}(h)$ and
$\Lambda_{1}(E_{1})=\Lambda_{1,1}(E_{1},\mathbb{R})$.

  * •

$\mathcal{F}(E_{1},E_{2})$ denotes the set of measurable functions from
$E_{1}$ to $E_{2}$.

  * •

For any $h\in\mathcal{F}(E_{1},E_{2})$ and $\epsilon>0$, $B(h,\epsilon)$
denotes the ball of radius $\epsilon$ of $\mathcal{F}(E_{1},E_{2})$ centered
at $h$, that is,
$B(h,\epsilon)=\big{\\{}f\in\mathcal{F}(E_{1},E_{2}),~{}\|f-h\|_{\infty}\leq\epsilon\big{\\}}$.

  * •

For any $\mathcal{H}\subset\mathcal{F}(E_{1},E_{2})$, the $\epsilon$-covering
number $\mathcal{N}(\mathcal{H},\epsilon)$ of $\mathcal{H}$ is the minimal
number of balls of radius $\epsilon$ needed to cover $\mathcal{H}$; that is,

$\mathcal{N}(\mathcal{H},\epsilon)=\inf\Big{\\{}m\geq 1~{}:\exists
h_{1},\cdots,h_{m}\in\mathcal{H}\text{ such that
}\mathcal{H}\subset\bigcup_{i=1}^{m}B(h_{i},\epsilon)\Big{\\}}.$

  * •

For any $U\subset E_{1}$, $\overline{U}$ denotes the closure of $U$.

  * •

For any function $g:\mathbb{R}\rightarrow\mathbb{R}$, $g^{\prime}$ and
$g^{\prime\prime}$ denote the first and second order derivatives of $g$. Also,
for all $x\in\mathbb{R}$, we set $g^{\prime}(x+):=\lim_{\epsilon\downarrow
0}\big{(}g(x+\epsilon)-g(x)\big{)}/\epsilon$ and
$g^{\prime}(x-):=\lim_{\epsilon\downarrow
0}\big{(}g(x-\epsilon)-g(x)\big{)}/\epsilon$.

  * •

For any bounded set $U\subset E_{1}$, $\|U\|=\sup_{x\in U}\|x\|$.

  * •

For all $x=(x_{1},\cdots,x_{d})\in\mathbb{R}^{d}$ and
$\beta=(\beta_{1},\cdots,\beta_{d})\in\mathbb{N}_{0}^{d}$,
$x^{\beta}=x_{1}^{\beta_{1}}\cdot\ldots\cdot x_{d}^{\beta_{d}}$ and
$\beta!=\beta_{1}!\cdot\ldots\cdot\beta_{d}!$.

###  2.2 Some assumptions and weak dependence

Let us set the following assumptions on the input space $\mathcal{X}$ and the
loss function $\ell$.

  1. (A1):

$\mathcal{X}\subset\mathbb{R}^{{d_{x}}}$ is a compact set.
Under (A1), we set

$R:=\max(1,4\|{\mathcal{X}}\|).$ (2.1)

  2. (A2):

There exists $\mathcal{K}_{\ell}>0$ such that,
$\ell\in\Lambda_{1,\mathcal{K}_{\ell}}(\mathbb{R}^{d_{y}}\times\mathcal{Y})$.

For all $L,N,S,B,F>0$, we set

$M_{L,N,S,B,F}:=\max\Big{(}\sup_{h\in\mathcal{H}_{\sigma,{d_{x}},{d_{y}}}}\sup_{z\in\mathcal{Z}}\ell(h,z),1\Big{)}\text{
and
}G_{L,N,S,B,F}:=\sup_{h_{1},h_{2}\in\mathcal{H}_{\sigma,{d_{x}},{d_{y}}},h_{1}\neq
h_{2}}\sup_{z\in\mathcal{Z}}\frac{|\ell(h_{1},z)-\ell(h_{2},z)|}{\|h_{1}-h_{2}\|_{\infty}},$
(2.2)

with
$\mathcal{H}_{\sigma,{d_{x}},{d_{y}}}=\mathcal{H}_{\sigma,{d_{x}},{d_{y}}}(L,N,S,B,F)$.
When it is clear in the context and no confusion can arise, we use $M,G$ for
$M_{L,N,S,B,F}$ and $G_{L,N,S,B,F}$ respectively. Under the assumption (A1)
and if ${\mathcal{Y}}$ is bounded, we have,

$\widetilde{M}_{F}:=\sup_{(y_{1},y_{2})\in
B\big{(}\max(F,\|{\mathcal{Y}}\|)\big{)}}\ell(y_{1},y_{2})<\infty,$ (2.3)

where
$B\big{(}\max(F,\|{\mathcal{Y}}\|)\big{)}\subset\mathbb{R}^{d_{y}}\times\mathbb{R}^{d_{y}}$
is the ball of radius $\max(F,\|{\mathcal{Y}}\|)$, centered at 0. Let
$z_{0}=(x_{0},y_{0})$ fixed. In addition to (A2), we have for all
$z=(x,y)\in{\mathcal{Z}}$ and $h\in\mathcal{H}_{\sigma,{d_{x}},{d_{y}}}$,

$\displaystyle\ell(h,z)$
$\displaystyle\leq\ell(h,z_{0})+|\ell(h,z)-\ell(h,z_{0})|\leq\ell(h(x_{0}),y_{0})+|\ell(h(x),y)-\ell(h(x_{0}),y_{0})|$
$\displaystyle\leq\widetilde{M}_{F}+{\mathcal{K}}_{\ell}(\|x-x_{0}\|+\|y-y_{0}\|)\leq\widetilde{M}_{F}+2{\mathcal{K}}_{\ell}(\|{\mathcal{X}}\|+\|{\mathcal{Y}}\|).$
(2.4)

In this case, we get,

$M_{L,N,S,B,F}\leq\max\big{(}\widetilde{M}_{F}+2{\mathcal{K}}_{\ell}(\|{\mathcal{X}}\|+\|{\mathcal{Y}}\|),1\big{)}<\infty,$
(2.5)

and this bound does not depend on $L,N,S,B$. Also, under (A2), we get for all
$h_{1},h_{2}\in\mathcal{H}_{\sigma,{d_{x}},{d_{y}}}$ with $h_{1}\neq h_{2}$
and $z=(x,y)\in{\mathcal{Z}}$ such that $h_{1}(x)\neq h_{2}(x)$,

$\frac{|\ell(h_{1},z)-\ell(h_{2},z)|}{\|h_{1}-h_{2}\|_{\infty}}=\frac{|\ell(h_{1}(x),y)-\ell(h_{2}(x),y)|}{\|(h_{1}(x),y)-(h_{2}(x),y)\|}\times\frac{\|(h_{1}(x),y)-(h_{2}(x),y)\|}{\|h_{1}-h_{2}\|_{\infty}}\leq{\mathcal{K}}_{\ell}.$
(2.6)

Hence, $G_{L,N,S,B,F}\leq{\mathcal{K}}_{\ell}$ and this bound does not depend
on $L,N,S,B,F$.

Let us give the definition of the weak dependence in a general context, see
[8] and [4]. Let $E$ be a separable Banach space.

###### Definition 2.1

An $E$-valued process $(Z_{t})_{t\in\mathbb{Z}}$ is said to be
$(\Lambda_{1}(E),\psi,\epsilon)$-weakly dependent if there exists a function
$\psi:[0,\infty)^{2}\times\mathbb{N}^{2}\rightarrow[0,\infty)$ and a sequence
$\epsilon=(\epsilon(r))_{r\in\mathbb{N}}$ decreasing to zero at infinity such
that, for any $g_{1},g_{2}\in\Lambda_{1}(E)$ with
$g_{1}:E^{u}\rightarrow\mathbb{R}$, $g_{2}:E^{v}\rightarrow\mathbb{R}$
($u,v\in\mathbb{N}$) and for any $u$-tuple $(s_{1},\cdots,s_{u})$ and any
$v$-tuple $(t_{1},\cdots,t_{v})$ with $s_{1}\leq\cdots\leq s_{u}\leq
s_{u}+r\leq t_{1}\leq\cdots\leq t_{v}$, the following inequality is fulfilled:

$\left|\mathrm{Cov}\left(g_{1}(Z_{s_{1}},\cdots,Z_{s_{u}}),g_{2}(Z_{t_{1}},\cdots,Z_{t_{v}})\right)\right|\leq\psi\left(\mathrm{Lip}(g_{1}),\mathrm{Lip}(g_{2}),u,v\right)\epsilon(r).$

The following choices of $\psi$ (see also [4]) are well known examples.

  * •

$\psi\left(\mathrm{Lip}(g_{1}),\mathrm{Lip}(g_{2}),u,v\right)=v\mathrm{Lip}(g_{2})$:
the $\theta$-weak dependence, then denote $\epsilon(r)=\theta(r)$;

  * •

$\psi\left(\mathrm{Lip}(g_{1}),\mathrm{Lip}(g_{2}),u,v\right)=u\mathrm{Lip}(g_{1})+v\mathrm{Lip}(g_{2})$:
the $\eta$-weak dependence, then denote $\epsilon(r)=\eta(r)$;

  * •

$\psi\left(\mathrm{Lip}(g_{1}),\mathrm{Lip}(g_{2}),u,v\right)=uv\mathrm{Lip}(g_{1})\cdot\mathrm{Lip}(g_{2})$:
the $\kappa$-weak dependence, then denote $\epsilon(r)=\kappa(r)$;

  * •

$\psi\left(\mathrm{Lip}(g_{1}),\mathrm{Lip}(g_{2}),u,v\right)=u\mathrm{Lip}(g_{1})+v\mathrm{Lip}(g_{2})+uv\mathrm{Lip}(g_{1})\cdot\mathrm{Lip}(g_{2})$:
the $\lambda$-weak dependence, then denote $\epsilon(r)=\lambda(r)$.

Let us set now the weak dependence assumption.

  1. (A3):

The process $\\{Z_{t}=(X_{t},Y_{t}),~{}t\in\mathbb{Z}\\}$ is stationary
ergodic and $(\Lambda_{1}(\mathcal{Z}),\psi,\epsilon)$-weakly dependent with
$\epsilon_{r}=\mathcal{O}(r^{-\gamma})$ for some $\gamma>3$.

Numerous classical models satisfy (A3); for example, ARMAX, TARX, GARCH-X,
ARMAX-GARCH, APARCH-X ([10]), multivariate INGARCH (with exponential family
conditional distribution), see for instance [7], [5]. Other examples such as
APARCH-X$(\delta,\infty)$, ARX($\infty$)-ARCH($\infty$) introduced by [6] also
satisfy (A3), see [7].

##  3 Hölder smooth functions approximation by DNNs

Let $U\subseteq\mathbb{R}^{d_{x}}$. For any
$\beta=(\beta_{1},\cdots,\beta_{d_{x}})^{T}\in\mathbb{N}^{d_{x}}$ and
$x=(x_{1},\cdots,x_{d_{x}})^{T}\in U$, we set

$|\beta|=\sum_{i=i}^{d_{x}}\beta_{i}\text{ and
}\partial^{\beta}=\dfrac{\partial^{|\beta|}}{\partial
x_{1}^{\beta_{1}}\cdots\partial x_{d_{x}}^{\beta_{d_{x}}}}.$

For any $s>0$, the Hölder space $\mathcal{C}^{s}(U)$ is a set of vector valued
functions $h:U\rightarrow\mathbb{R}^{d_{y}}$ such that, for any
$\beta\in\mathbb{N}^{d_{x}}$ with $|\beta|\leq[s]$,
$\|\partial^{\beta}h\|_{\infty}<\infty$ and for any
$\beta\in\mathbb{N}^{d_{x}}$ with $|\beta|=[s]$,
$Lip_{s-[s]}(\partial^{\beta}h)<\infty$. This space is equipped with the norm

$\|h\|_{\mathcal{C}^{s}(U)}=\sum_{0\leq|\beta|\leq[s]}\|\partial^{\beta}h\|_{\infty}+\sum_{|\beta|=[s]}Lip_{s-[s]}(\partial^{\beta}h).$

For $s>0$, $U\subseteq\mathbb{R}^{d_{x}}$, $\mathcal{C}^{s}(\overline{U})$
with the norm $\|\cdot\|_{\mathcal{C}^{s}(\overline{U})}$ is a Banach space
(see for example [24], [9]). For any $s>0$, $U\subseteq\mathbb{R}^{d_{x}}$ and
$\mathcal{K}>0$, set

$\mathcal{C}^{s,\mathcal{K}}(U)=\big{\\{}h\in\mathcal{C}^{s}(U),~{}\|h\|_{\mathcal{C}^{s}(U)}\leq\mathcal{K}\big{\\}}.$

Let us consider the following definition, see also [18], [19].

###### Definition 3.1

Let a function $g:\mathbb{R}\rightarrow\mathbb{R}$.

  1. 1.

$g$ is continuous piecewise linear (or ”piecewise linear” for notational
simplicity) if it is continuous and there exists $K$ ($K\in\mathbb{N}$) break
points $a_{1},\cdots,a_{K}\in\mathbb{R}$ with $a_{1}\leq a_{2}\leq\cdots\leq
a_{K}$ such that, for any $k=1,\cdots,K$, $g^{\prime}(a_{k}-)\neq
g^{\prime}(a_{k}+)$ and $g$ is linear on
$(-\infty,a_{1}],[a_{1},a_{2}],\cdots[a_{K},\infty)$.

  2. 2.

$g$ is locally quadratic if there exits an interval $(a,b)$ on which $g$ is
three times continuously differentiable with bounded derivatives and there
exists $t\in(a,b)$ such that $g^{\prime}(t)\neq 0$ and
$g^{\prime\prime}(t)\neq 0$.

The classical activation functions $\sigma(z)=\max(z,0)$ (ReLU) is piecewise
linear and $\sigma(z)=1/(1+e^{-z})$ (sigmoid) is locally quadratic (see [18]).
Consider the following assumption on the activation function $\sigma$.

  1. (A4):

There exists $\mathcal{K}_{\sigma}>0$ such that
$\sigma\in\Lambda_{1,\mathcal{K}_{\sigma}}(\mathbb{R})$. Moreover, $\sigma$ is
either piecewise linear or locally quadratic and fixes a segment
$I\subseteq[0,1]$.

Recall that, $\sigma$ fixes the segment $I$ if $\sigma(z)=z$ for all $z\in I$.
Here are some examples of activation functions satisfying (A4).

  * •

ReLU: $\sigma(z)=\max(z,0)$.

  * •

Leaky ReLU: $\sigma(z)=\max(z,az)$, for $a\in(0,1)$.

  * •

Exponential linear unit (ELU) [3]:
$\sigma(z)=a(e^{z}-1)\textrm{\dsrom{1}}(z\leq 0)+z\textrm{\dsrom{1}}(z>0)$,
for $a>0$.

  * •

Inverse square root linear unit (ISRLU) [2]:
$\sigma(z)=\frac{z}{\sqrt{1+az^{2}}}\textrm{\dsrom{1}}(z\leq
0)+z\textrm{\dsrom{1}}(z>0)$, for $a>0$.

  * •

SignReLu [16]:
$\sigma(z)=a\frac{z}{1-z}\textrm{\dsrom{1}}(z<0)+z\textrm{\dsrom{1}}(z\geq
0)$, for $a\in(0,1)$.

The following theorem provides an approximation of any Hölder smooth function
by DNNs with a large class of activation functions.

###### Theorem 3.2

Assume that (A1) and (A4) hold. Let $s,\mathcal{K}>0$ and
$h:\mathcal{X}\rightarrow\mathbb{R}^{d_{y}}$ with
$h\in\mathcal{C}^{s,\mathcal{K}}(\mathcal{X})$. There exist
$L_{0},N_{0},S_{0},B_{0}>0$ depending only on $d_{x},d_{y},s,\mathcal{K}$, $R$
(defined at (2.1)) and $\sigma$ such that, for all $\epsilon>0$, there exists
a neural network
$h_{\epsilon}\in\mathcal{H}_{\sigma,{d_{x}},{d_{y}}}(L_{0}\log_{+}(1/\epsilon),N_{0}\epsilon^{-{d_{x}}/s},S_{0}\epsilon^{-{d_{x}}/s}\log_{+}(1/\epsilon),B_{0}\epsilon^{-4(d_{x}/s+1})$
satisfying,

$\|h-h_{\epsilon}\|_{\infty,\mathcal{X}}\leq\epsilon,$ (3.1)

where $\log_{+}x=\max(1,\log x)$ for all $x>0$.

Theorem 3.2 is an extension of Theorem 1 of [18] to vector valued functions
defined on any compact subset of $\mathbb{R}^{d_{x}}$. This theorem also
extends Theorem 1 of [21] to deep neural networks with a broad class of
activation functions.

##  4 Excess risk bound

Consider the learning problem presented in the introduction, based on the
observations $D_{n}=\\{Z_{1}=(X_{1},Y_{1}),\cdots,Z_{n}=(X_{n},Y_{n})\\}$
which is a trajectory of a stationary and ergodic process
$\\{Z_{t}=(X_{t},Y_{t}),~{}t\in\mathbb{Z}\\}$, taking values in
$\mathcal{Z}=\mathcal{X}\times\mathcal{Y}$, with
$\mathcal{Y}\subset\mathbb{R}$, that is, $d_{y}=1$. We focus on the class of
DNNs $\mathcal{H}_{\sigma,{d_{x}},1}(L_{n},N_{n},S_{n},B_{n},F_{n})$ (defined
at (1.6)), for some $L_{n},N_{n},S_{n},B_{n},F_{n}>0$. Set for the sequel,
$\mathcal{H}_{\sigma,n}:=\mathcal{H}_{\sigma,{d_{x}},1}(L_{n},N_{n},S_{n},B_{n},F_{n})$.
Also, denote,

$M_{n}:=M_{L_{n},N_{n},S_{n},B_{n},F_{n}},~{}G_{n}:=G_{L_{n},N_{n},S_{n},B_{n},F_{n}}\text{
and }\widetilde{M}_{n}:=M_{F_{n}},$ (4.1)

where $M_{L_{n},N_{n},S_{n},B_{n},F_{n}}$, $G_{L_{n},N_{n},S_{n},B_{n},F_{n}}$
and $M_{F_{n}}$ are defined in (2.2) and (2.3). The neural network obtained
from the empirical risk minimization (ERM) algorithm is given by,

$\widehat{h}_{n}=\underset{h\in\mathcal{H}_{\sigma,n}}{\operatorname*{argmin}}\widehat{R}_{n}(h),$
(4.2)

where $\widehat{R}_{n}$ is defined at (1.3). The excess risk is given by
$R(\widehat{h}_{n})-\inf_{h\in\mathcal{F}(\mathcal{X},\mathcal{Y})}R(h)$,
where $R$ is defined at (1.1). This excess risk has the following well known
decomposition,

$R(\widehat{h}_{n})-\underset{h\in\mathcal{F}(\mathcal{X},\mathcal{Y})}{\inf}R(h)=\underset{\text{Estimation
error}}{\underbrace{R(\widehat{h}_{n})-\underset{h\in\mathcal{H}_{\sigma,n}}{\inf}R(h)}}+\underset{\text{Approximation
error}}{\underbrace{\underset{h\in\mathcal{H}_{\sigma,n}}{\inf}R(h)-\underset{h\in\mathcal{F}(\mathcal{X},\mathcal{Y})}{\inf}R(h)}}.$
(4.3)

In the sequel, we consider the target neural network
$h_{\mathcal{H}_{\sigma,n}}\in\mathcal{H}_{\sigma,n}$, assumed to exist and
satisfies,

$R(h_{\mathcal{H}_{\sigma,n}})=\underset{h\in\mathcal{H}_{\sigma,n}}{\inf}R(h).$
(4.4)

A major concern in statistical learning theory is to solve the problem of the
best trade-off between the estimation and approximation errors. This is done
by taking into account the complexity of the class of hypothesis functions,
which must not be too large, nor too small.

In this section, we focus on the convergence rate of the excess risk, and set
an additional assumption.

  1. (A5):

There exists a predictor $h^{*}\in\mathcal{F}(\mathcal{X},\mathcal{Y})$,
satisfying $R(h^{*})=\inf_{h\in\mathcal{F}(\mathcal{X},\mathcal{Y})}R(h)$ and
$h^{*}\in\mathcal{C}^{s,\mathcal{K}}(\mathcal{X})$ for some
$s,{\mathcal{K}}>0$.

The following theorem provides a bound of the excess risk.

###### Theorem 4.1

Assume that (A1)-(A5) hold. Let $\eta,\nu\in(0,1)$ and $\alpha>2+d_{x}/s$.
Then, there exist universal constants $L_{0},N_{0},S_{0},B_{0}>0$ and
$n_{0}\geq 1$, depending on
${\mathcal{K}}_{\ell},C_{\sigma},d_{x},\eta,\nu,\alpha,s,L_{0},N_{0},S_{0},B_{0}$,
such that, for any $n\geq n_{0}$, $F_{n}>0$, if for the DNN’s class

$\mathcal{H}_{\sigma,n}=\mathcal{H}_{\sigma,{d_{x}},1}\Big{(}(L_{0}/\alpha)\log
n,N_{0}n^{d_{x}/(s\alpha)},(S_{0}/\alpha)n^{d_{x}/(s\alpha)}\log
n,n^{4(d_{x}/s+1)/\alpha},F_{n}\Big{)},$

$M_{n}<\infty$, then, we have with probability at least $1-\eta$,

$R(\widehat{h}_{n})-\underset{h\in\mathcal{F}(\mathcal{X},\mathcal{Y})}{\inf}R(h)\leq\dfrac{2M_{n}+{\mathcal{K}}_{\ell}}{n^{1/\alpha}}+\Bigg{(}\frac{\log(2C_{1}\log
n/\eta)}{C_{n,2}}\Bigg{)}^{1/2}$ (4.5)

for some constant $C_{1}>0$, where $\widehat{h}_{n}$ is defined in (4.2),
$M_{n}$ is given in (4.1) and (2.2), and $C_{n,2}$ is given in (5.21).

###### Remark 4.2

If ${\mathcal{Y}}$ is bounded, and by choosing a constant sequence
$F_{n}=F>0$, then, from (2.5), (4.1) and (4.5), we get,

$R(\widehat{h}_{n})-\underset{h\in\mathcal{F}(\mathcal{X},\mathcal{Y})}{\inf}R(h)\leq\dfrac{2\max\big{(}\widetilde{M}_{F}+2{\mathcal{K}}_{\ell}(\|{\mathcal{X}}\|+\|{\mathcal{Y}}\|),1\big{)}+{\mathcal{K}}_{\ell}}{n^{1/\alpha}}+\Bigg{(}\frac{\log(2C_{1}\log
n/\eta)}{C_{n,2}}\Bigg{)}^{1/2}.$ (4.6)

Which shows that, the convergence rate of the excess risk is less than
$\mathcal{O}(n^{-1/\alpha})$ for all $\alpha>2+d_{x}/s$. This rate is close to
$\mathcal{O}(n^{-1/2})$ when $s\gg d_{x}$. Note that, the rate obtained in
[18] is close to $\mathcal{O}(1/n)$ when $s\gg d_{x}$. But, these findings are
in specific cases of regression, classification, independent observations and
with an input space ${\mathcal{X}}=[0,1]^{d_{x}}$. The results obtained here
are in a general setting and for a large class of dependent processes.

##  5 Proofs of the main results

###  5.1 Proof of Theorem 3.2

First of all, remark the parallelization property for two networks of the same
depth: for any $d_{1},d_{2}\in\mathbb{N}$ and
$L,N_{1},N_{2},S_{1},S_{2},B_{1},B_{2}>0$,

$h_{1}\in\mathcal{H}_{\sigma,{d_{x}},d_{1}}(L,N_{1},S_{1},B_{1}),~{}h_{2}\in\mathcal{H}_{\sigma,{d_{x}},d_{2}}(L,N_{2},S_{2},B_{2})\Rightarrow(h_{1},h_{2})\in\mathcal{H}_{\sigma,{d_{x}},d_{1}+d_{2}}(L,N_{1}+N_{2},S_{1}+S_{2},B_{1}+B_{2}).$

Thus, it suffices to prove the theorem with $d_{y}=1$. To do this, we split
the proof into two parts, as in the proof of Theorem 1 in [21]. Let
$s,\mathcal{K},\epsilon>0$ and $h\in\mathcal{C}^{s,\mathcal{K}}(\mathcal{X})$.
Without loss of generality, we assume that $\epsilon<1$ and that, $\sigma$
fixes the segment $I=[1/4,3/4]$.

(1) Assume that $\mathcal{X}\subset[1/4,3/4]^{d_{x}}$.
(i) Case of piecewise linear activation functions.
Let $\rho$ be the ReLU activation function. From Theorem 1 in [21], for any
$m\geq 1$, $M\geq\max\\{5^{d_{x}},(s+1)^{d_{x}},(\mathcal{K}+1)e^{d_{x}}\\}$,
there exists a network
$\widetilde{h}_{1}\in\mathcal{H}_{\rho,{d_{x}},1}(L_{1},N_{1},S_{1},1)$ with
$L_{1}=9+(m+5)(1+\lceil\log_{2}\max(d_{x},s)\rceil)$, $N_{1}=6(d_{x}+\lceil
s\rceil)M$ and $S_{1}\leq 142(d_{x}+s+1)^{3+d_{x}}M(m+6)$, such that,

$\|\widetilde{h}_{1}-h\|_{\infty,\mathcal{X}}\leq(2\mathcal{K}R^{s}+1)(1+d_{x}^{2}+s^{2})6^{d_{x}}M2^{-m}+\mathcal{K}(9R)^{s}M^{-s/d_{x}}.$
(5.1)

From Lemma A1 of [18], there exists a network, with the activation function
$\sigma$, that produces the same output of this ReLU neural network; that is,
there exists
$\widetilde{h}_{2}\in\mathcal{H}_{\sigma,{d_{x}},1}(L_{2},N_{2},S_{2},B_{2})$
satisfying,

$\|\widetilde{h}_{1}-\widetilde{h}_{2}\|_{\infty,\mathcal{X}}=0,$

with $L_{2}=L_{1}$, $N_{2}=2N_{1}$, $S_{2}=4S_{1}+2L_{1}N_{1}+1$ and for some
$B_{2}>0$ depending on $\sigma$. Therefore, we get from (5.1)

$\|h-\widetilde{h}_{2}\|_{\infty,\mathcal{X}}\leq(2\mathcal{K}R^{s}+1)(1+d_{x}^{2}+s^{2})6^{d_{x}}M2^{-m}+\mathcal{K}(9R)^{s}M^{-s/d_{x}}.$
(5.2)

By taking
$M=\max\Big{\\{}5^{d_{x}},(s+1)^{d_{x}},(\mathcal{K}+1)e^{d_{x}},(9R)^{d_{x}}(2{\mathcal{K}})^{d_{x}/s}\epsilon^{-d_{x}/s}\Big{\\}}$
and
$m=\max\Big{\\{}\log_{2}\Big{(}2(2{\mathcal{K}}R^{s}+1)(1+d_{x}^{2}+s^{2})6^{d_{x}}M\epsilon^{-1}\Big{)},1\Big{\\}}$
in (5.2), we get

$\|h-\widetilde{h}_{2}\|_{\infty,\mathcal{X}}\leq\epsilon.$ (5.3)

For these choices of $M$ and $m$, we have, $M\leq
5^{d_{x}}+(s+1)^{d_{x}}+(\mathcal{K}+1)e^{d_{x}}+9^{d_{x}}(2{\mathcal{K}})^{d_{x}/s}\epsilon^{-d_{x}/s}$
and $m\leq
1+\log_{2}\Big{(}2({\mathcal{K}}+1)(1+d_{x}^{2}+s^{2})6^{d_{x}}M\epsilon^{-1}\Big{)}$.
Hence, one can easily see that:

$L_{2}=L_{1}=9+(m+5)(1+\lceil\log_{2}\max(d_{x},s)\rceil)\leq
L_{0}\log(1/\epsilon),~{}N_{2}=2N_{1}=12(d_{x}+\lceil s\rceil)M\leq
N_{0}\epsilon^{-d_{x}/s},$ $S_{2}=4S_{1}+2L_{1}N_{1}+1\leq
568(d_{x}+s+1)^{3+d_{x}}M(m+6)+2L_{0}\big{(}\log(1/\epsilon)\big{)}N_{0}\epsilon^{-d_{x}/s}+1\leq
S_{0}\epsilon^{-d_{x}/s}\log(1/\epsilon),$

and $B_{2}\leq B_{0}\epsilon^{-4(d_{x}/s+1)}$ for some
$L_{0},N_{0},S_{0},B_{0}>0$ depending on $d_{x},s,{\mathcal{K}},R$ and
$\sigma$. Thus, the result holds for piecewise linear activation functions and
when $\mathcal{X}\subset[1/4,3/4]^{d_{x}}$.

(ii) Case of locally quadratic activation functions.
Let $\mathfrak{M}\in\mathbb{N}$. Consider the grid points with length
$1/\mathfrak{M}$ inside the $d_{x}$-dimensional unit hypercube
$[0,1]^{d_{x}}$,

$\mathbb{G}_{d_{x},\mathfrak{M}}:=\Big{\\{}\dfrac{1}{\mathfrak{M}}(m_{1},\cdots,m_{d_{x}}),~{}m_{j}\in\\{0,1,\cdots\mathfrak{M}\\},j=1,\cdots,d_{x}\Big{\\}}.$
(5.4)

For all $z\in{\mathcal{X}}\cap\mathbb{G}_{d_{x},\mathfrak{M}}$, denote by
$P^{s}_{z,\mathfrak{M}}h(\cdot)$ the $\lfloor s\rfloor$-th order Taylor
polynomial of $h$ around $\boldsymbol{z}$, that is

$P^{s}_{z}h(x)=\sum_{\beta\in\mathbb{N}_{0}^{d_{x}},|\beta|\leq s}(\partial
f)(z)\dfrac{(x-z)^{\beta}}{\beta!}.$ (5.5)

When $z\in[0,1]^{d_{x}}\setminus{\mathcal{X}}$, set
$z^{*}=\underset{\widetilde{z}\in\operatorname*{argmin}_{u\in{\mathcal{X}}\cap\mathbb{G}_{d_{x},\mathfrak{M}}}(\|z-u\|)}{\operatorname*{argmin}}\|\widetilde{z}\|$.
One can easily check that, $\boldsymbol{z}^{*}$ exists and is unique.
Therefore, set

$P^{s}_{z}h(x)=P^{s}_{z^{*}}h(x).$ (5.6)

Define for all
$x=(x_{1},\ldots,x_{d_{x}})^{\prime},z=(z_{1},\ldots,z_{d_{x}})^{\prime}\in\mathbb{G}_{d_{x},\mathfrak{M}}$,

$P^{s}h(x)=\sum_{z\in\mathbb{G}_{d_{x},\mathfrak{M}}}P^{s}_{z}h(x)\prod_{j=1}^{d_{x}}\big{(}1-\mathfrak{M}|x_{j}-z_{j}|\big{)}_{+}.$
(5.7)

For $\mathfrak{M}\geq 4$, we get from the proof of Theorem 1 in [21],

$\|P^{s}h-h\|_{\infty,{\mathcal{X}}}\leq{\mathcal{K}}3^{s}\mathfrak{M}^{-s}.$
(5.8)

According to the proof of Theorem 1 in [18], for some
$L_{3},N_{3},S_{3},B_{3}>0$, one can find a neural network,

$\widetilde{h}_{3}\in\mathcal{H}_{\sigma,{d_{x}},1}\Big{(}L_{3}\log(1/\epsilon),N_{3}\epsilon^{-d_{x}/s},S_{3}\epsilon^{-d_{x}/s}\log(1/\epsilon),B_{3}\epsilon^{-4(d_{x}/s+1)}\Big{)}$

such that,

$\|P^{s}h-\widetilde{h}_{3}\|_{\infty,\mathcal{X}}\leq C_{1}\epsilon~{}\text{
for some }C_{1}>0.$ (5.9)

By taking $\mathfrak{M}=[\epsilon^{-1/s}]$, the result follows from (5.8) and
(5.9).

(2) The general case of any compact set ${\mathcal{X}}$.
As in the proof of Theorem 1 in [21], define the affine map,

$T:\mathbb{R}^{d_{x}}\rightarrow\mathbb{R}^{d_{x}},~{}x\mapsto
x/R+(1/2,\ldots,1/2)^{T},$ (5.10)

where $R$ is defined in (2.1). By setting
${\mathcal{X}}^{\prime}:=T({\mathcal{X}})$, one can see that
${\mathcal{X}}^{\prime}\subseteq[1/4,3/4]^{d_{x}}$. Also, one can easily
verify that, since $h\in\mathcal{C}^{s,\mathcal{K}}({\mathcal{X}})$, then,
$g:=h(T^{-1})\in\mathcal{C}^{s,R^{s}{\mathcal{K}}}({\mathcal{X}}^{\prime})$.
Therefore, let us apply the result of the first part to
$g\in\mathcal{C}^{s,R^{s}{\mathcal{K}}}({\mathcal{X}}^{\prime})$. So, for some
$L_{0},N_{0},S_{0},B_{0}>0$ depending only on $d_{x},s,\mathcal{K}$ and $R$,
there exists a neural network
$\widetilde{g}\in\mathcal{H}_{\sigma,{d_{x}},1}(L_{0}\log_{+}(1/\epsilon),N_{0}\epsilon^{-{d_{x}}/s},S_{0}\epsilon^{-{d_{x}}/s}\log_{+}(1/\epsilon),B_{0}\epsilon^{-4(d_{x}/s+1)})$
satisfying,

$\|g-\widetilde{g}\|_{\infty,{\mathcal{X}}^{\prime}}\leq\epsilon.$ (5.11)

Now, consider the neural network,

$\widetilde{h}:=\widetilde{g}\circ\sigma_{0}\circ T,\text{ where
}\sigma_{0}(z_{1}\ldots,z_{d_{x}})=\big{(}\sigma(z_{1}),\ldots,\sigma(z_{d_{x}})\big{)}\text{
for all }(z_{1}\ldots,z_{d_{x}})\in\mathbb{R}^{d_{x}}.$ (5.12)

Since $T$ is an affine transformation, the neural network $\widetilde{h}$ is
obtained from $\widetilde{g}$ by adding one hidden layer, which also adds
$2{d_{x}}$ non-zero parameters. That is,

$\widetilde{h}\in\mathcal{H}_{\sigma,{d_{x}},1}\Big{(}(L_{0}+1)\log_{+}(1/\epsilon),N_{0}\epsilon^{-{d_{x}}/s},(S_{0}+2d_{x})\epsilon^{-{d_{x}}/s}\log_{+}(1/\epsilon),\max(B_{0},1)\epsilon^{-4(d_{x}/s+1)}\Big{)}.$
(5.13)

Also, $T({\mathcal{X}})={\mathcal{X}}^{\prime}\subseteq[1/4,3/4]$ and, since
this set is fixed by $\sigma$, we get from (5.11),

$\|h-\widetilde{h}\|_{\infty,{\mathcal{X}}}=\|g\circ
T-\widetilde{g}\circ\sigma_{0}\circ
T\|_{\infty,{\mathcal{X}}}\leq\|g-\widetilde{g}\|_{\infty,{\mathcal{X}}^{\prime}}\leq\epsilon.$
(5.14)

Thus, the result holds from (5.13) and (5.14). This completes the proof of the
theorem.

$\blacksquare$

###  5.2 Proof of Theorem 4.1

Let $L_{n},N_{n},S_{n},B_{n},F_{n}>0$ and consider the DNN’s class
$\mathcal{H}_{\sigma,n}=\mathcal{H}_{\sigma,{d_{x}},1}(L_{n},N_{n},S_{n},B_{n},F_{n})\subset\mathcal{F}(\mathcal{X},\mathcal{Y})$.
Under (4.4), (A5) and the decomposition (4.3), we get,

$R(\widehat{h}_{n})-\underset{h\in\mathcal{F}(\mathcal{X},\mathcal{Y})}{\inf}R(h)=R(\widehat{h}_{n})-R(h^{*})=\Big{(}R(\widehat{h}_{n})-R(h_{\mathcal{H}_{\sigma,n}})\Big{)}+\Big{(}R(h_{\mathcal{H}_{\sigma,n}})-R(h^{*})\Big{)}.$
(5.15)

Let $\eta,\nu\in(0,1)$ and $\alpha>2+d_{x}/s$. Let us deal first with the
approximation error. Observe that, from the Lipschitz property of $\ell$, it
holds for all $h\in\mathcal{H}_{\sigma,n}$, that,

$R(h)-R(h^{*})=\mathbb{E}_{Z_{0}}\big{[}\ell\big{(}h(X_{0}),Y_{0}\big{)}\big{]}-\mathbb{E}_{Z_{0}}\big{[}\ell\big{(}h^{*}(X_{0}),Y_{0}\big{)}\big{]}\leq{\mathcal{K}}_{\ell}\mathbb{E}_{X_{0}}|h(X_{0})-h^{*}(X_{0})|.$

Hence, we get,

$\displaystyle R(h_{\mathcal{H}_{\sigma,n}})-R(h^{*})$
$\displaystyle=\underset{h\in\mathcal{H}_{\sigma,n}}{\inf}R(h)-R(h^{*})=\underset{h\in\mathcal{H}_{\sigma,n}}{\inf}\Big{(}R(h)-R(h^{*})\Big{)}$
$\displaystyle\leq{\mathcal{K}}_{\ell}\underset{h\in\mathcal{H}_{\sigma,n}}{\inf}\Big{(}\mathbb{E}_{X_{0}}\big{|}h(X_{0})-h^{*}(X_{0})\big{|}\Big{)}$
(5.16)

Since $h^{*}\in\mathcal{C}^{s,\mathcal{K}}(\mathcal{X})$, for
$\epsilon=n^{-1/\alpha}$ in Theorem 3.2, one can find positive constants
$L_{0},N_{0},S_{0},B_{0}$ such that, with

$L_{n}=\frac{L_{0}}{\alpha}\log
n,N_{n}=N_{0}n^{d_{x}/(s\alpha)},S_{n}=\frac{S_{0}}{\alpha}n^{d_{x}/(s\alpha)}\log
n\text{ and }B_{n}=B_{0}n^{4(d_{x}/s+1)/\alpha}$ (5.17)

there is a neural network
$h_{n}\in\mathcal{H}_{\sigma,n}=\mathcal{H}_{\sigma,{d_{x}},1}(L_{n},N_{n},S_{n},B_{n},F_{n})$
satisfying,

$\|h_{n}-h^{*}\|_{\infty,{\mathcal{X}}}\leq\frac{1}{n^{1/\alpha}}.$

Therefore, in addition to (5.2), it holds that,

$R(h_{\mathcal{H}_{\sigma,n}})-R(h^{*})\leq\frac{{\mathcal{K}}_{\ell}}{n^{1/\alpha}}.$
(5.18)

Let us consider now the estimation error over the class
$\mathcal{H}_{\sigma,n}=\mathcal{H}_{\sigma,{d_{x}},1}(L_{n},N_{n},S_{n},B_{n},F_{n})$
where $L_{n},N_{n},S_{n},B_{n}$ are given in (5.17). Under (A1)-(A3), one can
find a constant $C_{n,1}>0$ (see [11]) such that, for all
$h\in\mathcal{H}_{\sigma,n}$,

$\mathbb{E}\Big{[}\Big{(}\sum_{i=1}^{n}\Big{(}\ell(h(X_{i}),Y_{i})-\mathbb{E}[\ell(h(X_{0}),Y_{0})]\Big{)}\Big{)}^{2}\Big{]}\leq
C_{n,1}n.$ (5.19)

With the assumption $\alpha>2+d_{x}/s$, one can easily see that the condition
(4.7) of Theorem 4.3 in [12] holds. From this theorem, there exists a constant
$C_{1}>0$ and $\widetilde{n}_{0}$ depending on
${\mathcal{K}}_{\ell},C_{\sigma},d_{x},\eta,\nu,\alpha,s$ such that, for any
$n\geq\widetilde{n}_{0}$ and with probability at least $1-\eta$, the
estimation error satisfies,

$R(\widehat{h}_{n})-R(h_{\mathcal{H}_{\sigma,n}})\leq\dfrac{2M_{n}}{n^{1/\alpha}}+\Bigg{(}\frac{\log(2C_{1}\log
n/\eta)}{C_{n,2}}\Bigg{)}^{1/2},$ (5.20)

where,

$C_{n,2}=\frac{n^{2}}{nC_{n,1}+\log n\;n^{\nu-1/4}(2M_{n})^{\nu}/C_{n,1}},$
(5.21)

with $C_{n,1}$ defined in (5.19). Thus, the result follows from (5.15), (5.18)
and (5.20). This completes the proof of the theorem.

$\blacksquare$

## References

  * [1] Bauer, B., and Kohler, M. On deep learning as a remedy for the curse of dimensionality in nonparametric regression.  The Annals of Statistics 47, 4 (2019), 2261 – 2285.
  * [2] Carlile, B., Delamarter, G., Kinney, P., Marti, A., and Whitney, B. Improving deep learning by inverse square root linear units (isrlus).  arXiv preprint arXiv:1710.09967 (2017).
  * [3] Clevert, D.-A., Unterthiner, T., and Hochreiter, S. Fast and accurate deep network learning by exponential linear units (elus).  arXiv preprint arXiv:1511.07289 (2015).
  * [4] Dedecker, J., Doukhan, P., Lang, G., León, J. R., Louhichi, S., and Prieur, C. Weak dependence: With Examples and Applications.  Lecture Notes in Statistics 190, Springer-Verlag, New York, 2007\.
  * [5] Diop, M. L., and Kengne, W. A general procedure for change-point detection in multivariate time series.  TEST (2022), 1–33.
  * [6] Diop, M. L., and Kengne, W. Inference and model selection in general causal time series with exogenous covariates.  Electronic Journal of Statistics 16, 1 (2022), 116–157.
  * [7] Diop, M. L., and Kengne, W. Statistical learning for $\psi$-weakly dependent processes.  arXiv preprint arXiv:2210.00088 (2022).
  * [8] Doukhan, P., and Louhichi, S. A new weak dependence condition and applications to moment inequalities.  Stochastic processes and their applications 84, 2 (1999), 313–342.
  * [9] Ern, A., and Guermond, J.-L. Theory and practice of finite elements, vol. 159.  Springer, 2004.
  * [10] Francq, C., et al. Qml inference for volatility models with covariates.  Econometric Theory 35, 1 (2019), 37–72.
  * [11] Hwang, E., and Shin, D. W. A study on moment inequalities under a weak dependence.  Journal of the Korean Statistical Society 42, 1 (2013), 133–141.
  * [12] Kengne, W., and Modou, W. Deep learning for $\psi$-weakly dependent processes.  arXiv preprint arXiv:2302.00333 (2023).
  * [13] Kim, Y., Ohn, I., and Kim, D. Fast convergence rates of deep neural networks for classification.  Neural Networks 138 (2021), 179–197.
  * [14] Kohler, M., and Krzyzak, A. On the rate of convergence of a deep recurrent neural network estimate in a regression problem with dependent data.  arXiv preprint arXiv:2011.00328 (2020).
  * [15] Kurisu, D., Fukami, R., and Koike, Y. Adaptive deep learning for nonparametric time series regression.  arXiv preprint arXiv:2207.02546 (2022).
  * [16] Lin, G., and Shen, W. Research on convolutional neural network based on improved relu piecewise activation function.  Procedia computer science 131 (2018), 977–984.
  * [17] Ma, M., and Safikhani, A. Theoretical analysis of deep neural networks for temporally dependent observations.  arXiv preprint arXiv:2210.11530 (2022).
  * [18] Ohn, I., and Kim, Y. Smooth function approximation by deep neural networks with general activation functions.  Entropy 21, 7 (2019), 627.
  * [19] Ohn, I., and Kim, Y. Nonconvex sparse regularization for deep neural networks and its optimality.  Neural Computation 34, 2 (2022), 476–517.
  * [20] Petersen, P., and Voigtlaender, F. Optimal approximation of piecewise smooth functions using deep relu neural networks.  Neural Networks 108 (2018), 296–330.
  * [21] Schmidt-Hieber, J. Deep relu network approximation of functions on a manifold.  arXiv preprint arXiv:1908.00695 (2019).
  * [22] Schmidt-Hieber, J. Nonparametric regression using deep neural networks with relu activation function.  The Annals of Statistics (2020), 1875 – 1897.
  * [23] Suzuki, T. Adaptivity of deep relu network for learning in besov and mixed smooth besov spaces: optimal rate and curse of dimensionality.  arXiv preprint arXiv:1810.08033 (2018).
  * [24] Triebel, H. Theory of function spaces II, vol. 84 of Monogr. Math., Basel.  Basel: Birkhäuser Verlag, 1992.
  * [25] Yarotsky, D. Error bounds for approximations with deep relu networks.  Neural Networks 94 (2017), 103–114.


# Scattering oblique plane microscopy for in-vivo blood cell imaging

Gregory N. McKay  1 Ryan C. Niemeier  1 Carlos Castro-González  2 and Nicholas
J. Durr1  1Department of Biomedical Engineering, Department of Electrical and
Computer Engineering, Johns Hopkins University
2Leuko Labs Inc., Boston, MA, USA
<EMAIL_ADDRESS>
###### Abstract

Oblique plane microscopy (OPM) enables high speed, volumetric fluorescence
imaging through a single-objective geometry. While these advantages have
positioned OPM as a valuable tool to probe biological questions in animal
models, its potential for in vivo human imaging is largely unexplored due to
its typical use with exogeneous fluorescent dyes. Here we introduce a
scattering-contrast oblique plane microscope (sOPM) and demonstrate label-free
imaging of blood cells flowing through human capillaries in vivo. The sOPM
illuminates a capillary bed in the ventral tongue with an oblique light sheet,
and images side- and back- scattered signal from blood cells. By synchronizing
sOPM with a conventional capillaroscope, we acquire paired widefield and axial
images of blood cells flowing through a capillary loop. The widefield
capillaroscope image provides absorption contrast and confirms the presence of
red blood cells (RBCs), while the sOPM image may aid in determining whether
optical absorption gaps (OAGs) between RBCs have cellular or acellular
composition. Further, we demonstrate consequential differences between
fluorescence and scattering versions of OPM by imaging the same polystyrene
beads sequentially with each technique. Lastly, we substantiate in vivo
observations by imaging isolated red blood cells, white blood cells, and
platelets in vitro using 3D agar phantoms. These results demonstrate a
promising new avenue towards in vivo blood analysis.

††journal: boe††articletype: Research Article

http://durr.jhu.edu/

##  1 Introduction

The recent application of modern microscopy techniques to human capillary
imaging promises non-invasive alternatives to conventional, phlebotomy-based
clinical blood tests. For example, neutropenia screening has recently been
demonstrated non-invasively by imaging nailfold capillaries, relying solely on
intrinsic absorption-based contrast produced by hemoglobin within red blood
cells [1, 2]. Capillaroscopy holds the potential to enable safer, at-home
screening for immunocompromised patients. This includes cancer patients who
are at increased risk of infection from the side effects of chemotherapy, and
who have infrequent access to phlebotomy-based blood cell counts in hospital
settings. Low-cost, mobile phone-based versions of this technology have been
developed, indicating this technique could have further impact as a point-of-
care tool in low- and middle- income countries that lack access to laboratory
equipment [3, 4]. Neutropenia screening with nailfold capillaroscopy relies on
detecting and quantifying optical absorption gaps (OAGs) that are most likely
to contain a neutrophil, based on features described in literature [5, 6, 7,
8, 9, 10]. OAGs appear as gaps of negative absorption contrast between red
blood cells within a capillary loop, and there is strong evidence they can be
caused by circulating leukocytes or blood plasma [5, 6, 7, 9]. Currently,
there is no method of determining the composition of an OAG. To produce
quantitative white blood cell counts non-invasively, there is a need for tools
that elucidate the composition of OAGs flowing through human capillaries.

Techniques such as optical coherence tomography [11, 12, 13] and reflectance
confocal imaging [14] could be applied to in vivo cytometry, however compared
to OPM, they are typically lower resolution and lower speed, respectively.
Spectrally encoded confocal microscopy [15, 16, 10] can increase confocal
imaging speed but requires additional system complexity. A conventional
scattering light sheet microscope with two orthogonal objectives can also
probe scattering signal in bulk tissue [17], however is not compatible
simultaneously with widefield capillaroscopy as presented here. Oblique back-
illumination capillaroscopy [18], a relatively simple, widefield imaging
technique, could prove useful in resolving OAGs in superficial capillaries,
though only images a single depth.

For the task of detecting blood cells flowing in capillaries in vivo, and in
particular detecting white blood cells, scattering-contrast oblique plane
microscopy (sOPM) provides several advantageous features. From a microscopy
perspective, human capillaries are superficially embedded within a turbid,
semi-infinite medium, and thus, a single-objective, epi-illumination geometry
is ideal [19, 20]. Further, an oblique light sheet probes a range of tissue
depths simultaneously with an en face view, which is useful for finding and
monitoring capillaries embedded within tissue at different depths. sOPM can
utilize low-cost high-speed cameras to meet the frame-rate demands of
capturing blood cells that travel up to 1mm/s at the capillary level [21, 22].
Though it broadens with depth in scattering media such as skin, the oblique
light sheet provides optical sectioning, illuminating primarily a thin plane
of tissue conjugate with the sensor, helping reduce out-of-focus signal. A
well-formed light sheet can illuminate blood cells passing single-file through
a capillary, analogous to a conventional scattering-based flow cytometer [23].
In fact, Dunsby suggested the use of OPM for flow cytometry in his seminal
work [19]. In conventional flow cytometry, the intrinsic side- and forward-
scattering signal from laser-illuminated blood cells provides identifying
information on cellular granularity and size [24, 25]. In contrast, sOPM
detects side- and back-scattered light.

In this paper, we build an OPM and compare fluorescence and scattering
contrast by imaging the same polystyrene bead sample with each technique. We
also use sOPM to image the intrinsic scattering signal from blood cells
flowing in a human capillary in vivo. By simultaneously acquiring data of the
same capillary with sOPM and a conventional widefield capillaroscope, we
demonstrate that OAGs that appear to be comprised of a white blood cell can be
distinguished with sOPM from those that appear to be plasma gaps. Finally, we
confirm a similar scattering measurement in vitro from isolated blood cells
embedded in a 3D agar phantom.

##  2 Methods

###  2.1 Optical system

A schematic of the optical system used for the experiments in this paper is
shown in Figure 1. The system is comprised of two main components: (1) the
sOPM, and (2) a widefield capillaroscope.

The sOPM system is illuminated by either a super luminescent diode (SLD, 840nm
Exalos EBD270001-03) collimated with a 15 mm biconvex lens, or a laser diode
(658nm, Thorlabs L658P040) collimated and spatially filtered (SF) with a 50
$\mu$m pinhole placed at the Fourier plane of a 2x beam expander (25 mm and 50
mm biconvex lenses). Sources are switchable using a flip mirror (FM) in the
common illumination path, and are focused to a line on one mirror of a
galvanometer mirror pair (GM, Thorlabs GVS012) by a cylindrical lens (f = 50
mm). The line on the mirror is made conjugate with the back focal plane of
objective 1 (Nikon 40x 1.15NA APO LWD WI $\lambda$S WD = 0.59-0.61) by a 4F
system (RL1, f = 75mm and RL2, f = 150mm). The image of the light sheet at the
back focal plane is offset from the objective’s optic axis, creating an
oblique light sheet (LS) in sample space. The GMs were rotated during
alignment to tune the light sheet angle to $\alpha$ = 36∘. The system is
aligned such that the working distance of objective 1 is located at the sample
surface. Scattered light (SL, purple) is collected by objective 1, redirected
off a 50:50 beamsplitter (BS, Thorlabs BSW29), and relayed to objective 2
(Nikon 20x 0.75NA Plan Apo DIC N2 WD = 1mm) by a second relay system (RL3, f =
100mm, RL4, f = 150mm). A circular iris (Thorlabs, SM1D12C) is placed at the
back aperture of objective 2 to reduce its effective numerical aperture and
control aberrations. The collected light is imaged to an oblique image plane
(OIP) with net magnification from the sample space of 1.33x, to match the
index of refraction ratio between objective 1 and objective 2 sample space.
Finally, objective 3 (Nikon finite-conjugate 40x 0.65NA 210 ELWD) re-images
the tilted light sheet on the sOPM sensor (PCO.edge 5.5 sCMOS). For
fluorescent bead imaging, a 680nm LP filter (EF, Omega RPE680LP) is added in
the second 4F system. The magnification of the sOPM system is 53.2x,
considering the index of refraction 1.33 and the 40x magnification of
objective 3.

A widefield capillaroscope system is included to enable simultaneous
acquisition of the two different signals. It consists of a green LED
(Superbright 1W XLamp LED, 88lm, 527nm) condensed by an aspheric lens (f =
20mm, Thorlabs ACL2520U-A), and reflected off a 605nm long-pass dichroic
mirror (DM1, Thorlabs DMLP605R). The LED produces brightfield illumination
(BFI) on the sample, that is strongly absorbed by hemoglobin within red blood
cells. Remitted light is collected by objective 1, redirected off the
beamsplitter, and focused onto the capillaroscope sensor (Imaging Source DMK
33UX252) by RL3 and a second dichroic mirror (DM2, Thorlabs DMLP605R). The
magnification of the capillaroscope is approximately 20x, due to RL3, acting
as tube lens with half the standard focal length specified for objective 1.

Figure 1: (a) Layout of the sOPM (red and purple beamlines) and the widefield
capillaroscope (green beamlines). A flip mirror (FM) is used to alternate
between 658nm laser diode (LD) or 840nm super luminescent diode (SLD) light
sheet (LS) illumination. The LD is shaped by a spatial filter (SF), and the
840nm diode is collimated with a collimating lens (CL1). Both sources pass
through a cylindrical lens (CyL), which forms a light sheet onto galvanometric
mirrors (GM), which is then imaged to the back aperture of objective 1 by
relay lenses (RL) 1 $\&$ 2\. Scattered light (SL) is collected by the same
objective, reflected off a beamsplitter (BS), and relayed by another 4F system
(RL3 and RL4) to the back aperture of objective 2, forming a 1.33x
magnification oblique image plane (OIP). Finally, a finite conjugate objective
3 images the oblique image plane onto the sOPM sensor. For fluorescence-mode
imaging, an emission filter (EF) is placed prior to objective 2. For
aberration control, a circular iris is placed at the back aperture of
objective 2. Widefield capillaroscopy with brightfield illumination (BFI) is
performed with a collimated 527nm LED introduced through a dichroic mirror
(DM1). Light is collected by the same objective and directed to the
capillaroscope sensor by a second dichroic mirror (DM2). (b) An enlarged
region of interest of the sample space shows a capillary loop within the light
sheet and the imaging coordinate system. Note that z’ refers to the direction
of the light sheet. Scattered light is detected through a coverslip (CS) and
water immersion (W).

###  2.2 Polystyrene bead phantoms

Fluorescent polystyrene beads (4$\mu$m diameter, 1$\%$ solids, Bangslabs
FSFR006) were embedded in 1$\%$ w/v aqueous agar solution with a 30x dilution.
Bead solution was pipetted into 800$\mu$m deep silicone isolators (Invitrogen
secure-seal hybridization chamber gasket), which was mounted on top of a
coverslip ($\\#$1.5) for imaging. The 658nm laser diode was used for
illumination, and an exposure time of 50ms was used for scattering imaging.
Since the scattering signal was significantly more intense than fluorescence,
the image was dominated by scattered light in this case. For fluorescent
imaging, an emission filter (Omega RPE680LP) was inserted into the detection
path, blocking the scattering signal, and an exposure time of 500ms was used.

###  2.3 In vivo capillary imaging

In vivo human imaging was conducted in a protocol approved by the Johns
Hopkins University Institutional Review Board (IRB00204985). A single healthy,
human volunteer was enrolled in the study. The participant’s ventral tongue
was imaged at 100Hz with the 658nm laser diode, using vacuum stabilization to
keep the tissue stationary (see [18] for details). A digital hardware trigger
was used (Arduino Uno) to synchronize image acquisition between the sOPM and
capillaroscope sensors.

###  2.4 Blood cell phantoms

To support the in vivo blood cell scattering results, we created in vitro
blood cell phantoms using 1.5 $\%$ w/v low melting point (LMP) agar solution
in a 0.9 mM NaCl buffer solution. Blood products were purchased from Zen-Bio,
including leukocyte-depleted erythrocytes (SER-LDRBC-SDS), buffy coat (SER-BC-
SDS), and platelet-rich concentrate (SER-PC-SDS). White blood cell isolation
was conducted on the buffy coat sample, using centrifugation (300g, 5 minutes)
and AKC red blood cell lysis buffer (ThermoFisher A1049201). The platelet
sample was further concentrated 8x through centrifugation (1500g, 15 minutes),
pelleting, decanting, and resuspension. Purified blood products were diluted
20x with LMP agar solution, and pipetted into 800 $\mu$m deep silicone
isolators. Images of the purified products were taken with the widefield
capillaroscope to verify successful cell purification (Figure 2).

Figure 2: Purified blood cell products were obtained and confirmed with the
capillaroscope - (a) leukocyte-depleted red blood cells, (b) purified white
blood cells from buffy coat, and (c) concentrated platelets.

##  3 Results

###  3.1 Polystyrene bead phantoms

Polystyrene bead phantoms provided insight into a few key phenomena. First, we
observed that their image appeared to become severely aberrated at sample
depths of z > 95 $\mu$m (z’ > 160 $\mu$m) (Figure 3(a) $\&$ (c)). This is
consistent with the axial range of aberration-free remote focusing predicted
by Botcherby et al.’s analysis adapted to our system parameters [20]. We
observed that the aberrations at these imaging depths could be reduced with an
iris at the back aperture of objective 2 (Figure 3(b) $\&$ (d)), which reduces
the effective numerical aperture of the objective. A 4 mm diameter iris, which
creates an effective numerical aperture for objective 2 of NA${}_{eff}=0.33$,
provided a reasonable balance of imaging depth and bead resolution.
Additionally, we observed that more polystyrene beads came into focus when we
stopped down the iris (Figure 3(c) vs. (d)). We believe this is due to an
increase in depth of field afforded by the lower numerical aperture, now
focusing light from particles located in planes slightly above and below the
center axis of the light sheet. While this technique does improve particle
resolution by controlling aberrations, the collected scattering signal was
significantly reduced (Figure 3(c) vs. (d)).

Figure 3: (a) Aberration and depth of field control demonstrated with 4 $\mu$m
polystyrene beads and a variable iris placed at the back focal plane of
objective 2. Full frame (a) and region of interest (c) taken with objective 2
NA $=0.75$. Full frame (b) and region of interest (d) taken with objective 2
NA${}_{eff}=0.33$. An increase in particle number, likely due to depth of
field increase, and improved particle visibility, from aberration control, are
both observed by decreasing the effective numerical aperture of collection.
The vertical line of intensity superficial to the sample is formed by specular
reflection from the coverslip.

We also observed that polystyrene beads imaged by the sOPM system appeared
consistently smaller than what was predicted by the magnification and the 4
$\mu$m specified bead diameter (Figure 4(a)). In scattering detection, a FWHM
of 1.1 $\mu$m was observed along the y-axis, and 3 $\mu$m along the z’ axis.
However, when the same beads were imaged by only their fluorescent signal by
placing a 680nm long-pass emission filter in the detection path and increasing
the exposure time, the beads appeared much closer to their expected size
(Figure 4(b)). With fluorescent detection, a FWHM of 3.5 $\mu$m was observed
along the y-axis, and 5.9 $\mu$m along the z’ axis. We believe the discrepancy
between the two signals is due to higher order scattering artifact (lensing
and mirroring) arising from the large index of refraction mismatch between the
polystyrene beads and the agar phantom [26]. The small, spherically shaped
beads may act as miniature convex mirrors, creating a demagnified virtual
image within the bead itself. In the case of fluorescent imaging, specular
reflection is blocked by the emission filter, and the signal of the entire
fluorescent bead is observed.

Figure 4: (a) Scattering- and (b) fluorescence-OPM images of the same 4 $\mu$m
polystyrene beads in agar. The same beads imaged with scattering contrast
appear nearly half the bead size, likely a virtual image from reflected light
off the spherical bead.

###  3.2 In vivo capillary imaging

We next imaged capillaries on the ventral tongue of a healthy human
participant. We used vacuum stabilization to keep the tongue stationary [18]
and imaged with both scattering-contrast OPM and absorption contrast widefield
capillaroscopy. A time-course measurement from this experiment is displayed in
Figure 5. Each panel (a)-(c) highlights a distinct event that occurred while
imaging the same capillary, with widefield capillaroscopy images placed
directly above the temporally corresponding sOPM images, and temporally
adjacent frames in each column ($\Delta$t = 10 ms). In Figure 5(d), a
quantitative comparison between line profiles of the image modalities is
presented. The Michelson contrast of the capillary in the capillaroscope is
compared to the mean scattering intensity of the sOPM signal. When red blood
cells pass through the capillary (Figure 5(a)), we observe a relatively weak
and consistent signal in the widefield capillaroscope (due to hemoglobin
absorption) and a relatively strong and consistent signal in sOPM (due to
intracellular scattering). When a pair of white blood cells pass through the
capillary (Figure 5(b)), a distinct drop in capillary contrast is observed
with the widefield system from the absence of hemoglobin. For this event, the
sOPM system registers a scattering signal from the pair of white blood cells
that is similar in intensity to that from the red blood cells. Finally, in
Figure 5(c), a larger OAG is observed passing through the capillary, where the
presence or absence of white blood cell(s) is more ambiguous. In this case, a
wider but similar drop in contrast is observed in the widefield
capillaroscope. For this event however, the sOPM system also demonstrates a
marked decrease in scattering intensity, suggesting the OAG is acellular in
nature and could be caused simply by blood plasma.

Figure 5: Paired widefield capillaroscopy and sOPM images of an in vivo human
ventral tongue capillary. Each column represents 10 ms time difference, light
sheet (LS) location denoted by red overlay. (a) Red blood cells pass through
the capillary loop as dark shadows in widefield capillaroscopy imaging (top),
and at the same time generate strong scattering signal in sOPM image (bottom).
(b) A tightly packed pair of white blood cells (WBCs) passes around the
capillary loop and generates a similarly strong scattering signal to adjacent
red blood cells in sOPM imaging. (c) A large OAG passes through the capillary
loop, producing a marked decrease in scattering intensity observed by the sOPM
sensor, suggesting acellular origin. (d) The Michelson contrast of the
capillary in the capillaroscope image is compared to the mean scattering
intensity from sOPM.

###  3.3 Blood cell phantoms

To validate the in vivo blood cell scattering results, we imaged isolated
blood components in the sOPM system (Figure 6). The left column ((a) $\&$ (d))
shows RBCs, the middle column ((b) $\&$ (e)) WBCs, and the right column ((c)
$\&$ (f)) show platelets. The top row ((a)-(c)) demonstrate RBC, WBC, and
platelet samples, respectively, with 658nm LD illumination, while the bottom
row ((d)-(f)) show the same particles illuminated with the 840nm SLD. With
these experiments, we observe that red blood cells and white blood cells
produce similar scattering contrast in vitro and in vivo. Further, we observe
that the use of a SLD reduces speckle noise and makes blood cells easier to
resolve. This is especially apparent with smaller particles, such as
platelets, whose sizes are similar to the speckle grains of the 658nm LD.
While the 840nm SLD in our experiment was not powerful enough to be used in
our current system for in vivo imaging, it is clear the use of a broadband
source or other methods of speckle reduction can significantly improve the
signal to noise ratio of sOPM.

Figure 6: Agar blood cell phantom imaging. Panels (a) $\&$ (d) show isolated
red blood cells, (b) $\&$ (e) show white blood cells, and (c) $\&$ (f) show
platelets. Each sample was first imaged with the 658nm laser diode (top), and
second with an 840nm SLD (bottom).

##  4 Discussion

With its epi-illumination geometry, reliance on endogenous signal, and
capability for high-speed, optically-sectioned imaging, scattering oblique
plane microscopy could serve as a useful tool for in vivo blood analysis. Our
results show that the endogenous scattering signal of blood cells within human
capillaries can be imaged non-invasively at 100 Hz. We observe that white
blood cells and red blood cells produce a similar strength of scattered
signal, while larger, acellular OAGs demonstrate a marked reduction in
scattered signal. While the sOPM presented here could not distinguish white
blood cells and red blood cells, it can distinguish between acellular plasma
gaps and blood cells. In combination with widefield absorption-based
capillaroscopy, which can distinguish between red blood cells and white blood
cells, the three of these major constituents of capillary blood (red blood
cell, white blood cell, and plasma gaps) can be classified. Recently, the
frequency of OAGs in nailfold capillaries has been inversely correlated to
severe neutropenia [1, 2]. Although the OAGs identified in this approach
fulfill certain features that maximize their likelihood of representing
neutrophils, they still might be either plasma or white blood cell in origin,
which may limit the expansion of this technique to provide quantitative white
blood cell counts. sOPM has the potential to reduce this ambiguity. However,
in order to image capillaries in the nailfold, which are typically 150-400
$\mu$m beneath the tissue surface [27], improvements are needed to enable sOPM
at larger depths. This may be possible with lower magnification objective
lenses, longer wavelengths, and potentially incorporating confocal line-
scanning to improve optical sectioning [28, 29, 30]. It may also be useful to
test sOPM with an anamorphic lens design [31] or utilizing a diffractive
approach to help increase the field of view [32].

Additional future work should focus on developing scattering contrast
techniques that distinguish between red blood cells and white blood cells. We
believe it is worth investigating scattering spectral dependence,
polarization-sensitivity, and scattering angle-resolved detection for this
purpose. Additionally, it is clear from our in vitro blood cell phantom
results that speckle noise is degrading the signal. Thus, the use of a higher
power broadband source or other means of speckle reduction such as a
commercial laser speckle reducer, rotating diffuser, or despeckling algorithms
[33] are worth investigating. Finally, volumetric tissue imaging can be
achieved by incorporating scanning and descanning galvanometric mirrors into
the sOPM design, similar to how SCAPE is used for fluorescence imaging [34].
With improved designs, scattering oblique plane microscopy may find use in
dermatologic, ophthalmic, and endoscopic settings, similar to optical
coherence tomography, for evaluating bulk tissue non-invasively with B-mode
imaging. The ability for sOPM to detect side- and back-scattered light could
enable further cell and tissue discrimination, analogous to how conventional
flow cytometry extracts information through measuring side- and forward-
scattered light.

##  5 Conclusion

Human capillary imaging with new microscopy techniques has the potential to
reveal important clinical biomarkers that may enable non-invasive blood
analysis. For example, non-invasive severe neutropenia screening has been
shown possible by quantifying certain OAGs in conventional capillaroscopy
videos [1, 2]. However, despite its success to screen for neutropenia, the
exact composition of OAGs is ambiguous with widefield capillaroscopy, which
may limit its application to provide quantitative white blood cell count.
Scattering oblique plane microscopy can provide additional and complimentary
signal to discriminate cellular versus acellular OAGs in human capillaries in
vivo, helping to classify blood columns into the three major components of red
blood cells, white blood cells, and plasma.

## Funding

This research was supported in part with a gift from Fifth Generation, Inc.,
the Johns Hopkins Medical Science Training Program Fellowship, and the
National Institutes of Health (R44CA228920).

## Acknowledgments

The authors would like to thank Milind Rajadhyaksha and Charles A. DiMarzio
for helping interpret scattering vs. fluorescent signals. The authors would
also like to thank Ian Butterworth, Aurélien Bourquard, Álvaro Sánchez-Ferro
and the team at Leuko Labs for their ongoing insight, support, and feedback
about the project.

## Disclosures

The authors are co-inventors on a provisional patent application assigned to
Johns Hopkins University. They may be entitled to future royalties from
intellectual property related to the technologies described in this article.

## References

  * [1] A. Bourquard, A. Pablo Trinidad, I. Butterworth, Á. Sánchez Ferro, C. Cerrato, K. Humala, M. F. Urdiola, C. D. Rio, B. Valles, J. M. Tucker Schwartz, E. S. Lee, B. J. Vakoc, T. P. Padera, M. J. Ledesma Carbayo, Y.-B. Chen, E. P. Hochberg, M. L. Gray, and C. Castro González, “Non-invasive detection of severe neutropenia in chemotherapy patients by optical imaging of nailfold microcirculation,” Scientific Reports 8:5301, 1–12 (2018).
  * [2] A. Pablo-Trinidad, I. Butterworth, M. J. Ledesma-Carbayo, T. Vettenburg, A. Sanchez-Ferro, L. Soenksen, N. J. Durr, A. Munoz-Barrutia, C. Cerrato, K. Humala, M. F. Urdiol, C. Del Rio, B. Valles, Y.-B. Chen, E. P. Hochberg, C. Castro-González, and A. Bourquard, “Automated detection of neutropenia using noninvasive video microscopy of superficial capillaries,” American Journal of Hematology 94, E219–E222 (2019).
  * [3] C. Castro-González, I. Butterworth, A. Bourquard, and A. Sanchez-Ferro, “Systems, Apparatus, and Methods for Analyzing Blood Cell Dynamics (US 9,984,277 B2),” United States Patent (2018).
  * [4] G. N. McKay, N. Mohan, I. Butterworth, A. Bourquard, A. Sanchez-Ferro, C. Castro-González, and N. J. Durr, “Visualization of blood cell contrast in nailfold capillaries with high-speed reverse lens mobile phone microscopy,” Biomedical Optics Express 11, 2268–2276 (2020).
  * [5] G. W. Schmid-Schonbein, S. Usami, R. Skalak, and S. Chien, “The Interaction of Leukocytes and Erythrocytes Capillary and Postcapillary Vessels,” Microvascular Research 19, 45–70 (1980).
  * [6] S. H. Sinclair, M. Azar-cavanagh, K. A. Soper, R. F. Tuma, and H. N. Mayrovirz, “Investigation of the Source of the Blue Field Entoptic Phenomenon,” Investigative Ophthalmology and Visual Science 30, 668–673 (2018).
  * [7] A. Roggan, M. Friebel, K. Dorschel, A. Hahn, and G. Muller, “Optical Properties of Circulating Human Blood in the Wavelength Range 400-2500nm,” Journal of Biomedical Optics 4, 36–46 (1999).
  * [8] G. Fuchsjäger-Mayrl, M. Malec, E. Polska, B. Jilma, M. Wolzt, and L. Schmetterer, “Effects of granulocyte colony stimulating factor on retinal leukocyte and erythrocyte flux in the human retina,” Investigative Ophthalmology and Visual Science 43, 1520–1524 (2002).
  * [9] A. Uji, M. Hangai, S. Ooto, K. Takayama, N. Arakawa, H. Imamura, K. Nozato, and N. Yoshimura, “The Source of Moving Particles in Parafoveal Capillaries Detected by Adaptive Optics Scanning Laser Ophthalmoscopy,” Investigative Ophthalmology and Visual Science 53, 171–178 (2012).
  * [10] L. Golan, D. Yeheskely-hayon, L. Minai, and E. J. Dann, “Noninvasive imaging of flowing blood cells using label-free spectrally encoded flow cytometry,” Biomedical Optics Express 3, 628–630 (2012).
  * [11] L. Ginner, T. Schmoll, A. Kumar, M. Salas, N. Pricoupenko, L. M. Wurster, and R. A. Leitgeb, “Holographic line field en-face OCT with digital adaptive optics in the retina in vivo,” Biomedical Optics Express 9, 472 (2018).
  * [12] C.-L. Chen and R. K. Wang, “Optical coherence tomography based angiography [Invited],” Biomedical Optics Express 8, 1056 (2017).
  * [13] R. A. Leitgeb, “En face optical coherence tomography: a technology review [Invited],” Biomedical Optics Express 10, 2177 (2019).
  * [14] I. Saknite, Z. Zhao, J. R. Patrinely, M. Byrne, M. Jagasia, and E. R. Tkaczyk, “Individual cell motion in healthy human skin microvasculature by reflectance confocal video microscopy,” Microcirculation 27, 1–11 (2020).
  * [15] G. J. Tearney, R. H. Webb, and B. E. Bouma, “Spectrally encoded confocal microscopy,” Optics Letters 23, 1152–1154 (1998).
  * [16] L. Golan and D. Yelin, “Flow cytometry using spectrally encoded confocal microscopy,” Optics Letters 35, 2218 (2010).
  * [17] C. D. Nguyen, P. K. O’Neal, N. Kulkarni, E. Yang, and D. Kang, “Scattering-Based Light-Sheet Microscopy for Rapid Cellular Imaging of Fresh Tissue,” Lasers in Surgery and Medicine pp. 1–8 (2020).
  * [18] G. N. McKay, N. Mohan, and N. J. Durr, “Imaging human blood cells in vivo with oblique back-illumination capillaroscopy,” Biomedical Optics Express 11, 2373–2382 (2020).
  * [19] C. Dunsby, “Optically sectioned imaging by oblique plane microscopy,” Optics Express 16, 186–196 (2009).
  * [20] E. J. Botcherby, R. Juskaitis, M. J. Booth, and T. Wilson, “An optical technique for remote focusing in microscopy,” Optics Communications 281, 880–887 (2008).
  * [21] A. Bollinger, P. Butti, J. P. Barras, H. Trachsler, and W. Siegenthaler, “Red blood cell velocity in nailfold capillaries of man measured by a television microscopy technique,” Microvascular Research 7, 61–72 (1974).
  * [22] N. Mugii, M. Hasegawa, Y. Hamaguchi, C. Tanaka, K. Kaji, K. Komura, I. Ueda-Hayakawa, S. Horie, M. Ikuta, K. Tachino, F. Ogawa, S. Sato, M. Fujimoto, and K. Takehara, “Reduced red blood cell velocity in nail-fold capillaries as a sensitive and specific indicator of microcirculation injury in systemic sclerosis,” Rheumatology 48, 696–703 (2009).
  * [23] G. Henel and J. L. Schmitz, “Basic Theory and Cinical Applications of Flow Cytometry,” Laboratory Medicine 38, 428–436 (2007).
  * [24] S. H. Cho, J. M. Godin, C.-H. Chen, W. Qiao, H. Lee, and Y.-H. Lo, “Review Article: Recent advancements in optofluidic flow cytometer,” Biomicrofluidics 4 (2010).
  * [25] V. Ost, J. Neukammer, and H. Rinneberg, “Flow Cytometric Differentiation of Erythrocytes and Leukocytes in Dilute Whole Blood by lLght Scattering,” Cytometry 32, 191–197 (1998).
  * [26] P. Ledwig and F. E. Robles, “Quantitative 3D refractive index tomography of opaque samples in epi-mode,” Optica 8, 6 (2021).
  * [27] U. Baran, L. Shi, and R. K. Wang, “Capillary blood flow imaging within human finger cuticle using optical microangiography,” Journal of Biophotonics 8, 46–51 (2015).
  * [28] M. Rajadhyaksha, R. R. Anderson, and R. H. Webb, “Video-rate confocal scanning laser microscope for imaging human tissues in vivo,” Applied Optics 38, 2105–2115 (1999).
  * [29] B. Larson, S. Abeytunge, and M. Rajadhyaksha, “Performance of full-pupil line-scanning reflectance confocal microscopy in human skin and oral mucosa in vivo,” Biomedical Optics Express 2, 2055 (2011).
  * [30] A. K. Glaser, Y. Wang, and J. T. C. Liu, “Assessing the imaging performance of light sheet microscopies in highly scattering tissues,” Biomedical Optics Express 7, 129–138 (2016).
  * [31] W. Shao, K. Kilic, W. Yin, G. Wirak, X. Qin, H. Feng, D. Boas, C. V. Gabel, and J. Yi, “Wide field-of-view volumetric imaging by a mesoscopic oblique plane microscopy with low numerical aperture objectives,” bioRxiv (2020).
  * [32] M. Hoffman and B. Judkewitz, “Diffractive oblique plane microscopy,” Optica 6 (2019).
  * [33] T. L. Bobrow, F. Mahmood, M. Inserni, and N. J. Durr, “DeepLSR: A deep learning approach for laser speckle reduction,” Biomedical Optics Express 10, 2869–2882 (2018).
  * [34] M. B. Bouchard, V. Voleti, C. S. Mendes, C. Lacefield, W. B. Grueber, R. S. Mann, R. M. Bruno, and E. M. C. Hillman, “Swept confocally-aligned planar excitation (SCAPE) microscopy for high-speed volumetric imaging of behaving organisms,” Nature Photonics 9, 113–119 (2015).


$\varepsilon\to 0$, we have

$-\log\Pi(\|u\|_{s^{\prime},q}\leq\varepsilon)\asymp\varepsilon^{-\frac{1}{\frac{s-s^{\prime}}{d}-\frac{1}{q}}}$

###### Proof.

We can compute

$\Pi(\|u\|_{s^{\prime},q}\leq\varepsilon)=\mathbb{P}\left[\sum_{\ell=1}^{\infty}(\ell^{\tau_{q}(s^{\prime})-\tau_{q}(s)}\|\xi_{\ell}(\cdot)\|_{q})^{q}\leq\varepsilon^{q}\right]$

where $\mathbb{P}$ is the probability measure on the infinite product space
$\Omega=(L^{q}({\mathcal{T}}))^{\infty}$ as in Definition 4. From the proof of
Theorem 3.3 we know
$\|\xi\|_{q}^{q}=\sum_{\ell=1}^{\infty}\lambda_{\ell}^{\frac{q}{2}}\chi^{2}_{\ell}$
is an infinite mixture of $\chi^{2}(1)$ random variables, so the condition of
[Theorem 4.2 of 4] is trivially met and we have

$\log\mathbb{P}\left[\sum_{\ell=1}^{\infty}(\ell^{\tau_{q}(s^{\prime})-\tau_{q}(s)}\|\xi_{\ell}(\cdot)\|_{q})^{q}\leq\varepsilon^{q}\right]\asymp\varepsilon^{-\frac{1}{\tau_{q}(s)-\tau_{q}(s^{\prime})-\frac{1}{q}}}$

∎

The second lemma gives a upper bound of the first term of the concentration
function (25).

######  Lemma A.3 (Decentering function).

Assume $u^{\dagger}\in B^{s^{\dagger},q^{\dagger}}({\mathcal{Z}})$ for some
$s^{\dagger}>s^{\prime}$ and $q^{\dagger}\in[1,2]$. Then as $\varepsilon\to
0$, we have the following bounds

  1. (i)

If $q^{\dagger}\geq q$, we require $s^{\dagger}>s^{\prime}$:

$\inf_{h\in
B^{s,q}({\mathcal{Z}}):\|h-u^{\dagger}\|_{s^{\prime},q}\leq\varepsilon}\|h\|_{s,q}^{q}\lesssim\begin{cases}1,&\\!if\,s<s^{\dagger}\\\
(-\log\varepsilon)^{1-\frac{q}{q^{\dagger}}},&\\!if\,s=s^{\dagger}\\\
\varepsilon^{-\frac{s-s^{\dagger}}{s^{\dagger}-s^{\prime}}(q\wedge
q^{\dagger})},&\\!if\,s>s^{\dagger}\end{cases}$

  2. (ii)

If $q^{\dagger}<q$, we restrict $u^{\dagger}\in
B^{s^{\dagger},q^{\dagger},q}({\mathcal{Z}})\subsetneq
B^{s^{\dagger},q^{\dagger}}({\mathcal{Z}})$ and require
$s^{\dagger}>s^{\prime}-\frac{d}{q}+\frac{d}{q^{\dagger}}$:

$\inf_{h\in
B^{s,q}({\mathcal{Z}}):\|h-u^{\dagger}\|_{s^{\prime},q}\leq\varepsilon}\|h\|_{s,q}^{q}\lesssim\begin{cases}1,&\\!if\,s\leq
s^{\dagger}+\frac{d}{q}-\frac{d}{q^{\dagger}}\\\
\varepsilon^{-\frac{\frac{s-s^{\dagger}}{d}-\frac{1}{q}+\frac{1}{q^{\dagger}}}{\frac{s^{\dagger}-s^{\prime}}{d}+\frac{1}{q}-\frac{1}{q^{\dagger}}}q},&\\!if\,s>s^{\dagger}+\frac{d}{q}-\frac{d}{q^{\dagger}}\end{cases}$

###### Proof.

First of all, by Proposition 4.1 we have $B^{s^{\dagger},q^{\dagger}\wedge
q,q}({\mathcal{Z}})\subset B^{s^{\prime},q}$ for
$s^{\prime}<s^{\dagger}-\left(\frac{d}{q^{\dagger}}-\frac{d}{q}\right)_{+}$.

Next, for given spatial basis $\\{\phi_{\ell}\\}_{\ell=1}^{\infty}$ in
Definition 4, we identify $u^{\dagger}\in B^{s^{\dagger},q^{\dagger}}$ with
$u^{\dagger}_{\mathcal{T}}=\\{u^{\dagger}_{\ell}(\cdot)\\}_{\ell=1}^{\infty}\in\ell^{q^{\dagger},\tau_{q^{\dagger}}(s^{\dagger})}(L^{q^{\dagger}}({\mathcal{T}}))$.
Then we follow [1] to approximate $u^{\dagger}_{\mathcal{T}}$ with
$h_{1:L}=\\{u^{\dagger}_{\ell}(\cdot)\\}_{\ell=1}^{\infty}$ where
$u^{\dagger}_{\ell}(\cdot)\equiv 0$ for all $\ell>L$. Note
$h_{1:L}\in\ell^{q,\tau_{q}(s)}(L^{q}({\mathcal{T}}))$ for any finite
$L\in{\mathbb{N}}$. Identifying $h_{1:L}$ with $h\in B^{s,q}({\mathcal{Z}})$,
we use the similar argument as above to get

$\|h-u^{\dagger}\|_{s^{\prime},q}^{q}=\sum_{\ell=L+1}^{\infty}\ell^{\tau_{q}(s^{\prime})q}\|u^{\dagger}_{\ell}(\cdot)\|_{q}^{q}\leq\begin{cases}\|u^{\dagger}\|_{s^{\dagger},q^{\dagger}}^{q^{\dagger}}L^{\frac{s^{\prime}-s^{\dagger}}{d}q^{\dagger}},&\\!if\,q^{\dagger}=q\\\
\|u^{\dagger}\|_{s^{\dagger},q^{\dagger}}^{q}L^{\frac{s^{\prime}-s^{\dagger}}{d}q},&\\!if\,q^{\dagger}>q\\\
\|u^{\dagger}\|_{s^{\dagger},q^{\dagger},q}^{q}L^{\left(\frac{s^{\prime}-s^{\dagger}}{d}-\frac{1}{q}+\frac{1}{q^{\dagger}}\right)q},&\\!if\,q^{\dagger}<q\end{cases}$

Therefore, to have $\|h-u^{\dagger}\|_{s^{\prime},q}\leq\varepsilon$ we let

$L\gtrsim\begin{cases}\varepsilon^{-\frac{d}{s^{\dagger}-s^{\prime}}},&\\!if\,q^{\dagger}\geq
q\\\
\varepsilon^{-\frac{1}{\frac{s^{\dagger}-s^{\prime}}{d}+\frac{1}{q}-\frac{1}{q^{\dagger}}}},&\\!if\,q^{\dagger}<q\end{cases}$
(42)

On the other hand, the infimum is less than $\|h\|_{s,q}^{q}$ with above $h$,
which can be bounded as follows. If $q^{\dagger}=q$,

$\|h\|_{s,q}^{q}=\sum_{\ell=1}^{L}\ell^{\tau_{q^{\dagger}}(s)q^{\dagger}}\|u^{\dagger}_{\ell}(\cdot)\|_{q^{\dagger}}^{q^{\dagger}}\leq\begin{cases}\|u^{\dagger}\|_{s^{\dagger},q^{\dagger}}^{q^{\dagger}},&\\!if\,s\leq
s^{\dagger}\\\
\|u^{\dagger}\|_{s^{\dagger},q^{\dagger}}^{q^{\dagger}}L^{\frac{s-s^{\dagger}}{d}q^{\dagger}},&\\!if\,s>s^{\dagger}\end{cases}$

While if $q^{\dagger}>q$, by similar argument using Hölder inequality as in
(38),

$\|h\|_{s,q}^{q}=\sum_{\ell=1}^{L}\ell^{\tau_{q}(s)q}\|u^{\dagger}_{\ell}(\cdot)\|_{q}^{q}\leq\begin{cases}C\|u^{\dagger}\|_{s^{\dagger},q^{\dagger}}^{q},&\\!if\,s<s^{\dagger}\\\
\|u^{\dagger}\|_{s^{\dagger},q^{\dagger}}^{q}(\log
L)^{1-\frac{q}{q^{\dagger}}},&\\!if\,s=s^{\dagger}\\\
\|u^{\dagger}\|_{s^{\dagger},q^{\dagger}}^{q}L^{\frac{s-s^{\dagger}}{d}q},&\\!if\,s>s^{\dagger}\end{cases}$

And if $q^{\dagger}<q$, by similar argument as in (39),

$\|h\|_{s,q}^{q}=\sum_{\ell=1}^{L}\ell^{\tau_{q}(s)q}\|u^{\dagger}_{\ell}(\cdot)\|_{q}^{q}\leq\begin{cases}\|u^{\dagger}\|_{s^{\dagger},q^{\dagger},q}^{q},&\\!if\,s\leq
s^{\dagger}+\frac{d}{q}-\frac{d}{q^{\dagger}}\\\
\|u^{\dagger}\|_{s^{\dagger},q^{\dagger},q}^{q}L^{\left(\frac{s-s^{\dagger}}{d}-\frac{1}{q}+\frac{1}{q^{\dagger}}\right)q},&\\!if\,s>s^{\dagger}+\frac{d}{q}-\frac{d}{q^{\dagger}}\end{cases}$

Substituting $L$ in (42) to the above equations yields the conclusion. ∎

###### Proof of Theorem 4.8.

By Lemmas A.2 and A.3, we have the following bounds for the concentration
function (25) as $\varepsilon\to 0$, if $q^{\dagger}\geq q$,

$\varphi_{u^{\dagger}}(\varepsilon)\lesssim\begin{cases}1+\varepsilon^{-\frac{1}{\frac{s-s^{\prime}}{d}-\frac{1}{q}}},&\\!if\,s<s^{\dagger}\\\
(-\log\varepsilon)^{1-\frac{q}{q^{\dagger}}}+\varepsilon^{-\frac{1}{\frac{s-s^{\prime}}{d}-\frac{1}{q}}},&\\!if\,s=s^{\dagger}\\\
\varepsilon^{-\frac{s-s^{\dagger}}{s^{\dagger}-s^{\prime}}(q\wedge
q^{\dagger})}+\varepsilon^{-\frac{1}{\frac{s-s^{\prime}}{d}-\frac{1}{q}}},&\\!if\,s>s^{\dagger}\end{cases}$

For $s\leq s^{\dagger}$, the bound is dominated by
$\varepsilon^{-\frac{1}{\frac{s-s^{\prime}}{d}-\frac{1}{q}}}$. For the last
case, we need to determine a balancing point of $s$ for the two terms by
setting their powers equal. The calculation shows that if $s\leq
s^{\dagger}+\frac{d}{q}$, the bound is still dominated by
$\varepsilon^{-\frac{1}{\frac{s-s^{\prime}}{d}-\frac{1}{q}}}$, but otherwise
is dominated by
$\varepsilon^{-\frac{s-s^{\dagger}}{s^{\dagger}-s^{\prime}}q}$. Therefore, we
have

$\varphi_{u^{\dagger}}(\varepsilon)\lesssim\begin{cases}\varepsilon^{-\frac{1}{\frac{s-s^{\prime}}{d}-\frac{1}{q}}},&\\!if\,s\leq
s^{\dagger}+\frac{d}{q}\\\
\varepsilon^{-\frac{s-s^{\dagger}}{s^{\dagger}-s^{\prime}}q},&\\!if\,s>s^{\dagger}+\frac{d}{q}\end{cases}$

We need to determine minimal $\varepsilon_{n}$ such that
$\varphi_{u^{\dagger}}(\varepsilon_{n})\leq n\varepsilon_{n}^{2}$. Hence for
$q^{\dagger}\geq q$,

$\varepsilon_{n}\asymp\begin{cases}n^{-\frac{q(s-s^{\prime})-d}{2q(s-s^{\prime})+(q-2)d}},&\\!if\,s\leq
s^{\dagger}+\frac{d}{q}\\\
n^{-\frac{s^{\dagger}-s^{\prime}}{2(s^{\dagger}-s^{\prime})+q(s-s^{\dagger})}},&\\!if\,s>s^{\dagger}+\frac{d}{q}\end{cases}$

Now if $q^{\dagger}<q$, by similar argument we have the concentration function
(25) as $\varepsilon\to 0$

$\varphi_{u^{\dagger}}(\varepsilon)\lesssim\begin{cases}1+\varepsilon^{-\frac{1}{\frac{s-s^{\prime}}{d}-\frac{1}{q}}},&\\!if\,s\leq
s^{\dagger}+\frac{d}{q}-\frac{d}{q^{\dagger}}\\\
\varepsilon^{-\frac{\frac{s-s^{\dagger}}{d}-\frac{1}{q}+\frac{1}{q^{\dagger}}}{\frac{s^{\dagger}-s^{\prime}}{d}+\frac{1}{q}-\frac{1}{q^{\dagger}}}q}+\varepsilon^{-\frac{1}{\frac{s-s^{\prime}}{d}-\frac{1}{q}}},&\\!if\,s>s^{\dagger}+\frac{d}{q}-\frac{d}{q^{\dagger}}\end{cases}$

Thus the contraction rate for $q^{\dagger}<q$ becomes

$\varepsilon_{n}\asymp\begin{cases}n^{-\frac{q(s-s^{\prime})-d}{2q(s-s^{\prime})+(q-2)d}},&\\!if\,s\leq
s^{\dagger}+\frac{2d}{q}-\frac{d}{q^{\dagger}}\\\
n^{-\frac{s^{\dagger}-s^{\prime}+\frac{d}{q}-\frac{d}{q^{\dagger}}}{2(s^{\dagger}-s^{\prime})+q(s-s^{\dagger})-(q-2)(\frac{d}{q}-\frac{d}{q^{\dagger}})}},&\\!if\,s>s^{\dagger}+\frac{2d}{q}-\frac{d}{q^{\dagger}}\end{cases}$

Rewriting the equations into one yields the conclusion. ∎

See 4.9

###### Proof of Theorem 4.9.

Denote the upper bound of the first term in the concentration in Lemma A.3 as
$d(\varepsilon)$. By re-examining the proof of Theorem 4.8, we have the
concentration function (26) bounded as

$\varphi_{u^{\dagger},\kappa}(\varepsilon)\lesssim\kappa^{-q}d(\varepsilon)+(\varepsilon/\kappa)^{-\frac{1}{\frac{s-s^{\prime}}{d}-\frac{1}{q}}}$

The optimal choice, $\kappa\asymp
d(\varepsilon)^{\frac{1}{q}-\frac{d}{q^{2}(s-s^{\prime})}}\varepsilon^{\frac{d}{q(s-s^{\prime})}}$,
is made by balancing the above two terms. Hence the concentration function
bound becomes $\varphi_{u^{\dagger},\kappa}(\varepsilon)\lesssim
d(\varepsilon)^{\frac{d}{q(s-s^{\prime})}}\varepsilon^{-\frac{d}{s-s^{\prime}}}$.
Note, most bounds in Lemma A.3 appears in the format of
$d(\varepsilon)\asymp\varepsilon^{-b}$ except when $q^{\dagger}\geq q$ and
$s=s^{\dagger}$. We substitute in and force the derived rate to be minimax:

$n^{-\frac{1}{2+\frac{db}{q(s-s^{\prime})}+\frac{d}{s-s^{\prime}}}}\asymp
n^{-\frac{1}{2+\frac{d}{s^{\dagger}}}}$

which implies that $b(s)=q(\frac{s-s^{\prime}}{s^{\dagger}}-1)$ and the
corresponding scaling factor
$\kappa\asymp\varepsilon^{-\frac{s-s^{\prime}-\frac{d}{q}}{s^{\dagger}}+1}$.

Next we examine whether the bound
$\varepsilon^{-b(s)}=\varepsilon^{-q(\frac{s-s^{\prime}}{s^{\dagger}}-1)}$ can
be achieved as those $d(\varepsilon)$ in Lemma A.3 . If $q^{\dagger}\geq q$,
setting $b(s)=0$ leads to $s=s^{\prime}+s^{\dagger}$ contradicting with
$s<s^{\dagger}$; $b(s)=\frac{s-s^{\dagger}}{s^{\dagger}-s^{\prime}}(q\wedge
q^{\dagger})$ yields $s=s^{\dagger}+s^{\prime}-(s^{\dagger})^{2}/s^{\prime}$
contradicting with $s>s^{\dagger}$; Lastly, $s=s^{\dagger}$ does not solve
$(-\log\varepsilon)^{(1-\frac{q}{q^{\dagger}})\frac{d}{q(s-s^{\prime})}}\varepsilon^{-\frac{d}{s-s^{\prime}}}=\varepsilon^{-\frac{1}{2+\frac{d}{s^{\dagger}}}}$.
If $q^{\dagger}<q$, $s=s^{\prime}+s^{\dagger}$ does not satisfy $s\leq
s^{\dagger}-\left(\frac{d}{q^{\dagger}}-\frac{d}{q}\right)$; solving
$b(s)=\frac{\frac{s-s^{\dagger}}{d}-\frac{1}{q}+\frac{1}{q^{\dagger}}}{\frac{s^{\dagger}-s^{\prime}}{d}+\frac{1}{q}-\frac{1}{q^{\dagger}}}q$
gives
$s=\frac{s^{\dagger}(s^{\dagger}-s^{\prime})}{s^{\prime}+\left(\frac{d}{q^{\dagger}}-\frac{d}{q}\right)}+s^{\prime}$,
which can be shown
$s>s^{\dagger}-\left(\frac{d}{q^{\dagger}}-\frac{d}{q}\right)$. Hence,
substitute the only feasible $s$ and the minimax rate
$\varepsilon_{n}^{\dagger}$ into the above expression of $\kappa$ and the
scaling factor becomes
$\kappa_{n}\asymp(\varepsilon_{n}^{\dagger})^{-\frac{s^{\dagger}-s^{\prime}}{s^{\prime}+\left(\frac{d}{q^{\dagger}}-\frac{d}{q}\right)}+\frac{d}{qs^{\dagger}}+1}\asymp
n^{-\frac{1}{2s^{\dagger}+d}\left[-\frac{s^{\dagger}(s^{\dagger}-s^{\prime})}{s^{\prime}+\left(\frac{d}{q^{\dagger}}-\frac{d}{q}\right)}+\frac{d}{q}+s^{\dagger}\right]}$.
∎

##  Appendix B Inference

Algorithm 1 White-noise dimension-independent MCMC (wn-$\infty$-MCMC)

1:Initialize current state $u^{(0)}$ and transform it into the whitened space
$\zeta^{(0)}=T^{-1}(u^{(0)})$

2:Sample velocity $\eta^{(0)}\sim{\mathcal{N}}(0,I)$

3:Calculate current energy
$E_{0}=\Phi(\zeta^{(0)})-\frac{\varepsilon^{2}}{8}\|g(\zeta^{(0)})\|^{2}+\frac{1}{2}\log|\mathcal{K}(\zeta^{(0)})|$

4:for $i=0$ to $I-1$ do

5: Run
$\Psi_{\varepsilon}:(\zeta^{(i)},\eta^{(i)})\mapsto(\zeta^{(i+1)},\eta^{(i+1)})$
according to (33).

6: Update the energy $E_{0}\leftarrow E_{0}+\frac{\varepsilon}{2}(\langle
g(\zeta^{(i)}),\eta^{(i)}\rangle+\langle
g(\zeta^{(i+1)}),\eta^{(i+1)}\rangle)$

7:end for

8:Calculate new energy
$E_{1}=\Phi(\zeta^{(I)})-\frac{\varepsilon^{2}}{8}\|g(\zeta^{(I)})\|^{2}+\frac{1}{2}\log|\mathcal{K}(\zeta^{(I)})|$

9:Calculate acceptance probability $a=\exp(-E_{1}+E_{0})$ according to (34).

10:Accept $\zeta^{(I)}$ with probability $a$ for the next state
$\zeta^{\prime}$ or set $\zeta^{\prime}=\zeta^{(0)}$.

11:Record the next state $u^{\prime}=T(\zeta^{\prime})$ in the original space.

See 5.1

###### Proof of Proposition 5.1.

We can compute the joint density of ${\bm{\Xi}}$ and $\kappa$

$\displaystyle
p({\bm{\Xi}},\kappa)=\prod_{\ell=1}^{L}p(\bm{\xi}_{\ell}|\kappa)p(\kappa^{q})$
$\displaystyle=\left(\frac{q}{2}\right)^{L}(2\pi)^{-\frac{JL}{2}}|{\bf
C}_{0}|^{-\frac{L}{2}}\prod_{\ell=1}^{L}r_{0,\ell}^{(\frac{q}{2}-1)\frac{J}{2}}\kappa^{-\frac{q}{2}\cdot\frac{JL}{2}}\exp\left\\{-\kappa^{-\frac{q}{2}}\sum_{\ell=1}^{L}\frac{r_{0,\ell}^{\frac{q}{2}}}{2}\right\\}\frac{\beta^{\alpha}}{\Gamma(\alpha)}\kappa^{-\frac{q}{2}(\alpha+1)}\exp(-\beta\kappa^{-\frac{q}{2}})$
$\displaystyle\propto\left(\kappa^{\frac{q}{2}}\right)^{-(\alpha+\frac{JL}{2}+1)}\exp\left\\{-\kappa^{-\frac{q}{2}}\left(\beta+\frac{1}{2}\sum_{\ell=1}^{L}r_{0,\ell}^{\frac{q}{2}}\right)\right\\}$

By identifying the parameters for $\kappa^{\frac{q}{2}}$ we recognize that
$\kappa^{\frac{q}{2}}|{\bm{\Xi}}$ is another inverse-gamma with parameters
$\alpha^{\prime}$ and $\beta^{\prime}$ as given. ∎

##  Appendix C More Numerical Results

###  C.1 Simulation

Truth |  $q=0.5$ |  $q=1$ (STBP) |  $q=1.5$ |  $q=2$ (STGP)
---|---|---|---|---

---

---

---

---

Figure 10: MAP reconstruction of simulated annulus with $I=256\times 256$,
$J=100$. Columns from left to right: true images, MAP estimates by STBP models
with $q=0.5,1,1.5,2$ respectively. Rows from top to bottom: time step
$t_{j}=0.1,0.3,0.6,0.9$.

Figure 10 illustrates the regularization effect of parameter $q>0$ of STBP
priors in the simulated regression problem of a shrinking annulus. When the
regularization parameter $q$ ranges in $[0,2]$, the smaller $q$ is, the more
regularization it imposes hence the sharper MAP solution the corresponding
model renders compared with the truth. When $q=2$, STBP reduces to STGP which
returns the smoothest reconstruction with blurring boundaries. Even $q=0.5$ is
not in the main range of interest $[1,2]$ where the associated priors have
good properties, e.g. convexity, the resulting prior model still yields a
solution (the second column) with the lowest error among the models for
selective $q$’s.

###  C.2 Dynamic Tomogrpahy Reconstruction

Truth |  Observations |  STBP |  STGP |  time-uncorrelated
---|---|---|---|---

---

---

---

---

Figure 11: Reconstruction results of dynamic STEMPO tomography in the original
space. Columns from left to right: true images, sinograms, MAP estimates by
STBP, STGP and time-uncorrelated models respectively. Rows from top to bottom:
time step $j=0,6,13,19$.

In Figure 11, MAP estimates of the dynamic STEMPO tompography obtained by
optimizing the log-posterior 28 in the original space of ${\bm{\Xi}}$ are
compared among STBP, STGP, and time-uncorrelated models. Although we still
observe the better reconstruction by STBP (the third column) compared with the
other two (the forth and the last columns), these results are generally more
noisy with larger errors compared with those obtained by optimizing (31) in
the whitened space of $\bm{\mathrm{Z}}$, as illustrated in Figure 4. Such
comparison not only supports the superior performance of STBP, but also
highlights the benefit of the white-noise representation (29), which is also
verified in Figure 5.

Truth |  Observations |  STBP |  STGP |  time-uncorrelated
---|---|---|---|---

---

---

---

---

Figure 12: MAP reconstruction for the dynamic emoji tomography in the original
space. Columns from left to right: true images, sinograms, MAP estimates by
STBP, STGP and time-uncorrelated models respectively. Rows from top to bottom:
time step $j=6,14,22,30$.

Similarly in the example of dynamic reconstruction of emoji tomography, MAP
estimates in the original space shown in Figure 12 demonstrate similar
contrast among the three models, STBP, STGP and time-uncorrelated; but are
also more blurry compared with those obtain in the whitened space as in Figure
7.

Finally, Figure 13 compares the MCMC estimates for the emoji tomography
reconstruction in the whitened space by STBP and STGP models. They are all
more noisy compared with MAP estimates plotted in Figure 7 due to the limited
samples. The posterior standard deviations by the STBP model (the forth
column) possess slightly more recognizable spatial feature of smiling faces
than these by the STGP model (the last column).

Truth |  STBP (mean) |  STGP (mean) |  STBP (std) |  STGP (std)
---|---|---|---|---

---

---

---

---

Figure 13: MCMC reconstruction of dynamic emoji tomography in the whitened
space. Columns from left to right: true images, posterior mean estimates by
STBP and STGP, posterior standard deviation estimates by STBP and STGP models
respectively. Rows from top to bottom: time step $j=6,14,22,30$.

###  C.3 Navier-Stokes Inverse Problems

$j=0$ |  $j=6$ |  $j=12$ |  $j=18$ |  $j=24$
---|---|---|---|---

Figure 14: Observed and emulated Navier-Stokes equation solutions. Upper row:
true NSE trajectory solved by classical PDE solver; Lower row: emulated NSE
trajectory by FNO. Left to right: time step $j=0,6,12,18,24$
($t_{j}\in(10,40]$).

In the inverse problem governed by the Navier-Stokes equation (NSE), Figure 14
compares the true trajectory of the vorticity solved by the classical PDE
solver (upper row) for the time window $(10,40]$ with the one emulated by FNO
network (lower row). The negligible difference indicates that the trained FNO
network serves as a very precise emulator of the PDE solver that can
facilitate the Bayesian inference, which requires expensive repeated PDE
forward solving but it is now replaced by cheap emulation.

Truth |  STBP (mean) |  STGP (mean) |  STBP (std) |  STGP (std)
---|---|---|---|---

---

---

---

---

Figure 15: MCMC inverse solutions of Navier-Stokes equation in the whitened
space. Columns from left to right: true vorticity, posterior mean estimates by
STBP and STGP, posterior standard deviation estimates by STBP and STGP models
respectively. Rows from top to bottom: time step $j=0,3,6,9$.

Figure 15 compares the MCMC estimates for the first 10 unit time vorticity of
NSE obtained in the whitened space by STBP and STGP models. Albeit noisy, the
posterior mean estimates by STBP (the second column) manifest spatial features
closer to the truth compared with the those by STGP (the thrid column) model.
For the UQ, posterior standard deviation results are not very informative for
both models, but a few of them at later time ($j=6,9$) show more spatial
traits in the STBP model than in the STGP model.