# Linearization Turns Neural Operators into Function-Valued Gaussian Processes

Emilia Magnani Marvin Pförtner11footnotemark: 1 Tobias Weber11footnotemark: 1
Philipp Hennig
Tübingen AI Center
University of Tübingen
Tübingen, Germany Equal contribution.

###### Abstract

Modeling dynamical systems, e.g. in climate and engineering sciences, often
necessitates solving partial differential equations. Neural operators are deep
neural networks designed to learn nontrivial solution operators of such
differential equations from data. As for all statistical models, the
predictions of these models are imperfect and exhibit errors. Such errors are
particularly difficult to spot in the complex nonlinear behaviour of dynamical
systems. We introduce a new framework for approximate Bayesian uncertainty
quantification in neural operators using function-valued Gaussian processes.
Our approach can be interpreted as a probabilistic analogue of the concept of
currying from functional programming and provides a practical yet
theoretically sound way to apply the linearized Laplace approximation to
neural operators. In a case study on Fourier neural operators, we show that,
even for a discretized input, our method yields a Gaussian closure–a
structured Gaussian process posterior capturing the uncertainty in the output
function of the neural operator, which can be evaluated at an arbitrary set of
points. The method adds minimal prediction overhead, can be applied post-hoc
without retraining the neural operator, and scales to large models and
datasets. We showcase the efficacy of our approach through applications to
different types of partial differential equations.

##  1 Introduction

Partial differential equations (PDEs) are a powerful language for describing
the complex interactions that arise in dynamical systems. Solving PDEs is
therefore a crucial research topic for explaining the dynamics inherent in
physical, biological, and engineering systems. In scientific machine learning,
models predicting physical phenomena such as weather or climate are often
trained to approximate the solution of an underlying PDE from data.

Learning to solve PDEs is closely related to operator learning: Instead of
learning to solve a specific PDE, it can be beneficial to learn the operator
that maps a functional parameter of the PDE (such as initial values, boundary
conditions, force fields, or material parameters) to the solution associated
with the given parameter. This approach is powerful because it learns to solve
entire classes of PDEs simultaneously. Additionally, learning mappings between
function spaces can make models discretization invariant [Kovachki et al.,
2023], which is beneficial for training efficiency. This is in contrast to
classical numerical solvers like finite difference or finite element methods
that discretize the space on a mesh.

Neural operators, and in particular Fourier neural operators (FNOs), stand out
among the deep architectures that are able to effectively learn operators and
are now widely used in practice. Their applications span various domains such
as weather forecasting [Pathak et al., 2022, Bonev et al., 2023], fluid
dynamics [Grady et al., 2022, Renn et al., 2023, Li et al., 2022], and
automotive aerodynamics [Li et al., 2024].

The complex nature of dynamic systems makes errors in predictions difficult to
detect. Uncertainty quantification aims to account for these errors by
providing an estimate of prediction quality. However, current methods in
operator learning are usually not able to provide such an estimate, hindering
their utility in applications.

To address these issues, we develop the _neural operator Laplace
approximation_ (NOLA), a novel framework for approximate Bayesian uncertainty
quantification [Papamarkou et al., 2024] in neural operators using function-
valued Gaussian processes. Our method leverages the concept of currying in
functional programming to enable the application of the linearized Laplace
approximation [Immer et al., 2020] to neural operators. We then show that the
resulting approximate Bayesian posterior over the parameters of the neural
operator induces a function-valued Gaussian process belief over the operator
learned by the network. By considering Fourier neural operators, we show that
NOLA provides a structured Gaussian process posterior, capturing the
uncertainty in the neural operator’s output function and allowing for
efficient evaluation at arbitrary points. NOLA is practical, introduces
minimal additional computational overhead, and can be applied post-hoc,
without the necessity to retrain the neural operator. Moreover, it scales
efficiently to large models and datasets and, just as neural operators, is
applicable to different types of PDEs.

In section 2 we provide a brief overview of neural operators, (multi-output)
Gaussian processes, and the linearized Laplace approximation. In section 3 we
first develop Gaussian processes taking values in (infinite-dimensional)
Banach spaces of functions, as well as the notion of _Gaussian currying_ ,
which formalizes their equivalence to multi-output Gaussian processes. We then
use Gaussian currying to construct function-valued Gaussian processes from
neural operators with Gaussian weight posteriors. We discuss related work in
section 4 and showcase the efficacy on different PDE datasets in section 5.

##  2 Background

###  2.1 Neural operators

_Neural operators_ (NOs) [Kovachki et al., 2023] are neural network
architectures that map between (infinite-dimensional) Banach spaces of
functions. More precisely, a neural operator is a function
${\bm{F}}\colon{\mathbb{A}}\times{\mathbb{W}}\to{\mathbb{U}}$, where

  * •

${\mathbb{A}}$ is a Banach space of functions
${\bm{a}}\colon{\mathbb{D}}_{\mathbb{A}}\to\mathbb{R}^{d_{\mathbb{A}}^{\prime}}$
with domain ${\mathbb{D}}_{\mathbb{A}}\subset\mathbb{R}^{d_{\mathbb{A}}}$,

  * •

${\mathbb{U}}$ is a Banach space of functions
${\bm{u}}\colon{\mathbb{D}}_{\mathbb{U}}\to\mathbb{R}^{d_{\mathbb{U}}^{\prime}}$
with domain ${\mathbb{D}}_{\mathbb{U}}\subset\mathbb{R}^{d_{\mathbb{U}}}$,

  * •

${\mathbb{W}}$ is a set of parameters (typically
${\mathbb{W}}\subset\mathbb{R}^{p}$ or ${\mathbb{W}}\subset\mathbb{C}^{p}$).

To keep the training process computationally tractable, neural operators are
trained on datasets
$\\{({\bm{a}}^{(i)}({\bm{X}}_{{\mathbb{A}}}^{(i)}),{\bm{u}}^{(i)}({\bm{X}}_{{\mathbb{U}}}^{(i)}))\\}_{i=1}^{n}$
consisting of pairs of input and corresponding output functions
$({\bm{a}}^{(i)},{\bm{u}}^{(i)})\in{\mathbb{A}}\times{\mathbb{U}}$ that are
discretized at finitely many points
${\bm{X}}_{{\mathbb{A}}}^{(i)}\in({\mathbb{D}}_{\mathbb{A}})^{n_{\mathbb{A}}^{(i)}}$
and
${\bm{X}}_{{\mathbb{U}}}^{(i)}\in({\mathbb{D}}_{\mathbb{U}})^{n_{\mathbb{U}}^{(i)}}$,
respectively. The training objective is typically given by the empirical risk

$R({\bm{w}})=\frac{1}{n}\sum_{i=1}^{n}L({\bm{u}}^{(i)}({\bm{X}}_{{\mathbb{U}}}^{(i)}),{\bm{F}}({\bm{a}}^{(i)}({\bm{X}}_{{\mathbb{A}}}^{(i)}),{\bm{w}})({\bm{X}}_{{\mathbb{U}}}^{(i)}))$

or a regularized version of the empirical risk. Neural operators have
originally been motivated and are commonly used to learn the solution operator
of non-linear, parametric partial differential equations. In this case, we
typically have ${\mathbb{D}}_{\mathbb{A}}={\mathbb{D}}_{\mathbb{U}}$, the
input functions ${\bm{a}}\in{\mathbb{A}}$ correspond to parameters and/or
initial conditions of the PDE, and the output functions
${\bm{u}}\in{\mathbb{U}}$ are the corresponding solutions of the PDE (at later
time points). There are many different realizations of the abstract neural
operator framework, including low-rank neural operators [Kovachki et al.,
2023], (multipole) graph neural operators [Li et al., 2020a, b], and
(spherical) Fourier neural operators [Li et al., 2020c, Bonev et al., 2023].
Due to their recent popularity, the later parts of this work specifically
address Fourier neural operators.

####  2.1.1 Fourier neural operators

In section 3.2.1 we will focus in particular on _Fourier neural operators_
(FNOs) [Li et al., 2020c], neural operator architectures that apply all
spatially global operations in the spectral domain. An FNO ${\bm{F}}$
transforms a periodic input function ${\bm{a}}$ into a periodic output
function
${\bm{F}}({\bm{a}},{\bm{w}})({\bm{x}})\coloneqq{\bm{q}}({\bm{v}}^{(L)}({\bm{x}}),{\bm{w}}_{{\bm{q}}})$
with

${\bm{v}}^{(l+1)}_{i}({\bm{x}})\coloneqq\sigma^{(l)}\left(\sum_{j=1}^{d_{\bm{v}}^{\prime}}\mathcal{F}^{-1}\left(\left({\bm{R}}^{(l)}_{kij}\mathcal{F}\left({\bm{v}}^{(l)}_{j}\right)_{k}\right)_{k=1}^{k_{\text{max}}}\right)({\bm{x}})+{\bm{W}}^{(l)}_{ij}{\bm{v}}^{(l)}_{j}({\bm{x}})\right)$

for $l=1,\dotsc,L-1$ and
${\bm{v}}^{(1)}({\bm{x}})={\bm{p}}({\bm{a}}({\bm{x}}),{\bm{w}}_{{\bm{p}}})\in\mathbb{R}^{d_{\bm{v}}^{\prime}}$,
where $\mathcal{F}$ denotes the Fourier transform of a periodic
function.111More precisely, the operator
$\mathcal{F}\colon\mathrm{L}_{2}(\mathbb{T}^{d},\mathbb{R})\to\ell_{2}(\mathbb{C})$
maps a real-valued square-integrable function on the $d$-dimensional torus to
the coefficients of the corresponding Fourier series.
${\bm{p}}\colon\mathbb{R}^{d_{\mathbb{A}}^{\prime}}\times{\mathbb{W}}_{\bm{p}}\to\mathbb{R}^{d_{\bm{v}}^{\prime}}$
and
${\bm{q}}\colon\mathbb{R}^{d_{\bm{v}}^{\prime}}\times{\mathbb{W}}_{\bm{q}}\to\mathbb{R}^{d_{\mathbb{U}}^{\prime}}$
are parametric functions called _lifting_ and _projection_ , respectively.
${\bm{R}}^{(l)}\in\mathbb{C}^{k_{\text{max}}\times d_{\bm{v}}^{\prime}\times
d_{\bm{v}}^{\prime}}$ and
${\bm{W}}^{(l)}\in\mathbb{R}^{d_{\bm{v}}^{\prime}\times d_{\bm{v}}^{\prime}}$,
and
${\bm{w}}=({\bm{w}}_{{\bm{p}}},{\bm{W}}^{(1)},{\bm{R}}^{(1)},\dotsc,{\bm{W}}^{(L-1)},{\bm{R}}^{(L-1)},{\bm{w}}_{\bm{q}})$.
The map ${\bm{v}}^{(l)}\to{\bm{v}}^{(l+1)}$ is the $l$-th _Fourier layer_. If
the inputs ${\bm{a}}$ are discretized on a regular grid, $\mathcal{F}$ can be
computed by a real fast Fourier transform ($\operatorname{rfft}$).

###  2.2 (Multi-output) Gaussian processes

As we aim to generalize the notion of a Gaussian process later, we provide the
formal definition of Gaussian processes from mathematical statistics that is
not often used in machine learning. A _Gaussian process_ (GP) with index set
${\mathbb{A}}$ on a probability space $(\Omega,\mathcal{A},\mathrm{P})$ is a
function ${\mathrm{f}}\colon{\mathbb{A}}\times\Omega\to\mathbb{R}$ such that
$\omega\mapsto({\mathrm{f}}(a_{1},\omega),\dotsc,{\mathrm{f}}(a_{n},\omega))$
is an $\mathbb{R}^{n}$-valued Gaussian random variable for all
$n\in\mathbb{N}$ and $a_{1},\dotsc,a_{n}\in{\mathbb{A}}$. We use the shorthand
${\mathrm{f}}(a)\coloneqq{\mathrm{f}}(a,\cdot)$. The _mean function_ of
${\mathrm{f}}$ is given by
$a\mapsto\operatorname{\mathbb{E}}_{\mathrm{P}}\left[{\mathrm{f}}(a)\right]$
and the _covariance function_ of ${\mathrm{f}}$ is given by
$(a_{1},a_{2})\mapsto\operatorname{Cov}_{\mathrm{P}}\left[{\mathrm{f}}(a_{1}),{\mathrm{f}}(a_{2})\right]$.
We denote by ${\mathrm{f}}\sim{\operatorname{\mathcal{GP}}\left(m,k\right)}$
that ${\mathrm{f}}$ is a Gaussian process with mean function $m$ and
covariance function $k$.

If $\mathbb{R}^{{\mathbb{A}}}$ is equipped with the product topology $\tau$,
then $\omega\mapsto{\mathrm{f}}(\cdot,\omega)$ is a function-valued random
variable with values in
$(\mathbb{R}^{{\mathbb{A}}},\mathcal{B}\left(\tau\right))$. This justifies
interpreting GPs as probability measures over functions
${\mathbb{A}}\to\mathbb{R}$.

It is common to extend the concept of a GP to finitely many output dimensions.
A _$d^{\prime}$ -output Gaussian process_ with index set ${\mathbb{A}}$ on a
probability space $(\Omega,\mathcal{A},\mathrm{P})$ is a function
${\bm{\mathrm{f}}}\colon{\mathbb{A}}\times\Omega\to\mathbb{R}^{d^{\prime}}$
such that
$\omega\mapsto\begin{pmatrix}{\bm{\mathrm{f}}}(a_{1},\omega)^{\top}&\cdots&{\bm{\mathrm{f}}}(a_{n},\omega)^{\top}\end{pmatrix}^{\top}$
is an $\mathbb{R}^{n\cdot d^{\prime}}$-valued Gaussian random variable for all
$n\in\mathbb{N}$ and $a_{1},\dotsc,a_{n}\in{\mathbb{A}}$. We use the shorthand
${\bm{\mathrm{f}}}(a)\coloneqq{\bm{\mathrm{f}}}(a,\cdot)$. The _mean function_
of ${\bm{\mathrm{f}}}$ is given by
$a\mapsto\operatorname{\mathbb{E}}_{\mathrm{P}}\left[{\bm{\mathrm{f}}}(a)\right]\in\mathbb{R}^{d^{\prime}}$
and the _covariance function_ of ${\bm{\mathrm{f}}}$ is given by
$(a_{1},a_{2})\mapsto\operatorname{Cov}_{\mathrm{P}}\left[{\bm{\mathrm{f}}}(a_{1}),{\bm{\mathrm{f}}}(a_{2})\right]\in\mathbb{R}^{d^{\prime}\times
d^{\prime}}$. We denote by
${\bm{\mathrm{f}}}\sim{\operatorname{\mathcal{GP}}\left({\bm{m}},{\bm{K}}\right)}$
that ${\bm{\mathrm{f}}}$ is a multi-output Gaussian process with mean function
${\bm{m}}$ and covariance function ${\bm{K}}$. While the notion of a multi-
output Gaussian process might seem more general than the notion of a Gaussian
process, it is possible to “emulate” a function with multiple outputs by
augmenting the input space of a Gaussian process:

######  Lemma 2.1.

Let $(\Omega,\mathcal{A},\mathrm{P})$ be a probability space,
${\bm{\mathrm{f}}}\colon{\mathbb{A}}\times\Omega\to\mathbb{R}^{d^{\prime}}$,
$\mathbb{I}=\\{1,\dotsc,d^{\prime}\\}$, and
${\mathrm{f}}\colon({\mathbb{A}}\times\mathbb{I})\times\Omega\to\mathbb{R}$
with $({\bm{\mathrm{f}}}(a,\cdot))_{i}={\mathrm{f}}((a,i),\cdot)$ for all
$a\in{\mathbb{A}}$ and $i\in\mathbb{I}$ ($\mathrm{P}$-almost surely). Then
${\bm{\mathrm{f}}}\sim{\operatorname{\mathcal{GP}}\left({\bm{m}},{\bm{K}}\right)}$
if and only if
${\mathrm{f}}\sim{\operatorname{\mathcal{GP}}\left(m,k\right)}$, where

$({\bm{m}}(a))_{i}=m(a,i)$

for all $a\in{\mathbb{A}}$ and $i\in\mathbb{I}$, as well as

$({\bm{K}}(a_{1},a_{2}))_{ij}=k((a_{1},i),(a_{2},j))$

for all $a_{1},a_{2}\in{\mathbb{A}}$ and $i,j\in\mathbb{I}$.

###  2.3 Linearized Laplace approximation

The _linearized Laplace approximation_ (LLA) [MacKay, 1992a, b, Immer et al.,
2020] is a conceptually simple, yet effective Daxberger et al. [2021a] method
for obtaining an approximate posterior distribution over the parameters
${\bm{w}}\in\mathbb{R}^{p}$ of a neural network
${\bm{f}}\colon\mathbb{R}^{d}\times\mathbb{R}^{p}\to\mathbb{R}^{d^{\prime}}$.
It applies whenever the objective function $R$ used to train the neural
network is (equivalent to) a negative log-posterior

$R({\bm{w}})=-\log p({\bm{w}}\mid\mathcal{D})=-\log
p({\bm{w}})-\sum_{i=1}^{n}\log
p({\bm{y}}^{(i)}\mid{\bm{f}}({\bm{x}}^{(i)},{\bm{w}}))+\text{const.}$

of the network parameters given data
$\mathcal{D}=\\{({\bm{x}}^{(i)},{\bm{y}}^{(i)})\\}_{i=1}^{n}$. It is common
for the prior over the parameters to be Gaussian, in which case $-\log
p({\bm{w}})$ acts as an $\mathrm{L}2$-regularizer on the parameters. During
training, we attempt to find a local minimum ${\bm{w}}^{\star}$ of the
objective function $R$, i.e. a maximum a-posteriori (MAP) estimator of the
network parameters given the data.

Following Immer et al. [2020], we approximate the posterior of the network
weights as follows: First, we linearize the model using a first-order Taylor
approximation in the weights around the MAP estimator ${\bm{w}}^{\star}$

${\bm{f}}({\bm{x}},{\bm{w}})\approx{\bm{f}}^{\text{lin}}_{{\bm{w}}^{\star}}({\bm{x}},{\bm{w}})\coloneqq{\bm{f}}({\bm{x}},{\bm{w}}^{\star})+\left.\mathrm{D}_{{\bm{w}}}{\bm{f}}({\bm{x}},{\bm{w}})\right|_{{\bm{w}}={\bm{w}}^{\star}}({\bm{w}}-{\bm{w}}^{\star})$

Afterwards, we compute a second-order Taylor approximation of the negative
log-posterior $R^{\text{lin}}_{{\bm{w}}^{\star}}$ of the linearized network at
the MAP ${\bm{w}}^{\star}$

$\displaystyle R^{\text{lin}}_{{\bm{w}}^{\star}}({\bm{w}})$
$\displaystyle\coloneqq-\log p({\bm{w}})-\sum_{i=1}^{n}\log
p({\bm{y}}^{(i)}\mid{\bm{f}}^{\text{lin}}_{{\bm{w}}^{\star}}({\bm{x}}^{(i)},{\bm{w}}))+\text{const.}$
$\displaystyle\approx R({\bm{w}}^{\star})+\underbrace{\nabla
R\left({\bm{w}}^{\star}\right)^{\top}}_{\approx
0}({\bm{w}}-{\bm{w}}^{\star})+\frac{1}{2}({\bm{w}}-{\bm{w}}^{\star})^{\top}{\bm{P}}({\bm{w}}-{\bm{w}}^{\star})$
$\displaystyle=\frac{1}{2}({\bm{w}}-{\bm{w}}^{\star})^{\top}{\bm{P}}({\bm{w}}-{\bm{w}}^{\star})+\text{const.}$
(2.1)

with ${\bm{P}}\coloneqq-\left.\mathrm{H}_{{\bm{w}}}\log
p({\bm{w}})\right|_{{\bm{w}}={\bm{w}}^{\star}}+{\bm{G}}$, where

${\bm{G}}\coloneqq-\sum_{i=1}^{n}{\left.\mathrm{D}_{{\bm{w}}}{\bm{f}}({\bm{x}}^{(i)},{\bm{w}})\right|_{{\bm{w}}={\bm{w}}^{\star}}\left.\mathrm{H}_{{\bm{f}}}\log
p({\bm{y}}^{(i)}\mid{\bm{f}})\right|_{{\bm{f}}={\bm{f}}({\bm{x}}^{(i)},{\bm{w}}^{\star})}\left.\mathrm{D}_{{\bm{w}}}{\bm{f}}({\bm{x}}^{(i)},{\bm{w}})\right|_{{\bm{w}}={\bm{w}}^{\star}}^{\top}}$

is the so-called _generalized Gauss-Newton_ (GGN) matrix [Schraudolph, 2002].
The GGN is guaranteed to be positive-semidefinite. Equation 2.1 is the
negative log-density of a (potentially degenerate) multivariate Gaussian
distribution with mean ${\bm{w}}^{\star}$ and covariance matrix
${\bm{P}}^{\dagger}$, i.e.

$p({\bm{\mathrm{w}}}={\bm{w}}\mid\mathcal{D})=\exp R({\bm{w}})\approx\exp
R^{\text{lin}}_{{\bm{w}}^{\star}}({\bm{w}})\approx{\operatorname{\mathcal{N}}\left({\bm{w}};{\bm{w}}^{\star},{\bm{P}}^{\dagger}\right)}.$

This Gaussian distribution is referred to as the _linearized Laplace
approximation_ of $p({\bm{\mathrm{w}}}={\bm{w}}\mid\mathcal{D})$. Under the
linearized model, the approximate Gaussian posterior over the weights induces
a tractable posterior predictive over the output of the neural network [Khan
et al., 2019, Immer et al., 2020]. More precisely, using closure properties of
Gaussian distributions under affine maps, one can show that the pushforward of
the LLA posterior through
${\bm{w}}\mapsto{\bm{f}}^{\text{lin}}_{{\bm{w}}^{\star}}({\bm{x}},{\bm{w}})$
defines a ($d^{\prime}$-output) Gaussian process

${\bm{\mathrm{f}}}\mid\mathcal{D}\sim{\operatorname{\mathcal{GP}}\left({\bm{f}}(\,\cdot\,,{\bm{w}}^{\star}),({\bm{x}}_{1},{\bm{x}}_{2})\mapsto\left.\mathrm{D}_{{\bm{w}}}{\bm{f}}({\bm{x}}_{1},{\bm{w}})\right|_{{\bm{w}}={\bm{w}}^{\star}}{\bm{P}}^{\dagger}\left.\mathrm{D}_{{\bm{w}}}{\bm{f}}({\bm{x}}_{2},{\bm{w}})\right|_{{\bm{w}}={\bm{w}}^{\star}}^{\top}\right)}.$

##  3 NOLA: The neural operator Laplace approximation

In this section, we develop the _neural operator Laplace approximation_
(NOLA). NOLA constructs (approximate) Bayesian neural operators by applying
linearized Laplace approximation to neural operators post training. It can be
applied to existing trained models as a post-processing step and does not
require expensive retraining. Furthermore, NOLA employs the framework of
function-valued Gaussian processes to quantify uncertainty in the output of
neural operators. To that end, we first develop the concept of a function-
valued Gaussian process and draw an important parallel with currying in
functional programming. figure 1 illustrates the main steps comprising our
methodology.

CurryingGaussianCurryingLinearized LaplaceApproximationNOLAStep 0Step 1Step
2Step
3${\bm{a}}\in{\mathbb{A}}$${\bm{a}}\in{\mathbb{A}}$${\bm{a}}\in{\mathbb{A}}$${\bm{a}}\in{\mathbb{A}}$$\times$${\bm{x}}\in{\mathbb{D}}_{\mathbb{U}}$$\times$${\bm{x}}\in{\mathbb{D}}_{\mathbb{U}}$${\bm{w}}\in{\mathbb{W}}$$\begin{pmatrix}\leavevmode\hbox
to10.4pt{\vbox to11.4pt{\pgfpicture\makeatletter\hbox{\hskip
5.2pt\lower-1.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}\pgfsys@color@rgb@stroke{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{}{{}{{}}{}
{}{}{}\pgfsys@moveto{-5.0pt}{0.0pt}\pgfsys@lineto{5.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} {{}{}}{{}}{} {}{}{}{ { {\pgfsys@beginscope
\pgfsys@setdash{}{0.0pt}\pgfsys@miterjoin{} {} {} {}
\pgfsys@moveto{2.1633pt}{0.0pt}\pgfsys@lineto{0.2pt}{1.14899pt}\pgfsys@lineto{0.0pt}{0.0pt}\pgfsys@lineto{0.2pt}{-1.14899pt}\pgfsys@closepath\pgfsys@fillstroke\pgfsys@endscope}}
}{}{}{{}}{{ {\pgfsys@beginscope
\pgfsys@setdash{}{0.0pt}\pgfsys@miterjoin{}{}{}{}{}{}{}{}
\pgfsys@moveto{2.36916pt}{0.0pt}\pgfsys@curveto{2.36916pt}{0.599pt}{1.88358pt}{1.08458pt}{1.28458pt}{1.08458pt}\pgfsys@curveto{0.68558pt}{1.08458pt}{0.2pt}{0.599pt}{0.2pt}{0.0pt}\pgfsys@curveto{0.2pt}{-0.599pt}{0.68558pt}{-1.08458pt}{1.28458pt}{-1.08458pt}\pgfsys@curveto{1.88358pt}{-1.08458pt}{2.36916pt}{-0.599pt}{2.36916pt}{0.0pt}\pgfsys@closepath\pgfsys@moveto{1.28458pt}{0.0pt}\pgfsys@fillstroke\pgfsys@endscope}}
}{}{}{{}}\pgfsys@moveto{0.0pt}{1.36916pt}\pgfsys@lineto{0.0pt}{7.39032pt}\pgfsys@stroke\pgfsys@invoke{
}{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{
}\pgfsys@transformcm{0.0}{-1.0}{1.0}{0.0}{0.0pt}{1.56915pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}}{{}{{}}{}{}{{}}{{{}}{
{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{0.0}{1.0}{-1.0}{0.0}{0.0pt}{7.49031pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}} } \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\ \leavevmode\hbox
to10.4pt{\vbox to11.4pt{\pgfpicture\makeatletter\hbox{\hskip
5.2pt\lower-1.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}\pgfsys@color@rgb@stroke{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{}{{}{{}}{}
{}{}{}\pgfsys@moveto{-5.0pt}{0.0pt}\pgfsys@lineto{5.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} {{}{}}{{}}{} {}{}{}{
}{}{}{{}}{}{}{}{{}}\pgfsys@moveto{0.0pt}{1.36916pt}\pgfsys@lineto{0.0pt}{7.39032pt}\pgfsys@stroke\pgfsys@invoke{
}{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{
}\pgfsys@transformcm{0.0}{-1.0}{1.0}{0.0}{0.0pt}{1.56915pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}}{{}{{}}{}{}{{}}{{{}}{
{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{0.0}{1.0}{-1.0}{0.0}{0.0pt}{7.49031pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}} } \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\ \leavevmode\hbox
to10.4pt{\vbox to11.4pt{\pgfpicture\makeatletter\hbox{\hskip
5.2pt\lower-1.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}\pgfsys@color@rgb@stroke{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{}{{}{{}}{}
{}{}{}\pgfsys@moveto{-5.0pt}{0.0pt}\pgfsys@lineto{5.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} {{}{}}{{}}{} {}{}{}{
}{}{}{{}}{}{}{}{{}}\pgfsys@moveto{0.0pt}{1.36916pt}\pgfsys@lineto{0.0pt}{7.39032pt}\pgfsys@stroke\pgfsys@invoke{
}{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{
}\pgfsys@transformcm{0.0}{-1.0}{1.0}{0.0}{0.0pt}{1.56915pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}}{{}{{}}{}{}{{}}{{{}}{
{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{0.0}{1.0}{-1.0}{0.0}{0.0pt}{7.49031pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}} } \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\
\vdots\end{pmatrix}$${\bm{w}}\in{\mathbb{W}}$$\begin{pmatrix}\leavevmode\hbox
to10.4pt{\vbox to11.4pt{\pgfpicture\makeatletter\hbox{\hskip
5.2pt\lower-1.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}\pgfsys@color@rgb@stroke{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{}{{}{{}}{}
{}{}{}\pgfsys@moveto{-5.0pt}{0.0pt}\pgfsys@lineto{5.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} {{}{}}{{}}{} {}{}{}{
}{}{}{{}}{}{}{}{{}}\pgfsys@moveto{0.0pt}{1.36916pt}\pgfsys@lineto{0.0pt}{7.39032pt}\pgfsys@stroke\pgfsys@invoke{
}{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{
}\pgfsys@transformcm{0.0}{-1.0}{1.0}{0.0}{0.0pt}{1.56915pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}}{{}{{}}{}{}{{}}{{{}}{
{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{0.0}{1.0}{-1.0}{0.0}{0.0pt}{7.49031pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}} } \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\ \leavevmode\hbox
to10.4pt{\vbox to11.4pt{\pgfpicture\makeatletter\hbox{\hskip
5.2pt\lower-1.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}\pgfsys@color@rgb@stroke{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{}{{}{{}}{}
{}{}{}\pgfsys@moveto{-5.0pt}{0.0pt}\pgfsys@lineto{5.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} {{}{}}{{}}{} {}{}{}{
}{}{}{{}}{}{}{}{{}}\pgfsys@moveto{0.0pt}{1.36916pt}\pgfsys@lineto{0.0pt}{7.39032pt}\pgfsys@stroke\pgfsys@invoke{
}{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{
}\pgfsys@transformcm{0.0}{-1.0}{1.0}{0.0}{0.0pt}{1.56915pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}}{{}{{}}{}{}{{}}{{{}}{
{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{0.0}{1.0}{-1.0}{0.0}{0.0pt}{7.49031pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}} } \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\ \leavevmode\hbox
to10.4pt{\vbox to11.4pt{\pgfpicture\makeatletter\hbox{\hskip
5.2pt\lower-1.2pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}\pgfsys@color@rgb@stroke{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{}{}{{}{{}}{}
{}{}{}\pgfsys@moveto{-5.0pt}{0.0pt}\pgfsys@lineto{5.0pt}{0.0pt}\pgfsys@stroke\pgfsys@invoke{
} {{}{}}{{}}{} {}{}{}{
}{}{}{{}}{}{}{}{{}}\pgfsys@moveto{0.0pt}{1.36916pt}\pgfsys@lineto{0.0pt}{7.39032pt}\pgfsys@stroke\pgfsys@invoke{
}{{}{{}}{}{}{{}}{{{}}{{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgffillcolor}{rgb}{1,1,1}\pgfsys@color@gray@fill{1}\pgfsys@invoke{
}\pgfsys@transformcm{0.0}{-1.0}{1.0}{0.0}{0.0pt}{1.56915pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}}{{}{{}}{}{}{{}}{{{}}{
{{}}{\pgfsys@beginscope\pgfsys@invoke{
}\pgfsys@transformcm{0.0}{1.0}{-1.0}{0.0}{0.0pt}{7.49031pt}\pgfsys@invoke{
}\pgfsys@invoke{ \lxSVG@closescope }\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope}}{{}}}} } \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\
\vdots\end{pmatrix}$${\bm{\mathrm{w}}}\in{\mathbb{W}}$$\begin{pmatrix}\leavevmode\hbox
to9.98pt{\vbox to10.29pt{\pgfpicture\makeatletter\hbox{\hskip
5.2pt\lower-0.08891pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}\pgfsys@color@rgb@stroke{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{}{}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}
{}{}\pgfsys@moveto{-5.0pt}{0.11108pt}\pgfsys@lineto{-4.58328pt}{0.22797pt}\pgfsys@lineto{-4.16656pt}{0.43945pt}\pgfsys@lineto{-3.74985pt}{0.79575pt}\pgfsys@lineto{-3.33313pt}{1.35376pt}\pgfsys@lineto{-2.91641pt}{2.16324pt}\pgfsys@lineto{-2.4997pt}{3.24738pt}\pgfsys@lineto{-2.08298pt}{4.57947pt}\pgfsys@lineto{-1.66626pt}{6.06659pt}\pgfsys@lineto{-1.24954pt}{7.55005pt}\pgfsys@lineto{-0.83282pt}{8.82645pt}\pgfsys@lineto{-0.4161pt}{9.69315pt}\pgfsys@lineto{0.00061pt}{10.00002pt}\pgfsys@lineto{0.41733pt}{9.69147pt}\pgfsys@lineto{0.83405pt}{8.82324pt}\pgfsys@lineto{1.25076pt}{7.54562pt}\pgfsys@lineto{1.66748pt}{6.06232pt}\pgfsys@lineto{2.0842pt}{4.57535pt}\pgfsys@lineto{2.50092pt}{3.24387pt}\pgfsys@lineto{2.91763pt}{2.16049pt}\pgfsys@lineto{3.33435pt}{1.35162pt}\pgfsys@lineto{3.75107pt}{0.79453pt}\pgfsys@lineto{4.16779pt}{0.43854pt}\pgfsys@lineto{4.5845pt}{0.22751pt}\pgfsys@stroke\pgfsys@invoke{
} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\ \leavevmode\hbox
to9.98pt{\vbox to10.29pt{\pgfpicture\makeatletter\hbox{\hskip
5.2pt\lower-0.08891pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}\pgfsys@color@rgb@stroke{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{}{}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}
{}{}\pgfsys@moveto{-5.0pt}{0.11108pt}\pgfsys@lineto{-4.58328pt}{0.22797pt}\pgfsys@lineto{-4.16656pt}{0.43945pt}\pgfsys@lineto{-3.74985pt}{0.79575pt}\pgfsys@lineto{-3.33313pt}{1.35376pt}\pgfsys@lineto{-2.91641pt}{2.16324pt}\pgfsys@lineto{-2.4997pt}{3.24738pt}\pgfsys@lineto{-2.08298pt}{4.57947pt}\pgfsys@lineto{-1.66626pt}{6.06659pt}\pgfsys@lineto{-1.24954pt}{7.55005pt}\pgfsys@lineto{-0.83282pt}{8.82645pt}\pgfsys@lineto{-0.4161pt}{9.69315pt}\pgfsys@lineto{0.00061pt}{10.00002pt}\pgfsys@lineto{0.41733pt}{9.69147pt}\pgfsys@lineto{0.83405pt}{8.82324pt}\pgfsys@lineto{1.25076pt}{7.54562pt}\pgfsys@lineto{1.66748pt}{6.06232pt}\pgfsys@lineto{2.0842pt}{4.57535pt}\pgfsys@lineto{2.50092pt}{3.24387pt}\pgfsys@lineto{2.91763pt}{2.16049pt}\pgfsys@lineto{3.33435pt}{1.35162pt}\pgfsys@lineto{3.75107pt}{0.79453pt}\pgfsys@lineto{4.16779pt}{0.43854pt}\pgfsys@lineto{4.5845pt}{0.22751pt}\pgfsys@stroke\pgfsys@invoke{
} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\ \leavevmode\hbox
to9.98pt{\vbox to10.29pt{\pgfpicture\makeatletter\hbox{\hskip
5.2pt\lower-0.08891pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}\pgfsys@color@rgb@stroke{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{}{}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}
{}{}\pgfsys@moveto{-5.0pt}{0.11108pt}\pgfsys@lineto{-4.58328pt}{0.22797pt}\pgfsys@lineto{-4.16656pt}{0.43945pt}\pgfsys@lineto{-3.74985pt}{0.79575pt}\pgfsys@lineto{-3.33313pt}{1.35376pt}\pgfsys@lineto{-2.91641pt}{2.16324pt}\pgfsys@lineto{-2.4997pt}{3.24738pt}\pgfsys@lineto{-2.08298pt}{4.57947pt}\pgfsys@lineto{-1.66626pt}{6.06659pt}\pgfsys@lineto{-1.24954pt}{7.55005pt}\pgfsys@lineto{-0.83282pt}{8.82645pt}\pgfsys@lineto{-0.4161pt}{9.69315pt}\pgfsys@lineto{0.00061pt}{10.00002pt}\pgfsys@lineto{0.41733pt}{9.69147pt}\pgfsys@lineto{0.83405pt}{8.82324pt}\pgfsys@lineto{1.25076pt}{7.54562pt}\pgfsys@lineto{1.66748pt}{6.06232pt}\pgfsys@lineto{2.0842pt}{4.57535pt}\pgfsys@lineto{2.50092pt}{3.24387pt}\pgfsys@lineto{2.91763pt}{2.16049pt}\pgfsys@lineto{3.33435pt}{1.35162pt}\pgfsys@lineto{3.75107pt}{0.79453pt}\pgfsys@lineto{4.16779pt}{0.43854pt}\pgfsys@lineto{4.5845pt}{0.22751pt}\pgfsys@stroke\pgfsys@invoke{
} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\
\vdots\end{pmatrix}$${\bm{\mathrm{w}}}\in{\mathbb{W}}$$\begin{pmatrix}\leavevmode\hbox
to9.98pt{\vbox to10.29pt{\pgfpicture\makeatletter\hbox{\hskip
5.2pt\lower-0.08891pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}\pgfsys@color@rgb@stroke{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{}{}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}
{}{}\pgfsys@moveto{-5.0pt}{0.11108pt}\pgfsys@lineto{-4.58328pt}{0.22797pt}\pgfsys@lineto{-4.16656pt}{0.43945pt}\pgfsys@lineto{-3.74985pt}{0.79575pt}\pgfsys@lineto{-3.33313pt}{1.35376pt}\pgfsys@lineto{-2.91641pt}{2.16324pt}\pgfsys@lineto{-2.4997pt}{3.24738pt}\pgfsys@lineto{-2.08298pt}{4.57947pt}\pgfsys@lineto{-1.66626pt}{6.06659pt}\pgfsys@lineto{-1.24954pt}{7.55005pt}\pgfsys@lineto{-0.83282pt}{8.82645pt}\pgfsys@lineto{-0.4161pt}{9.69315pt}\pgfsys@lineto{0.00061pt}{10.00002pt}\pgfsys@lineto{0.41733pt}{9.69147pt}\pgfsys@lineto{0.83405pt}{8.82324pt}\pgfsys@lineto{1.25076pt}{7.54562pt}\pgfsys@lineto{1.66748pt}{6.06232pt}\pgfsys@lineto{2.0842pt}{4.57535pt}\pgfsys@lineto{2.50092pt}{3.24387pt}\pgfsys@lineto{2.91763pt}{2.16049pt}\pgfsys@lineto{3.33435pt}{1.35162pt}\pgfsys@lineto{3.75107pt}{0.79453pt}\pgfsys@lineto{4.16779pt}{0.43854pt}\pgfsys@lineto{4.5845pt}{0.22751pt}\pgfsys@stroke\pgfsys@invoke{
} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\ \leavevmode\hbox
to9.98pt{\vbox to10.29pt{\pgfpicture\makeatletter\hbox{\hskip
5.2pt\lower-0.08891pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}\pgfsys@color@rgb@stroke{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{}{}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}
{}{}\pgfsys@moveto{-5.0pt}{0.11108pt}\pgfsys@lineto{-4.58328pt}{0.22797pt}\pgfsys@lineto{-4.16656pt}{0.43945pt}\pgfsys@lineto{-3.74985pt}{0.79575pt}\pgfsys@lineto{-3.33313pt}{1.35376pt}\pgfsys@lineto{-2.91641pt}{2.16324pt}\pgfsys@lineto{-2.4997pt}{3.24738pt}\pgfsys@lineto{-2.08298pt}{4.57947pt}\pgfsys@lineto{-1.66626pt}{6.06659pt}\pgfsys@lineto{-1.24954pt}{7.55005pt}\pgfsys@lineto{-0.83282pt}{8.82645pt}\pgfsys@lineto{-0.4161pt}{9.69315pt}\pgfsys@lineto{0.00061pt}{10.00002pt}\pgfsys@lineto{0.41733pt}{9.69147pt}\pgfsys@lineto{0.83405pt}{8.82324pt}\pgfsys@lineto{1.25076pt}{7.54562pt}\pgfsys@lineto{1.66748pt}{6.06232pt}\pgfsys@lineto{2.0842pt}{4.57535pt}\pgfsys@lineto{2.50092pt}{3.24387pt}\pgfsys@lineto{2.91763pt}{2.16049pt}\pgfsys@lineto{3.33435pt}{1.35162pt}\pgfsys@lineto{3.75107pt}{0.79453pt}\pgfsys@lineto{4.16779pt}{0.43854pt}\pgfsys@lineto{4.5845pt}{0.22751pt}\pgfsys@stroke\pgfsys@invoke{
} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\ \leavevmode\hbox
to9.98pt{\vbox to10.29pt{\pgfpicture\makeatletter\hbox{\hskip
5.2pt\lower-0.08891pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}\pgfsys@color@rgb@stroke{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.48828125}{0.6484375}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}{}{}{{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}{}
{}{}\pgfsys@moveto{-5.0pt}{0.11108pt}\pgfsys@lineto{-4.58328pt}{0.22797pt}\pgfsys@lineto{-4.16656pt}{0.43945pt}\pgfsys@lineto{-3.74985pt}{0.79575pt}\pgfsys@lineto{-3.33313pt}{1.35376pt}\pgfsys@lineto{-2.91641pt}{2.16324pt}\pgfsys@lineto{-2.4997pt}{3.24738pt}\pgfsys@lineto{-2.08298pt}{4.57947pt}\pgfsys@lineto{-1.66626pt}{6.06659pt}\pgfsys@lineto{-1.24954pt}{7.55005pt}\pgfsys@lineto{-0.83282pt}{8.82645pt}\pgfsys@lineto{-0.4161pt}{9.69315pt}\pgfsys@lineto{0.00061pt}{10.00002pt}\pgfsys@lineto{0.41733pt}{9.69147pt}\pgfsys@lineto{0.83405pt}{8.82324pt}\pgfsys@lineto{1.25076pt}{7.54562pt}\pgfsys@lineto{1.66748pt}{6.06232pt}\pgfsys@lineto{2.0842pt}{4.57535pt}\pgfsys@lineto{2.50092pt}{3.24387pt}\pgfsys@lineto{2.91763pt}{2.16049pt}\pgfsys@lineto{3.33435pt}{1.35162pt}\pgfsys@lineto{3.75107pt}{0.79453pt}\pgfsys@lineto{4.16779pt}{0.43854pt}\pgfsys@lineto{4.5845pt}{0.22751pt}\pgfsys@stroke\pgfsys@invoke{
} } \pgfsys@invoke{\lxSVG@closescope }\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}\\\
\vdots\end{pmatrix}$${\bm{u}}\in{\mathbb{U}}$${\bm{u}}({\color[rgb]{0.6875,0.4296875,0.58984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.6875,0.4296875,0.58984375}{\bm{x}}})\in\mathbb{R}^{d_{\mathbb{U}}^{\prime}}$${\bm{\mathrm{u}}}({\color[rgb]{0.6875,0.4296875,0.58984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.6875,0.4296875,0.58984375}{\bm{x}}})\in\mathbb{R}^{d_{\mathbb{U}}^{\prime}}$${\bm{\mathrm{u}}}\in{\mathbb{U}}$$\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0.70703125,0.62890625,0.41015625}\definecolor[named]{pgfstrokecolor}{rgb}{0.70703125,0.62890625,0.41015625}{\bm{u}}}\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}={\bm{F}}(\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0,0.41015625,0.66796875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.41015625,0.66796875}{\bm{a}}}\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}},\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0.48828125,0.6484375,0.29296875}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}{\bm{w}}}\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}})$$\begin{aligned}
\leavevmode\hbox to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip
0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0.70703125,0.62890625,0.41015625}\definecolor[named]{pgfstrokecolor}{rgb}{0.70703125,0.62890625,0.41015625}{\bm{u}}}({\color[rgb]{0.6875,0.4296875,0.58984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.6875,0.4296875,0.58984375}{\bm{x}}})\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&={\bm{f}}((\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0,0.41015625,0.66796875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.41015625,0.66796875}{\bm{a}}}\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}},\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0.6875,0.4296875,0.58984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.6875,0.4296875,0.58984375}{\bm{x}}}\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}),\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0.48828125,0.6484375,0.29296875}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}{\bm{w}}}\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}})\\\
&={\bm{F}}({\color[rgb]{0,0.41015625,0.66796875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.41015625,0.66796875}{\bm{a}}},{\color[rgb]{0.48828125,0.6484375,0.29296875}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}{\bm{w}}})({\color[rgb]{0.6875,0.4296875,0.58984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.6875,0.4296875,0.58984375}{\bm{x}}})\end{aligned}$$\begin{aligned}
\leavevmode\hbox to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip
0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0.70703125,0.62890625,0.41015625}\definecolor[named]{pgfstrokecolor}{rgb}{0.70703125,0.62890625,0.41015625}{\bm{\mathrm{u}}}}({\color[rgb]{0.6875,0.4296875,0.58984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.6875,0.4296875,0.58984375}{\bm{x}}})\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&={\bm{\mathrm{f}}}(\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0,0.41015625,0.66796875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.41015625,0.66796875}{\bm{a}}}\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}},\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0.6875,0.4296875,0.58984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.6875,0.4296875,0.58984375}{\bm{x}}}\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}})\\\
&={\bm{f}}^{\text{lin}}_{\bm{\mu}}(({\color[rgb]{0,0.41015625,0.66796875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.41015625,0.66796875}{\bm{a}}},{\color[rgb]{0.6875,0.4296875,0.58984375}\definecolor[named]{pgfstrokecolor}{rgb}{0.6875,0.4296875,0.58984375}{\bm{x}}}),\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0.48828125,0.6484375,0.29296875}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}{\bm{\mathrm{w}}}}\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}})\end{aligned}$$\begin{aligned}
\leavevmode\hbox to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip
0.0pt\lower 0.0pt\hbox to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0.70703125,0.62890625,0.41015625}\definecolor[named]{pgfstrokecolor}{rgb}{0.70703125,0.62890625,0.41015625}{\bm{\mathrm{u}}}}\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}&={\mathrm{F}}(\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0,0.41015625,0.66796875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.41015625,0.66796875}{\bm{a}}}\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}})={\bm{\mathrm{f}}}({\color[rgb]{0,0.41015625,0.66796875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.41015625,0.66796875}{\bm{a}}},\,\cdot\,)\\\
&={\bm{f}}^{\text{lin}}_{\bm{\mu}}(({\color[rgb]{0,0.41015625,0.66796875}\definecolor[named]{pgfstrokecolor}{rgb}{0,0.41015625,0.66796875}{\bm{a}}},\,\cdot\,),\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}}{\color[rgb]{0.48828125,0.6484375,0.29296875}\definecolor[named]{pgfstrokecolor}{rgb}{0.48828125,0.6484375,0.29296875}{\bm{\mathrm{w}}}}\leavevmode\hbox
to0pt{\vbox to0pt{\pgfpicture\makeatletter\hbox{\hskip 0.0pt\lower 0.0pt\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{
}\definecolor[named]{pgfstrokecolor}{rgb}{0.1953125,0.25390625,0.29296875}\pgfsys@color@rgb@stroke{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@color@rgb@fill{0.1953125}{0.25390625}{0.29296875}\pgfsys@invoke{
}\pgfsys@setlinewidth{0.4pt}\pgfsys@invoke{ }\nullfont\hbox
to0.0pt{\pgfsys@beginscope\pgfsys@invoke{ }{} \pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hbox
to0.0pt{}{}{}{}\hss}\pgfsys@discardpath\pgfsys@invoke{\lxSVG@closescope
}\pgfsys@endscope\hss}}\lxSVG@closescope\endpgfpicture}})\end{aligned}$ Figure
1: Graphical illustration of the steps involved in computing a _neural
operator Laplace approximation_ (NOLA). A trained neural operator ${\bm{F}}$
(top left) is converted into an equivalent neural network ${\bm{f}}$ with
outputs in $\mathbb{R}^{d_{\mathbb{U}}^{\prime}}$ using (reverse) currying
(top right). It is then possible to apply the linearized Laplace approximation
to ${\bm{f}}$, which produces a Gaussian process posterior ${\bm{\mathrm{f}}}$
quantifying the uncertainty about the function learned by ${\bm{f}}$ (bottom
right). Finally, we can use Gaussian currying to transform ${\bm{\mathrm{f}}}$
into a function-valued Gaussian process posterior $\bm{\mathrm{F}}$ over the
operator learned by the neural operator ${\bm{F}}$ (bottom left).

###  3.1 Function-valued Gaussian processes and Gaussian Currying

As seen in section 2.3, given a Gaussian belief over the parameters of a
neural network
${\bm{f}}\colon\mathbb{R}^{d}\times\mathbb{R}^{p}\to\mathbb{R}^{d^{\prime}}$,
model linearization yields a (multi-output) Gaussian process belief over the
function learned by the neural network. However, this is not immediately
applicable to neural operators, since their outputs do not lie in
$\mathbb{R}^{d^{\prime}}$, but in a potentially infinite-dimensional Banach
space of functions. Hence, we need to generalize (multi-output) Gaussian
processes to the notion of a _Banach-valued Gaussian process_.
definitionDefBanachValuedGaussianProcess Let ${\mathbb{U}}$ be a real
separable Banach space. A _${\mathbb{U}}$ -valued Gaussian process_ with index
set ${\mathbb{A}}$ on a probability space $(\Omega,\mathcal{A},\mathrm{P})$ is
a function ${\mathrm{F}}\colon{\mathbb{A}}\times\Omega\to{\mathbb{U}}$ such
that
$\omega\mapsto({\mathrm{F}}(a_{1},\omega),\dotsc,{\mathrm{F}}(a_{n},\omega))$
is a joint222See remark A.1., i.e.
$({\mathbb{U}}^{n},\mathcal{B}\left({\mathbb{U}}\right)^{\otimes n})$-valued,
Gaussian random variable for all $n\in\mathbb{N}$ and
$a_{1},\dotsc,a_{n}\in{\mathbb{A}}$. As above, we use the shorthand
${\mathrm{F}}(a)\coloneqq{\mathrm{F}}(a,\cdot)$. Moreover, if we equip the
vector space ${\mathbb{U}}^{{\mathbb{A}}}$ of (linear and non-linear)
operators ${\mathbb{A}}\to{\mathbb{U}}$ with the product topology $\tau$, then
the map $\omega\mapsto{\mathrm{F}}(\cdot,\omega)$ is a Gaussian random
variable with values in
$({\mathbb{U}}^{{\mathbb{A}}},\mathcal{B}\left(\tau\right))$. This warrants
the interpretation of ${\mathbb{U}}$-valued Gaussian processes as _Gaussian
random operators_.

In the context of neural operators, ${\mathbb{U}}$ is a Banach space of
$\mathbb{R}^{d_{\mathbb{U}}^{\prime}}$-valued functions on a common domain
${\mathbb{D}}_{\mathbb{U}}$. In this case, we can show that
${\mathbb{U}}$-valued Gaussian processes are closely related to multi-output
Gaussian processes with an augmented input space. This is in analogy to lemma
2.1, but requires some additional technical assumptions.

######  Theorem 3.1 (Proof in section A.2).

Let $(\Omega,\mathcal{A},\mathrm{P})$ be a probability space and
${\mathbb{U}}$ a real separable Banach space of
$\mathbb{R}^{d^{\prime}}$-valued functions with domain
${\mathbb{D}}_{\mathbb{U}}$ such that all evaluation maps
$\delta_{\bm{x}}\colon{\mathbb{U}}\to\mathbb{R}^{d^{\prime}},{\bm{u}}\mapsto{\bm{u}}({\bm{x}})$
are continuous. Let
$\bm{\mathrm{F}}\colon{\mathbb{A}}\times\Omega\to{\mathbb{U}}$ and
${\bm{\mathrm{f}}}\colon({\mathbb{A}}\times{\mathbb{D}}_{\mathbb{U}})\times\Omega\to\mathbb{R}^{d^{\prime}}$
such that
$\bm{\mathrm{F}}({\bm{a}},\cdot)({\bm{x}})={\bm{\mathrm{f}}}(({\bm{a}},{\bm{x}}),\cdot)$
for all ${\bm{a}}\in{\mathbb{A}}$ and ${\bm{x}}\in{\mathbb{D}}_{\mathbb{U}}$
($\mathrm{P}$-almost surely). Then

  1. (i)

${\bm{\mathrm{f}}}$ is a $d^{\prime}$-output Gaussian process if
$\bm{\mathrm{F}}$ is a ${\mathbb{U}}$-valued Gaussian process,

and, if assumption A.1 holds for
$\mathbb{L}_{\delta}\coloneqq\\{{\bm{u}}\mapsto\delta_{\bm{x}}({\bm{u}})_{i}\colon{\bm{x}}\in{\mathbb{D}}_{\mathbb{U}}\land
i=1,\dotsc,d^{\prime}\\}$,

  1. (ii)

$\bm{\mathrm{F}}$ is a ${\mathbb{U}}$-valued Gaussian process if
${\bm{\mathrm{f}}}$ is a $d^{\prime}$-output Gaussian process.

theorem 3.1 provides a valuable insight into the abstract concept of function-
valued Gaussian processes. It establishes that function-valued Gaussian
processes are equivalent to (multi-output) Gaussian processes with augmented
input spaces. This equivalence enables the translation of abstract function-
valued objects into computationally feasible structures (real-valued Gaussian
processes).

##### Gaussian Currying

As a useful intuition, we note that theorem 3.1 constitutes a probabilistic
analogue of the concept of _currying_ from functional programming (and
category theory more generally). The theorem shows the equivalence of the
(vector-valued) Gaussian random function
${\bm{\mathrm{f}}}\colon{\mathbb{A}}\times{\mathbb{D}}_{\mathbb{U}}\to\mathbb{R}^{d^{\prime}}$
and the Gaussian random operator
$\bm{\mathrm{F}}\colon{\mathbb{A}}\to({\mathbb{D}}_{\mathbb{U}}\to\mathbb{R}^{d^{\prime}})$
with
$\bm{\mathrm{F}}({\bm{a}})({\bm{x}})\stackrel{{\scriptstyle\text{a.s.}}}{{=}}{\bm{\mathrm{f}}}({\bm{a}},{\bm{x}})$.

######  Example 3.1 (Currying a Continuous Bivariate Gaussian Process).

Let ${\mathrm{f}}\sim{\operatorname{\mathcal{GP}}\left(m,k\right)}$ be a
bivariate 2-output Gaussian process with index set $\mathbb{R}^{2}$ on
$(\Omega,\mathcal{A},\mathrm{P})$ with ($\mathrm{P}$-almost surely) continuous
paths. For instance, this assumption is fulfilled if $m$ is continuous and $k$
is a multivariate Matérn covariance function [Da Costa et al., 2023]. Then
$a\mapsto{\mathrm{f}}(a,\cdot)$ is a function-valued Gaussian process. More
precisely, theorem 3.1 shows that the map
${\mathrm{F}}\colon\mathbb{R}\times\Omega\to
C(\mathbb{R}),(a,\omega)\mapsto(x\mapsto{\mathrm{f}}((a,x),\omega))$ is a
$C(\mathbb{R})$-valued Gaussian process.

Thus, an intuitive way to understand function-valued Gaussian processes is as
objects that, when evaluated, return a Gaussian process. Currying can also be
used to relate the mean and covariance functions of function-valued (or more
generally Banach-valued) Gaussian processes, and their counterparts defined on
the corresponding multi-output Gaussian process. The rather technical
definitions of mean and covariance functions of Banach-valued Gaussian
processes are in section A.2.

###  3.2 Linearization turns neural operators into function-valued Gaussian
processes

Having introduced function-valued Gaussian processes in the previous section,
we can now use the notion to build NOLA, i.e. construct (approximate) Bayesian
neural operators. We delineate the key components of our methodology into
different steps, which are visually represented in figure 1.

##### Step 0

Let
${\bm{F}}\colon{\mathbb{A}}\times{\mathbb{W}}\to{\mathbb{U}}\subset(\mathbb{R}^{d_{\mathbb{U}}^{\prime}})^{{\mathbb{D}}_{\mathbb{U}}}$
be a neural operator as in section 2.1 with ${\mathbb{W}}=\mathbb{R}^{p}$.

##### Step 1

Using currying on ${\bm{F}}$, we define the function

${\bm{f}}\colon({\mathbb{A}}\times{\mathbb{D}}_{\mathbb{U}})\times{\mathbb{W}}\to\mathbb{R}^{d_{\mathbb{U}}^{\prime}},(({\bm{a}},{\bm{x}}),{\bm{w}})\mapsto{\bm{F}}({\bm{a}},{\bm{w}})({\bm{x}}).$

##### Step 2

Assume that we have a posterior Gaussian belief
${\bm{\mathrm{w}}}\sim{\operatorname{\mathcal{N}}\left({\bm{\mu}},{\bm{\Sigma}}\right)}$
over the parameters of the network. In this work, we will obtain the posterior
over ${\bm{\mathrm{w}}}$ via Laplace approximation, but it could also have
originated from other (approximate) inference schemes such as variational
inference or (approximate) moment matching. Since ${\bm{f}}$ has values in
$\mathbb{R}^{d_{\mathbb{U}}^{\prime}}$, we can, as in section 2.3, linearize
the model around ${\bm{\mu}}$:

${\bm{f}}(({\bm{a}},{\bm{x}}),{\bm{w}})\approx{\bm{f}}^{\text{lin}}_{\bm{\mu}}(({\bm{a}},{\bm{x}}),{\bm{w}})\coloneqq{\bm{f}}(({\bm{a}},{\bm{x}}),{\bm{\mu}})+\left.\mathrm{D}_{{\bm{w}}}{\bm{f}}(({\bm{a}},{\bm{x}}),{\bm{w}})\right|_{{\bm{w}}={\bm{\mu}}}({\bm{w}}-{\bm{\mu}})$

to arrive at an induced approximate $d_{\mathbb{U}}^{\prime}$-output Gaussian
process belief with index set ${\mathbb{A}}\times{\mathbb{D}}_{\mathbb{U}}$

${\bm{\mathrm{f}}}\sim{\operatorname{\mathcal{GP}}\left({\bm{f}}(\cdot,{\bm{\mu}}),(({\bm{a}}_{1},{\bm{x}}_{1}),({\bm{a}}_{2},{\bm{x}}_{2}))\mapsto\left.\mathrm{D}_{{\bm{w}}}{\bm{f}}(({\bm{a}}_{1},{\bm{x}}_{1}),{\bm{w}})\right|_{{\bm{w}}={\bm{\mu}}}{\bm{\Sigma}}\left.\mathrm{D}_{{\bm{w}}}{\bm{f}}(({\bm{a}}_{2},{\bm{x}}_{2}),{\bm{w}})\right|_{{\bm{w}}={\bm{\mu}}}^{\top}\right)}.$

##### Step 3

Finally, we can use Gaussian currying to construct a Gaussian random operator
from ${\bm{\mathrm{f}}}$. Namely, we define the function

$\bm{\mathrm{F}}\colon{\mathbb{A}}\times\Omega\to{\mathbb{U}},({\bm{a}},\omega)\mapsto({\bm{x}}\mapsto{\bm{\mathrm{f}}}(({\bm{a}},{\bm{x}}),\omega)).$

Under some technical assumptions about ${\mathbb{U}}$, theorem 3.1 then shows
that $\bm{\mathrm{F}}$ is a ${\mathbb{U}}$-valued Gaussian process. For the
spaces ${\mathbb{U}}$ considered in the context of neural operators, these
assumptions are virtually always met. Moreover,

$\displaystyle\operatorname{\mathbb{E}}\left[\bm{\mathrm{F}}({\bm{a}})({\bm{x}})\right]$
$\displaystyle={\bm{F}}({\bm{a}},{\bm{\mu}})({\bm{x}}),\qquad\qquad\text{and}$
$\displaystyle\operatorname{Cov}\left[\bm{\mathrm{F}}({\bm{a}}_{1})({\bm{x}}_{1}),\bm{\mathrm{F}}({\bm{a}}_{2})({\bm{x}}_{2})\right]$
$\displaystyle=\left.\mathrm{D}_{{\bm{w}}}{\bm{F}}({\bm{a}}_{1},{\bm{w}})({\bm{x}}_{1})\right|_{{\bm{w}}={\bm{\mu}}}{\bm{\Sigma}}\left.\mathrm{D}_{{\bm{w}}}{\bm{F}}({\bm{a}}_{2},{\bm{w}})({\bm{x}}_{2})\right|_{{\bm{w}}={\bm{\mu}}}^{\top}.$

####  3.2.1 Case study: Bayesian Fourier neural operators

The above exposition applies generally to operator-valued models. Applying it
specifically to Fourier neural operators leads to a particularly efficient
representation of the function-valued posterior process. For simplicity of
exposition, we limit ourselves to a so-called last-layer Laplace
approximation, in which only the parameters
${\bm{w}}_{L-1}\coloneqq({\bm{R}}^{(L-1)},{\bm{W}}^{(L-1)})$ are treated
probabilistically [Kristiadi et al., 2020]. However, we would like to point
out that it is possible to proceed with a full Laplace approximation over all
parameters ${\bm{w}}$. For an input ${\bm{a}}\in{\mathbb{A}}$, we can
factorize the FNO as

${\bm{F}}({\bm{a}},{\bm{w}})({\bm{x}})=\underbrace{({\bm{q}}(\cdot,{\bm{w}}_{{\bm{q}}})\circ\sigma^{(L-1)})}_{\eqqcolon\tilde{{\bm{q}}}}\left({\bm{z}}^{(L-1)}({\bm{x}},{\bm{w}}_{L-1})\right),$

with

$\displaystyle{\bm{z}}^{(L-1)}_{i}({\bm{x}},{\bm{w}}_{L-1})$
$\displaystyle\coloneqq\sum_{j=1}^{d_{\bm{v}}^{\prime}}\mathcal{F}^{-1}\left({\bm{R}}^{(L-1)}_{:ij}\odot\hat{{\bm{v}}}^{(L-1)}_{:j}\right)({\bm{x}})+\sum_{j=1}^{d_{\bm{v}}^{\prime}}{\bm{W}}_{ij}^{(L-1)}{\bm{v}}^{(L-1)}_{j}({\bm{x}})$
$\displaystyle=\sum_{j=1}^{d_{\bm{v}}^{\prime}}\sum_{k=1}^{k_{\text{max}}}\operatorname{Re}({\bm{R}}^{(L-1)}_{kij})\underbrace{\operatorname{Re}(\hat{{\bm{v}}}^{(L-1)}_{kj})\cos\left(\langle\omega_{k},{\bm{x}}\rangle\right)}_{\eqqcolon\phi_{kj}({\bm{x}})}$
$\displaystyle\qquad+\sum_{j=1}^{d_{\bm{v}}^{\prime}}\sum_{k=1}^{k_{\text{max}}}\operatorname{Im}({\bm{R}}^{(L-1)}_{kij})\underbrace{(-1)\operatorname{Im}(\hat{{\bm{v}}}^{(L-1)}_{kj})\sin\left(\langle\omega_{k},{\bm{x}}\rangle\right)}_{\eqqcolon\varphi_{kj}({\bm{x}})}$
$\displaystyle\qquad+\sum_{j=1}^{d_{\bm{v}}^{\prime}}{\bm{W}}_{ij}^{(L-1)}\underbrace{{\bm{v}}^{(L-1)}_{j}({\bm{x}})}_{\eqqcolon\psi_{j}({\bm{x}})},$

where
$\hat{{\bm{v}}}^{(L-1)}_{kj}\coloneqq\mathcal{F}({\bm{v}}^{(L-1)}_{j})_{k}\in\mathbb{C}$
for $k\in\\{1,\dotsc,k_{\text{max}}\\}$. We note that
${\bm{z}}^{(L-1)}({\bm{x}},{\bm{w}}_{L-1})$ is linear in ${\bm{w}}_{L-1}$.
Thus, assuming a Gaussian belief over
${\bm{w}}_{L-1}\cong(\operatorname{Re}({\bm{R}}^{(L-1)}),\operatorname{Im}({\bm{R}}^{(L-1)}),{\bm{W}}^{(L-1)})$
induces a (multi-output) Gaussian process belief
${\bm{\mathrm{z}}}^{(L-1)}\sim{\operatorname{\mathcal{GP}}\left({\bm{m}}_{{\bm{\mathrm{z}}}^{(L-1)}},{\bm{K}}_{{\bm{\mathrm{z}}}^{(L-1)}}\right)}$
with
${\bm{m}}_{{\bm{\mathrm{z}}}^{(L-1)}}={\bm{z}}^{(L-1)}({\bm{x}},{\bm{w}}_{L-1}^{\star})$
over ${\bm{z}}^{(L-1)}$. Moreover, ${\bm{\mathrm{z}}}^{(L-1)}$ is the sum of
three (dependent) parametric Gaussian processes with feature functions
$\phi_{kj}$, $\varphi_{kj}$, and $\psi_{j}$, respectively. Consequently, the
function-valued GP induced by the linearized FNO is given by

${\mathrm{F}}({\bm{a}})({\bm{x}})=\tilde{{\bm{q}}}({\bm{m}}_{{\bm{\mathrm{z}}}^{(L-1)}}({\bm{x}}))+\mathrm{D}\tilde{{\bm{q}}}\left({\bm{m}}_{{\bm{\mathrm{z}}}^{(L-1)}}({\bm{x}})\right)({\bm{\mathrm{z}}}^{(L-1)}({\bm{x}})-{\bm{m}}_{{\bm{\mathrm{z}}}^{(L-1)}}({\bm{x}})),$

i.e.
${\mathrm{F}}({\bm{a}})\sim{\operatorname{\mathcal{GP}}\left({\bm{m}}_{\bm{a}},{\bm{K}}_{\bm{a}}\right)}$
with

$\displaystyle{\bm{m}}_{\bm{a}}({\bm{x}})$
$\displaystyle={\bm{F}}({\bm{a}},{\bm{w}}^{\star})({\bm{x}}),\qquad\text{and}$
$\displaystyle{\bm{K}}_{\bm{a}}({\bm{x}}_{1},{\bm{x}}_{2})$
$\displaystyle=\mathrm{D}\tilde{{\bm{q}}}\left({\bm{m}}_{{\bm{\mathrm{z}}}^{(L-1)}}({\bm{x}}_{1})\right){\bm{K}}_{{\bm{\mathrm{z}}}^{(L-1)}}({\bm{x}}_{1},{\bm{x}}_{2})\mathrm{D}\tilde{{\bm{q}}}\left({\bm{m}}_{{\bm{\mathrm{z}}}^{(L-1)}}({\bm{x}}_{2})\right)^{\top}.$

If the input function ${\bm{a}}\in{\mathbb{A}}$ is discretized on a grid
${\bm{X}}_{{\mathbb{A}}}^{(i)}\in({\mathbb{D}}_{\mathbb{A}})^{n_{\mathbb{A}}^{(i)}}$,
we set
$\hat{{\bm{v}}}^{(L-1)}_{kj}\coloneqq\operatorname{rfft}({\bm{v}}^{(L-1)}_{j}({\bm{X}}_{{\mathbb{A}}}^{(i)}))_{k}$
$\in\mathbb{C}$ and $\psi_{j}({\bm{x}})$ interpolates
${\bm{v}}^{(L-1)}_{j}({\bm{X}}_{{\mathbb{A}}}^{(i)})$ (e.g. spline
interpolation or Fourier interpolation).

There are two practical benefits arising from this representation. First, note
that computing the moments of and drawing samples from
${\mathrm{F}}({\bm{a}})$ only needs access to the hidden state
${\bm{v}}^{(L-1)}$ of the neural operator. This means that we can evaluate the
Gaussian process belief at arbitrary output points
${\bm{x}}\in{\mathbb{D}}_{\mathbb{U}}$ without the need to compute more than
one forward pass of the neural operator. Secondly, due to the fact that the
Gaussian process belief ${\mathrm{F}}({\bm{a}})$ over the output function is
parametric, we can efficiently sample entire functions from it that can then
be lazily evaluated at arbitrary points. This is in contrast to general non-
parametric Gaussian processes, where one typically discretizes the GP before
drawing samples of the function values at the given finite set of points. Such
lazy functional samples can be used e.g. for active experimental design and
Bayesian optimization [Wilson et al., 2021].

###  3.3 Implementation

In the following, we focus on the linearized last-layer Laplace approximation
for extracting a tractable Gaussian belief over the weights of the Fourier
neural operator. Last-layer approximations, where the feature map
corresponding to the first $L-1$ layers is set to its MAP estimate [Snoek et
al., 2015, Kristiadi et al., 2020], allow the Laplace approximation to scale
to deeper architectures and have proven effective in both theoretical and
practical applications [Daxberger et al., 2021a, Kristiadi et al., 2020].
Since the final standard (MLP) decoding layer ${\bm{w}}_{{\bm{q}}}$ applies
only pointwise and therefore lacks global characteristics, we consider the
last spectral convolution with bias function instead. Specifically, we use the
final Fourier layer weights
${\bm{w}}_{L-1}\cong(\operatorname{Re}({\bm{R}}^{(L-1)}),\operatorname{Im}({\bm{R}}^{(L-1)}),{\bm{W}}^{(L-1)})$.

Common approaches for approximating the GGN of large layers - such as K-FAC or
diagonal [Daxberger et al., 2021a] \- leverage a block-diagonal structure that
trades cross-layer correlations for computational efficiency. However, since
the Fourier convolution forms a collection of linear layers applied in
parallel to different Fourier modes, such block approximations would yield a
belief over the weights that is uncorrelated across these modes, which seems
inaccurate due to the following inverse Fourier transformation. To address
this issue, we focus instead on finding a low-rank approximation $VV^{T}$ of
the inverse GGN via Lanczos iterations (similar to a truncated SVD) [Dangel et
al., 2023]. The resulting low-rank plus scaled diagonal approximation of the
posterior covariance yields an efficiently represented object suitable for
matrix-free implementations, allowing for example for computationally cheap
inversion via the Woodbury matrix identity.

The Gaussian belief of ${\bm{w}}$ is hence given by

${\bm{w}}\sim{\mathcal{N}}({\bm{w}}^{*},(n\cdot{\bm{G}}+\tau I)^{-1}),$

where $N$ is the number of input-output pairs following the Bayesian
perspective. For a given discretization, this covariance can be pushed forward
onto the output via efficiently implemented Jacobian-vector products following
Step 2 in Section 3.2.

##  4 Related work

The literature on neural operators has been extensively reviewed, with
Azizzadenesheli et al. [2024] providing a comprehensive overview of various
architectures. These include graph neural operators [Li et al., 2020b],
physics-informed neural operators [Li et al., 2021], multi-wavelet neural
operators [Gupta et al., 2021] and the widely used Fourier neural operators
[Li et al., 2020c]. Lanthaler et al. [2024] further contributes to this field
by quantifying the aliasing error resulting from discretization of FNOs and
obtaining algebraic rates of convergence in relation to grid resolution.

Despite the advancements in neural operator architectures, uncertainty
estimation remains an underexplored area. Some progress has been made by
Magnani et al. [2022], Kumar et al. [2024], Garg and Chakraborty [2023]. Garg
and Chakraborty [2023] employs variational inference to estimate the Bayesian
posterior over DeepONet predictions. The work in Magnani et al. [2022], Kumar
et al. [2024] is more closely related to the present work. Magnani et al.
[2022] use Laplace approximation to provide uncertainty estimates for graph
neural operators, but do not extend this to Fourier neural operators or
consider a function space approach. Kumar et al. [2024] incorporates a
Gaussian Process prior with a mean function derived from a Wavelet Neural
Operator, optimizing model hyperparameters by minimizing the negative log-
marginal likelihood. Other Bayesian operator frameworks have been considered
in Zou et al. [2022] and Garg and Chakraborty [2022]. Operator-valued kernels
and function-valued Gaussian processes have been studied in the Hilbert space
setting, e.g. by Micchelli and Pontil [2005], Kadri et al. [2016], and Owhadi
[2023]. Our approach, however, formulates the theory within the context of
Banach spaces, as neural operators are defined as mappings between such
spaces. Other relevant methods using Gaussian processes to solve partial
differential equations (PDEs) include the works of Chen et al. [2021], Batlle
et al. [2024], and Chen et al. [2023].

Laplace approximation, introduced to deep learning by Mackay [1992], has
gained popularity in the Bayesian deep learning community [Ritter et al.,
2018, Daxberger et al., 2021a, Kristiadi et al., 2020, Papamarkou et al.,
2024]. This is also due to its scalability, achieved through various
strategies including using log-posterior Hessian approximations [Ritter et
al., 2018, Martens, 2020], treating only a subset of the model
probabilistically [Daxberger et al., 2021b], employing linearized Laplace
[Foong et al., 2019, Immer et al., 2020], or using scalable Gaussian processes
methods [Deng et al., 2022, Ortega et al., 2023]. Other Bayesian deep learning
methods include variational inference [Graves, 2011, Blundell et al., 2015,
Khan et al., 2018, Zhang et al., 2018], Markov Chain Monte Carlo [Neal, 1996,
Welling and Teh, 2011, Zhang et al., 2020], or heuristic methods [Gal and
Ghahramani, 2016, Maddox et al., 2019].

##  5 Experiments

We evaluate our approach by quantifying the uncertainty of Fourier neural
operator predictions, that were trained on a limited amount of training
samples. Following the experimental setup in Brandstetter et al. [2022], we
train multiple Fourier neural operators on varying numbers of training
trajectories (i.e. $32$, $64$, and $128$ training samples, each having $256$
spatial and $140$ temporal points) for the Korteweg-de Vries (KdV,
$\partial_{t}u+u\partial_{x}u+\partial_{x}^{3}u=0$), the Kuramoto-Sivashinsky
(KS, $\partial_{t}u+\partial_{x}^{2}u+\partial_{x}^{4}u+u\partial_{x}u=0$) and
Burgers’ ($\partial_{t}u+u\partial_{x}u-\nu\partial_{x}^{2}u=0$ where $\nu>0$)
equations. Each neural operator is trained to predict twenty future time steps
based on the twenty previous ones. All models consist of four Fourier
convolutional layers with 16 Fourier modes and 32 hidden channels. Using the
above-outlined Laplace approximation, we model 33,824 parameters (more than
one-fifth of all parameters) probabilistically.

We evaluate NOLA on each trained model against input perturbations and weight
perturbations as baseline approaches, which can be seen as ensemble forecast:
the former is given by
${\bm{F}}_{{\bm{w}}}({\bm{a}}({\bm{x}})+\epsilon^{(k)}_{{\bm{x}}})_{k\in K}$
with $\epsilon^{(k)}_{{\bm{x}}}\sim{\mathcal{N}}(0,\tau^{-2}I_{|{\bm{x}}|})$,
and the latter by
$({\bm{F}}_{{\bm{w}}_{L-1}+\epsilon^{(k)}}({\bm{a}}({\bm{x}}))_{k}$ with
$\epsilon^{(k)}\sim{\mathcal{N}}(0,\tau^{-2}I_{|{\bm{w}}_{L-1}|})$. For NOLA,
we can estimate the standard deviation by the square root of the pushed-
forward output variance, while for both perturbation-based methods, we take
the empirical standard deviation of $|K|$ ensemble members, setting $|K|=100$.

Figure 2: Comparing NOLA, input perturbations, and weight perturbations at a
single predicted time point of a sample trajectory of the KdV equation. The
corresponding FNO has been trained on 32 training trajectories.

To quantify and compare the uncertainty, we consider the negative log-
likelihood (NLL), the root squared mean error (RMSE), and the q-statistic (Q)
defined as follows:

RMSE $\displaystyle=\sqrt{\frac{1}{c}\sum_{x\in{\bm{x}},i\leq
n}|{\bm{F}}({\bm{a}}_{i})(x)-{\bm{u}}_{i}(x)|^{2}},$ Q
$\displaystyle=\frac{1}{c}\sum_{x\in{\bm{x}},i\leq
n}\frac{|{\bm{F}}({\bm{a}}_{i})({\bm{x}})-{\bm{u}}_{i}({\bm{x}})|^{2}}{\operatorname{\mathbb{V}}\left[({\bm{a}}_{i})(x)\right]},\text{
and }$ NLL $\displaystyle=\frac{1}{2c}\sum_{x\in{\bm{x}},i\leq
n}\left[\log\left(2\pi\operatorname{\mathbb{V}}\left[{\bm{F}}({\bm{a}}_{i})(x)\right]\right)+\frac{|{\bm{F}}({\bm{a}}_{i})(x)-{\bm{u}}_{i}(x)|^{2}}{\operatorname{\mathbb{V}}\left[({\bm{a}}_{i})(x)\right]}\right],$

where $c$ is the number of summands and $n$ is the number of input-output
pairs considered for evaluation. The RMSE measures the accuracy of the
GP/ensemble mean compared to the true target and should be close to zero. The
Q estimate indicates posterior calibration and should be close to one. A lower
NLL reflects a balance between low standard deviations and error calibration.
For NOLA we compute the GGN on 128 input-output pairs of the training set. The
hyperparameter $\tau$ is calibrated for each method on the validation set and
the metrics are compared on $100$ samples of the test set, as shown in Table
1. A single sample prediction for a trajectory of the KdV equation can be
found in Figure 2.

Table 1: Evaluation metrics on 100 test samples.

| KdV | KS | Burgers
---|---|---|---
| NLL | RMSE | Q | NLL | RMSE | Q | NLL | RMSE | Q
32 training samples |  |  |  |  |  |  |  |  |
Input perturbations | 0.795 | 0.162 | 3.32 | -0.390 | 0.152 | 1.636 | 0.596 | 0.331 | 2.397
Weight perturbations | -0.306 | 0.169 | 1.929 | -0.539 | 0.148 | 1.712 | 0.909 | 0.34 | 3.208
NOLA | -0.577 | 0.163 | 1.152 | -0.543 | 0.148 | 1.125 | 0.443 | 0.335 | 0.534
64 training samples |  |  |  |  |  |  |  |  |
Input perturbations | -0.214 | 0.122 | 1.01 | -0.767 | 0.108 | 1.664 | -0.417 | 0.125 | 2.075
Weight perturbations | -0.691 | 0.121 | 1.422 | -0.862 | 0.106 | 1.786 | -0.322 | 0.126 | 2.565
NOLA | -0.920 | 0.119 | 1.003 | -0.872 | 0.106 | 1.273 | -0.517 | 0.123 | 1.987
128 training samples |  |  |  |  |  |  |  |  |
Input perturbations | -0.383 | 0.094 | 1.54 | -1.108 | 0.082 | 1.476 | -1.207 | 0.044 | 3.059
Weight perturbations | -0.913 | 0.095 | 0.558 | -1.121 | 0.081 | 1.908 | -1.397 | 0.042 | 2.505
NOLA | -1.191 | 0.091 | 1.311 | -1.158 | 0.081 | 1.417 | -1.719 | 0.042 | 1.055

While NOLA outperforms both baselines in the metrics sometimes only by a
margin, we note that the overall prediction error in the experiment is already
low and it is therefore important to test NOLA on more diverse tasks, where
more uncertainty can be captured. Observations from Figure 2 suggest that
NOLA’s sample paths align more closely with the true target function than
those from the baselines.

##  6 Conclusion

In this work, we developed a probabilistic framework for neural network-
learned operators using Banach-valued Gaussian processes. Our approach extends
Gaussian process theory into the domain of operators, facilitating the
treatment of infinite-dimensional function spaces. By demonstrating the
equivalence between Banach-valued Gaussian processes and multi-output Gaussian
processes with an augmented input space, we establish _Gaussian currying_ , a
probabilistic analogue of currying in functional programming.

This currying process transforms complex operator mappings into more
manageable and practically implementable forms, effectively modeling
relationships between infinite-dimensional spaces. By applying linearization
and Laplace approximation, neural operators can be viewed as function-valued
Gaussian processes, providing analytic uncertainty estimates over an operator
and yielding a continuous function that can be evaluated on any grid. This
function can also be analytically propagated to downstream analyses.

In our experiments, we show improvements over perturbation-based methods in
terms of common metrics. Especially, sample predictions generated with NOLA
exhibit more realistic behaviour. Future directions for this work include
investigating into the interpolation error that arises in constructing the
feature functions of the parametric Gaussian process in the Bayesian Fourier
neural operator. Moreover, we plan to apply our approach to a wider set of
PDEs, including multidimensional problems.

## Acknowledgments and Disclosure of Funding

The authors gratefully acknowledge financial support by the European Research
Council through ERC CoG Action 101123955 ANUBIS ; the DFG Cluster of
Excellence “Machine Learning - New Perspectives for Science”, EXC 2064/1,
project number 390727645; the German Federal Ministry of Education and
Research (BMBF) through the Tübingen AI Center (FKZ: 01IS18039A); the DFG SPP
2298 (Project HE 7114/5-1), and the Carl Zeiss Foundation, (project
"Certification and Foundations of Safe Machine Learning Systems in
Healthcare"), as well as funds from the Ministry of Science, Research and Arts
of the State of Baden-Württemberg. The authors thank the International Max
Planck Research School for Intelligent Systems (IMPRS-IS) for supporting
Emilia Magnani, Marvin Pförtner and Tobias Weber.

## References

  * Kovachki et al. [2023] Nikola Kovachki, Zongyi Li, Burigede Liu, Kamyar Azizzadenesheli, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Neural operator: Learning maps between function spaces with applications to PDEs.  _JMLR_ , 24(89):1–97, 2023.
  * Pathak et al. [2022] Jaideep Pathak, Shashank Subramanian, Peter Harrington, Sanjeev Raja, Ashesh Chattopadhyay, Morteza Mardani, Thorsten Kurth, David Hall, Zongyi Li, Kamyar Azizzadenesheli, et al.  Fourcastnet: A global data-driven high-resolution weather model using adaptive fourier neural operators.  _arXiv preprint arXiv:2202.11214_ , 2022.
  * Bonev et al. [2023] Boris Bonev, Thorsten Kurth, Christian Hundt, Jaideep Pathak, Maximilian Baust, Karthik Kashinath, and Anima Anandkumar.  Spherical Fourier neural operators: Learning stable dynamics on the sphere.  In _ICML_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 2806–2823. PMLR, 2023.
  * Grady et al. [2022] TJ Grady, Rishi Khan, Mathias Louboutin, Ziyi Yin, Philipp A Witte, Ranveer Chandra, Russell J Hewett, and Felix J Herrmann.  Towards large-scale learned solvers for parametric PDEs with model-parallel fourier neural operators.  _arXiv preprint arXiv:2204.01205_ , 2022.
  * Renn et al. [2023] Peter I Renn, Cong Wang, Sahin Lale, Zongyi Li, Anima Anandkumar, and Morteza Gharib.  Forecasting subcritical cylinder wakes with fourier neural operators.  _arXiv preprint arXiv:2301.08290_ , 2023.
  * Li et al. [2022] Zhijie Li, Wenhui Peng, Zelong Yuan, and Jianchun Wang.  Fourier neural operator approach to large eddy simulation of three-dimensional turbulence.  _Theoretical and Applied Mechanics Letters_ , 12(6):100389, 2022.
  * Li et al. [2024] Zongyi Li, Nikola Kovachki, Chris Choy, Boyi Li, Jean Kossaifi, Shourya Otta, Mohammad Amin Nabian, Maximilian Stadler, Christian Hundt, Kamyar Azizzadenesheli, et al.  Geometry-informed neural operator for large-scale 3d PDEs.  _NeurIPS_ , 36, 2024.
  * Papamarkou et al. [2024] Theodore Papamarkou, Maria Skoularidou, Konstantina Palla, Laurence Aitchison, Julyan Arbel, David Dunson, Maurizio Filippone, Vincent Fortuin, Philipp Hennig, José Miguel Hernández-Lobato, Aliaksandr Hubin, Alexander Immer, Theofanis Karaletsos, Mohammad Emtiyaz Khan, Agustinus Kristiadi, Yingzhen Li, Stephan Mandt, Christopher Nemeth, Michael A Osborne, Tim G. J. Rudner, David Rügamer, Yee Whye Teh, Max Welling, Andrew Gordon Wilson, and Ruqi Zhang.  Position: Bayesian deep learning is needed in the age of large-scale AI.  In _ICML_ , 2024.
  * Immer et al. [2020] Alexander Immer, Maciej Korzepa, and Matthias Bauer.  Improving predictions of Bayesian neural networks via local linearization.  In _AISTATS_ , 2020.
  * Li et al. [2020a] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Neural operator: Graph kernel network for partial differential equations.  In _ICLR 2020 Workshop on Integration of Deep Neural Models and Differential Equations_ , 2020a.
  * Li et al. [2020b] Zongyi Li, Nikola Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Multipole graph neural operator for parametric partial differential equations.  In _NeurIPS_ , volume 33, pages 6755–6766. Curran Associates, Inc., 2020b.
  * Li et al. [2020c] Zongyi Li, Nikola Borislavov Kovachki, Kamyar Azizzadenesheli, Burigede Liu, Kaushik Bhattacharya, Andrew Stuart, and Anima Anandkumar.  Fourier neural operator for parametric partial differential equations.  In _ICLR_ , 2020c.
  * MacKay [1992a] David JC MacKay.  Bayesian interpolation.  _Neural computation_ , 4(3):415–447, 1992a.
  * MacKay [1992b] David JC MacKay.  The evidence framework applied to classification networks.  _Neural computation_ , 4(5):720–736, 1992b.
  * Daxberger et al. [2021a] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig.  Laplace redux-effortless bayesian deep learning.  _NeurIPS_ , 2021a.
  * Schraudolph [2002] Nicol Schraudolph.  Fast curvature matrix-vector products for second-order gradient descent.  _Neural computation_ , 14:1723–38, 2002.
  * Khan et al. [2019] Mohammad Emtiyaz Khan, Alexander Immer, Ehsan Abedi, and Maciej Korzepa.  Approximate inference turns deep networks into Gaussian processes.  In _NeurIPS_ , 2019.
  * Da Costa et al. [2023] Nathaël Da Costa, Marvin Pförtner, Lancelot Da Costa, and Philipp Hennig.  Sample path regularity of Gaussian processes from the covariance kernel.  _arXiv preprint arXiv:2312.14886_ , 2023.  doi:10.48550/arXiv.2312.14886.
  * Kristiadi et al. [2020] Agustinus Kristiadi, Matthias Hein, and Philipp Hennig.  Being bayesian, even just a bit, fixes overconfidence in relu networks.  In _ICML_ , pages 5436–5446. PMLR, 2020.
  * Wilson et al. [2021] James T. Wilson, Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, and Marc Peter Deisenroth.  Pathwise conditioning of Gaussian processes.  _Journal of Machine Learning Research_ , 22(105):1–47, 2021.
  * Snoek et al. [2015] Jasper Snoek, Oren Rippel, Kevin Swersky, Ryan Kiros, Nadathur Satish, Narayanan Sundaram, Mostofa Patwary, Mr Prabhat, and Ryan Adams.  Scalable bayesian optimization using deep neural networks.  In _ICML_ , 2015.
  * Dangel et al. [2023] Felix Dangel, Lukas Tatzel, and Philipp Hennig.  Vivit: Curvature access through the generalized gauss-newton’s low-rank structure.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=DzJ7JfPXkE.
  * Azizzadenesheli et al. [2024] Kamyar Azizzadenesheli, Nikola Kovachki, Zongyi Li, Miguel Liu-Schiaffini, Jean Kossaifi, and Anima Anandkumar.  Neural operators for accelerating scientific simulations and design.  _Nature Reviews Physics_ , pages 1–9, 2024.
  * Li et al. [2021] Zongyi Li, Hongkai Zheng, Nikola Kovachki, David Jin, Haoxuan Chen, Burigede Liu, Kamyar Azizzadenesheli, and Anima Anandkumar.  Physics-informed neural operator for learning partial differential equations.  _arXiv preprint arXiv:2111.03794_ , 2021.
  * Gupta et al. [2021] Gaurav Gupta, Xiongye Xiao, and Paul Bogdan.  Multiwavelet-based operator learning for differential equations.  In _NeurIPS_ , volume 34, 2021.
  * Lanthaler et al. [2024] Samuel Lanthaler, Andrew M Stuart, and Margaret Trautner.  Discretization error of fourier neural operators.  _arXiv preprint arXiv:2405.02221_ , 2024.
  * Magnani et al. [2022] Emilia Magnani, Nicholas Krämer, Runa Eschenhagen, Lorenzo Rosasco, and Philipp Hennig.  Approximate bayesian neural operators: Uncertainty quantification for parametric PDEs, 2022.
  * Kumar et al. [2024] Sawan Kumar, Rajdip Nayek, and Souvik Chakraborty.  Neural operator induced gaussian process framework for probabilistic solution of parametric partial differential equations.  _arXiv preprint arXiv:2404.15618_ , 2024.
  * Garg and Chakraborty [2023] Shailesh Garg and Souvik Chakraborty.  Vb-deeponet: A bayesian operator learning framework for uncertainty quantification.  _Engineering Applications of Artificial Intelligence_ , 118:105685, 2023.  ISSN 0952-1976.
  * Zou et al. [2022] Zongren Zou, Xuhui Meng, Apostolos F Psaros, and George Em Karniadakis.  Neuraluq: A comprehensive library for uncertainty quantification in neural differential equations and operators.  _arXiv preprint arXiv:2208.11866_ , 2022.
  * Garg and Chakraborty [2022] Shailesh Garg and Souvik Chakraborty.  Variational bayes deep operator network: a data-driven bayesian solver for parametric differential equations.  _arXiv preprint arXiv:2206.05655_ , 2022.
  * Micchelli and Pontil [2005] Charles A Micchelli and Massimiliano Pontil.  On learning vector-valued functions.  _Neural computation_ , 17(1):177–204, 2005.
  * Kadri et al. [2016] Hachem Kadri, Emmanuel Duflos, Philippe Preux, Stéphane Canu, Alain Rakotomamonjy, and Julien Audiffren.  Operator-valued kernels for learning from functional response data.  _JMLR_ , 17(20):1–54, 2016.
  * Owhadi [2023] Houman Owhadi.  Do ideas have shape? idea registration as the continuous limit of artificial neural networks.  _Physica D: Nonlinear Phenomena_ , 444, 2023.
  * Chen et al. [2021] Yifan Chen, Bamdad Hosseini, Houman Owhadi, and Andrew M Stuart.  Solving and learning nonlinear PDEs with gaussian processes.  _Journal of Computational Physics_ , 447:110668, 2021.
  * Batlle et al. [2024] Pau Batlle, Matthieu Darcy, Bamdad Hosseini, and Houman Owhadi.  Kernel methods are competitive for operator learning.  _Journal of Computational Physics_ , 496:112549, 2024.
  * Chen et al. [2023] Yifan Chen, Houman Owhadi, and Florian Schäfer.  Sparse cholesky factorization for solving nonlinear pdes via gaussian processes.  _arXiv preprint arXiv:2304.01294_ , 2023.
  * Mackay [1992] David John Cameron Mackay.  _Bayesian methods for adaptive models_.  California Institute of Technology, 1992.
  * Ritter et al. [2018] Hippolyt Ritter, Aleksandar Botev, and David Barber.  A scalable laplace approximation for neural networks.  In _ICLR_ , 2018.
  * Martens [2020] James Martens.  New insights and perspectives on the natural gradient method.  _JMLR_ , 21(146):1–76, 2020.
  * Daxberger et al. [2021b] Erik Daxberger, Eric Nalisnick, James U Allingham, Javier Antoran, and Jose Miguel Hernandez-Lobato.  Bayesian deep learning via subnetwork inference.  In _ICML_ , volume 139 of _PMLR_ , 2021b.
  * Foong et al. [2019] Andrew YK Foong, Yingzhen Li, José Miguel Hernández-Lobato, and Richard E Turner.  ’In-Between’ uncertainty in bayesian neural networks.  _arXiv preprint arXiv:1906.11537_ , 2019.
  * Deng et al. [2022] Zhijie Deng, Feng Zhou, and Jun Zhu.  Accelerated linearized laplace approximation for bayesian deep learning.  _NeurIPS_ , 2022.
  * Ortega et al. [2023] Luis A Ortega, Simón Rodríguez Santana, and Daniel Hernández-Lobato.  Variational linearized laplace approximation for bayesian deep learning.  _arXiv preprint arXiv:2302.12565_ , 2023.
  * Graves [2011] Alex Graves.  Practical variational inference for neural networks.  In _NeurIPS_ , 2011.
  * Blundell et al. [2015] Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra.  Weight uncertainty in neural network.  In _ICML_ , 2015.
  * Khan et al. [2018] Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava.  Fast and scalable Bayesian deep learning by weight-perturbation in adam.  In _ICML_ , 2018.
  * Zhang et al. [2018] Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger B. Grosse.  Noisy natural gradient as variational inference.  In _ICML_ , 2018.
  * Neal [1996] Radford M. Neal.  _Bayesian Learning for Neural Networks_.  Springer-Verlag, Berlin, Heidelberg, 1996.  ISBN 0387947248.
  * Welling and Teh [2011] Max Welling and Yee Whye Teh.  Bayesian learning via stochastic gradient Langevin dynamics.  In _ICML_ , 2011.
  * Zhang et al. [2020] Ruqi Zhang, Chunyuan Li, Jianyi Zhang, Changyou Chen, and Andrew Gordon Wilson.  Cyclical stochastic gradient MCMC for Bayesian deep learning.  In _ICLR_ , 2020.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a Bayesian approximation: Representing model uncertainty in deep learning.  In _ICML_ , 2016.
  * Maddox et al. [2019] Wesley Maddox, T. Garipov, Pavel Izmailov, Dmitry P. Vetrov, and Andrew Gordon Wilson.  A simple baseline for Bayesian uncertainty in deep learning.  In _NeurIPS_ , 2019.
  * Brandstetter et al. [2022] Johannes Brandstetter, Max Welling, and Daniel E Worrall.  Lie point symmetry data augmentation for neural pde solvers.  _arXiv preprint arXiv:2202.07643_ , 2022.
  * Klenke [2014] Achim Klenke.  _Probability Theory: A Comprehensive Course_.  Universitext. Springer, London, second edition, 2014.
  * Pförtner et al. [2022] Marvin Pförtner, Ingo Steinwart, Philipp Hennig, and Jonathan Wenger.  Physics-informed gaussian process regression generalizes linear pde solvers.  _arXiv preprint arXiv:2212.12474_ , 2022.

## Appendix

##  Appendix A Theoretical Results

###  A.1 Gaussian measures on separable Banach spaces

We aim to quantify epistemic uncertainty about operators between (infinite-
dimensional) Banach spaces of functions using a Gaussian process framework. To
be able to define a Gaussian process belief over such operators, we hence need
a notion of a Gaussian random variables with values in (separable) Banach
spaces.

######  Definition A.1 (Gaussian Measure on separable Banach space).

Let ${\mathbb{U}}$ be a real separable Banach space. A probability measure
$\gamma$ on $({\mathbb{U}},\mathcal{B}\left({\mathbb{U}}\right))$ is called
_Gaussian_ if every continuous linear functional
$\ell\in{\mathbb{U}}^{\prime}$ is a univariate Gaussian random variable on
$({\mathbb{U}},\mathcal{B}\left({\mathbb{U}}\right),\gamma)$. A
${\mathbb{U}}$-valued random variable is called _Gaussian_ if its law is
Gaussian.

######  Remark A.1 (Jointly Gaussian Measure on separable Banach spaces).

In this work, we frequently need to construct joint Gaussian measures on
multiple separable Banach spaces. Fortunately, we can also leverage definition
A.1 for this. More formally, we aim to define a Gaussian measure on the
iterated Cartesian product ${\mathbb{U}}^{n}$ of a real separable Banach space
${\mathbb{U}}$. We equip ${\mathbb{U}}^{n}$ with the product sigma algebra
$\mathcal{B}\left({\mathbb{U}}\right)^{\otimes n}$. Since ${\mathbb{U}}$ is
Polish, we have $\mathcal{B}\left({\mathbb{U}}\right)^{\otimes
n}=\mathcal{B}\left(\tau\right)$, where $\tau$ is the product topology on
${\mathbb{U}}^{n}$ [Klenke, 2014, Theorem 14.8]. Moreover,
$({\mathbb{U}}^{n},\tau)$ is Banachable, i.e. there is a norm
$\lVert\cdot\rVert_{{\mathbb{U}}^{n}}$ that induces $\tau$ such that
$({\mathbb{U}}^{n},\lVert\cdot\rVert_{{\mathbb{U}}^{n}})$ is complete. Hence,
as in definition A.1, we call a probability measure $\gamma$ on
$({\mathbb{U}}^{n},\mathcal{B}\left({\mathbb{U}}\right)^{\otimes n})$
_Gaussian_ if every $\ell\in{\mathbb{U}}^{\prime}$ is a univariate Gaussian
random variable on
$({\mathbb{U}}^{n},\mathcal{B}\left({\mathbb{U}}\right)^{\otimes n},\gamma)$.

Similar to their finite-dimensional counterparts, Gaussian measures with
values in separable Banach spaces admit the definition of a mean and a
(cross-)covariance operator.

######  Proposition A.1 (Mean and Covariance Operator [see e.g. Pförtner et
al., 2022, Proposition B.2]).

Let $\gamma$ be a Gaussian measure on a real separable Banach space
${\mathbb{U}}$. There is a unique $m_{\gamma}\in{\mathbb{U}}$ with
$\ell(m_{\gamma})=\operatorname{\mathbb{E}}_{u\sim\gamma}\left[\ell(u)\right]$
for every continuous linear functional $\ell\in{\mathbb{U}}^{\prime}$,
referred to as the _mean_ of $\gamma$. Similarly, there is a unique bounded
linear operator
$\mathcal{C}_{\gamma}\colon{\mathbb{U}}^{\prime}\to{\mathbb{U}}$ with
$\ell_{1}(\mathcal{C}_{\gamma}(\ell_{2}))=\operatorname{Cov}_{u\sim\gamma}\left[\ell_{1}(u),\ell_{2}(u)\right]$
for any $\ell_{1},\ell_{2}\in{\mathbb{U}}^{\prime}$, the so-called _covariance
operator_ of $\gamma$. The mean and covariance operator of a Gaussian random
variable with values in ${\mathbb{U}}$ is defined accordingly.

######  Corollary A.2 (Cross-Covariance Operator).

Let ${\mathrm{u}}_{1},{\mathrm{u}}_{2}$ be jointly Gaussian random variables
on $(\Omega,\mathcal{A},\mathrm{P})$ with values in real separable Banach
spaces ${\mathbb{U}}_{1},{\mathbb{U}}_{2}$, respectively. There is a unique
bounded linear operator
$\mathcal{C}_{{\mathrm{u}}_{1},{\mathrm{u}}_{2}}\colon{\mathbb{U}}_{1}^{\prime}\to{\mathbb{U}}_{2}$
with
$\ell_{1}(\mathcal{C}_{{\mathrm{u}}_{1},{\mathrm{u}}_{2}}(\ell_{2}))=\operatorname{Cov}_{\mathrm{P}}\left[\ell_{1}({\mathrm{u}}_{1}),\ell_{2}({\mathrm{u}}_{2})\right]$
for all $\ell_{1}\in{\mathbb{U}}_{1}^{\prime}$ and
$\ell_{2}\in{\mathbb{U}}_{2}^{\prime}$. The operator
$\mathcal{C}_{{\mathrm{u}}_{1},{\mathrm{u}}_{2}}$ is referred to as the
_cross-covariance operator_ between ${\mathrm{u}}_{1}$ and ${\mathrm{u}}_{2}$.

###  A.2 Banach-valued Gaussian processes

Now we have all the necessary preliminaries to define a Gaussian random
process that takes values in real separable Banach spaces. *

As for (multi-output) Gaussian processes, we can also define mean and
covariance functions for Banach-valued Gaussian processes. However, their
definition is more technically involved.

######  Definition A.2.

Let ${\mathrm{F}}$ be a ${\mathbb{U}}$-valued Gaussian process with index set
${\mathbb{A}}$ on $(\Omega,\mathcal{A},\mathrm{P})$. The function

$\mathcal{M}\colon{\mathbb{A}}\to{\mathbb{U}},a\mapsto
m_{{\mathrm{F}}(a,\,\cdot\,)}$

is called the _mean function_ of ${\mathrm{F}}$ and the function

$\mathcal{K}\colon{\mathbb{A}}\times{\mathbb{A}}\to({\mathbb{U}}^{\prime}\to{\mathbb{U}}),(a_{1},a_{2})\mapsto\mathcal{C}_{{\mathrm{F}}(a_{1},\,\cdot\,),{\mathrm{F}}(a_{2},\,\cdot\,)}$

is referred to as the _covariance function_ of ${\mathrm{F}}$.

In the following, we aim to establish a correspondence between Banach-valued
Gaussian processes and ($\mathbb{R}$-valued) Gaussian processes. Unlike in
lemma 2.1, we need additional technical assumptions for this to work both
ways. Denote by
$\operatorname{scl}_{w*}({\mathbb{L}})\coloneqq\\{\ell\in{\mathbb{U}}^{\prime}\nonscript\>|\allowbreak\nonscript\>\mathopen{}\exists\\{\ell_{i}\\}_{i\in\mathbb{N}}\subset{\mathbb{L}}\colon\ell_{i}\to_{w*}\ell\\}$
the weak-* _sequential_ closure of a set
${\mathbb{L}}\subset{\mathbb{U}}^{\prime}$.

######  Assumption A.1.

Let ${\mathbb{U}}$ be a real separable Banach space and
$\mathbb{L}\subset{\mathbb{U}}^{\prime}$ a set of continuous linear
functionals on ${\mathbb{U}}$ such that there is an
$n_{\operatorname{scl}}\in\mathbb{N}_{0}$ with
$\operatorname{scl}_{w*}^{n_{\operatorname{scl}}}(\operatorname{span}{\mathbb{L}})={\mathbb{U}}^{\prime}$.

######  Theorem A.3.

Let $(\Omega,\mathcal{A},\mathrm{P})$ be a probability space, ${\mathbb{U}}$ a
real separable Banach space, and $\mathbb{L}\subset{\mathbb{U}}^{\prime}$ .
Let ${\mathrm{F}}\colon{\mathbb{A}}\times\Omega\to{\mathbb{U}}$ and
${\mathrm{f}}\colon({\mathbb{A}}\times\mathbb{L})\times\Omega\to\mathbb{R}$
such that $\ell({\mathrm{F}}(a,\cdot))={\mathrm{f}}((a,\ell),\cdot)$ for all
$a\in{\mathbb{A}}$ and $\ell\in\mathbb{L}$ ($\mathrm{P}$-almost surely). Then

  1. (i)

${\mathrm{F}}\sim{\operatorname{\mathcal{GP}}\left(\mathcal{M},\mathcal{K}\right)}$
implies ${\mathrm{f}}\sim{\operatorname{\mathcal{GP}}\left(m,k\right)}$,

and, if assumption A.1 holds,

  1. (ii)

${\mathrm{f}}\sim{\operatorname{\mathcal{GP}}\left(m,k\right)}$ implies
${\mathrm{F}}\sim{\operatorname{\mathcal{GP}}\left(\mathcal{M},\mathcal{K}\right)}$,

where,

  1. (iii)

in both cases,

$\ell(\mathcal{M}(a))=m(a,\ell)$

for all $a\in{\mathbb{A}}$ and $\ell\in\mathbb{L}$, as well as

$\ell_{1}(\mathcal{K}(a_{1},a_{2})(\ell_{2}))=k((a_{1},\ell_{1}),(a_{2},\ell_{2}))$

for all $a_{1},a_{2}\in{\mathbb{A}}$ and
$\ell_{1},\ell_{2}\in\mathbb{{\mathbb{U}}}$.

###### Proof.

First of all, note that $\ell\in({\mathbb{U}}^{n})^{\prime}$ if and only if
there are $\ell_{1},\dotsc,\ell_{n}\in{\mathbb{U}}^{\prime}$ such that

$\ell(u_{1},\dotsc,u_{n})=\sum_{i=1}^{n}\ell_{i}(u_{i}),$

where we equip ${\mathbb{U}}^{n}$ with the product topology.

  1. (i)

Let $a_{1},\dotsc,a_{n}\in{\mathbb{A}}$ and
$\ell_{1},\dotsc,\ell_{n}\in{\mathbb{U}}^{\prime}$. By the above, the linear
functionals

$\tilde{\ell}_{i}\colon{\mathbb{U}}^{n}\to\mathbb{R},(u_{1},\dotsc,u_{n})\mapsto\ell_{i}(u_{i})$

are continuous w.r.t. the product topology on ${\mathbb{U}}^{n}$. Moreover,
$\omega\mapsto({\mathrm{F}}(a_{1},\omega),\dotsc,{\mathrm{F}}(a_{n},\omega))$
is Gaussian by assumption. Hence,

$\displaystyle\omega\mapsto$
$\displaystyle\left(\tilde{\ell}_{i}({\mathrm{F}}(a_{1},\omega),\dotsc,{\mathrm{F}}(a_{n},\omega))\right)_{i=1}^{n}$
$\displaystyle=\left(\ell_{i}({\mathrm{F}}(a_{i},\omega))\right)_{i=1}^{n}$
$\displaystyle=\left({\mathrm{f}}((a_{i},\ell_{i}),\omega)\right)_{i=1}^{n}$

is Gaussian with values in $\mathbb{R}^{n}$.

  2. (ii)

Let $a_{1},\dotsc,a_{n}\in{\mathbb{A}}$. We have to show that
$\omega\mapsto({\mathrm{F}}(a_{1},\omega),\dotsc,{\mathrm{F}}(a_{n},\omega))$
is a Gaussian random variable with values in ${\mathbb{U}}^{n}$, i.e. that for
every $\ell\in({\mathbb{U}}^{n})^{\prime}$, the random variable
$\omega\mapsto\ell({\mathrm{F}}(a_{1},\omega),\dotsc,{\mathrm{F}}(a_{n},\omega))$
is Gaussian with values in $\mathbb{R}$. By the above, we know that there are
$\ell_{1},\dotsc,\ell_{n}\in{\mathbb{U}}^{\prime}$ such that

$\omega\mapsto\ell({\mathrm{F}}(a_{1},\omega),\dotsc,{\mathrm{F}}(a_{n},\omega))=\sum_{i=1}^{m}\ell_{i}({\mathrm{F}}(a_{i},\omega))=\sum_{i=1}^{m}{\mathrm{f}}((a_{i},\ell_{i}),\omega)$

Claim. Let $m\in\mathbb{N}_{0}$ and
$\ell_{1},\dotsc,\ell_{n}\in\operatorname{scl}_{w*}^{m}(\operatorname{span}{\mathbb{L}})$.
Then $\omega\mapsto\sum_{i=1}^{n}{\mathrm{f}}((a_{i},\ell_{i}),\omega)$ is
Gaussian.

Proof of Claim. This is a straightforward modification of Theorem B.6 from
[Pförtner et al., 2022].

By assumption A.1, there is $m=n_{\operatorname{scl}}$ such that
$\operatorname{scl}_{w*}^{m}(\operatorname{span}{\mathbb{L}})={\mathbb{U}}^{\prime}$.
Hence, the statement follows from the claim.

  3. (iii)

Using the fact that Bochner integrals and bounded linear operators commute, we
obtain

$\displaystyle\ell(\mathcal{M}(a))$
$\displaystyle=\ell\left(\int_{\Omega}{\mathrm{F}}(a,\omega)\mathrm{P}\left(\mathrm{d}\omega\right)\right)$
$\displaystyle=\int_{\Omega}\underbrace{\ell({\mathrm{F}}(a,\omega))missing}_{\stackrel{{\scriptstyle\text{a.s.}}}{{=}}{\mathrm{f}}((a,\ell),\omega)}\mathrm{P}\left(\mathrm{d}\omega\right)$
$\displaystyle=\operatorname{\mathbb{E}}_{\mathrm{P}}\left[{\mathrm{f}}(a,\ell)\right]$
$\displaystyle=m(a,\ell)$

and

$\displaystyle\ell_{1}(\mathcal{K}(a_{1},a_{2})(\ell_{2}))$
$\displaystyle=\ell_{1}\left(\int_{\Omega}\ell_{2}({\mathrm{F}}(a_{2},\omega)-\mathcal{M}(a_{2}))({\mathrm{F}}(a_{1},\omega)-\mathcal{M}(a_{1}))\mathrm{P}\left(\mathrm{d}\omega\right)\right)$
$\displaystyle=\int_{\Omega}\ell_{2}({\mathrm{F}}(a_{2},\omega)-\mathcal{M}(a_{2}))\ell_{1}\left({\mathrm{F}}(a_{1},\omega)-\mathcal{M}(a_{1})\right)\mathrm{P}\left(\mathrm{d}\omega\right)$
$\displaystyle=\int_{\Omega}(\underbrace{\ell_{1}({\mathrm{F}}(a_{1},\omega))missing}_{\stackrel{{\scriptstyle\text{a.s.}}}{{=}}{\mathrm{f}}((a_{1},\ell_{1}),\omega)}-\underbrace{\ell_{1}(\mathcal{M}(a_{1}))missing}_{=m(a_{1},\ell_{1})})(\underbrace{\ell_{2}({\mathrm{F}}(a_{2},\omega))missing}_{\stackrel{{\scriptstyle\text{a.s.}}}{{=}}{\mathrm{f}}((a_{2},\ell_{2}),\omega)}-\underbrace{\ell_{2}(\mathcal{M}(a_{2}))missing}_{=m(a_{2},\ell_{2})})\mathrm{P}\left(\mathrm{d}\omega\right)$
$\displaystyle=\operatorname{Cov}_{\mathrm{P}}\left[{\mathrm{f}}(a_{1},\ell_{1}),{\mathrm{f}}(a_{2},\ell_{2})\right]$
$\displaystyle=k((a_{1},\ell_{1}),(a_{2},\ell_{2})).$

∎

######  Corollary A.4.

Let $(\Omega,\mathcal{A},\mathrm{P})$ be a probability space and
${\mathbb{U}}$ a real separable Banach space of real-valued functions on a
common domain ${\mathbb{D}}_{\mathbb{U}}$ with continuous point evaluation
functionals $\delta_{x}\colon{\mathbb{U}}\to\mathbb{R},u\mapsto u(x)$. Let
${\mathrm{F}}\colon{\mathbb{A}}\times\Omega\to{\mathbb{U}}$ and
${\mathrm{f}}\colon({\mathbb{A}}\times{\mathbb{D}}_{\mathbb{U}})\times\Omega\to\mathbb{R}$
such that ${\mathrm{F}}(a,\cdot)(x)={\mathrm{f}}((a,x),\cdot)$ for all
$a\in{\mathbb{A}}$ and $x\in{\mathbb{D}}_{\mathbb{U}}$ ($\mathrm{P}$-almost
surely). Then

  1. (i)

${\mathrm{F}}\sim{\operatorname{\mathcal{GP}}\left(\mathcal{M},\mathcal{K}\right)}$
implies ${\mathrm{f}}\sim{\operatorname{\mathcal{GP}}\left(m,k\right)}$,

and, if assumption A.1 holds333 For instance, this is the case if
${\mathbb{U}}$ is a separable RKHS,
${\mathbb{U}}=C({\mathbb{D}}_{\mathbb{U}})$ and ${\mathbb{D}}_{\mathbb{U}}$ is
a compact metric space, or
${\mathbb{U}}=C^{k}(\overline{{\mathbb{D}}_{\mathbb{U}}})$ and
${\mathbb{D}}_{\mathbb{U}}\subset\mathbb{R}^{d}$, which follows from
Propositions B.6, B.7, and B.10 in [Pförtner et al., 2022].  for
$\mathbb{L}_{\delta}\coloneqq\\{\delta_{x}\colon
x\in{\mathbb{D}}_{\mathbb{U}}\\}$,

  1. (ii)

${\mathrm{f}}\sim{\operatorname{\mathcal{GP}}\left(m,k\right)}$ implies
${\mathrm{F}}\sim{\operatorname{\mathcal{GP}}\left(\mathcal{M},\mathcal{K}\right)}$,

where,

  1. (iii)

in both cases,

$\mathcal{M}(a)(x)=m(a,x)$

for all $a\in{\mathbb{A}}$ and $x\in{\mathbb{D}}_{\mathbb{U}}$, as well as

$\mathcal{K}(a_{1},a_{2})(\ell_{2})(x_{1})=\ell_{2}(x_{2}\mapsto
k((a_{1},x_{1}),(a_{2},x_{2})))$

for all $a_{1},a_{2}\in{\mathbb{A}}$, $\ell_{2}\in{\mathbb{U}}$, and
$x_{1}\in{\mathbb{D}}_{\mathbb{U}}$.

Finally, theorem A.3 from the main text is a corollary of the results
developed above.

###### Proof of theorem 3.1.

Follows from corollaries A.4 and 2.1. ∎