# Introduction to Infinite Dimensional Statistics
and Applications

Jan Mandel
University of Colorado Denver

(Version )

###### Contents

  1. 1 Introduction
  2. 2 Motivation: Spatial stochastic models and data assimilation
    1. 2.1 Continuum and discretization
    2. 2.2 From least squares to Bayes theorem
    3. 2.3 Bayes theorem in infinite dimension
    4. 2.4 Random field as a stochastic process
    5. 2.5 Functions of the Laplace operator, random sine series, and the heat kernel
    6. 2.6 Perils of probability in infinite dimension
    7. 2.7 Kalman filter
  3. 3 Hilbert spaces
    1. 3.1 A quick review of basic concepts
    2. 3.2 Tensor product
    3. 3.3 Compact linear operators
    4. 3.4 Trace class operators
    5. 3.5 Hilbert-Schmidt operators
  4. 4 Probability on Hilbert spaces
    1. 4.1 Random elements
    2. 4.2 $L^{p}$ spaces
    3. 4.3 Mean value of random element
    4. 4.4 Mean of tensor product
    5. 4.5 Covariance
    6. 4.6 Convergence of random elements
    7. 4.7 Karhunen-Loève explansion and random orthonormal series
  5. 5 Inequalities
    1. 5.1 Cauchy-Schwarz inequality
    2. 5.2 Hölder’s inequality
    3. 5.3 Chebychev’s inequality
    4. 5.4 Marcinkiewicz-Zygmund inequality
  6. 6 Properties of covariance
    1. 6.1 Independent random elements
    2. 6.2 Bounds on covariance
    3. 6.3 Sample mean and covariance
  7. 7 Laws of large numbers
    1. 7.1 $L^{2}$ and weak law of large numbers
    2. 7.2 $L^{p}$ law of large numbers
    3. 7.3 Convergence of sample covariance
    4. 7.4 Strong laws of large numbers
  8. 8 Gaussian measures
  9. 9 White noise
    1. 9.1 Classical definition of white noise stochastic process
    2. 9.2 White noise as a weak random variable
    3. 9.3 Whitening of a random vector and white noise mapping
    4. 9.4 White noise mapping
  10. 10 Convergence analysis of the EnKF in the large ensemble limit
    1. 10.1 Optimal statistical interpolation
    2. 10.2 Kalman filter
    3. 10.3 The ensemble Kalman filter
    4. 10.4 Finite exchangeable sequences
    5. 10.5 Convergence of the EnKF in the large ensemble limit
  11. 11 The ETKF
    1. 11.1 Base scheme
    2. 11.2 Formulation in the ensemble space
    3. 11.3 Scalar data
  12. A Other approaches to convergence of sample covariance
  13. B White noise measures
  14. C Linear Algebra

##  1 Introduction

These notes started to educate ourselves and to collect some background for
our future work, with the hope that perhaps they will be useful to others
also. The selection of the material is motivated by efforts to simplify and
generalize our analysis of the ensemble Kalman filter [29, 30, 35, 44].
Earlier versions served for short courses at INP-ENSEEIHT and L’École
Nationale de la Météorologie, Toulouse, France, in December 2012, 2013, and
2014.

Many if not all results are more or less elementary or available in the
literature, but we need to fill some holes (which are undoubtely statements so
trivial that the authors we use do not consider them holes at all) or make
straightforward extensions, and then we do the proofs in sufficient detail for
reference. The proofs might be included in a shortened form in a future paper
on an application, so we may present separate proofs for a more general and a
less general case rather than strive for the shortest writing. For the same
reason, we may spell out the argument rather than rely on a reference to a
numbered equation earlier.

The basic Hilbert space framework is based on [15, Ch. 1], but then we deviate
in several ways. Our focus is on the (relatively elementary) algebra of random
elements and mappings rather than deep properties of the probability measures
on infinitely dimensional spaces as found in standard monographs, such as [9,
33, 40], and in fact much of the contents of these notes is taken for granted
in those advanced sources and not even spelled out. For example, [40] notes
that many statements in finite dimension hold true in Hilbert spaces, but he
does not see the need prove the Hilbert space extensions. [57] uses arguments
which carry over to infinite dimension. Our consistent preference for $L^{p}$
norms rather than moments is somewhat unusual in statistics, even if the norms
are commonly used in, for example [11, 40]. The use of $L^{p}$ norms allows us
to use the properties of the norm easily, and it results in a style very
similar to $L^{p}$ spaces in real analysis, which we consider an advantage. We
are interested in estimates with explicit constants and explicit rates of
convergence rather that almost sure convergence or convergence under the
weakest possible conditions.

Although many results extend in a straightforward way to Banach spaces (or at
least reflexive ones), we prefer the simpler expression afforded by the less
general Hilbert space setting. The prerequisites for reading these notes are
only some Lebesgue integral and measure-theoretic probability, and basic
concepts from functional analysis, on the level of introductory graduate
courses. We review some results that should be known from such basic courses,
but we omit their proofs. Instead, we prove only the more unusual statements,
in more detail.

##  2 Motivation: Spatial stochastic models and data assimilation

Much of the recent development in _data assimilation_ and _quantification
uncertainty_ can be described as _statistics of the solutions of partial
differential equations_ , or _statistics of scientific computing_. Here is
picture of an eclectic collection of books I saw in Summer 2012 at the
Parallel Algorithms Group at CERFACS, which says it all:

###  2.1 Continuum and discretization

Physical models are formulated as partial differential equations, which have
solutions in spaces of functions, which are infinitely dimensional. The
computational realization of those models relies on discretizations, resulting
in models with finite dimensional state. Data assimilation methods rely on
multivariate statitics, applied to the discretized, finite dimensional models.
Finer resolution models result in models with higher dimension.

This higher dimension is generally just an artifact of the discretization.
Sometimes we have lower effective dimension: Nonlinear systems, low-
dimensional attractor. Curse of dimensionality: the performance of stochastic
algorithms sometimes deteriorates with the system dimension. But sometimes
not, the physical PDE model has so many scales that refining the
discretization keeps changing the solution. Then the effective dimension of
the model may be large and it has not been captured by the numerical model
yet… but then the adequacy of the numerical model is questionable in the first
place.

Standard approach in numerical PDEs: solution in infinitely dimensional
function space, finite-dimensional is approximation. Similarly here,
stochastic models in infinite dimension, fininite dimensional is an
approximation.

Probability on infinitely-dimensional spaces can be tricky. There are many
statements that should be intuitively true but are not. Combines statistics
and functional analysis, so sometimes called functional statistics.

Asymptotics of interest:

  * •

large number of samples

  * •

large number of time steps

  * •

large dimension from finer discretization

In following sections, we review some of the theoretical foundations and prove
an asymptotic result, convergence of the Ensemble Kalman Filter (EnKF) in the
limit for large number of samples (i.e., ensemble members).

###  2.2 From least squares to Bayes theorem

For symmetric positive definite matrix $A$, denote the vector norm
$\left|u\right|_{A}=\left(u^{\mathrm{T}}Au\right)^{1/2}$.

Data assimilation can be posed as solving approximately the inverse problem
[12, 55]

$\mathcal{H}\left(u\right)=d$

where $u\in U$ is unknown system state, $\mathcal{H}$ is observation function,
and $y$ is the data. Solving by least squares gives

$\left|\mathcal{H}\left(u\right)-d\right|_{R^{-1}}^{2}\rightarrow\min_{u\in
U}.$

However $u$ can be not unique or be uncontrollably large, so we add a
Tikhonov-like regularization term to control the departure from some fixed
$u^{f}$, and get

$\left|\mathcal{H}\left(u\right)-d\right|_{R^{-1}}^{2}+\left|u-u^{f}\right|_{Q^{-1}}^{2}\rightarrow\min_{u\in
U},$

which is equivalent to

$e^{-\frac{1}{2}\left|\mathcal{H}\left(u\right)-d\right|_{R^{-1}}^{2}}e^{-\frac{1}{2}\left|u-u^{f}\right|_{Q^{-1}}^{2}}\rightarrow\max_{u\in
U},$

which has the interpretation of maximizing the _analysis_ probability density
from the Bayes theorem,

$u^{a}=\arg\max_{u\in U}p^{a}\left(u\right),\quad p^{a}\left(u\right)\propto
p\left(d|u\right)p^{f}\left(u\right),$

where $\propto$ means “proportional to”,

$p^{f}\left(u\right)\propto e^{-\frac{1}{2}\left|u-u^{f}\right|_{Q^{-1}}^{2}}$

is the _forecast_ probability density, and

$p(d|u)=e^{-\frac{1}{2}\left|\mathcal{H}\left(u\right)-d\right|_{R^{-1}}^{2}}$
(1)

the _data likelihood_. The forecast is our best knowledge about the state
before the data, expressed as a probability density. Data likelihood describes
the data value as well as its error distribution. Here, That is,
$\mathcal{H}\left(u\right)$ is what the data would be if there were no errors
and the truth were $u$, and $R$ is the covariance of the normally distributed
data error. That is, given $u$, the data is assumed to be distributed as
$d\sim N\left(\mathcal{H}\left(u\right),R\right)$.

###  2.3 Bayes theorem in infinite dimension

All is good when the dimension of the state is finite. However, recall that by
definition, $p$ is a probability density of a measure $\mu$ on
$\mathbb{R}^{n}$ defined by

$\mu\left(A\right)=\int_{A}p(u)d\nu\left(u\right)$ (2)

for all Borel sets $A\subset U$, that is open, closed, or obtained by
countably many set operations from open or closed sets. Such sets are called
_Borel sets_. The integration in (2) is with respect to the Lebesgue measure
$\nu$ (i.e. integration as we know it). But there is no Lebesgue measure on an
infinitely dimensional space (Theorem 4).

The key to applying the Bayes theorem in an infinitely dimensional space is to
realize that the primary object is the measure and not its density. So, write
the Bayes theorem for densities as

$p^{a}(u)\propto p(d|u)p^{f}(u)$

and integrate over a Borel set $A\subset U$,

$\int_{A}p^{a}(u)du\propto\int_{A}p(d|u)p^{f}(u)du.$

Then, the analysis probability measure $\mu^{a}$ is given by

$\mu^{a}\left(A\right)=c\int_{A}p(d|u)d\mu^{f}(u)\quad\text{for all Borel sets
}A\subset U,$ (3)

where the constant is determined from the condition that

$\mu^{a}\left(U\right)=c\int_{U}p(d|u)d\mu^{f}(u)=1.$ (4)

The relation (3) between the measures $\mu^{f}$ and $\mu^{a}$ is called the
_Radon-Nykodym derivative_ and denoted as

$\frac{d\mu^{a}}{d\mu^{f}}\left(u\right)=cp(d|u).$

Unlike densities, the Radon-Nykodym derivative carries over the the general
case of infinitely dimensional space. But, how do we know that

$\int_{U}p(d|u)d\mu^{f}(u)>0$

in (3)?

All is good when the data in (1) is finite dimensional, or, more generally,
$C^{-1}$ is an operator defined everywhere (but we will see that then the data
likelihood cannot come from a probability density, like in finite dimension).
Then the data likelihood $p^{f}\left(u\right)\propto
e^{-\frac{1}{2}\left|u-u^{f}\right|_{Q^{-1}}^{2}}$ is positive everywhere, and
the integral of a nonegative function which is positive on a set of positive
measure is positive.

###  2.4 Random field as a stochastic process

The usual approach in geostatistics [13] is the random field as a stochastic
process, that is a collection of random variables $\left\\{U_{x}\right\\}$,
one for every point $x$ in some domain. The mean at $x$ and covariance between
two points $x$ and $y$ are defined pointwise

$\displaystyle\overline{U}_{x}$ $\displaystyle=E\left(U_{x}\right)$
$\displaystyle C\left(x,y\right)$
$\displaystyle=E\left(\left(U_{x}-\overline{U}_{x}\right)\left(U_{y}-\overline{U}_{y}\right)\right)$

If the mean does not depend on $x$ and the covariance depends on the
difference $x-y$ only,

$C_{U}\left(x,y\right)=f\left(x-y\right)$

the random field $U$ is called stationary and $f$ is the covariance function
of $U$.

One common way of generating a random field is as a sum of a series of
functions with random coefficients,

$U_{x}=m\left(x\right)+\sum_{n=1}^{\infty}c_{n}^{1/2}\xi_{n}V_{n}\left(x\right),$
(5)

where $c_{n}\geq 0$ are constants, $\xi_{n}$ are uncorrelated random variables
with $E\left(\xi_{n}\right)=0$, $E\left(\xi_{n}^{2}\right)=1$, and $V_{n}$ are
orthonormal functions on the spatial domain $S$, that is
${\displaystyle\int\limits_{S}}v_{m}v_{n}dx=0$ if $m\neq n$ and $1$ if $m=n$.
Then

$\displaystyle\overline{U}_{x}$ $\displaystyle=m\left(x\right)$ $\displaystyle
C\left(x,y\right)$
$\displaystyle=E\left[\left(\sum_{m=1}^{\infty}c_{m}^{1/2}\xi_{m}v_{m}\left(x\right)\right)\left(\sum_{n=1}^{\infty}c_{n}^{1/2}\xi_{n}v_{n}\left(y\right)\right)\right]$
$\displaystyle=E\left[\sum_{m=1}^{\infty}c_{m}\xi_{m}^{2}v_{m}\left(x\right)v_{m}\left(y\right)\right]$
$\displaystyle=\sum_{m=1}^{\infty}c_{m}v_{m}\left(x\right)v_{m}\left(y\right)$
(6)

The covariance operator is now a mapping that assigns to a function $w$ the
function $\boldsymbol{C}w$, defined by

$\left(\boldsymbol{C}w\right)\left(x\right)={\displaystyle\int\limits_{S}}C\left(x,y\right)w\left(y\right)dy.$

and $c_{n}$ are eigenvalues and $v_{n}$ the eigenvectors of the covariance
operator $\boldsymbol{C}$, that is,

$\boldsymbol{C}v_{n}=c_{n}v_{n},$

because

$\displaystyle\left(\boldsymbol{C}v_{n}\right)\left(x\right)$
$\displaystyle={\displaystyle\int\limits_{S}}\sum_{m=1}^{\infty}c_{m}v_{m}\left(x\right)v_{m}\left(y\right)v_{n}\left(y\right)dy$
$\displaystyle=\sum_{m=1}^{\infty}c_{m}v_{m}\left(x\right){\displaystyle\int\limits_{S}}v_{m}\left(y\right)v_{n}\left(y\right)dy=c_{n}v_{n}\left(x\right)$

assuming that switching the integral and the infinite sum is justified. See,
e.g., [47] studies of the convergence of random series. We will see later that
an operator $\boldsymbol{C}$ is the covariance of a probability measure if and
only if it has finite trace,

$\operatorname*{Tr}\boldsymbol{C}=\sum_{n=1}^{\infty}c_{n}<\infty.$ (7)

When the trace condition (7) is not satisfied, random series (5) not give a
random element in the usual sense or even converge. The extreme case is white
noise, where all $c_{n}=1$.

We find it more convenient to consider a realization of the random process
$U_{x}$ a random function (also called a path), and even better a random
element of some Hilbert space, which leads to considerable simplification. We
are then led naturally to the study of random elements and probability
measures on Hilbert spaces.

In computational applications, the model runs in a finite dimensional space,
such as values on a grid in the spatial domain $S$, and one can expect that
the statistical properties of the model then approach the infinite-dimensional
case as the grid gets refined. In some methods such as stochastic Galerkin
[3], rigorous analysis of such approximations is available. Engineering
literature, however, typically relies on heuristic arguments and computational
experiments, e.g. [62].

###  2.5 Functions of the Laplace operator, random sine series, and the heat
kernel

Can one use a function of the Laplace operator

$-\Delta=-\frac{\partial^{2}}{\partial x^{2}}-\frac{\partial^{2}}{\partial
y^{2}}$

as on the rectangle as a covariance operator? Consider the Laplace operator on
a rectangle

$S=\left(0,a\right)\times\left(0,b\right),$

with boundary conditions $u=0$ on $\partial S$. The eigenvectors and
eigenvalues of $-\Delta$,

$-\Delta u_{k\ell}=\lambda_{k\ell}u_{k\ell}$

are

$u_{k\ell}\left(x,y\right)=\sin\left(\frac{x}{a}k\pi\right)\sin\left(\frac{y}{b}\ell\pi\right)$
$\lambda_{k\ell}=\left(\frac{k\pi}{a}\right)^{2}+\left(\frac{\ell\pi}{b}\right)^{2}.$

Using the negative powers of the Laplacian
$\left(-\Delta\right)^{{}^{-\alpha}}$ as the covariance gives the random sum

$U\left(x,y\right)=\sum_{k=1}^{\infty}\sum_{\ell=1}^{\infty}\lambda_{k\ell}^{-\alpha/2}\xi_{k\ell}\sin\left(\frac{x}{a}k\pi\right)\sin\left(\frac{y}{b}\ell\pi\right)$
(8)

where $\xi_{k\ell}\sim N\left(0,1\right)$ are independent. The numbers
$\left(\lambda_{k\ell}\right)^{-\alpha}$ are the eigenvalues of the operator
$\left(-\Delta\right)^{{}^{-\alpha}}$. Now consider when $U$ is square
integrable almost surely:

$\displaystyle
E\left[\int_{0}^{a}\int_{0}^{b}\left|U\left(x,y\right)\right|^{2}dydx\right]$
$\displaystyle=\sum_{k=1}^{\infty}\sum_{\ell=1}^{\infty}\lambda_{k\ell}^{-\alpha}E\left[\left|\xi_{k\ell}\right|^{2}\right]\int_{0}^{a}\int_{0}^{b}\sin^{2}\left(\frac{x}{a}k\pi\right)\sin^{2}\left(\frac{y}{b}\ell\pi\right)dydx$
$\displaystyle=\operatorname*{const}\sum_{k=1}^{\infty}\sum_{\ell=1}^{\infty}\lambda_{k\ell}^{-\alpha}<\infty.$

Substituting and by summation over diagonals $k+\ell=n$, we see that up to a
constant multiplication factor,

$\displaystyle\sum_{k=1}^{\infty}\sum_{\ell=1}^{\infty}\lambda_{k\ell}^{-\alpha}$
$\displaystyle=\operatorname*{const}\sum_{k=1}^{\infty}\sum_{\ell=1}^{\infty}\left(\frac{1}{k^{2}+\ell^{2}}\right)^{\alpha}$
$\displaystyle\approx\operatorname*{const}\sum_{k=1}^{\infty}\sum_{\ell=1}^{\infty}\left(\frac{1}{k+\ell}\right)^{2\alpha}$
$\displaystyle=\sum_{n=1}^{\infty}\sum_{\ell=1}^{n}\left(\frac{1}{n}\right)^{2\alpha}$
$\displaystyle=\sum_{n=1}^{\infty}n\left(\frac{1}{n}\right)^{2\alpha}=\sum_{n=1}^{\infty}n^{1-2\alpha}<\infty$

if and only if $1-2\alpha<-1$, that is $\alpha>1$. Hence, we have:

###### Theorem 1

$E\left[\int_{0}^{a}\int_{0}^{b}\left|U\left(x,y\right)\right|^{2}dydx\right]<\infty$
a.s. if and only if $\alpha>1$. In particular, if $\alpha>1$, then the series
(8) converges to a sum $U\in L^{2}\left(S\right)$ a.s.

In a similar way, we show that higher powers $\alpha$ make the random
functions smooth. Consider the partial derivatives of $U$ of order $s=p+q$,

$\displaystyle E\int_{0}^{a}\int_{0}^{b}\left|\frac{\partial^{s}U}{\partial
x^{p}y^{q}}\right|^{2}dydx$
$\displaystyle=\operatorname*{const}\sum_{k=1}^{\infty}\sum_{\ell=1}^{\infty}\frac{k^{2p}\ell^{2q}}{\lambda_{k\ell}^{\alpha}}$
$\displaystyle=\operatorname*{const}\sum_{k=1}^{\infty}\sum_{\ell=1}^{\infty}\frac{k^{2p}\ell^{2q}}{\left(k^{2}+\ell^{2}\right)^{\alpha}}$
$\displaystyle\approx\operatorname*{const}\sum_{k=1}^{\infty}\sum_{\ell=1}^{\infty}\frac{\left(k+\ell\right)^{2s}}{\left(k+\ell\right)^{2\alpha}}$
$\displaystyle=\operatorname*{const}\sum_{n=1}^{\infty}n^{2s-2\alpha+1}<\infty$

if and only if $2s-2\alpha+1<-1$, that is $\alpha>1+s$. Functions with all
partial derivatives up to order $s$ square integrable are said to be in the
Sobolev space $H^{s}\left(S\right)$, hence we have

$U\in H^{s}\left(S\right)\text{ if }\alpha>1+s\text{, }s\geq 0.$

Note that this result holds even if $s$ is not integer, and in fact one can
define Sobolev spaces with non-integer exponent precisely like this, by
differentiating Fourier series. By the _Sobolev embedding theorem_ , in two
dimensions, functions in $H^{s}$ are continuous if $s>1$. Consequently, $U$
given by (8) is almost surely continuous if $\alpha>2$, differentiable if
$\alpha>3$, etc.

The heat kernel

$K_{T}:u_{0}\mapsto u_{T}$

is defined by the solution $u_{T}=u\left(T,x\right)$ of the heat equation

$\frac{\partial u}{\partial t}=\Delta u\text{ in }S$ (9)

with initial condition $u\left(0,x\right)=u_{0}$ and $u\left(t,x\right)=0$ on
$\partial S$. Just like in elementary differential equations, this equation
has the solution

$u\left(t\right)=e^{-t\Delta}u\left(0\right),$

where the exponential of the operator is defined by subtituting in the power
series,

$e^{-t\Delta}=I-t\Delta+\frac{1}{2!}t^{2}\Delta^{2}-\frac{1}{3!}t^{3}\Delta^{3}+\cdots$

The eigenvectors of the function of an operator remain the same and the
eigenvalues transform by substituting them into the function according to the
spectral mapping theorem. For the example when
$S=\left(0,a\right)\times\left(0,b\right)$ as above, we have eigenvalues of
the heat kernel $K_{T}$,

$e^{-T\lambda_{k\ell}}=e^{-T\left[\left(\frac{k\pi}{a}\right)^{2}+\left(\frac{\ell\pi}{b}\right)^{2}\right]}$

which go to zero faster than any $\left(\lambda_{k\ell}\right)^{-\alpha}$.
Consequently, random function with heat kernel covariance has continuous
partial derivatives of all orders a.s. The use of the Green’s function of the
Laplace equation (that is, $\left(-\Delta^{-1}\right)$ for covariance was
suggested in [31], and several functions of the Laplace operator including the
heat kernel were proposed in [49, 58].

Finite dimensional versions of the operations with the random field and its
covariance are easily and cheaply implemented by FFT. Let

$h_{x}=\frac{a}{m+1},\text{ }h_{y}=\frac{b}{n+1}.$ (10)

Discretizing the operator $-\Delta$ by finite differences on the the uniform
mesh with the nodes

$\left(ih_{x},jh_{y}\right),\quad i=1,\ldots,m,\text{ }j=1,\ldots,n.$

we obtain the linear operator $L_{mn}:\mathbb{R}^{m\times
n}\rightarrow\mathbb{R}^{m\times n}$, given by

$L_{mn}:u\mapsto v,\quad
v_{ij}=\frac{-u_{i+1,j}+2u_{ij}-u_{i-1,j}}{h_{x}^{2}}+\frac{-u_{i,j-1}+2u_{ij}-u_{i,j-1}}{h_{y}^{2}},$

where the values off the grid are taken as
$u_{0j}=u_{m+1,j}=u_{i,0}=u_{i,n+1}=0$. The eigenvectors of $L_{mn}$ are
$u_{k\ell}$, given by

$\left[u_{k\ell}\right]_{ij}=\sin\left(\frac{i}{m+1}k\pi\right)\sin\left(\frac{j}{n+1}\ell\pi\right),\quad
k=1,\ldots,m,\text{ }\ell=1,\ldots,n,$ (11)

and using the trigonometric identity

$-\sin(A-B)+2\sin A-\sin(A+B)=2\sin A(1-\cos B)=4\sin A\sin^{2}\frac{B}{2}$

we have the corresponding eigenvalues $\lambda_{k\ell}$, given by

$\lambda_{mn,k\ell}=4\left(\frac{\sin\frac{\pi}{2\left(m+1\right)}k}{\frac{a}{m+1}}\right)^{2}+4\left(\frac{\sin\frac{\pi}{2\left(n+1\right)}\ell}{\frac{b}{n+1}}\right)^{2},\quad
k=1,\ldots,m,\text{ }\ell=1,\ldots,n.$ (12)

The eigenvalues of the discretized operator approach the exact eigenvalues

$\lim_{m,n\rightarrow\infty}\lambda_{mn,k\ell}=\left(\frac{k\pi}{a}\right)^{2}+\left(\frac{\ell\pi}{b}\right)^{2},$

for fixed eigenvalue number $k,\ell$. The decomposition of any vector in the
basis of eigenvectors is computed by the Fast Fourier Transform (FFT).

[Matlab demo comes here]

###  2.6 Perils of probability in infinite dimension

Definitions of measure and density need to be taken seriously now - intuitive
approach is no longer good enough. Sigma-additive translation-invariant or
rotation invariant measure finite on balls does not exist in infinite
dimension.

No Lebesgue measure, no density in the usual sense. What happens to Bayes
theorem? Density-free formulation by integrals (Radon Nikodym theorem). No
need for reference measure. Data likelihood need not be density. In fact, we
have seen that it better should not be.

  * •

What exactly it means a random variable, random element? The answer will
depend on the topology.

  * •

Can we do everything without probability densities?

  * •

What happens to the Bayes theorem and data assimilation?

  * •

How do you even integrate functions with values in infinite dimensional
spaces? The mean of a random variable is integral.

  * •

What are the requirements on the coefficients of a series to guarantee
convergence a.s., convergence to a continuous function a.s.?

  * •

Elements of a Hilbert space need not be functions, so there need not be any
concept of values at points. Then, what exactly is covariance and how does it
relate to what we know for finite dimension, and for a stochastic process?

  * •

How do common statistics formulas and inequalities carry over to infinite
dimension?

  * •

Many standard arguments in statistics are usually done term-by-term. Now what
in infinite dimension?

  * •

How do the laws of large numbers carry over?

  * •

What is white noise (a random vector with identity as the covariance) in an
infinitely dimensional space?

###  2.7 Kalman filter

Data assimilation, also known as sequential statistical estimation, or cycle
between the application of the Bayes theorem, called analysis stem, and
densities gaussian, represented by mean and covariance matrix, advances
covariance by linear model. Needs covariance matrix to evaluate the Bayes
theorem.

Extended Kalman filter and variants: approximation of the covariance when the
model is not linear.

Cannot maintain covariance matrix when the system dimension is large.

Ensemble Kalman filter EnKF: replaces the covariance by sample covariance
computed from an ensemble of simulations. Still assumes that the distributions
are gaussian, but formulas do not depend on it, so usually used anyway.

Particle filter: represents the densities by a weighted ensemble, ensemble
members now called particles. Application of the Bayes theorem updates the
weights. Needs many particles to cover the locations where the densities are
large.

Curse of dimensionality: when the dimension of the system state grows,
exponentially more particles are needed [6]. But not always - why? Depends on
state distribution. When the effective dimension is low, convergence is good.
We will show that this happens always when the state probability distribution
is actually a probability distribution on the state space. This will depend on
the behavior of the eigenvalues of the covariance - they need to drop off fast
enough.. The assumptions in [6] that the eigenvalues are bounded away from $0$
make the state distribution similar to white noise. Contrary to the lore, this
applies to both the EnKF and the particle filter [5].

Curse of dimensionality? Not for probability measures!

|
---|---

Constant covariance eigenvalues $\lambda_{n}=1$ and the inverse law
$\lambda_{n}=1/n$ are not probability measures in the limit because
$\sum_{n=1}^{\infty}\lambda_{n}=\infty$.
Inverse square law $\lambda_{n}=1/n^{2}$ gives a probability measure because
$\sum_{n=1}^{\infty}\lambda_{n}<\infty$.
m=25 uniformly sampled data points from 1D state, N=10 ensemble members. From
[5, Fig. 4.7].

##  3 Hilbert spaces

We first review some material on Hilbert spaces to set the stage.

###  3.1 A quick review of basic concepts

The material in this section can be found in any textbook of introductory
functional analysis and many others, for example [32, 37].

A complete inner product space is called a _Hilbert space_. A space is
separable if it has a countable dense subset. We will work with separable
spaces only. Let $H$ be a real, separable Hilbert space equipped with the
inner product $\left\langle\cdot,\cdot\right\rangle$ and the norm
$\left|\cdot\right|$. We use single bars because double bars
$\left\|\cdot\right\|$ will be reserved for stochastic norms. If $x,y\in H$
and $\left\langle x,y\right\rangle=0$, then $x$ and $y$ are orthogonal, which
is denoted as $x\perp y$.

The symbol $\left|\cdot\right|$ will also denote the operator norm on the
space of continous (or, equivalently, bounded) linear operators
$\left[H,K\right]$, where $H$ and $K$ are Hilbert spaces, and $\left[H\right]$
stands for $\left[H,H\right]$. The space $\left[H,K\right]$ is equipped with
the norm

$\left|A\right|_{\left[H,K\right]}=\sup_{u\in H,u\neq
0}\frac{\left|Au\right|_{K}}{\left|u\right|_{H}}$

which makes it a Banach space (a complete normed linear space), but the norm
is not induced by an inner product, so $\left[H,K\right]$ is not a Hilbert
space. The Euclidean space $\mathbb{R}^{n}$ is a Hilbert space with the inner
product $\left\langle a,b\right\rangle=a^{\mathrm{T}}b$. We will use
subscripts to designate the space, such as in
$\left\langle\cdot,\cdot\right\rangle_{H}$, only if there is a danger of
confusion.

The space of all linear functionals on $H$ is called the _algebraic dual_ of
$H$, and denoted by $H^{\\#}$. The space of all continuous (or, equivalently,
bounded) linear functionals on $H$ is the dual space of $H$, and denoted by
$H^{\prime}$. If $v\in H$, then the mapping

$f:u\mapsto\left\langle u,v\right\rangle$

is clearly a bounded linear functional, and its norm is

$\left|f\right|_{H^{\prime}}=\sup_{u\in H,u\neq 0}\frac{\left|\left\langle
u,v\right\rangle\right|}{\left|u\right|}=\left|v\right|.$

The _Riesz representation theorem_ states that every bounded linear functional
$f$ on $H$ is of this form, $f:u\mapsto\left\langle u,v\right\rangle$ for some
$v\in H$. This allows to consider the spaces $H$ and $H^{\prime}$ to be
identified: for a linear functional $F$ on $H$, we use the notation

$F\left(u\right)=\left\langle u,F\right\rangle,$ (13)

called _duality pairing_. If $F$ is bounded, $F$ is considered to coincide
with an element of $H$ and $\left\langle\cdot,\cdot\right\rangle$ with the
inner product. However, we will use the notation (13) for unbounded
functionals also. Then $H=H^{\prime}\subset H^{\\#}$ and the duality pairing
$\left\langle\cdot,\cdot\right\rangle$ extends the inner product to $H\times
H^{\\#}$_._

One important consequence of the Riesz representation theorem is
representation of bounded bilinear forms by bounded operators, which we will
need to define covariance operators. For simplicity (and the application we
have in mind), consider the case of real space $H$ only.

###### Lemma 2

If $a\left(u,v\right)$ is a bilinear form $H\times H\rightarrow\mathbb{R}$
such that

$\forall u,v\in H:\left|a\left(u,v\right)\right|\leq
M\left|u\right|\left|v\right|,$

then there exists unique linear operator on $H$ such that

$\forall u,v\in H:\left\langle u,Av\right\rangle=a\left(u,v\right),$

and $A\in\left[H\right]$ with $\left|A\right|\leq M$.

Proof. The map $u\mapsto a\left(u,v\right)$ for a fixed $v$ is a bounded
linear functional, so it can be written as $\left\langle u,Av\right\rangle$
for some $Av.$

Similarly, for any $A\in\left[H,K\right]$, where $H$ and $K$ are Hilbert
spaces, the Riesz representation theorem implies the existence of the adjoint
operators, which is defined uniquely by

$\left\langle u,Av\right\rangle_{K}=\left\langle
A^{\ast}u,v\right\rangle_{H},\quad\forall u\in K,v\in H.$

An operator $A\in\left[H\right]$ is called self-adjoint if $A^{\ast}=A$,
positive definite if $\left\langle Au,u\right\rangle>0$ for all $u\in H$,
$u\neq 0$, and positive semidefinite if $\left\langle Au,u\right\rangle\geq 0$
for all $u\in H$.

A  _complete orthonormal system_ $\left\\{e_{n}\right\\}\subset H$ is a
sequence of vectors $e_{n}$ such that

$\left\langle e_{m},e_{n}\right\rangle=0\text{ if }m\neq n\text{,
}\left\langle e_{n},e_{n}\right\rangle=1,$

and the _span of_ $\left\\{e_{n}\right\\}$_is dense_ in $H$. A separable
Hilbert space has a complete orthonormal system (in fact, infinitely many).
Given a complete orthonormal system $\left\\{e_{n}\right\\}$, any vector $u\in
H$ can be decomposed uniquely into an _abstract Fourier series_

$u=\sum_{n=1}^{\infty}c_{n}e_{n},\quad c_{n}=\left\langle
u,e_{n}\right\rangle,$ (14)

and the _Parseval equality_ holds,

$\sum_{n=1}^{\infty}\left|c_{n}\right|^{2}=\left|u\right|^{2}.$ (15)

In many physical systems, $\left|u\right|^{2}$ corresponds to a total energy
of the system, and $\left|c_{n}\right|^{2}$ is the energy of mode $n$ (such as
a vibration or turbulence mode).

The space $\left[H\right]$ of all bounded linear operators on an infinite
dimensional space is a Banach space and it is not separable: the diagonal
operator

$\sum_{n=1}^{\infty}c_{n}e_{n}\mapsto\sum_{n=1}^{\infty}d_{n}c_{n}e_{n},\quad
d_{n}=0\text{ or }1$

is in $\left[H\right]$, but the distance between any such two operators in
$\left[H\right]$ is at least $1$, and there are uncountably many sequences of
zeros and ones. In fact, the space $\left[H\right]$ is so general that in any
Banach space can be embedded into the space $\left[H\right]$ for some Hilbert
space $H$. The space $\left[H\right]$ will not be very useful because no
special properties can hold for $\left[H\right]$ which would not be true for
an arbitrary Banach space. Later, we will look at some subspaces of
$\left[H\right]$ with some useful properties.

###  3.2 Tensor product

When $x$ and $y$ are vectors in $\mathbb{R}^{n}$, their product
$xy^{\mathrm{T}}$ is a rank-one matrix. Note that for $z\in\mathbb{R}^{n}$,

$\left(xy^{\mathrm{T}}\right)z=x\left(y^{\mathrm{T}}z\right)=x\left\langle
z,y\right\rangle$ (16)

Therefore, the replacement for a rank-one matrix in Hilbert space is the
tensor product $x\otimes y$, which is a linear operator defined the same way
as (16),

$x\otimes y\in\left[H\right],\quad\left(x\otimes y\right)z=x\left\langle
z,y\right\rangle\quad\forall z\in H$

or, equivalently, through the Riesz representation theorem,

$x\otimes y\in\left[H\right],\quad\left\langle u,\left(x\otimes
y\right)v\right\rangle=\left\langle u,x\right\rangle\left\langle
y,v\right\rangle\quad\forall u,v\in H.$ (17)

See, for example, [16, page 25]. Then,

$\left|x\otimes y\right|=\left|x\right|\left|y\right|$ (18)

This is clear when $y=0$. For $y\neq 0$,

$\frac{\left|\left(x\otimes
y\right)z\right|}{\left|z\right|}=\frac{\left|x\left\langle
y,z\right\rangle\right|}{\left|z\right|}=\frac{\left|x\right|\left|\left\langle
y,z\right\rangle\right|}{\left|z\right|}\leq\frac{\left|x\right|\left|y\right|\left|z\right|}{\left|z\right|}=\left|x\right|\left|y\right|$

with equality when $z=y$. Continuity of the tensor product follows,

$\displaystyle\left|x\otimes y-w\otimes z\right|$
$\displaystyle=\left|\left(x-w\right)\otimes
y+w\otimes\left(y-z\right)\right|$
$\displaystyle\leq\left|z-w\right|\left|y\right|+\left|y-z\right|\left|w\right|.$
(19)

If $\left|x\right|=1$, $x\otimes x$ is the orthogonal projection on the span
of $x$:

$\left(x\otimes x\right)z=x\left\langle x,z\right\rangle=0\Leftrightarrow
z\perp x,\quad\left(x\otimes x\right)x=x\left\langle x,x\right\rangle=x.$

###  3.3 Compact linear operators

A linenar operator $A:H\rightarrow K$ is compact if for every bounded sequence
$\left\\{u_{n}\right\\}\subset H$, $\left\\{Au_{n}\right\\}$ has a convergent
subsequence in $K$. A compact operator is bounded. Linear operator
$A\in\left[H,K\right]$ is compact if and only if there is a sequence of
operators $A_{n}$ with finite-dimensional range which converge to $A$ in the
operator norm, $\lim_{n\rightarrow 0}\left|A-A_{n}\right|=0$.

The spectrum $\sigma\left(A\right)$ of a linear operator $A$ is the set of all
$\lambda\in\mathbb{C}$ such that $\left(\lambda I-A\right)^{-1}$ does not
exist as an operator in $\left[H\right]$. This concept of spectrum generalizes
eigenvalues in finite dimension. The spectrum is a closed subset of
$\mathbb{C}$. The spectrum of a linear operator can be quite general, however
the spectrum of a compact operator in $\left[H\right]$ consists of isolated
points, which are eigenvalues of finite multiplicity, and, unless $H$ is
finite dimensional, also zero, which can be the only accumulation point of the
spectrum. The _spectral theorem for selfadjoint compact operators_ guarantees
that if $A=A^{\ast}$ is compact, then there is a complete orthonormal system
$\left\\{u_{n}\right\\}$ consisting of eigenvectors of $A$,

$Au_{n}=\lambda_{n}u_{n},\quad\left\langle u_{m},u_{n}\right\rangle=0\text{ if
}m\neq n\text{, }\left|u_{n}\right|=1,$

the eigenvalues $\lambda_{n}$ of $A$ are real, and $A$ is the infinite sum

$A=\sum_{n=1}^{\infty}\lambda_{n}u_{n}\otimes u_{n},$ (20)

which converges in the operator norm. The tensor products $u_{n}\otimes u_{n}$
are called spectral projections.

The _spectral decomposition_ (20) allows an easy definition of functions of a
compact operator: if a function $f$ is continuous on the spectrum
$\sigma\left(A\right)\subset\mathbb{R}$ of a compact self-adjoint operator
$A$, then

$f\left(A\right)=\sum_{n=1}^{\infty}f\left(\lambda_{n}\right)u_{n}\otimes
u_{n},$ (21)

and the action of $f\left(A\right)$ on a vector

$v=\sum_{n=1}^{\infty}c_{n}u_{n}$

is obtained by multiplying the Fourier coefficients $c_{n}$ by
$f\left(\lambda_{n}\right)$,

$f\left(A\right)v=\sum_{n=1}^{\infty}f\left(\lambda_{n}\right)u_{n}\left\langle
v,u_{n}\right\rangle=\sum_{n=1}^{\infty}f\left(\lambda_{n}\right)c_{n}u_{n}.$
(22)

The definition of $f\left(A\right)$ is consistent with the definition of
positive integer powers of $A$. More generally, when $f$ can be expanded into
a power series convergent on an open circle in $\mathbb{C}$ containing
$\sigma\left(A\right)$, $f\left(A\right)$ is obtained also by simply
substituting the operator $A$ in the power series. According to the _spectral
mapping theorem_ , the spectrum of $f\left(A\right)$ is

$\sigma\left(f\left(A\right)\right)=f\left(\sigma\left(A\right)\right)=\left\\{f\left(\lambda\right)|\lambda\in\sigma\left(A\right)\right\\}.$

This is again consistent with the behavior of eigenvalues of powers of
matrices.

The definition (21) also applies to negative powers $A^{-\alpha}$, $\alpha>0$,
of self-adjoint positive definite compact operators, but the result is an
_unbounded operator defined only on the dense subspace_
$A^{\alpha}\left(H\right)$ _of_ $H$. Note that $0$ cannot be an eigenvalue of
$A$, because $A$ is assumed to be positive definite, so all $\lambda_{n}>0$.
Then we define $A^{-\alpha}v$ only for

$v\in A^{\alpha}\left(H\right)=\left\\{A^{\alpha}w|w\in
H\right\\}=\left\\{\left.\sum_{n=1}^{\infty}c_{n}u_{n}\right|c_{n}=\lambda_{n}^{\alpha}d_{n},\sum_{n=1}^{\infty}\left|d_{n}\right|^{2}<\infty\right\\}$

by

$A^{-\alpha}v=\sum_{n=1}^{\infty}\lambda_{n}^{-\alpha}c_{n}u_{n},\quad
v=\sum_{n=1}^{\infty}c_{n}u_{n}.$

It is easy to see that $A^{-\alpha}$ is not a bounded operator because

$\frac{\left|A^{-\alpha}u_{n}\right|}{\left|u_{n}\right|}=\lambda_{n}^{-\alpha}\rightarrow\infty\text{
as }n\rightarrow\infty$

because $0$ is the only accumulation point of the eigenvalues $\lambda_{n}>0$.
The space $A^{\alpha}\left(H\right)$ for $\alpha>0$ is a dense subspace of
$H$, because it contains all linear combinations (which are defined as having
finitely many terms) of the complete orthonormal sets of eigenvectors
$\left\\{u_{n}\right\\}$ of $A$.

If $A$ is compact, then $A^{\ast}A$ is compact self-adjoint positive
semidefinite operator. The square roots of the eigenvalues of $A^{\ast}A$ are
called _singular values_ of $A$, and they are denoted by $\sigma_{n}$. With
the corresponding eigenvectors $v_{n}$ of $A^{\ast}A$, we have

$\left(A^{\ast}A\right)^{1/2}=\sum_{n=1}^{\infty}\sigma_{n}v_{n}\otimes
v_{n},\quad A^{\ast}Av_{n}=\sigma_{n}^{2}v_{n}.$

It is an easy exercise to verify that, just like in finite dimension,

$\left|A\right|=\max\sigma_{n},$

and, in addition, if $A$ is self-adjoint and positive semidefinite, then the
singular values and eigenvalues are the same, $\sigma_{n}=\lambda_{n}$, and
$\left(A^{\ast}A\right)^{1/2}=\left(A^{2}\right)^{1/2}=A$.

###  3.4 Trace class operators

An operator $A\in\left[H\right]$ is trace class operator if for some
orthonormal sequence $\left\\{e_{n}\right\\}$ in $H,$

$\sum_{n=1}^{\infty}\left\langle\left(A^{\ast}A\right)^{1/2}e_{n},e_{n}\right\rangle<\infty.$

The space of all trace class operators is denoted by
$\operatorname*{Tr}\left(H\right)$ and it is a Banach space equipped with the
norm

$\left|A\right|_{\operatorname*{Tr}\left(H\right)}=\sum_{n=1}^{\infty}\left\langle\left(A^{\ast}A\right)^{1/2}e_{n},e_{n}\right\rangle=\operatorname*{Tr}\left(A^{\ast}A\right)^{1/2},$

where the trace of a selfadjoint operator
$A\in\operatorname*{Tr}\left(H\right)$, $A=A^{\ast}$ is defined by

$\operatorname*{Tr}A=\sum_{n=1}^{\infty}\left\langle
Ae_{n},e_{n}\right\rangle$

The value of $\operatorname*{Tr}A$ and thus
$\left|A\right|_{\operatorname*{Tr}\left(H\right)}$ do not depend on the
choice of the orthonormal sequence $\left\\{e_{n}\right\\}$: if
$\left\\{f_{m}\right\\}$ is another orthonormal sequence, we expand

$Ae_{n}=\sum_{m=1}^{\infty}\left\langle Ae_{n},f_{m}\right\rangle f_{m}$

and we get

$\displaystyle\sum_{n=1}^{\infty}\left\langle Ae_{n},e_{n}\right\rangle$
$\displaystyle=\sum_{n=1}^{\infty}\left\langle\sum_{m=1}^{\infty}\left\langle
Ae_{n},f_{m}\right\rangle f_{m},e_{n}\right\rangle$
$\displaystyle=\sum_{n=1}^{\infty}\sum_{m=1}^{\infty}\left\langle
Ae_{n},f_{m}\right\rangle\left\langle f_{m},e_{n}\right\rangle$
$\displaystyle=\sum_{m=1}^{\infty}\sum_{n=1}^{\infty}\left\langle
Af_{m},e_{n}\right\rangle\left\langle e_{n},f_{m}\right\rangle$
$\displaystyle=\sum_{m=1}^{\infty}\left\langle\sum_{n=1}^{\infty}\left\langle
Af_{m},e_{n}\right\rangle e_{n},f_{m}\right\rangle$
$\displaystyle=\sum_{m=1}^{\infty}\left\langle Af_{m},f_{m}\right\rangle.$

Trace class operators are compact, therefore all points of their spectrum are
eigenvalues of finite multiplicity except possibly for the point zero, and
zero is the only possible cluster point of the spectrum. Trace class operators
will play an important role as covariance operators of random elements in a
Hilbert space.

###  3.5 Hilbert-Schmidt operators

An operator $A\in\left[H\right]$ is Hilbert-Schmidt operator if for an
orthonormal sequence $\left\\{e_{n}\right\\}$ in $H,$

$\sum_{n=1}^{\infty}\left\langle
Ae_{n},Ae_{n}\right\rangle=\sum_{n=1}^{\infty}\left\langle
A^{\ast}Ae_{n},e_{n}\right\rangle=\sum_{n=1}^{\infty}\left|Ae_{n}\right|^{2}<\infty.$

The space of all Hilbert-Schmidt operators is denoted by $HS\left(H\right)$
and it is a separable Hilbert space equipped with the inner product

$\left\langle
A,B\right\rangle_{HS\left(H\right)}=\sum_{n=1}^{\infty}\left\langle
Ae_{n},Be_{n}\right\rangle,$ (23)

and the norm

$\left|A\right|_{HS\left(H\right)}=\left\langle
A,A\right\rangle_{HS\left(H\right)}^{1/2}=\left(\sum_{n=1}^{\infty}\left|Ae_{n}\right|^{2}\right)^{1/2},$

which again do not depend on the choice of $\left\\{e_{n}\right\\}$, just like
the trace. Hilbert-Schmidt operators are also compact.

With $\sigma_{n}$ the singular values of $A$ and $e_{n}$ the singular vectors,
we see immediately that

$\max\sigma_{n}\leq\left(\sum_{n=1}^{\infty}\sigma_{n}^{2}\right)^{1/2}\leq\sum_{n=1}^{\infty}\sigma_{n},$

thus

$\left|A\right|\leq\left|A\right|_{HS\left(H\right)}\leq\left|A\right|_{\operatorname*{Tr}\left(H\right)}.$
(24)

In particular, every trace-class operator is also a Hilbert-Schmidt operator,
which is a bounded operator,

$\left[H\right]\supset
HS\left(H\right)\supset\operatorname*{Tr}\left(H\right).$

The Hilbert-Schmidt norm of a tensor product $x\otimes x$ can be computed
easily.

###### Lemma 3

For any $x\in H$,

$\left|x\otimes x\right|_{HS\left(H\right)}=\left|x\right|_{H}^{2}.$ (25)

Proof. Let $\left\\{e_{n}\right\\}$ be an orthonormal sequence in $H$. Then,

$\displaystyle\left|x\otimes x\right|_{HS\left(H\right)}^{2}$
$\displaystyle=\sum_{n=1}^{\infty}\left|\left(x\otimes
x\right)e_{k}\right|_{H}^{2}$
$\displaystyle=\sum_{n=1}^{\infty}\left|x\left\langle
x,e_{k}\right\rangle_{H}\right|_{H}^{2}$
$\displaystyle=\left|x\right|_{H}^{2}\sum_{n=1}^{\infty}\left|\left\langle
x,e_{k}\right\rangle\right|^{2}$
$\displaystyle=\left|x\right|_{H}^{2}\left|x\right|_{H}^{2}$

by Parseval’s equality.

In the finite-dimensional case with $H=\mathbb{R}^{m}$, $\left\langle
u,v\right\rangle=\sum_{n=1}^{m}u_{n}v_{n}$, the Hilbert-Schmidt operator norm
is the Frobenius matrix norm: With $e_{n}$ the canonical basis vectors, we
have

$\left|A\right|_{HS\left(H\right)}^{2}=\sum_{n=1}^{m}\left|Ae_{n}\right|^{2}=\sum_{n=1}^{m}\left|A_{:,n}\right|^{2}=\sum_{n=1}^{m}\sum_{k=1}^{m}a_{kn}^{2},$

where $A_{:,n}$ denotes the $n-th$ column of $A$.

The utility of the space Hilbert-Schmidt operators comes from the facts that
it is a Hilbert space, and that sample covariances are Hilbert-Schmidt
operators.

See, for example [16, 33, 37], and a short introduction in [14] for more
details on trace-class and Hilbert-Schmidt operators.

##  4 Probability on Hilbert spaces

Because we need to question all established concepts to see what survives in
infinite dimension, we need to revisit the basics of probability theory. The
intuitive probability is not good enough here; to prove anything and navigate
safely, we need measure-theoretic probability. To start with, the concept of
probablity density (with respect to Lebesgue measure), which is ubiquitous in
finite dimensional probability, is no longer available, because there is
Lebesgue measure in infinite dimension.

###### Theorem 4

Let $U$ be infinitely dimensional normed linear space. There is no measure on
Borel sets of $U$ that is invariant to shifts or invariant to rotations, and
positive but finite on all balls.

Proof. We show the proof on an example. Consider the space $\ell^{2}$ of real
sequences $v=\left\\{v_{n}\right\\}$ with the norm
$\left|v\right|=\left(v_{1}^{2}+v_{2}^{2}+\cdots\right)^{1/2}$, and suppose
that $\mu$ is such measure on $\ell^{2}$. The unit vectors

$\displaystyle e_{1}$ $\displaystyle=\left(1,0,0,\ldots\right),$
$\displaystyle e_{2}$ $\displaystyle=\left(0,1,0,\ldots\right),$

satisfy $\left|e_{n}\right|=1$ and $\left|e_{m}-e_{n}\right|>1$, so any two
open balls $B_{1/2}\left(e_{n}\right)=\left\\{v\in
V:\left|v-e_{n}\right|<1/2\right\\}$ do not instersect, and their union is
contained in a bigger ball $B_{2}\left(0\right)=\left\\{v\in
V:\left|v\right|<2\right\\}$ centered at zero,

$B_{1/2}\left(e_{1}\right)\cup B_{1/2}\left(e_{2}\right)\cup\cdots\subset
B_{2}\left(0\right).$

Since the measure $\mu$ is rotation (or translation) invariant,
$\mu\left(B_{1/2}\left(e_{n}\right)\right)=\mu\left(B_{1/2}\left(e_{1}\right)\right)$
for all $n$. Thus,

$\mu\left(B_{1/2}\left(e_{1}\right)\right)+\mu\left(B_{1/2}\left(e_{1}\right)\right)+\cdots\leq\mu\left(B_{2}\left(0\right)\right),$

which is a contradiction since $\mu\left(B_{1/2}\left(e_{1}\right)\right)>0$
and $\mu\left(B_{2}\left(0\right)\right)<\infty$.

###### Remark 5

Note we said “normed linear” space. That’s so that we know what open sets and
therefore Borel sets are. We also used the norm to define the balls that
served in the contradition. How can you get around the contradiction if you
need to? Change the definition of open sets so that open balls are not
measurable. That is, use a different topology. See Chapter 9.

###  4.1 Random elements

Let $\left(\Omega,\mathcal{F},\mu\right)$ be a probability space, that is, a
measure space such that $\mu\left(\Omega\right)=1$. Here, $\mathcal{F}$ is the
system of subsets $S$ of $\Omega$ such that the measure $\mu\left(S\right)$ is
defined, and it is a $\sigma$-algebra,

$\emptyset\in\mathcal{F,\quad}A\in\mathcal{F\Longrightarrow}\Omega\mathcal{\setminus
A\in
F,\quad}A_{1},A_{2},\ldots\in\mathcal{F\Longrightarrow}{\displaystyle\bigcup\limits_{n=1}^{\infty}}A_{n}\in\mathcal{F}$

and $\Pr$ is a measure: $\mu\left(A\right)\geq 0$ defined for all
$A\in\mathcal{F}$, $\mu\left(\emptyset\right)=0$, and $\mu$ is
$\sigma$-additive:

$A_{1},A_{2},\ldots\in\mathcal{F}\text{ disjoint
}\Longrightarrow\mu\left({\displaystyle\bigcup\limits_{n=1}^{\infty}}A_{n}\right)={\displaystyle\sum\limits_{n=1}^{\infty}}\mu\left(A_{n}\right).$

We emphasize that $\sigma$_-additivity is required for the usual probability
theory to go through_.

_Random variable_ is a real-valued measurable function $X$ on
$\left(\Omega,\mathcal{F}\right)$, that is, the inverse image
$X^{-1}\left(A\right)=\left\\{\omega\in\Omega|X\left(\omega\right)\in
A\right\\}\in\mathcal{F}$ for any Borel set $A\subset\mathbb{R}$.111Unlike in
Real Analysis, non-measurable functions in probability are not uncommon. When
$f$ is not measurable, it simply means that it can distinguish subsets of
$\Omega$ which are not visible by the $\sigma$-algebra $\mathcal{F}$. For
example, if you throw a dice and cannot distinguish between $2$ and $3$ (say,
those sides are smeared), any function that depends on this distinction is not
measurable.222Borel sets are the smallest $\sigma$-algebra containing all open
sets.. The notation

$\Pr\left(X\in
A\right)=\mu\left(X^{-1}\left(A\right)\right)=\mu\left(\left\\{\omega\in\Omega|X\left(\omega\right)\in
A\right\\}\right)$

for the probability that the value of $X$ is in $A$ is used. The mean value
(if it exists) of a random variable $X$ is the abstract (Lebesgue) integral
with respect to the measure $\mu$,

$E\left(X\right)=\int_{\Omega}X\left(\omega\right)\mu\left(d\omega\right).$

To simplify notation, the measure $\mu$ in the integrals is assumed and we
write simply

$E\left(X\right)=\int_{\Omega}X\left(\omega\right)d\omega.$

Now let $H$ be a separable Hilbert space. The $\sigma$-algebra of all Borel
sets in $H$ is denoted by $\mathcal{B}\left(H\right)$. A random element $X$
with values in $H$ is a measurable function $X:\Omega\rightarrow H$, that is,
such that333The usual definition of measurable function with values in a
metric space.

$\forall B\in\mathcal{B}\left(H\right):X^{-1}\left(B\right)\in\mathcal{F}$
(26)

Given an $H$-valued random element $X$,

$\mu_{X}\left(B\right)=\mu\left(X^{-1}\left(B\right)\right)=\Pr\left(X\in
B\right)$

is a probability measure on $\mathcal{B}\left(H\right)$ (commonly called just
a probability measure on $H$), called the _distribution_ of $X$.

###  4.2 $L^{p}$ spaces

Since a continuous function of a measurable function is measurable, it follows
that $\left|X\right|$, and $\left\langle u,X\right\rangle$, for any $u\in H$,
are random variables444The latter property is called weak measurability, while
measurable in the usual sense (26) is called strongly measurable. When $H$ is
separable, weak measurability is equivalent to strong measurability by Pettis
measurability theorem [51].. For $1\leq p<\infty$, define
$L^{p}=L^{p}\left(\Omega,H\right)=L^{p}\left(\Omega,\mathcal{F},\mu,H\right)$
as the space of all $H$-valued random elements $X$ such that the _moment_

$E\left(\left|X\right|^{p}\right)=\int_{\Omega}\left|X\left(\omega\right)\right|^{p}d\omega<\infty$

As usual, we do not distinguish between elements of $L^{p}$ equal a.s. The
space $L^{p}$ equipped with the norm

$\left\|X\right\|_{p}=\left(E\left(\left|X\right|^{p}\right)\right)^{1/p}$

is a Banach space, and $L^{2}\left(\Omega,H\right)$ equipped with the inner
product

$E\left(\left\langle U,V\right\rangle\right)=\int_{\Omega}\left\langle
U\left(\omega\right),V\left(\omega\right)\right\rangle d\omega$

is a Hilbert space555The corresponding definition of $L^{\infty}$ is the space
of all random elements $X$ such that for some $c\in\mathbb{R}$,
$\left|X\right|\leq c$ a.s., but are not concerned with the case $p=\infty$
here.666In the case $H=\mathbb{R}$, these are the same spaces as the $L^{p}$
spaces of real functions of a real variable, except on an abstract set
$\Omega$, without any topology, rather than on $\mathbb{R}$. Abstract
properties of the measure and integral that do not require the topology of
$\mathbb{R}$, carry over..

In particular, given a Hilbert space $K$ and a probability measure $\mu$ on
$K$, the space

$L^{p}\left(K,\mu\right)=L^{p}\left(K,\mathcal{B}\left(K\right),\mu,\mathbb{R}\right)$

is the $L^{p}$ space of all real functions on $H$ that are measurable on $H$
with respect to the Borel sets of $H$, and such that
$\int_{H}\left|f\right|^{p}d\mu<\infty$.

If $X\in L^{2}\left(\Omega,H\right)$, then by the Cauchy inequality on
$L^{2}\left(\Omega,H\right)$,

$\left\|X\right\|_{1}=\left|E\left(\left|X\right|\right)\right|\leq\left\|X\right\|_{2}$

because

$\left\|X\right\|_{1}=E\left(\left|X\right|\right)=E\left(\left|X\right|\cdot
1\right)\leq\left\|X\right\|_{2}\left\|1\right\|_{2}=\left\|X\right\|_{2},$

so

$L^{2}\left(\Omega,H\right)\subset L^{1}\left(\Omega,H\right).$

More generally,

$p>r\geq 1\Longrightarrow L^{p}\left(\Omega,H\right)\subset
L^{r}\left(\Omega,H\right).$

###  4.3 Mean value of random element

For $X\in L^{2}\left(\Omega,H\right)$, the mean value $E\left(X\right)$
defined by777Thus,
$E\left(U\right)={\textstyle\int\nolimits_{\Omega}}Ud\omega$ is the Gelfand-
Pettis integral [51] (the weak integral), not the Bochner integral (the strong
integral).

$E\left(X\right)\in H,\quad\left\langle
v,E\left(X\right)\right\rangle=E\left(\left\langle
v,X\right\rangle\right),\quad\forall v\in H,$ (27)

exists and is unique from the Riesz representation theorem:888For every
bounded linear functional $f$ on $H$, there exists a unique $v\in H$ such that
$\left\langle u,v\right\rangle=f\left(v\right)$ $\forall v\in H$. In addition,
$\left|u\right|_{H}=\left|f\right|_{H^{\prime}}$. Since

$\left|E\left(\left\langle v,X\right\rangle\right)\right|\leq
E\left(\left|v\right|\left|X\right|\right)\leq\left|v\right|E\left(\left|X\right|\right),$

by the Cauchy inequality on $H$, the mapping $v\mapsto E\left(\left\langle
v,X\right\rangle\right)$ is a bounded linear functional on $H$, therefore
there exists unique

$u\in H\text{ such that }\left\langle v,u\right\rangle=E\left(\left\langle
v,X\right\rangle\right)\text{ for all }v\in H.$

Note that when $H=\mathbb{R}^{n}$ and $v=e_{k}$, the $k$-th coordinate vector,
then $\left\langle v,E\left(X\right)\right\rangle$ is simply entry $k$ of the
vector $E\left(X\right)$.

For _operator-valued random element_ $U\in
L^{1}\left(\Omega,\left[H\right]\right)$, the mean value $E\left(U\right)$,
defined by

$E\left(U\right)\in\left[H\right],\quad\left\langle
u,E\left(U\right)v\right\rangle=E\left(\left\langle
u,Uv\right\rangle\right)\quad\forall u,v\in H,$ (28)

exists in $\left[H\right]$ and it is unique, from Lemma 2 and the Cauchy
inequality in $H$,

$\left|E\left(\left\langle
u,Uv\right\rangle\right)\right|\leq\left|E\left(\left|u\right|\left|U\right|\left|v\right|\right)\right|=\left|u\right|\left|v\right|E\left(\left|U\right|\right).$

From standard properties of the abstract integral [36],

$\left|E\left(X\right)\right|\leq
E\left(\left|X\right|\right)=\left\|X\right\|_{1}.$

A random variable $X$ and its distribution are called _centered_ if
$E\left(X\right)=0$.

A random element $X$ is a constant if $X\left(\omega\right)$ is the same for
all $\omega\in\Omega$. A constant random element with values in $H$ can be
considered as simply a (non-random, deterministic) element of $H$, and we can
write $X\in H$. If $X$ is a constant, then

$\left\|X\right\|_{p}=\left(E\left(\left|X\right|^{p}\right)\right)^{1/p}=\left|X\right|,\quad
1\leq p<\infty.$ (29)

In particular, a constant $H$-valued random element is in all spaces $L^{p}$,
$p\geq 1$.

###  4.4 Mean of tensor product

The tensor product with a constant and the mean commute: If $X,Y\in
L^{1}\left(\Omega,H\right)$, $x,y\in H,$ then

$E\left(X\otimes y\right)=E\left(X\right)\otimes y,$ (30)

and

$E\left(x\otimes Y\right)=x\otimes E\left(Y\right)$ (31)

Indeed, if and $u,v\in H$, then

$\displaystyle\left\langle u,E\left(X\otimes y\right)v\right\rangle$
$\displaystyle=E\left(\left\langle u,\left(X\otimes
y\right)v\right\rangle\right)$ $\displaystyle=E\left(\left\langle
u,X\right\rangle\left\langle y,v\right\rangle\right)$
$\displaystyle=\left\langle u,E\left(X\right)\right\rangle\left\langle
y,v\right\rangle$ $\displaystyle=\left\langle u,\left(E\left(X\right)\otimes
y\right)v\right\rangle$

by the definition (28) of the mean of a random mapping, the definition (17) of
tensor product, the definition (27) of mean of a random vector, and again the
the definition (17) of tensor product. This shows (30). The proof of (31) is
similar. See also Lemma 2.

###  4.5 Covariance

The covariance of random elements $X$, $Y$ is the linear operator
$\operatorname*{Cov}\left(X,Y\right)$, defined by999The existence and
uniqueness of $\operatorname*{Cov}\left(X,Y\right)$ follows again from Riesz
representation theorem. See [15, Ch. 1] for more details. However we used the
Riesz representation theorem in the definition of tensor product and the mean
of the tensor product… it is exactly the same but we need it several times so
we did it just once.

$\left\langle
u,\operatorname*{Cov}\left(X,Y\right)v\right\rangle=E\left(\left\langle
u,X-E\left(X\right)\right\rangle\left\langle
v,Y-E\left(Y\right)\right\rangle\right)\quad\forall u,v\in H,$ (32)

$\operatorname*{Cov}\left(X\right)$ stands for
$\operatorname*{Cov}\left(X,X\right)$. Clearly, if
$\operatorname*{Cov}\left(X\right)$ exists, it is self-adjoint and positive
semidefinite.

###### Theorem 6

If $X,Y\in L^{2}\left(\Omega,H\right)$,
$\operatorname*{Cov}\left(X,Y\right)\in\left[H\right]$ exists and

$\displaystyle\operatorname*{Cov}\left(X,Y\right)$
$\displaystyle=E\left(\left(X-E\left(X\right)\right)\otimes\left(Y-E\left(Y\right)\right)\right)$
$\displaystyle=E\left(X\otimes Y\right)-E\left(X\right)\otimes
E\left(Y\right).$ (33)

Proof. Writing

$\left\langle u,X-E\left(X\right)\right\rangle\left\langle
v,Y-E\left(Y\right)\right\rangle=\left\langle
u,\left(X-E\left(X\right)\right)\otimes\left(Y-E\left(Y\right)\right)v\right\rangle$

we have from the definition of the mean of a linear operator (28),

$\displaystyle\left\langle
u,\operatorname*{Cov}\left(X,Y\right)v\right\rangle$
$\displaystyle=E\left(\left\langle
u,\left(X-E\left(X\right)\right)\otimes\left(Y-E\left(Y\right)\right)v\right\rangle\right)$
$\displaystyle=\left\langle
u,E\left(\left(X-E\left(X\right)\right)\otimes\left(Y-E\left(Y\right)\right)\right)v\right\rangle$

and by the Cauchy inequality and triangle inequality

$\displaystyle\left|E\left(\left\langle
u,X-E\left(X\right)\right\rangle\left\langle
v,Y-E\left(Y\right)\right\rangle\right)\right|$
$\displaystyle\leq\left|E\left(\left\langle
u,X-E\left(X\right)\right\rangle\right)\right|\left|E\left(\left\langle
v,Y-E\left(Y\right)\right\rangle\right)\right|$
$\displaystyle\leq\left|E\left(\left|u\right|\left|X-E\left(X\right)\right|\right)\right|\left|E\left(\left|v\right|\left|Y-E\left(Y\right)\right|\right)\right|$
$\displaystyle\leq\left|u\right|\left\|X-E\left(X\right)\right\|_{2}\left|v\right|\left\|Y-E\left(Y\right)\right\|_{2}$
$\displaystyle\leq
4\left|u\right|\left|v\right|\left\|X\right\|\left\|Y\right\|_{2}$

for any $u,v\in H$. Therefore, $\operatorname*{Cov}\left(X,Y\right)$ exists
and $\operatorname*{Cov}\left(X,Y\right)$ $\in L\left(H\right)$ by Lemma 2
(corollary of Riesz representation theorem). To prove (33),

$\displaystyle\operatorname*{Cov}\left(X,Y\right)$
$\displaystyle=E\left(\left(X-E\left(X\right)\right)\otimes\left(Y-E\left(Y\right)\right)\right)$
$\displaystyle=E\left(X\otimes Y-X\otimes
E\left(Y\right)-E\left(X\right)\otimes Y+E\left(X\right)\otimes
E\left(Y\right)\right)$ $\displaystyle=E\left(X\otimes
Y\right)-E\left(X\right)\otimes E\left(Y\right)-E\left(X\right)\otimes
E\left(Y\right)+E\left(X\right)\otimes E\left(Y\right)$
$\displaystyle=E\left(X\otimes Y\right)-E\left(X\right)\otimes
E\left(Y\right),$

using (30) and (31).

###### Theorem 7

Suppose that $X\in L^{1}\left(\Omega,H\right)$. Then $X\in
L^{2}\left(\Omega,H\right)$ if and only if $\operatorname*{Cov}\left(X\right)$
exists and $\operatorname*{Tr}\operatorname*{Cov}\left(X\right)<\infty$. In
fact,

$\left\|X-E\left(X\right)\right\|_{L^{2}\left(\Omega,H\right)}^{2}=\operatorname*{Tr}\operatorname*{Cov}\left(X\right).$

Proof. We follow [33, proof of Theorem 2.1]. Without loss of generality,
assume $E\left(X\right)=0$. Suppose for the moment that
$C=\operatorname*{Cov}\left(X\right)$ exists. Then

$\left\langle u,Cv\right\rangle=E\left(\left\langle
X,u\right\rangle\left\langle X,v\right\rangle\right)\quad\forall u,v\in H.$
(34)

Choose a complete orthonormal set $\left\\{e_{n}\right\\}$. For a fixed
$\omega$, decompose $X\left(\omega\right)$ in the abstract Fourier series

$X\left(\omega\right)=\sum_{n=1}^{\infty}\,\left\langle
X\left(\omega\right),e_{n}\right\rangle e_{n}$

convergent in the norm of $H$. By Parseval equality,

$\left|X\left(\omega\right)\right|^{2}=\sum_{n=1}^{\infty}\left|\,\left\langle
X\left(\omega\right),e_{n}\right\rangle\right|^{2}.$

Because this is a series of nonegative random variables (=real-valued
measurable functions of $\omega$), by the monotone convergence theorem,

$\int_{\Omega}\left|X\left(\omega\right)\right|^{2}d\omega=\sum_{n=1}^{\infty}\int_{\Omega}\left|\,\left\langle
X\left(\omega\right),e_{n}\right\rangle\right|^{2}d\omega=\sum_{n=1}^{\infty}\left\langle
Ce_{n},e_{n}\right\rangle=\operatorname*{Tr}C$ (35)

because $\left\langle Ce_{n},e_{n}\right\rangle=\lambda_{n}$.

Let $\operatorname*{Tr}C<\infty$. Then $X\in L^{2}\left(\Omega,H\right)$ from
(35).

Let $X\in L^{2}\left(\Omega,H\right)$. By Theorem 6, $C$ exists, and
$\operatorname*{Tr}C<\infty$ from (35).

###### Remark 8

When $H=\mathbb{R}^{n}$ and $u=e_{k}$, $v=e_{\ell}$, and the linear operator
$\operatorname*{Cov}\left(X,Y\right)$ is identified with its matrix, then
$\left\langle u,\operatorname*{Cov}\left(X,Y\right)v\right\rangle$ is simply
the entry $k$, $\ell$ of the covariance matrix. In the case when $n=1$,
$\operatorname*{Cov}\left(X\right)$ is also called the variance of the random
variable $X$ and denoted by $\operatorname*{Var}\left(X\right)$.

###  4.6 Convergence of random elements

There are many types of convergence of random variables. With many types of
convergence in infinitely dimensional spaces, there are many, many more. But
here just replace the absolute value by the norm in $H$.

_Convergence in norm almost surely (a.s.)_ is defined as

$\displaystyle X_{k}\rightarrow X\text{ in }H\text{ a.s.}$
$\displaystyle\Leftrightarrow\lim_{k\rightarrow\infty}\left|X_{k}\left(\omega\right)-X\left(\omega\right)\right|=0\quad\forall\omega\in\Omega\setminus\Theta,\text{\quad}\mu\left(\Theta\right)=0$
$\displaystyle\Leftrightarrow\Pr\left(\lim_{k\rightarrow\infty}\left|X_{k}\left(\omega\right)-X\left(\omega\right)\right|=0\right)=1$

Since $L^{p}\left(\Omega,H\right)$ is a normed space, we have the usual notion
of _convergence in the_ $L^{p}$_norm_ ,

$X_{k}\rightarrow X\text{ in }L^{p}\Longleftrightarrow X\in
L^{p}\left(\Omega,H\right)\wedge\left\|X_{k}-X\right\|_{p}\rightarrow 0.$

_Convergence in distribution_ of random elements, denoted by
$X_{k}\Longrightarrow X$, is defined as weak convergence of their
distributions,

$\lim_{k\rightarrow\infty}{\displaystyle\int\limits_{H}}\phi
d\mu_{X_{k}}={\displaystyle\int\limits_{H}}\phi d\mu_{X}\quad\forall\phi\in
C_{b}\left(H\right),$ (36)

where $C_{b}\left(H\right)$ is the space of all continuous bounded functions
on $H$.

_Convergence in probability_ of random elements is defined as

$\lim_{k\rightarrow\infty}\Pr\left(\left|X_{k}-X\right|>\varepsilon\right)=0\quad\forall\varepsilon>0.$

Convergence a.s. implies convergence in probability, which implies convergence
in distribution. Convergence in distribution to a constant implies convergence
in probability [26, Lemma 4.7].101010In fact, in any metric space.

Convergence in $L^{p}$ implies convergence a.s. and in probability, but the
converse is in general not true. However, we have the converse under an
additional boundedness assumption. This result is a strengthening of the
Lebesgue dominated convergence theorem.

######  Lemma 9 (Uniform integrability)

If $\left\\{X_{k}\right\\}$ is a bounded sequence in
$L^{p}\left(\Omega,H\right)$ and $X_{k}\rightarrow X$ in probability, then
$\left\|X_{k}-X\right\|_{q}\rightarrow 0$ for all $1\leq q<p$.

Proof. It is well known that for random variables, the lemma follows from
uniform integrability [7, p. 338]. For random elements $X_{k}$, $X$ on $H$,
consider the random variables $U_{k}=\left|X_{k}-X\right|$.

######  Lemma 10 (Slutsky’s theorem, [11, page 254, Corollary 2])

If $\left\\{X_{k}\right\\}$, $\left\\{Y_{k}\right\\}$are random variables,
$X_{k}\Longrightarrow X$, $Y_{k}\Longrightarrow Y$, and $Y$ is constant, then
$X_{k}Y_{k}\Longrightarrow XY$ and $X_{k}+Y_{k}\Longrightarrow X+Y$.

######  Lemma 11 (Slutsky’s theorem, [25, Theorem 18.8, page 161])

If $\left\\{X_{k}\right\\}$, $\left\\{Y_{k}\right\\}$are random elements in
$\mathbb{R}^{m}$, $X_{k}\Longrightarrow X$, $Y_{k}\Longrightarrow Y$, and $Y$
is constant, $X_{k}+Y_{k}\Longrightarrow X+Y$.

######  Lemma 12 (Continuous mapping theorem, [57, Theorem 2.3])

If $X_{n}$ are random elements with values on a metric space $M$, and
$X_{k},X$ random elements with values in $M$, $f$ function from $M$ to another
metric space, the probability that $X$ attains value where $f$ is
discontinuous is zero, then $X_{k}\rightarrow X$ implies
$f\left(X_{k}\right)\rightarrow f\left(X\right)$. This is true for convergence
in distribution, convergence in probability, and convergence almost surely.

###  4.7 Karhunen-Loève explansion and random orthonormal series

Suppose $U\in L^{2}\left(\Omega,H\right)$ with the covariance operator
$C=\operatorname*{Cov}\left(X\right)$. Since $C$ is compact self-adjoint
operator, there exists a complete orthonormal system $\left\\{u_{n}\right\\}$
of eigenvectors of $U$:

$Cu_{n}=\lambda_{n}u_{n}.$

Assume for the moment that $E\left(U\right)=0$ and for a fixed
$\omega\in\Omega$, expand $U$ in an abstract Fourier series using the
orthonormal system $\left\\{u_{n}\right\\}$:

$U\left(\omega\right)={\textstyle\sum\nolimits_{n=1}^{\infty}}\theta_{n}\left(\omega\right)u_{n},\text{
where }\theta_{n}\left(\omega\right)=\left\langle
U\left(\omega\right),u_{n}\right\rangle.$ (37)

So the sum (37) converges in $H$ a.s. Compute:

$\displaystyle E\left(\theta_{n}\right)$ $\displaystyle=E\left(\left\langle
U,u_{m}\right\rangle\right)=\left\langle E\left(U\right),u_{m}\right\rangle=0$
$\displaystyle E\left(\theta_{m}\theta_{n}\right)$
$\displaystyle=E\left(\left\langle U,u_{m}\right\rangle\left\langle
U,u_{n}\right\rangle\right)=\left\langle
u_{m},Cu_{n}\right\rangle=\left\\{\begin{array}[c]{c}\lambda_{n}\text{ if
}m=n\\\ 0\text{ if }m\neq 0\end{array}\right.$

Back to the of general $E\left(U\right)$ and substituting
$\theta_{n}=\lambda_{n}^{1/2}\xi_{n}$ (and choosing suitable $\xi_{n}$ if
$\lambda_{n}=0$) we have $E\left(\xi_{m}\xi_{n}\right)=\delta_{mn}$ and (37)
becomes the _Karhunen-Loève expansion,_

$U=E\left(U\right)+{\textstyle\sum\nolimits_{n=1}^{\infty}}\lambda_{n}^{1/2}\xi_{n}u_{n}\text{,\quad}E\left(\xi_{n}\right)=0.$
(38)

convergent a.s. in $H$ and _double-orthogonal_ :

$\left\langle
u_{m},u_{n}\right\rangle_{H}=\delta_{mn},\quad\left\langle\xi_{m},\xi_{n}\right\rangle_{L^{2}\left(\Omega\right)}=\delta_{mn}$

because
$\left\langle\xi_{m},\xi_{n}\right\rangle_{L^{2}\left(\Omega\right)}=E\left(\xi_{m}\xi_{n}\right)$.
See [42] for further details.

Karhunen-Loève explansion is used in practice in many ways. Here are some:

  1. 1.

_Data analysis_ : Estimate the covariance from data (realizations of $U$) by
sample covariance, and compute the eigenvalues and eigenvectors to represent
the random element $U$. This is known as _Principal Component Analysis (PCA),
Karhunen-Loève transform (KLT), or Proper Orthogonal Decomposition (POD)_. The
components with several largest eigenvalues $\lambda_{n}$ are responsible for
most of the variance in the random element $U$. (The practical computation is
done by SVD of the data minus sample mean, which is much less expensive and
less prone to numerical errors than computing the sample covariance first.)

  2. 2.

Prescribe the covariance so that it has suitable eigenvectors, and use the
Karhunen-Loève expansion to _generate the random element_ $U$ (rather, the
first few terms to generate a version of $U$ in finite dimension). This is the
method of random Fourier series in Section 2.5, where the covariance was
chosen so that it has trigonometric functions as its eigenvectors.

  3. 3.

Use the first few terms Karhunen-Loève expansion of the to build random
coefficients and assumed form of the solution (called trial space in
variational methods) of _stochastic partial differential equations_. The
solution is then found numerically as a _deterministic function of a small
number random variables_ $\xi_{1},\ldots,\xi_{n}$. This is the foundation of
methods such as _stochastic Galerkin_ [2], _stochastic collocation_ [21] and
_polynomial chaos_ [62].

##  5 Inequalities

###  5.1 Cauchy-Schwarz inequality

In the Hilbert space $H$, the Cauchy-Schwarz inequality reads

$\left|\left\langle X,Y\right\rangle\right|\leq\left|X\right|\left|Y\right|.$
(39)

For random variables $X$ and $Y$, the Cauchy-Schwarz inequality in
$L^{2}\left(\Omega,\mathbb{R}\right)$,

$\left|\int_{\Omega}XYd\omega\right|^{2}\leq\int_{\Omega}\left|X\right|^{2}d\omega\int_{\Omega}\left|Y\right|^{2}d\omega$

becomes

$\left|E\left(XY\right)\right|\leq\left\|X\right\|_{2}\left\|Y\right\|_{2}.$
(40)

This is a special case (with $H=\mathbb{R}$) of Cauchy-Schwarz inequality in
the Hilbert space $L^{2}\left(\Omega,H\right)$ below.

###### Lemma 13

If $X,Y\in L^{2}\left(\Omega,H\right)$, then

$\displaystyle\left|E\left(\left\langle X,Y\right\rangle\right)\right|$
$\displaystyle\leq\left\|X\right\|_{2}\left\|Y\right\|_{2},$ (41)
$\displaystyle\left\|X\otimes Y\right\|_{1}$
$\displaystyle\leq\left\|X\right\|_{2}\left\|Y\right\|_{2},$ (42)

and

$\left\|X\right\|_{1}\leq\left\|X\right\|_{2}.$ (43)

Proof. Inequality (41) is Cauchy-Schwarz inequality in the Hilbert space
$L^{2}\left(\Omega,H\right)$. We have used it few times already, this is for
completeness only. Inequality (42) follows from (18),

$\left|E\left(X\otimes Y\right)\right|\leq E\left(\left|X\otimes
Y\right|\right)=E\left(\left|X\right|\left|Y\right|\right)\leq\left\|X\right\|_{2},$

using Cauchy-Schwarz inequality for the random variables $\left|X\right|$ and
$\left|Y\right|$. From (41),

$E\left(\left|X\right|\right)=E\left(1\left|X\right|\right)\leq
E\left(1\right)^{1/2}E\left(\left|X\right|^{2}\right)^{1/2}=\left\|X\right\|_{2},$

which yields (43).

###### Lemma 14

For any $U$, $V\in L^{p}\left(\Omega,H\right)$,

$\displaystyle\left\|U\otimes V\right\|_{p}$
$\displaystyle=\left\|\left|U\right|\left|V\right|\right\|_{p}=\left(E\left(\left|U\right|^{p}\left|V\right|^{p}\right)\right)^{1/p}$
(44) $\displaystyle\leq
E\left(\left|U\right|^{2p}\right)^{1/2p}E\left(\left|U\right|^{2p}\right)^{1/2p}=\left\|U\right\|_{2p}\left\|U\right\|_{2p}.$

Proof. The proof follows from Cauchy-Schwarz inequality for the random
variables $\left|U\right|^{p}$ and $\left|V\right|^{p}$.

###  5.2 Hölder’s inequality

Let $1<p,q<\infty$, and $1/p+1/q=1$. For random variables $X$ and $Y$,
Hölder’s inequality for integrals,

$\left|\int_{\Omega}XYd\omega\right|\leq\left(\int_{\Omega}\left|X\right|^{p}d\omega\right)^{1/p}\left(\int_{\Omega}\left|Y\right|^{p}d\omega\right)^{1/q},$

becomes

$\left|E\left(XY\right)\right|\leq
E\left(\left|X\right|^{p}\right)^{1/p}E\left(\left|Y\right|^{q}\right)^{1/q}.$
(45)

Estimates for $H$-valued random elements follow.

###### Lemma 15

If $1<p,q<\infty$, $1/p+1/q=1$, and $X\in L^{p}\left(\Omega,H\right)$, $Y\in
L^{q}\left(\Omega,H\right)$, then

$\displaystyle E\left(\left|X\right|\left|Y\right|\right)$
$\displaystyle\leq\left\|X\right\|_{p}\left\|Y\right\|_{q},$ (46)
$\displaystyle\left\|\left\langle X,Y\right\rangle\right\|_{1}$
$\displaystyle\leq\left\|X\right\|_{p}\left\|Y\right\|_{q},$ (47)
$\displaystyle\left\|X\otimes Y\right\|_{1}$
$\displaystyle\leq\left\|X\right\|_{p}\left\|Y\right\|_{q},$ (48)

Proof. Inequality (46) is Hölder’s inequality (45) applied to the random
variables $\left|X\right|$ and $\left|Y\right|$. (47) follows from (46) by
Cauchy-Schwarz inequality (39) in $H$,

$E\left(\left|\left\langle X,Y\right\rangle\right|\right)\leq
E\left(\left|X\right|\left|Y\right|\right).$

Finally, (18),

$E\left(\left|X\otimes Y\right|\right)\leq
E\left(\left|X\right|\left|Y\right|\right)$

and (46) give (48).

Lemma 15 provides bounds on the $L^{1}\left(\Omega,H\right)$ norm. Bounds on
higher norms are similar.

###### Lemma 16

If $1<p,q<\infty$, $1/p+1/q=1$, $s\geq 1$, and $X\in
L^{ps}\left(\Omega,H\right)$, $Y\in L^{qs}\left(\Omega,H\right)$, then

$\displaystyle\left\|\left|X\right|\left|Y\right|\right\|_{s}$
$\displaystyle\leq\left\|X\right\|_{ps}\left\|Y\right\|_{qs},$ (49)
$\displaystyle\left\|\left\langle X,Y\right\rangle\right\|_{s}$
$\displaystyle\leq\left\|X\right\|_{ps}\left\|Y\right\|_{qs},$ (50)
$\displaystyle\left\|X\otimes Y\right\|_{s}$
$\displaystyle\leq\left\|X\right\|_{ps}\left\|Y\right\|_{qs}.$ (51)

Proof. By Hölder inequality for the random variables $\left|X\right|^{s}$ and
$\left|Y\right|^{s}$,

$E\left(\left|X\right|^{s}\left|Y\right|^{s}\right)\leq
E\left(\left|X\right|^{sp}\right)^{1/p}E\left(\left|Y\right|^{sq}\right)^{1/q}.$

The proofs of (50) and (51) carry over from the proof of Lemma 15.

We now generalize (43) to arbitrary $L^{p}$ spaces.

###### Lemma 17

If $1\leq s<t$ and $X\in L^{t}\left(\Omega,H\right)$, then

$\left\|X\right\|_{s}\leq\left\|X\right\|_{t}.$ (52)

Proof. Choose $p>1$ so that $t=ps$, apply Hölder inequality for the random
variables $\left|X\right|^{s}$ and $1$,

$E\left(\left|X\right|^{s}1^{s}\right)\leq
E\left(\left|X\right|^{sp}\right)^{1/p}E\left(1^{sq}\right)^{1/q}$

and note that $\left\|1\right\|_{qs}=1$ since $1$ is constant.

###  5.3 Chebychev’s inequality

The generalized Chebychev’s inequality for a nonnegative random variable $U$
states that if $f$ is a measurable, nonnegative, nondecreasing function on
$\left(0,+\infty\right)$, then

$\Pr\left(f\left(U\right)>t\right)\leq\frac{1}{f\left(t\right)}E\left(f\left(U\right)\right).$

For $X\in L^{p}$, the choices $f\left(t\right)=t$ and $U=\left|X\right|^{p}$
yield

$\Pr\left(\left|X\right|^{p}>t\right)\leq\frac{1}{t}E\left(\left|X\right|^{p}\right),\quad\forall
t>0,$

or, equivalently, by the substitution $t=\theta^{p}$,

$\Pr\left(\left|X\right|>\theta\right)\leq\left(\frac{\left\|X\right\|_{p}}{\theta}\right)^{p},\quad\forall\theta>0,\quad
p\geq 1.$ (53)

###  5.4 Marcinkiewicz-Zygmund inequality

######  Theorem 18 ([46], [11, p. 367])

If $1\leq p<+\infty$ and $X_{k}$, $k=1,\ldots,n$, are independent random
variables such that $E\left(X_{k}\right)=0$ and
$E\left(\left|X_{k}\right|^{p}\right)<+\infty$, then

$A_{p}E\left(\left(\sum_{k=1}^{n}\left|X_{k}\right|^{2}\right)^{p/2}\right)\leq
E\left(\left|\sum_{k=1}^{n}X_{k}\right|^{p}\right)\leq
B_{p}E\left(\left(\sum_{k=1}^{n}\left|X_{k}\right|^{2}\right)^{p/2}\right)$
(54)

where $A_{p}$ and $B_{p}$ are positive constants, which depend only on $p$.

Marcinkiewicz-Zygmund inequality in terms of norms becomes

$A_{p}\left\|\left(\sum_{k=1}^{n}\left|X_{k}\right|^{2}\right)^{1/2}\right\|_{p}\leq\left\|\sum_{k=1}^{n}X_{k}\right\|_{p}\leq
B_{p}\left\|\left(\sum_{k=1}^{n}\left|X_{k}\right|^{2}\right)^{1/2}\right\|_{p},$

still for random variables only. For $p=2$, the inequality in a Hilbert space
holds with $A_{2}=B_{2}=1,$

$E\left(\left|\sum_{k=1}^{n}X_{k}\right|^{2}\right)=E\left(\sum_{i=1}^{n}\sum_{j=1}^{n}\left\langle
X_{i},X_{j}\right\rangle\right)=E\left(\sum_{i=1}^{n}\left|X_{i}\right|^{2}\right),$

since $E\left(X_{k}\right)=0$, which is the well-known property

$\operatorname{Var}\left(\sum_{k=1}^{n}X_{k}\right)=\sum_{i=1}^{n}\operatorname{Var}\left(X_{i}\right).$

However, an extension from $\mathbb{R}$ to Banach spaces relies on the theory
of geometry of Banach spaces. The Marcinkiewicz-Zygmund inequality is not
valid in Banach spaces in general, and in fact it defines a special type of
Banach spaces. We will concentrate on the upper bound, cf. Corollary 27 below.

###### Definition 19

Let $1\leq p\leq 2$. A separable Banach space $U$ is said to be of Rademacher
type $p$ if there exists $C$ such that for every $n$ and for all
$x_{1},\ldots,x_{n}\in U$ and for every sequence $r_{i}$ of real independent
random variables with $\Pr\left(r_{i}=\pm 1/2\right)=1/2$,

$E\left(\left|\sum_{i=1}^{n}r_{i}x_{i}\right|\right)\leq
C\left(\sum_{i=1}^{n}\left|x_{i}\right|^{p}\right)^{1/p}.$ (55)

######  Remark 20 ([1, p. 158])

Rademacher type is defined for $p\leq 2$ only because the only Banach space
that satifies (55) with $p>2$ is the trivial space $\left\\{0\right\\}$.

###### Example 21

Every separable Banach space is of type $1$. By the triangle inequality,

$E\left(\left|\sum_{i=1}^{n}r_{i}x_{i}\right|\right)\leq
E\left(\sum_{i=1}^{n}\left|r_{i}\right|\left|x_{i}\right|\right)=\frac{1}{2}\sum_{i=1}^{n}\left|x_{i}\right|.$

###### Example 22

The space $l^{1}$ is not of type $p$ for any $p>1$. Consider
$x_{i}=\left(0,\ldots,0,1,0,\ldots\right)$ with the $1$ in the $i$-the place.
Then,

$E\left(\left|\sum_{i=1}^{n}r_{i}x_{i}\right|\right)=E\left(\sum_{i=1}^{n}\left|r_{i}\right|\right)=\frac{n}{2},\text{but
}\left(\sum_{i=1}^{n}\left|x_{i}\right|^{p}\right)^{1/p}=n^{1/p}.$

###### Example 23

The space $\operatorname*{Tr}H$ of trace class operators on a Hilbert space
$H$ is is not of type $p$ for any $p>1$, because it contains a copy of
$l^{1}$, namely, diagonal matrices.

######  Example 24 ([1, p. 159])

A separable Hilbert space $H$ is of Rademacher type 2. Let
$x_{1},\ldots,x_{n}\in H$. Then

$\displaystyle E\left(\left|\sum_{i=1}^{n}r_{i}x_{i}\right|\right)^{2}$
$\displaystyle\leq
E\left(\left|\sum_{i=1}^{n}r_{i}x_{i}\right|^{2}\right)=E\left(\sum_{i=1}^{n}\sum_{j=1}^{n}r_{i}r_{j}\left(x_{i},x_{j}\right)\right)$
$\displaystyle=E\left(\sum_{i=1}^{n}r_{i}^{2}\left|x_{i}\right|^{2}\right)=\frac{1}{4}\left(\sum_{i=1}^{n}\left|x_{i}\right|^{2}\right),$

by the independence of $r_{i}$.

###### Remark 25

It follows that for any $\sigma$-finite measure, $L^{2}$ is type 2 since it is
a separable Hilbert space. In addition, $L^{p}$, $p>2$, is also type 2 [1, p.
160], and $L^{p}$, $1<p<2$ is type $p$ ([1, Exercise 1, p. 202], [23]).

######  Proposition 26 ([61, page 120, Proposition 2.1], [1, Theorem 7.2 (2)]
for $q=1$)

Let $1\leq p\leq 2$ and $q\geq 1$. Banach space $U$ is of Rademacher type $p$
if and only if there exists constant $C$ such that for every $n$ and for any
sequence $X_{i}$ of independent random elements in $U$ with
$E\left(X_{i}\right)=0,$

$E\left(\left|\sum_{i=1}^{n}X_{i}\right|^{q}\right)\leq
CE\left(\sum_{i=1}^{n}\left|X_{i}\right|^{p}\right)^{q/p}$ (56)

###### Corollary 27

The upper bound in Marcinkiewicz-Zygmund inequality (54) holds in a separable
Banach space $U$ if and only if $U$ is of Rademacher type $2$.

######  Corollary 28 (Marcinkiewicz-Zygmund inequality in Hilbert space)

If $1\leq q<+\infty$ and $X_{k}$, $k=1,\ldots,n$, are independent random
elements in a separable Hilbert space $H$ such that $E\left(X_{k}\right)=0$
and $E\left(\left|X_{k}\right|^{q}\right)<+\infty$, then

$E\left(\left|\sum_{i=1}^{n}X_{i}\right|^{q}\right)\leq
B_{p}E\left(\sum_{i=1}^{n}\left|x_{i}\right|^{2}\right)^{q/2},$ (57)

where $B_{p}$ depends on $p$ only.

Proof. Hilbert space is of Rademacher type 2, so it is enough to set $p=2$ in
(56). Since every separable Hilbert space $H$ is isometric to $l^{2}$, the
constant $B_{p}$ is determined on $l^{2}$ and it does not depend on the
particular space $H$.

###### Remark 29

Strangely, even if Rademacher type is a property of the Banach space, and so
one would expect a formulation as an inequality that does not involve
probability, no such characterization seems to be known. (The probabilistic
formulation does rely on the norm only, but in a complicated manner: the norm
defines Borel measures and thus random variables.) However, Banach spaces of
Rademacher of type $p$ can be characterized as not containing subspaces
isomorphic to $l^{q}$, $q>p$, in a (rather complicated) approximate sense
[40].

##  6 Properties of covariance

###  6.1 Independent random elements

We now generalize the well-known property that indepent random variables are
uncorrelated. Note that the variables need to be in $L^{2}$ to guarantee that
the correlation is defined.

###### Lemma 30

If $X,$ $Y\in L^{2}\left(\Omega,H\right)$ are independent, then
$\operatorname*{Cov}\left(X,Y\right)=0.$

Proof. Adding constants to $X$ and $Y$ does not change independence, so
without loss of generality assume that $E\left(X\right)=E\left(Y\right)=0$.
Let $u,v\in H$. From the Cauchy inequality in $H$, $\left|\left\langle
u,X\left(\omega\right)\right\rangle\right|^{2}\leq\left|u\right|\left|X\left(\omega\right)\right|$,
it follows that the random variable $\left\langle u,X\right\rangle\in
L^{2}\left(\Omega,\mathbb{R}\right)$. Similarly, $\left\langle
u,X\right\rangle\in L^{2}\left(\Omega,\mathbb{R}\right)$. Since $\left\langle
v,X\right\rangle$ and $\left\langle v,Y\right\rangle$ are independent,

$E\left(\left\langle u,X\right\rangle\left\langle
v,Y\right\rangle\right)=E\left(\left\langle
u,X\right\rangle\right)E\left(\left\langle
v,Y\right\rangle\right)=\left\langle
u,E\left(X\right)\right\rangle\left\langle v,E\left(Y\right)\right\rangle=0,$

using the definition of the mean (27).

###  6.2 Bounds on covariance

The following estimate generalizes a well-known inequality for random vectors
in $\mathbb{R}^{n}$. Taking advantage or tensor products and properties of the
norm, we can carry over a straightforward proof for $\mathbb{R}^{n}$.

###### Lemma 31

If $X$, $Y\in L^{2}\left(\Omega,H\right)$, then

$\left|\operatorname*{Cov}\left(X,Y\right)\right|\leq
E\left(\left|X\right|\left|Y\right|\right)+\left|E\left(X\right)\right|\left|E\left(Y\right)\right|.$

In particular,

$\left|\operatorname*{Cov}\left(X,Y\right)\right|\leq
2\left\|X\right\|_{2}\left\|Y\right\|_{2}.$ (58)

and, if, in addition, $E\left(X\right)=E\left(Y\right)=0$, then

$\left|\operatorname*{Cov}\left(X,Y\right)\right|\leq\left\|X\right\|_{2}\left\|Y\right\|_{2}.$
(59)

Proof. From (33), the triangle inequality, the property of the integral
$\left|E\left(U\right)\right|\leq E\left(\left|U\right|\right)$, the equality
$\left|x\otimes y\right|=\left|x\right|\left|y\right|$, and Cauchy-Schwarz
inequality,

$\displaystyle\left|\operatorname*{Cov}\left(X,Y\right)\right|$
$\displaystyle=\left|E\left(X\otimes Y\right)-E\left(X\right)\otimes
E\left(Y\right)\right|$ $\displaystyle\leq\left|E\left(X\otimes
Y\right)\right|+\left|E\left(X\right)\otimes E\left(Y\right)\right|$
$\displaystyle\leq E\left(\left|X\otimes
Y\right|\right)+\left|E\left(X\right)\otimes E\left(Y\right)\right|$
$\displaystyle\leq
E\left(\left|X\right|\left|Y\right|\right)+\left|E\left(X\right)\right|\left|E\left(Y\right)\right|.$

The rest follows from Cauchy-Schwarz inequality,
$E\left(\left|X\right|\left|Y\right|\right)\leq\left\|X\right\|_{2}\left\|Y\right\|_{2}$,
$\left|E\left(X\right)\right|\leq
E\left(\left|X\right|1\right)\leq\left\|X\right\|_{2}$.

Note that the same proof, using Hölder’s inequality instead of Cauchy-Schwarz
inequality, yields

$\left|\operatorname*{Cov}\left(X,Y\right)\right|\leq
2\left\|X\right\|_{p}\left\|Y\right\|_{q},\quad
p,q>1,\quad\frac{1}{p}+\frac{1}{q}=1.$ (60)

Here is continuity of covariance.

###### Lemma 32

If $X$, $Y\in L^{2}\left(\Omega,H\right)$, then

$\left|\operatorname*{Cov}(X,X)-\operatorname*{Cov}(Y,Y)\right|\leq
2\left\|X-Y\right\|_{2}\left(\left\|Y\right\|_{2}+\left\|X\right\|_{2}\right).$
(61)

Proof. Since $\operatorname*{Cov}$ is bilinear,

$\operatorname*{Cov}(X,X)-\operatorname*{Cov}(Y,Y)=\operatorname*{Cov}(X-Y,Y)+\operatorname*{Cov}(X,X-Y),$

which gives

$\left|\operatorname*{Cov}(X,X)-\operatorname*{Cov}(Y,Y)\right|\leq
2\left\|X-Y\right\|_{2}\left\|Y\right\|_{2}+2\left\|X\right\|_{2}\left\|X-Y\right\|_{2}$

from (58).

###  6.3 Sample mean and covariance

Given $X_{k}\in H$, $k=1,\ldots,n$, the sample mean is defined by

$E_{n}\left(X_{k}\right)=\frac{1}{n}\sum_{k=1}^{n}X_{k}.$ (62)

Given also $Y_{k}$, $k=1,\ldots,n$, the sample covariance
$C_{n}\left(X_{k},Y_{k}\right)$ is defined as

$C_{n}(X_{k},Y_{k})=E_{n}\left(X_{k}-E_{n}\left(X_{k}\right)\right)\otimes\left(Y_{k}-E_{n}\left(Y_{k}\right)\right),$

Again, we write $C_{n}\left(X_{k}\right)$ for $C_{n}\left(X_{k},X_{k}\right)$.
We will use the sample mean notation also with more general terms, for example

$E_{n}\left(\left|X_{k}\right|^{2}\right)=\frac{1}{n}\sum_{k=1}^{n}\left|X_{k}\right|^{2}.$

We first estimate sample covariance pointwise.

###### Lemma 33

If $X_{k},Y_{k}\in H$, $k=1,\ldots,n$, then

$\displaystyle C_{n}(X_{k},Y_{k})$ $\displaystyle=E_{n}\left(X_{k}\otimes
Y_{k}\right)-E_{n}\left(X_{k}\right)\otimes E_{n}\left(Y_{k}\right)$ (63)
$\displaystyle\left|C_{n}\left(X_{k},Y_{k}\right)\right|$ $\displaystyle\leq
2E_{n}\left(\left|X_{k}\right|^{2}\right)^{1/2}E_{n}\left(\left|Y_{k}\right|^{2}\right)^{1/2},$
(64) $\displaystyle\left|C_{n}(X_{k})-C_{n}(Y_{k})\right|$ $\displaystyle\leq
2E_{n}\left(\left|X_{k}-Y_{k}\right|^{2}\right)^{1/2}\left(E_{n}\left(\left|X_{k}\right|^{2}\right)^{1/2}+E_{n}\left(\left|Y_{k}\right|^{2}\right)^{1/2}\right).$
(65)

Proof. These are same as the properties of covariance (33), (58), (61), and
the proofs are exactly the same. Alternatively, one can consider random
variables $X$ and $Y$ that attain the values of $X_{k}$, $Y_{k}$,
$k=1,\ldots,n$, respectively, with equal probability, and apply (33), (58),
and (61).

If $X_{k}$ and $Y_{k}$ are random variables, then the sample mean is
$H$-valued random element, while the sample covariance is
$\left[H\right]$-valued random element. We estimate these random elements in
$L^{p}$ norms.

###### Lemma 34

For all $p\geq 1$, if $X_{k}\in L^{p}\left(\Omega,H\right)$, $k=1,\ldots,n$,
are identically distributed and $Y_{k}\in L^{p}\left(\Omega,H\right)$,
$k=1,\ldots,n$, are identically distributed, then

$\displaystyle\left\|E_{n}\left(X_{k}\right)\right\|_{p}$
$\displaystyle\leq\left\|X_{1}\right\|_{p},$ (66)
$\displaystyle\left\|C_{n}\left(X_{k},Y_{k}\right)\right\|_{p}$
$\displaystyle\leq 2\left\|X_{1}\right\|_{2p}\left\|Y_{1}\right\|_{2p}.$ (67)

Proof. Inequality (66) follows immediately from the triangle inequality,

$\left\|E_{n}\left(X_{k}\right)\right\|_{p}=\left\|\frac{1}{n}\sum_{k=1}^{n}X_{k}\right\|_{p}\leq\frac{1}{n}\sum_{k=1}^{n}\left\|X_{k}\right\|_{p}=\left\|X_{k}\right\|_{p}.$
(68)

Then, by the same argument as in (68) and from (44),

$\left\|E_{n}\left(X_{k}\otimes
Y_{k}\right)\right\|_{p}=\left\|\frac{1}{n}\sum_{k=1}^{n}X_{k}\otimes
Y_{k}\right\|_{p}\leq\left\|X_{k}\otimes
Y_{k}\right\|_{p}\leq\left\|X_{k}\right\|_{2p}\left\|Y_{k}\right\|_{2p}$

and by using (44) again and then (68),

$\left\|E_{n}\left(X_{k}\right)\otimes
E_{n}\left(Y_{k}\right)\right\|_{p}\leq\left\|E\left(X_{1}\right)\right\|_{2p}\left\|E\left(Y_{1}\right)\right\|_{2p}\leq\left\|X_{1}\right\|_{2p}\left\|Y_{1}\right\|_{2p},$

which gives

$\displaystyle\left\|C_{n}\left(X_{k},Y_{k}\right)\right\|_{p}$
$\displaystyle\leq\left\|E_{n}\left(X_{k}\otimes
Y_{k}\right)\right\|_{p}+\left\|E_{n}\left(X_{k}\right)\otimes
E_{n}\left(Y_{k}\right)\right\|_{p}$
$\displaystyle\leq\left\|X_{1}\right\|_{2p}\left\|Y_{1}\right\|_{2p}+\left\|X_{1}\right\|_{2p}\left\|Y_{1}\right\|_{2p},$

and completes the proof of (67).

###### Lemma 35

If $X_{k}\in L^{4}\left(\Omega,H\right)$, $k=1,\ldots,n$, are identically
distributed and $Y_{k}\in L^{4}\left(\Omega,H\right)$, $k=1,\ldots,n$, are
identically distributed, then

$\left\|C_{n}(X_{k})-C_{n}(Y_{k})\right\|_{2}\leq\sqrt{8}\max_{k}\left\|X_{k}-Y_{k}\right\|_{4}\sqrt{\left\|X_{k}\right\|_{4}^{2}+\left\|Y_{k}\right\|_{4}^{2}}$
(69)

Proof. From (65) and the inequality $\left(a+b\right)^{2}\leq
2\left(a^{2}+b^{2}\right),$ we have

$\left|C_{n}(X_{k})-C_{n}(Y_{k})\right|^{2}\leq
8E_{n}\left(\left|X_{k}-Y_{k}\right|^{2}\right)\left(E_{n}\left(\left|X_{k}\right|^{2}\right)+E_{n}\left(\left|Y_{k}\right|^{2}\right)\right).$

Integrating and using the Cauchy-Schwarz inequality for random variables for
each term yields

$\displaystyle E\left(\left|C_{n}(X_{k})-C_{n}(Y_{k})\right|^{2}\right)$
$\displaystyle\leq
8E\left(E_{n}\left(\left|X_{k}-Y_{k}\right|^{2}\right)\left[E_{n}\left(\left|X_{k}\right|^{2}\right)+E_{n}\left(\left|Y_{k}\right|^{2}\right)\right]\right)$
(70) $\displaystyle\leq
8\left\|E_{n}\left(\left|X_{k}-Y_{k}\right|^{2}\right)\right\|_{2}\left[\left\|E_{n}\left(\left|X_{k}\right|^{2}\right)\right\|_{2}+\left\|E_{n}\left(\left|X_{k}\right|^{2}\right)\right\|_{2}\right],$

where by Cauchy-Schwarz inequality,

$\left\|E_{n}\left(\left|X_{k}\right|^{2}\right)\right\|_{2}\leq\max_{k=1:n}\left\|X_{k}\right\|_{4}^{2}$
(71)

since

$\displaystyle\left\|E_{n}\left(\left|X_{k}\right|^{2}\right)\right\|_{2}^{2}$
$\displaystyle=E\left(\left(\frac{1}{n}\sum_{k=1}^{n}\left|X_{k}\right|^{2}\right)^{2}\right)$
$\displaystyle=\frac{1}{n^{2}}\sum_{k=1}^{n}\sum_{\ell=1}^{n}E\left(\left|X_{k}\right|^{2}\left|X_{\ell}\right|^{2}\right)$
$\displaystyle\leq\frac{1}{n^{2}}\sum_{k=1}^{n}\sum_{\ell=1}^{n}E\left(\left|X_{k}\right|^{4}\right)^{1/2}E\left(\left|X_{\ell}\right|^{4}\right)^{1/2}$
$\displaystyle\leq\max_{k=1:n}E\left(\left|X_{k}\right|^{4}\right)=\max_{k=1:n}\left\|X_{k}\right\|_{4}^{4},$

and similarly for the other terms. Now taking the square root of (70) and
using (71) three times gives the desired estimate (69).

##  7 Laws of large numbers

###  7.1 $L^{2}$ and weak law of large numbers

The following theorem generalizes the weak law of large numbers to the Hilbert
space setting. The usual proof caries over.

###### Lemma 36

Let $X_{k}\in L^{2}\left(\Omega,H\right)$ be i.i.d. Then

$\left\|E_{n}\left(X_{k}\right)-E\left(X_{1}\right)\right\|_{2}\leq\frac{1}{\sqrt{n}}\left\|X_{1}-E\left(X_{1}\right)\right\|_{2}\leq\frac{2}{\sqrt{n}}\left\|X_{1}\right\|_{2}$
(72)

and $E_{n}\left(X_{k}\right)\Rightarrow E\left(X_{1}\right)$ as
$n\rightarrow\infty$.

Proof. First, without loss of generality, let $E\left(X_{1}\right)=0$. Since
$X_{k}$ are independent, they are uncorrelated (Lemma 30), and

$\displaystyle\left\|E_{n}\left(X_{k}\right)\right\|_{2}^{2}$
$\displaystyle=E\left(\left|\left(\frac{1}{N}\sum_{k=1}^{n}X_{k}\right)\right|^{2}\right)=\frac{1}{n^{2}}\sum_{k=1}^{n}\sum_{\ell=1}^{n}E\left(\left\langle
X_{k},X_{\ell}\right\rangle\right)$
$\displaystyle=\frac{1}{n}E\left(\left|X_{1}\right|^{2}\right)=\frac{1}{n}\left\|X_{1}\right\|_{2}^{2}.$

For general $E\left(X_{1}\right)$, from
$E_{n}\left(X_{k}-E\left(X_{1}\right)\right)=E_{n}\left(X_{k}\right)-E\left(X_{1}\right)$
and from the triangle inequality,

$\left\|E_{n}\left(X_{k}\right)-E\left(X_{1}\right)\right\|_{2}=\frac{1}{\sqrt{n}}\left\|X_{1}-E\left(X_{1}\right)\right\|_{2}\leq\frac{1}{\sqrt{n}}\left(\left\|X_{1}\right\|_{2}+\left\|E\left(X_{1}\right)\right\|_{2}\right).$

Since $E\left(X_{1}\right)$ is a constant and by properties of the integral
and monotonicity $L^{p}$ norms,

$\left\|E\left(X_{1}\right)\right\|_{2}=\left|E\left(X_{1}\right)\right|\leq
E\left(\left|X_{1}\right|\right)=\left\|X_{1}\right\|_{1}\leq\left\|X_{1}\right\|_{2},$

which gives (72). Now by Chebyschev’s inequality for the random variable
$\left|E_{n}\left(X_{k}\right)-E\left(X_{1}\right)\right|$, for any
$\varepsilon>0$,

$\Pr\left(\left|E_{n}\left(X_{k}\right)-E\left(X_{1}\right)\right|>\varepsilon\right)\leq\left(\frac{\left\|E_{n}\left(X_{k}\right)-E\left(X_{1}\right)\right\|_{2}}{\varepsilon}\right)^{2}=\frac{4\left\|X_{1}\right\|_{2}^{2}}{n\varepsilon^{2}}\rightarrow
0,$

thus $\ E_{n}\left(X_{k}\right)\rightarrow E\left(X_{1}\right)$ in
probability. The result now follows from the fact that for random elements in
a metric space, convergence in probability to a constant implies convergence
in distribution [26, Lemma 4.7].

###  7.2 $L^{p}$ law of large numbers

Marcinkiewicz-Zygmund inequality allows to prove a variant of the weak law of
large numbers in stronger $L_{p}$ norms. The following statement is
essentially the same as [11, Corollary 2, page 368], just with more detail,
and using the Marzinkiewicz-Zygmund inequality in Hilbert spaces.

###### Lemma 37

Let $H$ be a Hilbert space, $X_{i}\in L^{p}\left(\Omega,H\right)$ be i.i.d.,
$p\geq 2$, and $E_{n}\left(X_{k}\right)=\frac{1}{n}\sum_{k=1}^{n}X_{k}$. Then,

$\left\|E_{n}\left(X_{k}\right)-E\left(X_{1}\right)\right\|_{p}\leq\frac{C_{p}}{\sqrt{n}}\left\|X_{1}-E\left(X_{1}\right)\right\|_{p}\leq\frac{2C_{p}}{\sqrt{n}}\left\|X_{1}\right\|_{p},$
(73)

where $C_{p}$ depends on $p$ only.

Proof. If $p=2$, the statement becomes (72). Let $p>2$, and, without loss of
generality, assume first that $E\left(X_{1}\right)=0$. By Hölder’s inequality,

$\sum_{k=1}^{n}\left|X_{k}\right|^{2}=\sum_{k=1}^{n}1\left(\left|X_{k}\right|^{2}\right)\leq
n^{\left(p-2\right)/p}\left(\sum_{k=1}^{n}\left(\left|X_{k}\right|^{2}\right)^{p/2}\right)^{2/p}=n^{\left(p-2\right)/p}\left(\sum_{k=1}^{n}\left|X_{k}\right|^{p}\right)^{2/p},$
(74)

thus, using Marcinkiewicz-Zygmund inequality (57),

$\displaystyle E\left(\left|\sum_{k=1}^{n}X_{k}\right|^{p}\right)$
$\displaystyle\leq
B_{p}E\left(\left(\sum_{k=1}^{n}\left|X_{k}\right|^{2}\right)^{p/2}\right)$
$\displaystyle\leq
B_{p}n^{p/2-1}E\left(\sum_{k=1}^{n}\left|X_{k}\right|^{p}\right)$
$\displaystyle=B_{p}n^{p/2}E\left(\left|X_{1}\right|^{p}\right)$

because $\frac{p}{2}\frac{p-2}{p}=\frac{p}{2}-1$ and $X_{k}$ are identically
distributed. Consequently,

$\left\|\sum_{k=1}^{n}X_{k}\right\|_{p}\leq
B_{p}^{1/p}n^{1/2}\left\|X_{1}\right\|_{p},$

and the first inequality in (73) follows. The rest follows from the triangle
inequality.

###  7.3 Convergence of sample covariance

Recall that from (63),

$C_{n}(X_{k},Y_{k})=E_{n}\left(X_{k}\otimes
Y_{k}\right)-E_{n}\left(X_{k}\right)\otimes E_{n}\left(Y_{k}\right),$

while

$\operatorname*{Cov}\left(X_{1},Y_{1}\right)=E\left(X_{1}\otimes
Y_{1}\right)-E\left(X_{1}\right)\otimes E\left(Y_{1}\right).$

We need convergence of $C_{n}(X_{k},Y_{k})$ to
$\operatorname*{Cov}(X_{1},Y_{1})$ as $n\rightarrow\infty$, in some sense,
i.e., the laws of large numbers for the sample covariance.

Since covariance is in the Hilbert-Schmidt space $HS\left(H\right)$, which is
a Hilbert space, we can use laws of large numbers in the space of Hilbert-
Schmidt operators.

###### Lemma 38

If $p\geq 2$, $H$ is Hilbert space, and $X_{k}\in L^{2p}\left(\Omega,H\right)$
are i.i.d. Then

$\left(E\left|C_{n}\left(X_{k}\right)-\operatorname*{Cov}\left(X_{1}\right)\right|_{HS\left(H\right)}^{p}\right)^{1/p}\leq\left(\frac{2C_{p}}{\sqrt{n}}+\frac{4C_{2p}^{2}}{n}\right)\left\|X_{1}\right\|_{2p}^{2}.$
(75)

where $C_{r}$ is a constant which depends on $r$ only; in particular,
$C_{2}=1.$

Proof. Without loss of generality, let $E\left(X_{1}\right)=0$. Then

$C_{n}\left(X_{k}\otimes Y_{k}\right)-\operatorname*{Cov}\left(X_{1}\otimes
Y_{1}\right)=\left(E_{n}\left(X_{k}\otimes Y_{k}\right)-E\left(X_{1}\otimes
Y_{1}\right)\right)+E_{n}\left(X_{k}\right)\otimes E_{n}\left(Y_{k}\right).$
(76)

The first term in (76) is estimated by

$\displaystyle\left(E\left|E_{n}\left(X_{k}\otimes
X_{k}\right)-E\left(X_{1}\otimes
X_{1}\right)\right|_{HS\left(H\right)}^{p}\right)^{1/p}$
$\displaystyle\leq\frac{2C_{p}}{\sqrt{n}}\left(E\left|X_{1}\otimes
X_{1}\right|_{HS\left(H\right)}^{p}\right)^{1/p}$
$\displaystyle=\frac{2C_{p}}{\sqrt{n}}\left(E\left|X_{1}\right|_{H}^{2p}\right)^{1/p}=\frac{2C_{p}}{\sqrt{n}}\left\|X\right\|_{2p}^{2}$

from the $L^{p}$ law of large numbers (73) in the Hilbert space
$HS\left(H\right)$, and computing the Hilbert-Schmidt norm of tensor product
by (25).

For the second term in (76), first note that

$\left|E_{n}\left(X_{k}\right)\otimes
E_{n}\left(X_{k}\right)\right|_{HS\left(H\right)}=\left|E_{n}\left(X_{k}\right)\right|_{H}^{2}.$

Then, (18) and the $L^{p}$ law of large numbers in $H$ yield

$\left(E\left|E_{n}\left(X_{k}\right)\otimes
E_{n}\left(X_{k}\right)\right|_{HS\left(H\right)}^{p}\right)^{1/p}=\left(E\left|E_{n}\left(X_{k}\right)\right|_{H}^{2p}\right)^{1/p}=\left\|E_{n}\left(X\right)-0\right\|_{2p}^{2}\leq\left(\frac{2C_{2p}}{\sqrt{n}}\left\|X_{1}\right\|_{2p}\right)^{2}.$

The proof is concluded by the use of the triangle inequality.

Lemma 38 proves convergence of the sample covariance in the Hilbert-Schmidt
norm, which is weaker than the trace norm, which would be the natural norm for
covariances. However, the space $\operatorname*{Tr}\left(H\right)$ of trace
class operators is not a Hilbert space.

###### Corollary 39

Using the inequality between the operator norm and the Hilbert-Schmidt norm
(24), we have from (75) that

$\left\|C_{n}\left(X_{k}\right)-\operatorname*{Cov}\left(X_{1}\right)\right\|_{p}\leq\left(\frac{2C_{p}}{\sqrt{n}}+\frac{4C_{2p}^{2}}{n}\right)\left\|X_{1}\right\|_{2p}^{2}.$

###### Remark 40

The argument developed in Lemma 37, Lemma 38, and Corollary 39 is essentially
a reformulation the proof of Lemma 3.3 in [39]. We have extended the argument
to infinite dimension and made explicit the fact that the bounds are
independent of the dimension of the space.

###  7.4 Strong laws of large numbers

######  Theorem 41 ([40, Theorem 7.9])

Let $0<p<2$, and $\left(X_{i}\right)$ be i.i.d. random elements with values in
Banach space $B$, and $S_{n}=X_{1}+\cdots+X_{n}$. Then

$\frac{S_{n}}{n^{1/p}}\rightarrow 0\text{ almost surely}$

if and only if

$E\left(\left|X_{1}\right|^{p}\right)<\infty\text{ and
}\frac{S_{n}}{n^{1/p}}\rightarrow 0\text{ in probability.}$

######  Theorem 42 ([40, Corollary 7.10])

Let $\left(X_{i}\right)$ be i.i.d. random elements with values in Banach space
$B$ with $E\left(\left|X_{1}\right|\right)<\infty$, and
$E_{n}=\frac{1}{n}\left(X_{1}+\cdots+X_{n}\right)$. Then

$E_{n}\rightarrow E\left(X\right)\text{ almost surely.}$

##  8 Gaussian measures

Given a measure $\mu$ on a Hilbert space $H$, its Fourier transform is the
function

$\widehat{\mu}:h\in H\mapsto\int_{H}e^{-i\left\langle
h,u\right\rangle}\mu\left(du\right).$

More generally, if $U$ is a linear topological space and $\mu$ a measure on
the dual space $U^{\prime}$ (the space for all continous linear functionals on
$U$), the Fourier transform of $\mu$ is the function

$\widehat{\mu}:v\in U\mapsto\int_{U^{\prime}}e^{-i\left\langle
v,u\right\rangle}\mu\left(du\right),$

where $\left\langle v,u\right\rangle=u\left(v\right)$.

###### Proposition 43

Given a separable Hilbert space $H$, $a\in H$, and $Q$ symmetric positive
semidefinite linear operator on $H$ of trace class, then there exists unique
probability measure $\mu$ on $H$ such that

$\widehat{\mu}\left(h\right)=e^{i\left\langle
a,h\right\rangle-\frac{1}{2}\left\langle Qh,h\right\rangle}.$ (77)

The measure $\mu$ has mean $a$ and covariance $Q$. If $X$ is a random element
with values in $H$ and with distribution $\mu$, and $u\in H$, then

$\left\langle u,X\right\rangle\sim N\left(\left\langle
a,u\right\rangle,\left\langle u,Qu\right\rangle\right).$

The Gaussian measure satisfying (77) is denoted by $N\left(a,Q\right)$.

###### Proposition 44

If $\mu$ is a Gaussian measure on a Hilbert space $H$ and $1\leq p<\infty$,
then

$M_{p}\left(\mu\right)=\int_{H}\left|x\right|^{p}\mu\left(dx\right)<\infty.$
(78)

That is, Gaussian measure has finite moments of all orders.

The space $L^{p}\left(H,\mu\right)$ for a Gaussian measure $\mu$ has some
addional properties.

###### Proposition 45

If $H$ is a Hilbert space, $\mu$ a centered Gaussian measure, and $1\leq
p<\infty$. Then $H^{\prime}\subset L^{p}\left(H,\mu\right)$ and the embedding
$H^{\prime}\rightarrow L^{p}\left(H,\mu\right)$ is continous. If $H$ is
infinitely dimensional, the $H^{\prime}$ and the $L^{p}\left(H,\mu\right)$
norm are not equivalent on $H^{\prime}$.

Proof. Let $v\in H^{\prime}$. Then

$\displaystyle\left|v\right|_{L^{p}\left(H,\mu\right)}$
$\displaystyle=\left(\int_{H}\left|\left\langle
v,u\right\rangle\right|^{p}\mu\left(du\right)\right)^{1/p}\leq\left(\int_{H}\left|v\right|_{H^{\prime}}^{p}\left|u\right|_{H}^{p}\mu\left(du\right)\right)^{1/p}=$
(79)
$\displaystyle=\left|v\right|_{H^{\prime}}^{p}\left(\int_{H}\left|u\right|_{H}^{p}\mu\left(du\right)\right)^{1/p}=\left|v\right|_{H^{\prime}}M_{p}\left(\mu\right)^{1/p},$
(80)

with $M_{p}\left(\mu\right)<\infty$ by (78), which proves that
$H^{\prime}\subset L^{p}\left(H,\mu\right)$ with continous embedding. To show
that the opposite inequality does not hold, let $1\leq p<\infty$ and
$\left\\{e_{k}\right\\}$ be an orthonormal basis of eigenvectors of the
covariance of $\mu$ with the corresponding eigenvalues $\lambda_{k}$, and
define linear functionals $v_{n}:H\rightarrow\mathbb{R}$ by

$v_{n}:u={\textstyle\sum\nolimits_{k=1}^{\infty}}u_{k}e_{k}\mapsto u_{n}.$

Then

$\left|v_{n}\right|_{L^{p}\left(H,\mu\right)}=c_{n}\left(\int_{-\infty}^{+\infty}\left|x\right|^{p}e^{-x^{2}/2\lambda_{n}}dx\right)^{1/p}=C_{p}\lambda_{n}^{1/p}\rightarrow
0\text{ as }n\rightarrow\infty,$

for some constant $C_{p}$, while

$\left|v_{n}\right|_{H^{\prime}}=\sup\left\\{u_{n}:{\textstyle\sum\nolimits_{k=1}^{\infty}}u_{k}^{2}=1\right\\}=1.$

Note that the standard definition of the dual norm gives,

$\left|u\right|_{L^{p}\left(H,\mu\right)^{\prime}}=\sup\left\\{\left|\left\langle
u,v\right\rangle\right|:v\in H,\text{ }\int_{H}\left|\left\langle
x,v\right\rangle\right|^{p}\mu\left(dx\right)\leq 1\right\\},\quad u\in H.$
(81)

###### Lemma 46

The space $L^{p}\left(H,\mu\right)^{\prime}\cap H$ is continously embedded and
dense in $H.$

Proof. From (79), for any $u\in H$

$\displaystyle\left|u\right|_{H}$
$\displaystyle=\sup\left\\{\left|\left\langle u,v\right\rangle\right|:v\in
H^{\prime},\text{ }\left|v\right|_{H^{\prime}}\leq 1\right\\}$
$\displaystyle\leq\sup\left\\{\left|\left\langle u,v\right\rangle\right|:v\in
H^{\prime},\text{ }\left|v\right|_{L^{p}\left(H,\mu\right)}\leq C\right\\}$
$\displaystyle=C\sup\left\\{\left|\left\langle u,v\right\rangle\right|:v\in
H^{\prime},\text{ }\left|v\right|_{L^{p}\left(H,\mu\right)}\leq 1\right\\}$
$\displaystyle\leq C\left|u\right|_{L^{p}\left(H,\mu\right)^{\prime}}.$

##  9 White noise

###  9.1 Classical definition of white noise stochastic process

The classical concept of Gaussian white noise is a random processes, that is a
collection of random variables $F\left(t\right):\Omega\rightarrow$
$\mathbb{R}$, $t\in\left[0,1\right]$, such that

$\displaystyle E\left(F\left(t\right)\right)$ $\displaystyle=0\text{ for all
}t\in\left[0,1\right],$
$\displaystyle\operatorname*{Cov}\left(F\left(r\right),F\left(s\right)\right)$
$\displaystyle=E\left(F\left(r\right)F\left(s\right)\right)=\left\\{\begin{array}[c]{c}0\text{
if }r\neq s,\\\ 1\text{ if }r=s.\end{array}\right.$

Alternatively, $F$ is thought of as a random element whose values are real
function on $\left[0,1\right]$, called paths. White noise $F$ is gaussian if
the distribution of $F\left(t\right)$ is gaussian for all $t$, that is,
$F\left(t\right)\sim N\left(0,1\right)$. Similarly, white noise vector $F$ in
$\mathbb{R}^{n}$ is a random vector with zero mean and covariance equal to the
identity $I$, and gaussian white noise if $F\sim N(0,I)$. However, for random
elements $F$ with values in an infinitely dimensional Hilbert space, this is
not possible, because the covariance of a random element must be a trace class
operator, while $\operatorname*{Tr}I=+\infty$. Thus, a white noise random
element with values in a Hilbert space cannot exist, and something must give
way.

###  9.2 White noise as a weak random variable

One natural generalization of white noise in $\mathbb{R}^{n}$ to a Hilbert
space $H$ is a weak random variable. The space of all (not necessarily
bounded) linear functionals on $H$ is called the algebraic dual of $H$ and
denoted by $H^{\\#}$. The value of linear functional $F\in H^{\\#}$ at $u\in
H$ is denoted by $\left\langle u,F\right\rangle$. Recall that random variable
means a measurable function on some probability space
$\left(\Omega,\mathcal{F}\right)$.

###### Definition 47

Let $H$ be a Hilbert space. A map $F:\Omega\rightarrow H^{\\#}$ is a weak
random variable if for every $u\in H$, $\left\langle u,F\right\rangle$ is a
random variable.

The definitions of mean and covariance remain the same as in (27) and (32).

###### Definition 48

For a weak random variable $F:\Omega\rightarrow H^{\\#}$, the mean value
$E\left(X\right)$ (if it exists) is defined by

$E\left(F\right)\in H,\quad\left\langle
v,E\left(F\right)\right\rangle=E\left(\left\langle
v,F\right\rangle\right)\quad\forall v\in H$

and its covariance (if it exists) is the linear operator
$\operatorname*{Cov}\left(F\right)$, defined by

$\left\langle
u,\operatorname*{Cov}\left(F\right)v\right\rangle=E\left(\left\langle
u,F-E\left(F\right)\right\rangle\left\langle
v,F-E\left(F\right)\right\rangle\right)\quad\forall u,v\in H,$ (82)

###### Definition 49

A weak random variable $F$ is gaussian if for any $v_{1},\ldots,v_{n}$, the
random vector $\left(\left\langle v_{1},F\right\rangle,\ldots,\left\langle
v_{n},F\right\rangle\right)$ is gaussian.

###### Definition 50

White noise on a Hilbert space $H$ is a gaussian weak random variable $F$ such
that $E\left(F\right)=0$ and $\operatorname*{Cov}\left(F\right)=I.$

###### Remark 51

It follows from Definition 50 that white noise $F$ satisfies

$\left\langle
u,\operatorname*{Cov}\left(F\right)v\right\rangle=E\left(\left\langle
u,F\right\rangle\left\langle v,F\right\rangle\right)=\left\langle
u,v\right\rangle\quad\forall u,v\in H.$ (83)

Condition (83) can be stated equivalently that the map

$u\in H\mapsto\left\langle u,F\left(\cdot\right)\right\rangle\in
L^{2}\left(\Omega\right)$

is isometry.

The definition of noise as a weak random variable means that the noise $F$
does not need to be measurable, only averages of the form $\left\langle
u,F\right\rangle$ are required to be measurable.

###### Remark 52

It follows immediately from Definition 50 that Gaussian white noise $F$
satisfies $E\left(\left\langle v,F\right\rangle\left\langle
u,F\right\rangle\right)=\left\langle u,v\right\rangle$ and taking $u=v$, we
have

$E\left(\left|\left\langle u,F\right\rangle\right|^{2}\right)=1\quad\forall
u\in H,\left|u\right|=1.$ (84)

While, in this sense, the norm of the random linear functional
$u\mapsto\left\langle u,F\right\rangle$ is one in squared average, and (84)
looks similar to the definition of the norm of a functional,

$\left\|F\left(\omega\right)\right\|_{H^{\prime}}=\sup_{u\in
H,\left\|u\right\|=1}\left|\left\langle
u,F\left(\omega\right)\right\rangle\right|,$

(84) does not say that linear functionals $u\mapsto\left\langle
u,F\left(\omega\right)\right\rangle$ for a fixed $\omega$ are bounded, i.e.,
(84) does not necessarily imply that

$\sup_{u\in H,\left|u\right|=1}\left|\left\langle
u,F\left(\omega\right)\right\rangle\right|^{2}<\infty\quad\forall\omega\in\Omega.$
(85)

(It would be good to find an example where (85) is false.)

###### Remark 53

However, we do not know yet if such weak random variable exists. In [4], white
noise is constructed using a theory of finitely additive measures instead of
the standard measure theory. However, we will proceed as in [15], just with
few more details, and then interpret the result in the sense of Definition 50.

###  9.3 Whitening of a random vector and white noise mapping

A classical way how to construct a white noise vector in finite dimension is
by “whitening” an existing random vector. Given a random vector $X\sim
N\left(0,Q\right)$ on $\mathbb{R}^{n}$, whitening of the vector $X$ is defined
as the random vector $Q^{-1/2}X$. Clearly,

$\displaystyle\operatorname*{Cov}\left(Q^{-1/2}X\right)$
$\displaystyle=E\left(Q^{-1/2}X\left(Q^{-1/2}X\right)^{\mathrm{T}}\right)=E\left(Q^{-1/2}XX^{\mathrm{T}}Q^{-1/2}\right)$
$\displaystyle=Q^{-1/2}E\left(XX^{\mathrm{T}}\right)Q^{-1/2}=Q^{-1/2}QQ^{-1/2}=I$

so $Q^{-1/2}X$ is indeed white noise.

Suppose that $X$ is a random element with distribution $\mu$, where
$\mu=N\left(0,Q\right)$ is a Gaussian measure on an infinitely dimensional
Hilbert space $H$ with nondegenerate covariance operator $Q$, that is, $\ker
Q=\left\\{0\right\\}$. Since $Q$ is self-adjoint compact operator, there is a
complete orthonormal sequence $\left\\{v_{k}\right\\}$ consisting of the
eigenvectors of $Q$ with eigenvalues $\lambda_{1}\geq\lambda_{2}\geq\cdots>0$,
and $Q$ has the spectral decomposition

$Q=\sum_{k=1}^{\infty}\lambda_{k}v_{k}\otimes v_{k}.$

Any real power of $Q$ is then defined by

$Q^{\theta}=\sum_{k=1}^{\infty}\lambda_{k}^{\theta}v_{k}\otimes
v_{k},\quad\theta\in\mathbb{R}.$

Since $Q$ is of trace class, $\sum_{k=1}^{\infty}\lambda_{k}<\infty$, and,
consequently, $\lim_{k\rightarrow\infty}\lambda_{k}=0$.

Unfortunately, the construction of white noise by whitening does not carry
over immediately to an infinitely dimensional Hilbert space because $Q^{-1/2}$
is an unbounded operator, defined only on the subspace

$\mathcal{D}\left(Q^{-1/2}\right)=Q^{1/2}\left(H\right)=\left\\{\sum_{k=1}^{\infty}\lambda_{k}^{1/2}c_{k}v_{k}:\sum_{k=1}^{\infty}c_{k}^{2}<\infty\right\\}\subset
H,$

which is called the _Cameron-Martin space_ of the measure $\mu$. A definition
of white noise as $Q^{-1/2}X$, $X\sim\mu$, then runs into a difficulty: it is
not true that $Q^{-1/2}$ is defined $\mu$-a.e. in $H$; in fact, exactly the
opposite is true [15, Proposition 1.27],

$\mu\left(Q^{1/2}\left(H\right)\right)=0.$

Consequently, $Q^{-1/2}X$ is not a random element, because it is not defined
a.s. Furthermore, the operator $Q^{-1/2}$ cannot be extended to all of $H$ by
continuity using the $H$-norm: The Cameron-Martin space
$Q^{1/2}\left(H\right)$ is dense in $H$ but the operator $Q^{-1/2}:$
$Q^{1/2}\left(H\right)\rightarrow H$ is unbounded, that is, not continuous in
the $H$ norm.

###  9.4 White noise mapping

Extending $Q^{-1/2}$ by continuity is possible if one can find some other
Banach space $V\supset H$ such that $Q^{-1/2}$ is continuous from
$\left(Q^{1/2}\left(H\right),\left\|\cdot\right\|\right)$ to
$\left(V,\left\|\cdot\right\|_{V}\right)$. Choose $V=L^{2}\left(H,\mu\right)$.
This is a space of real-valued functions on $H$, not necessarily linear, with
the inner product

$\left\langle
F,G\right\rangle_{L^{2}\left(H,\mu\right)}=\int_{H}FGd\mu=\int_{H}F\left(u\right)G\left(u\right)\mu\left(du\right).$

An element $u\in H$ is identified with the bounded linear functional
$x\mapsto\left\langle x,u\right\rangle$ in $H^{\prime}$, and then
$H=H^{\prime}$ by the Riesz representation theorem. Since

$\left|u\right|_{L^{2}\left(H,\mu\right)}=\left(\int_{H}\left|\left\langle
x,u\right\rangle\right|^{2}\mu\left(dx\right)\right)^{1/2}\leq\operatorname*{const}\left|u\right|_{H}\quad\forall
u\in H$ (86)

by Proposition 45, it holds that

$H=H^{\prime}\subset L^{2}\left(H,\mu\right).$

with continuous inclusion. The $L^{2}\left(H,\mu\right)$ inner product becomes
for $u$,

$v\in H\left\langle
u,v\right\rangle_{L^{2}\left(H,\mu\right)}=\int_{H}\left\langle
x,u\right\rangle\left\langle x,v\right\rangle\mu\left(dx\right)=\left\langle
Qu,v\right\rangle$

by the definition of covariance (82). Now $Q^{-1/2}$ is continuous, in fact
isometry from $Q^{1/2}\left(H\right)$ to $L^{2}\left(H,\mu\right)$:

$\displaystyle\left\langle
Q^{-1/2}w,Q^{-1/2}z\right\rangle_{L^{2}\left(H,\mu\right)}$
$\displaystyle=\int_{H}\left\langle x,Q^{-1/2}w\right\rangle\left\langle
x,Q^{-1/2}z\right\rangle\mu\left(dx\right)$ $\displaystyle=\left\langle
Q^{-1/2}w,QQ^{-1/2}z\right\rangle=\left\langle w,z\right\rangle.$

Hence, $Q^{-1/2}$ can be uniquely extended to a continuous mapping (which is
also isometry)

$W:H\rightarrow L^{2}\left(H,\mu\right),\quad\left\langle
Ww,Wz\right\rangle_{L^{2}\left(H,\mu\right)}=\left\langle w,z\right\rangle,$
(87)

called the _white noise mapping_[15, p. 23].

Since the $L^{2}\left(H,\mu\right)$ limit of a sequence of linear functionals
is clearly a linear functional, $W\left(z\right)$ are still linear functionals
on $H$, so $W\left(z\right)\in H^{\\#}\cap L^{2}\left(H,\mu\right)$. However,
for $z\in H\setminus Q^{1/2}\left(H\right)$, they are not necessarily bounded
linear functionals, because the $L^{2}\left(H,\mu\right)$ limit of a sequence
of bounded linear functionals on $H$ does not need to be a bounded functional.

###### Proposition 54

Let $\mu$ be a centered Gaussian measure on Hilbert space $H$, $W:H\rightarrow
H^{\\#}\cap L^{2}\left(H,\mu\right)$ the white noise mapping, and
$X:\Omega\rightarrow H$ a random element with distribution $\mu$. Define
$F:\Omega\rightarrow H^{\\#}\cap L^{2}\left(H,\mu\right)$ by
$F\left(\omega\right)=WX\left(\omega\right)$. Then $\left\langle
u,F\right\rangle$ is a Gaussian random variable for every $u\in H$, and

$E\left(\left\langle v,F\right\rangle\left\langle
u,F\right\rangle\right)=\left\langle u,v\right\rangle,\quad\forall u,v\in H.$
(88)

Proof. We already know that for a fixed $\omega$, $F\left(\omega\right)$ is a
linear functional, and the mapping $WX\left(\omega\right)$ is linear because
the mapping $u\in H\mapsto$ $W_{u}\in$ $L^{2}\left(H,\mu\right)$ is linear,
being a continuous extension a linear mapping on a dense subspace of $H$. By
substitution from (87),

$\displaystyle E\left(\left\langle v,F\right\rangle\left\langle
u,F\right\rangle\right)$ $\displaystyle=\int_{\Omega}\left\langle
v,F\left(\omega\right)\right\rangle\left\langle
u,F\left(\omega\right)\right\rangle d\omega$
$\displaystyle=\int_{\Omega}\left\langle
v,WX\left(\omega\right)\right\rangle\left\langle
u,WX\left(\omega\right)\right\rangle d\omega$
$\displaystyle=\int_{H}\left\langle v,Wx\right\rangle\left\langle
u,Wx\right\rangle\mu\left(dx\right)$ $\displaystyle=\int_{H}\left\langle
x,Wv\right\rangle\left\langle x,Wu\right\rangle\mu\left(dx\right)$
$\displaystyle=\left\langle Wv,QWu\right\rangle$ $\displaystyle=\left\langle
Q^{-1/2}v,QQ^{-1/2}u\right\rangle$ $\displaystyle=\left\langle
v,u\right\rangle$

for all $u,v\in Q^{1/2}\left(H\right)$. Because $Q^{1/2}\left(H\right)$ is
dense in $H$, the covariance property (88) follows by continuity.

##  10 Convergence analysis of the EnKF in the large ensemble limit

###  10.1 Optimal statistical interpolation

Consider a stochastic model with the state $V\sim N\left(\mu,Q\right)$ in
$\mathbb{R}^{n}$, and a given observation (i.e., data) vector $d$. The
$n\times n$ matrix $Q$ is called the background covariance. The correspondence
between the state and the data is given by a linear observation operator $H$
and data error covariance $R$: the data is assumed to be distributed as $d\sim
N\left(Hv,R\right)$, given the value of the state $V$ equal to $v$. That is,
$Hv$ is what the data would be if there were no errors and the truth were $v$,
and $R$ is the covariance of the normally distributed data error. Then the
probability density of the state is

$p_{V}\left(v\right)\propto
e^{-\frac{1}{2}\left(v-\mu\right)^{\mathrm{T}}Q^{-1}\left(v-\mu\right)}$

where $\propto$ means proportional, and the data likelihood is

$p\left(d|v\right)\propto
e^{-\frac{1}{2}\left(Hv-d\right)^{\mathrm{T}}R^{-1}\left(Hv-d\right)}.$

From the Bayes theorem, the probability density of the model state after the
data is taken into account, called the analysis $V^{a}$, is

$p_{V^{a}}(v)\propto p(d|v)p_{V}(v)\propto
e^{-\frac{1}{2}\left[\left(v-\mu\right)^{\mathrm{T}}Q^{-1}\left(v-\mu\right)+\left(Hv-d\right)^{\mathrm{T}}R^{-1}\left(Hv-d\right)\right]}$

Note that the exponent involves a quadratic function of $v$, which we can
write in the form

$\left(v-\mu\right)^{\mathrm{T}}Q^{-1}\left(v-\mu\right)+\left(Hv-d\right)^{\mathrm{T}}R^{-1}\left(Hv-d\right)=\left(v-\mu^{a}\right)^{\mathrm{T}}\left(Q^{a}\right)^{-1}\left(v-\mu^{a}\right)$

and compare the quadratic and linear terms:

$\displaystyle v^{\mathrm{T}}Q^{-1}v+v^{\mathrm{T}}H^{\mathrm{T}}R^{-1}Hv$
$\displaystyle=v^{\mathrm{T}}\left(Q^{a}\right)^{-1}v$
$\displaystyle-2v^{\mathrm{T}}Q^{-1}\mu-2v^{\mathrm{T}}H^{\mathrm{T}}R^{-1}d$
$\displaystyle=-2v^{\mathrm{T}}\left(Q^{a}\right)^{-1}\mu^{a}$

which gives

$\displaystyle Q^{-1}+H^{\mathrm{T}}R^{-1}H$
$\displaystyle=\left(Q^{a}\right)^{-1}$ (89) $\displaystyle
Q^{-1}\mu+H^{\mathrm{T}}R^{-1}d$
$\displaystyle=\left(Q^{a}\right)^{-1}\mu^{a}$ (90)

which gives

$\displaystyle Q^{a}$
$\displaystyle=\left(Q^{-1}+H^{\mathrm{T}}R^{-1}H\right)^{-1}$
$\displaystyle\mu^{a}$
$\displaystyle=Q^{a}\left(Q^{-1}\mu+H^{\mathrm{T}}R^{-1}d\right)$

Using the Sherman-Morrison-Woodbury formula

$\left(A+UCV\right)^{-1}=A^{-1}-A^{-1}U\left(C^{-1}+VA^{-1}U\right)^{-1}VA^{-1},$
(91)

we have

$\displaystyle Q^{a}$
$\displaystyle=\left(\underbrace{Q^{-1}}_{A}+\underbrace{H^{\mathrm{T}}}_{U}\underbrace{R^{-1}}_{C}\underbrace{H}_{V}\right)^{-1}$
(92) $\displaystyle=Q-QH^{\mathrm{T}}\left(R+HQH^{\mathrm{T}}\right)^{-1}HQ$
$\displaystyle=\left(I-QH^{\mathrm{T}}\left(R+HQH^{\mathrm{T}}\right)^{-1}H\right)Q$
(93) $\displaystyle=(I-KH)Q,$ (94)

where $K$ is the Kalman gain matrix, given by

$K=QH^{\mathrm{T}}(HQH^{\mathrm{T}}+R)^{-1},$ (95)

and

$\displaystyle\mu^{a}$
$\displaystyle=Q^{a}\left(Q^{-1}\mu+H^{\mathrm{T}}R^{-1}d\right)$
$\displaystyle=(I-KH)Q\left(Q^{-1}\mu+H^{\mathrm{T}}R^{-1}d\right)$
$\displaystyle=(I-KH)\mu+\left(I-QH^{\mathrm{T}}\left(R+HQH^{\mathrm{T}}\right)^{-1}H\right)QH^{\mathrm{T}}R^{-1}d$
$\displaystyle=(I-KH)\mu+Kd$ $\displaystyle=\mu-K\left(H\mu-d\right),$

since

$\displaystyle\left(I-QH^{\mathrm{T}}\left(R+HQH^{\mathrm{T}}\right)^{-1}H\right)QH^{\mathrm{T}}R^{-1}$
$\displaystyle=QH^{\mathrm{T}}\left(I-\left(R+HQH^{\mathrm{T}}\right)^{-1}HQH^{\mathrm{T}}\right)R^{-1}$
$\displaystyle=QH^{\mathrm{T}}\left(R+HQH^{\mathrm{T}}\right)^{-1}\left(\left(R+HQH^{\mathrm{T}}\right)-HQH^{\mathrm{T}}\right)R^{-1}$
$\displaystyle=QH^{\mathrm{T}}(HQH^{\mathrm{T}}+R)^{-1}=K$

That is, the analysis distribution is

$V^{a}\sim N\left(\mu^{a},Q^{a}\right)$

where $\mu^{a}$ is given by the least squares estimate $v=\mu^{a}$,

$\left(v-\mu\right)^{\mathrm{T}}Q^{-1}\left(v-\mu\right)+\left(Hv-d\right)^{\mathrm{T}}R^{-1}\left(Hv-d\right)\rightarrow\min_{v}$
(96)

with the optimality conditions

$Q^{-1}\left(v-\mu\right)+H^{\mathrm{T}}R^{-1}\left(Hv-d\right)=0$

which gives

$\mu^{a}=\left(Q^{-1}+H^{\mathrm{T}}R^{-1}H\right)^{-1}\left(Q^{-1}\mu+H^{\mathrm{T}}R^{-1}d\right).$
(97)

and

$\mu^{a}=\mu+K(d-H\mu).$ (98)

When only the mean is advanced by the model and the background covariance is
obtained in other ways (typically by an expert judgement involving data
analysis), the method (98)-(95) is called optimal statistical interpolation
[28].

###  10.2 Kalman filter

Here is how a linear model transform state with normal distribution. Suppose
$u\propto N\left(\mu_{k}^{a},Q_{k}^{a}\right)$ is the analysis in step $k$
with linear model $Au+f$ acting on it. Then the forecast in the next step
$k+1$ is

$Au+f\propto N\left(A\mu_{k}^{a}+f,AQ_{k}^{a}A^{\top}\right)$

The output of the model is the forecast for the next cycle (time step)

$u\propto
N\left(\mu_{k+1}^{f},Q_{k+1}^{f}\right),\quad\mu_{k+1}^{f}=A\mu_{k}^{a}+f,\
Q_{k+1}^{f}=AQ_{k}^{a}A^{\top}$

This is how the original filter was formulated for a linear model [27]. The
model can of course be different from step to step, $A_{k}u+f_{k}$. When the
model is nonlinear, we use the first order approximation

$\mathcal{A}\left(u\right)\approx\mathcal{A}\left(u_{k}\right)+J\left(u_{k}\right)\left(u-u_{k}\right)$

and take

$u\propto
N\left(\mu_{k+1}^{f},Q_{k+1}^{f}\right),\quad\mu_{k+1}^{f}=\mathcal{A}\left(\mu_{k}^{a}\right),\
Q_{k+1}^{f}=J\left(u_{k}\right)Q_{k}^{a}J\left(u_{k}\right)^{\top}+D,$

where $D>0$ is regularization (oftem diagonal boosting) boosting matrix. These
are the equations of the extended Kalman filter. The extension to nonlinear
case [48] made many applications possible, and is still a de-facto standard in
navigation including GPS.

For practical use, regulariation is important - we add a positive definite
matrix $D$ to the covariance, usually diagonal. This prevents the degeneration
of the Kalman filter covariance to zero, because in optimal statistical
interpolation, the variance always decreases. The regularization “accounts
for” errors in the model vs. the reality, which the data come from.

###  10.3 The ensemble Kalman filter

To avoid manipulating the covariance matrix and to allow to use the model as a
black box, the ensemble Kalman filter approximates the state distribution
$N\left(\mu,Q\right)$ by a collection of of random vectors
$X_{k}\in\mathbb{R}^{n}$,

$X=[X_{1},\ldots,X_{N}]$ (99)

called an ensemble. Note that in general the ensemble is _not_ a sample
(i.i.d. set of random elements) from $N\left(\mu,Q\right)$; the analysis step
(101) below breaks the independence of the ensemble members. It is convenient
to operate with an ensemble as a matrix with the columns $X_{k}$. The analysis
step of the ensemble Kalman filter (EnKF) consists of creating the perturbed
data ensemble,

$D=[D_{1},\ldots,D_{N}],\quad D_{k}\sim N\left(d,R\right),$ (100)

independent of $X$, and the analysis ensemble $X^{a}$, by

$X^{a}=X+K_{N}(D-HX),$ (101)

where $K_{N}$ is the approximate Kalman gain matrix, given by

$K_{N}=Q_{N}H^{\mathrm{T}}(HQ_{N}H^{\mathrm{T}}+R)^{-1},$ (102)

and $Q_{N}$ is the sample covariance computed from the ensemble $X$,

$Q_{N}=\frac{1}{N-1}\sum_{k=1}^{N}A_{k}A_{k}^{\mathrm{T}},\quad
A_{k}=\left(X_{k}-\overline{X}\right),\quad\overline{X}=\frac{1}{N}\sum_{k=1}^{N}X_{k}.$
(103)

For practical application, note that $\overline{X}=\frac{1}{N}Xe$, where $e$
is the column vector of all ones of length $N$, and then

$HQ_{N}H^{\mathrm{T}}=\frac{1}{N-1}BB^{\mathrm{T}},$

where

$B=HX-HX\frac{ee^{\mathrm{T}}}{N}$

and use the Sherman-Morrison-Woodbury formula to (102) to get

$(HQ_{N}H^{\mathrm{T}}+R)^{-1}=R^{-1}\left[I-\frac{1}{N-1}B\left(I+\frac{B^{\mathrm{T}}R^{-1}B}{N-1}\right)^{-1}B_{k}^{\mathrm{T}}R_{k}^{-1}\right]$
(104)

We can write the ensemble covariance as

$\displaystyle Q_{N}$
$\displaystyle=\frac{1}{N-1}\left(X-X\frac{ee^{\mathrm{T}}}{N}\right)\left(X-X\frac{ee^{\mathrm{T}}}{N}\right)^{\mathrm{T}}$
$\displaystyle=X\frac{1}{N-1}\left(I-\frac{ee^{\mathrm{T}}}{N}\right)X^{\mathrm{T}},$

and the analysis ensemble (101) in the transformation form as linear
combination of the forecast forecast ensemble,

$X^{a}=XT,\quad
T=I+\frac{1}{N-1}\left(I-\frac{ee^{\mathrm{T}}}{N}\right)X^{\mathrm{T}}H^{\mathrm{T}}(HQ_{N}H^{\mathrm{T}}+R)^{-1}(D-HX).$
(105)

After the analysis step (101), each member of the ensemble is advanced by the
model (LABEL:eq:model) independently,

$X^{f}=\mathcal{M}\left(X^{a}\right),$ (106)

then we use $X^{f}$ in place of $X$ above, and the process repeats.

This is the version of EnKF from [10], distinguished by the use of the
randomized data (100). See [18] for futher details and variants. The reason
for the data perturbation (100) is to make the sample covariance of the
analysis what it should be.

###### Lemma 55

Suppose $U\sim N\left(\mu,Q\right)$ is a random vector with values in
$\mathbb{R}^{n}$, and $D\sim N\left(d,R\right)$ is a random vectors with
values in $\mathbb{R}^{m}$, $U$ and $D$ are independent, and
$H\in\mathbb{R}^{m\times n}$ is a matrix. Define

$U^{a}=U+K(D-HU),\quad K=QH^{\mathrm{T}}(HQH^{\mathrm{T}}+R)^{-1}.$ (107)

Then $U^{a}\sim N\left(\mu^{a},Q^{a}\right)$, where

$\mu^{a}=\mu+K(d-H\mu),\quad Q^{a}=\left(I-KH\right)Q$ (108)

Proof. Since the $D$ is gaussian and $K$ and $H$ are non-random, $U^{a}$ is
also gaussian. Gaussian distribution is uniquely specified by its mean and
covariance. Taking the mean in (107) shows that
$\mu^{a}=E\left(U_{k}^{a}\right)$. Using the independence of $U$ and $D$, the
computation from [10]

$\displaystyle E\left\langle U^{a}-\mu^{a},U^{a}-\mu^{a}\right\rangle$
$\displaystyle=E\left\langle
U-\mu+K(\left(D-d\right)-H\left(U-\mu\right)),U-\mu+K(\left(D-d\right)-H\left(U-\mu\right))\right\rangle$
$\displaystyle=E\left\langle\left(I-KH\right)\left(U-\mu\right)),\left(I-KH\right)\left(U-\mu\right))\right\rangle+E\left\langle
K\left(D-d\right),K\left(D-d\right)\right\rangle$
$\displaystyle=\left(I-KH\right)Q\left(I-KH\right)^{\top}+KRK^{\top}$
$\displaystyle=Q-KHQ-QH^{\top}K^{\top}+KHQH^{\top}K^{\top}+KRK^{\top}$
$\displaystyle=\left(I-KH\right)Q-QH^{\top}K^{\top}+K\left(HQH^{\top}+R\right)K^{\top}$
$\displaystyle=\left(I-KH\right)Q-QH^{\top}K^{\top}+QH(HQH^{\mathrm{T}}+R)^{-1}\left(HQH^{\top}+R\right)K^{\top}$
$\displaystyle=\left(I-KH\right)Q$

shows that $\operatorname{Cov}U^{a}=Q^{a}$ from (108).

Applying Theorem

###### Corollary 56

Given the forecast enemble

$U=[U_{1},\ldots,U_{N}],\quad U_{k}\sim N\left(\mu,Q\right)\text{ i.i.d.,}$

and the perturbed data ensemble

$D=[D_{1},\ldots,D_{N}],\quad D_{k}\sim N\left(d,R\right),$

independent of $U$, define $U^{a}$ by the EnKF analysis formula with the exact
covariance,

$U^{a}=U+K(D-HU).\quad K=QH^{\mathrm{T}}(HQH^{\mathrm{T}}+R)^{-1}$ (109)

Then

$U^{a}=[U_{1}^{a},\ldots,U_{N}^{a}],\quad U_{k}^{a}\sim
N\left(\mu^{a},Q^{a}\right)\text{ i.i.d}.$ (110)

###  10.4 Finite exchangeable sequences

An $N$-tuple of random elements $\left[X_{1},\ldots,X_{N}\right]$ with values
in $H\times\ldots\times H=H^{N}$ is exchangeable if their joint distribution
is invariant to a permutation of the indices; that is, for any permutation
$\pi$ of the numbers $1,\ldots,N$ and any Borel set $B\subset H^{N}$,

$\Pr\left(\left[X_{\pi\left(1\right)},\ldots,X_{\pi\left(N\right)}\right]\in
B\right)=\Pr\left(\left[X_{1},\ldots,X_{N}\right]\in B\right).$

Clearly, an i.i.d. sequence is exchangeable. The following lemmas are almost
obvious.

###### Lemma 57

If random elements $Y_{1},\ldots,Y_{N}$ are exchangeable, $Z_{1},\ldots,Z_{N}$
are exchangeable, and $Y_{1},\ldots,Y_{N}$ are independent of
$Z_{1},\ldots,Z_{N}$, then $Y_{1}+Z_{1},\ldots,Y_{N}+Z_{N}$ are exchangeable.

######  Lemma 58 ([44, Lemma 1])

If random elements $Z_{1},\ldots,Z_{N}$ are exchangeable and
$Y_{k}=F\left(Z_{1},\ldots,Z_{N},Z_{k}\right)$, where $F$ is measurable, and
permutation invariant in the first $N$ arguments, then $Y_{1},\ldots,Y_{N}$
are also exchangeable.

###  10.5 Convergence of the EnKF in the large ensemble limit

Here is a simplified presentation of the proof from [44]. Clearly, the hope is
that by a suitable law of large numbers, $Q_{N}\rightarrow Q$, as
$N\rightarrow\infty$, then $K_{N}\rightarrow K$, and, in some sense, the EnKF
analysis step (101) is asymptotically correct. The use of Slutsky’s theorem
for showing that $K_{N}\rightarrow K$ was suggested in [19]. However, the
actual formulation of the result and its proof were done only later in [39,
44]. We follow [44] here, with some simplifications discovered since [44] was
written.

Here we need to introduce the analysis cycle index $m$, which we were avoiding
so far in order not to complicate the notation. We will drop it later whenever
possible. The data assimilation process starts with an initial ensemble
$X^{\left(0\right)}$, which we assume to be gaussian i.i.d., and it proceeds
through a sequence of analysis cycle. First advance the initial ensemble by
the model (106),
$X^{\left(1\right)}=\mathcal{M}^{\left(1\right)}\left(X^{\left(0\right)}\right)$.
The analysis $X^{\left(1\right),a}$ is obtained by the use of (101) with
$X^{\left(1\right)}$ in place of $X$. In each cycle $m>1$, the ensemble is
advanced by the model (106),
$X^{\left(m\right)}=\mathcal{M}^{\left(m\right)}\left(X^{\left(m-1\right),a}\right)$
and the analysis step (101) is applied with $X^{\left(m\right)}$ in place of
$X$, giving $X^{\left(m\right),a}$. The data $d^{\left(m\right)}$ is given in
each step, and the observation matrix $H^{\left(m\right)}$ and data covariance
$R^{\left(m\right)}$ can also change from step to step.

To maintain gaussian distribution at least in the limit as
$N\rightarrow\infty$, we assume that the model in each step $m$ is linear, as
in (LABEL:eq:model), with $A=A^{\left(m\right)}$ and $b=b^{\left(m\right)}$.

Instead of proving properties of the ensembles $X^{\left(m\right)}$ alone, we
want to show that they approach the ensembles $U^{\left(m\right)}$, obtained
by using the exact state covariance $Q^{\left(m\right)}$ in each step. Thus,
we put $U^{\left(0\right)}=X^{\left(0\right)}$ (which is i.i.d. gaussian), and
define $U^{\left(m\right)}$ in the same way as $X^{\left(m\right)}$, except
that we use the exact state covariance in every step following (109), instead
of the sample covariance from the ensemble $X^{\left(m\right)}$.

We will need the property that for each $m$, the ordered pairs
$\left[X_{k}^{\left(m\right)};U_{k}^{\left(m\right)}\right]$ are identically
distributed for all $k=1,\ldots,m$. We find it convenient to write the pairs
vertically as

$\left[X_{k}^{\left(m\right)};U_{k}^{\left(m\right)}\right]=\left[\begin{array}[c]{c}X_{k}^{\left(m\right)}\\\
U_{k}^{\left(m\right)}\end{array}\right],\quad k=1,\ldots,N,$ (111)

and prove a stronger property, namely that they are exchangeable. A set of of
random elements is called _exchangeable_ if their joint distribution does not