###### Contents

  1. 1 Nonparametric affine manifolds
    1. 1.1 _Affine_ space
    2. 1.2 _Affine_ manifold
    3. 1.3 _Affine_ bundle
  2. 2 Non-parametric _statistical_ affine manifolds
    1. 2.1 _Exponential_ affine manifold
  3. 3 Banach spaces of random variables as coordinate spaces
    1. 3.1 Continuous and continuously differentiable density functions on a bounded domain of $\mathbb{R}^{d}$.
    2. 3.2 Probability densities in Orlicz spaces
  4. 4 Exponential statistical _bundles_
    1. 4.1 Maximal exponential model
    2. 4.2 Covariant derivatives, tensor bundle, acceleration
    3. 4.3 Gradients on the Tensor Bundles
    4. 4.4 Lagrangian and Hamiltonian Formalisms on the full Statistical Bundle

11institutetext: Goffredo Chirco 22institutetext: Dipartimento di Fisica
“Ettore Pancini”, Università degli Studi di Napoli Federico II, Complesso
Universitario di Monte Sant’Angelo, Via Cinthia, 21, 80126 Napoli, Italy
22email<EMAIL_ADDRESS>33institutetext: Giovanni Pistone
44institutetext: de Castro Statistics, Collegio Carlo Alberto, piazza Vincenzo
Arbarello 8, 10122 Torino, Italy 44email<EMAIL_ADDRESS>
# Dually affine Information Geometry modeled on a Banach space

Goffredo Chirco and Giovanni Pistone

###### Abstract

In this chapter, we study Information Geometry from a particular non-
parametric or functional point of view. The basic model is a probabilities
subset usually specified by regularity conditions. For example, probability
measures mutually absolutely continuous or probability densities with a given
degree of smoothness. We construct a manifold structure by giving an atlas of
charts as mappings from probabilities to a Banach space. The charts we use are
quite peculiar in that we consider only instances where the transition
mappings are affine. We chose a particular expression of the tangent and
cotangent bundles in this affine setting.

## Overview

This chapter consists of two parts. In the first part, Sec. 1 and 2, we first
present a version of the affine geometry of statistical models based on Weyl’s
axioms and notions from Inferential Statistics. Then, we present the
fundamental structures of differential geometry in terms of velocity, natural
gradient, parallel transport, acceleration. In the second part, Sec. 3, 4, we
present specific cases of model Banach spaces. Finally, we discuss the case of
Orlicz spaces of exponential growth in detail.

Our approach covers only the dually flat affine structure of non-parametric
Information Geometry. Other chapters treat approaches based on a Riemannian
structure or Fréchet spaces as coordinate’s spaces.

The use of differential geometry in the study of statistical models was first
devised by C.R. Rao rao:45  who recognized that the _Fisher Information
matrix_ fisher:26  defines a Riemannian manifold on the model parameters. It
was later recognized by B. Efron efron:1975 ; efron:1978  that the affine
geometry of the _exponential families_ provides a more interesting geometric
setup that connects with such fundamental topics as entropy, information, and
Gibbs-Boltzmann distribution. It was the work of Čenkov cenkov:1982 , and
Amari amari:85 , to reconnect the two branches in the topic we call
Information Geometry (IG). Pistone and Sempi pistone|sempi:95  first presented
the nonparametric version we describe here. A further enlargement of the scope
of IG to include geometries that depend on the geometry of the sample space is
the object of current research and not treated here..

##  1 Nonparametric affine manifolds

In this chapter, we present a non-parametric version of the Amari-Nagaoka
theory of Information Geometry (IG), see amari|nagaoka:2000 . Due to the
specific issues of the non-parametric setup, we adopt Bourbaki’s definition of
a differentiable manifold.111For nonparametric Differential Geometry, see
(bourbaki:71variete, , § 5), (lang:1995, , Chapter II),
(abraham|marsden|ratiu:1988, , Ch. 3). Before starting, we want to make a
general _caveat_. The geometric structure of interest in IG is much more
specific than the highly general one used in most Differential Geometry
textbooks. First, it is an affine geometry; second, it applies to spaces of
probability measures. For this reason, several geometric objects appear in a
specific expression. We always use a specially given set of charts and always
give a presentation compatible with the specific statistical intuition of IG.

A _chart_ on a set $M$ is a triple $(s,U,B)$, where $s$ is a 1-to-1 mapping
from its domain $U\subset M$ to an open subset $S(U)$ of a topological vector
space $B$. A _topological vector space_ is a vector space endowed with a
topology such that all the vector space operations are continuous. We will
briefly talk about top-linear mappings to mean continuous linear mappings
between topological linear spaces. Normed vector spaces and Banach spaces are
instances of topological vector spaces. In most of our applications, the
topological vector space of interest will be a Banach space of random
variables.222We will not use any sophisticated notion from Functional
Analysis. For basic reference, see (abraham|marsden|ratiu:1988, , Ch. 2),
rudin:1987-3rd

The manifold structure depends on the given atlas of charts. We repeat below
the basic definitions to stress that we do not assume the coordinate spaces to
be finite-dimensional nor require it to be the same for each chart.

######  Definition 1 ($C^{k}$-atlas)

Let $M$ be a given set and let $B_{\alpha}$, $\alpha\in A$, be a family of
Banach spaces. For each $\alpha\in$A, $(s_{\alpha},U_{\alpha},B_{\alpha})$ is
a chart on $M$, that is, $s_{\alpha}\colon U_{\alpha}\to B_{\alpha}$ is 1-to-1
from $U_{\alpha}\subset M$ to the open set $s_{\alpha}(U_{\alpha})$. We assume
that each couple of charts, say $(s_{\alpha},U_{\alpha},B_{\alpha})$ and
$(s_{\beta},U_{\beta},B_{\beta})$, either have disjoint domains,
$U_{\alpha}\cap U_{\beta}=\emptyset$, or the _transition mapping_ (or, overlap
mapping)

$s_{\beta}\circ s^{-1}_{\alpha}\colon s_{\alpha}(U_{\alpha}\cap U_{\beta})\to
s_{\beta}(U_{\alpha}\cap U_{\beta})$ (1)

is a 1-to-1 $C^{k}$ mapping between open sets. If such condition holds, the
spaces $B_{\alpha}$ and $B_{\beta}$ are isomorfic as topological vector spaces
and the set of all charts is a $C^{k}-$_atlas_ $\mathcal{A}$.

Let us stress that there is a diffeomorphism between open sets of the
corresponding spaces of coordinates for each couple of overlapping domains,
say $B_{1}$ and $B_{2}$. The implicit function theorem implies the existence
of 1-to-1 continuous linear mapping between $B_{1}$ and $B_{2}$. Hence the
assumption about the existence of a top-linear isomorphism is unavoidable.
This remark is critical in the infinite-dimensional case, and checking it will
be a constant concern for us. Moreover, we will see that the specification of
a convenient family of such isomorphisms is a way to specify an affine
structure.

######  Definition 2 ($C^{k}$-manifold)

Two atlases are equivalent if their union is again an atlas. A class of
equivalent atlases is a _manifold_.

The two well known examples below present instances of manifold in this sense
and are of special interest in IG.333See, for example, the pictures in
(abraham|marsden|ratiu:1988, , ?). Both example are related to the
$\alpha$-_embedding_ as introduced in amari|nagaoka:2000 .

######  Example 1 (The stereo-graphic projection of the unit sphere of
$L^{2}(\mu)$)

In the probability space $(\Omega,\mathcal{F},\mu)$, the unit sphere of
$L^{2}(\mu)$ with norm
$\left\|\rho\right\|_{\mu}^{2}=\int\left|\rho\right|^{2}\ d\mu$, is
$M=\left\\{\rho\in L^{2}(\mu)\,\middle|\,\left\|\rho\right\|_{\mu}=1\right\\}$
For each $\alpha\in M$, define a chart’s domain to be
$U_{\alpha}=M\setminus\left\\{-\alpha\right\\}$ and the coordinates’ space to
be the Hilbert space of random variables which are orthogonal to $\alpha$,
$B_{\alpha}=\left\\{u\in L^{2}(\mu)\,\middle|\,\left\langle
u,\alpha\right\rangle_{\mu}=0\right\\}$. The $\alpha$-chart is the stereo-
graphic projection from $-\alpha$ to the hyper-plane $B_{\alpha}+\alpha$, that
is, the vector $(s_{\alpha}(\beta)+2\alpha)$ is aligned with the vector
$\beta+\alpha$. If $u=s_{\alpha}(\beta)$, then
$u+2\alpha=\theta(\beta+\alpha$), were
$2=\theta(\left\langle\beta,\alpha\right\rangle+1)$ and
$\theta=1+\left\|u/2\right\|^{2}$. It follows

$\displaystyle
s_{\alpha}\colon\beta\mapsto\frac{2}{1+\left\langle\alpha,\beta\right\rangle}(\beta-\left\langle\alpha,\beta\right\rangle\alpha)$
(2) $\displaystyle S^{-1}_{\alpha}\colon
u\mapsto\frac{1}{1+\left\|u/2\right\|^{2}}(u+(1-\left\|u/2\right\|^{2})\alpha)$
(3)

One can check that $s_{\alpha}$, $\alpha\in M$, is an atlas of the sphere
considered as a submanifold of $L^{2}(\mu)$.

The previous example is related with one of the possible construction of the
manifold structure on probability densities. In fact, the sphere is mappend
onto the set of probability densities by the square function.

######  Example 2 (Square root embedding)

we can map the unit sphere $M$ into the affine subspace
$L^{1}_{1}(\mu)=\left\\{p\in L^{1}(\mu)\,\middle|\,\int p\ d\mu=1\right\\}$ by
$\rho\mapsto\rho^{2}$, so that the image is the set $P$ of probability
densities. Such a mapping is not 1-to-1, but nevertheless it has been used to
define a geometry on $P$. If $t\mapsto p(t)$ is a curve in the set of
probability densities, then $\rho(t)=\sqrt{p(t)}$ is a uniquely defined curve
in $M$. We could define $T_{p}(q)=s_{\sqrt{p}}(\sqrt{q})\in B_{\sqrt{p}}\,$
and $\,T^{-1}_{p}(u)=S^{-1}_{\sqrt{p}}(u)$ to get from Eq.s (2) and (3) the
mappings

$\displaystyle T_{p}\colon q\mapsto\frac{2}{1+\int\sqrt{pq}\
d\mu}\left(\sqrt{q}-\int\sqrt{pq}\ d\mu\ \sqrt{p}\,\right)$ (4) $\displaystyle
T^{-1}_{p}\colon
u\mapsto\left(\frac{2}{1+\left\|u/2\right\|^{2}}\left(u-(1-\left\|u/2\right\|^{2})\sqrt{p}\,\right)\right)^{2}$
(5)

The use of $B_{\sqrt{p}}=\left\\{u\in L^{2}(\mu)\,\middle|\,\int u\sqrt{p}\
d\mu=0\right\\}$ might seem unnatural. In fact, if $p>0$, the $u\mapsto
p^{-1/2}u$ maps $B_{\sqrt{p}}$ to $\left\\{v\in
L^{2}(\sqrt{p}\cdot\mu)\,\middle|\,\int vp\ d\mu=0\right\\}$, with the chart

$p^{1/2}s_{\sqrt{p}}(\sqrt{q})=\frac{2}{1+\int\sqrt{pq}\
d\mu}\left(\sqrt{\frac{q}{p}}-\int\sqrt{\frac{q}{p}}\ p\ d\mu\ \right)$ (6)

We do not further discuss this topic here, see the relevant chapter in this
Handbook.

###  1.1 _Affine_ space

The manifolds of interest here fall in the particular class of _affine
manifolds_. We begin with a general definition and will turn to the specific
case of statistical manifolds later on. If the reader is interested in
comparing such a general approach to our main applications, she may refer to
the example in Sec. 2.1.

The word ”affine” above refers to the geometrical construction of vectors
associated with displacement according to classical H. Weyl’s axioms of an
affine space.444See (weyl:1952, , § I.2). The L. Schwartz’ textbook
(schwartz:1981, , § III.1) has the same presentation in a slightly different
language.

Let be given a set $M$ and a real finite-dimensional vector space $V$. A
_displacement_ mapping is a mapping

$M\times M\ni(P,Q)\mapsto\overrightarrow{PQ}\in V\ ,$ (7)

such that

  1. 1.

for each fixed $P$ the partial mapping $s_{P}\colon
Q\mapsto\overrightarrow{PQ}$ is _1-to-1 and onto_ , and

  2. 2.

the _parallelogram law_ or _Chasles rule_ ,
$\overrightarrow{PQ}+\overrightarrow{QR}=\overrightarrow{PR}$, holds true.

The notation $Q=P+\overrightarrow{PQ}$ is also useful to show that the vector
space $V$ acts on the set $M$. From the parallelogram law follows that
$\overrightarrow{PP}=0$ and $\overrightarrow{PQ}+\overrightarrow{QP}=0$. The
structure $(M,V,\overrightarrow{\phantom{pq}})$ is called _affine space_. The
corresponding affine manifold is derived from the atlas of charts $s_{P}\colon
M\to V$, $P\in M$. Notice that the change-of-chart is the choice of a new
origin.555Such a structure supports a full geometrical development, see the
Nomizu and Sazaki monograph nomizu|sasaki:94 .

Because of our non-parametric perspective, we re-define the object to fit our
Def. 1 of a manifold. That is, we consider a generalization of Weyl’s axioms
that allows for a family of (possibly infinite-dimensional) top linear spaces
instead of a single finite-dimensional vector space.

######  Definition 3 (Affine space)

Let $M$ be a set and let $B_{\mu}$, $\mu\in M$, be a family of toplinear
spaces. Let $({\mathbb{U}}_{\nu}^{\mu})$, $\nu,\mu\in M$ be a family of
toplinear isomorphism ${\mathbb{U}}_{\nu}^{\mu}\colon B_{\nu}\to B_{\mu}$
satisfying the cocycle condition666Compare (lang:1995, , p. 42).

AF0

${\mathbb{U}}_{\mu}^{\mu}=I$ and
${\mathbb{U}}_{\nu}^{\rho}{\mathbb{U}}_{\mu}^{\nu}={\mathbb{U}}_{\mu}^{\rho}$.

${\mathbb{U}}_{\nu}^{\mu}$ is the _parallel transport_ from $B_{\nu}$ onto
$B_{\mu}$. Consider a _displacement_ mapping

$\operatorname{\mathbb{S}}\colon(\nu,\mu)\mapsto s_{\nu}(\mu)\in B_{\nu}$ (8)

defined on a subset of the product space
$\operatorname{dom}\left(\operatorname{\mathbb{S}}\right)\subset M\times M$.
We assume

AF1

For each fixed $\nu$ the mapping $M_{\nu}\ni\mu\mapsto
s_{\nu}(\mu)=\operatorname{\mathbb{S}}(\nu,\mu)$ is injective.

AF2

$\operatorname{\mathbb{S}}(\mu_{1},\mu_{2})+{\mathbb{U}}_{\mu_{2}}^{\mu_{1}}\operatorname{\mathbb{S}}(\mu_{2},\mu_{3})=\operatorname{\mathbb{S}}(\mu_{1},\mu_{3})$.

We will say that the structure $(M,(B_{\mu})_{\mu\in
M},({\mathbb{U}}_{\nu}^{\mu})_{\mu,\nu\in M},\operatorname{\mathbb{S}})$ is an
_affine space_.

When the vector space is constant $B_{\mu}=B$ and the parallel transport maps
are the identity, we recover Weyl’s definition. Note that in our defintion the
$s_{\mu}$ map is not required to be defined on all of $M$ and to be
surjective.

Def. 3(AF2) with $\mu_{1}=\mu_{3}=\nu$ and $\mu_{2}=\mu$ becomes

$\operatorname{\mathbb{S}}(\nu,\mu)+{\mathbb{U}}_{\mu}^{\nu}\operatorname{\mathbb{S}}(\mu,\nu)=0\
.$ (9)

Let us compute, where defined, the _change-of-origin_ map $s_{\mu}\circ
s_{\nu}^{-1}$ in an affine space. At $\rho=s_{\nu}^{-1}(w)$, $w\in B_{\nu}$,
it holds

$s_{\mu}\circ
s_{\nu}^{-1}(w)=s_{\mu}(\rho)=s_{\mu}(\nu)+{\mathbb{U}}_{\nu}^{\mu}s_{\nu}(\rho)=s_{\mu}(\nu)+{\mathbb{U}}_{\nu}^{\mu}w\
.$ (10)

The change-of-origin map exteds to a toplinear isomorphism which is an affine
map.

###  1.2 _Affine_ manifold

An affine space provides a family of candidates to charts $s_{\nu}\colon
M_{\nu}\to B_{\nu}$, $\nu\in M$, that we could use as an atlas. We want to add
a smoothness condition.

######  Definition 4 (Affine manifold)

Let $\left(M,(B_{\mu})_{\mu\in M},({\mathbb{U}}_{\nu}^{\mu})_{\mu,\nu\in
M},\operatorname{\mathbb{S}}\right)$ be an affine space as in Def. 3.

AF3

We assume that for each $\nu$, the image set $s_{\nu}(M_{\nu})$ is a
neighborhood of 0 in $B_{\mu}$.

That is, its interior $s_{\nu}(M_{\nu})^{\circ}$ is an open set containing
$s_{\nu}(\nu)=0$. Define the coordinates domains as
$U_{\nu}=s_{\nu}^{-1}\left(s_{\nu}(M)^{\circ}\right)$, so that
$(s_{\nu},U_{\nu},B_{\nu})$ is a chart on $M$. Such a chart is said to have
_origin_ $\nu$. It follows, that such charts are compatible and the resulting
manifold is by definition the _affine manifold_ associated to the given affine
space.

###### Proof

Clearly, all assumptions of Def. 1 hold true, but the fact that the domains in
eq. (1) are both open. For each couple $\mu,\nu\in M$, we have defined
$U_{\mu}=s_{\mu}^{-1}\left(s_{\mu}(M)^{\circ}\right)$ and
$U_{\nu}=s_{\nu}^{-1}\left(s_{\nu}(M)^{\circ}\right)$. Use the change-of-
origin equation (10) to see that

$s_{\nu}(U_{\nu}\cap U_{\mu})=s_{\nu}(M)^{\circ}\cap s_{\nu}\circ
s_{\mu}^{-1}(s_{\mu}(M)^{\circ})=\\\
s_{\nu}(M)^{\circ}\cap(s_{\nu}(\mu)+{\mathbb{U}}_{\mu}^{\nu}s_{\mu}(M)^{\circ})$
(11)

is open. ∎

Given the affine atlas, we can locally express the displacement mapping
$\operatorname{\mathbb{S}}$ with respect of any origin $\sigma\in M$. From the
parallelogram law for the points $\sigma,\nu,\mu$, we write

$s_{\sigma}(\nu)+{\mathbb{U}}_{\nu}^{\sigma}\operatorname{\mathbb{S}}(\nu,\mu)=s_{\sigma}(\mu)\
.$ (12)

Because of the cocyle property Def 3(AF0), we can write

$\operatorname{\mathbb{S}}(\nu,\mu)={\mathbb{U}}_{\sigma}^{\nu}\left(s_{\sigma}(\mu)-s_{\sigma}(\nu)\right)\
,$ (13)

which in turn implies the expression in the chart at $\sigma$ of the
displacement is given by

$\operatorname{\mathbb{S}}_{\sigma}(u,v)=\operatorname{\mathbb{S}}(S^{-1}_{\sigma}(u),S^{-1}_{\sigma}(v))={\mathbb{U}}_{\sigma}^{S^{-1}_{\sigma}(u)}(v-u)\
.$ (14)

The expression $\operatorname{\mathbb{S}}_{\sigma}$ is affine in the second
variable, hence the derivative777We use the notation $Df(u)[k]$ to denote the
derivative of the function $f$ in the direction $k$. Alternative notations are
$D_{h}f(u)$, $Df(u)\cdot k$, $f^{\prime}(u)k$. The notation with square
brackets is used, for example, in absil|mahony|sepulchre:2008 .

$\left.\frac{d}{dt}\operatorname{\mathbb{S}}_{\sigma}(u,v+tk)\right|_{t=0}=D_{2}\operatorname{\mathbb{S}}_{\sigma}(u,v)[k]={\mathbb{U}}_{\sigma}^{S^{-1}_{\sigma}(u)}k\
.$ (15)

Using (15) and (14), we find that the expression of the displacement solves a
differential equation, namely

$\operatorname{\mathbb{S}}_{\sigma}(u,v)=D_{2}\operatorname{\mathbb{S}}_{\sigma}(u,v)[v-u]\
.$ (16)

The previous equation provides a differential equation for the derivative in
the first variable. In fact, the first partial derivative in the direction $h$
is

$D_{1}\operatorname{\mathbb{S}}_{\sigma}(u,v)[h]=D_{1}D_{2}\operatorname{\mathbb{S}}_{\sigma}(u,v)[h,v-u]-D_{2}\operatorname{\mathbb{S}}_{\sigma}(u,v)[h]\
.$ (17)

######  Example 3 (A running example: compact sample space, continuous
positive densities)

Let $\Omega$ be a compact metric space and let $\mathcal{B}$ be its Borel
$\sigma$-algebra. We are given a reference finite measure $m$ on
$(\Omega,\mathcal{B})$. $C(\Omega)$ is the Banach space of continuous
functions on $\Omega$ with the uniform norm
$\left\|f\right\|_{\infty}=\sup_{x\in\Omega}\left|fx)\right|$. Let $M$ be the
set of _strictly positive continuous probability density_ $p$, that is,
positive funtions in $C(\Omega)$ such that $\int f\ dm=1$. It is easy to see
that $M$ an open convex subset of the affine space $B=\left\\{f\in
C(\Omega)\,\middle|\,\int f\ dm=1\right\\}$. We note that the previous set up
applies, in particular, to the case where $\Omega$ is finite and $C(\Omega)$
is the space of all real vectors with indeces in $\Omega$.

There are many interesting ways to give to the set $M$ an affine geometry. We
introduce here 3 cases to be discussed in detail in the following.

_Flat:_

As $M$ is an open convex subset of an affine space $B$, it inherits the affine
geometry of the larger space and the displacement vector is simply
$\operatorname{\mathbb{S}}(p,q)=q-p\in B_{1}$, where $B_{1}$ is the vector
space parallel to $B$, $B_{1}=\left\\{f\in C(\Omega)\,\middle|\,\int f\
dm=0\right\\}$. That is $s_{p}(q)=q-p$ and $s_{p}^{-1}(u)=u+p$ for all $u\in
B_{1}$ such that $u+p>0$.

_Mixture:_

If $B_{p}=\left\\{f\in C(\Omega)\,\middle|\,\int f\ p\cdot dm=0\right\\}$ and
$\prescript{\text{m}}{}{\mathbb{U}}_{p}^{q}u=\frac{q}{p}u$, then we can define
$s_{p}(q)=\frac{q}{p}-1$ for all $q\in M$. The parallelogram law as follows
from

$\left(\frac{q}{p}-1\right)+\frac{p}{q}\left(\frac{q}{r}-1\right)=\frac{r}{p}-1\
.$ (18)

The inverse mapping is $S^{-1}_{p}(u)=(1+u)\cdot p$ and it is defined on the
open set $U_{p}=\left\\{u\in B_{p}\,\middle|\,u>-1\right\\}$. The expression
of the displacement is $\operatorname{\mathbb{S}}_{p}(u,v)=(1+u)^{-1}(v-u)$
and the partial derivatives are
$D_{1}\operatorname{\mathbb{S}}_{p}(u,v)[h]=-(1+u)^{-2}(1+v)h$ and
$D_{2}\operatorname{\mathbb{S}}_{p}(u,v)[k]=(1+u)^{-1}k$.

_Exponential:_

With the same $B_{p}$ as above, we define
$\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}u=u-\int u\ q\cdot dm$ and define
$s_{p}(q)=\log\frac{q}{p}-\int\log\frac{q}{p}\ p\cdot dm$. The parallelogram
law follows from

$\left(\log\frac{q}{p}-\int\log\frac{q}{p}\ p\cdot dm\right)+\\\
\left(\log\frac{r}{q}-\int\log\frac{r}{q}\ p\cdot
dm-\int\left(\log\frac{r}{q}-\int\log\frac{r}{q}\ p\cdot dm\right)\
dm\right)=\\\ \left(\log\frac{r}{p}-\int\log\frac{r}{p}\ p\cdot dm\right)$
(19)

The inverse of the chart is easily seen to be

$s_{p}^{-}(u)=\exp\left(u-K_{p}(u)\right)\cdot p\ ,\quad
K_{p}(u)=\log\int\mathrm{e}^{u}\ p\ dm\ ,\quad u\in B_{p}\ .$ (20)

The expression of the displacement is

$\operatorname{\mathbb{S}}_{r}(u,v)=(v-u)-\int(v-u)\mathrm{e}^{u-K_{r}(u)}\ r\
dm\ .$ (21)

###  1.3 _Affine_ bundle

The specific form of the atlas defining the affine manifold allows the
extension of the same atlas to define an affine bundle.

######  Definition 5 (Affine bundle)

Given the affine manifold $\mathcal{M}$ of def. 4, consider the linear bundle

$SM=\left\\{(\mu,v)\,\middle|\,\mu\in M,v\in B_{\mu}\right\\}\ .$ (22)

The equation

$SM\times
SM\ni((\nu,u),(\mu,v))\mapsto(s_{\nu}(\mu),{\mathbb{U}}_{\mu}^{\nu}v)\in
B_{\nu}\times B_{\nu}$ (23)

defines a displacement on the bundle. For each $\nu$ define the chart

$s_{\nu}\colon SM\ni(\mu,v)\mapsto(s_{\nu}(\mu),{\mathbb{U}}_{\mu}^{\nu}v)\in
B_{\nu}\times B_{\nu}$ (24)

to define the affine bundle $S\mathcal{M}$ as a manifold. Equivalently, we can
say that $S\mathcal{M}$ is a linear bundle with trivialization

$s_{\nu}\colon(\mu,v)\mapsto(s_{\nu}(\mu),{\mathbb{U}}_{\mu}^{\nu}v)\ .$ (25)

The affine bundle is a convenient expression of the tangent bundle of the
affine manifold if we define the velocity as follows.

######  Definition 6 (Velocity)

The _velocity_ of the smooth curve $t\mapsto\gamma(t)$ of the affine manifold
is the curve $t\mapsto(\gamma(t),\overset{\star}{\gamma}(t))$ of the affine
bundle whose second component is

$\overset{\star}{\gamma}(t)=\lim_{h\to
0}h^{-1}(s_{\gamma(t)}(\gamma(t+h))=\left.\frac{d}{dh}s_{\gamma(t)}(\gamma(t+h))\right|_{h=0}\
.$ (26)

By (24) and Def. 3(AF2) applied to the points, the expression in the chart
centered at $\nu$ of $\overset{\star}{\gamma}(t)$ is

${\mathbb{U}}_{\gamma(t)}^{\nu}\overset{\star}{\gamma}(t)=\lim_{h\to
0}h^{-1}{\mathbb{U}}_{\gamma(t)}^{\nu}s_{\gamma(t)}(\gamma(t+h))=\\\
\lim_{h\to
0}h^{-1}\left(s_{\nu}(\gamma(t+h))-s_{\nu}(\gamma(t)\right)=\frac{d}{dt}s_{\nu}(\gamma(t))\
,$ (27)

and, conversely,

$\overset{\star}{\gamma}(t)={\mathbb{U}}_{\nu}^{\gamma(t)}\frac{d}{dt}s_{\nu}(\gamma(t))\
.$ (28)

######  Definition 7 (Integral curve and flow of a section)

Let $F$ be a section of the affine bundle, that is,
$(\mu,F(\mu))\in\mathcal{S}\mathcal{M}$. An _integral curve_ of the section
$F$ is a curve $t\mapsto\gamma(t)$ such that
$\overset{\star}{\gamma}(t)=F(\gamma(t))$. A _flow_ of the section $F$ is a
mapping

$M\times I\ni(\nu,t)\mapsto\Gamma_{t}(\nu)$ (29)

such that for each $\nu$ the curve $t\mapsto\Gamma_{t}(\nu)$ is an integral
curve and $\gamma(0,\nu)=\nu$.

######  Definition 8 (Auto-parallel curve)

A curve $I\colon t\mapsto\gamma(t)$ is _auto-parallel_ in the affine bundle if

$\overset{\star}{\gamma}(t)={\mathbb{U}}_{\gamma(s)}^{\gamma(t)}\overset{\star}{\gamma}(s)\quad
s,t\in I\ .$ (30)

###### Proposition 1

The following conditions are equivalent.

  1. 1.

The curve $\gamma$ is autoparallel.

  2. 2.

The expression of the cuve in each chart is affine.

  3. 3.

For all $s,t$

$\gamma(t)=S^{-1}_{\gamma(s)}\left((t-s)\overset{\star}{\gamma}(s)\right)\ .$
(31)

###### Proof

From eq. (30) we see that the velocity is constant in each chart,
${\mathbb{U}}_{\gamma(t)}^{\nu}\overset{\star}{\gamma}(t)={\mathbb{U}}_{\gamma(s)}^{\nu}\overset{\star}{\gamma}(s)$,
and, by (27), we have that
$\frac{d}{dt}s_{\nu}(\gamma(t)=\frac{d}{ds}s_{\nu}(\gamma(s)$. Hence $t\mapsto
s_{\nu}(t)$ is an affine curve,
$s_{\nu}(\gamma(t))-s_{\nu}(\gamma(s))=(t-s)\frac{d}{ds}s_{\nu}(\gamma(s))=(t-s){\mathbb{U}}_{\gamma(s)}^{\nu}\overset{\star}{\gamma}(s)$.
Now put $\nu=\gamma(t)$ to get
$s_{\gamma(s)}(\gamma(t))=(t-s)\overset{\star}{\gamma}(s)$ hence (31). And
conversely. ∎

###### Proposition 2

The affine bundle is an affine manifold for the displacement mapping

$((\nu,v),(\mu,w))\mapsto\left(\operatorname{\mathbb{S}}(\mu,\nu),{\mathbb{U}}_{\mu}^{\nu}w-v\right)\in
B_{\nu}\times B_{\nu}$ (32)

and the transports ${\mathbb{U}}_{\nu}^{\mu}\times{\mathbb{U}}_{\nu}^{\mu}$.

###### Proof

Check all the properties AF0–3. ∎

######  Definition 9 (Acceleration)

Consider the curve $t\mapsto\mu(t)$ with velocity
$t\mapsto\overset{\star}{\mu}(t)$. The acceleration
$t\mapsto\overset{\star\star}{\mu}(t)$ is defined as the velocity
$t\mapsto(\mu(t),\overset{\star}{\mu}(t))$.

$(\overset{\star}{\mu}(t),\overset{\star\star}{\mu}(t))=\lim_{h\to
0}h^{-1}s_{\mu(t),\overset{\star}{\mu}(t)}(\mu(t+h),\overset{\star}{\mu}(t+h))\
.$ (33)

Especially, for all $\mu\in M$,

$\overset{\star\star}{\mu}(t)={\mathbb{U}}_{\mu}^{\mu(t)}\frac{d}{dt}{\mathbb{U}}_{\mu(t)}^{\mu}\overset{\star}{\mu}(t)\
.$ (34)

From this equation, it follows that

###### Proposition 3

A curve with 0 acceleration is auto-parallel.

######  Example 4 (Running example: follows from Example 3)

If $t\mapsto p(t)$ is a curve, let us compute the velocity by implicitely
assuming enough smoothness to justify all the computations . The velocity in
the mixture manifold is

$\overset{\star}{p}(t)=\lim_{h\to
0}h^{-1}\prescript{\text{m}}{}{\mathbb{U}}_{p(t+h)}^{p(t)}\operatorname{\mathbb{S}}(p(t),p(t+h))=\\\
\lim_{h\to
0}h^{-1}\left(\frac{p(t)}{p(t+h)}\left(\frac{p(t+h)}{p(t)}-1\right)\right)=\frac{\dot{p}(t)}{p(t)}\
.$ (35)

The velocity in the exponential manifold is

$\overset{\star}{p}(t)=\lim_{h\to
0}h^{-1}\prescript{\text{e}}{}{\mathbb{U}}_{p(t+h)}^{p(t)}\operatorname{\mathbb{S}}(p(t),p(t+h))=\\\
\lim_{h\to 0}h^{-1}\left(\log\frac{p(t+h)}{p(t)}-\int\log\frac{p(t+h)}{p(t)}\
p(t)\ dm\right)=\frac{\dot{p}(t)}{p(t)}\ .$ (36)

It is remarkable that the expression of the velocity is the same in both
cases. In the statistical literature the quantity $\overset{\star}{p}(t)$ is
called the _Fisher’s score_ of the 1-dimensional model $p(t)$.888This notion
is due to R. Fisher. See, for example, the textbook efron|hastie:2016 . The
exponential velocity for a curve of the form of a Gibbs model
$p(t)\propto\mathrm{e}^{\alpha(t)U}\cdot p$, that is
$p(t)=\mathrm{e}^{\alpha(t)U-\psi(t)}\cdot p$, is

$\overset{\star}{p}(t)=\frac{d}{dt}\left(\alpha(t)U-\psi(t)\right)=\dot{\alpha}(t)U-\dot{\psi}(t)=\dot{\alpha}(t)\left(U-\int
U\ p(t)\ dm\right)\ .$ (37)

In this case, the quantity $\overset{\star}{p}(t)$ is seen as
$\dot{\alpha}(t)$ times the _fluctuation_ $\left(U-\int U\ p(t)\
dm\right)$.999See any textbook of Statistical Physiscs, for example
landau|lifshits:1980

Let us compute the acceleration in both cases. In the mixture case,

$\overset{\star\star}{p}(t)=\prescript{\text{m}}{}{\mathbb{U}}_{p}^{p(t)}\frac{d}{dt}\prescript{\text{m}}{}{\mathbb{U}}_{p(t)}^{p}\overset{\star}{p}(t)=\\\
\frac{p}{p(t)}\frac{d}{dt}\frac{p(t)}{p}\frac{\dot{p}(t)}{p(t)}=\frac{\ddot{p}(t)}{p(t)}\
.$ (38)

In the exponential case,

$\overset{\star\star}{p}(t)=\prescript{\text{e}}{}{\mathbb{U}}_{p}^{p(t)}\frac{d}{dt}\prescript{\text{e}}{}{\mathbb{U}}_{p(t)}^{p}\overset{\star}{p}(t)=\prescript{\text{e}}{}{\mathbb{U}}_{p}^{p(t)}\frac{d}{dt}\left(\frac{\dot{p}(t)}{p(t)}-\int\frac{\dot{p}(t)}{p(t)}\
p\ dm\right)=\\\
\frac{\ddot{p}(t)}{p(t)}-\left(\frac{\dot{p}(t)}{p(t)}\right)^{2}-\int\left(\frac{\ddot{p}(t)}{p(t)}-\left(\frac{\dot{p}(t)}{p(t)}\right)^{2}\right)\
p(t)\ dm=\\\
\frac{\ddot{p}(t)}{p(t)}-\left(\frac{\dot{p}(t)}{p(t)}\right)^{2}+\int\left(\frac{\dot{p}(t)}{p(t)}\right)^{2}\
p(t)\ dm\ .$ (39)

For the Gibbs model above, the exponential accelleration is proportional to
the velocity, namely

$\overset{\star\star}{p}(t)=\ddot{\alpha}(t)\left(U-\int U\ p(t)\
dm\right)=\frac{\ddot{\alpha}(t)}{\dot{\alpha}(t)}\overset{\star}{p}(t)\ .$
(40)

The auto-parallel curves in the mixture geometry are of the form
$\gamma(t)=\gamma(0)+\dot{\gamma}(0)t=(1+\overset{\star}{\gamma}(0))\gamma(0)=(1-t)\gamma(0)+t\gamma(1)$.
The last expression explains the name. In the exponential geometry, the form
of the auto-parallel curve is derived from Eq. (31),
$\gamma(t)=S^{-1}_{\gamma(0)}(t\overset{\star}{(}\gamma(0))=\mathrm{e}^{t\overset{\star}{\gamma}(0)-K_{\gamma(0)}(\overset{\star}{\gamma}(0))}\cdot\gamma(0)$,
that is, it is an exponential family. The auto-parallel interval is
$\gamma(t)\propto\gamma(0)^{1-t}\gamma(1)^{t}$.

######  Definition 10 (Duality)

Let be given two affine manifolds on the same base set $M$,
$\mathcal{M}_{i}=(M,(B^{i}_{\mu})_{\mu\in
M},(\prescript{i}{}{{\mathbb{U}}_{\nu}^{\mu}})_{\mu,\nu\in
M},\prescript{i}{}{\operatorname{\mathbb{S}}})$, $i=1,2$, and let be given for
each $\mu\in M$ a duality pairing

$B^{1}_{\mu}\times B^{2}_{\mu}\ni(u_{1},u_{2})\mapsto\left\langle
u_{1},u_{2}\right\rangle_{\mu}\ .$ (41)

The affine manifolds $\mathcal{M}_{1}$ and $M_{2}$ are in duality if for all
$\mu,\nu\in M$, $u\in B_{\mu}$, $v\in B^{2}_{\nu}$, it holds

$\left\langle
u,\prescript{2}{}{{\mathbb{U}}_{\nu}^{\mu}v}\right\rangle_{\mu}=\left\langle\prescript{1}{}{{\mathbb{U}}_{\mu}^{\nu}u},v\right\rangle_{\nu}\
.$ (42)

######  Example 5 (Duality. Follows from Examples 3 and 4)

In the present case, the mixture and the exponential fibers are equal,
$\prescript{\text{m}}{}{B}_{p}=\prescript{\text{e}}{}{B}_{p}=B_{p}$, and there
is a separating pairing $\left\langle u,v\right\rangle_{p}=\int u\ v\ p\ dm$.
The mixture affine manifold and the exponential affine manifold are actually
dual. Let us check this. For $u\in B_{p}$ and $v\in B_{q}$

$\left\langle\prescript{\text{m}}{}{\mathbb{U}}_{p}^{q}u,v\right\rangle_{q}=\int\frac{p}{q}u\
v\ q\ dm=\int u\ v\ p\ dm=\\\ \int u\left(v-\int vp\,dm\right)\ p\
dm=\left\langle
y,\prescript{\text{e}}{}{\mathbb{U}}_{q}^{p}v\right\rangle_{p}\ .$ (43)

######  Definition 11 (Gradient)

Consider a $M$ which is base of two dual affine manifolds $\mathcal{M}_{1}$
and $\mathcal{M}_{2}$. A real function $\phi$ on $\mathcal{M}_{1}$ has a
gradient $\operatorname{grad}\phi$ if $\operatorname{grad}\phi$ is a section
of the affine bundle $S\mathcal{M}_{2}$ and for all smooth curve
$t\mapsto\gamma(t)\in M$ it holds

$\frac{d}{dt}\phi(\gamma(t))=\left\langle\operatorname{grad}\phi(\gamma(t)),\overset{\star}{\gamma}(t)\right\rangle_{\sigma}\
.$ (44)

The gradient as defined above is related, but does not coincide, with the
_natural gradient_ of S.I. Amari. Let us express the gradient
$\operatorname{grad}$ in a chart with origin $\sigma$ with the ordinary
gradient $\nabla_{\sigma}$. In the 1-chart, it holds
$\phi\circ\gamma(t)=\left(\phi\circ\prescript{1}{}{S}^{-1}_{\sigma}\right)\circ\left(\prescript{1}{}{s}_{\sigma}\circ\gamma(t)\right)=\phi_{\sigma}\circ\left(\prescript{1}{}{s}_{\sigma}\circ\gamma(t)\right)$,
where $\phi_{\sigma}\colon U_{\sigma}\to\mathbb{R}$ is the expression of
$\phi$, so that

$\frac{d}{dt}\phi\circ\gamma(t)=d\phi_{\sigma}\left[\frac{d}{dt}\left(\prescript{1}{}{s}_{\sigma}\circ\gamma(t)\right)\right]=d\phi_{\sigma}\left[\prescript{1}{}{{\mathbb{U}}_{\gamma(t)}^{\sigma}}\overset{\star}{\gamma}(t)\right]=\\\
\left\langle\nabla_{\sigma}\phi_{\sigma},\prescript{1}{}{{\mathbb{U}}_{\gamma(t)}^{\sigma}}\overset{\star}{\gamma}(t)\right\rangle_{\sigma}=\left\langle\prescript{2}{}{{\mathbb{U}}_{\sigma}^{\gamma(t)}}\nabla_{\sigma}\phi_{\sigma},\overset{\star}{\gamma}(t)\right\rangle_{\gamma(t)}\
,$ (45)

where we have used (28) and $\nabla_{\sigma}$ denotes the gradient computed in
the duality of $\prescript{1}{}{B}_{\sigma}$ with
$\prescript{2}{}{B}_{\sigma}$. In conclusion, the gradient of $\phi\colon M$
equals the gradient of $\phi_{\sigma}\colon\prescript{1}{}{B}_{\sigma}$,

$\operatorname{grad}\phi(\mu)=\prescript{2}{}{{\mathbb{U}}_{\sigma}^{\mu}}\nabla_{\sigma}\phi_{\sigma}\circ\prescript{1}{}{s}_{\sigma}(\mu)=\nabla_{\mu}\phi_{\mu}(0).$
(46)

######  Example 6 (Gradient of the entropy. Follows from Examples 3,4, and 5)

The entropy is $\operatorname{\mathcal{H}}\left(q\right)=-\int q\log q\ dm$.
The expression of the entropy of $q$ at $p$ is
$\operatorname{\mathcal{H}}_{p}(v)=-\int(1+v)p\log((1+v)p)\ dm$, $v\in B_{p}$.
Let $t\mapsto v(t)$ be a smooth curve in $B_{q}$ with $v(0)=0$. We have

$\left.\frac{d}{dt}\operatorname{\mathcal{H}}_{p}(v(t))\right|_{t=0}=\left.\int(1+\log((1+v(t))p)\dot{v}(t)\
p\ dm\right|_{t=0}=\\\ \left.\left\langle-\log((1+v(t))p)+\int\log((1+v(t))p)\
dm,\dot{v}(t)\right\rangle_{p}\right|_{t=0}=\\\ \left\langle-\log
q-\operatorname{\mathcal{H}}\left(q\right),\dot{v}(0)\right\rangle_{p}\ .$
(47)

In conclusion
$\operatorname{grad}_{\text{m}}\operatorname{\mathcal{H}}\left(q\right)=-\log
q+\operatorname{\mathcal{H}}\left(q\right)$. The same result holds for
$\operatorname{grad}_{\text{e}}$.

In the exponential geometry, as
$\overset{\star}{\gamma}(t)=\frac{d}{dt}\log\gamma(t)$, the gradient flow
equation for the entropy
$\overset{\star}{\gamma}(t)=-\operatorname{grad}\operatorname{\mathcal{H}}\left(\gamma(t)\right)$
becomes

$\frac{d}{dt}\log\gamma(t)=\log\gamma(t)-\int\gamma(t)\log\gamma(t)\ dm$ (48)

If we compute the acceleration, we find the remarkable result
$\overset{\star\star}{\gamma}(t)=\overset{\star}{\gamma}(t)$. The curve
$\gamma(t)=\mathrm{e}^{a(t)v-K_{p}(v)}\cdot p$ has
$\overset{\star}{\gamma}(t)=\dot{a}(t)(v-\int v\ \gamma(t)\ dm)$, and
acceleration $\overset{\star\star}{\gamma}(t)=\ddot{a}(t)(v-\int v\ \gamma(t)\
dm)=\frac{\ddot{a}(t)}{\dot{a}(t)}\overset{\star}{\gamma}(t)$. We want
$\dot{a}(t)=\ddot{a}(t)$, that is $a(t)=c\mathrm{e}^{t}+b$.

######  Example 7 (Differentiable densities)

In this example, we discuss how to construct an affine manifold for
differentiable densities. Let $\Omega$ be a bounded domain in $\mathbb{R}^{d}$
and let $m$ denote the Lebesgue measure on $\Omega$. Let $C_{b}^{n}(\Omega)$
be the Banach space of functions which are continuous on $\overline{\Omega}$
and $n$-times differentiable on $\Omega$, with bounded partial derivatives.
The norm is
$\left\|f\right\|_{d}=\left\|f\right\|_{\infty}+\sum_{i=1}^{d}\left\|\partial_{i}f\right\|_{\infty}$.
Let $M_{1}$ be the affine subspace where $\int f\ dm=1$. Convex subset $M$ of
positive functions is an open subset of $M_{1}$. It is the set of positive
differentiable density functions on $\Omega$.

A displacement map101010This displacement appears in hivarinen:2005  based on
derivatives is given by

$\operatorname{\mathbb{S}}(p,q)=\nabla\log\frac{q}{p}=\frac{\nabla
q}{q}-\frac{\nabla p}{p}$ (49)

Let us define the fibers as

$\bm{B}_{p}=\left\\{\bm{u}\in
C_{b}^{n-1}(\Omega;\mathbb{R}^{d})\,\middle|\,\bm{u}=\nabla U,U\in
C_{b}^{n}(\Omega),\int U\ p\ dm=0\right\\}\ .$ (50)

All the fibers are equal and we can assume trivial transports. The semi-group
property is clear. Let us compute the inverse of a chart. If
$B_{p}=\left\\{U\in C_{b}^{n}(\Omega)\,\middle|\,\int U\ p\ dm=0\right\\}$
then $s_{p}(\bm{u})^{-1}=q$ if $\log\frac{q}{p}=U\in B_{p}$ and $\bm{u}=\nabla
U$. It follows that, in the notations of Example 3, we can write

$s_{p}^{-1}(\bm{u})=\mathrm{e}^{U-K_{p}(U)}\cdot p\ ,\quad\bm{u}=\nabla U\ .$
(51)

The bundle on $M$ with fibers $\bm{B}_{p}$ has an interesting inner
product,111111The first reference is otto:2001 .

$\bm{\langle}\bm{u},\bm{v}\bm{\rangle}_{p}=\int\nabla U\cdot\nabla V\ p\ dm\
.$ (52)

We do not further follow this example, but refer to the relevant chapter in
this Handbook.

##  2 Non-parametric _statistical_ affine manifolds

Let us focus now on affine manifolds whose base set is the vector space of
signed measures $\mathbb{M}$ on a given measurable space
$(\Omega,\mathcal{F})$.121212We refer to ay|jost|le|schwachhofer:2017igbook
for a full treatment of Information Geometry in $\mathcal{M}$. See the
relevant chapter in this Handbook. We refer to rudin:1987-3rd  for basic
Functional Analysis and Measure theory. More advanced textbooks are
aliprantis|border:2006  and brezis:2011fasspde  We recall that the set of
finite measures on a measurable space $(X,\mathcal{X})$ is a lattice and a
convex pointed cone.131313See (aliprantis|border:2006, , Ch. 10-11). A _signed
measure_ is the numerical difference of two finite measures,
$\mu=\mu_{1}-\mu_{2}$. There exists a unique minimal decomposition
$\mu=\mu_{+}-\mu_{-}$, the _Jordan decomposition_ , where the positive part
$\mu_{+}$ and the negative part $\mu_{-}$ have disjoint supports, that is,
$\mu_{-}\wedge\mu_{+}=0$. The measure $\left|\mu\right|=\mu_{+}+\mu_{-}$ is
the absolute value of $\mu$ and
$\mu\mapsto\left\|\mu\right\|_{\operatorname{TM}}=\int d\left|\mu\right|$ is
the _total variation norm_. The affine subspace of signed measures with total
mass $\tau$ is denoted $\mathbb{M}_{\tau}$. In particular, we are interested
with the affine subspace $\mathbb{M}_{1}$ and in the vector subspace
$\mathbb{M}_{0}$. The affine subspace $\mathbb{M}_{1}$ contains the closed
convex set of probability measures $\mathbb{P}$.

The integral induces a natural pairing on
$\mathbb{M}\times\mathcal{L}^{\infty}$, that is, $(\mu,f)\mapsto\int f\ d\mu$.
The space $\mathbb{M}_{0}$ is closed in the weak topology. We have
$\left\langle\mu,f\right\rangle\leq\left\|f\right\|_{\infty}\left\|f\right\|_{\operatorname{TM}}$,
and the weak convergence induced on $\mathcal{L}^{\infty}$ implies the point-
wise convergence.

For each curve $t\mapsto\mu(t)$ in $\mathbb{M}_{1}$ and for any topology on
$\mathbb{M}$ compatible with the operations of vector space such that
$\mathbb{M}_{0}$ is closed, it holds
$\frac{d}{dt}\mu(t)=\dot{\mu}(t)\in\mathbb{M}_{0}$ provided the curve is
differentiable at $t$, that is

$\lim_{h\to 0}\left(h^{-1}(\mu(t+h)-\mu(t))-\dot{\mu}(t)\right)=0$ (53)

.

The following special case is of high interest. Assume that the curve stays in
$\mathbb{P}$. In such a case, $\mu_{s}(A)=0$ implies that $s$ is a minimum
point of $t\mapsto\mu_{t}(A)$, hence $\dot{\mu}_{s}(A)=0$ so that the absolute
continuity $\dot{\mu}_{t}<<\mu_{t}$ holds. The _Fisher’s score_
$\overset{\star}{\mu}_{t}=\frac{d\dot{\mu}_{t}}{d\mu_{t}}\in L^{1}(\mu_{t})$
is defined for smooth statistical models. Notice that
$\int\overset{\star}{\mu}_{t}\ d\mu_{t}=\dot{\mu}_{t}(X)=0$.

For any topological vector space on $\mathbb{M}$ such the the mapping
$\mu\mapsto\int f\ d\mu=\left\langle f,\mu\right\rangle$ is continuous, the
_Fisher-Rao equation_ holds,

$\frac{d}{dt}\int f\ d\mu_{t}=\int f\ d\dot{\mu}_{t}=\int\left(f-\int f\
d\mu_{t}\right)\ d\dot{\mu}_{t}=\\\ \int\left(f-\int f\ d\mu_{t}\right)\
\frac{d\dot{\mu}_{t}}{d\mu_{t}}\ d\mu_{t}=\int\left(f-\int f\ d\mu_{t}\right)\
\overset{\star}{\mu}_{t}\ d\mu_{t}=\\\ \left\langle f-\int f\
d\mu_{t},\overset{\star}{\mu}_{t}\right\rangle_{\mu_{t}}\ .$ (54)

Notice that $\overset{\star}{\mu}_{t}\in L^{1}(\mu_{t})$, that is, possibly a
different space for each $t$.

The existence of a common dominating measure is an option to be considered,
that is, $\mu(t)=p_{t}\cdot\mu$. Assume moreover $p(t)>0$ $\mu$-almost surely.
In such a case,

$\overset{\star}{\mu}(t)=\frac{\dot{p}_{t}\cdot\mu}{p_{t}\cdot\mu}=\frac{\dot{p}_{t}}{p_{t}}=\frac{d}{dt}\log
p_{t}\quad\text{$\mu$-a.s.}$ (55)

This basic scheme has several variations in the various instances of
statistical affine spaces. A few schemes of examples, non further developed in
this chapter, are listed below. However, we will focus on the cases described
in the following sections.

######  Example 8 ($\mathbb{P}$ on a measurable space with the total variation
norm)

Let the base manifold be $\mathcal{M}=\mathbb{M}_{1}$ and let the fibers be
$B_{\mu}=\mathbb{M}_{0}$ for all $\mu\in\mathbb{P}$. In this case, the affine
structure is simply the affine structure of the affine space $\mathbb{M}_{1}$.
Recall that $\mathbb{M}$ is a Banach space for the total variation norm. This
space has even more structure. It is a _Dedekind complete Banach lattice_ for
the total variation norm.141414See (aliprantis|border:2006, , Ch. 8–9). That
is, any bounded above set (respectively, below) in the natural order of signed
measures has an upper (respectively, a lower) limit. In particular,
$\mu\vee\nu$ and $\mu\wedge\nu$ exist, and open intervals are an open set. We
define a displacement by $\operatorname{\mathbb{S}}(\nu,\mu)=\mu-\nu$, that
is, $s_{\nu}\colon\mathcal{P}\ni\mu\mapsto\mu-\nu\in M_{0}$. For each $\nu$
the mapping $\mu\mapsto S_{\nu}(\mu)$ is clearly 1-to-1 and the image is the
set of all $\xi\in\mathbb{M}_{0}$. The dual space is the space of bounded
measurable functions with the topology of bounded convergence. This space
supports many special structures, particularly the geometry of scores and an
embedded Riemannian geometry.151515This construction was done in detail in the
monograph ay|jost|le|schwachhofer:2017igbook .

######  Example 9 (Positive probability densities with the $L^{1}(m)$
topology)

Let us take $M=\left\\{\rho\in L^{1}(m)\,\middle|\,\rho\geq 0,\int\rho\
dm=1\right\\}$ and $B=L^{1}_{0}(m)$. These are subsets of case in Ex. 8. The
mapping $s_{\eta}\colon M\ni\rho\mapsto\rho-\eta\in B$ is 1-to-1. Consider
that the image of $s_{\eta}$ has empty interior in the $L^{1}(m)$ topology if
the sample space is not finite. This, in fact, is a counter-example showing
that no trivial construction is feasible in infinite dimension.

######  Example 10 (Continuous probability densities with respect to a Borel
probability measure on a compact space)

This has already been used in the previous section as an introductory example.
We assume $\Omega$ is metric and compact.161616This assumption is satisfied in
the finite state space case. A large part of the literature in non-parametric
Information Geometry is actually cased on this assumption. The tutorial
pistone:2020-NPCS  is a presentation along such lines. We consider a reference
Radon measure $m$, that is, a positive, hence continuous, linear functional on
$C(\Omega)$. We construct an affine space with base $M=\left\\{p\in
C(\Omega)\,\middle|\,p>0,\int p\ d\mu=1\right\\}$ by setting
$\operatorname{\mathbb{S}}pq=q-p$. In this case, $s_{p}\colon M\to
U_{p}=\left\\{u\in C(\Omega)\,\middle|\,u+p>0,\int u\ dm=0\right\\}$ The space
of vectors is $B=\left\\{u\in C(\Omega)\,\middle|\,\int u\ dm=0\right\\}$. We
want to show that $U_{p}$ is open in $B$. In fact, if $u\in U_{p}$, there is
an $\epsilon>0$ such that $u-\epsilon>p$, $\epsilon=\min(u+p)$. If we define
$B_{p}=\left\\{u\in C(\Omega)\,\middle|\,\int u\ p\ dm=0\right\\}$, then the
velocity is $\overset{\star}{p}_{t}=\dot{p}_{t}/p_{t}\in B_{p_{t}}$.

######  Example 11 (Arens-Eells)

Let $(\Omega,d)$ be a metric space with Borel measurable space
$(\Omega,\mathcal{B})$. Let $B$ be the vector space of all signed measures
$\xi$ of the form

$\xi=\int(\delta_{x}-\delta_{y})\ a(dx,dy)\quad\text{with $a$ a signed measure
on $\Omega\times\Omega$}\ .$ (56)

That is, $f\in\mathcal{L}^{\infty}(\mathcal{B})$,

$\int f\ d\xi=\int(f(x)-f(y))\ a(dx,dy),$ (57)

in particular, $\xi(\Omega)=0$. The Arens-Eells norm is

$\xi\mapsto\sup\left\\{\int
d(x,y)a(dx,dy)\,\,\middle|\,\,\xi=\int(\delta_{x}-\delta_{y})\
a(dx,dy)\right\\}$ (58)

is its Arens-Eells norm. Take $M$ to be a maximal set of probability measures
such that $\nu,\mu\in M$ implies $\mu-\nu\in B$.171717This is clear on a
finite state space, otherwise see (aliprantis|border:2006, , Ch. 15) and
(weaver:2018-la-2nd-ed, , § 3.1).

###  2.1 _Exponential_ affine manifold

In this section, we set the exponential affine geometry already described in
Examples 3 to 5 in a larger framework. There are many feasible choices for the
Banach spaces acting as coordinate spaces. However, not all settings will
work, and we consider it important to spell out general requirements. The
following sections will describe two specific choices of Banach spaces.

Let the base set $M$ be the set of all probability measures equivalent to a
reference $\sigma$-finite measure $m$. That is, $\mu=p\cdot m$ and $p>0$
$m$-a.s. The equivalent set of all positive $m$-desities is the maximal
possible base set. Many models we will define below apply to a smaller set of
densities.

As all the measures $p\cdot m$, $p\in M$, are equivalent, the vector spaces of
$p\cdot m$-equivalent classes of real random variables are equal,
$L^{0}(p\cdot m)=L^{0}(m)$. $L^{0}(m)$ is a topological vector space with the
convergence in $m$-measure. However, the bundle $M\times L^{0}(m)$ seems too
big to support an exponential geometric structure because the random variable
$U$ and $\mathrm{e}^{U}$ will not always be $m$-integrable unless the state
space is finite.

Let us assume that $M$ is a set of positive $m$-probability densities possibly
smaller than the maximal one. Each probability density $p\in M$ defines its
own Banach space of integrable random variables $L^{1}(p)=L^{1}(p\cdot m)$. In
general, the spaces are not equal for different densities. A suffient
condition for $L^{1}(p)=L^{1}(q)$ for all of all $p,q\in M$, is that the
density ratio is bounded above and below for all couples, $k\leq q/p\leq K$
for some $0<k\leq K$. In fact,

$\int\left|f\right|\ q\ dm=\int\left|f\right|\frac{q}{p}\ p\ dm\leq
K\int\left|f\right|\ p\ dm\ .$ (59)

We want a set of random variables that are integrable for all densities in our
base set $M$. That is, we look for a topological vector space $B$ of
$m$-classes of random variables such that

$B\hookrightarrow\cap_{p\in M}L^{1}(p)\hookrightarrow L^{0}(m)\ .$ (60)

######  Example 12 (Compact sample space, continuous densities)

If $\Omega$ is compact and $p,q$ are assumed to be continuous, then a bound in
Eq. (59) always exists.

######  Example 13 (Bounded random variables)

Clearly, that $\cap_{p\in M}L^{1}(p)=L^{\infty}(m)$, so that one could
restrict the attention to bounded random variable only. This does not seem to
produce a model with sufficient applicability. Frequntly, useful random
variables are unbounded.

Given a vector space $B$ satisfying Eq. (60), we define the family of spaces

$B_{p}=\left\\{u\in B\,\middle|\,\int u\ p\ dm=0\right\\}$ (61)

together with the transports

$\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}\colon B_{p}\ni\mapsto
u-{\operatorname{\mathbb{E}}}_{q}\left[u\right]\in B_{q}\ .$ (62)

Notice that the transports compose correctly as a cocycle,

$\prescript{\text{e}}{}{\mathbb{U}}_{p}^{p}u=u\quad\text{and}\quad\prescript{\text{e}}{}{\mathbb{U}}_{p}^{r}=\prescript{\text{e}}{}{\mathbb{U}}_{q}^{r}\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}$
(63)

Moreover, we assume $M$ is such that for each couple $p,q\in M$ the
$\log$-ratio is well defined in $B$, $\log\frac{q}{p}\in B$. An _affine space_
is now defined by the displacement mapping

$\operatorname{\mathbb{S}}\colon(p,q)\mapsto\log\frac{q}{p}-{\operatorname{\mathbb{E}}}_{p}\left[\log\frac{q}{p}\right]\in
B_{p}\subset L^{1}_{0}(p)\ .$ (64)

In fact, the parallelogram law holds,

$\left(\log\frac{q}{p}-{\operatorname{\mathbb{E}}}_{p}\left[\log\frac{q}{p}\right]\right)+\prescript{\text{e}}{}{\mathbb{U}}_{q}^{p}\left(\log\frac{r}{q}-{\operatorname{\mathbb{E}}}_{q}\left[\log\frac{r}{q}\right]\right)=\\\
\log\frac{q}{p}-{\operatorname{\mathbb{E}}}_{p}\left[\log\frac{q}{p}\right]+\log\frac{r}{p}-\log\frac{q}{p}-{\operatorname{\mathbb{E}}}_{p}\left[\log\frac{r}{p}-\log\frac{q}{p}\right]\\\
\log\frac{r}{p}-{\operatorname{\mathbb{E}}}_{p}\left[\log\frac{r}{p}\right]$
(65)

Finally, we want to check that the charts

$s_{p}\colon M\ni
q\mapsto\log\frac{q}{p}-{\operatorname{\mathbb{E}}}_{p}\left[\log\frac{q}{p}\right]$
(66)

with

$s_{p}^{-1}\colon u\mapsto\mathrm{e}^{u-K_{p}(u)}\cdot p\ ,\quad
K_{p}(u)={\operatorname{\mathbb{E}}}_{p}\left[\mathrm{e}^{u}\right]\ .$ (67)

are such that the image of each $s_{p}$ is a neighborhood of 0 in $B_{p}$.

The velocity (26) of the curve

$t\mapsto p(t)=\mathrm{e}^{u(t)-K_{p}(u(t))}\cdot p$ (68)

is

$\overset{\star}{p}(t)=\frac{d}{dt}\log
p(t)=\dot{(}u(t))-\frac{d}{dt}K(u(t))=\prescript{\text{e}}{}{\mathbb{U}}_{p}^{p(t)}\dot{u}(t)\
,$ (69)

where the last equality follows from a well-known property of the derivative
of the cumulant generating function.181818A classical reference for
exponential families is brown:86

The auto-parallel curves are the the exponential families

$t\mapsto\mathrm{e}^{tu-K_{p}(tu)}\cdot p\quad u\in B_{p}\ .$ (70)

Let us compute the accelleration with Eq. (34)

$\overset{\star\star}{p}(t)=\prescript{\text{e}}{}{\mathbb{U}}_{p}^{p(t)}\frac{d}{dt}\prescript{\text{e}}{}{\mathbb{U}}_{p(t)}^{p}\overset{\star}{p}(t)=\\\
\prescript{\text{e}}{}{\mathbb{U}}_{p}^{p(t)}\frac{d}{dt}\left(\frac{\dot{p}(t)}{p(t)}-\int\frac{\dot{p}(t)}{p(t)}\
p\ bm\right)=\\\
\prescript{\text{e}}{}{\mathbb{U}}_{p}^{p(t)}\left(\frac{\ddot{p}(t)}{p(t)}-\left(\frac{\dot{p}(t)}{p(t)}\right)^{2}-\int\frac{\ddot{p}(t)}{p(t)}-\left(\frac{\dot{p}(t)}{p(t)}\right)^{2}\
p\ dm\right)=\\\
\frac{\ddot{p}(t)}{p(t)}-\left(\frac{\dot{p}(t)}{p(t)}\right)^{2}+\int\left(\frac{\dot{p}(t)}{p(t)}\right)^{2}\
p(t)\ dm\ .$ (71)

###### Example 14

We have seen that $\overset{\star\star}{p}(t)=0$ implies $p(t)=\mathrm{e}^{tu-
K_{p}(tu)}\cdot p$, $p(0)=p$, $u\in B_{p}$. In a different time scale,
$q(t)=\exp(a(t)v-K_{p}(a(t)v)\cdot p$, $\dot{a}>0$, $v\in B_{p}$. We have

$\overset{\star}{q}(t)=\dot{a}(t)\left(v-\int v\ q(t)\
dm\right)\quad\overset{\star\star}{q}(t)=\ddot{a}(t)\left(v-\int v\ q(t)\
dm\right)$ (72)

so that

$\overset{\star\star}{q}(t)=\frac{\ddot{a}(t)}{\dot{a}(t)}\overset{\star}{q}(t)=\frac{d}{dt}\log\dot{a}(t)\overset{\star}{q}(t)\
.$ (73)

In particular, $a(t)=-1/t$, yields the equation
$t\overset{\star\star}{q}(t)+2\overset{\star}{q}(t)=0$.191919A classical
reference for Boltzmann-Gibbs is landau|lifshits:1980

##  3 Banach spaces of random variables as coordinate spaces

In our definition of affine statistical manifold we consider a set of
probability measures $M$ and, for each $\nu\in M$, a mapping $s_{\nu}\colon
M\to B_{\nu}$, where $B_{\nu}$ is a topological vector space, a _top-linear
space_. The specific needs of the modeling dictate the choice of the
displacement map, the only restriction being the parallelogram law. The choice
of the family of top-linear spaces $B_{\nu}$ and the family of parallel
transport ${\mathbb{U}}_{\nu}^{\mu}$, $\nu,\mu\in M$, could be challenging.
There are two topological requirements:

  1. 1.

${\mathbb{U}}_{\nu}^{\mu}$ is an isomorphism of $B\nu$ onto $B_{\mu}$, and

  2. 2.

The set image of $M$ with $s_{\nu}$, that is the set of all coordinates, is
open in $B_{\nu}$.

The previous requirements are strict, but nevertheless there is a wide choice
of possible set-ups. The number of possible variations is too large to provide
an exhaustive list. As it is in other fields, Functional Analysis provides a
toolbox of methods to adapt to each specific case. For convenience of the
reader and for future reference, we offer in Table 1 of the displacementn maps
we have introduced in the previous sections.

chart | $s_{p}(q)$ | $S^{-1}(u)$
---|---|---
mixture | $u=\frac{q}{p-1}$ | $q=(1+u)\cdot p$
exponential |  $u=\log\frac{q}{p}-\int\log\frac{q}{p}\,p\,dm$ | $q=\mathrm{e}^{u-K_{p}(u)}\cdot p$
Hyvärinen | ${\bm{u}}=\nabla\log\frac{q}{p}$ |  $q=\mathrm{e}^{U-K_{p}(U)}\cdot p\quad{\bm{u}}=\nabla U$
Table 1: Cases of affine charts

We will discuss two instances of the base set $M$.

§3.1

Continuous and continuously differentiable probability functions on an open
real domain.

§3.2

Densities in Orlicz spaces.

###  3.1 Continuous and continuously differentiable density functions on a
bounded domain of $\mathbb{R}^{d}$.

This is actually the simplest case and we have already presented it in
Examples 3, 4, 5. In applications where one can assume bounded densities for a
standard reference measure, one can focus on specific classes of bounded
functions, such as bounded measurable functions or continuous functions on a
compact metric space. However, we note that such a setup is not suitable in
many statistical applications. For example, Gaussian probability measures on
$\mathbb{R}^{d}$ do not transform via bounded factors. However, there is a
considerable scope of application in cases where the sample space is naturally
bounded, for example, in some applications of Statistical Physics or Data
Science. Specifically, we will restrict our discussion to real bounded domains
and continuous densities or continuously differentiable densities.

Let $\Omega$ be an open bounded domain of $\mathbb{R}^{d}$. Let $m$ be a
probability measure on $\Omega$ such that $\int f\ dm=0$ implies $f=0$ for all
continuous non-negative $f$. For example, the uniform probability measure
$\int f\ dm=\int_{\Omega}f(x)\ dx/\int_{\Omega}dx$ will work, while a finite
mixture of Dirac measures will not.

Let us take as model space the vector space $B_{1}=\left\\{u\in
C^{k}_{b}(\Omega)\,\middle|\,\int{u}\,f\ dm=0\right\\}$, where $k$ is a given
non-negative integer. As $B_{1}$ is a closed vector subspace of
$C^{k}_{b}(\Omega)$, it is a Banach space for the restriction of the uniform
norm of $C^{k}_{b}(\Omega)$. We can define the the mixture and the exponential
displacement of table 1. The base set is the set of $C^{k}_{b}(\Omega)$ and
positive densities, that is,

$M=\left\\{p\in C^{k}_{b}(\Omega)\,\middle|\,p(x)>0,x\in\overline{\Omega},\int
p\ dm=1\right\\}\ .$ (74)

The fibers are $B_{p}=\left\\{u\in C^{k}_{b}(\Omega)\,\middle|\,\int u\ p\
dm=0\right\\}$. All the check of the properties of affine manifold are easy
and some depend on the fact every $f\in C^{1}_{b}(\Omega)$, $f>0$, is bounded
below by a positive constant.

Let us consider the Hyvärinen chart. Consider the case $k=2$. For each couple
$q,p$ of $C^{2}_{b}(\Omega)$ and positive densities, the vector random
variable $\bm{u}=\nabla\log\frac{q}{p}$ belongs to
$C^{1}_{b}(\Omega;\mathbb{R}^{d})$. If $\nabla U=\bm{u}$ and $\int U\ p\
dm=0$, then $q=\mathrm{e}^{U-K_{p}(U)}\cdot p$.

###  3.2 Probability densities in Orlicz spaces

In the theory of Lebesgue spaces, one evaluates the integrated value of a
function $f$ in some power scale $\left|f\right|^{\alpha}$,
$1\leq\alpha<\infty$. Then, one computes the norm as
$\left\|f\right\|_{\alpha}=\left(\int\left|f\right|^{\alpha}\right)^{1/\alpha}$.
In Orlicz spaces, we use a more general scale $\Phi(f)$, for example,
$\Phi(f)=\cosh f-1$. The presentation below is not the most general possible,
but nevertheless it covers all the examples in our scope.202020The monograph
by Adams and Fournier (adams|fournier:2003, , Ch. 8) and the Musielak
monograph (musielak:1983, , Ch. I-II) provide basic references to this topic.

If $\phi\in C[0,+\infty[$ satisfies:

  1. 1.

$\phi(0)=0$,

  2. 2.

$\phi$ is strictly increasing, and

  3. 3.

$\lim_{u\to+\infty}\phi(u)=+\infty$,

its primitive function

$\Phi(x)=\int_{0}^{x}\phi(u)\ du\ ,\quad x\geq 0\ ,$ (75)

is strictly convex and a diffeomorphism of $]0,\infty[$. The function $\Phi$
is extended to $\mathbb{R}$ by symmetry, $\Phi(x)=\Phi(\left|x\right|)$ and it
is called _Young function_.212121Notice that $\left|x\right|^{\alpha}$,
$\alpha>1$ is included in our definition, while $\left|x\right|$ is not. If
more generality is needed, see (adams|fournier:2003, , §8.2) and
(musielak:1983, , §7.1).

The inverse function $\psi=\phi^{-1}$ has the same properties 1) to 3) as
$\phi$, so that its primitive

$\Psi(y)=\int_{0}^{y}\psi(v)\ dv\ ,\quad y\geq 0\ ,$ (76)

is again a Young function. The couple $(\Phi,\Psi)$, is a couple of
_conjugate_ Young functions. The relation is symmetric and we write both
$\Psi=\Phi^{*}$ and $\Phi=\Psi^{*}$.

The following properties are easy to check. The _Young inequality_ holds true,

$\Phi(x)+\Psi(y)\geq xy\ ,\quad x,y\geq 0\ ,$ (77)

and the _Legendre equality_ holds true ,

$\Phi(x)+\Psi(\phi(x))=x\phi(x)\ ,\quad x\geq 0\ ,$ (78)

that is, the _Legendre transform_ coincides with the convex conjugate,

$\Psi(y)=xy-\Phi(\phi^{-1}(y))=\inf_{x}\left(xy-\Phi(x)\right)\ ,$ (79)

Table 2 collects the examples we are going to use in the
following.222222Indeed, the previous theory is just a very special case of
convex duality, cf. (ekeland|temam:1999convex2nd, , Ch. I).

$\Phi=\Psi_{*}$ |  $\Psi=\Phi_{*}$
---|---
$x^{\alpha}/\alpha$ |  $\int_{0}^{y}v^{1/(\alpha-1)}\ dv=y^{\beta}/\beta\ ,\quad 1/\alpha+1/\beta=1$
$\exp_{2}(x)=\mathrm{e}^{x}-1-x$ |  $(\exp_{2})_{*}(y)=\int_{0}^{y}\log(1+v)\ dv=(1+y)\log(1+y)-y$
$\cosh_{2}(x)=\cosh x-1$ |  $(\cosh_{2})_{*}(y)=\int_{0}^{y}\sinh^{-1}(v)\ dv=y\sinh^{-1}y-\sqrt{1+y^{2}}$
$\operatorname{gauss}_{(x)}=\exp\left(\frac{1}{2}x^{2}\right)-1$ |  no closed form
Table 2: Examples of Young functions. In the table first line
$\alpha,\beta>1$.

Given a probability space $(X,\mathcal{X},\mu)$, we denote by $L^{0}(\mu)$ the
space of $\mu$-classes of real random variables.

###### Definition 12

Given a Young function $\Phi$ and a probability measure $\mu$, the _Orlicz
space_ $L_{\Phi}\left(\mu\right)$ is the vector subspace of $f\in L^{0}(\mu)$
such that $\int\Phi(\rho^{-1}f)\ d\mu$ is finite for some $\rho>0$.

###### Proposition 4

$L_{\Phi}\left(\mu\right)$ is a Banach space for the norm whose closed unit
ball is $\left\\{f\in L^{0}(\mu)\,\middle|\,\int\Phi(\left|f\right|)\ d\mu\leq
1\right\\}$.

The vector space property is easy to check by using the convexity of the Young
function $\Phi$. The norm implicitly defined above is called _Luxemburg norm_
,. Explicitly,

$\left\|f\right\|_{L_{\Phi}\left(\mu\right)}\leq\rho\quad\text{if, and only
if,}\quad\int\Phi(\rho^{-1}\left|f\right|)\ d\mu\leq 1\ ,$ (80)

that is,

$\left\|f\right\|_{L_{\Phi}\left(\mu\right)}=\inf\left\\{\rho\,\middle|\,\int\Phi(\rho^{-1}\left|f\right|)\
d\mu\leq 1\right\\}\ .$ (81)

We refer to the standard monographs on Orlicz spaces for detailed proofs o the
proposition above.232323See (adams|fournier:2003, , § 8.7-11) and
(musielak:1983, , §I.1). See a proof of completeness in (musielak:1983, , Th.
7.7). The next remark list some special features of this class of Banach
spaces.

###### Remark 1

  1. 1.

If $c$ is constant function, then
$\left\|c\right\|_{L_{\Phi}\left(\mu\right)}=c$ if, and only if $\Phi(1)=1$,
which is the case for power functions, but is not the case for the other
examples in the table.

  2. 2.

In the case $\Phi(x)=\left|x\right|^{\alpha}$, $1<\alpha<\infty$, the
Luxemburg norm equals the Lebesgue norm. If
$\Phi(x)=\alpha^{-1}\left|x\right|^{\alpha}$, then the Luxemburg norm equals
$\alpha^{-1/\alpha}$ $\times$ the Lebesgue norm.

  3. 3.

We have assumed the reference measure $\mu$ to be a probability measure. This
is nor required in the general theory of Orlicz spaces. It is a specific
feature of the application we a looking for.

  4. 4.

The convergence of a sequence $(f_{n})$ to zero in $L_{\Phi}\left(\mu\right)$,
that is $\lim_{n\to\infty
0}\left\|f_{n}\right\|_{L_{\Phi}\left(\mu\right)}=0$, is not equivalent
$\lim_{n\to\infty}\int\Phi(f_{n})\ d\mu=0$. In fact, it is required that, for
all $\epsilon>0$, it holds
$\left\|\epsilon^{-1}f_{n}\right\|_{L_{\Phi}\left(\mu\right)}\leq 1$
definitively. The condition of norm convergence in terms of integrals is

$\int\Phi(\epsilon^{-1}f_{n})\ d\mu\leq 1\quad\text{definitively for all
$\epsilon>0$.}$ (82)

Now, for all $0<\lambda<1$ it holds $\Phi(\lambda x)\leq\lambda\Phi(x)$, so
that

$\int\Phi(\epsilon^{-1}f_{n})\
d\mu\leq\lambda\int\Phi((\lambda\epsilon)^{-1}f_{n})\
d\mu\leq\lambda\quad\text{definitively for all $\epsilon>0$.}$ (83)

In conclusion,

$\lim_{n\to\infty
0}\left\|f_{n}\right\|_{L_{\Phi}\left(\mu\right)}=0\quad\Leftrightarrow\quad\lim_{n\to\infty}\int\Phi(\epsilon^{-1}f_{n})\
d\mu=0\ ,\quad\epsilon>0\ .$ (84)

  5. 5.

If a growth condition of the form $\Phi(ax)\leq C(a)\Phi(x)$, $a>0$, holds,
then the condition $\lim_{n\to\infty}\int\Phi(f_{n})\ d\mu=0$ clearly implies
(84). This is the case of the power functions
$\left|ax\right|^{\alpha}=a^{\alpha}\left|x\right|^{\alpha}$. This is not the
case for $\exp_{2}$ of Table 2 because $\exp_{2}(cx)/\exp_{x}(x)$ is unbouded
for $x\geq 0$. This issue is important for the duality between conjugate
spaces, see below.

For each couple of conjugate Young function $\Phi$ and $\Psi=\Phi^{*}$ we have
a couple of conjugate Orlicz spaces with a duality pairing. In fact,
integration of the Young inequality (77) gives

$\int\left|uv\right|\ d\mu\leq\int\Phi(\left|u\right|)\
d\mu+\int\Psi(\left|v\right|)\ d\mu\ .$ (85)

The duality pairing is

$L_{\Phi}\left(\mu\right)\times
L_{\Psi}\left(\mu\right)\ni(u,v)\mapsto\left\langle
u,v\right\rangle_{\mu}=\int uv\ d\mu\ .$ (86)

If the norms of $u$ and $v$ in (85) are both 1, the LHSXSXS is bounded by 2,
that is,

$\left\langle u,v\right\rangle_{\mu}\leq
2\left\|u\right\|_{L_{\Phi}\left(\mu\right)}\left\|v\right\|_{L_{\Phi_{*}}\left(\mu\right)}\
.$ (87)

Each element of an Orlicz space is associated via the duality pairing (86) to
a linear continuous functional of the conjugate. However, an Orlicz space is
the dual Banach space of of its conjugate in particular cases only, see below.
However, an equivalent norm can be defined via the duality pairing, namely,
the _Orlicz norm_ ,

$\left\|f\right\|_{{L_{\Psi}\left(\mu\right)}^{*}}=\sup\left\\{\left\langle
f,g\right\rangle_{\mu}\,\middle|\,\left\|f\right\|_{L_{\Psi}\left(\mu\right)}\leq
1\right\\}\ .$ (88)

By bounding the pairing with (87), we see that
$\left\|f\right\|_{{L_{\Phi}\left(\mu\right)}^{*}}\leq
2\left\|f\right\|_{L_{\Psi}\left(\mu\right)}$. Because of this inequality,
(88) defines a continuous norm on $L_{\Psi}\left(\mu\right)$ and $\left\langle
f,g\right\rangle_{\mu}\leq\left\|f\right\|_{L_{\Psi}\left(\mu\right)^{*}}\left\|g\right\|_{L_{\Psi}\left(\mu\right)}$.
More, the Luxembourg norm and the Orlicz norm are actually equivalent. Let us
show that $\left\|f\right\|_{L_{\Phi}\left(\mu\right)^{*}}\leq 1$ whenever
$\left\|f\right\|_{L_{\Psi}\left(\mu\right)^{*}}\leq 1$. The conjugation
relation extends to integrals,

$\int\Phi(f)\ d\mu=\sup\left\\{\left\langle
f,g\right\rangle_{\mu}-\int\Psi(g)\ d\mu\,\middle|\,g\in
L_{\Psi}\left(\mu\right)\right\\}\ .$ (89)

but we can compute the $\sup$ on a smaller set because

$\sup\left\\{\left\langle f,g\right\rangle_{\mu}-\int\Psi(g)\
d\mu\,\middle|\,\int\Psi(g)d\mu>1\right\\}\leq\\\
\sup\left\\{\left\|g\right\|_{L_{\Psi}\left(\mu\right)}-\int\Psi(g)\
d\mu\,\middle|\,\int\Psi(g)d\mu>1\right\\}\leq 0\ .$ (90)

With that, (89) becomes

$\int\Phi(f)\ d\mu=\sup\left\\{\left\langle
f,g\right\rangle_{\mu}-\int\Psi(g)d\mu\,\middle|\,\left\|f\right\|_{L_{\Psi}\left(\mu\right)}\leq
1\right\\}\leq 1$ (91)

and the bound is proved.

Other equivalent norms are of interest and will be discussed later in specific
instances of the Young function or the base measure $\mu$.

The domination relation between Young functions imply continuous injection
properties for the corresponding Orlicz spaces. We will say that $\Phi_{2}$
_eventually dominates_ $\Phi_{1}$, written $\Phi_{1}\prec\Phi_{2}$, if there
are positive constants $a,b$ and a non-negative $\bar{x}$ such that
$\Phi_{1}(x)\leq a\Phi_{2}(bx)$ for all $x\geq\bar{x}$. As, in our case, $\mu$
is a probability measure, the continuous embedding
$L_{\Phi_{2}}\left(\mu\right)\to L_{\Phi_{1}}\left(\mu\right)$ holds if, and
only if, $\Phi_{1}\prec\Phi_{2}$. If $\Phi_{1}\prec\Phi_{2}$, then
$(\Phi_{2})_{*}\prec(\Phi_{1})_{*}$. See (adams|fournier:2003, , Th. 8.12) or
(musielak:1983, , Th. 8.5).

When there exists a function $C$ such that $\Psi(ax)\leq C(a)\Psi(x)$ for all
$a\geq 0$. In such a case, the conjugate $L_{\Psi^{*}}\left(\mu\right)$ is the
dual Banach space of $L_{\Psi}\left(\mu\right)$, and bounded functions are a
dense set.242424We do not develop this classical topic here because it is not
much relevant in our application to IG. See (adams|fournier:2003, , §8.17-20).

We now move to the discussion of the examples to be used in the one version of
affine statistical manifold, that is, the cases from Tab. 2.

The spaces corresponding to the power functions coincide with the ordinary
Lebesgue spaces. The norm are related by

$\left\|f\right\|_{L_{\Phi_{\alpha}}\left(\mu\right)}=\alpha^{1/\alpha}\left\|f\right\|_{L^{\alpha}(\mu)}\
.$ (92)

The embedding conditions hold. The spaces are dual of each other.

The Young function $\exp_{2}$ and $\cosh_{2}$ are equivalent and the Orlicz
spaces are isomorphic equal as vector spaces and isomorphic,
$L_{\cosh_{2}}\left(\mu\right)\leftrightarrow L_{\exp_{2}}\left(\mu\right)$.
This example is of special interest for us as they provide the model spaces
for a non-parametric version of Information Geometry, see Sec. 4 below. They
both are eventually dominated by $\operatorname{gauss}_{2}$ and eventually
dominate all powers, that is,

$L^{\infty}(\mu)\hookrightarrow
L_{\operatorname{gauss}_{2}}\left(\mu\right)\hookrightarrow
L_{\exp_{2}}\left(\mu\right)\approx
L_{\cosh_{2}}\left(\mu\right)\hookrightarrow\\\ L^{\alpha}(\mu)\hookrightarrow
L^{2}(\mu)\hookrightarrow L^{\beta}(\mu)\\\ \hookrightarrow
L_{\exp_{2}^{*}}\left(\mu\right)\approx
L_{\cosh_{2}^{*}}\left(\mu\right)\hookrightarrow
L_{\operatorname{gauss}_{2}^{*}}\left(\mu\right)\hookrightarrow L^{1}(\mu)\ ,$
(93)

where $\alpha>2$ and $1<\beta<2$ are conjugate, $1/\alpha+1/\beta=1$. Each
space at the left of $L^{2}(\mu)$ is the dual of one space at the right.

The Orlicz space $L_{\exp_{2}}\left(\mu\right)=L_{\cosh_{2}}\left(\mu\right)$
is actually known with different many names is various chapters of Statistics.
The proposition below provides equivalent definitions.

###### Proposition 5

  1. 1.

A function belongs to the space $L_{\cosh_{2}}\left(\mu\right)$ if, and only
if, its _moment generating function $\lambda\mapsto\int\mathrm{e}^{\lambda f}$
is finite in a neighborhood of 0_. In turn, this implies that the moment
generating function is analytic at 0.252525See (brown:86, , Ch. 2).

  2. 2.

The same property is equivalent to a large deviation inequality.262626See
(wainwright:2019-hds, , Ch. 2). Precisely, a function $f$ belongs to
$L_{\cosh_{2}}\left(\mu\right)$ if, and only if, it is _sub-exponential_ ,
that is, there exist constants $C_{1},C_{2}>0$ such that

$\mu(\left|f\right|\geq t)\leq C_{1}\exp\left(-C_{2}t\right)\ ,\quad t\geq 0\
.$ (94)

###### Proof

The first statement is immediate. If
$\left\|f\right\|_{L_{\cosh_{2}}\left(\mu\right)}=\rho$, then
$\int\mathrm{e}^{\rho^{-1}\left|f\right|}\ d\mu\leq 4$. It follows that

$\mu(\left|f\right|>t)=\mu\left(\mathrm{e}^{\rho^{-1}\left|f\right|}>\mathrm{e}^{\rho^{-1}t}\right)\leq\left(\int\mathrm{e}^{\rho^{-1}\left|f\right|}\
d\mu\right)\mathrm{e}^{-\rho^{-1}t}\leq 4\mathrm{e}^{-\rho^{-1}t}\ .$ (95)

The sub-exponential inequality holds with $C_{1}=4$ and
$C_{2}=\left\|f\right\|_{L_{\cosh_{2}}\left(\mu\right)}^{-1}$. Conversely, for
all $\lambda>0$,

$\int\mathrm{e}^{\lambda f}\
d\mu\leq\int_{1}^{\infty}\mu\left(\mathrm{e}^{\lambda f^{+}}>t\right)\ dt\leq
C_{1}\int_{0}^{\infty}\mathrm{e}^{-(C_{2}\lambda^{-1}-1)s}\ ds\ .$ (96)

The right-hand side is finite if $\lambda<C_{2}$ and the same bound holds for
$-f$. ∎

###### Remark 2

  1. 1.

A sub-exponential random variable is of particular interest in applications
because they admit an explicit exponential bound in the Law of Large Numbers.
Another class of interest consists of the _sub-Gaussian_ random variables,
that is, those random variables whose square is sub-exponential.272727See
wainwright:2019-hds .

  2. 2.

The theory of sub-exponential random variables provides an _equivalent norm
for the space_ $L_{\cosh_{2}}\left(\mu\right)$.282828See
buldygin|kozachenko:2000  or siri|trivellato:2021-SPL . The norm is

$f\mapsto\sup_{k}\left((2k)!^{-1}\int f^{2k}\
d\mu\right)^{1/2k}=\left\bracevert f\right\bracevert_{\cosh_{2}}\ .$ (97)

Let us prove the equivalence. If
$\left\|f\right\|_{L_{\cosh_{2}}\left(\mu\right)}\leq 1$, then

$1\geq\int\cosh_{2}f\ d\mu\geq\frac{1}{(2k)!}\int f^{2k}\ d\mu\quad\text{for
all $k=1,2,\dots$,}$ (98)

so that $1\geq\left\bracevert f\right\bracevert_{\cosh_{2}}$. Conversely, if
the latter inequality holds, then

$\int\cosh_{2}(f/\sqrt{2})\ d\mu=\sum_{k=1}^{\infty}\frac{1}{(2k)!}\int
f^{2k}\ d\mu\left(\frac{1}{2}\right)^{k}\leq 1\ ,$ (99)

so that $\left\|f\right\|_{L_{\cosh_{2}}\left(\mu\right)}\leq\sqrt{2}$.

  3. 3.

It is covenient to introduce a further notation. For each Young function
$\Phi$, the function $\overline{\Phi}(x)=\Phi(x^{2})$ is again a Young
function such that
$\left\|f\right\|_{L_{\overline{\Phi}}\left(\mu\right)}\leq\lambda$ if, and
only if,
$\left\|\left|f\right|^{2}\right\|_{L_{\Phi}\left(\mu\right)}\leq\lambda^{2}$.
_We will denote the resulting space by_ $L_{\Phi}^{2}\left(\mu\right)$. For
example, $\operatorname{gauss}_{2}$ and $\overline{\cosh_{2}}$ are
$\prec$-equivalent , hence the isomorphisn
$L_{\operatorname{gauss}_{2}}\left(\mu\right)\leftrightarrow
L_{\cosh_{2}}^{2}\left(\mu\right)$. As an application of this notation,
consider that for each increasing convex $\Phi$ it holds
$\Phi(fg)\leq\Phi((f^{2}+g^{2})/2)\leq(\Phi(f^{2})+\Phi(g^{2}))/2$. It follows
that when the $L_{\Phi}^{2}\left(\mu\right)$-norm of $f$ and of $g$ is bounded
by one, the $L_{\Phi}\left(\mu\right)$-norm of $f$, $g$, and $fg$, are all
bounded by one. The space $L_{\cosh_{2}}\left(\mu\right)$ has a continuous
injection in the Fréchet space
$L^{\infty-0}(\mu)=\cap_{\alpha>1}L^{\alpha}(\mu)$, which is an algebra. When
we need the product, we can either assume the factors are both sub-Gaussian or
move up the functional framework to the Lebesgue spaces’ intersection.

######  Example 15 (Gaussian exponential Orlicz space)

Let us now discuss other special issues of Orlicz spaces by focusing on a case
of specific interest in IG, that is, the _Gaussian exponential Orlicz space_
$L_{\cosh_{2}}\left(\gamma\right)$, with $\gamma$ the standard $n$-variate
Gaussian density. We note that that _dominated convergence does not hold in
this space_. In fact, the squared-norm function $f(x)=\left|x\right|^{2}$
belongs to the Gaussian exponential Orlicz space
$L_{\cosh_{2}}\left(\gamma\right)$ because

$\int\cosh_{2}(\lambda f(x))\ \gamma(x)dx<\infty\quad\text{for
all}\quad\lambda<1/2\ .$ (100)

The sequence $f_{N}(x)=f(x)(\left|x\right|\leq N)$ converges to $f$ point-wise
and in all $L^{\alpha}(\gamma)$, $1\leq\alpha<\infty$. However, the
convergence does not hold in the Gaussian exponential Orlicz space. We see
that, for all $\lambda\geq 1/2$,

$\int\cosh_{2}(\lambda(f(x)-f_{N}(x)))\
\gamma(x)dx=\int_{\left|x\right|>N}\cosh_{2}(\lambda f(x))\
\gamma(x)dx=\infty\ ,$

while convergence would imply

$\limsup_{N\to\infty}\int\cosh_{2}(\lambda(f(x)-f_{N}(x)))\ \gamma(x)dx\leq
1\quad\text{for all $\lambda>0$}\ .$ (101)

In the same spirit, one must observe that the closure in
$L_{\cosh_{2}}\left(\gamma\right)$ of the vector space of bounded functions is
called _Orlicz class_ $M_{\cosh_{2}}(\gamma)$ and is strictly smaller than the
full Orlicz space. Precisely, one can prove that $f\in M_{\cosh_{2}}(\gamma)$
if, and only if. the moment generating function
$\lambda\mapsto\int\mathrm{e}^{\lambda f(x)}\ \gamma(x)dx$ is finite for all
$\lambda$.292929See pistone:2018 . An example is $f(x)=x$. Bounded convergence
holds in the Orlicz class. Assume $f\in M_{\cosh_{2}}(\gamma)$ and consider
the sequence $f_{N}(x)=(\left|x\right|\leq N)f(x)$. Now,

$\int\cosh_{2}(\lambda(f(x)-f_{N}(x))\ \gamma(x)dx=\int_{\left|x\right|\geq
N}\cosh_{2}(\lambda f(x))\ \gamma(x)dz$ (102)

converges to 0 as $N\to\infty$.

##  4 Exponential statistical _bundles_

We now show how to apply the construction of the exponential and mixture
affine bundles in the case where the fibers are Orlicz spaces as defined in
the previous sections. Precisely, we consider two equivalent conjugate couples
of Young function from Table 2,

$\displaystyle\exp_{2}(x)$
$\displaystyle=\mathrm{e}^{x}-1-x=\int_{0}^{x}(x-s)\exp s\ ds\ .$ (103)
$\displaystyle\exp_{2}^{*}(y)$
$\displaystyle=(1+y)\log(1+y)-y=\int_{0}^{y}\frac{y-s}{1+s}\ ds\ .$ (104)
$\displaystyle\cosh_{2}(x)$ $\displaystyle=\cosh x-1=\int_{0}^{x}(x-s)\cosh s\
ds\ .$ (105) $\displaystyle\cosh_{2}^{*}(y)$
$\displaystyle=y\sinh^{-1}y-\sqrt{1+y^{2}}=\int_{0}^{y}\frac{y-s}{\sqrt{1+s^{2}}}\
ds\ .$ (106)

The integral form is convenient in proving useful inequalities. The identity
of the Orlicz spaces from the Young functions (103) and (105) follows from the
inequalities $\cosh_{2}(x)\leq\exp_{2}(x)\leq 2\cosh_{2}(x)$, for $x\geq 0$,
which, in turn, follow from $\cosh s\leq\exp s\leq 2\cosh s$. For all $a,y>0$,

$\exp_{2}^{*}(ay)=\int_{0}^{ay}\frac{ay-s}{1+s}\
ds=a\int_{0}^{y}\frac{a(y-t)}{1+at}\
dt\leq\max(a,a^{2})\int_{0}^{y}\frac{y-t}{1+t}\ dt\ ,$ (107)

and similarly for $\cosh_{2}^{*}$. The growth bounds

$\exp_{2}^{*}(ay)\leq
a\max(1,a)\exp_{2}^{*}(y)\quad\text{and}\quad\cosh_{2}^{*}(ay)\leq
a\max(1,a)\cosh_{2}^{*}(y)$ (108)

imply a a bound on the Luxemburg norm. If $\int\exp_{2}^{*}(f)\ dm<\infty$,
then

$\int\exp_{2}^{*}(\rho^{-1}f)\
dm\leq\rho^{-1}\max(1,\rho^{-1})\int\exp_{2}^{*}(f)\ dm<\infty\ ,\quad\rho>0\
.$ (109)

It follows that

$\left\|f\right\|_{L_{\exp_{2}^{*}}\left(m\right)}^{-1}\max\left(1,\left\|f\right\|_{L_{\exp_{2}^{*}}\left(m\right)}^{-1}\right)\int\exp_{2}^{*}\left(f\right)\
dm\geq\\\
\int\exp_{2}^{*}\left(\left\|f\right\|_{L_{\exp_{2}^{*}}\left(m\right)}^{-1}f\right)\
dm=1\ ,$ (110)

and

$\left\|f\right\|_{L_{\exp_{2}^{*}}\left(m\right)}\min\left(1,\left\|f\right\|_{L_{\exp_{2}^{*}}\left(m\right)}\right)\leq\int\exp_{2}^{*}\left(f\right)\
dm\ .$ (111)

The bounds (108) imply that the conjugate space equals the dual Banach
space.303030This is a general result, see (musielak:1983, , §13) or
(adams|fournier:2003, , §8.17–20). Here we sketch the argument in our special
case. From the bound in Eq. (109), we see that the norm is defined with
equality, that is, $\left\|f\right\|_{L_{\exp_{2}^{*}}\left(m\right)}=1$ if,
and only if, $\int\exp_{2}^{*}(f)\ dm=1$. Now, Eq. (104) shows that
$\exp_{2}^{*}(y)$ is smaller than $y^{2}/2=\int_{0}^{y}(y-s)\ ds$. Because of
that, we have the injection $L^{2}(m)\hookrightarrow
L_{\exp_{2}^{*}}\left(m\right)$, and hence the dual injection
$\left(L_{\exp_{2}^{*}}\left(m\right)\right)^{*}\hookrightarrow(L^{2}(m))^{*}=L^{2}(m)$,
so that each element of the dual is identified with a random variable. That
is, a linear functional on $L_{\exp_{2}^{*}}\left(m\right)$ of norm $k$ is of
the form $f\mapsto\left\langle f,g\right\rangle_{m}$ with $\left|\left\langle
f,g\right\rangle_{m}\right|\leq
k\left\|f\right\|_{L_{\exp_{2}^{*}}\left(m\right)}$. The dual is identified
with $L_{\exp_{2}}\left(m\right)$ with the Orlicz norm. In conclusion

$L_{\exp_{2}}\left(\mu\right)={L_{\exp_{2}^{*}}\left(\mu\right)}^{*}\quad\text{and}\quad
L_{\cosh_{2}^{*}}\left(\mu\right)={L_{\cosh_{2}^{*}}\left(\mu\right)}^{*}\ .$
(112)

The reverse duality, that is, reflexivity, does not hold unless the sample
space is finite. That is, the dual of $L_{\exp_{2}}\left(\mu\right)$ contains
functionals which are not representable as functions. This is similar to the
well known case $(L^{1}(m))^{*}=L^{\infty}(m)$ and
$(L^{\infty}(m))^{*}\subsetneq L^{1}(m)$.313131See the general references
already cited above.

######  Remark 3 (Analytic bi-lateral Laplace transform)

The random variable $u$ belongs to $L_{\exp_{2}}\left(m\right)$ if, and only
if, the Laplace transform of the image probability measure $u_{*}(m)$ is
finite on an open interval containing 0. In such a case, the Laplace transform
itself is analytic at 0.323232See, for example, malliavin:1995

######  Remark 4 (Densities with finite entropy)

Let us discuss the relation between the following properties: 1) the
probability density $p$ belongs to the conjugate space
$L_{\exp_{2}^{*}}\left(m\right)$; 2) the density $p$ has integrable logarithm
$-\infty<\int\log p\ dm\leq 0$; 3) the density $p$ has finite entropy
$0\leq\int p\log p\ dm<+\infty$.333333A classical reference is
cover|thomas:2006 . The function $y\mapsto y\log y$, $y>0$, is convex, hence
the increment is bounded by the derivatives at the extreme points, $\log
y+1\leq(1+y)\log(1+y)-y\log y\leq\log(1+y)+1\leq y+1$. From the upper bound,
integration gives $\int\exp_{2}^{*}(p)\ dm\leq\int p\log p\ dm+1$, that is, 3)
implies 1). From the lower bound, if 2) holds, then $\int p\log p\
dm\leq\int\exp_{2}^{*}(p)\ dm-\int\log p\ dm$, that is, 1) and 2) imply 3).

In the next section, we define a class of densities such that the affine
structure based on these Orlicz spaces applies.

###  4.1 Maximal exponential model

In this section, we apply the general methods of Sec. 2 to the specific case
of a Banach manifold modeled on the Orlicz space
$L_{\cosh_{2}}\left(\mu\right)$.343434The idea of _maximal exponential_ model
has been introduced in pistone|sempi:95 , cena|pistone:2007 ,
pistone|rogantin:99 ; the extension to the statistical _bundle_ was introduced
in gibilisco|pistone:98 pistone:2013GSI  and pistone:2018 .

First, we define the moment functional and the cumulant functional. These are
non-parametric versions of the moment generating function and the cumulant
generating function, respectively. Given a probability measure $\mu$ on the
measurable space $(\Omega,\mathcal{B})$, we define $B_{\mu}=\left\\{u\in
L_{\cosh_{2}}\left(\mu\right)\,\middle|\,\int u\ d\mu=0\right\\}$. It is a
Banach space when the Luxemburg norm of $L_{\cosh_{2}}\left(\mu\right)$ is
restricted to the sub-space. The _moment functional_ is the convex mapping

$M_{\mu}\colon B_{\mu}\ni u\mapsto\int\mathrm{e}^{u}\ d\mu\in]0,\infty]\ .$
(113)

The _proper domain_ of $M_{\mu}$, $\left\\{u\in
B_{\mu}\,\middle|\,M_{\mu}(u)<\infty\right\\}$, is a convex subset of
$B_{\mu}$ that contains the unit ball of $L_{\cosh_{2}}\left(\mu\right)$. In
fact, $\int\cosh_{2}u\ d\mu\leq 1$ implies $\int\mathrm{e}^{u}\
d\mu=M_{\mu}(u)\leq 4$. It follows that the interior of the proper domain of
$M_{\mu}$ is an open convex set, $\mathcal{S}(\mu)=\left\\{u\in
B_{\mu}\,\middle|\,k_{\mu}(u)<\infty\right\\}^{\circ}$.

The _cumulant functional_ is defined by for $u\in\mathcal{s}_{\mu}$ by
$K_{\mu}(u)=\log M_{\mu}(u)$. For all $u\in\mathcal{s}_{\mu}$ and $h\in
B_{\mu}$, the mapping $t\mapsto M_{\mu}(h+th)=\int\mathrm{e}^{th}\
\mathrm{e}^{u}\ d\mu$ is the Laplace transform of the random variable $h$ with
respect to the finite measure $\mathrm{e}^{u}\cdot d\mu$ and it is defined on
a neighborhood of 0. It follows from standard results353535See, for example,
brown:86 . that the mapping is infinitely differentiable at $t=0$ with $k$’th
derivative

$\left.\frac{\partial M_{\mu}(u+\sum_{j=1}^{k}t_{j}h_{j})}{\partial
t_{1}\cdots\partial t_{k}}\right|_{t_{,}\dots,t_{k}=0}=\int h_{1}\cdots h_{k}\
\mathrm{e}^{u}\ d\mu\ .$ (114)

The multi-linear mapping $B_{\mu}\times\cdots\times
B_{\mu}\ni(h_{1},\dots,h_{k})\to h_{1}\cdots h_{k}\mathrm{e}^{u}$ is bounded
into $L^{1}(\mu)$. In fact,

$\int\left|h_{1}\cdots h_{k}\right|\mathrm{e}^{u}\
d\mu\leq\left(\int\left|h_{1}\cdots h_{k}\right|^{n}\
d\mu\right)^{1/n}\left(\int\mathrm{e}^{nu/(n-1)}\ d\mu\right)^{(n-1)/n}$ (115)

and we can chose $n$ such that $nu/(n-1)\in s_{\mu}$. For such an $n$, the
first factor is bounded by

$\int\left|h_{1}\cdots h_{k}\right|^{n}\ d\mu\leq
n!^{k}\int\mathrm{e}^{h_{1}+\cdots h_{k}}\ d\mu\ ,$ (116)

where the RHS integral is bounded if $h_{1}+\cdot+h_{k}\in\mathcal{S}(\mu)$.
This proves the boundedness. In other words, the mapping $\mathcal{s}_{\mu}\ni
u\mapsto M_{\mu}(u)$ is infinitely Gateaux-differentiable and the derivatives
are continuous linear operators.363636We do not discuss here stronger
differentiability conditions, see cena:2002 . In fact, the moment functional
is Frechét-differentiable on $\mathcal{S}(\mu)$ and analytic on the open unit
ball. We will denote the derivative by

$D^{k}M_{\mu}(u)\cdot(h_{1}\cdots h_{k})=\int h^{k}\mathrm{e}^{u}\ d\mu\ .$
(117)

The relevance for us of the moment functional and of the cumulant functional
lies in the fact they are the normalising constants (or partition functions)
in probability densities of exponential form. The support $M$ of the affine
manifold of interests here is the _maximal exponential model_ 373737The idea
to study the geometry of statistical models by embedding them in a larger
exponential family is due to efron:1975  and efron:1978 . The possibility of a
non-parametric set-up for the statistical bundle was suggested first in
dawid:75  and dawid:1977as . See the review paper kass:89 .
$\operatorname{\mathcal{E}}\left(\mu\right)$ consisting of all probability
densities of the exponential form

$q=M_{\mu}(u)^{-1}\mathrm{e}^{u}=\exp\left(u-K_{\mu}(u)\right)\ ,\quad
u\in\mathcal{s}_{\mu}\ .$ (118)

The following ”portmanteau” proposition is crucial in proving the consistency
of the affine structure of our Orlicz space setup. It shows the existence of a
statistical bundle with base $\operatorname{\mathcal{E}}\left(\mu\right)$,
fibers which are closed subspaces of $L_{\cosh_{2}}\left(\mu\right)$, and a
proper co-cycle of parallel transports.

###### Proposition 6

For all densities $p,q\in\operatorname{\mathcal{E}}\left(\mu\right)$ the
following propositions are equivalent.

  1. 1.

$q=\mathrm{e}^{u-K_{p}(u)}\cdot p\ ,$ (119)

where $u\in L_{(\cosh-1)}\left(\mu\right)$, $\int u\ p\ d\mu=0$, and $u$
belongs to the interior of the proper domain of the convex function
$K_{p}=\int\mathrm{e}^{u}\ p\ d\mu$.

  2. 2.

The densities $p$ and $q$ are connected by an open exponential arc, that is,
there exists a one-dimensional exponential family
$r_{\theta}\propto\mathrm{e}^{\theta U}$ with $\theta\in I$, $r_{0}=p$,
$r_{1}=q$, and $[0,1]\subset I$.

  3. 3.

$L_{\cosh_{2}}\left(p\right)=L_{\cosh_{2}}\left(q\right)$ and the norms are
equivalent;

  4. 4.

$p/q\in\cup_{\alpha>1}L^{\alpha}(q)$ and $q/p\in\cup_{\alpha>1}L^{\alpha}(p)$.

  5. 5.

The mapping $v\mapsto\frac{q}{p}v$ is an isomorphism of
$L_{\cosh_{2}^{*}}\left(p\right)$ onto $L_{\cosh_{2}^{*}}\left(q\right)$

######  Proof (Main argument only)

We give only part of the proof.383838See cena|pistone:2007 ;
santacroce|siri|trivellato:2016 ; santacroce|siri|trivellato:2018 ;
siri|trivellato:2021-SPL  for a detailed proof and some further developments.
Precisely, we prove a generalization of the implication 2$\Rightarrow$3. Let
$F$ be logarithmically convex on $\mathbb{R}$ and such that $\Phi=F-1$ is a
Young function. For example, the assumption holds for both $F(x)=\cosh x$ and
$F(x)=\mathrm{e}^{x^{2}/2}$. For all real $A$ and $B$, the function

$\mathbb{R}^{2}\ni(\lambda,t)\mapsto F(\lambda
A)\mathrm{e}^{tB}=\exp\left(\log F(\lambda A)+tB\right)$ (120)

is convex and so is the integral

$C(\lambda,t)=\int F(\lambda f(x))\ \mathrm{e}^{tu(x)}\ p(x)\ \mu(dx)\ ,$
(121)

where $f\in L_{\Phi}\left(\gamma\right)$ with and $u\in
L_{\cosh_{2}}\left(p\right)$ with $\int u(x)\ p(x)\ dx=0$. Without restriction
of generality, assume $\left\|f\right\|_{L_{\Phi}\left(p\right)}=1$. Let us
derive two marginal inequalities. First, for $t=0$, the definition of
Luxemburg norm gives

$C(\lambda,0)=\int F(\lambda f)\ p(x)\ \mu(dx)\leq 2\ ,\quad-1\leq\lambda\leq
1\ .$ (122)

Second, for $\lambda=0$, consider $K_{p}(tu)=\log\int\mathrm{e}^{tu}\
p(x)\mu(dx)$, where $t$ belongs to an an open interval $I$ containing $[0,1]$
and such that $K_{p}(tu)<+\infty$. It follows that

$C(0,t)=\int\mathrm{e}^{tu}\ p(x)\mu(dx)=\mathrm{e}^{K_{p}(tu)}<+\infty\ .$
(123)

Choose a $t>1$ in $I$ and consider the convex combination

$\left(\frac{t-1}{t},1\right)=\frac{t-1}{t}(1,0)+\frac{1}{t}(0,t)$ (124)

and the inequality

$C\left(\frac{t-1}{t},1\right)\leq\frac{t-1}{t}C(1,0)+\frac{1}{t}C(0,t)\leq
2\frac{t-1}{t}+\frac{1}{t}\mathrm{e}^{K_{1}(tu)}\ .$ (125)

Now,

$\int\Phi\left(\frac{t-1}{t}f(x)\right)\mathrm{e}^{u(x)-K_{1}(u)}p(x)\
\mu(dx)=\\\ \int
F\left(\frac{t-1}{t}f(x)\right)\mathrm{e}^{u(x)-K_{p}(u)}p(x)\ \,u(dx)-1=\\\
\mathrm{e}^{-K_{p}(u)}C\left(\frac{t-1}{t},1\right)-1\leq\mathrm{e}^{-K_{p}(u)}\left(2\frac{t-1}{t}+\frac{1}{t}\mathrm{e}^{K_{p}(tu)}\right)-1\
.$

As the RHS is finite, we have proved that $f\in L_{\Phi}\left(q\right)$ for
$q=\mathrm{e}^{u-K_{p}(u)}$. Conversely, a similar argument shows the other
implication. We have proved that all Orlicz spaces $L_{\Phi}\left(p\right)$,
$p\in\operatorname{\mathcal{E}}\left(\gamma\right)$ are equal. In turn,
equality of spaces implies the equivalence norms. It is possible to derive
explicit bounds by choosing a $t$ such that the RHS is smaller or equal to 1.
∎

If $p,q$ are densities in the maximal exponential model, then there is an
$\epsilon>0$ such that the combination $r=(1-\lambda)p+\lambda q$,
$-\epsilon<\lambda<1+\epsilon$ is there too. That is, _the maximal exponential
model is a convex set and is open on lines_. This is proved by Prop.
6(4).393939See the detailed proof in santacroce|siri|trivellato:2016 .

Let us recall a few notation from Sec. 2.

$\displaystyle B_{p}$ $\displaystyle=$ $\displaystyle\left\\{u\in
L_{\cosh_{2}}\left(\mu\right)\,\middle|\,\int u\ p\ d\mu=0\right\\}$ (126)
$\displaystyle\mathrm{e}_{p}(u)$ $\displaystyle=$
$\displaystyle\exp\left(u-K_{p}(u)\right)\cdot p$ (127)
$\displaystyle\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}u$ $\displaystyle=$
$\displaystyle u-\int u\ p\ d\mu$ (128)
$\displaystyle\prescript{\text{m}}{}{\mathbb{U}}_{p}^{q}u$ $\displaystyle=$
$\displaystyle\frac{q}{p}u$ (129)

The spaces $B_{p}$, $p\in\operatorname{\mathcal{E}}\left(\mu\right)$, of Eq.
(126) will be the fibers of the statistical bundle. Given
$p,q\in\operatorname{\mathcal{E}}\left(\mu\right)$, Prop. 6(3) shows that the
Banach spaces $B_{p}$ and $B_{q}$ are vector sub-spaces of co-dimension 1 of
the two isomorphic space $L_{\cosh_{2}}\left(p\right)\approx
L_{\cosh_{2}}\left(q\right)$. The mapping
$\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}\colon B_{p}\to B_{q}$ of Eq. (128)
is indeed such an automorphism.

According to Prop. 6(1), for each given
$p\in\operatorname{\mathcal{E}}\left(\mu\right)$, every other
$q\in\operatorname{\mathcal{E}}\left(\mu\right)$ is of the form
$\mathrm{e}_{p}(u)$ for some $u\in B_{p}$. Precisely,
$\log\frac{q}{p}=u-K_{p}(u)$ with $\int\log\frac{q}{p}\ p\ d\mu=-K_{p}(u)$ and
$u=\log\frac{q}{p}-\int\log\frac{q}{p}\ p\ d\mu$. It follows that the mapping

$\operatorname{\mathcal{E}}\left(\mu\right)^{2}\ni(p,q)\mapsto\operatorname{\mathbb{S}}(p,q)=\log\frac{q}{p}-\int\log\frac{q}{p}\
p\ d\mu\in B_{p}$ (130)

is well defined and, moreover,

$s_{p}\colon\operatorname{\mathcal{E}}\left(\mu\right)\ni
q\mapsto\operatorname{\mathbb{S}}(p,q)\in B_{p}$ (131)

is 1-to-1 with image the open set $\left\\{u\in
B_{p}\,\middle|\,K_{p}(u)<\infty\right\\}^{\circ}$.

Eq. (128) defines a linear continuous invertible operator from $B_{p}$ onto
$B_{q}$. The co-cycle properties hold:
$\prescript{\text{e}}{}{\mathbb{U}}_{p}^{p}$ is the identity and
$\prescript{\text{e}}{}{\mathbb{U}}_{q}^{r}\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}=\prescript{\text{e}}{}{\mathbb{U}}_{p}^{r}$.
In turn, we see that
$\left(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}\right)^{-1}=\prescript{\text{e}}{}{\mathbb{U}}_{q}^{p}$.

We have thus proved that the statistical bundle

$S\operatorname{\mathcal{E}}\left(\mu\right)=\left\\{(p,u)\,\middle|\,p\in\operatorname{\mathcal{E}}\left(\mu\right),u\in
B_{p}\right\\}$ (132)

admits the family of parallel transports
$\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}\colon B_{p}\to B_{q}$. With
respect to this family, the map $\operatorname{\mathbb{S}}$ of Eq. (130) is
easily seen to be an affine displacement. Moreover, the image of the chart an
open set. All requirements for an affine statistical manifold are met.

Let us discuss the duality. Define

$\prescript{*}{}{B}_{b}=\left\\{v\in
L_{\exp_{2}^{*}}\left(\mu\right)\,\middle|\,\int vp\ d\mu=0\right\\}\ .$ (133)

It is a Banach space for the restriction of the Luxemburg norm. We use the
pre-script notation to remember that $B_{p}$ is the dual of
$\prescript{*}{}{B}_{p}$, that is,
$\left(\prescript{*}{}{B}_{p}\right)^{*}=B_{p}$. In the pairing $B_{p}$,
$\prescript{*}{}{B}_{p}$, it holds

$\left\langle\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}u,v\right\rangle_{q}=\left\langle
u,\prescript{\text{m}}{}{\mathbb{U}}_{q}^{p}v\right\rangle_{p}\ ,$ (134)

for all $p,q\in\operatorname{\mathcal{E}}\left(\mu\right)$, $u\in B_{p}$ and
$v\in\prescript{*}{}{B}_{p}$.

We can define the conjugate affine system with

$\prescript{*}{}{s}_{p}\colon q\mapsto\frac{q}{p}-1\ .$ (135)

Let us check that $\prescript{*}{}{s}_{p}(q)\in\prescript{*}{}{B}_{p}$.
Clearly, $\int\left(\frac{q}{p}-1\right)\ p\ d\mu=0$. From
$\exp_{2}^{*}(y)=(1+y)\log(1+y)-y$, we find

$\int\exp_{2}^{*}(s_{p}(q))\ p\ d\mu=\int\frac{q}{p}\log\frac{q}{p}p\
d\mu-\int\frac{q}{p}\ d\mu+1=-\int\log\frac{p}{q}\ q\ d\mu\ .$ (136)

The last term of the equality is finite because it is the opposite of the
normalising constant of the exponential representation of $p$ with respect to
$q$. The existence of a family of transports is shown in Prop. 6(5).

We give below a few results about the fist three derivatives of the cumulant
funtional. As the values of the Gateaux derivatives are directional
derivatives, all equalities below reduce to well known properties of the usual
cumulant generating functions.

$DK_{p}(u)[h]=\int h\ \mathrm{e}^{u-K_{p}(u)}\ d\mu=\int h\ \mathrm{e}_{p}(u)\
d\mu\ .$ (137)
$D^{2}K_{p}(u)[h_{1},h_{2}]=\int(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h_{1})(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h_{2})\
\mathrm{e}_{p}(u)\ d\mu=\\\
\left\langle\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h_{1},\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h_{2}\right\rangle_{\mathrm{e}_{p}(u)}\
.$ (138)
$D^{3}K_{p}(u)[h_{1},h_{2},h_{3}]=\int(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h_{1})(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h_{2})(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h_{3})\
\mathrm{e}_{p}(u)\ d\mu\ .$ (139)

###### Remark 5

The equations above show that the expected value
${\operatorname{\mathbb{E}}}_{q}\left[h\right]$, the covariance
$\operatorname{Cov}_{q}\left(h_{1},h_{2}\right)$, and the triple covariance
$\operatorname{Cov}_{q}\left(h_{1},h_{2},h_{3}\right)$, all actually depend on
a convex functions, namely, the cumulant functional. In fact, what we have
here is a special case of Hessian geometry.404040See shima:2007 .

######  Remark 6 (Entropy: cf. Ex. 6 and Rem. 4)

The entropy is finite on all of
$q\in\operatorname{\mathcal{E}}\left(\mu\right)$ because
$-\operatorname{\mathcal{H}}\left(q\right)=\operatorname{D}\left(q\middle|1\right)$
and its expression in terms of $K_{1}$ and $DK_{1}$ is

$\int q\log q\ d\mu=\int(u-K_{1}(u))\mathrm{e}_{1}(u)\
d\mu=DK_{1}(u)[u]-K_{1}(u)\ .$ (140)

The random variable $\log q=u-K_{1}(u)$ is integrable, with $\int\log q\
d\mu=-K_{p}(u)$. It follows from the argument in Remark 4 that each density
$p\in\operatorname{\mathcal{E}}\left(\mu\right)$ is a an element of
$L_{\exp_{2}^{*}}\left(\mu\right)$.

We have the mapping between convex sets

$B_{1}\supset\mathcal{s}_{\mu}\ni
u\mapsto\mathrm{e}_{1}(u)\in\operatorname{\mathcal{E}}\left(\mu\right)\subset\prescript{*}{}{B}_{1}+1\
.$ (141)

It should be noted that
$p\mapsto\prescript{*}{}{s}(p)=p-1\in\prescript{*}{}{B}_{1}$ is not a chart of
$\operatorname{\mathcal{E}}\left(\mu\right)$ because the image is not an open
set in general. It is in fact the restriction of an affine chart defined on a
larger base set, the affine subspace of $L_{\cosh_{2}^{*}}\left(\mu\right)$
generated the maximal exponential model, that is, $\prescript{*}{}{B}_{1}+1$.

We conclude this section with examples showing how a classical topics of IG
are presented in our formalism.

######  Example 16 (First and second variation of the KL divergence)

Given $p,q\in\operatorname{\mathcal{E}}\left(m\right)$, we can write
$q=\mathrm{e}^{u-K_{p}(u)}\cdot p=\mathrm{e}_{p}(u)$, that is, $u=s_{p}(q)$.
It follows that the value at $u$ of the expression of the Kullback-Leibler
divergence $q\mapsto\operatorname{D}\left(q\middle|p\right)$ in the chart
$s_{p}$ is

$\operatorname{D}\left(q\middle|p\right)=\int\log\frac{q}{p}\ q\
dm=\int(u-K_{p}(u))\ \mathrm{e}_{p}(u)\ p\ dm=\\\ \int u\ \mathrm{e}_{p}(u)\
p\ dm-K_{p}(u)=DK_{p}(u)[u]-K_{p}(u)\ .$ (142)

The derivative in the direction $h$ is

$D(DK_{p}(u)[u]-K_{p}(u))[h]=D^{2}K_{p}(u)[u,h]+DK_{p}(u)[h]-DK_{p}(u)[h]=\\\
D^{2}K_{p}(u)[u,h]=\left\langle\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}u,\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}h\right\rangle_{q}=\left\langle\prescript{\text{m}}{}{\mathbb{U}}_{q}^{p}\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}u,h\right\rangle_{p}\
.$ (143)

The second derivative in the directions $h$ and $k$ is

$D^{2}(DK_{p}(u)[u]-K_{p}(u))[h,k]=\\\
D(D^{2}K_{p}(u)[u,h])[k]=D^{3}K_{p}(u)[u,h,k]+D^{2}K_{p}(u)[k,h]=\\\
\int(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}u)(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}h)(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}k)\
q\
d\mu+\int(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}h)(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}k)\
q\ d\mu$ (144)

Both $\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}h$ and
$\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}k$ are in the fiber $B_{q}\subset
L_{\cosh_{2}}\left(q\right)$ but, in general, their product
$(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}h)(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}k)$
is an element of $\cap_{\alpha>1}L^{\alpha}(q)$ only. If actually $h,k\in
L_{\operatorname{gauss}_{2}}\left(q\right)$, a simple algebraic expansion
presents the symmetric part of the bilinear operator as a function of the
product $hk-\int hk\ p\ dm\in B_{p}$.

######  Example 17 (Sub-models with constant expectation)

Let be given a constant $b\in\mathbb{R}$, a reference density
$p\in\operatorname{\mathcal{E}}\left(m\right)$, and a random variable $f\in
L_{\cosh_{2}}\left(m\right)$ such that $\int f\ p\ dm=b$. Consider the subset
$E(f,p)$ of the maximal exponential model
$\operatorname{\mathcal{E}}\left(m\right)$ consisting of all densities $q$
such that $\int f\ q\ dm=\int f\ p\ dm$. It is a relatively open convex set.
Note that $f-b\in B_{p}$ and the condition can be equivalently rewritten in
terms of the coordinate $u=s_{p}(q)$. Namely,
$\mathrm{e}_{p}(u)=\mathrm{e}^{u-K_{p}(u)}\cdot p=q\in E(B,p)$, if, and only
if,

$0=\int(f-b)\ \mathrm{e}_{p}(u)\ dm=DK_{p}(u)[f-b]\ .$ (145)

Any tangent vector $h$ satisfies
$D^{2}K_{p}(u)[h,f-b]=\operatorname{Cov}_{q}\left(h,f-b\right)=0$ For a smooth
$F\colon\operatorname{\mathcal{E}}\left(m\right)\to\mathbb{R}$, the expression
at $p$ is $F_{p}(u)=F\circ\mathrm{e}_{p}(u)$. Any extremal point of $F$
restricted to $E(f,p)$ satisfies $DF_{p}(u)[h]=0$.

Consider the entropy $\operatorname{\mathcal{H}}\left(q\right)=\int\log q\ q\
dm$. As seen in Rem. 4, we have
$\operatorname{\mathcal{H}}_{p}(u)=DK_{1}(u)[u]-K_{1}(u)$ and
$D\operatorname{\mathcal{H}}_{1}(u)[h]=\operatorname{Cov}_{\mathrm{e}_{1}(u)}\left(h,u\right)$,
so that the stationarity condition is
$\operatorname{Cov}_{\mathrm{e}_{1}(u)}\left(h,f\right)=0$ whenever
$\operatorname{Cov}_{\mathrm{e}_{1}(u)}\left(h,f-b\right)$. In conclusion
$u\propto(f-b)$ and the stationary point has the form
$q=\mathrm{e}^{\theta(f-b)-K_{p}(\theta(f-b))}$.

A similar argument holds in the case of a finite number of random variables
with given expected values. In the case of an infinite dimensional subspace of
random variables, the subspace must be splitting.

######  Example 18 (Pythagorean theorem for the KL divergence)

The Kullback-Leibler divergence is
$\operatorname{D}\left(p\middle|q\right)=\int\log\frac{p}{q}\ p\ dm$. Consider
positive densities $q=\mathrm{e}^{u-K_{p}(u)}\cdot p$,
$q\in\operatorname{\mathcal{E}}\left(p\right)$, and $r=(1+v)\cdot p$,
$v\in\prescript{*}{}{B}_{p}$. A simple computation shows that

$\operatorname{D}\left(r\middle|q\right)+\left\langle
u,v\right\rangle_{p}=\operatorname{D}\left(r\middle|p\right)+\operatorname{D}\left(p\middle|q\right)\
.$

The case where $\left\langle u,v\right\rangle_{p}=0$ is sometimes called
Pythagorean Theorem for divergences. The result actually implies a conjugation
statement. In fact, $\left\langle
u,v\right\rangle_{p}\leq\operatorname{D}\left(r\middle|p\right)+\operatorname{D}\left(p\middle|q\right)$
and $r=q$ gives $\left\langle
u,v\right\rangle_{p}=\operatorname{D}\left(q\middle|p\right)+\operatorname{D}\left(p\middle|q\right)$.

###  4.2 Covariant derivatives, tensor bundle, acceleration

In discussing higher-order geometry, one needs to define bundles whose fibers
are the product of multiple copies of the mixture and exponential fibers.

As a first example, the _full bundle_ is

$\prescript{1}{}{S}^{1}\operatorname{\mathcal{E}}\left(\mu\right)=\left\\{(q,\eta,w)\,\middle|\,q\in\operatorname{\mathcal{E}}\left(\mu\right),\eta\in{}^{*}S_{q}\operatorname{\mathcal{E}}\left(\mu\right),w\in
S_{q}\operatorname{\mathcal{E}}\left(\mu\right)\right\\}\ .$ (146)

There is a duality pairing
${}^{*}S_{q}\operatorname{\mathcal{E}}\left(\mu\right)\times
S_{q}\operatorname{\mathcal{E}}\left(\mu\right)\ni(\eta,w)\mapsto\left\langle\eta,w\right\rangle_{q}$
and the dual of ${}^{*}S_{q}\operatorname{\mathcal{E}}\left(\mu\right)$ is
$S_{q}\operatorname{\mathcal{E}}\left(\mu\right)$. The full bundle is our
setup to discuss second-order geometry.

More generally, ${}^{h}\\!S^{k}$ will denote the case with $h$ mixture factors
and $k$ exponential factors. We use both notations for expected values in the
following sections, $\int F\
pd\mu={\operatorname{\mathbb{E}}}_{p}\left[F\right]$.

Let us compute the expression of the velocity at time $t$ of a smooth curve in
the exponential bundle:

$t\mapsto\gamma(t)=(q(t),w(t))\in
S\\!\operatorname{\mathcal{E}}\left(\mu\right)=\prescript{0}{}{S}^{1}\operatorname{\mathcal{E}}\left(\mu\right)\
,$ (147)

where $q(t)\in\operatorname{\mathcal{E}}\left(\mu\right)$ and $w(t)\in
S_{q(t)}\operatorname{\mathcal{E}}\left(\mu\right)$ is a $q(t)$-centered
random variable in the Orlicz space $L_{\cosh_{2}}\left(\mu\right)$.

In the chart centered at $p$, the expression of the curve is

$\gamma_{p}(t)=s_{p}(\gamma(t))=\left(s_{p}(q(t)),\prescript{\text{e}}{}{\mathbb{U}}_{q(t)}^{p}w(t)\right)\
,$ (148)

and, consequently, the time derivative has two components,

$\frac{d}{dt}s_{p}(q(t))=\frac{d}{dt}\left(\log\frac{q(t)}{p}-{\operatorname{\mathbb{E}}}_{p}\left[\log\frac{q(t)}{p}\right]\right)=\frac{\dot{q}(t)}{q(t)}-{\operatorname{\mathbb{E}}}_{p}\left[\frac{\dot{q}(t)}{q(t)}\right]=\\\
\prescript{\text{e}}{}{\mathbb{U}}_{q(t)}^{p}\frac{\dot{q}(t)}{q(t)}=\prescript{\text{e}}{}{\mathbb{U}}_{q(t)}^{p}\frac{d}{dt}\log
q(t)\ ,$ (149)

and

$\frac{d}{dt}\prescript{\text{e}}{}{\mathbb{U}}_{q(t)}^{p}w(t)=\frac{d}{dt}\left(w(t)-{\operatorname{\mathbb{E}}}_{p}\left[w(t)\right]\right)=\dot{w}(t)-{\operatorname{\mathbb{E}}}_{p}\left[\dot{w}(t)\right]\
.$ (150)

By expressing the tangent at each time $t$ in the chart centered at the
current position $q(t)$, from the first component we obtain the _velocity_ ,

$\overset{\star}{q}(t)=\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q(t)}\frac{d}{dt}s_{p}(q(t))=\dot{u}(t)-{\operatorname{\mathbb{E}}}_{q(t)}\left[\dot{u}(t)\right]=\frac{d}{dt}\log
q(t)=\frac{\dot{q}(t)}{q(t)}\ .$ (151)

Notice that $t\mapsto(q(t),\overset{\star}{q}(t))$ is a curve in the
exponential statistical bundle whose expression in the chart centered at $p$
is $t\mapsto(u(t),\dot{u}(t))$. The mapping $q\mapsto(q,\overset{\star}{q})$
is the _lift_ of the curve to the statistical bundle.

Let us turn to the interpretation of the second component in (150). In terms
of the exponential parallel transport in (62), we define an exponential
_covariant_ derivative by setting

$\frac{\operatorname{D}}{dt}w(t)=\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q(t)}\frac{d}{dt}\prescript{\text{e}}{}{\mathbb{U}}_{q(t)}^{p}w(t)=\\\
\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q(t)}\left(\dot{w}(t)-{\operatorname{\mathbb{E}}}_{p}\left[\dot{w}(t)\right]\right)=\dot{w}(t)-{\operatorname{\mathbb{E}}}_{q(t)}\left[\dot{w}(t)\right]\
.$ (152)

The notation $\,\frac{\operatorname{D}}{dt}\,$ will generically denote the
covariant time derivative in a given transport or connection, whose choice
will depend on the context. When necessary, we use
$\operatorname{D_{\text{e}}}$, $\operatorname{D_{\text{m}}}$, or similar
notations.

Let us do now the computation in the _mixture bundle_. The curve is

$t\mapsto\zeta(t)=(q(t),\eta(t))\in{}^{*}\\!S\\!\operatorname{\mathcal{E}}\left(\mu\right)=\prescript{1}{}{S}^{0}\operatorname{\mathcal{E}}\left(\mu\right)$
(153)

The computation in the first component is the same as above. The expression of
the second component in chart is
$\prescript{\text{m}}{}{\mathbb{U}}_{q(t)}^{p}\eta(t)=\frac{q(t)}{p}\eta(t)$.
This gives the derivation

$\frac{d}{dt}\prescript{\text{m}}{}{\mathbb{U}}_{q(t)}^{p}\eta(t)=\frac{d}{dt}\frac{q(t)}{p}\eta(t)=\frac{1}{p}\left(\dot{q}(t)\eta(t)+q(t)\dot{\eta}(t)\right)\
,$ (154)

which, in turn, defines the mixture covariant derivative as

$\frac{\operatorname{D}}{dt}\eta(t)=\prescript{\text{m}}{}{\mathbb{U}}_{p}^{q(t)}\frac{d}{dt}\prescript{\text{m}}{}{\mathbb{U}}_{q(t)}^{p}\eta(t)=\\\
\frac{p}{q(t)}\frac{1}{p}\left(\dot{q}(t)\eta(t)+q(t)\dot{\eta}(t)\right)=\overset{\star}{q}(t)\eta(t)+\dot{\eta}(t)\
.$ (155)

A basic computation in the full statistical bundle is the the variation of the
duality pairing. The couple of covariant derivatives in (152), (155) are
compatible with the duality pairing.

######  Proposition 7 (Duality of the covariant derivatives)

For each smooth curve in the full statistical bundle,

$t\mapsto(q(t),\eta(t),w(t))\in\prescript{1}{}{S}^{1}\operatorname{\mathcal{E}}\left(\mu\right)\
,$ (156)

it holds

$\frac{d}{dt}\left\langle\eta(t),w(t)\right\rangle_{q(t)}=\left\langle\frac{\operatorname{D_{\text{m}}}}{dt}\eta(t),w(t)\right\rangle_{q(t)}+\left\langle\eta(t),\frac{\operatorname{D_{\text{e}}}}{dt}w(t)\right\rangle_{q(t)}\
.$ (157)

###### Proof

In an arbitrary reference density $p$,

$\frac{d}{dt}\left\langle\eta(t),w(t)\right\rangle_{q(t)}=\frac{d}{dt}\left\langle\prescript{\text{m}}{}{\mathbb{U}}_{q(t)}^{p}\eta(t),\prescript{\text{e}}{}{\mathbb{U}}_{q(t)}^{p}w(t)\right\rangle_{p}=\\\
\left\langle\frac{d}{dt}\prescript{\text{m}}{}{\mathbb{U}}_{q(t)}^{p}\eta(t),\prescript{\text{e}}{}{\mathbb{U}}_{q(t)}^{p}w(t)\right\rangle_{p}+\left\langle\prescript{\text{m}}{}{\mathbb{U}}_{q(t)}^{p}\eta(t),\frac{d}{dt}\prescript{\text{e}}{}{\mathbb{U}}_{q(t)}^{p}w(t)\right\rangle_{p}=\\\
\left\langle\prescript{\text{m}}{}{\mathbb{U}}_{p}^{q(t)}\frac{d}{dt}\prescript{\text{m}}{}{\mathbb{U}}_{q(t)}^{p}\eta(t),w(t)\right\rangle_{q(t)}+\left\langle\eta(t),\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q(t)}\frac{d}{dt}\prescript{\text{e}}{}{\mathbb{U}}_{q(t)}^{p}w(t)\right\rangle_{p}\
,$ (158)

which is Eq. (157). ∎

######  Remark 7 (Riemannian derivative)

In amari|nagaoka:2000  an other covariant derivative is defined for
$t\mapsto(q(t),w(t))\in S\\!\operatorname{\mathcal{E}}\left(\mu\right)$.
Because of the embedding
$S\\!\operatorname{\mathcal{E}}\left(\mu\right)\subset{}^{*}\\!S\\!\operatorname{\mathcal{E}}\left(\mu\right)$,
we can define

$\frac{\operatorname{D_{0}}}{dt}w(t)=\frac{1}{2}\left(\frac{\operatorname{D_{\text{m}}}}{dt}w(t)+\frac{\operatorname{D_{\text{e}}}}{dt}w(t)\right)$
(159)

The remarkable property of this derivative is the compatibility with the inner
product. If
$t\mapsto(q(t),v(t),w(t))\in\prescript{0}{}{S}^{2}\operatorname{\mathcal{E}}\left(\mu\right)$,
a straightforward computation shows that

$\frac{d}{dt}\left\langle
v(t),w(t)\right\rangle_{q(t)}=\left\langle\frac{\operatorname{D_{0}}}{dt}v(t),w(t)\right\rangle_{q(t)}+\left\langle
v(t),\frac{\operatorname{D_{0}}}{dt}w(t)\right\rangle_{q(t)}\ .$ (160)

Let us show that both the covariant derivatives we have defined in Eq.s (152)
and (155) actually deserve their name.414141We adapt the presentation of
(docarmo:1992, , § 2.2). All curves and fields are assumed to be smooth.

###### Proposition 8

Both exponential and mixture covariant derivatives satisfy the following
equations.

$\displaystyle\frac{\operatorname{D}}{dt}(X(t)+Y(t))=\frac{\operatorname{D}}{dt}X(t)+\frac{\operatorname{D}}{dt}Y(t)$
(161)
$\displaystyle\frac{\operatorname{D}}{dt}f(t)v(t)=\dot{f}(t)X(t)+f(t)\frac{\operatorname{D}}{dt}X(t)$
(162)

###### Proof

Both equations follow immediately from the definitions, that is
$\frac{\operatorname{D}}{dt}X(t)={\mathbb{U}}_{p}^{q(t)}\frac{d}{dt}{\mathbb{U}}_{q(t)}^{p}X(t)$.
∎

###### Remark 8

Let us justify our unusual presentation of a classical topic such as covariant
derivatives in short. As the manifold structure we discuss is actually an
affine space with global charts, a number of notions are presented in a direct
way, namely, the geometry of the tangent bundle of the manifold is described
in terms of explicitly defined parallel transports on its expression as a
statistical bundle.424242This was done first in gibilisco|pistone:98 . Because
of that, and because of the non-parametric set-up, and because of the
statistical application of interest, the covariant derivatives are given as
operations on smooth curves.

The usual covariant derivatives of vector fields434343As described, for
example, in (lang:1995, , Ch. VIII) could be defined, whether it is needed, as
follows. If $F$ is a smooth section of the statistical bundle, $F(q)$ in the
fiber at $q$, for each smooth curve $t\mapsto q(t)$ one could compute
$\frac{\operatorname{D}}{dt}F(q(t))$ and looks at its representation as a
linear operator $DF(q(t))$ applied to $\overset{\star}{q}(t)$. Here $DF$ would
be the covariant derivative as defined on smooth sections. As we have a
specific representation of the linear operators of fibers represented
precisely by the full statistical bundle
$\prescript{1}{}{S}^{1}\operatorname{\mathcal{E}}\left(\mu\right)$, we prefer
to talk about covariant gradients instead of covariant derivatives. This is
done in the following section.

We now define the _second statistical bundle_ to be

$\prescript{0}{}{S}^{3}\operatorname{\mathcal{E}}\left(\mu\right)=\left\\{(q,w_{1},w_{2},w_{3})\,\middle|\,(q\in\operatorname{\mathcal{E}}\left(\mu\right),w_{1},w_{2},w_{3}\in
S_{q}\operatorname{\mathcal{E}}\left(\mu\right)\right\\}\ ,$ (163)

with charts centered at $p\in\operatorname{\mathcal{E}}\left(\mu\right)$
defined by

$s_{p}(q,w_{1},w_{2},w_{3})=\left(s_{p}(q),\prescript{\text{e}}{}{\mathbb{U}}_{q}^{p}w_{1},\prescript{\text{e}}{}{\mathbb{U}}_{q}^{p}w_{2},\prescript{\text{e}}{}{\mathbb{U}}_{q}^{p}w_{3}\right)\
.$ (164)

The second bundle is an expression of the statistical bundle of the
exponential statistical bundle,
$SS\\!\operatorname{\mathcal{E}}\left(\mu\right)$. That is, for each curve
$t\mapsto\gamma(t)=(q(t),w(t))\in
S\\!\operatorname{\mathcal{E}}\left(\mu\right)$, we define its _velocity at
$t$_ to be

$\overset{\star}{\gamma}(t)=\left(q(t),w(t),\overset{\star}{q}(t),\frac{\operatorname{D}}{dt}w(t)\right)\
,$ (165)

because $t\mapsto\overset{\star}{\gamma}(t)$ is a curve in the second
statistical bundle and that its expression in the chart at $p$ has the last
two components equal to the values given in (149) and (150), respectively. The
corresponding notion of gradient will be discussed in the next section.

For each smooth curve $t\mapsto q(t)$, the velocity of its lift
$t\mapsto\chi(t)=(q(t),\overset{\star}{q}(t))$ is

$\overset{\star}{\chi}(t)=\left(q(t),\overset{\star}{q}(t),\overset{\star}{q}(t),\overset{\star\star}{q}(t)\right)\
,$ (166)

where $\overset{\star\star}{q}(t)$ defines the _exponential acceleration_ at
$t$,

$\overset{\star\star}{q}(t)=\frac{\operatorname{D}}{dt}\overset{\star}{q}(t)=\frac{d}{dt}\frac{\dot{q}(t)}{q(t)}-{\operatorname{\mathbb{E}}}_{q(t)}\left[\frac{d}{dt}\frac{\dot{q}(t)}{q(t)}\right]=\frac{\ddot{q}(t)}{q(t)}-\Big{(}\overset{\star}{q}(t)^{2}-{\operatorname{\mathbb{E}}}_{q(t)}\left[\overset{\star}{q}(t)^{2}\right]\Big{)}\
.$ (167)

As the two middle components of the RHS of Eq. (166) are equal, the
acceleration is actually defined in
$\prescript{0}{}{S}^{2}\operatorname{\mathcal{E}}\left(\mu\right)$.

######  Remark 9 (e-Geodesic)

The acceleration defined above has the one-dimensional exponential families as
(differential) geodesic. Every exponential (Gibbs) curve $t\mapsto
q(t)=\mathrm{e}_{p}(tu)$ has velocity $\overset{\star}{q}(t)=u-dK(tu)[u]$, so
that the acceleration is $\overset{\star\star}{q}(t)=0$. Conversely, if one
writes $v(t)=\log q(t)$, then

$0=\overset{\star\star}{q}(t)=\ddot{v}(t)+{\operatorname{\mathbb{E}}}_{q(t)}\left[\ddot{v}(t)\right]\
,$ (168)

so that $v(x;t)=tv(x)+c$.

We have defined the _exponential acceleration_
$\frac{\operatorname{D_{\text{e}}}}{dt}\overset{\star}{q}(t)=\overset{\star\star}{q}(t)$
via exponential transport in (167). Similarly, we define the _mixture
acceleration_ , via mixture transport, as

$\frac{\operatorname{D_{\text{m}}}}{dt}\overset{\star}{q}(t)={\mathbb{U}}_{p}^{q(t)}\frac{d}{dt}{\mathbb{U}}_{q(t)}^{p}\overset{\star}{q}(t)=\ddot{q}(t)/q(t)\
.$ (169)

######  Example 19 (The SIR model as a second order equation)

The Kermack and McKendrick SIR model is a differential equation on the
positive probability densities on $\left\\{S,I,R\right\\}$. As an equation in
the statistical bundle is

$\begin{cases}\overset{\star}{p}(S;t)&=-\beta p(I;t)\\\
\overset{\star}{p}(I;t)&=\beta p(S;t)-\gamma\\\
\overset{\star}{p}(R;t)&=\gamma p(I;t)/p(R;t)\end{cases}$ (170)

The mixture acceleration $\prescript{m}{}{\operatorname{D}}^{2}p=\ddot{p}/p$
as a function of $(p(t),\overset{\star}{p}(t))$ is given by an equation which
is linear in each fiber:

$\prescript{m}{}{\operatorname{D}}^{2}p(t)=\begin{bmatrix}-\beta p(I;t)&-\beta
p(I;t)&0\cr\beta p(S;t)&(\beta p(S;t)-\gamma)&0\cr
0&\gamma&0\end{bmatrix}\overset{\star}{p}(t)$ (171)

###  4.3 Gradients on the Tensor Bundles

Given a scalar field
$F\colon\operatorname{\mathcal{E}}\left(\mu\right)\to\mathbb{R}$ the
_gradient_ of $F$ is the section $q\mapsto\operatorname{grad}F(q)$ of the
mixture bundle ${}^{*}\\!S\\!\operatorname{\mathcal{E}}\left(\mu\right)$ such
that for all smooth curve $t\mapsto
q(t)\in\operatorname{\mathcal{E}}\left(\mu\right)$ it holds

$\frac{d}{dt}F(q(t))=\left\langle\operatorname{grad}F(q(t)),\overset{\star}{q}(t)\right\rangle_{q(t)}\
.$ (172)

######  Example 20 (Gradient of the entropy)

The natural gradient can be computed in some cases without recourse to the
computation in charts. For example, if conditions for existence and smoothness
are satisfied, the derivative of the entropy function
$\operatorname{\mathcal{H}}\left(q(t)\right)={\operatorname{\mathbb{E}}}_{1}\left[q(t)\log
q(t)\right]$ along the curve $t\mapsto q(t)$ is

$\frac{d}{dt}\operatorname{\mathcal{H}}\left(q(t)\right)=-\frac{d}{dt}{\operatorname{\mathbb{E}}}_{1}\left[q(t)\log
q(t)\right]=-{\operatorname{\mathbb{E}}}_{1}\left[\dot{q}(t)(\log
q(t)+1)\right]=\\\ -{\operatorname{\mathbb{E}}}_{q(t)}\left[\log
q(t)\overset{\star}{q}(t)\right]=\left\langle-\log
q(t)-\operatorname{\mathcal{H}}\left(q(t)\right),\overset{\star}{q}(t)\right\rangle_{q(t)}\
,$ (173)

hence $\operatorname{grad}\operatorname{\mathcal{H}}\left(q\right)=-\log
q-\operatorname{\mathcal{H}}\left(q\right)$.

More precisely, in coordinates, we have

$\operatorname{\mathcal{H}}\left(\mathrm{e}_{p}(u(t))\right)=-\int(u(t)-K_{p}(u(t))\
\mathrm{e}_{p}(u(t))\ d\mu=\\\ -DK_{p}(u(t))[u(t)]+K_{p}(u(t)]\ .$ (174)

With cancellations and centering, the derivative is

$D^{2}K_{p}(u(t)][u(t),\dot{u}(t)]=\left\langle-\log
q(t)-{\operatorname{\mathbb{E}}}_{q(t)}\left[-\log
q(t)\right],\overset{\star}{q}(t)\right\rangle_{q(t)}\ .$ (175)

In chart, the gradient is expressed as a function of the ordinary gradient
$\nabla F_{p}$ of $F$. In the generic chart at $p$, with $q=\mathrm{e}_{p}(u)$
and $F(q)=F_{p}(u)$, it holds

$\left\langle\operatorname{grad}F(q(t)),\overset{\star}{q}(t)\right\rangle_{q(t)}=\frac{d}{dt}F(q(t))=\frac{d}{dt}F_{p}(u(t))=DF_{p}(u(t))[\dot{u}(t)]=\\\
DF_{p}(u(t))[{\mathbb{U}}_{q(t)}^{p}\overset{\star}{q}(t)]=\left\langle
p^{-1}\nabla
F_{p}(u(t)),{\mathbb{U}}_{q(t)}^{p}\overset{\star}{q}(t)\right\rangle_{p}=\\\
\left\langle{\mathbb{U}}_{p}^{q(t)}p^{-1}\nabla
F_{p}(u(t)),\overset{\star}{q}(t)\right\rangle_{q(t)}=\left\langle
q^{-1}\nabla F_{p}(u(t)),\overset{\star}{q}(t)\right\rangle_{q(t)}=\\\
\left\langle q^{-1}\nabla
F_{p}(u(t))-{\operatorname{\mathbb{E}}}_{q(t)}\left[q^{-1}\nabla
F_{p}(u(t))\right],\overset{\star}{q}(t)\right\rangle_{q(t)}\ ,$ (176)

where, in the pairing, mixture and exponential transports consistently acts on
the fibers.

######  Remark 10 (Natural gradient)

The definition of gradient above is a non parametric version of the natural
gradient introduced by S-I Amari.444444See amari:1998natural . Consider a
$d$-dimensional statistical model $\Theta\ni\theta\mapsto
q(\theta)\in\operatorname{\mathcal{E}}\left(\mu\right)$,
$\Theta\subset\mathbb{R}^{d}$ open. The variation along the curve
$\theta_{i}\mapsto q(\theta)$ is

$\frac{\partial}{\partial\theta_{i}}F(q(\theta))=\left\langle\operatorname{grad}F(\theta),\frac{\partial}{\partial\theta_{i}}\log
q(\theta)\right\rangle_{q(\theta(t))}\ .$ (177)

Assume there is a $q(\theta)$-orthogonal projection of
$\operatorname{grad}F(q(\theta))$ onto the space generated by
$\frac{\partial}{\partial\theta_{1}}\log
q(\theta),\dots,\frac{\partial}{\partial\theta_{d}}\log q(\theta)$.

$\frac{\partial}{\partial\theta_{i}}F(q(\theta))=\sum_{j=1}^{d}\widehat{F}_{j}(\theta)\left\langle\frac{\partial}{\partial\theta_{j}}\log
q(\theta),\frac{\partial}{\partial\theta_{i}}\log
q(\theta)\right\rangle_{q(\theta(t))}=\sum_{j=1}^{d}\widehat{F}_{j}(\theta)I_{ij}(\theta)\
,$ (178)

where the matrix $I(\theta)$ is the _Fisher matrix_ of the given model. The
last equation presents the natural gradient $\widehat{F}_{j}(\theta)$ as the
inverse Fisher matrix applied to the ordinary gradient with respect to the
parameters. Notice that the non-parametric setup clarifies an explicit set of
assumptions to justify the computation.

######  Example 21 (Gradient Flow of the entropy: cf. Ex. 6)

The integral curves of the gradient flow equation

$\overset{\star}{q}(t)=\operatorname{grad}\operatorname{\mathcal{H}}\left(q(t)\right)$
(179)

are exponential families of the form $q(t)\propto q(0)^{\mathrm{e}^{-t}}$. In
fact, if we write the equation in $\mathbb{R}^{N}$, we get the quasi-linear
ODE

$\frac{d}{dt}\log q(t)=-\log q(t)-\operatorname{\mathcal{H}}\left(q(t)\right)\
,$ (180)

and, in turn,

$\log q(t)=\mathrm{e}^{-t}\log
q(0)-\mathrm{e}^{-t}\int_{0}^{t}\mathrm{e}^{s}\operatorname{\mathcal{H}}\left(q(t)\right)\
ds\ .$ (181)

The behaviour as $t\to\pm\infty$ and other properties follow easily.454545See
pistone:2013Entropy .

Given a section $q\mapsto F(q)\in
S_{q}\operatorname{\mathcal{E}}\left(\mu\right)$, the variation of the entropy
along the integral curves, $\overset{\star}{q}(t)=F(q(t))$, is

$\frac{d}{dt}\operatorname{\mathcal{H}}\left(q(t)\right)=\left\langle\operatorname{grad}\operatorname{\mathcal{H}}\left(q(t)\right),F(q(t))\right\rangle_{q(t)}\\\
=-\left\langle\log
q(t)+\operatorname{grad}\operatorname{\mathcal{H}}\left(q(t)\right),F(q(t))\right\rangle_{q(t)}\
.$ (182)

For example, the condition for entropy production becomes

$\left\langle\log
q+\operatorname{\mathcal{H}}\left(q(t)\right),F(q)\right\rangle_{q}={\operatorname{\mathbb{E}}}_{q}\left[\log
qF(q)\right]<0\ .$ (183)

The definition of the natural gradient can be generalized to functions defined
on the full statistical bundle
$\prescript{1}{}{S}^{1}\operatorname{\mathcal{E}}\left(\mu\right)$. Precisely,
let be given a real function
$F\colon\prescript{1}{}{S}^{1}\operatorname{\mathcal{E}}\left(\mu\right)\times\mathcal{D}\to\mathbb{R}$,
where $\mathcal{D}$ a domain of $\mathbb{R}^{k}$. For a generic smooth curve

$t\mapsto(q(t),\eta(t),w(t),c(t))\in\prescript{1}{}{S}^{1}\operatorname{\mathcal{E}}\left(\mu\right)\times\mathcal{D}\
,$ (184)

we want to write

$\frac{d}{dt}F\big{(}q(t),\eta(t),w(t),c(t)\big{)}=\\\
\left\langle\operatorname{grad}F\big{(}q(t),\eta(t),w(t),c(t)\big{)},\overset{\star}{q}(t)\right\rangle_{q(t)}+\left\langle\frac{\operatorname{D}}{dt}\eta(t),\operatorname{grad}_{\text{m}}F\big{(}q(t),\eta(t),w(t),c(t)\big{)}\right\rangle_{q(t)}+\\\
\left\langle\operatorname{grad}_{\text{e}}F\big{(}q(t),\eta(t),w(t),c(t)\big{)},\frac{\operatorname{D}}{dt}w(t)\right\rangle_{q(t)}+\nabla
F\big{(}q(t),\eta(t),w(t),c(t)\big{)}\cdot\dot{c}(t)\ ,$ (185)

where the four components of the gradient are

$\prescript{1}{}{S}^{1}\operatorname{\mathcal{E}}\left(\mu\right)\times\mathcal{D}\ni(q,\eta,w,c)\mapsto\begin{cases}(q,\operatorname{grad}F\big{(}q,\eta,w,c\big{)})\in{}^{*}S_{q}\operatorname{\mathcal{E}}\left(\mu\right)\\\
(q,\operatorname{grad}_{\text{m}}F\big{(}q,\eta,w,c\big{)})\in
S_{q}\operatorname{\mathcal{E}}\left(\mu\right)\\\
(q,\operatorname{grad}_{\text{e}}F\big{(}q,\eta,w,c\big{)})\in{}^{*}S_{q}\operatorname{\mathcal{E}}\left(\mu\right)\\\
(q,\nabla
F\big{(}q,\eta,w,c\big{)})\in\operatorname{\mathcal{E}}\left(\mu\right)\times\mathbb{R}^{k}\end{cases}$
(186)

###### Proposition 9

The total derivative (185) holds true, where

  1. 1.

$\operatorname{grad}F\big{(}q,\eta,w,c\big{)}$ is the natural gradient of

$q\mapsto F(q,{\mathbb{U}}_{p}^{q}\zeta,{\mathbb{U}}_{p}^{q}v,c)\ ,$ (187)

that is, with the representation in $p$-chart

$F_{p}(u,\zeta,w,c)=F(\mathrm{e}_{p}(u),{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}\zeta,{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}v,c)\
,$ (188)

it is defined by

$\left\langle\operatorname{grad}F(q,\zeta,w,c),\overset{\star}{q}\right\rangle_{q}=d_{1}F_{p}(u,\zeta,w,c)\left[{\mathbb{U}}_{q}^{p}\overset{\star}{q}\right]\
,\quad(q,\overset{\star}{q})\in
S\\!\operatorname{\mathcal{E}}\left(\mu\right)\ ;$ (189)

  2. 2.

$\operatorname{grad}_{\text{m}}F\big{(}q,\eta,w,c\big{)}$ and
$\operatorname{grad}_{\text{e}}F\big{(}q,\eta,w,c\big{)}$ are the fiber
gradients;

  3. 3.

$\nabla F\big{(}q,\eta,w,c\big{)}$ is the Euclidean gradient w.r.t. the last
variable.

###### Proof

Let us fix a reference density $p$ and express both the given function and the
generic curve in the chart at $p$. We can write the total derivative as

$\frac{d}{dt}F\big{(}q(t),\eta(t),w(t),c(t)\big{)}=\frac{d}{dt}F_{p}(u(t),\zeta(t),v(t),c(t))=\\\
D_{1}F_{p}\big{(}u(t),\zeta(t),v(t),c(t)\big{)}\big{[}\dot{u}(t)\big{]}+D_{2}F_{p}\big{(}u(t),\zeta(t),v(t),c(t)\big{)}\big{[}\dot{\zeta}(t)\big{]}+\\\
D_{3}F_{p}\big{(}u(t),\zeta(t),v(t),c(t)\big{)}\big{[}\dot{v}(t)\big{]}+D_{4}F_{p}\big{(}u(t),\zeta(t),v(t),c(t)\big{)}\big{[}\dot{c}(t)\big{]}\
.$

In the equation above, $D_{j}$, with $j=1,\dots,4$, denotes the partial
derivative with respect to the $j$-th variable of $F_{p}$,which is intended to
provide a linear operator represented by the appropriate dual vector, that is,
the value of the proper gradient.

The last term of the total derivative does not require any comment and we can
write it as the ordinary Euclidean gradient:

$D_{4}F_{p}\big{(}u(t),\zeta(t),v(t),c(t)\big{)}\big{[}\dot{c}(t)\big{]}=\nabla
F_{p}\big{(}u(t),\zeta(t),v(t),c(t)\big{)}\cdot\dot{c}(t)\ .$ (190)

Let us consider together the second and the third term. This is a computation
of the fiber derivative and does not involve the representation in chart.
Given $\alpha\in{}^{*}S_{p}\operatorname{\mathcal{E}}\left(\mu\right)$ and
$\beta\in S_{p}\operatorname{\mathcal{E}}\left(\mu\right)$, that is,
$(\alpha,\beta)\in\prescript{1}{}{S}^{1}_{p}\operatorname{\mathcal{E}}\left(\mu\right)$,
we have

$D_{2}F_{p}(u,\zeta,v,c)[\alpha]+D_{3}F_{p}(u,\zeta,v,c)[\beta]=\left.\frac{d}{dt}F_{p}(u,\zeta+t\alpha,w+t\beta,c)\right|_{t=0}=\\\
\left.\frac{d}{dt}F(q,\eta+t{\mathbb{U}}_{p}^{q}\alpha,v+t{\mathbb{U}}_{p}^{q}\beta,c)\right|_{t=0}=\mathbb{F}F(q,\eta,w,c)[({\mathbb{U}}_{p}^{q}\alpha,{\mathbb{U}}_{p}^{q}\beta)]=\\\
\left\langle{\mathbb{U}}_{p}^{q}\alpha,\operatorname{grad}_{\text{m}}F(q,\eta,w,c)\right\rangle_{q}+\left\langle\operatorname{grad}_{\text{e}}F(q,\eta,w,c),{\mathbb{U}}_{p}^{q}\beta\right\rangle_{q}\
,$

where $\mathbb{F}$ denotes the fiber derivative in
$\prescript{1}{}{S}^{1}_{q}\operatorname{\mathcal{E}}\left(\mu\right)$, which
is expressed, in turn, with the relevant gradients. Consider that the inner
product always has ${}^{*}S_{q}\operatorname{\mathcal{E}}\left(\mu\right)$
first, followed by $S_{q}\operatorname{\mathcal{E}}\left(\mu\right)$ and that
the subscript to the $\operatorname{grad}$ symbol displays which component of
the full bundle is considered.

We have that

$\frac{\operatorname{D}}{dt}w(t)={\mathbb{U}}_{p}^{q(t)}\dot{v}(t)\
,\quad\frac{\operatorname{D}}{dt}\eta(t)={\mathbb{U}}_{p}^{q(t)}\dot{\zeta}(t)\
.$ (191)

Putting together all the contributions, we have proved that

$\frac{d}{dt}F\big{(}q(t),\eta(t),w(t),c(t)\big{)}=\\\
D_{1}F_{p}\big{(}u(t),\zeta(t),v(t),c(t)\big{)}\big{[}{\mathbb{U}}_{\mathrm{e}_{p}(u(t))}^{p}\overset{\star}{q}(t)\big{]}+\left\langle\frac{\operatorname{D}}{dt}\eta(t),\operatorname{grad}_{\text{m}}F\big{(}q(t),\eta(t),w(t),c(t)\big{)}\right\rangle_{q(t)}+\\\
\left\langle\operatorname{grad}_{\text{e}}F\big{(}q(t),\eta(t),w(t),c(t)\big{)},\frac{\operatorname{D}}{dt}w(t)\right\rangle_{q(t)}+\nabla
F\big{(}q(t),\eta(t),w(t),c(t)\big{)}\cdot\dot{c}(t)\ ,$

To identify the first term in the total derivative above, consider
the“geodesic” case,

$q(t)=\mathrm{e}_{p}(u(t)),\quad\eta(t)={\mathbb{U}}_{p}^{\mathrm{e}_{p}(u(t))}\zeta,\quad
w(t)={\mathbb{U}}_{p}^{\mathrm{e}_{p}(u(t))}v,\quad c(t)=c\ ,$ (192)

so that the first term reduces to
$D_{1}F_{p}(u(t),\zeta,v,c)[{\mathbb{U}}_{\mathrm{e}_{p}(u(t))}^{p}\overset{\star}{q}(t)]$.
It follows that the proper way to compute the first gradient is to consider
the function on $\operatorname{\mathcal{E}}\left(\mu\right)$ defined by

$q\mapsto
F_{\zeta,v,c}(q)=F(q,{\mathbb{U}}_{p}^{q}\zeta,{\mathbb{U}}_{p}^{q}v,c)$ (193)

which has a natural gradient whose chart representation is precisely that
first term. ∎

We have so concluded the computation of the total derivative of a parametric
function on the full bundle. Notice that the computation of the natural
gradient for $\operatorname{grad}_{\text{m}}F\big{(}q,\eta,w,c\big{)}$ and
$\operatorname{grad}_{\text{e}}F\big{(}q,\eta,w,c\big{)}$ is done by fixing
the variables in the fibers to be translations of fixed ones.

We are going to discuss the following examples of gradient flow on the full
statistical bundle: the scalar function $L(q,w)=\frac{1}{2}\left\langle
w,w\right\rangle_{q}$; the _cumulant_ function $L(q,w)=K_{q}(w)$; the
_conjugate cumulant_ function
$H(q,\eta)={\operatorname{\mathbb{E}}}_{q}\left[(1+\eta)\log(1+\eta)\right]$.

######  Example 22 (Scalar function $\frac{1}{2}\left\langle
w,w\right\rangle_{q}$)

On the statistical bundle, consider the scalar function given by the pairing

$L(q,w)=\frac{1}{2}\left\langle w,w\right\rangle_{q}\ .$ (194)

In chart, we have
$L_{p}(u,v)=L\left(\mathrm{e}_{p}(u),\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}v\right)=D^{2}K_{p}(u)[v,v]$
from Eq. (138), where $K_{p}(u)$ is the expression in the chart at $p$ of
Kullback–Leibler divergence of
$q\mapsto\operatorname{D}\left(p\middle|q\right)$.

From Eq. (139), we write the derivative with respect to $u$ in the direction
$h$ as

$\frac{1}{2}D^{3}K_{p}(u)[v,v,h]=\frac{1}{2}{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[\left(v-{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[v\right]\right)^{2}\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h\right]=\\\
\frac{1}{2}\left\langle
w^{2}-{\operatorname{\mathbb{E}}}_{q}\left[w\right]^{2},\prescript{\text{e}}{}{\mathbb{U}}_{p}^{q}h\right\rangle_{q}$
(195)

which, in turn, identifies the gradient as
$\operatorname{grad}\frac{1}{2}\left\langle
w,w\right\rangle_{q}=\frac{1}{2}(w^{2}-{\operatorname{\mathbb{E}}}_{q}\left[w^{2}\right])\in{}^{*}\\!S\\!\operatorname{\mathcal{E}}\left(\mu\right)$.
The exponential gradient is
$\operatorname{grad}_{\text{e}}\frac{1}{2}\left\langle
w,w\right\rangle_{q}=w$.

######  Example 23 (Cumulant functional)

If $L(q,w)=K_{q}(w)$, then

$L_{p}(u,v)=K_{\mathrm{e}_{p}(u)}\left(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}v\right)=\log{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[\mathrm{e}^{v-{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[v\right]}\right]x=\\\
\log{\operatorname{\mathbb{E}}}_{p}\left[\mathrm{e}^{u-K_{p}(u)+v-{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[v\right]}\right]=\log{\operatorname{\mathbb{E}}}_{p}\left[\mathrm{e}^{u+v-K_{p}(u)-dK_{p}(u)[v]}\right]=\\\
K_{p}(u+v)-K_{p}(u)-dK_{p}(u)[v]\ .$ (196)

The derivative with respect to $u$ in the direction $h$ is

$DK_{p}(u+v)[h]-DK_{p}(u)[h]-D^{2}K_{p}(u)[v,h]=\\\
{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u+v)}\left[h\right]-{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[h\right]-{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[\left({\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}v\right)\left({\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h\right)\right]=\\\
{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[\frac{\mathrm{e}_{p}(u+v)}{\mathrm{e}_{p}(u)}h\right]-{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[h\right]-{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[w\left({\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h\right)\right]=\\\
{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[\frac{\mathrm{e}_{p}(u+v)}{\mathrm{e}_{p}(u)}\left({\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h\right)\right]-\left\langle
w,{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h\right\rangle_{q}\ .$ (197)

The expected value of the factor
$\frac{\mathrm{e}_{p}(u+v)}{\mathrm{e}_{p}(u)}$ in the first term of the RHS
equals

${\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[\mathrm{e}^{v-(K_{p}(u+v)-K_{p}(u))}\left(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h\right)\right]=\\\
{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[\mathrm{e}^{v-(K_{\mathrm{e}_{p}(u)}({\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}v)+dK_{p}(u)[v])}\left({\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h\right)\right]=\\\
{\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[\mathrm{e}^{\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}v-K_{\mathrm{e}_{p}(u)}({\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}v)}\left(\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h\right)\right]=\\\
{\operatorname{\mathbb{E}}}_{q}\left[\mathrm{e}^{w-K_{q}(w)}\left({\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h\right)\right]=\left\langle\frac{e_{q}(w)}{q}-1,\prescript{\text{e}}{}{\mathbb{U}}_{p}^{\mathrm{e}_{p}(u)}h\right\rangle_{q}\
.$ (198)

In conclusion,
$\operatorname{grad}K_{q}(w)=\left(\frac{e_{q}(w)}{q}-1\right)-w$. The
expectation gradient is easily seen to be

$\operatorname{grad}_{\text{e}}K_{q}(w)=\frac{\mathrm{e}_{q}(w)}{q}-1\ .$
(199)

######  Example 24 (Conjugate cumulant functional)

The conjugate cumulant functional

${}^{*}\\!S\\!\operatorname{\mathcal{E}}\left(\mu\right)\colon(q,\eta)\mapsto
H(q,\eta)={\operatorname{\mathbb{E}}}_{q}\left[(1+\eta)\log(1+\eta)\right]\
,\quad\eta>-1\ ,$ (200)

is the Legendre transform of the cumulant function $K_{q}$,

$H(q,\eta)=\left\langle\eta,(\operatorname{grad}K_{q})^{-1}(\eta)\right\rangle_{q}-K_{q}\left((\operatorname{grad}K_{q})^{-1}(\eta)\right)\
.$ (201)

In particular, the fiber gradient of $H_{q}$ is
$\operatorname{grad}_{\text{m}}H(q,\eta)=\log(1+\eta)-{\operatorname{\mathbb{E}}}_{q}\left[\log(1+\eta)\right]$
which is the inverse of the fiber gradient of $K_{q}$. Notice that
$r=(1+\eta)q$ is a density, and
$\operatorname{D}\left(r\middle|q\right)=H(q,\eta)$.

Let us compute the gradient. The expression of the conjugate cumulant
functional in the chart at $p$ is

$H_{p}(u,\zeta)={\operatorname{\mathbb{E}}}_{\mathrm{e}_{p}(u)}\left[\left(1+\frac{p}{\mathrm{e}_{p}(u)}\zeta\right)\log\left(1+\frac{p}{\mathrm{e}_{p}(u)}\zeta\right)\right]=\\\
{\operatorname{\mathbb{E}}}_{p}\left[\left(\frac{\mathrm{e}_{p}(u)}{p}+\zeta\right)\log\left(1+\frac{p}{\mathrm{e}_{p}(u)}\zeta\right)\right]\
.$ (202)

As, for $h\in S_{p}\operatorname{\mathcal{E}}\left(\mu\right)$,

$\displaystyle