section

#  Applications of Structural Statistics:
Geometric Inference in Exponential Families

P. Michl  orcid.org/0000-0002-6398-0654

###### Abstract

Exponential families comprise a broad class of statistical models and
parametric families like normal distributions, binomial distributions, gamma
distributions or exponential distributions. Thereby the formal representation
of its probability distributions induces a confined intrinsic structure, which
appears to be that of a dually flat statistical manifold. Conversely it can be
shown, that any dually flat statistical manifold, which is given by a regular
Bregman divergence uniquely induced a regular exponential family, such that
exponential families may - with some restrictions - be regarded as a universal
representation of dually flat statistical manifolds. This article reviews the
pioneering work of Shun’ichi Amari about the intrinsic structure of
exponential families in terms of structural stratistics.

Keywords: Exponential Family, Statistical Manifold, Geometric Inference

##  1 Introduction

In accordance to it’s historical roots classical statistical theory has been
formulated to describe repeatable experiments in terms of random variables. A
drawback that accompanies this language is the difficulty to integrate and
describe abstract structural knowledge. Rising theories like deep learning and
complex networks dynamics, however, impressively demonstrate, that statistical
modeling and inference can greatly benefit from the integration of abtract
structural assumptions and in particular in the domains of complex natural
data.

It is therefore not surprising that this development led to a growing interest
in alternative approaches to formulate statistical theory. In particular S.
Amari pursued a fundamentally different approach by focusing on the embedding
function space of the probability distributions [1]. This view motivated the
reformulation of statistical theory by means of structural statistics [2, 3].
An important application and showcase are exponential families, which can be
completely characterized their geometric structure in terms of dually flat
statistical manifolds.

##  2 Primary affine structure of Exponential Families

######  Definition (Exponential family).

_Let $(\Omega,\,\mathcal{F_{\theta}})$ be a statistical model over a
measurable space $(\Omega,\,\Sigma_{\Omega})$. Then $(\Omega,\,\mathcal{F})$
is termed an exponential family if and only if there exists an invertible
function_

$\xi:\mathrm{dom}\theta\rightarrow\mathbb{R}^{n}$

_a sufficient statistic_

$T:(\Omega,\,\Sigma_{\Omega})\rightarrow(\mathbb{R}^{n},\,\mathcal{B}(\mathbb{R}^{n}))$

_and a scalar function_

$f:\mathrm{dom}\theta\rightarrow\mathbb{R}$

_such that for any $P_{\theta}\in\mathcal{F}$ and $\sigma\in\Sigma_{\Omega}$
it holds that:_

$\displaystyle P_{\theta}[\sigma]=$
$\displaystyle\int_{\sigma}\exp\left(\boldsymbol{\xi}(\theta_{P})\cdot\boldsymbol{T}(\omega)\right)\mathrm{d}\mu(\omega)$
(2.1) $\displaystyle\cdot\int_{\sigma}\exp f(\theta_{P})\mathrm{d}\mu(\omega)$

Since $T$ is a sufficient statistic a Markov morphism, given by
$\mathcal{T}(P)[\sigma]=P[T^{-1}(\sigma)]$ is globally invertible and its
restriction to $(\Omega,\,\mathcal{F})$ yields a statistical isomorphism
$\mathfrak{T}\in\mathrm{iso(\mathbf{Stat})}$ to a statistical model
$(X,\,\mathcal{P})\coloneqq\mathrm{img}\,\mathfrak{T}$ over a measurable space
$(X,\,\Sigma_{X})$. Then $\eta\coloneqq\theta\circ\xi^{-1}$ is an identifiable
parametrisation of $(X,\,\mathcal{P})$ and by the definition

$\psi\coloneqq\xi^{-1}\circ f\circ\xi$

it follows that:

$P_{\eta}[\sigma]=\int_{\sigma}\exp\left(\boldsymbol{\eta}_{P}\cdot\boldsymbol{x}-\psi(\boldsymbol{\eta}_{P})\right)\mathrm{d}\boldsymbol{x},\,\forall\sigma\in\Sigma_{X}$
(2.2)

Without loss of generality any exponential family, as defined in 2.1, may
therefore be assumed to be given by probability distributions with a
representation 2.2. This representation is termed the canonical form of an
exponential family and the parametrisation $\eta$ the canonical, or natural
parametrisation.

######  Definition (Natural parametrisation).

_Let $(X,\,\mathcal{P})\in\mathrm{ob}(\mathbf{Stat})$ be an exponential family
in canonical form, then the corresponding canonical parametrisation $\eta$ is
termed a natural parametrisation of $(X,\,\mathcal{P})$ and the parameter
vectors $\boldsymbol{\eta}_{P}\in\mathrm{dom}\,\eta\subseteq\mathbb{R}^{n}$
are termed natural parameters. _Remark _: Exponential families, given by the
notation $(X,\,\mathcal{P}_{\eta})$ implicate a canonical form and a natural
parametrisation $\eta$._

The function $\psi:\mathbb{R}^{n}\to\mathbb{R}$, given by an exponential
family in canonical form is known as the cumulant generating function and may
be regarded as a normalisation factor, that implements the normalisation
condition of the probability distribution:

$\int_{X}\exp\left(\boldsymbol{\eta}_{P}\cdot\boldsymbol{x}-\psi(\boldsymbol{\eta}_{P})\right)\mathrm{d}\boldsymbol{x}=1$
(2.3)

Since $\psi(\boldsymbol{\eta}_{P})$ is independent of $\boldsymbol{x}$ it may
be pulled out of the integral and a rearrangement of equation 2.3 yields:

$\psi(\boldsymbol{\eta}_{P})=\log\int_{X}\exp(\boldsymbol{\eta}_{P}\cdot\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
(2.4)

Due to this dependency the cumulant generating function relates different
statistical properties.

######  Lemma 1.

Let $\psi$ be the cumulant generating function of an exponential family
_$(X,\,\mathcal{P}_{\eta})$_ , then $\psi$ is convex with respect to $\eta$
and it’s first and second order derivatives are given by:

$\displaystyle\nabla_{\eta}\psi(\boldsymbol{\eta}_{P})$ $\displaystyle=$
$\displaystyle\mathrm{E}_{\eta}[\boldsymbol{x}]$ (2.5)
$\displaystyle\nabla_{\eta}^{2}\psi(\boldsymbol{\eta}_{P})$ $\displaystyle=$
$\displaystyle\mathrm{Var_{\eta}}[\boldsymbol{x}]$ (2.6)

where $\mathrm{E}_{\eta}[\boldsymbol{x}]\in\mathbb{R}^{n}$ and
$\mathrm{Var_{\eta}}[\boldsymbol{x}]\in\mathbb{R}^{n}$ respectively denote the
vectorial expectation and variance of $\boldsymbol{x}$ with respect to
$P_{\eta}$.

###### Proof.

Let $P_{\eta}\in\mathcal{P}$ and let $p_{\eta}$ be the density function of
$P_{\eta}$ over $(X,\,\Sigma_{X})$. Then the normalization condition is:

$\int_{X}p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}=1$ (2.7)

The partial derivation $\partial_{i}$ of equation 2.7 to the natural parameter
$\eta_{i}$ yields:

$\displaystyle\partial_{i}\int_{X}p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
(2.8) $\displaystyle=$
$\displaystyle\int_{X}\mathrm{\partial_{i}exp}\left(\sum_{i=1}^{n}\eta_{i}x_{i}-\psi(\boldsymbol{\eta}_{P})\right)\mathrm{d}\boldsymbol{x}$
$\displaystyle=$
$\displaystyle\int_{X}(x_{i}-\partial_{i}\psi(\boldsymbol{\eta}_{P}))p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}\stackrel{{\scriptstyle\ref{eq:expfam:normalization}}}{{=}}0$

Therefore:

$\displaystyle\nabla_{\eta}\psi(\boldsymbol{\eta}_{P})$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:expfam:normalization}}}{{=}}$
$\displaystyle\nabla_{\eta}\psi(\boldsymbol{\eta}_{P})\int_{X}p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
$\displaystyle=$
$\displaystyle\int_{X}\nabla_{\eta}\psi(\boldsymbol{\eta}_{P})p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:expfam:partialf}}}{{=}}$
$\displaystyle\int_{X}\boldsymbol{x}p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
$\displaystyle=$ $\displaystyle\mathrm{E_{\eta}}[\boldsymbol{x}]$

This proves equation 2.5. A further partial derivation of 2.8 with respect to
the natural parameter $\eta_{j}$ yields:

$\displaystyle 0$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:expfam:partialf}}}{{=}}$
$\displaystyle\partial_{j}\int_{X}(x_{i}-\partial_{i}\psi(\boldsymbol{\eta}_{P}))p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
$\displaystyle=$
$\displaystyle\int_{X}(x_{i}-\partial_{i}\psi(\boldsymbol{\eta}_{P}))\partial_{j}p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
$\displaystyle-\int_{X}\partial_{j}\partial_{i}\psi(\boldsymbol{\eta}_{P})\int_{X}p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:expfam:partialf}}}{{=}}$
$\displaystyle\int_{X}(x_{i}-\partial_{i}\psi(\boldsymbol{\eta}_{P}))(x_{j}-\partial_{j}\psi(\boldsymbol{\eta}))p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
$\displaystyle-\partial_{j}\partial_{i}\psi(\boldsymbol{\eta}_{P})$

Therefore:

$\displaystyle\nabla_{\eta}^{2}\psi(\boldsymbol{\eta}_{P})$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:expfam:dpartialf}}}{{=}}$
$\displaystyle\int_{X}(\boldsymbol{x}-\nabla_{\eta}\psi(\boldsymbol{\eta}_{P}))(\boldsymbol{x}-\nabla_{\eta}\psi(\boldsymbol{\eta}_{P}))p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:expfam:nablaf}}}{{=}}$
$\displaystyle\int_{X}(\boldsymbol{x}-\mathrm{E}[\boldsymbol{x}])^{2}p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
$\displaystyle=$ $\displaystyle\mathrm{Var_{\eta}}[x]$

This proves equation 2.6. Furthermore since:

$\nabla_{\eta}^{2}\psi(\boldsymbol{\eta}_{p})=\mathrm{Var_{\eta}}[x]\geq
0,\,\forall\boldsymbol{\eta}_{p}\in\mathrm{img}\eta$

∎

The Hessian matrix of $\psi$ is positive definite and therefore $\psi$ is
convex. The convexity of $\psi$ may therefore be used to induce a Riemannian
metric by the Bregman divergence.

######  Lemma 2.

Let $\psi$ be the cumulant generating function of an exponential family
_$(X,\,\mathcal{P}_{\eta})$._ Then the Bregman divergence given by $D_{\psi}$,
is the dual Kullback-Leibler divergence, such that:

$D_{\psi}[P\parallel Q]=D_{\mathrm{KL}}^{*}[P\parallel Q],\,\forall
P,\,Q\in\mathcal{P}$ (2.10)

###### Proof.

Let $P,\,Q\in\mathcal{P}$ and $p,\,q$ their respectively density functions
over $(X,\,\Sigma_{X})$. By calculating:

$\log
p_{\eta}(\boldsymbol{x})=\boldsymbol{\eta}_{p}\cdot\boldsymbol{x}-\psi(\boldsymbol{\eta}_{p})$
(2.11)

the Bregman divergence is derived by:

$\displaystyle D_{\psi}[P\parallel Q]$
$\displaystyle\stackrel{{\scriptstyle}}{{=}}$
$\displaystyle\psi(\boldsymbol{\eta}_{P})-\psi(\boldsymbol{\eta}_{Q})$
$\displaystyle-\nabla_{\eta}\psi(\boldsymbol{\eta}_{Q})\cdot(\boldsymbol{\eta}_{Q}-\boldsymbol{\eta}_{P})$
$\displaystyle=$
$\displaystyle\left(\int_{X}(\boldsymbol{\eta}_{Q}\cdot\boldsymbol{x})q(\boldsymbol{x})\mathrm{d}\boldsymbol{x}-\psi(\boldsymbol{\eta}_{Q})\right)$
$\displaystyle-\left(\int_{X}(\boldsymbol{\eta}_{P}\cdot\boldsymbol{x})q(\boldsymbol{x})\mathrm{d}\boldsymbol{x}-\psi(\boldsymbol{\eta}_{P})\right)$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:expfam:logp}}}{{=}}$
$\displaystyle\int_{X}q_{\eta}(\boldsymbol{x})\log
q(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
$\displaystyle-\int_{X}q(\boldsymbol{x})\log
p(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$ $\displaystyle=$
$\displaystyle\int_{X}q(\boldsymbol{x})\log\frac{q(\boldsymbol{x})}{p(\boldsymbol{x})}\mathrm{d}\boldsymbol{x}$
$\displaystyle\stackrel{{\scriptstyle}}{{=}}$ $\displaystyle
D_{\mathrm{KL}}^{*}[P\parallel Q]$

∎

######  Lemma 3.

Let $\psi$ be the cumulant generating function of an exponential family
_$(X,\,\mathcal{P}_{\eta})$._ Then the Bregman divergence given by $D_{\psi}$,
induces a Riemannian metric, which is given by the Fisher information:

$g_{P,\psi}=\mathrm{I}[P\mid X]$ (2.12)

###### Proof.

Let $P\in\mathcal{P}$ and $p$ the probability density function of $P$ over
$(X,\,\Sigma_{X})$, then:

$\nabla_{\eta}\log
p_{\eta}(\boldsymbol{x})=\nabla_{\eta}(\boldsymbol{\eta}_{P}\cdot\boldsymbol{x}-\psi(\boldsymbol{\eta}_{P}))=\boldsymbol{x}-\nabla_{\eta}\psi(\boldsymbol{\eta}_{P})$
(2.13)

The Riemannian metric, which is induced by the Bregman divergence $D_{\psi}$
is therefore given by:

$\displaystyle g_{P,\psi}$ $\displaystyle\stackrel{{\scriptstyle}}{{=}}$
$\displaystyle\nabla_{\eta}^{2}\psi(\boldsymbol{\eta}_{P})$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:expfam:dpartialf}}}{{=}}$
$\displaystyle\int_{X}(\boldsymbol{x}-\nabla_{\eta}\psi(\boldsymbol{\eta}_{P}))^{2}p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:expfam:dellogp}}}{{=}}$
$\displaystyle\int_{X}(\nabla_{\eta}\log
p_{\eta}(\boldsymbol{x}))^{2}p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
$\displaystyle\stackrel{{\scriptstyle\mathrm{def}}}{{=}}$
$\displaystyle\mathrm{I}[P_{\eta}\mid X]$

∎

######  Lemma 4.

Let $\psi$ be the cumulant generating function of an exponential family
$(X,\,\mathcal{P}_{\eta})$ and $(X,\,\mathcal{P}_{\eta},\,D_{\psi})$ the
Riemannian statistical manifold, given by the Bregman divergence $D_{\psi}$.
Then the $\eta$-affine geodesics in $(X,\,\mathcal{P}_{\eta},\,D_{\psi})$ are
given by exponential families.

###### Proof.

The $\eta$-affine geodesics in $(X,\,\mathcal{P_{\eta}},\,D_{\psi})$ are given
by affine linear curves in the $\eta$-parametrisation. For
$P,\,Q\in\mathcal{P}$ and
$\boldsymbol{\eta}_{P,Q}(t)=(1-t)\boldsymbol{\eta}_{P}+t\boldsymbol{\eta}_{Q}$,
with $t\in[0,\,1]$ let $\gamma_{P,Q}(t)$ be the $\eta$-affine geodesic
connecting $P$ with $Q$ and $\mathrm{d}_{X}\gamma_{P,Q}(t)$ the probability
density function of $\gamma_{P,Q}(t)$ over $(X,\,\Sigma_{X})$. Then then
representation of $\mathrm{d}_{X}\gamma_{P,Q}(t)$ in the
$\eta$-parametrisation is given by:

$\mathrm{d}_{X}\gamma_{P,Q}(t)(\boldsymbol{x})=\exp\left(t(\boldsymbol{\eta}_{Q}-\boldsymbol{\eta}_{P})\cdot\boldsymbol{x}+\boldsymbol{\eta}_{P}\cdot\boldsymbol{x}-\psi_{\eta}(t)\right)$

Let $p,\,q$ be the respective probability densities of $P$ and $Q$ over
$(X,\,\Sigma_{X})$. A $\log$-transformation and a subsequent substitution of
$\boldsymbol{\eta}_{P}$ and $\boldsymbol{\eta}_{Q}$ by equation 2.11 yields:

$\log\mathrm{d}_{X}\gamma_{P,Q}(t)(\boldsymbol{x})=(1-t)\log
p(\boldsymbol{x})+t\log q(\boldsymbol{x})-\psi(t)$

A further $\exp$-transformation and subsequent integration over a measurable
set $\sigma\in\Sigma$ gives:

$\displaystyle\gamma_{P,Q}(t)[\sigma]$
$\displaystyle=\int_{\sigma}\exp\left((1-t)\log
p(\boldsymbol{x})\right)\mathrm{d}\boldsymbol{x}$
$\displaystyle\cdot\int_{\sigma}\exp\left(t\log
q(\boldsymbol{x})-\psi(t)\right)\mathrm{d}\boldsymbol{x}$

For varying $P,\,Q\in\mathcal{P}$ this yields a generic representation of
$\eta$-affine geodesics in $(X,\,\mathcal{P_{\eta}},\,D_{\psi})$, by are
exponential families with respect to the curve parameter $t$. ∎

######  Lemma 5.

Let $\psi$ be the cumulant generating function of an exponential family
$(X,\,\mathcal{P}_{\eta})$ and $(X,\,\mathcal{P}_{\eta},\,D_{\psi})$ the
Riemannian statistical manifold, given by the Bregman divergence $D_{\psi}$.
Then the $\eta$-affine geodesics in $(X,\,\mathcal{P}_{\eta},\,D_{\psi})$ are
flat with respect to the Fisher information.

##  3 Dual affine structure of Exponential Families

In the purpose, to study the structure, obtained by a Legendre transformation
a further parametric family is introduced, that is shown to cover this dual
structure. This parametric family is the mixture family.

######  Definition (Mixture family).

_Let_

$(\Omega,\,\mathcal{F_{\theta}})\in\mathrm{ob}(\mathbf{Stat})$

_be a statistical model over a measurable space $(\Omega,\,\Sigma_{\Omega})$.
Then $(\Omega,\,\mathcal{F})$ is a mixture family if and only if there is an
invertible function $w:\mathrm{dom}\theta\rightarrow\mathbb{R}^{n+1}$ with
$w_{i}(\theta_{p})>0,\,\forall i$ and
$\sum_{i=0}^{n}w_{i}(\theta_{p})=1,\,\forall\theta_{p}$ and $n+1$ pairwise
independent probability distributions $Q_{i}$ over
$(\Omega,\,\Sigma_{\Omega})$, such that for all
$P_{\theta}\in\mathcal{F_{\theta}}$:_

$P_{\theta}[\sigma]=\sum_{i=0}^{n}w_{i}(\theta_{p})Q_{i}[\sigma],\,\forall\sigma\in\Sigma_{\Omega}$
(3.1)

The normalization condition can be used to restrict the number of parameters,
by the definition:

$R(\theta_{p})[\sigma]=\left(1-\sum_{i=1}^{n}w_{i}(\theta_{p})\right)Q_{0}[\sigma]$
(3.2)

Therefore:__

$P_{\theta}[\sigma]=\sum_{i=1}^{n}w_{i}(\theta_{p})Q_{i}[\sigma]+R(\theta_{p})[\sigma]$

This transformation is a globally invertible Markov morphism, and its
restriction to $(\Omega,\,\mathcal{F})$ yields a statistical isomorphism
$\mathfrak{T}\in\mathrm{iso(\mathbf{Stat})}$ to a statistical model
$(X,\,\mathcal{P})\coloneqq\mathrm{img}\mathcal{\mathfrak{T}}$. Then
$\mu\coloneqq w\circ\theta^{-1}$ is an identifiable parametrisation of
$(X,\,\mathcal{P})$ and since $w$ defines a probability distribution over
$\theta$ it may be regarded as the expected value
$\mathrm{E_{\theta}}[\boldsymbol{x}]$ with respect to $P_{\theta}$. By the
definition of $\varphi\coloneqq-w^{-1}\circ R\circ w$ it follows that:__

$P_{\mu}[\sigma]=\boldsymbol{\mu}_{P}\cdot\boldsymbol{Q}[\sigma]-\varphi(\boldsymbol{\mu}_{P})[\sigma]$
(3.3)

Without loss of generality any mixture family may therefore be assumed to be
given by equations 3.3. This representation is termed the canonical form of a
mixture family and the parametrisation $\mu$ the expectation parametrisation.

######  Definition (Expectation parametrisation).

_Let_

$(X,\,\mathcal{P}_{\theta})\in\mathrm{ob}(\mathbf{Stat})$

_Then a parametrisation $\mu$ with $\mathrm{dom}\mu\subseteq\mathbb{R}^{n}$,
which is given by $\boldsymbol{\mu}_{P}=\mathrm{E_{\theta}}[\boldsymbol{x}]$,
where $\mathrm{E_{\theta}}[\boldsymbol{x}]$ denotes the vectorial expectation
of $\boldsymbol{x}$ with respect to $P_{\theta}$, is termed an expectation
parametrisation with respect to $\theta$ and parameter vectors
$\boldsymbol{\mu}_{P}\in\mathrm{dom}\mu$ are termed expectation parameters._

######  Lemma 6.

Let $\psi$ be the cumulant generating function of an exponential family
_$(X,\,\mathcal{P}_{\eta})$_ and $\varphi$ the negative entropy
$-\mathrm{H}[P_{\eta}\mid X]$. Then the dual parametrisation $\eta{}^{*}$ is
given by the expectation parametrisation $\mu$ and the Legendre dual function
$\psi^{*}$ by $\varphi$, such that:

$\displaystyle\boldsymbol{\eta}_{P}^{*}$ $\displaystyle=$
$\displaystyle\boldsymbol{\mu}_{P}$ (3.4)
$\displaystyle\psi^{*}(\boldsymbol{\eta}_{P}^{*})$ $\displaystyle=$
$\displaystyle\varphi(\boldsymbol{\mu}_{P})$ (3.5)

###### Proof.

Let $P_{\eta}\in\mathcal{P}$ and $p_{\eta}$ the density function of $P_{\eta}$
over $(X,\,\Sigma_{X})$. Then from equation 2.5 is follows, that:

$\boldsymbol{\eta}_{P}^{*}=\nabla_{\eta}\psi(\boldsymbol{\eta}_{P})\stackrel{{\scriptstyle\ref{eq:expfam:nablaf}}}{{=}}\mathrm{E_{\eta}}[\boldsymbol{x}]\stackrel{{\scriptstyle\mathrm{def}}}{{=}}\boldsymbol{\mu}_{P}$

the dual parametrisation $\boldsymbol{\eta}_{P}^{*}$ directly follows from .
The Legendre dual function $\psi^{*}$ is derived by:

$\displaystyle\psi^{*}(\boldsymbol{\eta}_{P}^{*})$
$\displaystyle\stackrel{{\scriptstyle}}{{=}}$
$\displaystyle\boldsymbol{\eta}_{P}\cdot\nabla_{\eta}\psi(\boldsymbol{\eta}_{P})-\psi(\boldsymbol{\eta}_{P})$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:expfam:normalization}}}{{=}}$
$\displaystyle\int_{X}(\boldsymbol{\eta}_{P}\cdot\boldsymbol{x})p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}-\psi(\boldsymbol{\eta}_{P})\int_{X}p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$
$\displaystyle=$
$\displaystyle\int_{X}p_{\eta}(\boldsymbol{x})(\boldsymbol{\eta}_{P}\cdot\boldsymbol{x}-\psi(\boldsymbol{\eta}_{P}))\mathrm{d}\boldsymbol{x}$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:expfam:func}}}{{=}}$
$\displaystyle\int_{X}p_{\eta}(\boldsymbol{x})\log
p_{\eta}(\boldsymbol{x})\mathrm{d}\boldsymbol{x}$ $\displaystyle=$
$\displaystyle-\mathrm{H}[P_{\eta}\mid X]$

∎

######  Lemma 7.

Let $\psi$ be the cumulant generating function of an exponential family
_$(X,\,\mathcal{P}_{\eta})$_. Then the The Bregman divergence, given by the
Legendre dual function $\varphi$ is the Kullback-Leibler divergence:

$D_{\varphi}[P\parallel Q]=D_{\mathrm{KL}}[P\parallel Q]$

###### Proof.

From lemma LABEL:lem:3.9, equation LABEL:eq:divergence_dual_divergence it
follows, that:

$\displaystyle D_{\varphi}[P\parallel Q]$
$\displaystyle\stackrel{{\scriptstyle\text{\ref{eq:divergence_dual_divergence}}}}{{=}}$
$\displaystyle D_{\varphi^{*}}[Q_{\mu}^{*}\parallel P_{\mu}^{*}]$
$\displaystyle\stackrel{{\scriptstyle\mathrm{def}}}{{=}}$ $\displaystyle
D_{\psi}[Q_{\eta}\parallel P_{\eta}]$
$\displaystyle\stackrel{{\scriptstyle\text{\ref{eq:lem:3.14:1}}}}{{=}}$
$\displaystyle D_{\mathrm{KL}}[P\parallel Q]$

∎

######  Lemma 8.

Let $\psi$ be the cumulant generating function of an exponential family
_$(X,\,\mathcal{P}_{\eta})$_. Then the Bregman divergence $D_{\varphi}$ of the
Legendre dual function $\varphi=\psi^{*}$, induces a Riemannian metric, which
is given by the inverse Fisher information:

$g_{P,\varphi}=\mathrm{I}[P\mid X]^{-1}$ (3.6)

###### Proof.

By applying theorem LABEL:prop:3.2 the Bregman divergence
$D_{\varphi}=D_{\psi^{*}}$ induces the dual Riemannian metric
$g_{P,\psi}^{*}$, which by definition is inverse to the Riemannian metric,
induced by $D_{\psi}$, such that $g_{P,\varphi}=g_{P,\psi}^{-1},\,\forall
P\in\mathcal{P}$. From lemma 3 it follows, that
$g_{P,\varphi}=\mathrm{I}[P\mid X]^{-1}.$ ∎

######  Lemma 9.

Let $\psi$ be the cumulant generating function of an exponential family
$(X,\,\mathcal{P}_{\eta})$ and $(X,\,\mathcal{P},\,D_{\varphi})$ the
Riemannian statistical manifold, given by the Bregman divergence
$D_{\varphi}$of the Legendre dual function $\varphi=\psi^{*}$. Then the
$\mu$-affine geodesics in $(X,\,\mathcal{P}_{\mu},\,D_{\psi})$ are given by
mixture families.

###### Proof.

$\mu$-affine geodesics in $(X,\,\mathcal{P}_{\mu},\,D_{\psi})$ are given by
affine linear curves in the $\mu$-parametrisation. For $P,\,Q\in\mathcal{P}$
and
$\boldsymbol{\mu}_{P,Q}(t)=(1-t)\boldsymbol{\mu}_{P}+t\boldsymbol{\mu}_{Q}$,
with $t\in[0,\,1]$ let $\gamma_{P,Q}(t)$ be the $\mu$-affine geodesic
connecting $P$ with $Q$ and $\mathrm{d}_{x}\gamma_{P,Q}(t)$ the probability
density of $\gamma_{P,Q}(t)$ over $(X,\,\Sigma_{X})$, then:

$\mathrm{d}_{X}\gamma_{P,Q}(t)(\boldsymbol{x})=(1-t){}_{\boldsymbol{\mu}}(\boldsymbol{x})+tq_{\boldsymbol{\mu}}(\boldsymbol{x})$

An integration over $\sigma$ yields:

$\gamma_{P,Q}(t)=\int_{\sigma}\left((1-t)p(\boldsymbol{x})+tq(\boldsymbol{x})\right)\mathrm{d}\boldsymbol{x}$

This is a mixture of probability distributions with respect to a mixing
parameter $t$. ∎

######  Lemma 10.

Let $\psi$ be the cumulant generating function of an exponential family
$(X,\,\mathcal{P}_{\eta})$. Then the $\mu$-affine geodesics are flat with
regard to the Riemannian metric, induced by the Bregman divergence
$D_{\varphi}$of the Legendre dual function $\varphi=\psi^{*}$.

##  4 Dually flat structure

######  Theorem 11 (_Structure of Exponential Families_).

__Let $(X,\,\mathcal{P})$ be an exponential family. Then there exists a
Bregman divergence $D_{\psi}$ such that $(X,\,\mathcal{P_{\eta}},\,D_{\psi})$
is a dually flat statistical manifold with respect to the Riemannian metric,
induced by $D_{\psi}$.

###### Proof.

Let $\eta$ be the natural parametrisation and $\psi$ the cumulant generating
function of $(X,\,\mathcal{P})$. Let further be $D_{\psi}$ the Bregman
divergence over $(X,\,\mathcal{P})$ with respect to $\psi$, then
$(X,\,\mathcal{P_{\eta}},\,D_{\psi})$ is a Riemannian statistical manifold
with a Bregman divergence $D_{\psi}$. By lemma 3 it follows that $\eta$ is an
affine parametrisation and by lemma 5 that the $\eta$-affine geodesics are
flat with respect to the Riemannian metric, induced by $D_{\psi}$. Let $\mu$
be the expectation parametrisation of $(X,\,\mathcal{P})$ with respect to
$\eta$, then by lemma 1 it follows, that $\mu=\eta^{*}$, by lemma 8, that
$\mu$ is an affine parametrisation of $(X,\,\mathcal{P},\,D_{\psi^{*}})$ and
by lemma 10 that the $\mu$-affine geodesics are flat with respect to the
Riemannian metric, induced by $D_{\varphi}$, where $\varphi=\psi^{*}$.
Therefore the conditions of lemma LABEL:lem:3.10 are satisfied and
$(X,\,\mathcal{P_{\xi}},\,D_{\psi})$ is a dually flat statistical manifold. ∎

Together the primary and dual affine structure of an exponential family induce
a dually flat structure, which is characterized by the $\eta$-affine and
$\mu$-affine geodesics, with respect to the Fisher information metric and its
dual metric. Thereby the $\eta$-affine geodesics are represented by
exponential families over the curve parameter $t$ an the $\mu$-affine
geodesics by mixture families. This relationship allows a characterisation of
the intrinsic dually flat structure, which is independent of its
parametrisation. This structure is given by an _$e$ -affine structure_, that
preserves the exponential family representation within its primary affine
structure and an _$m$ -affine structure_ that preserves the dual
representation within the dual affine structure. Therefore geodesics and
geodesic projections in the $e$-affine structure are respectively termed _$e$
-geodesics_, denoted by $\gamma_{e}$ and _$e$ -affine projections_, denoted by
$\pi_{e}$. Furthermore the geodesics and geodesic projections in the
$m$-affine structure are respectively termed _$m$ -geodesics_, denoted by
$\gamma_{m}$ and _$m$ -affine projections_, denoted by $\pi_{m}$. With respect
to submanifolds, a smooth submanifold is termed _$e$ -flat_ if it has a linear
embedding within the $e$-affine structure and _$m$ -flat_ if it has a linear
embedding within the $m$-affine structure.

######  Corollary 12.

Let $(X,\,\mathcal{P})$ be an exponential family and $P\in\mathcal{P}$. Then
the geodesic projection of $P$ to an $m$-flat submanifold is uniquely given by
an $e$-affine projection $\pi_{e}$ and the geodesic projection of $P$ to an
$e$-flat submanifold is uniquely given by an $m$-affine projection $\pi_{m}$.

$(X,\,\mathcal{P})$$P$$\pi_{m}$$m\textrm{-flat}$$e\textrm{-flat}$$\pi_{e}$
Figure 4.1: Unique projection in Exponential Families

###### Proof.

Let $(X,\,\mathcal{P})$ be an exponential family. Then theorem 11 states, that
the structure of $(X,\,\mathcal{P})$ is given by a dually flat statistical
manifold $(X,\,\mathcal{P}_{\eta},\,D_{\psi})$, where the Riemannian metric is
induced by a Bregman divergence $D_{\psi}$ with respect to the natural
parametrisation $\eta$. Let $(X,\,\mathcal{Q})$ be an $m$-flat submanifold,
then $(X,\,\mathcal{Q})$ is flat with regard to the Riemannian metric, induced
by the expectation parametrisation and therefore flat with respect to
$D_{\psi^{*}}$. Conversely let $(X,\,\mathcal{S})$ be an $e$-flat submanifold,
then $(X,\,\mathcal{Q})$ is flat with regard to the Riemannian metric, induced
by the natural parametrisation and therefore flat with respect to $D_{\psi}$.
Therefore the conditions of corollary LABEL:cor:3.2 are satisfied, which
proves the corollary. ∎

##  5 Maximum Entropy Estimation in Exponential Families

The sample space $(X,\,\Sigma)$ of a statistical model $(X,\,\mathcal{P})$ is
generated by a statistic $T$, that induces a probability distribution $P_{X}$
from the probability space of an underlying statistical population
$(\Omega,\,\mathcal{F},\,P)$. To this end also the sample space may be
regarded as a probability space, whereat the occurrence of the probability
distribution $P_{X}$ is hypothetical.

######  Lemma 13.

Let $(X,\,\mathcal{P}_{\eta})$ be an exponential family over a sample space
$(X,\,\Sigma)$, and $P_{U}$ the uniform distribution over $(X,\,\Sigma)$. Then
$P_{U}\in\mathcal{P}$.

###### Proof.

Let $\mu$ be a $\sigma$-finite reference measure over $(X,\,\Sigma)$, then the
probability density $u$ of the uniform distribution is given by:

$u_{X}(x)\coloneqq\begin{cases}\frac{1}{\mu(X)}&x\in X\\\ 0&x\notin
X\end{cases}$

Let $\eta$ be the canonical parametrisation of $(X,\,\mathcal{P})$, then the
densities of any $P\in\mathcal{P}$ may be written as:

$p_{\eta}(\boldsymbol{x})=\exp\left(\boldsymbol{\eta}_{P}\cdot\boldsymbol{x}-\psi(\boldsymbol{\eta}_{P})\right)$

Where the cumulative generating function $\psi(\boldsymbol{\eta}_{P})$ is
given by:

$\psi(\boldsymbol{\eta}_{P})=\log\int_{X}\exp(\boldsymbol{\eta}_{P}\cdot\boldsymbol{x})\mathrm{d}\mu(\boldsymbol{x})$

Let $\boldsymbol{\eta}_{P}=0$, then:

$\displaystyle p_{\eta}(\boldsymbol{x})$ $\displaystyle=$
$\displaystyle\exp\left(0\cdot\boldsymbol{x}-\psi(0)\right)$ $\displaystyle=$
$\displaystyle\exp\left(\log\int_{X}\mathrm{d}\mu(\boldsymbol{x})\right)^{-1}$
$\displaystyle=$ $\displaystyle\mu(X)^{-1}$

This is the density of the uniform distribution over $(X,\,\Sigma)$. ∎

######  Lemma 14.

Let $(X,\,\Sigma)$ be a measurable space and $P_{U}$ the uniform distribution
over $(X,\,\Sigma)$. Then let $P$ be an arbitrary probability distribution
over $(X,\,\Sigma)$, then:

$\mathrm{H}[P_{U}\mid X]\geq\mathcal{\mathrm{H}}[P\mid X]$

###### Proof.

Let $u_{X}$ be the probability density of $P_{U}$, then the entropy of $P_{U}$
over $(X,\,\Sigma)$ is:

$\displaystyle\mathrm{H}[P_{U}\mid X]$ $\displaystyle=$
$\displaystyle-\int_{X}u_{X}(x)\log u_{X}(x)\mathrm{d}\mu(x)$ $\displaystyle=$
$\displaystyle-\int_{X}\mu(X)^{-1}\log\mu(X)^{-1}\mathrm{d}\mu(x)$
$\displaystyle=$ $\displaystyle\log\mu(X)^{-1}$

Let $P$ be an arbitrary probability distribution over $(X,\,\Sigma)$ with
density $p$, then:

$\displaystyle D_{\mathrm{KL}}[P\parallel P_{U}]$ $\displaystyle=$
$\displaystyle\int_{X}p(x)\log\frac{p(x)}{u_{X}(x)}\mathrm{d}\mu(x)$
$\displaystyle=$ $\displaystyle\int_{X}p(x)\log p(x)\mathrm{d}\mu(x)$
$\displaystyle-\int_{X}p(x)\log u_{X}(x)\mathrm{d}\mu(x)$ $\displaystyle=$
$\displaystyle-\mathrm{H}[P\mid X]-\log\mu(X)^{-1}$ $\displaystyle=$
$\displaystyle-\mathcal{\mathrm{H}}[P\mid X]+\mathcal{H}[P_{U}\mid X]\geq 0$

Therefore:

$\mathcal{H}[P_{U}\mid X]\geq\mathcal{H}[P\mid X],\,\forall P$

∎

######  Lemma 15.

Let $(X,\,\mathcal{P})$ be an exponential family over a sample space
$(X,\,\Sigma)$. Then a maximum entropy estimation of $(X,\,\mathcal{P})$ is
given by the uniform probability distribution over $(X,\,\mathcal{P})$.

###### Proof.

From lemma 13 it follows, that $P_{U}\in\mathcal{P}$. Furthermore lemma 14
proves that $P_{U}$ maximizes the entropy among all $P\in\mathcal{P}$. ∎

######  Theorem 16.

Let $(X,\,\mathcal{P})$ be an exponential family over a sample space
$(X,\,\Sigma)$ and $(X,\,\mathcal{Q})$ a smooth submanifold, then a maximum
entropy estimation of $(X,\,\mathcal{Q})$ is given by a geodesic projection of
the uniform probability distribution over $(X,\,\mathcal{P})$ to
$(X,\,\mathcal{Q})$.

$(X,\,\mathcal{P})$$(X,\,\mathcal{Q})$$P_{U}$$\pi$$\hat{\theta}_{\mathrm{ME}}$
Figure 5.1: ME estimation in Exponential Families

###### Proof.

[todo] ∎

By assuming a given event $\sigma\in\Sigma$ within a sample space
$(X,\,\Sigma)$, the principle of maximum entropy emphasizes a probability
distribution, that minimizes additional assumptions. Then lemma 15 states,
that the maximum entropy estimation of a single observation within the set of
all probability distributions is given by the uniform probability distribution
over this observation, such that:

$u_{\sigma}(x)\coloneqq\begin{cases}\frac{1}{\mu(\sigma)}&,\,x\in\sigma\\\
0&,\,x\notin\sigma\end{cases}$

The dispensation of any additional knowledge except the observation itself
determines the uniform probability distribution as the empirical probability
of a single observation. This shall be extended to repeated observations. Let
$\boldsymbol{\sigma}=(\sigma_{i})_{i\in I}$ be a repeated observation in
$(X,\,\Sigma)$, then the density function of a finite measure over
$(X,\,\Sigma)$ is given by the arithmetic mean of the uniform distributions of
the individual single observations:

$f_{\boldsymbol{\sigma}}(x)=\frac{1}{|I|}\sum_{i\in
I}u_{\sigma_{i}}(x)=\frac{1}{|I|}\sum_{i\in
I}\frac{\delta_{x}(\sigma_{i})}{\mu(\sigma_{i})}$

Where $\mu$ is a $\sigma$-finite reference measure and $\delta$ the Dirac
measure and defined by:

$\delta_{x}(\sigma)=\begin{cases}1&,\,x\in\sigma\\\
0&,\,x\notin\sigma\end{cases}$

Then $f_{\boldsymbol{\sigma}}$ is a probability density function over
$(X,\,\Sigma)$, since $f_{\boldsymbol{\sigma}}(x)>0,\,\forall x\in X$ and:

$\displaystyle\int_{X}f_{\boldsymbol{\sigma}}(x)\mathrm{d}\mu(x)$
$\displaystyle=$ $\displaystyle\int_{X}\frac{1}{|I|}\sum_{i\in
I}\frac{\delta_{x}(\sigma_{i})}{\mu(\sigma_{i})}\mathrm{d}\mu(x)$
$\displaystyle=$ $\displaystyle\frac{1}{|I|}\sum_{i\in
I}\frac{1}{\mu(\sigma_{i})}\left(\int_{X}\delta_{x}(\sigma_{i})\mathrm{d}\mu(x)\right)$
$\displaystyle=$ $\displaystyle\frac{1}{|I|}\sum_{i\in
I}\frac{\mu(X\cap\sigma_{i})}{\mu(\sigma_{i})}=1$

Furthermore the assumptions given by $f_{\boldsymbol{\sigma}}$ are equal the
knowledge which is given by the repeated observation $\boldsymbol{\sigma}$.
This determines $f_{\boldsymbol{\sigma}}$ as the density of an empirical
probability distribution.

######  Definition (Empirical probability distribution).

_Let $(X,\,\Sigma)$ be a measurable space,
$\boldsymbol{\sigma}=(\sigma_{i})_{i\in I}$ a repeated observation in
$(X,\,\Sigma)$ and $\mu$ a $\sigma$-finite reference measure over
$(X,\,\Sigma)$. Then the empirical probability distribution of
$\boldsymbol{\sigma}$ over $(X,\,\Sigma)$ is given by:_

$P_{\boldsymbol{\sigma}}[A]\coloneqq\int_{A}\frac{1}{|I|}\sum_{i\in
I}\frac{\delta_{x}(\sigma_{i})}{\mu(\sigma_{i})}\mathrm{d}\mu(x),\,\forall
A\in\Sigma$

######  Proposition 17.

Let $(X,\,\Sigma)$ be a sample space, $P_{X}$ be observable distribution of
$(X,\,\Sigma)$ and $\boldsymbol{\sigma}(n)=(\sigma_{i})_{i\leq n}$ the partial
sequences of a repeated observation
$\boldsymbol{\sigma}=(\sigma_{i})_{i\in\mathbb{N}}$ in $(X,\,\Sigma)$. Then
the sequence of the empirical probability distributions
$P_{\boldsymbol{\sigma}(n)}$ converges to $P_{X}$ as $n\to\infty$.

###### Proof.

Let $p_{X}$ be the probability density function of $P_{X}$. Then due to the
strong law of large numbers it follows, that:

$\lim_{n\to\infty}\frac{1}{n}\sum_{i\in
1}^{n}\frac{\delta_{x}(\sigma_{i})}{\mu(\sigma_{i})}\stackrel{{\scriptstyle
a.s.}}{{=}}\mathrm{E}[\delta_{x}(y)]=\int_{X}\delta_{x}(y)\mathrm{d}p_{X}(y)$
(5.1)

Let $A\in\Sigma$, then the limit of the empirical probability distributions
$P_{n}[A]$ is given by:

$\displaystyle\lim_{n\to\infty}P_{n}[A]$ $\displaystyle=$
$\displaystyle\lim_{n\to\infty}\int_{A}\frac{1}{n}\sum_{i\in
1}^{n}\frac{\delta_{\\{x\\}}(\sigma_{i})}{\mu(\sigma_{i})}\mathrm{d}\mu(x)$
$\displaystyle=$
$\displaystyle\int_{A}\left(\lim_{n\to\infty}\frac{1}{n}\sum_{i\in
1}^{n}\frac{\delta_{\\{x\\}}(\sigma_{i})}{\mu(\sigma_{i})}\right)\mathrm{d}\mu(x)$
$\displaystyle\stackrel{{\scriptstyle a.s.}}{{=}}$
$\displaystyle\int_{A}\int_{X}\delta_{\\{x\\}}(y)\mathrm{d}p_{X}(y)\mathrm{d}\mu(x)$
$\displaystyle=$ $\displaystyle\int_{A}p_{X}(x)\mathrm{d}\mu(x)$
$\displaystyle=$ $\displaystyle P_{X}[A]$

∎

##  6 Maximum Likelihood Estimation in Exponential Families

######  Lemma 18.

Let $(X,\,\mathcal{Q})$ be a statistical model over a sample space
$(X,\,\Sigma)$ and $\boldsymbol{\sigma}$ a finite repeated observation in
$(X,\,\Sigma)$. Then the MLE has the following representation:

$\hat{\theta}_{\mathrm{ML}}\in\arg\max_{\theta}\sum_{i=1}^{n}\log\mathrm{L}[Q_{\theta}\mid\sigma_{i}]$
(6.1)

###### Proof.

Since the $\log$ transformation is strictly monotonous over
$\mathrm{img}\mathrm{L}\subseteq[0,\,1]$ it directly follows, that:

$\arg\max_{\theta}\mathrm{L}[Q_{\theta}\mid\boldsymbol{\sigma}]=\arg\max_{\theta}\log\mathrm{L}[Q_{\theta}\mid\boldsymbol{\sigma}]$

Furthermore due to pairwise independence of the individual observations are it
follows, that:

$\log\mathrm{L}[Q_{\theta}\mid\boldsymbol{\sigma}]=\log\mathrm{\prod_{i=1}^{n}}\mathrm{L}[Q_{\theta}\mid\sigma_{i}]=\sum_{i=1}^{n}\log\mathrm{L}[Q_{\theta}\mid\sigma_{i}]$

And therefore:

$\displaystyle\hat{\theta}_{\mathrm{ML}}$ $\displaystyle\in$
$\displaystyle\arg\max_{\theta}\mathrm{L}[Q_{\theta}\mid\boldsymbol{\sigma}]$
$\displaystyle=$
$\displaystyle\arg\max_{\theta}\sum_{i=1}^{n}\log\mathrm{L}[Q_{\theta}\mid\sigma_{i}]$

∎

######  Lemma 19.

Let $(X,\,\mathcal{Q})$ be a statistical model over a sample space
$(X,\,\Sigma)$ and $\boldsymbol{\sigma}$ a finite repeated observation in
$(X,\,\Sigma)$. Let further be $p_{\boldsymbol{\sigma}}$ the empirical
probability density with respect to $\boldsymbol{\sigma}$. Then the MLE has
the following representation:

$\hat{\theta}_{\mathrm{ML}}\in\arg\max_{\theta}\int_{X}p_{\boldsymbol{\sigma}}(x)\log
q_{\theta}(x)\mathrm{d}\mu(x)$ (6.2)

###### Proof.

By substitution of the empirical probability density
$p_{\boldsymbol{\sigma}}(x)$ it follows that:

$\displaystyle\int_{X}p_{\boldsymbol{\sigma}}(x)\log
q_{\theta}(x)\mathrm{d}\mu(x)$ $\displaystyle=$
$\displaystyle\int_{X}\left(\frac{1}{n}\sum_{i\in
1}^{n}\frac{1}{\mu(\sigma_{i})}\delta_{\\{x\\}}(\sigma_{i})\right)\log
q_{\theta}(x)\mathrm{d}\mu(x)$ $\displaystyle=$
$\displaystyle\frac{1}{n}\sum_{i\in
1}^{n}\frac{1}{\mu(\sigma_{i})}\int_{\sigma_{i}}\log
q_{\theta}(x)\mathrm{d}\mu(x)$ $\displaystyle=$
$\displaystyle\frac{1}{n}\sum_{i\in
1}^{n}\frac{1}{\mu(\sigma_{i})}\left(\mu(\sigma_{i})\log\mathrm{L}[Q_{\theta}\mid\sigma_{i}]-\mu(\sigma_{i})\right)$
$\displaystyle=$ $\displaystyle\frac{1}{n}\sum_{i\in
1}^{n}\log\mathrm{L}[Q_{\theta}\mid\sigma_{i}]-1$

The maximization of equation 6 with respect to $\theta$ therefore yields the
following identity:

$\displaystyle\arg\max_{\theta}\int_{X}p_{\boldsymbol{\sigma}}(x)\log
q_{\theta}(x)\mathrm{d}\mu(x)$ $\displaystyle=$
$\displaystyle\arg\max_{\theta}\left(\frac{1}{n}\sum_{i\in
I}\log\mathcal{\mathrm{L}}[Q_{\theta}\mid\sigma_{i}]-1\right)$
$\displaystyle=$ $\displaystyle\arg\max_{\theta}\sum_{i\in
I}\log\mathrm{L}[Q_{\theta}\mid\sigma_{i}]$

By lemma 18, equation 6.1 it follows, that:

$\hat{\theta}_{\mathrm{ML}}\in\arg\max_{\theta}\int_{X}p_{\boldsymbol{\sigma}}(x)\log
q_{\theta}(x)\mathrm{d}\mu(x)$

∎

######  Theorem 20.

Let $(X,\,\mathcal{P})$ be an exponential family over a sample space
$(X,\,\Sigma)$. Let further be $(X,\,\mathcal{Q})$ a smooth submanifold of
$(X,\,\mathcal{P})$ and $\boldsymbol{\sigma}$ a repeated observation in
$(X,\,\Sigma)$. Then a maximum likelihood estimation of $(X,\,\mathcal{Q})$
respective to $\boldsymbol{\sigma}$ is given by the geodesic projection of the
empirical probability $P_{\boldsymbol{\sigma}}$ to $(X,\,\mathcal{Q})$ in
$(X,\,\mathcal{P})$.

$(X,\,\mathcal{P})$$(X,\,\mathcal{Q})$$P_{\boldsymbol{\sigma}}$$\pi$$\hat{\theta}_{\mathrm{ML}}$
Figure 6.1: ML estimation in Exponential Families

###### Proof.

Let $(X,\,\mathcal{P}_{\eta})$ be an exponential family over a sample space
$(X,\,\Sigma)$ and $(X,\,\mathcal{Q})$ a smooth submanifold of
$(X,\,\mathcal{P})$. Then there exists a Bregman divergence $D_{\psi}$, such
that $(X,\,\mathcal{P}_{\eta},\,D_{\psi})$ is a dually flat statistical
manifold and $\eta$ is the $e$-affine parametrisation of
$(X,\,\mathcal{P}_{\eta},\,D_{\psi})$. Then
$(X,\,\mathcal{Q}_{\eta},\,D_{\psi})$ is a smooth Riemannian submanifold of
$(X,\,\mathcal{P}_{\eta},\,D_{\psi})$ with respect to the induced Riemannian
metric $D_{\psi}$. Let $\boldsymbol{\sigma}$ be a repeated observation over
$(X,\,\Sigma)$ and $P_{\boldsymbol{\sigma}}$ its empirical probability
distribution.

Then $P_{\boldsymbol{\sigma}}$ is given in expectation parameters in
$(X,\,\Sigma)$ and therefore in an $m$-affine parametrisation within
Riemannian Manifold $(X,\,\mathcal{P}_{\eta},\,D_{\psi})$. By applying the
projection theorem the geodesic projection of $P_{\boldsymbol{\sigma}}$ to
$(X,\,\mathcal{Q}_{\eta},\,D_{\psi})$, equals the dual affine projection, and
therefore by a point $Q\in\mathcal{Q}$, that minimizes the Bregman divergence
$D_{\psi}[Q\parallel P_{\boldsymbol{\sigma}}]$. The geodesic distance is
therefore given by:

$\displaystyle d(P_{\boldsymbol{\sigma}},\,Q)$ $\displaystyle=$ $\displaystyle
D_{\psi}[Q\parallel P_{\boldsymbol{\sigma}}]$ $\displaystyle=$ $\displaystyle
D_{\mathrm{KL}}[P_{\boldsymbol{\sigma}}\parallel Q]$ $\displaystyle=$
$\displaystyle\int_{X}p_{\boldsymbol{\sigma}}(x)\log\frac{p_{\boldsymbol{\sigma}}(x)}{q(x)}\mathrm{d}\mu(x)$
$\displaystyle=$ $\displaystyle\int_{X}p_{\boldsymbol{\sigma}}(x)\log
p_{\boldsymbol{\sigma}}(x)\mathrm{d}\mu(x)$
$\displaystyle-\int_{X}p_{\boldsymbol{\sigma}}(x)\log q(x)\mathrm{d}\mu(x)$
$\displaystyle=$ $\displaystyle-\mathrm{H}[P_{\sigma}\mid
X]-\int_{X}p_{\boldsymbol{\sigma}}(x)\log q(x)\mathrm{d}\mu(x)$

The minimization of $d(P_{\boldsymbol{\sigma}},\,Q)$ with respect to the
natural parametrisation $\eta$ therefore yields the following identity:

$\arg\min_{\eta}d(P_{\boldsymbol{\sigma}},\,Q_{\eta})=\arg\max_{\eta}\int_{X}p_{\boldsymbol{\sigma}}(x)\log
q_{\eta}(x)\mathrm{d}\mu(x)$ (6.4)

By equation 6.2 and lemma, equation 6.4 it follows, that:

$\displaystyle\hat{\theta}_{\mathrm{ML}}$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:lem:4.3:1}}}{{\in}}$
$\displaystyle\arg\max_{\eta}\int_{X}p_{\boldsymbol{\sigma}}(x)\log
q_{\eta}(x)\mathrm{d}\mu(x)$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:prop:4.1:1}}}{{=}}$
$\displaystyle\arg\min_{\eta}d(P_{\boldsymbol{\sigma}},\,Q_{\eta})$

∎

##  7 Latent variable models

Exponential families, as introduced in the previous sections, statistically
relate random variables over a common statistical population by their common
probability distribution in the sample space. In many cases however the
intrinsic structure of this relationship has a natural decomposition by the
introduction of latent random variables, that are not directly observable from
the statistical population but assumed to affect the observations. This is of
particular importance for the modelling of statistical populations with
complex network structures. In this case the properties of the network may be
incorporated by the conditional transition probabilities between observable
and latent random variables.

In completely observable statistical models, the probability distributions may
be estimated by the empirical probability distributions of repeated
observations. In latent variable models however the conditional transition
probabilities $p(v\mid h)$ and $p(h\mid v)$ between the observables $v\in V$
and the latent variables $h\in H$ in general prevent this inference. The only
exception is given if $p(v\mid h)$ and $p(h\mid v)$ are uniform distributed,
such that for any given $v$ any $h$ has the same probability with respect to
$v$ and vice versa. In this case estimations decompose into independent
estimations of the observable variables and the latent variables. If the
conditional transition probabilities, however are not uniform distributed,
they have to be taken into account for estimations. This also applies to
empirical distributions. Let $\boldsymbol{\sigma}(n)=(\sigma_{i})_{i\leq n}$
be the partial sequences of a repeated observation
$\boldsymbol{\sigma}=(\sigma_{i})_{i\in\mathbb{N}}$ in $(V,\,\Sigma_{v})$,
then by proposition 17 it follows, that the empirical probabilities
$P_{\boldsymbol{\sigma}(n)}$ converge in distribution to the true probability
distribution $P_{V}$ of $(V,\,\Sigma_{v})$. Since $P_{V}$ however is the
marginal distribution of the observables in _$(X,\,\Sigma)$_ the common
empirical probabilities over _$(X,\,\Sigma)$_ are constituted by an empirical
probability of a repeated observation and a conditional transition
probability. The probability density $p_{\boldsymbol{\sigma}}$ of an empirical
probability over $(X,\,\Sigma)$ is therefore given by:

$p_{\boldsymbol{\sigma}}(x)=p_{\boldsymbol{\sigma}}(v,\,h)=p_{\boldsymbol{\sigma}}(v)p(h\mid
v),\,\forall x\in X$

In the presence of continuous variables it is useful to restrict the empirical
probability distributions to “non pathological” cases. This restriction
defines statistical model with respect to empirical observations.

######  Definition (Empirical model).

_Let $(X,\,\Sigma)$ be a partially observable measurable space with $X=V\times
H$. Then a statistical model $(X,\,\mathcal{E})$ is termed an empirical model
over $(X,\,\Sigma)$ if $\mathcal{E}$ comprises all empirical probability
distributions over $(X,\,\Sigma)$, which are constituted by a finite repeated
observation and a conditional transition probability of a given set
$\mathcal{T}$. If $\mathcal{T}$ is the set of all absolutely continuous
conditional transition probabilities, then $(X,\,\mathcal{E})$ is termed an
absolutely continuous empirical model._

##  8 Exponential families with latent variables

Let $(X,\,\mathcal{P})$ be an exponential family over a partially observable
measurable space $(X,\,\mathcal{P})$. Then due to theorem 11 the structure of
$(X,\,\mathcal{P})$ is that of a dually flat manifold such that there exists a
parametrisation $\eta$ and a convex function $\psi$, that allow to regard
$(X,\,\mathcal{P})$ as a dually flat statistical manifold, given by
$(X,\,\mathcal{P}_{\eta},\,D_{\psi})$. In the purpose of observation based
estimations in the presence of latent variables, the obstacle that has to be
taken is the extension of $(X,\,\mathcal{P}_{\eta},\,D_{\psi})$ to a dually
flat embedding space $(X,\,\mathcal{U},\,D)$, that covers
$(X,\,\mathcal{P}_{\eta},\,D_{\psi})$ as well as the empirical model
$(X,\,\mathcal{E})$. For arbitrary empirical models however this embedding
does generally not exist, for which $(X,\,\mathcal{E})$ is assumed to be
absolutely continuous. Then $(X,\,\mathcal{E})$ is generally an infinite
dimensional exponential family.

######  Lemma 21.

Let $(X,\,\mathcal{E})$ be an absolutely continuous empirical model over a
partially observable measurable space $(X,\,\Sigma)$. Then $(X,\,\mathcal{E})$
is an infinite dimensional exponential family and its probability densities
are given by:

$\displaystyle p_{\boldsymbol{\sigma}}(\boldsymbol{v},\,\boldsymbol{h})$
$\displaystyle=\frac{\exp\int_{H}\eta_{s}(\boldsymbol{v},\,\boldsymbol{r})\delta(\boldsymbol{h}-\boldsymbol{r})\mathrm{d}\boldsymbol{r}}{\exp\psi_{s}(\eta_{s}(\boldsymbol{v},\,\boldsymbol{h}))}$

where $\boldsymbol{v}\mathbb{\in R}^{k}$ and $\boldsymbol{h}\in\mathbb{R}^{l}$
respectively denote the observable and latent variables,
$\eta_{s}(\boldsymbol{v},\,\boldsymbol{h})$ scalar coefficients and $\psi_{s}$
the cumulant generating function, given by:

$\psi_{s}(\eta_{s}(\boldsymbol{v},\,\boldsymbol{h}))=\log\int_{\mathbb{R}^{l}}\exp(\eta_{s}(\boldsymbol{v},\,\boldsymbol{r}))\mathrm{d}\boldsymbol{r}$
(8.1)

###### Proof.

Due to the definition of an empirical model the probability density
$p_{\boldsymbol{\sigma}}$ of any $P\in\mathcal{E}$ may be written as:

$p_{\boldsymbol{\sigma}}(\boldsymbol{v},\,\boldsymbol{h})=\frac{1}{n}\sum_{i=1}^{n}\delta(\boldsymbol{v}-\boldsymbol{s}_{i})p(\boldsymbol{h}\mid\boldsymbol{s}_{i})$
(8.2)

Furthermore any conditional transition probability
$p(\boldsymbol{h}\mid\boldsymbol{v})$ is given by:

$p(\boldsymbol{h}\mid\boldsymbol{v})=\int_{H}\delta(\boldsymbol{h}-\boldsymbol{r})p(\boldsymbol{r}\mid\boldsymbol{v})\mathrm{d}\boldsymbol{r}$
(8.3)

Therefore by equation 8.2 and 8.3 it follows, that:

$p_{s}(\boldsymbol{v},\,\boldsymbol{h})=\frac{1}{n}\sum_{i=1}^{n}\delta(\boldsymbol{v}-\boldsymbol{s}_{i})\int_{\mathbb{R}^{l}}\delta(\boldsymbol{h}-\boldsymbol{r})p(\boldsymbol{r}\mid\boldsymbol{s}_{i})\mathrm{d}\boldsymbol{r}$
(8.4)

Since $p(\boldsymbol{h}\mid\boldsymbol{v})$ are absolutely continues equation
8.4 may be rewritten to:

$p_{s}(\boldsymbol{v},\,\boldsymbol{h})=\int_{H}\left(\frac{1}{n}\sum_{i=1}^{n}\delta(\boldsymbol{v}-\boldsymbol{s}_{i})p(\boldsymbol{r}\mid\boldsymbol{s}_{i})\right)\delta(\boldsymbol{h}-\boldsymbol{r})\mathrm{d}\boldsymbol{r}$

By the substitution

$\mu_{s}(\boldsymbol{v},\,\boldsymbol{r})=\frac{1}{n}\sum_{i=1}^{n}\delta(\boldsymbol{v}-\boldsymbol{s}_{i})p(\boldsymbol{r}\mid\boldsymbol{s}_{i})$
(8.5)

it follows, that the the empirical probabilities in $\mathcal{E}$ may be
written as a mixture with mixing coefficients
$\mu_{s}(\boldsymbol{v},\,\boldsymbol{r})$:

$p_{s}(\boldsymbol{v},\,\boldsymbol{h})=\int_{H}\mu_{s}(\boldsymbol{v},\,\boldsymbol{r})\delta(\boldsymbol{h}-\boldsymbol{r})\mathrm{d}\boldsymbol{r}$
(8.6)

This shows, that $\mathcal{E}$ is an infinite dimensional mixture family. By a
further transformation:

$\eta_{s}(\boldsymbol{v},\,\boldsymbol{h})=\log\mu_{s}(\boldsymbol{v},\,\boldsymbol{h})+\psi_{s}$

a dual representation of equation 8.6 is obtained, with:

$\displaystyle p_{s}(\boldsymbol{v},\,\boldsymbol{h})=$
$\displaystyle\frac{\exp\int_{\mathbb{R}^{l}}\eta_{s}(\boldsymbol{v},\,\boldsymbol{r})\delta(\boldsymbol{h}-\boldsymbol{r})\mathrm{d}\boldsymbol{r}}{\exp\psi_{s}(\eta_{s}(\boldsymbol{v},\,\boldsymbol{h}))}$
(8.7)

This shows, that $(X,\,\mathcal{E})$ is also an infinite dimensional
exponential family with natural variables
$\eta_{s}(\boldsymbol{v},\,\boldsymbol{r})$ and the cumulative generating
function is given by

$\text{
}\psi_{s}(\eta_{s}(\boldsymbol{v},\,\boldsymbol{h}))=\log\int_{\mathbb{R}^{l}}\exp(\eta_{s}(\boldsymbol{v},\,\boldsymbol{r}))\mathrm{d}\boldsymbol{r}$

∎

######  Lemma 22.

Let $(X,\,\mathcal{P})$ be an exponential family over a partially observable
measurable space $(X,\,\Sigma)$ and $(X,\,\mathcal{E})$ an absolutely
continuous empirical model over $(X,\,\Sigma)$. Then there exists a dually
flat statistical manifold $(X,\,\mathcal{U},\,D)$, that covers
$(X,\,\mathcal{P})$ and $(X,\,\mathcal{E})$ as dually flat submanifolds.

###### Proof.

With respect to the partially observable measurable space $(X,\,\Sigma)$ any
$p\in\mathcal{P}$ may be written as
$p(\boldsymbol{x})=p(\boldsymbol{v},\,\boldsymbol{h})$ and any
$p_{\boldsymbol{s}}\in\mathcal{E}$ as
$p_{s}(\boldsymbol{x})=p_{s}(\boldsymbol{v},\,\boldsymbol{h})$. Let
$\mathcal{U}$ be given by:

$\mathcal{U}=\left\\{q\mid
q(\boldsymbol{v},\,\boldsymbol{h})=p(\boldsymbol{v},\,\boldsymbol{h})p_{s}(\boldsymbol{v},\,\boldsymbol{h}),\,p\in\mathcal{P},\,p_{s}\in\mathcal{E}\right\\}$

Since $(X,\,\mathcal{P})$ is an exponential family it has a natural
parametrisation $\boldsymbol{\eta}_{\mathcal{P}}$ and due to Lemma 21 the
continuous empirical model $(X,\,\mathcal{E})$ is an infinite dimensional
exponential family with natural coefficients
$\boldsymbol{\eta}_{\mathcal{E}}=(\eta_{s}(\boldsymbol{v},\,\boldsymbol{h})_{v,h})^{T}$.
Then a parametrisation of $(X,\,\mathcal{U})$ is given by:

$\boldsymbol{\eta}=(\boldsymbol{\eta}_{\mathcal{P}},\,\boldsymbol{\eta}_{\mathcal{E}})^{T}$

Let further be $\psi_{\mathcal{P}}$ the cumulant generating function of
$(X,\,\mathcal{P})$ and $\psi_{\mathcal{E}}$ the cumulant generating function
of $(X,\,\mathcal{E})$, then $\psi_{\mathcal{P}}$ and $\psi_{\mathcal{E}}$ are
convex functions and therefore a convex function over $\boldsymbol{\eta}$ is
given by:

$\psi(\boldsymbol{\eta})=\psi_{\mathcal{P}}(\boldsymbol{\eta}_{\mathcal{P}})+\psi_{\mathcal{E}}(\boldsymbol{\eta}_{\mathcal{E}})$

This allows the definition of a Bregman divergence $D_{\psi}$, such that
$D_{\psi}$ induces Riemannian metric over $(X,\,\mathcal{U})$. By substitution
of equation 8.7 it follows that the application of $\boldsymbol{\eta}$ to
$(X,\,\mathcal{U})$ yields a parametric representation, which is given by:

$\displaystyle q_{\boldsymbol{\eta}}(\boldsymbol{v},\,\boldsymbol{h})$
$\displaystyle=\exp\left(\int_{\mathbb{R}^{l}}\eta_{s}(\boldsymbol{v},\,\boldsymbol{r})\delta(\boldsymbol{h}-\boldsymbol{r})\mathrm{d}\boldsymbol{r}\right.$
$\displaystyle\left.+\boldsymbol{\eta}_{\mathcal{P}}\cdot(\boldsymbol{v},\,\boldsymbol{h})^{T}-\psi(\boldsymbol{\eta})\right)$

This shows that $(X,\,\mathcal{U}_{\boldsymbol{\eta}})$ is an exponential
family and furthermore, that
$(X,\,\mathcal{U}_{\boldsymbol{\eta}},\,D_{\psi})$ is a dually flat
statistical manifold, that covers $(X,\,\mathcal{P})$ and $(X,\,\mathcal{E})$
as smooth submanifolds. Since the projections
$\boldsymbol{\eta}\to\boldsymbol{\eta}_{\mathcal{P}}$ and
$\boldsymbol{\eta}\to\boldsymbol{\eta}_{\mathcal{E}}$ are linear in the
$e$-parametrisation $(X,\,\mathcal{P})$ and $(X,\,\mathcal{E})$ are $e$-flat
with respect to the induced metric $D_{\psi}$. Let
$\boldsymbol{\mu}_{\mathcal{P}}$ be the expectation parameters of
$(X,\,\mathcal{P})$ and
$\boldsymbol{\mu}_{\mathcal{E}}=(\mu_{s}(\boldsymbol{v},\,\boldsymbol{h})_{v,h})^{T}$
expectation coefficients $(X,\,\mathcal{E})$, then the dual parametrisation
$\boldsymbol{\mu}$ is given by:

$\displaystyle\boldsymbol{\mu}$ $\displaystyle=$
$\displaystyle\nabla\psi_{\mathcal{P}}(\boldsymbol{\eta}_{\mathcal{P}})+\nabla\psi_{\mathcal{E}}(\boldsymbol{\eta}_{\mathcal{E}})$
$\displaystyle=$
$\displaystyle(\boldsymbol{\mu}_{\mathcal{P}},\,0)^{T}+(0,\,\boldsymbol{\mu}_{\mathcal{E}})^{T}$
$\displaystyle=$
$\displaystyle(\boldsymbol{\mu}_{\mathcal{P}},\,\boldsymbol{\mu}_{\mathcal{E}})^{T}$

Since the projections $\boldsymbol{\mu}\to\boldsymbol{\mu}_{\mathcal{P}}$ and
$\boldsymbol{\mu}\to\boldsymbol{\mu}_{\mathcal{E}}$ are linear in the
$m$-parametrisation $(X,\,\mathcal{P})$ and $(X,\,\mathcal{E})$ are $m$-flat
with respect to the induced metric $D_{\psi}$. Therefore
$(X,\,\mathcal{P},\,D_{\psi})$ and $(X,\,\mathcal{E},\,D_{\psi})$ are a dually
flat submanifolds of $(X,\,\mathcal{U},\,D_{\psi})$. ∎

##  9 Maximum Likelihood Estimation in Exponential Families with Latent
Variables

Due to the existence of a simply connected embedding space
$(X,\,\mathcal{U},\,D_{\varphi})$ that covers $(X,\,\mathcal{P})$ as well as
$(X,\,\mathcal{E})$, also arbitrary smooth submanifolds $(X,\,\mathcal{Q})$ of
$(X,\,\mathcal{P})$ may be connected to submanifolds
$(X,\,\mathcal{E}_{\boldsymbol{\sigma}})$ of $(X,\,\mathcal{E})$, given by a
repeated observation $\boldsymbol{\sigma}$. Since
$(X,\,\mathcal{U},\,D_{\varphi})$ is furthermore a Riemannian statistical
manifold and $(X,\,\mathcal{Q})$ and $(X,\,\mathcal{E}_{\boldsymbol{\sigma}})$
are smooth submanifolds of $(X,\,\mathcal{U},\,D_{\varphi})$ geodesics between
$(X,\,\mathcal{Q})$ and $(X,\,\mathcal{E}_{\boldsymbol{\sigma}})$ are given by
the induced Riemannian metric $D_{\varphi}$.

######  Theorem 23.

Let $(X,\,\mathcal{P})$ be an exponential family over a partially observable
measurable space $(X,\,\Sigma)$ and $(X,\,\mathcal{E})$ an absolutely
continuous empirical model over $(X,\,\Sigma)$. Let further be
$(X,\,\mathcal{Q})$ a smooth submanifold of $(X,\,\mathcal{P})$, and
$\boldsymbol{\sigma}$ a repeated observation in $(X,\,\Sigma)$. Then a maximum
likelihood estimation of $(X,\,\mathcal{Q})$ respective to
$\boldsymbol{\sigma}$ is given by a minimal geodesic projection of
$(X,\,\mathcal{E}_{\boldsymbol{\sigma}})$ to $(X,\,\mathcal{Q})$.

$(X,\,\mathcal{U})$$(X,\,\mathcal{Q})$$\hat{P}_{\boldsymbol{\sigma}}$$\pi$$(X,\,\mathcal{E}_{\boldsymbol{\sigma}})$$\hat{\theta}_{\mathrm{ML}}$
Figure 9.1: ML estimation in latent variable Exponential Families

###### Proof.

Since $(X,\,\mathcal{P})$ is an exponential family and $(X,\,\mathcal{E})$ a
continuous empirical model, there exists a common dually flat embedding space
$(X,\,\mathcal{U}_{\eta},\,D_{\psi})$, such that $(X,\,\mathcal{P})$ and
$(X,\,\mathcal{E})$ are dually flat submanifolds, with respect to the Bregman
divergence $D_{\psi}$. Since the maximum likelihood estimation of
$\mathcal{Q}$ with respect to the repeated observation $\boldsymbol{\sigma}$
is independent of its chosen parametrisation, it may be obtained by the
natural parametrisation of the embedding space
$(X,\,\mathcal{U}_{\eta},\,D_{\psi})$, such that:__

$\hat{\theta}_{ML}\in\arg\max_{\eta}\mathrm{L}[Q_{\eta}\mid\boldsymbol{\sigma}]$

Then theorem has an equivalent formulation, given by:

$\arg\max_{\eta}\mathrm{L}[Q_{\eta}\mid\boldsymbol{\sigma}]=\arg\min_{\eta}d(P_{\eta},\,Q_{\eta})$
(9.1)

Since $(X,\,\mathcal{Q})$ is a smooth submanifold of $(X,\,\mathcal{P})$ and
$(X,\,\mathcal{P})$ of $(X,\,\mathcal{U})$ it follows, that
$(X,\,\mathcal{Q})$ is also a smooth submanifold of $(X,\,\mathcal{U})$ and
since $(X,\,\mathcal{U},\,D_{\psi})$ is a dually flat statistical manifold,
the projection theorem is satisfied. Therefore the geodesic projection
$\pi:\mathcal{E}_{\boldsymbol{\sigma}}\to\mathcal{Q}$ of a fixed point
$P\in\mathcal{E}_{\boldsymbol{\sigma}}$ to $\mathcal{Q}$ is given by the
minimal dual affine projection, and thus by a point $Q_{\eta}\in\mathcal{Q}$,
that minimizes $D_{\psi}[Q_{\eta}\parallel P]$, such that:

$\arg\min_{\eta}d(P,\,Q_{\eta})=\arg\min_{\eta}D_{\psi}[Q_{\eta}\parallel P]$

Then the minimal geodesic projection of $\mathcal{E}_{\boldsymbol{\sigma}}$ to
$\mathcal{Q}$ is given by points
$P_{\boldsymbol{\eta}}\in\mathcal{E}_{\boldsymbol{\sigma}}$ and
$Q_{\boldsymbol{\eta}}\in\mathcal{Q}$, that minimize
$D_{\psi}[Q_{\boldsymbol{\eta}}\parallel P_{\boldsymbol{\eta}}]$:

$\arg\min_{\boldsymbol{\eta}}d(P_{\boldsymbol{\eta}},\,Q_{\boldsymbol{\eta}})=\arg\min_{\boldsymbol{\eta}}D_{\psi}[Q_{\boldsymbol{\eta}}\parallel
P_{\boldsymbol{\eta}}]$ (9.2)

Since $\boldsymbol{\eta}$ is the natural parametrisation of the exponential
family $(X,\,\mathcal{U})$, the Bregman divergence $D_{\psi}$ is given by the
dual Kullback-Leibler divergence in natural parameters. Therefore it follows,
that:

$\arg\min_{\boldsymbol{\eta}}D_{\psi}[Q_{\boldsymbol{\eta}}\parallel
P_{\boldsymbol{\eta}}]=\arg\min_{\boldsymbol{\eta}}D_{\mathrm{KL}}[P_{\boldsymbol{\eta}}\parallel
Q_{\boldsymbol{\eta}}]$ (9.3)

Without loss of generality let $\boldsymbol{v}:(X_{v},\,\Sigma_{v})\to V$ be
the vectorial observable random variable and
$\boldsymbol{h}:(X_{h},\,\Sigma_{h})\to H$ the vectorial latent random
variable. Then $\mathrm{d}\boldsymbol{v}$ and $\mathrm{d}\boldsymbol{h}$
denote the Lebesgue measures in $V$ and $H$ and:

$\displaystyle D_{\mathrm{KL}}[P_{\boldsymbol{\eta}}\parallel
Q_{\boldsymbol{\eta}}]$ $\displaystyle=$
$\displaystyle\int_{X}p_{\boldsymbol{\eta}}(\boldsymbol{x})\log\frac{p_{\boldsymbol{\eta}}(\boldsymbol{x})}{q_{\boldsymbol{\eta}}(\boldsymbol{x})}\mathrm{d}\boldsymbol{x}$
$\displaystyle=$
$\displaystyle\int_{H,\boldsymbol{\sigma}}p_{\boldsymbol{\eta}}(\boldsymbol{v},\,\boldsymbol{h})\log\frac{p_{\boldsymbol{\eta}}(\boldsymbol{v},\,\boldsymbol{h})}{q_{\boldsymbol{\eta}}(\boldsymbol{v},\,\boldsymbol{h})}\mathrm{d}\boldsymbol{v}\mathrm{d}\boldsymbol{h}$

The common empirical distributions
$p_{\boldsymbol{\eta}}(\boldsymbol{v},\,\boldsymbol{h})$ are defined by the
marginal empirical densities $p_{\boldsymbol{\eta}}(\boldsymbol{v})$ of the
observable variables and conditional transition probabilities
$p_{\boldsymbol{\eta}}(\boldsymbol{h}\mid\boldsymbol{v})$ of the latent
variables by
$p_{\boldsymbol{\eta}}(\boldsymbol{v},\,\boldsymbol{h})=p_{\boldsymbol{\eta}}(\boldsymbol{v})p_{\boldsymbol{\eta}}(\boldsymbol{h}\mid\boldsymbol{v})$,
such that:

$\displaystyle D_{\mathrm{KL}}[P_{\boldsymbol{\eta}}\parallel
Q_{\boldsymbol{\eta}}]$ $\displaystyle=$
$\displaystyle\int_{H,\boldsymbol{\sigma}}p_{\boldsymbol{\eta}}(\boldsymbol{v})p_{\boldsymbol{\eta}}(\boldsymbol{h}\mid\boldsymbol{v})\log\frac{p_{\boldsymbol{\eta}}(\boldsymbol{v})p_{\boldsymbol{\eta}}(\boldsymbol{h}\mid\boldsymbol{v})}{q_{\boldsymbol{\eta}}(\boldsymbol{v},\,\boldsymbol{h})}\mathrm{d}\boldsymbol{v}\mathrm{d}\boldsymbol{h}$

Furthermore the $\log$ function allows to write the product into a sum and to
substitute the addends by functionals, such that:

$\displaystyle D_{\mathrm{KL}}[P_{\boldsymbol{\eta}}\parallel
Q_{\boldsymbol{\eta}}]$ $\displaystyle=$ $\displaystyle
F_{1}(P_{\boldsymbol{\eta}},\,Q_{\boldsymbol{\eta}})+F_{2}(P_{\boldsymbol{\eta}},\,Q_{\boldsymbol{\eta}})+F_{3}(P_{\boldsymbol{\eta}},\,Q_{\boldsymbol{\eta}})$

With:

$\displaystyle F_{1}(P_{\boldsymbol{\eta}},\,Q_{\boldsymbol{\eta}})$
$\displaystyle\coloneqq\int_{H,\boldsymbol{\sigma}}p_{\boldsymbol{\eta}}(\boldsymbol{v})p_{\boldsymbol{\eta}}(\boldsymbol{h}\mid\boldsymbol{v})\log
p_{\boldsymbol{\eta}}(\boldsymbol{v})\mathrm{d}\boldsymbol{v}\mathrm{d}\boldsymbol{h}$
$\displaystyle F_{2}(P_{\boldsymbol{\eta}},\,Q_{\boldsymbol{\eta}})$
$\displaystyle\coloneqq\int_{H,\boldsymbol{\sigma}}p_{\boldsymbol{\eta}}(\boldsymbol{v})p_{\boldsymbol{\eta}}(\boldsymbol{h}\mid\boldsymbol{v})\log
p_{\boldsymbol{\eta}}(\boldsymbol{h}\mid\boldsymbol{v})\mathrm{d}\boldsymbol{v}\mathrm{d}\boldsymbol{h}$
$\displaystyle F_{3}(P_{\boldsymbol{\eta}},\,Q_{\boldsymbol{\eta}})$
$\displaystyle\coloneqq-\int_{H,\boldsymbol{\sigma}}p_{\boldsymbol{\eta}}(\boldsymbol{v})p_{\boldsymbol{\eta}}(\boldsymbol{h}\mid\boldsymbol{v})\log
q_{\boldsymbol{\eta}}(\boldsymbol{v},\,\boldsymbol{h})\mathrm{d}\boldsymbol{v}\mathrm{d}\boldsymbol{h}$

Then it follows, that:

$\displaystyle F_{1}(P_{\boldsymbol{\eta}},\,Q_{\boldsymbol{\eta}})$
$\displaystyle=\int_{\boldsymbol{\sigma}}p_{\boldsymbol{\eta}}(\boldsymbol{v})\log
p_{\boldsymbol{\eta}}(\boldsymbol{v})\left(\int_{H}p_{\boldsymbol{\eta}}(\boldsymbol{h}\mid\boldsymbol{v})\mathrm{d}\boldsymbol{h}\right)\mathrm{d}\boldsymbol{v}$
$\displaystyle=\int_{\boldsymbol{\sigma}}p_{\boldsymbol{\eta}}(\boldsymbol{v})\log
p_{\boldsymbol{\eta}}(\boldsymbol{v})\mathrm{d}\boldsymbol{v}$
$\displaystyle=-\mathrm{H}[P_{\eta}\mid\boldsymbol{\sigma}]$

And furthermore::

$\displaystyle F_{2}(P_{\boldsymbol{\eta}},\,Q_{\boldsymbol{\eta}})$
$\displaystyle=\int_{\boldsymbol{\sigma}}p_{\boldsymbol{\eta}}(\boldsymbol{v})\left(\int_{H}p_{\boldsymbol{\eta}}(\boldsymbol{h}\mid\boldsymbol{v})\log
p_{\boldsymbol{\eta}}(\boldsymbol{h}\mid\boldsymbol{v})\mathrm{d}\boldsymbol{h}\right)\mathrm{d}\boldsymbol{v}$
$\displaystyle=-\int_{\boldsymbol{\sigma}}p_{\boldsymbol{\eta}}(\boldsymbol{v})\mathrm{H}[p(\boldsymbol{h}\mid\boldsymbol{v})\mid
H]\mathrm{d}\boldsymbol{v}$
$\displaystyle=-\mathrm{H}[p(\boldsymbol{h}\mid\boldsymbol{v})\mid(H,\,\boldsymbol{\sigma})]$

Then $F_{1}$ and $F_{2}$ are completely determined by $\boldsymbol{\sigma}$,
such that their minima do not depend on the choice of $P_{\boldsymbol{\eta}}$
and $Q_{\boldsymbol{\eta}}$. Therefore:

$\displaystyle\arg\min_{\boldsymbol{\eta}}D_{\mathrm{KL}}[P_{\boldsymbol{\eta}}\parallel
Q_{\boldsymbol{\eta}}]$ (9.4)
$\displaystyle=\arg\min_{\boldsymbol{\eta}}F_{3}(P_{\boldsymbol{\eta}},\,Q_{\boldsymbol{\eta}})$
$\displaystyle=\arg\max_{\boldsymbol{\eta}}\int_{H,\boldsymbol{\sigma}}p_{\boldsymbol{\eta}}(\boldsymbol{v})p_{\boldsymbol{\eta}}(\boldsymbol{h}\mid\boldsymbol{v})\log
q_{\boldsymbol{\eta}}(\boldsymbol{v},\,\boldsymbol{h})\mathrm{d}\boldsymbol{v}\mathrm{d}\boldsymbol{h}$
$\displaystyle=\arg\max_{\boldsymbol{\eta}}\int_{X}p_{\boldsymbol{\eta}}(x)\log
q_{\boldsymbol{\eta}}(x)\mathrm{d}\mu(x)$

Since $p_{\boldsymbol{\eta}}(x)$ are the empirical probabilities over
$(X,\,\Sigma)$ the conditions of lemma , equation 6.2, are satisfied, such
that

$\displaystyle\arg\max_{\boldsymbol{\eta}}\mathrm{L}[Q_{\boldsymbol{\eta}}\mid\boldsymbol{\sigma}]$
$\displaystyle\stackrel{{\scriptstyle}}{{=}}\arg\max_{\boldsymbol{\eta}}\int_{X}p_{\boldsymbol{\eta}}(x)\log
q_{\boldsymbol{\eta}}(x)\mathrm{d}\mu(x)$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:thm:4.1:4}}}{{=}}\arg\min_{\boldsymbol{\eta}}D_{\mathrm{KL}}[P_{\boldsymbol{\eta}}\parallel
Q_{\boldsymbol{\eta}}]$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:thm:4.1:3}}}{{=}}\arg\min_{\boldsymbol{\eta}}D_{\psi}[Q_{\boldsymbol{\eta}}\parallel
P_{\boldsymbol{\eta}}]$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:thm:4.1:2}}}{{=}}\arg\min_{\boldsymbol{\eta}}d(P_{\boldsymbol{\eta}},\,Q_{\boldsymbol{\eta}})$

∎

##  10 Alternating minimization

Let $(X,\,\mathcal{P},\,D_{\psi})$ be a dually flat statistical manifold and
$(X,\,\mathcal{Q})$ and $(X,\,\mathcal{E})$ smooth submanifolds of
$(X,\,\mathcal{P},\,D_{\psi})$. Then the divergence $D_{\psi}$ between
$(X,\,\mathcal{Q})$ to $(X,\,\mathcal{E})$ is defined by the minimal
divergence of its respective points, such that:

$D_{\psi}[\mathcal{Q}\parallel\mathcal{S}]=\min_{Q\in\mathcal{Q},\,S\in\mathcal{S}}D_{\psi}[Q\parallel
S]=D_{\psi}[\hat{Q}\parallel\hat{S}]$

where $\hat{Q}\in\mathcal{Q}$ and $\hat{S}\in\mathcal{S}$ minimize
$D_{\psi}[Q\parallel S]$. By applying Amari’s projection theorem it follows,
that the pair $(\hat{Q},\,\hat{S})$ also minimizes the geodesic distance and
therefore has to be regarded as a pair of closest points between
$(X,\,\mathcal{Q})$ and $(X,\,\mathcal{S})$. Furthermore the dually flat
structure allows an iterative alternating geodesic projection between
$(X,\,\mathcal{Q})$ and $(X,\,\mathcal{S})$ to asymptotically approximate
$\hat{Q}$ and $\hat{S}$. This is termed alternating minimization.

######  Definition (_Alternating minimization_).

Let $(X,\,\mathcal{U})$ be an exponential family with smooth submanifolds
$(X,\,\mathcal{Q})$ and $(X,\,\mathcal{S})$. Then the alternating minimization
from $\mathcal{S}$ to $\mathcal{Q}$ iteratively defines a sequence
$(Q_{n},\,S_{n})_{n\in\mathbb{N}_{0}}$ of elements in
$(\mathcal{Q},\,\mathcal{S})$, which is given by _:_

Begin:

_Let $S_{0}\in\mathcal{S}$ be arbitrary_

Iteration _:_

($m$-step)_ $Q_{n}$ is given by a geodesic projection of $S_{n}$ to
$\mathcal{Q}$:_

$Q_{n}=\pi(Q_{n})=\arg\min_{Q\in\mathcal{Q}}d(S_{n},\,Q)$

($e$-step)_ $S_{n+1}$ is given by a dual geodesic projection of $Q_{n}$ to
$\mathcal{S}$:_

$S_{n+1}=\pi^{*}(Q_{n})=\arg\min_{S\in\mathcal{S}}d(S,\,Q_{n})$

######  Theorem 24 (_Alternating minimization_).

__Let $(X,\,\mathcal{U})$ be an exponential family model with smooth
submanifolds $(X,\,\mathcal{Q})$ and $(X,\,\mathcal{S})$. Then the alternating
minimization algorithm converges in a pair of probability distributions, that
locally minimize the geodesic distance between $(X,\,\mathcal{Q})$ and
$(X,\,\mathcal{S})$.

$(X,\,\mathcal{U},\,D)$$(X,\,\mathcal{Q})$$S_{0}$$(X,\,\mathcal{S})$$S_{1}$$S_{2}$$S_{3}$$Q_{0}$$Q_{1}$$Q_{3}$$Q_{2}$
Figure 10.1: Alternating minimization in Exponential Families

###### Proof.

Let $(X,\,\mathcal{U})$ be an exponential family, then the Bregman divergence
of its cumulative generating function $\psi$ induces a Riemannian metric and
$(X,\,\mathcal{U},\,D_{\psi})$ is a simply connected dually flat statistical
manifold. Then $(X,\,\mathcal{Q},\,D_{\psi})$ and
$(X,\,\mathcal{S},\,D_{\psi})$ are Riemannian submanifolds of
$(X,\,\mathcal{U},\,D_{\psi})$, with respect to the induced Riemannian metric.
In the $m$-step, the geodesic projection $\pi:\mathcal{S}\to\mathcal{Q}$ is
given by the dual affine projection in the $m$-parametrisation. Then the dual
affine projection minimizes the Bregman divergence and the Pythagorean theorem
yields:

$\displaystyle d(Q_{n},\,S_{n})$ (10.1)
$\displaystyle=D_{\psi}[S_{n}\parallel\pi(S_{n})]$ $\displaystyle\leq
D_{\psi}[S_{n}\parallel\pi(S_{n})]+D_{\psi}[\pi(S_{n})\parallel R]$
$\displaystyle=D_{\psi}[S_{n}\parallel Q_{n-1}]=d(Q_{n-1},\,S_{n})$

In the $e$-step dual geodesic projection $\pi^{*}:\mathcal{Q}\to\mathcal{S}$
is given by the affine projection in the $e$-parametrisation. Then the affine
projection minimizes the dual Bregman divergence and the Pythagorean theorem
yields:

$\displaystyle d(Q_{n},\,S_{n+1})$ (10.2)
$\displaystyle=D_{\psi^{*}}[\pi(Q_{n})\parallel Q_{n}]$ $\displaystyle\leq
D_{\psi^{*}}[R\parallel\pi(Q_{n})]+D_{\psi^{*}}[\pi(Q_{n})\parallel Q_{n}]$
$\displaystyle=D_{\psi^{*}}[S_{n}\parallel Q_{n}]=d(Q_{n},\,S_{n})$

This proves, that $d(Q_{n},\,S_{n})$ monotonously decreases, since:

$\displaystyle d(Q_{n+1},\,S_{n+1})$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:thm:4.3:2}}}{{\leq}}d(Q_{n},\,S_{n+1})$
$\displaystyle\stackrel{{\scriptstyle\ref{eq:thm:4.3:1}}}{{\leq}}d(Q_{n},\,S_{n}),\,\forall
n\in\mathbb{N}_{0}$

Furthermore $d(Q_{n},\,S_{n})$ is bounded bellow by:

$d(Q_{n},\,S_{n})=D_{\psi}[S_{n}\parallel S_{n}]\geq 0,\,\forall
n\in\mathbb{N}_{0}$

This proves, that $d(Q_{n},\,S_{n})$ converges against a local minimum. ∎

######  Corollary 25.

Let $(X,\,\mathcal{U})$ be an exponential family with an $e$-flat submanifold
$(X,\,\mathcal{Q})$ and an $m$-flat submanifold $(X,\,\mathcal{S})$. Then the
alternating minimization algorithm from $\mathcal{S}$ to $\mathcal{Q}$
converges against points, that globally minimize the geodesic distance between
$(X,\,\mathcal{Q})$ and $(X,\,\mathcal{S})$.

###### Proof.

Let __ $(Q_{n},\,S_{n})_{n\in\mathbb{N}_{0}}$ be a sequence, given by the
alternating minimization algorithm from $\mathcal{S}$ to $\mathcal{Q}$, then
due to theorem 24 $(Q_{n},\,S_{n})$ converges against against points, that
locally minimize the geodesic distance between $(X,\,\mathcal{Q})$ and
$(X,\,\mathcal{S})$. Since $(X,\,\mathcal{Q})$ is $e$-flat and
$(X,\,\mathcal{S})$ is $m$-flat corollary LABEL:cor:3.2 is satisfied, such
that this geodesic distance in unique and therefore $(Q_{n},\,S_{n})$
converges against points, that globally minimize the geodesic distance between
$(X,\,\mathcal{Q})$ and $(X,\,\mathcal{S})$. ∎

######  Corollary 26.

Let $(X,\,\mathcal{P})$ be an exponential family over a partially observable
measurable space $(X,\,\Sigma)$ and $(X,\,\mathcal{E})$ an absolutely
continuous empirical model over $(X,\,\Sigma)$. Let further be
$(X,\,\mathcal{Q})$ an $e$-flat submanifold of $(X,\,\mathcal{P})$, and
$\boldsymbol{\sigma}$ a repeated observation over $(X,\,\Sigma)$. Then a
maximum likelihood estimation of $(X,\,\mathcal{Q})$ respective to
$\boldsymbol{\sigma}$ is given by the limit of alternating minimization
algorithm from $\mathcal{E}_{\boldsymbol{\sigma}}$ to $\mathcal{Q}$.

###### Proof.

Since $(X,\,\mathcal{P})$ is an exponential family and $(X,\,\mathcal{E})$ an
absolutely continuous empirical model, there exists a common dually flat
embedding space $(X,\,\mathcal{U}_{\boldsymbol{\eta}},\,D_{\psi})$, such that
$(X,\,\mathcal{P})$ and $(X,\,\mathcal{E})$ are dually flat submanifolds, with
respect to the Bregman divergence $D_{\psi}$. Then
$(X,\,\mathcal{E}_{\boldsymbol{\sigma}})$ is $m$-flat in
$(X,\,\mathcal{U}_{\boldsymbol{\eta}},\,D_{\psi})$ and by definition
$(X,\,\mathcal{Q})$ is $e$-flat in $(X,\,\mathcal{P})$ and therefore also in
$(X,\,\mathcal{U}_{\boldsymbol{\eta}},\,D_{\psi})$. This allows the
application of corollary 25. ∎

## References

  * [1] Shun-ichi Amari.  Differential geometrical theory of statistics.  Differential geometry in statistical inference, pages 19–94, 1987\.
  * [2] Patrick Michl.  Foundations of Structural Statistics: Topological Statistical Theory.  ArXiv Preprint, 2019.
  * [3] Patrick Michl.  Foundations of Structural Statistics: Statistical Manifolds.  ArXiv Preprint, 2020.