# Geometry Perspective of Estimating Learning Capability of Neural Networks

Ankan Dutta<EMAIL_ADDRESS>Department of Mechanical Engineering, Jadavpur
University, Kolkata 32, India  Arnab Rakshit<EMAIL_ADDRESS>Artificial Intelligence Lab, Department of Electronics and Telecommunication
Engineering, Jadavpur University, Kolkata 32, India

###### Abstract

The paper uses statistical and differential geometric motivation to acquire
prior information about the learning capability of an artificial neural
network on a given dataset. The paper considers a broad class of neural
networks with generalized architecture performing simple least square
regression with stochastic gradient descent (SGD). The system characteristics
at two critical epochs in the learning trajectory are analyzed. During some
epochs of the training phase, the system reaches equilibrium with the
generalization capability attaining a maximum. The system can also be coherent
with localized, non-equilibrium states, which is characterized by the
stabilization of the Hessian matrix. The paper proves that neural networks
with higher generalization capability will have a slower convergence rate. The
relationship between the generalization capability with the stability of the
neural network has also been discussed. By correlating the principles of high-
energy physics with the learning theory of neural networks, the paper
establishes a variant of the Complexity-Action conjecture from an artificial
neural network perspective.

Neural Networks, Differential Geometry, Learning Capability, Dynamical System

##  I Introduction

Much of the success of Artificial Neural Network (ANN) has been credited to
the universal function approximator property of multilayered ANN HORNIK1991251
; 10.5555/3086952 ; kidger2020universal ; NIPS2017_7203 . ANN with a high
number of trainable parameters can approximate any given function with nearly
zero training error. Although, this doesn’t assure the generalization capacity
of ANN on the testing dataset zhang2016understanding ; arora2019finegrained ;
arora2018stronger . In recent work, Zhang et al zhang2016understanding  showed
that the neural networks can easily fit random labels with zero training error
but the test error or generalization error increases with an increase in
randomness in labeling. The generalization error precisely captures the
distortion in the underlying meaningful structure of data caused due to random
labeling. Hence, generalization plays the most vital role in deciding how well
a neural network model learns a dataset. The optimization of the loss function
in the training period is an empirically easy task and corresponds to
memorization rather than learning. The generalization capability of a model is
related to the eigenspectrum of the Hessian matrix of the loss function
chaudhari2016entropysgd , with wider minima producing better generalization
capability wang2018identifying ; sagun2016eigenvalues ;
chaudhari2016entropysgd . Laurent et al 10.5555/3305381.3305487  on the
contrary, argued that sharp minima can have generalization property. Although
generalization is not completely understood, it is always assumed that the
generalization capability of a neural network is highly dependent on the
architecture of the neural network. Using a data-dependent complexity measure,
Arora et al arora2019finegrained  showed that the generalization bound is
independent of network size. On the other hand intuitively, one can argue that
higher training speed will reduce the probability of exploring a better
optimal parameter set, thus reducing the generalization capability. The exact
dependence of convergence rate and generalization capability is less
documented. Here, the paper focuses on the relationship between generalization
capability and convergence rate with the architecture of ANN, for a given
dataset.

In the past literature, the learning trajectory has been looked up from two
different perspectives: a statistical viewpoint and an information-theoretic
viewpoint. From a statistical aspect, the learning trajectory represents the
evolution of eigenvalues of the hessian matrix of the loss function
jastrzbski2018relation  and evolution of variance of weights implying the
generalization capability of a neural network jastrzbski2018relation . From an
information-theoretic perspective, the learning trajectory (or information
flow) consists of the fitting and compression phase of mutual information
between layers shwartzziv2017opening  of a neural network and is dependent on
the training dataset and architecture pmlr-v97-goldfeld19a . For a particular
dataset, different learning trajectory is seen for different neural networks
pmlr-v97-goldfeld19a . Here, the paper focuses on the statistical and
differential geometric analysis of the learning trajectory. Recent work,
Lampinen et al lampinen2018analytic  established the nonlinear dynamics of
generalization in deep linear networks. The work by lampinen2018analytic ;
saxe2013exact  developed the analytic solutions to the training and testing
error of deep networks as a function of training time. However, the work
lampinen2018analytic ; saxe2013exact  developed the analytical solutions for a
teacher-student network with a fixed architecture where teacher networks
generate training data for more complex (or more layered) student network.
Here, the paper considers a more generalized architecture framework of ANN
with single input-single output data. Learning trajectory manifests the
complete dynamical behavior of neural networks. The paper analyses two
critical phases of training. The early phase epoch when the largest (or few
large) eigenvalues of the hessian matrix reaches its highest value
jastrzbski2018relation  and the late phase epoch after which the eigenvalues
of the hessian matrix completely stabilizes. Focusing only on two critical
epochs reduces the complexity of analysis drastically and hence, one can term
the analysis of learning trajectory as the analysis of learning capability at
the extremal epochs. Learning capability denotes both the effective capability
zhang2016understanding  of a neural network to optimize the loss function (or
minimize training error) and also the generalization capability (or minimize
testing error) 10.5555/3295222.3295344 .

In 2013, Yan et al YanE4185  argued that the states with local lowest
potential or point attractor state represent a particular memory. The
stability of these attractor states is crucial for memory storage and
retrieval. The paper focuses on the stability and thermodynamics of neural
circuits using Lyapunov exponent and probability flux. The analysis
exclusively focuses on the stability and thermodynamics of the neural circuit
when the ANN reaches its maximum generalization capability. In doing so, the
paper establishes that there is always a trade-off between generalization
capability and convergence rate, and the system will execute limit cycles or
oscillations during maximum generalization capability. The whole analysis is
based upon the correspondence between the trajectory of parameters in a
Riemannian manifold called Diffusion metric Fioresi2020  with the trajectory
of particles in the space-time where parameters form the generalized
coordinate system. The correspondence opens up the application of principles
of high-energy physics in the context of learning theory. From this
motivation, the paper establishes a variant of the famous Complexity-Action
conjecture from an ANN perspective. The correspondence has been made between
the complexity of a parameter distribution and the action of parameters on the
learning manifold.

The paper constructs the mathematical background in Section II, based on which
the whole analysis is carried out in Section III—Section VI. In Section II the
neural network is represented as a generalized parameterized function, whose
loss function is optimized by updating the parameters based on SGD. A matrix
called architecture-dataset matrix is introduced which indicates a relation
between the architecture of the neural network and the dataset on which it is
to be trained. In Section II the paper establishes a relationship between
Diffusion matrix and the architecture-dataset matrix. The section also proves
important learning characteristics of the neural network at two extremal
epochs. In Section IV the paper analyze the stability using eigenvalues of the
architecture-dataset matrix. The relationship between the architecture matrix
and the curvature of the diffusion metric is established in Section V.
Moreover, in Section VI the correlation of the principles of high energy
physics to the learning dynamics of neural networks is epitomized by
establishing the Complexity-Action correspondence. The major contributions and
results of the paper are summarized in Section VII.

##  II Mathematical background

A neural network gets trained on a sampled training dataset and optimizes the
loss function iteratively using a learning algorithm. Here stochastic gradient
descent (SGD) is employed as the learning algorithm. Let us consider an input
data stream $x=\\{x_{i}\\}_{i=1}^{N}$ generated randomly from a Gaussian
distribution $P(x)$ to map onto a targeted data stream $\hat{y}$. The pair
$\\{x_{i},\hat{y_{i}}\\}_{i=1}^{N}$ representing
$(x_{1},\hat{y_{1}}),(x_{2},\hat{y_{2}}),...,(x_{N},\hat{y_{N}})$ form the
training dataset. The neural network gets trained to learn the mapping of
$x\mapsto\hat{y}$. The neural network produces $y$ as a noisy functional form
of $x$ at every training epoch, learning to predict $\hat{y}$. Mathematically,
$y$ can be represented by the equation $1$ as given below

$y=c(\alpha;x)+\eta$ (1)

where $\eta$ denotes a Gaussian noise of standard deviation $\sigma$. The
function $c(\alpha;x)$ characterized with parameter $\alpha$ represents the
architecture and weight arrangement of the neural network completely. The
function $c$ can be represented as the linear combination of its basis
function.

$c(\alpha;x)=\sum_{\mu=1}^{K}\alpha^{\mu}\phi_{\mu}(x)$ (2)

where $\phi=\\{\phi_{\mu}\\}_{\mu=1}^{K}$ are basis functions of $c$. Here,
the co-efficient of basis functions is denoted as the parameters
$\alpha=\\{\alpha^{\mu}\\}_{\mu=1}^{K}$. Conventionally, any neural network of
layers $L$ is represented by $c=(\phi^{\prime}_{L}\circ
w_{L}\circ\phi^{\prime}_{L-1}\circ...\circ w_{2}\circ\phi^{\prime}_{1}\circ
w_{1})(x)$ where weights $w=\\{w_{l}\\}_{l=1}^{L}$ and $\alpha$ are related by
some function $g$, given by $\alpha^{\mu}=g^{\mu}(w)$. Activation functions
are represented by $\phi^{\prime}=\\{\phi^{\prime}_{l}\\}_{l=1}^{L}$. It is
important to note that the weights $w$ can be considered as independent of
each other but the parameters $\alpha$ are not independent of each other. In
fact, the dependency of $\alpha^{\mu}$ and $\alpha^{\nu}$ can be given by the
Jacobian:

$\mathcal{G}^{\mu}_{\nu}=\frac{\partial\alpha^{\mu}}{\partial\alpha^{\nu}}=\left\\{\begin{tabular}[]{cc}$\sum_{l\in\\{L(\nu)\\}}\frac{\partial
g^{\mu}(w)}{\partial w_{l}}\frac{\partial w_{l}}{\partial
g^{\nu}(w)}$&$\mu\not=\nu$\\\ 1&$\mu=\nu$\end{tabular}\right\\}$ (3)

where $\\{L(\nu)\\}$ is the collection of indexes $l$ for which
$\frac{\partial g^{\nu}(w)}{\partial w^{l}}\not=0$ which indicates that we are
spanning over the weights indexes which will change $g^{\nu}(w)$. We denote
the collection of elements $\mathcal{G}^{\mu}_{\nu}$ as a jacobian matrix $G$,
where $G_{\mu\nu}=\mathcal{G}^{\mu}_{\nu}$. The matrix $G$ is not constant to
the architecture-dataset pair, $G$ changes with epochs as weights evolves with
time.The connectivity between the parameters can be controlled by implicitly
changing the element of the matrix $G$. One can change the elements of matrix
$G$ inherently by changing the number of layers or changing the dropout rate
in the architecture of the neural network. The elements of matrix $G$ signify
the dependence of parameters on each other thus it is important in designing
of architectures. Note that the matrix $G$ becomes an identity matrix when all
the parameters are independent of each other.

The neural network optimizes the parameter $\alpha$ to the optimal parameter
$\bar{\alpha}$ according to the loss function
$f=\frac{1}{N}\sum_{i=1}^{N}(\hat{y}_{i}-y_{i})^{2}$ (here mean squared error)
to produce the stream of pair $\\{x,\hat{y}\\}$ characterised by
$\bar{\alpha}$. The loss function $f$ can be approximated as an energy term
$2\sigma^{2}E_{N}$ bialek2000predictability  with the assumptions of
$N\rightarrow\infty$ and neglecting the fluctuations. The energy term $E_{N}$
can be shown as

$\begin{split}\frac{f}{2\sigma^{2}}\hskip
5.69054pt\widetilde{\rightarrow}&E_{N}(\alpha;\\{x,\hat{y}\\})=\frac{1}{2}\\\
&+\frac{1}{2\sigma^{2}}\sum_{\mu,\nu=1}^{K}(\alpha^{\mu}-\bar{\alpha}^{\mu})A_{\mu\nu}^{\infty}(\alpha^{\nu}-\bar{\alpha}^{\nu})\end{split}$
(4)

where $A^{\infty}$ is a $K\times K$ symmetric matrix representing the
architecture and activation functions of the neural network along with the
training dataset. So we call $A^{\infty}$ an architecture-dataset matrix,
denoted by

$A_{\mu\nu}^{\infty}=\lim_{N\rightarrow\infty}\frac{1}{N}\sum_{i=1}^{N}\phi_{\mu}(x_{i})\phi_{\nu}(x_{i})=\Big{<}\phi_{\mu}(x)\phi_{\nu}(x)\Big{>}$
(5)

The variance $\sigma^{2}$ is constant and an intrinsic property of the dataset
sampled from the distribution $P(x)$. Thus the variance $\sigma^{2}$ is
independent of parameters $\alpha$. Considering the basis functions $\phi$
only to be a function of $x$, it is easy to notice that the hessian of loss
function w.r.t. parameters $\alpha$ is given by

$\begin{split}H_{\zeta\eta}=\frac{\partial^{2}f}{\partial\alpha^{\zeta}\partial\alpha^{\eta}}=&2\sum_{\mu,\nu=1}^{K}\mathcal{G}^{\mu}_{\eta\zeta}A^{\infty}_{\mu\nu}(\alpha^{\nu}-\bar{\alpha}^{\nu})\\\
&+2\sum_{\mu,\nu=1}^{K}\mathcal{G}^{\mu}_{\eta}A^{\infty}_{\mu\nu}\mathcal{G}^{\nu}_{\zeta}\end{split}$
(6)

where
$\mathcal{G}^{\mu}_{\eta\zeta}=\frac{\partial^{2}\alpha^{\mu}}{\partial\alpha^{\eta}\partial\alpha^{\zeta}}$,
considering $\alpha$ to be continuous and symmetric in partial derivatives,
applying Schwarz’s theorem shows
$\mathcal{G}^{\mu}_{\eta\zeta}=\mathcal{G}^{\mu}_{\zeta\eta}$. One can also
note that when the parameters $\alpha^{\mu}\rightarrow\bar{\alpha}^{\mu}$,
then the Hessian $H\rightarrow 2GA^{\infty}G$.

Recent study, Fioresi2020  defined diffusion matrix and showed that the noise
in stochastic gradient descent (SGD) during training is highly anisotropic.
The diffusion matrix is basically the co-variance matrix of the gradient of
the loss function $f$. Being consistent with the notation of Fioresi2020 , the
diffusion matrix is given by

$D_{\mu\nu}=\frac{1}{N}\sum_{i=1}^{N}\frac{\partial
f_{i}}{\partial\alpha^{\mu}}\frac{\partial
f_{i}}{\partial\alpha^{\nu}}-\frac{1}{N^{2}}\sum_{i,j=1}^{N}\frac{\partial
f_{i}}{\partial\alpha^{\mu}}\frac{\partial f_{j}}{\partial\alpha^{\nu}}$ (7)

The diffusion matrix $D$ is a co-variance matrix, thus it is positive semi-
definite. Recently chaudhari2017stochastic  showed that the SGD tend to
minimise a separate potential or loss function $\Phi$ (not related to basis
functions $\phi$) rather than the original loss function $f$. The potential is
defined as

$\Phi=-\beta^{-1}\log(\rho^{ss}(\alpha))$ (8)

where temperature $\beta^{-1}=\frac{\eta}{2b}$, $\eta$ being the learning rate
and $b$ is the batch size. $\rho^{ss}(\alpha)$ is the steady state
distribution of parameters. The study Fioresi2020  showed the relation between
the potential $\Phi$ and loss function $f$ as $\nabla\Phi=\widetilde{D}\nabla
f$ where diffusion metric $\widetilde{D}=I_{K\times K}+\epsilon D$ and
$\epsilon$ is a scalar number constraint to $\epsilon<1/\lambda_{D_{max}}$.
The diffusion metric is used to derive the geodesic equation which showed the
dynamics of weights in anisotropic noise.

##  III Architecture & diffusion matrix

The relation between architecture-dataset matrix and diffusion matrix shows
how the architecture-dataset pair of a neural network governs the
stochasticity in SGD.

Identity 1: Using equation $(7)$ and a similar treatment as
bialek2000predictability  (See A), the relation between Architecture matrix
$A^{\infty}$ and Diffusion matrix $D$ is established below

$\begin{split}D_{\mu\nu}\widetilde{\rightarrow}&4[\sum_{p,k,\eta,\zeta}^{K}\mathcal{G}^{\eta}_{\mu}\mathcal{G}^{\zeta}_{\nu}(\alpha^{p}-\bar{\alpha}^{p})\\{A_{p\eta\zeta
k}^{\infty}-A_{p\eta}^{\infty}A_{\zeta k}^{\infty}\\}\\\
&\quad\quad\quad\quad\quad\quad\times(\alpha^{k}-\bar{\alpha}^{k})+\sigma^{2}\sum_{\eta,\zeta}^{K}\mathcal{G}^{\eta}_{\mu}\mathcal{G}^{\zeta}_{\nu}A_{\eta\zeta}^{\infty}]\\\
\text{where}\hskip 5.69054pt&A_{p\eta\zeta
k}^{\infty}=\lim_{N\rightarrow\infty}\frac{1}{N}\sum_{i=1}^{N}\phi_{p}(x_{i})\phi_{\eta}(x_{i})\phi_{\zeta}(x_{i})\phi_{k}(x_{i})\end{split}$
(9)

The matrix $D$ is evaluated under an approximation for large
$N\rightarrow\infty$ and neglecting the fluctuations owing to the symbol
$\widetilde{\rightarrow}$ as used in bialek2000predictability . Here, an
approximated diffusion matrix $D^{\infty}$ is introduced, which would bring
equality to the equation $(9)$ as given below

$\begin{split}D^{\infty}&=4\sigma^{2}GA^{\infty}G+4C^{\infty}\\\
\text{where}\hskip
5.69054ptC^{\infty}_{\mu\nu}&=\sum_{p,k,\eta,\zeta}(\alpha^{p}-\bar{\alpha}^{p})\mathcal{G}^{\mu}_{\eta}\mathcal{G}^{\nu}_{\zeta}\\{A_{p\eta\zeta
k}^{\infty}-A_{p\eta}^{\infty}A_{\zeta k}^{\infty}\\}\\\
&\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\times(\alpha^{k}-\bar{\alpha}^{k})\end{split}$
(10)

Lemma $1$: The matrix $C^{\infty}$ is a symmetric matrix. It is a positive
semi-definite matrix if $GA^{\infty}G$ is positive semi-definite. The elements
of matrix $C^{\infty}$ are given by

$\begin{split}&\sum_{\mu\nu}y^{\mu}C^{\infty}_{\mu\nu}y^{\nu}=2\sum_{\mu,\nu,\eta,\zeta}V^{\mu\eta\zeta}V^{\mu\zeta\eta}\text{var}(M_{\zeta\eta}(x))\\\
&+\Big{[}2\sum_{\mu,\nu,\zeta,\eta}y^{\nu}\mathcal{G}^{\nu}_{\zeta}A^{\infty}_{\eta\zeta}\mathcal{G}^{\mu}_{\eta}y^{\mu}\Big{]}\Big{[}2\sum_{p}(\alpha^{p}-\bar{\alpha}^{p})^{2}\text{var}(\phi_{p}(x))\Big{]}\end{split}$
(11)

where matrix $V^{\mu
p\eta}=y^{\mu}\mathcal{G}^{\mu}_{\eta}(\alpha^{p}-\bar{\alpha}^{p})$ for any
vector $y$ and $M_{\zeta q}(x)=\phi_{\zeta}(x)\phi_{q}(x)$. (See B)

Using equation $(10)$ and $(11)$, the approximated diffusion matrix
$D^{\infty}$ can be represented as:

$\begin{split}y^{T}D^{\infty}y=&4\sigma^{2}y^{T}GA^{\infty}Gy+8y^{T}GEGy+\\\
&(4\Delta\alpha^{T}F\Delta\alpha)(4y^{T}GA^{\infty}Gy)\geq 0\end{split}$ (12)

where $E$ is a scalar $\Delta\alpha^{T}Y\Delta\alpha$ and
$Y_{\mu\nu}=\text{var}(M_{\mu\nu}(x))$ and $\Delta\alpha=\alpha-\bar{\alpha}$.
The elements of matrix $F$ is given by
$F_{\mu\nu}=\delta_{\mu\nu}\text{var}(\phi_{\mu}(x))$. As matrix $Y$ is a
variance of the elements of matrix $M$, the scalar $E$ is always positive.
Moreover, the matrix $F$ is form of a variance of the elements of the basis
function $\phi$. So, the matrix $F$ is positive semi-definite matrix. So, when
matrix $GA^{\infty}G$ is positive semi-definite, the matrix $D^{\infty}$ is
semi-definite, though the converse is not necessarily true. But the inequality
comes from the fact the approximated diffusion matrix is also a positive semi-
definite matrix similar to the original diffusion matrix. So, the matrix
$GA^{\infty}G$ is constrained to satisfy the inequality which will be analysed
later in Theorem $1$. Approximated diffusion matrix $D^{\infty}$ is always a
positive-semi definite matrix as it is the covariance matrix of noise in SGD.
But if $GA^{\infty}G$ is negative semi-definite, then matrix $D^{\infty}$ can
also be indefinite or negative semi-definite. This contradicts the inequality
$(12)$, which constrains the learning trajectory.

We look at the extreme critical epochs when bias
$b=\frac{1}{K}\sum_{\mu}\Delta\alpha_{\mu}\rightarrow 0$ with maximum
parameter variance $\sigma_{\alpha}^{2}$ and when bias $b\rightarrow 0$ with
minimum parameter variance $\sigma_{\alpha}^{2}\rightarrow 0$.

Theorem $1$: (See C)
Part 1
If all of the following conditions are satisfied:
Condition 1: The matrix $GA^{\infty}G$ is negative-semi definite i.e.
$\max\lambda_{GA^{\infty}G}<0$
Condition 2: Epochs for which bias $b\rightarrow 0$
Condition 3: Epochs for which variance of parameters is at its minimum i.e.
$\sigma_{\alpha}^{2}\rightarrow 0$

Then for these epochs,

  1. 1.

The maximum eigenvalue of the matrix $GA^{\infty}G$ i.e.
$\max\lambda_{GA^{\infty}G}\rightarrow 0^{-}$

  2. 2.

The matrix $D^{\infty}$ tends towards null matrix i.e. $D^{\infty}\rightarrow
0^{+}$

  3. 3.

The order of $\Delta\alpha$ is constrained to
$\mathcal{O}(\Delta\alpha)\geq\sqrt{\mathcal{O}\Big{(}\frac{\sigma^{2}|\max\lambda_{GA^{\infty}G}|}{\text{det}(F^{2})|\max\lambda_{G^{2}}|-\text{det}(F)|\max\lambda_{GA^{\infty}G}|}\Big{)}}$

  4. 4.

The Hessian of loss function stabilizes i.e. $\lambda_{H}\rightarrow 0$

Part 2
Another observation is:
Having Condition 1, Condition 2 satisfied but with
Condition 3: Epochs for which maximum variance of parameters
$\sigma_{\alpha}^{2}$ is reached
Condition 4: We define a matrix $J_{\mu l}=\frac{\partial g^{\mu}(w)}{\partial
w^{l}}$ and it is full-rank i.e. $\text{rank}(J)=\min\\{K,L\\}$

Then for these epochs,

  1. 1.

The parameters reaches its stationary point $\dot{\alpha}^{\mu}\rightarrow 0$
$\forall\mu$ and the system is in equilibrium.

  2. 2.

The matrix $G,H$ and $D^{\infty}$ also reaches its stationary point i.e.
$\frac{\partial G}{\partial t}\rightarrow 0,\frac{\partial H}{\partial
t}\rightarrow 0$ and $\frac{\partial D^{\infty}}{\partial t}\rightarrow 0$

  3. 3.

The maximum reached by the variance of parameters is given by
$\max\sigma_{\alpha}^{2}\sim\frac{1}{\beta\det(A^{\infty})}$

In recent literature, jastrzbski2018relation  showed that the eigenvalue of
hessian stabilizes for some epochs, which is an inference of the Theorem $1$:
part 1. For epochs satisfying Theorem $1$: Part 1, there will be isotropic
diffusion $D^{\infty}\rightarrow 0$, so there will be a simple (not
stochastic!) gradient descent as Levi-Civita connection coefficients
$\Gamma=0$. In other words, the low rank of the diffusion matrix
chaudhari2017stochastic  suggests that the majority of learning is having
simple gradient descent for most of the directions, as the parameters are near
critical points with minimum variance. Moreover, the matrix
$D^{\infty}\rightarrow 0$ corresponds to the absence of diffusion or coherent
process YanE4185  owing to the non-equilibrium phases and limit cycles shown
by chaudhari2017stochastic . Yan et al YanE4185  argued that these coherent
systems are a crucial part of the stability of the continuous memories in the
human brain. Inference drawn from the part 1 of Theorem $1$ is thus consistent
with the works chaudhari2017stochastic ; YanE4185 . The Part 2 of Theorem $1$
shows that the eigenvalues of hessian at some epochs reaches its maximum
value, which is consistent with the work jastrzbski2018relation . Our work
further shows that the system is in equilibrium at these epochs.

##  IV Stability analysis of Neural Network

We define approximated diffusion metric $\widetilde{D}^{\infty}=I_{K\times
K}+\epsilon D^{\infty}$, using equation $(13)$. The Lagrangian $\mathcal{L}$
is defined as
$\mathcal{L}=\sqrt{\sum_{\mu,\nu}^{K}\widetilde{D}_{\mu\nu}^{\infty}\frac{\partial\alpha^{\mu}}{\partial
t}\frac{\partial\alpha^{\nu}}{\partial t}}-V(\alpha,\dot{\alpha})$. The
external force for the parameter $\alpha^{\mu}$ is denoted by
$F_{\mu}=-\frac{\partial V}{\partial\alpha^{\mu}}$. Recent study Fioresi2020
used $\frac{\partial V}{\partial\alpha^{\mu}}=\frac{\partial}{\partial
t}(\frac{\partial f}{\partial\alpha^{\mu}})$ to derive the SGD equations for
parameters $\alpha$.

Identity 2: The potential $V$ under which the parameters have a geodesics path
analogous to SGD is given by (See D)

$V(\dot{\alpha})=\sum_{\mu=1}^{K}H_{\mu\mu}\int(\dot{\alpha}^{\mu})^{2}dt$
(13)

where $H$ is the hessian matrix of the function w.r.t. parameters $\alpha$.
Now using equation $(10)$ and the definition of Lagrangian, one can relate the
action $S=\int Ldt$ with the architecture matrix $A^{\infty}$ given by
equation $(14)$

$S=\int_{0}^{T}\Big{\\{}\sqrt{\begin{aligned}
&\sum_{\mu=1}^{K}\dot{\alpha}^{\mu}\dot{\alpha}_{\mu}+4\epsilon\sum_{\mu,\nu=1}^{K}\dot{\alpha}^{\mu}(C_{\mu\nu}^{\infty}\\\
&\quad\quad\quad+\sigma^{2}\sum_{\eta,\zeta=1}^{K}\mathcal{G}^{\mu}_{\eta}\mathcal{G}^{\nu}_{\zeta}A_{\eta\zeta}^{\infty})\dot{\alpha}^{\nu}\end{aligned}}-V(\dot{\alpha})\Big{\\}}dt$
(14)

where total training time $T>0$.

Following equation $(14)$, the action is coordinate invariant MIT2002symmetry
but depends on the path travelled under the learning rule $\dot{\alpha}$ and
is analogous to SGD Fioresi2020  given by the geodesic equation

$\begin{split}\frac{\partial\alpha}{\partial t}=&-(I-\epsilon
D^{\infty})\frac{\partial f}{\partial\alpha}\\\
\Rightarrow\frac{\partial\alpha}{\partial
t}=&-2GA^{\infty}(\alpha-\bar{\alpha})+2\epsilon
D^{\infty}GA^{\infty}(\alpha-\bar{\alpha})\end{split}$ (15)

The above equations are the dynamical equations governing the parameter
evolution with epochs. Considering the matrix $G=I$, the evolution of
parameters can be given as

$\alpha(t)=\bar{\alpha}+\exp(-tA^{\infty})[\alpha(0)-\bar{\alpha}]+\mathcal{O}(\epsilon)$
(16)

The above equation is equivalent to the evolution of parameters shown in
saxe2013exact  as an exact solution for deep linear networks. Moreover,
equation $(15)$ is a generalized form as it can not only capture deep linear
networks but any other generalized architecture where the matrix $G$ is a
function of $\alpha$. It is evident from equation $(16)$ that the eigenvalues
of the matrix $A^{\infty}$ decides the stability of the neural network. If the
matrix $A^{\infty}$ is positive definite, the system is stable. Whereas, if at
least one of the eigenvalue of the matrix $A^{\infty}$ is negative, a chaotic
nature will arise. There can also be limit cycles, which will arise if the
matrix $A^{\infty}$ is singular. Combining the stability analysis from
equation $(16)$ for the epochs in part $2$ of Theorem $1$ with the matrix
$G=I$, it becomes evident that the system reaches unstable equilibrium for
negative semi-definite matrix $A^{\infty}$. Keeping the generalization
capability of a neural network i.e. $\max\sigma_{\alpha}^{2}$ constant, the
quantity $\beta\det(A^{\infty})$ remains a constant according to part $2$ of
Theorem $1$,. Using the dynamical equation $(16)$, the inverse temperature
$\beta=\frac{2b}{\eta}$ act as a time-constant factor, or in other words, the
inverse of learning rate $1/\eta$ act as a time-constant factor, keeping batch
size $b$ constant. Thus, the paper establishes the time-constant factor in
equation $(16)$ for a generalized architecture, which corresponds to the time-
constant of the dynamical equations used in saxe2013exact  for teacher-student
network.

The problem of searching an optimal architecture-dataset matrix $A^{\infty}$
(architectures reaching maximum generalization capability with the highest
possible convergence rate) takes into account two aspects: generalization
capability i.e. $\max\sigma_{\alpha}^{2}$ and time-scale of parameter
optimization. The generalization capability tends towards infinity when the
architecture-dataset matrix tends towards singular $\det A^{\infty}\rightarrow
0$, using part 2 of Theorem $1$. On the other hand, the time-scale of
parameter optimization is given by the eigenvalues of the architecture-dataset
matrix $A^{\infty}$, which should be a large positive number resulting in a
faster optimization rate. Yet, the architecture-dataset matrix must have all
positive eigenvalue for the stability of the optimization. To increase the
generalization capability $\max\sigma_{\alpha}^{2}\rightarrow\infty$, the
matrix $A^{\infty}$ needs to be singular with at least one of the eigenvalue
$\lambda\rightarrow 0^{-}$ to satisfy epochs in part $2$ of Theorem $1$. This
condition asserts unstable limit cycles in phase space. So, it can be
concluded that increasing the generalization capability of a neural network
will give rise to limit cycles in the phase portrait of parameters and
decreases the convergence or optimization rate.

##  V Ricci scalar of learning manifold

Identity $3$: The Ricci scalar is given by (See E)

$R=\frac{\epsilon}{2}\sum_{i,j,p}\Big{\\{}2\frac{\partial^{2}D_{ip}^{\infty}}{\partial\alpha^{j}\partial\alpha^{i}}-\frac{\partial^{2}D_{ii}^{\infty}}{\partial\alpha^{j}\partial\alpha^{p}}-\frac{\partial^{2}D_{pp}^{\infty}}{\partial\alpha^{i}\partial\alpha^{i}}\Big{\\}}$
(17)

One can simplify the equation $(17)$ by constraining the matrix $G$. For
architectures having independent parameters $\alpha$, the matrix $G$ becomes
identity. Otherwise, in any generalized architecture for which parameters
$\alpha$ becomes independent for some epochs, the Ricci scalar is constant
given by

$\begin{split}R=&2\epsilon\sum_{i,j,p}(2A^{\infty}_{iipj}+A^{\infty}_{pi}A^{\infty}_{ij}+A^{\infty}_{ip}A^{\infty}_{pi}-A^{\infty}_{piij}\\\
&\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad-A^{\infty}_{ippi}-2A^{\infty}_{ii}A^{\infty}_{pj})\end{split}$
(18)

Identity 4: The Einstein tensor $E$ for architectures with $G=I$ can be
related with architecture matrix by (See F)

$\begin{split}E_{ik}&=2\epsilon\sum_{j,p}(A^{\infty}_{jpik}-A^{\infty}_{jp}A^{\infty}_{ik}+A^{\infty}_{ikpj}-A^{\infty}_{ik}A^{\infty}_{ip}\\\
&\quad\quad\quad\quad\quad\quad-A^{\infty}_{jkip}+A^{\infty}_{jk}A^{\infty}_{ip}-A^{\infty}_{ippk}+A^{\infty}_{ip}A^{\infty}_{pk})\\\
&-\frac{1}{2}\delta_{ik}\sum_{i,j,p}(2A^{\infty}_{iipj}+A^{\infty}_{pi}A^{\infty}_{ij}+A^{\infty}_{ip}A^{\infty}_{pi}-A^{\infty}_{piij}\\\
&\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad-A^{\infty}_{ippi}-2A^{\infty}_{ii}A^{\infty}_{pj})\end{split}$
(19)

Combining equation $(19)$ with Einstein’s field equation $E=K_{0}T_{s}$, where
$T_{s}$ is the energy-momentum tensor and $K_{0}>0$ a constant corresponding
to Einstein’s constant, it is apparent that the energy-momentum tensor $T_{s}$
depends on the architecture-dataset matrix. Thus for $G=I$, the mass
configuration is constant given by equation $(19)$. The architecture-data pair
has a constant corresponding mass configuration (energy density, momentum and
stress), which controls the trajectory of parameters in the learning manifold
of neural networks. For generalized architectures, as matrix $G$ evolves with
time, the corresponding mass configuration dynamically changes in the
background manifold. This opens up a possibility to control the learning
trajectory of the neural network using the matrix $G$.

##  VI Complexity-Action Conjecture

The trajectory of parameters in the Riemannian manifold i.e. diffusion metric,
can be correlated with the trajectory of a particle in the space-time, and
correlations with the principles of high energy physics is a possibility. One
such attempt is made in this section by correlating a famous conjecture in the
high energy physics called Complexity-Action Conjecture to the learning
dynamics of neural network.

Recent literature Kanwal_2017  defined complexity of a distribution $p$ by the
KL-divergence between $p$ and a distribution $p^{c}$ whose complexity is zero.
Complexity zero Kanwal_2017  is equivalent to the total information flow
($IF$) between past distribution $p^{c}(t^{-})$ and future distribution
$p^{c}(t^{+})$ being zero. Total information flow is given by

$IF(p(t^{-})\rightarrow
p(t^{+}))=\sum_{v}S(p_{v}(t^{+})|p_{v}(t^{-}))-S(p(t^{+})|p(t^{-}))$ (20)

where $S$ is the entropy measure and $p_{v}$ are the distribution of
microstates ensemble $v$. Here we defined a class of distribution $p^{c}$ for
which the complexity is $0$. The section applies this defination of complexity
on the parameter distribution.

Corollary 2: Both Dirac-delta distribution $p^{\delta}$ and an equilibrium
distribution $p^{ss}$ belongs to the distribution class $p^{c}$. (See G)

The time at which the parameter distribution reaches complexity $0$ is taken
as $T$. The parameter distribution $p(t)$ is characterised with $\alpha(t)$
which learns to reach at critical point $\bar{\alpha}$ by a learning rule
(here SGD). We define a parameter distribution $p(\alpha)$ whose parameter
$\alpha$ is $\epsilon^{\prime}-$near to the critical points $\bar{\alpha}$
i.e. $|\alpha^{\prime}-\bar{\alpha}|\sim\epsilon^{\prime}$ at time step
$T-\Delta t$, where $\Delta t\rightarrow 0$ and reaching its maximum
generalization capacity simultaneously. The complexity of distribution $p$ is
equal to the KL divergence between distribution $p$ characterised by
$\alpha=\bar{\alpha}+\epsilon^{\prime}$ & $p^{c}$ which is given by

$C(p,T-\Delta t)=KL(p(\bar{\alpha}+\epsilon^{\prime})||p^{c}(\bar{\alpha}))$
(21)

Theorem 2: The action for epochs satisfying part 1 of Theorem $1$ and part 2
of Theorem $1$ is given by (See H)

$S=2\sigma^{2}\epsilon C$ (22)

A similar argument has been made by relating quantum complexity and action of
a certain spacetime region called a Wheeler-DeWitt patch Brown_2016 . Here the
correspondence has been made between complexity of a parameter distribution
and action of parameters on the learning manifold.

##  VII Conclusion

A theoretical framework is proposed to estimate the learning capability of a
neural network for a particular dataset prior to its training. The paper
establishes how the architecture-dataset matrix governs the noise of the SGD
during training. At some epochs of the training phase, the system can show
absence of noise or diffusion phenomena with localized, non-equilibrium
states. Stabilisation of the hessian matrix is associated with such epochs.
The system can also reach equilibrium when the variance of parameters reaches
its maximum. The paper proves that neural networks will have a slower
convergence rate if its generalization capability is large. The stability of
the neural network has also been discussed using the eigenvalues of the
architecture-dataset matrix. The paper establishes the relationship between
the architecture-dataset matrix and the curvature of the diffusion metric. The
relationship signifies the correspondence of the trajectory of parameters in
the Diffusion metric with the trajectory of particles in the space-time
coordinates. Such correspondence correlates the principles of high-energy
physics with the learning theory of neural networks, using which the paper
establishes the Complexity-Action conjecture from an ANN perspective. Here,
the Complexity-Action correspondence has been made between the complexity of
the parameter distribution and the action of parameters on the learning
manifold. Using the analysis, one can also correlate any neuroscience model
(e.g. Hodgkin-Huxley model) with the generalized architecture of the neural
network as an extension of the work. The characteristics of learning in the
brain thus can be correlated with the learning dynamics of the neural network
established in the paper, among which few are discussed using the stability
and thermodynamics of the neural network. The correlation can not only help
the neuroscience community to understand the mechanism of the human brain, and
but also the computer science community to build better neural architectures.

## Acknowledgements

A.R. acknowledges the financial support received from Council of Scientific &
Industrial Research (CSIR), Govt. of India.

## References

  * [1] Kurt Hornik.  Approximation capabilities of multilayer feedforward networks.  Neural Networks, 4(2):251 – 257, 1991.
  * [2] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep Learning.  The MIT Press, 2016.
  * [3] Patrick Kidger and Terry Lyons.  Universal approximation with deep narrow networks, 2020.
  * [4] Zhou Lu, Hongming Pu, Feicheng Wang, Zhiqiang Hu, and Liwei Wang.  The expressive power of neural networks: A view from the width.  In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 6231–6239. Curran Associates, Inc., 2017.
  * [5] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals.  Understanding deep learning requires rethinking generalization, 2016.
  * [6] Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang.  Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks, 2019.
  * [7] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang.  Stronger generalization bounds for deep nets via a compression approach, 2018.
  * [8] Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina.  Entropy-sgd: Biasing gradient descent into wide valleys, 2016.
  * [9] Huan Wang, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher.  Identifying generalization properties in neural networks, 2018.
  * [10] Levent Sagun, Leon Bottou, and Yann LeCun.  Eigenvalues of the hessian in deep learning: Singularity and beyond, 2016\.
  * [11] Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio.  Sharp minima can generalize for deep nets.  In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 1019–1028. JMLR.org, 2017.
  * [12] Stanisław Jastrzębski, Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey.  On the relation between the sharpest directions of dnn loss and the sgd step length, 2018.
  * [13] Ravid Shwartz-Ziv and Naftali Tishby.  Opening the black box of deep neural networks via information, 2017.
  * [14] Ziv Goldfeld, Ewout Van Den Berg, Kristjan Greenewald, Igor Melnyk, Nam Nguyen, Brian Kingsbury, and Yury Polyanskiy.  Estimating information flow in deep neural networks.  In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2299–2308, Long Beach, California, USA, 09–15 Jun 2019. PMLR.
  * [15] Andrew K. Lampinen and Surya Ganguli.  An analytic theory of generalization dynamics and transfer learning in deep linear networks, 2018.
  * [16] Andrew M. Saxe, James L. McClelland, and Surya Ganguli.  Exact solutions to the nonlinear dynamics of learning in deep linear neural networks, 2013.
  * [17] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro.  Exploring generalization in deep learning.  In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS’17, page 5949–5958, Red Hook, NY, USA, 2017. Curran Associates Inc.
  * [18] Han Yan, Lei Zhao, Liang Hu, Xidi Wang, Erkang Wang, and Jin Wang.  Nonequilibrium landscape theory of neural networks.  Proceedings of the National Academy of Sciences, 110(45):E4185–E4194, 2013.
  * [19] Soatto Stefano Fioresi Rita, Chaudhari Pratik.  A geometric interpretation of stochastic gradient descent using diffusion metrics.  Entropy 22, no. 1: 101, 2020.
  * [20] William Bialek, Ilya Nemenman, and Naftali Tishby.  Predictability, complexity and learning, 2000.
  * [21] Pratik Chaudhari and Stefano Soatto.  Stochastic gradient descent performs variational inference, converges to limit cycles for deep networks, 2017.
  * [22] Edmund Bertschinger.  Symmetry transformations, the einstein-hilbert action, and gauge invariance.  MIT, 2002.
  * [23] Maxinder Kanwal, Joshua Grochow, and Nihat Ay.  Comparing information-theoretic measures of complexity in boltzmann machines.  Entropy, 19(7):310, Jul 2017.
  * [24] Adam R. Brown, Daniel A. Roberts, Leonard Susskind, Brian Swingle, and Ying Zhao.  Holographic complexity equals bulk action?  Physical Review Letters, 116(19), May 2016.
  * [25] George W. Bohrnstedt and Arthur S. Goldberger.  On the exact covariance of products of random variables.  Journal of the American Statistical Association, 64(328):1439–1442, 1969.

## Appendix

###  VII.1 Identity 1

Proof:
Using the definition of diffusion matrix $D$ from equation $(7)$

$D_{\mu\nu}=\frac{4}{N}\sum_{i}\\{\hat{y}_{i}-c(\alpha;x_{i})\\}\\{\hat{y}_{i}-c(\alpha;x_{i})\\}\frac{\partial
c(\alpha;x_{i})}{\partial\alpha^{\mu}}\frac{\partial
c(\alpha;x_{i})}{\partial\alpha^{\nu}}\\\
-\frac{4}{N^{2}}\sum_{i,j}\\{\hat{y}_{i}-c(\alpha;x_{i})\\}\\{\hat{y}_{j}-c(\alpha;x_{j})\\}\frac{\partial
c(\alpha;x_{i})}{\partial\alpha^{\mu}}\frac{\partial
c(\alpha;x_{j})}{\partial\alpha^{\nu}}\\\
=\frac{4}{N}\sum_{i}\\{\hat{y}_{i}-c(\alpha;x_{i})\\}^{2}\sum_{\eta,\zeta}\mathcal{G}^{\eta}_{\mu}\mathcal{G}^{\zeta}_{\nu}\phi_{\zeta}(x_{i})\phi_{\eta}(x_{i})-\frac{4}{N^{2}}\sum_{i}\\{\hat{y}_{i}-c(\alpha;x_{i})\\}\sum_{\eta}\mathcal{G}^{\eta}_{\mu}\phi_{\eta}(x_{i})\sum_{j}\\{\hat{y}_{j}-c(\alpha;x_{j})\\}\sum_{\zeta}\mathcal{G}^{\zeta}_{\nu}\phi_{\zeta}(x_{i})$
(23)

Using a similar approximation argument as [20], we let $N\rightarrow\infty$
and neglect the fluctuations.

$\begin{split}\frac{\partial
c(\alpha;x_{i})}{\partial\alpha^{\mu}}&=\sum_{\eta}\mathcal{G}^{\eta}_{\mu}\phi_{\eta}(x_{i})\hskip
5.69054pt[\text{as the basis functions are only functions of }x]\\\
\lim_{N\rightarrow\infty}\sum_{i=1}^{N}\hat{y}_{i}^{2}&=\Big{[}\sum_{\mu,\nu}\bar{\alpha}^{\mu}\sum_{i}\phi_{\mu}(x_{i})\phi_{\nu}(x_{i})\bar{\alpha}^{\nu}+\sigma^{2}\Big{]}\\\
\lim_{N\rightarrow\infty}\sum_{i=1}^{N}\hat{y}_{i}\phi_{\mu}(x_{i})&=\sum_{\nu}\sum_{i}\phi_{\mu}(x_{i})\phi_{\nu}(x_{i})\bar{\alpha}^{\nu}\end{split}$
(24)

Using above relations one can evaluated the diffusion matrix as

$D_{\mu\nu}^{\infty}\widetilde{\rightarrow}\frac{4}{N}(\\{\sum_{\eta,\zeta,p,q}\mathcal{G}^{\zeta}_{\mu}\mathcal{G}^{\eta}_{\nu}\bar{\alpha}^{p}[\sum_{i}\phi_{\zeta}(x_{i})\phi_{\eta}(x_{i})\phi_{p}(x_{i})\phi_{q}(x_{i})]\bar{\alpha}^{q}+\sigma^{2}\sum_{i}\phi_{\eta}(x_{i})\phi_{\zeta}(x_{i})\\}\\\
-2\sum_{\eta,\zeta,p,q}\mathcal{G}^{\zeta}_{\mu}\mathcal{G}^{\eta}_{\nu}\alpha^{p}[\sum_{i}\phi_{\eta}(x_{i})\phi_{\zeta}(x_{i})\phi_{p}(x_{i})\phi_{q}(x_{i})]\bar{\alpha}^{q}+\sum_{\eta,\zeta,p,q}\mathcal{G}^{\zeta}_{\mu}\mathcal{G}^{\eta}_{\nu}\alpha^{p}[\sum_{i}\phi_{\eta}(x_{i})\phi_{\zeta}(x_{i})\phi_{p}(x_{i})\phi_{q}(x_{i})]\alpha^{q})\\\
-\frac{4}{N^{2}}\\{\sum_{p,\eta}(\alpha^{p}-\bar{\alpha}^{p})\sum_{i}\mathcal{G}^{\eta}_{\mu}\phi_{\eta}(x_{i})\phi_{p}(x_{i})\\}\\{\sum_{q,\zeta}(\alpha^{q}-\bar{\alpha}^{q})\sum_{j}\mathcal{G}^{\zeta}_{\nu}\phi_{\zeta}(x_{j})\phi_{q}(x_{j})\\}\\\
\Rightarrow\frac{D_{\mu\nu}^{\infty}}{4}\widetilde{\rightarrow}\sigma^{2}\frac{1}{N}\sum_{i=1}^{N}\mathcal{G}^{\zeta}_{\mu}\mathcal{G}^{\eta}_{\nu}\phi_{\zeta}(x_{i})\phi_{\eta}(x_{i})+\sum_{\eta,\zeta,\eta,p,q}^{K}\mathcal{G}^{\zeta}_{\mu}\mathcal{G}^{\eta}_{\nu}(\alpha^{p}-\bar{\alpha}^{p})\\{\frac{1}{N}\sum_{i=1}^{N}\phi_{p}(x_{i})\phi_{\zeta}(x_{i})\phi_{\eta}(x_{i})\phi_{q}(x_{i})\\\
-\frac{1}{N}\sum_{i=1}^{N}\phi_{p}(x_{i})\phi_{\eta}(x_{i})\frac{1}{N}\sum_{i}^{N}\phi_{\zeta}(x_{i})\phi_{q}(x_{i})\\}(\alpha^{q}-\bar{\alpha}^{q})$
(25)

which can be represented as follows:

$\frac{D_{\mu\nu}}{4}\widetilde{\rightarrow}\sum_{p,k,\eta,\zeta}^{K}\mathcal{G}^{\eta}_{\mu}\mathcal{G}^{\zeta}_{\nu}(\alpha^{p}-\bar{\alpha}^{p})\\{A_{p\eta\zeta
k}^{\infty}-A_{p\eta}^{\infty}A_{\zeta
k}^{\infty}\\}(\alpha^{k}-\bar{\alpha}^{k})+\sigma^{2}\sum_{\eta,\zeta}^{K}\mathcal{G}^{\eta}_{\mu}\mathcal{G}^{\zeta}_{\nu}A_{\eta\zeta}^{\infty}$

where $4-$ rank tensor $A_{p\eta\zeta
q}^{\infty}=\lim_{N\rightarrow\infty}\frac{1}{N}\sum_{i=1}^{N}\phi_{p}(x_{i})\phi_{\eta}(x_{i})\phi_{\zeta}(x_{i})\phi_{q}(x_{i})$
and matrix $A^{\infty}$ is given by equation $(5)$.

###  VII.2 Lemma 1

Proof:

$\displaystyle A_{p\mu\nu q}^{\infty}-A_{p\mu}^{\infty}A_{\nu q}^{\infty}$
$\displaystyle=\lim_{N\rightarrow\infty}\frac{1}{N}\Sigma_{i=1}^{N}\phi_{p}(x_{i})\phi_{\mu}(x_{i})\phi_{\nu}(x_{i})\phi_{q}(x_{i})-\frac{1}{N}\Sigma_{i=1}^{N}\phi_{p}(x_{i})\phi_{\mu}(x_{i})\frac{1}{N}\Sigma_{i=1}^{N}\phi_{\nu}(x_{i})\phi_{q}(x_{i})$
$\displaystyle=\Big{<}\phi_{p}(x)\phi_{\mu}(x)\phi_{\nu}(x)\phi_{q}(x)\Big{>}-\Big{<}\phi_{p}(x)\phi_{\mu}(x)\Big{>}\Big{<}\phi_{\nu}(x)\phi_{q}(x)\Big{>}$
$\displaystyle=\text{cov}(\phi_{\nu}(x)\phi_{q}(x),\phi_{p}(x)\phi_{\mu}(x))$
$\displaystyle=\text{cov}(\phi_{p}(x)\phi_{\mu}(x),\phi_{\nu}(x)\phi_{q}(x))^{T}$
$\displaystyle=\\{A_{q\nu\mu p}^{\infty}-A_{\nu
q}^{\infty}A_{p\mu}^{\infty}\\}^{T}$

$\displaystyle C^{\infty}_{\mu\nu}$
$\displaystyle=\Sigma_{p,q}(\alpha^{p}-\bar{\alpha}^{p})\\{A_{p\mu\nu
q}^{\infty}-A_{p\mu}^{\infty}A_{\nu
q}^{\infty}\\}(\alpha^{q}-\bar{\alpha}^{q})$
$\displaystyle=\Sigma_{p,q}(\alpha^{p}-\bar{\alpha}^{p})\\{A_{q\nu\mu
p}^{\infty}-A_{p\mu}^{\infty}A_{\nu
q}^{\infty}\\}(\alpha^{q}-\bar{\alpha}^{q})\text{[using above relation]}$
$\displaystyle=\Sigma_{p,q}(\alpha^{p}-\bar{\alpha}^{p})\\{A_{p\nu\mu
q}^{\infty}-A_{p\nu}^{\infty}A_{\mu
q}^{\infty}\\}(\alpha^{q}-\bar{\alpha}^{q})\text{[interchanging dummy
variables]}$ $\displaystyle=C^{\infty}_{\nu\mu}$

Therefore the matrix $C^{\infty}$ is a symmetric matrix (though it is obvious
from the fact that both the matrix $D^{\infty}$ and $GA^{\infty}G$ is
symmetric)

For any vector $y$

$\begin{split}\sum_{\mu\nu}y^{\mu}C^{\infty}_{\mu\nu}y^{\nu}&=\sum_{\mu,p,q,\nu,\zeta,\eta}y^{\mu}(\alpha^{p}-\bar{\alpha}^{p})\mathcal{G}^{\zeta}_{\nu}\mathcal{G}^{\eta}_{\mu}\\{A_{p\eta\zeta
q}^{\infty}-A_{p\eta}^{\infty}A_{\zeta
q}^{\infty}\\}(\alpha^{q}-\bar{\alpha}^{q})y^{\nu}\\\
&=\sum_{\mu,p,q,\nu,\zeta,\eta}y^{\mu}(\alpha^{p}-\bar{\alpha}^{p})\mathcal{G}^{\zeta}_{\nu}\mathcal{G}^{\eta}_{\mu}\text{cov}(\phi_{\zeta}(x)\phi_{q}(x),\phi_{p}(x)\phi_{\eta}(x))(\alpha^{q}-\bar{\alpha}^{q})y^{\nu}\\\
&=\sum_{\mu,p,q,\nu,\eta,\zeta}V^{\nu q\zeta}\text{cov}(M_{\zeta q}(x),M_{\eta
p}(x))V^{\mu p\eta}\\\
&=\sum_{\mu,p,q,\nu,\eta,\zeta}\mathcal{G}^{\zeta}_{\nu}\mathcal{G}^{\eta}_{\mu}(\delta_{\eta\zeta}\delta_{pq}+\delta_{\zeta
q}\delta_{p\eta}+\delta_{\zeta\eta}+\delta_{\zeta q}\\\
&+\delta_{p\eta}+\delta_{pq})V^{\nu q\zeta}\text{cov}(M_{\zeta q}(x),M_{\eta
p}(x))V^{\mu p\eta}\hskip 5.69054pt[\text{using conditions J}_{1:6}]\\\
&=2\sum_{\mu,\nu,\eta,\zeta}V^{\mu\eta\zeta}V^{\mu\zeta\eta}\text{var}(M_{\zeta\eta}(x))+4\sum_{\mu,\nu,p,\zeta,\eta}V^{\mu
p\eta}\Big{<}\phi_{\zeta}(x)\Big{>}\Big{<}\phi_{\eta}(x)\Big{>}\text{var}(\phi_{p}(x))V^{\nu
p\zeta}\hskip 5.69054pt[\text{using }J_{7}]\\\
&=2\sum_{\mu,\nu,\zeta,\eta}V^{\mu\eta\zeta}V^{\mu\zeta\eta}\text{var}(M_{\zeta\eta}(x))\\\
&+4\sum_{\mu,\nu,p,\zeta,\eta}V^{\mu
p\zeta}\Big{<}\phi_{\zeta}(x)\phi_{\eta}(x)\Big{>}\text{var}(\phi_{p}(x))V^{\nu
p\eta}[\text{co variance of basis functions is 0}]\\\
&=2\sum_{\mu,\nu,\eta,\zeta}V^{\mu\eta\zeta}V^{\mu\zeta\eta}\text{var}(M_{\zeta\eta}(x))+4\sum_{\mu,\nu,p,\zeta,\eta}V^{\mu
p\zeta}A^{\infty}_{\zeta\eta}\text{var}(\phi_{p}(x))V^{\nu p\eta}\\\
&=2\sum_{\mu,\nu,\eta,\zeta}V^{\mu\eta\zeta}V^{\mu\zeta\eta}\text{var}(M_{\zeta\eta}(x))+\Big{[}2\sum_{\mu,\nu,\zeta,\eta}y^{\nu}\mathcal{G}^{\nu}_{\zeta}A^{\infty}_{\eta\zeta}\mathcal{G}^{\mu}_{\eta}y^{\mu}\Big{]}\Big{[}2\sum_{p}(\alpha^{p}-\bar{\alpha}^{p})^{2}\text{var}(\phi_{p}(x))\Big{]}\end{split}$
(26)

Here matrix $V^{\mu
p\eta}=y^{\mu}\mathcal{G}^{\mu}_{\eta}(\alpha^{p}-\bar{\alpha}^{p})$ and
$M_{\zeta q}(x)=\phi_{\zeta}(x)\phi_{q}(x)$.

Conditions $J$ for cross-covariance to be non-zero are
$J_{1}:(\eta-\zeta)^{2}+(p-q)^{2}=0;J_{2}:(\zeta-q)^{2}+(\eta-p)^{2}=0;$
$J_{3}:\zeta=\eta;$ $J_{4}:\eta=p;$ $J_{5}:p=q;$ $J_{6}:\zeta=q$. Because all
the basis functions are independent of each other.

$J_{7}:$ In literature [25] showed the co-variance of product of random
variables can be given as

$\displaystyle\text{cov}(xy,uv)$
$\displaystyle=\mathrm{E}(x)\mathrm{E}(u)\text{cov}(y,v)+\mathrm{E}(x)\mathrm{E}(v)\text{cov}(y,u)+\mathrm{E}(y)\mathrm{E}(u)\text{cov}(x,v)$
$\displaystyle+\mathrm{E}(y)\mathrm{E}(v)\text{cov}(x,u)+\text{cov}(x,u)\text{cov}(y,v)+\text{cov}(x,v)\text{cov}(y,u)$

where $x,y,u,w$ are random variables. The covariance of product is $0$ for
independent $x,y,u,w$. Two variables can be dependent only when they are equal
(using condition $J_{1:6}$), thus the covariance is non-zero and given by
$\text{cov}(xy,uy)=\mathrm{E}(x)\mathrm{E}(u)\text{var}(y)$

###  VII.3 Theorem 1

Proof:

One can rearrange equation $(12)$ as

$4\underset{(a)}{\underbrace{y^{T}GA^{\infty}Gy}}\underset{(b)\geq
0}{\Big{[}\underbrace{\sigma^{2}+4\Delta\alpha^{T}F\Delta\alpha}\Big{]}}+8\Big{(}\underset{(c)\geq
0}{\underbrace{y^{T}G\Delta\alpha^{T}Y\Delta\alpha Gy}}\Big{)}\geq 0$ (27)

for all vector $y$. The variable $(a)$ can be positive or negative depending
on the definiteness of $GA^{\infty}G$ matrix. If $(a)\geq 0$, the equation
$(26)$ is always true. The equation $(27)$ can be represented for negative
semi-definite matrix $y^{T}GA^{\infty}Gy<0$ or
$\mathrm{R}_{y}(\lambda_{GA^{\infty}G})<0$ as:

$\begin{split}y^{T}\frac{2G\Delta\alpha^{T}V\Delta\alpha
G}{\sigma^{2}+4\Delta\alpha^{T}F\Delta\alpha}y\geq&\Big{|}y^{T}GA^{\infty}Gy\Big{|}\\\
\Rightarrow\mathrm{R}_{y}(\frac{2G\Delta\alpha^{T}Y\Delta\alpha
G}{\sigma^{2}+4\Delta\alpha^{T}F\Delta\alpha})\geq&\Big{|}\mathrm{R}_{y}(GA^{\infty}G)\Big{|}\end{split}$
(28)

where Rayleigh quotient $\mathrm{R}_{y}(X)=\frac{y^{T}Xy}{y^{T}y}$, as $y$ can
be any vector, we drop the subscript $y$ from the Rayleigh quotient. For a
negative Rayleigh quotient $\mathrm{R}(X)<0$, the inequality
$\min\lambda_{X}\leq\mathrm{R}(X)\leq\max\lambda_{X}$ can be expressed as
$|\min\lambda_{X}|\geq|\mathrm{R}(X)|\geq|\max\lambda_{X}|$.

For part 1, as bias $b\rightarrow 0$ and variance of parameters
$\sigma_{\alpha}^{2}\rightarrow 0$, equation $(28)$ can be written as

$\begin{split}\lim_{\sigma_{\alpha}^{2}\rightarrow 0,b\rightarrow
0}\mathrm{R}(\frac{2G\mathrm{R}_{\alpha_{Y}}(\lambda_{Y})G}{\frac{\sigma^{2}}{\sigma_{\alpha}^{2}+b^{2}}+4\mathrm{R}_{\alpha_{F}}(\lambda_{F})})\geq&\Big{|}\mathrm{R}(GA^{\infty}G)\Big{|}\hskip
5.69054pt[\text{Dividing denominator and numerator by
}\Delta\alpha^{T}\Delta\alpha]\\\ \lim_{\epsilon^{\prime}\rightarrow
0}\epsilon^{\prime}\geq&\Big{|}\mathrm{R}(GA^{\infty}G)\Big{|}\geq\Big{|}\max\lambda_{GA^{\infty}G}\Big{|}\\\
\text{Therefore,}\hskip 5.69054pt\mathrm{R}(GA^{\infty}G)\rightarrow
0^{-}\hskip 5.69054pt&\&\hskip 5.69054pt\max\lambda_{GA^{\infty}G}\rightarrow
0^{-}\end{split}$ (29)

where
$\mathrm{R}_{\alpha_{Y}}(\lambda_{Y})=\frac{\Delta\alpha^{T}Y\Delta\alpha}{\Delta\alpha^{T}\Delta\alpha}$.
Now using equation $(10)$, the diffusion matrix $D^{\infty}\rightarrow
4\sigma^{2}GA^{\infty}G$ when $\Delta\alpha\rightarrow 0$. Given the matrix
$GA^{\infty}G$ is negative semi-definite, equation $(12)$ shows that maximum
of eigenvalue of diffusion matrix
$\mathrm{R}(D^{\infty})\leq\max\lambda_{GA^{\infty}G}\rightarrow 0^{-}$, a
contradiction ! So the order of $\epsilon^{\prime}$ must be constrained in
$\mathcal{O}(\epsilon^{\prime})>\mathcal{O}(|\max\lambda_{GA^{\infty}G}|)$
which is important to avoid the contradiction.

Rewriting equation $(27)$, the inequality can be rewritten as

$2\mathrm{R}(G^{2})\Big{[}\Delta\alpha^{T}Y\Delta\alpha\Big{]}\geq|\mathrm{R}(GA^{\infty}G)|\Big{[}\sigma^{2}+4\Delta\alpha^{T}F\Delta\alpha\Big{]}$
(30)

The order of matrix $Y$ i.e.
$\mathcal{O}(Y)=\mathcal{O}(\text{var}(\phi^{2}))=\mathcal{O}(\text{var}(\phi)^{2})=\mathcal{O}(F^{2})$.
One can represent the inequality in $(28)$ in terms of order magnitude as

$\begin{split}2\frac{\max\lambda_{G^{2}}}{|\max\lambda_{GA^{\infty}G}|}\geq\frac{2\mathrm{R}(G^{2})}{|\mathrm{R}(GA^{\infty}G)|}&\geq\frac{\sigma^{2}+4\Delta\alpha^{T}F\Delta\alpha}{\Delta\alpha^{T}Y\Delta\alpha}\\\
\mathcal{O}\Big{(}\frac{\max\lambda_{G^{2}}}{|\max\lambda_{GA^{\infty}G}|}\Big{)}&\geq\mathcal{O}\Big{(}\frac{\sigma^{2}}{\Delta\alpha^{T}Y\Delta\alpha}\Big{)}+\mathcal{O}(\frac{1}{\text{det}(F)})\\\
\mathcal{O}\Big{(}\frac{\max\lambda_{G^{2}}}{|\max\lambda_{GA^{\infty}G}|}-\frac{1}{\text{det}(F)}\Big{)}&\geq\mathcal{O}\Big{(}\frac{\sigma^{2}}{\Delta\alpha^{T}Y\Delta\alpha}\Big{)}\\\
\mathcal{O}(\Delta\alpha^{T}Y\Delta\alpha)&\geq\mathcal{O}\Big{(}\frac{\sigma^{2}}{\frac{\max\lambda_{G^{2}}}{|\max\lambda_{GA^{\infty}G}|}-\frac{1}{\text{det}(F)}}\Big{)}\\\
\mathcal{O}(\Delta\alpha)&\geq\sqrt{\mathcal{O}\Big{(}\frac{\sigma^{2}|\max\lambda_{GA^{\infty}G}|}{\text{det}(F^{2})|\max\lambda_{G^{2}}|-\text{det}(F)|\max\lambda_{GA^{\infty}G}|}\Big{)}}\end{split}$
(31)

Though both $\Delta\alpha$ and $\max\lambda_{GA^{\infty}G}$ tends towards $0$,
the order of $\Delta\alpha$ is still restricted to inequality $(31)$ making
the diffusion matrix $D^{\infty}\rightarrow 0^{+}$. Moreover using equation
$(6)$, the Hessian $H=2GA^{\infty}G+2GA^{\infty}\Delta\alpha$, shows the
expected value of $H$ also tends to $\mathrm{R}(H)\rightarrow 0^{-}$ for
$\mathcal{O}(\Delta\alpha)<\mathcal{O}(\det G)$ and the expected value of
$\mathrm{R}(H)\rightarrow 0^{+}$ for
$\mathcal{O}(\Delta\alpha)>\mathcal{O}(\det G)$. In both case, the eigenvalue
of hessian tends towards $0$.

For part 2, consider the phase portrait of parameters with $\alpha$ and
$\dot{\alpha}$ as the state vectors. According to Lioville’s theorem, in non-
dissipative system the phase flow remains incompressible i.e. the area of
phase portait remains constant. In other words, the spread or variance of one
state vector competes with the variance of the other state vector owing to the
area conservation. Here, as bias $\sigma_{\alpha}^{2}$ is maximum, the
variance of velocity of parameters $\sigma_{\dot{\alpha}}^{2}\rightarrow 0$
according to Lioville’s theorem. Moreover, as bias attains its minimum value
$b\rightarrow 0$ the expected velocity of parameter
$\Big{<}\dot{\alpha}\Big{>}\rightarrow 0$. Combining both mean and variance of
the momenta $\dot{\alpha}$ becoming $0$, almost all of the index $\mu$ of
$\dot{\alpha}^{\mu}\rightarrow 0$. Using condition $4$ of Theorem $1$, the
velocity of parameters can be related as:

$\begin{split}\frac{\partial\alpha^{\nu}}{\partial
t}=&\sum_{l\in\\{L(\nu)\\}}\frac{\partial g^{\nu}(w)}{\partial
w^{l}}\frac{\partial w^{l}}{\partial t}\\\ \frac{\partial\alpha}{\partial
t}=&J\frac{\partial w}{\partial t}\\\
J^{\dagger}\frac{\partial\alpha}{\partial t}=&\frac{\partial w}{\partial t}\\\
\frac{\partial w}{\partial t}=&0\end{split}$ (32)

where $J^{\dagger}$ is a left inverse of $J$. Thus the system is in
equilibrium. From equation $(28)$, considering
$\max\sigma_{\alpha}^{2}>>\sigma^{2}$,

$\begin{split}\mathrm{R}(\frac{G\mathrm{R}_{\alpha_{Y}}(\lambda_{Y})G}{2\mathrm{R}_{\alpha_{F}}(\lambda_{F})})\geq&\Big{|}\mathrm{R}(GA^{\infty}G)\Big{|}\\\
\mathrm{R}(G\Big{\\{}I\frac{\mathrm{R}_{\alpha_{V}}(\lambda_{V})}{2\mathrm{R}_{\alpha_{F}}(\lambda_{F})}+A^{\infty}\Big{\\}}G)\geq&0\\\
\end{split}$ (33)

The evolution of matrix $G$ plays an important role in the system.

$\begin{split}\frac{\partial\mathcal{G}^{\mu}_{\nu}}{\partial
t}=&\sum_{l,p=1}^{L}\frac{\partial^{2}g^{\mu}(w)}{\partial w^{p}\partial
w^{l}}\frac{\partial w^{l}}{\partial g^{\nu}(w)}\dot{w}^{p}+\frac{\partial
g^{\mu}(w)}{\partial w^{l}}\frac{\partial}{\partial
w^{p}}\Big{(}\frac{1}{\frac{\partial g^{\nu}(w)}{\partial
w^{l}}}\Big{)}\dot{w}^{p}\end{split}$ (34)

So for $\dot{w}=0$, the matrix $G$ has reached a stationary point i.e.
$\dot{G}=0$ as well. Similarly, the diffusion matrix $D^{\infty}$ becomes
stationary and also the hessian matrix is stationary using equation $(10)$ and
equation $(6)$, respectively.

Till now, the parameter $\alpha$ is denoted as a vector whose mean and
variance changes with epochs. So, the parameters vector $\alpha$ is collected
from one individual trajectory of the parameters across epochs. One can
generalize by collecting multiple such trajectories. The parameter vector now
generalizes to parameter matrix, each column representing $\alpha^{\mu}$
$\forall\mu$ as a vector, having a mean of $\Big{<}\alpha^{\mu}\Big{>}$
$\forall\mu$. Each column vector of parameters will have a corresponding
expected ’optimal’ parameter $\Big{<}\bar{\alpha}^{\mu}\Big{>}$ $\forall\mu$.
Hence, the bias $b=|\Big{<}\alpha\Big{>}-\Big{<}\bar{\alpha}\Big{>}|$ is a
vector. The variance of parameters $\sigma_{\alpha}^{2}$ becomes the co-
variance matrix $\Sigma_{\alpha}$. The matrix $G$ is then defined by
$\mathcal{G}^{\mu}_{\nu}=\frac{\partial\Big{<}\alpha^{\mu}\Big{>}}{\partial\Big{<}\alpha^{\nu}\Big{>}}$.
The distribution of parameters is given by
$\mathcal{N}(\Big{<}\alpha\Big{>},\Sigma_{\alpha})$. For a single roll-out of
parameters $\vec{\alpha}$ across epochs, the potential $\Phi$ can be written
from equation $(8)$

$\begin{split}\Phi(\vec{\alpha})=&-\beta^{-1}\log(\rho^{ss}(\alpha))\\\
=&\beta^{-1}\Big{[}\frac{K}{2}\log|\Sigma_{\alpha}|+\frac{K}{2}\log(2\pi)+\frac{1}{2}(\vec{\alpha}-\Big{<}\alpha\Big{>})^{T}\Sigma^{-1}_{\alpha}(\vec{\alpha}-\Big{<}\alpha\Big{>})\Big{]}\\\
\Rightarrow\frac{\partial\Phi(\vec{\alpha})}{\partial\Big{<}\alpha^{\eta}\Big{>}}=&2\beta^{-1}\sum_{\mu,\nu=1}^{K}(\vec{\alpha}-\Big{<}\alpha^{\mu}\Big{>})\Sigma_{\alpha}^{-1}(\mu,\nu)\mathcal{G}^{\nu}_{\eta}\\\
\Rightarrow\nabla\Phi(\vec{\alpha})=&2\beta^{-1}G\Sigma_{\alpha}^{-1}(\vec{\alpha}-\Big{<}\alpha\Big{>})\end{split}$
(35)

On the other hand, $\nabla\Phi(\alpha)=\widetilde{D^{\infty}}\nabla f$, where
$\nabla f=2GA^{\infty}(\vec{\alpha}-\Big{<}\bar{\alpha}\Big{>})$. As bias
$b\rightarrow\vec{0}$, the expected value of parameter
$\Big{<}\alpha\Big{>}\rightarrow\Big{<}\bar{\alpha}\Big{>}$, thus equating two
equation

$\begin{split}\beta\widetilde{D^{\infty}}GA^{\infty}(\vec{\alpha}-\Big{<}\bar{\alpha}\Big{>})=&G\Sigma_{\alpha}^{-1}(\vec{\alpha}-\Big{<}\bar{\alpha}\Big{>})\\\
\Rightarrow\beta\Sigma_{\alpha}G^{-1}\widetilde{D^{\infty}}GA^{\infty}=&I\\\
\Rightarrow\beta\Sigma_{\alpha}A^{\infty}+\epsilon\beta\Sigma_{\alpha}G^{-1}D^{\infty}GA^{\infty}=&I\\\
\Rightarrow\det(\Sigma_{\alpha})=\frac{1}{\beta\det(A^{\infty})}+\mathcal{O}(\epsilon)\end{split}$
(36)

Reducing the dimensionality, the variance of parameters
$\sigma_{\alpha}^{2}\sim\det(\Sigma_{\alpha})=\frac{1}{\beta\det(A^{\infty})}$.

###  VII.4 Identity 2

Proof:

Evaluating the potential used by [19],

$\begin{split}\frac{\partial
V}{\partial\alpha^{\mu}}&=\frac{\partial}{\partial t}(\frac{\partial
f}{\partial\alpha^{\mu}})\\\ \Rightarrow\frac{\partial V}{\partial
t}&=\sum_{\mu}\dot{\alpha}^{\mu}\frac{\partial}{\partial t}(\frac{\partial
f}{\partial\alpha^{\mu}})\hskip 5.69054pt\text{(following rules of Einstein's
summation notation)}\\\ \Rightarrow
V&=\sum_{\mu}\int\dot{\alpha}^{\mu}\frac{\partial}{\partial t}(\frac{\partial
f}{\partial\alpha^{\mu}})dt\\\ &=\sum_{\mu}\dot{\alpha}^{\mu}\frac{\partial
f}{\partial\alpha^{\mu}}-\sum_{\mu}\int\ddot{\alpha}^{\mu}\frac{\partial
f}{\partial\alpha^{\mu}}dt\\\ &=\sum_{\mu}\dot{\alpha}^{\mu}\frac{\partial
f}{\partial\alpha^{\mu}}-\sum_{\mu}\int\frac{\ddot{\alpha}^{\mu}}{\dot{\alpha}^{\mu}}\frac{\partial
f}{\partial\alpha^{\mu}}d\alpha^{\mu}\\\
&=\sum_{\mu}\dot{\alpha}^{\mu}\frac{\partial
f}{\partial\alpha^{\mu}}-\sum_{\mu}\Big{\\{}\frac{\partial
f}{\partial\alpha^{\mu}}\dot{\alpha}^{\mu}-\frac{\partial^{2}f}{\partial\alpha^{\mu}\partial\alpha^{\mu}}\int(\dot{\alpha}^{\mu})^{2}dt\Big{\\}}\\\
&=\sum_{\mu}H_{\mu\mu}\int(\dot{\alpha}^{\mu})^{2}dt\end{split}$ (37)

where $H$ is the hessian matrix of the loss function. If the learning
principle of one parameter coordinate $\alpha^{\mu}$ would depend on
$\alpha^{\nu}$ then the potential $V$ would have been a summation over both
dummy variable $\mu,\nu$.

###  VII.5 Identity 3

Proof:
The Ricci Tensor is given by on the approximated diffusion metric
$\widetilde{D}^{\infty}=I_{K\times K}+\epsilon D^{\infty}$

$\begin{split}R_{ijk}^{j}&=\frac{\partial\Gamma_{ik}^{j}}{\partial\alpha^{j}}-\frac{\partial\Gamma_{jk}^{j}}{\partial\alpha^{i}}+\sum_{p}(\Gamma_{ik}^{p}\Gamma_{jp}^{j}-\Gamma_{jk}^{p}\Gamma_{ip}^{j})\\\
&=\frac{\epsilon}{2}\sum_{p}\Big{[}\frac{\partial}{\partial\alpha^{j}}\Big{\\{}\delta_{jp}(\frac{\partial
D_{pi}^{\infty}}{\partial\alpha^{k}}+\frac{\partial
D_{pk}^{\infty}}{\partial\alpha^{i}}-\frac{\partial
D_{ki}^{\infty}}{\partial\alpha^{p}})\Big{\\}}-\frac{\partial}{\partial\alpha^{i}}\Big{\\{}\delta_{jp}(\frac{\partial
D_{pj}^{\infty}}{\partial\alpha^{k}}+\frac{\partial
D_{pk}^{\infty}}{\partial\alpha^{j}}-\frac{\partial
D_{kj}^{\infty}}{\partial\alpha^{p}})\Big{\\}}\Big{]}+\mathcal{O}(\epsilon^{2})\\\
&=\frac{\epsilon}{2}\sum_{p}\Big{\\{}\frac{\partial^{2}D_{pi}^{\infty}}{\partial\alpha^{j}\partial\alpha^{k}}+\frac{\partial^{2}D_{kp}^{\infty}}{\partial\alpha^{i}\partial\alpha^{j}}-\frac{\partial^{2}D_{ki}^{\infty}}{\partial\alpha^{j}\partial\alpha^{p}}-\frac{\partial^{2}D_{pp}^{\infty}}{\partial\alpha^{i}\partial\alpha^{k}}\Big{\\}}\\\
\Rightarrow
R_{ik}&=\frac{\epsilon}{2}\sum_{j,p}\Big{\\{}\frac{\partial^{2}D_{pi}^{\infty}}{\partial\alpha^{j}\partial\alpha^{k}}+\frac{\partial^{2}D_{kp}^{\infty}}{\partial\alpha^{i}\partial\alpha^{j}}-\frac{\partial^{2}D_{ki}^{\infty}}{\partial\alpha^{j}\partial\alpha^{p}}-\frac{\partial^{2}D_{pp}^{\infty}}{\partial\alpha^{i}\partial\alpha^{k}}\Big{\\}}\\\
\Rightarrow
R&=\frac{\epsilon}{2}\sum_{i,k}\delta_{ik}\Big{[}\sum_{j,p}\Big{\\{}\frac{\partial^{2}D_{pi}^{\infty}}{\partial\alpha^{j}\partial\alpha^{k}}+\frac{\partial^{2}D_{kp}^{\infty}}{\partial\alpha^{i}\partial\alpha^{j}}-\frac{\partial^{2}D_{ki}^{\infty}}{\partial\alpha^{j}\partial\alpha^{p}}-\frac{\partial^{2}D_{pp}^{\infty}}{\partial\alpha^{i}\partial\alpha^{k}}\Big{\\}}\Big{]}+\mathcal{O}(\epsilon^{2})\\\
&=\frac{\epsilon}{2}\sum_{i,j,p}\Big{\\{}2\frac{\partial^{2}D_{ip}^{\infty}}{\partial\alpha^{j}\partial\alpha^{i}}-\frac{\partial^{2}D_{ii}^{\infty}}{\partial\alpha^{j}\partial\alpha^{p}}-\frac{\partial^{2}D_{pp}^{\infty}}{\partial\alpha^{i}\partial\alpha^{i}}\Big{\\}}\end{split}$
(38)

For matrix $G=I$, the Diffusion matrix is given by

$D^{\infty}(\alpha)=4\sigma^{2}A^{\infty}+4C^{\infty}(\alpha)$ (39)

where
$C^{\infty}_{\mu\nu}=\sum_{p,k}(\alpha^{p}-\bar{\alpha}^{p})\\{A_{p\mu\nu
k}^{\infty}-A_{p\mu}^{\infty}A_{\nu
k}^{\infty}\\}(\alpha^{k}-\bar{\alpha}^{k})$. So the Ricci Scalar can be
represented for these architecture as

$\begin{split}R=&2\epsilon\sum_{i,j,p}\Big{\\{}2\frac{\partial^{2}C_{ip}^{\infty}}{\partial\alpha^{j}\partial\alpha^{i}}-\frac{\partial^{2}C_{ii}^{\infty}}{\partial\alpha^{j}\partial\alpha^{p}}-\frac{\partial^{2}C_{pp}^{\infty}}{\partial\alpha^{i}\partial\alpha^{i}}\Big{\\}}\\\
=&2\epsilon\sum_{i,j,p}(2A^{\infty}_{iipj}+A^{\infty}_{pi}A^{\infty}_{ij}+A^{\infty}_{ip}A^{\infty}_{pi}-A^{\infty}_{piij}-A^{\infty}_{ippi}-2A^{\infty}_{ii}A^{\infty}_{pj})\end{split}$
(40)

The Ricci scalar is a constant when matrix $G=I$, in other words the
parameters or coordinates are independent of each other.

###  VII.6 Identity 4

Proof:
The Einstein tensor can be given for architectures with $G=I$ as

$\begin{split}G_{ik}&=\frac{\epsilon}{2}\sum_{j,p}\Big{\\{}\frac{\partial^{2}D_{pi}^{\infty}}{\partial\alpha^{j}\partial\alpha^{k}}+\frac{\partial^{2}D_{kp}^{\infty}}{\partial\alpha^{i}\partial\alpha^{j}}-\frac{\partial^{2}D_{ki}^{\infty}}{\partial\alpha^{j}\partial\alpha^{p}}-\frac{\partial^{2}D_{pp}^{\infty}}{\partial\alpha^{i}\partial\alpha^{k}}\Big{\\}}-\frac{1}{2}\delta_{ik}\sum_{i,j,p}\Big{\\{}2\frac{\partial^{2}D_{ip}^{\infty}}{\partial\alpha^{j}\partial\alpha^{i}}-\frac{\partial^{2}D_{ii}^{\infty}}{\partial\alpha^{j}\partial\alpha^{p}}-\frac{\partial^{2}D_{pp}^{\infty}}{\partial\alpha^{i}\partial\alpha^{i}}\Big{\\}}+\mathcal{O}(\epsilon^{2})\\\
&=2\epsilon\sum_{j,p}(A^{\infty}_{jpik}-A^{\infty}_{jp}A^{\infty}_{ik}+A^{\infty}_{ikpj}-A^{\infty}_{ik}A^{\infty}_{ip}-A^{\infty}_{jkip}+A^{\infty}_{jk}A^{\infty}_{ip}-A^{\infty}_{ippk}+A^{\infty}_{ip}A^{\infty}_{pk})\\\
&-\frac{1}{2}\delta_{ik}\sum_{i,j,p}(2A^{\infty}_{iipj}+A^{\infty}_{pi}A^{\infty}_{ij}+A^{\infty}_{ip}A^{\infty}_{pi}-A^{\infty}_{piij}-A^{\infty}_{ippi}-2A^{\infty}_{ii}A^{\infty}_{pj})\end{split}$
(41)

###  VII.7 Corollary 2

For dirac delta function $p^{\delta}$, there is only one microstate $\nu=1$.
So $IF=C=0$. On the other hand, the future equilibrium distribution
$p^{ss}(t^{+})$ can be completely predicted by the past distribution
$p^{ss}(t^{-})$ thus $H(p^{ss}(t^{+})|p^{ss}(t^{-}))=0$. Moreover as the
distribution is in equilibrium, probability of micro states $v$ are stationary
$H(p_{v}^{ss}(t^{+})|p_{v}^{ss}(t^{-}))=0$ $\forall v$. Thus the $IF$ and
complexity of any distribution in equilibrium is zero.

###  VII.8 Theorem 2

Proof:
For $\Delta t\rightarrow 0$,

$\begin{split}C(T-\Delta
t)&=D_{KL}(p(\bar{\alpha}+\epsilon^{\prime})||p^{c}(\bar{\alpha}))\\\
&=\frac{1}{2}\sum_{\mu,\nu}^{K}\Delta\alpha^{\mu}\Big{[}\frac{1}{N}\sum_{i=1}^{N}\nabla_{\mu}\log
p^{t}((x_{i},y_{i})|\alpha)\nabla_{\nu}\log
p^{t}((x_{i},y_{i})|\alpha)\Big{]}_{\mu\nu}\Delta\alpha^{\nu}\\\
&=\frac{1}{4\sigma^{2}}\sum_{\mu,\nu}^{K}\Delta\alpha^{\mu}\Big{[}\frac{1}{N}\sum_{i=1}^{N}\nabla_{\mu}f_{i}\nabla_{\nu}f_{i}\Big{]}_{\mu\nu}\Delta\alpha^{\nu}\\\
&=\frac{1}{\sigma^{2}}\sum_{p,k,\eta,\zeta,\mu,\nu}^{K}\mathcal{G}^{\eta}_{\mu}\mathcal{G}^{\zeta}_{\nu}\Delta\alpha^{\mu}(\alpha^{p}-\bar{\alpha}^{p})A_{p\eta\zeta
k}^{\infty}(\alpha^{k}-\bar{\alpha}^{k})\Delta\alpha^{\nu}+\sum_{\zeta,\eta,\mu,\nu}^{K}\Delta\alpha^{\mu}\mathcal{G}^{\eta}_{\mu}\mathcal{G}^{\zeta}_{\nu}A_{\eta\zeta}^{\infty}\Delta\alpha^{\nu}\hskip
5.69054pt\Big{[}\text{see equation $(22-24)$}\Big{]}\\\
\Rightarrow\frac{\partial C}{\partial t}\Delta
t&=\frac{1}{\sigma^{2}}\sum_{p,k,\eta,\zeta,\mu,\nu}^{K}\mathcal{G}^{\eta}_{\mu}\mathcal{G}^{\zeta}_{\nu}\Delta\alpha^{\mu}(\alpha^{p}-\bar{\alpha}^{p})A_{p\eta\zeta
k}^{\infty}(\alpha^{k}-\bar{\alpha}^{k})\Delta\alpha^{\nu}+\sum_{\zeta,\eta,\nu,\mu}^{K}\Delta\alpha^{\mu}\mathcal{G}^{\eta}_{\mu}\mathcal{G}^{\zeta}_{\nu}A_{\eta\zeta}^{\infty}\Delta\alpha^{\nu}\hskip
5.69054pt[C(T)=0]\end{split}$ (42)

Rewriting an approximation of equation $(14)$ using the definition of
Lagrangian $L$ for small $\epsilon<<1$ and $\Delta t\rightarrow 0$.

$\begin{split}Ldt&=\sum_{\mu}^{K}d\alpha^{\mu}d\alpha_{\mu}+\sum_{\mu,\nu}^{K}\frac{\epsilon}{2}D_{\mu\nu}^{\infty}d\alpha_{\mu}d\alpha_{\nu}-V(\dot{\alpha})dt\\\
&=\sum_{\mu}^{K}d\alpha^{\mu}d\alpha_{\mu}+2\sigma^{2}\epsilon\frac{\partial
C}{\partial
t}dt-2\epsilon\sum_{\mu,\nu,p,k,\eta,\zeta}^{K}\mathcal{G}^{\eta}_{\mu}\mathcal{G}^{\zeta}_{\nu}d\alpha^{\mu}(\alpha^{p}-\bar{\alpha}^{p})A_{p\eta}^{\infty}A_{\zeta
k}^{\infty}(\alpha^{k}-\bar{\alpha}^{k})d\alpha^{\nu}-V(\dot{\alpha})dt\\\
\frac{dS}{dt}&=\sum_{\mu}^{K}\dot{\alpha}^{\mu}\dot{\alpha}_{\mu}dt+2\sigma^{2}\epsilon\frac{\partial
C}{\partial
t}-2\epsilon\sum_{\mu,\nu,p,k,\eta,\zeta}^{K}\mathcal{G}^{\eta}_{\mu}\mathcal{G}^{\zeta}_{\nu}\dot{\alpha}^{\mu}(\alpha^{p}-\bar{\alpha}^{p})A_{p\eta}^{\infty}A_{\zeta
k}^{\infty}(\alpha^{k}-\bar{\alpha}^{k})\dot{\alpha}^{\nu}-V(\dot{\alpha})\end{split}$
(43)

For $\alpha\rightarrow\bar{\alpha}$ and $dt\rightarrow 0$, the rate of change
of action can be equated with

$\begin{split}\frac{dS}{dt}=2\sigma^{2}\epsilon\frac{\partial C}{\partial
t}-\sum_{\mu}^{K}H_{\mu\mu}\int(\dot{\alpha}^{\mu})^{2}dt\\\ \end{split}$ (44)

Epochs satisfying part 1 of Theorem $1$ , the hessian matrix tends toward a
null matrix, thus $H_{\mu\mu}=0$ $\forall\mu$ which makes the equation $(44)$
satisfy $S=2\sigma^{2}\epsilon C$. Epochs satisfying part 2 of Theorem $1$,
the system reaches equilibrium $\dot{\alpha}\rightarrow 0$ thus making the
equation $(44)$ satisfy $S=2\sigma^{2}\epsilon C$.