# Invariance Properties of the Natural Gradient in Overparametrised
Systems111A previous version of this paper [14] was presented at the
International Conference on Geometric Science of Information 2021 in Paris.
Here invariance properties for proper parametrisations were discussed. The
current paper treats invariance for both proper and non-proper
parametrisations.

Jesse van Oostrum  Hamburg University of Technology, Hamburg, Germany
Johannes Müller  Max Planck Institute for Mathematics in the Sciences,
Leipzig, Germany  Nihat Ay  Hamburg University of Technology, Hamburg, Germany
Leipzig University, Leipzig, Germany  Santa Fe Institute, Santa Fe, USA

###### Abstract

The natural gradient field is a vector field that lives on a model equipped
with a distinguished Riemannian metric, e.g. the Fisher-Rao metric, and
represents the direction of steepest ascent of an objective function on the
model with respect to this metric. In practice, one tries to obtain the
corresponding direction on the parameter space by multiplying the ordinary
gradient by the inverse of the Gram matrix associated with the metric. We
refer to this vector on the parameter space as the natural parameter gradient.
In this paper we study when the pushforward of the natural parameter gradient
is equal to the natural gradient. Furthermore we investigate the invariance
properties of the natural parameter gradient. Both questions are addressed in
an overparametrised setting.

##  1 Introduction

Within the field of deep learning, gradient methods have become ubiquitous
tools for parameter optimisation. Standard gradient optimisation procedures
use the vector of coordinate derivatives of the objective function as the
update direction of the parameters. This is implicitly assuming a Euclidean
geometry on the space of parameters. It can be argued that this is not always
the most natural choice of geometry. Instead one can choose a more natural
geometry for the problem at hand and then determine the Riemannian gradient of
the objective function for this natural geometry, resulting in the so called
_natural gradient_. The _natural gradient method_ is the optimisation
algorithm that performs discrete parameter updates in the direction of the
natural gradient. This method was first proposed by Amari [1] using the
geometry induced by the Fisher-Rao metric. It is an active field of study
within information geometry [3, 10, 6] and has been shown extremely effective
in many applications [4, 16, 17]. More recently, also other geometries on the
model have been studied, such as the Wasserstein geometry [11, 9]. The natural
gradient is defined independently of a specific parametrisation. Although it
is an open problem, there is work supporting the idea that the efficiency of
learning of the method is due to this invariance [18].

In practice, the update direction of the parameters is given by the ordinary
gradient multiplied by the inverse of the Gram matrix associated with the
metric on the model. We will refer to this vector on the parameter space as
the _natural parameter gradient_. In order to determine whether this direction
is desired we have to map this vector to the model, since it is the location
on the model, not on the parameter space, that determines the performance of
the model. In non-overparametrised systems it can be shown that in a non-
singular point on the model, the pushforward of the natural parameter gradient
is equal to the natural gradient. Furthermore the natural parameter gradient
can be called parametrisation invariant in this case [12]. In many practical
applications of machine learning, and in particular deep learning, one deals
however with overparametrised models, in which different directions on the
parameter space correspond to a single direction on the model. In this case,
the Gram matrix is degenerate and we use a generalised inverse to calculate
the natural parameter gradient. In this paper, we will investigate whether the
pushforward of the natural parameter gradient remains equal to the natural
gradient in the overparametrised setting.

The Moore-Penrose (MP) inverse is the canonical choice of generalised inverse
for the natural parameter gradient [5]. The definition of the MP inverse is
based on the Euclidean inner product defined on the parameter space. Using the
MP inverse is therefore thought to affect the parametrisation invariance of
the natural parameter gradient [13], and thus potentially the performance of
the natural gradient method. In this paper we propose two different notions of
invariance. The first evaluates the invariance of the natural parameter
gradient by examining the behaviour of its pushforward on the model. The
second looks at the behaviour on the parameter space itself. Since the
location and direction on the model is what matters, we argue that the former
is of greater importance.

##  2 The natural gradient

Let $(\mathcal{Z},g)$ be a Riemannian manifold, $\Xi$ be a parameter space
that we assume to be an open subset of $\mathbb{R}^{d}$,
$\phi\colon\Xi\to\mathcal{Z}$ a smooth map (taking the role of a
parametrisation222Within the context of differential geometry, a
parametrisation usually means a local diffeomorphism onto its image. This is
also referred to as a coordinate system. In our context, we use a more general
definition of parametrisation, where $\phi\colon\Xi\to\mathcal{Z}$ no longer
needs to be a diffeomorphism onto its image but only smooth.),
$\mathcal{M}\coloneqq\phi(\Xi)\subset\mathcal{Z}$ a model333Note that in the
literature this is also called a parametrised model, statistical manifold, or
in the context of machine learning, a neuromanifold. We choose the word model,
however, to emphasize that we do not assume a manifold structure on
$\mathcal{M}$., and $\mathcal{L}\colon\mathcal{Z}\to\mathbb{R}$ a smooth
(objective) function (see Figure 1). We call $p\in\mathcal{M}$ non-singular if
$\mathcal{M}$ is locally an embedded submanifold of $\mathcal{Z}$ around $p$
and we denote the set of non-singular points with
$\operatorname{Smooth}(\mathcal{M})$. A point $p$ is called singular if it is
not non-singular. The Riemannian gradient of $\mathcal{L}$ on $\mathcal{Z}$ is
defined implicitly as follows:

$g_{p}\left(\operatorname{grad}_{p}^{\mathcal{Z}}\mathcal{L},\cdot\right)=d\mathcal{L}_{p}(\cdot).$
(1)

By the Riesz representation theorem, this defines the gradient uniquely.

######  Definition 1 (Natural gradient).

For $p\in\operatorname{Smooth}(\mathcal{M})$ the Riemannian gradient of
$\mathcal{L}|_{\mathcal{M}}$ on the model $\mathcal{M}$ is called the _natural
gradient_ and is denoted $\operatorname{grad}^{\mathcal{M}}_{p}\mathcal{L}$.

It is easy to show that:

$\operatorname{grad}^{\mathcal{M}}_{p}\mathcal{L}=\Pi_{p}(\operatorname{grad}^{\mathcal{Z}}_{p}\mathcal{L}),$
(2)

where $\Pi_{p}$ is the projection onto $T_{p}\mathcal{M}$. We define the
pushforward of the tangent vector on the parameter space through the
parametrisation as $\partial_{i}(\xi)\coloneqq
d\phi_{\xi}\left(\left.\frac{\partial}{\partial\xi^{i}}\right|_{\xi}\right)$,
and the Gram matrix $G(\xi)$ by $G_{ij}(\xi)\coloneqq
g_{\phi(\xi)}\left(\partial_{i}(\xi),\partial_{j}(\xi)\right)$. We denote the
vector of coordinate derivatives with
$\nabla_{\xi}\mathcal{L}\coloneqq\left(\partial_{1}(\xi)\mathcal{L},...,\partial_{d}(\xi)\mathcal{L}\right)=\left(\frac{\partial\mathcal{L}\circ\phi}{\partial\xi^{1}}(\xi),...,\frac{\partial\mathcal{L}\circ\phi}{\partial\xi^{d}}(\xi)\right)\in\mathbb{R}^{d}$.
Let $\xi$ be such that $\phi(\xi)\in\operatorname{Smooth}(\mathcal{M})$. We
say that a parametrisation is proper in $\xi$ when:
$\operatorname{span}\left(\left\\{\partial_{1}(\xi),...,\partial_{d}(\xi)\right\\}\right)=T_{\phi(\xi)}\mathcal{M}$.
Furthermore, following the Einstein summation convention, we write
$a^{i}b_{i}$ for the sum $\sum_{i}a^{i}b_{i}$.

$\xi$$\mathbb{R}$$p$$\Xi$$\mathcal{Z}$$\mathcal{M}$$\frac{\partial{}}{\partial{\xi^{2}}}|_{\xi}$$\frac{\partial{}}{\partial{\xi^{1}}}|_{\xi}$$\partial_{1}(\xi)$$\partial_{2}(\xi)$$\phi$$\mathcal{L}$
Figure 1: Parametrisation and objective function

######  Definition 2 (Generalised inverse).

A _generalised inverse_ of an $n\times m$ matrix $A$, denoted $A^{+}$, is an
$m\times n$ matrix satisfying the following property:

$AA^{+}A=A.$ (3)

Note that this definition implies that for $w\in\mathbb{R}^{n}$ in the image
of A, i.e. $w=Av$ for some $v\in\mathbb{R}^{m}$, we have:

$AA^{+}w=AA^{+}Av=Av=w.$ (4)

This shows that $AA^{+}$ is the identity operator on the image of $A$.

######  Definition 3 (Natural parameter gradient).

We define the _natural parameter gradient_ to be the following vector on the
parameter space:

$\widetilde{\nabla}_{\xi}\mathcal{L}\coloneqq\left(G^{+}(\xi)\nabla_{\xi}\mathcal{L}\right)^{i}\left.\frac{\partial{}}{\partial{\xi^{i}}}\right|_{\xi}.$
(5)

The pushforward of this vector, given by:

$d\phi_{\xi}\widetilde{\nabla}_{\xi}\mathcal{L}=\left(G^{+}(\xi)\nabla_{\xi}\mathcal{L}\right)^{i}\partial_{i},$
(6)

is called the _natural parameter gradient on $\mathcal{M}$_.

Often, the natural parameter gradient is denoted with matrix notation as
follows:

$\widetilde{\nabla}_{\xi}\mathcal{L}\coloneqq
G^{+}(\xi)\nabla_{\xi}\mathcal{L},$ (7)

where an identification between the canonical basis of $\mathbb{R}^{d}$ and
the vectors $\left.\frac{\partial{}}{\partial{\xi^{i}}}\right|_{\xi}$ is made
implicitly.

We are now in the position to state the main result of the paper:

######  Theorem 1.

Let $\xi\in\Xi$ and $p=\phi(\xi)\in\mathcal{M}$. We have:

$d\phi_{\xi}\widetilde{\nabla}_{\xi}\mathcal{L}=\Pi_{\xi}\left(\operatorname{grad}_{p}^{\mathcal{Z}}\mathcal{L}\right),$
(8)

where $\Pi_{\xi}$ is the projection onto
$\operatorname{span}\\{\partial_{i}(\xi)\\}_{i}$. In particular, when
$\phi(\xi)$ is non-singular and
$\operatorname{span}\\{\partial_{i}(\xi)\\}_{i}=T_{p}\mathcal{M}$ we have:

$d\phi_{\xi}\widetilde{\nabla}_{\xi}\mathcal{L}=\operatorname{grad}^{\mathcal{M}}_{p}\mathcal{L}.$
(9)

This theorem implies that under certain conditions the pushforward of the
natural parameter gradient is equal to the natural gradient. Furthermore we
see that in general the natural parameter gradient on $\mathcal{M}$ is
dependent on the choice of parametrisation through $\Pi_{\xi}$, but becomes
invariant when the coordinate vectors span the full tangent space of
$\mathcal{M}$. In the next section we will study in more detail the invariance
properties of the natural parameter gradient.

The proof of Theorem 1 will be based on the following result from linear
algebra:

######  Lemma 1.

Let $(V,\langle\cdot,\cdot\rangle)$ be a finite-dimensional inner product
space and $V^{*}$ its dual space. Let $\\{e_{i}\\}_{i\in\\{1,...,d\\}}\subset
V$ (not necessarily linearly independent), $G$ the matrix defined by
$G_{ij}=\langle e_{i},e_{j}\rangle$, $\omega\in V^{*}$, $v$ such that $\langle
v,\cdot\rangle=\omega(\cdot)$ and $\Pi$ the projection on the space
$\operatorname{span}\\{e_{i}\\}_{i}$. Then,

$\Pi(v)=\left(G^{+}\right)^{ij}\omega(e_{j})e_{i}.$ (10)

###### Proof.

Start by noting $\Pi(v)$ is uniquely defined by the fact that
$\langle\Pi(v),w\rangle=\omega(w)$, for
$w\in\operatorname{span}\\{e_{i}\\}_{i}$ and $\langle\Pi(v),w\rangle=0$ for
$w\in\left(\operatorname{span}\\{e_{i}\\}_{i}\right)^{\perp}$. Since the RHS
of (10) lies in the span of $\\{e_{i}\\}_{i}$, it remains to show that for an
arbitrary vector $w=w^{i}e_{i}\in\operatorname{span}\\{e_{i}\\}_{i}$ we have:

$\langle\left(G^{+}\right)^{ij}\omega(e_{j})e_{i},w^{k}e_{k}\rangle=\omega(w).$
(11)

Working out the LHS gives:

$\displaystyle\langle\left(G^{+}\right)^{ij}\omega(e_{j})e_{i},w^{k}e_{k}\rangle$
$\displaystyle=G_{ik}\left(G^{+}\right)^{ij}\omega(e_{j})w^{k}$ (12)
$\displaystyle=w^{k}G_{ki}\left(G^{+}\right)^{ij}G_{jl}v^{l}$ (13)
$\displaystyle=w^{k}G_{kl}v^{l}$ (14) $\displaystyle=w^{k}\omega(e_{k})$ (15)
$\displaystyle=\omega(w),$ (16)

where we use the fact that: $\omega(e_{i})=G_{ij}v^{j}$ and the symmetry of
$G$ in the second equality.

∎

###### Proof of Theorem 1.

We now let $T_{p}\mathcal{M}$ take the role of $V$, $d\mathcal{L}_{p}$ the
role of $\omega$, $\partial_{i}(\xi)$ the role of $e_{i}$, and
$\operatorname{grad}_{p}\mathcal{L}$ the role of $v$. Equation (8) now follows
immediately. When the tangent vectors $\\{\partial_{i}(\xi)\\}_{i}$ span the
whole tangent space of $\mathcal{M}$ at $p$, $\Pi_{\xi}$ becomes the identity
on $T_{p}\mathcal{M}$. This gives Equation (9). ∎

##  3 Invariance properties of the natural parameter gradient

In this section we study the invariance properties of the natural parameter
gradient by using an alternative parametrisation of $\mathcal{M}$ given by:

$\psi\colon\Theta\ni\theta\mapsto\psi(\theta)\in\mathcal{Z}.$ (17)

Note that $G^{+}(\xi),\nabla_{\xi}\mathcal{L}$ and $\partial_{i}(\xi)$ in the
definition of $d\phi_{\xi}\widetilde{\nabla}_{\xi}\mathcal{L}$ all implicitly
depend on the parametrisation $\phi$. For an alternative parametrisation
$\psi$ we will therefore write: ${\partial}_{i}(\theta)\coloneqq
d\psi_{\theta}\left(\frac{\partial{}}{\partial{\theta}}|_{\theta}\right)$,
${G}_{ij}(\theta)\coloneqq
g_{\psi(\theta)}\left({\partial}_{i}(\theta),{\partial}_{j}(\theta)\right)$,
and
$\nabla_{\theta}\mathcal{L}\coloneqq({\partial}_{1}(\theta)\mathcal{L},...,{\partial}_{d}(\theta)\mathcal{L})$.

The invariance properties can be studied from the perspective of the model and
from the perspective of the parameter space itself. Since the former is of
more importance, we will start with this one.

$\theta$$\xi$$p$$\Theta$$\Xi$$\mathcal{M}$$f$$\psi$$\phi$$\frac{\partial{}}{\partial{\theta^{2}}}|_{\theta}$$\frac{\partial{}}{\partial{\theta^{1}}}|_{\theta}$$\frac{\partial{}}{\partial{\xi^{2}}}|_{\xi}$$\frac{\partial{}}{\partial{\xi^{1}}}|_{\xi}$$\partial_{i}(\xi)$${\partial}_{i}(\theta)$
Figure 2: Two parametrisations of $\mathcal{M}$

###  3.1 Parametrisation dependence and reparametrisation invariance on the
model

A parametrisation can be used to represent tangent vectors on the model space
by elements of $\mathbb{R}^{d}$. A representation (of vectors on
$\mathcal{M}$) can be interpreted as the map
$\mathcal{O}\colon(\phi,\xi)\mapsto\mathcal{O}(\phi,\xi)\in T_{\xi}\Xi\
(\cong\mathbb{R}^{d})$ that takes a parametrisation-coordinate pair and
assigns a tangent vector on the parameter space to it. The natural parameter
gradient defined by
$\widetilde{\nabla}_{\xi}\mathcal{L}=\left(G^{+}(\xi)\nabla_{\xi}\mathcal{L}\right)^{i}\left.\frac{\partial{}}{\partial{\xi^{i}}}\right|_{\xi}$
in Equation (5) is an example of a representation, where the dependence on
$\phi$ on the RHS is implicit. Naively, one could define invariance of a
representation in the following way:

######  Definition 4 (Parametrisation independence).

Let $\mathcal{M}$ be a model. A representation $\mathcal{O}(\cdot,\cdot)$ is
called _parametrisation independent_ if for any pair $\phi,\psi$ of
parametrisations of $\mathcal{M}$, and coordinates $\xi,\theta$ such that
$\psi(\theta)=\phi(\xi)$, the following holds:

$d\psi_{\theta}\mathcal{O}(\psi,\theta)=d\phi_{\xi}\mathcal{O}(\phi,\xi).$
(18)

It turns out that this is not a very useful definition. As we will see, no
non-trivial representation can be parametrisation independent in the sense of
this definition. We will illustrate this in Example 1 and 2 below for the
natural parameter gradient on specific models. A formal proof can be found in
the Appendix A.1.
In order to overcome the limitation of Definition 4, we propose the following
more suitable definition of invariance of a representation:

######  Definition 5 (Reparametrisation invariance).

Let $\mathcal{M}$ be a model. A representation $\mathcal{O}(\cdot,\cdot)$ is
called _reparametrisation invariant_ if for any pair $\phi,\psi$ of
parametrisations of $\mathcal{M}$, such that $\psi=\phi\circ f$ for a
diffeomorphism $f\colon\Theta\to\Xi$, and coordinates $\xi,\theta$ such that
$\theta=f^{-1}(\xi)$, the equality (18) holds.

Due to the extra requirement of the existence of the reparametrisation
function $f$ in Definition 5, we get the following central result of this
paper:

######  Theorem 2.

The natural parameter gradient is reparametrisation invariant.

###### Proof.

By Definition 5, we need to show that for $\psi=\phi\circ f$ and
$\theta=f^{-1}(\xi)$ we have:

$d\psi_{\theta}\widetilde{\nabla}_{\theta}\mathcal{L}=d\phi_{\xi}\widetilde{\nabla}_{\xi}\mathcal{L}.$
(19)

Since the differential $df_{\theta}$ is surjective, we have that
$\operatorname{span}\\{\partial_{i}(\xi)\\}_{i}=\operatorname{span}\\{{\partial}_{j}(\theta)\\}_{j}$.
Therefore, by using equation Equation (8) of Theorem 1, we get:

$d\psi_{\theta}\widetilde{\nabla}_{\theta}\mathcal{L}=\Pi_{\theta}\left(\operatorname{grad}_{\psi(\theta)}^{\mathcal{Z}}\mathcal{L}\right)=\Pi_{\xi}\left(\operatorname{grad}_{\phi(\xi)}^{\mathcal{Z}}\mathcal{L}\right)=d\phi_{\xi}\widetilde{\nabla}_{\xi}\mathcal{L},$
(20)

which is what we wanted to show. ∎

######  Remark 1.

Note that under the extra assumptions that $\mathcal{M}$ is a smooth manifold
and all parametrisations are required to be diffeomorphisms Definitions 4 and
5 are equivalent, since the composition $f=\psi\circ\phi^{-1}$ is a
diffeomorphism. These assumptions are often implicitly made when referring to
the invariance of the natural gradient. However, as we will see below, this is
no longer the case in our more general setting.

#### Example 1

This example will be of a graphical nature. Consider the parametrisation
$\phi$ that is the composition of the 2 maps in Figure 3. Now let
$\psi\colon\Xi\to\mathcal{M}$ be this parametrisation but with a 90 degree
rotation around $\phi(\xi)$ applied before projecting down to $\mathcal{M}$.

$\Xi$$\xi$$\mathcal{M}$$\phi(\xi)$ Figure 3: Parametrisation with a non-
surjective span of the parameter vectors

Note that the spans of the parameter vectors have trivial intersection as
depicted in Figure 4(a). This immediately implies that for any representation
we have: $d\psi_{\theta}\mathcal{O}(\psi,\theta)\neq
d\phi_{\xi}\mathcal{O}(\phi,\xi)$ except when both sides are equal to zero. In
particular, we can let the natural gradient be as in Figure 4(b). We know from
Theorem 1 that the natural parameter gradients on $\mathcal{M}$ will be the
projections of the natural gradient onto the respective spans of the parameter
vectors as depicted in the figure. Note that the projection should be
orthogonal with respect to the inner product $g_{p}$, which we have chosen
here to be Euclidean for ease of illustration.

$\mathcal{M}$$\phi(\xi)$$\operatorname{span}\\{d\phi_{\xi}(\frac{\partial{}}{\partial{\xi^{i}}}|_{\xi})\\}_{i}$$\operatorname{span}\\{d\psi_{\xi}(\frac{\partial{}}{\partial{\xi^{j}}}|_{\xi})\\}_{j}$
(a) Spans of both sets of parameter vectors

$\mathcal{M}$$\phi(\xi)$$\operatorname{grad}^{\mathcal{M}}_{\phi(\xi)}\mathcal{L}$$d\phi_{\xi}\widetilde{\nabla}_{\xi}\mathcal{L}$$d\psi_{{\xi}}\widetilde{\nabla}_{{\xi}}\mathcal{L}$
(b) Projected gradient vectors

Figure 4: Spans and gradient vectors on the model of both parametrisations

This example shows that for non-singular points, we can construct two
parametrisations that give different natural parameter gradients on the same
point of the model. Note that this is not in violation of Theorem 2 since
there does not exist a diffeomorphism $f$ such that $\psi=\phi\circ f$.

#### Example 2

$\Xi$$\mathbb{R}^{2}$$\mathcal{M}$$\phi$$p$$\xi_{1}$$\xi_{2}$ Figure 5:
Parametrisation that contains a singular point

Let us consider the case in which $\phi$ is a smooth map from an interval on
the real line to $\mathbb{R}^{2}$ as depicted in Figure 5. We have that
$\xi_{1}$ and $\xi_{2}$ are both mapped to the same point $p$ in
$\mathbb{R}^{2}$. Note that $\mathcal{M}$ is in this case not a locally
embedded submanifold around $p$ and thus $p$ is a singular point. Note that
$G{(\xi_{1}})$ is a real number different from zero and therefore non-
degenerate. Calculating the natural parameter gradient on $\mathcal{M}$ for
$\xi=\xi_{1}$ gives:

$\displaystyle d\phi_{\xi_{1}}\widetilde{\nabla}_{\xi_{1}}\mathcal{L}$
$\displaystyle=G^{+}(\xi_{1})\nabla_{\xi}\mathcal{L}\ \partial(\xi_{1})$ (21)
$\displaystyle=G^{-1}(\xi_{1})\frac{\partial\mathcal{L}\circ\phi}{\partial\xi}(\xi_{1})\partial(\xi_{1}).$
(22)

Since
$G^{-1}(\xi_{1})\frac{\partial\mathcal{L}\circ\phi}{\partial\xi}(\xi_{1})$ is
a scalar, the resulting vector will lie in the span of $\partial(\xi_{1})$
illustrated by the blue arrows in the figure.

Now let $f\colon\Theta\to\Xi$ be a diffeomorphism such that
$f(\theta_{1})=\xi_{2}$. An alternative parametrisation of $\mathcal{M}$ is
given by:

$\psi=\phi\circ f.$ (23)

Calculating the natural parameter gradient at $\theta_{1}$ for this
parametrisation gives:

$d\psi_{{\theta_{1}}}\widetilde{\nabla}_{{\theta_{1}}}\mathcal{L}={G}^{-1}({\theta_{1}})\frac{\partial\mathcal{L}\circ\psi}{\partial\theta}(\theta_{1})\partial(\theta_{1}).$
(24)

Note that this vector is in the span of $\partial(\theta_{1})$ denoted by the
red arrows in the figure and therefore in general different from (22). This
shows that when
$\operatorname{span}\\{\partial_{i}(\xi)\\}_{i}\neq\operatorname{span}\\{{\partial}_{j}(\theta)\\}_{j}$
the outcome of
$\left(G^{+}(\xi)\nabla_{\xi}\mathcal{L}\right)^{i}\partial_{i}(\xi)$ can be
dependent on the choice of parametrisation and therefore the natural parameter
gradient is not parametrisation independent. Note however that this result is
not in contradiction with Theorem 2 since we do not have
$\theta_{1}=f^{-1}(\xi_{1})$. See Appendix A.2 for a worked-out example of
this.

###  3.2 Reparametrisation (in)variance on the parameter space

In the previous section we have looked at the invariance properties of the
natural parameter gradient from the perspective of the model. One can also
study the invariance properties from the perspective of the parameter space as
is done for example in Section 12 of [12]. Translating the definition of
invariance given there to our notation gives the following:

######  Definition 6 (Reparametrisation invariance on the parameter space).

A representation $\mathcal{O}(\cdot,\cdot)$ is called _reparametrisation
invariant on the parameter space_ if for any pair of parametrisations
$\phi,\psi$ such that $\psi=\phi\circ f$ for a diffeomorphism
$f\colon\Theta\to\Xi$, and coordinates $\xi,\theta$ such that
$\theta=f^{-1}(\xi)$, we have:

$df_{\theta}\mathcal{O}(\psi,\theta)=\mathcal{O}(\phi,\xi).$ (25)

Note that reparametrisation invariance on the parameter space implies
reparametrisation invariance on the model as defined in Definition 5.
Furthermore, it can be shown that when $\mathcal{M}$ is a smooth manifold and
all parametrisations are required to be diffeomorphisms, like in Remark 1,
this definition is equivalent to Definitions 4 and 5. In that case, the
natural parameter gradient satisfies Equation (25). As we will see below, this
is not true for general $\phi$. We would like to argue, however, that this is
not a suitable definition of invariance, since multiple vectors on the
parameter space can be mapped to the same vector on the model. Therefore
inequality on the parameter space does not have to imply inequality on the
model.

$\theta$$\xi$$p$$\Theta$$\Xi$$\mathcal{M}$$f$$\psi$$\phi$$\widetilde{\nabla}_{\xi}\mathcal{L}$$\widetilde{\nabla}_{\theta}\mathcal{L}$$df_{\theta}\widetilde{\nabla}_{\theta}\mathcal{L}$
$d\phi_{\xi}\widetilde{\nabla}_{\xi}\mathcal{L}=$
---
$d\phi_{\xi}df_{\theta}\widetilde{\nabla}_{\theta}\mathcal{L}$

Figure 6: Two parametrisations of $\mathcal{M}$ with different gradient
vectors on the parameter space

We will now make the above explicit. Let us choose the MP inverse as
generalised inverse for $\widetilde{\nabla}_{\xi}\mathcal{L}$ and consider an
alternative parametrisation $\psi=\phi\circ f$ for a diffeomorphism
$f\colon\Theta\to\Xi$ (see Figure 6). We denote the matrix of partial
derivatives of $f$ at ${\theta}$ with
$F_{i}^{j}({\theta})=\frac{\partial{f^{j}}}{\partial{{\theta}^{i}}}({\theta})$.
For $\xi=f({\theta})$ we get the following relations:

$\displaystyle{\partial}_{i}({\theta})$
$\displaystyle=F_{i}^{j}({\theta})\,\partial_{j}(\xi)$ (26)
$\displaystyle\nabla_{\theta}\mathcal{L}$
$\displaystyle=F({\theta})\,\nabla_{\xi}\mathcal{L}$ (27)
$\displaystyle{G}({\theta})$
$\displaystyle=F({\theta})\,G(\xi)\,F^{T}({\theta}).$ (28)

We map $\widetilde{\nabla}_{\theta}\mathcal{L}$ to $T_{\xi}\Xi$ through
$df_{\theta}$ and get:

$\displaystyle df_{\theta}\widetilde{\nabla}_{\theta}\mathcal{L}$
$\displaystyle=df_{\theta}\left(\left({G}^{+}({\theta})\nabla_{\theta}\mathcal{L}\right)^{i}\left.\frac{\partial}{\partial{\theta}^{i}}\right|_{\theta}\right)$
(29)
$\displaystyle=\left(\left(F({\theta})G(\xi)F^{T}({\theta})\right)^{+}F({\theta})\,\nabla_{\xi}\mathcal{L}\right)^{i}F_{i}^{j}({\theta})\left.\frac{\partial{}}{\partial{\xi^{j}}}\right|_{\xi}$
(30)
$\displaystyle=\left(F^{T}({\theta})\left(F({\theta})G(\xi)F^{T}({\theta})\right)^{+}F({\theta})\,\nabla_{\xi}\mathcal{L}\right)^{j}\left.\frac{\partial{}}{\partial{\xi^{j}}}\right|_{\xi}.$
(31)

We will write $y_{\Xi},y_{\Theta}$ for the coefficients of
$\widetilde{\nabla}_{\xi}\mathcal{L}$ and
$df_{\theta}\widetilde{\nabla}_{\theta}\mathcal{L}$ respectively. From Theorem
1 and the fact that $F(\theta)$ is of full rank we know that
$F(\theta)\,\nabla_{\xi}\mathcal{L}$ lies in the image of
$F(\theta)G(\xi)F^{T}(\theta)$. Therefore, by the definition of the MP
inverse, we have that
$\left(F(\theta)G(\xi)F^{T}(\theta)\right)^{+}F(\theta)\,\nabla_{\xi}\mathcal{L}=\operatorname*{arg\,min}_{x}\\{||x||:F(\theta)G(\xi)F^{T}(\theta)x=F(\theta)\,\nabla_{\xi}\mathcal{L}\\}$,
where $||\cdot||$ is the Euclidean norm on $\mathbb{R}^{d}$. The coefficients
in (31) become:

$\displaystyle y_{\Theta}$
$\displaystyle=F^{T}({\theta})\left(F({\theta})G(\xi)F^{T}({\theta})\right)^{+}F({\theta})\,\nabla_{\xi}\mathcal{L}$
(32)
$\displaystyle=\operatorname*{arg\,min}_{y}\left\\{||\left(F^{T}({\theta})\right)^{-1}y||:G(\xi)y=\nabla_{\xi}\mathcal{L}\right\\},$
(33)

where we substitute $y=F^{T}(\theta)x$ in the last line.

######  Remark 2.

Note that $||\left(F^{T}\right)^{-1}(\cdot)||$ is the pushforward of the norm
on $\Theta$ through $f$. This shows nicely the equivalence of the gradient for
on the one hand constructing a different parametrisation $(\psi)$, and on the
other hand defining a different inner product
$\left(||\left(F^{T}\right)^{-1}(\cdot)||\right)$ for the existing
parametrisation $(\phi)$.

Comparing the result to the natural parameter gradient on $\Xi$ gives:

$\displaystyle\widetilde{\nabla}_{\xi}\mathcal{L}$
$\displaystyle=\left(G^{+}(\xi)\nabla_{\xi}\mathcal{L}\right)^{i}\left.\frac{\partial{}}{\partial{\xi^{i}}}\right|_{\xi}$
(34)
$\displaystyle=(y_{\Xi})^{i}\left.\frac{\partial{}}{\partial{\xi^{i}}}\right|_{\xi}$
(35) $\displaystyle y_{\Xi}$
$\displaystyle=\operatorname*{arg\,min}_{y}\\{||y||:G(\xi)y=\nabla_{\xi}\mathcal{L}\\}.$
(36)

Because the norms in (33) and (36) are different, generally $y_{\Theta}\neq
y_{\Xi}$. However, both satisfy $G(\xi)y=\nabla_{\xi}\mathcal{L}$ and
therefore $G(\xi)(y_{\Theta}-y_{\Xi})=0$. This implies:

$\displaystyle
d\phi_{\xi}\left(df_{\theta}\widetilde{\nabla}_{\theta}\mathcal{L}-\widetilde{\nabla}_{\xi}\mathcal{L}\right)$
$\displaystyle=\left(y_{\Theta}-y_{\Xi}\right)^{i}\partial_{i}(\xi)$ (37)
$\displaystyle=0,$ (38)

where the last equality can be verified by taking the norm on the RHS of (37)
using that it is non-degenerate, like so:

$||\left(y_{\Theta}-y_{\Xi}\right)^{i}\partial_{i}(\xi)||^{2}_{g}=\left(y_{\Theta}-y_{\Xi}\right)^{T}G(\xi)\left(y_{\Theta}-y_{\Xi}\right)=0.$
(39)

This shows that for overparametrised systems the natural parameter gradient is
not reparametrisation invariant on the parameter space. However, as implied by
Theorem 1, the dependency on the parametrisation disappears when the gradient
is mapped to the model. See Appendix A.3 for a worked-out example of the above
discussion.

##  4 Practical considerations for the natural gradient method

The natural gradient method is performed by updating the current parameter
vector $\xi\in\Xi$ in the direction of the vector
$G^{+}(\xi)\nabla_{\xi}\mathcal{L}\in\mathbb{R}^{d}$. In case of a constrained
parameter space, i.e. $\Xi$ is not the full space $\mathbb{R}^{d}$, such as
the space of covariance matrices, one runs the risk of stepping outside the
parameter space, see also [2]. This is called a constraint violation. One can
use backprojection [8], addition of a penalty, and weight clipping [7] to
avoid these violations. Note that for a variety of neural network
applications, including many supervised learning tasks, the parameter space is
unconstrained. For these models however, the generalised inverse is often hard
to compute due to the high number of parameters. In this context, the Woodbury
matrix identity with damping is often used instead [15]. Investigating these
topics further falls outside the scope of this paper.

#### Reparametrisation (in)variance of the natural gradient method trajectory

We have shown in Theorem 2 that from the perspective of the model, the natural
parameter gradient is reparametrisation invariant. That is, for two
parametrisations $\phi,\psi$ for which $\psi=\phi\circ f$ for a diffeomorphism
$f$ and $\theta=f^{-1}(\xi)$, we have that,

$d\phi_{\xi}\widetilde{\nabla}_{\xi}\mathcal{L}=\left(G^{+}(\xi)\nabla_{\xi}\mathcal{L}\right)^{i}\partial_{i}=\left({G}^{+}(\theta)\nabla_{\theta}\mathcal{L}\right)^{j}{\partial}_{j}=d\psi_{\theta}\widetilde{\nabla}_{\theta}\mathcal{L}.$
(40)

The natural gradient method is performed by updating the current parameter
vector $\xi\in\mathbb{R}^{d}$ in the direction of the vector
$G^{+}(\xi)\nabla_{\xi}\mathcal{L}\in\mathbb{R}^{d}$. Equation (40) implies
that if we would update the parameters for both parametrisations an
infinitesimal amount, this would give us the same result on the model. We
would like to emphasise however that updating the parameters by a finite
amount will in general result in different locations on the model. Therefore
the natural gradient method trajectory is dependent on the choice of
parametrisation. This is however not an issue specific to overparametrised
models but with the natural gradient method in general. See Section 12 of [12]
for exact bounds on the invariance.

#### Occurrence of non-proper points

We saw in the proof of Theorem 2 that when
$\operatorname{span}\\{\partial_{i}(\xi)\\}_{i}=\operatorname{span}\\{{\partial}_{j}(\theta)\\}_{j}$
for two parametrisations $\phi$ and $\psi$ with $\phi(\xi)=\psi(\theta)$, we
have
$d\phi_{\xi}\widetilde{\nabla}_{\xi}\mathcal{L}=d\psi_{\theta}\widetilde{\nabla}_{\theta}\mathcal{L}$.
For $\phi(\xi)\in\operatorname{Smooth}(\mathcal{M})$ note that this equality
holds in particular when
$\operatorname{span}\\{\partial_{i}(\xi)\\}_{i}=\operatorname{span}\\{{\partial}_{j}(\theta)\\}_{j}=T_{p}\mathcal{M}$,
i.e. $\phi$ is proper in $\xi$. Therefore we will now study when this is the
case. We start by recalling some basic facts from smooth manifold theory: Let
$M,N$ be smooth manifolds and $F:M\to N$ a smooth map. We call a point $p\in
M$ a regular point if $dF_{p}:T_{p}M\to T_{F(p)}N$ is surjective and a
critical point otherwise. A point $q\in N$ is called a regular value if all
the elements in $F^{-1}(q)$ are regular points, and a critical value
otherwise. If $M$ is $n$-dimensional, we say that a subset $S\subset M$ has
measure zero in $M$, if for every smooth chart $(U,\psi)$ for $M$, the subset
$\psi(S\cap U)\subset\mathbb{R}^{n}$ has $n$-dimensional measure zero. That
is: $\forall\delta>0$, there exists a countable cover of $\psi(S\cap U)$
consisting of open rectangles, the sum of whose volumes is less than $\delta$.
We have the following result based on Sard’s theorem:

######  Proposition 1.

If $\operatorname{Smooth}(\mathcal{M})$ is a manifold, then the image of the
set of points for which $\phi$ is not proper has measure zero in
$\operatorname{Smooth}(\mathcal{M})$.

###### Proof.

From the definition of $\operatorname{Smooth}(\mathcal{M})$ we know that for
every $p\in\operatorname{Smooth}(\mathcal{M})$ there exists a $U_{p}$ open in
$\mathcal{Z}$ such that $U_{p}\cap\mathcal{M}$ is an embedded submanifold of
$\mathcal{Z}$. Let
$U\coloneqq\bigcup_{p\in\operatorname{Smooth}(\mathcal{M})}U_{p}$. Note that:
$U\cap\mathcal{M}=\operatorname{Smooth}(\mathcal{M})$ and therefore
$\phi^{-1}(U)=\phi^{-1}(\operatorname{Smooth}(\mathcal{M}))$. Since $U$ is
open in $\mathcal{Z}$, $\phi^{-1}(\operatorname{Smooth}(\mathcal{M}))$ is an
open subset of $\Xi$ and thus an embedded submanifold. Therefore we can
consider the map:

$\phi|_{\phi^{-1}(\operatorname{Smooth}(\mathcal{M}))}:\phi^{-1}(\operatorname{Smooth}(\mathcal{M}))\to\operatorname{Smooth}(\mathcal{M})$
(41)

and note that the image of the set of points for which $\phi$ is not proper is
equal to the set of critical values of
$\phi|_{\phi^{-1}(\operatorname{Smooth}(\mathcal{M}))}$ in
$\operatorname{Smooth}(\mathcal{M})$. A simple application of Sard’s theorem
gives the result. ∎

This proposition implies that when $\operatorname{Smooth}(\mathcal{M})$ is a
manifold, the set of points for which the pushforward of the natural parameter
gradient is unequal to the natural gradient has measure zero in
$\operatorname{Smooth}(\mathcal{M})$.

##  5 Conclusion

In this paper we have studied the natural parameter gradient, which was
defined as the update direction of the natural gradient method, and its
pushforward to the model in an overparametrised setting. We have seen that the
latter is equal to the natural gradient under certain conditions. Furthermore
we have proposed different notions of invariance and studied whether the
natural parameter gradient satisfies these. From the perspective of the model,
we have seen that the natural parameter gradient is reparametrisation
invariant but that it is not parametrisation independent. Additionally, we saw
that the natural parameter gradient is not reparametrisation invariant on the
parameter space. We have argued, however, that this notion is less suitable in
an overparametrised setting since multiple vectors on the parameter space can
correspond to the same vector on the model. Finally we have given some
practical considerations for the natural gradient method.

## Acknowledgements

JvO and NA acknowledge the support of the Deutsche Forschungsgemeinschaft
Priority Programme “The Active Self” (SPP 2134). JM acknowledges support by
the ERC under the European Union’s Horizon 2020 research and innovation
programme (grant agreement no 757983), by the International Max Planck
Research School for Mathematics in the Sciences and the Evangelisches
Studienwerk Villigst e.V..

## References

  * [1] Shun-Ichi Amari  “Natural gradient works efficiently in learning”  In _Neural computation_ 10.2 MIT Press, 1998, pp. 251–276
  * [2] Martin Arjovsky, Soumith Chintala and Léon Bottou  “Wasserstein generative adversarial networks”  In _International conference on machine learning_ , 2017, pp. 214–223  PMLR
  * [3] Nihat Ay  “On the locality of the natural gradient for learning in deep Bayesian networks”  In _Information Geometry_ Springer, 2020, pp. 1–49
  * [4] Nihat Ay, Guido Montúfar and Johannes Rauh  “Selection criteria for neuromanifolds of stochastic dynamics”  In _Advances in Cognitive Neurodynamics_ Springer, 2013, pp. 147–154
  * [5] Alberto Bernacchia, Mate Lengyel and Guillaume Hennequin  “Exact natural gradient in deep linear networks and its application to the nonlinear case”  In _Advances in Neural Information Processing Systems_ 31 Curran Associates, Inc., 2018
  * [6] Roger Grosse and James Martens  “A Kronecker-factored approximate Fisher matrix for convolution layers”  In _International Conference on Machine Learning_ , 2016, pp. 573–582  PMLR
  * [7] Ishaan Gulrajani et al.  “Improved training of Wasserstein GANs”  In _Advances in neural information processing systems_ 30, 2017
  * [8] Harold J. Kushner and Dean S. Clark  “Stochastic Approximation Methods for Constrained and Unconstrained Systems”  Springer New York, 1978
  * [9] Wuchen Li and Guido Montúfar  “Natural gradient via optimal transport”  In _Information Geometry_ 1.2 Springer, 2018, pp. 181–214
  * [10] Wu Lin, Frank Nielsen, Mohammad Emtiyaz Khan and Mark Schmidt  “Tractable structured natural gradient descent using local parameterizations”  In _International Conference on Machine Learning_ , PMLR, 2021, pp. 6680–6691
  * [11] Luigi Malagò, Luigi Montrucchio and Giovanni Pistone  “Wasserstein Riemannian geometry of Gaussian densities”  In _Information Geometry_ 1.2 Springer, 2018, pp. 137–179
  * [12] James Martens  “New Insights and Perspectives on the Natural Gradient Method”  In _Journal of Machine Learning Research_ 21, 2020, pp. 1–76
  * [13] Yann Ollivier  “Riemannian metrics for neural networks I: feedforward networks”  In _Information and Inference: A Journal of the IMA_ 4.2 Oxford University Press, 2015, pp. 108–153
  * [14] Jesse van Oostrum and Nihat Ay  “Parametrisation Independence of the Natural Gradient in Overparametrised Systems”  In _International Conference on Geometric Science of Information_ , 2021, pp. 726–735  Springer
  * [15] Sidak Pal Singh and Dan Alistarh  “WoodFisher: Efficient Second-Order Approximation for Neural Network Compression”  arXiv, 2020  DOI: 10.48550/ARXIV.2004.14340
  * [16] Hado Van Hasselt  “Reinforcement learning in continuous state and action spaces”  In _Reinforcement learning_ Springer, 2012, pp. 207–251
  * [17] Csongor Várady, Riccardo Volpi, Luigi Malagò and Nihat Ay  “Natural wake-sleep algorithm”  In _Deep Learning through Information Geometry Worksop_ , 2020
  * [18] Guodong Zhang, James Martens and Roger Grosse  “Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networks”  arXiv, 2019  DOI: 10.48550/ARXIV.1905.10961

##  Appendix A Appendix

###  A.1 On the limitation of Definition 4

######  Proposition 2.

No non-trivial representation of a vector can be parametrisation independent
in the sense of Definition 4. More precisely, any representation satisfying
Definition 4 has the property that for all parametrisation-coordinate pairs
$\phi,\xi$ the following holds:

$d\phi_{\xi}\mathcal{O}(\phi,\xi)=0.$ (42)

###### Proof.

Let $\mathcal{M}$ be a model and assume that $\mathcal{O}$ is a representation
satisfying Definition 4. Let $\phi$ be a parametrisation of $\mathcal{M}$ and
$\xi^{*}\in\Xi$ be a fixed (arbitrary) element on the domain of $\phi$. Now
consider the following function:

$\displaystyle f\colon\mathbb{R}^{d}$ $\displaystyle\to\mathbb{R}^{d}$ (43)
$\displaystyle\theta=\left(\theta^{1},..,\theta^{d}\right)$
$\displaystyle\mapsto\left(\left(\theta^{1}\right)^{3},...,\left(\theta^{d}\right)^{3}\right)+\xi^{*}.$
(44)

We define $\Theta=f^{-1}(\Xi)$ and $\psi=\phi\circ f|_{\Theta}$. First note
that, since $f$ is continuous, $\Theta$ is an open set. Secondly, since $f$ is
surjective, we have $\psi(\Theta)=\mathcal{M}$. Therefore $\psi$ is a
parametrisation of $\mathcal{M}$. It is easy to see that the differential of
$f$ at $\theta=0$, $df_{0}$, is equal to zero and therefore by the chain rule
we have $d\psi_{0}=d(\phi\circ f|_{\Theta})_{0}=d\phi_{f(0)}\circ df_{0}=0$.
Furthermore we have: $\psi(0)=\phi(\xi^{*})$. Therefore in order for
$\mathcal{O}$ to satisfy equation (18) we need that:

$d\phi_{\xi}^{*}\mathcal{O}(\phi,\xi^{*})=d\psi_{0}\mathcal{O}(\psi,0)=0.$
(45)

Since $\xi^{*}$ was chosen arbitrarily, this implies that $\mathcal{O}$ is a
trivial representation. ∎

######  Remark 3.

The function $f$ used in the proof above is actually a homeomorphism since it
is a continuous bijection and its inverse, given by
$f^{-1}(\xi)=\left(\sqrt[3]{\xi^{1}-\left(\xi^{*}\right)^{1}},...,\sqrt[3]{\xi^{d}-\left(\xi^{*}\right)^{d}}\right)$,
is also continuous. The inverse is however not differentiable and therefore
$f$ is not a diffeomorphism, which is required for Definition 5.

###  A.2 Example calculation of parametrisation dependence on the model

We illustrate Example 2 in Section 3.1 with a specific calculation. Let us
consider the following parametrisation:

$\displaystyle\phi\colon(-1/8\pi,5/8\pi)$
$\displaystyle\to(\mathbb{R}^{2},\bar{g})$ (46) $\displaystyle\xi$
$\displaystyle\mapsto(x,y)=(\sin(2\xi),\sin(\xi)).$ (47)

This gives $\xi_{1}=0,\xi_{2}=\frac{1}{2}\pi$ in the above discussion. We get
the following calculation for $d\phi_{0}\widetilde{\nabla}_{0}\mathcal{L}$:

$\displaystyle\partial(\xi)$
$\displaystyle=d\phi_{\xi}\left(\left.\frac{\partial{}}{\partial{\xi}}\right|_{\xi}\right)$
(48)
$\displaystyle=2\cos(2\xi)\left.\frac{\partial{}}{\partial{x}}\right|_{\phi(\xi)}+\cos(\xi)\left.\frac{\partial{}}{\partial{y}}\right|_{\phi(\xi)}$
(49) $\displaystyle\partial(\xi_{1})$
$\displaystyle=2\left.\frac{\partial{}}{\partial{x}}\right|_{(0,0)}+\left.\frac{\partial{}}{\partial{y}}\right|_{(0,0)}$
(50) $\displaystyle G(\xi_{1})$ $\displaystyle=2^{2}+1^{1}=5$ (51)
$\displaystyle d\phi_{\xi_{1}}\widetilde{\nabla}_{\xi_{1}}\mathcal{L}$
$\displaystyle=\frac{1}{5}\left.\frac{\partial{}}{\partial{\xi}}\right|_{\xi=0}\bigg{(}\mathcal{L}(\sin(2\xi),\sin(\xi))\bigg{)}\left(2\left.\frac{\partial{}}{\partial{x}}\right|_{(0,0)}+\left.\frac{\partial{}}{\partial{y}}\right|_{(0,0)}\right).$
(52)

Now let:

$\displaystyle f\colon(-1/8\pi,5/8\pi)$ $\displaystyle\to(-1/8\pi,5/8\pi)$
(53) $\displaystyle\theta$ $\displaystyle\mapsto-(\theta-\frac{1}{4}\pi).$
(54)

This implies that $\theta_{1}=f^{-1}(\xi_{2})=0$. We define the alternative
parametrisation $\psi=\phi\circ f$. Note that we have
$\psi(\theta)=(-\sin(2\theta),\sin(\theta))$ and thus
$\psi(\theta_{1})=\phi(\xi_{1})=(0,0)$. A similar calculation as before gives:

$\displaystyle{\partial}(\theta_{1})$
$\displaystyle=-2\left.\frac{\partial{}}{\partial{x}}\right|_{(0,0)}+\left.\frac{\partial{}}{\partial{y}}\right|_{(0,0)}$
(55) $\displaystyle
d\psi_{{\theta_{1}}}\widetilde{\nabla}_{{\theta_{1}}}\mathcal{L}$
$\displaystyle=\frac{1}{5}\left.\frac{\partial{}}{\partial{\theta}}\right|_{\theta=0}\bigg{(}\mathcal{L}(-\sin(2\theta),\sin(\theta))\bigg{)}\left(-2\left.\frac{\partial{}}{\partial{x}}\right|_{(0,0)}+\left.\frac{\partial{}}{\partial{y}}\right|_{(0,0)}\right).$
(56)

Note that because $\partial(\xi_{1})\neq{\partial}(\theta_{1})$, (52) and (56)
are not equal to each other. We can therefore conclude that the natural
parameter gradient is not parametrisation independent.

###  A.3 Example calculation of reparametrisation (in)variance on the
parameter space

We illustrate the discussion in Section 3.2 with a specific calculation. Let
us consider the following setting:

$\displaystyle\mathcal{Z}$ $\displaystyle=\Xi=\Theta=\mathbb{R}^{2}$ (57)
$\displaystyle\phi(\xi_{1},\xi_{2})$ $\displaystyle=(\xi_{1}+\xi_{2},0)$ (58)
$\displaystyle f({\theta}_{1},{\theta}_{2})$
$\displaystyle=(2{\theta}_{1},{\theta}_{2})$ (59)
$\displaystyle\mathcal{L}(x,y)$ $\displaystyle=x^{2}$ (60)

Plugging this into the expressions derived above gives:

$\displaystyle\psi({\theta}_{1},{\theta}_{2})$
$\displaystyle=(2{\theta}_{1}+{\theta}_{2},0)$ (61)
$\displaystyle\partial_{1}(\xi)$
$\displaystyle=\partial_{2}(\xi)=\left.\frac{\partial{}}{\partial{x}}\right|_{\phi(\xi)}$
$\displaystyle F({\theta})=\begin{bmatrix}2&0\\\ 0&1\end{bmatrix}$ (62)
$\displaystyle{\partial}_{1}({\theta})$
$\displaystyle=2\left.\frac{\partial{}}{\partial{x}}\right|_{\psi({\theta})}$
$\displaystyle G(\xi)=\begin{bmatrix}1&1\\\ 1&1\end{bmatrix}$ (63)
$\displaystyle{\partial}_{2}({\theta})$
$\displaystyle=\left.\frac{\partial{}}{\partial{x}}\right|_{\psi({\theta})}$
$\displaystyle{G}({\theta})=\begin{bmatrix}4&2\\\ 2&1\end{bmatrix}$ (64)
$\displaystyle\nabla_{\xi}\mathcal{L}$
$\displaystyle=(2(\xi_{1}+\xi_{2}),2(\xi_{1}+\xi_{2}))$ (65)
$\displaystyle\nabla_{\theta}\mathcal{L}$
$\displaystyle=(4(2{\theta}_{1}+{\theta}_{2}),2(2{\theta}_{1}+{\theta}_{2})).$
(66)

Now we fix ${\theta}=(1,1)$ and $\xi=f({\theta})=(2,1)$. We start by computing
$y_{\Xi}$. From the above we know that:

$y_{\Xi}=\operatorname*{arg\,min}_{y}\\{||y||:G(\xi)y=\nabla_{\xi}\mathcal{L}\\}.$
(67)

It can be easily verified that this gives $y_{\Xi}=(3,3)$. For $y_{\Theta}$ we
get:

$y_{\Theta}=\operatorname*{arg\,min}_{y}\\{||\left(F^{T}(\theta)\right)^{-1}y||:G(\xi)y=\nabla_{\xi}\mathcal{L}\\},$
(68)

which gives: $y_{\Theta}=(4\frac{4}{5},1\frac{1}{5})$. Evidently we have
$y_{\Xi}\neq y_{\Theta}$. Note however that when we map the difference of the
two gradient vectors from $T_{(2,1)}\Xi$ to $T_{(3,0)}\mathcal{M}$ through
$d\phi_{(2,1)}$ we get:

$\displaystyle
d\phi_{\xi}\left(df_{\theta}\widetilde{\nabla}_{\theta}\mathcal{L}-\widetilde{\nabla}_{\xi}\mathcal{L}\right)$
$\displaystyle=\left(y_{\Theta}-y_{\Xi}\right)^{i}\partial_{i}(\xi)$ (69)
$\displaystyle=(4\frac{4}{5}-3)\frac{\partial{}}{\partial{x}}|_{(3,0)}+(1\frac{1}{5}-3)\frac{\partial{}}{\partial{x}}|_{(3,0)}$
(70) $\displaystyle=0.$ (71)

This shows that the natural parameter gradient is in general not
reparametrisation invariant on the parameter space, but that the dependency on
the parametrisation disappears when the gradient is mapped to the model.


* [17] Weibing Huang, Charles Albert Lehalle, and Mathieu Rosenbaum.  Simulating and analyzing order book data: The queue-reactive model.  Journal of the American Statistical Association, 110:107–122, 1 2015.
  * [18] Sungil Kim and Heeyoung Kim.  A new metric of absolute percentage error for intermittent demand forecasts.  International Journal of Forecasting, 32(3):669–679, 2016.
  * [19] Charles-Albert Lehalle, Othmane Mounjid, and Mathieu Rosenbaum.  Optimal liquidity-based trading tactics.  Stochastic Systems, 11(4):368–390, 2018.
  * [20] Andrew W. Lo, A. Craig MacKinlay, and June Zhang.  Econometric models of limit-order executions.  Journal of Financial Economics, 65(1):31–71, 2002.
  * [21] Lisa Lorentzen and Haakon Waadeland.  Continued Fractions, volume 1.  Atlantis Press, 2 edition, (2008).
  * [22] Costis Maglaras, Ciamac C. Moallemi, and Muye Wang.  A deep learning approach to estimating fill probabilities in a limit order book.  Quantitative Finance, 22(11):1989–2003, 2022.
  * [23] William H. Press and Saul A. Teukolsky.  Evaluating Continued Fractions and Computing Exponential Integrals.  Computers in Physics, 2(5):88–89, 1988.
  * [24] Eric Smith, J. Doyne Farmer, László Gillemot, and Supriya Krishnamurthy.  Statistical theory of the continuous double auction Statistical theory of the continuous double auction.  Quantitative Finance, 3(6):1–36, 2003.
  * [25] David A. Swanson.  On the Relationship among Values of the same Summary Measure of Error when used across Multiple Characteristics at the same point in time: An Examination of MALPE and MAPE.  Review of Economics and Finance, 5(1):1–14, 2015.
  * [26] AFM Market Watch Edition 8 - Algorithmic Trading.  Technical report, 2023.
  * [27] Ian J. Thompson and A.R. Barnett.  Coulomb and Bessel functions of complex arguments and order.  Journal of Computational Physics, 64(2):490–509, 1986.
  * [28] Ioane Muni Toke and Nakahiro Yoshida.  Modelling intensities of order flows in a limit order book.  Quantitative Finance, 17:683–701, 5 2017.
  * [29] Frank Wilcoxon.  Individual Comparisons by Ranking Methods.  Biometrics Bulletin, 1(6):80, 1945.

##  Appendix A Proofs

###  A.1 Proposition 1

Proof: Since the arrivals follow a Poisson process, the time spent in state
$i$ is exponentially distributed with parameter $\lambda_{i}+\mu_{i}$ and
therefore has density function
$(\lambda_{i}+\mu_{i})e^{-(\lambda_{i}+\mu_{i})t}$. Furthermore, with
probability $\frac{\lambda_{i}}{\lambda_{i}+\mu_{i}}$ the next state is $i+1$
and with probability $\frac{\mu_{i}}{\lambda_{i}+\mu_{i}}$ the next state is
$i-1$. Let $f_{i,i-1}(t)$ be the density function of $\sigma_{i,i-1}$, then we
have

$\displaystyle f_{i,i-1}(t)$
$\displaystyle=\frac{\mu_{i}}{\lambda_{i}+\mu_{i}}(\lambda_{i}+\mu_{i})e^{-(\lambda_{i}+\mu_{i})t}+\frac{\lambda_{i}}{\lambda_{i}+\mu_{i}}(\lambda_{i}+\mu_{i})e^{-(\lambda_{i}+\mu_{i})t}*f_{i+1,i}(t)*f_{i,i-1}(t)$
(142)
$\displaystyle=\mu_{i}e^{-(\lambda_{i}+\mu_{i})t}+\lambda_{i}e^{-(\lambda_{i}+\mu_{i})t}*f_{i+1,i}(t)*f_{i,i-1}(t),$

where the operator $*$ denotes convolution, i.e. $f*g$ is the convolution of
two functions $f$ and $g$. Since this Laplace transform (142) is given by the
product of the separate Laplace transforms, we can now use a useful property
of the Laplace transform of a convolution of (probability) functions to
continue the calculation. Let $\mathcal{L}[f*g](s)$ denote the Laplace
transform of the convolution $f*g$, then

$\mathcal{L}[f*g](s)=\mathcal{L}[g](s)\mathcal{L}[g](s)=\hat{f}(s)\hat{g}(s).$
(143)

Now by taking the (one-sided) Laplace transform on both sides of Equation
(142) and using

$\displaystyle\int_{0}^{\infty}e^{-st}e^{-(\lambda_{i}+\mu_{i})t}\mathrm{d}t$
$\displaystyle=\int_{0}^{\infty}e^{-(\lambda_{i}+\mu_{i}+s)t}\mathrm{d}t$
(144)
$\displaystyle=-\frac{1}{\lambda_{i}+\mu_{i}+s}e^{-(\lambda_{i}+\mu_{i}+s)t}\bigg{|}_{0}^{\infty}$
$\displaystyle=\frac{1}{\lambda_{i}+\mu_{i}+s},$

we obtain

$\hat{f}_{i,i-1}(s)=\frac{\mu_{i}}{\lambda_{i}+\mu_{i}+s}+\frac{\lambda_{i}}{\lambda_{i}+\mu_{i}+s}\hat{f}_{i+1,i}(s)\hat{f}_{i,i-1}(s).$
(145)

Rewriting this expression leads to

$\displaystyle\hat{f}_{i,i-1}(s)$
$\displaystyle=\frac{\mu_{i}}{\lambda+\mu_{i}+s-\lambda\hat{f}_{i+1,i}(s)}$
(146)
$\displaystyle=\cfrac{\mu_{i}}{\lambda_{i}+\mu_{i}+s-\cfrac{\lambda_{i}\mu_{i+1}}{\lambda_{i}+\mu_{i+1}+s-\cfrac{\lambda_{i}\mu_{i+2}}{\lambda_{i}+\mu_{i+2}+s-\cdots}}}$
$\displaystyle=-\frac{1}{\lambda_{i-1}}\text{{\huge
K}}_{k=i}^{\infty}\frac{-\lambda_{k-1}\mu_{k}}{\lambda_{k}+\mu_{k}+s}.$

###  A.2 Lemma 2

Proof: We have $F^{\prime}=f$ (a.s.),

$\hat{F}(s)=\int_{-\infty}^{\infty}e^{-st}F(t)\mathrm{d}t\quad\mathrm{and}\quad\hat{F}^{\prime}(s)=\hat{f}(s)=\int_{-\infty}^{\infty}e^{-st}f(t)\mathrm{d}t.$

By integration by parts, we have

$\displaystyle\hat{f}(s)=\hat{F}^{\prime}(s)$
$\displaystyle=\int_{-\infty}^{\infty}e^{-st}F^{\prime}(t)\mathrm{d}t$ (147)
$\displaystyle=e^{-st}F(t)\bigg{|}_{-\infty}^{\infty}+s\int_{-\infty}^{\infty}e^{-st}F(t)\mathrm{d}t$
(148)
$\displaystyle=s\int_{-\infty}^{\infty}e^{-st}F(t)\mathrm{d}t=s\hat{F}(s).$
(149)

###  A.3 Lemma 3

Proof: Since $X$ is exponentially distributed with rate $\Lambda$ for all
$t\geq 0$, and is independent of $Y$, we have

$\displaystyle\mathbb{P}[X\wedge Y<t]$
$\displaystyle=1-\mathbb{P}[X>t]\mathbb{P}[Y>t]$ (150)
$\displaystyle=1-e^{-\Lambda t}(1-F_{Y}(t)),$

where $F_{Y}$ is the cdf of $Y$. It then follows that $f_{X\wedge Y}(t)$ is
given by:

$\displaystyle f_{X\wedge Y}(t)$
$\displaystyle=\frac{\mathrm{d}}{\mathrm{d}t}\bigg{(}1-(1-F_{Y}(t))e^{-\Lambda
t}\bigg{)}$ (151)
$\displaystyle=\frac{\mathrm{d}}{\mathrm{d}t}\bigg{(}1-e^{-\Lambda
t}+e^{-\Lambda t}F_{Y}(t)\bigg{)}$ $\displaystyle=\Lambda e^{-\Lambda
t}+e^{-\Lambda t}f_{Y}(t)-\Lambda F_{Y}(t)e^{-\Lambda t}$
$\displaystyle=e^{-\Lambda t}\bigg{(}f_{Y}(t)+\Lambda(1-F_{Y}(t))\bigg{)},$

for all $t\geq 0$. The Laplace transform of $f_{X\wedge Y}$ is then given by

$\displaystyle\hat{f}_{X\wedge Y}(s)$
$\displaystyle=\int_{-\infty}^{\infty}e^{-st}f_{X\wedge Y}(t)\mathrm{d}t$
(152) $\displaystyle=\int_{0}^{\infty}e^{-st}e^{-\Lambda
t}\bigg{(}f_{Y}(t)+\Lambda(1-F_{Y}(t))\bigg{)}\mathrm{d}t$
$\displaystyle=\int_{0}^{\infty}e^{-(s+\Lambda)t}f_{Y}(t)\mathrm{d}t+\Lambda\int_{0}^{\infty}e^{-(s+\Lambda)t}(1-F_{Y}(t))\mathrm{d}t$
$\displaystyle=\hat{f}_{Y}(s+\Lambda)+\Lambda\int_{0}^{\infty}e^{-(s+\Lambda)t}(1-F_{Y}(t))\mathrm{d}t.$

By integration by parts, we have for the second part of the last equality

$\displaystyle\Lambda\int_{0}^{\infty}e^{-(s+\Lambda)t}(1-F_{Y}(t))\mathrm{d}t$
$\displaystyle=\Lambda\bigg{(}\bigg{[}(1-F_{Y}(t))\cdot-\frac{1}{s+\Lambda}e^{-(s+\Lambda)t}\bigg{]}_{0}^{\infty}-\frac{1}{s+\Lambda}\int_{0}^{\infty}e^{-(s+\Lambda)t}f_{Y}(t)\mathrm{d}t\bigg{)}$
(153)
$\displaystyle=\Lambda\bigg{(}\frac{1}{s+\Lambda}-\frac{1}{s+\Lambda}\hat{f}_{Y}(s+\Lambda)\bigg{)}$
$\displaystyle=\frac{\Lambda}{s+\Lambda}\big{(}1-\hat{f}_{Y}(s+\Lambda)\big{)}.$

Therefore, we obtain:

$\hat{f}_{X\wedge
Y}(s)=\hat{f}_{Y}(s+\Lambda)+\frac{\Lambda}{s+\Lambda}\big{(}1-\hat{f}_{Y}(s+\Lambda)\big{)}.$
(154)