See pages 1-last of HRG_PAMIv2.pdf


Transferability of Winning Lottery Tickets in Neural Network Differential
Equation Solvers.

Edward Prideaux-Ghee
June 2023

Recent work has shown that renormalisation group theory is a useful framework
with which to describe the process of pruning neural networks via iterative
magnitude pruning. This report formally describes the link between RG theory
and IMP and extends previous results around the Lottery Ticket Hypothesis and
Elastic Lottery Hypothesis to Hamiltonian Neural Networks for solving
differential equations. We find lottery tickets for two Hamiltonian Neural
Networks and demonstrate transferability between the two systems, with
accuracy being dependent on integration times. The universality of the two
systems is then analysed using tools from an RG perspective.

###### Contents

  1. 1 Introduction
  2. 2 Renormalisation Group and Scaling
    1. 2.1 Ising Model and Block Spins
    2. 2.2 RG
    3. 2.3 Fixed points and relevant parameters
    4. 2.4 Scaling of Variables
    5. 2.5 Universality Classes
  3. 3 DNNs and Iterative Magnitude Pruning (IMP)
  4. 4 IMP as an RG
    1. 4.1 Proof IMP is an RG scheme
    2. 4.2 IMP Flow
  5. 5 Neural Network DE Solvers
    1. 5.1 Nonlinear Oscillator
    2. 5.2 IMP Flow
    3. 5.3 Chaotic Hénon-Heiles
    4. 5.4 Universality
  6. 6 Conclusions
  7. References

##  1 Introduction

Within the past few decades, machine learning has developed into an ever more
popular and successful method in solving a variety of problems in fields such
as computer vision and natural language processing (NLP). A subset of these
methods is deep neural networks (DNNs) in which potentially billions of
parameters are trained to find an optimal solution to a problem [1]. The large
size of these networks means they demand extensive computation power as well
as a lot of memory. However, many DNNs are over-parameterised [2] which means
that there is unnecessary computing power being used. This has led to research
into the Lottery Ticket Hypothesis [3] \- a form of transfer learning - which
states:

######  Lottery Ticket Hypothesis.

A dense neural network contains a sparse subnetwork that, when trained under
the same initialisation, can perform with similar or even better accuracy than
the original network when trained for the same number of epochs.

One current method for finding sparse subnetworks with potentially similar
performance to the full network is through a process of pruning: removing
weights from the network. There are many different types of pruning
procedures, such as magnitude based, random and single-shot
[4],[5],[6],[7],[8],[9] with very little consensus on which methods should be
used in different situations. In this report, we will focus on one common
technique - Iterative Magnitude Pruning (IMP) - as it has been shown to give a
”state-of-the-art sparsity-accuracy trade-off” [10]. Winning Lottery Tickets
have been found in several areas such as computer vision [3],[11],[12][13] and
natural language processing (NLP) [14] however the methods used to find them
are very computationally expensive. Therefore, research has been done into the
universality of winning lottery tickets across different tasks. The existence
of universal lottery tickets has been proven [15] suggesting that they could
be a general occurrence across several fields. In fact, transferability of
winning lottery tickets has been shown in several fields [16],[17],[18]
including across networks with different architectures [19]. Until recently,
there was very little theory around the transferability of lottery tickets and
therefore transfer experiments were the only indication as to whether it would
be successful or not. However, work by Redman et.al.[20] has provided a link
between sparsifying neural networks and renormalisation group (RG) theory.
Specifically, it has been shown that IMP is a renormalisation group scheme and
therefore tools from RG theory can be used to study the LTH. RG theory has
historically been used predominantly in statistical mechanics, explaining
universal behaviour in thermodynamic systems during a phase transition [21].
One important parallel between RG theory and pruning DNNs is the concept of
power-law scaling. During a phase transition, variables scale via a power-law
relationship with a critical exponent. The same behaviour has been observed in
IMP [22] with error scaling via the following relationship within a critical
parameter density

$\epsilon\sim cd^{-\gamma}$ (1)

where $\epsilon$ is error, $d$ is density and $\gamma$ is a constant.

One area where the LTH and universality have been less explored is that of
scientific solvers. These address a range of problems, such as handling high-
dimensionality [23],[24],[25][26], and have been applied to a wide range of
situations [27],[28],[29],[30],[31],[32]. Specifically, we will be looking
into neural network differential equation (DE) solvers. These aim to solve an
optimisation problem to provide an analytic, closed form differentiable
function as a solution [33]. These types of solvers have been studied
individually [34] [35] [36] [37] but there exists very little knowledge of
universality between solvers. As an example, we take Hamiltonian Neural
Networks [36] as they provide useful test cases which should hopefully give an
insight into the wider nature of scientific solvers.

##  2 Renormalisation Group and Scaling

In this section, we aim to formally define a renormalisation group (RG) so
that we may later use its related tools for our case of understanding IMP and
the LTH. First, we use the example of block spins in the 2D Ising model from
thermodynamics to introduce the concept of power scaling macro-observables
near a phase transition [38]. This idea is then developed further to give the
concept of the renormalisation group and universality. The development of this
theory will follow the work of Goldenfeld (Lectures on Phase Transitions and
the Renormalization Group, chapter 9, p229-p256) [21] which views
renormalisation group from a thermodynamic phase transition perspective. For a
quantum field theory perspective see [39].

Conceptually, the procedure of the renormalisation group as thought of by
Kandoff contains three steps. The first step is course-graining which involves
reducing the resolution of the system by increasing the smallest length scale
from $a$ to $la>a$. [40]. Then in order to restore the original resolution,
the system is rescaled by reducing all length scales by factor $l$. The final
stage is to renormalise the system so that variables vary on the same scale as
originally.

###  2.1 Ising Model and Block Spins

Consider a system of N spins arranged on a 2-dimensional square lattice with
spacing $a$. For now, we make the assumptions of locality and rotational and
translational symmetry in the Hamiltonian. That is each spin only interacts
with an external magnetic field and its nearest neighbours, with couplings
determined by the constants $k_{1}$ and $k_{2}$ respectively. In this case, we
have the Hamiltonian of the system [21]:

$\mathcal{H}(\textbf{s,k})=-\sum_{i}k_{1}s_{1}-\sum_{\langle
i,j\rangle}k_{2}s_{i}s_{j}$ (2)

Where $s_{i}$ are the spins $s_{i}\in\\{-1,+1\\}$, $k_{1}$ and $k_{2}$ are the
strengths of the coupling constants and $\langle\cdot,\cdot\rangle$ denotes
the nearest neighbours in the lattice. This is the simplest model except the
trivial case where atoms only interact with themselves. As we are interested
in behaviour near critical points and phase transitions for our analysis of
IMP, we denote the singular part of the free energy of the system near the
phase transition as $f_{s}(t,h)$[21]. Within the vicinity of the phase
transition, we must consider quantities on the length scale of the correlation
length $\xi$, which describes the length scale on which variables are
correlated.

The system can then be investigated at different scales by using a process of
”course graining” to view a block of multiple spins as one singular spin with
properties aggregated from the constituent sub-spins [21]. If we take a block
of spins of side length $al$, we then have a system of $Nl^{-2}$ blocks each
containing $l^{2}$ individual spins. We define this block-spin transformation
in the following way:

$s^{\prime}=\frac{1}{|\bar{m}_{l}|}\frac{1}{l^{2}}\sum_{i\in I}s_{i}$ (3)

where I is the set of indices of spins in the new block and $\bar{m}_{l}$ is
the average magnetisation of the block defined by

$\bar{m}_{l}=\frac{1}{l_{2}}\sum_{i\in I}\langle s_{i}\rangle.$ (4)

Re-normalising the block spins like this ensures that they can take the same
values as the original individual spins ($\pm 1$). Now we assume that, similar
to the original system, each block spin interacts only with its nearest
neighbouring blocks and the external field. Hence we can define new coupling
constants $k_{i}^{\prime}$ which determine the nature of these interactions.
This assumption leads us to finding the new effective Hamiltonian of the
system to be:

$\mathcal{H}(\textbf{s',k'})=-k_{1}^{\prime}\sum_{i}s_{i}^{\prime}-k_{2}^{\prime}\sum_{\langle
i,j\rangle}s_{i}^{\prime}s_{j}^{\prime}$ (5)

which is of the same form as the original Hamiltonian. Considering the
correlation length $\xi$, initially we had $\xi_{1}$ measured on a scale of
the lattice spacing $a$. After performing the course graining, we now have
$\xi_{l}$ which is measured on a scale of the new spacing between blocks $la$.
This therefore gives the relation $\xi=\xi_{l}la=\xi_{1}a$ which gives:

$\xi_{l}=\xi_{1}/l$ (6)

One key property near the critical point during phase transition is the
divergence of the correlation length $\xi\rightarrow\infty$. As $l>1$,
$\xi_{l}<\xi_{1}$ which means the new system is further from the critical
point than in the original system, and hence the new system is at an effective
reduced temperature $t^{\prime}$ and magnetisation $h^{\prime}$ [21].
Combining these results, the form of the free energy of the effective system
$f_{s}(t^{\prime},h^{\prime})$ is the same as the original system but with new
reduced temperature and magnetisation variables.

$f_{s}(t^{\prime},h^{\prime})=l^{2}f_{s}(t,h)$ (7)

Therefore, the process of course graining has retained the structure of the
system, although with scaled parameters. So, denoting this block spin
transformation as $\mathcal{R}$, we can describe the effect on the Hamiltonian
as:

$\mathcal{R}\mathcal{H}(\textbf{s,k})=\mathcal{H}(\textbf{s}^{\prime},\mathcal{T}\mathbf{k})=\mathcal{H}(\textbf{s',k'})$
(8)

Where $s^{\prime},k^{\prime}$ are the new spins and coupling constants
respectively, and the mapping of the coupling constants is denoted by
$\mathcal{T}:\mathbb{R}^{K}\rightarrow\mathbb{R}^{K}$. As $\mathcal{R}$ is
repeatedly applied, we get a flow of coupling constants, known as the RG flow,
which is determined by the eigenvectors of $\mathcal{T}$ when linearized near
a fixed point. The corresponding eigenvalues are classified into three
classes: relevant if $\lambda_{i}>1$; irrelevant if $\lambda_{i}<1$; or
marginal if $\lambda_{i}=1$.

As our interest is in understanding the power law scaling, we further assume
that the temperature and magnetism in the critical region transform under
course graining by:

$\begin{split}t^{\prime}=tl^{y_{t}}\hskip 21.68121pty_{t}>0\\\
h^{\prime}=hl^{y_{h}}\hskip 21.68121pty_{h}>0\end{split}$ (9)

Inputting this into (7) gives us

$f_{s}(t,h)=l^{-2}f_{s}(tl^{y_{t}},hl^{y_{h}})$ (10)

As there are no restrictions on the level of scaling being implemented ($l$),
taking $l$ such that $|t|l^{y_{t}}=1$ allows for the convenient form:

$f_{s}(t,h)=|t|^{d/y_{t}}f_{s}(1,h|t|^{-y_{h}/y_{t}})$ (11)

This is solely a function of $h|t|^{-y_{h}/y_{t}}$ and hence can be written

$f_{s}(t,h)=|t|^{d/y_{t}}F_{f}(h|t|^{-y_{h}/y_{t}})$ (12)

where $F_{f}(x)=f_{s}(1,x)$. This can therefore be written in the form of the
static scaling hypothesis by denoting $2-\alpha=\frac{d}{y_{t}}$ and
$\Delta=\frac{y_{h}}{y_{t}}$:

$f_{s}(t,h)=|t|^{2-\alpha}F_{f}(h|t|^{\Delta})$ (13)

This is a specific example of how a course-graining procedure can lead to the
observed scaling of macro-observables by discounting the irrelevant couplings
between degrees of freedom on a smaller scale.

###  2.2 RG

The above example of block spins in the 2D Ising model demonstrates how it
makes sense to take this form of the scaling for certain variables when near
the critical point. However, the assumptions made cannot make sense when
extended to different situations. For example, in the case where spins do not
interact with their neighbours but do interact with spins further away. In
this case, it would not make sense for block spins, comprised of multiple
spins, to not interact with their neighbours. Therefore, the concepts of
renormalisation group theory are now developed for a more general Hamiltonian.

In order to concretely define a renormalisation group, the course-graining
process used above must be formally analysed, and the origin of singular
behaviour must be understood. As in the block spin case, the fundamental
concept is that if given a system with interactions on a scale $a$, we take
blocks of length $la$, then rescaling and normalising the resulting system
gives one similar to the original system with respect to the degrees of
freedom. This gives us the characteristic that slightly different systems (e.g
interactions between nearest neighbours/next-nearest neighbours) can have the
same effective Hamiltonians after course-graining which is important in the
context of universality classes. For example, in the block spin case where
individual spins do not interact with their nearest neighbours, the resulting
system after course-graining could be similar to that where individual spins
do.

Now we consider Hamiltonians of the general form

$\mathcal{H}=\sum_{n}K_{n}\Theta_{n}\\{S\\}$ (14)

with coupling constants $K_{n}$, and $\Theta_{n}$ local operators as functions
of the degrees of freedom $\\{S\\}$ [21]. Now we perform course-graining
similarly to before, collecting degrees of freedom together in a linear block
of length $la$. We call this general transformation a ”renormalisation group
transformation” $R_{l}$. As before, we are free to choose the level of course-
graining $l$ so can denote the transformation of coupling constants:

$[\textbf{k}^{\prime}]=\mathcal{R}_{l}[\textbf{k}]$ (15)

The renormalisation transformations $R_{l}$ in fact form a semi-group where
successive transformations with course-graining scales $l_{1},l_{2}$ are
equivalent to a single transformation with scale $l_{1}\cdot l_{2}$ [21]:

$R_{l_{1}l_{2}}[K]=R_{l_{2}}\circ R_{l_{1}}[K]$ (16)

In order to calculate $R_{l}$, we now properly define the course graining
procedure. First we will need to utilise the partition function $Z_{n}$ and
quantity $g$ relating to free energy per degree of freedom.

$\begin{split}Z_{N}[K]=&Tr\hskip 7.22743pte^{\mathcal{H}}\\\
g[K]=&\frac{1}{N}\log Z_{N}[K]\end{split}$ (17)

In order to perform the course graining process and reduce the number of
degrees of freedom, we perform a partial trace over the degrees of freedom.

$\begin{split}e^{\mathcal{H}^{\prime}_{N}\\{[K^{\prime}],S_{I}^{\prime}\\}}&=Tr^{\prime}_{\\{S_{i}\\}}e^{\mathcal{H}_{N}\\{[K],S_{I}\\}}\\\
&=Tr_{\\{S_{i}\\}}\mathcal{P}(S_{i},S_{I}^{\prime})e^{\mathcal{H}_{N}\\{[K],S_{I}\\}}\end{split}$
(18)

Where $Tr_{\\{S_{i}\\}}$ is the trace operator over the values that $S_{i}$
can take {$\pm 1$}. Here, to allow the trace to be unrestricted, a projection
operator $P(S_{i},S^{\prime}_{I})$ is used. This is constructed in such a way
that the block degrees of freedom $S_{I}$ take the same values as the original
degrees of freedom, thus maintaining the characteristics of the original
system.

As an example of how the projection operator is constructed, we use the case
of the Ising 2D system as previous explained. Having Ising spins on a 2D
lattice, we use an RG transformation with blocks of length $(2l+1)a$ so that
there is an odd number of spins $(2l+1)^{2}$ in each block. In order for the
block spins have the same values as the original degrees of freedom, we define

$S^{\prime}_{I}=sign(\sum_{i\in I}S_{i})=\pm 1$ (19)

Therefore, there is the associated projection operator

$P(S_{i},S^{\prime}_{I})=\Pi_{I}\delta(S^{\prime}_{I}-sign(\Sigma_{i\in
I}S_{i}))$ (20)

where $\delta$ is a Kronecker delta function. Clearly this is not a unique RG
transformation that ensures $S^{\prime}_{I}=\pm 1$. However the projection
operator must satisfy the following three properties [21]:

  1. i

$P(s_{i},s_{I}^{\prime})\geq 0$

  2. ii

$P(s_{i},s_{I}^{\prime})$ respects the symmetries of the system

  3. iii

$\sum_{\\{s_{I}^{\prime}\\}}P(s_{i},s_{I}^{\prime})=1$

Condition (i) is necessary to ensure that the exponential of the effective
Hamiltonian $e^{\mathcal{H}^{\prime}_{N}\\{[K^{\prime}],S_{I}^{\prime}\\}}\geq
0$ and is thus $\mathcal{H}^{\prime}$ is well defined as the effective block
spin Hamiltonian.

The second condition (ii) guarantees that there exist no new forms of
couplings or symmetries in the new system that were not possible in the
original system i.e. the effective Hamiltonian can be written in the same form
as originally, but with different values of coefficients. i.e. Given a system
of N degrees of freedom described by Hamiltonian:

$\mathcal{H}_{N}=NK_{0}+h\sum_{i}S_{i}+K_{1}\sum_{i,j}S_{i}S_{j}+...$ (21)

the effective Hamiltonian can be written in an equivalent way:

$\mathcal{H^{\prime}}_{N^{\prime}}=N^{\prime}K^{\prime}_{0}+h^{\prime}\sum_{I}S^{\prime}_{I}+K^{\prime}_{1}\sum_{I,J}S^{\prime}_{I}S^{\prime}_{J}+...$
(22)

This can include cases where certain coefficients in the original system are
zero but are not zero in the effective system. For example, in a spin system
where every pair of spins interact, but there are no interactions of spin
triples ($K_{2}\sum_{I,J,K}S_{I}S_{J}S_{K},\hskip 3.61371ptK_{2}=0$), the
effective system could have interactions of this type ($K^{\prime}_{2}\neq
0$).

Finally, condition (iii) ensures that there is a unique well-defined mapping
of old degrees of freedom to the new ones. Furthermore, in the probabilistic
case, this condition ensures it is well-defined by ensuring the sum of all the
probabilities of transformed variables is 1. Furthermore, (iii) means that the
partition function $Z_{N}$ is invariant under RG transformation [21]:

$\begin{split}Z_{N}^{\prime}[K^{\prime}]=&Tr_{\\{S^{\prime}_{I}\\}}\hskip
3.61371pte^{\mathcal{H}^{\prime}_{N^{\prime}}\\{[K^{\prime}],S^{\prime}_{I}\\}}\\\
=&Tr_{\\{S^{\prime}_{I}\\}}Tr_{\\{S_{i}\\}}\hskip
3.61371ptP(S_{i},S^{\prime}_{I})e^{\mathcal{H}_{N}\\{[K],S_{i}\\}}\\\
=&Tr_{\\{S_{i}\\}}\hskip 3.61371pte^{\mathcal{H}_{N}\\{[K],S_{i}\\}}\cdot 1\\\
=&Z_{N}[K]\end{split}$ (23)

As this is invariant, we can clearly see that the previously defined quantity
$g$ for the free energy follows the relation:

$\begin{split}g[K^{\prime}]=&\frac{1}{N^{\prime}}\log
Z_{N^{\prime}}[K^{\prime}]\\\ =&\frac{1}{l^{-d}N}\log Z_{N}[K]\\\
=&l^{d}g[K]\end{split}$ (24)

clearly conserving total energy as the free energy per degree of freedom has
been scaled by the scaling factor $l^{d}$. One benefit of RG theory is that it
is easier to establish approximate parameters [k’] as opposed to calculating
the partition function.

###  2.3 Fixed points and relevant parameters

Now in order to analyse the behaviour of systems after repeated applications
of the RG transformations, we consider the ”flow” of parameters $[K^{(n)}]$.
This group of constants from all sets of initial parameters $[K^{(0)}]$ is
known as the renormalisation group flow. It has been noted [21] that the
trajectories of parameters are usually attracted to certain fixed points, near
to which the systems demonstrate characteristic scaling behaviour.

Denote a fixed point of the RG transformation $R_{l}[K]$ in parameter space as
$[K^{*}]$. It therefore has the property

$[K^{*}]=R_{l}[K^{*}]$ (25)

Looking at the correlation length at the fixed point, relation (6) gives

$\xi[K*]=\xi[K*]/l$ (26)

which implies $\xi$ is either $\infty$ of $0$. These two cases are denoted as
”critical” and ”trivial” fixed points respectively.

Within a vicinity of the fixed point, the parameters and Hamiltonian can be
written

$K_{n}=K^{*}_{n}+\delta\hat{K}_{n},\hskip
36.135pt\mathcal{H}=\mathcal{H}^{*}+\delta\hat{\mathcal{H}}$ (27)

where $\delta$ is a small parameter. Performing the RG transformation $R_{l}$
gives

$R_{l}[K_{n}]=K^{\prime}_{n}=K^{*}_{n}+\delta\hat{K}^{\prime}_{n}$ (28)

where $\hat{K}^{\prime}_{n}$ can be represented by the Taylor expansion:

$\hat{K}^{\prime}_{n}=K^{*}_{n}+\sum_{m}\frac{\partial\hat{K}^{\prime}_{n}}{\partial
K_{m}}|_{K=K^{*}}\cdot\delta K_{m}+O((\delta K)^{2})$ (29)

so $\delta\hat{K}^{\prime}_{m}=\sum_{m}M_{nm}\delta K_{m}$ where
$M_{nm}=\frac{\partial\hat{K}^{\prime}_{n}}{\partial K_{m}}|_{K=K^{*}}$ is the
linearisation of the RG transformation near the fixed point. To analyse the RG
flow in the vicinity of the fixed point $K^{*}$, the eigenvalues and
eigenvectors of this linearisation are required. Denote the eigenvalues of
$M^{(l)}$ (associated with transformation scale $l$) as $\lambda^{(l)}_{i}$
with corresponding eigenvectors $\textbf{v}^{(l)}_{i}$. Due to the
associativity of the RG transformation, we have the following property:

$\begin{split}&M^{(l)}M^{(l^{\prime})}=M^{(ll^{\prime})}\\\
\implies&\lambda^{(l)}_{i}\lambda^{(l^{\prime})}_{i}=\lambda^{(ll^{\prime})}_{i}\end{split}$
(30)

Clearly we can see that $\lambda_{i}^{1}=1$ as
$\lambda^{(l)}_{i}\lambda^{1}_{i}=\lambda^{(l)}_{i}$. Therefore,
differentiating and setting $l^{\prime}=1$, we get the differential equation:

$\begin{split}&l\frac{d\lambda^{(l)}}{dl}=y_{i}\lambda^{(l)}\\\
\rightarrow&\lambda^{(l)}=l^{y_{i}}\end{split}$ (31)

where $y_{i}$ is independent of $l$ and is to be determined.

The different values of $y_{i}$ for each eigenvector are important as they
describe in what directions the components of $\delta K$ grow or shrink.

  1. 1.

$y_{i}>0$ ($|\lambda_{i}|>1$) implies $\delta K$ grows in the $\textbf{v}_{i}$
direction.

  2. 2.

$y_{i}=0$ ($|\lambda_{i}|=1$) implies $\delta K$ is invariant in the
$\textbf{v}_{i}$ direction.

  3. 3.

$y_{i}<0$ ($|\lambda_{i}|<1$) implies $\delta K$ shrinks in the
$\textbf{v}_{i}$ direction.

In case (1) they are called ”relevant” eigenvalues/eigenvectors, in case (2),
they are called ”marginal” eigenvalues/eigenvectors, and case (3) are called
”irrelevant” eigenvalues/eigenvectors.

Therefore, the relevant and irrelevant directions determine the flow of
parameters near a fixed point, and hence the critical behaviour. When the
initial point is slightly away from the critical manifold of a fixed point (in
parameter space), it will begin to travel towards the fixed point until it
gets repelled according to the relevant eigenvectors.

This gives one part to the idea of universality as different initial
conditions in parameter space can develop the same critical behaviour when
$\mathcal{R}$ is repeatedly applied.

###  2.4 Scaling of Variables

Now that the critical behaviour has been explained using RG theory, it can be
seen how RG leads to the scaling behaviour around criticality.

In the example of the Ising model, taking temperature and magnetism as $T$ and
$H$, there are two relevant directions to consider: $t$ and $h$. As stated
earlier, the singular part of the free energy density obeys the relation:

$f_{s}(t,h)=l^{-d}f_{s}(t^{\prime},h^{\prime})$

Additionally, $T$ and $H$ both undergo individual course-graining processes
$\mathcal{R}_{l}^{T}$ and $\mathcal{R}_{l}^{H}$.

Following the RG analysis, using the translated variables

$\begin{split}\Delta T=&T-T^{*}\\\ \Delta H=&H-H^{*}\end{split}$ (32)

we find the linearisation of the RG transformation

$\begin{pmatrix}\Delta T^{\prime}\\\ \Delta
H^{\prime}\end{pmatrix}=M\begin{pmatrix}\Delta T\\\ \Delta H\end{pmatrix}$

where

$M=\begin{pmatrix}\frac{\partial R_{l}^{T}}{\partial T}&\frac{\partial
R_{l}^{T}}{\partial H}\\\ \frac{\partial R_{l}^{H}}{\partial T}&\frac{\partial
R_{l}^{H}}{\partial H}\end{pmatrix}_{T=T^{*},H=H^{*}}.$

Then writing the eigenvalues as before ($\lambda_{l}^{t}=l^{y_{t}}$,
$\lambda_{l}^{h}=l^{y_{h}}$), the linearised RG transformation becomes

$\begin{pmatrix}t^{\prime}\\\
h^{\prime}\end{pmatrix}=\begin{pmatrix}\lambda_{l}^{t}&0\\\
0&\lambda_{l}^{h}\end{pmatrix}\begin{pmatrix}t\\\ h\end{pmatrix}$ (33)

Now considering the free energy and correlation length, it can be seen that
free energy evolves as

$f_{s}(t,h)=l^{-nd}f_{s}(t^{(n)},h^{(n)})=l^{-nd}f_{s}(l^{ny_{t}}t,l^{ny_{h}}h)$
(34)

which is of the same form as in the block spin case (10).

Correlation length evolves according to (6) hence after n iterations of the
renormalisation group transformation

$\xi(t,h)=l^{n}\xi(l^{ny_{t}}t,l^{ny_{h}}h)$ (35)

As the only restriction on $l$ is that $l>0$, taking $l^{n}=bt^{-1/y_{t}}$ for
some arbitrary value b gives what is known as the static scaling hypothesis:

$f_{s}(t,h)=t^{d/y_{t}}b^{-d}f_{s}(b,h/t^{y_{h}/y_{t}})$ (36)

The static scaling hypothesis is usually written in the form

$f_{s}(t,h)=t^{2-\alpha}F_{f}(\frac{h}{t^{\Delta}})$ (37)

where $F_{f}(x)=f_{s}(1,x)$, and $\alpha,\Delta$ are critical exponents [21].
Therefore, RG has provided a method to approximate the critical exponents from
the eigenvalues of the linearised RG transformation:

$\begin{split}2-\alpha=&\frac{d}{y_{t}}\\\
\Delta=&\frac{y_{h}}{y_{t}}\end{split}$ (38)

###  2.5 Universality Classes

As described in the LTH, we are very interested in the idea of transferability
across multiple different systems in order to improve training time.
Therefore, the concept of universality classes is now briefly discussed in
more detail.

It has been seen that iterating the renormalisation group transformation
causes the coupling constants to flow towards a fixed point, irrelevant of
initial conditions. However, assuming the start point was off of the critical
manifold, eventually, the irrelevant directions will have been ”iterated out”
[41] leaving only the relevant directions controlling the macro-behaviour of
the system. Therefore, different systems that share the same critical
exponents will have the same relevant directions, and therefore demonstrate
the same behaviour in this limit.

##  3 DNNs and Iterative Magnitude Pruning (IMP)

Now that the surrounding RG theory has been introduce, it must now be applied
to the context of DNNs and pruning for winning lottery tickets. Therefore, the
structures of neural networks and mechanisms of pruning are now set up.

The case that will be considered is that of a feed-forward DNN being pruned
via iterative magnitude pruning (IMP). A feed-forward DNN consists of layers
of neurons/units, with neurons in different layers being interconnected by
weighted links. When an input is given to the first layer (input layer) of
neurons, the following layers have values dependent on the values in the
previous layer and the weighted connections. Denoting the weights connecting
unit $i$ in layer $l$ and unit $j$ in layer $l+1$ as $\theta_{ij}$, the value
of neuron $j$ in layer $n$ can be written
$a_{j}=h(\sum_{i\in\textbf{n}}w_{ij}a_{i}+b)$, where $b$ is an added bias term
and $h$ is a non-linear activation function (eg relu, tanh…).

Once this process had continued through the network and reached the
final/output layer, a final function can be applied to perform a task, such as
data classification (sigmoid). In cases where supervised learning is used, the
performance of the network can be measured via a loss function comparing the
accuracy of the output compared to a known result (e.g % of images correctly
classified). However, in cases of unsupervised learning, the output accuracy
can be quantified by comparing the performance to a desired result. This is
controlled via a loss function $\mathcal{L}$. Gradient descent with respect to
the loss is then performed on the weights in the network to minimise the loss.
This stage is called backpropagation. Iterating this process (if
hyperparameters are carefully selected) leads to improved performance at the
task as the parameters are optimized to minimise the loss.

As previously stated, training DNNs is a computationally expensive task which
has motivated research into a variety of methods to increase efficiency of
training. This includes recent work in the areas of transfer learning [42],
pruning [4], and utilising Koopman Operator Theory [43]. We are most
interested in reducing the over-parameterisation of DNNs and finding sparse
subnetworks with similar performance to the original fully connected network
and hence will be using pruning. IMP is one such method of doing this,
removing weights with the smallest magnitude after t iterations of training
the network. For more detailed analysis of the theory behind IMP, see
[44],[45]. An algorithm for IMP, removing one parameter from the network each
iteration of pruning, can be written as follows [44]:

Algorithm 1: IMP
For loss function $\mathcal{L}:\mathbb{R}^{p}\rightarrow\mathbb{R}$, training
time $T\in\mathbb{R}_{+}$, initial weights $w^{init}\in\mathbb{R}^{p}$ and
$q<p$ iterations of pruning:

  * •

Set $M=\mathbf{1}_{P}$

  * •

For $k=0$ to $q$:
Initialise $w^{(k)}(0)=Mw^{init}$
Train $\dot{w}^{(k)}(t)=-M\nabla L(w^{(k)}(t))$ for $t\in[0,T]$
Set $i=$ argmin${}_{j\in[p]}\\{|w_{j}^{(k)}|:M_{jj}=1\\}$
Set $M_{ii}=0$

  * •

Return $w^{(q)}(T)$

When applying the IMP algorithm to a DNN, it can either be applied across the
whole network, or to individual layers, removing a specified proportion of
non-zero weights each iteration. In practice, this is a computationally
expensive process so to reduce the number of times that the network must be
trained, a percentage of the remaining weights must be removed each iteration.
This must be large enough to reduce the number of iterations, be small enough
to not over-prune.

Work by Frankle and Carbin [3] shows the importance of resetting the weights
to their original values before retraining. This is because when finding a
potential lottery ticket, the initialisation is crucial to the efficiency of
training the subnetwork and random initialisation results in decreased
accuracy. This is because the mask is specific to the initialisation and
therefore will only find a sparsified subnetwork with similar efficiency if
the initial point is the same, as stated in the LTH.

Furthermore, once pruning is complete, the sparsified network must be fully
trained again from the initial point in order to give accurate results. This
is because the just pruned network is not fully optimised due to certain
connections being set to zero, even if they were small.

The importance of the initialisation in terms of the LTH means that a winning
ticket is comprised of both the mask $M$ and the weight initialisation $w(0)$.

##  4 IMP as an RG

###  4.1 Proof IMP is an RG scheme

In order to use the RG tools defined in the previous section for the purposes
of analysing IMP, it must be shown that IMP can be seen as a renormalization
group transformation. To do this, comparisons are made between DNNs with IMP
and the 2D Ising model in order to motivate the construction of a similar
course graining process to that of the block spins. It can in fact be shown
that they are both systems of an equivalent form. Although some features of RG
theory are not yet fully understood, such as correlation length, following the
work of Redman et.al.[20], we can prove IMP is a renormalisation group
operator and hence gather useful tools from this link.

On a conceptual basis, it is clear to see how a DNN under IMP can be
visualised as an Ising model with course-graining. A DNN can be visualised as
a lattice structure of nodes with each node characterised by the activation
value $a_{j}=h(\sum_{i\in\textbf{n}}w_{ij}a_{i}+b)$. Then each node is
connected by a weighted connection to the nodes in the neighbouring layer.

Taking a more formal approach, we can immediately see the similarities between
the effects of IMP $\mathcal{I}$ operator and the $\mathcal{R}$ operator from
the block spin system. Consider a DNN with parameters $\boldsymbol{\theta}$,
activations a, and loss function $\mathcal{L}$. Applying the IMP operator
$\mathcal{I}$ gives:

$\mathcal{I}\mathcal{L}(\textbf{a},\boldsymbol{\theta})=\mathcal{L}(\textbf{a}^{\prime},\boldsymbol{\theta}^{\prime})=\mathcal{L}(\textbf{a}^{\prime},\mathcal{T}\boldsymbol{\theta})$
(39)

This is analogous to the $\mathcal{R}$ operator acting on the Hamiltonian in
the classical spin system, taking $\mathcal{L}$ as analogous to $\mathcal{H}$,
a for s and $\boldsymbol{\theta}$ for K. Here the operator
$\mathcal{T}:\mathbb{R}^{N}\rightarrow\mathbb{R}^{N}$ (N being the total
number of parameters in the network) is a composition of a masking operator
$\mathcal{M}$ and a training operator $\mathcal{F}$, thus only retraining the
non-pruned parameters. $\mathcal{M}$ is defined by the chosen method of
pruning, which in our case is magnitude pruning, and $\mathcal{F}$ is defined
by the optimizer used and whether the weights are being rewound to their
initialisation.

Hence the full process of IMP can be written:

$\mathcal{I}^{n}\mathcal{L}(\textbf{a}^{0},\boldsymbol{\theta}^{0})=\mathcal{L}(\textbf{a}^{\prime
n-1},\boldsymbol{\theta}^{\prime n-1})=\mathcal{L}(\textbf{a}^{\prime
n-1},\mathcal{T}^{n}\boldsymbol{\theta}^{0})$ (40)

This sequence of evolving parameters $\\{\boldsymbol{\theta}^{i}\\}_{i}$,
determined by the eigenvectors of $\mathcal{T}$, gives an equivalent to the RG
flow described before. In the context of IMP, we call this sequence the IMP
flow.

In order to show that this is an RG scheme, we consider analogous elements of
the system to that of the block spin model. Analogous to the spins s is the
activations a of each unit in the model; the couplings between spins are
analogous to the model parameters $\boldsymbol{\theta}$; and the Hamiltonian
$\mathcal{H}$ is analogous to the loss function
$\mathcal{L}(\textbf{a},\boldsymbol{\theta})$ [20].

For a general DNN, we define the activation $a_{j}^{(i)}$ of the j-th unit in
layer i as:

$a_{j}^{(i)}=h[\sum_{k}g_{k}(\textbf{a},\boldsymbol{\theta})]$ (41)

where $g_{k}(\textbf{a},\boldsymbol{\theta})$ are the functions that determine
the effect of other parameters and activations on $a_{j}^{(i)}$ and $h$ is the
selected activation function. For example, in a feed-forward DNN, each
activation is the sum of the product of the previous layers’ activations and
the weights of parameters connecting them. There is also an additional bias
term added to each activation. Thus, there are two functions:

  1. 1.

Bias: $g_{0}=\theta_{j}^{(i)}$

  2. 2.

Weighted input from previous layer:
$g_{1}=\sum_{k=1}^{N^{i-1}}\theta_{jk}^{(i)}a_{k}^{(i-1)}$

Where $\theta_{j}^{(i)}$ are biases for unit $j$ in layer $i$,
$\theta_{jk}^{(i)}$ are weights connecting $a_{k}^{(i-1)}$ to $a_{j}^{(i)}$
and $N^{i-1}$ is the number of activations in layer ($i$-1).

Therefore after applying the IMP operator $\mathcal{I}$, the parameters
$\boldsymbol{\theta}$ have been transformed hence giving:

$a_{j}^{\prime(i)}=h[\sum_{k}g_{k}(\textbf{a}^{\prime},\mathcal{T}\boldsymbol{\theta})]=h[\sum_{k}g_{k}(\textbf{a}^{\prime},\mathcal{F}\circ\mathcal{M}\boldsymbol{\theta})]$
(42)

Therefore, we can define the associated projection operator for $\mathcal{I}$
in the same way as we did for the block spin case.

$P(a_{j}^{(i)},a_{j}^{\prime(i)})=\prod_{I}\delta(a_{j}^{\prime(i)}-h[\sum_{k}g_{k}(\textbf{a}^{\prime},\boldsymbol{\theta}^{\prime})])$
(43)

To ensure this is an RG projection operator, it can verified that it meets the
previously defined requirements of a projection operator (2.2): [20]

  1. i

As $P()$ is a product of Kronecker deltas, it clearly satisfies (i)
$P(s_{i},s_{I}^{\prime})\geq 0$

  2. ii

Property (ii) is met as the process of IMP only removes parameters from the
model. Therefore, until the point where every weight in a layer has been
pruned (layer collapse), the activations $a_{j}^{(i)}$ still have the same
form, and thus the loss function is still of the same form.

  3. iii

Property (iii) is met when the masking and refining operators $\mathcal{M}$
and $\mathcal{F}$ are deterministic. This requires fixing the seed so that the
sampling order of test and train data (in the case of supervised learning) is
fixed for every epoch. This ensures that $P$ is a unique projection operator.

As the constructed projection operator fulfils the requirements of a
renormalisation group projection operator, it has been shown that
$\mathcal{I}$ is and RG operator.

###  4.2 IMP Flow

Now that working in an RG framework has been justified, we can use the tools
available in this theory to analyse IMP behaviour. Clearly understanding the
critical manifold is of great importance as it can give a clear indication of
which parameters are relevant and irrelevant during pruning. The parameters
$\theta_{i}$ are transformed by the transformation $\mathcal{T}$. Therefore,
by finding eigenfunctions of $\mathcal{T}$, we can find the relevant and
irrelevant eigenvalues/eigenvectors [20]. This is important in the context of
the LTH as different models which have the same relevant eigenvectors
$\textbf{v}_{i}$ with $\lambda_{i}>1$, should share the same subset of
parameters which remain after pruning.

One proposed eigenfunction of $\mathcal{T}$ by [20] which will be considered
relates to the proportion of total magnitude of parameters remaining in each
layer after n iterations of IMP. Consider the function

$M_{i}(n)=\frac{\sum_{j}|\theta^{(i)}_{j}(n)|}{\sum_{i}\sum_{j}|\theta_{j}^{(i)}(n)|}$
(44)

where $\theta_{j}^{(i)}(n)$ is the jth weight in layer i after n iterations of
IMP. In order to find the eigenvalues associated with this eigenfunction, we
use the following relation

$M_{i}(n+1)=\lambda_{i}M_{i}(n)$ (45)

Therefore, to find $\lambda_{i}$ we invert this to get
$\lambda_{i}=\frac{M_{i}(n+1)}{M_{i}(n)}$. As in the process of defining
renormalisation groups, in order to having a meaningful value for the
relevance of each layer, we write $\lambda_{i}=l^{\sigma}_{i}$ where
$\sigma_{i}$ is invariant of the course-graining $l$. Therefore, $\sigma_{i}$
may be used to compare behaviour with different models which may use a
different pruning rate $l$.

##  5 Neural Network DE Solvers

To support the use of RG tools in finding universal lottery tickets, we apply
this theory to the case of neural networks for solving differential equations.
This is an important field as, unlike traditional solvers which generate a
solution at a range of individual points, DNNs can provide a differentiable
solution in a closed analytic form [33]. Another advantage is that they are
very capable when solving high dimensional problems while exploiting parallel
computing. They have been found to be applicable to a wide range of PDEs and
dynamical systems [46],[35],[33],[28],[24],[47],[48] so RG theory can allow us
to find ”similar” systems amongst these to help improve training times by
utilising transferable lottery tickets.

A useful example to consider is Hamiltonian Neural Networks (HNNs). These are
designed to use unsupervised learning to solve problems while conserving
physical properties - such as energy - and are applicable to wide range of
problems of different complexities in Hamiltonian mechanics [49]. Even though
these networks are less complex than some solvers, they provide a useful
insight into how RG tools can be used to analyse IMP and universality in
scientific solvers for well-posed problems.

As an example, we perform IMP on a existing neural network. The codebase for
this can be found here [50]. Work by Mattheakis et.al.[35] explores the
behaviour of Hamiltonian neural networks being used to solve Hamilton’s
equations in order to obtain equations of motion for two different dynamical
systems - a non-linear oscillator and a Hénon-Heiles system. For more analysis
of these models see [35, 51, 52].

###  5.1 Nonlinear Oscillator

The first case considered is a neural network designed to solve the nonlinear
one-dimensional an-harmonic oscillator described by the Hamiltonian

$\mathcal{H}=\frac{p^{2}}{2}+\frac{x^{2}}{2}+\frac{x^{4}}{4}$ (46)

where mass and natural frequency are taken to be 1. The Hamiltonian
$\mathcal{H}$ represents the total energy $E$ of the system while the
associated Hamilton’s equations for this system, given by
$\dot{x}=\frac{\partial H}{\partial p}$ and $\dot{p}=-\frac{\partial
H}{\partial x}$, are:

$\dot{x}=p,\hskip 72.26999pt\dot{p}=-(x+x^{3})$ (47)

These are solved by the neural network by minimising the loss function

$L=\frac{1}{K}\sum_{n=1}^{K}[(\dot{\hat{x}}^{(n)}-\dot{\hat{p}}^{(n)})^{2}+(\dot{\hat{p}}^{(n)}+\dot{\hat{x}}^{(n)}+(\dot{\hat{x}}^{(n)})^{3})^{2}]$
(48)

which is the mean square error from Hamilton’s equations.

The neural network used to solve this optimisation problem has 2 hidden
layers, each with 50 neurons, and an output layer of 2 neurons (corresponding
to the two degrees of freedom in the problem). Therefore, there are a total of
(50 + 50x50 + 50x2 =) 2650 weights in the neural network and 102 biases. The
hyperparameters were set to the same values as used by Mattheakis et.al.
(lr=$8\cdot 10^{-3}$, $5\cdot 10^{4}$ epochs, …) as these were shown to
achieve a high accuracy solution which we aim to maintain during pruning.

For now, we will ignore the effect of pruning biases directly, as much of that
behaviour will be implicitly included when pruning weights. This is because
when weights connecting to a neuron are pruned, the bias at that neuron
becomes less significant.

In order to obtain an effective theory of IMP in neural networks for solving
DEs, different pruning procedures are performed in order to understand
different scalings that exist in the model. First, the effect of pruning
individual layers is explored. Therefore, to carry out the previously
explained algorithm in this case, we only use the masking operator over
parameters in the desired layer - by which the process for one iteration of
pruning on layer $l$ becomes:

  * •

Fully train neural network

  * •

Prune $p\%$ of remaining weights in layer $l$

  * •

Reset remaining weights to initial values

  * •

Fully train remaining parameters

To ensure IMP is implemented correctly, the weights are reset to the initial
random values after each pruning iteration. Furthermore, to ensure that the
pruned weights are no longer trained, the corresponding gradient is set to
zero during the backpropagation stage.

We expect to find similar behaviour to that found by Rosenfeld et.al.[22],
namely that error scales with a power law relationship $\epsilon=cd^{-\gamma}$
within in a critical region of pruning. This would support the use of RG
theory to analyse IMP. For each layer, numerical experiments have been
performed, pruning at 1%, 5% and 10% each iteration to gain an understanding
of how pruning rate can affect the efficiency of IMP. Work by Vandersmissen
et.al.[12] has suggested that the lottery ticket hypothesis is not affected by
pruning rate so we expect to see similar behaviour but at different
resolutions. Additionally, during each experiment, the layer was only pruned
until 10% of weights remain to prevent full layer collapse. Below are the
results for these experiments:

(a) Layer 1, 1%

(b) layer 1, 5%

(c) layer 1, 10%

Figure 1: Pruning of layer 1 with an initial 50 weights

(a) layer 2, 1%

(b) layer 2, 5%

(c) layer 2, 10%

Figure 2: Pruning of layer 2 with an initial 2500 weights

(a) layer 3, 1%

(b) layer 3, 5%

(c) layer 3, 10%

Figure 3: Pruning of layer 3 with an initial 100 weights

Here we can clearly begin to see the expected behaviour in loss. As more of
the parameters are pruned, we see that the network is less able to learn how
to solve the equations accurately. From the finest level of pruning (1%) we
can see how there is a critical percentage of weights that can be pruned in
each layer before we see the power-law relationship between layer density and
the error in the solution (e.g layer 1 $\approx$ 50%). This exemplifies the
LTH as there are clearly redundant weights which can be removed without
negatively impacting on accuracy. This is because other weights in the network
are able to account for the lack of weights in other layers. This explains
why, when pruning layers individually, the deeper layers can withstand less
pruning before an increase in error. When weights are removed front the top of
the network, there are many layers below which can account for this. However,
when weights are removed from deeper in the network, information is being lost
from layers above which cannot be accounted for below.

Using the gradient of the graphs in the interval where the power-law
relationship is visible, the critical exponents corresponding to error when
pruning each layer individually can be observed. The case of 1% pruning is
used as this gives the finest resolution of the behaviour in each layer.

layer | $\gamma$
---|---
layer 1 | 1.36
layer 2 | 0.44
layer 3 | 1.22
Table 1: Critical exponents retaining to error as a function of density of on
individual layer.

This gives an idea of each layer’s sensitivity to pruning. For example,
pruning either the input or output layer has the most significant impact on
the accuracy of the model.

Furthermore, comparing the different pruning rate, we see that each rate shows
the same behaviour but to a different resolution. Therefore IMP should be able
to find a winning ticket for a range of pruning rates, so long as it is not
too high. This corroborates findings by Vandermissen et.al.[12].

So far, only the ”macro” effect of pruning on the error from the network one
layer at a time. The next experiment to be considered is that where all layers
are pruned together. This gives a more detailed picture of how the coupling in
the network develops during pruning, and therefore relates strongly with the
RG theory previously explained. Therefore, comparing critical exponents from
this experiment will be able to be used to give some idea of transferability.
For this experiment, at each pruning stage, the 5% of smallest weights from
the entire network are set to 0. This is the most useful case in finding
potential lottery tickets as one aims to remove as many parameters from the
network as possible without negatively affecting accuracy. Similarly to in the
previous experiments, each layer is no longer pruned after reaching 5% density
to prevent layer collapse.

As described early in section 4, this scenario directly compares to a 2D Ising
model where we considered a lattice of activations coupled by weights in the
network. The results we find for this case give a fuller picture of the
potential of IMP in finding winning tickets and the power-law scaling of error
during pruning.

Figure 4: Error of NN solving the non-linear oscillator system during IMP
pruning at 5%.

In these results, the behaviour of power law scaling within a certain critical
region can clearly be observed, which helped motivate the use of RG theory to
describe IMP. Furthermore, this corroborates findings from Rosenfeld et
al.(2021) [53] which show that different machine learning models have three
different periods during pruning. The first of these is the low-error plateau
where error is similar to that of the fully-connected network. The end of this
region is the point of interest when trying to find a winning lottery ticket.
The next stage is where the power-law scaling is observed. As stated earlier,
this is modelled by $\epsilon\approx cd^{-\gamma}$. The final segment is where
the error levels off at a high value as the network is no longer to learn
anything significant. This is called the high-error plateau
$\epsilon^{\uparrow}$. [53]

Extracting the gradient of the log-log graph for the power-law scaling section
of the pruning process gives the critical exponent for the error as a function
of density. So, for this system, the critical component $\gamma=9.61$.

###  5.2 IMP Flow

For the case where parameters were pruned from any layer in the model, we use
the procedure previously defined in (44) to obtain a value for $\sigma_{i}$ in
each layer of the DNN to gather a representation of the significance of each
layer in the pruning process.

For the Non-linear Oscillator, the $\sigma_{i}$ from each eigenvalue for each
hidden layer are as follows.

$\sigma_{1}$ | 0.313
---|---
$\sigma_{2}$ | -0.010
$\sigma_{3}$ | 0.736
Table 2: Critical exponents corresponding to magnitude remaining in each layer
of the non-linear oscillator system throughout pruning.

These results were collected from pruning 5% of the networks weights during
each iteration of IMP, however they are independent of the pruning rate.
Clearly we see that this supports the data from the previous section that
layers 1 and 3 are relevant whereas layer 2 is not relevant (using the
definition of relevant and irrelevant eigenvalues from previously).

###  5.3 Chaotic Hénon-Heiles

As previously stated, one of the most useful links between the LTH and RG
theory is the concept of universality. In application, once one successful
mask has been found to give efficient results on one network, it may be
transferred to another network without having to re-perform the
computationally expensive pruning process.

Work by Morcos et al.[18] has already shown that for certain tasks such as
computer vision, winning tickets can be transferred across different
tasks/models. Furthermore, work by Redman et.al.[20] has suggested that in the
cases where this is possible, the blocks/layers have similar relevance
($\sigma_{i}$) and can therefore be considered in the same universality class.

Therefore, in an attempt to find another model in the same universality class
as the non-linear oscillator solver analysed, we analyse a new neural network
DE solver for a Hénon-Heiles oscillator system. [54]

The Hénon-Heiles system is a chaotic system describing the non-linear planar
trajectory of body around a galactic centre. The degrees of freedom in the
system are $(x,y,p_{x},p_{y})$, representing position and momentum
respectively.

The total energy of the system is given by the following Hamiltonian [35]:

$\mathcal{H}=\frac{1}{2}(p_{x}^{2}+p_{y}^{2})+\frac{1}{2}(x^{2}+y^{2})+(x^{2}y-\frac{1}{3}y^{3})$
(49)

Therefore we can extract Hamilton’s equations:

$\begin{split}\dot{x}=\frac{\partial\mathcal{H}}{\partial p_{x}}&=p_{x}\\\
\dot{y}=\frac{\partial\mathcal{H}}{\partial p_{y}}&=p_{y}\\\
\dot{p_{x}}=-\frac{\partial\mathcal{H}}{\partial x}&=-(x+2xy)\\\
\dot{p_{y}}=-\frac{\partial\mathcal{H}}{\partial
y}&=-(y+x^{2}-y^{2})\end{split}$ (50)

As in the one-dimensional case, the loss function is taken to be the mean
squared error from Hamilton’s equations in order to conserve the Hamiltonian:

$L=\frac{1}{K}\sum_{n=0}^{K}[(\dot{\hat{x}}^{(n)}-\dot{\hat{p}}_{x}^{(n)})^{2}+(\dot{\hat{y}}^{(n)}-\dot{\hat{p}}_{y}^{(n)})^{2}+(\dot{\hat{p}}_{x}^{(n)}+\hat{x}^{(n)}+2\hat{x}^{(n)}\hat{y}^{(n)})^{2}+(\dot{\hat{p}}_{y}^{(n)}+\hat{y}^{(n)}+(\hat{x}^{(n)})^{2}-(\hat{y}^{(n)})^{2})^{2}]$
(51)

The neural network used to solve this problem is of a very similar structure
to that of the non-linear oscillator. However, it has some slight differences.
As before, two hidden layers of 50 neurons are used, but the output layer is
now 4 nodes instead of 2, to account for the extra dimension in the problem.
In order to keep the rest of the system similar, the same activation function
($\sin(\cdot)$) and optimiser (Adam) are used. With both structural
similarities and similarities in the problems being solved, it is hoped that
the two systems considered are in the same universality class.

Repeating the procedure as for the non-linear oscillator system, the IMP flow
of parameters in the new system is found. The $\sigma_{i}$ corresponding to
each eigenvalue is found to be as follows:

| Non-linear Oscillator | Hénon-Heiles Oscillator
---|---|---
$\sigma_{1}$ | 0.313 | 0.375
$\sigma_{2}$ | -0.010 | -0.012
$\sigma_{3}$ | 0.736 | 0.630
Table 3: Critical exponents corresponding to magnitude remaining in each layer
of both systems.

Clearly we can see that the two systems have the same relevant layers and
similar values for $\sigma_{i}$ in each layer. This therefore suggests that a
winning ticket can be transferred between the two models.

Plotting the error as the model is pruned, we see that it follows a similar
behaviour to that of the non-linear oscillator in that the periods of low-
error plateau, power-law scaling, and high-error plateau fall within similar
regions of density.

Figure 5: Error of NN solving Hénon-Heiles system during IMP at 5% pruning
rate

Furthermore, relating IMP to pruning, the two systems have similar values of
critical exponent $\gamma$, from equation (1), for the macro-observable error.
Therefore, according to RG theory, the two systems should exhibit the same
behaviour under the IMP operator.

###  5.4 Universality

So far, it has been observed that these two similar systems have the same
relevant layers and similar $\sigma_{i}$ and $\gamma$ values. This implies
that a winning ticket in one case can be transferred between the two systems
and still provide improved performance. However, the difference in
architecture means that the mask and initialisation cannot simply be carried
between the two networks. Therefore, we utilise a strategy as suggested by
experiments around the Elastic Lottery Hypothesis (ELH) [19].

The Elastic Lottery Hypothesis is an extension of the LTH as it suggests that
a winning ticket for a certain system can be transferred to a different system
with different architecture while still maintaining performance and improving
training time. In order to do this, winning tickets must be ”stretched” or
”squeezed” into the new architecture. For the case being considered here, the
winning ticket for the non-linear oscillator must be stretched into the wider
Hénon-Heile architecture. The suggested method for doing this, which has shown
success in other types of neural network [19], is by duplicating a block/layer
of the winning ticket to extend it. Therefore, a winning ticket for the non-
linear oscillator must have the third layer duplicated in order to be applied
to the Hénon-Heiles system.

Now using this method to stretch the winning ticket, the masks found while
pruning the non-linear oscillator are now applied to the Hénon-Heiles network.
Due to the nature of two different systems, the effect of certain
hyperparameters must also be investigated. Although hyperparameters such as
width, depth and learning rate are easily kept the same, the time span for
which the system is being solved over is also important. In the non-linear
oscillator case, the network is solving Hamilton’s Equations for 200 equally
spaced points on the time interval $[0,4\pi]$. Therefore, it would be useful
to know whether the lottery ticket for this model is applicable to solutions
over multiple time intervals. Plotting the results below, we fit a curve to
gather the general trend for each system.

Figure 6: Transferability of winning ticket to Hénon-Heiles system solving
over different integration times.

What can be seen is that a lottery ticket is most effective when applied to a
Hénon-Heiles system solving over a similar time period. This is because the
complexity and hyperparameters of the original system must have the capability
of solving the new system effectively. Solving over a longer integration time
requires a network of greater complexity and size. Hence the periods of low-
error plateau, power-law scaling, and high-error plateau are much less
visible. The optimal integration time to be used is dependent on the system
being considered. The best candidate would be one which provides a similar
error for the fully-connected network.

Comparing the results for $t=4\pi$ with existing experiments in the field of
computer vision, it is seen that transferability is not as strong in neural
network DE solvers. One potential reason for this is that the neural networks
in this case are far smaller than in convolutional networks. For example,
Resnet-50 models contain several millions of trainable parameters, whereas
simple DE solvers contain less than 10,000. Therefore, there is a more clearly
defined region for the low-error plateau as each trainable parameter is far
less significant in the entire network, and hence has less effect when
removed.

Repeating this process in the other direction, we take masks throughout the
pruning process of the Hénon-Heiles network and apply them to the non-linear
oscillator system. Due to the differing architectures of the systems, the
ticket must now be ”squeezed” as opposed to stretched into the new network.
This involves removing blocks from the lottery ticket. In this case, half of
the final layer of weights will be removed so that it outputs 2 values as
opposed to 4. Work by Chen et.al.[19] suggests that systems are not sensitive
to whether consecutive or non-consecutive blocks are removed. Therefore we
decided to remove the second and fourth blocks, as these correspond to
position and momentum in the second dimension.

Figure 7: Transferability of winning ticket to non-linear oscillator system

As we can see, the transferability works in both directions with systems
showing similar behaviour at similar densities. Again, the value of error is
dependent on the integration time. As transferability works in both
directions, it is strong evidence that the two systems are in fact in the same
universality class.

##  6 Conclusions

By comparing DNNs under IMP with extensively studied thermodynamic systems, we
have been able to motivate the use of RG theory to explain neural network
behaviour during pruning. Following the work of Redman et.al.[20], we were
able to formally show that IMP is a renormalisation group transformation. As
renormalisation group theory has been the primary method of explaining
universality across different systems sharing the same critical exponents
during a phase transition, we aimed to extend existing results in the field of
computer vision to neural network DE solvers. In the context of the winning
lottery ticket hypothesis, this involved transferring lottery tickets from a
Hamiltonian Neural Network solving Hamilton’s equations in a non-linear
oscillator system to one solving Hamilton’s equations in the Hénon-Heiles
system. Firstly, by pruning individual layers of the non-linear oscillator
system, we saw that the system is in fact over-parameterised and therefore a
winning ticket does exist to increase training time without negatively
impacting accuracy. Then pruning both the non-linear oscillator and the Hénon-
Heiles system showed that they both have the save relevant layers (shown by
$\sigma_{i}$ values). Hence work by Redman et.al. suggests that these two
systems will show transferability of winning lottery tickets. Experiments for
this showed that transferring lottery tickets preserves the pruning behaviour
in the new system. However, the success of this depends on the integration
time used in each system. As winning lottery tickets only enable a neural
network to achieve the accuracy of the fully connected network, the fully
connected network must be capable of reaching a high enough accuracy.
Therefore, transferring winning lottery tickets is most successful when the
new system is similarly over-parameterised and has a similar accuracy to the
original system. To solve for larger integration times, larger networks with
more complex architectures must be used. The issue when doing this is that
either the winning ticket must be stretched to fit the new architecture, or
the system must be pruned again to find a new winning ticket. The experiments
performed provide further supporting evidence for results found by Chen
et.al.[19] on the elastic lottery hypothesis that duplicating winning tickets
into wider network architectures can provide high accuracy during transfer.

### Future Directions

There are many steps that can be taken in furthering the knowledge of
transferability of winning lottery tickets between different neural network
models to solve DEs. Firstly, the behaviour of deeper networks and whether
this leads to more disparities in layer relevance. Currently, we have only
looked at simple networks with 2 hidden layers, but more complex systems and
longer integration times will require more complex architectures, in which
pruning will be even more important to reduce computation time. This also
enables research into transferability between models of different depths,
extending knowledge around the Elastic Lottery Hypothesis. Additionally,
considering more systems will allow for them to be categorised by their
critical exponent $\gamma$ to find different classes of ”similar” solvers. As
using RG theory to analyse pruning IMP has shown promising results, filling in
gaps in the connection between the two, such as correlation functions, could
give new insights into the pruning process.

## References

  * Schmidhuber [2015] Jürgen Schmidhuber.  Deep learning in neural networks: An overview.  _Neural Networks_ , 61:85–117, jan 2015.  doi: 10.1016/j.neunet.2014.09.003.
  * Dauphin and Bengio [2013] Yann N. Dauphin and Yoshua Bengio.  Big neural networks waste capacity, 2013.
  * Frankle and Carbin [2018] Jonathan Frankle and Michael Carbin.  The lottery ticket hypothesis: Finding sparse, trainable neural networks.  2018\.  doi: 10.48550/ARXIV.1803.03635.  URL https://arxiv.org/abs/1803.03635.
  * Vadera and Ameen [2021] Sunil Vadera and Salem Ameen.  Methods for pruning deep neural networks, 2021.
  * Lee et al. [2019] Namhoon Lee, Thalaiyasingam Ajanthan, and Philip H. S. Torr.  Snip: Single-shot network pruning based on connection sensitivity, 2019\.
  * Yu et al. [2021] Shixing Yu, Zhewei Yao, Amir Gholami, Zhen Dong, Sehoon Kim, Michael W Mahoney, and Kurt Keutzer.  Hessian-aware pruning and optimal neural implant, 2021.
  * Hayou et al. [2021] Soufiane Hayou, Jean-Francois Ton, Arnaud Doucet, and Yee Whye Teh.  Robust pruning at initialization, 2021.
  * Liu et al. [2020] Jiayi Liu, Samarth Tripathi, Unmesh Kurup, and Mohak Shah.  Pruning algorithms to accelerate convolutional neural networks for edge applications: A survey.  https://arxiv.org/abs/2005.04275, 2020.
  * Wimmer et al. [2023] Paul Wimmer, Jens Mehnert, and Alexandru Paul Condurache.  Dimensionality reduced training by pruning and freezing parts of a deep neural network: a survey.  _Artificial Intelligence Review_ , may 2023.  doi: 10.1007/s10462-023-10489-1.
  * Gale et al. [2019] Trevor Gale, Erich Elsen, and Sara Hooker.  The state of sparsity in deep neural networks, 2019.  URL https://arxiv.org/abs/1902.09574.
  * Chen et al. [2020] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Zhangyang Wang, and Michael Carbin.  The lottery ticket hypothesis for pre-trained bert networks, 2020.
  * Vandersmissen and Oramas [2023] Benjamin Vandersmissen and Jose Oramas.  Considering layerwise importance in the lottery ticket hypothesis, 2023\.
  * Chen et al. [2021a] Tianlong Chen, Jonathan Frankle, Shiyu Chang, Sijia Liu, Yang Zhang, Michael Carbin, and Zhangyang Wang.  The lottery tickets hypothesis for supervised and self-supervised pre-training in computer vision models, 2021a.
  * Michel et al. [2019] Paul Michel, Omer Levy, and Graham Neubig.  Are sixteen heads really better than one?, 2019.
  * Burkholz et al. [2022] Rebekka Burkholz, Nilanjana Laha, Rajarshi Mukherjee, and Alkis Gotovos.  On the existence of universal lottery tickets, 2022.
  * Kong et al. [2022] Meihao Kong, Jing Huo, Wenbin Li, Jing Wu, Yu-Kun Lai, and Yang Gao.  Playing lottery tickets in style transfer models, 2022.
  * Mehta [2019] Rahul Mehta.  Sparse transfer learning via winning lottery tickets, 2019.
  * Morcos et al. [2019] Ari S. Morcos, Haonan Yu, Michela Paganini, and Yuandong Tian.  One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers, 2019.
  * Chen et al. [2021b] Xiaohan Chen, Yu Cheng, Shuohang Wang, Zhe Gan, Jingjing Liu, and Zhangyang Wang.  The elastic lottery ticket hypothesis, 2021b.
  * Redman et al. [2021] William T. Redman, Tianlong Chen, Zhangyang Wang, and Akshunna S. Dogra.  Universality of winning tickets: A renormalization group perspective.  2021\.  doi: 10.48550/ARXIV.2110.03210.  URL https://arxiv.org/abs/2110.03210.
  * Goldenfeld [1992] Nigel Goldenfeld.  _Lectures on Phase Transitions and the Renormalization Group_.  Frontiers in Physics. Westview Press, 1992.  ISBN 9780201554090.
  * Rosenfeld et al. [2020] Jonathan S. Rosenfeld, Jonathan Frankle, Michael Carbin, and Nir Shavit.  On the predictability of pruning across scales, 2020.  URL https://arxiv.org/abs/2006.10621.
  * E et al. [2017] Weinan E, Jiequn Han, and Arnulf Jentzen.  Deep learning-based numerical methods for high-dimensional parabolic partial differential equations and backward stochastic differential equations.  _Communications in Mathematics and Statistics_ , 5(4):349–380, nov 2017.  doi: 10.1007/s40304-017-0117-6.  URL https://doi.org/10.1007%2Fs40304-017-0117-6.
  * Sirignano and Spiliopoulos [2018] Justin Sirignano and Konstantinos Spiliopoulos.  Dgm: A deep learning algorithm for solving partial differential equations.  _Journal of Computational Physics_ , 375:1339–1364, 2018\.  ISSN 0021-9991.  doi: https://doi.org/10.1016/j.jcp.2018.08.029.
  * Guidetti et al. [2021] Veronica Guidetti, Francesco Muia, Yvette Welling, and Alexander Westphal.  dnnsolve: an efficient nn-based pde solver, 2021.
  * Zeng et al. [2022] Shaojie Zeng, Zong Zhang, and Qingsong Zou.  Adaptive deep neural networks methods for high-dimensional partial differential equations.  _Journal of Computational Physics_ , 463:111232, 2022.  ISSN 0021-9991.  doi: https://doi.org/10.1016/j.jcp.2022.111232.  URL https://www.sciencedirect.com/science/article/pii/S0021999122002947.
  * Raissi et al. [2018] Maziar Raissi, Paris Perdikaris, and George Em Karniadakis.  Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.  _Journal of Computational Physics_ , 378, 11 2018.  ISSN 0021-9991.  doi: 10.1016/j.jcp.2018.10.045.
  * Lu et al. [2021] Lu Lu, Pengzhan Jin, Guofei Pang, Zhongqiang Zhang, and George Em Karniadakis.  Learning nonlinear operators via DeepONet based on the universal approximation theorem of operators.  _Nature Machine Intelligence_ , 3(3):218–229, mar 2021.  doi: 10.1038/s42256-021-00302-5.
  * Ba et al. [2023] Demba Ba, Akshunna S. Dogra, Rikab Gambhir, Abiy Tasissa, and Jesse Thaler.  Shaper: Can you hear the shape of a jet?, 2023.
  * Michoski et al. [2020] Craig Michoski, Miloš Milosavljević, Todd Oliver, and David R. Hatch.  Solving differential equations using deep neural networks.  _Neurocomputing_ , 399:193–212, 2020.  ISSN 0925-2312.  doi: https://doi.org/10.1016/j.neucom.2020.02.015.  URL https://www.sciencedirect.com/science/article/pii/S0925231220301909.
  * Shen et al. [2020] Xing Shen, Xiaoliang Cheng, and Kewei Liang.  Deep euler method: solving odes by approximating the local truncation error of the euler method, 2020.
  * Brunton et al. [2016] Steven L. Brunton, Joshua L. Proctor, and J. Nathan Kutz.  Discovering governing equations from data by sparse identification of nonlinear dynamical systems.  _Proceedings of the National Academy of Sciences_ , 113(15):3932–3937, mar 2016.  doi: 10.1073/pnas.1517384113.
  * Lagaris et al. [1998] I.E. Lagaris, A. Likas, and D.I. Fotiadis.  Artificial neural networks for solving ordinary and partial differential equations.  _IEEE Transactions on Neural Networks_ , 9(5):987–1000, 1998.  doi: 10.1109/72.712178.
  * Dogra [2020] Akshunna S. Dogra.  Dynamical systems and neural networks, 2020.
  * Mattheakis et al. [2022] Marios Mattheakis, David Sondak, Akshunna S. Dogra, and Pavlos Protopapas.  Hamiltonian neural networks for solving equations of motion.  _Physical Review E_ , 105(6), jun 2022.  doi: 10.1103/physreve.105.065305.  URL https://doi.org/10.1103%2Fphysreve.105.065305.
  * Greydanus et al. [2019] Sam Greydanus, Misko Dzamba, and Jason Yosinski.  Hamiltonian neural networks, 2019.
  * Berg and Nyström [2018] Jens Berg and Kaj Nyström.  A unified deep artificial neural network approach to partial differential equations in complex geometries.  _Neurocomputing_ , 317:28–41, 2018.  ISSN 0925-2312.  doi: https://doi.org/10.1016/j.neucom.2018.06.056.  URL https://www.sciencedirect.com/science/article/pii/S092523121830794X.
  * Wilson [1971] Kenneth G. Wilson.  Renormalization group and critical phenomena. i. renormalization group and the kadanoff scaling picture.  _Phys. Rev. B_ , 4:3174–3183, Nov 1971.  doi: 10.1103/PhysRevB.4.3174.
  * Costello [2011] Kevin Costello.  _Renormalization and Effective Field Theory_ , volume 170 of _Mathematical Surveys and Monographs_.  American Mathematical Society, 2011.  ISBN 9781470413972.
  * Simons [1997] Ben Simons.  Phase transitions and collective phenomena lecture notes, 1997.
  * Sullivan [2019] E Sullivan.  Universality caused: the case of renormalization group explanation.  9, 2019.  URL https://doi.org/10.1007/s13194-019-0260-x.
  * Zhuang et al. [2020] Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He.  A comprehensive survey on transfer learning, 2020.
  * Dogra and Redman [2020] A. S. Dogra and W. T. Redman.  Optimizing neural networks via koopman operator theory.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 33, 2020.
  * Elesedy et al. [2021] Bryn Elesedy, Varun Kanade, and Yee Whye Teh.  Lottery tickets in linear models: An analysis of iterative magnitude pruning, 2021.
  * Maene et al. [2021] Jaron Maene, Mingxiao Li, and Marie-Francine Moens.  Towards understanding iterative magnitude pruning: Why lottery tickets win, 2021.
  * Han et al. [2018] Jiequn Han, Arnulf Jentzen, and Weinan E.  Solving high-dimensional partial differential equations using deep learning.  _Proceedings of the National Academy of Sciences_ , 115(34):8505–8510, aug 2018.  doi: 10.1073/pnas.1718942115.
  * Mattey and Ghosh [2022] Revanth Mattey and Susanta Ghosh.  A novel sequential method to train physics informed neural networks for allen cahn and cahn hilliard equations.  _Computer Methods in Applied Mechanics and Engineering_ , 390:114474, feb 2022.  doi: 10.1016/j.cma.2021.114474.
  * Hornung et al. [2020] Fabian Hornung, Arnulf Jentzen, and Diyora Salimova.  Space-time deep neural network approximations for high-dimensional partial differential equations, 2020.
  * Taylor [2005] John R. Taylor.  _Classical Mechanics_.  University Science Books Mill Valley, 2005.
  * [50] https://github.com/MathePhysics/M4R_23_epg.
  * Dogra et al. [2023] A. S. Dogra, J. B. Lai, and M. Peev.  Neural network differential equation solvers allow unsupervised error analysis and correction.  _arXiv_ , 2023.
  * Dogra [2023] A. S. Dogra.  v tangent kernels.  _arXiv_ , 2023.
  * Rosenfeld et al. [2021] Jonathan S Rosenfeld, Jonathan Frankle, Michael Carbin, and Nir Shavit.  On the predictability of pruning across scales.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 9075–9083. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/rosenfeld21a.html.
  * Henon and Heiles [1964] Michel Henon and Carl Heiles.  The applicability of the third integral of motion: Some numerical experiments.  69:73, February 1964.  doi: 10.1086/109234.