Separable Hamiltonian Neural Networks

1st Zi-Yu Khoo
School of Computing

National University of Singapore

Singapore, Singapore

2nd Jonathan Sze Choong Low
Singapore Institute of Manufacturing Technology

Agency for Science, Technology and Research

Singapore, Singapore

3rd Stéphane Bressan
School of Computing

National University of Singapore

Singapore, Singapore

Journal of  Class Files, Vol. 14, No. 8, August 2015
Shell et al.: Bare Demo of IEEEtran.cls for IEEE Journals

The modelling of dynamical systems from discrete observations is a challenge faced by modern scientific and engineering data systems. Hamiltonian systems are one such fundamental and ubiquitous class of dynamical systems. Hamiltonian neural networks are state-of-the-art models that unsupervised-ly regress the Hamiltonian of a dynamical system from discrete observations of its vector field under the learning bias of Hamilton’s equations.
Yet Hamiltonian dynamics are often complicated, especially in higher dimensions where the state space of the Hamiltonian system is large relative to the number of samples. A recently discovered remedy to alleviate the complexity between state variables in the state space is to leverage the additive separability of the Hamiltonian system and embed that additive separability into the Hamiltonian neural network.
Following the nomenclature of physics-informed machine learning, we propose three separable Hamiltonian neural networks. These models embed additive separability within Hamiltonian neural networks. The first model uses additive separability to quadratically scale the amount of data for training Hamiltonian neural networks. The second model embeds additive separability within the loss function of the Hamiltonian neural network. The third model embeds additive separability through the architecture of the Hamiltonian neural network using conjoined multilayer perceptions. We empirically compare the three models against state-of-the-art Hamiltonian neural networks, and demonstrate that the separable Hamiltonian neural networks, which alleviate complexity between the state variables, are more effective at regressing the Hamiltonian and its vector field.

Hamiltonian dynamical systems, physics-informed neural networks, physics properties, system symmetries, additive separability

§ INTRODUCTION

Modelling dynamical systems is a core challenge for science and engineering. The movement of a pendulum, the wave function of a quantum-mechanical system, the movement of fluids around the wing of a plane, the weather patterns under climate change, and the populations forming an ecosystem are spatio-temporal behaviours of physical phenomena described by dynamical systems. Hamiltonian systems <cit.> are a class of dynamical systems governed by Hamilton's equations, which indicate conservation of the Hamiltonian value of the system.

A recent advancement in the modelling of dynamical systems is Hamiltonian neural networks <cit.>, which are physics-informed neural networks with learning biases given by Hamilton's equations and their corollaries <cit.>. Hamiltonian neural networks are universal function approximators <cit.> capable of modelling non-linear multivariate functions <cit.>. They use a learning bias <cit.> based on physics information regarding Hamilton's equations <cit.> to aid the neural network in converging towards solutions that adhere to physics laws <cit.>. Hamiltonian neural networks unsupervised-ly regress the vector field and the Hamiltonian of a dynamical system from discrete observations of its state space or vector field under the learning bias of Hamilton's equations, and outperform their physics-uninformed counterparts in doing so <cit.>.

However, Hamiltonian dynamics are often complicated and chaotic, especially in higher dimensional systems such as the Toda Lattice and Henon Heiles systems. The state space of the Hamiltonian system is large relative to the number of samples. A redeeming feature of these systems is their additive separability. As highlighted by Gruver et al. in the ICML 2022 spotlight paper, additive separability "allowed the physics-informed neural network to avoid[...] artificial complexity from its coordinate system" (the input variables) and improve its performance <cit.>. This motivates further informing Hamiltonian neural networks of additive separability to alleviate the complexity between state variables of Hamiltonian systems.

The main technical contribution of this work is the embedding of additive separability into a Hamiltonian neural network to regress a Hamiltonian and vector field. We propose three Hamiltonian neural networks that independently embed additive separability using three modes of biases. We call this family of Hamiltonian neural networks, separable Hamiltonian neural networks. The three separable Hamiltonian neural networks follow the nomenclature of Karniadakis et al. <cit.> and embed additive separability using three modes of biases: observational bias, learning bias, and inductive bias. The first model embeds an observational bias by training on newly generated data that embody separability. The second model embeds a learning bias through the loss function of the Hamiltonian neural network. The third model embeds an inductive bias through the architecture of the Hamiltonian neural network by means of conjoined multilayer perceptrons.

We empirically evaluate the performance of the proposed models against a baseline Hamiltonian neural network on a variety of representative additively separable Hamiltonian systems. We compare their performances in regressing an additively separable Hamiltonian and a vector field.

In this paper, Section <ref> presents the necessary background on dynamical, Hamiltonian and separable Hamiltonian systems. Section <ref> synthesises the related work and positions the main contribution. Section <ref> introduces the proposed separable Hamiltonian neural networks. Section <ref> compares the proposed models against the baseline Hamiltonian neural network in regressing additively separable Hamiltonians and vector fields. Section <ref> concludes the paper.

§ BACKGROUND

§.§ Dynamical Systems and Hamiltonian Systems

Dynamical systems theory <cit.> studies the temporal dynamics, or time evolution, of dynamical systems. The phase or state space of a dynamical system is a $\mathbb{R}^{2\times n}$ multidimensional space which represents all possible states of a dynamical system comprising the combination of the system's $2 \times n$ state variables <cit.>, also called degrees of freedom, or parameters. Without loss of generality, we consider autonomous, time-independent systems. The dynamics of an autonomous system are captured by the vector field <cit.> of dimension $\mathbb{R}^{2\times n}$, formed by the time derivatives of the state variables.

A Hamiltonian system <cit.> is a dynamical system characterised by a smooth, real-valued scalar function of the state variables <cit.> called the Hamiltonian function or simply the Hamiltonian, $H \in \mathbb{R}$. The Hamiltonian system is governed by Hamilton's equations <cit.>, a system of $2 \times n$ differential equations given in Equation <ref> that define the vector field $F(x, y) = \left( \dv{x}{t}, \dv{y}{t}\right)$ <cit.>. A Hamiltonian with $2\times n$ state variables has a dimension, or axis, of $n$. Conventionally, the set of variables can be evenly split into two sets called generalised variables, noted $\vec{x}$ for the position, and $\vec{y}$ for momentum.

\begin{align}
\dv{\vec{x}}{t} = \pdv{H(\vec{x},\vec{y})}{\vec{y}}, \ \ \ \ \dv{\vec{y}}{t} = -\pdv{H(\vec{x},\vec{y})}{\vec{x}} . \label{eqn:defnHamiltonian}
\end{align}

A classic example of a Hamiltonian system is the non-linear pendulum. Figure <ref> shows the vector field of a non-linear pendulum in its two-dimensional phase space with state variables position (angle when oscillating in a plane) and momentum (mass multiplied by velocity).
The vector field is formed by the time derivatives of the state variables. For a non-linear pendulum of unitary mass, the Hamiltonian is the sum of kinetic and potential energy.
The heatmap in Figure <ref> shows the value of the Hamiltonian in the phase space.

(a) Non-linear pendulum vector field (black arrows) and Hamiltonian (heatamap), (b) Random samples of the non-linear pendulum vector field.

§.§ Separable Hamiltonian Systems

A Hamiltonian system is separable if the Hamiltonian can be separated into additive terms, each dependent on either $\vec{x}$ or $\vec{y}$, where $\vec{x}$ and $\vec{y}$ are disjoint subsets of the state variables of a Hamiltonian system <cit.>. The separable Hamiltonian is defined $H(\vec{x},\vec{y}) = T(\vec{x}) + V(\vec{y})$ where $T$ and $V$ are arbitrary functions <cit.>. Furthermore, the mixed partial derivative of the separable Hamiltonian is zero following Equation <ref>.

\begin{equation}
\pdv{H}{x}{y} = \pdv{}{x}\left( \pdv{(T(\vec{x}) + V(\vec{y}))}{y}  \right) =  \pdv{}{x} \left( \pdv{V(\vec{y})}{y}\right) = 0 . \label{eqn:addsep_hamiltonian_mixedpartial}
\end{equation}

§.§ Examples of Separable Hamiltonian Systems

For illustration and comparative empirical evaluation of the models presented, we consider eight Hamiltonian systems shown in Table <ref>, of which five are classical and mechanical and three are abstract.
These allow the models to demonstrate their performance in predicting a range of Hamiltonian dynamics comprising different functions, different values of $n$, and well-behaved or chaotic dynamics, from observations.

Equations and $n$ of each Hamiltonian system. Abstract systems are indicated with an asterisk beside their names.

System     Equation     $n$

Non-linear Pendulum     $H(x,y) = \frac{y^2}{2} + (1-\cos{x})$     1

Trigonometric*     $H(x,y) = \sin^2(x) +\cos^2(y)$     1

Arctangent*     $H(x,y) = \arctan(x^2) + \arctan(y^2)$     1

Logarithmic*     $H(x,y) = \frac{2}{3}\times x - \log(x)+\frac{4}{3}\times y - \log{y}$     1

Anisotropic Oscillator     $H(x_1,x_2,y_1,y_2) = \sqrt{y_1^2+y_2^2+1} + \frac{x_1^2+x_2^2}{2} + \frac{0\times x_1^4 +0.05\times x_2^4}{4}$     2

Henon Heiles     $H(x_1,x_2,y_1,y_2) = \frac{y_1^2 + y_2^2}{2} + \frac{x_1^2 + x_2^2}{2} + x_1^2 \times x_2 - \frac{x_2^3}{3}$     2

Toda Lattice     $H(x_1,x_2,x_3,y_1,y_2,y_3) = \frac{y_1^2+y_2^2+y_3^2}{2}+\exp(x_1-x_2)+\exp(x_2-x_3)+\exp(x_3-x_1)-3$     3

Coupled Oscillator     $H(x_1,x_2,x_3,y_1,y_2,y_3) = \frac{y_1^2 + y_2^2 + y_3^2}{2} +  \frac{(x_2-x_1)^2}{2} + \frac{(x_3-x_2)^2}{2} $     3

§ RELATED WORK

§.§ Predicting Function Values and Vector Fields

There are multiple statistical methods to regress correlated vector-valued functions <cit.>.
Hastie et al. <cit.> addressed the regressing of vector fields using multiple output regression and machine learning. State-of-the-art works regress the vector field of a dynamical system using neural networks that model ordinary <cit.> and partial <cit.> differential equations.

For the regressing of Hamiltonian dynamical systems, Bertalan et al. <cit.> and Greydanus et al. <cit.> independently use physics-informed machine learning methods to regress the value of the Hamiltonian from multiple evenly-spaced samples along multiple Hamiltonian trajectories. Our work emulates theirs, by embedding Hamilton's equations within the loss function of a neural network to regress the Hamiltonian <cit.> and using automatic differentiation of the regressed Hamiltonian to yield the regressed vector field <cit.>. However, our work uses instantaneous observations of the Hamiltonian vector field, which are sufficient to train the Hamiltonian neural network.

Recent advancements in regressing Hamiltonian vector fields use neural ordinary differential equations <cit.> and leverage the symplectic property of the Hamiltonian. They use symplectic integration to regress the Hamiltonian vector field. Some further leverage Hamiltonian separability <cit.> by using a Leapfrog integrator. Others additionally require the Hamiltonian to be mechanical <cit.>. Neural ordinary differential equations-based works require trajectories of the Hamiltonian system as input while our model only requires instantaneous observations of the Hamiltonian vector field.

§.§ Biases in Neural Networks

Karniadakis et al. focus on three modes of biasing a regression model: observational bias, learning bias, and inductive bias <cit.>.
Observational biases are introduced directly through data that embody the underlying physics, or carefully crafted data augmentation procedures. With sufficient data to cover the input domain of a regression task, machine learning methods have demonstrated remarkable power in achieving accurate interpolation between the dots <cit.>.
Learning biases are soft constraints introduced by appropriate loss functions, constraints and inference algorithms that modulate the training phase of a machine learning model to explicitly favour convergence towards solutions that adhere to the underlying physics <cit.>.
Inductive biases are prior assumptions incorporated by tailored interventions to a machine learning model architecture, so regressions are guaranteed to implicitly and strictly satisfy a set of given physical laws <cit.>.

Hamiltonian neural networks leverage learning biases and use Hamilton's equations as soft constraints in the loss function of the neural network to favour convergence toward the Hamiltonian<cit.>.

Seminal work on embedding additive separability within neural networks utilised block diagonal matrices to regress elements of an additively separable finite element problem <cit.>.

Recently, Zhong et al. <cit.> leveraged the additive separability of mechanical Hamiltonians as an inductive bias to design neural ordinary differential equations.
Gruver et al. <cit.> empirically examined these neural networks and found that their improved generalization resulted from the bias of a second-order structure <cit.>, which arose from the additive separability of the modelled mechanical Hamiltonian and "allowed the physics-informed neural network to avoid[...] artificial complexity from its coordinate system (the input variables)" and improve its performance <cit.>. Our work also exploits the additive separability of the modelled Hamiltonian.

Our work
incorporates knowledge regarding the additive separability of the Hamiltonian function within the Hamiltonian neural network in the style of Karniadakis' physics-informed machine learning <cit.> so that regressions sought of the Hamiltonian function and vector field are guaranteed to implicitly or explicitly satisfy this separability.

§ METHODOLOGY

Four Hamiltonian neural networks are compared for the task of regressing the Hamiltonian and vector field of a Hamiltonian system. One, the baseline, is uninformed of the additive separability of the Hamiltonian system. Three proposed models are informed via observational, learning and inductive biases respectively. Subsections <ref>, <ref>, <ref>, and <ref> introduce the four models. Section <ref> empirically compares the models on their abilities to perform the task.

§.§ The Baseline Hamiltonian Neural Network

We adapt Hamiltonian neural networks (HNNs) <cit.> (leftmost, Figure <ref>) for the task of regressing the Hamiltonian and vector field of a Hamiltonian system from random samples of the vector field. Hamiltonian neural networks inform a neural network that a system is Hamiltonian by embedding a Hamiltonian learning bias into the neural network.

Architecture of the baseline Hamiltonian neural network and proposed separable Hamiltonian neural network with observational bias (left), proposed separable Hamiltonian neural network with learning bias (centre), and proposed separable Hamiltonian neural network with inductive bias (right).

Equation <ref> defines the loss function of the Hamiltonian neural network. $f_0$ is an arbitrary pinning term that "pins" the regressed Hamiltonian to one among several solutions that are modulo and additive constant, and reduces the search space for convergence.
$f_1$ and $f_2$ are Hamilton's equations corresponding to Equation <ref>. $f_0$, $f_1$ and $f_2$ introduce biases that favour the convergence of the Hamiltonian neural network toward the underlying physics of the regressed Hamiltonian. Equation $f_*$ defines the loss function of the neural network as a linear combination of equations $f_0$ to $f_2$. $c_k$ is the coefficient of each $f_k$. One can assume that $c_k = 1$ although additional knowledge of the system can be used to emphasise any $f_k$ <cit.>.

\begin{align}
f_0 &= \left( \hat{H}(\vec{x_0},\vec{y_0}) - H_0 \right)^2, &f_1 &= \left(\pdv{\hat{H}}{\vec{y}} - \dv{\vec{x}}{t} \right)^2,& \notag \\
f_2 &= \left(\pdv{\hat{H}}{\vec{x}} + \dv{\vec{y}}{t} \right)^2, & f_*&(\vec{x},\vec{y},\dv{\vec{x}}{t},\dv{\vec{y}}{t};w) = \sum^2_{k=0} c_k f_k.
\label{eqn:MLPloss1}
\end{align}

To perform the task of regressing the Hamiltonian and vector field of a $n$ dimensional Hamiltonian system, the Hamiltonian neural network uses instantaneous, random samples of the $2\times n$ state variables and the $2\times n$ vectors, like those seen in Figure <ref>. The state variables are input to the model. The output of the model is the regression surrogate, $\hat{H}$, which is an estimator of the Hamiltonian $H$. The training is supervised through the loss function, which uses the $2\times n$ vectors corresponding to the $2\times n$ state variables that were input to the model. Via gradient descent, $f_*$ is minimised. The derivative of the surrogate $\hat{H}$ at all input state variables is the surrogate vector field. It is computed via automatic differentiation <cit.> of the Hamiltonian neural network.

The separable Hamiltonian neural networks introduced in subsections <ref>, <ref> and <ref> adopt the Hamiltonian learning bias of the Hamiltonian neural network and further embed biases regarding additive separability. They perform the task of regressing the Hamiltonian and vector field in the same way.

§.§ Embedding a Separability Observational Bias within a Hamiltonian Neural Network

The baseline Hamiltonian neural network can be informed of additive separability by embedding an observational bias into the Hamiltonian neural network. Given data of the vector field comprising instantaneous state variables and vectors, additive separability is used to quadratically scale the amount of data. Training the Hamiltonian neural network on the new data embeds the observational bias and allows the model to regress a surrogate Hamiltonian that reflects the additive separability of the data. The model, with an embedded observational bias, is a separable Hamiltonian neural network.

Given original data comprising samples of tuples $(\vec{x},\vec{y})$, new samples are generated. Additive separability means the generation comprises new combinations of the $\vec{x}$ and $\vec{y}$ from the original data. As the Hamiltonian is additively separable, the first derivatives of the new samples are dependent solely on $\vec{x}$ or $\vec{y}$, and can be inferred from the original samples at the respective values of $\vec{x}$ or $\vec{y}$. The amount of data available to train the Hamiltonian neural network has increased.

Consider original data comprising two samples $(x_1,y_1)$ and $(x_2,y_2)$ in blue in Figure <ref>. With additive separability, two new samples created are $(x_1,y_2)$ and $(x_2,y_1)$ in red. Figure <ref> shows that as more samples from the original data (in blue) are available, quadratically more new samples (in red) can be created. Generally, up to $N\times(N-1)$ new samples can be created from original data comprising $N$ samples.

(a) $N\times(N-1) = 2$ new samples are created from $N=2$ samples, (b) $N\times(N-1) = 12$ new samples are created from $N=4$ samples.

The observational bias creates more data to improve coverage of the input domain of the Hamiltonian regression task, and the regressed surrogate Hamiltonian reflects the additive separability of the data. This improves regression performance but increases the time taken for forward and backward propagation of the separable Hamiltonian neural network. In selecting the optimal number of samples, there is a trade-off between regression performance and training time.

§.§ Embedding a Separability Learning Bias within a Hamiltonian Neural Network

The baseline Hamiltonian neural network can be informed of additive separability by embedding a learning bias. The resulting separable Hamiltonian neural network with learning bias (centre, Figure <ref>) favours convergence towards a surrogate Hamiltonian that is additively separable.

$f_3$ in Equation <ref> is the mixed partial derivative of the surrogate Hamiltonian $\hat{H}$ corresponding to Equation <ref>. $f^{sep}_*$ is the loss function of the separable Hamiltonian neural network with learning bias. It is a linear combination of equations $f_0$ to $f_3$ from Equation <ref> and <ref>. It introduces a bias that favours convergence of the Hamiltonian neural network toward a surrogate Hamiltonian that is additively separable.

\begin{align}
f_3 &= \left( \pdv{\hat{H}}{x}{y} \right)^2 \forall x \in \vec{x}, y \in \vec{y}, & f^{sep}_*= \sum^3_{k=0} c_k f_k .
\label{eqn:MLPloss2}
\end{align}

A larger $c_3$ allows the separable Hamiltonian neural network to emphasise $f_3$ and additive separability of the surrogate Hamiltonian, but may also decrease the emphasis of $f_0$, $f_1$ and $f_2$, presenting a trade-off in the optimal value of $c_3$.

§.§ Embedding a Separability Inductive Bias within a Hamiltonian Neural Network

A baseline Hamiltonian neural network can be informed of additive separability by embedding an inductive bias. The resulting separable Hamiltonian neural network with inductive bias (rightmost, Figure <ref>) regresses a surrogate Hamiltonian that implicitly and strictly satisfies additive separability.

The proposed separable Hamiltonian neural network with inductive bias is not fully-connected. The model comprises two smaller neural networks with the same number of layers, conjoined only at the output layer. Each smaller conjoined neural network has one output. Their sum (indicated by the dotted lines in Figure <ref>) is the surrogate Hamiltonian. The architecture of the proposed separable Hamiltonian neural network ensures additive separability as each smaller conjoined neural network has an input of either $\vec{x}$ or $\vec{y}$, and is, therefore, a function of either $\vec{x}$ or $\vec{y}$. During forward propagation, the sum of the conjoined neural networks ensures the surrogate Hamiltonian is always the sum of two independent functions of $\vec{x}$ and $\vec{y}$.

The mixed partial derivative of the surrogate Hamiltonian is by design always zero. Additive separability is strictly satisfied. The conjoined neural networks are trained simultaneously in parallel to minimise training time.

The separable Hamiltonian neural network has two smaller conjoined neural networks that can be trained consecutively or simultaneously in parallel. Furthermore, the summation layer of the separable Hamiltonian neural network can utilise a simple sum or introduce weights and biases. The different implementations for the separable Hamiltonian neural network with inductive bias present a trade-off in finding its optimal implementation.

§ PERFORMANCE EVALUATION

The four models are compared on the task of regressing the Hamiltonian and vector field of a Hamiltonian system. To reach the comparison, the proposed separable Hamiltonian neural networks must first be empirically studied and optimised. Thereafter, the baseline Hamiltonian neural network and three optimised models can be compared.

In Experiment 1, the optimal number of samples to create from the original data for the separable Hamiltonian neural network with observational bias is empirically studied and identified. In Experiment 2, the optimal value of $c_3$ for the separable Hamiltonian neural network with learning bias is empirically studied and identified. In Experiment 3, the optimal implementation for the separable Hamiltonian neural network with inductive bias is empirically studied and identified. Finally, in Experiment 4, the baseline model and three optimised separable Hamiltonian neural networks are compared on the task of regressing the Hamiltonian and vector field, and their respective training times.

In finding the optimal implementations of the three informed variants, we only compare their performance on the task of regressing the vector field. This is because regressing the Hamiltonian involves finding the real-valued sum of the integrated vector field. Errors made by the models in regressing the vector field may be cancelled out when regressing the Hamiltonian. Therefore, a model that regresses the Hamiltonian well may not regress the vector field well, but a model that regresses the Hamiltonian vector field well can also regress the Hamiltonian well. For completeness, and to demonstrate this phenomenon, we regress and present results for both the Hamiltonian and the vector field in Experiment 4.

For the regression of the Hamiltonian, the performance of the models is measured by the absolute or L1-error between the surrogate Hamiltonian of a model and the true Hamiltonian from test data. The absolute error is computed following Equation <ref>. For the regression of the vector field, the performance of the models is measured by the vector error between the derivative of the surrogate Hamiltonian of a model and the true vector field from test data. The vector error is computed following Equation <ref> <cit.>. $\hat{v}$ is the regressed vector and $v$ is the true vector. The test data set comprises $d = s^{2n}$ vectors, with $s=10$ evenly spaced states in each dimension of the phase space for each system.

\begin{equation}
E_H =\frac{1}{d} \sum^{d}_{k=1} ||\hat{H}_k-H_k||_1,  \label{eqn:abserror}
\end{equation}

\begin{equation}
E_V =\frac{1}{d} \sum^{d}_{k=1} \frac{||\hat{v}_k-v_{k}||_2}{||v_{k}||_2} .
\label{eqn:vectorerror}
\end{equation}

All experiments are evaluated over the Hamiltonian systems shown in Table <ref>. The general experimental setup for all models in all experiments is as follows. Training data comprising $512$ samples of the state variables and vector field are generated uniformly at random within the sampling domain for each Hamiltonian system. The sampling domains are shown in columns 2 and 3 of Table <ref>. The samples comprise tuples of the state variables $(\vec{x},\vec{y})$ and their corresponding vectors or time derivatives. The models to be experimented on are designed with two hidden layers, an Adam optimizer and softplus activation. In training, 20% of the training data is set aside as validation data for a dynamic stopping criterion using validation-based early stopping <cit.> and a batch size of 80 is used. All models have an input layer with width $2\times n$, two hidden layers with width shown in columns 6 and 8 of Figure <ref>, and one output layer with width one. Samples of state variables are input to the models. Samples of the vector field are used in the loss function of the models. All models are trained until convergence. In order to find an optimal bias-variance trade-off, the training will terminate if there is no decrease in the validation loss for 4,000 epochs in a row. The output of the models is the surrogate Hamiltonian. The surrogate vector field is computed via automatic differentiation of the surrogate Hamiltonian with respect to its inputs $\vec{x}$ and $\vec{y}$. It is equivalent to the vector field following Equation <ref>.
All models are trained in  on a GeForce GTX1080 GPU with 32 GB RAM. The complete code in Python and results for the models discussed are available at <github.com/zykhoo/SeparableNNs>. All experiments are repeated for 20 random seeds.

Configuration of the Hamiltonian neural network (HNN), separable Hamiltonian neural network with observational bias (sHNN-O), separable Hamiltonian neural network with learning bias (sHNN-L), and separable Hamiltonian neural network with inductive bias (sHNN-I) for each experiment.

2cSampling domain of     2c     2cHNN, sHNN-O and sHNN-L     2csHNN-I

Experiment     $x$     $y$     $n$     Learning Rate     Width     Total Params     Width     Total Params

Non-linear Pendulum     $[-2\pi, 2\pi]$     $[-1.2, 1.2]$     1     0.001     16 <cit.>     337     11     332

Trigo     $[-2,2]$     $[-2,2]$     1     0.001     16     337     11     332

Arctangent     $[-2,2]$     $[-2,2]$     1     0.001     16     337     11     332

Logarithmic     $[0.75,2.25]$     $[0.5,1.5]$     1     0.001     32     1185     22     1146

Anisotropic Oscillator     $[-0.5, 0.5]$     $[-0.5, 0.5]$     2     0.001     32     1249     22     1190

Henon Heiles     $[-0.5, 0.5]$     $[-0.5, 0.5]$     2     0.01     32     1249     22     1190

Toda Lattice     $[-0.5, 0.5]$     $[-0.5, 0.5]$     3     0.01     31     1241     22     1234

Coupled Oscillator     $[-0.5, 0.5]$     $[-0.5, 0.5]$     3     0.01     31     1241     22     1234

§.§ Experiment 1: Optimising the Separable Hamiltonian Neural Network with Observational Bias

This subsection details the experimental setup to empirically study the trade-off between regression performance and regression time taken for the separable Hamiltonian neural network with observational bias, by determining the optimal number of new samples to create from data with $N$ samples.

§.§.§ Experimental Setup

With original training data of size $512$, 20% of the data is first set aside as validation data. The remaining 80% or $N=409$ of training data comprising samples $(\vec{x}_i,\vec{y}_i)$ and time derivatives $(\dv[]{\vec{x}_i}{t}, \dv[]{\vec{y}_i}{t}) \forall i \in N$, is doubled by creating new data comprising samples $(\vec{x}_i,\vec{y}_{i+1})$ with time derivatives $(\dv[]{\vec{x}_i}{t}, \dv[]{\vec{y}_{i+1}}{t}) \forall i \in N$, then appending this new data to the original data. Generally, the training data can be increased $m$ times by creating new data comprising samples $(x_i,y_{i+m})$ and with time derivatives $(\dv[]{x_i}{t}, \dv[]{y_{i+m}}{t}) \forall m=[1,k]$, then appending all $m$ sets of new samples to the original data.
The separable Hamiltonian neural network with observational bias is set up following the leftmost architecture in Figure <ref>. All details of the setup, training and evaluation follow from the description above in Section <ref>.
The separable Hamiltonian neural network is trained and evaluated for models where $k=\{1,2,3,4,5,10,20,30,40,50,100,200,300,400\}$.

§.§.§ Experimental Results
We report the vector error and training time for the separable Hamiltonian neural network with observational bias in Tables <ref> and <ref>.

Vector error $E$ and standard error (in brackets) of the separable Hamiltonian neural network with observational bias with  $k=\{1,2,3,4,5,10,20,30,40,50,100,200,300,400\}$.

Experiment     $k = 1$     $k = 2$     $k = 3$     $k = 4$     $k = 5$

Non-linear Pendulum      7.27E-03 (1.38E-03)     6.80E-03 (1.68E-03)     5.15E-03 (7.06E-04)     4.38E-03 (2.96E-04)     4.41E-03 (4.53E-04)

Trigo     9.47E-03 (8.36E-04)     8.17E-03 (7.17E-04)     7.80E-03 (6.11E-04)     7.39E-03 (6.31E-04)     6.36E-03 (3.14E-04)

Arctangent     3.81E-03 (2.51E-04)     3.55E-03 (2.49E-04)     2.92E-03 (1.83E-04)     2.89E-03 (2.27E-04)     2.54E-03 (2.36E-04)

Logarithmic     3.36E-03 (1.75E-04)     3.02E-03 (1.72E-04)     2.77E-03 (1.54E-04)     2.65E-03 (1.33E-04)     2.43E-03 (1.47E-04)

Anisotropic Oscillator      4.80E-03 (3.33E-04)     4.53E-03 (4.52E-04)     3.60E-03 (2.53E-04)     2.65E-03 (1.84E-04)     3.29E-03 (3.19E-04)

Henon Heiles      7.62E-03 (7.38E-04)     5.48E-03 (3.78E-04)     6.20E-03 (6.77E-04)     4.52E-03 (6.84E-04)     4.53E-03 (4.21E-04)

Toda Lattice      7.11E-03 (5.14E-04)     4.65E-03 (3.57E-04)     4.36E-03 (3.38E-04)     3.65E-03 (2.65E-04)     4.04E-03 (4.78E-04)

Coupled Oscillator     6.97E-03 (4.14E-04)     5.35E-03 (5.00E-04)     4.69E-03 (2.96E-04)     4.20E-03 (5.15E-04)     3.74E-03 (3.44E-04)

Experiment     $k = 10$     $k = 20$     $k = 30$     $k = 40$     $k = 50$

Non-linear Pendulum     4.26E-03 (8.72E-04)     5.35E-03 (1.13E-03)     3.24E-03 (5.53E-04)     4.99E-03 (2.10E-03)     3.40E-03 (4.07E-04)

Trigo      5.40E-03 (5.56E-05)     4.97E-03 (8.53E-04)     4.85E-03 (8.15E-05)     4.38E-03 (4.32E-04)     3.82E-03 (2.32E-04)

Arctangent     2.17E-03 (1.74E-04)     2.28E-03 (3.25E-04)     1.69E-03 (1.81E-04)     2.10E-03 (2.00E-04)     1.30E-03 (1.46E-04)

Logarithmic     1.35E-03 (1.96E-04)     1.25E-03 (3.31E-04)     1.04E-03 (2.54E-04)     1.23E-03 (1.50E-05)     7.54E-04 (2.97E-04)

Anisotropic Oscillator     1.91E-03 (2.36E-04)     1.96E-03 (2.75E-04)     1.41E-03 (1.40E-04)     2.44E-03 (1.33E-04)     2.07E-03 (2.76E-04)

Henon Heiles     2.63E-03 (6.49E-04)     1.93E-03 (9.93E-05)     1.73E-03 (2.30E-04)     2.50E-03 (4.19E-06)     2.91E-03 (1.06E-03)

Toda Lattice     2.28E-03 (3.87E-04)     2.29E-03 (4.94E-04)     3.12E-03 (8.54E-04)     3.36E-03 (4.20E-04)     3.08E-03 (1.60E-03)

Coupled Oscillator      4.73E-03 (1.21E-03)     3.49E-03 (7.17E-04)     2.98E-03 (5.56E-04)     2.47E-03 (4.62E-04)     2.01E-03 (5.88E-05)

Experiment     $k = 100$     $k = 200$     $k = 300$     $k = 400$

Non-linear Pendulum     2.33E-03 (4.31E-04)     1.47E-03 (7.34E-04)     1.94E-03 (9.91E-04)     9.15E-04 (7.87E-05)

Trigo     4.11E-03 (9.36E-04)     3.92E-03 (2.97E-04)     1.52E-03 (3.49E-04)     1.13E-03 (3.79E-04)

Arctangent     1.32E-03 (3.00E-04)     7.36E-04 (1.04E-04)     5.49E-04 (3.11E-05)     6.78E-04 (1.81E-04)

Logarithmic     4.59E-04 (1.40E-04)     3.71E-04 (1.20E-05)     6.43E-04 (1.01E-04)     7.50E-04 (9.63E-05)

Anisotropic Oscillator      1.23E-03 (TBC)     1.46E-03 (TBC)     1.34E-03 (TBC)     1.70E-03 (TBC)

Henon Heiles     2.51E-03 (3.73E-04)     2.01E-03 (3.04E-04)     2.10E-03 (TBC)     2.15E-03 (TBC)

Toda Lattice     2.46E-03 (TBC)     2.36E-03 (TBC)     2.74E-03 (TBC)     4.05E-02 (TBC)

Coupled Oscillator     1.58E-03 (TBC)     2.47E-03 (TBC)     1.89E-03 (TBC)     2.03E-03 (TBC)

Time taken in seconds and number of epochs (in brackets) until convergence for the separable Hamiltonian neural network with observational bias   $k=\{1,2,3,4,5,10,20,30,40,50,100,200,300,400\}$.

Experiment     $k = 1$     $k = 2$     $k = 3$     $k = 4$     $k = 5$

Non-linear Pendulum      736.63 (12558)     818.47 (9816)     1098.08 (9724)     1357.76 (10084)     1423.56 (8923)

Trigo      802.01 (13269)     899.80 (10342)     1182.96 (10388)     1319.57 (9627)     1574.60 (9888)

Arctangent     821.36 (13283)     880.43 (10445)     1016.77 (9104)     1108.92 (8203)     1244.61 (7874)

Logarithmic      435.95 (7355)     525.97 (6211)     597.76 (5610)     717.89 (5481)     845.54 (5393)

Anisotropic Oscillator     350.13 (5630)     469.20 (5445)     584.54 (5201)     754.54 (5392)     826.12 (5012)

Henon Heiles     436.05 (6690)     589.43 (6572)     648.91 (5662)     789.63 (5670)     952.46 (5654)

Toda Lattice     583.22 (9296)     740.80 (8446)     970.64 (8229)     1065.19 (7380)     1293.25 (7582)

Coupled Oscillator     423.40 (7275)     638.25 (7658)     708.63 (6567)     895.03 (6742)     976.32 (6119)

Experiment     $k = 10$     $k = 20$     $k = 30$     $k = 40$     $k = 50$

Non-linear Pendulum      1816.62 (6198)     3172.87 (5353)     5034.65 (5814)     5677.65 (5290)     6637.13 (4800)

Trigo      2821.06 (10259)     3906.90 (7540)     4750.75 (6495)     6390.46 (6378)     6957.24 (5354)

Arctangent     1955.68 (6207)     3938.59 (6674)     4581.49 (5372)     6035.19 (5190)     7235.74 (5146)

Logarithmic      1293.17 (4426)     2228.54 (4432)     3122.86 (4108)     4371.33 (4222)     5577.60 (4092)

Anisotropic Oscillator     1206.20 (4202)     2503.29 (4283)     3314.66 (4238)     4551.16 (4138)     5577.62 (4161)

Henon Heiles      1584.12 (4680)     2994.16 (4282)     4192.28 (4417)     5458.34 (4474)     6028.65 (4260)

Toda Lattice      1785.62 (5999)     3557.63 (6579)     4088.23 (5177)     5469.58 (5296)     5653.24 (4479)

Coupled Oscillator     1974.52 (6268)     3062.54 (4920)     4112.55 (4446)     5547.47 (4687)     6387.00 (4521)

Experiment     $k = 100$     $k = 200$     $k = 300$     $k = 400$

Non-linear Pendulum     10723.34 (4400)     22522.65 (4589)     31044.18 (4217)     40573.54 (4132)

Trigo      10720.68 (4464)     22595.35 (4682)     31543.92 (4348)     40967.67 (4226)

Arctangent     11544.85 (4723)     21399.34 (4352)     31693.65 (4290)     43807.09 (4458)

Logarithmic      10094.42 (4038)     19920.57 (4036)     30200.35 (4082)     39784.93 (4016)

Anisotropic Oscillator     9934.18 (4016)     19956.70 (4040)     29440.12 (4016)     39049.51 (4004)

Henon Heiles     10250.28 (4123)     20243.83 (4044)     30279.43 (4074)     40095.46 (4021)

Toda Lattice     11298.29 (4584)     20373.09 (4240)     30254.50 (4125)     40696.08 (4132)

Coupled Oscillator     11162.42 (4624)     20088.37 (4085)     30427.31 (4105)     40281.04 (4046)

Generally, from Tables <ref> and <ref>, as $k$ increases, the vector error decreases and training time increases. The decrease in vector error is not proportional to the amount of data. The marginal improvement in vector error decreases as $k$ increases. Furthermore, the time taken for each epoch increases proportionately to $k$. However, the training time does not, because as $k$ increases, the number of epochs required for convergence decreases. The time taken for the model to converge when $k=1$ (where the number of samples is doubled from the original data) is not 200 times more than the time taken for the model to converge when $k=400$ (where the number of samples is four hundred times from the original data). These suggest that smaller values of $k$ are sufficient to cover the input domain and emphasise the additive separability of the regression task. Nonetheless, in general, the model with $k=400$ regresses the vector field best, and the model with $k=1$ is the fastest. A good trade-off that balances reducing the vector error and training time is the model with $k=2$.

§.§ Experiment 2: Optimising the Separable Hamiltonian Neural Network with Learning Bias

This subsection details the experimental setup to empirically study the trade-off between emphasing additive separability of the Hamiltonian regression task and $f_0$, $f_1$ and $f_2$ in Equation <ref>. The optimal value of $c_3$ is empirically determined.

§.§.§ Experimental Setup

The separable Hamiltonian neural network with learning bias is set up following the middle architecture in Figure <ref> with loss function Equation <ref>. All details of the setup, training and evaluation follow from the description above in Section <ref>.
The separable Hamiltonian neural network is trained and evaluated for cases where $c_3=\{0.25, 0.50, 1.00, 2.00, 4.00\}$.

§.§.§ Experimental Results
We report the vector error calculated following Equation <ref> for the separable Hamiltonian neural network with learning bias in Table <ref>.

Vector error $E$ and standard error (in brackets) of the separable Hamiltonian neural network with learning bias with $c_3=\{0.25, 0.50, 1.00, 2.00, 4.00\}$.

Experiment     $c_3 = 0.25$     $c_3 = 0.50$     $c_3 = 1.00$     $c_3 = 2.00$     $c_3 = 4.00$

Non-linear Pendulum      1.15E-02 (2.07E-03)     1.19E-02 (1.87E-03)     9.97E-03 (1.72E-03)     9.68E-03 (1.85E-03)     9.66E-03 (1.16E-03)

Trigo     1.15E-02 (8.88E-04)     1.17E-02 (1.13E-03)     1.22E-02 (1.17E-03)     1.15E-02 (1.12E-03)     1.10E-02 (1.00E-03)

Arctangent     5.00E-03 (4.08E-04)     4.85E-03 (4.01E-04)     4.64E-03 (4.04E-04)     4.49E-03 (3.24E-04)     4.47E-03 (4.25E-04)

Logarithmic     4.19E-03 (2.22E-04)     4.12E-03 (2.37E-04)     4.00E-03 (2.41E-04)     4.00E-03 (2.50E-04)     4.03E-03 (2.70E-04)

Anisotropic Oscillator      5.50E-03 (5.50E-04)     5.70E-03 (5.19E-04)     5.67E-03 (5.69E-04)     4.79E-03 (4.61E-04)     4.75E-03 (4.13E-04)

Henon Heiles     8.65E-03 (1.15E-03)     8.91E-03 (7.33E-04)     8.04E-03 (6.12E-04)     7.74E-03 (7.27E-04)     6.63E-03 (5.19E-04)

Toda Lattice      7.44E-03 (5.88E-04)     6.04E-03 (4.92E-04)     5.39E-03 (3.17E-04)     5.63E-03 (4.34E-04)     6.12E-03 (6.35E-04)

Coupled Oscillator     7.24E-03 (4.76E-04)     6.85E-03 (4.87E-04)     5.82E-03 (4.86E-04)     6.30E-03 (5.64E-04)     5.84E-03 (3.40E-04)

Time taken in seconds and number of epochs (in brackets) until convergence for the separable Hamiltonian neural network with learning bias with $c_3=\{0.25, 0.50, 1.00, 2.00, 4.00\}$.

Experiment     $c_3 = 0.25$     $c_3 = 0.50$     $c_3 = 1.00$     $c_3 = 2.00$     $c_3 = 4.00$

Non-linear Pendulum     961.73 (16183)     964.55 (15979)     1041.61 (16804)     1053.62 (17960)     1107.71 (18221)

Trigo      761.74 (16861)     708.33 (15463)     656.71 (14551)     685.51 (15170)     681.98 (15004)

Arctangent     1086.58 (17434)     1089.23 (17386)     1215.22 (19444)     1039.68 (17071)     1094.83 (18033)

Logarithmic      366.54 (7879)     356.74 (7871)     376.92 (8356)     365.03 (8089)     363.11 (7918)

Anisotropic Oscillator     379.00 (6330)     388.16 (6686)     389.89 (6534)     431.07 (7170)     444.30 (7324)

Henon Heiles      553.52 (9227)     511.45 (8414)     535.14 (8849)     543.65 (9189)     624.28 (10286)

Toda Lattice      965.10 (11483)     1011.75 (12107)     1112.39 (13648)     1060.29 (12705)     1110.75 (13082)

Coupled Oscillator      777.76 (10196)     791.64 (10310)     915.80 (12093)     862.42 (11520)     843.84 (11183)

Table <ref> shows that as $c_3$ increases, the vector error decreases. Table <ref> shows that as $c_3$ increases, the number of epochs and training time required increases. These suggest that as the value of $c_3$ increases, the emphasis on additive separability increases, and the separable Hamiltonian neural network with inductive bias places less emphasis on learning Hamilton's equations. As a result, the number of epochs required for the model to learn the surrogate Hamiltonian and converge increases, increasing training time. The model with $c_3 = 4.00$ regresses the vector field best. The model with $c_3 = 0.25$ is the fastest. A good trade-off that balances the importance of additive separability and Hamilton's equations appears to be $c_3=1.00$ as it often outperforms the model when $c_3=2.00$.
Using $c_3=1.00$ is optimal, and balances the importance of additive separability and Hamilton's equations.

§.§ Experiment 3: Optimising the Separable Hamiltonian Neural Network with Inductive Bias

This subsection details the experimental setup to empirically study the trade-off between different implementations of the separable Hamiltonian neural network with inductive bias.

§.§.§ Experimental Setup

The separable Hamiltonian neural network with inductive bias is set up following the rightmost architecture in Figure <ref>. The width of each smaller conjoined neural network is shown in the second last column of Table <ref>.

The proposed separable Hamiltonian neural network with inductive bias trains the two conjoined neural networks in two ways. Firstly, consecutively, where $\vec{x}$ is first input to the first conjoined neural network, which outputs $f(\vec{x})$, then $\vec{y}$ is input to the second conjoined neural network, which outputs $g(\vec{y})$. Their sum is the surrogate Hamiltonian. Secondly, simultaneously and in parallel, by designing each layer of the separable Hamiltonian neural network such that it appends the respective layers of the two conjoined neural networks. For the case where the model is trained in parallel, forward propagation through the proposed model is computed as $x_{n} = \sigma(W_{n} \times x_{n-1} + B_{n})$ where $\sigma$ is the activation function, $x_{n}$ is the output of layer $n$ calculated from the output of layer $n-1$, $W_{n}$ is the weight matrix of layer $n$ of shape $2\times L_n \times L_{n-1}$, and $L_n$ and $L_{n-1}$ are the widths of layers $n$ and $n-1$ respectively, and $B_{n}$ is the bias matrix of layer $n$ of shape $2\times 1 \times L_n$. The $2$ in both the weight matrix and bias matrix corresponds to the two disjoint subsets of the state variables of the additively separable Hamiltonian. With this architecture, both smaller conjoined neural networks are trained simultaneously in parallel, with each forward and backward propagation using one graphics processing unit.

The separable Hamiltonian neural network with inductive bias is also trained and evaluated for five possible implementations of the summation layer. The zeroth implementation is a simple summation (sHNN-I (0)). The first implementation is a linear layer with fixed and equal weights, and no bias (sHNN-I (1)). The second implementation is a linear layer with fixed and equal weights, and a trainable bias (sHNN-I (2)). The third implementation is a linear layer with trainable weights, and no bias (sHNN-I (3)). The fourth implementation is a linear layer with trainable weights and bias (sHNN-I (4)). For the zeroth and first implementations, the last column in Table <ref> shows the number of parameters of the model. The second, third and fourth implementations have one, two and three additional parameters respectively. All other details of the setup, training and evaluation follow from the description above in Section <ref>.

§.§.§ Experimental Results

We report the vector error calculated following Equation <ref> for the separable Hamiltonian neural network with inductive bias in Tables <ref> and <ref>.

Vector error $E$ and standard error (in brackets) of the separable Hamiltonian neural network with inductive bias.

Experiment     Parallel     Consecutive

Non-linear Pendulum      8.54E-03 (1.25E-03)     9.22E-03 (1.17E-03)

Trigo     8.19E-03 (9.10E-04)     8.18E-03 (9.35E-04)

Arctangent     5.42E-03 (1.23E-03)     5.03E-03 (9.67E-04)

Logarithmic     4.70E-03 (2.53E-04)     4.71E-03 (2.57E-04)

Anisotropic Oscillator     5.43E-03 (4.84E-04)     5.41E-03 (3.76E-04)

Henon Heiles     7.14E-03 (6.41E-04)     8.22E-03 (7.35E-04)

Toda Lattice      7.28E-03 (8.89E-04)     6.36E-03 (3.57E-04)

Coupled Oscillator     6.43E-03 (4.51E-04)     6.58E-03 (4.33E-04)

Time taken in seconds and number of epochs (in brackets) until convergence for the separable Hamiltonian neural network with inductive bias .

Experiment     Parallel     Consecutive

Non-linear Pendulum     781.37 (17218)     1103.43 (17468)

Trigo      523.00 (12265)     749.89 (12214)

Arctangent     1272.94 (28813)     1782.73 (28641)

Logarithmic     333.90 (7580)     492.30 (7935)

Anisotropic Oscillator     282.49 (6022)     377.48 (5647)

Henon Heiles     341.88 (7223)     473.61 (7322)

Toda Lattice      453.37 (9721)     597.27 (9104)

Coupled Oscillator     347.43 (7195)     474.03 (7029)

From Table <ref> it is observed that training the conjoined neural networks in parallel and consecutively results in different vector errors and training time. Intuitively, they should have the same vector errors, but closer analysis reveals they have different floating point errors which cause their vector errors to diverge after many iterations. Generally, both separable Hamiltonian neural networks perform well in regressing the vector field as they have similar vector errors. However, from Table <ref>, it is observed that the separable Hamiltonian neural network with conjoined neural networks trained in parallel is consistently faster than that which is trained consecutively. Therefore, it is preferred that the conjoined neural networks are trained in parallel.

Vector error $E$ and standard error (in brackets) of the separable Hamiltonian neural network with inductive bias.

Experiment     sHNN-I (0)     sHNN-I (1)     sHNN-I (2)     sHNN-I (3)     sHNN-I (4)

Non-linear Pendulum      8.54E-03 (1.25E-03)     8.54E-03 (1.25E-03)     1.07E-02 (1.45E-03)     3.11E-02 (1.01E-02)     1.66E-02 (4.77E-03)

Trigo     8.19E-03 (9.10E-04)     8.19E-03 (9.10E-04)     8.11E-03 (9.81E-04)     1.27E-02 (3.24E-03)     1.40E-02 (3.51E-03)

Arctangent     5.42E-03 (1.23E-03)     5.42E-03 (1.23E-03)     5.53E-03 (9.02E-04)     1.15E-02 (4.67E-03)     6.98E-03 (1.40E-03)

Logarithmic      4.70E-03 (2.53E-04)     4.70E-03 (2.53E-04)     4.74E-03 (2.03E-04)     4.61E-03 (3.59E-04)     5.23E-03 (3.83E-04)

Anisotropic Oscillator     5.43E-03 (4.84E-04)     5.43E-03 (4.84E-04)     5.46E-03 (4.85E-04)     6.16E-03 (7.06E-04)     7.49E-03 (8.47E-04)

Henon Heiles     7.14E-03 (6.41E-04)     7.14E-03 (6.41E-04)     8.19E-03 (6.57E-04)     1.08E-02 (1.74E-03)     8.52E-03 (7.90E-04)

Toda Lattice     7.28E-03 (8.89E-04)     7.28E-03 (8.89E-04)     6.48E-03 (4.09E-04)     7.32E-03 (8.31E-04)     6.71E-03 (5.54E-04)

Coupled Oscillator      6.43E-03 (4.51E-04)     6.43E-03 (4.51E-04)     7.58E-03 (6.59E-04)     7.86E-03 (7.57E-04)     8.07E-03 (6.44E-04)

Time taken in seconds and number of epochs (in brackets) until convergence for the separable Hamiltonian neural network with inductive bias.

Experiment     sHNN-I (0)     sHNN-I (1)     sHNN-I (2)     sHNN-I (3)     sHNN-I (4)

Non-linear Pendulum      781.07 (17218)     782.08 (17218)     666.85 (14302)     954.76 (20334)     751.63 (15924)

Trigo     538.09 (12265)     579.91 (12265)     587.27 (12133)     921.46 (19324)     926.06 (18776)

Arctangent     1305.04 (28813)     1325.54 (28813)     1833.63 (39295)     1104.49 (23381)     1646.29 (34049)

Logarithmic     325.69 (7580)     384.96 (7580)     414.82 (8107)     453.58 (8593)     482.48 (9147)

Anisotropic Oscillator     264.34 (6022)     305.55 (6022)     302.17 (5927)     333.65 (6298)     323.18 (6055)

Henon Heiles     320.37 (7223)     414.94 (7223)     410.67 (7199)     424.65 (7393)     456.86 (8186)

Toda Lattice     465.70 (9721)     526.01 (9721)     559.81 (9930)     555.06 (9919)     528.27 (9280)

Coupled Oscillator     327.42 (7195)     459.48 (7195)     436.48 (6853)     494.67 (7691)     565.67 (8667)

From Table <ref>, it can be observed that the simple summation (sHNN-I (0)) and linear layer with fixed and equal weights (sHNN-I (1)) have the same vector errors because they are identical in implementation. However, from Table <ref>, it is observed that the simple summation (sHNN-I (0)) is consistently faster than the linear layer with fixed and equal weights (sHNN-I (1)). Generally, these two models also have the lowest vector errors among the five possible last-layer implementations. The second, third and fourth implementations have more trainable parameters but do not regress the vector field well. Additional trainable weights or a bias obfuscate the contributions of the conjoined neural network toward the surrogate Hamiltonian and introduce unnecessary complexities when regressing the surrogate Hamiltonian.
Therefore, the zeroth implementation, with a simple summation, is the preferred implementation for the summation layer of the separable Hamiltonian neural network with inductive bias.

§.§ Experiment 4: Comparing the Four Variants on the Task of Regressing the Hamiltonian and Vector Field

This subsection details the experimental setup to empirically compare the four models on the task of regressing the Hamiltonian and vector field of a Hamiltonian system. The four models are the baseline Hamiltonian neural network and the three proposed separable Hamiltonian neural networks with observational, learning and inductive biases.

§.§.§ Experimental Setup

The Hamiltonian neural network is set up following the leftmost architecture in Figure <ref>. From subsections <ref>, <ref> and <ref>, the optimal implementations of the various separable Hamiltonian neural networks are used. These are the separable Hamiltonian neural network with observational bias where $k=2$, the separable Hamiltonian neural network with learning bias where $c_3=1.00$ and the separable Hamiltonian neural network with inductive bias with a summation in the last layer. All details of the setup, training and evaluation follow from the description above in Section <ref>.

§.§.§ Experimental Results
We report the L1-error in regressing the Hamiltonian following Equation <ref> and the vector error calculated following Equation <ref> for all models in Table <ref> and Table <ref> respectively. We report the time taken to train each model in seconds in Table <ref>.

Mean squared error and standard error (in brackets) in regressing the Hamiltonian for the baseline HNN, proposed sHNN-O, proposed sHNN-L and proposed sHNN-I trained till convergence. Percentages indicate percentage improvement from baseline.

Experiment     Baseline HNN     Proposed sHNN-O     Proposed sHNN-L     Proposed sHNN-I

Non-linear Pendulum       5.42E-04 (2.95E-04)     1.14E-04 (6.25E-05)     3.90E-04 (1.94E-04)     2.06E-04 (1.25E-04)

Trigo       3.79E-05 (9.37E-06)     1.77E-05 (4.81E-06)     4.15E-05 (8.34E-06)     9.05E-06 (2.55E-06)

Arctangent      1.46E-05 (3.06E-06)     5.07E-06 (1.31E-06)     1.07E-05 (3.72E-06)     8.89E-05 (4.32E-05)

Logarithmic       3.36E-06 (8.60E-07)     3.68E-06 (1.90E-06)     5.65E-06 (1.50E-06)     2.09E-06 (7.54E-07)

Anisotropic Oscillator       7.00E-06 (2.49E-06)     4.34E-06 (1.80E-06)     9.56E-06 (2.59E-06)     4.51E-06 (1.35E-06)

Henon Heiles     6.91E-06 (1.94E-06)     2.23E-06 (4.94E-07)     6.14E-06 (1.65E-06)     7.29E-06 (1.68E-06)

Toda Lattice      2.71E-05 (5.39E-06)     1.01E-05 (3.15E-06)     2.15E-05 (4.07E-06)     1.99E-05 (3.81E-06)

Coupled Oscillator        2.56E-05 (9.94E-06)     8.77E-06 (2.98E-06)     1.14E-05 (3.38E-06)     1.51E-05 (4.18E-06)

Vector error $E$ and standard error (in brackets) of the baseline Hamiltonian neural network and proposed separable Hamiltonian neural networks with observational, learning and inductive biases.

Experiment     Baseline HNN     Proposed sHNN-O     Proposed sHNN-L     Proposed sHNN-I

Non-linear Pendulum     1.24E-02 (2.50E-03)     6.80E-03 (1.68E-03)     9.97E-03 (1.72E-03)     8.54E-03 (1.25E-03)

Trigo       1.18E-02 (1.08E-03)     8.17E-03 (7.17E-04)     1.22E-02 (1.17E-03)     8.19E-03 (9.10E-04)

Arctangent      5.09E-03 (2.89E-04)     3.55E-03 (2.49E-04)     4.64E-03 (4.04E-04)     5.42E-03 (1.23E-03)

Logarithmic       4.32E-03 (2.21E-04)     3.02E-03 (1.72E-04)     4.00E-03 (2.41E-04)     4.70E-03 (2.53E-04)

Anisotropic Oscillator     7.33E-03 (6.34E-04)     4.53E-03 (4.52E-04)     5.67E-03 (5.69E-04)     5.43E-03 (4.84E-04)

Henon Heiles      1.11E-02 (8.87E-04)     5.48E-03 (3.78E-04)     8.04E-03 (6.12E-04)     7.14E-03 (6.41E-04)

Toda Lattice       9.47E-03 (4.90E-04)     4.65E-03 (3.57E-04)     5.39E-03 (3.17E-04)     7.28E-03 (8.89E-04)

Coupled Oscillator      1.10E-02 (1.40E-03)     5.35E-03 (5.00E-04)     5.82E-03 (4.86E-04)     6.43E-03 (4.51E-04)

Time taken in seconds and number of epochs (in brackets) until convergence for the baseline HNN, proposed sHNN-O, proposed sHNN-L and proposed sHNN-I.

Experiment     Baseline HNN     Proposed sHNN-O     Proposed sHNN-L     Proposed sHNN-I

Non-linear Pendulum     672.12 (18213.6)     818.47 (9816.35)     1041.61 (16803.75)     781.07 (17217.95)

Trigo      490.53 (14701.8)     899.80 (10341.85)     656.71 (14550.9)     538.09 (12264.8)

Arctangent      586.52 (16307.0)     880.43 (10445.35)     1215.22 (19443.75)     1305.04 (28812.8)

Logarithmic      284.68 (8353.7)     525.97 (6210.55)     376.92 (8356.4)     325.69 (7579.55)

Anisotropic Oscillator      220.00 (6527.05)     469.20 (5445.35)     389.89 (6533.7)     264.34 (6022.45)

Henon Heiles      295.71 (8510.95)     589.43 (6572.05)     535.14 (8848.7)     320.37 (7222.85)

Toda Lattice     408.06 (11196.0)     740.80 (8446.45)     1112.39 (13647.75)     465.70 (9720.65)

Coupled Oscillator       317.71 (9255.3)     638.25 (7657.7)     915.80 (12092.8)     327.42 (7195.0)

From Tables <ref> and <ref>, we observe that in general, all proposed separable Hamiltonian neural networks regress the Hamiltonian and vector field with a lower absolute error and vector error than the baseline Hamiltonian neural network. The proposed models leverage physics information regarding separability to penalise or prevent interaction between the state variables and this reduces the complexity of the Hamiltonian regression problem. The regression of the Hamiltonian and vector field are therefore improved. However, from Table <ref>, the baseline model is faster than all proposed models.

Generally, from Tables <ref>, <ref> and <ref>, we observe that among the three proposed models, the separable Hamiltonian neural network with observational bias has the lowest absolute error and vector error, while the separable Hamiltonian neural network with inductive bias is the fastest. The observational bias generates more samples of the data, and this emphasises additive separability and covers the input domain of the Hamiltonian system to ease the interpolation task of the model. Conversely, the models with learning and inductive bias only rely on emphasising additive separability to regress the Hamiltonian and vector field. The additional effect of the observational bias in covering the input domain of the Hamiltonian system allows the model to regress the vector field and Hamiltonian better. The model with inductive bias generally outperforms the model with learning bias as it restricts regressions of the Hamiltonian and vector field to strictly satisfy separability, therefore forcing the model to simplify a complex regression problem into two smaller ones.

It is also observed that the relative performance of the models between Tables <ref> and <ref> changes. Regressing the Hamiltonian involves finding the sum of the integrated vector field and errors made by the models in regressing the vector field may be cancelled out when regressing the Hamiltonian.

From Table <ref>, the proposed models generally require fewer epochs to converge as the knowledge of additive separability reduces the complexity of the Hamiltonian regression problem. However, the baseline model is the fastest to train.
This is because the time taken for each epoch for the proposed models is longer. Compared to the baseline model, the proposed model with inductive bias is slower due to its conjoined architecture with higher dimensional weight and bias matrices that slightly increase the forward and backward propagation time for each epoch. The proposed model with learning bias is even slower due to the additional time taken to compute the mixed partial derivative. The proposed model with observational bias is the slowest as it has several times more samples that linearly scale the training time per epoch given the same batch size.
Among the proposed models, the model with inductive bias generally requires the fewest number of epochs to converge and less time per epoch. It is the fastest proposed model.

The separable Hamiltonian neural network with inductive bias is the optimal separable Hamiltonian neural network as it outperforms the baseline in regressing the Hamiltonian and vector field and has the smallest trade-off in training time.

§ CONCLUSION

Four models are compared for the task of regressing the Hamiltonian and vector field of a Hamiltonian system from discrete observations of the vector field. One, the baseline Hamiltonian neural network, is uninformed of the additive separability of the Hamiltonian system. Three proposed separable Hamiltonian neural networks are informed via observational, learning and inductive biases respectively.
All proposed separable Hamiltonian neural network models leverage additive separability to avoid artificial complexity between state variables. They are more effective than the baseline in regressing Hamiltonian vector fields and can converge within fewer epochs, but are generally slower in training. The best model is the separable Hamiltonian neural network with inductive bias as it outperforms the baseline in the regression tasks and has the smallest trade-off in training time.

We are now studying separable Hamiltonian neural networks that are simultaneously embedded with multiple biases. Preliminary results show that models embedded with both observational and inductive biases can regress Hamiltonian vector fields best. We are also working on using an inductive bias to recover the kinetic and potential energies of a Hamiltonian system for better interpretability, and dynamically testing for and embedding separability as an inductive bias by rewiring the Hamiltonian neural network on the fly.

§ PROOF OF THE FIRST ZONKLAR EQUATION

Appendix one text goes here.

Appendix two text goes here.