# About some compression algorithms

Orchidea Maria Lecian
<EMAIL_ADDRESS>
ICRA c/o Physics Department, Sapienza University of Rome,
Piazzale Aldo Moro, 5- 00185 Rome, Italy;
Faculty of Civil and Industrial Engineering, Sapienza University of Rome,
Via Eudossiana, 18- 00184 Rome, Italy;
Faculty of Information Engineering, Sapienza University of Rome,
Via Eudossiana, 18- 00184 Rome, Italy;
Brunello B. Tirozzi
<EMAIL_ADDRESS>
Department of Physics,
University La Sapienza of Rome, Italy.

(June 2023)

## Abstract

We use neural network algorithms for finding compression methods of images in
the framework of iterated function systems which is a collection of the
transformations of the interval $(0,1)$ satisfying suitable properties.

## Key-words:

Fractal compression; neural networks; Hertz method.

##  I Introduction

Be the series $x_{1},...,x_{n}$. A function $g$ has to be found, such that

$x_{t+1}=g(x_{t}).$ (1)

The series can be the trajectory of a dynamical system or the realisation of
some stochastic process, where the latter can be obtained after experimental
evidence.
Two approaches are going to be discussed.
The auto-regressive integrated moving average (ARIMA) models are already apt
to provide with a solution of this problem; nevertheless there is the
limitation to their application since they are valid for stationary processes
only, and it is sometimes necessary to implement many prime differences to
obtain from the series the initial data of a stationary succession, which
peculiarity augments the characteristic dimension of of the process
artificially. The neural networks can be applied to any succession without the
need of the property of stationarity; they are generally used to reconstruct
any series of data from any non-linear non-deterministic transformation.
The paper is organised as follows.
In Section I, the possibility to apply neural networks to reconstruct series
of data form non-linear non-deterministic transformations is presented.
In Section II, the Hertz method is analysed.
In Section III, the partition of the phase-space methodology is put forward.
In Section IV, patterns-learning neural networks are exposed.
In Section V, the Hertz method is further investigated.
In Section VI, layers networks are exhibited.
In Section VII, the images-compression methods are set.
In Section VIII, the contractive transformations are further characterised.
In Section IX, the qualities of the winner neuron are revelaed.
In Section X, prospective studies are envisaged.

##  II The Hertz method

As an example, the Hertz reconstruction method ref13  allows one to determine
with much precision the graphic of the chaotic transformation of the interval
$(0,1)$ in itself,i.e.

$x_{n+1}=4x_{n}(1-x_{n}),\ \ x_{n}\in(0,1).$ (2)

In general, the data to be analysed will be apt to be grouped in
$d$-dimensional vectors, for which the block of the first $d$ data can be
considered as the components of the first vector, the second block of $d$ data
can be considered as the components of the second vector, and so on, according
to the following scheme:

$\displaystyle\underline{x}=(x^{1}_{m}),\ \ m=1,...,d,$ (3a) $\displaystyle
x^{k}_{m}=x_{kd+m},\ \ m=1,...,d,$ (3b)

i.e. such that $d$ is the characteristic dimension of the process. In the
$1$-dimensional case here considered, $d$ can be identified with the
correlation length between the data which is given from the smallest $k$ such
that

$Ex_{t}x_{t+k}=(Ex_{t})^{2}.$

If one has also a geometric structure, data have to be considered, which have
dimension equal to that of the MANIFOLD to which they belong; afterwords, the
implementation in the above can be applied. In general, on has thus to
consider a transformation

$\underline{x}^{k+1}=\underline{g}(\underline{x}^{k}),\ \
\underline{x}^{k}\in\mathbb{R}^{m}.$ (4)

##  III The Hertz method: the partition of the phase-space methodology

For the sake of simplicity, one can consider the case $m=1$; it is worth
recalling the method of Hertz et al. ref13 .
The method consists in establishing a partition of the state space $X$, which
is taken as the same of the interval $(0,1)$ for the sake of directness, where
the partition consists of intervals (blocks) $B_{\alpha,a}=1,...,M$; in other
words,

$X=\cup_{\alpha=1}^{\alpha=M}B_{\alpha},\ \ B_{\alpha}\cap B_{\beta}=0\ \
\forall\alpha\neq\beta.$ (5)

The dimension of these blocks is not necessarily constant, as one can learn
from the recursive process used for their determination, which is illustrated
in the below.
For each block $B_{\alpha}$, the mean value $x_{\alpha}$ is defined, of the
data which belong to the block; the dispersion $\sigma^{2}_{\alpha}$ of the is
defined as well as

$x_{\alpha}=\frac{1}{N_{\alpha}}\sum_{x\in
B_{\alpha}}x\sigma_{\alpha}^{2}=\frac{1}{N_{\alpha}}\sum_{x\in
B_{\alpha}}(x-x_{\alpha})^{2},$ (6)

where $N_{\alpha}$ is the number of data which belong to the block
$B_{\alpha}$. The structure $B_{1},...,B_{M}$ is determined as follows.
As a first step, the whole space $X$ is divided into two blocks $B_{L}$ and
$B_{R}$, where the condition is imposed, that the sum

$\sigma_{B_{L}}^{2}+\sigma_{B_{R}}^{2}$ (7)

is the least possible one.
within this construction, it is fundamental that each block contain a number
of points which is greater than a certain quantity.
As an example, n the case of data from geology, the condition $N_{\alpha}>16$
has to be imposed ref10 : differently, it is impossible to reconstruct the
series. This condition, indeed, allows one to avoid the occurrence of extreme
situations, such as the possibility that the leftmost block $B_{L}$ contains
only one datum $N_{L}=1$, and that all the other data belong to the rightmost
block $N_{R}=N-1$. The lower limit of the $N_{\alpha}$ is determined also
after the request that the introduction of the quantities $x_{\alpha}$ and
$\sigma_{\alpha}^{2}$ be of interest.
This construction is repeated in each of the blocks $B_{L}$ and $B_{R}$ until
the values of $N_{\alpha}$ reach the lower limit; finally, a certain number of
blocks is constructed, each containing a certain group of data. The presented
reconstruction method consists in the approximation of $g$ in each block with
an opportune constant, i.e. in finding a local approximation of $g$ in each
$B_{\alpha}$.
For these purposes, for each $B_{\alpha}$ a characteristic function
$f_{\alpha}(x)$ of the block is here newly defined as

$f_{\alpha}(x)=e^{-\frac{(x-x_{\alpha})^{2}}{2\sigma_{\alpha}^{2}}};\ \
P_{\alpha}=\frac{f_{\alpha}(x)}{\sum_{\alpha=1}^{\alpha=M}f_{\alpha}(x)}.$ (8)

In Eq. (8), the function $P_{\alpha}$ is obviously defined to acquire the
approximate value of $1$ for $x\in B_{\alpha}$,as the other functions
$f_{\beta}(x)$ are almost vanishing for $x\notin B_{\beta}$; it is therefore
possible to demonstrate straightforward that, in Eq. (8), the most relevant
contribution in the denominator is $f_{\alpha}(x)$ for $x\in B_{\alpha}$,i.e.
from which it is straightforward demonstrated that the fraction equals $1$.
$\Box$
The procedure is most effective the smaller the $\sigma_{\gamma}^{2}$: for
this reason, the construction of the blocks $B_{\alpha}$ is accomplished under
the request that the condition of minimum hold. The function $g(x)$ is
approximated by a function $f(x)$ which is almost constant in the blocks
$B_{\alpha}$, and which takes the value $f^{\alpha}$ in such a block as

$f(x)=\sum_{\alpha=1}^{\alpha^{M}}f^{\alpha}P_{\alpha}(x).$ (9)

The constants $f^{\alpha}$ are determined by imposing the minimum of the Error
function

$E=\sum_{i=1}^{i=N-1}(x_{+1}-f(x_{i}))^{2}.$ (10)

As before, the minimum can be found by the Monte-Carlo method by identifying
$E(\\{f^{\alpha}\\})$ with the energy of the system.

##  IV Patterns-learning neural networks

Furthermore, it is interesting to consider the present algorithm also from a
different point of view, i.e. as a particular neural network, which learns the
patterns $x_{1},...,x_{n}$.
It is the present purpose to newly demonstrate the important concept which s
at the basis of all the neural networks applied to the analysis and to the
recognition of the data.
The following neural structure is considered.
The value $x$ assigned to the input neuron is passed to the intermediate units
$1,...,M$. The unit $\alpha$ il qualified after a non-linear output function
$P_{\alpha}(x)$ defined in Eq. (8); $f^{1},...,f^{M}$ are the weights which
connect the output units with the neurons $1,...,M$. The output neuron
receives a quantity

$f(x)=\sum_{\alpha}f^{\alpha}P_{\alpha}(x).$ (11)

According to the neural language, the network learns the patterns
$(x_{i},x_{i+1})$ by varying the weights $f^{\alpha}$ (where the evolution of
the weights corresponds, in this language, to the learning process), such that
the difference between the wanted output $x_{i+1}$ in correspondence of the
input $x_{i}$ and the effective output $f(x_{i})$ be the least; this procedure
corresponds to the calculation of the minimum of Eq. (10). The law of
evolution of the weights can be given after the probabilistic change of the
Monte-Carlo method, or by the gradient method

$\Delta f^{\alpha}=-\eta(k)\frac{\partial E}{\partial f^{\alpha}},$ (12)

where $\eta(k)$ is a coefficient decaying with $k$, $k$ being the number of
iterations of the learning process, and $\Delta f^{\alpha}$ is the variation
of the weight $f^{\alpha}$. The reason of the decreasing behaviour of
$\eta(k)$ is explained after considering $E$ as a quadratic function of the
variable $x$. If $x$ is on the left of the minimum, it undergoes a positive
increase, while if $x$ is on the right of the minimum, it undergoes a negative
decrease, as $\Delta x$ is proportional to the derivative of $E$ with respect
to $x$; nevertheless, as the variable approaches the minimum, the changes
might make the variable $x$ go from a zone of increase to a zone of decrease,
and so to make $x$ oscillate around the limit. In the case of a quadratic
function, the first derivative vanishes linearly around the minimum, and
therefore these oscillations are damped; in the case of any quadratic function
of more variables this effect can be missing: it is thus better to make use of
the convergence factor $\eta(k)$. The choice of an opportune $eta(k)$ is the
analogue of the variation of the temperature in the Monte-Carlo method, and
there exists a theoretical approach for the determination of $\eta(k)$ ref14 .

##  V More about the Hertz method

The Hertz methods becomes some more complicated if one wants to obtain a
better approximation. In this case, it is better to approximate locally $g(x)$
with linear functions inside the block

$f(x)=\sum_{\alpha=1}^{\alpha=N}(a^{\alpha}+b^{\alpha}x)P_{\alpha}(x).$ (13)

One can imagine that the number of neurons doubles; each neuron becomes a
couple in which each element has a different output function.
After the scheme Eq. (13), an excellent reconstruction of the function
$g(x)=4x(1-x)$ was rendered, as well as of the data pertinent to the ratios of
the concentrations of the isotopes of Sn ref10  considered starting from 540
millions of years ago. It is interesting to note that the plot of the series
reconstructed by this method extrapolated at the period before 540 million
years is inside the uncertainty of $30\%$ established after the predictions of
the ARIMA model.
Given any time series which exhibits a stochastic behaviour, it is suited to
make use of both approaches, in a way to gain control of the likelihood of the
predictions.
This method applies also to the distributions of thickness of the rings of the
trees ref15 . Nevertheless, the construction of the blocks $B_{\alpha}$ was
accomplished with good success without working on the data directly, but
grouping the data which are obtained from those measured after a
transformation, which somehow takes into account the different behaviour of
the growth of the trees with respect to their age. It is important to note
that the neural approach to this type of problems is made easier according to
the fact that there exist on the internet many knots of public domain which
make available to the users neural-networks simulators which work in unix
environment within the graphic system X-window. These programs can be easily
copied by ftp if the computer on which the user works is connected to the web.
As an example of neural-network simulators, it is important to consider one
constructed by the Hinton group (Toronto), which has many learning methods,
i.e. many methods to minimize the cost function $E$ Eq. (10). It is therefore
possible to compare many methods of minimization and to chose the most
appropriate one as far the current problem is concerned. This simulator
contains also a graphic display of the weights which are found, and,
therefore, of the solution of the problem of the minimum.The Hertz et al.’s
neural network can therefore be used to reconstruct any functional dependence
within a time series. The interesting feature of this method is the
possibility to accomplish an analysis about the structure and about the
distribution of the data. This method seems the best one to address problems
of such type, as in comparison with the Kohonen Networks.
It is also possible to address this problem by means of a neural network
without this introductory analysis about the data distribution. These networks
are of common use in many applicative problems , and they are named ’back
propagation’ networks after the name of the algorithm used within the learning
of the network. It is possible to consider the three-layers network: one input
layer, an output layer and an intermediate layer. The set of the neural layers
and of the connection between the layers forms the architecture of the neural
network one is going to investigate next.

##  VI More about layers networks

Let $d$ be the characteristic dimension of the time series determined after
the methods discussed in the above. It is natural to hypothesize that the
input network consist of $d$ neurons, to each of which only one component is
presented by the input vectors $\underline{x}\equiv x_{1},...,x_{d}$. The $d$
input neurons are linked with all the $M$ neurons within the intermediate
layer through the weights

$W^{1}_{ij},\ \ i=1,...,M,\ \ j=1,...,d.$ (14)

When a $d$-dimensional datum $\underline{x}$ is presented to the input layer,
each knot $i$ of the intermediate layer receives a signal spelled after a
formula analogous to that of the Hopfield model

$\sum_{j=1}^{j=d}W^{1}_{ij}x_{j},\ \ i=1,...,M,$ (15)

and, on its turn, the very same neuron emits a signal which is defined after
its non-linear output function $z_{i}$ s.t.

$z_{i}=\sigma_{1}\left(\sum_{j=1}^{j=d}W^{1}_{ij}x_{j}\right),$ (16)

for which, for instance,

$\sigma(x)=\frac{1}{1+e^{-\lambda x}},$ (17)

in the cases which are included between $0$ and $1$.
The output signals from the $M$ intermediate neurons arrive to the output
layer by means of a law analogous to that Eq. (15), such that the generic
neuron $i$ from the output layer receives a signal equal to

$\sum_{j=1}^{j=M}W^{2}_{ij}z_{j}.$ (18)

The $W^{2}_{ij}$, $j=1,...,M$ are the synaptic couplings between the
intermediate layer and the output one. The generic output emits a neuron $i$
emits a signal spelled as

$y_{i}=\sigma_{2}\left(\sum_{j=1}^{j=M}W^{2}_{ij}z_{j}\right).$ (19)

This network has the purpose to admit as an output vector $\underline{y}\equiv
y_{1},...,d$ the vector $\underline{x}^{k+1}$ when the date pertinent to the
time $k$, i.e. $\underline{x}^{k}$ is presented to the output layer. It is
therefore necessary to determine the set of synaptic links $W^{1}_{ij}$,
$W^{2}_{lk}$ in a way such that the difference between the output of the
network $\underline{y}$ and the wanted output $\underline{x}^{k+1}$ be
minimized. It is natural to introduce a cost function of a form

$E=\sum_{k=1}^{k=N}\sum_{l=1}^{l=d}\left(x_{l}^{k+1}-\sigma_{2}(W^{2}_{lj}\sigma_{1}(\sum_{m=1}^{m=d}W^{1}_{jm}x^{k}_{m}\right)^{2}.$
(20)

The function $E$ can be minimized again by means of the Monte-Carlo method or
by means of the method of the gradient. As it is a complicated function
depending on the weights, maxima and minima are expected to be present, such
that, in general, there is the need to apply the ’simulated annealing’ method
chapIV . After the analysis of some time series, it was observed that the
Hertz et al.’s network and that based on the ’back propagation’ provide with
the same results as reconstruction; nevertheless, the convergence time of the
weights of the latter network is much longer than that of the former network
is the case of data issued form systems of stochastic behaviour.

##  VII Images-compression methods

Three images-compression methods will be considered.
The Bransley ref16  method is based on the ’iterated function system’ (IFS),
which are affine and contractive transformations of the plane.
The neural-network-of-’back propagation’ method is a method based on a one-
intermediate-layer neural network with an input layer and an output layer.
The vector-quantization method is realised via Kohonen networks. This approach
is very general and applies to the analysis of any time series,i.e.so that it
could be treated within the framework of the Hopfield model sectionIII . It is
possible to reach enormous compression factors via the IFS; neverthelss, the
reconstruction mechanism is not clear, and is probably base on the use of
other quantization techniques, and on a huge data-base which associates the
true image to each compressed image.
As an instance to explain the IFS method, it is useful for the sake of
simplicity to make use of a ’black and white’ image which appears on the
screen of a calculator; in other words, a succession of data is considered,
$x_{1},...,x_{n}$, where $x_{i}$ can acquire the values $0$ or $1$, and $N$ is
the total number of pixels (discrete lattice) which define the screen. The
fundamental hypothesis in the Bransley method is that every image can be
divided into two different types of subsets, i.e. one in which there is a
simple structure whose contours are defined by regular geometric lines, whose
approximation by simple function holds. The further method to be considered is
one based on the assumption that a complicated structure can be approximated
by a fractal set. As instances of fractal structures, the Cantor set and the
Serpinski triangle are examples. Almost all of these fractals can be generated
by the application of a well-determined family of affine contractive
transformations of the plane an infinite number of times.

##  VIII More about contractive transformations

A set of contractive transformations $w_{1}(z),...,w_{k}(z)$ of the plane
$z=(x,y)\in\mathbb{R}^{2}$ constitutes a family of IFS maps if the following
hypotheses are satisfied:

$w_{i}(z)\equiv(a_{i}x+b_{i}y+c_{i},d_{i}x+e_{i}y+f_{i}),\ \ i=1,...,M,$ (21)

and

$d(w_{i}(z_{1}),w()_{i}(z_{2})\leq sd(z_{1},z_{2}),\ \ \forall i=1,...,k,\\\
\forall z_{1},z_{2}\in\mathbb{R}^{2},$ (22)

with $0<s<1$.
In other words, all the linear transformations which belong to a certain IFS
are contractive with the same coefficient of contraction.
Be $W(z)$ the transformation generated by these maps, i.e.

$W(z)=\bigcap_{i=1}^{i=k}w_{i}(z).$ (23)

It is straightforward to demonstrate that, if the transformation $W(z)$ is
generated by a IFS, by impementing $N$ iterations of the map $W$ starting from
a point $z$ chosen at random with unifrm probability in a compact subset
$K\in\mathbb{R}^{2}$, the limit

$lim_{N\rightarrow\infty}(W^{o})^{N}(z)$ (24)

exists, it does not depend on the choice of $z$ and is a fractal. An example
is given after the image which is obtained by applying the following linear
transformations ref16 :

$\displaystyle x_{1}=0.5x+1$ (25a) $\displaystyle y_{1}=0.5+1$ (25b)
$\displaystyle x_{1}=0.5x+1$ (25c) $\displaystyle y_{1}=0.5y+50$ (25d)
$\displaystyle x_{1}=0.x+50$ (25e) $\displaystyle y_{1}=0.5y+50.$ (25f)

$(W^{O})^{N}(z)$ converges with respect to the Hausdorff metric defined in the
space of the subsets of a compact $K$ in the plane. Given two subsets
$A,B\subset K$, the Hausdorff distance $h_{A,B}$ can be determined in the
following way

$\displaystyle h_{1}(x)=inf_{y\in B}d(x,y),$ (26a) $\displaystyle
h_{1}=sup_{x\in A}h(x),$ (26b) $\displaystyle h_{2}(y)=inf_{x\in A}d(y,x),$
(26c) $\displaystyle h_{2}=sup_{y\in B}h_{2}(y),$ (26d) $\displaystyle
h_{A,B}=max(h_{1},h_{2}).$ (26e)

As solely images realised on a lattice are here dealt with, the Hausdorff
distance can well be substituted by the Hamming distance. In the simple case
of black-and-white images, there happens that any two images $A$ and $B$ are
two subsets of the rectangular lattice $K$ of dimension $N\times M$ given by
the screen, and it is possible to associate to $A$ ($B$) the vector
$\underline{x}_{A}(i),\ \ i\in K,\ \ (\underline{x}_{B},\ \ i\in K)$ such that

$x_{A}(i)=1\ \ if\ \ i\in A,x_{A}(i)=0\ \ if\ \ i\in K-A.$ (27)

It is possible to define the distance between the two images $A$ and $B$,
$d(A,B)$, thus as the Hamming distance between thee two vectors
$\underline{x}_{A}$ and $\underline{x}_{b}$ as

$d(A,B)=\frac{1}{N\times M}\sum_{i\in
K}\mid\underline{x}_{A}(i)-\underline{x}_{B}(i)\mid.$ (28)

The convergence theorem is valid also for non-discretised figures, as stated
in the above. Using the Hamming distance allows one to save a factor $N$ at
each step of the procedure (which is demonstrated in the below); indeed, to
determine $h_{A,B}$ one needs $N$ operations.
A fractal figure $A$ can be enormously compressed simply after substituting
the vector $\underline{x}_{A}$ which determines it by the coefficients of the
transformations $w_{1},...,w_{M}$ which are part of the IFS which generates
it. Within this framework, a fractal is defined as any set which can be
generated after some IFS. An US trading company founded by Bransley himself
elaborated a program, now in commerce, which manages to reduce a picture by
Rembrandt in one file of of extension of $50k$, and, form this, it manages to
reconstruct the original picture with a very high precision.
This program apparently solves the following inverse problem: given an image
$L=\\{x_{1},...,x_{n}$, the determination is requested of the transformation
generated by an IFS $w_{1}(z),...,w_{M}(z)$ as

$W(z)=\cup_{i=1}^{i=k}w_{i}(z),$ (29)

which approximates it in the sense that for any point $z$ chosen in the
lattice which defines the pixels f the screen of the calculator, one has

$lim_{m\rightarrow\infty}d(W^{m}(z),L)=0.$ (30)

It is difficult to solve such a problem unless one has a previous catalogue of
model-images and of transformations IFS which generate it. There exists a
very-well known theorem ref16  which partially simplifies this the problem: if
one manages to determine $\epsilon$ such that $\exists
w=\\{w_{1},...,w_{k}\\}$ for which

$d(w(L),L)\leq\epsilon,$ (31)

where $w(L)=\cup_{x\in L}w(x)$, then

$lim_{m\rightarrow\infty}d(w^{m}(L),L)\leq\frac{\epsilon}{1-s};$ (32)

then, the solution of the problem Eq. (30) allows one to approximate at best
the given image $L$. Problem Eq. (30) is not trivial; nevertheless, it is
possible to try to solve it by Monte-Carlo method. For this, one considers
$d(w(L),L)$ as a cost function to be minimised, the states variables are
considered as the coefficients $(a_{i},b_{i},c_{i},f_{i})_{i=1}^{k}$ of the
IFS $W(w_{1},...,w_{k})$; one thereafter discretises the space of
coefficients, and, then, starting from a random initial configuration
$\underline{\eta}\equiv(a^{0}_{1},b_{1}^{0},...,f_{k}^{0})$, one introduces
the transitions
$\underline{(}\eta)\rightarrow\underline{\eta}_{e}^{\prime}=\underline{\eta}_{e}\pm
h$ with probability

$e^{-\beta\Delta}/1+e^{-\beta\Delta}$ (33)

where

$\Delta=d(w(\underline{\eta^{\prime}})(L),L)-d(w(\underline{\eta})(L),L)$ (34)

if $\Delta>0$; and with probability

$\frac{1}{1+e^{-\beta\Delta}}$ (35)

if $\Delta<0$. This method was applied in the fractal leaf-like structure
ref16 , and to the Serpinski triangle: the results were not satisfying, as it
was not possible to reach $\Delta<0.64$ which provides one with a non-suitable
reconstruction ref17 . By dividing the image in square-like boxes of sides
$20\times 20$ pixels, and then by using an approach which is more similar to
the classical vector-quantization one, and by trying to compress and
reconstruct the IFS in each of these blocks, it was possible to obtain a
result which was nevertheless not of interest.
The substitution of the Hamming distance with the Hausdorff distance renders
the procedure less apt from the point of view of the time requested by the
calculator, as the Hausdorff distance requires $2N^{2}$ operations, but the
numerical results on the limiting value of $\Delta$ were of the same type.
Probably the fractal approach is not to be used by itself for each block, but
to be associated with other types of compression. As suggested by Hertz ref13
, it is possible to make use of a network which only one intermediate layer.
It is necessary to consider now a network with $M=2P$ input neurons, $M/2$
intermediate neurons, and $M$ output neurons.
The vector $x_{1},...,x_{M}$ is given to the input layer, where $M$ is the
umber of pixels inside such one block. The datum $x_{1}$ is presented to the
input of the first neuron, the datum $x_{2}$ is presented to the input of the
second neuron, and so on. The neurons from the intermediate layer are
connected with all the input neurons and with all the output neurons. Be
$z_{1},...,z_{[}M/2]$ the values of the output of the neurons of the
intermediate layer; the following decomposition holds

$z_{k}=\sigma_{1}\left(sum_{j=1}^{j=M}w_{kj}^{1}x_{j}\right),\ \
k=1,...,[M/2];$ (36)

the output is written as

$y_{l}=\sigma_{2}\left(w_{lk}^{2}z_{k}\right),\ \ l=1,...,M,$ (37)

where $w_{lk}^{2}$ and $w_{kj}^{1}$ are the synaptic-coupling intensities
between the intermediate layer and that of output and between the output state
and the intermediate one respectively; and $\sigma_{1}$ and $\sigma_{2}$ are
the two neural non-linear output functions. It is now the suitable duty to
implement the a learning of the net in a way such that, for each black-and-
white intensity vector $x_{1}^{\alpha},...,x_{M}^{\alpha}$ , associated to
each of the $Q$ blocks $B_{\alpha}$, $\alpha=1,...,Q$ in which the image is
divided, the output of the net is still $x_{1}^{\alpha},...,x_{M}^{\alpha}$:
in this case, it is crucial to outline that the image has been compresses of a
factor $2$ because the following step is possible

$\bigcup_{\alpha=1}^{\alpha=Q}(x_{1}^{\alpha},...,x_{M}^{\alpha})\rightarrow\bigcup_{\alpha=1}^{\alpha=Q}(z_{1}^{\alpha},...,z_{[M/2]}^{\alpha}).$
(38)

It is obtained by minimising the following cost function VEDI

$E=\sum_{\alpha=1}^{\alpha=Q}\sum_{l=1}^{l=M}\left(x_{l}^{\alpha}-\sigma_{2}\left(\sum_{k=1}^{k=[M/2]}w_{lk}^{2}\sigma_{1}\sum_{j=1}^{j=M}w_{kj}^{1}x_{j}^{\alpha}\right)\right)^{2}.$
(39)

The procedure can be iterated by using a net of the same type, whose input
layer is the intermediate layer of the previous one, and an intermediate layer
consisting of $[[N/2]2]$ and so on. The reconstruction obtained after the
algorithm Eq. (38) allows one to obtain results which are much improved with
respect to those obtained within the IFS method; nevertheless, the procedure
requests many iterations before an interesting reduction factor is obtained,
and it requires again the use of other techniques to get significant results.
The problem caused by the repetition of this method is that the number of
component of all the vectors $\underline{z}_{\alpha}$ obtained plus the number
of the bytes necessary to individuate the weights of the networks used to
complete the ’compressed’ vectors $\underline{z}_{\alpha}$ is equal or greater
than the number of bytes that determine the original image. The number id
equal in the case of a fractal leaf-like structure.Of course it is possible to
try to find a minimum $M$ each time such a network is used. The most general
approach consists in reducing the initial number of bytes which constitutes
the image before applying the compression techniques.
The vector quantization is therefore described. The discussion about the
problem of the images brought to the introduction of this method, which has
nevertheless very general features. An image on a high-resolution screen i
seen as a continuous form; this way, the discrete aspect connected with the
lattice of pixels which constitute the screen. The idea of quantization
consists in the approximation of the color or of the shade of gray in that
portion of image in which these features do not have significant variations.
According to the previously-accomplished analysis, the image is divided in
sets such that, inside each of these sets, there are small variation of
intensity of black and white. One associates the mean value of the intensity
in the zone with each of these ’small items’ of the partition. It is easily
shown that this algorithm is a particular case of the following procedure,
which is named vector quantization. On starts form observing that, given
$a_{1},...,x_{N}$ are the values of the intensities of gray in one of such
homogeneous zones, and given $a$ their mean value, $a$ is the minimum of the
following cost function

$E=\sum_{i=1}^{i=N}(x_{i}-a)^{2}.$ (40)

It is now possible to consider a more general situation, in which there is a
compact subspace $K\subset\mathcal{R^{d}}$ and a set of data $\underline{v}\in
K$ distributed according to a probability distribution
$P(\underline{v})d\underline{v}$. According to this paradigm, it is possible
to make a comparison with the case in which there are colors; accordingly, one
considers an intensity vector instead of a scalar, each component of which
corresponding to the intensity of a determined colour in a certain point of
the image. The set $K$ is the set of possible vectors $v$ associated with the
image. It is possible to generalise the latter observation by posing the
probelm of determining the vectors $\underline{w}_{1},...,\underline{w}_{m}$
such that the following cost function be continuous

$E=\sum_{i=1}^{i=m}\int_{A}(\underline{w}_{i})d\underline{v}_{i}P(\underline{v})\mid\mid\underline{v}-\underline{w}_{i}\mid\mid^{2}$
(41)

Where the sets $A(\underline{w}_{i}),i=1,...,m$ there items of the Vornoi
partition of the set $K$, which is defined in the following way:

$A(\underline{w}_{i})\equiv\\{\underline{w}\in
K:\mid\mid\underline{w}-\underline{w}_{i}\mid\mid<\mid\mid\underline{w}-\underline{w}_{j}\mid\mid,\
\ \forall j\neq i\\}.$ (42)

It is straightforward to verify that $A(\underline{w}_{i})\cap
A(\underline{w}_{j})=0$ for $i\neq j$, and that
$\cup_{i=1}^{i=m}A\underline{w}_{i}=K$.
It is necessary now to apply the method of the gradient to find the minimum of
$E$, as

$\Delta\underline{w}_{i}=-\eta\int_{A(\underline{w}_{i})}d\underline{v}P(\underline{v})(\underline{v}-\underline{v_{i}}),$
(43)

where $\eta$ is an opportune coefficient which varies at the number of
iterations of the procedure of updating of the $\underline{w}_{i}$ varies,
where the latter procedure is one of the same type which has been applied in
Section I. It is needed to introduce at the present step the formula which
approximates the integrals defined on a limited support via the summation of
random numbers:

$\int_{A}f(x)dx\simeq\sum_{i}f(x_{i}).$ (44)

In Eq. (44), $x_{i}$ in the summation are random numberrs chosen in the set
$A$ with probability $dx$. Byputting togehter Eq. (43) with Eq. (44) and by
introducing an index $n$, where the latter is counting the number of
iterations, it is possible to write the method of the gradient for $E$ in the
following way:

$\displaystyle\underline{w}_{i}(n+1)=\underline{w}_{i}(n)-\eta(\underline{v}_{n}-\underline{w}_{i}(n))\
\ if\ \ \underline{v}_{n}\in A(\underline{w}_{i}(n)),$
$\displaystyle\underline{w}_{j}(n+1)=\underline{w}_{j}(n)\ \ if\ \
\underline{v}(n)\notin A(\underline{w}_{j}(n))\ \ i,j=1,2,...,m.$

This procedure is easily ’translated’ into an algorithm of neural learning.
$m$ neurons are considered, and to each neuron $i$ there is initially
associated a vector $\underline{w}_{i}(0)$, where the latter is randomly
chosen, i.e. $i=1,...,m$, and a set of data as that in the above. The
$\underline{w}_{i}$ are named ’weight vectors at the time $n$’, and they are
equivalent to the synaptic intensities intorduced in the other neural
networks. Be $\underline{v}(n)$ a given succession of vectors in $K$, each of
which is chosen with probability $P(\underline{v})$, and be $\eta(n)$ a
succession of positive constants opportunely chosen. The previous algorithm is
equivalent to the following learning rule:

$\displaystyle\underline{w}_{i}(n+1)=\underline{w}_{i}(n)-\eta(n)(\underline{v}_{n}-\underline{w}_{i}(n))\
\
\mid\mid\underline{v}(n)-\underline{w}_{i}(n)\mid\mid<\mid\mid\underline{v}(n)-\underline{w}_{j}(n)\mid\mid\
\ \forall j\neq i,$ $\displaystyle\underline{w}_{j}(n+1)=\underline{w}_{j}(n)\
\ otherwise,\ \ i,j=1,...,m.$

Eq. (46) demonstrates the learning of the Kohonen networks in the particular
case where the weight of the neuron $i$ is the only item varying, which
satisfies the condition

$\mid\mid\underline{v}(n)-\underline{w}_{i}(n)\mid\mid<\mid\mid\underline{v}(n)-\underline{w}_{j}(n)\mid\mid\
\ \forall j\neq i.$ (47)

##  IX The winning neuron

As an insight form Eq. (47), the neuron whose weight is qualified by the
minimum distance from the vector $\underline{v}$ is named ’the winning
neuron’. In the case there should be more than one winning neuron, one neuron
is randomly chosen among the winning neurons. This dynamics is a particular
case of the Kohonen dynamics, in which the weights of the neurons which belong
to a neighbourhood of the winning neuron are varying, too, where the
neighbourhood is decreasing with $n$. The set of vectors
$\underline{w}_{i}=lim_{n\rightarrow\infty}\underline{w}_{i}(n)$ is a
generalisation of the average of a homogeneous set of data, and it can be also
considered as a set which is obtained by compressing the data which belong to
$K$. The compression consists in associating the vector $\underline{w}_{i}$ to
all the vectors which belong to $A(\underline{w}_{i})$. The Kohonen network
can be used for the recognition of the voice and the limiting weights obtained
are the characteristic vectors which represent the phonemes of a language
which is being spoken. The recognition is obtained as follows: if a datum
$\underline{v}$ is presented to the network, the $\underline{v}$ becomes
active only if the neuron which satisfies the condition of the winning neuron:
$\underline{v}\in A(\underline{w}_{i})$. In ref14  a theorem is demonstrated,
which provides with the sufficient conditions for which the weights
$\underline{w}_{i}(n)$ be convergent in probability to a limiting value. The
learning process is a stochastic process, as the samples which are presented
to the network are taken with a certain probability distribution, and thus the
convergence of the weights to a given limit acquires a meaning only within a
probabilistic framework.

##  X Perspectives

There are two conditions requested in ref14 : the first condition is a certain
type of decreasing of $\eta$; the second condition is that the derivative of a
certain Ljapunov function, defined on $K^{m}$, be always negative but except
for a limiting value. In other words, there must exist an absolute minimum for
the convergence in the probability be accomplished, as it must always happen
in the cases when these types of interpretations are applied. Unfortunately,
such a Ljapunov function is rather complicated, and there are several local
minima which correspond to the metastable states found in ref18 . Letting the
neighbourhood of the winning neuron vary with $n$ allows one to avoid these
metastable states; nevertheless, up to now no reason has been justified to
demonstrate it. Differently, in ref14 , an alternative method is proposed,
which is based on the analogy of this situation with the research of the
absolute minimum for the desordered sysyems.
The paper is organised as follows.
In Section I, the application of neural networks to reconstruct series of data
form non-linear non-deterministic transformations is manifested.
In Section II, the Hertz method is surveyed.
In Section III, the partition of the phase-space methodology is investigated.
In Section IV, patterns-learning neural networks are expressed.
In Section V, the Hertz method is further studied.
In Section VI, layers networks are proposed.
In Section VII, the images-compression methods are investigated.
In Section VIII, the contractive transformations are further scrutinised.
In Section IX, the qualities of the winner neuron are indicated.
In Section X, prospective researches are confronted.

## References

  * (1) K. Stokbo, D. K. Umbaerger, J. A. Hertz, Nordita internal note, Blegdamsevej 17, 2100 Copenhagen 0, Denmark, 1990.
  * (2) G. Cimino, Laurea Thesis, Academic year 1992-1993.
  * (3) J. Feng, B. Tirozzi, Preprint, Mathematics Department, University of Rome ’La Sapienza’, Piazzale ALdo Moro, 5- 00185 Rome, Italy.
  * (4) G. Rotundo, Supplemento al N. 9 di Comunicazioni Scientifiche Italiane, pagg. 113-127 (1993).
  * (5) D. P. Kroese, T. Brereton, T. Taimre, Z. I. Botev, Why the Monte Carlo method is so important today, WIREs Comput Stat. 6, 386 (2014).
  * (6) M. Bransley, Fractals Everywhere, San Diego, Academic Press Inc (1988).
  * (7) B. Tirozzi, O. M. Lecian, in preparation,
  * (8) B. B. Mandelbrot, Fractals and Chaos: The Mandelbrot Set and Beyond, Springer, New York, USA (2010).
  * (9) E. Obermayer, K. Shulten, K. Ritter, Self-organizing maps: stationary states, metastability and convergence rate, Biological Cybernetics, 67, 35 (1992).


25% | 0.054 | 0.156 | 0.069 | 0.178 | 0.107 | 0.217 | 0.169 | 0.304 | 0.265 | 0.364 | 0.180 | 0.292 | 0.106 | 0.236 | 0.080 | 0.189 | 0.090 | 0.203 | 0.140 | 0.262 | 0.102 | 0.227 | 0.259 | 0.298 | 0.098 | 0.197 | 0.192 | 0.266 | 0.127 | 0.224
37.5% | 0.072 | 0.180 | 0.084 | 0.196 | 0.120 | 0.230 | 0.220 | 0.347 | 0.296 | 0.382 | 0.215 | 0.318 | 0.124 | 0.258 | 0.102 | 0.212 | 0.109 | 0.222 | 0.174 | 0.293 | 0.135 | 0.261 | 0.323 | 0.326 | 0.119 | 0.215 | 0.215 | 0.277 | 0.160 | 0.245
50% | 0.107 | 0.216 | 0.102 | 0.215 | 0.141 | 0.248 | 0.293 | 0.402 | 0.334 | 0.404 | 0.257 | 0.347 | 0.165 | 0.299 | 0.133 | 0.240 | 0.137 | 0.248 | 0.215 | 0.325 | 0.179 | 0.298 | 0.423 | 0.366 | 0.158 | 0.242 | 0.257 | 0.297 | 0.235 | 0.279
Avg | 0.069 | 0.173 | 0.078 | 0.187 | 0.115 | 0.224 | 0.202 | 0.329 | 0.284 | 0.373 | 0.201 | 0.306 | 0.117 | 0.246 | 0.094 | 0.201 | 0.103 | 0.214 | 0.161 | 0.279 | 0.122 | 0.245 | 0.304 | 0.317 | 0.114 | 0.209 | 0.211 | 0.275 | 0.157 | 0.239
$ETTh2$ | 12.5% | 0.039 | 0.125 | 0.040 | 0.130 | 0.057 | 0.152 | 0.187 | 0.319 | 0.101 | 0.231 | 0.100 | 0.216 | 0.095 | 0.212 | 0.042 | 0.133 | 0.044 | 0.138 | 0.305 | 0.431 | 0.163 | 0.289 | 0.090 | 0.167 | 0.058 | 0.134 | 0.085 | 0.162 | 0.091 | 0.172
25% | 0.044 | 0.135 | 0.046 | 0.141 | 0.061 | 0.158 | 0.279 | 0.390 | 0.115 | 0.246 | 0.127 | 0.247 | 0.137 | 0.258 | 0.049 | 0.147 | 0.050 | 0.149 | 0.322 | 0.444 | 0.206 | 0.331 | 0.097 | 0.176 | 0.060 | 0.138 | 0.088 | 0.165 | 0.101 | 0.179
37.5% | 0.051 | 0.147 | 0.052 | 0.151 | 0.067 | 0.166 | 0.400 | 0.465 | 0.126 | 0.257 | 0.158 | 0.276 | 0.187 | 0.304 | 0.056 | 0.158 | 0.060 | 0.163 | 0.353 | 0.462 | 0.252 | 0.370 | 0.105 | 0.185 | 0.064 | 0.144 | 0.091 | 0.169 | 0.118 | 0.190
50% | 0.059 | 0.158 | 0.060 | 0.162 | 0.073 | 0.174 | 0.602 | 0.572 | 0.136 | 0.268 | 0.183 | 0.299 | 0.232 | 0.341 | 0.065 | 0.170 | 0.068 | 0.173 | 0.369 | 0.472 | 0.316 | 0.419 | 0.118 | 0.199 | 0.071 | 0.153 | 0.097 | 0.176 | 0.150 | 0.205
Avg | 0.048 | 0.141 | 0.049 | 0.146 | 0.065 | 0.163 | 0.367 | 0.436 | 0.119 | 0.250 | 0.142 | 0.259 | 0.163 | 0.279 | 0.053 | 0.152 | 0.055 | 0.156 | 0.337 | 0.452 | 0.234 | 0.352 | 0.102 | 0.182 | 0.063 | 0.142 | 0.090 | 0.168 | 0.115 | 0.186
$ECL$ | 12.5% | 0.080 | 0.194 | 0.085 | 0.202 | 0.055 | 0.160 | 0.196 | 0.321 | 0.102 | 0.229 | 0.092 | 0.214 | 0.107 | 0.237 | 0.093 | 0.210 | 0.089 | 0.210 | 0.218 | 0.326 | 0.190 | 0.308 | 0.214 | 0.293 | 0.079 | 0.182 | 0.181 | 0.271 | 0.091 | 0.196
25% | 0.087 | 0.203 | 0.089 | 0.206 | 0.065 | 0.175 | 0.207 | 0.332 | 0.121 | 0.252 | 0.118 | 0.247 | 0.120 | 0.251 | 0.097 | 0.214 | 0.096 | 0.220 | 0.219 | 0.326 | 0.197 | 0.312 | 0.266 | 0.324 | 0.094 | 0.199 | 0.194 | 0.280 | 0.115 | 0.217
37.5% | 0.094 | 0.211 | 0.094 | 0.213 | 0.076 | 0.189 | 0.219 | 0.344 | 0.141 | 0.273 | 0.144 | 0.276 | 0.136 | 0.266 | 0.102 | 0.220 | 0.104 | 0.229 | 0.222 | 0.328 | 0.203 | 0.315 | 0.339 | 0.366 | 0.117 | 0.223 | 0.220 | 0.296 | 0.152 | 0.245
50% | 0.101 | 0.220 | 0.100 | 0.221 | 0.091 | 0.208 | 0.235 | 0.357 | 0.160 | 0.293 | 0.175 | 0.305 | 0.158 | 0.284 | 0.108 | 0.228 | 0.113 | 0.239 | 0.228 | 0.331 | 0.210 | 0.319 | 0.447 | 0.424 | 0.156 | 0.259 | 0.264 | 0.324 | 0.222 | 0.286
Avg | 0.090 | 0.207 | 0.092 | 0.210 | 0.072 | 0.183 | 0.214 | 0.339 | 0.131 | 0.262 | 0.132 | 0.260 | 0.130 | 0.259 | 0.100 | 0.218 | 0.101 | 0.225 | 0.222 | 0.328 | 0.200 | 0.313 | 0.316 | 0.352 | 0.112 | 0.216 | 0.215 | 0.293 | 0.145 | 0.236
$Weather$ | 12.5% | 0.026 | 0.049 | 0.025 | 0.045 | 0.029 | 0.049 | 0.057 | 0.141 | 0.047 | 0.101 | 0.039 | 0.084 | 0.041 | 0.107 | 0.027 | 0.051 | 0.026 | 0.047 | 0.037 | 0.093 | 0.031 | 0.076 | 0.041 | 0.042 | 0.026 | 0.031 | 0.038 | 0.041 | 0.038 | 0.036
25% | 0.028 | 0.052 | 0.029 | 0.052 | 0.031 | 0.053 | 0.065 | 0.155 | 0.052 | 0.111 | 0.048 | 0.103 | 0.064 | 0.163 | 0.029 | 0.056 | 0.030 | 0.054 | 0.042 | 0.100 | 0.035 | 0.082 | 0.045 | 0.045 | 0.027 | 0.032 | 0.040 | 0.041 | 0.043 | 0.039
37.5% | 0.033 | 0.060 | 0.031 | 0.057 | 0.035 | 0.058 | 0.081 | 0.180 | 0.058 | 0.121 | 0.057 | 0.117 | 0.107 | 0.229 | 0.033 | 0.062 | 0.032 | 0.060 | 0.049 | 0.111 | 0.040 | 0.091 | 0.048 | 0.049 | 0.030 | 0.034 | 0.042 | 0.043 | 0.055 | 0.043
50% | 0.037 | 0.065 | 0.034 | 0.062 | 0.038 | 0.063 | 0.102 | 0.207 | 0.065 | 0.133 | 0.066 | 0.134 | 0.183 | 0.312 | 0.037 | 0.068 | 0.037 | 0.067 | 0.053 | 0.114 | 0.046 | 0.099 | 0.054 | 0.054 | 0.033 | 0.037 | 0.044 | 0.045 | 0.069 | 0.048
Avg | 0.031 | 0.056 | 0.030 | 0.054 | 0.060 | 0.144 | 0.076 | 0.171 | 0.055 | 0.117 | 0.052 | 0.110 | 0.099 | 0.203 | 0.032 | 0.059 | 0.031 | 0.057 | 0.045 | 0.104 | 0.038 | 0.087 | 0.047 | 0.048 | 0.029 | 0.034 | 0.041 | 0.043 | 0.051 | 0.042

Table 19: Results for the imputation task in the time-steps missing at random
setting. Results averaged across 4 different masking rates: {12.5%, 25%,
37.5%, 50%}. Statistical interpolation methods such as forward and backward
fill (naive), linear, nearest, and cubic interpolation perform better than
many transformer-based baselines. Therefore, we consider the much harder,
patches missing at random setting in our experiments.

Dataset | Mask Ratio | $\mathtt{MOMENT_{0}}$ | $\mathtt{MOMENT_{LP}}$ | GPT4TS | TimesNet | Naive | Linear | Nearest | Cubic
---|---|---|---|---|---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
Weather | 0.125 | 0.085 | 0.131 | 0.033 | 0.073 | 0.036 | 0.076 | 0.035 | 0.096 | 0.105 | 0.089 | 0.050 | 0.055 | 0.069 | 0.067 | 0.373 | 0.115
0.250 | 0.079 | 0.130 | 0.036 | 0.078 | 0.030 | 0.071 | 0.037 | 0.100 | 0.127 | 0.104 | 0.075 | 0.065 | 0.094 | 0.076 | 0.297 | 0.125
0.375 | 0.081 | 0.128 | 0.034 | 0.075 | 0.030 | 0.070 | 0.035 | 0.099 | 0.120 | 0.111 | 0.066 | 0.069 | 0.082 | 0.080 | 0.904 | 0.176
0.500 | 0.081 | 0.129 | 0.035 | 0.075 | 0.026 | 0.069 | 0.035 | 0.098 | 0.124 | 0.127 | 0.066 | 0.078 | 0.086 | 0.090 | 0.831 | 0.197
Mean | 0.082 | 0.130 | 0.035 | 0.075 | 0.031 | 0.071 | 0.036 | 0.098 | 0.119 | 0.108 | 0.065 | 0.067 | 0.083 | 0.078 | 0.601 | 0.153
ETTh1 | 0.125 | 0.430 | 0.417 | 0.160 | 0.239 | 0.183 | 0.242 | 0.158 | 0.254 | 1.008 | 0.602 | 0.583 | 0.466 | 0.712 | 0.523 | 0.985 | 0.661
0.250 | 0.373 | 0.392 | 0.142 | 0.238 | 0.278 | 0.267 | 0.154 | 0.261 | 1.311 | 0.686 | 0.833 | 0.540 | 0.954 | 0.581 | 1.433 | 0.772
0.375 | 0.398 | 0.398 | 0.121 | 0.228 | 0.232 | 0.263 | 0.195 | 0.274 | 1.317 | 0.703 | 0.843 | 0.572 | 0.973 | 0.613 | 2.615 | 1.028
0.500 | 0.408 | 0.403 | 0.132 | 0.231 | 0.213 | 0.243 | 0.192 | 0.267 | 1.103 | 0.643 | 0.840 | 0.559 | 0.963 | 0.601 | 3.681 | 1.204
Mean | 0.402 | 0.403 | 0.139 | 0.234 | 0.227 | 0.254 | 0.175 | 0.264 | 1.185 | 0.658 | 0.775 | 0.534 | 0.900 | 0.579 | 2.178 | 0.916
ETTh2 | 0.125 | 0.122 | 0.235 | 0.051 | 0.150 | 0.115 | 0.215 | 0.163 | 0.277 | 0.196 | 0.285 | 0.105 | 0.208 | 0.134 | 0.225 | 0.452 | 0.404
0.250 | 0.127 | 0.237 | 0.079 | 0.177 | 0.114 | 0.216 | 0.161 | 0.280 | 0.210 | 0.291 | 0.120 | 0.220 | 0.154 | 0.240 | 0.831 | 0.524
0.375 | 0.124 | 0.237 | 0.056 | 0.155 | 0.110 | 0.216 | 0.170 | 0.291 | 0.229 | 0.310 | 0.142 | 0.243 | 0.175 | 0.262 | 1.571 | 0.672
0.500 | 0.127 | 0.242 | 0.056 | 0.154 | 0.098 | 0.207 | 0.186 | 0.295 | 0.265 | 0.329 | 0.171 | 0.264 | 0.199 | 0.279 | 4.823 | 0.966
Mean | 0.125 | 0.238 | 0.061 | 0.159 | 0.109 | 0.213 | 0.170 | 0.286 | 0.225 | 0.304 | 0.135 | 0.234 | 0.166 | 0.252 | 1.920 | 0.641
ETTm1 | 0.125 | 0.179 | 0.278 | 0.069 | 0.170 | 0.078 | 0.147 | 0.089 | 0.200 | 0.273 | 0.293 | 0.094 | 0.183 | 0.147 | 0.217 | 0.334 | 0.345
0.250 | 0.206 | 0.290 | 0.071 | 0.169 | 0.071 | 0.144 | 0.080 | 0.194 | 0.395 | 0.341 | 0.114 | 0.202 | 0.171 | 0.234 | 0.539 | 0.424
0.375 | 0.209 | 0.289 | 0.069 | 0.163 | 0.076 | 0.146 | 0.091 | 0.199 | 0.475 | 0.378 | 0.188 | 0.242 | 0.257 | 0.274 | 0.842 | 0.528
0.500 | 0.215 | 0.294 | 0.086 | 0.169 | 0.081 | 0.149 | 0.088 | 0.197 | 0.679 | 0.448 | 0.265 | 0.291 | 0.346 | 0.316 | 1.715 | 0.680
Mean | 0.202 | 0.288 | 0.074 | 0.168 | 0.076 | 0.146 | 0.087 | 0.198 | 0.455 | 0.365 | 0.165 | 0.229 | 0.230 | 0.260 | 0.858 | 0.494
ETTm2 | 0.125 | 0.076 | 0.183 | 0.032 | 0.108 | 0.043 | 0.126 | 0.128 | 0.233 | 0.087 | 0.164 | 0.049 | 0.117 | 0.062 | 0.132 | 0.237 | 0.262
0.250 | 0.084 | 0.187 | 0.029 | 0.105 | 0.046 | 0.129 | 0.101 | 0.207 | 0.104 | 0.182 | 0.057 | 0.132 | 0.073 | 0.146 | 0.373 | 0.309
0.375 | 0.076 | 0.181 | 0.032 | 0.109 | 0.059 | 0.137 | 0.116 | 0.225 | 0.115 | 0.196 | 0.063 | 0.141 | 0.078 | 0.154 | 0.626 | 0.376
0.500 | 0.077 | 0.183 | 0.031 | 0.110 | 0.059 | 0.140 | 0.103 | 0.212 | 0.144 | 0.222 | 0.080 | 0.162 | 0.102 | 0.177 | 0.899 | 0.477
Mean | 0.078 | 0.184 | 0.031 | 0.108 | 0.052 | 0.133 | 0.112 | 0.220 | 0.113 | 0.191 | 0.062 | 0.138 | 0.079 | 0.152 | 0.534 | 0.356
Electricity | 0.125 | 0.251 | 0.370 | 0.095 | 0.211 | 0.069 | 0.180 | 0.126 | 0.248 | 1.350 | 0.818 | 0.458 | 0.466 | 0.608 | 0.492 | 0.924 | 0.610
0.250 | 0.249 | 0.372 | 0.093 | 0.211 | 0.073 | 0.184 | 0.121 | 0.246 | 1.447 | 0.857 | 0.654 | 0.554 | 0.815 | 0.582 | 1.619 | 0.769
0.375 | 0.250 | 0.371 | 0.094 | 0.211 | 0.071 | 0.181 | 0.125 | 0.248 | 1.518 | 0.888 | 0.836 | 0.637 | 1.031 | 0.675 | 2.507 | 0.959
0.500 | 0.250 | 0.371 | 0.092 | 0.210 | 0.075 | 0.185 | 0.126 | 0.249 | 1.581 | 0.915 | 1.002 | 0.712 | 1.239 | 0.766 | 3.978 | 1.213
Mean | 0.250 | 0.371 | 0.094 | 0.211 | 0.072 | 0.183 | 0.124 | 0.248 | 1.474 | 0.869 | 0.737 | 0.592 | 0.923 | 0.629 | 2.257 | 0.888

Table 20: Imputation Results. MOMENT achieves state-of-the-art imputation
results in both zero-shot and linear probe fine-tuning settings.

###  D.5 What is MOMENT Learning?

To investigate what MOMENT is learning, we conducted a series of experiments
using synthetically generated sine waves to evaluate MOMENT’s ability to
capture changes in trend, amplitude, frequencies, baselines, and phase of
time-series. In each experiment, $c$ controls the factor of interest, for
example the power of the trend polynomial $c\in(\frac{1}{8},8)$ Oreshkin et
al. [2020] (Fig. 8), and frequency $c\in(1,32)$ of the generated sine waves
(Fig. 8). We generate multiple sine waves by varying $c$, derive their
sequence-level representations using MOMENT (Sec. 3.4), and visualize them in
a 2- dimensional space using PCA and t-SNE van der Maaten [2014] in Fig. 4 and
Fig. 6.

We also study the composition of the learnable mask embedding and the
relationship between frequency and reconstruction error in a zero-shot
setting. We find that the learned mask embedding is approximately composed of
numbers drawn from the standard normal and that MOMENT can reconstruct lower
frequency signals better. We observed a curious spike in reconstruction error
around time-series of frequency $c=64$. (Fig. 10)

(i) Trend |  (ii) Amplitude |  (iii) Frequency |  (iv) Basline Shift |  (v) Auto-correlation
---|---|---|---|---
Figure 6: What is MOMENT learning? Structure in the PCA (top) and t-SNE
(bottom) visualizations of the embeddings of synthetically generated sinusoids
suggest that MOMENT can capture subtle trend, scale, frequency, and auto-
correlation information. $\epsilon$ denotes gaussian noise with $0$ mean and
$0.1$ standard deviation. $c$ controls the factor of interest, i.e. the power
of the trend polynomial, amplitude, and frequency of the sine waves in
experiments (i), (ii) & (iii), respectively. Figure 7: PCA (top) and t-SNE
(bottom) visualizations of representations learned by MOMENT on the 5 largest
UCR datasets. Different colors represent different classes. Even without
dataset-specific fine-tuning, MOMENT learns distinct representations for
different classes (i) Trend
---
(ii) Amplitude
(iii) Frequency
(iv) Baseline Shift
(v) Phase
Figure 8: Examples of sinusoids used in the interpretability experiments.  |  |  |
---|---|---|---
Figure 9: Masking using a [MASK] tokens allows MOMENT to reconstruct time-
series in a zero-shot setting. Since zeros contain information, they bias
model predictions. For two datasets ETTh1 and Weather, we mask the time-series
with zeros on the left and special mask tokens on the right. Figure 10: (Left)
MOMENT can reconstruct lower frequency time-series better in a zero-shot
setting. (Right) The learned mask token is approximately composed of numbers
drawn from a standard normal.

###  D.6 Impact of Model Size

We studied the impact of scaling the size of the model and training data on
zero-shot forecasting, imputation, and anomaly detection performance. As shown
in Fig. x, we found that increasing the size of the model generally improved
zero-shot performance (lower MSE and sMAPE, higher VUS-ROC). Since varying the
size of the pre-training dataset is expensive, we instead look at the zero-
shot performance of model checkpoints before completing the first epoch. Our
findings suggest that increasing the diversity in training data may also
improve zero-shot performance.

###  D.7 Training losses

Figure 11: Training losses (MSE). A dashed vertical line denotes the first
epoch. All models were trained with a batch size of 131072 patches. (left)
Larger models obtain lower training loss. right Eventually, randomly
initialized MOMENT-small outperform the same model initialized with Flan-T5
weights. The figure on the right is in log scale.

###  D.8 Efficiency Analysis

Model | ETTh1-96
---|---
Total Param. (M) | Trainable Param. (M) | Mem. (MiB)
MOMENT | 347.53 | 6.29 | 2079
GPT4TS | 82.28 | 1.12 | 1031
TimesNet | 0.89 | 0.89 | 683
Time-LLM | 3623.71 | 254.37 | 4537

Table 21: Efficiency analysis of MOMENT against other forecasting models on
the ETTh1 with prediction horizon set to 96. MOMENT outperforms all the listed
models and has a fraction of parameters as the most recent LLM-based
forecasting method.

##  Appendix E Transparency Index

Sub-domain | Indicator | MOMENT
---|---|---
Data | Data size | 1
Data sources | 1
Data creators | 0
Data source selection | 1
Data curation | 1
Data augmentation | 1
Harmful data filtration | 0
Copyrighted data | 1
Data license | 1
Personal information in data | 1
Data Labor | Use of human labor | 1
Employment of data laborers | 1
Geographic distribution of data laborers | 1
Wages | 1
Instructions for creating data | 1
Labor protections | 1
Third party partners | 1
Data Access | Queryable external data access | 1
Direct external data access | 1
Compute | Compute usage | 1
Development duration | 1
Compute hardware | 1
Hardware owner | 1
Energy usage | 1
Carbon emissions | 1
Broader environmental impact | 0
Methods | Model stages | 1
Model objectives | 1
Core frameworks | 1
Additional dependencies | 1
Data Mitigations | Mitigations for privacy | 0
Mitigations for copyright | 0
| Upstream Subtotal | 75%

Sub-domain | Indicator | MOMENT
---|---|---
Model Basics | Input modality | 1
Output modality | 1
Model components | 1
Model size | 1
Model architecture | 1
Centralized model documentation | 1
Model Access | External model access protocol | 1
Blackbox external model access | 1
Full external model access | 1
Capabilities | Capabilities description | 1
Capabilities demonstration | 1
Evaluation of capabilities | 1
External reproducibility of capabilities evaluation | 0
Third party capabilities evaluation | 0
Limitations | Limitations description | 1
Limitations demonstration | 1
Third party evaluation of limitations | 0
Risks | Risks description | 1
Risks demonstration | 0
Unintentional harm evaluation | 0
External reproducibility of unintentional harm evaluation | 0
Intentional harm evaluation | 0
External reproducibility of intentional harm evaluation | 0
Third party risks evaluation | 0
Model Mitigations | Mitigations description | 0
Mitigations demonstration | 0
Mitigations | Mitigations evaluation | 0
External reproducibility of mitigations evaluation | 0
Third party mitigations evaluation | 0
Trustworthiness | Trustworthiness evaluation | 0
External reproducibility of trustworthiness evaluation | 0
Inference | Inference duration evaluation | 1
Inference compute evaluation | 1
| Model Subtotal | 51.5%

Table 22: Expected (left) upstream and (right) model transparency scores.
MOMENT has one of the highest upstream transparency. Our model transparency
scores are lower due to (third-party) harm, mitigations, trustworthiness
evaluation, which are not well understood for time-series modeling.

##  Appendix F Results Sources

Task | Method | Type | Reimplementation/ Rerun | Source
---|---|---|---|---
Long-horizon Forecasting | Time-LLM | LLM-based | $\checkmark$ | Time-LLM
GPT4TS | $\times$ | One Fits All
| PatchTST, Fedformer, Autoformer,
---
Stationary, ETSformer, LightTS,
Informer, Reformer
Transformer-based | $\times$ | One Fits All
Pyraformer, LogTrans | $\times$ | TimesNet
TimesNet, DLinear | Deep learning | $\times$ | One Fits All
N-BEATS | $\checkmark$ | N-BEATS
Short-horizon Forecasting | GPT4TS | LLM-based | $\checkmark$ | One Fits All
TimesNet | Deep learning | $\checkmark$ | TimesNet
N-BEATS | $\checkmark$ | N-BEATS
| AutoARIMA, AutoTheta, AutoETS,
---
Seasonal Naive, Naive, Random Walk
Statistical learning | $\checkmark$ | Nixtla Statsforecast Repository
Classification | GPT4TS | LLM-based | $\checkmark$ | One Fits All
TimesNet | Deep learning | $\checkmark$ | TimesNet
TS2Vec, T-Loss, TNC, TS-TCC, TST | Unsupervised Representation learning | $\times$ | TS2Vec
| CNN, Encoder, FCN, MCNN, MLP,
---
ResNet, t-LeNet, TWIESN
Deep learning | $\times$ | DL4TSC Repository
DTW | Statistical learning | $\times$ | TS2Vec
Anomaly Detection | GPT4TS | LLM-based | $\checkmark$ | One Fits All
TimesNet | Deep learning | $\checkmark$ | TimesNet
Anomaly Transformer | Transformer-based | $\checkmark$ | Anomaly Transformer
DGHL | Deep learning | $\checkmark$ | Time-Series Model Selection
$k$-NN | Statistical learning | $\checkmark$ | Time-Series Model Selection
Imputation | GPT4TS | LLM-based | $\checkmark$ | One Fits All
TimesNet | Deep learning | $\checkmark$ | TimesNet
| PatchTST, ETSformer, LightTS,
---
Fedformer,Stationary, Autoformer,
Informer, Reformer
Transformer-based | $\times$ | One Fits All
DLinear | Deep learning | $\times$ | One Fits All
Naive | Statistical learning | $\checkmark$ |  Pandas FFill, Pandas BFill
Linear, Nearest, Cubic | $\checkmark$ | Scipy Interp1D

Table 23: Source for the results for each baseline for all downstream task.

##  Appendix G Radar Plot

We generate a radar plot (Fig. 1) to visually compare MOMENT with GPT4TS and
TimesNet. The values obtained by each method for a given task are min-max
normalized with respect to the other methods for each of the 5 downstream
tasks. For imputation, long- and short-horizon forecasting, we report $1-$ the
normalized MSE or sMAPE for the methods on the weather and (subset of) M4
datasets, respectively. For classification and anomaly detection, we report
the average accuracy and VUS-ROC of the methods across all the datasets.