ifaamas [AAMAS ’23]Proc. of the 22nd International Conference on Autonomous
Agents and Multiagent Systems (AAMAS 2023)May 29 – June 2, 2023 London, United
KingdomA. Ricci, W. Yeoh, N. Agmon, B. An (eds.) 2023 2023 0 0 0 180 Institute
of Advanced Technology, University of Science and Technology of China Hefei
230026 China G60 STI Valley Industry & Innovation Institute, Jiaxing
University Jiaxing 314001 China corresponding author Key Laboratory of Medical
Electronics and Digital Health of Zhejiang Province, Jiaxing University
Jiaxing 314001 China Engineering Research Center of Intelligent Human Health
Situation Awareness of Zhejiang Provincey, Jiaxing University Jiaxing 314001
China

# Infomaxformer: Maximum Entropy Transformer for Long Time-Series Forecasting
Problem

Peiwang Tang<EMAIL_ADDRESS>and  Xianchao Zhang
<EMAIL_ADDRESS>
###### Abstract.

The Transformer architecture yields state-of-the-art results in many tasks
such as natural language processing (NLP) and computer vision (CV), since the
ability to efficiently capture the precise long-range dependency coupling
between input sequences. With this advanced capability, however, the quadratic
time complexity and high memory usage prevents the Transformer from dealing
with long time-series forecasting problem (LTFP). To address these
difficulties: (i) we revisit the learned attention patterns of the vanilla
self-attention, redesigned the calculation method of self-attention based the
Maximum Entropy Principle. (ii) we propose a new method to sparse the self-
attention, which can prevent the loss of more important self-attention scores
due to random sampling.(iii) We propose Keys/Values Distilling method
motivated that a large amount of feature in the original self-attention map is
redundant, which can further reduce the time and spatial complexity and make
it possible to input longer time-series. Finally, we propose a method that
combines the encoder-decoder architecture with seasonal-trend decomposition,
i.e., using the encoder-decoder architecture to capture more specific seasonal
parts. A large number of experiments on several large-scale datasets show that
our Infomaxformer is obviously superior to the existing methods. We expect
this to open up a new solution for Transformer to solve LTFP, and exploring
the ability of the Transformer architecture to capture much longer temporal
dependencies.

###### Key words and phrases:

Maximum Entropy, Transformer, Time-Series, Forecasting

##  1\. Introduction

Defined as an ordered dataset formed with time change, time-series refer to a
series of ordered observations acquired according to time sequence
cryer1986time , which is widely used in commercial and industrial fields, such
as biomedical field liu2018learning , economic and financial field
di2016artificial , electric power zhou2021informer  and transportation field
yin2016multivariate . As an important part of time-series analysis, time-
series forecasting mainly analyze the trend, periodicity, volatility and other
time-series patterns of time-series by using the time-series data observed in
history and the relevant rules that have been mastered, so as to predict the
situation in the future lai2018modeling ; bai2018empirical ; wu2021autoformer
. In practical applications, we can use a large number of past time-series to
achieve long-term prediction for the future, i.e., long time-series
forecasting problem (LTFP). Recent deep prediction models have made great
progress, especially Transformer based models jiang2022accurate ;
lim2021temporal ; liu2021pyraformer ; wu2020adversarial ; tang2022mtsmae . The
Transformer vaswani2017attention  shows better performance than the recurrent
neural network (RNN) model in modeling the long-term dependence of sequence
data, and has achieved the best results in the natural language processing
(NLP) devlin2018bert ; radford2018improving  and computer vision (CV) 2021An ;
he2022masked  fields, since its advanced self-attention mechanism.

However, there are still some problems in solving LTFP of existing Transformer
models. First, the self-attention mechanism has high performance, but also
brings high time complexity and memory usage liu2021swin ; yu2021glance .
Although some large-scale Transformer models have produced impressive results
in the NLP and CV fields radford2019language ; bao2021beit , they often
require dozens or even hundreds of GPUs during training, which limits the
possibility of Transformer models to solve LTFP. Although there have been some
researches on reducing the time complexity and memory usage of the self-
attention mechanism, only realize a limited reduce of complexity to
$\mathcal{O}(LlogL)$ zhou2021informer ; li2019enhancing ; kitaev2019reformer .
Moreover, some methods for reducing the complexity only randomly select dot-
product pairs, which will cause some performance loss and lead to the long-
term dependence of the sequences that cannot be well captured by the self-
attention mechanism. Second, it is unreliable to find the time dependence
directly from the time-series, because these dependencies may be masked by the
entangled temporal patterns.

Figure 1. Infomaxformer architecture

In order to better solve LTFP, our work explicitly and deeply discussed the
above problems, studied the sparsity of self-attention mechanism, decomposed
the time-series, and updated the network components. Finally, we have
conducted extensive experiments on five different datasets. The final
experimental results show that our proposed Infomaxformer can significantly
improve the accuracy of prediction, and is superior to other state-of-the-art
models. The contributions of this paper are summarized as follows:

  * •

We review the calculation method of self-attention mechanism from the
perspective of information entropy shannon1948mathematical , and sparse the
calculation of self-attention by using the Maximum Entropy Principle
jaynes1957information  to reduce the time complexity.

  * •

In view of the data characteristic that local information of time-series is
heavy spatial redundancy, we propose the Keys/Values Distilling method, which
can further reduce the time and space complexity to $\mathcal{O}(L)$, and help
the model to accept longer sequence inputs.

  * •

In order to decompose time-series and explain complex time-series patterns, we
propose a decomposition method, which is combined with self-attention
mechanism, to process complex time-series and extract more useful features.

  * •

We have conducted extensive experiments on datasets in many different fields,
and the final results show that our proposed model achieves the most advanced
performance in a variety of experimental settings.

(a) Softmax scores at Head1@Encoder layer

(b) Softmax scores at Head7@Encoder layer

Figure 2. The Softmax scores in the self-attention from canonical Transformer
trained on ETTh1 dataset

##  2\. Related Work

###  2.1. Time-Series Forecasting

The classical convolutional neural network (CNN) krizhevsky2012imagenet  model
can extract the local information unrelated to the spatial position in the
data lim2021time . In order to allow CNN to be used in the time-series,
scholars designed multi-layer causal convolutions to ensure that only past
information can be used for prediction borovykh2017conditional ;
bai2018empirical . For the processing of long-term dependencies, the Temporal
Convolutional Network (TCN) introduces the dilated convolutions, which changes
the interval of original look-back window from 1 to $d_{l}$, where $d_{l}$ is
a layer-specific division rate. In traditional modeling, recurrent neural
networ (RNN) is also widely used in the field of time-series prediction owing
to its architecture naturally supports inputs and outputs with sequential
relationships salinas2020deepar ; rangapuram2018deep ; lim2020recurrent ;
tang2022features . The main idea is to use the memory state of RNN neurons to
store all past effective information. However, RNN variants may be limited in
learning the long-term dependency in the data. Since all the information in
the past will decay with time and the difficult for RNN to learn the long-term
memory hochreiter2001gradient . Long Short-Term Memory networks (LSTM)
hochreiter1997long  introduces some different operation gates to solve this
problem, but it does not solve the long-term dependency well. To further these
effort, attention mechanism is proposed to help the neural network to learn
long-term memory information bahdanau2014neural . In short, the attention
mechanism of time-series is to calculate the dynamic weight, find the weighted
sum of past hidden states, and predict the output value with the summed state.
In this way, the vector used for prediction can contain information that
predicts a more informative time point for the current time point fan2019multi
; jiang2022accurate ; lim2021temporal .

###  2.2. Sparse Attention

In the standard self-attention mechanism, each token needs to pay attention to
all other tokens vaswani2017attention . However, for the trained transformer,
the learned attention matrix A is often very sparse across most data points
child2019generating . Therefore, the computational complexity can be reduced
by limiting the number of queries that want to participate in the query-key
pairs through the incorporating structural bias. The existing methods can be
divided into two categories: position-based and content-based sparse attention
lin2021survey . In position-based sparse attention, the attention matrix is
limited to some predefined patterns ye2019bp ; parmar2018image . Although
these spark patterns change in different ways, some of them can be decomposed
into some atomic sparse patterns, e.g., global attention, band attention,
dilated attention, random attention, block local attention guo2019star ;
beltagy2020longformer . Many spark patterns include one or more of the above
atomic sparse patterns zaheer2020big . Another work is to create a sparse
graph based on the input content. A simple method is to select keywords that
may have a large similarity score with a given query. In order to construct
the sparse graph effectively, the maximum inner product search problem can be
repeated, i.e, the key with the maximum dot product can be found by a query
without calculating all dot-product terms zhou2021informer ; li2019enhancing .
For example, Routing transformer roy2021efficient  uses K-means clustering to
cluster queries and keys on the same group of centroid vectors. Each query
only focuses on the keys belonging to the same cluster. Reformer
kitaev2019reformer  uses location sensitive hashing (LSH) to select key-value
pairs for each query. The proposed LSH allows each token to attend only to the
tokens in the same hash bucket. In Informer zhou2021informer , based on query
and key similarity sampling dot-product pairs, ProbSparse self-attention is
proposed to reduce the time complexity of Transformer to $\mathcal{O}(L\log
L)$ and allows it to accept longer input.

##  3\. Methodology

The problem of long time-series forecasting is to input the past sequence
$\mathcal{X}=\left\\{x_{1},\cdots,x_{L_{x}}|x_{i}\in\mathbb{R}^{d_{x}}\right\\}$,
and the output is to predict corresponding future sequence
$\mathcal{Y}=\left\\{y_{L_{x}+1},\cdots,y_{L_{x}+L_{y}}|y_{i}\in\mathbb{R}^{d_{y}}\right\\}$,
where $L_{x}$ and $L_{y}$ are the lengths of input and output sequences
respectively, and $d_{x}$ and $d_{y}$ are the feature dimensions of input
$\mathcal{X}$ and output $\mathcal{Y}$ respectively. The LTFP encourages a
longer input’s length $L_{x}$ and a longer output’s length $L_{y}$ than
previous works.

Our proposed Infomaxformer holds the encoder-decoder architecture and combines
it with the decomposition structure to solve LTFP. Please refer to Figure 1
for an overview and the following sections for details.

###  3.1. Vanilla Self-attention Mechanism

The scaled dot-product attention mechanism in original Transformer
vaswani2017attention  performs as:

$Attention(\textbf{Q},\textbf{K},\textbf{V})=Softmax(\frac{\textbf{Q}\textbf{K}^{T}}{\sqrt{d}})\textbf{V}$
(1)

i.e., $Attention$ is defined as an operation of ternary matrix, where
$\textbf{Q}(queries)\in\mathbb{R}^{L_{Q}\times d}$,
$\textbf{K}(keys)\in\mathbb{R}^{L_{K}\times d}$,
$\textbf{V}(values)\in\mathbb{R}^{L_{V}\times d}$, and $d$ is the feature
dimension. To further discuss the self-attention mechanism, the $Softmax$
function is expanded, and use $q_{i}$, $k_{i}$ and $v_{i}$ to represent the
$i$-th row in Q, K and V respectively. For the time-series with input length
$L$, $i$ represents the $i$-th data. Therefore, the original self-attention
mechanism for the $i$-th data can be expressed as:

$\mathcal{A}(i)=\sum_{j}^{L}\frac{e^{\frac{q_{i}k_{j}^{T}}{\sqrt{d}}}}{\sum_{l}^{L}e^{\frac{q_{i}k_{l}^{T}}{\sqrt{d}}}}v_{j}=\sum_{j}^{L}\frac{k(q_{i},k_{j})}{\sum_{l}^{L}k(q_{i},k_{l})}v_{j}$
(2)

which $k(q_{i},k_{j})=exp(q_{i}k_{j}^{T}/\sqrt{d})$ tsai2019transformer .

Let $p(q_{i},k_{j})=k(q_{i},k_{j})/\sum_{l}^{L}k(q_{i},k_{l})$, $Attention$
can be abbreviated as:

$\mathcal{A}(i)=\sum_{j}^{L}p\left(q_{i},k_{j}\right)v_{j}$ (3)

where $p\left(q_{i},k_{j}\right)$ is the probability of $v_{i}$, then
$\mathcal{A}(i)$ is the expectation of matrix V. For the probability
$p(q_{i},k_{j})$, it requires the quadratic times dot-product computation and
$\mathcal{O}(L_{Q}L_{K})$ memory usage, which is the main reason why the
traditional self-attention mechanism cannot handle long time-series (it is
easy to lead to out-of-memory), and also the main disadvantage that limits its
prediction ability.

Many previous studies have shown that the probability distribution of self-
attention mechanism has potential sparsity child2019generating ; lin2021survey
, and a selection strategy is designed for all $p(q_{i},k_{j})$ without
significantly affecting the performance of the model ye2019bp ;
parmar2018image ; guo2019star ; beltagy2020longformer . To motivate our
approach, we first revisit the learned attention patterns of the vanilla self-
attention and make a qualitative evaluation. According to Figure 2, in the
first layer of encoder, the scores follows an obvious long tail distribution,
and the Softmax scores has obvious blocking phenomenon, especially in the
second and third layers. So a few dot-product pairs contribute to the major
attention, and others generate negligible attention. Then, how to “select”
them?

###  3.2. Reformulation via the Lens of Information Entropy

We now provide the intuition to reformulate Equation (3) via the lens of
information entropy shannon1948mathematical . Information entropy is a basic
concept of information theory, which describes the uncertainty of possible
events of information sources. Its formula is as follows:

$H\left(x_{i}\right)=-\sum_{i=1}^{L}p(x_{i})lnp(x_{i})$ (4)

where $p(x_{i})$ represents the probability that the random event $X$ is
$x_{i}$. Any information has redundancy, which is related to the occurrence
probability (uncertainty) of each symbol in the information. The probability
and the amout of information generated are positively correlated. Information
is used to eliminate random uncertainty, and information entropy is a measure
of the amount of information needed to eliminate uncertainty, i.e., the amount
of information that an unknown event may contain.

Maximum Entropy Principle When only some knowledge about the unknown
distribution is mastered, the probability distribution with the largest
entropy value should be selected jaynes1957information .

It is difficult to determine the probability distribution of random variables.
Generally, only the average values or the values under certain limited
conditions can be measured. There can be many (even infinite) distributions
that meet the measured values. The maximum entropy principle is a criterion
for selecting the statistical characteristics of random variables that best
meet the objective conditions, also known as the Maximum Information
Principle. Based on this principle, it is effective to select a distribution
with maximum entropy as the distribution of the random variable.

Maximum Entropy Self-attention From Equation (3), the $i$-th query’s attention
on all the keys are defined as a probability distributions $\textbf{p}_{i}$
and the output is its composition with values V. According to the maximum
entropy principle, the dominant dot-product pairs encourage the corresponding
entropy of $\textbf{p}_{i}$ to be maximum. However, the traversing of all the
$\textbf{p}_{i}$ still needs to calculate each dot-product pair, i.e., the
time complexity is $\mathcal{O}(L^{2})$. Motivated by this, we propose a very
simple but effective approximation method to obtain the query information
entropy measurement.

###### Proposition

For all probability distributions $\textbf{p}_{i}$ and $\textbf{p}_{j}$, if
$\sigma_{p_{i}}<\sigma_{p_{j}}$, it can be considered that $H(i)>H(j)$.

If the $i$-th query’s $\textbf{p}_{i}$ gains a smaller variance, its
information entropy is larger and has a higher possibility to contain the
dominate dot-product pairs. Variance is a measure of the degree of dispersion
of a group of data. The variance of data subject to the same distribution is
the same, so we only need to randomly sample constant $U$ from K to calculate
the variance of the $i$-th query’s probability distribution $\textbf{p}_{i}$,
which only need to calculate $\mathcal{O}(L_{Q})$ dot-product for each query-
key lookup and the layer memory usage maintains $\mathcal{O}(L_{Q})$. Then,
select sparse Top-$u$ from Q as $\bar{\textbf{Q}}$ to calculate the standard
dot-product pair, so the time complexity and memory usage maintains
$\mathcal{O}(uL_{K})$. However, the rest of queries can’t be left without any
calculation.

In the case of discrete sources, for discrete sources with L symbols, the
information entropy can reach the maximum value only when they appear with
equal probability, that is, the average uncertainty of sources with equal
probability distribution is the maximum

Based on the proposed measurement and Theorem 3.2, we have the maximum entropy
self-attention, i.e., MEA (the pseudo-code is in Appendix):

$\mathcal{A}(i)=\begin{cases}\sum_{j}^{L}p\left(q_{i},k_{j}\right)v_{j}&\text{
, if top-u}\\\ \\\ \sum_{j}^{L}v_{j}/L&\text{, otherwise}\end{cases}$ (5)

###  3.3. Embedding Method

Figure 3. The Embedding Method

As shown in Figure 3, the input embedding consists of three parts, a scalar, a
local position and a global time stamp. We use scalar projection SP, local
position embedding PE vaswani2017attention  and time embedding TE
zhou2021informer  to deal with the above parts respectively:

$SP=Conv1d(x^{t}_{i})$ (6)
$\begin{split}PE_{\left(i,2j\right)}=sin\left(i/10000^{2j/d_{model}}\right)\\\
PE_{\left(i,2j+1\right)}=cos\left(i/10000^{2j/d_{model}}\right)\end{split}$
(7) $TE=E(month)+E(day)+E(hour)+E(minute)$ (8)

For the Equation (6), we project the scalar context $x^{t}_{i}$ into
$d_{model}$-dim vector with 1-D convolutional filters. The kernel width is 3,
stride is 1, the input channel is $d_{x}$ and the output channel is
$d_{model}$. For the Equation (7), $i\in\left\\{1,\dots,L_{x}\right\\}$,
$j\in\left\\{1,\dots,\left\lfloor d_{model}/2\right\rfloor\right\\}$,
$d_{model}$ is the feature dimension after embedding. For the Equation (8),
$E$ is a learnable stamp embeddings with limited vocab size (up to 60, namely
taking minutes as the finest granularity).

For the three different features finally obtained, instead of the method of
addition zhou2021informer ; wu2021autoformer , we stacked them together and
reduced their dimension through a two-dimensional convolution, that is, the
input channel of convolution is 3 and the output channel is 1:

$\mathcal{X}=Conv2d(Stack(SP,PE,TE))$ (9)

where the kernel width and stride is $(1,1)$.

###  3.4. Keys/Values Distilling

In previous works, a feed-forward network with a single hidden layer is
proposed to linearly project the queries, keys and values vaswani2017attention
; zhou2021informer . As the natural consequence of the original sequence
linear project, the queries, keys and values have a lot of redundant features.
We use the distilling operation to privilege the superior keys and values with
dominating features and make a focused feature map in the self-attention
mechanism. It trims the time dimension of the input sharply, does not
arbitrarily delete the feature of the input sequence, but recombines them into
a new heads weights matrix.

Figure 4. The Keys/Values Distilling

As shown in Figure 4, we distilling keys/values using a three-step convolution
operation:

$\begin{split}\mathcal{K}_{1}&=Maxpool2d(Relu(Conv2d(\mathcal{X})))\\\
\mathcal{K}_{2}&=Maxpool2d(Relu(Conv2d(\mathcal{K}_{1})))\\\
\mathcal{K}_{3}&=Maxpool2d(Relu(Conv2d(\mathcal{K}_{2})))\end{split}$ (10)

where $\mathcal{X}\in\mathbb{R}^{L_{x}\times d_{model}}$. $Conv2d()$ performs
an 2-D convolutional filters (kernel size=(2, 2)) with the $Relu$ activation
function, the number of input channels is the number of heads, and the number
of output channels is $h$ times the number of input channels, so after three-
step convolution, the number of output channels, i.e., the number of heads, is
$h^{3}$ (this can be modified as needed). $Maxpool2d()$ performs an 2-D max
pooling (kernel size and stride is ($l$, h)). Therefore, after three-step
convolution,
$\mathcal{K}_{3}\in\mathbb{R}^{\frac{L_{X}}{l^{3}}\times\frac{d_{model}}{h^{3}}}$,
i.e. $\mathcal{K}\in\mathbb{R}^{L_{K}\times\frac{d_{model}}{h^{3}}}$,
$\mathcal{V}\in\mathbb{R}^{L_{V}\times\frac{d_{model}}{h^{3}}}$, $L_{K}$ and
$L_{V}$ is ${L_{X}}/{l^{3}}$.

Complexity Analysis: Now we know that $L_{K}$ is ${L_{X}}/{l^{3}}$, so the
time complexity and space complexity of our MEA is
$\mathcal{O}(u{L_{X}}/{l^{3}})$. We set $u=c\sqrt{L_{Q}}$, $l=L^{1/6}_{X}$,
$c$ is a constant sampling factor, $u$ varies linearly with $L_{Q}$, so:

$u{L_{X}}/{l^{3}}=c\sqrt{L_{Q}}{L_{X}}/{(L^{1/6}_{X})^{3}}=c\sqrt{L_{Q}}\sqrt{L_{X}}$
(11)

where $L_{Q}=L_{X}=L$. Consequently, our time complexity and space complexit
can reach linear $\mathcal{O}(L)$.

###  3.5. Time-Series Decomposition

In order to make long-term prediction under the input of long time-series, we
use the concept of decomposition to learn complex time patterns, which can
separate the time-series into trend and seasonal hyndman2018forecasting ;
cleveland1990stl ; wu2021autoformer . These two parts respectively represent
two features including long-term development trend and seasonality of the
time-series, which are different in different time-series. To overcome such a
problem, we introduce a time-series decomposition block (TSD), which can
propose the development trend and seasonality of the time-series from the
input. Specifically, we use the moving average to smooth out periodic
fluctuations, extract long-term trends, and highlight seasonality. For an
input sequence $\mathcal{X}\in\mathbb{R}^{L\times d}$ of length $L$, this
process can be formulated as:

$\begin{split}\mathcal{X}_{t}&=AvgPool(\mathcal{X})\\\
\mathcal{X}_{s}&=\mathcal{X}-\mathcal{X}_{t}\end{split}$ (12)

where $\mathcal{X}_{s},\mathcal{X}_{t}\in\mathbb{R}^{L\times d}$ represent
extracted seasonal and trend, respectively. We use
$\mathcal{X}_{t},\mathcal{X}_{s}=TimeSeriesDecomp(\mathcal{X})$ to summarize
above equations.

Table 1. Multivariate long time-series forecasting results on five cases. A
lower MSE or MAE indicates a better prediction, and we use black numbers to
indicate the best performance. Due to the limitation of memory, the batch size
of some models is changed to 16, which is indicated by underlined numbers. The
‘-’ indicates that there is still not enough memory after the batch size is
changed to 16

Methods | Infomaxformer | Autoformer | Informer | Reformer | LogTrans | Transformer | LSTM
---|---|---|---|---|---|---|---
Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
ECL | 24 | 0.190 | 0.305 | 0.195 | 0.312 | 0.310 | 0.400 | 0.280 | 0.381 | 0.231 | 0.338 | 0.244 | 0.350 | 0.338 | 0.419
48 | 0.209 | 0.321 | 0.221 | 0.332 | 0.357 | 0.425 | 0.273 | 0.370 | 0.287 | 0.373 | 0.260 | 0.359 | 0.334 | 0.412
96 | 0.218 | 0.328 | 0.230 | 0.340 | 0.367 | 0.434 | 0.291 | 0.381 | 0.292 | 0.377 | 0.278 | 0.373 | 0.330 | 0.409
192 | 0.239 | 0.346 | 0.280 | 0.347 | 0.362 | 0.434 | 0.344 | 0.420 | 0.295 | 0.385 | 0.286 | 0.377 | 0.327 | 0.407
384 | 0.278 | 0.377 | - | 0.450 | 0.483 | 0.327 | 0.404 | 0.323 | 0.397 | 0.290 | 0.378 | 0.320 | 0.403
ETTh1 | 24 | 0.478 | 0.489 | 0.499 | 0.516 | 1.130 | 0.871 | 0.624 | 0.578 | 0.505 | 0.513 | 0.882 | 0.723 | 1.232 | 0.839
48 | 0.552 | 0.528 | 0.562 | 0.552 | 1.231 | 0.905 | 0.727 | 0.638 | 0.568 | 0.544 | 1.311 | 0.954 | 1.261 | 0.873
96 | 0.564 | 0.543 | 0.611 | 0.579 | 1.345 | 0.953 | 0.930 | 0.743 | 0.714 | 0.629 | 1.957 | 1.199 | 1.268 | 0.873
192 | 0.556 | 0.544 | 0.724 | 0.625 | 1.643 | 1.061 | 1.124 | 0.821 | 0.865 | 0.717 | 1.758 | 1.136 | 1.266 | 0.871
384 | 0.597 | 0.570 | - | 1.499 | 1.004 | 1.270 | 0.862 | 0.952 | 0.749 | 1.405 | 0.981 | 1.273 | 0.873
ETTh2 | 24 | 0.436 | 0.476 | 0.437 | 0.493 | 2.048 | 1.173 | 0.975 | 0.787 | 0.621 | 0.617 | 1.016 | 0.814 | 3.291 | 1.385
48 | 0.629 | 0.544 | 0.658 | 0.643 | 3.047 | 1.471 | 1.652 | 1.027 | 1.168 | 0.985 | 2.199 | 1.242 | 3.378 | 1.408
96 | 0.593 | 0.533 | 0.686 | 0.646 | 6.882 | 2.258 | 3.301 | 1.427 | 2.279 | 1.265 | 5.862 | 2.052 | 3.488 | 1.432
192 | 0.703 | 0.607 | 0.792 | 0.671 | 5.070 | 1.885 | 3.774 | 1.617 | 4.207 | 1.776 | 4.045 | 1.675 | 3.489 | 1.434
384 | 0.575 | 0.557 | - | 4.080 | 1.669 | 3.363 | 1.465 | 3.032 | 1.526 | 3.549 | 1.516 | 3.486 | 1.430
ETTm1 | 24 | 0.330 | 0.397 | 0.414 | 0.438 | 0.354 | 0.401 | 0.430 | 0.453 | 0.882 | 0.666 | 0.355 | 0.403 | 1.121 | 0.791
48 | 0.418 | 0.454 | 0.537 | 0.505 | 0.533 | 0.521 | 0.578 | 0.544 | 0.951 | 0.707 | 0.514 | 0.526 | 1.130 | 0.799
96 | 0.494 | 0.502 | 0.545 | 0.517 | 0.592 | 0.571 | 0.710 | 0.612 | 0.558 | 0.540 | 0.740 | 0.657 | 1.141 | 0.805
192 | 0.558 | 0.531 | 0.605 | 0.534 | 0.768 | 0.682 | 0.896 | 0.702 | 0.591 | 0.565 | 0.700 | 0.641 | 1.141 | 0.806
384 | 0.611 | 0.561 | - | 0.938 | 0.765 | 1.072 | 0.781 | 0.767 | 0.650 | 0.838 | 0.719 | 1.153 | 0.810
Weather | 24 | 0.307 | 0.357 | 0.455 | 0.489 | 0.348 | 0.401 | 0.370 | 0.426 | 0.385 | 0.427 | 0.326 | 0.378 | 0.492 | 0.500
48 | 0.381 | 0.422 | 0.544 | 0.542 | 0.488 | 0.505 | 0.443 | 0.475 | 0.498 | 0.505 | 0.447 | 0.664 | 0.497 | 0.504
96 | 0.456 | 0.481 | 0.554 | 0.546 | 0.603 | 0.574 | 0.511 | 0.520 | 0.562 | 0.550 | 0.548 | 0.532 | 0.500 | 0.506
192 | 0.508 | 0.517 | 0.585 | 0.560 | 0.700 | 0.632 | 0.537 | 0.541 | 0.591 | 0.570 | 0.620 | 0.575 | 0.503 | 0.517
384 | 0.511 | 0.514 | - | 0.681 | 0.621 | 0.536 | 0.535 | 0.622 | 0.585 | 0.630 | 0.579 | 0.513 | 0.524

###  3.6. Encoder and Decoder

Encoder: As shown in Figure 1, the encoder focuses on modeling of the seasonal
part. Our encoder layers are composed of two sub-blocks. The first is a MEA
mechanism, and the second is a simple, position-wise fully connected feed-
forward network (MLP). We employ residual connections he2016deep  around each
of the sub-blocks, but unlike previous structures vaswani2017attention ;
2021An , layer normalization ba2016layer  was not performed. The input of the
encoder is only the seasonal part $\mathcal{X}_{s}$ of the input sequence
$\mathcal{X}$, and the output only contains the seasonal information of the
past and will be used as cross information to help the decoder better predict
the seasonal information of the future sequence.

$\begin{split}\mathcal{X}_{t},\mathcal{X}_{s}&=TimeSeriesDecomp(\mathcal{X})\\\
\mathcal{X}^{n}_{s1}&=\mathcal{X}^{n-1}_{s}+MEA(\mathcal{X}^{n-1}_{s})\\\
\mathcal{X}^{n}_{s}&=\mathcal{X}^{n}_{s1}+MLP(\mathcal{X}^{n}_{s1})\end{split}$
(13)

where $n=1\dots N$, $\mathcal{X}^{0}_{t}=\mathcal{X}_{t}$,
$\mathcal{X}_{eno}=\mathcal{X}^{N}_{t}$, $N$ is the number of layers of the
encoder, and MLP consists of two 1-D convolution operations.

Decoder: In addition to the two sub-blocks in each encoder layer, the decoder
in classic Transformer also inserts a third sub-block in the two sub-blocks,
which performs self-attention on the output of the encoder layers. On this
basis, we insert the fourth sub-block in the decoder, i.e., the time-series
decomposition block. Similar to the encoder, we employ residual connections
around each of the sub-blocks, but did not perform layer normalization. We
input the following vectors to the decoder:

$\mathcal{X}_{dei}=Concat(\mathcal{X}_{label},\mathcal{X}_{0})\in\mathbb{R}^{(L_{label}+L_{y})\times
d_{model}}$ (14)

where $\mathcal{X}_{label}\in\mathbb{R}^{L_{label}\times d_{model}}$ is start
token, $L_{label}$ is the label length,
$\mathcal{X}_{0}\in\mathbb{R}^{L_{y}\times d_{model}}$ is a placeholder for
the target sequence (set the scalar to 0). By setting masked dot-products to
negative infinity, masked multi-head attention is applied to the MEA
calculation (MMEA). This masking ensures that the prediction of position $i$
can only rely on the known outputs of positions less than $i$, which avoids
auto-regressive. A fully connected layer acquires the final output, and its
outsize is $d_{y}$. For the time-series, the trend changes are not obvious,
but the specific seasonal is different. We use the decoder to predict the
seasonal of future data, and use the average of input data to approximate the
trend part of future data.

$\begin{split}\mathcal{X}^{n}_{1}&=\mathcal{X}^{n-1}+MEA(\mathcal{X}^{n-1})\\\
\mathcal{X}^{n}_{t},\mathcal{X}^{n}_{s}&=TimeSeriesDecomp(\mathcal{X}^{n}_{1})\\\
\mathcal{X}^{n}_{s1}&=\mathcal{X}^{n}_{s}+MMEA(\mathcal{X}^{n}_{s},\mathcal{X}_{eno})\\\
\mathcal{X}^{n}&=\mathcal{X}^{n}_{s1}+MLP(\mathcal{X}^{n}_{s1})+\mathcal{X}^{n}_{t}\\\
\mathcal{Y}&=Mean(\mathcal{X}_{t})+\mathcal{X}^{M}\end{split}$ (15)

where $n=1\dots M$, $\mathcal{X}^{0}=\mathcal{X}_{dei}$, $M$ is the number of
layers of the decoder.

Loss Function: Our loss function is calculated by the mean square error (MSE)
between the model output data $y_{o}$ and the real data $y$. and the loss is
propagated back from the decoder’s outputs across the entire model.

Table 2. Complexity analysis of different forecasting models. The ${\star}$ denotes applying generative style decoder zhou2021informer . Methods | Training | Forecasting
---|---|---
Time | Memory | Steps
Infomaxformer | $\mathcal{O}(L)$ | $\mathcal{O}(L)$ | 1
Autoformer | $\mathcal{O}(L\log L)$ | $\mathcal{O}(L\log L)$ | 1
Informer | $\mathcal{O}(L\log L)$ | $\mathcal{O}(L\log L)$ | 1
Transformer | $\mathcal{O}(L^{2})$ | $\mathcal{O}(L^{2})$ | $L$
LogTrans | $\mathcal{O}(L\log L)$ | $\mathcal{O}(L^{2})$ | 1⋆
Reformer | $\mathcal{O}(L\log L)$ | $\mathcal{O}(L\log L)$ | $L$
LSTM | $\mathcal{O}(L)$ | $\mathcal{O}(L)$ | $L$
Table 3. Different input lengths for two prediction lengths in ETTh1. The ‘-’
indicates failure for the out-of-memory

Predicition length | 336 | 480
---|---|---
Encoder’s input | 336 | 480 | 720 | 960 | 1200 | 1440 | 480 | 720 | 960 | 1200 | 1440
Informer | MSE | 1.474 | 1.576 | 1.644 | 1.607 | 1.580 | 1.766 | 1.441 | 1.531 | 1.508 | 1.428 | 1.520
MAE | 0.999 | 1.024 | 1.045 | 1.043 | 1.037 | 1.124 | 0.971 | 0.998 | 0.997 | 0.967 | 1.017
Reformer | MSE | 1.000 | 1.043 | 1.200 | 1.159 | - | - | 1.148 | 1.259 | - | - | -
MAE | 0.766 | 0.790 | 0.842 | 0.829 | 0.827 | 0.870
Transformer | MSE | 1.085 | 1.161 | 1.630 | 1.922 | - | - | 1.083 | 1.410 | - | - | -
MAE | 0.830 | 0.858 | 1.065 | 1.159 | 0.836 | 0.969
LogTrans | MSE | 1.136 | 1.127 | 1.059 | 1.075 | 1.060 | 1.067 | 0.888 | 0.940 | 0.892 | 0.885 | 0.914
MAE | 0.851 | 0.849 | 0.817 | 0.823 | 0.818 | 0.816 | 0.733 | 0.764 | 0.736 | 0.732 | 0.745
Autoformer | MSE | 0.581 | 0.560 | 0.748 | - | - | - | 0.573 | - | - | - | -
MAE | 0.547 | 0.547 | 0.650 | 0.558
Infomaxformer | MSE | 0.567 | 0.556 | 0.544 | 0.566 | 0.568 | 0.624 | 0.537 | 0.583 | 0.597 | 0.599 | 0.709
MAE | 0.545 | 0.547 | 0.543 | 0.563 | 0.564 | 0.602 | 0.551 | 0.581 | 0.587 | 0.585 | 0.653

##  4\. Experiment

###  4.1. Datasets

ETT(Electricity Transformer Temperature):111ETT dataset was acquired at
zhou2021informer . The ETT is a key indicator for long-term deployment of the
electric power, which collects data from two counties in China from July 2016
to July 2018 for a total of two years.

ECL (Electricity Consuming Load):222ECL dataset was acquired at
https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014. It
collects the electricity consumption (Kwh) of 321 customers. Due to the lack
of data li2019enhancing , we follow the settings in Informer zhou2021informer
to convert the dataset to hourly consumption for 2 years.

Weather:333Weather dataset was acquired at
https://www.ncei.noaa.gov/data/local-climatological-data/. This dataset
contains the local climatologicale data of nearly 1600 locations in the United
States. The data are collected by once an hour from 2010 to 2013.

###  4.2. Experimental Details

Baselines: We selected six methods as comparison, including Transformer
vaswani2017attention , four latest state-of-the-art Transformer-based models:
Reformer kitaev2019reformer , LogTrans li2019enhancing , Informer
zhou2021informer , Autoformer wu2021autoformer , and one RNN-based models:
LSTM (Long Short-Term Memory networks) hochreiter1997long .

Experiment setting: Our experiment was implemented in Pytoch paszke2019pytorch
, and all the experiments are conducted on a single Nvidia RTX 3090 GPU (24GB
memory). The input of each dataset is zero-mean normalized. We use two
evaluation metrics, including mean square error (MSE):
$MSE=\frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{d}\frac{(y-\hat{y})^{2}}{d}$ and
mean absolute error (MAE):
$MAE=\frac{1}{n}\sum_{i=1}^{n}\sum_{j=1}^{d}\frac{\left|y-\hat{y}\right|}{d}$,
where $n$ is the length of the sequence and $d$ is the dimension of data at
each time point. We use these two evaluation metrics on each prediction window
to calculate the average of forecasts and roll the whole set with $stride=1$.

Our implementation details follows common practice of Informer
zhou2021informer  training, and all experiments are repeated five times. We
use Adam kingma2014adam  optimizer for optimization with a learning rate
starts from $1e^{-4}$, decaying two times smaller every epoch, and the batch
size is 32. The number of encoder layers is 3 and the number of decoder layers
is 2. There is no limit to the total number of epochs, with appropriate early
stopping, i.e., when the loss of the validation set does not decrease on three
epochs, the training will be stopped. More detailed settings can be found in
Appendix 4.2.

###  4.3. Multivariate Time-series Forecasting

To compare the performance of different prediction lengths, we fixed the input
length $L_{x}$ to 784 and gradually extended the prediction length $L_{y}$,
i.e., {24, 48, 96, 192, 384}, representing {6h, 12h, 24h, 48h, 96h} in ETTm,
{1d, 2d, 4d, 8d, 16d} in {ETTh, ECL, Weather}, and we set the length of the
label to double $L_{y}$.

As shown in Table 1, our proposed Infomaxformer model achieves the best
performance in all benchmarks and all predicted length settings. Although the
performance of Autoformer is closest to our model, it can only set the batch
size to 32 when the prediction length is 24. When the prediction length is
384, the batch size to 16 will also lead to out-of-memory. This shows that our
proposed Infomaxformer model can increase the prediction ability, while
greatly reducing the use of memory. In addition, we also found that with the
increase of prediction length, the prediction performance of Infomaxformer is
more stable, and there is no sudden drop in performance, which means that
Infomaxformer maintains good long-term robustness. Low memory usage, good
robustness, high-performance prediction, etc., which is very meaningful for
practical applications, and our model has the above advantages.

(a) Sampling Factor $c$

(b) Sampling Factor $U$

Figure 5. The parameter sensitivity of two components in Infomaxformer

###  4.4. Parameter Sensitivity

We perform the sensitivity analysis of the proposed Infomaxformer model on
ETTh1.

Input Length: As shown in the Table 3, we gradually extended the size of the
input sequence $L_{x}$, i.e., {336, 480, 720, 960, 1200, 1440}, while keeping
the predicted length $L_{y}$ unchanged, and the length $L_{label}$ of the
label sequence is consistent with $L_{y}$. Our $L_{y}$ selected two values,
336 and 480. In Table 3, it can be seen that increasing the input length will
lead to the decrease of MSE and MAE, because long input will bring repeated
short-term patterns. However, as the input sequence increases, there may be
more dependencies between the inputs, and the influence of noise in the input
data will also increase. Some models can not effectively eliminate the
influence of these noises and can not better grasp the dependence of long
time-series, so MSE and MAE may increase. In this experiment, the batch size
of Autoformer is set to 16, because too large batch size will directly lead to
out-of-memory. In Table 2, we make statistics on the time complexity and
memory usage of various models. It can be seen that there is still a certain
gap between the memory usage of many model theories and the actual
application. It seems that the Autoformer with memory usage of
$\mathcal{O}(L\log L)$ is not better than the original Transformer with memory
usage of $\mathcal{O}(L^{2})$.

Sampling Factor $c$: The sampling factor $c$ controls the information
bandwidth of MEA in Equation (5). We start with small factors (=1) and
gradually increase to large factors (=9). As can be seen in Figure 5(a), the
performance of our Infomaxformer does not change much, and it is not similar
to the case that the performance of Informer is slightly improved with the
change of sampling factor. This is because our initial sampling is $\sqrt{L}$,
not $\log L$ in Informer zhou2021informer , so enough dominant queries is
selected to calculate the dot-product to prevent the loss of important
features.

Sampling Factor $U$: The sampling factor $U$ controls how many keys are
selected to calculate the variance. Although the variance of data subject to
the same distribution is the same, too few data samples will cause the
calculated variance to be not the actual variance. It can be seen from Figure
5(b) that when $U$ is too low, the performance of the Infomaxformer model will
indeed be affected. However, when $U$ gradually increases, the performance of
the model tends to be stable.

Table 4. Ablation study of the Infomaxformer

Encoder’s input | 720 | 960 | 1200 | 1440
---|---|---|---|---
Infomaxformer | MSE | 0.566 | 0.588 | 0.633 | 0.573
MAE | 0.579 | 0.585 | 0.617 | 0.585
Infomaxformer1 | MSE | 1.326 | 1.237 | 0.880 | 1.290
MAE | 0.911 | 0.891 | 0.736 | 0.882
Infomaxformer2 | MSE | 0.906 | 0.681 | 0.839 | 0.831
MAE | 0.763 | 0.638 | 0.727 | 0.720
Infomaxformer3 | MSE | 1.074 | 1.018 | 0.964 | 1.121
MAE | 0.822 | 0.823 | 0.789 | 0.846

###  4.5. Ablation Study

We also performed some additional experiments for ablation analysis on ETTh1.
Infomaxformer1 indicates that Keys/Values Distilling is replaced with the
original projection operation, Infomaxformer2 indicates that MEA is replaced
with the canonical self-attention mechanism, and Infomaxformer3 indicates that
we have not taken our proposed Time-Series Decomposition. In this experiment,
we set the predicted length $L_{y}$ to 720 and select four ultra long input
lengths. As shown in the Table 4, without any part of the Infomaxformer model,
the performance will be degraded. Only the complete Infomaxformer can achieve
the best performance. The impact of MEA mechanism on the performance of
Infomaxformer is not as obvious as the other two, because the mechanism
focuses on sparing self-attention and reducing time complexity.

Table 5. Comparative experiment of different Decomposition Block in
Infomaxformer and Autoformer

Predicition length | 96 | 192 | 384 | 720
---|---|---|---|---
TSD+MEA | MSE | 0.554 | 0.496 | 0.567 | 0.551
MAE | 0.540 | 0.513 | 0.555 | 0.559
SDB+MEA | MSE | 0.709 | 0.770 | 0.683 | 0.677
MAE | 0.624 | 0.662 | 0.626 | 0.626
TSD+AC | MSE | 0.706 | 0.677 | - | -
MAE | 0.627 | 0.605 | - | -
SDB+AC | MSE | 0.623 | 0.699 | - | -
MAE | 0.577 | 0.619 | - | -

Different Time-Series Decomposition: In order to better compare the TSD
proposed by us with the Series decomposition block (SDB) proposed by
Autoformer wu2021autoformer , we freely combined MEA, Auto-Correlation (AC)
wu2021autoformer  with TSD, SDB. As shown in the Table 5, we found an
interesting phenomenon. No matter Infomaxformer (TSD+MEA) or Autoformer
(SDB+AC), only the complete state model has the best performance. Our TSD is
inferior to SDB in short sequence prediction ($L_{y}=96$), but superior to SDB
in long sequence prediction ($L_{y}=192$). The sparsity of MEA results in the
model being able to output longer sequences, but it is precisely because of
this sparsity that the performance of MEA is inferior to AC when memory
allows. However, the perfect combination of TSD and MEA proposed by us can
output longer sequences and maintain better prediction performance.

##  5\. Conclusion

In this paper, we studied the long time-series forecasting problem (LTFP) and
proposed Infomaxformer to predict time-series. Specifically, we designed the
Maximum Enterprise Self-attention mechanism and Keys/Values Distilling
operation to deal with the challenges of quadratic time complexity and
quadratic memory usage in vanilla Transformers. Finally, we reduce the time
complexity and memory usage to $\mathcal{O}(L)$. In addition, the well-
designed time-series decomposition and the perfect combination with
Transformer architecture can effectively deal with the complex time-series
patterns of time-series, thus alleviating the limitations of the traditional
decomposition architecture. The experiments on real-word data have proved the
effectiveness of Infomaxformer in improving the prediction ability of LTFP.

## References

  * (1) J. L. Ba, J. R. Kiros, and G. E. Hinton, Layer normalization, arXiv preprint arXiv:1607.06450, (2016).
  * (2) D. Bahdanau, K. Cho, and Y. Bengio, Neural machine translation by jointly learning to align and translate, arXiv preprint arXiv:1409.0473, (2014).
  * (3) S. Bai, J. Z. Kolter, and V. Koltun, An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, arXiv preprint arXiv:1803.01271, (2018).
  * (4) H. Bao, L. Dong, and F. Wei, Beit: Bert pre-training of image transformers, arXiv preprint arXiv:2106.08254, (2021).
  * (5) I. Beltagy, M. E. Peters, and A. Cohan, Longformer: The long-document transformer, arXiv preprint arXiv:2004.05150, (2020).
  * (6) A. Borovykh, S. Bohte, and C. W. Oosterlee, Conditional time series forecasting with convolutional neural networks, arXiv preprint arXiv:1703.04691, (2017).
  * (7) T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems, 33 (2020), pp. 1877–1901.
  * (8) M. Chen, A. Radford, R. Child, J. Wu, H. Jun, D. Luan, and I. Sutskever, Generative pretraining from pixels, in International conference on machine learning, PMLR, 2020, pp. 1691–1703.
  * (9) R. Child, S. Gray, A. Radford, and I. Sutskever, Generating long sequences with sparse transformers, arXiv preprint arXiv:1904.10509, (2019).
  * (10) R. B. Cleveland, W. S. Cleveland, J. E. McRae, and I. Terpenning, Stl: A seasonal-trend decomposition, J. Off. Stat, 6 (1990), pp. 3–73.
  * (11) J. D. Cryer, Time series analysis, vol. 286, Springer, 1986.
  * (12) J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, (2018).
  * (13) L. Di Persio and O. Honchar, Artificial neural networks architectures for stock price prediction: Comparisons and applications, International journal of circuits, systems and signal processing, 10 (2016), pp. 403–413.
  * (14) C. Doersch, A. Gupta, and A. A. Efros, Unsupervised visual representation learning by context prediction, in Proceedings of the IEEE international conference on computer vision, 2015, pp. 1422–1430.
  * (15) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, and S. a. Gelly, An image is worth 16x16 words: Transformers for image recognition at scale, in International Conference on Learning Representations, 2021.
  * (16) C. Fan, Y. Zhang, Y. Pan, X. Li, C. Zhang, R. Yuan, D. Wu, W. Wang, J. Pei, and H. Huang, Multi-horizon time series forecasting with temporal attention learning, in Proceedings of the 25th ACM SIGKDD International conference on knowledge discovery & data mining, 2019, pp. 2527–2535.
  * (17) X. Geng, H. Liu, L. Lee, D. Schuurams, S. Levine, and P. Abbeel, Multimodal masked autoencoders learn transferable representations, arXiv preprint arXiv:2205.14204, (2022).
  * (18) P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He, Accurate, large minibatch sgd: Training imagenet in 1 hour, arXiv preprint arXiv:1706.02677, (2017).
  * (19) Q. Guo, X. Qiu, P. Liu, Y. Shao, X. Xue, and Z. Zhang, Star-transformer, in Proceedings of NAACL-HLT, 2019, pp. 1315–1325.
  * (20) K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick, Masked autoencoders are scalable vision learners, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 16000–16009.
  * (21) K. He, X. Zhang, S. Ren, and J. Sun, Deep residual learning for image recognition, in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * (22) G. E. Hinton and R. Zemel, Autoencoders, minimum description length and helmholtz free energy, Advances in neural information processing systems, 6 (1993).
  * (23) S. Hochreiter, Y. Bengio, P. Frasconi, J. Schmidhuber, et al., Gradient flow in recurrent nets: the difficulty of learning long-term dependencies, 2001.
  * (24) S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural computation, 9 (1997), pp. 1735–1780.
  * (25) R. J. Hyndman and G. Athanasopoulos, Forecasting: principles and practice, OTexts, 2018.
  * (26) I. Loshchilov and F. Hutter, Sgdr: Stochastic gradient descent with warm restarts, in ICLR, 2017.
  * (27) E. T. Jaynes, Information theory and statistical mechanics, Physical review, 106 (1957), p. 620.
  * (28) H. Jiang, M. Cui, D. W. K. Ng, and L. Dai, Accurate channel prediction based on transformer: Making mobility negligible, IEEE Journal on Selected Areas in Communications, (2022).
  * (29) D. P. Kingma and J. Ba, Adam: A method for stochastic optimization, in International Conference on Learning Representations, 2015.
  * (30) N. Kitaev, L. Kaiser, and A. Levskaya, Reformer: The efficient transformer, in ICLR, 2019.
  * (31) A. Krizhevsky, I. Sutskever, and G. E. Hinton, Imagenet classification with deep convolutional neural networks, Advances in neural information processing systems, 25 (2012).
  * (32) G. Lai, W.-C. Chang, Y. Yang, and H. Liu, Modeling long-and short-term temporal patterns with deep neural networks, in The 41st international ACM SIGIR conference on research & development in information retrieval, 2018, pp. 95–104.
  * (33) S. Li, X. Jin, Y. Xuan, X. Zhou, W. Chen, Y.-X. Wang, and X. Yan, Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, Advances in neural information processing systems, 32 (2019).
  * (34) X. Li, W. Wang, L. Yang, and J. Yang, Uniform masking: Enabling mae pre-training for pyramid-based vision transformers with locality, arXiv preprint arXiv:2205.10063, (2022).
  * (35) B. Lim, S. Ö. Arık, N. Loeff, and T. Pfister, Temporal fusion transformers for interpretable multi-horizon time series forecasting, International Journal of Forecasting, 37 (2021), pp. 1748–1764.
  * (36) B. Lim and S. Zohren, Time-series forecasting with deep learning: a survey, Philosophical Transactions of the Royal Society A, 379 (2021), p. 20200209.
  * (37) B. Lim, S. Zohren, and S. Roberts, Recurrent neural filters: Learning independent bayesian filtering steps for time series prediction, in 2020 International Joint Conference on Neural Networks (IJCNN), IEEE, 2020, pp. 1–8.
  * (38) T. Lin, Y. Wang, X. Liu, and X. Qiu, A survey of transformers, arXiv preprint arXiv:2106.04554, (2021).
  * (39) L. Liu, J. Shen, M. Zhang, Z. Wang, and J. Tang, Learning the joint representation of heterogeneous temporal events for clinical endpoint prediction, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, 2018.
  * (40) S. Liu, H. Yu, C. Liao, J. Li, W. Lin, A. X. Liu, and S. Dustdar, Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting, in International Conference on Learning Representations, 2021.
  * (41) Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, Swin transformer: Hierarchical vision transformer using shifted windows, in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10012–10022.
  * (42) I. Loshchilov and F. Hutter, Decoupled weight decay regularization, in ICLR, 2019.
  * (43) E. J. Mcshane, Jensen’s inequality, Bulletin of the American Mathematical Society, 43 (1937), pp. 521–527.
  * (44) M. Noroozi and P. Favaro, Unsupervised learning of visual representations by solving jigsaw puzzles, in European conference on computer vision, Springer, 2016, pp. 69–84.
  * (45) N. Parmar, A. Vaswani, J. Uszkoreit, L. Kaiser, N. Shazeer, A. Ku, and D. Tran, Image transformer, in International conference on machine learning, PMLR, 2018, pp. 4055–4064.
  * (46) A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al., Pytorch: An imperative style, high-performance deep learning library, Advances in neural information processing systems, 32 (2019).
  * (47) A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al., Improving language understanding by generative pre-training, (2018).
  * (48) A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al., Language models are unsupervised multitask learners, OpenAI blog, 1 (2019), p. 9.
  * (49) S. S. Rangapuram, M. W. Seeger, J. Gasthaus, L. Stella, Y. Wang, and T. Januschowski, Deep state space models for time series forecasting, Advances in neural information processing systems, 31 (2018).
  * (50) A. Roy, M. Saffar, A. Vaswani, and D. Grangier, Efficient content-based sparse attention with routing transformers, Transactions of the Association for Computational Linguistics, 9 (2021), pp. 53–68.
  * (51) D. Salinas, V. Flunkert, J. Gasthaus, and T. Januschowski, Deepar: Probabilistic forecasting with autoregressive recurrent networks, International Journal of Forecasting, 36 (2020), pp. 1181–1191.
  * (52) C. E. Shannon, A mathematical theory of communication, The Bell system technical journal, 27 (1948), pp. 379–423.
  * (53) P. Tang and X. Zhang, Features fusion framework for multimodal irregular time-series events, in Pacific Rim International Conference on Artificial Intelligence, Springer, 2022, pp. 366–379.
  * (54) , Mtsmae: Masked autoencoders for multivariate time-series forecasting, arXiv preprint arXiv:2210.02199, (2022).
  * (55) Y.-H. H. Tsai, S. Bai, M. Yamada, L.-P. Morency, and R. Salakhutdinov, Transformer dissection: An unified understanding for transformer’s attention via the lens of kernel, in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 4344–4353.
  * (56) A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, Attention is all you need, Advances in neural information processing systems, 30 (2017).
  * (57) X. Wang and A. Gupta, Unsupervised learning of visual representations using videos, in Proceedings of the IEEE international conference on computer vision, 2015, pp. 2794–2802.
  * (58) H. Wu, J. Xu, J. Wang, and M. Long, Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, Advances in Neural Information Processing Systems, 34 (2021), pp. 22419–22430.
  * (59) S. Wu, X. Xiao, Q. Ding, P. Zhao, Y. Wei, and J. Huang, Adversarial sparse transformer for time series forecasting, Advances in neural information processing systems, 33 (2020), pp. 17105–17115.
  * (60) Z. Xie, Z. Zhang, Y. Cao, Y. Lin, J. Bao, Z. Yao, Q. Dai, and H. Hu, Simmim: A simple framework for masked image modeling, in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 9653–9663.
  * (61) Z. Ye, Q. Guo, Q. Gan, X. Qiu, and Z. Zhang, Bp-transformer: Modelling long-range context via binary partitioning, arXiv preprint arXiv:1911.04070, (2019).
  * (62) Y. Yin and P. Shang, Multivariate multiscale sample entropy of traffic time series, Nonlinear Dynamics, 86 (2016), pp. 479–488.
  * (63) Q. Yu, Y. Xia, Y. Bai, Y. Lu, A. L. Yuille, and W. Shen, Glance-and-gaze vision transformer, Advances in Neural Information Processing Systems, 34 (2021), pp. 12992–13003.
  * (64) M. Zaheer, G. Guruganesh, K. A. Dubey, J. Ainslie, C. Alberti, S. Ontanon, P. Pham, A. Ravula, Q. Wang, L. Yang, et al., Big bird: Transformers for longer sequences, Advances in Neural Information Processing Systems, 33 (2020), pp. 17283–17297.
  * (65) H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang, Informer: Beyond efficient transformer for long sequence time-series forecasting, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 2021, pp. 11106–11115.