# TimeXer: Empowering Transformers for Time Series
Forecasting with Exogenous Variables

Yuxuan Wang  Haixu Wu  Jiaxiang Dong  Yong Liu  Yunzhong Qiu  Haoran Zhang
Jianmin Wang  Mingsheng Long

###### Abstract

Recent studies have demonstrated remarkable performance in time series
forecasting. However, due to the partially-observed nature of real-world
applications, solely focusing on the target of interest, so-called _endogenous
variables_ , is usually insufficient to guarantee accurate forecasting.
Notably, a system is often recorded into multiple variables, where the
_exogenous series_ can provide valuable external information for endogenous
variables. Thus, unlike prior well-established multivariate or univariate
forecasting that either treats all the variables equally or overlooks
exogenous information, this paper focuses on a practical setting, which is
time series forecasting with exogenous variables. We propose a novel
framework, TimeXer, to utilize external information to enhance the forecasting
of endogenous variables. With a deftly designed embedding layer, TimeXer
empowers the canonical Transformer architecture with the ability to reconcile
endogenous and exogenous information, where patch-wise self-attention and
variate-wise cross-attention are employed. Moreover, a global endogenous
variate token is adopted to effectively bridge the exogenous series into
endogenous temporal patches. Experimentally, TimeXer significantly improves
time series forecasting with exogenous variables and achieves consistent
state-of-the-art performance in twelve real-world forecasting benchmarks.

##  1 Introduction

Figure 1: Comparison between different forecasting paradigms. The inputs of
forecasting with exogenous variables include multiple external variates
$\\{\mathbf{z}^{(1)},\cdots,\mathbf{z}^{(C)}\\}$, which serve as auxiliary
information and are not required for forecasting.

Time series forecasting is of pressing demand in real-world scenarios and can
be widely used in various domains, such as weather (Wu et al., 2023b; Zhang et
al., 2023), electricity market (Weron, 2014), and transportation (Lv et al.,
2014). As shown in Figure 1, existing forecasting paradigms can be summarized
into three distant categories. In comparison with univariate and multivariate
forecasting, forecasting with exogenous variables introduces auxiliary
information to facilitate the forecasting of the endogenous variable.
Exogenous variables are prevalent and indispensable in practical applications
since the variations within time series data are often influenced by external
factors, such as economic indicators, demographic changes, and societal
events. For instance, electricity prices are highly dependent on the supply
and demand of the market, and it is virtually impossible to predict future
prices solely based on historical data. Incorporating external factors allows
for a more comprehensive understanding of the interrelationships and causality
among various variables, leading to better and more reliable performance, as
well as interpretability.

Unlike previous well-established settings, since only the endogenous variable
is of interest, forecasting with exogenous variables poses unique challenges.
First, time series are often affected by multiple factors, which requires
models to reconcile discrepancy and dependency among the _endogenous_ and
_exogenous_ variables. Regarding exogenous variables equally with endogenous
ones will not only cause significant time and memory complexity but also
involve unnecessary interaction from endogenous series to external
information. Second, the effects of external factors on endogenous series
could be continuous and time-lagged. Time series in real-world scenarios are
often irregular, and exogenous variables may encounter problems such as
missing data, misaligned lengths, and misaligned sampling times.

Given the crucial role of exogenous variables in time series forecasting,
classical statistical models have already included exogenous variables as part
of inputs, distinguishing them from endogenous ones. Despite the success of
deep models in capturing intricate temporal dependencies in time series data,
there are few models that support external input. Existing exogenous works
usually struggle to capture complex long-term temporal dependencies (Olivares
et al., 2023). Transformers (Vaswani et al., 2017) have exhibited remarkable
performance in time series forecasting thanks to their ability to capture both
temporal dependencies and multivariate correlations. However, existing
Transformer-based forecasters are not originally designed for forecasting with
exogenous variables. Represented by PatchTST (Nie et al., 2022), existing
variate-independent models are only capable of capturing temporal dependencies
but fail to capture multivariate correlations. Conversely, iTransformer (Liu
et al., 2023) succeeds in capturing interrelationships between variables by
using attention mechanisms on variate tokens but is unable to capture temporal
variation between different sub-series since the whole series is embedded to a
variate token by a temporal linear projection.

Beyond complicated designs, we discover that it is possible to make the
canonical Transformers seamlessly adapt to exogenous variables without
modifying the component or architecture. Specifically, in this paper, we
excavate the potential of Transformer architecture and propose a Time Series
Transformer with eXogenous variables (TimeXer). By utilizing the series-level
representation of time series data, TimeXer can capture correlations between
multiple exogenous and endogenous variables through cross-attention. Since
capturing the temporal dependencies within the endogenous series is
indispensable for accurate prediction, TimeXer also includes patch-wise
representations for endogenous series. This design allows TimeXer to capture
the endogenous temporal dependencies using self-attention. Besides, the
endogenous variate token effectively diffuses the external information to
corresponding endogenous temporal patches. Extensive experiments demonstrate
that TimeXer achieves state-of-the-art performance on various real-world
datasets. Our contributions are summarized as:

  * •

Motivated by the universality and importance of exogenous variables in time
series forecasting, we empower the Transformer architecture to the inclusion
of exogenous variables without architectural modifications.

  * •

We propose TimeXer that utilizes different levels of representations of
endogenous and exogenous variables to capture temporal dependencies by self-
attention over patch-level endogenous temporal tokens and multivariate
correlations by cross-attention over variate tokens.

  * •

Extensive experiments on real-world datasets reveal that TimeXer can better
utilize exogenous information to facilitate prediction than other forecasting
models.

##  2 Related Works

#### Time Series Forecasting Models

In recent years, deep models have achieved great progress in the field of time
series analysis. Among all downstream tasks, time series forecasting is an
essential problem where a large volume of work has been proposed to handle the
intricate temporal dependencies, which can be divided into four categories:
RNN-based, CNN-based, MLP-based, and Transformer-based models. RNN-based
models, including LSTNet (Lai et al., 2018) and DeepAR (Salinas et al., 2020),
are suited for handling sequential temporal data, while they are restricted by
gradient vanishing and inefficiency problems. With the power of convolutional
kernels, recent works, such as TimesNet (Wu et al., 2023a), SCINet (Liu et
al., 2022a) empower CNN in time series modeling and achieve comparable
performance with a Transformer-based model. Recently, MLP-based models led by
N-BEATS (Oreshkin et al., 2019) and DLinear (Zeng et al., 2023) have emerged
as competitive forecasters in the time series community.

Motivated by the great success in the field of natural language processing
(Devlin et al., 2018) and computer vision (Liu et al., 2021b), Transformers
have garnered significant interest in time series data due to their ability to
capture long-term temporal dependencies and complex multivariate correlations.
Categorized based on the granularity of representation learning, Transformer-
based models can be divided into point-wise, patch-wise, and series-wise as
shown in Figure 2. Due to the serial nature of time series, most previous
works use a point-wise representation of time series data and apply attention
mechanisms to capture the correlations among different time points. Therefore,
many efficient Transformers (Zhou et al., 2021; Wu et al., 2021; Liu et al.,
2021a; Zhou et al., 2022) were proposed to reduce the complexity caused by
point-wise modeling. Informer (Zhou et al., 2021) designs a ProbSparse self-
attention to reduce the quadratic complexity in time and memory. Autoformer
(Wu et al., 2021) replaces canoical self-attention with Auto-correlation to
discover the sub-series similarity within time series data. Pyraformer (Liu et
al., 2021a) develops a pyramidal attention module to capture both short- and
long-temporal dependencies with linear time and space complexity. Considering
point-wise representations fall short in revealing local semantic information
lies in the temporal variation, PatchTST (Nie et al., 2022) split time series
data into subseries-level patches and then capture dependencies between
patches. Beyond capturing the patch-level temporal dependencies within one
single series, recent approaches have endeavored to capture interdependencies
among patches from different variables over time. Crossformer (Zhang & Yan,
2022) introduces a Two-Stage Attention layer to efficiently capture the cross-
time and cross-variate dependencies of each patch. Further expanding the
receptive field, iTransformer utilizes the global representation of the whole
series and applies attention to these series-wise representations to capture
multivariate correlations. Yet, most of these methods only focus on
multivariate or univariate time series forecasting and do not conduct special
designs for exogenous variables, which is different from our setting in this
paper.

Figure 2: Illustration of different levels of representation for time series
data, ranging from point-wise, patch-wise to series-wise.

#### Forecasting with Exogenous Variables

Time series forecasting with exogenous variables has been widely discussed in
classical statistical methods. A vast majority of statistical methods have
been extended to include exogenous variables as part of input. Extending the
acknowledged ARIMA model, ARIMAX (Williams, 2001) and SARIMAX (Vagropoulos et
al., 2016) incorporate the correlations between exogenous and endogenous
variables along with the auto-regression on endogenous variables. Although
time series modeling methods have evolved from statistical to deep models,
most of the existing deep models incorporating covariates, such as Temporal
Fusion Transformer (TFT) (Lim et al., 2021), primarily focus on variable
selection. Some approaches, including NBEATSx (Olivares et al., 2023) and TiDE
(Das et al., 2023) contend that forecasting models are capable of accessing
future values of exogenous variables during the prediction of endogenous
variables. It is notable that previous models concatenate exogenous features
with endogenous features at each time point and then map them to a latent
space, necessitating the alignment of the endogenous and exogenous series in
time. However, time series in real-world applications often suffer from
problems such as missing value and uneven sampling, which leads to significant
challenges in modeling the effects of exogenous variables on endogenous
variables. In contrast, TimeXer introduces external information to Transformer
architecture through a deftly designed embedding strategy, which can
effectively introduce the external information into patch-wise representations
of endogenous variables, thereby being enable to adapt to time-lagged or data-
missing records.

##  3 TimeXer

In time series forecasting with exogenous variables, the endogenous time
series represents the values to be predicted, while the exogenous variables
are additional factors that may impact the endogenous series.

#### Problem Definition

Given an endogenous time series
$\mathbf{x}_{1:L}=\\{x_{1},x_{2},...,x_{L}\\}\in\mathbb{R}^{L\times 1}$ and
corresponding exogenous series
$\mathbf{z}_{1:L^{\prime}}=\\{\mathbf{z}^{(1)}_{1:L^{\prime}},\mathbf{z}^{(2)}_{1:L^{\prime}},...,\mathbf{z}^{(C)}_{1:L^{\prime}}\\}\in\mathbb{R}^{L\times
C}$, where $L$ is the length of the look-back window of the endogenous
variable, $x_{i}$ denotes the value at the $i$-th time points, $L^{\prime}$ is
the length of the look-back window of exogenous variables, $C$ is the number
of exogenous variables, and $\mathbf{z}^{(i)}_{1:L^{\prime}}$ represents the
$i$-th exogenous variable. Note that the length of endogenous series $L$ and
exogenous series $L^{\prime}$ can be different. The goal of forecasting model
$f$ is to predict the future $S$ time steps
$\widehat{\mathbf{x}}=\\{x_{L+1},x_{L+2},...,x_{L+S}\\}$ based on both
historical observations $\mathbf{x}_{1:L}$ and corresponding exogenous series
$\mathbf{z}_{1:L^{\prime}}$, which can be formulated as follows:

$\mathbf{\widehat{x}}=f\left(\mathbf{x}_{1:L},\mathbf{z}_{1:L^{\prime}}\right).$
(1)

Figure 3: The overall structure of TimeXer. TimeXer empowers the canonical
Transformer to the processing of exogenous variables. (a) Both patch embedding
and variate embedding are utilized for the endogenous variables to obtain
multiple temporal tokens and a variate token respectively. (b) Each exogenous
variable is embedded as a variate token through variate embedding. (c) Self-
attention is applied to the endogenous temporal tokens to capture patch-wise
dependencies. (d) Cross-attention is adopted to model the series-level
dependencies over endogenous and exogenous variables.

#### Structure Overview

As shown in Figure 3, our proposed TimeXer repurposes canonical Transformer
without modifying any component, where variate embedding and patch embedding
are introduced to address the difference between exogenous and endogenous
variables. TimeXer adopts vanilla self-attention and cross-attention to
capture temporal-wise and variate-wise dependencies respectively.

#### Variate Embedding

Most of the existing Transformer-based forecasting models embed each time
point or a sub-series of time series data as a temporal token and apply self-
attention to learn the temporal dependencies. Recent work (Liu et al., 2023)
attempts to tokenize the whole time series as a variate token and adopt
Transformer architecture to capture multivariate correlation. It is notable
that in time series forecasting with exogenous variables, the model
facilitates the prediction of the endogenous variable by capturing the
correlations between endogenous and exogenous variables. Thus using too fine-
grained representations of exogenous variables not only introduces significant
computational complexity but also may introduce unnecessary noises due to, for
instance, time misalignment. Therefore, we utilize the series-global
representation of both endogenous and exogenous series to model mutual
interactions between different variables. Technically, TimeXer adopts a
variate embedding to embed each series into a variate token, which can be
summarized as follows:

$\displaystyle\mathbf{V}_{en}$
$\displaystyle=\operatorname{EnVariateEmbed}\left(\mathbf{x}\right),$ (2)
$\displaystyle\mathbf{V}_{ex,i}$
$\displaystyle=\operatorname{ExVariateEmbed}\big{(}\mathbf{z}^{(i)}\big{)}\
\text{$i\in\\{1,\cdots,C\\}$}.$

The two variate-embedding layers
$\operatorname{EnVariateEmbed}:\mathbb{R}^{L}\to\mathbb{R}^{D}$ and
$\operatorname{ExVariateEmbed}:\mathbb{R}^{L^{\prime}}\to\mathbb{R}^{D}$ are
instantiated as trainable linear projectors, where $L$ and $L^{\prime}$ are
the look-back length of endogenous and exogenous series respectively. In this
way, the embedded vector contains information of the whole series.
$\mathbf{V}_{ex}=\\{\mathbf{V}_{ex,i}\\}_{i=1}^{C}$ denotes the set of
representations for multiple exogenous series.

#### Patch Embedding

Simultaneously, since the temporal causality is crucial for the forecast of
the endogenous variable, we include the patch-wise representation to finely
model the endogenous variable. Concretely, the endogenous series is split into
non-overlap patches and each patch is projected to a temporal token. The patch
embedding of the endogenous variable can be formulated as:

$\displaystyle\\{\mathbf{s}_{1},\mathbf{s}_{2},...,\mathbf{s}_{N}\\}$
$\displaystyle=\operatorname{Patchify}\left(\mathbf{x}\right),$ (3)
$\displaystyle\mathbf{P}_{en}$
$\displaystyle=\operatorname{PatchEmbed}\left(\mathbf{s}_{1},\mathbf{s}_{2},...,\mathbf{s}_{N}\right).$

Denote the patch length as $P$, $N=\lfloor\frac{L}{P}\rfloor$ is the number of
patches of the endogenous series, and $\mathbf{s}_{i}$ denotes the $i$-th
patch in the patch sequence. The $\operatorname{PatchEmbed}()$ map each patch
of length $P$ into $D$ dimensions with a trainable linear projection and
position embedding. In this way, the patch embedding of the endogenous
variable $\mathbf{P}_{en}\in\mathbb{R}^{N\times D}$ contains $N$ patch-wise
temporal tokens of $D$ dimension.

#### Patch-wise Self-Attention

In time series forecasting, it is essential for the model to capture the
intrinsic temporal dependencies within the endogenous variable. Considering
the exogenous variables that do not need to be predicted by the model, TimeXer
applies attention to all endogenous tokens. Concretely, the embedded vector of
endogenous variables contains multiple patch-wise temporal tokens
$\mathbf{\widehat{P}}_{en}$ and a series-wise variate token
$\mathbf{\widehat{V}}_{en}$, where the variate token serves as a global token
of the whole series, incorporating information from other temporal tokens.
This provides a comprehensive view of the temporal dependencies within the
endogenous variable, as well as better interaction with the series-wise
variate tokens of the exogenous variables. The process can be formalized as:

$\mathbf{\widehat{P}}^{l}_{en},\mathbf{\widehat{V}}^{l}_{en}=\operatorname{LN}\Big{(}\left[\mathbf{P}^{l}_{en},\mathbf{V}^{l}_{en}\right]+\operatorname{Self-
Attn}\left(\left[\mathbf{P}^{l}_{en},\mathbf{V}^{l}_{en}\right]\right)\Big{)},$
(4)

where $l\in\\{0,\cdots,M-1\\}$ denotes the $l$-th TimeXer block.
$\mathbf{P}^{l}_{en}$ and $\mathbf{V}^{l}_{en}$ are the patch-wise and
variate-wise token of the endogenous variable respectively, where
$\mathbf{P}^{0}_{en},\mathbf{V}^{0}_{en}$ are the embedded
$\mathbf{P}_{en},\mathbf{V}_{en}$. Here, $\left[\cdot,\cdot\right]$ denotes
the concatenation of the patch-wise and series-wise tokens of the endogenous
variable along the sequence dimension. $\operatorname{Self-Attn}(\cdot)$
denotes the Multi-head Self-Attention over the endogenous tokens.
$\operatorname{LN}(\cdot)$ denotes Layer Normalization.

#### Variate-wise Cross-Attention

Cross-attention has been widely used in multi-modal learning (Li et al., 2021)
to extract the dependencies between two different embedding vectors. In
TimeXer, the cross-attention layer takes the endogenous variable as query and
the exogenous variable as key and value to build the connection between the
two types of variables. Since the exogenous variables are embedded as series-
level variate tokens and are processed by the self-attention layer, we use the
variate token of the endogenous variable to integrate information from
exogenous variables. Then the output of cross-attention layers is then
concatenated with the other endogenous temporal tokens. The above process can
be formalized as:

$\displaystyle\mathbf{V}^{l+1}_{en}$
$\displaystyle=\operatorname{LN}\left(\mathbf{\widehat{V}}^{l}_{en}+\operatorname{Cross-
Attn}\big{(}\mathbf{\widehat{V}}^{l}_{en},\mathbf{V}_{ex},\mathbf{V}_{ex}\big{)}\right).$
(5)

Here, $\operatorname{Cross-Attn}(\cdot)$ denotes the Multi-head Cross-
Attention between the endogenous and exogenous variables, where the query
$\mathbf{\widehat{V}}^{l}_{en}$ is the endogenous variate token from the
$l$-th TimeXer block, keys and values are the exogenous variate token
$\mathbf{V}^{l}_{ex}$. The output of the cross-attention layer
$\mathbf{\widehat{V}}^{l}_{en}$ will become the input of the next self-
attention layer. In this way, the multivariate correlations learned by the
endogenous variate token can be further diffused to the endogenous temporal
tokens, realizing the information transfer between exogenous variate tokens
and endogenous temporal tokens.

Forecasting Loss As aforementioned in the problem definition, since the
exogenous variables do not need to be predicted by the model, we apply a
linear projection to the decoder output to yield the prediction of endogenous
variables. Finally, we opt to use the L2 loss to measure the discrepancy
between the prediction and the ground truth:

$\displaystyle\mathbf{\widehat{x}}$
$\displaystyle={\rm{Projection}}(\mathbf{V}^{L}_{en}),$ (6)
$\displaystyle\rm{Loss}$
$\displaystyle=\sum_{i=1}^{S}\|\mathbf{x}_{i}-\widehat{\mathbf{x}}_{i}\|_{2}^{2}.$

where $L$ is the number of TimeXer block,
$\mathbf{\widehat{x}}\in\mathbb{R}^{S\times 1}$ is the prediction of future
$S$ time steps of the endogenous variable.

##  4 Experiments

To verify the model’s effectiveness, we extensively evaluate our proposed
TimeXer on a diverse range of real-world time series datasets from different
domains.

#### Datasets

To completely evaluate the forecasting capability of our proposed TimeXer, we
conduct both short-term forecasting and long-term forecasting settings. For
short-term forecasting tasks, we consider Electricity Price short-term
Forecasting datasets (EPF) (Lago et al., 2021), which is a real-world
forecasting benchmark derived from five major power market data spanning six
years each. Each dataset contains electricity price as an endogenous variable
and two influential exogenous variables in practice. As for long-term
forecasting, we adopt seven well-established public long-term multivariate
forecasting benchmarks to perform forecasting with exogenous variables. We
summarize the domain information of endogenous and exogenous variables for
each dataset in Table 1.

Table 1: Dataset descriptions. Note that the number of endogenous variables is
1 for all datasets, corresponding to multiple exogenous variables. Detailed
descriptions are listed in the Appendix 5.

Dataset |  Exogenous |  Endogenous |  #Num
---|---|---|---
Electricity |  Electricity |  Electricity |  320
Weather |  Climate |  Climate |  20
ETTh |  Energy |  Temperature |  6
ETTm |  Energy |  Temperature |  6
Traffic |  Transportation |  Transportation |  820
NP |  Energy |  Price |  2
PJM |  Energy |  Price |  2
BE |  Energy |  Price |  2
FR |  Energy |  Price |  2
DE |  Energy |  Price |  2

#### Baselines

We include 10 state-of-the-art deep forecasting models as our baselines,
including Transformer-based model: iTransformer (Liu et al., 2023), PatchTST
(Nie et al., 2022), Crossformer (Zhang & Yan, 2022), Stationary (Liu et al.,
2022b), Autoformer (Wu et al., 2021), CNN-based model: TimesNet (Wu et al.,
2023a), SCINet (Liu et al., 2022a), and Linear-based model: RLinear (Li et
al., 2023), DLiear (Zeng et al., 2023), TiDE (Das et al., 2023). Among these
models, TiDE is an advanced recent forecaster elaborated for exogenous
variables.

#### Implementation Details

For short-term forecasting on EPF datasets, we set the patch length to 24 and
follow the standard protocol in short-term electricity price forecasting
(Olivares et al., 2023), where the input length and predict length are set to
168 and 24. For long-term forecasting datasets, we uniformly set the patch
length to 16 and fix the length of the lookback series as 96 and the
prediction length varies $\\{96,192,336,720\\}$.

Table 2: Full results of the short-term forecasting task on EPF dataset. We
follow the standard protocol in short-term electricity price forecasting,
where the input length and predict length are set to 168 and 24 respectively
for all baselines. Avg means the average results from all five datasets. The
character “.” in the Transformers indicates the name of *former.

Model |  TimeXer |  iTrans. |  RLinear |  PatchTST |  Cross. |  TiDE |  TimesNet |  DLinear |  SCINet |  Stationary |  Auto.
---|---|---|---|---|---|---|---|---|---|---|---
(Ours) |  (2023) |  (2023) |  (2022) |  (2022) |  (2023) |  (2023a) |  (2023) |  (2022a) |  (2022b) |  (2021)
Metric |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE
NP |  0.238 |  0.268 |  0.265 |  0.300 |  0.335 |  0.340 |  0.267 |  0.284 |  0.245 |  0.289 |  0.335 |  0.340 |  0.250 |  0.289 |  0.309 |  0.321 |  0.373 |  0.368 |  0.294 |  0.308 |  0.402 |  0.398
PJM |  0.088 |  0.188 |  0.097 |  0.197 |  0.124 |  0.229 |  0.106 |  0.209 |  0.149 |  0.198 |  0.124 |  0.228 |  0.097 |  0.195 |  0.108 |  0.215 |  0.143 |  0.259 |  0.122 |  0.228 |  0.168 |  0.267
BE |  0.374 |  0.241 |  0.394 |  0.270 |  0.520 |  0.337 |  0.403 |  0.264 |  0.436 |  0.294 |  0.523 |  0.336 |  0.419 |  0.288 |  0.463 |  0.313 |  0.731 |  0.412 |  0.433 |  0.289 |  0.500 |  0.333
FR |  0.381 |  0.211 |  0.439 |  0.233 |  0.507 |  0.290 |  0.411 |  0.220 |  0.440 |  0.216 |  0.510 |  0.290 |  0.431 |  0.234 |  0.429 |  0.260 |  0.855 |  0.384 |  0.466 |  0.242 |  0.519 |  0.295
DE |  0.440 |  0.418 |  0.479 |  0.443 |  0.574 |  0.498 |  0.461 |  0.432 |  0.540 |  0.423 |  0.568 |  0.496 |  0.502 |  0.446 |  0.520 |  0.463 |  0.565 |  0.497 |  0.483 |  0.447 |  0.674 |  0.544
AVG |  0.304 |  0.265 |  0.335 |  0.289 |  0.412 |  0.339 |  0.330 |  0.282 |  0.362 |  0.284 |  0.412 |  0.338 |  0.340 |  0.290 |  0.366 |  0.314 |  0.533 |  0.384 |  0.360 |  0.303 |  0.453 |  0.368

Table 3: Full results of the long-term forecasting with exogenous variables.
We compare extensive competitive models under different prediction lengths
following the setting of iTransformer (Liu et al., 2023). The input sequence
length is set to 96 for all baselines. Results are averaged from all
prediction lengths. The character “.” in the Transformers indicates the name
of *former. “-” denotes out of memory (OOM) problem. The complete results are
listed in the Appendix 7.

Model |  TimeXer |  iTrans. |  RLinear |  PatchTST |  Cross. |  TiDE |  TimesNet |  DLinear |  SCINet |  Stationary |  Auto.
---|---|---|---|---|---|---|---|---|---|---|---
(Ours) |  (2023) |  (2023) |  (2022) |  (2022) |  (2023) |  (2023a) |  (2023) |  (2022a) |  (2022b) |  (2021)
Metric |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE
ECL |  0.336 |  0.414 |  0.365 |  0.442 |  0.444 |  0.486 |  0.394 |  0.446 |  0.344 |  0.412 |  0.419 |  0.468 |  0.410 |  0.476 |  0.393 |  0.457 |  0.427 |  0.490 |  0.372 |  0.450 |  0.495 |  0.528
Weather |  0.002 |  0.031 |  0.002 |  0.031 |  0.002 |  0.029 |  0.002 |  0.031 |  0.005 |  0.055 |  0.002 |  0.029 |  0.097 |  0.115 |  0.006 |  0.066 |  0.007 |  0.071 |  0.002 |  0.031 |  0.006 |  0.060
ETTh1 |  0.074 |  0.210 |  0.075 |  0.211 |  0.084 |  0.224 |  0.078 |  0.215 |  0.285 |  0.447 |  0.083 |  0.223 |  0.076 |  0.215 |  0.116 |  0.259 |  0.437 |  0.565 |  0.110 |  0.256 |  0.130 |  0.282
ETTh2 |  0.183 |  0.337 |  0.199 |  0.352 |  0.205 |  0.356 |  0.192 |  0.345 |  1.027 |  0.873 |  0.205 |  0.356 |  0.210 |  0.362 |  0.224 |  0.369 |  1.155 |  0.955 |  0.262 |  0.405 |  0.242 |  0.386
ETTm1 |  0.051 |  0.169 |  0.053 |  0.175 |  0.053 |  0.173 |  0.053 |  0.173 |  0.411 |  0.548 |  0.053 |  0.173 |  0.054 |  0.175 |  0.066 |  0.188 |  0.098 |  0.241 |  0.077 |  0.204 |  0.085 |  0.230
ETTm2 |  0.116 |  0.252 |  0.127 |  0.267 |  0.122 |  0.261 |  0.120 |  0.258 |  0.976 |  0.769 |  0.122 |  0.261 |  0.129 |  0.271 |  0.126 |  0.263 |  0.685 |  0.713 |  0.207 |  0.333 |  0.154 |  0.305
Traffic |  0.150 |  0.227 |  0.161 |  0.246 |  0.324 |  0.412 |  0.173 |  0.253 |  - |  - |  0.324 |  0.411 |  0.171 |  0.264 |  0.323 |  0.404 |  0.447 |  0.500 |  0.361 |  0.361 |  0.302 |  0.353

###  4.1 Main Results

Comprehensive short-term and long-term forecasting results are listed in Table
2 and Table 3. A lower MSE or MAE indicates more accurate forecasting results.

Short-term Electricity Price Forecasting task is derived from real-world
scenarios, and presents a unique challenge for the forecasting model for the
endogenous variable have been shown to be highly correlated with two exogenous
variables in the dataset. As shown in Table 2, TimeXer achieves consistent
state-of-the-art performance on all five datasets, outperforming various
baseline models. Notably, linear forecasters fail to triumph over Transformer-
based forecasters in this task. In contrast, with attention mechanisms,
Transformer-based models are better equipped to capture relationships between
intra-time and inter-time series data. It is notable that for channel-
independent models led by DLinear and PathTST, while they include exogenous
variables as part of the input, they solely perform univariate forecasting on
the endogenous variable. In contrast, TimeXer effectively integrates
information from exogenous variables and surpasses the performance of
univariate forecasting. Similar to TimeXer, Crossformer divides all input
series into segments and captures multivariate correlations over all segments;
however, it fails to outperform other baselines, suggesting that modeling all
variables at a granular level introduces unnecessary noise into the
forecasting. Conversely, iTransformer neglects the temporal-wise attention
module, indicating that there are still limitations in capturing temporal
dependencies solely through linear projection. Therefore, TimeXer strikes a
balance between avoiding distractions caused by fine-grained modeling of
exogenous variables and addressing inaccuracies resulting from coarse-grained
temporal modeling of endogenous variables, ultimately achieving state-of-the-
art performance.

We also evaluate the performance of TimeXer on established long-term
forecasting benchmarks. Unlike short-term forecasts, which contain only two
releated exogenous variables, the long-term multivariate dataset has a large
number of exogenous variables, and not all exogenous variables are related to
the endogenous variable. As listed in Table 3, TimeXer shows great performance
in the vast majority datasets, which demonstrates that TimeXer are effective
in capturing beneficial information for forecasting.

###  4.2 Abalation Studies

#### Masking Exogenous Variables

Real-world time series encounter problems such as missing data that result in
low-quality data. In this section, we use random masking to replicate these
situations and further explore the forecasting performance when fed low-
quality data. Previous works (Dong et al., 2023) have demonstrated that the
semantic information of time series lies in temporal variation, We use
complete, high-quality historical data for the endogenous variables and
progressively reduce the quality of the data for the exogenous variables by
increasing the masking ratio from 0% (i.e., using complete historical data for
the exogenous variables) to 99%. It can be observed that with the decrease in
the quality of the exogenous series, the forecasting performance of the model
also decreases. Notably, our model maintains a competitive performance when
the exogenous series are masked by a small amount, which indicates that our
proposed TimeXer is capable of supporting low-quality data scenarios.

Figure 4: Forecasting performance with the masked exogenous series on three
EPF datasets, simulating the missing data scenario.

#### Varying Different Embeddings

In our TimeXer, we utilize inverted embedding to obtain series-level
representations of exogenous and endogenous variables. Besides, patch
embedding is utilized to get patch-level temporal representations of the
endogenous variable. To elaborate on the validity of our embedding strategy
for endogenous and exogenous variables, we conducted detailed ablations.
Specifically, for exogenous variables, we replace existing series-level
representations with temporal-wise representations. For endogenous variables,
we remove different components of the embedded vector. As shown in Table 4,
among various architectural designs, TimeXer exhibits superior performance
across all datasets. Compared to the second design which replaces the variate
token for each exogenous variable with multiple temporal tokens to capture
interrelationships between the different variables, TimeXer achieves superior
forecasting performance with lower complexity. Notably, the performance of the
third and fourth designs indicates that for endogenous variables, both
temporal and variate tokens are indispensable.

Table 4: Ablation Results. _Ex._ and _En._ are abbreviations for Exogenous
variable and Endogenous variable. _T_ and _V_ denote temporal token and
variate token respectively. Results are averaged from all prediction lengths.
Full results are listed in the Appendix.

Design | En. | Ex. | ETTh2 | ETTm2 | Traffic
---|---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE
Ours | T+V | V | 0.183 | 0.337 | 0.116 | 0.252 | 0.150 | 0.227
Replace | T+V | T | 0.192 | 0.343 | 0.122 | 0.259 | 0.158 | 0.239
w/o | w/o V | V | 0.197 | 0.347 | 0.117 | 0.254 | 0.158 | 0.239
w/o T | V | 0.188 | 0.341 | 0.118 | 0.256 | 0.152 | 0.233

###  4.3 Model Analysis

Figure 5: Visualization of learned attention map alongside the endogenous time
series and the exogenous time series with highest and lowest attention scores.

#### Analysis of Variate-wise Correlations

TimeXer adopts cross-attention on series-wise variate tokens to capture the
multivariate correlation between the endogenous and exogenous variables,
enhancing the interpretability of the learned attention map. To validate the
rationale behind attention on variate tokens, we visualize the learned
attention map alongside the time series with the highest and lowest attention
scores. As illustrated in Figure reffig:attn-vis, the case study on the
Weather dataset demonstrates that TimeXer is able to distinguish between
exogenous variables that exhibit similarities with the endogenous variable and
those with little to no correlation. The former receives greater attention,
while the latter is given less attention, resulting in a more focused and
interpretable attention map. Furthermore, physical interpretations for the
visualized are provided. For the endogenous variable CO2-Concentration, there
is indeed a strong correlation between it and Air Density, while the Maximum
Wind Velocity a relatively minor impact, which validates the effectiveness of
cross attention in TimeXer. Moreover, the cross-attention enables the model to
identify and emphasize influential exogenous variables, thus improving the
interpretability.

Figure 6: Forecasting performance with the look-back length of exogenous
variables varying from $\\{96,192,336,512,720\\}$ with a fixed look-back
length of the endogenous variable to 96. Different styles of lines represent
different prediction lengths. In most cases, the forecasting performance
benefits from the increase of exogenous look-back length.

#### Representation Analysis

To evaluate the performance of TimeXer from the perspective of representation,
we adopt centered kernel alignment (CKA) similarity (Kornblith et al., 2019)
analysis. Previous works (Wu et al., 2023a; Dong et al., 2023) reveal that for
low-level time series task including forecasting, there is a great similarity
among representations from the different layers and a higher similarity
corresponds to a better performance. As shown in Figure 7, the representations
from the first and the last layers of TimeXer enjoy great similarities,
verifying that TimeXer can learn the appropriate representations for the
prediction. It is notable that iTransformer does not distinguish between
endogenous and exogenous variables and the output of the model contains
representation of all variables. However, the result of the CKA analysis shows
that despite the high similarity of the series representations of all
variables, the representation of the endogenous variables was not well
learned. This result also suggests that directly applying a multivariate model
to perform forecasting with exogenous variables introduces unnecessary noise
into the model thus interfering with its forecasting performance.

Figure 7: Comparison of representation CKA similarity between TimeXer and
competitive baselines. _iTrm-All_ denotes the series representation of all
variables learned by iTransformer, _iTrm-En_ is the learned series
representation of the endogenous variable.

#### Increasing Exogenous Look-back Length

Theoretically, the forecasting performance of the model could potentially
benefit from an increased length of the look-back window, as a longer
historical context can provide a more comprehensive modeling. However, the
attention will be distracted when the window becomes excessively long. In
TimeXer, we use the series-level representation of exogenous variables which
allows for the misalignment between endogenous and exogenous variables. This
is particularly valuable in real-world scenarios where the endogenous series
may be collected from a newly introduced sensor that has limited historical
data. In such cases, using a longer exogenous series can effectively augment
the available information for more robust predictions. Based on this
motivation, we evaluate the performance of TimeXer in Figure 6 with increased
exogenous look-back length. We fix the length of the endogenous variable to 96
and increase the length of exogenous variables. The results reveal that
extending the look-back length of exogenous variables indeed yields
improvements in forecasting performance.

#### Extend to Multivariate Forecasting

The major difference between time series forecasting with exogenous variables
and multivariate forecasting lies in the fact that the model does not require
predicting the exogenous variables. In principle, multivariate forecasting can
be viewed as the independent prediction of each variable in multivariate data,
and for each variable, the other variables are exogenous to facilitate the
prediction. Motivated by this insight, we endeavored to extend the
capabilities of TimeXer to encompass multivariate forecasting scenarios.
Technically, we consider the variables in the dataset as mutually independent
endogenous variables, with each variable considering all other variables
except itself as exogenous variables. Leveraging the established long-term
multivariate forecasting benchmarks and experimental settings from
iTransformer (Liu et al., 2023), we conduct mainstream multivariate
forecasting task and evaluate the performance of our proposed TimeXer with
state-of-the-art forecasting models. We present the results averaged from all
four prediction lengths in Figure 8. It can be observed that TimeXer exhibits
competitive performance in multivariate forecasting tasks with baseline
models, highlighting its effectiveness and generality.

Figure 8: Performance Comparison between TimeXer with state-of-art models.
Full results are listed in Appendix 8.

##  5 Conclusion

Considering the prevalence of exogenous variables in real-world forecasting
scenarios, we empower the canonical Transformer architecture with the ability
to reconcile endogenous and exogenous information without architectural
modifications. With a deftly designed embedding strategy, our proposed TimeXer
is able to capture both endogenous temporal dependencies and multivariate
correlations between endogenous and exogenous variables. Experimental results
demonstrate that our proposed TimeXer achieves state-of-the-art performance in
both the short- and long-term forecasting task with exogenous variables. Going
beyond the experimental setting, TimeXer demonstrates its potential in a
variety of complex real-world forecasting scenarios containing exogenous
variables, including challenges such as low data quality and misalignment
between series.

## References

  * Das et al. (2023) Das, A., Kong, W., Leach, A., Sen, R., and Yu, R.  Long-term forecasting with tide: Time-series dense encoder.  _arXiv preprint arXiv:2304.08424_ , 2023.
  * Devlin et al. (2018) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Dong et al. (2023) Dong, J., Wu, H., Zhang, H., Zhang, L., Wang, J., and Long, M.  Simmtm: A simple pre-training framework for masked time-series modeling.  In _NeurIPS_ , 2023.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kornblith et al. (2019) Kornblith, S., Norouzi, M., Lee, H., and Hinton, G.  Similarity of neural network representations revisited.  In _International conference on machine learning_ , pp. 3519–3529. PMLR, 2019.
  * Lago et al. (2021) Lago, J., Marcjasz, G., De Schutter, B., and Weron, R.  Forecasting day-ahead electricity prices: A review of state-of-the-art algorithms, best practices and an open-access benchmark.  _Applied Energy_ , 293:116983, 2021.
  * Lai et al. (2018) Lai, G., Chang, W.-C., Yang, Y., and Liu, H.  Modeling long-and short-term temporal patterns with deep neural networks.  In _The 41st international ACM SIGIR conference on research & development in information retrieval_, pp. 95–104, 2018.
  * Li et al. (2021) Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., and Hoi, S. C. H.  Align before fuse: Vision and language representation learning with momentum distillation.  _Advances in neural information processing systems_ , 34:9694–9705, 2021.
  * Li et al. (2019) Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.-X., and Yan, X.  Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting.  _Advances in neural information processing systems_ , 32, 2019.
  * Li et al. (2023) Li, Z., Qi, S., Li, Y., and Xu, Z.  Revisiting long-term time series forecasting: An investigation on linear mapping.  _arXiv preprint arXiv:2305.10721_ , 2023.
  * Lim et al. (2021) Lim, B., Arık, S. Ö., Loeff, N., and Pfister, T.  Temporal fusion transformers for interpretable multi-horizon time series forecasting.  _International Journal of Forecasting_ , 37(4):1748–1764, 2021.
  * Liu et al. (2022a) Liu, M., Zeng, A., Chen, M., Xu, Z., Lai, Q., Ma, L., and Xu, Q.  Scinet: Time series modeling and forecasting with sample convolution and interaction.  _Advances in Neural Information Processing Systems_ , 35:5816–5828, 2022a.
  * Liu et al. (2021a) Liu, S., Yu, H., Liao, C., Li, J., Lin, W., Liu, A. X., and Dustdar, S.  Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting.  In _International conference on learning representations_ , 2021a.
  * Liu et al. (2022b) Liu, Y., Wu, H., Wang, J., and Long, M.  Non-stationary transformers: Exploring the stationarity in time series forecasting.  2022b.
  * Liu et al. (2023) Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., and Long, M.  itransformer: Inverted transformers are effective for time series forecasting.  _arXiv preprint arXiv:2310.06625_ , 2023.
  * Liu et al. (2021b) Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 10012–10022, 2021b.
  * Lv et al. (2014) Lv, Y., Duan, Y., Kang, W., Li, Z., and Wang, F.-Y.  Traffic flow prediction with big data: A deep learning approach.  _IEEE Transactions on Intelligent Transportation Systems_ , 16(2):865–873, 2014.
  * Nie et al. (2022) Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J.  A time series is worth 64 words: Long-term forecasting with transformers.  _arXiv preprint arXiv:2211.14730_ , 2022.
  * Olivares et al. (2023) Olivares, K. G., Challu, C., Marcjasz, G., Weron, R., and Dubrawski, A.  Neural basis expansion analysis with exogenous variables: Forecasting electricity prices with nbeatsx.  _International Journal of Forecasting_ , 39(2):884–900, 2023.
  * Oreshkin et al. (2019) Oreshkin, B. N., Carpov, D., Chapados, N., and Bengio, Y.  N-beats: Neural basis expansion analysis for interpretable time series forecasting.  _arXiv preprint arXiv:1905.10437_ , 2019.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * Salinas et al. (2020) Salinas, D., Flunkert, V., Gasthaus, J., and Januschowski, T.  Deepar: Probabilistic forecasting with autoregressive recurrent networks.  _International Journal of Forecasting_ , 36(3):1181–1191, 2020.
  * Vagropoulos et al. (2016) Vagropoulos, S. I., Chouliaras, G., Kardakos, E. G., Simoglou, C. K., and Bakirtzis, A. G.  Comparison of sarimax, sarima, modified sarima and ann-based models for short-term pv generation forecasting.  In _2016 IEEE international energy conference (ENERGYCON)_ , pp. 1–6. IEEE, 2016.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Weron (2014) Weron, R.  Electricity price forecasting: A review of the state-of-the-art with a look into the future.  _International journal of forecasting_ , 30(4):1030–1081, 2014.
  * Williams (2001) Williams, B. M.  Multivariate vehicular traffic flow prediction: evaluation of arimax modeling.  _Transportation Research Record_ , 1776(1):194–200, 2001.
  * Wu et al. (2021) Wu, H., Xu, J., Wang, J., and Long, M.  Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.  _Advances in Neural Information Processing Systems_ , 34:22419–22430, 2021.
  * Wu et al. (2023a) Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M.  Timesnet: Temporal 2d-variation modeling for general time series analysis.  In _International Conference on Learning Representations_ , 2023a.
  * Wu et al. (2023b) Wu, H., Zhou, H., Long, M., and Wang, J.  Interpretable weather forecasting for worldwide stations with a unified deep model.  _Nature Machine Intelligence_ , pp. 1–10, 2023b.
  * Zeng et al. (2023) Zeng, A., Chen, M., Zhang, L., and Xu, Q.  Are transformers effective for time series forecasting?  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 37, pp. 11121–11128, 2023.
  * Zhang & Yan (2022) Zhang, Y. and Yan, J.  Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Zhang et al. (2023) Zhang, Y., Long, M., Chen, K., Xing, L., Jin, R., Jordan, M. I., and Wang, J.  Skilful nowcasting of extreme precipitation with nowcastnet.  _Nature_ , 619(7970):526–532, 2023.
  * Zhou et al. (2021) Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pp. 11106–11115, 2021.
  * Zhou et al. (2022) Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., and Jin, R.  Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting.  In _International Conference on Machine Learning_ , pp. 27268–27286. PMLR, 2022.

##  Appendix A Implementation Details

###  A.1 Dataset Descriptions

We conduct long-term forecasting experiments on 7 real-world datasets to
evaluate the performance of our proposed TimeXer, including: (1) ECL (Li et
al., 2019) includes hourly electricity consumption data from 321 clients. We
take the electricity consumption of the last client as an endogenous variable
and other clients as exogenous variables. (2) Weather (Zhou et al., 2021)
records 21 meteorological factors collected every 10 minutes from the Weather
Station of the Max Planck Biogeochemistry Institute in 2020. In our
experiment, we use the Wet Bulb factor as the endogenous variable to be
predicted and the other indicators as exogenous variables. (3) ETT (Zhou et
al., 2021) contains four subsets where ETTh1 and ETTh2 are hourly recorded,
and ETTm1 and ETTm2 are recorded every 15 minutes. The endogenous variable is
the oil temperature and the exogenous variables are 6 power load features. (4)
Traffic (Wu et al., 2023a) records hourly road occupancy rates measured by 862
sensors of San Francisco Bay area freeways. We take the measurement of the
last sensor as an endogenous variable and others as exogenous variables.

In addition to the public multivariate time series datasets, we perform short-
term forecasting on the Electricity Price Forecasting Datasets (Lago et al.,
2021), which contains five datasets representing five different day-ahead
electricity markets spanning six years each. Here are the descriptions of the
datasets: (1) NP represents The Nord Pool electricity market, recording the
hourly electricity price, and corresponding grid load and wind power forecast
from 2013-01-01 to 2018-12-24. (2) PJM represents the Pennsylvania-New Jersey-
Maryland market, which contains the zonal electricity price in the
Commonwealth Edison (COMED), and corresponding System load and COMED load
forecast from 2013-01-01 to 2018-12-24. (3) BE represents Belgium’s
electricity market, recording the hourly electricity price, load forecast in
Belgium, and generation forecast in France from 2011-01-09 to 2016-12-31. (4)
FR represents the electricity market in France, recording the hourly prices,
and corresponding load and generation forecast from 2012-01-09 to 2017-12-31.
(5) DE represents the German electricity market, recording the hourly prices,
the zonal load forecast in the TSO Amprion zone, and the wind and solar
generation forecasts from 2012-01-09 to 2017-12-31.

Table 5: Dataset descriptions. _Ex._ and _En._ are abbreviations for the
Exogenous variable and Endogenous variable, respectively. The dataset size is
organized in (Train, Validation, Test)

Dataset | #Num | Ex. Descriptions | En. Descriptions | Sampling Frequency | Dataset Size
---|---|---|---|---|---
Electricity | 320 | Electricity Consumption | Electricity Consumption | 1 Hour | (18317, 2633, 5261)
Weather | 20 | Climate Feature | CO2-Concentration | 10 Minutes | (36792, 5271, 10540)
ETTh | 6 | Power Load Feature | Oil Temperature | 1 Hour | (8545, 2881, 2881)
ETTm | 6 | Power Load Feature | Oil Temperature | 15 Minutes | (34465, 11521, 11521)
Traffic | 861 | Road Occupancy Rates | Road Occupancy Rates | 1 Hour | (12185, 1757, 3509)
NP | 2 | Grid Load, Wind Power | Nord Pool Electricity Price | 1 Hour | (36500, 5219, 10460)
PJM | 2 | System Load, SyZonal COMED load | Pennsylvania-New Jersey-Maryland | 1 Hour | (36500, 5219, 10460)
Electricity Price
BE | 2 | Generation, System Load | Belgium’s Electricity Price | 1 Hour | (36500, 5219, 10460)
FR | 2 | Generation, System Load | France’s Electricity Price | 1 Hour | (36500, 5219, 10460)
DE | 2 | Wind power, Ampirion zonal load | German’s Electricity Price | 1 Hour | (36500, 5219, 10460)

###  A.2 Implementation Details

All the experiments are implemented in PyTorch (Paszke et al., 2019) and
conducted on a single NVIDIA 4090 24GB GPU. We utilize ADAM (Kingma & Ba,
2014) with an initial learning rate $10^{-4}$ and L2 loss for the model
optimization. The training process is fixed to $10$ epochs with an early
stopping. We set the number of TimeXer blocks in our proposed model
$L\in\\{1,2,3\\}$. The dimension of series representations $d_{model}$ is
searched from $\\{128,256,512\\}$. The patch length is uniformly set to 16 for
long-term forecasts and 24 for short-term forecasts. We reproduced the
compared baseline models based on the benchmark of TimesNet (Wu et al., 2023a)
Repository.

##  Appendix B Model Efficiency

To evaluate the efficiency of TimeXer, we evaluate the training speed and
memory usage of TimeXer with six baseline models with identical hidden
dimensions and batch sizes. Specifically, we choose ECL and NP datasets,
corresponding to the two representative cases of excessive (320 exogenous
variables) and few (2 exogenous variables) variables, respectively. It can be
observed that when there are few exogenous variables, our model has no
significant advantage in efficiency, but the forecasting performance is
optimal. The results indicate that although our model exhibits optimal
forecasting performance, there is little efficiency advantage when dealing
with a small number of exogenous variables. It is notable that when faced with
numerous variables TimeXer demonstrates its advantage by outperforming
iTransformer in terms of memory footprint. This is because, in TimeXer, the
Cross-Attention layer learns the dependencies between exogenous and endogenous
variables, whereas iTransformer learns the multivariate relationships between
all the variables, leading to higher computational complexity.

Figure 9: Model efficiency comparison on ECL and NP datasets.

##  Appendix C Ablation Study

###  C.1 Varying Patch Length

In this section, we experiment with the effect of patch lengths on the
forecasting performance. We fix the look-back window to 96 and vary the patch
lengths $P\in\\{4,6,8,12,16,24\\}$. We conduct experiments on all datasets
with consistent parameters except patch-length, and the results are shown in
Figure 10. The results are the averaged MSE under four different prediction
length $S\in\\{96,192,336,720\\}$. It can be seen that the average prediction
performance does not vary dramatically with different patch lengths, which
indicates that our model is robust to the patch length hyperparameter.
Notably, the model generally performs lower when the patch length is small,
which is probably because small patches are not enough to represent the
semantic information in time series data.

Figure 10: Hyper-parameter sensitivity analysis of TimeXer on twelve real-
world forecasting benchmarks.

###  C.2 Varying Different Embedding

In our above-mentioned analysis, we apply different embedding strategies to
the endogenous and exogenous variables. For the endogenous variable, we apply
patch embedding and variate embedding to capture both temporal dependencies
and cross-attention between different variables. To verify the rationality of
this setting, we select two additional settings. First, we replace variate
embedding with patch embedding on exogenous variables, while keeping the
embedding strategy on the endogenous variable fixed. Second, we modify the
embedding design on the endogenous variable by removing either temporal
embedding or patch embedding. We calculate the average MSE loss on four
different prediction lengths to fully evaluate the performance of each
setting.

Temporal dependencies and cross-attention are both important in the endogenous
variable embedding. Our numerical experiments in Table 4 verify that taking
down either kind of embedding from the endogenous variable weakens the
averaged performance. Precisely, the two different embedding choices can be
seen as two independent threads of the forecasting process. First, we believe
that the evolving trend of a time-series variable in the future shall follow
the trend that it has been following in the past. The patch embedding gains
the temporal dependencies of how the endogenous variable itself has been
evolving, which is important evidence of the future evolution of this
endogenous variable. Second, we also believe that the evolving trend of a
time-series variable shall be strongly related to its covariant. In
Transformer-based deep learning models, researchers found cross-attention a
powerful tool to estimate variable relations. The variate embedding gains the
cross-attention between the endogenous variable and all exogenous variables,
which is important evidence of how exogenous variables are influencing the
endogenous variable during evolution. By combining these two kinds of
information, we achieve better average performance than using only one kind on
most datasets.

Cross-attention information of exogenous variables leads to better performance
generally. When replacing patch embedding with temporal embedding, the average
MSE loss slightly rises compared to our original design. This indicates that
extracting cross-attention information from all exogenous variables better is
more beneficial than perceiving temporal dependencies for most time-series
learning tasks. Intuitively, exogenous variables serve as background
information for forecasting the endogenous variables. Thus, the model should
learn the relationship between exogenous variables and the specific endogenous
variable to achieve better performance. In this case, we use cross-attention
to represent the correlation between different exogenous variables, which we
believe has enabled better construction of this endogenous-exogenous variable
relation during the machine learning process. Temporal embedding, however, is
not able to learn the correlation between different variables, which leads to
this performance decline in Table 4.

Table 6: Full results of the ablation study.

Design | Endogenous | Exogenous | Horizon | ETTh2 | ETTm2 | Traffic
---|---|---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE
Ours | Temporal+Variate | Variate | 96 | 0.130 | 0.278 | 0.062 | 0.180 | 0.145 | 0.219
192 | 0.179 | 0.330 | 0.095 | 0.229 | 0.146 | 0.220
336 | 0.209 | 0.366 | 0.127 | 0.270 | 0.145 | 0.224
720 | 0.215 | 0.372 | 0.180 | 0.330 | 0.165 | 0.246
Avg | 0.183 | 0.337 | 0.116 | 0.252 | 0.150 | 0.227
Replace | Temporal+Variate | Temporal | 96 | 0.131 | 0.279 | 0.067 | 0.186 | 0.155 | 0.234
192 | 0.180 | 0.332 | 0.100 | 0.235 | 0.152 | 0.231
336 | 0.222 | 0.375 | 0.135 | 0.280 | 0.151 | 0.233
720 | 0.234 | 0.387 | 0.185 | 0.335 | 0.174 | 0.258
Avg | 0.192 | 0.343 | 0.122 | 0.259 | 0.158 | 0.239
w/o | w/o Variate | Variate | 96 | 0.132 | 0.279 | 0.064 | 0.183 | 0.153 | 0.230
192 | 0.183 | 0.335 | 0.099 | 0.236 | 0.152 | 0.230
336 | 0.221 | 0.375 | 0.127 | 0.270 | 0.151 | 0.234
720 | 0.249 | 0.398 | 0.179 | 0.329 | 0.175 | 0.260
Avg | 0.197 | 0.347 | 0.117 | 0.255 | 0.158 | 0.239
w/o Temporal | Variate | 96 | 0.133 | 0.282 | 0.063 | 0.183 | 0.149 | 0.226
192 | 0.176 | 0.329 | 0.097 | 0.234 | 0.147 | 0.224
336 | 0.211 | 0.367 | 0.129 | 0.275 | 0.145 | 0.230
720 | 0.234 | 0.387 | 0.181 | 0.332 | 0.166 | 0.251
Avg | 0.188 | 0.341 | 0.118 | 0.256 | 0.152 | 0.233

Our original design of embedding strategy is motivated by our innovative
cognitive pattern of beneficial influential factors in forecasting tasks. By
using this form of embedding setting, we attempt to make Transformer models
learn temporal dependencies of the foretasted endogenous variable on one hand
while learning the cross-attention relations between endogenous and exogenous
variables on the other hand. We also make the model possess the cross-
attention correlation between different exogenous variables at the same time
to help the model better understand the relations between endogenous and
exogenous variables. The results of this ablation study provide a solid
factual basis for our viewpoint.

##  Appendix D Showcase

To have an intuitive concept of the forecasting process, we visually present
endogenous and exogenous variables from selected datasets BE, DE, and PJM in
Figure 11, Figure 12, and Figure 13. In each example, we take down the ground
truth values of endogenous and exogenous variables in datasets, as well as the
forecasting results of the endogenous variable. Our examples cover six
different models: TimeXer, Crossformer, iTransformer, PatchTST, TiDE, and
DLinear. All models take in a sequence of data with a length of 168 and
perform forecasting tasks on a prediction length of 24. Points of inflection
on curves are used to evaluate the forecasting quality. If the predicted value
remains within a range of 0.05 compared to the ground truth, we take this
prediction as a successful forecast and put a green circle of 0.05 radius
around it. Otherwise, we take this prediction as an out-of-range forecast and
put a red circle of 0.05 radius to mark its failure.

Figure 11: Showcases of TimeXer in forecasting with exogenous variables from
BE datasets. The two leftmost plots of the title ”Generation” and ”System
Load” are the exogenous variables in the BE dataset. TimeXer outperforms all
of its challengers by predicting all 4 injections in 24 prediction time
points. Figure 12: Showcases of TimeXer in forecasting with exogenous
variables from DE datasets. The two leftmost plots of the title ”System Load”
and ”Zonal COMED Load” are the exogenous variables in the DE dataset. TimeXer
outperforms all of its challengers by predicting all 4 injections in 24
prediction time points. Figure 13: Showcases of TimeXer in forecasting with
exogenous variables from PJM datasets. The two leftmost plots of the title
”System Load” and ”Zonal COMED Load” are the exogenous variables in the PJM
dataset. TimeXer outperforms all of its challengers by predicting all 4
injections in 24 prediction time points.

By counting the green and red circles on all injection points in Figure 11,
Figure 12, and Figure 13, it is clear that the TimeXer can forecast target
endogenous more precisely, especially at inflection points where we marked
with green circles. Other models tend to vibrate nearby or go beyond the
ground truth, indicating that TimeXer is more robust than existing models. By
learning from the endogenous variable’s temporal dependencies and relations
between endogenous and exogenous variables, TimeXer not only acquires abundant
contextual information about its own history but also obtains nutritive
relation information about correlated variables. Such structure makes TimeXer
more aware of the potential pattern of the target dataset, leading to enhanced
forecasting performance compared to known Transformer-based models.

##  Appendix E Full Results

Due to the limited space of the main text, all the experiment results are
summarized into CKA analysis, long-term forecasting results, and long-term
multivariate forecasting results. The CKA analysis is neatly presented in
Figure 9. The two forecasting results are categorized and indexed in Table 7
and Table 8.

###  E.1 Full Results of CKA Analysis

o evaluate the performance of TimeXer from the perspective of representation,
we adopt centered kernel alignment (CKA) similarity (Kornblith et al., 2019)
analysis. Due to the paper limit, we provide results from three EPF datasets
here. As shown in Fig. (reffig:cka), the representations of the first and last
layers of TimeXer have a great similarity, reaching more than 80% on all
datasets, which verifies that TimeXer is able to learn appropriate
representations for prediction.

Figure 14: Series Representation Analysis on three EPF datasets. _iTrm-All_
denotes the series representation of all variables learned by iTransformer,
_iTrm-En_ is the learned series representation of the endogenous variable.

###  E.2 Full Results of Long-term Forecasting

To evaluate the performance of our proposed TimeXer, we conduct long-term
forecasting with exogenous variables on acknowledged real-world multivariate
datasets. The look-back length is set to 96, and the prediction length varies
from $\\{96,192,336,720\\}$. The results are listed in Table 7.

Table 7: Full results of the long-term forecasting with exogenous variables
task.

Models |  TimeXer |  iTrans. |  RLinear |  PatchTST |  Cross. |  TiDE |  TimesNet |  DLinear |  SCINet |  Stationary |  Auto.
---|---|---|---|---|---|---|---|---|---|---|---
(Ours) |  (2023) |  (2023) |  (2022) |  (2022) |  (2023) |  (2023a) |  (2023) |  (2022a) |  (2022b) |  (2021)
Metric |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE
ECL |  96 |  0.282 |  0.380 |  0.299 |  0.403 |  0.433 |  0.480 |  0.339 |  0.412 |  0.265 |  0.364 |  0.405 |  0.459 |  0.342 |  0.437 |  0.387 |  0.451 |  0.390 |  0.462 |  0.298 |  0.407 |  0.432 |  0.502
192 |  0.319 |  0.399 |  0.321 |  0.413 |  0.407 |  0.461 |  0.361 |  0.425 |  0.313 |  0.390 |  0.383 |  0.442 |  0.384 |  0.461 |  0.365 |  0.436 |  0.375 |  0.456 |  0.340 |  0.433 |  0.492 |  0.492
336 |  0.362 |  0.429 |  0.379 |  0.446 |  0.440 |  0.481 |  0.393 |  0.440 |  0.380 |  0.431 |  0.418 |  0.464 |  0.439 |  0.493 |  0.391 |  0.453 |  0.468 |  0.519 |  0.405 |  0.471 |  0.508 |  0.548
720 |  0.380 |  0.450 |  0.461 |  0.504 |  0.495 |  0.523 |  0.482 |  0.507 |  0.418 |  0.463 |  0.471 |  0.507 |  0.473 |  0.514 |  0.428 |  0.487 |  0.477 |  0.524 |  0.444 |  0.489 |  0.547 |  0.569
AVG |  0.336 |  0.414 |  0.365 |  0.442 |  0.444 |  0.486 |  0.394 |  0.446 |  0.344 |  0.412 |  0.419 |  0.468 |  0.410 |  0.476 |  0.393 |  0.457 |  0.427 |  0.490 |  0.372 |  0.450 |  0.495 |  0.528
Weather |  96 |  0.001 |  0.026 |  0.001 |  0.026 |  0.001 |  0.025 |  0.001 |  0.027 |  0.004 |  0.048 |  0.001 |  0.025 |  0.002 |  0.029 |  0.006 |  0.062 |  0.006 |  0.064 |  0.001 |  0.028 |  0.007 |  0.066
192 |  0.002 |  0.030 |  0.002 |  0.029 |  0.001 |  0.028 |  0.002 |  0.030 |  0.005 |  0.053 |  0.001 |  0.028 |  0.002 |  0.031 |  0.006 |  0.066 |  0.007 |  0.071 |  0.002 |  0.030 |  0.007 |  0.061
336 |  0.002 |  0.031 |  0.002 |  0.031 |  0.002 |  0.029 |  0.002 |  0.032 |  0.004 |  0.051 |  0.002 |  0.029 |  0.002 |  0.031 |  0.006 |  0.068 |  0.008 |  0.072 |  0.002 |  0.030 |  0.007 |  0.062
720 |  0.002 |  0.035 |  0.002 |  0.036 |  0.002 |  0.033 |  0.002 |  0.036 |  0.007 |  0.067 |  0.002 |  0.033 |  0.381 |  0.368 |  0.007 |  0.070 |  0.008 |  0.074 |  0.002 |  0.033 |  0.005 |  0.053
AVG |  0.002 |  0.031 |  0.002 |  0.031 |  0.002 |  0.029 |  0.002 |  0.031 |  0.005 |  0.055 |  0.002 |  0.029 |  0.097 |  0.115 |  0.006 |  0.066 |  0.007 |  0.071 |  0.002 |  0.031 |  0.006 |  0.060
ETTh1 |  96 |  0.055 |  0.180 |  0.057 |  0.183 |  0.059 |  0.185 |  0.055 |  0.178 |  0.133 |  0.297 |  0.059 |  0.184 |  0.059 |  0.188 |  0.065 |  0.188 |  0.343 |  0.502 |  0.072 |  0.204 |  0.119 |  0.263
192 |  0.073 |  0.206 |  0.074 |  0.209 |  0.078 |  0.214 |  0.072 |  0.206 |  0.232 |  0.409 |  0.078 |  0.214 |  0.080 |  0.217 |  0.088 |  0.222 |  0.393 |  0.533 |  0.095 |  0.238 |  0.132 |  0.286
336 |  0.085 |  0.229 |  0.084 |  0.223 |  0.093 |  0.240 |  0.087 |  0.231 |  0.244 |  0.423 |  0.093 |  0.240 |  0.083 |  0.224 |  0.110 |  0.257 |  0.406 |  0.537 |  0.110 |  0.261 |  0.126 |  0.278
720 |  0.082 |  0.227 |  0.084 |  0.229 |  0.106 |  0.256 |  0.098 |  0.247 |  0.530 |  0.660 |  0.104 |  0.255 |  0.083 |  0.231 |  0.202 |  0.371 |  0.604 |  0.690 |  0.164 |  0.321 |  0.143 |  0.299
AVG |  0.074 |  0.210 |  0.075 |  0.211 |  0.084 |  0.224 |  0.078 |  0.215 |  0.285 |  0.447 |  0.083 |  0.223 |  0.076 |  0.215 |  0.116 |  0.259 |  0.437 |  0.565 |  0.110 |  0.256 |  0.130 |  0.282
ETTh2 |  96 |  0.130 |  0.278 |  0.137 |  0.287 |  0.136 |  0.286 |  0.136 |  0.285 |  0.261 |  0.413 |  0.136 |  0.285 |  0.159 |  0.310 |  0.135 |  0.282 |  0.763 |  0.767 |  0.186 |  0.333 |  0.184 |  0.335
192 |  0.179 |  0.330 |  0.187 |  0.341 |  0.187 |  0.339 |  0.185 |  0.337 |  1.240 |  1.028 |  0.187 |  0.339 |  0.196 |  0.351 |  0.188 |  0.335 |  1.080 |  0.929 |  0.226 |  0.375 |  0.214 |  0.364
336 |  0.209 |  0.366 |  0.221 |  0.376 |  0.231 |  0.384 |  0.217 |  0.373 |  0.974 |  0.874 |  0.231 |  0.384 |  0.232 |  0.385 |  0.238 |  0.385 |  1.159 |  0.960 |  0.302 |  0.443 |  0.269 |  0.405
720 |  0.215 |  0.372 |  0.253 |  0.403 |  0.267 |  0.417 |  0.229 |  0.384 |  1.633 |  1.177 |  0.267 |  0.417 |  0.254 |  0.403 |  0.336 |  0.475 |  1.615 |  1.163 |  0.335 |  0.471 |  0.303 |  0.440
AVG |  0.183 |  0.337 |  0.199 |  0.352 |  0.205 |  0.356 |  0.192 |  0.345 |  1.027 |  0.873 |  0.205 |  0.356 |  0.210 |  0.362 |  0.224 |  0.369 |  1.155 |  0.955 |  0.262 |  0.405 |  0.242 |  0.386
ETTm1 |  96 |  0.027 |  0.123 |  0.029 |  0.128 |  0.030 |  0.129 |  0.029 |  0.126 |  0.171 |  0.355 |  0.030 |  0.129 |  0.029 |  0.128 |  0.034 |  0.135 |  0.050 |  0.173 |  0.034 |  0.138 |  0.097 |  0.251
192 |  0.042 |  0.156 |  0.045 |  0.163 |  0.044 |  0.160 |  0.045 |  0.160 |  0.293 |  0.474 |  0.044 |  0.160 |  0.044 |  0.160 |  0.055 |  0.173 |  0.083 |  0.227 |  0.060 |  0.182 |  0.062 |  0.197
336 |  0.056 |  0.181 |  0.060 |  0.190 |  0.057 |  0.184 |  0.058 |  0.184 |  0.330 |  0.503 |  0.057 |  0.184 |  0.061 |  0.190 |  0.078 |  0.210 |  0.110 |  0.261 |  0.087 |  0.222 |  0.083 |  0.230
720 |  0.078 |  0.213 |  0.079 |  0.218 |  0.080 |  0.217 |  0.082 |  0.221 |  0.852 |  0.861 |  0.080 |  0.217 |  0.083 |  0.223 |  0.098 |  0.234 |  0.152 |  0.305 |  0.127 |  0.275 |  0.100 |  0.245
AVG |  0.051 |  0.169 |  0.053 |  0.175 |  0.053 |  0.173 |  0.053 |  0.173 |  0.411 |  0.548 |  0.053 |  0.173 |  0.054 |  0.175 |  0.066 |  0.188 |  0.098 |  0.241 |  0.077 |  0.204 |  0.085 |  0.230
ETTm2 |  96 |  0.062 |  0.180 |  0.071 |  0.194 |  0.074 |  0.199 |  0.068 |  0.188 |  0.149 |  0.309 |  0.073 |  0.199 |  0.073 |  0.200 |  0.072 |  0.195 |  0.253 |  0.427 |  0.098 |  0.229 |  0.133 |  0.282
192 |  0.095 |  0.229 |  0.108 |  0.247 |  0.104 |  0.241 |  0.100 |  0.236 |  0.686 |  0.740 |  0.104 |  0.241 |  0.106 |  0.247 |  0.105 |  0.240 |  0.592 |  0.677 |  0.161 |  0.302 |  0.143 |  0.294
336 |  0.127 |  0.270 |  0.140 |  0.288 |  0.131 |  0.276 |  0.128 |  0.271 |  0.546 |  0.602 |  0.131 |  0.276 |  0.150 |  0.296 |  0.136 |  0.280 |  0.777 |  0.790 |  0.243 |  0.362 |  0.156 |  0.308
720 |  0.180 |  0.330 |  0.188 |  0.340 |  0.180 |  0.329 |  0.185 |  0.335 |  2.524 |  1.424 |  0.180 |  0.329 |  0.186 |  0.338 |  0.191 |  0.335 |  1.117 |  0.960 |  0.326 |  0.441 |  0.184 |  0.333
AVG |  0.116 |  0.252 |  0.127 |  0.267 |  0.122 |  0.261 |  0.120 |  0.258 |  0.976 |  0.769 |  0.122 |  0.261 |  0.129 |  0.271 |  0.126 |  0.263 |  0.685 |  0.713 |  0.207 |  0.333 |  0.154 |  0.305
Traffic |  96 |  0.145 |  0.219 |  0.156 |  0.236 |  0.350 |  0.431 |  0.176 |  0.253 |  0.154 |  0.230 |  0.350 |  0.430 |  0.154 |  0.249 |  0.350 |  0.430 |  0.371 |  0.448 |  0.214 |  0.323 |  0.290 |  0.290
192 |  0.146 |  0.220 |  0.156 |  0.237 |  0.314 |  0.404 |  0.162 |  0.243 |  0.180 |  0.256 |  0.316 |  0.405 |  0.164 |  0.255 |  0.302 |  0.387 |  0.450 |  0.503 |  0.195 |  0.307 |  0.291 |  0.291
336 |  0.145 |  0.224 |  0.154 |  0.243 |  0.305 |  0.399 |  0.164 |  0.248 |  - |  - |  0.305 |  0.398 |  0.167 |  0.259 |  0.298 |  0.384 |  0.447 |  0.501 |  0.198 |  0.309 |  0.322 |  0.416
720 |  0.165 |  0.246 |  0.177 |  0.268 |  0.328 |  0.415 |  0.189 |  0.267 |  - |  - |  0.305 |  0.398 |  0.197 |  0.292 |  0.340 |  0.416 |  0.521 |  0.548 |  0.835 |  0.507 |  0.307 |  0.414
AVG |  0.150 |  0.227 |  0.161 |  0.246 |  0.324 |  0.412 |  0.173 |  0.253 |  - |  - |  0.315 |  0.403 |  0.171 |  0.264 |  0.323 |  0.404 |  0.447 |  0.500 |  0.361 |  0.361 |  0.302 |  0.353
$1^{\text{st}}$ Count |  26 |  24 |  0 |  1 |  6 |  6 |  2 |  2 |  2 |  3 |  6 |  6 |  1 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0

###  E.3 Full Results of Long-term Multivariate Forecasting

To evaluate the generality of our proposed TimeXer, we conduct long-term
multivariate forecasting on existing real-world multivariate benchmarks. The
look-back length is set to 96, and the prediction length varies from
$\\{96,192,336,720\\}$. The results are listed in Table 8.

Table 8: Full results of the long-term multivariate forecasting task.

Models |  TimeXer |  iTrans. |  RLinear |  PatchTST |  Cross. |  TiDE |  TimesNet |  DLinear |  SCINet |  Stationary |  Auto.
---|---|---|---|---|---|---|---|---|---|---|---
(Ours) |  (2023) |  (2023) |  (2022) |  (2022) |  (2023) |  (2023a) |  (2023) |  (2022a) |  (2022b) |  (2021)
Metric |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE
ECL |  96 |  0.151 |  0.247 |  0.148 |  0.240 |  0.201 |  0.281 |  0.195 |  0.285 |  0.219 |  0.314 |  0.237 |  0.329 |  0.168 |  0.272 |  0.197 |  0.282 |  0.247 |  0.345 |  0.169 |  0.273 |  0.201 |  0.317
192 |  0.165 |  0.261 |  0.162 |  0.253 |  0.201 |  0.283 |  0.199 |  0.289 |  0.231 |  0.322 |  0.236 |  0.330 |  0.184 |  0.289 |  0.196 |  0.285 |  0.257 |  0.355 |  0.182 |  0.286 |  0.222 |  0.334
336 |  0.183 |  0.280 |  0.178 |  0.269 |  0.215 |  0.298 |  0.215 |  0.305 |  0.246 |  0.337 |  0.249 |  0.344 |  0.198 |  0.300 |  0.209 |  0.301 |  0.269 |  0.369 |  0.200 |  0.304 |  0.231 |  0.338
720 |  0.220 |  0.309 |  0.225 |  0.317 |  0.257 |  0.331 |  0.256 |  0.337 |  0.280 |  0.363 |  0.284 |  0.373 |  0.220 |  0.320 |  0.245 |  0.333 |  0.299 |  0.390 |  0.222 |  0.321 |  0.254 |  0.361
Avg |  0.180 |  0.274 |  0.178 |  0.270 |  0.219 |  0.298 |  0.216 |  0.304 |  0.244 |  0.334 |  0.251 |  0.344 |  0.192 |  0.295 |  0.212 |  0.300 |  0.268 |  0.365 |  0.193 |  0.296 |  0.227 |  0.338
Weather |  96 |  0.168 |  0.209 |  0.174 |  0.214 |  0.192 |  0.232 |  0.177 |  0.218 |  0.158 |  0.230 |  0.202 |  0.261 |  0.172 |  0.220 |  0.196 |  0.255 |  0.221 |  0.306 |  0.173 |  0.223 |  0.266 |  0.336
192 |  0.220 |  0.254 |  0.221 |  0.254 |  0.240 |  0.271 |  0.225 |  0.259 |  0.206 |  0.277 |  0.242 |  0.298 |  0.219 |  0.261 |  0.237 |  0.296 |  0.261 |  0.340 |  0.245 |  0.285 |  0.307 |  0.367
336 |  0.276 |  0.296 |  0.278 |  0.296 |  0.292 |  0.307 |  0.278 |  0.297 |  0.272 |  0.335 |  0.287 |  0.335 |  0.280 |  0.306 |  0.283 |  0.335 |  0.309 |  0.378 |  0.321 |  0.338 |  0.359 |  0.395
720 |  0.353 |  0.347 |  0.358 |  0.349 |  0.364 |  0.353 |  0.354 |  0.348 |  0.398 |  0.418 |  0.351 |  0.386 |  0.365 |  0.359 |  0.345 |  0.381 |  0.377 |  0.427 |  0.414 |  0.410 |  0.419 |  0.428
Avg |  0.254 |  0.276 |  0.258 |  0.279 |  0.272 |  0.291 |  0.259 |  0.281 |  0.259 |  0.315 |  0.271 |  0.320 |  0.259 |  0.287 |  0.265 |  0.317 |  0.292 |  0.363 |  0.288 |  0.314 |  0.338 |  0.382
ETTh1 |  96 |  0.377 |  0.397 |  0.386 |  0.405 |  0.386 |  0.395 |  0.414 |  0.419 |  0.423 |  0.448 |  0.479 |  0.464 |  0.384 |  0.402 |  0.386 |  0.400 |  0.654 |  0.599 |  0.513 |  0.491 |  0.449 |  0.459
192 |  0.425 |  0.426 |  0.441 |  0.436 |  0.437 |  0.424 |  0.460 |  0.445 |  0.471 |  0.474 |  0.525 |  0.492 |  0.436 |  0.429 |  0.437 |  0.432 |  0.719 |  0.631 |  0.534 |  0.504 |  0.500 |  0.482
336 |  0.457 |  0.441 |  0.487 |  0.458 |  0.479 |  0.446 |  0.501 |  0.466 |  0.570 |  0.546 |  0.565 |  0.515 |  0.491 |  0.469 |  0.481 |  0.459 |  0.778 |  0.659 |  0.588 |  0.535 |  0.521 |  0.496
720 |  0.464 |  0.463 |  0.503 |  0.491 |  0.481 |  0.470 |  0.500 |  0.488 |  0.653 |  0.621 |  0.594 |  0.558 |  0.521 |  0.500 |  0.519 |  0.516 |  0.836 |  0.699 |  0.643 |  0.616 |  0.514 |  0.512
Avg |  0.431 |  0.432 |  0.454 |  0.447 |  0.446 |  0.434 |  0.469 |  0.454 |  0.529 |  0.522 |  0.541 |  0.507 |  0.458 |  0.450 |  0.456 |  0.452 |  0.747 |  0.647 |  0.570 |  0.537 |  0.496 |  0.487
ETTh2 |  96 |  0.289 |  0.340 |  0.297 |  0.349 |  0.288 |  0.338 |  0.302 |  0.348 |  0.745 |  0.584 |  0.400 |  0.440 |  0.340 |  0.374 |  0.333 |  0.387 |  0.707 |  0.621 |  0.476 |  0.458 |  0.346 |  0.388
192 |  0.370 |  0.391 |  0.380 |  0.400 |  0.374 |  0.390 |  0.388 |  0.400 |  0.877 |  0.656 |  0.528 |  0.509 |  0.402 |  0.414 |  0.477 |  0.476 |  0.860 |  0.689 |  0.512 |  0.493 |  0.456 |  0.452
336 |  0.422 |  0.434 |  0.428 |  0.432 |  0.415 |  0.426 |  0.426 |  0.433 |  1.043 |  0.731 |  0.643 |  0.571 |  0.452 |  0.452 |  0.594 |  0.541 |  1.000 |  0.744 |  0.552 |  0.551 |  0.482 |  0.486
720 |  0.429 |  0.445 |  0.427 |  0.445 |  0.420 |  0.440 |  0.431 |  0.446 |  1.104 |  0.763 |  0.874 |  0.679 |  0.462 |  0.468 |  0.831 |  0.657 |  1.249 |  0.838 |  0.562 |  0.560 |  0.515 |  0.511
Avg |  0.378 |  0.403 |  0.383 |  0.407 |  0.374 |  0.398 |  0.387 |  0.407 |  0.942 |  0.684 |  0.611 |  0.550 |  0.414 |  0.427 |  0.559 |  0.515 |  0.954 |  0.723 |  0.526 |  0.516 |  0.450 |  0.459
ETTm1 |  96 |  0.309 |  0.352 |  0.334 |  0.368 |  0.355 |  0.376 |  0.329 |  0.367 |  0.404 |  0.426 |  0.364 |  0.387 |  0.338 |  0.375 |  0.345 |  0.372 |  0.418 |  0.438 |  0.386 |  0.398 |  0.505 |  0.475
192 |  0.355 |  0.378 |  0.377 |  0.391 |  0.391 |  0.392 |  0.367 |  0.385 |  0.450 |  0.451 |  0.398 |  0.404 |  0.374 |  0.387 |  0.380 |  0.389 |  0.426 |  0.441 |  0.459 |  0.444 |  0.553 |  0.496
336 |  0.387 |  0.399 |  0.426 |  0.420 |  0.424 |  0.415 |  0.399 |  0.410 |  0.532 |  0.515 |  0.428 |  0.425 |  0.410 |  0.411 |  0.413 |  0.413 |  0.445 |  0.459 |  0.495 |  0.464 |  0.621 |  0.537
720 |  0.448 |  0.435 |  0.491 |  0.459 |  0.487 |  0.450 |  0.454 |  0.439 |  0.666 |  0.589 |  0.487 |  0.461 |  0.478 |  0.450 |  0.474 |  0.453 |  0.595 |  0.550 |  0.585 |  0.516 |  0.671 |  0.561
Avg |  0.375 |  0.391 |  0.407 |  0.410 |  0.414 |  0.407 |  0.387 |  0.400 |  0.513 |  0.496 |  0.419 |  0.419 |  0.400 |  0.406 |  0.403 |  0.407 |  0.485 |  0.481 |  0.481 |  0.456 |  0.588 |  0.517
ETTm2 |  96 |  0.171 |  0.255 |  0.180 |  0.264 |  0.182 |  0.265 |  0.175 |  0.259 |  0.287 |  0.366 |  0.207 |  0.305 |  0.187 |  0.267 |  0.193 |  0.292 |  0.286 |  0.377 |  0.192 |  0.274 |  0.255 |  0.339
192 |  0.238 |  0.300 |  0.250 |  0.309 |  0.246 |  0.304 |  0.241 |  0.302 |  0.414 |  0.492 |  0.290 |  0.364 |  0.249 |  0.309 |  0.284 |  0.362 |  0.399 |  0.445 |  0.280 |  0.339 |  0.281 |  0.340
336 |  0.301 |  0.340 |  0.311 |  0.348 |  0.307 |  0.342 |  0.305 |  0.343 |  0.597 |  0.542 |  0.377 |  0.422 |  0.321 |  0.351 |  0.369 |  0.427 |  0.637 |  0.591 |  0.334 |  0.361 |  0.339 |  0.372
720 |  0.401 |  0.397 |  0.412 |  0.407 |  0.407 |  0.398 |  0.402 |  0.400 |  1.730 |  1.042 |  0.558 |  0.524 |  0.408 |  0.403 |  0.554 |  0.522 |  0.960 |  0.735 |  0.417 |  0.413 |  0.433 |  0.432
Avg |  0.278 |  0.323 |  0.288 |  0.332 |  0.286 |  0.327 |  0.281 |  0.326 |  0.757 |  0.610 |  0.358 |  0.404 |  0.291 |  0.333 |  0.350 |  0.401 |  0.571 |  0.537 |  0.306 |  0.347 |  0.327 |  0.371
Traffic |  96 |  0.416 |  0.280 |  0.395 |  0.268 |  0.649 |  0.389 |  0.544 |  0.359 |  0.522 |  0.290 |  0.805 |  0.493 |  0.593 |  0.321 |  0.650 |  0.396 |  0.788 |  0.499 |  0.612 |  0.338 |  0.613 |  0.388
192 |  0.435 |  0.288 |  0.417 |  0.276 |  0.601 |  0.366 |  0.540 |  0.354 |  0.530 |  0.293 |  0.756 |  0.474 |  0.617 |  0.336 |  0.598 |  0.370 |  0.789 |  0.505 |  0.613 |  0.340 |  0.616 |  0.382
336 |  0.451 |  0.296 |  0.433 |  0.283 |  0.609 |  0.369 |  0.551 |  0.358 |  0.558 |  0.305 |  0.762 |  0.477 |  0.629 |  0.336 |  0.605 |  0.373 |  0.797 |  0.508 |  0.618 |  0.328 |  0.622 |  0.337
720 |  0.484 |  0.314 |  0.467 |  0.302 |  0.647 |  0.387 |  0.586 |  0.375 |  0.589 |  0.328 |  0.719 |  0.449 |  0.640 |  0.350 |  0.645 |  0.394 |  0.841 |  0.523 |  0.653 |  0.355 |  0.660 |  0.408
Avg |  0.447 |  0.295 |  0.428 |  0.282 |  0.626 |  0.378 |  0.555 |  0.362 |  0.550 |  0.304 |  0.760 |  0.473 |  0.620 |  0.336 |  0.625 |  0.383 |  0.804 |  0.509 |  0.624 |  0.340 |  0.628 |  0.379
$1^{\text{st}}$ Count |  18 |  20 |  9 |  11 |  4 |  6 |  0 |  0 |  3 |  0 |  0 |  0 |  1 |  0 |  1 |  0 |  0 |  0 |  0 |  0 |  0 |  0