# Chimera: Effectively Modeling Multivariate Time Series with 2-Dimensional
State Space Models

Ali Behrouz  Michele Santacatterina  Ramin Zabih

###### Abstract

Modeling multivariate time series is a well-established problem with a wide
range of applications from healthcare to financial markets. It, however, is
extremely challenging as it requires methods to (1) have high expressive power
of representing complicated dependencies along the time axis to capture both
long-term progression and seasonal patterns, (2) capture the inter-variate
dependencies when it is informative, (3) dynamically model the dependencies of
variate and time dimensions, and (4) have efficient training and inference for
very long sequences. Traditional State Space Models (SSMs) are classical
approaches for univariate time series modeling due to their simplicity and
expressive power to represent _linear_ dependencies. They, however, have
fundamentally limited expressive power to capture non-linear dependencies, are
slow in practice, and fail to model the inter-variate information flow.
Despite recent attempts to improve the expressive power of SSMs by using deep
structured SSMs, the existing methods are either limited to univariate time
series, fail to model complex patterns (e.g., seasonal patterns), fail to
dynamically model the dependencies of variate and time dimensions, and/or are
input-independent. We present Chimera, an expressive variation of the
2-dimensional SSMs with careful design of parameters to maintain high
expressive power while keeping the training complexity linear. Using two SSM
heads with different discretization processes and input-dependent parameters,
Chimera is provably able to learn long-term progression, seasonal patterns,
and desirable dynamic autoregressive processes. To improve the efficiency of
complex 2D recurrence, we present a fast training using a new 2-dimensional
parallel selective scan. We further present and discuss 2-dimensional Mamba
and Mamba-2 as the spacial cases of our 2D SSM. Our experimental evaluation
shows the superior performance of Chimera on extensive and diverse benchmarks,
including ECG and speech time series classification, long-term and short-term
time series forecasting, and time series anomaly detection.

##  1 Introduction

Modeling time series is a well-established problem with a wide range of
applications from healthcare [6, 36] to financial markets [21, 60] and energy
management [87]. The complex nature of time series data, its diverse domains
of applicability, and its broad range of tasks (e.g., classification [6, 72],
imputation [77, 51], anomaly detection [6, 67], and forecasting [87]),
however, raise fundamental challenges to design effective and generalizable
models: (1) The higher-order, seasonal, and long-term patterns in time series
require an effective model to be able to expressively capture complex and
autoregressive dependencies; (2) In the presence of multiple variates of time
series, an effective model need to capture the complex dynamics of the
dependencies between time and variate axes. More specifically, most existing
multivariate models seem to suffer from overfitting especially when the target
time series is not correlated with other covariates [82]. Accordingly, an
effective model needs to adaptively learn to select (resp. filter) informative
(resp. irrelevant) variates; (3) The diverse set of domains and tasks requires
effective models to be free of manual pre-processing and domain knowledge and
instead adaptively learn them; and (4) Due to the processing of very long
sequences, effective methods need efficient training and inference.

Classical methods (e.g., State Space Models [30, 2], ARIMA [5], SARIMA [9],
Exponential Smoothing (ETS) [75]) require manual data preprocessing and model
selection, and often are not able to capture complex _non-linear_ dynamics.
The raise of deep learning methods and more specifically Transformers [71] has
led to significant research efforts to address the limitation of classical
methods and develop effective deep models [44, 14, 76, 79, 78, 46, 89, 39, 86,
47]. Unfortunately, most existing deep models struggle to achieve all the
above four criteria. The main body of research in this direction has focused
on designing attention modules that use the special traits of time series [79,
76]. However, the inherent permutation equivariance of attentions contradicts
the causal nature of time series and often results in suboptimal performance
compared to simple linear methods [82]. Moreover, they often either overlook
difference of seasonal and long-term trend or use non-learnable methods to
handle them [76].

A considerable subset of deep models overlook the importance of modeling the
dependencies of variates [82, 84, 56]. These dependencies, however, are not
always useful; specifically when the target time series is not correlated with
other covariates [12]. Despite several studies exploring the importance of
learning cross variate dependencies [86, 47, 12], there has been no universal
standard and the conclusion has been different depending on the domain and
benchmarks. Accordingly, we argue that an effective model need to _adaptively_
learn to capture the dependencies of variates in a data-dependent manner. In
this direction, recently, [47] argue that attention mechanisms are more
effective when they are used across variates, showing the importance of
modeling complex non-linear dependencies across the variate axis in a data-
dependent manner. However, the quadratic complexity of Transformers challenges
the model on multivariate time series with a large number of variates (e.g.,
brain activity signals [6] or traffic forecasting [87]), limiting the
efficient training and inference (see subsection 4.1, and subsection 4.2).

Figure 1: The Overview of Contributions and Architecture of Chimera. We
present a 2-dimensional SSM with careful and expressive parameterization. It
uses different learnable discretization processes to learn seasonal and long-
term progression patterns, and leverages a parallelizable and fast training
process by re-formulating the 2D input dependent recurrence as a 2D prefix sum
problem.

The objective of this study is to develop a provably expressive model for
multivariate time series that not only can model the dynamics of the
depenendencies along both time and variates, but it also takes advantage of
fast training and inference. To this end, we present a Chimera, a three-headed
two-dimensional State Space Model (SSM) that is based on linear layers along
(i) time, (ii) variates, (iii) time$\rightarrow$variate, and (iv)
variate$\rightarrow$time. Chimera has a careful parameterization based on the
pair of companion and diagonal matrices (see Figure 1), which is provably
expressive to recover both classical methods [75, 5, 9], linear attentions,
and recent SSM-based models [8, 55]. It further uses an adaptive module based
on a 2D SSM with an especially designed discretization process to capture
seasonal patterns. While our theoretical results and design of Chimera
guarantee the first three criteria of an effective model, due to its 2D
recurrence, the naive implementation of Chimera results in slow training. To
address this issue, we reformulate its 2D recurrence as the prefix sum problem
with a 2-dimensional associative operators. This new formulation can be done
in parallel and has hardware-friendly implementation, resulting in much faster
training and inference.

We discuss new variants of our 2D SSM in Section 3.2 by limiting its
transition matrices. The resulted models can be seen as the generalization of
Mamba [23] and Mamba-2 [16] to 2-dimensional data. While the main focus of
this paper is on time series data, these presented models due to their 2D
inductive bias are potentially suitable for other high dimensional data
modalities such as images, videos, and multi-channel audio.

In our experimental evaluation, we explore the performance of Chimera in a
wide range of tasks: ECG and audio speech time series classification, long-
and short-term time series forecasting, and anomaly detection tasks. We find
that Chimera achieve superior or on par performance with state-of-the-art
methods, while having faster training and less memory consumption. We perform
a case study on the human brain activity signals [6] to show (1) the
effectiveness of Chimera and (2) evaluate the importance of modeling the
dynamics of the variates dependencies.

##  2 Preliminaries

Notations. In this paper we mainly focus on classification and forecasting
tasks. Note that anomaly detection can be seen as a binary classification
task, where $0$ means “normall” and $1$ means “anomaly”. We let
$\mathbf{X}=\\{\mathbf{x}_{1},\dots,\mathbf{x}_{N}\\}\in\mathbb{R}^{N\times
T}$ be the input sequences, where $N$ is the number of variates and $T$ is the
time steps. We use $\mathbf{x}_{v,t}$ to refer to the value of the series $v$
at time $t$. In classification (anomaly detection) tasks, we aim to classify
input sequences and for forecasting tasks, given an input sequence
$\mathbf{x}_{i}$, we aim to predict
$\tilde{\mathbf{x}}_{i}\in\mathbb{R}^{1\times H}$, i.e., the next $H$ time
steps for variate $\mathbf{x}_{i}$, where $H$ is called horizon. In 2D SSMs
formulation, for a 2-dimensional vector $x\in\mathbb{C}^{1}$, we use $x^{(1)}$
and $x^{(2)}$ to refer to its real and imaginary components, respectively.

Multi-Dimensional State Space Models. We build our approach on the continuous
State Space Model (SSM) but later we make each component of Chimera discrete
by a designed discretization process. For additional discussion on 1D SSMs see
Appendix A. Given parameters
$\mathbf{A}_{\tau_{1}}\in\mathbb{R}^{N^{(\tau_{1})}\times N^{(\tau_{1})}}$,
$\mathbf{B}_{\tau_{2}}\in\mathbb{C}^{N^{(\tau_{2})}\times 1}$, and
$\mathbf{C}\in\mathbb{C}^{N_{1}\times N_{2}}$ for $\tau_{1}\in\\{1,...,4\\}$
and $\tau_{2}\in\\{1,2\\}$, the general form of the time-invariant 2D SSM is
the map $\mathbf{x}\in\mathbb{C}^{1}\mapsto\mathbf{y}\in\mathbb{C}^{1}$
defined by the linear Partial Differential Equation (PDE) with initial
condition $h(0,0)=0$:

$\displaystyle\frac{\partial}{\partial
t^{(1)}}h\left(t^{(1)},t^{(2)}\right)=\left(\mathbf{A}_{1}h^{(1)}\left(t^{(1)},t^{(2)}\right),\mathbf{A}_{2}h^{(2)}\left(t^{(1)},t^{(2)}\right)\right)+\mathbf{B}_{1}\mathbf{x}\left(t^{(1)},t^{(2)}\right),$
(1) $\displaystyle\frac{\partial}{\partial
t^{(2)}}h\left(t^{(1)},t^{(2)}\right)=\left(\mathbf{A}_{3}h^{(1)}\left(t^{(1)},t^{(2)}\right),\mathbf{A}_{4}h^{(2)}\left(t^{(1)},t^{(2)}\right)\right)+\mathbf{B}_{2}\mathbf{x}\left(t^{(1)},t^{(2)}\right),$
(2)
$\displaystyle\mathbf{y}\left(t^{(1)},t^{(2)}\right)=\langle\mathbf{C},\mathbf{x}\left(t^{(1)},t^{(2)}\right)\rangle.$
(3)

Contrary to the multi-dimensional SSMs discussed by [23, 26], in which multi-
dimension refers to the dimension of the input but with one time variable, the
above formulation uses two variables, meaning that the mapping is from a 2D
grid to a 2D grid.

(Seasonal) Autoregressive Process. Autoregressive process is a basic yet
essential premise for time series modeling, which models the causal nature of
time series. Given $p\in\mathbb{N}$, $\mathbf{x}_{k}\in\mathbb{R}^{d}$, the
simple linear autoregressive relationships between $\mathbf{x}_{k}$ and its
past samples $\mathbf{x}_{k-1},\mathbf{x}_{k-2},\dots,\mathbf{x}_{k-p}$ can be
modeled as
$\mathbf{x}_{k}=\phi_{1}\mathbf{x}_{k-1}+\phi_{2}\mathbf{x}_{k-2}+\dots,\phi_{p}\mathbf{x}_{k-p},$
where $\phi_{1},\dots,\phi_{p}$ are coefficients. This is called
$\text{AR}(p)$. Similarly, in the presence of seasonal patterns, the seasonal
autoregressive process, $\text{SAR}(p,q,s)$, is:

$\displaystyle\mathbf{x}_{k}=\phi_{1}\mathbf{x}_{k-1}+\phi_{2}\mathbf{x}_{k-2}+\dots,\phi_{p}\mathbf{x}_{k-p}{\color[rgb]{0.4,0.4,0.8}\definecolor[named]{pgfstrokecolor}{rgb}{0.4,0.4,0.8}+\eta_{1}\mathbf{x}_{k-s}+\eta_{2}\mathbf{x}_{k-2s}+\dots+\eta_{q}\mathbf{x}_{k-qs}},$
(4)

where $s$ is the frequency of seasonality, and $\phi_{1},\dots,\phi_{p}$ and
$\eta_{1},\dots,\eta_{q}$ are coefficients. Note that one can simply extend
the above formulation to multivariate time series by letting coefficients to
be vectors and replace the product with element-wise product.

##  3 Chimera: A Three-headed 2-Dimensional State Space Model

In this section, we first present a mathematical model for multivariate time
series data and then based on this model, we present a neural architecture
that can satisfy all the criteria discussed in §1.

###  3.1 Motivations & Chimera Model

SSMs have been long-standing methods for modeling time series [30, 2], mainly
due to their simplicity and expressive power to represent complicated and
autoregressive dependencies. Their states, however, are the function of a
single-variable (e.g., time). Multivariate time series, on the other hand,
require capturing dependencies along both time and variate dimensions,
requiring the current state of the model to be the function of both time and
variate. Classical 2D SSMs [40, 19, 18, 32], however, struggle to achieve good
performance compared to recent advanced deep learning methods as they are :
(1) only able to capture linear dependencies, (2) discrete by design, having a
pre-determined resolution, and so cannot simply model seasonal patterns, (3)
slow in practice for large datasets, (4) their update parameters are static
and cannot capture the dynamics of dependencies. Deep learning-based methods
[87, 12, 47], on the other hand, potentially are able to address a subset of
the above limitations, while having their own drawbacks (discussed in §1). In
this section, we start with _continuous_ SSMs due to their connection to both
classical methods [30, 2] and recent breakthrough in deep learning [26, 23].
We then discuss our contributions on how to take the advantages of the best of
both worlds, addressing all the abovementioned limitations.

Discrete 2D SSM. We use 2-dimensional SSMs, introduced in Equation 1-3, to
model multivariate time series, where the first axis corresponds to the time
dimension and the second axis is the variates. Accordingly, each state is a
function of both time and variates. The first stage is to transform the
continuous form of 2D SSMs to discrete form. Given the step size $\Delta_{1}$
and $\Delta_{2}$, which represent the resolution of the input along the axes,
discrete form of the input is defined as
$\mathbf{x}_{k,\ell}=\mathbf{x}(k\Delta_{1},\ell\Delta_{2})$. Using Zero-Order
Hold (ZOH) method, we can discretize the input as (see Appendix C for
details):

$\displaystyle\begin{pmatrix}h^{(1)}_{k,\ell+1}\\\
h^{(2)}_{k+1,\ell}\end{pmatrix}=\begin{pmatrix}\bar{\mathbf{A}}_{1}&\bar{\mathbf{A}}_{2}\\\
\bar{\mathbf{A}}_{3}&\bar{\mathbf{A}}_{4}\\\
\end{pmatrix}\begin{pmatrix}h^{(1)}_{k,\ell}\\\
h^{(2)}_{k,\ell}\end{pmatrix}+\begin{pmatrix}\bar{\mathbf{B}}_{1}\\\
\bar{\mathbf{B}}_{2}\end{pmatrix}\otimes\begin{pmatrix}\bar{\mathbf{x}}_{k,\ell+1}\\\
\bar{\mathbf{x}}_{k+1,\ell}\end{pmatrix},$ (5)

where
$\bar{\mathbf{A}}_{i}=\exp\left(\Delta_{\left\lfloor\frac{i+1}{2}\right\rfloor}\mathbf{A}_{i}\right)$
for $i=1,2,3,4$,
$\bar{\mathbf{B}}_{1}=\begin{bmatrix}\mathbf{A}_{1}^{{-1}}\left(\bar{\mathbf{A}}_{1}-\mathbf{I}\right)\mathbf{B}^{(1)}_{1}\\\
\mathbf{A}_{2}^{{-1}}\left(\bar{\mathbf{A}}_{2}-\mathbf{I}\right)\mathbf{B}^{(2)}_{1}\end{bmatrix}$,
and
$\bar{\mathbf{B}}_{2}=\begin{bmatrix}\mathbf{A}_{3}^{{-1}}\left(\bar{\mathbf{A}}_{3}-\mathbf{I}\right)\mathbf{B}^{(1)}_{2}\\\
\mathbf{A}_{4}^{{-1}}\left(\bar{\mathbf{A}}_{4}-\mathbf{I}\right)\mathbf{B}^{(2)}_{2}\end{bmatrix}$.

Note that this formulation can also be viewed as the modification of the
discrete Roesser’s SSM model [40] when we add a lag of $1$ in the inputs
$\begin{pmatrix}\bar{\mathbf{x}}_{i,j}\\\
\bar{\mathbf{x}}_{i,j}\end{pmatrix}$. This modification, however, misses the
discretization step, which is an important step in our model. We later use the
discretization step to (1) empower the model to select (resp. filter) relevant
(resp. irrelevant) information, (2) adaptively adjust the resolution of the
method, capturing seasonal patterns.

From now on, we use $t$ (resp. $v$) to refer to the index along the time
(resp. variate) dimension. Therefore, for the sake of simplicity, we
reformulate Equation 5 as follows:

$\displaystyle
h^{(1)}_{v,t+1}=\bar{\mathbf{A}}_{1}h^{(1)}_{v,t}+\bar{\mathbf{A}}_{2}h^{(2)}_{v,t}+\bar{\mathbf{B}}_{1}\mathbf{x}_{v,t+1},$
(6) $\displaystyle
h^{(2)}_{v+1,t}=\bar{\mathbf{A}}_{3}h^{(1)}_{v,t}+\bar{\mathbf{A}}_{4}h^{(2)}_{v,t}+\bar{\mathbf{B}}_{2}\mathbf{x}_{v+1,t},$
(7)
$\displaystyle\mathbf{y}_{v,t}=\mathbf{C}_{1}h^{(1)}_{v,t}+\mathbf{C}_{2}h^{(2)}_{v,t},$
(8)

where
$\bar{\mathbf{A}}_{1},\bar{\mathbf{A}}_{2},\bar{\mathbf{A}}_{3},\bar{\mathbf{A}}_{4}\in\mathbb{R}^{N\times
N}$, $\bar{\mathbf{B}}_{1},\bar{\mathbf{B}}_{2}\in\mathbb{R}^{N\times 1}$, and
$\mathbf{C}_{1},\mathbf{C}_{2}\in\mathbb{R}^{1\times N}$ are parameters of the
model, $h^{(1)}_{v,t},h^{(2)}_{v,t}\in\mathbb{R}^{N\times d}$ are hidden
states, and $\mathbf{x}_{v,t}\in\mathbb{R}^{1\times d}$ is the input. In this
formulation, intuitively, $h^{(1)}_{v,t}$ is the hidden state that carries
cross-time information (each state depends on its previous time stamp but
within the same variate), where $\bar{\mathbf{A}}_{1}$ and
$\bar{\mathbf{A}}_{2}$ control the emphasis on past cross-time and cross-
variate information, respectively. Similarly, $h^{(2)}_{v,t}$ is the hidden
state that carries cross-variate information (each state depends on other
variates but with the same time stamp). Later in this section, we discuss to
modify the model to bi-directional setting along the variate dimension, to
enhance information flow along this non-causal dimension.

Interpretation of Discretization. Time series data are often sampled from an
underlying continuous process [74, 31]. In these cases, variable $\Delta_{1}$
in the discretization of the time axis can be interpreted as resolution or the
sampling rate from the underlying continuous data. However, discretization
along the variate axis, which is discrete by its nature, or when working
directly with discrete data [37] is an unintuitive process, and raise
questions about its significance. The discretization step in 1D SSMs has deep
connections to gating mechanisms of RNNs [68, 23], automatically ensures that
the model is normalized [29], and results in desirable properties such as
resolution invariance [55].

######  Proposition 3.1.

The 2D discrete SSM introduced in Equation 6-8 with parameters
$(\\{\bar{\mathbf{A}}_{i}\\},\\{\bar{\mathbf{B}}_{i}\\},\\{\bar{\mathbf{C}}_{i}\\},k\Delta_{1},\ell\Delta_{2})$
evolves at a rate $k$ (resp. $\ell$) times as fast as the 2D discrete SSM with
parameters
$(\\{\bar{\mathbf{A}}_{i}\\},\\{\bar{\mathbf{B}}_{i}\\},\\{\bar{\mathbf{C}}_{i}\\},\Delta_{1},\ell\Delta_{2})$
(resp.
$(\\{\bar{\mathbf{A}}_{i}\\},\\{\bar{\mathbf{B}}_{i}\\},\\{\bar{\mathbf{C}}_{i}\\},k\Delta_{1},\Delta_{2})$).

Accordingly, parameters $\Delta_{1}$ can be viewed as the controller of the
length of dependencies that the model captures. That is, based on the above
result, we see the discretization along the time axis as the setting of the
resolution or sampling rate: while small $\Delta_{1}$ can capture long-term
progression, larger $\Delta_{1}$ captures seasonal patterns. For now, we see
the discretization along the variate axis as a mechanism similar to gating in
RNNs [27, 23], where $\Delta_{2}$ controls the length of the model context.
Larger values of $\Delta_{2}$ means less context window, ignoring other
variates, while smaller values of $\Delta_{2}$ means more emphesis on the
dependencies of variates. Later, inspired by [23], we discuss making
$\Delta_{2}$ as the function of the input, resulting in a selection mechanism
that filters irrelevant variates.

Figure 2: Different forms of Chimera. (Top-Left) Chimera has a recurrence form
(bi-directional along the variates), which also can be computed as a global
convolution in training. (Top-Right) In forecasting, we present the
multivariate closed-loop to improve the performance for long horizons.
(Bottom) Using data-dependent parameters, Chimera training can be done as a
parallel 2D scan.

Structure of Transition Matrices. For Chimera to be expressive and able to
recover autoregressive process, hidden states $h^{(1)}_{v,t}$ should carry
information about _past_ time stamps. While making all the parameters in
$\mathbf{A}_{i}$ learnable allows the model to learn any arbitrary structure
for $\mathbf{A}_{i}$, previous studies show that this is not possible unless
the structure of transition matrices are restricted [25, 28]. To this end,
inspired by [84] that argue that companion matrices are effective to capture
the dependencies along the time dimension, we restrict $\mathbf{A}_{1}$ and
$\mathbf{A}_{2}$ matrices to have companion structure:

$\displaystyle\mathbf{A}_{i}=\begin{pmatrix}0&0&\dots&0&a^{(i)}_{1}\\\
1&0&\dots&0&a^{(i)}_{2}\\\ 0&1&\dots&0&a^{(i)}_{3}\\\
\vdots&\vdots&\ddots&\vdots&\vdots\\\
0&0&\dots&1&a^{(i)}_{N}\end{pmatrix}=\underset{\text{Shift
Matrix}}{\underbrace{\begin{pmatrix}0&0&\dots&0&0\\\ 1&0&\dots&0&0\\\
0&1&\dots&0&0\\\ \vdots&\vdots&\ddots&\vdots&\vdots\\\
0&0&\dots&1&0\end{pmatrix}}}+\underset{\text{Low-rank
Matrix}}{\underbrace{\begin{pmatrix}0&0&\dots&0&a^{(i)}_{1}\\\
0&0&\dots&0&a^{(i)}_{2}\\\ 0&0&\dots&0&a^{(i)}_{3}\\\
\vdots&\vdots&\ddots&\vdots&\vdots\\\ 0&0&\dots&0&a^{(i)}_{N}\end{pmatrix}}},$
(9)

for $i=1,2$. Note that these two matrices are responsible to fuse the
information along the time axis (see Figure 2). Not only this formulation is
shown to be effective for capturing dependencies along the time dimension [84]
(also see Theorem 3.4), but it also can help us to compute the power of
$\mathbf{A}_{1}$ and $\mathbf{A}_{2}$ faster in the convolutional form, as
discussed by [84]. Also, for $\mathbf{A}_{3}$ and $\mathbf{A}_{4}$, we observe
that even a simpler structure of diagonal matrices is effective to fuse
information along the variate dimension. Not only these simple structured
matrices make the training of the model faster, but they also are proven to be
effective [25].

Bi-Directionality. The causal nature of the 2D SSM result in limited
information flow along the variate dimension as variate are not ordered. To
overcome this challenge, inspired by the bi-directional 1D SSMs [73], we use
two different modules for forward and backward pass along the variate
dimension:

$\displaystyle
h^{(1)^{f}}_{v,t+1}=\bar{\mathbf{A}}^{f}_{1}h^{(1)}_{v,t}+\bar{\mathbf{A}}^{f}_{2}h^{(2)^{f}}_{v,t}+\bar{\mathbf{B}}^{f}_{1}\mathbf{x}_{v,t+1},$
$\displaystyle
h^{(1)^{b}}_{v,t+1}=\bar{\mathbf{A}}^{b}_{1}h^{(1)}_{v,t}+\bar{\mathbf{A}}^{b}_{2}h^{(2)}_{v,t}+\bar{\mathbf{B}}^{b}_{1}\mathbf{x}_{v,t+1},$
(10) $\displaystyle
h^{(2)^{f}}_{v+1,t}=\bar{\mathbf{A}}^{f}_{3}h^{(1)}_{v,t}+\bar{\mathbf{A}}^{f}_{4}h^{(2)^{f}}_{v,t}+\bar{\mathbf{B}}^{f}_{2}\mathbf{x}_{v+1,t},$
$\displaystyle
h^{(2)^{b}}_{v-1,t}=\bar{\mathbf{A}}^{b}_{3}h^{(1)^{b}}_{v,t}+\bar{\mathbf{A}}^{b}_{4}h^{(2)^{b}}_{v,t}+\bar{\mathbf{B}}^{b}_{2}\mathbf{x}_{v-1,t},$
(11)
$\displaystyle\mathbf{y}^{f}_{v,t}=\mathbf{C}^{f}_{1}h^{(1)^{f}}_{v,t}+\mathbf{C}^{f}_{2}h^{(2)^{f}}_{v,t},$
(12)
$\displaystyle\mathbf{y}^{b}_{v,t}=\mathbf{C}^{b}_{1}h^{(1)^{b}}_{v,t}+\mathbf{C}^{b}_{2}h^{(2)^{b}}_{v,t},$
(13)
$\displaystyle\mathbf{y}_{v,t}=\mathbf{y}^{f}_{v,t}+\mathbf{y}^{b}_{v,t},$
(14)

where
$\bar{\mathbf{A}}^{\tau}_{1},\bar{\mathbf{A}}^{\tau}_{2},\bar{\mathbf{A}}^{\tau}_{3},\bar{\mathbf{A}}^{\tau}_{4}\in\mathbb{R}^{N\times
N}$,
$\bar{\mathbf{B}}^{\tau}_{1},\bar{\mathbf{B}}^{\tau}_{2}\in\mathbb{R}^{N\times
1}$, and $\mathbf{C}^{\tau}_{1},\mathbf{C}^{\tau}_{2}\in\mathbb{R}^{1\times
N}$ are parameters of the model,
$h^{(1)^{\tau}}_{v,t},h^{(2)^{\tau}}_{v,t}\in\mathbb{R}^{N\times d}$ are
hidden states, $\mathbf{x}_{v,t}\in\mathbb{R}^{1\times d}$ is the input, and
$\tau\in\\{f,b\\}$. Figure 2 illustrates the bi-directional recurrence process
in Chimera. For the sake of simplicity, we continue with unidirectional pass,
but adapting them for bi-directional setting is simple as we use two separate
blocks, each of which for a direction.

Convolution Form. Similar to 1D SSMs [26], our _data-independent_ formulation
can be viewed as a convolution with a kernel $\mathbf{K}$. This formulation
not only results in faster training by providing the ability of parallel
processing, but it also connect Chimera with very recent studies of modern
convolution-based architecture for time series [51]. Applying the recurrent
rules in Equation 6-8, we can write the output as:

$\displaystyle\mathbf{y}_{v,t}=\sum_{1\leq\hat{v}\leq v}\sum_{1\leq\hat{t}\leq
t}\left(\mathbf{C}_{1}\mathbf{K}^{(1)}_{\hat{v},\hat{t}}+\mathbf{C}_{2}\mathbf{K}^{(2)}_{\hat{v},\hat{t}}\right)\mathbf{x}_{\hat{v},\hat{t}},$
(15)

where kernels
$\mathbf{K}^{(\tau)}_{\hat{v},\hat{t}}=\sum_{(z_{1},\dots,z_{5})\in\mathbf{P}^{(\tau)}}q_{i}\>\bar{\mathbf{A}}_{1}^{p_{1}}\bar{\mathbf{A}}_{2}^{p_{2}}\bar{\mathbf{A}}_{3}^{p_{3}}\bar{\mathbf{A}}_{4}^{p_{4}}\bar{\mathbf{B}}_{p_{5}}$,
and $\mathbf{P}^{(\tau)}$ is the partitioning of the paths from the starting
point to $(\hat{v},\hat{t})$ for $\tau\in\\{1,2\\}$. As discussed by [4], if
the power of $\bar{\mathbf{A}}_{i}$s are given and cached, calculating the
partitioning of all paths can be done very efficiently (near-linearly) as it
the generalization of pascal triangle. To calculate the power of
$\bar{\mathbf{A}}_{i}$, note that we use diagonal matrices as the structure of
$\bar{\mathbf{A}}_{3},$ and $\bar{\mathbf{A}}_{4}$, and so computing their
powers is very fast. On the other hand, for $\bar{\mathbf{A}}_{1}$ and
$\bar{\mathbf{A}}_{2}$ with companion structures, we can use sparse matrix
multiplication, which results in linear complexity in terms of the sequence
length.

Data-Dependent Parameters. As discussed earlier, parameters
$\bar{\mathbf{A}}_{1}$ and $\bar{\mathbf{A}}_{2}$ controls the emphasis on
past cross-time and cross-variate information. Similarly, parameters
$\Delta_{1}$ and $\bar{\mathbf{B}}_{1}$ controls the emphasis on the current
input and historical data. Since these parameters are data-independent, one
can interpret them as a global feature of the system. In complex systems
(e.g., human neural activity), however, the emphasis depends on the current
input, requiring these parameters to be the function of the input (see §4.1).
The input-dependency of parameters allows the model to select relevant and
filter irrelevant information for each input data, providing a similar
mechanism as transformers [23]. Additionally, as we argue earlier, depending
on the data, the model needs to adaptively learn if mixing information along
the variates is useful. Making parameters input-dependent further overcomes
this challenge and lets our model to mix relevant and filter irrelevant
variates for the modeling of a variate of interest. One of our main technical
contributions is to let $\bar{\mathbf{B}}_{i}$, $\bar{\mathbf{C}}_{i}$, and
$\Delta_{i}$ for $i\in\\{1,2\\}$ be the function of the input
$\mathbf{x}_{v,t}$. This input-dependent 2D SSM, unfortunately, does not have
the convolution form, limiting the scalability and efficiency of the training.
We overcome this challenge by computing the model recurrently with a new 2D
scan.

2D Selective Scan. Inspired by the scanning in 1D SSMs [66, 23], we present an
algorithm to decrease the sequential steps that are required to calculate
hidden states. Given $p,q$, each of which with 6 elements, we first define
operation $\divideontimes$ as: ($\odot$ is matrix-matrix and $\otimes$ is
matrix-vector multiplication)

$\displaystyle p\divideontimes q=\begin{pmatrix}p_{1}&p_{2}&p_{3}\\\
p_{4}&p_{5}&p_{6}\end{pmatrix}\divideontimes\begin{pmatrix}q_{1}&q_{2}&q_{3}\\\
q_{4}&q_{5}&q_{6}\end{pmatrix}=\begin{pmatrix}q_{1}\odot p_{1}&q_{2}\odot
p_{2}&q_{1}\otimes p_{3}+q_{2}\otimes p_{6}+q_{3}\\\ q_{4}\odot
p_{4}&q_{5}\odot p_{5}&q_{4}\otimes p_{3}+q_{5}\otimes
p_{6}+q_{6}\end{pmatrix}$

The proofs of the next two theorems are in Appendix E.

######  Theorem 3.2.

Operator $\divideontimes$ is associative: Given $p,q,$ and $r$, we have:
$\left(p\divideontimes q\right)\divideontimes
r=p\divideontimes\left(q\divideontimes r\right)$.

######  Theorem 3.3.

2D SSM recurrence can be done in parallel using parallel prefix sum algorithms
with associative operator $\divideontimes$.

###  3.2 New Variants of 2D SSM: 2D Mamba and 2D Mamba-2

Figure 2 (Top-Left) shows the recurrence form of our 2D SSM. Each small square
is a state of the system, i.e., the state of a variate at a certain time
stamp. 2D SSM considers two hidden states for each state (represented by two
colors: light red and blue), encoding the information along the time (red) and
variate (blue), respectively. Furthermore, each arrow represents a transition
matrix $\mathbf{A}_{i}$ that decides to how information need to be fused. In
this section, we discuss different variants of our 2D SSM by limiting its
parameters.

2D Mamba. We let $\mathbf{A}_{2}=\mathbf{A}_{3}=\mathbf{0}$ in the formulation
of our 2D SSM. The resulting model is equivalent to:

$\displaystyle
h^{(1)}_{v,t+1}=\bar{\mathbf{A}}_{1}h^{(1)}_{v,t}+\bar{\mathbf{B}}_{1}\mathbf{x}_{v,t+1},$
(16) $\displaystyle
h^{(2)}_{v+1,t}=\bar{\mathbf{A}}_{4}h^{(2)}_{v,t}+\bar{\mathbf{B}}_{2}\mathbf{x}_{v+1,t},$
(17)
$\displaystyle\mathbf{y}_{v,t}=\mathbf{C}_{1}h^{(1)}_{v,t}+\mathbf{C}_{2}h^{(2)}_{v,t},$
(18)

where $\bar{\mathbf{A}}_{1}=\exp\left(\Delta_{1}\mathbf{A}_{1}\right)$,
$\bar{\mathbf{A}}_{2}=\exp\left(\Delta_{2}\mathbf{A}_{2}\right)$,
$\bar{\mathbf{B}}_{1}=\begin{bmatrix}\mathbf{A}_{1}^{{-1}}\left(\bar{\mathbf{A}}_{1}-\mathbf{I}\right)\mathbf{B}^{(1)}_{1}\\\
\mathbf{0}\end{bmatrix}$, and
$\bar{\mathbf{B}}_{2}=\begin{bmatrix}\mathbf{0}\\\
\mathbf{A}_{4}^{{-1}}\left(\bar{\mathbf{A}}_{4}-\mathbf{I}\right)\mathbf{B}^{(2)}_{2}\end{bmatrix}$.
This formulation with data-dependent parameters, is equivalent to using two S6
blocks [23] each of which along a dimension. Notably, these two S6 blocks are
not separate as the output $\mathbf{y}_{v,t}$ is based on both hidden states
$h^{(1)}_{v,t}$ and $h^{(2)}_{v,t}$, capturing 2D inductive bias.

2D Mamba-2. Recently, [16] present Mamba-2 that re-formulates S6 block using
structured semi-separable matrices, resulting in more efficient training and
ability of having larger recurrent state sizes. Although we leave the
exploration of how generic 2D SSMs can be re-formulated by tensors (see
Section 5 for further discussion), the special case of
$\mathbf{A}_{2}=\mathbf{A}_{3}=\mathbf{0}$, similar to the above formulation,
can be re-formulated as two SSD blocks [16] each of which along a dimension.
Furthermore, for bi-directionality along the variates, one can use quasi-
separable structured matrices, which inherently captures bi-directionality as
discussed by [8]:

$\displaystyle\mathbf{y}_{v,t}$
$\displaystyle=\underset{{\color[rgb]{0.56640625,0.046875,0.24609375}\definecolor[named]{pgfstrokecolor}{rgb}{0.56640625,0.046875,0.24609375}\text{SSD
Block}}}{\underbrace{\begin{pmatrix}{\mathbf{C}_{1_{v,1}}}{\bar{\mathbf{B}}}_{1_{v,1}}&0&\dots&0\\\
{\mathbf{C}_{1_{v,2}}}\mathbf{A}_{1_{v,2}}{\bar{\mathbf{B}}}_{1_{v,1}}&{\mathbf{C}_{1_{v,2}}}{\bar{\mathbf{B}}}_{1_{v,2}}&\dots&0\\\
\vdots&\vdots&\ddots&\vdots\\\
{\mathbf{C}_{1_{v,t}}}\left(\prod_{i=2}^{t}\mathbf{A}_{1_{v,i}}\right){\bar{\mathbf{B}}}_{1_{v,1}}&{\mathbf{C}_{1_{v,t}}}\left(\prod_{i=3}^{t}\mathbf{A}_{1_{v,i}}\right){\bar{\mathbf{B}}}_{1_{v,2}}&\dots&{\mathbf{C}_{1_{v,t}}}{\bar{\mathbf{B}}}_{1_{v,t}}\\\
\end{pmatrix}}}\mathbf{x}_{v,:}$ (19)
$\displaystyle+\underset{{\color[rgb]{0.56640625,0.046875,0.24609375}\definecolor[named]{pgfstrokecolor}{rgb}{0.56640625,0.046875,0.24609375}\text{Quasi-
Separable
Block}}}{\underbrace{\begin{pmatrix}{\color[rgb]{0.56640625,0.046875,0.24609375}\definecolor[named]{pgfstrokecolor}{rgb}{0.56640625,0.046875,0.24609375}\gamma_{1}}&{\color[rgb]{0.56640625,0.046875,0.24609375}\definecolor[named]{pgfstrokecolor}{rgb}{0.56640625,0.046875,0.24609375}{\mathbf{C}^{\prime}_{2_{v-1,t}}}\mathbf{A}^{\prime}_{4_{v-1,t}}{\bar{\mathbf{B}}^{\prime}}_{2_{v,t}}}&\dots&{\color[rgb]{0.56640625,0.046875,0.24609375}\definecolor[named]{pgfstrokecolor}{rgb}{0.56640625,0.046875,0.24609375}{\mathbf{C}^{\prime}_{2_{v,t}}}\left(\prod_{i=1}^{v-1}\mathbf{A}^{\prime}_{4_{i,t}}\right){\bar{\mathbf{B}}^{\prime}}_{2_{1,t}}}\\\
{\mathbf{C}_{2_{2,t}}}\mathbf{A}_{4_{2,t}}{\bar{\mathbf{B}}}_{2_{1,t}}&{\color[rgb]{0.56640625,0.046875,0.24609375}\definecolor[named]{pgfstrokecolor}{rgb}{0.56640625,0.046875,0.24609375}\gamma_{2}}&\dots&{\color[rgb]{0.56640625,0.046875,0.24609375}\definecolor[named]{pgfstrokecolor}{rgb}{0.56640625,0.046875,0.24609375}{\mathbf{C}^{\prime}_{2_{v-1,t}}}\left(\prod_{i=2}^{v-1}\mathbf{A}^{\prime}_{4_{i,t}}\right){\bar{\mathbf{B}}^{\prime}}_{2_{2,t}}}\\\
\vdots&\vdots&\ddots&\vdots\\\
{\mathbf{C}_{2_{v,t}}}\left(\prod_{i=2}^{v}\mathbf{A}_{4_{i,t}}\right){\bar{\mathbf{B}}}_{2_{1,t}}&{\mathbf{C}_{2_{v,t}}}\left(\prod_{i=3}^{v}\mathbf{A}_{4_{i,t}}\right){\bar{\mathbf{B}}}_{2_{2,t}}&\dots&{\color[rgb]{0.56640625,0.046875,0.24609375}\definecolor[named]{pgfstrokecolor}{rgb}{0.56640625,0.046875,0.24609375}\gamma_{v}}\\\
\end{pmatrix}}}\mathbf{x}_{:,t},$ (20)

where $\mathbf{x}_{v,:}$ and $\mathbf{x}_{:,t}$ are the vectors when we fix
$v$ and $t$ in input $\mathbf{x}$, respectively.

###  3.3 Chimera Neural Architecture

In this section, we use a stack of our 2D SSMs, with non-linearity in between,
to enhance the expressive power and capabilities of the abovementioned 2D SSM.
To this end, similar to deep SSM models [84], we allow all parameters to be
learnable and in each layer we use multiple 2D SSMs, each of which with its
own responsibility. Also, in the data-dependent variant of Chimera, we let
parameters ${\mathbf{B}}_{i},\mathbf{C}_{i},$ and $\Delta_{i}$ for
$i\in\\{1,2\\}$ be the function of the input $\mathbf{x}$:

$\displaystyle{\mathbf{B}}_{i}=\texttt{Linear}_{{\mathbf{B}}_{i}}(x),\qquad{\mathbf{C}}_{i}=\texttt{Linear}_{{\mathbf{C}}_{i}}(x),\qquad\Delta_{i}=\texttt{Softplus}\left(\texttt{Linear}_{\Delta_{i}}(x)\right).$
(21)

Chimera follows the commonly used decomposition of time series, and decomposes
them into trend components and seasonal patterns. it, however, uses special
traits of 2D SSM to capture these terms.

Seasonal Patterns. To capture the multi-resolution seasonal patterns, we take
advantage of the discretization process. Proposition 3.1 states that if
$\mathbf{x}(v,t)\mapsto\mathbf{y}(v,t)$ with parameters
$(\\{\bar{\mathbf{A}}_{i}\\},\\{\bar{\mathbf{B}}_{i}\\},\\{\bar{\mathbf{C}}_{i}\\},\Delta_{1},\Delta_{2})$
then $\mathbf{x}(v,kt)\mapsto\mathbf{y}(v,kt)$ with
$(\\{\bar{\mathbf{A}}_{i}\\},\\{\bar{\mathbf{B}}_{i}\\},\\{\bar{\mathbf{C}}_{i}\\},k\Delta_{1},\Delta_{2})$.
Accordingly, we use $\texttt{2D-SSM}(.)$ module with a separate learnable
$\Delta_{s}$ that is responsible to learn the best resolution to capture
seasonal patterns. Another interpretation for this module is based on
SAR$(p,q,s)$ (Equation 4). In this case, $\Delta_{s}$ aims to learn a proper
parameter $s$ to capture seasonal patterns. Since we expect the resolution
before and after this module matches, we add additional re-discretization
module (a simple linear layer), after this module.

Trend Components. The second module of Chimera,
$\texttt{2D-SSM}_{t}\left(.\right)$ simply uses a sequence of multiple 2D SSMs
to learn trend components. Proper combination of the outputs of this and the
previous modules can capture both seasonal and trend components.

Both Modules Together. We followed previous studies [69] and consider residual
connection modeling for learning trend and seasonal patterns. Given input data
$\tilde{\mathbf{X}}_{0}=\mathbf{X}$, and $\ell=0,\dots,\mathcal{L}$, we have:

$\displaystyle\hat{\mathbf{X}}_{\ell+1}=\texttt{2D-SSM}_{t}\left(\tilde{\mathbf{X}}_{\ell}\right),$
(22) $\displaystyle\tilde{\mathbf{X}}_{\ell+1}=\texttt{Re-
Discretization}\left(\texttt{2D-SSM}_{s}\left(\tilde{\mathbf{X}}_{\ell}-\hat{\mathbf{X}}_{\ell+1}\right)\right).$
(23)

Figure 1 illustrate the architecture of Chimera. Due to the ability of our 2D
SSM to recover smoothing techniques (see Theorem 3.4), this combination of
modules for trend and seasonal patterns can be viewed as a generalization of
traditional methods that use moving average with residual connection to model
seasonality [69].

Gating with Linear Mapping. Inspired by the success of gated recurrent and
SSM-based models [61, 23], we use a head of a fully connected layer with Swish
[62], resulting in SwiGLU variant [70]. While we validate the significance of
this head, this

Closed-Loop 2D SSM Decoder. To enhance the generalizability and the ability of
our model for longer-horizon, we extend the closed-loop decoder module [84],
which is similar to autoregression, to multivariate time series. We use
distinct processes for the inputs and outputs, using additional matrices
$\mathbf{D}_{1}$ and $\mathbf{D}_{2}$ in each decoder 2D SSM, we model future
input time-steps explicitly:

$\displaystyle\mathbf{y}_{v,t}=\mathbf{C}_{1}h^{(1)}_{v,t}+\mathbf{C}_{2}h^{(2)}_{v,t},$
(24)
$\displaystyle\mathbf{u}_{v,t}=\mathbf{D}_{1}h^{(1)}_{v,t}+\mathbf{D}_{2}h^{(2)}_{v,t},$
(25)

where $\mathbf{u}_{v,t}$ is the next input and $\mathbf{y}_{v,t}$ is the
output. Note that the other parts (recurrence) are the same as Equation 6.
Figure 2 illustrate the architecture of closed-loop 2D SSM.

###  3.4 Theoretical Justification

In this section, we provide some theoretical evidences for the performance of
Chimera. These results are mostly revisiting the theorems by [84] and [4], and
extending them for Chimera. In the first theorem, we show that Chimera
recovers several classic methods, and pre-processing steps as it can recover
SpaceTime [84] and additionally because of its design, it can recover SARIMA
[9]:

######  Theorem 3.4.

Chimera can represent seasonal autoregressive process, SARIMA [9], SpaceTime
[84], and so ARIMA [5], exponential smoothing [75], and controllable linear
time–invariant systems [13].

######  Theorem 3.5.

Chimera can represent S4nd [55], TSM2 [8], and TSMixer [12].

Next theorem compares the expressiveness of Chimera with some existing 2D deep
SSMs. Since Chimera can recover 2DSSM [4], it can express full-rank kernels
with a constant number of parameters:

######  Theorem 3.6.

Similar to 2DSSM [4], Chimera can express full-rank kernels with
$\mathcal{O}\left(1\right)$ parameters, while existing deep SSMs [55, 8]
require $\mathcal{O}\left(N\right)$ parameters to express $N$-rank kernels.

##  4 Experiments

Goals and Baselines. We evaluate Chimera on a wide range of time series tasks.
In § 4.1 we compare Chimera with the state-of-the-art general multivariate
time series models [77, 51, 44, 76, 79, 89, 86, 47, 8, 17, 45, 58] on long-
term forecasting and classification tasks. In the next part, we test the
performance of Chimera in short-term forecasting. In § 4.1 we perform a case
study on human neural activity to classify seen images, which requires
capturing complex dynamic dependencies of variates, to test the ability of
Chimera in capturing cross-variate information and the significance of data-
dependency. In § 4.2, we evaluate the significance of the Chimera’s components
by performing ablation studies. In § 4.2, we evaluate whether the superior
performance of Chimera coincide with its efficiency. Finally, we test the
Chimera’s generalizability on unseen variates and further evaluate its ability
to filter irrelevant context in § 4.3. The details and additional experiments
are in Appendix G.

Table 1: Average Performance on long-term forecasting task. The first and
second results are highlighted in red (bold) and orange (underline). Full
results are reported in Appendix G.

|  Chimera |  TSM2 |  Simba |  TCN |  iTransformer |  RLinear |  PatchTST |  Crossformer |  TiDE |  TimesNet |  DLinear
---|---|---|---|---|---|---|---|---|---|---|---
(ours) |  behrouz2024mambamixer |  patro2024simba |  donghao2024moderntcn |  liu2024itransformer |  li2023revisiting |  nie2023a |  zhang2022crossformer |  das2023longterm |  wu2023timesnet |  zeng2023transformers
|  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE
ETTm1 |  0.345 |  0.377 |  0.361 |  - |  0.383 |  0.396 |  0.351 |  0.381 |  0.407 |  0.410 |  0.414 |  0.407 |  0.387 |  0.400 |  0.513 |  0.496 |  0.419 |  0.419 |  0.400 |  0.406 |  0.403 |  0.407
ETTm2 |  0.250 |  0.316 |  0.267 |  - |  0.271 |  0.327 |  0.253 |  0.314 |  0.288 |  0.332 |  0.286 |  0.327 |  0.281 |  0.326 |  0.757 |  0.610 |  0.358 |  0.404 |  0.291 |  0.333 |  0.350 |  0.401
ETTh1 |  0.405 |  0.424 |  0.403 |  - |  0.441 |  0.432 |  0.404 |  0.420 |  0.454 |  0.447 |  0.446 |  0.434 |  0.469 |  0.454 |  0.529 |  0.522 |  0.541 |  0.507 |  0.458 |  0.450 |  0.456 |  0.452
ETTh2 |  0.318 |  0.375 |  0.333 |  - |  0.361 |  0.391 |  0.322 |  0.379 |  0.383 |  0.407 |  0.374 |  0.398 |  0.387 |  0.407 |  0.942 |  0.684 |  0.611 |  0.550 |  0.414 |  0.427 |  0.559 |  0.515
ECL |  0.154 |  0.249 |  0.169 |  - |  0.185 |  0.274 |  0.156 |  0.253 |  0.178 |  0.270 |  0.219 |  0.298 |  0.205 |  0.290 |  0.244 |  0.334 |  0.251 |  0.344 |  0.192 |  0.295 |  0.212 |  0.300
Exchange |  0.311 |  0.358 |  0.443 |  - |  - |  - |  0.302 |  0.366 |  0.360 |  0.403 |  0.378 |  0.417 |  0.367 |  0.404 |  0.940 |  0.707 |  0.370 |  0.413 |  0.416 |  0.443 |  0.354 |  0.414
Traffic |  0.403 |  0.286 |  0.420 |  - |  0.493 |  0.291 |  0.398 |  0.270 |  0.428 |  0.282 |  0.626 |  0.378 |  0.481 |  0.304 |  0.550 |  0.304 |  0.760 |  0.473 |  0.620 |  0.336 |  0.625 |  0.383
Weather |  0.219 |  0.258 |  0.239 |  - |  0.255 |  0.280 |  0.224 |  0.264 |  0.258 |  0.278 |  0.272 |  0.291 |  0.259 |  0.281 |  0.259 |  0.315 |  0.271 |  0.320 |  0.259 |  0.287 |  0.265 |  0.317
$1^{\text{st}}$ Count |  5 |  5 |  1 |  - |  0 |  0 |  2 |  3 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0

###  4.1 Main Results: Classification and Forecasting

Long-Term Forecasting. We perform experiments in long-term forecasting task on
benchmark datasets [87]. Table 1 reports the average of results over different
horizons (for the results of each see Table 8). Chimera shows outstanding
performance, achieving the best or the second best results in all the datasets
and outperforms baselines in 5 out of 8 benchmarks. Notably, it surpasses
extensively studied MLP-based and Transformer-based models while being more
efficient (see subsection 4.1, subsection 4.2, and Appendix G), providing a
better balance of performance and efficiency. It further _significantly_
outperforms recurrent models, including very recent Mamba-based architectures
[8, 58], unleashing the potential of classical models, SSMs, when are
carefully designed in deep learning settings.

Classification and Anomaly Detection. We evaluate the performance of Chimera
in ECG classification on PTB-XL dataset [72] (see subsection 4.1), speech
classification [74](subsection 4.1), 10 multivariate datasets from UEA Time
Series Classification Archive [3] (see subsection 4.2 and Table 10), and
anomaly detection tasks on five widely-used benchmarks: SMD [67], SWaT [54],
PSM [1] and SMAP [34] (see subsection 4.2 and Table 11). For each benchmark,
we use the state-of-the-art methods that are applicable to the task as the
baselines. subsection 4.1 reports the performance of Chimera and baselines on
ECG classification tasks. Chimera outperforms all the baselines in 4/6 tasks,
while achieving the second best results on the remaining tasks. Since these
tasks are univariate time series, we attribute the outstanding performance of
Chimera, specifically compared to SpaceTime [84], to its ability of capturing
seasonal patterns and its input-dependent parameters, resulting in dynamically
learn dependencies.

subsection 4.1 reports the results on speech audio classification task, which
require long-range modeling of time series. Due to the length of the sequence
(16K), LSSL [26] and Transformer [71] has out of memory (OOM) issue, showing
the efficiency of Chimera compared to alternative backbones.

Finally, we report the summary of the results in multivariate time series
classification and anomaly detection tasks in subsection 4.2. The full list of
results can be found in Table 10 and Table 11. Chimera shows outstanding
performance, achieving highest average accuracy and F1 score in classification
and anomaly detection tasks even compared to very recent and state-of-the-art
methods [77, 51].

Table 2: ECG statement classification on PTB-XL (100 Hz version).

Tasks | All | Diag | Sub-diag | Super-diag | Form | Rhythm
---|---|---|---|---|---|---
Chimera | 0.941 | 0.947 | 0.935 | 0.930 | 0.901 | 0.975
SpaceTime [84] | 0.936 | 0.941 | 0.933 | 0.929 | 0.883 | 0.967
S4 [26] | 0.938 | 0.939 | 0.929 | 0.931 | 0.895 | 0.977
Inception | 0.925 | 0.931 | 0.930 | 0.921 | 0.899 | 0.953
xRN-101 | 0.925 | 0.937 | 0.929 | 0.928 | 0.896 | 0.957
LSTM | 0.907 | 0.927 | 0.928 | 0.927 | 0.851 | 0.953
Transformer | 0.857 | 0.876 | 0.882 | 0.887 | 0.771 | 0.831

Table 3: Speech classification.

Method | Acc. (%)
---|---
Chimera | 98.40
SpaceTime | 97.29
S4 | 98.32
LSSL | OOM
WaveGan-D | 96.25
Transformer | OOM

Short-Term Forecasting. Our evaluation on short-term forecasting tasks on M4
benchmark datasets [22] reports in Table 4 (Full list in Table 9), which also
shows the superior performance of Chimera compared to baselines.

Table 4: Short-term forecasting task on the M4 dataset. Full results are
reported in Appendix G.

Models |  Chimera |  ModernTCN |  PatchTST |  TimesNet |  N-HiTS |  N-BEATS∗ |  ETS$\ast$ |  LightTS |  DLinear |  FED$\ast$ |  Stationary |  Auto$\ast$ |  Pyra$\ast$ |  In$\ast$ |  Re$\ast$ |  LSTM
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
(ours) |  donghao2024moderntcn |  nie2023a |  wu2023timesnet |  challu2022n |  oreshkin2019n |  woo2022etsformer |  Zhang2022LessIM |  Zeng2022AreTE |  zhou2022fedformer |  Liu2022NonstationaryTR |  wu2021autoformer |  liu2021pyraformer |  zhou2021informer |  kitaev2020reformer |  Hochreiter1997LongSM
Weighted |  Average |  SMAPE |  11.618 |  11.698 |  11.807 |  11.829 |  11.927 |  11.851 |  14.718 |  13.525 |  13.639 |  12.840 |  12.780 |  12.909 |  16.987 |  14.086 |  18.200 |  160.031
MASE |  1.528 |  1.556 |  1.590 |  1.585 |  1.613 |  1.599 |  2.408 |  2.111 |  2.095 |  1.701 |  1.756 |  1.771 |  3.265 |  2.718 |  4.223 |  25.788
OWA |  0.827 |  0.838 |  0.851 |  0.851 |  0.861 |  0.855 |  1.172 |  1.051 |  1.051 |  0.918 |  0.930 |  0.939 |  1.480 |  1.230 |  1.775 |  12.642

Case Study of Brain Activity. Input dependency is a must to capture the
dynamic of dependencies. To support this claim, we use BVFC [6] (multivariate
time series only), which aim to classify seen images by its corresponding
brain activity response. This task, requires focusing more on the dependencies
of brain units and their responses rather than the actual time series. Also,
since each window corresponds to a specific image, the model needs to capture
the dependencies based on the current window, requiring to be input-dependent.
Results are reported in subsection 4.2. Chimera significantly outperforms all
the baselines including our Chimera but without data-dependent parameters
(convolution form). Due to the large number of brain units, i.e., 9K, in the
first dataset, transformer-based methods face OOM issue. However, they are
also data-dependent and so shows the second best results in second and third
datasets. This results support the significance of data-dependency in Chimera.

###  4.2 Ablation Study and Efficiency

To evaluate the significance of the Chimera’s design, we perform ablation
studies and remove one of the components at each time, keeping other parts
unchanged. subsection 4.2 reports the results. The first row reports the
Chimera’s performance, while row 2 uses unidirectional recurrence along the
variate dimension, row 3 removes the gating mechanism, row 4 uses convolution
form (data-independent), and row 5 removes the module for seasonal patterns.
The results show that all the components of Chimera contributes to its
performance.

Table 5: Image classification by brain activity (Acc. %).

Method | Chimera | Chimera (ind.) | SpaceTime | S4 | iTrans. | Trans. | DLinear
---|---|---|---|---|---|---|---
(ours) | (ours) |  zhang2023effectively |  gu2022efficiently |  liu2024itransformer |  vaswani2017attention |  Zeng2022AreTE
BVFC (9K) | 69.41 | 62.36 | 41.20 | 40.89 | OOM | OOM | 39.74
BVFC (1K) | 58.99 | 50.25 | 34.31 | 35.19 | 54.18 | 43.60 | 33.09
BVFC (400) | 51.08 | 45.17 | 33.58 | 33.76 | 48.22 | 38.05 | 32.73

Table 6: Ablation study on the Chimera’s design.

Method | ETTh1 | ETTm1 | ETTh2
---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE
Chimera | 0.405 | 0.424 | 0.345 | 0.377 | 0.318 | 0.375
Uni.-directional | 0.409 | 0.429 | 0.354 | 0.385 | 0.326 | 0.381
w/o Gating | 0.418 | 0.433 | 0.351 | 0.384 | 0.321 | 0.379
Input-independent | 0.471 | 0.498 | 0.361 | 0.389 | 0.372 | 0.401
w/o seasonal | 0.426 | 0.431 | 0.357 | 0.382 | 0.331 | 0.386

Figure 3: Classification and anomaly detection performance. Full list with
additional baselines is in Appendix G. Figure 4: Wall-clock scaling.

Length of Time Series. We perform experiments on the effect of the sequence
length on the efficiency of Chimera and baselines. The results are reported in
subsection 4.2. Chimera scales linearly with respect to the sequence length
and has smoother scaling than S4 [26] and Transformers [71]. These results
also highlight the significance of our algorithm that uses 2D parallel scans
for training Chimera. This algorithm results in $\approx\times 4$ faster
training, which is very closed to the convolutional format _without_ data
dependency. Chimera also has a close running time to SpaceTime [84], which has
1D recurrent.

###  4.3 Selection Mechanism Along Time and Variate

Variate Generalization. We argue that the data-dependency with discretization
allows the model to filter the irrelevant context based on the input,
resulting in more generalizability. Inspired by [47], we train our model (and
baseline) on 20% of variates and evaluate its generalizability to unseen
variates. The results are reported in subsection 4.3. Chimera has on par
generalizability compared to Transformers (when applied along the variate
dimension), which we attributes to its data-dependent parameters as Chimera
with convolution form performs poorly on unseen variates.

Context Filtering. Increasing the lookback length does not necessarily result
in better performance for Transformers [47]. Due to the selection mechanism of
Chimera, we expect it to filter irrelevant information and monotonically
performs better. subsection 4.3 reports the Chimera’s performance (w/ and w/o
data-dependency) and transformer-based baselines [87, 78] while varying the
lookback length. Chimera due to its selection mechanism monotonically performs
better with increasing the lookback.

Figure 5: Selection results in generalization to unseen variates. Figure 6:
Effect of lookback length.

##  5 Conclusion and Future Work

This paper presents Chimera, a three-headed 2-dimensional SSM model with
provably high expressive power. Chimera is based on 2D SSMs with careful
design of parameters that allows it to dynamically and simultaneously capture
the dependencies along both time and variate dimensions. We provide different
views of our 2D SSM for efficient training, and present a data-dependent
formulation with a fast implementation using 2D scans. Chimera uses two
different modules to capture trend and seasonal patterns and its
discretization process allows these modules to adjust the resolution at each
time stamp and for each variate. Our experimental and theoretical results
support the effectiveness and efficiency of Chimera in a wide range of tasks.

Other Data Modalities. While the parameterization of Chimera is designed to
expressively model time series data, the overall architecture of Chimera and
our data-dependent 2D SSM with its 2D scan form in training are potentially
applicable for other higher dimensional data types, e.g., images, videos,
multi-channel speech, etc. Despite recent attempts to design effective SSM-
based vision models [59], the existing models suffer from the lack of 2D
spatial inductive bias. Our 2D SSM, however, is able to provide 2D inductive
bias, potentially being more effective than existing 1D selective SSMs.
Accordingly, a promising direction is to explore the potential of 2D selective
SSMs for other high dimensional data modalities and different tasks.

Variants of Chimera. As discussed in Section 3.2, different variants of
Chimera result in the extension of well-known architectures like Mamba [23] to
2-dimensional data, or extension of methods like S4ND [55], and 2DSSM [4] to
have data-dependent weights. Despite the fact that our formulation of the 2D
SSM with discretization and data dependent parameters provides a more general
framework to extend SSMs to higher-dimensional data, it does not necessarily
mean that for any data modalities and network size, its generic form can
achieve the best result. While our experimental evaluation is limited to the
generic form of our 2D SSM and Chimera, it is a promising future direction to
see if limiting transition matrix $\mathbf{A}_{i}$ (i.e., 2D Mamba, 2D
Mamba-2) can result in more powerful models. We leave the experimental
evaluations of these spacial cases of our 2D SSM for future work.

Efficiency. While our 2D scan decreases the number of required recurrence to
compute the hidden states, its still based on a naive implementation of
parallel scan. There is a potential for further improvement of 2D parallel
scan’s efficiency by using more hardware-aware implementations similar to
selective scan by [23].

## References

  * [1] Ahmed Abdulaal, Zhuanghua Liu and Tomer Lancewicki  “Practical approach to asynchronous multivariate time series anomaly detection and localization”  In _Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining_, 2021, pp. 2485–2494
  * [2] Masanao Aoki  “State space modeling of time series”  Springer Science & Business Media, 2013
  * [3] Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam and Eamonn Keogh  “The UEA multivariate time series classification archive, 2018”  In _arXiv preprint arXiv:1811.00075_ , 2018
  * [4] Ethan Baron, Itamar Zimerman and Lior Wolf  “A 2-Dimensional State Space Layer for Spatial Inductive Bias”  In _The Twelfth International Conference on Learning Representations_ , 2024  URL: https://openreview.net/forum?id=BGkqypmGvm
  * [5] David J Bartholomew  “Time Series Analysis Forecasting and Control.”  JSTOR, 1971
  * [6] Ali Behrouz, Parsa Delavari and Farnoosh Hashemi  “Unsupervised Representation Learning of Brain Activity via Bridging Voxel Activity and Functional Connectivity”  In _International conference on machine learning (ICML)_ , 2024
  * [7] Ali Behrouz and Farnoosh Hashemi  “Graph Mamba: Towards Learning on Graphs with State Space Models”  In _arXiv preprint arXiv:2402.08678_ , 2024
  * [8] Ali Behrouz, Michele Santacatterina and Ramin Zabih  “Mambamixer: Efficient selective state space models with dual token and channel selection”  In _arXiv preprint arXiv:2403.19888_ , 2024
  * [9] Michael Bender and Slobodan Simonovic  “Time-series modeling for long-range stream-flow forecasting”  In _Journal of Water Resources Planning and Management_ 120.6 American Society of Civil Engineers, 1994, pp. 857–870
  * [10] George EP Box and Gwilym M Jenkins  “Some recent advances in forecasting and control”  In _Journal of the Royal Statistical Society. Series C (Applied Statistics)_ 17.2 JSTOR, 1968, pp. 91–109
  * [11] Cristian Challu, Kin G Olivares, Boris N Oreshkin, Federico Garza, Max Mergenthaler and Artur Dubrawski  “N-HiTS: Neural Hierarchical Interpolation for Time Series Forecasting”  In _arXiv preprint arXiv:2201.12886_ , 2022
  * [12] Si-An Chen, Chun-Liang Li, Nate Yoder, Sercan O Arik and Tomas Pfister  “Tsmixer: An all-mlp architecture for time series forecasting”  In _arXiv preprint arXiv:2303.06053_ , 2023
  * [13] Chi-Tsong Chen  “Linear system theory and design”  Saunders college publishing, 1984
  * [14] Zonglei Chen, Minbo Ma, Tianrui Li, Hongjun Wang and Chongshou Li  “Long sequence time-series forecasting with deep learning: A survey”  In _Information Fusion_ 97 Elsevier, 2023, pp. 101819
  * [15] Junyoung Chung, Caglar Gulcehre, KyungHyun Cho and Yoshua Bengio  “Empirical evaluation of gated recurrent neural networks on sequence modeling”  In _arXiv preprint arXiv:1412.3555_ , 2014
  * [16] Tri Dao and Albert Gu  “Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality”  In _International Conference on Machine Learning (ICML)_ , 2024
  * [17] Abhimanyu Das, Weihao Kong, Andrew Leach, Shaan K Mathur, Rajat Sen and Rose Yu  “Long-term Forecasting with TiDE: Time-series Dense Encoder”  In _Transactions on Machine Learning Research_ , 2023  URL: https://openreview.net/forum?id=pCbC3aQB5W
  * [18] Rikus Eising  “Realization and stabilization of 2-D systems”  In _IEEE Transactions on Automatic Control_ 23.5 IEEE, 1978, pp. 793–799
  * [19] Ettore Fornasini and Giovanni Marchesini  “Doubly-indexed dynamical systems: State-space models and structural properties”  In _Mathematical systems theory_ 12.1 Springer, 1978, pp. 59–72
  * [20] Jean-Yves Franceschi, Aymeric Dieuleveut and Martin Jaggi  “Unsupervised Scalable Representation Learning for Multivariate Time Series”  In _NeurIPS_ , 2019
  * [21] Kelum Gajamannage, Yonggi Park and Dilhani I Jayathilake  “Real-time forecasting of time series in financial markets using sequentially trained dual-LSTMs”  In _Expert Systems with Applications_ 223 Elsevier, 2023, pp. 119879
  * [22] Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I Webb, Rob J Hyndman and Pablo Montero-Manso  “Monash time series forecasting archive”  In _arXiv preprint arXiv:2105.06643_ , 2021
  * [23] Albert Gu and Tri Dao  “Mamba: Linear-time sequence modeling with selective state spaces”  In _arXiv preprint arXiv:2312.00752_ , 2023
  * [24] Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra and Christopher Ré  “Hippo: Recurrent memory with optimal polynomial projections”  In _Advances in neural information processing systems_ 33, 2020, pp. 1474–1487
  * [25] Albert Gu, Karan Goel, Ankit Gupta and Christopher Ré  “On the Parameterization and Initialization of Diagonal State Space Models”  In _Advances in Neural Information Processing Systems_ , 2022  URL: https://openreview.net/forum?id=yJE7iQSAep
  * [26] Albert Gu, Karan Goel and Christopher Re  “Efficiently Modeling Long Sequences with Structured State Spaces”  In _International Conference on Learning Representations_ , 2022  URL: https://openreview.net/forum?id=uYLFoz1vlAC
  * [27] Albert Gu, Caglar Gulcehre, Thomas Paine, Matt Hoffman and Razvan Pascanu  “Improving the gating mechanism of recurrent neural networks”  In _International Conference on Machine Learning_ , 2020, pp. 3800–3809  PMLR
  * [28] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra and Christopher Ré  “Combining recurrent, convolutional, and continuous-time models with linear state space layers”  In _Advances in neural information processing systems_ 34, 2021, pp. 572–585
  * [29] Albert Gu, Isys Johnson, Aman Timalsina, Atri Rudra and Christopher Re  “How to Train your HIPPO: State Space Models with Generalized Orthogonal Basis Projections”  In _International Conference on Learning Representations_ , 2023  URL: https://openreview.net/forum?id=klK17OQ3KB
  * [30] Andrew C Harvey  “Forecasting, structural time series models and the Kalman filter”  In _Cambridge university press_ Cambridge university press, 1990
  * [31] Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kidder, Anna Corriveau, Maryam Vaziri-Pashkam and Chris I Baker  “THINGS-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and behavior”  In _Elife_ 12 eLife Sciences Publications Limited, 2023, pp. e82580
  * [32] Ts Hinamoto  “Realizations of a state-space model from two-dimensional input-output map”  In _IEEE Transactions on Circuits and Systems_ 27.1 IEEE, 1980, pp. 36–44
  * [33] S. Hochreiter and J. Schmidhuber  “Long Short-Term Memory”  In _Neural Comput._ , 1997
  * [34] Kyle Hundman, Valentino Constantinou, Christopher Laporte, Ian Colwell and Tom Soderstrom  “Detecting spacecraft anomalies using lstms and nonparametric dynamic thresholding”  In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, 2018, pp. 387–395
  * [35] Romain Ilbert, Ambroise Odonnat, Vasilii Feofanov, Aladin Virmaux, Giuseppe Paolo, Themis Palpanas and Ievgen Redko  “Unlocking the Potential of Transformers in Time Series Forecasting with Sharpness-Aware Minimization and Channel-Wise Attention”  In _arXiv preprint arXiv:2402.10198_ , 2024
  * [36] Plamen Ch Ivanov, Luis A Nunes Amaral, Ary L Goldberger, Shlomo Havlin, Michael G Rosenblum, Zbigniew R Struzik and H Eugene Stanley  “Multifractality in human heartbeat dynamics”  In _Nature_ 399.6735 Nature Publishing Group UK London, 1999, pp. 461–465
  * [37] Alistair EW Johnson, Lucas Bulgarelli, Lu Shen, Alvin Gayles, Ayad Shammout, Steven Horng, Tom J Pollard, Sicheng Hao, Benjamin Moody and Brian Gow  “MIMIC-IV, a freely accessible electronic health record dataset”  In _Scientific data_ 10.1 Nature Publishing Group UK London, 2023, pp. 1
  * [38] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas and François Fleuret  “Transformers are rnns: Fast autoregressive transformers with linear attention”  In _International conference on machine learning_ , 2020, pp. 5156–5165  PMLR
  * [39] Nikita Kitaev, Lukasz Kaiser and Anselm Levskaya  “Reformer: The Efficient Transformer”  In _ICLR_ , 2020
  * [40] Sun-Yuan Kung, Bernard C Levy, Martin Morf and Thomas Kailath  “New results in 2-D systems theory, Part II: 2-D state-space models—realization and the notions of controllability, observability, and minimality”  In _Proceedings of the IEEE_ 65.6 IEEE, 1977, pp. 945–961
  * [41] Guokun Lai, Wei-Cheng Chang, Yiming Yang and Hanxiao Liu  “Modeling long-and short-term temporal patterns with deep neural networks”  In _SIGIR_ , 2018
  * [42] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang and Xifeng Yan  “Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecasting”  In _NeurIPS_ , 2019
  * [43] Zhe Li, Shiyi Qi, Yiduo Li and Zenglin Xu  “Revisiting long-term time series forecasting: An investigation on linear mapping”  In _arXiv preprint arXiv:2305.10721_ , 2023
  * [44] Bryan Lim and Stefan Zohren  “Time-series forecasting with deep learning: a survey”  In _Philosophical Transactions of the Royal Society A_ 379.2194 The Royal Society Publishing, 2021, pp. 20200209
  * [45] Minhao Liu, Ailing Zeng, Muxi Chen, Zhijian Xu, Qiuxia Lai, Lingna Ma and Qiang Xu  “Scinet: Time series modeling and forecasting with sample convolution and interaction”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 5816–5828
  * [46] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu and Schahram Dustdar  “Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting”  In _International conference on learning representations_ , 2021
  * [47] Yong Liu, Tengge Hu, Haoran Zhang, Haixu Wu, Shiyu Wang, Lintao Ma and Mingsheng Long  “iTransformer: Inverted Transformers Are Effective for Time Series Forecasting”  In _The Twelfth International Conference on Learning Representations_ , 2024  URL: https://openreview.net/forum?id=JePfAI8fah
  * [48] Yong Liu, Haixu Wu, Jianmin Wang and Mingsheng Long  “Non-stationary transformers: Exploring the stationarity in time series forecasting”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 9881–9893
  * [49] Yong Liu, Haixu Wu, Jianmin Wang and Mingsheng Long  “Non-stationary Transformers: Rethinking the Stationarity in Time Series Forecasting”  In _NeurIPS_ , 2022
  * [50] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye and Yunfan Liu  “Vmamba: Visual state space model”  In _arXiv preprint arXiv:2401.10166_ , 2024
  * [51] Donghao Luo and Xue Wang  “ModernTCN: A Modern Pure Convolution Structure for General Time Series Analysis”  In _The Twelfth International Conference on Learning Representations_ , 2024  URL: https://openreview.net/forum?id=vpJMJerXHU
  * [52] Jun Ma, Feifei Li and Bo Wang  “U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation”  In _arXiv preprint arXiv:2401.04722_ , 2024
  * [53] Eric Martin and Chris Cundy  “Parallelizing Linear Recurrent Neural Nets Over Sequence Length”  In _International Conference on Learning Representations_ , 2018  URL: https://openreview.net/forum?id=HyUNwulC-
  * [54] Aditya P Mathur and Nils Ole Tippenhauer  “SWaT: A water treatment testbed for research and training on ICS security”  In _2016 international workshop on cyber-physical systems for smart water networks (CySWater)_ , 2016, pp. 31–36  IEEE
  * [55] Eric Nguyen, Karan Goel, Albert Gu, Gordon Downs, Preey Shah, Tri Dao, Stephen Baccus and Christopher Ré  “S4nd: Modeling images and videos as multidimensional signals with state spaces”  In _Advances in neural information processing systems_ 35, 2022, pp. 2846–2861
  * [56] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong and Jayant Kalagnanam  “A Time Series is Worth 64 Words: Long-term Forecasting with Transformers”  In _The Eleventh International Conference on Learning Representations_ , 2023  URL: https://openreview.net/forum?id=Jbdc0vTOcol
  * [57] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados and Yoshua Bengio  “N-BEATS: Neural basis expansion analysis for interpretable time series forecasting”  In _ICLR_ , 2019
  * [58] Badri N. Patro and Vijay S. Agneeswaran  “SiMBA: Simplified Mamba-Based Architecture for Vision and Multivariate Time series”, 2024  arXiv:2403.15360 [cs.CV]
  * [59] Badri Narayana Patro and Vijay Srinivas Agneeswaran  “Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challenges”  In _arXiv preprint arXiv:2404.16112_ , 2024
  * [60] Steve Pincus and Rudolf E Kalman  “Irregularity, volatility, risk, and financial market time series”  In _Proceedings of the National Academy of Sciences_ 101.38 National Acad Sciences, 2004, pp. 13709–13714
  * [61] Zhen Qin, Songlin Yang and Yiran Zhong  “Hierarchically gated recurrent neural network for sequence modeling”  In _Advances in Neural Information Processing Systems_ 36, 2023
  * [62] Prajit Ramachandran, Barret Zoph and Quoc V Le  “Searching for activation functions”  In _arXiv preprint arXiv:1710.05941_ , 2017
  * [63] David Salinas, Valentin Flunkert, Jan Gasthaus and Tim Januschowski  “DeepAR: Probabilistic forecasting with autoregressive recurrent networks”  In _International journal of forecasting_ 36.3 Elsevier, 2020, pp. 1181–1191
  * [64] Yair Schiff, Chia-Hsiang Kao, Aaron Gokaslan, Tri Dao, Albert Gu and Volodymyr Kuleshov  “Caduceus: Bi-directional equivariant long-range dna sequence modeling”  In _arXiv preprint arXiv:2403.03234_ , 2024
  * [65] Imanol Schlag, Kazuki Irie and Jürgen Schmidhuber  “Linear transformers are secretly fast weight programmers”  In _International Conference on Machine Learning_ , 2021, pp. 9355–9366  PMLR
  * [66] Jimmy T.H. Smith, Andrew Warrington and Scott Linderman  “Simplified State Space Layers for Sequence Modeling”  In _The Eleventh International Conference on Learning Representations_ , 2023  URL: https://openreview.net/forum?id=Ai8Hw3AXqks
  * [67] Ya Su, Youjian Zhao, Chenhao Niu, Rong Liu, Wei Sun and Dan Pei  “Robust anomaly detection for multivariate time series through stochastic recurrent neural network”  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, 2019, pp. 2828–2837
  * [68] Corentin Tallec and Yann Ollivier  “Can recurrent neural networks warp time?”  In _International Conference on Learning Representations_ , 2018  URL: https://openreview.net/forum?id=SJcKhk-Ab
  * [69] William Toner and Luke Darlow  “An Analysis of Linear Time Series Forecasting Models”  In _International conference on machine learning (ICML)_ , 2024
  * [70] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro and Faisal Azhar  “Llama: Open and efficient foundation language models”  In _arXiv preprint arXiv:2302.13971_ , 2023
  * [71] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser and Illia Polosukhin  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [72] Patrick Wagner, Nils Strodthoff, Ralf-Dieter Bousseljot, Dieter Kreiseler, Fatima I Lunze, Wojciech Samek and Tobias Schaeffter  “PTB-XL, a large publicly available electrocardiography dataset”  In _Scientific data_ 7.1 Nature Publishing Group, 2020, pp. 1–15
  * [73] Junxiong Wang, Jing Nathan Yan, Albert Gu and Alexander Rush  “Pretraining Without Attention”  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ Singapore: Association for Computational Linguistics, 2023, pp. 58–69  DOI: 10.18653/v1/2023.findings-emnlp.5
  * [74] Pete Warden  “Speech commands: A dataset for limited-vocabulary speech recognition”  In _arXiv preprint arXiv:1804.03209_ , 2018
  * [75] Peter R Winters  “Forecasting sales by exponentially weighted moving averages”  In _Management science_ 6.3 INFORMS, 1960, pp. 324–342
  * [76] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar and Steven C.. Hoi  “ETSformer: Exponential Smoothing Transformers for Time-series Forecasting”  In _arXiv preprint arXiv:2202.01381_ , 2022
  * [77] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang and Mingsheng Long  “TimesNet: Temporal 2D-Variation Modeling for General Time Series Analysis”  In _The Eleventh International Conference on Learning Representations_ , 2023  URL: https://openreview.net/forum?id=ju_Uqw384Oq
  * [78] Haixu Wu, Jialong Wu, Jiehui Xu, Jianmin Wang and Mingsheng Long  “Flowformer: Linearizing Transformers with Conservation Flows”  In _ICML_ , 2022
  * [79] Haixu Wu, Jiehui Xu, Jianmin Wang and Mingsheng Long  “Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting”  In _Advances in neural information processing systems_ 34, 2021, pp. 22419–22430
  * [80] Jiehui Xu, Haixu Wu, Jianmin Wang and Mingsheng Long  “Anomaly Transformer: Time Series Anomaly Detection with Association Discrepancy”  In _ICLR_ , 2021
  * [81] Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda and Yoon Kim  “Gated linear attention transformers with hardware-efficient training”  In _International conference on machine learning (ICML)_ , 2024
  * [82] Ailing Zeng, Muxi Chen, Lei Zhang and Qiang Xu  “Are Transformers Effective for Time Series Forecasting?”  In _AAAI_ , 2023
  * [83] Ailing Zeng, Muxi Chen, Lei Zhang and Qiang Xu  “Are transformers effective for time series forecasting?”  In _Proceedings of the AAAI conference on artificial intelligence_ 37, 2023, pp. 11121–11128
  * [84] Michael Zhang, Khaled Kamal Saab, Michael Poli, Tri Dao, Karan Goel and Christopher Re  “Effectively Modeling Time Series with Simple Discrete State Spaces”  In _The Eleventh International Conference on Learning Representations_ , 2023  URL: https://openreview.net/forum?id=2EpjkjzdCAa
  * [85] T. Zhang, Yizhuo Zhang, Wei Cao, J. Bian, Xiaohan Yi, Shun Zheng and Jian Li  “Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structures”  In _arXiv preprint arXiv:2207.01186_ , 2022
  * [86] Yunhao Zhang and Junchi Yan  “Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting”  In _The eleventh international conference on learning representations_ , 2023
  * [87] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong and Wancai Zhang  “Informer: Beyond efficient transformer for long sequence time-series forecasting”  In _Proceedings of the AAAI conference on artificial intelligence_ 35, 2021, pp. 11106–11115
  * [88] Tian Zhou, Ziqing Ma, Qingsong Wen, Liang Sun, Tao Yao, Wotao Yin and Rong Jin  “Film: Frequency improved legendre memory model for long-term time series forecasting”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 12677–12690
  * [89] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun and Rong Jin  “Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting”  In _International conference on machine learning_ , 2022, pp. 27268–27286  PMLR

##  Appendix A Background

###  A.1 1D Space State Models

1D Space State Models (SSMs) are linear time-invariant systems that map input
sequence $x(t)\in\mathbb{R}^{L}\mapsto y(t)\in\mathbb{R}^{L}$ [2]. SSMs use a
latent state $h(t)\in\mathbb{R}^{N\times L}$, transition parameter
$\mathbf{A}\in\mathbb{R}^{N\times N}$, and projection parameters
$\mathbf{B}\in\mathbb{R}^{N\times 1},\mathbf{C}\in\mathbb{R}^{1\times N}$ to
model the input and output as:

$\displaystyle h^{\prime}(t)=\mathbf{A}\>h(t)+\mathbf{B}\>x(t),\qquad\qquad
y(t)=\mathbf{C}\>h(t).$ (26)

Most existing SSMs [26, 23, 8], first discretize the signals
$\mathbf{A},\mathbf{B},$ and $\mathbf{C}$. That is, using a parameter
$\bm{\Delta}$ and zero-order hold, the discretized formulation is defined as:

$\displaystyle
h_{t}=\bar{\mathbf{A}}\>h_{t-1}+\bar{\mathbf{B}}\>x_{t},\qquad\qquad
y_{t}=\mathbf{C}\>h_{t},$ (27)

where $\bar{\mathbf{A}}=\exp\left(\bm{\Delta}\mathbf{A}\right)$ and
$\bar{\mathbf{B}}=\left(\bm{\Delta}\mathbf{A}\right)^{-1}\left(\exp\left(\bm{\Delta}\mathbf{A}-I\right)\right)\>.\>\bm{\Delta}\mathbf{B}$.
[24] show that discrete SSMs can be interpreted as both convolutions and
recurrent networks: i.e.,

$\displaystyle\bar{\mathbf{K}}=\left({\mathbf{C}}\bar{\mathbf{B}},{\mathbf{C}}\bar{\mathbf{A}}\bar{\mathbf{B}},\dots,{\mathbf{C}}\bar{\mathbf{A}}^{L-1}\bar{\mathbf{B}}\right),$
$\displaystyle y=x\ast\bar{\mathbf{K}},$ (28)

which makes their training and inference very efficient as a convolution and
recurrent model, respectively.

###  A.2 Data Dependency

Above discrete SSMs are based on data-independent parameters. That is,
parameters $\bm{\Delta},$ $\bar{\mathbf{A}}$, $\bar{\mathbf{B}}$, and
${\mathbf{C}}$ are time invariant and are the same for any input. [23] argue
that this time invariance has the cost of limiting SSMs effectiveness in
compressing context into a smaller state [23]. To overcome this challenge,
they present a selective SSMs (S6) block that effectively selects relevant
context by enabling dependence of the parameters $\bar{\mathbf{B}}$,
$\bar{\mathbf{C}}$, and $\bm{\Delta}$ on the input $x_{t}$, i.e.:

$\displaystyle\bar{\mathbf{B}}_{t}=\texttt{Linear}_{\textbf{B}}(x_{t})$ (29)
$\displaystyle\bar{\mathbf{C}}_{t}=\texttt{Linear}_{\textbf{C}}(x_{t})$ (30)
$\displaystyle\bm{\Delta}_{t}=\texttt{Softplus}\left(\texttt{Linear}_{\bm{\Delta}}(x_{t})\right),$
(31)

where $\texttt{Linear}(.)$ is a linear projection and
$\texttt{Softplus}(.)=\log(1+\exp(.))$. This data dependency comes at the cost
of efficiency as the model cannot be trained as a convolution. To overcome
this challenge, [23] show that the linear recurrence in Equation 1 can be
formulated as an associative scan [53], which accepts efficient parallel
algorithms.

##  Appendix B Additional Related Work

Classical Approach. Modeling time series data is a long-standing problem and
has attracted much attention during the past 60 years. There have been several
mathematical models to capture the time series traits like exponential
smoothing[75], autoregressive integrated moving average (ARIMA) [5], SARIMA
[9], Box-Jenkins method [10], and more recently state-space models [30, 2].
Despite their more interpretability, these methods usually fail to capture
non-linear dependencies and also often require manually analyzing time series
features (e.g., trend or seasonality), resulting in lack of generalizability.

Recurrent and Deep State Space Models. Another group of relevant studies to
ours is deep sequence models. A common class of architectures for sequence
modeling are recurrent neural networks such as like GRUs [15], DeepAR [63],
LSTMs [33]. The main drawback of RNNs is their potential for
vanishing/exploding gradients and also their slow training. Recently, linear
attention methods with fast training attracted attention [81, 38, 65]. [38]
show that these methods have recurrent formulation and can be fast in
inference.

Recently, deep state space models have attracted much attention as the
alternative of Transformers [71], due to their fast training and inference
[24]. These methods are the combination of traditional SSMs with deep neural
networks by directly parameterizing the layers of a neural network with
multiple linear SSMs, and overcome common recurrent training drawbacks by
leveraging the convolutional view of SSMs [24, 25, 28, 26, 66]. Recently, [23]
present a new formulation of deep SSMs by allowing the parameters to be the
function of inputs. This architecture shows promissing potential in various
domains like NLP [23], vision [52, 50, 8], graphs [7], DNA modeling [23, 64].

All the above methods are design for 1D data, meaning that the states depends
on one variable. There are, however, a few studies that uses 2D SSMs in deep
learning settings. S4ND [55] uses continuous signals to model images. These
methods not only consider two separate SSM for the axes, but it also directly
treat the system as a continuous system without discretization step.
Furthermore, S4ND has data-independent parameters. Another similar approach is
2DSSM [4], that models images as discrete signals. That is, the initial SSM
model is discrete and again there is a lack of discretization step, which is
important for time series as we discussed earlier. Also, their method again is
based on data-independent parameters. Both S4ND and 2DSSM can be computed as a
convolution. We, however, present a new scanning technique for fast training
of 2D SSMs, even with input-dependent parameters.

Other methods. Transformer-based models have attracted much attention over
recent years for multivariate time series forecasting, when modeling the
complex relationships of co-variates or along the time dimension is required
[89, 39, 86, 82, 87, 46, 79, 35, 56]. Several studies have focused on
designing more efficient and effective attentions with using special traits of
time series [76]. Some other studies have focused on extracting long-term
information for better forecasting [56, 88]. In addition to transformers,
linear models also have shown promising results [77, 12]. For example, [12]
present TSMixer, an all-MLP architecture for time series forecasting, with
promising performance. Due to the expressive power of our 2D SSM, these linear
methods sometimes can be viewed as a special case of 2D SSMs. Recently,
convolution-based models for time series have shown promising results [51].
These methods by using global kernels enhance the global receptive field. Our
data-independent formulation of Chimera is connected to this line of work as
it can be written as a global convolution.

##  Appendix C Details of the Discretization

Given PDE with initial condition $h(0,0)=0$:

$\displaystyle\frac{\partial}{\partial
t^{(1)}}h^{(1)}\left(t^{(1)},t^{(2)}\right)$
$\displaystyle=\left(\mathbf{A}_{1}h^{(1)}\left(t^{(1)},t^{(2)}\right),\mathbf{A}_{2}h^{(2)}\left(t^{(1)},t^{(2)}\right)\right)+\mathbf{B}_{1}\mathbf{x}\left(t^{(1)},t^{(2)}\right),$
(32) $\displaystyle\frac{\partial}{\partial
t^{(1)}}h^{(2)}\left(t^{(1)},t^{(2)}\right)$
$\displaystyle=\left(\mathbf{A}_{1}h^{(1)}\left(t^{(1)},t^{(2)}\right),\mathbf{A}_{2}h^{(2)}\left(t^{(1)},t^{(2)}\right)\right)+\mathbf{B}_{1}\mathbf{x}\left(t^{(1)},t^{(2)}\right),$
(33) $\displaystyle\frac{\partial}{\partial
t^{(2)}}h^{(1)}\left(t^{(1)},t^{(2)}\right)$
$\displaystyle=\left(\mathbf{A}_{3}h^{(1)}\left(t^{(1)},t^{(2)}\right),\mathbf{A}_{4}h^{(2)}\left(t^{(1)},t^{(2)}\right)\right)+\mathbf{B}_{2}\mathbf{x}\left(t^{(1)},t^{(2)}\right),$
(34) $\displaystyle\frac{\partial}{\partial
t^{(2)}}h^{(2)}\left(t^{(1)},t^{(2)}\right)$
$\displaystyle=\left(\mathbf{A}_{3}h^{(1)}\left(t^{(1)},t^{(2)}\right),\mathbf{A}_{4}h^{(2)}\left(t^{(1)},t^{(2)}\right)\right)+\mathbf{B}_{2}\mathbf{x}\left(t^{(1)},t^{(2)}\right),$
(35)

over the sampling intervals $[k\Delta t^{(1)},(k+1)\Delta t^{(1)}]$ and
$[\ell\Delta t^{(2)},(\ell+1)\Delta t^{(2)}]$ we have:

$\displaystyle\int_{k\Delta t^{(1)}}^{(k+1)\Delta
t^{(1)}}\frac{\partial}{\partial
t^{(1)}}h^{(1)}\left(t^{(1)},t^{(2)}\right)\,dt^{(1)}$ $\displaystyle=\>$
$\displaystyle\int_{k\Delta t^{(1)}}^{(k+1)\Delta
t^{(1)}}\left(\mathbf{A}_{1}h^{(1)}\left(t^{(1)},t^{(2)}\right)+\mathbf{B}^{(1)}_{1}\mathbf{x}^{(1)}\left(t^{(1)},t^{(2)}\right)\right)\,dt^{(1)}$
(36)

and so:

$\displaystyle\int_{k\Delta t^{(1)}}^{(k+1)\Delta
t^{(1)}}\frac{\partial}{\partial
t^{(1)}}h^{(2)}\left(t^{(1)},t^{(2)}\right)\,dt^{(1)}$ $\displaystyle=\>$
$\displaystyle\int_{k\Delta t^{(1)}}^{(k+1)\Delta
t^{(1)}}\left(\mathbf{A}_{2}h^{(2)}\left(t^{(1)},t^{(2)}\right)+\mathbf{B}^{(2)}_{1}\mathbf{x}^{(2)}\left(t^{(1)},t^{(2)}\right)\right)\,dt^{(1)}$
(37)

Similarly, for the second equation we have:

$\displaystyle\int_{\ell\Delta t^{(2)}}^{(\ell+1)\Delta
t^{(2)}}\frac{\partial}{\partial
t^{(2)}}h^{(1)}\left(t^{(1)},t^{(2)}\right)\,dt^{(2)}$ $\displaystyle=\>$
$\displaystyle\int_{\ell\Delta t^{(2)}}^{(\ell+1)\Delta
t^{(2)}}\left(\mathbf{A}_{3}h^{(1)}\left(t^{(1)},t^{(2)}\right)+\mathbf{B}^{(1)}_{2}\mathbf{x}^{(1)}\left(t^{(1)},t^{(2)}\right)\right)\,dt^{(2)}$
(38)

and so:

$\displaystyle\int_{\ell\Delta t^{(2)}}^{(\ell+1)\Delta
t^{(2)}}\frac{\partial}{\partial
t^{(2)}}h^{(2)}\left(t^{(1)},t^{(2)}\right)\,dt^{(2)}$ $\displaystyle=\>$
$\displaystyle\int_{\ell\Delta t^{(2)}}^{(\ell+1)\Delta
t^{(2)}}\left(\mathbf{A}_{4}h^{(2)}\left(t^{(1)},t^{(2)}\right)+\mathbf{B}^{(2)}_{2}\mathbf{x}^{(2)}\left(t^{(1)},t^{(2)}\right)\right)\,dt^{(2)}$
(39)

Next, the integrals can be simplified as:

$\displaystyle h^{(1)}\left((k+1)\Delta t^{(1)},t^{(2)}\right)$
$\displaystyle=\>$ $\displaystyle e^{\mathbf{A}_{1}\Delta
t^{(1)}}h^{(1)}\left(k\Delta t^{(1)},t^{(2)}\right)+\int_{k\Delta
t^{(1)}}^{(k+1)\Delta t^{(1)}}e^{\mathbf{A}_{1}(t^{(1)}-k\Delta
t^{(1)})}\mathbf{B}^{(1)}_{1}\mathbf{x}^{(1)}\left(t^{(1)},t^{(2)}\right)\,dt^{(1)},$
(40)

and

$\displaystyle h^{(2)}\left((k+1)\Delta t^{(1)},t^{(2)}\right)$
$\displaystyle=\>$ $\displaystyle e^{\mathbf{A}_{2}\Delta
t^{(1)}}h^{(2)}\left(k\Delta t^{(1)},t^{(2)}\right)+\int_{k\Delta
t^{(1)}}^{(k+1)\Delta t^{(1)}}e^{\mathbf{A}_{2}(t^{(1)}-k\Delta
t^{(1)})}\mathbf{B}^{(2)}_{1}\mathbf{x}^{(2)}\left(t^{(1)},t^{(2)}\right)\,dt^{(1)},$
(41)

and similarly for the third and fourth equations we have:

$\displaystyle h^{(1)}\left(t^{(1)},(\ell+1)\Delta t^{(2)}\right)$
$\displaystyle=\>$ $\displaystyle e^{\mathbf{A}_{3}\Delta
t^{(2)}}h^{(1)}\left(t^{(1)},\ell\Delta t^{(2)}\right)+\int_{\ell\Delta
t^{(2)}}^{(\ell+1)\Delta t^{(2)}}e^{\mathbf{A}_{3}(t^{(2)}-\ell\Delta
t^{(2)})}\mathbf{B}_{2}^{(1)}\mathbf{x}^{(1)}\left(t^{(2)},t^{(1)}\right)\,dt^{(2)}$
(42)

and

$\displaystyle h^{(2)}\left(t^{(1)},(\ell+1)\Delta t^{(2)}\right)$
$\displaystyle=\>$ $\displaystyle e^{\mathbf{A}_{4}\Delta
t^{(2)}}h^{(2)}\left(t^{(1)},\ell\Delta t^{(2)}\right)+\int_{\ell\Delta
t^{(2)}}^{(\ell+1)\Delta t^{(2)}}e^{\mathbf{A}_{4}(t^{(2)}-\ell\Delta
t^{(2)})}\mathbf{B}_{2}^{(2)}\mathbf{x}^{(2)}\left(t^{(2)},t^{(1)}\right)\,dt^{(2)}$
(43)

Using ZOH assumption, we have:

$\displaystyle\int_{0}^{\Delta
t^{(1)}}e^{\mathbf{A}_{1}s}\,ds=\mathbf{A}^{(1)^{-1}}\left(e^{\mathbf{A}_{1}\Delta
t^{(1)}}-\mathbf{I}\right)$ $\displaystyle\int_{0}^{\Delta
t^{(1)}}e^{\mathbf{A}_{2}s}\,ds=\mathbf{A}^{(2)^{-1}}\left(e^{\mathbf{A}_{2}\Delta
t^{(1)}}-\mathbf{I}\right)$ (44) $\displaystyle\int_{0}^{\Delta
t^{(2)}}e^{\mathbf{A}_{3}s}\,ds=\mathbf{A}^{(3)^{-1}}\left(e^{\mathbf{A}_{3}\Delta
t^{(2)}}-\mathbf{I}\right)$ (45) $\displaystyle\int_{0}^{\Delta
t^{(2)}}e^{\mathbf{A}_{4}s}\,ds=\mathbf{A}^{(4)^{-1}}\left(e^{\mathbf{A}_{4}\Delta
t^{(2)}}-\mathbf{I}\right)$ (46)

Accordingly, the discretized form is as follows:

$\displaystyle h^{(1)}_{k+1,\ell}=e^{\mathbf{A}_{1}\Delta
t^{(1)}}h^{(1)}_{k,\ell}+\mathbf{A}^{(1)^{-1}}\left(e^{\mathbf{A}_{1}\Delta
t^{(1)}}-\mathbf{I}\right)\mathbf{B}^{(1)}_{1}\mathbf{x}^{(1)}_{k+1,\ell}$
(47) $\displaystyle h^{(2)}_{k+1,\ell}=e^{\mathbf{A}_{2}\Delta
t^{(1)}}h^{(2)}_{k,\ell}+\mathbf{A}^{(2)^{-1}}\left(e^{\mathbf{A}_{2}\Delta
t^{(1)}}-\mathbf{I}\right)\mathbf{B}^{(2)}_{1}\mathbf{x}^{(2)}_{k+1,\ell}$
(48) $\displaystyle h^{(1)}_{k,\ell+1}=e^{\mathbf{A}_{3}\Delta
t^{(2)}}h^{(1)}_{k,\ell}+\mathbf{A}^{(3)^{-1}}\left(e^{\mathbf{A}_{3}\Delta
t^{(2)}}-\mathbf{I}\right)\mathbf{B}^{(1)}_{2}\mathbf{x}^{(1)}_{k,\ell+1}$
(49) $\displaystyle h^{(2)}_{k,\ell+1}=e^{\mathbf{A}_{4}\Delta
t^{(2)}}h^{(2)}_{k,\ell}+\mathbf{A}^{(4)^{-1}}\left(e^{\mathbf{A}_{4}\Delta
t^{(2)}}-\mathbf{I}\right)\mathbf{B}^{(2)}_{2}\mathbf{x}^{(2)}_{k,\ell+1},$
(50)

which means that:

$\displaystyle\bar{\mathbf{A}}_{1}=\exp\left(\mathbf{A}_{1}\Delta_{1}\right),$
(51)
$\displaystyle\bar{\mathbf{A}}_{2}=\exp\left(\mathbf{A}_{2}\Delta_{1}\right),$
(52)
$\displaystyle\bar{\mathbf{A}}_{3}=\exp\left(\mathbf{A}_{3}\Delta_{2}\right),$
(53)
$\displaystyle\bar{\mathbf{A}}_{4}=\exp\left(\mathbf{A}_{4}\Delta_{2}\right),$
(54)

and

$\displaystyle\bar{\mathbf{B}}_{1}=\begin{bmatrix}\mathbf{A}^{(1)^{-1}}\left(e^{\mathbf{A}_{1}\Delta_{1}}-\mathbf{I}\right)\mathbf{B}^{(1)}_{1}\\\
\mathbf{A}^{(2)^{-1}}\left(e^{\mathbf{A}_{2}\Delta_{1}}-\mathbf{I}\right)\mathbf{B}^{(2)}_{1}\end{bmatrix},$
(56)
$\displaystyle\bar{\mathbf{B}}_{2}=\begin{bmatrix}\mathbf{A}^{(3)^{-1}}\left(e^{\mathbf{A}_{3}\Delta_{2}}-\mathbf{I}\right)\mathbf{B}^{(1)}_{2}\\\
\mathbf{A}^{(4)^{-1}}\left(e^{\mathbf{A}_{4}\Delta_{2}}-\mathbf{I}\right)\mathbf{B}^{(2)}_{2}\end{bmatrix}.$
(57)

##  Appendix D Details of the Structure of Transition Matrices

######  Definition D.1 (Companion Matrix).

A matrix $A\in\mathbb{R}^{N\times N}$ has companion form if it can be written
as:

$\displaystyle A=\begin{pmatrix}0&0&\dots&0&a_{1}\\\ 1&0&\dots&0&a_{2}\\\
0&1&\dots&0&a_{3}\\\ \vdots&\vdots&\ddots&\vdots&\vdots\\\
0&0&\dots&0&a_{N_{1}}\\\ 0&0&\dots&1&a_{N}\end{pmatrix}.$ (58)

These matrices can be decompose into a shift and a low-rank matrix. That is:

$\displaystyle A=\begin{pmatrix}0&0&\dots&0&a_{1}\\\ 1&0&\dots&0&a_{2}\\\
0&1&\dots&0&a_{3}\\\ \vdots&\vdots&\ddots&\vdots&\vdots\\\
0&0&\dots&0&a_{N_{1}}\\\ 0&0&\dots&1&a_{N}\end{pmatrix}=\underset{\text{Shift
Matrix}}{\underbrace{\begin{pmatrix}0&0&\dots&0&0\\\ 1&0&\dots&0&0\\\
0&1&\dots&0&0\\\ \vdots&\vdots&\ddots&\vdots&\vdots\\\ 0&0&\dots&0&0\\\
0&0&\dots&1&0\end{pmatrix}}}+\underset{\text{Low-rank
Matrix}}{\underbrace{\begin{pmatrix}0&0&\dots&0&a_{1}\\\ 0&0&\dots&0&a_{2}\\\
0&0&\dots&0&a_{3}\\\ \vdots&\vdots&\ddots&\vdots&\vdots\\\
0&0&\dots&0&a_{N_{1}}\\\ 0&0&\dots&0&a_{N}\end{pmatrix}}}.$ (59)

This formulation can help us to compute the power of $A$ faster in the
convolutional form, as discussed by [84].

##  Appendix E Theoretical Results

###  E.1 Proof of Theorem 3.2

In this part, we want to prove that $\divideontimes$ is associative. This
operator is defined as:

$\displaystyle p\divideontimes q=\begin{pmatrix}p_{1}&p_{2}&p_{3}\\\
p_{4}&p_{5}&p_{6}\end{pmatrix}\divideontimes\begin{pmatrix}q_{1}&q_{2}&q_{3}\\\
q_{4}&q_{5}&q_{6}\end{pmatrix}=\begin{pmatrix}q_{1}\odot p_{1}&q_{2}\odot
p_{2}&q_{1}\otimes p_{3}+q_{2}\otimes p_{6}+q_{3}\\\ q_{4}\odot
p_{4}&q_{5}\odot p_{5}&q_{4}\otimes p_{3}+q_{5}\otimes
p_{6}+q_{6}\end{pmatrix}$

Accordingly, we have:

$\displaystyle(p\divideontimes q)\divideontimes r=\begin{pmatrix}q_{1}\odot
p_{1}&q_{2}\odot p_{2}&q_{1}\otimes p_{3}+q_{2}\otimes p_{6}+q_{3}\\\
q_{4}\odot p_{4}&q_{5}\odot p_{5}&q_{4}\otimes p_{3}+q_{5}\otimes
p_{6}+q_{6}\end{pmatrix}\divideontimes\begin{pmatrix}r_{1}&r_{2}&r_{3}\\\
r_{4}&r_{5}&r_{6}\end{pmatrix},$ (60)

re-using the definition of $\divideontimes$, we have:

$\displaystyle(p\divideontimes q)\divideontimes r$
$\displaystyle=\begin{pmatrix}q_{1}\odot p_{1}&q_{2}\odot p_{2}&q_{1}\otimes
p_{3}+q_{2}\otimes p_{6}+q_{3}\\\ q_{4}\odot p_{4}&q_{5}\odot
p_{5}&q_{4}\otimes p_{3}+q_{5}\otimes
p_{6}+q_{6}\end{pmatrix}\divideontimes\begin{pmatrix}r_{1}&r_{2}&r_{3}\\\
r_{4}&r_{5}&r_{6}\end{pmatrix}$ (61)
$\displaystyle=\begin{pmatrix}r_{1}\odot(q_{1}\odot
p_{1})&r_{2}\odot(q_{2}\odot p_{2})&r_{1}\otimes(q_{1}\otimes
p_{3}+q_{2}\otimes p_{6}+q_{3})+r_{2}\odot(q_{4}\otimes p_{3}+q_{5}\otimes
p_{6}+q_{6})+r_{3}\\\ r_{4}\odot(q_{4}\odot p_{4})&r_{5}\odot(q_{5}\odot
p_{5})&r_{4}\otimes(q_{1}\otimes p_{3}+q_{2}\otimes
p_{6}+q_{3})+r_{4}\otimes(q_{4}\otimes p_{3}+q_{5}\otimes
p_{6}+q_{6})+r_{6}\end{pmatrix}$ (62)

Using the fact that $\odot$ and $\otimes$ are associative, we have:

$\displaystyle(p\divideontimes q)\divideontimes r$
$\displaystyle=\begin{pmatrix}q_{1}\odot p_{1}&q_{2}\odot p_{2}&q_{1}\otimes
p_{3}+q_{2}\otimes p_{6}+q_{3}\\\ q_{4}\odot p_{4}&q_{5}\odot
p_{5}&q_{4}\otimes p_{3}+q_{5}\otimes
p_{6}+q_{6}\end{pmatrix}\divideontimes\begin{pmatrix}r_{1}&r_{2}&r_{3}\\\
r_{4}&r_{5}&r_{6}\end{pmatrix}$ (63)
$\displaystyle=\begin{pmatrix}r_{1}\odot(q_{1}\odot
p_{1})&r_{2}\odot(q_{2}\odot p_{2})&r_{1}\otimes(q_{1}\otimes
p_{3}+q_{2}\otimes p_{6}+q_{3})+r_{2}\odot(q_{4}\otimes p_{3}+q_{5}\otimes
p_{6}+q_{6})+r_{3}\\\ r_{4}\odot(q_{4}\odot p_{4})&r_{5}\odot(q_{5}\odot
p_{5})&r_{4}\otimes(q_{1}\otimes p_{3}+q_{2}\otimes
p_{6}+q_{3})+r_{4}\otimes(q_{4}\otimes p_{3}+q_{5}\otimes
p_{6}+q_{6})+r_{6}\end{pmatrix}$ (64)
$\displaystyle=\begin{pmatrix}p_{1}&p_{2}&p_{3}\\\
p_{4}&p_{5}&p_{6}\end{pmatrix}\divideontimes\begin{pmatrix}r_{1}\odot
q_{1}&r_{2}\odot q_{2}&r_{1}\otimes q_{3}+r_{2}\otimes q_{6}+r_{3}\\\
r_{4}\odot q_{4}&r_{5}\odot q_{5}&r_{4}\otimes q_{3}+r_{5}\otimes
q_{6}+r_{6}\end{pmatrix}$ (65) $\displaystyle=p\divideontimes(q\divideontimes
r),$ (66)

which proves the theorem.

###  E.2 Proof of Theorem 3.3

For each $v,t$, we can pre-compute $\mathbf{B}_{1}\mathbf{x}_{v,t}$ and
$\mathbf{B}_{2}\mathbf{x}_{v,t+1}$. Accordingly, all the following parameters
are pre-computed:

$\displaystyle
c^{(i,j,k,\ell)}_{v,t}=\begin{pmatrix}\mathbf{A}_{1}&\mathbf{A}_{2}&\mathbf{B}_{1}\mathbf{x}_{v+i,t+j}\\\
\mathbf{A}_{3}&\mathbf{A}_{4}&\mathbf{B}_{2}\mathbf{x}_{v+k,t+\ell}\end{pmatrix},$
(67)

for all inputs $\mathbf{x}_{v,t}$ and $i,j,k,\ell\in\\{0,1\\}$. Now, starting
from $\begin{pmatrix}\mathbf{S}^{(1)}_{0,0}\\\
\mathbf{S}^{(2)}_{0,0}\end{pmatrix}=\begin{pmatrix}I&I&0\\\
I&I&0\end{pmatrix}$, we have:

$\displaystyle\begin{pmatrix}\mathbf{S}_{0,1}\\\
\mathbf{S}_{1,0}\end{pmatrix}$ $\displaystyle=\begin{pmatrix}I&I&0\\\
I&I&0\end{pmatrix}\divideontimes\begin{pmatrix}\mathbf{A}_{1}&\mathbf{A}_{2}&\mathbf{B}_{1}\mathbf{x}_{0,1}\\\
\mathbf{A}_{3}&\mathbf{A}_{4}&\mathbf{B}_{2}\mathbf{x}_{1,0}\end{pmatrix}$
(68)
$\displaystyle=\begin{pmatrix}\mathbf{A}_{1}&\mathbf{A}_{2}&\mathbf{B}_{1}\mathbf{x}_{0,1}\\\
\mathbf{A}_{3}&\mathbf{A}_{4}&\mathbf{B}_{2}\mathbf{x}_{1,0}\end{pmatrix}.$
(69)

Re-using operator $\divideontimes$, we have:

$\displaystyle\begin{pmatrix}\mathbf{S}_{1,1}\\\
\mathbf{S}_{1,1}\end{pmatrix}$
$\displaystyle=\begin{pmatrix}\mathbf{S}_{0,1}\\\
\mathbf{S}_{1,0}\end{pmatrix}\divideontimes\underset{\text{Pre-
computed}}{\underbrace{\begin{pmatrix}\mathbf{A}_{1}&\mathbf{A}_{2}&\mathbf{B}_{1}\mathbf{x}_{1,1}\\\
\mathbf{A}_{3}&\mathbf{A}_{4}&\mathbf{B}_{2}\mathbf{x}_{1,1}\end{pmatrix}}}$
(70)
$\displaystyle=\begin{pmatrix}\mathbf{A}_{1}^{2}&\mathbf{A}_{2}^{2}&\mathbf{A}_{1}\mathbf{B}_{1}\mathbf{x}_{0,1}+\mathbf{A}_{2}\mathbf{B}_{2}\mathbf{x}_{1,0}+\mathbf{B}_{1}\mathbf{x}_{1,1}\\\
\mathbf{A}_{3}&\mathbf{A}_{4}^{2}&\mathbf{A}_{3}\mathbf{B}_{1}\mathbf{x}_{0,1}+\mathbf{A}_{4}\mathbf{B}_{2}\mathbf{x}_{1,0}+\mathbf{B}_{2}\mathbf{x}_{1,1}\end{pmatrix}$
(71)

Looking at the third element of each row, these elements are calculating the
hidden states of the recurrent (it can be shown by a straightforward
induction). Accordingly, using this operation, we can recursively calculate
the the outputs of 2D SSM.

However, using Theorem 3.2, we know that this is an associative operation, so
instead of calculating in the recurrent form, we can use parallel pre-fix sum
make this computation parallel, decreasing the sequential operations required
to calculate the hidden states. Note that since our above operation can model
the problem as an parallel prefix, all the algorithms for this problem can be
used to enhance the efficiency.

###  E.3 Proof of Theorem 3.4

To prove this theorem, we need to (1) show that Chimera can recover SpaceTime.
Given this, since SpaceTime is capable of recovering ARIMA [5], exponential
smoothing [75], and controllable linear time–invariant systems [13], we can
conclude that Chimera can also recover these methods. Then, (2) we need to
prove that Chimera can recover SARIMA. This is the model that SpaceTime is not
capable of recovering due to the additional seasonal terms.

Note that using $\mathbf{A}_{2}=\mathbf{A}_{3}=\mathbf{A}_{4}=0$, results in a
1D SSM, with companion matrix as the structure of $\mathbf{A}_{1}$, which is
SpaceTime. Accordingly, SpaceTime is a special case of Chimera when the
recurrence only happen along the time direction.

Note that as discussed in Proposition 3.1, multiplying the discretization
parameter $\Delta$ results in multiplying the steps. Accordingly, using $s$ as
the $\Delta$ in our seasonal module and also letting
$\mathbf{A}_{2}=\mathbf{A}_{3}=\mathbf{A}_{4}=0$ for the seasonal module, we
can model the seasonal terms in the formulation of SAR$(p,q,s)$, meaning that
Chimera can also recover SARIMA which is ARIMA with seasonal terms. Note that
the reason that Chimera is capable of such modeling is that it uses two heads
separately for trend and seasonal terms. Therefore, using different
discretization parameters, each can model their own corresponding terms in
SAR$(p,q,s)$.

###  E.4 Proof of Theorem 3.5

Similar to the above, using $\mathbf{A}_{2}=\mathbf{A}_{3}=0$, our formulation
is equivalent to S4D, while we use diagonal matrices as the structure of
$\mathbf{A}_{1}$. Similarly, as discussed by [8], MambaMixer is equivalent to
S4ND but on patched data. Using our Theorem 5, we can recover linear layers,
resulting in recovering TSMixer by setting $\mathbf{A}_{2}=\mathbf{A}_{3}=0$.

###  E.5 Proof of Theorem 3.6

We in fact will show that restricting Chimera results in recovering 2DSSM [4].
As discussed earlier, this method do not use discretization and initially
starts from a discrete system. Also, it uses input-independent parameters.
Therefore, we use
$\texttt{Linear}_{\Delta_{1}}(.)=\texttt{Linear}_{\Delta_{2}}(.)$ as broadcast
function, and restrict Chimera to have input-independent parameters, then
Chimera can recover 2DSSM [4].

##  Appendix F Experimental Settings

We provide the description of datasets in Table 7.

Table 7: Dataset descriptions. The dataset size is organized in (Train, Validation, Test). Tasks | Dataset | Dim | Series Length | Dataset Size |  Information (Frequency)
---|---|---|---|---|---
| ETTm1, ETTm2 | 7 |  {96, 192, 336, 720} | (34465, 11521, 11521) |  Electricity (15 mins)
| ETTh1, ETTh2 | 7 |  {96, 192, 336, 720} | (8545, 2881, 2881) |  Electricity (15 mins)
Forecasting | Electricity | 321 |  {96, 192, 336, 720} | (18317, 2633, 5261) |  Electricity (Hourly)
(Long-term) | Traffic | 862 |  {96, 192, 336, 720} | (12185, 1757, 3509) |  Transportation (Hourly)
| Weather | 21 |  {96, 192, 336, 720} | (36792, 5271, 10540) |  Weather (10 mins)
| Exchange | 8 |  {96, 192, 336, 720} | (5120, 665, 1422) |  Exchange rate (Daily)
| ILI | 7 |  {24, 36, 48, 60} | (617, 74, 170) |  Illness (Weekly)
| M4-Yearly | 1 | 6 | (23000, 0, 23000) |  Demographic
| M4-Quarterly | 1 | 8 | (24000, 0, 24000) |  Finance
Forecasting | M4-Monthly | 1 | 18 | (48000, 0, 48000) |  Industry
(short-term) | M4-Weakly | 1 | 13 | (359, 0, 359) |  Macro
| M4-Daily | 1 | 14 | (4227, 0, 4227) |  Micro
| M4-Hourly | 1 | 48 | (414, 0, 414) |  Other
|  EthanolConcentration | 3 | 1751 | (261, 0, 263) |  Alcohol Industry
|  FaceDetection | 144 | 62 | (5890, 0, 3524) |  Face (250Hz)
|  Handwriting | 3 | 152 | (150, 0, 850) |  Handwriting
|  Heartbeat | 61 | 405 | (204, 0, 205) |  Heart Beat
Classification |  JapaneseVowels | 12 | 29 | (270, 0, 370) |  Voice
(UEA) |  PEMS-SF | 963 | 144 | (267, 0, 173) |  Transportation (Daily)
|  SelfRegulationSCP1 | 6 | 896 | (268, 0, 293) |  Health (256Hz)
|  SelfRegulationSCP2 | 7 | 1152 | (200, 0, 180) |  Health (256Hz)
|  SpokenArabicDigits | 13 | 93 | (6599, 0, 2199) |  Voice (11025Hz)
|  UWaveGestureLibrary | 3 | 315 | (120, 0, 320) |  Gesture
| SMD | 38 | 100 | (566724, 141681, 708420) |  Server Machine
Anomaly | MSL | 55 | 100 | (44653, 11664, 73729) |  Spacecraft
Detection | SMAP | 25 | 100 | (108146, 27037, 427617) |  Spacecraft
| SWaT | 51 | 100 | (396000, 99000, 449919) |  Infrastructure
| PSM | 25 | 100 | (105984, 26497, 87841) |  Server Machine

###  F.1 Baselines

In our experiments, we use the following baselines:

  * •

Table 8: TSM2 [8], Simba [58], TCN [51], iTransformer [47], RLinear [43],
PatchTST [56], Crossformer [86], TiDE [17], TimesNet [77], DLinear [83],
SCINet [45], FEDformer [89], Stationary [48], Autoformer [79]

  * •

Table 9: ModernTCN [51], PatchTST [56], TimesNet [77], N-HiTS [11], N-BEATS∗
[57], ETSformer [76], LightTS [85], DLinear [82], FEDformer [89], Stationary
[49], Autoformer [79], Pyraformer [46], Informer [87], Reformer [39], LSTM
[33]

  * •

Table 10: LSTM [33], LSTNet [41], LSSL [26], Trans.former [71], Reformer [39],
Informer [87], Pyraformer [46], Autoformer [79], Station. [49], FEDformer
[89], ETSformer [76], Flowformer [78], DLinear [82], LightTS. [85], TimesNet
[77], PatchTST [56], MTCN [51]

For the results of the baselines, we re-use the results reported by [77], or
from the original cited papers.

##  Appendix G Additional Experimental Results

###  G.1 Long Term Forecasting Full Results

The complete results of long term forecasting are reported in Table 8.

Table 8: Long-term forecasting task with different horizons $\mathbf{H}$. The
first, second, and third best results are highlighted in red (bold), orange
(underline), and purple.

|  Chimera |  TSM2 |  Simba |  TCN |  iTransformer |  RLinear |  PatchTST |  Crossformer |  TiDE |  TimesNet |  DLinear |  SCINet |  FEDformer |  Stationary |  Autoformer
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
(ours) |  behrouz2024mambamixer |  patro2024simba |  donghao2024moderntcn |  liu2024itransformer |  li2023revisiting |  nie2023a |  zhang2022crossformer |  das2023longterm |  wu2023timesnet |  zeng2023transformers |  liu2022scinet |  zhou2022fedformer |  liu2022non |  wu2021autoformer
|  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE
ETTm1 |  96 |  0.293 |  0.351 |  0.322 |  - |  0.324 |  0.360 |  0.292 |  0.346 |  0.334 |  0.368 |  0.355 |  0.376 |  0.329 |  0.367 |  0.404 |  0.426 |  0.364 |  0.387 |  0.338 |  0.375 |  0.345 |  0.372 |  0.418 |  0.438 |  0.379 |  0.419 |  0.386 |  0.398 |  0.505 |  0.475
192 |  0.329 |  0.362 |  0.349 |  - |  0.363 |  0.382 |  0.332 |  0.368 |  0.377 |  0.391 |  0.391 |  0.392 |  0.367 |  0.385 |  0.450 |  0.451 |  0.398 |  0.404 |  0.374 |  0.387 |  0.380 |  0.389 |  0.439 |  0.450 |  0.426 |  0.441 |  0.459 |  0.444 |  0.553 |  0.496
336 |  0.352 |  0.383 |  0.366 |  - |  0.395 |  0.405 |  0.365 |  0.391 |  0.426 |  0.420 |  0.424 |  0.415 |  0.399 |  0.410 |  0.532 |  0.515 |  0.428 |  0.425 |  0.410 |  0.411 |  0.413 |  0.413 |  0.490 |  0.485 |  0.445 |  0.459 |  0.495 |  0.464 |  0.621 |  0.537
720 |  0.408 |  0.412 |  0.407 |  - |  0.451 |  0.437 |  0.416 |  0.417 |  0.491 |  0.459 |  0.487 |  0.450 |  0.454 |  0.439 |  0.666 |  0.589 |  0.487 |  0.461 |  0.478 |  0.450 |  0.474 |  0.453 |  0.595 |  0.550 |  0.543 |  0.490 |  0.585 |  0.516 |  0.671 |  0.561
Avg |  0.345 |  0.377 |  0.361 |  - |  0.383 |  0.396 |  0.351 |  0.381 |  0.407 |  0.410 |  0.414 |  0.407 |  0.387 |  0.400 |  0.513 |  0.496 |  0.419 |  0.419 |  0.400 |  0.406 |  0.403 |  0.407 |  0.485 |  0.481 |  0.448 |  0.452 |  0.481 |  0.456 |  0.588 |  0.517
ETTm2 |  96 |  0.168 |  0.261 |  0.173 |  - |  0.177 |  0.263 |  0.166 |  0.256 |  0.180 |  0.264 |  0.182 |  0.265 |  0.175 |  0.259 |  0.287 |  0.366 |  0.207 |  0.305 |  0.187 |  0.267 |  0.193 |  0.292 |  0.286 |  0.377 |  0.203 |  0.287 |  0.192 |  0.274 |  0.255 |  0.339
192 |  0.215 |  0.289 |  0.230 |  - |  0.245 |  0.306 |  0.222 |  0.293 |  0.250 |  0.309 |  0.246 |  0.304 |  0.241 |  0.302 |  0.414 |  0.492 |  0.290 |  0.364 |  0.249 |  0.309 |  0.284 |  0.362 |  0.399 |  0.445 |  0.269 |  0.328 |  0.280 |  0.339 |  0.281 |  0.340
336 |  0.278 |  0.337 |  0.279 |  - |  0.304 |  0.343 |  0.272 |  0.324 |  0.311 |  0.348 |  0.307 |  0.342 |  0.305 |  0.343 |  0.597 |  0.542 |  0.377 |  0.422 |  0.321 |  0.351 |  0.369 |  0.427 |  0.637 |  0.591 |  0.325 |  0.366 |  0.334 |  0.361 |  0.339 |  0.372
720 |  0.341 |  0.378 |  0.388 |  - |  0.400 |  0.399 |  0.351 |  0.381 |  0.412 |  0.407 |  0.407 |  0.398 |  0.402 |  0.400 |  1.730 |  1.042 |  0.558 |  0.524 |  0.408 |  0.403 |  0.554 |  0.522 |  0.960 |  0.735 |  0.421 |  0.415 |  0.417 |  0.413 |  0.433 |  0.432
Avg |  0.250 |  0.316 |  0.267 |  - |  0.271 |  0.327 |  0.253 |  0.314 |  0.288 |  0.332 |  0.286 |  0.327 |  0.281 |  0.326 |  0.757 |  0.610 |  0.358 |  0.404 |  0.291 |  0.333 |  0.350 |  0.401 |  0.571 |  0.537 |  0.305 |  0.349 |  0.306 |  0.347 |  0.327 |  0.371
ETTh1 |  96 |  0.362 |  0.391 |  0.375 |  - |  0.379 |  0.395 |  0.368 |  0.394 |  0.386 |  0.405 |  0.386 |  0.395 |  0.414 |  0.419 |  0.423 |  0.448 |  0.479 |  0.464 |  0.384 |  0.402 |  0.386 |  0.400 |  0.654 |  0.599 |  0.376 |  0.419 |  0.513 |  0.491 |  0.449 |  0.459
192 |  0.398 |  0.415 |  0.398 |  - |  0.432 |  0.424 |  0.405 |  0.413 |  0.441 |  0.436 |  0.437 |  0.424 |  0.460 |  0.445 |  0.471 |  0.474 |  0.525 |  0.492 |  0.436 |  0.429 |  0.437 |  0.432 |  0.719 |  0.631 |  0.420 |  0.448 |  0.534 |  0.504 |  0.500 |  0.482
336 |  0.402 |  0.416 |  0.419 |  - |  0.473 |  0.443 |  0.391 |  0.412 |  0.487 |  0.458 |  0.479 |  0.446 |  0.501 |  0.466 |  0.570 |  0.546 |  0.565 |  0.515 |  0.491 |  0.469 |  0.481 |  0.459 |  0.778 |  0.659 |  0.459 |  0.465 |  0.588 |  0.535 |  0.521 |  0.496
720 |  0.458 |  0.477 |  0.422 |  - |  0.483 |  0.469 |  0.450 |  0.461 |  0.503 |  0.491 |  0.481 |  0.470 |  0.500 |  0.488 |  0.653 |  0.621 |  0.594 |  0.558 |  0.521 |  0.500 |  0.519 |  0.516 |  0.836 |  0.699 |  0.506 |  0.507 |  0.643 |  0.616 |  0.514 |  0.512
Avg |  0.405 |  0.424 |  0.403 |  - |  0.441 |  0.432 |  0.404 |  0.420 |  0.454 |  0.447 |  0.446 |  0.434 |  0.469 |  0.454 |  0.529 |  0.522 |  0.541 |  0.507 |  0.458 |  0.450 |  0.456 |  0.452 |  0.747 |  0.647 |  0.440 |  0.460 |  0.570 |  0.537 |  0.496 |  0.487
ETTh2 |  96 |  0.257 |  0.325 |  0.253 |  - |  0.290 |  0.339 |  0.263 |  0.332 |  0.297 |  0.349 |  0.288 |  0.338 |  0.302 |  0.348 |  0.745 |  0.584 |  0.400 |  0.440 |  0.340 |  0.374 |  0.333 |  0.387 |  0.707 |  0.621 |  0.358 |  0.397 |  0.476 |  0.458 |  0.346 |  0.388
192 |  0.314 |  0.369 |  0.334 |  - |  0.373 |  0.390 |  0.320 |  0.374 |  0.380 |  0.400 |  0.374 |  0.390 |  0.388 |  0.400 |  0.877 |  0.656 |  0.528 |  0.509 |  0.402 |  0.414 |  0.477 |  0.476 |  0.860 |  0.689 |  0.429 |  0.439 |  0.512 |  0.493 |  0.456 |  0.452
336 |  0.316 |  0.381 |  0.347 |  - |  0.376 |  0.406 |  0.313 |  0.376 |  0.428 |  0.432 |  0.415 |  0.426 |  0.426 |  0.433 |  1.043 |  0.731 |  0.643 |  0.571 |  0.452 |  0.452 |  0.594 |  0.541 |  1.000 |  0.744 |  0.496 |  0.487 |  0.552 |  0.551 |  0.482 |  0.486
720 |  0.388 |  0.427 |  0.401 |  - |  0.407 |  0.431 |  0.392 |  0.433 |  0.427 |  0.445 |  0.420 |  0.440 |  0.431 |  0.446 |  1.104 |  0.763 |  0.874 |  0.679 |  0.462 |  0.468 |  0.831 |  0.657 |  1.249 |  0.838 |  0.463 |  0.474 |  0.562 |  0.560 |  0.515 |  0.511
Avg |  0.318 |  0.375 |  0.333 |  - |  0.361 |  0.391 |  0.322 |  0.379 |  0.383 |  0.407 |  0.374 |  0.398 |  0.387 |  0.407 |  0.942 |  0.684 |  0.611 |  0.550 |  0.414 |  0.427 |  0.559 |  0.515 |  0.954 |  0.723 |  0.437 |  0.449 |  0.526 |  0.516 |  0.450 |  0.459
ECL |  96 |  0.132 |  0.234 |  0.142 |  - |  0.165 |  0.253 |  0.129 |  0.226 |  0.148 |  0.240 |  0.201 |  0.281 |  0.181 |  0.270 |  0.219 |  0.314 |  0.237 |  0.329 |  0.168 |  0.272 |  0.197 |  0.282 |  0.247 |  0.345 |  0.193 |  0.308 |  0.169 |  0.273 |  0.201 |  0.317
192 |  0.144 |  0.223 |  0.153 |  - |  0.173 |  0.262 |  0.143 |  0.239 |  0.162 |  0.253 |  0.201 |  0.283 |  0.188 |  0.274 |  0.231 |  0.322 |  0.236 |  0.330 |  0.184 |  0.289 |  0.196 |  0.285 |  0.257 |  0.355 |  0.201 |  0.315 |  0.182 |  0.286 |  0.222 |  0.334
336 |  0.156 |  0.259 |  0.175 |  - |  0.188 |  0.277 |  0.161 |  0.259 |  0.178 |  0.269 |  0.215 |  0.298 |  0.204 |  0.293 |  0.246 |  0.337 |  0.249 |  0.344 |  0.198 |  0.300 |  0.209 |  0.301 |  0.269 |  0.369 |  0.214 |  0.329 |  0.200 |  0.304 |  0.231 |  0.338
720 |  0.184 |  0.280 |  0.209 |  - |  0.214 |  0.305 |  0.191 |  0.286 |  0.225 |  0.317 |  0.257 |  0.331 |  0.246 |  0.324 |  0.280 |  0.363 |  0.284 |  0.373 |  0.220 |  0.320 |  0.245 |  0.333 |  0.299 |  0.390 |  0.246 |  0.355 |  0.222 |  0.321 |  0.254 |  0.361
Avg |  0.154 |  0.249 |  0.169 |  - |  0.185 |  0.274 |  0.156 |  0.253 |  0.178 |  0.270 |  0.219 |  0.298 |  0.205 |  0.290 |  0.244 |  0.334 |  0.251 |  0.344 |  0.192 |  0.295 |  0.212 |  0.300 |  0.268 |  0.365 |  0.214 |  0.327 |  0.193 |  0.296 |  0.227 |  0.338
Exchange |  96 |  0.077 |  0.198 |  0.163 |  - |  - |  - |  0.080 |  0.196 |  0.086 |  0.206 |  0.093 |  0.217 |  0.088 |  0.205 |  0.256 |  0.367 |  0.094 |  0.218 |  0.107 |  0.234 |  0.088 |  0.218 |  0.267 |  0.396 |  0.148 |  0.278 |  0.111 |  0.237 |  0.197 |  0.323
192 |  0.159 |  0.270 |  0.229 |  - |  - |  - |  0.166 |  0.288 |  0.177 |  0.299 |  0.184 |  0.307 |  0.176 |  0.299 |  0.470 |  0.509 |  0.184 |  0.307 |  0.226 |  0.344 |  0.176 |  0.315 |  0.351 |  0.459 |  0.271 |  0.315 |  0.219 |  0.335 |  0.300 |  0.369
336 |  0.311 |  0.344 |  0.383 |  - |  - |  - |  0.307 |  0.398 |  0.331 |  0.417 |  0.351 |  0.432 |  0.301 |  0.397 |  1.268 |  0.883 |  0.349 |  0.431 |  0.367 |  0.448 |  0.313 |  0.427 |  1.324 |  0.853 |  0.460 |  0.427 |  0.421 |  0.476 |  0.509 |  0.524
720 |  0.697 |  0.623 |  0.999 |  - |  - |  - |  0.656 |  0.582 |  0.847 |  0.691 |  0.886 |  0.714 |  0.901 |  0.714 |  1.767 |  1.068 |  0.852 |  0.698 |  0.964 |  0.746 |  0.839 |  0.695 |  1.058 |  0.797 |  1.195 |  0.695 |  1.092 |  0.769 |  1.447 |  0.941
Avg |  0.311 |  0.358 |  0.443 |  - |  - |  - |  0.302 |  0.366 |  0.360 |  0.403 |  0.378 |  0.417 |  0.367 |  0.404 |  0.940 |  0.707 |  0.370 |  0.413 |  0.416 |  0.443 |  0.354 |  0.414 |  0.750 |  0.626 |  0.519 |  0.429 |  0.461 |  0.454 |  0.613 |  0.539
Traffic |  96 |  0.366 |  0.248 |  0.396 |  - |  0.468 |  0.268 |  0.368 |  0.253 |  0.395 |  0.268 |  0.649 |  0.389 |  0.462 |  0.295 |  0.522 |  0.290 |  0.805 |  0.493 |  0.593 |  0.321 |  0.650 |  0.396 |  0.788 |  0.499 |  0.587 |  0.366 |  0.612 |  0.338 |  0.613 |  0.388
192 |  0.394 |  0.292 |  0.408 |  - |  0.413 |  0.317 |  0.379 |  0.261 |  0.417 |  0.276 |  0.601 |  0.366 |  0.466 |  0.296 |  0.530 |  0.293 |  0.756 |  0.474 |  0.617 |  0.336 |  0.598 |  0.370 |  0.789 |  0.505 |  0.604 |  0.373 |  0.613 |  0.340 |  0.616 |  0.382
336 |  0.409 |  0.311 |  0.427 |  - |  0.529 |  0.284 |  0.397 |  0.270 |  0.433 |  0.283 |  0.609 |  0.369 |  0.482 |  0.304 |  0.558 |  0.305 |  0.762 |  0.477 |  0.629 |  0.336 |  0.605 |  0.373 |  0.797 |  0.508 |  0.621 |  0.383 |  0.618 |  0.328 |  0.622 |  0.337
720 |  0.443 |  0.294 |  0.449 |  - |  0.564 |  0.297 |  0.440 |  0.296 |  0.467 |  0.302 |  0.647 |  0.387 |  0.514 |  0.322 |  0.589 |  0.328 |  0.719 |  0.449 |  0.640 |  0.350 |  0.645 |  0.394 |  0.841 |  0.523 |  0.626 |  0.382 |  0.653 |  0.355 |  0.660 |  0.408
Avg |  0.403 |  0.286 |  0.420 |  - |  0.493 |  0.291 |  0.398 |  0.270 |  0.428 |  0.282 |  0.626 |  0.378 |  0.481 |  0.304 |  0.550 |  0.304 |  0.760 |  0.473 |  0.620 |  0.336 |  0.625 |  0.383 |  0.804 |  0.509 |  0.610 |  0.376 |  0.624 |  0.340 |  0.628 |  0.379
Weather |  96 |  0.146 |  0.206 |  0.161 |  - |  0.176 |  0.219 |  0.149 |  0.200 |  0.174 |  0.214 |  0.192 |  0.232 |  0.177 |  0.218 |  0.158 |  0.230 |  0.202 |  0.261 |  0.172 |  0.220 |  0.196 |  0.255 |  0.221 |  0.306 |  0.217 |  0.296 |  0.173 |  0.223 |  0.266 |  0.336
192 |  0.189 |  0.239 |  0.208 |  - |  0.222 |  0.260 |  0.196 |  0.245 |  0.221 |  0.254 |  0.240 |  0.271 |  0.225 |  0.259 |  0.206 |  0.277 |  0.242 |  0.298 |  0.219 |  0.261 |  0.237 |  0.296 |  0.261 |  0.340 |  0.276 |  0.336 |  0.245 |  0.285 |  0.307 |  0.367
336 |  0.244 |  0.281 |  0.252 |  - |  0.275 |  0.297 |  0.238 |  0.277 |  0.278 |  0.296 |  0.292 |  0.307 |  0.278 |  0.297 |  0.272 |  0.335 |  0.287 |  0.335 |  0.280 |  0.306 |  0.283 |  0.335 |  0.309 |  0.378 |  0.339 |  0.380 |  0.321 |  0.338 |  0.359 |  0.395
720 |  0.297 |  0.309 |  0.337 |  - |  0.350 |  0.349 |  0.314 |  0.334 |  0.358 |  0.347 |  0.364 |  0.353 |  0.354 |  0.348 |  0.398 |  0.418 |  0.351 |  0.386 |  0.365 |  0.359 |  0.345 |  0.381 |  0.377 |  0.427 |  0.403 |  0.428 |  0.414 |  0.410 |  0.419 |  0.428
Avg |  0.219 |  0.258 |  0.239 |  - |  0.255 |  0.280 |  0.224 |  0.264 |  0.258 |  0.278 |  0.272 |  0.291 |  0.259 |  0.281 |  0.259 |  0.315 |  0.271 |  0.320 |  0.259 |  0.287 |  0.265 |  0.317 |  0.292 |  0.363 |  0.309 |  0.360 |  0.288 |  0.314 |  0.338 |  0.382

###  G.2 Short-Term Forecasting

The complete results of short term forecasting are reported in Table 9.

Table 9: Full results for the short-term forecasting task in the M4 dataset.
$\ast.$ in the Transformers indicates the name of $\ast$former. _Stationary_
means the Non-stationary Transformer.

Models |  Chimera |  ModernTCN |  PatchTST |  TimesNet |  N-HiTS |  N-BEATS∗ |  ETS$\ast$ |  LightTS |  DLinear |  FED$\ast$ |  Stationary |  Auto$\ast$ |  Pyra$\ast$ |  In$\ast$ |  Re$\ast$ |  LSTM
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
(ours) |  donghao2024moderntcn |  nie2023a |  wu2023timesnet |  challu2022n |  oreshkin2019n |  woo2022etsformer |  Zhang2022LessIM |  Zeng2022AreTE |  zhou2022fedformer |  Liu2022NonstationaryTR |  wu2021autoformer |  liu2021pyraformer |  zhou2021informer |  kitaev2020reformer |  Hochreiter1997LongSM
Yearly |  SMAPE |  13.107 |  13.226 |  13.258 |  13.387 |  13.418 |  13.436 |  18.009 |  14.247 |  16.965 |  13.728 |  13.717 |  13.974 |  15.530 |  14.727 |  16.169 |  176.040
MASE |  2.902 |  2.957 |  2.985 |  2.996 |  3.045 |  3.043 |  4.487 |  3.109 |  4.283 |  3.048 |  3.078 |  3.134 |  3.711 |  3.418 |  3.800 |  31.033
OWA |  0.767 |  0.777 |  0.781 |  0.786 |  0.793 |  0.794 |  1.115 |  0.827 |  1.058 |  0.803 |  0.807 |  0.822 |  0.942 |  0.881 |  0.973 |  9.290
Quarterly |  SMAPE |  9.892 |  9.971 |  10.179 |  10.100 |  10.202 |  10.124 |  13.376 |  11.364 |  12.145 |  10.792 |  10.958 |  11.338 |  15.449 |  11.360 |  13.313 |  172.808
MASE |  1.105 |  1.167 |  0.803 |  1.182 |  1.194 |  1.169 |  1.906 |  1.328 |  1.520 |  1.283 |  1.325 |  1.365 |  2.350 |  1.401 |  1.775 |  19.753
OWA |  0.853 |  0.878 |  0.803 |  0.890 |  0.899 |  0.886 |  1.302 |  1.000 |  1.106 |  0.958 |  0.981 |  1.012 |  1.558 |  1.027 |  1.252 |  15.049
Monthly |  SMAPE |  12.549 |  12.556 |  12.641 |  12.670 |  12.791 |  12.677 |  14.588 |  14.014 |  13.514 |  14.260 |  13.917 |  13.958 |  17.642 |  14.062 |  20.128 |  143.237
MASE |  0.914 |  0.917 |  0.930 |  0.933 |  0.969 |  0.937 |  1.368 |  1.053 |  1.037 |  1.102 |  1.097 |  1.103 |  1.913 |  1.141 |  2.614 |  16.551
OWA |  0.864 |  0.866 |  0.876 |  0.878 |  0.899 |  0.880 |  1.149 |  0.981 |  0.956 |  1.012 |  0.998 |  1.002 |  1.511 |  1.024 |  1.927 |  12.747
Others |  SMAPE |  4.685 |  4.715 |  4.946 |  4.891 |  5.061 |  4.925 |  7.267 |  15.880 |  6.709 |  4.954 |  6.302 |  5.485 |  24.786 |  24.460 |  32.491 |  186.282
MASE |  3.007 |  3.107 |  2.985 |  3.302 |  3.216 |  3.391 |  5.240 |  11.434 |  4.953 |  3.264 |  4.064 |  3.865 |  18.581 |  20.960 |  33.355 |  119.294
OWA |  0.983 |  0.986 |  1.044 |  1.035 |  1.040 |  1.053 |  1.591 |  3.474 |  1.487 |  1.036 |  1.304 |  1.187 |  5.538 |  5.013 |  8.679 |  38.411
Weighted |  Average |  SMAPE |  11.618 |  11.698 |  11.807 |  11.829 |  11.927 |  11.851 |  14.718 |  13.525 |  13.639 |  12.840 |  12.780 |  12.909 |  16.987 |  14.086 |  18.200 |  160.031
MASE |  1.528 |  1.556 |  1.590 |  1.585 |  1.613 |  1.599 |  2.408 |  2.111 |  2.095 |  1.701 |  1.756 |  1.771 |  3.265 |  2.718 |  4.223 |  25.788
OWA |  0.827 |  0.838 |  0.851 |  0.851 |  0.861 |  0.855 |  1.172 |  1.051 |  1.051 |  0.918 |  0.930 |  0.939 |  1.480 |  1.230 |  1.775 |  12.642

###  G.3 Classification

The complete results of time series classification are reported in Table 10.

Table 10: Full results for the classification task (accuracy %). We omit
“former” from the names of Transformer-based methods. For all methods, the
standard deviation is less than 0.1%.

Datasets / Models |  LSTM |  LSTNet |  LSSL |  Trans. |  Re. |  In. |  Pyra. |  Auto. |  Station. |  FED. |  /ETS. |  /Flow. |  /DLinear |  /LightTS. |  /TimesNet |  /PatchTST/ |  MTCN/ |  Chimera
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
Hochreiter1997LongSM |  2018Modeling |  gu2022efficiently |  vaswani2017attention |  kitaev2020reformer |  zhou2021informer |  liu2021pyraformer |  wu2021autoformer |  Liu2022NonstationaryTR |  zhou2022fedformer |  woo2022etsformer |  wu2022flowformer |  Zeng2022AreTE |  Zhang2022LessIM |  wu2023timesnet |  nie2023a |  donghao2024moderntcn |  (ours)
EthanolConcentration |  32.3 |  39.9 |  31.1 |  32.7 |  31.9 |  31.6 |  30.8 |  31.6 |  32.7 |  31.2 |  28.1 |  33.8 |  32.6 |  29.7 |  35.7 |  32.8 |  36.3 |  39.8
FaceDetection |  57.7 |  65.7 |  66.7 |  67.3 |  68.6 |  67.0 |  65.7 |  68.4 |  68.0 |  66.0 |  66.3 |  67.6 |  68.0 |  67.5 |  68.6 |  68.3 |  70.8 |  70.4
Handwriting |  15.2 |  25.8 |  24.6 |  32.0 |  27.4 |  32.8 |  29.4 |  36.7 |  31.6 |  28.0 |  32.5 |  33.8 |  27.0 |  26.1 |  32.1 |  29.6 |  30.6 |  32.9
Heartbeat |  72.2 |  77.1 |  72.7 |  76.1 |  77.1 |  80.5 |  75.6 |  74.6 |  73.7 |  73.7 |  71.2 |  77.6 |  75.1 |  75.1 |  78.0 |  74.9 |  77.2 |  81.3
JapaneseVowels |  79.7 |  98.1 |  98.4 |  98.7 |  97.8 |  98.9 |  98.4 |  96.2 |  99.2 |  98.4 |  95.9 |  98.9 |  96.2 |  96.2 |  98.4 |  97.5 |  98.8 |  99.1
PEMS-SF |  39.9 |  86.7 |  86.1 |  82.1 |  82.7 |  81.5 |  83.2 |  82.7 |  87.3 |  80.9 |  86.0 |  83.8 |  75.1 |  88.4 |  89.6 |  89.3 |  89.1 |  89.5
SelfRegulationSCP1 |  68.9 |  84.0 |  90.8 |  92.2 |  90.4 |  90.1 |  88.1 |  84.0 |  89.4 |  88.7 |  89.6 |  92.5 |  87.3 |  89.8 |  91.8 |  90.7 |  93.4 |  93.7
SelfRegulationSCP2 |  46.6 |  52.8 |  52.2 |  53.9 |  56.7 |  53.3 |  53.3 |  50.6 |  57.2 |  54.4 |  55.0 |  56.1 |  50.5 |  51.1 |  57.2 |  57.8 |  60.3 |  59.9
SpokenArabicDigits |  31.9 |  100.0 |  100.0 |  98.4 |  97.0 |  100.0 |  99.6 |  100.0 |  100.0 |  100.0 |  100.0 |  98.8 |  81.4 |  100.0 |  99.0 |  98.3 |  98.7 |  100.0
UWaveGestureLibrary |  41.2 |  87.8 |  85.9 |  85.6 |  85.6 |  85.6 |  83.4 |  85.9 |  87.5 |  85.3 |  85.0 |  86.6 |  82.1 |  80.3 |  85.3 |  85.8 |  86.7 |  86.7
Average Accuracy |  48.6 |  71.8 |  70.9 |  71.9 |  71.5 |  72.1 |  70.8 |  71.1 |  72.7 |  70.7 |  71.0 |  73.0 |  67.5 |  70.4 |  73.6 |  72.5 |  74.2 |  75.3

###  G.4 Anomaly Detection

The complete results of anomaly detection tasks are reported in Table 11.

Table 11: Full results for the anomaly detection task. The P, R and F1 represent the precision, recall and F1-score (%) respectively. A higher value of P, R and F1 indicates a better performance. Datasets |  SMD |  MSL |  SMAP |  SWaT |  PSM |  Avg F1
---|---|---|---|---|---|---
Metrics |  P |  R |  F1 |  P |  R |  F1 |  P |  R |  F1 |  P |  R |  F1 |  P |  R |  F1 |  (%)
LSTM |  Hochreiter1997LongSM |  78.52 |  65.47 |  71.41 |  78.04 |  86.22 |  81.93 |  91.06 |  57.49 |  70.48 |  78.06 |  91.72 |  84.34 |  69.24 |  99.53 |  81.67 |  77.97
Transformer |  vaswani2017attention |  83.58 |  76.13 |  79.56 |  71.57 |  87.37 |  78.68 |  89.37 |  57.12 |  69.70 |  68.84 |  96.53 |  80.37 |  62.75 |  96.56 |  76.07 |  76.88
LogTrans |  2019Enhancing |  83.46 |  70.13 |  76.21 |  73.05 |  87.37 |  79.57 |  89.15 |  57.59 |  69.97 |  68.67 |  97.32 |  80.52 |  63.06 |  98.00 |  76.74 |  76.60
TCN |  Franceschi2019UnsupervisedSR |  84.06 |  79.07 |  81.49 |  75.11 |  82.44 |  78.60 |  86.90 |  59.23 |  70.45 |  76.59 |  95.71 |  85.09 |  54.59 |  99.77 |  70.57 |  77.24
Reformer |  kitaev2020reformer |  82.58 |  69.24 |  75.32 |  85.51 |  83.31 |  84.40 |  90.91 |  57.44 |  70.40 |  72.50 |  96.53 |  82.80 |  59.93 |  95.38 |  73.61 |  77.31
Informer |  zhou2021informer |  86.60 |  77.23 |  81.65 |  81.77 |  86.48 |  84.06 |  90.11 |  57.13 |  69.92 |  70.29 |  96.75 |  81.43 |  64.27 |  96.33 |  77.10 |  78.83
Anomaly∗ |  xu2021anomaly |  88.91 |  82.23 |  85.49 |  79.61 |  87.37 |  83.31 |  91.85 |  58.11 |  71.18 |  72.51 |  97.32 |  83.10 |  68.35 |  94.72 |  79.40 |  80.50
Pyraformer |  liu2021pyraformer |  85.61 |  80.61 |  83.04 |  83.81 |  85.93 |  84.86 |  92.54 |  57.71 |  71.09 |  87.92 |  96.00 |  91.78 |  71.67 |  96.02 |  82.08 |  82.57
Autoformer |  wu2021autoformer |  88.06 |  82.35 |  85.11 |  77.27 |  80.92 |  79.05 |  90.40 |  58.62 |  71.12 |  89.85 |  95.81 |  92.74 |  99.08 |  88.15 |  93.29 |  84.26
LSSL |  gu2022efficiently |  78.51 |  65.32 |  71.31 |  77.55 |  88.18 |  82.53 |  89.43 |  53.43 |  66.90 |  79.05 |  93.72 |  85.76 |  66.02 |  92.93 |  77.20 |  76.74
Stationary |  Liu2022NonstationaryTR |  88.33 |  81.21 |  84.62 |  68.55 |  89.14 |  77.50 |  89.37 |  59.02 |  71.09 |  68.03 |  96.75 |  79.88 |  97.82 |  96.76 |  97.29 |  82.08
DLinear |  Zeng2022AreTE |  83.62 |  71.52 |  77.10 |  84.34 |  85.42 |  84.88 |  92.32 |  55.41 |  69.26 |  80.91 |  95.30 |  87.52 |  98.28 |  89.26 |  93.55 |  82.46
ETSformer |  woo2022etsformer |  87.44 |  79.23 |  83.13 |  85.13 |  84.93 |  85.03 |  92.25 |  55.75 |  69.50 |  90.02 |  80.36 |  84.91 |  99.31 |  85.28 |  91.76 |  82.87
LightTS |  Zhang2022LessIM |  87.10 |  78.42 |  82.53 |  82.40 |  75.78 |  78.95 |  92.58 |  55.27 |  69.21 |  91.98 |  94.72 |  93.33 |  98.37 |  95.97 |  97.15 |  84.23
FEDformer |  zhou2022fedformer |  87.95 |  82.39 |  85.08 |  77.14 |  80.07 |  78.57 |  90.47 |  58.10 |  70.76 |  90.17 |  96.42 |  93.19 |  97.31 |  97.16 |  97.23 |  84.97
TimesNet (I) |  wu2023timesnet |  87.76 |  82.63 |  85.12 |  82.97 |  85.42 |  84.18 |  91.50 |  57.80 |  70.85 |  88.31 |  96.24 |  92.10 |  98.22 |  92.21 |  95.21 |  85.49
TimesNet (R) |  wu2023timesnet |  88.66 |  83.14 |  85.81 |  83.92 |  86.42 |  85.15 |  92.52 |  58.29 |  71.52 |  86.76 |  97.32 |  91.74 |  98.19 |  96.76 |  97.47 |  86.34
CrossFormer |  zhang2022crossformer |  83.6 |  76.61 |  79.70 |  84.68 |  83.71 |  84.19 |  92.04 |  55.37 |  69.14 |  88.49 |  93.48 |  90.92 |  97.16 |  89.73 |  93.30 |  83.45
PatchTST |  nie2023a |  87.42 |  81.65 |  84.44 |  84.07 |  86.23 |  85.14 |  92.43 |  57.51 |  70.91 |  80.70 |  94.93 |  87.24 |  98.87 |  93.99 |  96.37 |  84.82
ModernTCN |  donghao2024moderntcn |  87.86 |  83.85 |  85.81 |  83.94 |  85.93 |  84.92 |  93.17 |  57.69 |  71.26 |  91.83 |  95.98 |  93.86 |  98.09 |  96.38 |  97.23 |  86.62
Chimera |  (ours) |  87.74 |  83.29 |  85.46 |  84.01 |  86.83 |  85.39 |  93.05 |  58.12 |  71.55 |  92.18 |  95.93 |  94.01 |  97.30 |  96.19 |  96.74 |  86.69