# FreqTSF: Time Series Forecasting Via Simulating Frequency Kramer-Kronig
Relations

Rujia Shen
Faculty of Computing
Harbin Institute of Technology
Harbin, Heilongjiang, China
<EMAIL_ADDRESS>
&Liangliang Liu
Faculty of Computing
Harbin Institute of Technology
Harbin, Heilongjiang, China
<EMAIL_ADDRESS>
&Boran Wang
Faculty of Computing
Harbin Institute of Technology
Shenzhen, Guangdong, China
<EMAIL_ADDRESS>
&Yi Guan
Faculty of Computing
Harbin Institute of Technology
Harbin, Heilongjiang, China
<EMAIL_ADDRESS>
&Yang Yang
School of Artificial Intelligence
Changchu University of Science and Technalogy
Changchu, Jilin, China
<EMAIL_ADDRESS>
& Jingchi Jiang
The Artificial Intelligence Institute
Harbin Institute of Technology
Harbin, Heilongjiang, China
<EMAIL_ADDRESS>

###### Abstract

Time series forecasting (TSF) is immensely important in extensive
applications, such as electricity transformation, financial trade, medical
monitoring, and smart agriculture. Although Transformer-based methods can
handle time series data, their ability to predict long-term time series is
limited due to the “anti-order" nature of the self-attention mechanism. To
address this problem, we focus on frequency domain to weaken the impact of
order in TSF and propose the FreqBlock, where we first obtain frequency
representations through the Frequency Transform Module. Subsequently, a newly
designed Frequency Cross Attention is used to obtian enhanced frequency
representations between the real and imaginary parts, thus establishing a link
between the attention mechanism and the inherent Kramer-Kronig relations
(KKRs). Our backbone network, FreqTSF, adopts a residual structure by
concatenating multiple FreqBlocks to simulate KKRs in the frequency domain and
avoid degradation problems. On a theoretical level, we demonstrate that the
proposed two modules can significantly reduce the time and memory complexity
from $\mathcal{O}(L^{2})$ to $\mathcal{O}(L)$ for each FreqBlock computation.
Empirical studies on four benchmark datasets show that FreqTSF achieves an
overall relative MSE reduction of 15% and an overall relative MAE reduction of
11% compared to the state-of-the-art methods. The code will be available soon.

_Keywords_ Time series forecasting $\cdot$ Frequency Transform Module $\cdot$
Frequency Cross Attention $\cdot$ Residual structure $\cdot$ Frequency
representations

##  1 Introduction

Time series forecasting (TSF) has become increasingly widespread in real-world
applications, such as energy (Alvarez et al., 2010; Mohammed and Al-Bazi,
2022), finance (Rime et al., 2010; Rodrigues et al., 2020), medical treatment
(Li et al., 2021; Khadem et al., 2023), and agriculture (Saini et al., 2020).
Due to its significant practical value in production and daily life, TSF has
received considerable attention.

Significantly benefiting from the robust nonlinear modeling capabilities of
deep models, numerous methods have emerged to analyze the complex variations
in real-world time series data. Among these methods, approaches based on
recurrent neural networks (RNNs) have achieved remarkable results (Medsker and
Jain, 2001; Bengio et al., 1994; Hochreiter and Schmidhuber, 1997; Chung et
al., 2014; Gu et al., 2021). However, due to problems such as the vanishing or
exploding gradient (Pascanu et al., 2013), RNNs often face performance
limitations, which render them unable to capture long-term dependencies.
Inspired by the recent successes in NLP and CV (Vaswani et al., 2017; Kenton
and Toutanova, 2019; Dosovitskiy et al., 2020; Rao et al., 2021), Transformer
has been introduced to TSF, aiming to address gradient-related issues and
capture long-term dependencies (Kitaev et al., 2019; Zhou et al., 2021; Liu et
al., 2021, 2022). Unfortunately, Transformer methods fail to capture
regularities in the temporal dimension due to the “anti-order" nature of self-
attention (Zeng et al., 2023; Yildirim and Akon, 2023). Additionally, finding
reliable dependencies directly from scattered time points is challenging for
attention mechanisms because temporal dependencies may be deeply hidden in
complex temporal patterns (Wu et al., 2021). Some approaches tend to use the
simplest multilayer perceptron (MLP) to avoid problems caused by attention
(Zeng et al., 2023; Zhang et al., 2022). While there are numerous approaches
(Wu et al., 2021; Zhou et al., 2022; Woo et al., 2022; Wu et al., 2022) that
currently shift the focus from the time domain to the frequency domain, they
typically exploit attention mechanisms after Fast Fourier Transforms (FFT),
thereby overlooking intra- and inter-variable variations. Here, as depicted in
Figure 1, intra-variable variations refer to the changes within each variable
over time, while inter-variable variations refer to the mutual influence
between different variables.

Figure 1: Intra- and inter-variable variations.

In this paper, we adopt two approaches to capture the long-term dependencies
of time series data and overcome the problems of context information loss
caused by the “anti-order" nature of attention mechanisms. Firstly, inspired
by the significant advantages of frequency-domain methods in TSF (Wu et al.,
2021; Zhou et al., 2022; Woo et al., 2022; Wu et al., 2022), we shift our
perspective to the frequency domain to handle intricate temporal patterns and
weaken the impact of order in TSF, which constitute the inherent mechanisms of
long-term dependencies in time series data. Recognizing that real-world time
series often exhibit multi-periodicity, such as daily and annual variations in
electricity transformations and regular weekly and quarterly changes in
exchange rates, we use the Short-Time Fourier Transform (STFT) with predefined
windows of different sizes. STFT decouples overlapping and interacting
periodic signals from the time domain into the frequency domain for better
deal with intra-variable and inter-variable variations. Our second approach is
to make the Transformer suitable for feature extraction in the frequency
domain. We introduce Kramer-Kronig relations (KKRs), which are mathematical
mechanisms in the frequency domain, into the attention mechanism. We redesign
the attention-based structure to extract real and imaginary parts, which
simulates KKRs in the frequency domain. Relevant experiments have confirmed
that these two representations contribute to TSF tasks. By combining these two
approaches, we propose FreqTSF as a new model for TSF.

Our contributions are four-fold:

  * •

We propose a time series forecasting framework called FreqTSF, formed by
residual connections of FreqBlocks, to better capture the global KKR
properties of time series in the frequency domain.

  * •

We introduce the Frequency Cross Attention into the FreqBlock structure to
capture enhanced mechanisms within time series through frequency domain
mapping. The Frequency Cross Attention builds the ability to capture the
cross-temporal interactions of time series signals.

  * •

In particular, we effectively reduce the time complexity and memory complexity
from $\mathcal{O}(L^{2})$ to $\mathcal{O}(L)$ for each FreqBlock computation,
as verified both theoretically and empirically.

  * •

Experimental results show that FreqTSF consistently achieves the state-of-the-
art TSF performance with around 15% MSE reduction.

##  2 Related Work

In this section, related work includes the problem statement (Section 2.1),
the summarization of TSF methods (Section 2.2), the standard method for
transforming time domain information into frequency domain information - the
Short Time Fourier Transform ( Section 2.3), and Kramers-Kronig relations
between the real and imaginary parts of causal systems in the frequency domain
(Section 2.4), which form the theoretical foundation of our work.

###  2.1 Problem Statement

Given data points
$X=\\{x^{t}_{1},...,x^{t}_{D}\\}^{L}_{t=1}\in\mathbb{R}^{D\times L}$ in a
lookback window of time series as input, where $L$ is the size of the window,
$D\geq 1$ is the number of variables, and $x^{t}_{i}$ is the value of the
$i$-th variable at the $t$-th time step. The task of TSF is to predict the
forecasting horizon
$\hat{X}=\\{\hat{x}^{t}_{1},...,\hat{x}^{t}_{D}\\}^{L+T}_{t=L+1}\in\mathbb{R}^{D\times
T}$. In this paper, our research is based on multi-step TSF, i.e., $T>1$.

###  2.2 Time Series Forecasting

Data-driven time series forecasting allows researchers to understand the
evolution of systems without the need to construct precise physical laws (Zhou
et al., 2022). With the development of deep neural networks, RNNs, specialized
for sequence data analysis, have been proposed and achieved remarkable
results. However, due to the gradient problems, RNN (Medsker and Jain, 2001),
LSTM (Hochreiter and Schmidhuber, 1997), GRU (Chung et al., 2014) and LSSL (Gu
et al., 2021) still need help adapting to long-term time series forecasting
tasks.

Transformer has been developed and applied to time series forecasting. As a
core component of Transformer, the attention mechanism plays a crucial role in
identifying dependencies between different variables at various time points.
Many improvements have been proposed to make the attention mechanism suitable
for time series forecasting at lower complexity (Kitaev et al., 2019; Zhou et
al., 2021; Liu et al., 2021, 2022). However, complex temporal patterns may
mask such correlations and are challenging to manifest as significant trends
or periodicities in the time domain (Wu et al., 2021). Therefore, some
researches aim to use the simple multilayer perceptron (MLP) to deal with the
above problems (Zeng et al., 2023; Zhang et al., 2022). Other methods shift
the perspective from time domain to frequency domain, and then discover the
inherent trends or periodicities of time series (Wu et al., 2021; Zhou et al.,
2022; Woo et al., 2022; Wu et al., 2022). However, most of the SOTA models
(Zeng et al., 2023; Zhang et al., 2022; Wu et al., 2022) still process time
series data in the time domain without modeling the potential frequency domain
causal mechanisms of time series data, facing challenges in dealing with
complex temporal variations. More discussions about TSF methods are in the
Appendix.

###  2.3 Short-Time Fourier Transform

A commonly used method for transforming time series data from the time domain
to the frequency domain is the Short-Time Fourier Transform (STFT) (Cvetkovic,
2000; Durak and Arikan, 2003). The fundamental component of STFT, consisting
of multiple windows of different sizes, is the Fourier transform, which can be
accelerated using FFT. STFT initially divides more extended time series into
shorter segments of equal length and then computes the Fourier transform
separately for each of these shorter segments. As the window slides along the
time axis until it reaches the end, the STFT of the time series is computed,
resulting in a two-dimensional representation of the series. Moreover,
different window sizes form multi-scale STFT results. Mathematically, this
process can be expressed as:

$\begin{split}\text{STFT}\\{X\\}(\omega,\tau)=\sum_{t=1}^{L}x^{t}w(t-\tau)e^{-i\omega
t},\end{split}$ (1)

where $w(\tau)$ represents the window function, usually a Hann window, $X$
denotes the time series to be transformed and $\omega$ is the angular
frequency with $i^{2}=-1$. The $\text{STFT}\\{X\\}$ is essentially the Fourier
transform of $x^{t}w(t-\tau)$, representing a function that captures the
variations of the signal concerning both time domain and frequency domain.

###  2.4 Kramers-Kronig Relations

In control theory, a causal system is defined as one in which the output
depends on past and present inputs but not on future ones (Karimi and
Hamilton, 2003). Such a relationship is also consistent with the inherent
generation mechanism of time series data, where the data unfolds gradually
over time, and future data points do not precede those with a time delay.
Therefore, in this study, we approximate time series data as a causal system
for analysis. In the following, we will elucidate a theoretical constraint on
a causal system in the frequency domain, specifically regarding the
interdependence of the real and imaginary parts in terms of Kramers-Kronig
relations (King, 2009). The constraints on the relationship between the real
and imaginary parts can guide the design and training of FreqTSF, ensuring the
mapping of the time domain to the frequency domain without altering the
intrinsic mechanisms.

###### Theorem 1

If $X(t)$ is a causal time function with respect to time $t$, i.e. ,
$X(t)=\begin{cases}0,&t<0\\\ X(t),&t\geq 0\end{cases}$, and does not contain
the continuous-time unit impulse function $\delta(t)=\begin{cases}0,&t\neq
0\\\ \infty,&t=0\end{cases}$ and derivatives at $t=0$. Suppose the Fourier
transform $\mathcal{F}$ of $X(t)$ is represented in the form of the real part
$Re$ and the imaginary part $Im$, i.e., $X(t)\leftrightarrow
F(\omega)=Re(\omega)+iIm(\omega)$. Then, we have:

$\begin{split}&Re(\omega)=\frac{1}{\pi}\int_{-\infty}^{\infty}\frac{Im(\sigma)}{\omega-\sigma}d\sigma,\\\
&Im(\omega)=-\frac{1}{\pi}\int_{-\infty}^{\infty}\frac{Re(\sigma)}{\omega-\sigma}d\sigma.\end{split}$
(2)

The proof for Theorem 1 is shown in the Appendix. The Equation 2 (Cizek, 1970;
Benitez et al., 2001) states that for any causal time function, the real and
imaginary parts of its Fourier transform satisfy the KKRs. These relations
represent bidirectional mathematical connections between the real and
imaginary parts of complex functions. From the Equation 2, it can be
intuitively observed that the representations of the real and imaginary parts
exhibit periodic characteristics based on $\omega$. In addition, the real and
imaginary parts themselves can reflect the trend characteristics of time
series, making the expression of the two parts based on KKRs conducive to
effective forecasting.

Furthermore, in the frequency domain, we can obtain a multi-scale
representation and multi-frequency components of time series through the STFT.
This transform not only improves the computational efficiency but also
effectively reduces the computational cost of time series analysis (Xu et al.,
2020). However, the significance of KKRs formed by the real and imaginary
parts in the frequency domain, which characterize the properties of time
series variables in the frequency domain, has not been recognized. To the best
of our knowledge, our proposed method is the first work to achieve TSF by
utilizing real and imaginary parts of KKRs in the frequency domain to improve
the accuracy and reliability of the model.

Figure 2: FreqTSF: a structure composed of residual connections.

##  3 FreqTSF

In this section, we propose the FreqTSF, depicted in Figure 2, with a modular
architecture to model the KKRs of time series in the frequency domain derived
from different variables. More details about the overview of the framework are
in the Appendix. To capture intra-variable and inter-variable variations,
shown in Figure 1, we design a set of residual-connected FreqBlocks within the
FreqTSF, which adopt the Frequency Transform Module, followed by the Frequency
Cross Attention, to adaptively allocate attention weights to the real and
imaginary parts of the intra-variable variations. Simultaneously, it also
integrates inter-variable information through a parameter-efficient inception
block.

###  3.1 Frequency Transform Module

The Frequency Transform Module transforms the input into frequency components
using the STFT:

$\begin{split}C_{i,p,k}=\text{STFT}\\{X_{i}\\}(\omega_{k},\tau_{p})=\sum_{t=1}^{L}x_{i}^{t}w_{p}(t-\tau_{p})e^{-i\omega_{k}t},\end{split}$
(3)

where $X_{i}$ is the $i$-th time series, $w_{p}(\tau_{p})$ represents the
$p$-th window function with center $\tau_{p}$ (totally $P$ window functions
with the set of window functions denoted as $W=\\{w_{p}\\}_{p=1...P}$), and
$\omega_{k}$ is the $k$-th angular frequency. $C_{i,p,k}$ denotes the $k$-th
result of the STFT using the $p$-th predefined window function for the $i$-th
time series (including the real and imaginary parts). The Equation 3 is
further decomposed into the real and imaginary parts based on
$e^{-i\omega_{k}t}=\cos(-\omega_{k}t)+i\sin(-\omega_{k}t)=\cos(\omega_{k}t)-i\sin(\omega_{k}t)$
as follows:

$\begin{split}C^{\text{Re}}_{i,p,k}&=\sum_{t=1}^{L}x_{i}^{t}w_{p}(t-\tau_{p})\cos\omega_{k}t,\\\
C^{\text{Im}}_{i,p,k}&=-\sum_{t=1}^{L}x_{i}^{t}w_{p}(t-\tau_{p})\sin\omega_{k}t,\end{split}$
(4)

where $C^{\text{Re}}_{i,p,k}$ and $C^{\text{Im}}_{i,p,k}$ represent the $k$-th
real and imaginary components, respectively, of the $i$-th time series
obtained by decomposition using the $p$-th window. Considering the sparsity of
the frequency domain, we select only the top-$M$ amplitude values to reduce
the computational burden of the attention mechanism. Then we obtain the most
significant real components $C^{\text{Re}}_{i,p,1},...,C^{\text{Re}}_{i,p,M}$
and imaginary components $C^{\text{Im}}_{i,p,1},...,C^{\text{Im}}_{i,p,M}$,
where $M$ is a hyperparameter. Additionally, we concatenate the two parts
obtained through all $P$ windows to get $\text{Re}\in\mathbb{R}^{D\times
P\times M}$ and $\text{Im}\in\mathbb{R}^{D\times P\times M}$ with
$\text{Amp}(\cdot)$ denoting the amplitude values:

$\begin{split}&\\{C^{\text{Re}}_{i,p,1},...,C^{\text{Re}}_{i,p,M}\\},\\{C^{\text{Im}}_{i,p,1},...,C^{\text{Im}}_{i,p,M}\\}=\underset{k\in\\{1,...,M\\}}{\arg\text{Top-M}}(\text{Amp}(C^{\text{Re}}_{i,p,k},C^{\text{Im}}_{i,p,k})),\\\
&\text{Re}=\underset{i\in\\{1,...,D\\},p\in\\{1,...,P\\},m\in\\{1,...,M\\}}{\text{Concatenate}}(C^{\text{Re}}_{i,p,m}),\\\
&\text{Im}=\underset{i\in\\{1,...,D\\},p\in\\{1,...,P\\},m\in\\{1,...,M\\}}{\text{Concatenate}}(C^{\text{Im}}_{i,p,m}).\end{split}$
(5)

###  3.2 Frequency Cross Attention

Based on the Theorem 1, we can obtain the corresponding imaginary part from
the real part based on the KKRs, and vice versa. The key idea is to compute an
importance score between the real and imaginary parts of each variable, which
can characterize the KKRs. Therefore, we design the Frequency Cross Attention,
which includes two parts, $B^{\text{Re}}$ and $B^{\text{Im}}$, where
$B^{\text{Re}}$ characterizes the ability of the real part to preserve the
imaginary part, while $B^{\text{Im}}$ establishes the ability of the imaginary
part to transform into the real part, as illustrated in Figure 2 right.

We consider the above establishments as extensions of the relationships in the
frequency domain. Therefore, to learn the correct representation of the real
part, we need to enrich the real part Re with the information from the
imaginary part Im, resulting in the KKR-compliant real part
$\tilde{\text{Re}}$ and vice versa. To allow the model to jointly attend to
information from different representation subspaces, we use $H$ heads of
cross-attention mechanisms. For the $h$-th head, we use linear mappings to
transform the real part Re into $Q^{\text{Re}}_{h}$, $K^{\text{Re}}_{h}$, and
$V^{\text{Re}}_{h}$, and the imaginary part Im into $Q^{\text{Im}}_{h}$,
$K^{\text{Im}}_{h}$, and $V^{\text{Im}}_{h}$:

$\begin{split}Q_{h}^{\text{Re}}=\text{Re}^{T}W^{\text{Re},Q}_{h}+b^{\text{Re},Q}_{h},&Q_{h}^{\text{Im}}=\text{Im}^{T}W^{\text{Im},Q}_{h}+b^{\text{Im},Q}_{h},\\\
K_{h}^{\text{Re}}=\text{Re}^{T}W^{\text{Re},K}_{h}+b^{\text{Re},K}_{h},&K_{h}^{\text{Im}}=\text{Im}^{T}W^{\text{Im},K}_{h}+b^{\text{Im},K}_{h},\\\
V_{h}^{\text{Re}}=\text{Re}^{T}W^{\text{Re},V}_{h}+b^{\text{Re},V}_{h},&V_{h}^{\text{Im}}=\text{Im}^{T}W^{\text{Im},V}_{h}+b^{\text{Im},V}_{h}.\end{split}$
(6)

Here, $W^{*,Q/K}_{h}\in\mathbb{R}^{D\times d_{K}}$,
$W^{*,V}_{h}\in\mathbb{R}^{D\times d_{V}}$,
$b^{*,Q/K}_{h}\in\mathbb{R}^{d_{K}}$, and $b^{*,V}_{h}\in\mathbb{R}^{d_{V}}$
are learnable parameters. Therefore,
$Q_{h}^{*},K_{h}^{*}\in\mathbb{R}^{P\times M\times d_{K}}$,
$V_{h}^{*}\in\mathbb{R}^{P\times M\times d_{V}}$. Then, when calculating
$\text{Attn}^{\text{Im}}_{h}$, the representation of the real part Re is
enriched with the imaginary part Im. In contrast, when calculating
$\text{Attn}^{\text{Re}}_{h}$, the representation of the real part Re needs to
be filtered by learnable weights to select information that can establish the
KKRs with the imaginary part Im:

$\begin{split}\text{Attn}^{\text{Im}}_{h}(Q_{h}^{\text{Im}},K_{h}^{\text{Im}},V_{h}^{\text{Re}})&=\text{softmax}(\frac{Q_{h}^{\text{Im}}(K^{\text{Im}}_{h})^{T}}{\sqrt{d_{K}}})V_{h}^{\text{Re}},\\\
\text{Attn}^{\text{Re}}_{h}(Q_{h}^{\text{Re}},K_{h}^{\text{Re}},V_{h}^{\text{Im}})&=\text{softmax}(\frac{Q_{h}^{\text{Re}}(K^{\text{Re}}_{h})^{T}}{\sqrt{d_{K}}})V_{h}^{\text{Im}}.\end{split}$
(7)

After computing the attention weight matrices for all $H$ real attention heads
and imaginary attention heads, we concatenate all
$\text{Attn}^{\text{Im}}_{h}$ and $\text{Attn}^{\text{Re}}_{h}$ and then apply
linear transformations to map them to the real part
$\tilde{\text{Re}}\in\mathbb{R}^{P\times M\times d_{\text{hidden}}}$ and
imaginary part $\tilde{\text{Im}}\in\mathbb{R}^{P\times M\times
d_{\text{hidden}}}$ with implicit KKRs.

$\begin{split}\tilde{\text{Re}}&=\text{MultiHead}(Q^{\text{Im}},K^{\text{Im}},V^{\text{Re}})=\text{Concat}(\text{Attn}^{\text{Im}}_{1},...,\text{Attn}^{\text{Im}}_{H})W^{\text{Re}},\\\
\tilde{\text{Im}}&=\text{MultiHead}(Q^{\text{Re}},K^{\text{Re}},V^{\text{Im}})=\text{Concat}(\text{Attn}^{\text{Re}}_{1},...,\text{Attn}^{\text{Re}}_{H})W^{\text{Im}}.\end{split}$
(8)

Here, $H$ denotes the number of parallel attention heads, and the projections
are parameter matrices $W^{\text{Re}},W^{\text{Im}}\in\mathbb{R}^{Hd_{V}\times
d_{\text{hidden}}}$. Thus, we achieve adaptive attention weight allocation for
the real and imaginary parts within the variables by Frequency Cross Attention
while determining the KKRs.

###  3.3 FreqBlock

As shown in Figure 2, we organize the FreqBlock in a residual way (He et al.,
2016). Specifically, for the length-$L$ input time series
$X\in\mathbb{R}^{D\times L}$, we first apply a normalization block at the very
beginning to deal with the nonstationary characteristics arising from varying
mean and standard deviation. Additionally, a denormalization block is applied
at the very end to transform the model outputs back to the original statistics
(Liu et al., 2022). Subsequently, for the normalized input time series
$X^{\prime}\in\mathbb{R}^{D\times L}$, we utilize the linear embedding layer
Embed to project the column inputs into deep features
$X^{0}\in\mathbb{R}^{E\times L}$. For the $n$th layer of the FreqTSF
(containing a total of $N$ layers) with the input
$X^{n-1}\in\mathbb{R}^{E\times L}$:

$\begin{split}X^{n}=\text{FreqBlock}^{n}(X^{n-1})+X^{n-1}.\end{split}$ (9)

As depicted in Figure 2, in the $n$th FreqBlock, the process consists of two
successive phases in the frequency domain: capturing intra-variable causal
variations and integrating inter-variable information.

####  3.3.1 Capturing Intra-variable Causal Variations

In each FreqBlock, we apply the Frequency Transform Module and the Frequency
Cross Attention in sequence. The deep features $X^{n-1}\in\mathbb{R}^{E\times
L}$ undergo the Frequency Transform Module to obtain enhanced real and
imaginary parts in the frequency domain, which are selectively Top-$M$
enhanced in the frequency domain. Using the Frequency Cross Attention, we
connect the real part $\tilde{\text{Re}}$ and the imaginary part
$\tilde{\text{Im}}$ with the KKRs. This process allows us to efficiently
derive informative causal representations using a parameter-efficient
inception block (Szegedy et al., 2015). The formalized procedure is:

$\begin{split}&\text{Re}^{n},\text{Im}^{n}=\text{Frequency Transform
Module}^{n}(X^{n-1}),\\\
&\tilde{\text{Re}}^{n},\tilde{\text{Im}}^{n}=\text{Frequency Cross
Attention}^{n}(\text{Re}^{n},\text{Im}^{n}),\\\
&\hat{X}^{n}=\text{Inception}^{n}(\text{Linear}^{n}(\text{Concat}(\tilde{\text{Re}}^{n},\tilde{\text{Im}}^{n}))),\end{split}$
(10)

where $\text{Re}^{n},\text{Im}^{n}\in\mathbb{R}^{E\times P\times M}$ are the
enhanced real and imaginary parts obtained by encoding $X^{n-1}$ through the
$n$-th Frequency Transform Module, where $P$ is the number of windows in the
STFT, and $M$ is a predefined parameter as mentioned in Section 3.1.
$\tilde{\text{Re}}^{n},\tilde{\text{Im}}^{n}\in\mathbb{R}^{P\times M\times
d_{\text{hidden}}}$ represent the real and imaginary parts with potential
KKRs.

####  3.3.2 Integrating Inter-variable Information

After the Frequency Cross Attention, we concatenate the real and imaginary
parts with KKRs and then process them through a linear mapping
$\text{Linear}^{n}$ with learnable parameters
$W^{n}\in\mathbb{R}^{d_{\text{hidden}}\times E}$ and $b^{n}\in\mathbb{R}^{E}$.
Subsequently, we pass the result through a parameter-efficient inception block
denoted as $\text{Inception}^{n}(\cdot)$, which incorporates multi-scale
kernels and is widely recognized as a prominent architectural component in
computer vision. The learned representations serve as the output of the $n$-th
FreqBlock, combining inter-variable information that captures correlations
between variables through their mutual influence.

###  3.4 Efficient computation

For each FreqBlock, the time complexity is the sum of the time complexity of
the STFT in the Frequency Transform Module and the attention mechanism in the
Frequency Cross Attention. The time complexity of the STFT with a window size
of $||w_{p}||$ is typically expressed as
$\mathcal{O}(||w_{p}||\log||w_{p}||)$. This is because applying the FFT to
each window requires computational effort of
$\mathcal{O}(||w_{p}||\log||w_{p}||)$, and for a time series data with length
$L$ and variable number $D$, the total time complexity for $P$ windows is
$\mathcal{O}(\sum_{p=1}^{P}LD||w_{p}||\log||w_{p}||)$. Considering that
$W,D,P$ are usually predefined constants and are typically much smaller than
$L$, the time complexity is finally $\mathcal{O}(L)$.

On the other hand, the memory complexity of FreqTSF depends on the
requirements of storing intermediate results in the STFT and attention weight
matrices in the Frequency Cross Attention. For the STFT, the FFT results must
be stored for each window, resulting in a cost of
$\mathcal{O}(\sum_{p=1}^{P}DL)$. Therefore, the memory complexity of the
Frequency Cross Attention is mainly due to the storage of attention weights
and intermediate results. However, since the Frequency Transform Module
freezes the input size for Frequency Cross Attention, i.e.,
$\text{Re}^{n},\text{Im}^{n}\in\mathbb{R}^{E\times P\times M}$, where $E,P,M$
are predefined hyperparameters, the complexity of Frequency Cross Attention in
this work is $\mathcal{O}(EPM)=\mathcal{O}(1)$. Furthermore, for $D,P$, which
are generally predefined constants and typically much smaller than $L$, the
memory complexity of FreqTSF is $\mathcal{O}(L)$.

##  4 Experiments

In this section, we compare the well-acknowledged and advanced models in time
series forecasting, including the RNN-based models: LSTM (Hochreiter and
Schmidhuber, 1997) and LSSL (Gu et al., 2021); CNN-based Model: TimesNet (Wu
et al., 2022); MLP-based models: LightTS (Zhang et al., 2022) and DLinear
(Zeng et al., 2023); Transformer-based models: Reformer (Kitaev et al., 2019),
Informer (Zhou et al., 2021), Pyraformer (Liu et al., 2021), Autoformer (Wu et
al., 2021), FEDformer (Zhou et al., 2022), Non-stationary Transformer (Liu et
al., 2022) and ETSformer (Woo et al., 2022). Overall, 12 baselines are
discussed in the Appendix and included for a comprehensive comparison.

To thoroughly evaluate the performance of FreqTSF, we evaluate the proposed
FreqTSF on two real-world benchmarks and two widely acknowledged time series
simulators, covering mainstream applications in TSF: energy (the ETT dataset
(Zhou et al., 2021)), economics (the Exchange dataset (Lai et al., 2018)),
medicine (the Type 1 Diabetes Mellitus Simulator (T1DMs) (Man et al., 2014)),
and agriculture (the Lintul3 (Shibu et al., 2010)).

Similar to TimesNet (Wu et al., 2022), We follow standard protocol and split
all datasets into training, validation and test set in chronological order by
the ratio of 6:2:2 for the ETT dataset and 7:1:2 for the other datasets. We
use the mean square error (MSE) and mean absolute error (MAE) to evaluate the
accuracy of TSF. Details about datasets and metrics are shown in the Appendix.

Table 1: The MSEs and MAEs of the time series forecasting. The cumulative
number of first predictions for each model is tabulated in the last row, which
serves as a comprehensive metric for evaluating model performance. The best
results are colored.

Models | FreqTSF | TimesNet | ETSformer | LightTS | DLinear | FEDformer | Stationary | Autoformer | Pyraformer | Informer | Reformer | LSSL | LSTM
---|---|---|---|---|---|---|---|---|---|---|---|---|---
Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
ETTh1 | 96 | 0.214 | 0.279 | 0.384 | 0.402 | 0.494 | 0.479 | 0.424 | 0.432 | 0.386 | 0.400 | 0.376 | 0.419 | 0.513 | 0.491 | 0.449 | 0.459 | 0.664 | 0.612 | 0.865 | 0.713 | 0.837 | 0.728 | 0.548 | 0.528 | 1.044 | 0.773
192 | 0.267 | 0.326 | 0.436 | 0.429 | 0.538 | 0.504 | 0.475 | 0.462 | 0.437 | 0.432 | 0.420 | 0.448 | 0.534 | 0.504 | 0.500 | 0.482 | 0.790 | 0.681 | 1.008 | 0.792 | 0.923 | 0.766 | 0.542 | 0.526 | 1.217 | 0.832
336 | 0.338 | 0.369 | 0.491 | 0.469 | 0.574 | 0.521 | 0.518 | 0.488 | 0.481 | 0.459 | 0.459 | 0.465 | 0.588 | 0.535 | 0.521 | 0.496 | 0.891 | 0.738 | 1.107 | 0.809 | 1.097 | 0.835 | 1.298 | 0.942 | 1.259 | 0.841
720 | 0.420 | 0.408 | 0.521 | 0.500 | 0.562 | 0.535 | 0.547 | 0.533 | 0.519 | 0.516 | 0.506 | 0.507 | 0.643 | 0.616 | 0.514 | 0.512 | 0.963 | 0.782 | 1.181 | 0.865 | 1.257 | 0.889 | 0.721 | 0.659 | 1.271 | 0.838
Avg | 0.309 | 0.350 | 0.458 | 0.450 | 0.542 | 0.510 | 0.491 | 0.479 | 0.456 | 0.452 | 0.440 | 0.460 | 0.570 | 0.537 | 0.496 | 0.487 | 0.827 | 0.703 | 1.040 | 0.795 | 1.029 | 0.805 | 0.777 | 0.664 | 1.198 | 0.821
ETTh2 | 96 | 0.312 | 0.360 | 0.340 | 0.374 | 0.340 | 0.391 | 0.397 | 0.437 | 0.333 | 0.387 | 0.358 | 0.397 | 0.476 | 0.458 | 0.346 | 0.388 | 0.645 | 0.597 | 3.755 | 1.525 | 2.626 | 1.317 | 1.616 | 1.036 | 2.522 | 1.278
192 | 0.401 | 0.410 | 0.402 | 0.414 | 0.430 | 0.439 | 0.520 | 0.504 | 0.477 | 0.476 | 0.429 | 0.439 | 0.512 | 0.493 | 0.456 | 0.452 | 0.788 | 0.683 | 5.602 | 1.931 | 11.12 | 2.979 | 2.083 | 1.197 | 3.312 | 1.384
336 | 0.455 | 0.454 | 0.452 | 0.452 | 0.485 | 0.479 | 0.626 | 0.559 | 0.594 | 0.541 | 0.496 | 0.487 | 0.552 | 0.551 | 0.482 | 0.486 | 0.907 | 0.747 | 4.721 | 1.835 | 9.323 | 2.769 | 2.970 | 1.439 | 3.291 | 1.388
720 | 0.487 | 0.480 | 0.462 | 0.468 | 0.500 | 0.497 | 0.863 | 0.672 | 0.831 | 0.657 | 0.463 | 0.474 | 0.562 | 0.560 | 0.515 | 0.511 | 0.963 | 0.783 | 3.647 | 1.625 | 3.874 | 1.697 | 2.576 | 1.363 | 3.257 | 1.357
Avg | 0.414 | 0.426 | 0.414 | 0.427 | 0.439 | 0.452 | 0.602 | 0.543 | 0.559 | 0.515 | 0.437 | 0.449 | 0.526 | 0.516 | 0.450 | 0.459 | 0.826 | 0.703 | 4.431 | 1.729 | 6.736 | 2.191 | 2.311 | 1.259 | 3.095 | 1.352
Exchange | 96 | 0.099 | 0.228 | 0.107 | 0.234 | 0.085 | 0.204 | 0.116 | 0.262 | 0.088 | 0.218 | 0.148 | 0.278 | 0.111 | 0.237 | 0.197 | 0.323 | 1.748 | 1.105 | 0.847 | 0.752 | 1.065 | 0.829 | 0.395 | 0.474 | 1.453 | 1.049
192 | 0.195 | 0.323 | 0.226 | 0.344 | 0.182 | 0.303 | 0.215 | 0.359 | 0.176 | 0.315 | 0.271 | 0.380 | 0.219 | 0.335 | 0.300 | 0.369 | 1.874 | 1.151 | 1.204 | 0.895 | 1.188 | 0.906 | 0.776 | 0.698 | 1.846 | 1.179
336 | 0.355 | 0.434 | 0.367 | 0.448 | 0.348 | 0.428 | 0.377 | 0.466 | 0.313 | 0.427 | 0.460 | 0.500 | 0.421 | 0.476 | 0.509 | 0.524 | 1.943 | 1.172 | 1.672 | 1.036 | 1.357 | 0.976 | 1.029 | 0.797 | 2.136 | 1.231
720 | 1.044 | 0.781 | 0.964 | 0.746 | 1.025 | 0.774 | 0.831 | 0.699 | 0.839 | 0.695 | 1.195 | 0.841 | 1.092 | 0.769 | 1.447 | 0.941 | 2.085 | 1.206 | 2.478 | 1.310 | 1.510 | 1.016 | 2.283 | 1.222 | 2.984 | 1.427
Avg | 0.423 | 0.441 | 0.416 | 0.443 | 0.410 | 0.427 | 0.385 | 0.447 | 0.354 | 0.414 | 0.519 | 0.500 | 0.461 | 0.454 | 0.613 | 0.539 | 1.913 | 1.159 | 1.550 | 0.998 | 1.280 | 0.932 | 1.121 | 0.798 | 2.105 | 1.221
T1DMs | 96 | 0.06 | 0.03 | 0.13 | 0.06 | 0.17 | 0.18 | 0.07 | 0.07 | 0.39 | 0.32 | 0.07 | 0.08 | 0.19 | 0.11 | 0.28 | 0.26 | 0.01 | 0.02 | 0.01 | 0.02 | 0.01 | 0.04 | 0.66 | 0.37 | 0.01 | 0.03
192 | 0.12 | 0.07 | 0.12 | 0.07 | 0.34 | 0.33 | 0.08 | 0.11 | 0.43 | 0.36 | 0.06 | 0.10 | 0.20 | 0.15 | 0.56 | 0.45 | 0.01 | 0.03 | 0.01 | 0.04 | 0.03 | 0.06 | 3.52 | 0.81 | 0.01 | 0.03
336 | 0.15 | 0.09 | 0.10 | 0.07 | 0.49 | 0.44 | 0.07 | 0.11 | 0.45 | 0.39 | 0.06 | 0.12 | 0.16 | 0.14 | 0.33 | 0.31 | 0.01 | 0.03 | 0.01 | 0.04 | 0.07 | 0.10 | 2.07 | 0.61 | 0.01 | 0.03
720 | 0.60 | 0.50 | 0.11 | 0.09 | 0.56 | 0.47 | 0.07 | 0.11 | 0.38 | 0.34 | 0.12 | 0.20 | 0.20 | 0.15 | 0.60 | 0.46 | 0.01 | 0.04 | 0.01 | 0.05 | 0.06 | 0.10 | 2.12 | 0.61 | 0.01 | 0.04
Avg | 0.23 | 0.17 | 0.12 | 0.07 | 0.39 | 0.35 | 0.07 | 0.10 | 0.41 | 0.35 | 0.08 | 0.12 | 0.19 | 0.14 | 0.44 | 0.37 | 0.01 | 0.03 | 0.01 | 0.04 | 0.04 | 0.08 | 2.09 | 0.60 | 0.01 | 0.03
Lintul3 | 7 | 2.81 | 0.42 | 5.72 | 0.77 | 59.94 | 3.40 | 3.69 | 0.77 | 172.22 | 6.42 | 23.32 | 1.99 | 6.73 | 0.82 | 15.32 | 1.62 | 11.43 | 1.53 | 17.78 | 1.79 | 16.85 | 1.65 | 5.99 | 1.19 | 2.40 | 0.71
14 | 2.87 | 0.60 | 12.85 | 0.96 | 43.23 | 2.96 | 13.69 | 1.35 | 366.81 | 9.22 | 10.77 | 1.51 | 19.03 | 1.19 | 75.06 | 3.24 | 8.76 | 1.31 | 18.48 | 1.82 | 28.31 | 2.06 | 8.25 | 1.28 | 2.12 | 0.68
21 | 2.49 | 0.81 | 28.86 | 1.28 | 135.42 | 4.89 | 8.01 | 1.21 | 600.12 | 11.75 | 11.31 | 1.50 | 21.17 | 0.99 | 90.10 | 3.48 | 8.58 | 1.34 | 15.04 | 1.73 | 33.19 | 2.21 | 7.76 | 1.30 | 2.12 | 0.69
30 | 4.56 | 0.99 | 64.22 | 1.85 | 82.08 | 3.58 | 10.40 | 1.34 | 861.78 | 14.02 | 12.48 | 1.52 | 34.37 | 1.18 | 305.86 | 5.92 | 9.56 | 1.43 | 14.82 | 1.78 | 40.82 | 2.39 | 9.44 | 1.25 | 4.33 | 0.94
Avg | 3.18 | 0.71 | 27.91 | 1.21 | 80.16 | 3.70 | 8.94 | 1.16 | 500.23 | 10.35 | 14.47 | 1.63 | 20.33 | 1.05 | 121.59 | 3.57 | 9.58 | 1.40 | 16.53 | 1.78 | 54.79 | 2.08 | 7.86 | 1.26 | 2.74 | 0.76
$1^{st}$ Count | 19 | 5 | 3 | 1 | 6 | 1 | 0 | 0 | 10 | 0 | 0 | 0 | 7

###  4.1 Main Results

For better comparison, we follow the experimental settings of TimesNet (Wu et
al., 2022) and Autoformer (Wu et al., 2021). Expressly, for the ETT, Exchange
and T1DMs datasets, the lookback window length is consistently set to 96, and
the forecasting lengths for both training and evaluation are fixed to be 96,
192, 336, and 720, respectively. Furthermore, in the case of the Lintul3
dataset, the fixed lookback window length is 30, with corresponding
forecasting lengths for both training and evaluation set to 7, 14, 21, and 30.

The results of TSF are presented in Table 1. We count the number of first
predictions achieved by each model across various datasets, which serves as a
comprehensive metric for evaluating model performance. Experimental results
show that FreqTSF achieves the best overall performance across all benchmark
datasets. Specifically, FreqTSF achieves state-of-the-art performance in over
30% of the cases. Compared to TimesNet and Dlinear, FreqTSF exhibits an
overall relative reduction of 15% in MSEs and 11% in MAEs. Notably, on
specific datasets, such as ETTh1, the improvement exceeds 30%. While FreqTSF
performs slightly worse than TimesNet, ETSformer, LightTS, and DLinear on the
Exchange dataset, the results of these five methods are close together,
indicating that FreqTSF remains highly competitive on this dataset. On the
T1DMs dataset, the systematic variations in human blood glucose levels
contribute to a highly regular pattern characterized by a sharp postprandial
rise followed by a gradual decline to a stable baseline. Consequently, all
baselines, including FreqTSF, exhibit remarkably low MSEs and MAEs due to the
predictable nature of these glycemic fluctuations. FreqTSF demonstrates
competitive MAE performance compared to the optimal LSTM on the Lintul3
dataset. However, the relative MSEs are more considerable, possibly due to
abrupt numerical variations in the Lintul3 dataset. FreqTSF, together with the
Frequency Transform Module, may discard such variations as noise, leading to
inaccurate fitting of certain variables. Nevertheless, overall, this confirms
the effectiveness of FreqTSF in TSF. This improvement is attributed to the
design of the Frequency Transform Module and Frequency Cross Attention, which
will be further verified in the subsequent ablation experiments.

###  4.2 Model Efficiency Analysis

To summarize the model performance and efficiency, we conduct a comparison of
parameter scale and GPU memory size on the ETTh1 dataset using five baseline
models that achieved the first MSE and MAE in TSF (i.e. TimesNet, ETSformer,
LightTS, Dlinear, and FEDformer). Among these models, our proposed FreqTSF and
TimesNet exhibit nearly identical parameter sizes and GPU memory, with
parameter sizes of 4.6MB and 4.5MB and GPU memory sizes of 1703MiB for both.
In contrast, the MLP-based models, such as LightTS and Dlinear, have
relatively smaller parameter sizes and GPU memory, averaging around 0.04MB and
1699MiB, respectively. However, these models may struggle to simulate time
series data effectively due to their small parameter size. On the other hand,
ETSformer and FEDformer have parameter sizes and GPU memory similar to MLP-
based models, but they fail to capture the extreme values in time series data.
We attribute the parameter size of our model to the adoption of the inception
block at the end of the FreqBlock, a design choice shared with TimesNet.
Despite the increased parameters, FreqTSF achieves the best overall
performance within an acceptable single-GPU model size.

##  5 Discussion

This study presents, for the first time, an attention-based algorithm for
simulating the KKRs in the frequency domain. Major findings include: (1) our
proposed FreqTSF demonstrates significantly superior performance compared to
baseline TSF algorithms in terms of both accuracy and computational
complexity; (2) through ablation experiments on modules, shown in the
Appendix, the combination of the two proposed modules (the Frequency Transform
Module and the Frequency Cross Attention) leads to a significant improvement
in model performance; (3) ablation experiments on the frequency domain filter
in the Appendix confirm the importance of $M$, which affects the final model
performance; (4) ablation experiments on hyperparameter sensitivity in the
Appendix show the robustness and reproducibility of FreqTSF. However, it is
worth noting that Table 1 presents the performances of FreqTSF compared to
state-of-the-art baselines. Nevertheless, for Exchange and T1DMs, the
performances of FreqTSF do not exhibit statistical significance when compared
to other methods. Therefore, we will perform a more thorough analysis to
determine the reasons for this result.

Looking more closely at FreqTSF’s performance on the Exchange dataset, while
the relative gaps between FreqTSF and the best-performing models range from 7%
to 25%, the absolute gap averages only 0.15. For the T1DMs dataset, while
FreqTSF has larger relative gaps compared to the best-performing models, the
absolute gap ranges from 0.01 to 0.16. We argue that based on these
observations, FreqTSF remains competitive as its absolute gaps with state-of-
the-art baselines are not substantial.

A possible reason for these observations is that the Exchange dataset contains
a significant number of mutations. FreqTSF first transforms time series data
from the time domain to the frequency domain within each FreqBlock,
automatically categorizing mutations as noise and consequently ignoring them.
This disregard may contribute significantly to the poor performance of FreqTSF
on the Exchange dataset. In addition, for the T1DMs dataset, physiological
indicators exhibit periodic variations due to the fixed timing of daily
carbohydrate intake (three meals per day). This simple periodicity and regular
fluctuation is easily captured by Pyraformer, which uses pyramidal attention
to capture hierarchical multi-scale temporal dependencies in the time domain.
Conversely, our frequency domain transformation module uses STFT with windows
of different sizes, which can capture such periodic changes. However, by
transforming the time domain to the frequency domain, the effectiveness in
capturing these simple patterns may be reduced.

##  6 Conclusion, Limitations and Future Works

In this paper, we propose FreqTSF that leverages FreqBlocks connected in a
residual way, resulting in state-of-the-art performance. Each FreqBlock
employs the STFT for the multi-period signal analysis of variables,
facilitating the acquisition of intra-variable time series KKRs from a
frequency perspective. The proposed Frequency Cross Attention fully exploits
the KKRs of the real and imaginary parts in the frequency domain of causal
systems, demonstrating its effectiveness in time series forecasting. Extensive
experiments demonstrate that FreqTSF achieves superior forecasting performance
with minimal resource consumption on four benchmark datasets, outperforming 12
state-of-the-art algorithms. Furthermore, FreqTSF reduces the time and memory
complexity of each FreqBlock computation from $\mathcal{O}(L^{2})$ to
$\mathcal{O}(L)$.

While FreqTSF demonstrates exceptional performance, there are still some
limitations. For example, FreqTSF may incorrectly categorize mutations in
time-series data as noise and overlook the accurate modeling of simple
periodic curves. In addition, FreqTSF currently relies solely on attention to
learn the KKR representation, but the use of neural ordinary differential
equations (ODEs) (Chen et al., 2018) could potentially provide a superior
solution. Therefore, to further improve and overcome the limitations of
FreqTSF, in our future work, we intend to utilize both time- and frequency-
domain information to improve model performance. In addition, we aim to
explore large-scale pre-training methods specifically tailored to time series
applications, using FreqTSF as the underlying architecture. This approach has
the potential to bring significant benefits to a wide range of downstream
tasks.

##  7 Acknowledgments

This study was supported in part by a grant from the National Key Research and
Development Program of China [2021ZD0110900] and the National Natural Science
Foundation of China [62006063].

## References

  * Alvarez et al. [2010] Francisco Martinez Alvarez, Alicia Troncoso, Jose C Riquelme, and Jesus S Aguilar Ruiz.  Energy time series forecasting based on pattern sequence similarity.  _IEEE Transactions on Knowledge and Data Engineering_ , 23(8):1230–1243, 2010.
  * Mohammed and Al-Bazi [2022] Nooriya A Mohammed and Ammar Al-Bazi.  An adaptive backpropagation algorithm for long-term electricity load forecasting.  _Neural Computing and Applications_ , 34(1):477–491, 2022.
  * Rime et al. [2010] Dagfinn Rime, Lucio Sarno, and Elvira Sojli.  Exchange rate forecasting, order flow and macroeconomic information.  _Journal of International Economics_ , 80(1):72–88, 2010.
  * Rodrigues et al. [2020] Paulo Canas Rodrigues, Olushina Olawale Awe, Jonatha Sousa Pimentel, and Rahim Mahmoudvand.  Modelling the behaviour of currency exchange rates with singular spectrum analysis and artificial neural networks.  _Stats_ , 3(2):137–157, 2020.
  * Li et al. [2021] Li Li, Jie Sun, Liemin Ruan, and Qifa Song.  Time-series analysis of continuous glucose monitoring data to predict treatment efficacy in patients with t2dm.  _The Journal of Clinical Endocrinology & Metabolism_, 106(8):2187–2197, 2021.
  * Khadem et al. [2023] Heydar Khadem, Hoda Nemat, Jackie Elliott, and Mohammed Benaissa.  Blood glucose level time series forecasting: Nested deep ensemble learning lag fusion.  _Bioengineering_ , 10(4):487, 2023.
  * Saini et al. [2020] Umesh Saini, Rajesh Kumar, Vipin Jain, and MU Krishnajith.  Univariant time series forecasting of agriculture load by using lstm and gru rnns.  In _2020 IEEE Students Conference on Engineering & Systems (SCES)_, pages 1–6. IEEE, 2020.
  * Medsker and Jain [2001] Larry R Medsker and LC Jain.  Recurrent neural networks.  _Design and Applications_ , 5(64-67):2, 2001.
  * Bengio et al. [1994] Yoshua Bengio, Patrice Simard, and Paolo Frasconi.  Learning long-term dependencies with gradient descent is difficult.  _IEEE transactions on neural networks_ , 5(2):157–166, 1994.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Chung et al. [2014] Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  In _NIPS 2014 Workshop on Deep Learning, December 2014_ , 2014.
  * Gu et al. [2021] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré.  Combining recurrent, convolutional, and continuous-time models with linear state space layers.  _Advances in neural information processing systems_ , 34:572–585, 2021.
  * Pascanu et al. [2013] Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio.  On the difficulty of training recurrent neural networks.  In _International conference on machine learning_ , pages 1310–1318. Pmlr, 2013.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Kenton and Toutanova [2019] Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of naacL-HLT_ , volume 1, page 2, 2019.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2020.
  * Rao et al. [2021] Yongming Rao, Wenliang Zhao, Zheng Zhu, Jiwen Lu, and Jie Zhou.  Global filter networks for image classification.  _Advances in neural information processing systems_ , 34:980–993, 2021.
  * Kitaev et al. [2019] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.  Reformer: The efficient transformer.  In _International Conference on Learning Representations_ , 2019.
  * Zhou et al. [2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pages 11106–11115, 2021.
  * Liu et al. [2021] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar.  Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting.  In _International conference on learning representations_ , 2021.
  * Liu et al. [2022] Yong Liu, Haixu Wu, Jianmin Wang, and Mingsheng Long.  Non-stationary transformers: Exploring the stationarity in time series forecasting.  _Advances in Neural Information Processing Systems_ , 35:9881–9893, 2022.
  * Zeng et al. [2023] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu.  Are transformers effective for time series forecasting?  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 37, pages 11121–11128, 2023.
  * Yildirim and Akon [2023] Esma Yildirim and Arafat Akon.  Predicting short-term variations in end-to-end cloud data transfer throughput using neural networks.  _IEEE Access_ , 2023.
  * Wu et al. [2021] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long.  Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.  _Advances in Neural Information Processing Systems_ , 34:22419–22430, 2021.
  * Zhang et al. [2022] Tianping Zhang, Yizhuo Zhang, Wei Cao, Jiang Bian, Xiaohan Yi, Shun Zheng, and Jian Li.  Less is more: Fast multivariate time series forecasting with light sampling-oriented mlp structures.  _arXiv preprint arXiv:2207.01186_ , 2022.
  * Zhou et al. [2022] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin.  Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting.  In _International Conference on Machine Learning_ , pages 27268–27286. PMLR, 2022.
  * Woo et al. [2022] Gerald Woo, Chenghao Liu, Doyen Sahoo, Akshat Kumar, and Steven Hoi.  Etsformer: Exponential smoothing transformers for time-series forecasting.  _arXiv preprint arXiv:2202.01381_ , 2022.
  * Wu et al. [2022] Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long.  Timesnet: Temporal 2d-variation modeling for general time series analysis.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Cvetkovic [2000] Zoran Cvetkovic.  On discrete short-time fourier analysis.  _IEEE transactions on signal processing_ , 48(9):2628–2640, 2000.
  * Durak and Arikan [2003] Lutfiye Durak and Orhan Arikan.  Short-time fourier transform: two fundamental properties and an optimal implementation.  _IEEE Transactions on Signal Processing_ , 51(5):1231–1242, 2003.
  * Karimi and Hamilton [2003] Kamran Karimi and Howard J Hamilton.  Distinguishing causal and acausal temporal relations.  In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_ , pages 234–240. Springer, 2003.
  * King [2009] Frederick W King.  _Hilbert Transforms: Volume 2_ , volume 2.  Cambridge University Press, 2009.
  * Cizek [1970] Vaclav Cizek.  Discrete hilbert transform.  _IEEE Transactions on Audio and Electroacoustics_ , 18(4):340–343, 1970.
  * Benitez et al. [2001] D Benitez, PA Gaydecki, A Zaidi, and AP Fitzpatrick.  The use of the hilbert transform in ecg signal analysis.  _Computers in biology and medicine_ , 31(5):399–406, 2001.
  * Xu et al. [2020] Kai Xu, Minghai Qin, Fei Sun, Yuhao Wang, Yen-Kuang Chen, and Fengbo Ren.  Learning in the frequency domain.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1740–1749, 2020.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Szegedy et al. [2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1–9, 2015.
  * Lai et al. [2018] Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu.  Modeling long-and short-term temporal patterns with deep neural networks.  In _The 41st international ACM SIGIR conference on research & development in information retrieval_, pages 95–104, 2018.
  * Man et al. [2014] Chiara Dalla Man, Francesco Micheletto, Dayu Lv, Marc Breton, Boris Kovatchev, and Claudio Cobelli.  The uva/padova type 1 diabetes simulator: new features.  _Journal of diabetes science and technology_ , 8(1):26–34, 2014.
  * Shibu et al. [2010] Muhammed E Shibu, Peter A Leffelaar, Herman Van Keulen, and Pramod K Aggarwal.  Lintul3, a simulation model for nitrogen-limited situations: Application to rice.  _European Journal of Agronomy_ , 32(4):255–271, 2010.
  * Chen et al. [2018] Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.  Neural ordinary differential equations.  _Advances in neural information processing systems_ , 31, 2018.
  * Box and Jenkins [1968] George EP Box and Gwilym M Jenkins.  Some recent advances in forecasting and control.  _Journal of the Royal Statistical Society. Series C (Applied Statistics)_ , 17(2):91–109, 1968.
  * Box et al. [2015] George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung.  _Time series analysis: forecasting and control_.  John Wiley & Sons, 2015.
  * Weisstein [2014] Eric W Weisstein.  Convolution theorem.  _From MathWorld-A Wolfram Web Resource, 2006a. URL http://mathworld. wolfram. com/ConvolutionTheorem. html_ , 2014.

## Appendix

###  A.1 Time Series Forecasting

Many classical methods, such as ARIMA [Box and Jenkins, 1968, Box et al.,
2015], follow a Markovian process by constructing autoregressive TSF models
based on predefined time patterns. However, this autoregressive process
becomes ineffective in real-world nonlinear and non-stationary conditions.

With the flourishing development of deep neural networks, RNNs, specialized
for sequence data analysis, have been proposed and achieved remarkable
results. Vanilla RNN [Medsker and Jain, 2001], as the simplest form in the RNN
family with a single hidden layer and recurrent connections, often suffers
from the problems of vanishing or exploding gradient, which makes it
challenging for long sequence prediction [Bengio et al., 1994]. To address the
gradient problems, Long Short-Term Memory (LSTM) [Hochreiter and Schmidhuber,
1997] and Gated Recurrent Unit (GRU) [Chung et al., 2014] employ gated
structures to control the flow of information, allowing better handling of
dependencies in long sequences. LSSL [Gu et al., 2021] conceptually unifies
the strengths of RNNs and maps a sequence by simply simulating a linear
continuous-time state-space representation. However, due to inherent design
flaws, recurrent models still need help adapting to long-term TSF tasks.

Transformer has been developed and applied to TSF. As a core component of the
Transformer, the attention mechanism plays a crucial role in identifying
dependencies between different variables at various time points. Reformer
[Kitaev et al., 2019] addresses the problem of quadratic computational
complexity in the attention mechanism. It replaces dot-product attention with
locality-sensitive hashing to reduce the time complexity of attention
computation and employs reversible residual layers to reduce memory
complexity. Informer [Zhou et al., 2021] proposes the probspare self-attention
to reduce the time and memory complexity to $\mathcal{O}(L\log L)$. Pyraformer
[Liu et al., 2021] adopts pyramidal attention to capture hierarchical multi-
scale temporal dependencies, thereby reducing the computational cost. Non-
stationary Transformer [Liu et al., 2022] focuses on improving the forecasting
ability to model the non-stationary data. It introduces series
stationarization and de-stationary attention and demonstrates its
effectiveness in forecasting non-stationary data prediction through both
theoretical analysis and experiments. However, relying on the attention
mechanism in the above Transformer-based methods to directly discover
correlations between variables at different time points is unreliable. Complex
temporal patterns may mask such correlations and are challenging to manifest
as significant trends or periodicities in the time domain [Wu et al., 2021].
Furthermore, the primary capability of the Transformer comes from its
attention mechanism, which inevitably leads to the loss of temporal
information [Zeng et al., 2023].

Some researches aim to use the simple multilayer perceptron (MLP) to deal with
the above problems. LightTS [Zhang et al., 2022] applies an MLP-based
structure on top of two delicate down-sampling strategies, including interval
and continuous sampling. Dlinear [Zeng et al., 2023] introduces an
embarrassingly simple one-layer linear model and employs direct multistep
forecasting, achieving high accuracy. Other approaches shift the focus from
the time domain to the frequency domain. Autoformer [Wu et al., 2021] proposes
autocorrelation mechanisms which first employ period-based dependencies and
then aggregate time delay using FFT, for accelerated computation. Fedformer
[Zhou et al., 2022] adopts an expert mixture design to improve trend and
periodicity decomposition and proposes a sparse attention mechanism in the
frequency domain, ensuring efficiency while improving prediction accuracy.
ETSformer [Woo et al., 2022] employs exponential smoothing attention and
frequency attention mechanisms to replace the self-attention mechanism in the
vanilla Transformer, improving both accuracy and efficiency. TimesNet [Wu et
al., 2022] provides a new approach to TSF by decomposing time series into
multiple periods using predefined windows. Through the interaction of intra-
period and inter-period information, it connects generic inception blocks to
complete TSF. However, models mentioned above process time series data without
modeling intra- and inter-variable variations. As a result, they still face
challenges in dealing with complex temporal variations.

###  A.2 Theoretical Proof

###### Theorem 1

If $X(t)$ is a causal time function with respect to time $t$, i.e. ,
$X(t)=\begin{cases}0,&t<0\\\ X(t),&t\geq 0\end{cases}$, and does not contain
the continuous-time unit impulse function $\delta(t)=\begin{cases}0,&t\neq
0\\\ \infty,&t=0\end{cases}$ and derivatives at $t=0$. Suppose the Fourier
transform $\mathcal{F}$ of $X(t)$ is represented in the form of the real part
$Re$ and the imaginary part $Im$, i.e., $X(t)\leftrightarrow
F(\omega)=Re(\omega)+iIm(\omega)$. Then, we have:

$\begin{split}&Re(\omega)=\frac{1}{\pi}\int_{-\infty}^{\infty}\frac{Im(\sigma)}{\omega-\sigma}d\sigma,\\\
&Im(\omega)=-\frac{1}{\pi}\int_{-\infty}^{\infty}\frac{Re(\sigma)}{\omega-\sigma}d\sigma.\end{split}$
(A.1)

###### Proof 1

$X(t)$ can be expressed as $X(t)=X(t)\epsilon(t)$ by introducing the unit step
function $\epsilon(t)=\begin{cases}1,&t>0\\\ 0,&t\leq 0\end{cases}$. Taking
the Fourier transform of both sides of $X(t)=X(t)\epsilon(t)$, based on the
frequency domain convolution theorem [Weisstein, 2014], we have:

$\begin{split}&\mathcal{F}\\{X(t)\\}=Re(\omega)+iIm(\omega),\\\
&\mathcal{F}\\{X(t)\epsilon(t)\\}=\frac{1}{2\pi}[Re(\omega)+iIm(\omega)]\times[\pi\delta(\omega)+\frac{1}{i\omega}].\end{split}$
(A.2)

Using $\mathcal{F}\\{X(t)\\}=\mathcal{F}\\{X(t)\epsilon(t)\\}$ and then we
have:

$\begin{split}Re(\omega)+iIm(\omega)&=\frac{1}{2\pi}[Re(\omega)+iIm(\omega)]\times[\pi\delta(\omega)+\frac{1}{i\omega}]\\\
&=[\frac{Re(\omega)}{2}+\frac{1}{2\pi}Im(\omega)\times\frac{1}{\omega}]+i[\frac{Im(\omega)}{2}-\frac{1}{2\pi}Re(\omega)\times\frac{1}{\omega}].\end{split}$
(A.3)

Equating the real and imaginary parts on both sides of the equation and
rearranging, we get:

$\begin{split}&Re(\omega)=\frac{1}{\pi}Im(\omega)\times\frac{1}{\omega}=\frac{1}{\pi}\int_{-\infty}^{\infty}\frac{Im(\sigma)}{\omega-\sigma}d\sigma,\\\
&Im(\omega)=-\frac{1}{\pi}Re(\omega)\times\frac{1}{\omega}=-\frac{1}{\pi}\int_{-\infty}^{\infty}\frac{Re(\sigma)}{\omega-\sigma}d\sigma.\end{split}$
(A.4)

So far, the Equation A.1 has been proven.

###  A.3 General Overview of FreqTSF

The input lookback window, $X\in\mathbb{R}^{D\times L}$, undergoes
normalization and embedding modules to obtain $X^{0}\in\mathbb{R}^{E\times
L}$. It is then fed into a stack of residual FreqBlocks, and the output of the
$N$th-layer FreqBlock, after denormalization, serves as the forecasting
horizon $\hat{X}\in\mathbb{R}^{D\times T}$. The $n$th FreqBlock is defined as
$X^{n}=\text{FreqBlock}^{n}(X^{n-1})+X^{n-1}$. First, the FreqBlock utilizes
Equation 3 to apply the STFT with windows of various sizes to the time series
data, obtaining the corresponding real and imaginary parts in the frequency
domain, which follows the Top-$M$ frequency enhancement (Equation 5). Next,
the FreqBlock establishes intra-variable KKRs by applying the Frequency Cross
Attention module between the enhanced real and imaginary parts. Finally, the
FreqBlock integrates inter-variable information through a parameter-efficient
inception block. The Frequency Cross Attention module leverages KKR-based
attention to construct mathematical mechanisms, and concatenates the real part
$\tilde{\text{Re}}$ and the imaginary part $\tilde{\text{Im}}$ of the
attention output (Equation 8), representing the hidden information with
inherent KKRs in the frequency domain for output.

Figure A.1: FreqTSF: a structure composed of residual connections.

###  A.4 Datasets

To thoroughly evaluate the performance of FreqTSF, we evaluate the proposed
FreqTSF on two real-world benchmarks and two widely acknowledged time series
simulators, covering mainstream applications in TSF: energy (the ETT dataset
[Zhou et al., 2021]), economics (the Exchange dataset [Lai et al., 2018]),
medicine (the Type 1 Diabetes Mellitus Simulator (T1DMs) [Man et al., 2014]),
and agriculture (the Lintul3 [Shibu et al., 2010]).

  * •

The ETT dataset [Zhou et al., 2021] comprises data collected from electricity
transformers, recorded every 15 minutes from July 2016 to July 2018. The
dataset contains seven time-series variables, including load and oil
temperature. Specifically, ETTh1 and ETTh2 are variant datasets with hourly
granularity.

  * •

The Exchange dataset [Lai et al., 2018] contains daily exchange rates for
eight countries recorded from 1990 to 2016, encompassing eight time-series
variables.

  * •

The Type 1 Diabetes Mellitus Simulator (T1DMs) [Man et al., 2014] elucidates
the temporal evolution of patient states in response to different
distributions of exogenous insulin intervention doses. This simulator has been
accepted by the FDA as a viable alternative to preclinical animal testing for
the evaluation of novel treatment modalities for Type 1 Diabetes Mellitus. In
this study, simulations are performed using default settings tailored for
adult patients to generate temporal data sequences.

  * •

Lintul3 [Shibu et al., 2010] is a simple crop growth model designed to
simulate growth under water and nitrogen-limited conditions. In this
experiment, we simulate the growth process of spring wheat in Northeast China
from January 2002 to August 2002. The dataset includes 20 time-series
variables, including two dimensions of action variables and 18 dimensions of
crop trait variables.

We follow standard protocol and split all datasets into training, validation
and test set in chronological order by the ratio of 6:2:2 for the ETT dataset
and 7:1:2 for the other datasets.

###  A.5 Evaluation Metrics

We use the mean square error (MSE) and mean absolute error (MAE) to evaluate
the accuracy of TSF, which is calculated as follows:

$\begin{split}\text{MSE}(\hat{X},{X})&=\frac{1}{D\times
T}\sum_{d=1}^{D}\sum_{t=L+1}^{L+T}(\hat{x}_{d}^{t}-{x}_{d}^{t})^{2}\\\
\text{MAE}(\hat{X},{X})&=\frac{1}{D\times
T}\sum_{d=1}^{D}\sum_{t=L+1}^{L+T}|\hat{x}_{d}^{t}-{x}_{d}^{t}|\end{split}$
(A.5)

Here, $\hat{X}\in\mathbb{R}^{D\times T}$ represents the estimated time series,
while ${X}$ denotes the corresponding ground-truth.

###  A.6 Implementation details

Similar to TimesNet [Wu et al., 2022], our proposed FreqTSF is trained using
the L2 loss and optimized using the Adam optimizer with an initial learning
rate of $10^{-3}$. The batch size is set to 32. All experiments are repeated
five times, implemented in PyTorch, and conducted on a single NVIDIA RTX TITAN
24GB GPU.

Note that we set the embedding dimension $E$ to 32 and map the input
$X\in\mathbb{R}^{D\times L}$ to $X^{\prime}\in\mathbb{R}^{32\times L}$.
Subsequently, we employ $N=4$ FreqBlocks with residual connections to avoid
the degradation issues. In each FreqBlock, we utilize $P=3$ windows with sizes
$W=\\{100,50,20\\}$, all of which are Hann windows. We make the top-$M=10$
selection for the STFT result in each window to reduce computational
complexity. The dimensions $d_{K}$, $d_{V}$, and $d_{\text{hidden}}$ in the
Frequency Cross Attention are all set to 64. Within the inception block, we
use six kernels of different sizes to fuse inter-variable information from
different STFT windows. We find that the dimensions of the hidden layers and
the number of kernels are not critical factors affecting the performance of
FreqTSF. Therefore, we conduct ablation experiments on the number of
FreqBlocks $N$, the embedding dimension $E$, the window $W$, and the frequency
domain filter $M$, which will be discussed in Section A.9 and A.8.2.

FreqTSF

TimesNet

ETSformer

LightTS

DLinear

FEDformer

Figure A.2: Visualization of ETTh1 predictions by different models under the
input-96-predict-720 setting. The black lines stand for the input, the gray
lines stand for the ground truth and the orange lines stand for predicted
values.

###  A.7 Showcases

To dissect the contributions of each module within our proposed FreqTSF, we
conduct a comparative analysis of prediction results on the ETTh1 dataset
using five baseline models (TimesNet, ETSformer, LightTS, DLinear, and
FEDformer). These baselines have previously demonstrated proficiency with
first MSE and MAE performance on the ETT and Exchange real-world datasets. In
Figure A.2, it is evident that the two modules enhance the forecasting
capabilities of the Transformer model from different perspectives,
particularly in capturing intricate details of the prediction curve, which are
conducted on the ETTh1 dataset.

FreqTSF, TimesNet, and FEDformer exhibit higher forecasting accuracy among
these six models. In contrast, other models suffer from deficiencies in
capturing trends and periodicities, leading to imprecise predictions.
Additionally, we observe that the prediction curves of TimesNet and FEDformer
are relatively smoother, failing to capture abrupt changes in the time series
at certain time points. ETSformer struggles with predicting extreme points, as
evidenced by inaccurate predictions of both the lowest and highest values in
the figure. Moreover, LightTS and DLinear, both MLP-based methods, exhibit
significant cumulative errors that result in increasingly inaccurate
predictions over time. In comparison, FreqTSF stands out by effectively
capturing both trends and periodicities, as well as abrupt changes in the time
series at specific time points. We believe that this phenomenon is attributed
to the Frequency Cross Attention, which provides a consolidated representation
of the real and imaginary parts in the frequency domain, thereby making
enhanced trend and periodicity features causal. This module, followed by the
inception blocks, enables adaptive integration of information between
variables, resulting in precise predictive performance. This result
underscores the effectiveness of FreqTSF in simultaneously capturing various
characteristics of time series, offering a comprehensive and accurate
forecasting solution.

Table 2: Ablation Studies for Modules: Three variants of FreqTSF are compared
with vanilla FreqTSF.

Models | FreqTSF-FTM-FCA | FreqTSF-FCA | FreqTSF-FTM | FreqTSF
---|---|---|---|---
Metrics | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
96 | 0.427 | 0.438 | 0.406 | 0.415 | 0.397 | 0.411 | 0.214 | 0.297
192 | 0.451 | 0.453 | 0.449 | 0.443 | 0.448 | 0.442 | 0.267 | 0.326
336 | 0.498 | 0.479 | 0.491 | 0.468 | 0.483 | 0.461 | 0.338 | 0.369
720 | 0.562 | 0.535 | 0.519 | 0.481 | 0.491 | 0.480 | 0.420 | 0.408
Avg | 0.484 | 0.476 | 0.466 | 0.451 | 0.454 | 0.448 | 0.309 | 0.350

###  A.8 Ablation Studies

In this section, two types of ablation experiments are conducted. The module
ablation studies are designed to validate the effectiveness of the two
proposed modules. Moreover, the ablation studies for $M$ aim to assess the
significance of the critical parameter $M$, which determines the extent to
which noise introduced by meaningless frequencies is mitigated in the context
of this study.

####  A.8.1 Ablation Studies for Modules

In this section, we conduct ablation experiments on ETTh1 dataset to validate
the effectiveness of the Frequency Transform Module and the Frequency Cross
Attention. The current state-of-the-art results of FreqTSF, which utilizes
both the Frequency Transform Module and the Frequency Cross Attention, serve
as the baseline. Three ablation variants of FreqTSF are tested: 1) FreqTSF-
FTM-FCA: neither the Frequency Transform Module nor the Frequency Cross
Attention is used; 2) FreqTSF-FCA: the Frequency Transform Module, and the
vanilla attention mechanism are used to capture frequency domain information;
3) FreqTSF-FTM: the Frequency Cross Attention is used without the Frequency
Transform Module. The ablated versions of FreqTSF and the SOTA model are
compared in Table 2.

As shown in Table 2, FreqTSF, including the Frequency Transform Module and the
Frequency Cross Attention, shows nearly 100% improvement over FreqTSF-FTM-FCA,
demonstrating improvement in all four cases. When the Frequency Transform
Module is added, FreqTSF-FCA achieves an additional 4% improvement over
FreqTSF-FTM-FCA. We posit that the Frequency Transform Module can effectively
remove extraneous frequency information (such as noise) from the frequency
domain, Transform functional frequency components, and thereby facilitate the
model’s understanding of trend-related information in time series data,
ultimately improving accuracy. With the addition of Frequency Cross Attention,
FreqTSF-FTM achieves a 6% improvement over FreqTSF-FTM-FCA. We attribute this
improvement to the ability of Frequency Cross Attention to simulate time
series modeling based on the inherent mechanics of causal systems through the
establishment of real and imaginary KKRs. This improvement contributes to the
reliability of the model. As for the combination of the Frequency Transform
Module and Frequency Cross Attention it not only achieves denoising effects
but also captures the causal characteristics of the system, resulting in
state-of-the-art performance, which validates the effectiveness of the
designed Frequency Transform Module and Frequency Cross Attention in capturing
frequency domain information.

####  A.8.2 Ablation Studies for Frequency Domain Filter

Figure A.3: Ablation Studies for Frequency Domain Filter.

We introduce a hyperparameter $M$ in the Equation 5 to eliminate irrelevant
information in the frequency domain. In this section, $M=\infty$ denotes the
absence of the Top-$M$ selection. The ablation analysis for $M$ is presented
in Figure A.3. Experimental results on ETTh1 dataset show that, when
$T=\\{96,192,336,720\\}$ respectively, MSEs and MAEs tend to initially
decrease (or remain constant) and then increase with increasing $M$. We posit
that the choice of $M$ significantly affects the final model performance,
possibly because $M$ directly influences the amount of information within deep
representations. Furthermore, when $M=\infty$, an excessive amount of
information in the frequency domain results in Frequency Cross Attention being
unable to capture critical information. Conversely, when $M=5$, insufficient
frequency domain information hinders accurate predictions. Hence, in this
study, considering both efficiency and performance, we set $M=10$ for TSF.

###  A.9 Hyperparameter Sensitivity

We introduce a hyperparameter $W$ in the Equation 3 to predefine the sizes of
the windows in the STFT, enriching the information obtained from the Fourier
transform through different dimensions. Therefore, we conduct ablation
experiments on ETTh1 dataset for the embedding dimension $E$ and the number of
FreqBlocks $N$. Table 3 provides the sensitivity analysis for the above three
hyperparameters. Besides, we can also find out the following observations:
The number of FreqBlocks $N$: We observe that $N$ should be increased with the
growth of the forecasting length $T$, but its impact on the model accuracy is
not substantial. In the experiments with $T=96$ and $T=192$, across the above
eight comparison experiments with fixed $E$ and $W$, when $N=2$ it leads to
lower MSEs and MAEs in 7/8 cases. However, in the experiments with $T=336$ and
$T=720$, it achieves lower MSEs and MAEs in 6/8 cases when $N=4$.
Nevertheless, the choice of $N$ does not significantly reduce the MSEs and
MAEs. Thus, we believe that the model performance is more robust to the choice
of $N$.
The embedding dimension $E$: We find that $E$ should also increase with the
growth of the forecasting length $T$, and $E$ has a notable impact on the
accuracy of FreqTSF. In the experiments with $T=96$, we observe that holding
$N$ and $W$ constant, the MSEs and MAEs are lower when $E=32$ compared to
$E=64$. However, in the experiments with $T=192$, $T=336$, and $T=720$,
FreqTSF achieves lower MSEs and MAEs when $E=64$. This effect can be
attributed to the fact that $E$ directly affects the amount of information in
deep representations.
The window sizes $W$: We find that the effect of the window sizes on the
FreqTSF decreases as $T$ increases. In the experiments with $T=96$ and
$T=192$, keeping $N$ and $E$ constant, we observe that it leads to lower MSEs
and MAEs in 6/8 cases with window lengths of $\\{100,50,20\\}$. However, in
the experiments with $T=336$ and $T=720$, changes in the STFT window size have
little effect on the MSEs and MAEs. Thus, we claim that the model performance
is more robust to the choice of $W$.

Table 3: Sensitivity analysis is performed on the hyperparameters, including
FreqBlock number $N$, embedding dimension $E$, and window $W$ in FreqTSF.

$N$ | $E$ | $W$ | T=96 | T=192 | T=336 | T=720
---|---|---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
2 | 32 | 100,50,20 | 0.214 | 0.297 | 0.280 | 0.335 | 0.365 | 0.390 | 0.436 | 0.421
100,50 | 0.230 | 0.303 | 0.294 | 0.347 | 0.344 | 0.373 | 0.437 | 0.422
64 | 100,50,20 | 0.227 | 0.305 | 0.267 | 0.326 | 0.338 | 0.369 | 0.430 | 0.419
100,50 | 0.246 | 0.314 | 0.277 | 0.332 | 0.343 | 0.373 | 0.420 | 0.408
4 | 32 | 100,50,20 | 0.229 | 0.305 | 0.285 | 0.338 | 0.341 | 0.371 | 0.433 | 0.419
100,50 | 0.235 | 0.307 | 0.286 | 0.339 | 0.339 | 0.370 | 0.434 | 0.420
64 | 100,50,20 | 0.252 | 0.325 | 0.285 | 0.338 | 0.339 | 0.370 | 0.431 | 0.417
100,50 | 0.254 | 0.326 | 0.285 | 0.338 | 0.339 | 0.370 | 0.432 | 0.419