# FTMixer: Frequency and Time Domain Representations Fusion for Time Series
Modeling

Zhengnan Li
Communication University of China
Beijing, China
<EMAIL_ADDRESS>Yunxiao Qin
Communication University of China
Beijing, China
<EMAIL_ADDRESS>Xilong Cheng
Communication University of China
Beijing, China
<EMAIL_ADDRESS>Yuting Tan
Communication University of China
Beijing, China
<EMAIL_ADDRESS>
###### Abstract

Time series data can be represented in both the time and frequency domains,
with the time domain emphasizing local dependencies and the frequency domain
highlighting global dependencies. To harness the strengths of both domains in
capturing local and global dependencies, we propose the Frequency and Time
Domain Mixer (FTMixer). To exploit the global characteristics of the frequency
domain, we introduce the Frequency Channel Convolution (FCC) module, designed
to capture global inter-series dependencies. Inspired by the windowing concept
in frequency domain transformations, we present the Windowing Frequency
Convolution (WFC) module to capture local dependencies. The WFC module first
applies frequency transformation within each window, followed by convolution
across windows. Furthermore, to better capture these local dependencies, we
employ channel-independent scheme to mix the time domain and frequency domain
patches. Notably, FTMixer employs the Discrete Cosine Transformation (DCT)
with real numbers instead of the complex-number-based Discrete Fourier
Transformation (DFT), enabling direct utilization of modern deep learning
operators in the frequency domain. Extensive experimental results across seven
real-world long-term time series datasets demonstrate the superiority of
FTMixer, in terms of both forecasting performance and computational
efficiency. Code is available at https://github.com/FMLYD/FTMixer

##  1 Introduction

Time series data, characterized by sequential order and temporal dependencies,
encapsulate valuable insights into the dynamics of various systems and
processes (Liu et al. (2023a, 2024b); Cheng et al. (2024); Lim et al. (2020)).
Diverse datasets such as stock prices, traffic flows (Wu et al. (2021)), and
ECL consumption pose unique challenges and opportunities for computational
analysis (Trindade (2015); Ye et al. (2024b); Lai et al. (2018)) . Each
demands tailored approaches to effectively capture its inherent properties
(Xue et al. (2024); Liu et al. (2023c); Oreshkin et al. (2020)).

Figure 1: (a) Visualizations of the learned patterns of MLPs in the time
domain and the frequency domain. Learning in the frequency domain can identify
clearer diagonal dependencies and key patterns than the time domain. (b) The
predictions in frequency domain and time domain. We can discover that the
output in frequency domain is more smooth and focus on the low-frequency
signals.

In recent years, advancements in deep learning have revolutionized time series
analysis, with a primary focus on time-domain exploration. Among them,
Transformer-based methods (Liu et al. (2023a); Zhang and Yan (2022); Zhou et
al. (2022b); Nie et al. (2022); Wu et al. (2021)) and MLP-based methods (Zeng
et al. (2022); Kim et al. (2021); Han et al. (2024); Das et al. (2023); Toner
and Darlow (2024)) dominate this field. Most previous methods learn time
series in the time domain and achieve promising performance (Qi et al.
(2024)).

In the time domain, representations focus on capturing local dependencies at
specific time points within the data, providing insights into how channels
evolve over time and interact locally. Conversely, frequency domain
representations transform the data into constituent frequencies, offering a
broader perspective that enables the analysis of global dependencies and
periodicities spanning the entire dataset. As depicted in Figure 1 (a), the
weights of the time domain MLP exhibit a more complex spread, with diagonal
lines suggesting periodic information but overall reflecting localized, time-
specific dependencies. However, the weights of the frequency domain MLP reveal
sparse yet concentrated activations along the diagonal, indicating significant
contributions from few frequency components and capturing broad patterns
within the data. This contrast emphasizes the different strengths of each
domain in capturing and representing underlying patterns within the data.

Several studies have leveraged the frequency domain to analyze time series
data, enhancing feature extraction and capturing complex patterns (Wu et al.
(2022); Zhou et al. (2022b); Xu et al. (2023); Wang et al. (2024); Yi et al.
(2023)). For example, FEDformer (Zhou et al. (2022b)) integrates spectral data
to enhance features, while TimesNet (Wu et al. (2022)) utilizes Temporal
Convolutional Networks (TCNs) for feature extraction, emphasizing the
significance of the Fast Fourier Transform (FFT) for periodicity extraction.
Despite these advancements, the full potential of the frequency domain in time
series analysis remains untapped.

However, challenges persist in developing powerful frequency-domain models.
Previous frequency methods often employ Discrete Fourier Transformation (DFT)
(Xu et al. (2023); Zhou et al. (2022b); Wu et al. (2022)), which involves
complex representations. Unfortunately, the widely-used techniques in deep
learning, such as Batch Normalization and activation functions, on complex
numbers is under-explored. Moreover, as depicted in Figure 1 (b), the
frequency domain may over-focus on global dependencies, leading to a lack of
high-frequency information. Additionally, effectively combining the frequency
and time domains for time series forecasting requires further exploration.

In this paper, we introduce Frequency and Time domain Mixer (FTMixer), a novel
approach that efficiently incorporates both time and frequency domains to
handle local and global dependencies simultaneously. First, to unlock the
potential of the frequency domain with deep learning models, we introduce the
Discrete Cosine Transformation (DCT) (Ahmed et al. (1974)). Compared to the
Discrete Fourier Transform (DFT) (Ahmed et al. (1974)), DCT operates
exclusively on real numbers, making it more suitable for modern deep learning
techniques. To capture inter-series global dependencies, we propose a novel
Frequency Channel Convolution (FCC). The key idea of FCC is to embedding the
entire sequence in the frequency domain before convolution. To better capture
local dependencies, we draw inspiration from the windowing DFT (Stasiński
(2002); Xu et al. (2020)) and propose a novel Windowing Frequency Convolution
(WFC) module. Specifically, WFC segments the sequence into patches with
diverse window-scales and performs convolution within the patches under each
scale. To aggregate the frequency domain and time domain, we propose to mix
the frequency domain patches and time domain patches by Depth-Wise Separable
Convolution. Finally, to better leverage the advantages of both the frequency
domain and time domain, we propose to compute the loss function separately in
the frequency domain and time domain.

Extensive experiments on seven datasets demonstrate that FTMixer achieves
superior performance compared to state-of-the-art methods. For instance, on
the ETTh1 dataset, FTMixer achieves almost a 6% improvement over the state-of-
the-art method.

##  2 Related Work

###  2.1 Frequency-Aware Time Series Analysis

Time series analysis plays a crucial role in various domains, including
finance, public health, and weather forecasting (Wu et al. (2021)). Recent
years have witnessed significant development in this field driven by deep
learning models specifically designed for time series tasks. Among these
models, three prominent architectures have garnered considerable attention:
Multi-Layer Perceptrons (MLPs), Transformers, and Temporal Convolutional
Networks (TCNs).

Inspired by their success in natural language processing, Transformers have
been adapted for time series analysis with remarkable results (e.g., Li et al.
(2023), Ni et al. (2024), Ye et al. (2024a)). Examples include Autoformer (Wu
et al. (2021)), which utilizes attention mechanisms to decompose sequences,
PatchTST (Nie et al. (2022)) which segments sequences inspired by the Vision
Transformer (ViT) architecture, and iTransformer (Liu et al. (2023a)) that
embeds the entire sequence then computing attention across channel dimensions.

Known for their simplicity and effectiveness, MLPs have also found application
in time series analysis (e.g., Kim et al. (2021), Liu et al. (2023b), Toner
and Darlow (2024), Li et al. (2024), Zeng et al. (2022), Xu et al. (2023), Das
et al. (2023),Challu et al. (2022)). DLinear (Zeng et al. (2022)), for
instance, performs trend-season decomposition and learns using two MLPs.
RLinear Kim et al. (2021) implements reversible instance norm and achieves
impressive performance. Additionally, FITS (Xu et al. (2023)) directly learns
in the frequency domain, leading to surprising results.

Temporal Convolutional Networks (TCNs) are another class of deep learning
models excelling at capturing local dependencies within time series data
(e.g., Cheng et al. (2024), Wu et al. (2022), Wang et al. (2022), Luo and Wang
(2024)). TimesNet (Wu et al. (2022)) utilizes TCNs for feature extraction,
with a particular focus on leveraging Fast Fourier Transform (FFT) for
periodicity extraction. ModernTCN (Luo and Wang (2024)), drawing inspiration
from transformers, captures inter-series and cross-time information
simultaneously. ConvTimeNet (Cheng et al. (2024)) proposes a novel patch
method to determine the suitable length of the patch window, enhancing the
adaptability of TCNs to various time series datasets.

Despite the advancements, most existing methods capture patterns in the time
domain, neglecting the potential of the frequency domain . This motivates our
exploration of Frequency-Aware Time Series Analysis, which leverages insights
from the frequency domain to enhance prediction accuracy.

####  2.1.1 Frequency-Aware Time Series Analysis

Several successful approaches have demonstrated the value of incorporating
frequency domain information. FEDFormer Zhou et al. (2022b) and FiLM Zhou et
al. (2022a) integrate frequency information as additional features, enriching
the model’s ability to discern long-term periodic patterns while
simultaneously expediting computation. TimesNet (Wu et al. (2022)) adopts FFT
for decomposing periodic patterns, followed by denoising via high-frequency
filtering. FreTS (Yi et al. (2023)) forecasts time series by leveraging both
inner-series and inter-series information. On the other hand, FITS (Xu et al.
(2023)) achieves improved performance by training sequences directly in the
frequency domain using a fully connected layer. However, a single linear model
often proves insufficient for capturing non-linear patterns in the frequency
domain. Additionally, the effectiveness of traditional deep learning
techniques like activation functions and batch normalization on complex number
data (used in the DFT) remains uncertain.

This work addresses these limitations by introducing the Discrete Cosine
Transform (DCT) for the first time in time series analysis. Compared to the
Discrete Fourier Transform (DFT) (Ahmed et al. (1974)), DCT operates
exclusively on real numbers, making it more suitable for modern deep learning
techniques. Furthermore, DCT utilizes only amplitude to represent the
frequency domain information, simplifying the computation of the loss function
in the frequency domain. These advantages of DCT pave the way for a novel and
potentially more effective approach to frequency-aware time series
forecasting.

recommendation systems Liu et al. (2024a); Yang et al. (2024).

##  3 Methodology

###  3.1 Prelimiary

####  3.1.1 Problem Definition

Let $[{X}_{1},{X}_{2},\cdots,{X}_{T}]\in\mathbb{R}^{N\times T}$ stand for the
regularly sampled multi-channel time series dataset with $N$ series and $T$
timestamps, where ${X}_{t}\in\mathbb{R}^{N}$ denotes the multi-channel values
of $N$ distinct series at timestamp $t$. We consider a time series lookback
window of length-$L$ at each timestamp $t$ as the model input, namely
$\mathbf{X}_{t}=[{X}_{t-L+1},{X}_{t-L+2},\cdots,{X}_{t}]\in\mathbb{R}^{N\times
L}$; also, we consider a horizon window of length-$\tau$ at timestamp $t$ as
the prediction target, denoted as
$\mathbf{Y}_{t}=[{X}_{t+1},{X}_{t+2},\cdots,{X}_{t+\tau}]\in\mathbb{R}^{N\times\tau}$.
Then the time series forecasting formulation is to use historical observations
$\mathbf{X}_{t}$ to predict future values ${\mathbf{Y}}_{t}$. For simplicity,
we shorten the model input $\mathbf{X}_{t}$ as
$\mathbf{X}=[{X}_{1},{X}_{2},\cdots,{X}_{L}]\in\mathbb{R}^{N\times L}$ and
reformulate the prediction target as
$\mathbf{Y}=[{X}_{L+1},{X}_{L+2},\cdots,{X}_{L+\tau}]\in\mathbb{R}^{N\times\tau}$,
in the rest of the paper.

####  3.1.2 Discrete Cosine Transformation

Our methodology hinges on the utilization of the Discrete Cosine
Transformation (DCT) to transmute input data into the frequency domain. Here
we elucidate the fundamental concepts of the Cosine Transformation and Fourier
Transformation, forging a vital linkage between the two, before delving into
the intricacies of the Discrete Cosine Transformation.

The Cosine Transform emerges as a specialized variant within the broader
spectrum of the Fourier Transform. While the Fourier Transform disassembles a
function into both sine and cosine constituents, the Cosine Transform
singularly concentrates on the cosine components. This emphasis proves
particularly advantageous in scenarios where the function exhibits symmetry,
thereby facilitating a more streamlined transformational process.

The continuous Fourier Transform of a function $f(t)$ is articulated as:

$F(\omega)=\int_{-\infty}^{\infty}f(t)e^{-i\omega t}\,dt,$

where $F(\omega)$ symbolizes the frequency domain representation of $f(t)$.
For an even function $f_{e}(t)$, the Fourier Transform can be expressed solely
in terms of cosine functions, owing to its symmetry:

$F(\omega)=\int_{-\infty}^{\infty}f_{e}(t)\cos(\omega t)\,dt.$

This profound connection between continuous Cosine Transformation and Fourier
Transformation underscores the elegance of cosine components in symmetric
functions. Moving forward, we establish a formal relationship between the
Discrete Cosine Transformation and the Discrete Fourier Transformation,
enunciated through the following theorem:

Theorem 1: The Discrete Cosine Transform (DCT) of a sequence can be derived
from the Discrete Fourier Transform (DFT) of a symmetrically extended version
of the sequence. The proof can be found in Appendix LABEL:appendix:proof.

We define the DCT for a sequence of channels as:

$\bar{x}_{k}=\sum_{i=1}^{L}x_{i}\cos\left(\frac{\pi}{L}\left(i+\frac{1}{2}\right)k\right),$
(1)

where $x_{i}$ represents the $i$-th element of the series
$\mathbf{X}\in\mathbb{R}^{L}$, and $\mathbf{X}$ denotes the sequence in the
time domain. $L$ signifies the length of $\mathcal{X}$, and $\bar{x}_{k}$
denotes the $k$-th frequency component in the DCT frequency domain
coefficients, with $k\in[1,2,\ldots,L]$.

Utilizing Eq. 1, we derive
$\bar{\mathcal{X}}=[\bar{x}_{1},\bar{x}_{2},\ldots,\bar{x}_{L}]$, representing
the frequency features of $\mathcal{X}$.

Moreover, the DCT is reversible, allowing for the transformation of frequency
domain coefficients back to the time domain via the inverse Discrete Cosine
Transform (iDCT), articulated as:

$x_{i}=\frac{1}{2}\bar{x}_{1}+\sum_{k=2}^{L}\bar{x}_{k}\cos\left(\frac{\pi}{L}\left(k+\frac{1}{2}\right)i\right).$
(2)

Our proposed methodology harnesses the Discrete Cosine Transformation (DCT) to
transition input data into the frequency domain. Renowned in signal
processing, the DCT facilitates the conversion of spatial domain data into its
frequency domain counterpart by emphasizing its cosine constituents.
Distinguished by its operational efficiency with real numbers, the DCT mirrors
the Discrete Fourier Transform (DFT) albeit with a spatial domain extension,
thereby simplifying integration with deep neural networks via existing deep
learning frameworks.

###  3.2 Overall Architecture

To leverage both time and frequency domains to capture intricate patterns
within time series data, we propose a novel method Frequency and Time domain
Mixer (FTMixer). As shown in Figure 2, we propose a novel Frequency Channel
Convolution (FCC) module to extract inter-series information in the frequency
domain. Inspired by the concept of windowing in frequency domain
transformations, we also introduce a novel module termed Windowing Frequency
Convolution (WFC). Additionally, the proposed FTMixer incorporates RevIN Kim
et al. (2021) to bolster robustness. These components collectively form a
cohesive structure that balances local and global feature extraction for time
series analysis. In summary, the Model Structure of FTMixer can be represented
as:

$\left\\{\begin{array}[]{lr}\mathbf{Z}_{\text{FCC}}=f_{\text{FCC}}(\mathbf{X}),\\\
\mathbf{Z}_{\text{DS}}=f_{\text{DS}}(\text{Concate}(\mathcal{F}_{\text{WFC}}(\mathbf{X})),\\\
{\mathbf{Z}}=\mathbf{Z}_{\text{FCC}}+\mathbf{Z}_{\text{DS}},\\\
\hat{\mathbf{Y}}=f_{\text{Pre}}(\mathbf{Z}),\end{array}\right.$ (3)

where $\hat{\mathbf{Y}}$ stands for the output and $\mathbf{X}$ represents the
input. $\mathcal{F}_{\text{WFC}}$ is the mapping function that maps the WFC
module to all channels in the input $\mathbf{X}$, as Figure 2 shows.
$f_{\text{DS}}$ and $f_{\text{Pre}}$ are the depth-wise separative convolution
(DS-Conv) and the model predictor.

Figure 2: The framework of FTMixer. FTMixer comprises two main modules: FCC
and WFC. An example $\mathbf{X}$ containing four channels is visualized here
for easier understanding. The model predictor is a Linear layer.

###  3.3 Frequency Channel Convolution

The FCC module is designed to capture inter-series dependencies in the
frequency domain. However, convolution often emphasizes local dependencies,
neglecting broader global dependencies due to its local invariance.
Recognizing that the frequency domain is more conducive to revealing global
dependencies compared to the time domain, we suggest embedding the entire
sequence into the frequency domain prior to convolution. To achieve this, we
independently apply the Discrete Cosine Transform (DCT) to each channel of the
input, thereby obtaining comprehensive frequency domain representations of the
sequence.

$\mathbf{X}_{f}=\text{Embedding}(\text{DCT}({\mathbf{X}}))$ (4)

Here, $\mathbf{X}_{f}$ represents the frequency domain representations of
$\mathbf{X}$. Drawing inspiration from transformers, we propose to convolve
along the series feature dimensions after embedding. This convolution can be
formulated as:

$\vspace{-3pt}\mathbf{Z}_{\text{FCC}}=\text{iDCT}(\text{Conv1d}(\mathbf{X}_{f}))$
(5)

###  3.4 Windowing Frequency Convolution

Previous frequency-domain based models commonly focus solely on the frequency
domain representation of the entire sequence, potentially leading to similar
representations for two distinct sequences in the time domain. Drawing
inspiration from the windowing concept in frequency domain transformations, we
propose performing DCT within multi-scale windows to capture fine-grained
information. Specifically, as Figure 2 shows, in the WFC module, we firstly
segment each channel of the input sequence into patches with diverse scales
and then perform DCT within each patch to obtain the local frequency domain
representation. To capture inter-patch information, we perform convolution
across these patches. Finally, we transform them back to the time domain and
add $\mathbf{Z}_{f}$, which can be formulated as:

$\vspace{-3pt}\left\\{\begin{array}[]{lr}\mathbf{Z}_{\mathbf{P}_{j}}=\text{Embedding}\Big{(}\text{iDCT}\big{(}\text{Conv}(\text{DCT}(\mathbf{P}_{j}))\big{)}+\mathbf{P}_{j}\Big{)},\\\
{\mathbf{Z}_{\text{WFC}}}=\text{Concate}(\mathbf{Z}_{\mathbf{P}_{1}},\mathbf{Z}_{\mathbf{P}_{2}},\ldots,\mathbf{Z}_{\mathbf{P}_{n}}),\end{array}\right.$
(6)

where $\mathbf{P}_{j}$ denotes the $j$-th patch (e.g., $\mathbf{P}_{1}$ or
$\mathbf{P}_{2}$ in Figure 2) in the WFC module. $\mathbf{Z}_{\mathbf{P}_{j}}$
is the feature obtained for the $j$-th patch after the embedding layer.
${\mathbf{Z}_{\text{WFC}}}$ is the output of the WFC module for the current
input channel. Note that we separately apply the WFC module on each input
channel, which can be seen in Figure 2.

###  3.5 Depth-Wise Separable Convolution

After obtain ${\mathbf{Z}_{\text{WFC}}}$ for all input channel, we first
concate them and then apply Depth-Wise Separable Convolution (DS Convolution),
to further process the obtained feature. DS Convolution decouples the learning
of intra-patch dimensions from the learning of inter-patch dimensions. It is
more computationally efficient than vanilla convolution and comprises two
components: Depth-Wise Convolution (DW Conv) and Point-Wise Convolution (PW
Conv). DW Conv aggregates inter-patch information through grouped convolution,
while PW Conv operates akin to a Feed Forward Network (FFN) to extract intra-
patch information.

###  3.6 Loss Function

To thoroughly leverage the advantages of the frequency and time domains, we
propose calculating the loss function specifically in each domain. For the
time domain, we use Mean Squared Error (MSE) as the loss function, while for
the frequency domain, we utilize Mean Absolute Error (MAE) loss instead of
MSE. The reason why we use L1 loss function in frequency domain is that
different frequency components often exhibit vastly varying magnitudes,
rendering squared loss methods unstable. The experimental results in Appendix
LABEL:sec:abl-loss prove that both the two loss functions are helpful to
improve the performance. The overall loss function of our method can be
expressed as:

$\vspace{-3pt}\left\\{\begin{array}[]{lr}\vspace{3pt}\mathcal{L}_{\text{time}}=\sum_{i=1}^{\tau}\frac{||\mathbf{Y}_{i}-F(\mathbf{X})_{i}||^{2}}{\tau},\\\
\vspace{1pt}\mathcal{L}_{\text{fre}}=\sum_{i=1}^{\tau}\frac{||\mathbf{Y}_{i}-F(\mathbf{X})_{i}||}{\tau},\\\
\vspace{3pt}\mathcal{L}_{\text{total}}=\mathcal{L}_{\text{time}}+\mathcal{L}_{\text{fre}}.\end{array}\right.$
(7)

##  4 Experiment

###  4.1 Experiment Settings

In this section, we evaluate the efficacy of FTMixer on time series
forecasting, and anomaly detection tasks. We show that our FTMixer can serve
as a foundation model with competitive performance on these tasks.

Table 1: The Statistics of the seven datasets used in our experiments. Datasets | ETTh1&2 | ETTm1&2 | Traffic | ECL | Weather
---|---|---|---|---|---
Channels | 7 | 7 | 862 | 321 | 21
Timesteps | 17,420 | 69,680 | 17,544 | 26,304 | 52,696
Granularity | 1 hour | 5 min | 1 hour | 1 hour | 10 min

##### Datasets.

Our study delves into the analysis of seven widely-used real-world multi-
channel time series forecasting datasets. These datasets encompass diverse
domains, including ECL Transformer Temperature (ETTh1, ETTh2, ETTm1, and
ETTm2) Zhou et al. (2021), ECL, Traffic, and Weather, as utilized by
Autoformer Wu et al. (2021). For fairness in comparison, we adhere to a
standardized protocol Liu et al. (2023a), dividing all forecasting datasets
into training, validation, and test sets. Specifically, we employ a ratio of
6:2:2 for the ETT dataset and 7:1:2 for the remaining datasets. Refer to Table
1 for an overview of the characteristics of these datasets.

Table 2: Full forecasting results on different prediction lengths
$\in\\{96,192,336,720\\}$. Lower MSE and MAE indicate better performance. We
highlight the best performance with red bold text.

Methods | FTMixer | ModernTCN | TSLANet | iTransformer | PatchTST | Crossformer | FEDformer | RLinear | Dlinear | TimesNet
---|---|---|---|---|---|---|---|---|---|---
Methods | —— | ICLR 2024 | ICML 2024 | ICLR 2024 | ICLR 2023 | ICLR 2023 | ICML 2022 | ICLR 2022 | AAAI 2023 | ICLR 2023
Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
$ETTh1$ | 96 | 0.356 | 0.388 | 0.391 | 0.410 | 0.382 | 0.406 | 0.386 | 0.405 | 0.382 | 0.401 | 0.423 | 0.448 | 0.376 | 0.419 | 0.386 | 0.395 | 0.375 | 0.399 | 0.384 | 0.402
192 | 0.401 | 0.410 | 0.416 | 0.423 | 0.422 | 0.435 | 0.441 | 0.436 | 0.428 | 0.425 | 0.471 | 0.474 | 0.420 | 0.448 | 0.437 | 0.424 | 0.405 | 0.416 | 0.436 | 0.429
336 | 0.422 | 0.425 | 0.437 | 0.435 | 0.443 | 0.451 | 0.487 | 0.458 | 0.451 | 0.436 | 0.570 | 0.546 | 0.459 | 0.465 | 0.479 | 0.446 | 0.439 | 0.443 | 0.491 | 0.469
720 | 0.430 | 0.454 | 0.461 | 0.470 | 0.499 | 0.498 | 0.503 | 0.491 | 0.452 | 0.459 | 0.653 | 0.621 | 0.506 | 0.507 | 0.481 | 0.470 | 0.472 | 0.490 | 0.521 | 0.500
Avg | 0.402 | 0.419 | 0.426 | 0.434 | 0.434 | 0.447 | 0.454 | 0.448 | 0.428 | 0.430 | 0.529 | 0.522 | 0.440 | 0.460 | 0.446 | 0.434 | 0.423 | 0.437 | 0.458 | 0.450
$ETTh2$ | 96 | 0.275 | 0.335 | 0.279 | 0.344 | 0.289 | 0.351 | 0.297 | 0.349 | 0.285 | 0.340 | 0.745 | 0.584 | 0.358 | 0.397 | 0.288 | 0.338 | 0.289 | 0.353 | 0.340 | 0.374
192 | 0.336 | 0.375 | 0.342 | 0.387 | 0.342 | 0.388 | 0.380 | 0.400 | 0.356 | 0.386 | 0.877 | 0.656 | 0.429 | 0.439 | 0.374 | 0.390 | 0.383 | 0.418 | 0.402 | 0.414
336 | 0.359 | 0.398 | 0.363 | 0.402 | 0.371 | 0.413 | 0.428 | 0.432 | 0.365 | 0.405 | 1.043 | 0.731 | 0.496 | 0.487 | 0.415 | 0.426 | 0.448 | 0.465 | 0.452 | 0.452
720 | 0.388 | 0.427 | 0.390 | 0.428 | 0.417 | 0.450 | 0.427 | 0.445 | 0.395 | 0.427 | 1.104 | 0.763 | 0.463 | 0.474 | 0.420 | 0.440 | 0.605 | 0.551 | 0.462 | 0.468
Avg | 0.339 | 0.383 | 0.344 | 0.390 | 0.355 | 0.401 | 0.383 | 0.407 | 0.347 | 0.387 | 0.942 | 0.684 | 0.437 | 0.449 | 0.374 | 0.399 | 0.431 | 0.447 | 0.414 | 0.427
$ETTm1$ | 96 | 0.284 | 0.334 | 0.293 | 0.345 | 0.286 | 0.340 | 0.334 | 0.368 | 0.291 | 0.340 | 0.404 | 0.426 | 0.379 | 0.419 | 0.355 | 0.376 | 0.299 | 0.343 | 0.338 | 0.375
192 | 0.321 | 0.361 | 0.336 | 0.372 | 0.329 | 0.372 | 0.377 | 0.391 | 0.328 | 0.365 | 0.450 | 0.451 | 0.426 | 0.441 | 0.391 | 0.392 | 0.335 | 0.365 | 0.374 | 0.387
336 | 0.355 | 0.384 | 0.370 | 0.391 | 0.356 | 0.387 | 0.426 | 0.420 | 0.365 | 0.389 | 0.532 | 0.515 | 0.445 | 0.459 | 0.424 | 0.415 | 0.369 | 0.386 | 0.410 | 0.411
720 | 0.415 | 0.417 | 0.422 | 0.419 | 0.417 | 0.418 | 0.491 | 0.459 | 0.422 | 0.423 | 0.666 | 0.589 | 0.543 | 0.490 | 0.487 | 0.450 | 0.425 | 0.421 | 0.478 | 0.450
Avg | 0.343 | 0.373 | 0.355 | 0.382 | 0.347 | 0.380 | 0.407 | 0.410 | 0.352 | 0.379 | 0.513 | 0.495 | 0.448 | 0.452 | 0.414 | 0.408 | 0.357 | 0.379 | 0.400 | 0.406
$ETTm2$ | 96 | 0.163 | 0.252 | 0.168 | 0.257 | 0.167 | 0.262 | 0.180 | 0.264 | 0.169 | 0.254 | 0.287 | 0.366 | 0.203 | 0.287 | 0.182 | 0.265 | 0.167 | 0.260 | 0.187 | 0.267
192 | 0.219 | 0.287 | 0.225 | 0.297 | 0.230 | 0.305 | 0.250 | 0.309 | 0.230 | 0.294 | 0.414 | 0.492 | 0.269 | 0.328 | 0.246 | 0.304 | 0.224 | 0.303 | 0.249 | 0.309
336 | 0.269 | 0.320 | 0.273 | 0.328 | 0.284 | 0.337 | 0.311 | 0.348 | 0.280 | 0.329 | 0.597 | 0.542 | 0.325 | 0.366 | 0.307 | 0.342 | 0.281 | 0.342 | 0.321 | 0.351
720 | 0.351 | 0.377 | 0.370 | 0.390 | 0.368 | 0.391 | 0.412 | 0.407 | 0.378 | 0.386 | 1.730 | 1.042 | 0.421 | 0.415 | 0.407 | 0.398 | 0.397 | 0.421 | 0.408 | 0.403
Avg | 0.250 | 0.309 | 0.259 | 0.318 | 0.262 | 0.324 | 0.288 | 0.332 | 0.264 | 0.316 | 0.757 | 0.611 | 0.305 | 0.349 | 0.286 | 0.327 | 0.267 | 0.332 | 0.291 | 0.333
$Traffic$ | 96 | 0.362 | 0.238 | 0.425 | 0.298 | 0.372 | 0.261 | 0.395 | 0.268 | 0.401 | 0.267 | 0.522 | 0.290 | 0.587 | 0.366 | 0.649 | 0.389 | 0.410 | 0.282 | 0.593 | 0.321
192 | 0.382 | 0.252 | 0.435 | 0.302 | 0.388 | 0.266 | 0.417 | 0.276 | 0.406 | 0.268 | 0.530 | 0.293 | 0.604 | 0.373 | 0.601 | 0.366 | 0.423 | 0.287 | 0.617 | 0.336
336 | 0.389 | 0.256 | 0.446 | 0.306 | 0.394 | 0.269 | 0.433 | 0.283 | 0.421 | 0.277 | 0.558 | 0.305 | 0.621 | 0.383 | 0.609 | 0.369 | 0.436 | 0.296 | 0.629 | 0.336
720 | 0.425 | 0.281 | 0.452 | 0.311 | 0.430 | 0.289 | 0.467 | 0.302 | 0.452 | 0.297 | 0.589 | 0.328 | 0.626 | 0.382 | 0.647 | 0.387 | 0.466 | 0.315 | 0.640 | 0.350
Avg | 0.390 | 0.257 | 0.440 | 0.304 | 0.396 | 0.271 | 0.428 | 0.282 | 0.420 | 0.277 | 0.550 | 0.304 | 0.610 | 0.376 | 0.627 | 0.378 | 0.434 | 0.295 | 0.620 | 0.336
$Weather$ | 96 | 0.143 | 0.187 | 0.150 | 0.204 | 0.148 | 0.198 | 0.174 | 0.214 | 0.160 | 0.204 | 0.158 | 0.230 | 0.217 | 0.296 | 0.192 | 0.232 | 0.176 | 0.237 | 0.172 | 0.220
192 | 0.188 | 0.232 | 0.196 | 0.247 | 0.194 | 0.242 | 0.221 | 0.254 | 0.204 | 0.245 | 0.206 | 0.277 | 0.276 | 0.336 | 0.240 | 0.271 | 0.220 | 0.282 | 0.219 | 0.261
336 | 0.241 | 0.276 | 0.247 | 0.286 | 0.245 | 0.282 | 0.278 | 0.296 | 0.257 | 0.285 | 0.272 | 0.335 | 0.339 | 0.380 | 0.292 | 0.307 | 0.265 | 0.319 | 0.280 | 0.306
720 | 0.318 | 0.332 | 0.330 | 0.339 | 0.325 | 0.337 | 0.358 | 0.349 | 0.329 | 0.338 | 0.398 | 0.418 | 0.403 | 0.428 | 0.364 | 0.353 | 0.323 | 0.362 | 0.365 | 0.359
Avg | 0.223 | 0.257 | 0.231 | 0.269 | 0.228 | 0.265 | 0.258 | 0.278 | 0.238 | 0.268 | 0.259 | 0.315 | 0.309 | 0.360 | 0.272 | 0.291 | 0.246 | 0.300 | 0.259 | 0.287
$ECL$ | 96 | 0.127 | 0.217 | 0.142 | 0.345 | 0.136 | 0.229 | 0.148 | 0.240 | 0.138 | 0.230 | 0.219 | 0.314 | 0.193 | 0.308 | 0.201 | 0.281 | 0.140 | 0.237 | 0.168 | 0.272
192 | 0.145 | 0.235 | 0.156 | 0.25 | 0.152 | 0.244 | 0.162 | 0.253 | 0.149 | 0.243 | 0.231 | 0.322 | 0.201 | 0.315 | 0.201 | 0.283 | 0.153 | 0.249 | 0.184 | 0.289
336 | 0.163 | 0.262 | 0.174 | 0.269 | 0.168 | 0.262 | 0.178 | 0.269 | 0.169 | 0.262 | 0.246 | 0.337 | 0.214 | 0.329 | 0.215 | 0.298 | 0.169 | 0.267 | 0.198 | 0.300
720 | 0.199 | 0.285 | 0.211 | 0.297 | 0.205 | 0.293 | 0.225 | 0.317 | 0.211 | 0.299 | 0.280 | 0.363 | 0.246 | 0.355 | 0.257 | 0.331 | 0.203 | 0.301 | 0.220 | 0.320
Avg | 0.159 | 0.249 | 0.171 | 0.290 | 0.165 | 0.257 | 0.178 | 0.270 | 0.167 | 0.259 | 0.244 | 0.334 | 0.214 | 0.327 | 0.219 | 0.298 | 0.166 | 0.264 | 0.193 | 0.295

##### Evaluation protocol.

Our evaluation framework, inspired by TimesNet Wu et al. (2022), hinges on two
key metrics: Mean Squared Error (MSE) and Mean Absolute Error (MAE). To ensure
a fair comparison, we adhere to a consistent evaluation protocol. Herein, the
historical horizon’s length is uniformly set at $T=96$ across all models,
while prediction lengths ($F$) are selected from the set ${96,192,336,720}$.

##### Baseline setting.

We compare FTMixer against a variety of state-of-the-art baselines.
Transformer-based baselines include iTransformer, PatchTST, Crossformer,
FEDformer, and Autoformer. MLP-based baselines include RLinear Kim et al.
(2021) and DLinear Zeng et al. (2022). Besides, we also consider the
Convolutional-based baseline SCINet Liu et al. (2022), the general-purpose
time series models TimesNet, another frequency domain related baseline TSLANet
Eldele et al. (2024), and ModernTCN Luo and Wang (2024).

##### Inplemental Details

Similar to the settings from Zhou et al. (2023), we set the look-back window
to 336 for the ETT dataset, 512 for the Traffic, Weather,and ECL datasets. We
also incorporate the data normalization block, and reverse instance norm in
the forecasting task Kim et al. (2021). For the baselines, we report the best
results in their original works if their settings are consistent with ours,
otherwise, we re-run their official codes to obtain the performance for fair
comparison. Notable, the official codes of some baselines contain a bug
dropping the last batch in test phase. For these baselines, we fix the bug and
re-run them. All reported results are the means under 10 random seeds.

###  4.2 Experimental Results

Quantitative Comparison. Table 2 presents the comprehensive forecasting
results, with the best-performing performances highlighted in bold and the
second-best highlighted in blue. Lower values of Mean Squared Error (MSE) and
Mean Absolute Error (MAE) indicate better predictive performance. It is
evident that FTMixer consistently demonstrates the most promising predictive
performance across all datasets.

To provide further insight, we select three datasets for clarification:

ETTh1 Dataset: The dataset ETTh1 exhibits both global and complex local multi-
scale dependencies. In this scenario, FTMixer achieves notably superior
results, with an average MSE improvement of 4.9% compared to the second-best
model, PatchTST. Analysis in Table 3 indicates that both Weighted Fusion
Context (WFC) and Fusion Context Vector (FCV) are effective on this dataset.
Particularly, WFC plays a crucial role in capturing multi-scale local
dependencies, contributing significantly to the model’s performance when
dealing with datasets characterized by complex local dependencies. Weather
Dataset: This dataset is characterized by local dependencies with considerable
noise. Despite these challenges, FTMixer successfully captures intricate
patterns within the data, leading to significantly promising results. Patching
methods, employed to effectively capture important local patterns and
relationships within the time series, play a vital role in maintaining good
performance. Furthermore, our WFC enhances these local patterns by effectively
capturing multi-scale local dependencies. ECL Dataset: This dataset exhibits
significant global dependencies. Even in this scenario, FTMixer outperforms
other models, achieving the best performance. The robustness of FTMixer across
datasets with varying characteristics underscores its efficacy in capturing
both global and local dependencies effectively. This meticulous analysis
reveals the robustness and effectiveness of FTMixer across diverse datasets
with different dependency structures and noise levels.

##  5  Model Analysis

###  5.1 Ablation Study

| ETTh1 | ECL | Weather | ETTm2
---|---|---|---|---
w/o WFC | 0.447 | 0.186 | 0.285 | 0.293
w/o FCC | 0.427 | 0.171 | 0.252 | 0.258
Ours | 0.402 | 0.159 | 0.223 | 0.250
Table 3: The ablation experimental results about WFC and FCC.

In this subsection , we assess the contribution of the different components in
our model, where we report the performance of the model when removing each
component individually. The ablation study of loss function can be found in
Appendix LABEL:sec:abl-loss.

The effectiveness of WFC. WFC is responsible for capturing local dependencies
and improve robustness. WFC significantly contributes to the performance on
the datasets with obivious local dependencies, for instance weather and ETTh1.
As shown in Table 3, removing WFC yields a notable decline in performance, its
absence results in higher MSE values in the forecasting task of 0.447 and
0.186 for the ETTh1 and ECL datasets. This underscores the WFC’s critical role
in feature extraction and capturing complex multi-scale local dependencies.

The effectiveness of FCC. FCC aims at capturing global dependencies and inter-
series information which is vital for performance on datasets with intricate
inter-series dependencies. As illustrated in Table 3, removing FCC
significantly harms the performance on ECL dataset.

###  5.2 Computational Efficiency

Cost | Benchmark | FTMixer | ModernTCN | PatchTST
---|---|---|---|---
Time | ETTh1 | 0.021s | 0.043s | 0.027s
ECL | 0.023s | 0.141s | 0.036s
Memory | ETTh1 | 272M | 316M | 448M
ECL | 304MB | 3350MB | 828MB
Table 4: The computational cost comparison between the proposed FTMixer,
iTransformer, and PatchTST. The batch size for ETTh1 dataset here is $32$,
while the batch size for ECL dataset here is $1$.

In this subsection, we delve into the computational efficiency analysis of the
proposed FTMixer. FTMixer stands out as a purely Temporal Convolutional
Network (TCN)-based method, distinguished by its streamlined computational
overhead. Unlike Transformer-based methodologies, which typically entail a
computational complexity of $O(T^{2})$ per layer, FTMixer significantly
mitigates this burden to $O(KT)$, where $K$ denotes the size of the
convolution kernel. The computational complexity of the Feature Vision
Convolution (FCC) component within FTMixer is expressed as $O(TMK)$, where $M$
signifies the number of channels and $K$ denotes the size of the convolution
kernel. To substantiate the efficiency claims of FTMixer, experiments were
conducted on the ETTh1 and ECL datasets. As delineated in Table 4, FTMixer
showcased superior efficiency when compared to two prominent state-of-the-art
methods ModernTCN and PatchTST. ModernTCN’s efficiency diminishes as the
number of channels escalates, attributable to the linear complexity of its
convolution kernel concerning channel count. For instance, the ETTh1 dataset
comprises 10 channels, while the ECL dataset comprises 321 channels. In stark
contrast, our FTMixer component demonstrates less memory costs under identical
conditions.

###  5.3 Visualization

Figure 3: The visualization of the prediction of FTMixer, PatchTST, and
ModernTCN on ETTh1. Notably, te proposed FTMixer performs the best

Here we visualize the predictions on the ETTh1 dataset. As illustrated in
Figure 3, our FTMixer model demonstrates superior prediction accuracy compared
to the two baselines, PatchTST and ModernTCN. Unlike these time domain
methods, FTMixer reveals more periodic information and effectively captures
preciser global dependencies.

| ETTh1 | Weather
---|---|---
| MSE | MAE | MSE | MAE
96 | 0.368 | 0.390 | 0.165 | 0.207
192 | 0.360 | 0.389 | 0.153 | 0.195
336 | 0.357 | 0.388 | 0.143 | 0.187
720 | 0.355 | 0.387 | 0.142 | 0.189
Table 5: The performance of the proposed FTMixer under diverse input lengths.

###  5.4 Varying Input Length

In this section, we conduct experiments with varying input lengths
$L\in{96,192,336,720}$, while keeping the predicted length fixed at $96$. As
shown in Table 5, our model’s performance improves with longer input lengths,
highlighting the input length efficiency of our model. A frequency component
in the input may resemble a frequency component in the entire dataset. Thanks
to the Wavelet Frequency Component (WFC) and Feature Vision Convolution (FCC),
our model can capture the main frequency component. With longer input lengths,
this frequency component becomes more representative of the entire dataset,
thereby enhancing the model’s length efficiency.

##  6 Conclusion

This paper investigates the potential of combining information from both the
time domain and the frequency domain for time series forecasting tasks. We
propose a novel method called FTMixer, which leverages the Discrete Cosine
Transform (DCT) to integrate time-series data with its corresponding frequency
domain representation. Extensive experiments demonstrate the effectiveness of
FTMixer and highlight the value of frequency domain information in time series
forecasting. FTMixer achieves state-of-the-art performance while maintaining
computational efficiency. We believe these findings can stimulate greater
interest in the role of the frequency domain for time series forecasting
tasks.

## References

  * Ahmed et al. [1974] N. Ahmed, T. Natarajan, and K. R. Rao.  Discrete cosine transform.  _IEEE transactions on Computers_ , 100(1):90–93, 1974.
  * Challu et al. [2022] C. Challu, K. G. Olivares, B. N. Oreshkin, F. Garza, M. Mergenthaler-Canseco, and A. Dubrawski.  N-hits: Neural hierarchical interpolation for time series forecasting, 2022.
  * Cheng et al. [2024] M. Cheng, J. Yang, T. Pan, Q. Liu, and Z. Li.  Convtimenet: A deep hierarchical fully convolutional model for multivariate time series analysis.  _arXiv preprint arXiv:2403.01493_ , 2024.
  * Das et al. [2023] A. Das, W. Kong, A. Leach, R. Sen, and R. Yu.  Long-term forecasting with tide: Time-series dense encoder.  _arXiv preprint arXiv:2304.08424_ , 2023.
  * Eldele et al. [2024] E. Eldele, M. Ragab, Z. Chen, M. Wu, and X. Li.  Tslanet: Rethinking transformers for time series representation learning.  In _International Conference on Machine Learning_ , 2024.
  * Han et al. [2024] L. Han, X.-Y. Chen, H.-J. Ye, and D.-C. Zhan.  Softs: Efficient multivariate time series forecasting with series-core fusion, 2024.
  * Kim et al. [2021] T. Kim, J. Kim, Y. Tae, C. Park, J.-H. Choi, and J. Choo.  Reversible instance normalization for accurate time-series forecasting against distribution shift.  In _International Conference on Learning Representations_ , 2021.
  * Lai et al. [2018] G. Lai, W.-C. Chang, Y. Yang, and H. Liu.  Modeling long- and short-term temporal patterns with deep neural networks, 2018.
  * Li et al. [2023] B. Li, W. Cui, L. Zhang, C. Zhu, W. Wang, I. W. Tsang, and J. T. Zhou.  Difformer: Multi-resolutional differencing transformer with dynamic ranging for time series analysis.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(11):13586–13598, 2023.  doi: 10.1109/TPAMI.2023.3293516.
  * Li et al. [2024] Z. Li, R. Cai, Z. Yang, H. Huang, G. Chen, Y. Shen, Z. Chen, X. Song, Z. Hao, and K. Zhang.  When and how: Learning identifiable latent states for nonstationary time series forecasting, 2024.
  * Lim et al. [2020] B. Lim, S. O. Arik, N. Loeff, and T. Pfister.  Temporal fusion transformers for interpretable multi-horizon time series forecasting, 2020.
  * Liu et al. [2024a] C. Liu, J. Lin, J. Wang, H. Liu, and J. Caverlee.  Mamba4rec: Towards efficient sequential recommendation with selective state space models, 2024a.
  * Liu et al. [2022] M. Liu, A. Zeng, M. Chen, Z. Xu, Q. Lai, L. Ma, and Q. Xu.  Scinet: Time series modeling and forecasting with sample convolution and interaction.  _Advances in Neural Information Processing Systems_ , 35:5816–5828, 2022.
  * Liu et al. [2023a] Y. Liu, T. Hu, H. Zhang, H. Wu, S. Wang, L. Ma, and M. Long.  itransformer: Inverted transformers are effective for time series forecasting.  _arXiv preprint arXiv:2310.06625_ , 2023a.
  * Liu et al. [2023b] Y. Liu, C. Li, J. Wang, and M. Long.  Koopa: Learning non-stationary time series dynamics with koopman predictors, 2023b.
  * Liu et al. [2023c] Y. Liu, H. Wu, J. Wang, and M. Long.  Non-stationary transformers: Exploring the stationarity in time series forecasting, 2023c.
  * Liu et al. [2024b] Y. Liu, H. Zhang, C. Li, X. Huang, J. Wang, and M. Long.  Timer: Transformers for time series analysis at scale, 2024b.
  * Luo and Wang [2024] D. Luo and X. Wang.  Moderntcn: A modern pure convolution structure for general time series analysis.  In _The Twelfth International Conference on Learning Representations_ , 2024.
  * Ni et al. [2024] Z. Ni, H. Yu, S. Liu, J. Li, and W. Lin.  Basisformer: Attention-based time series forecasting with learnable and interpretable basis, 2024.
  * Nie et al. [2022] Y. Nie, N. H. Nguyen, P. Sinthong, and J. Kalagnanam.  A time series is worth 64 words: Long-term forecasting with transformers.  _arXiv preprint arXiv:2211.14730_ , 2022.
  * Oreshkin et al. [2020] B. N. Oreshkin, D. Carpov, N. Chapados, and Y. Bengio.  N-beats: Neural basis expansion analysis for interpretable time series forecasting, 2020.
  * Qi et al. [2024] S. Qi, Z. Xu, Y. Li, L. Wen, Q. Wen, Q. Wang, and Y. Qi.  Pdetime: Rethinking long-term multivariate time series forecasting from the perspective of partial differential equations, 2024.
  * Stasiński [2002] R. Stasiński.  Dct computation using real-valued dft algorithms.  In _2002 11th European Signal Processing Conference_ , pages 1–4, 2002.
  * Toner and Darlow [2024] W. Toner and L. Darlow.  An analysis of linear time series forecasting models, 2024.
  * Trindade [2015] A. Trindade.  ElectricityLoadDiagrams20112014.  UCI Machine Learning Repository, 2015.  DOI:https://doi.org/10.24432/C58C86.
  * Wang et al. [2022] H. Wang, J. Peng, F. Huang, J. Wang, J. Chen, and Y. Xiao.  Micn: Multi-scale local and global context modeling for long-term series forecasting.  In _The Eleventh International Conference on Learning Representations_ , 2022.
  * Wang et al. [2024] H. Wang, L. Pan, Z. Chen, D. Yang, S. Zhang, Y. Yang, X. Liu, H. Li, and D. Tao.  Fredf: Learning to forecast in frequency domain.  _arXiv preprint arXiv:2402.02399_ , 2024.
  * Wu et al. [2021] H. Wu, J. Xu, J. Wang, and M. Long.  Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.  _Advances in neural information processing systems_ , 34:22419–22430, 2021.
  * Wu et al. [2022] H. Wu, T. Hu, Y. Liu, H. Zhou, J. Wang, and M. Long.  Timesnet: Temporal 2d-variation modeling for general time series analysis.  In _The eleventh international conference on learning representations_ , 2022.
  * Xu et al. [2020] K. Xu, M. Qin, F. Sun, Y. Wang, Y.-K. Chen, and F. Ren.  Learning in the frequency domain.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 1740–1749, 2020.
  * Xu et al. [2023] Z. Xu, A. Zeng, and Q. Xu.  Fits: Modeling time series with $10k$ parameters.  _arXiv preprint arXiv:2307.03756_ , 2023.
  * Xue et al. [2024] W. Xue, T. Zhou, Q. Wen, J. Gao, B. Ding, and R. Jin.  Card: Channel aligned robust blend transformer for time series forecasting, 2024.
  * Yang et al. [2024] J. Yang, Y. Li, J. Zhao, H. Wang, M. Ma, J. Ma, Z. Ren, M. Zhang, X. Xin, Z. Chen, and P. Ren.  Uncovering selective state space model’s capabilities in lifelong sequential recommendation, 2024.
  * Ye et al. [2024a] H. Ye, J. Chen, S. Gong, F. Jiang, T. Zhang, J. Chen, and X. Gao.  Atfnet: Adaptive time-frequency ensembled network for long-term time series forecasting, 2024a.
  * Ye et al. [2024b] J. Ye, W. Zhang, K. Yi, Y. Yu, Z. Li, J. Li, and F. Tsung.  A survey of time series foundation models: Generalizing time series representation with large language model, 2024b.
  * Yi et al. [2023] K. Yi, Q. Zhang, W. Fan, S. Wang, P. Wang, H. He, D. Lian, N. An, L. Cao, and Z. Niu.  Frequency-domain mlps are more effective learners in time series forecasting, 2023.
  * Zeng et al. [2022] A. Zeng, M. Chen, L. Zhang, and Q. Xu.  Are transformers effective for time series forecasting?, 2022.
  * Zhang and Yan [2022] Y. Zhang and J. Yan.  Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting.  In _The eleventh international conference on learning representations_ , 2022.
  * Zhou et al. [2021] H. Zhou, S. Zhang, J. Peng, S. Zhang, J. Li, H. Xiong, and W. Zhang.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  In _The Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Conference_ , volume 35, pages 11106–11115. AAAI Press, 2021.
  * Zhou et al. [2022a] T. Zhou, Z. Ma, X. wang, Q. Wen, L. Sun, T. Yao, W. Yin, and R. Jin.  Film: Frequency improved legendre memory model for long-term time series forecasting, 2022a.
  * Zhou et al. [2022b] T. Zhou, Z. Ma, Q. Wen, X. Wang, L. Sun, and R. Jin.  Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting.  In _International conference on machine learning_ , pages 27268–27286. PMLR, 2022b.
  * Zhou et al. [2023] T. Zhou, P. Niu, X. Wang, L. Sun, and R. Jin.  One fits all:power general time series analysis by pretrained lm, 2023.


# A Data-driven Nonlinear Recharge Controller for Energy Storage in Frequency
Regulation

Wenting Ma  Dept. of Electrical Engineering
Tsinghua University
Beijing, China
<EMAIL_ADDRESS>Bolun Xu Wenting Ma was a visiting student at
Columbia University in Spring 2020 supported by the Columbia College Visiting
Student Program and Tsinghua University’s International Class of Energy
Internet Program. Dept. of Earth and Environmental Engineering
Columbia University
New York, NY, USA
<EMAIL_ADDRESS>
###### Abstract

Battery energy storage boosts up the response speed of power system frequency
regulation, but must be recharged carefully to minimize the distortion to the
frequency regulation response. This paper proposes a nonlinear feedback
controller to optimize the recharge for storage resources in frequency
regulation. This controller is designed using a data-driven best-hindsight
optimization framework, the resulting nonlinear recharge controller’s gain
depends on the storage state of charge as well as its power and energy rating.
The developed controller is compared with two benchmark automatic generation
control designs, one is a proportional-integral-based control from PJM
Interconnection, the other one is based on linear-quadratic regulator.
Simulation results using real area control error data from PJM Interconnection
show the proposed controller achieves smaller deviations in both the area
control error and the storage state of charge compared to the two benchmark
controllers under various storage configurations.

###### Index Terms:

Frequency regulation, Energy storage, Nonlinear control, Data-driven control

##  I Introduction

Frequency regulation was one of the first grid-scale applications for battery
energy storage (BES) and was considered an ideal BES application due to its
high requirement of response speed and low requirement of energy capacity.
Power system frequency stability depends on the balance between generations
and demands, and is critical to the reliable supply of electric energy. Units
participating in frequency regulation by adjusting their active power output
according to automatic generation control (AGC) signals instructed by the
system operator. With the increasing penetration of intermittent renewable
generations, conventional thermal units can hardly follow the disturbance
dynamics resulting from the loss of rotational inertia and increasing time-
variant generation [1], and BES becomes an ideal choice to boost-up the
response speed regulation services and reduce the capacity procurement
requirement [2].

Yet, a new challenge arises in systems where BES participants are taking up a
significant market share in frequency regulation services: _to manage the
recharge need of these BES while minimizing the distortion to the regulation
response_. PJM Interconnection, the largest system operator in North America,
was a pioneer in integrating BES and its regulation market share of BES peaked
50% in 2018 [3]. These BES follow a filtered AGC signal (RegD signal) to make
sure the net energy consumption is less than 15 minutes (normalized by the
rated power of the BES), so that BES with limited energy capacity can be
utilized more effectively than through the traditional regulation signal (RegA
signal) [4]. However, PJM has experienced cases that the recharge need for
these storage out-weights the AGC, or these BES have no more energy left when
the system is still experiencing a contingency. As a result, a higher BES
ratio can worsen the regulation quality and weaken the system reliability, and
PJM has revised its AGC design and enforced a 40% cap on storage participation
[5].

The key challenge in designing AGC for storage is to incorporate its
charge/discharge efficiency and tight energy limits, which are nonlinear
dynamics that cannot be well addressed by common linear system controllers
such as proportional-integral (PI) or linear-quadratic regulator (LQR) [6]. On
the other hand, frequency fluctuations in power systems are non-Gaussian due
to human interactions and renewable intermittency [7], which also degrades the
performance of traditional LQR controllers, while emerging researches on data-
driven AGC have not considered the recharge need for storage resources [8].
The novelty of this paper is the design of a controller that learns the
optimal control policies from historical regulation signals without assuming
any underlying distributions and actively incorporates storage constraints.
The policy is implemented as a lookup table and can be seamlessly incorporated
into existing PI-based AGC designs for managing storage recharge, while the
rest of the AGC, especially the control loop for conventional generators,
remains unchanged. The main contributions of this paper can be summarized as
follows:

  * •

We design a nonlinear feedback controller to manage storage state of charge
(SoC) in frequency regulation using a data-driven best-hindsight optimization
[9] approach.

  * •

We conclude best-hindsight solutions into a feedback gain lookup table and
incorporate it with AGC for conventional generators.

  * •

We tested the performance of the controller against benchmark AGCs under
different BES configurations using historical PJM regulation signals.

The rest of the paper is organized as follows: Section II presents the problem
formulation and introduces benchmark controllers, Section III describes the
best-hindsight optimization problem and policy selection of proposed BES
controller. Section IV evaluates the performance of the controller by
simulation, and performs a quantitative correlation analysis between initial
battery states and controller gains, and finally section V concludes with our
findings.

##  II Formulation and Benchmarks

We start by introducing the set-up for the frequency regulation simulation
environment and unit response models, then introduce the two benchmark AGC
designs. We also introduce an anti-windup design into the AGC to adapt linear
controllers for handling tightly constrained BES units.

###  II-A Simulation Framework

To compare the performance of different controllers, we first set up a
frequency regulation simulation environment according to the PJM AGC design
manual [5]. Fig. 1 illustrates the flowchart of this model. Signal input into
this simulation is the uncorrected area control error (ACE) signal, which can
be reconstructed using the historical PJM regulation signals, unit capacity,
and the unit response model (See Section II-B). The ACE signal is the sum of
the corrected ACE signal plus the response from all regulation units, and it
is feed into the AGC and generates RegA and RegD signals to conventional
generators and BES participants. Responses from all regulation units are added
to the ACE. Note that this framework is similar to signal tracking and does
not consider the frequency dynamics, because in U.S. frequency regulation
belongs to the secondary frequency control tier and its propose is to mitigate
ACE and relieve the primary response, instead of stabilizing system dynamics
[10].

Figure 1: The AGC block diagram.

###  II-B Modeling of the Conventional Generators and BES

Conventional generator response model [11] is described in Fig. 2 (a). After
the controller sends RegA signal to generators, a deadband is applied to
prevent excessively frequent response. $G_{g}(s)=1/(T_{g}s+1)$ is the governor
dynamics. Power output of generators $P_{g}$ is constrained by ramp rates and
maximum generation capacity.

Figure 2: (a) Conventional generator model. (b) BES model

The response model of BES is described in Fig. 2 (b), which considers charging
and discharging efficiency as well as SoC constraints. $e$ is the SoC of BES,
and is fed back to the controller.

###  II-C PJM Conditional Neutrality Controller

Beginning on Jan.9, 2017, the PJM’s new conditional neutrality AGC [5] was put
into production. It is a hybrid proportional-integral (PI) controller
including a RegD integral feedback loop so that RegD is energy neutral. This
AGC sends slow RegA signal to conventional generators with limited ramping
capability and unlimited energy, and send the remaining RegD signal to faster
resources with limited energy[4].

Figure 3: Structure of PJM Conditional Neutrality Controller.

The structure of this controller is shown in Fig. 3, where $P_{ACE}$ denotes
filtered ACE, $G_{f1}(s)=1/(T_{a}s+1)$ is the low pass filter for RegA signal,
and $G_{f2}(s)=1/(T_{d}s+1)$ is the low pass filter for RegD signal. The PI
control implemented in this controller is given as the following:

$\displaystyle P_{AGC}$ $\displaystyle=-K_{P}P_{ACE}-K_{I}I_{ACE}$ (1a)
$\displaystyle I_{ACE}$ $\displaystyle=\int P_{ACE}dt$ (1b)

$P_{AGC}$ is the signal output of PI controller, $K_{P}$ and $K_{I}$ denotes
controller gains of the proportional term and integral term. Parameters of
limiters are set according to output power constraints on RegA and RegD
resources within the area.

###  II-D LQR-based AGC

The PJM controller determines the RegD signal as the residual between the
actual AGC and the low-passed RegA signal. This prohibits adjustments of the
control policy to further minimize corrected ACE ($P_{ACE}$ in Fig. 1) for a
linear time-invariant system. Besides, a regulating action performed by BES
suffers efficiency losses so that storage will slowly become empty. We
introduce another benchmark AGC based on LQR to determine the trade-off
between AGC performance and regulation costs. Although the use of LQR in
designing AGC is questionable because the regulation system is nonlinear and
ACE disturbances are not Gaussian, our goal is to provide another benchmark in
the comparison, and this LQR-based AGC does show competitive performance
compared to the PJM AGC in simulation.

The continues time-domain LQR AGC formulation is

$\displaystyle\min_{x,u}\;$ $\displaystyle
J:=\int_{t_{0}}^{\infty}x^{T}Qx+u^{T}Ru$ (2a)
$\displaystyle\dot{x}=Ax+Bu+\omega$ (2b) The LQR AGC uses a pre-set PI control
for RegA and optimizes the RegD control, the states include ACE, RegA,
integral of ACE, and BES SoC. The control is the RegD signal. $\displaystyle
x=$ $\displaystyle[P_{ACE},RegA,I_{ACE},e]^{T}$ (2c) $\displaystyle u_{t}=$
$\displaystyle RegD$ (2d) Based on the modeling approach of system dynamics
proposed in [12], the state and input matrices $A$ and $B$ can be given as
$\displaystyle A=$
$\displaystyle\begin{bmatrix}-\frac{1}{M}&\frac{1}{M}&0&0\\\
-\frac{K_{P}}{T_{a}}&-\frac{1}{T_{a}}&-\frac{1}{T_{a}}&0\\\ K_{I}&0&0&0\\\
0&0&0&0\end{bmatrix}$ (2e) $\displaystyle B=$
$\displaystyle\begin{bmatrix}\frac{1}{M}&0&0&1\end{bmatrix}$ (2f) where $M$ is
the normalized inertia constant. The LQR generates a state-feedback control
for RegD as $u=-Kx$ (2g) where $K$ is the regulation control gain of system
states.

###  II-E Anti-windup Control

We add anti-windup control [13] to all AGC designs to avoid integral wind-up
in the integrals caused by control saturation. Nonentities in regulation
units, including governor dead-bands, ramp rates, and storage energy
constraints, introduces nonlinear dynamics and sometimes even cause
instability [14]. When the actuator saturates as reached a constraint, the
integral term in (1a) tends to accumulate the control error and dominate the
controller, so called _wind-up_ , and a return to normal control operations
becomes difficult due to the large integral value. Wind-up effects are more
likely to occur in systems with high BES share as storage may frequently reach
upper or lower energy limits [3], though they are less noticeable in
traditional AGC because conventional generators are less likely to be
constrained,

We modify the integrator in (1b) by adding a comparing term of the actual
input and commanded input. As shown in Fig. 1, we suppose certain states of
actuators can be measured and fed back to the controller. This includes the
actual generators output $P_{g}$, BES power output $P_{e}$, and SoC of BES
$e$. The integrator is now written as:

$I_{ACE}=\int[P_{ACE}+(RegA-P_{g})+(RegD-P_{e})]dt$ (3)

The scheme is not in effect when system’s actual output is synchronized with
the control signal, but feeds back the mismatch when non-linearities are
encountered.

##  III Optimization-aided Controller Design

We introduce a data-driven best-hindsight optimization problem based on the
existing PJM AGC to derive the optimal recharge policy for BES given their
nonlinear dynamics, and implement this policy as a look-up table. Our storage
control (RegD) is defined as a PI control combined with a nonlinear SoC
feedback control as

$\displaystyle
RegD_{t}=-K^{D}_{p}P_{ACE,t}-K^{D}_{I}I_{ACE,t}-f_{SoC}(e_{t})e_{t}$ (4)

where $K^{D}_{p}$ and $K^{D}_{I}$ are proportional and integral control gains
for the RegD signal in $MW$, while $f_{SoC}(e_{t})$ is the SoC feedback
control gain and it is a function dependent on the storage state-of-charge
value $e_{t}$ in $MW\cdot h$.

We choose the SoC feedback gain as the only nonlinear control component based
on the observation that during the design process the PI gain does not show
meaningful dependency on the storage SoC. Also, system operators usually have
pretty good knowledge about the choice of PI gains for their AGC but are less
familiar with how to manage storage SoC, therefore this study focused on the
SoC feedback gain and assume it to be a scalar function to the storage SoC.

###  III-A Best-hindsight Optimization Formulation

We use a best-hindsight framework to design the feedback gain for state-of-
charge $f_{soc}(e_{t})$ in order to better incorporate the BES non-linearity
due to efficiency and energy limits subjects to the non-Gaussian ACE signal
with unknown distribution ($p<0.001$ in both Jarque-Bera test and Lilliefors
test). In the best-hindsight framework, we solve for the optimal SoC feedback
gain $K_{e}$ over a relatively short period $T$ with a starting time index
$t_{0}$ using historical ACE data as a deterministic optimization problem. We
repeatedly solve this optimization for various $t_{0}$ and initial SoC
$e_{0}$, and record all optimized SoC feedback gains ($K_{e}$), which we
eventually concluded it into a function with respect to the storage state-of-
charge. Such that the control policy is data-driven as it depends on the
historical realizations of the ACE. The formulation for this deterministic
optimization problem is

$\displaystyle\min_{K_{e}}J$
$\displaystyle=\Sigma_{t=t_{0}}^{t_{0}+T}[(P_{ACE,t})^{2}+w_{e}\cdot
e_{t}^{2}]$ (5a) s.t. $\displaystyle RegA_{t}$
$\displaystyle=-K_{p}P_{ACE,t}-K_{I}I_{ACE,t}$ (5b) $\displaystyle RegD_{t}$
$\displaystyle=-K^{D}_{p}P_{ACE,t}-K^{D}_{I}I_{ACE,t}-K_{e}e_{t}$ (5c) RegD
control as in (4) system and unit response models as in Fig. 1 & 2

This program optimizes a SoC feedback gain $K_{e}$ that minimizes the
objective using given PI control settings. (5a) is the cost function, where
$w_{e}$ is the weight coefficient on its SoC deviation term. By modifying the
value of $w_{e}$, the relative importance between reducing output ACE and
keeping SoC close to its desired state is adjusted. (5b) calculates the RegA
signal using a pre-set PI control setting, and (5c) calculates the RegD signal
using a different PI control setting and a SoC feedback gain $K_{e}$, which is
the decision variable in this problem. The two regulation signals are then
sent into unit response models as shown in Fig. 2 according to the simulation
framework in Fig. 1.

###  III-B Solution Method

We obtain the feedback gain function $f_{SoC}$ by fitting results of $K_{e}$
in (5) to a uni-variate function depending on storage SoC $e$. (5) is a
nonlinear optimization problem because of the quadratic objective and the use
of saturation functions, and it is solved using the fmincon function from
MATLAB.

We train $f_{SoC}$ using historical ACE signal $w_{t}$ over a period
$t\in\\{1,2,\dotsc,H\\}$ with following steps.

  1. 1.

Discretize value range of SoC $e\in[0,E]$ into $N=500$ bins:
$\\{e_{n}|n\in\\{1,2,\dotsc,N\\}\\}$.

  2. 2.

For $t_{0}=1:(H-T)$

Solve (5), record the solution $K_{e}$ and its corresponding initial SoC level
$e$: $K_{e}(t_{0})$, $e(t_{0})$.

  3. 3.

$f_{SoC}(e)$ is obtained by computing the average value of recorded $K_{e}$s:

$f_{SoC}(e)=\frac{\sum_{t_{0}=1}^{H-T}K_{e}(t_{0})\mathbbm{1}\\{e_{n-1}\leq
e(t_{0})<e_{n}\\}}{\sum_{t_{0}=1}^{H-T}\mathbbm{1}\\{e_{n-1}\leq
e(t_{0})<e_{n}\\}}$ (6)

###  III-C Training Results and Insights

We fix RegA capacity to $C_{a}=400MW$ according to PJM regulation requirements
[5]. We train the lookup table $f_{SoC}(e)$ under various BES configurations.
The obtained feedback gain function $f_{SoC}(e)$ is nonlinear, as visualized
Fig. 4. Here, a larger BES energy capacity is represented by a longer
discharge time. Feedback gains from different BES settings share some common
features, such as the gain is at minimal when the SoC is around the reference
50% level, and increases as the SoC approaches storage limits. An interesting
result is that the feedback gain is reduced when BES is nearly empty (SoC less
than 10%), which is likely due to that the system is experiencing
contingencies and has a persistent demand for regulation generation that
depletes the storage. A large recharge power in this scenario may add too much
stress to the AGC and worsen the overall regulation quality, which is a
similar concept to PJM’s current practice of manually disabling the energy
neutrality control during contingencies.

Figure 4: Averaged recharge control policy function $f_{SoC}(e)$ under various
BES configurations. The feedback gain is applied to the SoC deviation to the
reference $50\%$ target in MWh units.

##  IV Simulation and Results

We present simulation results of the proposed controller and benchmark
controllers using real uncorrected ACE data from PJM, i.e., the ACE of the
system assuming there is no regulation response. The uncorrected ACE data is
reconstructed using historical regulation signals and unit response models
according to the PJM AGC manual [5]. 30 days of 2-second-resolution data are
used to construct the proposed recharge policy, and the controller performance
is evaluated using another 15 days of data. All AGC have the same setting of
PI control gains for conventional regulation units (RegA) with $K_{P}=0$,
$K_{I}=0.4$. The rest of the benchmark PJM controller parameters are based on
the PJM manual [5]. The benchmark LQR controller is tuned to achieve the best
control performance using cost weights $Q(1,1)=1$, $Q(2,2)=(C_{d}/C_{a})^{2}$,
$Q(3,3)=0$, $Q(4,4)=1/E^{2}$, and $R=1$. The resulting state-feedback control
gain $K$ is $K=[0.4309,-0.0339,0.1210,8]$. In our simulations, the LQR control
performance is not sensitive to different storage settings so the same $K$ is
used for all storage cases. For the proposed controller, we set the RegD PI
control setting to $K^{D}_{P}=1$ and $K^{D}_{I}=0.8$, the window length $T$
for best-hindsight optimization in (5a) to 15 minutes. The SoC deviation
weight $w_{e}$ is tuned using an exhaustive search in each storage setting to
achieve the best overall control performance.

We simulate 400MW RegA capacities (i.e. conventional generators) with varying
RegD capacities as well as the BES energy capacity. We assume all RegD is
provided only by BES, and assume all BES units have identical physical
parameters, initial SoC, and response characteristic, so all BES units can be
aggregated as a single BES in the design and simulations. The BES round-trip
efficiency is 85%. Fig. 5 demonstrates comparisons of the uncorrected ACE,
responses from different AGC designs, and the resulting total SoC from all
storage.

Figure 5: Comparison of controller performance. Overall, the three controllers provide similar shapes of ACE control results and SoC profiles, while the proposed control achieves the smallest ACE deviations and SoC deviations.  TABLE I: Controller Performance Under Different BES Configurations BES | $\overline{(P_{ACE})^{2}}/\times 10^{3}$ | $\overline{e^{2}}$
---|---|---
Config. | Prop. | LQR | PJM | Prop. | LQR | PJM
200MW/15min | 32.6 | 34.3 | 36.5 | 110.7 | 101.9 | 116.1
200MW/20min | 31.0 | 32.4 | 33.9 | 152.3 | 160.1 | 174.7
200MW/30min | 29.5 | 29.8 | 30.6 | 216.4 | 310.9 | 323.4
200MW/60min | 24.5 | 26.4 | 26.3 | 730.3 | 1015.7 | 1008.6
300MW/15min | 28.6 | 29.8 | 31.3 | 158.7 | 206.5 | 176.7
400MW/15min | 26.6 | 29.8 | 28.8 | 189.2 | 370.4 | 218.9

We use two indexes to compare the control performance of different AGC
designs: the average values of output ACE square $\overline{(P_{ACE})^{2}}$
and the SoC deviation square $\overline{e^{2}}$, where we set $e_{0}=50\%$ as
our desired state of BES. Table I shows the results over the 15 day test
period. Note that under the BES configuration of 200MW/60min, we adjust the
parameter settings for the proposed controller to $T$=30min and $w_{e}$ = 10
in order to obtain best performance. The proposed controller is observed to
reduce output ACE error by up to 10.74% compared to LQR controller, and 10.68%
compared to PJM controller. Moreover, it reduces the SoC deviation by up to
48.92% compared to LQR controller, and 33.09% compared to PJM controller. This
improvement in reducing extreme charge and discharge is especially significant
for BES with greater energy capacity such as 200MW/30min and 200WM/60min.

It is worth noting that the performance of the proposed controller depends
heavily upon the tuning of SoC deviation weight setting $w_{e}$. The current
setting of $w_{e}=60$ achieves the overall best performance under a BES
configuration of 300MW/15min, where output ACE error is reduced by 4.16%
compared to LQR controller, and SoC deviation is reduced by 10.20%. Meanwhile,
an improvement is expected by tuning $w_{e}$ under a rather under-performing
configuration of BES.

##  V Conclusion

We developed a data-driven nonlinear feedback control to optimize the recharge
need for battery energy storage in frequency regulation. This feedback
controller uses a best-hindsight framework that models non-linear system
dynamics and non-stochastic disturbances, and can incorporate with any
existing AGC design. Simulation results using real ACE data shows the proposed
controller achieved significant improvements compared to two benchmark AGC in
both controlling ACE and preventing excessive use of ESS that might exacerbate
degrading. Moreover, it is capable of adapting to different BES configuration
by adjusting relative weights accordingly. The correlation between control
gains and initial SoC is analyzed to help understand the effect of BES
capacity constraints on the controller’s choice of control policy.

The key insight from our study is that energy storage recharge can be better
managed with nonlinear controllers, which were not considered by any existing
AGC design. We hope results from this paper could incentivize new ideas on how
to better incorporate storage resources in ancillary services to achieve
better grid reliability and sustainability. Future directions includes
generalize this control method over diverse storage resources and to large-
scale interconnected systems composed of multiple balancing areas. Our results
show that new monitoring schemes and market designs are required to improve
the utilization for storage resources.

## References

  * [1] A. Ulbig, T. S. Borsche, and G. Andersson, “Impact of low rotational inertia on power system stability and operation,” _IFAC Proceedings Volumes_ , vol. 47, no. 3, pp. 7290–7297, 2014.
  * [2] Y. V. Makarov, S. Lu, J. Ma, and T. B. Nguyen, “Assessing the value of regulation resources based on their time response characteristics,” Pacific Northwest National Lab.(PNNL), Richland, WA (United States), Tech. Rep., 2008\.
  * [3] B. Xu, Y. Shi, D. S. Kirschen, and B. Zhang, “Optimal battery participation in frequency regulation markets,” _IEEE Transactions on Power Systems_ , vol. 33, no. 6, pp. 6715–6725, 2018.
  * [4] T. Wang, “Battery assisted conventional generator in pjm frequency regulation market,” in _2019 IEEE Power & Energy Society General Meeting (PESGM)_. IEEE, 2019, pp. 1–5.
  * [5] “Implementation and rationale for pjm’s conditional neutrality regulation signals,” [Available Online] http://pjm.com/ /media/committeesgroups/task-forces/rmistf/postings/regulation-market-whitepaper.ashx.
  * [6] A. Isidori, _Nonlinear control systems_. Springer Science & Business Media, 2013.
  * [7] B. Schäfer, C. Beck, K. Aihara, D. Witthaut, and M. Timme, “Non-gaussian power grid frequency fluctuations characterized by lévy-stable laws and superstatistics,” _Nature Energy_ , vol. 3, no. 2, pp. 119–126, 2018.
  * [8] P. Hidalgo-Gonzalez, R. Henriquez-Auba, D. S. Callaway, and C. J. Tomlin, “Frequency regulation using data-driven controllers in power grids with variable inertia due to renewable energy,” in _2019 IEEE Power & Energy Society General Meeting (PESGM)_. IEEE, 2019, pp. 1–5.
  * [9] E. Hazan, S. Kakade, and K. Singh, “The nonstochastic control problem,” in _Algorithmic Learning Theory_. PMLR, 2020, pp. 408–421.
  * [10] B. Xu, Y. Dvorkin, D. S. Kirschen, C. A. Silva-Monroy, and J.-P. Watson, “A comparison of policies on the participation of storage in us frequency regulation markets,” in _2016 IEEE Power and Energy Society General Meeting (PESGM)_. IEEE, 2016, pp. 1–5.
  * [11] K. Doenges, I. Egido, L. Sigrist, E. Lobato, and L. Rouco, “Improving agc performance in power systems with regulation response accuracy margins using battery energy storage system (bess),” _IEEE Transactions on Power Systems_ , 2019.
  * [12] Q. Liu and M. D. Ilić, “Enhanced automatic generation control (e-agc) for future electric energy systems,” in _2012 IEEE Power and Energy Society General Meeting_. IEEE, 2012, pp. 1–8.
  * [13] A. Zheng, M. V. Kothare, and M. Morari, “Anti-windup design for internal model control,” _International Journal of Control_ , vol. 60, no. 5, pp. 1015–1024, 1994.
  * [14] W. Tan, S. Chang, and R. Zhou, “Load frequency control of power systems with non-linearities,” _IET Generation, Transmission & Distribution_, vol. 11, no. 17, pp. 4307–4313, 2017.