#  LightCTS: A Lightweight Framework for Correlated Time Series Forecasting

Zhichen Lai†, Dalin Zhang†∗, Huan Li‡∗, Christian S. Jensen†, Hua Lu§, Yan
Zhao† †Department of Computer Science, Aalborg University, Denmark
‡College of Computer Science and Technology, Zhejiang University, China
§Department of People and Technology, Roskilde University, Denmark

(2023)

###### Abstract.

Correlated time series (CTS) forecasting plays an essential role in many
practical applications, such as traffic management and server load control.
Many deep learning models have been proposed to improve the accuracy of CTS
forecasting. However, while models have become increasingly complex and
computationally intensive, they struggle to improve accuracy. Pursuing a
different direction, this study aims instead to enable much more efficient,
lightweight models that preserve accuracy while being able to be deployed on
resource-constrained devices. To achieve this goal, we characterize popular
CTS forecasting models and yield two observations that indicate directions for
lightweight CTS forecasting. On this basis, we propose the LightCTS framework
that adopts plain stacking of temporal and spatial operators instead of
alternate stacking which is much more computationally expensive. Moreover,
LightCTS features light temporal and spatial operator modules, called L-TCN
and GL-Former, that offer improved computational efficiency without
compromising their feature extraction capabilities. LightCTS also encompasses
a last-shot compression scheme to reduce redundant temporal features and speed
up subsequent computations. Experiments with single-step and multi-step
forecasting benchmark datasets show that LightCTS is capable of nearly state-
of-the-art accuracy at much reduced computational and storage overheads.

correlated time series forecasting, lightweight neural networks

*corresponding authors: D. Zhang<EMAIL_ADDRESS>and H. Li<EMAIL_ADDRESS>

††journalyear: 2023††copyright: acmcopyright††conference: the 2023
International Conference on Management of Data; June 18–23, 2023; Seattle, WA,
USA††price: 15.00††doi: nnnnn/nnnnnn.nnnnn††ccs: Information systems Spatial-
temporal systems††ccs: Information systems Data mining

##  1\. Introduction

Figure 1. DL-based CTS forecasting frameworks using (a) alternate stacking and
(b) plain stacking.

Driven in part by the availability of increasingly advanced and affordable
sensor technologies, cyber-physical systems (CPSs) (Derler et al., 2011) are
being deployed at a rapid pace. In a typical CPS, multiple sensors sample
physical processes of interest and emit multiple time series with
correlations. One example is sensors that sample power production by
photovoltaic installations in a geographical region (Lai et al., 2018).
Extracting and exploiting correlations in correlated time series (CTS) is
important in many applications, such as the forecasting of traffic situations
(Cirstea et al., 2021; Wu et al., 2021; Papadimitriou and Yu, 2006; Yuan et
al., 2020), air quality (Du et al., 2019), server loads (Faloutsos et al.,
2019; Ma et al., 2018), social activity (Zhu and Shasha, 2002), and wind farm
maintenance (Cheng et al., 2022). In this study, we focus on CTS forecasting.
One significant application occurs in the predictive maintenance of wind
turbines (Cheng et al., 2022), which are often deployed in remote and harsh
locations. Accurate and instant forecasts of a turbine’s operating status,
e.g., covering pitch speed and active power, can enable identification of
potential failures, thereby enabling timely maintenance, and thus reducing
regular maintenance costs, decreases in generated power, and potential safety
hazards (Lai et al., 2022). Hence, forecasting has attracted extensive
research attention.

Deep learning (DL) techniques have recently shown impressive CTS forecasting
performance. A variety of DL modules, such as convolutional neural networks
(CNNs) (Yu et al., 2018b; Wu et al., 2019, 2020; Guo et al., 2019; Huang et
al., 2020; Chen et al., 2022; Rao et al., 2022), recurrent neural networks
(RNNs) (Bai et al., 2020; Chen et al., 2020a; Li et al., 2018; Chang et al.,
2018; Shih et al., 2019), graph convolutional networks (GCNs) (Li et al.,
2018; Pan et al., 2021; Wu et al., 2019, 2021; Guo et al., 2019; Chen et al.,
2022; Rao et al., 2022), and Transformers (Xu et al., 2020; Park et al., 2020;
Zhou et al., 2021; Wu et al., 2021), are used to construct operators for
extracting temporal features from individual time series or spatial features
across correlated time series. These two categories of operators are referred
to as temporal operators (T-operators) and spatial operators (S-operators)
(see the categorization in Table 2). Studies (Yu et al., 2018b; Wu et al.,
2021) show that such operators are effective at feature extraction and enable
state-of-the-art CTS forecasting accuracy. We analyze the commonalities of
these DL-based models and present a generic framework as shown in Figure 1(a).
The framework starts with an embedding module that ingests the raw CTS data;
then, multiple spatio-temporal blocks (ST-blocks) are stacked, each consisting
of a sequence of alternate S- and T-operators for extracting high-order
spatio-temporal features (ST-features); the framework ends with a module that
aggregates ST-features at different depths with residual connections and
outputs a forecast.

However, DL-based CTS models are often large, and inferencing is often
computationally expensive. This limits the possibilities of deploying CTS
forecasting on resource-constrained edge computing devices, which is otherwise
attractive in CPS applications due to the decentralized computation and
potentially increased service responsiveness (Shi et al., 2016). As an
example, there is a pressing need to monitor and forecast the working status
of wind turbines in real time for maintenance (Cheng et al., 2022). However,
wind turbines are often deployed offshore or in high latitudes, and
transmitting their operational data to a remote data center is costly,
lagging, and fragile to the quality of networking. To detect potential faults
in a timely manner and respond on board, an appealing solution is to place a
lightweight model on an edge device operating locally. Microcontroller units
(MCUs) are widely used edge devices in industry due to their stability and low
cost (Sudharsan et al., 2021). However, they have very limited memory. The
popular STM32F4 series of MCUs have up to 3Mb of memory, which is insufficient
for the deployment of state-of-art CTS models like GwNet (Wu et al., 2019).
Moreover, as existing CTS models are not specifically designed for lightweight
applications, simply lightening these models degrades their performance
dramatically (see Table 10).

Moreover, we observe that although recent studies on CTS forecasting focus
mainly on improving accuracy, progress has almost come to a standstill. For
example, AutoCts (Wu et al., 2021), a state-of-the-art forecasting model,
improves GwNet (Wu et al., 2019), a previous state-of-the-art model, by up to
0.06 miles per hour (mph) in terms of mean absolute error (MAE) on traffic
speed forecasting. AutoCts models are much larger and are obtained through
neural architecture search, which involves the training and evaluation of
thousands of large models, which may take up to hundreds of GPU hours,
incurring considerable $CO_{2}$ emissions (Chen et al., 2021). This situation,
characterized by increasingly larger and computationally expensive models with
diminishing accuracy improvements, motivates a different direction, where we
instead aim to achieve lightweight models with competitive accuracy. This will
enable edge computing as well as overall emissions savings (Shi et al., 2016).

Although several lightweight techniques have been proposed in computer vision
(Tan and Le, 2019; Zhang et al., 2018; Han et al., 2020; Li et al., 2021),
these techniques are not readily applicable to CTS forecasting. A key reason
is that lightweight computer vision models focus mainly on simplifying 2D and
3D convolutions for image and video data, while CTS models involve instead 1D
convolution of temporal features and graph convolution of spatial features. In
addition, recent lightweight Transformers (Mehta and Rastegari, 2021; Liu et
al., 2021) reduce the computational cost of the self-attention mechanism by
utilizing the similarities among adjacent and multi-scale image patches, which
are not applicable to CTS data.

We propose LightCTS, a framework that enables lightweight CTS forecasting at
significantly reduced computational cost while retaining forecasting accuracy
comparable to the state-of-the-art. We start with a comprehensive analysis of
existing CTS models, placing them in a generic framework (see Figure 1(a)) and
scrutinizing the computational and storage overheads of their components, both
theoretically and empirically. The analysis yields important observations (see
Section 3) and points to two directions for achieving lightness, namely 1)
simplifying computations associated with ST-feature extraction and 2)
optimizing the generic CTS architecture and compressing redundant temporal
dimensions for costly S-operators.

By following these directions, LightCTS offers a set of novel lightweight
techniques. First, LightCTS includes a novel T-operator module called _Light
Temporal Convolutional Network_ (L-TCN) and a novel S-operator module called
_GlobalLocal TransFormer_ (GL-Former) for temporal and spatial feature
extraction, respectively. Both L-TCN and GL-Former adopt grouping strategies
to reduce the full connections between adjacent layers to local in-group
connections, thus achieving lower complexity than the vanilla TCN and
Transformer. Moreover, LightCTS adopts a simple but effective _plain stacking_
based architecture (see Figure 1(b)) that decouples temporal and spatial
feature extraction and renders the compression and reduction of intermediate
features more flexible. Along with plain stacking, a _last-shot compression_
scheme is employed that retains only the last time-step slice of temporal
features extracted by T-operators. This scheme reduces the features that are
fed to subsequent components with only minor information loss, as TCNs tend to
capture the most significant features in the last time step (Lea et al.,
2016). The plain stacking and last-shot compression combine to considerably
lower the computational and storage overheads of the subsequent S-operators as
well as of the aggregation and output module.

Considering both single-step and multi-step CTS forecasting, we conduct
extensive experiments to evaluate LightCTS on six benchmark datasets. We find
that LightCTS achieves accuracy comparable to those of state-of-the-art
models, but with much lower computational and storage overheads. We have made
our implementation publicly available111https://github.com/AI4CTS/lightcts.

The contributions of the paper are summarized as follows.

  * •

We propose LightCTS, a novel lightweight CTS forecasting framework. To the
best of our knowledge, this is the first study of lightweight DL-based CTS
forecasting.

  * •

We analyze the architectures, S/T-operators, and resource costs of mainstream
CTS models, and identify key opportunities for achieving lightness.

  * •

We contribute L-TCN and GL-Former, two novel lightweight T- and S-operator
modules, targeting the extraction of ST-features of CTS. We also propose a
plain stacking pattern and a last-shot compression scheme, targeting a
reduction of the sizes of the inputs to S-operators and the aggregation and
output module.

  * •

We report on experimental findings for different tasks, offering evidence that
LightCTS is capable of state-of-the-art accuracy while reducing computational
costs and model sizes very substantially.

Section 2 covers the definition of CTS and its forecasting tasks; Section 3
analyzes the commonalities of existing CTS models; Section 4 details the
design of KDCTS; Section 5 reports on the experimental study; Section 6 covers
related work on CTS forecasting and knowledge distillation of DL models;
finally, Section 7 concludes and presents research directions.

##  2\. Preliminaries

This section covers preliminaries of CTS and formalizes the problem of CTS
forecasting. Frequently used notations are summarized in Table 1.

Table 1. Summary of notation.

Notation | Description
---|---
$X$ | An indexed set of correlated time series (CTS)
$\mathtt{N}$ | Number of time series in $X$
$\mathtt{T}$ | Number of time steps in $X$
$\mathtt{D}$ | Embedding size of S/T-operators
$\mathtt{P}$ | Number of historical time steps used in CTS forecasting
$\mathtt{Q}$ | Number of future time steps of CTS forecasting

###  2.1. Correlated Time Series

In a cyber-physical system (Derler et al., 2011), $\mathtt{N}$ devices each
generate timestamped data, yielding $\mathtt{N}$ time series. The time series
are called correlated time series (CTS) (Wu et al., 2021), denoted as
$X\in\mathbb{R}^{\mathtt{N}\times\mathtt{T}\times\mathtt{F}}$, where
$\mathtt{T}$ and $\mathtt{F}$ denote the number of time steps and the number
of sensor measurements per time step, respectively. For example, in a wind
turbine farm consisting of 30 turbines, each turbine may emit wind speed and
wind direction measurements at each time step; thus, if the system emits
measurements for 500 time steps, we get $X\in\mathbb{R}^{30\times 500\times
2}$.

Given the $\mathtt{N}$ time series, two kinds of correlations occur: temporal
correlations within time series and spatial correlations across different time
series. On the one hand, consecutive measurements in a time series are
naturally correlated. On the other hand, concurrent measurements by different
devices may be correlated due to, e.g., the spatial proximity of the devices.
For example, traffic flows reported by sensors on connected road segments are
naturally correlated (Pedersen et al., 2020; Wu et al., 2021).

###  2.2. CTS Forecasting Problems

We consider single-step and multi-step CTS forecasting. First, single-step CTS
forecasting aims to predict the $\mathtt{Q}$-th future time step
($\mathtt{Q}\geq 1$); formally,

(1)
$\hat{X}_{t+\mathtt{P+Q}}\leftarrow\mathcal{SF}(X_{t+\mathtt{1}},\ldots,X_{t+\mathtt{P}}),$

where $t$ indexes the beginning time step, $\mathtt{P}$ is the number of
historical time steps used for forecasting, $\hat{X}_{t+\mathtt{P+Q}}$ denotes
the predicted CTS at the future $(t+\mathtt{P+Q})$-th time step, and
$\mathcal{SF}$ denotes a single-step CTS forecasting model.

Next, multi-step CTS forecasting aims to predict $\mathtt{Q}$ ($\mathtt{Q}>1$)
consecutive future time steps in one pass; formally,

(2)
$\\{\hat{X}_{t+\mathtt{P+1}},\ldots,\hat{X}_{t+\mathtt{P+Q}}\\}\leftarrow\mathcal{MF}(X_{t+\mathtt{1}},\ldots,X_{t+\mathtt{P}}),$

where $\mathcal{MF}$ denotes a multi-step CTS forecasting model.

For both problems, it is essential to extract the temporal dynamics in each
time series and the spatial correlations among different time series from the
historical data. To this end, deep learning (DL) techniques with powerful
temporal and spatial feature extraction capabilities have been used widely in
CTS models. A review of existing DL-based CTS models is provided in Section 6.
Due to the characteristics of the neural network operators used to extract
spatial and temporal features, the training and inferencing of DL-based CTS
models incur considerable computational and storage overheads. In this study,
_we aim to enable lightweight CTS forecasting models (i.e., models with fewer
computations and parameters) with forecasting accuracy comparable to the
state-of-the-art CTS forecasting models._

##  3\. Analyses and Directions

In Section 3.1, we place existing DL-based CTS modeling proposals in a generic
framework, map the complexity of their internal operators, and investigate the
computational and storage overheads of representative models. Based on this
analysis, we identify directions for achieving a lightweight CTS framework in
Section 3.2.

###  3.1. Analysis of Existing CTS Models

####  3.1.1. Generic Framework

To gain insight into the prospects of lightening DL-based CTS models, we
consider representative proposals (Li et al., 2018; Yu et al., 2018b; Wu et
al., 2019; Bai et al., 2020; Wu et al., 2020; Xu et al., 2020; Park et al.,
2020; Guo et al., 2019; Chen et al., 2020a; Shih et al., 2019; Wu et al.,
2021; Zhou et al., 2021; Grigsby et al., 2021; Chang et al., 2018; Chen et
al., 2022; Huang et al., 2020; Cirstea et al., 2021; Rao et al., 2022). Figure
1(a) shows a generic framework for these models. Generally, a CTS model has
three components: (1) an _embedding module_ that transforms the raw CTS into
latent representations; (2) a stack of _spatio-temporal blocks_ (ST-blocks)
that extract spatial and temporal correlations as high-order features; and (3)
an _aggregation and output module_ that aggregates ST-features from the ST-
blocks and outputs the result, which is either a single-step or a multi-step
forecast (see Section 2.2).

Being responsible for extracting temporal and spatial correlations, ST-blocks
make up the key component of a CTS model. Typically, an ST-block includes
alternating stacks of _temporal operators_ (T-operators) and _spatial
operators_ (S-operators). For example, the alternate stacking pattern can be
$\langle T,S\rangle$, $\langle S,T\rangle$, $\langle T,S,T\rangle$, etc. The
S/T-operators are the basic ingredients for extracting comprehensible
features.

####  3.1.2. S/T-operators

We proceed to study the S/T-operators employed by state-of-the-art models (Li
et al., 2018; Yu et al., 2018b; Wu et al., 2019; Bai et al., 2020; Wu et al.,
2020; Xu et al., 2020; Park et al., 2020; Guo et al., 2019; Chen et al.,
2020a; Shih et al., 2019; Wu et al., 2021; Zhou et al., 2021; Grigsby et al.,
2021; Chang et al., 2018; Chen et al., 2022; Huang et al., 2020; Cirstea et
al., 2021; Rao et al., 2022). Specifically, we analyze each operator’s time
complexity in terms of _FLOPs_ (floating-point operations) and space
complexity in terms of _the number of model parameters_. Referring to Table 2,
we categorize popular S- and T-operators into different families based on the
base operators that they extend. Considering that there are only minor
differences between the operators in a family (e.g., applying different
attention mechanisms or convolution kernels), we report on the time and space
complexity of the base operators in Table 2. We refer interested readers to
the supplemental material (lig, 2023) for a detailed analysis of base
operators.

Table 2. Categorization and analysis of ST-block operators.

Type | Family | Literature | Time Complexity | Space Complexity
---|---|---|---|---
T-operator | CNN |  |  (Yu et al., 2018b; Wu et al., 2019; Guo et al., 2019; Wu et al., 2020; Cirstea et al., 2021)
---
(Huang et al., 2020; Wu et al., 2021; Chen et al., 2022; Rao et al., 2022)
$\mathcal{O}(\mathtt{D}^{2}\cdot\mathtt{N}\cdot\mathtt{P})$ | $\mathcal{O}(\mathtt{D}^{2})$
RNN |  |  (Bai et al., 2020; Chen et al., 2020a; Chang et al., 2018)
---
(Li et al., 2018; Cirstea et al., 2021; Shih et al., 2019)
$\mathcal{O}(\mathtt{D}^{2}\cdot\mathtt{N}\cdot\mathtt{P})$ | $\mathcal{O}(\mathtt{D}^{2})$
Transformer |  |  (Xu et al., 2020; Park et al., 2020)
---
(Zhou et al., 2021; Wu et al., 2021)
$\mathcal{O}(\mathtt{D}\cdot\mathtt{N}\cdot\mathtt{P}\cdot(\mathtt{P}+\mathtt{D}))$ | $\mathcal{O}(\mathtt{D}^{2})$
S-operator | GCN |  |  (Li et al., 2018; Guo et al., 2019; Wu et al., 2019, 2021)
---
(Cirstea et al., 2021; Chen et al., 2022; Rao et al., 2022)
$\mathcal{O}(\mathtt{D}\cdot\mathtt{N}\cdot\mathtt{P}\cdot(\mathtt{N}+\mathtt{D}))$ | $\mathcal{O}(\mathtt{D}^{2})$
Transformer |  |  (Xu et al., 2020; Park et al., 2020)
---
(Grigsby et al., 2021; Wu et al., 2021)
$\mathcal{O}(\mathtt{D}\cdot\mathtt{N}\cdot\mathtt{P}\cdot(\mathtt{N}+\mathtt{D}))$ | $\mathcal{O}(\mathtt{D}^{2})$

Embedding size $\mathtt{D}$; time series number $\mathtt{N}$; historical time
steps $\mathtt{P}$.

Three main T-operator families are identified: (1) _CNN-based T-operators_ (Yu
et al., 2018b; Wu et al., 2019, 2020; Guo et al., 2019; Huang et al., 2020;
Chen et al., 2022; Cirstea et al., 2021; Rao et al., 2022), specifically
Temporal Convolutional Networks (TCNs), that apply dilated causal convolutions
to time series data; (2) _RNN-based T-operators_ , such as long short term
memory networks (LSTMs) (Shih et al., 2019) and gated recurrent unit networks
(GRUs) (Bai et al., 2020; Chen et al., 2020a; Chang et al., 2018; Li et al.,
2018; Cirstea et al., 2021), that process time series based on a recursive
mechanism; and (3) _Transformer-based T-operators_ (Xu et al., 2020; Park et
al., 2020; Zhou et al., 2021; Wu et al., 2021) that adopt the attention
mechanism to establish self-interactions of input time steps, enabling
weighted temporal information extraction over long sequences. While all
T-operator families have the same space complexity, the time complexity of the
operators in the Transformer family is larger than those of the operators in
the CNN and RNN families because of their large-size matrix multiplication (Wu
et al., 2021). Further, although the RNN family operators have the same time
complexity regarding FLOPs as the CNN family operators, the former adopt a
sequential computation scheme that significantly reduces the actual
efficiency. Thus, CNN-based T-operators are the most promising for lightweight
CTS models. Recent TCN models, including GwNet (Wu et al., 2019), MtGnn (Wu et
al., 2020), and Fogs (Rao et al., 2022), achieve the state-of-the-art
accuracies.

There are roughly two S-operator families: (1) _GCN-based S-operators_ ,
specifically Chebyshev GCNs (Guo et al., 2019; Cirstea et al., 2021; Chen et
al., 2022; Rao et al., 2022) or Diffusion GCNs (Li et al., 2018; Wu et al.,
2019, 2021), utilize predefined or learned spatial adjacency matrices to
capture high-order spatial correlations and (2) _Transformer-based
S-operators_ (Xu et al., 2020; Park et al., 2020; Grigsby et al., 2021; Wu et
al., 2021) cast attention operations across different time series to obtain
their weighted spatial correlations. Theoretically, GCNs and Transformers
incur the same space and time complexities for S-operators (see Table 2).
Moreover, no existing studies compare their CTS forecasting performance in the
same setting. We thus include experiments that compare these two S-operators
in our LightCTS framework. The results, in Section 5.4 and Table 9, show that
a Transformer-based S-operator achieves the best accuracy in our framework.

####  3.1.3. FLOP and Parameter Use in CTS Models

To investigate the resource consumption of each component of CTS models, we
analyze FLOPs and parameters via a case study. We select three
representatives, namely Fogs (Rao et al., 2022) as the most accurate model,
MtGnn (Wu et al., 2020) as the most lightweight model, and GwNet (Wu et al.,
2019) as a widely used model. They are also all included for comparison in the
experimental study in Section 5. We select the METR-LA dataset (see Section
5.1.1) and use the architectures reported by the original papers. Table 3
shows the distribution of FLOPs and parameters associated with different
components of CTS models, namely the embedding module, the T-operators and
S-operators in the ST-component, and the aggregation and output module.

Table 3. Distribution of FLOPs and parameters in CTS models.

Model | Metric | Input | T- | S- | Aggregation
---|---|---|---|---|---
Embedding | operators | operators | and Output
Fogs | FLOPs | 0.01% | 2.13% | 95.72% | 2.15%
Parameters | 0.01% | 3.25% | 77.19% | 19.55%
MtGnn | FLOPs | 0.12% | 23.54% | 70.19% | 6.15%
Parameters | 0.02% | 75.52% | 9.49% | 14.97%
GwNet | FLOPs | 0.03% | 6.70% | 75.23% | 18.04%
Parameters | 0.03% | 10.77% | 19.94% | 69.26%

As expected, a significant portion of FLOPs and parameters occur in the S- and
T-operators that make up the core component of a CTS model. Moreover,
S-operators consume many more FLOPs than T-operators. Surprisingly, the
aggregation and output module is also responsible for many FLOPs and
parameters. This is likely because the aggregation and output module has to
process massive ST-features extracted by all ST-blocks.

###  3.2. Observations and Resulting Directions

We highlight the main observations from the above analyses and identify
promising directions for designing LightCTS.

###### Observation 1.

S- and T-operators, which make up the main component of CTS models (Section
3.1.1), incur significant computational and storage overheads (Table 3). Their
time and space complexities are both proportional to $\mathtt{D}^{2}$ (Table
2).

Observation 1 indicates that it is essential to lighten S- and T-operators. As
presented in Table 2, the time and space complexities of all S/T-operators are
positively correlated with $\mathtt{N}$, $\mathtt{P}$, and $\mathtt{D^{2}}$.
The numbers of time series $\mathtt{N}$ and historical time steps $\mathtt{P}$
are decided by the raw CTS data and should be left unaltered in a CTS model.
Therefore, manipulating the embedding size $\mathtt{D}$ of S/T-operators is a
direction for achieving lightness. Many studies (Wu et al., 2021; Chen et al.,
2020a; Wu et al., 2020), however, have shown that simply reducing $\mathtt{D}$
inevitably degrades forecasting accuracy. Instead, we propose to simplify and
reduce the neural network computations associated with $\mathtt{D}$, to be
detailed in Sections 4.2 and 4.4 for T- and S-operators, respectively.

###### Observation 2.

S-operators consume many more resources, especially FLOPs, than T-operators
(Table 3).

One reason for Observation 2 is that S-operators usually have higher time
complexity than T-operators (see Table 2). The complexity of S-operators can
be reduced by manipulating $\mathtt{D}$ as discussed for Observation 1.
Another major reason is that the input to S-operators must retain the temporal
dimensions even if these contribute little to extracting spatial correlations.
This is a consequence of the alternate stacking pattern of the existing
generic CTS framework (see Section 3.1.1). Specifically, since executions of
S- and T-operators intermixed, some T-operators occur after S-operators. To
allow such T-operators to properly extract temporal features, the temporal
dimensions must be preserved. Although T-operators also receive redundant
spatial information due to intermixing, S-operators are affected more due to
their higher time complexity. Moreover, the spatial dimension $\mathtt{N}$
cannot be compressed as the aim is to forecast future values for all
$\mathtt{N}$ time series.

This observation suggests that a new stacking pattern of S/T-operators that
compresses temporal features before applying S-operators. We thus propose to
design a new stacking pattern that decouples T-operators and S-operators, to
be detailed in Section 4.1. In addition, we devise a temporal feature
compression scheme to condense the input feature maps for S-operators while
preserving key temporal information to be used in the final forecasting, to be
detailed in Section 4.3. As mentioned in Section 3.1.3, the aggregation and
output module takes all features extracted by ST-blocks as input, which incurs
considerable FLOPs and parameters. As a by-product of the temporal feature
compression, the intermediate T- and S-features can be downsized, which leads
to a significant reduction in FLOPs and parameters for the aggregation and
output module.

##  4\. Construction of LightCTS

We start by presenting the new _plain stacking_ pattern in Section 4.1; next,
we detail the light T-operator module in Section 4.2 and its subsequent _last-
shot compression_ in Section 4.3; we then present the light S-operator module
in Section 4.4 and the assembly of LightCTS in Section 4.5.

###  4.1. Architecture with Plain Stacking

As illustrated in Figure 1(a), the conventional CTS architecture relies on ST-
blocks, each of which stacks S- and T-operators alternately. Such an alternate
stacking pattern maintains a feature size of
$\mathtt{N}\times\mathtt{P}\times\mathtt{D}$ throughout feature extraction.
Indeed, the temporal dimension $\mathtt{P}$ is only considered in T-operators
and is disregarded in S-operators. In other words, the output representation
unnecessarily increases the computational and storage overheads of
S-operators.

We deviate from the alternate stacking pattern and instead combine a
T-operator module consisting of $\mathtt{L}_{T}$ T-operators and an S-operator
module consisting of $\mathtt{L}_{S}$ S-operators serially, yielding what we
call the _plain stacking_ scheme. Figure 1(b) presents the novel architecture,
which enjoys several benefits. First, temporal and spatial feature extraction
are decoupled, allowing compression of the temporal dimension before applying
the more complex S-operators (see Observation 2). In particular, we propose a
_last-shot compression_ scheme (to be detailed in Section 4.3) that reduces
the output size of temporal feature extraction from
$\mathtt{N}\times\mathtt{P}\times\mathtt{D}$ to $\mathtt{N}\times\mathtt{D}$.

Moreover, the computational overhead of the aggregation and output module is
also reduced by the feature compression. Indeed, the architecture further
reduces costs by taking as input only the final features of the temporal and
spatial feature extraction phases, instead of the features from all stacked
ST-blocks in the conventional architecture.

The plain stacking architecture does not lower the effectiveness of feature
extraction but reduces computational costs, as will be shown in Section 5.4.
To construct a specific LightCTS model using the new architecture, we design a
light T-operator module L-TCN and a light S-operator module GL-Former,
presented in Sections 4.2 and 4.4.

###  4.2. L-TCN

####  4.2.1. Background of TCN

We propose an Light Temporal Convolutional Network (L-TCN) that is based on
the Gated TCN (Wu et al., 2019), which incorporates the gating mechanism into
standard TCNs to control the temporal information flow. A TCN adopts dilated
causal convolutions (DCC) (Yu and Koltun, 2016; van den Oord et al., 2016) to
capture both long- and short-term temporal patterns in a non-recursive
fashion, thus alleviating the issue of gradient explosion in RNNs.

Figure 2 (left) illustrates the basic structure of a standard TCN with three
DCC layers. In Layer 1, a convolutional filter slides over the input without
skipping any values in the filter (i.e., _dilation rate_ $\delta$ = 1); in
Layer 2, a convolutional filter that skips one value in the middle (i.e.,
$\delta$ = 2) is applied to the output of Layer 1; in Layer 3, convolutional
operations are performed with three skipped values (i.e., $\delta$ = 4) in the
filter. By stacking multiple DCC layers, a.k.a. TCN layers, with gradually
increased dilation rates, a TCN T-operator module with exponentially enlarged
receptive fields is built. As seen in Figure 2 (left), the receptive field of
the last layer’s rightmost node can cover the entire time series length of the
input data, which implies that the plain stacking does not ignore important
spatio-temporal correlations.

With the raw input CTS
${X}\in\mathbb{R}^{\mathtt{N}\times\mathtt{P}\times\mathtt{F}}$ being embedded
into the latent representation
${H}\in\mathbb{R}^{\mathtt{N}\times\mathtt{P}\times\mathtt{D}}$ by the
embedding module, a TCN layer cast on ${H}$ is formalized as follows.

$\displaystyle\operatorname{TCN}(H\mid\delta,\mathtt{K})=H^{\prime},\;\text{where}$
$\displaystyle
H^{\prime}[i;t;d]=\sum\nolimits_{k=0}^{\mathtt{K}-1}\big{(}H[i;{t-\delta\times
k};:]\cdot\text{W}^{d}[k;:]\big{)}$

is the $d$-th ($d\in[0,\mathtt{D})$) output feature map at time step $t$
($t\in[0,\mathtt{P})$) of the $i$-th ($i\in[0,\mathtt{N})$) time series,
$\text{W}^{d}\in\mathbb{R}^{\mathtt{D}\times\mathtt{K}}$ is the $d$-th
convolutional filter, and $\mathtt{K}$ (often as small as 2 or 3) and $\delta$
are the kernel size and dilation rate, respectively. To keep the temporal
dimension of $\mathtt{P}$ constant in the output, zero padding is applied to
the input of each layer (van den Oord et al., 2016).

Figure 2. A TCN with layers of (a) standard convolution (TCN), (b) group
convolution (GTCN), and (c) shuffled group convolution (SGTCN). The TCN
consists of three layers with dilation rates $\delta=\\{1,2,4\\}$ and the
kernel size $\mathtt{K}$ = 2.

####  4.2.2. Lightening TCN

Previous studies (Wu et al., 2020; Chen et al., 2022) show that directly
reducing the embedding size $\mathtt{D}$ inevitably lowers the representation
capability of the model, resulting in subpar accuracy. Therefore, we propose
instead to lighten the standard TCN using a grouping strategy. This is
motivated by observations of previous studies (Xie et al., 2017; Zhang et al.,
2018) that a TCN has redundant connections between adjacent TCN layers and
thus can be optimized. Specifically, the standard TCN layer in Figure 2(a) has
full connections between input and output feature maps, while the grouping
strategy in Figure 2(b) first partitions the input TCN feature maps into
$\mathtt{G}^{T}$ equal-sized, consecutive, and non-overlapping groups, and
then performs convolutions within the groups. A group TCN layer, i.e., a TCN
layer with the grouping strategy, is represented as follows.

$\operatorname{GTCN}(H\mid\mathtt{G}^{T})=\operatorname{concat}(\\{\operatorname{TCN}(H_{j}\mid\delta,\mathtt{K})\\}_{j=1}^{\mathtt{G}^{T}}),$

where
$H_{j}=[:,:,\frac{\mathtt{D}\times(j-1)}{\mathtt{G}^{T}}:\frac{\mathtt{D}\times
j}{\mathtt{G}^{T}}]\in\mathbb{R}^{\mathtt{N}\times\mathtt{P}\times\frac{\mathtt{D}}{{\mathtt{G}^{T}}}}$
is the $j$-th group of input feature maps and $\operatorname{concat}(\cdot)$
denotes the concatenation operation. The time and space complexity of each
group are
$\mathcal{O}\big{(}(\frac{\mathtt{D}}{\mathtt{G}^{T}})^{2}\cdot\mathtt{N}\cdot\mathtt{P}\big{)}$
and $\mathcal{O}\big{(}(\frac{\mathtt{D}}{\mathtt{G}^{T}})^{2}\big{)}$,
respectively. Therefore, the time and space complexity of a group TCN layer
with $\mathtt{G}^{T}$ groups is
$\mathcal{O}(\frac{\mathtt{D}^{2}}{\mathtt{G}^{T}}\cdot\mathtt{N}\cdot\mathtt{P})$
and $\mathcal{O}(\frac{\mathtt{D}^{2}}{\mathtt{G}^{T}})$, respectively, which
is $\frac{1}{\mathtt{G}^{T}}$ of the corresponding standard TCN. For example,
in Figure 2(b), $\mathtt{D}$ = 4, input and output feature maps are split into
$\mathtt{G}^{T}$ = 2 groups, and each group consists of
$\mathtt{D}/\mathtt{G}^{T}$ = 2 feature maps. The number of convolution
filters is consequently reduced from $\mathtt{D}^{2}$ = 16 to
$\mathtt{D}^{2}/\mathtt{G}^{T}$ = 8.

One drawback of the naive grouping strategy is the lack of information
exchange among groups. Thus, we propose to use a shuffled grouping strategy to
support communications among feature map groups. As depicted in Figure 2(c),
shuffling allows group convolutions to obtain input from different groups,
with one input feature map contributing to all groups. In the implementation,
we stipulate that the number of feature maps in each input group is divisible
by the number of output groups. The shuffling enhances the naive grouping
strategy in a simple but effective way to enable inter-group communication
without increasing model complexity.

The group number $\mathtt{G}^{T}$ is a model structure hyperparameter that
controls the balance between the lightness of an L-TCN and its capacity to
extract temporal information. Intuitively, a larger $\mathtt{G}^{T}$ improves
L-TCN’s lightness but reduces its capacity. Therefore, it is important to tune
$\mathtt{G}^{T}$. First, $\mathtt{G}^{T}$ belongs to a small set of candidate
values because it can only be a factor of the embedding size (e.g., {2, 4, 8,
16, 32} for $\mathtt{D}$ = 64). Therefore, we employ grid search on the small
number of candidates to maximize $\mathtt{G}^{T}$ while maintaining nearly
state-of-the-art accuracy. Alternatively, it is possible to search for an
optimal $\mathtt{G}^{T}$ more efficiently by applying advanced multi-objective
hyperparameter optimization approaches (Morales-Hernández et al., 2022), such
as multi-objective Bayesian optimization. The group number tuning discussed
here also applies to the grouping techniques we use in other modules. The
effect of varying $\mathtt{G}^{T}$ is studied empirically in Section 5.3.2.

Following previous studies (Wu et al., 2019, 2021), we adopt the gating
mechanism to decide the ratio of temporal information extracted by a shuffled
group TCN layer to flow through the model. Thus, an L-TCN layer is given as
follows.

$\operatorname{L-TCN}(H)=\operatorname{tanh}(\operatorname{SGTCN}_{{o}}(H\mid\mathtt{G}^{T}))\odot\sigma(\operatorname{SGTCN}_{{g}}(H\mid\mathtt{G}^{T})),$

where $\operatorname{SGTCN}_{{o}}$ and $\operatorname{SGTCN}_{{g}}$ are two
parallel _shuffled group_ TCN branches: the former extracts temporal features
and the latter controls the ratio at which the features are passed along. The
gating ratio is achieved by the sigmoid function $\sigma(\cdot)$ and is
applied to every temporal feature element by the element-wise product $\odot$.

To sum up, an L-TCN layer reduces the time and space complexities to
$\frac{1}{\mathtt{G}^{T}}$ of the standard TCN’s counterpart, i.e., to
$\mathcal{O}(\frac{\mathtt{D}^{2}}{\mathtt{G}^{T}}\cdot\mathtt{N}\cdot\mathtt{P})$
and $\mathcal{O}(\frac{\mathtt{D}^{2}}{\mathtt{G}^{T}})$, respectively. So
far, the output feature map is of size
$\mathtt{N}\times\mathtt{P}\times\mathtt{D}$. We proceed to present a last-
shot compression scheme to reduce the input size for subsequent S-operators.

###  4.3. Last-shot Compression

Inspired by the success of residual connections (He et al., 2016), popular CTS
models aggregate the features from every ST-block to get the output. The
bottom left corner of Figure 3 illustrates how such a classical aggregation
scheme is applied to the L-TCN. In particular, the features extracted by each
L-TCN layer are summed to obtain an aggregated feature tensor of shape
$\mathtt{N}\times\mathtt{P}\times\mathtt{D}$, with the temporal dimension
$\mathtt{P}$ preserved for subsequent spatial feature extraction. If we are
able to compress the temporal features extracted by each L-TCN layer, the
computational overheads of the following S-operators and the aggregation and
output module will be reduced. Note that these two components are both costly
(see Table 3).

To achieve this, we propose a simple but effective mechanism, named _last-shot
compression_. The idea comes from the intuition that more distant temporal
features are less important for the forecast at the current moment (Cao and
Tay, 2003; Tay and Cao, 2002; de Freitas et al., 1999). To put it simply,
last-shot compression retains only the features at the most recent time step
as the output of each L-TCN layer. An illustration is given in Figure 2
(left), where the rightmost snippets of each layer’s output (corresponding to
the last time step) are extracted and summed as the input to the subsequent
components.

One concern may be that such aggressive compression will completely lose the
information from the previous ($\mathtt{P}-1$) time steps of the input feature
map. This is not so because the stacking of dilated convolutions by L-TCNs
ensures that the last time step feature of each layer preserves information
from several most recent input time steps at different ranges. For example, in
Figure 2 (left), the last time step feature from Layer 1 captures the 7th and
8th input time steps, while that from Layer 2 captures the 5th to the 8th
input steps, and that from Layer 3 captures all eight input time steps. Hence,
if we aggregate only the last time step feature of each L-TCN layer, we still
perceive the full input, but focus more on recent input time steps while
performing higher compression on more distant input time steps. The ablation
study in Section 5.4 shows that this last-shot compression achieves impressive
cost reductions while maintaining model accuracy.

Figure 3. Feature aggregation after last-shot compression vs the classical
feature aggregation (Wu et al., 2021) (bottom left).

The last-shot compression and the following feature aggregation are
illustrated in Figure 3. Let
$H^{b}\in\mathbb{R}^{\mathtt{N}\times\mathtt{P}\times\mathtt{D}}$ be the
output feature of the $b$-th ($1\leq b\leq\mathtt{L}_{T}$) L-TCN layer, and
let $O^{b}=H^{b}[:,\mathtt{P}-1,:]$ $\in\mathbb{R}^{\mathtt{N}\times
1\times\mathtt{D}}$ be the last-step feature of $H^{b}$. The aggregation sums
the last-step features of all layers, i.e.,
$H=\sum\nolimits_{b=1}^{\mathtt{L}_{T}}O^{b}$. The aggregated feature $H$ is
then sent to a squeeze and excitation (SE) module (Hu et al., 2018) for
attentive feature representation:

(3)
$H^{T}=H\cdot\sigma(\text{W}_{s2}\cdot\operatorname{ReLU}(\text{W}_{s1}\cdot
H^{\circ})),$

where $H^{\circ}=\operatorname{GlobalAvgPool}(H)\in\mathbb{R}^{\mathtt{D}}$ is
achieved through the global average pooling (Hu et al., 2018), and
$H^{T}\in\mathbb{R}^{\mathtt{N}\times\mathtt{D}}$ is the final temporal
feature. Given the reduction ratio $r$ in the SE module,
$\text{W}_{s1}\in\mathbb{R}^{\frac{\mathtt{D}}{r}\times\mathtt{D}}$ and
$\text{W}_{s2}\in\mathbb{R}^{\mathtt{D}\times\frac{\mathtt{D}}{r}}$ are weight
matrices to squeeze the representation that in turn is rescaled back to
produce attentions with a sigmoid function $\sigma(\cdot)$ over the original,
aggregated feature $H$.

Compared to the classical aggregation scheme that leads to the feature size of
$\mathtt{N}\times\mathtt{P}\times\mathtt{D}$, the last-shot compression
achieves the feature size of $\mathtt{N}\times\mathtt{D}$ with the temporal
dimension having been flattened from $\mathtt{P}$ to 1. Despite this
reduction, the gradually enlarged receptive field of L-TCN (see Section 4.2.1)
ensures that the last-shot compression upon L-TCN can obtain the temporal
features across all time steps. This property enables further extraction of
spatial correlations over time. The space and time complexities of the
subsequent spatial feature extraction are thus reduced by a factor of
$1/\mathtt{P}$.

###  4.4. GL-Former

####  4.4.1. Background and Overall Design

Transformers occur in many state-of-the-art CTS forecasting models (Xu et al.,
2020; Park et al., 2020; Zhou et al., 2021; Wu et al., 2021). Aiming for light
yet effective S-operators, we propose a GlobalLocal TransFormer (GL-Former)
module that aims to extract both global-scale and local-scale spatial
correlations among different time series. This way, GL-Former eliminates the
exhaustive global-scale spatial correlation extraction seen in the standard
Transformer (Vaswani et al., 2017). We proceed to give preliminaries of the
standard Transformer and then detail the GL-Former design that targets
accuracy and lightness.

A standard Transformer consists of an _encoding layer_ and $\mathtt{L}_{s}$
_attention blocks_. Given the input
$H^{T}\in\mathbb{R}^{\mathtt{N}\times\mathtt{D}}$ generated by the last-shot
compression scheme (see Equation 3), a _positional encoding mechanism_ (PE)
(Vaswani et al., 2017) is introduced to the encoding layer. The reason is that
by default, Transformers’ self-attention operations are unable to interpret
the permutations of the input nodes. Specifically, the encoding layer converts
$H^{T}$ to a learned positional encoding $H^{\text{PE}}$ by incorporating the
identity information of CTS nodes:

$H^{\text{PE}}=H^{T}+\text{W}^{\text{PE}},$

where $\text{W}^{\text{PE}}\in\mathbb{R}^{\mathtt{N}\times\mathtt{D}}$ is a
learnable matrix capturing the identity information. The resulting encoding
$H^{\text{PE}}$ is fed to sequential attention blocks, each of which consists
of a _multi-head attention_ (MHA) layer and a _feed-forward network_ (FFN)
layer.

An MHA layer is a concatenation of $\mathtt{h}$ repeated self-attention
modules (heads) in parallel:

(4)
$\operatorname{MHA}(H^{\text{PE}})=\operatorname{concat}(\\{\operatorname{head}_{i}(H^{\text{PE}})\\}_{i=1}^{\mathtt{h}})$
(5)
$\operatorname{head}_{i}(H^{\text{PE}})=\operatorname{softmax}\big{(}H^{\text{I}}\big{)}\cdot{V}_{i}$
(6)
$H^{\text{I}}=({{Q}_{i}\cdot{K}_{i}^{\mathsf{T}}})/{\sqrt{\mathtt{D}/\mathtt{h}}}$
(7)
$Q_{i}=H^{\text{PE}}\cdot\text{W}^{Q}_{i},K_{i}=H^{\text{PE}}\cdot\text{W}^{K}_{i},V_{i}=\text{H}^{\text{PE}}\cdot\text{W}^{V}_{i},$

where all learnable matrices
$\text{W}^{*}_{i}\in\mathbb{R}^{\mathtt{D}\times\frac{\mathtt{D}}{\mathtt{h}}}$
are used to convert the embedding size from $\mathtt{D}$ to
$\mathtt{D}/\mathtt{h}$. In Equation 5, each head
$\operatorname{head}_{i}(\cdot)$ is a self-attention module that produces
attention scores for its input
$H^{\text{PE}}\in\mathbb{R}^{\mathtt{N}\times\mathtt{D}}$ by making the input
interact with itself. MHA provides impressive power to encode multiple
relationships.

An FFN layer merges the output
$H^{\text{MHA}}\in\mathbb{R}^{\mathtt{N}\times\mathtt{D}}$ of Equation 4 and
provides non-linear activations via two fully-connected layers:

(8)
$\operatorname{FFN}(H^{\text{MHA}})=\operatorname{FFN}^{(1)}(H^{\text{MHA}})\cdot\text{W}_{2}+\text{b}_{2}$
(9)
$\operatorname{FFN}^{(1)}(H^{\text{MHA}})=\operatorname{ReLU}(H^{\text{MHA}}\cdot\text{W}_{1}+\text{b}_{1}),$

where $\text{W}_{1}\in\mathbb{R}^{\mathtt{D}\times\mathtt{D}^{\prime}}$,
$\text{W}_{2}\in\mathbb{R}^{\mathtt{D}^{\prime}\times\mathtt{D}}$,
$\text{b}_{1}\in\mathbb{R}^{\mathtt{D}^{\prime}}$, and
$\text{b}_{2}\in\mathbb{R}^{\mathtt{D}}$ are learnable matrices and biases.
The two layers first enlarge the embedding size from $\mathtt{D}$ to
$\mathtt{D}^{\prime}$, which is typically a quadruple of $\mathtt{D}$ (Vaswani
et al., 2017), and then scale it back to $\mathtt{D}$. The feature generated
by the ($\mathtt{L}_{S}$)-th attention block is the final output of the
Transformer, i.e., $H^{S}$.

A standard attention block introduced above consists of an MHA and an FFN
layer and captures global-scale spatial correlations among all CTS nodes.
Although using _global attention blocks_ yields powerful feature extraction
capabilities, a model that learns such complicated information can be quite
hard to train. Injecting prior knowledge into the model is a sensible way to
increase model training efficiency, as the model does not need to extrapolate
the knowledge from the data itself. Further, the prior knowledge may offer
more information beyond the training data, thus helping to regularize the
model and prevent overfitting (Vladimirova et al., 2019). Specifically, in our
problem setting, using prior knowledge in the data, such as the explicit
spatial proximity information of CTS nodes (modeled as an
$\mathtt{N}$-by-$\mathtt{N}$ adjacency matrix), we can focus on extracting
correlations for only those pairs of nodes that are potentially relevant and
can omit computations for other pairs. This kind of attention block, which we
call a _local attention block_ , is detailed in Section 4.4.2. As shown in
Figure 4, an example GL-Former is a sequence of alternating global and local
attention blocks. The alternation combats the information loss on local
attentions. Note that the numbers of global and local attention blocks are not
necessarily the same. For example, one global attention could be followed by
two local attentions. In addition, we adopt the grouping strategy to ease the
computations of the MHA and FFN layers in attention blocks, to be detailed in
Section 4.4.3.

Figure 4. An example of GL-Former consisting of $\mathtt{L}_{S}$ alternating
global attention blocks and local attention blocks.

####  4.4.2. Local Attention Block

The computation of a local attention follows that of a global attention
(Equations 4 to 7), except that a mask function is applied in Equation 6 to
retain pairs of relevant nodes. Specifically, the mask function
$\operatorname{mask}(Z,M)$ hides an element $Z[i,j]$ in the feature matrix $Z$
if element $M[i,j]$ of the mask matrix is false; formally,

$Z[i,j]=\begin{cases}Z[i,j],&\text{if~{}}M[i,j]\text{~{}is~{}}\texttt{true}\\\
-\infty,&\text{otherwise}\end{cases}$

A hidden value is set to $-\infty$ because the masked feature matrix is sent
to a $\operatorname{softmax}(\cdot)$ function (see Equation 5). Besides, a
domain may be associated with several adjacency matrices. For example, one
adjacency matrix may capture the correlations in terms of the distance between
a pair of nodes, while another may capture the correlations in terms of the
data dependency between nodes. In this setting, we obtain mask matrix $M$ by
aggregating all relevant adjacency matrices of the domain, i.e.,
$M=\sum_{i}A_{i}$, where $A_{i}$ is one of the adjacency matrices. Notably,
$A_{i}$ is a sparse adjacency matrix thresholded by a filtering function (Wu
et al., 2019), which reduces the impact of noise and makes the model more
robust (Chen et al., 2022).

With the mask function, the variable $H^{\text{I}}$ in Equation 6 is
calculated as follows for a local attention block.

(10)
$H^{\text{I}}=\operatorname{mask}\big{(}({{Q}_{i}\cdot{K}_{i}^{\mathsf{T}}})/{\sqrt{\mathtt{D}/\mathtt{h}}},M\big{)}$

####  4.4.3. L-MHA and L-FNN

We propose the light MHA (L-MHA) that adopts a grouping strategy similar to
the one proposed for L-TCN in Section 4.2. The input encoding $H^{\text{PE}}$
of L-MHA is first partitioned into $\mathtt{G}^{M}$ groups, and then the
original MHA (see Equation 4) is applied to each group $H^{\text{PE}}_{j}$
($1\leq j\leq\mathtt{G}^{M}$). The L-MHA is given as follows.

$\displaystyle\operatorname{L-MHA}(H^{\text{PE}})$
$\displaystyle=\operatorname{concat}(\\{\operatorname{MHA}(H^{\text{PE}}_{j})\\}_{j=1}^{\mathtt{G}^{M}}),\;\text{where}$
$\displaystyle H^{\text{PE}}_{j}$
$\displaystyle=H^{\text{PE}}[:,\frac{\mathtt{D}\times(j-1)}{\mathtt{G}^{M}}:\frac{\mathtt{D}\times
j}{\mathtt{G}^{M}}]$

We apply multi-head attention with the same number $\mathtt{h}$ of heads for
each partitioned group. The output of $\operatorname{MHA}(H^{\text{PE}}_{j})$
is of size $\mathtt{N}\times\frac{\mathtt{D}}{\mathtt{G}^{M}}$, and the final
output of L-MHA is of size $\mathtt{N}\times\mathtt{D}$, the same as the
standard MHA in Equation 4. Still, the time and space complexities of L-MHA
are a fraction ${1}/{\mathtt{G}^{M}}$ of those of the standard MHA.

Likewise, we implement a light FFN (L-FFN) that partitions the input features
into $\mathtt{G}^{F}$ groups. We only apply the grouping strategy to the
second layer of an original FFN (see Equation 8) and the computation in the
first layer remains as shown in Equation 9. The first layer
$\operatorname{FFN}^{(1)}$ encapsulates the only non-linear activation in an
attention block, and lightening it will reduce accuracy considerably. As a
result, L-FFN is processed as follows.

$\displaystyle\operatorname{L-FFN}(H^{\text{MHA}})$
$\displaystyle=\operatorname{concat}(\\{\operatorname{FFN}(H^{\text{MHA}}_{j})\\})_{j=1}^{\mathtt{G}^{F}},\;\text{where}$
$\displaystyle H^{\text{MHA}}_{j}$
$\displaystyle=H^{\text{MHA}}[:,\frac{\mathtt{D}\times(j-1)}{\mathtt{G}^{F}}:\frac{\mathtt{D}\times
j}{\mathtt{G}^{F}}]$

As only the second FFN layer is lightened, the complexity of L-FFN is
${(1+1/\mathtt{G}^{F})}/{(1+1)}$ as the case for the standard FFN counterpart.

###  4.5. Assembling LightCTS

We compose LightCTS using the plain stacking architecture from Figure 1(b) and
using the proposed light T- and S-operator modules (i.e., L-TCN and GL-
Former). In particular, we use one CNN layer to implement the embedding module
and configure the aggregation and output module with two fully-connected
layers:

$\hat{Y}=\operatorname{ReLU}((H^{S}+H^{T})\cdot\text{W}^{o}_{1}+\text{b}^{o}_{1})\cdot\text{W}^{o}_{2}+\text{b}^{o}_{2},$

where $\hat{Y}\in\mathbb{R}^{\mathtt{N}\times\mathtt{L}}$ is the forecast
result, $\mathtt{L}=1$ or $\mathtt{Q}$ denotes the forecast dimension
depending on whether single-step or multi-step forecasting is performed;
$H^{T}$ and $H^{S}$ are the output features of the last-shot compression and
GL-Former, respectively; and $\text{W}^{o}_{1}$, $\text{W}^{o}_{2}$,
$\text{b}^{o}_{1}$, and $\text{b}^{o}_{2}$ are learnable matrices and biases.

Finally, we follow previous work (Rao et al., 2022; Wu et al., 2021, 2019) and
employ mean absolute error (MAE) as the loss function.

It should be noted that this simple plain stacking architecture does not lower
the effectiveness of feature extraction. The key reason is twofold: 1) the
last-shot compression (detailed in Section 4.3) with the proposed T-operator
module L-TCN (detailed in Section 4.2) are able to emphasize the most recent
timestep’s features and to reduce the noise in feature maps, thus improving
the subsequent spatial feature extraction; and 2) the proposed S-operator
module GL-Former (detailed in Section 4.4) addresses both local and global-
scale spatial correlations. By considering adjacency matrices in the attention
block through a mask, GL-Former can capture the prior knowledge of spatial
correlations, which the standard Transformer is unable to do, thus enhancing
the capability of spatial feature extraction. We report on empirical studies
in Section 5.4 that offer evidence of the effectiveness of the proposed last-
shot compression, GL-Former, and plain stacking.

##  5\. Experiments

We evaluate LightCTS on both multi-step and single-step CTS forecasting tasks.
We include many commonly-used benchmark datasets, four for multi-step
forecasting (two on traffic flow forecasting and two on traffic speed
forecasting) and two for single-step forecasting. These benchmarks are
associated with different accuracy metrics. To enable direct and fair
comparisons, we use the metrics employed in the original papers. The code and
datasets are made available (lig, 2023).

###  5.1. Multi-Step Forecasting

####  5.1.1. Datasets

  * •

PEMS04 (Song et al., 2020) is a traffic flow dataset collected from 307
sensors on 29 roads in San Francisco Bay Area during January – March 2018. The
traffic flow ranges from 0 to 919 with a mean of 91.74.

  * •

PEMS08 (Song et al., 2020) is a traffic flow dataset collected from 170
sensors on 8 roads in San Bernardino during July – September 2016. The traffic
flow ranges from 0 to 1,147 with a mean of 98.17.

  * •

METR-LA (Li et al., 2018) contains traffic speed data in mph gathered during
March – June 2012 from 207 loop detectors, from the road network of Los
Angeles County. The speed ranges from 0 to 70 mph with a mean of 53.98 mph.

  * •

PEMS-BAY (Li et al., 2018) is a traffic speed dataset gathered from 325 loop
detectors in the road network in the San Francisco Bay Area. The speed ranges
from 0 to 85.1 mph with a mean of 62.62 mph.

The sampling interval of each dataset is 5 minutes. The datasets are organized
and split (i.e., in train:validation:test) as in previous studies (Wu et al.,
2019, 2021). The statistics are summarized in Table 4.

Table 4. Dataset statistics for multi-step forecasting.

Dataset | Data type | $\mathtt{N}$ | $\mathtt{T}$ | $\mathtt{P}$ | $\mathtt{Q}$ | Split Ratio
---|---|---|---|---|---|---
PEMS04 | Traffic flow | 307 | 16,992 | 12 | 12 | 6:2:2
PEMS08 | Traffic flow | 170 | 17,856 | 12 | 12 | 6:2:2
METR-LA | Traffic speed | 207 | 34,272 | 12 | 12 | 7:1:2
PEMS-BAY | Traffic speed | 325 | 52,116 | 12 | 12 | 7:1:2

####  5.1.2. Metrics

We consider accuracy and lightness as follows.

  * •

Accuracy Metrics. Following existing multi-step forecasting studies (Li et
al., 2018; Yu et al., 2018b; Wu et al., 2019, 2021), we use mean absolute
error (MAE), root mean squared error (RMSE), and mean absolute percentage
error (MAPE) to measure accuracy comprehensively. The three metrics capture
the forecasting accuracy from different perspectives: MAE gives equal weights
to all errors, RMSE focuses on the most severe errors, and MAPE highlights the
errors when ground truth values are small. Lower MAE, RMSE, and MAPE indicate
higher forecasting accuracy.

  * •

Lightness Metrics. Consistent with the existing conventions (Howard et al.,
2017; Zhang et al., 2018) and to eliminate the influence of different DL
platforms and operating system conditions (e.g., multiple concurrent running
programs), we evaluate the lightness of CTS forecasting models using FLOPs and
the number of parameters222FLOPs and parameter counts are captured by _Fvcore_
(fvc, 2023) from Facebook Research. considered during _inferencing_. This
accords with existing studies (Zhang et al., 2018; Li et al., 2021; Tan and
Le, 2019). In addition, we report the latency and peak memory use (abbreviated
as Peak Mem) of models during inferencing on a low-computational-resource
device (see Section 5.1.4). The results are for practical reference and will
vary depending on hardware, software, implementation, and other factors.

####  5.1.3. CTS Forecasting Models for Comparisons

All models are implemented using their original code; and if using the same
dataset, we report the original results.

  * •

DcRnn (Li et al., 2018). A relatively early DL-based model that adopts
diffusion GCNs and GRUs for S- and T-operators, respectively.

  * •

GwNet (Wu et al., 2019). A widely used benchmark model that integrates
adaptive GCNs and TCNs for S- and T-operators, respectively.

  * •

AgCrn (Bai et al., 2020). A comprehensive but costly model that considers
dynamic spatial correlations through different time steps.

  * •

MtGnn (Wu et al., 2020). A successor of GwNet with new graph learning layers
and an optimized overall structure.

  * •

AutoCts (Wu et al., 2021). An automated framework that allows heterogeneous
ST-blocks with different S/T-operators and their connections through automatic
search.

  * •

EnhanceNet (Cirstea et al., 2021). A framework that uses distinct filters for
each time series and dynamic adjacency matrices to capture spatial
correlations over time. TCNs are used to implement T-operators.

  * •

Fogs (Rao et al., 2022). A recent model uses first-order gradients to avoid
fitting irregularly-shaped distributions.

  * •

AutoCts-KDf/AutoCts-KDp. We construct two compressed variants of
AutoCts333AutoCts is selected as the teacher model for compression as it
generally achieves the highest effectiveness among all competitor methods in
our study. using knowledge distillation (KD) for regression tasks (Takamoto et
al., 2020). Specifically, AutoCts-KDf and AutoCts-KDp are compressed to have
nearly the same numbers of FLOPs and parameters as LightCTS.

####  5.1.4. Implementation Details

All models are trained on a server with an NVIDIA Tesla P100 GPU. To
investigate models’ inferencing performance in constrained computing
environments, we employ an X86 device with a 380 MHz CPU.

The L-TCN has ($\mathtt{L}_{T}$ = 4) layers with the dilation rate $\delta$ of
each layer set to [1, 2, 4, 8] to ensure that the receptive field of the last
layer can cover the entire time series length of the input CTS. The GL-Former
has ($\mathtt{L}_{S}$ = 6) attention blocks for METR-LA and PEMS-BAY, and
$\mathtt{L}_{S}$ = 4 for PEMS04 and PEMS08. The stacking pattern is one global
block followed by one local block, as shown in Figure 4. Following parameter
tuning, we set the embedding size $\mathtt{D}$ = 48 for METR-LA, $\mathtt{D}$
= 64 for the other three datasets. For all datasets, the group numbers
$\mathtt{G}^{T}$ = 4, $\mathtt{G}^{M}$ = $\mathtt{G}^{F}$ = 2, and the
reduction ratio $r$ of the SE module is set to 8. We adopt the Adam optimizer
with a learning rate of 0.002 to train models for 250 epochs.

####  5.1.5. Overall Comparisons

Following existing conventions for direct and fair comparison (Wu et al.,
2021; Rao et al., 2022; Wu et al., 2020), we report the average accuracy over
all 12 future time steps for the PEMS04 and PEMS08 datasets and report the
accuracy at the 3rd, 6th, and 12th time steps for the METR-LA and PEMS-BAY
datasets. Tables 5 and 6 show the results for both the accuracy and lightness
measures. In this section, the best results are in bold, and the second-best
results are underlined.

Considering accuracy on PEMS04 and PEMS08 datasets, Table 5 shows that
LightCTS achieves the best MAE and RMSE results and ranks second regarding
MAPE. Although Fogs achieves the best MAPE on both datasets, it is only
marginally better than LightCTS (less than 0.1%), and its MAE and RMSE rank
only around 4th among all models. The accuracy results on METR-LA and PEMS-BAY
datasets in Table 6 show that LightCTS achieves the best MAE, at least the
second-best MAPE, and a competitive RMSE (top-3 in most cases). While LightCTS
does not always rank in the top-3 in terms of RMSE, it is only negligibly
below. For example, for the 15th-minute forecast on METR-LA dataset, LightCTS
achieves an RMSE of 5.16 to rank 4th, while the top-3 models achieve 5.11,
5.14, and 5.15; the maximum margin is only 0.05, corresponding to a speed of
0.05 mph traffic speed. Given that there are always fluctuations across real-
world datasets, such a small performance difference is insignificant.

Considering lightness, Tables 5 and 6 show that LightCTS has significantly
fewer FLOPs and model parameters and achieves lower latency and peak memory
use than all other models, with the exception that AgCrn achieves slightly
fewer parameters and lower peak memory use on PEMS08 dataset. However, AgCrn
is much less accurate. LightCTS clearly uses fewer resources than the two most
accurate competing models, EnhanceNet and AutoCts (e.g., less than 1/6 and
1/10 FLOPs), while maintaining comparable accuracy.

Besides, although KD (Takamoto et al., 2020) enables creating the AutoCts-KDf
and AutoCts-KDp models that have FLOPs and parameter counts comparable to
those of LightCTS, their effectiveness metrics are significantly lower, as
seen in Tables 5 and 6.

In summary, LightCTS offers substantially reduced computational and storage
overheads while providing top-tier accuracy at multi-step CTS forecasting.
LightCTS thus offers unique value because it enables CTS forecasting with no
accuracy penalty using limited resources, which are often found in real-world
applications. In addition, it also lowers costs when deployed in non-
constrained settings, such as servers.

Table 5. Accuracy and lightness comparison for multi-step traffic flow
forecasting.

Data | Models |  | FLOPs
---
(unit: M)
| Params
---
(unit: K)
| Latency
---
(unit: s)
| Peak Mem
---
(unit: Mb)
MAE | RMSE | MAPE
PEMS04 | DcRnn | 3739 | 371 | 22.9 | 8.1 | 24.70 | 38.12 | 17.12%
GwNet | 1277 | 311 | 4.8 | 6.8 | 19.16 | 30.46 | 13.26%
AgCrn | 3936 | 749 | 9.5 | 19.2 | 19.83 | 32.26 | 12.97%
MtGnn | 393 | 547 | 5.3 | 12.1 | 19.32 | 31.57 | 13.52%
AutoCts | 2043 | 368 | 5.2 | 7.8 | 19.13 | 30.44 | 12.89%
EnhanceNet | 969 | 283 | 4.2 | 6.0 | 19.11 | 30.34 | 14.33%
Fogs | 5936 | 2366 | 14.6 | 42.9 | 19.34 | 31.20 | 12.71%
AutoCts-KDf | 196 | 17 | 3.0 | 5.3 | 25.17 | 38.85 | 17.06%
AutoCts-KDp | 1278 | 186 | 4.4 | 6.8 | 22.83 | 35.21 | 15.55%
LightCTS | 147 | 185 | 1.1 | 4.7 | 18.79 | 30.14 | 12.80%
PEMS08 | DcRnn | 2070 | 371 | 14.9 | 4.8 | 17.86 | 27.83 | 11.45%
GwNet | 479 | 309 | 1.0 | 4.0 | 15.13 | 24.07 | 10.10%
AgCrn | 726 | 150 | 3.2 | 2.6 | 15.95 | 25.22 | 10.09%
MtGnn | 153 | 352 | 2.7 | 4.6 | 15.71 | 24.62 | 10.03%
AutoCts | 808 | 366 | 3.7 | 4.7 | 14.82 | 23.64 | 9.51%
EnhanceNet | 365 | 275 | 1.7 | 3.8 | 14.82 | 23.60 | 9.58%
Fogs | 1949 | 1294 | 6.6 | 14.2 | 14.92 | 24.09 | 9.42%
AutoCts-KDf | 100 | 22 | 1.3 | 3.3 | 19.47 | 30.26 | 13.77%
AutoCts-KDp | 485 | 181 | 1.9 | 4.0 | 16.89 | 26.40 | 11.08%
LightCTS | 70 | 177 | 0.4 | 2.8 | 14.63 | 23.49 | 9.43%

Table 6. Accuracy and lightness comparison for multi-step traffic speed forecasting. Data | Model | FLOPs | Params | Latency | Peak Mem | 15 mins | 30 mins | 60 mins
---|---|---|---|---|---|---|---|---
METR-LA |  | (unit: M) | (unit: K) | (unit: s) | (unit: Mb) | MAE | RMSE | MAPE | MAE | RMSE | MAPE | MAE | RMSE | MAPE
DcRnn | 2521 | 436 | 16.4 | 13.6 | 2.77 | 5.38 | 7.30% | 3.15 | 6.45 | 8.80% | 3.60 | 7.60 | 10.50%
GwNet | 658 | 309 | 1.7 | 8.9 | 2.69 | 5.15 | 6.90% | 3.07 | 6.22 | 8.37% | 3.53 | 7.37 | 10.01%
AgCrn | 2453 | 748 | 7.5 | 22.6 | 2.83 | 5.45 | 7.56% | 3.20 | 6.55 | 8.79% | 3.58 | 7.41 | 10.13%
MtGnn | 208 | 405 | 3.9 | 11.9 | 2.69 | 5.18 | 6.86% | 3.05 | 6.17 | 8.19% | 3.49 | 7.23 | 9.87%
AutoCts | 1090 | 366 | 2.8 | 11.4 | 2.67 | 5.11 | 6.80% | 3.05 | 6.11 | 8.15% | 3.47 | 7.14 | 9.81%
EnhanceNet | 648 | 453 | 2.6 | 13.4 | 2.69 | 5.14 | 6.93% | 3.06 | 6.10 | 8.29% | 3.49 | 7.23 | 9.96%
Fogs | 2858 | 1524 | 7.4 | 45.9 | 2.72 | 5.20 | 7.05% | 3.12 | 6.30 | 8.60% | 3.64 | 7.61 | 10.62%
AutoCts-KDf | 95 | 15 | 1.6 | 5.9 | 3.04 | 5.80 | 8.49% | 3.57 | 7.03 | 10.49% | 4.19 | 8.34 | 12.73%
AutoCts-KDp | 595 | 155 | 2.2 | 7.1 | 2.78 | 5.21 | 7.33% | 3.18 | 6.23 | 9.00% | 3.64 | 7.28 | 10.97%
LightCTS | 71 | 133 | 0.3 | 5.6 | 2.67 | 5.16 | 6.82% | 3.03 | 6.16 | 8.11% | 3.42 | 7.21 | 9.46%
PEMS-BAY | DcRnn | 5386 | 436 | 22.6 | 13.8 | 1.38 | 2.95 | 2.90% | 1.74 | 3.97 | 3.90% | 2.07 | 4.74 | 4.90%
GwNet | 1408 | 312 | 3.7 | 10.3 | 1.30 | 2.74 | 2.73% | 1.63 | 3.70 | 3.67% | 1.95 | 4.52 | 4.63%
AgCrn | 4224 | 749 | 10.1 | 22.7 | 1.35 | 2.83 | 2.87% | 1.69 | 3.81 | 3.84% | 1.96 | 4.52 | 4.67%
MtGnn | 432 | 573 | 7.6 | 19.2 | 1.32 | 2.79 | 2.77% | 1.65 | 3.74 | 3.69% | 1.94 | 4.49 | 4.53%
AutoCts | 2295 | 369 | 5.9 | 11.9 | 1.30 | 2.71 | 2.69% | 1.61 | 3.62 | 3.55% | 1.89 | 4.32 | 4.36%
EnhanceNet | 1442 | 474 | 5.4 | 14.2 | 1.33 | 2.81 | 2.80% | 1.64 | 3.72 | 3.65% | 1.93 | 4.47 | 4.51%
Fogs | 6608 | 2551 | 16.3 | 76.5 | 1.38 | 2.91 | 2.94% | 1.73 | 3.93 | 3.97% | 2.09 | 4.71 | 4.96%
AutoCts-KDf | 218 | 18 | 3.4 | 9.5 | 1.42 | 2.92 | 2.95% | 1.78 | 4.02 | 3.99% | 2.11 | 4.82 | 5.06%
AutoCts-KDp | 1431 | 248 | 4.4 | 10.4 | 1.38 | 2.85 | 2.87% | 1.72 | 3.82 | 3.86% | 2.06 | 4.62 | 4.73%
LightCTS | 208 | 236 | 1.2 | 9.2 | 1.30 | 2.75 | 2.71% | 1.61 | 3.65 | 3.59% | 1.89 | 4.32 | 4.39%
Table 7. Accuracy and lightness comparison for single-step CTS forecasting. Data | Model | FLOPs | Params | Latency | Peak Mem | 3-th | 6-th | 12-th | 24-th
---|---|---|---|---|---|---|---|---|---
|  | (unit: M) | (unit: K) | (unit: s) | (unit: Mb) | RRSE | CORR | RRSE | CORR | RRSE | CORR | RRSE | CORR
Solar-Energy | DsaNet | 914 | 6377 | 0.8 | 32.5 | 0.1822 | 0.9842 | 0.2450 | 0.9701 | 0.3287 | 0.9444 | 0.4389 | 0.8943
MtGnn | 1090 | 348 | 0.5 | 9.9 | 0.1778 | 0.9852 | 0.2348 | 0.9726 | 0.3109 | 0.9509 | 0.4270 | 0.9031
MaGnn | 492 | 105 | 0.4 | 9.2 | 0.1771 | 0.9853 | 0.2361 | 0.9724 | 0.3105 | 0.9539 | 0.4108 | 0.9097
AutoCts | 2237 | 91 | 1.1 | 17.6 | 0.1750 | 0.9853 | 0.2298 | 0.9763 | 0.2957 | 0.9566 | 0.4143 | 0.9097
AutoCts-KDf | 418 | 12 | 0.4 | 9.2 | 0.1802 | 0.9834 | 0.2463 | 0.9696 | 0.3332 | 0.9403 | 0.4277 | 0.9021
AutoCts-KDp | 1196 | 41 | 0.7 | 13.4 | 0.1785 | 0.9844 | 0.2371 | 0.9736 | 0.3288 | 0.9435 | 0.4196 | 0.9043
LightCTS | 169 | 38 | 0.2 | 8.6 | 0.1714 | 0.9864 | 0.2202 | 0.9765 | 0.2955 | 0.9568 | 0.4129 | 0.9084
Electricity | DsaNet | 2262 | 6377 | 1.2 | 53.9 | 0.0855 | 0.9264 | 0.0963 | 0.9040 | 0.1020 | 0.8910 | 0.1044 | 0.8898
MtGnn | 4800 | 362 | 1.5 | 21.4 | 0.0745 | 0.9474 | 0.0878 | 0.9316 | 0.0916 | 0.9278 | 0.0953 | 0.9234
MaGnn | 2215 | 120 | 0.8 | 20.3 | 0.0745 | 0.9476 | 0.0876 | 0.9323 | 0.0908 | 0.9282 | 0.0963 | 0.9217
AutoCts | 8740 | 95 | 3.2 | 21.3 | 0.0743 | 0.9477 | 0.0865 | 0.9315 | 0.0932 | 0.9247 | 0.0947 | 0.9239
AutoCts-KDf | 1858 | 16 | 1.8 | 12.4 | 0.0818 | 0.9292 | 0.0949 | 0.9148 | 0.1003 | 0.9007 | 0.1018 | 0.8935
AutoCts-KDp | 3937 | 33 | 2.3 | 17.3 | 0.0764 | 0.9442 | 0.0899 | 0.9275 | 0.0934 | 0.9188 | 0.0983 | 0.9071
LightCTS | 239 | 27 | 0.4 | 10.0 | 0.0736 | 0.9445 | 0.0831 | 0.9343 | 0.0898 | 0.9261 | 0.0952 | 0.9215

###  5.2. Single-Step Forecasting

####  5.2.1. Datasets

  * •

Solar-Energy (Lai et al., 2018) contains records of solar power production in
megawatt-hour (MWh) collected from 137 photovoltaic plants in Alabama during
2006. The records range from 0 to 88.9 MWh with a mean of 6.4 MWh. The
sampling interval is 10 minutes.

  * •

Electricity (Lai et al., 2018) contains records of electricity consumption in
kilowatt-hour (kWh) for 321 clients in Portugal during 2012 – 2014. The values
range from 0 to 764,000 kWh, with an average of 2,514 kWh. The sampling
interval is 15 minutes.

The data organization and data splitting follow existing work (Wu et al.,
2020, 2021). Dataset statistics are summarized in Table 8.

Table 8. Dataset statistics for single-step forecasting. Dataset | $\mathtt{N}$ | $\mathtt{T}$ | $\mathtt{P}$ | $\mathtt{Q}$ | Split Ratio
---|---|---|---|---|---
Solar-Energy | 137 | 52,560 | 168 | {3, 6, 12, 24} | 6:2:2
Electricity | 321 | 26,304 | 168 | {3, 6, 12, 24} | 6:2:2

####  5.2.2. Metrics

We use the same lightness metrics as the multi-step forecasting task. We use
root relative squared error (RRSE) and correlation coefficient (CORR) to
measure forecasting accuracy, which are the conventional metrics used in
single-step CTS forecasting (Wu et al., 2021; Chen et al., 2022; Chang et al.,
2018). Specifically, RRSE indicates how well a model performs w.r.t. the
average of the true values, while CORR measures the strength of the linear
correlation between the forecast results and the true values. The more
accurate the model, the lower the RRSE and the higher the CORR.

####  5.2.3. CTS Models in Comparisons

We include two CTS models that are specifically designed for single-step
forecasting:

  * •

DsaNet (Huang et al., 2019). A dual self-attention network for multivariate
time series forecasting.

  * •

MaGnn (Chen et al., 2022). A multi-branch model that extracts temporal
features at different time scales.

In addition, we include MtGnn (Wu et al., 2020) and AutoCts (Wu et al., 2021)
(see Section 5.1.3) as they also support single-step forecasting. We use the
settings that achieve the best accuracy for the comparison models; or if using
the same dataset, we report their original results.

####  5.2.4. Implementation Details

To build LightCTS for single-step forecasting, we do almost the same as for
multi-step forecasting. The differences are as follows. According to parameter
tuning, we set $\mathtt{D}$ = 32 for the Solar-Energy dataset and $\mathtt{D}$
= 24 for the Electricity dataset, and we set the dilation rates in the
($\mathtt{L}_{T}$ = 8) L-TCN layers to [1, 2, 4, 8, 16, 32, 48, 64]. We adopt
a GL-Former with ($\mathtt{L}_{S}$ = 2) attention blocks (i.e., one global
attention followed by one local attention). We adopt the Adam optimizer with a
learning rate of 0.0005 to train models for 100 epochs.

####  5.2.5. Overall Comparisons

Table 7 shows the results on Solar-Energy and Electricity datasets. In
accordance with existing research (Chen et al., 2022; Wu et al., 2020), we
report single-step forecasting results for the 3rd, 6th, 12th, and 24th future
time steps.

We observe similar trends as for multi-step forecasting. In terms of accuracy,
LightCTS achieves the best performance on most of the comparison items, and it
is very competitive on the others. Next, LightCTS is the most lightweight
model with far fewer FLOPs and parameters than all competitors. For example,
LightCTS uses less than 1/10 (_resp._ 1/3) FLOPs than AutoCts (_resp._ MaGnn).
In addition, LightCTS has the lowest latency and peak memory use among all
baselines. LightCTS’s low requirement of computing resource means a great
potential to be deployed in resource-constrained environments.

###  5.3. Parameter Study

We study systematically the impact of key LightCTS hyperparameters, including
the embedding size $\mathtt{D}$, the group number $\mathtt{G}^{T}$ of L-TCN,
and the attention block number $\mathtt{L}_{S}$ of GL-Former. These
hyperparameters are selected as they are adjustable and affect model
performance noticeably. We summarize the results in Figures 5 and 6 for multi-
step forecasting on PEMS08 and single-step forecasting on Solar-Energy,
respectively. We report the results on the other datasets in the supplemental
material (lig, 2023).

####  5.3.1. Impact of Embedding Size $\mathtt{D}$

Figure 5(a) shows the impact of $\mathtt{D}$ on model accuracy and lightness
of single-step forecasting on the PEMS08 dataset. Both the FLOPs and number of
parameters increase steadily as $\mathtt{D}$ increases, and so do the latency
and peak memory use.

Considering accuracy, as $\mathtt{D}$ grows from 32 to 64, MAE, RMSE, and MAPE
decrease moderately. However, when $\mathtt{D}$ goes up from 64 to 80, the
forecasting errors increase slightly. The reason may be that a smaller
$\mathtt{D}$ restricts the model’s ability to extract ST-features, while a
larger $\mathtt{D}$ may introduce redundancy into the model and make it
difficult to train, and can lead to overfitting. In Figure 6(a), the single-
step forecasting exhibits similar trends as the multi-step counterpart. The
FLOPs and number of parameters increase while the forecasting errors drop
first and climb up afterwards.

The results are consistent with those of existing studies (Wu et al., 2020;
Chen et al., 2022)—directly cutting the embedding size to a small value
inevitably reduces model accuracy. Thus, a new design for manipulating
$\mathtt{D}$, such as our L-TCN and GL-Former, is effective at enabling
lightweight and accurate CTS forecasting models.

Figure 5. Impact of (a) embedding size $\mathtt{D}$, (b) group number
$\mathtt{G}^{T}$, and (c) attention block number $\mathtt{L}_{S}$ in GL-Former
for multi-step forecasting on PEMS08 dataset. Figure 6. Impact of (a)
embedding size $\mathtt{D}$, (b) group number $\mathtt{G}^{T}$, and (c)
attention block number $\mathtt{L}_{S}$ in GL-Former for single-step
forecasting on Solar-Energy dataset.

####  5.3.2. Impact of Group Number $\mathtt{G}^{T}$

We further investigate the impact of the group number $\mathtt{G}^{T}$ of
L-TCN on the performance and lightness of LightCTS. This parameter controls
the tradeoff between L-TCN’s capacity for temporal information extraction and
its lightness.

The multi-step forecasting results in Figure 5(b) show that when
$\mathtt{G}^{T}$=4, the evaluation errors reach a minimum that is below the
value for $\mathtt{G}^{T}$=1, i.e., the standard TCN without grouping. This
implies that our grouping strategy can remove redundant connections without
compromising forecasting accuracy. However, due to information loss, the
evaluation errors increase substantially when the group size is increased to
8. A similar pattern is seen for single-step forecasting in Figure 6(b), where
the best accuracy is also obtained when $\mathtt{G}^{T}$=4.

Further, additional experiments show that similar patterns appear for the
other group number hyperparameters, namely $\mathtt{G}^{M}$ and
$\mathtt{G}^{F}$. For brevity, we report on these experiments in the
supplemental material (lig, 2023).

####  5.3.3. Impact of Attention Block Number $\mathtt{L}_{S}$ in GL-Former

LightCTS supports an elastic number of attention blocks in GL-Former. Hence,
it is of interest to understand how the GL-Former attention block number
$\mathtt{L}_{S}$ affects accuracy and lightness. The number $\mathtt{L}_{T}$
of L-TCN layers is also tunable, with its value being relevant to the input
time series $\mathtt{P}$ for producing sufficient receptive fields. We thus
focus on varying and testing the attention block number $\mathtt{L}_{S}$.

As shown in Figures 5(c) and 6(c), for both multi-step and single-step
forecasting, as GL-Former goes deeper (i.e., larger $\mathtt{L}_{S}$), the
evaluation errors first decrease and then increase. This happens because
although deeper models can theoretically better extract information, they are
prone to overfitting or hard to train (Vladimirova et al., 2019). Notably, in
Figure 6(c), since the T-operators consume a dominant proportion of FLOPs due
to the large input length ($\mathtt{P}$ = 168) of time series, the overall
FLOPs and peak memory use of LightCTS increase only slightly when
$\mathtt{L}_{S}$ is increased.

###  5.4. Ablation Study

Table 9. Ablation study on PEMS08. Refer to Section 5.4 for the details of the
models.

Model |  | FLOPs
---
(unit: M)
| Params
---
(unit: K)
| Latency
---
(unit: s)
| Peak Mem
---
(unit: Mb)
MAE | RMSE | MAPE
LightCTS | 70 | 177 | 0.4 | 2.8 | 14.63 | 23.49 | 9.43%
LightCTS$\backslash$T | 149 | 226 | 0.5 | 3.1 | 14.64 | 23.55 | 9.48%
LightCTS$\backslash$LS | 390 | 285 | 0.6 | 3.7 | 15.31 | 24.21 | 10.55%
LightCTS$\backslash$M | 75 | 239 | 0.5 | 3.0 | 14.57 | 23.48 | 9.48%
LightCTS$\backslash$F | 75 | 238 | 0.5 | 3.0 | 14.64 | 23.59 | 9.66%
LightCTS$\backslash$LA | 70 | 177 | 0.4 | 2.8 | 14.70 | 23.71 | 9.55%
LightCTS-A | 390 | 285 | 0.6 | 3.7 | 15.64 | 24.52 | 10.96%
LightCTS-CGCN | 53 | 74 | 0.3 | 1.7 | 16.53 | 26.34 | 10.63%
LightCTS-DGCN | 95 | 154 | 0.4 | 2.6 | 16.24 | 25.79 | 10.72%

We conduct an ablation study on PEMS08 dataset to understand the contribution
of each component in LightCTS. Specifically, we implement a group of LightCTS
variants by removing one of the lightweight components and observe the impact
on both accuracy and lightness. The variants include:

  * •

LightCTS$\backslash$T that substitutes the L-TCN with the standard TCN as the
T-operator module.

  * •

LightCTS$\backslash$LS that substitutes the last-shot compression with the
classical full-shot aggregation method (see Figure 1(a)),

  * •

LightCTS$\backslash$M that substitutes the L-MHA component of the GL-Former
with the standard MHA component.

  * •

LightCTS$\backslash$F that substitutes the L-FFN component of the GL-Former
with the standard FFN component.

  * •

LightCTS$\backslash$LA that substitutes all local attention blocks with global
attention blocks.

We also introduce three other LightCTS variants to assess our design choices:

  * •

LightCTS-A that adopts the alternate stacking pattern instead of the proposed
plain stacking.

  * •

LightCTS-CGCN that substitutes the GL-Former with Chebyshev GCNs (Defferrard
et al., 2016) as the S-operator module.

  * •

LightCTS-DGCN that substitutes the GL-Former with Diffusion GCNs (Gasteiger et
al., 2019) as the S-operator module.

Table 9 shows the results. We make the following observations. 1) LightCTS
achieves almost the best accuracy with much fewer FLOPs and parameters when
all the lightweight techniques are deployed, implying that the proposed
modules (i.e., L-TCN and GL-Former) and the last-shot compression are more
efficient than their standard counterparts. 2) LightCTS$\backslash$LA is
inferior to LightCTS in terms of accuracy, indicating that the local attention
block utilizing prior knowledge of spatial information does help achieve
better forecasting performance. 3) The comparison between LightCTS and
LightCTS-A indicates that the plain stacking strategy is much better at
reducing overheads than the standard alternate stacking strategy, while
simultaneously improving the forecasting accuracy. 4) When comparing LightCTS-
CGCN, LightCTS-DGCN, and LightCTS, we observe that although LightCTS consumes
more resources, its accuracy surpasses the GCN-based models’ by a large
margin. Thus, we choose the Transformer-based S-operators in LightCTS.

###  5.5. Studies on Memory Constraints

We evaluate the performance of LightCTS under memory constraints of 3Mb,
2.5Mb, and 2Mb, as commonly found in commodity MCUs (STM, 2023). We include
representative baselines: AutoCts (the most accurate), AutoCts-KDf/AutoCts-KDp
(KD variants of AutoCts), AgCrn (the least peak memory use), and GwNet (the
lowest latency). For fair comparisons, we adjust the embedding size
$\mathtt{D}$ for all models to fit into the constrained memory while
maintaining their structures and components. Results on PEMS08 are presented
in Table 10. Results on other datasets are available elsewhere (lig, 2023).

Even with the lowest possible embedding size, AutoCts and its variants are
unable to comply with the memory constraints due to their large intermediate
results. This is concrete evidence of the inapplicability of these models to
resource-constrained devices such as MCUs. While GwNet and AgCrn are able to
meet the memory constraints, they experience significant accuracy loss. In
contrast, LightCTS shows the least accuracy degradation and surpasses the
baseline models significantly with the lowest latency under all studied
constraints. These results demonstrate the need for specialized lightweight
designs as state-of-the-art CTS models without such considerations fail to
achieve satisfactory accuracy when downscaled for low-memory settings.

Table 10. Models vs memory constraints on PEMS08.

| Mem Constraint
---
(unit: Mb)
Model |  | FLOPs
---
(unit: M)
| Params
---
(unit: K)
| Latency
---
(unit: s)
| Peak Mem
---
(unit: Mb)
MAE | RMSE | MAPE
| LightCTS | 70 | 177 | 0.4 | 2.8 | 14.63 | 23.49 | 9.43%
3 | GwNet | 328 | 178 | 1.0 | 2.9 | 17.40 | 27.34 | 11.14%
| AgCrn | 726 | 150 | 3.2 | 2.6 | 15.95 | 25.22 | 10.09%
| LightCTS | 42 | 103 | 0.3 | 2.5 | 14.82 | 23.78 | 9.66%
2.5 | GwNet | 137 | 49 | 0.8 | 2.4 | 18.00 | 28.20 | 11.49%
| AgCrn | 586 | 115 | 2.9 | 2.5 | 16.72 | 26.26 | 11.27%
| LightCTS | 8 | 15 | 0.2 | 2.0 | 16.70 | 26.28 | 10.75%
2 | GwNet | 40 | 10 | 0.6 | 2.0 | 19.38 | 30.14 | 13.58%
| AgCrn | 100 | 11 | 2.2 | 2.0 | 19.17 | 29.71 | 13.06%

##  6\. Related Work

DL-based Models for CTS Forecasting. Deep learning models dominate CTS
forecasting. Different studies involve different S/T-operators. GCNs and GCN
variants are the most common S-operators (Li et al., 2018; Yu et al., 2018b;
Bai et al., 2020; Wu et al., 2020; Chen et al., 2022). In addition to building
graphs using prior knowledge in GCNs, learned graphs (Wu et al., 2019, 2020;
Chen et al., 2022) and adaptive graphs (Bai et al., 2020) demonstrate
advantages in capturing dynamic and implicit correlations among time series.
Further, TCNs (Yu et al., 2018b; Wu et al., 2019, 2020; Guo et al., 2019;
Huang et al., 2020; Chen et al., 2022) and RNNs (Bai et al., 2020; Chen et
al., 2020a; Chang et al., 2018; Li et al., 2018; Shih et al., 2019) are the
most widely adopted T-operators. Recently, the Transformer and its variants
have been employed as S-operators (Xu et al., 2020; Park et al., 2020; Zhou et
al., 2021; Wu et al., 2021) and T-operators (Xu et al., 2020; Park et al.,
2020; Zhou et al., 2021; Wu et al., 2021), due to the powerful correlation
modeling abilities of Transformers. Neural architecture search (NAS) has been
introduced to automatically select appropriate S/T-operators, resulting in
competitive performance without having to design a CTS forecasting model
manually (Wu et al., 2021; Pan et al., 2021).

All existing studies focus on improving forecasting accuracy. However,
progress is slowing down and becoming marginal (see Tables 6 and 7). In
contrast, LightCTS contributes lightweight S/T-operators and enables
lightweight CTS models (w.r.t. computation and model size) without
compromising forecasting accuracy. In this sense, LightCTS renders forecasting
more cost-effective and extends its potential applicability to edge devices in
CPSs.

Lightweight DL Models. Developing light DL models is motivated by the
requirements of real-time and mobile applications. There are two streams of
related work (Chen et al., 2020b): compressing well-trained big models and
designing lightweight models from scratch. The first stream has been well
studied in areas like CV (Krizhevsky et al., 2012; Han et al., 2015; Yu et
al., 2018a) and NLP (Michel et al., 2019; Jiao et al., 2020; Wang et al.,
2020) fields. However, LightCTS falls outside this stream as there are no
compelling well-trained CTS models.

Next, impressive advances in the design of lightweight models from scratch
have also been widely reported in CV (Zhang et al., 2018; Sandler et al.,
2018; Tan and Le, 2019; Howard et al., 2017, 2019; Wu et al., 2018). A popular
operator is the depth-wise separable convolution that decouples the standard
convolution into intra- and inter-feature map computation. It serves as the
basic block of the famous MobileNets (Howard et al., 2017) and Xception
(Chollet, 2017). A follow-up work is the inverted bottleneck structure
(Sandler et al., 2018) that finds a narrow-wide-narrow convolution to achieve
reduced computations while competitive accuracy. From a different perspective,
EfficientNet (Tan and Le, 2019) aims to scale existing modules to meet certain
constraints rather than designing new efficient modules. However, the
aforementioned lightweight modules cannot be applied directly to CTS
forecasting because of the inherently different data structures and tasks
involved. For example, these methods focus mainly on simplifying 2D and 3D
convolutions for extracting local features of image and video data, while CTS
models require uncovering long-term temporal dynamics and non-uniform spatial
correlations. Given this gap, we identify potential directions to achieve
lightness of CTS forecasting based on a careful study of existing CTS models.
LightCTS offers a plain stacking architecture together with a last-shot
compression to efficiently deal with the heterogeneity of S/T-operators, which
is unique compared to models used in CV. We also design L-TCN and GL-Former
according to the exclusive characteristics of temporal dynamics and spatial
correlations in CTS.

##  7\. Conclusion

We present LightCTS, a new framework for lightweight forecasting of correlated
time series (CTS) that achieves comparable accuracy to state-of-the-art CTS
forecasting models but consumes much fewer computational resources. LightCTS
integrates a set of novel computational cost reduction techniques, notably a
plain stacking architecture, the L-TCN (Light Temporal Convolutional Network)
and GL-Former (GlobalLocal TransFormer) modules for extracting spatio-temporal
features, and a last-shot compression scheme for reducing redundant,
intermediate features. Comprehensive experiments offer evidence that LightCTS
is capable of providing state-of-the-art CTS forecasting accuracy with much
fewer FLOPs and parameters than existing CTS forecasting proposals.

## References

  * (1)
  * fvc (2023) 2023\.  Fvcore.  https://github.com/facebookresearch/fvcore.  (Accessed Jan 2023).
  * lig (2023) 2023\.  LightCTS project: Code, datasets and supplemental material.  https://github.com/AI4CTS/lightcts.  (Accessed Jan 2023).
  * STM (2023) 2023\.  Microcontrollers and microprocessors, STMicroelectronics.  https://www.st.com/en/microcontrollers-microprocessors.html.  (Accessed Jan 2023).
  * Bai et al. (2020) Lei Bai, Lina Yao, Can Li, Xianzhi Wang, and Can Wang. 2020.  Adaptive graph convolutional recurrent network for traffic forecasting. In _NeurIPS_. 17804–17815.
  * Cao and Tay (2003) Li-Juan Cao and Francis Eng Hock Tay. 2003.  Support vector machine with adaptive parameters in financial time series forecasting.  _TNNLS_ 14, 6 (2003), 1506–1518.
  * Chang et al. (2018) Yen-Yu Chang, Fan-Yun Sun, Yueh-Hua Wu, and Shou-De Lin. 2018\.  A memory-network based solution for multivariate time-series forecasting.  _arXiv preprint arXiv:1809.02105_ (2018).
  * Chen et al. (2021) Boyu Chen, Peixia Li, Baopu Li, Chen Lin, Chuming Li, Ming Sun, Junjie Yan, and Wanli Ouyang. 2021\.  BN-NAS: Neural architecture search with batch normalization. In _CVPR_. 307–316.
  * Chen et al. (2022) Ling Chen, Donghui Chen, Zongjiang Shang, Youdong Zhang, Bo Wen, and Chenghu Yang. 2022\.  Multi-scale adaptive graph neural network for multivariate time series forecasting.  _arXiv preprint arXiv:2201.04828_ (2022).
  * Chen et al. (2020a) Weiqi Chen, Ling Chen, Yu Xie, Wei Cao, Yusong Gao, and Xiaojie Feng. 2020a.  Multi-range attentive bicomponent graph convolutional network for traffic forecasting. In _AAAI_. 3529–3536.
  * Chen et al. (2020b) Yanjiao Chen, Baolin Zheng, Zihan Zhang, Qian Wang, Chao Shen, and Qian Zhang. 2020b.  Deep learning on mobile and embedded devices: State-of-the-art, challenges, and future directions.  _ACM Comput. Surv._ 53, 4 (2020), 1–37.
  * Cheng et al. (2022) Xu Cheng, Fan Shi, Xiufeng Liu, Meng Zhao, and Shengyong Chen. 2022.  A novel deep class-imbalanced semisupervised model for wind turbine blade icing detection.  _TNNLS_ 33, 6 (2022), 2558–2570.
  * Chollet (2017) François Chollet. 2017\.  Xception: Deep learning with depthwise separable convolutions. In _CVPR_. 1251–1258.
  * Cirstea et al. (2021) Razvan-Gabriel Cirstea, Tung Kieu, Chenjuan Guo, Bin Yang, and Sinno Jialin Pan. 2021\.  Enhancenet: Plugin neural networks for enhancing correlated time series forecasting. In _ICDE_. 1739–1750.
  * de Freitas et al. (1999) Nando de Freitas, Marta Milo, Philip Clarkson, Mahesan Niranjan, and Andrew Gee. 1999.  Sequential support vector machines. In _IEEE Signal Processing Society Workshop_. 31–40.
  * Defferrard et al. (2016) Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. 2016.  Convolutional neural networks on graphs with fast localized spectral filtering. In _NIPS_. 3837–3845.
  * Derler et al. (2011) Patricia Derler, Edward A Lee, and Alberto Sangiovanni Vincentelli. 2011\.  Modeling cyber-physical systems.  _Proc. IEEE_ 100, 1 (2011), 13–28.
  * Du et al. (2019) Shengdong Du, Tianrui Li, Yan Yang, and Shi-Jinn Horng. 2019\.  Deep air quality forecasting using hybrid deep learning framework.  _TKDE_ 33, 6 (2019), 2412–2424.
  * Faloutsos et al. (2019) Christos Faloutsos, Jan Gasthaus, Tim Januschowski, and Yuyang Wang. 2019.  Classical and contemporary approaches to big time series forecasting. In _SIGMOD_. 2042–2047.
  * Gasteiger et al. (2019) Johannes Gasteiger, Stefan Weißenberger, and Stephan Günnemann. 2019\.  Diffusion improves graph learning. In _NeurIPS_. 13333–13345.
  * Grigsby et al. (2021) Jake Grigsby, Zhe Wang, and Yanjun Qi. 2021.  Long-range transformers for dynamic spatiotemporal forecasting.  _arXiv preprint arXiv:2109.12218_ (2021).
  * Guo et al. (2019) Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. 2019.  Attention based spatial-temporal graph convolutional networks for traffic flow forecasting. In _AAAI_. 922–929.
  * Han et al. (2020) Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. 2020\.  Ghostnet: More features from cheap operations. In _CVPR_. 1580–1589.
  * Han et al. (2015) Song Han, Jeff Pool, John Tran, and William Dally. 2015\.  Learning both weights and connections for efficient neural network. In _NIPS_. 1135–1143.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _CVPR_. 770–778.
  * Howard et al. (2019) Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. 2019\.  Searching for mobilenetv3. In _CVPR_. 1314–1324.
  * Howard et al. (2017) Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. 2017\.  Mobilenets: Efficient convolutional neural networks for mobile vision applications.  _arXiv preprint arXiv:1704.04861_ (2017).
  * Hu et al. (2018) Jie Hu, Li Shen, and Gang Sun. 2018.  Squeeze-and-excitation networks. In _CVPR_. 7132–7141.
  * Huang et al. (2020) Rongzhou Huang, Chuyin Huang, Yubao Liu, Genan Dai, and Weiyang Kong. 2020.  LSGCN: Long short-term traffic prediction with graph convolutional networks. In _IJCAI_. 2355–2361.
  * Huang et al. (2019) Siteng Huang, Donglin Wang, Xuehan Wu, and Ao Tang. 2019\.  Dsanet: Dual self-attention network for multivariate time series forecasting. In _CIKM_. 2129–2132.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020\.  TinyBERT: Distilling BERT for natural language understanding. In _EMNLP_. 4163–4174.
  * Krizhevsky et al. (2012) Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012\.  Imagenet classification with deep convolutional neural networks. In _NIPS_. 1106–1114.
  * Lai et al. (2018) Guokun Lai, Wei-Cheng Chang, Yiming Yang, and Hanxiao Liu. 2018\.  Modeling long-and short-term temporal patterns with deep neural networks. In _SIGIR_. 95–104.
  * Lai et al. (2022) Zhichen Lai, Xu Cheng, Xiufeng Liu, Lizhen Huang, and Yongping Liu. 2022.  Multiscale wavelet-driven graph convolutional network for blade icing detection of wind turbines.  _IEEE Sensors Journal_ 22, 22 (2022), 21974–21985.
  * Lea et al. (2016) Colin Lea, Rene Vidal, Austin Reiter, and Gregory D Hager. 2016\.  Temporal convolutional networks: A unified approach to action segmentation. In _ECCV_. 47–54.
  * Li et al. (2021) Changlin Li, Guangrun Wang, Bing Wang, Xiaodan Liang, Zhihui Li, and Xiaojun Chang. 2021\.  Dynamic slimmable network. In _CVPR_. 8607–8617.
  * Li et al. (2018) Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. 2018\.  Diffusion convolutional recurrent neural network: Data-driven traffic forecasting. In _ICLR_. 1–10.
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin transformer: Hierarchical vision transformer using shifted windows. In _CVPR_. 10012–10022.
  * Ma et al. (2018) Lin Ma, Dana Van Aken, Ahmed Hefny, Gustavo Mezerhane, Andrew Pavlo, and Geoffrey J Gordon. 2018\.  Query-based workload forecasting for self-driving database management systems. In _SIGMOD_. 631–645.
  * Mehta and Rastegari (2021) Sachin Mehta and Mohammad Rastegari. 2021.  MobileViT: Light-weight, general-purpose, and mobile-friendly vision transformer. In _ICLR_.
  * Michel et al. (2019) Paul Michel, Omer Levy, and Graham Neubig. 2019.  Are sixteen heads really better than one?. In _NeurIPS_. 14014–14024.
  * Morales-Hernández et al. (2022) Alejandro Morales-Hernández, Inneke Van Nieuwenhuyse, and Sebastian Rojas Gonzalez. 2022.  A survey on multi-objective hyperparameter optimization algorithms for Machine Learning.  _Artificial Intelligence Review_ (2022), 1–51.
  * Pan et al. (2021) Zheyi Pan, Songyu Ke, Xiaodu Yang, Yuxuan Liang, Yong Yu, Junbo Zhang, and Yu Zheng. 2021.  AutoSTG: Neural architecture search for predictions of spatio-temporal graph. In _WWW_. 1846–1855.
  * Papadimitriou and Yu (2006) Spiros Papadimitriou and Philip Yu. 2006.  Optimal multi-scale patterns in time series streams. In _SIGMOD_. 647–658.
  * Park et al. (2020) Cheonbok Park, Chunggi Lee, Hyojin Bahng, Yunwon Tae, Seungmin Jin, Kihwan Kim, Sungahn Ko, and Jaegul Choo. 2020\.  ST-GRAT: A novel spatio-temporal graph attention networks for accurately forecasting dynamically changing road speed. In _CIKM_. 1215–1224.
  * Pedersen et al. (2020) Simon Aagaard Pedersen, Bin Yang, and Christian S Jensen. 2020\.  Anytime stochastic routing with hybrid learning.  _PVLDB_ 13, 9 (2020), 1555–1567.
  * Rao et al. (2022) Xuan Rao, Hao Wang, Liang Zhang, Jing Li, Shuo Shang, and Peng Han. 2022\.  FOGS: First-order gradient supervision with learning-based graph for traffic flow forecasting. In _IJCAI_.
  * Sandler et al. (2018) Mark Sandler, Andrew G. Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. 2018.  MobileNetV2: Inverted residuals and linear bottlenecks. In _CVPR_. 4510–4520.
  * Shi et al. (2016) Weisong Shi, Jie Cao, Quan Zhang, Youhuizi Li, and Lanyu Xu. 2016.  Edge computing: Vision and challenges.  _IEEE Internet Things J._ 3, 5 (2016), 637–646.
  * Shih et al. (2019) Shun-Yao Shih, Fan-Keng Sun, and Hung-yi Lee. 2019.  Temporal pattern attention for multivariate time series forecasting.  _Mach. Learn._ 108, 8 (2019), 1421–1441.
  * Song et al. (2020) Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. 2020\.  Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting. In _AAAI_. 914–921.
  * Sudharsan et al. (2021) Bharath Sudharsan, John G Breslin, and Muhammad Intizar Ali. 2021\.  Ml-mcu: A framework to train ml classifiers on mcu-based iot edge devices.  _IEEE Internet Things J._ (2021).
  * Takamoto et al. (2020) Makoto Takamoto, Yusuke Morishita, and Hitoshi Imaoka. 2020\.  An efficient method of training small models for regression problems with knowledge distillation. In _MIPR_. IEEE, 67–72.
  * Tan and Le (2019) Mingxing Tan and Quoc Le. 2019.  Efficientnet: Rethinking model scaling for convolutional neural networks. In _ICML_. 6105–6114.
  * Tay and Cao (2002) Francis EH Tay and LJ Cao. 2002.  Modified support vector machines in financial time series forecasting.  _Neurocomputing_ 48, 1-4 (2002), 847–861.
  * van den Oord et al. (2016) Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew W. Senior, and Koray Kavukcuoglu. 2016.  WaveNet: A generative model for raw audio. In _ISCA Speech Synthesis Workshop_. 125\.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _NeurIPS_ , Vol. 30.
  * Vladimirova et al. (2019) Mariia Vladimirova, Jakob Verbeek, Pablo Mesejo, and Julyan Arbel. 2019.  Understanding priors in Bayesian neural networks at the unit level. In _ICML_. 6458–6467.
  * Wang et al. (2020) Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020.  Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In _NeurIPS_. 5776–5788.
  * Wu et al. (2018) Bichen Wu, Alvin Wan, Xiangyu Yue, Peter Jin, Sicheng Zhao, Noah Golmant, Amir Gholaminejad, Joseph Gonzalez, and Kurt Keutzer. 2018.  Shift: A zero flop, zero parameter alternative to spatial convolutions. In _CVPR_. 9127–9135.
  * Wu et al. (2021) Xinle Wu, Dalin Zhang, Chenjuan Guo, Chaoyang He, Bin Yang, and Christian S Jensen. 2021\.  AutoCTS: Automated correlated time series forecasting.  _PVLDB_ 15, 4 (2021), 971–983.
  * Wu et al. (2020) Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, Xiaojun Chang, and Chengqi Zhang. 2020\.  Connecting the dots: Multivariate time series forecasting with graph neural networks. In _KDD_. 753–763.
  * Wu et al. (2019) Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. 2019.  Graph WaveNet for deep spatial-temporal graph modeling. In _IJCAI_. 1907–1913.
  * Xie et al. (2017) Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. 2017.  Aggregated residual transformations for deep neural networks. In _CVPR_. 1492–1500.
  * Xu et al. (2020) Mingxing Xu, Wenrui Dai, Chunmiao Liu, Xing Gao, Weiyao Lin, Guo-Jun Qi, and Hongkai Xiong. 2020.  Spatial-temporal transformer networks for traffic flow forecasting.  _arXiv preprint arXiv:2001.02908_ (2020).
  * Yu et al. (2018b) Bing Yu, Haoteng Yin, and Zhanxing Zhu. 2018b.  Spatio-temporal graph convolutional networks: a deep learning framework for traffic forecasting. In _IJCAI_. 3634–3640.
  * Yu and Koltun (2016) Fisher Yu and Vladlen Koltun. 2016.  Multi-scale context aggregation by dilated convolutions. In _ICLR_. 1–10.
  * Yu et al. (2018a) Ruichi Yu, Ang Li, Chun-Fu Chen, Jui-Hsin Lai, Vlad I Morariu, Xintong Han, Mingfei Gao, Ching-Yung Lin, and Larry S Davis. 2018a.  NISP: Pruning networks using neuron importance score propagation. In _CVPR_. 9194–9203.
  * Yuan et al. (2020) Haitao Yuan, Guoliang Li, Zhifeng Bao, and Ling Feng. 2020\.  Effective travel time estimation: When historical trajectories over road networks matter. In _SIGMOD_. 2135–2149.
  * Zhang et al. (2018) Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. 2018\.  Shufflenet: An extremely efficient convolutional neural network for mobile devices. In _CVPR_. 6848–6856.
  * Zhou et al. (2021) Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang. 2021.  Informer: Beyond efficient transformer for long sequence time-series forecasting. In _AAAI_. 11106–11115.
  * Zhu and Shasha (2002) Yunyue Zhu and Dennis Shasha. 2002.  Statstream: Statistical monitoring of thousands of data streams in real time.  (2002), 358–369.