*[table]position=top

# Characterization of a CMOS pixel sensor for charged particle tracking

L. Li  L. Zhang, 11footnotetext: Corresponding author. J. N. Dong  J. Liu  and
M. Wang

###### Abstract

A prototype of the CMOS pixel sensor named Supix-1has been fabricated and
tested in order to investigate the feasibility of a pixelated tracker for a
proposed Higgs factory, namely, the Circular Electron-Positron Collider
(CEPC). The sensor, taped out with a 180CMOS Image Sensor (CIS) process,
consists of nine different pixel arrays varying in pixel pitches, diode sizes
and geometries in order to study the particle detection performance of
enlarged pixels. The test was carried out with a ${}^{55}\text{{Fe}}$
radioactive source. Two soft X-ray peaks observed were used to calibrate the
charge to voltage factor of the sensor. The pixel-wise equivalent noise
charge, charge collection efficiency and signal-to-noise ratio were evaluated.
A reconstruction method for clustering pixels of a signal has been developed
and the cluster-wise performance was studied as well. The test results show
that pixels with the area as large as of $21\text{\times}84\text{\,}\um$ have
satisfactory noise level and charge collection performance, meeting general
requirements for a pixel sensor. This contribution demonstrates that the CMOS
pixel sensor with enlarged pitches, using the CIS technology, can be used in
tracking for upcoming collider detectors akin to the CEPC.

##  1 Introduction

The discovery of the Higgs boson [1, 2] opens a new window for studying the
fundamental physics principles of the Standard Model (SM) and exploring beyond
the SM. The Circular Electron-Positron Collider (CEPC) [3] is a proposed Higgs
factory to conduct precision measurements of the Higgs boson. In the
conceptual design of the CEPC detector [4], the tracking system involves a
silicon tracker, with silicon microstrip sensors as the basic technology.
However, a fully pixelated silicon tracker with large pixels is also under
consideration and the CMOS Pixel Sensor (CPS) has gained particular interest
due to high granularity and low material budget. The CEPC high precision
tracking requires a general condition of the single point resolution,
$\sigma_{\text{sp}}<7\um$, on the transverse plane with regard to the magnetic
field. In the longitudinal direction, however, the resolution is undefined,
but a loose requirement at least a few times larger than $7\um$ is
anticipated. Without considering the effect of charge sharing, the nominal
pixel size will be about $20\text{\times}80\text{\,}\mathrm{\mathsf{\mu}m}$ or
larger. With the standard CMOS Image Sensor (CIS) process, the CPS pixels
realized so far used in high-energy physics experiments are relatively small.
For instance, a CPS based vertex detector for the STAR experiment at the
Relativistic Heavy Ion Collider (RHIC), Brookhaven National Laboratory,
employs pixels of $20.7\text{\times}20.7\text{\,}\mathrm{\mathsf{\mu}m}$ [5].
ALPIDE, another CPS chip which is going to be used for the major upgrade of
the inner tracking system for the ALICE experiment at Large Hadron Collider
(LHC), CERN, is developed in
$27\text{\times}29\text{\,}\mathrm{\mathsf{\mu}m}$ pixels [6].

Figure 1: Design of the Supix-1sensor: (a) a photograph and the schematic of
in-pixel circuit, (b) illustration of a CMOS pixel cross section and (c) the
staggered arrangement of diodes in sectors with the largest x-pitch.

In order to investigate the feasibility of relatively large pixels based on
the standard CMOS imaging process, a prototype chip, named Supix-1(Shandong
University PIXel), has been fabricated and tested. The chip is designed with
three different pixel sizes, up to
$21\text{\times}84\text{\,}\mathrm{\mathsf{\mu}m}$, and variant diode
geometries. The main purpose of the design is to explore the interplay of the
pixel pitch and the diode geometry on the charge collection efficiency.
Details of the chip design can be found in ref. [7], while some
characteristics are briefed here. As shown in Figure 1, the chip contains nine
sectors (only sectors A0, A2, A5, A7 and A8 were accessible during the test)
and has an overall area of $2\text{\times}8\text{\,}\mathrm{m}\mathrm{m}$.
Each sector contains 64 rows and 16 columns (labelled from C0 to C15) of
pixels. The sensitive areas and the pixel pitches of relevant sectors are
listed in Table 1. The schematic of in-pixel circuit is also shown in Figure
1, employing a 3T architecture. The collected charge in a pixel is converted
into the signal voltage through an N-well/P-epitaxial layer diode (D1), as
illustrated in Figure 1. Combinations of various diode geometries and pixel
pitches are implemented in different sectors, as listed in Table 1. The diode
surface is defined as the area of the N-well and the diode footprint is the
total area formed by the surrounding P-well. A staggered pixel design is
implemented in sectors with the largest x-pitch
($84\text{\,}\mathrm{\mathsf{\mu}m}$), as shown in Figure 1, to enhance charge
collection. The analog signal of each sector is read out in a rolling shutter
mode. The readout time for a frame is $32\text{\,}\mathrm{\mathsf{\mu}s}$ with
a basic clock of $2\text{\,}\mathrm{M}\mathrm{H}\mathrm{z}$.

In this article, we present the test of Supix-1with a radioactive source of
${}^{55}\text{{Fe}}$. The procedure for characterizing the sensor performance,
taking the sector of A0 as an example, is described. The manuscript is
organized as follows: a brief introduction of the test system is given in
Section 2, steps of the test are elaborated in Section 3, test results are
discussed in Section 4, and we conclude in Section 5.

##  2 The test system for Supix-1

As shown in Figure 2, the test system for Supix-1consists of readout
electronics and a data acquisition system (SupixDAQ). The readout electronics
involves a device under test (DUT) board, an analog-to-digital conversion
(ADC) board, a field programmable gate array (FPGA) board as well as a
personal computer (PC).

The DUT board

connects to the Supix-1chip by wire bonding and drives analog signal out of
the chip.

The ADC board

digitizes the analog signal with 16-bit ADCs (LTC2323-16) and transmits
digitized data to the FPGA board through a FPGA mezzanine card (FMC).

The FPGA board

configures the ADC board and the DUT board and transmits the digitized data to
the PC for data taking through a PCI-express interface. The FPGA is Xilinx
Kintex-7 FPGA KC-705.

The PC

manages data acquisition and records the data.

Figure 2: (a) Test system for the Supix-1sensor and (b) working flow of the
system.

The readout electronics was calibrated with the input voltage from a signal
generator (Tektronix AFG3252C). As shown in Figure 3, the linearity is good in
the range of the input voltage. The electronics calibration constant is
defined as

$A\equiv\frac{\text{ADC}}{\text{Voltage}},$ (2.1)

where ADC represents the ADC count. The calibration resulted in $A=$7.76\pm
0.04\text{\,}\mathrm{A}\mathrm{D}\mathrm{C}\mathrm{/}\mathrm{m}\mathrm{V}$$.

Figure 3: Calibration for the readout electronics: (a) the ADC output of an
input square wave with $1\text{\,}\mathrm{V}$ amplitude and 1frequency and (b)
the ADC output as a function of the input voltage together with a linear
function fit.

As the readout of Supix-1is in rolling shutter, the SupixDAQ takes data in
frames continuously and implements the correlated double sampling (CDS)
method. As demonstrated in Figure 4, the ADC distribution of a frame is
subtracted by that of a preceding frame, resulting in the CDS distribution.
The CDS can cancel the pixel-wise fluctuation of baseline and suppress the
fixed pattern noise significantly. Hence, the CDS value of ADC, also called
ADC, will be used in the work unless explicitly distinguished.

Figure 4: Demonstration of the correlated double sampling (CDS) method: (a)
the ADC output of a frame, (b) the ADC output of a preceding frame and (c) the
result of (a) minus (b), namely CDS of the frame.

In order to calibrate noise pixel-by-pixel, a _noise run_ is taken before
taking data with a radioactive source, namely a _calibration run_. As an
example, the noise level before and after CDS of each pixel in the sector of
A0 is shown in Figure 5(a). Both distributions of the mean and the standard
deviation of the pixel-by-pixel noise are shown in Figure 5(b) and (c),
separately. Values of the latter are noise calibration constants. During the
test, because the pixels of a few columns (C0, C1, C4 and C7) did not work
properly and the reconstruction of the pixel cluster would be studied, only
the pixels of columns from C8 to C15 were used.

Figure 5: Results of a noise run for the sector of A0: (a) profiles of the
mean and standard deviation of the pixel-by-pixel distribution of the raw ADC
(up) and the ADC after CDS (down), and distributions of the CDS mean (b) and
standard deviation (c) of pixels. Only pixels of the columns from C8 to C15
are shown. Figure 6: (a) The fake-trigger rate (left axis) and the fake-hit
rate (right axis) versus the trigger threshold, and (b) the SNR distribution
of the seed pixel. See the text for explanations.

The noise calibration constants are used in the calibration run to define the
hit pixel, with the condition of $-\text{CDS}>8\sigma$. A frame having at
least one hit is triggered for recording for off-line analysis. The _trigger
threshold_ of $8\sigma$ is chosen to suppress the noise hit as many as
possible while losing the physical hit as few as possible. Figure 6(a) shows
the fake-trigger rate and the fake-hit rate as functions of the trigger
threshold for a noise run. For the threshold beyond $5\sigma$, the fake-
trigger rate approaches zero and the fake-hit rate decreases exponentially as
the trigger threshold increases. Figure 6(b) shows, for a calibration run, the
distribution of the signal-to-noise ratio (SNR) of the seed pixel, which has
the largest value in a frame, and the SNR is just $-\text{CDS}/\sigma$.
Obviously the cut on $8\sigma$ loses physical hits rarely, if any. In
addition, the SNR distribution of all pixels for the same run is shown in
Figure 10(a). With the chosen threshold, the rates of fake trigger and fake
hit are $(7\pm
3.)\text{\times}{10}^{-6}\text{\,}\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{g}\mathrm{e}\mathrm{r}^{-1}$
and $(1.8\pm
0.6)\text{\times}{10}^{-8}\text{\,}\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}^{-1}\,\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{g}\mathrm{e}\mathrm{r}^{-1}$,
respectively.

Figure 7:  Sensor performance under ${}^{55}\text{{Fe}}$ irradiation: (a) hit
map of pixels, (b) distribution of the number of hit pixels per frame and (c)
distribution of the ADC after CDS of the hit pixel.

A hit map of the pixel for a sector is shown in Figure 7(a). There are much
more hits in the first two rows than in the others, which happens in all
sectors. The cause is unknown yet, and the two rows of pixels were excluded
from searching the seed pixel. Corresponding distributions of the number of
hit pixels per frame and the CDS of hit pixels are shown in Figure 7(b) and
Figure 7(c), respectively. As the CDS value of the hit pixel is negative, we
use its opposite value in the following text for convenience.

##  3 The chip test with ${}^{55}\text{{Fe}}$

###  3.1 The sensor gain calibration

The sensor gain calibration is carried out using the soft X-rays of
${}^{55}\text{{Fe}}$, of which k-$\alpha$ (5.9) accounts for 24.4% and
k-$\beta$ (6.5) for 2.9% in yield. A 5.9photon can generate about 1640
electron-hole pairs when absorbed in silicon bulk, assuming a constant value,
3.6at room temperature, is required to create an electron-hole pair [8]. In
most cases, the ionized electrons are collected by several pixels, forming a
pixel cluster, through thermal diffusion in the epitaxial layer. Specifically,
the electrons will be collected by a single pixel if the photons are absorbed
in the depleted region of that pixel, forming the full energy peak [9]. Figure
8 shows, for instance, a distribution of the seed-pixel ADC, on which two
full-energy peaks (_calibration peaks_) around 169 and 185 can be observed,
corresponding to the X-rays of k-$\alpha$ and k-$\beta$, respectively.

Figure 8: The distribution of the seed-pixel ADC, irradiated with
${}^{55}\text{{Fe}}$, together with a full spectrum fit. See the text for
explanations.

As shown in Figure 8, the full spectrum can be fitted well with a combination
of component functions. The addition of a crystal ball function (CB, blue
dashed line) for k-$\alpha$ peak and a Gaussian function ($\text{G}_{\beta}$)
for k-$\beta$ peak is used to describe the calibration peaks. The power-law
tail of the crystal ball function is justified considering the partial energy
deposition in the depleted region. The peak with the highest amplitude, named
the _collection peak_ , is described by two Gaussian functions (G1 and G2)
superposed on the crystal-ball tail. There are two ways that ionized electrons
can contribute to the collection peak: (a) directly by an incident X-ray
photon in the vicinity of the seed pixel, and (b) by charge sharing from
neighbor pixels. Hence, the use of two Gaussians could be justified. The
Landau distribution fails to describe the collection peak because most
incident X-ray photons are fully absorbed in the epitaxial layer. Towards the
low end of the spectrum, an exponential function (E) is added to describe the
pedestal associated with the electronic noise.

The pixel gain is defined as

$G=\frac{\text{ADC}_{\text{peak}}-\text{ADC}_{\text{pedestal}}}{\text{Q}},$
(3.1)

where $Q$ is for charge. Assuming the linear response, the gain can be
obtained by comparing the locations of the two calibration peaks, with no need
for pedestal subtraction. After the gain calibration, the charge-to-voltage
factor (CVF) can be determined by

$\text{CVF}=\frac{G}{A},$ (3.2)

where $A$ is the electronics calibration factor in eq. (2.1). Also, the fixed
pattern noise shown in Figure 5(c) can be transformed into the equivalent
noise charge (ENC),

$\text{ENC}=\frac{\left\langle\text{RMS}_{\text{noise}}\right\rangle}{G},$
(3.3)

where $\left\langle\text{RMS}_{\text{noise}}\right\rangle$ is the mean of
$\text{RMS}_{\text{CDS}}$ in Figure 5(c). The ENC and the CVF are inversely
correlated.

We use the fitted location of the collection peak, namely the most probable
value (MPV) of ADC, to characterize the charge collection efficiency (CCE) and
the SNR of the pixel for a sector,

$\displaystyle\text{CCE}_{\text{pixel}}$
$\displaystyle=\frac{\text{ADC}_{\text{MPV,
seed}}}{\text{ADC}_{\text{k-}\alpha,\text{seed}}},$ (3.4)
$\displaystyle\text{SNR}_{\text{pixel}}$
$\displaystyle=\frac{\text{ADC}_{\text{MPV,
seed}}}{\left\langle\text{RMS}_{\text{noise}}\right\rangle},$ (3.5)

where $\text{ADC}_{\text{k-}\alpha,\text{seed}}$ is the location of the
k-$\alpha$ peak. Results for all sectors are summarized in Table 1.

###  3.2 The anomalous pixel output and a correction method

An anomaly was observed when we studied reconstruction for the pixel cluster.
For each frame in a $5\times 5$ array centering at the seed pixel, we figured
out the top five pixels, including the seed pixel, based on the ordering of
ADC. Figure 9(a) shows the relative position distribution of the top five
pixels. Among four adjacent pixels, the frequency of the pixel in the same
column as the seed but being read out immediately after the seed, tagged as
PIX_F, is anomalously high. In Figure 9(b), the ADC of each adjacent pixel is
drawn as a function of the seed-pixel ADC, and the anomaly is significant.
Particularly, in the ADC region greater than 170, where the seed pixel absorbs
mostly all ionized charges, the ADC of the three normal pixels fluctuates
around zero. In comparison, the ADC of the anomalous pixel increases linearly
as the seed ADC increases. The anomaly might be related to the rolling shutter
readout, but remains unknown yet.

Figure 9:  (a) Distribution of the relative position of the top five pixels
ordered in ADC within a $5\times 5$ array centering at the seed pixel per
frame, and (b) the ADC of each adjacent pixel and the corrected ADC of the
anomalous pixel as a function of the seed-pixel ADC. The tag PIX_F stands for
the anomalous pixel. See the text for explanations.

An empirical method was developed to correct the output of the anomalous
pixel, that is also illustrated in Figure 9(b). A linear function, $f_{a}(x)$,
where $x$ stands for ADC of the seed pixel, is obtained via fitting the
anomalous curve in the range of $x>170$. Another linear function, $f_{b}(x)$,
is obtained via fitting the normal curves in the range of $130<x<170$. Then a
linear combination of the two functions, $\alpha f_{a}+\beta f_{b}$, is fitted
against the anomalous curve in the same fit range as that of $f_{b}$ to
determine the coefficients $\alpha$ and $\beta$. Finally, the anomaly is
corrected with the equation

$\text{PIX\\_F\\_cor}=\frac{\text{PIX\\_F}(x)-\alpha f_{a}(x)}{\beta},$ (3.6)

where PIX_F and PIX_F_cor are the original and corrected ADCs of the anomalous
pixel, respectively. As shown in Figure 9(b), the correction (cyan curve) is
reasonable, but it is somehow underestimated in the range of
$\text{CDS}_{\text{seed}}<100\,$\mathrm{A}\mathrm{D}\mathrm{C}$$. The effect
of the bias to the pixel clustering is negligible based on further study.

###  3.3 Reconstruction of the pixel cluster

Most X-ray photons of ${}^{55}\text{{Fe}}$ deposit their energy in the
epitaxial layer and the ionized electrons are collected by a cluster of
pixels. Hence, the clustering algorithm as well as the property of the cluster
was studied. We tried several methods to reconstruct the cluster of pixels.
Their performance was similar and one of them is reported here.

Figure 10: Reconstruction for the cluster of pixel: (a) distributions of the
pixel SNR for a noise run and a calibration run, (b) distribution of the
number of pixels per cluster, (c) distribution of the cluster SNR, and (d)
distribution of the cluster ADC and a fit for the distribution, together with
the distribution of the seed-pixel ADC for comparison. See the text for
explanations.

A cluster is defined by a group of adjacent fired pixels, which have signals
above a _clustering threshold_. The cluster reconstruction starts with a seed
pixel, selects neighboring fired pixels including diagonal ones, and repeats
the selection for newly selected pixels until there are no fired neighbors
anymore or reaching an edge of the pixel array. The clustering threshold for
the fired pixel is based on the SNR of individual pixels. As indicated in
Figure 10(a), the threshold is set to $3.1$, corresponding to less than one
per mil of noise pixels leaked into clustering pixels. The number of pixels
and the SNR of the reconstructed cluster are shown in Figure 10(b) and (c),
respectively. The cluster SNR is evaluated as

$\text{SNR}_{\text{cluster}}=\frac{\sum_{i}\text{ADC}_{i}}{\sqrt{\sum_{i}\sigma_{i}^{2}}},$
(3.7)

where $\text{ADC}_{i}$ and $\sigma_{i}$ are the ADC and the noise of the
$i$-th pixel. The numerator is just the ADC of a cluster. Most of the clusters
consist of 4 or 5 pixels and have the SNR greater than 25. On the SNR
distribution, there is a small peak at about 85, which attributes mainly to
the single-pixel clusters.

The distribution of the cluster ADC is shown in Figure 10(d), together with
that of the seed pixel for comparison. The distribution can be fitted well
with a few Gaussian functions superposed with a crystal ball function. The
peak of the distribution shifts toward the left with regard to the k-$\alpha$
peak. The shift, indeed, depends on the clustering threshold, which is a
compromise between the noise pixel and the physically fired pixel. We can use
the location of the peak to define the charge collection efficiency of the
cluster,

$\text{CCE}_{\text{cluster}}=\frac{\text{ADC}_{\text{MPV,
cluster}}}{\text{ADC}_{\text{k-}\alpha,\text{seed}}}.$ (3.8)

Both of the $\text{SNR}_{\text{cluster}}$ and $\text{CCE}_{\text{cluster}}$
can characterize the performance of the sensor under test. Results are
summarized in Table 1, where the MPV of $\text{SNR}_{\text{cluster}}$, as
shown in Figure 10(c), is reported.

##  4 Test results

All available sectors have been tested with the previously described
procedure. Test results as well as geometrical parameters of the sensor are
listed in Table 1, where the pixel-wise quantities are defined in Section 3.1
and the cluster-wise quantities are defined in Section 3.3.

For all sectors, the CVFs and the ENCs of the pixels are consistent within
uncertainties. The means of pixel-by-pixel ENC distributions in each sector
are all below $14\text{\,}\mathrm{e}^{-}$, and have the RMS spreads of about
40%. Taking A0 as a reference, variations of the means among sectors are
within 10%, less than the uncertainties.

As for the pixel-wise performance of charge collection, the SNRs are
consistent within uncertainties, whereas the CCEs, and the means of SNRs
exhibit geometry dependence similarly. The sectors of A2 and A0 have almost
the same geometry configuration except that the x-pitch of A2 is four times
that of A0. The CCE of A2 decreases about 19% of that of A0, while the mean of
SNR decreases about 6% only. The sectors of A2, A5 and A8, consisting of the
largest pixels, have the same pixel pitches and different areas of the diode
surface and footprint. Among them, the CCE and SNR of A8 are the best, very
close to those of A0, whereas those of A5 are the worst. It implies that not
only the absolute size of the diode surface but also the surface to footprint
ratio influences the performance. The sectors of A7 and A8 have similar diode
dimensions, but the x-pitch of A7 is half of A8’s. In addition, the diodes of
A8 have a staggered arrangement, therefore the diode distribution density is
half of that of A7 evenly. That explains why A7, due to charge sharing, has
smaller CCE than A8 does. The observations confirm our previous simulation for
the sensor design, as shown in Figure 3 and 4 in ref. [7].

For the cluster-wise performance, all sectors perform well, having CCEs and
SNRs greater than 90% and 30, respectively. Again the SNRs are consistent
within uncertainties, and their means depend weakly on geometrical
configuration. On the other hand, the CCEs are almost geometry independent,
within about 5% variation referring to the CCE of A0, and A7 performs the
best.

Table 1: Geometrical parameters of the Supix-1sensor and test results with a ${}^{55}\text{{Fe}}$ radioactive source. CVF is for the charge-to-voltage factor, ENC is for the equivalent noise charge, CCE is for the charge collection efficiency and SNR is for the signal-to-noise ratio. The uncertainties of the CCEs are on the most probable values, while the uncertainties of the others are of the distribution. See the text for more explanations.  Sector | A0 | A2 | A5 | A7 | A8
---|---|---|---|---|---
Sensitive Area x (mm) | 0.3 | 1.3 | 1.3 | 0.7 | 1.3
Sensitive Area y (mm) | 1.3
Pixel Pitch x ($\mu$m) | 21 | 84 | 84 | 42 | 84
Pixel Pitch y ($\mu$m) | 21
Diode Surface ($\mu$m2) | 8 | 8 | 12 | 20 | 20
Diode Footprint ($\mu$m2) | 11 | 11 | 18 | 44 | 50
Pixel | CVF ($\mu\mathrm{V}\mathrm{/}\mathrm{e}$) | 13$\pm$5 | 14$\pm$5 | 12$\pm$4 | 14$\pm$4 | 13$\pm$5
ENC (e) | 13$\pm$5 | 12$\pm$5 | 13$\pm$5 | 12$\pm$4 | 12$\pm$5
CCE (%) | 39.1$\pm$1.0 | 31.7$\pm$0.7 | 27.5$\pm$0.6 | 30.4$\pm$0.8 | 37.2$\pm$0.7
SNR | 52$\pm$11 | 49$\pm$10 | 36$\pm$8 | 42$\pm$9 | 53$\pm$11
Cluster | CCE (%) | 95.6$\pm$0.5 | 90.3$\pm$0.4 | 93.8$\pm$0.5 | 96.8$\pm$0.4 | 92.4$\pm$0.6
SNR | 36$\pm$3 | 33$\pm$2 | 32$\pm$2 | 36$\pm$3 | 36$\pm$3

##  5 Conclusions

Using a radioactive source of ${}^{55}\text{{Fe}}$, we have characterized the
Supix-1pixel sensor implemented in a $180\text{\,}\mathrm{n}\mathrm{m}$ CMOS
imaging process. The sensor, consisting of sectors with different pixel
configurations, is aimed to investigate detection performance of the large
pixel, meeting requirements of a proposed pixelated silicon tracker at CEPC.
The full energy peaks corresponding to the k-$\alpha$ and k-$\beta$ X-rays of
${}^{55}\text{{Fe}}$ have been observed and are used to calibrate the pixel
gain and to obtain the pixel-wise CVF, ENC, CCE and SNR. A preliminary
algorithm to reconstruct the signal cluster of pixels has been developed and
the cluster-wise CCE and SNR are determined. During the study of
reconstruction, an anomaly in the pixel output is observed and an empirical
correction method has been developed.

With our chosen threshold for data taking, the fake-hit rate is in the order
of
${10}^{-8}\text{\,}\mathrm{p}\mathrm{i}\mathrm{x}\mathrm{e}\mathrm{l}^{-1}\,\mathrm{t}\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{g}\mathrm{e}\mathrm{r}^{-1}$.
The pixels of all available sectors have low noise, with all ENCs being about
$13\text{\,}\mathrm{e}^{-}$ and almost the same within the uncertainty of
about $5\text{\,}\mathrm{e}^{-}$. For different sectors, the pixel-wise CCEs
depend on the geometry configuration significantly, whereas the cluster-wise
CCEs do not and are greater than 90% for all. The SNRs of both pixel and
cluster in all sectors are high, mostly above 30, and are consistent within
uncertainties.

Our test results demonstrate that large pixels, like those in the sector of
A8, satisfy typical requirements of pixel detectors in the sense of noise and
charge collection efficiency. They are practical for tracking at CEPC, since
CVF and ENC do not constrain pixel performance at the large sizes. In
addition, the pixel performance of charge collection depends not only on the
pitch size but also on the diode design, including the areas of surface and
footprint as well as the arrangement of position. It would be potentially
possible to explore the design of even larger pixels, optimized towards
specific requirements of different detectors.

## Acknowledgments

This work has been supported by the National Natural Science Foundation of
China(U1232202, U2032203 and 12075142), the Ministry of Science and Technology
of China (2018YFA0404302) and Shandong Provincial Natural Science Foundation
(ZR2020MA102).

## References

  * [1] ATLAS collaboration, _Observation of a new particle in the search for the Standard Model Higgs boson with the ATLAS detector at the LHC_ , _Phys. Lett. B_ 716 (2012) 1 [arXiv:1207.7214].
  * [2] CMS collaboration, _Observation of a New Boson at a Mass of 125 GeV with the CMS Experiment at the LHC_ , _Phys. Lett. B_ 716 (2012) 30 [arXiv:1207.7235].
  * [3] The CEPC Study Group, “CEPC Conceptual Design Report: Volume 1 - Accelerator.” 2018, arXiv:1809.00285.
  * [4] The CEPC Study Group, “CEPC Conceptual Design Report: Volume 2 - Physics & Detector.” 2018, arXiv:1811.10545.
  * [5] G. Contin et al., _The STAR MAPS-based PiXeL detector_ , _Nucl. Instrum. Meth. A_ 907 (2018) 60 [arXiv:1710.02176].
  * [6] ALICE collaboration, _The ALPIDE pixel sensor chip for the upgrade of the ALICE Inner Tracking System_ , _Nucl. Instrum. Meth. A_ 845 (2017) 583.
  * [7] L. Zhang, M. Fu, Y. Zhang et al., _Investigation of CMOS pixel sensor with 0.18 $\mu$m CMOS technology for high-precision tracking detector_, _JINST_ 12 (2017) C01011.
  * [8] Particle Data Group collaboration, _Review of Particle Physics_ , _PTEP_ 2020 (2020) 083C01.
  * [9] G. Deptuch et al., _Design and testing of monolithic active pixel sensors for charged particle tracking_ , _IEEE Trans. Nucl. Sci._ 49 (2002) 601.


resize[2][!]  rescale[2][]

# Long-term Forecasting with
TiDE: Time-series Dense Encoder

Abhimanyu Das<EMAIL_ADDRESS>
Google Research Weihao Kong<EMAIL_ADDRESS>
Google Research Andrew Leach<EMAIL_ADDRESS>
Google Cloud Shaan Mathur<EMAIL_ADDRESS>
Google Cloud Rajat Sen<EMAIL_ADDRESS>
Google Research Rose Yu<EMAIL_ADDRESS>
University of California San Diego

###### Abstract

Recent work has shown that simple linear models can outperform several
Transformer based approaches in long term time-series forecasting. Motivated
by this, we propose a Multi-layer Perceptron (MLP) based encoder-decoder
model, Time-series Dense Encoder (TiDE), for long-term time-series forecasting
that enjoys the simplicity and speed of linear models while also being able to
handle covariates and non-linear dependencies. Theoretically, we prove that
the simplest linear analogue of our model can achieve near optimal error rate
for linear dynamical systems (LDS) under some assumptions. Empirically, we
show that our method can match or outperform prior approaches on popular long-
term time-series forecasting benchmarks while being 5-10x faster than the best
Transformer based model.††Author names are arranged in alphabetical order of
last names.

##  1 Introduction

Long-term forecasting, which is to predict several steps into the future given
a long context or look-back, is one of the most fundamental problems in time
series analysis, with broad applications in energy, finance, and
transportation. Deep learning models (Wu et al., 2021; Nie et al., 2022) have
emerged as a popular approach for forecasting rich, multivariate, time series
data, often outperforming classical statistical approaches such as ARIMA or
GARCH (Box et al., 2015). In several forecasting competitions such as the M5
competition (Makridakis et al., 2020) and IARAI Traffic4cast contest (Kreil et
al., 2020), deep neural networks performed quite well.

Various neural network architectures have been explored for forecasting,
ranging from recurrent neural networks to convolutional networks to graph
neural networks. For sequence modeling tasks in domains such as language,
speech and vision, Transformers (Vaswani et al., 2017) have emerged as the
most successful model, even outperforming recurrent neural networks
(LSTMs)(Hochreiter and Schmidhuber, 1997). Subsequently, there has been a
surge of Transformer-based forecasting papers (Wu et al., 2021; Zhou et al.,
2021; 2022) in the time-series community that have claimed state-of-the-art
(SoTA) forecasting performance for long-horizon tasks. However, recent work
(Zeng et al., 2023) has shown that these Transformerss-based architectures may
not be as powerful as one might expect for time series forecasting, and can be
easily outperformed by a simple linear model on forecasting benchmarks. Such a
linear model however has deficiencies since it is ill-suited for modeling non-
linear dependencies among the time-series sequence and the time-independent
covariates. Indeed, a very recent paper (Nie et al., 2022) proposed a new
Transformer-based architecture that obtains SoTA performance for deep neural
networks on the standard multivariate forecasting benchmarks.

In this paper, we present a simple and effective deep learning architecture
for long-term forecasting that obtains superior performance when compared to
existing SoTA neural network based models on the time series forecasting
benchmarks. Our Multi-Layer Perceptron (MLP)-based model is embarrassingly
simple without any self-attention, recurrent or convolutional mechanism.
Therefore, it enjoys a linear computational scaling in terms of the context
and horizon lengths unlike many Transformer-based solutions.

The main contributions of this work are as follows:

  * •

We propose the Time-series Dense Encoder (TiDE) model architecture for long-
term time series forecasting. TiDE encodes the past of a time-series along
with covariates using dense MLPs and then decodes time-series along with
future covariates, again using dense MLPs.

  * •

We analyze a simplified linear analogue of our model and prove that this
linear model can achieve near optimal error rate in linear dynamical systems
(LDS) (Kalman, 1963) when the design matrix of the LDS has maximum singular
value bounded away from $1$. We empirically verify this on a simulated dataset
where the linear model outperforms LSTMs and Transformers.

  * •

On popular real-world long-term forecasting benchmarks, our model achieves
better or similar performance compared to prior neural network based baselines
($>$10% lower Mean Squared Error on the largest dataset). At the same time,
TiDE is 5x faster in terms of inference and more than 10x faster in training
when compared to the best Transformer based model.

##  2 Background and Related Work

Models for long-term forecasting can be broadly divided into either
multivariate models or univariate models.

Multivariate models use the past of all the interrelated time-series variables
and predict the future of all the time-series as a joint function of those
pasts. This includes the classical VAR models (Zivot and Wang, 2006). We will
mostly focus on the prior work on neural network based models for long-term
forecasting. LongTrans (Li et al., 2019a) uses attention layer with LogSparse
design to capture local information with near linear space and computational
complexity. Informer (Zhou et al., 2021) uses the ProbSparse self-attention
mechanism to achieve sub-quadratic dependency on the length of the context.
Autoformer (Wu et al., 2021) uses trend and seasonal decomposition with sub-
quadratic self attention mechanism. FEDFormer (Zhou et al., 2022) uses a
frequency enchanced structure while Pyraformer (Liu et al., 2021) uses
pyramidal self-attention that has linear complexity and can attend to
different granularities. The common theme among the above works is the use of
sub-quadratic approximations for the full self attention mechanism, that have
been also been used in other domains (Wang et al., 2020).

On the other hand, univariate models predict the future of a time-series
variable as a function of only the past of the same time-series and covariate
features. In other words, the past of other time-series is not part of the
input during inference. There are two kinds of univariate models, local and
global. Local univariate models are usually trained per time-series variable
and inference is done per time-series as well. Different variables have
different models. Classical models like AR, ARIMA, exponential smoothing
models (McKenzie, 1984) and the Box-Jenkins methodology (Box and Jenkins,
1968) belong in this category. We would refer the reader to (Box et al., 2015)
for an in depth discussion of these methods.

Global univariate models ignore the variable information and train one shared
model for all the time series on the whole dataset. This category mainly
includes deep learning based architectures like (Salinas et al., 2020). In the
context of long-term forecasting, recently it was observed that a simple
linear global univariate model can outperform the transformer based
multivariate approaches for long-term forecasting (Zeng et al., 2023). DLinear
(Zeng et al., 2023) learns a linear mapping from context to horizon, pointing
to deficiencies in sub-quadratic approximations to the self-attention
mechanism. Indeed, a very recent model, PatchTST (Nie et al., 2022) has shown
that feeding contiguous patches of time-series as tokens to the vanilla self-
attention mechanism can beat the performance of DLinear in long-term
forecasting benchmarks. MLPs have been used for time-series forecasting in the
popular N-BEATS model (Oreshkin et al., ) and later extended in a follow up
work (Challu et al., 2023) that uses multi-rate sampling for better
efficiency. However, these methods do not explicitly mention supporting
covariates and fall short of PatchTST in long-horizon benchmarks.

Recent work has improved the efficacy of RNNs (Kag et al., 2020; Lukoševičius
and Uselis, 2022; Rusch and Mishra, 2020; Li et al., 2019b) and applied
parameter efficient SSMs (Gu et al., ; Gupta et al., 2022) to modeling long
range dependencies in sequences. They have demonstrated improvement over some
transformer based architectures on sequence modeling benchmarks including
speech and 1-D pixel level image classification tasks. We compare our method
to S4 model (Gu et al., ), which is the only such method that has been applied
to global univariate and global multivariate forecasting.

Note that all categories of models can be fairly compared on the task of
multivariate long-term forecasting if they are evaluated on the same test set
for the same task, which is the protocol that we follow in Section 5.

##  3 Problem Setting

Before we describe the problem setting we will need to setup some general
notation.

###  3.1 Notation

We will denote matrices by bold capital letters like
$\bm{{X}}\in\mathbb{R}^{N\times T}$. The slice notation $i:j$ denotes the set
$\\{i,i+1,\cdots j\\}$ and $[n]:=\\{1,2,\cdots,n\\}$. The individual rows and
columns are always treated as column vectors unless otherwise specified. We
can also use sets to select sub-matrices i.e
$\bm{{X}}[\mathcal{I},\mathcal{J}]$ denotes the sub-matrix with rows in
$\mathcal{I}$ and columns in $\mathcal{J}$. $\bm{{X}}[:,j]$ means selecting
the $j$-th column while $\bm{{X}}[i,:]$ means the $i$-th row. The notation
$[{\bm{v}};{\bm{u}}]$ will denote the concatenation of the two column vectors
and the same notation can be used for matrices along a dimension.

###  3.2 Multivariate Forecasting

In this section we first abstract out the core problem in long-term
multivariate forecasting. There are $N$ time-series in the dataset. The look-
back of the $i$-th time-series will be denoted by $\mathbf{y}^{(i)}_{1:L}$,
while the horizon is denoted by $\mathbf{y}^{(i)}_{L+1:L+H}$. The task of the
forecaster is to predict the horizon time-points given access to the look-
back.

In many forecasting scenarios, there might be dynamic and static covariates
that are known in advance. With slight abuse of notation, we will use
${\bm{x}}^{(i)}_{t}\in\mathbb{R}^{r}$ to denote the $r$-dimensional dynamic
covariates of time-series $i$ at time $t$. For instance, they can be global
covariates (common to all time-series) such as day of the week, holidays etc
or specific to a time-series for instance the discount of a particular product
on a particular day in a demand forecasting use case. We can also have static
attributes of a time-series denoted by ${\bm{a}}^{(i)}$ such as features of a
product in retail that do not change with time. In many applications, these
covariates are vital for accurate forecasting and a good model architecture
should have provisions to handle them.

The forecaster can be thought of as a function that maps the history
$\mathbf{y}^{(i)}_{1:L}$, the dynamic covariates ${\bm{x}}^{(i)}_{1:L+H}$ and
the static attributes ${\bm{a}}^{(i)}$ to an accurate prediction of the
future, i.e.,

$f:\mathopen{}\mathclose{{}\left(\mathopen{}\mathclose{{}\left\\{\mathbf{y}^{(i)}_{1:L}}\right\\}_{i=1}^{N},\mathopen{}\mathclose{{}\left\\{{\bm{x}}^{(i)}_{1:L+H}}\right\\}_{i=1}^{N},\mathopen{}\mathclose{{}\left\\{{\bm{a}}^{(i)}}\right\\}_{i=1}^{N}}\right)\longrightarrow\mathopen{}\mathclose{{}\left\\{\hat{\mathbf{y}}^{(i)}_{L+1:L+H}}\right\\}_{i=1}^{N}.$
(1)

The accuracy of the prediction will be measured by a metric that quantifies
their closeness to the actual values. For instance, if the metric is Mean
Squared Error (MSE), then the goodness of fit is measured by,

$\displaystyle\mathrm{MSE}\mathopen{}\mathclose{{}\left(\mathopen{}\mathclose{{}\left\\{\mathbf{y}^{(i)}_{L+1:L+H}}\right\\}_{i=1}^{N},\mathopen{}\mathclose{{}\left\\{\hat{\mathbf{y}}^{(i)}_{L+1:L+H}}\right\\}_{i=1}^{N}}\right)=\frac{1}{NH}\sum_{i=1}^{N}\mathopen{}\mathclose{{}\left\|{\mathbf{y}^{(i)}_{L+1:L+H}-\hat{\mathbf{y}}^{(i)}_{L+1:L+H}}}\right\|_{2}^{2}.$
(2)

##  4 Model

Recently, it has been observed that simple linear models (Zeng et al., 2023)
can outperform Transformers based models in several long-term forecasting
benchmarks. On the other hand, linear models will fall short when there are
inherent non-linearities in the dependence of the future on the past.
Furthermore, linear models would not be able to model the dependence of the
prediction on the covariates as evidenced by the fact that (Zeng et al., 2023)
do not use time-covariates as they hurt performance.

In this section, we introduce a simple and efficient MLP based architecture
for long-term time-series forecasting. In our model we add non-linearities in
the form of MLPs in a manner that can handle past data and covariates. The
model is dubbed TiDE (Time-series Dense Encoder) as it encodes the past of a
time-series along with covariates using dense MLP’s and then decodes the
encoded time-series along with future covariates.

Figure 1: Overview of TiDE architecture. The dynamic covariates per time-point
are mapped to a lower dimensional space using a feature projection step. Then
the encoder combines the look-back along with the projected covariates with
the static attributes to form an encoding. The decoder maps this encoding to a
vector per time-step in the horizon. Then a temporal decoder combines this
vector (per time-step) with the projected features of that time-step in the
horizon to form the final predictions. We also add a global linear residual
connection from the look-back to the horizon.

An overview of our architecture has been presented in Figure 1. Our model is
applied in a channel independent manner (the term was used in (Nie et al.,
2022)) i.e the input to the model is the past and covariates of one time-
series at a time
$\mathopen{}\mathclose{{}\left(\mathbf{y}^{(i)}_{1:L},{\bm{x}}^{(i)}_{1:L},{\bm{a}}^{(i)}}\right)$
and it maps it to the prediction of that time-series
$\hat{\mathbf{y}}^{(i)}_{L+1:L+H}$. Note that the weights of the model are
trained globally using the whole dataset. A key component in our model is the
MLP residual block towards the right of the figure.

Residual Block. We use the residual block as the basic layer in our
architecture. It is an MLP with one hidden layer with ReLU activation. It also
has a skip connection that is fully linear. We use dropout on the linear layer
that maps the hidden layer to the output and also use layer norm at the
output.

We separate the model into encoding and decoding sections. The encoding
section has a novel feature projection step followed by a dense MLP encoder.
The decoder section consists of a dense decoder followed by a novel temporal
decoder. Note that the dense encoder (green block with $n_{e}$ layers) and
decoder blocks (yellow block with $n_{d}$ layers) in Figure 1 can be merged
into a single block. For the sake of exposition we keep them separate as we
tune the hidden layer size in the two blocks separately. Also the last layer
of the decoder block is unique in the sense that its output dimension needs to
be $H\times p$ before the reshape operation.

###  4.1 Encoding

The task of the encoding step is to map the past and the covariates of a time-
series to a dense representation of the features. The encoding in our model
has two key steps.

Feature Projection. We use a residual block to map ${\bm{x}}^{(i)}_{t}$ at
each time-step (both in the look-back and the horizon) into a lower
dimensional projection of size $\tilde{r}\ll r$ (temporalWidth). This
operation can be described as,

$\displaystyle\tilde{{\bm{x}}}^{(i)}_{t}=\mathrm{ResidualBlock}\mathopen{}\mathclose{{}\left({\bm{x}}^{(i)}_{t}}\right).$
(3)

This is essentially a dimensionality reduction step since flattening the
dynamic covariates for the whole look-back and horizon would lead to an input
vector of size $(L+H)r$ which can be prohibitively large. On the other hand,
flattening the reduced features would only lead to a dimension of
$(L+H)\tilde{r}$.

Dense Encoder. As the input to the dense encoder, we stack and flatten all the
past and future projected covariates, concatenate them with the static
attributes and the past of the time-series. Then we map them to an embedding
using an encoder which contains multiple residual blocks. This can be written
as,

$\displaystyle{\bm{e}}^{(i)}=\mathrm{Encoder}\mathopen{}\mathclose{{}\left(\mathbf{y}^{(i)}_{1:L};\tilde{{\bm{x}}}^{(i)}_{1:L+H};{\bm{a}}^{(i)}}\right)$
(4)

The encoder internal layer sizes are all set to hiddenSize and the total
number of layers in the encoder is set to $n_{e}$ (numEncoderLayers).

###  4.2 Decoding

The decoding in our model maps the encoded hidden representations into future
predictions of time series. It also comprises of two operations, dense decoder
and temporal decoder.

Dense Decoder. The first decoding unit is a stacking of several residual
blocks like the encoder with the same hidden layer sizes. It takes as an input
the encoding ${\bm{e}}^{(i)}$ and maps it to a vector ${\bm{g}}^{(i)}$ of size
$H\times p$ where $p$ is the decoderOutputDim. This vector is then reshaped to
a matrix $\bm{{D}}^{(i)}\in\mathbb{R}^{d\times H}$. The $t$-th column i.e
${\bm{d}}^{(i)}_{t}$ can be thought of as the decoded vector for the $t$-th
time-period in the horizon for all $t\in[H]$. This whole operation can be
described as,

$\displaystyle{\bm{g}}^{(i)}$
$\displaystyle=\mathrm{Decoder}\mathopen{}\mathclose{{}\left({\bm{e}}^{(i)}}\right)~{}~{}~{}\in\mathbb{R}^{p.H}$
$\displaystyle\bm{{D}}^{(i)}$
$\displaystyle=\mathrm{Reshape}\mathopen{}\mathclose{{}\left({\bm{g}}^{(i)}}\right)~{}~{}~{}\in\mathbb{R}^{p\times
H}.$

The number of layers in the dense decoder is $n_{d}$ (numDecoderLayers).

Temporal Decoder. Finally, we use the temporal decoder to generate the final
predictions. The temporal decoder is just a residual block with output size
$1$ that maps the decoded vector ${\bm{d}}^{(i)}_{t}$ at $t$-th horizon time-
step concatenated with the projected covariates $\tilde{{\bm{x}}}^{(i)}_{L+t}$
i.e

$\displaystyle\hat{\mathbf{y}}^{(i)}_{L+t}=\mathrm{TemporalDecoder}\mathopen{}\mathclose{{}\left({\bm{d}}^{(i)}_{t};\tilde{{\bm{x}}}^{(i)}_{L+t}}\right)~{}~{}~{}\forall
t\in[H].$

This operation adds a "highway" from the future covariates at time-step $L+t$
to the prediction at time-step $L+t$. This can be useful if some covariates
have a strong direct effect on a particular time-step’s actual value. For
instance, in retail demand forecasting a holiday like Mother’s day might
strongly affect the sales of certain gift items. Such signals can be lost or
take longer for the model to learn in absence of such a highway. We denote the
hyperparameter controlling the hidden size of the temporal decoder as
temporalDecoderHidden.

Finally, we add a global residual connection that linearly maps the look-back
$\mathbf{y}^{(i)}_{1:L}$ to a vector the size of the horizon which is added to
the prediction $\hat{\mathbf{y}}^{(i)}_{L+1:L+H}$. This ensures that a purely
linear model like the one in (Zeng et al., 2023) is always a subclass of our
model.

Training and Evaluation. The model is trained using mini-batch gradient
descent where each batch consists of a batchSize number of time-series and the
corresponding look-back and horizon time-points. We use MSE as the training
loss. Each epoch consists of all look-back and horizon pairs that can be
constructed from the training period i.e. two mini-batches can have
overlapping time-points. This was standard practice in all prior work on long-
term forecasting (Zeng et al., 2023; Liu et al., 2021; Wu et al., 2021; Li et
al., 2019a).

The model is evaluated on a test set on every (look-back, horizon) pair that
can be constructed from the test set. This is usually known as rolling
validation/evaluation. A similar evaluation on a validation set can be used
optionally used to tune parameters for model selection.

##  5 Experimental Results

In this section we present our main experimental results on popular long-term
forecasting benchmarks. We also perform an ablation study that shows the
usefulness of the temporal decoder.

###  5.1 Long-Term Time-Series Forecasting

Datasets. We use seven commonly used long-term forecasting benchmark datasets:
Weather, Traffic, Electricity and 4 ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2).
We refer the reader to (Wu et al., 2021) for a detailed discussion on the
datasets. In Table 1 we provide some statistics about the datasets. Note that
Traffic and Electricity are the largest datasets with $>800$ and $>300$ time-
series each having tens of thousands of time-points. Since we are only
interested in long-term forecasting results in this section, we omit the
shorter horizon ILI dataset.

Dataset | #Time-Series | #Time-Points | Frequency
---|---|---|---
Electricity | 321 | 26304 | 1 Hour
Traffic | 862 | 17544 | 1 Hour
Weather | 21 | 52696 | 10 Minutes
ETTh1 | 7 | 17420 | 1 Hour
ETTh2 | 7 | 17420 | 1 Hour
ETTm1 | 7 | 69680 | 15 Minutes
ETTm2 | 7 | 69680 | 15 Minutes
Table 1: Summary of datasets.

Models | TiDE | PatchTST/64 | N-HiTS | DLinear | FEDformer | Autoformer | Informer | Pyraformer | LogTrans |
---|---|---|---|---|---|---|---|---|---|---
Metric | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE |
Weather | 96 | 0.166 | 0.222 | 0.149 | 0.198 | 0.158 | 0.195 | 0.176 | 0.237 | 0.238 | 0.314 | 0.249 | 0.329 | 0.354 | 0.405 | 0.896 | 0.556 | 0.458 | 0.490 |
192 | 0.209 | 0.263 | 0.194 | 0.241 | 0.211 | 0.247 | 0.220 | 0.282 | 0.275 | 0.329 | 0.325 | 0.370 | 0.419 | 0.434 | 0.622 | 0.624 | 0.658 | 0.589 |
336 | 0.254 | 0.301 | 0.245 | 0.282 | 0.274 | 0.300 | 0.265 | 0.319 | 0.339 | 0.377 | 0.351 | 0.391 | 0.583 | 0.543 | 0.739 | 0.753 | 0.797 | 0.652 |
720 | 0.313 | 0.340 | 0.314 | 0.334 | 0.401 | 0.413 | 0.323 | 0.362 | 0.389 | 0.409 | 0.415 | 0.426 | 0.916 | 0.705 | 1.004 | 0.934 | 0.869 | 0.675 |
Traffic | 96 | 0.336 | 0.253 | 0.360 | 0.249 | 0.402 | 0.282 | 0.410 | 0.282 | 0.576 | 0.359 | 0.597 | 0.371 | 0.733 | 0.410 | 2.085 | 0.468 | 0.684 | 0.384 |
192 | 0.346 | 0.257 | 0.379 | 0.256 | 0.420 | 0.297 | 0.423 | 0.287 | 0.610 | 0.380 | 0.607 | 0.382 | 0.777 | 0.435 | 0.867 | 0.467 | 0.685 | 0.390 |
336 | 0.355 | 0.260 | 0.392 | 0.264 | 0.448 | 0.313 | 0.436 | 0.296 | 0.608 | 0.375 | 0.623 | 0.387 | 0.776 | 0.434 | 0.869 | 0.469 | 0.734 | 0.408 |
720 | 0.386 | 0.273 | 0.432 | 0.286 | 0.539 | 0.353 | 0.466 | 0.315 | 0.621 | 0.375 | 0.639 | 0.395 | 0.827 | 0.466 | 0.881 | 0.473 | 0.717 | 0.396 |
Electricity | 96 | 0.132 | 0.229 | 0.129 | 0.222 | 0.147 | 0.249 | 0.140 | 0.237 | 0.186 | 0.302 | 0.196 | 0.313 | 0.304 | 0.393 | 0.386 | 0.449 | 0.258 | 0.357 |
192 | 0.147 | 0.243 | 0.147 | 0.240 | 0.167 | 0.269 | 0.153 | 0.249 | 0.197 | 0.311 | 0.211 | 0.324 | 0.327 | 0.417 | 0.386 | 0.443 | 0.266 | 0.368 |
336 | 0.161 | 0.261 | 0.163 | 0.259 | 0.186 | 0.290 | 0.169 | 0.267 | 0.213 | 0.328 | 0.214 | 0.327 | 0.333 | 0.422 | 0.378 | 0.443 | 0.280 | 0.380 |
720 | 0.196 | 0.294 | 0.197 | 0.290 | 0.243 | 0.340 | 0.203 | 0.301 | 0.233 | 0.344 | 0.236 | 0.342 | 0.351 | 0.427 | 0.376 | 0.445 | 0.283 | 0.376 |
ETTh1 | 96 | 0.375 | 0.398 | 0.379 | 0.401 | 0.378 | 0.393 | 0.375 | 0.399 | 0.376 | 0.415 | 0.435 | 0.446 | 0.941 | 0.769 | 0.664 | 0.612 | 0.878 | 0.740 |
192 | 0.412 | 0.422 | 0.413 | 0.429 | 0.427 | 0.436 | 0.412 | 0.420 | 0.423 | 0.446 | 0.456 | 0.457 | 1.007 | 0.786 | 0.790 | 0.681 | 1.037 | 0.824 |
336 | 0.435 | 0.433 | 0.435 | 0.436 | 0.458 | 0.484 | 0.439 | 0.443 | 0.444 | 0.462 | 0.486 | 0.487 | 1.038 | 0.784 | 0.891 | 0.738 | 1.238 | 0.932 |
720 | 0.454 | 0.465 | 0.446 | 0.464 | 0.472 | 0.561 | 0.501 | 0.490 | 0.469 | 0.492 | 0.515 | 0.517 | 1.144 | 0.857 | 0.963 | 0.782 | 1.135 | 0.852 |
ETTh2 | 96 | 0.270 | 0.336 | 0.274 | 0.337 | 0.274 | 0.345 | 0.289 | 0.353 | 0.332 | 0.374 | 0.332 | 0.368 | 1.549 | 0.952 | 0.645 | 0.597 | 2.116 | 1.197 |
192 | 0.332 | 0.380 | 0.338 | 0.376 | 0.353 | 0.401 | 0.383 | 0.418 | 0.407 | 0.446 | 0.426 | 0.434 | 3.792 | 1.542 | 0.788 | 0.683 | 4.315 | 1.635 |
336 | 0.360 | 0.407 | 0.363 | 0.397 | 0.382 | 0.425 | 0.448 | 0.465 | 0.400 | 0.447 | 0.477 | 0.479 | 4.215 | 1.642 | 0.907 | 0.747 | 1.124 | 1.604 |
720 | 0.419 | 0.451 | 0.393 | 0.430 | 0.625 | 0.557 | 0.605 | 0.551 | 0.412 | 0.469 | 0.453 | 0.490 | 3.656 | 1.619 | 0.963 | 0.783 | 3.188 | 1.540 |
ETTm1 | 96 | 0.306 | 0.349 | 0.293 | 0.346 | 0.302 | 0.350 | 0.299 | 0.343 | 0.326 | 0.390 | 0.510 | 0.492 | 0.626 | 0.560 | 0.543 | 0.510 | 0.600 | 0.546 |
192 | 0.335 | 0.366 | 0.333 | 0.370 | 0.347 | 0.383 | 0.335 | 0.365 | 0.365 | 0.415 | 0.514 | 0.495 | 0.725 | 0.619 | 0.557 | 0.537 | 0.837 | 0.700 |
336 | 0.364 | 0.384 | 0.369 | 0.392 | 0.369 | 0.402 | 0.369 | 0.386 | 0.392 | 0.425 | 0.510 | 0.492 | 1.005 | 0.741 | 0.754 | 0.655 | 1.124 | 0.832 |
720 | 0.413 | 0.413 | 0.416 | 0.420 | 0.431 | 0.441 | 0.425 | 0.421 | 0.446 | 0.458 | 0.527 | 0.493 | 1.133 | 0.845 | 0.908 | 0.724 | 1.153 | 0.820 |
ETTm2 | 96 | 0.161 | 0.251 | 0.166 | 0.256 | 0.176 | 0.255 | 0.167 | 0.260 | 0.180 | 0.271 | 0.205 | 0.293 | 0.355 | 0.462 | 0.435 | 0.507 | 0.768 | 0.642 |
192 | 0.215 | 0.289 | 0.223 | 0.296 | 0.245 | 0.305 | 0.224 | 0.303 | 0.252 | 0.318 | 0.278 | 0.336 | 0.595 | 0.586 | 0.730 | 0.673 | 0.989 | 0.757 |
336 | 0.267 | 0.326 | 0.274 | 0.329 | 0.295 | 0.346 | 0.281 | 0.342 | 0.324 | 0.364 | 0.343 | 0.379 | 1.270 | 0.871 | 1.201 | 0.845 | 1.334 | 0.872 |
720 | 0.352 | 0.383 | 0.362 | 0.385 | 0.401 | 0.413 | 0.397 | 0.421 | 0.410 | 0.420 | 0.414 | 0.419 | 3.001 | 1.267 | 3.625 | 1.451 | 3.048 | 1.328 |

Table 2: Multivariate long-term forecasting results with our model.
$T\in\\{96,192,336,720\\}$ for all datasets. The best results including the
ones that cannot be statistically distinguished from the best mean numbers are
in bold. We calculate standard error intervals for our method over 5 runs. The
rest of the numbers are taken from the results from (Nie et al., 2022)222Note
that there was a bug in the original test dataloader that affected the result
significantly in smaller datasets like ETTh1 and ETTh2. We report the PatchTST
results after correcting this bug in the dataloader.. All metrics are reported
on standard normalized datasets. We provide the standard errors for our method
in Table 5 in Appendix B.

Baselines and Setup. We choose SOTA Transformers based models for time-series
including Fedformer (Zhou et al., 2022), Autoformer (Wu et al., 2021),
Informer (Zhou et al., 2021), Pyraformer (Liu et al., 2021) and LongTrans (Li
et al., 2019a). Recently, DLinear (Zeng et al., 2023) showed that simple
linear models can outperform the above methods and therefore DLinear serves as
an important baseline. We include N-HiTS (Challu et al., 2023) which is an
improvement over the famous NBeats (Oreshkin et al., ) model. Finally we
compare with PatchTST (Nie et al., 2022) where they showed that vanilla
Transformers applied to time-series patches can be very effective. The results
for all Transformer based baselines are reported from (Nie et al., 2022).

For each method, the look-back window was tuned in
$\\{24,48,96,192,336,720\\}$. We report the DLinear numbers directly from the
original paper (Zeng et al., 2023). For our method we always use context
length of $720$ for all horizon lengths in $\\{96,192,336,720\\}$. All models
were trained using MSE as the training loss. In all the datasets, the
train:validation:test ratio is 7:1:2 as dictated by prior work. Note that all
the experiments are performed on standard normalized datasets (using the mean
and the standard deviations in the training period) in order to be consitent
with prior work (Wu et al., 2021).

Our Model. We use the architecture described in Figure 1. We tune our hyper-
parameters using the validation set rolling validation error. We provide
details about our hyper-parameters in Appendix B.3. As global dynamic
covariates, we use simple time-derived features like minute of the hour, hour
of the day, day of the week etc which are normalized similar to (Alexandrov et
al., 2020). Note that these features were turned off in DLinear since it was
observed to hurt the performance of the linear model, however our model can
easily handle such features. Our model is trained in Tensorflow (Abadi, 2016)
and we optimize using the default settings of the Adam optimizer (Kingma and
Ba, 2014). We provide our implementation in the supplementary with scripts to
reproduce the results in Table 2.

Results.  We present Mean Squared Error (MSE) and Mean Absolute Error (MSE)
for all datasets and methods in Table 2. For our model we report the mean
metric out of 5 independent runs for each setting. The bold-faced numbers are
from the best model or within statistical significance of the best model in
terms of two standard error intervals. Note that different predictive
statistics are optimal for different target metrics (Awasthi et al., 2021;
Gneiting, 2011) and therefore we should look at a target metric that is
closely aligned with the training loss in this case. Since all models were
trained using MSE let us focus on that column for comparisons.

We can see that TiDE, PatchTST, N-HiTS and DLinear are much better than the
other baselines in all datasets. This can be attributed to the fact that sub-
quadratic approximations to the full self-attention mechanism is perhaps not
best suited for long term forecasting. The same was observed in the PatchTST
(Nie et al., 2022) where it was shown that full self attention over patches
was much more effective even when applied in a channel dependent manner. For a
more in depth discussion about the pitfalls of sub-quadratic attention
approximation in the context of forecasting, we refer the reader to Section 3
of (Zeng et al., 2023). In Appendix A, we prove that a linear analogue of our
model can be optimal for predicting linear dynamical systems when compared
against sequence models, thus shedding some light on why our model and even
simpler models like DLinear can be so competitive for long context and/or
horizon forecasting.

Further, we outperform DLinear significantly in all settings except for
horizon 192 in ETTh1 where the performances are equal. This shows the value of
the additional non-linearity in our model. In some datasets like Weather and
ETTh1, N-HiTS performs similar to TiDE and PatchTST for horizon 96, but fails
to uphold the performance for longer horizons. In all datasets except Weather,
we either outperform PatchTST or perform within its statistical significance
for most horizons. In the Weather dataset, PatchTST peforms the best for
horizons 96-336 while our model is the most performant for horizon 720. In the
biggest dataset (Traffic), we significantly outperform PatchTST in all
settings. For instance, for horizon 720 our prediction is 10.6% better than
PatchTST in MSE. We provide additional results in Appendix 5 including a
comparison with the S4 model (Gu et al., ) in Table 6.

###  5.2 Demand Forecasting

In order to showcase our model’s ability to handle static attrubutes and
complex dynamic covariates we use the M5 forecasting competition benchmarks
(Makridakis et al., 2022). We follow the convention in the example
notebook333https://github.com/awslabs/gluonts/blob/dev/examples/m5_gluonts_template.ipynb
released by the authors of (Alexandrov et al., 2020). The dataset consists of
more than 30k time-series with static attributes like hierarchical categories
and dynamic covariates like promotions. More details of the setup are
available in Appendix 5.

We present the competition metric (WRMSSE) results on the test set
corresponding to the private leader-board in Table 3. We compare with DeepAR
(Salinas et al., 2020) whose implementation can handle all covariates and also
PatchTST (the best model from Table 2). Note that the implementation of
PatchTST (Nie et al., 2022) does not handle covariates. We report the score
over 3 independent runs along with the corresponding standard errors. We can
see that PatchTST performs poorly as it does not use covariates. Our model
using all the covariates outperforms DeepAR (that also uses all the
covariates) by as much as 20%. For the sake of ablation, we also provide the
metric for our model that uses only date derived features as covariates. There
is a degradation in performance from not using the dataset specific covariates
but even so this version of the model also outperforms the other baselines.

Model | Covariates | Test WRMSSE
---|---|---
TiDE | Static + Dynamic | $\mathbf{0.611\pm 0.009}$
TiDE | Date only | $0.637\pm 0.005$
DeepAR | Static + Dynamic | $0.789\pm 0.025$
PatchTST | None | $0.976\pm 0.014$
Table 3: M5 forecasting results on the private test set. We report the
competition metric (averaged across three runs) for each model. We also list
the covariates used by all models.

###  5.3 Training and Inference Efficiency

(a) Inference time per batch in microseconds

(b) Training time for one epoch in seconds.

Figure 2: In (a) we show the inference time per batch on the electricity
dataset. In (b) we show the corresponding training times for one epoch. In
both the figures the y-axis is plotted in log-scale. Note that the PatchTST
model ran out of GPU memory for look-back $L\geq 1440$.

In the previous section we have seen that TiDE outperforms all methods except
PatchTST by a large margin while it performs better or comparable to PatchTST
in all datasets except Weather. Next, we would like to demonstrate that TiDE
is much more efficient than PatchTST in terms of both training and inference
times.

Firstly, we would like to note that inference scales as
$\tilde{O}(n_{e}h^{2}+hL)$ for the encoder in TiDE, where $n_{e}$ is the
number of layers in the encoder, $h$ is the hidden size of the internal layers
and $L$ is the look-back. On the other hand, inference in PatchTST encoder
would scale as $\tilde{O}(Kn_{a}L^{2}/P^{2})$, where $K$ is the size of the
key in self-attention, $P$ is the patch-size and $n_{a}$ is the number of
attention layers. The quadratic scaling in $L$ can be prohibitive for very
long contexts. Also, the amount of memory required is quadratic in $L$ for the
vanilla Transformer architecture used in PatchTST 444Note that sub-quadratic
memory attention mechanism does exist (Dao et al., 2022) but has not been used
in PatchTST. The quadratic computation seems to be unavoidable since
approximations like Pyraformer seem to perform much worse than PatchTST..

We demonstrate these effects in practice in Figure 2. For the comparison to be
fair we carry out the experiment using the data loader in the Autoformer (Wu
et al., 2021) code base that was used in all subsequent papers. We use the
electricity dataset with batch size $8$, that is each batch has a shape of
$8\times 321\times L$ because the electricity dataset has $321$ time-series.
We report the inference time for one batch and the training time for one epoch
for TiDE and PatchTST as the look-back ($L$) is increased from $192$ to
$2880$. We can see that there is an order of magnitude difference in inference
time. The differences in training time is even more stark with PatchTST being
much more sensitive to the look-back size. Further PatchTST runs out of memory
for $L\geq 1440$. Thus our model achieves better or similar accuracy while
being much more computation and memory efficient. All the experiments in this
section were performed using a single NVIDIA T4 GPU on the same machine with
64 core Intel(R) Xeon(R) CPU @ 2.30GHz.

###  5.4 Ablation Study

#### Temporal Decoder.

The use of the temporal decoder for adaptation to future covariates is perhaps
one of the most interesting components of our model. Therefore in this section
we would like to show the usefulness of that component with a semi-synthetic
example using the electricity dataset.

Figure 3: We plot the actuals vs the predictions from TiDE with and without
the temporal decoder after just one epoch of training on the modified
electricity dataset. The red part of the horizontal line indicates an event of
Type A occuring.

We derive a new dataset from the electricity dataset, where we add numerical
features for two kinds of events. When an event of Type A occurs the value of
a time-series is increased by a factor which is uniformly chosen between
$[3,3.2]$. When an event of Type B occurs the value of a time-series is
decreased by a factor which is uniformly chosen between $[2,2.2]$. Only 80% of
the time-series are affected by these events and the time-series id’s that
fall in this bracket are chosen randomly. There are 4 numerical covariates
that indicate Type A and 4 that indicate Type B. When Type A event occurs the
Type A covariates are drawn from an isotropic Gaussian with mean
$[1.0,2.0,2.0,1.0]$ and variance $0.1$ for every coordinate. On the other hand
in the absence of Type A events Type A covariates are drawn from an isotropic
Gaussian with mean $[0.0,0.0,0.0,0.0]$. Thus these covariates serve as noisy
indicators of the event. We follow a similar pattern for Type B events and
covariates but with different means. Whenever these events occur they occur
for 24 contiguous hours.

In order to showcase that the use of the temporal decoder can learn such
patterns derived from the covariates faster, we plot the predictions from the
TiDE model with and without the temporal decoder after just one epoch of
training on the modified electricity dataset in Figure 3. The red part of the
horizontal line indicates the occurrence of Type A events. We can see that the
use of temporal decoder has a slight advantage during that time-period. But
more importantly, in the time instances following the event the model without
the temporal decoder is thrown off possibly because it has not yet readjusted
its past to what it should have been without the event. This effect is
negligible in the model which uses the temporal decoder, even after just one
epoch of training.

Figure 4: We plot the Test MSE on the traffic dataset as a function of
different context sizes for three different horizon length tasks. Each plot is
an average of 5 runs with the 2 standard error interval plotted.

Models | TiDE | TiDE (no res.)
---|---|---
Electricity | 96 | $\mathbf{0.132\pm 0.003}$ | $\mathbf{0.229\pm 0.001}$ | $0.136\pm 0.001$ | $0.235\pm 0.002$
192 | $\mathbf{0.147\pm 0.003}$ | $\mathbf{0.243\pm 0.001}$ | $0.153\pm 0.001$ | $0.253\pm 0.001$
336 | $\mathbf{0.161\pm 0.001}$ | $\mathbf{0.261\pm 0.002}$ | $0.172\pm 0.003$ | $0.274\pm 0.002$
720 | $0.196\pm 0.002$ | $0.294\pm 0.001$ | $0.196\pm 0.003$ | $0.295\pm 0.002$

Figure 5: We perform an ablation study by presenting results from our model
without any residual connections, on the electricity benchmark. We average
over 5 runs for all the numbers and present the corresponding standard errors.

#### Context Size.

In Figure 5 we study the dependence of prediction accuracy with context size
on the Traffic dataset. We plot the results for multiple horizon length tasks
and show that in all cases our methods performance becomes better with
increasing context size as expected. This is contrast to some of the
transformer based methods like Fedformer, Informer as shown in (Zeng et al.,
2023).

#### Residual Connections.

In Table 5 we perform an ablation study of the residual connections on the
electricity dataset. In the model dubbed TiDE (no res) we remove all the
residual connections including the ones in the residual block as well as the
global linear residual connection. In horizons 96-336 we see a statistically
significant drop in performance without the residual connections.

##  6 Conclusion

We propose a simple MLP based encoder decoder model that matches or supersedes
the performance of prior neural network baselines on popular long-term
forecasting benchmarks. At the same time, our model is 5-10x faster than the
best Transformer based baselines. Our study shows that self attention might
not be necessary to learn the periodicity and trend patterns at least for
these long-term forecasting benchmarks.

Our theoretical analysis partly explains why this could be the case by proving
that linear models can achieve near optimal rate when the ground truth is
generated from a linear dynamical system. However, for future work it would be
interesting to rigorously analyze MLPs and Transformer (including non-
linearity) under some simple mathematical model for time-series data and
potentially quantify the (dis)advantages of these architectures for different
levels of seasonality and trends. Also, note that transformers are generally
more parameter efficient than MLPs while being much more memory and compute
intensive. This could be a limitation while training extremely large scale
pre-trained models but the benefits of that line of work are not immediately
clear in time-series forecasting and is beyond the scope of this work.

## References

  * Abadi [2016] Martín Abadi.  Tensorflow: learning functions at scale.  In _Proceedings of the 21st ACM SIGPLAN International Conference on Functional Programming_ , pages 1–1, 2016.
  * Alexandrov et al. [2020] Alexander Alexandrov, Konstantinos Benidis, Michael Bohlke-Schneider, Valentin Flunkert, Jan Gasthaus, Tim Januschowski, Danielle C Maddix, Syama Rangapuram, David Salinas, Jasper Schulz, et al.  Gluonts: Probabilistic and neural time series modeling in python.  _The Journal of Machine Learning Research_ , 21(1):4629–4634, 2020.
  * Awasthi et al. [2021] Pranjal Awasthi, Abhimanyu Das, Rajat Sen, and Ananda Theertha Suresh.  On the benefits of maximum likelihood estimation for regression and forecasting.  In _International Conference on Learning Representations_ , 2021.
  * Bartlett and Mendelson [2002] Peter L Bartlett and Shahar Mendelson.  Rademacher and gaussian complexities: Risk bounds and structural results.  _Journal of Machine Learning Research_ , 3(Nov):463–482, 2002.
  * Box and Jenkins [1968] George EP Box and Gwilym M Jenkins.  Some recent advances in forecasting and control.  _Journal of the Royal Statistical Society. Series C (Applied Statistics)_ , 17(2):91–109, 1968.
  * Box et al. [2015] George EP Box, Gwilym M Jenkins, Gregory C Reinsel, and Greta M Ljung.  _Time series analysis: forecasting and control_.  John Wiley & Sons, 2015.
  * Challu et al. [2023] Cristian Challu, Kin G. Olivares, Boris N. Oreshkin, Federico Garza, Max Mergenthaler, and Artur Dubrawski.  NHITS: Neural Hierarchical Interpolation for Time Series forecasting.  In _The Association for the Advancement of Artificial Intelligence Conference 2023 (AAAI 2023)_ , 2023.  URL https://arxiv.org/abs/2201.12886.
  * Dao et al. [2022] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré.  Flashattention: Fast and memory-efficient exact attention with io-awareness.  _Advances in Neural Information Processing Systems_ , 35:16344–16359, 2022.
  * Gneiting [2011] Tilmann Gneiting.  Making and evaluating point forecasts.  _Journal of the American Statistical Association_ , 106(494):746–762, 2011.
  * [10] Albert Gu, Karan Goel, and Christopher Re.  Efficiently modeling long sequences with structured state spaces.  In _International Conference on Learning Representations_.
  * Gupta et al. [2022] Ankit Gupta, Albert Gu, and Jonathan Berant.  Diagonal state spaces are as effective as structured state spaces.  _Advances in Neural Information Processing Systems_ , 35:22982–22994, 2022.
  * Hazan et al. [2017] Elad Hazan, Karan Singh, and Cyril Zhang.  Learning linear dynamical systems via spectral filtering.  _Advances in Neural Information Processing Systems_ , 30, 2017.
  * Hochreiter and Schmidhuber [1997] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * Kag et al. [2020] Anil Kag, Ziming Zhang, and Venkatesh Saligrama.  Rnns incrementally evolving on an equilibrium manifold: A panacea for vanishing and exploding gradients?  In _International Conference on Learning Representations_ , 2020.
  * Kalman [1963] Rudolf Emil Kalman.  Mathematical description of linear dynamical systems.  _Journal of the Society for Industrial and Applied Mathematics, Series A: Control_ , 1(2):152–192, 1963.
  * Kim et al. [2021] Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo.  Reversible instance normalization for accurate time-series forecasting against distribution shift.  In _International Conference on Learning Representations_ , 2021.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kreil et al. [2020] David P Kreil, Michael K Kopp, David Jonietz, Moritz Neun, Aleksandra Gruca, Pedro Herruzo, Henry Martin, Ali Soleymani, and Sepp Hochreiter.  The surprising efficiency of framing geo-spatial time series forecasting as a video prediction task–insights from the iarai traffic4cast competition at neurips 2019.  In _NeurIPS 2019 Competition and Demonstration Track_ , pages 232–241. PMLR, 2020.
  * Li et al. [2019a] Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xiang Wang, and Xifeng Yan.  Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting.  _Advances in neural information processing systems_ , 32, 2019a.
  * Li et al. [2019b] Shuai Li, Wanqing Li, Chris Cook, and Yanbo Gao.  Deep independently recurrent neural network (indrnn).  _arXiv preprint arXiv:1910.06251_ , 2019b.
  * Liu et al. [2021] Shizhan Liu, Hang Yu, Cong Liao, Jianguo Li, Weiyao Lin, Alex X Liu, and Schahram Dustdar.  Pyraformer: Low-complexity pyramidal attention for long-range time series modeling and forecasting.  In _International conference on learning representations_ , 2021.
  * Lukoševičius and Uselis [2022] Mantas Lukoševičius and Arnas Uselis.  Time-adaptive recurrent neural networks.  _arXiv preprint arXiv:2204.05192_ , 2022.
  * Makridakis et al. [2020] S Makridakis, E Spiliotis, and V Assimakopoulos.  The m5 accuracy competition: Results, findings and conclusions.  _Int J Forecast_ , 2020.
  * Makridakis et al. [2022] Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos.  M5 accuracy competition: Results, findings, and conclusions.  _International Journal of Forecasting_ , 38(4):1346–1364, 2022.
  * McKenzie [1984] ED McKenzie.  General exponential smoothing and the equivalent arma process.  _Journal of Forecasting_ , 3(3):333–344, 1984.
  * Nie et al. [2022] Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam.  A time series is worth 64 words: Long-term forecasting with transformers.  _International conference on learning representations_ , 2022.
  * [27] Boris N Oreshkin, Dmitri Carpov, Nicolas Chapados, and Yoshua Bengio.  N-beats: Neural basis expansion analysis for interpretable time series forecasting.  In _International Conference on Learning Representations_.
  * Rusch and Mishra [2020] T Konstantin Rusch and Siddhartha Mishra.  Coupled oscillatory recurrent neural network (cornn): An accurate and (gradient) stable architecture for learning long time dependencies.  _arXiv preprint arXiv:2010.00951_ , 2020.
  * Salinas et al. [2020] David Salinas, Valentin Flunkert, Jan Gasthaus, and Tim Januschowski.  Deepar: Probabilistic forecasting with autoregressive recurrent networks.  _International Journal of Forecasting_ , 36(3):1181–1191, 2020.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. [2020] Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma.  Linformer: Self-attention with linear complexity.  _arXiv preprint arXiv:2006.04768_ , 2020.
  * Wu et al. [2021] Haixu Wu, Jiehui Xu, Jianmin Wang, and Mingsheng Long.  Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.  _Advances in Neural Information Processing Systems_ , 34:22419–22430, 2021.
  * Zeng et al. [2023] Ailing Zeng, Muxi Chen, Lei Zhang, and Qiang Xu.  Are transformers effective for time series forecasting?  _Proceedings of the AAAI conference on artificial intelligence_ , 2023.
  * Zhou et al. [2021] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  In _Proceedings of the AAAI conference on artificial intelligence_ , 2021.
  * Zhou et al. [2022] Tian Zhou, Ziqing Ma, Qingsong Wen, Xue Wang, Liang Sun, and Rong Jin.  Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting.  In _International Conference on Machine Learning_ , pages 27268–27286. PMLR, 2022.
  * Zivot and Wang [2006] Eric Zivot and Jiahui Wang.  Vector autoregressive models for multivariate time series.  _Modeling financial time series with S-PLUS®_ , pages 385–429, 2006.

##  Appendix A Theoretical Analysis under Linear Dynamical Systems

To gain insights into our design, we will now analyze the simplest linear
analogue of our model. In our model if all the residual connections are active
and the size of the encoding is greater than or equal to the length of the
horizon, then it reduces to a linear map from the context and the covariates
to the horizon. We study this version for the case when the data is generated
from a Linear Dynamical System (LDS), that has been a popular mathematical
model for systems evolving with time [Kalman, 1963]. We will prove that under
some conditions, a linear model that maps the past and the covariates of a
finite context to the future can be optimal for prediction in a LDS.

###  A.1 Theoretical Results

We formally define a linear dynamical system (LDS) as follows,

######  Definition A.1.

A linear dynamical system (LDS) is a map from a sequence of input vectors
$x_{1},\ldots,x_{T}\in\mathbb{R}^{n}$ to output (response) vectors
$y_{1},\ldots,y_{T}\in\mathbb{R}^{m}$ of the form

$\displaystyle h_{t+1}$ $\displaystyle=Ah_{t}+Bx_{t}+\eta_{t}$ (5)
$\displaystyle y_{t}$ $\displaystyle=Ch_{t}+Dx_{t}+\xi_{t},$ (6)

where $h_{0},\ldots,h_{T}\in\mathbb{R}^{d}$ is a sequence of hidden states,
$A,B,C,D$ are matrices of appropriate dimension, and
$\eta_{t}\in\mathbb{R}^{d},\xi_{t}\in\mathbb{R}^{m}$ are (possibly stochastic)
noise vectors. The $x_{t}$’s can be thought of as covariates for the time-
series $y_{t}$.

Given an LDS with parameter $\Theta=(A,B,C,D,h_{0}=0)$, we define the LDS
predictor as follows,

######  Definition A.2 (LDS predictor).

$\displaystyle\hat{y}_{t}$
$\displaystyle=y_{t-1}+(CB+D)x_{t}-Dx_{t-1}+\sum_{i=1}^{t-1}C(A^{i}-A^{i-1})Bx_{t-i}$
(7)

For a fixed sequence length $T$, we consider a roll-out of the system
$\\{(x_{t},y_{t})\\}_{t=1}^{T}$ to be a single example. In particular, we
define $(X=(x_{1},y_{1},\ldots,x_{T-1},y_{T-1},x_{T}),Y=y_{T})$ where $X$
contains be the full information that is available for the model to predict
observation $y_{T}$. Trained on $N$ i.i.d. samples $\\{(X_{i},Y_{i})\\}$, the
goal of the model is to predict $Y_{i}$ from $X_{i}$.

We assume the samples satisfy
$\mathopen{}\mathclose{{}\left\|{x_{t}}}\right\|_{2},\mathopen{}\mathclose{{}\left\|{y_{t}}}\right\|_{2}\leq
c$ for a constant $c$. We compete against the class of functions in
$\mathcal{H}$ restricted to contain LDSs with parameters
$\Theta=(A,B,C,D,h_{0}=0)$ such that $0\preccurlyeq A\preccurlyeq\gamma\cdot
I$ where $\gamma<1$ is a constant and
$\mathopen{}\mathclose{{}\left\|{B}}\right\|_{F},\mathopen{}\mathclose{{}\left\|{C}}\right\|_{F},\mathopen{}\mathclose{{}\left\|{D}}\right\|_{F}\leq
c$ for an absolute constant $c$. For error metrics, we consider squared loss
function
$\ell_{X,Y}=\mathopen{}\mathclose{{}\left\|{h(x_{1},y_{1},\ldots,x_{T-1},y_{T-1},x_{T})-y_{T}}}\right\|^{2}$.
For an empirical sample set $S$, let $\ell_{S}(h)=\frac{1}{|S|}\sum_{(X,Y)\in
S}\ell_{X,Y}(h)$. Similarly, for a distribution $\mathcal{D}$, let
$\ell_{\mathcal{D}}(h)=\mathbb{E}_{(X,Y)\sim\mathcal{D}}[\ell_{X,Y}(h)]$.

Now we define our auto-regressive hypothesis class. Let
$\tilde{X}\in\mathbb{R}^{(k+1)n+m}$ be the concatenated vector

$\mathopen{}\mathclose{{}\left[\begin{matrix}x_{t-k}&x_{t-k+1}&\cdots&x_{t-1}&\;x_{t}\;&\;y_{t-1}\end{matrix}}\right],$

and $f_{M}(\tilde{X})=M\tilde{X},M\in\mathbb{R}^{m\times(k+1)n+m}$. Our
hypothesis class is defined as
$\hat{\mathcal{H}}=\\{f_{M}|\mathopen{}\mathclose{{}\left\|{M}}\right\|_{F}\leq
O(1)\\}$.

We are ready to state our main theorem.

######  Proposition A.3 (Generalization bound of learning LDS with auto-
regressive algorithm).

Choose any $\varepsilon>0$. Let $S=\\{(X_{i},Y_{i})\\}_{i=1}^{N}$ be a set of
i.i.d. training samples from a distribution $\mathcal{D}$. Let
$\hat{h}\vcentcolon=\operatorname*{argmin}_{h\in\hat{\mathcal{H}}}\ell_{S}(h)$
with a choice of $k=\Theta(\log(1/\varepsilon))$. Let
$h^{*}\vcentcolon=\operatorname*{argmin}_{h^{*}\in\mathcal{H}}\ell_{\mathcal{D}}(h)$
be the loss minimizer over the set of $LDS$ predictors. Then, with probability
at least $1-\delta$, it holds that

$\ell_{\mathcal{D}}(\hat{h})-\min_{h\in\mathcal{H}}\ell_{\mathcal{D}}(h)\leq\varepsilon+\frac{O\mathopen{}\mathclose{{}\left(\log(1/\varepsilon)\sqrt{\log
1/\delta}}\right)}{\sqrt{N}}.$

The above result shows that the linear autoregressive predictor with a short
look-back window is competitive against the best LDS predictor where the
largest eigenvalue of the transition matrix $A$ is strictly smaller than $1$.
In Appendix A.2 we compare linear models with LSTM’s and Transformers on long-
term forecasting tasks on data generated by LDS, thus validating our
theoretical results.

###### Proof of Proposition A.3.

The proof proceeds as follows: First we show that an auto-regressive model
with look by window length $\Omega(\log(1/\varepsilon))$ can approximate an
LDS with error $\varepsilon$. Second, we prove a simple bound on the
Rademacher complexity of the class of auto-regressive model we considered,
which implies a generalization bound of our algorithm. Combining both results
yields our main result.

######  Proposition A.4 (Approximating LDS with auto-regressive model).

Let $\hat{y}_{T}$ be the predictions made by an LDS
$\Theta=(A,B,C,D,h_{0}=0)$. Then, for any $\varepsilon>0$, with a choice of
$k=\Omega\mathopen{}\mathclose{{}\left({\log(1/\varepsilon)}}\right)$, there
exists an $M_{\Theta}\in\mathbb{R}^{m\times(k+1)n+m}$ such that

$\mathopen{}\mathclose{{}\left\|{M_{\Theta}\tilde{X}-y_{T}}}\right\|^{2}\leq\mathopen{}\mathclose{{}\left\|{\hat{y}_{T}-y_{T}}}\right\|^{2}+\varepsilon.$

###### Proof.

This proposition is an analog of Theorem 3 in [Hazan et al., 2017]. We
construct $M_{\Theta}$ as the block matrix

$\mathopen{}\mathclose{{}\left[\begin{matrix}M^{(k-1)}&M^{(k-2)}&\cdots&M^{(1)}&M^{(x^{\prime})}&M^{(x)}&M^{(y)}\end{matrix}}\right],$

where the blocks’ dimensions are chosen to align with $\tilde{X}_{t}$, the
concatenated vector

$\mathopen{}\mathclose{{}\left[\begin{matrix}x_{t-k}&x_{t-k+1}&\cdots&x_{t-1}&\;x_{t}\;&\;y_{t-1}\end{matrix}}\right],$

so that the prediction is the block matrix-vector product

$M_{\Theta}\tilde{X}_{t}=\sum_{j=1}^{k-1}M^{(j)}x_{t-1-j}+M^{(x^{\prime})}x_{t-1}+M^{(x)}x_{t}+M^{(y)}y_{t-1}.$

Our construction is as follows:

  * •

$M^{(j)}=C(A^{j+1}-A^{j})B$, for each $1\leq j\leq k-1$.

  * •

$M^{(x^{\prime})}=C(A-I)B-D,\quad M^{(x)}=CB+D,\quad M^{(y)}=I_{m\times m}$.

The prediction of the LDS is by definition

$\displaystyle\hat{y}_{t}$
$\displaystyle=y_{t-1}+(CB+D)x_{t}-Dx_{t-1}+\sum_{i=1}^{t-1}C(A^{i}-A^{i-1})Bx_{t-i}$

We conclude that

$\hat{y}_{t}=M_{\Theta}\tilde{X}_{t}+\sum_{i=k+2}^{T}C(A^{i}-A^{i-1})Bx_{t-i}.$

This implies

$\displaystyle\mathopen{}\mathclose{{}\left\|{M_{\Theta}\tilde{X}_{t}-y_{t}}}\right\|^{2}$
$\displaystyle\leq$
$\displaystyle\mathopen{}\mathclose{{}\left\|{\hat{y}_{t}-y_{t}}}\right\|^{2}+2\mathopen{}\mathclose{{}\left\|{\hat{y}_{t}-y_{t}}}\right\|\mathopen{}\mathclose{{}\left\|{\sum_{i=k+2}^{T}C(A^{i}-A^{i-1})Bx_{t-i}}}\right\|+\mathopen{}\mathclose{{}\left\|{\sum_{i=k+2}^{T}C(A^{i}-A^{i-1})Bx_{t-i}}}\right\|^{2}$
$\displaystyle\leq$
$\displaystyle\mathopen{}\mathclose{{}\left\|{\hat{y}_{t}-y_{t}}}\right\|^{2}+O(\frac{\gamma^{k+2}+\gamma^{2k+4}}{1-\gamma})\leq\mathopen{}\mathclose{{}\left\|{\hat{y}_{t}-y_{t}}}\right\|+\varepsilon$

∎

The empirical Rademacher complexity of $\hat{\mathcal{H}}$ on $N$ samples,
with this restriction that $\mathopen{}\mathclose{{}\left\|{M}}\right\|=O(1)$,
satisfies

$\mathcal{R}_{N}(\hat{\mathcal{H}})\leq
O\mathopen{}\mathclose{{}\left(\frac{1}{\sqrt{N}}}\right).$

It’s easy to check that
$\mathopen{}\mathclose{{}\left\|{M_{\Theta}}}\right\|_{F}\leq
O\mathopen{}\mathclose{{}\left(\sqrt{\sum_{i=0}^{k}\gamma^{i}}}\right)=O(1)$,
which falls in the feasible set of our algorithm. The maximum loss
$\ell_{\max}$ of the hypothesis in model class $\hat{\mathcal{H}}$ is bounded
by $O(k)$. The lipschitz constant of loss function in the matrix $M$ is
$G_{\mathrm{max}}\leq\mathopen{}\mathclose{{}\left\|{M\tilde{X}_{t}-y_{t}}}\right\|_{2}\cdot\mathopen{}\mathclose{{}\left\|{\tilde{X}_{t}}}\right\|_{2}\leq
O(k)$

With all of these facts in hand, a standard Rademacher complexity-dependent
generalization bound holds in the improper hypothesis class
$\hat{\mathcal{H}}$ (see, e.g. [Bartlett and Mendelson, 2002]):

######  Lemma A.5 (Generalization via Rademacher complexity).

With probability at least $1-\delta$, it holds that

$\displaystyle\ell_{\mathcal{D}}(\hat{h})-\ell_{\mathcal{D}}(\hat{h}^{*})$
$\displaystyle\leq
G_{\mathrm{max}}\mathcal{R}_{N}(\hat{\mathcal{H}})+\ell_{\mathrm{max}}\sqrt{\frac{8\ln
2/\delta}{N}}$

With the stated choice of $k$, an upper bound for the RHS of Lemma A.5 is

$\frac{O\mathopen{}\mathclose{{}\left(\log(1/\varepsilon)\sqrt{\log
1/\delta}}\right)}{\sqrt{N}}.$

Combining this with the approximation result (Proposition A.4) yields the
theorem. ∎

###  A.2 Experimental Results on Synthetic Datasets

Dataset. We evaluate several models on a synthetic dataset generated from a
linear dynamical system. The transition matrix $A$ is a $30\times 30$
dimension Wishart random matrix normalized to have operator norm equals
$0.95$. The noise $\eta_{t}$ in the state transition follows from a
$30$-dimensional Gaussian distribution. The input at each time-step $x_{t}$
follows from a $5$-dimensional standard Gaussian distribution, which is
observable to the model. Finally, We add seasonality of $6$ different
periodicity by adding cosine signal as the input to the linear dynamical
system, but hidden from the prediction model. We generate $4$ different time
series which shares the same model parameter, input $x_{t}$ and seasonality
input, with the only difference coming from the randomness in the state
transition. We set look-back window to be length $320$, and horizon also to
length $320$. For each time-series, we use the first $1640$ steps for
training, next $740$ steps for validation, and the final $740$ steps for
testing, which results in $4000$ examples for training, $400$ examples for
validation, and $400$ examples for testing.

Baselines and Setup. We evaluate three models on our synthetic dataset:
linear, long short-term memory (LSTM) and Transformer. Our linear model is a
direct linear map between the history in look-back window and future. We use a
one layer LSTM with dimension 128. For Transformer, we use a two layer self-
attention layers with dimension 128, combined with a one hidden layer feed-
forward network with 128 hidden units.

Results. We present Mean Squared Error (MSE) for all models in Table 4. For
all models, we report the mean and standard deviation out of 3 independent
runs for each setting. The bold-faced numbers are from the best model or
within statistical significance of the best model in terms of two standard
error intervals. We also plot the actuals (ground truth) vs the predictions
from Linear, LSTM and Transformer models. We see that Transformer captures the
lower frequency seasonality of the time-series but seems to not be able to
leverage the inputs/covariates to predict the short term variation of the
value, LSTM seems to not capture the trend/seasonality correctly, while Linear
model’s prediction is the closest to the truth, which matches the metric
result in Table 4.

Model | MSE
---|---
Linear | $\mathbf{0.510\pm 0.001}$
LSTM | $1.455\pm 0.455$
Transformer | $0.731\pm 0.041$
Table 4: Mean Squared Error (MSE) of Linear, LSTM and Transformer models on
synthetic time-series. Figure 6: We plot the actuals vs the predictions from
Linear, LSTM and Transformer models.

##  Appendix B More Experimental Details

###  B.1 Additional Experiments

#### Comparison against S4.

In Table 6 we present the results of our model along side that of the S4 model
[Gu et al., ]. The numbers of S4 are directly taken from Table-14 of the
original paper. It can be seen that TiDE vastly outperforms the S4 model on
the time-series benchmarks.

#### M5 Forecasting.

We will now provide more details about the M5 forecasting experiments. We
follow the setup used in the notebook linked in Section 5 that was released by
the authors of [Alexandrov et al., 2020]. The list of dynamic features include
date derived features, promotion features like snap_CA, snap_TX, snap_WI,
even_type_1 and event_type_2. It also includes static attributes like
category_id, store_id, department_id and item_id. The categorical features are
embedded into learnable embeddings.

DeepAR [Salinas et al., 2020] has suggested using the zero-inflated negative
binomial loss likelihood as the loss function for sparse count data in the
dataset. Therefore we use this loss function for our model and the DeepAR
model. All models are trained to a maximum of 100 epochs with an early
stopping patience of 5.

Models | TiDE
---|---
Electricity | 96 | ${0.132\pm 0.003}$ | ${0.229\pm 0.001}$
192 | ${0.147\pm 0.003}$ | ${0.243\pm 0.001}$
336 | ${0.161\pm 0.001}$ | ${0.261\pm 0.002}$
720 | $0.196\pm 0.002$ | $0.294\pm 0.001$
Traffic | 96 | ${0.336\pm 0.001}$ | ${0.253\pm 0.001}$
192 | ${0.346\pm 0.001}$ | ${0.257\pm 0.002}$
336 | ${0.355\pm 0.001}$ | ${0.260\pm 0.001}$
720 | $0.386\pm 0.002$ | $0.273\pm 0.0005$
Weather | 96 | ${0.166\pm 0.0005}$ | ${0.222\pm 0.0005}$
192 | ${0.209\pm 0.002}$ | ${0.263\pm 0.0001}$
336 | ${0.254\pm 0.002}$ | ${0.301\pm 0.0001}$
720 | $0.313\pm 0.001$ | $0.340\pm 0.0002$
ETTm2 | 96 | ${0.161\pm 0.0002}$ | ${0.251\pm 0.0003}$
192 | ${0.215\pm 0.0001}$ | ${0.289\pm 0.0004}$
336 | ${0.267\pm 0.0001}$ | ${0.326\pm 0.0002}$
720 | $0.352\pm 0.0002$ | $0.383\pm 0.0002$
ETTm1 | 96 | ${0.306\pm 0.0001}$ | ${0.349\pm 0.0002}$
192 | ${0.335\pm 0.0002}$ | ${0.366\pm 0.0002}$
336 | ${0.364\pm 0.0004}$ | ${0.384\pm 0.0001}$
720 | $0.413\pm 0.0001$ | $0.413\pm 0.0001$
ETTh1 | 96 | ${0.375\pm 0.0003}$ | ${0.398\pm 0.0002}$
192 | ${0.412\pm 0.0002}$ | ${0.422\pm 0.0001}$
336 | ${0.435\pm 0.0001}$ | ${0.433\pm 0.0001}$
720 | $0.454\pm 0.0003$ | $0.465\pm 0.0001$
ETTh2 | 96 | ${0.270\pm 0.0005}$ | ${0.336\pm 0.0007}$
192 | ${0.332\pm 0.001}$ | ${0.380\pm 0.002}$
336 | ${0.360\pm 0.001}$ | ${0.407\pm 0.001}$
720 | $0.419\pm 0.005$ | $0.451\pm 0.002$
Table 5: We provide standard error bars for our method over 5 independent runs. Models | TiDE | S4
---|---|---
Metric | MSE | MAE | MSE | MAE
Weather | 336 | 0.254 | 0.301 | 0.531 | 0.539
720 | 0.313 | 0.340 | 0.578 | 0.578
ETTh1 | 336 | 0.435 | 0.433 | 1.407 | 0.910
720 | 0.454 | 0.465 | 1.162 | 0.842
ETTh2 | 336 | 0.360 | 0.407 | 0.531 | 0.539
720 | 0.419 | 0.451 | 2.650 | 1.340
Electricity | 336 | 0.161 | 0.261 | 0.531 | 0.539
720 | 0.196 | 0.294 | 0.578 | 0.578
Table 6: We benchmark our model’s performance against that of S4. The S4
results are taken from Table 14 of the original paper [Gu et al., ].

###  B.2 Data Loader

Each training batch consists of a look-back $\bm{{Y}}[\mathcal{B},t-L:t-1]$
and a horizon $\bm{{Y}}[\mathcal{B},t:t+H-1]$. Here, $t$ can range from $L+1$
to $H$ steps before the end of the training set. $\mathcal{B}$ denotes the
indices of the time-series in the batch and the batchSize can be set as a
hyper-parameter. When batchSize is greater than $N$, all the time-series are
loaded in a batch.

We also load time derived features as covariates. The time-stamps
corresponding to time-indices $t-L:t+H-1$ are converted to periodic features
like minute of the hour, hour of the day, day of the week etc normalized to
the scale $[-0.5,0.5]$ as done in GluonTS [Alexandrov et al., 2020]. In total
we have 8 such features, many of which can stay constant depending on the
granularity of the dataset.

###  B.3 Hyperparameters

Recall that in Section 4, we had the following hyper-parameters temporalWidth,
hiddenSize, numEncoderLayers, numDecoderLayers, decoderOutputDim and
temporalDecoderHidden. We also have hyper-parameters layerNorm and
dropoutLevel that denote the global model level layer norm on/off and the
probability of dropout. We also tune the maximum learningRate which is the
input to a cosine decay learning rate schedule. In all our experiments
batchSize is fixed to $512$ and temporalWidth is fixed to $4$. We also tune
whether reversible instance normalization [Kim et al., 2021] is turned on or
off. The tuning range of the hparams are provided in Table 7. We use the
validation loss to tune the hyper-parameters per dataset.

Parameter | Range
---|---
hiddenSize | [256, 512, 1024]
numEncoderLayers | [1, 2, 3]
numDecoderLayers | [1, 2, 3]
decoderOutputDim | [4, 8, 16, 32]
temporalDecoderHidden | [32, 64, 128]
dropoutLevel | [0.0, 0.1, 0.2, 0.3, 0.5]
layerNorm | [True, False]
learningRate | Log-scale in [1e-5, 1e-2]
revIn | [True, False]
Table 7: Ranges of different hyper-paramaters

We report the specific hyper-parameters chosen for each dataset in Table 8.

Dataset | hiddenSize | numEncoderLayers | numDecoderLayers | decoderOutputDim | temporalDecoderHidden | dropoutLevel | layerNorm | learningRate | revIn
---|---|---|---|---|---|---|---|---|---
Traffic | 256 | 1 | 1 | 16 | 64 | 0.3 | False | 6.55e-5 | True
Electricity | 1024 | 2 | 2 | 8 | 64 | 0.5 | True | 9.99e-4 | False
ETTm1 | 1024 | 1 | 1 | 8 | 128 | 0.5 | True | 8.39e-5 | False
ETTm2 | 512 | 2 | 2 | 16 | 128 | 0.0 | True | 2.52e-4 | True
ETTh1 | 256 | 2 | 2 | 8 | 128 | 0.3 | True | 3.82e-5 | True
ETTh2 | 512 | 2 | 2 | 32 | 16 | 0.2 | True | 2.24e-4 | True
Weather | 512 | 1 | 1 | 8 | 16 | 0.0 | True | 3.01e-5 | False

Table 8: The hyper-parameters for different experimental settings.