eccentricity stellar halo debris in Milky Way analogs the EAGLE simulations in
order to derive a correspondence between mass and metallicity for comparison
with APOGEE. Their observed trends of progenitor mass with [Fe/H] and
[$\alpha$/Fe] (see their figure 11) would by no means exclude a progenitor
with stellar mass $\sim 1.5\times 10^{8}$ $\mathrm{M}_{\odot}$ given a typical
[Fe/H] for GS/E of $-1.2$ (Horta et al., 2023a). Horta et al. (2023b) use the
FIRE simulations to investigate the properties of merger remnants in the halos
of Milky Way analogs, and their findings would indeed be challenging to
reconcile with our low stellar mass for GS/E. They show that fully phase-mixed
remnants with [Fe/H] comparable to GS/E tend to have higher stellar masses.
These comparisons suggest that while there may be some tension in placing such
a light GS/E in a cosmological context, it would be by no means an outlier
when compared with other remnants in the Milky Way stellar halo.

We posit that the reason for our detection of a substantially lower mass for
GS/E compared with other studies is that our kinematic selections are far more
restrictive. LBM22 compared literature kinematic selections for GS/E with
their derived high-purity selections, finding that some commonly used
selections for GS/E can achieve purity as low as 0.5-0.6, and many achieve
purity in the range of 0.6-0.7. This adds a substantial degree of
contamination to samples selected in this way, artificially increasing
inferred masses. This effect is compounded, however, by the fact that a
contaminated sample will also yield a different fit to the GS/E sample, one
which is more likely to resemble the low-$\beta$ stellar halo. We specifically
attribute the large derived mass of the $\sqrt{J_{\mathrm{R}}}-L_{\mathrm{z}}$
selection sample to this exact behaviour, since the purity using our original
DF models is 0.78 and it drops to 0.52 when using our improved kinematic
effective selection function.

####  5.3.2 The density profile of GS/E

We find that the density profile of GS/E is described by a shallow power law
in the inner parts of the Galaxy, and steepens between 20 and 30 kpc. These
findings are in good agreement with recent studies such as Han et al. (2022),
who detect breaks at 12 and 28 kpc. Our power law models with two breaks find
those radii to be at about 15-20 kpc and 30-40 kpc, which is broadly
consistent with the findings of Han et al. (2022). These authors find inner,
middle, and outer power laws of 1.7, 3.1, and 4.6 respectively. These power
laws are roughly consistent with both our BPL and DBPL findings for the AD
selection, where for the BPL profile we have inner power laws of about 1.2
steepening to about 3.8, and for the DBPL profile the inner, middle, and outer
power laws are about 0.9, 3.0, and 6.2. We are therefore shallower in the
inner Galaxy and steeper in the outer Galaxy when compared with the results of
Han et al. (2022). Our break radius is also consistent with the findings of
MB20 who see a break radius of 25 kpc. They do find a much steeper inner power
law ($\alpha\sim 3.5$), but they were not specifically targeting GS/E in their
fits. in general, the finding of a break radius between 15-30 kpc is in broad
agreement with other literature findings (Sesar et al., 2011; Deason et al.,
2011; Xue et al., 2015; Deason et al., 2019).

Interestingly, we find that our best-fitting density profiles for GS/E are
similar in orientation to those found by Iorio & Belokurov (2019), who noted
that the major axis of the GS/E density profile is broadly aligned with both
the Hercules-Aquila Cloud and Virgo Overdensity. Specifically, the major axis
of our density profiles is rotated by $90\degr-100\degr$ (specifically
$99.5^{+13.9}_{-14.9}$ for our best-fitting AD model) in the Galactocentric
$X-Y$ plane towards the direction of Galactic rotation (positive $Y$), and
inclined about $16\degr$ below the plane. Iorio & Belokurov (2019) find that
the principal axis is rotated about $20\degr$ from the Galactocentric Y-axis
(about $70\degr$ towards anti-rotation from the X-axis) and and inclination of
about $20\degr$ (oriented the same as our model). When we cast our rotation
angles in the same frame as theirs we would find rotation of about $9.5\degr$
from the Y-axis and $80.5\degr$ towards anti-rotation from the X-axis. These
results are very similar (c.f. our Figure 10 with their Figures 2 and 3),
reinforcing the idea that the Hercules-Aquila Cloud and Virgo Overdensity are
related to the GS/E remnant. Similar findings were also recently reported by
Han et al. (2022), although they find the rotation of the principal axis with
respect to the Sun-Galactic center axis to be only $24\degr$, but a similar
inclination of the principal axis of $25\degr$.

Regarding axis scale ratios, we find that the GS/E remnant is significantly
more triaxial than other studies have reported. Han et al. (2022) find axis
ratios of 1:0.8:0.7 for the GS/E remnant, and Iorio et al. (2018) finds
1:0.8:0.5-0.6, while we find axis ratios 1:0.55:0.45. The cause of this
mismatch could be attributed to our selections, and perhaps the GS/E is truly
more triaxial than these other studies have noted. A more likely
interpretation of the difference in these findings is that GS/E changes in
shape with increasing Galactocentric radius. The model of Iorio et al. (2018)
has variable flattening with radius, while the shape parameters of the models
used by Han et al. (2022), like ours, are fixed with radius. It may be that
surveys which are sensitive to different ranges of Galactocentric radius (due
to the differences in observational footprint, choice of tracer, etc) find
different shape parameters for GS/E. It is curious though that the results of
Han et al. (2022), using data from the H3 survey which should probe larger
Galactocentric radii compared with APOGEE which probe smaller Galactocentric
radii, should find less triaxiality since one might generally expect large
degrees of triaxiality at larger Galactocentric radii. Nevertheless, when
these results are collated with recent findings of complex structure
associated with the GS/E remnant at large Galactocentric radii by Chandra et
al. (2022), we can likely say that GS/E has a complicated shape-radius
dependency.

##  6 Summary and Conclusions

In this work we have measured the mass of the GS/E remnant using high purity
samples of APOGEE DR16 red giants selected using kinematics and chemistry. We
employ a density modelling approach which accounts for the APOGEE survey
selection function, dust obscuration, and the density of the tracer population
in colour-magnitude space. To this method we add a novel technique for
correcting the biases induced by kinematic selections which is based the
creation of a kinematic effective selection function using fiducial
distribution function models of the Milky Way stellar halo. We identify GS/E
in multiple kinematic spaces using high-purity kinematic selections based on
the work of LBM22, and we fit triaxial rotated density profiles of varying
radial complexity to these samples. We validate our density modelling
approach, and specifically our new kinematic effective selection functions,
using mock APOGEE data. Our main results can be summarized as follows:

  * •

We find that our kinematically-selected GS/E samples are well fit by density
profiles which have relatively shallow or flat power law indices in the inner
galaxy ($0.5<\alpha_{1}<1.5$) and drop sharply at $\sim~{}20-30$ kpc to
$\alpha>3.5$ at larger radii.

  * •

The best-fitting GS/E density profiles are triaxial, with axis ratios
$1:0.55:0.45$. The principal axis is rotated by about $100\degr$ towards the
direction of Galactic rotation and is inclined $16\degr$ below the plane,
which is in good agreement with previous studies.

  * •

We determine the mass of GS/E to be $1.45\
^{+0.92}_{-0.51}\,10^{8}\,M_{\odot}$, which is lower than found in previous
studies. An assessment of sources of systematic error suggests they are
smaller in magnitude than the statistical uncertainty quoted above. We
attribute our finding of a lower mass to our use of restrictive kinematic cuts
which produces a high-purity sample of GS/E stars and removes contamination
from stellar halo populations with lower velocity anisotropy.

  * •

A stellar mass of $1.5\times 10^{8}\,\mathrm{M}_{\odot}$ corresponds to a
total halo mass of GS/E of $\approx 5-6\times 10^{10}\,\mathrm{M}_{\odot}$
using standard stellar-mass–to–halo-mass relations. Given that the Milky Way’s
halo mass at the time of the merger was likely $\approx 5\times
10^{11}\,\mathrm{M}_{\odot}$, GS/E constituted a minor 1:8-1:10 merger.

  * •

We also fit density profiles to the whole stellar halo, finding that the best-
fitting profiles have power law index around $\alpha=2.5$, with no evidence
for breaks in the density profile. We find the mass of the stellar halo to be
between $6.7-8.4\times 10^{8}$ $\mathrm{M}_{\odot}$ out to 70 kpc. GS/E
therefore represents only a modest fraction of the mass of the stellar halo,
about 15 per cent at the low end or 25 per cent at the high end.

We have shown evidence that the GS/E remnant is lighter than many previous
studies have suggested. We argue that our precise kinematic selections for
GS/E driven by fiducial stellar halo distribution function models leads to
less contamination by stellar halo populations with low velocity anisotropy.
While we are confident in our results, they represent the first application of
a new technique for integrating kinematic selections into Galactic density
modelling through the use of distribution functions. Additional work needs to
be done to explore distribution functions which more accurately represent the
stellar populations known to exist in the stellar halo, including triaxial,
rotating distribution functions for example. We hope that simulations may be
illuminating for this purpose, and plan to study this technique in the context
of cosmological N-body simulations. We envision a future where stellar halo
density modelling does not rely on kinematic selections to define samples, but
involves modelling of the full six dimensions of phase space.

## Acknowledgements

JMML and JB acknowledge financial support from NSERC (funding reference number
RGPIN-2020-04712) and an Ontario Early Researcher Award (ER16-12-061). We
thank Josh Speagle and Ting Li for helpful discussions and comments on the
manuscript. This work has made use of data from the European Space Agency
(ESA) mission Gaia (https://www.cosmos.esa.int/gaia), processed by the Gaia
Data Processing and Analysis Consortium (DPAC,
https://www.cosmos.esa.int/web/gaia/dpac/consortium). Funding for the DPAC has
been provided by national institutions, in particular the institutions
participating in the Gaia Multilateral Agreement. Funding for the Sloan
Digital Sky Survey IV has been provided by the Alfred P. Sloan Foundation, the
U.S. Department of Energy Office of Science, and the Participating
Institutions. SDSS-IV acknowledges support and resources from the Center for
High-Performance Computing at the University of Utah. The SDSS web site is
www.sdss.org.

## Data Availability

The APOGEE DR16 data used in this article are available at:
https://www.sdss.org/dr16. The Gaia data used in this article are available
at: https://gea.esac.esa.int/archive/.

## References

  * Abdurro’uf et al. (2022) Abdurro’uf et al., 2022, ApJS, 259, 35
  * Ahumada et al. (2020) Ahumada R., et al., 2020, ApJS, 249, 3
  * Akaike (1974) Akaike H., 1974, IEEE Transactions on Automatic Control, 19, 716
  * Amarante et al. (2022) Amarante J. A. S., Debattista V. P., Beraldo e Silva L., Laporte C. F. P., Deg N., 2022, ApJ, 937, 12
  * Amorisco (2017) Amorisco N. C., 2017, MNRAS, 464, 2882
  * Beaton et al. (2021) Beaton R. L., et al., 2021, AJ, 162, 302
  * Behroozi et al. (2013) Behroozi P. S., Wechsler R. H., Conroy C., 2013, ApJ, 770, 57
  * Belokurov & Kravtsov (2022) Belokurov V., Kravtsov A., 2022, MNRAS, 514, 689
  * Belokurov et al. (2018) Belokurov V., Erkal D., Evans N. W., Koposov S. E., Deason A. J., 2018, MNRAS, 478, 611
  * Bennett & Bovy (2019) Bennett M., Bovy J., 2019, MNRAS, 482, 1417
  * Binney (2012) Binney J., 2012, MNRAS, 426, 1324
  * Binney (2014) Binney J., 2014, MNRAS, 440, 787
  * Binney (2018) Binney J., 2018, MNRAS, 474, 2706
  * Binney & Tremaine (2008) Binney J., Tremaine S., 2008, Galactic Dynamics: Second Edition. Princeton University Press, Princeton, NJ
  * Bland-Hawthorn & Gerhard (2016) Bland-Hawthorn J., Gerhard O., 2016, ARA&A, 54, 529
  * Blanton et al. (2017) Blanton M. R., et al., 2017, AJ, 154, 28
  * Bovy (2015) Bovy J., 2015, ApJS, 216, 29
  * Bovy et al. (2012) Bovy J., Rix H.-W., Liu C., Hogg D. W., Beers T. C., Lee Y. S., 2012, ApJ, 753, 148
  * Bovy et al. (2016a) Bovy J., Rix H.-W., Green G. M., Schlafly E. F., Finkbeiner D. P., 2016a, ApJ, 818, 130
  * Bovy et al. (2016b) Bovy J., Rix H.-W., Schlafly E. F., Nidever D. L., Holtzman J. A., Shetrone M., Beers T. C., 2016b, ApJ, 823, 30
  * Bowen & Vaughan (1973) Bowen I. S., Vaughan A. H. J., 1973, Appl. Opt., 12, 1430
  * Bressan et al. (2012) Bressan A., Marigo P., Girardi L., Salasnich B., Dal Cero C., Rubele S., Nanni A., 2012, MNRAS, 427, 127
  * Bullock & Johnston (2005) Bullock J. S., Johnston K. V., 2005, ApJ, 635, 931
  * Chabrier (2003) Chabrier G., 2003, PASP, 115, 763
  * Chandra et al. (2022) Chandra V., et al., 2022, arXiv e-prints, p. arXiv:2212.00806
  * Conroy et al. (2019) Conroy C., et al., 2019, ApJ, 883, 107
  * Cui et al. (2012) Cui X.-Q., et al., 2012, Research in Astronomy and Astrophysics, 12, 1197
  * Das et al. (2020) Das P., Hawkins K., Jofré P., 2020, MNRAS, 493, 5195
  * Deason et al. (2011) Deason A. J., Belokurov V., Evans N. W., 2011, MNRAS, 416, 2903
  * Deason et al. (2019) Deason A. J., Belokurov V., Sanders J. L., 2019, MNRAS, 490, 3426
  * Drimmel et al. (2003) Drimmel R., Cabrera-Lavers A., López-Corredoira M., 2003, A&A, 409, 205
  * Eisenstein et al. (2011) Eisenstein D. J., et al., 2011, AJ, 142, 72
  * Fattahi et al. (2019) Fattahi A., et al., 2019, MNRAS, 484, 4471
  * Fernandes et al. (2023) Fernandes L., et al., 2023, MNRAS, 519, 3611
  * Feuillet et al. (2020) Feuillet D. K., Feltzing S., Sahlholdt C. L., Casagrande L., 2020, MNRAS, 497, 109
  * Foreman-Mackey et al. (2013) Foreman-Mackey D., Hogg D. W., Lang D., Goodman J., 2013, PASP, 125, 306
  * Freeman & Bland-Hawthorn (2002) Freeman K., Bland-Hawthorn J., 2002, ARA&A, 40, 487
  * GRAVITY Collaboration et al. (2021) GRAVITY Collaboration et al., 2021, A&A, 647, A59
  * Gaia Collaboration et al. (2016) Gaia Collaboration et al., 2016, A&A, 595, A1
  * Gaia Collaboration et al. (2022) Gaia Collaboration et al., 2022, arXiv e-prints, p. arXiv:2208.00211
  * García Pérez et al. (2016) García Pérez A. E., et al., 2016, AJ, 151, 144
  * Goodman & Weare (2010) Goodman J., Weare J., 2010, Communications in Applied Mathematics and Computational Science, 5, 65
  * Grand et al. (2017) Grand R. J. J., et al., 2017, MNRAS, 467, 179
  * Green et al. (2015) Green G. M., et al., 2015, ApJ, 810, 25
  * Gunn et al. (2006) Gunn J. E., et al., 2006, AJ, 131, 2332
  * Han et al. (2022) Han J. J., et al., 2022, AJ, 164, 249
  * Harris (1996) Harris W. E., 1996, AJ, 112, 1487
  * Hasselquist et al. (2021) Hasselquist S., et al., 2021, ApJ, 923, 172
  * Hawkins et al. (2015) Hawkins K., Jofré P., Masseron T., Gilmore G., 2015, MNRAS, 453, 758
  * Haywood et al. (2018) Haywood M., Di Matteo P., Lehnert M. D., Snaith O., Khoperskov S., Gómez A., 2018, ApJ, 863, 113
  * Helmi (2020) Helmi A., 2020, ARA&A, 58, 205
  * Helmi et al. (1999) Helmi A., White S. D. M., de Zeeuw P. T., Zhao H., 1999, Nature, 402, 53
  * Helmi et al. (2018) Helmi A., Babusiaux C., Koppelman H. H., Massari D., Veljanoski J., Brown A. G. A., 2018, Nature, 563, 85
  * Hernquist (1990) Hernquist L., 1990, ApJ, 356, 359
  * Holtzman et al. (2018) Holtzman J. A., et al., 2018, AJ, 156, 125
  * Horta et al. (2021a) Horta D., et al., 2021a, MNRAS, 500, 1385
  * Horta et al. (2021b) Horta D., et al., 2021b, MNRAS, 500, 5462
  * Horta et al. (2023a) Horta D., et al., 2023a, MNRAS, 520, 5671
  * Horta et al. (2023b) Horta D., et al., 2023b, ApJ, 943, 158
  * Iorio & Belokurov (2019) Iorio G., Belokurov V., 2019, MNRAS, 482, 3868
  * Iorio & Belokurov (2021) Iorio G., Belokurov V., 2021, MNRAS, 502, 5686
  * Iorio et al. (2018) Iorio G., Belokurov V., Erkal D., Koposov S. E., Nipoti C., Fraternali F., 2018, MNRAS, 474, 2142
  * Jean-Baptiste et al. (2017) Jean-Baptiste I., Di Matteo P., Haywood M., Gómez A., Montuori M., Combes F., Semelin B., 2017, A&A, 604, A106
  * Johnston et al. (2008) Johnston K. V., Bullock J. S., Sharma S., Font A., Robertson B. E., Leitner S. N., 2008, ApJ, 689, 936
  * Jönsson et al. (2020) Jönsson H., et al., 2020, AJ, 160, 120
  * Lancaster et al. (2019) Lancaster L., Koposov S. E., Belokurov V., Evans N. W., Deason A. J., 2019, MNRAS, 486, 378
  * Lane et al. (2022) Lane J. M. M., Bovy J., Mackereth J. T., 2022, MNRAS, 510, 5119
  * Leung & Bovy (2019a) Leung H. W., Bovy J., 2019a, MNRAS, 483, 3255
  * Leung & Bovy (2019b) Leung H. W., Bovy J., 2019b, MNRAS, 489, 2079
  * Mackereth & Bovy (2018) Mackereth J. T., Bovy J., 2018, PASP, 130, 114501
  * Mackereth & Bovy (2020) Mackereth J. T., Bovy J., 2020, MNRAS, 492, 3631
  * Mackereth et al. (2017) Mackereth J. T., et al., 2017, MNRAS, 471, 3057
  * Mackereth et al. (2018) Mackereth J. T., Crain R. A., Schiavon R. P., Schaye J., Theuns T., Schaller M., 2018, MNRAS, 477, 5072
  * Mackereth et al. (2019) Mackereth J. T., et al., 2019, MNRAS, 482, 3426
  * Majewski et al. (2011) Majewski S. R., Zasowski G., Nidever D. L., 2011, ApJ, 739, 25
  * Majewski et al. (2017) Majewski S. R., et al., 2017, AJ, 154, 94
  * Marshall et al. (2006) Marshall D. J., Robin A. C., Reylé C., Schultheis M., Picaud S., 2006, A&A, 453, 635
  * Martell et al. (2017) Martell S. L., et al., 2017, MNRAS, 465, 3203
  * Montalbán et al. (2020) Montalbán J., et al., 2020, arXiv e-prints, p. arXiv:2006.01783
  * Monty et al. (2020) Monty S., Venn K. A., Lane J. M. M., Lokhorst D., Yong D., 2020, MNRAS, 497, 1236
  * Myeong et al. (2019) Myeong G. C., Vasiliev E., Iorio G., Evans N. W., Belokurov V., 2019, MNRAS, 488, 1235
  * Nadler et al. (2020) Nadler E. O., et al., 2020, ApJ, 893, 48
  * Naidu et al. (2020) Naidu R. P., Conroy C., Bonaca A., Johnson B. D., Ting Y.-S., Caldwell N., Zaritsky D., Cargile P. A., 2020, ApJ, 901, 48
  * Naidu et al. (2021) Naidu R. P., et al., 2021, arXiv e-prints, p. arXiv:2103.03251
  * Naidu et al. (2022) Naidu R. P., et al., 2022, arXiv e-prints, p. arXiv:2204.09057
  * Nidever et al. (2015) Nidever D. L., et al., 2015, AJ, 150, 173
  * Posti et al. (2015) Posti L., Binney J., Nipoti C., Ciotti L., 2015, MNRAS, 447, 3060
  * Powell (1964) Powell M. J. D., 1964, The Computer Journal, 7, 155
  * Read et al. (2017) Read J. I., Iorio G., Agertz O., Fraternali F., 2017, MNRAS, 467, 2019
  * Rix & Bovy (2013) Rix H.-W., Bovy J., 2013, A&ARv, 21, 61
  * Sanders (2012) Sanders J., 2012, MNRAS, 426, 128
  * Sanders & Binney (2015) Sanders J. L., Binney J., 2015, MNRAS, 447, 2479
  * Santana et al. (2021) Santana F. A., et al., 2021, AJ, 162, 303
  * Schaye et al. (2015) Schaye J., et al., 2015, MNRAS, 446, 521
  * Schönrich et al. (2010) Schönrich R., Binney J., Dehnen W., 2010, MNRAS, 403, 1829
  * Schwarz (1978) Schwarz G., 1978, The Annals of Statistics, 6, 461
  * Searle & Zinn (1978) Searle L., Zinn R., 1978, ApJ, 225, 357
  * Sesar et al. (2011) Sesar B., Jurić M., Ivezić Ž., 2011, ApJ, 731, 4
  * Shetrone et al. (2015) Shetrone M., et al., 2015, ApJS, 221, 24
  * Skrutskie et al. (2006) Skrutskie M. F., et al., 2006, AJ, 131, 1163
  * Smith et al. (2021) Smith V. V., et al., 2021, AJ, 161, 254
  * Vincenzo et al. (2019) Vincenzo F., Spitoni E., Calura F., Matteucci F., Silva Aguirre V., Miglio A., Cescutti G., 2019, MNRAS, 487, L47
  * Wetzel et al. (2016) Wetzel A. R., Hopkins P. F., Kim J.-h., Faucher-Giguère C.-A., Kereš D., Quataert E., 2016, ApJ, 827, L23
  * White & Frenk (1991) White S. D. M., Frenk C. S., 1991, ApJ, 379, 52
  * Williams & Evans (2015) Williams A. A., Evans N. W., 2015, MNRAS, 448, 1360
  * Wilson et al. (2019) Wilson J. C., et al., 2019, PASP, 131, 055001
  * Xue et al. (2015) Xue X.-X., Rix H.-W., Ma Z., Morrison H., Bovy J., Sesar B., Janesh W., 2015, ApJ, 809, 144
  * Yanny et al. (2009) Yanny B., et al., 2009, AJ, 137, 4377
  * Zamora et al. (2015) Zamora O., et al., 2015, AJ, 149, 181
  * Zasowski et al. (2013) Zasowski G., et al., 2013, AJ, 146, 81
  * Zasowski et al. (2017) Zasowski G., et al., 2017, AJ, 154, 198

##  Appendix A Validation of the Method with Mock Data

Here we show the results of applying our density fitting technique to mock
data, as described in § 3.6. Figure 11 shows the posterior distributions for
fits to mock data with and without disk contamination, as well as mock data
created using kinematic cuts that mirror those used to select the GS/E
samples. The contours of the posterior distributions are set at 1 and
2-$\sigma$. The black lines show the input parameters for the models. The
posterior distributions always contain the input model parameters, yet are
broad in general. This is due to the fact that we chose to fit a lightweight
mock halo (only $2\times 10^{8}$ M⊙ total mass) to test whether our modelling
approach would work with a small number of stars. Since each stellar halo
realization is subject to a certain degree of randomness we generate multiple
stellar halos with similar parameters and a variety of masses. We find that
the input parameters are always contained in the posterior, and that there are
no major systematic over- or underestimations of input parameters beyond those
noted in § 3.6.

Figure 11: Corner plot showing the posterior samples of the fits to mock data.
Contours are placed at the 1 and 2-$\sigma$ levels. The black lines show the
fit to the whole mock sample without disk contamination, the grey line shows
the fit including to the whole sample including disk contamination, and the
coloured lines show the fits to the mock GS/E kinematic samples without disk
contamination. the solid black lines show the input parameters for the models,
and the dashed black lines show the mass expected for the kinematically
selected GS/E analog populations (half that of the whole mock).

##  Appendix B Completeness of the modified kinematic effective selection
function

Here we show the kinematic effective selection function derived based on our
fits to the GS/E and whole-halo populations described in § 4.4. Figures 12 and
13 are identical to Figures 5 and 6, except they use this new kinematic
effective selection function (denoted $\mathfrak{S}_{2}^{\prime}$). Onto
Figure 12 we add the median of $\mathfrak{S}_{2}^{\prime}$ and
$\mathfrak{S}_{2}$, calculated across all fields at each distance modulus, to
aid efficient comparison of typical values.

Figure 12: Same as Figure 5 but for the secondary kinematic effective
selection function (denoted $\mathfrak{S}_{2}^{\prime}$) described in § 4.4.
Overlaid on each panel are the median, calculated at each distane modulus,
both of this secondary kinematic effective selection function
$\mathfrak{S}_{2}^{\prime}$ (black solid line) and the original kinematic
effective selection function $\mathfrak{S}_{2}$ (black dashed line) presented
in Figure 12. Figure 13: Same as Figure 6 but for the secondary kinematic
effective selection function described in § 4.4.


# PDETime: Rethinking Long-Term Multivariate Time Series Forecasting from the
perspective of partial differential equations

Shiyi Qi  Zenglin Xu  Yiduo Li  Liangjian Wen  Qingsong Wen  Qifan Wang  Yuan
Qi

###### Abstract

Recent advancements in deep learning have led to the development of various
models for long-term multivariate time-series forecasting (LMTF), many of
which have shown promising results. Generally, the focus has been on
historical-value-based models, which rely on past observations to predict
future series. Notably, a new trend has emerged with time-index-based models,
offering a more nuanced understanding of the continuous dynamics underlying
time series. Unlike these two types of models that aggregate the information
of spatial domains or temporal domains, in this paper, we consider
multivariate time series as spatiotemporal data regularly sampled from a
continuous dynamical system, which can be represented by partial differential
equations (PDEs), with the spatial domain being fixed. Building on this
perspective, we present PDETime, a novel LMTF model inspired by the principles
of Neural PDE solvers, following the encoding-integration-decoding operations.
Our extensive experimentation across seven diverse real-world LMTF datasets
reveals that PDETime not only adapts effectively to the intrinsic
spatiotemporal nature of the data but also sets new benchmarks, achieving
state-of-the-art results.

Machine Learning, ICML

##  1 Introduction

Multivariate time series forecasting plays a pivotal role in diverse
applications such as weather prediction (Angryk et al., 2020), energy
consumption (Demirel et al., 2012), healthcare (Matsubara et al., 2014), and
traffic flow estimation (Li et al., 2017). In the field of multivariate time
series forecasting, two typical approaches have emerged: historical-value-
based (Zhou et al., 2021; Wu et al., 2021; Zeng et al., 2023; Nie et al.,
2023), and time-index-based models (Woo et al., 2023; Naour et al., 2023).
Historical-value-based models predict future time steps utilizing historical
observations, $\hat{x}_{t+1}=f_{\theta}(x_{t},x_{t-1},...)$, while time-index-
based models only use the corresponding time-index features,
$\hat{x}_{t+1}=f_{\theta}(t+1)$. Historical-value-based models have been
widely explored due to their simplicity and effectiveness, making them the
state-of-the-art models in the field. However, it is important to acknowledge
that multivariate time series data are often discretely sampled from
continuous dynamical systems. This characteristic poses a challenge for
historical-value-based models, as they tend to capture spurious correlations
limited to the sampling frequency (Gong et al., 2017; Woo et al., 2023).
Moreover, despite their widespread use and effectiveness, the progress in
developing historical-value-based models has encountered bottlenecks (Li et
al., 2023), making it challenging to further improve their performance. This
limitation motivates the exploration of alternative approaches that can
overcome the shortcomings of historical-value-based models and potentially
deliver more robust forecasting models.

In recent years, deep time-index-based models have gradually been studied (Woo
et al., 2023; Naour et al., 2023). These models intrinsically avoid the
aforementioned problem of historical-value-based models, by mapping the time-
index features to target predictions in continuous space through implicit
neural representations (Tancik et al., 2020; Sitzmann et al., 2020) (INRs).
However, time-index-based models are only characterized by time-index
coordinates, which makes them ineffective in modeling complex time series data
and lacking in exploratory capabilities.

In this work, we propose a novel perspective by treating multivariate time
series as spatiotemporal data discretely sampled from a continuous dynamical
system, described by partial differential equations (PDEs) with fixed spatial
domains as described in Eq 3.

From the PDE perspective, we observe that historical-value-based and time-
index-based models can be considered as simplified versions of Eq 4, where
either the temporal or spatial information is removed.

By examining Eq 4, we can interpret historical-value-based models as implicit
inverse problems. These models infer spatial domain information, such as the
position and physical properties of sensors, from historical observations. By
utilizing the initial condition and spatial domain information, these models
predict future sequences. Mathematically, historical-value-based models can be
formulated as:

$\displaystyle\hat{x}_{\tau}=x_{\tau_{0}}+\int_{\tau_{0}}^{\tau}f_{\theta}(x)d\tau$
(1)

where $x$ represents the spatial domain information.

Time-index-based models, on the other hand, can be seen as models that
consider only the temporal domain. These models solely rely on time-index
coordinates for prediction and do not explicitly incorporate spatial
information. In this case, the prediction is formulated as:

$\displaystyle\hat{x}_{\tau}=\int_{\tau_{0}}^{\tau}f_{\theta}(t_{\tau})d\tau$
(2)

Motivated by the limitations of existing approaches and inspired by neural PDE
solvers, we introduce PDETime, a PDE-based model for LMTF. PDETime considers
LMTF as an Initial Value Problem, explicitly incorporating both spatial and
temporal domains and requiring the specification of an initial condition.

Specifically, PDETime first predicts the partial derivative term
$E_{\theta}(x_{his},t_{\tau})=\alpha_{\tau}\approx\frac{\partial
u(x,t_{\tau})}{\partial t_{\tau}}$ utilizing an encoder in latent space.
Unlike traditional PDE problems, the spatial domain information of LMT is
unknown. Therefore, the encoder utilizes historical observations to estimate
the spatial domain information. Subsequently, a numerical solver is employed
to calculate the integral term
$z_{\tau}=\int_{\tau_{0}}^{\tau}\alpha_{\tau}d\tau$. The proposed solver
effectively mitigates the error issue and enhances the stability of the
prediction results compared to traditional ODE solvers (Chen et al., 2018).
Finally, PDETime employs a decoder to map the integral term from the latent
space to the spatial domains and predict the results as
$\hat{x}_{\tau}=x_{\tau_{0}}+D_{\phi}(z_{\tau})$. Similar to time-index-based
models, PDETime also utilizes meta-optimization to enhance its ability to
extrapolate across the forecast horizon. Additionally, PDETime can be
simplified into a historical-value-based or time-index-based model by
discarding either the time or spatial domains, respectively.

In short, we summarize the key contributions of this work:

  * •

We propose a novel perspective for LMTF by considering it as spatiotemporal
data regularly sampled from PDEs along the temporal domains.

  * •

We introduce PDETime, a PDE-based model inspired by neural PDE solvers, which
tackles LMTF as an Initial Value Problem of PDEs. PDETime incorporates
encoding-integration-decoding operations and leverages meta-optimization to
extrapolate future series.

  * •

We extensively evaluate the proposed model on seven real-world benchmarks
across multiple domains under the long-term setting. Our empirical studies
demonstrate that PDETime consistently achieves state-of-the-art performance.

##  2 Related Work

Multivariate Time Series Forecasting. With the progressive breakthrough made
in deep learning, deep models have been proposed to tackle various time series
forecasting applications. Depending on whether temporal or spatial is
utilized, these models are classified into historical-value-based (Zhou et
al., 2021, 2022; Zeng et al., 2023; Nie et al., 2023; Zhang & Yan, 2023; Liu
et al., 2024, 2022b, 2022a; Wu et al., 2023), and time-index-based models (Woo
et al., 2023). Historical-value-based models, predicting target time steps
utilizing historical observations, have been extensively developed and made
significant progress in which a large body of work that tries to apply
Transformer to forecast long-term series in recent years (Wen et al., 2023).
Early works like Informer (Zhou et al., 2021) and LongTrans (Li et al., 2019)
were focused on designing novel mechanism to reduce the complexity of the
original attention mechanism, thus capturing long-term dependency to achieve
better performance. Afterwards, efforts were made to extract better temporal
features to enhance the performance of the model (Wu et al., 2021; Zhou et
al., 2022). Recent work (Zeng et al., 2023) has found that a single linear
channel-independent model can outperform complex transformer-based models.
Therefore, the very recent channel-independent models like PatchTST (Nie et
al., 2023) and DLinear (Zeng et al., 2023) have become state-of-the-art. In
contrast, time-index-based models (Woo et al., 2023; Fons et al., 2022; Jiang
et al., 2023; Naour et al., 2023) are a kind of coordinated-based models,
mapping coordinates to values, which was represented by INRs. These models
have received less attention and their performance still lags behind
historical-value-based models. PDETime, unlike previous works, considers
multivariate time series as spatiotemporal data and approaches the prediction
target sequences from the perspective of partial differential equations
(PDEs).

Implicit Neural Representations. Implicit Neural Representations are the class
of works representing signals as a continuous function parameterized by multi-
layer perceptions (MLPs) (Tancik et al., 2020; Sitzmann et al., 2020) (instead
of using the traditional discrete representation). These neural networks have
been used to learn differentiable representations of various objects such as
images (Henzler et al., 2020), shapes (Liu et al., 2020, 2019), and textures
(Oechsle et al., 2019). However, there is limited research on INRs for times
series (Fons et al., 2022; Jiang et al., 2023; Woo et al., 2023; Naour et al.,
2023; Jeong & Shin, 2022). And previous works mainly focused on time series
generation and anomaly detection (Fons et al., 2022; Jeong & Shin, 2022).
DeepTime (Woo et al., 2023) is the work designed to learn a set of basis INR
functions for forecasting, however, its performance is worse than historical-
value-based models. In this work, we use INRs to represent spatial domains and
temporal domains.

Neural PDE Solvers. Neural PDE solvers which are used for temporal PDEs, are
laying the foundations of what is becoming both a rapidly growing and
impactful area of research. These neural PDE solvers fall into two broad
categories, neural operator methods and autoregressive methods. The neural
operator methods (Kovachki et al., 2021; Li et al., 2020; Lu et al., 2021)
treat the mapping from initial conditions to solutions as time $t$ as an
input-output mapping learnable via supervised learning. For a given PDE and
given initial conditions $u_{0}$, the neural operator $\mathcal{M}$ is trained
to satisfy $\mathcal{M}(t,u_{0})=u(t)$. However, these methods are not
designed to generalize to dynamics for out-of-distribution $t$. In contrast,
the autoregressive methods (Bar-Sinai et al., 2019; Greenfeld et al., 2019;
Hsieh et al., 2019; Yin et al., 2022; Brandstetter et al., 2021) solve the
PDEs iteratively. The solution of autoregressive methods at time $t+\Delta t$
as $u(t+\Delta)=\mathcal{A}(u(t),\Delta t)$. In this work, We consider
multivariate time series as data sampled from a continuous dynamical system
according to a regular time discretization, which can be described by partial
differential equations. For the given initial condition $x_{\tau_{0}}$, we use
the numerous solvers (Euler solver) to simulate target time step $x_{\tau}$.

##  3 Method

Figure 1: The framework of proposed PDETime which consists of an Encoder
$E_{\theta}$, a Solver, and a Decoder $D_{\phi}$. Given the initial condition
$x_{\tau_{0}}$, PDETime first simulates $\frac{\partial
u(x,t_{\tau})}{\partial t_{\tau}}$ at each time step $t_{\tau}$ using the
Encoder $E_{\theta}(x_{his},t_{\tau},\tau)$; then uses the Solver to compute
$\int_{\tau_{0}}^{\tau}\frac{\partial u(x,t_{\tau})}{\partial
t_{\tau}}dt_{\tau}$, which is a numerical solver; finally, the Decoder maps
integral term $z_{\tau}$ from latent space to special domains and predict the
final results $\hat{x}_{\tau}=x_{\tau_{0}}+D_{\phi}(z_{\tau})$.

###  3.1 Problem Formulation

In contrast to previous works (Zhou et al., 2021; Woo et al., 2023), we regard
multivariate time series as the spatiotemporal data regularly sampled from
partial differential equations along the temporal domain, which has the
following form:

$\displaystyle\mathcal{F}(u,\frac{\partial u}{\partial t},\frac{\partial
u}{\partial x^{1}},...,\frac{\partial^{2}u}{\partial
t^{2}},\frac{\partial^{2}u}{\partial x^{2}},...)=0,$ $\displaystyle
u(x,t):\Omega\times\mathcal{T}\rightarrow\mathcal{V},$ (3)

subject to initial and boundary conditions. Here $u$ is a spatio-temporal
dependent and multi-dimensional continuous vector field,
$\Omega\in\mathbb{R}^{C}$ and $\mathcal{T}\in\mathbb{R}$ denote the spatial
and temporal domains, respectively. For multivariate time series data, we
regard channel attributes as spatial domains (e.g., the position and physical
properties of sensors). The value of the spatial domains, $x$, is fixed but
unknown (e.g., the position of sensors is fixed but unknown), while the value
of the temporal domains is known. We treat LMTF as the Initial Value Problems,
where $u(x_{t},t)\in\mathbb{R}^{C}$ at any time $t$ is required to infer
$u(x_{t^{\prime}},t^{\prime})\in\mathbb{R}^{C}$.

Consequently, we forecast the target series by the following formula:

$u(x,t)=u(x,t_{0})+\int_{t_{0}}^{t}\frac{\partial
u(x,\tau)}{\partial\tau}d\tau.$ (4)

DETime initiates its process with a single initial condition, denoted as
$u(x,t_{0})$, and leverages neural networks to project the system’s dynamics
forward in time. The procedure unfolds in three distinct steps. Firstly,
PDETime generates a latent vector, $\alpha_{\tau}$ of a predefined dimension
$d$, utilizing an encoder function,
$E_{\theta}:\Omega\times\mathcal{T}\rightarrow\mathbb{R}^{d}$ (denoted as the
ENC step). Subsequently, it employs an Euler solver, a numerical method, to
approximate the integral term, $z_{t}=\int_{t_{0}}^{t}\alpha_{\tau}d\tau$,
effectively capturing the system’s evolution over time (denoted as the SOL
step). In the final step, PDETime translates the latent vectors, $z_{t}$, back
into the spatial domain using a decoder,
$D_{\phi}:\mathbb{R}^{d}\rightarrow\mathcal{V}$ to reconstruct the spatial
function (denoted as the DEC step). This results in the following model,
illustrated in Figure 1,

$\displaystyle(ENC)\,\,\alpha_{\tau}$
$\displaystyle=E_{\theta}(x_{his},t_{\tau},\tau)$ (5)
$\displaystyle(SOL)\,\,\,\,z_{\tau}$
$\displaystyle=\int_{\tau_{0}}^{\tau}\alpha_{\tau}d\tau$ (6)
$\displaystyle(DEC)\,\,x_{\tau}$ $\displaystyle=D_{\phi}(z_{\tau}).$ (7)

We describe the details of the components in Section 3.2.

###  3.2 Components of PDETime

####  3.2.1 Encoder: $\alpha_{\tau}=E_{\theta}(x_{his},t_{\tau},\tau)$

The Encoder computes the latent vector $\alpha_{\tau}$ representing
$\frac{\partial u(x,t_{\tau})}{\partial t_{\tau}}$. In contrast to previous
works (Chen et al., 2022; Yin et al., 2022) that compute $\frac{\partial
u(x,t_{\tau})}{\partial t_{\tau}}$ using auto-differentiation powered by deep
learning frameworks, which often suffers from poor stability due to
uncontrolled temporal discretizations, we propose directly simulating
$\frac{\partial u(x,t_{\tau})}{\partial t_{\tau}}$ by a carefully designed
neural networks $E_{\theta}(x_{his},t_{\tau},\tau)$. The overall architecture
of $E_{\theta}(x_{his},t_{\tau},\tau)$ is shown in Figure 2. As mentioned in
Section 3, since the spatial domains of MTS data are fixed and unknown, we can
only utilize the historical data $x_{his}\in\mathbb{R}^{H\times C}$ to assist
the neural network in fitting the value of spatial domains $x$. To represent
the high frequency of $\tau$, $x_{his}$, and $t_{\tau}$, we use concatenated
Fourier Features (CFF) (Woo et al., 2023; Tancik et al., 2020) and SIREN
(Sitzmann et al., 2020) with $k$ layers.

$\displaystyle\tau^{(i)}$
$\displaystyle=\text{GELU}(W_{\tau}^{(i-1)}\tau^{(i-1)}+b_{\tau}^{(i-1)})$
$\displaystyle x_{\tau}^{(i)}$
$\displaystyle=\sin(W_{x}^{(i-1)}x_{\tau}^{(i-1)}+b_{x}^{(i-1)})$
$\displaystyle t_{\tau}^{(i)}$
$\displaystyle=\sin(W_{t}^{(i-1)}t_{\tau}^{(i-1)}+b_{t}^{(i-1)}),\,\,\,i=1,...,k$
(8)

where $x_{\tau}^{(0)}\in\mathbb{R}^{H\times C}=x_{his}$,
$t_{\tau}^{(0)}\in\mathbb{R}^{t}$ is the temporal feature, and
$\tau\in\mathbb{R}$ is the index feature where $\tau_{i}=\frac{i}{H+L}$ for
$i=0,1,...,H+L$, $L$ and $H$ are the look-back and horizon length,
respectively. We employ CFF to represent $\tau^{(0)}$, i.e.
$\tau^{(0)}=[\sin(2\pi\textbf{B}_{1}\tau),\cos(2\pi\textbf{B}_{1}\tau),...,\sin(2\pi\textbf{B}_{s}\tau),\cos(2\pi\textbf{B}_{s}\tau)]$
where $\textbf{B}_{s}\in\mathbb{R}^{\frac{d}{2}\times c}$ is sampled from
$\mathcal{N}(0,2^{s})$.

After representing $\tau\in\mathbb{R}^{d}$, $t_{\tau}\in\mathbb{R}^{t}$, and
$x_{\tau}\in\mathbb{R}^{d\times C}$ with INRs, the Encoder aggregates
$x_{\tau}$ and $t_{\tau}$ using $\tau$, formulated as
$f_{\theta}(\tau,t_{\tau},x_{\tau})$:

$\displaystyle\tau$ $\displaystyle=\tau+\text{Attn}(\tau,x_{\tau})$
$\displaystyle\tau$ $\displaystyle=\tau+\text{Linear}([\tau;t_{\tau}])$ (9)

where $[\cdot;\cdot]$ is is the row-wise stacking operation. In this work, we
utilize attention mechanism (Vaswani et al., 2017) and linear mapping to
aggregate the information of spatial domains and temporal domains,
respectively. However, various networks can be designed to aggregate these two
types of information.

Unlike previous works (Chen et al., 2022; Yin et al., 2022), we do not
directly make $\alpha_{\tau}=\frac{\partial u(x,t_{\tau})}{\partial
t_{\tau}}$, but rather make
$\int_{\tau_{0}}^{\tau}\alpha_{\tau}d\tau=\int_{\tau_{0}}^{\tau}\frac{\partial
u(x,t_{\tau})}{\partial t_{\tau}}d\tau$ in latent space, without the need for
autoregressive calculation of $\alpha_{t}$ which can effectively alleviate the
error accumulation problem and make the prediction results more stable
(Brandstetter et al., 2021).

The pseudocode of the Encoder of PDETime is summarized in Appendix A.3.

Figure 2: The architecture of the Encoder which is used to simulate
$\frac{\partial u(x,t_{\tau})}{\partial t_{\tau}}$. In this work, we use INRs
to represent $x_{his}$, $t_{\tau}$, and $\tau$, and then use the Attention
mechanism and linear mapping to aggregate the information of spatial domains
and temporal domains.

####  3.2.2 Solver: $z_{t}=\int_{\tau_{0}}^{\tau}\alpha_{\tau}$

The Solver introduces a numerical solver (Euler Solver) to compute the
integral term $z_{\tau}=\int_{\tau_{0}}^{\tau}\alpha_{\tau}d\tau$, which can
be approximated as:

$\displaystyle z_{\tau}$ $\displaystyle=\int_{\tau_{0}}^{\tau}\frac{\partial
u(x_{\tau},t_{\tau})}{\partial
t_{\tau}}d\tau\approx\sum_{\tau=\tau_{0}}^{\tau}\frac{\partial
u(x_{\tau},t_{\tau})}{\partial t_{\tau}}\ast\Delta\tau$
$\displaystyle\approx\sum_{\tau=\tau_{0}}^{\tau}E_{\theta}(x_{his},t_{\tau},\tau)\ast\Delta\tau$
(10)

where we set $\Delta\tau=1$ for convenience. However, as shown in Figure 3
(a), directly compute
$z_{\tau}=\sum_{\tau=\tau_{0}}^{\tau}E_{\theta}(x_{his},t_{\tau},\tau)\ast\Delta\tau=\sum_{\tau=\tau_{0}}^{\tau}\alpha_{\tau}\ast\Delta\tau$
through Eq 10 can easily lead to error accumulation and gradient problems
(Rubanova et al., 2019; Wu et al., 2022; Brandstetter et al., 2021). To
address these issues, we propose a modified approach, as illustrated in Figure
3 (b). We divide the sequence into non-overlapping patches of length $S$,
where $\frac{H+L}{S}$ patches are obtained. For $\tau(H+L)\mod S=0$, we
directly estimate the integral term as
$z_{\tau}=f_{\psi}(x_{his},t_{\tau},\tau)$ using a neural network $f_{\psi}$.
Otherwise, we use the numerical solver to estimate the integral term with the
integration lower limit $\lfloor\frac{\tau\cdot(H+L)}{S}\rfloor\cdot S$. This
modification results in the following formula for the numerical solver:

$\displaystyle z_{t}$
$\displaystyle=f_{\psi}(x_{\tau^{\prime}},t_{\tau^{\prime}},\tau^{\prime})+\int_{\tau^{\prime}}^{\tau}f_{\theta}(x_{\tau},t_{\tau},\tau)d\tau,$
$\displaystyle\tau^{\prime}$
$\displaystyle=\lfloor\frac{\tau*(H+L)}{s}\rfloor\ast S$ (11)

Furthermore, as shown in Figure 3 (b), Eq 11 breaks the continuity. To address
this, we introduce an additional objective function to ensure continuity as
much as possible:

$\displaystyle\mathcal{L}_{c}=\mathcal{L}(f_{\psi}(x_{\tau},t_{\tau},\tau),f_{\psi}(x_{\tau^{\prime}},t_{\tau^{\prime}},\tau^{\prime})+\int_{\tau^{\prime}}^{\tau}f_{\theta}(x_{\tau},t_{\tau},\tau)d\tau)$
$\displaystyle\mathrm{s.t.}\;\tau\ast(H+L)\mod
S=0,\;\tau^{\prime}=\frac{\tau\ast(L+H)-S}{(H+L)}.$ (12)

The experimental results have shown that incorporating $\mathcal{L}_{c}$
improves the stability of PDETime.

The pseudocode of the Solver of PDETime is summarized in Appendix A.4.

Figure 3: The framework of standard and variant of numerical solver proposed
in our work.

####  3.2.3 Decoder: $x_{\tau}=D_{\phi}(z_{\tau})+x_{\tau_{0}}$

After estimating the integral term in latent space, we utilize a Decoder
$D_{\phi}$ to decode $z_{\tau}$ into the spatial function. To tackle the
distribution shift problem, we introduce a meta-optimization framework (Woo et
al., 2023; Bertinetto et al., 2018) to enhance the extrapolation capability of
PDETime. Specifically, we split the time series into look-back window
$\textbf{X}_{t-L:t}=[x_{t-L},...,x_{t}]\in\mathbb{R}^{L\times C}$ and horizon
window $\textbf{Y}_{t+1:t+H}=[x_{t+1},...,x_{t+H}]\in\mathbb{R}^{H\times C}$
pairs. We then use the parameters $\phi$ and $\theta$ to adapt the look-back
window and horizon window through a bi-level problem:

$\displaystyle\theta^{\ast}=\arg\min_{\theta}\sum_{\tau=\tau_{t+1}}^{\tau_{t+H}}\mathcal{L}_{p}(D_{\phi}(\int_{\tau_{0}}^{\tau}E_{\theta}(x_{his},t_{\tau},\tau)),x_{\tau})$
$\displaystyle\phi^{\ast}=\arg\min_{\phi}\sum_{\tau=\tau_{t-L}}^{\tau_{t}}\mathcal{L}_{r}(D_{\phi}(\int_{\tau_{0}}^{\tau}E_{\theta}(x_{his},t_{\tau},\tau)),x_{\tau})$
(13)

where $\mathcal{L}_{r}$ and $\mathcal{L}_{p}$ denote the reconstruction and
prediction loss, respectively (which will be described in detail in Section
3.3). During training, PDETime optimizes both $\theta$ and $\phi$; while
during testing, it only optimizes $\phi$ to enhance the extrapolation. To
ensure speed and efficiency, we employ a single ridge regression for
$D_{\phi}$ (Bertinetto et al., 2018).

###  3.3 Optimization

Different from previous works (Zhou et al., 2021, 2022; Wu et al., 2021; Woo
et al., 2023) that directly predict the future time steps with the objective
$\mathcal{L}_{p}=\frac{1}{H}\mathcal{L}(x_{\tau},\hat{x}_{\tau})$. As
described as Eq 4, we consider the LMTF as the Initial Value Problem in PDE,
and the initial condition $x_{\tau_{0}}$is known (we use the latest time step
in the historical series as the initial condition). Thus the objective of
PDETime is to estimate the integral term $\int_{\tau_{0}}^{\tau}\frac{\partial
u(x,t_{\tau})}{\partial t_{\tau}}d\tau$. To achieve this, we define the
prediction objective as:

$\displaystyle\mathcal{L}_{p}=\frac{1}{H}\mathcal{L}(D_{\phi}(z_{\tau}),x_{\tau}-x_{\tau_{0}})$
(14)

where $z_{\tau}=\int_{\tau_{0}}^{\tau}E_{\theta}(x_{his},t_{\tau},\tau)d\tau$
as described in Section 3.2.1. Additionally, $\mathcal{L}_{r}$ has the same
form $\mathcal{L}_{r}=\frac{1}{L}(D_{\phi}(z_{\tau}),x_{\tau}-x_{\tau_{0}})$.

In addition to the prediction objective, we also aim to ensure that the first-
order difference of the predicted sequence matches that of the target
sequence. Therefore, we introduce the optimization objective:

$\displaystyle\mathcal{L}_{f}=\frac{1}{H}\sum_{\tau=\tau_{t+1}}^{\tau_{t+H}}\mathcal{L}(D_{\phi}(z_{\tau})-D_{\phi}(z_{\tau-1}),x_{\tau}-x_{\tau-1})$
(15)

Combining $\mathcal{L}_{p}$, $\mathcal{L}_{c}$, and $\mathcal{L}_{f}$, the
training objective is:

$\displaystyle\mathcal{L}_{p}=\mathcal{L}_{p}+\mathcal{L}_{c}+\mathcal{L}_{f}.$
(16)

##  4 Experiments

###  4.1 Experimental Settings

Datasets. We extensively include 7 real-world datasets in our experiments,
including four ETT datasets (ETTh1, ETTh2, ETTm1, ETTm2) (Zhou et al., 2021).
Electricity, Weather and Traffic (Wu et al., 2021), covering energy,
transportation and weather domains (See Appendix A.1.1 for more details on the
datasets). To ensure a fair evaluation, we follow the standard protocol of
dividing each dataset into the training, validation and testing subsets
according to the chronological order. The split ratio is 6:2:2 for the ETT
dataset and 7:1:2 for the others (Zhou et al., 2021; Wu et al., 2021). We set
the length of the lookback series as 512 for PatchTST, 336 for DLinear, and 96
for other historical-value-based models. The experimental settings of DeepTime
remain consistent with the original settings (Woo et al., 2023). The
prediction length varies in $\\{96,192,336,720\\}$.

Comparison methods. We carefully choose 10 well-acknowledged models (9
historical-value-based models and 1 time-index-based model) as our benchmarks,
including (1) Transformer-based models: Autoformer (Wu et al., 2021),
FEDformer (Zhou et al., 2022), Stationary (Liu et al., 2022b), Crossformer
(Zhang & Yan, 2023), PatchTST (Nie et al., 2023), and iTransformer (Liu et
al., 2024); (2) Linear-based models: DLinear (Zeng et al., 2023) ; (3) CNN-
based models: SCINet (Liu et al., 2022a), TimesNet (Wu et al., 2023); (4)
Time-index-based model: DeepTime (Woo et al., 2023). (See Appendix A.1.2 for
details of these baselines)

Implementation Details. Our method is trained with the Smooth L1 (Girshick,
2015) loss using the ADAM (Kingma & Ba, 2014) with the initial learning rate
selected from $\\{10^{-4},10^{-3},5\times 10^{-3}\\}$. Batch size is set to
32. All experiments are repeated 3 times with fixed seed 2024, implemented in
Pytorch (Paszke et al., 2019) and conducted on a single NVIDIA RTX 3090 GPUs.
Following DeepTime (Woo et al., 2023), we set the look-back length as
$L=\mu\ast H$, where $\mu$ is a multiplier which decides the length of the
look-back windows. We search through the values $\mu=[1,3,5,7,9]$, and select
the best value based on the validation loss. We set layers of INRs $k=5$ by
default, and select the best results from $N=\\{1,2,3,5\\}$. We summarize the
temporal features used in this work in Appendix A.2.

###  4.2 Main Results and Ablation Study

Table 1: Full results of the long-term forecasting task. We compare extensive competitive models under different prediction lengths following the setting of PatchTST (2023). The input sequence length is set to 336 and 512 for DLinear and PatchTST, and 96 for other historical-value-based baselines. _Avg_ means the average results from all four prediction lengths. Models |  PDETime |  iTransformer |  PatchTST |  Crossformer |  DeepTime |  TimesNet |  DLinear |  SCINet |  FEDformer |  Stationary |  Autoformer
---|---|---|---|---|---|---|---|---|---|---|---
(Ours) |  (2024) |  (2023) |  (2023) |  (2023) |  (2023) |  (2023) |  (2022a) |  (2022) |  (2022b) |  (2021)
Metric |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE
ETTm1 |  96 |  0.292 |  0.335 |  0.334 |  0.368 |  0.293 |  0.346 |  0.404 |  0.426 |  0.305 |  0.347 |  0.338 |  0.375 |  0.299 |  0.343 |  0.418 |  0.438 |  0.379 |  0.419 |  0.386 |  0.398 |  0.505 |  0.475
192 |  0.329 |  0.359 |  0.377 |  0.391 |  0.333 |  0.370 |  0.450 |  0.451 |  0.340 |  0.371 |  0.374 |  0.387 |  0.335 |  0.365 |  0.439 |  0.450 |  0.426 |  0.441 |  0.459 |  0.444 |  0.553 |  0.496
336 |  0.346 |  0.374 |  0.426 |  0.420 |  0.369 |  0.392 |  0.532 |  0.515 |  0.362 |  0.387 |  0.410 |  0.411 |  0.369 |  0.386 |  0.490 |  0.485 |  0.445 |  0.459 |  0.495 |  0.464 |  0.621 |  0.537
720 |  0.395 |  0.404 |  0.491 |  0.459 |  0.416 |  0.420 |  0.666 |  0.589 |  0.399 |  0.414 |  0.478 |  0.450 |  0.425 |  0.421 |  0.595 |  0.550 |  0.543 |  0.490 |  0.585 |  0.516 |  0.671 |  0.561
Avg |  0.340 |  0.368 |  0.407 |  0.410 |  0.352 |  0.382 |  0.513 |  0.496 |  0.351 |  0.379 |  0.400 |  0.406 |  0.357 |  0.378 |  0.485 |  0.481 |  0.448 |  0.452 |  0.481 |  0.456 |  0.588 |  0.517
ETTm2 |  96 |  0.158 |  0.244 |  0.180 |  0.264 |  0.166 |  0.256 |  0.287 |  0.366 |  0.166 |  0.257 |  0.187 |  0.267 |  0.167 |  0.260 |  0.286 |  0.377 |  0.203 |  0.287 |  0.192 |  0.274 |  0.255 |  0.339
192 |  0.213 |  0.283 |  0.250 |  0.309 |  0.223 |  0.296 |  0.414 |  0.492 |  0.225 |  0.302 |  0.249 |  0.309 |  0.224 |  0.303 |  0.399 |  0.445 |  0.269 |  0.328 |  0.280 |  0.339 |  0.281 |  0.340
336 |  0.262 |  0.318 |  0.311 |  0.348 |  0.274 |  0.329 |  0.597 |  0.542 |  0.277 |  0.336 |  0.321 |  0.351 |  0.281 |  0.342 |  0.637 |  0.591 |  0.325 |  0.366 |  0.334 |  0.361 |  0.339 |  0.372
720 |  0.334 |  0.336 |  0.412 |  0.407 |  0.362 |  0.385 |  1.730 |  1.042 |  0.383 |  0.409 |  0.408 |  0.403 |  0.397 |  0.421 |  0.960 |  0.735 |  0.421 |  0.415 |  0.417 |  0.413 |  0.433 |  0.432
Avg |  0.241 |  0.295 |  0.288 |  0.332 |  0.256 |  0.316 |  0.757 |  0.610 |  0.262 |  0.326 |  0.291 |  0.333 |  0.267 |  0.331 |  0.571 |  0.537 |  0.305 |  0.349 |  0.306 |  0.347 |  0.327 |  0.371
ETTh1 |  96 |  0.356 |  0.381 |  0.386 |  0.405 |  0.379 |  0.401 |  0.423 |  0.448 |  0.371 |  0.396 |  0.384 |  0.402 |  0.375 |  0.399 |  0.654 |  0.599 |  0.376 |  0.419 |  0.513 |  0.491 |  0.449 |  0.459
192 |  0.397 |  0.406 |  0.441 |  0.436 |  0.413 |  0.429 |  0.471 |  0.474 |  0.403 |  0.420 |  0.436 |  0.429 |  0.405 |  0.416 |  0.719 |  0.631 |  0.420 |  0.448 |  0.534 |  0.504 |  0.500 |  0.482
336 |  0.420 |  0.419 |  0.487 |  0.458 |  0.435 |  0.436 |  0.570 |  0.546 |  0.433 |  0.436 |  0.491 |  0.469 |  0.439 |  0.443 |  0.778 |  0.659 |  0.459 |  0.465 |  0.588 |  0.535 |  0.521 |  0.496
720 |  0.425 |  0.446 |  0.503 |  0.491 |  0.446 |  0.464 |  0.653 |  0.621 |  0.474 |  0.492 |  0.521 |  0.500 |  0.472 |  0.490 |  0.836 |  0.699 |  0.506 |  0.507 |  0.643 |  0.616 |  0.514 |  0.512
Avg |  0.399 |  0.413 |  0.454 |  0.447 |  0.418 |  0.432 |  0.529 |  0.522 |  0.420 |  0.436 |  0.458 |  0.450 |  0.423 |  0.437 |  0.747 |  0.647 |  0.440 |  0.460 |  0.570 |  0.537 |  0.496 |  0.487
ETTh2 |  96 |  0.268 |  0.330 |  0.297 |  0.349 |  0.274 |  0.335 |  0.745 |  0.584 |  0.287 |  0.352 |  0.340 |  0.374 |  0.289 |  0.353 |  0.707 |  0.621 |  0.358 |  0.397 |  0.476 |  0.458 |  0.346 |  0.388
192 |  0.331 |  0.370 |  0.380 |  0.400 |  0.342 |  0.382 |  0.877 |  0.656 |  0.383 |  0.412 |  0.402 |  0.414 |  0.383 |  0.418 |  0.860 |  0.689 |  0.429 |  0.439 |  0.512 |  0.493 |  0.456 |  0.452
336 |  0.358 |  0.395 |  0.428 |  0.432 |  0.365 |  0.404 |  1.043 |  0.731 |  0.523 |  0.501 |  0.452 |  0.452 |  0.448 |  0.465 |  1.000 |  0.744 |  0.496 |  0.487 |  0.552 |  0.551 |  0.482 |  0.486
720 |  0.380 |  0.421 |  0.427 |  0.445 |  0.393 |  0.430 |  1.104 |  0.763 |  0.765 |  0.624 |  0.462 |  0.468 |  0.605 |  0.551 |  1.249 |  0.838 |  0.463 |  0.474 |  0.562 |  0.560 |  0.515 |  0.511
Avg |  0.334 |  0.379 |  0.383 |  0.407 |  0.343 |  0.387 |  0.942 |  0.684 |  0.489 |  0.472 |  0.414 |  0.427 |  0.431 |  0.446 |  0.954 |  0.723 |  0.437 |  0.449 |  0.526 |  0.516 |  0.450 |  0.459
ECL |  96 |  0.129 |  0.222 |  0.148 |  0.240 |  0.129 |  0.222 |  0.219 |  0.314 |  0.137 |  0.238 |  0.168 |  0.272 |  0.140 |  0.237 |  0.247 |  0.345 |  0.193 |  0.308 |  0.169 |  0.273 |  0.201 |  0.317
192 |  0.143 |  0.235 |  0.162 |  0.253 |  0.147 |  0.240 |  0.231 |  0.322 |  0.152 |  0.252 |  0.184 |  0.289 |  0.153 |  0.249 |  0.257 |  0.355 |  0.201 |  0.315 |  0.182 |  0.286 |  0.222 |  0.334
336 |  0.152 |  0.248 |  0.178 |  0.269 |  0.163 |  0.259 |  0.246 |  0.337 |  0.166 |  0.268 |  0.198 |  0.300 |  0.169 |  0.267 |  0.269 |  0.369 |  0.214 |  0.329 |  0.200 |  0.304 |  0.231 |  0.338
720 |  0.176 |  0.272 |  0.225 |  0.317 |  0.197 |  0.290 |  0.280 |  0.363 |  0.201 |  0.302 |  0.220 |  0.320 |  0.203 |  0.301 |  0.299 |  0.390 |  0.246 |  0.355 |  0.222 |  0.321 |  0.254 |  0.361
Avg |  0.150 |  0.244 |  0.178 |  0.270 |  0.159 |  0.252 |  0.244 |  0.334 |  0.164 |  0.265 |  0.192 |  0.295 |  0.166 |  0.263 |  0.268 |  0.365 |  0.214 |  0.327 |  0.193 |  0.296 |  0.227 |  0.338
Traffic |  96 |  0.330 |  0.232 |  0.395 |  0.268 |  0.360 |  0.249 |  0.522 |  0.290 |  0.390 |  0.275 |  0.593 |  0.321 |  0.410 |  0.282 |  0.788 |  0.499 |  0.587 |  0.366 |  0.612 |  0.338 |  0.613 |  0.388
192 |  0.332 |  0.232 |  0.417 |  0.276 |  0.379 |  0.256 |  0.530 |  0.293 |  0.402 |  0.278 |  0.617 |  0.336 |  0.423 |  0.287 |  0.789 |  0.505 |  0.604 |  0.373 |  0.613 |  0.340 |  0.616 |  0.382
336 |  0.342 |  0.236 |  0.433 |  0.283 |  0.392 |  0.264 |  0.558 |  0.305 |  0.415 |  0.288 |  0.629 |  0.336 |  0.436 |  0.296 |  0.797 |  0.508 |  0.621 |  0.383 |  0.618 |  0.328 |  0.622 |  0.337
720 |  0.365 |  0.244 |  0.467 |  0.302 |  0.432 |  0.286 |  0.589 |  0.328 |  0.449 |  0.307 |  0.640 |  0.350 |  0.466 |  0.315 |  0.841 |  0.523 |  0.626 |  0.382 |  0.653 |  0.355 |  0.660 |  0.408
Avg |  0.342 |  0.236 |  0.428 |  0.282 |  0.390 |  0.263 |  0.550 |  0.304 |  0.414 |  0.287 |  0.620 |  0.336 |  0.433 |  0.295 |  0.804 |  0.509 |  0.610 |  0.376 |  0.624 |  0.340 |  0.628 |  0.379
Weather |  96 |  0.157 |  0.203 |  0.174 |  0.214 |  0.149 |  0.198 |  0.158 |  0.230 |  0.166 |  0.221 |  0.172 |  0.220 |  0.176 |  0.237 |  0.221 |  0.306 |  0.217 |  0.296 |  0.173 |  0.223 |  0.266 |  0.336
192 |  0.200 |  0.246 |  0.221 |  0.254 |  0.194 |  0.241 |  0.206 |  0.277 |  0.207 |  0.261 |  0.219 |  0.261 |  0.220 |  0.282 |  0.261 |  0.340 |  0.276 |  0.336 |  0.245 |  0.285 |  0.307 |  0.367
336 |  0.241 |  0.281 |  0.278 |  0.296 |  0.245 |  0.282 |  0.272 |  0.335 |  0.251 |  0.298 |  0.280 |  0.306 |  0.265 |  0.319 |  0.309 |  0.378 |  0.339 |  0.380 |  0.321 |  0.338 |  0.359 |  0.395
720 |  0.291 |  0.324 |  0.358 |  0.349 |  0.314 |  0.334 |  0.398 |  0.418 |  0.301 |  0.338 |  0.365 |  0.359 |  0.323 |  0.362 |  0.377 |  0.427 |  0.403 |  0.428 |  0.414 |  0.410 |  0.419 |  0.428
Avg |  0.222 |  0.263 |  0.258 |  0.279 |  0.225 |  0.263 |  0.259 |  0.315 |  0.231 |  0.286 |  0.259 |  0.287 |  0.246 |  0.300 |  0.292 |  0.363 |  0.309 |  0.360 |  0.288 |  0.314 |  0.338 |  0.382
$1^{\text{st}}$ Count |  33 |  33 |  0 |  0 |  2 |  2 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0 |  0

Comprehensive forecasting results are listed in Table 1 with the best in Bold
and the second underlined. The lower MSE/MAE indicates the more accurate
prediction result. Overall, PDETime achieves the best performance on most
settings across seven real-world datasets compared with historical-value-based
and time-index-based models. Additionally, experimental results also show that
the performance of the proposed PDETime changes quite steadily as the
prediction length $H$ increases. For instance, the MSE of PDETime increases
from 0.330 to 0.365 on the Traffic dataset, while the MSE of PatchTST
increases from 0.360 to 0.432, which is the SOTA historical-value-based model.
This phenomenon was observed in other datasets and settings as well,
indicating that PDETime retains better long-term robustness, which is
meaningful for real-world practical applications.

Table 2: Ablation study on variants of PDETime. -Temporal refers that removing the temporal domain feature $t_{\tau}$; -Spatial refers that removing the historical observations $x_{his}$; - Initial refers that removing the initial condition $x_{\tau_{0}}$. The best results are highlighted in bold. Dataset |  Models |  PDETime |  -Temporal |  -Spatial |  -Initial |  -Temporal -Spatial |  \- All
---|---|---|---|---|---|---|---
Metric |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE
Traffic |  96 |  0.330 |  0.232 |  0.336 |  0.236 |  0.329 |  0.232 |  0.334 |  0.235 |  0.394 |  0.268 |  0.401 |  0.269
192 |  0.332 |  0.232 |  0.368 |  0.247 |  0.336 |  0.234 |  0.334 |  0.232 |  0.407 |  0.269 |  0.413 |  0.270
336 |  0.342 |  0.236 |  0.378 |  0.251 |  0.344 |  0.236 |  0.343 |  0.236 |  0.419 |  0.273 |  0.426 |  0.272
720 |  0.365 |  0.244 |  0.406 |  0.265 |  0.371 |  0.250 |  0.368 |  0.250 |  0.453 |  0.291 |  0.671 |  0.406
Weather |  96 |  0.157 |  0.203 |  0.158 |  0.205 |  0.159 |  0.205 |  0.169 |  0.213 |  0.159 |  0.205 |  0.166 |  0.212
192 |  0.200 |  0.246 |  0.206 |  0.253 |  0.198 |  0.243 |  0.208 |  0.248 |  0.198 |  0.243 |  0.208 |  0.250
336 |  0.241 |  0.281 |  0.240 |  0.278 |  0.246 |  0.282 |  0.245 |  0.287 |  0.240 |  0.277 |  0.244 |  0.283
720 |  0.291 |  0.324 |  0.292 |  0.323 |  0.290 |  0.322 |  0.300 |  0.337 |  0.294 |  0.327 |  0.299 |  0.337

We perform ablation studies on the Traffic and Weather datasets to validate
the effect of temporal feature $t_{\tau}$, spatial feature $x_{his}$ and
initial condition $x_{\tau_{0}}$. The results are presented in Table 2. 1) The
initial condition $x_{t_{0}}$ is useful on most settings. As mentioned in
Section 3, we treat LMTF as Initial Value Problem, thus the effectiveness of
$x_{\tau_{0}}$ validates the correctness of PDETime. 2) The impact of spatial
features $x_{his}$ on PDETime is limited. This may be due to the fact that the
true spatial domains $x$ are unknown, and we utilize the historical
observations $x_{his}$ to simulate $x$. However, this approach may not be
effective in capturing the true spatial features. 3) The influence of temporal
feature $t_{\tau}$ on PDETime various significantly across different datasets.
Experimental results have shown that $t_{\tau}$ is highly beneficial in the
Traffic dataset, but its effect on Weather dataset is limited. For example,
the period of Traffic dataset may be one day or one week, making it easier for
PDETime to learn temporal features. On the other hand, the period of Weather
dataset may be one year or longer, but the dataset only contains one year of
data. As a result, PDETime cannot capture the complete temporal features in
this case.

We conduct an additional ablation study on Traffic to evaluate the ability of
different INRs to extract features of $x_{his}$, $t_{\tau}$, and $\tau$. In
this study, we compared the performance of using the GELU or Tanh activation
function instead of sine in SIREN and making
$\tau^{(0)}=[\text{GELU}(2\pi\textbf{B}_{1}\tau),\text{GELU}(2\pi\textbf{B}_{1}\tau),...]$
or
$\tau^{(0)}=[\text{Tanh}(2\pi\textbf{B}_{1}\tau),\text{Tanh}(2\pi\textbf{B}_{1}\tau),...]$.
Table 4 presents the experimental results, we observe that the sine function
(periodic functions) can extract features better than other non-decreasing
activation functions. This is because the smooth, non-periodic activation
functions fail to accurately model high-frequency information (Sitzmann et
al., 2020). Time series data is often periodic, and the periodic nature of the
sine function makes it more effective in extracting time series features.

###  4.3 Effects of Hyper-parameters

Figure 4: Evaluation on hyper-parameter impact. (a) MSE against hyper-
parameter layers of INRs $k$ in Forecaster on ETTh1. (b) MSE against hyper-
parameter layers of aggregation module $N$ in Forecaster on ETTh1. (c) MSE
against hyper-parameter patch length $S$ in Estimator on ETTh1.

We evaluate the effect of four hyper-parameters: look-back window $L$, number
of INRs layers $k$, number of aggregation layers $N$, and patch length $S$ on
the ETTh1 and ETTh2 datasets. First, we perform a sensitivity on the look-back
window $L=\mu\ast H$, where $H$ is the experimental setting. The results are
presented in Table 3. We observe that the test error decreases as $\mu$
increases, plateauing and even increasing slightly as $\mu$ grows extremely
large when the horizon window is small. However, under a large horizon window,
the test error increases as $\mu$ increases. Next, we evaluate the hyper-
parameters $N$ and $k$ on PDETime, as shown in Figure 4 (a) and (b)
respectively. We find that the performance of PDETime remains stable when
$k\geq 3$. Additionally, the number of aggregation layers $N$ has a limited
impact on PDETime. Furthermore, we investigate the effect of patch length $S$
on PDETime, as illustrated in Figure 4 (c). We varied the patch length from 2
to 48 and evaluate MSE with different horizon windows. As the patch length $S$
increased, the prediction accuracy of PDETime initially improved, reached a
peak, and then started to decline. However, the accuracy remains relatively
stable throughout. We also extended the patch length to $S=H$. In this case,
PDETime performed poorly, indicating that the accumulation of errors has a
significant impact on the performance of PDETime. Overall, these analyses
provide insights into the effects of different hyper-parameters on the
performance of PDETime and can guide the selection of appropriate settings for
achieving optimal results.

Due to space limitations, we put extra experiments on robustness and
visualization experiments in Appendix A.5 and Appendix A.6, respectively.

Table 3: Analysis on the look-back window length, based on the multiplier on horizon length, $L=\mu\ast H$. The best results are highlighted in bold. Dataset |  Horizon |  96 |  192 |  336 |  720
---|---|---|---|---|---
$\mu$ |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE
ETTh1 |  1 |  0.378 |  0.386 |  0.415 |  0.411 |  0.421 |  0.420 |  0.425 |  0.446
3 |  0.359 |  0.382 |  0.394 |  0.404 |  0.427 |  0.421 |  0.443 |  0.460
5 |  0.360 |  0.385 |  0.396 |  0.405 |  0.421 |  0.420 |  0.495 |  0.501
7 |  0.354 |  0.381 |  0.398 |  0.405 |  0.427 |  0.429 |  0.545 |  0.532
9 |  0.356 |  0.381 |  0.397 |  0.406 |  0.446 |  0.440 |  1.220 |  0.882
ETTh2 |  1 |  0.288 |  0.335 |  0.357 |  0.381 |  0.380 |  0.404 |  0.380 |  0.421
3 |  0.276 |  0.331 |  0.339 |  0.374 |  0.358 |  0.395 |  0.422 |  0.456
5 |  0.275 |  0.333 |  0.331 |  0.370 |  0.360 |  0.408 |  0.622 |  0.576
7 |  0.268 |  0.330 |  0.331 |  0.378 |  0.384 |  0.427 |  0.624 |  0.595
9 |  0.272 |  0.331 |  0.331 |  0.378 |  0.412 |  0.451 |  0.797 |  0.689
Table 4: Analysis on INRs. PDETime refers to our proposed approach. GELU and Tanh refer to replacing SIREN and CFF with GELU or Tanh activation, respectively. The best results are highlighted in bold. Dataset |  Method |  PDETime |  GELU |  Tanh
---|---|---|---|---
Metric |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE
Traffic |  96 |  0.330 |  0.232 |  0.332 |  0.237 |  0.338 |  0.233
192 |  0.332 |  0.232 |  0.338 |  0.241 |  0.339 |  0.235
336 |  0.342 |  0.236 |  0.348 |  0.244 |  0.348 |  0.238
720 |  0.365 |  0.244 |  0.376 |  0.252 |  0.366 |  0.244

##  5 Conclusion and Future Work

In this paper, we propose a novel LMTS framework PDETime, based on neural PDE
solvers, which consists of Encoder, Solver, and Decoder. Specifically, the
Encoder simulates the partial differentiation of the original function over
time at each time step in latent space in parallel. The solver is responsible
for computing the integral term with improved stability. Finally, the Decoder
maps the integral term from latent space into the spatial function and
predicts the target series given the initial condition. Additionally, we
incorporate meta-optimization techniques to enhance the ability of PDETime to
extrapolate future series. Extensive experimental results show that PDETime
achieves state-of-the-art performance across forecasting benchmarks on various
real-world datasets. We also perform ablation studies to identify the key
components contributing to the success of PDETime.

Future Work Firstly, while our proposed neural PDE solver, PDETime, has shown
promising results for long-term multivariate time series forecasting, there
are other types of neural PDE solvers that could potentially be applied to
this task. Exploring these alternative neural PDE solvers and comparing their
performance on LMTF could be an interesting future direction. Furthermore, our
ablation studies have revealed that the effectiveness of the historical
observations (spatial domain) on PDETime is limited, and the impact of the
temporal domain varies significantly across different datasets. Designing
effective network structures that better utilize the spatial and temporal
domains could be an important area for future research. Finally, our approach
of rethinking long-term multivariate time series forecasting from the
perspective of partial differential equations has led to state-of-the-art
performance. Exploring other perspectives and frameworks to tackle this task
could be a promising direction for future research.

## Broader Impacts

Our research is dedicated to innovating time series forecasting techniques to
push the boundaries of time series analysis further. While our primary
objective is to enhance predictive accuracy and efficiency, we are also
mindful of the broader ethical considerations that accompany technological
progress in this area. While immediate societal impacts may not be apparent,
we acknowledge the importance of ongoing vigilance regarding the ethical use
of these advancements. It is essential to continuously assess and address
potential implications to ensure responsible development and application in
various sectors.

## References

  * Angryk et al. (2020) Angryk, R. A., Martens, P. C., Aydin, B., Kempton, D., Mahajan, S. S., Basodi, S., Ahmadzadeh, A., Cai, X., Filali Boubrahimi, S., Hamdi, S. M., et al.  Multivariate time series dataset for space weather data analytics.  _Scientific data_ , 7(1):227, 2020.
  * Bar-Sinai et al. (2019) Bar-Sinai, Y., Hoyer, S., Hickey, J., and Brenner, M. P.  Learning data-driven discretizations for partial differential equations.  _Proceedings of the National Academy of Sciences_ , 116(31):15344–15349, 2019.
  * Bertinetto et al. (2018) Bertinetto, L., Henriques, J. F., Torr, P. H., and Vedaldi, A.  Meta-learning with differentiable closed-form solvers.  _arXiv preprint arXiv:1805.08136_ , 2018.
  * Brandstetter et al. (2021) Brandstetter, J., Worrall, D. E., and Welling, M.  Message passing neural pde solvers.  In _International Conference on Learning Representations_ , 2021.
  * Chen et al. (2022) Chen, P. Y., Xiang, J., Cho, D. H., Chang, Y., Pershing, G., Maia, H. T., Chiaramonte, M., Carlberg, K., and Grinspun, E.  Crom: Continuous reduced-order modeling of pdes using implicit neural representations.  _arXiv preprint arXiv:2206.02607_ , 2022.
  * Chen et al. (2018) Chen, R. T., Rubanova, Y., Bettencourt, J., and Duvenaud, D. K.  Neural ordinary differential equations.  _Advances in neural information processing systems_ , 31, 2018.
  * Demirel et al. (2012) Demirel, Ö. F., Zaim, S., Çalişkan, A., and Özuyar, P.  Forecasting natural gas consumption in istanbul using neural networks and multivariate time series methods.  _Turkish Journal of Electrical Engineering and Computer Sciences_ , 20(5):695–711, 2012.
  * Fons et al. (2022) Fons, E., Sztrajman, A., El-Laham, Y., Iosifidis, A., and Vyetrenko, S.  Hypertime: Implicit neural representation for time series.  _arXiv preprint arXiv:2208.05836_ , 2022.
  * Girshick (2015) Girshick, R.  Fast r-cnn.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 1440–1448, 2015.
  * Gong et al. (2017) Gong, M., Zhang, K., Schölkopf, B., Glymour, C., and Tao, D.  Causal discovery from temporally aggregated time series.  In _Uncertainty in artificial intelligence: proceedings of the… conference. Conference on Uncertainty in Artificial Intelligence_ , volume 2017. NIH Public Access, 2017.
  * Greenfeld et al. (2019) Greenfeld, D., Galun, M., Basri, R., Yavneh, I., and Kimmel, R.  Learning to optimize multigrid pde solvers.  In _International Conference on Machine Learning_ , pp. 2415–2423. PMLR, 2019.
  * Henzler et al. (2020) Henzler, P., Mitra, N. J., and Ritschel, T.  Learning a neural 3d texture space from 2d exemplars.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 8356–8364, 2020.
  * Hsieh et al. (2019) Hsieh, J.-T., Zhao, S., Eismann, S., Mirabella, L., and Ermon, S.  Learning neural pde solvers with convergence guarantees.  _arXiv preprint arXiv:1906.01200_ , 2019.
  * Jeong & Shin (2022) Jeong, K.-J. and Shin, Y.-M.  Time-series anomaly detection with implicit neural representation.  _arXiv preprint arXiv:2201.11950_ , 2022.
  * Jiang et al. (2023) Jiang, W., Boominathan, V., and Veeraraghavan, A.  Nert: Implicit neural representations for general unsupervised turbulence mitigation.  _arXiv preprint arXiv:2308.00622_ , 2023.
  * Kingma & Ba (2014) Kingma, D. P. and Ba, J.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kovachki et al. (2021) Kovachki, N., Li, Z., Liu, B., Azizzadenesheli, K., Bhattacharya, K., Stuart, A., and Anandkumar, A.  Neural operator: Learning maps between function spaces.  _arXiv preprint arXiv:2108.08481_ , 2021.
  * Li et al. (2019) Li, S., Jin, X., Xuan, Y., Zhou, X., Chen, W., Wang, Y.-X., and Yan, X.  Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting.  _Advances in neural information processing systems_ , 32, 2019.
  * Li et al. (2017) Li, Y., Yu, R., Shahabi, C., and Liu, Y.  Diffusion convolutional recurrent neural network: Data-driven traffic forecasting.  _arXiv preprint arXiv:1707.01926_ , 2017.
  * Li et al. (2020) Li, Z., Kovachki, N., Azizzadenesheli, K., Liu, B., Bhattacharya, K., Stuart, A., and Anandkumar, A.  Fourier neural operator for parametric partial differential equations.  _arXiv preprint arXiv:2010.08895_ , 2020.
  * Li et al. (2023) Li, Z., Qi, S., Li, Y., and Xu, Z.  Revisiting long-term time series forecasting: An investigation on linear mapping.  _arXiv preprint arXiv:2305.10721_ , 2023.
  * Liu et al. (2022a) Liu, M., Zeng, A., Chen, M., Xu, Z., Lai, Q., Ma, L., and Xu, Q.  Scinet: Time series modeling and forecasting with sample convolution and interaction.  _Advances in Neural Information Processing Systems_ , 35:5816–5828, 2022a.
  * Liu et al. (2019) Liu, S., Saito, S., Chen, W., and Li, H.  Learning to infer implicit surfaces without 3d supervision.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Liu et al. (2020) Liu, S., Zhang, Y., Peng, S., Shi, B., Pollefeys, M., and Cui, Z.  Dist: Rendering deep implicit signed distance function with differentiable sphere tracing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2019–2028, 2020.
  * Liu et al. (2022b) Liu, Y., Wu, H., Wang, J., and Long, M.  Non-stationary transformers: Exploring the stationarity in time series forecasting.  _Advances in Neural Information Processing Systems_ , 35:9881–9893, 2022b.
  * Liu et al. (2024) Liu, Y., Hu, T., Zhang, H., Wu, H., Wang, S., Ma, L., and Long, M.  iTransformer: Inverted transformers are effective for time series forecasting.  In _International Conference on Learning Representations_ , 2024.
  * Lu et al. (2021) Lu, L., Jin, P., Pang, G., Zhang, Z., and Karniadakis, G. E.  Learning nonlinear operators via deeponet based on the universal approximation theorem of operators.  _Nature machine intelligence_ , 3(3):218–229, 2021.
  * Matsubara et al. (2014) Matsubara, Y., Sakurai, Y., Van Panhuis, W. G., and Faloutsos, C.  Funnel: automatic mining of spatially coevolving epidemics.  In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ , pp. 105–114, 2014.
  * Naour et al. (2023) Naour, E. L., Serrano, L., Migus, L., Yin, Y., Agoua, G., Baskiotis, N., Guigue, V., et al.  Time series continuous modeling for imputation and forecasting with implicit neural representations.  _arXiv preprint arXiv:2306.05880_ , 2023.
  * Nie et al. (2023) Nie, Y., Nguyen, N. H., Sinthong, P., and Kalagnanam, J.  A time series is worth 64 words: Long-term forecasting with transformers.  In _International Conference on Learning Representations_ , 2023.
  * Oechsle et al. (2019) Oechsle, M., Mescheder, L., Niemeyer, M., Strauss, T., and Geiger, A.  Texture fields: Learning texture representations in function space.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pp. 4531–4540, 2019.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32, 2019.
  * Rubanova et al. (2019) Rubanova, Y., Chen, R. T., and Duvenaud, D. K.  Latent ordinary differential equations for irregularly-sampled time series.  _Advances in neural information processing systems_ , 32, 2019.
  * Sitzmann et al. (2020) Sitzmann, V., Martel, J., Bergman, A., Lindell, D., and Wetzstein, G.  Implicit neural representations with periodic activation functions.  _Advances in neural information processing systems_ , 33:7462–7473, 2020.
  * Tancik et al. (2020) Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., and Ng, R.  Fourier features let networks learn high frequency functions in low dimensional domains.  _Advances in Neural Information Processing Systems_ , 33:7537–7547, 2020.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wen et al. (2023) Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., and Sun, L.  Transformers in time series: A survey.  In _International Joint Conference on Artificial Intelligence(IJCAI)_ , 2023.
  * Woo et al. (2023) Woo, G., Liu, C., Sahoo, D., Kumar, A., and Hoi, S.  Learning deep time-index models for time series forecasting.  In _International Conference on Machine Learning_ , pp. 37217–37237. PMLR, 2023.
  * Wu et al. (2021) Wu, H., Xu, J., Wang, J., and Long, M.  Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting.  _Advances in Neural Information Processing Systems_ , 34:22419–22430, 2021.
  * Wu et al. (2023) Wu, H., Hu, T., Liu, Y., Zhou, H., Wang, J., and Long, M.  Timesnet: Temporal 2d-variation modeling for general time series analysis.  In _International Conference on Learning Representations_ , 2023.
  * Wu et al. (2022) Wu, T., Maruyama, T., and Leskovec, J.  Learning to accelerate partial differential equations via latent global evolution.  _Advances in Neural Information Processing Systems_ , 35:2240–2253, 2022.
  * Yin et al. (2022) Yin, Y., Kirchmeyer, M., Franceschi, J.-Y., Rakotomamonjy, A., and Gallinari, P.  Continuous pde dynamics forecasting with implicit neural representations.  _arXiv preprint arXiv:2209.14855_ , 2022.
  * Zeng et al. (2023) Zeng, A., Chen, M., Zhang, L., and Xu, Q.  Are transformers effective for time series forecasting?  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 37, pp. 11121–11128, 2023.
  * Zhang & Yan (2023) Zhang, Y. and Yan, J.  Crossformer: Transformer utilizing cross-dimension dependency for multivariate time series forecasting.  In _International Conference on Learning Representations_ , 2023.
  * Zhou et al. (2021) Zhou, H., Zhang, S., Peng, J., Zhang, S., Li, J., Xiong, H., and Zhang, W.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 35, pp. 11106–11115, 2021.
  * Zhou et al. (2022) Zhou, T., Ma, Z., Wen, Q., Wang, X., Sun, L., and Jin, R.  Fedformer: Frequency enhanced decomposed transformer for long-term series forecasting.  In _International Conference on Machine Learning_ , pp. 27268–27286. PMLR, 2022.

##  Appendix A Appendix

In this section, we present the experimental details of PDETime. The
organization of this section is as follows:

  * •

Appendix A.1 provides details on the datasets and baselines.

  * •

Appendix A.2 provides details of time features used in this work.

  * •

Appendix A.3 provides pseudocode of the Encoder of PDETime.

  * •

Appendix A.4 provides pseudocode of the Solver of PDETime.

  * •

Appendix A.5 presents the results of the robustness experiments.

  * •

Appendix A.6 visualizes the prediction results of PDETime on the Traffic
dataset.

###  A.1 Experimental Details

####  A.1.1 Datasets

we use the most popular multivariate datasets in LMTF, including ETT,
Electricity, Traffic and Weather:

  * •

The ETT (Zhou et al., 2021) (Electricity Transformer Temperature) dataset
contains two years of data from two separate countries in China with intervals
of 1-hour level (ETTh) and 15-minute level (ETTm) collected from electricity
transformers. Each time step contains six power load features and oil
temperature.

  * •

The Electricity 111
https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014.
dataset describes 321 clients’ hourly electricity consumption from 2012 to
2014.

  * •

The Traffic 222http://pems.dot.ca.gov. dataset contains the road occupancy
rates from various sensors on San Francisco Bay area freeways, which is
provided by California Department of Transportation.

  * •

the Weather 333https://www.bgc-jena.mpg.de/wetter/. dataset contains 21
meteorological indicators collected at around 1,600 landmarks in the United
States.

Table 5 presents key characteristics of the seven datasets. The dimensions of
each dataset range from 7 to 862, with frequencies ranging from 10 minutes to
7 days. The length of the datasets varies from 966 to 69,680 data points. We
split all datasets into training, validation, and test sets in chronological
order, using a ratio of 6:2:2 for the ETT dataset and 7:1:2 for the remaining
datasets.

Table 5: Statics of Dataset Characteristics. Datasets | ETTh1 | ETTh2 | ETTm1 | ETTm2 | Electricity | Traffic | Weather
---|---|---|---|---|---|---|---
Dimension | 7 | 7 | 7 | 7 | 321 | 862 | 21
Frequency | 1 hour | 1 hour | 15 min | 15 min | 1 hour | 1 hour | 10 min
Length | 17420 | 17420 | 69680 | 69680 | 26304 | 52696 | 17544

####  A.1.2 Baselines

We choose SOTA and the most representative LMTF models as our baselines,
including historical-value-based and time-index-based models, as follows:

  * •

PatchTST (Nie et al., 2023): the current historical-value-based SOTA models.
It utilizes channel-independent and patch techniques and achieves the highest
performance by utilizing the native Transformer.

  * •

DLinear (Zeng et al., 2023): a highly insightful work that employs simple
linear models and trend decomposition techniques, outperforming all
Transformer-based models at the time.

  * •

Crossformer (Zhang & Yan, 2023): similar to PatchTST, it utilizes the patch
technique commonly used in the CV domain. However, unlike PatchTST’s
independent channel design, it leverages cross-dimension dependency to enhance
LMTF performance.

  * •

FEDformer (Zhou et al., 2022): it employs trend decomposition and Fourier
transformation techniques to improve the performance of Transformer-based
models in LMTF. It was the best-performing Transformer-based model before
Dlinear.

  * •

Autoformer (Wu et al., 2021): it combines trend decomposition techniques with
an auto-correlation mechanism, inspiring subsequent work such as FEDformer.

  * •

Stationary (Liu et al., 2022b): it proposes a De-stationary Attention to
alleviate the over-stationarization problem.

  * •

iTransformer (Liu et al., 2024): it different from pervious works that embed
multivariate points of each time step as a (temporal) token, it embeds the
whole time series of each variate independently into a (variate) token, which
is the extreme case of Patching.

  * •

TimesNet (Wu et al., 2023): it transforms the 1D time series into a set of 2D
tensors based on multiple periods and uses a parameter-efficient inception
block to analyze time series.

  * •

SCINet (Liu et al., 2022a): it proposes a recursive downsample-convolve-
interact architecture to aggregate multiple resolution features with complex
temporal dynamics.

  * •

DeepTime (Woo et al., 2023): it is the first time-index-based model in long-
term multivariate time-series forecasting.

###  A.2 Temporal Features

Depending on the sampling frequency, the temporal feature $t_{\tau}$ of each
dataset is also different. We will introduce the temporal feature of each data
set in detail:

  * •

ETTm and Weather: day-of-year, month-of-year, day-of-week, hour-of-day,
minute-of-hour.

  * •

ETTh, Traffic, and Electricity: day-of-year, month-of-year, day-of-week, hour-
of-day.

we also normalize these features into [0,1] range.

###  A.3 Pseudocode

We provide the pseudo-code of Encoder and Solver in Algorithms 1.

Algorithm 1 Pseudocode of the aggregation layer of Encoder

1:Time-index feature $\tau$, temporal feature $t_{\tau}$ and spatial feature
$x_{his}$.

2:$\tau$, $t_{\tau}$, $x_{his}$ = Linear($\tau$), Linear($t_{\tau}$),
Linear($x_{his}$) $\triangleright$ $\tau\in\mathbb{R}^{B\times L\times d}$,
$x_{his}\in\mathbb{R}^{B\times C\times d}$, $t_{\tau}\in\mathbb{R}^{B\times
L\times t}$

3:$\tau$, $t_{\tau}$, $x_{his}$ = Norm(GeLU($\tau$)), Norm($\sin(t_{\tau})$),
Norm(GeLU($x_{his}$))

4:$\tau$ = Norm($\tau$ \+ Attn($\tau$, $x_{his}$))

5:$t_{\tau}$=torch.cat(($\tau$,$t_{\tau}$),dim=-1) $\triangleright$
$t_{\tau}\in\mathbb{R}^{B\times L\times(d+t)}$

6:$\tau$=Norm($\tau$ \+ Linear($t_{\tau}$)) $\triangleright$
$\tau\in\mathbb{R}^{B\times L\times d}$

7:return $\tau$

###  A.4 Pseudocode

We provide the pseudo-code of the Solver in Algorithms 2.

Algorithm 2 Pseudocode of the numerical Solver

1:time-index feature $\tau\in\mathbb{R}^{B\times L\times d}$, batch size $B$,
input length $L$ and dimension $d$, and patch length $S$.

2:dudt = MLP ($\tau$) $\triangleright$ dudt $\in\mathbb{R}^{B\times L\times
d}$

3:u = MLP ($\tau$) $\triangleright$ u $\in\mathbb{R}^{B\times L\times d}$

4:u=u.reshape(B, -1, S, d) $\triangleright$ u
$\in\mathbb{R}^{B\times\frac{L}{S}}\times S\times d$

5:dudt = -dudt

6:dudt = torch.flip(dudt, [2])

7:dudt = dudt_[:,:,:-1,:] $\triangleright$ dudt
$\in\mathbb{R}^{B\times\frac{L}{S}\times(S-1)\times d}$

8:dudt = torch.cat((u[:,:,-1:,:], dudt), dim=-2) $\triangleright$ dudt
$\in\mathbb{R}^{B\times\frac{L}{S}}\times S\times d$

9:intdudt = torch.cumsum(dudt, dim=-2)

10:intdudt = torch.flip(dudt, [2])

11:intdudt = intdudt.reshape(B, L , d) $\triangleright$ intdudt
$\in\mathbb{R}^{B\times L\times d}$

12:intdudt = MLP(intdudt) $\triangleright$ intdudt $\in\mathbb{R}^{B\times
L\times d}$

13:return intdudt

###  A.5 Experimental Results of Robustness

The experimental results of the robustness of our algorithm based on Solver
and Initial condition are summarized in Table 6. We also test the
effectiveness of continuity loss $\mathcal{L}_{c}$ in Table 7. The
experimental results in Table 6 demonstrate that PDETime can achieve strong
performance on the ETT dataset even when using only the Solver or initial
value conditions, without explicitly incorporating spatial and temporal
information. This finding suggests that PDETime has the potential to perform
well even in scenarios with limited data availability. Additionally, we
conducted an analysis on the ETTh1 and ETTh2 datasets to investigate the
impact of the loss term $\mathcal{L}_{c}$. Our findings demonstrate that
incorporating $\mathcal{L}_{c}$ into PDETime can enhance its robustness.

Table 6: Analysis on Solver and Initial condition. INRs refers to only using INRs to represent $\tau$; + Initial refers to aggregating initial condition $x_{\tau_{0}}$; +Solver refers to using numerical solvers to compute integral terms in latent space. The best results are highlighted in bold. Dataset |  Method |  INRs |  INRs+Initial |  INRs+Solver
---|---|---|---|---
Metric |  MSE |  MAE |  MSE |  MAE |  MSE |  MAE
ETTh1 |  96 |  0.371 |  0.396 |  0.364 |  0.384 |  0.358 |  0.381
192 |  0.403 |  0.420 |  0.402 |  0.409 |  0.398 |  0.407
336 |  0.433 |  0.436 |  0.428 |  0.420 |  0.348 |  0.238
720 |  0.474 |  0.492 |  0.439 |  0.452 |  0.455 |  0.476
ETTh2 |  96 |  0.287 |  0.352 |  0.270 |  0.330 |  0.285 |  0.342
192 |  0.383 |  0.412 |  0.331 |  0.372 |  0.345 |  0.379
336 |  0.523 |  0.501 |  0.373 |  0.405 |  0.357 |  0.399
720 |  0.765 |  0.624 |  0.392 |  0.429 |  0.412 |  0.444
Table 7: Analysis on the effectiveness of loss term $\mathcal{L}_{c}$.  Dataset |  Method |  PDETime |  PDETime-$\mathcal{L}_{c}$
---|---|---|---
Metric |  MSE |  MAE |  MSE |  MAE
ETTh1 | 96 | 0.356 | 0.381 | 0.357 | 0.381
192 | 0.397 | 0.406 | 0.393 | 0.405
336 | 0.420 | 0.419 | 0.422 | 0.420
720 | 0.425 | 0.419 | 0.446 | 0.458
ETTh2 | 96 | 0.268 | 0.330 | 0.271 | 0.330
192 | 0.331 | 0.370 | 0.341 | 0.373
336 | 0.358 | 0.395 | 0.363 | 0.397
720 | 0.380 | 0.421 | 0.396 | 0.434

###  A.6 Visualization

We visualize the prediction results of PDETime on the Traffic dataset. As
illustrated in Figure 5, for prediction lengths $H=96,192,336,720$, the
prediction curve closely aligns with the ground-truth curves, indicating the
outstanding predictive performance of PDETime. Meanwhile, PDETime demonstrates
effectiveness in capturing periods of time features.

Figure 5: Visualization of PDETime’s prediction results on Traffic with
different prediction length.