$\displaystyle f(t_{0})\leq A+B\int_{0}^{t_{0}}s^{-\gamma}f(t_{0}-s)ds.$

Then, there exists a constant $C$ depending only on $B$ and $\gamma$ so that

$\displaystyle f(t)\leq CA,\quad\forall 0\leq t\leq 1.$

See Lemma 2.2 in [48].

###  4.4 Proof for the main theorem 3

The previous section shows that the solution $\phi$ to the system (20)
verifies the bound

$\displaystyle|\phi(t,x)|\lesssim(1-t+r)^{-\frac{5+2\epsilon-p}{2p_{1}^{\prime}}}\mathcal{M}(t)^{\frac{1}{p_{1}^{\prime}}}\lesssim(1-t)^{-\frac{5+2\epsilon-p}{2p_{1}^{\prime}}}\lesssim(1-t)^{-\frac{5-p+\epsilon_{2}}{p+1}}$

for some small positive constant $\epsilon_{2}$ (depending on $p$, $\epsilon$
and $\epsilon_{1}$). By the definition of $\phi(t,x)$ and $A$ in
$\mathbb{R}^{1+3}$, the solution
$\tilde{\phi}(\tilde{t},\tilde{x}_{1},\tilde{x}_{2})$ to the equation (16)
also verifies the bound

$\displaystyle|\tilde{\phi}(\tilde{t},\tilde{x}_{1},\tilde{x}_{2})|=|\phi|\lesssim(1-\tilde{t})^{-\frac{5-p+\epsilon_{2}}{p+1}}.$

Finally in view of the conformal transformation, the original solution
$\phi(t,x_{1},x_{2})$ (by abusing the notation, the original solution $\phi$
is different to $\phi(t,x_{1},x_{2},x_{3})$ defined in $\mathbb{R}^{1+3}$) to
(1) in $\mathbb{R}^{1+2}$ then satisfies the pointwise bound

$\displaystyle|\phi(t,x_{1},x_{2})|$
$\displaystyle\lesssim\Lambda^{-\frac{1}{2}}|\tilde{\phi}(\tilde{t},\tilde{x}_{1},\tilde{x}_{2})|\lesssim(2+t+r_{1})^{-\frac{1}{2}}(2+t-r_{1})^{-\frac{1}{2}}(2+t-r_{1})^{\frac{5-p+\epsilon_{2}}{p+1}}$
$\displaystyle\lesssim(2+t+r_{1})^{-\frac{1}{2}}(2+|t-r_{1}|)^{-\frac{3p-9-2\epsilon_{2}}{2(p+1)}}.$

Here $r_{1}=\sqrt{x_{1}^{2}+x_{2}^{2}}$ and the solution $\phi$ vanishes out
side the cone $t\leq r_{1}-1$ since the initial data are compactly supported.
We thus finished the proof for Theorem 3.

## References

  * [1] S. Alinhac.  The null condition for quasilinear wave equations in two space dimensions I.  Invent. Math., 145(3):597–618, 2001.
  * [2] S. Alinhac.  The null condition for quasilinear wave equations in two space dimensions. II.  Amer. J. Math., 123(6):1071–1101, 2001.
  * [3] H. Brézis and T. Gallouet.  Nonlinear Schrödinger evolution equations.  Nonlinear Anal., 4(4):677–681, 1980.
  * [4] H. Brézis and S. Wainger.  A note on limiting cases of Sobolev embeddings and convolution inequalities.  Comm. Partial Differential Equations, 5(7):773–789, 1980.
  * [5] D. Chae and K. Choe.  Global existence in the Cauchy problem of the relativistic Chern-Simons-Higgs theory.  Nonlinearity, 15(3):747–758, 2002.
  * [6] M. Chae and S. Oh.  Small data global existence and decay for relativistic Chern-Simons equations.  Ann. Henri Poincaré, 18(6):2123–2198, 2017.
  * [7] Y. Cho and S. Hong.  Low regularity solutions to the non-abelian Chern-Simons-Higgs system in the Lorenz gauge.  NoDEA Nonlinear Differential Equations Appl., 28(6):Paper No. 70, 25, 2021.
  * [8] M. Dafermos and I. Rodnianski.  A new physical-space approach to decay for the wave equation with applications to black hole spacetimes.  In XVIth International Congress on Mathematical Physics, pages 421–432. World Sci. Publ., Hackensack, NJ, 2010.
  * [9] J. Delort, D. Fang, and R. Xue.  Global existence of small solutions for quadratic quasilinear Klein-Gordon systems in two space dimensions.  J. Funct. Anal., 211(2):288–323, 2004.
  * [10] S. Dong, Y. Ma, and X. Yuan.  Asymptotic behavior of 2D wave-Klein-Gordon coupled system under null condition.  Bull. Sci. Math., 187:Paper No. 103313, 43, 2023.
  * [11] G. Dunne.  Aspects of Chern-Simons theory.  In Aspects topologiques de la physique en basse dimension/Topological aspects of low dimensional systems (Les Houches, 1998), pages 177–263. EDP Sci., Les Ulis, 1999.
  * [12] D.Wei and S. Yang.  On the global behaviors for defocusing semilinear wave equations in $\mathbb{R}^{1+2}$.  Anal. PDE, 15(1):245–272, 2022.
  * [13] D. Gilbarg and N.Trudinger.  Elliptic Partial Differential Equations of Second Order.  Springer-Verlag, Berlin, reprint of the 1998 edition edition, 2001.
  * [14] J. Ginibre and G. Velo.  The global Cauchy problem for the nonlinear Klein-Gordon equation.  Math. Z., 189(4):487–505, 1985.
  * [15] J. Ginibre and G. Velo.  Conformal invariance and time decay for nonlinear wave equations. I, II.  Ann. Inst. H. Poincaré Phys. Théor., 47(3):221–261, 263–276, 1987.
  * [16] J. Ginibre and G. Velo.  Generalized Strichartz inequalities for the wave equation.  J. Funct. Anal., 133(1):50–68, 1995.
  * [17] S. Girvin.  Introduction to the fractional quantum Hall effect.  In The quantum Hall effect, volume 45 of Prog. Math. Phys., pages 133–162. Birkhäuser, Basel, 2005.
  * [18] R. Glassey and H. Pecher.  Time decay for nonlinear wave equations in two space dimensions.  Manuscripta Math., 38(3):387–400, 1982.
  * [19] J. Hong, Y. Kim, and P. Pac.  Multivortex solutions of the abelian Chern-Simons-Higgs theory.  Phys. Rev. Lett., 64(19):2230–2233, 1990.
  * [20] H. Huh.  Low regularity solutions of the Chern-Simons-Higgs equations.  Nonlinearity, 18(6):2581–2589, 2005.
  * [21] H. Huh.  Local and global solutions of the Chern-Simons-Higgs system.  J. Funct. Anal., 242(2):526–549, 2007.
  * [22] H. Huh.  Towards the Chern-Simons-Higgs equation with finite energy.  Discrete Contin. Dyn. Syst., 30(4):1145–1159, 2011.
  * [23] H. Huh.  Global energy solutions of Chern-Simons-Higgs equations in one space dimension.  J. Math. Anal. Appl., 420(1):781–791, 2014.
  * [24] H. Huh.  Conformal identity of the Chern-Simons gauged wave equations and its applications.  J. Hyperbolic Differ. Equ., 14(2):341–347, 2017.
  * [25] H. Huh.  $L^{\infty}$ bounds for Maxwell-gauged equations in $\mathbb{R}^{1+1}$ and their applications.  J. Math. Anal. Appl., 488(2):124064, 10, 2020.
  * [26] H. Huh and S. Oh.  Low regularity solutions to the Chern-Simons-Dirac and the Chern-Simons-Higgs equations in the Lorenz gauge.  Comm. Partial Differential Equations, 41(3):375–397, 2016.
  * [27] H. Huh and J. Yim.  $L^{\infty}$ bounds for Chern-Simons gauged equations in $\mathbb{R}^{1+1}$ and their applications.  J. Math. Phys., 62(5):Paper No. 051502, 9, 2021.
  * [28] A. Ionescu and B. Pausader.  Global regularity of solutions of the Einstein-Klein-Gordon system: a review.  Quart. Appl. Math., 78(2):277–303, 2020.
  * [29] R. Jackiw and E. Weinberg.  Self-dual Chern-Simons vortices.  Phys. Rev. Lett., 64(19):2234–2237, 1990.
  * [30] A. Jaffe and C. Taubes.  Vortices and monopoles, volume 2 of Progress in Physics.  Birkhäuser, Boston, MA, 1980.  Structure of static gauge theories.
  * [31] M. Keel and T. Tao.  Endpoint Strichartz estimates.  Amer. J. Math., 120(5):955–980, 1998.
  * [32] S. Klainerman.  Uniform decay estimates and the lorentz invariance of the classical wave equation.  Comm. Pure Appl. Math., 38(3):321–332, 1985.
  * [33] S. Klainerman and I. Rodnianski.  Sharp trace theorems for null hypersurfaces on Einstein metrics with finite curvature flux.  Geom. Funct. Anal., 16(1):164–229, 2006.
  * [34] S. Klainerman and I. Rodnianski.  A Kirchoff-Sobolev parametrix for the wave equation and applications.  J. Hyperbolic Differ. Equ., 4(3):401–433, 2007.
  * [35] S. Klainerman, Q. Wang, and S. Yang.  Global solution for massive Maxwell-Klein-Gordon equations.  Comm. Pure Appl. Math., 73(1):63–109, 2020.
  * [36] H. Lindblad and T. Tao.  Asymptotic decay for a one-dimensional nonlinear wave equation.  Anal. PDE, 5(2):411–422, 2012.
  * [37] G. Luli, S. Yang, and P. Yu.  On one-dimension semi-linear wave equations with null conditions.  Adv. Math., 329:174–188, 2018.
  * [38] M. Nolasco and G. Tarantello.  Double vortex condensates in the Chern-Simons-Higgs theory.  Calc. Var. Partial Differential Equations, 9(1):31–94, 1999.
  * [39] M. Okamoto.  Well-posedness of the Cauchy problem for the Chern-Simons-Dirac system in two dimensions.  J. Hyperbolic Differ. Equ., 10(4):735–771, 2013.
  * [40] H. Pecher.  Decay of solutions of nonlinear wave equations in three space dimensions.  J. Funct. Anal., 46(2):221–229, 1982.
  * [41] H. Pecher.  Global well-posedness in energy space for the Chern-Simons-Higgs system in temporal gauge.  J. Hyperbolic Differ. Equ., 13(2):331–351, 2016.
  * [42] S. Selberg and A. Tesfahun.  Global well-posedness of the Chern-Simons-Higgs equations with finite energy.  Discrete Contin. Dyn. Syst., 33(6):2531–2546, 2013.
  * [43] J. Spruck and Y. Yang.  Topological solutions in the self-dual Chern-Simons theory: existence and approximation.  Ann. Inst. H. Poincaré C Anal. Non Linéaire, 12(1):75–97, 1995.
  * [44] E. Stein.  Harmonic analysis: real-variable methods, orthogonality, and oscillatory integrals, volume 43 of Princeton Mathematical Series.  Princeton University Press, Princeton, NJ, 1993.  With the assistance of Timothy S. Murphy, Monographs in Harmonic Analysis, III.
  * [45] W. Strauss.  Decay and asymptotics for $\Box u=F(u)$.  J. Functional Analysis, 2:409–457, 1968.
  * [46] G. Tarantello.  Multiple condensate solutions for the Chern-Simons-Higgs theory.  J. Math. Phys., 37(8):3769–3796, 1996.
  * [47] W. von Wahl.  Some decay-estimates for nonlinear wave equations.  J. Functional Analysis, 9:490–495, 1972.
  * [48] D. Wei, S. Yang, and P. Yu.  On the global dynamics of Yang-Mills-Higgs equations.  2022\.  arXiv:2203.12143.
  * [49] S. Yang.  Global behaviors of defocusing semilinear wave equations.  Ann. Sci. Éc. Norm. Supér. (4), 55(2):405–428, 2022.
  * [50] S. Yang.  Pointwise decay for semilinear wave equations in $\mathbb{R}^{1+3}$.  J. Funct. Anal., 283(2):Paper No. 109486, 59, 2022.
  * [51] S. Yang and P. Yu.  On global dynamics of the Maxwell-Klein-Gordon equations.  Camb. J. Math., 7(4):365–467, 2019.
  * [52] J. Yuan.  Local well-posedness of non-abelian Chern-Simons-Higgs system in the Lorenz gauge.  J. Math. Phys., 58(7):071505, 10, 2017.

School of Mathematical Sciences, Peking University, Beijing, China

E-mail address<EMAIL_ADDRESS>
Beijing International Center for Mathematical Research, Peking University,
Beijing, China

E-mail address<EMAIL_ADDRESS>


# Spatial-Temporal Synchronous Graph Transformer network (STSGT) for COVID-19
forecasting

Soumyanil Banerjee  Department of Computer ScienceWayne State
UniversityDetroit, MIUSA<EMAIL_ADDRESS>,  Ming Dong  Department of
Computer ScienceWayne State UniversityDetroit, MIUSA<EMAIL_ADDRESS>and
Weisong Shi  Department of Computer ScienceWayne State UniversityDetroit,
MIUSA<EMAIL_ADDRESS>
###### Abstract.

COVID-19 has become a matter of serious concern over the last few years. It
has adversely affected numerous people around the globe and has led to the
loss of billions of dollars of business capital. In this paper, we propose a
novel Spatial-Temporal Synchronous Graph Transformer network (STSGT) to
capture the complex spatial and temporal dependency of the COVID-19 time
series data and forecast the future status of an evolving pandemic. The layers
of STSGT combine the graph convolution network (GCN) with the self-attention
mechanism of transformers on a synchronous spatial-temporal graph to capture
the dynamically changing pattern of the COVID time series. The spatial-
temporal synchronous graph simultaneously captures the spatial and temporal
dependencies between the vertices of the graph at a given and subsequent time-
steps, which helps capture the heterogeneity in the time series and improve
the forecasting accuracy. Our extensive experiments on two publicly available
real-world COVID-19 time series datasets demonstrate that STSGT significantly
outperforms state-of-the-art algorithms that were designed for spatial-
temporal forecasting tasks. Specifically, on average over a 12-day horizon, we
observe a potential improvement of 12.19% and 3.42% in Mean Absolute Error
(MAE) over the next best algorithm while forecasting the daily infected and
death cases respectively for the 50 states of US and Washington, D.C.
Additionally, STSGT also outperformed others when forecasting the daily
infected cases at the state level, e.g., for all the counties in the State of
Michigan. The code and models are publicly available at
https://github.com/soumbane/STSGT.

spatial-temporal graphs, COVID-19 forecasting, transformers

††copyright: none

##  1\. Introduction

The Coronavirus disease 2019 (COVID-19) is caused by severe acute respiratory
syndrome coronavirus 2 (SARS-CoV-2). The first known case was identified in
Wuhan, China in December 2019, and it has since rapidly spread around the
globe. The coronavirus causes infection to the upper respiratory tract which
could prove fatal if the immune system does not respond (Harapan et al.,
2020). The World Health Organization (WHO) declared COVID-19 as a global
pandemic in March, 2020 (Cucinotta and Vanelli, 2020). Since then, COVID has
disrupted many businesses and households worldwide. Millions of people around
the globe got infected with the virus and numerous families lost their loved
ones (Koh et al., 2021) due to complications that aroused from the virus.
Small businesses have been the ones that were the most adversely affected due
to COVID (Bartik et al., 2020). The global economy and financial markets were
also severely affected due to COVID (Pak et al., 2020). This in turn affected
a lot of people worldwide who lost their jobs due to the economic downturn.

Deep learning has revolutionized the areas of computer vision, natural
language processing, robotics, medical image processing and many others. Deep
learning removes the feature engineering process of traditional machine
learning and learns in an end-to-end manner by automatically extracting
meaningful and important patterns from the input data. For example, many
researchers have used deep learning models for detecting COVID-19 from medical
images such as X-rays (Minaee et al., 2020; Awan et al., 2021). However, such
methods could only be applied for persons already affected by COVID and these
are not ideal for prevention of COVID-19.

In such situations, the prevention of COVID-19 becomes a matter of serious
concern as a lot of lives could be saved if we could prevent the disease. The
most important question that arises regarding the prevention of COVID is
whether we can forecast the future status of the disease. In other words, if
we could forecast the total number of infected and death cases in advance,
then a lot of precautions could be taken to stop the disease from spreading.
This leads us to the problem of time series analysis of COVID cases, where we
make use of the past historical COVID data to forecast the future COVID cases.
It is also equally important to forecast the COVID cases over the long-term
since this would provide some time for the health care workers to prepare for
difficult situations and help them serve the infected patients in a more
efficient way. Additionally, a long-term forecasting would enable the common
people to take safety precautions and prevent the disease.

Graph Neural Networks (GNNs) are a special type of machine learning algorithms
that considers a graph as input and learns to predict the attributes of the
vertices and edges of the graph. Recently, GNNs have been used for numerous
applications where the data could be modeled as a graph. GNNs could also be
combined with time series analysis algorithms to construct the spatial-
temporal graph neural network. One of the classic cases where a spatial-
temporal graph neural network has been used is traffic forecasting. In traffic
forecasting, many researchers have proposed complex spatial-temporal graph
neural networks to accurately capture the complex pattern in the spatial-
temporal time series data. However, there are very few research works that
deal with time series forecasting with COVID data. Some researchers have used
spatial-temporal graph neural networks to forecast the COVID cases (Kapoor et
al., 2020) but such methods were applied during the start of the pandemic
(summer 2020) and as such the size of the dataset was too small to draw an
inference. Hence, it becomes critically important that we develop a deep
learning model which uses the COVID data from the start of the pandemic to
recent times, so that this model could be used for prevention of the disease
more effectively.

In this paper, we develop a novel Spatial-Temporal Synchronous Graph
Transformer network (STSGT) which uses the historical COVID data to forecast
the future infected and death cases for all the US states. In STSGT, first we
denote the US states as vertices of a graph and the daily infected or death
cases as the vertex features for a given time-step. The physical distances
between the states are used as an adjacency matrix. Then, a large spatial-
temporal synchronous graph is generated from the historical time-steps, where
each day denote a time-step. This novel process ensures that we can capture
the complex non-linear spatial-temporal dependencies between adjacent time-
steps in a synchronous fashion. We apply the multi-head self-attention
mechanism of transformers (Vaswani et al., 2017) on the spatial-temporal
synchronous graph to infer the dependency of a vertex of the graph with
another vertex. After each vertex finds its importance to all other vertices
based on attention scores, we use the graph convolution network (GCN) (Kipf
and Welling, 2017) to aggregate the features (infected or death cases) from
neighboring vertices and update the current vertex of the graph. Finally, we
forecast the future COVID cases for both short-term, i.e., next day
forecasting, and long-term, i.e., forecasting over the next 12 days.

The main contributions of our work are summarized as follows:

  * •

We propose a novel Spatial-Temporal Synchronous Graph Transformer network
(STSGT) which can simultaneously capture the complex spatial and temporal
dependencies of the COVID time series data and accurately forecast the future
COVID cases by utilizing the multi-head self-attention mechanism of
transformers combined with a graph convolution network (GCN).

  * •

To the best of our knowledge, this is the first work that performs such a
thorough and detailed analysis with real world COVID-19 time series datasets
over a long period of time from the start of the pandemic in March 2020 to Nov
2021 (right before the start of the Omicron wave).

  * •

Extensive experiments on two real-world public datasets demonstrate that our
model STSGT could accurately forecast the daily infected and death cases at
both the national (for all the US states) and state (e.g., for all the
counties of the state of Michigan) levels. Our model significantly outperforms
other state-of-the-art models designed for spatial-temporal forecasting tasks.

The rest of the paper is organized as follows: Section 2 reviews the relevant
research works and their applications. Section 3 describes the details of our
proposed model STSGT along with its components. Section 4 describes the
experimental setup and the results of our experiments. Section 5 presents
discussion and future work. Lastly, Section 6 presents our conclusion.

##  2\. Related Works

###  2.1. Graph Neural Networks

Graph Neural Networks (GNNs) have been significantly useful when the input
data does not have a fixed structure as in an image or text and could be
represented as a graph with a set of vertices, edges and an adjacency matrix.
GNNs and its variants have been employed for numerous applications such as
recommendation systems (Wu et al., 2020), protein interface prediction (Fout
et al., 2017) and semantic segmentation (Qi et al., 2017). For example, a
combination of Convolution Neural Networks (CNNs) and graph relation networks
have been used for brain connectivity network analysis (Banerjee et al.,
2020).

Graph Convolution Network (GCN) (Kipf and Welling, 2017) was first developed
in a transductive setting to apply the convolutional operation on graphs in
the spectral domain. The attention mechanism was introduced in the spatial
domain into the feature aggregation process of GCNs by Graph Attention Network
(GAT) (Veličković et al., 2018), which learns the weights to aggregate
features from the neighboring vertices. The first application of graph
representation learning to large-scale graphs in an inductive setting was
introduced by GraphSAGE (Hamilton et al., 2017). Cluster-GCN (Chiang et al.,
2019) removed the redundant computation in the neighborhood sampling of
GraphSAGE by clustering the graph and then train a GCN. A major limitation of
increasing the number of layers of GCN is that it leads to over-smoothing,
where vertex representations become indistinguishable as the number of layers
of the graph network increases (Li et al., 2019, 2018a; Xu et al., 2018; Rong
et al., 2019; Zhao and Akoglu, 2019).

###  2.2. Spatial-Temporal Forecasting

Spatial-Temporal forecasting is a fundamentally important problem and has lot
of useful applications such as weather forecasting (Tekin et al., 2021) and
action recognition (Yan et al., 2018). In smart transportation, traffic
forecasting is one of the most challenging problems where the goal is to
forecast the future traffic status based on historical observations. Spatial-
temporal graph neural networks have been extensively used for traffic
forecasting. Spatio-temporal graph convolution networks (STGCN) (Yu et al.,
2018) used a GCN and a gated 1-D CNN to capture spatial and temporal
dependencies respectively. Diffusion convolutional recurrent neural networks
(DCRNN) (Li et al., 2018b) used the diffusion convolution and the diffusion
convolution gated recurrent unit to model the spatial and temporal
dependencies respectively. An attention-based approach has been combined with
GCN and used with the weekly, daily and hourly traffic data in ASTGCN (Guo et
al., 2019). Graph WaveNet (Wu et al., 2019) uses a gated temporal
convolutional network (TCN) (Lea et al., 2016) with a dilation factor and a
GCN for capturing the long-range temporal and spatial dependencies
respectively for traffic forecasting. Spatial-temporal synchronous graph
convolutional networks (STSGCN) (Song et al., 2020) applied the GCN on a
synchronous spatial-temporal graph to capture the complex spatial-temporal
dependencies and heterogeneity in traffic data. Spatial-temporal fusion graph
neural networks (STFGNN) (Li and Zhu, 2021) fused a temporal graph with the
spatial graph to forecast the future traffic conditions. Spatial-temporal
graph neural networks has also recently been used to forecast the COVID-19
cases (Kapoor et al., 2020).

###  2.3. Transformers

One of the most prominent architectures that have revolutionized the Natural
Language Processing (NLP) field are transformers (Vaswani et al., 2017).
Transformers introduced the multi-head self-attention mechanism which could
capture the relationship between every pair of words in a sentence in a
parallel manner, thereby improving over the performance of serial
architectures such as long short-term memory (LSTM) (Hochreiter and
Schmidhuber, 1997) and gated recurrent unit (GRU) (Chung et al., 2014), which
have the vanishing gradient problem (Hochreiter, 1998). Transformers and its
recent variants such as BERT (Devlin et al., 2018) and large language models
such as GPT-3 (Brown et al., 2020) have been used for numerous NLP
applications such as machine translation, sentiment classification, speech
recognition and many others.

The self-attention mechanism of transformers have also been used in computer
vision by vision transformers (Dosovitskiy et al., 2020) for image recognition
at scale. Recently, the self-attention mechanism of transformers has been
adopted to the spatial domain of GNNs with graphormer (Ying et al., 2021) and
to the spatial-temporal forecasting domain with the spatial-temporal
transformer network (STTN) (Xu et al., 2020).

##  3\. Our Approach

Our proposed Spatial-Temporal Synchronous Graph Transformer network (STSGT)
could be summarized with three main points: 1) Generate the samples from the
original time series data with a sliding window approach; 2) For each sample,
connect each vertex with itself at the previous and the next time steps to
construct a spatial-temporal synchronous graph; 3) Use STSGT layers to capture
the complex spatial-temporal correlations and then forecast the future status
of the spatial-temporal graph network.

In STSGT, we generate a large synchronous spatial-temporal graph with several
days historical data and then calculate the multi-head self-attention between
every pair of vertices of this large graph. This novel process not only
captures the relationship between the COVID infected or death cases between
neighboring states or counties in a given day but also simultaneously captures
the relationship between those states or counties in the previous and
following days. Note that STSGT is fundamentally different from existing
approaches for spatial-temporal forecasting. First, in most other graph neural
network-based spatial-temporal models, a GCN is used to capture the spatial
dependency for a given time-step, and a separate 1D convolution unit or
attention unit is used to capture the temporal dependency between the time-
steps, while STSGT used the multi-head self-attentions to capture both spatial
and temporal dependency synchronously. In the literature, the most similar
model to our approach is STTN (Xu et al., 2020), which has separate spatial
and temporal transformer units and hence cannot simultaneously capture the
complex spatial-temporal dependencies between adjacent time steps. The
synchronous mechanism in our model helps us to achieve better performance in
forecasting.

In the following sections, we provide a detailed discussion on every component
of STSGT.

###  3.1. Generate samples using a Sliding Window

We use a sliding window approach to generate the samples from the COVID-19
time series datasets as shown in Figure 1.

Each of the time step comprises of a spatial graph $G=(V,E,A)$, where $V$ and
$E$ denote the set of vertices and edges of $G$ respectively. The number of
vertices of the graph is denoted as $N$, where $N=|V|$. “$A$” denotes the
adjacency matrix of $G$, where $A\in\mathbb{R}^{N\times N}$. For each vertex
$N$ of the graph $G$, there are certain features $F$ that are associated with
it. Hence, we denote graph signal matrix for each time-step $t$ of the time
series as $X_{(t)}$, where $X_{(t)}\in\mathbb{R}^{N\times F}$.

Figure 1. Generating the samples from the training data using a sliding window
approach.

Then, we can formulate the problem of COVID-19 time series forecasting as
learning the function $f$ which uses the historical $M$ observations to
forecast the future $H$ observations:

(1)
$\hat{X}_{(t+1)},\hat{X}_{(t+2)},...,\hat{X}_{(t+H)}=f\left(X_{(t)},X_{(t-1)},...,X_{(t-M+1)}\right)$

where, $\hat{X}$ and $X$ denote the $H$ forecasted and $M$ ground truth
observations respectively.

###  3.2. Construction of the Spatial-Temporal Synchronous Graph

The goal behind the construction of the spatial-temporal synchronous graph is
to capture the complex spatial-temporal dependencies that are present between
vertices of previous and subsequent time-steps. To achieve this goal, we
connected each vertex $N$ of the graph $G$ to itself in the previous and next
time-step for the entire time period $M$ as illustrated in Figure 2. This
results in a larger spatial-temporal graph $\tilde{G}$ where a given vertex at
a particular time-step can aggregate information from another neighborhood
vertex of previous and subsequent time-steps. The larger spatial-temporal
graph also helps in capturing the long-range dependencies between the vertices
of different time-steps.

Figure 2. Spatial-Temporal Synchronous Graph $\tilde{G}$ generated from the
historical $M$ time steps. Here, we have illustrated the case when $M=12$.

The adjacency matrix of the spatial graph $G$ with $N$ vertices at a given
time-step $t$ is given as $A\in\mathbb{R}^{N\times N}$. Hence, the spatial-
temporal synchronous graph $\tilde{G}$ that we construct will have its
adjacency matrix $\tilde{A}$, where $\tilde{A}\in\mathbb{R}^{MN\times MN}$. In
this large graph $\tilde{G}$, a given vertex’s index $p(1\leq p\leq N)$ in the
spatial graph $G$ will be mapped to a new index in the large spatial-temporal
graph $\tilde{G}$, and the new index will be: $(t-1)N+p$, where $t(1\leq t\leq
M)$ is the time-step number in the larger spatial-temporal synchronous graph
$\tilde{G}$.

In the large spatial-temporal synchronous graph $\tilde{G}$, when a given
vertex at time-step $t_{i}$ is connected to itself at subsequent time-step
$t_{j}$, the corresponding value in the large adjacency matrix $\tilde{A}$
with $MN$ vertices is set to be 1. Hence, the adjacency matrix of $\tilde{A}$
is formulated as:

(2) $\tilde{A}_{i,j}=\begin{cases}1,&\text{if $v_{i}$ connects to $v_{j}$}\\\
0,&\text{otherwise}\end{cases}$

where $v_{i}$ and $v_{j}$ denotes the same vertex $i$ and $j$ respectively in
subsequent time-steps of the spatial-temporal synchronous graph $\tilde{G}$.

Figure 3. Adjacency Matrix $\tilde{A}$ of the Spatial-Temporal Synchronous
Graph $\tilde{G}$. Here, we have illustrated the case when $M=12$.

We have shown the adjacency matrix $\tilde{A}$ of the spatial-temporal
synchronous graph $\tilde{G}$ in Figure 3. The diagonal of $\tilde{A}$ are the
adjacency matrices $A(t_{k})$ of the corresponding spatial graphs $G(t_{k})$
for time-steps $t_{k}$ where $1\leq k\leq M$. Hence, we can say that
$\tilde{A}_{i,i}=A(t_{k})$, where $i=(k-1)N$ to $kN$ for $1\leq k\leq M$.

The off-diagonal matrices of $\tilde{A}$ indicate the connection between each
vertex to itself in the previous or subsequent time-steps. Hence, we can say
that the upper-triangular side of $\tilde{A}$, i.e. $A(t_{k}\rightarrow
t_{k+1})$ for $1\leq k\leq(M-1)$ is formed as: $\tilde{A}_{i,j}=1$, where
$i=(k-1)N+p$, $j=kN+p$ and $i\neq j$, for $1\leq k\leq(M-1)$ and $1\leq p\leq
N$. In a similar manner, the lower triangular side of $\tilde{A}$, i.e.
$A(t_{k+1}\rightarrow t_{k})$ for $1\leq k\leq(M-1)$ is symmetric to the upper
triangular side $A(t_{k}\rightarrow t_{k+1})$ for $1\leq k\leq(M-1)$. All the
other elements are zero as indicated in Figure 3.

In our approach, the spatial adjacency matrix $A$ for graph $G(t_{k})$ at
time-step $t_{k}$ is a weighted adjacency matrix formed by the physical
distances between the vertices of $G(t_{k})$ which we denote as the states of
US or the counties in a state. Hence, for this adjacency matrix $A$, the
feature aggregation process of GCN in our model’s STSGT layer will not be
affected as each vertex of $G(t_{k})$ has a different influence on its
neighboring vertices. But for the adjacency matrix $\tilde{A}$ of the large
spatial-temporal synchronous graph $\tilde{G}$, the off-diagonal matrices
$A(t_{k}\rightarrow t_{k+1})$ and $A(t_{k+1}\rightarrow t_{k})$ ($1\leq
k\leq(M-1)$) have entries that are $1$. Our goal is to learn the temporal
dependency between same vertices in adjacent time-steps ($v_{i}$ and $v_{j}$)
in an end-to-end fashion. This would make our feature aggregation process more
general and flexible since the features between vertices of adjacent time-
steps would be aggregated based on the learned temporal dependency. Hence, we
define a mask matrix $\tilde{A}_{mask}$ with a similar shape as $\tilde{A}$,
i.e. $\tilde{A}_{mask}\in\mathbb{R}^{MN\times MN}$. We calculate the Hadamard-
product between the mask matrix and the adjacency matrix $\tilde{A}$ to form
an updated matrix as follows:

(3) $\tilde{A}_{updated}=\tilde{A}_{mask}\circ\tilde{A}$

where the mask matrix $\tilde{A}_{mask}$ contains the learnable weights
between the vertices of adjacent time-steps $v_{i}$ and $v_{j}$ during the
training process, $\circ$ denotes the Hadamard-product and
$\tilde{A}_{updated}\in\mathbb{R}^{MN\times MN}$ denotes the updated adjacency
matrix. Subsequently, $\tilde{A}_{updated}$ is used for all GCN operations on
the spatial-temporal synchronous graph $\tilde{G}$.

###  3.3. Spatial-Temporal Synchronous Graph Transformer Network

The architecture of our proposed Spatial-Temporal Synchronous Transformer
network (STSGT) is shown in Figure 4.

Figure 4. The architecture of our proposed Spatial-Temporal Synchronous Graph
Transformer network (STSGT).

The input of STSGT is generated from the graph signal matrix
$X_{(t)}\in\mathbb{R}^{N\times F}$ by considering $M$ continuous time-steps.
Hence, we generate the spatial-temporal synchronous graph signal matrix
$\tilde{X}\in\mathbb{R}^{M\times N\times F}$ which is basically the graph
signal matrix of the large spatial-temporal synchronous graph $\tilde{G}$.

The input graph signal matrix is passed through a single 1-D convolutional
layer to map the input to a higher dimensional space. Hence, the output of the
1-D convolutional layer is the spatial-temporal synchronous graph signal
matrix $\tilde{X}\in\mathbb{R}^{M\times N\times C_{in}}$ where $C_{in}>F$, and
then $\tilde{X}$ is passed on to the STSGT layers.

####  3.3.1. Spatial-Temporal Synchronous Graph Transformer layer

The architecture of the STSGT Layer is illustrated in Figure 5.

Figure 5. The architecture of one Spatial-Temporal Synchronous Graph
Transformer layer. The adjacency matrix of the spatial-temporal synchronous
graph $\tilde{A}_{updated}$ is used by the GCN.

Each layer comprises of the Spatial-Temporal Synchronous Transformer (STST)
for calculating the self-attention between the vertices of the spatial-
temporal synchronous graph $\tilde{G}$. A conventional GCN follows the STST to
aggregate the information obtained through self-attention mechanism, from
neighboring vertices of all vertices of $\tilde{X}$. A GCN layer on the output
of STST is defined as:

(4)
$\tilde{X^{\prime\prime}}=\sigma\left(\tilde{A}_{updated}\tilde{X^{\prime}}W+b\right)$

where $\tilde{X^{\prime}}\in\mathbb{R}^{M\times N\times C^{\prime}}$ is the
output of STST, $\tilde{A}_{updated}\in\mathbb{R}^{MN\times MN}$ is the
updated spatial-temporal synchronous adjacency matrix,
$W\in\mathbb{R}^{C^{\prime}\times C^{\prime\prime}}$ is the learnable weight
matrix, $b\in\mathbb{R}^{C^{\prime\prime}}$ is the learnable bias, $\sigma$ is
the non-linearity such as ReLU applied after the graph convolution, and
$\tilde{X^{\prime\prime}}\in\mathbb{R}^{M\times N\times C^{\prime\prime}}$ is
the output of the GCN layer.

Additionally, residual connections are added as shown in Figure 5 to allow the
gradients to back-propagate to the initial layers of the model during the
training process. Finally, the output of a STSGT layer is an updated spatial-
temporal graph signal matrix $\tilde{X^{\prime\prime}}\in\mathbb{R}^{M\times
N\times C^{\prime\prime}}$ which is passed on to the next STSGT layer.

####  3.3.2. Temporal and Spatial Encodings

Temporal and spatial encodings are added to the input spatial-temporal graph
signal matrix $\tilde{X}\in\mathbb{R}^{M\times N\times C_{in}}$, before
passing it on to the first STSGT layer. This helps the model to distinguish
between the different time-steps (temporal encodings) and the different
vertices of a given time-step (spatial encodings). These temporal and spatial
encodings are added as follows:

(5) $\tilde{X}=\tilde{X}+T_{enc}+S_{enc}$

where the graph signal matrix after temporal and spatial encoding is
$\tilde{X}\in\mathbb{R}^{M\times N\times C_{in}}$. The temporal encoding added
is $T_{enc}\in\mathbb{R}^{M\times 1\times C_{in}}$ which basically assigns
same values for all vertices $N$ of a given time-step $t$ and different values
for vertices in different time-steps. The spatial encoding added is
$S_{enc}\in\mathbb{R}^{1\times N\times C_{in}}$ which basically assigns same
values for a given vertex $N$ across all time-steps and different values for
vertices in the same time-step $t$.

####  3.3.3. Spatial-Temporal Synchronous Self-Attention

The architecture of the Spatial-Temporal Synchronous Transformer (STST) within
each STSGT layer is illustrated in Figure 6 (left). First, the layer
normalization is applied to the input graph signal matrix
$\tilde{X}\in\mathbb{R}^{M\times N\times C_{in}}$. Subsequently, the output of
the layer norm is passed to a multi-head spatial-temporal synchronous self-
attention mechanism as illustrated in Figure 6 (right). Layer normalization is
applied again to the output of the multi-head self-attention. Finally, a three
layer Multi-Layer Perceptron (MLP) is used to output the encoded graph signal
matrix $\tilde{X^{\prime}}\in\mathbb{R}^{M\times N\times C^{\prime}}$.

The architecture of each head of the spatial-temporal synchronous self-
attention mechanism is illustrated in Figure 6 (right). The input to each head
is the spatial-temporal graph signal matrix $\tilde{X}\in\mathbb{R}^{M\times
N\times C_{in}}$. This matrix is reshaped to $\tilde{X}\in\mathbb{R}^{MN\times
C_{in}}$. Then $\tilde{X}$ is projected to three higher dimensional latent
subspaces by generating the Query(Q), Key(K) and Value(V) matrices. The Q, K
and V matrices are generated as follows:

(6) $Q=\tilde{X}W_{Q}\qquad K=\tilde{X}W_{K}\qquad V=\tilde{X}W_{V}$

where, $Q\in\mathbb{R}^{MN\times d_{q}}$ and $W_{Q}\in\mathbb{R}^{C_{in}\times
d_{q}}$, $K\in\mathbb{R}^{MN\times d_{k}}$ and
$W_{K}\in\mathbb{R}^{C_{in}\times d_{k}}$, $V\in\mathbb{R}^{MN\times d_{v}}$
and $W_{V}\in\mathbb{R}^{C_{in}\times d_{v}}$. Here, $d_{q}=d_{k}=d{v}$ and
the projection matrices $W_{Q},W_{K},W_{V}$ are feed forward neural networks.

Figure 6. (left) The architecture of STST and (right) The architecture of one
head of the Spatial-Temporal Synchronous Self-Attention within each STST.

In a similar manner as the encoder of the transformer (Vaswani et al., 2017),
we calculate the similarity or attention score $S$ between the query and key
matrices in the higher dimensional latent subspace. The attention score $S$ is
calculated as:

(7) $S=softmax\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)$

where, $S\in\mathbb{R}^{MN\times MN}$ is the attention score obtained from
self-attention between all the vertices of the spatial-temporal synchronous
graph $\tilde{G}$ with graph signal matrix $\tilde{X}$. $Q$ and $K$ are the
query and key matrices respectively, and $d_{k}$ is the dimension of the key
matrix $K$. The softmax function normalizes the scores and the scaling factor
$\frac{1}{\sqrt{d_{k}}}$ prevents the saturation due to the softmax function.

Finally, the self-attention scores are multiplied by the value matrix $V$ to
update the features of each vertex of $\tilde{X}$. The update is done as
follows:

(8) $L=SV$

where $S$ and $V$ are the attention score and value matrix respectively. The
matrix $L\in\mathbb{R}^{MN\times d_{v}}$ obtained with equation 8 is passed
through a fully connected layer to update the output spatial-temporal
synchronous graph signal matrix $\tilde{X}_{enc}\in\mathbb{R}^{MN\times C}$.

####  3.3.4. Forecasting the future COVID status

We forecast the future values of the spatial-temporal COVID time series with
an output layer (see Figure 4) which consists of two 1D convolution layers to
map the output of the last STSGT layer to the original dimension of the time
series data. Let us denote the output of the last STSGT layer in Figure 4 as
$\tilde{X}\in\mathbb{R}^{M\times N\times C^{\prime}}$, which is then
transposed and reshaped to $\tilde{X}^{T}\in\mathbb{R}^{N\times MC^{\prime}}$.
We then use two 1D convolution layers on $\tilde{X}^{T}$ as follows:

(9) $\hat{X}=\sigma\left(\tilde{X}^{T}W_{1}+b_{1}\right).W_{2}+b_{2}$

where, $\hat{X}\in\mathbb{R}^{N\times H}$ is output of our STSGT model,
$\sigma$ is the ReLU non-linearity, $W_{1}\in\mathbb{R}^{MC^{\prime}\times
C^{\prime\prime}}$ and $b_{1}\in\mathbb{R}^{C^{\prime\prime}}$ are the
learnable weights and bias of the first 1D convolution layer and
$W_{2}\in\mathbb{R}^{C^{\prime\prime}\times H}$ and $b_{2}\in\mathbb{R}^{H}$
are the learnable weights and bias of the second 1D convolution layer
respectively. Here, $C^{\prime\prime}$ is the hidden layer dimension of the
output layers and $H$ is the length of the forecasting horizon. Finally, the
output $\hat{X}$ is transposed to $\hat{X}^{T}\in\mathbb{R}^{H\times N}$.

It should be noted that we forecast all the $H$ horizon time-steps
$\hat{X}_{(t+1)},\hat{X}_{(t+2)},...,\hat{X}_{(t+H)}$ at the same time instead
of recursively forecasting one of the horizon time-step at a given time as
done in previous works (Song et al., 2020; Li et al., 2018b; Yu et al., 2018).

##  4\. Experiments

We evaluated our STSGT framework with three main tasks: 1) forecasting the
COVID-19 daily infected cases for the 50 states of US and Washington, D.C.; 2)
forecasting the COVID-19 daily infected cases at the state level with 83
counties for the state of Michigan; and 3) forecasting the COVID-19 daily
death cases for the 50 states of US and Washington, D.C. We compared the
performance of our proposed approach with baselines and state-of-the-art
spatial-temporal forecasting algorithms. All our experiments were performed
using the PyTorch 1.8.0 framework (Paszke et al., 2019).

###  4.1. Datasets

####  4.1.1. John Hopkins University (JHU) COVID-19 Dataset

We used the publicly available COVID-19 Data Repository by the Center for
Systems Science and Engineering (CSSE) at Johns Hopkins University
(https://github.com/CSSEGISandData/COVID-19) (Dong et al., 2020).
Specifically, we used the time series dataset which is updated regularly with
the number of infected and death cases across all states of the US. We used
the time series data from March 15, 2020 to Nov 30, 2021 spanning 626 days.

####  4.1.2. New York Times (NYT) COVID-19 Dataset

We used the publicly available data from The New York Times, based on reports
from state and local health agencies (NYT, 2021). Similar to the JHU dataset,
we used the time series data for the daily infected and death cases across all
US states. We used the time series data from March 18, 2020 to Nov 30, 2021
spanning 623 days.

###  4.2. Comparison Methods

####  4.2.1. Baseline Method

We used the following baseline method for spatial-temporal forecasting, to
compare with our proposed approach.

  * •

Auto Regressive Integrated Moving Average (ARIMA) (Li et al., 2018b) \- ARIMA
consists of an auto-regressive (AR) and a moving average (MA) component and is
used to forecast future values of a stationary time series. For our
experiments, we used an order of (5, 1, 0) for ARIMA. That is, we used 5 AR
and 0 MA components with first order difference (d=1) on the input time
series. We used ARIMA as a baseline comparison method since it is one of the
most common algorithms for time-series analysis, and it has been extensively
used as a comparison method by several state-of-the-art spatial-temporal
forecasting algorithms.

####  4.2.2. State-of-the-art Spatial-temporal Forecasting Algorithms

We compared our proposed approach with the following state-of-the-art methods
that were designed for spatial-temporal forecasting. For each of the models
below, the number of features or channels for each vertex of the graph is 1,
which is the daily infected or death cases. The number of vertices for the
graph per time-step is 51 (50 US states and Washington, D.C.) and 83 (for
Michigan). The history (M) and horizon (H) for the sliding window is selected
as 12 for all the spatial-temporal forecasting algorithms below. The default
values are used for the hyper-parameters in each model, which we detailed in
their descriptions below.

  * •

Spatial-Temporal Graph Convolution Network (STGCN) (Yu et al., 2018) \- STGCN
uses the spatial-temporal convolution blocks to forecast the horizon. Each of
its spatial-temporal blocks consists of spectral graph convolution to capture
the spatial dependency between vertices of same time-step and an independent
temporal gated convolution block to capture the temporal dependency between
the time-steps. We used the STGCN implementation
(https://github.com/FelixOpolka/STGCN-PyTorch) with 16 spatial and output
channels for each STGCN block.

  * •

Attention-based Spatial-Temporal Graph Convolution Network (ASTGCN(r)) (Guo et
al., 2019) \- ASTGCN concatenates the output of several spatial-temporal
blocks used independently on the hourly, daily and weekly spatial-temporal
data to forecast the next hour status. Each of the spatial-temporal blocks
consists of spectral graph convolution to capture the spatial dependency and a
combination of spatial and temporal attention mechanism to capture the dynamic
spatial and temporal correlations of the spatial-temporal data. We used only
the recent component of ASTGCN (i.e., ASTGCN(r)), equivalent to only using the
hourly data, to have a fair comparison with other methods. We used the ASTGCN
implementation (https://github.com/guoshnBJTU/ASTGCN-r-pytorch) with 2 ASTGCN
blocks, 16 time filters with stride 1 and 16 Chebyshev filters of order 2.

  * •

Graph WaveNet (Wu et al., 2019) \- Graph WaveNet comprises of multiple layers
with skip connections where each layer consists of two gated temporal
convolutional networks (TCN) (Lea et al., 2016). A dilation factor is used for
each TCN to consider the long-range temporal dependencies of the time series.
The gated TCN is followed by an independent graph convolutional layer to
capture the spatial dependencies. We used the Graph WaveNet implementation
(https://github.com/nnzhan/Graph-WaveNet) with the “double transition”
adjacency matrix. We used 2 blocks with each block having 2 layers. We used 16
residual and dilation channels along with 128 skip channels and 256 end
channels. A $(1,2)$ kernel size was used for the filter and gate convolutions.

  * •

Spatial-Temporal Transformer Network (STTN) (Xu et al., 2020) \- STTN has
several spatial-temporal blocks, and each of these blocks consists of an
independent spatial and temporal transformer. The spatial transformer combines
self-attention with a GCN using a gated mechanism to capture the spatial
dependency. The output of the spatial transformer is passed to the temporal
transformer which uses the self-attention mechanism to capture the temporal
dependencies. We used the STTN implementation (https://github.com/wubin5/STTN)
with 2 layers of spatial and temporal transformer, embedding size of 16 for
each vertex of the graph, a 400-dimensional temporal encoding for the temporal
transformer, a Chebyshev polynomial of order 3 for the spatial GCN and 2 heads
for the multi-head self-attention for both spatial and temporal transformer.

###  4.3. Evaluation Metrics

In this paper, we use the following three metrics to evaluate the performance
of all the models.

####  4.3.1. Mean Absolute Error (MAE)

The Mean Absolute Error (MAE) is defined as:

(10) $MAE=\frac{1}{N}\sum_{i}\left(\mid y_{t}[i]-y_{p}[i]\mid\right)$

####  4.3.2. Root Mean Square Log Error (RMSLE)

The Root Mean Square Log Error (RMSLE) is defined as:

(11)
$RMSLE=\sqrt{\frac{1}{N}\sum_{i}\left[\log(y_{t}[i]+1)-\log(y_{p}[i]+1)\right]^{2}}$

where, $\log$ denotes the natural logarithm.

####  4.3.3. Root Mean Square Error (RMSE)

The Root Mean Square Error (RMSE) is defined as:

(12) $RMSE=\sqrt{\frac{1}{N}\sum_{i}\left(y_{t}[i]-y_{p}[i]\right)^{2}}$

In all of the above equations, $N$ is the total number of samples in the
dataset, $y_{t}$ is the number of ground truth cases and $y_{p}$ is the
forecasted number of cases. A lower MAE, RMSLE and RMSE indicate that the
forecasted number of cases is closer to the ground truth number of cases.

###  4.4. Implementation Details

For all our experiments, we used the past 12 days history (M=12) to forecast
the next 12 days horizon (H=12) using the sliding window approach shown in
Figure 1. The rationale behind the choice of 12 days history and horizon is
that most of the recent state-of-the-art spatial-temporal forecasting
algorithms (Guo et al., 2019),(Wu et al., 2019),(Yu et al., 2018) use the
previous 12 time-steps to forecast the next 3, 6, 9 or 12 time-steps. We
generate the adjacency matrix $A$ for the spatial graph $G$ at a given time-
step by the euclidean distances between the states of the US. This distance is
calculated by considering the coordinates of each state which in turn is the
average of the latitude and longitude of all the counties for a given state.
After generating the adjacency matrix, we normalized it with the maximum value
and used a maximum threshold value of $0.3$ to generate the final adjacency
matrix for all US states and the counties at the state level. The same
adjacency matrix was used for both JHU and NYT datasets to forecast the daily
infected and death cases for all 50 US states and Washington, D.C. A separate
adjacency matrix was generated at the state level, e.g. the state of Michigan,
to forecast the daily infected cases for all the counties.

We divided the entire dataset in chronological order with $80\%$ training,
$10\%$ validation and $10\%$ testing. In other words, we used the data from
March 15, 2020 to July 28, 2021 as the training set, July 29 to Sept 28, 2021
as the validation set and Sept 29 to Nov 30, 2021 as the testing set. For the
NYT dataset, March 18, 2020 was used as the start date as the cases for one of
the states were not reported till this date. It should be noted that cross-
validation cannot be performed since the validation dataset has to follow the
chronological order. Z-score normalization was applied to the inputs, i.e.,
the train, validation and test datasets were separately normalized by
subtracting the mean and dividing by the standard deviation. We used the Mean
Absolute Error (MAE) as the loss function and trained all the models for $100$
epochs with early stopping by monitoring the MAE on the validation set. We
clipped the gradients to $L2=5$ and used a batch size of 16 for training all
models with a learning rate of $0.001$ without using any learning rate decay.

The performance of our proposed STSGT framework is dependent on several hyper-
parameters. These hyper-parameters include the number of STSGT layers, the
number of attention heads for multi-head attention in each STST layer, the
dimension of the Query(Q), Key(K) and Value(V) matrices for calculating the
self-attention within each attention head, and the length of the history time-
steps (M) for 12 horizon time-steps (H=12) in the sliding window (see Figure
1). We empirically decided these hyper-parameters such that they are
consistent with other spatial-temporal forecasting algorithms and thus we can
have a fair comparison. Specifically, we used 2 STSGT layers in our model.
Within each STSGT layer, we used 2 STST+GCN layer combinations. For the self-
attention, we selected the Query(Q), Key(K) and Value(V) dimensions to be 16
with 2 attention heads in each of the STST layers. Additionally, for the
sliding window with 12 horizon time-steps (H=12), we choose 12 historical
time-steps (M=12).

###  4.5. Experimental Results

####  4.5.1. Forecasting JHU COVID-19 Daily Infected Cases for US States

The results of using our proposed STSGT model and other models to forecast the
daily infected cases for the 50 US states and Washington, D.C. using the JHU
dataset are summarized in Table 1. We summed the daily infected cases for all
counties of a given state to generate the ground truth number of infected
cases for that state. Clearly, our proposed STSGT significantly outperformed
all the other models by achieving the lowest MAE, RMSLE and RMSE for both
short-term horizon (next day, $H=1$) and long-term horizon indicated by the
mean over the next 12 days ($H=12$). When comparing with the second best model
ASTGCN(r), we observe that the mean MAE over the long-term, i.e. 12 days
horizon, reduces by 131.22 while the RMSLE and RMSE reduces by 0.081 and
397.81 respectively. An interesting observation is that ASTGCN(r) and our
proposed STSGT can better forecast the longer time horizons for all the states
of US, compared to other models as seen by a lower 12 days mean MAE when
compared to day 1 MAE.

Table 1. Results demonstrating the MAE, RMSLE and RMSE of the daily infected cases of 50 US states and Washington, D.C. with JHU data. The lowest MAE, RMSLE and RMSE are marked in bold. Algorithm |  | Forecasting
---
Horizon
MAE | RMSLE | RMSE
ARIMA (Li et al., 2018b) | Day 1 $(H=1)$ | 2204.42 | 5.996 | 3724.19
12 Days Mean | 2206.14 | 6.069 | 3895.20
STGCN (Yu et al., 2018) | Day 1 $(H=1)$ | 1243.48 | 0.869 | 2518.78
12 Days Mean | 1296.86 | 0.905 | 2369.93
ASTGCN(r) (Guo et al., 2019) | Day 1 $(H=1)$ | 1078.06 | 0.822 | 2369.56
12 Days Mean | 1076.29 | 0.903 | 2102.68
Graph WaveNet (Wu et al., 2019) | Day 1 $(H=1)$ | 1140.02 | 0.917 | 2203.16
12 Days Mean | 1315.56 | 0.969 | 2263.27
STTN (Xu et al., 2020) | Day 1 $(H=1)$ | 1172.34 | 0.948 | 2475.33
12 Days Mean | 1198.16 | 0.984 | 2191.91
STSGT (ours) | Day 1 $(H=1)$ | 978.390 | 0.802 | 2129.74
12 Days Mean | 945.070 | 0.822 | 1704.87

A comparison of the average infected cases over the next 12 days horizon
$(H=12)$ for four of the most affected US states due to COVID, is shown in
Figure 7. We compared the average ground truth and forecasted average infected
cases for the states of New York, Florida, California and Texas for a period
of 12 days ranging from Nov 6-17, 2021, which is a time frame from our testing
set. We use the past 12 days historical data $(M=12)$ ranging from Oct 25-Nov
5, 2021, to forecast the next 12 days horizon $(H=12)$ infected cases (Nov
6-17, 2021). In Figure 7, we observe that the average number of forecasted
infected cases with our proposed STSGT is much closer to the ground truth
number of average infected cases for most of the states, when compared to
state-of-the-art spatial-temporal forecasting methods.

Figure 7. A comparison of the average infected cases for the next 12 days
horizon $(H=12)$ for four US states.

####  4.5.2. Forecasting JHU COVID-19 Daily Death Cases for US States

The results of using our proposed STSGT model and other models to forecast the
daily death cases for the 50 US states and Washington, D.C. using the JHU
dataset are summarized in Table 2. Similar to Table 1, we summed the daily
death cases for all counties of a given state to generate the ground truth
number of death cases for that state. Our proposed STSGT outperformed all the
other models by achieving the lowest MAE for the short-term horizon (H=1) and
the lowest MAE, RMSLE and RMSE for the long-term horizon indicated by the mean
over the next 12 days (H=12). When comparing with the second best model
ASTGCN(r), we observe that the mean MAE over the long-term, i.e. 12 days
horizon, reduces by 0.73 while the RMSLE and RMSE reduces by 0.005 and 0.11
respectively. We observe that even though ASTGCN(r) achieves lower RMSLE and
RMSE for forecasting the next day (day 1) death cases, our proposed model
STSGT could better forecast the death cases over the long-term (12 days) as
observed by a lower MAE, RMSLE and RMSE.

Table 2. Results demonstrating the MAE, RMSLE and RMSE of the daily death cases of 50 US states and Washington, D.C. with JHU data. The lowest MAE, RMSLE and RMSE are marked in bold. Algorithm |  | Forecasting
---
Horizon
MAE | RMSLE | RMSE
ARIMA (Li et al., 2018b) | Day 1 $(H=1)$ | 42.62 | 2.753 | 74.89
12 Days Mean | 43.66 | 2.698 | 110.60
STGCN (Yu et al., 2018) | Day 1 $(H=1)$ | 26.25 | 0.966 | 55.58
12 Days Mean | 25.23 | 0.952 | 54.15
ASTGCN(r) (Guo et al., 2019) | Day 1 $(H=1)$ | 21.20 | 0.849 | 47.73
12 Days Mean | 21.36 | 0.873 | 47.25
Graph WaveNet (Wu et al., 2019) | Day 1 $(H=1)$ | 23.02 | 0.889 | 48.99
12 Days Mean | 25.03 | 0.922 | 51.16
STTN (Xu et al., 2020) | Day 1 $(H=1)$ | 21.31 | 0.863 | 48.52
12 Days Mean | 22.05 | 0.892 | 48.23
STSGT (ours) | Day 1 $(H=1)$ | 20.73 | 0.855 | 48.30
12 Days Mean | 20.63 | 0.868 | 47.14

####  4.5.3. Forecasting JHU COVID-19 Daily Infected Cases for Michigan

The results of using our proposed STSGT model and other models to forecast the
daily infected cases at the state level, e.g., the state of Michigan, using
the JHU dataset, are summarized in Table 3. In this case, the daily infected
cases of the individual counties of Michigan served as the ground truth. We
excluded the correctional facilities, counties denoted as “Out of Michigan”
and unassigned counties, thus resulting in 83 counties in total. Hence, we
used 83 vertices for the graph per time-step (i.e. for each day). As shown in
Table 3, our proposed STSGT significantly outperformed all the other models by
achieving the lowest MAE, RMSLE and RMSE for both short-term horizon (H=1) and
long-term horizon indicated by the mean over the next 12 days (H=12). When
comparing with the second best model STTN, we observe that the mean MAE over
the long-term horizon (12 days) reduces by 9.86 while the RMSLE and RMSE
reduces by 0.267 and 17.49 respectively. This experiment clearly demonstrates
that our model could perform very well at the state level and hence our model
could be used for forecasting infected and death cases for any US state.

Table 3. Results demonstrating the MAE, RMSLE and RMSE of the daily infected cases for 83 counties of the state of Michigan with JHU data. The lowest MAE, RMSLE and RMSE are marked in bold. Algorithm |  | Forecasting
---
Horizon
MAE | RMSLE | RMSE
ARIMA (Li et al., 2018b) | Day 1 $(H=1)$ | 124.52 | 3.851 | 270.78
12 Days Mean | 142.46 | 4.006 | 322.06
STGCN (Yu et al., 2018) | Day 1 $(H=1)$ | 48.30 | 0.739 | 95.99
12 Days Mean | 57.31 | 0.753 | 126.73
ASTGCN(r) (Guo et al., 2019) | Day 1 $(H=1)$ | 47.50 | 0.772 | 97.14
12 Days Mean | 55.20 | 0.785 | 122.10
Graph WaveNet (Wu et al., 2019) | Day 1 $(H=1)$ | 41.34 | 0.658 | 87.97
12 Days Mean | 64.05 | 0.879 | 143.61
STTN (Xu et al., 2020) | Day 1 $(H=1)$ | 42.50 | 0.812 | 81.37
12 Days Mean | 49.33 | 0.810 | 108.31
STSGT (ours) | Day 1 $(H=1)$ | 22.04 | 0.430 | 40.78
12 Days Mean | 39.47 | 0.543 | 90.82
Figure 8. (top) Daily Infected cases for Wayne County, Michigan and (bottom)
Daily Infected cases for Oakland County, Michigan from October 4 to November
5, 2021.

The two most populated counties in Michigan are Wayne county with a population
over 1.7 million and Oakland county with a population over 1.2 million. We
plotted the ground truth and forecasted number of infected cases for Wayne and
Oakland counties for a time frame in the testing set ranging from Oct 4-Nov 5,
2021 as shown in Figure 8. We use the 12 days historical data $(M=12)$ to
forecast the next day infected cases $(H=1)$ in Figure 8. Specifically, we use
the data from Sept 22-Oct 3 to forecast the infected cases for Oct 4, then the
data from Sept 24-Oct 5 to forecast the infected cases for Oct 6 and so on. We
removed the days when the ground truth infected cases were zero as these cases
were not reported in the JHU dataset for Michigan. In Figure 8, we compare
STSGT with other spatial-temporal forecasting algorithms and clearly observe
that the forecasted number of infected cases with STSGT is much closer to the
ground truth number of infected cases.

####  4.5.4. Forecasting NYT COVID-19 Daily Infected Cases for US States

The results of using our proposed STSGT model and other models to forecast the
daily infected cases for the 50 US states and Washington, D.C. using the NYT
dataset are summarized in Table 4. For the NYT dataset, the daily infected
cases were reported for each state thereby eliminating the need for the
county-wise summation process as done for the JHU dataset. Clearly, our
proposed STSGT significantly outperformed all the other models by achieving
the lowest MAE and RMSE for the short-term horizon (H=1) and the lowest MAE,
RMSLE and RMSE for the long-term horizon indicated by the mean over the next
12 days (H=12). When comparing with the second best model ASTGCN(r), we
observe that the mean MAE over the long-term horizon (12 days) reduces by
52.77 while the RMSLE and RMSE reduces by 0.010 and 136.08 respectively. This
clearly shows the advantage of our model for forecasting over the long-term
horizon.

Table 4. Results demonstrating the MAE, RMSLE and RMSE of the daily infected cases of 50 US states and Washington, D.C. with NYT data. The lowest MAE, RMSLE and RMSE are marked in bold. Algorithm |  | Forecasting
---
Horizon
MAE | RMSLE | RMSE
ARIMA (Li et al., 2018b) | Day 1 $(H=1)$ | 2152.48 | 5.935 | 3311.26
12 Days Mean | 2185.04 | 5.968 | 3676.43
STGCN (Yu et al., 2018) | Day 1 $(H=1)$ | 1340.13 | 0.975 | 2303.10
12 Days Mean | 1384.35 | 0.997 | 2357.91
ASTGCN(r) (Guo et al., 2019) | Day 1 $(H=1)$ | 939.840 | 0.852 | 1582.46
12 Days Mean | 976.420 | 0.879 | 1626.42
Graph WaveNet (Wu et al., 2019) | Day 1 $(H=1)$ | 960.840 | 0.854 | 1506.59
12 Days Mean | 1173.00 | 0.919 | 1905.55
STTN (Xu et al., 2020) | Day 1 $(H=1)$ | 1125.88 | 0.978 | 1794.09
12 Days Mean | 1177.85 | 1.029 | 1854.19
STSGT (ours) | Day 1 $(H=1)$ | 903.590 | 0.864 | 1465.55
12 Days Mean | 923.650 | 0.869 | 1490.34

####  4.5.5. Forecasting NYT COVID-19 Daily Death Cases for US States

The results of using our proposed STSGT model and other models to forecast the
daily death cases for the 50 US states and Washington, D.C. using the NYT
dataset are summarized in Table 5. Similar to Table 4, the daily death cases
were reported for each state thereby eliminating the need for the county-wise
summation process as done for the JHU dataset. Clearly, our proposed STSGT
significantly outperformed all the other models by achieving the lowest MAE
and RMSLE for the short-term horizon ($H=1$) and the lowest MAE, RMSLE and
RMSE for the long-term horizon indicated by the mean over the next 12 days
($H=12$). When comparing with the second best model ASTGCN(r) for the long-
term (12 days mean), we observe that the mean MAE over 12 days horizon reduces
by 2.02 while the RMSLE and RMSE reduces by 0.025 and 0.02 respectively.

Table 5. Results demonstrating the MAE, RMSLE and RMSE of the daily death cases of 50 US states and Washington, D.C. with NYT data. The lowest MAE, RMSLE and RMSE are marked in bold. Algorithm |  | Forecasting
---
Horizon
MAE | RMSLE | RMSE
ARIMA (Li et al., 2018b) | Day 1 $(H=1)$ | 49.25 | 2.693 | 107.41
12 Days Mean | 49.36 | 2.661 | 138.96
STGCN (Yu et al., 2018) | Day 1 $(H=1)$ | 28.86 | 0.949 | 67.85
12 Days Mean | 28.43 | 0.959 | 63.87
ASTGCN(r) (Guo et al., 2019) | Day 1 $(H=1)$ | 25.90 | 0.944 | 62.07
12 Days Mean | 24.40 | 0.932 | 57.47
Graph WaveNet (Wu et al., 2019) | Day 1 $(H=1)$ | 24.73 | 0.921 | 60.07
12 Days Mean | 26.18 | 0.965 | 60.57
STTN (Xu et al., 2020) | Day 1 $(H=1)$ | 29.85 | 1.091 | 69.55
12 Days Mean | 28.09 | 1.079 | 62.97
STSGT (ours) | Day 1 $(H=1)$ | 22.88 | 0.912 | 61.39
12 Days Mean | 22.38 | 0.907 | 57.45

##  5\. Discussion and Future Work

The present study investigated the impact and utility of spatial-temporal
forecasting for COVID-19 time series analysis. Our proposed novel STSGT could
accurately forecast the daily infected and death cases across the 50 US states
and Washington, D.C. STSGT also performed extremely well at the state level by
accurately forecasting the daily infected cases for all the counties of
Michigan. Hence, our model could help the common people and the government by
taking necessary precautions so that COVID could be prevented.

It should be noted that we have selected mid-March (March 15 or 18, 2020) as
the start date for our analysis as the pandemic started in almost all of the
US states around that date, and there were significant number of infected and
death cases after mid-March. We chose Nov 30, 2021 as the end date of our
analysis as the Omicron variant of COVID started dominating after this date,
and the number of daily cases spiked to a new high. Hence, the daily cases
from around December 1, 2021 until the end of Omicron wave (around Feb 15,
2022) formed a different pattern than what we observed from the start of the
pandemic. We believe that our model needs to be re-trained again with the
Omicron wave’s new data pattern to achieve accurate forecasting, and we leave
this for future work. In future, we also plan to incorporate an efficient
self-attention mechanism to further accelerate the training process of STSGT
while maintaining similar performance as our current approach.

##  6\. Conclusion

In this paper, we proposed a novel model which can synchronously capture the
complex spatial-temporal dependencies in the COVID data using the multi-head
self-attention. We show through extensive experiments on two real-world public
datasets that our model can outperform the existing models specifically
designed for spatial-temporal forecasting. Our proposed model could have a
significant impact on society by making the common people more aware and help
the government eradicate this pandemic in the near future.

###### Acknowledgements.

This work was partially supported by National Science Foundation (Grant No.
2027251).

## References

  * (1)
  * NYT (2021) 2021\.  The New York Times (2021). Coronavirus (Covid-19) Data in the United States.  https://github.com/nytimes/covid-19-data.  Retrieved [03-April-2022].
  * Awan et al. (2021) Mazhar Javed Awan, Muhammad Haseeb Bilal, Awais Yasin, Haitham Nobanee, Nabeel Sabir Khan, and Azlan Mohd Zain. 2021.  Detection of COVID-19 in chest X-ray images: A big data enabled deep learning approach.  _Int. J. Environ. Res. Public Health_ 18, 19 (Sept. 2021), 10147.
  * Banerjee et al. (2020) Soumyanil Banerjee, Ming Dong, Min-Hee Lee, Nolan O’Hara, Csaba Juhasz, Eishi Asano, and Jeong-Won Jeong. 2020.  Deep relational reasoning for the prediction of language impairment and postoperative seizure outcome using preoperative DWI connectome data of children with focal epilepsy.  _IEEE Transactions on Medical Imaging_ 40, 3 (2020), 793–804.
  * Bartik et al. (2020) Alexander W. Bartik, Marianne Bertrand, Zoe Cullen, Edward L. Glaeser, Michael Luca, and Christopher Stanton. 2020.  The impact of COVID-19 on small business outcomes and expectations.  _Proceedings of the National Academy of Sciences_ 117, 30 (2020), 17656–17666.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chiang et al. (2019) Wei-Lin Chiang, Xuanqing Liu, Si Si, Yang Li, Samy Bengio, and Cho-Jui Hsieh. 2019\.  Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks. In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 257–266.
  * Chung et al. (2014) Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _arXiv preprint arXiv:1412.3555_ (2014).
  * Cucinotta and Vanelli (2020) Domenico Cucinotta and Maurizio Vanelli. 2020.  WHO declares COVID-19 a pandemic.  _Acta Biomed._ 91, 1 (March 2020), 157–160.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Dong et al. (2020) Ensheng Dong, Hongru Du, and Lauren Gardner. 2020.  An interactive web-based dashboard to track COVID-19 in real time.  _Lancet Infect. Dis._ 20, 5 (May 2020), 533–534.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).
  * Fout et al. (2017) Alex Fout, Jonathon Byrd, Basir Shariat, and Asa Ben-Hur. 2017\.  Protein interface prediction using graph convolutional networks.  _Advances in neural information processing systems_ 30 (2017).
  * Guo et al. (2019) Shengnan Guo, Youfang Lin, Ning Feng, Chao Song, and Huaiyu Wan. 2019.  Attention based spatial-temporal graph convolutional networks for traffic flow forecasting. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 922–929.
  * Hamilton et al. (2017) William L. Hamilton, Rex Ying, and Jure Leskovec. 2017\.  Inductive Representation Learning on Large Graphs. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_. 1025–1035.
  * Harapan et al. (2020) Harapan Harapan, Naoya Itoh, Amanda Yufika, Wira Winardi, Synat Keam, Haypheng Te, Dewi Megawati, Zinatul Hayati, Abram L. Wagner, and Mudatsir Mudatsir. 2020.  Coronavirus disease 2019 (COVID-19): A literature review.  _Journal of Infection and Public Health_ 13, 5 (2020), 667–673.
  * Hochreiter (1998) Sepp Hochreiter. 1998\.  The vanishing gradient problem during learning recurrent neural nets and problem solutions.  _International Journal of Uncertainty, Fuzziness and Knowledge-Based Systems_ 6, 02 (1998), 107–116.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ 9, 8 (1997), 1735–1780.
  * Kapoor et al. (2020) Amol Kapoor, Xue Ben, Luyang Liu, Bryan Perozzi, Matt Barnes, Martin Blais, and Shawn O’Banion. 2020.  Examining covid-19 forecasting using spatio-temporal graph neural networks.  _arXiv preprint arXiv:2007.03113_ (2020).
  * Kipf and Welling (2017) Thomas N Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  _In: International Conference on Learning Representations_ (2017).
  * Koh et al. (2021) Howard K. Koh, Alan C. Geller, and Tyler J. VanderWeele. 2021\.  Deaths From COVID-19.  _JAMA_ 325, 2 (01 2021), 133–134.
  * Lea et al. (2016) Colin Lea, Rene Vidal, Austin Reiter, and Gregory D Hager. 2016\.  Temporal convolutional networks: A unified approach to action segmentation. In _European Conference on Computer Vision_. Springer, 47–54.
  * Li et al. (2019) G. Li, M. Müller, A. Thabet, and B. Ghanem. 2019\.  DeepGCNs: Can GCNs Go As Deep As CNNs?. In _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_. 9266–9275.
  * Li and Zhu (2021) Mengzhang Li and Zhanxing Zhu. 2021.  Spatial-temporal fusion graph neural networks for traffic flow forecasting. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 35. 4189–4196.
  * Li et al. (2018a) Qimai Li, Zhichao Han, and Xiao-Ming Wu. 2018a.  Deeper insights into graph convolutional networks for semi-supervised learning.  _arXiv preprint arXiv:1801.07606_ (2018).
  * Li et al. (2018b) Yaguang Li, Rose Yu, Cyrus Shahabi, and Yan Liu. 2018b.  Diffusion Convolutional Recurrent Neural Network: Data-Driven Traffic Forecasting. In _International Conference on Learning Representations (ICLR ’18)_.
  * Minaee et al. (2020) Shervin Minaee, Rahele Kafieh, Milan Sonka, Shakib Yazdani, and Ghazaleh Jamalipour Soufi. 2020\.  Deep-COVID: Predicting COVID-19 from chest X-ray images using deep transfer learning.  _Medical Image Analysis_ 65 (2020), 101794.
  * Pak et al. (2020) Anton Pak, Oyelola A. Adegboye, Adeshina I. Adekunle, Kazi M. Rahman, Emma S. McBryde, and Damon P. Eisen. 2020.  Economic Consequences of the COVID-19 Outbreak: the Need for Epidemic Preparedness.  _Frontiers in Public Health_ 8 (2020).
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems 32_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024–8035.
  * Qi et al. (2017) Xiaojuan Qi, Renjie Liao, Jiaya Jia, Sanja Fidler, and Raquel Urtasun. 2017.  3d graph neural networks for rgbd semantic segmentation. In _Proceedings of the IEEE International Conference on Computer Vision_. 5199–5208.
  * Rong et al. (2019) Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. 2019\.  Dropedge: Towards deep graph convolutional networks on node classification. In _International Conference on Learning Representations_.
  * Song et al. (2020) Chao Song, Youfang Lin, Shengnan Guo, and Huaiyu Wan. 2020\.  Spatial-temporal synchronous graph convolutional networks: A new framework for spatial-temporal network data forecasting. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 914–921.
  * Tekin et al. (2021) Selim Furkan Tekin, Oguzhan Karaahmetoglu, Fatih Ilhan, Ismail Balaban, and Suleyman Serdar Kozat. 2021.  Spatio-temporal weather forecasting and attention mechanism on convolutional lstms.  _arXiv preprint arXiv:2102.00696_ (2021).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Veličković et al. (2018) Petar Veličković et al. 2018\.  Graph Attention Networks.  _International Conference on Learning Representations_ (2018).
  * Wu et al. (2020) Shiwen Wu, Fei Sun, Wentao Zhang, and Bin Cui. 2020\.  Graph neural networks in recommender systems: a survey.  _arXiv preprint arXiv:2011.02260_ (2020).
  * Wu et al. (2019) Zonghan Wu, Shirui Pan, Guodong Long, Jing Jiang, and Chengqi Zhang. 2019.  Graph WaveNet for Deep Spatial-Temporal Graph Modeling. In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_. 1907–1913.
  * Xu et al. (2018) Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. 2018.  Representation learning on graphs with jumping knowledge networks.  _arXiv preprint arXiv:1806.03536_ (2018).
  * Xu et al. (2020) Mingxing Xu, Wenrui Dai, Chunmiao Liu, Xing Gao, Weiyao Lin, Guo-Jun Qi, and Hongkai Xiong. 2020.  Spatial-temporal transformer networks for traffic flow forecasting.  _arXiv preprint arXiv:2001.02908_ (2020).
  * Yan et al. (2018) Sijie Yan, Yuanjun Xiong, and Dahua Lin. 2018.  Spatial temporal graph convolutional networks for skeleton-based action recognition. In _Thirty-second AAAI conference on artificial intelligence_.
  * Ying et al. (2021) Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. 2021\.  Do Transformers Really Perform Badly for Graph Representation?  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Yu et al. (2018) Bing Yu, Haoteng Yin, and Zhanxing Zhu. 2018.  Spatio-temporal Graph Convolutional Networks: A Deep Learning Framework for Traffic Forecasting. In _Proceedings of the 27th International Joint Conference on Artificial Intelligence (IJCAI)_.
  * Zhao and Akoglu (2019) Lingxiao Zhao and Leman Akoglu. 2019.  Pairnorm: Tackling oversmoothing in gnns.  _arXiv preprint arXiv:1909.12223_ (2019).