# STAN: Spatio-Temporal Attention Network for Next Location Recommendation

Yingtao Luo1, Qiang Liu2,3,*, Zhaocheng Liu4 1University of Washington2Center
for Research on Intelligent Perception and Computing, Institute of Automation,
Chinese Academy of Sciences3School of Artificial Intelligence, University of
Chinese Academy of Sciences4Renmin University of China<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
(2021)

###### Abstract.

The next location recommendation is at the core of various location-based
applications. Current state-of-the-art models have attempted to solve spatial
sparsity with hierarchical gridding and model temporal relation with explicit
time intervals, while some vital questions remain unsolved. Non-adjacent
locations and non-consecutive visits provide non-trivial correlations for
understanding a user’s behavior but were rarely considered. To aggregate all
relevant visits from user trajectory and recall the most plausible candidates
from weighted representations, here we propose a Spatio-Temporal Attention
Network (STAN) for location recommendation. STAN explicitly exploits relative
spatiotemporal information of all the check-ins with self-attention layers
along the trajectory. This improvement allows a point-to-point interaction
between non-adjacent locations and non-consecutive check-ins with explicit
spatio-temporal effect. STAN uses a bi-layer attention architecture that
firstly aggregates spatiotemporal correlation within user trajectory and then
recalls the target with consideration of personalized item frequency (PIF). By
visualization, we show that STAN is in line with the above intuition.
Experimental results unequivocally show that our model outperforms the
existing state-of-the-art methods by 9-17%.

Point-of-Interest; recommendation; attention; spatiotemporal

††journalyear: 2021††copyright: iw3c2w3††conference: Proceedings of the Web
Conference 2021; April 19–23, 2021; Ljubljana, Slovenia††booktitle:
Proceedings of the Web Conference 2021 (WWW ’21), April 19–23, 2021,
Ljubljana, Slovenia††doi: 10.1145/3442381.3449998††isbn:
978-1-4503-8312-7/21/04††ccs: Information systems Location based
services††ccs: Information systems Data mining††ccs: Human-centered computing
Ubiquitous and mobile computing design and evaluation methods

##  1\. Introduction

Next Point-of-Interest (POI) recommendation raises intensive studies in recent
years owing to the growth of location-based services such as Yelp, Foursquare
and Uber. The large volume of historical check-in data gives service providers
invaluable information to understand user preferences on next movements, as
the historical trajectories reveal the user’s behavioral pattern in making
every decision. Meanwhile, such a system can also provide users with the
convenience to decide where to go and how to plan the day, based on previous
visits as well as current status (Feng et al., 2015; Gao et al., 2013; Zhang
et al., 2017; Han et al., 2020; Liu et al., 2014).

Previous approaches have extensively studied various aspects and proposed many
models to make a personalized recommendation. Early models mainly focus on
sequential transitions, such as Markov chains (Rendle, 2010). Later on,
recurrent neural networks (RNNs) with memory mechanism improved recommendation
precision, inspiring following works (Hidasi et al., 2015; Zhu et al., 2017;
Sun et al., 2020; Feng et al., 2018) to propose RNN variants to better extract
the long periodic and short sequential features of user trajectories. Besides
sequential regularities, researchers have exploited temporal and spatial
relation to assist sequential recommendation (Liu et al., 2016a). The recent
state-of-the-art models fed time intervals and/or spatial distances between
two consecutive visits to explicitly represent the effect of the
spatiotemporal gap between each movement. Prior works have also addressed the
sparsity problem of spatiotemporal information by discretely denoting time in
hours and partitioning spatial areas by hierarchical grids (Yang et al., 2020;
Zhao et al., 2020; Lian et al., 2020). Besides, they modified neural
architectures (Zhao et al., 2019; Yao et al., 2017; Tang and Wang, 2018) or
stacked extra modules (Guo et al., 2020; Sun et al., 2020; Chen et al., 2020)
to integrate these additional information.

Figure 1. A trajectory example showing the relation between non-consecutive
visits and non-adjacent locations. The map shows the spatial distribution of
visited locations, which are named by figures from 0 to 6. The timeline shows
the temporal distribution of visited locations from Friday to Monday. Solid
marks represent restaurants. Hollow marks 0, 1, 2 represent home, work place,
and shopping mall, respectively. Restaurants 3, 4, 5 and 6 are functionally
relevant but are temporally non-successive and spatially distanced.

With the continuously upcoming novel models pushing forward our understanding
of mobility prediction, several key problems remain unsolved. 1) First, the
correlations between non-adjacent locations and non-contiguous visits have not
been learned effectively. The mobility of users may depend more on relevant
locations visited a few days ago rather than irrelevant locations visited just
now. Moreover, it is not rare for a user to visit distanced locations that are
functionally relevant/similar. In a special example shown in Figure 1, a user
always dines at a certain restaurant near the workplace on Friday evening, go
to some shopping malls on Saturday morning, and dine at a random restaurant
near a mall on Saturday evening. In this case, the user has de facto made two
non-consecutive visits to non-adjacent restaurants, where the explicit spatial
distances between home and shopping malls and the explicit temporal interval
between meals provide non-trivial information for predicting the exact
location for Saturday dinner. However, most current models focused on spatial
and/or temporal differences between current and future steps while ignoring
spatiotemporal correlation within the trajectory. 2) Second, the previously
practiced hierarchical gridding for spatial discretization is insensitive to
spatial distance. The gridding-based attention network aggregates neighboring
locations but cannot perceive spatial distance. Grids that are close to each
other reflect no difference to those that are not, tossing a lot of spatial
information. 3) Third, previous models extensively overlooked personalized
item frequency (PIF) (Hu et al., 2020; Wang et al., 2019b; Ren et al., 2019).
Repeated visits to the same place reflect the frequency, which emphasizes the
importance of the repeated locations and the possibility of users revisiting.
Previous RNN-based models and self-attention models can hardly reflect PIF due
to the memory mechanism and normalization operation, respectively.

To this end, we proposed STAN, a Spatio-Temporal Self-Attention Network for
the next location recommendation. In
STAN111https://github.com/yingtaoluo/Spatial-Temporal-Attention-Network-for-
POI-Recommendation, we design a self-attention layer for aggregating important
locations within the historical trajectory and another self-attention layer
for recalling the most plausible candidates, both with the consideration of a
point-to-point explicit spatiotemporal effect. Self-attention layers can
assign different weights to each visit within the trajectory, which overcomes
the long-term dependency problem of the commonly used recurrent layers. The
bi-layer system allows effective aggregation that considers PIF. We employ
linear interpolation for the embedding of spatiotemporal transition matrix to
address the sparsity problem, which is sensitive to spatial distance, unlike
GPS gridding. STAN can learn correlations between non-adjacent locations and
non-contiguous visits owing to the spatiotemporal effect of all check-ins fed
into the model.

To summarize, our contributions are listed as follows:

  * •

We propose STAN, a spatiotemporal bi-attention model, to fully consider the
spatiotemporal effect for aggregating relevant locations. To our best
recollection, STAN is the first model in POI recommendation that explicitly
incorporates spatiotemporal correlation to learn the regularities between non-
adjacent locations and non-contiguous visits.

  * •

We replace the GPS gridding with a simple linear interpolation technique for
spatial discretization, which can recover spatial distances and reflect user
spatial preference, instead of merely aggregating neighbors. We integrate this
method into STAN for more accurate representation.

  * •

We specifically propose a bi-attention architecture for PIF. The first layer
aggregates relevant locations within the trajectory for updated
representation, so that the second layer can match the target to all check-
ins, including repetition.

  * •

Experiments on four real-world datasets are conducted to evaluate the
performances of the proposed method. The result shows that the proposed STAN
outperforms the accuracy of state-of-the-art models by more than 10%.

##  2\. Related Works

In this section, we briefly review some works on sequential recommendation and
the next POI recommendation. The next POI recommendation can be viewed as a
special sub-task of sequential recommendation with spatial information.

###  2.1. Sequential Recommendation

The sequential recommendation was mainly modeled by two schools of models:
Markov-based models and deep learning-based models.

Markov-based models predict the probability of the next behavior via a
transition matrix. Due to the sparsity of sequential data, the Markov model
can hardly capture the transition of intermittent visits. Matrix factorization
models (Koren et al., 2009; Rendle, 2010) are proposed to approach this
problem, with further extensions (Cheng et al., 2013; He and McAuley, 2016)
find that explicit spatial and temporal information help a lot with
recommendation performance. In general, Markov-based models mainly focus on
the transition probability between two consecutive visits.

Challenged by the flaws of Markov models, deep learning-based models thrive to
replace them. Among them, models based on RNN (Zhang et al., 2014) are
representative and quickly develop as strong baselines. They have achieved
satisfactory performances on variety of tasks, such as session-based
recommendation (Hidasi et al., 2015; Li et al., 2017), next basket
recommendation (Yu et al., 2016) and next item recommendation (Zhou et al.,
2019; Chen et al., 2018). Meanwhile, time intervals between adjacent behaviors
are incorporated in the RNN-based recommendation models (Liu et al., 2016b;
Zhu et al., 2017), for better preserving the dynamic characteristics of user
history. Besides RNN, other deep learning methods are also considered. For
example, metric embedding algorithms (Feng et al., 2015, 2020), convolutional
neural networks (Tang and Wang, 2018; Yuan et al., 2019; Wang et al., 2019a),
reinforcement learning algorithms (Massimo and Ricci, 2018), and graph network
(Wu et al., 2019; Yu et al., 2020) are proposed one by one for sequential
recommendation. Recently, researchers extensively use self-attention (Vaswani
et al., 2017) for sequential recommendation, where a model named SASRec (Kang
and McAuley, 2018) is proposed. Based on SASRec, time intervals within user
sequence are considered (Li et al., 2020; Ye et al., 2020). Moreover, as
discussed in (Hu et al., 2020), Personalized Item Frequency (PIF) is very
important for sequential recommendations. RNN-based sequential recommenders
have been proven to be unable for effectively capturing PIF. In models based
on self-attention, PIF is also hard to capture due to the normalization in
attention modules. After normalization, the representation of previous
histories is reduced to a single vector of embedding dimension. Matching each
candidate with this representation can hardly reflect PIF information.

###  2.2. Next POI Recommendation

Most existing next POI recommendation models are based on RNN. STRNN (Liu et
al., 2016a) uses temporal and spatial intervals between every two consecutive
visits as explicit information to improve model performance, which has also
been applied in public security evaluation (Wu et al., 2016). SERM (Yao et
al., 2017) jointly learns temporal and semantic contexts that reflect user
preference. DeepMove (Feng et al., 2018) combines an attention layer for
learning long-term periodicity with a recurrent layer for learning short-term
sequential regularity and learned from highly correlated trajectories.
Regarding the use of spatiotemporal information in the next location
recommendation, many previous works only used explicit spatiotemporal
intervals between two successive visits in a recurrent layer. STRNN (Liu et
al., 2016a) directly uses spatiotemporal intervals between successive visits
in a recurrent neural network. Then, Time-LSTM(Zhu et al., 2017) proposes to
add time gates to the LSTM structure to better adapt the spatiotemporal
effect. STGN (Zhao et al., 2019) further enhances the LSTM structure by adding
spatiotemporal gates. ATST-LSTM (Huang et al., 2019) uses an attention
mechanism to assist LSTM in assigning different weights to each check-in,
which starts to use attention but still only considered successive visits.
LSTPM (Sun et al., 2020) proposes a geo-dilated RNN that aggregates locations
visited recently, but only for short-term preference. Inspired by sequential
item recommendation (Kang and McAuley, 2018), GeoSAN (Lian et al., 2020) uses
self-attention model in next location recommendation that allows point-to-
point interaction within the trajectory. However, GeoSAN ignores the explicit
modeling of time intervals and spatial distances, as the gridding method for
spatial discretization used in GeoSAN can not well capture the exact
distances. In other words, all previous methods have not effectively
considered non-trivial correlations between non-adjacent locations and non-
contiguous visits. Moreover, these models also have problems in modeling PIF
information.

##  3\. Preliminaries

In this section, we give problem formulations and term definitions. We denote
the set of user, location and time as
$\textit{U}=\\{u_{1},u_{2},...,u_{\textit{U}}\\}$,
$\textit{{L}}=\\{l_{1},l_{2},...,l_{\textit{L}}\\}$,
$\textit{{T}}=\\{t_{1},t_{2},...,t_{\textit{{T}}}\\}$, respectively.

Historical Trajectory. The trajectory of user $u_{i}$ is temporally ordered
check-ins. Each check-in $r_{k}$ within the trajectory of user $u_{i}$ is a
tuple $(u_{i},l_{k},t_{k})$, in which $l_{k}$ is the location and $t_{k}$ is
the timestamp. Each user may have a variable-length trajectory
$\textit{tra}(u_{i})=\\{r_{1},r_{2},...,r_{{m}_{i}}\\}$. We transform each
trajectory into a fixed-length sequence
$\textit{seq}(u_{i})=\\{r_{1},r_{2},...,r_{n}\\}$, with $n$ as the maximum
length we consider. If $n<m_{i}$, we only consider the most recent $n$ check-
ins. If $n>m_{i}$, we pad zeros to the right until the sequence length is $n$
and mask off the padding items during calculation.

Trajectory Spatio-Temporal Relation Matrix. We model time intervals and
geographical distances as the explicit spatio-temporal relation between two
visited locations. We denote temporal interval between $i$-th and $j$-th
visits as $\Delta_{ij}^{t}=|t_{i}-t_{j}|$, and denote spatial distance between
the GPS location of $i$-th visit and the GPS location of $j$-th visit as
$\Delta_{ij}^{s}=Haversine(GPS_{i},GPS_{j})$. Specifically, the trajectory
spatial relation matrix $\Delta^{s}\in\mathbb{R}^{n\times n}$ and the
trajectory temporal relation matrix $\Delta^{t}\in\mathbb{R}^{n\times n}$ are
separately represented as:

(1)
$\displaystyle\Delta^{t,s}=\begin{bmatrix}\Delta_{11}^{t,s}&\Delta_{12}^{t,s}&\dots&\Delta_{1n}^{t,s}\\\\[5.0pt]
\Delta_{21}^{t,s}&\Delta_{22}^{t,s}&\dots&\Delta_{2n}^{t,s}\\\
\vdots&\vdots&\ddots&\vdots\\\\[2.0pt]
\Delta_{n1}^{t,s}&\Delta_{n2}^{t,s}&\dots&\Delta_{nn}^{t,s}\end{bmatrix}$

Candidate Spatio-Temporal Relation Matrix. Besides the internal explicit
relation, we also consider a next spatiotemporal matrix in the paper. It
calculates the distance between each location candidate $i\in[1,L]$ and each
location of the check-ins $j\in[1,n]$ as
$N_{ij}^{s}=Haversine(GPS_{i},GPS_{j})$, and represents the time intervals
between $t_{m+1}$ and $\\{t_{1},t_{2},...,t_{m}\\}$ that are repeated L times
to expand into 2D as $N_{ij}^{t}=|t_{m+1}-t_{j}|$. The candidate spatial
relation matrix $N^{s}\in\mathbb{R}^{L\times n}$ and the candidate temporal
relation matrix $N^{t}\in\mathbb{R}^{L\times n}$ are separately represented
as:

(2) $\displaystyle
N^{t,s}=\begin{bmatrix}N_{11}^{t,s}&N_{12}^{t,s}&\dots&N_{1n}^{t,s}\\\\[5.0pt]
N_{21}^{t,s}&N_{22}^{t,s}&\dots&N_{2n}^{t,s}\\\
\vdots&\vdots&\ddots&\vdots\\\\[2.0pt]
N_{L1}^{t,s}&N_{L2}^{t,s}&\dots&N_{Ln}^{t,s}\end{bmatrix}$

Mobility Prediction. Given the user trajectory $(r_{1},r_{2},...,r_{m})$, the
location candidates $L=\\{l_{1},l_{2},...,l_{\textit{L}}\\}$, the spatio-
temporal relation matrix $\Delta^{t,s}$, and the next spatio-temporal matrix
$N^{t,s}$, our goal is to find the desired output $l\in r_{m+1}$.

##  4\. The Proposed Framework

Our proposed Spatio-Temporal Attention Network (STAN) consists of: 1) a
multimodal embedding module that learns the dense representations of user,
location, time, and spatiotemporal effect; 2) a self-attention aggregation
layer that aggregates important relevant locations within the user trajectory
to update the representation of each check-in; 3) an attention matching layer
that calculates softmax probability from weighted check-in representations to
compute the probability of each location candidate for next location; 4) a
balanced sampler that use a positive sample and several negative samples to
compute the cross-entropy loss. The neural architecture of the proposed STAN
is shown in Figure 2.

Figure 2. The architecture of the proposed STAN model.

###  4.1. Multimodal Embedding Module

The multi-modal embedding module consists of two parts, namely a trajectory
embedding layer and a spatio-temporal embedding layer.

####  4.1.1. User Trajectory Embedding Layer

A multi-modal embedding layer is used to encode user, location and time into
latent representations. For user, location and time, we denote their embedded
representations as $e^{u}$$\in\mathbb{R}^{\textit{$d$}}$,
$e^{l}$$\in\mathbb{R}^{\textit{$d$}}$, $e^{t}$$\in\mathbb{R}^{\textit{$d$}}$,
respectively. The embedding module is incorporated into the other modules to
transform the scalars into dense vectors to reduce computation and improve
representation. Here, the continuous timestamp is divided by $7\times 24=168$
hours that represents the exact hour in a week, which maps the original time
into 168 dimensions. This temporal discretization can indicate the exact time
in a day or a week, reflecting periodicity. Therefore, the input dimensions of
the embeddings $e^{u}$, $e^{l}$ and $e^{t}$ are U, L, and 168, respectively.
The output of user trajectory embedding layer for each check-in $r$ is the sum
$e^{r}=e^{u}+e^{l}+e^{t}\in\mathbb{R}^{d}$. For the embedding of each user
sequence $\textit{seq}(u_{i})=\\{r_{1},r_{2},...,r_{n}\\}$, we denote as
$E(u_{i})=\\{e^{r_{1}},e^{r_{2}},...,e^{r_{n}}\\}\in\mathbb{R}^{n\times\textit{d}}$.

####  4.1.2. Spatio-Temporal Embedding Layer

A unit embedding layer is used for the dense representation of spatial and
temporal differences with an hour and hundred meters as basic units,
respectively. Recall that if we regard the maximum space or time intervals as
the number of embeddings and discretize all the intervals, it can easily lead
to a sparse relation encoding. This layer multiplies the space and time
intervals each with a unit embedding vector $e_{\Delta s}$ and $e_{\Delta t}$,
respectively. The unit embedding vectors reflect the continuous spatiotemporal
context with the basic unit and avoid sparsity encoding with the dense
dimensions. Especially, we can use this technique that is sensitive to spatial
distance to replace hierarchical gridding method, which only aggregates
adjacent locations and is not capable to represent spatial distance. In
mathematics, the spatiotemporal difference embedding is
$e_{ij}^{\Delta}\in\mathbb{R}^{d}$:

(5) $\displaystyle\left\\{\begin{array}[]{lr}e_{ij}^{\Delta
t}=\Delta_{ij}^{t}\times e_{\Delta t}\\\\[5.0pt] e_{ij}^{\Delta
s}=\Delta_{ij}^{s}\times e_{\Delta s}\end{array}\right.$

Inspired by (Liu et al., 2016a; Liu et al., 2017; Liu et al., 2020), we may
also consider an alternative interpolation embedding layer that sets a upper-
bound unit embedding vector and a lower-bound unit embedding vector and
represents the explicit intervals as a linear interpolation, which is an
approximation to the unit embedding layer. In experiments, the two methods
have similar efficiency. The interpolation embedding is calculated as:

(8) $\displaystyle\left\\{\begin{array}[]{lr}e_{ij}^{\Delta
t}=\dfrac{e_{\Delta t}^{sup}(Upper(\Delta t)-\Delta t)+e_{\Delta
t}^{inf}(\Delta t-Lower(\Delta t))}{Upper(\Delta t)-Lower(\Delta
t)}\\\\[12.0pt] e_{ij}^{\Delta s}=\dfrac{e_{\Delta s}^{sup}(Upper(\Delta
s)-\Delta s)+e_{\Delta s}^{inf}(\Delta s-Lower(\Delta s))}{Upper(\Delta
s)-Lower(\Delta s)}\\\\[6.0pt] \end{array}\right.$

This layer processes two matrices: the trajectory spatio-temporal relation
matrix and the candidate spatio-temporal relation matrix, as described in
preliminaries. Their embeddings are $E(\Delta^{t})\in\mathbb{R}^{n\times
n\times d}$, $E(\Delta^{s})\in\mathbb{R}^{n\times n\times d}$,
$E(N^{t})\in\mathbb{R}^{L\times n\times d}$, and
$E(N^{s})\in\mathbb{R}^{L\times n\times d}$. We can use a weighted sum of the
last dimension and add spatial and temporal embeddings together to create:

(11)
$\displaystyle\left\\{\begin{array}[]{lr}E(\Delta)=Sum(E(\Delta^{t}))+Sum(E(\Delta^{s}))\in\mathbb{R}^{n\times
n}\\\\[6.0pt] E(N)=Sum(E(N^{t}))+Sum(E(N^{s}))\in\mathbb{R}^{L\times
n}\end{array}\right.$

###  4.2. Self-Attention Aggregation Layer

Inspired by self-attention mechanisms, we propose an extensional module to
consider the different spatial distances and time intervals between two visits
in a trajectory. This module aims at aggregating relevant visited locations
and updating the representation of each visit. Self-attention layer can
capture long-term dependency and assign different weights to each visit within
the trajectory. This point-to-point interaction within the trajectory allows
the layer to assign more weights to relevant visits. Moreover, we can easily
incorporate the explicit spatio-temporal intervals into the interaction. Given
the user embedded trajectory matrix $E(u)$ with non-padding length
$m^{\prime}$ and the spatio-temporal relation matrices $E(\Delta)$, this layer
firstly construct a mask matrix $M\in\mathbb{R}^{n\times n}$ with upper left
elements $\mathbb{R}^{m^{\prime}\times m^{\prime}}$ being ones and other
elements being zeros. Then the layer computes a new sequence $S$ after
converting them through distinct parameter matrices
$W_{Q},W_{K},W_{V}\in\mathbb{R}^{\textit{d}\times\textit{d}}$ as

(12) $\displaystyle S(u)=Attention(E(u)W_{Q},E(u)W_{K},E(u)W_{V},E(\Delta),M)$

with

(13) $\displaystyle
Attention(Q,K,V,\Delta,M)=\left(M*softmax(\dfrac{QK^{T}+\Delta}{\sqrt{d}})\right)V$

Here, only the mask and softmax attention are multiplied element by element,
while others use matrix multiplication. It is very important for us to
consider causality that only the first $m^{\prime}$ visits in the trajectory
are fed into the model while predicting the $(m^{\prime}+1)$-st location.
Therefore, during training, we use all the $m^{\prime}\in[1,m]$ to mask the
input sequence and accordingly to the selected label. We can get
$S(u)\in\mathbb{R}^{n\times\textit{d}}$ as the updated representation of the
user trajectory. Another alternative implementation is to feed explicit
spatio-temporal intervals into both $E(u)W_{K}$ and $E(u)W_{V}$, as TiSASRec
(Li et al., 2020) did. However, in experiments, we found out the two methods
have similar performances. Our implementation is in a more concise form using
only matrix multiplication instead of element-wise calculation.

###  4.3. Attention Matching Layer

This module aims at recalling the most plausible candidates from all the L
locations by matching with the updated representation of the user trajectory.
Given the updated trajectory representation
$S(u)\in\mathbb{R}^{n\times\textit{d}}$, the embedded location candidates
$E(l)=\\{e^{l}_{1},e^{l}_{2},...,e^{l}_{L}\\}\in\mathbb{R}^{L\times\textit{d}}$,
and the embedding of the candidate spatio-temporal relation matrix
$E(N)\in\mathbb{R}^{L\times n}$, this layer computes the probability of each
location candidate to be the next location as

(14) $\displaystyle A(u)=Matching(E(l),S(u),E(N))$

with

(15) $\displaystyle
Matching(Q,K,N)=Sum\left(softmax\left(\dfrac{QK^{T}+N}{\sqrt{d}}\right)\right)$

Here, the $Sum$ operation is a weighted sum of the last dimension, converting
the dimension of $A(u)$ to be $\mathbb{R}^{L}$. In Eq.(8), we show that the
updated representations of check-ins all participate in the matching of each
candidate location, unlike other self-attention models that reduce the PIF
information. This is due to the design of a bi-layer system that firstly
aggregates relevant locations and then recalls from representations with
consideration of PIF.

###  4.4. Balanced Sampler

Due to the unbalanced scale of positive and negative samples in $A(u)$,
optimizing the cross-entropy loss is no longer efficient as the loss weights
little on the momentum to push forward the correct prediction. It would be
normal to observe that as the loss goes down, the recall rate also goes down.
Given the user $i$’s sequence $seq(u_{i})$, the matching probability of each
candidate location $a_{j}\in A(u_{i})$ for $j\in[1,L]$, and the label $l_{k}$
with number of order $k$ in the location set $L$, the ordinary cross-entropy
loss is written as:

(16) $\displaystyle-\sum_{i}\sum_{m_{i}}\left(log\sigma(a_{k})+\sum_{j=1,j\neq
k}^{L}log(1-\sigma(a_{j}))\right)$

In this form, for every positive sample $a_{k}$, we need to compute $L-1$
negative samples in the meantime. Other implementations also extensively used
binary cross-entropy loss that computes only one negative sample along with a
positive sample. However, this may also leave many non-label samples unused
throughout the entire training. Here, we can simply set the number of negative
samples used in cross-entropy loss as a hyperparameter s. Here we propose a
balanced sampler for randomly sampling negative samples at each step of
training. Consequently, we update the random seed for the negative sampler
after each training step. The loss is calculated as

(17)
$\displaystyle-\sum_{i}\sum_{m_{i}}\left(log\sigma(a_{k})+\sum_{\begin{subarray}{c}(j_{1},j_{2},...,j_{s})\in[1,L]\\\
(j_{1},j_{2},...,j_{s})\neq k\end{subarray}}log(1-\sigma(a_{j}))\right)$

##  5\. Experiments

In this section, we show our empirical results to make a fair comparison with
other models quantitatively. We show a table of datasets, a table of
recommendation performance under the evaluation of top$k$ recall rates,
figures of model stability, and the visualization of attention weights in STAN
aggregation.

###  5.1. Datasets

We evaluate our proposed STAN model on four real-world datasets: Gowalla
222http://snap.stanford.edu/data/loc-gowalla.html, SIN
333https://www.ntu.edu.sg/home/gaocong/data/poidata.zip, TKY and NYC
444http://www-public.imtbs-tsp.eu/~zhang_da/pub/dataset_tsmc2014.zip. The
numbers of users, locations, and check-ins in each dataset are shown in Table
1. In experiments, we use the original raw datasets that only contain the GPS
of each location and user check-in records, and pre-process them following
each work’s protocol. In regard to the pre-processing technique of datasets,
many previous works used sliced trajectory with a fixed-length window or
maximum time interval. We follow each work’s setup, although this could
prevent the model from learning long-time dependency. For each user that has
$m$ check-ins, we divide a dataset into training, validation, and test
datasets. The number of training set is $m-3$, with the first
$m^{\prime}\in[1,m-3]$ check-ins as input sequence and the $[2,m-2]$-nd
visited location as label; the validation set uses the first $m-2$ check-ins
as input sequence and the $(m-1)$-st visited location as label; the test set
uses the first $m-1$ check-ins as input sequence and the $m$-th visited
location as label. The split of datasets follows the causality that no future
data is used in the prediction of future data.

Table 1. Basic dataset statistics. | Gowalla | TKY | SIN | NYC
---|---|---|---|---
#users | 53008 | 2245 | 2032 | 1064
#locations | 121944 | 7872 | 3662 | 5136
#check-ins | 3302414 | 447571 | 179721 | 147939
Table 2. Recommendation performance comparison with baselines. | Gowalla |  | TKY |  | SIN |  | NYC
---|---|---|---|---|---|---|---
| Recall@5 | Recall@10 |  | Recall@5 | Recall@10 |  | Recall@5 | Recall@10 |  | Recall@5 | Recall@10
STRNN | 0.1664 | 0.2567 |  | 0.1836 | 0.2791 |  | 0.1791 | 0.2016 |  | 0.2365 | 0.2802
DeepMove | 0.1959 | 0.2699 |  | 0.2684 | 0.3509 |  | 0.2389 | 0.3155 |  | 0.3268 | 0.4014
STGN | 0.1528 | 0.2422 |  | 0.1940 | 0.2710 |  | 0.2292 | 0.2727 |  | 0.2439 | 0.3015
ARNN | 0.1810 | 0.2745 |  | 0.1852 | 0.2696 |  | 0.1817 | 0.2538 |  | 0.1970 | 0.3483
LSTPM | 0.2015 | 0.2701 |  | 0.2568 | 0.3310 |  | 0.2579 | 0.3327 |  | 0.2791 | 0.3564
TiSASRec | 0.2411 | 0.3546 |  | 0.3031 | 0.3693 |  | 0.2963 | 0.3753 |  | 0.3664 | 0.5020
GeoSAN | 0.2764 | 0.3645 |  | 0.2957 | 0.3740 |  | 0.3397 | 0.3943 |  | 0.4006 | 0.5267
STAN | 0.3016 | 0.3998 |  | 0.3461 | 0.4264 |  | 0.3751 | 0.4301 |  | 0.4669 | 0.5962
Improvement | 9.12% | 9.68% |  | 17.04% | 14.01% |  | 10.42% | 9.08% |  | 16.55% | 13.20%

###  5.2. Baseline Models

We compare our STAN with the following baselines:

  * •

STRNN (Liu et al., 2016a): an invariant RNN model that incorporates spatio-
temporal features between consecutive visits.

  * •

DeepMove (Feng et al., 2018): a state-of-the-art model with recurrent and
attention layers to capture periodicity.

  * •

STGN (Zhao et al., 2019): a state-of-the-art model that adds time and distance
interval gates to LSTM.

  * •

ARNN (Guo et al., 2020): a state-of-the-art model that uses semantic and
spatial information to construct knowledge graph and improve the performance
of sequential LSTM model.

  * •

LSTPM (Sun et al., 2020): a state-of-the-art model that combines long-term and
short-term sequential models for recommendation.

  * •

TiSASRec (Li et al., 2020): a state-of-the-art model that uses self-attention
layers with explicit time intervals for sequential recommendation, but it uses
no spatial information.

  * •

GeoSAN (Lian et al., 2020): a state-of-the-art model that uses hierarchical
gridding of GPS locations for spatial discretization and uses self-attention
layers for matching, without use of explicit spatio-temporal interval.

###  5.3. Evaluation Matrices

We adopt the top$k$ recall rates, Recall@5 and Recall@10, to evaluate
recommendation performance. Recall@k counts the rate of true positive samples
in all positive samples, which in our case means the rate of the label in the
top$k$ probability samples. For evaluation, we drop the balanced sampler
module and directly recall the target from A, the output of the attention
matching layer. The larger the Recall@k, the better the performance.

###  5.4. Settings

There are two kinds of hyperparameters: (i) common hyperparameters that are
shared by all models; (ii) unique hyperparameters that depend on each model’s
framework. We train the common hyperparameters on a simple recurrent neural
network and then apply them to all models, which helps reduce the training
burden. The embedding dimension $d$ to 50 for TKY, SIN and NYC datasets and 10
for gowalla dataset. We use the Adam optimizer with default betas, the
learning rate of 0.003, the dropout rate of 0.2, the training epoch of 50, and
the maximum length for trajectory sequence of 100. Fixing these common
hyperparameters, we fine-tune the unique hyperparameters for each model. In
our model, the number of negative samples in the balanced sampler is optimal
at 10.

###  5.5. Recommendation Performance

Table 2 shows the recommendation performance of our model and baselines on the
four datasets. All the differences between different methods are statistically
significant ($error<5e^{-5}$). We use a T-test with a p-value of 0.01 to
evaluate the performance improvement provided by STAN. Here, we use the
averaged performance run by 10 times and reject the H0 hypothesis. Therefore,
we know the improvement of STAN is statistically significant.

We can see that our model unequivocally outperforms all compared models with
9%-17% improvement in recall rates. We show in Figures 3 and 4 that the model
is stable under hyperparameter tuning. Among baseline models, self-attention
models such as TiSASRec and GeoSAN clearly have better performances over RNN-
based models. It is not a surprise since previous RNN-based models often use
sliced short trajectories instead of long trajectories, which tossed long-term
periodicity and can hardly capture the exact influence of each visits towards
the next movement. It should be noted that we do not use any semantic
information to construct knowledge graph to perform meta-path in ARNN, as
semantic analysis was not performed by other baselines in the comparison.

Among RNN-based models, LSTPM and DeepMove have relatively better
performances, due to their consideration of periodicity. Among self-attention
models, TiSASRec used temporal intervals and GeoSAN considered geographical
partitions. Only STAN fully considers the spatio-temporal intervals within the
sequences for modeling non-consecutive visits and non-adjacent locations, and
modifies attention architecture to adapt PIF information instead of inheriting
the transformer (Vaswani et al., 2017) structure directly. In addition,
because STRNN and TiSASRec both use temporal intervals, we can compare their
performances to evaluate the improvement provided by self-attention modules
versus recurrent layers.

We can also refer to Table 3, where the $-ALL$ model represents a variant STAN
model without spatio-temporal intervals and the balanced sampler. $-ALL$ model
is different from ordinary self-attention models only on the bi-layer system,
which considers PIF information. $-ALL$ has a slightly worse performance than
GeoSAN on the recall rates of the four datasets, but is slightly better than
TiSASRec and much better than RNN-based models. This tells us that the bi-
layer system which considers PIF is approximately as important as time
intervals incorporated into the attention systems.

Table 3. Ablation Analysis, in which we compare different modules in STAN. | Gowalla |  | TKY |  | SIN |  | NYC
---|---|---|---|---|---|---|---
| Recall@5 | Recall@10 |  | Recall@5 | Recall@10 |  | Recall@5 | Recall@10 |  | Recall@5 | Recall@10
STAN | 0.3016 | 0.3998 |  | 0.3461 | 0.4264 |  | 0.3751 | 0.4301 |  | 0.4669 | 0.5962
-TIM-BS | 0.2835 | 0.3718 |  | 0.3006 | 0.3819 |  | 0.3416 | 0.3873 |  | 0.4126 | 0.5245
-EWTI-BS | 0.2794 | 0.3717 |  | 0.3052 | 0.3781 |  | 0.3404 | 0.3890 |  | 0.4083 | 0.5272
-TIM | 0.2946 | 0.3925 |  | 0.3315 | 0.4099 |  | 0.3643 | 0.4176 |  | 0.4495 | 0.5814
-SIM-BS | 0.2823 | 0.3729 |  | 0.3123 | 0.3865 |  | 0.3337 | 0.3901 |  | 0.4126 | 0.5299
-EWSI-BS | 0.2812 | 0.3724 |  | 0.3132 | 0.3794 |  | 0.3313 | 0.3916 |  | 0.4124 | 0.5277
-SIM | 0.2977 | 0.3908 |  | 0.3405 | 0.4141 |  | 0.3636 | 0.4165 |  | 0.4502 | 0.5860
-ALL | 0.2645 | 0.3531 |  | 0.2867 | 0.3660 |  | 0.3239 | 0.3776 |  | 0.3896 | 0.5094
Figure 3. Impact of embedding dimension.

###  5.6. Ablation Study

To analyze different modules in our model, we conduct an ablation study in
this section. We denote the based model as STAN, with spatio-temporal
intervals and a balanced sampler. We drop different components to form
variants. The components are listed as:

  * •

SIM (Spatial Intervals in Matrix): This denotes the explicit spatial intervals
we use within the trajectory as a matrix.

  * •

EWSI (Element-Wise Spatial Intervals): This denotes the element-wise spatial
intervals following the structure of TiSASRec (Li et al., 2020).

  * •

TIM (Temporal Intervals in Matrix): This denotes the explicit temporal
intervals we use within the trajectory as a matrix.

  * •

EWTI (Element-Wise Temporal Intervals): This denotes the element-wise temporal
intervals following the structure of TiSASRec (Li et al., 2020).

  * •

BS (Balanced Sampler): Balanced sampler for calculating loss.

Table 3 shows the results of the ablation study. We find that a balanced
sampler is crucial for improving the recommendation performance, which
provides a nearly 5-12% increase in recall rates. Spatial and temporal
intervals can explicitly express the correlation between non-consecutive
visits and non-adjacent locations. Adding spatial distances and temporal
intervals all provide nearly 4-8% increase in recall rates. We also find that
our method to introduce spatio-temporal correlations is equivalent to the
method used in TiSASRec (Li et al., 2020), while our method is easier to
implement and can be computationally convenient due to its matrix form. The
worst condition is that none of the spatio-temporal intervals nor balanced
sampler is used, in which the Recall@5 and Recall@10 decrease drastically.
Even so, this $-ALL$ ablated model still outperforms previously reported RNN-
based models such as DeepMove, STRNN, and STGN. $-ALL$ model with the bi-layer
system can consider PIF information. This explains why $-ALL$ still has a
better performance over TiSASRec and RNN-based models. This tells us that the
bi-layer system which considers PIF is as important as time intervals
incorporated into self-attention systems.

Figure 4. Impact of number of negative samples.

###  5.7. Stability Study

####  5.7.1. Embedding dimension

We vary the dimension of embedding $d$ in the multimodal embedding module from
10 to 60 with step 10. Figure 3 shows that $d=50$ is the best dimension for
trajectory and spatio-temporal embedding. In general, the recommendation
performance of our model is insensitive to the hyperparameter $d$, with less
than 6% change rate for the Gowalla dataset and less than 2% change rate for
other datasets. As long as $d$ is large than 30, the change in recommendation
performance will be less than 0.5%, which can be ignored.

####  5.7.2. Number of negative samples

We experiment a series of number of negative samples $s=[1,10,20,30,40,50]$ in
the balanced sampler. Figure 4 shows that the number of negative samples less
than 20 can all produce stable recommendations for all datasets. STAN is
specifically insensitive to the number of negative samples for the Gowalla
dataset, which has as many as 121944 locations. This indicates that the larger
the dataset, the larger the optimal number of negative samples. As the number
of negative samples increases, the balanced loss will tend to the ordinary
cross-entropy loss. In Table 3, we found that the balanced sampler is crucial
for improving recommendation performance. If the number of negative samples is
above the threshold, the recall rate will drop drastically.

###  5.8. Interpretability Study

To understand the mechanism of STAN, the aggregation of non-consecutive visits
and non-adjacent locations performed by the self-attention aggregation layer
is at the core. We visualize the correlation matrix $Cor$ of the attention
weights in Figure 5. Each element $Cor_{i,j}$ of the matrix represents the
weighted influence of $j$-th visited location on $i$-th visited location. The
correlation matrix is calculated as the softmax of the multiplication of query
and key in the self-attention aggregation layer. The value of each element in
this correlation matrix is either tending to 1 or 0, as a result of softmax
operation. Using the correlation matrix to times the original check-in
embeddings, we can update the representations of the trajectory. Figure 5 is
based on a slice of real user trajectory example that is discussed in
Introduction Section and Figure 1.

Here, different locations are classified and named by numbers from 0 to 6. By
query of the exact GPS, we find that locations 0, 1, 2 are home, workplace,
and shopping mall, respectively. Locations 3, 4, 5 and 6 are restaurants.
Figure 5(a) shows the spatial correlation of visited locations that is
attained by Figure 5(b), where locations with the yellow-colored marks and
locations within the range of the same dark circles are aggregated together.
This shows that not only adjacent locations but also non-adjacent locations
are correlated. Locations 3, 4, 5 and 6 are all restaurants and are often
visited at the exact time for meals. We can tell from the correlation matrix
that they are relevant, despite that they are spatially distanced. The
temporal order of this trajectory example is shown in the timeline of Figure
1. This is a sliced sparse trajectory as we edit off the irrelevant visits to
focus on the correlation of restaurants. The time and order of these
restaurants being visited are not consecutive but are still aggregated
together. This gives evidence that visited temporally non-consecutive
locations may be correlated. Both shreds of evidence in space and time
demonstrate our motivation.

##  6\. Conclusion

In this work, we propose a spatio-temporal attention network, abbreviated as
STAN. We use a real trajectory example to illustrate the functional relevance
between non-adjacent locations and non-consecutive visits, and propose to
learn the explicit spatio-temporal correlations within the trajectory using a
bi-attention system. This architecture firstly aggregates spatio-temporal
intervals within the trajectory and then recalls the target. Because all the
representations of the trajectory are weighted, the recall of the target fully
considers the effect of personalized item frequency (PIF). We propose a
balanced sampler for matching calculating cross-entropy loss, which
outperforms the commonly practiced binary and/or ordinary cross-entropy loss.
We perform comprehensive ablation study, stability study, and interpretability
study in the experimental section. We prove an improvement of recall rates by
the proposed components and very robust stability against hyperparameters’
variation. We also propose to replace the hierarchical gridding method for
spatial discretization with a simple linear interpolation technique, which can
reflect the continuous spatial distance while providing dense representation.
Experimental comparison with baseline models unequivocally demonstrates the
superiority of our model, as STAN improves recall rates to new records that
surpass the state-of-the-art models by 9-17%.

Figure 5. The mechanism of STAN. (a) An example map showing the aggregation of
visited locations. The locations with the same colored marks and locations
within the range of the same dark circles are aggregated. This gives solid
evidence that non-adjacent locations may be correlated and aggregated in our
model. (b) The correlation matrix. Here, we take the softmax of the
multiplication of query and key in the self-attention aggregation layer as a
correlation matrix, which is used to update the representation of check-ins.

###### Acknowledgements.

This work is supported by National Key Research and Development Program
(2018YFB1402605, 2018YFB1402600), National Natural Science Foundation of China
(U19B2038, 61772528), Beijing National Natural Science Foundation (4182066).

## References

  * (1)
  * Chen et al. (2018) Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2018.  Sequential recommendation with user memory networks. In _Proceedings of the eleventh ACM international conference on web search and data mining_. 108–116.
  * Chen et al. (2020) Yile Chen, Cheng Long, Gao Cong, and Chenliang Li. 2020\.  Context-aware deep model for joint mobility and time prediction. In _Proceedings of the 13th International Conference on Web Search and Data Mining_. 106–114.
  * Cheng et al. (2013) Chen Cheng, Haiqin Yang, Michael R. Lyu, and Irwin King. 2013\.  Where You like to Go next: Successive Point-of-Interest Recommendation. In _Proceedings of the Twenty-Third International Joint Conference on Artificial Intelligence_ (Beijing, China) _(IJCAI ’13)_. AAAI Press, 2605¨C2611.
  * Feng et al. (2018) Jie Feng, Yong Li, Chao Zhang, Funing Sun, Fanchao Meng, Ang Guo, and Depeng Jin. 2018.  Deepmove: Predicting human mobility with attentional recurrent networks. In _Proceedings of the 2018 world wide web conference_. 1459–1468.
  * Feng et al. (2015) Shanshan Feng, Xutao Li, Yifeng Zeng, Gao Cong, and Yeow Meng Chee. 2015.  Personalized ranking metric embedding for next new poi recommendation. In _IJCAI’15 Proceedings of the 24th International Conference on Artificial Intelligence_. ACM, 2069–2075.
  * Feng et al. (2020) Shanshan Feng, Lucas Vinh Tran, Gao Cong, Lisi Chen, Jing Li, and Fan Li. 2020\.  HME: A Hyperbolic Metric Embedding Approach for Next-POI Recommendation. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1429–1438.
  * Gao et al. (2013) Huiji Gao, Jiliang Tang, Xia Hu, and Huan Liu. 2013\.  Exploring Temporal Effects for Location Recommendation on Location-Based Social Networks. In _Proceedings of the 7th ACM Conference on Recommender Systems_ (Hong Kong, China). Association for Computing Machinery, New York, NY, USA, 93¨C100.
  * Guo et al. (2020) Qing Guo, Zhu Sun, Jie Zhang, and Yin-Leng Theng. 2020\.  An Attentional Recurrent Neural Network for Personalized Next Location Recommendation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 83–90.
  * Han et al. (2020) Peng Han, Zhongxiao Li, Yong Liu, Peilin Zhao, Jing Li, Hao Wang, and Shuo Shang. 2020.  Contextualized Point-of-Interest Recommendation. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , Christian Bessiere (Ed.). International Joint Conferences on Artificial Intelligence Organization, 2484–2490.  https://doi.org/10.24963/ijcai.2020/344
  * He and McAuley (2016) Ruining He and Julian McAuley. 2016.  Fusing similarity models with markov chains for sparse sequential recommendation. In _2016 IEEE 16th International Conference on Data Mining (ICDM)_. IEEE, 191–200.
  * Hidasi et al. (2015) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2015.  Session-based recommendations with recurrent neural networks.  _arXiv preprint arXiv:1511.06939_ (2015).
  * Hu et al. (2020) Haoji Hu, Xiangnan He, Jinyang Gao, and Zhi-Li Zhang. 2020\.  Modeling Personalized Item Frequency Information for Next-basket Recommendation.  _arXiv preprint arXiv:2006.00556_ (2020).
  * Huang et al. (2019) Liwei Huang, Yutao Ma, Shibo Wang, and Yanbo Liu. 2019\.  An attention-based spatiotemporal lstm network for next poi recommendation.  _IEEE Transactions on Services Computing_ (2019).
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _2018 IEEE International Conference on Data Mining (ICDM)_. IEEE, 197–206.
  * Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009\.  Matrix factorization techniques for recommender systems.  _Computer_ 42, 8 (2009), 30–37.
  * Li et al. (2017) Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017\.  Neural attentive session-based recommendation. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. 1419–1428.
  * Li et al. (2020) Jiacheng Li, Yujie Wang, and Julian McAuley. 2020.  Time Interval Aware Self-Attention for Sequential Recommendation. In _Proceedings of the 13th International Conference on Web Search and Data Mining_. 322–330.
  * Lian et al. (2020) Defu Lian, Yongji Wu, Yong Ge, Xing Xie, and Enhong Chen. 2020.  Geography-Aware Sequential Location Recommendation. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2009–2019.
  * Liu et al. (2020) Qiang Liu, Zhaocheng Liu, and Haoli Zhang. 2020.  An empirical study on feature discretization.  _arXiv preprint arXiv:2004.12602_ (2020).
  * Liu et al. (2016b) Qiang Liu, Shu Wu, Diyi Wang, Zhaokang Li, and Liang Wang. 2016b.  Context-aware sequential recommendation. In _2016 IEEE 16th International Conference on Data Mining (ICDM)_. IEEE, 1053–1058.
  * Liu et al. (2017) Qiang Liu, Shu Wu, and Liang Wang. 2017.  Multi-behavioral sequential prediction with recurrent log-bilinear model.  _IEEE Transactions on Knowledge and Data Engineering_ 29, 6 (2017), 1254–1267.
  * Liu et al. (2016a) Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2016a.  Predicting the next Location: A Recurrent Model with Spatial and Temporal Contexts. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_ (Phoenix, Arizona) _(AAAI’16)_. AAAI Press, 194¨C200.
  * Liu et al. (2014) Yong Liu, Wei Wei, Aixin Sun, and Chunyan Miao. 2014\.  Exploiting geographical neighborhood characteristics for location recommendation. In _Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management_. 739–748.
  * Massimo and Ricci (2018) David Massimo and Francesco Ricci. 2018.  Harnessing a generalised user behaviour model for next-POI recommendation. In _Proceedings of the 12th ACM Conference on Recommender Systems_. 402–406.
  * Ren et al. (2019) Pengjie Ren, Zhumin Chen, Jing Li, Zhaochun Ren, Jun Ma, and Maarten de Rijke. 2019\.  RepeatNet: A repeat aware neural recommendation machine for session-based recommendation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 4806–4813.
  * Rendle (2010) Steffen Rendle. 2010\.  Factorization machines. In _2010 IEEE International Conference on Data Mining_. IEEE, 995–1000.
  * Sun et al. (2020) Ke Sun, Tieyun Qian, Tong Chen, Yile Liang, Quoc Viet Hung Nguyen, and Hongzhi Yin. 2020.  Where to Go Next: Modeling Long-and Short-Term User Preferences for Point-of-Interest Recommendation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 214–221.
  * Tang and Wang (2018) Jiaxi Tang and Ke Wang. 2018.  Personalized top-n sequential recommendation via convolutional sequence embedding. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_. 565–573.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Advances in neural information processing systems_. 5998–6008.
  * Wang et al. (2019b) Chenyang Wang, Min Zhang, Weizhi Ma, Yiqun Liu, and Shaoping Ma. 2019b.  Modeling item-specific temporal dynamics of repeat consumption for recommender systems. In _The World Wide Web Conference_. 1977–1987.
  * Wang et al. (2019a) Jingyi Wang, Qiang Liu, Zhaocheng Liu, and Shu Wu. 2019a.  Towards Accurate and Interpretable Sequential Prediction: A CNN & Attention-Based Feature Extractor. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. 1703–1712.
  * Wu et al. (2016) Shu Wu, Qiang Liu, Ping Bai, Liang Wang, and Tieniu Tan. 2016.  SAPE: A system for situation-aware public security evaluation. In _AAAI_.
  * Wu et al. (2019) Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019\.  Session-based recommendation with graph neural networks. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Yang et al. (2020) Dingqi Yang, Benjamin Fankhauser, Paolo Rosso, and Philippe Cudre-Mauroux. 2020.  Location Prediction over Sparse User Mobility Traces Using RNNs: Flashback in Hidden States!. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_. 2184–2190.
  * Yao et al. (2017) Di Yao, Chao Zhang, Jianhui Huang, and Jingping Bi. 2017\.  Serm: A recurrent model for next location prediction in semantic trajectories. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. 2411–2414.
  * Ye et al. (2020) Wenwen Ye, Shuaiqiang Wang, Xu Chen, Xuepeng Wang, Zheng Qin, and Dawei Yin. 2020\.  Time Matters: Sequential Recommendation with Complex Temporal Information. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1459–1468.
  * Yu et al. (2016) Feng Yu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2016.  A dynamic recurrent model for next basket recommendation. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_. 729–732.
  * Yu et al. (2020) Feng Yu, Yanqiao Zhu, Qiang Liu, Shu Wu, Liang Wang, and Tieniu Tan. 2020.  TAGNN: Target attentive graph neural networks for session-based recommendation. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1921–1924.
  * Yuan et al. (2019) Fajie Yuan, Alexandros Karatzoglou, Ioannis Arapakis, Joemon M Jose, and Xiangnan He. 2019.  A simple convolutional generative network for next item recommendation. In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_. 582–590.
  * Zhang et al. (2014) Yuyu Zhang, Hanjun Dai, Chang Xu, Jun Feng, Taifeng Wang, Jiang Bian, Bin Wang, and Tie-Yan Liu. 2014\.  Sequential click prediction for sponsored search with recurrent neural networks.  _arXiv preprint arXiv:1404.5772_ (2014).
  * Zhang et al. (2017) Zhiqian Zhang, Chenliang Li, Zhiyong Wu, Aixin Sun, Dengpan Ye, and Xiangyang Luo. 2017\.  NEXT: A Neural Network Framework for Next POI Recommendation.  _CoRR_ abs/1704.04576 (2017).  arXiv:1704.04576 http://arxiv.org/abs/1704.04576
  * Zhao et al. (2020) Kangzhi Zhao, Yong Zhang, Hongzhi Yin, Jin Wang, Kai Zheng, Xiaofang Zhou, and Chunxiao Xing. 2020.  Discovering Subsequence Patterns for Next POI Recommendation.  (2020).
  * Zhao et al. (2019) Pengpeng Zhao, Haifeng Zhu, Yanchi Liu, Jiajie Xu, Zhixu Li, Fuzhen Zhuang, Victor S Sheng, and Xiaofang Zhou. 2019\.  Where to go next: A spatio-temporal gated network for next poi recommendation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 5877–5884.
  * Zhou et al. (2019) Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019.  Deep interest evolution network for click-through rate prediction. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 5941–5948.
  * Zhu et al. (2017) Yu Zhu, Hao Li, Yikang Liao, Beidou Wang, Ziyu Guan, Haifeng Liu, and Deng Cai. 2017.  What to Do Next: Modeling User Behaviors by Time-LSTM.. In _IJCAI_ , Vol. 17. 3602–3608.