[1]

# EasyTPP: Towards Open Benchmarking the Temporal Point Processes

Siqiao Xue1 Xiaoming Shi1 Zhixuan Chu1 Yan Wang1 Fan Zhou1
Hongyan Hao1 Caigao Jiang1 Chen Pan1 Yi Xu1 James Y. Zhang1
Qingsong Wen2 Jun Zhou1 Hongyuan Mei3
1Ant Group 2Alibaba DAMO Academy 3TTIC
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Continuous-time event sequences play a vital role in real-world domains such
as healthcare, finance, online shopping, social networks, and so on. To model
such data, temporal point processes (TPPs) have emerged as the most advanced
generative models, making a significant impact in both academic and
application communities. Despite the emergence of many powerful models in
recent years, there is still no comprehensive benchmark to evaluate them. This
lack of standardization impedes researchers and practitioners from comparing
methods and reproducing results, potentially slowing down progress in this
field. In this paper, we present EasyTPP, which aims to establish a central
benchmark for evaluating TPPs. Compared to previous work that also contributed
datasets, our EasyTPP has three unique contributions to the community: (i) a
comprehensive implementation of eight highly cited neural TPPs with the
integration of commonly used evaluation metrics and datasets; (ii) a
standardized benchmarking pipeline for a transparent and thorough comparison
of different methods on different datasets; (iii) a universal framework
supporting multiple ML libraries (e.g., PyTorch and TensorFlow) as well as
custom implementations. Our benchmark is open-sourced: all the data and
implementation can be found at this Github
repository111https://github.com/ant-research/EasyTemporalPointProcess.. We
will actively maintain this benchmark and welcome contributions from other
researchers and practitioners. Our benchmark will help promote reproducible
research in this field, thus accelerating research progress as well as making
more significant real-world impacts.

##  1 Introduction

Continuous-time event sequences are ubiquitous in various real-world domains,
such as neural spike trains in neuroscience (Williams et al., 2020), orders in
financial transactions (Jin et al., 2020), and user page viewing behavior in
the e-commerce platform (Hernandez et al., 2017). To model these event
sequences, temporal point processes (TPPs) are commonly used, which specify
the probability of each event type’s instantaneous occurrence, also known as
the _intensity function_ , conditioned on the past event history. Classical
TPPs, such as Poisson processes (Daley & Vere-Jones, 2007) and Hawkes
processes (Hawkes, 1971), have a well-established mathematical foundation and
have been widely used to model traffic (Cramér, 1969), finance (Hasbrouck,
1991) and seismology (Ogata, 1988) for several decades. However, the strong
parametric assumptions in these models constrain their ability to capture the
complexity of real-world phenomena. To overcome the limitations of classical
TPPs, Du et al. (2016) introduced the concept of neural TPPs, which can
leverage the power of neural networks to learn complex dependencies and
improve computational efficiency (Shchur et al., 2021). Since then, numerous
advancements have been made in this field, as evidenced by the rapidly growing
literature on neural TPPs since 2016. Recent reviews have documented the
extensive methodological developments in TPPs, which have expanded their
applicability to various real-world scenarios. As shown in Figure 1 and
section F.1, the number of research papers on TPPs has been steadily
increasing, indicating the growing interest and potential impact of this
research area. These advancements have enabled more accurate and flexible
modeling of event sequences in diverse fields.

As the field of TPPs continues to advance with the introduction of new
architectures and applications, there is an increasing need for a standardized
benchmarking platform that facilitates transparent and meaningful comparisons
between methods. Such a platform would enable researchers to easily evaluate
their newly proposed methods against the already available methods and ensure
that practitioners select the best methodology for their specific problem. A
standardized benchmarking platform is indispensable in providing researchers
and practitioners with a common ground to compare different methods and assess
their performance. It would enable rigorous evaluation of various models and
methods and ensure that the results are reproducible and reliable.
Additionally, such a platform would help in identifying the strengths and
weaknesses of each method, leading to the development of more effective and
efficient models. Overall, a standardized benchmark is an essential
prerequisite for the continued progress of TPPs. It would provide a foundation
for the development of more sophisticated models that can be applied to a wide
range of real-world applications.

Figure 1: ArXiv submissions over time on TPPs

In this work, we introduce EasyTPP, an open-source Python library with the
following distinct merits: (i) comprehensiveness: EasyTPP provides $8$ most-
cited and competitive baselines for researchers to benchmark new models across
a variety of evaluation metrics and datasets; (ii) standardization: a
standardized benchmarking pipeline has been established to enable transparent
comparisons of the various methods; (iii) generality and extensibility:
EasyTPP is versatile, supporting both TensorFlow (Abadi et al., 2016) and
PyTorch (Paszke et al., 2019), two of the most widely used optimization
frameworks, as well as a generic abstraction layer that enables custom
implementations.

##  2 Related work

#### Temporal Point Processes.

Over recent years, a large variety of TPP models have been proposed, aiming to
model event sequences with varying sorts of properties. Many of them are built
on recurrent neural networks (Du et al., 2016; Mei & Eisner, 2017; Xiao et
al., 2017; Omi et al., 2019; Shchur et al., 2020; Mei et al., 2020). Models of
this kind enjoy continuous state spaces and flexible transition functions,
thus achieving superior performance on many real-world datasets, compared to
the classical Hawkes process (Hawkes, 1971). To properly capture the long-
range and complex dependency in the sequence, the attention and transformer
techniques (Vaswani et al., 2017) have been adapted to TPPs (Zuo et al., 2020;
Zhang et al., 2020; Yang et al., 2022a; Wen et al., 2023) and makes further
improvements on predictive performance. Apart from standard learning tasks,
other applications of neural TPPs include long-horizon event prediction
(Deshpande et al., 2021; Yang et al., 2022b; Xue et al., 2022), spatio-
temporal event prediction (Chen et al., 2021; Yang et al., 2018), event
evolution on dynamic graphs (Xue et al., 2021; Trivedi et al., 2019; Shi et
al., 2023) and interactive event modeling (Upadhyay et al., 2018; Qu et al.,
2023). Despite significant progress made in academia, the existing studies
usually perform model evaluations and comparisons in an ad-hoc manner, e.g.,
by introducing non-generative structures, using different experimental
settings or implementing under different ML frameworks such as TensorFlow and
PyTorch. Such conventions not only increase the difficulty in reproducing
these methods but also may lead to inconsistent experimental results among
them, which largely limits the credibility and practical value of research
results in this field.

#### Open Benchmarking on TPPs.

The significant attention attracted by TPPs in recent years naturally leads to
a high demand for an open benchmark to fairly compare against baseline models.
Open benchmarking is valuable not only to promote research progress but also
for practitioners to easily assess the models and apply them to industrial
datasets. While many efforts have been made in the domains of recommender
systems (Zhu et al., 2021), computer vision (Deng et al., 2009), and natural
language processing (Wang et al., 2019), benchmarking in the field of TPPs is
an under-explored topic. Tick (Bacry et al., 2017) and pyhawkes
222https://github.com/slinderman/pyhawkes. are two well-known libraries that
focus on statistical learning for classical TPPs, which are not suitable for
the state-of-the-art neural models. Poppy
333https://github.com/HongtengXu/PoPPy. is a PyTorch-based toolbox for neural
TPPs, but it has not been actively maintained since three years ago and has
not implemented any recent state-of-the-art methods. To the best of our
knowledge, EasyTPP is the first package that provides open benchmarking for
the popular neural TPPs.

##  3 Preliminaries

#### Definition.

Suppose we are given a fixed time interval $[0,T]$ over which an event
sequence is observed. Suppose there are $I$ events in the sequence at times
$0<t_{1}<\ldots<t_{I}\leq T$. We denote the sequence as
$x_{[0,T]}=(t_{1},k_{1}),\ldots,(t_{I},k_{I})$ where each
$k_{i}\in\\{1,\ldots,K\\}$ is a discrete event type. Note that representations
in terms of time $t_{i}$ and the corresponding inter-event time
$\tau_{i}=t_{i}-t_{i-1}$ are isomorphic, we use them interchangeably. TPPs are
probabilistic models for such event sequences. If we use $p_{k}(t\mid
x_{[0,t)})$ to denote the probability that an event of type $k$ occurs over
the infinitesimal interval $[t,t+dt)$, then the probability that nothing
occurs will be $1-\sum_{k=1}^{K}p_{k}(t\mid x_{[0,t)})$. Formally, the
distribution of a TPP can be characterized by the intensity $\lambda_{k}(t\mid
x_{[0,t)})\geq 0$ for each event type $k$ at each time $t>0$ such that
$p_{k}(t\mid x_{[0,t)})=\lambda_{k}(t\mid x_{[0,t)})dt$.

#### Neural TPPs.

A neural TPP model autoregressively generates events one after another via
neural networks. For the $i$-th event $(t_{i},k_{i})$, it computes the
embedding of the event $\bm{e}_{i}\in\mathbb{R}^{D}$ via an embedding layer
and the hidden state $\bm{h}_{i}$ gets updated conditioned on $\bm{e}_{i}$ and
the previous state $\bm{h}_{i-1}$. Then one can draw the next event
conditioned on the hidden state $\bm{h}_{i}$:

$\displaystyle
t_{i+1},k_{i+1}\sim\mathbb{P}_{\theta}(t_{i+1},k_{i+1}|\bm{h}_{i}),\quad\bm{h}_{i}=f_{update}(\bm{h}_{i-1},\bm{e}_{i}),$
(1)

where $f_{update}$ denotes a recurrent encoder, which could be either RNN (Du
et al., 2016; Mei & Eisner, 2017) or more expressive attention-based recursion
layer (Zhang et al., 2020; Zuo et al., 2020; Yang et al., 2022a). A new line
of research models the evolution of the states completely in continuous time:

$\displaystyle\bm{h}_{i-}=f_{evo}(\bm{h}_{i-1},t_{i-1},t_{i})\quad\text{between
event times}$ (2)
$\displaystyle\bm{h}_{i}=f_{update}(\bm{h}_{i-},\bm{e}_{i})\quad\text{at event
time $t_{i}$}$ (3)

The state evolution in equation 2 is generally governed by an ordinary
differential equation (ODE) (Rubanova et al., 2019). For a broad and fair
comparison, in EasyTPP, we implement not only recurrent TPPs but also an ODE-
based continuous-time state model.

#### Learning TPPs.

Negative log-likelihood (NLL) is the default training objective for both
classical and neural TPPs. The NLL of a TPP given the entire event sequence
$x_{[0,T]}$ is

$\displaystyle\sum_{i=1}^{I}\log\lambda_{k_{i}}(t_{i}\mid
x_{[0,t_{i})})-\int_{t=0}^{T}\sum_{k=1}^{K}\lambda_{k}(t\mid x_{[0,t)})dt$ (4)

Derivations of this formula can be found in previous works (Hawkes, 1971; Mei
& Eisner, 2017).

##  4 Benchmarking Process

Figure 2 presents the open benchmarking pipeline for neural TPPs, which is
implemented in EasyTPP. In summary, the pipeline consists of the following key
components.

#### Data Preprocess.

Following common practices, we split the set of sequences into the train,
validation, and test set with a fixed ratio. To feed the sequences of varying
lengths into the model, in EasyTPP, we pad all sequences to the same length,
then use the "sequence$\\_$mask" tensor to identify which event tokens are
padding. As we implemented several variants of attention-based TPPs, we also
generated the "attention$\\_$mask" to mask all the future positions at each
event to avoid “peeking into the future”.

#### Training.

We can estimate the model parameters by locally maximizing the NLL in equation
4 with any stochastic gradient method. Note that computing the NLL can be
challenging due to the presence of the integral in the second term in equation
4. In EasyTPP, by default, we approximate the integral by Monte-Carlo
estimation to compute the overall NLL (see section C.1). Nonetheless, EasyTPP
also incorporates some neural TPPs (e.g., the intensity-free model (Shchur et
al., 2020)), which allow us to compute the NLL analytically, which is more
computationally efficient.

#### Sampling.

Given the learned parameters, we apply the minimum Bayes risk (MBR) principle
to predict the time and type with the lowest expected loss. A recipe can be
found in section C.2. Note that other methods exist for predicting a TPP, such
as adding an MLP layer to directly output the time and type prediction (Zuo et
al., 2020; Zhang et al., 2020). However, as we aim to build a generative model
of event sequences, we believe the principal way to make predictions based on
continuous-time generative model is thinning algorithm (Ogata, 1988). In
EasyTPP, a batch-wise thinning algorithm is consistently used when evaluating
the predictive performance of TPPs.

#### Hyperparameter Tuning.

Most studies specified the detailed hyper-parameters of their models in the
papers. However, with the modified code fitted in the EasyTPP framework or the
new splits of datasets, it may be inappropriate to use the same hyper-
parameters. Besides the classical grid search method, we also integrate
_Optuna_ (Akiba et al., 2019) in our framework to automatically search optimal
hyperparameters and prune unpromising trials for faster results.

We hope that the definition of our open benchmarking pipeline could provide
guidance for fair comparisons and reproducible works in TPPs.

Figure 2: An open benchmarking pipeline using EasyTPP.

##  5 EasyTPP’s Software Interface

High Level Software Architecture. The purpose of building EasyTPP is to
provide a simple and standardized framework to allow users to apply different
state-of-the-art (SOTA) TPPs to arbitrary data sets. For researchers, EasyTPP
provides an implementation interface to integrate new recourse methods in an
easy-to-use way, which allows them to compare their method to already existing
methods. For industrial practitioners, the availability of benchmarking code
helps them easily assess the applicability of TPP models for their own
problems.

High-level visualization of the EasyTPP’s software architecture is depicted in
Figure 9. _Data Preprocess_ component provides a common way to access the
event data across the software and maintains information about the features.
For the _Model_ component, the library provides the possibility to use
existing methods or extend the users’ custom methods and implementations. A
_wrapper_ encapsulates the black-box models along with the trainer and
sampler. The primary purpose of the wrapper is to provide a common interface
to easily fit in the training and evaluation pipeline, independently of their
framework (e.g., PyTorch, TensorFlow). The running of the pipeline is
parameterized by the configuration class - _RunnerConfig_ (without hyper-
parameter tuning) and _HPOConfig_ (with hyper-parameter tuning).

Figure 3: Architecture of the EasyTPP library. The dashed arrows show the
different implementation possibilities, either to use pre-defined SOTA TPP
models or provide a custom implementation. All dependencies between the
configurations and modules are visualized by solid arrows with additional
descriptions. Overall, the running of the pipeline is parameterized by the
configuration classes - _RunnerConfig_ (w/o hyper tuning) and _HPOConfig_
(with hyper tuning).

Why and How does EasyTPP support both PyTorch and TensorFlow? PyTorch and
TensorFlow are the two most popular Deep Learning (DL) frameworks today.
PyTorch has a reputation for being a research-focused framework, and indeed,
most of the authors have implemented TPPs in PyTorch, which are used as
references by EasyTPP. On the other hand, TensorFlow has been widely used in
real world applications. For example, Microsoft recommender
444https://github.com/microsoft/recommenders., NVIDIA Merlin
555https://developer.nvidia.com/nvidia-merlin. and Alibaba EasyRec
666https://github.com/alibaba/EasyRec. are well-known industrial user modeling
systems with TensorFlow as the backend. In recent works, TPPs have been
introduced to better capture the evolution of the user preference in
continuous-time (Bao & Zhang, 2021; Fan et al., 2021; Bai et al., 2019). To
support the use of TPPs by industrial practitioners, we implement an
equivalent set of TPPs in TensorFlow. As a result, EasyTPP not only helps
researchers analyze the strengths and bottlenecks of existing models, but also
facilitates the deployment of TPPs in industrial applications.

See Appendix B for more details on the interface and examples of difference
user cases.

##  6 Experimental Evaluation

###  6.1 Experimental Setup

We comprehensively evaluate 8 widely-cited state-of-the-art models in our
benchmark:

  * •

Two RNN-based models: Recurrent marked temporal point process (RMTPP) (Du et
al., 2016) and neural Hawkes Process (NHP) (Mei & Eisner, 2017).

  * •

Three attention-based models: self-attentive Hawkes pocess (SAHP) (Zhang et
al., 2020), transformer Hawkes process (THP) (Zuo et al., 2020), attentive
neural Hawkes process (AttNHP) (Yang et al., 2022a).

  * •

One TPP with he fully neural network based intensity: FullyNN (Omi et al.,
2019).

  * •

One intensity-free model IFTPP (Shchur et al., 2020).

  * •

One TPP with the hidden state evolution governed by a neural ODE: ODETPP. It
is a simplified version of the TPP proposed by Chen et al. (2021) by removing
the spatial component.

We conduct experiments on $1$ synthetic and $5$ real-world datasets from
popular works that contain diverse characteristics in terms of their
application domains and temporal statistics (see Table 1):

  * •

Synthetic. This dataset contains synthetic event sequences from a univariate
Hawkes process sampled using Tick (Bacry et al., 2017) whose conditional
intensity function is defined by
$\lambda(t)=\mu+\sum_{t_{i}<t}\alpha\beta\cdot\text{exp}{-\beta(t-t_{i})}$
with $\mu=0.2,\alpha=0.8,\beta=1.0$. We randomly sampled disjoint train, dev,
and test sets with $1200$, $200$ and $400$ sequences.

  * •

Amazon(Ni, 2018). This dataset includes time-stamped user product reviews
behavior from January, 2008 to October, 2018. Each user has a sequence of
produce review events with each event containing the timestamp and category of
the reviewed product, with each category corresponding to an event type. We
work on a subset of $5200$ most active users with an average sequence length
of $70$ and then end up with $K=16$ event types.

  * •

Retweet (Ke Zhou & Song, 2013). This dataset contains time-stamped user
retweet event sequences. The events are categorized into $K=3$ types: retweets
by “small,” “medium” and “large” users. Small users have fewer than $120$
followers, medium users have fewer than $1363$, and the rest are large users.
We work on a subset of $5200$ active users with an average sequence length of
$70$.

  * •

Taxi (Whong, 2014). This dataset tracks the time-stamped taxi pick-up and
drop-off events across the five boroughs of the New York City; each (borough,
pick-up or drop-off) combination defines an event type, so there are $K=10$
event types in total. We work on a randomly sampled subset of $2000$ drivers
with an average sequence length of $39$.

  * •

Taobao (Xue et al., 2022). This dataset contains time-stamped user click
behaviors on Taobao shopping pages from November 25 to December 03, 2017. Each
user has a sequence of item click events with each event containing the
timestamp and the category of the item. The categories of all items are first
ranked by frequencies and the top $19$ are kept while the rest are merged into
one category, with each category corresponding to an event type. We work on a
subset of $4800$ most active users with an average sequence length of $150$
and then end up with $K=20$ event types.

  * •

StackOverflow (Leskovec & Krevl, 2014). This dataset has two years of user
awards on a question-answering website: each user received a sequence of
badges and there are $K=22$ different kinds of badges in total. We work on a
subset of $2200$ active users with an average sequence length of $65$.

All preprocessed datasets are available at Google
Drive777https://drive.google.com/drive/u/0/folders/1f8k82-NL6KFKuNMsUwozmbzDSFycYvz7..

Dataset | $K$ | # of Event Tokens | Sequence Length
---|---|---|---
|  |  Train |  Dev |  Test |  Min |  Mean |  Max
Retweet |  $3$ |  $369000$ |  $62000$ |  $61000$ |  10 |  $41$ |  $97$
Taobao |  $17$ |  $350000$ |  $53000$ |  $101000$ |  $3$ |  $51$ |  $94$
Amazon |  $16$ |  $288000$ |  $12000$ |  $30000$ |  $14$ |  $44$ |  $94$
Taxi |  $10$ |  $51000$ |  $7000$ |  $14000$ |  $36$ |  $37$ |  $38$
StackOverflow |  $22$ |  $90000$ |  $25000$ |  $26000$ |  $41$ |  $65$ |  $101$
Hawkes-1D |  $1$ |  $55000$ |  $7000$ |  $15000$ |  $62$ |  $79$ |  $95$
Table 1: Statistics of each dataset.

Evaluation Protocol. We keep the model architectures as the original
implementations in their papers. For a fair comparison, we use the same
training procedure for all the models: we used the same optimizer (Adam
(Kingma & Ba, 2015) with default parameters), biases initialized with zeros,
no learning rate decay, the same maximum number of training epochs, and early
stopping criterion (based on log-likelihood on the held-out dev set) for all
models.

We mainly examine the models in two standard scenarios.

  * •

Goodness-of-fit: we fit the models on the train set and measure the log-
probability they assign to the held-out data.

  * •

Next-event prediction: we use the minimum Bayes risk (MBR) principle to
predict the next event time given only the preceding events, as well as its
type given both its true time and the preceding events. We evaluate the time
and type prediction by RMSE and error rate, respectively.

In addition, we propose a new evaluation task: the long-horizon prediction.
Given the prefix of each held-out sequence $x_{[0,T]}$, we autoregressively
predict the next events in a future horizon $\hat{x}_{(T,T^{{}^{\prime}}]}$.
It is evaluated by measuring the optimal transport distance (OTD), a type of
edit distance for event sequences (Mei et al., 2019), between the prediction
$\hat{x}_{(T,T^{{}^{\prime}}]}$ and ground truth $x_{(T,T^{{}^{\prime}}]}$. As
pointed out by Xue et al. (2022), long-horizon prediction of event sequences
is essential in various real-world domains, and this task provides new insight
into the predictive performance of the models.

It is worth noting that: i) IFTPP does not explicitly calculate intensity,
which is not applicable for the thinning sampling required in our benchmarking
process; ii) FullyNN, faithfully implemented based on the author’s version,
does not support multi-type event sequences. Therefore these two models are
excluded from the prediction task.

###  6.2 Results and Analysis

#### Main Results on Goodness-of-Fit and Next-Event Prediction.

The results are displayed in Figure 4 and Figure 5:

  * •

Figure 4 reports the log-likelihood on three held-out datasets for all the
methods. We find IFTPP outperforms all the competitors because it evaluates
the log-likelihood in a close form while the others (RMTPP, NHP, THP, AttNHP,
ODETPP) compute the intensity function via Monte Carlo integration, causing
numerical approximation errors. FullyNN method, which also exactly computes
the log-likelihood, has the worst fitness performance. As Shchur et al. (2020)
points out, the PDF of FullyNN does not integrate to $1$ due to a suboptimal
choice of the network architecture, therefore causing a negative impact on the
performance.

  * •

Figure 5 reports the time and type prediction results on three real datasets.
We find there is no single winner against all the other methods. Attention-
based methods (SAHP, THP, AttNHP) generally perform better than or close to
non-attention methods (RMTPP, NHP, ODETPP) on Amazon, Taobao, and
Stackoverflow, while NHP is the winner on both Retweet and Taxi. We see that
NHP is a comparable strong baseline with attention-based TPPs. This is not too
surprising because similar results have been reported in previous studies
(Yang et al., 2022a; Shi et al., 2023).

Please see Appendix E for the complete results on all the datasets. With a
growing number of TPP methods proposed, we will continuously expand the
catalog of models and datasets and actively update the benchmark in our Github
repository.

(a) Synthetic 1-D Hawkes

(b) Retweet

(c) Taxi

Figure 4: Performance of all the methods on the goodness-of-fit task on
synthetic Hawkes, Retweet, and Taxi data. A higher score is better. All
methods are implemented in PyTorch.

(a) Amazon

(b) Retweet

(c) Taxi

(d) Taobao

(e) StackOverflow

(f) Amazon

(g) Retweet

(h) Taxi

(i) Taobao

(j) StackOverflow

Figure 5: Performance of all the methods on next-event’s time prediction
(first row) and next-event’s type prediction (second row) on five real
datasets. Lower score is better. All methods are implemented in PyTorch. As
clarified, FullyNN and IFTPP are not applicable for the prediction tasks.

#### Analysis-I: Long Horizon Prediction.

We evaluate the long horizon prediction task on Retweet and Taxi datasets. On
both datasets, we set the prediction horizon to be the one that approximately
has $5$ and $10$ events, respectively. Shown in Figure 7 and Figure 7, we find
that AttNHP and THP are two co-winners on Retweet and THP is a single winner
on Taxi. Nonetheless, the margin of the winner over the competitors is small.
Due to the fact that these models are autoregressive and locally normalized,
they are all exposed to cascading errors. To fix this issue, one could resort
to globally normalized models (Xue et al., 2022), which is out of the scope of
the paper.

Figure 6: Long Horizon Prediction on Retweet data: left (avg prediction
horizon $5$ events) vs. right (avg prediction horizon $10$ events).

Figure 7: Long Horizon Prediction on Taxi data: left (avg prediction horizon
$5$ events) vs. right (avg prediction horizon $10$ events).

#### Analysis-II: Models with Different Frameworks: PyTorch vs. TensorFlow.

Researchers normally implement their experiments and models for specific ML
frameworks. For example, recently proposed methods are mostly restricted to
PyTorch and are not applicable to TensorFlow models. As explained in section
5, to facilitate the use of TPPs, we implement two equivalent sets of methods
in PyTorch and TensorFlow. Figure 8 shows the performance of the next event
prediction with models in TensorFlow: under the same training procedure, they
all give close results to those in PyTorch (see Figure 5). To conclude,
although the code could not be exactly the same, the two sets of models
produce similar performance in terms of predictive ability.

(a) Amazon

(b) Amazon

(c) Retweet

(d) Retweet

(e) Taxi

(f) Taxi

Figure 8: Performance of all the methods on next-event’s time prediction
(first row) and next-event’s type prediction (second row). All methods are
implemented in TensorFlow.

##  7 Conclusion

In this work, we presented EasyTPP, a versatile benchmarking platform for the
standardized and transparent comparison of TPP methods on different integrated
data sets. With a growing open-source community, EasyTPP can evolve to be the
main library for benchmarking TPPs. We hope that this work contributes to
further advances in the research.

##  8 Limitation and Societal Impacts

#### Limitations.

Our framework mainly implements models with neural networks, which are known
to be data-hungry. Although they worked well in our experiments, it might
still suffer compared to non-neural models if starved of data.

#### Societal Impacts.

By releasing the benchmarking code and data, we hope to facilitate the
modeling of continuous-time sequential data in many domains. However, our
method may be applied to unethical ends. For example, its abilities of better
fitting data and making more accurate predictions could potentially be used
for unwanted tracking of individual behavior, e.g. for surveillance.

## References

  * Abadi et al. (2016) Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., Devin, M., Ghemawat, S., Irving, G., Isard, M., et al.  Tensorflow: a system for large-scale machine learning.  In _OSDI’16_ , volume 16, pp. 265–283, 2016.
  * Akiba et al. (2019) Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M.  Optuna: A next-generation hyperparameter optimization framework.  In _Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2019.
  * Bacry et al. (2017) Bacry, E., Bompaire, M., Gaïffas, S., and Poulsen, S.  tick: a Python library for statistical learning, with a particular emphasis on time-dependent modeling.  _ArXiv e-prints_ , 2017.
  * Bai et al. (2019) Bai, T., Zou, L., Zhao, W. X., Du, P., Liu, W., Nie, J.-Y., and Wen, J.-R.  Ctrec: A long-short demands evolution model for continuous-time recommendation.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR’19, pp. 675–684, New York, NY, USA, 2019. Association for Computing Machinery.  ISBN 9781450361729.
  * Bao & Zhang (2021) Bao, J. and Zhang, Y.  Time-aware recommender system via continuous-time modeling.  In _Proceedings of the 30th ACM International Conference on Information and Knowledge Management_ , CIKM ’21, pp. 2872–2876, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450384469.
  * Boyd et al. (2020) Boyd, A., Bamler, R., Mandt, S., and Smyth, P.  User-dependent neural sequence models for continuous-time event data.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2020.
  * Chen et al. (2021) Chen, R. T., Amos, B., and Nickel, M.  Neural spatio-temporal point processes.  _ICLR_ , 2021.
  * Cramér (1969) Cramér, H.  Historical review of Filip Lundberg’s works on risk theory.  _Scandinavian Actuarial Journal_ , 1969(sup3):6–12, 1969.
  * Daley & Vere-Jones (2007) Daley, D. J. and Vere-Jones, D.  _An Introduction to the Theory of Point Processes, Volume II: General Theory and Structure_.  Springer, 2007.
  * Deng et al. (2009) Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 248–255, 2009.
  * Deshpande et al. (2021) Deshpande, P., Marathe, K., De, A., and Sarawagi, S.  Long horizon forecasting with temporal point processes.  _WSDM_ , abs/2101.02815, 2021.
  * Du et al. (2016) Du, N., Dai, H., Trivedi, R., Upadhyay, U., Gomez-Rodriguez, M., and Song, L.  Recurrent marked temporal point processes: Embedding event history to vector.  In _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2016.
  * Fan et al. (2021) Fan, Z., Liu, Z., Zhang, J., Xiong, Y., Zheng, L., and Yu, P. S.  Continuous-time sequential recommendation with temporal graph collaborative transformer.  In _Proceedings of the 30th ACM International Conference on Information and Knowledge Management_ , CIKM ’21, pp. 433–442, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450384469.
  * Hasbrouck (1991) Hasbrouck, J.  Measuring the information content of stock trades.  _The Journal of Finance_ , 46(1):179–207, 1991\.
  * Hawkes (1971) Hawkes, A. G.  Spectra of some self-exciting and mutually exciting point processes.  _Biometrika_ , 1971.
  * Hernandez et al. (2017) Hernandez, S., Álvarez, P., Fabra, J., and Ezpeleta, J.  Analysis of users’ behavior in structured e-commerce websites.  _IEEE Access_ , 5:11941–11958, 2017.
  * Jin et al. (2020) Jin, Z., Guo, S., Chen, N., Weiskopf, D., Gotz, D., and Cao, N.  Visual causality analysis of event sequence data.  _IEEE transactions on visualization and computer graphics_ , 27(2):1343–1352, 2020.
  * Ke Zhou & Song (2013) Ke Zhou, H. Z. and Song, L.  Learning triggering kernels for multi-dimensional hawkes processes.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2013.
  * Kingma & Ba (2015) Kingma, D. and Ba, J.  Adam: A method for stochastic optimization.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2015.
  * Leskovec & Krevl (2014) Leskovec, J. and Krevl, A.  SNAP Datasets: Stanford large network dataset collection, 2014.
  * Lewis & Shedler (1979) Lewis, P. A. and Shedler, G. S.  Simulation of nonhomogeneous Poisson processes by thinning.  _Naval Research Logistics Quarterly_ , 1979.
  * Liniger (2009) Liniger, T. J.  _Multivariate Hawkes processes_.  Diss., Eidgenössische Technische Hochschule ETH Zürich, Nr. 18403, 2009.
  * Mei & Eisner (2017) Mei, H. and Eisner, J.  The neural Hawkes process: A neurally self-modulating multivariate point process.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Mei et al. (2019) Mei, H., Qin, G., and Eisner, J.  Imputing missing events in continuous-time event streams.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2019.
  * Mei et al. (2020) Mei, H., Qin, G., Xu, M., and Eisner, J.  Neural Datalog through time: Informed temporal modeling via logical specification.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2020.
  * Ni (2018) Ni, J.  Amazon review data, 2018\.
  * Ogata (1988) Ogata, Y.  Statistical models for earthquake occurrences and residual analysis for point processes.  _J. Am. Stat. Assoc_ , 83(401):9–27, 1988.
  * Omi et al. (2019) Omi, T., Ueda, N., and Aihara, K.  Fully neural network based model for general temporal point processes.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2019.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.  Pytorch: An imperative style, high-performance deep learning library.  _arXiv preprint arXiv:1912.01703_ , 2019.
  * Qu et al. (2023) Qu, C., Tan, X., Xue, S., Shi, X., Zhang, J., and Mei, H.  Bellman meets hawkes: Model-based reinforcement learning via temporal point processes.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2023.
  * Rubanova et al. (2019) Rubanova, Y., Chen, R. T., and Duvenaud, D. K.  Latent ordinary differential equations for irregularly-sampled time series.  _Advances in Neural Information Processing Systems (NeurIPS)_ , 32, 2019.
  * Shchur et al. (2020) Shchur, O., Biloš, M., and Günnemann, S.  Intensity-free learning of temporal point processes.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2020.
  * Shchur et al. (2021) Shchur, O., Türkmen, A. C., Januschowski, T., and Günnemann, S.  Neural temporal point processes: A review.  In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI’21)_ , 2021.
  * Shi et al. (2023) Shi, X., Xue, S., Wang, K., Zhou, F., Zhang, J. Y., Zhou, J., Tan, C., and Mei, H.  Language models can improve event prediction by few-shot abductive reasoning.  _arXiv preprint arXiv:2305.16646_ , 2023.
  * Trivedi et al. (2019) Trivedi, R., Farajtabar, M., Biswal, P., and Zha, H.  DyRep: Learning representations over dynamic graphs.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2019.
  * Upadhyay et al. (2018) Upadhyay, U., De, A., and Gomez-Rodriguez, M.  Deep reinforcement learning of marked temporal point processes.  _NeurIPS_ , 2018.
  * Vaswani et al. (2017) Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., and Polosukhin, I.  Attention is all you need.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2017.
  * Wang et al. (2019) Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  _ICLR_ , 2019.
  * Wen et al. (2023) Wen, Q., Zhou, T., Zhang, C., Chen, W., Ma, Z., Yan, J., and Sun, L.  Transformers in time series: A survey.  In _International Joint Conference on Artificial Intelligence(IJCAI)_ , 2023.
  * Whong (2014) Whong, C.  FOILing NYC’s taxi trip data, 2014.
  * Williams et al. (2020) Williams, A., Degleris, A., Wang, Y., and Linderman, S.  Point process models for sequence detection in high-dimensional neural spike trains.  _Advances in neural information processing systems_ , 33:14350–14361, 2020.
  * Xiao et al. (2017) Xiao, S., Yan, J., Yang, X., Zha, H., and Chu, S.  Modeling the intensity function of point process via recurrent neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2017.
  * Xue et al. (2021) Xue, S., Shi, X., Hao, H., Ma, L., Zhang, J., Wang, S., and Wang, S.  A graph regularized point process model for event propagation sequence.  In _2021 International Joint Conference on Neural Networks (IJCNN)_ , pp. 1–7, 2021.
  * Xue et al. (2022) Xue, S., Shi, X., Zhang, Y. J., and Mei, H.  Hypro: A hybridly normalized probabilistic model for long-horizon prediction of event sequences.  In _Advances in Neural Information Processing Systems (NeurIPS)_ , 2022.
  * Yang et al. (2022a) Yang, C., Mei, H., and Eisner, J.  Transformer embeddings of irregularly spaced events and their participants.  In _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2022a.
  * Yang et al. (2022b) Yang, C., Wu, Q., Wen, Q., Zhou, Z., Sun, L., and Yan, J.  Towards out-of-distribution sequential event prediction: A causal treatment.  In _Advances in Neural Information Processing Systems_ , 2022b.
  * Yang et al. (2018) Yang, G., Cai, Y., and Reddy, C. K.  Recurrent spatio-temporal point process for check-in time prediction.  In _CIKM_ , 2018.
  * Zhang et al. (2020) Zhang, Q., Lipani, A., Kirnap, O., and Yilmaz, E.  Self-attentive Hawkes process.  In _Proceedings of the International Conference on Machine Learning (ICML)_ , 2020.
  * Zhu et al. (2021) Zhu, J., Liu, J., Yang, S., Zhang, Q., and He, X.  Open benchmarking for click-through rate prediction.  In Demartini, G., Zuccon, G., Culpepper, J. S., Huang, Z., and Tong, H. (eds.), _CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021_ , pp. 2759–2769. ACM, 2021.
  * Zuo et al. (2020) Zuo, S., Jiang, H., Li, Z., Zhao, T., and Zha, H.  Transformer Hawkes process.  In _International Conference on Machine Learning_ , pp. 11692–11702. PMLR, 2020.

##  Appendix A Limitation and Societal Impacts

#### Limitations.

Our framework mainly implements models with neural networks, which are known
to be data-hungry. Although they worked well in our experiments, it might
still suffer compared to non-neural models if starved of data.

#### Societal Impacts.

By releasing the benchmarking code and data, we hope to facilitate the
modeling of continuous-time sequential data in many domains. However, our
method may be applied to unethical ends. For example, its abilities of better
fitting data and making more accurate predictions could potentially be used
for unwanted tracking of individual behavior, e.g. for surveillance.

##  Appendix B EasyTPP’s Software Interface Details

In this section, we describe the architecture of our open-source benchmarking
software EasyTPP in more detail and provide examples of different use cases
and their implementation.

###  B.1 High Level Software Architecture

The purpose of building EasyTPP is to provide a simple and standardized
framework to allow users to apply different state-of-the-art (SOTA) TPPs to
arbitrary data sets. For researchers, EasyTPP provides an implementation
interface to integrate new recourse methods in an easy-to-use way, which
allows them to compare their method to already existing methods. For
industrial practitioners, the availability of benchmarking code helps them
easily assess the applicability of TPP models for their own problems.

A high level visualization of the EasyTPP’s software architecture is depicted
in Figure 9. _Data Preprocess_ component provides a common way to access the
event data across the software and maintains information about the features.
For the _Model_ component, the library provides the possibility to use
existing methods or extend the users’ custom methods and implementations. A
_wrapper_ encapsulates the black-box models along with the trainer and
sampler. The primary purpose of the wrapper is to provide a common interface
to easily fit in the training and evaluation pipeline, independently of their
framework (e.g., PyTorch, TensorFlow). See section B.2 and section B.3 for
details. The running of the pipeline is parameterized by the configuration
class - _RunnerConfig_ (without hyper-parameter tuning) and _HPOConfig_ (with
hyper-parameter tuning).

Figure 9: Architecture of the EasyTPP library. The dashed arrows show the
different implementation possibilities, either to use pre-defined SOTA TPP
models or provide a custom implementation. All dependencies between these
objects are visualized by solid arrows with an additional description. The
running of the pipeline is parameterized by the configuration classes -
_RunnerConfig_ (w/o hyper tuning) and _HPOConfig_ (with hyper tuning).

###  B.2 Why does EasyTPP support both TensorFlow and PyTorch

TensorFlow and PyTorch are the two most popular Deep Learning (DL) frameworks
today. PyTorch has a reputation for being a research-focused framework, and
indeed, most of the authors have implemented TPPs in PyTorch, which are used
as references by EasyTPP. On the other hand, TensorFlow has been widely used
in real world applications. For example, Microsoft recommender
888https://github.com/microsoft/recommenders., NVIDIA Merlin
999https://developer.nvidia.com/nvidia-merlin. and Alibaba EasyRec
101010https://github.com/alibaba/EasyRec. are well-known industrial user
modeling systems with TensorFlow as the backend. In recent works, TPPs have
been introduced to better capture the evolution of the user preference in
continuous-time (Bao & Zhang, 2021; Fan et al., 2021; Bai et al., 2019). To
support the use of TPPs by industrial practitioners, we implement an
equivalent set of TPPs in TensorFlow. As a result, EasyTPP not only helps
researchers analyze the strengths and bottlenecks of existing models, but also
facilitates the deployment of TPPs in industrial applications.

###  B.3 How does EasyTPP support both PyTorch and TensorFlow

We implement two equivalent sets of data loaders, models, trainers, thinning
samplers in TensorFlow and PyTorch, respectively, then use wrappers to
encapsulate them so that they have the same API exposed in the whole training
and evaluation pipeline. See Figure 10.

Figure 10: Illustration of TensorFlow and PyTorch Wrappers in the EasyTPP
library.

###  B.4 EasyTPP for Researchers

The research groups can inherit from the _BaseModel_ to implement their own
method in EasyTPP. This opens up a way of standardized and consistent
comparisons between different TPPs when exploring new models.

Specifically, if we want to customize a TPP in PyTorch, we need to initialize
the model by inheriting the class _TorchBaseModel_ :

⬇

1

2from easy_tpp.model.torch_model.torch_basemodel import TorchBaseModel

3

4# Custom Torch TPP implementations need to

5# inherit from the TorchBaseModel interface

6class NewModel(TorchBaseModel):

7 def __init__(self, model_config):

8 super(NewModel, self).__init__(model_config)

9

10 # Forward along the sequence, output the states / intensities at the event
times

11 def forward(self, batch):

12 ...

13 return states

14

15

16 # Compute the loglikelihood loss

17 def loglike_loss(self, batch):

18 ....

19 return loglike

20

21 # Compute the intensities at given sampling times

22 # Used in the Thinning sampler

23 def compute_intensities_at_sample_times(self, batch, sample_times,
**kwargs):

24 ...

25 return intensities

Listing 1: Pseudo implementation of customizing a TPP model in PyTorch using
EasyTPP.

Equivalent, if we want to customize a TPP in TensorFlow, we need to initialize
the model by inheriting the class _TfBaseModel_ :

⬇

1

2from easy_tpp.model.torch_model.tf_basemodel import TfBaseModel

3

4# Custom Torch TPP implementations need to

5# inherit from the TorchBaseModel interface

6class NewModel(TfBaseModel):

7 def __init__(self, model_config):

8 super(NewModel, self).__init__(model_config)

9

10 # Forward along the sequence, output the states / intensities at the event
times

11 def forward(self, batch):

12 ...

13 return states

14

15

16 # Compute the loglikelihood loss

17 def loglike_loss(self, batch):

18 ....

19 return loglike

20

21 # Compute the intensities at given sampling times

22 # Used in the Thinning sampler

23 def compute_intensities_at_sample_times(self, batch, sample_times,
**kwargs):

24 ...

25 return intensities

Listing 2: Pseudo implementation of customizing a TPP model in TensorFlow
using EasyTPP.

###  B.5 EasyTPP as a Modeling Library

A common usage of the package is to train and evaluate some standard TPPs.
This can be done by loading black-box-models and data sets from our provided
datasets, or by user-defined models and datasets via integration with the
defined interfaces. Listing LABEL:fig:library shows an implementation example
of a simple use-case, fitting a TPP model method to a preprocessed dataset
from our library.

⬇

1import argparse

2

3from easy_tpp.config_factory import Config

4from easy_tpp.runner import Runner

5

6

7def main():

8 parser = argparse.ArgumentParser()

9

10 parser.add_argument(’--config_dir’,

11 type=str,

12 required=False,

13 default=’configs/experiment_config.yaml’,

14 help=’Dir of configuration yaml to train and evaluate the model.’)

15

16 parser.add_argument(’--experiment_id’,

17 type=str,

18 required=False,

19 default=’IntensityFree_train’,

20 help=’Experiment id in the config file.’)

21

22 args = parser.parse_args()

23

24 # Build up the configuation for the runner

25 config = Config.build_from_yaml_file(args.config_dir,
experiment_id=args.experiment_id)

26

27 # Intialize the runner for the pipeline

28 model_runner = Runner.build_from_config(config)

29

30 # Start running

31 model_runner.run()

32

33

34if __name__ == ’__main__’:

35 main()

Listing 3: Example implementation of running a TPP model using EasyTPP.

##  Appendix C Model Implementation Details

We have implemented the following TPPs

  * •

Recurrent marked temporal point process (RMTPP) (Du et al., 2016). We
implemented both the Tensorflow and PyTorch version of RMTPP by our own.

  * •

Neural Hawkes process (NHP) (Mei & Eisner, 2017) and Attentive neural Hawkes
process (AttNHP) (Yang et al., 2022a). The Pytorch implementation mostly comes
from the code from the public GitHub repository at
https://github.com/yangalan123/anhp-andtt (Yang et al., 2022a) with MIT
License. We developed the Tensorflow version of NHP and ttNHP by our own.

  * •

Self-attentive Hawkes process (SAHP) (Zhang et al., 2020) and transformer
Hawkes process (THP) (Zuo et al., 2020). We rewrote the PyTorch versions of
SAHP and THP based on the public Github repository at
https://github.com/yangalan123/anhp-andtt (Yang et al., 2022a) with MIT
License. We developed the Tensorflow versions of the two models by our own.

  * •

Intensity-free TPP (IFTPP) (Shchur et al., 2020). The Pytorch implementation
mostly comes from the code from the public GitHub repository at
https://github.com/shchur/ifl-tpp (Shchur et al., 2020) with MIT License. We
implemented a Tensorflow version by our own.

  * •

Fully network based TPP (FullyNN) (Omi et al., 2019). We rewrote both the
Tensorflow and PyTorch versions of the model faithfully based on the author’s
code at https://github.com/omitakahiro/NeuralNetworkPointProcess. Please not
that the model only considers the number of the types to be one, i.e., the
sequence’s $K=1$.

  * •

ODE-based TPP (ODETPP) (Chen et al., 2021). We implement a TPP model, in both
Tensorflow and PyTorch, with a continuous-time state evolution governed by a
neural ODE. It is basically the spatial-temporal point process (Chen et al.,
2021) without the spatial component.

###  C.1 Likelihood Computation Details

In this section, we discuss the implementation details of NLL computation in
equation 4.

The integral term in equation 4 is computed using the Monte Carlo
approximation given by Mei & Eisner (2017, Algorithm 1), which samples times
$t$. This yields an unbiased stochastic gradient. For the number of Monte
Carlo samples, we follow the practice of Mei & Eisner (2017): namely, at
training time, we match the number of samples to the number of observed events
at training time, a reasonable and fast choice, but to estimate log-likelihood
when tuning hyperparameters or reporting final results, we take 10 times as
many samples.

At each sampled time $t$, the Monte Carlo method still requires a summation
over all events to obtain $\lambda(t)$. This summation can be expensive when
there are many event types. This is not a serious problem for our EasyTPP
implementation since it can leverage GPU parallelism.

###  C.2 Next Event Prediction

It is possible to sample event sequences exactly from any intensity-based
model in EasyTPP, using the thinning algorithm that is traditionally used for
autoregressive point processes (Lewis & Shedler, 1979; Liniger, 2009). In
general, to apply the thinning algorithm to sample the next event at time
$\geq t_{0}$, it is necessary to have an upper bound on
$\\{\lambda_{{e}}(t):t\in[t_{0},\infty)\\}$ for each event type $t$. An
explicit construction for the NHP (or AttNHP) model was given by Mei & Eisner
(2017, Appendix B.3).

Section 4 includes a task-based evaluation where we try to predict the _time_
and _type_ of just the next event. More precisely, for each event in each
held-out sequence, we attempt to predict its time given only the preceding
events, as well as its type given both its true time and the preceding events.

We evaluate the time prediction with average L2 loss (yielding a root-mean-
squared error, or RMSE) and evaluate the argument prediction with average 0-1
loss (yielding an error rate).

Following Mei & Eisner (2017), we use the minimum Bayes risk (MBR) principle
to predict the time and type with the lowest expected loss. For completeness,
we repeat the general recipe in this section.

For the $i$th event, its time $t_{i}$ has density
$p_{i}(t)=\lambda(t)\exp(-\int_{t_{i-1}}^{t}\lambda(t^{\prime})dt^{\prime})$.
We choose $\int_{t_{i-1}}^{\infty}tp_{i}(t)dt$ as the time prediction because
it has the lowest expected L2 loss. The integral can be estimated using i.i.d.
samples of $t_{i}$ drawn from $p_{i}(t)$ by the thinning algorithm.

Given the next event time $t_{i}$, we choose the most probable type
$\operatorname{argmax}_{e}\lambda_{{e}}(t_{i})$ as the type prediction because
it minimizes expected 0-1 loss.

###  C.3 Long Horizon Prediction

The TPP models are typically autoregressive: predicting each future event is
conditioned on all the previously predicted events. Following the approach in
(Xue et al., 2022), we set up a prediction horizon and use OTD to measure the
divergence between the ground truth sequence and the predicted sequence within
the horizon. For more details about the setup and evaluation protocal, please
see Section 5 in Xue et al. (2022).

##  Appendix D Dataset Details

To comprehensively evaluate the models, we preprocessed one synthetic and five
real-world datasets from widely-cited works that contain diverse
characteristics in terms of their application domains and temporal statistics.

  * •

Synthetic. This dataset contains synthetic event sequences from a univariate
Hawkes process sampled using Tick (Bacry et al., 2017) whose conditional
intensity function is defined by

$\lambda(t)=\mu+\sum_{t_{i}<t}\alpha\beta\cdot\text{exp}{-\beta(t-t_{i})}$

with $\mu=0.2,\alpha=0.8,\beta=1.0$. We randomly sampled disjoint train, dev,
and test sets with $1200$, $200$ and $400$ sequences.

  * •

Amazon (Ni, 2018). This dataset includes time-stamped user product reviews
behavior from January, 2008 to October, 2018. Each user has a sequence of
produce review events with each event containing the timestamp and category of
the reviewed product, with each category corresponding to an event type. We
work on a subset of $5200$ most active users with an average sequence length
of $70$ and then end up with $K=16$ event types.

  * •

Retweet (Ke Zhou & Song, 2013). This dataset contains time-stamped user
retweet event sequences. The events are categorized into $K=3$ types: retweets
by “small,” “medium” and “large” users. Small users have fewer than $120$
followers, medium users have fewer than $1363$, and the rest are large users.
We work on a subset of $5200$ most active users with an average sequence
length of $70$.

  * •

Taxi (Whong, 2014). This dataset tracks the time-stamped taxi pick-up and
drop-off events across the five boroughs of the New York City; each (borough,
pick-up or drop-off) combination defines an event type, so there are $K=10$
event types in total. We work on a randomly sampled subset of $2000$ drivers
and each driver has a sequence. We randomly sampled disjoint train, dev and
test sets with $1400$, $200$ and $400$ sequences.

  * •

Taobao (Xue et al., 2022). This dataset contains time-stamped user click
behaviors on Taobao shopping pages from November 25 to December 03, 2017. Each
user has a sequence of item click events with each event containing the
timestamp and the category of the item. The categories of all items are first
ranked by frequencies and the top $19$ are kept while the rest are merged into
one category, with each category corresponding to an event type. We work on a
subset of $4800$ most active users with an average sequence length of $150$
and then end up with $K=20$ event types.

  * •

StackOverflow (Leskovec & Krevl, 2014). This dataset has two years of user
awards on a question-answering website: each user received a sequence of
badges and there are $K=22$ different kinds of badges in total. We randomly
sampled disjoint train, dev and test sets with $1400,400$ and $400$ sequences
from the dataset.

Table 2 shows statistics about each dataset mentioned above.

Dataset | $K$ | # of Event Tokens | Sequence Length
---|---|---|---
|  |  Train |  Dev |  Test |  Min |  Mean |  Max
Retweet |  $3$ |  $369000$ |  $62000$ |  $61000$ |  10 |  $41$ |  $97$
Taobao |  $17$ |  $350000$ |  $53000$ |  $101000$ |  $3$ |  $51$ |  $94$
Amazon |  $16$ |  $288000$ |  $12000$ |  $30000$ |  $14$ |  $44$ |  $94$
Taxi |  $10$ |  $51000$ |  $7000$ |  $14000$ |  $36$ |  $37$ |  $38$
StackOverflow |  $22$ |  $90000$ |  $25000$ |  $26000$ |  $41$ |  $65$ |  $101$
Hawkes-1D |  $1$ |  $55000$ |  $7000$ |  $15000$ |  $62$ |  $79$ |  $95$
Table 2: Statistics of each dataset.

##  Appendix E Experiment Details

Training Details. For TPPs, the main hyperparameters to tune are the hidden
dimension $D$ of the neural network and the number of layers $L$ of the
attention structure (if applicable). In practice, the optimal $D$ for a model
was usually $16,32,64$; the optimal $L$ was usually $1,2,3,4$. To train the
parameters for a given generator, we performed early stopping based on log-
likelihood on the held-out dev set. The chosen parameters for the main
experiments are given in Table 3.

Computation Cost. All the experiments were conducted on a server with $256$G
RAM, a $64$ logical cores CPU (Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz)
and one NVIDIA Tesla P100 GPU for acceleration. For training, the batch size
is 256 by default. On all the dataset, the training of AttNHP takes most of
the time (i.e., around 4 hours) while other models take less than 2 hours.

More Results. We plot the additional results of next event prediction task
with models implemented in TensorFlow in Figure 11. We will continue update
the benchmarking results on new datasets and new models in our Github
repository.

(a) Taobao

(b) Taobao

(c) StackOverflow

(d) StackOverflow

Figure 11: Performance of all the methods on next-event’s time prediction (first row) and next-event’s type prediction (second row) on Taobao and StackOverflow dataset . All methods are implemented in TensorFlow. Model |  Description |  Value Used
---|---|---
|  $hidden\\_size$ |  $32$
|  $time\\_emb\\_size$ |  $16$
RMTPP |  $num\\_layers$ |  $2$
|  $hidden\\_size$ |  $64$
|  $time\\_emb\\_size$ |  $16$
NHP |  $num\\_layers$ |  $2$
|  $hidden\\_size$ |  $32$
|  $time\\_emb\\_size$ |  $16$
SAHP |  $num\\_layers$ |  $2$
|  $num\\_heads$ |  $2$
|  $hidden\\_size$ |  $64$
|  $time\\_emb\\_size$ |  $16$
THP |  $num\\_layers$ |  $2$
|  $num\\_heads$ |  $2$
|  $hidden\\_size$ |  $32$
|  $time\\_emb\\_size$ |  $16$
AttNHP |  $num\\_layers$ |  $1$
|  $num\\_heads$ |  $2$
|  $hidden\\_size$ |  $32$
ODETPP |  $time\\_emb\\_size$ |  $16$
|  $num\\_layers$ |  $2$
|  $hidden\\_size$ |  $32$
FullyNN |  $time\\_emb\\_size$ |  $16$
|  $num\\_layers$ |  $2$
|  $hidden\\_size$ |  $32$
IntensityFree |  $time\\_emb\\_size$ |  $16$
|  $num\\_layers$ |  $2$
Table 3: Descriptions and values of hyperparameters used for models.

##  Appendix F Additional Note

###  F.1 Citation Count in ArXiv

We search the TPP-related articles in ArXiv 111111https://arxiv.org/ using
their own search engine in three folds:

  * •

Temporal point process: we search through the abstract of articles which
contains the term ’temporal point process’.

  * •

Hawkes process: we search through the abstract of articles with the term
’hawkes process’ but without the term ’temporal point process’.

  * •

Temporal event sequence: we search through the abstract of articles which
include the term ’temporal event sequence’ but exclude the term ’hawkes
process’ and ’temporal point process’.

We group the articles found out by the search engine by years and report it in
section 1.