# Universal Transformer Hawkes Process with Adaptive Recursive Iteration

Lu-ning Zhang<EMAIL_ADDRESS>Jian-wei Liu<EMAIL_ADDRESS>Zhi-yan
Song  Xin Zuo  Department of Automation, College of Information Science and
Engineering, China University of Petroleum , Beijing, Beijing, China

###### Abstract

Asynchronous events sequences are widely distributed in the natural world and
human activities, such as earthquakes records, users’ activities in social
media and so on. How to distill the information from these seemingly
disorganized data is a persistent topic that researchers focus on. The one of
the most useful model is the point process model, and on the basis, the
researchers obtain many noticeable results. Moreover, in recent years, point
process models on the foundation of neural networks, especially recurrent
neural networks (RNN) are proposed and compare with the traditional models,
their performance are greatly improved. Enlighten by transformer model, which
can learning sequence data efficiently without recurrent and convolutional
structure, transformer Hawkes process is come out, and achieves state-of-the-
art performance. However, there is some research proving that the re-
introduction of recursive calculations in transformer can further improve
transformer’s performance. Thus, we come out with a new kind of transformer
Hawkes process model, universal transformer Hawkes process (UTHP), which
contains both recursive mechanism and self-attention mechanism, and to improve
the local perception ability of the model, we also introduce convolutional
neural network (CNN) in the position-wise-feed-forward part. We conduct
experiments on several datasets to validate the effectiveness of UTHP and
explore the changes after the introduction of the recursive mechanism. These
experiments on multiple datasets demonstrate that the performance of our
proposed new model has a certain improvement compared with the previous state-
of-the-art models.

###### keywords:

Hawkes process, universal transformer Hawkes process, convolutional neural
network, recursive calculation, self-attention mechanism

††journal: arXiv

##  1 Introduction

In this era of informatization and data in trajectories, nature and human
activities will be recorded as a large amount of asynchronous sequential data,
for instance, the record of occurrence of earthquakes and aftershocks [1],
transaction history of financial markets [2], electronic health record [3],
equipment failure history [4] and user behavior in social networks [5, 6],
such as Weibo, Twitter, Facebook, etc. all are asynchronous events sequences
data.

These asynchronous sequences contain valuable knowledge and information,
researchers utilize a variety of methods to better mine these knowledge and
information from the data. Among a diversity of methods, point process is one
of the most widely used methods in this field, and in point process models,
Hawkes process [7] is one of the most commonly used model. The self-exciting
progress in Hawkes process fits the interaction between events to some extent,
thus the application of Hawkes process achieves certain results in sequence
analysis.

For example, Zhou et al. present alternating direction method of multipliers
(ADMM)-based algorithm to learn Hawkes process to discover the hidden network
of social influence [7], Xu et al. make use of non-parametric Hawkes process
to reveal the Granger causality between the users activity on watching
internet protocol television (IPTV) [8]. Hansen et al. demonstrate the great
expressive potential of Hawkes process in neuroscience with least absolute
shrinkage and selection operator (LASSO) method [9]. Reynaud-Bouret and
Schbath provide a new way to detect the favored or avoided distances between
genomic events along deoxyribonucleic acid (DNA) sequences [10]. Zhang et al.
[4] modify the traditional Hawkes process, introducing the time-dependent
background intensity to Hawkes process to analyze the background probability
of failure and relationship between failures in compressor station.

However, traditional Hawkes process only considers the positive superposition
of effect of historical events, which severely constrains the fitting ability
of this type of model. Meanwhile, the lack of nonlinear operations in
traditional Hawkes process also sets an upper limit for Hawkes’ expressive
ability. Thus, in recent years, due to the strong fitting ability of neural
networks and especially the sequence modeling ability of RNN, the research
direction in this field is transferred to the neural point process model. For
instance, Du et al. [11] embed the information of sequence data (including
time-stamp and event type) into RNN, and come up with the recurrent marked
temporal point processes (RMTPP) to model the conditional intensity function
by nonlinear dependency of the history. And similar to RMTPP, Mei et al. [12]
propose the continuous-time LSTM (Long Short-Term Memory) to model the
conditional intensity of point process, which is called as neural Hawkes
process, and in this continuous neural Hawkes process, the influence of the
previous event decays with time continuously. Xiao et al. use two RNN to model
the conditional intensity function, one is used to process time stamp
information, and the other is used to process historical event information
[13].

Inevitably, these RNN based-models also inherit the drawbacks of RNN, for
instance, it may take a long time for the patient to develop symptoms due to
certain sequel, which has obvious long-term characteristics, such as diabetes,
cancer and other chronic diseases, while these RNN-based models are hard to
reveal the long-term dependency between the distant events in the sequences
[14]. The ideal point process model should be able to solve these problems.
Moreover, in the training of RNN-based models, such problems as vanishing and
exploding Gradients [15] often occur, and then affect the performance of the
model.

It’s worth noting that in traditional sequence learning problem, such as
machine translation [16] and speech recognition [17], transformer model [18]
based on self-attention [19] mechanism achieves distinct performance
improvement without application of CNN and RNN, meanwhile, the transformer
structure free-from recurrent modules makes the model have higher
computational efficiency. These achievements give a new insight on the
development of sequential data learning. On account of this fact, Zhang et al.
present self-attention Hawkes process [20], furthermore, Zuo et al. [21]
propose transformer Hawkes process based on the attention mechanism and
encoder structure in transformer. This model utilizes pure transformer
structure without using RNN and CNN, and achieves state-of-the-art
performance, but there is still much room for improvement in the transformer
models, for instance, transformer simply stack the encoder layer to learn
sequence data, and foregoes the recursive bias learning in RNN, while this
recursive learning might be more important than commonly believed.

Dehghani et al. point out that re-introduce recurrence calculation in
transformer maybe promote the performance of transformer, which is called as
universal transformer [22], this model combines the advantage of transformer
and RNN, organically combines the self-attention and recurrence learning
mechanism. In recurrence process, Dehghani et al. make use of the ACT
mechanism [23] to decide when the recurrence process will halt. The
experimental results of universal transformer demonstrate that the
effectiveness of the combination of self-attention mechanism and recurrence
mechanism.

Based on the achievements of universal transformer, we tend to work out a new
framework of transformer Hawkes process based on the idea of universal
transformer, we name it as universal transformer Hawkes process. We introduce
the recurrent structure in transformer Hawkes process, and make our model to
achieve Turing completeness compared with previous transformer model.
Moreover, we add a convolution module to the position-wise-feed-forward-layer,
which enhance the local perception ability of universal transformer Hawkes
process. We conduct the experiments on multiple dataset to compare with state-
of-the-art baselines, to validate the effectiveness of our model. We also
demonstrate whether the additional RNN layers will have a positive impact on
fitting mutual interdependence among the events in the sequence. In addition,
to demonstrate the effectiveness of the ACT mechanism, we compare the
performances of universal transformer with and without the ACT mechanism, and
verify that the halting mechanism of dynamic iteration will make the model
perform better overall.

Our paper is organized as follows. In Section 2, we introduce the related work
about Hawkes process and point process in view of the neural network. In
Section 3, we are going to instruct our model in details, including structure,
condition intensity function, prediction tasks and training process. At last,
Section 4 lists our experimental results to illustrate the advantages of
universal transformer Hawkes process and t ACT mechanism. At last, Section 5
concludes the article.

##  2 Related work

###  2.1 Hawkes process

Hawkes process has form shown as the following:

$\lambda(t)=\mu(t)+\sum\limits_{i:t_{i}<t}{\phi(t-t_{i})}$ (1)

where $\mu(t)$ is background intensity function, indicates the background
probability of event occurrence, $\phi(t)$ is the impact function, which used
to measure the historical event influence, and
$\sum\limits_{i:t_{i}<t}{\phi(t-t_{i})}$ records the impact of all historical
events on the current instant. Traditional Hawkes process model in Eq. 1
assumes the positive superposition of past historical impact. Until now, there
are many variants of traditional Hawkes process.

Zhao et al. firstly make use of Hawkes process to model the Twitter data to
predict the final number of reshares, which only have small relative error
[6]. Xu et al. use the non-parametric method to represent the Hawkes process,
and utilize a corresponding learning algorithm to get the better performance
of the model, and then the Granger causality on of IPTV user on watching
program is obtained [8]. Kobayashi and Lamhbiotte present a time-dependent
Hawkes process, whose impact functions are time-dependent. They validate the
method on Twitter data and prove that there is a systematic improvement to the
previous method [24]. Yang et al. develop an online learning method of Hawkes
process based on the nonparametric method [25]. In 2018, Alan reviews and
summarizes the application of the Hawkes process proposed by him in the
financial field [26]. Mohler presents a novel modulated Hawkes process, to
quickly identify risks and trigger appropriate public safety responses in
communities to prevent a range of social harm events, such as crime, drug
abuse, traffic accident and medical emergencies [27].

Zhang et al. [4] make significant improvement to the Hawkes process model, the
Weibull background intensity is used instead of constant background intensity.
Weibull background intensity is a kind of time-dependent background intensity,
which can better describe the trend of the base possibility of the event over
time. After verifying the effectiveness of this Weibull-Hawkes process and the
corresponding learning algorithm, based on this new model, Zhang et al.
analyze the failure sequence of the compressor station and list the trend of
background probability of failures in the compressor station over time and the
Granger causality between the failures, and some suggestions are made for the
production of the compressor station.

###  2.2 Transformer and Universal Transformer

In 2017, Vaswani et al. propose transformer model [18], which makes full use
of self-attention [19] module, and discards the CNN and RNN structure,
achieves a great improvement in the field of sequence learning, such as
natural language processing [28].

However, recent research shows that the recurrent learning in RNN can have a
greater role beyond imagination, thus, Dehghani et al. propose universal
transformer [22] which combine the recurrent learning and self-attention
mechanism, moreover, in order to better allocate model computing resources,
the ACT mechanism [23] is introduced into models, then, this model achieves
better results than transformers.

###  2.3 Neural Hawkes Process

In general, the neural network has a stronger nonlinear fitting ability than
other models, especially RNN is better at learning sequence data than other
neural networks, so it has been often used in the processing of sequence data
such as speech and text.

Therefore, researchers also hope to use RNN to fit asynchronous event sequence
data. Du et al. present RMTPP models [11], to learn the history effect via
RNN, including history event type and time-stamp. For the first time, this
neural model abandons the overly strong assumptions of the Hawkes process and
other point process models, and achieves greater improvements. Xiao et al.
make use of two RNN to model the event sequence, one of them is used to model
the background intensity and the other is used to model the impact of
historical events [13]. This method allows a black-box treatment for modeling
the conditional intensity function, and end to end training can make model
event sequences easier. Mei and Eisner propose a new form of LSTM [12], the
continuous time LSTM, whose state can decay to another with time, and based on
this LSTM they come up with the neural Hawkes process to model the
asynchronous event sequence, which obtains better model performance.

While the shortcomings of RNN gradually appear, researchers tend to use better
neural networks to model event sequences. Zhang et al. [20] firstly utilize
self-attention mechanism to get the Hawkes process of the sequence. And based
on the achievement of transformer, and enlighten by [18], Zuo et al. utilize
the encoder structure in transformer, and propose the transformer Hawkes
process (THP) [21], this model encodes and converts event sequence data into
hidden representations, and then maps hidden representations into continuous
conditional intensity functions.

##  3 Proposed Model

Table 1: Nomenclature

Symbols | Description | Variable names in ACT mechanism | Size
---|---|---|---
$S_{e}$ | The dataset of sequences | / | /
$s_{n}$ | The $n$-th sequence | / | /
$I_{n}$ | The length of $n$-th sequence | / | /
$N$ | The total number of sequences | / | $\mathbb{N}^{+}$
$C$ | The total number of type of events in sequences | / | $\mathbb{N}^{+}$
$t_{i}$ | The time stamp of $i$-th event | / | $\mathbb{R}$
$c_{i}$ | The event type of $i$-th event | / | $\mathbb{N}^{+}$
$D$ | The model dimension of universal transformer | / | $\mathbb{N}^{+}$
$\bm{K}$ | The embedding matrix of event type | / | $\mathbb{R}^{D\times C}$
max_n | The max iteration times of ACT mechanism | / | $\mathbb{N}^{+}$
$\mathbb{I}(\cdot)$ | Indicator function | / | /
EncodingLayer | The encoding layer in ACT mechanism | / | /
$T_{h}$ | The threshold in ACT mechanism | threshold | $\mathbb{R}$
$\bm{W}_{p}$ | The weight p in ACT mechanism | weight_p | $\mathbb{R}^{D\times 1}$
$S$ | The state variable in ACT mechanism | state | $\mathbb{R}^{I_{n}\times D}$
$P_{S}$ | The previous state variable in ACT mechanism | previous_state | $\mathbb{R}^{I_{n}\times D}$
$h_{p}$ | The halting probability variable in ACT mechanism | halting_probability | $\mathbb{R}^{D}$
$R_{e}$ | The remainders variable in ACT mechanism | remainders | $\mathbb{R}^{D}$
$n$ | The upgrade times of state in ACT mechanism | n_updatas | $\mathbb{R}^{D}$
$S_{r}$ | The still running variable in ACT mechanism | still_running | $\mathbb{R}^{D}$
$n_{h}$ | The new halted variable in ACT mechanism | new_halted | $\mathbb{R}^{D}$
$W$ | The update weight of ACT mechanism | update_weight | $\mathbb{R}^{D}$
$L$ | he number of multi-head attention | / | $\mathbb{R}$
$\left\\{{{\bm{A}}_{l}}\right\\}_{l=1}^{L}$ | The number of multi-head attention | / | $\mathbb{R}^{D_{V}}$
${D_{K}}$ | The dimension of query and key vector | / | $\mathbb{N}^{+}$
${D_{V}}$ | The dimension of value vector | / | $\mathbb{N}^{+}$
$\left\\{{{\bm{Q}}_{l}}\right\\}_{l=1}^{L}$ | The query variable of multi-head attention | / | $\mathbb{R}^{I_{n}\times D_{K}}$
$\left\\{{{\bm{K}}_{l}}\right\\}_{l=1}^{L}$ | The key variable of multi-head attention | / | $\mathbb{R}^{I_{n}\times D_{K}}$
$\left\\{{{\bm{V}}_{l}}\right\\}_{l=1}^{L}$ | The value variable of multi-head attention | / | $\mathbb{R}^{I_{n}\times D_{V}}$
$\left\\{{{\bm{W}}_{Q}^{l}}\right\\}_{l=1}^{L}$ | The query matrix of multi-head attention | / | $\mathbb{R}^{D\times D_{K}}$
$\left\\{{{\bm{W}}_{K}^{l}}\right\\}_{l=1}^{L}$ | The key matrix of multi-head attention | / | $\mathbb{R}^{D\times D_{K}}$
$\left\\{{{\bm{W}}_{v}^{l}}\right\\}_{l=1}^{L}$ | The value matrix of multi-head attention | / | $\mathbb{R}^{D\times D_{K}}$
$W_{multi}$ | The aggregation matrix of multi-head attention | / | $\mathbb{R}^{LD_{V}\times D}$
$\bm{A}^{\prime}$ | Obtained from $\bm{A}$ passes through fully connected layer FC1 | / | $\mathbb{R}^{I_{n}\times D}$
$\bm{A}^{\prime\prime}$ | Obtained from $\bm{A}^{\prime}$ passes through CNN layer and fully connected layer FC2 | / | $\mathbb{R}^{I_{n}\times D_{H}}$
$D_{RNN}$ | The dimension of RNN in the model | / | $\mathbb{N}^{+}$
$S^{\prime}$ | Obtained from , the output of ACT mechanism, passes through fully connected layer FC3 | / | $\mathbb{R}^{I_{n}\times D_{RNN}}$
$S^{\prime\prime}$ | Obtained from passes through RNN and fully connected layer FC4 | / | $\mathbb{R}^{I_{n}\times D}$
$\bm{H}$ | The hidden representation of sequence | / | $\mathbb{R}^{I_{n}\times D}$
$\bm{h}(t_{i})$ | The hidden representation of i-th event | / | $\mathbb{R}^{D}$
$\mathcal{H}_{t}$ | The previous history at time t | / | /
$b_{c}$ | The background intensity of event type c | / | $\mathbb{R}$
${\alpha_{c}}$ | Thecontinuous parameter of conditional intensity function of event type | / | $\mathbb{R}^{1\times D}$
${\bm{w}}_{c}^{T}$ | Historical weight parameter of conditional intensity function of event type | / | $\mathbb{R}^{1\times D}$
$\bm{W}_{time}$ | The prediction parameter of time-stamp | / | $\mathbb{R}^{1\times D}$
$\bm{W}_{type}$ | The prediction parameter of event type | / | $\mathbb{R}^{C\times D}$

For sequences of asynchronous events, we need to determine what model they
have. The symbols used in the paper are shown in Table 1, and in general, we
formulate it as following: assume that there are $N$ sequences in the dataset
of the asynchronous events, represented as $S_{e}=\\{s_{n}\\}_{n=1}^{N}$ , and
for each sequence, note that their lengths are not the same, for the $n$-th
sequence $s_{n}=\\{t_{i},c_{i}\\}_{i=1}^{I_{n}}$ , its length is $I_{n}$ ,
each sequence $s_{n}$ is composed with $I_{n}$ tuples, $t_{i}$ is the time-
stamp of $i$-th event, and $c_{i}\in C$ is the corresponding event type. In
the forward phase, for arbitrary sequence
$s_{n}=\\{t_{i},c_{i}\\}_{i=1}^{I_{n}}$ , the entire sequence could be
directly input to the universal transformer, supposing the model dimension is
$D$ , then all these events in the sequences (including their time-stamp) can
be represent by their corresponding $D$-dimensional vectors, then this
sequence can be described by the hidden representation of event sequence
${\bm{H}}\in\mathbb{R}^{D\times I_{N}}$ . The continuous conditional intensity
function can be calculated by ${\bm{H}}\in\mathbb{R}^{D\times I_{N}}$ and the
equation we proposed in section 3.2.

###  3.1 Universal Transformer Hawkes Process

First, we need to map the asynchronous event sequence into the temporal
encodings, which denote the occurring times of events, and event-type
encodings. According to point process theory, the time-stamps of events in the
event sequences are the coordinate on the temporal axis, in other words, time-
stamps are temporal position of events. Analogous to the text sequence, the
time-stamps in the event sequence are equivalent to the position of the words,
and the event type embedding vectors are equivalent to the semantic vectors of
the words. Thus, for the temporal encodings, we can leverage the position
encoding approaches in nature language processing to encode the time-stamps of
events, similar as [18,21], the position encoder for the occurring time-stamp
of event is shown as Eq.2:

$[{\bm{x}}(t_{i})]_{j}=\left\\{{\begin{array}[]{*{20}c}{\cos\left({t_{i}/10000^{\frac{{j-1}}{D}}}\right),{\rm{if}}\>j\>{\rm{is}}\>{\rm{odd}},}\\\
{\sin\left({t_{i}/10000^{\frac{j}{D}}}\right),{\rm{if}}\>j\>{\rm{is}}\>{\rm{even}}.}\\\
\end{array}}\right.$ (2)

Thus, for the time-stamp $t_{i}$ of event, its corresponding temporal encoding
is ${\bm{x}}(t_{i})\in\mathbb{R}^{D}$ , where $D$ is the model dimension of
universal transformer, and in order to get event-type encoding, we set
embedding matrix ${\bm{K}}\in\mathbb{R}^{D\times C}$, and each kind of event
is corresponding to an one-hot encoding ${\bm{c}}_{i}\in\mathbb{R}^{C}$ , then
we can get the event-type encoding ${\bm{Kc}}_{i}\in\mathbb{R}^{D}$ . We
define that $\bm{X}^{T}$ and $({\bm{KC}}_{n})^{T}$ are the corresponding
temporal encoding and event-type encoding of the sequence, where
${bm{X}}=\\{{bm{x}}(t_{1}),{bm{x}}(t_{2}),...,{bm{x}}(t_{I_{n}})\\}\in\mathbb{R}^{D\times
I_{n}}$ and ${bm{C}}_{n}=[c_{1},c_{2},...,c_{I_{n}}]\in\mathbb{R}^{C\times
I_{n}}$ .

In order to get the hidden representation of event sequence, we need to input
the above temporal and event-type encodings into the recurrent part in
universal transformer, which is shown as Fig. 1 and Fig. 2.

Figure 1: Schematic diagram of universal transformer with pure recurrence, the
model simply uses the hidden state as a recurrent variable  Figure 2:
Schematic diagram of universal transformer based on ACT algorithm, in the
model, ACT algorithm provides a measure of how well of learning of different
elements in the variable.

Different with traditional transformer, for universal transformer, the
temporal and event-type encodings will be alternatively and recursively
refined by attention and convolutional layers until a certain stop condition
is met, then the final hidden representation will be acquired. In universal
transformer with pure recurrence illustrated in Fig. 1, the hidden state will
iterate the predetermined number of times, and the final hidden state is set
as the hidden representation of event sequence. The corresponding
implementation process is shown as the Algorithm 1:

Algorithm 1 Universal Transformer with pure recurrence.

0: The maximum number of iterations: max_n, event-type encoding
$({\bm{KC}}_{n})^{T}$, temporal encoding (position encoding) $\bm{X}^{T}$ and
EncodingLayer in model.

0: Hidden representation of event sequence ${\bm{H}}\in\mathbb{R}^{D\times
I_{N}}$ (State)

Initialize state $n\leftarrow 0$, ${\bm{S}}\leftarrow({\bm{KC}}_{n})^{T}$ .

2: while $(n<\rm{max\\_n})$ do

${\bm{S}}\leftarrow{\bm{S}}+{{}}{\bm{X}}^{T}$

4: ${\bm{S}}\leftarrow
output\;of\;{\rm{the}}\;i{\rm{-th}}\;{\rm{encoding\;layer}}({\bm{S}})$

end while

6: return ${\bm{H}}\leftarrow{\bm{S}}$

In iterative process, some symbols, such as words and phonemes, in our
application scenario they are the temporal and event-type encodings, are
usually more ambiguous than other elements in the sequence, the more ambiguous
these symbols are, and the more refining processes are required. However, the
above algorithm simply refines equally among each element in the sequence, it
is easy to see that this mechanism is too rough and doesn’t meet the
requirement we mentioned.

Thus, similar as [22], we adopt the ACT mechanism [23] in our proposed
universal transformer Hawkes process, which is a mechanism can dynamically
adjust the number of refinedness for each symbol in the sequence. The ACT
mechanism updates the symbols in the sequence according to their state
parameter. Since ACT method is used in speech recognition and other scenarios
and cannot be directly applied to event sequence data, we are going to make
some certain modifications to the concrete implementation procedure of the ACT
mechanism. The modified ACT mechanism is illustrated in Algorithm 2, and the
information flow diagram of ACT mechanism in Universal transformer is shown in
Fig. 3.

Algorithm 2 Universal Transformer based on ACT mechanism.

0: The maximum number of iterations: max_n, event-type encoding
$({\bm{KC}}_{n})^{T}$, temporal encoding (position encoding) $\bm{X}^{T}$,
threshold $T_{h}$ and EncodingLayer in
model.Paremeter:${\bm{W}}_{p}\in\mathbb{R}^{D\times 1}$

0: Hidden representation of event sequence ${\bm{H}}\in\mathbb{R}^{D\times
I_{N}}$ (State)

Initialize state $n\leftarrow{0},R_{e}\leftarrow 0,h_{p}\leftarrow
0,P_{s}\leftarrow 0$ and ${\bm{S}}\leftarrow({\bm{KC}}_{n})^{T}$ .

2: while there is any element in $\left({(h_{p}>T_{h})\&(n<\max\\_n)}\right)$
is true, do

${\bm{S}}\leftarrow{\bm{S}}+{{}}{\bm{X}}^{T}$

4: $p\leftarrow Sigmoid(S{\bm{W}}_{p})$

$S_{r}\leftarrow\mathbb{I}(h_{p}<1.0)$

6: $N_{h}\leftarrow(h_{p}+(p*\mathbb{I}(S_{r}>T_{h}))*S_{r}$

$S_{r}\leftarrow(h_{p}+(p*\mathbb{I}(S_{r}\geqslant T_{h}))*S_{r}$

8: $h_{p}\leftarrow h_{p}+p*S_{r}$

$R_{e}\leftarrow R_{e}+N_{h}*(1-h_{p})$

10: $h_{p}\leftarrow h_{p}+N_{h}*R_{e}$

$n\leftarrow n+S_{r}+N_{h}$

12: $W\leftarrow p*S_{r}+N_{h}*R_{e}$

$S\leftarrow EncodingLayer(S)$

14: $P_{S}\leftarrow S*W+P_{S}*(1-W)$

end while

16: return ${\bm{H}}\leftarrow{{P_{S}}}$

Figure 3: Diagram of universal transformer based on ACT mechanism, the figure
shows the information flow and update process of different variables

We can see that universal transformer without ACT mechanism and RNN have
obvious similarities, in both models, there is only state calculated
recursively in the iteration, and universal transformer with ACT mechanism is
more similar to LSTM, because there are multiple variables to be updated in
the recurrence, and the updated weights are to some extent consistent with the
forgetting gate in LSTM. These facts show that universal transformer with ACT
mechanism will have a stronger learning ability than universal transformer
without ACT mechanism.

Figure 4: The diagram of encoding layer in universal transformer, the self-
attention mechanism of universal transformer is similar to the normal
transformer, and we modify position-wise-feed-forward part, add a
convolutional layer between the two fully connected layers to enhance the
local perception ability of the model.

In both algorithms, we all are going to feed the state in Algorithm 1 and 2
into the encoding layer, whose flow-chart is illustrated in Fig. 4, encoding
layer is composed with two parts, i.e., multi-head attention part and
position-wise-feed-forward part. In encoding layer, we also utilize the
commonly used techniques in transformer to relieve the difficulty in training,
such as the residual connection [29] and layer normalization [30].

In multi-head attention part, we also adopt dot-product attention, which can
be written as Eq.3:

${\bm{A}}=Softmax\left({\frac{{{\bm{QK}}^{T}}}{{\sqrt{D_{K}}}}}\right){\bm{V}}$
(3)

where

${\bm{Q}}={\bm{SW}}_{Q},{\bm{K}}={\bm{SW}}_{K},{\bm{V}}={\bm{SW}}_{V}$ (4)

state $S$ is the input to the encoding layer, $\bm{Q}$,$\bm{K}$,and $\bm{V}$
are respectively query, key and value matrices, which are obtained by
different linear transformations
${\bm{W}}_{Q},{\bm{W}}_{K}\in\mathbb{R}^{D\times D_{K}}$ ,
${\bm{W}}_{V}\in\mathbb{R}^{D\times D_{V}}$ of state $S$ . And in order to
improve the diversity and expressive ability of model, we make use of multi-
head self-attention to make the model perform better. If we utilize L heads
self-attention, then, we can get $L$ outputs
${\bm{A}}_{1},{\bm{A}}_{2},...,{\bm{A}}_{L}$ , which are computed by different
sets of matrices
$\left\\{{{\bm{W}}_{Q}^{l},{\bm{W}}_{K}^{l},{\bm{W}}_{V}^{l}}\right\\}_{l=1}^{L}$
. The formula for multi-head attention is shown as Eq.5:

${\bm{A}}=\left[{{\bm{A}}_{1},{\bm{A}}_{2},...,{\bm{A}}_{L}}\right]{\bm{W}}_{multi}$
(5)

In Eq.5, ${\bm{W}}_{multi}\in\mathbb{R}^{LD_{V}\times D}$ is the aggregation
matrix. It is worth noting that the events in the sequence occur in sequential
order, thus, it has the characteristic called as “future invisibility”, we
need to block the impact of future events on current time. We utilize the
masked self-attention mechanism similar as [18]. For example, we set
${\bm{E}}={\bm{QK}}^{T}$ , the masked self-attention mechanism will set the
element in the $i$-th row in $\bm{E}$ ranged in
${\bm{E}}(i,i+1),{\bm{E}}(i,i+2),..,{\bm{E}}(i,D)$ to negative infinite, which
will ensure that the self-attention weight of future events obtained through
softmax function is 0.

After getting the attention output ${\bm{A}}$ it will be fed into position-
wise-feed-forward part, which is shown in the right part of Fig. 4. To improve
model’s local perception of the sequence, we introduce the CNN module in
position-wise-feed-forward part, more specifically, fully connected layer FC1
transforms attention output ${\bm{A}}$ into a higher-dimensional linear matrix
${\bm{A^{\prime}}}\in\mathbb{R}^{I_{n}\times D_{H}}$ . Then it pass through
the convolutional layer, nonlinear layer, and pooling layer, and fully
connected layer FC2 will convert it to the original dimension
${\bm{A^{\prime\prime}}}\in\mathbb{R}^{I_{n}\times D}$.

After the Algorithm 1 or 2 completely stop, we will get the output $H$, and at
last, will be as the input of postprocessing part, which is constituted with
fully connected layer FC3, RNN layer and final fully connected layer FC4. This
structure is motivated by [21],[31], and in fact the additional RNN can make
model better fit the sequential data, in a nutshell, the FC3 transforms input
$S$ or $P_{S}$ to $S^{\prime}\in\mathbb{R}^{I_{n}\times D_{{\rm{RNN}}}}$ and
$S^{\prime}$ will be recurrently iterated by the RNN, note that the types of
RNN can be selected as LSTM or GRU, then the FC4 converts $S^{\prime}$ into
$S^{\prime\prime}\in\mathbb{R}^{I_{n}\times D}$ , at last, $S^{\prime\prime}$
will be assigned to the hidden representation
${\bm{H}}\in\mathbb{R}^{I_{n}\times D}$ of the sequence of events. The
effectiveness of postprocessing part in UTHP is decided by the characteristic
of the dataset, and we will discuss this issue in subsection 4.3.2 and
analyzed it in experiments.

###  3.2 Conditional Intensity Function

In general, point processes are supposed to be described by their
corresponding conditional intensity functions. However, after encoding
operation of universal transformer, we can only get the discrete conditional
intensity established by hidden representation of event sequence. Therefore,
we need to construct the continuous conditional intensity function on the
foundation of hidden representation of event sequence.

For every kind of event, we denote
$\lambda_{c}(t\left|{\mathcal{H}_{t}}\right.)$ as corresponding conditional
intensity function of UTHP, and here $\mathcal{H}_{t}=(t_{i},c_{i}):t_{i}<t$
is the history of past events at time t. Similar as [21], the form of
conditional intensity function is shown as Eq.6:

$\lambda_{c}(t\left|{\mathcal{H}_{t}}\right.)=f(b_{c}+\alpha_{c}\frac{{t-t_{i}}}{{t_{i}}}+{\bm{w}}_{c}^{T}{\bm{h}}(t_{i}))$
(6)

In Eq.6, $t\in[t_{i},t_{i+1})$, and $b_{c}$ is the background intensity,
indicates the possibility of an event occurring without considering historical
information. $\alpha_{c}\frac{{t-t_{i}}}{{t_{i}}}$ is continuous conditional
intensity function in the interval $[t_{i},t_{i+1})$ , where $\alpha_{c}$ is
predefined parameter and can be modified to trainable parameter in subsequent
improvements, and ${\bm{w}}_{c}^{T}{\bm{h}}(t_{i}))$ represents the history
impact to the conditional intensity function. $f(\cdot)$is the softplus
function, which is a smooth improvement of the ReLU nonlinear function.
Softness parameter of softplus function is $\beta$ , using this nonlinear
function makes the conditional intensity function smoother and make the new
model have a better expressive ability. And the overall conditional intensity
function for the whole sequence is given as follow:

$\lambda(t\left|{\mathcal{H}_{t}}\right.){\rm{=}}\sum\limits_{c=1}^{C}{\lambda_{c}(t\left|{\mathcal{H}_{t}}\right.)}$
(7)

###  3.3 Prediction

In point process theory, prediction is one of the most important tasks. For
instance, it can be used to predict when does the patient develop which
disease, and when and what events will happen in social networks. After
obtaining the conditional intensity function, in view of [32], we can predict
the possible types and occurrence times of future events according to
conditional intensity function and Eq.8:

$\begin{array}[]{l}p(\left.t\right|{\cal H}_{t})=\lambda(\left.t\right|{\cal
H}_{t})\exp t(-\int_{t_{i}}^{t}{\lambda(\left.s\right|{\cal H}_{t})}ds)\\\
\hat{t}_{i+1}=\int_{t_{i}}^{\infty}{t\cdot p(\left.t\right|{\cal H}_{t})dt}\\\
\hat{c}_{i+1}=\mathop{\arg\max}\limits_{c}\frac{{\lambda_{c}(\hat{t}_{i+1}|{\cal
H}_{i+1})}}{{\lambda(\hat{t}_{i+1}|{\cal H}_{i+1})}}\\\ \end{array}$ (8)

The existence of the integral of the conditional intensity function in Eq.8
makes the prediction become difficult, although we can use the Monte Carlo
sampling method [33] to calculate it, this is still very inefficient. Due to
the powerful fitting ability of neural network, we make use of neural work to
predict the future event type and time-stamps, which can be shown as Eq.9:

$\begin{array}[]{l}\hat{t}_{i+1}={\bm{W}}_{time}{\bm{h}}(t_{i})\\\
\widehat{\bf{p}}_{i+1}=Softmax({\bm{W}}_{type}{\bm{h}}(t_{i}))\\\
\hat{c}_{i+1}=\mathop{\arg\max}\limits_{c}{\bm{\hat{p}}}_{i+1}(c)\\\
\end{array}$ (9)

According to [32], given the sequence $s_{n}=\\{t_{i},c_{i}\\}_{i=1}^{I_{n}}$
, and conditional intensity function, we can get its log-likelihood
expression:

$L(s_{n})=\sum\limits_{i=1}^{I_{n}}{\log}\,\lambda(\left.{t_{i}}\right|\mathcal{H}_{i})-\int_{t_{1}}^{t_{I_{n}}}{\lambda(\left.t\right|\mathcal{H}_{t})}dt$
(10)

Assuming there are $N$ sequences, then the model parameters can be solved by
maximum log-likelihood principle:

$\max\sum\nolimits_{n=1}^{N}{L(s_{n})}$

However, there is difficulty in solving
$\Lambda=\int_{t_{1}}^{t_{I_{n}}}{\lambda\left({\left.t\right|\mathcal{H}_{t}}\right)}dt$
because we use universal transformer to obtain the conditional intensity
function, it is hard to obtain the closed-form of $\Lambda$ , fortunately, we
have two alternative methods to solve it. The first method is the Monte Carlo
sampling integration method [33], which is shown in Eq.11:

$\hat{\Lambda}_{MC}=\sum\limits_{i=2}^{L}{\left({t_{i}-t_{i-1}}\right)}(\frac{1}{M}\sum\limits_{m=1}^{M}{\lambda(u_{m})})$
(11)

where $u_{m}\sim U(t_{i-1},t_{i})$ , and $u_{m}$ is sampled from the uniform
distribution in support set $[Ut_{i-1},t_{i}]$. $\hat{\Lambda}_{MC}$
calculated by this method is an unbiased estimate of $\Lambda$. The second
method is numerical analysis method [34], for instance, based on the
trapezoidal rule, we can get the estimate as shown in Eq.12:

$\hat{\Lambda}_{NI}=\sum\limits_{i=2}^{I_{n}}{\frac{{t_{i}-t_{i-1}}}{2}}(\lambda(\left.{t_{i}}\right|\mathcal{H}_{i})+\lambda(\left.{t_{i-1}}\right|\mathcal{H}_{i-1}))$
(12)

In spite of the bigger estimate error than the Monte Carlo method,
$\hat{\Lambda}_{NI}$ is still a valid approximation of $\Lambda$ . After
dealing with the solution of $\Lambda$ , we are going to define the prediction
loss of time-stamp and event type, for a sequence of length $I_{n}$ , we will
make $I_{n}-1$ predictions of time-stamp and event type, in other words, we
will not predict the first event. Therefore, the predict loss of next time-
stamp and event prediction for sequence $s_{n}$ is shown as Eq.13 and 14:

$L_{time}(s_{n})=\sum\nolimits_{i=2}^{I_{n}}{(t_{i}-\hat{t}_{i})^{2}}$ (13)

$L_{type}(s_{n})=\sum\nolimits_{i=2}^{I_{n}}{-{\mathbf{c}}}_{i}^{T}\log(\widehat{\mathbf{p}}_{i})$
(14)

where $\bm{c}_{i}$ is one-hot encoding with the ground-truth event type . In
summary, given the dataset including the sequences$\\{s_{n}\\}_{n=1}^{N}$ , we
need to solve:

$\min\sum\limits_{n=1}^{N}{-L(s_{n})}+\alpha_{type}L_{type}(s_{n})+\alpha_{time}L_{time}(s_{n})$
(15)

where $\alpha_{type}$ and $\alpha_{time}$ are hyper-parameters, which will
help keep training stable. This objective function can be efficiently solved
by stochastic gradient optimization algorithm, such as adaptive moment
estimation (ADAM) [35], and we use the default parameters of ADAM: learning
rate=0.0001, betas are 0.9 and 0.999, eps = $10^{-8}$, weight_decay = 0, for
all these experiments, when we solves log-likelihood of models, we set
$\alpha_{type}=0$ and $\alpha_{time}=0$ , and when performing prediction
tasks, we fix $\alpha_{type}=1$ and $\alpha_{time}=0.01$ .

##  4 Experiments

We compare our model with three baselines on six events sequence datasets, we
evaluate these models by per-event-loglikelihood (in nats), root mean square
error (RMSE) and event prediction accuracy on held-out test sets, we first
introduce the details of the data set and baselines, and then list our
experimental results.

###  4.1 Datasets

In this subsection, we utilize benchmark six datasets of event sequence to
conduct experiments, Table 2 introduces the characteristics of each dataset.

Table 2: Characteristics of datasets. Each row corresponds to the description of this data set, including the total type number of events, the smallest, average, and largest sequence length. Dataset | C | Sequence Length
---|---|---
Min | Aver. | Max
Synthetic | 5 | 20 | 60 | 100
Retweets | 3 | 50 | 109 | 264
MemeTrack | 5000 | 1 | 3 | 31
MIMIC-II | 75 | 2 | 4 | 33
StackOverflow | 22 | 41 | 72 | 736
Financial | 2 | 829 | 2074 | 3319

Synthetic [12]: Mei generates a set of synthetic sequences based on Hawkes
process with random sampling the parameters and thinning algorithm.

Retweets [6]: This dataset contains lots of sequences of tweets, every
sequence corresponds to an origin tweet (for instance, the original content
some user post) and its following retweets. The time and label of user of each
retweet is recorded in the sequence, and users are divided into three
categories based on the number of their followers: “small”, “medium”, and
“large”.

MemeTrack [36]: This dataset contains 42000 different meme posting in 5000
websites in the duration of 10 months, each sequence represents this meme from
its birth to its final disappearance, and each event in the sequence
corresponds to a time stamp and a website id.

StackOverflow [36]: StackOverflow is a famous programming question-answering
website. StackOverflow rewards users with badges to encourage them to
participate in community activities, meanwhile, the same badge can be given to
the same user, this dataset includes the lots of users reward histories during
two years, which are treated as sequences, and each event in sequences
indicates the acquisition of badge.

Electrical Medical Records [37]: MIMIC-II dataset contains patients’ visit
record to a hospital’s ICU in the course of seven years. Each patient’s record
is treated as a sequence, and each event contains the corresponding time-stamp
and diagnosis.

Financial Transactions [11]: This financial dataset contains large number of
short-term transaction records of a stock in one day. The operation of each
transaction is recorded as the event, and this dataset contains several long
sequences, which only have two kinds of events: “buy” and “sell”, and the unit
of the time-stamp is millisecond.

###  4.2 Baselines

RMTPP [11]: Du et al. come up with a recurrent network, which can model the
event types and time-stamp in the sequence by embedding history to the vector.

NHP [12]: Mei and Eisner present the neural Hawkes process based on the
continuous-time LSTM, which has decay property of history events impact.

THP [21]: On the foundation of existing achievement of transformer, Zuo et al.
propose the transformer Hawkes process, which achieves state-of-the-art
performance. We get the following experimental results based on the model and
the hyper-parameter they provide.

For NHP model, we also use ADAM optimizer with same hyper-parameter i.e., we
set learning rate=0.0001, betas are 0.9 and 0.999, eps =$10^{-8}$ ,
weight_decay = 0, and the number of hidden nodes of the continuous-time LSTM
is 64, for Synthetic, Retweets and MemeTrack dataset, we set batch size to 16.
The hyper-parameter configurations of THP and UTHP are given in section 4.5.

###  4.3 Experimental results and comparison

In this part, we are going to compare the performance of UTHP and baselines on
datasets of multiple event sequences. First, we utilize loglike (per-event-
loglikelihood) as the measurement of models’ performance on Synthetic,
Retweets and MemeTrack datasets, which is shown in Table 3:

Table 3: The loglike on three dataset, from left to right, each column is a different model, and each row is a different dataset Datasets | RMTPP | NHP | THP | UTHP
---|---|---|---|---
Synthetic | \ | -1.33 | 0.834 | 0.796
Retweets | -5.99 | -5.06 | -4.69 | -4.75
MemeTrack | -6.04 | -6.23 | 0.68 | 0.31

From Table 3, we can see that transformer models achieve obvious improvement
than previous one, and although our UTHP model does not reach the state-of-
the-art level, it still achieves second best. After this, we compare the
performance of different models on complex datasets, including StackOverflow,
MIMIC-II and Financial datasets. Because of the importance of event and time
prediction of point process, while we compare the performance of models on
these datasets, the multiple metric criteria will be adopted, involving per-
event-log-likelihood, the predict accuracy for event and RMSE of time-stamp
prediction. For transformer based model, we utilize Eq. 9 to make predictions,
and for RNN based model, Eq. 10 is used. Table 4, Table 5 and Table 6
summarize the experimental results on these datasets.

Table 4: Different models’ performance on StackOverflow dataset, each row is the different metric criteria, including accuracy, RMSE and loglike (per-event-loglikelihood) StackOverflow | RMTPP | NHP | THP | UTHP
---|---|---|---|---
Accuracy | 45.9 | 46.3 | 46.8 | 46.9
RMSE(d) | 9.78 | 9.83 | 4.99 | 4.42
Loglike | -2.60 | -2.55 | -0.56 | -0.55
Table 5: Different models’ performance on MIMIC-II dataset. MIMIC-II | RMTPP | NHP | THP | UTHP
---|---|---|---|---
Accuracy | 81.2 | 83.2 | 83.2 | 84.4
RMSE(d) | 6.12 | 6.13 | 0.86 | 0.85
Loglike | -1.35 | -1.38 | -0.14 | -0.18
Table 6: Different models’ performance on Financial dataset. Financial | RMTPP | NHP | THP | UTHP
---|---|---|---|---
Accuracy | 61.95 | 62.20 | 62.23 | 62.52
RMSE(s) | 1.56 | 1.56 | 0.02575 | 0.02574
Loglike | -3.89 | -3.60 | -1.39 | -1.16

From these results, we can see that our proposed UTHP model achieves obvious
improvement than the baselines among the different scenarios, different data
sets have markedly different characteristics, for instance, the number of
classes of MIMIC-II dataset is 75, and the average length is only 4, while the
number of classes of Financial dataset is only 2, and the average length is
2074. In all of these datasets, our model improves relatively significantly.
This suggests that UTHP can better model complex asynchronous event sequences
and learn long-term and short-term event dependencies than existing baseline
models.

(a) MIMIC-II (b) StackOverflow (c) Fianancial

Figure 5: Prediction accuracies of RMTPP, NHP, THP and UTHP. Based on the five
times train-dev-test partition, five experiments are performed on each
dataset, the mean and standard deviations of different models are depicted.

(a) MIMIC-II (b) StackOverflow (c) Fianancial (d) Zoom in of RMSE of THP and
UTHP on the datasets

Figure 6: The mean and standard deviation for different models obtained from
the five experiments in term of RMSE metric for RMTPP, NHP, THP and UTHP.

Fig. 5 and Fig.6 visualizes the accuracies of different models and RMSE of
baselines and UHP, due to the dispersion of the data, the error bars are wide.
We can find out that the results of UTHP better than the other baselines.

###  4.4 Ablation study

In this subsection, we are going to discuss the impact to the performance of
three significance part in models, the postprocessing part, CNN module in the
position-wise-feed-forward part and ACT mechanism, we demonstrate
experimentally how the presence of these three parts affects the performance
of the model.

####  4.4.1 Ablation study of the postprocessing part

As we mentioned in subsection 3.1, the introduction of the additional
postprocessing part will have an impact on the transformer-based models, and
the impact on the model varies with the dataset. In this subsection, we will
demonstrate the impact of the introduction of RNN.

Previous experimental results prove that RNN (including LSTM and GRU) can
successfully capture the short-term dependencies in the sequence, but not for
long-term dependencies. Thus, we can assume that if we introduce the RNN in
the additional postprocessing part to the transformer-based model, it will
have better performance to capture the dependencies between the events on the
dataset with short-term characteristic, and have relatively poor performance
of dependencies fitting on the dataset with long-term characteristic.

To validate whether the facts are consistent with our assumptions, validation
experiments are carried out on the dataset with long-term property, i.e.,
MIMIC-II, and dataset with short-term property, i.e., Financial dataset, and
the event prediction accuracies are used to measure how well the model fits
the dependencies between events. The experimental results are shown as Table 7
and Fig. 7:

Table 7: Consider that whether there is an additional RNN layer in the postprocessing part, UTHP and THP’s event prediction accuracy rates on MIMIC-II dataset and financial dataset. | MIMIC-II | Financial
---|---|---
| With RNN | Without RNN | With RNN | Without RNN
| Accuracy of THP
---
81.44 | 83.18 | 62.23 | 60.22
| Accuracy of UTHP
---
83.34 | 84.43 | 62.52 | 62.04

From the experimental results, we can discover that for transformer-based
model, addition RNN layer in the postprocessing part has a negative effect on
the experimental results of MIMIC-II dataset, and has a positive impact on the
financial dataset. These facts verify that our hypothesis, RNN do enhance the
fitting ability to events short-term dependencies, and weaken the fitting
ability to the events long-term dependencies.

(a) MIMIC-II (b) Financial

Figure 7: Visualization of ablation experimental results with and without an
additional RNN layer in the postprocessing part.

In addition, UTHP model demonstrates its robustness without RNN on the
financial dataset. The reason is that our model already has a recursive
structure, which helps it better model short-term data. This also reflects the
value of UTHP model we proposed.

####  4.4.2 Ablation study of CNN module in the position-wise-feed-forward
part

In general, CNN module will enhance the local perception ability of the model,
which is the reason why we add CNN module into the position-wise-feed-forward
part. In order to judge whether the introduction of the CNN module is
effective, we conduct the comparison experiments, compare the UTHP based on
ACT algorithm with CNN and without CNN module in position-wise-feed-forward
part on StackOverflow, MIMIC-II and Financial dataset. The experiment results
are shown as Table 8 and Fig. 8.

Table 8: Model performance comparisons of UTHP based on ACT algorithm with CNN and without CNN module in position-wise-feed-forward part | UTHP | With CNN | Without CNN
---|---|---|---
Accuracy | StackOverflow | 46.87 | 46.76
MIMIC-II | 84.43 | 83.83
Financial | 62.52 | 62.52
Figure 8: Accuracy comparison of UTHP based on ACT algorithm with CNN and
without CNN module in position-wise-feed-forward part.

####  4.4.3 Ablation study of ACT mechanism

From the above experimental results, we can find out that the introduction of
CNN module in position-wise-feed-forward part obviously raises the event
prediction accuracy. This fact indicates that CNN module enhance the ability
to model connections between events of UTHP through its local perception
characteristics, which indicates that the introduction of CNN module in
position-wise-feed-forward part is effective.

In subsection 3.1, we introduce the reasons for adding the ACT mechanism to
the model, ACT mechanism will dynamically regulate the calculation of input in
the transformer, make the model use more computing resources to where it is
needed more. The concept of this effect is similar to and complementary to the
self-attention mechanism, and theoretically makes the model perform better.
And in order to investigate the effect of the ACT mechanism on UTHP, we
conducted a series of comparative experiments, we compare pure UTHP under
different recursive iteration times (the corresponding algorithm is Algorithm
1) and UTHP with ACT mechanism (the corresponding algorithm is Algorithm 2 and
maximum number of iterations max_n=2). These results are summarized in Table
9:

Table 9: Model performance comparisons of UTHP based on ACT mechanism and UTHP with pure recurrence | Times of iteration | ACT | 1 | 2 | 3 | 4
---|---|---|---|---|---|---
StackOverflow | Accuracy | 46.87 | 46.67 | 46.70 | 46.70 | 46.72
RMSE | 4.42 | 4.52 | 4.47 | 4.47 | 4.47
Loglike | -0.55 | -0.60 | -0.58 | -0.58 | -0.57
MIMIC-II | Accuracy | 84.43 | 82.34 | 82.63 | 83.05 | 81.95
RMSE | 0.85 | 0.90 | 0.87 | 0.89 | 0.85
Loglike | -0.18 | -0.21 | -0.19 | -0.18 | -0.17
Financial | Accuracy | 62.52 | 62.52 | 62.45 | 62.43 | 62.29
RMSE | 0.02574 | 0.02574 | 0.02572 | 0.02571 | 0.02572
Loglike | -1.16 | -1.24 | -0.76 | -0.80 | -0.77

For the pure recurrence of UTHP, we considered four different iteration times
as the first row in Table 9, and when the number of iterations is once, it
indicates that hidden representation of sequences will directly output the
model, rather than recurrently iterate in the model, from the results we can
find out that UTHP with ACT mechanism performs than pure recurrence UTHP
overall. Especially in terms of the accuracies of event prediction, UTHP with
ACT mechanism are all better than UTHP with pure recurrence.

For a more intuitive comparison, we visualize these experimental results and
error bars, as shown in Fig. 9, Fig. 10, and Fig. 11.

(a) Accuracy (b) RMSE (c) Loglike

Figure 9: Model performance curves of UTHP with ACT mechanism and pure UTHP on
StackOverflow dataset.

(a) Accuracy (b) RMSE (c) Loglike

Figure 10: Model performance curves of UTHP with ACT mechanism and pure UTHP
on MIMIC-II dataset.

(a) Accuracy (b) RMSE (c) Loglike

Figure 11: Model performance curves of UTHP with ACT mechanism and pure UTHP
on Financial dataset.

From these figures, we can see that compared to the error bar, UTHP with ACT
mechanism has a clear gap with pure UTHP when it performs well, such as the
accuracy, RMSE and Loglike curves on StackOverflow dataset, and the accuracy
curves on MIMIC-II dataset. And comparing with the standard deviation, when
pure UTHP results are better than UTHP with mechanism, the difference is not
so obvious. This further verifies that the introduction of the ACT mechanism
will improve the overall performance of UTHP.

In addition, with the increasing of iteration times, the Loglike on these
datasets of pure UTHP are also increasing. The fact represents that the
increment of iteration items enhance the modeling ability of UTHP.

###  4.5 Fewer model parameters of UTHP

Because of the recurrence structure of UTHP, there is only one encoding layer
in our model, while THP utilize the stacking of multiple encoding layer, in
other words, UTHP model share the parameters in the encoding layer. By this
way, we find that our proposed architectures improve model performance, and
also reduce model parameters.

For instance, in subsection 4.3, on each dataset, UTHP and THP adopt the
similar hyper-parameters, which is shown as Table 10 and Table 11, the only
difference is that UTHP has only one encoding layer, and there is a CNN module
in the position-wise-feed-forward part of encoding layer.

Table 10: Hyperparameter configurations of THP. Dataset | Synthetic | Retweets | MemeTrack | Financial | MIMIC-II |  | StackOverflow
---
$D$ | 64 | 64 | 64 | 128 | 64 | 512
$D_{H}$ | 256 | 256 | 256 | 2048 | 256 | 1024
$D_{\rm{RNN}}$ | 128 | 128 | 128 | 128 | 0 | 128
$D_{Q}=D_{V}$ | 16 | 16 | 16 | 64 | 16 | 512
Heads of attention | 3 | 3 | 3 | 6 | 3 | 4
Layers of transformer | 2 | 2 | 2 | 2 | 2 | 2
Dropout | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
Table 11: Hyperparameter configurations of UTHP. Dataset | Synthetic | Retweets | MemeTrack | Financial | MIMIC-II |  | StackOverflow
---
$D$ | 64 | 64 | 64 | 128 | 64 | 512
$D_{H}$ | 256 | 256 | 256 | 2048 | 256 | 1024
$D_{\rm{RNN}}$ | 128 | 128 | 128 | 128 | 0 | 128
$D_{Q}=D_{V}$ | 16 | 16 | 16 | 64 | 16 | 512
Heads of attention | 3 | 3 | 3 | 6 | 3 | 4
Max_n | 2 | 2 | 2 | 2 | 2 | 2
Dropout | 0.1 | 0.1 | 0.1 | 0.1 | 0.1 | 0.1
Size of convolution kernel | 3 | 3 | 3 | 3 | 3 | 3
Stride of convolution kernel | 2 | 2 | 2 | 2 | 2 | 2
Padding | 0 | 0 | 0 | 0 | 0 | 0
Stride of the pooling module | 2 | 2 | 2 | 2 | 2 | 2
Size of the pooling module | 2 | 2 | 2 | 2 | 2 | 2
Table 12: The numbers of parameters of the THP and UTHP on different datasets. Number of parameters | THP | UTHP
---|---|---
Synthetic | 245765 | 129494
Retweet | 137795 | 46356
MemeTrack | 1102216 | 1010777
MIMIC-II | 151691 | 60252
StackOverflow | 21022742 | 5281319
Financial | 4343298 | 724627

As we can see from the Table 12, although we introduce a CNN module in
position-wise-feed-forward part which increases the additional parameters in
UTHP, it still has less parameter than THP. The parameter number of the THP’s
parameter is approximately 6 to 1.1 times of the UTHP’s one. The fewer number
of event types in the dataset, the fewer parameters for UTHP compared to THP.
Combined with the experimental results in subsection 4.3, our model uses fewer
parameters to achieve better performance, which reflects the improvement of
our model.

##  5 Conclusions and future works

In this paper, we come up with UTHP, a new neural point process model to
analyze the asynchronous event sequence. UTHP combines the self-attention
mechanism in transformer and the recurrence mechanism in RNN, this operation
allows our model to organically integrate the advantages of RNN and
transformer, moreover, in order to make UTHP to adaptively determine when to
stop refining processes of hidden variables, we introduce the ACT mechanism to
UTHP. Experimental results verify that our model performs better than the
baselines overall with fewer model parameters, and we also discuss the impact
of the existence of the additional RNN layer and the ACT mechanism in the
ablation study.

##  6 Acknowledgment

Thanks for Hong-yuan Mei and Si-miao Zuo for their generous help in my
research state, their help greatly improved our research.

## References

  * [1] Y. Ogata, Space-time point-process models for earthquake occurrences, Annals of the Institute of Statistical Mathematics 50 (2) (1998) 379–402.
  * [2] E. Bacry, I. Mastromatteo, J.-F. Muzy, Hawkes processes in finance, Market Microstructure and Liquidity 1 (01) (2015) 1550005.
  * [3] L. Wang, W. Zhang, X. He, H. Zha, Supervised reinforcement learning with recurrent neural network for dynamic treatment recommendation, in: Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018, pp. 2447–2456.
  * [4] L.-n. Zhang, J.-w. Liu, X. Zuo, Survival analysis of failures based on hawkes process with weibull base intensity, Engineering Applications of Artificial Intelligence 93 (2020) 103709.
  * [5] K. Zhou, H. Zha, L. Song, Learning social infectivity in sparse low-rank networks using multi-dimensional hawkes processes, in: Artificial Intelligence and Statistics, 2013, pp. 641–649.
  * [6] Q. Zhao, M. A. Erdogdu, H. Y. He, A. Rajaraman, J. Leskovec, Seismic: A self-exciting point process model for predicting tweet popularity, in: Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, 2015, pp. 1513–1522.
  * [7] A. G. Hawkes, Spectra of some self-exciting and mutually exciting point processes, Biometrika 58 (1) (1971) 83–90.
  * [8] H. Xu, M. Farajtabar, H. Zha, Learning granger causality for hawkes processes, in: International Conference on Machine Learning, 2016, pp. 1717–1726.
  * [9] N. R. Hansen, P. Reynaud-Bouret, V. Rivoirard, et al., Lasso and probabilistic inequalities for multivariate point processes, Bernoulli 21 (1) (2015) 83–143.
  * [10] P. Reynaud-Bouret, S. Schbath, et al., Adaptive estimation for hawkes processes; application to genome analysis, The Annals of Statistics 38 (5) (2010) 2781–2822.
  * [11] N. Du, H. Dai, R. Trivedi, U. Upadhyay, M. Gomez-Rodriguez, L. Song, Recurrent marked temporal point processes: Embedding event history to vector, in: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2016, pp. 1555–1564.
  * [12] H. Mei, J. M. Eisner, The neural hawkes process: A neurally self-modulating multivariate point process, in: Advances in Neural Information Processing Systems, 2017, pp. 6754–6764.
  * [13] S. Xiao, J. Yan, X. Yang, H. Zha, S. M. Chu, Modeling the intensity function of point process via recurrent neural networks, in: Thirty-First AAAI Conference on Artificial Intelligence, 2017.
  * [14] Y. Bengio, P. Simard, P. Frasconi, Learning long-term dependencies with gradient descent is difficult, IEEE transactions on neural networks 5 (2) (1994) 157–166.
  * [15] R. Pascanu, T. Mikolov, Y. Bengio, On the difficulty of training recurrent neural networks, in: International conference on machine learning, 2013, pp. 1310–1318.
  * [16] A. Raganato, J. Tiedemann, et al., An analysis of encoder representations in transformer-based machine translation, in: Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, The Association for Computational Linguistics, 2018.
  * [17] L. Dong, S. Xu, B. Xu, Speech-transformer: a no-recurrence sequence-to-sequence model for speech recognition, in: 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), IEEE, 2018, pp. 5884–5888.
  * [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in neural information processing systems, 2017, pp. 5998–6008.
  * [19] D. Bahdanau, K. Cho, Y. Bengio, Neural machine translation by jointly learning to align and translate, in: Y. Bengio, Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
URL http://arxiv.org/abs/1409.0473

  * [20] Q. Zhang, A. Lipani, O. Kirnap, E. Yilmaz, Self-attentive hawkes process, ICML, 2020\.
  * [21] S. Zuo, H. Jiang, Z. Li, T. Zhao, H. Zha, Transformer hawkes process, in: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, Vol. 119 of Proceedings of Machine Learning Research, PMLR, 2020, pp. 11692–11702.
URL http://proceedings.mlr.press/v119/zuo20a.html

  * [22] M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, L. Kaiser, Universal transformers, in: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019.
URL https://openreview.net/forum?id=HyzdRiR9Y7

  * [23] A. Graves, Adaptive computation time for recurrent neural networks, arXiv preprint arXiv:1603.08983 (2016).
  * [24] R. Kobayashi, R. Lambiotte, Tideh: Time-dependent hawkes process for predicting retweet dynamics, arXiv preprint arXiv:1603.09449 (2016).
  * [25] Y. Yang, J. Etesami, N. He, N. Kiyavash, Online learning for multivariate hawkes processes, Advances in Neural Information Processing Systems 30 (2017) 4937–4946.
  * [26] A. G. Hawkes, Hawkes processes and their applications to finance: a review, Quantitative Finance 18 (2) (2018) 193–198.
  * [27] G. Mohler, J. Carter, R. Raje, Improving social harm indices with a modulated hawkes process, International Journal of Forecasting 34 (3) (2018) 431–439.
  * [28] K. Chowdhary, Natural language processing, in: Fundamentals of Artificial Intelligence, Springer, 2020, pp. 603–649.
  * [29] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [30] J. L. Ba, J. R. Kiros, G. E. Hinton, Layer normalization, arXiv preprint arXiv:1607.06450 (2016).
  * [31] C. Wang, M. Li, A. J. Smola, Language models with transformers, arXiv preprint arXiv:1904.09408 (2019).
  * [32] D. J. Daley, D. Vere-Jones, An introduction to the theory of point processes: volume II: general theory and structure, Springer Science & Business Media, 2007\.
  * [33] C. Robert, G. Casella, Monte Carlo statistical methods, Springer Science & Business Media, 2013.
  * [34] J. Stoer, R. Bulirsch, Introduction to numerical analysis, Vol. 12, Springer Science & Business Media, 2013.
  * [35] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6980 (2014).
  * [36] J. Leskovec, A. Krevl, Snap datasets: Stanford large network dataset collection (2014).
  * [37] A. E. Johnson, T. J. Pollard, L. Shen, H. L. Li-Wei, M. Feng, M. Ghassemi, B. Moody, P. Szolovits, L. A. Celi, R. G. Mark, Mimic-iii, a freely accessible critical care database, Scientific data 3 (1) (2016) 1–9.


* [25] J. L. Marin and S. Aubry, Breathers in nonlinear lattices: numerical calculation from the anticontinous limit, Nonlinearity, 9 (1996).
  * [26] S. H. Mousavi, A. B. Khanikaev, and Z. Wang, Topologically protected elastic waves in phononic metamaterials, Nature Communications, 6 (2015).
  * [27] L. Nirenberg, Topics in Nonlinear Functional Analysis, AMS, 1974.
  * [28] T. Ozawa, H. M. Price, A. Amo, N. Goldman, M. Hafezi, L. Lu, M. C. Rechtsman, D. Schuster, J. Simon, O. Zilberberg, and I. Carusotto, Topological photonics, Reviews of Modern Physics, 91 (2019).
  * [29] R. K. Pal, J. Vila, M. Leamy, and M. Ruzzene, Amplitude-dependent topological edge states in nonlinear phononic lattices, Physical Review E, 97 (2018), p. 032209, https://doi.org/https://doi.org/10.1103/PhysRevE.97.032209.
  * [30] N. Pernet, P. St-Jean, D. Solnyshkov, G. Malpuech, N. C. Zambon, Q. Fontaine, B. Real, O. Jamadi, A. Lemaitre, M. Morassi, L. L. Gratiet, T. Baptiste, A. Harouri, I. Sagnes, A. Amo, S. Ravets, and J. Bloch, Gap solitons in a one-dimensional driven-dissipative topological lattice, Nature Physics, 18 (2022), p. 678–684, https://doi.org/https://doi.org/10.1038/s41567-022-01599-8.
  * [31] J. Shapiro and M. I. Weinstein, Is the continuum SSH model topological?, https://arxiv.org/abs/2107.09146, (2022).
  * [32] S. Q. Shen, Topolgical insulators: Dirac equation in condensed matter (2nd edition), Springer, 2017.
  * [33] D. Smirnova, D. Leykam, Y. Chong, and Y. Kivshar, Nonlinear topological photonics, Applied Physics Reviews, 7 (2020), p. 021306, https://doi.org/https://doi.org/10.1063/1.5142397.
  * [34] D. A. Smirnova, L. A. Smirnov, D. Leykam, and Y. S. Kivshar, Topological edge states and gap solitons in the nonlinear dirac model., Lasers and Photonics Reviews, 13 (2019), p. 1900223.
  * [35] A. Soffer and M. I. Weinstein, Resonances, radiation damping, and instability of hamiltonian nonlinear waves, Inventiones Mathematicae, 136 (1999), pp. 9–74.
  * [36] A. Soffer and M. I. Weinstein, Selection of the ground state for nonlinear schroedinger equations, Reviews in Mathematical Physics, 16 (2004), pp. 977–1071.
  * [37] A. Soffer and M. I. Weinstein, Theory of nonlinear dispersive waves and selection of the ground state, Phys. Rev. Lett., 95 (2005), p. 213905.
  * [38] D. D. Solnyshkov, O. Bleu, B. Teklu, and G. Malpuech, Chirality of topological gap solitons in bosonic dimer chains, Physical Review Letters, 118 (2017), p. 023901, https://doi.org/https://doi.org/10.1103/PhysRevLett.118.023901.
  * [39] W. P. Su, J. R. Schrieffer, and A. J. Heeger, Solitons in polyacetylene, Physical Review Letters, 42 (1979), p. 1698, https://doi.org/https://doi.org/10.1103/PhysRevLett.42.1698.
  * [40] C. Sulem and P. Sulem, The Nonlinear Schrödinger Equation Self-Focusing and Wave Collapse, Springer, 1999.
  * [41] P. Wang, L. Lu, and K. Bertoldi, Topological phononic crystals with one-way elastic edge waves, Physical Review Letters, 115 (2015).
  * [42] M. I. Weinstein, Excitation thresholds for nonlinear localized modes on lattices, Nonlinearity, 12 (1999).
  * [43] M. I. Weinstein, Localized states and their dynamics in the nonlinear Schroedinger / Gross-Pitaeveskii equation: Analysis and Applications, Frontiers in Applied Dynamics: Reviews and Tutorials, 3 (2015).