# Self-Supervised Learning in Event Sequences: A Comparative Study and Hybrid
Approach of Generative Modeling and Contrastive Learning

Viktor Moskvoretskii1,3,∗ Dmitry Osin1,∗ Egor Shvetsov1 Igor Udovichenko1
Maxim Zhelnin1 Andrey Dukhovny2 Anna Zhimerikina2 Albert Efimov2 Evgeny
Burnaev1
1Skolkovo Institute of Science and Technology
2Sberbank
3HSE University
{V.Moskvoretskii, d.osin, e.shvetsov, i.udovichenko, M.Zhelnin,
<EMAIL_ADDRESS>{Zhimerikina.A.Y, AADukhovny<EMAIL_ADDRESS>
###### Abstract

This study investigates self-supervised learning techniques to obtain
representations of Event Sequences (EvS). It is a key modality in various
applications, including but not limited to banking, e-commerce, and
healthcare.

We perform a comprehensive study of generative and contrastive approaches in
self-supervised learning, applying them both independently. We find that there
is no single supreme method. Consequently, we explore the potential benefits
of combining these approaches. To achieve this goal, we introduce a novel
method that aligns generative and contrastive embeddings as distinct
modalities, drawing inspiration from contemporary multimodal research.

Generative and contrastive approaches are often treated as mutually exclusive,
leaving a gap for their combined exploration. Our results demonstrate that
this aligned model performs at least on par with, and mostly surpasses,
existing methods and is more universal across a variety of tasks. Furthermore,
we demonstrate that self-supervised methods consistently outperform the
supervised approach on our datasets.

**footnotetext: These authors contributed equally to this work

##  1 Introduction

Motivation. Event Sequences (EvS) are widely used in various real-world
applications, including medicine Waring et al. (2020), biology Valeri et al.
(2023), social medial analysis Liguori et al. (2023), fault diagnosis Shao et
al. (2019), churn prediction Jain et al. (2021); Sudharsan and Ganesh (2022),
customer segmentation Carnein and Trautmann (2019), fraud detection Xie et al.
(2022) and more Zhuzhel et al. (2023); Fursov et al. (2021). As a result,
there is a growing need to effectively model such data. Most self-supervised
methods are focused on images, text, speech, or time-series. In Computer
Vision (CV), contrastive learning methods have achieved state-of-the-art
results as a pre-training strategy before supervised fine-tuning He et al.
(2021); Chen et al. (2020); Caron et al. (2021). In Natural Language
Processing (NLP), most modern models rely on generative pre-training Radford
and Narasimhan (2018); Devlin et al. (2019). However, these approaches are
largely unexplored for EvS.

The aim of our work is (1) to study generative and contrastive self-supervised
approaches for pre-training and representation learning in the domain of EvS,
(2) and to determine if they can complement each other. We study two ways to
combine these approaches. For the first, we use a combined loss function. For
the second, drawing inspiration from contemporary multimodal studies Radford
et al. (2021); Zhai et al. (2023); Moskvoretskii and Kuznetsov (2023) we
consider differently pre-trained models as distinct modalities and employ
alignment techniques.

##### Contributions.

We are the first to our knowledge to examine the generative modeling for pre-
training in the domain of EvS. We introduce a novel method called the
Multimodal-Learning Event Model (MLEM) that aligns two pre-training
strategies. Specifically, we use a pre-trained contrastive model to align a
generative model. Our results have demonstrated that, on average, the aligned
model outperforms any single method across a diverse range of tasks and
datasets. Furthermore, our results indicate that self-supervised methods for
pre-training outperform supervised approaches on all the datasets we have
examined.

Furthermore, our study uncovers two significant insights. First, we find that
the generative approach consistently achieves superior performance on tasks
related to the Temporal Point Process (TPP), such as predicting the next event
type and event time.

Second, we find that most of the methods are robust to perturbations along the
time axis. However, they significantly degrade in performance on a synthetic
dataset where the time component plays a crucial role. This observation
suggests that some event sequences can be sufficiently classified using a ”Bag
of Words” approach, where the order of events is not as important.

We provide the source code for all the experiments described in this
paper.***The source code of our work is publicly available at
https://github.com/VityaVitalich/MLEM.

##  2 Related work

In the following section, we will review various generative, contrastive,
hybrid, and other related methods and works.

Generative methods for pre-training and representation learning have seen
notable development in recent years in both NLP Radford and Narasimhan (2018);
Devlin et al. (2019); Raffel et al. (2023) and CV He et al. (2021); Assran et
al. (2023); Bachmann et al. (2022). In NLP one of the generative pre-training
procedures is the next token prediction used in BERT Devlin et al. (2019).

In Lin et al. (2022) authors study generative modeling of EvS for TPP tasks.
Similarly to NLP, they use next-event prediction as their main training
objective. While EvS has been studied in such a generative context, the
authors in Lin et al. (2022) did not specifically investigate generative
modeling as a representation of learning or pre-training. The similarities
between autoregressive modeling in TPP and the success of generative
approaches in NLP have prompted us to consider the potential for applying
generative-style self-supervised learning and pre-training to the EvS domain.

Contrastive methods are a widely recognized in He et al. (2020); Grill et al.
(2020). These methods typically draw closer the embeddings of variations of
the same object and distance them from those of different objects. In CoLES
Babaev et al. (2022) authors study contrastive approach for EvS. In their
work, the positive samples consist of subsequences derived from a single
user’s events, while the negative samples are obtained from events of
different users. We utilize this method to examine the efficacy of the
contrastive approach in our study.

Hybrid methods. Our work falls into the category of hybrid self-supervised
approaches Qi et al. (2023); Yang et al. (2023); Lazarow et al. (2017); Oquab
et al. (2023); Zhou et al. (2022); Kim et al. (2021); Kim and Ye (2022);
Shukla and Marlin (2021). Few studies have merged generative and contrastive
methods, typically focusing on loss combination. DINOv2 Oquab et al. (2023)
updates the traditional contrastive loss from DINO Caron et al. (2021) with
iBOT’s Zhou et al. (2022) reconstruction loss through Masked Image Modeling.
Similarly, GCRL Kim et al. (2021) combines these losses via a weighted sum,
significantly outperforming models that are purely generative or contrastive.
EBCLR Kim and Ye (2022) also adopts this approach in the context of energy-
based models, using a sum of contrastive and generative losses.

Other hybrid methods focus on combining generative and supervised learning Liu
and Abbeel (2020). One such method, mTAND Shukla and Marlin (2021), has been
applied for EvS classification.

Supervised learning is commonly used for the EvS classification task. Some
works focus on modeling the underlying dynamics of the EvS process. For
example, irregular samples can be modeled by employing dynamic modeling with
Ordinary Differential Equations Rubanova et al. (2019). In contrast, SeqNAS
Udovichenko et al. (2024) adopts Neural Architecture Search to identify
optimal architectures without strong prior assumptions about the underlying
process. A key finding is the effectiveness of Recurrent Neural Networks
(RNNs) in modeling EvS, leading us to incorporate an RNN encoder in our model.
Furthermore, the authors propose benchmark datasets for EvS classification. We
selected the largest datasets from the benchmark, as self-supervised methods
require substantial data volumes.

Figure 1: A scheme for contrastive pre-training and representation learning is
illustrated using two different sequences,{$S_{1},S_{2}$}, where
$S\in\mathcal{R}^{l\times f}$ and $l$ is a sequence and sequence length,
respectively. The sub-sequences
{$S_{1}^{{}^{\prime}},S_{2}^{{}^{\prime}},S_{1}^{{}^{\prime\prime}},S_{2}^{{}^{\prime\prime}}$}
are sampled using a subsequence sampler described in Section 4.1, and the
latent representations
{$h_{1}^{{}^{\prime}},h_{2}^{{}^{\prime}},h_{1}^{{}^{\prime\prime}},h_{2}^{{}^{\prime\prime}}$},
where $h\in\mathcal{R}^{m}$, are produced by the contrastive encoder model
$\mathcal{E}_{c}$. In our work, we utilize the GRU (Gated Recurrent Unit) and
extract the last hidden state as $h$. The term $L^{con}$ denotes the
contrastive loss. Figure 2: In our generative method, the bottleneck encoder
$\mathcal{E}_{g}$ encodes the entire sequence $S\in\mathcal{R}^{l\times f}$
into $h\in\mathcal{R}^{m}$, $l$ denotes sequence lengths and $f$ and $m$
corresponding hidden sizes. In our work, we utilize the GRU (Gated Recurrent
Unit) and extract the last hidden state as $h$. For generation, we employ a
transformer decoder with recursive generation conditioned on $h$. The term
$L^{LM}$ denotes the reconstruction loss, and $[BOS]$ represents the Beginning
Of Sequence token. Figure 3: MLEM approach for EvS. We illustrate it with two
different sequences {$S_{1},S_{2}$}. Both the contrastive CON and generative
GEN encoders map the sequences {$S_{1},S_{2}$} into corresponding
{$h_{1}^{c},h_{2}^{c},h_{1}^{g},h_{2}^{g}$} latent representations. We align
the representations from different models using SigLIP and compute the
alignment loss, $L^{align}$ to train GEN. Simultaneously we train GEN encoder
with another reconstruction objective $L^{LM}$. In the end, only the GEN
encoder is used as the final model for fine-tuning and obtaining
representations. Throughout the procedure, we use a pre-trained CON encoder
with frozen weights.

##  3 Event Sequences Preliminaries

Datasets used in this work can be described as sets of sequences:
$C=\\{(S_{1},y_{1}),(S_{2},y_{2}),\ldots\\}$, where $y_{i}$ is an attribute
corresponding to the entire sequence or some target. Each
$S_{i}=((t^{1}_{i},\Upsilon^{1}_{i}),(t^{2}_{i},\Upsilon^{2}_{i}),\ldots)$ is
a sequence of events $x^{j}_{i}=(t^{j}_{i},\Upsilon^{j}_{i})$, where
$t_{i}^{j}$ represents the time when the event $x_{i}^{j}$ occurred and
$t_{i}^{j}\leq t^{j+1}_{i}$. The set of sub-events
$\Upsilon^{j}_{i}=\\{k^{1},k^{2},\ldots\\}$ describes the event $x_{i}^{j}$.
It is important to note that in this work, the terms Event Sequences (EvS) and
Irregularly Sampled Time-series (ISTS) are used interchangeably, as the
occurrence of measurement or sampling for the latter can be seen as an event.

##  4 Methodology

###  4.1 Contrastive Learning

Contrastive Learning encodes a sequence $S$ into a compact representation
$f_{e}:S_{i}\rightarrow h_{i}\in\mathcal{R}^{m}$, by bringing positive pairs
(i.e., semantically similar objects) closer to each other in the embedding
space, while pushing negative pairs (i.e., dissimilar objects) further apart.
In the CoLES Babaev et al. (2022) framework, the authors suggest constructing
a set of positive pairs by sampling sub-sequences from a sequence
$S_{i}\rightarrow\\{S_{i}^{{}^{\prime}},S_{i}^{{}^{\prime\prime}}\\}$, where
each element in $\\{S_{i}^{{}^{\prime}},S_{i}^{{}^{\prime\prime}}\\}$ is
shorter than $S_{i}$ and elements in
$\\{S_{i}^{{}^{\prime}},S_{i}^{{}^{\prime\prime}}\\}$ may intersect, the
number of sub-sequences is not limited to two. We adopt this approach and
refer to it as the subsequence sampler in Figure 1. Further, we utilize loss
function (1) from Hadsell et al. (2006). The overall procedure is illustrated
in Figure 1.

$\begin{split}L^{con}&=\frac{1}{|C|}\sum_{i=1}^{|C|}\sum_{j=1}^{|C|}\Bigg{(}z_{ij}\frac{1}{2}||h_{i}-h_{j}||_{2}^{2}\\\
&+(1-z_{ij})\frac{1}{2}\max\\{0,\rho-||h_{i}-h_{j}||_{2}\\}^{2}\Bigg{)}\end{split}$
(1)

Here, $z_{ij}\in\\{0,1\\}$ denotes if objects are semantically similar, $h$ is
the sequence embedding, $\rho$ \- the minimal margin between dissimilar
objects and $C=\\{S_{1},S_{2},\ldots\\}$ is a set of sequences.

###  4.2 Generative Modeling

We employ generative modeling to train our bottleneck encoder. Although our
methodology is similar to Sequence-to-Sequence models, it differs as we use a
bottleneck encoder to derive a single vector representation for the entire
sequence.

For the encoder, we utilize RNN, which extracts the final hidden state $h$,
which is subsequently passed into a Transformer decoder Vaswani et al. (2017).
The decoder reconstructs the entire sequence, conditioned on $h$ via the
Cross-Attention mechanism, in an auto-regressive manner. This entire process
is illustrated in Figure 2.

To train the model we use the next event prediction objective, which is
similar to training language models. Nonetheless, as each event in the
sequence $x^{j}_{i}=(t^{j}_{i},\Upsilon^{j}_{i})$ contain multiple sub-events
$\Upsilon=\\{k^{1},k^{2},\ldots\\}$, we need to reconstruct all of them. To
this end, we apply the cross-entropy loss for each categorical $k$ and the
Mean Squared Error (MSE) loss for each real-valued $k$. The final loss is a
cumulative sum of the losses for all elements in
$\Upsilon=\\{k^{1},k^{2},\ldots\\}$, plus the MSE loss for time $t$. We denote
this loss as $L^{LM}$.

Intuitively, this procedure requires our encoder to develop a representation
informative enough for the decoder to accurately reconstruct the entire
sequence. Since we map all our sequences $S$ into $h\in\mathcal{R}^{m}$ we
call all encoders bottleneck encoders.

All details related to the model can be found in Section 5.3.

Method | ABank | Age | PhysioNet | Pendulum | TaoBao
---|---|---|---|---|---
| ROC-AUC | Accuracy | ROC-AUC | MSE | ROC-AUC
Supervised | $0.768\pm 0.000$ | $0.602\pm 0.005$ | $0.790\pm 0.021$ | $\underline{0.33\pm 0.02}$ | $0.684\pm 0.002$
Contrastive | $0.729\pm 0.033$ | $0.638\pm 0.007$ | $\mathbf{0.815\pm 0.013}$ | $\mathbf{0.26\pm 0.02}$ | $0.679\pm 0.003$
Generative | $\underline{0.788\pm 0.003}$ | $0.639\pm 0.007$ | $0.787\pm 0.007$ | $\mathbf{0.26\pm 0.03}$ | $\mathbf{0.695\pm 0.004}$
Naïve | $0.658\pm 0.020$ | $0.638\pm 0.007$ | $0.759\pm 0.024$ | $\mathbf{0.26\pm 0.04}$ | $0.691\pm 0.002$
MLEM | $\mathbf{0.790\pm 0.000}$ | $\mathbf{0.642\pm 0.005}$ | $0.780\pm 0.004$ | $\mathbf{0.26\pm 0.05}$ | $\mathbf{0.695\pm 0.002}$

Table 1: Evaluation of self-supervised methods fine-tuned using supervised
loss. ABank, PhysioNet and TaoBao are reported with ROC-AUC, AGE reported with
accuracy and Pendulum with MSE.

###  4.3 Aligning generative encoder

To align the embeddings obtained through generative modeling with those
acquired through contrastive learning, we treat them as distinct modalities
and employ a contrastive aligning procedure inspired by CLIP Radford et al.
(2021). We utilize the recent SigLIP loss Zhai et al. (2023) for this purpose.
The resulting aligned encoder model is referred to as the Multimodal-Learning
Event Model (MLEM).

Overall MLEM training procedure is exactly the same as for the generative
model described in Section 4.2, except the alignment, which incurs an
additional loss (2) resulting in the total loss (3). Training MLEM requires a
pre-trained contrastive encoder. Using this encoder we train a generative
model and align its embeddings with the embeddings produced by the contrastive
encoder. Contrastive encoder weights are not updated during training. Both
generative $\mathcal{E}_{g}$ and contrastive $\mathcal{E}_{c}$ encoders
receive a set of sequences $C=\\{S_{1},S_{2},\ldots\\}$ and map each sequence
$S$ into corresponding hidden states $h^{g}$ and $h^{c}$, then the goal of
MLEM alignment is to pull embeddings obtained from the same sequence closer to
each other and to push embeddings from different sequences further away. This
procedure is illustrated in Figure 3. Below is the alignment loss we use:

$L^{align}=\frac{1}{|C|}\sum_{i=1}^{|C|}\sum_{j=1}^{|C|}\underbrace{\log\frac{1}{1+e^{z_{ij}(-t\cdot{h^{g}_{i}}\cdot
h^{c}_{j}+b)}}}_{L^{align}_{ij}}$ (2)

Here, $z_{ij}\in\\{-1,1\\}$ indicates if a pair of embeddings originated from
the same sequence $S$, $t$ and $b$ denote the temperature and bias,
respectively, both are learnable parameters.

Total loss to train MLEM:

$L=\alpha L^{LM}(S,\hat{S})+\beta
L^{align}(\mathcal{E}_{g}(S),\mathcal{E}_{c}(S))$ (3)

Here, $S$ and $\hat{S}$ denote original and reconstructed sequences, $\alpha$
and $\beta$ are hyperparameters that adjust the strength of alignment.

All technical details can be found in Section 5.3.

###  4.4 Naïve method

As a baseline, we also examine a straightforward, or Naïve, approach that
merges generative and contrastive methods. This is achieved by incorporating a
contrastive objective into the generative model, akin to methodologies used in
prior studies Kim et al. (2021); Oquab et al. (2023). The final loss is a
weighted sum of objectives $L=\alpha L^{LM}+\beta L^{con}$. One of the
downsides of this implementation is that we can not utilize full-length
sequences because sub-sequence sampling is required for contrastive procedure.
Further, we compare Naïve method with MLEM and show that while it is very
close by design it leads to inferior results.

##  5 Experiments

###  5.1 Datasets

In our study, we use five EvS datasets formally outlined in Section 3.

  * •

A subset of EvS datasets from SeqNAS benchmark Udovichenko et al. (2024), more
precisely: ABank, Age, TaoBao. ABank and Age are bank transactions and TaoBao
is user activity.

  * •

PhysioNet 2012 comprises highly sparse medical event data, with the goal of
predicting mortality.

  * •

Pendulum is a synthetic dataset simulating pendulum motion with the objective
of predicting its length using a sequence of coordinates. This dataset was
created specifically to evaluate a model’s capability to effectively
incorporate the time component in sequence modeling.

Each dataset contains a range of features, both numerical and categorical. All
datasets, with the exception of ABank, consist of irregularly sampled time
steps. Comprehensive details for each dataset are available in the
supplementary materials.

| ABank | Age | PhysioNet | Pendulum | TaoBao
---|---|---|---|---|---
| ROC-AUC | TPP Acc. | Acc. | TPP Acc. | ROC-AUC | TPP MSE | MSE | TPP MSE | ROC-AUC | TPP Acc.
| Linear Probing
Contrastive | $0.678\pm 0.003$ | $0.37$ | $0.623\pm 0.010$ | $0.28$ | $\mathbf{0.819\pm 0.003}$ | $1.05$ | $\mathbf{0.37\pm 0.02}$ | $0.8$ | $0.680\pm 0.001$ | $0.22$
Generative | $0.753\pm 0.003$ | $0.43$ | $0.610\pm 0.004$ | $0.3$ | $0.759\pm 0.010$ | $\mathbf{0.93}$ | $0.74\pm 0.09$ | $0.8$ | $\mathbf{0.689\pm 0.005}$ | $\mathbf{0.35}$
Naïve | $0.703\pm 0.002$ | $0.37$ | $0.602\pm 0.003$ | $\mathbf{0.31}$ | $0.733\pm 0.005$ | $\mathbf{0.93}$ | $0.59\pm 0.07$ | $0.8$ | $0.680\pm 0.003$ | $0.32$
MLEM | $\mathbf{0.757\pm 0.000}$ | $\mathbf{0.43}$ | $\mathbf{0.627\pm 0.000}$ | $0.29$ | $0.762\pm 0.010$ | $\mathbf{0.93}$ | $0.41\pm 0.01$ | $\mathbf{0.79}$ | $0.683\pm 0.003$ | $\mathbf{0.35}$
| Non-linear Probing
Contrastive | $0.691\pm 0.004$ | $0.37$ | $0.629\pm 0.010$ | $0.28$ | $\mathbf{0.822\pm 0.001}$ | $1.05$ | $\mathbf{0.37\pm 0.00}$ | $0.8$ | $0.686\pm 0.001$ | $0.22$
Generative | $0.758\pm 0.003$ | $0.43$ | $0.618\pm 0.004$ | $0.3$ | $0.764\pm 0.006$ | $\mathbf{0.93}$ | $0.63\pm 0.02$ | $0.8$ | $0.684\pm 0.002$ | $\mathbf{0.35}$
Naïve | $0.704\pm 0.005$ | $0.37$ | $0.608\pm 0.002$ | $\mathbf{0.31}$ | $0.774\pm 0.008$ | $\mathbf{0.93}$ | $0.57\pm 0.05$ | $0.8$ | $\mathbf{0.690\pm 0.002}$ | $0.32$
MLEM | $\mathbf{0.759\pm 0.003}$ | $\mathbf{0.43}$ | $\mathbf{0.634\pm 0.005}$ | $0.29$ | $0.780\pm 0.001$ | $\mathbf{0.93}$ | $0.40\pm 0.01$ | $\mathbf{0.79}$ | $0.688\pm 0.002$ | $\mathbf{0.35}$

Table 2: Evaluation of self-supervised methods using linear and non-linear
probing for downstream tasks and TPP tasks. The table reports the mean and
standard deviation calculated from three different seeds for downstream task
metrics. For TPP tasks, only the mean values from three seeds are presented,
as the standard deviation was consistently $0.00$. In the table, the best-
performing values are highlighted in bold, while the second-best values are
underlined.

###  5.2 Evaluation objectives

To evaluate the effectiveness of self-supervised training strategies, our
study focuses on two key aspects: the quality of the embeddings and the
effectiveness of fine-tuning the entire network, following self-supervised
studies Dubois et al. (2024, 2021).

####  5.2.1 Main Objectives

To assess the quality of embeddings, we primarily rely on metrics from
downstream tasks. This involves utilizing both linear and non-linear probing
methods. For non-linear probing, we employ Gradient Boosting algorithms with
LGBM Ke et al. (2017) applied directly to the embeddings.

We evaluate embeddings on several tasks, including the prediction of an
attribute or a target $y_{i}$ given the entire $S_{i}$ and the prediction of
consequent event $x^{j}$ given a set of
$S_{i}=\\{x_{i}^{1},\ldots,x^{j-1}_{i}\\}$. The second objective addresses the
TPP task, which involves predicting the type of the next event or the time of
the next event’s arrival. We have processed each dataset such that the target
is either the category of the next event (for datasets that include this
feature) or the time until the next event (for datasets without a primary
event category).

####  5.2.2 Secondary Objectives

We incorporate additional metrics to assess the quality of embeddings.
Specifically, we utilize anisotropy and intrinsic dimension, drawing on
insights from previous research Nakada and Imaizumi (2020); Razzhigaev et al.
(2023). In Section 6.3, we compare our findings with the results presented in
the aforementioned works.

Anisotropy assesses the non-uniform distribution of embeddings in space,
providing insights into the contextualization of our embedding. Lower
anisotropy in embeddings has been linked to better model performance Ait-Saada
and Nadif (2023). In line with the approach used in Razzhigaev et al. (2023),
we compute anisotropy as the ratio of the highest singular value to the sum of
all singular values:

$Anisotropy(H)=\frac{\sigma_{1}^{2}}{\sum\limits_{i}\sigma_{i}^{2}}$

The intrinsic dimension evaluates the optimal dimensionality of data, shedding
light on the core information captured by the embeddings. To determine the
intrinsic dimension, we employ the method proposed in Facco et al. (2017).
This method examines how the volume of an $n$-dimensional sphere changes with
dimension $d$. Further details are available in the original paper or in
Razzhigaev et al. (2023).

###  5.3 Models

Feature embeddings. To transform features into a numerical vector, we use an
embedding layer for categorical features and linear projection for numerical
ones. For ABank, Age and TaoBao we set the dimension of each feature to 32.
For Pendulum and PhysioNet dimension is set to 8. Additionally, we use the
time difference between events as a separate feature.

For consistency in our experiments, we use the same encoder architecture
across all training strategies and datasets. We use a single-layer GRU with a
hidden size of 512, we take the last hidden state as sequence embedding. The
GRU is selected for its proven effectiveness in encoding time-ordered
sequences and its extensive application in relevant literature, providing a
reliable baseline for our study Rubanova et al. (2019); Tonekaboni et al.
(2021); Yoon et al. (2019); Udovichenko et al. (2024).

For Supervised model, we use the aforementioned encoder with a linear head.

Contrastive learning approach is described at 4.1 and uses the same encoder as
mentioned above. Furthermore, we integrate a Feed Forward Projector atop the
GRU for loss calculation, a technique commonly adopted in several studies for
enhanced performance Grill et al. (2020); Oquab et al. (2023).

For the generative modeling, we employ a vanilla transformer decoder
configured with LayerNorm, consisting of 3 layers, 2 heads, a hidden size of
128, and a Feed Forward hidden size of 256. To ensure training stability, we
also apply LayerNorm to the sequence embedding produced by the encoder.
Additionally, a projection layer is used atop the decoder to predict each
feature.

###  5.4 Training details

To maintain consistency in our experiment, we trained all models using
identical parameters. Models were trained for 100 epochs on datasets with
fewer than 100K sequences and for 40 epochs on larger datasets. The learning
rate was set at 1e-3, weight decay at 3e-3, and the batch size was fixed at
128. For the MLEM model, we set $\alpha$ at 1 and $\beta$ at 10. We averaged
the results across three different seeds to ensure reliability and
reproducibility.

##  6 Results

###  6.1 Self-supervised pre-training for fine-tuning

Here we evaluate the effectiveness of self-supervised pre-training strategies
prior to supervised fine-tuning the entire model. For fine-tuning, we employed
pre-trained encoders with randomly initialized linear layers and trained the
model for 10 epochs. The results presented in Table 1 indicate that, among all
the evaluated pre-training methods, MLEM consistently achieves the most
favorable results across all datasets, with the exception of the highly sparse
PhysioNet dataset. Furthermore, it is noteworthy that supervised learning
without self-supervised pre-training consistently produces inferior results.

###  6.2 Embeddings probing

We conducted a comprehensive analysis of the quality of embeddings using both
linear and non-linear probing techniques, results are presented in Table 2. We
evaluated performance on various tasks described in Section 5.2.1, including
EvS classification and regression tasks for the entire sequence, as well as
TPP tasks. Despite some variations in absolute values, the overall trends and
rankings of methods remain consistent across both probing strategies.

Our assessment reveals that neither the contrastive nor the generative
approach consistently outperforms the other. However, the MLEM model
consistently demonstrates superior performance across most datasets. Notably,
when MLEM is not the top performer, it often ranks second, highlighting its
versatility. By effectively integrating the advantages of both contrastive and
generative techniques, this approach consistently delivers strong overall
performance across a range of datasets.

Figure 4: Mean percentage change averaged across datasets. The X-axis
represents the dropout probability, while the Y-axis indicates the mean change
in the linear probing downstream metric, compared to the metric with no
dropout.

###  6.3 Anisotropy and Intrinsic Dimension

Method | Anisotropy $\downarrow$ | Intrinsic Dimension $\uparrow$
---|---|---
Contrastive | $0.11\pm 0.04$ | $10.15\pm 6.12$
Generative | $0.08\pm 0.03$ | $14.86\pm 10.12$
Naïve | $0.07\pm 0.04$ | $11.26\pm 6.51$
MLEM | $\mathbf{0.06\pm 0.03}$ | $\mathbf{15.86\pm 9.02}$
Table 3: Average anisotropy and intrinsic dimension across datasets

We conducted an evaluation of anisotropy and the intrinsic dimensions of
embeddings from various pre-training strategies, as detailed in 5.2.2. As a
result, we show that MLEM has the highest intrinsic dimension and the lowest
anisotropy, on average, across all datasets. This finding may indicate that
our pre-training strategy effectively enables the embeddings to encapsulate a
greater amount of information and ensures their uniform distribution in space.
The results are presented in Table 3.

Contrary to the findings in Dubois et al. (2024), we did not observe any
correlation between intrinsic dimension and downstream performance metrics.
Similarly, we did not find any correlation for anisotropy, which aligns with
the aforementioned work.

Figure 5: Correlation plots for anisotropy and intrinsic dimension with
normalized performance.

Figure 6: The t-SNE visualizations showcase embeddings resulting from various
pre-training strategies. The top row displays the embeddings for the Age
dataset, while the bottom row illustrates those for the TaoBao dataset. Each
point represents a sequence $S_{i}$ from a given dataset, colored accordingly
to the corresponding attribute $y_{i}$. For Age, there are 4 classes and 2
classes for TaoBao

###  6.4 Visualization

The findings related to anisotropy and intrinsic dimension align with the
t-SNE visualization of the Age and TaoBao datasets shown in Figure 6. While
contrastive embeddings typically form distinct clusters, the generative
approaches and MLEM provide a more complex embedding space.

##  7 Embedding Robustness

Our study also explores the robustness of model embeddings to specific types
of perturbations. We focus on examining the impact of dropout and the
shuffling of events within one sequence on the performance of the embeddings
on downstream tasks.

###  7.1 Sensitivity to data omission

To investigate the effects of data omission, we applied a random dropout
strategy, removing entire events from a sequence along the time axis, with
varying probabilities $p_{dropout}\in\\{0.1,0.3,0.5,0.7\\}$. This allowed us
to examine the impact of different levels of data removal on the embeddings,
as seen in Figure 4, by measuring the average percentage decrease in
downstream task metrics.

MLEM’s performance slightly decreases at $0.1$ dropout probability and more so
at higher levels. This indicates that MLEM is more sensitive to dropout and
requires the presence of all the samples rather than depending on one specific
sample. This property can be used in a sensitivity analysis or related
applications.

Additionally, the generative modeling embeddings exhibited an intriguing
behavior: they not only resisted the decline but actually surpassed their
original performance at a dropout rate of 0.5. To draw reliable conclusions
based on this finding, further work is required with a larger number of
datasets and various models.

###  7.2 Sensitivity to perturbations

In subsequent tests, we examined the impact of perturbing events within a
sequence on downstream task performance. The results, presented in Table 4,
show that perturbing sequence elements had minimal to no effect on the
results. This suggests that current models may be employing a ’Bag-of-Words’
approach and considering a set of events rather than the sequential nature of
the data. One might assume that this discrepancy is due to not suitable models
or training strategies. However, this hypothesis is not supported by a decline
in performance on the pendulum dataset, where the time component is crucial
and the same models were used.

This raises the question of whether we could further enhance model performance
by explicitly considering the sequential nature of the data, or if this is
solely a property of the datasets themselves.

Method | ABank | Age | PhysioNet | Pendulum | TaoBao
---|---|---|---|---|---
Contrastive | +0.47% | -1.93% | -0.37% | -367.57% | -0.15%
Generative | -0.93% | +0.16% | -6.19% | -91.89% | -0.29%
Naive | -1.42% | +0.99% | -3.00% | -147.46% | 0.00%
MLEM | -1.45% | -0.16% | -6.56% | -285.37% | +0.29%

Table 4: Percentage change in linear probing performance after shuffling
events order

##  8 Computational complexity

To obtain the MLEM encoder, we need to train two models, which is not
desirable. However, to demonstrate the practicality of our method, we compare
MLEM with SeqNas Udovichenko et al. (2024) in terms of performance and
computational efficiency. SeqNas employs an architectural search to identify
the most optimal supervised architecture for the task at hand. We posit that
SeqNas represents an upper bound for downstream metrics. Table 5 presents
results from the two largest datasets, examined by us, demonstrating that MLEM
achieves performance near the upper bound set by SeqNas, but with
significantly lower computational cost.

Method | ABank | Age
---|---|---
| ROC-AUC | GPU hours | Accuracy | GPU hours
MLEM | $0.790\pm 0.000$ | $25$ | $0.642\pm 0.005$ | $9$
SeqNas | $0.7963\pm 0.001$ | $288$ | $0.645\pm 0.002$ | $80$

Table 5: Comparison of downstream metrics and computational costs for training
and fine-tuning MLEM versus SeqNas.

##  9 Discussion

We observe that in our settings, MLEM outperforms the Naïve method, despite
their initial similarities. One possible explanation for the success of our
method can be attributed to viewing MLEM as a Knowledge Distillation Hinton et
al. (2015) procedure. However, further work is required to confirm this
hypothesis.

We observed that data perturbation does not significantly impact model
performance, suggesting a near-time-invariance. This could be attributed to
the nature of the examined datasets or the absence of appropriate modeling
approaches. Both hypotheses suggest the potential for future research in time-
sensitive modeling.

We also observe consistent underperformance of methods that include generative
approach on the PhysioNet dataset. This could be attributed to the high rate
of missing values, which complicates the generation of accurate predictions
for subsequent steps. Furthermore, the relatively small size of the dataset
may be a contributing factor, given that generative modeling is generally
dependent on large amounts of data.

##  10 Conclusion

We proposed a novel approach to combine multiple self-supervised strategies
and demonstrated that this approach yields superior results compared to any
single self-supervised method. We believe this opens up a new direction for
enhancing self-supervised learning.

Our study revealed that self-supervised pre-trained models, especially those
with generative pre-training, outperform purely supervised methods, with MLEM
showing the most promise. This suggests potential benefits in exploring
different pre-training and architectural options.

In our work, we are the first, to our knowledge, to apply generative modeling
to event sequences for representation learning and pre-training. Our findings
show that neither generative nor contrastive pre-training clearly surpasses
the other in effectiveness. To address this, we developed the MLEM method,
which innovatively merges contrastive learning with generative modeling. MLEM
aligns these two approaches as separate modalities, inspired by recent
advances in multimodal research, and it demonstrates enhanced performance on
most datasets for both primary and secondary goals.

## References

  * Ait-Saada and Nadif [2023] Mira Ait-Saada and Mohamed Nadif.  Is anisotropy truly harmful? a case study on text clustering.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1194–1203, Toronto, Canada, July 2023. Association for Computational Linguistics.
  * Assran et al. [2023] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas.  Self-supervised learning from images with a joint-embedding predictive architecture, 2023.
  * Babaev et al. [2022] Dmitrii Babaev, Nikita Ovsov, Ivan Kireev, Maria Ivanova, Gleb Gusev, Ivan Nazarov, and Alexander Tuzhilin.  Coles: contrastive learning for event sequences with self-supervision.  In Proceedings of the 2022 International Conference on Management of Data, pages 1190–1199, 2022.
  * Bachmann et al. [2022] Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir.  Multimae: Multi-modal multi-task masked autoencoders, 2022.
  * Carnein and Trautmann [2019] Matthias Carnein and Heike Trautmann.  Customer segmentation based on transactional data using stream clustering.  In Advances in Knowledge Discovery and Data Mining: 23rd Pacific-Asia Conference, PAKDD 2019, Macau, China, April 14-17, 2019, Proceedings, Part I 23, pages 280–292. Springer, 2019.
  * Caron et al. [2021] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers, 2021.
  * Chen et al. [2020] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations, 2020.
  * Devlin et al. [2019] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * Dubois et al. [2021] Yann Dubois, Douwe Kiela, David J. Schwab, and Ramakrishna Vedantam.  Learning optimal representations with the decodable information bottleneck, 2021.
  * Dubois et al. [2024] Yann Dubois, Tatsunori Hashimoto, and Percy Liang.  Evaluating self-supervised learning via risk decomposition, 2024.
  * Facco et al. [2017] Elena Facco, Maria d’Errico, Alex Rodriguez, and Alessandro Laio.  Estimating the intrinsic dimension of datasets by a minimal neighborhood information.  Scientific Reports, 7(1), September 2017.
  * Fursov et al. [2021] Ivan Fursov, Matvey Morozov, Nina Kaploukhaya, Elizaveta Kovtun, Rodrigo Rivera-Castro, Gleb Gusev, Dmitry Babaev, Ivan Kireev, Alexey Zaytsev, and Evgeny Burnaev.  Adversarial attacks on deep models for financial transaction records, 2021.
  * Grill et al. [2020] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko.  Bootstrap your own latent: A new approach to self-supervised learning, 2020.
  * Hadsell et al. [2006] Raia Hadsell, Sumit Chopra, and Yann Lecun.  Dimensionality reduction by learning an invariant mapping.  pages 1735 – 1742, 02 2006.
  * He et al. [2020] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning, 2020.
  * He et al. [2021] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners, 2021.
  * Hinton et al. [2015] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * Jain et al. [2021] Hemlata Jain, Ajay Khunteta, and Sumit Srivastava.  Telecom churn prediction and used techniques, datasets and performance measures: a review.  Telecommunication Systems, 76:613–630, 2021.
  * Ke et al. [2017] Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu.  Lightgbm: A highly efficient gradient boosting decision tree.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
  * Kim and Ye [2022] Beomsu Kim and Jong Chul Ye.  Energy-based contrastive learning of visual representations, 2022.
  * Kim et al. [2021] Saehoon Kim, Sungwoong Kim, and Juho Lee.  Hybrid generative-contrastive representation learning, 2021.
  * Lazarow et al. [2017] Justin Lazarow, Long Jin, and Zhuowen Tu.  Introspective neural networks for generative modeling.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2774–2783, 2017.
  * Liguori et al. [2023] Angelica Liguori, Luciano Caroprese, Marco Minici, Bruno Veloso, Francesco Spinnato, Mirco Nanni, Giuseppe Manco, and Joao Gama.  Modeling events and interactions through temporal processes–a survey.  arXiv preprint arXiv:2303.06067, 2023.
  * Lin et al. [2022] Haitao Lin, Lirong Wu, Guojiang Zhao, Pai Liu, and Stan Z Li.  Exploring generative neural temporal point process.  arXiv preprint arXiv:2208.01874, 2022.
  * Liu and Abbeel [2020] Hao Liu and Pieter Abbeel.  Hybrid discriminative-generative training via contrastive learning.  arXiv preprint arXiv:2007.09070, 2020.
  * Moskvoretskii and Kuznetsov [2023] Viktor Moskvoretskii and Denis Kuznetsov.  Imad: Image-augmented multi-modal dialogue.  arXiv preprint arXiv:2305.10512, 2023.
  * Nakada and Imaizumi [2020] Ryumei Nakada and Masaaki Imaizumi.  Adaptive approximation and generalization of deep neural network with intrinsic dimensionality, 2020.
  * Oquab et al. [2023] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  Dinov2: Learning robust visual features without supervision, 2023.
  * Qi et al. [2023] Zekun Qi, Runpei Dong, Guofan Fan, Zheng Ge, Xiangyu Zhang, Kaisheng Ma, and Li Yi.  Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining.  arXiv preprint arXiv:2302.02318, 2023.
  * Radford and Narasimhan [2018] Alec Radford and Karthik Narasimhan.  Improving language understanding by generative pre-training.  2018\.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision, 2021.
  * Raffel et al. [2023] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu.  Exploring the limits of transfer learning with a unified text-to-text transformer, 2023.
  * Razzhigaev et al. [2023] Anton Razzhigaev, Matvey Mikhalchuk, Elizaveta Goncharova, Ivan Oseledets, Denis Dimitrov, and Andrey Kuznetsov.  The shape of learning: Anisotropy and intrinsic dimensions in transformer-based models, 2023.
  * Rubanova et al. [2019] Yulia Rubanova, Ricky T. Q. Chen, and David Duvenaud.  Latent odes for irregularly-sampled time series, 2019.
  * Shao et al. [2019] Siyu Shao, Ruqiang Yan, Yadong Lu, Peng Wang, and Robert X Gao.  Dcnn-based multi-signal induction motor fault diagnosis.  IEEE Transactions on Instrumentation and Measurement, 69(6):2658–2669, 2019.
  * Shukla and Marlin [2021] Satya Narayan Shukla and Benjamin M. Marlin.  Multi-time attention networks for irregularly sampled time series, 2021.
  * Sudharsan and Ganesh [2022] R Sudharsan and EN Ganesh.  A swish rnn based customer churn prediction for the telecom industry with a novel feature selection strategy.  Connection Science, 34(1):1855–1876, 2022.
  * Tonekaboni et al. [2021] Sana Tonekaboni, Danny Eytan, and Anna Goldenberg.  Unsupervised representation learning for time series with temporal neighborhood coding, 2021.
  * Udovichenko et al. [2024] Igor Udovichenko, Egor Shvetsov, Denis Divitsky, Dmitry Osin, Ilya Trofimov, Ivan Sukharev, Anatoliy Glushenko, Dmitry Berestnev, and Evgeny Burnaev.  SeqNAS: Neural architecture search for event sequence classification.  IEEE Access, 12:3898–3909, 2024.
  * Valeri et al. [2023] Jacqueline A Valeri, Luis R Soenksen, Katherine M Collins, Pradeep Ramesh, George Cai, Rani Powers, Nicolaas M Angenent-Mari, Diogo M Camacho, Felix Wong, Timothy K Lu, et al.  Bioautomated: An end-to-end automated machine learning tool for explanation and design of biological sequences.  Cell Systems, 14(6):525–542, 2023.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * Waring et al. [2020] Jonathan Waring, Charlotta Lindvall, and Renato Umeton.  Automated machine learning: Review of the state-of-the-art and opportunities for healthcare.  Artificial intelligence in medicine, 104:101822, 2020.
  * Xie et al. [2022] Yu Xie, Guanjun Liu, Chungang Yan, Changjun Jiang, and MengChu Zhou.  Time-aware attention-based gated network for credit card fraud detection by extracting transactional behaviors.  IEEE Transactions on Computational Social Systems, 2022.
  * Yang et al. [2023] Yonghui Yang, Zhengwei Wu, Le Wu, Kun Zhang, Richang Hong, Zhiqiang Zhang, Jun Zhou, and Meng Wang.  Generative-contrastive graph learning for recommendation.  2023\.
  * Yoon et al. [2019] Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar.  Time-series generative adversarial networks.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * Zhai et al. [2023] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer.  Sigmoid loss for language image pre-training, 2023.
  * Zhou et al. [2022] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong.  ibot: Image bert pre-training with online tokenizer, 2022.
  * Zhuzhel et al. [2023] Vladislav Zhuzhel, Vsevolod Grabar, Galina Boeva, Artem Zabolotnyi, Alexander Stepikin, Vladimir Zholobov, Maria Ivanova, Mikhail Orlov, Ivan Kireev, Evgeny Burnaev, et al.  Continuous-time convolutions model of event sequences.  arXiv preprint arXiv:2302.06247, 2023.

## Detailed Dataset Information

Detailed information regarding all the datasets is presented in Table 6.

ABank is a dataset of bank transactions with regularly sampled time-steps,
representing transaction IDs rather than the transaction times. The goal is
binary classification: predicting whether a user will default.

Age consists of bank transactions with irregularly sampled time-steps. The
task is to categorize users into one of four age groups.

PhysioNet features medical measurements with irregularly sampled time-steps.
The target is binary classification: predicting in-hospital mortality.

Pendulum is a synthetic dataset created from pendulum simulations with
irregularly sampled time-steps, using the Hawkes process. It includes features
like the x and y coordinates of the pendulum, normalized by its length. Each
observation has a unique real-valued pendulum length, and the task is to
predict this length. The detailed dataset synthesis procedure is described in
section Pendulum dataset generation.

Taobao captures user activity on the TaoBao platform with irregular time-
steps. The binary classification task is to predict whether a user will make a
payment in the next 7 days.

For all datasets, categories that appeared fewer than 500 times were
consolidated into a single category. Event time values were normalized to a
unit interval for consistency. In the case of the PhysioNet dataset, features
were aggregated at 360-second intervals. If a feature appeared at least once
within this timeframe, its average was calculated. If a feature was absent, it
was marked with a value of -1. No other manual feature generation or specific
preprocessing was performed, unless otherwise specified.

Train - test - validation splits. For test sets we followed existing
literature Babaev et al. [2022]; Udovichenko et al. [2024], with the exception
of the PhysioNet dataset, for which we created the split. The train and
validation sets for each training session were random in fixed proportions.
All the metrics are reported on test sets. In cases where sequences exceeded a
specified length, we selected the $N$ most recent events, with $N$ being the
sequence length as defined in dataset Table 6.

| ABank | Age | Taobao | PhysioNet | Pendulum
---|---|---|---|---|---
# Observations | 2.7B | 26B | 7.8M | 0.5M | 8.9M
Mean observations per user | 280 | 881 | 806 | 72 | 89
Observations std. per user | 270 | 124 | 1042 | 21 | 11
Max observations per user in modeling | 200 | 1000 | 1000 | 200 | 200
# Sequences | 963M | 30K | 9K | 8K | 100K
# Classes | 2 | 4 | 2 | 2 | -
# Categorical features | 16 | 1 | 2 | 3 | 0
# Real-valued features | 3 | 1 | 0 | 38 | 2
Target | Default | Age group | Payment | Mortality | Length

Table 6: Statistics of datasets

## Pendulum dataset generation

We developed a pendulum dataset to assess models when time dependency is
crucial.

We simulated pendulum motion with different lengths and sampled coordinates
with irregular time intervals, which were derived using a sampling method
based on the Hawkes process. Therefore, our dataset consists of sequences
where each event is represented with time and two coordinates, each sequence
is generated with different pendulum length.

We opted to Hawkes process to emphasize the critical role of accurate event
timing in successful model performance corresponding to real world
applications.

To model the Hawkes process, we consider the following intensity function
$\lambda(t)$ that is given by (4).

$\lambda(t)=\mu+\sum_{t_{i}<t}\alpha e^{-\beta(t-t_{i})}$ (4)

We used following parameters for the Hawkes process:

  * •

$\mu$ is the base intensity, was fixed at 10;

  * •

$\alpha$ is the excitation factor, was chosen to be 0.2;

  * •

$\beta$ is the decay factor, was set to 1.

  * •

$t_{i}$ are the times of previous events before time $t$.

The example of generated event times with these parameters is depicted in
Figure 8.

Figure 7: The figure illustrates the pendulum motion at various instances,
with time steps determined by a Hawkes process. It captures the pendulum’s
trajectory using only the normalized planar coordinates at these sampled
times. Figure 8: Example of temporal distribution of events ranging from 0 to
5 seconds. Each event is marked with a star along the timeline. The y-axis
serves only as a technical aid to separate the events for clarity and does not
convey any additional information.

These parameter values were selected with the intention of generating
sequences that contained approximately 100 events each. Additionally, this
specific combination of $\mu$, $\alpha$, and $\beta$ was designed to create
sequences where events would be densely clustered during certain intervals and
sparsely distributed during others. This configuration allowed us to simulate
scenarios that closely mimic real-world dynamics, where event occurrences can
fluctuate between periods of high and low activity.

To model the pendulum we consider the second-order differential equation:

$\theta^{\prime\prime}+\left(\frac{b}{m}\right)\theta^{\prime}+\left(\frac{g}{L}\right)\sin(\theta)=0$
(5)

where,

  * •

$\theta^{\prime\prime}$ is the Angular Acceleration,

  * •

$\theta^{\prime}$ is the Angular Velocity,

  * •

$\theta$ is the Angular Displacement,

  * •

$b$ is the Damping Factor,

  * •

$g=9.81\,\text{m/s}^{2}$ is the acceleration due to gravity,

  * •

$L$ is the Length of pendulum,

  * •

$m$ is the Mass of bob in kg.

To convert this second-order differential equation into two first-order
differential equations, we let $\theta_{1}=\theta$ and
$\theta_{2}=\theta^{\prime}$, which gives us:

$\theta_{2}^{\prime}=\theta^{\prime\prime}=-\left(\frac{b}{m}\right)\theta_{2}-\left(\frac{g}{L}\right)\sin(\theta_{1})$
(6) $\theta_{1}^{\prime}=\theta_{2}$ (7)

Thus, the first-order differential equations for the pendulum simulation are:

$\displaystyle\theta_{2}^{\prime}$
$\displaystyle=-\left(\frac{b}{m}\right)\theta_{2}-\left(\frac{g}{L}\right)\sin(\theta_{1})$
(8) $\displaystyle\theta_{1}^{\prime}$ $\displaystyle=\theta_{2}$ (9)

In our simulations, we fixed the damping factor $b=0.5$ and the mass of the
bob $m=1$. The length $L$ of the pendulum is taken from a uniform distribution
$U(0.5,5)$, representing a range of possible lengths from 0.5 to 5 meters. The
initial angular displacement $\theta$ and the initial angular velocity
$\theta^{\prime}$ are both taken from a uniform distribution $U(1,9)$, which
provides a range of initial conditions in radians and radians per second,
respectively.

Our primary objective is to predict the length of the pendulum, denoted as
$L$, using the normalized coordinates $x$ and $y$ on the plane. These
coordinates are scaled with respect to the pendulum’s length, such that the
trajectory of the pendulum is represented in a unitless fashion. This
normalization allows us to abstract the pendulum’s motion from its actual
physical dimensions and instead focus on the pattern of movement. An
illustrative example of this motion is presented in Figure 7, where the path
traced by the pendulum bob is depicted over time.

## Generated datasets

We conducted a preliminary evaluation of our decoder’s ability to reconstruct
sequences from their embeddings. We hypothesize that while the regenerated
sequences may exhibit slight deviations from the original data, the overall
distribution of features across whole dataset should align. To investigate
this, we compared distributions of features in generated datasets, results are
visualized in Figure 9 for Age dataset.

There is a notable resemblance between the generated sequences and the actual
data regarding the distribution of numerical feature ”amount”, particularly
around the mean. However, the model struggles with accurately reproducing the
timing and the key dataset feature—the user group. The MLEM tends to
overrepresent the most frequent classes while underrepresenting less common
ones. Moreover, the Generative model despite of the mostly same performance
exhibits unexpected behaviour, overproducing some of the rarer classes.

These findings suggest directions for potential improvements, more
specifically: improving time component modeling, applying different generation
approaches, and studying different architecture designs such as enhancing
either the encoder’s or decoder’s performance.

Figure 9: Distributions of features for real and generated Age dataset. True
denotes distributions for real dataset, Generative and MLEM denote
distributions for datasets generated from embeddigs obtain via Generative and
MLEM approaches.