# Exploiting Session Information in BERT-based Session-aware Sequential
Recommendation

Jinseok Seol, Youngrok Ko, Sang-goo Lee  Seoul National UniversityRepublic of
Korea jamie, yrko1<EMAIL_ADDRESS>
(2022)

###### Abstract.

In recommendation systems, utilizing the user interaction history as
sequential information has resulted in great performance improvement. However,
in many online services, user interactions are commonly grouped by sessions
that presumably share preferences, which requires a different approach from
ordinary sequence representation techniques. To this end, sequence
representation models with a hierarchical structure or various viewpoints have
been developed but with a rather complex network structure. In this paper, we
propose three methods to improve recommendation performance by exploiting
session information while minimizing additional parameters in a BERT-based
sequential recommendation model: using session tokens, adding session segment
embeddings, and a time-aware self-attention. We demonstrate the feasibility of
the proposed methods through experiments on widely used recommendation
datasets.

Session-aware Recommendation, Sequential Recommendation, Temporal Self-
Attention

††copyright: acmcopyright††journalyear: 2022††doi:
10.1145/3477495.3531910††conference: SIGIR ’22: The 45th International ACM
SIGIR Conference on Research and Development in Information Retrieval; July
11-15, 2022; Madrid, Spain††booktitle: SIGIR ’22: The 45th International ACM
SIGIR Conference on Research and Development in Information Retrieval, July
11-15, 2022, Madrid, Spain††price: 15.00††isbn: 978-1-4503-XXXX-X/22/02

##  1\. Introduction

After the great success of the collaborative filtering algorithm (Sarwar et
al., 2001), the recommendation system has gone through another breakthrough
with leverage of sequential information (Hidasi et al., 2015; Quadrana et al.,
2018; Kang and McAuley, 2018; Sun et al., 2019). Models using RNN (Hidasi et
al., 2015) or attention mechanism (Kang and McAuley, 2018; Sun et al., 2019),
which has been successfully studied in natural language processing (Vaswani et
al., 2017; Devlin et al., 2018), recommend products that fit user taste by
understanding and analyzing the user’s interaction history as sequence
information. However, when using a recommendation system in real-life service,
a sequence encoding technique different from natural language processing is
required because user activities are performed in the unit of sessions
(Quadrana et al., 2017). As one of the research directions, session-based
recommendation, which recommends using only a relatively short history of the
current session, has been widely studied (Hidasi et al., 2015; Li et al.,
2017; Liu et al., 2018; Wu et al., 2019; Cho et al., 2021). Meanwhile, in the
case with known user history, the session-aware model is also being studied,
that is, a method of making a recommendation considering the user’s long-term
preference and short-term preference at the same time (Ma et al., 2020; Zhang
et al., 2020b; Xu et al., 2021; Latifi et al., 2021). In this case, the
sequence is grouped through hierarchy or represented into graphs (Ye et al.,
2020b; Wang et al., 2020a; Zhang et al., 2020a; Fan et al., 2021; Sun et al.,
2022; Zhang et al., 2022a). Moreover, temporal information can be induced as
part of inductive bias (Ma et al., 2020; Li et al., 2020; Cho et al., 2020) or
various user intents can be considered (Zheng et al., 2019; Ren et al., 2020;
Cen et al., 2020; Wang et al., 2020b; Wu et al., 2021; Yu et al., 2021; Zhang
et al., 2022b; Chen et al., 2022; Zang et al., 2022). However, these session-
aware methods commonly come with a rather complex model structure in order to
explicitly represent the session information.

On the other hand, in the case of the BERT model (Vaswani et al., 2017; Devlin
et al., 2018), which is having great success in natural language processing,
the model is trained with a masked language model (MLM) using a learnable mask
token. In addition, to perform the task of inferring the relationship between
two sentences (Conneau et al., 2017; Reimers and Gurevych, 2019; He et al.,
2020), the input sentences are distinguished through sentence segment
embeddings. When applying BERT in the recommendation system (Sun et al.,
2019), it is difficult to take advantage of the pretraining using MLM, but
these components still have room for utilization in recommendation models (de
Souza Pereira Moreira et al., 2021; Zhang et al., 2021a).

As another element of sequential recommendation, temporal information can be
used, which normally does not exist in other sequence modeling tasks (Xu et
al., 2019; Ye et al., 2020a; Zhang et al., 2021b). In the interaction history
of the user, not only the order of the item but also the time the interaction
was performed is important (Ye et al., 2020a). By exploiting the temporal
information, it is possible to take long-term and short-term interests into
account and analyze preferences more thoroughly. For time information, usually
expressed as unix timestamp, feature engineering is usually required due to a
unit problem (Lei et al., 2019; Wang et al., 2019), is often transformed into
triangular functions based on Bochner’s Theorem (Xu et al., 2019) so that it
can be used for vector calculation. Recently, using learnable parameters for
temporal embedding is proposed to minimize feature engineering (Fan et al.,
2021) which can utilize absolute and relative time information as needed.

In this paper, taking ideas from the aforementioned components, we propose
three methods to embed session-awareness into a model with minimum additional
model structure: using session tokens, adding session segment embeddings, and
a temporal self-attention. We conduct experiments on widely used
recommendation datasets, namely MovieLens and Steam (Kang and McAuley, 2018)
and demonstrate the feasibility of the proposed methods in terms of
recommendation performance gain compared to additional parameters required.

##  2\. Related Work

###  2.1. Sequential Recommendation

The sequential recommendation model has been developed by applying the models
that have been successful in the NLP field. RNN-based (Hidasi et al., 2015; Li
et al., 2017; Quadrana et al., 2018), CNN-based (Tang and Wang, 2018),
Attention-based (Kang and McAuley, 2018; Huang et al., 2018a; Zhang et al.,
2019; Sun et al., 2019; Wu et al., 2020; Zhang et al., [n.d.]; Han et al.,
2021; Fan et al., 2022) have been presented. Detailed model improvements have
been made to suit each environment (Huang et al., 2018b; Liu et al., 2018; Ma
et al., 2019; Cho et al., 2021; Qiu et al., 2021).

###  2.2. Session-aware Recommendation

Unlike session-based, session-aware recommendation is a method that uses the
relationship between sessions for each user and makes recommendations by
structurally decoupling long-term and short-term preferences from a slightly
more diverse perspective (Latifi et al., 2021). There are models that reflect
the hierarchical structure (Quadrana et al., 2017; Wu and Yan, 2017; Phuong et
al., 2019; Cui et al., 2019), the relationship between multiple sessions (Ren
et al., 2019; Tang et al., 2019; Zhu et al., 2020; Zhang et al., 2020b; Li et
al., 2021; Xu et al., 2021).

###  2.3. Time-aware Models

Various models have been developed to utilize temporal information with
continuous values such as unix timestamp in sequence data. There are models
that construct temporal embedding (Xu et al., 2019; Ye et al., 2020a; Zhang et
al., 2021b; Shabani et al., 2022) and apply to the real environment (Lei et
al., 2019; Cho et al., 2020; Filipovic et al., 2020), and improve sequence
encoding performance by reflecting time difference to attention (Wang et al.,
2019; Li et al., 2020; Ma et al., 2020).

Figure 1. Input layer from model architecture showing proposed methods:
session token (ST), session segment embedding (SSE), and temporal self-
attention (TAS).

##  3\. Approach

###  3.1. Background

The BERT-based sequential recommendation model is trained through the masked
language model (Devlin et al., 2018). First, as model input, a user’s
interaction history is cut to a fixed length $L$, the item embedding is set as
the token embedding, and the learnable positional embedding is added according
to the position of an item. We stack a few Transformer (Vaswani et al., 2017)
layers and predict items through the softmax classifier in the last layer. In
the training step, a randomly selected item is replaced with a mask token, and
we train the model to predict the item in the corresponding position. At the
inference step, the last-positioned item token is set to a mask token in order
to predict the next item.

###  3.2. Session Token

The simplest way to distinguish sessions in the sequence of items is to insert
a delimiter between item sequences (Ma et al., 2020). A learnable extra token
called session token (ST) is inserted between sessions as if it is an item
embedding. Unlike the padding token, it is not excluded from attention, and it
has the effect of shifting the positional embedding by one per session. The
advantage of this method is that it can indicate whether the input is a new
session or not at inference time.

###  3.3. Session Segment Embedding

Segment embedding can be used as another way to differentiate sessions. When
inferring the relationship between two sentences in natural language
processing, segment embedding is added to the input representation to
distinguish two sentences (Conneau et al., 2017). Similarly, we use learnable
session segment embedding (SSE) that indicates the ordinal position of
sessions, which can be thought of as another positional embedding that
provides a hierarchy of sequences. Note that similar to the session token,
this method can also indicate whether it is a new session or not at inference
time.

For $p$-th item $i$ in $j$-th session of a user, our input representation
becomes: $x=IE_{i}+PE_{p}+SSE_{j}$ where $IE$ is an item embedding, $PE$ is a
positional embedding from BERT, and $SSE$ is a session segment embedding. The
maximum number of sessions is limited so that only the most recent $m$
sessions are considered. As in the implementation of positional embedding,
ordinals are attached in the most recent order and padding is filled to match
the model input length $L$.

###  3.4. Temporal Self-Attention

Among many temporal encoding methods for the time-aware sequential
recommendation (Wang et al., 2019; Fan et al., 2021; Xu et al., 2019; Zhang et
al., 2021b; Cho et al., 2020; Ye et al., 2020a; Zang et al., 2022), we adopt
the learnable temporal encoding methodology from (Fan et al., 2021) and the
representation concatenation for the self-attention scheme from (Xu et al.,
2019). With this strategy, we can consider both long or short-term preference
if necessary since temporal characteristics are implicitly reflected through
training the temporal parameters, rather than explicitly engineering the unix
timestamp features (Cho et al., 2020).

For a timestamp $t$, we define a temporal encoding (TE) as follows:

(1)
$TE(t)=\left[\cos(\omega_{1}t+\theta_{1})\,\,\cos(\omega_{2}t+\theta_{2})\,\,\cdots\,\,\cos(\omega_{d_{T}}t+\theta_{d_{T}})\right]^{\top},$

where $d_{T}$ is a temporal dimension, and $\omega_{i}$, $\theta_{i}$ are
learnable parameters. We concatenate temporal encoding vectors $T$ to the
input representation $X$, which gives us a temporal self-attention (TAS) as
follows:

(2)
$TAS(X,T)=\text{softmax}\left(\frac{[X\,\,T][X\,\,T]^{\top}}{\sqrt{d_{X}+d_{T}}}\right)X,$

where $d_{X}$ is an input dimension of $X$. Here we can see that the attention
weight $a_{ij}$ between $(x_{i},t_{i})$ and $(x_{j},t_{j})$ is calculated as

(3) $a_{ij}=x_{i}^{\top}x_{j}+TE(t_{i})^{\top}TE(t_{j}),$

so that the weight becomes sum of self-attentiveness and temporal
attentiveness (Xu et al., 2019).

For multi-layered and multi-headed Transformer layers, we concatenate TE on
each layer and head. Note that TE can be trained on each layer or head
separately, but empirically no significant improvements were found.

###  3.5. Model Architecture

The input representation layer including all proposed methods is shown in
Figure 1. The rest part of the model is identical to BERT4Rec (Sun et al.,
2019). Note that the difference from SASRec (Kang and McAuley, 2018), which
uses an autoregressive decoder, is that information other than item embedding
such as positional embedding, session segment embedding, and temporal encoding
can be utilized at inference time for the to-be-predicted item.

Table 1. Dataset statistics after preprocessing. $Q_{i}$ denotes $i$-th quantile. Dataset | Steam | ML-1M | ML-20M
---|---|---|---
#users | 6,330 | 1,196 | 23,404
#items | 4,331 | 3,327 | 12,239
#rows | 49,163 | 158,498 | 1,981,866
density | 0.18% | 3.98% | 0.69%
| $Q_{1}$ | 5 | 73 | 35
item/user | $Q_{2}$ | 6 | 137 | 68
| $Q_{3}$ | 8 | 200 | 124
| $Q_{1}$ | 2 | 2 | 2
session/user | $Q_{2}$ | 2 | 2 | 2
| $Q_{3}$ | 2 | 3 | 3
| $Q_{1}$ | 2 | 8 | 6
item/session | $Q_{2}$ | 3 | 26 | 15
| $Q_{3}$ | 3 | 70 | 39

##  4\. Experiment

###  4.1. Experimental Design

####  4.1.1. Datasets and preprocessing

We evaluate our approach on three real-world recommendation datasets:
MovieLens 1M and 20M and Steam (Kang and McAuley, 2018). In the preprocessing
step, similar to (Kang and McAuley, 2018), each review or rating information
was considered as an implicit positive interaction. For the quality of the
dataset, items with an interaction count of 5 or less were removed, and users
with a history length of 5 or less were also removed. For time-related
information, unix timestamp units are used, and as in (Cho et al., 2021),
sessions are separated if there is no activity for 1-day. In order to create a
multi-session environment, the number of sessions per user is limited to 2 or
more, the number of items configured per session is 2 or more, and only 200
recent items are used. The statistics of datasets after preprocessing are
shown in Table 1. For each user, the last item was used as a test item, the
second most recent item as validation, and the rest as a training set. Note
that items in the validation were also enclosed when testing.

####  4.1.2. Evaluation metrics

As evaluation metrics, Recall and NDCG were used (He et al., 2017). For each
user, 100 negative items not in interaction history were randomly sampled and
ranked together with the ground-truth item. As is known from (Krichene and
Rendle, 2020), random negatives are biased samplers, so we additionally
construct _popular negatives_ and _all negatives_. Popular negatives take
sorted items based on popularity as negative items, which act as a kind of
“hard negatives”. In the case of all negatives, the entire item including the
user’s positive items is used as a candidate set and ranked together. In a
real-world commercial recommendation system, post-processing is often
conducted after selecting recommendation candidate items, thus all negatives
metric presumably imitates real-world performance.

####  4.1.3. Implementation details

We use PyTorch implementation of the standard BERT implementation (Sun et al.,
2019) and trained from scratch using the AdamW (Loshchilov and Hutter, 2018)
optimizer on a Titan Xp GPU. All hyper-parameters were tuned through grid
search, and we report the one with the best performance in the final result.
Hidden dimension is searched among [8, 16, 32, 64, 128, 256], mask probability
from [0.05, 0.1, 0.15, 0.2, 0.25, 0.3], number of attention head from [1, 2,
4, 8], max session segments from [2, 3, 4, 5], and temporal dimension from [8,
16, 32, 64]. Based on dataset statistics, the max sequence length is set to
200, 100, and 15 for ML-1M, ML-20M, and Steam respectively.

All source codes including model implementation, dataset preprocessing,
experiments with hyper-parameter tuning is released to the public in
https://github.com/theeluwin/session-aware-bert4rec.

(a) Max Session Segments

(b) Temporal Dimension

Figure 2. Impact of hyper-parameters on ML-1M dataset. Table 2. Recommendation performance comparison with ablation study on various evaluation metrics. Bold and underline text indicate the best and the second-best score, respectively. Dataset | Model | #params | R@10 Ran. | N@10 Ran. | R@10 Pop. | N@10 Pop. | R@10 All | N@10 All
---|---|---|---|---|---|---|---|---
Steam | POP | - | 0.7553 | 0.5214 | 0.1803 | 0.0922 | 0.1761 | 0.0897
SASRec | 5,385,708 | 0.7834 | 0.6726 | 0.5523 | 0.5007 | 0.5164 | 0.4610
BERT4Rec | 5,385,708 | 0.7987 | 0.6915 | 0.5670 | 0.5196 | 0.5313 | 0.4782
\+ ST | 5,385,964 | 0.8063 | 0.7080 | 0.6033 | 0.5602 | 0.5578 | 0.5199
\+ SSE | 5,386,988 | 0.8060 | 0.7045 | 0.5919 | 0.5482 | 0.5528 | 0.5138
\+ TSA | 5,389,996 | 0.8079 | 0.7129 | 0.5997 | 0.5562 | 0.5577 | 0.5179
\+ ST + SSE | 5,387,244 | 0.8011 | 0.7031 | 0.5978 | 0.5543 | 0.5558 | 0.5147
\+ ST + TSA | 5,390,252 |  0.8120 | 0.7093 |  0.6030 |  0.5596 | 0.5616 |  0.5187
\+ SSE + TSA | 5,391,276 | 0.8125 |  0.7103 | 0.5997 | 0.5551 | 0.5566 | 0.5181
\+ ST + SSE + TSA | 5,391,532 | 0.8093 | 0.7078 | 0.5979 | 0.5486 |  0.5581 | 0.5118
ML-1M | POP | - | 0.4030 | 0.2196 | 0.0284 | 0.0130 | 0.0151 | 0.0066
SASRec | 4,918,016 | 0.7199 | 0.4962 | 0.4189 | 0.2674 | 0.1480 |  0.0742
BERT4Rec | 4,918,016 | 0.7341 | 0.5100 | 0.5025 | 0.3211 | 0.1129 | 0.0508
\+ ST | 4,918,272 |  0.7508 | 0.5160 | 0.4900 | 0.3271 | 0.1338 | 0.0618
\+ SSE | 4,918,784 | 0.7383 | 0.5260 | 0.5109 |  0.3348 | 0.1455 | 0.0678
\+ TSA | 4,919,136 | 0.7408 | 0.5259 | 0.4933 | 0.3239 |  0.1522 | 0.0790
\+ ST + SSE | 4,919,040 | 0.7533 | 0.5230 |  0.5134 | 0.3362 | 0.1497 | 0.0739
\+ ST + TSA | 4,919,392 | 0.7441 | 0.5256 | 0.5125 | 0.3328 | 0.1564 | 0.0728
\+ SSE + TSA | 4,919,904 | 0.7492 |  0.5270 | 0.4900 | 0.3086 | 0.1513 | 0.0704
\+ ST + SSE + TSA | 4,920,160 | 0.7458 | 0.5298 | 0.5217 | 0.3283 | 0.1505 | 0.0701
ML-20M | POP | - | 0.6844 | 0.4121 | 0.0429 | 0.0219 | 0.0281 | 0.0140
SASRec | 9,464,272 | 0.9014 | 0.6954 | 0.4370 | 0.2839 | 0.1489 | 0.0807
BERT4Rec | 9,464,272 | 0.9053 | 0.6944 | 0.4729 | 0.3051 | 0.1381 | 0.0724
\+ ST | 9,464,528 | 0.9058 | 0.6958 | 0.4985 | 0.3287 | 0.1436 | 0.0750
\+ SSE | 9,465,552 | 0.9070 | 0.6975 | 0.4711 | 0.3079 | 0.1450 | 0.0740
\+ TSA | 9,481,040 |  0.9114 | 0.7010 | 0.4690 | 0.3025 | 0.1439 | 0.0754
\+ ST + SSE | 9,465,808 | 0.9069 | 0.6967 | 0.4845 | 0.3183 | 0.1418 | 0.0735
\+ ST + TSA | 9,481,296 | 0.9105 | 0.7024 |  0.4960 |  0.3258 | 0.1499 |  0.0790
\+ SSE + TSA | 9,482,320 | 0.9111 |  0.7029 | 0.4847 | 0.3202 | 0.1447 | 0.0756
\+ ST + SSE + TSA | 9,482,576 | 0.9114 | 0.7029 | 0.4921 | 0.3211 |  0.1492 | 0.0786

###  4.2. Recommendation Performance

As shown in Table 2, all of our proposed methods outperform the baseline.
Although combinations of our methods do not necessarily help the task
performance, each method encourages at least some aspects of the evaluation
metric or dataset. With proper selection of session information exploitation,
we can get significant performance gain with a very little increment of the
number of model parameters. More specifically, in the case of a session token,
the parameters are only added as much as one additional item. In the case of
session segment embedding, the parameters are increased by the number of
segments, which are not many in most cases. For the temporal embedding, only
twice as many additional parameters as the temporal dimension are used.
Altogether, only about 0.1% of additional parameters are used, which is minor
compared to the parameters that make up the transformer layers and item
embeddings. Meanwhile, performance on NDCG with all negatives is improved from
8% to 50% or more in BERT4Rec.

###  4.3. Impact of Hyper-parameters

We also conducted experiments on the impact of session-related hyper-
parameters, namely max session segments and temporal dimension, as shown in
Figure 2. The performance between popular negatives and all negatives was
expected to have a high correlation, but the experimental results show that
the optimal hyper-parameter is slightly different depending on the evaluation
metric. In the case of max session segments (Figure 2(a)), a significant
performance gain in all negatives is achieved only when the value exceeds a
certain level, as expected in Table 1. For the case of the temporal dimension
(Figure 2(b)), hyper-parameter tuning for each dataset is required, which can
be seen as a drawback as mentioned in the experimental results of (Fan et al.,
2021).

##  5\. Limitations and Future Works

In this paper, we demonstrated exploiting session information for BERT-based,
session-aware sequential recommendations through a learnable session token,
session segment embedding, and temporal self-attention. Although experiment
results show the potential of the proposed methods, there are limitations on
datasets with different statistical characteristics related to the number of
sessions per user and the number of items per session. Specifically,
additional experiments were performed on datasets including Amazon reviews,
LastFM, RetailRocket, and Diginetica, but it was difficult to acquire
performance gain, which is presumed to be due to differences in data sparsity
and session-related statistical characteristics. As an ongoing work, we will
proceed and develop novel methodologies to utilize session information more
thoroughly and consistently in session-aware sequential recommendation tasks.

## Acknowledgments

This work was made in collaboration with Seoul National University, IntelliSys
Co., Ltd. Also, this work was partly supported by Institute of Information &
communications Technology Planning & Evaluation (IITP) grant funded by the
Korean government (MSIT) [No.2021-0-00302, AI Fashion Designer: Mega-Trend and
Merchandizing Knowledge Aware AI Fashion Designer Solution], [No.2021-0-02068,
Artificial Intelligence Innovation Hub (Artificial Intelligence Institute,
Seoul National University)], [No.2021-0-01343, Artificial Intelligence
Graduate School Program (Seoul National University)].

## References

  * (1)
  * Cen et al. (2020) Yukuo Cen, Jianwei Zhang, Xu Zou, Chang Zhou, Hongxia Yang, and Jie Tang. 2020\.  Controllable multi-interest framework for recommendation. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2942–2951.
  * Chen et al. (2022) Yongjun Chen, Zhiwei Liu, Jia Li, Julian McAuley, and Caiming Xiong. 2022.  Intent Contrastive Learning for Sequential Recommendation.  _arXiv preprint arXiv:2202.02519_ (2022).
  * Cho et al. (2021) Junsu Cho, SeongKu Kang, Dongmin Hyun, and Hwanjo Yu. 2021\.  Unsupervised Proxy Selection for Session-based Recommender Systems. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 327–336.
  * Cho et al. (2020) Sung Min Cho, Eunhyeok Park, and Sungjoo Yoo. 2020.  MEANTIME: Mixture of attention mechanisms with multi-temporal embeddings for sequential recommendation. In _Fourteenth ACM Conference on Recommender Systems_. 515–520.
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017.  Supervised learning of universal sentence representations from natural language inference data.  _arXiv preprint arXiv:1705.02364_ (2017).
  * Cui et al. (2019) Qiang Cui, Shu Wu, Yan Huang, and Liang Wang. 2019\.  A hierarchical contextual attention-based network for sequential recommendation.  _Neurocomputing_ 358 (2019), 141–149.
  * de Souza Pereira Moreira et al. (2021) Gabriel de Souza Pereira Moreira, Sara Rabhi, Jeong Min Lee, Ronay Ak, and Even Oldridge. 2021\.  Transformers4Rec: Bridging the Gap between NLP and Sequential/Session-Based Recommendation. In _Fifteenth ACM Conference on Recommender Systems_. 143–153.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Fan et al. (2022) Ziwei Fan, Zhiwei Liu, Yu Wang, Alice Wang, Zahra Nazari, Lei Zheng, Hao Peng, and Philip S Yu. 2022\.  Sequential Recommendation via Stochastic Self-Attention.  _arXiv preprint arXiv:2201.06035_ (2022).
  * Fan et al. (2021) Ziwei Fan, Zhiwei Liu, Jiawei Zhang, Yun Xiong, Lei Zheng, and Philip S Yu. 2021\.  Continuous-time sequential recommendation with temporal graph collaborative transformer. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 433–442.
  * Filipovic et al. (2020) Milena Filipovic, Blagoj Mitrevski, Diego Antognini, Emma Lejal Glaude, Boi Faltings, and Claudiu Musat. 2020.  Modeling Online Behavior in Recommender Systems: The Importance of Temporal Context.  _arXiv preprint arXiv:2009.08978_ (2020).
  * Han et al. (2021) Peiyao Han, Nan Wang, Kun Li, and Xiaokun Li. 2021\.  CASR: A Collaborative Attention Model for Session-based Recommendation. In _2021 IEEE 45th Annual Computers, Software, and Applications Conference (COMPSAC)_. IEEE, 290–296.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020\.  Deberta: Decoding-enhanced bert with disentangled attention.  _arXiv preprint arXiv:2006.03654_ (2020).
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017\.  Neural collaborative filtering. In _Proceedings of the 26th international conference on world wide web_. 173–182.
  * Hidasi et al. (2015) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2015.  Session-based recommendations with recurrent neural networks.  _arXiv preprint arXiv:1511.06939_ (2015).
  * Huang et al. (2018b) Jin Huang, Wayne Xin Zhao, Hongjian Dou, Ji-Rong Wen, and Edward Y Chang. 2018b.  Improving sequential recommendation with knowledge-enhanced memory networks. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_. 505–514.
  * Huang et al. (2018a) Xiaowen Huang, Shengsheng Qian, Quan Fang, Jitao Sang, and Changsheng Xu. 2018a.  Csan: Contextual self-attention network for user sequential recommendation. In _Proceedings of the 26th ACM international conference on Multimedia_. 447–455.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _2018 IEEE International Conference on Data Mining (ICDM)_. IEEE, 197–206.
  * Krichene and Rendle (2020) Walid Krichene and Steffen Rendle. 2020.  On sampled metrics for item recommendation. In _Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining_. 1748–1757.
  * Latifi et al. (2021) Sara Latifi, Noemi Mauro, and Dietmar Jannach. 2021.  Session-aware recommendation: A surprising quest for the state-of-the-art.  _Information Sciences_ 573 (2021), 291–315.
  * Lei et al. (2019) Chenyi Lei, Shouling Ji, and Zhao Li. 2019.  Tissa: A time slice self-attention approach for modeling sequential user behaviors. In _The World Wide Web Conference_. 2964–2970.
  * Li et al. (2017) Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017\.  Neural attentive session-based recommendation. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. 1419–1428.
  * Li et al. (2020) Jiacheng Li, Yujie Wang, and Julian McAuley. 2020.  Time interval aware self-attention for sequential recommendation. In _Proceedings of the 13th international conference on web search and data mining_. 322–330.
  * Li et al. (2021) Yang Li, Tong Chen, Peng-Fei Zhang, and Hongzhi Yin. 2021\.  Lightweight self-attentive sequential recommendation. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 967–977.
  * Liu et al. (2018) Qiao Liu, Yifu Zeng, Refuoe Mokhosi, and Haibin Zhang. 2018\.  STAMP: short-term attention/memory priority model for session-based recommendation. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1831–1839.
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Fixing weight decay regularization in adam.  (2018).
  * Ma et al. (2019) Chen Ma, Peng Kang, and Xue Liu. 2019.  Hierarchical gating networks for sequential recommendation. In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_. 825–833.
  * Ma et al. (2020) Yifei Ma, Balakrishnan Narayanaswamy, Haibin Lin, and Hao Ding. 2020.  Temporal-contextual recommendation in real-time. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2291–2299.
  * Phuong et al. (2019) Tu Minh Phuong, Tran Cong Thanh, and Ngo Xuan Bach. 2019\.  Neural session-aware recommendation.  _IEEE Access_ 7 (2019), 86884–86896.
  * Qiu et al. (2021) Ruihong Qiu, Zi Huang, Tong Chen, and Hongzhi Yin. 2021\.  Exploiting positional information for session-based recommendation.  _ACM Transactions on Information Systems (TOIS)_ 40, 2 (2021), 1–24.
  * Quadrana et al. (2018) Massimo Quadrana, Paolo Cremonesi, and Dietmar Jannach. 2018\.  Sequence-aware recommender systems.  _ACM Computing Surveys (CSUR)_ 51, 4 (2018), 1–36.
  * Quadrana et al. (2017) Massimo Quadrana, Alexandros Karatzoglou, Balázs Hidasi, and Paolo Cremonesi. 2017\.  Personalizing session-based recommendations with hierarchical recurrent neural networks. In _proceedings of the Eleventh ACM Conference on Recommender Systems_. 130–137.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Ren et al. (2019) Pengjie Ren, Zhumin Chen, Jing Li, Zhaochun Ren, Jun Ma, and Maarten De Rijke. 2019\.  Repeatnet: A repeat aware neural recommendation machine for session-based recommendation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 4806–4813.
  * Ren et al. (2020) Ruiyang Ren, Zhaoyang Liu, Yaliang Li, Wayne Xin Zhao, Hui Wang, Bolin Ding, and Ji-Rong Wen. 2020.  Sequential recommendation with self-attentive multi-adversarial network. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 89–98.
  * Sarwar et al. (2001) Badrul Sarwar, George Karypis, Joseph Konstan, and John Riedl. 2001.  Item-based collaborative filtering recommendation algorithms. In _Proceedings of the 10th international conference on World Wide Web_. 285–295.
  * Shabani et al. (2022) Mostafa Shabani, Dat Thanh Tran, Martin Magris, Juho Kanniainen, and Alexandros Iosifidis. 2022\.  Multi-head Temporal Attention-Augmented Bilinear Network for Financial time series prediction.  _arXiv preprint arXiv:2201.05459_ (2022).
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In _Proceedings of the 28th ACM international conference on information and knowledge management_. 1441–1450.
  * Sun et al. (2022) Zhongchuan Sun, Bin Wu, Youwei Wang, and Yangdong Ye. 2022\.  Sequential Graph Collaborative Filtering.  _Information Sciences_ (2022).
  * Tang et al. (2019) Jiaxi Tang, Francois Belletti, Sagar Jain, Minmin Chen, Alex Beutel, Can Xu, and Ed H. Chi. 2019.  Towards neural mixture recommender for long range dependent user sequences. In _The World Wide Web Conference_. 1782–1793.
  * Tang and Wang (2018) Jiaxi Tang and Ke Wang. 2018.  Personalized top-n sequential recommendation via convolutional sequence embedding. In _Proceedings of the eleventh ACM international conference on web search and data mining_. 565–573.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2020b) Chenyang Wang, Weizhi Ma, Min Zhang, Chong Chen, Yiqun Liu, and Shaoping Ma. 2020b.  Toward dynamic user intention: Temporal evolutionary effects of item relations in sequential recommendation.  _ACM Transactions on Information Systems (TOIS)_ 39, 2 (2020), 1–33.
  * Wang et al. (2020a) Jianling Wang, Kaize Ding, Liangjie Hong, Huan Liu, and James Caverlee. 2020a.  Next-item recommendation with sequential hypergraphs. In _Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval_. 1101–1110.
  * Wang et al. (2019) Yikai Wang, Liang Zhang, Quanyu Dai, Fuchun Sun, Bo Zhang, Yang He, Weipeng Yan, and Yongjun Bao. 2019\.  Regularized adversarial sampling and deep time-aware attention for click-through rate prediction. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. 349–358.
  * Wu and Yan (2017) Chen Wu and Ming Yan. 2017\.  Session-aware information embedding for e-commerce product recommendation. In _Proceedings of the 2017 ACM on conference on information and knowledge management_. 2379–2382.
  * Wu et al. (2020) Liwei Wu, Shuqing Li, Cho-Jui Hsieh, and James Sharpnack. 2020\.  SSE-PT: Sequential recommendation via personalized transformer. In _Fourteenth ACM Conference on Recommender Systems_. 328–337.
  * Wu et al. (2019) Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019\.  Session-based recommendation with graph neural networks. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 346–353.
  * Wu et al. (2021) Yongji Wu, Lu Yin, Defu Lian, Mingyang Yin, Neil Zhenqiang Gong, Jingren Zhou, and Hongxia Yang. 2021.  Rethinking Lifelong Sequential Recommendation with Incremental Multi-Interest Attention.  _arXiv preprint arXiv:2105.14060_ (2021).
  * Xu et al. (2021) Chengfeng Xu, Jian Feng, Pengpeng Zhao, Fuzhen Zhuang, Deqing Wang, Yanchi Liu, and Victor S Sheng. 2021.  Long-and short-term self-attention network for sequential recommendation.  _Neurocomputing_ 423 (2021), 580–589.
  * Xu et al. (2019) Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. 2019.  Self-attention with functional time representation learning.  _Advances in neural information processing systems_ 32 (2019).
  * Ye et al. (2020b) Rui Ye, Qing Zhang, and Hengliang Luo. 2020b.  Cross-Session Aware Temporal Convolutional Network for Session-based Recommendation. In _2020 International Conference on Data Mining Workshops (ICDMW)_. IEEE, 220–226.
  * Ye et al. (2020a) Wenwen Ye, Shuaiqiang Wang, Xu Chen, Xuepeng Wang, Zheng Qin, and Dawei Yin. 2020a.  Time matters: Sequential recommendation with complex temporal information. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1459–1468.
  * Yu et al. (2021) Bo Yu, Ruoqian Zhang, Wei Chen, and Junhua Fang. 2021\.  Graph neural network based model for multi-behavior session-based recommendation.  _GeoInformatica_ (2021), 1–19.
  * Zang et al. (2022) Tianzi Zang, Yanmin Zhu, Jing Zhu, Yanan Xu, and Haobing Liu. 2022.  MPAN: Multi-parallel attention network for session-based recommendation.  _Neurocomputing_ 471 (2022), 230–241.
  * Zhang et al. ([n.d.]) Beichuan Zhang, Zhijiao Xiao, and Shenghua Zhong. [n.d.].  Deep Self-Attention for Sequential Recommendation.  ([n. d.]).
  * Zhang et al. (2021b) Lu-ning Zhang, Jian-wei Liu, Zhi-yan Song, and Xin Zuo. 2021b.  Temporal attention augmented transformer Hawkes process.  _Neural Computing and Applications_ (2021), 1–15.
  * Zhang et al. (2020b) Mengqi Zhang, Shu Wu, Meng Gao, Xin Jiang, Ke Xu, and Liang Wang. 2020b.  Personalized graph neural networks with attention mechanism for session-aware recommendation.  _IEEE Transactions on Knowledge and Data Engineering_ (2020).
  * Zhang et al. (2022b) Mengqi Zhang, Shu Wu, Xueli Yu, Qiang Liu, and Liang Wang. 2022b.  Dynamic graph neural networks for sequential recommendation.  _IEEE Transactions on Knowledge and Data Engineering_ (2022).
  * Zhang et al. (2022a) Qian Zhang, Shoujin Wang, Wenpeng Lu, Chong Feng, Xueping Peng, and Qingxiang Wang. 2022a.  Rethinking Adjacent Dependency in Session-based Recommendations.  _arXiv preprint arXiv:2201.12532_ (2022).
  * Zhang et al. (2019) Tingting Zhang, Pengpeng Zhao, Yanchi Liu, Victor S Sheng, Jiajie Xu, Deqing Wang, Guanfeng Liu, and Xiaofang Zhou. 2019\.  Feature-level Deeper Self-Attention Network for Sequential Recommendation.. In _IJCAI_. 4320–4326.
  * Zhang et al. (2021a) Yuhui Zhang, Hao Ding, Zeren Shui, Yifei Ma, James Zou, Anoop Deoras, and Hao Wang. 2021a.  Language Models as Recommender Systems: Evaluations and Limitations. In _I (Still) Can’t Believe It’s Not Better! NeurIPS 2021 Workshop_.
  * Zhang et al. (2020a) Yin Zhang, Yun He, Jianling Wang, and James Caverlee. 2020a.  Adaptive hierarchical translation-based sequential recommendation. In _Proceedings of The Web Conference 2020_. 2984–2990.
  * Zheng et al. (2019) Yujia Zheng, Siyi Liu, and Zailei Zhou. 2019.  Balancing multi-level interactions for session-based recommendation.  _arXiv preprint arXiv:1910.13527_ (2019).
  * Zhu et al. (2020) Jing Zhu, Yanan Xu, and Yanmin Zhu. 2020.  Modeling Long-Term and Short-Term Interests with Parallel Attentions for Session-Based Recommendation. In _International Conference on Database Systems for Advanced Applications_. Springer, 654–669.