# Investigating the Robustness of Sequential Recommender Systems Against
Training Data Perturbations: an Empirical Study

Filippo Betello  0009-0006-0945-9688 Sapienza University of RomeRomeItaly
<EMAIL_ADDRESS>,  Federico Siciliano
0000-0003-1339-6983 Sapienza University of RomeRomeItaly
<EMAIL_ADDRESS>,  Pushkar Mishra  0000-0002-1653-6198 Meta
AILondonUnited Kingdom<EMAIL_ADDRESS>and  Fabrizio Silvestri
0000-0001-7669-9055 Sapienza University of RomeRomeItaly
<EMAIL_ADDRESS>
###### Abstract.

Sequential Recommender Systems (SRSs) have been widely used to model user
behavior over time, but their robustness in the face of perturbations to
training data is a critical issue. In this paper, we conduct an empirical
study to investigate the effects of removing items at different positions
within a temporally ordered sequence. We evaluate two different SRS models on
multiple datasets, measuring their performance using Normalized Discounted
Cumulative Gain (NDCG) and Rank Sensitivity List metrics. Our results
demonstrate that removing items at the end of the sequence significantly
impacts performance, with NDCG decreasing up to 60%, while removing items from
the beginning or middle has no significant effect. These findings highlight
the importance of considering the position of the perturbed items in the
training data and shall inform the design of more robust SRSs.

Recommender systems, Sequential recommender systems, Robustness

††ccs: Information systems Recommender systems††ccs: Information systems
Recommender systems††ccs: Information systems Evaluation of retrieval
results††ccs: Computing methodologies Neural networks

##  1\. Introduction

Recommender systems have become ubiquitous in our daily lives (Adomavicius and
Tuzhilin, 2005), playing a key role in helping users navigate the vast amounts
of information available online. Thanks to the global spread of e-commerce
services, social media and streaming platforms, recommender systems have
become increasingly important for personalized content delivery and user
engagement (Zhang et al., 2019). Typically, users’ current interests are
influenced by their past interactions and are therefore often dynamic and
evolving (Wang et al., 2019b). In recent years, Sequential Recommender Systems
(SRSs) have emerged as a popular approach to modeling user behavior over time
(Quadrana et al., 2018), leveraging the temporal dependencies in users’
interaction sequences to make more accurate predictions.

To improve the effectiveness of SRSs, various techniques have been developed,
including Recurrent Neural Networks (RNNs) (Wu et al., 2017), Attention
mechanisms (Kang and McAuley, 2018), and graph-based models (Wang et al.,
2019a). These approaches have shown promising results in capturing complex
user behavior patterns and providing personalized recommendations. However,
despite their success, the robustness of SRSs against perturbations in the
training data remains an open research question(Li et al., 2021).

In this study, we investigate the impact of items removal from users’
interaction sequences on the performance of SRSs. We conduct an empirical
analysis to understand the correlation between the degree and location of
perturbation in the training data, and the resulting performance degradation.
Specifically, we temporally order the sequences of interactions for each user
and remove a variable number of items, ranging from 1 to 10, from the
beginning, middle and end of the sequence.

Experiments can be envisioned in a real-world scenario where disruptions may
occur due to users using different services for the same purpose. In such
cases, service providers may have only part of the user’s data, while the rest
is owned by their competitors. Nevertheless, the provider needs to train its
recommender system with this incomplete data set, and it wants the model to be
resilient in the face of such perturbations.

We test two widely used SRSs, SASRec (Kang and McAuley, 2018) and GRU4Rec
(Hidasi et al., 2016), on four popular benchmark datasets: MovieLens 100K
(Harper and Konstan, 2015), MovieLens 1M (Harper and Konstan, 2015),
Foursquare (Yang et al., 2016) in both Tokyo and New York versions.

We use the RecBole (Zhao et al., 2021) library to evaluate the performance of
the models, employing metrics such as Normalized Discounted Cumulative Gain
(NDCG), Precision, Recall and Mean Reciprocal Rank (MRR), and to ensure
reproducibility of the results. Moreover, in order to better evaluate the
resulting ranking lists, we make use of the Rank List Sensitivity metrics (Oh
et al., 2022).

We can summarise our contributions as follows:

  * •

Our investigation shows that the most recent items in a user’s interaction
sequence are critical for accurate recommendation performance: when these
items are removed, there is a significant drop in all the performance metrics
for both the models and across all the datasets.

  * •

Conversely, when the oldest or middle items are removed, the performance
degradation is not as pronounced.

  * •

We observe that in some cases the drop in performance is proportional to the
number of items removed.

##  2\. Related works

###  2.1. Sequential Recommender Systems

SRSs are algorithms that leverage a user’s past interactions with items to
make personalized recommendations over time. Compared to traditional
recommender systems, SRSs consider the order and timing of these interactions,
allowing for more accurate predictions of a user’s preferences and behaviors
(Wang et al., 2019b). SRSs have been widely used in various applications,
including e-commerce (Schafer et al., 2001; Hwangbo et al., 2018), social
media (Guy et al., 2010; Amato et al., 2017), and music streaming platforms
(Schedl et al., 2015, 2018; Afchar et al., 2022).

Various techniques have been proposed to implement SRSs. Initially, Markov
Chain models were employed in sequential recommendation (Fouss et al., 2005b,
a); they are probabilistic models that assume the future state of a sequence
only depends on the current state. However, these models may struggle to
capture complex dependencies in long-term sequences, and hence, have
limitations. In recent years, Recurrent Neural Networks (RNNs) have emerged as
one of the most promising approach in this field (Donkers et al., 2017; Hidasi
et al., 2016; Quadrana et al., 2017): these methods encode the users’
historical preferences into a vector that gets updated at each time step, and
it is used to predict the next item in the sequence. Despite their success,
RNNs may face challenges in dealing with long-term dependencies and generating
diverse recommendations. Another approach comes from the use of the attention
mechanism (Vaswani et al., 2017): two different examples are SASRec (Kang and
McAuley, 2018) and BERT4Rec (Sun et al., 2019) architectures. This method
dynamically weighs the importance of different parts of the sequence, to
better capture the important features and improve the prediction accuracy.
Recently, Graph Neural Networks have become popular in the field of
recommendations system (Wu et al., 2022; Purificato et al., 2023), especially
in the sequential domain (Chang et al., 2021; Fan et al., 2021).

###  2.2. Robustness in Sequential Recommender Systems

Robustness is an important aspect of SRSs as they are vulnerable to noisy and
incomplete data. Surveys on the robustness of recommender systems (Hurley,
2011; Burke et al., 2015) discussed the challenges in developing robust
recommender systems and presented various techniques for improving the
robustness of recommender systems. Robustness can be measured by Fairness and
Diversity. Fairness is one of the most critical aspects of recommender
systems: Singh _et al._ (Singh and Joachims, 2018) proposes a framework for
defining Fairness constraints on rankings, along with the effective algorithms
needed to compute utility-maximizing rankings under such Fairness constraints.
Conversely, (Li et al., 2021) analyzes this problem from the user’s point of
view, differentiates users into advantaged and disadvantaged groups, and
proposes a re-ranking method to mitigate the bias. On the other hand,
Diversity is widely discussed in (Kunaver and Požrl, 2017): the authors
present a survey on the various techniques that are been developed in the
literature.

Many tools have been developed to test the robustness of different algorithms.
(O’Mahony et al., 2004), provided both a formalisation of the topic and a
framework; the same was done more recently by (Ovaisi et al., 2022a), who
developed a toolkit called RGRecSys. The toolkit provides a unified framework
for evaluating the robustness of SRSs and can be used to compare different
algorithms under different scenarios. In Natural Language Processing (NLP),
robustness can be tested using the Robustness Gym library (Goel et al., 2021):
it unifies four standard evaluation paradigms, namely subpopulations,
transformations, evaluation sets, and adversarial attack. Inspired by this
work, the authors of (Ovaisi et al., 2022b) present a library (RGRecSys) where
it is possible to compare different recommender models under different
scenarios (a subset of interest, under distributional shift, under
transformation on user or item features, under Cross-Site Request Forgery
(CSRF) (Barth et al., 2008) and under sparse data).

Recent work has focused on studying the problem and trying to increase
robustness in different ways. (Tang et al., 2023) focuses on robustness in the
sense of training stability. Their method attempts to improve gradient descent
by combining gradient clipping and normalization. (Oh et al., 2022) instead
investigated the Rank List Sensitivity of SRSs to interaction perturbations.
They showed that even small perturbations in user-item interactions can lead
to significant changes in the recommendations, and proposed a metric to
measure the rank list sensitivity of recommender systems.

Our work expands on this, going on to analyze not only other models and
datasets, but above all making a more accurate investigation of the effect
that different types of perturbation can have on the models’ performance. In
fact, while (Oh et al., 2022) perturbs a single interaction in the whole
dataset, we perturb the sequences of all users and analyze the performance as
the number of perturbations changes.

##  3\. Methodology

###  3.1. Setting

In a Sequential Recommendation setting, each user $u$ is represented by a
temporally ordered sequence of items $S_{u}$ it has interacted with:
$S_{u}=[I_{1},I_{2},...,I_{j},...,I_{L-1},I_{L}]$, where $L$ is the length of
the sequence for user $u$. For simplicity of notation, we consider only one
user, so we use $L$, whereas it would be more correct to use $L_{u}$, as the
number of interactions varies between users.

For the training data perturbations, we have considered three different
scenarios which we will refer to in Section 4:

  * •

Remove $n$ items at the beginning of the sequence:
$[I_{n+1},I_{n+2},...,I_{L-1}]$.

  * •

Remove $n$ elements in the middle of the sequence:
$[I_{1},I_{2},...,I_{\left\lfloor\frac{L-1-n}{2}\right\rfloor},I_{\left\lfloor\frac{L-1+n}{2}\right\rfloor}...,I_{L-1}]$.

  * •

Remove $n$ elements from the end of the sequence:
$[I_{1},I_{2},...,I_{L-1-N}]$.

with $n\in\\{1,2,...,10\\}$. In practice, the data is first separated into
training, validation and test set. Subsequently, only the training data are
perturbed, with a methodology dependent on the scenario considered and the
model is then trained on these. The models, trained on data perturbed in a
different manner, are therefore always tested on the same data.

The reason why $I_{L}$ does not appear in the formulation is that, before the
training phase, $I_{L}$ is separated from the training data and assigned to
the test data. Therefore, we exclude the last element $I_{L}$ from our removal
method as it is reserved for testing the model’s performance.

###  3.2. Real-world setting

The lack of user-object interactions in real-world scenarios is a common
event, as user interactions are spread across multiple services. For example,
in the case of movies or TV shows, a particular user may watch them through
broadcast TV, in cinemas or multiple streaming services.

In our case, the three considered scenarios correspond to three specific real-
world events:

  * •

Removing $n$ items at the beginning of the sequence: This represents a user
who signs up for a service without having used it before, so all their past
interactions, i.e., those at the beginning of the complete sequence, were
performed on other services.

  * •

Removing $n$ elements in the middle of the sequence: This represents a user
who takes a break from using the service for a certain period and then resumes
using it, but any interactions they had during the considered period are not
available to the service provider in question.

  * •

Removing $n$ elements from the end of the sequence: This represents a user who
has stopped using the service, so the service provider loses all interactions
the user has from that point onward. The service provider still has an
interest in winning the user back through their own platform or other means
such as advertising, and thus, it is essential to have a robust model to
continue providing relevant items to the user.

We might assume that removing the latest interactions will lead to the most
significant decrease in performance, as those reflect the users’ current
interests most accurately. Furthermore, it is likely that items at the
beginning of a long sequence are less relevant. Removing items in the middle
of the sequence can have an impact on performance in a less easily predictable
manner. However, given a long enough sequence, the effect is expected to be
minimal.

###  3.3. Datasets

Table 1. Dataset statistics after preprocessing Dataset | Users | Items | Interactions | Average $\frac{\mathrm{Actions}}{\mathrm{User}}$ | Median $\frac{\mathrm{Actions}}{\mathrm{User}}$
---|---|---|---|---|---
MovieLens 1M | 6040 | 3952 | 1M | 165 | 96
MovieLens 100K | 943 | 1682 | 100K | 106 | 65
Foursquare Tokyo | 2293 | 61858 | 537703 | 250 | 153
Foursquare New York | 1083 | 38333 | 227428 | 210 | 173

We use four different datasets:

MovieLens (Harper and Konstan, 2015) $\rightarrow$ This benchmark dataset is
often used to test recommender systems. In this work, we use the MovieLens
100K version, which contains 100,000 interactions for 943 users and 1682
items, and MovieLens 1M version, which contains one million interactions for
6040 users and 3952 items. The average number of interactions for each user in
the MovieLens 100K dataset is 106, while for MovieLens 1M dataset is
approximately 165. The ratings in MovieLens 1M and MovieLens 100K datasets are
made on a 5-star scale.

Foursquare (Yang et al., 2014) $\rightarrow$ This dataset contains check-ins
from New York City and Tokyo collected over a period of approximately ten
months (from April 12 2012 to February 16 2013). It contains 227,428 New York
City check-ins made by more than 1080 users with about 38,000 items, and
573,703 Tokyo check-ins made by 2293 users with approximately 62,000 items.
The average number of interactions for each user in the New York City dataset
is approximately 210, while for the Tokyo dataset is approximately 250. In
this case, the interaction type is check-ins.

The statistics for all the datasets are shown in Table 1.

We select datasets widely used in the literature and with a high number of
interactions per user. The limitation in dataset selection arises from our
intention to assess the robustness against the removal of up to 10 elements.
Therefore, the dataset must satisfy the following constraint:

(1) $L_{u}>10\quad\forall u\in U$

where $L_{u}$ is the number of interactions of user $u$, i.e. the length of
the sequence $S_{u}$ of interactions, and $U$ is the set of all users in the
dataset. If the condition is not met, we would delete all items for a
particular user who has less than 10 interactions. Therefore, we could not
train the model on that particular user.

###  3.4. Architectures

In our study, we use two different architectures to validate the results.

SASRec (Kang and McAuley, 2018) $\rightarrow$ In order to comprehend the
user’s preferences, it uses self-attention processes to determine the
importance of each interaction between the user and the item. We use the model
implemented in RecBole (Zhao et al., 2021), with the default hyperparameters
of the library.

GRU4Rec (Hidasi et al., 2016) $\rightarrow$ It is a recurrent neural network
architecture that uses gated recurrent units (GRUs) (Cho et al., 2014) to
improve the accuracy of the prediction of the click-through rate (CTR).

We choose to use these two models because both have demonstrated excellent
performance in several benchmarks and have been widely cited in the
literature. Furthermore, as one model employs attention while the other
utilizes RNN, their network functioning differs, which makes evaluating their
behavior under training perturbations particularly interesting. We use the
models’ implementation provided by the RecBole Python library (Zhao et al.,
2021). The default hyperparameters are kept for each model as configured in
the library.

###  3.5. Metrics

To evaluate the performance of the models, we use the traditional evaluation
metrics used for Sequential Recommendation: Precision, Recall, MRR and NDCG.
All the metrics are calculated with a cut-off $K$ of 20.

To mitigate excessive computation, a typical practice involves randomly
selecting 100 negative items for each user and subsequently comparing the
ranking of these items with the positive one. The term positive item is used
to refer to the next item in the sequence, which is the one to be predicted.
On the other hand, the term negative item is used to describe any item that is
not present in the user’s sequence and is thus considered irrelevant to the
user.

Moreover, we decide to further investigate the stability of the recommendation
models. We borrow from (Oh et al., 2022) the Rank Sensitivity List (RLS)
metric: it compares two recommendation lists, one coming from the model which
is trained without perturbations (i.e. without removing any items) with the
one coming from the model that has been trained in adversarial conditions
(i.e. with removing items). RLS employs a similarity measure, which can be
chosen from two possible options:

  * •

Rank Bias Overlap (RBO) (Webber et al., 2010) measures the similarity of
orderings between two rank lists. Higher values indicate that the items in the
two lists are arranged similarly;

  * •

Jaccard Similarity (JAC) (Jaccard, 1912) is a normalized measure of the
similarity of the contents of two sets. A model is stable if its Jaccard score
is close to 1.

Having a model $M$, two ranked lists $X$ and $Y$, and a similarity function
$sim$ between the two ranked lists, we can formalize the RLS metric as:

(2) $\boldsymbol{\mathrm{RLS}}=\\!\frac{1}{|X|}\\!\sum_{\forall X_{k}\in
X}\\!\\!\\!\\!\text{sim}(R_{M}^{X_{k}},R_{M}^{X_{k}})\quad\boldsymbol{\mathrm{RBO(X,Y)}}\\!=\\!(1-p)\\!\sum_{d=1}^{|I|}p^{d-1}\frac{|X[1:d]\cap
Y[1:d]|}{d}\quad\boldsymbol{\mathrm{JAC(X,Y)}}=\\!\frac{|X\cap Y|}{|X\cup Y|}$

###  3.6. Experiments

All the experiments are performed on a single NVIDIA RTX 3090 with 10496 CUDA
cores and 24 GB of VRAM. The batch size is fixed to 4096. Adam optimizer is
used with a fixed learning rate of $5*10^{-4}$. The number of epochs is set to
300, but in order to avoid overfitting, we early stop the training when the
NDCG@20 does not improve for 50 epochs.

The RecBole library was utilized for conducting all the experiments,
encompassing data preprocessing, model configuration, training, and testing.
This comprehensive library ensures the reproducibility of the entire process.

##  4\. Results

###  4.1. Intrinsic Models Instability

Table 2. Variation of metrics between two seeds. Metrics for the 4 datasets considered for GRU4Rec and SASRec. For Precision (Prec.), Recall, MRR and NDCG, it is shown the percentage variation between the obtained performance using two different initialization seeds for the models. For RLS using RBO (RBO) and RLS using Jaccard (JAC), the value of the metric is shown without leading zeroes. All metrics are computed with a cut-off of 20. For each metric, the value corresponding to the dataset where a model is less robust is highlighted in bold. For each dataset, the value corresponding to the model that is the least robust of the two given a metric is underlined. | SASRec | GRU4Rec
---|---|---
| Prec. | Recall | MRR | NDCG | RBO | JAC | Prec. | Recall | MRR | NDCG | RBO | JAC
ML 100k | 0.5% | 0.5% | 0.5% | 0.3% | .466 | .489 | 1.7% | 1.7% | 1.2% | 0.1% | .337 | .413
ML 1M | 0.3% | 0.4% | 0.5% | 0.5% | .549 | .569 | 0.2% | 0.2% | 3.5% | 2.5% | .311 | .347
FS NYC | 0.9% | 0.9% | 0.0% | 0.3% | .398 | .273 | 0.1% | 0.1% | 0.6% | 0.1% | .110 | .083
FS TKY | 1.4% | 1.4% | 0.4% | 0.02% | .418 | .267 | 0.8% | 0.8% | 4.7% | 3.4% | .210 | .165

To measure the inherent robustness of the models, i.e. in the baseline case
(without removal of items), we train the model twice using different
initialization seeds and compare the rankings obtained using the percentage
variation between the two rankings. The results are shown in Table 2.

The percentage discrepancy (in absolute value) between the Precision, Recall,
MRR and NDCG values obtained using two different initialization seeds is
shown. It can be seen that in general the discrepancy is negligible: almost
always less than 1%. On the other hand, the two RLS metrics, calculated using
RBO and Jaccard respectively, show us the similarity between the two rankings
produced with different initialization seeds. It is worth recalling that a
robust model should yield a RLS score that approaches 1. However, in this
case, the values deviate significantly from this ideal threshold. These
combined results indicate to us that the models converge to an adequate
performance beyond the initialization seed, but that the actual rankings
produced are heavily influenced by it.

The bold values show for each metric, the dataset where the least robust
result was obtained. None of the datasets stands out more than the others, but
we can mention that Foursquare Tokyo seems to give more problems regarding
standard evaluation metrics (Precision, Recall, MRR and NDCG), while for
Foursquare New York City it seems more difficult to produce stable rankings
(see RLS-RBO and RLS-JAC).

The underlined values instead compare, for each metric and dataset, which of
the two models is the least robust. If we consider metrics that do not take
into account the position of the positive item in the ranking, i.e. Precision
and Recall, GRU4Rec seems more robust than SASRec. If, on the other hand, we
look at metrics that penalize relevant items in positions too low in the
ranking, we see that the opposite happens. This suggests to us that GRU4Rec is
able to return a better set of results, but in a less relevant order than
SASRec does. As proof of this, if we check the RLS metrics, we see that
GRU4Rec is always the least robust model as the initialization seed changes.

###  4.2. Comparison of the position of removal

Table 3. Variation of metrics between two seeds. Metrics for the 3 scenarios
considered for GRU4Rec on MovieLens 1M and SASRec on Foursquare Tokyo. For
Precision (Prec.), Recall, MRR and NDCG, it is shown the percentage variation
between removing ten items and the reference value. For RLS using RBO (RBO)
and RLS using Jaccard (JAC), the value of the metric is shown without leading
zeroes. All metrics are computed with a cut-off of 20. For each metric, in
bold it is highlighted the value representing the less robust model.

| SASRec - Foursquare Tokyo | GRU4Rec - MovieLens 1M
---|---|---
| Prec. | Recall | MRR | NDCG | RBO | JAC | Prec. | Recall | MRR | NDCG | RBO | JAC
Beginning | -1.33% | -1.35% | -0.09% | -1.10% | .421 | .267 | -0.22% | -0.23% | -2.44% | -1.61% | .222 | .258
Middle | -1.29% | -1.07% | -0.03% | -0.62% | .430 | .269 | -0.47% | -0.48% | -4.99% | -3.37% | .219 | .254
End | -4.89% | -4.9% | -6.99% | -3.33% | .223 | .174 | -17.4% | -17.4% | -85.5% | -56.0% | .061 | .089

The percentage variation between removing ten items and removing none
(reference value) in the training set is displayed in Table 3. We report this
value for the Precision (Prec.), Recall, MRR and NDCG metrics, all calculated
with a cut-off of 20, for the three scenarios considered. From Table 3 it can
be seen that the removal of items from the beginning or middle of the sequence
does not greatly hinder the performance of the model; only a small decrease is
observed, which may simply be due to the slightly reduced amount of total
training data. Instead, it can be observed how removing items from the end of
the sequence leads to a drastic reduction in metrics: in the case of GRU4Rec
applied to the MovieLens 1M dataset, the NDCG more than halves.

Finally, we can see how the difference between the three settings, although
maintaining the same trend, is less marked for SASRec applied to the
Foursquare Tokyo dataset. This may be due to the generally higher performance
of SASRec (Kang and McAuley, 2018). A more in-depth analysis is presented in
Section 4.4.

Table 3 also shows the two RLS metrics, computed using RBO and Jaccard
similarity, on the same model-dataset pairs They are computed by comparing the
ranking obtained when removing 10 items with those of the baseline. Table 3
shows us how removing items at the end of the sequence leads to considerable
variation in the rankings produced by the models. Especially in the case of
GRU4Rec, the values approach 0, meaning that the produced rankings share
almost no items.

Our results are in contrast to those of (Oh et al., 2022), which instead claim
that an initial perturbation of a user sequence leads to a higher impact on
the RLS. However, their experimental setting is different than ours, as
explained in 2.

###  4.3. Effect of the number of elements removed

(a) HR@20 SASRec MovieLens 100k

(b) MRR GRU4Rec MovieLens 1M

(c) RLS-JAC@20 SASRec MovieLens 100k

(d) RLS-RBO@20 GRU4Rec MovieLens 1M

Figure 1. Plots of various metrics for the MovieLens 100k and 1M datasets as
the number of removed elements increases. The value of the baseline is shown
as a horizontal line (continuous line), while the other dashed lines show the
value of the metric as the number of items removed changes, respectively at
the beginning, in the middle and at the end of the sequence.

As we discussed in the previous section, removing elements that are at the
beginning or in the middle of the temporally ordered sequence has no effect on
performance. This is also confirmed by Figure 1, where we can also see,
however, that for the above-mentioned cases there is no variation as the
number of removed elements increases. The deviation of the RLS displayed in
Figure 1(d) will be analyzed in more detail in Section 4.4. For the remaining
setting, the one where we remove items at the end of the sequence, the effect
of the number of items removed is evident: the metrics drop drastically as the
number of items removed increases. This result holds true for both models
considered and for all four datasets tested.

###  4.4. Differences between the datasets

(a) NDCG@20 SASRec MovieLens 100k

(b) NDCG@20 SASRec MovieLens 1M

(c) NDCG@20 SASRec Foursquare Tokyo

(d) NDCG@20 SASRec Foursquare New York City

(e) RBO@20 SASRec MovieLens 100k

(f) RBO@20 SASRec MovieLens 1M

(g) RBO@20 SASRec Foursquare Tokyo

(h) RBO@20 SASRec Foursquare New York City

Figure 2. Plots of the NDCG and RLS-RBO metrics for SASRec on the four
datasets. The value of the baseline is shown as a horizontal line (continuous
line), while the other dashed lines show the value of the metric as the number
of items removed changes, respectively at the beginning, in the middle and at
the end of the sequence.

Figure 2 shows the performance for the three different settings for the SASRec
model applied to all datasets. From Figures 2(a), 2(b), 2(e), 2(f) we see that
the downward trend of the metric when removing items at the end of the
sequence is a characteristic of the MovieLens dataset: both NDCG@20 and RLS-
RBO@20 show a decrease when increasing the number of removed items. We
hypothesize that this is happening because the average number of actions per
user and the number of items (see Table 1) are not that large compared to the
number of items removed. On the other hand, the Foursquare datasets (2(c),
2(d), 2(g), 2(h)) do not suffer major performance degradation, probably due to
the higher average number of actions per user and the number of items (see
Table 1) than MovieLens. In addition to this, and probably for the same
motivation, the degradation of the RLS metrics is lower with respect to that
displayed in the MovieLens datasets. Finally, it is interesting to note that
on MovieLens 1M, there is a consistent performance degradation even when
elements at the beginning and in the middle of the temporally ordered sequence
are removed. This can be observed as a small decrease in the NDCG@20 (Figure
2(b)), but a sharp decrease in the value of the RLS-RBO (Figures 1(d), 2(f)).
This means that even if the model performs approximately the same, the
rankings produced vary greatly. The cause may be the fact that the MovieLens
1M dataset, among those considered, has the largest number of users and
interactions.

##  5\. Conclusion

In this work, we have analyzed the importance of the position of items in a
temporally ordered sequence for training SRSs. Our results demonstrate the
importance of the most recent elements in users’ interactions sequence: when
these items are removed from the training data, there is a significant drop in
all evaluation metrics for all case studies investigated. Furthermore, this
reduction is proportional to the number of elements removed. Conversely, the
performance degradation is not as pronounced when the oldest elements (towards
the beginning) or the middle elements in the sequence are removed. We
validated our hypothesis using four different datasets, MovieLens, both 100K
and 1M versions, and Foursquare, both New York and Tokyo versions, and two
different models, SASRec and GRU4Rec. We used traditional evaluation metrics
such as NDCG and Recall, but also RLS, a metric specifically designed to
measure Sensitivity. Future work in this direction could first extend our
results to more models and more datasets, and then investigate a way to make
the models robust to the removal of training data. We hypothesize that the
solution may lie in using different training strategies, like random masking,
Robust loss functions (Bucarelli et al., 2023; Wani et al., 2023), or
different optimization algorithms. While we focused on Sequential Recommender
Systems, our investigation would be relevant for other approaches as well.

###### Acknowledgements.

This work was partially supported by projects FAIR (PE0000013) and SERICS
(PE00000014) under the MUR National Recovery and Resilience Plan funded by the
European Union - NextGenerationEU, by Sapienza’s project FedSSL: Federated
Self-supervised Learning with applications to Automatic Health Diagnosis and
by ERC Starting Grant No. 802554 (SPECGEO) and PRIN 2020 project n.2020TA3K9N
”LEGO.AI”. Supported also by the ERC Advanced Grant 788893 AMDROMA, EC
H2020RIA project “SoBigData++” (871042), PNRR MUR project
IR0000013-SoBigData.it.

## References

  * (1)
  * Adomavicius and Tuzhilin (2005) G. Adomavicius and A. Tuzhilin. 2005.  Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions.  _IEEE Transactions on Knowledge and Data Engineering_ 17, 6 (2005), 734–749.  https://doi.org/10.1109/TKDE.2005.99
  * Afchar et al. (2022) Darius Afchar, Alessandro Melchiorre, Markus Schedl, Romain Hennequin, Elena Epure, and Manuel Moussallam. 2022.  Explainability in music recommender systems.  _AI Magazine_ 43, 2 (2022), 190–208.
  * Amato et al. (2017) Flora Amato, Vincenzo Moscato, Antonio Picariello, and Giancarlo Sperlí. 2017.  Recommendation in social media networks. In _2017 IEEE Third International Conference on Multimedia Big Data (BigMM)_. IEEE, 213–216.
  * Barth et al. (2008) Adam Barth, Collin Jackson, and John C. Mitchell. 2008\.  Robust Defenses for Cross-Site Request Forgery. In _Proceedings of the 15th ACM Conference on Computer and Communications Security_ (Alexandria, Virginia, USA) _(CCS ’08)_. Association for Computing Machinery, New York, NY, USA, 75–88.  https://doi.org/10.1145/1455770.1455782
  * Bucarelli et al. (2023) Maria Sofia Bucarelli, Lucas Cassano, Federico Siciliano, Amin Mantrach, and Fabrizio Silvestri. 2023\.  Leveraging Inter-Rater Agreement for Classification in the Presence of Noisy Labels. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 3439–3448.
  * Burke et al. (2015) Robin Burke, Michael P O’Mahony, and Neil J Hurley. 2015\.  Robust collaborative recommendation.  _Recommender systems handbook_ (2015), 961–995.
  * Chang et al. (2021) Jianxin Chang, Chen Gao, Yu Zheng, Yiqun Hui, Yanan Niu, Yang Song, Depeng Jin, and Yong Li. 2021\.  Sequential Recommendation with Graph Neural Networks. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Virtual Event, Canada) _(SIGIR ’21)_. Association for Computing Machinery, New York, NY, USA, 378–387.  https://doi.org/10.1145/3404835.3462968
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014\.  Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation.  arXiv:1406.1078 [cs.CL]
  * Donkers et al. (2017) Tim Donkers, Benedikt Loepp, and Jürgen Ziegler. 2017\.  Sequential User-Based Recurrent Neural Network Recommendations. In _Proceedings of the Eleventh ACM Conference on Recommender Systems_ (Como, Italy) _(RecSys ’17)_. Association for Computing Machinery, New York, NY, USA, 152–160.  https://doi.org/10.1145/3109859.3109877
  * Fan et al. (2021) Ziwei Fan, Zhiwei Liu, Jiawei Zhang, Yun Xiong, Lei Zheng, and Philip S. Yu. 2021\.  Continuous-Time Sequential Recommendation with Temporal Graph Collaborative Transformer. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_ (Virtual Event, Queensland, Australia) _(CIKM ’21)_. Association for Computing Machinery, New York, NY, USA, 433–442.  https://doi.org/10.1145/3459637.3482242
  * Fouss et al. (2005a) Francois Fouss, Stephane Faulkner, Manuel Kolp, Alain Pirotte, Marco Saerens, et al. 2005a.  Web Recommendation System Based on a Markov-Chainmodel.. In _ICEIS (4)_. 56–63.
  * Fouss et al. (2005b) Francois Fouss, Alain Pirotte, and Marco Saerens. 2005b.  A novel way of computing similarities between nodes of a graph, with application to collaborative recommendation. In _The 2005 IEEE/WIC/ACM International Conference on Web Intelligence (WI’05)_. IEEE, 550–556.
  * Goel et al. (2021) Karan Goel, Nazneen Rajani, Jesse Vig, Samson Tan, Jason Wu, Stephan Zheng, Caiming Xiong, Mohit Bansal, and Christopher Ré. 2021.  Robustness Gym: Unifying the NLP Evaluation Landscape.  arXiv:2101.04840 [cs.CL]
  * Guy et al. (2010) Ido Guy, Naama Zwerdling, Inbal Ronen, David Carmel, and Erel Uziel. 2010.  Social Media Recommendation Based on People and Tags. In _Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Geneva, Switzerland) _(SIGIR ’10)_. Association for Computing Machinery, New York, NY, USA, 194–201.  https://doi.org/10.1145/1835449.1835484
  * Harper and Konstan (2015) F. Maxwell Harper and Joseph A. Konstan. 2015.  The MovieLens Datasets: History and Context.  _ACM Trans. Interact. Intell. Syst._ 5, 4, Article 19 (dec 2015), 19 pages.  https://doi.org/10.1145/2827872
  * Hidasi et al. (2016) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016.  Session-based Recommendations with Recurrent Neural Networks.  arXiv:1511.06939 [cs.LG]
  * Hurley (2011) Neil J. Hurley. 2011\.  Robustness of Recommender Systems. In _Proceedings of the Fifth ACM Conference on Recommender Systems_ (Chicago, Illinois, USA) _(RecSys ’11)_. Association for Computing Machinery, New York, NY, USA, 9–10.  https://doi.org/10.1145/2043932.2043937
  * Hwangbo et al. (2018) Hyunwoo Hwangbo, Yang Sok Kim, and Kyung Jin Cha. 2018\.  Recommendation system development for fashion retail e-commerce.  _Electronic Commerce Research and Applications_ 28 (2018), 94–101.
  * Jaccard (1912) Paul Jaccard. 1912\.  The distribution of the flora in the alpine zone. 1\.  _New Phytologist_ 11, 2 (1912), 37–50.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _2018 IEEE International Conference on Data Mining (ICDM)_. IEEE, 197–206.
  * Kunaver and Požrl (2017) Matevž Kunaver and Tomaž Požrl. 2017.  Diversity in recommender systems – A survey.  _Knowledge-Based Systems_ 123 (2017), 154–162.  https://doi.org/10.1016/j.knosys.2017.02.009
  * Li et al. (2021) Yunqi Li, Hanxiong Chen, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2021.  User-Oriented Fairness in Recommendation. In _Proceedings of the Web Conference 2021_ (Ljubljana, Slovenia) _(WWW ’21)_. Association for Computing Machinery, New York, NY, USA, 624–632.  https://doi.org/10.1145/3442381.3449866
  * Oh et al. (2022) Sejoon Oh, Berk Ustun, Julian McAuley, and Srijan Kumar. 2022\.  Rank List Sensitivity of Recommender Systems to Interaction Perturbations. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_ (Atlanta, GA, USA) _(CIKM ’22)_. Association for Computing Machinery, New York, NY, USA, 1584–1594.  https://doi.org/10.1145/3511808.3557425
  * O’Mahony et al. (2004) Michael O’Mahony, Neil Hurley, Nicholas Kushmerick, and Guénolé Silvestre. 2004\.  Collaborative recommendation: A robustness analysis.  _ACM Transactions on Internet Technology (TOIT)_ 4, 4 (2004), 344–377.
  * Ovaisi et al. (2022a) Zohreh Ovaisi, Shelby Heinecke, Jia Li, Yongfeng Zhang, Elena Zheleva, and Caiming Xiong. 2022a.  RGRecSys: A Toolkit for Robustness Evaluation of Recommender Systems. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_. 1597–1600.
  * Ovaisi et al. (2022b) Zohreh Ovaisi, Shelby Heinecke, Jia Li, Yongfeng Zhang, Elena Zheleva, and Caiming Xiong. 2022b.  RGRecSys: A Toolkit for Robustness Evaluation of Recommender Systems. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ (Virtual Event, AZ, USA) _(WSDM ’22)_. Association for Computing Machinery, New York, NY, USA, 1597–1600.  https://doi.org/10.1145/3488560.3502192
  * Purificato et al. (2023) Antonio Purificato, Giulia Cassarà, Pietro Liò, and Fabrizio Silvestri. 2023.  Sheaf Neural Networks for Graph-based Recommender Systems.  arXiv:2304.09097 [cs.IR]
  * Quadrana et al. (2018) Massimo Quadrana, Paolo Cremonesi, and Dietmar Jannach. 2018\.  Sequence-aware recommender systems.  _ACM Computing Surveys (CSUR)_ 51, 4 (2018), 1–36.
  * Quadrana et al. (2017) Massimo Quadrana, Alexandros Karatzoglou, Balázs Hidasi, and Paolo Cremonesi. 2017\.  Personalizing Session-Based Recommendations with Hierarchical Recurrent Neural Networks. In _Proceedings of the Eleventh ACM Conference on Recommender Systems_ (Como, Italy) _(RecSys ’17)_. Association for Computing Machinery, New York, NY, USA, 130–137.  https://doi.org/10.1145/3109859.3109896
  * Schafer et al. (2001) J Ben Schafer, Joseph A Konstan, and John Riedl. 2001\.  E-commerce recommendation applications.  _Data Mining and Knowledge Discovery_ 5 (2001), 115–153.
  * Schedl et al. (2015) Markus Schedl, Peter Knees, Brian McFee, Dmitry Bogdanov, and Marius Kaminskas. 2015.  Music recommender systems.  _Recommender Systems Handbook_ (2015), 453–492.
  * Schedl et al. (2018) Markus Schedl, Hamed Zamani, Ching-Wei Chen, Yashar Deldjoo, and Mehdi Elahi. 2018.  Current challenges and visions in music recommender systems research.  _International Journal of Multimedia Information Retrieval_ 7 (2018), 95–116.
  * Singh and Joachims (2018) Ashudeep Singh and Thorsten Joachims. 2018.  Fairness of Exposure in Rankings. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery Data Mining_ (London, United Kingdom) _(KDD ’18)_. Association for Computing Machinery, New York, NY, USA, 2219–2228.  https://doi.org/10.1145/3219819.3220088
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ (Beijing, China) _(CIKM ’19)_. Association for Computing Machinery, New York, NY, USA, 1441–1450.  https://doi.org/10.1145/3357384.3357895
  * Tang et al. (2023) Jiaxi Tang, Yoel Drori, Daryl Chang, Maheswaran Sathiamoorthy, Justin Gilmer, Li Wei, Xinyang Yi, Lichan Hong, and Ed H Chi. 2023.  Improving Training Stability for Multitask Ranking Models in Recommender Systems.  _arXiv preprint arXiv:2302.09178_ (2023).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All You Need. In _Proceedings of the 31st International Conference on Neural Information Processing Systems_ (Long Beach, California, USA) _(NIPS’17)_. Curran Associates Inc., Red Hook, NY, USA, 6000–6010.
  * Wang et al. (2019b) Shoujin Wang, Liang Hu, Yan Wang, Longbing Cao, Quan Z Sheng, and Mehmet Orgun. 2019b.  Sequential recommender systems: challenges, progress and prospects.  _arXiv preprint arXiv:2001.04830_ (2019).
  * Wang et al. (2019a) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019a.  Neural Graph Collaborative Filtering. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Paris, France) _(SIGIR’19)_. Association for Computing Machinery, New York, NY, USA, 165–174.  https://doi.org/10.1145/3331184.3331267
  * Wani et al. (2023) Farooq Ahmad Wani, Maria Sofia Bucarelli, and Fabrizio Silvestri. 2023.  Combining Distance to Class Centroids and Outlier Discounting for Improved Learning with Noisy Labels.  arXiv:2303.09470 [cs.LG]
  * Webber et al. (2010) William Webber, Alistair Moffat, and Justin Zobel. 2010\.  A Similarity Measure for Indefinite Rankings.  _ACM Trans. Inf. Syst._ 28, 4, Article 20 (nov 2010), 38 pages.  https://doi.org/10.1145/1852102.1852106
  * Wu et al. (2017) Chao-Yuan Wu, Amr Ahmed, Alex Beutel, Alexander J. Smola, and How Jing. 2017.  Recurrent Recommender Networks. In _Proceedings of the Tenth ACM International Conference on Web Search and Data Mining_ (Cambridge, United Kingdom) _(WSDM ’17)_. Association for Computing Machinery, New York, NY, USA, 495–503.  https://doi.org/10.1145/3018661.3018689
  * Wu et al. (2022) Shiwen Wu, Fei Sun, Wentao Zhang, Xu Xie, and Bin Cui. 2022.  Graph neural networks in recommender systems: a survey.  _Comput. Surveys_ 55, 5 (2022), 1–37.
  * Yang et al. (2016) Dingqi Yang, Daqing Zhang, and Bingqing Qu. 2016\.  Participatory cultural mapping based on collective behavior data in location-based social networks.  _ACM Transactions on Intelligent Systems and Technology (TIST)_ 7, 3 (2016), 1–23.
  * Yang et al. (2014) Dingqi Yang, Daqing Zhang, Vincent W Zheng, and Zhiyong Yu. 2014.  Modeling user activity preference by leveraging user spatial temporal characteristics in LBSNs.  _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ 45, 1 (2014), 129–142.
  * Zhang et al. (2019) Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019\.  Deep Learning Based Recommender System: A Survey and New Perspectives.  _ACM Comput. Surv._ 52, 1, Article 5 (feb 2019), 38 pages.  https://doi.org/10.1145/3285029
  * Zhao et al. (2021) Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, Yingqian Min, Zhichao Feng, Xinyan Fan, Xu Chen, Pengfei Wang, Wendi Ji, Yaliang Li, Xiaoling Wang, and Ji-Rong Wen. 2021.  RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms.  arXiv:2011.01731 [cs.IR]