# A Systematic Review and Replicability Study of BERT4Rec for Sequential
Recommendation

Aleksandr Petrov<EMAIL_ADDRESS>University of GlasgowUnited
Kingdom and  Craig Macdonald<EMAIL_ADDRESS>University of
GlasgowUnited Kingdom

(2022)

###### Abstract.

BERT4Rec is an effective model for sequential recommendation based on the
Transformer architecture. In the original publication, BERT4Rec claimed
superiority over other available sequential recommendation approaches (e.g.
SASRec), and it is now frequently being used as a state-of-the art baseline
for sequential recommendations. However, not all subsequent publications
confirmed this result and proposed other models that were shown to outperform
BERT4Rec in effectiveness. In this paper we systematically review all
publications that compare BERT4Rec with another popular Transformer-based
model, namely SASRec, and show that BERT4Rec results are not consistent within
these publications. To understand the reasons behind this inconsistency, we
analyse the available implementations of BERT4Rec and show that we fail to
reproduce results of the original BERT4Rec publication when using their
default configuration parameters. However, we are able to replicate the
reported results with the original code if training for a much longer amount
of time (up to 30x) compared to the default configuration. We also propose our
own implementation of BERT4Rec based on the Hugging Face Transformers library,
which we demonstrate replicates the originally reported results on 3 out 4
datasets, while requiring up to 95% less training time to converge. Overall,
from our systematic review and detailed experiments, we conclude that BERT4Rec
does indeed exhibit state-of-the-art effectiveness for sequential
recommendation, but only when trained for a sufficient amount of time.
Additionally, we show that our implementation can further benefit from
adapting other Transformer architectures that are available in the Hugging
Face Transformers library (e.g. using disentangled attention, as provided by
DeBERTa, or larger hidden layer size cf. ALBERT). For example, on the
MovieLens-1M dataset, we demonstrate that both these models can improve
BERT4Rec performance by up to 9%. Moreover, we show that an ALBERT-based
BERT4Rec model achieves better performance on that dataset than state-of-the-
art results reported in the most recent publications.

††journalyear: 2022††copyright: acmcopyright††conference: Sixteenth ACM
Conference on Recommender Systems; September 18–23, 2022; Seattle, WA,
USA††booktitle: Sixteenth ACM Conference on Recommender Systems (RecSys ’22),
September 18–23, 2022, Seattle, WA, USA††price: 15.00††doi:
10.1145/3523227.3548487††isbn: 978-1-4503-9278-5/22/09

##  1\. Introduction

_Sequential Recommender Systems_ are a class of recommender systems that use
the order of interactions to make recommendations. The goal of such
recommendation models is to predict the next user-item interaction by the
sequence of user’s past interactions. This goal is also known as _next item
prediction task_. While early sequential recommender systems applied Markov
Chains (Rendle et al., 2010), more recently neural networks based models have
been shown to outperform traditional models (Hidasi et al., 2016; Tang and
Wang, 2018; Yuan et al., 2019; Ma et al., 2019). Since the arrival of the
Transformer neural architecture (Vaswani et al., 2017) and, in particular the
BERT (Devlin et al., 2019) language model, Transformer-based sequential
recommendations models, such as SASRec (Kang and McAuley, 2018), S3Rec (Zhou
et al., 2020), LightSANs (Fan et al., 2021a), NOVA-BERT (Liu et al., 2021c)
and DuoRec (Qiu et al., 2022) have achieved state-of-the-art performance in
next item prediction.

Figure 1. Comparison of four implementations of BERT4Rec (Original, RecBole,
BERT4Rec-VAE-Pytorch, Ours/BERT4Rec) with two baseline models (MF-BPR, SASRec)
and two more advanced Transformer-based models (Ours/ALBERT4Rec,
Ours/DeBERTa4Rec) on the MovieLens-1M dataset. All points represent models
with their default configuration. The dashed line represents NDCG@10 of the
original BERT4Rec implementation trained for different amounts of training
time.

In particular, BERT4Rec is a highly-cited Transformer-based sequence
recommendation model proposed by Sun et al. (Sun et al., 2019), which adapts
the BERT language model, by equating tokens with items and sentences with
sequences of item interactions. Sun et al. (Sun et al., 2019) argued that, one
of the main advantages of BERT4Rec compared to other Transformer-based models,
such as SASRec, is that it uses an item masking training task (Devlin et al.,
2019) – this argument was further investigated and confirmed by Petrov and
Macdonald (Petrov and Macdonald, 2022). Applied to recommendation, the main
idea of this training task is to replace random items in the training
sequences with a special _[mask]_ token and train the model to recover these
masked tokens; we describe details of this task in Section 3.3.

In the original publication (Sun et al., 2019), BERT4Rec was claimed to
achieve significant superiority over existing neural and traditional
approaches, however subsequent publications by a number of different authors
(e.g. (Zhou et al., 2020; Yue et al., 2021; Fan et al., 2021b)) did not
confirm this superiority. Our goal therefore is to understand the reasons
behind this discrepancy. In particular, we analyse three open source
implementations of BERT4Rec with an aim to reproduce the originally reported
results using these implementations. We find that there is a marked
discrepancy in both effectiveness and efficiency of these implementations. We
also demonstrate that training original implementation of BERT4Rec with its
default configuration results in an underfitted model, and it requires up to
30x more training time in order to replicate the originally reported results.

Ultimately, we show that, with proper configuration, BERT4Rec achieves better
performance than earlier models such as SASRec, and that some results reported
in subsequent papers are based on underfitted versions of BERT4Rec. Moreover,
we show that an appropriately trained BERT4Rec can match or outperform later
models (e.g. DuoRec (Qiu et al., 2022), LightSANs (Fan et al., 2021a) & NOVA-
BERT (Liu et al., 2021c)) and therefore may still be used as a state-of-the-
art sequential recommendation model.

In addition, we propose our own implementation of BERT4Rec that is based on a
popular Hugging Face Transformers library (Wolf et al., 2019). Hugging Face
Transformers is a popular machine learning library with a large supporting
community. For instance, its GitHub repository has almost 62K stars and 15K
forks,111https://github.com/huggingface/transformers which makes it the second
most popular machine learning library on GitHub.222After Tensorflow, according
to https://github.com/EvanLi/Github-Ranking It has became a de facto standard
for publishing Transformer-based models and therefore contains well optimised
and efficient versions of Transformer-based models. Based on this and the fact
that it was already successfully applied for recommendations (de Souza Pereira
Moreira et al., 2021), we expect that using it as a backbone for building
Transformer-based recommenders should be both effective and efficient. Indeed,
we show that our implementation with default configuration replicates the
results published by Sun et al. (Sun et al., 2019) and requires up to 95% less
time to achieve these results. Moreover, the BERT architecture in our
implementation can be easily replaced with other models available in the
Hugging Face Transformers library. We demonstrate by using two examples
(DeBERTa (He et al., 2020) and ALBERT (Lan et al., 2019)) that such
replacement can lead to further improved effectiveness.

Figure 1 summarises the results of our findings on the MovieLens-1M dataset.
Each point on the figure corresponds to a default configuration of a
recommendation model and the dashed line represents performance of the
original BERT4Rec code when trained for different amounts of time. The figure
shows that depending on the chosen BERT4Rec implementation, the observed
NDCG@10 effectiveness can vary from 0.0546 for RecBole implementation to 0.156
(our implementation, 3x difference). It also shows that the required training
time also depends on the implementation. For example, the BERT4Rec-VAE
implementation requires 18 minutes to train with default configuration whereas
the original implementation requires 44 minutes while achieving poorer NDCG@10
than BERT4Rec-VAE. The figure also show that the original implementation can
reach the same level of performance as the best implementation, however it
requires much more training compared to what is specified in the default
configuration. The figure also portrays results of our implementation based on
DeBERTa (He et al., 2020) and ALBERT (Lan et al., 2019) Transformers and shows
that these models can outperform BERT4Rec, however they require more time than
our BERT4Rec implementation. Nevertheless, their training times are still 85%
smaller than what is required for the full convergence of the original
BERT4Rec implementation.

In short, the contributions of the paper are: (1) a systematic review of the
papers comparing BERT4Rec and SASRec, which shows that the results of such
comparisons are not consistent; (2) an analysis of the available
implementations of BERT4Rec, which shows that frequently these implementations
fail to reproduce results reported in (Sun et al., 2019) when trained with
their default parameters, (3) an analysis of impact of the training time to
recommendation performance of the original BERT4Rec implementation; (4) a new
Hugging Face based implementation of BERT4Rec that successfully replicates the
results reported in (Sun et al., 2019); (5) two new recommendation models:
DeBERTa4Rec and ALBERT4Rec that improve quality of our implementation via
replacing BERT model with DeBERTa and ALBERT respectively. The code and the
documentation for this paper can be found in the virtual appendix in our
GitHub repository.333https://github.com/asash/bert4rec_repro

The rest of the paper is organised as follows: Section 2 formally introduces
the sequential recommendation problem and the main approaches to solve it;
section 3 describes details of SASRec and BERT4Rec, popular Transformer-based
sequential recommendations models; Section 4 contains a systematic review of
papers comparing SASRec and BERT4Rec; Section 5 describes available BERT4Rec
implementations; Sections 6 and 7 describe experimental setup and the results
of the experiments respectively; Section 8 put this work in context, wrt.
related work and performances observed in recent publications; Section 9
summarises the results and contains final remarks.

##  2\. Background

The goal of the sequential recommendation is to predict the next interaction
in a sequence. Consider a set of users $u\in U$, where each user has made a
sequence of interactions $s\in S=\\{i_{1},i_{2},i_{3}...i_{t}\\}$, and
$i_{k}\in I$ denotes items. The task of the recommender system is to predict
next element $i_{t+1}$ in the sequence. More formally, the goal of the
sequential recommender system $M$ is to produce a list of $K$ items ranked by
the probability of being the sequence continuation:

$M(s)\rightarrow\\{i_{s_{1}},i_{s_{2}},i_{s_{3}},i_{s_{4}}...\ i_{s_{K}}\\}$

Some of the first sequential models were based on Markov Chains (Rendle et
al., 2010), however over the last few years the best models for sequential
recommendations are based on neural networks. Early neural network-based
architectures for sequential recommendations included recurrent neural
networks (Hidasi et al., 2016; Hidasi and Karatzoglou, 2018) and convolutional
neural networks (Tang and Wang, 2018; Yuan et al., 2019), however, following
the advancements in natural language processing, more recent models have
adopted the self-attention mechanism (Li et al., 2017; Liu et al., 2018) and
the Transformer architecture (Sun et al., 2019; Zhou et al., 2020; Kang and
McAuley, 2018; Petrov and Makarov, 2021). Transformer-based models
consistently outperform both traditional (non-neural) models, as well as
earlier neural architectures (Sun et al., 2019; Kang and McAuley, 2018; Zhou
et al., 2020).

Two popular Transformer-based models are SASRec (Kang and McAuley, 2018) and
BERT4rec (Sun et al., 2019). The main difference between these two models is
the training task: during training SASRec only uses past interactions to
predict future ones, whereas BERT4Rec adopts items masking, which allows to
use both past and future interactions to recover masked items. We describe the
details of these models and their training tasks in Section 3. BERT4Rec was
published one year later than SASRec, and in the original publication it uses
SASRec as a baseline. In the original publication, the authors reported
improvements over SASRec on all four datasets they used for comparison.
Indeed, for example by the NDCG@10 metric, the authors reported 14.02%
improvement on the Beauty dataset, 5.31% on the Steam dataset, 10.32% on the
MovieLens-1M dataset, and 14.47% on the MovieLens-20M dataset. However, these
results were not consistently confirmed in some subsequent publications: for
example Zhou et al. (Zhou et al., 2020) reported -5.73% NDCG@10 decrease on
Beauty dataset, Qiu et al. (Qiu et al., 2022) reported -47.01% on MovieLens-1M
and Yue et al. (Yue et al., 2021) reported -0.8% on Steam. This paper analyses
the reasons for this discrepancy. In the next section, we describe the
architectures of the SASRec and BERT4rec models.

##  3\. SASRec and BERT4Rec

###  3.1. Transformer

Both SASRec and BERT4Rec are based on the Transformer (Vaswani et al., 2017)
architecture, which was, originally designed for natural language processing.
The original Transformer architecture consists of the encoder and decoder
parts, however recommendation models use only the encoder part of a
Transformer.

Essentially, a Transformer encoder is a sequence-to-sequence model, that
translates a sequence of tokens to a sequence of vectors. It encodes the
sequence of the input tokens using an embedding layer, sums these embeddings
with positional embeddings (vectors encoding position of the token in the
sequence), and passes them through several Transformer blocks. A Transformer
block is the core component of the architecture that consists of a multi-head
self attention layer, a pointwise feed-forward layer and residual connections.
We refer to the original publication (Vaswani et al., 2017) for more details
about the architecture. In order to use this architecture, SASRec and BERT4Rec
use sequences of item ids as the inputs. Each input sequence represents a
history of interactions of a single user. An output of the Transformer encoder
is a sequence of vectors. Both SASRec and BERT4Rec project each vector from
the encoder output sequence to the score distribution over items via
multiplying these vectors by the matrix of the learnable item embeddings.
Figure 2 illustrates architectures of both models. We now describe details of
the two models and highlight their key differences.

(a) SASRec

(b) BERT4Rec

Figure 2.  Architectures of SASRec & BERT4Rec. SASRec uses casual
(unidirectional) Self Attention, whereas BERT4rec uses regular (bidirectional)
Self Attention. SASRec aims to predict the input sequence shifted right,
whereas BERT4Rec recovers masked items.

###  3.2. SASRec

Figure 2a illustrates the SASRec model. The model uses a sequence of items (a
history of interactions of a single user) as its input and predicts the same
sequence shifted by one element to the right. This means, that the last
element in the predicted sequence corresponds to the future next iteration,
which exactly matches next item prediction - the main task of the sequential
recommendation. Indeed, at the inference time, SASRec only uses the last
element in the predicted sequence.

SASRec uses the causal (unidirectional) version of self attention: to predict
element $i_{k}$ of the output sequence it can only access elements
$1,2....i_{k}$ from the input sequence. Casual self attention prevents the
model from experiencing information leakage: with regular (bidirectional) self
attention, the task would be trivial - the model would just copy element
$i_{k+1}$ from the input sequence as its $k^{th}$ output.

###  3.3. BERT4Rec

Figure 2b illustrates BERT4Rec architecture. As we can see, it is very similar
to the SASRec architecture. One important difference is that BERT4Rec uses
regular (bidirectional) self attention instead of casual attention. BERT4Rec
also uses different training task - instead of predicting the shifted
sequence, it employs an item masking training task. At the training time, some
of the items in the input sequences are replaced with a special _[mask]_
token. The goal of the model is to recover these masked tokens. Items masking
task allows to generate more training samples - up to ${n\choose k}$, where
$n$ is the sequence length and $k$ is the number of masked items - out of a
single sequence. At the inference time, BERT4Rec adds a _[mask]_ token to the
end of the sequence of interactions and then for each item in the catalogue
BERT4Rec computes probability of being this [mask] token.

Items masking is loosely connected to the next item prediction task, which
makes it harder to train. However, as shown in the original BERT4Rec
publication (Sun et al., 2019), its bidirectional nature may be advantageous
and improve the quality of the model. Nevertheless, as we show in the next
section, the question whether or not BERT4Rec is a superior model compared to
SASRec is still an open question and recent publications do not give a
definitive answer to this question.

##  4\. Systematic review of SASRec and BERT4Rec performance

The results published in the original BERT4Rec paper (Sun et al., 2019) are
indeed promising. According to the publication, BERT4Rec outperformed the
second best model, SASRec, on four different datasets (Beauty (He and McAuley,
2016), Steam (Kang and McAuley, 2018), MovieLens-1M (Harper and Konstan, 2015)
and MovieLens-20M (Harper and Konstan, 2015)) and across six different
evaluation metrics (Recall@{1, 5, 10}, NDCG@{5, 10} and MRR). In order to
validate this result and estimate its reproducibility, we perform a systematic
review of the papers that compare SASRec and BERT4Rec. We formulate two
hypotheses that we test in our systematic review:

  1. H1

Overall, BERT4Rec is not systematically better than SASRec in the published
literature.

  2. H2

Even when experimental setup is similar (e.g. experiment on the same
datasets), the outcome of the comparison of BERT4Rec and SASRec may be
different, which indicates poor replicability of BERT4Rec results.

Hypothesis H1 addresses a general question if BERT4Rec is indeed a superior
model when compared to SASRec; i.e. it outperforms SASRec across a large
number of different datasets and metrics. Hypothesis H2 addresses the question
of reproducibility; we look in particular datasets and metrics and check if
the experiment results are consistent within this experimental setup.

###  4.1. Review Methodology

To test Hypotheses H1 and H2 we conducted a review of all papers citing the
original BERT4Rec publication (Sun et al., 2019) according to the Google
Scholar website.444http://scholar.gooogle.com From these papers, we chose to
include only those papers that used both BERT4Rec and SASRec models for their
experiments, and examined their observed performances. We further excluded
papers that were not peer reviewed to minimise chances of relying on non-
verified experiments.555Other examples of excluded publications include a
paper withdrawn by authors – and only available in the web archive – and an
MSc student thesis.

We considered performing a meta analysis, by aggregating improvements across
multiple papers (e.g. “average improvement by BERT4Rec in NDCG@10 over
SASRec”). However, we observe that the identified papers applied various
evaluation measures (e.g. varying rank cutoffs) and methodology (e.g.
different negatives sampling strategies, see Section 6.4). Instead we
considered three outcomes of the experiments: ”BERT4Rec wins”, ”SASRec wins”
and ”Tie”, and relied on counting. In particular, we counted that a particular
model wins a comparison if it was better according to all metrics used in the
experiment. If a model was better according to one subset of metrics and worse
according to another, we counted a tie.

To test the overall superiority of BERT4Rec over SASRec (Hypothesis H1), we
determine the total numbers of each possible outcome (BERT4Rec wins; SASRec
wins; Tie) and check whether or not there is any considerable amount of
situations when SASRec wins over BERT4Rec. To test results reproducibility on
different datasets (Hypothesis H2), we aggregate the results by datasets and
count experiment outcomes for each dataset independently.

###  4.2. Systematic Review Results

We reviewed the 370 papers citing BERT4Rec according to Google Scholar on
25/03/2022.666The spreadsheet of analysed papers is also included in our
virtual appendix. Following the inclusion criteria (comparison with SASRec) we
found 58 publications containing such comparisons, and thereafter excluded 18
(not peer-reviewed etc.). This left a total of 40 publications that compare
BERT4Rec and SASRec, making a total with 134 total comparisons on 46 different
datasets (3.35 comparisons per paper on average, 2.91 comparisons per dataset
on average).

####  4.2.1. H1. Overall results consistency

Table 1 summarises the results of our systematic review. As we can see from
the ”Total” row of the table, BERT4Rec indeed wins the majority of the
comparisons (86 out of 134, 64%). However, the numbers of comparisons, which
SASRec won (32 out of 134, 24%) or at least achieved a tie (16 out of 134,
12%) is not negligible: 36% overall. Therefore we can conclude that BERT4Rec
is not consistently superior compared to SASRec in the published literature,
which validates Hypothesis H1 and not consistent with the original BERT4Rec
paper. We now investigate if this inconsistency can be explained by the
differences in datasets, i.e. if the reason is that BERT4Rec is better on some
datasets, but SASRec is better on others.

####  4.2.2. H2. Poor replicability of BERT4Rec

We now turn again to Table 1 and analyse the results of the systematic review
aggregated by dataset. The table includes all datasets from our review
appearing in at least 5 papers. From the table, we observe that the proportion
of outcomes is indeed highly dependent on the dataset. For example, on the
both ML-20M and Steam datasets, BERT4Rec won 7 out 8 comparisons (88%),
whereas on Sport it won just 1 out of 6 (17%) and on Toys it won 0 out of 5
(0%) comparisons. Therefore, it appears that the disparity in the overall
results can be explained by some salient characteristics of the datasets.

However there is still some level of inconsistencies within datasets. For
example, on the two most popular datasets, Beauty and ML-1M, the proportion of
experiments which BERT4Rec did not win roughly matches the overall result: 7
out of 19 for Beauty (37%) and 5 out of 13 for ML-1M (28%). Importantly, the
original BERT4Rec paper (Sun et al., 2019) found that BERT4Rec was superior by
a statistically significant margin on both of these datasets, and as we can
see a large number of papers failed to replicate this result, which validates
our Hypothesis H2.

Overall, we argue that the lack of agreement in the observed results is
problematic: it means that papers have used different versions of BERT4Rec
with different configurations and ultimately many of these papers may be using
poor configurations of BERT4Rec as their baselines. This leads us to
investigate the available open source implementations of BERT4Rec and their
hyperparameter settings.

Table 1. Results of BERT4Rec vs SASRec comparisons in the peer-reviewed publications. Bold denotes model with more wins on a dataset. Asterisk (*) denotes datasets used in the original BERT4Rec paper (Sun et al., 2019). Only the datasets with appearing in at least 5 papers are presented (8 datasets out of 46), however the comparison results on all other 38 datasets are included in the ”Total” numbers. dataset |  total |  BERT4Rec wins |  SASRec wins |  Ties |  BERT4Rec wins papers |  SASRec wins papers |  Ties papers
---|---|---|---|---|---|---|---
Beauty* (He and McAuley, 2016) | 19 |  12 (63%) |  5 (26%) |  2 (11%) |  (Sun et al., 2019), (Ma et al., 2020), (Zhou et al., 2021), (Liu et al., 2021b), (Li et al., 2021d), (Cho et al., 2020), (Zhang et al., 2020), (Tong et al., 2021), (Yue et al., 2021), (He et al., 2021), (Wu et al., 2021), (Sun et al., 2022) |  (Zhou et al., 2020), (Li et al., 2021b), (Zhang et al., 2021), (Wang et al., 2022), (Qiu et al., 2022) |  (Bian et al., 2021), (Dallmann et al., 2021)
ML-1M* (Harper and Konstan, 2015) | 18 |  13 (72%) |  3 (17%) |  2 (11%) |  (Sun et al., 2019), (Ma et al., 2020), (Zhou et al., 2021), (Li et al., 2021d), (Cho et al., 2020), (Kang et al., 2021), (Tong et al., 2021), (He et al., 2021), (Wu et al., 2021), (Song et al., 2021), (Padungkiatwattana et al., 2022), (Potter et al., 2022), (Li et al., 2022) |  (Fan et al., 2021a), (Yue et al., 2021), (Qiu et al., 2022) |  (Zhang et al., 2020), (Dallmann et al., 2021)
Yelp (Asghar, 2016) | 10 |  6 (60%) |  4 (40%) |  0 (0%) |  (Zhou et al., 2020), (Amjadi et al., 2021), (Bian et al., 2021), (Wang et al., 2022), (Qiu et al., 2022), (Padungkiatwattana et al., 2022) |  (Fan et al., 2021a), (Li et al., 2021b), (Zhang et al., 2021), (Li et al., 2021c) |
Steam* (Pathak et al., 2017) | 8 |  7 (88%) |  1 (12%) |  0 (0%) |  (Sun et al., 2019), (Ma et al., 2020), (Zhou et al., 2021), (Li et al., 2021d), (Zhang et al., 2020), (Dallmann et al., 2021), (Wu et al., 2021) |  (Yue et al., 2021) |
ML-20M* (Harper and Konstan, 2015) | 8 |  7 (88%) |  0 (0%) |  1 (12%) |  (Sun et al., 2019), (Ma et al., 2020), (Zhao et al., 2021b), (Li et al., 2021d), (Cho et al., 2020), (Wu et al., 2021), (Potter et al., 2022) |  |  (Dallmann et al., 2021)
Sports (He and McAuley, 2016) | 6 |  1 (17%) |  4 (67%) |  1 (17%) |  (Li et al., 2022) |  (Zhou et al., 2020), (Li et al., 2021b), (Zhang et al., 2021), (Qiu et al., 2022) |  (Bian et al., 2021)
LastFM (Cantador et al., 2011) | 6 |  4 (67%) |  2 (33%) |  0 (0%) |  (Zhou et al., 2020), (Kang et al., 2021), (Tong et al., 2021), (Li et al., 2022) |  (Amjadi et al., 2021), (Zhang et al., 2021) |
Toys (He and McAuley, 2016) | 5 |  0 (0%) |  5 (100%) |  0 (0%) |  |  (Zhou et al., 2020), (Fan et al., 2021b), (Li et al., 2021b), (Zhang et al., 2021), (Bian et al., 2021) |
Total | 134 |  86 (64%) | 32 (23 %) | 16 (12 %) |  |  |

##  5\. BERT4Rec implementations

To understand the reasons for the poor replicability of BERT4Rec results,
demonstrated by our systematic review, we analyse the available BERT4Rec
implementations. During our systematic review, we identified three available
BERT4Rec implementations cited in the papers: (1) the original implementation,
provided by the authors of the original paper (Sun et al., 2019); (2) RecBole
(Zhao et al., 2021a) \- a library that contains a large number of
recommendation models, including BERT4Rec (3) BERT4Rec-VAE - a project that
implements of BERT4Rec and Variational Autoencoder (Liang et al., 2018) models
using PyTorch.777https://github.com/jaywonchung/BERT4Rec-VAE-Pytorch

In addition to these three existing implementations, we note the
Transformers4Rec project (de Souza Pereira Moreira et al., 2021), which is
based on the popular Hugging Face Transformers natural language processing
library (Wolf et al., 2019). Transformers4Rec does not include an
implementation of BERT4Rec, however, inspired by this project we implement our
own version of BERT4Rec that uses Hugging Face’s version of BERT as a
backbone. Indeed, given the popularity of the Hugging Face Transformers
library, we expect that models implemented using it will be highly efficient
and may outperform other available implementations.

Salient characteristics of all four BERT4Rec implementations are listed in
Table 2. We now turn to the experiments with these four implementations, which
help us to understand poor replicability of BERT4Rec results.

Table 2. BERT4Rec implementations used in our experiments. Implementation | GitHub URL | Framework |  GitHub stars |  Example papers
---|---|---|---|---
Original | FeiSun/BERT4Rec | Tensorflow v1 | 390 | (Hu et al., 2021; Huang et al., 2021; Liu et al., 2021a)
RecBole | RUCAIBox/RecBole | PyTorch | 1,800 | (Fan et al., 2021a; Bian et al., 2021; Li et al., 2021a)
BERT4Rec-VAE | jaywonchung/BERT4Rec-VAE-Pytorch | PyTorch | 183 | (Rappaz et al., 2021; Zeng et al., 2021; Tran et al., 2022)
Ours/Hugging Face | asash/bert4rec_repro | Tensorflow v2 | N/A | N/A

##  6\. Experimental Setup

###  6.1. Research Questions

We aim to address following research questions with our experiments:

  1. RQ1

Can we replicate the state-of-the-art results reported in (Sun et al., 2019)
using all of the available BERT4Rec implementations, applying their default
configurations?

  2. RQ2

What is the effect of the training time on the performance of the original
BERT4Rec implementation?

  3. RQ3

Can our Hugging Face-based implementation of BERT4Rec benefit from replacing
BERT with another language model available in the Hugging Face Transformers
library?

###  6.2. Datasets

Because our goal is to examine replicability of the BERT4Rec, we use same four
datasets as used in the original publication (Sun et al., 2019), namely ML-1M
(Harper and Konstan, 2015), Beauty (He and McAuley, 2016), Steam (Kang and
McAuley, 2018) and ML-20M (Harper and Konstan, 2015). These datasets also
represent 4 of the 5 most popular datasets in our systematic review. To ensure
quality of the data and following the common practice (Kang and McAuley, 2018;
Rendle et al., 2010; Tang and Wang, 2018; He et al., 2017), Sun et al. (Sun et
al., 2019) discarded sequences with less than 5 interactions. In order to stay
consistent with their experimental setup, for ML-1M, Steam, and Beauty we use
the preprocessed versions of the datasets from the original BERT4Rec
repository.888https://github.com/FeiSun/BERT4Rec/tree/master/data Table 3
presents salient characteristics of our experimental datasets.

We split each dataset into train, validation and test partitions using a
Leave-One-Out strategy: for each user, we hold the final interaction for the
test set; for a validation set, we select the second last action for 2048
randomly selected users from each dataset; the remaining interactions are used
for training.

Table 3. Experimental datasets. Dataset | Users | Items | Interactions | Avg. len. | Sparsity
---|---|---|---|---|---
ML-1M | 6,040 | 3,416 | 999,611 | 165.49 | 0.9515
Steam | 281,428 | 13,044 | 3,488,885 | 12.398 | 0.9990
Beauty | 40,226 | 54,542 | 353,962 | 8.79 | 0.9998
ML-20M | 138,493 | 26,744 | 20,000,263 | 144.41 | 0.9946

###  6.3. Models

For our replicability experiment (RQ1), we use four BERT4Rec implementations,
as described in Section 5. For all four implementations, we apply their
default data prepossessing pipelines and default model parameters. Table 4
lists the default parameter values. The original BERT4Rec version includes
slightly different parameters tuned for each different datasets and we use the
ones specified for the ML-20M dataset; however according to the original
publication (Sun et al., 2019) these changes in parameters within the limits
specified in the configuration files have only a limited effect and therefore
should not change overall model performance. For the ML-1M and ML-20M datasets
we also experiment with a so-called “longer seq” version, where we increase
maximum sequence length, because the average number of items per user is much
larger in these two datasets. Overall, the configuration of our implementation
of BERT4Rec is similar to the original model, with a notable difference in the
training stopping criteria: to ensure that the models are fully trained, we
use an early stopping mechanism; we measure the value of loss function on the
validation data and stop training if validation loss did not improve for 200
epochs.

We also use two baselines in our replicability experiments: (1) MF-BPR (Rendle
et al., 2009) \- a classic matrix factorisation based approach with a pairwise
BPR loss. We use the implementation of this model from the LightFM library
(Kula, 2015) and set the number of latent components to 128; (2) SASRec (Kang
and McAuley, 2018) \- a Transformer-based model, described in Section 3.2. We
use an adaptation of the original code for this
model.999https://github.com/kang205/SASRec For SASRec we set sequence length
to 50, embedding size to 50 and use 2 transformer blocks; akkording to Kang et
al. these parameters are within the range where SASRec shows reasonable
performance.

In the extra training experiment (RQ2) we use the default configuration of the
original BERT4Rec model with the exception of number of training steps, which
we vary between 200,000 and 12,800,000.

Finally, to compare BERT with other models available in the Hugging Face
Transformers library (RQ3), we experiment with two recent Transformer-based
architectures: (1) DeBERTa (He et al., 2020) \- a model that improves BERT
with a disentangled attention mechanism (He et al., 2020) where each word is
encoded using two vectors (a vector for content and a vector for position);
(2) ALBERT (Lan et al., 2019) \- a model that improves BERT via separating the
size of the hidden state of the vocabulary embedding from the number of the
hidden layers. It also introduces cross-layer parameter sharing, which allows
to decrease overall number of parameters in the network. Following the
BERT4Rec naming convention, we call these two models DeBERTa4Rec and
ALBERT4Rec. For these two models we use same model configuration parameters as
for BERT4Rec with the exception of longer sequence length, which for which we
considered values from {50, 100, 200} and report the results for the best-
performing value 200 according to NDCG@10 metric.

###  6.4. Metrics

Following Sun et al. (Sun et al., 2019), we evaluate the models using the
Leave-One-Out strategy and focus on two ranking-based metrics: NDCG and
Recall@K on the test data.101010Original BERT4Rec publication uses name Hit
Rate (HR) instead of Recall. Hit Rate represent probability of successful
prediction of one single interaction in the next-item prediction task. In case
of sequential recommendation, when there is only one true positive item per
user, this metric is equivalent to Recall, and we prefer more conventional
name. The authors report results using _sampled metrics_ : for each positive
item they sample 100 negative items \- these 101 items are then ranked for
evaluation. In particular, they use a popularity-based sampling of negatives:
the probability of sampling an item as a negative is proportional to its
overall popularity in the dataset. However, sampled metrics are known to be
problematic. Indeed, it has been shown in a number of recent publications
(Krichene and Rendle, 2020; Cañamares and Castells, 2020; Dallmann et al.,
2021) that sampled metrics are not always consistent with full, unsampled
versions (where all items are ranked for each user) and can lead to incorrect
model comparisons. Nevertheless, because our goal is to examine the
reproducibility of the BERT4Rec model, we apply sampled metrics in to compare
our results with the results reported in the original paper. In addition,
following the recommendations in (Krichene and Rendle, 2020) and (Cañamares
and Castells, 2020), we report results on _unsampled metrics_ , where we rank
all items from the dataset at evaluation time. However, in experiments where
we do not need to compare with the results reported in the original
publication (Sun et al., 2019) we only report unsampled metrics.

Table 4. Default parameters of the BERT4Rec implementations. Implementation | Original | RecBole | BERT4Rec-VAE | Ours | Ours (longer seq)
---|---|---|---|---|---
Sequence length | 200 | 50 | 100 | 50 | 100
Training stopping criteria | 400,000 steps | 300 epochs | 200 epochs |  | Early stopping:
---
200 epochs
| Early stopping:
---
200 epochs
Item masking probability | 0.2 | 0.2 | 0.15 | 0.2 | 0.2
Embedding size | 64 | 64 | 256 | 64 | 64
Transformer blocks | 2 | 2 | 2 | 2 | 2
Attention heads | 2 | 2 | 4 | 2 | 2

###  6.5. Criteria for a Successful Replication

There is some level of randomness involved in deep neural networks training.
Indeed, even when the same model is trained multiple times, random weights
initialisation and random data shuffling can lead to small changes in the
resulting effectiveness metrics. BERT4Rec also randomly masks items during
training, which also slightly increases the randomness of the process.
Furthermore, as mentioned above, the evaluation metrics used by the original
BERT4Rec publication (Sun et al., 2019) involve sampling random items and
therefore may change even when evaluating exactly same model.

Thus, it is almost impossible to exactly replicate the reported results.
Therefore, we need to define some interval around the values reported in (Sun
et al., 2019) within which we can say that our model replicates originally
reported results. Because we only know a single measurement per dataset, we
can not rely on hypothesis testing methods, such as paired t-test, so we need
a heuristic for defining the tolerance interval.

According to Madhyastha and Jain (Madhyastha and Jain, 2019), the standard
deviation in classification metrics for instances of the same deep neural
model trained with different random seeds can reach 2.65%. If we assume that
ranking metrics, such as NDCG and Recall, exhibit similar variance under
different model instances, this suggests that a larger tolerance is needed for
defining equivalence. Hence, in this paper we define that a _model replicates
the results reported in the original publication on a metric, if the metric
value is equal to the originally reported value within a relative tolerance of
$\pm 5\%$._

##  7\. Experimental Results

###  7.1. RQ1. Replicability with default configurations.

We first analyse if the available implementations of BERT4Rec are able to
replicate the originally reported results, when trained with their default
configurations. Table 5 compares the results achieved by the models trained
with their default configuration with the results reported in the original
BERT4Rec paper (Sun et al., 2019). As we can see from the table, both versions
of our Hugging Face-based implementation replicate the reported results on 3
datasets out of 4 (all except Beauty) for the popularity-sampled Recall@10
metric and on 2 datasets out 4 for the popularity-sampled NDCG@10 metric.
Moreover, BERT4Rec-VAE replicates the originally reported results on 2
datasets for the popularity-sampled Recall@10 metrics (both versions of
MovieLens) and on one dataset for NDCG@10 metric (ML-1M). However, the
original and RecBole implementations of BERT4Rec fail to replicate the
originally reported results on all four datasets by a large margin (e.g.
original implementation is 46.47% worse according to Recall@10 metric on
Steam).

The table also reports comparison of the BERT4Rec implementations with the two
baseline recommender models. As we can see, the best version of BERT4Rec
always statistically significantly (according to the 2-tailed t-test with
Bonferoni multiple test correction, $pvalue<0.05$) outperforms both MF-BPR and
SASRec on all four datasets and on both sampled and unsampled metrics.
BERT4Rec-VAE performs better than the baselines on all four datasets, and the
Ours and Ours (longer seq) versions of BERT4Rec perform better than baselines
on three out of four datasets (except Beauty). SASRec performs better than the
original and RecBole implementations on three datasets out of four datasets
(except Beauty), which echoes the inconsistencies in our systematic review.
Further, comparisons of the RecBole and original implementations with MF-BPR
on Steam and ML-20M can lead to different conclusions if we look to sampled or
unsampled metrics. For example according to sampled Recall@10, the RecBole
implementation is 33% worse than MF-BPR on Steam dataset, however at the same
time it is 86% better than MF-BPR according to full Recall@10. This
discrepancy is in line with the results reported in (Cañamares and Castells,
2020; Krichene and Rendle, 2020; Dallmann et al., 2021) and shows importance
of using unsampled metrics in the research.

From the table, we also see that there is big discrepancy in training
times.111111We report training times on our hardware configuration: 16 (out of
32) cores of an AMD Ryzen 3975WX CPU; NVIDIA A6000 GPU; 128GB memory. For
example, on the ML-1M dataset, training BERT4Rec-VAE takes only 18 minutes,
whereas training the RecBole version takes 5.6 hours. At the same time,
BERT4Rec-VAE achieves 2.3 times better performance than RecBole, according to
unsampled Recall@10.

Unfortunately, we fail to replicate results reported in (Sun et al., 2019) for
the Beauty dataset. In attempts to achieve originally reported results, we
also ran experiments with the original implementation with a configuration
tuned for this specific dataset, tried training of the original model for up
to 16x more training time, and tried using the evaluation framework and
metrics from the original codebase. None of these attempts allowed us to reach
metric values significantly better than we obtained from the BERT4Rec-VAE
implementation, which are 22.68% worse than the reported values.

Overall, the answer to the RQ1 depends on the implementation and on the
dataset. According to popularity-sampled Recall@10, we can replicate the
originally reported results with our implementation in 3/4 cases, in 2/4 cases
using BERT4Rec-VAE and in 0/4 cases using original and RecBole
implementations. Furthermore, for unsampled metrics, we can observe similar
conclusions: BERT4Rec models that fail to replicate the originally reported
(sampled) results also performed poorly on unsampled metrics as well.

Table 5. Replicability of the originally reported BERT4Rec results

| Model | Popularity-sampled | Unsampled |  Training Time
---|---|---|---|---
| Recall@10 | NDCG@10 | Recall@10 | NDCG@10
Baselines | MF-BPR | 0.5134 (-26.34%)† | 0.2736 (-43.21%)† | 0.0740† | 0.0377† | 58
SASRec | 0.6370 (-8.61%)† | 0.4033 (-16.29%)† | 0.1993† | 0.1078† | 316
BERT4Rec versions | Original | 0.5215 (-25,18%)† | 0.3042 (-36.86%)† | 0.1518† | 0.0806† | 2,665
RecBole | 0.4562 (-34.55%)† | 0.2589† (-46.26%)† | 0.1061† | 0.0546† | 20,499
BERT4Rec-VAE | 0.6698 (-3.90%)† | 0.4533 (-5.29%)† | 0.2394† | 0.1314† | 1,085
Ours | 0.6865 (-1.51%) | 0.4602 (-4.48%) | 0.2584 | 0.1392 | 3,679
Ours (longer seq) |  missing0.6975 (+0.07%) |  missing0.4751 (-1.39%) |  0.2821 |  0.1516 | 2,889
Reported (Sun et al., 2019) | BERT4Rec | 0.6970 | 0.4818 | N/A | N/A | N/A
(a) ML-1M Dataset

| Model | Popularity-sampled | Unsampled |  Training Time
---|---|---|---|---
| Recall@10 | NDCG@10 | Recall@10 | NDCG@10
Baselines | MF-BPR | 0.3466 (-13.63%)† | 0.1842 (-18.53%)† | 0.0398† | 0.0207† | 162
SASRec | 0.3744 (-6.70%)† | 0.2052 (-9.24%)† | 0.1198† | 0.0482† | 3,614
BERT4Rec versions | Original | 0.2148 (-46.47%)† | 0.1064 (-52.94%)† | 0.0737† | 0.0375† | 4,847
RecBole | 0.2325 (-42.06%)† | 0.1177 (-47.94)† | 0.0744† | 0.0377† | 83,816
BERT4Rec-VAE | 0.3520 (-12.29%)† | 0.1941 (-14.15%)† | 0.1237† | 0.0526† | 65,303
Ours |  missing0.3978 (-0.87%) |  missing0.2219 (-1.86%) |  0.1361 |  0.0734 | 117,651
Reported (Sun et al., 2019) | BERT4Rec | 0.4013 | 0.2261 | N/A | N/A | N/A
(b) Steam Dataset

| Model | Popularity-sampled | Unsampled |  Training Time
---|---|---|---|---
| Recall@10 | NDCG@10 | Recall@10 | NDCG@10
Baselines | MF-BPR | 0.2090 (-30.91%)† | 0.1089 (-41.47%)† | 0.0185† | 0.0090† | 58
SASRec | 0.1111 (-63.27%)† | 0.0524 (-71.83%)† | 0.0079† | 0.0036† | 316
BERT4Rec versions | Original | 0.1099 (-63.67%)† | 0.0567 (-69.55%)† | 0.0163† | 0.0079† | 3,249
RecBole | 0.1996 (-34.02%)† | 0.1103 (-40.76%)† | 0.0158† | 0.0079† | 11,024
BERT4Rec-VAE |  0.2339 (-22.68%) |  0.1407 (-24.44%) |  0.0331 |  0.0188 | 21,426
Ours | 0.1891 (-37.49)† | 0.0919 (-50.64%)† | 0.0166† | 0.0080† | 14,497
Reported (Sun et al., 2019) | BERT4Rec | 0.3025 | 0.1862 | N/A | N/A | N/A
(c) Beauty Dataset

| Model | Popularity-sampled | Unsampled |  Training Time
---|---|---|---|---
| Recall@10 | NDCG@10 | Recall@10 | NDCG@10
Baselines | MF-BPR | 0.6126 (-18.02%)† | 0.3424 (-35.88%)† | 0.0807† | 0.0407† | 197
SASRec | 0.6582 (-11.92%)† | 0.4002 (-25,06%)† | 0.1439† | 0.0724† | 3635
BERT4Rec versions | Original | 0.4027 (-46.11%)† | 0.2193† (-58.93%)† | 0.0939† | 0.0474† | 6,029
RecBole | 0.4611 (-38.30%)† | 0.2589 (-51.52%)† | 0.0906† | 0.0753† | 519,666
BERT4Rec-VAE |  missing0.7409 (-0.86%) |  missing0.5259 (-1.52%) |  0.2886 |  0.1732 | 23,030
Ours | 0.7127 (-4.63%)† | 0.4805 (-10.02%)† | 0.2393† | 0.1310† | 44,610
Ours (longer seq) | 0.7268 (-2.74)† | 0.4980 (-6.74%)† | 0.2514† | 0.1456† | 39,632
Reported (Sun et al., 2019) | BERT4Rec | 0.7473 | 0.5340 | N/A | N/A | N/A
(d) ML-20M Dataset

  * •

Bold denotes a successful replication of a metric reported in (Sun et al.,
2019), percentages show difference with the reported metric, underlined
denotes the best model by a metric, † denotes a statistically significant
difference with the best model on the paired t-test with Bonferoni multiple
testing correction ($pvalue<0.05$).

###  7.2. RQ2 Effects of training time on the original BERT4Rec implementation
performance

Sun et al. (Sun et al., 2019) did not report the amount of training they
needed to reach the state-of-the-art metrics. The amount of training specified
in the original BERT4Rec code is controlled by the parameter _training steps_
, which is set to 400,000 by default. To understand the effect of training
time, we vary the number of training steps from slightly less than default
(200,000, 0.5x compared to the default) to much larger (1,280,000, 32x
compared to the default), which results in a variation of training time from
23 minutes to 21.3 hours.  Figure 3 portrays the relationship between training
time and test performance of the original BERT4Rec implementation on the ML-1M
dataset using popularity-sampled metrics.121212We use popularity-sampled
metrics, because we need to compare results with the values reported by Sun et
al. (Sun et al., 2019) As can be observed from the figure, it is possible to
reproduce the originally reported results using the original BERT4Rec
implementation, however it requires many times more training time than the
default configuration. Indeed, the model needs to be trained almost 11 hours
instead of 42 minutes to replicate results on sampled Recall@10 (15x times)
and 21 hours to replicate results on sampled NDCG@10 (30x times). Looking back
at Figure 1, the effect of training time on unsampled version of NDCG follows
the same general trend: the performance that the model reaches with the
default configuration is 51.8% lower compared to what can be reach with 30x
increased amount of training.

Overall in answer to RQ2, we find that the number of training steps set in the
default configuration of the original BERT4Rec code is too small, and to reach
performance levels reported by Sun et al. (Sun et al., 2019), the training
time has to be increased up 30x times (from less than an hour to almost a day
of training). This change makes the model much harder to train, and for
example, makes hyperparameters tuning much less feasible, specifically with
limited amount of hardware. It also makes it very easy to mistakenly use an
underfitted version of BERT4Rec in the experiments. This likely explains
inconsistencies observed in our systematic review.

(a) Effect on popularity-sampled Recall@10

(b) Effect on popularity-sampled NDCG@10

Figure 3. Effect of the training time on the performance of the original
implementation of BERT4Rec. The dashed line represents the metric value
reported in (Sun et al., 2019), filled area represents $\pm$5% interval around
the reported value, within which we count the result as a ”replication” of the
originally reported result. Default configuration corresponds to the results
with the parameters specified in the original repository for the ML-1M
dataset.

###  7.3. RQ3 Other Transformers from Hugging Face for sequential
recommendation

To answer our final research question, we compare our BERT4Rec implementation
with two other models: DeBERTa4Rec (based on the DeBERTa (Lan et al., 2019)
architecture) and ALBERT4Rec (based on the ALBERT (He et al., 2020)
architecture). Both architectures propose improvements to the BERT
architecture (some of the improvements we describe in Section 6.3), and both
models were shown to outperform BERT on the GLUE benchmark in their native
natural language processing domain. Our goal is to experiment if these
improvements hold in the domain of sequential recommendations. Our Hugging
Face based implementation makes such an experiment very simple, as in the
Transformers library these models have compatible interfaces with BERT. We
also use same early stopping mechanism as we use in our version of BERT4Rec
(see Section 6.3).

The results for these new models on the ML-1M dataset are listed in Table 6.
As we can see from the table, the both models are better outperform BERT4Rec,
however only for ALBERT4Rec are the results statistically significant (+6.48%
Recall@10, +9.23% NDCG@10). These improvements allow us to conclude that the
answer to RQ3 is positive - our implementation of BERT4Rec can further other
models available in Hugging Face Transformers library. Overall, there are more
than 100 model architectures available in the Transformers library, and many
of these models can be used seamlessly instead of BERT with our BERT4Rec
implementations.

Table 6. Comparison of the Transformer models from the Hugging Face library. All results are reported on full (unsampled) metrics. The percentage shows the difference with BERT4Rec. Bold denotes the best model by a metric,† denotes statistically significant difference compared with BERT4Rec according to a paired t-test with Bonferroni multiple testing correction ($pvalue<0.05$).  Model | Recall@10 | NDCG@10 |  Training Time
---|---|---|---
BERT4Rec | 0.282 | 0.151 | 2,889
DeBERTa4Rec | 0.290 (+3.0%) | 0.159 (+2.3%) | 12,114
ALBERT4Rec | 0.300 (+6.4%)† | 0.165 (+9.2%)† | 12,453

##  8\. Related Work & Discussion

As observed in Section 4, there has been some difficulty in replicating the
BERT4Rec reported results in (Sun et al., 2019). In terms of related work, we
highlight Dacrema et al. (Dacrema et al., 2019) who showed that deep learning-
based recommendation papers frequently used weak configurations of baselines.
They demonstrated this on an example of simpler matrix factorisation
baselines, however in this work we show, that this problem also exists with
more complicated (and effective) models. Chin et al. (Chin et al., 2022)
showed that the dataset selection can also influence the conclusion about
models performance, which echoes the results of our systematic review.
Krichene et al. (Krichene and Rendle, 2020) showed that conclusions about
model performances may change when sampled or unsampled metrics are used –
this was recently specifically confirmed for sequential recommendation
(Dallmann et al., 2021). Our experiments also confirm these results. However,
none of the aforementioned works performed a systematic review about one
popular technique, nor examined the reasons why that technique could be
difficult to replicate.

Our results for RQ1 and RQ2 in Section 5 above suggest that available BERT4Rec
implementations are the cause of this difficulty, probably due to
underfitting. Indeed, some of the recent papers that seemingly have used an
underfitted BERT4Rec as a baseline for evaluation on the ML-1M dataset are
listed in Table 7. All these papers used unsampled metrics for evaluation, so
we can compare their reported values with a fully-trained BERT4Rec results. As
we can see from the table, the difference with our fully fitted version of
BERT4Rec ranges from -10% to -53%, the results that are inline with the ones
we observe when trained BERT4Rec implementations with default configurations.
Moreover, the results reported for the best models in these papers are rather
worse than those we observe for a fully-trained BERT4Rec (GRU4Rec+ -20.59%,
LightSANs -19.03%) or only marginally better (NOVA-BERT +1.55%, DuoRec +4.43%)
models. Furthermore, as shown Section 7.3, similar or even better improvements
can be achieved by a simple replacement of one Transformer with another.
According to this table, ALBERT4Rec improves the state-of-the-art result (+2%
Recall@10 compared to DuoRec), however this improvement needs to be verified
via direct comparison with statistical significance testing, rather than
comparing reported numbers. We leave these experiments as well as experiments
on more datasets and with other Transformer architectures available in the
Hugging Face Transformers Library for future research.

Table 7. BERT4Rec results and best model results reported in the literature for the ML-1M dataset. The results are copied from the respective publications. All results are reported on full (unsampled) metrics. The percentage shows the difference with our implementation of BERT4Rec. Bold denotes the best reported BERT4Rec result and the best result overall.  Publication |  | BERT4Rec
---
Recall@10
Best model |  | Best model
---
Recall@10
This paper | 0.282 (+0.0%) | ALBERT4Rec | 0.300 (+6.4%)
Dallmann et al. (Dallmann et al., 2021) | 0.160 (-43.2%) | GRU4Rec+ | 0.224 (-20.5%)
Qiu et al. (Qiu et al., 2022) | 0.132 (-53.1%) | DuoRec | 0.294 (+4.4%)
Fan et al. (Fan et al., 2021a) | 0.221 (-21.6%) | LightSANs | 0.228 (-19.0%)
Liu et al. (Liu et al., 2021c) | 0.252 (-10.5%) | NOVA-BERT | 0.286 (+1.5%)

##  9\. Conclusions

In this paper we conducted a systematic review of the papers comparing
BERT4Rec and SASRec and found that the reported results were not consistent
among these papers. To understand the reasons of the inconsistency, we
analysed the available BERT4Rec implementations and found that in many cases
they fail to replicate originally reported results when trained with their
default parameters. Furthermore, we showed that the original implementation
requires much more training time compared to the default configuration in
order to replicate originally reported results. This gives weight to the
argument that in some cases papers have used underfitted versions of BERT4Rec
as baselines.

We also proposed our own implementation of BERT4Rec based on the Hugging Face
Transformers library, which in most of the cases replicates the originally
reported results with the default configuration parameters. We showed that our
implementation achieves similar results to the most recent sequential
recommendation models, such as NOVA-BERT and DuoRec. We also showed our
implementation can be further improved by using other architectures available
in the Hugging Face Transformers library. We believe that this paper, as well
as our openly available code, will help the researchers to use appropriately
trained baselines and will move the science in the right direction.

## References

  * (1)
  * Amjadi et al. (2021) Mehrnaz Amjadi, Seyed Danial Mohseni Taheri, and Theja Tulabandhula. 2021.  Katrec: Knowledge aware attentive sequential recommendations. In _Proc. ICDS_. 305–320.
  * Asghar (2016) Nabiha Asghar. 2016\.  Yelp dataset challenge: Review rating prediction.  _arXiv preprint arXiv:1605.05362_ (2016).
  * Bian et al. (2021) Shuqing Bian, Wayne Xin Zhao, Kun Zhou, Jing Cai, Yancheng He, Cunxiang Yin, and Ji-Rong Wen. 2021.  Contrastive Curriculum Learning for Sequential User Behavior Modeling via Data Augmentation. In _Proc. CIKM_. 3737–3746.
  * Cañamares and Castells (2020) Rocío Cañamares and Pablo Castells. 2020.  On target item sampling in offline recommender system evaluation. In _Proc. RecSys_. 259–268.
  * Cantador et al. (2011) Iván Cantador, Peter Brusilovsky, and Tsvi Kuflik. 2011\.  2nd Workshop on Information Heterogeneity and Fusion in Recommender Systems (HetRec 2011). In _Proc. RecSys_ (Chicago, IL, USA) _(RecSys 2011)_. New York, NY, USA.
  * Chin et al. (2022) Jin Yao Chin, Yile Chen, and Gao Cong. 2022.  The Datasets Dilemma: How Much Do We Really Know About Recommendation Datasets?. In _Proc. WSDM_. 141–149.
  * Cho et al. (2020) Sung Min Cho, Eunhyeok Park, and Sungjoo Yoo. 2020.  MEANTIME: Mixture of attention mechanisms with multi-temporal embeddings for sequential recommendation. In _Proc. RecSys_. 515–520.
  * Dacrema et al. (2019) Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach. 2019\.  Are we really making much progress? A worrying analysis of recent neural recommendation approaches. In _Proc. RecSys_. 101–109.
  * Dallmann et al. (2021) Alexander Dallmann, Daniel Zoller, and Andreas Hotho. 2021\.  A Case Study on Sampling Strategies for Evaluating Neural Sequential Item Recommendation Models. In _Proc. RecSys_. 505–514.
  * de Souza Pereira Moreira et al. (2021) Gabriel de Souza Pereira Moreira, Sara Rabhi, Jeong Min Lee, Ronay Ak, and Even Oldridge. 2021\.  Transformers4Rec: Bridging the Gap between NLP and Sequential/Session-Based Recommendation. In _Proc. RecSys_. 143–153.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proc. NAACL-HLT_. 4171–4186.
  * Fan et al. (2021a) Xinyan Fan, Zheng Liu, Jianxun Lian, Wayne Xin Zhao, Xing Xie, and Ji-Rong Wen. 2021a.  Lighter and better: low-rank decomposed self-attention networks for next-item recommendation. In _Proc. SIGIR_. 1733–1737.
  * Fan et al. (2021b) Ziwei Fan, Zhiwei Liu, Jiawei Zhang, Yun Xiong, Lei Zheng, and Philip S Yu. 2021b.  Continuous-time sequential recommendation with temporal graph collaborative transformer. In _Proc. CIKM_. 433–442.
  * Harper and Konstan (2015) F Maxwell Harper and Joseph A Konstan. 2015.  The MovieLens datasets: History and context.  _ACM Transactions on Interactive Intelligent Systems (TIIS)_ 5, 4 (2015), 1–19.
  * He et al. (2020) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020\.  DeBERTa: Decoding-enhanced bert with disentangled attention. In _Proc. ICLR_.
  * He and McAuley (2016) Ruining He and Julian McAuley. 2016.  Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In _Proc. WWW_. 507–517.
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017\.  Neural collaborative filtering. In _Proc. WWW_. 173–182.
  * He et al. (2021) Zhankui He, Handong Zhao, Zhe Lin, Zhaowen Wang, Ajinkya Kale, and Julian Mcauley. 2021\.  Locker: Locally Constrained Self-Attentive Sequential Recommendation. In _Proc. CIKM_. 3088–3092.
  * Hidasi and Karatzoglou (2018) Balázs Hidasi and Alexandros Karatzoglou. 2018.  Recurrent neural networks with top-k gains for session-based recommendations. In _Proc. CIKM_. 843–852.
  * Hidasi et al. (2016) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016.  Session-based Recommendations with Recurrent Neural Networks. In _Proc. ICLR_.
  * Hu et al. (2021) Kaixi Hu, Lin Li, Qing Xie, Jianquan Liu, and Xiaohui Tao. 2021.  What is Next when Sequential Prediction Meets Implicitly Hard Interaction?. In _Proc. CIKM_. 710–719.
  * Huang et al. (2021) Liwei Huang, Yutao Ma, Yanbo Liu, Bohong Danny Du, Shuliang Wang, and Deyi Li. 2021\.  Position-enhanced and Time-aware Graph Convolutional Network for Sequential Recommendations.  _ACM Transactions on Information Systems (TOIS)_ (2021).
  * Kang et al. (2021) Taegwan Kang, Hwanhee Lee, Byeongjin Choe, and Kyomin Jung. 2021\.  Entangled bidirectional encoder to autoregressive decoder for sequential recommendation. In _Proc. SIGIR_. 1657–1661.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _Proc. ICDM_. 197–206.
  * Krichene and Rendle (2020) Walid Krichene and Steffen Rendle. 2020.  On sampled metrics for item recommendation. In _Proc. KDD_. 1748–1757.
  * Kula (2015) Maciej Kula. 2015\.  Metadata Embeddings for User and Item Cold-start Recommendations. In _Proc. Workshop on New Trends on Content-Based Recommender @ RecSys_ _(CEUR Workshop Proc., Vol. 1448)_. 14–21.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019\.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In _Proc. ICLR_.
  * Li et al. (2022) Chenglin Li, Mingjun Zhao, Huanming Zhang, Chenyun Yu, Lei Cheng, Guoqiang Shu, BeiBei Kong, and Di Niu. 2022\.  RecGURU: Adversarial Learning of Generalized User Representations for Cross-Domain Recommendation.  (2022), 571–581.
  * Li et al. (2021d) Haoyang Li, Xin Wang, Ziwei Zhang, Jianxin Ma, Peng Cui, and Wenwu Zhu. 2021d.  Intention-aware sequential recommendation with structured intent transition.  _IEEE Transactions on Knowledge and Data Engineering (TKDE)_ (2021).
  * Li et al. (2017) Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017\.  Neural attentive session-based recommendation. In _Proc. CIKM_. 1419–1428.
  * Li et al. (2021a) Yicong Li, Hongxu Chen, Xiangguo Sun, Zhenchao Sun, Lin Li, Lizhen Cui, Philip S Yu, and Guandong Xu. 2021a.  Hyperbolic hypergraphs for sequential recommendation. In _Proc. CIKM_. 988–997.
  * Li et al. (2021b) Yang Li, Tong Chen, Peng-Fei Zhang, and Hongzhi Yin. 2021b.  Lightweight self-attentive sequential recommendation. In _Proc. CIKM_. 967–977.
  * Li et al. (2021c) Yunzhe Li, Yue Ding, Bo Chen, Xin Xin, Yule Wang, Yuxiang Shi, Ruiming Tang, and Dong Wang. 2021c.  Extracting Attentive Social Temporal Excitation for Sequential Recommendation. In _Proc. CIKM_. 998–1007.
  * Liang et al. (2018) Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018.  Variational autoencoders for collaborative filtering. In _Proc. WWW_. 689–698.
  * Liu et al. (2021c) Chang Liu, Xiaoguang Li, Guohao Cai, Zhenhua Dong, Hong Zhu, and Lifeng Shang. 2021c.  Noninvasive Self-attention for Side Information Fusion in Sequential Recommendation. In _Proc. AAAI_. 4249–4256.
  * Liu et al. (2018) Qiao Liu, Yifu Zeng, Refuoe Mokhosi, and Haibin Zhang. 2018\.  STAMP: short-term attention/memory priority model for session-based recommendation. In _Proc. KDD_. 1831–1839.
  * Liu et al. (2021a) Zhiwei Liu, Yongjun Chen, Jia Li, Philip S Yu, Julian McAuley, and Caiming Xiong. 2021a.  Contrastive self-supervised sequential recommendation with robust augmentation.  _arXiv preprint arXiv:2108.06479_ (2021).
  * Liu et al. (2021b) Zhiwei Liu, Ziwei Fan, Yu Wang, and Philip S. Yu. 2021b.  Augmenting Sequential Recommendation with Pseudo-Prior Items via Reversely Pre-training Transformer. In _Proc. SIGIR_. 1608–1612.
  * Ma et al. (2019) Chen Ma, Peng Kang, and Xue Liu. 2019.  Hierarchical gating networks for sequential recommendation. In _Proc. KDD_. 825–833.
  * Ma et al. (2020) Jianxin Ma, Chang Zhou, Hongxia Yang, Peng Cui, Xin Wang, and Wenwu Zhu. 2020\.  Disentangled self-supervision in sequential recommenders. In _Proc. KDD_. 483–491.
  * Madhyastha and Jain (2019) Pranava Madhyastha and Rishabh Jain. 2019.  On model stability as a function of random seed.  _arXiv preprint arXiv:1909.10447_ (2019).
  * Padungkiatwattana et al. (2022) Umaporn Padungkiatwattana, Thitiya Sae-Diae, Saranya Maneeroj, and Atsuhiro Takasu. 2022.  ARERec: Attentive Local Interaction Model for Sequential Recommendation.  _IEEE Access_.
  * Pathak et al. (2017) Apurva Pathak, Kshitiz Gupta, and Julian McAuley. 2017\.  Generating and personalizing bundle recommendations on steam. In _Proc. SIGIR_. 1073–1076.
  * Petrov and Macdonald (2022) Aleksandr Petrov and Craig Macdonald. 2022.  Effective and Efficient Training for Sequential Recommendation using Recency Sampling.  _Proc. RecSys_ (2022).
  * Petrov and Makarov (2021) Aleksandr Petrov and Yuriy Makarov. 2021.  Attention-based neural re-ranking approach for next city in trip recommendations. In _Proc. WSDM WebTour_. 41–45.
  * Potter et al. (2022) Michael Potter, Hamlin Liu, Yash Lala, Christian Loanzon, and Yizhou Sun. 2022.  GRU4RecBE: A Hybrid Session-Based Movie Recommendation System (Student Abstract).  (2022).
  * Qiu et al. (2022) Ruihong Qiu, Zi Huang, Hongzhi Yin, and Zijian Wang. 2022\.  Contrastive learning for representation degeneration problem in sequential recommendation. In _Proc. WSDM_. 813–823.
  * Rappaz et al. (2021) Jérémie Rappaz, Julian McAuley, and Karl Aberer. 2021\.  Recommendation on Live-Streaming Platforms: Dynamic Availability and Repeat Consumption. In _Proc. RecSys_. 390–399.
  * Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009.  BPR: Bayesian personalized ranking from implicit feedback. In _Proc. CUAI_. 452–461.
  * Rendle et al. (2010) Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010\.  Factorizing personalized markov chains for next-basket recommendation. In _Proc. WWW_. 811–820.
  * Song et al. (2021) Zihan Song, Jiahao Yuan, Xiaoling Wang, and Wendi Ji. 2021\.  Capturing Multi-granularity Interests with Capsule Attentive Network for Sequential Recommendation. In _Proc. WISE_. 147–161.
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In _Proc. CIKM_. 1441–1450.
  * Sun et al. (2022) Zhongchuan Sun, Bin Wu, Youwei Wang, and Yangdong Ye. 2022\.  Sequential graph collaborative filtering.  _Information Sciences_ 592 (2022), 244–260.
  * Tang and Wang (2018) Jiaxi Tang and Ke Wang. 2018.  Personalized top-n sequential recommendation via convolutional sequence embedding. In _Proc. WSDM_. 565–573.
  * Tong et al. (2021) Xiaohai Tong, Pengfei Wang, Chenliang Li, Long Xia, and Shaozhang Niu. 2021.  Pattern-enhanced Contrastive Policy Learning Network for Sequential Recommendation. In _IJCAI_.
  * Tran et al. (2022) Quyen Tran, Lam Tran, Linh Chu Hai, Ngo Van Linh, and Khoat Than. 2022.  From Implicit to Explicit feedback: A deep neural network for modeling sequential behaviours and long-short term preferences of online users.  _Neurocomputing_ (2022).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Proc. NeurIPS_. 5998–6008.
  * Wang et al. (2022) Chenyang Wang, Weizhi Ma, and Chong Chen. 2022.  Sequential Recommendation with Multiple Contrast Signals.  _ACM Transactions on Information Systems (TOIS)_ (2022).
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019\.  Huggingface’s transformers: State-of-the-art natural language processing.  _arXiv preprint arXiv:1910.03771_ (2019).
  * Wu et al. (2021) Qitian Wu, Chenxiao Yang, Shuodian Yu, Xiaofeng Gao, and Guihai Chen. 2021.  Seq2Bubbles: Region-Based Embedding Learning for User Behaviors in Sequential Recommenders. In _Proc. CIKM_. 2160–2169.
  * Yuan et al. (2019) Fajie Yuan, Alexandros Karatzoglou, Ioannis Arapakis, Joemon M Jose, and Xiangnan He. 2019.  A simple convolutional generative network for next item recommendation. In _Proc. WSDM_. 582–590.
  * Yue et al. (2021) Zhenrui Yue, Zhankui He, Huimin Zeng, and Julian McAuley. 2021\.  Black-Box Attacks on Sequential Recommenders via Data-Free Model Extraction. In _Proc. RecSys_.
  * Zeng et al. (2021) Zheni Zeng, Chaojun Xiao, Yuan Yao, Ruobing Xie, Zhiyuan Liu, Fen Lin, Leyu Lin, and Maosong Sun. 2021\.  Knowledge transfer via pre-training for recommendation: A review and prospect.  _Frontiers in big Data_ (2021), 4.
  * Zhang et al. (2020) Lingxiao Zhang, Jiangpeng Yan, Yujiu Yang, and Li Xiu. 2020\.  Match4rec: A novel recommendation algorithm based on bidirectional encoder representation with the matching task. In _Proc. ICONIP_. Springer, 491–503.
  * Zhang et al. (2021) Yixin Zhang, Lizhen Cui, Wei He, Xudong Lu, and Shipeng Wang. 2021.  Behavioral data assists decisions: exploring the mental representation of digital-self.  _International Journal of Crowd Science_ (2021).
  * Zhao et al. (2021b) Pengyu Zhao, Tianxiao Shui, Yuanxing Zhang, Kecheng Xiao, and Kaigui Bian. 2021b.  Adversarial oracular seq2seq learning for sequential recommendation. In _Proc. ICJAI_.
  * Zhao et al. (2021a) Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, et al. 2021a.  Recbole: Towards a unified, comprehensive and efficient framework for recommendation algorithms. In _Proc. CIKM_. 4653–4664.
  * Zhou et al. (2021) Chang Zhou, Jianxin Ma, Jianwei Zhang, Jingren Zhou, and Hongxia Yang. 2021.  Contrastive learning for debiased candidate generation in large-scale recommender systems. In _Proc. KDD_. 3985–3995.
  * Zhou et al. (2020) Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. 2020\.  S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In _Proc. CIKM_. 1893–1902.


# Tile Computer-on-Module for the ATLAS Tile Calorimeter Phase-II upgrades

M G D Gololo,11footnotetext: Institute for Collider Particle Physics,
University of the Witwatersrand; F Carrió Argos, 22footnotetext: Instituto de
Física Corpuscular, University of Valencia; B Mellado

###### Abstract

The Tile PreProcessor (TilePPr) is the core element of the Tile Calorimeter
(TileCal) off-detector electronics for High-luminosity Large Hadron Collider
(HL-LHC). The TilePPr comprises FPGA-based boards to operate and read out the
TileCal on-detector electronics. The Tile Computer on Module (TileCoM)
mezzanine is embedded within TilePPr to carry out three main functionalities.
These include remote configuration of on-detector electronics and TilePPr
FPGAs, interface the TilePPr with the ATLAS Trigger and Data Acquisition
(TDAQ) system, and interfacing the TilePPr with the ATLAS Detector Control
System (DCS) by providing monitoring data. The TileCoM is a 10-layer board
with a Zynq UltraScale+ ZU2CG for processing data, interface components to
integrate with TilePPr and the power supply to be connected to the Advanced
Telecommunication Computing Architecture carrier. A CentOS embedded Linux is
deployed on the TileCoM to implement the required functionalities for the HL-
LHC. In this paper we present the hardware and firmware developments of the
TileCoM system in terms of remote programming, interface with ATLAS TDAQ
system and DCS system.

##  1 Introduction

The ATLAS experiment [1] is a general purpose detector originally designed to
study the products of proton-proton collisions. The basic principle of this
experiment involves the collision between two proton beams at the center of
the ATLAS detector generating particles in all directions. This is to study
energy and position measurements of electrons, photons, isolated hadrons,
$\tau$ leptons and jets.

Figure 1 shows a cross sectional view of the ATLAS Tile and Liquid Argon
calorimeters. The detector is divided into four logical sub-detectors
consisting of central long barrels (LBA, LBC) and extended barrels (EBA, EBC).
The Tile Calorimeter (TileCal) [2] is a sampling detector with iron as passive
medium and plastic scintillator tiles as active medium. The wavelength-
shifting (WLS) fiber grouping defines a three-dimensional cell structure. The
TileCal barrels are segmented into three layers (A, BC and D). A- and BC-cells
have dimensions of $\Delta\eta\times\Delta\phi=0.1\times 0.1$ in the first two
layers and $0.2\times 0.1$ in the last layer where $\eta$ represents the
pseudorapidity and $\phi$ represents the azimuthal angle. The WLS fibres are
used to collect light generated from each cell at two opposite edges. The WLS
fibres are bundled to define 4670 calorimetric cells. Light from cells is
routed to the PMTs via WLS fibers [3]. The PMTs and on-detector electronics
are on the outer radius of the module in a rigid steel girder called super-
drawers. The PMTs convert the light to electrical signals to be read out and
processed by on-detector electronics.

Figure 1:  The ATLAS detector [1]. Figure 2:  The ATLAS TileCal Phase-II
upgrade electronic chain system [4].

In the current system, these signals are digitized at the LHC frequency (40
MHz) with 10-bit Analog-to-Digital Converters (ADCs) and stored in pipeline
memories. In addition, the PMTs transmit analog trigger sums of cells to the
ATLAS Level-1 Calorimeter trigger system for particle identification and
trigger decision. All the front-end electronics are housed in Superdrawers
(SD), which are in the outermost part of the detector modules. The SD is
linked to the off-detector electronics via an optical link. Digitized detector
signals are transmitted to the Read-Out Driver (ROD) [5] in the off-detector
electronics. RODs are responsible for energy and time reconstruction, generate
busy, data integrity checking and lossless data compression. A total of 32 ROD
modules required to read out the entire TileCal detector.

##  2 The Phase-II upgrade

A major upgrade from the Large Hadron Collider (LHC) [6] to High Luminosity
LHC (HL-LHC) [7] will increase the instantaneous luminosity by a factor of 5
with a simultaneous proton-proton interactions per bunch crossing increasing
the total pile-up collisions up to 200. The HL-LHC aspires to provide an
integrated luminosity of $(3000-4000)$ fb-1. During 10 years of operation, an
improved ATLAS Trigger and Data AcQuisition (TDAQ) [8] system architecture
will have the capability to accommodate the trigger rates and amount of data
generated from the HL-LHC. In addition, the new on-detector electronics should
be capable of withstanding the HL-LHC radiation levels. Thus, a complete new
readout electronics is designed to accommodate the data acquisition system of
TileCal to the HL-LHC requirements.

Figure 3:  Tile calorimeter mini-drawer.

###  2.1 On-detector electronics

Figure 2 shows a complete redesign of the on- and off-detector readout
electronics system. The on-detector electronics are housed in SD where each SD
is composed of 4 minidrawers [9]. Figure 3 shows the Tile calorimeter mini-
drawer with organisation of the on-detector electronics. Each minidrawer
houses a maximum of 12 PMTs connected to FENICS cards, one Mainboard and one
Daughterboard to form an independent readout subsystem.. The FENICS on-
detector boards are used to shape and amplify the PMT signals in two gains.
One MainBoard [10] is allocated to each minidrawer and it is used to digitize
PMT signals by 12-bit dual ADCs. A DaughterBoard [11] is used to collect the
digital samples from the MainBoard ADCs. The collected data is then
transmitted through high speed link to the off-detector electronics. One Low
Voltage Power Supply (LVPS) [12] is used to power four minidrawers.

Figure 4:  The TilePPr Phase-II upgrade system [13].

###  2.2 Off-detector electronics

The off-detector electronics for the HL-LHC will be composed of 32 TilePPr
boards. Figure 4 shows a block diagram of one TilePPr and one Tile Trigger and
Data Acquisition interface (TDAQi). The TilePPr receive digital data from the
on-detector electronics at the accelerator frequency. Each TilePPr reads out
and operates up to eight TileCal modules where each Compact Processing Module
(CPM) [14] reads out 2 modules. The CPM serves as the core element of TilePPr
reading out and operating the TileCal modules. It is responsible for storing
the digitized samples in pipeline memories until the reception of a trigger
acceptance signal and online energy reconstruction. Then, the triggered data
is transmitted from the CPMs to the Front End LInk eXchange (FELIX) system
[15] after the reception from the ATLAS trigger system. The TDAQi receives the
reconstructed energy per cell from the CPMs and creates trigger objects [16].
These trigger objects are sent to the ATLAS trigger system for every bunch
crossing($\sim$25 ns).

The off-detector electronics are operated from Advanced Telecommunication
Computing Architecture (ATCA) shelves. The ATCA carrier host four CPMs and
three mezzanine cards: the Tile Computer-on-Module (TileCoM), the Gigabit
Ethernet Switch (GbE) switch and the Intelligent Platform Management
Controller (IPMC). The TileCoM is used as an entry point computer to remotely
access the entire readout electronic system of TileCal. The TileCoM is
accessed remotely according to the ATLAS system administration [17] within the
CERN ATLAS Technical and Control Network (ATCN) requirements. The TileCoM is a
System-on-Chip (SoC) based system that is connected to the CPMs, GbE and TDAQi
board, through the ATCA carrier.

##  3 The TileCoM hardware components

Figure 5:  Picture of the first prototype of the TileCoM. Figure 6:  TileCoM
block diagram with all the components.

The TileCoM is a 10-layer board which hosts a Zynq UltraScale + ZU2CG as shown
in Figure 5. Figure 6 shows TileCoM block diagram with all the components. The
Zynq UltraScale + ZU2CG includes major components such as Dual-core ARM
Cortex-R5F Based Real-Time Processing Unit, On-Chip Memory, ARM Mali-400 Based
GPU and interconnects to external components. The Zynq UltraScale+ ZU2CG can
be used to implement high-performance applications from a single platform. The
functionalities mentioned in Sections 4.2 to 4.4 are implemented on Processing
System (PS) and Programmable Logic (PL) to interact with the hardware and the
rest of ATLAS TileCal Phase-II upgrade electronic readout system.

The GbE and Inter-Integrated Circuit (I2C) protocols are used to interface
with Tile GbE Switch card and ATCA carrier, respectively. Whereas,
Transmission Control Protocol and the Internet Protocol (TCP/IP) is used as an
interconnection of TilePPr and DCS as detailed in Section 4.2. General Purpose
Input Output (GPIO) components are mostly used in remote programming
functionality explained in Section 4.4.

Figure 7:  The ATLAS TileCal remote programming architecture.

##  4 Software architecture

###  4.1 Deployment of an embedded Linux for TileCoM

The CERN Computing Security rules state that all computing devices in the
technical networks must be centrally managed as detailed in ATLAS SoC
requirements specification document [18]. Thus, the TileCoM will be fully
managed by ATLAS TDAQ System Administrators after its installation during the
Phase-II upgrade. According to the ATLAS SoC requirements, the CentOS and
PetaLinux distributions are the two preferred Linux-based OS for the
implementation of embedded solutions for the ATLAS Phase-II Upgrade.

The selected OS for the TileCoM is CentOS 7. The CentOS files as used in [19]
were used for the TileCoM OS. These files were loaded in the SD to boot the
embedded Linux. The three main parts of the CentOS 7 implemented on the
TileCoM are detailed below:

  * $\bullet$

User-space Linux applications: This is the user interface that allows users to
develop software and firmware applications. The Xilinx Virtual Cable (XVC)
[20] server, the IPbus and the OPC server are implemented on the user-space.

  * $\bullet$

Kernel version and common driver set: The kernel is configured specifically
for the TileCoM using the specific board support package from Xilinx.
Configuration, such as the setting kernel bootargs and user-space I/O drivers
are generated for CentOS 7.

  * $\bullet$

System tools and libraries: The user-space I/O drivers generated during kernel
configuration enables communication between the TileCoM I/O and the embedded
Linux. Libraries such as the Input Output (IIO) are used for the monitoring
functionality implemented in Section 4.4 to read from ADC sensors.

###  4.2 FPGA remote programming

Figure 8:  The Xilinx remote programming software and firmware architecture.

The TileCoM will be used to remotely program the on- and off-detector FPGAs.
The target FPGAs include the DaughterBoard and MainBoard FPGAs in the on-
detector electronics, and the CPM and TDAQi FPGAs in the off-detector
electronics. Each TileCoM configures four CPM Xilinx KU115 FPGAs, one TDAQi
Xilinx KU025 FPGA , 16 KU035 FPGAs (8 DaughterBoards), 32 Cyclone IV (8
Mainboards). FPGA configuration files, such as bitstreams and memory flash
files, will be sent through TCP/IP to the TileCoM from an external servers.
Then the TileCoM configures the desired FPGA using dedicated JTAG chains
implemented over the fibre for the on-detector electronics. Figure 7 shows a
block diagram defining the interconnection paths between the TileCoM and the
target FPGAs for remote programming.

The XVC [21] is used to implement the remote programming functionality on the
TileCoM. Figure 8 shows the design on the PL implemented using Intellectual
Property (IP) block designs from Xilinx to interconnect to the peripherals of
the TileCoM. IP blocks such as the binary counter, system management, PS
Advanced eXtensible Interface (AXI) interconnect peripherals, Debug Bridge and
the GPIO slaves. The AXI bus interfaces are mainly used transfer data between
PS and PL. The Debug Bridge is the main core for this functionality. It
provides a mechanism to establish the communication between the debug cores
and non-JTAG interfaces (for example, Ethernet/PCIe). The XVC server is
implemented on the ARM processor compiled in C programming language and
interfaces with the low-level design using embedded Linux. The ATCA carrier
includes one master JTAG chain for the CPMs and TDAQi, and one remote JTAG
chain per two Tile Modules, where each remote JTAG chain per CPM reads out two
TileCal modules. Xilinx Vivado tool runs in an external server to send
bitstreams and memory flash files.

###  4.3 Interface with the ATLAS TDAQ system

The off-detector electronics are divided into two main parts that consist of
FPGA boards: TilePPr, and TDAQi. Figure 9 shows a block diagram of the TilePPr
and TDAQ system interface. This functionality is mainly focused on the
communication of TilePPr and TDAQi with the ATLAS TDAQ system. The main
purpose of this communication is slow control and configuration of the on- and
off-detector electronics for calibration and operation purposes. This involves
configuration of ATCA carrier and CPMs in terms of clocking, power, optical
modules, GbE switch and other peripherals. To accomplish this functionality,
an IPbus based communication system is used [22]. This is a control protocol
for reading and modifying memory-mapped FPGA resources based on 32-bit
registers and addresses. It is used in conjunction with a remote software
capability implemented using the IPbus Ironman software [23] which implements
TCP/IP and UDP protocol for communication purposes.

Ironman is a Python framework that implements a single-threaded, reactor based
event-driven callback for the server and client communication. This framework
is used to standardize communication protocols on embedded processors. The
framework is mainly divided into three blocks on the PS of the TileCoM: the
server, SoC client and the internal communication interface. The server block
establishes main communication between the TileCoM and the client user. The
SoC client includes hardware maps that link the server and the internal
communication. The SoC client analyze the packet the packet and decide if the
packet includes valid address, valid permissions and valid data. If the packet
is valid, then the internal communication block builds up a response. Lastly,
the internal communication interface the TileCoM and the TDAQ system. The last
block establishes communication by using the hardware interface of the
framework to interface the TDAQ system through the PL of the TileCoM. The
implementation of Ironman server on the TileCoM makes it possible for the TDAQ
system to interface with the TileCoM through the ATCN.

Figure 9:  The ATLAS TileCal TilePPr-TDAQ system interface block diagram.

###  4.4 Interface with the ATLAS DCS system functionality

This functionality focus on both the communication, data acquisition and
publication of the monitoring data to the DCS. This is achieved by
implementing two Open Platform Communications (OPC) servers on the TileCoM to
communicate with the TileCal Phase-II upgrade electronics system and ATLAS
DCS. Figure 10 presents the general block diagram of the TilePPr and DCS
interface functionality [24]. This block diagram shows the connection with all
the components of the off-detector electronics. The servers will run
concurrently on the TileCoM reading approximately 2000 sensors per TilePPr.
All the 2000 sensors are read from the on-detector electronics. The acquired
data from TilePPr will be published to the DCS at a minimum frequency of 0.1
Hz [25].

The first Xilinx Analog-to-Digital Converter (XADC) OPC server monitors the
power consumption, optics diagnostics and temperature sensors. The second OPC
server is implemented to interface with the ATCA carrier, TDAQi and CPM
sensors. Additionally, this second server reads out local ATCA carrier sensors
through the $I^{2}C$ interface, and reads out sensor data from CPM and TDAQi
through GbE using the software IPbus implementation as detailed in Section
4.3.

Figure 10:  The ATLAS TilePPr and DCS interface functionality.

####  4.4.1 OPC Implementation

The Quick OPC UA server generation framework (Quasar) [26] is a software
development framework used for generic and efficient development of OPC UA
servers. The Quasar designs are created in XML formats and OPC UA executables
are generated to run the servers. This framework reduces development cycle and
enables the developers to integrate their devices. Configuration files are
also generated in XML formats for client connectivity and server data
representation. To integrate this framework to the TileCoM XADC, the IIO
library is added to the custom C++ code. A memory map is used for this
functionality to map the address of the sensors from the XADC. This layer
communicates with the XML design file to form a tree structure of the sensors
included in the design. Thus, XML configuration file handles the data
representation between the server and the client.

A few number of client applications are available to test publication of data
acquired by the OPCUA server such as the Supervisory Control and Data
Acquisition (SCADA: Siemens WinCC OA). This server was tested with both the UA
expert and SCADA for publication of data for DCS.

##  5 TileCoM integration tests results

Figure 11:  The integration of TilePPr boards test bench.

Figure 11 shows the test bench used to test the three main TileCoM
functionalities. The test bench includes one TileCal GbE Switch, one network
router, one Avnet Ultra96-V2 Zynq UltraScale+ MPSoC TileCoM evaluation board
and one Xilinx VC707 evaluation board as a CPM Emulator. The network router is
used to provide external access to all the boards integrating the test bench.
The TileCal GbE Switch is used as an interface for interconnecting all the
TilePPr modules. The Avnet Ultra96-V2 Zynq UltraScale+ MPSoC TileCoM
evaluation board is connected to the computer via UART to display the embedded
Linux terminal and control different applications developed for the TileCoM.

Table 1: The IP block design TileCoM FPGA resources. FPGA resource |  Utilization (%) |  Utilization |  Available
---|---|---|---
Look-UP Tables |  3.17% |  2238 |  70560
Flip Flops |  1.70% |  2396 |  141120
RAM blocks |  0.35% |  102 |  28800
Input/Output (IO) |  7.32% |  6 |  82
Global clock buffer |  2.55% |  5 |  196

Table 1 shows the number of used resources on the TileCoM FPGA. The remote
programming functionality was tested with the PL side of the TileCoM. The XVC
server application software is executed on the PS of the TileCoM. The PL was
accessed remotely using Vivado software. A bitstream to program the TileCoM
was sent remotely using the Vivado software.

The ATLAS DCS system implementation results are shown below. Figure 12 shows a
plot for temperature sensor readings taken for a period of 24 hours running
the OPCUA server on the TileCoM. The temperature readings are in the range of
48 degrees Celsius for the PS temperature sensor. The temperature of the board
increased during data acquisition period, remaining in the threshold values
according to the recommended operating values from the manufacturer [27]. The
voltage values shown in Figure 13 are within the threshold values recommended
by the FPGA manufacturer.

Figure 12:  The OPCUATileXADC Temperature and Voltage sensor data. Figure 13:
The OPCUATileXADC Voltage sensor data.

##  6 Conclusions

The ATLAS TileCal Phase-II upgrade system implies the complete redesign of the
on- and off-detector readout electronics in order to accommodate the current
DAQ system to the new requirements for the HL-LHC. This contribution focuses
on the TileCoM developments as part of the new readout electronics and a key
element of the TDAQ and DCS elements of the ATLAS TileCal at the HL-LHC. The
TileCoM is a SoC-based board running an embedded CentOS 7 Linux system, which
interfaces between the TileCal readout electronics and the ATLAS TDAQ and DCS
systems. Three main functionalities are implemented on the TileCoM in software
and firmware implementation. The remote programming functionality is used to
remotely program the Mainboard, Daughterboard, CPM and TDAQi FPGAs. The XVC
server application running on the TileCoM sends bitstream and memory flash
files through TCP/IP to the TileCoM from external servers. The second
functionality interfaces with the on- and off-detector electronics with the
TDAQ for configuration and slow control via GbE. Different calibrations runs
through the TileCoM. This functionality also includes monitoring of ATCA on-
board sensors, CPM sensors and on-detector electronics, configuration of the
ATCA carrier and CPMs. Lastly, the third functionality involves an
implementation of the communication between the TilePPr and the DCS. Two OPCUA
servers are implemented on the TileCoM to read out on- and off-detector
electronics sensor data. In the final application, approximately 2000 sensors
from the on-detector electronics are monitored by each TileCoM module. These
sensor data is transmitted in real-time to the DCS.

## Acknowledgments

The authors want to thank the South African Department of Science and
Innovation, and the National Research Foundation for their continued support
through the SA-CERN program and other forms of support. The Research Office of
the University of the Witwatersrand is also acknowledged and the Instituto de
Fisica Corpuscular.

## References

  * [1] Aad G et al. (ATLAS) 2008 JINST 3 S08003
  * [2] Aad G et al. (ATLAS) 2010 Eur. Phys. J. C 70 1193–1236
  * [3] Abdallah J et al. (ATLAS TileCal) 2013 JINST 8 P01005
  * [4] Valdes Santurio E (ATLAS Tile Calorimeter System) 2017 J. Phys. Conf. Ser. 928 012024
  * [5] Valero A 2020 Operational experience and evolution of the ATLAS Tile Hadronic Calorimeter Read-Out Drivers 22nd IEEE Real Time Conference
  * [6] Emery J et al. 2010 JINST 5 C12044
  * [7] Mckenzie R P (ATLAS Tile Calorimeter System) 2021 PoS LHCP2021 246
  * [8] Izzo V (ATLAS) 2021 PoS LHCP2020 094
  * [9] Valdes Santurio E (ATLAS Tile Calorimeter System) 2019 Nucl. Instrum. Meth. A 936 115–116
  * [10] Tang F, et al. (ATLAS Tile Calorimeter Group) 2014 Design of Main Board for ATLAS TileCal Demonstrator 19th Real Time Conference
  * [11] Santurio E V et al. 2021 IEEE Trans. Nucl. Sci. 68 2414–2420
  * [12] Nkadimeng E et al. (ATLAS) 2019 Low Voltage Power Supply production, hardware upgrade and testing for the ATLAS TileCal Front-End Electronics system SAIP2021
  * [13] Carrió F A and Valero A (ATLAS Tile Calorimeter) 2020 Nucl. Instrum. Meth. A 958 162487
  * [14] Carrió F A (ATLAS) 2021 IEEE Trans. Nucl. Sci. 68 1944–1951
  * [15] Paramonov A (ATLAS TDAQ) 2021 EPJ Web Conf. 251 04006
  * [16] Yue X 2019 Tile TDAQ interface module for the Phase-II Upgrade of the ATLAS Tile Calorimeter 2019 IEEE Nuclear Science Symposium (NSS) and Medical Imaging Conference (MIC) pp 1–4
  * [17] Ballestrero S et al. 2017 J. Phys. Conf. Ser. 898 082002
  * [18] ATLAS 2021 Soc for atlas subsystems: Requirements document for hl-lhc Tech. Rep. ATU-GE-ES-0005 ATLAS
  * [19] Spiwoks R et al. 2020 CentOS Linux for the ATLAS MUCTPI Upgrade 22nd IEEE Real Time Conference
  * [20] Xilinx 2021 Designing an xvc project for remote debugging of zynq ultrascale+ devices Tech. Rep. 000032879 Xilinx 974879
  * [21] Clark A and Bielich L 2015 Xilinx Virtual Cable Running on Zynq-7000 Using the PetaLinux Tools XAPP1251
  * [22] Ghabrous Larrea C et al. 2015 JINST 10 C02019
  * [23] Spiwoks R 2019 System-on-Chip (SoC): Experience and Recommendations
  * [24] Martins F (ATLAS) 2016 The ATLAS tile calorimeter DCS for run 2 2016 IEEE Nuclear Science Symposium and Medical Imaging Conference p 8069837
  * [25] ATLAS 2020 Dcs: Requirements for hl-lhc : Atlas phase-ii upgrade project Tech. Rep. ATL-COM-UPGRADE-2020-013 ATLAS
  * [26] Schlenker S et al. 2015 quasar - A Generic Framework for Rapid Development of OPC UA Servers 108 4
  * [27] Xilinx 2022 Zynq ultrascale+ mpsoc data sheet Tech. Rep. DS925 Xilinx v1.20