# Towards Universal Sequence Representation Learning
for Recommender Systems

Yupeng Hou∗†<EMAIL_ADDRESS>Gaoling School of Artificial Intelligence,
Renmin University of China ,  Shanlei Mu∗<EMAIL_ADDRESS>School of
Information, Renmin University of China ,  Wayne Xin Zhao†‡🖂
<EMAIL_ADDRESS>Gaoling School of Artificial Intelligence, Renmin
University of China ,  Yaliang Li<EMAIL_ADDRESS>Alibaba Group ,
Bolin Ding<EMAIL_ADDRESS>Alibaba Group and  Ji-Rong Wen†
<EMAIL_ADDRESS>Gaoling School of Artificial Intelligence, Renmin University
of China

(2022)

###### Abstract.

In order to develop effective sequential recommenders, a series of sequence
representation learning (SRL) methods are proposed to model historical user
behaviors. Most existing SRL methods rely on explicit item IDs for developing
the sequence models to better capture user preference. Though effective to
some extent, these methods are difficult to be transferred to new
recommendation scenarios, due to the limitation by explicitly modeling item
IDs. To tackle this issue, we present a novel universal sequence
representation learning approach, named UniSRec. The proposed approach
utilizes the associated description text of items to learn transferable
representations across different recommendation scenarios. For learning
_universal item representations_ , we design a lightweight item encoding
architecture based on parametric whitening and mixture-of-experts enhanced
adaptor. For learning _universal sequence representations_ , we introduce two
contrastive pre-training tasks by sampling multi-domain negatives. With the
pre-trained universal sequence representation model, our approach can be
effectively transferred to new recommendation domains or platforms in a
parameter-efficient way, under either inductive or transductive settings.
Extensive experiments conducted on real-world datasets demonstrate the
effectiveness of the proposed approach. Especially, our approach also leads to
a performance improvement in a cross-platform setting, showing the strong
transferability of the proposed universal SRL method. The code and pre-trained
model are available at: https://github.com/RUCAIBox/UniSRec.

Sequential Recommendation, Universal Representation Learning

$*$ Equal contribution.

$\dagger$ Beijing Key Laboratory of Big Data Management and Analysis Methods.

$\ddagger$ Beijing Academy of Artificial Intelligence, Beijing, 100084, China.

🖂 Corresponding author.

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August
14–18, 2022; Washington, DC, USA.††booktitle: Proceedings of the 28th ACM
SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22), August
14–18, 2022, Washington, DC, USA††price: 15.00††isbn:
978-1-4503-9385-0/22/08††doi: 10.1145/3534678.3539381††ccs: Information
systems Recommender systems

##  1\. Introduction

In the literature of recommender systems, sequential recommendation is a
widely studied task (Hidasi et al., 2016; Kang and McAuley, 2018), aiming to
recommend suitable items to a user given her/his historical interaction
records. Various methods have been proposed to improve the performance of
sequential recommendation, from early matrix factorization (_e.g.,_ FPMC
(Rendle et al., 2010)) to recent sequence neural networks (_e.g.,_ GRU4Rec
(Hidasi et al., 2016), Caser (Tang and Wang, 2018) and Transformer (Kang and
McAuley, 2018; Zhou et al., 2020)). These approaches have largely raised the
performance bar of sequential recommendation.

Though the adopted techniques are different, the core idea of existing methods
is similar: it first formulates the user behavior as a chronologically-ordered
interaction sequence with the items, and then develops effective architectures
for capturing the sequential interaction characteristics that reflect the user
preference. In this way, the learned sequence models can predict the likely
items to be interacted by a user, given the observed sequential context.
Framed in the paradigm of representation learning (Bengio et al., 2013), such
an approach essentially aims to conduct a sequence representation learning
(SRL) model based on historical user behavior data, which needs to effectively
capture the item characteristics and sequential interaction characteristics.
The capacity of the designed SRL model directly affects the performance of
sequential recommendation.

Despite the progress, most of existing SRL methods for recommendation rely on
explicit _item IDs_ for developing the sequence models (Hidasi et al., 2016;
Kang and McAuley, 2018). A major issue with this modeling way is that the
learned model is difficult to be transferred to new recommendation scenarios,
even when the underlying data forms are exactly the same. Such an issue limits
the reuse of the recommendation model across domains (Zhu et al., 2021). As a
common case, we need to re-train a sequential recommender from scratch when
adapting to a new domain, which is tedious and resource-consuming. In
addition, existing sequential recommenders usually suffer from the
recommendation of cold-start items having inadequate interactions with users.
For addressing the above issue, a number of studies have been proposed by
learning either semantic mapping (Zhu et al., 2019) or transferable components
(Li et al., 2022), in order to bridge the domain gap and enhance the item
representations. However, these existing attempts cannot fully solve the
fundamental issue caused by explicitly modeling item IDs. More recently,
increasing evidence shows that natural language text can play the role of
general semantic bridge across different tasks or domains (Devlin et al.,
2019) (including the recommendation tasks, _e.g.,_ zero-shot recommendation
(Ding et al., 2021)), with the remarkable success of pre-trained language
models (PLM).

Inspired by the recent progress of language intelligence (Vaswani et al.,
2017; Devlin et al., 2019), we aim to design a new SRL approach for learning
more generalizable sequence representations, by breaking the limit of explicit
ID modeling. The core idea is to utilize the associated description text
(_e.g.,_ product description, product title or brand) of an item, called _item
text_ , to learn transferable representations across different domains.
Although previous attempts have shown such an approach is promising (Ding et
al., 2021), there are still two major challenges to be solved. First, the
textual semantic space is not directly suited for the recommendation tasks. It
is not clear how to model and utilize item texts for improving the
recommendation performance, since directly introducing raw textual
representations as additional features may lead to suboptimal results. Second,
it is difficult to leverage multi-domain data for improving the target domain,
where the _seesaw phenomenon_ (referring to learning from multiple kinds of
domain-specific patterns is conflict or oscillating) often appears (Tang et
al., 2020).

To address the above issues, we propose the universal sequence representation
learning approach, named UniSRec. Our approach takes general interaction
sequences as input, and learns universal ID-agnostic representations based on
a pre-training approach. Specially, we focus on two key points that learn
_universal item representation_ and _universal sequence representations_. For
learning universal item representations, we design a lightweight architecture
based on parametric whitening and mixture-of-experts enhanced adaptor, which
can derive more isotropic semantic representations as well as enhance the
domain fusion and adaptation. For learning universal sequence representations,
we introduce two kinds of contrastive learning tasks, namely sequence-item and
sequence-sequence contrastive tasks, by sampling multi-domain negatives. Based
on the above methods, the pre-trained model can be effectively transferred to
a new recommendation scenario in a parameter-efficient way, under either
inductive or transductive settings.

To evaluate the proposed approach UniSRec, we conduct extensive experiments on
real-world datasets from different application domains and platforms.
Experimental results demonstrate that the proposed approach can effectively
utilize data from multiple domains to learn universal and transferable
representations. Especially, the results on cross-platform experiments show
that recommendation performance can be improved with the universal sequence
representation model pre-trained on other platforms without overlapping users
or items.

##  2\. Methodology

In this section, we present the proposed Universal Sequence representation
learning approach for Recommendation, named as UniSRec. Given historical user
behavioral sequences from a mixture of multiple domains, we aim to learn
universal item and sequence representations that can effectively transfer and
generalize to new recommendation scenarios (_e.g.,_ new domains or platforms)
in a parameter-efficient way.

###  2.1. Overview of the Approach

Our approach takes general interaction sequences as input and learns universal
representations based on a pre-training approach. We then formulate the task
and overview the proposed approach.

General input formulation. We formulate the behavior sequence of a user in a
general form of interaction sequence $s=\\{i_{1},i_{2},\cdots i_{n}\\}$, (in a
chronological order of interaction time), where each interacted item $i$ is
associated with a unique item ID and a description text (_e.g.,_ the product
description, item title or brand). We call the description text of an item $i$
_item text_ , denoted by $t_{i}=\\{w_{1},w_{2},\cdots,w_{c}\\}$, where the
words $w_{j}$ are from a shared vocabulary and $c$ denotes the truncated
length of item text. Here, each sequence contains all the interaction behavior
of a user at some specific domain, and a user can generate multiple behavior
sequences at different domains or platforms. As discussed before, as there are
large semantic gap between different domains, we don’t simply mix the behavior
data of a user. Instead, we treat the multiple interaction sequences of a user
as different sequences, without explicitly maintaining user IDs for each
sequence. Note that unlike other pre-training based recommendation methods
(Zhou et al., 2020; Yuan et al., 2020), item IDs are only auxiliary
information in our approach, and we mainly utilize item text to derive
generalizable ID-agnostic representations. Unless specified, item IDs will not
be used as input of our approach.

Solutions. To learn transferable representations across domains, we identify
two key problems for achieving this purpose, _i.e.,_ learning universal item
representation and sequence representation, since _items_ and _sequences_ are
the basic data forms in our general formulation. For learning universal item
representations (Section 2.2), we focus on the domain fusion and adaptation
with an MoE-enhanced adaptor based on parametric whitening. For learning
universal sequence representations (Section 2.3), we introduce two kinds of
contrastive learning tasks, namely sequence-item and sequence-sequence
contrastive tasks, by sampling multi-domain negatives. Based on the above
methods, the pre-trained model can be effectively transferred to a new
recommendation scenario in a parameter-efficient way, under either inductive
or transductive settings (Section 2.4). The overall framework of the proposed
approach UniSRec is depicted in Figure 1.

Figure 1. The overall framework of the proposed universal sequence
representation learning approach (UniSRec).

###  2.2. Universal Textual Item Representation

The first step toward universal sequential behavior modeling is to represent
items from various recommendation scenarios (_e.g.,_ domains or platforms)
into a unified semantic space. In previous studies (Hidasi et al., 2016; Kang
and McAuley, 2018), item representations are usually learned in _transductive
learning_ setting, where item IDs are pre-given and ID embeddings are learned
as item representations. Such a way largely limits the transferability of item
representations, since the vocabularies of item IDs are usually different
across domains.

Our solution is to learn transferable item representations based on the
associated item text, which describes the item characteristics in the form of
natural language. As more and more evidence shows (Devlin et al., 2019),
natural language provides a general data form to bridge the semantic gap
across different tasks or domains. Based on this idea, we first utilize the
pre-trained language model (PLMs) to learn the text embeddings. Since the
derived text representations from different domains are likely to span
different semantic spaces (even with the same text encoder), we propose the
techniques of parametric whitening and mixture-of-experts (MoE) enhanced
adaptor to transform the text semantics into a universal form suited to the
recommendation tasks.

####  2.2.1. Textual Item Encoding via Pre-trained Language Model

Considering the excellent language modeling capacity of PLMs, we utilize the
widely used BERT model (Devlin et al., 2019) to learn universal text
representations for representing items. Given an item $i$ and its
corresponding text $t_{i}$, we firstly concatenate (1) a special symbol [CLS],
(2) the words of item text $\\{w_{1},w_{2},\ldots,w_{c}\\}$, in order and
derive the input sequence for BERT. Then we feed the concatenated sequence
into the BERT model, and we have:

(1) $\bm{x}_{i}=\operatorname{BERT}([\texttt{[CLS]};w_{1},\ldots,w_{c}]),$

where $\bm{x}_{i}\in\mathbb{R}^{d_{W}}$ is the final hidden vector
corresponding to the first input token ([CLS]), and “[;]” denotes the
concatenation operation.

####  2.2.2. Semantic Transformation via Parametric Whitening

Though we can obtain semantic representations from BERT, they are not directly
suited for the recommendation tasks. Existing studies (Li et al., 2020) have
found that BERT induces a non-smooth anisotropic semantic space for general
texts. The case will become more severe when we mix item texts from multiple
domains with a large semantic gap. Inspired by recent works on whitening-based
methods (Su et al., 2021; Huang et al., 2021), we conduct a simple linear
transformation to transform original BERT representations for deriving
isotropic semantic representations.

Different from the original whitening methods (Su et al., 2021; Huang et al.,
2021) with pre-set mean and variance, we incorporate learnable parameters in
the whitening transformation for better generalizability on unseen domains.
Formally, we have:

(2) $\widetilde{\bm{x}}_{i}=(\bm{x}_{i}-\bm{b})\cdot\bm{W}_{1},$

where $\bm{b}\in\mathbb{R}^{d_{W}}$ and $\bm{W}_{1}\in\mathbb{R}^{d_{W}\times
d_{V}}$ are learnable parameters, and
$\widetilde{\bm{x}}_{i}\in\mathbb{R}^{d_{V}}$ is the transformed
representation. In this way, the anisotropy issue of learned representations
can be alleviated, which is useful to learn universal semantic
representations. For the efficiency consideration, we don’t introduce complex
non-linear architectures, such as flow-based generative models (Li et al.,
2020), which will be studied in future work.

####  2.2.3. Domain Fusion and Adaptation via MoE-enhanced Adaptor

With the above whitening transformation, our model can learn more isotropic
semantic representations. In order to learn universal item representations,
another important issue is how to transfer and fuse information across
domains, since there is usually a large semantic gap between different
domains. For example,the top frequent words of item text are quite different
across domains, _e.g.,_ _natural_ , _sweet_ , _fresh_ for food domain and
_war_ , _love_ , _story_ for movies domain. A straightforward approach is to
map the original BERT embeddings into some shared semantic space (Li et al.,
2020). However, it will lead to limited representation capacity for migrating
the domain bias. As our solution, we learn multiple whitening embeddings for
an item, and utilize an adaptive combination of these embeddings as the
universal item representations. Not limited to a simple single mapping between
item representations, we aim to establish a more flexible representation
mechanism to capture the semantic relatedness for domain fusion and
adaptation.

To implement our idea, we employ the mixture-of-expert (MoE) architecture
(Shazeer et al., 2017) for learning more generalizable item representations.
Specially, we incorporate $G$ whitening transformation modules as the
_experts_ , and then construct the MoE-enhanced adaptor based on a
parameterized router:

(3) $\bm{v}_{i}=\sum_{k=1}^{G}g_{k}\cdot\widetilde{\bm{x}}_{i}^{(k)},$

where $\widetilde{\bm{x}}_{i}^{(k)}$ is the output of the $k$-th whitening
transformation module (Eqn. (2)) and $g_{k}$ is the corresponding combination
weight from the _gating router_ , defined as follows:

(4) $\displaystyle\bm{g}=$
$\displaystyle\operatorname{Softmax}\left(\bm{x}_{i}\cdot\bm{W}_{2}+\bm{\delta}\right),$
(5) $\displaystyle\bm{\delta}=$
$\displaystyle\operatorname{Norm}()\cdot\operatorname{Softplus}\left(\bm{x}_{i}\cdot\bm{W}_{3}\right).$

In this equation, we utilize original BERT embedding $\bm{x}_{i}$ as input of
the router module, as it contains domain-specific semantic bias. Furthermore,
we incorporate learnable parameter matrices
$\bm{W}_{2},\bm{W}_{3}\in\mathbb{R}^{d_{W}\times G}$ to adaptively adjust the
weights of experts $\bm{g}\in\mathbb{R}^{G}$. In order to balance the expert
load, we utilize $\operatorname{Norm}()$ to generate random Gaussian noise,
controlled by parameter $\bm{W}_{2}$.

The merits of the MoE-enhanced adaptor are threefold. Firstly, the
representation of a single item is enhanced by learning multiple whitening
transformations. Second, we no longer require a direct semantic mapping across
domains, but instead utilize a learnable gating mechanism to adaptively
establish the semantic relatedness for domain fusion and adaptation. Third,
the lightweight adaptor endows the flexibility of parameter-efficient fine-
tuning when adapting to new domains (detailed in Section 2.4).

###  2.3. Universal Sequence Representation

Since different domains usually correspond to varying user behavioral
patterns, it may not work well to simply mix interaction sequences from
multiple domains for pre-training. And, it is likely to lead to the _seesaw
phenomenon_ (Tang et al., 2020) that the learning from multiple domain-
specific behavioral patterns can be conflict. Our solution is to introduce two
kinds of contrastive learning tasks, which can further enhance the fusion and
adaptation of different domains in deriving the item representations. In what
follows, we firstly introduce the base behavior encoder architecture, and then
present the proposed contrastive pre-training tasks that enhance the sequence
representations in universal semantic space.

####  2.3.1. Self-attentive Sequence Encoding

Given a sequence of universal item representations, we further utilize a user
behavior encoder to obtain the sequence representation. We aim to construct
the sequential patterns based on the learned universal textual item
representations, but not item IDs. Here, we adopt a widely used self-attentive
architecture (Vaswani et al., 2017), _i.e.,_ Transformers. Specially, it
consists stacks of multi-head self-attention layers (denoted by
$\operatorname{MHAttn}(\cdot)$) and point-wise feed-forward networks
(multilayer perceptron activated by $\operatorname{ReLU}$, denoted by
$\operatorname{FFN}(\cdot)$). We sum the learned text representations (_i.e.,_
$\bm{v}_{i}$ in Eqn. (3)) and the absolute position embeddings $\bm{p}_{j}$ as
input at position $j$. The input and the update process can be formalized as
following:

(6) $\displaystyle\bm{f}^{0}_{j}=$ $\displaystyle\ \bm{v}_{i}+\bm{p}_{j},$ (7)
$\displaystyle\bm{F}^{l+1}=$
$\displaystyle\operatorname{FFN}(\operatorname{MHAttn}(\bm{F}^{l})),$

where $\bm{F}^{l}=[\bm{f}^{l}_{0};\ldots;\bm{f}^{l}_{n}]$ denotes the
concatenated representations at each position in the $l$-th layer. We take the
final hidden vector $\bm{f}_{n}^{L}$ corresponding to the $n$-th (last)
position as the sequence representation (a total number of $L$ layers in the
behavior encoder).

####  2.3.2. Multi-domain Sequential Representation Pre-training

Given interaction sequences from multiple domains, we next study how to design
suitable optimization objectives to derive the outputs of sequential encoder
in the unified representation space. By contrasting sequences and items from
different domains, we aim to alleviate the seesaw phenomenon and capture their
semantic correlation in the pre-training stage. For this purpose, we design
the following sequence-item and sequence-sequence contrastive tasks.

Sequence-item contrastive task. The sequence-item contrastive task aims to
capture the intrinsic correlation between sequential contexts (_i.e.,_ the
observed subsequence) and potential next items in an interaction sequence.
Different from previous next-item prediction task (Kang and McAuley, 2018; Lin
et al., 2022) (using _in-domain_ negatives), for a given sequence, we adopt
_across-domain_ items as negatives. Such a way can enhance both the semantic
fusion and adaptation across domains, which is helpful to learn universal
sequence representations.

We consider a batch setting of $B$ training instances, where each training
instance is a pair of the sequential context (containing the proceeding items)
and the positive next item. We first encode them into embedding
representations
$\\{\langle\bm{s}_{1},\bm{v}_{1}\rangle,\ldots,\langle\bm{s}_{B},\bm{v}_{B}\rangle\\}$,
where $\bm{s}$ represents the normalized contextual sequence representations,
and $\bm{v}$ denotes the representation of the positive next item. Then, we
formalize the sequence-item contrastive loss as follows:

(8)
$\ell_{S-I}=-\sum_{j=1}^{B}\log\frac{\exp{\left(\bm{s}_{j}\cdot\bm{v}_{j}/\tau\right)}}{\sum_{j^{\prime}=1}^{B}\exp{\left(\bm{s}_{j}\cdot\bm{v}_{j^{\prime}}/\tau\right)}},$

where in-batch items are regarded as negative instances and $\tau$ is a
temperature parameter. As batches are constructed randomly, the in-batch
negative instances $\\{\bm{v}_{j^{\prime}}\\}$ will contain items from a
mixture of multiple domains.

Sequence-sequence contrastive task. Besides the above item-level pre-training
task, we further propose a sequence-level pre-training task, by conducting the
contrastive learning among multi-domain interaction sequences. The object is
to discriminate the representations of augmented sequences from multi-domain
sequences. We consider two kinds of augmentation strategies: (1) _Item drop_
refers to randomly dropping a fixed ratio of items in the original sequence,
and (2) _Word drop_ refers to randomly dropping words in item text. Given a
target sequence (with the representation $\bm{s}_{j}$), the augmented ones are
considered as positives (with the representation $\widetilde{\bm{s}}_{j}$),
while other in-batch ones are considered as negatives. The sequence-sequence
contrastive loss can be formally presented as following:

(9)
$\ell_{S-S}=-\sum_{j=1}^{B}\log\frac{\exp{\left(\bm{s}_{j}\cdot\widetilde{\bm{s}}_{j}/\tau\right)}}{\sum_{j^{\prime}=1}^{B}\exp{\left(\bm{s}_{j}\cdot\bm{s}_{j^{\prime}}/\tau\right)}}.$

Similar to Eqn. (8), as batches are constructed randomly, the in-batch
negative instances naturally contain sequences from multiple domains. In the
implementation, we preprocess the augmented item text using _word drop_ for
efficient pre-training, as the BERT representations of item text can be
obtained during preprocessing.

Multi-task learning. At the pre-training stage, we leverage a multi-task
training strategy to jointly optimize the proposed sequence-item contrastive
loss in Eqn. (8) and sequence-sequence contrastive loss in Eqn. (9):

(10) $\mathcal{L}_{\text{PT}}=\ell_{S-I}+\lambda\cdot\ell_{S-S},$

where $\lambda$ is a hyper-parameter to control the weight of sequence-
sequence contrastive loss. The pre-trained model is to be fine-tuned for
adapting to new domains.

###  2.4. Parameter-Efficient Fine-tuning

To adapt to a new domain, previous pre-training based recommendation methods
(Zhou et al., 2020; Ding et al., 2021) usually require fine-tuning the whole
network architecture, which is time-consuming and less flexible. Since our
model can learn universal representations for interaction sequences, our idea
is to fix the parameters of the major architecture, while only fine-tuning a
small proportion of parameters from the MoE-enhanced adaptor (Section 2.2.3)
for incorporating necessary adaptation. We find that the proposed MoE-enhanced
adaptor can quickly adapt to unseen domains, fusing the pre-trained model with
new domain characteristics. To be specific, we consider two fine-tuning
settings, either _inductive_ or _transductive_ , based on whether item IDs in
the target domain can be accessed.

Inductive setting. The first setting considers the test cases of recommending
new items from an unseen domain, which can’t be well solved by an ID-based
recommendation model. The proposed model doesn’t rely on item IDs, so that it
can learn universal text representations for new items. Given a training
sequence from the target domain, we firstly encode the sequential context
($i_{1}\rightarrow i_{t}$) and the candidate item $i_{t+1}$ into universal
representations as $\bm{s}$ and $\bm{v}_{i_{t+1}}$. Then we predict the next
item according to the following probability:

(11) $P_{I}(i_{t+1}|s)=\operatorname{Softmax}(\bm{s}\cdot\bm{v}_{i_{t+1}}),$

where we compute the softmax probability over the candidate set (the positive
item and a number of sampled negatives). The parameters to be tuned are those
in $\bm{b}$ and $\bm{W}_{r}$ in Eqn. (2).

Transductive setting. The second setting assumes that nearly all the items of
the target domain have appeared in the training set, and we can also learn ID
embeddings since item IDs are available. In this setting, to represent an
item, we combine the textual embedding $\bm{v}_{i_{t+1}}$ and ID embedding
$\bm{e}_{i_{t+1}}$ as the final item representation. Thus, we have the
following prediction probability:

(12)
$P_{T}(i_{t+1}|s)=\operatorname{Softmax}\left(\widetilde{\bm{s}}\cdot(\bm{v}_{i_{t+1}}+\bm{e}_{i_{t+1}})\right),$

where $\widetilde{\bm{s}}$ denotes the enhanced universal sequence
representation by adding ID embeddings in the input (Eqn. (7)). Note that the
rest parameters of the sequence encoder are still fixed in this setting.

For each setting, we optimize the widely used cross-entropy loss to fine-tune
parameters of MoE-enhanced adaptors. After fine-tuning, we predict the
probability distribution of next item for given sequences via Eqn. (11) and
Eqn. (12).

Table 1. Comparison of the transfer learning scenarios and application
settings of several approaches. $1\rightarrow 1$ denotes $1$ source domain to
$1$ target domain, and $M\rightarrow N$ denotes $M$ source domains to $N$
target domains. “Non-OL” denotes that the approach doesn’t require overlapped
users or items.

Methods | Transfer Learning Scenarios | Application Settings
---|---|---
$1\rightarrow 1$ | $M\rightarrow N$ | Non-OL | Transductive | Inductive
S3-Rec (Zhou et al., 2020) | ✗ | ✗ | ✗ | ✔ | ✗
PeterRec (Yuan et al., 2020) | ✔ | ✗ | ✗ | ✔ | ✗
RecGURU (Li et al., 2022) | ✔ | ✗ | ✔ | ✔ | ✗
ZESRec (Ding et al., 2021) | ✔ | ✗ | ✔ | ✗ | ✔
UniSRec (ours) | ✔ | ✔ | ✔ | ✔ | ✔

###  2.5. Discussion

In the literature of recommender systems, a large number of recommendation
models have been developed. Here, we make a brief comparison with related
recommendation models, in order to highlight the novelty and differences of
our approach.

General sequential approaches such as GRU4Rec (Hidasi et al., 2016) and SASRec
(Kang and McAuley, 2018) rely on explicit item IDs to construct the sequential
model, where they assume item IDs are pre-given. These approaches can’t
perform well under the cold-start setting with new items. As a comparison, our
approach aims to construct an ID-agnostic recommendation model that can
capture sequential patterns in a more general form of natural language.

Cross-domain approaches such as RecGURU (Li et al., 2022) propose to leverage
the auxiliary information from source domains to improve the performance on a
target domain. However, most approaches require overlapping users or items as
anchors. Besides, it is not easy to transfer and fuse multiple source domains
for improving the target domain. For our approach, we propose an MoE-enhanced
adaptor mechanism for domain fusion and adaptation based on universal textual
semantics.

Pre-training sequential approaches mainly pre-train their model on sequences
from (1) the current domain (S3-Rec (Zhou et al., 2020), IDA-SR (Mu et al.,
2022)), (2) other domains with overlapping users (PeterRec (Yuan et al.,
2020)) or (3) other closely related domains (ZESRec (Ding et al., 2021)).
However, none of these methods explore how to pre-train universal item and
sequence representations on multiple weakly related or irrelevant domains. In
contrast, our approach can learn more transferable representations that well
generalize to the target domains, from a mixture of multiple source domains.
The comparison of these approaches is presented in Table 1.

##  3\. Experiments

In this section, we first set up the experiments, and then present the results
and analysis.

###  3.1. Experimental Setup

Table 2. Statistics of the datasets after preprocessing. “Avg. $n$” denotes
the average length of item sequences. “Avg. $c$” denotes the average number of
tokens in item text.

Datasets | #Users | #Items | #Inters. | Avg. $n$ | Avg. $c$
---|---|---|---|---|---
Pre-trained | 1,361,408 | 446,975 | 14,029,229 | 13.51 | 139.34
\- Food | 115,349 | 39,670 | 1,027,413 | 8.91 | 153.40
\- CDs | 94,010 | 64,439 | 1,118,563 | 12.64 | 80.43
\- Kindle | 138,436 | 98,111 | 2,204,596 | 15.93 | 141.70
\- Movies | 281,700 | 59.203 | 3,226,731 | 11.45 | 97.54
\- Home | 731,913 | 185,552 | 6,451,926 | 8.82 | 168.89
Scientific | 8,442 | 4,385 | 59,427 | 7.04 | 182.87
Pantry | 13,101 | 4,898 | 126,962 | 9.69 | 83.17
Instruments | 24,962 | 9,964 | 208,926 | 8.37 | 165.18
Arts | 45,486 | 21,019 | 395,150 | 8.69 | 155.57
Office | 87,436 | 25,986 | 684,837 | 7.84 | 193.22
Online Retail | 16,520 | 3,469 | 519,906 | 26.90 | 27.80

Table 3. Performance comparison of different recommendation models. The best
and the second-best performances are denoted in bold and underlined fonts,
respectively. “Improv.” indicates the relative improvement ratios of the
proposed approach over the best performance baselines. “*” denotes that the
improvements are significant at the level of 0.01 with paired $t$-test.

Scenario | Dataset | Metric | SASRec | BERT4Rec | FDSA | S3-Rec | CCDR | RecGURU | ZESRec | UniSRect | UniSRect+ID | Improv.
---|---|---|---|---|---|---|---|---|---|---|---|---
Cross- Domain | Scientific | Recall@10 | 0.1080 | 0.0488 | 0.0899 | 0.0525 | 0.0695 | 0.1023 | 0.0851 |  0.1188* |  0.1235* | +14.35%
NDCG@10 | 0.0553 | 0.0243 | 0.0580 | 0.0275 | 0.0340 | 0.0572 | 0.0475 |  0.0641* |  0.0634* | +10.52%
Recall@50 | 0.2042 | 0.1185 | 0.1732 | 0.1418 | 0.1647 | 0.2022 | 0.1746 |  0.2394* |  0.2473* | +21.11%
NDCG@50 | 0.0760 | 0.0393 | 0.0759 | 0.0468 | 0.0546 | 0.0786 | 0.0670 |  0.0903* |  0.0904* | +15.01%
Pantry | Recall@10 | 0.0501 | 0.0308 | 0.0395 | 0.0444 | 0.0408 | 0.0469 | 0.0454 |  0.0636* |  0.0693* | +38.32%
NDCG@10 | 0.0218 | 0.0152 | 0.0209 | 0.0214 | 0.0203 | 0.0209 | 0.0230 |  0.0306* |  0.0311* | +35.22%
Recall@50 | 0.1322 | 0.1030 | 0.1151 | 0.1315 | 0.1262 | 0.1269 | 0.1141 |  0.1658* |  0.1827* | +38.20%
NDCG@50 | 0.0394 | 0.0305 | 0.0370 | 0.0400 | 0.0385 | 0.0379 | 0.0378 |  0.0527* |  0.0556* | +39.00%
Instruments | Recall@10 | 0.1118 | 0.0813 | 0.1070 | 0.1056 | 0.0848 | 0.1113 | 0.0783 |  0.1189* |  0.1267* | +13.33%
NDCG@10 | 0.0612 | 0.0620 | 0.0796 | 0.0713 | 0.0451 | 0.0681 | 0.0497 | 0.0680 |  0.0748* | $-$
Recall@50 | 0.2106 | 0.1454 | 0.1890 | 0.1927 | 0.1753 | 0.2068 | 0.1387 |  0.2255* |  0.2387* | +13.34%
NDCG@50 | 0.0826 | 0.0756 | 0.0972 | 0.0901 | 0.0647 | 0.0887 | 0.0627 | 0.0912 |  0.0991* | +1.95%
Arts | Recall@10 | 0.1108 | 0.0722 | 0.1002 | 0.1003 | 0.0671 | 0.1084 | 0.0664 | 0.1066 |  0.1239* | +11.82%
NDCG@10 | 0.0587 | 0.0479 | 0.0714 | 0.0601 | 0.0348 | 0.0651 | 0.0375 | 0.0586 | 0.0712 | $-$
Recall@50 | 0.2030 | 0.1367 | 0.1779 | 0.1888 | 0.1478 | 0.1979 | 0.1323 |  0.2049* |  0.2347* | +15.62%
NDCG@50 | 0.0788 | 0.0619 | 0.0883 | 0.0793 | 0.0523 | 0.0845 | 0.0518 | 0.0799 |  0.0955* | +8.15%
Office | Recall@10 | 0.1056 | 0.0825 | 0.1118 | 0.1030 | 0.0549 | 0.1145 | 0.0641 | 0.1013 |  0.1280* | +11.79%
NDCG@10 | 0.0710 | 0.0634 | 0.0868 | 0.0653 | 0.0290 | 0.0768 | 0.0391 | 0.0619 | 0.0831 | $-$
Recall@50 | 0.1627 | 0.1227 | 0.1665 | 0.1613 | 0.1095 | 0.1757 | 0.1113 | 0.1702 |  0.2016* | +14.74%
NDCG@50 | 0.0835 | 0.0721 | 0.0987 | 0.0780 | 0.0409 | 0.0901 | 0.0493 | 0.0769 | 0.0991 | +0.41%
Cross- Platform |  Online Retail | Recall@10 | 0.1460 | 0.1349 | 0.1490 | 0.1418 | 0.1347 | 0.1467 | 0.1103 | 0.1449 |  0.1537* | +3.15%
NDCG@10 | 0.0675 | 0.0653 | 0.0719 | 0.0654 | 0.0620 | 0.0658 | 0.0535 | 0.0677 | 0.0724 | +0.70%
Recall@50 | 0.3872 | 0.3540 | 0.3802 | 0.3702 | 0.3587 | 0.3885 | 0.2750 | 0.3604 | 0.3885 | 0.00%
NDCG@50 | 0.1201 | 0.1131 | 0.1223 | 0.1154 | 0.1108 | 0.1188 | 0.0896 | 0.1149 |  0.1239* | +1.31%

####  3.1.1. Datasets

To evaluate the performance of the proposed approach, we conduct experiments
in both cross-domain setting and cross-platform setting. The statistics of
datasets after preprocessing are summarized in Table 2.

(1) Pre-trained datasets: we select five categories from Amazon review
datasets (Ni et al., 2019), “ _Grocery and Gourmet Food_ ”, “ _Home and
Kitchen_ ”, “ _CDs and Vinyl_ ”, “ _Kindle Store_ ” and “ _Movies and TV_ ”,
as the source domain datasets for pre-training.

(2) Cross-domain datasets: we select another five categories from Amazon
review datasets (Ni et al., 2019), “ _Prime Pantry_ ”, “ _Industrial and
Scientific_ ”, “ _Musical Instruments_ ”, “ _Arts, Crafts and Sewing_ ” and “
_Office Products_ ”, as target domain datasets to evaluate the proposed
approach in cross-domain setting.

(3) Cross-platform datasets: we also select a dataset from different platforms
to evaluate the pre-trained universal sequence representation model in a
cross-platform setting. Online
Retail111https://www.kaggle.com/carrie1/ecommerce-data contains transactions
occurring between 01/12/2010 and 09/12/2011 from a UK-based online retail
platform, which does not contain shared users or items with the Amazon
platform.

Following previous works (Kang and McAuley, 2018; Zhou et al., 2020), we keep
the five-core datasets and filter users and items with fewer than five
interactions for all datasets. Then we group the interactions by users and
sort them by timestamp ascendingly. For item text, we concatenate fields
including _title_ , _categories_ and _brand_ in Amazon dataset and directly
use the _Description_ field in Online Retail dataset. We truncate item text
longer than $512$ tokens.

####  3.1.2. Compared Methods

We compare the proposed approach with the following baseline methods:

$\bullet$ SASRec (Kang and McAuley, 2018) adopts a self-attention network to
capture the user’s preference within a sequence.

$\bullet$ BERT4Rec (Sun et al., 2019) adapts the original text-based BERT
model with the cloze objective for modeling user behavior sequences.

$\bullet$ FDSA (Zhang et al., 2019) proposes to capture item and feature
transition patterns via self-attentive networks.

$\bullet$ S3-Rec (Zhou et al., 2020) pre-trains sequential models via mutual
information maximization objectives for feature fusion.

$\bullet$ CCDR (Xie et al., 2022a) proposes intra-domain and inter-domain
contrastive objects for cross-domain recommendation in matching. We extract
textual tags using TF-IDF algorithm, and then optimize the taxonomy-based
inter-CL objective.

$\bullet$ RecGURU (Li et al., 2022) proposes to pre-train user representations
via autoencoder in an adversarial learning paradigm. In our implementation, we
remove constraints on overlapped users.

$\bullet$ ZESRec (Ding et al., 2021) encodes item text via pre-trained
language model as item representations. ZESRec can be pre-trained on source
domain and directly applied to target domains for zero-shot recommendation.
For a fair comparison, we fine-tune the pre-trained model using item sequences
of the target domain.

For our approach, we firstly pre-train a universal sequence representation
model on the five source datasets that are introduced in Section 3.1.1. We
consider two major variants: (1) UniSRect denotes the model fine-tuned in
inductive setting using only item text; (2) UniSRect+ID denotes the model
fine-tuned in transductive setting using both item ID and item text.

####  3.1.3. Evaluation Settings

To evaluate the performance of the next item prediction task, we adopt two
widely used metrics Recall@$N$ and NDCG@$N$, where $N$ is set to 10 and 50.
Following previous works (Sun et al., 2019; Zhou et al., 2020; Zhao et al.,
2020), we apply the leave-one-out strategy for evaluation. For each user
interaction sequence, the last item is used as the test data, the item before
the last one is used as the validation data, and the remaining interaction
records are used for training. We rank the ground-truth item of each sequence
among all the other items for evaluation on test set, and finally report the
average score of all test users.

####  3.1.4. Implementation Details

We implement UniSRec using a popular open-source recommendation library
RecBole222https://recbole.io (Zhao et al., 2021). To ensure a fair comparison,
we optimize all the methods with Adam optimizer and carefully search the
hyper-parameters of all the compared methods. The batch size is set to 2,048.
We adopt early stopping with the patience of 10 epochs to prevent overfitting,
and NDCG@10 is set as the indicator. We tune the learning rate in
$\\{0.0003,0.001,0.003,0.01\\}$ and the embedding dimension in
$\\{64,128,300\\}$. We pre-train the proposed approach for $300$ epochs with
$\lambda=1e^{-3}$ and $G=8$ experts.

###  3.2. Overall Performance

We compare the proposed approach with the baseline methods on the five cross-
domain datasets and one cross-platform dataset. Note that, we fine-tune the
same pre-trained universal sequence representation model on these six datasets
for our approach. The results are reported in Table 3.

For the baseline methods, text-enhanced sequential recommendation methods
(_i.e.,_ FDSA and S3-Rec) perform better than the traditional sequential
recommendation methods (_i.e.,_ SASRec and BERT4Rec) on several datasets,
since item texts are used as auxiliary features to improve the performance.
The cross-domain methods CCDR and RecGURU do not perform well, because they
are still in a transductive setting. These methods become less effective
without explicit overlapping users between source and target domains. ZESRec
utilizes PLMs to encode item texts, but mainly reuses existing architectures
and pre-training tasks, which can’t fully leverage multi-domain interaction
data for improving the target domain.

Finally, by comparing the proposed approach UniSRect+ID with all the
baselines, it is clear that UniSRect+ID achieves the best performance in
almost all the cases. Different from these baselines, we derive universal
sequence representations via pre-training on multi-domain datasets. With the
specially designed parametric whitening module and MoE-enhanced adaptor
module, the learned universal item representations are more isotropic and
suitable for domain fusion and adaptation. Especially, the results on cross-
platform evaluation (_i.e.,_ Online Retail dataset) show that our approach can
be effectively transferred to a different platform via universal sequence
representation pre-training. Besides, the model fine-tuned in inductive
setting (without item IDs, UniSRect) also has a comparable performance with
other baselines. This further illustrates the effectiveness of the proposed
universal SRL approach.

Figure 2. Performance comparison w.r.t. different pre-training datasets on
“Scientific” and “Online Retail”. “All” denotes the model pre-trained on all
five datasets, and “None” denotes the training from scratch.

###  3.3. Further Analysis

####  3.3.1. Universal Pre-training Analysis

In this part, we compare and analyze the effectiveness of universal pre-
training. Specifically, we would like to examine whether the universal model
pre-trained on multiple datasets performs better than those models pre-trained
on one single source dataset, or those models without pre-training. The
experimental results are reported in Figure 2.

We can see that the model pre-trained on all the five datasets achieves better
performance than any model that pre-trained on a single dataset or without
pre-training. The results show that with the proposed universal sequence
representation learning approach, the pre-trained model can capture semantic
sequential patterns from multiple source domains to improve recommendation on
target domains or platforms.

Figure 3. Ablation study of UniSRec variants on “Scientific” and “Online
Retail”.

####  3.3.2. Ablation Study

In this part, we analyze how each of the proposed techniques or components
affects the final performance. We prepare four variants of the proposed
UniSRec model for comparisons, including (1) $w/o$ PW that replaces parametric
whitening with traditional linear layer, _i.e.,_ replacing Eqn. (2) by
$\widetilde{\bm{x}}_{i}=\bm{x}_{i}\cdot\bm{W}_{1}-\bm{b}$, (2) $w/o$ MoE
without MoE-enhanced adaptor ($G=1$ in Eqn. (3)), (3) $w/o$ S-I without
sequence-item contrastive task, and (4) $w/o$ S-S without sequence-sequence
contrastive task.

The experimental results of the proposed approach UniSRec and its variants are
reported in Figure 3. We can observe that all the proposed components are
useful to improve the recommendation performance. The variant $w/o$ MoE has
poor performances because the MoE-enhance adaptor is the key component to
improve the representation capacity for domain fusion and adaptation.

Figure 4. Performance comparison w.r.t. long-tail items on the “Scientific”
and “Online Retail” datasets. The bar graph represents the number of
interactions in test data for each group. The line chart represents the
improvement ratios for Recall@10 compared with SASRec.

####  3.3.3. Performance Comparison w.r.t. Long-tail Items.

One motivation to learn universal and transferable sequence representations is
to alleviate the cold-start recommendation issue. To verify this, we split the
test data into different groups according to the popularity of ground-truth
items in the training data, and then compare the improved ratio of Recall@10
score (_w.r.t._ the baseline SASRec) in each group. From Figure 4, we can
observe that the proposed approach outperforms the other baseline models in
most cases, especially when the ground truth item is unpopular, _e.g.,_ group
[0, 5) on Industrial and Scientific datasets and group [0, 20) on Online
Retail dataset. The results show that long-tail items can benefit from the
learned universal sequence representations.

###  3.4. Case Study

As shown in Table 3, we can see that our approach can achieve good performance
in a cross-platform setting (from _Amazon_ to _Online Retail_). To our
knowledge, in the literature of recommender systems, there are few studies
that can conduct cross-platform recommendation (Zhao et al., 2014). It is
interesting to study what kind of knowledge is actually transferred across
different platforms. For this purpose, we present an illustrative case in
Figure 5.

This example presents two short sequences of two different users from Amazon
and Online Retail, respectively. It can be observed that our approach doesn’t
rely on explicit item IDs to capture the user preference. Instead, it tries to
capture the semantic associations by modeling the sequential patterns.
Especially, the two short sequences correspond to a semantic transition from
the keyword of “ _dog_ ” to the keyword “ _cat_ ” as shown in the item title.
It shows that our approach can capture universal sequential patterns across
platforms in terms of general textual semantics.

Note that there might be also semantic correlations among other keywords.
Here, we select the keywords of “dog” and “cat” just for simplifying the
illustration. We leave a deep investigation of the universal representations
as future work.

Figure 5. The purchase history of a user in the source platform (top) and the
purchase history within an anonymous session in the target platform (bottom).
There are naturally no overlapping users and items between the two platforms.
This case shows that UniSRec can capture the universal semantic sequence
pattern (_e.g.,_ “Dog $\rightarrow$ Cat”) from large-scale pre-training, which
helps improve the recommendation performance of the target platform.

##  4\. Related Work

Sequential recommendation. To alleviate information overload, recommender
systems are widely studied and deployed in various real-world services.
Specially, it has been shown that sequential behaviors are important signals
to reflect user preferences, and thus sequential recommendation has recieved
much attention from both research and industry community (Rendle et al., 2010;
Hidasi et al., 2016). Early works on this topic adopt the Markov Chain
assumption by estimating item-item transition probability matrices (Rendle et
al., 2010). With the development of deep learning, Hidasi et al. (2016)
firstly introduce Gated Recurrent Units (GRU) to model sequential behaviors.
Then various kinds of neural network techniques are proposed to encode
interaction sequences, such as Recurrent Neural Network (RNN) (Li et al.,
2017), Transformer (Kang and McAuley, 2018; Sun et al., 2019; He et al., 2021;
Hou et al., 2022), Multilayer Perceptron (MLP) (Zhou et al., 2022) and Graph
Neural Network (GNN) (Wu et al., 2019; Chang et al., 2021). Besides directly
mining sequential patterns of IDs, some works are proposed to model sequences
with rich features (Zhang et al., 2019; Zhou et al., 2020) or additional self-
supervised signals (Yu et al., 2022; Zhou et al., 2020; Xie et al., 2022b).
However, item representations and model parameters of these methods are
usually restricted to specific data domains or platforms, making it difficult
to leverage multi-domain data to improve sequential recommendation.

Transfer learning in recommender systems. To deal with the data sparsity and
cold-start issues in the recommender systems, various works aim to leverage
behavior information from other domains (Zhu et al., 2021) or platforms (Lin
et al., 2019) to improve recommendation performance of the target domain or
both domains (Zhu et al., 2019). Most works rely on explicit overlapping data
for conducting the transfer across domains, such as common users (Hu et al.,
2018) or items (Singh and Gordon, 2008; Zhu et al., 2019), social networks
(Lin et al., 2019) and attributes (Tang et al., 2012). Recently, some works
attempt to learn universal user representations for different user-oriented
downstream tasks (Yuan et al., 2020). However, the need of explicitly shared
data still limits the application scope of the above methods. In this work, we
propose to represent items in a universal way via text-based PLMs for domain
adaptation. Different from the recently proposed zero-shot recommendation
methods (Ding et al., 2021), we do not require that the source and target
domains are closely related. With carefully designed universal pre-training
tasks and MoE-enhanced adaptor architecture, the proposed universal SRL
approach can be pre-trained on data from multiple source domains, and further
generalize to different domains without explicitly shared anchors.

##  5\. Conclusions

In this paper, we propose the universal sequence representation learning
approach for recommender systems, named _UniSRec_. Different from existing
sequential recommendation methods that reply on explicit item IDs for
representation learning, the proposed approach UniSRec utilizes item texts to
learn more transferable representations for sequential recommendation.
Specifically, we design a lightweight architecture based on parametric
whitening and MoE-enhanced adaptor to learn the universal item
representations. We further design two contrastive pre-training tasks to learn
universal sequence representations from multi-domain sequences. For future
work, we will consider collecting more recommendation data to train larger
user behavior models. Besides, we will explore more kinds of side information
to represent items and improve sequence representation learning, _e.g.,_
images and videos.

###### Acknowledgements.

This work was partially supported by National Natural Science Foundation of
China under Grant No. 61872369, Beijing Natural Science Foundation under Grant
No. 4222027, and Beijing Outstanding Young Scientist Program under Grant No.
BJJWZYJH012019100020098. This work was partially supported by Beijing Academy
of Artificial Intelligence (BAAI). Yupeng Hou was also partially supported by
Alibaba Group through Alibaba Research Intern Program. Xin Zhao is the
corresponding author.

## References

  * (1)
  * Bengio et al. (2013) Yoshua Bengio, Aaron C. Courville, and Pascal Vincent. 2013\.  Representation Learning: A Review and New Perspectives.  _TPAMI_ (2013).
  * Chang et al. (2021) Jianxin Chang, Chen Gao, Yu Zheng, Yiqun Hui, Yanan Niu, Yang Song, Depeng Jin, and Yong Li. 2021\.  Sequential Recommendation with Graph Neural Networks. In _SIGIR_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL_.
  * Ding et al. (2021) Hao Ding, Yifei Ma, Anoop Deoras, Yuyang Wang, and Hao Wang. 2021.  Zero-Shot Recommender Systems.  _arXiv preprint arXiv:2105.08318_ (2021).
  * He et al. (2021) Zhankui He, Handong Zhao, Zhe Lin, Zhaowen Wang, Ajinkya Kale, and Julian McAuley. 2021\.  Locally constrained self-attentive sequential recommendation. In _CIKM_.
  * Hidasi et al. (2016) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016.  Session-based Recommendations with Recurrent Neural Networks. In _ICLR_.
  * Hou et al. (2022) Yupeng Hou, Binbin Hu, Zhiqiang Zhang, and Wayne Xin Zhao. 2022\.  CORE: Simple and Effective Session-based Recommendation within Consistent Representation Space. In _SIGIR_.
  * Hu et al. (2018) Guangneng Hu, Yu Zhang, and Qiang Yang. 2018.  CoNet: Collaborative Cross Networks for Cross-Domain Recommendation. In _CIKM_.
  * Huang et al. (2021) Junjie Huang, Duyu Tang, Wanjun Zhong, Shuai Lu, Linjun Shou, Ming Gong, Daxin Jiang, and Nan Duan. 2021\.  WhiteningBERT: An Easy Unsupervised Sentence Embedding Approach. In _Findings of EMNLP_.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian J. McAuley. 2018.  Self-Attentive Sequential Recommendation. In _ICDM_.
  * Li et al. (2020) Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020\.  On the Sentence Embeddings from Pre-trained Language Models. In _EMNLP_.
  * Li et al. (2022) Chenglin Li, Mingjun Zhao, Huanming Zhang, Chenyun Yu, Lei Cheng, Guoqiang Shu, Beibei Kong, and Di Niu. 2022\.  RecGURU: Adversarial Learning of Generalized User Representations for Cross-Domain Recommendation. In _WSDM_.
  * Li et al. (2017) Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017\.  Neural Attentive Session-based Recommendation. In _CIKM_.
  * Lin et al. (2019) Tzu-Heng Lin, Chen Gao, and Yong Li. 2019.  CROSS: Cross-platform Recommendation for Social E-Commerce. In _SIGIR_.
  * Lin et al. (2022) Zihan Lin, Changxin Tian, Yupeng Hou, and Wayne Xin Zhao. 2022\.  Improving Graph Collaborative Filtering with Neighborhood-enriched Contrastive Learning. In _TheWebConf_.
  * Mu et al. (2022) Shanlei Mu, Yupeng Hou, Wayne Xin Zhao, Yaliang Li, and Bolin Ding. 2022.  ID-Agnostic User Behavior Pre-training for Sequential Recommendation.  _arXiv preprint arXiv:2206.02323_ (2022).
  * Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian J. McAuley. 2019.  Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects. In _EMNLP_.
  * Rendle et al. (2010) Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010\.  Factorizing personalized Markov chains for next-basket recommendation. In _WWW_.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017.  Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. In _ICLR_.
  * Singh and Gordon (2008) Ajit Paul Singh and Geoffrey J. Gordon. 2008.  Relational learning via collective matrix factorization. In _SIGKDD_.
  * Su et al. (2021) Jianlin Su, Jiarun Cao, Weijie Liu, and Yangyiwen Ou. 2021\.  Whitening Sentence Representations for Better Semantics and Faster Retrieval.  _arXiv preprint arXiv:2103.15316_ (2021).
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. In _CIKM_.
  * Tang et al. (2020) Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. 2020\.  Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations. In _RecSys_.
  * Tang and Wang (2018) Jiaxi Tang and Ke Wang. 2018.  Personalized Top-N Sequential Recommendation via Convolutional Sequence Embedding. In _WSDM_.
  * Tang et al. (2012) Jie Tang, Sen Wu, Jimeng Sun, and Hang Su. 2012\.  Cross-domain collaboration recommendation. In _SIGKDD_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017\.  Attention is All you Need. In _NeurIPS_.
  * Wu et al. (2019) Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019\.  Session-Based Recommendation with Graph Neural Networks. In _AAAI_.
  * Xie et al. (2022a) Ruobing Xie, Qi Liu, Liangdong Wang, Shukai Liu, Bo Zhang, and Leyu Lin. 2022a.  Contrastive Cross-domain Recommendation in Matching. In _SIGKDD_.
  * Xie et al. (2022b) Xu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu, Jinyang Gao, Bolin Ding, and Bin Cui. 2022b.  Contrastive learning for sequential recommendation. In _ICDE_.
  * Yu et al. (2022) Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Jundong Li, and Zi Huang. 2022\.  Self-Supervised Learning for Recommender Systems: A Survey.  _arXiv preprint arXiv:2203.15876_ (2022).
  * Yuan et al. (2020) Fajie Yuan, Xiangnan He, Alexandros Karatzoglou, and Liguang Zhang. 2020.  Parameter-Efficient Transfer from Sequential Behaviors for User Modeling and Recommendation. In _SIGIR_.
  * Zhang et al. (2019) Tingting Zhang, Pengpeng Zhao, Yanchi Liu, Victor S. Sheng, Jiajie Xu, Deqing Wang, Guanfeng Liu, and Xiaofang Zhou. 2019\.  Feature-level Deeper Self-Attention Network for Sequential Recommendation. In _IJCAI_.
  * Zhao et al. (2020) Wayne Xin Zhao, Junhua Chen, Pengfei Wang, Qi Gu, and Ji-Rong Wen. 2020.  Revisiting Alternative Experimental Settings for Evaluating Top-N Item Recommendation Algorithms. In _CIKM_.
  * Zhao et al. (2014) Wayne Xin Zhao, Yanwei Guo, Yulan He, Han Jiang, Yuexin Wu, and Xiaoming Li. 2014\.  We know what you want to buy: a demographic-based system for product recommendation on microblogs. In _KDD_.
  * Zhao et al. (2021) Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Yushuo Chen, Xingyu Pan, Kaiyuan Li, Yujie Lu, Hui Wang, Changxin Tian, Yingqian Min, Zhichao Feng, Xinyan Fan, Xu Chen, Pengfei Wang, Wendi Ji, Yaliang Li, Xiaoling Wang, and Ji-Rong Wen. 2021.  RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms. In _CIKM_.
  * Zhou et al. (2020) Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. 2020\.  S3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization. In _CIKM_.
  * Zhou et al. (2022) Kun Zhou, Hui Yu, Wayne Xin Zhao, and Ji-Rong Wen. 2022\.  Filter-enhanced MLP is All You Need for Sequential Recommendation. In _TheWebConf_.
  * Zhu et al. (2019) Feng Zhu, Chaochao Chen, Yan Wang, Guanfeng Liu, and Xiaolin Zheng. 2019.  DTCDR: A Framework for Dual-Target Cross-Domain Recommendation. In _CIKM_.
  * Zhu et al. (2021) Feng Zhu, Yan Wang, Chaochao Chen, Jun Zhou, Longfei Li, and Guanfeng Liu. 2021\.  Cross-Domain Recommendation: Challenges, Progress, and Prospects. In _IJCAI_.


Proceedings of the Second Annual LHCP ATL-PHYS-PROC-2014-097 August 12, 2014

Beyond Standard Model Higgs boson physics with the ATLAS experiment at the LHC

Nikolaos Rompotis

On behalf of the ATLAS Experiment,
Department of Physics,
University of Washington, Box 351560, Seattle WA 98195-1560, USA

> ABSTRACT
>
>
>
>
> The search for evidence of beyond Standard Model Higgs bosons is an integral
> part of the Higgs boson studies at the LHC. This article reviews recent
> beyond Standard Model Higgs boson searches using Run I LHC proton-proton
> collision data recorded by the ATLAS detector. In particular, searches for
> Higgs boson cascades, double Higgs boson production, scalar particles
> decaying to $\gamma\gamma$ pairs, flavor changing neutral currents involving
> Higgs bosons, and Higgs bosons decaying to invisible particles are
> discussed. No significant deviations from the background expectations are
> found and corresponding constraints on physics beyond the Standard Model are
> obtained.

> PRESENTED AT
>
>
>
>
> The Second Annual Conference
>
> on Large Hadron Collider Physics
>
> Columbia University, New York, U.S.A
>
> June 2-7, 2014

##  1 Introduction

The recent observation of a new particle with mass 125–126 GeV at the LHC [1,
2] opens a new era for particle physics. The detailed study of the production
and decay modes of the new particle provides invaluable input to answer the
question of whether this is indeed the long-sought Standard Model (SM) Higgs
boson [3, 4, 5, 6, 7, 8]. The first measurements indicate that the new
particle is indeed compatible with the SM Higgs boson, see e.g., [9].
Nevertheless, many more measurements and data will be needed to extract
reliable conclusions. This task is further complicated by the fact that many
beyond SM physics scenarios include a SM-like Higgs boson, which is part of an
extended scalar sector. In that case, searches for beyond SM Higgs bosons are
very interesting, since they provide direct information on a possibly extended
scalar sector, and hence they are complementary to the precise measurements of
the properties of the new particle.

This article reviews some of the recent searches for beyond SM Higgs bosons or
exotic properties of the recently discovered Higgs boson using proton-proton
collision data at 7 and 8 TeV center-of-mass energies as recorded by the ATLAS
detector [10].

##  2 Higgs Bosons in beyond SM physics scenarios

The scalar sector of the SM consists of a complex Higgs doublet, which after
the electroweak symmetry breaking leaves a single scalar boson in the theory.
Possible extensions of the scalar sector are restricted due to the fact that,
in general, they include unacceptably large corrections to the well-measured
quantity $\rho\equiv m_{W}^{2}/(m_{Z}^{2}\cos^{2}\theta)$, where $\theta$ is
the weak mixing angle [11]. The addition of doublets or singlets is such that
leading order corrections to $\rho$ vanish and, hence they can be compatible
with precision electroweak tests [12].

The introduction of one additional Higgs doublet defines a class of models,
which are collectively known as 2 Higgs doublet models (2HDM) [13]. These
models introduce five Higgs bosons, three of which are neutral and two of
which are charged. The Minimal Supersymmetric Standard Model (MSSM) [14, 15,
16, 17], which is a very popular realization of supersymmetry, is a particular
case of a 2HDM. The MSSM, as well as other 2HDMs, are compatible with existing
measurements, including the recently discovered Higgs boson [18, 19].

Apart from the introduction of new particles, extensions of the SM scalar
sector may affect the properties of the SM-like Higgs boson discovered at the
LHC. Enhancement of rare decays, completely new decays and production
mechanisms, and different couplings with respect to the SM expectations are
possible and may indicate connections to other puzzles, such as dark matter
(e.g., see Ref. [20] and references therein).

In the following, only a few recent ATLAS searches are described. As a
convention, the symbol $h$ will refer to the newly discovered Higgs boson with
mass $\sim 125$ GeV.

##  3 Some recent searches for beyond SM Higgs bosons with ATLAS

The existence of an extended scalar sector with more than one Higgs bosons
opens the possibility of cascade decays in which heavier Higgs bosons decay
into lighter ones. In this context, a Higgs boson cascade has been looked for
in Ref. [21] using 20.3 fb-1 of 8 TeV proton-proton collision data. A heavy
Higgs boson is produced via gluon-gluon fusion and initiates a cascade decay
that includes a charged scalar, $H^{\pm}$, a light neutral Higgs boson, $h$,
and $W$ bosons: $H\rightarrow W^{-}H^{+}\rightarrow W^{-}W^{+}h$, see Fig. 1.
The light neutral Higgs boson decays to $b\overline{b}$ and it is assumed to
have mass 125 GeV. The first of the $W$ bosons decays leptonically and
provides a way to trigger the events whereas the second one decays
hadronically maximizing the branching ratio. The final state shares the same
topology with $t\overline{t}$ events with similar $W$ boson decays, which is
the main background process for the search. A multivariate discriminant is
employed to exploit the kinematic differences between signal and
$t\overline{t}$ events. Upper limits on the production cross section of heavy
neutral Higgs boson, $H$, times the branching ratio BR$(H\rightarrow
W^{\mp}H^{\pm}\rightarrow W^{\mp}W^{\pm}h\rightarrow
W^{\mp}W^{\pm}b\overline{b})$ are derived as a function of its mass and the
charged scalar mass, as shown in Fig. 1.

Figure 1: The topology of a Higgs cascade search, described in Ref. [21], is
shown in (a). The observed upper limit for this process is shown in (b).

The pair production of Higgs bosons is very interesting and can be enhanced
due to the resonant decay of a heavy CP-even Higgs boson ($H\rightarrow hh$),
e.g., in a 2HDM, or even in a non-resonant way, e.g., in composite Higgs boson
models [22]. The analysis described in Ref. [23] uses 20.3 fb-1 of 8 TeV
proton-proton collision data to look for $hh$ production in the $h\rightarrow
b\overline{b}$ and $h\rightarrow\gamma\gamma$ channel. Both resonant and non-
resonant anomalous production of $hh$ pairs is searched for. Upper limits on
the production of a narrow-width heavy scalar boson decaying to $hh$ as a
function of its mass are shown in Fig. 2. The cross section for non-resonant
$hh$ production is constrained to be less than $2.2$ pb at 95% confidence
level. As a reminder, the SM $hh$ production cross section is $\sim 10$ fb,
i.e., about two orders of magnitude smaller than the sensitivity of this
search.

Figure 2: Upper limits from the search for a scalar particle decaying to $hh$
in the $b\overline{b}\gamma\gamma$ channel from Ref [23].

A search for a scalar particle that decays to a $\gamma\gamma$ pair and is in
the mass range 65–600 GeV is described in Ref. [24] and uses 20.3 fb-1 of 8
TeV proton-proton collision data. This analysis considers the SM-like Higgs
boson at $\sim 125$ GeV as a background to the search. Analytical functions
are employed to describe the shape of the $\gamma\gamma$ invariant mass
similar to those used in the SM Higgs boson search in the $\gamma\gamma$
channel [1]. The $\gamma\gamma$ mass range of this search is constrained by
trigger and background estimation requirements from the low mass side and by
the data statistics in the high mass side. Upper limits on the fiducial cross
section times the branching ratio $H\rightarrow\gamma\gamma$ are quoted, see
Fig. 3.

Figure 3: Upper limits at 95% confidence level of a search for a scalar
particle decaying to a $\gamma\gamma$ pair from Ref. [24].

Flavor changing neutral currents (FCNC) are in general enhanced in extended
Higgs sectors. In several occasions, models can be built that evade the tight
constraints on FCNC by flavor physics, like the type-III 2HDM [19]. In those
cases, the $thc$ coupling is predicted to be sizeable and close to the LHC
sensitivity [25]. A search for top quark decaying to $hq$, where $q$ denotes
some light quark, is reported in Ref. [26] and uses 4.7 fb-1 of 7 TeV and 20.3
fb-1 of 8 TeV proton-proton collision data. This analysis looks for
$t\overline{t}$ production in which one top decays to $hq$ and the other
semileptonically or in a fully hydronic way. An example of the $\gamma\gamma$
invariant mass after all selection requirements for the case where one of the
tops decays in a fully hydronic way is shown in Fig. 4. The observed
(expected) upper limit at 95% confidence level for the flavor changing
branching ratio is BR($t\rightarrow hq)<0.79(0.51)$%, see Fig. 4.

Figure 4: The distribution of the $\gamma\gamma$ mass for
$t\overline{t}\rightarrow hqWb$ candidates in which the $W$ boson decays
hadronically is shown in (a). Observed and expected CLs for the $t\rightarrow
hq$ search is shown in (b). Both figures are from Ref. [26].

A search for invisible decays of the Higgs boson discovered at the LHC can
provide access to dark matter through the Higgs portal [27]. A search reported
in Ref. [28] uses 4.5 fb-1 of 7 TeV and 20.3 fb-1 of 8 TeV proton-proton
collision data to look for the associated production of a Higgs boson with a
$Z$ boson as shown pictorially in Fig. 5. The $Z$ boson decays to an electron
or a muon pair and the Higgs boson is assumed to decay to invisible particles.
The final event sample is dominated by $Z$+jet events and the missing
transverse momentum distribution is examined for discrepancies with respect to
the SM prediction, see Fig. 5 Upper limits for the production of a Higgs boson
in association with a $Z$ boson, assuming that the Higgs boson decays always
to invisible particles, are shown in Fig. 5. This result is combined with
indirect constraints on invisible Higgs boson decays from Higgs boson coupling
measurements [29]. The observed (expected) combined constraint for the 125 GeV
Higgs boson on the invisible branching ratio is
BR$(h\rightarrow\mathrm{invisible})<37(39)$% at 95% confidence level.

Figure 5: The topology of the search for invisible Higgs boson decays
described in Ref. [28] is shown in (a). The missing transverse momentum after
the full selection and the upper limits on this process are shown in (b) and
(c), respectively.

##  4 Conclusions

The search for beyond SM Higgs bosons is highly motivated and has just started
having sensitivity to realistic scenarios. The discovery of a Higgs boson with
mass around 125 GeV has opened new possibilities for searches, especially
those that include the new particle in the final state. More data are needed
in order to explore the possibility of an extended scalar sector and, hence
the community is looking forward to the new LHC run.

## References

  * [1] ATLAS Collaboration, Phys. Lett. B 716, 1 (2012) [arXiv:1207.7214].
  * [2] CMS Collaboration, Phys. Lett. B 716, 30 (2012) [arXiv:1207.7235].
  * [3] F. Englert and R. Brout, Phys. Rev. Lett. 13, 321 (1964).
  * [4] P. W. Higgs, Phys. Lett. 12, 132 (1964).
  * [5] P. W. Higgs, Phys. Rev. Lett. 13, 508 (1964).
  * [6] G. S. Guralnik, C. R. Hagen and T. W. B. Kibble, Phys. Rev. Lett. 13, 585 (1964).
  * [7] P. W. Higgs, Phys. Rev. 145, 1156 (1966).
  * [8] T. W. B. Kibble, Phys. Rev. 155, 1554 (1967).
  * [9] ATLAS Collaboration, Phys. Lett. B 726, 88 (2013) [arXiv:1307.1427].
  * [10] ATLAS Collaboration, JINST 3, S08003 (2008).
  * [11] J. F. Gunion, H. E. Haber, G. L. Kane and S. Dawson, Front. Phys. 80, 1 (2000).
  * [12] M. Baak, M. Goebel, J. Haller, A. Hoecker, D. Kennedy, R. Kogler, K. Moenig and M. Schott et al., Eur. Phys. J. C 72, 2205 (2012) [arXiv:1209.2716].
  * [13] T. D. Lee, Phys. Rev. D 8, 1226 (1973).
  * [14] P. Fayet, Phys. Lett. B 64, 159 (1976).
  * [15] P. Fayet, Phys. Lett. B 69, 489 (1977).
  * [16] G. R. Farrar and P. Fayet, Phys. Lett. B 76, 575 (1978).
  * [17] P. Fayet, Phys. Lett. B 84, 416 (1979).
  * [18] P. Bechtle, S. Heinemeyer, O. Stal, T. Stefaniak, G. Weiglein and L. Zeune, Eur. Phys. J. C 73, 2354 (2013) [arXiv:1211.1955].
  * [19] G. C. Branco, P. M. Ferreira, L. Lavoura, M. N. Rebelo, M. Sher and J. P. Silva, Phys. Rept. 516, 1 (2012) [arXiv:1106.0034].
  * [20] D. Curtin, R. Essig, S. Gori, P. Jaiswal, A. Katz, T. Liu, Z. Liu and D. McKeen et al., [arXiv:1312.4992].
  * [21] ATLAS Collaboration, Phys. Rev. D 89, 032002 (2014) [arXiv:1312.1956].
  * [22] M. Gillioz, R. Grober, C. Grojean, M. Muhlleitner and E. Salvioni, JHEP 1210, 004 (2012) [arXiv:1206.7120].
  * [23] ATLAS Collaboration, [arXiv:1406.5053].
  * [24] ATLAS Collaboration, [arXiv:1407.6583].
  * [25] K. -F. Chen, W. -S. Hou, C. Kao and M. Kohda, Phys. Lett. B 725, 378 (2013) [arXiv:1304.8037].
  * [26] ATLAS Collaboration, JHEP 1406, 008 (2014) [arXiv:1403.6293].
  * [27] B. Patt and F. Wilczek, [arXiv:hep-ph/0605188].
  * [28] ATLAS Collaboration, Phys. Lett. B 732, 8 (2014) [arXiv:1402.3051].
  * [29] ATLAS collaboration, ATLAS-CONF-2014-010, https://cds.cern.ch/record/1670531 .