# ID-centric pre-training for recommendation

Yiqing Wu<EMAIL_ADDRESS>Institute of Computing Technology, Chinese
Academy of Sciences and University of Chinese Academy of
SciencesBeijingBeijingChina ,  Ruobing Xie<EMAIL_ADDRESS>TencentBeijingBeijingChina ,  Zhao Zhang<EMAIL_ADDRESS>Institute of
Computing Technology, Chinese Academy of SciencesBeijingBeijingChina ,  Xu
Zhang<EMAIL_ADDRESS>Institute of Computing Technology, Chinese
Academy of SciencesBeijingBeijingChina ,  Fuzhen Zhuang
<EMAIL_ADDRESS>Institute of Artificial Intelligence, Beihang
UniversityBeijingBeijingChina ,  Leyu Lin<EMAIL_ADDRESS>Tencent
BeijingBeijingChina ,  zhanhui Kang<EMAIL_ADDRESS>Tencent
BeijingBeijingChina and  Yongjun Xu<EMAIL_ADDRESS>Institute of Computing
Technology, Chinese Academy of Sciences BeijingBeijingChina

(2018; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Classical sequential recommendation models generally adopt ID embeddings to
store knowledge learned from user historical behaviors and represent items.
However, these unique IDs are challenging to be transferred to new domains.
With the thriving of pre-trained language model (PLM), some pioneer works
adopt PLM for pre-trained recommendation, where modality information (e.g.,
text) is considered universal across domains via PLM. Unfortunately, the
behavioral information in ID embeddings is still verified to be dominating in
PLM-based recommendation models compared to modality information and thus
limits these models’ performance. In this work, we propose a novel ID-centric
recommendation pre-training paradigm (IDP), which directly transfers
informative ID embeddings learned in pre-training domains to item
representations in new domains. Specifically, in pre-training stage, besides
the ID-based sequential model for recommendation, we also build a Cross-domain
ID-matcher (CDIM) learned by both behavioral and modality information. In the
tuning stage, modality information of new domain items is regarded as a cross-
domain bridge built by CDIM. We first leverage the textual information of
downstream domain items to retrieve behaviorally and semantically similar
items from pre-training domains using CDIM. Next, these retrieved pre-trained
ID embeddings, rather than certain textual embeddings, are directly adopted to
generate downstream new items’ embeddings. Through extensive experiments on
real-world datasets, both in cold and warm settings, we demonstrate that our
proposed model significantly outperforms all baselines. Codes will be released
upon acceptance.

Pre-trained Recommendation, ID-based Recommendation, Multi-domain
Recommendation

††copyright: acmlicensed††journalyear: 2018††doi: XXXXXXX.XXXXXXX††journal:
JACM††journalvolume: 37††journalnumber: 4††article: 111††publicationmonth:
8††ccs: Information systems Recommender systems

##  1\. Introduction

Sequential recommendation (SR) takes user‘s historical behavior sequence
chronologically as input and outputs the next predicted items according to
user preference (Kang and McAuley, 2018; Sun et al., 2019) and is applied to
websites’ recommender systems. Classical SR models mostly rely on ID
embeddings to represent items (i.e., projecting the unique ID of an item into
a dense embedding).

Figure 1. A example of ID-centric pre-training recommendation. The central
idea is to select related pre-training ID embeddings to generate item
embeddings in new domains.

Undoubtedly, the behavioral information hidden in user-item interactions forms
the foundation of recommendation. ID embeddings + ID-based sequential model
(e.g., (Kang and McAuley, 2018)) are perfectly suitable for storing such
information in SR and are dominating in practical SR systems. However, such an
ID-centric SR model cannot well deal with the cross-domain recommendation,
since the well-learned unique ID embeddings are challenging to directly
transfer to downstream new domains. Some efforts have explored cross-domain
recommendation (Hu et al., 2018; Tsai, 2018; Zhu et al., [n. d.], 2022), while
most of them rely on certain overlapping users or items with customized
networks, which are limited and not that universal. In fact, the
untransferability of ID embeddings is becoming a stumbling block in building a
universal SR model for multiple domains.

To overcome the untransferability of ID embeddings, some works attempt to
bypass IDs with modality information (e.g., texts). Recently, pre-trained
models have achieved substantial advancements CV and NLP fields (Devlin et
al., 2019; Mann et al., 2020; Liu et al., 2021). Inspired by this, a series of
pioneering initiatives have endeavored to utilize the abundant knowledge of
pre-trained language models (PLM) to address cross-domain adaptation and cold-
start problems in recommendation. Some works infer user preferences from
textual modality information (Hou et al., 2022; Li et al., 2023b). For
example, UniSRec (Hou et al., 2022) relies on item textual information with
PLM and parameter whitening to represent items in multiple domains. Some
studies further adopt PLM for sequential behavior modeling (Cui et al., 2022;
Zhang et al., 2023). However, some works demonstrate that there still exists a
huge gap between PLM (textual information) and recommendation (behavioral
information) (Gao et al., 2023; Liu et al., 2023; Hou et al., 2023). The help
of common knowledge in PLMs is marginal when we have sufficient user-item
interactions.

Despite the remarkable progress achieved by PLM-centric recommendation models,
we conclude three key factors that constrain the practical usage of existing
PLM-based recommendation models for multi-domain recommendation: (1)
Behavioral information in ID embeddings has been verified to be dominating for
SR compared to modality information, especially for relatively dense datasets.
The static common knowledge in PLM cannot directly infer dynamic personalized
preferences in SR. Sometimes even a well-tuned single-domain SR model (e.g.,
SASRec (Kang and McAuley, 2018)) can achieve better results (Gao et al.,
2023). Modality representations cannot fully replace ID embeddings, and
directly representing items with texts may hinder sufficient model training.
(2) Item modality information may contain noise and conflicts, which cannot
accurately and comprehensively represent items (i.e., textual correlation does
not always imply behavioral correlation). Even items with the same modality
information may represent different behavior patterns. (3) Currently,
industrial recommendation models are mostly based on IDs. Considering the
efficiency and costs, PLM-based models are too heavy and cost-intensive in
practical systems. Despite the great temptation of using LLM to “conquer”
recommendation, we still argue that: ID-based sequential modeling is still the
best practical and most efficient choice for recommendation currently, while
modality information is more suitable as supplementary features for zero-shot
scenarios.

Based on the above observations, we believe that establishing an ID-centric
Pre-trained Recommendation Model is crucial. Unfortunately, there have been no
prior attempts in this regard, and we aim to take the first step. To achieve
this, the core problem is _how to effectively transfer the informative
knowledge encoded in pre-trained ID-based sequential model to new domains_.
Naturally, item modality information (e.g., texts) widely exists and is
relatively universal across domains, which could be regarded as a bridge to
connect multi-domain ID embeddings in pre-training and downstream tasks as
shown in Fig.1. To achieve this, we design a novel ID-centric pre-training
framework (IDP) for multi-domain recommendation. Specifically, we first
optimize an ID-based sequential model (e.g., SASRec) on all interactions in
multiple pre-training datasets. Besides, since textual similarity does not
always equal behavioral similarity, we also build a cross-domain ID matcher
(CDIM) to connect multi-domain item IDs, which considers both behavioral and
textual modality correlations. In tuning, for each new item, we first retrieve
its top (behaviorally and textually) similar items in pre-training datasets
via item’s textual information via CDIM, and then aggregate their ID
embeddings as the new item’s initialization. Overall, the contributions of
this work are summarized as follows:

  * •

To the best of our knowledge, we are the first to build an ID-based pre-
trained model for multi-domain recommendation. It creatively represents new-
domain items via pre-trained ID embeddings, directly using IDs for main
recommendation and modality for multi-domain ID connections.

  * •

We implement IDP via CDIM, which utilizes behavior-tuned textual correlations
as bridges to aggregate well-learned ID embeddings for new items. CDIM adopts
the universal textual information to reuse pre-trained knowledge while
alleviating too many behavior-text conflicts.

  * •

Extensive experiments on $4$ datasets have verified the effectiveness and
universality of our ID-based pre-training. IDP is effective, efficient, easy-
to-deploy, and universal, which is welcomed to be quickly evaluated in the
industry.

##  2\. Related Works

###  2.1. Sequential Recommendation

Sequential recommendation models mainly leverage users’ chronological behavior
sequences to learn users’ preferences. Early works model sequence patterns by
Markov chains (MCs). Shani et al. (Shani et al., 2005) formalized sequential
recommendation as a sequential decision problem and adopted Markov Decision
Processes (MDPs) to solve it. Later (Rendle et al., 2010) combine the power of
MCs and MFs and achieve a promising result. High-order sequential relations
also be considered in the following work(He and McAuley, 2016; He et al.,
2017a). Recently, with the great progress of deep learning, ID-based
sequential models have achieved great success, which project item ID to unique
dense embedding. GRu4Rec introduces the GRU network to session-based
recommendation for ranking tasks. On the top of GRU4Rec, memory networks(Chen
et al., 2018; Huang et al., 2018), hierarchical structures(Quadrana et al.,
2017), attention mechanisms (Liu et al., 2018), reinforcement learning (Xin et
al., 2020) .etc are introduced. Expected recurrent neural network, other
models also employed for sequential recommendation. Caser (Tang and Wang,
2018) regards the user behavior sequence as a one-dimensional image and adopts
the Convolutional Neural Network (CNN) to capture sequence patterns.

Inspired by the success of Transformer and BERT (Vaswani et al., 2017; Devlin
et al., 2019), attention and transformer-based models are introduced to the
recommendation. DIN (Zhou et al., 2018) applies attention mechanisms between
the target item and user behavior sequence to capture the user interest.
SASRec (Kang and McAuley, 2018) utilizes the single-directional Transformer
block to model the sequential. Following BERT(Devlin et al., 2019), BERT4Rec
(Sun et al., 2019) adopt a bidirectional Transformer to incorporate user
behavior information from both directions. FDSA(Zhang et al., 2019) considers
the features of items and designs a feature-level self-attention block to
leverage the attribute information about items in the user behavior sequence.

###  2.2. Pre-training in Recommendation.

Recently, pre-training models have achieved great success in NLP (Devlin et
al., 2019; Liu et al., 2019) and CV (He et al., 2021; Radford et al., 2021).
They aim to transfer prior knowledge from general large-scale datasets to
downstream domains. While in recommendation, since IDs are unaligned, the
well-learned unique ID embeddings are challenging to transfer directly to
downstream new domains. To utilize the power of pre-trained model, some
single-domain pre-trained models (Yao et al., 2021; Wu et al., 2021) have been
proposed to improve recommendation accuracy of the inner domain. However,
these models lack transferability and are unable to apply to cross-domain
recommendation. Some works transfer well-learned knowledge by overlapped
users/items (Hu et al., 2018; Tsai, 2018; Zhu et al., [n. d.], 2022; Zang et
al., 2022; Yuan et al., 2020b). While overlapped users/items generally are
limited in the real world, thus they are lack of universality. To build a
universal model, some works attempt to design a text-based/modality-based pre-
trained model (Hou et al., 2022; Ding et al., 2021; Cui et al., 2022). Those
works introduce pre-trained language models (PLM) into recommendations and
attempt to replace ID-embedding with modality information. Specifically,
UniSRec (Hou et al., 2022) utilizes PLM to learn universal item
representations across different domains via items’ textual information.
However, recent studies show that ID-embedding is still irreplaceable (Gao et
al., 2023; Liu et al., 2023; Hou et al., 2023). Different from previous works,
we build an ID-centric pre-trained model, which directly transfers pre-trained
ID-embedding to downstream domains.

##  3\. Methodology

Table 1. Mathematical Notations Symbols | Description
---|---
$S$, $T$ | The pre-training multi-domain dataset and downstream domain
$U$, $V$ | The User set of and the Item set
$u$, $v$ | The user and item ID
$s_{u}$ | The behavior sequence of user $u$
$E,\mathbf{e_{v}}$ | The Item ID embedding Matrix and the ID embedding of item $v$
$P,\mathbf{p_{i}}$ | The Position Embedding Matrix and the position embedding of index $i$
| in behavior sequence.
$\mathbf{e_{u}}$ | The encoded user embedding
$\mathbf{h}_{u}^{0}$ | The input representation of sequential encoder
$Trm$ | The transformer block
$SimCSE$ | The encoder of SimCSE
$c_{v}$ | the textual information of item $v$
$t_{v}$ | the encoded textual representation of encoded item
$m,TOP(\cdot,\cdot)$ | The number of selected similar items of Pre-train
| domain and the function that finds top-m items
$V_{v^{T}}$ | The set of selected top-m similar items of the Pretraning domain
$f_{seq}$ | The sequential encoder of pre-training
$f_{seq}^{new}$ | The sequential encoder that re-trained on target downstream domain
$\Theta,\Theta^{*}$ | The parameters of the pre-trained sequential encoder and the parameters
| of fine-tuned sequential encoder
$\vartheta^{*}$ | The parameters of re-trained new sequential encoder by target downstream
| domain

###  3.1. Problem Statement

In this paper, we denote the user and item as $u\in U$ and $v\in V$, where $U$
and $V$ are the user set and item set. Each item is associated with a unique
ID $v_{i}$ and has a piece of text
$c_{v_{i}}=\\{w_{1},w_{2},...,{w_{|c_{v_{i}}|}}\\}$ that describes it, where
$w_{j}$ is the natural word in language. Besides, each user has a
corresponding chronological behavior sequence
$s_{u}=\\{v_{1}^{u},v_{2}^{u},...,v_{|s_{u}|}^{u}\\}$, where $|s_{u}|$ is the
length of user $u$’s behavior sequence. Each item $v_{i}$ is associated with a
trainable ID embedding denoted as $e_{v_{i}}$. We use $S$ and $T$ to denote
the pre-training multi-domain dataset and downstream domain dataset,
respectively. The items and users in pre-training domain and target domain are
denoted with $v_{i}^{S}$,$u_{i}^{S}$, $v_{i}^{T}$, and $u_{i}^{T}$,
respectively.

Sequential Recommendation aims to predict appropriate items that the user
would like to interact with next time $|s_{u}|+1$ based on the user behavior
sequence $s_{u}$. It can be formulated as follows:

(1) $v_{i}^{*}=\arg\max_{v_{i}\in V}P(v_{|s_{u}|+1}=v_{i}|s_{u}).$

###  3.2. Overview

In this paper, we aim to propose an ID-centric pre-trained recommendation
model, called IDP. We compare our IDP with other methods and show it in Fig.2.
The key difference between our approach and previous works lies in we do not
attempt to replace IDs with textual modality information. Instead, our IDP
directly transfers the behavioral information inherent in the upstream ID
embeddings to the downstream. We argue that ID-based recommendation still
dominates recommendation, and our IDP provides a new path to build a pre-
trained recommendation model.

Figure 2. Illustrations of our IDP and other pre-trained models. Different
from conventional pre-trained models that use texts (representations) as items
for behavior modeling, our IDP directly uses pre-trained ID embeddings to
generate new items.

The framework of IDP is illustrated in Fig.3. Our IDP mainly includes three
parts: (1) ID-based sequential model pre-training (2) Cross-domain ID Matcher
(3) downstream fine-tuning. Specifically, we first pre-train an ID-based
sequential model on a multi-domain pre-training dataset. Subsequently, in
order to model the behavior-aware item modality correlations in
recommendation, we train a Cross-domain ID Matcher (CDIM) with collaborative
signals. With the help of CDIM, we can leverage textual information as a
bridge to connect downstream items with pre-trained item embeddings. After
that, we utilize the selected pre-trained item embeddings obtained by CDIM to
generate ID embeddings for the downstream domain. Lastly, we fine-tune both
the pre-trained model and ID-generated ID embeddings for the new domain task.
Through this framework, our IDP could handle multiple new domains while taking
advantage of powerful and efficient ID-based sequential modeling. We summary
the

###  3.3. Pre-trained ID-based Sequential Model

In this section, we introduce the pre-training stage of our IDP. Sequential
recommendation has been widely studied in previous works (Kang and McAuley,
2018; Sun et al., 2019). Following (Zhou et al., 2020; Xie et al., 2022; Hou
et al., 2022), we adopt the classical SASRec (Kang and McAuley, 2018) as our
base sequential model. Note that our IDP is flexible and does not depend on
specific sequential models, it can be easily applied to other sequential
models. We also evaluate it in the experiment section. Specifically, for a
user sequence $s_{u}=\\{v_{1}^{u},v_{2}^{u},...,v_{|s_{u}|^{u}}\\}$, We input
it to the sequential model and it can be formulated as:

(2)
$\displaystyle\mathbf{e}_{u}=f_{seq}(s_{u}|\Theta,\mathbf{E},\mathbf{P}),where\,f_{seq}\,is\,SASRec,$

For $l=0$, $h_{u,i}^{0}$ is the ID embedding of $v_{i}^{u}$.

Following classical recommendation models (Kang and McAuley, 2018; Xie et al.,
2022), we use BPR loss (Rendle et al., 2009) to optimize our pre-training
model, formulated as:

(3) $\begin{split}L_{o}=-\sum_{(u,v_{i})\in S^{+}}\sum_{(u,v_{j})\in
S^{-}}\log\sigma(\mathbf{e}_{u}^{\top}\mathbf{e}_{v_{i}}-\mathbf{e}_{u}^{\top}\mathbf{e}_{v_{j}}),\quad
u\in U^{S},v_{i},v_{j}\in V^{S}.\end{split}$

We pre-train IDP on multi-domain datasets by simply merging all domains’
interactions without any user/item alignment.

Figure 3. Overall architecture of our IDP.

###  3.4. Cross-domain ID Matcher

For building an ID-based pre-training model, the biggest challenge lies in the
misalignment between the IDs in the pre-training multi-domain and those in the
downstream domain, which makes it impossible to directly utilize the upstream
ID embeddings. However, we notice that, in most systems, items have modality
information (e.g., title, content). Generally, the modality information is
universal across domains. Hence, we hope to utilize textual information as a
bridge to build a Cross-domain ID Matcher (CDIM). CDIM can match items in
downstream domains to appropriate items in pre-training domains. In this way,
we can utilize the pre-trained ID embedding for downstream tasks.

A straightforward solution is to utilize general textual modality similarity
for identifying similar items. However, merely considering modality similarity
falls short of achieving a proficient CDIM. There exist gaps between the
general semantic space and that in recommendation. Actually, in
recommendation, behavioral information is crucial and should also be taken
into account when measuring similarity. For example, paper diapers and beer
may not exhibit semantic similarity in the general semantic space, but they
might be considered similar in a recommendation context since new fathers
often purchase them together. Overall, we believe a proficient CDIM should
take into account two fundamental factors: (1) textual similarity. (2)
behavior similarity.

####  3.4.1. Textual similarity

Actually, measuring the similarity of two items for textual modality is
similar to matching two sentences. Previous works directly adopt BERT (Hou et
al., 2022) as the textual information encoder, while BERT is not designed for
the sentence-matching task. Considering this, we utilize SimCSE (Gao et al.,
2021) to encode the textual information. SimCSE employs contrastive learning
to enable the BERT model (Devlin et al., 2019) to generate high-quality
sentence embeddings, resulting in excellent performance on the sentence-
matching task. Specifically, given an item $v_{i}$ and corresponding textual
information $c_{v_{i}}=\\{w_{1},w_{2},...,{w_{|c_{v_{i}}|}}\\}$, we first
concatenate a special token $[CLS]$ with $c_{v_{i}}$, then we feed the
concatenated text into SimCSE as:

(4) $\displaystyle
t_{v_{i}}=SimCSE(\\{[CLS],w_{1},w_{2},...,{w_{|c_{v_{i}}|}}\\}).$

$t_{v_{i}}$ is the representation of $[CLS]$ token from the last layer of
$SimCSE$.

Furthermore, considering that SimCSE is training on the open-accessed corpus,
and may exist a gap between the recommendation corpus. We further fine-tune
SimCSE on recommendation data with a contrastive learning task. Similar to
SimCSE, we simply feed the same input textual information to the encoder
twice. In this way, we get two vectors $t_{v_{i}}$ and
$t_{v_{i}}^{{}^{\prime}}$, which are encoded by two randomly dropped encoders
(the encoder has dropout operation), respectively. We treat $t_{v_{i}}$ and
$t_{v_{i}}^{{}^{\prime}}$ as a positive pair. The contrastive learning task
can be formulated as:

(5)
$\begin{split}l_{v_{i}}^{text}=\frac{e^{sim(t_{v_{i}},t_{v_{i}}^{{}^{\prime}})/\tau}}{e^{sim(t_{v_{i}},t_{v_{i}}^{{}^{\prime}})/\tau}+\sum_{k}^{N-1}e^{sim(t_{v_{i}},t_{v_{k}})/\tau}}.\end{split}$

where $t_{i}$,$t_{i}^{{}^{\prime}}$ is the corresponding textual vectors
encoded by $SimCSE$, $\tau$ is the temperature parameter. $N$ is the batch
size, we adopt an in-batch negative sample strategy, where the other items’
textual vectors $t_{k}$ are regarded as negative instances.

####  3.4.2. Behavior-involved tuning

As discussed before, behavioral information is crucial and should also be
taken into account when measuring similarity in recommendation. Inspired by
the contrastive learning task of $SimCSE$, we utilize collaborative signals to
help model learning the behavior similarity in the recommendation scenario.
Specifically, we first train a recommendation model on multi-domain pre-
training datasets. In this work, we directly use the pre-trained sequential
model in the last step. Subsequently, for each item $v_{i}$ in pre-training
datasets, we obtain its most similar items
$V_{s}^{v_{i}}=(v_{s,1}^{v_{i}},v_{s,2}^{v_{i}}...,v_{s,k}^{v_{i}})$ in pre-
training datasets by embedding similarity of pre-trained model. Last we
conduct contrastive learning to fine-tune SimCSE. Here, We regard
$<v_{i},v_{j}>,v_{j}\in V_{s}^{v_{i}}$ as a positive pair and
$<v_{i},v_{k}>,v_{k}\notin V_{s}^{v_{i}}$ as a negative pair. For an item
$v_{i}$, the contrastive learning task can be formulated as:

(6) $\begin{split}l_{v_{i}}^{behavior}=\sum_{v_{j}\in
V_{s}^{v_{i}}}\frac{e^{sim(t_{v_{i}},t_{v_{j}})/\tau}}{e^{sim(t_{v_{i}},t_{v_{j}})/\tau}+\sum_{k}^{N-1}e^{sim(t_{v_{i}},t_{v_{k}})/\tau}}.\end{split}$

We adopt the multi-task learning strategy to fine-tune SimCSE, so as to make
the similarity of ID embeddings able to reflect both semantic (used to connect
multi-domain items via texts) and behavior (better for new-domain
initialization in recommendation) correlations. Specifically, the loss can be
represented as:

(7) $\begin{split}L=\sum_{v_{i}\in
V}l_{v_{i}}^{behavior}+l_{v_{i}}^{text}.\end{split}$

###  3.5. Generate ID Embedding in New Domains

In this section, we introduce how we use the pre-trained item ID embedding to
generate the item ID embedding of downstream scenarios. With the help of CDIM,
we can measure the similarity of items by textual information. Intuitively,
items that are similar may display analogous interaction patterns in the
recommendation, which is stored in ID embedding. Hence, for an item in the
downstream domain, we leverage the item’s textual information as a bridge to
connect this item with related pre-trained item ID embeddings to generate its
ID embedding. This process can be formulated as follows:

(8)
$\begin{split}V_{v_{i}^{T}}=TOP([sim(t_{v_{i}}^{T},t_{1}^{S}),sim(t_{i}^{T},t_{2}^{S}),...,sim(t_{i}^{T},t_{|V^{S}|}^{S})],m),\end{split}$

where $TOP(\cdot,\cdot)$ is the function that finds the top-$m$ items by
similarity scores given by CDIM, $V_{v_{i}^{T}}$ is the top-$m$ similar items.
Note that this process can be fast, we have analyzed the complexity of our
model in Sec. 3.10.

After that, we utilize the corresponding ID embeddings of the top-$m$ items to
generate an ID embedding for the item $v_{i}^{T}$ in the target domain.
Considering that, for an item $v_{i}^{T}$ in the target domain, the
significance of its similar items in the pre-trained domains may vary based on
their similarity to $v_{i}^{T}$. Therefore, for a given item $v_{i}^{T}$ and
its corresponding top-$m$ most similar items
${V_{v_{i}^{T}}=\\{{v_{1}^{S},v_{2}^{S},...,v_{m}^{S}}}\\}$ in the pre-
training domains, we employ normalized similarity as attention weights to
generate the ID embedding for $v_{i}^{T}$.

(9)
$\begin{split}score_{{v_{i}},{v_{j}}}&=sim(t_{v_{i}}^{T},t_{v_{j}}^{S})/\sum_{k}^{m}{sim(t_{v_{i}}^{T},t_{v_{k}}^{S})},\\\
e_{v_{i}}^{T}&=\sum_{k}^{m}{score_{i,k}*e_{v_{k}}^{S}},\end{split}$

where $e_{k}^{S}$ is the ID embedding of $v_{k}^{S}$, $e_{i}^{T}$ is the
generated ID embedding for $v_{i}^{T}$ in the downstream.

###  3.6. Applied to downstream domain

After generating ID embeddings for the target domain, we can apply the pre-
trained model to downstream domains. Different from previous methods that are
bonded with a pre-trained encoder, our IDP focuses on transfer the knowledge
contained in ID embedding. Thus our IDP is fixable It can be applied in the
following ways:

  * •

We can directly apply the model in downstream domains. In this setting, we
directly deploy the generated ID embedding and the pre-trained sequential
model to downstream domains without any tuning. This process can be formulated
as :

(10)
$\begin{split}\mathbf{e}_{u}^{T}=f_{seq}(s_{u}^{T}|\Theta,\mathbf{E}^{T},\mathbf{P}),\end{split}$

where the $f_{s}eq$ is the pre-trained sequential model, $\Theta$ is the
parameters of the pre-trained model, and it contains the parameters of the
embedding layer and sequential encoder .

  * •

we can fine-tune both pre-trained sequential encoder and generated ID
embedding on downstream data. This can be formulated as:

(11)
$\centering\begin{split}\mathbf{e}_{u}^{T}&=f_{seq}(s_{u}^{T}|\Theta^{*},\mathbf{E^{*}}^{T},\mathbf{P}^{*}),\\\
\Theta^{*},\mathbf{E^{*}}^{T},\mathbf{P}^{*}&=\arg\max_{\Theta,\mathbf{E}^{T},\mathbf{P}}P(v_{|s_{u}|+1}|s_{u},\Theta),u\in
U^{T},v\in V^{T},\end{split}\@add@centering$

where $\Theta^{*}$ is the fine-tuned parameters.

  * •

We can re-train another encoder (it can be a different architecture from the
pre-trained model) and fine-tune generated embedding on downstream data.

(12)
$\centering\begin{split}\mathbf{e}_{u}^{T}&=f_{seq}^{new}(s_{u}^{T}|\vartheta^{*},\mathbf{E^{*}}^{T},\mathbf{P}^{*}),\\\
\vartheta^{*},\mathbf{E^{*}}^{T},\mathbf{P}^{*}&=\arg\max_{\vartheta,\mathbf{E}^{T},\mathbf{P}}P(v_{|s_{u}|+1}|s_{u},\vartheta),u\in
U^{T},v\in V^{T},\end{split}\@add@centering$

where $\vartheta^{*}$ is the parameters of the new sequential model.

###  3.7. Fine-tuning Pre-trained Model

After generating ID embeddings for the target domain. We optimize pre-trained
model with the supervision of the target domain’s behavior data. Different
from previous methods (Hou et al., 2022) that are bonded with the pre-trained
encoder, our IDP is flexible as we transfer knowledge by ID embedding. Our IDP
can be applied in the following ways:

  * •

we can directly deploy the model in downstream domains with the generated new
ID embedding and pre-trained sequential encoder.

(1) we can directly deploy the model in downstream domains with the generated
new ID embedding and pre-trained sequential encoder. (2) we can fine-tune both
pre-trained sequential encoder and generated ID embedding on downstream data.
(3) We can re-train another encoder (it can be a different architecture from
the pre-trained model) and fine-tune generated embedding on downstream data.
This process can be formulated as follows:

(13)
$\begin{split}\Theta^{*}=\arg\max_{\Theta}P(v_{|s_{u}|+1}|s_{u},\Theta),u\in
U^{T},v\in V^{T},\end{split}$

where $\Theta$ is the parameters of fine-tuning, which varies depending on the
fine-tuning approach.

Algorithm 1 Training process of IDP

1:Pre-training multi-domains user set $\bm{U^{S}}$ and item set
$\bm{V^{S}}$;Pre-training dataset $\mathcal{X}^{T}$ ; Pre-trianing Sequential
recommendation encoder $\bm{f_{seq}}$; Target downstream domain user set
$\bm{U^{T}}$ and target downstream domain user set $\bm{V^{T}}$; Target
downstream data $\mathcal{X}^{T}$

2:

3:$\mathbf{\Theta}$, $\mathbf{E}^{S}$, $\mathbf{P}$,$f_{seq}\leftarrow$ Call
function: PRE-TRAINING SEQUENTIAL MODEL

4:Call function: TRAINING CDIM

5:$\mathbf{E}^{S}\leftarrow$ Call funciton:  GENERATING ID EMBEDDING

6:Fine-tuning by Eq.10,Eq.11 or Eq.12

7:

8:function Pre-training sequential model

9: Initialize the trainable parameters $\mathbf{\Theta}$ of encoder
$f_{seq}(\cdot)$, item embedding matrix $\mathbf{E}$ and position matrix
$\mathbf{P}$;

10: repeat

11: for batch data in $\mathcal{X}^{S}$ do

12: $\mathbf{e}_{u}\leftarrow$ encode the user behavior sequence
$f_{seq}$(bath data)

13: $L_{0}\leftarrow$ get the loss with Eq.3

14: Update $\mathbf{\Theta}$, $\mathbf{E}$ and $\mathbf{P}$ according $L_{0}$

15: end for

16: until the trainable parameters have converged.

17: return The pre-trained $\mathbf{E}$, $\mathbf{P}$, $\mathbf{\Theta}$ and
$f_{seq}$

18:end function

19:

20:function Training CDIM

21: repeat

22: $L^{text}\leftarrow$ calculate the textual contrastive learning loss with
Eq.5

23: $L^{behavior}\leftarrow$ calculate behavior-involved contrastive learning
loss with Eq.6

24: $\L\leftarrow L^{text}+L^{behavior}$

25: Update SimCSE by $L$

26: until the trainable parameters have converged.

27:end function

28:

29:function Generating ID embedding

30: $V_{v_{i}^{T}}\leftarrow$ get the most similar items in $V^{S}$ by Eq.8

31: $E^{T}\leftarrow$ generate item embedding with Eq.9

32:end function

###  3.8. Base Model

In this paper, Following (Zhou et al., 2020; Xie et al., 2022; Hou et al.,
2022), we adopt the classical SASRec (Kang and McAuley, 2018) as our base
sequential model. SASRec encodes the user behavior sequence by stacking the
transformer layers and achieving promising performance. The SASRec consists of
three sub-modules: an embedding layer, a multi-head attention module and a
position-wise feed-forward Network. Note that our IDP is flexible and does not
depend on specific sequential models, it can be easily applied to other
sequential models.

####  3.8.1. Embedding Layer

The embedding layer projects the high-dimensional one-hot item ID to low-
dimensional dense embedding by looking up an item embedding table
$\mathbf{E}\in\mathbb{R}^{|V|\times d}$. In addition, there is also a position
embedding table $\mathbf{P}\in\mathbb{R}^{T\times d}$ that projects the
position of items in sequence to dense embedding, where $T$ is the maximum
length of the sequence. Given a user behavior sequence
$s_{u}=\\{v_{1}^{u},v_{2}^{u},...,v_{|s_{u}|}^{u}\\}$, we can obtain the input
representation of user behavior sequence:

(14)
$\begin{split}\mathbf{h}_{u}^{0}&=\\{\mathbf{h}_{u,1}^{0},\\{\mathbf{h}_{u,2}^{0},...,\mathbf{h}_{u,|s_{u}|}^{0}\\},\\\
\mathbf{h}_{u,i}^{0}&=\mathbf{e}_{v_{i}}+\mathbf{p}_{i},\end{split}$

where $e_{i}\in E$ is the representation of item $v_{i}$, $p_{i}\in P$ is the
representation for position index $i$ and $h_{u,i}^{0}$ is the input
representation of $v_{i}$.

####  3.8.2. Self-attention Block

After the embedding layer, we employ the self-attention block to capture the
dependencies between items in the sequence and extract the preference of
users. Like transformer(Vaswani et al., 2017). The multi-head self-attention
mechanism is adopted to effectively extract selective information in different
subspace. Specifically, It first projects the input to $k$ subspace and
applies self-attention to each subspace. Then it combines the output of each
head by linear projecting the concatenated embedding. this process is
formulated as:

$\displaystyle MH(\mathbf{H}^{l})=concat(head_{1};head2;,...;head_{k})W^{O}$
(15) $\displaystyle
head_{i}=Attention(\mathbf{H}^{l}W_{i}^{Q},\mathbf{H}^{l}W_{i}^{K},\mathbf{H}^{l}W_{i}^{V})$

where $W_{i}^{Q},W_{i}^{K},W_{i}^{V}\in\mathbb{R}^{d\times
d\times\frac{d}{h}}$ and $W^{O}\in\mathbb{R}^{d\times d}$are the learnable
project matrix.

In this paper, we adopt the scaled dot-product attention as the attention
function:

(16) $\displaystyle Attention(Q,K,V)=softmax(\frac{QK^{\top}}{\sqrt{d/h}})V$

where $K,Q,V$ denote queries, keys and values vector, respectively. And
$\sqrt{d/h}$ is the scale factor to avoid large values of inner product.

####  3.8.3. Point-Wise Feed-Forward Network

The multi-head self-attention mechanism is effective in extracting the
relation between items, while the simple linear projection limits the capacity
of model. To endow the model with nonlinear interactions between different
dimensions, we adopt a _Position-wise Feed-Forward Network_ after the multi-
head self-attention mechanism sub-layer, which is formulated as:

$\displaystyle
PFFN(\mathbf{H}^{l})=[FFN(\mathbf{h}_{1}^{l})^{\top},...;FFN(\mathbf{h}_{t}^{l})^{\top}]^{\top}$
(17) $\displaystyle
FFN(\mathbf{h}_{i}^{l})=GELU(\mathbf{h}_{i}^{l}+b_{1})W_{2}+b2$

where $W_{1},b1,W_{2},b2$ are learnable parameters. $GELU$ is activation
function and is widely adopted in other works. (Vaswani et al., 2017; Devlin
et al., 2018; Sun et al., 2019)

####  3.8.4. Stack more Blocks

. Empirically, stacking more layers is benefit for modeling patterns in deep
learning. While model generally becomes difficult to train with it goes
deeper. Residual connection, dropout and layer normalization module are proved
effectively to alleviate this issue. So we apply them on our model:

(18) $\displaystyle LayerNorm(x+Dropout(sbulayer(x)))$

where $sublayer$ is the above self-attention block.

In summary, our sequence-based modeling module is represented as follows:

$\displaystyle\mathbf{H}^{0}=\mathbf{E}+\mathbf{P}$
$\displaystyle\mathbf{H}^{l}=Trm(\mathbf{H}^{l-1})$ $\displaystyle
Trm(\mathbf{H}^{l-1})=LayerNorm(\mathbf{A}^{l-1}+Dropout(PPFN(\mathbf{A}^{k-1})))$
(19)
$\displaystyle\mathbf{A}^{l-1}=LayerNorm(\mathbf{H}^{l-1},Dropout(MH(\mathbf{H}^{l-1})))$

Input a behavior sequence of a user, our multi-layer self-attention blocks
output a series of representations of all step $t$. In here, we adopt the last
step to output the sequence as the user’s local preference under this
behavior. We denote it as $\mathbf{e_{u}}=\bm{h}_{u,|s_{u}|}^{L}$.

###  3.9. Discussion on ID-centric Pre-training

Recently, there have been efforts to explore text-based pre-trained
recommendation (Hou et al., 2022), which aims to supplant item ID with natural
language. However, we argue that ID information (i.e. ID embedding) is
indispensable, which is more suitable to store luxuriant behavior information
in recommendation. We attempt to design an ID-centric pre-trained
recommendation model for the following reasons: (1) _Natural language is
difficult to provide a comprehensive portrayal of an item_. An item within the
recommendation system encompasses a multitude of information, including
behavioral logs, visual content, textual description, and so on. Textual
information constitutes only a portion of this broader context. Actually,
recent studies have corroborated this stance (Gao et al., 2023; Liu et al.,
2023; Hou et al., 2023). These studies demonstrate that even with the help of
large language models (LLMs), text-based collaborative filtering struggles to
outperform traditional collaborative filtering methods. (2) _ID-based pre-
training is far more efficient and low-cost compared to LLM-based models_. In
model serving, our IDP’s computation and memory costs are similar to its base
sequential encoder’s, while LLM-based models require expensive costs.
Moreover, in training, CDIM is only functioned for initialization. (3) _ID-
based SR models still dominate most real-world systems, and ID-based pre-
training is more flexible to be deployed in practice_. ID-based pre-trained
model is more similar to existing SR models, which is easy to deploy and
maintain, welcomed by the industry. Moreover, IDP is universal with different
base sequential models, which is more suitable to be enhanced with better
sequential modeling (e.g., new architecture or multi-interest modeling). In
contrast, existing PLM-based SR models generally bond with specific heavy
architectures, which are hard to use future techniques in SR modeling.

###  3.10. Complexity analysis

Now, we discuss the complexity of generating ID embeddings for the downstream
domain. If we directly search for the k most similar items for all items in
the downstream scenarios, the time complexity would be $O(MdN^{2}log(m))$
(accelerated by heap sort), where $M$ represents the number of downstream
scenarios, $N$ denotes the total number of items in the pre-training dataset,
$d$ denotes the ID embedding dimension, and $m$ is top-$m$ similar items. This
time complexity is unaffordable, especially when both $M$ and $N$ are
considerably large. However, thanks to efficient similarity search algorithms,
we can rapidly search out those items. For example, the time complexity can be
reduced to $O(Mlog(log(N)))$ by utilizing the HNSW algorithm (Malkov and
Yashunin, 2018). Thus, our approach is feasible in practice.

##  4\. Experiments

In this section, we aim to answer the following questions:

  * •

RQ1:How does IDP perform compared with other state-of-the-art (SOTA)
baselines?

  * •

RQ2: Can our proposed framework be generalized to other base models?
Furthermore, can our IDP be generalized to other downstream sequential models?

  * •

RQ3: How does IDP perform in the zero-shot setting?

  * •

RQ4: What are the effects of different modules in IDP?

  * •

RQ5: Can our IDP be utilized for helping inner-domain recommendation?

  * •

RQ6: Can our IDP be extended to multi-modal information?

  * •

RQ7: Can we derive some interesting insights by analyzing IDPs?

Table 2. Statistics of all datasets. Dataset | # user | # item | # instance | # avgn
---|---|---|---|---
Pre training | 1,125,233 | 729,305 | 13,920,393 | 19.09
Home | 599,247 | 315,932 | 6,284,657 | 19.89
CD | 82,891 | 109,840 | 1,266,516 | 11.53
Grocery | 102,274 | 84,303 | 1,105,163 | 13.11
Kindle | 118,061 | 140,592 | 2,188,322 | 15.57
Movie | 222,760 | 78,638 | 3,075,735 | 39.11
Downstream |  |  |  |
Arts | 1,576,189 | 302,370 | 2,726,742 | 9.02
Prime | 247,640 | 10,812 | 447,359 | 41.38
Instruments | 903,060 | 112,132 | 1,469,965 | 13.11
Office | 3,402,597 | 306,612 | 5,382,474 | 17.55

###  4.1. Dataset

We evaluate the proposed method on nine different categories of Amazon review
datasets, including pre-training datasets and downstream datasets. The
statistical information of datasets is shown in Table 1.

Pre-traning datasets We select five rating-only Amazon review datasets as the
pre-training datasets including ”Grocery and Gourmet Food”, ”Home and
Kitchen”, ”CDs and Vinyl”, ”Kindle Store” and ”Movies and TV ”.

Downstream datasets. To evaluate the proposed IDP, we conduct experiments on
both cross-domain datasets and cross-platform datasets. We select four rating-
only Amazon review datasets as the downstream datasets including ”Prime
Pantry”, ”Musical Instruments”, ”Arts, Crafts and Sewing” and ”Office
Products”.

Following (He et al., 2017b), we transformed all rating datasets into the
implicit dataset, in which each instance are labeled with 0 or 1 to indicate
whether a user rated an item. whether the user has rated the item For the pre-
training datasets, following previous works (Kang and McAuley, 2018; Zhou et
al., 2020), we discard the users and items with less than five interactions,
as the cold start problem of the pre-training domain is not our concern. We
first conduct pre-training on all five pre-trained models. Subsequently, we
fine-tune the pre-trained model separately on five downstream datasets to
validate the proposed method. For all datasets, We adopt the title of products
as the description text of items.

Table 3. Results on the multi-domain recommendation. The best and the second-best performances are bold and underlined. ”Impro.” denotes the relative improvement over the best baselines. All improvements are significant (t-test with p ${<}$ 0.05). Dataset | Model | NDCG@1 | NDCG@3 | NDCG@5 | HR@3 | HR@5 | MRR
---|---|---|---|---|---|---|---
Arts | GRU4Rec | 0.2343 | 0.3346 | 0.3698 | 0.4071 | 0.4928 | 0.3569
SASRec | 0.3228 | 0.4027 | 0.4298 | 0.4602 | 0.5262 | 0.4211
Bert4Rec | 0.2446 | 0.3485 | 0.3837 | 0.4234 | 0.5091 | 0.3691
DACDR | 0.2666 | 0.3601 | 0.3923 | 0.4275 | 0.5059 | 0.3805
PterRec | 0.2713 | 0.3613 | 0.3905 | 0.4259 | 0.4968 | 0.3799
FDSA | 0.2996 | 0.4005 | 0.4341 | 0.4731 | 0.5546 | 0.4219
Recformer | 0.3281 | 0.4137 | 0.4438 | 0.4754 | 0.5486 | 0.4366
UniSrecID+t | 0.3841 | 0.4926 | 0.5261 | 0.5699 | 0.6514 | 0.5081
IDP | 0.3404 | 0.4249 | 0.4528 | 0.4855 | 0.5532 | 0.4423
IDPID+t | 0.4176 | 0.5153 | 0.5454 | 0.5849 | 0.6581 | 0.5302
Impro | 8.02% | 4.41% | 3.54% | 2.56% | 1.02% | 4.17%
Instruments | GRU4Rec | 0.2382 | 0.3361 | 0.3728 | 0.4073 | 0.4964 | 0.3618
SASRec | 0.3121 | 0.3959 | 0.4243 | 0.4562 | 0.5252 | 0.4149
Bert4Rec | 0.2623 | 0.3648 | 0.4017 | 0.4390 | 0.5286 | 0.3877
DACDR | 0.2774 | 0.3858 | 0.4219 | 0.4640 | 0.5517 | 0.4047
PterRec | 0.2485 | 0.3409 | 0.3747 | 0.4077 | 0.4901 | 0.3655
FDSA | 0.3042 | 0.4108 | 0.4463 | 0.4873 | 0.5737 | 0.4318
Recformer | 0.2812 | 0.3382 | 0.3719 | 0.4015 | 0.4835 | 0.3666
UniSRecID+t | 0.3404 | 0.4492 | 0.4846 | 0.5274 | 0.6135 | 0.4686
IDP | 0.3286 | 0.4207 | 0.4504 | 0.4865 | 0.5588 | 0.4388
IDPID+t | 0.3977 | 0.4990 | 0.5305 | 0.5715 | 0.6481 | 0.5149
Impro. | 14.41% | 9.98% | 8.65% | 7.72% | 5.34% | 8.99%
Office | GRU4Rec | 0.2884 | 0.4052 | 0.4460 | 0.4896 | 0.5887 | 0.4267
SASRec | 0.3408 | 0.4429 | 0.4790 | 0.5169 | 0.6047 | 0.4619
Bert4Rec | 0.3101 | 0.4296 | 0.4681 | 0.5155 | 0.6089 | 0.4467
DACDR | 0.2879 | 0.4054 | 0.4462 | 0.4907 | 0.5896 | 0.4245
PterRec | 0.3082 | 0.4155 | 0.4501 | 0.4927 | 0.5768 | 0.4329
FDSA | 0.3317 | 0.4458 | 0.4832 | 0.5278 | 0.6185 | 0.4645
Recformer | 0.2521 | 0.3142 | 0.3382 | 0.3590 | 0.4176 | 0.3429
UniSRecID+t | 0.3899 | 0.5055 | 0.5418 | 0.5882 | 0.6763 | 0.5205
IDP | 0.3458 | 0.4487 | 0.4831 | 0.5227 | 0.6088 | 0.4668
IDPID+t | 0.4096 | 0.5186 | 0.5528 | 0.5965 | 0.6798 | 0.5331
Impro | 4.81% | 2.53% | 1.99% | 1.39% | 0.51% | 2.36%
Prime | GRU4Rec | 0.1094 | 0.1822 | 0.2163 | 0.2361 | 0.3193 | 0.2219
SASRec | 0.1520 | 0.2224 | 0.2536 | 0.2743 | 0.3503 | 0.2584
Bert4Rec | 0.0950 | 0.1561 | 0.1861 | 0.2013 | 0.2747 | 0.1977
DACDR | 0.1192 | 0.1896 | 0.2214 | 0.2415 | 0.3191 | 0.2268
PterRec | 0.1119 | 0.1804 | 0.2136 | 0.2314 | 0.3123 | 0.2209
FDSA | 0.1209 | 0.1877 | 0.2188 | 0.2370 | 0.3127 | 0.2261
Recformer | 0.1519 | 0.2069 | 0.2298 | 0.2450 | 0.3010 | 0.2395
UniSRecID+t | 0.1280 | 0.1991 | 0.2313 | 0.2518 | 0.3302 | 0.2372
IDP | 0.1526 | 0.2268 | 0.2595 | 0.2804 | 0.3598 | 0.2629
IDPID+t | 0.1842 | 0.2630 | 0.2953 | 0.3209 | 0.3996 | 0.2965
Impro. | 21.18% | 16.99% | 16.44% | 16.99% | 14.07% | 16.37%

###  4.2. Experimental Setting

Implement Details. The embedding sizes of items are set to 64 and the batch
size is 256 for all methods. We optimize all models with the Adam Optimizer
and carefully search for the hyper-parameters of all baselines. To avoid
overfitting, we adopt the early stop strategy with a patience of 20 epochs. We
adopt the random negative sampling strategy (Rendle et al., 2009) with 1
sampling number. For fair comparison, we employed the BPR loss for all
baseline models, as we observed that BPR loss outperformed cross-entropy loss
in performance. We conduct a grid search for hyper-parameters. We search for
models’ learning rates among $\\{1e^{-3},3e^{-4},1e^{-4},3e^{-5}\\}$.

Evaluation Protocols. We adopt the leave-one-out strategy to evaluate the
models’ performance. Following (Kang and McAuley, 2018; Zhou et al., 2020),
for each ground truth, we randomly sample $99$ items that the user did not
interact with under the target behavior as negative samples. We employ top-K
hit rate (HR@K), top-K normalized discounted cumulative gain (NDCG@K), and
mean Reciprocal Rank ( MRR), with $k=\\{1,3,5\\}$.

Specifically, the HR@k is calculated by the following:

(20) $\begin{split}HR@k=\frac{1}{|U|}\sum_{u\in U}\delta(\hat{R}(u,k)\cap
R(u)\neq\emptyset),\end{split}$

where $\hat{R}(u,k)$ is the top-k predicted item list of user $u$ that sorting
by predicted score. $R(u)$ is a ground-truth set of items of user $u$.
$\delta(\cdot)$ is the indicator function, where the $\delta(b)=1$ if $b$ is
true and 0 if false.

The $NDCG@k$ calculated by the following:

(21) $\displaystyle\qquad\operatorname{rel}(u,i)=\delta({\hat{R}(u,k)}^{i}\in
R(u)),$ $\displaystyle\operatorname{NDCG}(N)$
$\displaystyle=\frac{1}{Y}\sum_{i=1}^{N}\frac{2^{\operatorname{rel}(u,i)}-1}{\log(i+1)}\text{
where }Y=\sum_{i=1}^{N}\frac{1}{\log(i+1)},$

where ${\hat{R}(u,k)}^{i}$ is the $i$-th item of ranked item list
$\hat{R}(u,k)$. $\delta(\cdot)$ is the indicator function.

The $MRR$ is calculated by the following:

(22) $\begin{split}MRR=\frac{1}{|U|}\sum_{u\in U}rank_{u}\end{split}$

$rank_{u}$ is the rank position of the first relevant item in $\hat{R(u)}$.

###  4.3. Baselines

To validate our IDP, we conduct a comprehensive comparison. Specifically, we
compare our method with the following baselines:(1) GRU4Rec(Hidasi et al.,
2016) (2) SASRec (Kang and McAuley, 2018), (3) BERT4Rec (Sun et al., 2019),
(4) FDSA (Zhang et al., 2019), (5)Recformer(Li et al., 2023a), and
(6)UniSRecID+t (Hou et al., 2022).

  * •

GRU4Rec (Hidasi et al., 2016), which adopts GRU as users’ sequence encoder for
sequential recommenadtion.

  * •

SASRec (Kang and McAuley, 2018), which is a classical sequential
recommendation model that introduces self-attention in behavior modeling.

  * •

BERT4Rec (Sun et al., 2019), which adopts masked item prediction as an
optimizing object.

  * •

FDSA (Zhang et al., 2019), which adopts self-attentive networks to model item
and feature transition. In this paper, we adopt item descriptions as item
features.

  * •

DACDR (Zhang et al., 2021),which is a cross-domain recommendation model. It
achieves knowledge transfer across domains by embedding alignment through
common features. In this paper, we adopt text information as common feature.

  * •

PeterRec (Yuan et al., 2020a), which is a parameter-efficient transfer
learning recommendaiton model. It inserts model patches into pre-trained
models to achieve effective transfer learning for recommendation.

  * •

Recformer(Li et al., 2023a), which is a pure language model-based sequential
recommendation model. Recformer first translates user behavior sequence to a
natural language sentence. Then Recformer encodes translated sentences by
language model.

  * •

UniSRecID+t (Hou et al., 2022), which is a SOTA text-based pre-train model. In
the pre-training stage, UniSRec represents items with text embedding encoded
by a pre-trained language model. Then it encodes user sequential behaviors by
a sequential model based on text embedding. In the downstream domain, UniSRec
fine-tunes the pre-trained model by the designed MoE-enhanced adopter.
UniSRecID+t is the final version of UniSRec, which adds ID embedding in
downstream domains.

For our model, we adopt SASRec as our base model and pre-train SASRec on pre-
training datasets. Considering that some baselines adopt textual vectors as an
auxiliary feature (i.e. UniSRecID+t and FDSA), we also adopt textual vectors
as an additional feature for a fair comparison. Following UniSRec, we also add
ID embedding with the corresponding textual vector and input it to the
sequential encoder. The ID-only version is denoted as IDP and the text-
enhanced version is denoted as IDPID+t.

###  4.4. Overall Performance Comparison (RQ1)

The main results of our IDP are shown in Table LABEL:tab:main_table. We have:

(1) Compared to pure ID-based models (i.e., SASRec, GRU4Rec, BERT4Rec), our
IDP (adopting SASRec as the base model) significantly outperforms them on all
four datasets, demonstrating that our IDP effectively utilizes the
capabilities of ID-based pre-trained models. With the assistance of the
proposed CDIM, our IDP can use textual information associated with items as a
bridge to generate ID embeddings for downstream domains. As a result, our IDP
can flexibly transfer knowledge from the pre-trained model’s ID embeddings to
new domains. Furthermore, comparing text-based models (i.e. Recformer) with
ID-based models, the text-based models outperformed the ID-based models on the
Arts dataset. However, on other datasets, they performed less effectively than
the ID-based models. This indicates textual information still can not replace
the ID-based model.

(2) We note that text-enhanced models achieve better performance than ID-only
models. This can be attributed to that the textual information of items serves
as a useful auxiliary feature on those datasets. Furthermore, compared with
the text-enhanced method (i.e. FDSA and UniSRec), our IDPID+t still
significantly outperforms the SOTA pre-trained model UniSRec on all datasets.
Those results show the superiority of our IDP. UniSRec is a text-based pre-
trained model that transfers textual modality information to downstream
domains. In contrast to UniSRec, our IDP is an ID-based pre-trained model.
These experimental results further validate our opinion that modality
information can not replace ID embedding to store behavioral information in
recommendation.

(3) In comparison with other baselines, our model achieves substantial
improvements when confronted with downstream datasets of reduced scale.
Actually, in scenarios where the target domain data is relatively scarce
(e.g., Prime), knowledge from other domains is more needed. This demonstrates
our IDP can more effectively leverage the knowledge contained in the pre-
trained model, especially when the downstream domain lacks sufficient
behavioral information for items.

###  4.5. Universality of IDP (RQ2)

We validate the universality of our IDP from two perspectives: (1)the
universality of IDP on the pre-training base model. (2)the universality of IDP
on the downstream sequential model. (3) the universality of IDP on the cross-
platform scenarios.

####  4.5.1. The universality of IDP on the pre-training base model.

As discussed in the Introduction Section, our IDP transfers ID embedding to
downstream domains and thus does not depend on a specific sequential
recommendation model. To evaluate this, we change our pre-training base model
to other ID-based sequential models (i.e., BERT4Rec) and evaluate the
performance of pre-trained models on downstream datasets. The results are
shown in Table 4.

From this table, we find that (1) our IDP still works well on all datasets
when we change the pre-training base model. It shows the universality of our
IDP, which does not depend on a specific base model. In fact, notwithstanding
that the base models have different architectures, all of them are able to
learn the knowledge within the pre-training datasets. Our IDP serves as a
mechanism for facilitating knowledge transfer from the upstream to the
downstream dataset. (2) Similar to adopting SASRec, our IDP gets more
significant improvement when the dataset is smaller.

Table 4. Results of IDP with other sequential model in pre-training stage. IDPBERT4Rec adopts BERT4Rec as the base SR model both in pre-training stage and fine-tuning stage. All improvements are significant (p ${<}$ 0.05). Dataset | Model | HR@5 | NDCG@5
---|---|---|---
Instruments | BERT4Rec | 0.5286 | 0.4017
IDPBERT4Rec | 0.5563 | 0.4438
Arts | BERT4Rec | 0.5091 | 0.3837
IDPBERT4Rec | 0.5098 | 0.3968
Office | BERT4Rec | 0.6089 | 0.4681
IDPBERT4Rec | 0.6029 | 0.4687
Prime | BERT4Rec | 0.2747 | 0.1861
IDPBERT4Rec | 0.3466 | 0.2422

####  4.5.2. The universality of IDP on the downstream sequential model.

As introduced before, in real recommender systems, diverse teams tend to adopt
different sequential models based on their business requirements and system
architecture. The coupling of upstream and downstream sequential models (what
the previous works adopt) constrains the flexibility of utilizing pre-trained
recommendation models. This is one of the motivations why we try to propose an
ID-centric Pre-trained Model.

In this experiment, we adopt different sequential models for pre-training and
downstream fine-tuning. Specifically, we initially employ SASRec as the base
model to pre-train the ID embeddings. Subsequently, in the downstream phase,
we utilize BERT4Rec as the sequential model and fine-tune the generated ID
embedding. Here, we train BERT4Rec from scratch. The results are shown in
Table 5. From the table, we can find that our IDP still can improve downstream
performance even when the sequential model is different from upstream. It
shows the flexibility of our IDP, which does not bond with the pre-trained
sequential encoder. This characteristic makes IDP more likely to be widely
applied in the real world.

Table 5. A challenging setting that uses different SR models in pre-training/tuning. IDPB&S adopts SASRec as the base SR model in pre-training and uses a different BERT4Rec for fine-tuning. All improvements are significant (p ${<}$ 0.05). Dataset | Model | HR@5 | NDCG@5
---|---|---|---
Instruments | BERT4Rec | 0.5286 | 0.4017
IDPB&S | 0.5428 | 0.4299
Arts | BERT4Rec | 0.5091 | 0.3837
IDPB&S | 0.5149 | 0.3999
Office | BERT4Rec | 0.6089 | 0.4681
IDPB&S | 0.6130 | 0.4773
Prime | BERT4Rec | 0.2747 | 0.1861
IDPB&S | 0.3444 | 0.2406

####  4.5.3. The universality of IDP on the cross-platform scenarios

Table 6. Results on the cross-platform scenario. The best and the second-best performances are bold and underlined. ”Impro.” denotes the relative improvement over the best baselines. All improvements are significant (t-test with p ${<}$ 0.05). Metric | NDCG@1 | NDCG@3 | NDCG@5 | HR@3 | HR@5 | MRR
---|---|---|---|---|---|---
GRU4Rec | 0.3374 | 0.4794 | 0.5253 | 0.5808 | 0.6919 | 0.4945
SASRec | 0.3566 | 0.4948 | 0.5362 | 0.5927 | 0.6930 | 0.5082
BERT4Rec | 0.3258 | 0.4668 | 0.5117 | 0.5682 | 0.6765 | 0.4821
DACDR | 0.1444 | 0.2278 | 0.2649 | 0.2891 | 0.3791 | 0.2675
PterRec | 0.2187 | 0.3475 | 0.3961 | 0.4417 | 0.5599 | 0.3763
FDSA | 0.3313 | 0.4667 | 0.5107 | 0.5644 | 0.6710 | 0.4840
Recformer | 0.1332 | 0.2256 | 0.2685 | 0.2942 | 0.3987 | 0.2629
UniSRecID+t | 0.2902 | 0.4328 | 0.4810 | 0.5361 | 0.6531 | 0.4513
IDP | 0.3606 | 0.4966 | 0.5381 | 0.5935 | 0.6942 | 0.5105
IDPID+t | 0.3566 | 0.4904 | 0.5363 | 0.5874 | 0.6987 | 0.5062
Impro. | 1.10% | 0.36% | 0.35% | 0.10% | 0.80% | 0.45%

To demonstrate the universality of our IDP, we conduct a more challenging
cross-platform experiment. We pre-traine the IDP on the Amazon dataset and
fine-tune it on the Movielens-1M dataset. The results are shown in Table.6.
From the table, we can observe our IDP still achieved an improvement even in
cross-platform scenarios. Note that, we argue that it is a challenging task as
Amazon and Movielens are different types of platforms (shopping and video
websites), thus users may have different behavior patterns in them. It shows
the possibility of building a universal ID-centric pre-trained model.

###  4.6. Results of Zero-shot Scenario (RQ3)

In this section, we further evaluate the effectiveness of our IDP in a zero-
shot setting. In this setting, we directly apply the pre-trained model to
downstream datasets without tuning. Note that, as the UniSRecID+t can not be
adopted for zero-shot setting. We compare our method with the zero-shot
version of UniSRec (i.e. UniSRect). We demonstrate the results in Table.7.
Note that, for IDPID+t, we employ PCA to reduce the dimension with the encoded
textual vectors to the same dimension with ID embeddings. Following UniSRec,
we simply add textual vectors with generated ID embedding as the input of the
sequential model.

From this table, we can see that Our IDP is indeed effective. Even in zero-
shot scenarios, it still can help the system make decisions. Because our IDP
can transfer knowledge in the pre-training domains to other domains.
Furthermore, with the help of textual information, our IDP exhibits enhanced
performance and outperforms UniSRec by a substantial margin. This is
attributed to the universal nature of semantic information across different
domains and it is an important auxiliary information. Note that, different
from UniSRec which pre-trains an adapter to encode textual vectors, textual
vectors are directly used in IDP (i.e. reducing dimension with PCA). This also
highlights the flexibility of our IDP.

Table 7. Results on the zero-shot setting. All improvements are significant over baselines (t-test with p ${<}$ 0.05). Dataset | Model | HR@5 | NDCG@5
---|---|---|---
Arts | UniSRect | 0.2620 | 0.1920
IDP | 0.1079 | 0.0681
| IDP ID+t | 0.3234 | 0.2608
Instruments | UniSRect | 0.1693 | 0.1117
IDP | 0.0799 | 0.0489
| IDP ID+t | 0.2648 | 0.2002
Office | UniSRect | 0.2189 | 0.1556
IDP | 0.1337 | 0.0849
| IDPID+t | 0.2708 | 0.2177
Prime | UniSRect | 0.1598 | 0.1092
IDP | 0.0839 | 0.0519
| IDP ID+t | 0.1952 | 0.1487

###  4.7. Ablation Study (RQ4)

In this section, we conduct an ablation study to evaluate the effectiveness of
the proposed modules of our IDP. Precisely, we have (a) IDP w/o M, which
removes our matcher CDIM and only fine-tunes the pre-trained ID-based
sequential encoder on downstream domains, (b) IDP w/o BCL, which directly
utilizes the original SimCSE model as the cross-domain ID matcher without the
behavior-based CL tasks.

The results are reported in Fig. 4. We find (1) By comparing IDP with IDP w/o
M, we can see that our proposed CDIM can effectively transfer knowledge
contained in the pre-trained model via ID embeddings, which utilizes modality
information as a bridge. Besides, IDP w/o M even performs worse than the
single-domain SASRec. This indicates that only transferring the pre-trained
sequential encoder may lead to negative transfer effects. (2) By comparing IDP
with IDP without BCL, we can observe that the designed behavior-aware
contrastive learning task indeed functions effectively. It accomplishes this
by integrating behavioral information into textual vectors through the tuning
of SimCSE. As a result, CDIM is able to balance behavioral and textual
similarity across domains, facilitating the aggregation of ID embeddings for
initialization.

###  4.8. Representing Intra-domain Cold-start Items via Pre-trained Item
Embeddings (RQ5)

Figure 4. Ablation studies on Arts and Office datasets. All components of IDP are effective. Table 8. The result of generating ID embedding within inner-domain. Dataset | Metric | HR@3 | NDCG@3 | HR@5 | NDCG@5
---|---|---|---|---|---
Instruments | IDPInner | 0.1950 | 0.1655 | 0.2352 | 0.1819
Arts | IDPInner | 0.1294 | 0.1053 | 0.2705 | 0.1874

In this section, we aim to investigate the effects of generating IDs within
the inner domain. This experimental result has the ability to demonstrate
whether our IDP is capable of addressing inner-domain cold start issues.
Instead of generating ID embeddings from the upstream domain to the downstream
domain, we focus on generating ID embeddings within the inner domain.
Specifically, for a domain $A$, we first train a sequential model. Then we use
our IDP to generalize ID embedding for cold items (without any interaction).
Last we directly evaluate the model’s performance on cold items without any
tuning operation. Here, we randomly select 10% items as cold items and remove
them from the training dataset. The results are shown in Table 8, where
IDPInner is the model that re-generates ID embedding by our CDIM in the Inner
domain. From this table, we can observe that our IDP can still provide
reasonably accurate recommendations. This indicates that our IDP can be
flexibly applied to inner-domain item cold start scenarios.

###  4.9. Extending to Multi-modal Scenario

Table 9. Results on of extending textual modality information to multi-modal information. The best performances are bold and underlined. All improvements are significant (t-test with p ${<}$ 0.05). Dataset | Model | NDCG@1 | HR@3 | NDCG@3 | HR@5 | NDCG@5 | MRR
---|---|---|---|---|---|---|---
Arts | SASRec | 0.3228 | 0.4602 | 0.4027 | 0.5262 | 0.4298 | 0.4211
UniSRec | 0.3841 | 0.5699 | 0.4926 | 0.6514 | 0.5261 | 0.5081
IDP | 0.3404 | 0.4855 | 0.4249 | 0.5532 | 0.4528 | 0.4423
IDPID+t | 0.4176 | 0.5849 | 0.5153 | 0.6581 | 0.5454 | 0.5302
IDPMM | 0.3453 | 0.4860 | 0.4272 | 0.5493 | 0.4533 | 0.4436
IDP${}^{MM}_{ID+t}$ | 0.4185 | 0.5871 | 0.5169 | 0.6607 | 0.5472 | 0.5315
Office | SASRec | 0.3408 | 0.5169 | 0.4429 | 0.6047 | 0.4790 | 0.4619
UniSRec | 0.3899 | 0.5882 | 0.5055 | 0.6763 | 0.5418 | 0.5205
IDPID | 0.3458 | 0.5227 | 0.4487 | 0.6088 | 0.4831 | 0.4668
IDPID+t | 0.4096 | 0.5965 | 0.5186 | 0.6798 | 0.5528 | 0.5331
IDPMM | 0.3461 | 0.5242 | 0.4497 | 0.6087 | 0.4845 | 0.4681
IDP${}^{MM}_{ID+t}$ | 0.4056 | 0.5944 | 0.5156 | 0.6783 | 0.5502 | 0.5302
Instruments | SASRec | 0.3121 | 0.4562 | 0.3959 | 0.5252 | 0.4243 | 0.4149
UniSRec | 0.3404 | 0.5274 | 0.4492 | 0.6135 | 0.4846 | 0.4686
IDPID | 0.3286 | 0.4865 | 0.4207 | 0.5588 | 0.4504 | 0.4388
IDPID+t | 0.3977 | 0.5715 | 0.4990 | 0.6481 | 0.5305 | 0.5149
IDPMM | 0.3326 | 0.4867 | 0.4224 | 0.5568 | 0.4512 | 0.4401
IDP${}^{MM}_{ID+t}$ | 0.3998 | 0.5024 | 0.5341 | 0.5756 | 0.6525 | 0.5181
Prime | SASRec | 0.1520 | 0.2743 | 0.2224 | 0.3503 | 0.2536 | 0.2584
UniSRec | 0.1280 | 0.2518 | 0.1991 | 0.3302 | 0.2313 | 0.2372
IDPID | 0.1539 | 0.2804 | 0.2268 | 0.3598 | 0.2595 | 0.2629
IDPID+t | 0.1842 | 0.3209 | 0.2630 | 0.3996 | 0.2953 | 0.2965
IDPMM | 0.1522 | 0.2825 | 0.2273 | 0.3616 | 0.2598 | 0.2628
IDP${}^{MM}_{ID+t}$ | 0.1873 | 0.3213 | 0.2645 | 0.4012 | 0.2973 | 0.2986

In this paper, we aim to utilize the modality information to train our CDIM
and transfer the ID embedding to downstream domains. In the Sec. 3.4 we mainly
utilize the textual modality information as a bridge as it is more commonly
available in the real world. However, our IDP can easily be extended to multi-
modal scenarios. To evaluate it, we both utilize the image information and
textual information to train the CDIM. Specifically, we first utilize pre-
trained VIT(Dosovitskiy et al., 2020) to extract the image feature $im_{v}$ of
items. Then we fuse the encoded textual feature $t_{v}$ and image feature
$m_{v}$ by a two-layer MLP with a dropout layer, which is represented as
$mm_{v}$. Subsequently, we replace the textual representation $t_{v}$ in Eq.5
and Eq.6 with multi-modal representation $mm_{v}$, and conduct the modality
information contrastive learning (i.e., Eq.5) and behavior involved tuning
(i.e., Eq.6). After that, we get a multi-modal-based CDIM, which can be used
to transfer pre-trained ID embedding downstream.

To evaluate the performance of our IDP in a multi-modal scenario, we conduct
the following experiments (1) IDPmm, which utilize the multi-modal based CDIM
as the matcher (2) IDP${}^{mm}_{ID+mm}$, which is an multi-modal enhanced IDP
and utilize the multi-modal representation as an auxiliary feature.

We show the results in the Table.9. From this table, we have the following
observations: (1) All variants of our IDP outperform SASRec and all various of
our textual-enhanced IDP outperform UniSRec, which further demonstrates the
power of our IDP. (2) By comparing IDP and IDPMM, we found that introducing
multimodal information to train the CDIM module indeed helps improve the
performance of our IDP. (3) By further comparing C and D, we found that in the
case of text augmentation, C generally outperforms D. This further indicates
the benefit of introducing multimodal information to our IDP. Furthermore,
This experiment demonstrates the versatility of our IDP, which can be applied
in multi-modal scenarios.

###  4.10. Analysis of Sequential Encoder (RQ6)

In this paper, we propose the concept of ID-centric pre-trained models for the
first time. In this section, we conduct two exploratory experiments aimed at
providing some insights.

The effect of the pre-trained sequential encoder. Here, we aim to further
validate the impact of the pre-trained sequential encoder. Specifically,
instead of both utilizing a pre-trained encoder and generated ID-embedding for
downstream fine-tuning, we only adopt generated ID-embedding for downstream
fine-tuning and train the sequential encoder from scratch, denoted as IDP w/o
PS. We report our results in Table 10. We can observe that IDP performs worse
than IDP w/o PS in the Office dataset, whereas the performance is reversed in
the Prime dataset. This suggests that the utilization of a pre-trained
sequential encoder could potentially lead to negative transfer in specific
scenarios. In such cases, training the sequential encoder from scratch might
be a more suitable option.

Table 10. Performance comparison of IDP and IDP w/o PS. Dataset | Metric | HR@3 | NDCG@3 | HR@5 | NDCG@5
---|---|---|---|---|---
Instruments | IDP | 0.4865 | 0.4207 | 0.5588 | 0.4504
IDP w/o PS | 0.4843 | 0.4220 | 0.5522 | 0.4500
Prime | IDP | 0.2804 | 0.2268 | 0.3598 | 0.2595
IDP w/o PS | 2912 | 0.2364 | 0.3715 | 0.2693
Table 11. The result of exchanging sequential encoders. SASRecEX is the version of SASRec with the exchanged encoder. Dataset | Metric | HR@3 | NDCG@3 | HR@5 | NDCG@5
---|---|---|---|---|---
Instruments | SASRec | 0.4562 | 0.3959 | 0.5252 | 0.4243
SASRecEX | 0.4191 | 0.3553 | 0.4903 | 0.3846
Prime | SASRec | 0.2743 | 0.2224 | 0.3503 | 0.2536
SASRecEX | 0.2259 | 0.1810 | 0.2946 | 0.2092

The transferability of the sequence encoder. To analyze the transferability of
the sequential model’s encoder, we perform an exchange of the sequential
encoders between two sequential models that are trained on different domains.
Specifically, we train two sequential models on the ”Musical Instruments”
(Instruments) and ”Prime Pantry” (Prime) datasets, respectively. Afterward, we
interchange the sequential encoders of these two models while retaining their
respective embedding layers. We directly evaluate the performance of those
models on the test dataset without any fine-tuning operation. The results are
reported in Table 11, where SASRecEX represents the model with the exchanged
encoder.

From this table, it is evident that SASRecEX does not perform as well as
SASRec, which is an expected outcome since the encoder of SASRecEX is trained
on a different domain. Nevertheless, SASRecEX still demonstrates respectable
performance on both the Instrument and Prime datasets, despite the performance
gap compared to SASRec. It’s worth noting that there is a weak correlation
between ”prime” and ”Instrument.”

Based on these findings, we hypothesize that sequence models might contain
limited domain-specific knowledge, with the primary knowledge being stored
within ID embeddings. The encoder may primarily serve to capture the patterns
represented by user behavior sequences in a domain-agnostic manner. This
indicates the importance of knowledge transfer from ID embeddings when
designing an advanced pre-trained model.

##  5\. Conclusion And Future Work

In this work, we propose an ID-centric recommendation pre-training framework
(IDP) for MDR, which can effectively transfer the knowledge to downstream
domains. IDP directly aggregates well-learned ID embeddings in pre-training to
build new items’ initializations via the behavior-aware CDIM, which is
effective, universal, and flexible to be deployed with most existing
recommendation models in practice. In the future, we will consider utilizing
multi-modal information for knowledge cross-domain transfer.

## References

  * (1)
  * Chen et al. (2018) Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2018.  Sequential recommendation with user memory networks. In _Proceedings of the eleventh ACM international conference on web search and data mining_. 108–116.
  * Cui et al. (2022) Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022.  M6-Rec: Generative Pretrained Language Models are Open-Ended Recommender Systems.  _arXiv preprint arXiv:2205.08084_ (2022).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of NAACL_.
  * Ding et al. (2021) Hao Ding, Yifei Ma, Anoop Deoras, Yuyang Wang, and Hao Wang. 2021.  Zero-shot recommender systems.  _arXiv preprint arXiv:2105.08318_ (2021).
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple Contrastive Learning of Sentence Embeddings. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 6894–6910.  https://doi.org/10.18653/v1/2021.emnlp-main.552
  * Gao et al. (2023) Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang. 2023.  Chat-rec: Towards interactive and explainable llms-augmented recommender system.  _arXiv preprint arXiv:2303.14524_ (2023).
  * He et al. (2021) Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. 2021.  Masked autoencoders are scalable vision learners.  _CVPR_ (2021).
  * He et al. (2017a) Ruining He, Wang-Cheng Kang, and Julian McAuley. 2017a.  Translation-based recommendation. In _Proceedings of the eleventh ACM conference on recommender systems_. 161–169.
  * He and McAuley (2016) Ruining He and Julian McAuley. 2016.  Fusing similarity models with markov chains for sparse sequential recommendation. In _2016 IEEE 16th International Conference on Data Mining (ICDM)_. IEEE, 191–200.
  * He et al. (2017b) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017b.  Neural collaborative filtering. In _Proceedings of WWW_.
  * Hidasi et al. (2016) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016.  Session-based recommendations with recurrent neural networks. In _ICLR_.
  * Hou et al. (2022) Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. 2022.  Towards Universal Sequence Representation Learning for Recommender Systems. In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 585–593.
  * Hou et al. (2023) Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian McAuley, and Wayne Xin Zhao. 2023.  Large language models are zero-shot rankers for recommender systems.  _arXiv preprint arXiv:2305.08845_ (2023).
  * Hu et al. (2018) Guangneng Hu, Yu Zhang, and Qiang Yang. 2018.  Conet: Collaborative cross networks for cross-domain recommendation. In _Proceedings of CIKM_.
  * Huang et al. (2018) Jin Huang, Wayne Xin Zhao, Hongjian Dou, Ji-Rong Wen, and Edward Y Chang. 2018.  Improving sequential recommendation with knowledge-enhanced memory networks. In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_. 505–514.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _Proceedings of ICDM_.
  * Li et al. (2023a) Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, and Julian McAuley. 2023a.  Text Is All You Need: Learning Language Representations for Sequential Recommendation.  _arXiv preprint arXiv:2305.13731_ (2023).
  * Li et al. (2023b) Jinming Li, Wentao Zhang, Tian Wang, Guanglei Xiong, Alan Lu, and Gerard Medioni. 2023b.  GPT4Rec: A generative framework for personalized recommendation and user interests interpretation.  _arXiv preprint arXiv:2304.03879_ (2023).
  * Liu et al. (2023) Junling Liu, Chao Liu, Renjie Lv, Kang Zhou, and Yan Zhang. 2023.  Is chatgpt a good recommender? a preliminary study.  _arXiv preprint arXiv:2304.10149_ (2023).
  * Liu et al. (2018) Qiao Liu, Yifu Zeng, Refuoe Mokhosi, and Haibin Zhang. 2018.  STAMP: short-term attention/memory priority model for session-based recommendation. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1831–1839.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint_ (2019).
  * Liu et al. (2021) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021.  Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.. In _2021 IEEE/CVF International Conference on Computer Vision (ICCV)_.  https://doi.org/10.1109/iccv48922.2021.00986
  * Malkov and Yashunin (2018) Yu A Malkov and Dmitry A Yashunin. 2018.  Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs.  _IEEE transactions on pattern analysis and machine intelligence_ 42, 4 (2018), 824–836.
  * Mann et al. (2020) Ben Mann, N Ryder, M Subbiah, J Kaplan, P Dhariwal, A Neelakantan, P Shyam, G Sastry, A Askell, S Agarwal, et al. 2020\.  Language models are few-shot learners.  _arXiv preprint arXiv:2005.14165_ (2020).
  * Quadrana et al. (2017) Massimo Quadrana, Alexandros Karatzoglou, Balázs Hidasi, and Paolo Cremonesi. 2017.  Personalizing session-based recommendations with hierarchical recurrent neural networks. In _Proceedings of the Eleventh ACM Conference on Recommender Systems_. 130–137.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International Conference on Machine Learning_. PMLR, 8748–8763.
  * Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009.  BPR: Bayesian personalized ranking from implicit feedback.  _UAI_ (2009).
  * Rendle et al. (2010) Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010.  Factorizing personalized markov chains for next-basket recommendation. In _Proceedings of the 19th international conference on World wide web_. 811–820.
  * Shani et al. (2005) Guy Shani, David Heckerman, and Ronen I Brafman. 2005.  An MDP-based recommender system.  _JMLR_ (2005).
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In _Proceedings of the 28th ACM international conference on information and knowledge management_. 1441–1450.
  * Tang and Wang (2018) Jiaxi Tang and Ke Wang. 2018.  Personalized top-n sequential recommendation via convolutional sequence embedding. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_. 565–573.
  * Tsai (2018) Chun-Hua Tsai. 2018.  Diversity-Enhanced Recommendation Interface and Evaluation. In _Proceedings of the 2018 Conference on Human Information Interaction & Retrieval_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  (2017).
  * Wu et al. (2021) Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie. 2021.  Self-supervised graph learning for recommendation. In _Proceedings of SIGIR_.
  * Xie et al. (2022) Xu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu, Jinyang Gao, Jiandong Zhang, Bolin Ding, and Bin Cui. 2022.  Contrastive learning for sequential recommendation. In _ICDE_. IEEE.
  * Xin et al. (2020) Xin Xin, Alexandros Karatzoglou, Ioannis Arapakis, and Joemon M Jose. 2020.  Self-supervised reinforcement learning for recommender systems. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 931–940.
  * Yao et al. (2021) Tiansheng Yao, Xinyang Yi, Derek Zhiyuan Cheng, Felix Yu, Ting Chen, Aditya Menon, Lichan Hong, Ed H Chi, Steve Tjoa, Jieqi Kang, et al. 2021\.  Self-supervised Learning for Large-scale Item Recommendations. In _Proceedings of CIKM_.
  * Yuan et al. (2020a) Fajie Yuan, Xiangnan He, Alexandros Karatzoglou, and Liguang Zhang. 2020a.  Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. In _Proceedings of SIGIR_.
  * Yuan et al. (2020b) Fajie Yuan, Guoxiao Zhang, Alexandros Karatzoglou, Xiangnan He, JoemonM. Jose, Beibei Kong, and Yudong Li. 2020b.  One Person, One Model, One World: Learning Continual User Representation without Forgetting.  _Cornell University - arXiv,Cornell University - arXiv_ (Sep 2020).
  * Zang et al. (2022) Tianzi Zang, Yanmin Zhu, Haobing Liu, Ruohan Zhang, and Jiadi Yu. 2022.  A survey on cross-domain recommendation: taxonomies, methods, and future directions.  _ACM Transactions on Information Systems_ 41, 2 (2022), 1–39.
  * Zhang et al. (2023) Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. 2023.  Recommendation as instruction following: A large language model empowered recommendation approach.  _arXiv preprint arXiv:2305.07001_ (2023).
  * Zhang et al. (2021) Qian Zhang, Wenhui Liao, Guangquan Zhang, Bo Yuan, and Jie Lu. 2021.  A deep dual adversarial network for cross-domain recommendation.  _IEEE Transactions on Knowledge and Data Engineering_ (2021).
  * Zhang et al. (2019) Tingting Zhang, Pengpeng Zhao, Yanchi Liu, Victor S Sheng, Jiajie Xu, Deqing Wang, Guanfeng Liu, Xiaofang Zhou, et al. 2019\.  Feature-level Deeper Self-Attention Network for Sequential Recommendation.. In _IJCAI_. 4320–4326.
  * Zhou et al. (2018) Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018.  Deep interest network for click-through rate prediction. In _Proceedings of KDD_.
  * Zhou et al. (2020) Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. 2020.  S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization. In _CIKM_.
  * Zhu et al. ([n. d.]) Feng Zhu, Chaochao Chen, Yan Wang, Guanfeng Liu, and Xiaolin Zheng. [n. d.].  DTCDR: A Framework for Dual-Target Cross-Domain Recommendation.  ([n. d.]).
  * Zhu et al. (2022) Yongchun Zhu, Zhenwei Tang, Yudan Liu, Fuzhen Zhuang, Ruobing Xie, Xu Zhang, Leyu Lin, and Qing He. 2022.  Personalized transfer of user preferences for cross-domain recommendation. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_. 1507–1515.