# MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence
Representation for Recommendation

Jinpeng Wang  Tsinghua Shenzhen International Graduate School, Tsinghua
UniversityShenzhenChina<EMAIL_ADDRESS>,  Ziyun Zeng  Tsinghua
Shenzhen International Graduate School, Tsinghua UniversityShenzhenChina
<EMAIL_ADDRESS>,  Yunxiao Wang  Tsinghua Shenzhen
International Graduate School, Tsinghua UniversityShenzhenChina wang-
<EMAIL_ADDRESS>,  Yuting Wang  Tsinghua Shenzhen International
Graduate School, Tsinghua UniversityShenzhenChina
<EMAIL_ADDRESS>,  Xingyu Lu  Tsinghua Shenzhen International
Graduate School, Tsinghua UniversityShenzhenChina<EMAIL_ADDRESS>,  Tianxiang Li  Tsinghua Shenzhen International Graduate School, Tsinghua
UniversityShenzhenChina<EMAIL_ADDRESS>,  Jun Yuan  Huawei
Noah’s Ark LabShenzhenChina<EMAIL_ADDRESS>,  Rui Zhang${}^{\text{{\char
0\relax}}}$ www.ruizhang.info<EMAIL_ADDRESS>,  Hai-Tao Zheng  Tsinghua
Shenzhen International Graduate School, Tsinghua UniversityShenzhenChina Peng
Cheng LaboratoryShenzhenChina<EMAIL_ADDRESS>and  Shu-Tao
Xia${}^{\text{{\char 0\relax}}}$ Tsinghua Shenzhen International Graduate
School, Tsinghua UniversityShenzhenChina Research Center of Artificial
Intelligence, Peng Cheng LaboratoryShenzhenChina<EMAIL_ADDRESS>
(2023)

###### Abstract.

The goal of sequential recommendation (SR) is to predict a user’s potential
interested items based on her/his historical interaction sequences. Most
existing sequential recommenders are developed based on ID features, which,
despite their widespread use, often underperform with sparse IDs and struggle
with the cold-start problem. Besides, inconsistent ID mappings hinder the
model’s transferability, isolating similar recommendation domains that could
have been co-optimized. This paper aims to address these issues by exploring
the potential of multi-modal information in learning robust and generalizable
sequence representations. We propose MISSRec, a multi-modal pre-training and
transfer learning framework for SR. On the user side, we design a Transformer-
based encoder-decoder model, where the contextual encoder learns to capture
the sequence-level multi-modal user interests while a novel interest-aware
decoder is developed to grasp item-modality-interest relations for better
sequence representation. On the candidate item side, we adopt a dynamic fusion
module to produce user-adaptive item representation, providing more precise
matching between users and items. We pre-train the model with contrastive
learning objectives and fine-tune it in an efficient manner. Extensive
experiments demonstrate the effectiveness and flexibility of MISSRec,
promising a practical solution for real-world recommendation scenarios. Data
and code are available on https://github.com/gimpong/MM23-MISSRec.

multi-modal sequential recommendation, pre-training, parameter-efficient fine-
tuning, interest-aware sequence representation

✉ Corresponding authors.

††submissionid: 1218††journalyear: 2023††copyright:
rightsretained††conference: Proceedings of the 31st ACM International
Conference on Multimedia; October 29-November 3, 2023; Ottawa, ON,
Canada††booktitle: Proceedings of the 31st ACM International Conference on
Multimedia (MM ’23), October 29-November 3, 2023, Ottawa, ON, Canada††doi:
10.1145/3581783.3611967††isbn: 979-8-4007-0108-5/23/10††ccs: Information
systems Recommender systems††ccs: Information systems Multimedia information
systems††ccs: Information systems Personalization

##  1\. Introduction

Figure 1. Challenges of multi-modal sequential recommendation. (a) The multi-
modal synergy within each item can be user-dependent and dynamic. Users may
interact with the same item focusing on different modalities. (b) Redundant
interaction may overwhelm sparse but important interests. Modeling
interactions equally will lead to biased predictions.

Recommendation systems (Resnick and Varian, 1997; He et al., 2020a; Su et al.,
2021, 2022) are an important component in various application domains, from
e-commerce to content platforms, improving user engagement and satisfaction.
Sequential recommendation (SR) (Wang et al., 2019; Quadrana et al., 2018), a
popular methodology of recommendation, aims to predict users’ potential
interested items according to their historical interaction sequences.
Typically, this task is formulated as a representation learning problem, where
user embeddings are learned to capture underlying preferences behind the
interacted item sequence and to match with suitable item embeddings.

Current SR approaches (Kang and McAuley, 2018; Sun et al., 2019; Zhou et al.,
2020; Chen et al., 2022; Lin et al., 2023) predominantly rely on ID features
as input, which, despite their widespread uses in practice, exhibit two
primary shortcomings. (i) The robustness and scalability largely depend on the
distribution of user-item interaction. Due to the sparsity of interaction
data, it is challenging to learn accurate item and sequence representations
based on ID information, let alone the cold-start problem (Schein et al.,
2002) for new items. Meanwhile, ID-based approaches often exhibit a popularity
bias toward popular IDs (Abdollahpouri et al., 2020; Zhang et al., 2021a),
posing the fairness issue for the majority of infrequent IDs. (ii)
Transferring knowledge to new scenarios is hard because of inconsistent ID
mappings. It isolates similar domains that can be co-optimized, limiting the
applicability of ID-based models.

Given the ubiquity of text and images in real-world scenarios, leveraging
multi-modal data is promising to remedy the above shortcomings (Yang et al.,
2021). Particularly, both text and visual information play crucial roles in
attracting user attention – compelling titles can boost user engagement, while
visual elements like colors and shapes, can influence user decisions.
Considering remarkable progress in computer vision (Khan et al., 2022; Han et
al., 2022) and natural language understanding (Liu et al., 2023; Zhao et al.,
2023), they offer robust and generalizable semantic extraction capability to
item content. Therefore, we believe it necessary and also feasible to harness
multi-modal content understanding to the SR. However, such investigation
remains in a fledgling stage, as our literature review (§2) will show. _How
does multi-modal information impact SR? How to make better use of such
information in sequence modeling?_ These two questions drive our study in this
paper.

In general, utilizing multi-modal information in SR poses two challenges. (i)
The multi-modal synergy within each item is user-dependent and dynamic. Users
may interact with the same item for various reasons, and different modalities
contribute unequally to user interest in an item. Even for the same user-item
pair, such patterns can be time- and context-varying, making it difficult to
design effective multi-modal fusion. (ii) Information redundancy can overwhelm
essential user interests. Interaction sequences typically exhibit imbalanced
interest distribution. A user’s sequence usually contains lots of homogeneous
items, _e.g._ , daily necessities, while other informative items, _e.g._ ,
sports goods, may sparsely appear. If we treat all interactions equally in
user behavior modeling, it will result in overemphasis on specific kinds of
items and insufficient focus on others. We give an illustration of these
challenges in fig. 1.

To tackle these challenges, we propose MISSRec (Multi-modal IntereSt-aware
Sequential Recommendation), a multi-modal pre-training and transferring
framework for SR. In general, MISSRec takes interaction sequences with multi-
modal item information as input, learns ID-agnostic representations with an
interest-aware encoder-decoder model via self-supervised pre-training, and can
be efficiently adapted for multiple domains to enhance recommendation.
Specifically, (i) to bridge the semantic gap between general domains and
recommendation, we incorporate multi-modal feature adapters (Houlsby et al.,
2019; Bapna and Firat, 2019) to distill personalized semantics of general
multi-modal features in a parameter-efficient manner. (ii) To explore the
multi-modal synergy in the user-item interaction process, we introduce a
lightweight dynamic fusion strategy to produce user-specific item
representations. (iii) To mitigate the adverse effect of information
redundancy on sequence modeling, we introduce an interest discovery module to
mine global multi-modal interests among users. By associating items with
relevant interests via adaptive clustering, we convert multi-modal sequences
into a series of interest tokens. (iv) We design a novel Transformer-based
encoder-decoder architecture for interest-aware multi-modal sequence modeling,
where the contextual encoder captures the relations among multi-modal user
interests and the decoder grasps essential item-modality-interest patterns for
comprehensive sequence representation.

We conduct detailed experiments and show the effectiveness of MISSRec in
leveraging multi-modal data to learn robust and transferable sequence
representation. In particular, it exhibits better generalizability than ID-
based and text-based approaches and further alleviates the item cold-start
issue.

To summarize, we make the following contributions in this paper.

  * •

We highlight the significance and challenges of exploiting multi-modal
information for SR and propose an effective pre-training and efficient
transfer learning framework for it.

  * •

To capture the dynamic multi-modal synergy in the user-item interaction
process, we adopt a lightweight dynamic fusion module to produce user-adaptive
candidate item representation.

  * •

We introduce a multi-modal interest discovery module, based on which we
construct an interest-aware decoder to model item-modality-interest relations
for better sequence representation.

  * •

Extensive experiments of pre-training and downstream domain adaptation justify
the comprehensive merits of our approach.

##  2\. Related Works

###  2.1. Sequential Recommendation

Sequential recommendation (SR) aims to predict the next suitable item for a
given user based on her/his interacted item sequence (Wang et al., 2019;
Quadrana et al., 2018). Early SR approaches (He and McAuley, 2016a; Rendle et
al., 2010) exploited the Markov chain mechanism for sequential modeling, while
deep SR approaches explored CNN- (Tang and Wang, 2018; Yuan et al., 2019),
RNN- (Ni et al., 2018; Hidasi et al., 2016), MLP- (Zhou et al., 2022; Liang et
al., 2023) and attention-based (Kang and McAuley, 2018; Ma et al., 2019; Zhang
et al., 2019; Sun et al., 2019) architectures to model the transition patterns
in the interaction sequences. Besides, interest modeling (Zhou et al., 2018;
Li et al., 2019; Cen et al., 2020; Lin et al., 2023; Zhang et al., 2022) has
been another popular methodology for SR, where user interests were usually
implemented by attention or clustering. From other perspectives, extensive
efforts have been devoted to designing effective learning strategies for SR,
including temporal-aware learning tasks (Li et al., 2020; Tian et al., 2022;
Zhao, 2022) and contrastive learning objectives (Xie et al., 2022; Chen et
al., 2022; Xie et al., 2022; Zhou et al., 2020; Qiu et al., 2022; Wang et al.,
2023a). Note that most existing SR approaches are designed with ID features,
_e.g._ , item IDs or attribute IDs. They usually suffer from the cold-start
problem and fall short of transferability. Several SR approaches have
leveraged multi-modal item contents to mitigate these issues. For instance,
for each item, CSAN (Huang et al., 2018) aggregated multiple information
including multi-modal features and projected the result to obtain an
embedding, providing better item representation than the ID embedding alone.
MM-Rec (Wu et al., 2022) adopted pre-trained VL-BERT (Su et al., 2020) as the
multi-modal item encoder and built a single-tower model for SR. Inspired by
the success of MLP-Mixer (Tolstikhin et al., 2021) in CV, MMMLP (Liang et al.,
2023) adapted the architecture to multi-modal SR. On the basis of these works,
we further focus on learning universal sequence representations with multi-
modal item content. Our MISSRec contributes a pre-training and efficient
transfer framework for multi-modal SR, which can benefit real-world practice.

###  2.2. Pre-training and Transfer Learning in Recommendation

The paradigm of “pre-train and transfer” has become increasingly popular in
recommendation. Compared to cross-domain recommendation (Zhu et al., 2021),
this paradigm is more general as it does not require cross-domain
correspondence, _e.g._ , overlapped items. CLUE (Cheng et al., 2021) and
PeterRec (Yuan et al., 2020) designed ID-based sequence models and adopted
contrastive learning objectives (He et al., 2020b; Chen et al., 2020) for
model learning, where PeterRec further made the subsequent transfer parameter-
efficient. UPRec (Xiao et al., 2021) leveraged user profiles and social
information to construct auxiliary pre-text tasks. The assumption that both
pre-training and target domains share ID vocabulary restricts their
application scopes. By contrast, modality-based methods are more flexible. For
instance, text-based SR pre-training approaches (Hou et al., 2022, 2023)
utilized a pre-trained language model (PLM, _e.g._ , BERT (Devlin et al.,
2019)) as the frozen feature extractor in item representation modules, showing
efficacy and efficiency. Yuan et al. (2023) designed text-based and image-
based SR models under different scenarios, demonstrating the improvement from
the tunable feature extractors. Wang et al. (2022a) considered mixed modality
sequences where each item is either a text or an image. They also chose to
optimize the SR model and feature extractors jointly, improving the item
representation but largely downgrading learning efficiency. Furthermore, to
capture the multi-modal synergy within each item, two latest works (Zhang et
al., 2023; Song et al., 2023a) have focused on SR pre-training and transfer
learning with multi-modal item representations. Our MISSRec also belongs to
this setting, but we have fulfilled two improvements. (i) To our best
knowledge, we are the first to design interest-aware modeling for multi-modal
SR. (ii) We design user-adaptive candidate item fusion to model users’ dynamic
attention to different modalities.

###  2.3. Multi-modal Recommendation

Leveraging multi-modal information (_e.g._ , text and images) has been shown
to improve the recommendation efficacy in many applications, _e.g._ , news
feed (Zhang et al., 2021b; Xun et al., 2021; Liu et al., 2022b; Li et al.,
2022), short-video feed (Lei et al., 2021; Jiang et al., 2020; Cai et al.,
2021), fashion e-commerce (He and McAuley, 2016b; Song et al., 2023b), and can
effectively alleviate cold-start issues (Pan et al., 2022; Du et al., 2020).
Therefore, multi-modal recommendation (MMR) has become an active research
topic (Zhou et al., 2023b; Deldjoo et al., 2021, 2020). Early approaches
explored MMR with matrix factorization models (He and McAuley, 2016b; Chen et
al., 2017), while the latter ones mainly developed GNN-based models (Wei et
al., 2019, 2020; Sun et al., 2020) to demonstrate the efficacy. On this basis,
recent works have delved into self-supervised learning strategies (Zhou et
al., 2023a; Zhang et al., 2021c; Tao et al., 2022; Wei et al., 2023; Yi et
al., 2022; Dai et al., 2022) to enhance in-domain robustness or pre-training
strategies (Qiu et al., 2021; Liu et al., 2021, 2022a; Wang et al., 2023b;
Jiang et al., 2021; He et al., 2021) to improve cross-domain generalization.
Despite the remarkable progress in MMR, existing solutions were predominantly
designed for non-sequential scenarios, _e.g._ , collaborative filtering (Mao
et al., 2021), which are not very suitable for SR. Fortunately, our work
uncovers some challenges in multi-modal SR (see §1 and fig. 1) and fulfills
effective solutions. We hope it can provide a research basis for future work.

Figure 2. MISSRec consists of two stages. In the first stage, it extracts
multi-modal features from item content and transforms them into a token
sequence. Meanwhile, it adaptively converts the token sequence into multi-
modal interest tokens. Then, it encodes the mined interest sequence to capture
contextual cues for personalization. Next, by developing an interest-aware
decoding mechanism, it produces a comprehensive sequence embedding. Finally,
it adopts sequence-item and sequence-sequence contrastive objectives for pre-
training. In the second stage, it uses the sequence-item objective for fine-
tuning. Both transductive (_i.e._ , w/ item IDs) and inductive (_i.e._ , ID-
agnostic) tuning settings are supported. The downstream domain adaptation is
parameter-efficient as only the modality-specific adapters need to be tuned.
_This figure is best viewed in color._

##  3\. Method

###  3.1. Problem Formulation and Method Overview

Given the historical interaction sequence of the $i$-th user,
$S_{i}=[I_{1},I_{2},\cdots,I_{T_{i}}]$, ordered by timestamps, sequential
recommendation (SR) aims to predict the next item $I_{T_{i}+1}$ that the user
may interact with by learning representations for $S_{i}$ and candidate items.
Under the multi-modal settings, each item contains a unique item ID and multi-
modal content. Without loss of generality, we study leveraging text and image
modalities to improve SR in this paper. We propose MISSRec, a pre-training and
transferring framework, as illustrated in fig. 2. MISSRec consists of 7 main
components: (i, §3.2.1) Pre-trained, frozen text and image encoders for
extracting multi-modal features. (ii, §3.2.3) A pair of modality-specific
adapters to transform multi-modal features into input tokens, which bridge the
semantic gap between general features and personalization. In the fine-tuning
stage, they help with efficient model adaptation. (iii, §3.2.4) A dynamic
fusion module for generating user-adaptive candidate item representations.
(iv, §3.3.1) A multi-modal interest discovery module that generates user
interest tokens for the decoder. (v, §3.4.1) A contextual Transformer encoder
for the interest sequence that dynamically grasps important cues for
personalization. (vi, §3.4.2) An interest-aware Transformer decoder to capture
the item-modality-interest relation for better sequence representation. (vii,
§3.5 and §3.6) The pre-training and downstream fine-tuning tasks to achieve
transferable recommendation.

###  3.2. Universal Multi-modal Item Representation

####  3.2.1. Multi-modal Feature Extraction with Pre-trained Transformers

MISSRec utilizes multi-modal features to achieve universal representations.
Concretely, we take advantage of the pre-trained transformers for item content
understanding. Specifically, we use BERT (Devlin et al., 2019) and ViT
(Dosovitskiy et al., 2020) for text and visual feature extraction,
respectively. We adopt the cross-modal pre-trained version (Radford et al.,
2021) following Zhang et al. (2023). For an item $I$, we obtain its text
feature vector by $\bm{f}^{\text{t}}=\phi^{\text{t}}(I)$ and the image feature
vector by $\bm{f}^{\text{v}}=\phi^{\text{v}}(I)$, where $\phi^{\text{t}}$ and
$\phi^{\text{v}}$ denotes the text encoder the visual encoder, respectively.

####  3.2.2. Feature Augmentation

To enhance the robustness of the sequence representation model, we apply
dropout (Srivastava et al., 2014) as the feature augmentation. Given the text
features
$\bm{F}^{\text{t}}_{i}=[{\bm{f}^{\text{t}}_{1},\bm{f}^{\text{t}}_{2},\cdots,\bm{f}^{\text{t}}_{T_{i}}}]$,
we apply twice the feature augmentation and subsequently obtain
$\tilde{\bm{F}}_{i}^{\text{t}}=[{\tilde{\bm{f}}^{\text{t}}_{1},\tilde{\bm{f}}^{\text{t}}_{2},\cdots,\tilde{\bm{f}}^{\text{t}}_{T_{i}}}]$
and
$\tilde{\bm{F}}_{i}^{\text{t}^{\prime}}=[{\tilde{\bm{f}}^{\text{t}^{\prime}}_{1},\tilde{\bm{f}}^{\text{t}^{\prime}}_{2},\cdots,\tilde{\bm{f}}^{\text{t}^{\prime}}_{T_{i}}}]$
as the augmented text features. They serve as two positive views _w.r.t._
$\bm{F}^{\text{t}}_{i}$. The image features $\bm{F}^{\text{v}}_{i}$ follow
analogous augmentation procedure.

####  3.2.3. Bridge Domain Gap with Modality-specific Adapters

We adopt modality-specific adapters (Houlsby et al., 2019; Bapna and Firat,
2019) to reduce irrelevant semantics in multi-modal features and enhance the
personalization. We transform the augmented features and fuse the results as
item tokens, for example,
$\bm{X}_{i}=[{\bm{x}_{1},\bm{x}_{2},\cdots,\bm{x}_{L_{i}}}]=[\psi^{\text{t}}(\tilde{F}^{\text{t}}_{i});\psi^{\text{v}}(\tilde{F}^{\text{v}}_{i})]$.
$\psi^{\text{t}}$ and $\psi^{\text{v}}$ denote the text and visual adapters,
respectively.

In contrast to tunable feature extractors, adapters with frozen extractors
exhibits two strengths. (i) It allows (pre-) training of the sequence model
based on pre-extracted features, reducing time and memory overhead. (ii)
Adapters enable parameter-efficient transfer in downstream domains, which is
preferable to fine-tuning.

####  3.2.4. Candidate Item Representation with Dynamic Fusion

To model users’ dynamic attention to different modalities of candidate items,
we design a lightweight fusion module to generate user-adaptive item
representations. Given a candidate item $I_{k}$, we first encode it with
modality encoders and adapters to obtain modality embeddings, _i.e._ ,
$\bm{x}_{k}^{\text{t}}$ and $\bm{x}_{k}^{\text{v}}$. For the $i$-th user, we
compute its sequence representation $\bm{u}_{i}$ according to the procedures
in §3.3 and §3.4. Then, the adaptive representation of $I_{k}$ is defined by
the weighted fusion of modality embeddings, namely

(1)
$\displaystyle\bm{v}_{k}=\frac{e^{\alpha\cdot\langle\bm{u}_{i},\bm{x}_{k}^{\text{t}}\rangle}\cdot\bm{x}_{k}^{\text{t}}+e^{\alpha\cdot\langle\bm{u}_{i},\bm{x}_{k}^{\text{v}}\rangle}\cdot\bm{x}_{k}^{\text{v}}}{e^{\alpha\cdot\langle\bm{u}_{i},\bm{x}_{k}^{\text{t}}\rangle}+e^{\alpha\cdot\langle\bm{u}_{i},\bm{x}_{k}^{\text{v}}\rangle}}.$

$\langle\cdot,\cdot\rangle$ denotes the inner product. $\alpha\geq 0$ is a
concentration factor to balance two modalities. Besides the efficacy, the
fusion keeps efficiency as well, because it is factorizable when computing
matching scores. Specifically, let us define
$s^{\text{t}}_{i,k}=\langle\bm{u}_{i},\bm{v}_{j}^{\text{t}}\rangle$ and
$s^{\text{v}}_{i,k}=\langle\bm{u}_{i},\bm{v}_{j}^{\text{v}}\rangle$, the
matching score between $\bm{u}_{i}$ and $\bm{v}_{k}$ can be compute by

(2)
$\displaystyle\langle\bm{u}_{i},\bm{v}_{k}\rangle=\frac{s^{\text{t}}_{i,k}\cdot
e^{\alpha\cdot s^{\text{t}}_{i,k}}+s^{\text{v}}_{i,k}\cdot e^{\alpha\cdot
s^{\text{v}}_{i,k}}}{e^{\alpha\cdot s^{\text{t}}_{i,k}}+e^{\alpha\cdot
s^{\text{v}}_{i,k}}}.$

We can equivalently pre-compute modality-specific scores and fuse them to get
the overall score. In particular, dynamic fusion exhibits a transition between
mean and max poolings:
$\langle\bm{u}_{i},\bm{v}_{k}\rangle=(s^{\text{t}}_{i,k}+s^{\text{v}}_{i,k})/2$
when $\alpha=0$, while
$\langle\bm{u}_{i},\bm{v}_{k}\rangle=\max(s^{\text{t}}_{i,k},s^{\text{v}}_{i,k})$
when $\alpha\rightarrow+\infty$.

Figure 3. Convert item tokens to interest tokens via the multi-modal interest
discovery (MID) module. An item can be converted to different interests
according to multiple modalities.

###  3.3. Discover Multi-modal User Interests

Typically, item sequences reflect users’ implicit interests that are important
for SR. Interest modeling can reduce the negative impact of information
redundancy, helping to capture personalized semantics for precise
representation. In this sub-section, we introduce a multi-modal interest
discovery (MID) module to assist this goal.

####  3.3.1. Excavate and Index Interests via Adaptive Clustering

During the training process, item tokens are expected to convey both
transferable multi-modal information and personalized semantics. Hence, we
adopt a variant of $k$-nearest neighbor-based density peaks clustering
algorithm (DPC-KNN) (Du et al., 2016) on the whole item token set
$\mathcal{X}=\\{\bm{x}_{i}\\}_{i=1}^{N_{I}}$, to explore the distribution of
user interests before every training epoch starts. First, for the $i$-th
token, we compute its local density by considering its $k$-nearest neighbors:

(3) $\displaystyle\rho_{i}=\exp\left(-\frac{1}{k}\sum_{\bm{x}_{j}\in
k\text{NN}(\bm{x}_{i})}\left\lVert\bm{x}_{i}-\bm{x}_{j}\right\rVert_{2}^{2}\right).$

Then, for the $i$-th token, we compute its minimum distance to any other token
with higher local density, namely

(4)
$\displaystyle\delta_{i}=\begin{cases}\min_{j:\rho_{j}>\rho_{i}}\left\lVert\bm{x}_{i}-\bm{x}_{j}\right\rVert_{2},&\text{if
}\exists j\text{ s.t. }\rho_{j}>\rho_{i},\\\
\max_{j}\left\lVert\bm{x}_{i}-\bm{x}_{j}\right\rVert_{2},&\text{otherwise}.\end{cases}$

Next, we select $K_{\text{c}}$ cluster centroids with the highest
$\rho_{i}\times\delta_{i}$ scores as the interest prototypes. Finally, we
index each token by assigning it to the nearest centroid (_i.e._ , the
interest prototype).

####  3.3.2. Translate Multi-modal Item Tokens into Interest Tokens

As shown in figs. 2 and 3, the MID module receives item tokens, finds the
nearest interest prototype for each token, and returns irredundant interest
tokens. To accelerate token conversion, we can build the interest index of
each item token after clustering for fast lookup.

###  3.4. Multi-modal Interest-aware Sequence Model

We design an encoder-decoder model for sequence representation, which unifies
sequence modeling and multi-modal fusion.

####  3.4.1. Transformer-based Multi-modal Context Encoder

To pursue personalization, we expect the model to adaptively capture crucial
cues from the given context, _i.e._ , discovered interest tokens (§3.3.2),
denoted by ${\bm{\beta}_{1},\bm{\beta}_{2},\cdots,\bm{\beta}_{M_{i}}}$. We
construct an encoder with $K_{\text{e}}$ Transformer encoder blocks, each of
which follows the standard design (Vaswani et al., 2017). The context encoder
processes the input and generates an encoded token sequence
${\bm{\xi}_{1},\bm{\xi}_{2},\cdots,\bm{\xi}_{M_{i}}}$ for next-step decoding.

####  3.4.2. Transformer-based Interest-aware Decoder

After obtaining the encoded interest tokens
${\bm{\xi}_{1},\bm{\xi}_{2},\cdots,\bm{\xi}_{M_{i}}}$, we design an interest-
aware decoding process to model item-modality-interest relations for precise
and comprehensive sequence representation. Specifically, we construct a
decoder with $K_{\text{d}}$ Transformer decoder blocks. For the multi-head
cross-attention module in each decoder block, we take
${\bm{\xi}_{1},\bm{\xi}_{2},\cdots,\bm{\xi}_{M_{i}}}$ as the key and value. We
add item-wise positional embeddings
$\bm{P}_{i}=[{\bm{p}_{1},\bm{p}_{2},\cdots,\bm{p}_{L_{i}}}]$ to the item
tokens $\bm{X}_{i}=[{\bm{x}_{1},\bm{x}_{2},\cdots,\bm{x}_{L_{i}}}]$ and form
the decoding queries. Let us denote
$\bm{Y}_{i}=[{\bm{y}_{1},\bm{y}_{2},\cdots,\bm{y}_{L_{i}}}]$ as the decoded
output _w.r.t._ $\bm{X}_{i}$, then the embedding (_i.e._ , the sequence
representation) for the $i$-th user is produced by the output of the last
query token, namely $\bm{u}_{i}=\bm{y}_{L_{i}}$.

###  3.5. Self-supervised Contrastive Pre-training

To pursue universal representation for multi-modal interaction sequences, we
design two self-supervised contrastive learning (CL) tasks for pre-training
the model.

####  3.5.1. Sequence-item Contrastive Learning

To capture the correspondence between interaction sequences and candidate
items, we take sequence-item CL as a pre-training task. The objective _w.r.t._
$\bm{u}_{i}$:

(5)
$\ell_{i}^{S-I}=-\log\frac{\exp({\langle\bm{u}_{i},\bm{v}_{T_{i}+1}\rangle/\tau})}{\sum_{j=1}^{B}\exp({\langle\bm{u}_{i},\bm{v}_{T_{j}+1}\rangle/\tau})},$

where $B$ denotes the size of mini-batch. $\tau>0$ us a temperature factor.
The user-item matching scores are computed by eq. 2.

####  3.5.2. Sequence-sequence Contrastive Learning

Apart from sequence-item contrast, we leverage semantic invariance as a self-
supervising signal to enhance the robustness of the sequence model. Given
$\bm{u}_{i},\bm{u}_{i}^{\prime}$ as the representations of two augmented
sequences (see §3.2.2) _w.r.t._ the $i$-th user, we define the sequence-
sequence CL by

(6)
$\ell_{i}^{S-S}=-\log\frac{\exp({\langle\bm{u}_{i},\bm{u}_{i}^{\prime}\rangle/\tau})}{\sum_{j=1}^{B}\exp({\langle\bm{u}_{i},\bm{u}_{j}\rangle/\tau})+\exp({\langle\bm{u}_{i},\bm{u}_{j}^{\prime}\rangle/\tau})}.$

####  3.5.3. Overall Objectives

We sum up pre-train objectives as follows

(7) $\displaystyle\mathcal{L}_{\text{pre-
train}}=\frac{1}{B}\sum_{i=1}^{B}\left[\ell_{i}^{S-I}+\lambda\cdot\ell_{i}^{S-S}+\frac{\gamma}{M_{i}^{2}}\sum_{m,m^{\prime}=1}^{M_{i}}\langle\bm{\xi}_{m},\bm{\xi}_{m^{\prime}}\rangle\right].$

The last term is an orthogonal regularization to diversify interest-aware
decoded results. $\lambda,\gamma>0$ are the weights for different terms.

###  3.6. Efficient Fine-tuning

In order to transfer the recommendation knowledge to downstream domains and
enhance their performance, we further study efficient fine-tuning based on the
pre-trained sequence model. For the tuning objectives, we adopt sequence-item
CL and orthogonal regularization to directly optimize the next-item
prediction, namely,

(8) $\displaystyle\mathcal{L}_{\text{fine-
tune}}=\frac{1}{B}\sum_{i=1}^{B}\left[\ell_{i}^{S-I}+\frac{\gamma}{M_{i}^{2}}\sum_{m,m^{\prime}=1}^{M_{i}}\langle\bm{\xi}_{m},\bm{\xi}_{m^{\prime}}\rangle\right].$

As shown in the right block of fig. 2, we disable the feature augmentation and
the sequence-sequence contrast, so the sequence model only encodes once per
user, which is computation-efficient. Besides, only the item ID embedding
table (if included) and modality-specific adapters need to be tuned while
other modules keep frozen, which is also parameter-efficient (He et al.,
2022). Following Hou et al. (2022), we support _inductive_ and _transductive_
transfer settings for MISSRec.

####  3.6.1. Inductive Transfer

Given a target domain with lots of cold items, making robust predictions is
tricky for ID-based recommenders. We support an ID-agnostic solution to deal
with extreme sparsity, where the model predicts next item for the $i$-th user
by

(9)
$\displaystyle\text{Pr}_{\text{ind}}(I_{T_{i}+1}\mid{I_{1},I_{2},\cdots,I_{T_{i}}})=\text{softmax}(\langle\bm{u}_{i},\bm{v}_{T_{i}+1}\rangle).$

####  3.6.2. Transductive Transfer

Given a target domain of mostly warm items, we pursue more precise
recommendations by utilizing item IDs. Specifically, the model predicts next
item for the $i$-th user by

(10)
$\displaystyle\text{Pr}_{\text{trd}}(I_{T_{i}+1}\mid{I_{1},I_{2},\cdots,I_{T_{i}}})=\text{softmax}(\langle\bar{\bm{u}}_{i},\bm{v}_{T_{i}+1}+\bm{z}_{T_{i}+1}\rangle),$

where $\bm{z}_{T_{i}+1}$ is the ID embeddings of $I_{T_{i}+1}$.
$\bar{\bm{u}}_{i}$ denotes the encoded sequence representation with item IDs
(see the right block in fig. 2).

##  4\. Experiments

###  4.1. Research Questions

We evaluate the proposed method by conducting experiments on three datasets.
We aim to answer the following research questions:

  * RQ1:

Compared with state-of-the-art sequential recommendation models using various
types of information, can MISSRec achieve competitive performance in
downstream domains?

  * RQ2:

Is MISSRec better than other modality-based baselines in utilizing different
modalities? How much impact do multi-modal information and pre-training have
on its efficacy?

  * RQ3:

How do different designs contribute to MISSRec’s efficacy?

Table 1. Statistics of Pre-processed Datasets. “Cover.” denotes the image
coverage among the item set. “Avg. SL” denotes the average length of
interaction sequences.

Datasets | #Users | #Items | #Img. (Cover./%) | #Inters. | Avg. SL.
---|---|---|---|---|---
_Pre-trained_ | 1,361,408 | 446,975 | 94,151 (21.06%) | 14,029,229 | 13.51
\- Food | 115,349 | 39,670 | 29,990 (75.60%) | 1,027,413 | 8.91
\- CDs | 94,010 | 64,439 | 21,166 (32.85%) | 1,118,563 | 12.64
\- Kindle | 138,436 | 98,111 | 0 (0%) | 2,204,596 | 15.93
\- Movies | 281,700 | 59.203 | 8,675 (14.65%) | 3,226,731 | 11.45
\- Home | 731,913 | 185,552 | 34,320 (18.50%) | 6,451,926 | 8.82
Scientific | 8,442 | 4,385 | 1,585 (36.15%) | 59,427 | 7.04
Pantry | 13,101 | 4,898 | 4,587 (93.65%) | 126,962 | 9.69
Instruments | 24,962 | 9,964 | 6,289 (63.12%) | 208,926 | 8.37
Arts | 45,486 | 21,019 | 9,437 (44.90%) | 395,150 | 8.69
Office | 87,436 | 25,986 | 16,628 (63.99%) | 684,837 | 7.84

###  4.2. Experimental Setup

####  4.2.1. Datasets

We adopt 10 domains including “ _Grocery and Gourmet Food_ ”, “ _Home and
Kitchen_ ”, “ _CDs and Vinyl_ ”, “ _Kindle Store_ ”, “ _Movies and TV_ ”, “
_Prime Pantry_ ”, “ _Industrial and Scientific_ ”, “ _Musical Instruments_ ”,
“ _Arts, Crafts and Sewing_ ”, and “ _Office Products_ ”, from the standard
benchmark dataset, Amazon Review (Ni et al., 2019). To provide extensive
evaluations of the transferability, We divide the former 5 and the latter 5
into pre-training and downstream target domains, respectively. We follow Hou
et al. (2022) to pre-process for interaction data and retrieve text
information including _title_ , _categories_ and _brand_ from the metadata. To
support multi-modal inputs, we further crawl item images according to the URLs
in the metadata. The dataset statistics are reported in table 1. While texts
are fully available in the metadata, there are many missing images in the
dataset due to expired products or URLs. We still retain the items without
images in our experiments to keep a fair comparison with previous works (Hou
et al., 2022). As we will show below, our MISSRec can exhibit robust
improvement on incomplete multi-modal datasets.

####  4.2.2. Metrics

Following previous works (Hou et al., 2022; Zhou et al., 2020), we adopt two
standard metrics, _i.e._ , Recall (R@$K$) and NDCG (N@$K$), to evaluate the
retrieval performance. We set $K$ to 10 and 50 for showcases.

####  4.2.3. Baselines

We compare our method with 7 state-of-the-art sequential recommenders,
including (i) 2 pure ID-based recommenders: SASRec (Kang and McAuley, 2018)
and BERT4Rec (Sun et al., 2019); (ii) 1 attribute-aware recommender, S3-Rec
(Zhou et al., 2020); (iii) 4 text-based recommenders, FDSA (Zhang et al.,
2019), ZESRec (Ding et al., 2022), and UniSRec (Hou et al., 2022). We derive
an extra text-feature-based variant from SASRec regarding its simplicity and
effectiveness. For S3-Rec, we inherit previous works (Hou et al., 2022) to
embed the tags with textual features so as to support universal
representation. We adopt both transductive and inductive variants of UniSRec
for comparison. Therefore, we have 9 baselines in total.

Table 2. Comparisons on different target datasets. “T” and “V” stands for text
and visual features. “ _Improv_.” denotes the relative improvement of MISSRec
to the best baselines. The best and second-best results are in bold and
underlined, respectively.

Input Type & Model $\rightarrow$ |  | ID | T+ID | T+V+ID |  _Improv._ w/ ID |  | T | T+V |  _Improv._ w/o ID
---|---|---|---|---|---|---|---|---|---
Dataset | Metric |  | SASRec | BERT4Rec | FDSA | S3-Rec | UniSRec | MISSRec |  | SASRec | ZESRec | UniSRec | MISSRec
Scientific | R@10 |  | 0.1080 | 0.0488 | 0.0899 | 0.0525 | 0.1235 | 0.1360 | 10.12% |  | 0.0994 | 0.0851 | 0.1188 | 0.1278 | 7.58%
N@10 |  | 0.0553 | 0.0243 | 0.0580 | 0.0275 | 0.0634 | 0.0753 | 18.77% |  | 0.0561 | 0.0475 | 0.0641 | 0.0658 | 2.65%
R@50 |  | 0.2042 | 0.1185 | 0.1732 | 0.1418 | 0.2473 | 0.2431 | - |  | 0.2162 | 0.1746 | 0.2394 | 0.2375 | -
N@50 |  | 0.0760 | 0.0393 | 0.0759 | 0.0468 | 0.0904 | 0.0983 | 8.74% |  | 0.0815 | 0.0670 | 0.0903 | 0.0893 | -
Pantry | R@10 |  | 0.0501 | 0.0308 | 0.0395 | 0.0444 | 0.0693 | 0.0779 | 12.41% |  | 0.0585 | 0.0454 | 0.0636 | 0.0771 | 21.23%
N@10 |  | 0.0218 | 0.0152 | 0.0209 | 0.0214 | 0.0311 | 0.0365 | 17.36% |  | 0.0285 | 0.0230 | 0.0306 | 0.0345 | 12.75%
R@50 |  | 0.1322 | 0.1030 | 0.1151 | 0.1315 | 0.1827 | 0.1875 | 2.63% |  | 0.1647 | 0.1141 | 0.1658 | 0.1833 | 10.55%
N@50 |  | 0.0394 | 0.0305 | 0.0370 | 0.0400 | 0.0556 | 0.0598 | 7.55% |  | 0.0523 | 0.0378 | 0.0527 | 0.0571 | 8.35%
Instruments | R@10 |  | 0.1118 | 0.0813 | 0.1070 | 0.1056 | 0.1267 | 0.1300 | 2.60% |  | 0.1127 | 0.0783 | 0.1189 | 0.1201 | 1.01%
N@10 |  | 0.0612 | 0.0620 | 0.0796 | 0.0713 | 0.0748 | 0.0843 | 5.90% |  | 0.0661 | 0.0497 | 0.0680 | 0.0771 | 13.38%
R@50 |  | 0.2106 | 0.1454 | 0.1890 | 0.1927 | 0.2387 | 0.2370 | - |  | 0.2104 | 0.1387 | 0.2255 | 0.2218 | -
N@50 |  | 0.0826 | 0.0756 | 0.0972 | 0.0901 | 0.0991 | 0.1071 | 8.07% |  | 0.0873 | 0.0627 | 0.0912 | 0.0988 | 8.33%
Arts | R@10 |  | 0.1108 | 0.0722 | 0.1002 | 0.1003 | 0.1239 | 0.1314 | 6.05% |  | 0.0977 | 0.0664 | 0.1066 | 0.1119 | 4.97%
N@10 |  | 0.0587 | 0.0479 | 0.0714 | 0.0601 | 0.0712 | 0.0767 | 7.42% |  | 0.0562 | 0.0375 | 0.0586 | 0.0625 | 6.66%
R@50 |  | 0.2030 | 0.1367 | 0.1779 | 0.1888 | 0.2347 | 0.2410 | 2.68% |  | 0.1916 | 0.1323 | 0.2049 | 0.2100 | 2.49%
N@50 |  | 0.0788 | 0.0619 | 0.0883 | 0.0793 | 0.0955 | 0.1002 | 4.92% |  | 0.0766 | 0.0518 | 0.0799 | 0.0836 | 4.63%
Office | R@10 |  | 0.1056 | 0.0825 | 0.1118 | 0.1030 | 0.1280 | 0.1275 | - |  | 0.0929 | 0.0641 | 0.1013 | 0.1038 | 2.47%
N@10 |  | 0.0710 | 0.0634 | 0.0868 | 0.0653 | 0.0831 | 0.0856 | - |  | 0.0582 | 0.0391 | 0.0619 | 0.0666 | 7.59%
R@50 |  | 0.1627 | 0.1227 | 0.1665 | 0.1613 | 0.2016 | 0.2005 | - |  | 0.1580 | 0.1113 | 0.1702 | 0.1701 | -
N@50 |  | 0.0835 | 0.0721 | 0.0987 | 0.0780 | 0.0991 | 0.1012 | 2.12% |  | 0.0723 | 0.0493 | 0.0769 | 0.0808 | 5.07%

####  4.2.4. Implementation Details

We take Adam as the common optimizer in our comparison, and we carefully
choose suitable learning rates from $\\{3e^{-4},1e^{-3},3e^{-3},1e^{-2}\\}$
for different baselines. We take 100 epochs for pre-training. And for fine-
tuning, we implement early stopping with a patience of 10 to prevent over-
fitting. We adopt CLIP-B/32 (Radford et al., 2021; Wang et al., 2022b) as the
basic feature encoder to extract [CLS] features for each text and image. The
modality adapters then transform the features into the 300-d latent space for
sequence representation. In the MID module (§3.3), the number of interest
prototypes, $K_{\text{c}}$, is set to 2% of the item amount during pre-
training, and is set to 50% of the item amount during fine-tuning. To keep a
consistent parameter scale with baselines, we build our model with 1
Transformer encoder block and 1 decoder block, _i.e._ ,
$K_{\text{e}}=K_{\text{d}}=1$. The loss weights in eqs. 7 and 8 are set
$\lambda=1e^{-3}$ and $\gamma=1e^{-4}$, respectively. The temperature factor
for contrastive learning objectives, _e.g._ , eqs. 5 and 6, is set
$\tau=0.07$. The concentration factor $\alpha$ in §3.2.4 is set to a learnable
variable. The data and code have been released on
https://github.com/gimpong/MM23-MISSRec.

Figure 4. Performance comparison on long-tailed items. The histogram
represents the number of test samples (_i.e._ , interacted items) of different
popularities. The line chart shows the relative improvement to
SASRec${}_{\text{ID}}$ on Recall@10.

###  4.3. Comparasion with State-of-the-arts (RQ1)

####  4.3.1. Overall Performance

The overall comparison results on five different downstream datasets are shown
in table 2, from which we obtain three findings. (i) Text features can be
exploited as an effective alternative or supplement to ID features, and pre-
training can further enhance their efficacy. Comparing the SASRec variants
with different types of input, the text-based variant achieves competitive
performance and outperforms the ID-based variant on “Pantry” and “Instruments”
datasets. Besides, text-enhanced approaches FDSA and S3-Rec outperform ID-
based BERT4Rec by large margins. By adopting pre-training, UniSRec achieves
further improvement. (ii) ID information is still important for
personalization. Transductive methods generally perform better than inductive
ones. (iii) Under both transductive and inductive settings, our MISSRec
outperforms state-of-the-art baselines in most cases, and the margins are
considerable. We attribute the performance gain to three factors. First,
multi-modal information reflects more precise and comprehensive user
preferences, benefiting personalization. Second, the synergy of different
designs in MISSRec contributes to the efficacy of multi-modal sequence
representation. Third, pre-training further strengthened the above advantages.

####  4.3.2. Performance _w.r.t._ Different Item Frequencies

In this sub-section, we select the best performers from table 2 and compare
their efficacy _w.r.t._ long-tailed items. As shown in fig. 4, all modality-
based methods demonstrate effectiveness to ID-based SASRec on the least
frequent items, while the performance gains begin to vanish and even turn to
degradation as the items warm up. Compared with other baselines, MISSRec with
multi-modal learning shows the best few-short performance and also keeps
robust improvement with more frequent items, suggesting its feasibility in
addressing long-tailed and cold-start recommendation issues.

###  4.4. Multi-modal & Pre-training Analyses (RQ2)

Table 3. Model comparison _w.r.t._ various input types on the full-modality
data subset. We can see the superior capability of MISSRec in leveraging
modality features.

|  |  | Scientific |  | Office
---|---|---|---|---|---
Input Type | Variant |  | R@10 | R@50 | N@10 | N@50 |  | R@10 | R@50 | N@10 | N@50
ID+T | UniSRec |  | 0.1676 | 0.3200 | 0.0872 | 0.1208 |  | 0.1376 | 0.2152 | 0.0937 | 0.1104
MISSRec |  | 0.1697 | 0.3249 | 0.0890 | 0.1227 |  | 0.1398 | 0.2195 | 0.0953 | 0.1122
ID+V | UniSRec |  | 0.1631 | 0.3112 | 0.0857 | 0.1188 |  | 0.1324 | 0.2071 | 0.0921 | 0.1086
MISSRec |  | 0.1657 | 0.3165 | 0.0869 | 0.1198 |  | 0.1375 | 0.2152 | 0.0946 | 0.1113
ID+T+V | UniSRec |  | 0.1680 | 0.3209 | 0.0865 | 0.1194 |  | 0.1407 | 0.2203 | 0.0957 | 0.1133
MISSRec |  | 0.1711 | 0.3265 | 0.0890 | 0.1227 |  | 0.1421 | 0.2223 | 0.0966 | 0.1138
T | UniSRec |  | 0.1602 | 0.3202 | 0.0806 | 0.1148 |  | 0.1183 | 0.1893 | 0.0773 | 0.0926
MISSRec |  | 0.1619 | 0.3239 | 0.0814 | 0.1165 |  | 0.1186 | 0.1899 | 0.0787 | 0.0943
V | UniSRec |  | 0.1568 | 0.3135 | 0.0782 | 0.1118 |  | 0.1140 | 0.1823 | 0.0763 | 0.0910
MISSRec |  | 0.1575 | 0.3144 | 0.0789 | 0.1118 |  | 0.1152 | 0.1848 | 0.0771 | 0.0923
T+V | UniSRec |  | 0.1609 | 0.3216 | 0.0808 | 0.1141 |  | 0.1205 | 0.1929 | 0.0792 | 0.0945
MISSRec |  | 0.1633 | 0.3265 | 0.0812 | 0.1160 |  | 0.1209 | 0.1936 | 0.0798 | 0.0953

####  4.4.1. Comparison with Different Input Modalities

In this sub-section, we investigate the capacities of different modality-based
sequence models in leveraging multi-modal information. Specifically, we filter
out the items without images in “Scientific” and “Office” datasets. Then we
conduct experiments on the filtered subsets to enable horizontal comparison
among different settings, _e.g._ , visual-only _vs_ text-only. To ensure
objective evaluation, we adopt the same CLIP-B/32 feature extractors for all
comparison methods. We report the results in table 3. Interestingly, although
utilizing visual modality alone does not lead to satisfactory results, the
multi-modal synergy creates an effect of “1+1¿2”, which highlights the
significance of multi-modal sequential modeling. Besides, MISSRec demonstrates
robust and competitive performance under various settings.

Table 4. Analysis of the effect of pre-training.

Model: MISSRec |  | Scientific |  | Office
---|---|---|---|---
w/ ID? | Pre-train? |  | R@10 | R@50 | N@10 | N@50 |  | R@10 | R@50 | N@10 | N@50
✔ | ✔ |  | 0.1360 | 0.2431 | 0.0753 | 0.0983 |  | 0.1275 | 0.2005 | 0.0856 | 0.1012
✗ |  | 0.1282 | 0.2376 | 0.0711 | 0.0946 |  | 0.1269 | 0.2001 | 0.0848 | 0.1005
_Improv._ w/ ID |  | 6.08% | 2.31% | 5.91% | 3.91% |  | 0.47% | 0.20% | 0.94% | 0.70%
✗ | ✔ |  | 0.1278 | 0.2375 | 0.0658 | 0.0893 |  | 0.1038 | 0.1701 | 0.0666 | 0.0808
✗ |  | 0.1269 | 0.2354 | 0.0659 | 0.0891 |  | 0.1072 | 0.1726 | 0.0694 | 0.0834
_Improv._ w/o ID |  | 0.71% | 0.89% | -0.15% | 0.22% |  | -3.17% | -1.45% | -4.03% | -3.12%

####  4.4.2. Pre-training Analysis

Here we examine the effectiveness of pre-training to MISSRec. As shown in
table 4, we can see that pre-training enhances recommendation performance in
target domains under transductive settings. However, we observe the negative
transfer under inductive settings, especially on the larger dataset, Office. A
possible reason for the phenomenon is that the model can not adapt well to the
target data with a large domain gap accompanied by quite limited tunable
parameters (_i.e._ , adapters).

###  4.5. Model Analyses (RQ3)

To investigate the efficacy of different components, we construct 5 variants
of MISSRec${}_{\text{T+V+ID}}$ for comparison. Specifically, as presented by
the leftmost column in table 5, variant (1) removes the sequence-sequence
contrastive task (§3.5.2) from pre-training. Variant (2) removes the modality-
specific adapters (§3.2.3) so that the sequence model directly consumes 512-d
CLIP features. Variant (3) replaces the interest-aware decoder (§3.4.2) layer
with another encoder layer and builds a 2-layer contextual encoder that only
processes interest tokens. The sequence embedding is produced by mean-pooling
the output token sequence. Variant (4) only keeps the decoding structure and
builds a 2-layer decoder-only model. It also removes the cross-attention
module at each decoder layer due to the absence of encoder output. Variant (5)
removes the orthogonal regularization in learning objectives (eqs. 7 and 8).

####  4.5.1. Effects of Modality Adapters

Comparing variants (2) with (0), we can see significant performance decays,
owing to the semantic gap between general multi-modal features and user
interests. As a result, multi-modal features themselves as item
representations could not provide sufficient personalization for
recommendation, and modality adapters are required. On the other hand,
although end-to-end optimizing feature extractors with the recommender also
helps to mitigate the gap, as revealed by (Yuan et al., 2023; Wang et al.,
2022a), it will consume much more computation resources. Anyway, parameter-
efficient tuning with modality adapters can be preferable.

####  4.5.2. Effects of Learning Objectives

Comparing (1) and (0), we learn that $\ell_{i}^{\text{S-S}}$ in the pre-
training stage can enhance the representation robustness, as it helps to
improve downstream recommendation. Comparing (5) and (0), we find that the
orthogonal regularization leads to slightly better results on the Scientific
dataset but shows a negative impact on the Office dataset. We can infer that
the orthogonal regularization is sensitive to the domain or the weighting
factor, which requires careful choosing according to specific scenarios.

####  4.5.3. Effects of Interest-aware Decoding and Contextual Encoding

We can learn that variant (3) exhibits poor performance among the variants,
suggesting the necessity of the collaboration between context encoding and
interest-aware decoding. Besides, by comparing variants (4) and (0), variant
(4) without accessing encoded interest information also shows performance
degradation, which justifies the efficacy of interest mining and contextual
modeling.

Table 5. Ablation study with MISSRec.

|  | Scientific |  | Office
---|---|---|---|---
Variant |  | R@10 | R@50 | N@10 | N@50 |  | R@10 | R@50 | N@10 | N@50
(0) MISSRec |  | 0.1360 | 0.2431 | 0.0753 | 0.0983 |  | 0.1275 | 0.2005 | 0.0856 | 0.1012
(1) - w/o $\ell^{S-S}_{i}$ in Eq.(7) |  | 0.1337 | 0.2390 | 0.0710 | 0.0926 |  | 0.1273 | 0.2000 | 0.0847 | 0.1002
(2) - w/o Modality Adapters |  | 0.1058 | 0.1896 | 0.0582 | 0.0759 |  | 0.1029 | 0.1617 | 0.0708 | 0.0838
(3) - 2-Layer Encoder Only |  | 0.1220 | 0.2186 | 0.0680 | 0.0890 |  | 0.1175 | 0.1849 | 0.0823 | 0.0974
(4) - 2-Layer Decoder Only |  | 0.1172 | 0.2092 | 0.0647 | 0.0844 |  | 0.1003 | 0.1575 | 0.0696 | 0.0821
(5) - w/o Oth. Reg. in Eqs.(7-8) |  | 0.1355 | 0.2403 | 0.0745 | 0.0974 |  | 0.1286 | 0.2019 | 0.0873 | 0.1034

##  5\. Conclusions

This paper addresses the limitations of existing sequential recommendation
models that rely heavily on ID features by exploring the potential of multi-
modal information. We propose MISSRec, a novel multi-modal pre-training and
transfer learning framework for sequential recommendation, which effectively
tackles the cold-start problem and allows for efficient domain adaptation. By
utilizing a transformer-based contextual encoder, an interest-aware decoder, a
lightweight dynamic fusion module, MISSRec demonstrates improved performance
and generalizability compared to existing methods. Extensive experiments also
showcase the compatibility and robustness of MISSRec in handling incomplete or
missing modalities, reinforcing its pragmatic value for real-world scenarios.
More importantly, our paper suggests a promising direction for future research
in leveraging multi-modal information for sequential recommendation. We hope
MISSRec can provide some inspiration to further explorations.

###### Acknowledgements.

We want to thank the anonymous reviewers and the meta-reviewer for their
valuable comments and suggestions. This research is supported by National
Natural Science Foundation of China (Grant No.62171248 and Grant No.62276154),
the Natural Science Foundation of Guangdong Province (Grant No.
2023A1515012914), Shenzhen Science and Technology Program
(JCYJ20220818101012025), Basic Research Fund of Shenzhen City (Grant No.
JCYJ20210324120012033 and JSGG20210802154402007), the Major Key Project of PCL
for Experiments and Applications (PCL2021A07, and PCL2021A06), and Overseas
Cooperation Research Fund of Tsinghua Shenzhen International Graduate School
(HW2021008). We gratefully acknowledge the support of
MindSpore111https://www.mindspore.cn, which is a new deep learning framework
used for this research.

## References

  * (1)
  * Abdollahpouri et al. (2020) Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. 2020.  The connection between popularity bias, calibration, and fairness in recommendation. In _RecSys_.
  * Bapna and Firat (2019) Ankur Bapna and Orhan Firat. 2019.  Simple, Scalable Adaptation for Neural Machine Translation. In _EMNLP_.
  * Cai et al. (2021) Desheng Cai, Shengsheng Qian, Quan Fang, and Changsheng Xu. 2021\.  Heterogeneous hierarchical feature aggregation network for personalized micro-video recommendation.  _IEEE TMM_ (2021).
  * Cen et al. (2020) Yukuo Cen, Jianwei Zhang, Xu Zou, Chang Zhou, Hongxia Yang, and Jie Tang. 2020\.  Controllable multi-interest framework for recommendation. In _KDD_.
  * Chen et al. (2017) Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and Tat-Seng Chua. 2017\.  Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention. In _SIGIR_.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations. In _ICML_.
  * Chen et al. (2022) Yongjun Chen, Zhiwei Liu, Jia Li, Julian McAuley, and Caiming Xiong. 2022.  Intent contrastive learning for sequential recommendation. In _WWW_.
  * Cheng et al. (2021) Mingyue Cheng, Fajie Yuan, Qi Liu, Xin Xin, and Enhong Chen. 2021.  Learning transferable user representations with sequential behaviors via contrastive pre-training. In _ICDM_.
  * Dai et al. (2022) Quanyu Dai, Yalei Lv, Jieming Zhu, Junjie Ye, Zhenhua Dong, Rui Zhang, Shu-Tao Xia, and Ruiming Tang. 2022\.  LCD: Adaptive Label Correction for Denoising Music Recommendation. In _CIKM_.
  * Deldjoo et al. (2020) Yashar Deldjoo, Markus Schedl, Paolo Cremonesi, and Gabriella Pasi. 2020.  Recommender systems leveraging multimedia content.  _ACM CSUR_ (2020).
  * Deldjoo et al. (2021) Yashar Deldjoo, Markus Schedl, Balázs Hidasi, Yinwei Wei, and Xiangnan He. 2021.  Multimedia recommender systems: Algorithms and challenges.  In _Recommender systems handbook_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL_.
  * Ding et al. (2022) Hao Ding, Yifei Ma, Anoop Deoras, Yuyang Wang, and Hao Wang. 2022.  Zero-Shot Recommender Systems. In _ICLR Workshops_.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In _ICLR_.
  * Du et al. (2016) Mingjing Du, Shifei Ding, and Hongjie Jia. 2016.  Study on density peaks clustering based on k-nearest neighbors and principal component analysis.  _Elsevier KBS_ (2016).
  * Du et al. (2020) Xiaoyu Du, Xiang Wang, Xiangnan He, Zechao Li, Jinhui Tang, and Tat-Seng Chua. 2020\.  How to learn item representation for cold-start multimedia recommendation?. In _MM_.
  * Han et al. (2022) Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al. 2022\.  A survey on vision transformer.  _IEEE TPAMI_ (2022).
  * He et al. (2022) Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2022\.  Towards a Unified View of Parameter-Efficient Transfer Learning. In _ICLR_.
  * He et al. (2020b) Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020b.  Momentum contrast for unsupervised visual representation learning. In _CVPR_.
  * He et al. (2021) Li He, Hongxu Chen, Dingxian Wang, Shoaib Jameel, Philip Yu, and Guandong Xu. 2021\.  Click-through rate prediction with multi-modal hypergraphs. In _CIKM_.
  * He and McAuley (2016a) Ruining He and Julian McAuley. 2016a.  Fusing similarity models with markov chains for sparse sequential recommendation. In _ICDM_.
  * He and McAuley (2016b) Ruining He and Julian McAuley. 2016b.  VBPR: visual bayesian personalized ranking from implicit feedback. In _AAAI_.
  * He et al. (2020a) Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020a.  Lightgcn: Simplifying and powering graph convolution network for recommendation. In _SIGIR_.
  * Hidasi et al. (2016) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016.  Session-based recommendations with recurrent neural networks. In _ICLR_.
  * Hou et al. (2023) Yupeng Hou, Zhankui He, Julian McAuley, and Wayne Xin Zhao. 2023\.  Learning vector-quantized item representation for transferable sequential recommenders. In _WWW_.
  * Hou et al. (2022) Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. 2022\.  Towards Universal Sequence Representation Learning for Recommender Systems. In _KDD_.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-efficient transfer learning for NLP. In _ICML_.
  * Huang et al. (2018) Xiaowen Huang, Shengsheng Qian, Quan Fang, Jitao Sang, and Changsheng Xu. 2018.  Csan: Contextual self-attention network for user sequential recommendation. In _MM_.
  * Jiang et al. (2020) Hao Jiang, Wenjie Wang, Yinwei Wei, Zan Gao, Yinglong Wang, and Liqiang Nie. 2020\.  What aspect do you like: Multi-scale time-aware user interest modeling for micro-video recommendation. In _MM_.
  * Jiang et al. (2021) Xunqiang Jiang, Yuanfu Lu, Yuan Fang, and Chuan Shi. 2021\.  Contrastive pre-training of gnns on heterogeneous graphs. In _CIKM_.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _ICDM_.
  * Khan et al. (2022) Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. 2022\.  Transformers in vision: A survey.  _ACM CSUR_ (2022).
  * Lei et al. (2021) Chenyi Lei, Yong Liu, Lingzi Zhang, Guoxin Wang, Haihong Tang, Houqiang Li, and Chunyan Miao. 2021.  Semi: A sequential multi-modal information transfer network for e-commerce micro-video recommendations. In _KDD_. 3161–3171.
  * Li et al. (2019) Chao Li, Zhiyuan Liu, Mengmeng Wu, Yuchi Xu, Huan Zhao, Pipei Huang, Guoliang Kang, Qiwei Chen, Wei Li, and Dik Lun Lee. 2019\.  Multi-interest network with dynamic routing for recommendation at Tmall. In _CIKM_.
  * Li et al. (2020) Jiacheng Li, Yujie Wang, and Julian McAuley. 2020.  Time interval aware self-attention for sequential recommendation. In _WSDM_.
  * Li et al. (2022) Jian Li, Jieming Zhu, Qiwei Bi, Guohao Cai, Lifeng Shang, Zhenhua Dong, Xin Jiang, and Qun Liu. 2022\.  MINER: multi-interest matching network for news recommendation. In _ACL Findings_.
  * Liang et al. (2023) Jiahao Liang, Xiangyu Zhao, Muyang Li, Zijian Zhang, Wanyu Wang, Haochen Liu, and Zitao Liu. 2023.  MMMLP: Multi-modal Multilayer Perceptron for Sequential Recommendations. In _WWW_.
  * Lin et al. (2023) Guanyu Lin, Chen Gao, Yu Zheng, Jianxin Chang, Yanan Niu, Yang Song, Zhiheng Li, Depeng Jin, and Yong Li. 2023.  Dual-interest Factorization-heads Attention for Sequential Recommendation. In _WWW_.
  * Liu et al. (2023) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023\.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM CSUR_ (2023).
  * Liu et al. (2022b) Qijiong Liu, Jieming Zhu, Quanyu Dai, and Xiaoming Wu. 2022b.  Boosting deep ctr prediction with a plug-and-play pre-trainer for news recommendation. In _COLING_.
  * Liu et al. (2021) Yong Liu, Susen Yang, Chenyi Lei, Guoxin Wang, Haihong Tang, Juyong Zhang, Aixin Sun, and Chunyan Miao. 2021\.  Pre-training graph transformer with multimodal side information for recommendation. In _MM_.
  * Liu et al. (2022a) Zhuang Liu, Yunpu Ma, Matthias Schubert, Yuanxin Ouyang, and Zhang Xiong. 2022a.  Multi-Modal Contrastive Pre-training for Recommendation. In _ICMR_.
  * Ma et al. (2019) Chen Ma, Peng Kang, and Xue Liu. 2019.  Hierarchical gating networks for sequential recommendation. In _KDD_.
  * Mao et al. (2021) Kelong Mao, Jieming Zhu, Jinpeng Wang, Quanyu Dai, Zhenhua Dong, Xi Xiao, and Xiuqiang He. 2021.  SimpleX: A simple and strong baseline for collaborative filtering. In _CIKM_.
  * Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019.  Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In _EMNLP_.
  * Ni et al. (2018) Yabo Ni, Dan Ou, Shichen Liu, Xiang Li, Wenwu Ou, Anxiang Zeng, and Luo Si. 2018.  Perceive your users in depth: Learning universal user representations from multiple e-commerce tasks. In _KDD_.
  * Pan et al. (2022) Xingyu Pan, Yushuo Chen, Changxin Tian, Zihan Lin, Jinpeng Wang, He Hu, and Wayne Xin Zhao. 2022.  Multimodal Meta-Learning for Cold-Start Sequential Recommendation. In _CIKM_.
  * Qiu et al. (2022) Ruihong Qiu, Zi Huang, Hongzhi Yin, and Zijian Wang. 2022\.  Contrastive learning for representation degeneration problem in sequential recommendation. In _CIKM_.
  * Qiu et al. (2021) Zhaopeng Qiu, Xian Wu, Jingyue Gao, and Wei Fan. 2021\.  U-BERT: Pre-training user representations for improved recommendation. In _AAAI_.
  * Quadrana et al. (2018) Massimo Quadrana, Paolo Cremonesi, and Dietmar Jannach. 2018\.  Sequence-aware recommender systems.  _ACM CSUR_ (2018).
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _ICML_.
  * Rendle et al. (2010) Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010\.  Factorizing personalized markov chains for next-basket recommendation. In _WWW_.
  * Resnick and Varian (1997) Paul Resnick and Hal R Varian. 1997.  Recommender systems.  _Commun. ACM_ (1997).
  * Schein et al. (2002) Andrew I Schein, Alexandrin Popescul, Lyle H Ungar, and David M Pennock. 2002.  Methods and metrics for cold-start recommendations. In _SIGIR_.
  * Song et al. (2023a) Kunzhe Song, Qingfeng Sun, Can Xu, Kai Zheng, and Yaming Yang. 2023a.  Self-Supervised Multi-Modal Sequential Recommendation.  _arXiv preprint arXiv:2304.13277_ (2023).
  * Song et al. (2023b) Xuemeng Song, Chun Wang, Changchang Sun, Shanshan Feng, Min Zhou, and Liqiang Nie. 2023b.  MM-FRec: Multi-Modal Enhanced Fashion Item Recommendation.  _IEEE TKDE_ (2023).
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _JMLR_ (2014).
  * Su et al. (2020) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020\.  VL-BERT: Pre-training of Generic Visual-Linguistic Representations. In _ICLR_.
  * Su et al. (2021) Yixin Su, Rui Zhang, Sarah Erfani, and Zhenghua Xu. 2021\.  Detecting beneficial feature interactions for recommender systems. In _AAAI_.
  * Su et al. (2022) Yixin Su, Yunxiang Zhao, Sarah Erfani, Junhao Gan, and Rui Zhang. 2022.  Detecting arbitrary order beneficial feature interactions for recommender systems. In _KDD_.
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In _CIKM_.
  * Sun et al. (2020) Rui Sun, Xuezhi Cao, Yan Zhao, Junchen Wan, Kun Zhou, Fuzheng Zhang, Zhongyuan Wang, and Kai Zheng. 2020\.  Multi-modal knowledge graphs for recommender systems. In _CIKM_.
  * Tang and Wang (2018) Jiaxi Tang and Ke Wang. 2018.  Personalized top-n sequential recommendation via convolutional sequence embedding. In _WSDM_.
  * Tao et al. (2022) Zhulin Tao, Xiaohao Liu, Yewei Xia, Xiang Wang, Lifang Yang, Xianglin Huang, and Tat-Seng Chua. 2022.  Self-supervised learning for multimedia recommendation.  _IEEE TMM_ (2022).
  * Tian et al. (2022) Changxin Tian, Zihan Lin, Shuqing Bian, Jinpeng Wang, and Wayne Xin Zhao. 2022.  Temporal Contrastive Pre-Training for Sequential Recommendation. In _CIKM_.
  * Tolstikhin et al. (2021) Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. 2021\.  Mlp-mixer: An all-mlp architecture for vision.  _NIPS_ (2021).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _NIPS_.
  * Wang et al. (2023a) Chenyang Wang, Weizhi Ma, Chong Chen, Min Zhang, Yiqun Liu, and Shaoping Ma. 2023a.  Sequential recommendation with multiple contrast signals.  _ACM TOIS_ (2023).
  * Wang et al. (2023b) Hui Wang, Kun Zhou, Xin Zhao, Jingyuan Wang, and Ji-Rong Wen. 2023b.  Curriculum Pre-Training Heterogeneous Subgraph Transformer for Top-N Recommendation.  _ACM TOIS_ (2023).
  * Wang et al. (2022a) Jie Wang, Fajie Yuan, Mingyue Cheng, Joemon M Jose, Chenyun Yu, Beibei Kong, Zhijin Wang, Bo Hu, and Zang Li. 2022a.  TransRec: Learning Transferable Recommendation from Mixture-of-Modality Feedback.  _arXiv preprint arXiv:2206.06190_ (2022).
  * Wang et al. (2022b) Jinpeng Wang, Ziyun Zeng, Bin Chen, Yuting Wang, Dongliang Liao, Gongfu Li, Yiru Wang, Shu-Tao Xia, and Peng Cheng Intelligence. 2022b.  Hugs Are Better Than Handshakes: Unsupervised Cross-Modal Transformer Hashing with Multi-granularity Alignment. In _BMVC_.
  * Wang et al. (2019) Shoujin Wang, Liang Hu, Yan Wang, Longbing Cao, Quan Z. Sheng, and Mehmet Orgun. 2019\.  Sequential Recommender Systems: Challenges, Progress and Prospects. In _IJCAI_.
  * Wei et al. (2023) Wei Wei, Chao Huang, Lianghao Xia, and Chuxu Zhang. 2023\.  Multi-Modal Self-Supervised Learning for Recommendation.  _arXiv preprint arXiv:2302.10632_ (2023).
  * Wei et al. (2020) Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, and Tat-Seng Chua. 2020.  Graph-refined convolutional network for multimedia recommendation with implicit feedback. In _MM_.
  * Wei et al. (2019) Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2019\.  MMGCN: Multi-modal graph convolution network for personalized recommendation of micro-video. In _MM_.
  * Wu et al. (2022) Chuhan Wu, Fangzhao Wu, Tao Qi, Chao Zhang, Yongfeng Huang, and Tong Xu. 2022\.  MM-Rec: Visiolinguistic Model Empowered Multimodal News Recommendation. In _SIGIR_.
  * Xiao et al. (2021) Chaojun Xiao, Ruobing Xie, Yuan Yao, Zhiyuan Liu, Maosong Sun, Xu Zhang, and Leyu Lin. 2021.  UPRec: User-aware Pre-training for Recommender Systems.  _arXiv preprint arXiv:2102.10989_ (2021).
  * Xie et al. (2022) Xu Xie, Fei Sun, Zhaoyang Liu, Shiwen Wu, Jinyang Gao, Jiandong Zhang, Bolin Ding, and Bin Cui. 2022\.  Contrastive learning for sequential recommendation. In _ICDE_.
  * Xun et al. (2021) Jiahao Xun, Shengyu Zhang, Zhou Zhao, Jieming Zhu, Qi Zhang, Jingjie Li, Xiuqiang He, Xiaofei He, Tat-Seng Chua, and Fei Wu. 2021\.  Why do we click: visual impression-aware news recommendation. In _MM_.
  * Yang et al. (2021) Shiquan Yang, Rui Zhang, Sarah M Erfani, and Jey Han Lau. 2021\.  UniMF: A Unified Framework to Incorporate Multimodal Knowledge Bases into End-to-End Task-Oriented Dialogue Systems.. In _IJCAI_.
  * Yi et al. (2022) Zixuan Yi, Xi Wang, Iadh Ounis, and Craig Macdonald. 2022\.  Multi-modal graph contrastive learning for micro-video recommendation. In _SIGIR_.
  * Yuan et al. (2020) Fajie Yuan, Xiangnan He, Alexandros Karatzoglou, and Liguang Zhang. 2020.  Parameter-efficient transfer from sequential behaviors for user modeling and recommendation. In _SIGIR_.
  * Yuan et al. (2019) Fajie Yuan, Alexandros Karatzoglou, Ioannis Arapakis, Joemon M Jose, and Xiangnan He. 2019.  A simple convolutional generative network for next item recommendation. In _WSDM_.
  * Yuan et al. (2023) Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni. 2023\.  Where to Go Next for Recommender Systems? ID-vs. Modality-based recommender models revisited. In _SIGIR_.
  * Zhang et al. (2021c) Jinghao Zhang, Yanqiao Zhu, Qiang Liu, Shu Wu, Shuhui Wang, and Liang Wang. 2021c.  Mining latent structures for multimedia recommendation. In _MM_.
  * Zhang et al. (2023) Lingzi Zhang, Xin Zhou, and Zhiqi Shen. 2023.  Multimodal Pre-training Framework for Sequential Recommendation via Contrastive Learning.  _arXiv preprint arXiv:2303.11879_ (2023).
  * Zhang et al. (2021b) Qi Zhang, Jingjie Li, Qinglin Jia, Chuyuan Wang, Jieming Zhu, Zhaowei Wang, and Xiuqiang He. 2021b.  UNBERT: User-News Matching BERT for News Recommendation.. In _IJCAI_.
  * Zhang et al. (2022) Shengyu Zhang, Lingxiao Yang, Dong Yao, Yujie Lu, Fuli Feng, Zhou Zhao, Tat-Seng Chua, and Fei Wu. 2022\.  Re4: Learning to Re-contrast, Re-attend, Re-construct for Multi-interest Recommendation. In _WWW_.
  * Zhang et al. (2019) Tingting Zhang, Pengpeng Zhao, Yanchi Liu, Victor S Sheng, Jiajie Xu, Deqing Wang, Guanfeng Liu, Xiaofang Zhou, et al. 2019\.  Feature-level Deeper Self-Attention Network for Sequential Recommendation.. In _IJCAI_.
  * Zhang et al. (2021a) Yang Zhang, Fuli Feng, Xiangnan He, Tianxin Wei, Chonggang Song, Guohui Ling, and Yongdong Zhang. 2021a.  Causal intervention for leveraging popularity bias in recommendation. In _SIGIR_.
  * Zhao (2022) Qihang Zhao. 2022\.  RESETBERT4Rec: A pre-training model integrating time and user historical behavior for sequential recommendation. In _SIGIR_.
  * Zhao et al. (2023) Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023\.  A survey of large language models.  _arXiv preprint arXiv:2303.18223_ (2023).
  * Zhou et al. (2018) Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018\.  Deep interest network for click-through rate prediction. In _KDD_.
  * Zhou et al. (2023b) Hongyu Zhou, Xin Zhou, Zhiwei Zeng, Lingzi Zhang, and Zhiqi Shen. 2023b.  A Comprehensive Survey on Multimodal Recommender Systems: Taxonomy, Evaluation, and Future Directions.  _arXiv preprint arXiv:2302.04473_ (2023).
  * Zhou et al. (2020) Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen. 2020\.  S3-Rec: Self-supervised Learning for Sequential Recommendation with Mutual Information Maximization. In _CIKM_.
  * Zhou et al. (2022) Kun Zhou, Hui Yu, Wayne Xin Zhao, and Ji-Rong Wen. 2022\.  Filter-enhanced MLP is all you need for sequential recommendation. In _WWW_.
  * Zhou et al. (2023a) Xin Zhou, Hongyu Zhou, Yong Liu, Zhiwei Zeng, Chunyan Miao, Pengwei Wang, Yuan You, and Feijun Jiang. 2023a.  Bootstrap Latent Representations for Multi-Modal Recommendation. In _WWW_.
  * Zhu et al. (2021) Feng Zhu, Yan Wang, Chaochao Chen, Jun Zhou, Longfei Li, and Guanfeng Liu. 2021\.  Cross-domain recommendation: challenges, progress, and prospects. In _IJCAI_.