# Contextual Collaboration: Uniting Collaborative Filtering with Pre-trained
Language Models

Chen Wang<EMAIL_ADDRESS>0000-0001-5264-3305 University of Illinois
ChicagoUSA ,  Liangwei Yang<EMAIL_ADDRESS>0000-0001-5660-766X University of
Illinois ChicagoUSA ,  Zhiwei Liu<EMAIL_ADDRESS>Salesforce AIUSA ,
Xiaolong Liu<EMAIL_ADDRESS>,  Mingdai Yang<EMAIL_ADDRESS>University of
Illinois ChicagoUSA ,  Yueqing Liang<EMAIL_ADDRESS>Illinois Institute
of TechnologyUSA and  Philip S. Yu  University of Illinois ChicagoUSA
<EMAIL_ADDRESS>
(2018; 20 February 2007; 12 March 2009; 5 June 2009)

###### Abstract.

Traditional recommender systems have predominantly relied on identity
representations (IDs) to characterize users and items. In contrast, the
emergence of pre-trained language model (PLM) encoders has significantly
enriched the modeling of contextual item descriptions. While PLMs excel in
addressing few-shot, zero-shot, and unified modeling scenarios, they often
overlook the critical collaborative filtering signal. This omission gives rise
to two pivotal challenges: (1) Collaborative Contextualization, aiming for the
seamless integration of collaborative signals with contextual representations.
(2) The necessity to bridge the representation gap between ID-based and
contextual representations while preserving their contextual semantics.

In this paper, we introduce CollabContext, a novel model that skillfully
merges collaborative filtering signals with contextual representations,
aligning these representations within the contextual space while retaining
essential contextual semantics. Experimental results across three real-world
datasets showcase substantial improvements. Through its capability in
collaborative contextualization, CollabContext demonstrates remarkable
enhancements in recommendation performance, particularly in cold-start
scenarios.

The code is available after the conference accepts the paper.

Recommender System, Collaborative Filtering, Pre-trained Language Model

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06 Figure 1. Comparison of Collaborative
Filtering Alignment and Collaborative Contextualization Alignment in
Reciprocal Peer Tutoring for Collaborative Contextualization.

##  1\. Introduction

In the contemporary digital landscape, recommendation systems (RecSys) have
emerged as indispensable tools, greatly enhancing user experiences across a
wide range of online platforms, including e-commerce, content streaming,
services and social media networks (Bonab et al., 2021; Anderson et al.,
2020). These systems are instrumental in guiding users towards content,
products, or services that align with their preferences, thereby contributing
significantly to user satisfaction on online platforms. Collaborative
filtering (Sedhain et al., 2015; He et al., 2020) is the most widely-used
technique in RecSys, learning users’ preferences based on historical user-item
interactions. However, it neglects the rich contextual information embedded in
the user/item textual data. Recent advancements have seen the incorporation of
pre-trained language models (PLMs) (Devlin et al., 2019; Reimers and Gurevych,
2019), fine-tuned on extensive text corpora, significantly altering the
RecSys’ landscape (Sun et al., [n. d.]; Hou et al., [n. d.], 2022), which
provides new opportunities to amplify collaborative filtering-based methods by
incorporating user/item contextual data.

An inevitable challenge arises when combining information from the two
distinct domains (collaborative filtering, contextual information) as
Collaborative Contextualization: how to effectively incorporate collaborative
signals with contextual representations. Traditional collaborative filtering-
based models (Sedhain et al., 2015; He et al., 2020; Herlocker et al., 2000)
treat users and items as equal roles, initializing them based on random values
and learning them simultaneously. However, this approach becomes less suitable
when further considering contextual information, as the roles of the user/item
become distinct. Items, often with sufficient contextual information (Title,
Description, Brand, etc), can act as the role to provide contextual semantic
embedding space with PLMs encoding. However, users often with limited
contextual information due to privacy concerns, and they provide the
collaborative signal by actively interacting with items. Training users and
items in the same way as previous practices can diminish the opportunities for
collaborative contextualization. The second significant challenge, often
referred to as the Semantic Disparity between Contextual and Collaborative
spaces (Hou et al., 2022; Ding et al., 2021), stems from a fundamental
discrepancy in how item representations are generated. Typically, these
representations are derived from PLMs, which excel in understanding contextual
data but are often trained on diverse tasks unrelated to RecSys. When user and
item representations are learned simultaneously, they are brought into the
same unified representation space. As shown in Fig. 1, in the Collaborative
Filtering Alignment scenario, represented by the first set of ovals, user and
item spaces are initially separate, denoted by distinct ovals. These spaces
are indicated by dotted lines to emphasize their separation. Both user and
item spaces have arrows pointing toward a middle oval, symbolizing the
alignment of user and item representations into a unified space, referred to
as the Unified Representation Space. However, this alignment can inadvertently
give rise to a phenomenon we have termed as ”item representation
contamination” in this paper. During this process, the rich semantic
information initially encoded in item text representations by PLMs may be
lost. This gap between the semantic richness of contextual representations and
their transformation into a unified space not only poses a challenge in
preserving semantic information but also weakens the item’s representative
ability to effectively alleviate the cold-start problem by incorporating
contextual representations.

To cope with the aforementioned challenges, we propose a novel approach named
CollabContext to fuse collaborative signal and contextual information. In
CollabContext, users and items explicitly play different roles during the
training, which is inspired by Reciprocal Peer Tutoring (Fantuzzo et al., 9
10). In education pedagogy, Reciprocal Peer Tutoring indicates peers take on
dual roles as both learners and tutors to enrich their understanding of each
other through mutual teaching and learning. In CollabContext, we apply this
concept by transforming users and items into collaborative peers. Within this
collaborative paradigm, items initially act as peer tutors, actively guiding
the recommendation process by sharing their contextual features, which
significantly influence the recommendations. In contrast, users take on the
role of peer learners, proactively adjusting and refining their
representations based on item contextual information. Subsequently, the roles
reverse, with users becoming peer tutors, sharing their preferences and
behaviors, while items become peer learners, adapting their representations
based on user interactions. This reciprocal peer tutoring approach offers a
dynamic and effective way to incorporate collaborative filtering signals and
contextual representations for more accurate recommendations.

CollabContext marks a significant departure from collaborative filtering-based
RecSys, where user and item representations are typically trained
simultaneously. Instead, we adopt a dynamic approach inspired by the concept
of peer tutoring in education. In CollabContext, users and items assume
distinct yet interchanging roles, forming the essence of Reciprocal Peer
Tutoring. The uniqueness of our approach lies in the initial configuration:
item representations are generated through PLMs, while user representations
begin with random initializations. To kickstart the collaborative process, we
first designate items as tutors, with users taking on the role of learners.
When items serve as tutors, their froze representations act as a wellspring of
guidance, akin to peer tutors imparting knowledge to their peers.
Simultaneously, users, in their role as learners, actively engage in the
learning experience. They adapt their representations by incorporating
interactions and item representations, gradually trained to align with item’s
contextual semantic space. Once users have acquired sufficient knowledge, the
roles reverse: users become tutors, and items take on the mantle of learners.
However, instead of directly learning item representations, we employ a Multi-
Layer Perceptron (MLP) as an adapter to transform item representations. This
adapter is essential and flexible since it refine the item representation from
collaborative filtering signal during the user tutoring phase, while freezing
the item contextual representation. In the cold-start scenario, retaining the
item contextual representation enables us to deactivate the MLP and leverage
informative item contextual representations to enhance cold-start performance.
The Collaborative Contextualization Alignment, illustrated in Fig. 1, follows
a distinct path from conventional collaborative filtering. While there are
separate user and item spaces, the Unified Representation Space resides within
the item’s contextual semantic space, emphasizing that the alignment process
occurs within the context of item representations. This innovative approach
not only elevates recommendation quality but also enhances the system’s
ability to effectively handle cold-start scenarios.

To sum up, our contributions can be outlined as follows:

  1. (1)

Conceptually, we identify the problem, uncover the challenges and propose a
feasible solution for collaborative contextualization to bridge the gap
between collaborative filtering and the pre-trained language model.

  2. (2)

Methodologically, we design CollabContext that effectively incorporates both
collaborative filtering signal and contextualized information. It encodes user
embedding into the item’s contextualized embedding and keeps the rich semantic
information from the pre-trained language model.

  3. (3)

Experimentally, we conduct extensive experiments on three real-world datasets
under both warm and cold settings to validate the effectiveness of
CollabContext.

##  2\. Collaborative Contextualization

In this section, we introduce the new problem termed ”Collaborative
Contextualization”. Collaborative contextualization seeks to fuse contextual
information with collaborative filtering to provide more accurate personalized
recommendations. It is an inevitable problem that occurs between pre-trained
language models and collaborative filtering-based recommendations.

For recommendation task, we have a set of users
$\mathcal{U}=\\{u_{1},u_{2},...,\allowbreak u_{\left|\mathcal{U}\right|}\\}$,
a set of items
$\mathcal{I}=\\{i_{1},i_{2},...,i_{\left|\mathcal{I}\right|}\\}$ and a
historical interaction matrix R of size
$\left|\mathcal{U}\right|\times\left|\mathcal{I}\right|$. By treating
$\mathbf{R}$ as the adjacent matrix, we can also view the historical
interaction as a user-item bipartite graph
$\mathcal{G}(\mathcal{U,I,E})=\\{(u,i)|u\in\mathcal{U},i\in\mathcal{I},(u,i)\in\mathcal{E}\\}$,
where $\mathcal{E}$ is the edge set. There is an edge $(u,i)\in\mathcal{E}$ if
$\mathbf{R}_{u,i}=1$ with implicit feedback. In collaborative
contextualization, besides the collaborative filtering signal
$\mathcal{G}(\mathcal{U,I,E})$, we also have a contextual embedding for each
user/item with rich semantic information represented by $\mathbf{x}_{u}$ and
$\mathbf{x}_{i}$, respectively. Contextual embedding is encoded from context
information with corresponding pre-trained models as illustrated in Section
3.1. Encoded from pre-trained models, contextual embedding contains rich
semantic information, and collaborative contextualization seeks to bridge the
contextual embedding with the collaborative filtering signal to provide more
accurate recommendation. To be noted that the semantic richness for users and
items is different. For users, we tend to have much less context information
due to the privacy issue.

Figure 2. CollabContext comprises three key phases: the Item Tutoring Phase,
User Tutoring Phase, and Inference Phase. During the Item Tutoring Phase, the
model aligns user representations with the item contextual representation
space. In contrast, the User Tutoring Phase focuses on guiding item
representations to effectively incorporate collaborative signals while
preserving their contextual characteristics. Finally, in the Inference Phase,
the model leverages the acquired knowledge to provide personalized
recommendations by utilizing the learned user embeddings and transformed item
embeddings.

##  3\. Proposed Method

In this section, we introduce our innovative recommendation model.
CollabContext, is designed to address the challenge of enhancing RecSys by
effectively incorporating contextual information and Collaborative Filtering
(CF) signals. By leveraging historical user-item interactions, CollabContext
learns comprehensive contextual CF-incorporated representations for both users
and items. These representations not only successfully merge contextual and CF
signals but also yield substantial improvements in recommendation performance,
benefiting both general recommendation scenarios and challenging item cold-
start scenarios. In the subsequent subsections, we detail the architecture,
training phase, and inference phase of CollabContext, providing a
comprehensive overview of our novel approach for recommendation.

###  3.1. Contextual Item Representation

To leverage the robust language modeling capabilities offered by pre-trained
language models (PLMs), our approach allows for using any PLM as the encoder
to capture contextual item embeddings. For a given item $i$ with associated
contextual features, including the item title, category, and brand, we
concatenate these features into a single sentence
$S_{i}=[w_{1},w_{2},...,w_{c}]$, where $w$ is the text token and $c$ is the
total token number. $S_{i}$ is then used as input to the PLM, which results in
the following contextual representation for item $i$:

(1) $\textbf{x}_{i}=PLM(S_{i}),$

where $\textbf{x}_{i}\in\mathbb{R}^{d_{W}}$ is $i$’s contextual embedding, and
$d_{W}$ denotes the PLM’s output embedding size. We freeze $\textbf{x}_{i}$ as
the $0$-th layer hidden representation $\textbf{h}_{i}^{(0)}$ for graph
convolution in Section 3.2. This representation captures the rich semantic
information from the item’s contextual attributes, laying the foundation for
the fusion of contextual and collaborative filtering signals within
CollabContext.

###  3.2. Graph Aggregator

CollabContext is built upon graph aggregation to spread the rich semantic
contextual embedding obtained from Section 3.1. With the aggregation over
$\mathcal{G}(\mathcal{U,I,E})$, CollabContext updates user/item embedding
based on the collaborative filtering signal. Let’s denote the embeddings of
users as $\textbf{h}_{u}$ and the embeddings of items as $\textbf{h}_{i}$,
which is obtained by the graph aggregator:

(2)
$\textbf{h}_{u},\textbf{h}_{i}=\text{Aggregator}(\mathcal{G}(\mathcal{U,I,E}),\textbf{h}_{u}^{(0)},\textbf{h}_{i}^{(0)}),$

where $\textbf{h}_{u}^{(0)}$ and $\textbf{h}_{i}^{(0)}$ represent the
user/item initial embedding. $\textbf{h}_{i}^{(0)}$ is encoded from Section
3.1 and $\textbf{h}_{u}^{(0)}$ is randomly initialized embedding due to the
lack of sufficient context to encode contextual embedding. The Aggregator
performs aggregation on $\mathcal{G}(\mathcal{U,I,E})$ for $K$ layers to
smooth the embedding. For each layer’s aggregation, the computation is defined
as:

(3) $\displaystyle\textbf{h}_{u}^{(k+1)}$
$\displaystyle=\textbf{h}_{u}^{(k)}+\text{AGG}\left(\textbf{h}_{i}^{(k)},\forall
i\in\mathcal{N}(u)\right),$ (4) $\displaystyle\textbf{h}_{i}^{(k+1)}$
$\displaystyle=\textbf{h}_{i}^{(k)}+\text{AGG}\left(\textbf{h}_{u}^{(k)},\forall
u\in\mathcal{N}(i)\right),$

where $\textbf{h}_{u}^{(k)}$ represents the embedding of user $u$ at layer
$k$, and $\textbf{h}_{i}^{(k)}$ represents the embedding of item $i$ at layer
$k$. The function AGG denotes the aggregation function, which combines the
embeddings of neighboring nodes. To maintain generality, we use the most
widely used LGCN (He et al., 2020) as the aggregation function:

(5) $\text{AGG}(\textbf{h}_{i},\forall
i\in\mathcal{N}(u))=\sum_{i\in\mathcal{N}(u)}\frac{1}{\sqrt{|\mathcal{N}(u)|}\sqrt{|\mathcal{N}(i)|}}\textbf{h}_{i},$

where $\mathcal{N}(u)$ and $\mathcal{N}(i)$ represent the set of neighboring
nodes of $u$ and $i$. $\textbf{h}_{i}$ is the embedding of a item node $i$.
User aggregation is computed in the same way. It’s worth noting that the
aggregation function can be replaced with any graph aggregation function. In
the subsequent section, we delve into the training phase of CollabContext,
where we elucidate the process of learning comprehensive contextual CF-
incorporated representations for users and items, a pivotal step in our
innovative recommendation model.

###  3.3. Training Phase

In traditional bipartite graph learning for RecSys, a common practice involves
initializing user and item ID-based embeddings randomly and updating their
representations symmetrically. This symmetric learning paradigm entails
aggregating the representations of neighbors for each user and item.

However, symmetric learning, as traditionally employed, is suboptimal with
contextual information. There are two primary reasons for this suboptimality.
First, the initialization methods for users and items differ significantly.
Users’ representations are initialized as ID-based random representations,
while items’ representations are derived from Text-based representations
generated by pre-trained language models (PLMs). This discrepancy results in
two key differences: (1) Representation Space Mismatch: The ID-based random
initialization and Text-based PLM-generated representations are situated in
different representation spaces, making direct symmetrical aggregation
problematic. (2) Information Discrepancy: The ID-based random initialization
contains relatively meaningless information, while the Text-based PLM-
generated representations incorporate influential contextual information.
Combining these disparate representations in a symmetric manner can lead to
issues. One critical concern is the phenomenon we term ”item representation
contamination.” In traditional symmetric learning, during the aggregation
process, user ID-based random initialized representations are aggregated to
form the item representation, which is then integrated into the unified
representation space (depicted in Fig. 1). This aggregation includes
information from random initialization, which may not contribute to accurate
item representations. Consequently, the item representations can become
diluted and noisy, deviating from their intended informative space.

To address these challenges, we introduce CollabContext, a collaborative
contextualization model inspired by the educational pedagogy of Reciprocal
Peer Tutoring. In the spirit of this pedagogical concept, CollabContext
establishes a dynamic interchange of roles between users and items, akin to
the essence of Reciprocal Peer Tutoring. Initially, items assume the role of
tutors, guiding the learning process, while users adopt the role of learners,
actively engaging and adapting their representations by incorporating
interactions and item information to gain a deeper understanding of the
system. Subsequently, the roles reverse, with users becoming tutors and items
taking on the role of learners, facilitating a comprehensive knowledge
exchange. This approach effectively addresses the limitations of symmetric
learning and preserves the integrity of contextual information within item
representations.

In the following subsections, we will provide detailed insights into the Item
Tutoring Phase and User Tutoring Phase.

####  3.3.1. Item Tutoring Phase

In this phase, items take on the role of tutors, guiding the learning
experience. On the other hand, users assume the role of learners, actively
engaging and adapting their representations based on the guidance provided by
the items.

Leveraging the rich semantic information contained in item contextual
representations, our objective is to align user embeddings ($\textbf{h}_{u}$)
to the item representations ($\textbf{h}_{i}$) which has been shown in the
Fig. 2. To achieve this alignment and strengthen the normalized element-wise
similarity between a user’s representation and those of their interacted
items, we employ an alignment loss inspired by the DirectAU (Wang et al.,
2022) method. The alignment loss between user $u$ and item $i$ is defined as:

(6)
$l^{\mathcal{U}}_{align}=\frac{1}{|\mathcal{E}|}\sum_{(u,i)\in\mathcal{E}}||\textbf{h}_{u}-\text{freeze}(\textbf{h}_{i})||^{2},$

where $\text{freeze}(\textbf{h}_{i})$ indicates the frozen item embedding. To
prevent over-concentration in the representation space, the uniformity loss is
also added as the regularization. In our context, we compute and apply the
user uniformity loss $l^{\mathcal{U}}_{uniform}$ to optimize the learning of
user representations efficiently:

(7)
$l^{\mathcal{U}}_{uniform}=log\frac{1}{|\mathcal{U}|^{2}}\sum_{u\in\mathcal{U}}\sum_{u*\in\mathcal{U}}e^{-2||\textbf{h}_{u}-\textbf{h}_{u*}||}.$

These two loss metrics work in synergy to maintain proximities between
positive instances while dispersing random instances across the hypersphere.
The final loss function in the user representation learning stage is a
combination of the alignment loss and the user uniformity loss:

(8)
$\mathcal{L_{\mathcal{U}}}=l^{\mathcal{U}}_{align}+l^{\mathcal{U}}_{uniform}.$

The phase of user representation learning concludes upon meeting the
convergence criteria, which are predicated on the prediction scores achieved
on the validation set. Specifically, we employ NDCG@10 as the benchmark
metric, with an early stopping parameter set at 30. At this juncture, we
presume that users have assimilated adequate knowledge from both contextual
and collaborative filtering signals, given the current state of item
contextual representations. Following this, the subsequent user tutoring phase
is dedicated to refining these item contextual representations.

####  3.3.2. User Tutoring Phase

In contrast to aligning user representations with item contextual
representations to integrate collaborative and contextual signals, the User
Tutoring Phase introduces subtle adjustments when refining item
representations. In this phase, the primary goal is to preserve item
embeddings within the contextual representation space, thus retaining
contextual information while incorporating collaborative signals. To achieve
this, user representations remain fixed during this phase, leveraging the
knowledge gained from well-learned users. Rather than directly modifying item
representations, we employ a Multi-Layer Perceptron (MLP) to transform them,
as illustrated in Fig. 2(b). There are two main reasons for this approach.
First, item contextual representations alone often fail to capture
collaborative filtering signal, and they tend to become densely clustered (Hou
et al., 2022; Ding et al., 2021; Hou et al., [n. d.]), which can hinder
recommendation performance. Second, this approach allows us to preserve
informative item contextual knowledge for use in the cold-start setting
directly. It’s worth noting that applying the MLP to map new items, which it
hasn’t encountered during the training stage, might have a negative impact, as
demonstrated in Table 1. This highlights the importance of maintaining item
contextual knowledge for effective cold-start recommendations.

To refine item representations, we initially apply MLP, resulting in
$\widetilde{\textbf{h}}^{(0)}_{i}=\text{MLP}(\textbf{h}^{(0)}_{i})$, where
$\widetilde{\textbf{h}}^{(0)}_{i}$ denotes the transformed item
representations. Subsequently, we compute the convoluted user and item
representations as follows:

(9)
$\textbf{h}_{u},\widetilde{\textbf{h}}_{i}=\text{Aggregator}(\mathcal{G}(\mathcal{U,I,E}),\textbf{h}_{u}^{(0)},\widetilde{\textbf{h}}_{i}^{(0)}).$

The alignment loss employed in item representation learning mirrors the one
used in user representation learning but utilizes the transformed item
representation $\widetilde{\textbf{h}}_{i}$. The alignment loss in the User
Tutoring Phase is formulated as follows:

(10)
$l^{\mathcal{I}}_{align}=\frac{1}{|\mathcal{E}|}\sum_{(u,i)\in\mathcal{E}}||\text{freeze}(\textbf{h}_{u})-\widetilde{\textbf{h}}_{i}||^{2},$

where $\text{freeze}(\textbf{h}_{u})$ is the frozen user embedding to force
the training on the item. In addition to the alignment loss, we introduce a
uniformity loss for items to prevent over-concentration and ensure a well-
distributed representation space. This uniformity loss encourages item
representations to maintain suitable distances from each other, thereby
promoting diversity in the recommendation process. The uniformity loss for
items is defined as follows:

(11)
$l^{\mathcal{I}}_{uniform}=\log\frac{1}{|\mathcal{I}|^{2}}\sum_{i\in\mathcal{I}}\sum_{i*\in\mathcal{I}}e^{-2||\widetilde{\textbf{h}}_{i}-\widetilde{\textbf{h}}_{i*}||}.$

It fosters the even distribution of item representations within the
hypersphere, thereby enhancing the model’s ability to capture nuanced
differences between items with similar contextual features.

The final loss function for item representation learning is a combination of
the alignment loss and the item uniformity loss:

(12)
$\mathcal{L}_{\mathcal{I}}=l^{\mathcal{I}}_{align}+l^{\mathcal{I}}_{uniform}.$

Through this approach, we ensure that item representations capture both
contextual information and collaborative filtering signals, leading to
improved recommendation quality while retaining the flexibility to address
cold-start problems.

###  3.4. Inference Phase

During the inference phase, CollabFusion leverages the learned user embeddings
and transformed item embeddings to make personalized recommendations. The
recommendation score $s(u,i)$ for a user-item pair $(u,i)$ is determined
through the dot product between the user embedding $\textbf{h}_{u}$ and the
transformed item embedding $\widetilde{\textbf{h}}_{i}$:

(13) $s(u,i)=\textbf{h}_{u}^{T}\cdot\widetilde{\textbf{h}}_{i}.$

In the cold-start setting, we using the dot product between the user embedding
$\textbf{h}_{u}$ and the item contextual embedding $\textbf{h}_{i}$ to
calculate the recommendation score.

##  4\. Experiments

This section empirically evaluates the proposed CollabContext on three real-
world datasets. The goal is to answer the four following research questions
(RQs).

  * •

RQ1: What is the performance of CollabContext ?

  * •

RQ2: Does CollabContext still achieve the best in the challenging cold-start
recommendation ?

  * •

RQ3: How do different parts affect CollabContext ?

  * •

RQ4: Can CollabContext really keep the rich semantic contextualized
information ?

The detailed experiment setting and more experiment results are further
illustrated in the Appendix.

###  4.1. Dataset

To rigorously evaluate the performance of our proposed methodology, we conduct
experiments in both warm and cold settings. Key statistics of the preprocessed
datasets are summarized in Table 4. Specifically, we utilize three publicly
available real-world datasets from the Amazon Review
Dataset111https://jmcauley.ucsd.edu/data/amazon/: Electronics, Office
Products, and Grocery and Gourmet Food. These datasets have been widely
employed in prior recommendation system studies (Hou et al., [n. d.], 2022;
Wang et al., 2021a). To ensure the validity of both warm and cold settings, we
apply meticulous preprocessing to these datasets. Initially, in alignment with
previous work (Hou et al., [n. d.]), we employ a 5-core filtering strategy,
eliminating users and items with insufficient interactions. Subsequently, we
randomly select 5% of items to serve as the cold-start items, excising all
corresponding interactions from the preprocessed dataset to create a cold-
start item dataset. This ensures that cold-start items are only encountered
during the testing phase. The remaining dataset is partitioned into training,
validation, and testing subsets using an 80%-10%-10% split. For the contextual
features of items, we aggregate information from fields such as title,
categories, and brand in the Amazon dataset, truncating any item text
exceeding 512 tokens.

###  4.2. RQ1: Recommendatin Evaluation

Table 1. Warm Setting Comparison Table. The best and second-best results are bold and underlined, respectively. ${{}_{\text{ID}}}$ indicates ID-based models, and ${{}_{\text{TEXT}}}$ denotes models that employ item contextual representation for item embedding initialization. | Electronic | Office Products | Grocery and Gourmet Food
---|---|---|---
Model | R@10 | R@50 | N@10 | N@50 | R@10 | R@50 | N@10 | N@50 | R@10 | R@50 | N@10 | N@50
$\text{NeuMF}_{\text{ID}}$ | 0.0513 | 0.0675 | 0.0298 | 0.0339 | 0.1599 | 0.1868 | 0.1051 | 0.1113 | 0.1390 | 0.1644 | 0.0892 | 0.0951
$\text{DirectAU}_{\text{ID}}$ | 0.0546 | 0.0694 | 0.0292 | 0.0329 | 0.1661 | 0.1965 | 0.1000 | 0.1069 | 0.1438 | 0.1707 | 0.0844 | 0.0907
$\text{LightGCN}_{\text{ID}}$ | 0.0530 | 0.0857 | 0.0331 | 0.0413 | 0.1782 | 0.2177 | 0.1217 | 0.1307 | 0.1526 | 0.1917 | 0.1024 | 0.1114
$\text{SimpleX}_{\text{ID}}$ | 0.0558 | 0.1060 | 0.0305 | 0.0429 | 0.1727 | 0.2172 | 0.1091 | 0.1192 | 0.1519 | 0.1989 | 0.0920 | 0.1028
$\text{NCL}_{\text{ID}}$ | 0.0564 | 0.1008 | 0.0354 | 0.0464 | 0.1841 | 0.2304 | 0.1262 | 0.1368 | 0.1585 | 0.2063 | 0.1074 | 0.1183
$\text{NeuMF}_{\text{TEXT}}$ | 0.0389 | 0.0540 | 0.0237 | 0.0275 | 0.1271 | 0.1579 | 0.0846 | 0.0916 | 0.1166 | 0.1417 | 0.0736 | 0.0795
$\text{DirectAU}_{\text{TEXT}}$ | 0.0551 | 0.0718 | 0.0294 | 0.0336 | 0.1671 | 0.1963 | 0.1004 | 0.1071 | 0.1434 | 0.1703 | 0.0843 | 0.0906
$\text{LightGCN}_{\text{TEXT}}$ | 0.0560 | 0.0909 | 0.0320 | 0.0407 | 0.1782 | 0.2177 | 0.1217 | 0.1307 | 0.1525 | 0.1965 | 0.1030 | 0.1131
$\text{SimpleX}_{\text{TEXT}}$ | 0.0514 | 0.0931 | 0.0334 | 0.0437 | 0.1759 | 0.2121 | 0.1202 | 0.1284 | 0.1513 | 0.1891 | 0.1015 | 0.1102
$\text{NCL}_{\text{TEXT}}$ | 0.0553 | 0.0929 | 0.0324 | 0.0417 | 0.1767 | 0.2213 | 0.1174 | 0.1275 | 0.1547 | 0.2074 | 0.1021 | 0.1142
Wide&Deep | 0.0138 | 0.0485 | 0.0074 | 0.0159 | 0.1014 | 0.1428 | 0.0609 | 0.0705 | 0.0923 | 0.1303 | 0.0554 | 0.0642
DCNV2 | 0.0373 | 0.0575 | 0.0216 | 0.0266 | 0.1292 | 0.1648 | 0.0829 | 0.0910 | 0.1160 | 0.1516 | 0.0717 | 0.0799
$\text{CollabContext}_{w/o\text{MLP}}$ | 0.0562 | 0.0989 | 0.0367 | 0.0497 | 0.1439 | 0.1953 | 0.0998 | 0.1119 | 0.1188 | 0.1608 | 0.0873 | 0.0959
$\text{CollabContext}_{w\text{MLP}}$ | 0.0639 | 0.1069 | 0.0417 | 0.0523 | 0.1878 | 0.2315 | 0.1312 | 0.1393 | 0.1626 | 0.2098 | 0.1173 | 0.1267

We evaluate the performance of our proposed method against various baseline
approaches across three distinct datasets, with results detailed in Table 1.
For the sake of clarity and comparative analysis, we categorize the baseline
methods into three distinct classes: ID-based (denoted as ${}_{\text{ID}}$),
Text-based (denoted as ${}_{\text{TEXT}}$), and content-based, which includes
models such as Wide&Deep and DCNV2. In ID-based models, the embedding for both
the user and item is learned exclusively from their respective IDs and
interactions. Text-based models initialize item embeddings using contextual
representations of the items. content-based models, on the other hand,
incorporate item contextual representation as features for enhanced contextual
understanding. Our model includes two variations, denoted as
$\text{CollabContext}{w/o\text{MLP}}$ without MLP during the inference phase
and $\text{CollabContext}{w/\text{MLP}}$ with MLP during inference phase.

In a comparative analysis with established baseline methods, our proposed
CollabContext model with MLP during the inference phase consistently
demonstrates superior performance over both ID-based and Text enhanced
algorithms across a wide range of evaluation metrics. It validates
CollabContext can effectively incorporate collaborative filtering signal and
contextual information to set the state-of-the-art performance. By utilizing
item contextual representation, different baseline methods behave differently.
DirectAU, LightGCN, and SimpleX only exhibit very marginal gains compared with
their ID-based counterparts. In contrast, NeuMF and NCL experience significant
performance degradation when initialized with item contextual representations.
It shows previous methods are unable to incorporate both collaborative
filtering signal and the encoded contextual embedding from PLM effectively. It
is worth further comparing $\text{CollabContext}_{w/o\text{MLP}}$ and
$\text{CollabContext}_{w\text{MLP}}$. $\text{CollabContext}_{w\text{MLP}}$
achieves remarkable improvements on all datasets. It shows the necessity of
including a trainable MLP adaptor on the frozen item contextual embedding.

###  4.3. RQ2: Cold-start Evaluation

Table 2. Cold Setting Comparison Table. Notations consistent with the warm
setting comparison.

| Electronic | Office Products | Grocery and Gourmet Food
---|---|---|---
Model | R@10 | R@50 | N@10 | N@50 | R@10 | R@50 | N@10 | N@50 | R@10 | R@50 | N@10 | N@50
$\text{NeuMF}_{\text{TEXT}}$ | 0.0046 | 0.0250 | 0.0021 | 0.0066 | 0.0101 | 0.0558 | 0.0047 | 0.0145 | 0.0074 | 0.0413 | 0.0034 | 0.0107
$\text{DirectAU}_{\text{TEXT}}$ | 0.0048 | 0.0277 | 0.0023 | 0.0073 | 0.0118 | 0.0511 | 0.0057 | 0.0143 | 0.0082 | 0.0429 | 0.0038 | 0.0113
$\text{LightGCN}_{\text{TEXT}}$ | 0.0104 | 0.0443 | 0.0049 | 0.0124 | 0.0309 | 0.0960 | 0.0173 | 0.0315 | 0.0119 | 0.0522 | 0.0053 | 0.0140
$\text{SimpleX}_{\text{TEXT}}$ | 0.0098 | 0.0419 | 0.0049 | 0.0120 | 0.0122 | 0.0606 | 0.0064 | 0.0168 | 0.0113 | 0.0583 | 0.0056 | 0.0158
$\text{NCL}_{\text{TEXT}}$ | 0.0188 | 0.0716 | 0.0092 | 0.0209 | 0.0338 | 0.1150 | 0.0143 | 0.0320 | 0.0222 | 0.0778 | 0.0128 | 0.0248
Wide&Deep | 0.0038 | 0.0204 | 0.0018 | 0.0055 | 0.0140 | 0.1118 | 0.0060 | 0.0267 | 0.0150 | 0.1224 | 0.0072 | 0.0297
DCNV2 | 0.0057 | 0.0282 | 0.0029 | 0.0078 | 0.0107 | 0.0510 | 0.0047 | 0.0134 | 0.0082 | 0.0356 | 0.0039 | 0.0099
DroupoutNet | 0.0573 | 0.1203 | 0.0354 | 0.0497 | 0.1313 | 0.1925 | 0.0844 | 0.0982 | 0.1470 | 0.2446 | 0.0989 | 0.1208
Heater | 0.0036 | 0.0293 | 0.0017 | 0.0073 | 0.0078 | 0.0412 | 0.0034 | 0.0105 | 0.0271 | 0.0500 | 0.0095 | 0.0146
$\text{CollabContext}_{w/o\text{MLP}}$ | 0.0617 | 0.1498 | 0.0379 | 0.0578 | 0.1471 | 0.2696 | 0.0948 | 0.1220 | 0.1774 | 0.3026 | 0.1226 | 0.1507
$\text{CollabContext}_{w\text{MLP}}$ | 0.0076 | 0.0357 | 0.0030 | 0.0093 | 0.0104 | 0.0474 | 0.0041 | 0.0121 | 0.0422 | 0.2542 | 0.0223 | 0.0667

Table 2 provides an overview of the results of cold-start item
recommendations. Key findings include: 1) Our model,
$\text{CollabContext}_{w/o\text{MLP}}$, consistently outperforms the best-
performing baseline models in cold-start item recommendations. In contrast,
DropoutNet and Heater, despite utilizing the same item contextual
representations as item features, and employing pre-trained Matrix
Factorization (MF) to initialize user and item representations, fall short in
comparison to our model. This underscores the efficacy of CollabContext in
leveraging item contextual knowledge and highlights the benefits of its
unified representational space, seamlessly integrated into the item contextual
representation space, as illustrated in Fig. 1. This integration serves a dual
purpose: it retains the semantic richness of item contextual descriptions
while effectively addressing the cold-start item challenge. Importantly,
CollabContext accomplishes this without the need for auxiliary modules or
additional steps. 2) collaborative-based models struggle to harness the
informative item contextual representation for addressing the cold-start item
problem. This challenge arises from the Semantic Disparity between Contextual
and Collaborative spaces. Treating users and items as equal entities during
training leads to an alignment of the unified representation space into a new
space that diverges from the original item contextual representation space.
Consequently, when a new item is introduced, its contextual representation
deviates from the unified representation space, causing the model to encounter
difficulties in making recommendations for the new item.

###  4.4. RQ3: In-depth Analysis

Figure 3. Ablation study of CollabContext on Electronic

####  4.4.1. Ablation Study

In this subsection, we present a comprehensive analysis of the impact of each
proposed technique and component on both the warm and cold setting
performances. To facilitate a thorough comparison, we prepare two variants of
the CollabContext model: (1) a variant without Reciprocal Peer Tutoring (RPT),
denoted as $w/o$ RPT, maintaining the training strategy consistent with
collaborative models; and (2) a variant omits the User Tutoring Phase,
applying only item contextual representation.

The results of this ablation study are illustrated in Fig. 3. Notably, the
absence of Reciprocal Peer Tutoring (RPT), as observed in $w/o$ RPT, leads to
a significant reduction in model performance. This underscores the critical
importance of preserving the item contextual representation space.
Additionally, the exclusion of the User Tutoring Phase (UTP) reveals
interesting insights. In the warm setting, omitting the UTP harms performance,
as item contextual representations alone may tend to crowd together.
Conversely, in the cold setting, item contextual representations still retain
valuable context features that assist in addressing the cold-start challenge.

####  4.4.2. Impact of Diverse Pretrained Language Models (PLMs)

Numerous robust Pretrained Language Models (PLMs) hold the potential to
enhance RecSys by providing valuable item contextual representations. To
identify the most effective PLMs for our specific objective, we conducted an
evaluation of item contextual representations generated by four additional
PLMs. These PLMs have achieved varying rankings on the Massive Text Embedding
Benchmark (MTEB) leaderboard
222https://huggingface.co/spaces/mteb/leaderboard, highlighting their diverse
capabilities and potential contributions to the enhancement of RecSys. The
five PLMs evaluated include instructor-xl(Su et al., 2023), all-
MiniLM-L6-v2(Reimers and Gurevych, 2019), all-mpnet-base-v2(Reimers and
Gurevych, 2019), and bge-base-en-v1.5(Xiao et al., 2023) and bert-base-
unchased(Devlin et al., 2019). The experimental results are presented in Table
3, demonstrating significant improvements in the cold-start setting than the
default model. This underscores the vital importance of aligning the user
representation space with the item contextual representation space. Notably,
instructor-xl emerges as the top-performing PLM overall, as it can generate
text embeddings simply by providing the task instruction, without requiring
fine-tuning. For our experiments, we used the instruction ”Represent the
Amazon title:” with instructor-xl to generate the text embeddings.

Table 3. Comparison Table of PLMs. Notations consistent with warm setting comparison. Electronic | Warm Setting | Cold Setting
---|---|---
PLMs | R@10 | N@10 | R@10 | N@10
instructor-xl | 0.0639 | 0.0417 | 0.0612 | 0.0370
all-MiniLM-L6-v2 | 0.0633 | 0.0414 | 0.0563 | 0.0343
all-mpnet-base-v2 | 0.0641 | 0.0419 | 0.0556 | 0.0341
bge-base-en-v1.5 | 0.0636 | 0.0414 | 0.0561 | 0.0339
bert-base-unchased | 0.0631 | 0.0411 | 0.0352 | 0.0194

####  4.4.3. Should We Include Additional Item Tutoring and User Tutoring
Phases?

To investigate the advantages of further training user and item
representations beyond the initial item tutoring and user tutoring phases, we
conducted experiments involving continuous learning on these representations.
Fig. 4 presents the model’s performance in both warm and cold settings across
three datasets. In the plot, ’Item Tut’ indicates that the current phase is
the item tutoring phase, while ’User Tut’ designates the user tutoring phase.
The numbers on the x-axis represent the current training stage.

As depicted in Fig. 4, CollabContext achieved its best performance after the
first user tutoring phase, denoted as ’User Tut:1,’ across all three datasets.
This suggests that user and item representations do not require additional
separate training stages. Continuing to train the model beyond this point
results in a performance decline, possibly due to the significant deviation of
user and item representations from the item contextual representation, leading
to a loss of semantic information.

Figure 4. Overall performance in each training phase

####  4.4.4. Parameter Sensitivity

Figure 5. Parameter analysis of MLP on Electronic

We experimented with different configurations of the number of layers for the
MLP. The sensitivity results, shown in Fig. 5, reveal that the layer number is
not sensitive for model performance, and the highest performance is achieved
when using two layers with a hidden dimension of 768.

###  4.5. RQ4: Case Study

To provide visual evidence of CollabContext’s effectiveness in aligning user
representations with item contextual representations while preserving the
integrity of item-learned representations, we present a case study in Fig 6.
In the left figure, which represents the representation space following
traditional collaborative filtering alignment, the item contextual
representation (blue) is shown surrounding the user (green) and item (purple)
mapped representations by MLP. This spatial arrangement suggests that the item
contextual representation is not effectively integrated into the same space as
the user and item representations. In contrast, CollabContext, as shown in the
right figure, successfully aligns the user and item learned representations
within the item contextual representation space, ensuring that informative
contextual information is retained. This alignment contributes to significant
improvements in both warm and cold settings, showcasing the model’s enhanced
performance.

(a) CF Representation Space

(b) CC Representation Space

Figure 6. Comparison of representation space after model alignment. The left
figure illustrates the representation space following Collaborative Filtering
Alignment, while the right figure depicts the representation space after
Collaborative Contextualization Alignment. In both figures, the blue nodes
symbolize item contextual representations, the purple nodes represent item
mapped representations by MLP, and the green nodes denote user learned
representations.

##  5\. Related work

###  5.1. Collaborative Filtering

Collaborative Filtering (CF) is a widely used technique in modern RecSys. CF
models represent users and items as embeddings learned from historical
interactions. Graph Neural Networks (GNNs) have gained popularity in RecSys,
modeling user-item interactions as bipartite graphs to capture high-order
connectivity (Kipf and Welling, 2017). SpectralCF introduced spectral
convolution for better performance, especially for cold-start items (Zheng et
al., 2018). PinSAGE developed a random walk strategy for large-scale graphs
like Pinterest (Ying et al., 2018). NGCF aggregates information from neighbors
and incorporates collaborative signals (Wang et al., 2019). LightGCN
simplified NGCF, achieving better performance and shorter training time (He et
al., 2020). However, these collaborative models treat users and items equally,
learning their representations simultaneously, which may not be ideal for
incorporating informative item contextual representations.

###  5.2. Cold-start Recommendation

To address the cold-start problem, we typically integrate content features
into collaborative filtering (CF)-based recommendation models. These features
serve as a link to capture the collaborative signal for cold-start items. For
instance, models like DropoutNET (Volkovs et al., 2017) and CC-CC (Shi et al.,
2019) enhance CF-based models by incorporating item content features. Other
approaches, such as Heater (Zhu et al., 2020), explicitly model the
correlation between content and collaborative embeddings. In our approach, we
take a unique approach by directly combining collaborative filtering signals
with content information through collaborative contextualization. This
innovative approach greatly improves recommendation performance by seamlessly
blending collaborative and content-based information.

###  5.3. Pre-trained Language Models

General text embeddings are of paramount importance, finding wide utility not
only in common applications such as web search and question
answering(Karpukhin et al., 2020) but also in their foundational role in
enhancing large language models (Lewis et al., 2020; Izacard et al., 2022).
Unlike task-specific methods, general text embeddings must be versatile and
applicable across various contexts. In recent years, significant strides have
been made in this field, resulting in notable works like BERT (Devlin et al.,
2019), Instructor (Su et al., 2023), sentence-T5 (Ni et al., 2021a), Sentence-
Transformer (Reimers and Gurevych, 2019), C-Pack (Xiao et al., 2023), OpenAI
text embedding (Neelakantan et al., 2022), and more. In the realm of RecSys,
existing research has demonstrated the power and effectiveness of Pre-trained
Language Models (PLMs) in enhancing RecSys (Hou et al., [n. d.]; Ding et al.,
2021; Hou et al., 2022; Li et al., 2023), particularly in warm, cold, and few-
shot settings. However, most of these studies have primarily focused on
sequential recommendation scenarios, while our work centers on collaborative
filtering-based recommendations.

##  6\. Conclusion

In this paper, we study the collaborative contextualization problem to bridge
the gap between collaborative filtering and the pre-trained language model.
Taking advantage of the pre-trained language model, we first obtain the item
contextualized embedding with rich semantic information. Then, we propose
CollabContext to encode user embedding into the item’s contextual embedding
space based on the collaborative signal. CollabContext treats users and items
in different roles to better utilize both the collaborative filtering signal
and the rich contextual information on items. Experiments on three real-world
datasets under both warm and cold settings show our proposed CollabContext
surpasses current state-of-the-art methods. Our case study on learned
embedding space highlights that CollabContext can keep the semantic
information on the contextual embedding space from pre-trained language model.

## References

  * (1)
  * Anderson et al. (2020) Ashton Anderson, Lucas Maystre, Ian Anderson, Rishabh Mehrotra, and Mounia Lalmas. 2020.  Algorithmic effects on the diversity of consumption on spotify. In _Proceedings of The Web Conference 2020_. 2155–2165.
  * Bonab et al. (2021) Hamed Bonab, Mohammad Aliannejadi, Ali Vardasbi, Evangelos Kanoulas, and James Allan. 2021.  Cross-Market Product Recommendation. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. ACM.
  * Cao et al. (2023) Yuwei Cao, Liangwei Yang, Chen Wang, Zhiwei Liu, Hao Peng, Chenyu You, and Philip S. Yu. 2023.  Multi-task Item-attribute Graph Pre-training for Strict Cold-start Item Recommendation. In _Proceedings of the 17th ACM Conference on Recommender Systems, RecSys 2023, Singapore, Singapore, September 18-22, 2023_ , Jie Zhang, Li Chen, Shlomo Berkovsky, Min Zhang, Tommaso Di Noia, Justin Basilico, Luiz Pizzato, and Yang Song (Eds.). ACM, 322–333.  https://doi.org/10.1145/3604915.3608806
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016.  Wide & Deep Learning for Recommender Systems. In _Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, DLRS@RecSys 2016, Boston, MA, USA, September 15, 2016_ , Alexandros Karatzoglou, Balázs Hidasi, Domonkos Tikk, Oren Sar Shalom, Haggai Roitman, Bracha Shapira, and Lior Rokach (Eds.). ACM, 7–10.  https://doi.org/10.1145/2988450.2988454
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4171–4186.  https://doi.org/10.18653/v1/n19-1423
  * Ding et al. (2021) Hao Ding, Yifei Ma, Anoop Deoras, Yuyang Wang, and Hao Wang. 2021.  Zero-Shot Recommender Systems.  _CoRR_ abs/2105.08318 (2021).  arXiv:2105.08318 https://arxiv.org/abs/2105.08318
  * Fantuzzo et al. (9 10) John W Fantuzzo, Linda A Dimeff, and Shari L Fox. 1989-10.  Reciprocal Peer Tutoring: A Multimodal Assessment of Effectiveness with College Students.  _Teaching of psychology._ 16, 3 (1989-10).
  * He et al. (2020) Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020.  Lightgcn: Simplifying and powering graph convolution network for recommendation. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_. 639–648.
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017.  Neural Collaborative Filtering. In _Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017_ , Rick Barrett, Rick Cummings, Eugene Agichtein, and Evgeniy Gabrilovich (Eds.). ACM, 173–182.  https://doi.org/10.1145/3038912.3052569
  * Herlocker et al. (2000) Jonathan L. Herlocker, Joseph A. Konstan, and John Riedl. 2000.  Explaining collaborative filtering recommendations. In _CSCW 2000, Proceeding on the ACM 2000 Conference on Computer Supported Cooperative Work, Philadelphia, PA, USA, December 2-6, 2000_ , Wendy A. Kellogg and Steve Whittaker (Eds.). ACM, 241–250.  https://doi.org/10.1145/358916.358995
  * Hou et al. (2022) Yupeng Hou, Zhankui He, Julian J. McAuley, and Wayne Xin Zhao. 2022.  Learning Vector-Quantized Item Representation for Transferable Sequential Recommenders.  _CoRR_ abs/2210.12316 (2022).  https://doi.org/10.48550/arXiv.2210.12316 arXiv:2210.12316
  * Hou et al. ([n. d.]) Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. [n. d.].  Towards Universal Sequence Representation Learning for Recommender Systems. In _KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_.
  * Izacard et al. (2022) Gautier Izacard, Patrick S. H. Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022.  Few-shot Learning with Retrieval Augmented Language Models.  _CoRR_ abs/2208.03299 (2022).  https://doi.org/10.48550/arXiv.2208.03299 arXiv:2208.03299
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick S. H. Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense Passage Retrieval for Open-Domain Question Answering. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 6769–6781.  https://doi.org/10.18653/v1/2020.emnlp-main.550
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks. In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.  https://openreview.net/forum?id=SJU4ayYgl
  * Lewis et al. (2020) Patrick S. H. Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020.  Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (Eds.).  https://proceedings.neurips.cc/paper/2020/hash/6b493230205f780e1bc26945df7481e5-Abstract.html
  * Li et al. (2023) Jiacheng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang, and Julian J. McAuley. 2023.  Text Is All You Need: Learning Language Representations for Sequential Recommendation. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2023, Long Beach, CA, USA, August 6-10, 2023_ , Ambuj K. Singh, Yizhou Sun, Leman Akoglu, Dimitrios Gunopulos, Xifeng Yan, Ravi Kumar, Fatma Ozcan, and Jieping Ye (Eds.). ACM, 1258–1267.  https://doi.org/10.1145/3580305.3599519
  * Lin et al. (2022) Zihan Lin, Changxin Tian, Yupeng Hou, and Wayne Xin Zhao. 2022.  Improving Graph Collaborative Filtering with Neighborhood-enriched Contrastive Learning. In _WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022_ , Frédérique Laforest, Raphaël Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel Médini (Eds.). ACM, 2320–2329.  https://doi.org/10.1145/3485447.3512104
  * Mao et al. (2021) Kelong Mao, Jieming Zhu, Jinpeng Wang, Quanyu Dai, Zhenhua Dong, Xi Xiao, and Xiuqiang He. 2021.  SimpleX: A Simple and Strong Baseline for Collaborative Filtering. In _CIKM ’21: The 30th ACM International Conference on Information and Knowledge Management, Virtual Event, Queensland, Australia, November 1 - 5, 2021_ , Gianluca Demartini, Guido Zuccon, J. Shane Culpepper, Zi Huang, and Hanghang Tong (Eds.). ACM, 1243–1252.  https://doi.org/10.1145/3459637.3482297
  * Neelakantan et al. (2022) Arvind Neelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, et al. 2022\.  Text and code embeddings by contrastive pre-training.  _arXiv preprint arXiv:2201.10005_ (2022).
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ (2019).
  * Sedhain et al. (2015) Suvash Sedhain, Aditya Krishna Menon, Scott Sanner, and Lexing Xie. 2015.  AutoRec: Autoencoders Meet Collaborative Filtering. In _Proceedings of the 24th International Conference on World Wide Web Companion, WWW 2015, Florence, Italy, May 18-22, 2015 - Companion Volume_ , Aldo Gangemi, Stefano Leonardi, and Alessandro Panconesi (Eds.). ACM, 111–112.  https://doi.org/10.1145/2740908.2742726
  * Shi et al. (2019) Shaoyun Shi, Min Zhang, Xinxing Yu, Yongfeng Zhang, Bin Hao, Yiqun Liu, and Shaoping Ma. 2019.  Adaptive Feature Sampling for Recommendation with Missing Content Feature Values. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019_ , Wenwu Zhu, Dacheng Tao, Xueqi Cheng, Peng Cui, Elke A. Rundensteiner, David Carmel, Qi He, and Jeffrey Xu Yu (Eds.). ACM, 1451–1460.  https://doi.org/10.1145/3357384.3357942
  * Su et al. (2023) Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen-tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023.  One Embedder, Any Task: Instruction-Finetuned Text Embeddings. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_ , Anna Rogers, Jordan L. Boyd-Graber, and Naoaki Okazaki (Eds.). Association for Computational Linguistics, 1102–1121.  https://doi.org/10.18653/v1/2023.findings-acl.71
  * Sun et al. ([n. d.]) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. [n. d.].  BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019_.
  * Volkovs et al. (2017) Maksims Volkovs, Guang Wei Yu, and Tomi Poutanen. 2017.  DropoutNet: Addressing Cold Start in Recommender Systems. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 4957–4966.  https://proceedings.neurips.cc/paper/2017/hash/dbd22ba3bd0df8f385bdac3e9f8be207-Abstract.html
  * Wang et al. (2021a) Chen Wang, Yueqing Liang, Zhiwei Liu, Tao Zhang, and Philip S. Yu. 2021a.  Pre-training Graph Neural Network for Cross Domain Recommendation. In _Third IEEE International Conference on Cognitive Machine Intelligence, CogMI 2021, Atlanta, GA, USA, December 13-15, 2021_. IEEE, 140–145.  https://doi.org/10.1109/CogMI52975.2021.00026
  * Wang et al. (2022) Chenyang Wang, Yuanqing Yu, Weizhi Ma, Min Zhang, Chong Chen, Yiqun Liu, and Shaoping Ma. 2022.  Towards Representation Alignment and Uniformity in Collaborative Filtering. In _KDD ’22: The 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC, USA, August 14 - 18, 2022_ , Aidong Zhang and Huzefa Rangwala (Eds.). ACM, 1816–1825.  https://doi.org/10.1145/3534678.3539253
  * Wang et al. (2021b) Ruoxi Wang, Rakesh Shivanna, Derek Zhiyuan Cheng, Sagar Jain, Dong Lin, Lichan Hong, and Ed H. Chi. 2021b.  DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems. In _WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021_ , Jure Leskovec, Marko Grobelnik, Marc Najork, Jie Tang, and Leila Zia (Eds.). ACM / IW3C2, 1785–1797.  https://doi.org/10.1145/3442381.3450078
  * Wang et al. (2019) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019.  Neural Graph Collaborative Filtering. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019_ , Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer (Eds.). ACM, 165–174.  https://doi.org/10.1145/3331184.3331267
  * Wei et al. (2021) Yinwei Wei, Xiang Wang, Qi Li, Liqiang Nie, Yan Li, Xuanping Li, and Tat-Seng Chua. 2021.  Contrastive Learning for Cold-Start Recommendation. In _MM ’21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021_ , Heng Tao Shen, Yueting Zhuang, John R. Smith, Yang Yang, Pablo César, Florian Metze, and Balakrishnan Prabhakaran (Eds.). ACM, 5382–5390.  https://doi.org/10.1145/3474085.3475665
  * Xiao et al. (2023) Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023.  C-Pack: Packaged Resources To Advance General Chinese Embedding.  arXiv:2309.07597 [cs.CL]
  * Xu et al. (2022) Lanling Xu, Zhen Tian, Gaowei Zhang, Lei Wang, Junjie Zhang, Bowen Zheng, Yifan Li, Yupeng Hou, Xingyu Pan, Yushuo Chen, Wayne Xin Zhao, Xu Chen, and Ji-Rong Wen. 2022.  Recent Advances in RecBole: Extensions with more Practical Considerations.
  * Ying et al. (2018) Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. 2018.  Graph Convolutional Neural Networks for Web-Scale Recommender Systems. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018_, Yike Guo and Faisal Farooq (Eds.). ACM, 974–983.  https://doi.org/10.1145/3219819.3219890
  * Zheng et al. (2018) Lei Zheng, Chun-Ta Lu, Fei Jiang, Jiawei Zhang, and Philip S. Yu. 2018.  Spectral collaborative filtering. In _Proceedings of the 12th ACM Conference on Recommender Systems, RecSys 2018, Vancouver, BC, Canada, October 2-7, 2018_ , Sole Pera, Michael D. Ekstrand, Xavier Amatriain, and John O’Donovan (Eds.). ACM, 311–319.  https://doi.org/10.1145/3240323.3240343
  * Zhu et al. (2020) Ziwei Zhu, Shahin Sefati, Parsa Saadatpanah, and James Caverlee. 2020.  Recommendation for New Users and New Items via Randomized Training and Mixture-of-Experts Transformation. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_ , Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (Eds.). ACM, 1121–1130.  https://doi.org/10.1145/3397271.3401178

##  Appendix A appendix

###  A.1. Experimental Setup

####  A.1.1. Datasets

Table 4. The Statistics of Preprocessed Datasets: ’Avg.U’ represents the average number of interactions per user, ’Avg.I’ signifies the average number of interactions per item, and ’Cold-Items’ indicates the count of newly introduced items. | Electronic | Office Products | Gourmet Food
---|---|---|---
#Users | 81,512 | 51,493 | 66,268
#Items | 32,424 | 16,920 | 24,636
#Inters | 623,896 | 212,795 | 307,617
#Avg.U | 7.653 | 4.133 | 4.642
#Avg.I | 18.232 | 11.950 | 11.857
#Cold-Items | 1,797 | 888 | 1,310

####  A.1.2. Compared Methods

We compare the proposed approach with the following baseline methods:

  * •

NeuMF (He et al., 2017) is a neural network-enhanced matrix factorization
model that replaces the conventional dot product with a multi-layer perceptron
(MLP) to capture more nuanced user-item interactions.

  * •

DirectAU (Wang et al., 2022) introduces an innovative loss function that
evaluates representation quality in collaborative filtering (CF) based on
alignment and uniformity within the hypersphere. In our implementation, we
employ the alignment and uniformity loss, updating only the student role.

  * •

LightGCN (He et al., 2020) represents a state-of-the-art recommendation
algorithm grounded on Graph Convolutional Networks (GCN) (Kipf and Welling,
2017). It enhances performance by omitting feature transformations and
nonlinear activations.

  * •

SimpleX (Mao et al., 2021) proposes an easy-to-understand model with a unique
loss function that incorporates a larger set of negative samples and employs a
threshold to eliminate less informative ones. It also utilizes relative
weights to balance the contributions of positive-sample and negative-sample
losses.

  * •

NCL (Lin et al., 2022) offers a neighborhood-enriched contrastive learning
framework tailored for graph collaborative filtering. It explicitly captures
both structural and semantic neighbors as objects for contrastive learning.

  * •

Wide&Deep (Cheng et al., 2016) is a context-aware recommendation model that
trains both wide linear models and deep neural networks concurrently, aiming
to synergize the benefits of both memorization and generalization in RecSys.

  * •

DCNV2 (Wang et al., 2021b) is another context-aware recommendation model that
enhances the expressive power of Deep & Cross Networks (DCN) by extending the
original weight vector into a matrix. It also incorporates Mixture-of-Experts
(MoE) and low-rank techniques to reduce computational overhead.

  * •

DroupoutNet (Volkovs et al., 2017) employs a dropout operation during
training, randomly discarding portions of the collaborative embeddings. This
technique effectively addresses the cold-start problem by enhancing the
model’s robustness.

  * •

Heater (Zhu et al., 2020) utilizes the sum squared error (SSE) loss to model
collaborative embeddings based on content information. Additionally, it
employs a randomized training approach to further enhance the model’s
effectiveness.

####  A.1.3. Evaluation Settings

We evaluate our model’s recommendation performance using commonly employed
metrics in the field of Recommender Systems (RecSys): Recall@K and NDCG@K. By
default, we set the values of K to 10 and 50. The reported results are based
on the average scores across all users in the test set. These metrics consider
the rankings of items that users have not interacted with yet. In line with
established practices (He et al., 2020, 2017), we utilize a full-ranking
technique, which involves ranking all non-interacted items for each user. To
assess the model’s performance in a Cold-start setting, we follow the
procedures outlined in previous studies (Cao et al., 2023; Volkovs et al.,
2017; Zhu et al., 2020; Wei et al., 2021). In the Cold-start scenario, we
identify cold-start items by removing all training interactions for randomly
selected subsets of items.

###  A.2. Implementation Details

We implement CollabContext and other baseline models using the open-source
recommendation library, RecBole333https://recbole.io/docs/index.html (Xu et
al., 2022). For the sake of a fair comparison, we employ the Adam optimizer
across all methods and conduct meticulous hyperparameter tuning. The batch
size is configured at 1,024, and we implement early stopping with a patience
setting of 30 epochs to mitigate overfitting, using NDCG@10 as our evaluation
metric. Item text embeddings are generated utilizing a pretrained Instructor-
xl model444https://huggingface.co/hkunlp/instructor-xl (Su et al., 2023). The
dimensions for both user and item embeddings are fixed at 768. The instruction
we use is ”Represent the Amazon title:”.

For residual hyperparameters, we employ a grid search strategy to identify
optimal settings. Specifically, the learning rate is explored within the set
{0.0001, 0.001, 0.01}, and the weight decay coefficient is tuned among
{$1e^{-4}$, $1e^{-5}$, $1e^{-6}$}. For graph-based models, the number of
layers is evaluated over {1, 2, 3}. For content-based models, item contextual
representations serve as item features. Within the MLP in CollabContext, we
explore configurations with layer counts in {1, 2, 3}, hidden dimensions in
{384, 768, 1536}, and dropout rates in {0.2, 0.5}.

###  A.3. In-depth Discussions on CollabContext

####  A.3.1. Why Did We Choose Alignment and Uniformity Loss Over CE or BPR?

Our choice of using uniformity and alignment losses over cross-entropy (CE) or
Bayesian Personalized Ranking (BPR) loss is driven by the specific objective
of our model: to align the user representation space with the item contextual
representation space while preserving semantic item representations.

Alignment loss directly aligns with our motivation by encouraging the user and
item representations to be close in the same representation space. However, CE
loss, while sharing a similar objective of alignment, may not guarantee that
the aligned representations are adequately spaced apart. This distinction
makes alignment loss a more suitable choice for our task. Moreover, we
incorporate uniformity loss as it serves to prevent representations from
becoming excessively close, effectively maintaining a balanced distribution.
This is important to avoid over-concentration and promote diversity in the
representation space, which aligns with our goal of preserving the richness of
contextual semantics.

On the other hand, Bayesian Personalized Ranking (BPR) loss is designed to
optimize item rankings for individual users, enhancing personalized
recommendations. While it is valuable in its own right, it does not align with
our specific motivation of preserving semantic item representations through
the alignment of user and item spaces. Therefore, BPR loss is not aligned with
the objectives of our model.

###  A.4. Time Complexity

While CollabContext involves training in two phases, the increase in training
time is not substantial for two key reasons:

  * •

In terms of time complexity, the most computationally intensive part of
CollabContext is the uniformity loss. This loss component calculates
similarity between each pair of items and users, resulting in a time
complexity of $\mathcal{O}(|\mathcal{U}|^{2}+|\mathcal{I}|^{2}\times d)$.
Fortunately, during each stage, we only need to compute the representation
pairs that have been trained. For instance, during the Item Tutoring phase, we
only calculate user-user pairs, and during the User Tutoring phase, we only
compute item-item pairs. This reduces the time complexity to
$\mathcal{O}(max(|\mathcal{U}|^{2},|\mathcal{I}|^{2})\times d)$.

  * •

The User Tutoring phase converges quickly. Once user representations have been
aligned into the item contextual representation space, the item representation
learning from user presentations becomes more efficient. For example, in our
experiments, we trained for 25 epochs in the Electronic dataset, 57 epochs in
the Office Products dataset, and 75 epochs in the Grocery and Gourmet Food
dataset during the User Tutoring phase.

These factors collectively ensure that CollabContext’s training time remains
manageable even with the inclusion of additional phases.

###  A.5. Limitation and Future Works

This study is merely a preliminary exploration of collaborative
contextualization and carries several limitations. Firstly, we focused on
recommendation scenarios involving only contextual item information, leaving
room for future research to consider other context sources such as images and
videos. Secondly, we have yet to fully explore scenarios involving user
contextual information paired with item IDs, as well as those involving user
contextual information matched with item contextual information, as each of
these situations may entail distinct user and item roles. Lastly, the
applicability of PLM-generated contextual embeddings remains an open question,
especially when applied to different types of products such as books, music,
movies, or restaurants. Further research is needed to uncover the utility of
PLMs in these domains.

In our future work, one intriguing avenue of research revolves around
preserving the contextual representations of multiple domains while
effectively integrating collaborative filtering signals. To elaborate,
different domains possess distinct contextual features, and pre-trained
language models (PLMs) may encode these features into separate representation
spaces. This diversity can pose challenges when attempting to incorporate them
into a unified model. For instance, consider the domains of electronics and
fashion. PLMs may encode the contextual features of electronic products
differently from those of fashion items. Integrating recommendations across
these domains necessitates bridging these distinct representation spaces,
which presents a complex and interesting challenge.

Another compelling direction for future research centers on determining when a
user, in their role as a learner, has acquired sufficient knowledge to
transition to the role of tutor and vice versa for items. Currently, we rely
on a straightforward validity score as an indicator, which may lead to
premature or overly conservative transitions. Developing a novel indicator
that more accurately gauges the readiness for role-switching could lead to
substantial improvements in the model’s performance. This indicator should
strike a balance between stability and adaptability to optimize the learning
process.