# Parallel Knowledge Enhancement based Framework for Multi-behavior
Recommendation

Chang Meng<EMAIL_ADDRESS>Shenzhen International Graduate
School, Tsinghua UniversityShenzhenChina ,  Chenhao Zhai<EMAIL_ADDRESS>Shenzhen International Graduate School, Tsinghua UniversityShenzhenChina ,  Yu
Yang<EMAIL_ADDRESS>Shenzhen International Graduate School, Tsinghua
UniversityShenzhenChina ,  Hengyu Zhang<EMAIL_ADDRESS>Shenzhen International Graduate School, Tsinghua UniversityShenzhenChina and
Xiu Li<EMAIL_ADDRESS>Shenzhen International Graduate School,
Tsinghua UniversityShenzhenChina

(2023)

###### Abstract.

Multi-behavior recommendation algorithms aim to leverage the multiplex
interactions between users and items to learn users’ latent preferences.
Recent multi-behavior recommendation frameworks contain two steps: fusion and
prediction. In the fusion step, advanced neural networks are used to model the
hierarchical correlations between user behaviors. In the prediction step,
multiple signals are utilized to jointly optimize the model with a multi-task
learning (MTL) paradigm. However, recent approaches have not addressed the
issue caused by imbalanced data distribution in the fusion step, resulting in
the learned relationships being dominated by high-frequency behaviors. In the
prediction step, the existing methods use a gate mechanism to directly
aggregate expert information generated by coupling input, leading to negative
information transfer. To tackle these issues, we propose a Parallel Knowledge
Enhancement Framework (PKEF) for multi-behavior recommendation. Specifically,
we enhance the hierarchical information propagation in the fusion step using
parallel knowledge (PKF). Meanwhile, in the prediction step, we decouple the
representations to generate expert information and introduce a projection
mechanism during aggregation to eliminate gradient conflicts and alleviate
negative transfer (PME). We conduct comprehensive experiments on three real-
world datasets to validate the effectiveness of our model. The results further
demonstrate the rationality and effectiveness of the designed PKF and PME
modules. The source code and datasets are available at https://github.com/MC-
CV/PKEF.

Multi-behavior Recommendation, Multi-task, Knowledge Enhancement

††journalyear: 2023††copyright: rightsretained††conference: Proceedings of the
32nd ACM International Conference on Information and Knowledge Management;
October 21–25, 2023; Birmingham, United Kingdom††booktitle: Proceedings of the
32nd ACM International Conference on Information and Knowledge Management
(CIKM ’23), October 21–25, 2023, Birmingham, United Kingdom††doi:
10.1145/3583780.3615004††isbn: 979-8-4007-0124-5/23/10††ccs: Information
systems Recommender systems

##  1\. Introduction

Recommender systems are information filtering techniques designed to provide
personalized services based on user preferences. In our daily lives,
recommendation systems are widely used in various scenarios such as
e-commerce, social media, music, and video platforms. Early collaborative
filtering (CF) techniques (Su and Khoshgoftaar, 2009) made recommendations
based on users’ historical interactions with items, but they had limitations
in effectively utilizing diverse user behavior information for
recommendations. In the real world, user behavior goes beyond a single type
and includes various behaviors such as viewing, adding to cart, and
purchasing. Among them, we mainly focus on a specific behavior, namely target
behavior (e.g., buy) and considering other behaviors as auxiliary behaviors
(Gao et al., 2019a; Xia et al., 2020; Jin et al., 2020). These multiple
behavior signals carry rich user preferences, which can be leveraged to
comprehensively understand user needs and provide better services.

Recent researches have focused on effectively leveraging multiple behavior
signals for recommendations. Existing frameworks for multi-behavior
recommendation contain two steps: multi-behavior fusion and multi-behavior
prediction (He et al., 2023). In the fusion step, advanced neural networks are
applied to capture the correlations between users and items across multiple
behaviors. In the prediction step, multi-task learning (MTL) is devised to
further utilize the heterogeneous interaction information (Huang, 2021).

Multi-behavior Fusion. Early studies applied matrix factorization (Tang et
al., 2016; Krohn-Grimberghe et al., 2012; Singh and Gordon, 2008) to multi-
behavior recommendation. With the rise of deep learning, neural network-based
approaches (Xia et al., 2020; Gao et al., 2019a; Guo et al., 2019) have become
popular in multi-behavior fusion. These methods can model the complex
relationships between users and items, capturing richer user interests and
item features. Among them, graph neural networks (He et al., 2020; Wang et
al., 2019; Chen et al., 2020; Guo et al., 2023; Meng et al., 2022, 2023) have
been widely applied in multi-behavior recommendation due to their ability to
efficiently utilize high-order connectivity between users and items (Chen et
al., 2021; Xia et al., 2021b). For example, MBGCN (Jin et al., 2020) and GNMR
(Xia et al., 2021a) utilize graph neural networks to improve recommendation
performance. However, these methods do not consider using dependencies between
behaviors to assist model learning. In the real world, user behaviors often
follow a hierarchical order, such as view → cart → buy. User preference
information from upstream behaviors (e.g., view) can be used to assist
downstream tasks (e.g., cart and buy) (Gao et al., 2019a; Yan et al., 2022;
Cheng et al., 2023). CRGCN (Yan et al., 2022) and MB-CGCN (Cheng et al., 2023)
integrate the cascade dependencies between behaviors into graph convolutional
networks (GCNs), facilitating the learning of user and item embeddings. These
models, which consider behavioral hierarchy, have demonstrated better
performance compared to previous approaches.

Multi-behavior Prediction. Multi-task learning (MTL) is a commonly used
approach in multi-behavior prediction, as it can effectively utilize complex
heterogeneous signals from multiple tasks to jointly optimize the model.
Existing multi-task learning methods typically have coupled inputs for
different tasks (Caruana, 1997; Ma et al., 2018; Tang et al., 2020). They
generate multiple experts in different ways and aggregate the expert
information for subsequent tasks. For example, MMOE (Ma et al., 2018) utilizes
coupled representations to generate multiple experts and assigns learnable
weights to each task to aggregate the expert information. PLE (Tang et al.,
2020) further improves this approach by generating the specific experts for
each task on the basis of the shared experts for all tasks.

Figure 1. Histogram of user numbers w.r.t interaction numbers for different
behaviors. Figure 2. Illustration of the learned multi-behavioral correlations
in the cascade stream.

Although cascade graph convolutional networks and MTL-based multi-behavior
recommendation methods have made significant progress for multi-behavior
fusion and prediction respectively, they also have their limitations:

  * •

Ignorance of the imbalanced behavioral distribution. As shown in Figure 1
(plotting the data distribution), the interactions for different behaviors are
highly imbalanced. One behavior (e.g., view) may account for the majority of
the total interactions. In the cascade behavior modeling, this imbalance
problem is further exacerbated. As shown in Figure 2, in the cascade stream,
upstream behaviors have richer interaction information compared to downstream
behaviors. Thus, in the process of behavior propagation, the learned
relationships are dominated by upstream behaviors, leading to a biased
relationship learned by the model towards upstream behaviors, which interferes
with downstream behavior prediction.

  * •

Negative transfer problem. When training multiple tasks, the performance of
certain tasks can be negatively affected or interfered with by other tasks,
resulting in performance degradation. This is known as the negative transfer
phenomenon (Torrey and Shavlik, 2010). In multi-task learning, although
coupled inputs can share information from different behaviors, they can also
introduce potential gradient conflict issues (explained in Section 3.2.1).
Additionally, when aggregating expert information from different behaviors for
a specific task, noise from other behaviors is often introduced, leading to
negative transfer problems.

To address these two issues, we propose a Parallel Knowledge Enhancement based
Framework (PKEF) for Multi-behavior Recommendation. It consists of the
Parallel Knowledge Fusion module (PKF) and the Projection Disentangling Multi-
Experts network (PME). To address the first issue, PKF combines the cascade
and parallel paradigms, leveraging parallel knowledge for adaptive enhancement
of different behaviors’ representations while learning hierarchical
correlation information to correct the information bias caused by imbalanced
behavioral interactions.

To address the second issue, PME regards different behaviors as independent
tasks, generates corresponding expert informations for each behavior with
separate inputs, and aggregates the expert information from different
behaviors using learnable weights. Considering that the aggregation of
different behaviors may introduce noise during the learning process for a
specific behavior task, PME introduces a projection mechanism during
aggregation to disentangle the shared and unique parts for other behavioral
experts. The shared part is used for aggregation, avoiding the introduction of
harmful information. For the unique part, an auxiliary loss is designed for
optimizing, which promotes the effectiveness of complementary shared
information. PME alleviates the negative transfer phenomenon while solving the
gradient conflict problem (explained in Section 3.2.2).

In summary, our work makes the following contributions:

  * •

We investigate the issues of ignorance of the imbalanced behavioral
distribution in the cascade paradigm of multi-behavior recommendation and the
negative transfer phenomenon in MTL. We propose an innovative multi-behavior
recommendation framework (PKEF) to address these issues. It consists of the
Parallel Knowledge Fusion module (PKF) and the Projection Disentangling Multi-
Expert network (PME).

  * •

To achieve better recommendation performance, we address the issue of
imbalanced data distribution for different behaviors by enhancing the
hierarchical information propagation in the cascade process using parallel
knowledge (PKF). Additionally, we alleviate the gradient conflict introduced
by coupled MTL inputs and propose a projection-based denoising method to
remove harmful information between behaviors, effectively solving the negative
transfer problem (PME).

  * •

We conduct comprehensive experiments on three real-world datasets to
demonstrate the effectiveness of our model. Further experimental results
verify the rationality and effectiveness of the designed PKF and PME modules.

##  2\. Related Work

Multi-behavior Recommendation. Multi-behavior recommendation methods use
multiple user-item interactions to solve the data sparsity problem. In recent
years, this approach has attracted widespread attention.

Early multi-behavior recommendation methods usually handle multi-behavior data
by introducing multiple matrix factorization (Tang et al., 2016; Krohn-
Grimberghe et al., 2012; Singh and Gordon, 2008) or designing new sampling
strategies (Qiu et al., 2018; Loni et al., 2016; Guo et al., 2017). The former
one extends the traditional matrix factorization technique by conducting it on
multiple matrices with shared embeddings, such as CMF (Zhao et al., 2015). The
latter one uses multiple behaviors as auxiliary data and designs new sampling
strategies to enrich the training samples, such as MF-BPR (Loni et al., 2016)
and VALS (Ding et al., 2018), which introduce and improve negative sampling
strategies.

With the development of deep learning techniques (Yu et al., 2023; Zhou et
al., 2023; Liang et al., 2023), researchers have started to explore multi-
behavior recommendation models based on deep neural network (DNN) or graph
convolutional network (GCN). DNN-based models usually design models to learn
embeddings from each behavior and integrating them into the prediction of
target behaviors. For example, DIPN (Guo et al., 2019) and MATN (Xia et al.,
2020) use different attention mechanisms to model the relationship between
behaviors for embedding learning and aggregation. NMTR (Gao et al., 2019a)
differs from the above methods by using a multi-task learning model in which
all behaviors of the users serve as prediction targets and the prediction
scores of the previous behavior are passed to the next behavior for
prediction.

GCN-based models learn user embeddings by constructing a unified user-item
graph and performing graph convolution operations. GHCF (Chen et al., 2021)
explicitly models the high-order relationship between users and items through
GCN and performs multi-task learning to predict each behavior through a non-
sampling approach. MBGCN (Jin et al., 2020) takes behavior semantic into
account, capturing it by item-item propagation layer and combines behavior
semantic with behavior contributions learned from user-item propagation layer
for score prediction. The recently proposed CRGCN (Yan et al., 2022) and MB-
CGCN (Cheng et al., 2023) take into account the hierarchical correlation
between behaviors and achieve great performance by building cascaded graph
convolutional networks to capture user preferences. However, due to the
imbalanced distribution of the interactions among different behaviors, simply
employing cascaded networks will lead to the learned relationships being
dominated by high-frequency behaviors, which interferes with downstream
behavior prediction.

MTL for Recommendation. With the growing diversity of user interests, the
limitations of single-task learning in traditional recommender systems have
become more and more obvious, especially in the face of multiple signals. To
solve the above dilemma, in recent years, researchers have attempted to apply
multi-task learning to recommender systems. One model widely used in multi-
behavior recommendation is the shared bottom (Caruana, 1997) structure, where
each task shares the same bottom parameters to extract common features, while
the parameters at the top layer are independent. However, approaches based on
this structure (Chen et al., 2021; Gao et al., 2019a; Xia et al., 2021c) will
lead to negative transfer phenomenon and trigger a seesaw effect for tasks
with weak relevance. To solve these problems, MTL structures based on gated
expert algorithm are proposed. MOE (Jacobs et al., 1991) divides the shared
bottom structure into multiple experts that learn different features
separately. MMOE (Ma et al., 2018) extends MOE by introducing a task-specific
gating mechanism to obtain different fusion weights in multi-task learning.
PLE (Tang et al., 2020) further proposes to employ shared or task-specific
experts at the bottom layer and combine them adaptively through gating
networks. However, these methods use coupled inputs for multiple tasks, which
leads to the gradient conflict problem and negative transfer phenomenon, thus
affecting the model performance (Illustrated in Section 3.2.1).

##  3\. Problem Definition

###  3.1. Problem Definition

We define $u$ and $v$ as a user and an item, respectively. Meanwhile,
$\mathbf{U}$ and $\mathbf{V}$ denote the user and item sets, respectively. The
adjacency matrices of multiple behaviors can be represented by a set, i.e.,
$\mathcal{M}=\left\\{\mathbf{M}_{1},\mathbf{M}_{2},\cdots,\mathbf{M}_{K}\right\\}$,
where
$\mathbf{M}_{k}=\left[m_{(k)uv}\right]_{|\mathbf{U}|\times|\mathbf{V}|}\in\left\\{0,1\right\\}$
indicates whether the user $u$ interacted with the item $v$ under behavior
$k$. Furthermore, in order to represent the heterogeneous interaction
information of users and items more conveniently, we define the multiplex
user-item bipartite graph $\mathcal{G}=(\mathcal{H},\mathcal{E},\mathcal{M})$,
where $\mathcal{H}=\mathbf{U}\cup\mathbf{V}$,
$\mathcal{E}=\cup_{k=1}^{K}\mathcal{E}_{k}$ is the edge set including all
behavior records between users and items. In the multi-behavior
recommendation, we assume that $k\in\\{1,2,...,K\\}$, and the number
corresponds to the upstream and downstream relationships between behaviors.
The larger the number, the more downstream the behavior (i.e., $K$ is the most
downstream behavior). Last but not least, there exists a target behavior
(denotes as $\mathbf{M}_{K}$) to be optimized, which is purchasing (buying)
for e-commerce scenarios.

###  3.2. Gradient Issue in MTL

####  3.2.1. Gradient Conflict with Coupled Input

Most of the existing methods use the coupled input for MTL, as summarized in
Section 2. This may cause a gradient conflict issue in MTL which restricts
their learning ability for each task. As the classical MTL methods directly
couple the representations of different behaviors together with different
weights, we have:

(1)
$\displaystyle\mathbf{e}_{u}^{*}=\sum_{k=1}^{K}\lambda_{k}\mathbf{e}_{u}^{k},\mathbf{e}_{v}^{*}=\sum_{k=1}^{K}\lambda_{k}\mathbf{e}_{v}^{k},$

where $K$ is the number of behaviors, $\lambda_{k}$ is the weight of $k$-th
behavior. Taking $(\mathbf{e}_{u}^{*},\mathbf{e}_{v}^{*})$ as input for MTL,
the loss function can be formulated as:

(2) $\displaystyle\mathcal{L}_{uv}$
$\displaystyle=\sum_{k=1}^{K}L(f_{k}(\mathbf{e}_{u}^{*},\mathbf{e}_{v}^{*})-{o}_{uv}^{k}),$

where $\hat{o}_{uv}^{k}$ denotes the predictive probability that user $u$ will
interact with item $v$ under the k-th behavior, ${o}_{uv}^{k}$ is the true
label, $L(\cdot)$ is the loss function, and $f_{k}(\cdot)$ is the predictive
function in MTL models. Then we have:

(3)
${\partial{\mathcal{L}_{uv}}\over{\partial{(\mathbf{e}_{u}^{*}\circ\mathbf{e}_{v}^{*})}}}=\sum_{k=1}^{K}{\partial{f_{k}(\mathbf{e}_{u}^{*},\mathbf{e}_{v}^{*})}\over{\partial{(\mathbf{e}_{u}^{*}\circ\mathbf{e}_{v}^{*})}}}*{L^{{}^{\prime}}(f_{k}(\mathbf{e}_{u}^{*},\mathbf{e}_{v}^{*})-{o}_{uv}^{k})}=\sum_{k=1}^{K}{a_{uv}^{k}{\mathbf{r}^{k}}},$

where ($\circ$) is the hadamard product operation,
$a_{uv}^{k}=L^{{}^{\prime}}(f_{k}(\mathbf{e}_{u}^{*},\mathbf{e}_{v}^{*})-{o}_{uv}^{k})$
is a scalar.
$\mathbf{r}^{k}={\partial{f_{k}(\mathbf{e}_{u}^{*},\mathbf{e}_{v}^{*})}\over{\partial{(\mathbf{e}_{u}^{*}\circ\mathbf{e}_{v}^{*})}}}$.
As $\mathbf{r}^{k}$ denotes the derivative of a scalar to a vector, it is also
a vector. $\forall$ $k\in\\{1,2,\ldots,K\\}$, $a_{uv}^{k}{\mathbf{r}^{k}}$
determines the updating magnitude and direction of the vector
$\mathbf{e}_{u}^{*}\circ\mathbf{e}_{v}^{*}$. We can see that the gradients
from all behaviors are coupled, and they jointly optimize the same vector
$\mathbf{e}_{u}^{*}\circ\mathbf{e}_{v}^{*}$, which leading to gradient
conflicts. As a result, the harmful information coupled in the input affects
the learning of the target behavior information in the training process,
leading to negative transfer.

####  3.2.2. Projection Disentangling Multi-Experts with Separated Input

To handle the above problem, we first need to utilize the separated input of
each behavior to generate the behavior-specific expert information and
behavior-specific gating weight. Thus, we have:

(4)
$\mathbf{e}_{uv}^{k}=\mathbf{e}_{u}^{k}\circ\mathbf{e}_{v}^{k},\hat{\mathbf{g}}_{exp}^{k}=FC_{g}^{k}(\mathbf{e}_{u}^{k}||\mathbf{e}_{v}^{k})$

where $\hat{\mathbf{g}}_{exp}^{k}\in\mathbb{R}^{K\times 1}$ is the weight of
gate. $FC_{g}^{k}(\cdot)$ represents the behavior-specific fully connected
layer. Further, while aggregating the information from different experts, the
gating mechanism simultaneously introduce the negative information from other
experts. Thus, we need to extract the information that is useful to the
prediction of behavior $k$ from other experts
($\mathbf{e}_{uv}^{k^{\prime}}$). In details, we leverage a projection
mechanism and have:

(5)
$\mathbf{e}_{sha}^{k^{\prime},k}=a_{k^{\prime},k}\mathbf{e}_{uv}^{k},\hat{\mathbf{e}}_{exp}^{k}=\mathop{Concat}\limits_{k^{\prime}=1}^{K}(\mathbf{e}_{sha}^{k^{\prime},k})$

where $\mathbf{e}_{sha}^{k^{\prime},k}$ represents the shared information
extracted from $\mathbf{e}_{uv}^{k^{\prime}}$ with the guidance of
$\mathbf{e}_{uv}^{k}$.
$a_{k^{\prime},k}=\frac{\mathbf{e}_{uv}^{k^{\prime}}\cdot\mathbf{e}_{uv}^{k}}{|\mathbf{e}_{uv}^{k}||\mathbf{e}_{uv}^{k}|}$
is a scalar and can be flexibly adjusted. In practice, we adjust the scalar by
multiplying by a small value.

Finally, we analyse the optimization of input
$(\mathbf{e}_{u}^{t}\circ\mathbf{e}_{v}^{t})$, and have:

(6)
$\displaystyle{\partial{\mathcal{L}_{uv}}\over{\partial{(\mathbf{e}_{u}^{t}\circ\mathbf{e}_{v}^{t})}}}$
$\displaystyle=\sum_{k=1}^{K}\frac{\partial\hat{f}_{k}(\hat{\mathbf{g}}_{exp}^{k},\hat{\mathbf{e}}_{exp}^{k})}{\partial(\mathbf{e}_{u}^{t}\circ\mathbf{e}_{v}^{t})}$
$\displaystyle=\sum_{k=1}^{K}\frac{\partial\hat{f}_{k}(\hat{\mathbf{g}}_{exp}^{k},\mathbf{e}_{uv}^{k})}{\partial(\mathbf{e}_{u}^{t}\circ\mathbf{e}_{v}^{t})}=\frac{\partial\hat{f}_{t}(\hat{\mathbf{g}}_{exp}^{t},\mathbf{e}_{uv}^{t})}{\partial(\mathbf{e}_{u}^{t}\circ\mathbf{e}_{v}^{t})}$

where $\hat{f}_{k}(a,b)$ means the expressions with respect to variables $a$
and $b$ under behavior $k$. Without loss of generality, we can find that
$\forall t\in\\{1,2,\cdots,K\\}$, the gradient of each behavior optimizes
along the direction of their respective input (e.g., the gradient of the
$t$-th behavior optimizes $\mathbf{e}_{u}^{t}\circ\mathbf{e}_{v}^{t}$
independently), so that the gradient conflicts problem is successfully solved.

##  4\. METHODOLOGY

We devise a ”Parallel Knowledge Enhancement based Framework” (PKEF) for multi-
behavior recommendation, which contains two parts: (1) Parallel Knowledge
Fusion (PKF) module; (2) Projection Disentangling Multi-Experts (PME) network.
Figure 3 illustrates the technical details of the proposed framework.

Figure 3. Illustration of the proposed PKEF framework. ($\oplus$) denotes the
element-wise addition operation. Lines of different colors correspond to
representations of different colors in the Projection module (e.g., blue lines
denote $e_{ori}$ and green lines represent $e_{sha}$). For brevity, we
illustrate the knowledge fusion between cascade and parallel stream with a
projection scheme.

###  4.1. Embedding Layer

In industrial applications, users and items are often denoted as high-
dimensional one-hot vectors. However, to transform the high-dimensional sparse
vectors into low-dimensional dense embeddings, we apply the embedding lookup
operation for user $u$ and item $v$ to obtain the embedding vectors.
Generally, we have:

(7)
$\displaystyle\mathbf{E}_{u}=[\underbrace{\mathbf{x}_{u_{1}},\cdots,\mathbf{x}_{u_{|\mathbf{U}|}}}_{\text{users
embeddings }}],$
$\displaystyle\mathbf{x}_{u}=\operatorname{LookUp}\left(u,\mathbf{E}_{u}\right)$
$\displaystyle\mathbf{E}_{v}=[\underbrace{\mathbf{y}_{v_{1}},\cdots,\mathbf{y}_{v_{|\mathbf{V}|}}}_{\text{item
embeddings }}],$
$\displaystyle\mathbf{y}_{v}=\operatorname{LookUp}\left(v,\mathbf{E}_{v}\right)$

where $\mathbf{E}_{u}\in\mathbb{R}^{|\mathbf{U}|\times d}$ and
$\mathbf{E}_{v}\in\mathbb{R}^{|\mathbf{V}|\times d}$ are the embedding tables
for users and items, respectively, $|\mathbf{U}|$ and $|\mathbf{V}|$ are the
total number of users and items. $\mathbf{x}_{u}\in\mathbb{R}^{d}$ and
$\mathbf{y}_{v}\in\mathbb{R}^{d}$ denotes the embedding vectors of user $u$
and item $v$, and $d$ is the embedding size.

###  4.2. Parallel Knowledge Fusion

The recent multi-behavior methods ignore the imbalanced distribution of
different behaviors in the fusion step. Thus the learning of these models will
be more inclined to high-frequency behaviors, resulting in poor prediction
effects on target behaviors.

To solve the above problem, in our model, we combine both cascade and parallel
paradigms to learn complex interactions between multiple behaviors. Our
Parallel Knowledge Fusion (PKF) module utilizes parallel knowledge to enhance
the representations of different behaviors while learning hierarchical
correlation information, so as to correct the information bias caused by the
imbalance of behavior interaction distribution.

####  4.2.1. Cascade Correlation Learning.

As we have the adjacency matrices
$\mathbf{M}_{1},\mathbf{M}_{2},...,\mathbf{M}_{K}$ for different behaviors,
for convenience, we further process the matrices, and it can be formulated as:

(8) $\mathbf{A}_{k}=\left(\begin{array}[]{cc}0&\mathbf{M}_{k}\\\
\left(\mathbf{M}_{k}\right)^{T}&0\end{array}\right)$

where $\mathbf{M}_{k}$ is the user-item adjacency interaction matrix of
behavior $k$,
$\mathbf{M}_{k}\in\mathbb{R}^{(|\mathbf{U}|+|\mathbf{V}|)\times(|\mathbf{U}|+|\mathbf{V}|)}$,
$|\mathbf{U}|$ and $|\mathbf{V}|$ denote the number of users and items,
respectively. As graph neural networks (He et al., 2020; Wang et al., 2019)
have been widely used to model the high-order interactions between users and
items, we conduct a GNN-based paradigm to encode the information of each
behavior. Specifically, in each behavior $k$, we apply the message passing to
capture the high-order interaction information. Here, we simply leverage
LightGCN (He et al., 2020) as the GCN aggregator to aggregate information on
each layer $l$:

(9) $\mathbf{z}^{k,l+1}=\mathbf{\hat{A}}_{k}\mathbf{z}^{k,l}+\mathbf{z}^{k,l}$

where $\mathbf{z}^{k,l}=\mathbf{z}_{u}^{k,l}||\mathbf{z}_{v}^{k,l}$. $(||)$ is
the concatenate operation.
$\mathbf{\hat{A}}_{k}=\mathbf{D}^{-1}(\mathbf{A}_{k}+\mathbf{I})$ is the left
normalized adjacency matrix with added self-connections and $\mathbf{D}$ is a
diagonal degree matrix, which is defined as
$\mathbf{D}_{ii}=\sum_{j}(\mathbf{A}_{k}+\mathbf{I})_{ij}$. $\mathbf{I}$
denotes an identity matrix. And the initial
$\mathbf{z}^{1,0}=\mathbf{x}_{u}||\mathbf{y}_{v}$.

Further, following MB-CGCN (Cheng et al., 2023), we conduct a cascade paradigm
to learn the hierarchical correlation information of different behaviors. We
have:

(10) $\mathbf{z}^{k+1,0}=\mathbf{z}^{k,L_{k}}+\mathbf{z}^{k,0}$

where $L_{k}$ denotes the total layers of GNN of the $k$-th behavior. Here, we
apply a residual connection to combine the first and the last layer of the
upstream behavior representation as the input of the downstream behavior.

####  4.2.2. Parallel Interaction Enhancing.

In the previous part, we have modeled the hierarchical correlations of
different behaviors. However, as we have illustrated in the introduction,
imbalanced distribution of multiplex interactions will impact the learning of
target behavior. In order to handle this problem, we further conduct a
parallel learning paradigm which independently learns the representation of
each behavior, and then fusioning the knowledge on each layer corresponding to
the cascade stream.

Similar to the process of Equation 9, we first apply the same way to each
behavior, and have:

(11)
$\mathbf{p}^{k,l+1}=\mathbf{\hat{A}}_{k}\mathbf{p}^{k,l}+\mathbf{p}^{k,l}$

where $\mathbf{p}^{k,l}=\mathbf{p}_{u}^{k,l}||\mathbf{p}_{v}^{k,l}$. $(||)$ is
the concatenate operation. $\mathbf{\hat{A}}_{k}$ is the same as in Equation
9. And the initial $\mathbf{p}^{1,0}=\mathbf{x}_{u}||\mathbf{y}_{v}$.

Then we devise two schemes to fuse the knowledge between the parallel and
cascade streams, improving the Equation 9. Besides, we conduct comparison
experiments with other schemes (shown in Section 5.3.2). For simplicity, we
denote $\mathbf{e}_{par}^{k,l}=\mathbf{\hat{A}}_{k}\mathbf{p}^{k,l}$ and
$\mathbf{e}_{cas}^{k,l}=\mathbf{\hat{A}}_{k}\mathbf{z}^{k,l}$.

(1) Projection-enhanced Knowledge Fusion. This scheme is inspired by DUMN
(Bian et al., 2021), in which they used the representation projection
mechanism to decouple the implicit feedback representation by the explicit
feedback representation. Here, on each layer, we project the parallel
representation onto the cascade representation and use the part that is
collinear with it to enhance the cascade representation. It can be formulated
as:

(12) $\left\\{\begin{array}[]{c}\begin{aligned}
\mathbf{p}_{col}^{k,l}&=\frac{\mathbf{e}_{par}^{k,l}\cdot\mathbf{e}_{cas}^{k,l}}{|\mathbf{e}_{cas}^{k,l}|}\frac{\mathbf{e}_{cas}^{k,l}}{|\mathbf{e}_{cas}^{k,l}|}\\\
\mathbf{z}^{k,l+1}&=\mathbf{e}_{cas}^{k,l}+\mathbf{z}^{k,l}+\mathbf{p}_{col}^{k,l}\end{aligned}\end{array}\right.$

where ($\cdot$) is the vector inner product operation.
$\mathbf{e}_{par}^{k,l}$ and $\mathbf{e}_{cas}^{k,l}$ are the representations
of the parallel and cascade streams, respectively. $\mathbf{p}_{col}^{k,l}$
contains a mixture of behavior-specific and hierarchical correlation
information.

(2) Vanilla-enhanced Knowledge Fusion. Meanwhile, inspired by the vanilla
attention (Zhou et al., 2018), we devise a fusion scheme that has the similar
form with it, and have:

(13) $\left\\{\begin{array}[]{c}\begin{aligned}
\mathbf{w}^{k,l}&=Softmax(\mathbf{W}_{k,l}\mathbf{e}_{cas}^{k,l}+\mathbf{b}_{k,l})\\\
\mathbf{f}^{k,l}&=Concat([\mathbf{e}_{cas}^{k,l},\mathbf{e}_{par}^{k,l},\mathbf{e}_{cas}^{k,l}-\mathbf{e}_{par}^{k,l},\mathbf{e}_{cas}^{k,l}\circ\mathbf{e}_{par}^{k,l}])\\\
\mathbf{z}^{k,l+1}&=\mathbf{e}_{cas}^{k,l}+\mathbf{z}^{k,l}+\sum_{j}\mathbf{w}^{k,l}(j)\cdot\mathbf{f}^{k,l}(j)\end{aligned}\end{array}\right.$

where $(\circ)$ is the hadamard product operation,
$\mathbf{W}_{k,l}\in\mathbb{R}^{4\times 4d}$ and
$\mathbf{b}_{k,l}\in\mathbb{R}^{4\times 1}$ are feature transformation matrix
and bias matrix. $d$ is the dimension of embedding.

For the output of each behavior, we have:

(14)
$\mathbf{p}^{k,*}=\sum_{l=0}^{L_{k}}\mathbf{p}^{k,l},\mathbf{z}^{k,*}=\sum_{l=0}^{L_{k}}\mathbf{z}^{k,l}$

where $L_{k}$ is the number of GNN layers of the $k$-th behavior.

###  4.3. Projection Disentangling Multi-Experts Network

As we have obtained the representations of each behavior $k$ in the previous
section, we need to design a proper structure to further leverage the
multiplex signals with these representations. It has been verified in many
methods (Ma et al., 2018; Tang et al., 2020; Caruana, 1997) that a multi-task
learning module can perfectly handle this. The MTL structure first couples the
representations of all behaviors, then generate kinds of experts by the
coupled input, further applies a gating mechanism to aggregate the expert
information as the output, and finally utilizes the prediction losses of
different behaviors to jointly optimize the model.

However, the existing MTL structures utilize a couple representation as the
input while introducing noise from other behaviors while using gating
mechanisms to aggregate information from different experts. This leads to the
gradient conflict during the learning process. Thus, we proposed a well-
designed MTL module to handle the above problems. The following are details.

####  4.3.1. Generating of Experts.

As coupled input contains mixed information of different behaviors, making the
gradient coupled and conflict, we do not combine the multi-behavioral
representations together. We directly leverage each representation to generate
the behavior-specific experts:

(15) $\mathbf{q}^{k}=\mathbf{z}_{u}^{k,*}\circ\mathbf{z}_{v}^{k,*}$

where $(\circ)$ is the hadamard product operation.

####  4.3.2. Aggregating of Experts.

In order to alleviate the negative information transfer from other behavior-
specific experts, we improve the gating mechanism with a representation
projection mechanism. Take the behavior $k$ as an example, we have:

(16) $\left\\{\begin{array}[]{c}\begin{aligned}
\mathbf{q}_{sha}^{k^{\prime},k}&=\frac{\mathbf{q}^{k^{\prime}}\cdot\mathbf{q}^{k}}{|\mathbf{q}^{k}|}\frac{\mathbf{q}^{k}}{|\mathbf{q}^{k}|}\\\
\mathbf{q}_{uni}^{k^{\prime},k}&=\mathbf{q}^{k^{\prime}}-\mathbf{q}_{sha}^{k^{\prime},k}\end{aligned}\end{array}\right.$

where ($\cdot$) is the vector inner product operation.
$\mathbf{q}^{k^{\prime}}$ and $\mathbf{q}^{k}$ are the representations of the
$k^{\prime}$\- and $k$-th behavior, respectively.
$\mathbf{q}_{sha}^{k^{\prime},k}$ contains a mixture of the $k^{\prime}$\- and
$k$-th behavioral correlation information. $\mathbf{q}_{uni}^{k^{\prime},k}$,
which represents the unique part of the $k^{\prime}$-th behavior, and
$\mathbf{q}^{k}$, which denotes the $k$-th behavior, are distinctive and
should be as orthogonal as possible.

As we can see, the projection mechanism disentangle $\mathbf{q}^{k^{\prime}}$
by the guidance of $\mathbf{q}^{k}$, thus the shared and unique parts of other
behaviors can be further utilized to alleviate the negative transfer caused by
gating aggregation. To be specific, we take the shared representations
$\mathbf{q}_{sha}^{k^{\prime},k}(k^{\prime}\in\\{1,2,...,K\\}\cap
k^{\prime}\neq k)$ of other behaviors and $\mathbf{q}^{k}$ as targets of
aggregation by the $k$-th gate, and have:

(17) $\left\\{\begin{array}[]{c}\begin{aligned}
\mathbf{g}^{k}&=Softmax(\mathbf{W}_{g}\mathbf{z}^{k,*}+\mathbf{b}_{g})\\\
\hat{\mathbf{q}}^{k}&=\mathop{Concat}\limits_{k^{\prime}=1}^{K}(\mathbf{q}_{sha}^{k^{\prime},k})\\\
\hat{o}_{uv}^{k,cas}&=h^{k}(\sum_{j=1}^{K}{\mathbf{g}^{k}(j)\cdot\hat{\mathbf{q}}^{k}(j)})\end{aligned}\end{array}\right.$

where $\mathbf{W}_{g}\in\mathbb{R}^{K\times d}$ and
$\mathbf{b}_{g}\in\mathbb{R}^{K\times 1}$ are feature transformation matrix
and bias matrix, and $\mathbf{g}^{k}\in\mathbb{R}^{K\times 1}$ is the
attention vector which are used as selector to calculate the weighted sum of
all experts. $h^{k}(\cdot)$ is the tower function. $\hat{o}_{uv}^{k,cas}$ is
the prediction score of whether user $u$ will have interaction with item $v$
under behavior $k$ at the cascade stream.

Besides, we design a prediction task for the unique representation
$\mathbf{q}_{uni}^{k^{\prime},k}$. This task takes full advantage of the
mutually exclusive relationship between $\mathbf{q}_{uni}^{k^{\prime},k}$ and
$\mathbf{q}^{k}$, facilitating the learning of the $k$-th behavior. Details
are shown in Section 4.4.3.

###  4.4. Joint Optimization

####  4.4.1. Parallel Loss.

As we have obtained multi-behavioral representations from the parallel stream,
we design a parallel loss to help the learning of each representations. In
details, we have:

(18)
$\mathcal{L}^{par}=-\sum_{k=1}^{K}\sum_{(u,s,t)\in\mathcal{O}_{k}}\lambda_{k}*\textup{ln}\sigma(\hat{o}_{us}^{k,par}-\hat{o}_{ut}^{k,par})$

where $\hat{o}_{uv}^{k,par}=\mathbf{p}_{u}^{k,*}\cdot\mathbf{p}_{v}^{k,*}$ is
the prediction score of whether user $u$ will have interaction with item $v$
under behavior $k$ at the parallel stream. And we apply a Bayesian
Personalized Ranking (BPR) (Rendle et al., 2012) loss to optimize the model.
$\mathcal{O}_{k}=\left\\{(u,s,t)|(u,s)\in\mathcal{O}_{k}^{+},(u,t)\in\mathcal{O}_{k}^{-}\right\\}$
denotes the training dataset. $\mathcal{O}_{k}^{+}$ indicates observed
positive user-item interactions under behavior $k$ and $\mathcal{O}_{k}^{-}$
indicates unobserved user-item interactions under behavior $k$. $\lambda_{k}$
is the coefficient of behavior $k$. $\sigma$ is the Sigmoid function.

####  4.4.2. Cascade Loss.

Similar to the above, we devise a cascade loss for the cascade stream, and as
we have obtained the final prediction $\hat{o}_{uv}^{k,cas}$, we have:

(19)
$\mathcal{L}^{cas}=-\sum_{k=1}^{K}\sum_{(u,s,t)\in\mathcal{O}_{k}}\lambda_{k}*\textup{ln}\sigma(\hat{o}_{us}^{k,cas}-\hat{o}_{ut}^{k,cas})$

where the definition of parameters is the same to what in the parallel loss.

####  4.4.3. Unique Loss.

To make full use of the unique representation
$\mathbf{q}_{uni}^{k^{\prime},k}$, we design an auxiliary prediction task.
Specifically, we leverage $\mathbf{q}_{uni}^{k^{\prime},k}$ to predict the
interactive information of “$k^{\prime}$ without $k$”. In details, we have:

(20) $\mathcal{L}^{uni}=-\sum_{k=1}^{K}\sum\limits_{k^{\prime}=1\atop
k^{\prime}\neq
k}^{K}\sum_{(u,s,t)\in\mathcal{T}_{k^{\prime},k}}\lambda_{k}*\textup{ln}\sigma(\hat{o}_{us}^{k^{\prime},k,uni}-\hat{o}_{ut}^{k^{\prime},k,uni})$

where
$\hat{o}_{uv}^{k^{\prime},k,uni}=\mathbf{q}_{u,uni}^{k^{\prime},k}\cdot\mathbf{q}_{v,uni}^{k^{\prime},k}$
is the prediction score.
$\mathcal{T}_{k^{\prime},k}=\left\\{(u,s,t)|(u,s)\in\mathcal{O}_{k^{\prime}}^{+}\cap\mathcal{O}_{k}^{-},(u,t)\in\mathcal{O}_{k}^{-}\cup(\mathcal{O}_{k^{\prime}}^{+}\cap\mathcal{O}_{k}^{+})\right\\}$
denotes the training dataset. $\mathcal{O}_{k}^{+}$ indicates observed
positive user-item interactions under behavior $k$ and $\mathcal{O}_{k}^{-}$
indicates unobserved user-item interactions under behavior $k$. In short, we
remove from the behavioral adjacency matrix $\mathbf{M}_{k^{\prime}}$ the
positive items that $\mathbf{M}_{k^{\prime}}$ shares with $\mathbf{M}_{k}$.
Thus, we fully utilize the ”Only $k^{\prime}$” interactive information with
the help of the unique representations of behavior $k^{\prime}$.

In all, the final loss can be formulated as:

(21)
$\mathcal{L}(\Theta)=\mathcal{L}^{par}+\mathcal{L}^{cas}+\mathcal{L}^{uni}+\mu||\Theta||^{2}_{2}$

where $\Theta$ represents set of all model parameters. $\mu$ is the $L_{2}$
regularization coefficient for $\Theta$.

###  4.5. Complexity Analysis

####  4.5.1. Time Complexity.

The time complexity of PKEF primarily lies in the GNN parts, which consist of
cascade and parallel streams. Both the cascade and parallel parts have a
computational complexity of
$\sum_{k=1}^{K}{O\left(L^{k}\cdot\left|\mathcal{E}_{k}\right|\cdot d\right)}$.
Here, $\left|\mathcal{E}_{k}\right|$ represents the number of edges across all
graphs in the set $\mathcal{E}_{k}$, $K$ denotes the behavior number, $L_{k}$
refers to the number of GNN layers of the $k$-th behavior, and $d$ represents
the embedding size. Overall, the time complexity of PKEF is comparable to that
of existing GNN-based methods.

####  4.5.2. Space Complexity.

The learnable parameters in our proposed PKEF primarily come from the user and
item embeddings, denoted as $\mathbf{x}_{u}$ and $\mathbf{y}_{i}$
respectively. This is similar to existing methods. Furthermore, the dense
graphs $\mathcal{G}_{k}$ in the set $\mathcal{G}$ are transformed into sparse
behavior-specified matrices
$\mathbf{M}_{1},\mathbf{M}_{2},\cdots,\mathbf{M}_{K}$ for computational
purposes. This transformation allows us to perform computations without
requiring additional memory space to store the dense graphs. Hence, the memory
usage during the intermediate process remains within an acceptable range.

##  5\. Experiments

Table 1. Statistics of evaluation datasets.

Dataset | #User | #Item | #Interaction | #Target Interaction | #Interactive Behavior Type
---|---|---|---|---|---
Beibei | 21,716 | 7,977 | $3.3\times 10^{6}$ | 282,860 | {View,Cart,Buy}
Taobao | 15,449 | 11,953 | $1.2\times 10^{6}$ | 92,180 | {View,Cart,Buy}
Tmall | 41,738 | 11,953 | $2.3\times 10^{6}$ | 255,586 | {View,Collect,Cart,Buy}

###  5.1. Experimental Setting

####  5.1.1. Dataset Description

We follow MB-CGCN (Cheng et al., 2023) and CRGCN (Yan et al., 2022), and adopt
the same three datasets for evaluation, i.e., Beibei, Taobao and Tmall. For
these datasets, we adhere to previous studies’ methodology of removing
duplicates by retaining the earliest entry (Jin et al., 2020; Gao et al.,
2019b). Table 1 provides a summary of the statistical information for the
three datasets used in our experiments.

####  5.1.2. Evaluation Protocols

In all our experiments, we assess the performance of our proposed PKEF model
and baseline models based on the top-$k$ recommended items, using two
evaluation metrics: Hit Ratio (HR@k) and Normalized Discounted Cumulative Gain
(NDCG@k). Specifically, we set $k=10$ for our evaluations.

####  5.1.3. Baseline Models

To demonstrate the effectiveness of PKEF, we compare it with several state-of-
the-art methods, which can be divided into two categories: (1) Single-behavior
methods: MF-BPR (Rendle et al., 2012), NeuMF (He et al., 2017) and
LightGCN111https://github.com/kuandeng/LightGCN (He et al., 2020), (2) Multi-
behavior methods without MTL: RGCN (Schlichtkrull et al., 2018), GNMR (Xia et
al., 2021a), NMTR (Gao et al., 2019a),
MBGCN222https://github.com/tsinghua-fib-lab/MBGCN (Jin et al., 2020),
CRGCN333https://github.com/MingshiYan/CRGCN (Yan et al., 2022) and MB-
CGCN444https://github.com/SS-00-SS/MBCGCN (Cheng et al., 2023),

####  5.1.4. Parameter Settings

Our proposed PKEF is implemented in TensorFlow (Abadi et al., 2016). For a
fair comparison, following MB-CGCN (Cheng et al., 2023), we set the embedding
size to 64. We initialize the parameters using Xavier (Glorot and Bengio,
2010). The parameters are optimized by Adam (Kingma and Ba, 2014), while the
learning rate is set to $10^{-3}$. We search the number of GNN layers for
every behavior in {1,2,3,4} for user-item bipartite graph. In addition, we
adjust the loss coefficients for each behavior in {0,1/6,2/6,3/6,4/6,5/6,1}
and fix the sum of the coefficients for all actions as 1. Other parameters are
the same as MB-CGCN. All experiments are run for 5 times and average results
are reported. In addition, we conduct hyper-parameter analysis experiments
(shown in Section 5.4).

Table 2. The overall performance comparison. Boldface denotes the highest
score and underline indicates the results of the best baselines. $\star$
represents significance level $p$-value $<0.05$ of comparing PKEF with the
best baseline.

Model | Beibei | Taobao | Tmall
---|---|---|---
HR | NDCG | HR | NDCG | HR | NDCG
MF-BPR | 0.0191 | 0.0049 | 0.0076 | 0.0036 | 0.0230 | 0.0207
NeuMF | 0.0232 | 0.0135 | 0.0236 | 0.0128 | 0.0124 | 0.0062
LightGCN | 0.0391 | 0.0209 | 0.0411 | 0.0240 | 0.0393 | 0.0209
RGCN | 0.0363 | 0.0188 | 0.0215 | 0.0104 | 0.0316 | 0.0157
GNMR | 0.0413 | 0.0221 | 0.0368 | 0.0216 | 0.0393 | 0.0193
NMTR | 0.0429 | 0.0198 | 0.0282 | 0.0137 | 0.0536 | 0.0286
MBGCN | 0.0470 | 0.0259 | 0.0509 | 0.0294 | 0.0549 | 0.0285
CRGCN | 0.0459 | 0.0324 | 0.0855 | 0.0439 | 0.0840 | 0.0442
MB-CGCN | 0.0579 | 0.0381 | 0.1233 | 0.0677 | 0.0984 | 0.0558
PKEF | 0.1130⋆ | 0.0582⋆ | 0.1385⋆ | 0.0785⋆ | 0.1277⋆ | 0.0721⋆
Rel Impr. | 95.16% | 38.58% | 12.33% | 15.95% | 29.78% | 29.21%

###  5.2. Performance Comparison

Table 2 shows the performance of methods on three datasets with respect to
HR@10 and NDCG@10. We have the following findings:

  * •

Our PKEF model achieves the best performance across all three datasets.
Specifically, in terms of HR and NDCG metrics, PKEF outperforms the best
baselines on Beibei, Taobao, and Tmall datasets by 95.16$\%$, 12.33$\%$,
29.78$\%$ and 38.58$\%$, 15.95$\%$, 29.21$\%$, respectively. Our PKEF model
demonstrates significant enhancements in recommendation accuracy, particularly
when compared to the best baseline, MB-CGCN. This substantial progress
highlights the effectiveness of our model.

  * •

Multi-behavior models perform better than single-behavior models. For example,
MBGCN performs better than LightGCN. This indicates the superiority of
utilizing multiple types of interactions.

  * •

LightGCN consistently outperforms MF-BPR and NeuMF, while MBGCN outperforms
NMTR. This demonstrates the advantages of the GCN model, which leverages high-
order neighbor information on the user-item bipartite graph to learn
embeddings for users and items.

  * •

Finally, GNMR and MBGCN outperform RGCN by considering the contribution of
each behavior in the multi-behavioral fusion step. Compared to NMTR and MBGCN,
which only propose parallel learning paradigms during behavior fusion, CRGCN
and MB-CGCN explicitly incorporate the cascade relationships of multiple
behaviors during the fusion step, achieving performance that is second only to
our model. This indicates the necessity of considering hierarchical
correlation between behaviors.

###  5.3. Ablation Study

####  5.3.1. Impact of the Key Components

Table 3. Performances of different PKEF variants.

Model | Beibei | Taobao | Tmall
---|---|---|---
HR | NDCG | HR | NDCG | HR | NDCG
Base Model | 0.0734 | 0.0360 | 0.0849 | 0.0452 | 0.0829 | 0.0466
PKEF w/o PKF | 0.1096 | 0.0560 | 0.1121 | 0.0610 | 0.1205 | 0.0687
PKEF w/o PME | 0.0915 | 0.0460 | 0.0955 | 0.0532 | 0.0931 | 0.0514
PKEF | 0.1130 | 0.0582 | 0.1385 | 0.0785 | 0.1277 | 0.0721

To evaluate the effectiveness of sub-modules in our PKEF framework, we
consider three model variants: (1) Base Model: We remove both the PKF and PME
parts, so that the model only has the cascade stream and utilizing a bilinear
paradigm (separated input with a light-weight matrix transformation); (2) PKEF
w/o PKF: The PKF part is removed; (3) PKEF w/o PME: The PME part is replaced
with bilinear module. The performance of PKEF and its variants are summarized
in Table 3, and we come to these conclusions:

  * •

Comparing the performance of PKEF and its last two variants, we can find that
each variant brings about performance degradation when any key component is
removed or replaced with other modules. This demonstrates the rationality and
effectiveness of the two key designations.

  * •

It is worthwhile noticing that Base Model achieves the worst performance on
all three datasets compared to other variants with multi-behavior learning. In
particular, this variant has a performance decline up to 35.04%, 38.70%, and
35.08% in terms of HR (38.14%, 42.42%, and 35.37% in terms of NDCG) on Beibei,
Taobao, and Tmall datasets. This further demonstrates the effectiveness of the
combination of PKF and PME for solving the multi-behavior recommendation
problem.

####  5.3.2. Impact of the Knowledge Fusion Schemes

Table 4. Performances of different knowledge fusion schemes.

Model | Beibei | Taobao | Tmall
---|---|---|---
HR | NDCG | HR | NDCG | HR | NDCG
Summation | 0.0312 | 0.0173 | 0.1142 | 0.0635 | 0.0238 | 0.0127
Linear Trans. | 0.0847 | 0.0419 | 0.1185 | 0.0666 | 0.1084 | 0.0617
Vanilla Fusion | 0.1105 | 0.0568 | 0.1254 | 0.0707 | 0.1180 | 0.0674
Projection Fusion | 0.1130 | 0.0582 | 0.1385 | 0.0785 | 0.1277 | 0.0721

To further explore the forms of knowledge fusion between the parallel and
cascade streams, we devise two alternative schemes for general usage
(Illustrated in Section 4.2.2). Besides, we make a comparison between the
proposed two schemes with simple Summation (simply add the representation of
different streams up) and Linear Trans. (apply a linear transformation to
transfer the parallel knowledge). And as shown in Table 5.3.2, we can observe
that summation perform the worst among the four schemes. A probable reason is
that the distribution of the two representations of the streams is completely
different. So, simple summation may cause the harmful impact to the
distribution of representations. Besides, Linear Trans. leverage a implicit
way to transfer the knowledge, which may lead to a negative information
transfer when transfering the parallel knowledge. Vanilla Fusion weights the
fusion representations at different scales, alleviating the impact of
representation distribution. While the Projection Fusion utilizes a projection
mechanism to explicitly extract the useful information from the parallel
knowledge, and thus obtain the best performance on these three datasets.

####  5.3.3. Impact of the MTL module

To further demonstrate the superiority of our proposed PME in Multi-Task
Learning (MTL), we compare it with four state-of-the-art MTL models: Shared
Bottom (Caruana, 1997), Bilinear (Chen et al., 2021), MMOE (Ma et al., 2018),
and PLE (Tang et al., 2020). These models are applied on top of PKF for multi-
behavior recommendation. To ensure compatibility with the classical MTL models
(i.e., Shared Bottom, MMOE, and PLE), which expect the same input
representation, we weigh the $K$ separate representations generated by PKF to
obtain a unified input. The resulting variants are named PKF+SB, PKF+Bilinear,
PKF+MMOE, PKF+PLE, and PKF+PME. Table 5 summarizes the results. PKF+SB
performs the worst among all MTL models across all datasets. PKF+Bilinear,
which replaces the neural network’s prediction head with a light-weight matrix
transformation, shows better performance, which is likely due to reduced risk
of overfitting. Both PKF+MMOE and PKF+PLE employ gate networks with adaptive
attention weights for information fusion, outperforming the static and equally
weighted PKF+SB. Notably, our PME consistently outperforms all other models on
all datasets, reaffirming its effectiveness for MTL tasks.

Table 5. Performances of different MTL module.

Model | Beibei | Taobao | Tmall
---|---|---|---
HR | NDCG | HR | NDCG | HR | NDCG
PKF+SB | 0.0599 | 0.0287 | 0.0715 | 0.0381 | 0.0840 | 0.0472
PKF+Bilinear | 0.0915 | 0.0460 | 0.0955 | 0.0532 | 0.0931 | 0.0514
PKF+MMOE | 0.0830 | 0.0420 | 0.1091 | 0.0580 | 0.0890 | 0.0487
PKF+PLE | 0.0845 | 0.0431 | 0.1022 | 0.0545 | 0.0944 | 0.0517
PKF+PME | 0.1130 | 0.0582 | 0.1385 | 0.0785 | 0.1277 | 0.0721

Figure 4. Impact of GNN Layers for different behaviors.

###  5.4. Parameter Analysis

####  5.4.1. Impact of the number of layers

We investigate the impact of higher-order information on model performance by
varying the number of GNN layers. Specifically, we search the layer numbers in
the range of {1, 2, 3, 4} and use different numbers of layers for different
behaviors. The experimental results are shown in Figure 4, where the numbers
on each block indicate the layer for the buy behavior that achieves the best
performance while keeping the layer numbers fixed for the view and cart
behaviors. Due to lack of space, we only show the results on Beibei and
Taobao, the results of another dataset are similar.

Based on the results, it is evident that for both datasets, PKEF demonstrates
the highest performance when the GNN layers are configured as (4, 1, 1).
Furthermore, the influence of stacking different numbers of layers on
performance varied for different behaviors. There is a tendency to use deeper
propagation layers for the graph of view and shallower layers for downstream
behaviors such as buy. One possible reason is that the view behavior contains
richer interaction information and requires stacking more layers to capture
higher-order information for learning better user preferences. Whereas in
downstream behaviors with sparse interactions, excessive layers may introduce
noise and lead to overfitting.

####  5.4.2. Impact of the coefficients of different behaviors

We investigate the impact of the behavioral coefficient parameter
$\lambda_{k}$ on the performance of PKEF. There are three behavior types in
Beibei and Taobao (view, cart, and buy), which means there are three loss
coefficients $\lambda_{1}$, $\lambda_{2}$, and $\lambda_{3}$, respectively.
The value of $\lambda_{3}$ is determined when $\lambda_{1}$ and $\lambda_{2}$
are given. We use grid search in the range {0, 1/6, 2/6, 3/6, 4/6, 5/6, 1} and
plot the results for NDCG@10 (shown in Figure 5). For both datasets, PKEF
achieves the best performance with coefficient parameters set to (0, 4/6,
2/6), and the performance remains relatively consistent across different
parameters. This indicates that the model can effectively adapt to different
data distributions and has good generalization ability. The results of the
Tmall dataset, which we omit due to space constraints, reach similar
conclusions.

Figure 5. Impact of the Behavioral Coefficients.

###  5.5. Indepth Analysis

####  5.5.1. Case Study under Different Behavioral Correlations

We experimentally verify whether our model can alleviate potential gradient
conflicts. Specifically, we divide the test users into five user groups
according to the average Pearson correlation among all behaviors and select
subsets from each user group. To prevent the node degree from potentially
influencing the results (Wang et al., 2019), we keep similar average number of
user interactions among different subsets while maximizing the number of users
in each subset. For more rigorous results, we run experiments 5 times on each
dataset and plot the mean and fluctuation range on the figure. The
experimental results on Beibei and Taobao datasets are shown in Figure 6. We
find that PME consistently outperforms all other MTL methods across all user
groups, further demonstrating the superiority of PME for MTL. Additionally,
with the increase of Pearson correlation, the performance of PME grows more
rapidly compared to other MTL methods, while other MTL methods even show
fluctuations and decline. A possible reason is the negative transfer caused by
potential gradient conflicts when knowledge is transferred across different
tasks. We omit the results on the Tmall dataset due to space limitations,
which have consistent conclusions.

Figure 6. Average performances for user groups with different behavior
correlations.

####  5.5.2. Visualization of Gating Aggregation

We conduct experiments to compare the expert utilization between our PME model
and other gate-based models (MMOE and PLE). By visualizing the average weight
distribution of experts used for predicting the target behavior (shown in
Figure 7), we observe that PME achieves better differentiation among experts
compared to MMOE and PLE. We exclude gates used for other behaviors in our
analysis to solely focus on predicting the interaction probability of the
target behavior. Besides, in order to ensure the fairness of the comparison,
for MMOE and PLE, we fix the number of experts in Tmall to 4, and 3 in Beibei
and Taobao. While MMOE and PLE exhibit a nearly uniform distribution of
weights across all experts, PME selectively leverage information from
different behaviors, thereby avoiding potential conflicts. This demonstrates
the effectiveness of PME in utilizing diverse behavior information and
improving overall performance.

Figure 7. Expert utilization in gate-based models

Expert utilization in gate-based models

##  6\. Conclusion

In this paper, we propose the Parallel Knowledge Enhancement based Framework
(PKEF) for multi-behavior recommendation. To handle the problems of the
existing multi-behavior approches, we devise Parallel Knowledge Fusion (PKF)
module and Projection Disentangling Multi-Experts network (PME). PKF combines
cascade and parallel paradigms to enhance behavior representations, addressing
information bias caused by imbalanced behavioral interactions. PME treats each
behavior as an independent task, generating specific expert information for
each behavior using separate inputs. Besides, for each behavior, it leverages
a projection mechanism to disentangle the shared and specific parts from other
behaviors and aggregates the shared part while designing an auxiliary loss to
further utilize the unique part. Thus, the negative transfer is significantly
alleviated. Further, we perform extensive experiments on three real-world
datasets to validate the effectiveness of our PKEF. The results provide
further evidence of the rationale and effectiveness of the designed PKF and
PME modules.

###### Acknowledgements.

This work was partly supported by the Science and Technology Innovation
2030-Key Project (Grant No. 2021ZD0201404), Key Technology Projects in
Shenzhen (Grant No. JSGG20220831110203007), Shenzhen Key Laboratory of next
generation interactive media innovative technology (Grant No.
ZDSYS20210623092001004) and Aminer·ShenZhen·ScientificSuperBrain.

## References

  * (1)
  * Abadi et al. (2016) Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. 2016\.  $\\{$TensorFlow$\\}$: a system for $\\{$Large-Scale$\\}$ machine learning. In _12th USENIX symposium on operating systems design and implementation (OSDI 16)_. 265–283.
  * Bian et al. (2021) Zhi Bian, Shaojun Zhou, Hao Fu, Qihong Yang, Zhenqi Sun, Junjie Tang, Guiquan Liu, Kaikui Liu, and Xiaolong Li. 2021.  Denoising user-aware memory network for recommendation. In _Fifteenth ACM Conference on Recommender Systems_. 400–410.
  * Caruana (1997) Rich Caruana. 1997\.  Multitask learning.  _Machine learning_ 28, 1 (1997), 41–75.
  * Chen et al. (2021) Chong Chen, Weizhi Ma, Min Zhang, Zhaowei Wang, Xiuqiang He, Chenyang Wang, Yiqun Liu, and Shaoping Ma. 2021\.  Graph Heterogeneous Multi-Relational Recommendation. In _AAAI_ , Vol. 35. 3958–3966.
  * Chen et al. (2020) Lei Chen, Le Wu, Richang Hong, Kun Zhang, and Meng Wang. 2020.  Revisiting Graph Based Collaborative Filtering: A Linear Residual Graph Convolutional Network Approach. In _AAAI_ , Vol. 34. 27–34.
  * Cheng et al. (2023) Zhiyong Cheng, Sai Han, Fan Liu, Lei Zhu, Zan Gao, and Yuxin Peng. 2023.  Multi-Behavior Recommendation with Cascading Graph Convolution Networks. In _Proceedings of the ACM Web Conference 2023_. 1181–1189.
  * Ding et al. (2018) Jingtao Ding, Guanghui Yu, Xiangnan He, Yuhan Quan, Yong Li, Tat-Seng Chua, Depeng Jin, and Jiajie Yu. 2018\.  Improving Implicit Recommender Systems with View Data.. In _IJCAI_. 3343–3349.
  * Gao et al. (2019a) Chen Gao, Xiangnan He, Dahua Gan, Xiangning Chen, Fuli Feng, Yong Li, Tat-Seng Chua, and Depeng Jin. 2019a.  Neural multi-task recommendation from multi-behavior data. In _ICDE_. IEEE, 1554–1557.
  * Gao et al. (2019b) Chen Gao, Xiangnan He, Dahua Gan, Xiangning Chen, Fuli Feng, Yong Li, Tat-Seng Chua, Lina Yao, Yang Song, and Depeng Jin. 2019b.  Learning to recommend with multiple cascading behaviors.  _IEEE transactions on knowledge and data engineering_ 33, 6 (2019), 2588–2601.
  * Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010.  Understanding the difficulty of training deep feedforward neural networks. In _AISTATS_. 249–256.
  * Guo et al. (2017) Guibing Guo, Huihuai Qiu, Zhenhua Tan, Yuan Liu, Jing Ma, and Xingwei Wang. 2017\.  Resolving data sparsity by multi-type auxiliary implicit feedback for recommender systems.  _Knowledge-Based Systems_ 138 (2017), 202–207.
  * Guo et al. (2019) Long Guo, Lifeng Hua, Rongfei Jia, Binqiang Zhao, Xiaobo Wang, and Bin Cui. 2019\.  Buying or browsing?: Predicting real-time purchasing intent using attention-based deep network with multiple behavior. In _SIGKDD_. 1984–1992.
  * Guo et al. (2023) Wei Guo, Chang Meng, Enming Yuan, Zhicheng He, Huifeng Guo, Yingxue Zhang, Bo Chen, Yaochen Hu, Ruiming Tang, Xiu Li, et al. 2023\.  Compressed Interaction Graph based Framework for Multi-behavior Recommendation. In _Proceedings of the ACM Web Conference 2023_. 960–970.
  * He et al. (2020) Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020\.  LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation.  _arXiv preprint arXiv:2002.02126_ (2020).
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017\.  Neural collaborative filtering. In _WWW_. 173–182.
  * He et al. (2023) Zhicheng He, Weiwen Liu, Wei Guo, Jiarui Qin, Yingxue Zhang, Yaochen Hu, and Ruiming Tang. 2023.  A Survey on User Behavior Modeling in Recommender Systems.  _arXiv preprint arXiv:2302.11087_ (2023).
  * Huang (2021) Chao Huang. 2021\.  Recent Advances in Heterogeneous Relation Learning for Recommendation.  _arXiv preprint arXiv:2110.03455_ (2021).
  * Jacobs et al. (1991) Robert A Jacobs, Michael I Jordan, Steven J Nowlan, and Geoffrey E Hinton. 1991.  Adaptive mixtures of local experts.  _Neural computation_ 3, 1 (1991), 79–87.
  * Jin et al. (2020) Bowen Jin, Chen Gao, Xiangnan He, Depeng Jin, and Yong Li. 2020.  Multi-behavior recommendation with graph convolutional networks. In _SIGIR_.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ (2014).
  * Krohn-Grimberghe et al. (2012) Artus Krohn-Grimberghe, Lucas Drumond, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2012\.  Multi-relational matrix factorization using bayesian personalized ranking for social network data. In _Proceedings of the fifth ACM international conference on Web search and data mining_. 173–182.
  * Liang et al. (2023) Ke Liang, Yue Liu, Sihang Zhou, Wenxuan Tu, Yi Wen, Xihong Yang, Xiangjun Dong, and Xinwang Liu. 2023\.  Knowledge Graph Contrastive Learning Based on Relation-Symmetrical Structure.  _IEEE Transactions on Knowledge and Data Engineering_ (2023).
  * Loni et al. (2016) Babak Loni, Roberto Pagano, Martha Larson, and Alan Hanjalic. 2016.  Bayesian personalized ranking with multi-channel user feedback. In _Proceedings of the 10th ACM Conference on Recommender Systems_. 361–364.
  * Ma et al. (2018) Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi. 2018\.  Modeling task relationships in multi-task learning with multi-gate mixture-of-experts. In _SIGKDD_. 1930–1939.
  * Meng et al. (2023) Chang Meng, Hengyu Zhang, Wei Guo, Huifeng Guo, Haotian Liu, Yingxue Zhang, Hongkun Zheng, Ruiming Tang, Xiu Li, and Rui Zhang. 2023\.  Hierarchical Projection Enhanced Multi-Behavior Recommendation. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 4649–4660.
  * Meng et al. (2022) Chang Meng, Ziqi Zhao, Wei Guo, Yingxue Zhang, Haolun Wu, Chen Gao, Dong Li, Xiu Li, and Ruiming Tang. 2022.  Coarse-to-Fine Knowledge-Enhanced Multi-Interest Learning Framework for Multi-Behavior Recommendation.  _arXiv preprint arXiv:2208.01849_ (2022).
  * Qiu et al. (2018) Huihuai Qiu, Yun Liu, Guibing Guo, Zhu Sun, Jie Zhang, and Hai Thanh Nguyen. 2018\.  BPRH: Bayesian personalized ranking for heterogeneous implicit feedback.  _Information Sciences_ 453 (2018), 80–98.
  * Rendle et al. (2012) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2012.  BPR: Bayesian personalized ranking from implicit feedback.  _arXiv preprint arXiv:1205.2618_ (2012).
  * Schlichtkrull et al. (2018) Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018.  Modeling relational data with graph convolutional networks. In _European semantic web conference_. Springer, 593–607.
  * Singh and Gordon (2008) Ajit P Singh and Geoffrey J Gordon. 2008.  Relational learning via collective matrix factorization. In _Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_. 650–658.
  * Su and Khoshgoftaar (2009) Xiaoyuan Su and Taghi M Khoshgoftaar. 2009.  A survey of collaborative filtering techniques.  _Advances in artificial intelligence_ 2009 (2009).
  * Tang et al. (2020) Hongyan Tang, Junning Liu, Ming Zhao, and Xudong Gong. 2020\.  Progressive layered extraction (ple): A novel multi-task learning (mtl) model for personalized recommendations. In _RecSys_. 269–278.
  * Tang et al. (2016) Liang Tang, Bo Long, Bee-Chung Chen, and Deepak Agarwal. 2016\.  An empirical study on recommendation with multiple types of feedback. In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_. 283–292.
  * Torrey and Shavlik (2010) Lisa Torrey and Jude Shavlik. 2010.  Transfer learning.  In _Handbook of research on machine learning applications and trends: algorithms, methods, and techniques_. IGI global, 242–264.
  * Wang et al. (2019) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019.  Neural graph collaborative filtering. In _SIGIR_. 165–174.
  * Xia et al. (2021a) Lianghao Xia, Chao Huang, Yong Xu, Peng Dai, Mengyin Lu, and Liefeng Bo. 2021a.  Multi-Behavior Enhanced Recommendation with Cross-Interaction Collaborative Relation Modeling. In _ICDE_. IEEE, 1931–1936.
  * Xia et al. (2020) Lianghao Xia, Chao Huang, Yong Xu, Peng Dai, Bo Zhang, and Liefeng Bo. 2020.  Multiplex behavioral relation learning for recommendation via memory augmented transformer network. In _SIGIR_. 2397–2406.
  * Xia et al. (2021b) Lianghao Xia, Chao Huang, Yong Xu, Peng Dai, Xiyue Zhang, Hongsheng Yang, Jian Pei, and Liefeng Bo. 2021b.  Knowledge-enhanced hierarchical graph transformer network for multi-behavior recommendation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 4486–4493.
  * Xia et al. (2021c) Lianghao Xia, Yong Xu, Chao Huang, Peng Dai, and Liefeng Bo. 2021c.  Graph meta network for multi-behavior recommendation. In _SIGIR_. 757–766.
  * Yan et al. (2022) Mingshi Yan, Zhiyong Cheng, Chen Gao, Jing Sun, Fan Liu, Fuming Sun, and Haojie Li. 2022.  Cascading Residual Graph Convolutional Network for Multi-Behavior Recommendation.  _arXiv preprint arXiv:2205.13128_ (2022).
  * Yu et al. (2023) Chengqing Yu, Guangxi Yan, Chengming Yu, Yu Zhang, and Xiwei Mi. 2023.  A multi-factor driven spatiotemporal wind power prediction model based on ensemble deep graph attention reinforcement learning networks.  _Energy_ 263 (2023), 126034.
  * Zhao et al. (2015) Zhe Zhao, Zhiyuan Cheng, Lichan Hong, and Ed H Chi. 2015\.  Improving user topic interest profiles by behavior factorization. In _Proceedings of the 24th International Conference on World Wide Web_. 1406–1416.
  * Zhou et al. (2018) Chang Zhou, Jinze Bai, Junshuai Song, Xiaofei Liu, Zhengchao Zhao, Xiusi Chen, and Jun Gao. 2018.  Atrank: An attention-based user behavior modeling framework for recommendation. In _Thirty-Second AAAI Conference on Artificial Intelligence_.
  * Zhou et al. (2023) Yang Zhou, Zhuojia Yang, Qiang Sun, Chengqing Yu, and Chengming Yu. 2023.  An artificial intelligence model based on multi-step feature engineering and deep attention network for optical network performance monitoring.  _Optik_ 273 (2023), 170443.