# Graph Exploration Matters: Improving both individual-level and system-level
diversity in WeChat Feed Recommender

Shuai Yang  WeChat Search Department, TencentShenzhenChina
<EMAIL_ADDRESS>,  Lixin Zhang  WeChat Search Department,
TencentShenzhenChina<EMAIL_ADDRESS>,  Feng Xia  WeChat Search
Department, TencentBeijingChina<EMAIL_ADDRESS>and  Leyu Lin  WeChat
Search Department, TencentBeijingChina<EMAIL_ADDRESS>
(2018)

###### Abstract.

There are roughly three stages in real industrial recommendation systems,
candidates generation (retrieval), ranking and reranking. Individual-level
diversity and system-level diversity are both important for industrial
recommender systems. The former focus on each single user’s experience, while
the latter focus on the difference among users. Graph-based retrieval
strategies are inevitably hijacked by heavy users and popular items, leading
to the convergence of candidates for users and the lack of system-level
diversity. Meanwhile, in the reranking phase, Determinantal Point Process
(DPP) is deployed to increase individual-level diverisity. Heavily relying on
the semantic information of items, DPP suffers from clickbait and inaccurate
attributes. Besides, most studies only focus on one of the two levels of
diversity, and ignore the mutual influence among different stages in real
recommender systems. We argue that individual-level diversity and system-level
diversity should be viewed as an integrated problem, and we provide an
efficient and deployable solution for web-scale recommenders. Generally, we
propose to employ the retrieval graph information in diversity-based
reranking, by which to weaken the hidden similarity of items exposed to users,
and consequently gain more graph explorations to improve the system-level
diveristy. Besides, we argue that users’ propensity for diversity changes over
time in content feed recommendation. Therefore, with the explored graph, we
also propose to capture the user’s real-time personalized propensity to the
diversity. We implement and deploy the combined system in WeChat App’s Top
Stories used by hundreds of millions of users. Offline simulations and online
A/B tests show our solution can effectively improve both user engagement and
system revenue.

Diversified Recommendation, Graph-based Retrieval, Determinantal Point Process

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computer systems organization
Embedded systems††ccs: Computer systems organization Redundancy††ccs: Computer
systems organization Robotics††ccs: Networks Network reliability

Figure 1. Recommendation stages in WeChat Top Stories.

overview

##  1\. Introduction

Figure 2. Both individual-level diversity and system-level diversity are
important for content feed recommenders.

two-level of diversity

Industrial recommender systems are widely deployed in many online services
(Davidson et al., 2010; Baracskay et al., 2022; Sarwar et al., 2001; Li et
al., 2008, 2019b). WeChat, as the largest social instant messaging platform in
China, also provides content feed recommendation services named Top Stories.
Videos and articles are organized as a feed to be presented. Generally, most
real-world recommender systems follow a standard pipeline that can by divided
into three modules, namely retrieval, ranking and reranking (Covington et al.,
2016; Wang et al., 2020b). The main objective of the systems is to select top
k items most relevant to the users. However, the output list usually contains
repeated similar items, which makes users reluctant to explore the system
especially in content feed scenario. Besides, those highly ranked items
usually enjoy high popularity, and the diversity of each user’s behavior is
reduced as well. Consequently, it’s less possible for new items or those with
lower popularity to be exposed, and this eventually leads to the system-level
convergence. Since the it influences both the user-level experience and the
system-level diversity, many approaches are proposed to improve the diversity
of recommender systems.

An intuitive method is to optimize the diversity in reranking layer, where
ranked items are formed as the final feed exposed to users. Many algorithms,
such as Maximal Marginal Relevance (MMR) (Carbonell and Goldstein, 1998) and
Determinantal Point Process (DPP) (Ashkan et al., 2015; Chen et al., 2018;
Wang et al., 2020a; Wilhelm et al., 2018), are widely studied and deployed in
real-world recommenders. DPP-based model construct a kernel matrix that
contains pair-wise similarity and relevance, and shows more effectiveness and
efficiency than MMR. In real-world data, different users have different
diversity needs. Studies show that personalizing the diversity preference
could obtain online user engagement. Recent work (Wang et al., 2020a) adopts
behavior entropy to quantify user’s propensity to diversity, and provides a
fine-grained method to assign a personalized diversity factor to each user.

However, we observe that those approaches (Chen et al., 2018; Wang et al.,
2020a; Wilhelm et al., 2018) usually rely on the semantic information of
items, which cannot fully reveal the relevance of items, and it may fail when
the category information is not accurate. The entropy-based personalization
also views pair-wise distance of category as the same. While in real-world
classification, ”fashion” and ”music” are much more close than ”fashion” and
”technology”. Besides, by analizing the users behavior in WeChat Top Stories,
we observe that users’ preference to diversity changes over time. An example
is in Figure 3. This may caused by individual interest shift or the
distribution change of item pool, which are common cases in real-world
recommenders.

Another school of thought is to improve the diversity in the retrieval stage.
These methods focus on the user-wise diversity for long-term benefits. Unlike
DNN-based methods (Li et al., 2019b; Zhang et al., 2022) focusing on training
dissimilar user representations, GNN-based methods (He et al., 2020; Wang et
al., 2019; Yu et al., 2022; Xie et al., 2022; Yang et al., 2023) are devoted
to learn the connections between users and items. However, without a end-to-
end view of the whole system, those diversified retrieval methods could be
very weak to the final diversity to the users, because the top-ranked items in
the following stages are usually flooded by those with high popularity.

In this paper, we aim to improve both individual-level diversity and system-
level diversity in a personalized solution. In order to overcome the
limitations of existing methods, we propose a end-to-end diversification
framework by making full use of graph exploration. Generally, we first adopt
graph-based embedding retrieval model to match thousands of items from item
pool. Then the retrieved items are sifted to dozens by the ranking stage.
Based on the real-time recorded user behaviors and the graph embeddings of
items, a designed function is applied to capture user’s real-time propensity
to the degree of diversity. In reranking stage, we view the ranked items as
part of the huge graph in retrieval stage. We employ the graph embedding again
to describe the similarity of ranked items. The diversified reranking problem
is then converted to selecting a sub-graph with the best trade-off between
relevance and diversity. With the real-time personalized diversity factor, the
ranked scores and the graph-based similarity matrix, a kernel matrix can be
generated for DPP to improve individual-level diversity. The selected sub-
graph, is then formed as a feed for users to explore edges between diversified
items. These graph exploration, in return, are merged to the huge graph to
improve the system-level diversity. Since the graph-based embeddings are
trained once and used multiple times in this system, our solution is very
practical to be deployed in homogeneous feed recommenders. We conduct both
offline simulations and online evaluations on WeChat Topstories, which widely
serves hundreds of millions of users everyday.

The main contributions are concluded as follows:

  * •

We observe and analysis that user’s preference for diversity is not immutable
but keep changing. We propose a function to quantify user’s real-time
propensity to diversity.

  * •

We highlight the importance of both individual-level diversity and the system-
level diversity, and argue the effectiveness of end-to-end graph exploration
to improve the diversity in two levels.

  * •

We summarize our work as a framework and implement it in a frugal manner,
which has been deployed on a real-world recommendation system used by millions
of users. Our work is reusable and efficient to be transplanted.

##  2\. Related Work

Throughout the evolution of recommender systems, it went through shallow stage
(Koren et al., 2009; Rendle, 2010; Rendle et al., 2009), neural network stage
(Cheng et al., 2016; He et al., 2017; Guo et al., 2017), and the GNN stage (He
et al., 2020; Wang et al., 2019; Yu et al., 2022) to achieve higher accuracy.
Recommendation diversity is another essential factor that impacts user
experience, which is widely studied in real-world recommenders. For better
comparison with our work, we summarize the use of GNN in different phases of
recommendation (Gao et al., 2023), and then we elaborate on diversified
solutions for recommendation.

###  2.1. GNN in Recommender Systems

In real-world recommenders, GNN is adopted in different stages to act
different roles. In retrieval phase, GNN is usually employed as the embedding
trainer. The graphs can be roughly divided into user-item (He et al., 2020;
Wang et al., 2019) and item-item (Ying et al., 2018; Hamilton et al., 2017)
graphs. The graph architecture is usually constructed by analysing user’s
historical behaviors. These GNN-based models can capture similarity among
users and items by multiple times of information propagation. Specifically,
GraphSage (Hamilton et al., 2017) has shown its capability to be deployed in
web-scale recommenders (Ying et al., 2018). Combined with random walk,
GraphSage conducts an inductive representation learning on large item-item
graphs. It samples neighbors of the target node, aggregates and merges their
embeddings with the target node to update the representations.

Recent works also apply GNN in ranking stage, where accuracy is of first
importance. GNN is employed to produce dense embeddings of the input features,
and then the predictor use those embeddings to achieve ranked scores. These
methods (Li et al., 2019a; Wu et al., 2022) focus on the interaction among
features, which is hard to capture. Compared with DNN-based methods, GNN-based
ranking methods are seldom adopted in real-world feed recommendation due to
the constrained accuracy and the difficulty to be deployed.

Few works adopt GNN in reranking stage, where multiple goals other than
accuracy should be considered. IRGPR (Liu et al., 2020) considers both item
relationships and user preferences, incorporating one item relation graph to
capture item relationships with another user-item scoring graph to embed
personalized user intents into the propagation. However, it ignores the
diversity in reranking, which is one of the most important goal in reranking
stage.

###  2.2. Diversified Recommendation

Figure 3. Three users’ propensity to the degree of diversity changing in 10
days. Values are $f_{u}^{t}$ explained in section 3.2. A larger value means
user prefers more similar items.

user’s preference for diversity changes over time.

Recommendation diversity should be measured in two typical dimensions:
individual-level diversity and system-level diversity (Ge et al., 2010;
Herlocker et al., 2004; Gao et al., 2023). Individual-level diversity measures
the dissimilarity of the recommended items for each user, whereas system-level
diversity measures the dissimilarity of recommendation results of different
users. Both levels of diversity are important and should be considered as an
integrated problem.

In order to improve individual-level diversity, diversified reranking has been
widely studied. Maximal Marginal Relevance (MMR) (Carbonell and Goldstein,
1998) method considers both the relevance and the pair-wise similarity of
ranked items by point-wise rerank. DPP is an elegant probabilistic model with
the ability to express negative interactions. DPP-based models (Huang et al.,
2021; Ashkan et al., 2015; Chen et al., 2018; Wang et al., 2020a; Wilhelm et
al., 2018) construct a kernel matrix that contains pair-wise similarity and
relevance, and shows more effectiveness and efficiency than MMR. Fast greedy
maximum a posteriori inference for DPP (Fast-DPP) (Chen et al., 2018) makes it
easy to be deployed in web-scale recommender systems. In real-world data,
different users have different diversity needs (Wu et al., 2018; Chen et al.,
2017). Studies (Noia et al., 2014; Eskandanian et al., 2017) show that
personalizing the diversity preference could obtain online user engagement.
Based on DPP, (Wang et al., 2020a) adopts behavior entropy to quantify user’s
propensity to diversity, and provides a fine-grained method to assign a
personalized diversity factor to each user. It relies heavily on the pre-
defined semantic information, which cannot reveal the interaction similarity.
In our work, we observe that users’ preference to diversity changes over time.
This may caused by individual interest shifting or the distribution change of
item pool, which are common cases in real-world recommenders. Our approach
provides a function to capture both the user interests shifting and the item
pool change, and simply adopts Fast-DPP in reranking stage to improve
individual-level diversity.

However, as the last strategy before exposure, diversified reranking methods
focus on improving item-wise diversity showed to users but overlook the
system-level diversity, which is much more important for long-term gain. In
fact, retrieval should take more responsibility for diversity, since it
emphasizes the coverage of user-interested items rather than their specific
item ranks. Luckily, some works have been studied to improve the retrieval
diversity. For example, DNN-based method MIND (Li et al., 2019b) adopts a
multi-interest extractor layer with a variant dynamic routing in neural
network to extract user’s diverse interests. GNN-based methods are also
modified to capture long-tail items in graph. GraphDR (Xie et al., 2022)
builds a huge heterogeneous preference network to record different types of
user preferences, and conduct a fieldlevel heterogeneous graph attention
network for node aggregation. Both DGCN (Zheng et al., 2021) and DGRec (Yang
et al., 2023) focus on rebalancing the categories of retrieved items, which
also relies heavily on the pre-defined semantic information. As mentioned
previously, these graph-based models are hard to be deployed in ranking or
reranking stage, so in real-world web-scale recommeders they can just serve in
retrieval stage.

These methods focuses on either individual-level diversity or system-level
diversity separately. Our approach views the individual-level diversity and
the system-level diversity as an integrated problem, and attempts to solve it
in a simple yet effective solution. We make full use of the item-item graph
embedding in retrieval stage to capture use’s real-time preference to
diversity, reveal the interactive similarity of ranked items in reranking
stage. A sub-graph with the best trade-off between relevance and diversity is
generated by real-time personalized DPP, and the path is explored by real-
world users. Those graph exploration are then merged in the huge item-item
graph to increase the system-level diversity.

Figure 4. The overview of our solution for two levels of diversity. We
integrate the retrieval stage and the reranking stage by using the item-item
graph.

A woman and a girl in white dresses sit in an open car.

##  3\. METHOD

###  3.1. GNN-based Retrieval

GraphSage (Ying et al., 2018) allows embeddings to be efficiently generated
for unseen nodes by leveraging node feature information. It has been
successfully applied to web-scale recommender systems to compute embedddings
for billions nodes. It samples neighbors of one node, aggregates neighbourhood
embeddings to update node representations.

(1) $\displaystyle\mathbf{h}_{\mathcal{N}_{i}}^{l}$
$\displaystyle=\operatorname{AGGREGATE}_{l}\left(\left\\{\mathbf{h}_{j}^{l},\forall
j\in\mathcal{N}_{i}\right\\}\right),$ $\displaystyle\mathbf{h}_{i}^{l+1}$
$\displaystyle=\delta\left(\mathbf{W}^{l}\left[\mathbf{h}_{i}^{l}\|\mathbf{h}_{\mathcal{N}_{i}}^{l}\right]\right)$

where The embedding of graph node $i$ in the $l$-th propagation layer,
$\mathcal{N}_{i}$ is the neighborhood set of node $i$, $\delta$ is the
activation function, $\mathbf{W}$ is trainable matrix ,$\|$ is the
concatenation operation. In our work, we view the items as nodes in the graph,
and the edge is connected if two items are positively interacted in one user’s
behavior session. Item attributes are used as node features. Random walk is
adopted to sample neighbours of each node. Then we use the unsupervised
GraphSage to generate the embeddings of all the items. Based on user’s
historical positive interacted items, we can easily match the potential items
by computing the cosine similarity of item embeddings.

###  3.2. Real-time Personalization

With the observation of the temporal change of user’s preference to
individual-level diversity, we propose a scheme to capture user’s real-time
propensity to the degree of diversity, which is personalized as well.

First, a simple yet effective temporal sliding window is applied to filter
user’s recent positively interacted items $\mathbf{I}_{u}^{t}$. With the help
of the huge graph in retrieval stage, the similarity matrix
$\mathbf{S}_{\mathbf{I}_{u}^{t}}$ of these selected items from user $u$ can be
computed with normalized embeddings $\mathbf{e}_{u}^{t}$, and we use
$s_{\mathbf{u}}^{t}$ as the mean item similarity of user in time $t$:

(2)
$\displaystyle\mathbf{S}_{\mathbf{I}_{u}^{t}}=\left(\mathbf{e}_{u}^{t}*\mathbf{e}_{u}^{t}\top\right),$
$\displaystyle
s_{\mathbf{u}}^{t}=\operatorname{mean}_{i,j\in\mathbf{I}_{u}^{t},i\neq
j}\left(1-\mathbf{S}_{{\mathbf{I}_{u}^{t}}_{ij}}\right)$

Then $s_{\mathbf{u}}^{t}$ is updated in a vector $\mathbf{s}^{t}$, which
records all users’ real-time item similarity. We adopt z-score to standardize
all users’ $s_{\mathbf{u}}^{t}$, then sigmoid function is used to return a
value $f_{\mathbf{u}}^{t}$ in the range 0 to 1.

(3)
$f_{u}^{t}=\frac{1}{1+\exp\left(\frac{-(s_{u}^{t}-\bar{\mathbf{s}^{t}})}{\sigma_{\mathbf{s}^{t}}}\right)}.$

According to (Chen et al., 2013), user characteristics are correlated with
their historical behaviors in recommenders. We can simply make the assumption
that the conclusion that user $u$ with a large $f_{u}^{t}$ at time $t$ tends
to view more similar items. Thus this value can be viewed as user’s real-time
propensity to the degree of diversity. It will be more explainable when
combined with fast DPP in the next subsection.

Input: : Ranked items set: $Z$; Real-time personalized preference for
diversity of user $u$: $f_{u}^{t}$; Graph-based similarity matrix of ranked
items: $\mathbf{S}$; Ranked scores of items: $\mathbf{r}$; Output size $k$.

Output: Reranked top-k items set $Y$.

Compute intermediate scores $\mathbf{r}^{*}$ by equation (7).

Compute kernel matrix $\mathbf{L}$ by equation (6) using $\mathbf{r}^{*}$ and
$\mathbf{S}$ .

Initialize $Y=\\{\\}$.

while  _$|Z| >0$ and $|Y|<k$_ do

Select item $j$ by equation (8).

$Y\cup\\{j\\}$.

Return: $Y$.

Algorithm 1 Reranking Stage.

###  3.3. Reranking

It’s natural to adopt fast-DPP [11] in large sale recommendation systems for
efficiency. Positive semi-definite kernel matrix $\mathbf{L}$ can be generated
by using relevance scores $\mathbf{r}$ (usually the ranked score) and
normalized item embeddings $\mathbf{f}$ by

(4) $\displaystyle\mathbf{L}_{ij}=r_{i}r_{j}<\mathbf{f}_{i},\mathbf{f}_{j}>,$
$\displaystyle\mathbf{L}=Diag(\mathbf{r})*\mathbf{S}*Diag(\mathbf{r})$

where $\mathbf{S}$ is the similarity matrix of the ranked items. The log-
probability of selected item subset $Y$ can be written as

(5) $log\mathcal{P}(Y)\propto\theta*\sum\limits_{i\in
Y}r_{i}+(1-\theta)*log(det(\mathbf{S}_{Y})),\theta\in(0,1)$

where $det$ is the determinant of matrix, $\theta$ is the trade-off parameter
between relevance and diversity. A larger $\theta$ means more relevance of
results. When $\theta=0.5$, users obtain a equal importance between relevance
and diversity. The corresponding kernel with trade-off can be written as:

(6)
$\mathbf{L}^{{}^{\prime}}=Diag(exp(\alpha\mathbf{r}))*\mathbf{S}*Diag(exp(\alpha\mathbf{r})),\alpha=\theta/2(1-\theta).$

Study in (Chen et al., 2013) has shown that the historical behavior can
reflect user’s preference for diveristy. $f_{u}^{t}$ mentioned in section 3.2
forces the user similarity in a distribution ranging in 0 to 1, which follows
the fact that less users prefer either extremely more diversity or relevance .
Thus we use $f_{u}^{t}$ as the real-time trade-off parameter $\theta$ between
relevance and diveristy. In practice, the trade-off can be considered by
rewriting each ranked score as:

(7) $r^{*}_{i}=exp^{\frac{f_{u}^{t}}{2(1-f_{u}^{t})}*r_{i}},i\in Z.$

Figure 5. An illustration of graph exploration in proposed framework. An edge
between two nodes means they are both positively interacted by the same user
in one recommendation session. Shorter distance between items means higher
similarity.

A woman and a girl in white dresses sit in an open car.

Instead of using category-like semantic embedding of items as the input of
similarity computation, we again employ the embeddings from the huge item-item
graph to generate the similarity matrix $\mathbf{S}$ of ranked items. The
embeddings contain both item features information and their interacted
information. The former focuses on individual-level diversity, whereas the
latter is more important for system-level diversity.

Given the similarity matrix $\mathbf{S}$, the ranked scores $\mathbf{r}$ and
the real-time personalized diversity preference $f_{u}^{t}$, the kernel matrix
$\mathbf{L}$ can be generated. Fast-DPP converts problem of selecting subset
$Y$ from whole set $Z$ to greedily selecting item $j$ with the most promotion
to the determinants of the updated submatrix iteratively, since
$\mathcal{P}(Y)\propto det(\mathbf{L}_{Y})$:

(8) $j=\arg\max_{i\in Z\backslash
Y}\log\operatorname{det}\left(\mathbf{L}_{Y\cup\\{i\\}}\right)-\log\operatorname{det}\left(\mathbf{L}_{Y}\right)$

We formalize the deployed reranking stage in Algorithm 1.

###  3.4. Graph Exploration

With the components deconstructed above, we now explain how our framework
improve the individual-level diversity and system-level diversity in an
integrated manner. An example of top-5 items recommendation task is
illustrated in Figure 5.

Retrieval and ranking. The graph embeddings are online trained by using real-
time feedback from users. The graph embeddings are first used as a retrieval
strategy. After ranking, hundreds of selected items can be viewed as nodes in
the large graph. The ranked scores predict top-k items as the most relevant
items and usually very similar to user’s strong interest. Thus a higher
coverage of items in retrieval stage is needed.

Real-time personalized trade-off between relevance and diversity. As expounded
in section 3.2, combined with user’s history, $f_{u}^{t}$ is also generated by
the graph embeddings. This parameter contains user’s diversity information
based on the graph. It determines the degree of graph exploration.

Explore unseen items. The ranked items are not necessarily in the graph,
because in real-world recommenders there are dozens of retrieval strategies
other than graph-based retrieval. Luckily GraphSage can predict the unseen
items’ embedding by using the node features. In practice, the feature space is
usually the same as the existed graph nodes. Then the embedding can be
predicted without edges connected. Thus the unseen items can also be viewed as
part of the huge graph.

Explore unconnected edges. The graph is again used to generate the similarity
matrix of ranked items, which contains node features similarity and
interaction-level similarity. With the help of fast-DPP, top-k items are re-
selected. Since node features contain semantic and category-like information,
the diversified reranked items involve more semantic diversity, which improves
the individual-level diversity. Also, since the graph embedding contain
interaction-level similarity, items, usually together positively interacted by
the same user, are dispersed after DPP. The latter aims to improve the system-
level diversity by attempting to add new edges in the graph.

Graph update. The reranked items are then exposed to user as a feed. Items
with positive interaction in the same session are defined as correlated pairs,
and the edges are then added in the huge item-item graph. The recommender now
completes one round of graph exploration, and the graph is continuously
iterated after each recommendation lifecycle. In the next round, with the
added edges and nodes, the graph-based retrieval matches more diversified
items, which indicates a higher coverage of items and exactly improves the
system-level diversity Graph exploration can improve both levels of diversity
by leveraging the ”learning effect” mentioned in (Hohnhold et al., 2015).

###  3.5. Real-world Deployment

The graph-based retrieval model is implemented in C++ as an incremental
learning version to be easily updated. User’s behavior history are also
updated in a real-time manner. $s_{\mathbf{u}}^{t}$, which records all users’
real-time item similarity based on behavior history applied with a temporal
sliding window, is also updated in a real-time way. The size of the sliding
window can be defined as the space limit or the time limit. We find the latter
works better, because hot events often intensively change the data
distribution in content feed recommendation task. These real-time updated
parameters, together with the ranking models, are stored in online dict severs
to ensure the accuracy and diversity of the recommender system. The whole
framework reuse the graph embedding 3 times respectively in retrieval stage,
$f_{u}^{t}$ generation, and the similarity matrix generation in reranking
stage, which doesn’t rely on extra pre-trained embeddings or semantic
information. This is a saving solution, and it prevents the reranking DPP
suffering from clickbait or inaccurate categories, which is the common case in
content feed recommendation.

##  4\. Experimental results

Two stages of experiment are conducted in WeChat Top Stories videos
recommendation. We first conduct a 7-days online experiment to compare methods
with each other, and then the best method is launched for a long-term
observation. This is a frequently-used method in industrial recommenders. In
this section, we first prove the effectiveness of our work compared with
others, and then show the long-term gain since launched.

Table 1. Experimental Results in One Week method | IDL | RD | VV | Staytime | AD
---|---|---|---|---|---
Base | - | - | - | - | -
fast-DPP | +0.341 | * | +0.912 | * | +0.316
p-DPP | +0.367 | * | +1.117 | +0.391 | +0.684
ours | +2.003 | +4.678 | +2.397 | +0.725 | +1.143

Figure 6. An illustration of graph exploration in proposed framework. An edge
between two nodes means they are both positively interacted by the same user
in one recommendation session. Shorter distance between items means higher
similarity.

A woman and a girl in white dresses sit in an open car.

Baselines. Three baselines are compared in the 7 days online experiment. We
refer the existed rule-based diversified reranking as ”Base”, which limits the
number of items with the same categort or tags. Simply adopting fast-DPP (Chen
et al., 2018) based on the category-like similarity is referred as ”fast-DPP”.
The solution proposed in (Wang et al., 2020a) is p-DPP, which considers the
personal difference of propensity to diversity.

Metrics. We adopt intra-list average distance (ILAD) (Chen et al., 2018) to
measure the individual-level diversity, which is the average of all users’
mean distance of interacted items:

(9) $\operatorname{ILAD}=\operatorname{mean}_{u}\operatorname{mean}_{i,j\in
R_{u},i\neq j}\left(1-\mathbf{S}_{ij}\right)$

where $R_{u}$ denotes the item lists exposed to user $u$. And we propose to
measure the system-level diversity by computing the average distance of
retrieval results (RD), which is similar to ILAD in computation. In our
experiment, IDL is computed based on the item-item graph embeddings. We adopt
a series of important metrics in feed recommendation. To measure short-term
performance, per capita video views (VV), per capita playtime (Playtime), per
capita click through rate (CTR), and play completion rate (Completion) are
adopted. For long-term evaluation, per capita staytime (Staytime) and weekly
retention rate (Retention) are adopted. Besides, we also evaluate the
commercial revenue by per capita advertisements exposure (AD). The p-value is
set as 0.05. Online A/B experiments are conducted on buckets of users, and the
results are daily collected.

Compared with baselines. The results of a 7-days online experiment are showed
in Table 1. A ”*” means null hypothesis and can be viewed the same as the
baseline. Compared with the rule-based diversification, DPP-based methods can
significantly improve the individual-level diversity. Meanwhile, the
diversified items can improve users’ consumption in feed, which directly
demonstrates the effectiveness of DPP-based reranking. Compared with other
DPP-based methods, our work improves the retrieval diversity, which reveals
the effectiveness of graph exploration. This verifies that our framework can
simultaneously improve both individual-level diversity and system-level
diversity without reducing the accuracy. Our method gains the most online
increase, and is then launched for long-term observation.

Long-term experiment. As is shown in Figure 6, our method obtains consistent
increase in several aspects compared with the ”Base” method. For short-term
metrics, our method can significantly improve the click-through rate, which
reveals that diversified top-k items are more likely to be positively
interacted than those similar items. The observed growth in VV, Playtime, and
Completion also confirms this conclusion. We can observe the learning effect
in long-term gains. Users tend to spend more time in our feed, and the
retention rate also increase in a growing manner. The RD is also significantly
improved especially in the first few days, because the graph in trained in an
incremental way. With more graph exploration, the retrieval diversity is
continuously growing and it consequently affects the diversity in reranking
stage. This verifies that graph exploration can obtain a long-term gain in
system-level diversity. Besides, the commercial metric AD also increases,
because advertisements are regularly inserted in our feed. This proves that
our work can bring business growth.

Both of the experiments demonstrate that individual-level diversity and
system-level diversity can affect mutually when it comes to industrial feed
recommenders. Thus these two levels of diversity should be viewed as an
integrated problem. Graph exploration can efficiently and significantly
improve both of them without losing the online performance. Besides, compared
with methods training and loading another semantic embedding table, the reuse
of graph embedding saves storage and computing resources.

##  5\. Conclusions

Researchers focus on either individual-level diversity or system-level
diversity in one local stage of the recommendation systems, which ignores the
mutual influence with other stages. This makes it hard to achieve improvement
in standard three-stage industrial recommenders. In this paper, we treat
proving two levels of diversity as an integrated task from a systematic and
dynamic perspective. We discuss the importance of graph exploration in both
retrieval and reranking stage. The proposed framework reuses the graph in
different phases of recommender in an efficient way. With the natural
iteration of the system, our work obtains consistent improvement of short-
term, long-term, and commercial benefits in WeChat Top Stories. Besides, we
address that users’ propensity to the degree of diversity changes over time,
and a real-time personalization strategy is adopted to achieve a better trade-
off between relevance and diversity. Our work focuses on the practical issues
of industrial recommenders, and provide an efficient and deployable solution.

## References

  * (1)
  * Ashkan et al. (2015) Azin Ashkan, Branislav Kveton, Shlomo Berkovsky, and Zheng Wen. 2015.  Optimal Greedy Diversity for Recommendation. In _Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015_ , Qiang Yang and Michael J. Wooldridge (Eds.). AAAI Press, 1742–1748.  http://ijcai.org/Abstract/15/248
  * Baracskay et al. (2022) Ian Baracskay, Donald J. Baracskay III, Mehtab Iqbal, and Bart Piet Knijnenburg. 2022.  The Diversity of Music Recommender Systems. In _IUI 2022: 27th International Conference on Intelligent User Interfaces, Helsinki, Finland, March 22 - 25, 2022 - Companion Volume_ , Giulio Jacucci, Samuel Kaski, Cristina Conati, Simone Stumpf, Tuukka Ruotsalo, and Krzysztof Gajos (Eds.). ACM, 97–100.  https://doi.org/10.1145/3490100.3516474
  * Carbonell and Goldstein (1998) Jaime G. Carbonell and Jade Goldstein. 1998.  The Use of MMR, Diversity-Based Reranking for Reordering Documents and Producing Summaries. In _SIGIR ’98: Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, August 24-28 1998, Melbourne, Australia_ , W. Bruce Croft, Alistair Moffat, C. J. van Rijsbergen, Ross Wilkinson, and Justin Zobel (Eds.). ACM, 335–336.  https://doi.org/10.1145/290941.291025
  * Chen et al. (2013) Li Chen, Wen Wu, and Liang He. 2013.  How personality influences users’ needs for recommendation diversity?. In _2013 ACM SIGCHI Conference on Human Factors in Computing Systems, CHI ’13, Paris, France, April 27 - May 2, 2013, Extended Abstracts_ , Wendy E. Mackay, Stephen A. Brewster, and Susanne Bødker (Eds.). ACM, 829–834.  https://doi.org/10.1145/2468356.2468505
  * Chen et al. (2017) Li Chen, Wen Wu, and Liang He. 2017.  Personality and Recommendation Diversity.  In _Emotions and Personality in Personalized Services - Models, Evaluation and Applications_ , Marko Tkalcic, Berardina De Carolis, Marco de Gemmis, Ante Odic, and Andrej Kosir (Eds.). Springer, 201–225.  https://doi.org/10.1007/978-3-319-31413-6_11
  * Chen et al. (2018) Laming Chen, Guoxin Zhang, and Eric Zhou. 2018.  Fast Greedy MAP Inference for Determinantal Point Process to Improve Recommendation Diversity. In _Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montréal, Canada_ , Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (Eds.). 5627–5638.  https://proceedings.neurips.cc/paper/2018/hash/dbbf603ff0e99629dda5d75b6f75f966-Abstract.html
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, Rohan Anil, Zakaria Haque, Lichan Hong, Vihan Jain, Xiaobing Liu, and Hemal Shah. 2016\.  Wide & Deep Learning for Recommender Systems. In _Proceedings of the 1st Workshop on Deep Learning for Recommender Systems, DLRS@RecSys 2016, Boston, MA, USA, September 15, 2016_ , Alexandros Karatzoglou, Balázs Hidasi, Domonkos Tikk, Oren Sar Shalom, Haggai Roitman, Bracha Shapira, and Lior Rokach (Eds.). ACM, 7–10.  https://doi.org/10.1145/2988450.2988454
  * Covington et al. (2016) Paul Covington, Jay Adams, and Emre Sargin. 2016\.  Deep Neural Networks for YouTube Recommendations. In _Proceedings of the 10th ACM Conference on Recommender Systems, Boston, MA, USA, September 15-19, 2016_ , Shilad Sen, Werner Geyer, Jill Freyne, and Pablo Castells (Eds.). ACM, 191–198.  https://doi.org/10.1145/2959100.2959190
  * Davidson et al. (2010) James Davidson, Benjamin Liebald, Junning Liu, Palash Nandy, Taylor Van Vleet, Ullas Gargi, Sujoy Gupta, Yu He, Mike Lambert, Blake Livingston, and Dasarathi Sampath. 2010.  The YouTube video recommendation system. In _Proceedings of the 2010 ACM Conference on Recommender Systems, RecSys 2010, Barcelona, Spain, September 26-30, 2010_ , Xavier Amatriain, Marc Torrens, Paul Resnick, and Markus Zanker (Eds.). ACM, 293–296.  https://doi.org/10.1145/1864708.1864770
  * Eskandanian et al. (2017) Farzad Eskandanian, Bamshad Mobasher, and Robin Burke. 2017\.  A Clustering Approach for Personalizing Diversity in Collaborative Recommender Systems. In _Proceedings of the 25th Conference on User Modeling, Adaptation and Personalization, UMAP 2017, Bratislava, Slovakia, July 09 - 12, 2017_ , Mária Bieliková, Eelco Herder, Federica Cena, and Michel C. Desmarais (Eds.). ACM, 280–284.  https://doi.org/10.1145/3079628.3079699
  * Gao et al. (2023) Chen Gao, Yu Zheng, Nian Li, Yinfeng Li, Yingrong Qin, Jinghua Piao, Yuhan Quan, Jianxin Chang, Depeng Jin, Xiangnan He, and Yong Li. 2023.  A Survey of Graph Neural Networks for Recommender Systems: Challenges, Methods, and Directions.  _ACM Trans. Recomm. Syst._ 1, 1, Article 3 (mar 2023), 51 pages.  https://doi.org/10.1145/3568022
  * Ge et al. (2010) Mouzhi Ge, Carla Delgado-Battenfeld, and Dietmar Jannach. 2010\.  Beyond accuracy: evaluating recommender systems by coverage and serendipity. In _Proceedings of the 2010 ACM Conference on Recommender Systems, RecSys 2010, Barcelona, Spain, September 26-30, 2010_ , Xavier Amatriain, Marc Torrens, Paul Resnick, and Markus Zanker (Eds.). ACM, 257–260.  https://doi.org/10.1145/1864708.1864761
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  DeepFM: A Factorization-Machine based Neural Network for CTR Prediction. In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017_ , Carles Sierra (Ed.). ijcai.org, 1725–1731.  https://doi.org/10.24963/ijcai.2017/239
  * Hamilton et al. (2017) William L. Hamilton, Zhitao Ying, and Jure Leskovec. 2017\.  Inductive Representation Learning on Large Graphs. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (Eds.). 1024–1034.  https://proceedings.neurips.cc/paper/2017/hash/5dd9db5e033da9c6fb5ba83c7a7ebea9-Abstract.html
  * He et al. (2020) Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yong-Dong Zhang, and Meng Wang. 2020\.  LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020_ , Jimmy X. Huang, Yi Chang, Xueqi Cheng, Jaap Kamps, Vanessa Murdock, Ji-Rong Wen, and Yiqun Liu (Eds.). ACM, 639–648.  https://doi.org/10.1145/3397271.3401063
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017\.  Neural Collaborative Filtering. In _Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017_ , Rick Barrett, Rick Cummings, Eugene Agichtein, and Evgeniy Gabrilovich (Eds.). ACM, 173–182.  https://doi.org/10.1145/3038912.3052569
  * Herlocker et al. (2004) Jonathan L. Herlocker, Joseph A. Konstan, Loren G. Terveen, and John Riedl. 2004.  Evaluating collaborative filtering recommender systems.  _ACM Trans. Inf. Syst._ 22, 1 (2004), 5–53.  https://doi.org/10.1145/963770.963772
  * Hohnhold et al. (2015) Henning Hohnhold, Deirdre O’Brien, and Diane Tang. 2015\.  Focusing on the Long-Term: It’s Good for Users and Business. In _Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (Sydney, NSW, Australia) _(KDD ’15)_. Association for Computing Machinery, New York, NY, USA, 1849–1858.  https://doi.org/10.1145/2783258.2788583
  * Huang et al. (2021) Yanhua Huang, Weikun Wang, Lei Zhang, and Ruiwen Xu. 2021\.  Sliding Spectrum Decomposition for Diversified Recommendation. In _KDD ’21: The 27th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, Singapore, August 14-18, 2021_ , Feida Zhu, Beng Chin Ooi, and Chunyan Miao (Eds.). ACM, 3041–3049.  https://doi.org/10.1145/3447548.3467108
  * Koren et al. (2009) Yehuda Koren, Robert M. Bell, and Chris Volinsky. 2009\.  Matrix Factorization Techniques for Recommender Systems.  _Computer_ 42, 8 (2009), 30–37.  https://doi.org/10.1109/MC.2009.263
  * Li et al. (2019b) Chao Li, Zhiyuan Liu, Mengmeng Wu, Yuchi Xu, Huan Zhao, Pipei Huang, Guoliang Kang, Qiwei Chen, Wei Li, and Dik Lun Lee. 2019b.  Multi-Interest Network with Dynamic Routing for Recommendation at Tmall. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019_ , Wenwu Zhu, Dacheng Tao, Xueqi Cheng, Peng Cui, Elke A. Rundensteiner, David Carmel, Qi He, and Jeffrey Xu Yu (Eds.). ACM, 2615–2623.  https://doi.org/10.1145/3357384.3357814
  * Li et al. (2008) Qing-Cheng Li, Zhen-Hua Dong, and Tuo Li. 2008.  Research of information recommendation system based on reading behavior. In _2008 International Conference on Machine Learning and Cybernetics_ , Vol. 3. 1626–1631.  https://doi.org/10.1109/ICMLC.2008.4620666
  * Li et al. (2019a) Zekun Li, Zeyu Cui, Shu Wu, Xiaoyu Zhang, and Liang Wang. 2019a.  Fi-GNN: Modeling Feature Interactions via Graph Neural Networks for CTR Prediction. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM 2019, Beijing, China, November 3-7, 2019_ , Wenwu Zhu, Dacheng Tao, Xueqi Cheng, Peng Cui, Elke A. Rundensteiner, David Carmel, Qi He, and Jeffrey Xu Yu (Eds.). ACM, 539–548.  https://doi.org/10.1145/3357384.3357951
  * Liu et al. (2020) Weiwen Liu, Qing Liu, Ruiming Tang, Junyang Chen, Xiuqiang He, and Pheng-Ann Heng. 2020\.  Personalized Re-ranking with Item Relationships for E-commerce. In _CIKM ’20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020_ , Mathieu d’Aquin, Stefan Dietze, Claudia Hauff, Edward Curry, and Philippe Cudré-Mauroux (Eds.). ACM, 925–934.  https://doi.org/10.1145/3340531.3412332
  * Noia et al. (2014) Tommaso Di Noia, Vito Claudio Ostuni, Jessica Rosati, Paolo Tomeo, and Eugenio Di Sciascio. 2014\.  An analysis of users’ propensity toward diversity in recommendations. In _Eighth ACM Conference on Recommender Systems, RecSys ’14, Foster City, Silicon Valley, CA, USA - October 06 - 10, 2014_ , Alfred Kobsa, Michelle X. Zhou, Martin Ester, and Yehuda Koren (Eds.). ACM, 285–288.  https://doi.org/10.1145/2645710.2645774
  * Rendle (2010) Steffen Rendle. 2010\.  Factorization Machines. In _ICDM 2010, The 10th IEEE International Conference on Data Mining, Sydney, Australia, 14-17 December 2010_ , Geoffrey I. Webb, Bing Liu, Chengqi Zhang, Dimitrios Gunopulos, and Xindong Wu (Eds.). IEEE Computer Society, 995–1000.  https://doi.org/10.1109/ICDM.2010.127
  * Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009.  BPR: Bayesian Personalized Ranking from Implicit Feedback. In _UAI 2009, Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence, Montreal, QC, Canada, June 18-21, 2009_ , Jeff A. Bilmes and Andrew Y. Ng (Eds.). AUAI Press, 452–461.  https://www.auai.org/uai2009/papers/UAI2009_0139_48141db02b9f0b02bc7158819ebfa2c7.pdf
  * Sarwar et al. (2001) Badrul Munir Sarwar, George Karypis, Joseph A. Konstan, and John Riedl. 2001.  Item-based collaborative filtering recommendation algorithms. In _Proceedings of the Tenth International World Wide Web Conference, WWW 10, Hong Kong, China, May 1-5, 2001_ , Vincent Y. Shen, Nobuo Saito, Michael R. Lyu, and Mary Ellen Zurko (Eds.). ACM, 285–295.  https://doi.org/10.1145/371920.372071
  * Wang et al. (2019) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019.  Neural Graph Collaborative Filtering. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019_ , Benjamin Piwowarski, Max Chevalier, Éric Gaussier, Yoelle Maarek, Jian-Yun Nie, and Falk Scholer (Eds.). ACM, 165–174.  https://doi.org/10.1145/3331184.3331267
  * Wang et al. (2020a) Yichao Wang, Xiangyu Zhang, Zhirong Liu, Zhenhua Dong, Xinhua Feng, Ruiming Tang, and Xiuqiang He. 2020a.  Personalized Re-ranking for Improving Diversity in Live Recommender Systems.  _CoRR_ abs/2004.06390 (2020).  arXiv:2004.06390 https://arxiv.org/abs/2004.06390
  * Wang et al. (2020b) Zhe Wang, Liqin Zhao, Biye Jiang, Guorui Zhou, Xiaoqiang Zhu, and Kun Gai. 2020b.  COLD: Towards the Next Generation of Pre-Ranking System.  _CoRR_ abs/2007.16122 (2020).  arXiv:2007.16122 https://arxiv.org/abs/2007.16122
  * Wilhelm et al. (2018) Mark Wilhelm, Ajith Ramanathan, Alexander Bonomo, Sagar Jain, Ed H. Chi, and Jennifer Gillenwater. 2018.  Practical Diversified Recommendations on YouTube with Determinantal Point Processes. In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management, CIKM 2018, Torino, Italy, October 22-26, 2018_ , Alfredo Cuzzocrea, James Allan, Norman W. Paton, Divesh Srivastava, Rakesh Agrawal, Andrei Z. Broder, Mohammed J. Zaki, K. Selçuk Candan, Alexandros Labrinidis, Assaf Schuster, and Haixun Wang (Eds.). ACM, 2165–2173.  https://doi.org/10.1145/3269206.3272018
  * Wu et al. (2022) Jiancan Wu, Xiangnan He, Xiang Wang, Qifan Wang, Weijian Chen, Jianxun Lian, and Xing Xie. 2022.  Graph convolution machine for context-aware recommender system.  _Frontiers Comput. Sci._ 16, 6 (2022), 166614\.  https://doi.org/10.1007/s11704-021-0261-8
  * Wu et al. (2018) Wen Wu, Li Chen, and Yu Zhao. 2018.  Personalizing recommendation diversity based on user personality.  _User Model. User Adapt. Interact._ 28, 3 (2018), 237–276.  https://doi.org/10.1007/s11257-018-9205-x
  * Xie et al. (2022) Ruobing Xie, Qi Liu, Shukai Liu, Ziwei Zhang, Peng Cui, Bo Zhang, and Leyu Lin. 2022.  Improving Accuracy and Diversity in Matching of Recommendation With Diversified Preference Network.  _IEEE Trans. Big Data_ 8, 4 (2022), 955–967.  https://doi.org/10.1109/TBDATA.2021.3103263
  * Yang et al. (2023) Liangwei Yang, Shengjie Wang, Yunzhe Tao, Jiankai Sun, Xiaolong Liu, Philip S. Yu, and Taiqing Wang. 2023.  DGRec: Graph Neural Network for Recommendation with Diversified Embedding Generation. In _Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM 2023, Singapore, 27 February 2023 - 3 March 2023_ , Tat-Seng Chua, Hady W. Lauw, Luo Si, Evimaria Terzi, and Panayiotis Tsaparas (Eds.). ACM, 661–669.  https://doi.org/10.1145/3539597.3570472
  * Ying et al. (2018) Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L. Hamilton, and Jure Leskovec. 2018.  Graph Convolutional Neural Networks for Web-Scale Recommender Systems. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2018, London, UK, August 19-23, 2018_, Yike Guo and Faisal Farooq (Eds.). ACM, 974–983.  https://doi.org/10.1145/3219819.3219890
  * Yu et al. (2022) Wenhui Yu, Zixin Zhang, and Zheng Qin. 2022.  Low-Pass Graph Convolutional Network for Recommendation. In _Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022_. AAAI Press, 8954–8961.  https://ojs.aaai.org/index.php/AAAI/article/view/20878
  * Zhang et al. (2022) Shengyu Zhang, Lingxiao Yang, Dong Yao, Yujie Lu, Fuli Feng, Zhou Zhao, Tat-Seng Chua, and Fei Wu. 2022\.  Re4: Learning to Re-contrast, Re-attend, Re-construct for Multi-interest Recommendation. In _WWW ’22: The ACM Web Conference 2022, Virtual Event, Lyon, France, April 25 - 29, 2022_ , Frédérique Laforest, Raphaël Troncy, Elena Simperl, Deepak Agarwal, Aristides Gionis, Ivan Herman, and Lionel Médini (Eds.). ACM, 2216–2226.  https://doi.org/10.1145/3485447.3512094
  * Zheng et al. (2021) Yu Zheng, Chen Gao, Liang Chen, Depeng Jin, and Yong Li. 2021.  DGCN: Diversified Recommendation with Graph Convolutional Networks. In _WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021_ , Jure Leskovec, Marko Grobelnik, Marc Najork, Jie Tang, and Leila Zia (Eds.). ACM / IW3C2, 401–412.  https://doi.org/10.1145/3442381.3449835