# Disentangling the Potential Impacts of Papers into Diffusion, Conformity,
and Contribution Values

Zhikai Xue<EMAIL_ADDRESS>Guoxiu He<EMAIL_ADDRESS>Zhuoren
Jiang<EMAIL_ADDRESS>Yangyang Kang<EMAIL_ADDRESS>Star Zhao<EMAIL_ADDRESS>Wei Lu<EMAIL_ADDRESS>Faculty of Economics
and Management, East China Normal University, Shanghai, 200062, China
Institute of AI for Education, East China Normal University, Shanghai, 200062,
China  School of Public Affairs, Zhejiang University, Hangzhou, 310058, China
Alibaba Group, Hangzhou, 310058, China  Institute of Big Data (IBD), Fudan
University, Shanghai, 200433, China  National Institute of Intelligent
Evaluation and Governance, Fudan University, Shanghai, 200433, China  School
of Information Management, Wuhan University, Wuhan, 430072, China

###### Abstract

The potential impact of an academic paper is determined by various factors,
including its popularity and contribution. Existing models usually estimate
original citation counts based on static graphs and fail to differentiate
values from nuanced perspectives. In this study, we propose a novel graph
neural network to Disentangle the Potential impacts of Papers into Diffusion,
Conformity, and Contribution values (called DPPDCC). Given a target paper,
DPPDCC encodes temporal and structural features within the constructed dynamic
heterogeneous graph. Particularly, to capture the knowledge flow, we emphasize
the importance of comparative and co-cited/citing information between papers
and aggregate snapshots evolutionarily. To unravel popularity, we contrast
augmented graphs to extract the essence of diffusion and predict the
accumulated citation binning to model conformity. We further apply orthogonal
constraints to encourage distinct modeling of each perspective and preserve
the inherent value of contribution. To evaluate models’ generalization for
papers published at various times, we reformulate the problem by partitioning
data based on specific time points to mirror real-world conditions. Extensive
experimental results on three datasets demonstrate that DPPDCC significantly
outperforms baselines for previously, freshly, and immediately published
papers. Further analyses confirm its robust capabilities. We will make our
datasets and codes publicly available.

###### keywords:

Impact Prediction , Citation Network , Dynamic Heterogeneous Graph , Graph
Neural Network , Contrastive Learning

††journal: JASIST

##  1 Introduction

In paper retrieval and recommendation, ranking papers based on their
scientific impact can aid researchers to delve into intricate research
efforts. This is particularly significant considering the exponential annual
growth in the number of published papers (Lo et al., 2020; Chu and Evans,
2021; Xue et al., 2023). Given the inherent difficulties in quantifying the
precise numerical value of scientific impact, citation count is regularly
employed as a rough approximation (Evans and Reimer, 2009; Sinatra et al.,
2016; Jiang et al., 2021). Nevertheless, the current ranking depends on the
cumulative number of citations received to date, which merely represents the
impact within the prior research environment. Therefore, it becomes imperative
to cultivate a sophisticated model that can assess the potential impact of a
research paper, placing significant emphasis on the future (Wang et al.,
2013).

At a given point in time (e.g., at present), estimating the anticipated
increment in citations can effectively facilitate a fair comparison of the
potential impact between long-published papers and newly published ones.
Actually, the citations received by papers are not solely derived from the
influence of their contributions, but are also affected by a multitude of
other factors (Bornmann and Daniel, 2008; Case and Higgins, 2000). Hence,
rather than directly forecasting original citation counts, it is more
effective to disentangle citation increments received from extrinsic factors
and the actual contributing factor, making impact prediction more practical
and interpretable. Afterwards, we can identify exceptional works amidst the
vast array of publications by discerning the potential contributing impact. In
this work, we emphasize the popularity factor of a paper as a prominent
extrinsic factor affecting the increase of citations. More precisely, when
referring to the popularity factor of a paper, we are considering its citation
diffusion patterns and the collective conformity degree within the citation
network.

Regarding citation diffusion, we posit that influential papers assume an
amplifying effect in shaping the process of information dissemination within
the citation network (Page et al., 1998; Wu et al., 2019). This leads to
increased visibility of papers connected to them. In other words, papers cited
by highly-cited papers will receive significantly more citations than others.
The statistics of the dataset provide empirical support. For instance, in the
field of Computer Science, the former papers will receive an average of 1.5475
citations (after applying a logarithmic transformation), while the latter ones
will only receive an average of 0.8580 citations. Towards collective
conformity, existing studies have found that some researchers may tend to cite
papers associated with well-known entities, irrespective of their actual
relevance or genuine contribution (Thornley et al., 2015). This phenomenon is
substantiated by the future disparity (2.1917 vs. 0.8009) in papers with
different accumulated citations, which aligns with the concept of Matthew
effect in the academic domain (Allison and Stewart, 1974).

Therefore, by separating out the influence of the popularity, i.e., citation
diffusion and collective conformity, we can gain a closer approximation to the
genuine contributing impacts of papers. In essence, we aim to disentangle the
citation increment into diffusion, conformity, and contribution values (as
shown in Figure 1 (a)).

Figure 1: (a) We disentangle the citation increment into diffusion,
conformity, and contribution values for better interpretability. (b) To
imitate the evaluation of future dynamics of the citation context, we split
the dataset by the observation time point and categorize the samples into
previous, fresh, and immediate papers in terms of their publication time. (c)
We represent the heterogeneous subgraph schema with all metadata, where each
type of relation is bidirectional.

Typically, conventional research (Li et al., 2017; Cao et al., 2017; Yang et
al., 2022; Chen et al., 2022; Xu et al., 2022) on citation prediction imposes
stringent constraints to select favorable samples by removing lowly-cited
papers. Moreover, the division of these data into training, validation, and
testing is carried out randomly, disregarding the chronological order of paper
publications (Ruan et al., 2020; Abrishami and Aliakbary, 2019; Huang et al.,
2022; Xue et al., 2023). However, this problem setting deviates from real-
world scenarios, possibly causing a substantial distribution shift. Hence, the
model may face challenges in handling recently published papers. Random data
partitioning disregards new citation network updates, hampering the model’s
practicality and robustness in upcoming real-world test. This limitation
impedes the exploration of the potential value of papers reactivated in the
evolved context (Van Raan, 2004).

In contrast, our work emulates reality by performing a novel problem setting.
We include all papers without strict filtering to maintain consistency with
the data distribution of real-world citation network. As illustrated in Figure
1 (b), we partition the dataset based on particular observation time points.
This can introduce fresh papers that are not part of the training set
(previous papers) into the test set. We further consider immediate papers
(published at the test time point) that closely align with cold-start
scenarios to validate the robustness of our model. Thus, papers in the test
set can be divided into previous, fresh, and immediate, facilitating a more
practical evaluation. Notably, previous papers have different contexts and
future increments at the train and test time points. By employing this
approach, our model is incentivized to fit the dynamic citation context,
thereby enhancing its generalization.

Existing graph disentanglement studies usually aim at decomposing abstractive
hidden factors from either static or homogeneous graphs (see Section 2) (Ma et
al., 2019a; Yang et al., 2020; Wang et al., 2020). Instead, our focus lies in
disentangling the categorical characteristics of citation behaviors within
dynamic and heterogeneous citation graphs. In this study, tailored to each
target paper, we construct a Dynamic Heterogeneous Graph from the citation
network. The graph consists of a sequence of heterogeneous snapshots (see
Figure 1 (c)). This graph retains both the content of the paper and its
dynamic citation context. Based on the proposed graph, we devise a model that
could Disentangle the Potential impacts of Papers into Diffusion, Conformity,
and Contribution Values (called DPPDCC). This model utilizes Citation-aware
Relational Graph Convolutional Networks (R-GCNs) to capture the structural
features of the input sequence of snapshots. The outputs are then passed
through a Transformer Encoder, which allows for the incorporation of temporal
information. Specially, we design a novel module in modified R-GCN to
integrate the comparative and co-cited/citing information between paper nodes.
Furthermore, we propose a Type-specific Attention Readout mechanism to
aggregate information from each snapshot effectively. Upon acquiring the final
representation of the target paper, we proceed to disentangle the predicted
citation increment into diffusion, conformity, and contribution values
individually. For the diffusion, we employ Triplet Citation-aware Graph
Contrastive Learning to capture the popularity influence inherent in the
information diffusion process within the citation network. For the conformity,
we introduce an auxiliary task to classify bins of accumulated citations.
Additionally, we impose an orthogonal constraint on the representations of
disentangled perspectives in pair, guaranteeing that they encode distinct
features without overlap. Extensive experiments are conducted on three fields
of the real-world scientific dataset S2ORC (Lo et al., 2020). The results
illustrate the superior performance of DPPDCC compared to baseline models
across previous, fresh, and immediate papers. Ablation tests further highlight
the significance of the proposed components. Additionally, visualizations
offer insights into the model’s interpretability.

To sum up, the contributions of this research are as follows:

$\bullet$ We revise the problem formulation of impact prediction to better
align with real-world scenarios. Without filtering the predicted samples by
their citation counts, we partition the datasets based on specific time points
to maintain the dynamics of the citation context. Moreover, we employ
disentangled representation learning to extract the contributing impacts of
papers.

$\bullet$ We propose a novel model called DPPDCC that first encodes
comparative and co-cited/citing information evolutionarily within the Dynamic
Heterogeneous Graph through Citation-aware Graph Neural Network Encoder for
potential impact prediction. It further disentangles the citation increment
into diffusion, conformity, and contribution values.

$\bullet$ Experimental results show significantly superior to existing
baselines for previous, fresh, and immediate papers. Further analyses
illustrate that our model can reasonably make predictions.

##  2 Related Work

We review three lines of related work: Citation/Cascade Prediction, Dynamic
and Heterogeneous Graph Neural Network (GNN), and Disentangled Representation
Learning.

Citation prediction is a vital sub-task of automated academic evaluation as it
enables the estimation of potential impact. It is closely examined within
cascade prediction tasks, which share analogous graph structures and
objectives (Zhou et al., 2021). Citation count prediction places a focus on
the content and distinctive features of individual papers. In contrast,
cascade count prediction delves into the intricate relationships between
various entities, including papers and authors. Their approaches can be
broadly classified into three categories: stochastic models, feature-based
models, and deep learning models. Stochastic models can predict future
citation counts by fitting the curve of citation counts (Glänzel and Schubert,
1995), following Zipf-Mandelbrot’s law (Silagadze, 1997). Recently, machine
learning models have shown promising results by utilizing manually extracted
features (Yan et al., 2011; Yu et al., 2014; Ruan et al., 2020; Guille et al.,
2013; Tatar et al., 2014). More presently, deep neural networks have dominated
this task by applying advanced Natural Language Processing (NLP) and Computer
Vision (CV) methods to extract abstractive representations from paper content
(Abrishami and Aliakbary, 2019; Huang et al., 2022; Xue et al., 2023). For
cascade prediction, graph embeddings, sequence models, and GNN models are
employed to extract structural information from the underlying graph, and then
temporal information is encoded with sequence models (Li et al., 2017; Cao et
al., 2017; Yang et al., 2022; Chen et al., 2022; Xu et al., 2022). However,
existing models for citation and cascade prediction exhibit suboptimal
performance. They fail to fully exploit the valuable information presented in
paper content and scientific context within the citation network
simultaneously. Furthermore, their strict data selection and random data
splitting strategies may hinder their practicality. In contrast, our study
proposes to specifically revise these settings and utilizes both content and
dynamic context to conduct impact estimation.

GNNs (Sperduti and Starita, 1997; Gori et al., 2005; Scarselli et al., 2008;
Wu et al., 2020) are widely applied to handle non-Euclidean data like graphs.
Recently, research interest has surged in dynamic graphs, which incorporate
temporal information, as well as in heterogeneous graphs, which involve
multiple node types or edge types. These types of graphs are more prevalent in
practical scenarios and encapsulate more intricate information, compared to
static homogeneous graphs, driving the formulation and refinement of diverse
methodologies within the academic realm. The dynamic graph can be divided into
multiple snapshots at different time points. Previous models like DGCN
(Manessi et al., 2020), Dysat (Sankar et al., 2020), and ROLAND (You et al.,
2022) typically encode the snapshots with static GCN as the structural
encoder, followed by sequential models like Recurrent Neural Networks or
Transformer Encoder. Moreover, certain models incorporate GCN into RNN models
by configuring the parameters within RNN cells as GCN parameters (Pareja et
al., 2020). Within heterogeneous graphs, on one side, R-GCN serves as a
prominent model with its separate message-passing architecture (Schlichtkrull
et al., 2018). It conducts independent message passing within different
relations and aggregates them to update node representations. On the other
side, the relation-learning method transforms heterogeneous graphs into
homogeneous ones, while retaining node and edge type indicators to preserve
heterogeneous information (Hu et al., 2020; You et al., 2022). An example of
this technique is HGT (Hu et al., 2020), which utilizes multi-head attention
modules with specific parameters tailored to different node and edge types,
effectively modeling the complexity of heterogeneous graphs. Our study
proposes a Citation-aware GNN Encoder that effectively models both dynamic and
heterogeneous graphs of target papers, which are constructed from the citation
network.

Disentangled representation learning aims to identify and disentangle the
underlying explanatory factors (Bengio et al., 2013). The existing efforts
primarily focus on CV and NLP with Variational Auto-encoders (Ma et al.,
2019b; Locatello et al., 2019; Cai et al., 2019; Denton et al., 2017; Lee et
al., 2021). In graph learning tasks, inspired by capsule networks, they
partition node features into multiple hidden channels (Ma et al., 2019a).
Subsequent models strive to factorize the input graphs to facilitate distinct
message passing (Yang et al., 2020). Recent models extend their focus to
isolate causal and biased information presented in both the graph structure
and node features, with a particular emphasis on causal effects (Sui et al.,
2022; Fan et al., 2022). In addition, certain studies extend the disentangled
GNNs to specialized scenarios including recommendation, addressing more
complex graph structures involving either heterogeneous or dynamic graphs
(Wang et al., 2020; Zhang et al., 2022; Wen et al., 2022). In contrast to
previous studies, our approach acknowledges that the potential impact within
the citation network is influenced not only by the paper contribution but also
by popularity factors. These popularity factors encompass the amplifying
effect owing to the highly-cited paper nodes in information diffusion and the
manifestation of Matthew effect as collective conformity. Based on the dynamic
heterogeneous graph, we disentangle the potential impact into diffusion,
conformity, and contribution values, seeking a reasonable estimation.

##  3 Methodology

In this section, we introduce our proposed model DPPDCC in detail. The overall
framework of DPPDCC is depicted in Figure 2, which can be divided into two
parts: Citation-aware GNN Encoder (CGE) and Popularity-aware Disentanglement
Module (PDM).

Figure 2: The overall architecture of DPPDCC. DPPDCC first encodes the Dynamic
Heterogeneous Graph of the target paper with the Citation-aware GNN Encoder.
Based on the encoded representation of the target paper, DPPDCC then
disentangles its citation increment into diffusion, conformity, and
contribution values through three corresponding auxiliary tasks.

###  3.1 Problem Formulation and Notations

Before presenting our approach, we first introduce the problem formulation and
notations. In practice, the citation network comprises diverse entities and
relations that evolve dynamically each year. To capture the entity-aware and
time-variant characteristics, for each target paper, we construct a dynamic
heterogeneous graph from the citation network to predict its potential impact.

####  3.1.1 Dynamic Heterogeneous Graph

Given a target paper at the time step $t$, its heterogeneous subgraph is
defined as $G^{t}=(\nu^{t},\epsilon^{t},\phi,\varphi)$. Here, $\nu^{t}$
denotes the set of nodes surrounding the target paper at the time step $t$.
And $\epsilon^{t}$ represents the set of edges among $\nu^{t}$. Each node
$\nu$ and edge $\epsilon$ are associated with their type mapping functions
$\phi:\nu\rightarrow U$ and $\varphi:\epsilon\rightarrow R$, where $U$ and $R$
denote the types of nodes and edges (see Figure 1 (c)), respectively.

The backbone of the heterogeneous subgraph primarily consists of the cited and
citing relations. We collect $k$-hop cited/citing neighbors of the target
paper at the specific time $t$. For each $i$-th hop, to maintain the most
recent and significant information, we select the $K_{i}$ newest published and
then most highly cited papers. To address the potential lack of global
information, we incorporate metadata nodes associated with the papers (see
Figure 1 (c)). Prior to the observation time point, we consider a total of $T$
such subgraphs, each serving as a snapshot captured at distinct time steps.
These subgraphs organized into a sequence collectively constitute the dynamic
heterogeneous graph $G$ of the target paper.

####  3.1.2 Potential Impact Prediction

We assess the potential impact of the given target paper by estimating its
citation increment. It is quantified as the increased citation number $\Delta$
years after the observation time point.

###  3.2 Initialization

The initial embedding of the paper node is derived from the text embedding
obtained through the combination of title and abstract. Metadata node
embeddings in each snapshot are the average over all papers associated with
them in the global citation network at this time step. Specifically, the
extraction of text embeddings is accomplished using Sentence-BERT (Reimers and
Gurevych, 2019). For $i$-th node at the $t$-th time step, we have:

$\begin{split}&\bm{h}_{ti,paper}^{(0)}=TextEncoder([w_{1},w_{2},...,w_{n^{w}}])\\\
&\bm{h}_{ti,u}^{(0)}=avgpool([\bm{h}_{t1,paper}^{(0)},\bm{h}_{t2,paper}^{(0)},...,\bm{h}_{tm_{p},paper}^{(0)}]),\\\
&u\in\\{author,venue,time\\}\end{split}$ (1)

where $u$ denotes the node type, $TextEncoder$ denotes the text representation
model, and $m_{p}$ is the count of metadata node’s connected papers. To
initialize the $0$-th layer representation of the snapshot of the $i$-th
target paper at the $t$-th time step $\bm{h}_{ti,s}^{(0)}$, we mainly take
into account the contextual information from the corresponding time embedding
$\bm{h}_{t,time}^{(0)}$ of time step $t$. Since the time embedding only
preserves the global features, we utilize a gate module to approach specific
information from the $i$-th target paper embedding $\bm{h}_{i,p}^{(0)}$. The
snapshot representation $\bm{h}_{ti,s}^{(l)}$ of $l$-th layer will then be the
proxy to aggregate the corresponding heterogeneous subgraph:

$\begin{split}&\bm{h}_{ti,s}^{(0)}=\bm{a}_{ti}\times\bm{h}_{t,time}^{(0)}\\\
&\bm{a}_{ti}=\mathrm{sigmoid}(\bm{W}^{ipt}\bm{h}_{t,time}^{(0)}+\bm{W}^{tgt}\bm{h}_{i,p}^{(0)})\end{split}$
(2)

###  3.3 Citation-aware GNN Encoder

Citation-aware GNN Encoder (CGE) aims to capture distinct structural and
temporal features from the dynamic heterogeneous graph, which is facilitated
by leveraging knowledge diffusion theories derived from bibliometrics. To be
specific, the backbone of CGE consists of modified R-GCNs as the structural
encoder and a Transformer Encoder as the temporal encoder. We stack $L$
integrated layers to encode them alternately instead of separate modeling.
This allows us to exploit higher-order interactions between space and time. To
elaborate, for subgraphs at different time points, we first utilize distinct
Citation-aware R-GCNs to model their structural information separately.
Afterwards, a Type-specific Attention Snapshot Readout is employed to
summarize the information from these subgraphs based on the snapshot
representations. These aggregated representations are then fed into the
temporal encoder (Encoder of the Transformer) to capture the intricate
temporal relationships between different time points. By integrating
structural and temporal encoding, our approach could extract valuable insights
from the dynamic heterogeneous graph.

####  3.3.1 Citation-aware R-GCNs

Citation-aware R-GCNs are utilized to encode node hidden states $\bm{h}$
within the dynamic heterogeneous graph:

$\begin{split}&\bm{h}_{dst}^{(l+1)}=\underset{r\in\mathcal{R},r_{dst}=dst}{AGG}(f_{r}(g_{r},\bm{h}_{r_{src}}^{(l)},\bm{h}_{r_{dst}}^{(l)}))\\\
\end{split}$ (3)

where $\mathcal{R}$ is the set of all edges in the graph, $g_{r}$ is the
subgraph with relation $r$, and $AGG$ is the relation aggregation function.

Motivated by the Disruption-index (Wu et al., 2019), in “is cited by” edges,
if the source and target paper have more identical references, the source
paper may be less influenced by the target paper, resulting in a negative
effect; while, in “cites” edges, if the source and target paper have more
identical citations, they may share more commonness and are more closely
connected, leading to a positive effect. Thus, we introduce CompGAT, a novel
module based on GATv2 (Brody et al., 2021), to encode comparative and co-
cited/citing features between paper nodes. To capture the discrepancy between
source and target nodes, we encode the concatenation of their representations.
Besides the basic attention scores of GATv2, we incorporate co-cited or co-
citing strengths by assessing the normalized similarity of their references or
citations in the global citation network. Based on the adjacent matrix
$\bm{A}$, it is calculated as $\bm{C}=\mathrm{norm}(\bm{A}\bm{A}^{T})$, which
represents 2-hop relationships connected with intermediate paper nodes.
Finally, the attention is a mixed distribution resembling CopyNet (Gu et al.,
2016). For $i$-th paper node in $(l+1)$-th layer $\bm{h}_{i}^{(l+1)}$, we
have:

$\begin{split}&\bm{h}_{i}^{(l+1)}=\mathrm{LeakyReLU}\left(\mathrm{layernorm}(\bm{h}_{i}^{(l)}+\bm{h}_{in}^{(l+1)})\right)\\\
&\bm{h}_{in}^{(l+1)}=\sum_{j\in\mathcal{N}(i)}\alpha_{ij}^{(l)}\bm{W}_{c}^{(l)}[\bm{W}^{(l)}_{left}\bm{h}_{i}^{(l)}\|\bm{W}^{(l)}_{right}\bm{h}_{j}^{(l)}]\\\
&\alpha_{ij}^{(l)}=\lambda\mathrm{softmax_{i}^{c}}(c_{ij})+(1-\lambda)\mathrm{softmax_{i}^{e}}(e_{ij}^{(l)})\\\
&e_{ij}^{(l)}=\bm{w}_{a}^{(l)}\mathrm{LeakyReLU}\left(\bm{W}^{(l)}_{left}\bm{h}_{i}^{(l)}+\bm{W}^{(l)}_{right}\bm{h}_{j}^{(l)}\right)\end{split}$
(4)

where $c_{ij}\in\bm{C}$ is the normalized co-cited/citing strength, $\lambda$
is to balance the sum distribution of co-cited/citing strengths and basic
attention weights, and $\bm{w}_{a}^{(l)}$is the attention vector.
Additionally, we apply GIN (Xu et al., 2018) with sum-pooling for other edges
in the R-GCN:

$\begin{split}\bm{h}_{i,r_{dst}}^{(l+1)}=f_{\Theta,r}((1+\xi)\bm{h}_{i,r_{dst}}^{(l)}+AGG(\\{\bm{h}_{j,r_{src}}^{(l)},&j\in\mathcal{N}(i,r_{src})\\})),r\in\mathcal{R}^{normal}\end{split}$
(5)

where $\xi$ is the hyper-parameter to decide the retained original
information, $\mathcal{R}^{normal}$ is the set including all edges in classic
GIN, and $AGG$ is the neighborhood aggregation function of the GIN.

####  3.3.2 Type-specific Attention Snapshot Readout

To represent the time-aware information of a single snapshot $\bm{h}_{t,s}$,
we design an attention-based readout mechanism. It extracts the evolved
temporal information of different types of paper nodes (reference, citation,
and target paper) based on the publication age of the target paper.

$\begin{split}&\bm{h}_{t,s}^{(l+1)}=\bm{h}_{t,s}^{(l)}+\sum_{j\in\mathcal{N}(t)}\alpha_{tj}^{(l)}\bm{W}^{(l)}_{p}\bm{h}_{j,p}^{(l)}\\\
&\alpha_{tj}^{(l)}=\mathrm{softmax_{t}}(\hat{e}_{tj}^{(l)})\\\
&\hat{e}_{tj}^{(l)}={\bm{w}_{a}}^{(l)}\mathrm{LeakyReLU}\left(\bm{W}^{(l)}_{s}\bm{h}_{t,s}^{(l)}+\bm{W}^{(l)}_{p}\bm{h}_{j,p}^{(l)}\right)+e_{tj}^{(l)}\\\
&e_{tj}^{(l)}={\bm{w}_{t}}^{(l)}\mathrm{LeakyReLU}\left(f^{(l)}_{se}(\omega^{s}_{t})+f^{(l)}_{pe}(\omega^{p}_{j})\right)\end{split}$
(6)

where $\mathcal{N}(t)$ denotes the paper nodes within $t$-th 1-hop subgraph,
$f^{(l)}_{se}(\omega^{s}_{t})$ and $f^{(l)}_{pe}(\omega^{p}_{j})$ are snapshot
and paper type embeddings. The snapshot type is determined by binning the time
interval between time step $t$ and the publication time of the target paper.

####  3.3.3 Snapshot Temporal Encoder

To encode the temporal information across various snapshot times, we utilize a
4-layer Transformer Encoder. Compared with classic sequential models like RNN,
the self-attention mechanism captures more intricate relationships that extend
beyond strict ordinal dependencies:

$\begin{split}&\hat{\bm{H}}_{s}^{(l+1)}=\mathrm{MultiHeadSelfAttn}(\bm{H}_{s}^{(l+1)})\end{split}$
(7)

####  3.3.4 Final Output

We obtain the target paper representation $\bm{o}$ by avg-pooling across the
layers and then sum-pooling across the time steps, to maintain high-order
temporal information:

$\begin{split}&\tilde{\hat{\bm{H}}}_{s}=avgpool^{L}([\hat{\bm{H}}_{s}^{(1)},\hat{\bm{H}}_{s}^{(2)},\cdots,\hat{\bm{H}}_{s}^{(L)}])\\\
&\bm{o}=sumpool^{T}(\tilde{\hat{\bm{H}}}_{s})\end{split}$ (8)

###  3.4 Popularity-aware Disentanglement Module

We design the Popularity-aware Disentanglement Module (PDM) to disentangle the
citation count potentially received from different perspectives, including
citation diffusion, collective conformity, and actual contribution. Overall,
we sum up the individual predicted values to obtain the original citation
increment and apply different auxiliary tasks to help the disentanglement.
This approach enables the model to learn distinctive compositions for diverse
papers, thereby facilitating the identification and utilization of their
authentic contributions. We employ distinct Multilayer Perceptrons (MLPs) as
encoders to process information within the target perspectives, and then feed
them into another MLP separately to predict the corresponding citation
increment $\hat{y}_{i}^{v}$:

$\begin{split}&\hat{\bm{o}}_{i}^{v}=mlp^{v}_{enc}(\bm{o}_{i})\\\
&\hat{y}_{i}^{v}=mlp^{v}_{pred}(\hat{\bm{o}}_{i}^{v})\\\ \end{split}$ (9)

where $mlp^{d}_{enc}$ encodes specific aspects of the information and
$mlp^{v}_{pred}$ predicts the citation increment from the target perspective.
It is noteworthy that the three dedicated components are separately modeled to
disentangle distinct perspectives, and they are trained simultaneously to
facilitate comprehensive and concurrent learning.

####  3.4.1 Diffusion Encoder

The diffusion encoder extracts the popularity influence according to
information diffusion, which is represented as diversity affected by spreading
nodes’ degrees. We propose a Citation-aware Triplet Graph Contrastive Learning
method to extract the trunk of spreading in the dynamic graph. In various
snapshots, paper nodes are selectively omitted with probability, which are
weighted by their normalized global citations within the local subgraph.
Lowly-cited paper nodes are dropped for the positive view $z^{pos}$, as they
contribute less to spreading. Conversely, for the negative view $z^{neg}$,
highly-cited paper nodes are dropped, as they significantly impact the graph
semantics. For every sample, we contrast them solely with their augmented
views to preserve their unique characteristics from the diffusion perspective.
Through this approach, we amplify diffusion influence in the graph structure.
Contrasted with original sample $z^{ori}$, the diffusion loss
$\mathcal{L}_{dif}$ is:

$\begin{split}&\bm{z}_{i}^{ori\vee pos\vee
neg}=norm(mlp(\hat{\bm{o}}_{i}^{dif,ori\vee pos\vee neg}))\\\
&\mathcal{L}_{dif}=\frac{1}{N}\sum_{i=1}^{N}{-log\frac{exp(sim(\bm{z}^{ori},\bm{z}^{pos})/\tau)}{\sum^{2}{exp(sim(\bm{z}^{ori},\bm{z}^{neg\vee
pos})/\tau)}}}\end{split}$ (10)

where $sim(\cdot)$ is the similarity function like dot product and $\tau$ is
the temperature parameter.

####  3.4.2 Conformity Encoder

The conformity encoder extracts the popularity attributed to the reputation of
associated entities. Commonly, researchers tend to cite papers with
significant accumulated citations instead of other potentially more relevant
papers. In our work, we leverage the accumulated citations of the target paper
at the predicted time as a signal of the conformity. To extract its influence,
we divide the predicted samples into equal frequency bins to quantify Matthew
effect. By predicting the binning label $y_{im}^{con}$, we prompt the model to
learn the group differences resulting from the accumulated citations. Here,
the conformity loss $\mathcal{L}_{con}$ is:

$\begin{split}&\hat{y}_{im}^{con}=mlp(\hat{\bm{o}}_{i}^{con})\\\
&\mathcal{L}_{con}=\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}_{i}=-\frac{1}{N}\sum_{i=1}^{N}\sum_{m=1}^{M}y_{im}^{con}\log(\hat{y}_{im}^{con})\end{split}$
(11)

####  3.4.3 Contribution Encoder

After disentangling the popularity influences, the remaining citation
increment can serve as the genuine contribution. To further encourage distinct
encoding of various aspects for better approximation of contribution, we apply
an orthogonal regularization $\mathcal{L}_{ort}$ for each pair of
perspectives:

$\begin{split}&\mathcal{L}_{ort}=\frac{1}{N}\sum_{i=1}^{N}\frac{\|\hat{\bm{o}}_{i}^{v1}*\hat{\bm{o}}_{i}^{v2}\|^{2}}{\|\hat{\bm{o}}_{i}^{v1}\|*\|\hat{\bm{o}}_{i}^{v2}\|}\end{split}$
(12)

where $\|\cdot\|$ is L2-norm, $v1$ and $v2$ are each pair of three
perspectives.

####  3.4.4 Disentanglement Loss

We can obtain the final disentanglement loss $\mathcal{L}_{dis}$ by gathering
the losses of all perspectives:

$\begin{split}&\mathcal{L}_{dis}=\mathcal{L}_{dif}+\mathcal{L}_{con}+\mathcal{L}_{ort}\end{split}$
(13)

###  3.5 Training

Our task is to forecast the future increase in citations for the $i$-th paper.
Considering that the distribution of citation count is skew distribution, we
apply logarithmization to normalize the distribution. Thus, the main loss
function $\mathcal{L}_{reg}$ of the potential impact prediction task is the
mean squared error (MSE) between the target and predicted citation increment
with logarithmization: $y_{i}$ and $\hat{y}_{i}^{all}$.

$\begin{split}&\hat{y}_{i}^{all}=\hat{y}_{i}^{dif}+\hat{y}_{i}^{con}+\hat{y}_{i}\\\
&\mathcal{L}_{reg}=\frac{1}{N}\sum_{i=1}^{N}{(y_{i}-\hat{y}_{i}^{all})^{2}}\end{split}$
(14)

The final loss function $\mathcal{L}$ includes the main regression loss (MSE)
and the disentangled learning loss:

$\begin{split}&\mathcal{L}=\mathcal{L}_{reg}+\alpha\mathcal{L}_{dis}\end{split}$
(15)

where $\alpha$ controls the strength of disentanglement tasks.

##  4 Experimental Settings

In this section, we conduct extensive experiments and analyses on three
subsets extracted from the real-world scientific dataset S2ORC to validate
DPPDCC 111We will make our datasets and codes publicly available on Github..
Particularly, we plan to answer the following research questions (RQs):

$\bullet$ RQ1: Can DPPDCC improve the impact prediction task?

$\bullet$ RQ2: What is the role of each component in DPPDCC?

$\bullet$ RQ3: How sensitive is DPPDCC to hyper-parameters?

$\bullet$ RQ4: How does DPPDCC conduct the prediction?

###  4.1 Datasets

Our dataset is constructed using S2ORC (Lo et al., 2020), a comprehensive
repository comprising approximately 100 million scientific publications with
ample metadata information and unique identifiers. To optimize experimental
efficiency and account for the diverse characteristics of various domains, we
organize these publications into subsets based on their respective fields of
study. To establish the global citation network, we locate papers that possess
complete metadata (title, abstract, author IDs, and venue ID) or are high-
impact despite no venue. We select three fields from the dataset: computer
science (CS), chemistry (CHM), and psychology (PSY). The relevant statistics
are presented in Table 1.

Table 1: Detailed statistics for ground truth sample, nodes and edges of the citation network, as well as sample information of training, validation, and testing sets. dataset | CS | CHM | PSY
---|---|---|---
ground truth sample (paper) | 2,513,197 | 1,818,138 | 1,857,277
node | paper | 1,628,853 | 1,376,599 | 1,297,771
author | 1,598,925 | 1,946,073 | 1,585,595
venue | 12,524 | 8,389 | 13,775
time | 147 | 187 | 202
edge | cite | 11,534,431 | 10,382,698 | 13,401,112
write | 5,123,460 | 6,355,630 | 4,813,135
publish | 1,566,442 | 1,318,158 | 1,230,440
have | 1,628,853 | 1,376,599 | 1,297,771
sample | train (2012) | 183,105 | 227,304 | 219,499
val (2014) | 224,086 | 256,403 | 250,229
test (2017) | 300,000 | 300,000 | 300,000
qualified pool | 1,112,611 | 1,093,660 | 1,015,579

To select samples for testing, we target papers published prior to the test
observation point (see Figure 1 (b)). These papers should have complete
metadata and contain at least one reference, without any restrictions on the
citations they have received, so as to retain the genuine data distribution.
From the qualified pool, we randomly sample 300,000 papers to form the test
set. Additionally, we gather all the associated metadata for these papers. To
imitate the practical scenario, the training observation point is set 5 years
before the test observation point (equal to the observation time window $T$
and predicted interval $\Delta$) for non-overlapping. The validation
observation point is 3 years earlier than the test observation point. It is
crucial to emphasize that even though predicted samples in the training set
reappear in the test set, their input graphs and predicted values undergo
substantial variations due to the dynamic contextual changes. This inherent
challenge makes the task both challenging and significant. Moreover, we
meticulously categorize the papers in the results for thorough and detailed
analysis.

###  4.2 Baseline Models

The baseline models are mainly divided into citation/cascade prediction
models, dynamic GNN models, and disentangled GNN models:

Citation/Cascade prediction models: We apply classic and recent
citation/cascade prediction models to estimate the potential impacts of papers
based on content, citation graphs, or citation cascade graphs. SciBERT
(Beltagy et al., 2019) is fine-tuned to encode the title and abstract of the
paper. HINTS (Jiang et al., 2021) addresses the cold-start problem by applying
GCN on pseudo meta-data graphs. It uses a stochastic model to make
predictions. MUCas (Chen et al., 2022) uses time-aware sampling to extract
snapshots and utilizes MUG-caps as the encoder.

Dynamic GNN models: We employ recent dynamic GNN models to extract paper
representations from the dynamic graphs collected the same as our model,
taking the potential impact prediction as a graph regression task. EvolveGCN
(Pareja et al., 2020) integrates RNN structure within GNN modules. Dysat
(Sankar et al., 2020) incorporates multiple self-attention modules to encode
structural and temporal information. ROLAND (You et al., 2022) is a novel
dynamic heterogeneous GCN model that inherits static GCN methods.

Disentangled GNN models: We apply recent disentangled GNN models to extract
critical representations from the multi-hop subgraphs. DisenGCN (Ma et al.,
2019a) leverages neighborhood routing from capsule networks for
disentanglement. DisenHAN (Wang et al., 2020) builds upon DisenGCN and is
adapted to heterogeneous graphs for recommendation. CAL (Sui et al., 2022)
employs attention modules to estimate the causal and trivial masks for
structures and attributes and encodes them with specific GCNs separately. DisC
(Fan et al., 2022) estimates the causal/bias masks and further employs
causal/bias-aware loss functions as well as counterfactual unbiased training.
DIDA (Zhang et al., 2022) discovers invariant patterns within dynamic graphs
and applies random variant pattern swapping as the causal intervention.

###  4.3 Implementation Details

All baselines and our proposed model are implemented with PyTorch and DGL/PyG.
They are trained on an NVIDIA A800 80GB GPU and optimized with the Adam
optimizer (Kingma and Ba, 2014). We develop the models with the training and
validation sets and choose the final model with the best main metric on the
validation set. Each dataset is divided into previous papers (those included
in the training set), fresh papers (new additions in the test set), and
immediate papers (published at the test time point) for detailed analysis. We
adhere to the recommended settings for all baselines. In our model across all
datasets, the hop order $k$ is set to 2, corresponding top limits $K_{1}$ and
$K_{2}$ are set to 100 and 20, the learning rate is set to 1e-4, the number of
layers $L$ is set to 4, the co-cited/citing weight $\lambda$ is set to 0.5,
and the categories of equal frequency bins $M$ are set to 5. Additionally, the
batch size is set to 64 for CS and CHM, and 32 for PSY, and disentanglement
weight $\alpha$ is set to 0.5 for CS and PSY, and 1.0 for CHM.

###  4.4 Evaluation Metric

In our evaluation, we utilize two metrics: MALE (Mean Absolute Logarithmic
Error) and LogR2 (Logarithmic R-squared). MALE calculates the absolute error
between the target and predicted values after logarithmization:

$MALE=\frac{1}{N}\sum_{i=1}^{N}{|y_{i}-\hat{y}_{i}^{all}|}$ (16)

As for LogR2, it assesses the coefficient of determination, computed as the
proportion of variation in the dependent variable that can be predicted from
the independent variable, both after logarithmization.

$R^{2}=1-\frac{\sum_{i=1}^{N}{(y_{i}-\hat{y}_{i}^{all})^{2}}}{\sum_{i=1}^{N}{(y_{i}-\frac{1}{N}\sum_{i=1}^{N}y_{i})^{2}}}$
(17)

LogR2 becomes negative when the model’s predictions are worse than the ability
of mean value of the data, indicating an inability to capture the dataset’s
underlying trend. Given that our problem setting imitates practical scenarios
within dynamic contexts that demand the ability of extrapolation, encountering
negative values is both common and rational due to the inherent difficulty.
Therefore, by combining MALE and LogR2, we comprehensively estimate the
model’s direct prediction accuracy for individual samples and its capacity to
fit the broader trend exhibited by groups.

##  5 Results

###  5.1 Performance Comparison (RQ1)

To answer RQ1, we evaluate the performance of DPPDCC and other baselines. The
comparison results are reported in Table 2. We have the following
observations:

(1) All models perform well on previous papers presented in the training set
but experience degradation when dealing with fresh and immediate papers in the
test set, particularly in terms of the LogR2. Even the best baseline, MUCas,
struggles with immediate papers due to limited graph structures, as it is a
cascade prediction model designed for papers with sufficient citations.
However, dynamic GNN models can effectively encode multi-hop information to
complement the lack of cascade structures. Additionally, models like SciBERT
and HINTS can leverage prior knowledge, such as content information or
citation distribution, to mitigate these challenges.

(2) Both dynamic and heterogeneous graphs are important for our task. On one
side, since temporal information within dynamic graphs exposes the evolved
trend, it plays a crucial role. DIDA, which considers dynamic graphs, stands
out among the disentangled GNNs and performs most closely to MUCas across all
datasets. On the other side, leveraging heterogeneous information can enhance
DGNN performance, as demonstrated by ROLAND. However, the heterogeneous
information should be modeled properly, or it may lead to performance
degradation, which is demonstrated by DisenHAN compared with its backbone
DisenGCN in Psychology.

(3) While Disentanglement methods offer performance improvements, their
adaptation to dynamic heterogeneous graphs is essential to achieve optimal
flexibility. Notably, DIDA exhibits superior outcomes than ROLAND in LogR2 for
Chemistry, even when employing only homogeneous graphs. Moreover, in the
densest dataset, Psychology, Disentangled GNN models demonstrate commendable
performance. Among them, CAL outperforms both MUCas and ROLAND for fresh and
immediate papers. However, in other scenarios, apart from DIDA, these methods
struggle to compete with task-specific models and dynamic GNNs.

Table 2: Experimental results of performance comparison of DPPDCC with
baselines in MALE and LogR2. We divide the results into four categories:
previously, freshly, and immediately-published papers, as well as the total.
The best results are shown in bold, and the second results are underlined.
Significant improvements over best baseline results are marked with ∗.

| Computer Science | Chemistry | Psychology
---|---|---|---
MALE($\downarrow$) | total | prev | fresh | immed | total | prev | fresh | immed | total | prev | fresh | immed
SciBERT | 0.6847 | 0.6077 | 0.8054 | 0.7860 | 0.5451 | 0.4987 | 0.6903 | 0.6452 | 0.6395 | 0.6025 | 0.7402 | 0.7116
HINTS | 0.6799 | 0.6429 | 0.7378 | 0.7387 | 0.6024 | 0.5848 | 0.6574 | 0.6449 | 0.7749 | 0.7799 | 0.7614 | 0.7064
MUCas | 0.5079 | 0.4331 | 0.6250 | 0.8261 | 0.4691 | 0.4314 | 0.5869 | 0.8073 | 0.5207 | 0.4804 | 0.6306 | 0.7708
EGCN | 0.8157 | 0.8008 | 0.8392 | 0.8387 | 0.6424 | 0.6177 | 0.7196 | 0.6976 | 0.7424 | 0.7375 | 0.7560 | 0.7102
Dysat | 0.6248 | 0.5489 | 0.7437 | 0.8073 | 0.5442 | 0.5029 | 0.6733 | 0.6846 | 0.6861 | 0.6607 | 0.7555 | 0.7439
ROLAND | 0.5814 | 0.5166 | 0.6828 | 0.7157 | 0.5001 | 0.4592 | 0.6280 | 0.6406 | 0.5643 | 0.4996 | 0.7406 | 0.7926
DisenGCN | 0.8215 | 0.7816 | 0.8841 | 0.8485 | 0.6405 | 0.6079 | 0.7427 | 0.6987 | 0.6557 | 0.6316 | 0.7213 | 0.7081
DisenHAN | 0.8138 | 0.7411 | 0.9276 | 1.0686 | 0.6132 | 0.5817 | 0.7118 | 0.7313 | 0.7763 | 0.8077 | 0.6906 | 0.6476
CAL | 0.7323 | 0.6859 | 0.8050 | 0.7940 | 0.5864 | 0.5541 | 0.6873 | 0.6535 | 0.5631 | 0.5469 | 0.6073 | 0.6285
DisC | 0.6687 | 0.6144 | 0.7536 | 0.7900 | 0.5393 | 0.5037 | 0.6505 | 0.6409 | 0.5415 | 0.5159 | 0.6113 | 0.6369
DIDA | 0.5941 | 0.5018 | 0.7387 | 0.8181 | 0.4978 | 0.4642 | 0.6028 | 0.6426 | 0.5878 | 0.5604 | 0.6624 | 0.6681
DPPDCC | 0.4473 | 0.3783 | 0.5553 | 0.6035 | 0.4355 | 0.4071 | 0.5243 | 0.5802 | 0.4463 | 0.4138 | 0.5349 | 0.5633
#improve (%) | 11.93∗ | 12.65∗ | 11.15∗ | 15.68∗ | 7.16∗ | 5.64∗ | 10.67∗ | 9.42∗ | 14.29∗ | 13.86∗ | 11.92∗ | 10.38∗
LogR2($\uparrow$) | total | prev | fresh | immed | total | prev | fresh | immed | total | prev | fresh | immed
SciBERT | 0.3558 | 0.4424 | 0.1108 | 0.0843 | 0.4648 | 0.4709 | 0.1477 | 0.1653 | 0.4308 | 0.5041 | 0.0068 | -0.1095
HINTS | 0.2773 | 0.2028 | 0.2268 | 0.1718 | 0.3795 | 0.3119 | 0.2253 | 0.1633 | 0.1988 | 0.2007 | -0.0259 | -0.0804
MUCas | 0.6063 | 0.6757 | 0.4352 | -0.1079 | 0.6168 | 0.6306 | 0.3624 | -0.3479 | 0.6175 | 0.6846 | 0.2669 | -0.3521
EGCN | 0.1174 | 0.0531 | 0.0210 | -0.1460 | 0.2684 | 0.1913 | 0.0791 | 0.0035 | 0.2305 | 0.2469 | -0.0388 | -0.1386
Dysat | 0.4355 | 0.5029 | 0.2321 | 0.0089 | 0.4573 | 0.4411 | 0.1998 | 0.0582 | 0.2998 | 0.3342 | -0.0170 | -0.2377
ROLAND | 0.5245 | 0.5634 | 0.3764 | 0.2511 | 0.5410 | 0.5374 | 0.2942 | 0.1800 | 0.5540 | 0.6601 | 0.0429 | -0.3765
DisenGCN | 0.1428 | 0.1022 | 0.0206 | 0.0236 | 0.2735 | 0.2151 | 0.0332 | 0.0387 | 0.3945 | 0.4364 | 0.0758 | -0.0846
DisenHAN | 0.0054 | 0.1003 | -0.3220 | -0.8032 | 0.3793 | 0.3683 | 0.0630 | -0.1466 | 0.2455 | 0.2187 | 0.1387 | 0.0522
CAL | 0.2298 | 0.2264 | 0.0768 | 0.0427 | 0.3928 | 0.3707 | 0.1158 | 0.1135 | 0.5646 | 0.5975 | 0.3256 | 0.1166
DisC | 0.3434 | 0.3908 | 0.1472 | -0.0735 | 0.4770 | 0.4578 | 0.2387 | 0.1719 | 0.5915 | 0.6343 | 0.3233 | 0.0963
DIDA | 0.4703 | 0.5642 | 0.2391 | -0.0207 | 0.5760 | 0.5689 | 0.3587 | 0.1669 | 0.5118 | 0.5620 | 0.1950 | -0.0092
DPPDCC | 0.6871 | 0.7236 | 0.5754 | 0.4512 | 0.6503 | 0.6329 | 0.5042 | 0.3247 | 0.7037 | 0.7454 | 0.4701 | 0.2733
#improve (%) | 13.32∗ | 7.09∗ | 32.22∗ | 79.67∗ | 5.44∗ | 0.36∗ | 39.13∗ | 80.39∗ | 13.96∗ | 8.87∗ | 44.38∗ | 134.40∗

(4) Our proposed model surpasses all baselines, by effectively modeling the
dynamic heterogeneous graph and incorporating task-adjusted disentanglement.
This leads to an average improvement of about 10% in MALE across all
categories, along with a substantial advancement of over 30% in LogR2 for
fresh and immediate papers.  First, compared with traditional citation/cascade
prediction models, it exhibits superior generalization and practicability
through the fusion of content and context information within citation
networks, as well as the disentanglement power. Furthermore, we consider the
unique properties of citation networks to encode complex high-order structural
and temporal information, thereby outperforming common DGNNs. More
importantly, our model introduces interpretability from disentanglement models
without compromising performance and further adapts to the task for enhanced
improvement.

###  5.2 Ablation Test (RQ2)

To answer RQ2, ablation tests are performed on both CGE and PDM, highlighting
the significance of various modules. As demonstrated in Table 3, for CGE,
(-co-cited/citing) removes the proposed co-cited/citing strengths, (-CompGAT)
replaces the CompGAT with original GATv2 (Brody et al., 2021), and (-readout)
simply obtains the snapshot representation through avg-pooling over all paper
nodes without filtering. For PDM, (-diffusion) removes the diffusion
disentanglement, (-conformity) omits the conformity disentanglement, (-orthog)
disregards the orthogonal constraints, and (-disen) eliminates all
perspectives and replaces with a single MLP for prediction.

Table 3: Results of ablation studies according to MALE. model | CS | CHM | PSY
---|---|---|---
ours | 0.4473 | 0.4355 | 0.4463
CGE | -co-cited/citing | 0.4650 | 0.5014 | 0.4528
-CompGAT | 0.4722 | 0.4990 | 0.4582
-readout | 0.5116 | 0.4404 | 0.4932
PDM | -diffusion | 0.4593 | 0.4530 | 0.4573
-conformity | 0.5177 | 0.4747 | 0.4580
-orthog | 0.4930 | 0.4388 | 0.4502
-disen | 0.4606 | 0.4447 | 0.4670

We observe that: (1) Since removing any module leads to a performance
decrease, all modules play crucial roles in DPPDCC. (2) For CGE, the removal
of Type-specific Attention Readout may cause a notable performance drop. That
could be attributed to the presence of excessive noisy information within
multi-hop snapshots, which necessitates the filtering of aggregated
information for better concentration. Specially, in chemistry, since the
subgraphs are sparser with less information, the simple avg-pooling might be
sufficient to extract meaningful information. Furthermore, our proposed
CompGAT effectively captures distinct comparative information, and the
incorporation of co-cited/citing strengths further complements the multi-hop
information beyond the local neighborhood. (3) Eliminating any disentanglement
task adversely affects performance. Moreover, an evident decline is observed
after removing the whole disentanglement module, especially in Psychology.
Conformity rises as the paramount task among all disentanglement tasks. Its
focuses on the Matthew effect, which is widely pervasive and readily
discernible in reality.

###  5.3 Hype-parameters Test (RQ3)

To address RQ3, we aim to explore the model’s sensitivity to essential hyper-
parameters, such as $L$, $\lambda$, and $\alpha$, which respectively control
the order/hop information of CGE, the importance of co-cited/citing strengths,
and the weight of the disentanglement tasks relative to the main task. The
results about MALE with varying settings are depicted in Figure 3. We observe
that: (1) When the layer number $L$ continues to increase, the performance
will first increase and then decrease. Since we stack the structural and
temporal encoders simultaneously, the interaction between them is more
complicated and may also face the over-smoothing problem. The best number of
layers is 4 across all datasets. (2) The co-cited/citing weight $\lambda$ and
disentanglement weight $\alpha$ demonstrate greater stability compared with
$L$, especially in Psychology. Selecting the proper hyper-parameters can
further improve the performance, with 0.5 consistently proving to be a
moderate and effective choice.

(a) $L$

(b) $\lambda$

(c) $\alpha$

Figure 3: Results of hyper-parameters test in terms of MALE.

###  5.4 Disentanglement Visualization (RQ4)

To answer RQ4, we visualize the disentanglement composition with all positive
values in the test set of Chemistry. For papers containing negative
disentangled values, they always have nearly no increased citations.
Remarkably, over 80% of them feature negative because their contribution
values are the lowest across all perspectives, demonstrating the rationality
of our proposed method. For the stacked plot associated with time in Figure
4a, a clear trend emerges: the contribution proportion gradually increases,
while the conformity proportion undergoes a significant drop. This trend is
sensible since old papers tend to accumulate substantial citations, thus
attracting more researchers to cite them due to inertia. Additionally, the
diffusion proportion appears to be more stable across different times, as it
represents the potential hidden properties within diffusion. This stability
becomes more pronounced when papers are categorized into previous, fresh, and
immediate ones in Figure 4b. Diffusion proportions remain steady in both mean
and standard deviation. Moreover, contribution emerges as the most critical
perspective, with its dominant proportion. It further holds particular
significance for fresh and immediate papers. For immediate papers receiving
almost no citations, both the diffusion and conformity perspectives struggle
to offer predictive insights. However, the contribution enables the model to
anticipate the potential of samples in advance by analyzing their citation
context. This capability enhances the model’s robustness, particularly for
challenging samples close to cold-start scenarios. Furthermore, in Figure 4c,
when we bin the samples based on the predicted logarithmic values, we observe
a descending trend for conformity and a growing trend for contribution.
Notably, within the lowest bin group, the conformity proportion surpasses the
contribution proportion for the first time. The alignment between time and
impact groups can unveil an overall trend: in dynamic contexts, previous
papers tend to attract fewer new citations and may primarily owe to their
conformity, whereas recent and immediate papers are more likely to garner
increased citations owing to their genuine contributions.

(a) time trend

(b) group detail

(c) value bin

Figure 4: Visualization of disentangled value percentages in Chemistry. (a)
displays the trend that evolved with the publication time. (b) demonstrates
the detailed composition of papers categorized into previous, fresh, and
immediate ones. (c) is binning the samples based on the predicted values.

##  6 Conclusion

In this study, we propose a novel model named DPPDCC that disentangles the
potential impacts of papers into diffusion, conformity, and contribution
values. To better align with real-world conditions, we reformulate the problem
by partitioning the datasets based on the observation time point, without
excluding lowly-cited papers. Further, DPPDCC comprises two principal
components: Citation-aware GNN Encoder (CGE) and Popularity-aware
Disentanglement Module (PDM). Given the dynamic heterogeneous graph of a
target paper, CGE employs CompGAT to encode the comparative and co-
cited/citing information between papers, and utilizes Type-specific Snapshot
Readout to aggregate the information from each snapshot evolutionarily. Then,
PDM extracts the amplifying effects within information diffusion, and handles
the Matthew effect arising from the accumulated citations as the perspective
of collective conformity. Finally, it constrains each perspective to address
distinct aspects for preserving the genuine contribution. Experimental results
on three datasets demonstrate that DPPDCC significantly outperforms
alternative approaches for previous, fresh, and immediate papers.
Disentanglement visualization further demonstrates that DPPDCC can reliably
perform predictions.

In our future endeavors, we intend to develop extensive datasets spanning
diverse fields of study, aiming to simulate practical scenarios. This
initiative is geared towards enhancing the practicality and relevance of our
approach significantly. Additionally, we plan to incorporate additional
factors that consider citation intent, thereby enabling a more precise
evaluation of the genuine contributions made by academic papers. Furthermore,
we will conduct a thorough analysis of the model’s performance in identifying
novel papers within specific domains, comparing the results with relevant
studies. This comparative analysis will serve to elucidate the practical
implications of our proposed methodology.

## References

  * Abrishami and Aliakbary (2019) Abrishami, A., Aliakbary, S., 2019.  Predicting citation counts based on deep neural network learning techniques.  Journal of Informetrics 13, 485–499.
  * Allison and Stewart (1974) Allison, P.D., Stewart, J.A., 1974.  Productivity differences among scientists: Evidence for accumulative advantage.  American sociological review , 596–606.
  * Beltagy et al. (2019) Beltagy, I., Lo, K., Cohan, A., 2019.  Scibert: A pretrained language model for scientific text, in: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3615–3620.
  * Bengio et al. (2013) Bengio, Y., Courville, A., Vincent, P., 2013.  Representation learning: A review and new perspectives.  IEEE transactions on pattern analysis and machine intelligence 35, 1798–1828.
  * Bornmann and Daniel (2008) Bornmann, L., Daniel, H.D., 2008.  What do citation counts measure? a review of studies on citing behavior.  Journal of documentation 64, 45–80.
  * Brody et al. (2021) Brody, S., Alon, U., Yahav, E., 2021.  How attentive are graph attention networks?  International Conference on Learning Representations .
  * Cai et al. (2019) Cai, R., Li, Z., Wei, P., Qiao, J., Zhang, K., Hao, Z., 2019.  Learning disentangled semantic representation for domain adaptation, in: IJCAI: proceedings of the conference, NIH Public Access. p. 2060.
  * Cao et al. (2017) Cao, Q., Shen, H., Cen, K., Ouyang, W., Cheng, X., 2017.  Deephawkes: Bridging the gap between prediction and understanding of information cascades, in: Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pp. 1149–1158.
  * Case and Higgins (2000) Case, D.O., Higgins, G.M., 2000.  How can we investigate citation behavior? a study of reasons for citing literature in communication.  Journal of the American Society for Information Science 51, 635–645.
  * Chen et al. (2022) Chen, X., Zhang, F., Zhou, F., Bonsangue, M., 2022.  Multi-scale graph capsule with influence attention for information cascades prediction.  International Journal of Intelligent Systems 37, 2584–2611.
  * Chu and Evans (2021) Chu, J.S., Evans, J.A., 2021.  Slowed canonical progress in large fields of science.  Proceedings of the National Academy of Sciences 118.
  * Denton et al. (2017) Denton, E.L., et al., 2017.  Unsupervised learning of disentangled representations from video.  Advances in neural information processing systems 30.
  * Evans and Reimer (2009) Evans, J.A., Reimer, J., 2009.  Open access and global participation in science.  Science 323, 1025–1025.
  * Fan et al. (2022) Fan, S., Wang, X., Mo, Y., Shi, C., Tang, J., 2022.  Debiasing graph neural networks via learning disentangled causal substructure.  Advances in Neural Information Processing Systems 35, 24934–24946.
  * Glänzel and Schubert (1995) Glänzel, W., Schubert, A., 1995.  Predictive aspects of a stochastic model for citation processes.  Information processing & management 31, 69–80.
  * Gori et al. (2005) Gori, M., Monfardini, G., Scarselli, F., 2005.  A new model for learning in graph domains, in: Proceedings. 2005 IEEE international joint conference on neural networks, pp. 729–734.
  * Gu et al. (2016) Gu, J., Lu, Z., Li, H., Li, V.O., 2016.  Incorporating copying mechanism in sequence-to-sequence learning.  Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , 1631–1640.
  * Guille et al. (2013) Guille, A., Hacid, H., Favre, C., Zighed, D.A., 2013.  Information diffusion in online social networks: A survey.  ACM Sigmod Record 42, 17–28.
  * Hu et al. (2020) Hu, Z., Dong, Y., Wang, K., Sun, Y., 2020.  Heterogeneous graph transformer, in: Proceedings of the web conference 2020, pp. 2704–2710.
  * Huang et al. (2022) Huang, S., Huang, Y., Bu, Y., Lu, W., Qian, J., Wang, D., 2022.  Fine-grained citation count prediction via a transformer-based model with among-attention mechanism.  Information Processing & Management 59, 102799.
  * Jiang et al. (2021) Jiang, S., Koch, B., Sun, Y., 2021.  Hints: citation time series prediction for new publications via dynamic heterogeneous information network embedding, in: Proceedings of the Web Conference 2021, pp. 3158–3167.
  * Kingma and Ba (2014) Kingma, D.P., Ba, J., 2014.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980 .
  * Lee et al. (2021) Lee, J., Kim, E., Lee, J., Lee, J., Choo, J., 2021.  Learning debiased representation via disentangled feature augmentation.  Advances in Neural Information Processing Systems 34, 25123–25133.
  * Li et al. (2017) Li, C., Ma, J., Guo, X., Mei, Q., 2017.  Deepcas: An end-to-end predictor of information cascades, in: Proceedings of the 26th international conference on World Wide Web, pp. 577–586.
  * Lo et al. (2020) Lo, K., Wang, L.L., Neumann, M., Kinney, R., Weld, D.S., 2020.  S2orc: The semantic scholar open research corpus, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4969–4983.
  * Locatello et al. (2019) Locatello, F., Bauer, S., Lucic, M., Raetsch, G., Gelly, S., Schölkopf, B., Bachem, O., 2019.  Challenging common assumptions in the unsupervised learning of disentangled representations, in: international conference on machine learning, PMLR. pp. 4114–4124.
  * Ma et al. (2019a) Ma, J., Cui, P., Kuang, K., Wang, X., Zhu, W., 2019a.  Disentangled graph convolutional networks, in: International conference on machine learning, PMLR. pp. 4212–4221.
  * Ma et al. (2019b) Ma, J., Zhou, C., Cui, P., Yang, H., Zhu, W., 2019b.  Learning disentangled representations for recommendation.  Advances in neural information processing systems 32.
  * Manessi et al. (2020) Manessi, F., Rozza, A., Manzo, M., 2020.  Dynamic graph convolutional networks.  Pattern Recognition 97, 107000.
  * Page et al. (1998) Page, L., Brin, S., Motwani, R., Winograd, T., 1998.  The pagerank citation ranking: Bring order to the web.  Technical Report. Technical report, stanford University.
  * Pareja et al. (2020) Pareja, A., Domeniconi, G., Chen, J., Ma, T., Suzumura, T., Kanezashi, H., Kaler, T., Schardl, T., Leiserson, C., 2020.  Evolvegcn: Evolving graph convolutional networks for dynamic graphs, in: Proceedings of the AAAI Conference on Artificial Intelligence, pp. 5363–5370.
  * Reimers and Gurevych (2019) Reimers, N., Gurevych, I., 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , 3982–3992.
  * Ruan et al. (2020) Ruan, X., Zhu, Y., Li, J., Cheng, Y., 2020.  Predicting the citation counts of individual papers via a bp neural network.  Journal of Informetrics 14, 101039.
  * Sankar et al. (2020) Sankar, A., Wu, Y., Gou, L., Zhang, W., Yang, H., 2020.  Dysat: Deep neural representation learning on dynamic graphs via self-attention networks, in: Proceedings of the 13th international conference on web search and data mining, pp. 519–527.
  * Scarselli et al. (2008) Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G., 2008.  The graph neural network model.  IEEE transactions on neural networks 20, 61–80.
  * Schlichtkrull et al. (2018) Schlichtkrull, M., Kipf, T.N., Bloem, P., Van Den Berg, R., Titov, I., Welling, M., 2018.  Modeling relational data with graph convolutional networks, in: European semantic web conference, Springer. pp. 593–607.
  * Silagadze (1997) Silagadze, Z., 1997.  Citations and the zipf–mandelbrot law.  Complex Systems 11, 487–499.
  * Sinatra et al. (2016) Sinatra, R., Wang, D., Deville, P., Song, C., Barabási, A.L., 2016.  Quantifying the evolution of individual scientific impact.  Science 354, aaf5239.
  * Sperduti and Starita (1997) Sperduti, A., Starita, A., 1997.  Supervised neural networks for the classification of structures.  IEEE Transactions on Neural Networks 8, 714–735.
  * Sui et al. (2022) Sui, Y., Wang, X., Wu, J., Lin, M., He, X., Chua, T.S., 2022.  Causal attention for interpretable and generalizable graph classification, in: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1696–1705.
  * Tatar et al. (2014) Tatar, A., De Amorim, M.D., Fdida, S., Antoniadis, P., 2014.  A survey on predicting the popularity of web content.  Journal of Internet Services and Applications 5, 1–20.
  * Thornley et al. (2015) Thornley, C., Watkinson, A., Nicholas, D., Volentine, R., Jamali, H.R., Herman, E., Allard, S., Levine, K., Tenopir, C., 2015.  The role of trust and authority in the citation behaviour of researchers.  Information research .
  * Van Raan (2004) Van Raan, A.F., 2004.  Sleeping beauties in science.  Scientometrics 59, 467–472.
  * Wang et al. (2013) Wang, D., Song, C., Barabási, A.L., 2013.  Quantifying long-term scientific impact.  Science 342, 127–132.
  * Wang et al. (2020) Wang, Y., Tang, S., Lei, Y., Song, W., Wang, S., Zhang, M., 2020.  Disenhan: Disentangled heterogeneous graph attention network for recommendation, in: Proceedings of the 29th ACM international conference on information & knowledge management, pp. 1605–1614.
  * Wen et al. (2022) Wen, Q., Ouyang, Z., Zhang, J., Qian, Y., Ye, Y., Zhang, C., 2022.  Disentangled dynamic heterogeneous graph learning for opioid overdose prediction, in: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2009–2019.
  * Wu et al. (2019) Wu, L., Wang, D., Evans, J.A., 2019.  Large teams develop and small teams disrupt science and technology.  Nature 566, 378–382.
  * Wu et al. (2020) Wu, Z., Pan, S., Chen, F., Long, G., Zhang, C., Philip, S.Y., 2020.  A comprehensive survey on graph neural networks.  IEEE transactions on neural networks and learning systems 32, 4–24.
  * Xu et al. (2018) Xu, K., Hu, W., Leskovec, J., Jegelka, S., 2018.  How powerful are graph neural networks?  International Conference on Learning Representations .
  * Xu et al. (2022) Xu, X., Zhou, F., Zhang, K., Liu, S., 2022.  Ccgl: Contrastive cascade graph learning.  IEEE Transactions on Knowledge and Data Engineering .
  * Xue et al. (2023) Xue, Z., He, G., Liu, J., Jiang, Z., Zhao, S., Lu, W., 2023.  Re-examining lexical and semantic attention: Dual-view graph convolutions enhanced bert for academic paper rating.  Information Processing & Management 60, 103216.
  * Yan et al. (2011) Yan, R., Tang, J., Liu, X., Shan, D., Li, X., 2011.  Citation count prediction: learning to estimate future citations for literature, in: Proceedings of the 20th ACM international conference on Information and knowledge management, pp. 1247–1252.
  * Yang et al. (2022) Yang, C., Bao, P., Yan, R., Li, J., Li, X., 2022.  A graph temporal information learning framework for popularity prediction, in: Companion Proceedings of the Web Conference 2022, pp. 239–242.
  * Yang et al. (2020) Yang, Y., Feng, Z., Song, M., Wang, X., 2020.  Factorizable graph convolutional networks.  Advances in Neural Information Processing Systems 33, 20286–20296.
  * You et al. (2022) You, J., Du, T., Leskovec, J., 2022.  Roland: graph learning framework for dynamic graphs, in: Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 2358–2366.
  * Yu et al. (2014) Yu, T., Yu, G., Li, P.Y., Wang, L., 2014.  Citation impact prediction for scientific papers using stepwise regression analysis.  Scientometrics 101, 1233–1252.
  * Zhang et al. (2022) Zhang, Z., Wang, X., Zhang, Z., Li, H., Qin, Z., Zhu, W., 2022.  Dynamic graph neural networks under spatio-temporal distribution shift, in: Advances in Neural Information Processing Systems.
  * Zhou et al. (2021) Zhou, F., Xu, X., Trajcevski, G., Zhang, K., 2021.  A survey of information cascade analysis: Models, predictions, and recent advances.  ACM Computing Surveys (CSUR) 54, 1–36.