# Do the rich get richer? An empirical analysis of the Bitcoin transaction
network

Dániel Kondor<EMAIL_ADDRESS>Department of Physics of Complex Systems,
Eötvös Loránd University, Hungary
H-1117 Budapest, Pázmány Péter Sétány 1/A  Márton Pósfai  Department of
Physics of Complex Systems, Eötvös Loránd University, Hungary
H-1117 Budapest, Pázmány Péter Sétány 1/A  Department of Theoretical Physics,
Budapest University of Technology and Economics,
Budafoki út 8, Budapest H–1111, Hungary  István Csabai  Department of Physics
of Complex Systems, Eötvös Loránd University, Hungary
H-1117 Budapest, Pázmány Péter Sétány 1/A  Gábor Vattay  Department of Physics
of Complex Systems, Eötvös Loránd University, Hungary
H-1117 Budapest, Pázmány Péter Sétány 1/A

(August 27, 2024)

## Abstract

The possibility to analyze everyday monetary transactions is limited by the
scarcity of available data, as this kind of information is usually considered
highly sensitive. Present econophysics models are usually employed on presumed
random networks of interacting agents, and only some macroscopic properties
(e.g. the resulting wealth distribution) are compared to real-world data. In
this paper, we analyze Bitcoin, which is a novel digital currency system,
where the complete list of transactions is publicly available. Using this
dataset, we reconstruct the network of transactions and extract the time and
amount of each payment. We analyze the structure of the transaction network by
measuring network characteristics over time, such as the degree distribution,
degree correlations and clustering. We find that linear preferential
attachment drives the growth of the network. We also study the dynamics taking
place on the transaction network, i.e. the flow of money. We measure temporal
patterns and the wealth accumulation. Investigating the microscopic statistics
of money movement, we find that sublinear preferential attachment governs the
evolution of the wealth distribution. We report a scaling law between the
degree and wealth associated to individual nodes.

##  I Introduction

In the past two decades, network science has successfully contributed to many
diverse scientific fields. Indeed, many complex systems can be represented as
networks, ranging from biochemical systems, through the Internet and the World
Wide Web, to various social systems barabasi_review ; newman_reveiw ;
barabasibio ; pastor_book ; vespignani ; havlin ; critnet . Economics also
made use of the concepts of network science, gaining additional insight to the
more traditional approach networkfinancial1 ; networkfinancial2 ;
networkfinancial3 ; networkfinancial4 ; networkfinancial5 ; palla . Although a
large volume of financial data is available for research, information about
the everyday transactions of individuals is usually considered sensitive and
is kept private. In this paper, we analyze Bitcoin, a novel currency system,
where the complete list of transactions is accessible. We believe that this is
the first opportunity to investigate the movement of money in such detail.

Bitcoin is a decentralized digital cash system, there is no single overseeing
authority satoshi . The system operates as an online peer-to-peer network,
anyone can join by installing a client application and connecting it to the
network. The unit of the currency is one bitcoin (abbreviated as BTC), and the
smallest transferable amount is $10^{-8}\,\textrm{BTC}$. Instead of having a
bank account maintained by a central authority, each user has a Bitcoin
address, that consists of a pair of public and private keys. Existing bitcoins
are associated to the public key of their owner, and outgoing payments have to
be signed by the owner using his private key. To maintain privacy, a single
user may use multiple addresses. Each participating node stores the complete
list of previous transactions. Every new payment is announced on the network,
and the payment is validated by checking consistency with the entire
transaction history. To avoid fraud, it is necessary that the participants
agree on a single valid transaction history. This process is designed to be
computationally difficult, so an attacker can only hijack the system if he
possesses the majority of the computational power of participating parties.
Therefore the system is more secure if more resources are devoted to the
validation process. To provide incentive, new bitcoins are created
periodically and distributed among the nodes participating in these
computations. Another way to obtain bitcoins is to purchase them from someone
who already has bitcoins using traditional currency; the price of bitcoins is
completely determined by the market.

The Bitcoin system was proposed in 2008 by Satoshi Nakamoto, and the system
went online in January 2009 satoshi ; nakamoto ; shamir ; anonimity ;
structure . For over a year, it was only used by a few enthusiasts, and
bitcoins did not have any real-world value. A trading website called MtGox was
started in 2010, making the exchange of bitcoins and conventional money
significantly easier. More people and services joined the system, resulting a
steadily growing exchange rate. Starting from 2011, appearances in the
mainstream media drew wider public attention, which led to skyrocketing prices
accompanied by large fluctuations (see Fig. 1). Since the inception of Bitcoin
over 17 million transactions took place, and currently the market value of all
bitcoins in circulation exceeds 1 billion dollars. See the Methods section for
more details of the system and the data used in our analysis.

We download the complete list of transactions, and reconstruct the transaction
network: each node represents a Bitcoin address, and we draw a directed link
between two nodes if there was at least one transaction between the
corresponding addresses. In addition to the topology, we also obtain the time
and amount of every payment. Therefore, we are able to analyze both the
evolution of the network and the dynamical process taking place on it, i.e.
the flow and accumulation of bitcoins. To characterize the underlying network,
we investigate the evolution of basic network characteristics over time, such
as the degree distribution, degree correlations and clustering. Concerning the
dynamics, we measure the wealth statistics and the temporal patterns of
transactions. To explain the observed degree and wealth distribution, we
measure the microscopic growth statistics of the system. We provide evidence
that preferential attachment is an important factor shaping these
distributions. Preferential attachment is often referred to as the “rich get
richer” scheme, meaning that hubs grow faster than low-degree nodes. In the
case of Bitcoin, this is more than an analogy: we find that the wealth of
already rich nodes increases faster than the wealth of nodes with low balance;
furthermore, we find positive correlation between the wealth and the degree of
a node.

##  II Results

###  II.1 Evolution of the transaction network

Bitcoin is an evolving network: new nodes are added by creating new Bitcoin
addresses, and links are created if there is a transaction between two
previously unconnected addresses. The number of nodes steadily grows over time
with some fluctuations; especially noticeable is the large peak which
coincides with the first boom in the exchange rate in 2011 (Fig. 1) structure
. After five years Bitcoin now has $N=13,086,528$ nodes and $L=44,032,115$
links. To study the evolution of the network we measure the change of network
characteristics in function of time. We identify two distinct phases of
growth: (i) The _initial phase_ lasted until the fall of 2010, in this period
the system had low activity and was mostly used as an experiment. The network
measures are characterized by large fluctuations. (ii) After the initial phase
the Bitcoin started to function as a real currency, bitcoins gained real
value. The network measures converged to their typical value by mid-2011 and
they did not change significantly afterwards. We call this period the _trading
phase_.

Figure 1: The growth of the Bitcoin network. Number of addresses with nonzero
balance (green), addresses in participating in at least one transaction in one
week intervals (red) and the exchange price of bitcoins in US dollars
according to MtGox, the largest Bitcoin exchange site (blue). The black lines
are exponential functions bounding the growth of the network size; the
characteristic times are $188$ and $386$ days. Figure 2: Evolution of the
indegree distribution. Since the beginning of 2011, the shape of the
distribution does not change significantly. The black line shows a fitted
power-law for the final network; the exponent is $2.18$. The data is log-
binned for ease of visual inspection, the power-law is fitted on the original
data powerlaw . Figure 3: Evolution of the outdegree distribution. The black
line shows a fitted power-law for the final network; the exponent is $2.06$.
The data is log-binned for ease of visual inspection, the power-law is fitted
on the original data powerlaw . Figure 4: Evolution of the Gini coefficient of
the degree and the balance distributions. We observe the distinct initial
phase lasting until mid-2011. The trading phase is characterized by
approximately constant coefficients.

We first measure the degree distribution of the network. We find that both the
in- and the outdegree distributions are highly heterogeneous, and they can be
modeled with power-laws powerlaw . Figures 2 and 3 show the distribution of
indegrees and outdegrees at different points of time during the evolution of
the Bitcoin network. In the initial phase the number of nodes is low, and thus
fitting the data is prone to large error. In the trading phase, the exponents
of the distributions do not change significantly, and they are approximated by
power-laws $p_{\text{in}}(k_{\text{in}})\sim k_{\text{in}}^{-2.18}$ and
$p_{\text{out}}(k_{\text{out}})\sim k_{\text{out}}^{-2.06}$.

To further characterize the evolution of the degree distributions we calculate
the corresponding Gini coefficients in function of time (Fig. 4). The Gini
coefficient is mainly used in economics to characterize the inequality present
in the distribution of wealth, but it can be used to measure the heterogeneity
of any empirical distribution. In general, the Gini coefficient is defined as

$G=\frac{2\sum_{i=1}^{n}ix_{i}}{n\sum_{i=1}^{n}x_{i}}-\frac{n+1}{n}$ (1)

where $\\{x_{i}\\}$ is a sample of size $n$, and $x_{i}$ are monotonically
ordered, i.e. $x_{i}\leq x_{i+1}$. $G=0$ indicates perfect equality, i.e.
every node has the same wealth; and $G=1$ corresponds to complete inequality,
i.e. the complete wealth in the system is owned by a single individual. For
example, in the case of pure power-law distribution with $\alpha\geq 2$
exponent, the Gini coefficient is $G=1/(2\alpha-3)$ paretogini . This shows
the fact that smaller $\alpha$ exponents yield more heterogeneous wealth
distributions.

In the Bitcoin network we find that in the initial phase the Gini coefficient
of the indegree distribution is close to 1 and for the outdegree distribution
it is much lower. We speculate that in this phase a few users collected
bitcoins, and without the possibility to trade, they stored them on a single
address. In the second phase the coefficients quickly converge to
$G^{\text{in}}\approx 0.629$ and $G^{\text{out}}\approx 0.521$, indicating
that normal trade is characterized by both highly heterogeneous in- and
outdegree distributions.

Figure 5: Evolution of the clustering coefficient and the out-in degree
correlation coefficient. After the initial phase, both measures reach a
stationary value. Figure 6: The average indegree of neighbors in the function
of the outdegree $k^{\text{in}}_{\text{nn}}(k^{\text{out}})$. In networks
without degree correlations, the degree of connected nodes do not depend on
each other, therefore for such networks we expect that
$k^{\text{in}}_{\text{nn}}(k^{\text{out}})$ is constant. In the case of the
Bitcoin network, we observe a clear disassortative behavior:
$k^{\text{in}}_{\text{nn}}(k^{\text{out}})$ is a decreasing function,
indicating that nodes with high outdegree tend to connect to nodes with low
indegree.

To characterize the degree correlations we measure the Pearson correlation
coefficient of the out- and indegrees of connected node pairs:

$r=\frac{\sum_{e}(j^{\textrm{out}}_{e}-\overline{j^{\textrm{out}}})(k^{\textrm{in}}_{e}-\overline{k^{\textrm{in}}})}{L\sigma_{\text{out}}\sigma_{\text{in}}}.$
(2)

Here $j^{\textrm{out}}_{i}$ is the outdegree of the node at the _beginning_ of
link $e$, and $k^{\textrm{in}}_{i}$ is the indegree of the node at the _end_
of link $e$. The summation $\sum_{e}\cdot$ runs over all links,
$\overline{k^{\textrm{in}}}=\sum_{e}k^{\textrm{in}}_{e}/L$ and
$\sigma_{\text{in}}^{2}=\sum_{e}(k^{\textrm{in}}_{e}-\overline{k^{\textrm{in}}})^{2}/L$.
We calculate $\sigma_{\text{out}}$ and $\overline{j^{\textrm{out}}}$
similarly.

We find that the correlation coefficient is negative, except for only a brief
period in the initial phase. After mid-2010, the degree correlation
coefficient stays between $-0.01$ and $-0.05$, reaching a value of
$r\approx-0.014$ by 2013, suggesting that the network is disassortative (Fig.
5). However, small values of $r$ are hard to interpret: it was shown that for
large purely scale-free networks $r$ vanishes as the network size increases
menche . Therefore we compute the average nearest neighbor degree function
$k^{\text{in}}_{\text{nn}}(k^{\text{out}})$ for the final network;
$k^{\text{in}}_{\text{nn}}(k^{\text{out}})$ measures the average indegree of
the neighbors of nodes with outdegree $k^{\text{out}}$. We find clear
disassortative behavior (Fig. 6).

We also measure the average clustering coefficient

$C=\frac{1}{N}\sum_{v}\frac{\Delta_{v}}{d_{v}(d_{v}-1)/2},$ (3)

which measures the density of triangles in the network. Here the sum
$\sum_{v}\cdot$ runs over all nodes, and $\Delta_{v}$ is the number of
triangles containing node $v$. To calculate $\Delta_{v}$ we ignored the
directionality of the links; $d_{v}$ is the degree of node $v$ in the
undirected network.

\begin{overpic}[unit=1mm]{pa6a10y0a}
\put(12.0,27.0){\includegraphics{pa6inset}} \end{overpic} Figure 7: Rank
function for new link creation. The cumulative distribution function of the
$R$ values (see Eq. 5) for exponents $0.7,0.85,1,1.1,1.2$ and $1.35$. The
inset shows the Kolmogorov-Smirnoff error for these exponents.

In the initial phase $C$ is high, fluctuating around $0.15$ (see Fig. 5),
possibly a result of transactions taking place between addresses belonging to
a few enthusiasts trying out the Bitcoin system by moving money between their
own addresses. In the trading phase, the clustering coefficient reaches a
stationary value around $C\approx 0.05$, which is still higher than the
clustering coefficient for random networks with the same degree sequence
($C_{\text{rand}}\approx 0.0037(9)$).

To explain the observed broad degree distribution, we turn to the microscopic
statistics of link formation. Most real complex networks exhibit distributions
that can be approximated by power-laws. Preferential attachment was introduced
as a possible mechanism to explain the prevalence of this property barabasinew
. Indeed, direct measurements confirmed that preferential attachment governs
the evolution of many real systems, e.g. scientific citation networks newman ;
barabasipa ; perc2 , collaboration networks barabasipa2 , social networks
paonline ; youtube  or language use perc1 . In its original form, preferential
attachment describes the process when the probability of forming a new link is
proportional to the degree of the target node barabasi . In the past decade,
several generalizations and modifications of the original model were proposed,
aiming to reproduce further structural characteristics of real systems
krapivsky ; dorogovstev1 ; dorogovstev2 ; vazquez2 . Here, we investigate the
nonlinear preferential attachment model krapivsky , where the probability that
a new link connects to node $v$ is

$\pi(k_{v})=\frac{k_{v}^{\alpha}}{\sum_{w}k_{w}^{\alpha}},$ (4)

where $k_{v}$ is the indegree of node $v$, and $\alpha>0$. The probability
that the new link connects to _any_ node with degree $k$ is $\Pi(k)\sim
n_{k}(t)\pi(k)$, where $n_{k}(t)$ is the number of nodes with $k$ degree at
the time of the link formation. We cannot test directly our assumption,
because $\Pi(k)$ changes over time. To proceed we transform $\Pi(k)$ to a
uniform distribution by calculating the rank function $R(k,t)$ for each new
link given $\pi(k)$ and $n_{k}(t)$:

$R(k,t)=\frac{\sum_{j=0}^{k}n_{j}(t)j^{\alpha}}{\sum_{j=0}^{k_{\mathrm{max}}}n_{j}(t)j^{\alpha}}=\frac{\sum_{k_{v}<k}k_{v}^{\alpha}}{\sum_{v}k_{v}^{\alpha}}\textrm{.}$
(5)

If Eq. 4 holds, $R(k,t)$ is uniformly distributed in the interval $[0,1]$,
independently of $t$. Therefore, if we plot the cumulative distribution
function, we get a straight line for the correct exponent $\alpha$. To
determine the best exponent, we compare the empirical distribution of the $R$
values to the uniform distribution for different exponents by computing the
Kolmogorov-Smirnoff distance between the two distributions.

Evaluating our method for indegree distribution of the Bitcoin network, we
find good correspondence between the empirical data and the presumed
conditional probability function; the exponent giving the best fit is
$\alpha\approx 1$ (Fig. 7). This shows that the overall growth statistics
agree well with the preferential attachment process. Of course, preferential
attachment itself cannot explain the disassortative degree correlations and
the high clustering observed in the network. We argue that preferential
attachment is a key factor shaping the degree distribution, however, more
detailed investigation of the growth process is necessary to explain the
higher order correlations.

###  II.2 Dynamics of transactions

Figure 8: Distribution of time delay between transactions initiated from a
single Bitcoin address. We observe a power-law distribution close to the
widely observed $P(T)\sim T^{-1}$, the exponential cutoff corresponds to the
finite lifetime of the Bitcoin system.  Figure 9: Evolution of the
distribution of balances of individual Bitcoin addresses. The distributions
are shifted by arbitrary factors along the vertical axis for better visibility
of the separate lines. The black lines are stretched exponential and power-law
fits of the last empirical distribution. The tail can be approximated by a
power-law with exponent $-1.984$, however, the rest of the fit is
unsatisfactory. Therefore, we fit the distribution with a stretched
exponential distribution of form $P(b)\sim b^{-\gamma}e^{-(ab)^{1-\gamma}}$.
We find a better approximation of the whole distributions; the parameters are
$\gamma=0.873$ and $a=8014\,\textrm{BTC}^{-1}$.

In the this section, we analyze the detailed dynamics of money flow on the
transaction network. The increasing availability of digital traces of human
behavior revealed that various human activities, e.g. mobility patterns, phone
calls or email communication, are often characterized by heterogeneity bursts2
; kertesz ; mobility ; mobile . Here we show that the handling of money is not
an exception: we find heterogeneity in both balance distribution and temporal
patterns. We also investigate the microscopic statistics of transactions.

The state of node $v$ at time $t$ is given by the balance of the corresponding
address $b_{v}(t)$, i.e. the number of bitcoins associated to node $v$. The
transactions are directly available, and we can infer the balance of each node
based on the transaction list. Note that the overall quantity of bitcoins
increases over time: Bitcoin rewards users devoting computational power to
sustain the system.

We first investigate the temporal patterns of the system by measuring the
distribution of inactivity times $T$. The inactivity time is defined as the
time elapsed between two consecutive outgoing transactions from a node. We
find a broad distribution that can be approximated by the power-law $P(T)\sim
1/T$ (Fig. 8), in agreement with the behavior widely observed in various
complex systems bursts1 ; bursts2 ; poissonian_expl ; temporal_review .

It is well known that the wealth distribution of society is heterogeneous; the
often cited –and quantitatively not precise– 80-20 rule of Pareto states that
the top 20% of the population controls 80% of the total wealth. In line with
this, we find that the wealth distribution in the Bitcoin system is also
highly heterogeneous. The proper Pareto-like statement for the Bitcoin system
would be that the 6.28% of the addresses posesses the 93.72% of the total
wealth. We measure the distribution of balances at different points of time,
and we find a stable distribution. The tail of wealth distribution is
generally modeled with a power-law boltzman ; wealthchina ; forbes , following
this practice we find a power-law tail $\sim x^{-1.984}$ for balances $\gtrsim
50\textrm{BTC}$ (see Fig. 9). However, visual inspection of the fit is not
convincing: the scaling regime spans only the last few orders of magnitude,
and fails to reproduce the majority of the distribution. Instead we find that
the overall behavior is much better approximated by the stretched exponential
distribution $P(b)\sim b^{-\gamma}e^{-(ab)^{1-\gamma}}$, where $\gamma=0.873$
and $a=8014\,\textrm{BTC}^{-1}$.

To further investigate the evolution of the wealth distribution we measure the
Gini coefficient over time. We find that the distribution is characterized by
high values throughout the whole lifetime of the network, reaching a
stationary value around $G\approx 0.985$ in the trading phase (see Fig. 4).

To understand the origin of this heterogeneity, we turn to the microscopic
statistics of acquiring bitcoins. Similarly to the case of degree
distributions, the observed heterogeneous wealth distributions are often
explained by preferential attachment. Moreover, preferential attachment was
proposed significantly earlier in the context of wealth distributions than
complex networks simon . In economics preferential attachment is traditionally
called the Matthew effect or the “rich get richer phenomenon” matthew . It
states that the growth of the wealth of each individual is proportional to the
wealth of that individual. In line with this principle, several statistical
models were proposed to account for the heterogeneous wealth distribution
boltzman ; wealthpow ; pamoney1 ; pamoney2 .

Figure 10: Change of balances in one month windows. Increase (top) and
decrease (bottom, vertical axis is inverted) of node balances in one month
windows as a function of their balance at the beginning of each month. We show
the raw data (red), the average (green), median (blue) and logarithmic average
(magenta). The later three are calculated for logarithmically sized bins. We
find a clear positive correlation: addresses with high balance typically
increase their wealth more than addresses with low balance. The median and the
logarithmic average values almost coincide, which suggests multiplicative
fluctuations. The median and the logarithmic average increase approximately as
power-laws for several orders of magnitude. The black line is a power-law fit
for the double logarithmic data; the exponent is $0.857$.

To find evidence supporting this hypothesis, we first investigate the change
of balances in fixed time windows. We calculate the difference between the
balance of each address at the end and at the start of each month. We plot the
differences in function of the starting balances (Fig. 10). When the balance
increases, we observe a positive correlation: the average growth increases in
function of the starting balance, and it is approximated by the power-law
$\sim b^{0.857}$. This indicates the “rich get richer” phenomenon is indeed
present in the system. For decreasing balances, we find that a significant
number of addresses lose all their wealth in the time frame of one month. This
phenomenon is specific to Bitcoin: due to the privacy concerns of users, it is
generally considered a good practice to move unspent bitcoins to a new address
when carrying out a transaction newaddress .

To better quantify the preferential attachment, we carry out a similar
analysis to the previous section. However, there is a technical difference: in
the case of the evolution of the transaction network, for each event the
degree of a node increases by exactly one. In the case of the wealth
distribution there is no such constraint. To overcome this difficulty we
consider the increment of a node’s balance by one unit as an event, e.g. if
after a transaction $b_{v}$ increased by $\Delta b_{v}$, we consider it as
$\Delta b_{v}$ separate and simultaneous events. We only consider events when
the balance associated to an address increases, i.e. the address receives a
payment. We now calculate the rank function $R(b,t)$ defined in Eq. 5, and
plot the cumulative distribution function of the $R$ values observed
throughout the whole time evolution of the Bitcoin network (Fig. 11). Visual
inspection shows that no single exponent provides a satisfying result, meaning
that $\pi(b_{v})$ cannot be modeled by a simple power-law relationship like in
Eq. 4. However, we do find that the “average” behavior is best approximated by
exponents around $\alpha\approx 0.8$, suggesting that $\pi(b_{v})$ is a
sublinear function. In the context of network evolution, previous theoretical
work found that sublinear preferential attachment leads to a stationary
stretched exponential distribution krapivsky , in line with our observations.

We have investigated the evolution of both the transaction network and the
wealth distribution separately. However, it is clear that the two processes
are not independent. To study the connection between the two, we measure the
correlation between the indegree and balance associated to the individual
nodes. We plot the average balance of addresses as a function of their degrees
on Fig. 12. For degrees in the range of $1$–$3000$ (over $99.99\%$ of all
nodes with nonzero balance), the average balance is a monotonously increasing
function of the degree, and it is approximated by the power-law $b\sim
k_{\text{in}}^{0.617}$, indicating that the accumulated wealth and the number
of distinct transaction partners an individual has are inherently related.
Similar scaling was reported by Tseng et al., who conducted an online
experiment where volunteers traded on a virtual market pamoney2 .

\begin{overpic}[unit=1mm]{addrbalct2}
\put(46.0,14.0){\includegraphics{addrbalctinset}} \end{overpic} Figure 11:
Rank function for the growth of balances. The cumulative distribution function
of the $R$ values (see Eq. 5) for exponents $0$, $0.2$, $0.4$, $0.5$, $0.6$,
$0.7$, $0.85$ and $1$. The inset shows the maximum Kolmogorov-Smirnoff error
for these exponents. Here, the results are not as obvious as in the case of
link creation (Fig. 7; a simple power-law form like in Eq. 4 is not sufficient
to accurately model the statistics of money flow. On the other hand, the
“average” behavior shows a correlation between the balance and the increase of
the balance: the uncorrelated assumption ($\alpha=0$) clearly gives a much
worse approximate than the exponents that presume preferential attachment
($\alpha>0$). \begin{overpic}[unit=1mm]{degbal4}
\put(9.0,23.0){\includegraphics{degbalinset}} \end{overpic} Figure 12: Average
node balances as a function of the indegrees. We calculate the averages for
logarithmically sized bins. We find strong correlation between the balance and
the indegree of individual nodes. The main plot shows indegree values up to
$k_{\text{in}}\approx 3000$, only 75 nodes ($0.0063\%$) have higher indegree,
the averages calculated for such small sample result in high fluctuations (see
inset). We also measure both the Pearson and Spearman correlation coefficient:
The Pearson correlation coefficient of the full dataset is $0.00185041$, while
the Spearman rank correlation coefficient is $0.275881$. (Note that the
Pearson correlation coefficient measures the linear dependence between two
variables, while the Spearman coefficient evaluates monotonicity). We test the
statistical significance of the correlation by randomizing the dataset 1000
times and calculating the Spearman coefficient for each randomization. We find
that the average Spearman coefficient is $10^{-4}$ with a standard deviation
of $9.5\cdot 10^{-4}$, indicating that the correlation is indeed significant.

##  III Methods

###  III.1 The Bitcoin network

Bitcoin is based on a peer-to-peer network of users connected through the
Internet, where each node stores the list of previous transactions and
validates new transactions based on a proof-of-work system. Users announce new
transactions on this network, these transactions are formed into _blocks_ at
an approximately constant rate of one block per 10 minutes; blocks contain a
varying number of transactions. These blocks form the block-chain, where each
block references the previous block. Changing a previous transaction (e.g.
double spending) would require the recomputation of all blocks since then,
which becomes practically infeasible after a few blocks. To send or receive
bitcoins, each user needs at least one address, which is a pair of private and
public keys. The public key can be used for receiving bitcoins (users can send
money to each other referencing the recipient’s public key), while sending
bitcoins is achieved by signing the transaction with the private key. Each
transaction consists of one or more _inputs_ and _outputs_. In Fig. 13 we show
a schematic view of a typical Bitcoin transaction. Readers interested in the
technical details of the system can consult the original paper by Satoshi
Nakamoto satoshi  or the various resources available on the Internet bcweb ;
clients .

An important aspect of Bitcoin is how new bitcoins are created, and how new
users can acquire bitcoins. New bitcoins are generated when a new block is
formed as a reward to the users participating in block generation. The
generation of a valid new block involves solving a reverse hash problem, whose
difficulty can be set in a wide range. Participating in block generation is
referred to as _mining_ bitcoins. The nodes in the network regulate the block
generation process by adjusting the difficulty to match the processing power
currently available. As interest in the Bitcoin system grew, the effort
required to generate new blocks, and thus receive the newly available
bitcoins, has increased over 10 million fold; most miners today use
specialized hardware, requiring significant investments. Consequently, an
average Bitcoin user typically acquires bitcoins by either buying them at an
exchange site or receiving them as compensation for goods or services.

Due to the nature of the system, the record of all previous transactions since
its beginning are publicly available to anyone participating in the Bitcoin
network. From these records, one can recover the sending and receiving
addresses, the sum involved and the approximate time of the transaction. Such
detailed information is rarely available in financial systems, making the
Bitcoin network a valuable source of empirical data involving monetary
transactions. Of course, there are shortcomings: only the addresses involved
in the transactions are revealed, not the users themselves. While providing
complete anonymity is not among the stated goals of the Bitcoin project
anonimity2 , identifying addresses belonging to the same user can be difficult
anonimity , especially on a large scale. Each user can have an unlimited
number of Bitcoin addresses, which appear as separate nodes in the transaction
records. When constructing the network of users, these addresses would need to
be joined to a single entity.

Another issue arises not only for Bitcoin, but for most online social
datasets: It is hard to determine which observed phenomena are specific to the
system, and which results are general. We do not know to what extent the group
of people using the system can be considered as a representative sample of the
society. In the case of Bitcoin for example, due to the perceived anonymity of
the system, it is widely used for commerce of illegal items and substances
silkroad ; these types of transactions are probably overrepresented among
Bitcoin transactions. Ultimately, the validity of our results will be tested
if data becomes available from other sources, and comparison becomes possible.

###  III.2 Data

We installed the open-source bitcoind client and downloaded the blockchain
from the peer-to-peer network on May 7th, 2013. We modified the client to
extract the list of all transactions in a human-readable format. We downloaded
more precise timestamps of transactions from the blockchain.info website’s
archive. The data and the source code of the modified client program is
available at the project’s website web  or through the Casjobs web database
interface nmvo ; matray .

The data includes 235,000 blocks, which contain a total of 17,354,797
transactions. This dataset includes 13,086,528 addresses (i.e. addresses
appearing in at least one transaction); of these, 1,616,317 addresses were
active in the last month. The Bitcoin network itself does not store balances
associated with addresses, these can be calculated from the sum of received
and sent bitcoins for each address; preventing overspending is done by
requiring that the input of a transaction corresponds to the output of a
previous transaction. Using this method, we found that approximately one
million addresses had nonzero balance at the time of our analysis.

Figure 13: Schematic view of a Bitcoin transaction. Here we have four input
(I1–I4) and three output (O1–O3) addresses. Links in our analysis are created
pointing from each input to each output address.

##  IV Discussion

We have preformed detailed analysis of Bitcoin, a novel digital currency
system. A key difference from traditional currencies handled by banks is the
open nature of the Bitcoin: each transactions is publicly announced, providing
unprecedented opportunity to study monetary transactions of individuals. We
have downloaded and compiled the complete list of transactions, and we have
extracted the time and amount of each payment. We have studied the structure
and evolution of the transaction network, and we have investigated the
dynamics taking place on the network, i.e. the flow of bitcoins.

Measuring basic network characteristics in function of time, we have
identified two distinct phases in the lifetime of the system: (i) When the
system was new, no businesses accepted bitcoins as a form of payment,
therefore Bitcoin was more of an experiment than a real currency. This initial
phase is characterized by large fluctuations in network characteristics,
heterogeneous indegree- and homogeneous outdegree distribution. (ii) Later
Bitcoin received wider public attention, the increasing number of users
attracted services, and the system started to function as a real currency.
This trading phase is characterized by stable network measures, dissasortative
degree correlations and power-law in- and outdegree distributions. We have
measured the microscopic link formation statistics, finding that linear
preferential attachment drives the growth of the network.

To study the accumulation of bitcoins we have measured the wealth distribution
at different points in time. We have found that this distribution is highly
heterogeneous through out the lifetime of the system, and it converges to a
stable stretched exponential distribution in the trading phase. We have found
that sublinear preferential attachment drives the accumulation of wealth.
Investigating the correlation between the wealth distribution and network
topology, we have identified a scaling relation between the degree and wealth
associated to individual nodes, implying that the ability to attract new
connections and to gain wealth is fundamentally related.

We believe that the data presented in this paper has great potential to be
used for evaluating and refining econophysics models, as not only the bulk
properties, but also the microscopic statistics can be readily tested. To this
end, we make all the data used in this paper available online to the
scientific community in easily accessible formats web ; nmvo ; matray .

## Acknowledgments

The authors thank András Bodor and Philipp Hövel for many useful discussions
and suggestions. This work has been supported by the European Union under
grant agreement No. FP7-ICT-255987-FOC-II Project. The authors thank the
partial support of the European Union and the European Social Fund through
project FuturICT.hu (grant no.: TAMOP-4.2.2.C-11/1/KONV-2012-0013), the OTKA
7779 and the NAP 2005/KCKHA005 grants. EITKIC_12-1-2012-0001 project was
partially supported by the Hungarian Government, managed by the National
Development Agency, and financed by the Research and Technology Innovation
Fund and the MAKOG Foundation.

## References

  * (1) Albert, R. and Barabási, A.-L. (2002). Statistical mechanics of complex networks. Rev. Mod. Phys. 74 (1), 47
  * (2) Newman, M. E. J. (2003). The structure and function of complex networks. SIAM review 45 (2), 167–256
  * (3) Barabási A.-L., Oltvai Z. N. (2004). Network biology: understanding the cell’s functional organization. Nat. Rev. Genet. 5 101–113.
  * (4) Pastor-Satorras, R., and Vespignani, A. (2007). Evolution and structure of the Internet: A statistical physics approach. Cambridge University Press
  * (5) Barrat, A., Barthélemy, M., Vespignani, A. (2008). Dynamical Processes on Complex Networks. Cambridge University Press
  * (6) Cohen, R., and Havlin, S. (2010). Complex Networks. Structure, Robustness and Function. Cambridge University Press
  * (7) Dorogovtsev, S. N., Goltsev, A. V. and Mendes, J. F. F. (2008). Critical phenomena in complex networks. Rev. Mod. Phys. 80 (4) 1275–1335
  * (8) Catanzaro, M., and Buchanan, M. (2013). Network opportunity. Nature Physics, 9(3), 121–123.
  * (9) Caldarelli, G., Chessa, A., Pammolli, F., Gabrielli, A., and Puliga, M. (2013). Reconstructing a credit network. Nature Physics, 9(3), 125–126.
  * (10) Bargigli, L., and Gallegati, M. (2013). Finding Communities in Credit Networks. Economics, 7.
  * (11) Caldarelli, G. (2007). Scale-Free Networks. Oxford University Press
  * (12) Schweitzer, F., Fagiolo, G., Sornette, D., Vega-Redondo, F., Vespignani, A., and White, D. R. (2009). Economic networks: The new challenges. science Science, 325 (5939) 422
  * (13) Palla, G., Farkas, I., Derényi, I., Barabási, A.-L., and Vicsek, T. (2004). Reverse engineering of linking preferences from network restructuring. Phys. Rev. E, 70(4), 046115.
  * (14) Nakamoto, S. (2008). Bitcoin: A peer-to-peer electronic cash system. http://bitcoin.org/bitcoin.pdf
  * (15) Ron, D. and Shamir, A. (2012). Quantitative Analysis of the Full Bitcoin Transaction Graph. IACR Cryptology ePrint Archive, 2012:584. Available: http://eprint.iacr.org/2012/584. In: Financial Cryptography and Data Security, Springer, 2013.
  * (16) Reid, F. and Harrigan, M. (2011). An Analysis of Anonymity in the Bitcoin System. arXiv:1107.4524
  * (17) Ober, M., Katzenbeisser, S., and Hamacher, K. (2013). Structure and Anonymity of the Bitcoin Transaction Graph. Future Internet, 5(2), 237–250. doi:10.3390/fi5020237
  * (18) Note that the name Satoshi Nakamoto is widely believed to be a psuedonym. See e.g. `https://en.bitcoin.it/wiki/Satoshi_Nakamoto`
  * (19) Clauset, A., Shalizi, C. R., and Newman, M. E. J. (2009). Power-law distributions in empirical data. SIAM Review 51 (4), 661–703.
  * (20) Dorfman, R. (1979). Formula for the Gini Coefficient The Review of Economics and Statistics, 61 (1), 146–149
  * (21) Menche, J., Valleriani, A., and Lipowsky, R. (2010). Asymptotic properties of degree-correlated scale-free networks. Phys. Rev. E, 81(4), 046103.
  * (22) Barabási, A.-L. (2012). Network science: Luck or reason. Nature 489, 1–2
  * (23) Barabási, A.-L., Jeong, H., Néda, Z., Ravasz, E., Schbert, A., and Vicsek, T. (2002). Evolution of the social network of scientific collaborations. Physica A 311 590–614
  * (24) Newman, M. E. J. (2001). Clustering and preferential attachment in growing networks. Phys. Rev. E 64 025102
  * (25) Perc, M. (2013). Self-organization of progress across the century of physics. Scientific Reports, 3, 1720.
  * (26) Jeong, H., Néda, Z., and Barabási, A.-L. (2003). Measuring preferential attachment in evolving networks. Europhysics Letters, 567.
  * (27) Kunegis, J., Blattner, M., and Moser, C. (2013). Preferential attachment in online networks: Measurement and explanations. arXiv:1303.6271.
  * (28) Alan Mislove. Online Social Networks: Measurement, Analysis, and Applications to Distributed Information Systems. PhD thesis, Rice University, 2009.
  * (29) Perc, M. (2012). Evolution of the most common English words and phrases over the centuries. J. R. Soc. Interface, 9, 3323–3328
  * (30) Barabási, A.-L. and Albert, R. (1999). Emergence of scaling in random networks. Science 286 509–512
  * (31) Krapivsky, P. L., Redner, S., and Leyvraz, F. (2000). Connectivity of growing random networks. Phys. Rev. Lett., 85 (21), 4629.
  * (32) Dorogovtsev, S. N., Mendes, J. F. F., and Samukhin, A. N. (2000). Structure of growing networks with preferential linking. Phys. Rev. Lett., 85 (21), 4633.
  * (33) Dorogovtsev, S. N., and Mendes, J. F. F. (2000). Evolution of networks with aging of sites. _Phys. Rev. E_ , 62 (2), 1842.
  * (34) Vázquez, A. (2003). Growing network with local rules: Preferential attachment, clustering hierarchy, and degree correlations. Phys. Rev. E, 67 (5), 056104.
  * (35) Vázquez, A., Oliveira, J. G., Dezső, Z., Goh, K., Kondor, I., and Barabási, A.-L. (2006). Modeling bursts and heavy tails in human dynamics. Phys. Rev. E, 73, 036127.
  * (36) Jo, H., Karsai, M., Kertész, J., and Kaski, K. (2012). Circadian pattern and burstiness in mobile phone communication. New J. Phys., 14, 013055.
  * (37) Song, C., Qu, Z., Blumm, N., and Barabási, A.-L. (2010). Limits of predictability in human mobility. Science, 327, 1018–21.
  * (38) Lambiotte, R., Blondel, V. D., Kerchove, C., Huens, E., Prieur, C., Smoreda, Z., and Van Dooren, P. (2008). Geographical dispersal of mobile communication networks. Physica A, 387 (21), 5317–5325.
  * (39) Barabási, A.-L. (2005). The origin of bursts and heavy tails in human dynamics. Nature, 435 (7039), 207-211.
  * (40) Malmgren, R. D., Stouffer, D. B., Motter, A. E., and Amaral, L. A. (2008). A Poissonian explanation for heavy tails in e-mail communication. PNAS, 105 (47), 18153-18158.
  * (41) Holme, P., and Saramäki, J. (2012). Temporal networks. Physics Reports, 519 (3), 97-125.
  * (42) Yakovenko, V. M. and Rosser, J. B. (2009). Colloquium: Statistical mechanics of money, wealth, and income. Rev. Mod. Phys. 81 (4) 1703–1725
  * (43) Ning, D., and You-Gui, W. (2007). Power-law tail in the Chinese wealth distribution. Chinese Physics Letters, 24 (8), 2434.
  * (44) Klass, O. S., Biham, O., Levy, M., Malcai, O., and Solomon, S. (2006). The Forbes 400, the Pareto power-law and efficient markets. Eur. Phys. J. B, 55 (2), 143–147.
  * (45) Simon, H. A. (1955). On a class of skew distribution functions. Biometrika, 42 (3/4) 425–440.
  * (46) “For whosoever hath, to him shall be given, and he shall have more abundance: but whosoever hath not, from him shall be taken away even that he hath.” The Bible, Matthew 13:12, King James translation
  * (47) Ispolatov, S. and Krapivsky, P.L. and Redner, S. (1998). Wealth Distributions in Asset Exchange Models. Eur. Phys. J. B, 2 (2), 267–276.
  * (48) Garlaschelli, D., Battiston, S., Castri, M., Servedio, V. D. P., and Caldarelli, G. (2005). The scale-free topology of market investments. Physica A, 350, 491–499.
  * (49) Tseng, J., Li, S., and Wang, S. (2010). Experimental evidence for the interplay between individual wealth and transaction network. Eur. Phys. J. B, 73 69–74.
  * (50) “Most Bitcoin software and websites will help with this by generating a brand new address each time you perform a transaction.” https://en.bitcoin.it/wiki/Address
  * (51) http://bitcoin.org/about.html
  * (52) http://bitcoin.org/en/choose-your-wallet
  * (53) See e.g. https://en.bitcoin.it/wiki/Anonymity.
  * (54) Christin, N. (2012). Traveling the Silk Road: A measurement analysis of a large anonymous online marketplace. arXiv: 1207.7139
  * (55) Bitcoin project website at ELTE. http://www.vo.elte.hu/bitcoin
  * (56) Mátray, P., Csabai, I., Hága, P., Stéger, J., Dobos, L. and Vattay, G. (2007). Building a prototype for Network Measurement Virtual Observatory. Proc. ACM SIGMETRICS 2007 MineNet Workshop. San Diego, CA, USA. DOI: 10.1145/1269880.1269887
  * (57) CasJobs web database interface. http://nm.vo.elte.hu/casjobs


# LKPNR: LLM and KG for Personalized News Recommendation Framework

Chen Hao1, Xie Runfeng1, Cui Xiangyang1, Yan zhou1, Wang xin1, Xuanzhanwei1,
Zhang kai1
Corresponding authorCorresponding author

###### Abstract

Accurately recommending candidate news articles to users is a basic challenge
faced by personalized news recommendation systems. Traditional methods are
usually difficult to grasp the complex semantic information in news texts,
resulting in unsatisfactory recommendation results. Besides, these traditional
methods are more friendly to active users with rich historical behaviors.
However, they can not effectively solve the “long tail problem” of inactive
users. To address these issues, this research presents a novel general
framework that combines Large Language Models (LLM) and Knowledge Graphs (KG)
into semantic representations of traditional methods. In order to improve
semantic understanding in complex news texts, we use LLMs’ powerful text
understanding ability to generate news representations containing rich
semantic information. In addition, our method combines the information about
news entities and mines high-order structural information through multiple
hops in KG, thus alleviating the challenge of long tail distribution.
Experimental results demonstrate that compared with various traditional
models, the framework significantly improves the recommendation effect. The
successful integration of LLM and KG in our framework has established a
feasible path for achieving more accurate personalized recommendations in the
news field. Our code is available at https://github.com/Xuan-ZW/LKPNR.

##  1 Introduction

With the exponential growth of the Internet, an increasing number of
individuals are opting to access the most current global news via online
platforms, including MSN News. However, users often find themselves
overwhelmed by the sheer volume of available news content. Consequently, the
imperative for an effective news recommendation system becomes evident, as it
serves as a crucial tool in assisting users to navigate through this vast
information landscape. Such a system not only aids users in filtering copious
amounts of news but also employs personalized algorithms to proactively
present news items aligning with users’ genuine interests, thereby
significantly enhancing the fulfillment of their informational requirements(Wu
et al. 2021).

Figure 1: Differences between traditional news recommendation model (a) and
LKPNR framework (b)

The current research in this field primarily emphasizes the perspective of
representation learning(Bengio, Courville, and Vincent 2012), aiming to
enhance the learning of user and news representations separately. To
illustrate this, we take the MIND dataset(Wu et al. 2020) as a case study. The
user’s behavioral data within this dataset comprises a sequence of historical
clicks on news items, each of which is composed of title, abstract, category
and other related information. Earlier researches, exemplified by NAML(Wu et
al. 2019a) and NRMS(Wu et al. 2019c), have been employed to achieve improved
news representations. The methods often leverage CNN and LSTM for feature
extraction, coupled with feature fusion using attention mechanisms. However,
these existing studies exhibit certain limitations that deserve attention. One
such problem is insufficient news text feature extraction, this deficiency
hampers the effectiveness of the news representation process. Neglect of news
popularity and interconnections is another challenge, this neglect can
potentially lead the news recommender system into the challenging scenario of
the long-tail problem.

With the advent of ChatGPT, the realm of natural language processing has
seamlessly ushered in the era of Large Language models (LLMs). Notably,
significant models like ChatGLM2(Zeng et al. 2022), LLAMA2(Touvron et al.
2023), and RWKV(Peng et al. 2023) have surfaced within the open-source
community. LLMs, having undergone pre-training on vast corpora of textual
data, exhibit the ability to swiftly acclimate to the data distribution
pertaining to downstream tasks. Leveraging the exceptional language modeling
proficiency of LLMs, they adeptly uncover intricate linguistic relationships
and semantic nuances inherent within the text. This capacity allows for a more
robust contextual integration, thereby augmenting text comprehension and
facilitating the extraction of information-rich semantic features.

The long-tail problem(Yin et al. 2012) in recommendation systems is that a
significant majority, approximately 80%, of user clicks are concentrated on a
mere 20% of popular items. Consequently, this tendency results in
recommendation systems favoring these popular items, often overlooking less
popular ones, which, over time, detrimentally impacts the overall
effectiveness of recommendations. To address this long-tail challenge, recent
research(Guo et al. 2020; Wang et al. 2019) has explored the incorporation of
knowledge graphs (KGs) as supplementary information for recommender systems.
This innovative approach leverages graph-based learning(Ouyang et al. 2021) to
establish meaningful relationships among diverse items within the system.
Subsequently, this method harnesses additional item-specific information to
enhance the representation of long-tailed items, effectively mitigating the
issue of inadequate representation learning for such items.

To address the aforementioned challenges, we introduce LKPNR (LLM and KG for
Personalized News Recommendation framework), a personalized news
recommendation framework that links general news recommendation models with
the integration of KGs and LLMs. Capitalizing on the robust text comprehension
abilities of the LLM, we generate news representations imbued with
comprehensive semantic information for each news item, thereby mitigating the
shortcomings associated with the limited feature extraction capabilities
inherent in general news recommendation models. Concurrently, the
incorporation of subgraph structural representations, mined through multi-hop
inference within KG, serves to alleviate the issue of long-tail distribution
prevalent in news recommendation. By harnessing the strengths of both LLM and
KG, we observe a substantial enhancement in the model’s performance. The
differences between our proposed framework and the general news recommendation
model are shown in Figure 1. To summarize, our contributions are listed as
follows:

  * •

We propose LKPNR, a personalized news recommendation framework that fuses
general news recommendation models with KG and LLM. To the best of our
knowledge, this is the first work that combines both KG and LLM in the news
recommendation domain.

  * •

LKPNR can be flexibly combined with various general news recommendation
models. Leveraging LLM′s powerful text understanding and the news graph
structural relationships contained in the KG to inject additional information
into general news recommendation models.

  * •

Experiments on the MIND dataset show that LKPNR can significantly improves the
performance of general news recommendation models.

##  2 Related Work

### General News Recommendation Model

General news recommendation models typically involve encoding various aspects
of news, such as title and abstract independently. These encoded
representations are then interacted with separately to create a comprehensive
news representation. In a similar manner, historical news browsing sequences
are encoded and integrated to form a user representation. Subsequently, the
similarity between this user representation and the representations of
candidate news items is computed, enabling the prediction of whether the user
would find these candidates interested or not. Okura et al.(Okura et al.
2017)extract news features through a denoising self-encoder, while user
sequences are derived using RNN to obtain user features. Lian et al. proposed
DFM(Lian et al. 2018)using multi-channel inception blocks and attention
mechanism to tackle the issue of data diversity. An et al.(An et al. 2019)
employed GRU to model both long-term and short-term interests, resulting in
improved user representations. Wu et al. extended the attention
mechanism(Vaswani et al. 2017b) paradigm in several ways(Wu et al. 2019a, b).
Wang et al.(Wang et al. 2020) employ fine-grained interest matching using
dilated convolution and 3D convolution techniques.

### LLM-Powered News Recommendation Model

With the remarkable performance exhibited by LLM across diverse domains,
researchers have embarked on an exploration of its potential within the
recommendation domain. Kang et al.(Kang et al. 2023) conducted a comparative
study encompassing traditional collaborative filtering methods and LLM for
user rating prediction, examining zero-shot, few-shot, and fine-tuned
scenarios. Their research revealed intriguing insights. Likewise, Hou et
al.(Hou et al. 2023) devise various prompts to address the ranking predicament
in recommender systems. Gao et al.(Gao et al. 2023) transform user profiles
and historical interactions into prompts, leveraging ChatGPT′s in-context
learning capabilities for recommendation.

The above methods utilize LLM’s in-context learning capabilities by
constructing prompts to cope with downstream tasks in the general
recommendation domain, but the performance is far inferior to that of
traditional ID embedding-based fine-tuning methods. In the news recommendation
domain, LLM is also beginning to combine with traditional models. Liu et
al.(Liu et al. 2023) use ChatGPT to generate user profiles and augment the
news text, combining with a traditional recommendation model to achieve better
results. Li et al.(Li et al. 2023) based on the traditional recommendation
model, generated news representations by using LLM as a news encoder directly
to complete the news recommendation.

Figure 2: The framework of LKPNR. The lower left corner of figure is the input
news data into LKPNR, where the words marked in yellow indicates the entities.
Candidate news is encoded by LK-Aug news encoder that combines LLM and KG with
traditional general news encoder to obtain news representation. To obtain user
representation, LK-Aug User Encoder contains several LK-Aug News Encoders,
which encodes user’s historical click behaviors.

### KG-powered News Recommendation Model

Recommendation systems predominantly operate within a graph structure due to
the information-rich nature of the data, which leads to the widespread
adoption of GNN within the realm of news recommendation. Wang et al.(Wang et
al. 2018) proposed DKN, which constructs a graph of entities within the input
news and was utilized to seek neighboring nodes for expanding news
information. Another significant contribution by Mao et al(Mao, Zeng, and Wong
2021). involves the integration of a GCN with the user’s browsing history and
leverages both LSTM and attention mechanisms to extract textual information.
Yang et al.(Yang et al. 2023) employ a gated GNN, combining global
representations from other users with local representations to enhance the
personalized recommender system. Furthermore, several research endeavors aim
to amplify representation and inference capabilities by linking LLM and KG.
Sun et al.(Sun et al. 2021) utilized attention mechanism to facilitate
interaction between all tokens of the input text and KG entities, thus
augmenting the output of the language model with domain-specific knowledge
from the KG.

It is evident that the KG within the news domain holds substantial value in
terms of structured information and domain-specific knowledge. Its inherent
incompleteness and limited language understanding are effectively supplemented
by the extensive general knowledge encapsulated within LLM. Notably, there is
currently a dearth of work that combines LLM and KG within the news
recommendation field. Therefore, we propose LKPNR as a solution to bridge this
gap.

##  3 Problem Formulation

The personalized news recommendation system proposed takes the user’s news
clicks and the KG of the entities within the news as input. The user’s news
clicks refer to a number of pairs of user and news, denoted as
$\mathcal{C}=(u,n)\subseteq\mathcal{U}\times\mathcal{N}$, where $\mathcal{U}$
is the user set and $\mathcal{N}$ is the news set. The KG of the entities
contained in the news set $\mathcal{N}$ is a triple
$\mathcal{G}=(h,r,t)\subseteq\mathcal{E}\times\mathcal{R}\times\mathcal{E}$,
where $h,t\in\mathcal{E}$ represent the head and tail entities, and
$r\in\mathcal{R}$ represents the relation between them. Given the training
click set $\mathcal{H}$ and test click set $\mathcal{T}$, where
$\mathcal{H}\cup\mathcal{T}=\mathcal{C}$ and
$\mathcal{H}\cap\mathcal{T}=\phi$. Then the recommendation task can be
formulated as learning a matching function $\mathcal{F}$ from the training set
$\mathcal{H}$ and predicting the degree of user-news matching in the test set
$\mathcal{T}$ to obtain the score $\mathcal{M}$.

##  4 Framework

The overall recommendation system framework consists of three components: the
LLM and KG Augmented News Encoder (LK-Aug News Encoder), the LLM and KG
Augmented User Encoder (LK-Aug User Encoder), and the Click Predictor. The
overall recommendation framework is illustrated in Figure 2

### LK-Aug News Encoder

The LK-Aug News Encoder is composed of three sub-modules: General News
Encoder, LLM-Augmented Encoder and KG-Augmented Encoder.

#### General News Encoder.

The general news encoder is designed to learn a semantic representation of
news at the word level, which is achieved through a structure that typically
consists of two layers: the word embedding layer and the word fusion layer.
The word embedding layer utilizes an embedding matrix $E\in R^{(|v|\times d)}$
to convert each word $w$ occurring in the news title and abstract into an
embedding vector $e$. Here, $v$ denotes the number of words and $d$ denotes
the dimension of the word embedding. The fusion layer is a well-designed
component in the baseline experiments that interacts with the embedding
vectors of individual words via operations such as linear mapping, weighted
summing, and vector concatenation. These operations fuse the vectors to
produce a generic representation vector $r_{GNE}$ for the given news.

#### LLM-Aug Encoder.

We leverage powerful contextual understanding capability of LLM and rich open-
world knowledge to build news representations with comprehensive semantic
information for the input news to solve the problem of limited text feature
extraction capability in general news recommendation. The input news text is
created by concatenating news titles, news abstract, category and sub-category
through [SEP], as shown the Figure 2.

$S_{HS}=LLM(t)$ (1) $S_{HS\\_p}=mean\\_pooling(S_{HS})$ (2)

where $LLM(\cdot)$ is the LLM decoder which returns the hidden states of the
last four decoder layers, denoted $S_{HS}$, and $S_{HS_{p}}$ denotes the
output after taking mean pooling strategy on the sequence length dimension of
the each layer. After that $S_{HS_{p}}$ will be weighted and summed to get
$S_{W}$

$S_{W}=\sum\limits_{i=1}^{4}(a_{i}S_{HS\\_p}^{i})$ (3)

where $a_{i}$ denotes the learnable weights, $S_{HS\\_p}^{i}$ denotes the
hidden states of the $i-th$ layer. Finally the weighted hidden state $S_{W}$
will be projected to the text representation space by nonlinear mapping.

$r_{LLM}=\sigma(f_{l}(S_{W}))$ (4)

where $\sigma$ denotes the activation function, $f_{l}$ denotes the linear
transformation, and $r_{LLM}$ is the enhanced news representation.

#### KG-Aug Encoder

We feed the enhanced news representation $r_{LLM}$ into the nonlinear layer,
which is used to transform $r_{LLM}$ from the textual representation space to
the entity representation space.

$q=\sigma(f_{s}(r_{LLM}))$ (5)

where $f_{s}$ is a linear transformation. The query $q$ will have extensive
interactions with the entities in the KG.

Generally, the title and abstract of a piece of news will contain several
source entities. Considering the 1,2,…,n hop adjacent entities of these source
entities, we can extract a subgraph $g^{n}=(V,R)$ from the external KG, where
$V$ is the set of entities in the subgraph, $R$ is the set of edges connecting
entities. Taking the $k-th$ hop adjacent entity set as an example,
$V^{k}=\\{v_{i}^{k}\\}_{i=0}^{|M|}$ represents $M$ entities with $k$ hops from
the source entity. Then we get the embedding vector corresponding to each
entity through the wiki KG, denoted as $X^{k}=\\{x_{i}^{k}\\}_{i=0}^{|M|}$.
Given a query $q$ and a neighboring entity set $X^{k}$ with hop count $k$, the
KG-Augmented Encoder interacts the query with each vector $x_{i}^{k}$ in
$X^{k}$ to generate a news attention score for the entity, denoted as
$\alpha_{i}^{k}$.

$\alpha_{i}^{k}=Softmax(W^{T}_{k}[q;x_{i}^{k};q\circ x_{i}^{k}])$ (6)

where $W_{\alpha k}^{T}\in R^{3l\times 1}$, is a learnable parameter matrix,
$\circ$ is the element multiplication, and $[;]$ denotes the vector
connection.

The weighted representation $\hat{x}^{k}$ of a $k$-hop entity can be computed
as:

$\hat{x}^{k}=\sum_{i=1}^{M}{\alpha_{i}^{k}x_{i}^{k}}$ (7)

After that, we concatenate the weighted representation vectors of each hop and
project the concatenated vectors to the news representation space, denoted as
the KG representation $r_{KG}$

$r_{KG}=Q^{T}[\hat{x}^{1};...;\hat{x}^{n}]$ (8)

where $Q^{T}\in R^{nl\times o}$, $nl$ denotes the dimension after the
representation vector connection of each hop and $o$ represents the dimension
of the projected KG representation.

The final news representation vector $r_{n}$ is obtained by connecting three
news representations above.

$r_{n}=[r_{GNE};r_{KG};r_{LLM}]$ (9)

### LK-Aug User Encoder

The LK-Aug User Encoder learns representations of users based on their click
history on news. This module includes a News Embedding Layer (NEL) and a
Representation Fusion Layer (RFL). The NEL obtains the representation of the
news browsed by the user through the LK-Aug News Encoder, denoted as
$[r_{n}^{1},r_{n}^{2},…,r_{n}^{z}]$, where $z$ represents the length of the
historical browsing news sequence. The RFL transforms news representation
sequences into user representations $r_{u}$ through a series of fusion
methods, such as concatenation, mapping, attention, etc.

$[r_{n}^{1},r_{n}^{2},...,r_{n}^{k}]=NEL({h_{1},h_{2},...,h_{k}})$ (10)

$r_{u}=RFL([r^{1},r^{2},...,r^{k}])$ (11)

### Click Predictor and Model Training

Given candidate news and user representations $r_{n}$ and $r_{u}$, this module
is used to get the matching score between user u and the candidate news n. We
compute the dot-product$\hat{y}_{n,u}$ of $r_{n}$ and $r_{u}$ as the
unnormalized matching scores of users and news.

$\hat{y}_{n,u}=\left\langle r_{n},r_{u}\right\rangle$ (12)

Following previous general work, we use a negative sampling strategy for model
training. For the $i-th$ news exposure, we compute its unnormalized matching
score as $\hat{y}_{i}^{+}$. In addition, randomly select $K$ pieces of news as
the news that the user does not click, and its unnormalized matching score is
$[\hat{y}_{i,1}^{-},\hat{y}_{i,2}^{-},...,\hat{y}_{i,K}^{-},]$. We employ
softmax function to calculate the normalized matching score.

$p_{i}=\frac{exp(\hat{y}_{i}^{+})}{exp(\hat{y}_{i}^{+})+\sum_{j=1}^{K}{exp(\hat{y}_{i,j}^{-})}}$
(13)

In this way, the click prediction problem can be formulated as a K+1
classification problem. For the training set $\mathbcal{H}$, the loss function
$\mathbcal{L}$ of model training is the negative log likelihood of all
positive samples, which can be described as follows:

$\mathbcal{L}=-\sum_{i\in\mathbcal{H}}log(p_{i})$ (14)

Metric | NRMS | NAML
---|---|---
Orig | LKPNR | Orig | LKPNR
LKPNR-NRMS | w/o KG | w/o LLM | LKPNR-NAML | w/o KG | w/o LLM
AUC | 0.6802 | 0.7049 | 0.6997 | 0.6845 | 0.6842 | 0.7023 | 0.6995 | 0.6841
MRR | 0.3316 | 0.3492 | 0.3441 | 0.3308 | 0.3257 | 0.3423 | 0.3411 | 0.3319
nDCG@5 | 0.3661 | 0.3886 | 0.3846 | 0.3680 | 0.3621 | 0.3816 | 0.3810 | 0.3699
nDCG@10 | 0.4306 | 0.4514 | 0.4453 | 0.4310 | 0.4257 | 0.4440 | 0.4426 | 0.4325
Table 1: Performance of comparison results (Orig. denotes general news
recommendation, LKPNR-NRMS/NAML denotes NRMS/NAML with LKPNR, w/o KG denotes
remove the KG-Augmented Encoder, w/o LLM denotes remove the LLM-Augmented
Encoder)

##  5 Experiments

### Dataset

We conduct experiments on the MIND dataset, which is a dataset constructed
based on user click logs on the MSN online news site. We use the same
processing method as Mao et al. We randomly sample 200K user click logs from
the training and validation sets of the MIND dataset. Given the absence of
labels for the test set, we partition the original validation set into two
distinct segments: the experimental validation set and the experimental test
set. The specific information of the constructed sampled dataset is shown in
Table 2.

# users | 20000 | #user in train set | 189580
---|---|---|---
# news | 78602 | # news in train set | 75963
# training logs | 595186 | # training samples | 905297
Table 2: Statistics of MIND-200K dataset

### Implementation Details and Metrics

We select NAML and NRMS as our baseline models, setting the maximum sequence
length for user browsing history to 50 and adopting a negative sampling rate
of $k=4$. We maintain the parameter configurations of the General News Encoder
identical to the baseline. In the LLM-augmented encoder, the hidden states of
the LLM are projected to 500 dimensions. For the KG-augmented encoder, the
dimensions of entity embeddings are set to 100. Furthermore, we limit the
maximum number of neighboring nodes to 20 per source node, and the traversal
depth is constrained to a maximum of 2 hops. Throughout the training process,
learning rate employs 1e-4, batch size is set to 64, and early-stop strategy
is implemented. All experiments are conducted on the NVIDIA TESLA V100.

To evaluate the model’s performance, we use four widely recognized evaluation
metrics, specifically, the Area Under the ROC Curve (AUC), Mean Reciprocal
Rank (MRR), and Normalized Discounted Cumulative Gain (nDCG@5 and nDCG@10).

### Performance Comparison

We perform LKPNR on two baseline experiments. To further validate the efficacy
of our framework design, we conduct ablation experiments, the experimental
results are summarized in Table 1.

The experimental results show that the two baseline models have significant
improvement in all evaluation metrics (+2.47%/1.81% AUC, +1.76%/1.66% MRR,
+2.25%/1.95% nDCG@5, and +2.08%/1.83% nDCG@10 compared to NRMS/NAML
performance) with the augmentation of our framework. This performance
improvement derives from the fact that the news encoder of the baseline gains
better performance through the enhanced semantic information of the LLM and
the collaborative information of the KG. As depicted in Table 1, discernible
decrements in performance are across the spectrum of ablation variants
compared to our complete model, which demonstrates the efficacy of the
different components of our model. The removal of the LLM exhibits a
substantial impact on the overall performance of the model, demonstrating the
effectiveness of the augmenting semantic information for news representation.

### Performance of Different LLM

The characteristics of LLM can vary due to inconsistencies in the proportion
of data categories within the training data and model structural design.
Consequently, these differences lead to varying capabilities among LLMs,
reflected in their open-world knowledge, performance on diverse tasks and so
on, which leads to their distinct understanding of text. For instance,
ChatGLM2(Zeng et al. 2022) is trained on the same amount of Chinese and
English corpus, and can handle Chinese and English tasks with high quality.
LLAMA2(Touvron et al. 2023), trained on an extensive, high-quality English
dataset, demonstrates adeptness in handling various English tasks. RWKV(Peng
et al. 2023) exhibits a quicker reasoning speed and lower computational
complexity. In order to explore the impact of various LLMs on news
recommendation, we employ three cutting-edge models: ChatGLM2, LLAMA2, and
RWKV to generate enhanced news representations on the basis of the baseline
model NRMS. Table 3 shows the performance comparison of the enhanced news
representation of different LLMs in the recommendation task.

The outcomes of our experiments utilizing these three diverse LLMs are
detailed in the table below, and the results indicate that ChatGLM2 provides
the most effective enhancement for news recommendation when compared to both
LLAMA2 and RWKV. The reason for this phenomenon is that the training data of
ChatGLM2 may contain a certain proportion of English news.

Metrics | LKPNR
---|---
ChatGLM2-6B | LLAMA2-13B | RWKV-7B
AUC | 0.7049 | 0.6845 | 0.6771
MRR | 0.3492 | 0.3307 | 0.3300
nDCG@5 | 0.3886 | 0.3657 | 0.3631
nDCG@10 | 0.4514 | 0.4307 | 0.4218
Table 3: the performance comparison of the enhanced news representation of
different LLMs

### Effectiveness of KG Entity’s Query

We employ comparative experiments to ascertain optimal strategies for the
retrieval and integration of adjacent entities in a more efficient manner. The
incorporation of neighboring entity vectors in the news coding process can be
perceived as a mechanism that augments the collaborative information. By
fusing vectors of neighboring entities, the KG-Augmented Encoder is able to
gather information from multiple related entities and synergistically
integrate them into a single encoded representation. The crux of enhancing
news representation through the use of KG lies in the efficient extraction of
information from all neighboring entities. In KG-Augmented Encoder, the query
is obtained by projecting the textual representation of news into the entity
representation space. We have implemented an experiments to compare the
retrieval performance of the query converted by general news encoder and query
converted by LLM. In addition, direct weighted summation of all entities at
each hop may lead to relatively large information loss. We use multi-head
query to consider different representation spaces, i.e., the multiple weighted
summation of all entities with different weights can capture collaborative
information from different perspectives of the news and entities, and thus
improves the representational capability of KG-Augmented Encoder. The detailed
experimental results are shown in Figure 3.

Figure 3: Comparison of different queries under different num
heads(General_query denotes General News Encoder mapping query, LLM_query
denotes LLM mapping query)

The experimental results show that the retrieval performance of LLM mapping
query outperforms the General News Encoder mapping query. Compared with the
news representation produced by the General News Encoder, which is limited to
the specifics of the news text, the news representation of the LLMs contains
some open world knowledge about the news, and is thus able to understand the
information of neighboring entities more effectively. In addition, when
num_head=3, the LLM demonstrates its highest proficiency in mapping queries to
extract information. It suggests that expanding the representation space of
the query vector enhances its capability to gather entity information.

##  6 Case Study

Figure 4: Historical click sequence and candidate news for the sample user
Figure 5: The visualization of the attention weights to adjacent nodes

In this section, we demonstrate the characteristics of LLM and KG Augmented
News Encoder using visualization. Figure 4 shows a user’s click order of
historical news and current news candidates, as well as the detailed
categories, subcategories, titles and abstracts of these news. Before the
integration of KG, the user’s expression was transformed from the text and
category characteristics of these historical news clicks. This candidate news
has little correlation with any historical news clicked by users, and the
matching degree between the candidate news and the user vector is also very
low. After KG integration, give full consideration to adjacent nodes. Figure 4
contains the source and neighboring entities of the candidate news, and the
entities in the historical click news are highlighted in yellow. Figure 5
shows the attention of a query to neighboring nodes, and in Figure 4, the five
nodes with the highest attention are highlighted in blue. Although this
candidate news has a relatively low correlation with the user’s historical
click news, it has a large number of adjacent nodes with a high correlation
with the historical click news. The historical news sequence contains a number
of U.S. locations and celebrities, and the candidate news also includes many
of these entities in its adjacent nodes. This shows that the candidate news
has a deep potential connection with the historical click news, and the
matching degree has been greatly improved after considering the neighbor
nodes.

LLM and KG Augmented News Encoder considers potential connections between
candidate news and user’s history of clicking on news, which makes user-
candidate news matching beyond the understanding of news text.

##  7 Conclusion

In this work, we propose an innovative personalized news recommendation
framework LKPNR, which integrates a Large Language Model (LLM) and a Knowledge
Graph (KG). While combining the General News Encoder, the robust contextual
comprehension capability of the LLM allows us to derive news representations
imbued with semantic information. Simultaneously, we harness the news
relationship graph structure inherent in the KG to extract supplementary
collaborative news information, enhancing the efficacy of the news
recommendation system and alleviating the long tail problem to a certain
extent. The experimental results demonstrate the outstanding performance of
our proposed framework, leading to significant enhancements over the
traditional baseline.

## References

  * An et al. (2019) An, M.; Wu, F.; Wu, C.; Zhang, K.; Liu, Z.; and Xie, X. 2019.  Neural News Recommendation with Long- and Short-term User Representations.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Bengio, Courville, and Vincent (2012) Bengio, Y.; Courville, A. C.; and Vincent, P. 2012.  Representation Learning: A Review and New Perspectives.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 35: 1798–1828.
  * Clancey (1979) Clancey, W. J. 1979.  _Transfer of Rule-Based Expertise through a Tutorial Dialogue_.  Ph.D. diss., Dept. of Computer Science, Stanford Univ., Stanford, Calif.
  * Clancey (1983) Clancey, W. J. 1983.  Communication, Simulation, and Intelligent Agents: Implications of Personal Intelligent Machines for Medical Education.  In _Proceedings of the Eighth International Joint Conference on Artificial Intelligence (IJCAI-83)_ , 556–560. Menlo Park, Calif: IJCAI Organization.
  * Clancey (1984) Clancey, W. J. 1984.  Classification Problem Solving.  In _Proceedings of the Fourth National Conference on Artificial Intelligence_ , 45–54. Menlo Park, Calif.: AAAI Press.
  * Clancey (2021) Clancey, W. J. 2021.  The Engineering of Qualitative Models.  Forthcoming.
  * Engelmore and Morgan (1986) Engelmore, R.; and Morgan, A., eds. 1986.  _Blackboard Systems_.  Reading, Mass.: Addison-Wesley.
  * Gao et al. (2023) Gao, Y.; Sheng, T.; Xiang, Y.; Xiong, Y.; Wang, H.; and Zhang, J. 2023.  Chat-REC: Towards Interactive and Explainable LLMs-Augmented Recommender System.  _ArXiv_ , abs/2303.14524.
  * Guo et al. (2020) Guo, Q.; Zhuang, F.; Qin, C.; Zhu, H.; Xie, X.; Xiong, H.; and He, Q. 2020.  A Survey on Knowledge Graph-Based Recommender Systems.  _IEEE Transactions on Knowledge and Data Engineering_ , 34: 3549–3568.
  * Hasling, Clancey, and Rennels (1984) Hasling, D. W.; Clancey, W. J.; and Rennels, G. 1984.  Strategic explanations for a diagnostic consultation system.  _International Journal of Man-Machine Studies_ , 20(1): 3–19.
  * Hasling et al. (1983) Hasling, D. W.; Clancey, W. J.; Rennels, G. R.; and Test, T. 1983.  Strategic Explanations in Consultation—Duplicate.  _The International Journal of Man-Machine Studies_ , 20(1): 3–19.
  * Hou et al. (2023) Hou, Y.; Zhang, J.; Lin, Z.; Lu, H.; Xie, R.; McAuley, J.; and Zhao, W. X. 2023\.  Large Language Models are Zero-Shot Rankers for Recommender Systems.  _ArXiv_ , abs/2305.08845.
  * Kang et al. (2023) Kang, W.-C.; Ni, J.; Mehta, N.; Sathiamoorthy, M.; Hong, L.; Chi, E. H.; and Cheng, D. Z. 2023.  Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Prediction.  _ArXiv_ , abs/2305.06474.
  * Li et al. (2023) Li, R.; Deng, W.; Cheng, Y.-J.; Yuan, Z.; Zhang, J.; and Yuan, F. 2023.  Exploring the Upper Limits of Text-Based Collaborative Filtering Using Large Language Models: Discoveries and Insights.  _ArXiv_ , abs/2305.11700.
  * Lian et al. (2018) Lian, J.; Zhang, F.; Xie, X.; and Sun, G. 2018.  Towards Better Representation Learning for Personalized News Recommendation: a Multi-Channel Deep Fusion Approach.  In _International Joint Conference on Artificial Intelligence_.
  * Liu et al. (2023) Liu, Q.; Chen, N.; Sakai, T.; and Wu, X.-M. 2023.  A First Look at LLM-Powered Generative News Recommendation.  _ArXiv_ , abs/2305.06566.
  * Mao, Zeng, and Wong (2021) Mao, Z.; Zeng, X.; and Wong, K.-F. 2021.  Neural News Recommendation with Collaborative News Encoding and Structural User Encoding.  In _Conference on Empirical Methods in Natural Language Processing_.
  * NASA (2015) NASA. 2015.  Pluto: The ’Other’ Red Planet.  https://www.nasa.gov/nh/pluto-the-other-red-planet.  Accessed: 2018-12-06.
  * Okura et al. (2017) Okura, S.; Tagami, Y.; Ono, S.; and Tajima, A. 2017.  Embedding-based News Recommendation for Millions of Users.  _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_.
  * Ouyang et al. (2021) Ouyang, W. R.; Zhang, X.; Ren, S.; Li, L.; Zhang, K.; Luo, J.; Liu, Z.; and Du, Y. 2021.  Learning Graph Meta Embeddings for Cold-Start Ads in Click-Through Rate Prediction.  _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_.
  * Peng et al. (2023) Peng, B.; Alcaide, E.; Anthony, Q. G.; Albalak, A.; Arcadinho, S.; Cao, H.; Cheng, X.; Chung, M.; Grella, M.; Kranthikiran, G.; He, X.; Hou, H.; Kazienko, P.; Kocoń, J.; Kong, J.; Koptyra, B.; Lau, H.; Mantri, K. S. I.; Mom, F.; Saito, A.; Tang, X.; Wang, B.; Wind, J. S.; Wozniak, S.; Zhang, R.; Zhang, Z.; Zhao, Q.; Zhou, P.; Zhu, J.; and Zhu, R. 2023.  RWKV: Reinventing RNNs for the Transformer Era.  _ArXiv_ , abs/2305.13048.
  * Rice (1986) Rice, J. 1986.  Poligon: A System for Parallel Problem Solving.  Technical Report KSL-86-19, Dept. of Computer Science, Stanford Univ.
  * Robinson (1980a) Robinson, A. L. 1980a.  New Ways to Make Microcircuits Smaller.  _Science_ , 208(4447): 1019–1022.
  * Robinson (1980b) Robinson, A. L. 1980b.  New Ways to Make Microcircuits Smaller—Duplicate Entry.  _Science_ , 208: 1019–1026.
  * Sun et al. (2021) Sun, Y.; Shi, Q.; Qi, L.; and Zhang, Y. 2021.  JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering.  In _North American Chapter of the Association for Computational Linguistics_.
  * Touvron et al. (2023) Touvron, H.; Lavril, T.; Izacard, G.; Martinet, X.; Lachaux, M.-A.; Lacroix, T.; Rozière, B.; Goyal, N.; Hambro, E.; Azhar, F.; Rodriguez, A.; Joulin, A.; Grave, E.; and Lample, G. 2023.  LLaMA: Open and Efficient Foundation Language Models.  _ArXiv_ , abs/2302.13971.
  * Vaswani et al. (2017a) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017a.  Attention Is All You Need.  arXiv:1706.03762.
  * Vaswani et al. (2017b) Vaswani, A.; Shazeer, N. M.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, L.; and Polosukhin, I. 2017b.  Attention is All you Need.  In _NIPS_.
  * Wang et al. (2020) Wang, H.; Wu, F.; Liu, Z.; and Xie, X. 2020.  Fine-grained Interest Matching for Neural News Recommendation.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Wang et al. (2018) Wang, H.; Zhang, F.; Xie, X.; and Guo, M. 2018.  DKN: Deep Knowledge-Aware Network for News Recommendation.  _Proceedings of the 2018 World Wide Web Conference_.
  * Wang et al. (2019) Wang, H.; Zhao, M.; Xie, X.; Li, W.; and Guo, M. 2019.  Knowledge Graph Convolutional Networks for Recommender Systems.  _The World Wide Web Conference_.
  * Wu et al. (2019a) Wu, C.; Wu, F.; An, M.; Huang, J.; Huang, Y.; and Xie, X. 2019a.  Neural News Recommendation with Attentive Multi-View Learning.  In _International Joint Conference on Artificial Intelligence_.
  * Wu et al. (2019b) Wu, C.; Wu, F.; An, M.; Huang, J.; Huang, Y.; and Xie, X. 2019b.  NPA: Neural News Recommendation with Personalized Attention.  _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_.
  * Wu et al. (2019c) Wu, C.; Wu, F.; Ge, S.; Qi, T.; Huang, Y.; and Xie, X. 2019c.  Neural News Recommendation with Multi-Head Self-Attention.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Wu et al. (2021) Wu, C.; Wu, F.; Huang, Y.; and Xie, X. 2021.  Personalized News Recommendation: Methods and Challenges.  _ACM Transactions on Information Systems_ , 41: 1 – 50.
  * Wu et al. (2020) Wu, F.; Qiao, Y.; Chen, J.-H.; Wu, C.; Qi, T.; Lian, J.; Liu, D.; Xie, X.; Gao, J.; Wu, W.; and Zhou, M. 2020.  MIND: A Large-scale Dataset for News Recommendation.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Yang et al. (2023) Yang, B.; Liu, D.; Suzumura, T.; Dong, R.; and Li, I. 2023.  Going Beyond Local: Global Graph-Enhanced Personalized News Recommendations.  _ArXiv_ , abs/2307.06576.
  * Yin et al. (2012) Yin, H.; Cui, B.; Li, J.; Yao, J.; and Chen, C. 2012.  Challenging the Long Tail Recommendation.  _ArXiv_ , abs/1205.6700.
  * Zeng et al. (2022) Zeng, A.; Liu, X.; Du, Z.; Wang, Z.; Lai, H.; Ding, M.; Yang, Z.; Xu, Y.; Zheng, W.; Xia, X.; Tam, W. L.; Ma, Z.; Xue, Y.; Zhai, J.; Chen, W.; Zhang, P.; Dong, Y.; and Tang, J. 2022.  GLM-130B: An Open Bilingual Pre-trained Model.  _ArXiv_ , abs/2210.02414.