# Coded-Caching using Adaptive Transmission

Behrooz Makki, , Mohamed-Slim Alouini, B. Makki is with Ericsson Research,
Sweden<EMAIL_ADDRESS>M.-S. Alouini is with the King Abdullah
University of Science and Technology (KAUST), Thuwal, Makkah Province, Saudi
Arabia, Email<EMAIL_ADDRESS>
###### Abstract

Coded-caching is a promising technique to reduce the peak rate requirement of
backhaul links during high traffic periods. In this letter, we study the
effect of adaptive transmission on the performance of coded-caching based
networks. Particularly, concentrating on the reduction of backhaul peak load
during the high traffic periods, we develop adaptive rate and power allocation
schemes maximizing the network successful transmission probability, which is
defined as the probability of the event with all cache nodes decoding their
intended signals correctly. Moreover, we study the effect of different message
decoding and buffering schemes on the system performance. As we show, the
performance of coded-caching networks is considerably affected by rate/power
allocation as well as the message decoding/buffering schemes.

##  I Introduction

In 5G and beyond, the number of wireless devices, and their rate requirements,
will increase by orders of magnitude [1]. To cope with such demands, different
methods are proposed to improve the capacity and spectral efficiency. Here,
one of the promising techniques is network densification, i.e., the deployment
of many base stations (BSs) of different types such that there are more
resource blocks per unit area. The BSs, however, need to be connected to the
operators’ core network via a transport network, the problem which becomes
challenging as the number of the BSs/users increases. Particularly, the
increase of backhaul traffic may lead to backhaul congestion which, in turn,
leads to end-to-end latency increment. This is the main motivation for
wireless caching schemes reducing the backhaul load.

Caching is defined as storing popular reusable information at intermediate
nodes to reduce the backhauling load. Such a technique is of interest in
delay- and/or backhaul-constrained applications such as D2D, V2X and
integrated access and backhaul.

Not every type of information is cacheable, for example, interactive
applications such as gaming and voice calls. However, most of the network
traffic today, including trending tweets, breaking news and video, is
cacheable. Here, particular attention is paid to video. This is because,
according to [2], by 2022, $79\%$ of the world’s mobile traffic will be video.
For instance, Netflix and YouTube alone account for almost half of peak
downstream traffic in USA [3]. Also, video is typically long and pre-recorded,
which makes planning, prediction and segmentation easy. Specially, as
demonstrated in Fig. 1, video has high variation of the daily traffic profile,
and increases the backhaul peak rate during the night significantly. This is
important because the wireless network is designed based on the peak traffic.
Then, with a variant daily traffic profile, the network will be underutilized
most of the time, which is not economically viable. Thus, predicting the
videos of interest in the high-traffic (HT) periods and caching them in the
access points close to the devices during the low-traffic (LT) periods will
lead to considerable cost reduction. For these reasons, caching is currently
used by different content providers where, for instance, caches serve
approximately $60\%-80\%$ (resp. up to $90\%$) of the Netflix [4] (resp.
Facebook [5]) content requests.

Figure 1: Application traffic daily profile in western Europe [6, Fig. A3.3].

The initial (un-coded) caching schemes where based on distributing the same
information between the cache nodes, and minimizing the cache miss probability
defined as the probability of the event that the device’s requested file is
not in the cache node. Here, depending on the amount of the information
available about the sequence of the future requested files, different, e.g.,
Belady, highest-popularity-first, least-frequently-used and least-recently-
used, algorithms have been proposed [7]. Minimizing the cache miss probability
improves the average system performance. On the other hand, the fundamental
work of M. A. Maddah-ali and U. Niesen [8, 9] exploited the multicasting
opportunity of cache networks and network coding concept to introduce coded-
cashing, minimizing the worst-case backhaul peak rate in HT period. Also,
following [8, 9], there have been several works on the performance analysis of
coded-caching systems, e.g., [10, 11, 12]. In these works, it is mainly
concentrated on proper partitioning and distribution of the sub-packets
between the cache nodes while the wireless channel between the serve and the
cache nodes has been rarely studied, e.g. [13].

In this letter, we study the effect of adaptive data transmission and
different data decoding/buffering schemes on the performance of wireless
coded-caching networks. We present adaptive rate and power allocation schemes
between the sub-packets such that the network successful transmission
probability (STP) is maximized. Here, STP is defined as the probability of the
event that all cache nodes can decode their intended signals correctly.
Moreover, we investigate the effect of different decoding and buffering
schemes on the network STP. We concentrate on the worst-case peak backhaul
traffic cases where the cache nodes request for different signals during the
HT periods. As we show, the performance of coded-caching networks is
considerably affected by the decoding scheme as well as rate/power allocation.

##  II System Model

Figure 2: Coded-caching concept. Subplot A: Placements phase (at 2 AM) where
the cache nodes are filled in with different sub-packets. Subplot B: Delivery
phase (at 9 PM) where a single subpacket is braodcasted to the cache nodes.

As illustrated in Fig. 2, consider the simplest case of coded-caching networks
where a server connects to two cache nodes $\text{C}_{1}$ and $\text{C}_{2}$.
However, it is straightforward to extend the results to the cases with
different numbers of cache nodes. In general, coded-caching has two, namely,
placement and delivery, phases. During the placement phase, performed in the
LT period (say, at 2 AM), the server divides the packets, for instance,
packets $X$ and $\tilde{X}$, to sub-packets
$X_{1},X_{2},\tilde{X}_{1},\tilde{X}_{2}$ with $X=[X_{1},X_{2}]$,
$\tilde{X}=[\tilde{X}_{1}\tilde{X}_{2}],$ and fills in the caches with
different sub-packets having no knowledge, or probably a rough estimation, of
the cache nodes’ data requests during the HT period (see Fig. 2a). Assuming
the packets $X$ and $\tilde{X}$ to be of length $2L$, the sub-packets
$X_{1},X_{2},\tilde{X}_{1},\tilde{X}_{2}$ are of length $L$. In this way,
during the LT period, the server sends separate signals $[X_{1}\tilde{X}_{1}]$
and $[X_{2}\tilde{X}_{2}]$, each of length $2L,$ to $\text{C}_{1}$ and
$\text{C}_{2}$, respectively.

During the HT period (say, 9 PM), the server serves the cache nodes based on
their instantaneous data requests. Let $\bigoplus$ be the superposition
operator. Also, considering the worst-case scenario in terms of backhaul
traffic, assume that the caches $\text{C}_{1}$ and $\text{C}_{2}$ request for
different packets $X$ and $\tilde{X}$, respectively. Then, as demonstrated in
Fig. 2b, the server broadcasts a single sub-packet
$X_{2}\bigoplus\tilde{X}_{1}$ of length $L$. Also, using the accumulated
signals, each cache node may use different methods to decode its message of
interest (see Section III). In this way, as shown in [8, 9], coded-caching
reduces the peak HT backhaul traffic by $50\%$ because, unlike uncoded
caching, only a single sub-packet is broadcasted at, say, 9 PM. Finally, it is
interesting to note that the presented coded-caching approach is a specific
combination of the orthogonal multiple access (OMA) and NOMA (N:non) schemes
used in different time slots with proper packet partitioning and signal
decoding at the receivers.

Let us denote the server-cache $i$ channel coefficient by $h_{i},i=1,2,$ and
define the channel gains as $g_{i}=|h_{i}|^{2},i=1,2.$ We consider Rayleigh-
fading conditions with channel probability density functions (PDFs)
$f_{g_{i}}(u)=\lambda_{i}e^{-\lambda_{i}u},i=1,2,$ where $\lambda_{i},i=1,2,$
depends on the long-term channel quality. Then, the signals received by
$\text{C}_{1}$ and $\text{C}_{2}$ during the LT period are given by

$\displaystyle\left\\{\begin{matrix}\left[Y_{\text{C}_{1}}^{\text{LT}}(t)\,\tilde{Y}_{\text{C}_{1}}^{\text{LT}}(t)\right]=\sqrt{P}h_{1}^{\text{LT}}\left[X_{1}(t)\tilde{X}_{1}(t)\right]+\left[Z_{\text{C}_{1},1}(t)\,Z_{\text{C}_{1},2}(t)\right],t=1,\ldots,L\\\
\left[Y_{\text{C}_{2}}^{\text{LT}}(t)\,\tilde{Y}_{\text{C}_{2}}^{\text{LT}}(t)\right]=\sqrt{P}h_{2}^{\text{LT}}\left[X_{2}(t)\tilde{X}_{2}(t)\right]+\left[Z_{\text{C}_{2},1}(t)\,Z_{\text{C}_{2},2}(t)\right],t=1,\ldots,L\end{matrix}\right.$
(1)

while, at the HT period, the received signals are

$\displaystyle\left\\{\begin{matrix}Y_{C_{1}}^{\text{HT}}(t)=\sqrt{P}h_{1}^{\text{HT}}S(t)+Z_{\text{C}_{1},1}(t),t=1,\ldots,L\\\
Y_{C_{2}}^{\text{HT}}(t)=\sqrt{P}h_{2}^{\text{HT}}S(t)+Z_{\text{C}_{2},1}(t),t=1,\ldots,L\\\
S(t)=\alpha
X_{2}(t)+\sqrt{1-\alpha^{2}}\tilde{X}_{1}(t).\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\end{matrix}\right.$
(2)

Here, $h_{i}^{\text{LT}}$ and $h_{i}^{\text{HT}},i=1,2,$ represent the channel
realizations of $h_{i}$ at LT and HT periods, respectively. Then,
$Z_{\text{C}_{i},j},i,j=1,2,$ denotes the unit-variance additive Gaussian
noise, and $P$ is the server maximum transmit power. Also,
$X_{i},\tilde{X}_{i},i=1,2,$ are the unit-variance signals of the sub-packets,
$S(t)$ represents the unit-variance superimposed signals of $X_{2}$ and
$\tilde{X}_{1}$, and $\alpha\in[0,1]$ gives the power partitioning between
these signals.

##  III Analytical results

Let us denote the number of information bits in the packets $X$ and
$\tilde{X}$ by $K$ and $\tilde{K}$, respectively. Also, the information nats
are divided between the sub-packets according to

$\displaystyle\left\\{\begin{matrix}K=K_{1}+K_{2}\\\
\tilde{K}=\tilde{K}_{1}+\tilde{K}_{2},\end{matrix}\right.$ (3)

which, defining the code rates $R=\frac{K}{2L}$ and
$\tilde{R}=\frac{\tilde{K}}{2L},$ leads to

$\displaystyle\left\\{\begin{matrix}2R=R_{1}+R_{2}\\\
2\tilde{R}=\tilde{R}_{1}+\tilde{R}_{2}.\end{matrix}\right.$ (4)

Our goal is to design proper rate allocation between sub-packets, i.e.,
$R_{i},\tilde{R}_{i},i=1,2,$ and power split parameter $\alpha$ in (2) such
that the network STP is maximized. STP is defined as

$\displaystyle\Pr(\text{ST})=\frac{1}{2}\left(\Pr\left(\text{C}_{1}\,\text{successful}\right)+\Pr\left(\text{C}_{2}\,\text{successful}\right)\right),$
(5)

i.e., the probability of the event that both cache nodes can decode their
intended packets correctly. Depending on the buffering and the coding schemes
of the cache nodes, the network may lead to different STPs as follows.

###  III-A Joint Decoding at HT Period using Successive Interference
Cancellation

In one approach, referred to as Method 1 in the following, the cache nodes
buffer the signals received in the LT period, and use both maximum ratio
combining (MRC) and successive interference cancellation (SIC) for message
decoding at HT periods. Let us concentrate on cache node $\text{C}_{1}$.
Receiving $Y_{\text{C}_{1}}^{\text{HT}}$ in (2) during the HT period and with
$\tilde{Y}_{\text{C}_{1}}^{\text{LT}}$ in (1) already buffered, $\text{C}_{1}$
first uses MRC to decode $\tilde{X}_{1}(t)$. Then, with a successful decoding
of $\tilde{X}_{1}(t)$, node $\text{C}_{1}$ uses the SIC concept to remove
$\tilde{X}_{1}(t)$ from $Y_{\text{C}_{1}}^{\text{HT}}(t)$, leading to an
interference-free signal

$\displaystyle\mathcal{Y}_{\text{C}_{1}}^{\text{HT}}=\sqrt{P}h_{1}^{\text{HT}}\alpha
X_{2}(t)+Z_{\text{C}_{1},1}(t).$ (6)

Finally, the decoder generates the concatenated signal
$[Y_{\text{C}_{1}}^{\text{LT}}(t)\,\mathcal{Y}_{\text{C}_{1}}^{\text{HT}}(t)]$,
with $Y_{\text{C}_{1}}^{\text{LT}}(t)$ and
$\mathcal{Y}_{\text{C}_{1}}^{\text{HT}}(t)$ given in (1) and (6),
respectively, and decodes the packet $X$ in _one-shot._ In this way, and
considering the same procedure in $\text{C}_{2}$ by using (1)-(2), STP in (5)
is given by

$\displaystyle\Pr(\text{ST})=\frac{1}{2}(\eta_{1}\gamma_{1}+\eta_{2}\gamma_{2}),$
$\displaystyle\eta_{1}=\Pr\left(\log\left(1+\frac{(1-\alpha^{2})Pg_{1}^{\text{HT}}}{1+\alpha^{2}Pg_{1}^{\text{HT}}}+Pg_{1}^{\text{LT}}\right)\geq\tilde{R}_{1}\right),$
$\displaystyle\gamma_{1}=\Pr\left(\log\left(1+Pg_{1}^{\text{LT}}\right)+\log\left(1+\alpha^{2}Pg_{1}^{\text{HT}}\right)\geq
2R\right)$
$\displaystyle\eta_{2}=\Pr\left(\log\left(1+\frac{\alpha^{2}Pg_{2}^{\text{HT}}}{1+(1-\alpha^{2})Pg_{2}^{\text{HT}}}+Pg_{2}^{\text{LT}}\right)\geq
R_{2}\right),$
$\displaystyle\gamma_{2}=\Pr\left(\log\left(1+Pg_{2}^{\text{LT}}\right)+\log\left(1+(1-\alpha^{2})Pg_{2}^{\text{HT}}\right)\geq
2\tilde{R}\right).$ (7)

Here, $\eta_{1}$ (resp. $\eta_{2}$) is the probability of successful decoding
of $\tilde{X}_{1}(t)$ (resp. $X_{2}(t)$) at $\text{C}_{1}$ (resp.
$\text{C}_{2}$) using MRC. Then, $\gamma_{1}$ and $\gamma_{2}$ give the
probability that, removing the interference from the received signal in HT
period, the caches can decode their intended signals correctly. Note that in
$\gamma_{1}$ and $\gamma_{2}$ we have used the results on the maximum
achievable rates of parallel Gaussian channels.

Considering Rayleigh-fading conditions, we have

$\displaystyle\eta_{1}=1-\Pr\left(\frac{(1-\alpha^{2})Pg_{1}^{\text{HT}}}{1+\alpha^{2}Pg_{1}^{\text{HT}}}+Pg_{1}^{\text{LT}}\leq
e^{\tilde{R}_{1}}-1\right)=$ $\displaystyle
1-\int_{0}^{\frac{e^{\tilde{R}_{1}}-1}{P}}f_{g_{1}}(y)\Pr\left(\frac{(1-\alpha^{2})Pg_{1}^{\text{HT}}}{1+\alpha^{2}Pg_{1}^{\text{HT}}}\leq{e^{\tilde{R}_{1}}-1}-Py\right)\text{d}y$
$\displaystyle=e^{-\frac{\lambda_{1}\left({e^{\tilde{R}_{1}}-1}\right)}{P}}+\int_{\frac{\alpha^{2}e^{\tilde{R}_{1}}-1}{\alpha^{2}P}}^{\frac{e^{\tilde{R}_{1}}-1}{P}}e^{\lambda_{1}\left(y+\frac{{e^{\tilde{R}_{1}}-1}-Py}{\alpha^{2}P-(1-\alpha^{2})P\left({e^{\tilde{R}_{1}}-1}-Py\right)}\right)}\text{d}y,$
(8)

which can be calculated numerically. Also, following the same procedure, we
have

$\displaystyle\eta_{2}=e^{-\frac{\lambda_{2}\left(e^{R_{2}}-1\right)}{P}}+\int_{\frac{\left(1-\alpha^{2}\right)e^{R_{2}}-1}{\left(1-\alpha^{2}\right)P}}^{\frac{e^{R_{2}}-1}{P}}{\lambda_{2}e^{-\lambda_{2}\left(y+\frac{e^{R_{2}}-1-Py}{\alpha^{2}P-\left(1-\alpha^{2}\right)P\left(e^{R_{2}}-1-Py\right)}\right)}}\text{d}y.$
(9)

The terms $\gamma_{i},i=1,2,$ on the other hand, do not have closed-form or
easy-to-deal integration expressions. Thus, we use the Jensen’s inequality
$\frac{1}{n}\sum_{i=1}^{n}\log(1+x_{i})\leq\log\left(1+\frac{1}{n}\sum_{i=1}^{n}x_{i}\right)$
[14, Eq. (30)] and $f_{g_{i}}(u)=\lambda_{i}e^{-\lambda_{i}u},i=1,2,$ to
rephrase $\gamma_{i},i=1,2,$ as

$\displaystyle\gamma_{1}$ $\displaystyle\leq
1-\Pr\left(g_{1}^{\text{LT}}+\alpha^{2}g_{1}^{\text{HT}}\leq\frac{2}{P}\left(e^{R}-1\right)\right)$
$\displaystyle=1-\int_{0}^{\frac{2\left(e^{R}-1\right)}{P}}{f_{g_{1}}(x)\Pr\left(g_{1}^{\text{HT}}\leq\frac{\frac{2\left(e^{R}-1\right)}{P}-x}{\alpha^{2}}\right)}\text{d}x$
$\displaystyle=e^{\frac{-2\lambda_{1}\theta}{P}}+\frac{\alpha^{2}e^{\frac{-2\lambda_{1}\theta}{P\alpha^{2}}}}{\alpha^{2}-1}\left(1-e^{\frac{-2\lambda_{1}\left(\alpha^{2}-1\right)\theta}{P\alpha^{2}}}\right),$
(10) $\displaystyle\gamma_{2}$ $\displaystyle\leq
e^{\frac{-2\lambda_{2}\theta}{P}}+\frac{\left(\alpha^{2}-1\right)e^{\frac{-2\lambda_{2}\theta}{P\left(1-\alpha^{2}\right)}}}{\alpha^{2}}\left(1-e^{\frac{-2\lambda_{2}\alpha^{2}\theta}{P\left(\alpha^{2}-1\right)}}\right),$
(11)

where (11) follows the same procedure as in (III-A). In this way, the optimal
rate/power allocation maximizing STP is given by

$\displaystyle\max\frac{1}{2}\\{\eta_{1}\gamma_{1}+\eta_{2}\gamma_{2}\\}$
$\displaystyle\text{s.t.
}\,\alpha\in[0,1],R_{1}+R_{2}=2R,\,\tilde{R}_{1}+\tilde{R}_{2}=2\tilde{R},$
(12)

which can be effectively solved by, e.g., exhaustive search or the machine-
learning based scheme of [15].

###  III-B Joint Decoding at HT Period without SIC

Implementation of MRC and SIC, to decode and remove the interference,
increases the decoding complexity/delay. Also, SIC suffers from error
propagation problem, e.g., [16]. For these reasons, in Method 2, each cache
node decodes its intended packets in one-shot by considering the interference
as an additive noise. Here, (III-A) is rephrased as

$\displaystyle\Pr(\text{ST})=\frac{1}{2}(\bar{\gamma}_{1}+\bar{\gamma}_{2}),$
$\displaystyle\bar{\gamma}_{1}=\Pr\left(\log\left(1+Pg_{1}^{\text{LT}}\right)+\log\left(1+\frac{\alpha^{2}Pg_{1}^{\text{HT}}}{1+(1-\alpha^{2})Pg_{1}^{\text{HT}}}\right)\geq
2R\right)$
$\displaystyle\bar{\gamma}_{2}=\Pr\left(\log\left(1+Pg_{2}^{\text{LT}}\right)+\log\left(1+\frac{(1-\alpha^{2})Pg_{2}^{\text{HT}}}{1+\alpha^{2}Pg_{2}^{\text{HT}}}\right)\geq
2\tilde{R}\right),$ (13)

where, using the Jensen’s inequality and the same procedure as in (III-A), we
have

$\displaystyle\bar{\gamma}_{1}\leq
e^{-\frac{2\lambda_{1}\left(e^{R}-1\right)}{P}}+\int_{\frac{2\left(e^{R}-1\right)-\frac{\alpha^{2}}{1-\alpha^{2}}}{P}}^{\frac{2\left(e^{R}-1\right)}{P}}\lambda_{1}e^{-\lambda_{1}\left(x+\frac{2\left(e^{R}-1\right)-Px}{\alpha^{2}P+(1-\alpha^{2})P(Px-2\left(e^{R}-1\right))}\right)}\text{d}x,$
$\displaystyle\bar{\gamma}_{2}\leq
e^{-\frac{2\lambda_{2}\left(e^{\tilde{R}}-1\right)}{P}}+\int_{\frac{2\left(e^{\tilde{R}}-1\right)-\frac{1-\alpha^{2}}{\alpha^{2}}}{P}}^{\frac{2\left(e^{\tilde{R}}-1\right)}{P}}\lambda_{2}e^{-\lambda_{2}\left(x+\frac{2\left(e^{\tilde{R}}-1\right)-Px}{(1-\alpha^{2})P+\alpha^{2}P(Px-2\left(e^{\tilde{R}}-1\right))}\right)}\text{d}x,$
(14)

which can be calculated numerically.

Finally, note that, replacing (III-B) into (III-A), the optimal performance of
the cache nodes in Method 2 is independent of the rate split between the sub-
packets. This, although Method 1 gives the best performance in terms of the
worst-case peak traffic, may give an advantage to Method 2, compared to Method
1. This is because in Method 1 the rate split is performed by considering the
worst-case condition with the cache nodes requesting for different signals
during HT period. However, if the caches request for the same signals during
HT period, the rate split scheme of Method 1 is not necessarily optimal. As
opposed, in Method 2, the rate split is independent of the caches requested
signals in HT periods.

###  III-C Separate Decoding using SIC

In Methods 1-2, one needs to follow the coding schemes of incremental
redundancy hybrid automatic repeat request (HARQ)-based protocols or Raptor
codes, e.g., [17], where the message is decoded by concatenating different
sub-packets. Alternatively, in Method 3, we consider the case where, while MRC
and SIC are used to decode and remove the interference signal, respectively,
each cache node decodes its sub-packets of interest separately. That is,
considering $\text{C}_{1},$ $X_{1}$ (resp. $X_{2}$) is decoded during the LT
(resp. HT) period. In this case, the STP (III-A) is changed to

$\displaystyle\Pr\left(\text{ST}\right)=\frac{1}{2}\left(\eta_{1}\breve{\gamma}_{11}\breve{\gamma}_{12}+\eta_{2}\breve{\gamma}_{21}\breve{\gamma}_{22}\right),$
$\displaystyle\breve{\gamma}_{11}=\Pr\left(\log\left(1+Pg_{1}^{\text{LT}}\right)\geq
R_{1}\right)=e^{-\frac{\lambda_{1}\left(e^{R_{1}}-1\right)}{P}},$
$\displaystyle\breve{\gamma}_{12}=\Pr\left(\log\left(1+\alpha^{2}Pg_{1}^{\text{HT}}\right)\geq
R_{2}\right)=e^{-\frac{\lambda_{1}\left(e^{R_{2}}-1\right)}{\alpha^{2}P}}$
$\displaystyle\breve{\gamma}_{21}=\Pr\left(\log\left(1+(1-\alpha^{2})Pg_{2}^{\text{HT}}\right)\geq\tilde{R}_{1}\right)=e^{-\frac{\lambda_{2}\left(e^{\tilde{R}_{1}}-1\right)}{(1-\alpha^{2})P}},$
$\displaystyle\breve{\gamma}_{22}=\Pr\left(\log\left(1+Pg_{2}^{\text{LT}}\right)\geq\tilde{R}_{2}\right)=e^{-\frac{\lambda_{2}\left(e^{\tilde{R}_{2}}-1\right)}{P}},$
(15)

with $\eta_{i},i=1,2,$ given in (III-A), and (III-A) is adapted
correspondingly. In (III-C), $\breve{\gamma}_{11}$ is the probability that
$\text{C}_{1}$ decodes $X_{1}$ during the LT period. Also,
$\breve{\gamma}_{12}$ gives the probability that, after decoding and removing
$\tilde{X}_{1}$, the cache node $\text{C}_{1}$ correctly decodes $X_{2}$ in
the HT period. Also, the same arguments hold for $\breve{\gamma}_{2i},i=1,2.$

Note that, although Method 1 maximizes the achievable rate/STP, Method 3 has a
number of advantages including:

  * •

Low decoding complexity: Because, as opposed to Methods 1-2 decoding long
codewords of length $2L$, Method 3 is based on decoding sub-packets of length
$L$.

  * •

Efficient HARQ-based transmissions: In Methods 1-2, all packets are decoded
during the HT periods and, in case of decoding failure, the message is
retransmitted at that period. Such HARQ-based retransmissions increase the
backhauling load at HT period. As opposed, in Method 3, the decoding of the
first received sub-packets and all their required HARQ-based retransmissions
are performed during the LT period, which reduces the backhauling cost of
HARQ.

Finally, depending on the considered method, the buffering scheme of the
caches during the LT period may change. Particularly, in Methods 1-2 the
caches buffer the signals received during LT period without decoding. In
Method 3, however, the caches buffer the sub-packets successfully decoded
during LT period.

###  III-D Separate Decoding without SIC

To further reduce the complexity of Method 3, one can consider the case where,
while decoding the sub-packets separately, the cache nodes consider the
interference as an additive noise (Method 4). In this case, where the sub-
packets are decoded in different LT and HT periods without SIC, the STP is
given by

$\displaystyle\Pr\left(\text{ST}\right)=\frac{1}{2}(\breve{\gamma}_{11}\hat{\gamma}_{12}+\hat{\gamma}_{21}\breve{\gamma}_{22}),$
$\displaystyle\hat{\gamma}_{12}=\Pr\left(\log\left(1+\frac{\alpha^{2}Pg_{1}^{\text{HT}}}{1+(1-\alpha^{2})Pg_{1}^{\text{HT}}}\right)\geq
R_{2}\right)$
$\displaystyle\,\,\,\,\,\,\,\,=\left\\{\begin{matrix}0\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,&\text{if
}R_{2}\geq-\log\left(1-\alpha^{2}\right)\\\
e^{-\lambda_{1}\frac{\left(e^{R_{2}}-1\right)}{\alpha^{2}P-(1-\alpha^{2})P\left(e^{R_{2}}-1\right)}}&\text{otherwise}\end{matrix}\right.$
$\displaystyle\hat{\gamma}_{21}=\Pr\left(\log\left(1+\frac{(1-\alpha^{2})Pg_{2}^{\text{HT}}}{1+\alpha^{2}Pg_{2}^{\text{HT}}}\right)\geq\tilde{R}_{1}\right)$
$\displaystyle=\left\\{\begin{matrix}0\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,&\text{if
}\tilde{R}_{1}\geq-2\log\alpha\\\
e^{-\lambda_{2}\frac{\left(e^{\tilde{R}_{1}}-1\right)}{(1-\alpha^{2})P-\alpha^{2}P\left(e^{\tilde{R}_{1}}-1\right)}}&\text{otherwise}\end{matrix}\right.$
(16)

with $\breve{\gamma}_{11}$ and $\breve{\gamma}_{22}$ given in (III-C). Also,
in (III-D) we use Rayleigh channel PDFs and some manipulations to derive the
probabilities. For further comparisons between Methods 1-4, see Section IV.

##  IV Simulation Results

The simulation results are presented for the cases with $\lambda_{1}=1$ and
$\lambda_{2}=0.1$, i.e., with $10$ dB difference between the channel gains of
the server-cache links, and we define the transmission signal-to-noise ratio
(SNR) as $10\log_{10}P$, considering the additive noises to be unit-variance.
Note that we have evaluated the results for different parameter settings, and
they show the same qualitative conclusions as those presented in the
following. In Figs. 3-4, the results are obtained by optimizing the rate and
power allocation. Here, both exhaustive search and the genetic-algorithm based
scheme of [15] have been used which have ended up in the same results,
indicating the accuracy of the optimization process. In Fig. 5, we study the
effect of rate/power allocation.

Figure 3: Comparison between the STP of Methods 1-4, $\lambda_{1}=1,$
$\lambda_{2}=0.1,$ $R=\tilde{R}=1$ npcu. The results are obtained by optimal
rate and power allocation.

Considering $R=\tilde{R}=1$ nats-per-channel-use (npcu), Fig. 3 compares the
performance of Methods 1-4. Also, the figure verifies the tightness of the
Jensen’s inequality-based approximation results of (III-A)-(11). Then, Fig. 4
shows the STP versus the data rates $R=\tilde{R}$ for the cases with different
decoding/buffering methods and transmission SNRs.

Considering Methods 1 and 3, with joint and separate decoding on the sub-
packets, respectively, Fig. 5 studies the effect of optimal rate and power
allocation on the network STP. Particularly, the figure compares the optimal
results obtained by (III-A) with the cases using uniform power allocation,
i.e., with $\alpha=\frac{\sqrt{2}}{2}$ in (2), and/or equal rate split, i.e.,
$R_{i}=\tilde{R}_{i},i=1,2$. According to the figures, the following
conclusions can be drawn:

Figure 4: Achievable STP versus the data rate for different methods. The
results are obtained by optimal rate and power allocation and
$\lambda_{1}=1,\lambda_{2}=0.1.$
Figure 5: On the effect of optimal rate and power allocation in Methods 1 and
3, $R=\tilde{R}=1$ npcu and $\lambda_{1}=1,\lambda_{2}=0.1$.

  * •

The approximation results of (III-A)-(11) properly approximate the
probabilities $\gamma_{i},i=1,2$ (Fig. 3. Also, the same point is observed for
$\bar{\gamma}_{i},i=1,2,$ in (III-B) although not shown in the figure). Thus,
the approximations can be well utilized for the performance evaluation of
Methods 1-2, i.e., in the cases with joint decoding of the sub-packets.

  * •

Compared to the cases with separate decoding of sub-packets, i.e., Methods
3-4, considerable STP improvement is observed by joint decoding of the sub-
packet, i.e., Methods 1-2 (Figs. 3-4). However, as explained in Section III.C,
the STP increment of Methods 1-2 is at the cost of decoding complexity and
possible HARQ-based retransmissions at HT periods. On the other hand, for both
cases with joint and separate decoding of the sub-packets, using SIC-based
interference cancellation leads to marginal performance improvement at low
rates while its effect increases with the data rate (Fig. 4). Finally, as the
data rate increases, the performance gap between Methods 2 and 3 decreases,
i.e., one can reach the same STP as in the cases with joint sub-packet
decoding of interference-affected signals by separate sub-packets decoding if
the interference signals are removed using SIC. (Fig. 4).

  * •

For all parameter settings, Method 1 leads to the highest STP, compared to
Methods 2-4, at the cost of decoding delay/possible retransmissions at HT
periods (Figs. 3-4). For instance, with the parameter settings of Fig. 4 and
data rate 1.5 npcu, the implementation of Method 1 with transmit SNR 10 dB
results in the same STP, $80\%$, as in the cases with Method 4 and SNR 15 dB,
i.e., using advanced decoding methods leads to 5 dB gain in SNR (Fig. 4).

  * •

For both cases with and without interference cancellation (Methods 1 and 3),
optimal rate allocation leads to considerable STP increment (Fig. 5. The same
conclusion is observed in Methods 2 and 4, although not presented in the
figure). Also, the relative performance gain of optimal rate split increases
in the cases with interference cancellation. Finally, considering the
interference as additive noise, optimal power allocation between sub-packets
during HT period increases the STP. However, with interference cancellation
and joint decoding of sub-packets, the effect of optimal power allocation
between HT period sub-packets is marginal (Fig. 5).

##  V Conclusions

This paper studied the performance coded-caching networks in the cases with
adaptive rate/power allocation and different decoding/buffering schemes. As we
showed, joint decoding of the sub-packets at HT periods leads to considerable
performance improvement of coded-caching setups. Also, for different decoding
schemes, optimal rate split between the sub-packets increases the STP
considerably while optimal power allocation between the sub-packets of HT
period only improves the STP if SIC-based receiver is not implemented and the
sub-packets are decoded separately.

## References

  * [1] R. Rajatheva, et. al, ”Scoring the Terabit/s Goal:Broadband Connectivity in 6G,” arXiv:2008.07220, Aug. 2020, Available at: https://arxiv.org/pdf/2008.07220.pdf.
  * [2] Cisco, https://www.digitaltveurope.com/2019/02/20/cisco-79-of-worlds-mobile-traffic-to-be-video-by-2022/.
  * [3] D. Clark, S. Bauer, W. Lehr, K. Claffy, B. Dhamdhere, A. Huffaker, and M. Luckie, “Measurement and analysis of internet interconnection and congestion,” in _Proc. TPRC’2014_ , Tx, USA, Sept. 2014, pp. 1–5.
  * [4] Netflix, http://oc.nflxvideo.net/docs/OpenConnect-Deployment-Guide.pdf.
  * [5] Q. Huang, et. al, ”An Analysis of Facebook Photo Caching,” Proc. 20th ACM Symposium on Operating Systems Prnciples, Brighton, UK, 2013.
  * [6] Report ITU-R M.2370-0, ”IMT traffic estimates for the years 2020 to 2030”, Available at: https://www.itu.int/dms_pub/itu-r/opb/rep/R-REP-M.2370-2015-PDF-E.pdf.
  * [7] M. Zhang, H. Luo, and H. Zhang, “A survey of caching mechanisms in information-centric networking,” _IEEE Commun. Surveys Tuts._ , vol. 17, no. 3, pp. 1473–1499, Third quarter 2015.
  * [8] M. A. Maddah-Ali and U. Niesen, “Fundamental limits of caching,” _IEEE Trans. Inf. Theory_ , vol. 60, no. 5, pp. 2856–2867, May 2014.
  * [9] ——, “Decentralized coded caching attains order-optimal memory-rate tradeoff,” _IEEE/ACM Trans. Netw._ , vol. 23, no. 4, pp. 1029–1040, Aug. 2015.
  * [10] N. Naderializadeh, M. A. Maddah-Ali, and A. S. Avestimehr, “Fundamental limits of cache-aided interference management,” _IEEE Trans. Inf. Theory_ , vol. 63, no. 5, pp. 3092–3107, 2017.
  * [11] J. Zhang and O. Simeone, “Fundamental limits of cloud and cache-aided interference management with multi-antenna edge nodes,” _IEEE Trans. Inf. Theory_ , vol. 65, no. 8, pp. 5197–5214, 2019.
  * [12] K. Wan, G. Caire, ”On Coded Caching with Private Demands,” arXiv:1908.10821, June 2020, Available at: https://arxiv.org/pdf/1908.10821.pdf.
  * [13] S. Mohajer, I. Bergel, and G. Caire, “Cooperative wireless mobile caching: A signal processing perspective,” _IEEE Signal Proc. Mag._ , vol. 37, no. 2, pp. 18–38, Mar. 2020.
  * [14] B. Makki, T. Svensson, T. Eriksson, and M. Alouini, “On the performance of RF-FSO links with and without hybrid ARQ,” _IEEE Trans. Wireless Commun._ , vol. 15, no. 7, pp. 4928–4943, July 2016.
  * [15] B. Makki, T. Svensson, and M. Alouini, “On the throughput of large-but-finite MIMO networks using schedulers,” _IEEE Trans. Wireless Commun._ , vol. 18, no. 1, pp. 152–166, Jan. 2019.
  * [16] B. Makki, T. Svensson, M. Zorzi, ”An Error-limited NOMA-HARQ Approach using Short Packets,” arXiv:2006.14315, Available at: https://arxiv.org/pdf/2006.14315.pdf.
  * [17] B. Makki and T. Eriksson, “On hybrid ARQ and quantized CSI feedback schemes in quasi-static fading channels,” _IEEE Trans. Commun._ , vol. 60, no. 4, pp. 986–997, April 2012.


# A Context-Aware Citation Recommendation Model
with BERT and Graph Convolutional Networks

Chanwoo Jeong1 Sion Jang1 Hyuna Shin1 Eunjeong Park2,111Co-corresponding
Authors Sungchul Choi1,11footnotemark: 1
1 TEAMLAB, Gachon University, Republic of Korea
2 Papago, NAVER, Republic of Korea
{chanwooda0223, wayterren<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

With the tremendous growth in the number of scientific papers being published,
searching for references while writing a scientific paper is a time-consuming
process. A technique that could add a reference citation at the appropriate
place in a sentence will be beneficial. In this perspective, context-aware
citation recommendation has been researched upon for around two decades. Many
researchers have utilized the text data called the context sentence, which
surrounds the citation tag, and the metadata of the target paper to find the
appropriate cited research. However, the lack of well-organized benchmarking
datasets and no model that can attain high performance has made the research
difficult.

In this paper, we propose a deep learning based model and well-organized
dataset for context-aware paper citation recommendation. Our model comprises a
document encoder and a context encoder, which uses Graph Convolutional
Networks (GCN) layer and Bidirectional Encoder Representations from
Transformers (BERT), which is a pre-trained model of textual data. By
modifying the related PeerRead dataset, we propose a new dataset called
FullTextPeerRead containing context sentences to cited references and paper
metadata. To the best of our knowledge, This dataset is the first well-
organized dataset for context-aware paper recommendation. The results indicate
that the proposed model with the proposed datasets can attain state-of-the-art
performance and achieve a more than 28% improvement in mean average precision
(MAP) and recall@k.

##  1 Introduction

Have you ever had difficulty citing references while writing a scientific
paper? Most artificial intelligence researchers may have thought about finding
a solution for this problem a little easier at least once. As shown in Figure
1, a possible solution is to automatically find the information that should be
cited in a placeholder, such as [REF].

Figure 1: An example manuscript with citation placeholders to find fittable
references. This text is from BERT paper Devlin et al. (2018)

Finding a suitable scientific document–considering the surrounding text–that
can be used as a placeholder is called ”Context-aware citation recommendation”
He et al. (2010). The sentences on both sides of a placeholder are referred to
as ”context”. A context-aware citation recommendation task is a type of
supervised classification that is used to choose a paper suitable as the
placeholder based on its contents. In addition to context, by considering the
characteristics of a scientific document, the task has conducted with using
author, title, citation, journal (or conference name), etc., which are
metadata, or bibliometrics, of a scientific paper He et al. (2010, 2011); Liu
et al. (2015); Fontoura et al. (2013); Tang et al. (2014). In recent years,
there has been an increasing number of attempts to solve problems with the use
of a deep neural net; this resulted in an increase in the number of articles
Huang et al. (2015); Ebesu and Fang (2017); Yang et al. (2018).

Even though the task has been a relatively constantly researched area, one of
the most challenging aspects of this study is that there is no benchmarking
dataset against which proper performance can be measured. In general, this
task needs to use metadata along with the context surrounding the cited paper.
To the best of our knowledge, suitable datasets have not been disclosed. Among
the commonly used data, the ACL Anthology Network (AAN)
222http://clair.eecs.umich.edu/aan/index.php dataset does not provide paper
sentences and metadata in a preprocessed form, and the DBLP dataset
333https://dblp.uni-trier.de/xml/ only provides bibliographic information. In
a recently published Huang et al. (2015), CiteseerX datasets
444https://psu.app.box.com/v/refseer only provided context and citation
information and did not provide meta information simultaneously. As a result,
related studies have failed to use the same benchmarking dataset.

The purpose of this study is to provide datasets and state-of-the-art models
suitable for the context-aware paper recommendation task research, and in turn
provide researchers with an improved paper writing environment. The main
contributions of this study are as follows: First, we built reproducible
benchmarking datasets for the task. We preprocessed the existing AAN dataset
Radev et al. (2013, 2009); Dragomir R. Radev (2009) to fit the task, and
constructed new dataset called FullTextPeerRead using PeerRead
555https://github.com/allenai/PeerRead Kang et al. (2018). Second, we
constructed the state-of-the-art model for the task using BERT Devlin et al.
(2018) and Graph Convolution Networks (GCN) Kipf and Welling (2016a). Because
scientific papers contain textual contents data, and metadata that can be
represented as a graph, we use BERT, which recently proved to have the highest
performance level in the field of Natural Language Processing (NLP) for
textual data, and GCN for network-based metadata. Finally, we investigated
various factors to affect task performance through experiments.

##  2 Proposed Dataset

###  2.1 Dataset overview

We constructed two new datasets for the context-aware citation recommendation
task. We suggested revisions of the existing datasets, AAN Radev et al. (2013)
and FullTextPeerRead which is an expansion of PeerRead dataset Kang et al.
(2018). AAN and PeerRead datasets have well-organized bibliometric
information. The PeerRead dataset mainly provides peer reviews of submitted
papers in top-tier venues of the artificial intelligence field, along with
bibliometric information. Since all existing datasets of the task lack
information in the citation context, our focus is on gathering context
information with metadata. Therefore, the AAN and PeerRead datasets were
reprocessed to create the dataset.

###  2.2 Data acquistion

We used arXiv Vanity 666https://www.arxiv-vanity.com/ to create the dataset.
arXiv Vanity is a site that converts a latex-based PDF file to HTML document.
Our goal is to extract the context information on both sides of the citation
symbol, such as [1] or [Choi et al. 2016], together with the reference paper
information. To do this, we parsed latex into HTML via arXiv Vanity, and used
a regular expression to recognize the citation symbol in the document. Next,
the sentences on both sides of the citation symbol were stored on a database
with reference article information. We stored the collected information
together with the existing metadata and build it into a database.

The data actually collected was noisy because the latex documents were not
consistent format. After we automatically collected the necessary data, we
removed the noisy data manually. For example, in the case of CiteseerX
777https://psu.app.box.com/v/refseer, the citation symbol corresponding to the
placeholder is left in the context and data is provided. However, in this
case, the placeholder text itself was used for overfitting learning, so the
text can be used to tell the correct answer. Exactly, the reminded placeholder
can be used as crucial evidence of prediction. Therefore, our final work was
post-processed manually because this noise may remain when mechanically
collected.

###  2.3 Statics of dataset

Finally, the statics of the dataset we built are as shown in Table 1. The
number of extracted datasets were less than the original number of AAN or
PeerRead dataset because we needed to remove paper the PDFs that did not use
latex or were very noisy from being processed with arXiv Vanity. In Table 1
below, Total ”# of base papers” is a paper that cites other researches. We
have metadata information of the paper that is used as an input for the
classification task. ”# of cited papers” is a cited paper. In addition, we
extracted paragraph units on both sides of the citation symbol, and ”# of
citation context” means the sum of the number of sentences which are in the
extracted paragraph. Further, ”# of total papers” refers to the total number
of papers covered by base paper and cited paper, excluding duplicates.

Dataset name | AAN888http://bit.ly/2IDam8J | FullTextPeerRead999http://bit.ly/2Srkdht
---|---|---
# of total papers | 7,073 | 4,898
# of base papers | 5,576 | 3,761
# of cited papers | 2,417 | 2,478
# of citation context | 12,125 | 17,247
# Paper published year | 1965 - 2015 | 2007 - 2017
Table 1: Dataset description. Because the AAN policy prohibits disclosure of
modified data, we cannot disclose it. We will open the dataset after receiving
a grant from the AAN

##  3 A BERT-GCN Model for Context-Aware Citation Recommendation

###  3.1 Model overview

We construct the context-aware citation recommendation model using BERT Devlin
et al. (2018) and GCN Kipf and Welling (2016a). BERT is one of the highest
performing pre-trained models for NLP learning representation. We expect that
the learning presentation of context sentences, through pre-trained BERT, will
achieve a high performance. Scientific data, such as papers, also contain
various metadata, in addition to textual data. We use the GCN model to
represent the citation relationship between papers and to extract a learning
representation of them.

As shown in Figure 2, we construct a context encoder to extract textual
embedding, using BERT, and a citation encoder to extract graph embedding from
GCN. Each encoder is pre-trained with context data, and citation graph data is
extracted from the paper. Then data is inserted into the pre-trained models,
and concatenated embeddings are calculated by each encoder. Finally, after
passing the concatenated vectors to a feedforward neural net, the softmax
output layer is generated, and cross entropy is adopted as a loss function for
training.

Figure 2: A BERT-GCN model architecture

The structure of the proposed model is linked to the baseline CACR Yang et al.
(2018). CACR has both a paper encoder and a citation context encoder. CACR
demonstrates the performance of SOTA as the most recent context-aware citation
recommendation model using an AAN dataset and an LSTM model. In the CACR
model, a paper encoder was constructed using author, venue, and abstract
information in the paper. Our model constructed the citation encoder with GCN
solely using citation information.

###  3.2 Citation encoder

The citation encoder conducts unsupervised learning for citation, linking a
prediction with the GCN-based Variational Graph Auto-Encoders (VGAE) model
Kipf and Welling (2016b) by using citation relationships between papers as
input values. When paper information is used as input to a pre-trained GCN,
the model returns the relational learning representation as the embedding
vector. VGAE can capture the latent learning representation of graph data.

In existing research, it has been a challenge to convey the citation
relationship of a paper as Doc2Vec Le and Mikolov (2014) has been used to
encode paper information and summarize it after embedding the learning of the
individual meta-information. Our citation encoder complemented this by using
citation linking prediction information as a citation prediction feature.

####  3.2.1 Graph Convolutional Network (GCN) Layer

In our model, the role of the GCN layer is to abstract the citation network
graph information through a convolutional network. The GCN layer is used as an
inference model of VGAE. The formula of the GCN layer for VGAE is shown in
Equation 1.

$GCN(X,A)=\tilde{A}ReLU(\tilde{A}XW_{0})W_{1}\\\ $ (1)

The proposed model consists of two GCN layers. The layer uses two matrices as
input: the identity matrix $X$ and the adjacency matrix $A$, which is an $N$
by $N$ matrix. $N$ is the number of input paper. Learned from the first GCN
layer, layer parameter $W^{0}$ is used as the weight matrix for the second
layer. Each layer uses layer-wise propagation.

$\tilde{A}$ is a normalized adjacency matrix based on the diagonal degree
matrix D, as shown in Equation 2.

$\tilde{A}=D^{-\frac{1}{2}}AD^{-\frac{1}{2}}$ (2)

####  3.2.2 Variational Graph AutoEncoder (VGAE)

Figure 3: A architecture of Variational Graph AutoEncoder

VGAE is a model that applies the unsupervised learning method of the
Variational Autoencoder Rezende et al. (2014) to the graph structure using
GCN. VGAE learns the latent representation by minimizing the cost between
inference model and the generative model as shown in Equation 3.

$L=\mathbb{E}_{q(Z|X,A)}[\log p(A|Z)]-KL[q(Z|X,A)||p(Z)]$ (3)

The inference layer of VGAE learns the representation Z by to decrease the KL-
divergence loss between the normal distribution from the GCN layer result and
the Gaussian normal distribution, as shown in Equation 4.

$\displaystyle q(Z|X,A)$
$\displaystyle=\prod\;_{i=1}^{N}q(z_{i}|X,A),q(z_{i}|X,A)$ (4)
$\displaystyle=\mathcal{N}(z_{i}|\mu_{i},diag(\sigma_{i}^{2}))$

As a next step, the generative layer learns an adjacency matrix based on the
representation matrix $Z$ of the interference layer. The latent variable
$z_{i}$, $z_{j}$ is the inner product value of document $i$ and $j$. An
adjacency matrix is generated based on the latent variable through the inner
product between paper vectors, as shown in Equation 5. The generative model
defines the representation matrix $Z$ by reducing the difference between the
adjacency matrix $A$ generated by the generative model and the actual
adjacency matrix.

$\displaystyle p(A|Z)$
$\displaystyle=\prod\;_{i=1}^{N}\prod\;_{j=1}^{N}p(A_{ij}|z_{i},z_{j}),p(A_{ij}=1|z_{i},z_{j})$
(5) $\displaystyle=\sigma(z_{i}\top z_{j})$

##  4 Experiments

###  4.1 Experiments overview

We compare the proposed model with CACR Yang et al. (2018), one of the
existing SOTA models, with a focus on performance. We use the AAN and
FullTextPeerRead (FTPR) datasets in our experiments and use Mean Average
Precision (MAP), Mean Reciprocal Rank (MRR), and Recall@K as evaluation
metrics. The purpose of our experiments is to investigate the following
topics, including the overall performance of our model.

  * •

We compare the performance of the proposed model with the existing SOTA, the
CACR model, to gauge the outperformance of BERT and GCN over conventional
model.

  * •

We investigate differences in performance between models using BERT and GCN.
We used BERT for textual data and GCN for graph data. We analyze the impact of
each models on overall performance.

  * •

When using this model in a practical environment to recommend papers, we want
to verify that the model can recommend papers solely by looking at the
sentence on the left side of a citation symbol. In general, if researchers are
writing a paper, researchers expect to recommend reference papers based on the
contents which have been written so far without the entire document. We want
to check whether our model is available in this situation.

  * •

We check the performance of the model according to the length of textual data.
When using BERT, we check whether sentences, that are distant from the
citation symbol, are used as noise or useful information

  * •

We measure performance according to paper occurrence in the aggregate dataset.
Citation of specific papers is rare; however, we need to understand how our
model performs when it happens.

###  4.2 Experiments setting

####  4.2.1 Experimental dataset

In the experiment, the AAN dataset used data published before 2014, whereas
the FullTextPeerRead dataset comprised paper data published before 2018. After
interpreting the data, with sufficient context, and the metadata among
datasets, AAN and FullTextPeerRead yielded 6,500 and 4,898 papers,
respectively. The datasets were divided into two parts: the AAN dataset used
5,806 pre-2013 papers for the training set, and the remaining 973 others for
the test set prior to 2013, and the test set 973 for the test set. Regarding
the FullTextPeerRead dataset, 3,411 pre-2017 papers were used for the training
set, and 2,559 papers from 2017 were used for the test set. We applied two
text lengths, namely 50 and 100, to the citation context features, and
measured the related shift in performance. Furthermore, in order to test
performance according to the text input characteristics, we conducted an
experiment comparing single and pair context characteristics. For this
purpose, we used a single context consisting of a 100 text length on the left
side of a citation placeholder, and the pair context consisting of a 100 text
length on each side of the citation placeholder.

####  4.2.2 Evaluation metrics

For experimental evaluation, we use MAP, MRR, and Recall Top@K, which are
general metrics for information retrieval. The MAP measures average precision
reflecting the rank position regarding the retrieval list. This indicator is
based on the position of the corresponding label values for the K
recommendation list, and we measured the indicator with K=30. The MRR
indicator is defined as identifying the location of the first occurrence of
the actual labels in the recommendation list. Finally, Recall Top@K is defined
as an indicator of the actual label hit ratio in a Top@K recommendation list.
For this, we evaluate the Recall index through K = 5, 10, 30, 50, 80, 100

####  4.2.3 Parameter setting

We extracted the embedded context vectors and document vectors from the BERT
and GCN layers built in a separate learning process. In BERT, the number of
multi-head attention is 12, the encoder stack is 12, the total number of
epochs for learning is 30, the batch size is 16, and the Adam optimizer is
used. The learning rate is 2e-5, epsilon is 1e-6, beta1 is 0.9, beta2 is
0.999, and the weight decay rate is 0.01. We set the sequence length maximum
to 128, padding 0 if the length is shorter than 128, and the hidden size is
768.

With regard to GCN, the number of epochs is 200, the first hidden dimension is
same as the document size and the second hidden dimension is 768, the batch
size is same as the total document size(full-batch gradient descent), the
optimizer is the Adam optimizer Kingma and Ba (2014), and the learning rate is
0.01

###  4.3 Experiments results

####  4.3.1 Baseline Comparison

As shown in Table 2, our model delivers a significant performance improvement
over the existing CACR. Compared with the SOTA model, all our models showed a
three times performance improvement in MAP, MRR, and Recall@K indices,
approximately. In particular, Recall@5, which only sees five retrieval
citation papers, is a significant improvement.

In this experiment, both our model and CACR were used solely for papers with a
minimum fequency of five citations, and the learning was conducted by
considering fifty words on both sides of the citation symbol.

Dataset | Model | MAP | MRR | Recall@5 | Recall@10 | Recall@30 | Recall@50 | Recall@80
---|---|---|---|---|---|---|---|---
AAN | BERT-GCN | 0.6189 | 0.6036 | 0.6736 | 0.7109 | 0.7814 | 0.8162 | 0.8538
BERT-GCN-Left | 0.5967 | 0.5818 | 0.6459 | 0.6843 | 0.7506 | 0.785 | 0.8245
BERT | 0.6118 | 0.5971 | 0.6593 | 0.6976 | 0.7645 | 0.81 | 0.8257
BERT-Left | 0.5928 | 0.5789 | 0.6364 | 0.6678 | 0.7379 | 0.7793 | 0.8203
CACR Yang et al. (2018) | 0.2893 | 0.2917 | 0.3861 | 0.4531 | 0.5799 | 0.6573 | 0.721
FullTextPeerRead | BERT-GCN | 0.4181 | 0.4179 | 0.4864 | 0.5291 | 0.6036 | 0.6495 | 0.6994
BERT-GCN-Left | 0.3883 | 0.388 | 0.4455 | 0.4815 | 0.5539 | 0.5991 | 0.6499
BERT | 0.4152 | 0.415 | 0.4801 | 0.52 | 0.5926 | 0.6366 | 0.6887
BERT-Left | 0.3823 | 0.3821 | 0.4391 | 0.4755 | 0.5459 | 0.5885 | 0.6392
CACR Yang et al. (2018) | 0.1551 | 0.1549 | 0.2154 | 0.2761 | 0.4128 | 0.4794 | 0.5516
Table 2: MAP, MRR, and Recall@K scores for a frequency of over five citations
and fifty pair context sentences

We compared the performance by independently reproducing the Python code
related in the CACR paper. There was no detailed experimental information such
as frequency in the actual paper. Since no frequency was mentioned, we assume
that the performance described in the CACR paper is based on a frequency of
one, and we compare our performance with the performance described in the
paper of CACR as shown in Table 2. For MAP, MRR, and Recal@10, our model
outperform, but after Recall@10, it does underperform. Based on experience, we
guessed this to be a phenomenon that occurs when the classification label
value is returned with a high frequency of cited papers.

Dataset | Model | MAP | MRR | Recall@5 | Recall@10 | Recall@30 | Recall@50 | Recall@80
---|---|---|---|---|---|---|---|---
AAN | BERT-GCN | 0.4516 | 0.4323 | 0.4948 | 0.532 | 0.5839 | 0.6118 | 0.6335
CACR Yang et al. (2018) | 0.301 | 0.335 | - | 0.474 | 0.615 | 0.653 | 0.712
Table 3: MAP, MRR, Recall@K scores with a frequency of over one citation, 200
pair context sentences and CACR metric result

####  4.3.2 Impact of BERT and GCN

In all cases, when the GCN was added, the performance of the model improved.
Of interest is the impact that GCN has on the performance of BERT, and BERT-
left alone. As shown in Figure 4, the difference in recall performance between
BERT + GCN-Left and BERT-left differs from the difference in performance
difference between BERT + GCN and BERT. BERT-left has half the input of
context sentence of BERT. Hence, the impact of GCN is greater when the value
of the input words is small.

Figure 4: The effects of BERT, GCN and BERT-left

####  4.3.3 Impact of the left side context alone

From the viewpoint of an actual researcher writing a paper, we check the
performance when using the context sentence on the left side of the citation
symbol alone. As shown in Table2, the model using the left context alone has a
performance that is approximately 0.03 lower than the model using the entire
context for the context encoder. In other words, it is helpful to consider the
entire context on both sides of the quotation mark. However, adding GCN to the
BERT model considering the left context alone delivers performance
improvements over adding GCN to the BERT model that considers both contexts,
as the network information of the GCN can be helpful when textual information
is not sufficient.

Citation Context | Approaches |  Top-3 System
---|---|---
… inbold means significant ly better than the baseline according to [?] or p-value less than 0.01.baseline SMT system. The decoding … | BERT-GCN |  1\. Statistical Significance Tests For Machine Translation Evaluation (O)
2\. Bleu: A Method For Automatic Evaluation Of Machine Translation (X)
3\. On Coreference Resolution Performance Metrics (X)
BERT |  1\. Minimum Error Rate Training In Statistical Machine Translation (X)
2\. Statistical Phrase-Based Translation (X)
3\. Statistical Significance Tests For Machine Translation Evaluation (O)
CACR |  1\. TnT - A Statistical Part-Of-Speech Tagger (X)
2\. Stochastic Inversion Transduction Grammars And Bilingual Parsing Of
Parallel Corpora (X)
3\. A Maximum Entropy Model For Part-Of-Speech Tagging (X)
Many researchers have atte mpted to make use of cue p hrases especially for segme ntation both in prose [?] | BERT-GCN-Left |  1\. TextTiling: Segmenting Text Into Multi-Paragraph Subtopic Passages (O)
2\. Multi-Paragraph Segmentation Of Expository Text (X)
3\. Discourse Segmentation Of Multi-Party Conversation (X)
BERT-Left |  1\. Multi-Paragraph Segmentation Of Expository Text (X)
2\. Advances In Domain Independent Linear Text Segmentation (X)
3\. TextTiling: Segmenting Text Into Multi-Paragraph Subtopic Passages (O)

Table 4: A comparison of ground truth with the top five recommended citation
lists

####  4.3.4 Effect of length of the context sequence

We have already confirmed a shift in performance tied to the length of the
input context sentences. However, how much more can performance improve as the
number of sentences increases? As shown in Figure 5, when the context length
becomes a hundred or more, model performance becomes less impacted by
comparative context length. The context sentence length is relevant, but
beyond a certain point, its impact is actually reduced.

Figure 5: The shift in performance according to the length of sentence context
when frequency is one

####  4.3.5 Effect of paper citation frequency

Finally, we examine the impact of citation frequency on performance. The
results of the experiment with citation frequencies one, three, and five show
that performance improves when citation frequency is higher, as shown in Table
5. In general, papers not cited are not used for learning and can be processed
as sparse data even at the time of testing. We believe that the learning data
should be refined according to citation frequency for a well-functioning,
high-performance model.

Model | Frequency 1 | Frequency 3 | Frequency 5
---|---|---|---
BERT GCN | 0.4467 | 0.6063 | 0.6736
BERT | 0.4423 | 0.5971 | 0.6593
Table 5: A comparison of shifts in performance based on cited paper frequency

###  4.4 Recommendation Examples

Actual recommendation examples using our model can be found in Table 4. In
these examples, we observe that one of the biggest effects of GCN is that it
is possible to get a higher relevance similarity of cited paper through GCN
for similar articles when we look at textual data alone. Expanding the use of
GCN, would not only target similarities in the content of the papers but also
citation information in the previous thesis or citation information in the
current thesis. With regard to the use of GCN, the representation learning
information concerning citation is generated by identifying not only a
similarity in the content of the citing paper but also through the citation
information of the previous paper by the author or the information in the
cited paper of the current article. We think this information is very helpful
to improve the performance of the context-aware citation recommendation
process.

##  5 Conclusion

The proposed model for context-aware citation recommendation task delivers a
significant improvement in MAP, MRR and Recall@K over the existing model. The
basis for the breakthrough performance improvement is that the BERT model,
which has performed well in recent NLP tasks, is adapted to our context-aware
framework. Through the context encoding via BERT, our framework improves the
representation learning of the context side. In addition, we apply VGAE, which
comprises a GCN layer according to graph data to mitigate over-fitting to
local contexts when BERT is applied alone. The VGAE applied to our framework
citation encoder processes the paper citation network graph data into a paper
latent representation. The combination of the encoded paper network and the
encoded context is regularized, resulting in a performance increase over a
BERT-based model.

Regarding the context-aware citation recommendation study, existing datasets
are not up-to-data and there is no clear context detection. To address this
problem, we devised and released the FullTextPeerRead dataset. The proposed
dataset comprises updated papers to an extent including papers up-to 2017, and
provides a method to readily and accurately extract context metadata and has a
well-organized perspective.

## References

  * Devlin et al. [2018] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  arXiv e-prints, October 2018.
  * Dragomir R. Radev [2009] Bryan Gibson Pradeep Muthukrishnan Dragomir R. Radev, Mark Thomas Joseph.  A Bibliometric and Network Analysis of the field of Computational Linguistics.  Journal of the American Society for Information Science and Technology, 2009.
  * Ebesu and Fang [2017] Travis Ebesu and Yi Fang.  Neural citation network for context-aware citation recommendation.  In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’17, pages 1093–1096, New York, NY, USA, 2017. ACM.
  * Fontoura et al. [2013] Marcus Fontoura, H. Asthana, Pradeep B. Teregowda, Eric Treece, Madian Khabsa, Douglas Jordan, Stephen Carman, Prasenjit Mitra, C. Lee Giles, Bibek Paudel, Avishek Anand, Klaus Berberich, Stéphane Marchand-Maillet, Fidel Cacheda, Vreixo Formoso, and Victor Carneiro.  A supervised learning method for context-aware citation recommendation in a large corpus.  2013\.
  * He et al. [2010] Qi He, Jian Pei, Daniel Kifer, Prasenjit Mitra, and Lee Giles.  Context-aware citation recommendation.  In Proceedings of the 19th international conference on World wide web, pages 421–430. ACM, 2010.
  * He et al. [2011] Qi He, Daniel Kifer, Jian Pei, Prasenjit Mitra, and C Lee Giles.  Citation recommendation without author supervision.  In Proceedings of the fourth ACM international conference on Web search and data mining, pages 755–764. ACM, 2011.
  * Huang et al. [2015] Wenyi Huang, Zhaohui Wu, Chen Liang, Prasenjit Mitra, and C Lee Giles.  A neural probabilistic model for context based citation recommendation.  In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, pages 2404–2410. AAAI Press, 2015.
  * Kang et al. [2018] Dongyeop Kang, Waleed Ammar, Bhavana Dalvi, Madeleine van Zuylen, Sebastian Kohlmeier, Eduard Hovy, and Roy Schwartz.  A dataset of peer reviews (peerread): Collection, insights and nlp applications.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 1647–1661, 2018.
  * Kingma and Ba [2014] D. P. Kingma and J. Ba.  Adam: A Method for Stochastic Optimization.  arXiv e-prints, December 2014.
  * Kipf and Welling [2016a] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  arXiv preprint arXiv:1609.02907, 2016.
  * Kipf and Welling [2016b] Thomas N Kipf and Max Welling.  Variational graph auto-encoders.  NIPS Workshop on Bayesian Deep Learning, 2016.
  * Le and Mikolov [2014] Quoc Le and Tomas Mikolov.  Distributed representations of sentences and documents.  arXiv preprint arXiv:1405.4053, 2014.
  * Liu et al. [2015] H. Liu, X. Kong, X. Bai, W. Wang, T. M. Bekele, and F. Xia.  Context-based collaborative filtering for citation recommendation.  IEEE Access, 3:1695–1703, 2015.
  * Radev et al. [2009] Dragomir R. Radev, Pradeep Muthukrishnan, and Vahed Qazvinian.  The ACL anthology network corpus.  In Proceedings, ACL Workshop on Natural Language Processing and Information Retrieval for Digital Libraries, Singapore, 2009.
  * Radev et al. [2013] DragomirR. Radev, Pradeep Muthukrishnan, Vahed Qazvinian, and Amjad Abu-Jbara.  The acl anthology network corpus.  Language Resources and Evaluation, pages 1–26, 2013.
  * Rezende et al. [2014] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra.  Stochastic backpropagation and approximate inference in deep generative models.  In Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32, ICML’14, pages II–1278–II–1286. JMLR.org, 2014.
  * Tang et al. [2014] Xuewei Tang, Xiaojun Wan, and Xun Zhang.  Cross-language context-aware citation recommendation in scientific articles.  In Proceedings of the 37th International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR ’14, pages 817–826, New York, NY, USA, 2014. ACM.
  * Yang et al. [2018] L. Yang, Y. Zheng, X. Cai, H. Dai, D. Mu, L. Guo, and T. Dai.  A lstm based model for personalized context-aware citation recommendation.  IEEE Access, 6:59618–59627, 2018.