# PubGraph: A Large Scale Scientific Temporal Knowledge Graph

Kian Ahrabian, Xinwei Du, Richard Delwin Myloth, Arun Baalaaji Sankar
Ananthan, Jay Pujara

###### Abstract

Research publications are the primary vehicle for sharing scientific progress
in the form of new discoveries, methods, techniques, and insights.
Publications have been studied from the perspectives of both content analysis
and bibliometric structure, but a barrier to more comprehensive studies of
scientific research is a lack of publicly accessible large-scale data and
resources. In this paper, we present PubGraph, a new resource for studying
scientific progress that takes the form of a large-scale temporal knowledge
graph (KG). It contains more than 432M nodes and 15.49B edges mapped to the
popular Wikidata ontology. We extract three KGs with varying sizes from
PubGraph to allow experimentation at different scales. Using these KGs, we
introduce a new link prediction benchmark for transductive and inductive
settings with temporally-aligned training, validation, and testing partitions.
Moreover, we develop two new inductive learning methods better suited to
PubGraph, operating on unseen nodes without explicit features, scaling to
large KGs, and outperforming existing models. Our results demonstrate that
structural features of past citations are sufficient to produce high-quality
predictions about new publications. We also identify new challenges for KG
models, including an adversarial community-based link prediction setting,
zero-shot inductive learning, and large-scale learning.

##  1 Introduction

Scientific progress takes many forms, ranging from discovering new species to
repurposing extant models for novel tasks. Innovation in science has been
studied from a variety of perspectives, including the combination of scholarly
domains (Hofstra et al. 2020; Uzzi et al. 2013), sociological factors (De
Vaan, Stark, and Vedres 2015), and analogical reasoning (Hope et al. 2017;
Kang et al. 2022). However, many studies of this phenomenon have been limited
due to the difficulty in finding and using large-scale data for the domain. In
this paper, we address this obstacle by introducing PubGraph, a knowledge
graph (KG) with new resources and benchmarks, enabling the study of scientific
research at scale using structural patterns in citation and collaboration
networks. PubGraph provides a unique opportunity to compare multi-relational
KG models in both transductive and inductive settings at a billion-edge scale.

PubGraph is a large-scale multi-relational temporal KG built using the
OpenAlex catalog (Priem, Piwowar, and Orr 2022) and the Wikidata (Vrandečić
and Krötzsch 2014) ontology. It consists of more than 432M entities, including
authors, institutions, publication venues, and papers, and more than 15.49B
relationships for those entities. PubGraph also captures temporal information,
allowing the study of scientific works’ dynamics. In this paper, we
demonstrate that citation structure in PubGraph can be used, in concert with
KG link prediction models, to predict future citation links between research
papers with high accuracy.

Citations have been crucial in studying publications and their impact (Price
1965). Prior works have studied tasks on citations such as intent
classification (Jurgens et al. 2018; Cohan et al. 2019; Gururangan et al.
2020), recommendation (Bhagavatula et al. 2018; Färber and Sampath 2020), and
prediction (Cohan et al. 2020; Ostendorff et al. 2022). Many such studies use
the publication’s content, but content information may be difficult to obtain
and parse. Alternatively, citation structure has been shown (Liu et al. 2019;
Dietz, Bickel, and Scheffer 2007) to provide a strong signal for predictive
tasks. New large-scale graphs (Hu et al. 2021) for tasks such as link
prediction and node classification support using citation structure but are
limited in their relational information. Our work builds on these prior works
with new resources and benchmarks that are appropriate for a variety of tasks,
emphasizes the multi-relational structure via a knowledge graph, and includes
billions of edges.

Figure 1: Overview of the training and evaluation scheme. Intra-period current
links (black) are used for training in all experiment settings. Intra-period
future links (red) are used for evaluation in both validation and testing
phases in all experiment settings. Exo-period links (dotted blue) are used in
the training phase in transductive settings; however, in inductive settings,
these links are only used as auxiliary links during the evaluation phase.
Auxiliary links establish connections between seen training nodes and unseen
evaluation nodes.

One of the essential requirements for understanding scientific progress is to
predict meaningful connections between ideas and discoveries. The main
challenge of satisfying this requirement is to deal with new publications
requiring inference over unseen samples, i.e., inductive learning. In this
work, we explore this problem from a purely structural standpoint, framing the
research question as a link prediction task for KGs. The goal is to predict a
citation relationship between two research papers using information about
authors, institutions, venues, time, and other citations. This approach
provides insight into the utility of the relational and structural
collaborative information available among scholarly entities, which is
complementary to individual content-based features.

While KGs capture the structure necessary for this predictive task, many
models do not address this use case. Specifically, traditional KG-based
evaluations (Trouillon et al. 2016; Sun et al. 2019) are in an interpolated,
transductive setting where all entities (e.g., papers and authors) are known,
and a portion of their edges are missing. A better evaluation setting for
studying innovation is the extrapolated, inductive setting. Extrapolated
prediction requires train and test sets to be partitioned by a temporal
threshold, so model predictions are for a future time epoch. Inductive models
are capable of making predictions for previously unseen papers and authors. We
create benchmarks for KG models in the extrapolated setting with transductive
and inductive variations to support evaluating models that predict future
relationships between new publications. As part of our benchmarks, we
introduce an efficient, inductive, and featureless hybrid framework for large-
scale KGs, outperforming existing models.

The contributions of this work are summarized as follows:

  1. 1.

Creating of PubGraph, a billion-scale, multi-relational temporal KG using the
Wikidata ontology

  2. 2.

Introducing a large-scale extrapolated link prediction benchmarks for KG
models in both transductive and inductive settings

  3. 3.

Developing novel extensions for inductive graph models that improve
performance and scalability

  4. 4.

Identifying adversarial benchmark variants posing challenging new environments
for KG research

  5. 5.

Analyzing state-of-the-art static and temporal KG models for multi-relational
citation link prediction

##  2 Related Works

Knowledge Graphs: The research interest in KGs has increased in recent years.
These structured repositories store information in the form of triplets
representing a relation between two entities. One of the most prominent tasks
on KGs is link prediction which aims to complete a given inherently incomplete
KG (Bordes et al. 2013). However, they have also been applied to various
disciplines such as recommender systems (Wang et al. 2019d, b, a), question
answering (Zhang et al. 2018a; Narasimhan, Lazebnik, and Schwing 2018; Zhang
et al. 2018b), and text-based games (Adhikari et al. 2020; Ammanabrolu and
Hausknecht 2020) to name a few.

Knowledge Graph Embeddings: These embedding models aim to learn
representations for entities and relations in a given KG. Most of these models
follow the encoder-decoder framework for learning representations. The encoder
generates representations for entities and the relation, and the decoder
produces a score given the generated representations. Traditional models
(Bordes et al. 2013; Trouillon et al. 2016; Sun et al. 2019) use simple
shallow embeddings as the encoder and a scoring function as the decoder.

More recent models (Schlichtkrull et al. 2018; Dettmers et al. 2018) replace
shallow embeddings with Graph Neural Networks (GNN) as their encoders. These
complex encoders enable the models to gather information over the multi-hop
neighborhoods around the entities, increasing the reasoning power of the
models. However, the main downside of GNNs is their demanding computational
cost. This downside especially becomes critical for representation learning at
scale. As a result, almost all the past link prediction studies on large-scale
KGs rely on traditional models (Hu et al. 2021).

Large Scale Datasets: Closely related to our work is the MAG240M KG (Hu et al.
2021). However, there are three significant differences between these
datasets. First, we rely on OpenAlex instead of the Microsoft Academic Graph
(MAG) (Wang et al. 2020) to extract our KG. This choice is mainly due to the
discontinuation of MAG. Second, we add publication-venue relations as we
believe they potentially introduce a robust bias to the KG throughout the
learning process. Finally, in contrast to MAG240M, we only rely on the
structural information rather than the contents of the publications. It is
interesting to note that the nature of affiliations is dynamic and changes
through time, making it impossible to capture in a static structure.
Consequently, the imposed bias through author-institution relations could
potentially lead to degraded performance in static settings.

##  3 Building PubGraph

To create PubGraph, we rely on the OpenAlex catalog (Priem, Piwowar, and Orr
2022). OpenAlex is an open-source catalog of scholarly entities that provides
metadata for publications, authors, venues, concepts, and institutions. This
work focuses on the connections between publications, authors, venues, and
institutions. However, concepts and their links to publications are also used
in our analysis as supplementary information. Throughout both the
transformation and extraction steps, we use the KGTK toolkit (Ilievski et al.
2020) to run our queries. We release PubGraph under a CC-BY-SA license at TBD.

Dataset | #Nodes | #Edges | #Relations
---|---|---|---
ogbl-citation2 (Hu et al. 2020) | 2,927,963 | 30,561,187 | 1
Freebase (Bollacker et al. 2008) | 86,054,151 | 338,586,276 | 14,824
WikiKG90Mv2 (Hu et al. 2021) | 91,230,610 | 601,062,811 | 1,315
PubGraph | 432,172,002 | 15,499,528,916 | 51
PG-1M | 3,378,202 | 22,442,976 | 4
PG-10M | 25,312,490 | 315,225,337 | 4
PG-Full | 184,126,885 | 2,201,239,147 | 4

Table 1: Statistics of extracted datasets compared to the existing large-scale
datasets. The PG-Full dataset has more than 2x nodes and 3.6x edges compared
to the largest existing dataset.

###  3.1 Mapping to Wikidata

To transform the OpenAlex dump (taken on May 15th, 2022) into PubGraph, we
follow the well-known and well-studied Wikidata ontology (Vrandečić and
Krötzsch 2014). Specifically, we create a mapping between metadata information
from the OpenAlex dump to Wikidata properties. Using Wikidata enables broader
adoption of the KG and clear semantics for entities and relationships. We
further discuss the mapping process in Appendix E.

###  3.2 Sampling Methodology

Figure 2: Overview of the sampled PG-Full KG.

The full PubGraph KG contains a vast amount of information in the form of
literal values and sparse properties that are not easily usable by many KG
models. We extract subsets of PubGraph, designated as PG, to create easier-to-
use datasets for KG models. To extract PGs from the transformed data, we first
remove all the publications that have no citations and do not cite any other
papers to get PG-Full. Since these nodes are disconnected from other
publications, this step mitigates the sparsity problem and reduces the KG size
by a large margin.

Given the enormous size of the PG-Full, we create two small and medium-sized
sub-KGs to allow future studies at different scales. To this end, we use
snowball sampling (Goodman 1961) to extract PG-1M and PG-10M with 1M and 10M
publication nodes, respectively. After sampling, we remove any publication
without a publication date. Next, we extract all the publication-publication
(P2860), publication-author (P50), publication-venue (P1433), and author-
institution (P1416) links from the sampled data. We ensure to include all the
available author, venue, and institution links from the sampled publications
in the datasets. Finally, we split all the datasets temporally, using all the
publications before 2017 for training, 2017 up until 2020 for validation, and
2020 onward for testing. The details of the splits are provided in Appendix A.
Figure 2 illustrates an overview of the sampled KG.

To create the temporal datasets, we timestamp the links based on the
publication date of their respective publication. For example, the timestamp
of a citation link is the publication date of the citing paper. For each
author-institution link, we use the article’s publication date, which
indicates the author’s affiliation as the timestamp. Table 1 showcases the
statistics of the PubGraph and the sampled KGs along with a comparison to
existing large-scale datasets in the literature.

###  3.3 Data Quality

Metric | PG-1M | PG-10M | PG-Full
---|---|---|---
Mutual Citations (%) | 0.03 | 0.04 | 0.06
Authorship Completeness (%) | 99.97 | 99.97 | 99.92
Venue Completeness (%) | 92.37 | 90.25 | 75.34
Institution Completeness (%) | 81.45 | 71.21 | 45.77

Table 2: Validity and completeness metrics of sampled KGs.

To evaluate the quality of the extracted datasets, we check the validity and
completeness of our KGs. For validity, we look for potential mutual citations,
cases where two papers reference each other, violating strict temporal order.
This artifact may appear when articles have several revisions, but OpenAlex
only reports the earliest publication date. For completeness, we calculate
publication-author, publication-venue, and author-institution relations
completeness. Table 2 showcases these metrics on the extracted KGs. As evident
from the metrics, all the datasets exhibit an extremely low mutual citations
percentage which is evidence of their quality. Moreover, the small and medium-
sized KGs exhibit higher completeness metrics which we attribute to the forced
inclusion of all authors, venues, and institutions links.

##  4 Methodology

We design our experimental setup to accommodate different levels of
information availability. In the first setting, we study our research question
through the common information-rich graph completion environment, i.e., static
transductive, where each entity is seen many times in the training phase. We
then add additional temporal information to study the effects of dynamic
behaviors in the graph, i.e., temporal transductive. Finally, we investigate
our problem in an information-deprived environment, i.e., inductive, to study
adaptability and generalizability to unseen nodes.

###  4.1 Static Transductive

The static transductive setting applies when all entities are included in the
training set, and edges in the test set are missing at random. For this
setting, we choose ComplEx and RotatE as our experimental models. This choice
is mainly influenced by their superior expressive power (Sun et al. 2019) and
availability in the DGL-KE toolkit (Zheng et al. 2020), which we use
throughout these experiments.

Given a triplet $(s,r,o)$, ComplEx and RotatE produce scores as follows:

$\displaystyle f_{\text{ComplEx}}$ $\displaystyle=\text{Re}(\langle
E_{e}(s),E_{r}(r),\overline{E_{e}(o)}\rangle)$ (1) $\displaystyle
f_{\text{RotatE}}$ $\displaystyle=-\parallel E_{e}(s)\circ
E_{r}(r)-E_{e}(o)\parallel^{2}$ (2)

where $\langle.\rangle$ is the generalized dot product, $\circ$ is the
Hadamard product, $E_{e}$ is the entity embedding, and $E_{r}$ is the relation
embedding.

During the training phase, we use the self-adversarial negative sampling and
the log-sigmoid loss:

$\displaystyle p(s^{\prime},r,o^{\prime})$ $\displaystyle=\frac{\exp\alpha
f(s^{\prime},r,o^{\prime})}{\sum_{s^{\prime\prime},o^{\prime\prime}}\exp\alpha
f(s^{\prime\prime},r,o^{\prime\prime})}$ (3) $\displaystyle\mathcal{L}$
$\displaystyle=-\log\sigma(\gamma-f(s,r,o))$ (4)
$\displaystyle-\sum_{s^{\prime},o^{\prime}}p(s^{\prime},r,o^{\prime})\log\sigma(f(s^{\prime},r,o^{\prime})-\gamma)$

###  4.2 Temporal Transductive

The temporal transductive setting applies when all entities are present during
training, temporal information is included for edges, and missing edges for
evaluation only occur after a temporal threshold. We use diachronic
embeddings, which are temporal extensions of shallow embeddings. Diachronic
embeddings consist of a static part, i.e., shallow embedding, and a dynamic
part, i.e., high-dimensional sinusoid, which allows it to produce time-
conditioned embeddings. Consequently, they support extrapolation evaluation
settings in which we are required to have embeddings for unseen timestamps.
Given an entity $e$ and timestamp $t$, the diachronic embedding is calculated
as

$D_{e}(e,t)=E_{e}(e)\oplus[E_{A}(e)\sin(E_{F}(e)t+E_{\phi}(e))]$ (5)

where $\oplus$ is the concatenation operator, $E_{e}$, $E_{A}$, $E_{F}$, and
$E_{\phi}$ are all shallow embeddings and $E_{A}$, $E_{F}$, and $E_{\phi}$
respectively model the amplitude, frequency, and phase of the high-dimensional
sinusoid. Moreover, the dimension of the dynamic part is controlled through
the hyper-parameter $\psi$ in proportion to the embedding dimension.

For our experiments, we extend the existing TransE (DE-TransE) and DistMult
(DE-DistMult) implementation of DGL-KE to support diachronic embeddings. Given
a quadruple $(s,r,o,t)$, DE-TransE and DE-DistMult produce a score as follows:

$\displaystyle f_{\text{DE-TransE}}$ $\displaystyle=-\parallel
D_{e}(s,t)+E_{r}(r)-D_{e}(o,t)\parallel$ (6) $\displaystyle f_{\text{DE-
DistMult}}$ $\displaystyle=\langle D_{e}(s,t),E_{r}(r),D_{e}(o,t)\rangle$ (7)

Moreover, we apply dropout to the produced scores to regularize the model.

During the training phase, following the training environment described in the
original paper, we minimize the cross-entropy loss:

$\displaystyle\mathcal{L}=$
$\displaystyle-(\sum_{(s,r,o,t)\in\mathcal{G}}\frac{\exp
f(s,r,o,t)}{\sum_{o^{\prime}}\exp f(s,r,o^{\prime},t)}$ (8)
$\displaystyle+\sum_{(s,r,o,t)\in\mathcal{G}}\frac{\exp
f(s,r,o,t)}{\sum_{s^{\prime}}\exp f(s^{\prime},r,o,t)})$

###  4.3 Inductive

One of the most important aspects of a learning algorithm is its ability to
generalize to unseen samples. This aspect of machine learning on graphs is
addressed through the inductive setting. The inductive setting applies when
both entities and edges are missing after a temporal threshold. In general,
inductive learning algorithms rely on node features and multi-hop node
neighborhoods to generate embeddings for unseen nodes. Moreover, these
algorithms use the neural message-passing algorithm to aggregate information
from multi-hop node neighborhoods.

Given a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$ with $h^{0}_{u}=x_{u}$
(node features) and $h^{0}_{(u,v)}=y_{(u,v)}$ (edge features), the generalized
message-passing scheme at the $k$-th layer is defined as (Battaglia et al.
2018):

$\displaystyle h^{k}_{(u,v)}$
$\displaystyle=f_{\textit{edge}}(h^{k-1}_{(u,v)},h^{k-1}_{u},h^{k-1}_{v},h^{k-1}_{\mathcal{G}})$
(9) $\displaystyle m_{\mathcal{N}_{u}}$
$\displaystyle=f_{\textit{agg}}(\\{h^{k}_{(u,v)}\\}_{\forall
v\in\mathcal{N}_{u}})$ (10) $\displaystyle h^{k}_{u}$
$\displaystyle=f_{\textit{node}}(h^{k-1}_{u},m_{\mathcal{N}_{u}},h^{k-1}_{\mathcal{G}})$
(11) $\displaystyle h^{k}_{\mathcal{G}}$
$\displaystyle=f_{\textit{graph}}(h^{k-1}_{\mathcal{G}},\\{h^{k}_{u}\\}_{\forall
u\in\mathcal{V}},\\{h^{k}_{(u,v)}\\}_{\forall(u,v)\in\mathcal{E}})$ (12)

where $\mathcal{N}_{u}$ is the neighborhood around node $u$, $h^{k}_{(u,v)}$
is the edge representation, $h^{k}_{u}$ is the node representation, and
$h^{k}_{\mathcal{G}}$ is the graph representation.

Figure 3: Overview of the hybrid training and evaluation framework used for
GraphSAGE-H and RGCN-H.

One limitation of this formulation is the need for features for unseen nodes,
which are not part of PubGraph and may generally be unavailable. To address
these limitations, we propose two extensions to existing inductive methods to
allow them to work without node and edge features by leveraging network
structure. Our alternative generalized message-passing scheme, replaces
Equation 11 with

$\displaystyle h^{k}_{u}$
$\displaystyle=f_{\textit{node}}(m_{\mathcal{N}_{u}},h^{k-1}_{\mathcal{G}})\
.$ (13)

This approach is equivalent to removing self-loops during the aggregation
step. The main idea for this proposal is to build the representation of each
node solely based on its multi-hop neighborhood. This simple change allows us
to quickly generate representations for unseen nodes only using their links to
training nodes, i.e., auxiliary links. In this scheme, the model is forced to
distribute the presentation of each node among its neighbors. Hence, each
node’s embedding holds a soft aggregation of its neighborhood, potentially
leading to more robust learned representations. Conceptually, these
distributed representations have similarities to the NodePiece’s anchor nodes
(Galkin et al. 2021) as we could use them both to reconstruct the
representation of neighboring nodes. To study the effectiveness of this
approach, we transform two popular inductive learning frameworks using the
proposed aggregation scheme. We train all models by minimizing the cross-
entropy loss (Equation 8).

#### GraphSAGE-E

GraphSAGE’s original node representation function is defined as

$h^{k}_{u}=\sigma(W^{k}[h^{k-1}_{u}\oplus m_{\mathcal{N}_{u}}])$ (14)

where $\sigma$ is a non-linear function. Following the above-mentioned
aggregation scheme, we change this function to

$h^{k}_{u}=\sigma(W^{k}m_{\mathcal{N}_{u}})\ .$ (15)

#### RGCN-E

Similar to GraphSAGE, we modify RGCN’s node representation function from

$h^{k}_{u}=\sigma(\sum_{r\in\mathcal{R}}\sum_{v\in\mathcal{N}_{u}}\frac{1}{c_{u,r}}W^{k-1}_{r}h^{k-1}_{v}+W^{k-1}_{0}h^{k-1}_{u})$
(16)

where $\mathcal{R}$ is the relations set and $c_{u,r}=|\mathcal{N}^{r}_{u}|$,
to

$h^{k}_{u}=\sigma(\sum_{r\in\mathcal{R}}\sum_{v\in\mathcal{N}_{u}}\frac{1}{c_{u,r}}W^{k-1}_{r}h^{k-1}_{v})\
.$ (17)

For both GraphSAGE-E and RGCN-E, given two publications $(u,v)$ and a
$K$-layer model, we employ a dot product function to produce a score as
follows:

$f=\langle h^{K}_{u},h^{K}_{v}\rangle\ .$ (18)

Although our training setup for GraphSAGE-E and RGCN-E is a special case of
the oDistMult-ERAvg algorithm (Albooyeh, Goel, and Kazemi 2020), we employ a
more general and complex neighborhood encoder to increase the reasoning power
of our model. In contrast to the oDistMult and NodePiece original evaluation
setup, we evaluate the unseen-unseen (Xu et al. 2020) rather than the unseen-
seen links.

#### GraphSAGE-H and RGCN-H

Apart from the end-to-end models, we introduce a hybrid two-stage training
scheme. In this scheme, we first pre-train our model using static transductive
models to obtain node embeddings for training KG. Then, we freeze these
embeddings and use them as feature vectors to train our message-passing model
using the above-mentioned aggregation method. This combination allows us to
have the benefits of both shallow models, i.e., speed, and message-passing-
based models, i.e., more reasoning power. Figure 3 illustrates an overview of
the proposed hybrid scheme. Given the multiplicative nature of our scoring
function as defined in Equation 18, we use the embeddings from the best
ComplEx model to avoid any mismatches with the learned representation space.

#### GraphSAGE-D and RGCN-D

We compare our models with two simple baselines, which use relational node
degrees as feature vectors (Hamilton, Ying, and Leskovec 2017; Albooyeh, Goel,
and Kazemi 2020). In these models, for each node, a feature vector of size
$2|\mathcal{R}|$ is extracted where the $i$-th and $(|\mathcal{R}|+i)$-th
elements are, respectively, the incoming and outgoing node degrees with
relation type $r_{i}\in\mathcal{R}$.

#### ComplEx

As our final model, we include the results of the ComplEx model. Since ComplEx
is an inherently transductive model, for unseen nodes we assign randomly
initialized embeddings taken from $\mathcal{U}(-\mu,\mu)$ where
$\mu=(2+\gamma)/d$ (Sun et al. 2019) with $d$ being the embedding dimension.
This experiment acts as both a sanity check and a pseudo-random baseline for
the rest of our experiments.

##  5 Evaluation

In this section, we evaluate state-of-the-art KG link prediction models across
static transductive, temporal transductive, and temporal inductive settings.
Figure 1 illustrates our training and evaluation settings. We further evaluate
the impact of different feature classes with an ablation study and introduce a
more challenging adversarial negative sampling setting.

In our evaluations, we compare each positive sample with 1000 negative samples
to calculate filtered Mean Reciprocal Rank (MRR), Hits@10, and Hits@50 on the
tail entities. Apart from the static transductive setting, we report results
on the PG-1M dataset due to computing resource limitations. Finally, we limit
the training time on PG-1M, PG-10M, and PG-Full to 12 hours, two days, and
five days respectively. All the experiments are conducted using PyTorch
(Paszke et al. 2019) and DGL (Wang et al. 2019c) implementations on machines
with at least 64 CPU cores, 750GB of RAM, and either 2 Nvidia V100 or 4 Nvidia
RTX 2080 Ti GPUs. Appendix B provides the details on the hyperparameter tuning
procedure and the hyperparameters of the best-performing models.
Implementations are available under a CC-BY-SA license at TBD.

###  5.1 Transductive

We compare the static transductive models ComplEx (Trouillon et al. 2016) and
RotatE (Sun et al. 2019) with temporal models that use diachronic
representations (Goel et al. 2020), specifically DE-TransE and DE-DistMult.

Dataset | Model | MRR | Hits@1 | Hits@10 | Hits@50
---|---|---|---|---|---
PG-1M | ComplEx | 0.705 | 0.579 | 0.916 | 0.969
RotatE | 0.725 | 0.609 | 0.920 | 0.979
DE-TransE | 0.611 | 0.466 | 0.875 | 0.961
DE-DistMult | 0.589 | 0.456 | 0.836 | 0.947

Table 3: Link prediction results for static and temporal transductive settings
on the PG-1M dataset.

Table 3 presents the results of our experiments with static and temporal
transductive models on the PG-1M dataset. These results show that RotatE
performs best among all models, which we attribute to its superior
expressivity. However, counterintuitively, the temporal models fall far behind
the static models. We believe this is due to their inability to generalize to
unseen timestamps, which makes them unfit for this setting. Table 4 presents
the results of our experiments with static transductive models on PG-10M and
PG-Full datasets. RotatE and ComplEx record the best performance on PG-1M and
PG-Full, respectively. This performance boost for ComplEx is mainly due to its
lower computational cost, which results in more training steps in the
previously set training time, i.e., 120M vs. 60M in 5 days. As for the
improved performance compared to the PG-1M dataset, we believe this is due to
1) more training time for the model and 2) having a less incomplete set of
references for all the nodes, which leads to better-learned representations.

Dataset | Model | MRR | Hits@1 | Hits@10 | Hits@50
---|---|---|---|---|---
PG-10M | ComplEx | 0.677 | 0.562 | 0.872 | 0.945
RotatE | 0.750 | 0.656 | 0.910 | 0.974
PG-Full | ComplEx | 0.821 | 0.760 | 0.919 | 0.957
RotatE | 0.801 | 0.731 | 0.921 | 0.973

Table 4: Link prediction results for the static transductive setting on PG-10M
and PG-Full datasets.

###  5.2 Inductive

We evaluate the inductive setting using GraphSAGE (Hamilton, Ying, and
Leskovec 2017) and RGCN (Schlichtkrull et al. 2018). Additionally, we adapt
these models to an embedding-based inductive setting rather than a feature-
based one. We attempted experiments with additional state-of-the-art inductive
models oDistMult (Albooyeh, Goel, and Kazemi 2020) and NodePiece (Galkin et
al. 2021) using the original implementations provided by the authors.
Unfortunately, oDistMult required more than 750GB RAM, and the Nodepiece OOS-
LP setting required over 1000 hours for tokenization processing for our PG-1M
dataset.

Table 5 presents the results of our experiments in the inductive setting. The
results show that the hybrid variations outperform all of their counterparts.
Moreover, GraphSAGE-E and RGCN-E outperform their relational-degree-based
counterparts, making them straightforward substitutes for end-to-end
embedding-based inductive learning settings. These findings are exciting as
they provide scalable alternatives to the current state-of-the-art models.
Surprisingly, both end-to-end and hybrid GraphSAGE variations outperform their
RGCN counterparts. A potential explanatory factor may be the low number of
relation types (4) present in the PG-1M sample.

Model | MRR | Hits@1 | Hits@10 | Hits@50
---|---|---|---|---
ComplEx | 0.007 | 0.000 | 0.008 | 0.071
GraphSAGE-D | 0.149 | 0.062 | 0.324 | 0.664
RGCN-D | 0.166 | 0.071 | 0.364 | 0.709
GraphSAGE-E | 0.779 | 0.697 | 0.917 | 0.968
RGCN-E | 0.706 | 0.609 | 0.872 | 0.939
GraphSAGE-H | 0.784 | 0.697 | 0.928 | 0.973
RGCN-H | 0.746 | 0.645 | 0.919 | 0.977

Table 5: Link prediction results for the inductive setting.

###  5.3 Ablation Study

To study the influence of each node type on the model’s performance, we run an
ablation study on the PG-1M dataset. To this end, we first create all the
valid combinations of the node types. The criteria for a combination to be
valid are 1) publications must be included in the training as our task is to
predict links between publications and 2) institution nodes cannot be present
without author nodes; otherwise, because of the network topology (Figure 1)
institution nodes will be disjoint. Then, we benchmark our best-performing
static transductive model, i.e., RotatE, on these alternative datasets.

##### Discussion

Table 6 provides the results of our experiments. As evident from the results,
adding each node type improves the model’s performance. As expected,
publications are the most influential nodes and achieve relatively high
performance in a citation-only setting. The addition of both venues and
institutions only has a marginal, nonetheless positive, effect on the model’s
performance. Finally, when paired with publications, authors provide a
relative 38% MRR improvement over the standalone publications setting. We
attribute this boost of performance to two factors: 1) addition of authors
creates more pathways in the graph for the information to flow 2) author
representations act as dense summaries of more relevant publication nodes for
link prediction.

Variation | MRR | Hits@1 | Hits@10 | Hits@50
---|---|---|---|---
Full | 0.725 | 0.609 | 0.920 | 0.992
\- V | 0.713 | 0.598 | 0.909 | 0.969
\- I | 0.722 | 0.606 | 0.919 | 0.978
\- V - I | 0.715 | 0.600 | 0.909 | 0.970
\- A - I | 0.536 | 0.394 | 0.802 | 0.922
\- V - A - I | 0.518 | 0.380 | 0.780 | 0.900
Table 6: Ablation study on the PG-1M dataset. Legend: I: Institutions, A:
Authors, V: Venues.

###  5.4 Negative Sampling

Throughout our experiments, we employed the strategy of sampling 1000 negative
samples for each positive sample during the evaluation phase. However, this
strategy is prone to exhibiting inflated performance due to having no control
over the difficulty of the sampled nodes. For example, we might end up with
candidate nodes with different types than the target node, e.g., publication
vs. venue. In most cases, assuming the model is well-trained, and the data is
valid, the model will have an easier job assigning lower scores to these
nodes. Moreover, calculating the evaluation metrics on the complete set of
samples becomes increasingly more expensive as the size of the KG grows.
Hence, we propose three alternative strategies for negative sampling during
the evaluation phase. These strategies aim to find an efficient method that
could be used as a proxy for complete metric calculations. Our proposed
strategies are as follows:

  1. 1.

Entity Type: This is the most straightforward strategy in which we only sample
candidate nodes with the same type as the target node. For example, in our
case, we only sample from the publications.

  2. 2.

Time Constrained: Building upon our first strategy, we further add the
constraint of only sampling candidate nodes from the nodes within the
evaluation period. Intuitively, these unseen (inductive) or less seen
(transductive) nodes will pose more problems for the model during the
evaluation phase.

  3. 3.

Community: This strategy relies on the auxiliary outputs, i.e., communities,
that we generate from the PubGraph as described in Appendix C. Given a target
node, we sample candidate nodes only from its community. We hypothesize that
these nodes pose the most difficulty for the model during the evaluation
phase.

##### Discussion

Table 7 presents the results of our experiments with the aforementioned
negative sampling strategies. The reported times are for one evaluation run
over the complete testing set of the PG-1M dataset ($\sim$147K samples). As
evident from these results, the community-based method is the best proxy to
the full metrics calculation while still being significantly time efficient.
Even if we factor in the 11.5 hours (41400 seconds) that it takes to learn
communities for all the 91M publications, the difference in computation time
becomes much more significant when we have to repeat the evaluation process
over and over again, e.g., for validation, fine-tuning, etc. Moreover, the
full metrics are calculated on a GPU which is far more efficient than the
calculations on the CPU. It is important to note that the community-based
method is helpful in evaluation settings where the ground truth is known;
however, in settings where the ground truth is unknown, e.g., a deployed
model, there is no workaround to complete ranking computations as we have to
consider all the entities regardless. Appendix D provides more analysis of the
proposed approaches.

Variation |  #Negative Samples | MRR | Hits@1 | Hits@10 |  Time (Seconds)
---|---|---|---|---|---
Random | 1000 | 0.723 | 0.608 | 0.918 | 588 (CPU)
Entity Type | 1000 | 0.560 | 0.418 | 0.826 | 655 (CPU)
Time Constrained | 1000 | 0.577 | 0.449 | 0.817 | 601 (CPU)
Community | 1000 | 0.076 | 0.023 | 0.167 | 1008 (CPU)
Full |  $\sim$3.38M | 0.015 | 0.000 | 0.036 | 81987 (GPU)

Table 7: Negative sampling results on the PG-1M datasets.

##  6 Conclusion and Future Work

We extracted a novel large-scale KG from the OpenAlex catalog with more than
15.49 billion edges and 432 million nodes to study the innovation phenomena.
Moreover, we created small and medium-sized variations of the full KG to
enable future studies at different scales. Although we focused on the
extrapolated setting in this work, these KGs could be easily adapted in
studies with interpolated settings. Furthermore, given the suitability of
PubGraph for both transductive and inductive learning environments, we
conducted numerous experiments on many popular static transductive, temporal
transductive, and inductive models. We also presented a hybrid inductive
learning framework designed for large-scale KGs and adapted the existing
models to the featureless setting. Based on the results of our experiments,
the introduced hybrid framework outperforms its counterparts across all
models. This result is especially significant when working with large-scale
graphs, as it provides fast training while maintaining high performance.
Finally, we presented various analyses to understand our experiments’ results
better. Our analyses showcased the cost issues in the evaluation phase.
Moreover, our experiments shed light on the shortcomings of the temporal
transductive models in extrapolated settings. We also showcased the scaling
issues of current state-of-the-art models. As for future directions, apart
from the challenges and problems arising from the aforementioned shortcomings,
finding “surprising” and “should-have” citations is an exciting research
direction.

## References

  * Adhikari et al. (2020) Adhikari, A.; Yuan, X.; Côté, M.-A.; Zelinka, M.; Rondeau, M.-A.; Laroche, R.; Poupart, P.; Tang, J.; Trischler, A.; and Hamilton, W. 2020.  Learning dynamic belief graphs to generalize on text-based games.  _Advances in Neural Information Processing Systems_ , 33: 3045–3057.
  * Albooyeh, Goel, and Kazemi (2020) Albooyeh, M.; Goel, R.; and Kazemi, S. M. 2020.  Out-of-sample representation learning for knowledge graphs.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 2657–2666.
  * Ammanabrolu and Hausknecht (2020) Ammanabrolu, P.; and Hausknecht, M. 2020.  Graph constrained reinforcement learning for natural language action spaces.  _arXiv preprint arXiv:2001.08837_.
  * Battaglia et al. (2018) Battaglia, P. W.; Hamrick, J. B.; Bapst, V.; Sanchez-Gonzalez, A.; Zambaldi, V.; Malinowski, M.; Tacchetti, A.; Raposo, D.; Santoro, A.; Faulkner, R.; et al. 2018.  Relational inductive biases, deep learning, and graph networks.  _arXiv preprint arXiv:1806.01261_.
  * Bhagavatula et al. (2018) Bhagavatula, C.; Feldman, S.; Power, R.; and Ammar, W. 2018.  Content-based citation recommendation.  _arXiv preprint arXiv:1802.08301_.
  * Bollacker et al. (2008) Bollacker, K.; Evans, C.; Paritosh, P.; Sturge, T.; and Taylor, J. 2008.  Freebase: a collaboratively created graph database for structuring human knowledge.  In _Proceedings of the 2008 ACM SIGMOD international conference on Management of data_ , 1247–1250.
  * Bordes et al. (2013) Bordes, A.; Usunier, N.; Garcia-Duran, A.; Weston, J.; and Yakhnenko, O. 2013.  Translating embeddings for modeling multi-relational data.  _Advances in neural information processing systems_ , 26.
  * Cohan et al. (2019) Cohan, A.; Ammar, W.; Van Zuylen, M.; and Cady, F. 2019.  Structural scaffolds for citation intent classification in scientific publications.  _arXiv preprint arXiv:1904.01608_.
  * Cohan et al. (2020) Cohan, A.; Feldman, S.; Beltagy, I.; Downey, D.; and Weld, D. S. 2020.  Specter: Document-level representation learning using citation-informed transformers.  _arXiv preprint arXiv:2004.07180_.
  * De Vaan, Stark, and Vedres (2015) De Vaan, M.; Stark, D.; and Vedres, B. 2015.  Game changer: The topology of creativity.  _American Journal of Sociology_ , 120(4): 1144–1194.
  * Dettmers et al. (2018) Dettmers, T.; Minervini, P.; Stenetorp, P.; and Riedel, S. 2018.  Convolutional 2d knowledge graph embeddings.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32.
  * Dietz, Bickel, and Scheffer (2007) Dietz, L.; Bickel, S.; and Scheffer, T. 2007.  Unsupervised prediction of citation influences.  In _Proceedings of the 24th international conference on Machine learning_ , 233–240.
  * Färber and Sampath (2020) Färber, M.; and Sampath, A. 2020.  Hybridcite: A hybrid model for context-aware citation recommendation.  In _Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020_ , 117–126.
  * Galkin et al. (2021) Galkin, M.; Wu, J.; Denis, E.; and Hamilton, W. L. 2021.  Nodepiece: Compositional and parameter-efficient representations of large knowledge graphs.  _arXiv preprint arXiv:2106.12144_.
  * Goel et al. (2020) Goel, R.; Kazemi, S. M.; Brubaker, M.; and Poupart, P. 2020.  Diachronic embedding for temporal knowledge graph completion.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, 3988–3995.
  * Goodman (1961) Goodman, L. A. 1961.  Snowball sampling.  _The annals of mathematical statistics_ , 148–170.
  * Gururangan et al. (2020) Gururangan, S.; Marasović, A.; Swayamdipta, S.; Lo, K.; Beltagy, I.; Downey, D.; and Smith, N. A. 2020.  Don’t stop pretraining: adapt language models to domains and tasks.  _arXiv preprint arXiv:2004.10964_.
  * Hamilton, Ying, and Leskovec (2017) Hamilton, W.; Ying, Z.; and Leskovec, J. 2017.  Inductive representation learning on large graphs.  _Advances in neural information processing systems_ , 30.
  * Hofstra et al. (2020) Hofstra, B.; Kulkarni, V. V.; Munoz-Najar Galvez, S.; He, B.; Jurafsky, D.; and McFarland, D. A. 2020.  The diversity–innovation paradox in science.  _Proceedings of the National Academy of Sciences_ , 117(17): 9284–9291.
  * Hope et al. (2017) Hope, T.; Chan, J.; Kittur, A.; and Shahaf, D. 2017.  Accelerating innovation through analogy mining.  In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 235–243.
  * Hu et al. (2021) Hu, W.; Fey, M.; Ren, H.; Nakata, M.; Dong, Y.; and Leskovec, J. 2021.  Ogb-lsc: A large-scale challenge for machine learning on graphs.  _arXiv preprint arXiv:2103.09430_.
  * Hu et al. (2020) Hu, W.; Fey, M.; Zitnik, M.; Dong, Y.; Ren, H.; Liu, B.; Catasta, M.; and Leskovec, J. 2020.  Open graph benchmark: Datasets for machine learning on graphs.  _Advances in neural information processing systems_ , 33: 22118–22133.
  * Ilievski et al. (2020) Ilievski, F.; Garijo, D.; Chalupsky, H.; Divvala, N. T.; Yao, Y.; Rogers, C.; Li, R.; Liu, J.; Singh, A.; Schwabe, D.; et al. 2020.  KGTK: a toolkit for large knowledge graph manipulation and analysis.  In _International Semantic Web Conference_ , 278–293. Springer.
  * Jurgens et al. (2018) Jurgens, D.; Kumar, S.; Hoover, R.; McFarland, D.; and Jurafsky, D. 2018.  Measuring the evolution of a scientific field through citation frames.  _Transactions of the Association for Computational Linguistics_ , 6: 391–406.
  * Kang et al. (2022) Kang, H. B.; Qian, X.; Hope, T.; Shahaf, D.; Chan, J.; and Kittur, A. 2022.  Augmenting scientific creativity with an analogical search engine.  _ACM Transactions on Computer-Human Interaction_.
  * Liu et al. (2019) Liu, H.; Kou, H.; Yan, C.; and Qi, L. 2019.  Link prediction in paper citation network to construct paper correlation graph.  _EURASIP Journal on Wireless Communications and Networking_ , 2019(1): 1–12.
  * Narasimhan, Lazebnik, and Schwing (2018) Narasimhan, M.; Lazebnik, S.; and Schwing, A. 2018.  Out of the box: Reasoning with graph convolution nets for factual visual question answering.  _Advances in neural information processing systems_ , 31.
  * Ostendorff et al. (2022) Ostendorff, M.; Rethmeier, N.; Augenstein, I.; Gipp, B.; and Rehm, G. 2022.  Neighborhood contrastive learning for scientific document representations with citation embeddings.  _arXiv preprint arXiv:2202.06671_.
  * Paszke et al. (2019) Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32.
  * Price (1965) Price, D. J. D. S. 1965.  Networks of scientific papers: The pattern of bibliographic references indicates the nature of the scientific research front.  _Science_ , 149(3683): 510–515.
  * Priem, Piwowar, and Orr (2022) Priem, J.; Piwowar, H.; and Orr, R. 2022.  OpenAlex: A fully-open index of scholarly works, authors, venues, institutions, and concepts.  _arXiv preprint arXiv:2205.01833_.
  * Schlichtkrull et al. (2018) Schlichtkrull, M.; Kipf, T. N.; Bloem, P.; Berg, R. v. d.; Titov, I.; and Welling, M. 2018.  Modeling relational data with graph convolutional networks.  In _European semantic web conference_ , 593–607. Springer.
  * Sun et al. (2019) Sun, Z.; Deng, Z.-H.; Nie, J.-Y.; and Tang, J. 2019.  Rotate: Knowledge graph embedding by relational rotation in complex space.  _arXiv preprint arXiv:1902.10197_.
  * Traag, Waltman, and Van Eck (2019) Traag, V. A.; Waltman, L.; and Van Eck, N. J. 2019.  From Louvain to Leiden: guaranteeing well-connected communities.  _Scientific reports_ , 9(1): 1–12.
  * Trouillon et al. (2016) Trouillon, T.; Welbl, J.; Riedel, S.; Gaussier, É.; and Bouchard, G. 2016.  Complex embeddings for simple link prediction.  In _International conference on machine learning_ , 2071–2080. PMLR.
  * Uzzi et al. (2013) Uzzi, B.; Mukherjee, S.; Stringer, M.; and Jones, B. 2013.  Atypical combinations and scientific impact.  _Science_ , 342(6157): 468–472.
  * Vrandečić and Krötzsch (2014) Vrandečić, D.; and Krötzsch, M. 2014.  Wikidata: a free collaborative knowledgebase.  _Communications of the ACM_ , 57(10): 78–85.
  * Wang et al. (2019a) Wang, H.; Zhang, F.; Zhang, M.; Leskovec, J.; Zhao, M.; Li, W.; and Wang, Z. 2019a.  Knowledge-aware graph neural networks with label smoothness regularization for recommender systems.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, 968–977.
  * Wang et al. (2019b) Wang, H.; Zhao, M.; Xie, X.; Li, W.; and Guo, M. 2019b.  Knowledge graph convolutional networks for recommender systems.  In _The world wide web conference_ , 3307–3313.
  * Wang et al. (2020) Wang, K.; Shen, Z.; Huang, C.; Wu, C.-H.; Dong, Y.; and Kanakia, A. 2020.  Microsoft academic graph: When experts are not enough.  _Quantitative Science Studies_ , 1(1): 396–413.
  * Wang et al. (2019c) Wang, M.; Zheng, D.; Ye, Z.; Gan, Q.; Li, M.; Song, X.; Zhou, J.; Ma, C.; Yu, L.; Gai, Y.; Xiao, T.; He, T.; Karypis, G.; Li, J.; and Zhang, Z. 2019c.  Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks.  _arXiv preprint arXiv:1909.01315_.
  * Wang et al. (2019d) Wang, X.; He, X.; Cao, Y.; Liu, M.; and Chua, T.-S. 2019d.  Kgat: Knowledge graph attention network for recommendation.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, 950–958.
  * Xu et al. (2020) Xu, D.; Ruan, C.; Korpeoglu, E.; Kumar, S.; and Achan, K. 2020.  Inductive representation learning on temporal graphs.  _arXiv preprint arXiv:2002.07962_.
  * Zhang et al. (2018a) Zhang, Y.; Dai, H.; Kozareva, Z.; Smola, A. J.; and Song, L. 2018a.  Variational reasoning for question answering with knowledge graph.  In _Thirty-second AAAI conference on artificial intelligence_.
  * Zhang et al. (2018b) Zhang, Y.; Dai, H.; Toraman, K.; and Song, L. 2018b.  Kg^ 2: Learning to reason science exam questions with contextual knowledge graph embeddings.  _arXiv preprint arXiv:1805.12393_.
  * Zheng et al. (2020) Zheng, D.; Song, X.; Ma, C.; Tan, Z.; Ye, Z.; Dong, J.; Xiong, H.; Zhang, Z.; and Karypis, G. 2020.  Dgl-ke: Training knowledge graph embeddings at scale.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , 739–748.

##  Appendix A Dataset

Table 8 presents the statistics on the extracted splits of each dataset.
During the final experiment on the test set, we add all the edges in the
validation period to the training set, aligning the testing and validation
experiment settings.

Dataset | PG-1M | PG-10M | PG-Full
---|---|---|---
#Training (Validation) | 18,201,417 | 269,061,163 | 1,881,118,345
#Training (Testing) | 20,541,357 | 305,902,166 | 2,174,849,724
#Validation | 265,196 | 3,160,831 | 28,179,279
#Test | 147,106 | 2,396,547 | 26,389,424

Table 8: Statistics of datasets splits.

##  Appendix B Hyperparameters

###  B.1 Static Transductive

We tune the hyper-parameters of both models using the following set of values:
embedding dimensions $\in\\{50,100,200,400\\}$, learning rate
$\in\allowbreak\\{0.003\allowbreak,0.01\allowbreak,0.03,0.1,0.3\\}$, number of
negative samples
$\in\allowbreak\\{128\allowbreak,256,512\allowbreak,1024,2048\\}$,
regularization coefficient
$\in\\{0.0,\text{1e-9},\text{1e-8},\text{1e-7},\text{1e-6},\text{1e-5}\\}$,
$\alpha\in\\{0.25,0.5,1,2\\}$, and $\gamma\in\\{0.0,6,12,18,24\\}$. Due to the
high computational cost of running sweeps on larger KGs, we only tune the
parameters on the smaller PG-1M dataset and reuse the same hyper-parameters
for PG-10M and PG-Full.

Hyperparameter | ComplEx | RotatE
---|---|---
embedding dimension | 200 | 50
learning rate | 0.3 | 0.1
number of negative samples | 512 | 64
regularization coefficient | 1e-6 | 1e-6
$\alpha$ | 0.25 | 1
$\gamma$ | - | 6
Table 9: Best hyperparameters for the DE-X models.

###  B.2 Temporal Transductive

In addition to the original training environment, we experimented with
adversarial negative sampling. The hyper-parameters of both models are tuned
using the following set of values: embedding dimension $\in\\{50,100,200\\}$,
$\psi\in\\{0.08,0.16,0.32,0.64\\}$, learning rate
$\in\allowbreak\\{0.003\allowbreak,0.01,0.03,0.1,0.3\\}$, number of negative
samples $\in\\{128,256,512,1024\\}$, regularization coefficient
$\in\\{0.0,\text{1e-9},\text{1e-8},\text{1e-7},\text{1e-6},\text{1e-5}\\}$,
dropout $\in\\{0.0,0.1,0.2\\}$, $\alpha\in\\{0.25,0.5,1,2\\}$, and
$\gamma\in\\{0.0,6,12,\allowbreak 18,24\\}$.

Hyperparameter | TransE | DistMult
---|---|---
embedding dimension | 100 | 100
$\psi$ | 0.08 | 0.08
learning rate | 0.1 | 0.1
number of negative samples | 512 | 512
regularization coefficient | 0.0 | 1e-6
dropout | 0.0 | 0.1
$\alpha$ | 0.25 | -
$\gamma$ | 6 | -
Table 10: Best hyperparameters for the DE-X models.

###  B.3 Inductive

We only use 1-layer models throughout our experiments as running 2-layer
models proved to be infeasible memory-wise on our GPUs.

GraphSAGE-X: We tune the variations using the following set of values: number
of negative samples $\in\\{5,25,50\\}$, learning rate $\in\\{0.001,\allowbreak
0.003,0.01,0.03,0.1\\}$, fanouts $\in\\{5,15,25\\}$, model dimension
$\in\\{100,200,400\\}$, dropout $\in\\{0.0,0.1,0.2\\}$, aggregator type
$\in\\{\text{mean},\text{pool}\\}$, and normalization
$\in\\{\text{None},\allowbreak\text{Layer}\\}$.

Hyperparameter | -D | -H | -E
---|---|---|---
number of negative samples | 25 | 50 | 50
learning rate | 0.03 | 0.03 | 0.03
fanouts | 15 | 50 | 25
model dimension | 100 | 400 | 400
dropout | 0.0 | 0.1 | 0.1
aggregator type | mean | mean | mean
normalization | Layer | Layer | Layer
Table 11: Best hyperparameters for the GraphSAGE-X models.

RGCN-X: We tune the hyper-parameters of all the variations using the following
set of values: number of negative samples $\in\\{5,25,50\\}$, learning rate
$\in\\{0.001,\allowbreak 0.003,0.01,0.03,0.1\\}$, fanouts $\in\\{5,15,25\\}$,
model dimension $\in\\{100,200,400\\}$, number of bases $\in\\{8,16,32\\}$,
dropout $\in\\{0.0,0.1,0.2\\}$, and normalization
$\in\\{\text{None},\allowbreak\text{Layer}\\}$.

Hyperparameter | -D | -H | -E
---|---|---|---
number of negative samples | 25 | 25 | 50
learning rate | 0.01 | 0.003 | 0.03
fanouts | 25 | 15 | 25
model dimension | 100 | 400 | 200
number of bases | 8 | 8 | 8
dropout | 0.0 | 0.0 | 0.0
normalization | Layer | Layer | Layer
Table 12: Best hyperparameters for the RGCN-X models.

##  Appendix C Community Detection

As an auxiliary output, we use the Leiden algorithm (Traag, Waltman, and Van
Eck 2019) to detect communities in PubGraph. To this end, we first extract the
full citation network from all the publication-publication links. Then, we use
the Leiden algorithm to detect the communities in the citation network.

Throughout our experiments, we use the leidenalg
library111https://github.com/vtraag/leidenalg. We modify this implementation
to restrict the maximum number of generated communities. More specifically, to
restrict the number of communities to $N$, we implement the following
procedure:

  1. 1.

Create $N$ communities.

  2. 2.

Randomly assign each node to an arbitrary community.

  3. 3.

Execute the Leiden algorithm without allowing it to form new communities.

Moreover, we restrict the number of papers in a community to a max value to
avoid creating 1) large unfocused communities and 2) small insignificant
communities. Then, we experiment with four quality functions: Modular, RBER
(Reichardt and Bornholdt’s Potts model with an Erdős-Rényi null model),
Significance, and Surprise. Our experiments show that the Significance quality
function produces the highest quality communities.

Since OpenAlex concepts have a hierarchical structure, for each community, we
extract the ancestral graph of the concepts connected to the publications and
count the number of children for each root concept. Then, we select the
largest root concept for each community and calculate the percentage of the
papers that are children of that root concept. This metric is then used to
determine the communities’ quality when tuning the algorithm’s parameters.

We tune the Leiden algorithm on the extracted citation network with the
following parameters: maximum papers per community $\in\\{300,000,500,000\\}$,
number of communities $\in\\{3000,4000,5000,6000\\}$. Based on our
experiments, the highest quality communities are produced by the following
parameters: maximum papers per community $=300,000$, number of communities
$=3000$. Figure 4 illustrates our results on different number of communities.

Figure 4: Analysis of the effect of the number of communities on the quality
of communities.

##  Appendix D Negative Sampling

We further analyze the effect of the number of negative samples on the model’s
performance. Figure 5 presents the result of our experiments with varying
numbers of negative samples on all the introduced strategies. As expected, the
model’s performance rapidly drops with the increase of negative samples.
Moreover, the community-based negative sampling results act as an excellent
proxy at 5k negative samples and seem to be converging to the full variation
around 10k negative samples. This finding is further evidence of the
effectiveness of this method.

Figure 5: Analysis of the effect of negative samples count on the model’s
performance measured by MRR.

WikiData Properties
---
OpenAlex Metadata |  WikiData Property |  OpenAlex Metadata |  WikiData Property |  OpenAlex Metadata |  WikiData Property
OpenAlex Id |  P10283 (OpenAlex ID) | Type |  P2308 (class) | Country Code |  P299 (ISO 3166-1 numeric code)
MAG |  P6366 (Microsoft Academic ID) | Updated Date |  P5017 (last update) | City |  P131 (located in the administrative territorial entity)
Orcid |  P496 (ORCID ID) | Year |  P585 (point in time) | Geo Region |  P276 (location)
Grid Id |  P2427 (GRID ID) | Created Date |  P571 (inception) | Country |  P17 (country)
Scopus Id |  P1153 (Scopus author ID) | Date of Publication |  P577 (publication date) | Published in |  P1433 (published in)
ROR |  P6782 (ROR ID) | Author |  P50 (author) | Latitude & Longitude |  P625 (coordinate location)
Wikidata Id |  P1687 (Wikidata property) | Title |  P1476 (title) | Description |  P10358 (original catalog description)
Geonames Id |  P1566 (GeoNames ID) | Publisher |  P123 (publisher) | Ancestor |  P1038 (relative)
DOI |  P356 (DOI) | Related to |  P1659 (related properties) | Level/Position |  P1352 (ranking)
ISSN |  P236 (ISSN) | Works Count |  P3740 (number of works) | Twitter Handle |  P2002 (Twitter username)
ISSNL |  P7363 (ISSN-L) | Volume |  P478 (volume) | MeSH Qualifier Id |  P9341 (MeSH qualifier ID)
PMID |  P698 (PubMed ID) | Issue |  P433 (issue) | MeSH Descriptor Id |  P486 (MeSH descriptor ID)
PMCID |  P932 (PMCID) | Homepage URL |  P856 (official website) | Score |  P4271 (number of points/goals/set scored)
UMLS CUI |  P2892 (UMLS CUI) | OA status |  P6954 (online access status) | Relationship |  P2309 (relation)
Instance of |  P31 (instance of) | Affiliation |  P1416 (Affiliation) | Concept |  P921 (main subject)
Wikipedia |  P5178 (Wikipedia Library partner ID) | Cites work |  P2860 (Cites work) | Display Name |  P2561 (name)
Artificial Properties
OpenAlex Metadata |  Artificial Property |  OpenAlex Metadata |  Artificial Property |  OpenAlex Metadata |  Artificial Property
Cited by count |  P_cited_by_count | Is retracted |  P_is_retracted | UMLS AUI |  P_umls_aui

Table 13: OpenAlex metadata information mapping to WikiData properties.

##  Appendix E WikiData Ontology

Table 13 presents the mapping from OpenAlex metadata to WikiData properties,
which we created in the transformation phase. These mappings are selected such
that they best describe the metadata field. For those metadata with no
suitable parallel property, we create artificial ones to keep the KG as
complete as possible. We explain certain fields that might come across as
ambiguous as follows:

  1. 1.

Instance of $\rightarrow$ P31: “Instance of” indicates the class to which a
particular entity belongs. The entities are Work, Human, Concept, Institution,
and Venue.

  2. 2.

Type $\rightarrow$ P2308: “Type” indicates the type of work, e.g., journal
article, book, etc. (uses crossref’s type), and the type of institution, e.g.,
education, healthcare, etc. (controlled by ROR type)

  3. 3.

Related to $\rightarrow$ P1659: “Related to” indicates a related concept or an
affiliated institution.

  4. 4.

Ancestor $\rightarrow$ P1038: “Ancestor” is used to indicate the ancestors of
a concept in the concept hierarchical tree.

  5. 5.

Level/Position $\rightarrow$ P1352: “Level/Position” indicates the concept
level in the concept tree. Moreover, it is also used to indicate an author’s
position in a publication.

  6. 6.

Score $\rightarrow$ P4271: “Score” indicates how much two concepts are
related. It is further used to indicate how good of a match a concept is to a
publication.

##  Appendix F Surprising Citations

Query paper | Negative paper | Relative Score
---|---|---
BiPO 4 ‐Derived 2D Nanosheets for Efficient Electrocatalytic Reduction of CO2 to Liquid Fuel |  Carbon‐Based Metal‐Free Catalysts for Electrocatalytic Reduction of Nitrogen for Synthesis of Ammonia at Ambient Conditions | 2.927
High‐Performance Fullerene Free Polymer Solar Cells Based on New Thiazole‐Functionalized Benzo[1,2‐b:4,5‐b] dithiophene D‐A Copolymer Donors |  Chlorine substituted 2D-conjugated polymer for high-performance polymer solar cells with 13.1% efficiency via toluene processing | 1.934
Cationic vs. non-cationic polymeric vectors for nucleic acid delivery |  The promises and pitfalls of RNA-interference-based therapeutics | 1.109
Force‐Induced Turn‐On Persistent Room‐Temperature Phosphorescence in Purely Organic Luminogen |  How the Molecular Packing Affects the Room Temperature Phosphorescence in Pure Organi Compounds: Ingenious Molecular Design, Detailed Crystal Analysis, and Rational Theoretical Calculations | 2.605
Structures and Spin States of Iron(II) Complexes of Isomeric 2,6-Di(1,2,3-triazolyl)pyridine |  Structure function relationships in molecular spin-crossover complexes | 1.408
Table 14: Examples of “should have” citations.

Query paper | Positive paper | Rank | Explanation
---|---|---|---
Morphological evolution of self‐assembled PS‐g‐PA6 graft copolymer via in situ polymerization |  Analytical characterization of color changing waste polystyrene plastic | 527 |  Positive paper only has 2 citations and 4 references
All You Need Is Sleep: The Effects of Sleep Apnea and Treatment Benefits in the Heart Failure Patient |  Supplemental dataset on the influence of cardiac resynchronization therapy in pacing-induced cardiomyopathy and concomitant central sleep Apnea | 501 |  Positive paper only has 1 citation and 2 references
In-Situ Growth of MOF for Energy Conversion and Storage Devices |  Waste tyre pyrolysis – Impact of the process and its products on the environment | 622 |  Difficult to find the similarity based on the titles
Normal Force-Induced Highly Efficient Mechanical Sterilization of GaN Nanopillars |  Multidrug-Resistant Bacterial Infections in U.S. Hospitalized Patients, 2012–2017 | 812 |  Difficult to find the similarity based on the titles
Applications of Machine Learning in Bone and Mineral Research |  Comparison of Machine Learning Models to Predict Risk of Falling in Osteoporosis Elderly | 933 |  Bad performance of ComplEx

Table 15: Examples of “surprising” citations.

We further analyze the result of our ComplEx model by running a few case
studies on the “should-have”, i.e., high scores for negative samples, and the
“surprising,” i.e., low scores for positive samples, citations. For each query
paper and positive candidate pair, we randomly select 1000 negative papers and
rank the results based on the scores generated by the model.

Table 14 showcases some examples of “should-have” citations where none of the
positive papers are ranked first and presents the negative papers that are
ranked first. The relative score is calculated as the ratio of the negative
paper’s score to the positive paper’s. Table 15 presents the examples of
“surprising” citations where the positive papers are ranked lower than 500. In
some cases, the positive paper only has a few citations and references, which
leads to being under-trained. In other cases, it isn’t easy to find the
similarity even provided with the title. Finally, in some samples, even though
the titles show a high degree of similarity, ComplEx performs poorly.


# A Brief Survey of Open Radio Access Network (O-RAN) Security

YI-ZIH CHEN<EMAIL_ADDRESS>National Taiwan UniversityTaipeiR.O.C ,  PO-
JUNG SU<EMAIL_ADDRESS>National Taiwan UniversityTaipeiR.O.C ,
TERRANCE YU-HAO CHEN<EMAIL_ADDRESS>National Taiwan
UniversityTaipeiR.O.C and  CHI-TING LIU<EMAIL_ADDRESS>National Taiwan
UniversityTaipeiR.O.C

###### Abstract.

Open Radio Access Network (O-RAN), a novel architecture that separates the
traditional radio access network (RAN) into multiple disaggregated components,
leads a revolution in the telecommunication ecosystems. Compared to the
traditional RAN, the proposed O-RAN paradigm is more flexible and more cost-
effective for the operators, vendors, and the public. The key design
considerations of O-RAN include virtualization and intelligent capabilities in
order to meet the new requirements of 5G. However, because of the open nature
and the newly imported techniques in O-RAN architecture, the assessment of the
security in O-RAN architecture during its early development stage is crucial.
This project aims to present an investigation of the current ORAN architecture
from several attack surfaces, including (1) Architectural openness, (2) Cloud
and Virtualization, (3) Network slicing, and (4) Machine Learning. The
existing attack surfaces and corresponding mitigation methods of these attacks
are also surveyed and provided in this report, serving as a guiding principle
and valuable recommendation for the O-RAN implementers and framework
designers.

open radio access network, security, 5G

††copyright: none Figure 1. Logical Architecture of O-RAN proposed by O-RAN
alliance

##  1\. Introduction

Next-generation telecommunication technologies aim to satisfy the new demands
for enhanced performance, portability, elasticity and energy efficiency of
heterogeneous services. 5G and 6G networks adopt architectural transformations
to build a flexible, agile, and disaggregated system, which makes for the
evolution of wireless networks. Among all the new concepts, the Open Radio
Access Network (O-RAN) framework has been one of the most important
innovations in the field to serve as the prime infrastructure for future
mobile networks.

The objective of ORAN is to build a flexible, reliable, and cost-effective RAN
system by decomposing the Radio Access Network (RAN) into several open and
capable units based on cloud-native technology (Mimran et al., 2022; Klement
et al., 2022). Each component in ORAN is connected through open and
standardized interfaces, making it interoperable across different vendors and
third-party applications. Moreover, the ORAN framework also considers the
integration of machine learning technologies for better network resource
management such as resource allocation for consumers with different needs for
quality of service (QoS).

Fig. 1 shows the O-RAN framework proposed by the O-RAN alliance. Near the
bottom of the schematic, we could see the O-cloud component, which serves as a
cloud-based computing platform and hosts the functions such as the RAN
Intelligent Controller (RIC), Centralized Unit (CU), and Distributed Unit
(DU). Located in the middle of the schematic are the CU and DU, which are
derived from the conventional 3GPP-specified NG-RAN (Next Generation NodesBs
Radio Access Network) that is responsible for the digital data processing
computation. The splitting of CU and DU is based on the scenario of usage and
the trade-offs between performance and bandwidth. CU could be further divided
into the control plane (CP) and the user plane (UP) functions, which are also
referred to as the CU-CP and CU-UP. This division, known as the Control User
Plane Separation (CUPS), is based on various performance needs and could
improve the RAN function placements. Service Management and Orchestration
(SMO) Framework and the RAN Intelligent Controllers (RICs) are shown at the
top of the schematic. These functions aim to realize the automatic resource
management and control of the O-RAN network. The Non-Real-Time RIC (Non-RT
RIC) resides within SMO, supporting intelligent RAN management with policies
and ML-based algorithms. On the other hand, the Near Real-Time RIC (Near-RT
RIC) enables the near-real-time decision-making of O-RAN networks with a
logical function or by leveraging the ML model trained on Non-RT RIC. The time
response of the O-DU control loop is less than 10 milliseconds. For the Near-
RT RIC and Non-RT RIC, the time response is between 10 milliseconds to 1
second and more than 1 second respectively.

Dissecting the RAN architecture results in the extensions of additional
interfaces in the network, which could be identified as green lines in Fig. 1.
The O1-interface is responsible for the network management tasks (FCAPS:
Fault, Configuration, Accounting, Performance, Security) defined by the
international organization for standardization (ISO) model. This interface
supports the communication of network service management related to the
network functions. The O2-interface plays a role in the communication between
O-cloud and the SMO, helping the workload and resource allocation of the O-RAN
system. Security of this channel is extremely important (Mimran et al., 2022).
The Open Fronthaul Management Plane Interface helps the Network Management
Systems (NMSs) and the DUs with the RU administrations. The A1-interface is in
charge of connecting the Near-RT RIC and Non-RT RIC to transmit the data
stored in the SMO (e.g. the training data for the ML model, the enrichment
information of O-RAN, etc.) The E2-interface is used to connect the near-RT
RIC with the E2 nodes including CU, DU, and eNB to control and optimize
resource allocation. Generally, E2-interface supports all 3GPP-defined layers
and protocols. O-RAN alliance is working on the standardization of all the
above-mentioned interfaces, of which the need of security enhancement is
needed.

The openness of the proposed revolutionary architecture brings not only the
advantages but a vastly larger attack surface. With the concept of “secure by
design,” the discussion of security issues in ORAN architecture and the
related technologies is extremely important in the early development stages of
ORAN.

The remainder of the article is organized as follows. Section 2. defines our
review scope in terms of the identified attack surfaces in O-RAN. Section 3.
provides more detailed descriptions of possible attacks corresponding to the
attack surfaces and further presents the existing mitigation methods. Section
4. gives the literature review. Finally, in section 5., the conclusions and
recommendations are presented.

##  2\. Problem definition

This project would analyze the possible attack surfaces of the current O-RAN
framework. From the O-RAN architecture itself to the technologies including
virtualization and ML that are intended to be covered. The potential threats
and recommended mitigation methods are also provided.

We identify four risk areas of O-RAN according to a specific major aspect of
O-RAN framework based on the taxonomy proposed by Dudu Mimran et al (Mimran et
al., 2022). The following are the four risk areas we would look into in-depth:

###  2.1. Architectural openness

Openness is the core concept of O-RAN. The open framework offers several
advantages such as the reduction of operational costs and the opportunities
for small and medium-sized enterprises, which further accelerate the
development of RAN technologies. Nonetheless, openness comes along with more
interfaces and larger variances of security levels, which opens new attack
surfaces in the O-RAN network.

###  2.2. Cloud and Virtualization

In the O-RAN network, the RAN functions are disaggregated and deployed on the
edge clouds utilizing related cloud and virtualization technologies. Multiple
processes with different functionalities are allowed to be run on a single
unit with virtual machines (VMs) or containers. These procedures could be
monitored and managed in a centralized way, allowing a more flexible resource
allocation and the reduction of operational costs. Nonetheless, the explosion
of attack methods targeting the vulnerabilities in the cloud, including
virtualization and containerization, are highly transferable to the
virtualized RAN architecture as well. Therefore, this would also be a key part
of our research on which we will focus, including Image Manipulation Attacks,
Guest-to-Guest Attacks, Guest-to-Hypervisor Attacks, and Inconsistent Secure
Policies as illustrated in the fig. 2.

Figure 2. Cloud and Virtualization Technology Attack Surfaces.

A diagram that pinpoints the four attack surfaces of cloud and virtualization
that our research will focus on.

###  2.3. Network Slicing

The implementation of network slicing also plays an important role in the
O-RAN context based on the cloud-native nature of O-RAN. Network slicing
divides the physical networks into several virtual network slices, allowing
services for specific Quality of Service (QoS) requirements and offering
better resource allocations and isolation for end-to-end logical networks.
Although bringing the above advantages, network slicing technology also gives
rise to security and privacy issues that should be addressed.

###  2.4. Machine Learning

Intelligence is another key objective in the O-RAN system in order to support
complex cellular network management challenges such as resource scheduling,
modulation classification, spectrum sensing, etc. O-RAN framework defines
system-level intelligence units including SMO and RICs to manage the
distributed and disaggregated components of the system to meet the Enhanced
Mobile Broadband (eMBB), Ultra-reliable and Low Latency Communications
(URLLC), and Massive Machine Type Communication (mMTC) requirements of 5G.
These units enable the deployment of machine models and even third-party
machine learning applications.

The main difference between our project and previous works is that we pay more
attention to the areas of Cloud and Virtualization, Network Slicing, and
Machine Learning. Furthermore, we study the existing attack methods and
corresponding solutions or recommendations,

##  3\. Results: Threats and Mitigations

###  3.1. Architectural openness

O-RAN comprises several third Generation Partnership Project (3GPP) compliant
logical functional units. Every unit could communicate with each other through
open and standardized interfaces. The threats from the increasing numbers of
stakeholders and vendors should be carefully considered as the attack points
could be found along the whole supply chain and even to the third-party API
providers. O-RAN is also open to attacks that target traditional RANs or other
cellular protocols. The newly introduced channel might make O-RAN even more
susceptible to the attacks such as the Passive eavesdropping that extract
sensitive data from the communication between UE and RU, the Radio jamming
which aims to disrupt the transmission of radio signals, and the Side-channel
attacks that steal the user information from the user based on the physical
property. Proper guidelines, policies, and protocols must be defined and
followed to secure the O-RAN. Last but not least, it is extremely important to
adopt thorough and comprehensive security assessments (e.g. authentication)in
the transmitting protocols of the interfaces to guarantee secure data
transmissions between the units.

###  3.2. Cloud and Virtualization

As cloud computing is a broad field that is essentially an abstraction of all
IT services. It is destined to become more and more versatile and includes
more and more different features, it is technically impossible to analyze all
aspects of it in our work. Therefore, we would like to focus on cloud
computing services, which are likely to be the first major area of adoption
for O-RAN architecture in the future. In this section, for every attack
surface we mentioned, we would explain the concept, give an example of a
related attack, provide mitigations, and further explain its implications for
O-RAN.

####  3.2.1. Image Manipulation

Applications are stored and executed from binary image files in virtualized
environments. Some threats target the vulnerabilities in binary image files
during image creation and execution. For example, attackers may provide
malicious images containing older versions of OpenSSL and Unix Bash Shell to
victims. Utilizing Shellshock (CVE-2014-6271) and HeartBleed (CVE-2014-0160)
to gain illegal access.

Another vulnerability related to image creation is known as Template Images
Cloning (Grobauer et al., 2010). To provide nearly identical server images,
template images are often used to save computational resources. However, a
template image may have been manipulated so as to provide back-door access for
an attacker. Cloning can also leak private data to other users. In Amazon EC2,
users can provide template images for other users by turning a running image
into a template. Without strict management and configuration, it may contain
data that the user doesn’t wish to make public.

For image-related vulnerabilities, we would like to suggest users update
images to the latest versions automatically and use image filters and virtual
scanners frequently. Nüwa(Zhou et al., 2010) is a tool that enables efficient
and scalable offline patching of dormant VM images, while Mirage(Wei et al.,
2009) is an image management system to remove confidential information and
detect malicious images.

####  3.2.2. Guest-to-Guest Attack

In virtualized environments, applications share the same resources and may
share storage, a CPU, or a host OS. Several threats target those resources as
a means to attack other guest applications, which seem to be even more
prevalent in the context of virtualization. Attack methods include
communicating with other not mutually trusted guest applications, injecting
malicious execution code, leaking information through side-channel attacks,
denial of service (DoS) attacks on other guest applications, etc.

One kind of Guest-to-Guest attack we would like to further dive into is known
as the Flush+Reload side-channel attack as illustrated in (Yarom and Falkner,
2014). Due to a weakness in Intel x86 processors, the page sharing feature
that allows for copy-on-write leads to unexpected information leakages. This
attack focuses on the Last-Level-Cache (LLC) of the CPU which is shared among
multiple cores on the same processor die. With shared pages, the attacker can
ensure what is in and what is not in the cache hierarchy. Therefore, the
attacker can monitor the victim’s access to the memory line. The basic concept
can be summarized as the following.

  1. 1.

Attacker flushes the shared LLC with the x86 instruction clflush

  2. 2.

Victim [accesses / does not access] memory.

  3. 3.

Attacker reloads with [short / long] reload time.

  4. 4.

Attacker infers which memory line is accessed by the victim and thus what
instructions are executed

  5. 5.

Repeat and grind to extract more information

Possible mitigations against the attack include restricting the usage of the
clflush instruction and using newer CPUs, which feature non-inclusive memory
caches. For the former case, it is claimed that use cases of the instruction
are limited to the system function of maintaining cache coherence and
improving program performance. Therefore, it is simply sufficient to limit
clflush to the memory pages to which the process has write access. For the
latter case, non-inclusive cache memories, where cache memories in L1 do not
necessarily have to be in L2 in L3. Therefore, the attack becomes useless as
evicting a block from LLC does not mean evicting it from the cache completely.

Though this kind of attack may be harder to carry out on private clouds like
O-RAN where it is a lot harder to get intimate knowledge of the other tenants
running on the same infrastructure, it serves as a great example for us to see
how related attacks can extract information (private keys, credentials, etc. )
from other O-RAN services and how we could implement defenses against them.

####  3.2.3. Guest-to-Hypervisor Attack

Similar to Guest-to-Guest attacks, the Guest-to-Hypervisor attack also targets
shared resources between the hypervisor and guests. However, since the
hypervisor also acts as a single point of failure, these kinds of attacks
focus more on compromising the hypervisor in order to have more power to carry
out further attacks. Existing threats especially aim at harming the integrity
of the hypervisor, DoS attack on the hypervisor, or trying to avoid the
monitoring of VMMs (Virtual Machine Monitor), etc.

To more clearly explain the concept above, we would like to show a case in
which a Guest-to-Hypervisor Attack is carried out in order to pull off a
Guest-to-Guest Attack. In this scenario, the attacker tries to stall the live
migrations (Atya et al., 2017) of VMs on the same hypervisor so that it has
time to carry out the Flush+Reload Attack. As mentioned above, in order to
carry out a Flush+Reload Attack, the attacker needs to have sufficient
knowledge of the neighboring tenants in order to carry out the attack
successfully. Since the attack usually takes approximately a few ten minutes
($\approx$ 26 minutes) to complete, performing live migrations periodically
can greatly reduce side-channel attacks (Atya et al., 2017). Based on this
knowledge, the guest-to-hypervisor attack tries to create dirty pages and
cause bus contention deliberately in order to slow down the process of live
migrations, which increases the length of the attack time span and, thus, the
success rate.

Possible mitigations for the attack include setting resource usage limits on
the process of live migrations and allocating more resources for the migrating
VM to make it harder for the attacker VM to gather enough resources in order
to stall the migrating VM for a long enough duration.

As ORAN infrastructure gradually adopts a cloud-native approach, more and more
different functions will be virtualized. Therefore, how the hypervisor manages
all of these complex relations and defends against malicious attackers would
be an important area of concern when designing ORAN infrastructures.

####  3.2.4. Inconsistent Secure Policies

Since the core value of ORAN is its openness, misconfiguration resulting from
human errors where humans can be the developer of components, integrators,
engineers, and operators represent the highest risks in the system. Moreover,
as the complexity of cloud-based assets grows, the range of attacks on these
assets and the stakes concerned has also been on the rise. This includes the
exploitation of misconfigurations of cloud databases, storage, and
computational units. Not to mention, the perimeterless nature of cloud-based
systems also makes them vulnerable to malicious actors.

Many of these issues mentioned above already exist in our public cloud service
providers and, therefore, should also be kept in mind when designing future
ORAN architectures. One example is the widespread misconfiguration of the
well-known cloud storage service AWS S3 (Continella et al., 2018). Studies
that try to target the specific service and scan for as many buckets as
possible have revealed a daunting statistic. According to the work, out of all
the 240,000 S3 buckets found, 14% of them are publicly-listable, 11% of them
are readable, and 2% of them are writable. Readable buckets are likely to leak
private information such as user data or private credentials, while writable
buckets can be even more dangerous because of possible defacement, web
resource injections, and domain name trust takeovers, which could prove
dangerous for website owners and users simultaneously.

Possible mitigations for the attack include all kinds of education, awareness,
verification tools, and others. To give some examples, for bucket owners, a
tool to safely check the access policy of their buckets (including sensitive
text and file extension analysis) would be extremely helpful in ensuring the
deployment of safe configurations. On the other hand, for website users, we
could also provide a tool that checks for the existence of writable S3 buckets
and refuse from loading those dangerous web resources. Though solving the
human problem regarding security by and large is far from possible, we would
like to emphasize the importance of it in the process of designing open
systems, including O-RAN.

###  3.3. Network Slicing

By and large, the concept and technologies used in network slicing are similar
to those utilized in network virtualization, and so are the attacks and
mitigation methods. However, it can be seen as an independent technology that
deserves special attention. We would like to discuss the network-slicing-
specialized attack surfaces based on the survey by Ruxandra F.Olimid et al.
(Olimid and Nencioni, 2020)

####  3.3.1. Slice life cycle

Preparation.:

During this phase, a slice template defining the framework and the
configurations of a slice is created. Attacks that are similar to the Image
Manipulation might take place. A poorly designed or tampered with slice
template might affect all the slices built from it. As such, the secure
cryptographical protocols must be adopted during slice design and the
integrity of each slice template must be checked before deployment. The
disclosure of the template content should be prevented.

Instantiation, configuration, and activation.:

A new slice of resources and network functions would be built based on the
template in this phase. One should keep his/her eye open to the threats from
APIs. Adversaries might also forge a slice in this phase. To mitigate the
possible attacks, one should only use secure APIs on their slice and adopts
TLS for data transmission. For better defense, the APIs should also support
monitoring their own traffic log files by legal means.

Run time.:

The slice is in use and is vulnerable to the changes of the slice (e.g.
mitigation of slice configuration, permission setting, resource allocation) in
this phase. One should also be careful of data exposures and the availability
damage of the slice. Due to frequent access of the slice and the broad and
various users, there are a large variety of attacks during the preceding
phase. Overall, to defend against these attacks, the importance of
authenticity and integrity verification should be emphasized to deny the
adversary instance. A more detailed discussion of attacks and mitigation
methods during the slice operation is discussed in the Intra-slice security
and Inter-slice security sections.

Decommissioning.:

The slice would be eliminated in this phase when its mission is completed.
Inaccurate deactivation of slices might lead to the exposure of sensitive data
and malicious resource consumption. Destruction of the archive and the de-
allocation of network resources of the decommissioned slice should be
performed.

Other aspects of threats concern the interactions between different slices or
between entities within the same slice (e.g., resources, operation,
management).

####  3.3.2. Intra-slice security

This section discusses the threats that target and only influence a specific
slice. These kinds of threats include unauthorized access to slice resources,
attacks targeting the interconnection between slices, and the exploitation of
slice resources. End-to-end security of all communications must be considered
to defend against these kinds of attacks. Primary and secondary authentication
of the tenants should be adequately designed.

####  3.3.3. Inter-slice security

This section discusses the threats that target and influence other slices
running on the same physical resource. For example, a performance attack could
occupy all the resources on one physical infrastructure and the attacker could
execute other types of attack in the meantime due to a lack of services for
important security processes. Unauthorized access to sensitive data from a
low-security slice is also possible. The guarantee of slice isolation,
especially isolation from the slice with sensitive data, should be guaranteed
to prevent these attacks because security is only as strong as the weakest
link. Secure management is also recommended.

###  3.4. Machine Learning

Figure 3. ML workflow and vulnerabilities

Figure 3 shows a simplified ML workflow. The basic idea of ML is to take in a
large volume of data and their true labels in the training phase. The model
would adjust its parameters to a state where the predicted results and the
true labels have minimum prediction error. Once the model is well-trained,
users can query the model with input data, the model will compute the
corresponding predicted result. ML prediction has been used in a wide range of
fields such as image classification, natural language processing, marketing,
etc., and gained considerable success. In recent years, the usage of ML in
wireless communication has started to gain researchers’ attention. 5G
architecture also incorporates ML to improve the performance of the network.
However, ML is highly susceptible to even simple adversarial attacks. Without
proper consideration, ML approaches may expose the network to new threats.

####  3.4.1. ML Threats

We follow the work of He, Yingzhe, et al. (He et al., 2020) and respectively
discuss the three aspects of ML threats.

Training data.:

As the concept of ML is to delve into the data and seek intelligent
information. The feasibility and performance of an ML model are highly
dependent on the training data set. Poisoning attack exploits this weakness by
maliciously inserting mislabelled data into the training data set, degrading
the quality of training data and further affecting the quality of the model.
This attack could make the ML model unusable, violating the availability and
integrity of the model. Since the training data is crucial for model training,
the collection and management of data are also valuable assets. Model
inversion attack allows attackers to restore information in the training data
from the prediction results. Information such as the data memberships (whether
the given data is in the training data set) and data properties could be
stolen with this attack, leaking trade secrets and directly causing a loss to
the operating company.

Trained model.:

The training process takes a lot of time and costs in data collecting,
storage, and computing, the well-trained model is highly valuable to a
company. Model extraction attack allows attackers to recover the structure,
parameters, decision boundaries, etc., of the ML models by querying the model
and duplicating an approximate model without the need to perform additional
training. The duplication of ML models will cause enormous damage to the model
training company assets or can be useful for further attack.

Inputs and results of predictions.:

In the prediction phase, the input data can also be exploited by the attackers
to affect the outcome of the prediction. Evasion attacks carefully modified
the input data to produce adversarial examples, the change is unperceived by
humans while the model will wrongly classify the adversarial example.

The threats in the ML systems above expose O-RAN to new weaknesses. These
attacks not only harm the operating company’s interest but also make the
wireless network unusable. Performing poisoning attacks on the intelligent
components in O-RAN could lower the efficiency in network managing, or lead to
loss of availability. The model inversion attack may also leak users’ privacy.
The evasion attack can not only be performed to affect a victim’s prediction
result, a misbehaving UE can also intentionally input adversarial examples to
fool the ML models whose functionality relies on UEs’ input. In the traffic
steering use case, the attacker can fool the ML model, making its own
connection handover frequently between multiple cells (Bitton et al., 2022).
This could lead to the exhaustion of resources because the overhead of
handover is expensive.

####  3.4.2. mitigations

There are several techniques to make the ML model more tolerant of malicious
inputs. Adversarial training includes the adversarial examples in the training
data set, making the ML model gain inherent resistance against adversarial
inputs. Defensive distillation uses the knowledge extracted from the ML model
to improve its own resistance against adversarial examples (Papernot et al.,
2015). Concept drift continuously monitors the performance of ML model to
detect adversarial examples while adversarial detection focuses on the input
of the ML model.

Collecting users’ data for model training may also be a privacy issue,
cryptographic methods such as homomorphic encryption, multiparty computation,
and zero-knowledge argument schemes can be applied to protect privacy while
still allowing the computation for learning. Differential privacy protects the
privacy by adding noises in the prediction result to protect training data.

##  4\. Related work

DongHyun Je et al. (Je et al., 2021) analyzed the potential new threats due to
the introduction of new technologies in wireless communication systems from an
extensive aspect of 6G. Other groups focus on the ORAN concept and provide
more specific research. Dudu Mimran et al. (Mimran et al., 2022) proposed an
ontology and developed a process for ORAN security evaluation. A threat
analysis of ORAN based on the proposed process is conducted. However, their
research lacks a survey of possible solutions corresponding to the identified
threats. CT Shen et al. (Shen et al., 2022) studied several pieces of research
about information securities in 5G and ORAN yet they did not present a
detailed analysis of the reviewed cases from a higher point of view.

##  5\. Conclusion and future work

In this project, four main attack surfaces of O-RAN, including (1)
Architectural openness, (2) Cloud and Virtualization, (3) Network slicing, and
(4) Machine Learning are identified and followed by further study of the
possible solutions to the corresponding threats. Several examples of existing
attacks and mitigation methods are also discussed. Our results could serve as
a baseline and a guideline for operators or other implementers in building
cyber defense strategies for O-RAN networks. Although we agree that a vast
attack surface is created along with the introduction of the openness concept
in O-RAN systems. We believe that, through careful implementation of the
concept of security by design in every fragment and framework of O-RAN since
its early-development stage, O-RAN would evolve into a robust and secure
system in the long term. The enhancement of the security level of O-RAN still
needs the contribution of the community. More thorough and more practical
research of attacks and certified defense methods should be conducted in the
future. Continuous updates and re-evaluation according to the corresponding
specifications updated of O-RAN are also needed.

## References

  * (1)
  * Atya et al. (2017) Ahmed Atya, Azeem Aqil, Karim Khalil, Zhiyun Qian, Srikanth V. Krishnamurthy, and Thomas F. La Porta. 2017.  Stalling Live Migrations on the Cloud. In _11th USENIX Workshop on Offensive Technologies (WOOT 17)_. USENIX Association, Vancouver, BC.  https://www.usenix.org/conference/woot17/workshop-program/presentation/atya
  * Bitton et al. (2022) Ron Bitton, Dan Avraham, Eitan Klevansky, Dudu Mimran, Oleg Brodt, Heiko Lehmann, Yuval Elovici, and Asaf Shabtai. 2022\.  Adversarial Machine Learning Threat Analysis in Open Radio Access Networks.  https://doi.org/10.48550/ARXIV.2201.06093
  * Continella et al. (2018) Andrea Continella, Mario Polino, Marcello Pogliani, and Stefano Zanero. 2018.  There’s a Hole in That Bucket! A Large-Scale Analysis of Misconfigured S3 Buckets. In _Proceedings of the 34th Annual Computer Security Applications Conference_ (San Juan, PR, USA) _(ACSAC ’18)_. Association for Computing Machinery, New York, NY, USA, 702–711.  https://doi.org/10.1145/3274694.3274736
  * Grobauer et al. (2010) Bernd Grobauer, Tobias Walloschek, and Elmar Stocker. 2010\.  Understanding cloud computing vulnerabilities.  _IEEE Security & privacy_ 9, 2 (2010), 50–57.
  * He et al. (2020) Yingzhe He, Guozhu Meng, Kai Chen, Xingbo Hu, and Jinwen He. 2020.  Towards security threats of deep learning systems: A survey.  _IEEE Transactions on Software Engineering_ (2020).
  * Je et al. (2021) DongHyun Je, Jungsoo Jung, and Sunghyun Choi. 2021\.  Toward 6G Security: Technology Trends, Threats, and Solutions.  _IEEE Communications Standards Magazine_ 5, 3 (2021), 64–71.
  * Klement et al. (2022) Felix Klement, Stefan Katzenbeisser, Vincent Ulitzsch, Juliane Krämer, Slawomir Stanczak, Zoran Utkovski, Igor Bjelakovic, and Gerhard Wunder. 2022.  Open or not open: Are conventional radio access networks more secure and trustworthy than Open-RAN?  _arXiv preprint arXiv:2204.12227_ (2022).
  * Mimran et al. (2022) Dudu Mimran, Ron Bitton, Yehonatan Kfir, Eitan Klevansky, Oleg Brodt, Heiko Lehmann, Yuval Elovici, and Asaf Shabtai. 2022\.  Evaluating the Security of Open Radio Access Networks.  _CoRR_ abs/2201.06080 (2022).  arXiv:2201.06080 https://arxiv.org/abs/2201.06080
  * Olimid and Nencioni (2020) Ruxandra F Olimid and Gianfranco Nencioni. 2020.  5G network slicing: A security overview.  _IEEE Access_ 8 (2020), 99999–100009.
  * Papernot et al. (2015) Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. 2015.  Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks.  https://doi.org/10.48550/ARXIV.1511.04508
  * Shen et al. (2022) CT Shen, YY Xiao, YW Ma, JL Chen, Cheng-Mou Chiang, SJ Chen, and YC Pan. 2022.  Security Threat Analysis and Treatment Strategy for ORAN. In _2022 24th International Conference on Advanced Communication Technology (ICACT)_. IEEE, 417–422.
  * Wei et al. (2009) Jinpeng Wei, Xiaolan Zhang, Glenn Ammons, Vasanth Bala, and Peng Ning. 2009.  Managing security of virtual machine images in a cloud environment. In _Proceedings of the 2009 ACM workshop on Cloud computing security_. 91–96.
  * Yarom and Falkner (2014) Yuval Yarom and Katrina Falkner. 2014.  FLUSH+RELOAD: A High Resolution, Low Noise, L3 Cache Side-Channel Attack. In _23rd USENIX Security Symposium (USENIX Security 14)_. USENIX Association, San Diego, CA, 719–732.  https://www.usenix.org/conference/usenixsecurity14/technical-sessions/presentation/yarom
  * Zhou et al. (2010) Wu Zhou, Peng Ning, Xiaolan Zhang, Glenn Ammons, Ruowen Wang, and Vasanth Bala. 2010\.  Always up-to-date: scalable offline patching of vm images in a compute cloud. In _Proceedings of the 26th Annual Computer Security Applications Conference_. 377–386.