[0.55in]0pt1.8em

# PGB: A PubMed Graph Benchmark for Heterogeneous Network Representation
Learning

Eric W Lee  Joyce C Ho

###### Abstract

There has been a rapid growth in biomedical literature, yet capturing the
heterogeneity of the bibliographic information of these articles remains
relatively understudied. Although graph mining research via heterogeneous
graph neural networks has taken center stage, it remains unclear whether these
approaches capture the heterogeneity of the PubMed database, a vast digital
repository containing over 33 million articles. We introduce PubMed Graph
Benchmark (PGB), a new benchmark dataset for evaluating heterogeneous graph
embeddings for biomedical literature. PGB is one of the largest heterogeneous
networks to date and consists of 30 million English articles. The benchmark
contains rich metadata including abstract, authors, citations, MeSH terms,
MeSH hierarchy, and some other information. The benchmark contains an
evaluation task of 21 systematic reviews topics from 3 different datasets. In
PGB, we aggregate the metadata associated with the biomedical articles from
PubMed into a unified source and make the benchmark publicly available for any
future works.

##  1 Introduction

Academic graphs generated from bibliographic data serve as an essential data
source across many different fields. Analysis of such graphs can be used for
personalized article recommendations [48], retrieval of relevant articles [6],
understanding of trends in the field [2, 17], measuring academic influence and
novelty [54], and identifying relevant academic communities [12]. PubMed is an
example of an academic graph that contains over 33 million citations and
abstracts of literature related to biomedicine and health fields as well as
related disciplines such as life sciences, behavioral sciences, chemical
sciences, and bioengineering [11]. PubMed articles have been used to perform
numerous systematic reviews (SR) [16, 23, 42], evaluate biological processes
[27], identify protein-protein interactions [24], curate genes [46], and
extract biological networks [49]. To date, much of the work on PubMed
literature has focused on mining the text. However, the rich citation
structure can be utilized to automate the SR process and provide better
representation than their textual counterparts [28].

(a) PubMed article

(b) MeSH hierarchy

Figure 1: Example of PubMed Article (a) and the partial MeSH hierarchy (b)
that is associated with the article. For an article (a), the PubMed database
contains pmid, title, abstract, list of articles cited by, publication types,
list of MeSH terms, and list of substances (chemicals). The MeSh hierarchy (b)
shows the categorization of the MeSH terms to a broader concept.

For analysis of the citation graphs, low-dimensional representations, or
embeddings, of the graph’s nodes, serve as the fundamental analysis tool [10,
41, 51]. The idea is to learn a compact representation of each node that
preserves the structural information and properties of the graph. The graph
embedding can then be used for a variety of downstream tasks such as node
retrieval/recommendation [56], node classification [44], node clustering [36],
and link prediction [47]. In recent years, graph neural networks (GNNs) [5,
18, 26, 30, 38] have become pervasive throughout graph mining research due to
their impressive performances across various tasks. However, GNNs has been
predominantly studied in the homogeneous network setting, where there is only
a single node type and link-type [41, 50, 51]. Yet, an academic graph can
contain multiple objects (nodes) and link types (edges) which include author
information, venue information, subject, and terms. As such, researchers are
exploring extending GNNs to the heterogeneous information network (HIN) with
multiple node types and edges, each with potentially different side
information. Heterogeneous GNNs have been proposed to incorporate the node and
edge types [26, 45, 49, 52, 53].

Despite the development of such models, a recent paper demonstrated that in
fact the results generated by these state-of-the-art heterogeneous GNNs were
merely a mirage [32]. The lack of consistent experimental setup and
preprocessing of the data led to widely varying results. In some cases,
vanilla GNN models were better than their heterogeneous GNN counterparts. As
such, there have been recent developments toward developing benchmark graph
datasets. OGB was developed as a large-scale benchmark for a broad range of
graph machine learning tasks [22]. It encompasses various domains, including
bibliographic data (i.e., arXiv and Microsoft Academic Graph (MAG) [43]). The
ogbn-arxiv and obgn-papers100M dataset focus on the simple citation network
using a homogeneous network representation whereas the obgn-mag extracts a
heterogeneous information network from MAG and contains four different node
types (papers, authors, institutions, and topics) along with their links. Yet
OGB is more geared toward evaluating homogeneous GNNs rather than the
heterogeneous GNNs.

HGB [32] was developed as a new medium-scale benchmark dataset that spanned 11
heterogeneous networks including three bibliographic datasets: (1) DBLP, a
citation network of computer science that contains four nodes (authors,
papers, terms, and venues); (2) ACM, a citation network that spans papers from
5 conference venues that contains three nodes (authors, papers, and subjects);
and (3) PubMed, a network of genes, diseases, chemicals, and species extracted
from a large biomedical literature database using Named Entity Recognition
software. The experimental studies of GNN models on HGB demonstrated
substantially better performance on DBLP and ACM datasets than on the PubMed
dataset. This may not be surprising as many general domain text mining models
fail to generalize to biomedical literature [3, 15, 16], motivating a more
extensive study.

Unlike existing heterogeneous bibliographic benchmarks, PubMed contains rich
metadata beyond the citation structure. Figure 1(a) shows an example of a
PubMed article111Article can be found at
https://pubmed.ncbi.nlm.nih.gov/12429942/.. In addition to the author, venue,
and citation information that is commonly found in most bibliographic data,
each PubMed article contains data regarding the Chemical Substances within the
article, the type of article which characterizes the nature of the information
or the type of research support received, and Medical Subject Headings (MeSH)
terms which identify the broader concepts in the data. The categorical
information of chemical substances and publication types are not found in
DBLP, ACM, or MAG. Moreover, there are over 30,000 terms in the MeSH
vocabulary. Furthermore, the terms follow a hierarchical
taxonomy222https://www.nlm.nih.gov/mesh/intro_trees.html (see Figure 1(b) for
an example MeSH tree for some terms in the example), yet also have the unique
property that a term can belong to one or more trees. We note ACM has the
Computing Classification System (CCS) which is a hierarchical ontology but
does not allow a term to belong to multiple trees. Capturing this hierarchical
structure can potentially improve the representation; however, the data in
PubMed is incomplete as it does not perform author disambiguation.

We present PGB, a new benchmark dataset for evaluating heterogeneous graph
embeddings for biomedical literature. PGB is comprised of over 30 million
articles from the PubMed database. It leverages the citations and author
disambiguation capabilities of Semantic Scholar while also layering in the
rich metadata that is offered in PubMed including the MeSH Terms, Chemical
List, and Publication Type. PGB layers in the MeSH hierarchical structure for
all the terms associated with the article, unlike any of the existing graph
datasets. PGB also provides an evaluation task on biomedical abstract
screening on 21 SR topics from 3 different datasets. Our evaluation task is
different than the existing node-level and edge-level tasks provided in OGB
and HGB in that the same node can have different labels depending on the SR
content. By providing a high-quality and large-scale heterogeneous
bibliographic network with three different graph tasks and their associated
evaluation metrics, we can measure progress in a consistent and reproducible
fashion.

In addition to building PGB, we also perform extensive benchmark experiments
for the dataset using current state-of-the-art graph embedding methods
including 2 homogeneous GNNs and 3 heterogeneous graph embedding models.
Through the experiments, we highlight two research challenges associated with
generating embeddings for PubMed. First is the lack of scalability for many
existing GNNs as many of the state-of-the-art models were incapable of
processing the entire graph. Second is the inability to capture the
heterogeneity of the network as the models fail to achieve comparable
performance to other bibliographic networks. Our experimental results
illustrate the need for developing new scalable heterogeneous GNN models that
are capable of handling the rich metadata in PubMed, which is unlike any
existing bibliographic data.

Table 1: Comparison of existing bibliographic datasets with N denoting nodes, NT denoting node types, ET denoting edge types and HIER denoting a hierarchical structure on at least one of the nodes. | # N | # NT | # ET | HIER
---|---|---|---|---
PGB | 54,974,182 | 6 | 7 | ✓
ogbn-mag | 1,939,743 | 4 | 4 | ✗
ogbn-arxiv | 169,343 | 1 | 1 | ✗
ogbn-papers100M | 111,059,956 | 1 | 1 | ✗
HGB-DBLP | 26,128 | 4 | 6 | ✗
HGB-ACM | 10,942 | 4 | 6 | ✗
HGB-Freebase | 180,098 | 8 | 36 | ✗

##  2 Background

Bibliographic data is used in various tasks, for example, word embedding using
the title and abstract, network embedding using the citation, and author
network. Thus, many works have worked on constructing a benchmark for
bibliographic data such as OGB [22], HGB [32], and S2ORC [31]. Here we briefly
describe the three related academic paper benchmark datasets and their
limitations.

OGB [22] is a large-scale benchmark for graph machine learning tasks. It
encompasses a variety of domains such as social networks, biological networks,
molecular graphs, and knowledge graphs. OGB also has bibliographic data, for
example, ogbn-arxiv and ogbn-papers100M are citation networks that are
extracted from arxiv and MAG, respectively. Notably, both of these citation
networks are homogeneous networks with paper nodes and links that represent
the citation. Unlike ogbn-arxiv and ogbn-papers100M, OGB also has a
heterogeneous academic network, ogbn-mag, which is extracted from MAG. The
ogbn-mag dataset contains 4 different node types (i.e., papers, authors,
institutions, and topics) along with their relations. However, OGB focuses on
benchmarking graph machine learning methods on the large-scale homogeneous
network.

HGB [32] provided 11 medium-scale graph benchmark datasets for node
classification, link prediction, and knowledge-aware recommendation. For node
classification, it contains DBLP, IMDB, ACM, and Freebase [8] datasets, and
for link prediction, Amazon, LastFM, and PubMed datasets. For the PubMed
benchmark, it is the subset of a previously generated network of genes,
diseases, chemicals, and species filtered by domain experts [49]. Furthermore,
unlike PGB, the PubMed dataset does not reflect the bibliographic data
directly. Instead, for HGB, DBLP and ACM datasets serve as the lone benchmarks
for the bibliographic network. However, both datasets lack metadata that can
be helpful for learning node embeddings. Additionally, the benchmarks assume
each node has a single label, whereas labels can change depending on the
context.

The S2ORC [31] corpus is a large-scale academic paper corpus that is
constructed using the data from the Semantic Scholar literature corpus [1].
Articles in Semantic Scholar are derived from numerous sources which are
obtained directly from publishers such as MAG, arXiv, PubMed, and crawled from
the open Internet. Semantic Scholar clusters these papers based on title
similarity and DOI overlap, resulting in an initial set of approximately 200M
paper clusters. Using the Semantic Scholar literature corpus, S2ORC aggregated
the metadata of articles and cleaned the data to select canonical metadata
using external sources such as IEEE and DBLP. Although S2ORC contains
biomedical literature, it mainly focuses on the common metadata that exists
across all the articles. Since publication types, MeSH terms, and chemical
substances are only present in biomedical literature, such metadata is not
included in the dataset. Thus, developing embeddings that reflect the
heterogeneity of the PubMed database requires additional preprocessing. Table
1 summarizes the statistics of the existing bibliographic datasets.

##  3 Benchmark Construction

In this section, we introduce the framework to construct PGB, shown in Figure
2.

###  3.1 Paper Collection

Figure 2: Framework overview of the PGB.

PGB is constructed based on the S2ORC corpus [31]. S2ORC corpus is based on
the Semantic Scholar database and contains more complete citation information
than the PubMed database. However, there exist cases where the abstract only
exists in the Semantic Scholar database but not in the PubMed database. Since
PGB targets the biomedical literature, we extract articles that contain a
PubMed ID (PMID) from the S2ORC corpus.

####  3.1.1 Metadata Extraction

The S2ORC corpus only contains basic metadata of each PubMed paper (e.g.,
title, abstract, authors, year, and venue). In biomedical literature, unlike
general academic articles, there is important metadata that can serve an
important role such as Medical Subject Headings (MeSH) terms and publication
types. Additional partial information can be found in PubMed (see Figure 1).
To extract more detailed information, we use the Entrez
API333https://www.ncbi.nlm.nih.gov/books/NBK25501/ to retrieve the XML format
of the metadata of the biomedical literature and extract more detailed
information related to the article.

####  3.1.2 Extraction from PubMed

The PubMed metadata contains “Chemical List”, “Publication Type”, and “MeSH
Terms”. The chemical list provides the registry number of specific chemical
substances assigned by the Chemical Abstracts Service and the name of the
chemical substances. The publication type identifies the type of article
indexed for MEDLINE and characterizes the nature of the information, how it is
conveyed, and the type of research support received. For example, an article
can have a publication type of _Review_ , _Letter_ , _Retracted Publication_ ,
_Research Support, N.I.H._ , or _Clinical Conference_. Finally, MeSH terms are
used to characterize the content of the articles. MeSH terms are recorded with
the information whether they are the major topic or not. The major MeSH terms
denote that those are the most significant topics of the paper. The non-major
MeSH terms are used to identify concepts that have also been discussed in the
item, but that are not the primary topics. Using the Entrez API with the
associated PMIDs of the articles, we retrieve and include the name of the
chemical substances, the publication type, and both major and minor MeSH
terms.

####  3.1.3 Citation Extraction and Cleaning

While the PubMed database contains rich information on biomedical literature,
it contains few information about the citations. However, S2ORC corpus
extracted the citations from the collected pdf or LaTeX files on top of the
Semantic Scholar literature corpus. Thus, to construct PGB, we used the
citation information from the S2ORC corpus. S2ORC corpus includes both in and
out citations which refer that whether the paper is cited by another paper or
the paper cites another paper. We convert all the Semantic Scholar IDs into
PMIDs and remove papers that are not included in the PubMed database444While
this can potentially harm or bias the embedding, we did this to maintain
consistency in the article information in PGB.. In addition, we extract
citations from the Entrez API to add any missing citations that do not exist
in the Semantic Scholar database but exist in the PubMed database.

####  3.1.4 MeSH Terms Hierarchy

One important feature of MeSH terms is the hierarchical structure between MeSH
terms. MeSH terms can be categorized into broader MeSH terms that support the
categorization of the articles, as depicted in Figure 1(b). In the MeSH
hierarchy, the categories of different levels reveal the similarity at
different granularities, thus knowing the hierarchy can play an important role
in identifying similar articles. As shown in Figure 1(b), MeSH terms that are
assigned to the article can share the same parents or can be in a different
sub-tree. However, the Entrez API does not include the MeSH terms hierarchy.
Thus, we also extract the MeSH terms hierarchy
dataset555https://www.nlm.nih.gov/databases/download/mesh.html to identify the
position of the MeSH terms associated with each article.

The MeSH terms hierarchy dataset only contains the MeSH terms and the
associated tree numbers shown in Figure 1(b). For example, the MeSH terms
“Chromosome Segregation” (G04.144.220.
220.625) and “Mitosis” (G04.144.220.220.781) do not reveal the hierarchical
structure directly. Thus, retrieving the hierarchy information for each
article requires an additional step. To avoid the tedious process of
retrieving the MeSH hierarchy, we extract the tree number of each MeSH term
and assign it as a sub-field. When comparing two articles, if they do not have
the same MeSH terms but MeSH terms with the same parents (or within the same
sub-tree), then the two articles are potentially closely related.

##  4 PubMed Graph Benchmark (PGB)

Table 2: List of metadata included in the PGB, and the field name and the type in JSON format. Field Description | Field name | Field type | Explanation
---|---|---|---
PMID | pmid | str | Pubmed ID
Title | title | str | Title of the paper
Abstract | abstract | str | Abstract of the paper
Authors | authors | List[Dict] | list of authors
Year | year | int | Published year
Venue | venue | str | Venue of the paper
Publication Type | publication_type | str | Type of the article
Chemical List | chemicals | List | Name of the chemical substances
MeSH Terms | mesh | List[Dict] | List of MeSH terms
Inbound Citation | in_citation | List | List of PMID that cites the paper
Outbound Citation | out_citation | List | References of the paper
Table 3: Statistics of PGB. Name | Statistics
---|---
# of Articles | 30,872,730
# of Authors | 22,101,795
# of Articles with MeSH terms | 29,953,246
# of Articles with chemical list | 25,672,734
# of Articles with publication type | 27,185,396
# of Articles with inbound citations | 17,667,464
# of Articles with outbound citations | 8,080,795
Avg. Mesh terms per article | 10.70
Avg. chemical list per article | 3,85
Avg. inbound citations per article | 12.19
Avg. outbound citations per article | 25.83

The PGB contains 11 metadata which is shown in Table 2. The benchmark is
released and it is publicly available on Zenodo666The data is available at
https://zenodo.org/record/6406776#.YqrOKnbMKUk and the code associated with
its construction at https://github.com/ewhlee/PGB.. Table 2 also shows the
field name and the field type that are used to store PGB. The “authors” field
contains 4 subfields, “first”, “middle”, “last”, and “suffix”. For the
chemical list and the MeSH terms, we exclude the ids and only included the
name because the name itself is already unique. For the MeSH terms, we use 3
subfields to convey which MeSH terms are major or minor, The subfield “name”
refers to the name of the MeSH terms, the subfield “is_major” is set to a
true/false value to identify the major MeSH terms, and the subfield
“tree_number” is the MeSH hierarchy information. There can exist multiple
major MeSH terms for each article. We also included fields for validating
whether the inbound and outbound citation exists in the benchmark. The fields
are named “has_outbound_citations” and “has_inbound_citations”, and the value
is set to be either true or false. This helps users easily identify the
presence of citation information without parsing the list of citations.

The statistics of PGB are shown in Table 3. It contains 30,872,730 biomedical
literature and 221,01,795 authors. All the articles have PMID, title,
abstract, year, venue, and at least one author. However, there exist cases in
which any one of the fields is missing. The case that any metadata is missing
means that the metadata does not exist in both PubMed and S2ORC corpus. 42% of
articles do not have an inbound citation, and 73% of articles do not have
outbound citations. Table 3 also shows the average number of MeSH terms,
chemicals, and inbound and outbound citations. The average is computed on the
articles that contain the associated metadata. The size of PGB is about 60GB.

The metadata in PGB contains all necessary information to construct a
homogeneous or heterogeneous network. There are 5 node types (_P_ aper, _A_
uthor, _M_ eSH terms, _V_ enue, and publication _T_ ype) and 6 edge types
(P-P, P-A, A-A, P-M, P-V, P-T). The MeSH hierarchy can be added using another
edge type M-M. The constructed heterogeneous network can be used for node
classification to determine the topic of articles, link prediction for
citation recommendation, and SR for abstract/full-text screening. To
illustrate the usage of PGB, we perform experiments using both homogeneous and
heterogeneous network embedding models and evaluate the embedding for
identifying articles for an SR.

###  4.1 PGB Format

As described in Table 2, PGB contains 13 fields, and the description of each
field is explained in Section 4. The benchmark is released and publicly
available on Zenodo at https://zenodo.org/record/6406776#.YqrOKnbMKUk. We also
open-source the code to reconstruct the benchmark at
https://github.com/ewhlee/PGB. We hope that by releasing the code publicly,
the community can contribute to the maintenance of the benchmark dataset
(i.e., updating the graph or adding new tasks). Due to the size of the
benchmark ($\sim$60GB), PGB is split into 10 partitions. Each partition is a
zip file in Zenodo.

Figure 3: A simplified illustration of the SR screening process using
“Statins” from Cohen dataset [16].

###  4.2 License Information

PGB is released under the CC BY-NC 4.0 license and for non-commercial use. PGB
is constructed using the PubMed Entrez API and S2ORC. S2ORC is non-commercial
use and released under the same license (CC BY-NC 4.0 license). The PubMed
Entrez API does not require a signed license agreement to download the
publicly accessible data. However, we note that the associated PubMed metadata
(i.e., MeSH terms, Chemical list, and publication type) in PGB may not reflect
the most current data available on PubMed. The data can be re-updated using
the Github repository assuming no major changes in the type and format of the
machine-readable data. The usage guidelines and registration for the API key
are detailed in the electronic book
chapter777https://www.ncbi.nlm.nih.gov/books/NBK25497/#chapter2.Usage_Guidelines_and_Requiremen.

##  5 Experiments

In this section, we discuss the experimental settings for the SR task. The
heterogeneous networks are constructed based on subsets of PGB. Any other
datasets that are used throughout the experiments are explained in this
section.

###  5.1 Evaluation Task

Systematic reviews (SRs) are essential knowledge translation tools focused on
bridging the research-to-practice gap across a wide range of domains. In
health research, SRs aims to identify, evaluate, and summarize the findings of
all individual studies (which typically describe clinical trial results)
relevant to a clinical question, thereby making the available evidence more
accessible [13, 19, 20]. SRs serve as the basis for evidence-based medicine.

Figure 3 shows an illustration of the SR screening process. The first step is
to retrieve the initial list of articles using the combination of keywords and
MeSH terms related to the topic (3465 articles from the figure). Once the
initial list is retrieved, then reviewers go over the title and abstract
screening to get rid of irrelevant articles (95% of the articles). Finally,
the reviewers go over the full-text for articles passing the title and
abstract screening phase to collect the relevant articles to the topic (2.45%
of articles). However, the broad searches yield imprecise search results
(e.g., ¡2% relevant documents) and result in a labor-intensive process.
Current estimates for conducting an SR is 67 weeks from registration to
publication [9]. Clearly, this process is neither unsustainable or scalable,
especially given the exponential growth of biomedical literature [4].

Table 4: Statistics of all datasets used. Abs refers to the number of articles passing the abstract triage statuses and % shows the percentage. SR | Abs | Total | %
---|---|---|---
Cohen-ACEInhibitors | 183 | 2544 | 7.23
Cohen-ADHD | 84 | 851 | 9.87
Cohen-Antihistamines | 92 | 310 | 29.67
Cohen-AtypicalAntipsychotics | 363 | 1120 | 32.41
Cohen-BetaBlockers | 302 | 2072 | 14.57
Cohen-CalciumChannelBlocker | 279 | 1218 | 22.90
Cohen-Estrogens | 80 | 368 | 21.74
Cohen-NSAIDs | 88 | 393 | 22.39
Cohen-Opioids | 48 | 1915 | 2.51
Cohen-OralHypoglycemics | 139 | 503 | 27.63
Cohen-ProtonPumpInhibitors | 238 | 1333 | 17.85
Cohen-SkeletalMuscleRelaxants | 34 | 1643 | 2.56
Cohen-Statins | 173 | 3465 | 4.99
Cohen-Triptans | 218 | 671 | 32.48
Cohen-UrinaryIncontinence | 78 | 327 | 23.85
SWIFT-Transgenerational | 765 | 48638 | 1.57
SWIFT-PFOS-PFOA | 95 | 6331 | 1.50
SWIFT-BPA | 111 | 7700 | 1.44
CLEF-Prognosis-CD012661 | 192 | 3367 | 5.70
CLEF-DTA-CD008803 | 99 | 5220 | 1.89
CLEF-Intervention-CD005139 | 112 | 5392 | 2.07

###  5.2 Dataset

For the SR task, PGB contains three different datasets: the popular, and
publicly available dataset provided by Cohen _et al_. [16], SWIFT-Review [21]
and CLEFT-TAR [25].

The Cohen _et al_. dataset888https://dmice.ohsu.edu/cohenaa/systematic-drug-
class-review-data.html was the first SR dataset publicly released and was
based on comparing 15 classes of drugs to treat specific conditions [16]. The
evidence reports were completed by three evidence-based practice centers in
Oregon, Southern California, and the Research Triangle Institute / University
of North Carolina. The staff at these centers created search queries to
identify randomized controlled trials by combining the health conditions and
interventions. Another set of datasets is 3 sets provided by SWIFT-Review
[21].999https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-016-0263-z#Sec30
The dataset was generated by the National Toxicology Program (NTP) Office of
Health Assessment and Translation (OHAT). The last set is provided by CLEF
2019 e-Health TAR Lab [25] (Task 2)101010https://github.com/CLEF-TAR/tar which
focuses on retrieving relevant studies from during the abstract and title
screening phase of conducting an SR. From the CLEF-TAR dataset, we randomly
selected 3 sets which are the CD012661 topic from Prognosis, the CD008803
topic from DTA, and the CD005139 topic from Intervention. The title of the
topic CD012661 is “Development of type 2 diabetes mellitus in people with
intermediate hyperglycemia” [37]. The title of the topic CD008803 is “Optic
nerve head and fibre layer imaging for diagnosing glaucoma”[33]. The title of
the topic CD005139 is “Anti‐vascular endothelial growth factor for neovascular
age‐related macular degeneration” [39].

Each article, identified using the PMID, was triaged using a two-step process.
First, the abstract is reviewed to determine if it meets the inclusion
criteria of the SR. If the criteria are met, the full text of the article is
then reviewed to determine if the evidence should be summarized in the SR. PGB
targets the abstract screening process where most of the articles are
excluded. Table 4 summarizes the statistics for each SR topic. The abstract
(abs) denotes whether the article passed the title/abstract screening phase.
It is notable that the number of articles included after the abstract
screening varies from 1.44% to 32.48%, demonstrating a relatively large degree
of imbalance across the SR topics.

###  5.3 Evaluation Metrics

Several works also evaluate an SR task using the area under the receiver
operating curve (AUC) for predicting whether or not the abstract was screened
or not to report the results [15, 34]. Thus, we report AUC as well.

###  5.4 Baseline and Hyperparameters

We use the code and perform a parameter search around the neighborhood of
suggested parameters provided by the original paper. We briefly describe the
final parameters used and provide a link to the implementation. For each of
the implementations, we kept separate environment files to ensure that the
required Python packages were installed and the correct version as outlined in
the code.

  * •

SPECTER [14]: The SPECTER is an embedding model which learns the
representation of a document by computing the embeddings using a SciBERT model
[7] pre-trained on relatedness signals derived from the citation graph. We use
the embeddings for SPECTER provided by Semantic Scholar
API.111111https://www.semanticscholar.org/product/api The Semantic Scholar API
allows a paper search by using the PubMed ID to retrieve the Semantic Scholar
ID. Using this Semantic Scholar ID, we retrieve the SPECTER embeddings of each
document.

  * •

LINE [40]: LINE is a conventional homogeneous network embedding method that
uses first- and second-proximity. LINE uses the joint probability between two
nodes.121212https://github.com/DeepGraphLearning/graphvite. We set the number
of dimensions to 128 for both first- and second-proximity. The final embedding
is the concatenation of 2 proximities. As LINE is an unsupervised model, we
add a soft-max layer on top of the final embeddings.

  * •

GCN [26]: GCN is a graph convolutional network embedding model designed for a
homogeneous network.131313https://github.com/tkipf/gcn. GCN is trained in a
supervised setting using the SR task. We use the 500-dimension TF-IDF weighted
word vector provided by Namata _et al._ [35] as the node feature.

  * •

HAKE [55]: HAKE is a hierarchical-aware knowledge graph embedding model which
is not a GNN-based model but a translational distance model which describes
relations as translations from one node to the
other.141414https://github.com/MIRALab-USTC/KGE-HAKE. It uses radial
coordinates to embedding entities at different levels of the hierarchy and
uses angular coordinates to distinguish entities at the same level of the
hierarchy. HAKE uses the link-prediction task to learn the embeddings, and
thus it is an unsupervised model. For the supervised tasks, we add a soft-max
layer on top of the embeddings. We try [500, 1000] for the dimension size and
select 1000 as the validated parameter.

  * •

GAHNE [29]: GAHNE is a model to learn representations for
HIN.151515https:github.com/seanlxh/GAHNE. It converts the network into a
series of homogeneous sub-networks to capture semantic information. An
aggregation mechanism then fuses the sub-networks with supplemental
information from the whole network. Using the validation set, we process a
grid search using [0.01, 0.005, 0.001] for the learning rate, [0.0005, 0.001]
for L2 penalty, and [64, 128, 256] for the dimension size. The validated
parameters we used are a learning rate of 0.005, dropout of 0.5, L2 penalty of
0.001, and a dimension of 128. GAHNE is a supervised model and the model is
trained using the labels from SR topics.

  * •

ie-HGCN [52]: ie-HGCN is a GCN-based HIN embedding model that evaluates all
possible meta-paths and projects the representations of different types of
neighbor objects into a common semantic space using object- and type-level
aggregation.161616https://github.com/kepsail/ie-HGCN/. We use the supervised,
cross-entropy loss to learn the weights of the model. We set the number of
layers to be 5 and using the validation set, we tried [(128, 64, 32, 16),
(156, 128, 64, 32)] as the dimension size. The validated parameters used in
the results are 5 layers, with the dimensions of input, 128, 64, 32, and 16.
We use the same node feature as GCN.

LINE and HAKE are unsupervised models, and the other 3 GNN baselines are semi-
supervised models. For the homogeneous network, we only use the citation
information to construct the network. For HAKE, we use 3 node types (_P_ aper,
_M_ eSH terms, publication _T_ ype) with 4 edge types (P-P, P-M, M-M, P-T).
For the other 2 heterogeneous networks, we use 4 node types (_P_ aper, _V_
enue, _M_ eSH terms, publication _T_ ype) with 4 edge types (P-P, P-v, P-M,
and P-T)

###  5.5 Experimental Design

For all the SR topics, we use a subset of PGB as the baseline methods were
unable to scale when using the entire dataset. For each dataset (Cohen, SWIFT,
and CLEF-TAR), we start with articles in the dataset and trace the inbound and
outbound citations up to 2-hops to construct the subnetwork of PGB
(PGB_small), with about 1.2M, 3.4M, and 1.8M articles, respectively to Cohen,
SWIFT, and CLEF-TAR. In each subnetwork, we randomly split the train-
validation-test for the prediction task as 50%-25%-25% with 3 trials. The
reported AUC score is the average of 3 trials.

For training the network embedding, we use the provided codes for all the
baselines stated in the original paper. The validation set is used to tune the
hyperparameters for GAHNE and ie-HGCN. For all the baselines, we used a
machine with 1 Nvidia GeForce GTX 1080Ti and 11GB GPU memory.

Table 5: Performance results (AUC score) for the SR tasks. The AUC score is the average of 3 trials. The best score for each SR is bolded and the second highest is underlined. SR Topic | SPECTER | LINE | GCN | HAKE | GAHNE | ie-HGCN
---|---|---|---|---|---|---
Cohen-ACEInhibitors | 0.706 | 0.613 | 0.617 | 0.704 | 0.755 | 0.758
Cohen-ADHD | 0.578 | 0.585 | 0.598 | 0.586 | 0.639 | 0.643
Cohen-Antihistamines | 0.533 | 0.532 | 0.544 | 0.525 | 0.576 | 0.573
Cohen-AtypicalAntipsychotics | 0.661 | 0.589 | 0.599 | 0.673 | 0.723 | 0.72
Cohen-BetaBlockers | 0.716 | 0.617 | 0.625 | 0.714 | 0.766 | 0.761
Cohen-CalciumChannelBlockers | 0.609 | 0.624 | 0.636 | 0.617 | 0.668 | 0.665
Cohen-Estrogens | 0.653 | 0.569 | 0.577 | 0.66 | 0.716 | 0.712
Cohen-NSAIDS | 0.722 | 0.624 | 0.631 | 0.728 | 0.783 | 0.779
Cohen-Opiods | 0.709 | 0.625 | 0.635 | 0.709 | 0.759 | 0.764
Cohen-OralHypoglycemics | 0.528 | 0.547 | 0.553 | 0.526 | 0.58 | 0.576
Cohen-ProtonPumpInhibitors | 0.701 | 0.63 | 0.641 | 0.714 | 0.769 | 0.766
Cohen-SkeletalMuscleRelaxants | 0.712 | 0.631 | 0.643 | 0.716 | 0.762 | 0.769
Cohen-Statins | 0.693 | 0.597 | 0.609 | 0.689 | 0.741 | 0.748
Cohen-Triptans | 0.697 | 0.608 | 0.62 | 0.698 | 0.743 | 0.739
Cohen-UrinaryIncontinence | 0.709 | 0.624 | 0.636 | 0.71 | 0.767 | 0.762
SWIFT-Transgenerational | 0.706 | 0.652 | 0.658 | 0.711 | 0.769 | 0.774
SWIFT-PFOS-PFOA | 0.71 | 0.657 | 0.663 | 0.707 | 0.755 | 0.76
SWIFT-BPA | 0.664 | 0.593 | 0.607 | 0.672 | 0.742 | 0.749
CLEF-Prognosis-CD012661 | 0.692 | 0.625 | 0.637 | 0.685 | 0.708 | 0.71
CLEF-DTA-CD008803 | 0.631 | 0.622 | 0.642 | 0.644 | 0.712 | 0.718
CLEF-Intervention-CD005139 | 0.677 | 0.648 | 0.661 | 0.682 | 0.723 | 0.721

##  6 Evaluations

In this section, we discuss the performance of the SR task where the model
predicts the articles that pass the biomedical abstract screening phase. All
the results shown in this section use the subnetwork of each dataset (Cohen,
SWIFT-Review, and CLEF-TAR). We compare the performance of 1 language model
and 5 network embedding models on SR. The performance is reported in Table 5
in the average of AUC scores of 3 trials for each SR task. The best results
are bolded and the second-best results are underlined.

As shown in the table, the results of the heterogeneous network embedding
models (HAKE, GAHNE, and ie-HGCN) significantly outperform the homogeneous
network embedding models (LINE and GCN). This suggests that not only the
citation information but also other node types (venue, MeSH terms, and
publication type) helps to improve the performance of the systematic review
task. GAHNE and ie-HGCN outperform HAKE as HAKE is an unsupervised model while
others are semi-supervised models. However, by comparing the performance with
the homogeneous model, HAKE shows the importance of the hierarchical
information (MeSH hierarchy). The performance between GAHNE and ie-HGCN is
similar. Based on the result, it seems that ie-HGCN performs better when there
are more articles excluded from the abstract screening phase. In other words,
ie-HGCN shows strength when fewer articles survived the abstract screening.
For example, the “SWIFT-BPA” dataset has a total of 7700 articles in the
beginning but only 111 articles (1.44%) are selected. While ie-HGCN performs
better in cases when fewer articles are selected, GAHNE performs better in
cases when more papers are selected. For example, “Cohen-
AtypicalAntipsychotics” starts with 1120 articles, and 363 articles (32%)
passed the screening.

By comparing with the language model (SPECTER), it shows similar results with
HAKE. In other words, SPECTER outperforms the homogeneous network embedding
models (LINE and GCN) which only uses the citation network but underperforms
the heterogeneous network embedding models (GAHNE and ie-HGCN). Although
SPECTER is based on the transformer language model, it uses the document-level
relatedness from the citation graph. Thus, this helps SPECTER to outperform
the homogeneous network embedding models. As same as HAKE, GAHNE, and ie-HGCN
outperform SPECTER as SPECTER is an unsupervised model, while GAHNE and ie-
HGCN are semi-supervised models. However, the performance of SPECTER shows the
importance of both the abstract and the citation graph in the systematic
review process.

Overall, the performance across three datasets, Cohen, SWIFT-Review, and CLEF-
TAR shows a similar trend as we explained. The heterogeneous network embedding
model outperforms homogeneous network embedding models. Two unsupervised
models, SPECTER and HAKE show similar performance. And ie-HGCN outperforms
GAHNE as fewer articles are selected after the abstract screening phase.

##  7 Discussions and Limitations

Besides the baselines which are all embedding models, we also use a majority
class classifier for the SR task. The AUC score for using the majority class
classifier is 0.5 for all SR tasks. However, the accuracy is different for
each SR task. For example, “ADHD” is 0.891, and “Statins” is 0.95. The AUC
score is less sensitive to class imbalance as the minority class will have a
strong impact on the AUC score, which illustrates the difficulty of predicting
class imbalance datasets. On the other hand, accuracy is not sensitive to
class imbalance, thus it can be very high as our result even the minority
class is not well predicted.

We note that our work has several limitations that could be improved upon.
First, citations that were not a part of PubMed were not included. This was
done because there would have been insufficient and inconsistent metadata with
the rest of the articles in PGB, as well as the need to find some identifier
for the articles themselves to denote that they were distinct from the PubMed
articles. Second, many of the existing baseline methods were unable to scale
to the entire dataset. Such models typically require considerable computing
resources which we, unfortunately, did not have access to. More extensive
methods can be included for future work to better understand the performance
of state-of-the-art models. Finally, we note that the SR dataset is based on
old evidence reports, while there has been a considerable number of recent SRs
on various topics. We are in the process of expanding this to include more SRs
by collaborating with people who have conducted recent SRs in public health,
nursing, and medicine.

##  8 Conclusion

In this paper, we discuss the importance of biomedical literature and the
necessary metadata fields for research. There exist many studies that use
heterogeneous network embedding for various tasks such as node classification,
link prediction, and SR. However, it is time-consuming to aggregate the
necessary information from multiple sources to capture the rich bibliographic
data. We construct PGB, a biomedical literature bibliographic dataset, that
contains 11 fields of metadata. The strength of PGB is not only that it
contains multiple types of nodes and edges, but also captures a hierarchical
structure on one of its nodes. Our experimental results illustrate that the
scalability and the capability of handling rich metadata, especially the
hierarchical structure, for existing graph embedding models still remain open
challenges.

## References

  * [1] Waleed Ammar, Dirk Groeneveld, Chandra Bhagavatula, Iz Beltagy, Miles Crawford, Doug Downey, Jason Dunkelberger, Ahmed Elgohary, Sergey Feldman, Vu Ha, Rodney Kinney, Sebastian Kohlmeier, Kyle Lo, Tyler Murray, Hsu-Han Ooi, Matthew Peters, Joanna Power, Sam Skjonsberg, Lucy Wang, Chris Wilhelm, Zheng Yuan, Madeleine van Zuylen, and Oren Etzioni.  Construction of the literature graph in semantic scholar.  In Proc. of NAACL-HLT, 2018.
  * [2] Kimitaka Asatani, Junichiro Mori, Masanao Ochi, and Ichiro Sakata.  Detecting trends in academic research from a citation network using network representation learning.  PloS one, (5), 2018.
  * [3] Alexandra Bannach-Brown, Piotr Przybyła, James Thomas, Andrew SC Rice, Sophia Ananiadou, Jing Liao, and Malcolm Robert Macleod.  Machine learning algorithms for systematic review: reducing workload in a preclinical review of animal studies and reducing human screening error.  Systematic reviews, (1), 2019.
  * [4] Hilda Bastian, Paul Glasziou, and Iain Chalmers.  Seventy-five trials and eleven systematic reviews a day: How will we ever keep up?  PLOS Medicine, (9), 2010.
  * [5] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al.  Relational inductive biases, deep learning, and graph networks.  ArXiv preprint, 2018.
  * [6] Joeran Beel, Bela Gipp, Stefan Langer, and Corinna Breitinger.  Paper recommender systems: a literature survey.  International Journal on Digital Libraries, (4), 2016.
  * [7] Iz Beltagy, Kyle Lo, and Arman Cohan.  Scibert: A pretrained language model for scientific text.  arXiv preprint arXiv:1903.10676, 2019.
  * [8] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor.  Freebase: a collaboratively created graph database for structuring human knowledge.  In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247–1250. ACM, 2008.
  * [9] Rohit Borah, Andrew W Brown, Patrice L Capers, and Kathryn A Kaiser.  Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the prospero registry.  BMJ open, (2), 2017.
  * [10] Hongyun Cai, Vincent W Zheng, and Kevin Chen-Chuan Chang.  A comprehensive survey of graph embedding: Problems, techniques, and applications.  TKDE, (9), 2018.
  * [11] Kathi Canese and Sarah Weis.  Pubmed: the bibliographic database.  The NCBI Handbook, 2013.
  * [12] Tanmoy Chakraborty and Abhijnan Chakraborty.  Overcite: finding overlapping communities in citation network.  In Advances in Social Networks Analysis and Mining 2013, ASONAM ’13, Niagara, ON, Canada - August 25 - 29, 2013, 2013.
  * [13] Iain Chalmers, Larry V Hedges, and Harris Cooper.  A brief history of research synthesis.  Evaluation & the Health Professions, 25(1):12–37, June 2016\.
  * [14] Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S. Weld.  SPECTER: Document-level Representation Learning using Citation-informed Transformers.  In ACL, 2020.
  * [15] Aaron M Cohen.  Optimizing feature representation for automated systematic review work prioritization.  In AMIA annual symposium proceedings. American Medical Informatics Association, 2008.
  * [16] Aaron M Cohen, William R Hersh, Kim Peterson, and Po-Yin Yen.  Reducing workload in systematic review preparation using automated citation classification.  Journal of the American Medical Informatics Association, (2), 2006\.
  * [17] Suhendry Effendy and Roland HC Yap.  Analysing trends in computer science research: A preliminary study using the microsoft academic graph.  In Proceedings of the 26th international conference on world wide web companion, 2017.
  * [18] Marco Gori, Gabriele Monfardini, and Franco Scarselli.  A new model for learning in graph domains.  In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005. IEEE, 2005.
  * [19] David Gough and Diana Elbourne.  Systematic research synthesis to inform policy, practice and democratic debate.  Social Policy and Society, 1(3):225–236, July 2002.
  * [20] David Gough, Sandy Oliver, and James Thomas.  An introduction to systematic reviews.  Sage, second edition, 2017.
  * [21] Brian E Howard, Jason Phillips, Kyle Miller, Arpit Tandon, Deepak Mav, Mihir R Shah, Stephanie Holmgren, Katherine E Pelch, Vickie Walker, Andrew A Rooney, et al.  Swift-review: a text-mining workbench for systematic review.  Systematic reviews, 5:1–16, 2016.
  * [22] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec.  Open graph benchmark: Datasets for machine learning on graphs.  In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
  * [23] Sarthak Jain, Edward Banner, Jan-Willem van de Meent, Iain J. Marshall, and Byron C. Wallace.  Learning disentangled representations of texts with application to biomedical abstracts.  In Proc. of EMNLP, 2018.
  * [24] Hyunchul Jang, Jaesoo Lim, Joon-Ho Lim, Soo-Jun Park, Kyu-Chul Lee, and Seon-Hee Park.  Finding the evidence for protein-protein interactions from PubMed abstracts.  ArXiv preprint, 2022.
  * [25] Evangelos Kanoulas, Dan Li, Leif Azzopardi, and Rene Spijker.  Clef 2019 technology assisted reviews in empirical medicine overview.  In CEUR workshop proceedings, volume 2380, 2019.
  * [26] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In Proc. of ICLR, 2017.
  * [27] Martin Krallinger, Florian Leitner, and Alfonso Valencia.  Analysis of biological processes and diseases using text mining approaches.  Methods in molecular biology, 2010.
  * [28] Eric W Lee, Byron C Wallace, Karla I Galaviz, and Joyce C Ho.  Mmidas-ae: multi-modal missing data aware stacked autoencoder for biomedical abstract screening.  In Proceedings of the ACM Conference on Health, Inference, and Learning, 2020.
  * [29] Xiaohe Li, Lijie Wen, Chen Qian, and Jianmin Wang.  Gahne: Graph-aggregated heterogeneous network embedding.  In 2020 IEEE 32nd International Conference on Tools with Artificial Intelligence (ICTAI). IEEE, 2020.
  * [30] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard S. Zemel.  Gated graph sequence neural networks.  In Proc. of ICLR, 2016.
  * [31] Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld.  S2ORC: The semantic scholar open research corpus.  In Proc. of ACL, 2020.
  * [32] Qingsong Lv, Ming Ding, Qiang Liu, Yuxiang Chen, Wenzheng Feng, Siming He, Chang Zhou, Jianguo Jiang, Yuxiao Dong, and Jie Tang.  Are we really making much progress? revisiting, benchmarking and refining heterogeneous graph neural networks.  In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, 2021.
  * [33] Manuele Michelessi, Ersilia Lucenteforte, Francesco Oddone, Miriam Brazzelli, Mariacristina Parravano, Sara Franchi, Sueko M Ng, and Gianni Virgili.  Optic nerve head and fibre layer imaging for diagnosing glaucoma.  Cochrane Database of Systematic Reviews, (11), 2015.
  * [34] Makoto Miwa, James Thomas, Alison O’Mara-Eves, and Sophia Ananiadou.  Reducing systematic review workload through certainty-based screening.  Journal of biomedical informatics, 51:242–253, 2014.
  * [35] Galileo Namata, Ben London, Lise Getoor, Bert Huang, and UMD EDU.  Query-driven active surveying for collective classification.  In 10th International Workshop on Mining and Learning with Graphs, 2012.
  * [36] Feiping Nie, Wei Zhu, and Xuelong Li.  Unsupervised large graph embedding.  In Proc. of AAAI, 2017.
  * [37] Bernd Richter, Bianca Hemmingsen, Maria-Inti Metzendorf, and Yemisi Takwoingi.  Development of type 2 diabetes mellitus in people with intermediate hyperglycaemia.  Cochrane Database of Systematic Reviews, (10), 2018.
  * [38] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini.  The graph neural network model.  IEEE transactions on neural networks, (1), 2008.
  * [39] Sharon D Solomon, Kristina Lindsley, Satyanarayana S Vedula, Magdalena G Krzystolik, and Barbara S Hawkins.  Anti-vascular endothelial growth factor for neovascular age-related macular degeneration.  Cochrane Database of Systematic Reviews, (8), 2014.
  * [40] Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei.  LINE: large-scale information network embedding.  In Proceedings of the 24th International Conference on World Wide Web, WWW 2015, Florence, Italy, May 18-22, 2015, 2015.
  * [41] Anton Tsitsulin, Marina Munkhoeva, Davide Mottin, Panagiotis Karras, Ivan Oseledets, and Emmanuel Müller.  Frede: anytime graph embeddings.  Proceedings of the VLDB Endowment, (6), 2021.
  * [42] Siw Waffenschmidt, Marco Knelangen, Wiebke Sieben, Stefanie Bühn, and Dawid Pieper.  Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review.  BMC Medical Research Methodology, (132), 2019.
  * [43] Kuansan Wang, Zhihong Shen, Chiyuan Huang, Chieh-Han Wu, Yuxiao Dong, and Anshul Kanakia.  Microsoft academic graph: When experts are not enough.  Quantitative Science Studies, (1), 2020.
  * [44] Xiao Wang, Peng Cui, Jing Wang, Jian Pei, Wenwu Zhu, and Shiqiang Yang.  Community preserving network embedding.  In Proc. of AAAI, 2017.
  * [45] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S. Yu.  Heterogeneous graph attention network.  In The World Wide Web Conference, WWW 2019, San Francisco, CA, USA, May 13-17, 2019, 2019.
  * [46] Chih-Hsuan Wei, Bethany R. Harris, Donghui Li, Tanya Z. Berardini, Eva Huala, Hung-Yu Kao, and Zhiyong Lu.  Accelerating literature curation with text-mining tools: a case study of using PubTator to curate genes in PubMed abstracts.  ArXiv preprint, 20d.
  * [47] Xiaokai Wei, Linchuan Xu, Bokai Cao, and Philip S. Yu.  Cross view link prediction by learning noise-resilient representation consensus.  In Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017, 2017.
  * [48] Jevin D West, Ian Wesley-Smith, and Carl T Bergstrom.  A recommendation system based on hierarchical clustering of an article-level citation network.  IEEE Transactions on Big Data, (2), 2016.
  * [49] Carl Yang, Yuxin Xiao, Yu Zhang, Yizhou Sun, and Jiawei Han.  Heterogeneous network representation learning: A unified framework with survey and benchmark.  TKDE, 2020.
  * [50] Renchi Yang, Jieming Shi, Xiaokui Xiao, Yin Yang, and Sourav S Bhowmick.  Homogeneous network embedding for massive graphs via reweighted personalized pagerank.  Proceedings of the VLDB Endowment, (5), 2020.
  * [51] Renchi Yang, Jieming Shi, Xiaokui Xiao, Yin Yang, Juncheng Liu, and Sourav S Bhowmick.  Scaling attributed network embedding to massive graphs.  Proceedings of the VLDB Endowment, (1), 2020.
  * [52] Yaming Yang, Ziyu Guan, Jianxin Li, Jianbin Huang, and Wei Zhao.  Interpretable and efficient heterogeneous graph convolutional network.  ArXiv preprint, 2020.
  * [53] Seongjun Yun, Minbyul Jeong, Raehyun Kim, Jaewoo Kang, and Hyunwoo J. Kim.  Graph transformer networks.  In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019.
  * [54] Xinyuan Zhang, Qing Xie, and Min Song.  Measuring the impact of novelty, bibliometric, and academic-network factors on citation count using a neural network.  Journal of Informetrics, (2), 2021.
  * [55] Zhanqiu Zhang, Jianyu Cai, Yongdong Zhang, and Jie Wang.  Learning hierarchy-aware knowledge graph embeddings for link prediction.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3065–3072, 2020.
  * [56] Chang Zhou, Yuqiong Liu, Xiaofei Liu, Zhongyi Liu, and Jun Gao.  Scalable graph embedding for asymmetric proximity.  In Proc. of AAAI, 2017.