#  Accelerating COVID-19 research with graph mining and transformer-based
learning

Ilya Tyagin  0000-0002-0249-1072 Center for Bioinformatics and Computational
Biology
University of DelawareNewarkDE19713USA<EMAIL_ADDRESS>,  Ankit Kulshrestha
Computer and Information Sciences
University of DelawareNewarkDE19716USA<EMAIL_ADDRESS>,  Justin Sybrandt
0000-0001-5073-0122 School of Computing
Clemson University821 McMillan Rd.ClemsonSC29630USA<EMAIL_ADDRESS>,
Krish Matta  Charter School of WilmingtonWilmingtonDE19807USA
<EMAIL_ADDRESS>,  Michael Shtutman  Drug Discovery and
Biomedical Sciences
University of S. Carolina715 Sumter StreetColumbiaSC29208USA
<EMAIL_ADDRESS>and  Ilya Safro  Computer and Information Sciences
University of DelawareNewarkDE19716USA<EMAIL_ADDRESS>
(2020)

###### Abstract.

In 2020, the White House released the, “Call to Action to the Tech Community
on New Machine Readable COVID-19 Dataset,” wherein artificial intelligence
experts are asked to collect data and develop text mining techniques that can
help the science community answer high-priority scientific questions related
to COVID-19. The Allen Institute for AI and collaborators announced the
availability of a rapidly growing open dataset of publications, the COVID-19
Open Research Dataset (CORD-19). As the pace of research accelerates,
biomedical scientists struggle to stay current. To expedite their
investigations, scientists leverage hypothesis generation systems, which can
automatically inspect published papers to discover novel implicit connections.
We present automated general purpose hypothesis generation systems AGATHA-C
and AGATHA-GP for COVID-19 research. The systems are based on the graph mining
and transformer models. The systems are massively validated using
retrospective information rediscovery and proactive analysis involving human-
in-the-loop expert analysis. Both systems achieve high-quality predictions
across domains in fast computational time and are released to the broad
scientific community to accelerate biomedical research. In addition, by
performing the domain expert curated study, we show that the systems are able
to discover on-going research findings such as the relationship between
COVID-19 and oxytocin hormone.
Reproducibility: All code, details, and trained models are available at
https://github.com/IlyaTyagin/AGATHA-C-GP.

Hypothesis Generation, Literature-Based Discovery, Transformer Models,
Semantic Networks, Biomedical Recommendation,

††copyright: acmcopyright††journalyear: 2020††doi:
10.1145/1122445.1122456††conference: SIGKDD; November 1–5, 2021; ††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Applied computing
Bioinformatics††ccs: Applied computing Document management and text
processing††ccs: Computing methodologies Learning latent representations††ccs:
Computing methodologies Neural networks††ccs: Computing methodologies
Information extraction††ccs: Computing methodologies Semantic networks

##  1\. Introduction

Development of vaccines for COVID-19 is a major triumph of modern medicine and
humankind’s ability to accelerate scientific research. While we are all hoping
to see large-scale positive changes from fast mass adoption of the existing
vaccines, there remain significant open research questions around COVID-19.
The scientific community has a responsibility to do everything possible to
block the ongoing transmission of the dangerous virus and _accelerate research
to mitigate its consequences_. We present the following automated knowledge
discovery system in order to propose new tools that could compliment the
existing arsenal of techniques to accelerate biomedical and drug discovery
research for events like COVID-19.

The COVID-19 pandemic became one of the most important events in the
information space since the end of 2019. The pace of published scientific
information is unprecedented and spans all resolutions, from the news and pop-
science articles to drug design at the molecular level. The pace of scientific
research has already been a significant problem in science for years
(Spangler, 2015), and under current circumstances this factor becomes even
more pronounced. Several thousands papers are being added _weekly_ to
CORD-19111https://www.semanticscholar.org/cord19 (the dataset of publications
related to COVID-19) and even more in
MEDLINE222https://www.nlm.nih.gov/bsd/stats/cit_added.html. As a result,
groups working on similar problems may not be immediately aware of the other’s
findings, which can lead to inefficient investments and production delays.

Figure 1.  Number of new citations per week in CORD-19 dataset.

Under normal circumstances, the MEDLINE database of biomedical citations
receives approximately 950,000 new papers per year. Currently this database
indexes 31 million total citations. This pace challenges traditional research
methods, which often rely on human intuition when searching for relevant
information. As a result, the demand for modern AI solutions to help with the
automated analysis of scientific information is incredibly high. For instance,
the field of drug discovery has explored a range of AI analytical tools to
expedite new treatments (Gopalakrishnan et al., 2019). Designing lab
experiments and finding candidate chemical compounds is a costly and long-
lasting procedure, often taking years. To accelerate scientific discovery,
researchers came up with a family of strategies to utilize public knowledge
from databases like MEDLINE that are available through the National Institute
of Health (NIH), which facilitate _automated hypothesis generation_ (HG) also
known as literature-based discovery. Undiscovered public knowledge,
information that is implicitly present within available literature, but is not
yet explicitly known by an individual who can act on that information,
represents the target of our work.

Although, there are quite a few hypothesis generation (HG) systems
(Gopalakrishnan et al., 2019) including those we have previously proposed
(Sybrandt et al., 2017; Sybrandt et al., 2020), _none of them is currently
COVID-19 customized and available in the open domain to massively process
related queries_. In addition to the traditional requirements for HG systems,
such as high-quality results of hypotheses, interpretability and availability
for broad scientific community, a specific demand for COVID-19 data analysis
requires: (1) customization of the vocabulary and other logical units such as
subject-verb-object predicates; (2) customization of the training data that in
the reality of urgent research contains a lot of controversial and incorrect
information; (3) multiple models for different information resolutions (e.g.,
microscopic for drug design, and macroscopic for the population related
conclusions); and (4) validation on the on-going domain-specific discovery.

Our contribution: In this work we bridge this gap by releasing, AGATHA-C and
AGATHA-GP , reliable and easy to use HG systems that demonstrate state-of-the
art performance and validate their inference capabilities on both COVID-19
related and general biomedical data. To make them closely related to different
goals of COVID-19 research, they correspond to micro- (AGATHA-C, for COVID-19)
and macroscopic (AGATHA-GP, for general purpose) scales of knowledge
discovery. Both systems are trained on all existing biomedical literature
available through NIH and CORD-19 and able to process any queries to connect
biomedical concepts but AGATHA-C exhibits better results on the molecular
scale queries, e.g., those that are relevant to drug design, and AGATHA-GP
works better for general queries, e.g., establishing connections between
certain profession and COVID-19 transmission. As it will be explained later,
we emphasize that AGATHA is not a traditional _information retrieval_ system
that effectively searches for _existing_ information and thus cannot be
compared to them. Instead, AGATHA generates novel hypotheses.

Both systems are the next generation of the AGATHA knowledge network mining
transformer model (Sybrandt et al., 2020). _They substantially improve the
quality of the previous AGATHA by introducing new information layers into
multi-layered semantic knowledge network pipeline, and expanding new
information retrieval techniques that facilitate inference._ We present the
deep learning transformer-based AGATHA-C/GP models trained with up-to date
datasets and provide easy to use interface to broad scientific community to
conduct COVID-19 research. We validate the system via candidate ranking
(Sybrandt et al., 2018b; Sybrandt et al., 2020) using very recent scientific
publications containing findings absent in the training set. While the
original AGATHA has demonstrated state-of-the-art performance for the time of
its release, AGATHA and other systems were found to perform with notably lower
quality on extremely rapidly changing COVID-19 research. We demonstrate a
remarkable improvement on different types of queries with very fast query
process that allows massive validation. In addition, we demonstrate that the
proposed system can identify recently uncovered gene (BST2) and hormone
(oxytocin and melatonin) relationships to COVID-19, using only papers
published before these connections were discovered.

Reproducibility: All code, details, and pre-trained models are available at
https://github.com/IlyaTyagin/AGATHA-C-GP.

##  2\. Background and Related Work

The HG field has been present in information sciences for several decades. The
first notable approach was proposed by Swanson et al. in 1986 (Swanson, 1986),
which is called the A-B-C model. The concept of A-B-C model is to discover
intermediate (B) terms which occur in titles of publications for both terms A
(source) and C (target). In their experiments, Swanson et al. discovered an
implicit connection between Raynauld’s syndrome (term A) and fish oil (term C)
through blood viscosity (term B), which was mentioned in both sets. The
hypothesis that fish oil can be used for patients with Raynaud’s disease was
experimentally confirmed several years later (DiGiacomo et al., 1989). The key
idea of the proposed method is that all fragmented bits of information are
explicitly known, but their implicit relationships is what HG systems are
aimed to uncover.

We note the difference between HG (not only the A-B-C models) and traditional
information retrieval. The information retrieval techniques which represent
the vast majority of biomedical literature based discovery systems (including
those for COVID-19) are trained and (what is even more important) validated to
retrieve _existing information_ whereas the HG techniques predict
_undiscovered knowledge_ and thus must be massively validated on it. The HG
validation requires training the system strictly on historical data rather
than sampling it over the entire time.

The advances in machine and deep learning transformed the algorithms of HG
systems that are now able to process much larger information volumes
demonstrating higher quality predictions.

A number of works have been proposed to organize the CORD-19 literature into
structured graphs for different purposes . For instance, Basu _et al._ (Basu
et al., 2020) propose ERLKG - a knowledge graph built on CORD-19 with entities
corresponding to gene/chemical/disease names and the edges forming relations
between the concept. They use a fine tuned SciBERT model for both entity and
relation extraction. The main purpose of the knowledge graph is to predict a
link between a given chemical-disease and chemical-protein pair using a
trained GCN autoencoder (Kipf and Welling, 2017) approach. In another similar
work, Oniani _et al._ (Oniani et al., 2020) build a co-occurrence network on a
subset of CORD-19 with the edges corresponding to either gene-disease, gene-
mutation or chemical-disease type. The network is then embedded into latent
space using a node2vec walk. Link prediction is performed on the nodes by
training different classical machine learning algorithms. A major shortcoming
of these approaches is that they are limited to either specific kind of
entities or relations or both and as a result not only the scope of possible
new literature is narrowed but a lot of additional useful knowledge is
filtered out of the system. In contrast, our system does not limit itself to
specific entity or relation type and is able to capture much more information
from the same corpus.

A major interest of constructing knowledge graphs is to allow medical
researchers to re-purpose existing drugs for treating COVID-19. Zhang _et al._
(Zhang et al., 2020) develop a system that uses combined semantic predicates
from SemMedDB and CORD-19 (extracted using SemRep) to recommend drugs for
COVID-19 treatment. To improve the predications from CORD-19, the authors fine
tune various transformer based models on a manually annotated internal
dataset. Their resulting knowledge graph consists of 131,555 nodes and
2,558,935 edges. Our work on the other hand utilizes similar technologies and
produces a bigger graph with 287,356,836 nodes and 13,500,291,256 edges.
Moreover, we do not post-process extracted relations from SemRep and are still
able to achieve a higher RoC metric.

The most similar to our current work is the system proposed by (Nordon et al.,
2019). The authors use Electronic Medical Record (EMR) to generate candidates
for drug repurposing and then use a knowledge graph constructed from MEDLINE
documents to validate those candidates. In context of recent diseases like
COVID-19 for which EMRs are not as readily available, the system will be
limited in their candidate generation. Moreover, validation using only
SemMedDB publication data may yield subpar results since it is not updated as
frequently. In contrast, our method incorporates new sources of literature
during the graph construction phase itself and can be readily adapted to new
and emerging challenges in medicine. Other systems that are built for COVID-19
drug discovery include systems by (Martinc et al., 2020) and Kinderminer
(Kuusisto et al., 2017). The former tool uses fine-tuned SciBERT model to
generate contexualized embeddings given an initial seed set of words and the
latter system uses a keyword co-count algorithm to propose candidates for
COVID-19. We observe that our graph contains a larger variety of data sources
data than any of these tools and thus can produce broader set of hypotheses.

The vastness of COVID-19 literature also spurned the need for having systems
that could allow researchers and base users alike to get their COVID-19
queries answered. Systems like CKG (Wise _et al._) (Wise et al., 2020) and
SciSight (Hope _et al._) (Hope et al., 2020) currently provide this
functionality. While we do aim to provide an easy to use web-framework for
medical researchers, the scope of the aforementioned systems is beyond the
scope of our work. Unfortunately, no existing system out of those that are
trained to accept terms related to COVID-19 or SARS-CoV-2 provided an open
access for massive validation for a fair comparison with or was able to be
tested in multiple domains like AGATHA-C .

The lack of broader applicability of systems like these in the situation with
COVID-19 pandemic demonstrates that several major issues exist and require
immediate attention:
(1) Most of the existing HG systems are domain-specific (e.g., gene-disease
interactions) that is usually expressed in limiting the processed information
(e.g., significant filtering vocabulary and papers to a specific domain in
probabilistic topic modeling (Wang et al., 2011));
(2) A proper validation of HG system remains a technical problem because
multiple large-scale models have to trained with all heterogeneous data
carefully eliminated several years back;
(3) Moreover, a large number of HG systems are not massively validated at all
except of very old findings rediscovery (Smalheiser, 2017) or demonstrating of
just a few proactive examples in humanly curated investigation; and
(4) Interpretability and explainability of generated hypotheses remains a
major issue.

The UMLS Metathesaurus (Bodenreider, 2004b) is the NIH database containing
information about millions of concepts (both medical and general) and their
synonyms.

Metathesaurus accumulates information about its entries from more than 200
different vocabularies allowing to map and connect concepts from different
terminologies. Metathesaurus also keeps metadata about the concepts such as
semantic types and their hierarchy. The core unit of information in UMLS is
the concept unique identifier, or CUI. CUI is a codified representation of a
specific term, which includes its different atoms (spelling variants or
translations of the term on other languages), vocabulary entries, definitions
and other metadata.

SemRep (Arnold and Rahm, 2015) is a software kit developed by NIH for
extraction of semantic predicates (subject-verb-object triples) from the
provided corpus. It also allows to extract entities not involved in any
semantic predicate, if the corresponding option is selected.

The official example of possible SemRep output is: INPUT = “We used
hemofiltration to treat a patient with digoxin overdose that was complicated
by refractory hyperkalemia.”, OUTPUT = “Hemofiltration - TREATS - Patients;
Digoxin overdose - PROCESS_OF - Patients; hyperkalemia-COMPLICATES-Digoxin
overdose; Hemofiltration - TREATS (INFER) - Digoxin overdose”. SemRep handles
word sense disambiguation and performs terms mapping to the corresponding CUIs
from UMLS metathesaurus.

ScispaCy (Neumann et al., 2019) ScispaCy is a special version of spaCy
maintained by AllenAI, containing spaCy models for processing scientific and
bio-related texts. ScispaCy models are trained on different sources, such as
PMC-pretrained word2vec representations, MedMentions Entity linking Dataset
and so on. SciSpacy can handle various NLP tasks, such as NER, dependency
parsing and POS-tagging, where achieves state of the art performance.

SciBERT (Beltagy et al., 2019) is a BERT-like transformer pretrained language
model, where full-text scientific papers were used as a training dataset.
Embeddings are learned in a word-piece fashion, which makes them capture the
relationships between not only words in a sentence, but also between word
parts in each word.

FAISS (Johnson et al., 2017) is a library for fast approximate clustering and
similarity search between dense vectors. It scales to the huge datasets that
do not fit in RAM and can be used in a distributed fashion. FAISS is used in
our pipeline to perform $k$-means clustering of PQ-quantizated sentence
vectors to generate $k$-nearest neighbor edges for similar sentences (nodes)
in knowledge network.

PTBG (Lerer et al., 2019) (stands for PyTorch BigGraph) is a high-performance
graph embedding system allowing distributed training. It was designed to
handle large heterogeneous networks containing hundreds of millions of nodes
of different types and billions of typed edges. Distributed training is
achieved by computing embeddings on disjoint node sets.

AllenNLP Open Information Extraction. AllenNLP (Gardner et al., 2017) is a
powerful library developed by AllenAI that uses PyTorch backend to provide
deep-learning models for various natural processing tasks. Specifically,
AllenNLP Open Information Extraction provides a trained deep bi-LSTM model for
extracting predicates from unstructured text. An API is provided for running
inference in both single sentence and batch modes.

CORD-19 dataset (Wang et al., 2020) was released as a response to the world’s
COVID-19 pandemic to help data science experts and researchers to tackle the
challenge of answering the high priority scientific questions. It updates
daily and was created by the Allen Institute for AI in collaboration with
Microsoft Research, NLM, IBM and other organizations. At the time of writing
this paper it contains over 400.000 scientific abstracts and over 150.000
full-text papers about coronaviruses, primarily COVID-19.

MEDLINE is a database of NIH that includes almost 31 million citations (as of
2021) of scientific papers related to the biomedical and related fields. Some
of the citations are provided with MeSH (Medical Subject Headings) terms and
other metadata. MEDLINE is one of the largest and well-known resources for
biomedical text mining.

##  3\. Pipeline Summary

We briefly summarize the AGATHA semantic graph construction pipeline. It is
described in greater details in (Sybrandt et al., 2020).

Text pre-processing. The input for our system is a corpora of scientific
citations from the MEDLINE and CORD-19 datasets. These files contain titles
and abstracts for millions of biomedical papers. We filter non-English
documents, using the FastText Language Identification model (Joulin et al.,
2016) if the language is not provided. After that we split all abstracts into
sentences and process all sentences with ScispaCy library. From each sentence
we extract POS-annotated lemmas, entities and perform $n$-gram mining, where
$n\in[2,3,4]$ and $n$-grams are composed of frequently co-occurring lemmas.
Additionally, we associate all sentences with any relevant metadata, such as
the MeSH/UMLS (Bodenreider, 2004a) keywords provided along with the citation.

Figure 2. AGATHA multi-layered graph schema.

Semantic Graph Construction. We construct a semantic graph containing
different types of nodes, namely, sentences, entities, coded terms (from UMLS
and MeSH), $n$-grams, lemmas, and predicates following the schema depicted in
Figure 2. Edges between sentences are induced from the nearest-neighbors
network of sentence embeddings. We also include an edge between two sentences
that appear sequentially within the same abstract, counting the title as the
first sentence. Other edges can be inferred directly from the recorded
metadata. For instance, the node representing the entity “COVID-19” is
connected to every sentence and predicate that discuss COVID-19.

NLM UMLS implementation. The prior AGATHA semantic network only includes UMLS
terms that appear in SemMedDB predicates (Kilicoglu et al., 2012) which is a
major limitation. In this work we enrich the “Coded Term” layer by introducing
an additional preprocessing phase wherein we run the SemRep tool with full-
fielded output option ourselves on the entire input corpora. This phase would
be necessary as CORD-19 and most recent MEDLINE citations are not represented
within slowly updated SemMedDB. However, we find that we can substantially
increase the quality of recovered terms by applying these tools ourselves. By
doing that we not only enrich the “Coded Terms” semantic network layer, but
also introduce a significant number of uncovered previously semantic
predicates.

It happens because SemMedDB is a cumulative database, having various citations
in the database processed over many years with various versions of SemRep and
various UMLS releases available at different time periods.

To illustrate what was just said, let us consider the following example (PMID:
20109154): ”The results showed that V. cholerae O395 and also other related
enteric pathogens have the essential CASS components (CRISPR and cas genes) to
mediate a RNAi-like pathway.” The current SemRep version extracts the
following predicate: CRISPR-AFFECTS-RNAi, while SemMedDB does not contain any
predicates for this sentence. The year of publication of the corresponding
paper is 2009, but CRISPR term (C3658200) did not exist in the UMLS
metathesaurus on or before 2012, that is why at the time of adding this
citation to SemmedDB CRISPR-involved relation could not be identified.

Graph Embedding. The resulting semantic graph is a large undirected
heterogeneous network, where each node has its own type (as shown in Figure 2)
and each edge between nodes with types $u$ and $v$ corresponds to type $uv$.
At this point we additionally clarify that the constructed network is not a
traditional homogeneous knowledge graph. We embed the network using a
heterogeneous technique that captures node similarity through a biased
transformed dot product. By explicitly including a bias term for each node, we
capture a concepts overall affinity within the network that is critical for
such general terms as “coronavirus.” By learning transformations between each
pair of node types (e.g., between sentences and lemmas), we enable each type
to occupy embedding spaces with differing characteristics. Specifically, we
fit an embedding model that optimizes the following similarity measure:

(1)
$\mathcal{S}(u,v)=\hat{u}_{1}+\hat{v}_{1}+T^{uv}_{1}+\sum_{i=2}^{d}\hat{u}_{i}(\hat{v}_{i}+T^{uv}_{i}),$

where $d$ \- space dimensionality, $u,v$ are nodes in the semantic graph with
embeddings $\hat{u},\hat{v}$, and $T^{uv}$ is the directional transformation
vector between nodes of $u$’s type to nodes of $v$’s.

We use the PyTorch BigGraph heterogeneous graph embedding library to learn
$d=512$ dimensional vector embeddings for each node of our large semantic
graph. While fitting embeddings ($\hat{u}$) and transformation vectors
($T^{uv}$), we represent each edge of the semantic graph as two directed
edges. These learned values are optimized using softmax loss, where the
similarity for one edge is compared against the similarities of 100 negative
samples.

Ranking Semantic Predicates (Transformer model). After we obtain embeddings
per node in the semantic graph, we train AGATHA system ranking model. This
model is trained to rank published subject-object pairs above randomly
composed pairs of UMLS concepts (negative samples). Two coded terms, along
with a fixed-size random subsample of predicates containing each term are
input to this model. Graph embeddings for each term and predicate are fed into
stacked transformer encoder layers, which apply multi-headed self-attention
across the embedding set. The last set of encodings are averaged and the
result is projected to the unit interval, forming a scalar prediction for the
input’s “plausibility.”

Formally, the model to evaluate term pairs is defined as:

(2) $\displaystyle f(x,y)$
$\displaystyle=g\left(\begin{bmatrix}\hat{x}&\hat{y}&\hat{x^{\prime}_{1}}\ldots\hat{x^{\prime}_{k}}&\hat{y^{\prime}_{1}}\ldots\hat{y^{\prime}_{k}}\end{bmatrix}\right)$
$\displaystyle g(X)$ $\displaystyle=\text{sigmoid}(\mathcal{M}\Theta)$
$\displaystyle\mathcal{M}$
$\displaystyle=\frac{1}{|X|}\text{ColSum}\left(\mathcal{E}_{N}(\text{FeedForward}(X))\right)$
$\displaystyle\mathcal{E}_{0}(X)$ $\displaystyle=X$
$\displaystyle\mathcal{E}_{i+1}(X)$
$\displaystyle=\text{LayerNorm}\left(\text{FeedForward}(\mathcal{A}(X))+\mathcal{A}(X)\right)$
$\displaystyle\mathcal{A}(X)$
$\displaystyle=\text{LayerNorm}\left(\text{MultiHeadAttention}(X)+X\right),$

where each $x^{\prime}$ and $y^{\prime}$ are randomly sampled from the
neighborhoods of $x$ and $y$ respectively, and each $\hat{\cdot}$ denotes the
graph embedding of the given node. Furthermore, $\Theta$ represents a free
parameter, which is fit along with parameters internal to each FeedForward and
MultiHeadAttention layer, following the standard conventions for each.

The above model is fit using margin ranking loss, where predicates from the
training set are compared against a large set of negative samples. Additional
details pertaining to specific optimization choices surrounding this model are
present in the work originally proposing this model (Sybrandt et al., 2020).

##  4\. Augmenting Semantic Predicates with Deep Learning

Figure 3.  Predicate Extraction pipeline with Deep Learning based Open IE
system.

We used SemRep predicate extraction system in the first system, AGATHA-C , to
extract predicates from the abstracts. However, SemRep relies on expert coded
rules and heuristics to extract biomedical relations leading to significantly
fewer predicates for training. Thus, in order to augment the predicates (for
the second system, AGATHA-GP ) we decided to use a deep learning based
information extraction system by Stanvosky _et al._ (Stanovsky et al., 2018).
Figure 3 shows our overall predicate extraction pipeline. We index our
pipeline by box numbers and describe them below. Abstract Pre-processing [Box
1] . The input for the proposed semantic predicate extraction system is the
output files generated by SemRep tool with full-fielded output option enabled,
obtained from the preprocessing stage described in Sec. 3. As it was mentioned
previously, SemRep system extracts not only semantic triples, but also maps
entities found in the input corpus to their corresponding UMLS concept IDs,
this is the data which is used for the following method. The initial set of
records includes the sentence raw texts and extracted from them UMLS terms and
is augmented throughout the pipeline making it easier to extract final
predicates for downstream training.

Raw Predicate Extraction [Box 2 & 3]. We use a pre-trained instance of RnnOIE
(Stanovsky et al., 2018) provided as an API by AllenNLP. The model was trained
on the OIE2016 corpus. At a high level the model aims to learn a joint
embedding of individual words and their corresponding Beginning-Input-Output
(BIO) tags. The output of the model is a probability distribution over the BIO
tags. During inference the model selects specific phrases and groups them into
ARG0, V, ARG1 tags. By convention, we treat ARG0 as the subject and ARG1 as
the object in a subject-verb-object tuple. To speed up processing and scale it
to thousands of abstracts, we leverage model-parallelism across different
machines and run batch-mode inference on chunks of abstracts. Once the model
predictions have been extracted we extract the phrases with relevant tags into
raw predicates and augment them in the record. A subsequent filtering is
performed by extracting the terms matching with previously detected UMLS
concepts in the sentence.

Semnet Filtering [Box 4] Using a general purpose RnnOIE model has it’s own
challenges. During processing we noted that a lot of raw predicates were
either too general or contained too little meaning to be useful for training a
prediction model. To overcome this challenge we designed a corrective filter
to reduce noise and retain most useful predicates. We call this filter the
_semnet filter_.

Each UMLS concept has an associated semantic type (e.g., COVID-19 has an
associated semantic type of dsyn (disease)). This is useful for summarizing
large set of diverse text concepts into smaller number of categories. We used
the metadata from semantic types to construct two networks - a semantic
network and a hierarchical network. The semantic network consists of semantic
types as nodes and the edges imply a corresponding direct relation between
them. The hierarchical network is a network of a semantic type connected to
its more general semantic types. For example, a semantic type dsyn (disease)
is more generally associated with a biof (biological function) or a pathf
(pathological function). In order to filter a predicate, all edges emanating
from the subject’s semantic types are computed on a per-predicate basis. These
edges also include any specific-general concept relationships. If the object’s
semantic type is found to be in the candidate edge set, then we deem the
predicate as valid. In our experiments, we found that this filtering method
significantly eliminates predicates which do not directly pertain to the
biomedical domain.

Co-Count Filtering [Box 5]. A second strategy of pruning is to get information
about the terms that co-occur the most in our corpus. A larger degree of co-
occurrence implies a greater correlation between concepts. We use a normalized
frequency count as a scoring measure of co-occurrence. All predicates that
contain subject and object terms below a certain threshold are pruned from the
list of candidate predicates.

##  5\. Validation

A fair validation of HG systems is extremely challenging, as these models are
designed to predict novel connections that are unknown to even those who
evaluate the system (Sybrandt et al., 2018a). In addition, even if validated
by rediscovering findings using historical, the process is computationally
expensive because of the need to train multiple models to understand how many
months (or years) back, the HG system can predict the findings which requires
careful filtering of the used papers, vocabulary and other types of data. To
present our results in terms of its usefulness for urgent CORD-19-related HG,
we use a historical benchmark, which is conceptually described in (Sybrandt et
al., 2020). This technique is fully automated and does not require any domain
experts intervention.

Positive samples collection. We use SemRep and proposed pipeline in the
previous section approach to process the most recent CORD-19 citations, which
were published after the specific cut date making sure that the citations are
not included in the training set. After that we extract all subject-object
pairs from the obtained results and explicitly check that none of these pairs
are presented in the training set. Pairs mentioned in the CORD-19 less than
twice are filtered out from the validation set. Almost all of them are either
noisy or represent information that already appears in other pairs (e.g.,
because of the difference in grammar).

We also use the strategy of subdomain recommendation. This strategy works in
the following way. For each UMLS term we collect its semantic type (which is a
part of the metadata provided in UMLS metathesaurus) and group all extracted
SemRep pairs by the term-pair criteria (combination of subject and object
types). Then we identify the top-20 most common term-pairs subdomains and
construct the validation set from pairs belonging to these 20 subdomains.

Negative samples generation. To generate negative samples per domain, the
random sampling is used, that is, for each positive sample we keep its subject
and randomly sample the object belonging to the same semantic type as the
object of the source pair. We do this 10 times, thus having 10 negative
domain-specific samples for each positive sample. When the validation set is
generated, we apply our ranking criteria to it, obtaining a numerical score
value $s$ per each sample, where $s\in[0,1]$.

Evaluation metrics. We propose our approach as a recommendation system and to
report our results we use a combination of the following classification and
recommendation metrics.

  * •

Classification metrics: (1) Area under the receiver-operating-characteristic
curve (AUC ROC); (2) Area under the precision-recall curve (AUC PR).

  * •

Recommendation metrics: (1) Top-k precision (P.@k); (2) Average precision
(AP.@k).

We report these numbers in per subdomain manner to better understand how the
system performs with respect to specific task (e.g. drug repurposing).

##  6\. Results

To report results, we provide the performance measures for three AGATHA models
trained on the same input data (MEDLINE corpus and CORD-19 abstracts dataset):

  1. (1)

AGATHA-O : Baseline AGATHA model (Sybrandt et al., 2020);

  2. (2)

AGATHA-C : AGATHA-O with new UMLS layer and SemRep enrichment;

  3. (3)

AGATHA-GP : AGATHA-C with additional deep learning-based extracted and further
filtered predicates.

It is done in this particular manner because the major role in learning the
proposed ranking criteria depends heavily on the quality of extracted semantic
predicates and their number, as they form the training set for the AGATHA
ranking module. _At the moment of writing, no other general purpose and
available for public use HG system compliant with the three validation
criteria, namely, (a) ability to run thousands of queries in a reasonable
time, (b) ability to process COVID-19 related vocabulary, and (c) ability to
operate in multiple domains was available for comparison._ Comparison of the
baseline AGATHA-O is discussed in (Sybrandt et al., 2020).

The performance of both AGATHA-C and AGATHA-GP allows to run thousands of
queries in a very short time (in the order of minutes), making the validation
on a large number of samples possible. Unfortunately, given the current
circumstances, large-scale validation for the specific scientific subdomain
(COVID-19 related hypotheses) is hard to implement, because well-established
and reliable factual base is being actively developed at the moment and big
historic gap for the vocabulary simply does not exist (e.g., the COVID-19 term
is less than two years old). We, however, provide the validation set of
positive connections extracted from CORD-19 dataset citations added within the
time frame from October 28, 2020 to January 21, 2021, which numbered at 77
thousand abstracts.

The approximate running time with corresponding types of used hardware is
presented in Table 1. Each row corresponds to the stage in the AGATHA-C
/AGATHA-GP pipelines. The column “M” (machines) and CPU show the number of
machines and required CPUs, respectively. In the column “GPU” we indicate if
GPU was required or optional. For AGATHA training we used two NVIDIA V100 per
machine. The minimal requirements for RAM per machine are in column “RAM”. The
running time of queries is negligible.

Table 1.  Running time and hardware requirements.  Stage | Time | Hardware
---|---|---
|  | M | CPU | GPU | RAM
SemRep Processing | 2 d | 10-28 | 20+ | Opt | N/A
AllenNLP Predicates | 3 d | 28-40 | 20+ | Opt | N/A
Graph Construction | 10 d | 30+ | 20+ | Opt | 120GB+
Graph Conversion | 7 h | 1 | 40+ | Opt | 1TB+
Graph Embedding | 1 d | 20 | 24+ | Opt | 120GB+
AGATHA Training | 22 h | 5+ | 2+ | Yes | 300GB+
Network Adjacency | 1 d | 1 | 40+ | Opt | 1.5TB+

The overall training dataset contains 190.6 million sentences, which results
in 287 million nodes and 13.5 billion edges (AGATHA-C model).

Table 2.  Classification and recommendation quality metrics across recently popular COVID-19-related biomedical subdomains. Labels O, C and GP stand for AGATHA-O , AGATHA-C and AGATHA-GP models, respectively. Used abbreviations: phsu: Pharmacologic Substance; topp: Therapeutic or Preventive Procedure; humn: Human; menp: Mental Process; spco: Spatial Concept; aapp: Amino Acid, Peptide, or Protein; gngm: Gene or Genome; orch: Organic Chemical; fndg: Finding; geoa: Geographic Area; dsyn: Disease or Syndrome. Semantic pairs are presented in descending order according to the citations count of their corresponding predicates.  | ROC AUC | PR AUC | P.@10 | P.@100 | AP.@10 | AP.@100
---|---|---|---|---|---|---
| O | C | GP | O | C | GP | O | C | GP | O | C | GP | O | C | GP | O | C | GP
dsyn:dsyn | 0.83 | 0.89 | 0.88 | 0.35 | 0.41 | 0.44 | 0.60 | 0.50 | 0.70 | 0.54 | 0.54 | 0.55 | 0.70 | 0.49 | 0.72 | 0.56 | 0.54 | 0.62
phsu:dsyn | 0.86 | 0.91 | 0.91 | 0.36 | 0.43 | 0.47 | 0.30 | 0.60 | 0.60 | 0.45 | 0.51 | 0.57 | 0.77 | 0.73 | 0.90 | 0.47 | 0.55 | 0.65
fndg:dsyn | 0.86 | 0.93 | 0.92 | 0.42 | 0.53 | 0.57 | 0.70 | 0.70 | 1.00 | 0.54 | 0.58 | 0.66 | 0.90 | 0.70 | 1.00 | 0.65 | 0.64 | 0.78
dsyn:fndg | 0.80 | 0.89 | 0.90 | 0.29 | 0.42 | 0.43 | 0.40 | 0.60 | 0.60 | 0.39 | 0.46 | 0.49 | 0.46 | 0.86 | 0.55 | 0.44 | 0.59 | 0.54
fndg:humn | 0.80 | 0.89 | 0.89 | 0.37 | 0.46 | 0.47 | 0.90 | 0.80 | 0.80 | 0.43 | 0.54 | 0.58 | 0.96 | 0.86 | 0.70 | 0.71 | 0.66 | 0.68
dsyn:humn | 0.77 | 0.84 | 0.85 | 0.28 | 0.32 | 0.35 | 0.50 | 0.30 | 0.40 | 0.35 | 0.41 | 0.47 | 0.59 | 0.33 | 0.30 | 0.53 | 0.44 | 0.48
topp:dsyn | 0.87 | 0.92 | 0.92 | 0.38 | 0.50 | 0.50 | 0.40 | 0.80 | 0.60 | 0.51 | 0.58 | 0.61 | 0.33 | 0.70 | 0.81 | 0.48 | 0.65 | 0.63
orch:dsyn | 0.87 | 0.90 | 0.88 | 0.35 | 0.45 | 0.46 | 0.50 | 0.80 | 0.60 | 0.38 | 0.43 | 0.53 | 0.69 | 0.96 | 0.91 | 0.47 | 0.63 | 0.63
geoa:spco | 0.74 | 0.72 | 0.90 | 0.22 | 0.19 | 0.43 | 0.30 | 0.20 | 0.60 | 0.31 | 0.23 | 0.53 | 0.41 | 0.37 | 0.68 | 0.32 | 0.24 | 0.53
aapp:dsyn | 0.87 | 0.93 | 0.92 | 0.39 | 0.47 | 0.50 | 0.40 | 0.40 | 0.70 | 0.47 | 0.51 | 0.53 | 0.52 | 0.28 | 0.88 | 0.50 | 0.50 | 0.60
aapp:gngm | 0.73 | 0.85 | 0.84 | 0.20 | 0.30 | 0.34 | 0.30 | 0.00 | 0.60 | 0.20 | 0.36 | 0.36 | 0.32 | 0.00 | 0.59 | 0.32 | 0.33 | 0.44
orch:orch | 0.86 | 0.92 | 0.91 | 0.54 | 0.65 | 0.61 | 0.80 | 0.90 | 0.80 | 0.53 | 0.58 | 0.56 | 0.99 | 0.99 | 0.88 | 0.73 | 0.81 | 0.75
gngm:dsyn | 0.91 | 0.95 | 0.94 | 0.44 | 0.65 | 0.53 | 0.70 | 0.80 | 0.50 | 0.43 | 0.56 | 0.57 | 0.87 | 0.95 | 0.69 | 0.52 | 0.75 | 0.59
orch:phsu | 0.87 | 0.92 | 0.93 | 0.52 | 0.61 | 0.59 | 0.80 | 0.80 | 0.60 | 0.49 | 0.55 | 0.58 | 0.94 | 0.95 | 0.90 | 0.70 | 0.73 | 0.67
gngm:gngm | 0.71 | 0.85 | 0.85 | 0.18 | 0.35 | 0.34 | 0.30 | 0.60 | 0.40 | 0.20 | 0.33 | 0.36 | 0.35 | 0.65 | 0.35 | 0.22 | 0.47 | 0.45
phsu:fndg | 0.76 | 0.88 | 0.87 | 0.24 | 0.45 | 0.44 | 0.40 | 0.70 | 0.80 | 0.29 | 0.44 | 0.42 | 0.30 | 0.69 | 0.88 | 0.34 | 0.58 | 0.57
phsu:phsu | 0.86 | 0.91 | 0.92 | 0.53 | 0.56 | 0.58 | 0.90 | 0.80 | 0.80 | 0.45 | 0.47 | 0.53 | 0.98 | 0.82 | 0.90 | 0.72 | 0.70 | 0.69
orch:aapp | 0.90 | 0.93 | 0.93 | 0.57 | 0.61 | 0.68 | 0.80 | 0.70 | 1.00 | 0.49 | 0.53 | 0.58 | 0.96 | 0.74 | 1.00 | 0.74 | 0.71 | 0.78
geoa:menp | 0.88 | 0.89 | 0.94 | 0.40 | 0.50 | 0.61 | 0.60 | 0.60 | 0.60 | 0.42 | 0.47 | 0.58 | 0.59 | 0.84 | 0.59 | 0.47 | 0.63 | 0.68
phsu:orch | 0.88 | 0.94 | 0.94 | 0.49 | 0.62 | 0.55 | 0.70 | 0.80 | 0.60 | 0.44 | 0.48 | 0.51 | 0.92 | 0.95 | 0.92 | 0.62 | 0.74 | 0.61
Mean | 0.83 | 0.89 | 0.90 | 0.38 | 0.47 | 0.49 | 0.57 | 0.62 | 0.66 | 0.42 | 0.48 | 0.53 | 0.68 | 0.69 | 0.76 | 0.53 | 0.59 | 0.62

In Table 2, we compare aforementioned models using the metrics described in
the previous section. We present predicate types with NLM semantic type codes
(McCray et al., 2001) due to space restrictions. Both AGATHA-C and AGATHA-GP
models show significant gains when compared to AGATHA-O baseline model.
Benefits in the most problematic for the baseline model areas (e.g.,
(Geographic Area) $\rightarrow$ (Spatial Concept) denoted by (geoa,spco))
serve the best illustration for that, showing up to 0.25 advantage in ROC AUC
(AGATHA-GP ). Important biomedical domains, such as (Amino Acid, Peptide or
Protein) $\rightarrow$ (Disease or Syndrome) denoted by (aapp,dsyn) also show
noticeable improvements (0.07 for AGATHA-C ). Due to space limit we include
the results for only top-10 most popular subdomains. More can be found in the
long preprint version (Tyagin et al., 2021). Average ROC AUC value is
increased by 0.07.

Our validation strategy involves a big number of many-to-many queries, making
the area under precision-recall curve another very illustrative metric. This
is where the newly proposed models show even more drastic improvements over
the baseline AGATHA-O . For some subdomains, like (Organic Chemical)
$\rightarrow$ (Disease or Syndrome) (orch,dsyn) we observe that new models
improve the PR AUC score on more than 0.1. Average PR AUC value is increased
by 0.12.

##  7\. Emergent discovery case study

The proactive discovery of ongoing research findings is an important component
in the validation of hypothesis generation systems (Sybrandt et al., 2018b).
In particular, in the current uncertain situation with COVID-19 when a lot of
unintentionally incorrect discoveries are published, the validation must
include human-in-the-loop part even in limited capacity such as in (Aksenova
et al., 2019; Spangler et al., 2014). To demonstrate the predictive potential
of AGATHA-C we perform a case study on three COVID-19-related novel
connections manually selected by the domain expert. These connections were
published after the cut date before which any data used in training was
available to download at NIH.

At a low level, all AGATHA models use entity subsampling to calculate pairwise
ranking criteria, which means that the absolute numbers may fluctuate
slightly. Thus, to present the numeric scores, each experiment was repeated
100 times to compute the average and standard deviation that we present in
Figure 4.

AGATHA-C was tested whether it will be able to predict compounds potentially
applicable for the treatment of COVID-19 and the genes involved in the SARS-
CoV-2 pathogenesis. The data confirming cardiovascular protective effects of
hormone oxytocine were published recently (Diep, 2021; Wang and Wang, 2021).
The protective effect is linked to anti inflammatory activity of the hormone.
AGATHA-C ranked this connection at top 1.4 percent.

Figure 4.  Score distributions in case study experiment. Presented scores are
obtained with AGATHA-O and AGATHA-C models.

Similarly, we tested the prediction of the effects of the other hormone,
melatonin. Several publications, started from November 2020 (Cardinali et al.,
2020; Zimmermann and Curtis, 2020; Alschuler et al., 2020; Ho et al., 2021)
show the protective effects of melatonin, specifically for COVID-19
neurological complications. The activity was linked to anti-oxidative effects
of the melatonin. This connection was ranked at top 5.6 percent.

Our system ranked at top 7.6 percent the involvement of tetherin (BST2). The
results published in 2021 (Stewart et al., 2021) show that tetherin restricts
the secretion of SARS-CoV-2 viral particles and is downregulated by SARS-
CoV-2. Therefore, pharmacological activation of tetherin expression, or
inhibition of the degradation could be a promising direction of the
development of SARS-CoV-2 treatment.

To demonstrate AGATHA-C ranking capabilities, we use similar strategy to what
we proposed in the validation section, but now we randomly generate 500
negative samples for each pair of interest, maintaining the ratio of 1:500
between real-world connections and random noise. The goal of this experiment
is to rank the _real_ connections above randomly sampled pairs of the same
semantic network types. Illustration of this experiment is presented in Figure
4. In each of the three sub-figures, there are 501 results of scores for 501
pairs. One of the results is now known to be correct, while other 500 are
unknown. For example, for the first known meaningful pair COVID-19-melatonin,
we generated 500 pairs COVID-19-X pairs, where X is a randomly chosen hormone.
Because we are dealing with a medical domain, nobody can be 100% sure that all
these 500 are irrelevant. However, most of them are not likely to be related
to COVID-19. In the histograms of Fig. 4 we show a distribution of scores for
each 501 experiments for two models: baseline AGATHA-O and newly proposed
AGATHA-C . The solid lines indicate actual scores for the known pairs. To
conclude, if lab experiments were required to confirm the connections, the
known connections would be confirmed in the top 10 percent of predictions
which significantly reduces the research time and cost.

Table 3.  Comparison between AGATHA-C models trained with different cut dates and validated with recently discovered pairs.  Model | ROC AUC | PR AUC | AP.@10
---|---|---|---
C (Oct 2020) | 0.88 | 0.49 | 0.67
C (June 2021) | 0.90 | 0.55 | 0.77

We also test how inclusion of more recent training data affects AGATHA
performance. For that we take two models trained with the same methodology (C
models), but one model contains only training data limited to October 28th
2020 and another model is larger and contains more recent data (threshold:
June 23rd 2021). Both models were validated with the pairs firstly introduced
between June 24th 2021 and August 11th 2021. Results of this experiment
presented in Table 3. It shows that retraining the model using the proposed
method and more recent data yields in slightly better scores in all basic
metrics.

##  8\. Lessons Learned and Open Problems

Quality of the information retrieval pipelines. Information retrieval is an
important part of any HG pipeline. In order to uncover implicit connections,
the system should be able to capture existing explicit connections with as
much quality as possible. Given that human knowledge is usually stored in a
non-structured manner (e.g., scientific texts), the quality of systems that
process raw textual data, such as those that solve the named entity
recognition, or word sense disambiguation problems, is crucial.

We observed that the SemRep system performs better concept and relation
recognition when full abstracts are used as input data instead of single
sentences. SemRep also allows to perform optional sortal anaphora resolution
to extract co-references to the entities from neighbouring sentences, which
was shown to be useful in (Kilicoglu et al., 2016) and is used in this work.

“Positive” research bias. The absence of published negative research results
is a big problem for the HG field. With mostly positive results available,
often we have to generate negative examples through some kind of random
sampling. These negative samples likely do not adequately represent the real
nature of negatively confirmed scientific findings. Likely, one of the most
important future work directions in the area of HG is to accurately
distinguish and leverage positive and negative proposed results.

Domain experts involvement. When any hypothesis generation system is built,
one of the first questions a designer should address is extent that domain
experts are expected to participate in the pipeline. Modern decision-making
systems allow a fully automated discovery process (like the AGATHA system),
but this may not be sufficient. A domain expert who interfaces with a HG
system as a black box may not trust generated results or know how best to
interpret them. The challenge of interpretable hypothesis generation remains a
significant barrier to widespread adoption of these kinds of research tools.
For this we advocate using our “structural” learning HG system MOLIERE
(Sybrandt et al., 2017) in which with the topical modeling and network
analytic measures we interpret and explain the results.

The nature of input corpora. The question of what should be used as input to a
topic-modeling based hypothesis generation system is raised in (Sybrandt et
al., 2018a). Using full-text papers shows an improvement, but the trade-off
between run time and output quality was barely justifiable. However, deep
learning models have a greater potential for extracting useful information
from large input sources, and as it was demonstrated in our previous work
(Sybrandt et al., 2020), show significant performance advancements. Thus the
question of using full-text papers in deep learning-based hypothesis
generation systems should be addressed.

Knowledge resolution. Our newly proposed systems showed that the knowledge
resolution plays a major role in subdomain recommendation. To increase the
scope of model expertise (and the scope of potential applications beyond the
biomedical fields) we deliberately incorporate a general-purpose information
retrieval system RnnOIE into AGATHA-GP . This additional information results
in significant gains in broad subdomains like (Geographic Area) $\rightarrow$
(Idea or Concept) (geoa,idcn). At the same time, we observe that AGATHA-C
performs better in “microscopic” biomedical areas, e.g. (Organic Chemical)
$\rightarrow$ (Organic Chemical) (orch,orch), which raises the question of
choosing the appropriate model for every specific use case. Although, both
systems process all types of queries, the general purpose predicates
participated in training significantly improve “macroscopic” types of queries.

Predicate Extraction. One of the most important aspects of any hypothesis
generation system is to give it the ability to _reject_ hypothesis which are
not backed by any research. This task becomes difficult when we consider the
positive research bias of the existing literature. One way to approach this
problem can be to integrate relation extraction with sentiment modeling. In
this way, we would be able to capture not only the relations but also _how_
they are connected. Some work (Zhou et al., 2019; Eberts and Ulges, 2019) has
been proposed in this direction, but it remains an active area of research in
the biomedical domain. We aim to address this enhancement in a future work. In
our experiments, we found the LSTM based RnnOIE model to be one of the most
computationally demanding components of our pipeline. Speeding up inference
time and improving data processing efficiency at scale are two open problems
in this area.

Deployment Strategy. It is not easy to define a perfectly correct result in
biomedical hypothesis generation domain. Every year well known drugs
(sometimes used for generations) are retracted from the market because of the
newly discovered adversarial reactions, side effects or other contradictions.
Although, our system is publicly available, we cannot claim that it is fully
deployed because a full deployment with massive indication of good results
should always include a large series or proactive discovery supported with the
lab experiments. Thus, a dissemination of such system in academic biomedical
labs, and pharmaceutical research companies is imperative. Specifically,
important component of the full deployment that we are unable to test without
such broad collaborations is a verification of proprietary experimental
failure results as it is known that for a single working drug main molecule,
there are many more of those that have not been experimentally confirmed to be
useful. This negative experimental information is usually not published but
documented in internal proprietary journals.

Emergent vs. traditional research. Emergent research may show results, which
differ significantly from the results obtained with well-established and
widely used methods and datasets. One particularly important aspect of
emergent research (such as the one that we observe during COVID-19 pandemic)
is the validation strategy. When we started to work on the system there was no
publicly available high quality sufficiently large validation set, which could
have been used to evaluate the system at scale because COVID-19 related terms
have not been mentioned enough with other different terms. For the evaluation
purposes we use ”time slicing” technique in which we keep the most novel
research out of our training set. The problem with this approach is that in
some cases there is not enough time for a novel research to get proven or
refuted by a scientific community. This is especially sensitive issue in the
biomedical research. For example, in the original AGATHA model demonstration
(Sybrandt et al., 2020) we used a more sophisticated strategy to construct the
validation set, which included finding the most popular connections with non-
decreasing number of citations over time. It was possible because the
validation set was large enough to accommodate strict filtering and extracting
only a fraction of results. In this paper we had to significantly loosen these
criteria to keep the dataset size as large as possible to represent the
overall system performance.

##  9\. Conclusions

We present two graph mining transformer based models AGATHA-C and AGATHA-GP ,
for micro- and macroscopic scales of queries respectively, which are designed
to help domain experts solve high-priority research problems and accelerate
scientific discovery. We perform per-subdomain validation of these new models
on a rapidly changing COVID-19 focused dataset, composed of recently published
concept pairs and demonstrate that the proposed models achieve state-of-the-
art prediction quality. Both models significantly outperform the existing
baselines. We deploy the proposed models to the broad scientific community and
believe that our contribution can raise more interest in prospective
hypothesis generation applications.

## References

  * (1)
  * Aksenova et al. (2019) Marina Aksenova, Justin Sybrandt, Biyun Cui, Vitali Sikirzhytski, Hao Ji, Diana Odhiambo, Matthew D Lucius, Jill R Turner, Eugenia Broude, Edsel Peña, et al. 2019\.  Inhibition of the Dead Box RNA Helicase 3 prevents HIV-1 Tat and cocaine-induced neurotoxicity by targeting microglia activation.  _Journal of Neuroimmune Pharmacology_ (2019), 1–15.
  * Alschuler et al. (2020) Lise Alschuler, Ann Marie Chiasson, Randy Horwitz, Esther Sternberg, Robert Crocker, Andrew Weil, and Victoria Maizes. 2020.  Integrative medicine considerations for convalescence from mild-to-moderate COVID-19 disease.  _Explore_ (2020).
  * Arnold and Rahm (2015) Patrick Arnold and Erhard Rahm. 2015.  SemRep: A repository for semantic mapping.  _Datenbanksysteme für Business, Technologie und Web (BTW 2015)_ (2015).
  * Basu et al. (2020) Sayantan Basu et al. 2020\.  ERLKG: Entity Representation Learning and Knowledge Graph based association analysis of COVID-19 through mining of unstructured biomedical corpora. In _Proceedings of the First Workshop on Scholarly Document Processing_.
  * Beltagy et al. (2019) Iz Beltagy, Arman Cohan, and Kyle Lo. 2019.  Scibert: Pretrained contextualized embeddings for scientific text.  _arXiv preprint arXiv:1903.10676_ (2019).
  * Bodenreider (2004a) O. Bodenreider. 2004a.  The Unified Medical Language System (UMLS): integrating biomedical terminology.  _Nucleic Acids Res_ 32, Database issue (Jan 2004), D267–270.
  * Bodenreider (2004b) Olivier Bodenreider. 2004b.  The Unified Medical Language System (UMLS): Integrating Biomedical Terminology.
  * Cardinali et al. (2020) Daniel P Cardinali, Gregory M Brown, and Seithikurippu R Pandi-Perumal. 2020.  Can Melatonin Be a Potential “Silver Bullet” in Treating COVID-19 Patients?  _Diseases_ 8, 4 (2020), 44.
  * Diep (2021) Phuoc-Tan Diep. 2021\.  Is there an underlying link between COVID-19, ACE2, oxytocin and vitamin D?  _Medical Hypotheses_ 146 (2021), 110360.
  * DiGiacomo et al. (1989) R. A. DiGiacomo, J. M. Kremer, and D. M. Shah. 1989\.  Fish-oil dietary supplementation in patients with Raynaud’s phenomenon: a double-blind, controlled, prospective study.  _Am J Med_ 86, 2 (Feb 1989), 158–164.
  * Eberts and Ulges (2019) Markus Eberts and Adrian Ulges. 2019.  Span-based Joint Entity and Relation Extraction with Transformer Pre-training.  _arXiv:1909.07755 [cs]_ (Nov. 2019).  http://arxiv.org/abs/1909.07755 arXiv: 1909.07755.
  * Gardner et al. (2017) Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke S. Zettlemoyer. 2017.  AllenNLP: A Deep Semantic Natural Language Processing Platform.  arXiv:arXiv:1803.07640
  * Gopalakrishnan et al. (2019) Vishrawas Gopalakrishnan, Kishlay Jha, Wei Jin, and Aidong Zhang. 2019.  A survey on literature based discovery approaches in biomedical domain.  _Journal of biomedical informatics_ 93 (2019), 103141.
  * Ho et al. (2021) Ping Ho et al. 2021\.  Perspective Adjunctive Therapies for COVID-19: Beyond Antiviral Therapy.  _International Journal of Medical Sciences_ 18, 2 (2021), 314\.
  * Hope et al. (2020) Tom Hope, Jason Portenoy, Kishore Vasan, Jonathan Borchardt, Eric Horvitz, Daniel S. Weld, Marti A. Hearst, and Jevin West. 2020\.  SciSight: Combining faceted navigation and research group detection for COVID-19 exploratory scientific search.  arXiv:2005.12668 [cs.IR]
  * Johnson et al. (2017) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2017\.  Billion-scale similarity search with GPUs.  _arXiv preprint arXiv:1702.08734_ (2017).
  * Joulin et al. (2016) Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hérve Jégou, and Tomas Mikolov. 2016.  FastText.zip: Compressing text classification models.  _arXiv preprint arXiv:1612.03651_ (2016).
  * Kilicoglu et al. (2016) H. Kilicoglu et al. 2016\.  Sortal anaphora resolution to enhance relation extraction from biomedical literature.  _BMC Bioinformatics_ 17 (Apr 2016), 163.
  * Kilicoglu et al. (2012) Halil Kilicoglu, Dongwook Shin, Marcelo Fiszman, Graciela Rosemblat, and Thomas C. Rindflesch. 2012\.  SemMedDB: a PubMed-scale repository of biomedical semantic predications.  _Bioinform._ 28, 23 (2012), 3158–3160.  http://dblp.uni-trier.de/db/journals/bioinformatics/bioinformatics28.html#KilicogluSFRR12
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling. 2017.  Semi-Supervised Classification with Graph Convolutional Networks. In _International Conference on Learning Representations (ICLR)_.
  * Kuusisto et al. (2017) F. Kuusisto, J. Steill, Z. Kuang, J. Thomson, D. Page, and R. Stewart. 2017.  A Simple Text Mining Approach for Ranking Pairwise Associations in Biomedical Applications.  _AMIA Jt Summits Transl Sci Proc_ 2017 (2017), 166–174.
  * Lerer et al. (2019) Adam Lerer, Ledell Wu, Jiajun Shen, Timothee Lacroix, Luca Wehrstedt, Abhijit Bose, and Alex Peysakhovich. 2019.  PyTorch-BigGraph: A Large-scale Graph Embedding System. In _Proceedings of the 2nd SysML Conference_. Palo Alto, CA, USA.
  * Martinc et al. (2020) Matej Martinc et al. 2020\.  COVID-19 Therapy Target Discovery with Context-Aware Literature Mining. In _Discovery Science_. Springer International Publishing, Cham, 109–123.
  * McCray et al. (2001) A. T. McCray, A. Burgun, and O. Bodenreider. 2001.  Aggregating UMLS semantic types for reducing conceptual complexity.  _Stud Health Technol Inform_ 84, Pt 1 (2001), 216–220.
  * Neumann et al. (2019) Mark Neumann, Daniel King, Iz Beltagy, and Waleed Ammar. 2019\.  Scispacy: Fast and robust models for biomedical natural language processing.  _arXiv preprint arXiv:1902.07669_ (2019).
  * Nordon et al. (2019) Galia Nordon, G. Koren, V. Shalev, E. Horvitz, and Kira Radinsky. 2019.  Separating Wheat from Chaff: Joining Biomedical Knowledge and Patient Data for Repurposing Medications. In _AAAI_.
  * Oniani et al. (2020) David Oniani, Guoqian Jiang, Hongfang Liu, and Feichen Shen. 2020.  Constructing co-occurrence network embeddings to assist association extraction for COVID-19 and other coronavirus infectious diseases.  _Journal of the American Medical Informatics Association_ 27, 8 (05 2020), 1259–1267.
  * Smalheiser (2017) Neil R Smalheiser. 2017\.  Rediscovering Don Swanson: The past, present and future of literature-based discovery.  _Journal of Data and Information Science_ 2, 4 (2017), 43–64.
  * Spangler (2015) Scott Spangler. 2015\.  _Accelerating Discovery: Mining Unstructured Information for Hypothesis Generation_.  Chapman and Hall/CRC.
  * Spangler et al. (2014) Scott Spangler et al. 2014\.  Automated hypothesis generation based on mining scientific literature. In _Proceedings of the 20th ACM SIGKDD_. 1877–1886.
  * Stanovsky et al. (2018) Gabriel Stanovsky, Julian Michael, Luke Zettlemoyer, and Ido Dagan. 2018.  Supervised Open Information Extraction. In _Proceedings of The 16th Annual Conference of NAACL HLT_.
  * Stewart et al. (2021) Hazel Stewart, Kristoffer H Johansen, Naomi McGovern, Roberta Palmulli, George W Carnell, Jonathan Luke Heeney, Klaus Okkenhaug, Andrew Firth, Andrew A Peden, and James R Edgar. 2021.  SARS-CoV-2 spike downregulates tetherin to enhance viral spread.  _bioRxiv_ (2021), 2021–01.
  * Swanson (1986) Don R Swanson. 1986\.  Fish oil, Raynaud’s syndrome, and undiscovered public knowledge.  _Perspectives in biology and medicine_ 30, 1 (1986), 7–18.
  * Sybrandt et al. (2018a) Justin Sybrandt, Angelo Carrabba, Alexander Herzog, and Ilya Safro. 2018a.  Are Abstracts Enough for Hypothesis Generation?. In _2018 IEEE International Conference on Big Data_. 1504–1513.  https://doi.org/10.1109/bigdata.2018.8621974
  * Sybrandt et al. (2017) Justin Sybrandt, Michael Shtutman, and Ilya Safro. 2017\.  MOLIERE: Automatic Biomedical Hypothesis Generation System. In _Proceedings of the 23rd ACM SIGKDD_ (Halifax, NS, Canada) _(KDD ’17)_. ACM, New York, NY, USA, 1633–1642.  https://doi.org/10.1145/3097983.3098057
  * Sybrandt et al. (2018b) Justin Sybrandt, Micheal Shtutman, and Ilya Safro. 2018b.  Large-Scale Validation of Hypothesis Generation Systems via Candidate Ranking. In _2018 IEEE International Conference on Big Data_. 1494–1503.  https://doi.org/10.1109/bigdata.2018.8622637
  * Sybrandt et al. (2020) Justin Sybrandt, Ilya Tyagin, Michael Shtutman, and Ilya Safro. 2020.  _AGATHA: Automatic Graph Mining And Transformer Based Hypothesis Generation Approach_.  Association for Computing Machinery, New York, NY, USA, 2757–2764.  https://doi.org/10.1145/3340531.3412684
  * Tyagin et al. (2021) Ilya Tyagin, Ankit Kulshrestha, Justin Sybrandt, Krish Matta, Michael Shtutman, and Ilya Safro. 2021.  Accelerating COVID-19 research with graph mining and transformer-based learning.  arXiv:2102.07631 [cs.IR]
  * Wang et al. (2011) Huijun Wang, Ying Ding, Jie Tang, Xiao Dong, Bing He, Judy Qiu, and David J Wild. 2011.  Finding complex biological relationships in recent PubMed articles using Bio-LDA.  _PloS one_ 6, 3 (2011), e17243.
  * Wang et al. (2020) Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, K. Funk, Rodney Michael Kinney, Ziyang Liu, W. Merrill, P. Mooney, D. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Brandon Stilson Stilson, Alex D Wade, Kuansan Wang, Christopher Wilhelm, Boya Xie, Douglas M. Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020\.  CORD-19: The Covid-19 Open Research Dataset.  _ArXiv_ (2020).
  * Wang and Wang (2021) Stephani C Wang and Yu-Feng Wang. 2021.  Cardiovascular protective properties of oxytocin against COVID-19.  _Life Sciences_ (2021), 119130\.
  * Wise et al. (2020) Colby Wise, Vassilis N. Ioannidis, Miguel Romero Calvo, Xiang Song, George Price, Ninad Kulkarni, Ryan Brand, Parminder Bhatia, and George Karypis. 2020.  COVID-19 Knowledge Graph: Accelerating Information Retrieval and Discovery for Scientific Literature.  arXiv:2007.12731 [cs.IR]
  * Zhang et al. (2020) Rui Zhang, Dimitar Hristovski, Dalton Schutte, Andrej Kastrin, Marcelo Fiszman, and Halil Kilicoglu. 2020.  Drug Repurposing for COVID-19 via Knowledge Graph Completion.  arXiv:2010.09600 [cs.CL]
  * Zhou et al. (2019) Yan Zhou, Longtao Huang, Tao Guo, Jizhong Han, and Songlin Hu. 2019.  A Span-based Joint Model for Opinion Target Extraction and Target Sentiment Classification. In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence_. International Joint Conferences on Artificial Intelligence Organization, Macao, China, 5485–5491.  https://doi.org/10.24963/ijcai.2019/762
  * Zimmermann and Curtis (2020) Petra Zimmermann and Nigel Curtis. 2020.  Why is COVID-19 less severe in children? A review of the proposed mechanisms underlying the age-related difference in severity of SARS-CoV-2 infections.  _Archives of Disease in Childhood_ (2020).