#  Searching COVID-19 clinical research using graphical abstracts

Francesco Invernici1 Anna Bernasconi  PhD1,* Stefano Ceri  PhD1

## Abstract

Objective. Graphical abstracts are small graphs of concepts that visually
summarize the main findings of scientific articles. While graphical abstracts
are customarily used in scientific publications to anticipate and summarize
their main results, we propose them as a means for expressing graph searches
over existing literature.

Materials and methods. We consider the COVID-19 Open Research Dataset
(CORD-19), a corpus of more than one million abstracts; each of them is
described as a graph of co-occurring ontological terms, selected from the
Unified Medical Language System (UMLS) and the Ontology of Coronavirus
Infectious Disease (CIDO). Graphical abstracts are also expressed as graphs of
ontological terms, possibly augmented by utility terms describing their
interactions (e.g., “associated with”, “increases”, “induces”). We build a co-
occurrence network of concepts mentioned in the corpus; we then identify the
best matches of graphical abstracts on the network. We exploit graph database
technology and shortest-path queries.

Results. We build a large co-occurrence network, consisting of 128,249
entities and 47,198,965 relationships. A well-designed interface allows users
to explore the network by formulating or adapting queries in the form of an
abstract; it produces a bibliography of publications, globally ranked; each
publication is further associated with the specific parts of the abstract that
it explains, thereby allowing the user to understand each aspect of the
matching.

Discussion and Conclusion. Our approach supports the process of scientific
hypothesis formulation and evidence search; it can be reapplied to any
scientific domain, although our mastering of UMLS makes it most suited to
clinical domains.

Keywords: Clinical research, CORD-19, Graph search, Named entity recognition,
Neo4j, Co-occurrence network, Text mining.

## Objectives

While observing life science articles, we noted that they typically convey the
main research results in the form of graphical abstracts 1, 2, 3, 4; these are
very effective in expressing the main messages, as visual representations are
immediately perceived and complement textual descriptions (e.g., the text of
the abstract and introductory section). Thus, we explored the use of graphical
abstracts also for expressing queries; along 5, we conjectured that the life
science community would particularly benefit from such intuitive and easy
visualization/exploration tools. Our GRAPH-SEARCH system stems from this
motivation, providing both an intuitive visual language to express research
hypotheses and a friendly tool to explore relevant publications, which
highlights the relationships between the original graphical abstracts and an
underlying corpus of scientific evidence.

Since the COVID-19 outbreak in early 2020, important clinical research efforts
have been targeted at understanding the COVID-19 disease. More than one
million studies have been collected within the COVID-19 Open Research Dataset
(CORD-19 6), a corpus of manuscripts created to accelerate the research
against the disease. Their related abstracts represent a wealth of
information, which is -however- unstructured and thus hardly accessible or
searchable. Keyword-based search allows retrieving the documents that contain
some of the words in a target list 7, 8; this kind of search, however, does
not provide visual support, is not suited to expressing complex research
hypotheses, nor to compensating for missing specifications. The development of
frontend tools and visualizations for COVID-19 knowledge graphs has been
motivated in several cases 9, 10; GRAPH-SEARCH responds to this need, by
providing both an intuitive visual language to express COVID-19-related
clinical hypotheses and a friendly tool to explore relevant publications.

The visual language employed to express the user hypothesis takes inspiration
from the graphical abstract of Figure 1. Concepts are represented as nodes
including pictures, short textual descriptions, and abbreviations; directed
edges connect the nodes and are visually characterized by different
thicknesses and colors. Some concepts are associated with modifiers, such as
the small arrows on the left side to indicate the increase or the decrease of
specific effects. Boxes can also be used to group concepts with something in
common, or that are parts of the same sub-hypothesis. For instance, a portion
of the motivating graphical abstract (box on the right) describes the
correlation of the SARS-CoV-2 specific inactivation of ACE2 and the increased
concentration of AngII, leading to the three effects in the box at the bottom
(e.g., increase of vascular permeability).

Capturing all aspects of Figure 1 in a systematic way would be quite
difficult. Our graphical language captures some of them, by using - as query
language - hierarchical structures of concepts, interlinked by relationships,
and then defining a query semantics for extracting scientific evidence (i.e.,
publications) from the corpus, supporting the existence of such relationships.

Figure 1: Graphical abstract of ‘A Machine-Generated View of the Role of Blood
Glucose Levels in the Severity of COVID-19’ 1. Copyright © 2021 Logette _et
al._ , License CC BY 4.0

In order to support this idea, the underlying textual corpus must be first
analyzed and enriched; in our approach, the CORD-19 dataset is expressed in
the form of a co-occurrence network. First, we annotate all the abstracts with
terms from the Unified Medical Language System (UMLS, 11) - covering the
entire spectrum of medicine - and the Ontology of Coronavirus Infectious
Disease (CIDO, 12) - a community-driven open-source biomedical ontology for
COVID-19. Second, we build a comprehensive network that includes all relevant
clinical and biological concepts mentioned in the corpus, linking them based
on their co-occurrence in given abstracts.

Our GRAPH-SEARCH implementation is supported by a graphical interface
(available at http://gmql.eu/graph-search/) that allows to express the queries
and to interpret the results in terms of concepts explained by each discovered
reference, thus enabling the users to better qualify the query during the
interaction; in addition, users can read the textual abstracts of the
retrieved references. Such interactive exploration of the search space is
quite effective not only for exploring assumptions but also for progressively
adapting them as a result of existing evidence.

## Materials and Methods

### The CORD-19 dataset

CORD-19 is a corpus of academic publications about COVID-19 and related
coronavirus research; it was released and maintained by the Allen Institute
for AI, in collaboration with The White House Office of Science and Technology
Policy and other partners. Published articles and preprints were collected
from several archives, including PubMed, PubMedCentral, bioRxiv, and arXiv;
since its release, it has served as the basis of many COVID-19 text mining and
discovery systems 6. The final release of June 2, 2022 indexes more than 1
million publications. As summarized in Figure 2A, nearly 79% of the documents
in CORD-19 have an abstract. Out of them, around 41% have a full-text JSON
file available, while less than 11% of available full-text publications have
no abstract in the metadata table.

Figure 2: (A) Euler-Venn Diagram of the overlap of publications with abstract
with publications with full-text JSON from PDF or from PubMedCentral in
CORD-19. (B) Line plot showing the 10-base logarithm of the number of
publications (y-axis) per publish time date (x-axis).

We decided to focus on dataset records with an abstract. We detected the
language and filtered out those not in English. The file containing the
metadata of the dataset’s publications is a comma-separated table
(CORD-19metadata.csv), including: a unique identifier cord_uid for each
cluster of different records of the same publication, title, abstract,
publish_time, the journal’s abbreviated name, authors, and doi.

Records from CORD-19 are already harmonized 6, resulting in distinct cord_uid
keys; several records of the same publication are included. We deduplicated
them and retained just one record (the one published in a peer-reviewed
journal, if available, else the one richest in metadata). The distribution of
publication times is shown in Figure 2B. Spikes at the beginning of each year
correspond to publications whose publish time includes only the year field.
Publications prior to 2020 concern MERS, SARS, and coronavirus; we removed
them. Fuzzy matching of the abbreviated names was performed with a list of
full names obtained through Scopus 13. We included the number of received
citations (numCitedBy), obtained through SemanticScholar APIs 14.

### Co-occurrence Network

The co-occurrence network is constructed to support graph search; it consists
of entities and relationships mined from the title and abstract fields of the
metadata table. As sources of entities, we considered the UMLS and CIDO
ontologies. Entities of the network include as attributes the Name, optionally
an Umls_id when the entity is extracted from UMLS, and the Frequency
associated with the entity (i.e., number of documents in CORD-19 capturing
that concept).

While CIDO has a simple concept structure, UMLS concepts have a taxonomy that
includes macro-categories at a course level; each macro-category is further
characterized by a type. Currently, we consider the following UMLS macro-
categories: ACTIVITIES_AND_BEHAVIORS, ANATOMY, CHEMICALS_AND_DRUGS,
CONCEPTS_AND_IDEAS, DEVICES, DISORDERS, ENTITY, GENES_AND_MOLECOLAR_SEQUENCES,
GEOGRAPHIC_AREAS, LIVING_BEINGS, OBJECTS, OCCUPATIONS, ORGANIZATIONS,
PHENOMENA, PHYSIOLOGY, and PROCEDURES.

Relationships in the network express the co-occurrence of two entities in one
or more documents of CORD-19. Each relationship has the following attributes:
a Name (built as concatenation in alphabetic order of the names of the
entities that co-occur), a Frequency (the number of abstracts that mention
such co-occurring entities), and several statistical indicators of the
relationship’s strength within the corpus: the PMI value (Pointwise Mutual
Information estimator of the mutual information measured for the co-occurrence
15); the NPMI value (Normalized Pointwise Mutual Information, normalized by
Shannon’s self-information 16); and the Cramer’s V value (measuring the
significativity of the co-occurrence between two entities 17).

Figure 3: Co-occurrence network construction. Ontological terms are recognized
in a textual abstract using entity recognition, then, terms are connected to
each other using entity linking, eventually producing the network.

Figure 3 illustrates the process of ontology creation at a conceptual level.
The process applies to textual abstracts – we consider an excerpt of the text
of the publication providing the graphical abstract of Figure 1 – and consists
of an entity recognition task aiming to extract the known ontological terms
(either from UMLS or from CIDO) followed by an entity linking task;
eventually, we produce a network, whose entities are extracted terms and whose
relationships connect entities that co-occur, weighted by the strength of the
co-occurrence.

### Data provisioning and co-occurrence network construction

Figure 4: GRAPH-SEARCH data provision pipeline. Tasks are performed
sequentially, starting from the raw CORD-19metadata.csv file present in
CORD-19 and producing at the end a Neo4j database storing the co-occurrence
network.

The data provision workflow is represented in Figure 4; it follows the
extract-load-transform (ELT) paradigm. Data is extracted from CORD-19 and
loaded into the GRAPH SEARCH data storage system. We selected MariaDB 18, a
relational engine, for storing preparatory tables, and Neo4j 19, a graph data
engine, for storing the co-occurrence network.

The pipeline produces a co-occurrence network consisting of 128,249 entities
and 47,198,965 relationships, extracted from 662,105 initial publications.
Using the Neo4j Graph Data Science library 20, we verified that the graph is a
unique connected component—such a condition is essential to ensure that every
possible formulated graphical abstract can be matched on the network. In
addition, we produce the reverse index, i.e., a simple postings list whose
keys are the relationships of the network and whose elements are links to the
relevant publications where such relationships co-occur. We next further
describe the last four tasks.

#### Entity Mining.

It takes as input the curated and augmented metadata table and produces the
raw_entity table, performing typical Information Retrieval steps, such as
lexical analysis, removal of stopwords, stemming, and lemmatization. Then we
perform Named Entity Recognition (NER) – consisting of the identification and
extraction of entities from unstructured text and linking to UMLS and CIDO.

#### Entity Curation.

It aggregates the occurrences in the raw_entity table and outputs the
entity_materialized table, collecting all the entities to be employed as nodes
of the co-occurrence network. We exclude entities scoring a low similarity
with UMLS/CIDO concepts, using a normalized string similarity measure based on
the Levenshtein distance, with a threshold set to 0.7. We also include within
entities some utility terms that indicate level modifiers (such as ‘high’,
‘increased’) or causative connectors (i.e., ‘induces’).

#### Link mining.

It uses the raw_entity table and the entity_materialized table to generate the
bigram table (i.e., information on the links of the co-occurrence network) and
the bigram_publications table that we use as reverse index in the information
retrieval process.

#### Graph consolidation.

It selects data from the entity_materialized and bigram tables and migrates it
to the Neo4j instance to create the co-occurrence network. The relationships
of co-occurrence are chosen at this stage, based on their NPMI, which is the
point estimate of the Mutual Information, normalized by the Shannon Self-
Information within [-1,+1]; this compares the probability that the two
entities occur together. We exclude the relationships with NPMI$\leq$0, as a
non-positive NPMI indicates that the relationship is not significant.

### Graphical Abstract Search

A graphical abstract $A$ is a connected graph formed by nodes and undirected
relationships, where nodes are entities of $N$ and $rels(A)$ is a set of
arbitrary relationships connecting some pairs of entities in $A$. Both $A$ and
$N$ are connected graphs with undirected relationships; then, matching $A$
within $N$ can be seen as an instance of inexact graph matching 21.

All entities in $A$ are matched in $N$; then, for each relationship $r$ in
$rels(A)$, connecting nodes $\alpha$ and $\beta$, we retrieve the ‘shortest
paths’ within $N$ that connect $\alpha$ and $\beta$, i.e., a chain of
relationships $r^{\prime}_{1},r^{\prime}_{2}...r^{\prime}_{n}$, where
$r^{\prime}$ is in $rels(N)$, ${r^{\prime}}_{1}$ starts from node $\alpha$,
and ${r^{\prime}}_{n}$ ends in node $\beta$. Candidate shortest paths are
ranked by the average of the NPMI property associated with each relationship
along the path; we retain the top ten paths in the ranking. Shortest paths are
retrieved by using the All Pairs Shortest Path function allShortestPaths
available in Cypher, Neo4j v4.4 19.

We refer to the set of candidate shortest paths as expansion; the selection of
exactly one preferred path among candidates is performed interactively by the
user of the search system, as it is strictly domain/context-specific.

Then, we retrieve the relevant publications that mention concepts of the
matched graph, using the reverse index. We access the reverse index by
relationship name, using either $r$ when it appears in the relationships
$rels(N)$ of the co-occurrence network, or all the relationships
$r^{\prime}_{1},r^{\prime}_{2}...r^{\prime}_{n}$ appearing in the specified
$path(r)$. The score of a publication $P$ relative to an abstract $A$ (i.e.,
the number of explained relationships) is computed as follows:

$Score(P,A)=\sum_{r\in rels(A)}\frac{\sum_{r^{\prime}\in
path(r)}P_{r^{\prime}}}{|path(r)|}$ (1)

The addends of the external summation represent a score assigned to each
relationship $r$ in $A$. Each addend captures how well $P$ represents $r$; it
is equal to 1 if $P$ mentions directly the relationship of $A$ (e.g., when
$path(r)=r^{\prime}$, with length 1) or if $P$ mentions all the relationships
of $path(r)$. Otherwise, it equals a fraction of one, counting the number of
relationships $r^{\prime}_{1},r^{\prime}_{2}...r^{\prime}_{n}$ of $path(r)$
mentioned in $P$, divided by the length of $path(r)$.

Consider Figure 5 as an example of the four steps performed during the search:

  * •

Panel A: Create graphical abstract. Nodes are chosen among the concepts
existing in the network; relationships can be drawn between any pair of nodes.

  * •

Panel B: Find paths. For each pair of entities connected by a relationship in
the graphical abstract, the Neo4j graph is queried to find the shortest paths
(at most ten) with top average NPMI scores.

  * •

Panel C: Select paths. The user selects the most relevant path for each
original relationship that has been expanded.

  * •

Panel D: Retrieve publications and return ranking to the user. The system
collects the names of all the relationships from the expanded graphical
abstract (computed in step B and selected in step C) and exploits them to
retrieve the posting lists of publications (from the reverse index). It ranks
the publications by 1) the number of explained relationships of the original
graphical abstract (see Eq. 1); 2) the _sum_ of NPMI scores of the
relationships; and 3) the publication date.

Figure 5: (A) Example of a graphical abstract. (B) Match of graphical abstract
on the co-occurrence network, with the search of shortest paths (in the dashed
spaces, called expansions). Light green paths have the highest average NPMI of
each expansion. (C) Regardless of the suggested paths, users can select any
path (dark green). (D) A list of publications, ranked by their score, is
extracted; the score is computed using Eq. 1 and considers all the
relationships in the selected paths that are mentioned in the publication.

### Web Interface

The GRAPH-SEARCH application service exposes a web user interface to query the
built network; it contains a backend (web server that exposes a RESTful API
for high-level retrieval operations) and a frontend (user-friendly interface
that exploits the APIs). The frontend is built with the Vue.js framework and
the D3.js library for graph illustrations. The backend is written in Python
and includes 1) a swagger_server which implements the web service logic,
interfaces, and the models necessary to handle the persistence and
asynchronicity behaviors of a multi-user system; and 2) a core which
implements the retrieval strategy and provides high-level programming
interfaces for it.

## Results

We present a use case that emphasizes the strength of exploratory search over
graphs, by supporting users in selecting graph portions, considering/accepting
proposed expansions, and browsing results in terms of NMPI and explained
relationships. Several other use cases are available in the project’s WIKI
documentation.

Pairo-Castineira _et al._ 22 aims to reveal previously undescribed molecular
mechanisms of critical illness in patients with COVID-19 with genome-wide
studies. The results of such studies may provide therapeutic targets to
modulate the host immune response to promote survival. Inspired by this
publication, we create a graphical abstract including relevant human genes
that are related to higher or lower severity of COVID-19 (IFNAR2, CCR2, and
TYK2 genes) and we link them to the change in the severity of the disease (see
Figure 6A).

Figure 6: GRAPH-SEARCH screens dedicated to the use case. (A) Graphical
abstract; (B) Find paths; (C) Select paths; (D) First publication on the
results page.

Since the hypothesis is broad, we start the exploratory process focusing on a
subgraph of the graphical abstract (see nodes in red selected in Figure 6A).
Here, we only consider the effect of the increase of expression in the CCR2
gene. Figure 6B shows how GRAPH-SEARCH expands the path between the concepts
‘High’ and ‘Gene Expression’ (not otherwise connected in the co-occurrence
network). According to NPMI values, the most relevant concept connecting them
is ‘Up-Regulation (Physiology)’. Figure 6C shows that the path going through
this concept has been selected by the user among the other proposed. The
Results page (Figure 6D) shows a publication (Teixeira _et al._ 23) that
covers 4/5 explained relationships of the original graphical abstract. This
means that – out of the five original relationships of the selected portion of
the graphical abstract – only four are explained by the publication (all
except for the one between ‘Gene Expression’ and ‘High’). At this point, the
user can consider other portions of the graphical abstract, or the entire
abstract.

## Discussion

In this section, we first discuss the state of the art, by reviewing classic
approaches to search over networks, including the specific use of bio-
ontologies in information extraction systems and a close comparison with
COVID-19-specific search systems. We next briefly describe the user experience
of GRAPH-SEARCH.

### Network search

The task of searching and retrieving literature over networks with graph-based
queries can be considered through the subproblems that compose it. To query a
network with a graph-like query, a similarity measure between graphs has to be
defined. Existing methods in the context of graph databases include
definitions of graph edit distances and maximum common subgraphs 21, but a
more recent approach introduced a similarity measure based on a graph kernel
between pairs of documents, which exploits the shortest paths between nodes as
units to compare graphs 24. We also mention as a relevant related work G-Bean
25, a graph-based tool that exploits ontologies for graph-based query
expansion to support the user search intention discovery.

### Literature discovery/annotation with bio-ontologies

Bio-ontologies are often exploited to extract structured semantics out of
unstructured biomedical text data, including scientific literature, as
described in the survey by Liu _et al._26. Bio-ontologies are also applied in
natural language processing tasks, like NER 27. Moreover, Wang _et al._ 28
illustrates the application of bio-ontologies in retrieving biomedical
datasets, while Maraver _et al._ 29 emphasizes their role in literature search
facilitation and metadata organization. The potential for refining search
queries through ontology-guided expansion is also a recurring theme in the
biomedical literature for information retrieval; Díaz-Galiano _et al._ 30 and
Dong _et al._ 31 show query expansion methodologies using different medical
vocabularies. Lately, the integration of bio-ontologies with Language Models
has also gained traction within the context of bio-information extraction 32,
33.

### COVID-19-specific literature discovery

Two systems are focused on searching/exploring literature about COVID-19
searchable; they support enhanced keyword-based search but they do not offer
any graph-based search support.

LitCovid 7 was developed within the US National Institutes of Health (NIH) as
a comprehensive resource of literature on COVID-19 (372,221 publications at
the time of writing), continuously updated from PubMed. Publications are
manually screened to determine if they are relevant to COVID-19, they are
assigned to categories (such as overview, disease mechanism, transmission
dynamics, treatment, case report, and epidemic forecasting), associated
geographical location, and annotated with drug or chemical-related information
found in their title/abstract – if applicable. The updated version 34
introduces the long-covid category, adds annotations on variants and vaccines,
and supports topic categorization and entity recognition (with NER). The
interface allows to apply filters on Country, Journal, Drug, Variant, and
Vaccine and compose search strings. LitCovid positively compares its
performances to the classical keyword search of PubMed.

Outbreak.info Research Library 8 offers a searchable interface of COVID-19
publications with clinical trials, datasets, protocols, and other resources.
The data structure upon which the search is performed is supported by a
schema; entities are connected by links with various semantics. The visual
interface allows the use of some filters and keyword search; the R API allows
combining keywords with AND/OR connectors. An advanced query builder –
allowing to mix terms by any combination of AND, OR, and NOT and to compose
exact phrases – is declared as future work.

### User Experience

We performed an informal user study by asking five students in our laboratory
and five researchers external to our group, knowledgeable in biomedical and
clinical science, to explore simple research hypotheses based on personal
curiosity or ongoing research interest. After doing so, we assessed their user
experience along four dimensions.

#### Query language.

The users found it quite easy to translate their intended queries into the
graphical abstract paradigm; they appreciated the possibility of using
UMLS/CIDO concepts, as these are artifacts with which experts in biomedical
domains are quite familiar. Often the needed term could not immediately be
identified, but the concepts’ browser allowed them to find alternative terms
with limited effort. In some cases, they were disappointed by the language’s
implicit positive and conjunctive semantics, e.g., they wanted to confute a
given hypothesis but this is not supported. The use of ‘utility’ terms was
generally appreciated, but their effect on publication selection was not
always as expected.

#### Search process.

Users generally appreciated the overall search process, as steps are rather
intuitively chained one after the other within a search session, and well-
linked to search results. This is particularly true if compared to traditional
keyword-search-based interfaces, where users need to compose several paths in
separate tabs and merge/sort results manually. Most users appreciated the
possibility of being involved in the graph-driven search process in an
intuitive manner.

#### Query expansion.

This feature was positively received, as completions with intermediate nodes
in the expanded paths often suggested connections that were not anticipated by
the users at the time of query design. The identification of these paths along
the co-occurrence network also allowed them to find additional relevant
concepts, mentioned in the relevant abstracts. In many cases, they noted that
expansions contradicted their domain knowledge or simply did not correspond to
their expectations, but it was rather easy to disregard these cases and
continue with their search.

#### Results.

Users commented on the richness of the result set; they reported that the
relevance of found publications with respect to their initial query was from
‘quite high’ to ‘high’. They appreciated the ease of browsing/analyzing the
results, as sorting is made available using different measures (e.g., NPMI,
number of explained relations, and number of citations), and explained
relationships are graphically shown in small sub-graphs, allowing for
immediate inspection.

## Conclusion

GRAPH-SEARCH is the first search engine to propose the exploration of
scientific literature using the ‘graphical abstract’ paradigm. GRAPH-SEARCH
provides several unique features such as the possibility of describing
concepts using well-known ontologies, to establish co-occurrence relationships
between any two concepts of choice, to support search queries with concepts
proposed and ranked by the system, and to browse resulting publications
exploiting several visual and analytical measures.

The completeness and accuracy of the information captured in the co-occurrence
network strictly depend on the advances of the NER methods employed during the
steps of entity mining and linking. Other systems have employed expert
curation (e.g., LitCovid) or community-driven curation (e.g., Outbreak.info).
Although expert curation can improve the search experience, it does not
properly scale; we opted for the exploitation of well-known biomedical
ontologies such as UMLS/CIDO and to trust state-of-the-art NLP models used for
Entity Recognition in our data provision pipeline.

We plan to extend the graphical abstract search paradigm to knowledge graphs
with both entities and relationships, thereby enriching the expressivity of
graph queries. Then, we aim to formalize the use of graphical abstracts in the
context of graph databases, by approaching approximate graph search in the
context of subgraph matching, shortest path search, and conjunctive query
processing. We also aim to conduct extensive empirical studies, to measure the
effectiveness of graphical-abstract-search in terms of equal language
expressivity and richness/appropriateness of proposed results, and the user
satisfaction, analyzed along the dimensions of usability and support of
knowledge exploration.

## Data and code availability

The data processing pipeline is available as a Docker image at
https://hub.docker.com/r/frinve/graph-search. The application is available at
http://gmql.eu/graph-search. The WIKI with use cases is at
https://github.com/FrInve/graph-search/wiki/Use-Cases/.

## Acknowledgment

The authors would like to thank Luca Minotti for implementing the frontend of
the Web application.

## Conflict of interest statement

None declared.

## Funding

This paper is supported by PNRR-PE-AI FAIR project funded by the
NextGeneration EU program.

## References

  * 1 Logette E, Lorin C, Favreau C, Oshurko E, Coggan JS, Casalegno F, et al.  A machine-generated view of the role of blood glucose levels in the severity of COVID-19.  Frontiers in Public Health. 2021:1068.
  * 2 Grifoni A, Sidney J, Zhang Y, Scheuermann RH, Peters B, Sette A.  A sequence homology and bioinformatic approach can predict candidate targets for immune responses to SARS-CoV-2.  Cell Host & Microbe. 2020;27(4):671-80.
  * 3 Jackson B, Boni MF, Bull MJ, Colleran A, Colquhoun RM, Darby AC, et al.  Generation and transmission of interlineage recombinants in the SARS-CoV-2 pandemic.  Cell. 2021;184(20):5179-88.
  * 4 Singh R, Nagpal S, Pinna NK, Mande SS.  Tracking mutational semantics of SARS-CoV-2 genomes.  Scientific Reports. 2022;12:15704.
  * 5 Ware C.  Visual queries: The foundation of visual thinking.  In: Knowledge and information visualization: Searching for synergies. Springer; 2005. p. 27-35.
  * 6 Wang LL, Lo K, Chandrasekhar Y, Reas R, Yang J, Eide D, et al.  CORD-19: The COVID-19 Open Research Dataset.  arXiv preprint arXiv:200410706. 2020.
  * 7 Chen Q, Allot A, Lu Z.  LitCovid: an open database of COVID-19 literature.  Nucleic Acids Research. 2021;49(D1):D1534-40.
  * 8 Tsueng G, Mullen JL, Alkuzweny M, Cano M, Rush B, Haag E, et al.  Outbreak.info Research Library: A standardized, searchable platform to discover and explore COVID-19 resources.  Nature Methods. 2023;20(4):536-40.
  * 9 Kejriwal M.  Knowledge graphs and COVID-19: opportunities, challenges, and implementation.  Harvard Data Science Review. 2020;(Special Issue 1).
  * 10 Peng J, Xu D, Lee R, Xu S, Zhou Y, Wang K.  Expediting knowledge acquisition by a web framework for Knowledge Graph Exploration and Visualization (KGEV): case studies on COVID-19 and Human Phenotype Ontology.  BMC Medical Informatics and Decision Making. 2022;22(Suppl 2):147.
  * 11 Bodenreider O.  The unified medical language system (UMLS): integrating biomedical terminology.  Nucleic Acids Research. 2004;32(Suppl 1):D267-70.
  * 12 He Y, Yu H, Ong E, Wang Y, Liu Y, Huffman A, et al.  CIDO, a community-based ontology for coronavirus disease knowledge and data integration, sharing, and analysis.  Scientific Data. 2020;7:181.
  * 13 Rose ME, Kitchin JR.  pybliometrics: Scriptable bibliometrics using a Python interface to Scopus.  SoftwareX. 2019;10:100263.
  * 14 Silva D, Rohatgi S. semanticscholar; 2023.  Last accessed online: Sept 15th, 2023.  https://github.com/danielnsilva/semanticscholar.
  * 15 Church KW, Hanks P.  Word Association Norms, Mutual Information, and Lexicography.  Computational Linguistics. 1990;16(1):22-9.
  * 16 Bouma G.  Normalized (pointwise) mutual information in collocation extraction.  Proceedings of Conferences of the German Society for Computational Linguistics and Language Technology (GSCL). 2009;30:31-40.
  * 17 Cramér H.  Mathematical methods of statistics. vol. 43.  Princeton university press; 1999.
  * 18 MariaDB Foundation. MariaDB; 2023.  Last accessed online: Sept 15th, 2023.  https://mariadb.org/.
  * 19 Neo4j. Neo4j Graph Database; 2023.  Last accessed online: Sept 15th, 2023.  https://neo4j.com/.
  * 20 Neo4j. Neo4j Graph Data Science; 2023.  Last accessed online: Sept 15th, 2023.  https://github.com/neo4j/graph-data-science.
  * 21 Zhu Y, Qin L, Yu JX, Cheng H.  Finding top-k similar graphs in graph databases.  In: Proceedings of the 15th International Conference on Extending Database Technology; 2012. p. 456-67.
  * 22 Pairo-Castineira E, Clohisey S, Klaric L, Bretherick AD, Rawlik K, Pasko D, et al.  Genetic mechanisms of critical illness in COVID-19.  Nature. 2021;591(7848):92-8.
  * 23 Teixeira PC, Dorneles GP, Santana Filho PC, da Silva IM, Schipper LL, Postiga IA, et al.  Increased LPS levels coexist with systemic inflammation and result in monocyte activation in severe COVID-19 patients.  International Immunopharmacology. 2021;100:108125.
  * 24 Nikolentzos G, Meladianos P, Rousseau F, Stavrakas Y, Vazirgiannis M.  Shortest-Path Graph Kernels for Document Similarity.  In: Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark: Association for Computational Linguistics; 2017. p. 1890-900.
  * 25 Wang JZ, Zhang Y, Dong L, Li L, Srimani PK, Yu PS.  G-Bean: an ontology-graph based web tool for biomedical literature retrieval.  BMC Bioinformatics. 2014;15(Suppl 12):S1.
  * 26 Liu F, Chen J, Jagannatha A, Yu H.  Learning for biomedical information extraction: Methodological review of recent advances.  arXiv preprint arXiv:160607993. 2016.
  * 27 Gao S, Kotevska O, Sorokine A, Christian JB.  A pre-training and self-training approach for biomedical named entity recognition.  PloS One. 2021;16(2):e0246310.
  * 28 Wang X, Huang Z, van Harmelen F.  Ontology-based semantic similarity approach for biomedical dataset retrieval.  In: Health Information Science: 9th International Conference, HIS 2020, Amsterdam, The Netherlands, October 20–23, 2020, Proceedings 9. Springer; 2020. p. 49-60.
  * 29 Maraver P, Armañanzas R, Gillette TA, Ascoli GA.  PaperBot: open-source web-based search and metadata organization of scientific literature.  BMC Bioinformatics. 2019;20:50.
  * 30 Díaz-Galiano MC, Martín-Valdivia MT, Ureña-López L.  Query expansion with a medical ontology to improve a multimodal information retrieval system.  Computers in Biology and Medicine. 2009;39(4):396-403.
  * 31 Dong L, Srimani PK, Wang JZ.  Ontology graph based query expansion for biomedical information retrieval.  In: 2011 IEEE International Conference on Bioinformatics and Biomedicine. IEEE; 2011. p. 488-93.
  * 32 Fei H, Ren Y, Zhang Y, Ji D, Liang X.  Enriching Contextualized Language Model from Knowledge Graph for Biomedical Information Extraction.  Briefings in Bioinformatics. 2021 May;22(3):bbaa110.
  * 33 Kalyan KS, Rajasekharan A, Sangeetha S.  AMMU: a survey of transformer-based biomedical pretrained language models.  Journal of Biomedical Informatics. 2022;126:103982.
  * 34 Chen Q, Allot A, Leaman R, Wei CH, Aghaarabi E, Guerrerio JJ, et al.  LitCovid in 2022: an information resource for the COVID-19 literature.  Nucleic Acids Research. 2023;51(D1):D1512-8.