# Kirigami: large convolutional kernels improve deep learning-based RNA
secondary structure prediction

Marc Harary
Department of Data Science
Dana-Farber Cancer Institute
Boston, MA 02115, USA
<EMAIL_ADDRESS>&Chengxin Zhang
Department of Computational Medicine and Bioinformatics
University of Michigan
Ann Arbor, MI 48109, USA
<EMAIL_ADDRESS>
###### Abstract

We introduce a novel fully convolutional neural network (FCN) architecture for
predicting the secondary structure of ribonucleic acid (RNA) molecules.
Interpreting RNA structures as weighted graphs, we employ deep learning to
estimate the probability of base pairing between nucleotide residues. Unique
to our model are its massive 11-pixel kernels, which we argue provide a
distinct advantage for FCNs on the specialized domain of RNA secondary
structures. On a widely adopted, standardized test set comprised of 1,305
molecules, the accuracy of our method exceeds that of current state-of-the-art
(SOTA) secondary structure prediction software, achieving a Matthews
Correlation Coefficient (MCC) over 11-40% higher than that of other leading
methods on overall structures and 58-400% higher on pseudoknots specifically.

##  1 Introduction

In the context of ribonucleic acids (RNAs), secondary structure prediction
refers to the inference of the set of nucleotide base pairs corresponding to a
given primary structure [24]. Close parallels exist to contact prediction in
protein structure modeling, where deep learning methods have been successful
[45, 18]. On the other hand, the comparative scarcity of high-quality data
poses a significant challenge for the computational analysis of RNA secondary
structure [36], which has prompted the development of a wide gamut of
algorithms that draw from evolutionary biology, thermochemistry, statistics,
and deep learning to circumvent this limitation [36, 28, 46, 21, 27, 49, 41,
31].

Covariance techniques like R-scape [28] and PLMC [46] exploit comparative
phylogenetic analysis to identify pairwise co-occurrence of structural motifs
across multiple sequence alignments, though the shortage of available data
limits the reliability with which evolutionarily conserved structures can be
identified. Thermodynamic methods like RNAfold from ViennaRNA [21],
RNAstructure [27], ContextFold [47], and mfold [49] instead assign annotations
that minimize free energy (MFE) estimated from parameters in the Nearest
Neighbors database (NNDB) [42]. Their primary weakness lies in assuming a
strict equivalence between the empirically likeliest structure and the most
energetically stable, given that a non-trivial minority of molecules (e.g.,
snoRNAs) exist that do not occupy MFE states [34]. Moreover, thermochemical
algorithms typically employ dynamic programming [2], which cannot predict non-
nested base pairs and thereby diminishes accuracy on pseudoknotted molecules
[22] (Figure 2G-H). An additional class of traditional RNA algorithms,
stochastic context-free grammars like those of PPfold [41], borrow from
computational linguistics to model structural motifs like hairpin loops and
bulges as syntactical elements of a formal language, although they are also
highly dependent on empirical data for statistical estimates [34] and limited
to nested structures [22]. Finally, statistical models like IPknot [32] employ
other means of scoring secondary structures based on empirical likelihood to
estimate the maximum expected accuracy (MEA) solution, with some methods like
PETfold [34] unifying multiple approaches.

Recently, several pipelines based on deep learning have obtained results in
many cases superior to those of the foregoing [36, 31, 9]. SPOT-RNA [36]
employs transfer learning [25] and an ensemble of models composed of long
short-term memory [11, 33], residual neural network (ResNet) [10], and
multilayer perceptron [35] layers to achieve relatively high accuracy on both
nested and non-nested structures. Interpolating between biophysical and neural
algorithms, MXfold2 [31] consists of a smaller, though architecturally similar
model with convolutional [15] layers trained via a sophisticated loss function
that accounts for the NNDB parameters. While its performance on nested
structures is accordingly high, its reliance on dynamic post-processing limits
performance on pseudoknots. E2Efold’s [5] end-to-end differentiable algorithm
obtains poor results in comparison to approaches based either in deep learning
or otherwise [31, 9]. Most recently, UFold [9] achieves competitive accuracy
by means of convolutional and deconvolutional layers inspired by UNet [29], a
model originally designed for segmenting medical images.

Departing from the former architectures, the following work introduces
Kirigami, a fully convolutional network (FCN) [20] trained on secondary
structure annotations and architecturally inspired by recent developments in
computer vision [19]. It has been demonstrated that larger kernel sizes and an
absence of pooling and deconvolutional layers yield state-of-the-art (SOTA)
results for image classification [19], at least in some cases even
outperforming attention-based models like vision transformers [8]. By adapting
these innovations and expanding kernel sizes even further beyond those of most
existing convolutional networks for computer vision, we create a powerful
model for RNA secondary structure prediction that exceeds the performance of
several SOTA algorithms on a well-established test set [36, 31, 9]. Kirigami
is an open-source program under the MIT license with source code and training,
validation, and testing datasets available at https://github.com/marc-
harary/kirigami.

Figure 1: Flowchart of the Kirigami pipeline. Green nodes represent input-
output values, yellow miscellaneous operations, and blue trainable neural
network layers (or closely associated activation functions). Dashed red boxes
indicate network submodules. Constraints (I-IV) are indicated in their
corresponding nodes.

##  2 Results

###  2.1 Datasets

Kirigami was trained and tested on the datasets collected by the authors of
SPOT-RNA [36] from the bpRNA database [7], presently the largest collection of
RNA secondary structure annotations. From a range of families that include,
among other categories, tRNA, tmRNA, signal recognition particle RNA, and
RNase P RNA molecules, 13,419 samples were collected after filtration by CD-
HIT-EST [17] for $<80\%$ sequence identity to ensure non-redundancy. The
molecules were further divided into sets of 10,814, 1,300 and 1,305 for
training, validation, and testing and denoted TR0, VL0, and TS0, respectively.
We note that samples from the same RNA families were included in both TR0 and
TS0, unlike in some datasets like bpRNA-new used by UFold [9] and MXfold2
[31].

It should also be noted that higher resolution structures like those in the
Protein Databank (PDB) [3] are indeed available and have been used for
benchmarking past models like SPOT-RNA, for which a dataset of approximately
200 molecules was collated. However, we found that the comparative scarcity of
these data rendered obtaining high model consistency—and hence performing
reliable fine-tuning—prohibitively difficult. Model-wise differences in
performance were likewise unlikely to yield statistically significant results,
prompting our use of the bpRNA database for more robust and reproducible
experiments.

###  2.2 Overview of pipeline

Similar to previous studies [36, 9], we interpret RNA secondary structure
prediction as an image segmentation task in which 3D input tensors are
hierarchically filtered and mapped to outputs of equal height and width.
Analogous to pixels in a multichannel raster image, each element in the input
corresponds to a particular pair of nucleotides, while the output base-pairing
probabilities are analogous to a single-channel grayscale picture (Figure
2A-C). Correspondingly, the backbone of the Kirigami pipeline consists of a
32-block fully convolutional network (FCN) with residual skip connections and,
in our case, no pooling layers or dilated kernels. Our network is unique in
the size of its convolutional filters; kernel sizes were set to 11 rather than
the more standard values of 3 or 5 [36, 32, 9, 10, 15, 20]. A secondary post-
processing module imposes several constraints on the matrix output by the
network to remove pairs that (I) are non-symmetric, (II) are non-canonical,
(III) create sharp angles, or (IV) form multiplets (see Materials and
Methods).

Figure 2: Stages for post-processing using bpRNA_RFAM_35590 as an example. A.
Constraint I: symmetric predicted base-pairing probabilities
$\left(\mathbf{\hat{y}}_{1}\right)$ in log scale. The receptive field of an
kernel is indicated in the red dashed box. B. Adjacency matrix of the ground
truth $\left(\mathbf{y}\right)$. C. Final predicted adjacency matrix
$\left(\mathbf{\hat{y}}_{3}\right)$. D. Constraints II: Removal of sharp
angles $\left(\mathbf{\hat{y}}_{2}\right)$. In this case none are present. E.
Constraint III: Removal of non-canonical pairs
$\left(\mathbf{\hat{y}}_{3}\right)$. In this case none are present. F.
Constraint IV: Removal of multiplets $\left(\mathbf{\hat{y}}_{4}\right)$—in
this case none are present—and thresholding
$\left(\mathbf{\hat{y}}_{5}\right)$. G. Ball-and-stick representation of the
ground truth $\left(\mathbf{y}\right)$. Positions assigned using Pseudoviewer
[4]. H. Ball-and-stick representation of the final, thresholded structure
$\left(\mathbf{\hat{y}}_{5}\right)$.

###  2.3 Comparison with existing methods

On TS0, Kirigami achieves an average MCC of 0.706, which is 11.4%, 17.0%,
464.4%, 26.6%, 41.6%, 36.6%, 45.1%, 34.8%, and 40.7% higher than the MCCs of
UFold [9], SPOT-RNA [36], E2Efold [5], MXfold2 [31], RNAfold [21], PETfold
[34], PPfold [41], IPknot [32], and ContextFold [47], respectively (Table 1,
Figure 3A). Improvement is for the most part more pronounced on molecules with
pseudoknots, where the average MCC is 9.6%, 24.7%, 528.2%, 38.7%, 63.4%,
52.2%, 62.6%, 61.4%, and 37.9% higher, respectively. All differences are
statistically significant $\bigr{(}$Table 1, $p<1.76\times 10^{-11}\bigl{)}$.

Table 1: Performance Comparison of RNA Folding Algorithms

Dataset |  Kirigami |  UFold |  SPOT-RNA |  E2EFold |  MX2fold
---|---|---|---|---|---
Fully nested (N=1,176) |  0.710 |  0.636 |  0.611 |  0.127 |  0.566
Pseudoknotted (N=129) |  0.669 |  0.611 |  0.537 |  0.107 |  0.482
All (N=1,305) |  0.706 |  0.634 (1.76E-11) |  0.604 (5.40E-22) |  0.125 ($\sim$ 0) |  0.558 (4.62E-40)

Dataset |  RNAfold |  PETfold |  PPfold |  IPknot |  ContextFold
---|---|---|---|---|---
Fully nested (N=1,176) |  0.509 (2.90E-75) |  0.526 (4.32E-65) |  0.495 (4.38E-83) |  0.536 (1.54E-61) |  0.504 (1.65E-70)
Pseudoknotted (N=129) |  0.410 (2.90E-75) |  0.440 (4.32E-65) |  0.412 (4.38E-83) |  0.415 (1.54E-61) |  0.485 (1.65E-70)
All (N=1,305) |  0.499 (2.90E-75) |  0.517 (4.32E-65) |  0.487 (4.38E-83) |  0.524 (1.54E-61) |  0.502 (1.65E-70)

When considering pseudoknots in isolation from nested pairs, Kirigami obtains
a mean MCC of 0.615, which is 58.8%, 116.6%, 384.3%, 213.0%, 268.3%, 251.1%,
401.7%, 309.4% and 544.8% higher than its competitors (Table 3, Figure 3B).
The most dramatic difference lay in median MCC; Kirigami obtained a score of
0.797, whereas each of its alternatives exhibited a median less than or equal
to 0.

Table 2: Summary statistics for MCC on pseudoknot pairs in TS0 by model.

Statistic |  Kirigami |  UFold |  SPOT-RNA |  E2Efold |  MXfold2
---|---|---|---|---|---
Mean |  0.615 |  0.387 |  0.284 |  0.127 |  0.196
Median |  0.797 |  0.00 |  0.00 |  -1.13E-3 |  -1.41E-3

Statistic |  RNAfold |  PETfold |  PPfold |  IPknot |  ContextFold
---|---|---|---|---|---
Mean |  0.167 |  0.175 |  0.122 |  0.150 |  0.095
Median |  -1.87E-3 |  -1.73E-3 |  0.00 |  0.00 |  -1.25E-3

###  2.4 Ablation studies

To quantify the effect of kernel size on performance, we retrained otherwise
identical networks with varying kernel sizes. In other words, fixing
$N_{\text{channels}}=N_{\text{blocks}}=32$ (see Materials and Methods) and the
training procedure described below, networks were trained with smaller kernel
sizes of $k=9$, $k=7$, and $k=5$. Respectively, these resulted in MCCs of
0.680 $\left(p=1.35\times 10^{-2}\right)$, 0.681 $\left(p=1.62\times
10^{-2}\right)$, and 0.654 $\left(9.87\times 10^{-7}\right)$, equivalent to
percent decreases of 3.8%, 3.6%, and 7.4% with respect to the default
configuration (0.706, Table 2). One confounding factor, however, is network
size; larger kernels entail a larger number of total parameters and in turn
higher model capacity. We therefore tested a network with
$N_{\text{channels}}=32$, $N_{\text{blocks}}=64$, and $k=5$, which had
approximately 3.3 million parameters in comparison to the 3.2 million
parameters of the network with hyperparameters corresponding to
$N_{\text{channels}}=32$, $N_{\text{blocks}}=32$ and $k=7$. This modified
configuration exhibited a 5.42% decrease in MCC relative to the default
configuration from 0.706 to 0.668. Hence, the data consistently indicate that
kernel size—rather than total parameter number per se—improve prediction
accuracy.

Table 3: Mean MCC on TS0 for Kirigami by kernel size and number of blocks. $\mathbf{\left(k,N_{\text{blocks}}\right)}$ |  (11, 32) |  (9, 32) |  (7, 32) |  (5, 32) |  (5, 64)
---|---|---|---|---|---
Fully nested (N=1,176) |  0.710 |  0.687 |  0.687 |  0.663 |  0.674
Pseudoknotted (N=129) |  0.669 |  0.617 |  0.625 |  0.572 |  0.616
All (N=1,305) |  0.706 |  0.680 |  0.681 (1.35E-2) |  0.654 (9.18E-7) |  0.668 (3.06E-4)

###  2.5 Biophysical analysis

Using a software tool available in the ViennaRNA [21] suite, we computed the
average Gibbs free energy per mole per base (kJ/mol/b) as approximated by NNDB
[42] of both the ground truth secondary structures and those predicted by each
SOTA model, then computed the difference $\Delta G$ between the two estimates.
Because multiplets or sharp angles either cause the program to crash or return
a meaninglessly high output, they were removed (only for biophysical analysis)
from both ground truth structures and those predicted by models that do not
natively enforce such constraints. E2Efold [5] on average massively
underestimates ground truth stability, whereas the inverse holds for MXfold2
[31], RNAfold [21], PETfold [34], IPknot [32], and ContextFold [47]. Only the
neural models Kirigami, UFold [9], and SPOT-RNA [36], along with the maximum
expected accuracy (MEA) model PPfold [41], exhibit medians approximately equal
to the ground truth (Figure 4A).

We then regressed the model-wise MCC onto $\Delta G$ (Figure 4B). Expectedly,
the correlations were highest for the MFE models, namely RNAfold
$\left(r=0.528,\ p=1.11\times 10^{-94}\right)$, PETfold
$\left(r=0.459,p=5.35\times 10^{-69}\right)$, and MXfold2 $\left(r=0.583,\
p=1.44\times 10^{-119}\right)$, with ContextFold $\left(r=0.313,\ p=5.61\times
10^{-31}\right)$ being an exception. Lower (albeit still statistically
significant) correlations were observed for Kirigami $\left(r=0.251,\
p=3.84\times 10^{-20}\right)$, UFold $\left(r=0.293,\ p=2.91\times
10^{-27}\right)$, and SPOT-RNA $\left(r=0.270,\ p=2.78\times 10^{-23}\right)$.
As to the MEA algorithms, PPfold exhibited the lowest correlation
$\left(r=0.118,\ p=1.91\times 10^{-5}\right)$ while IPknot’s correlation
$\left(r=0.292,\ p=5.58\times 10^{-27}\right)$ was expectedly closer to those
of the neural models than to its biophysical counterparts.

Figure 3: A. Difference between the Gibbs free energy ($\Delta G$) of the
predicted structure and that of the ground truth by model as predicted by
ViennaRNA. B. MCC versus $\Delta G$ by model with the corresponding
correlation coefficient.

##  3 Discussion and Conclusion

In this work, we develop Kirigami, a deep learning model for RNA secondary
structure prediction. On a large benchmark dataset of 1,305 RNA structures
from the bpRNA [7] library, it achieves a mean MCC of 0.706, which is 11.4%,
26.6%, and 17.0% higher than the SOTA methods UFold [9], MXfold2 [31], and
SPOT-RNA [36], respectively, with an MCC of 0.615 on pseudoknots that is
58.8%, 213.0%, and 116.6% higher, respectively.

Along with these three neural networks, Kirigami significantly outperforms
other secondary structure prediction algorithms and particularly MFE models,
the growing superiority of neural to biophysical models paralleling the
“neuralization” of many aspects of protein structure prediction [1]. The
causes of the discrepancy are unclear; given that neural networks act as
universal function approximators [12, 48], an outstanding question in
secondary structure prediction is the nature of the function estimated when
models are trained on large datasets. One possibility is that neural networks
learn the energy landscape of RNA structures and are simply doing so to a
greater degree of accuracy than is possible via the relatively small number of
parameters that inform most biophysical models [21, 27, 49, 47, 42]. As is
shown in Figure 3A, UFold, SPOT-RNA, and Kirigami predict structures that not
only exhibit the highest MCC score, but also the smallest values of $\Delta
G$.

However, such improvements cannot be explained by neural models simply being
more capable of optimizing the landscape more efficiently, given that most MFE
models still typically overestimate molecular stability. Nor can it be the
case that Kirigami and other neural algorithms outperform biophysical methods
by avoiding an inductive bias towards strictly energetically minimizing
solutions—which, as noted above, can often limit MFE algorithms [34]—given
that correlations between the accuracy of the thermodynamic stability of the
networks’ predictions and their MCCs are lower. Thermodynamic accuracy, in
other words, cannot in general mediate between model type and performance on
secondary structure prediction. The nature of the mappings learned by Kirigami
and other neural models may therefore remain a topic of interest for future
inquiry.

As to the optimal architecture for neural networks in this domain, we
hypothesize that large-kernel fully convolutional networks [20] are ideal for
the one of the most challenging aspects of RNA secondary structure prediction,
namely the unique topology of output labels. Due to base stacking, most
molecules consist of a small quantity of large stems separated by unstructured
loops [7, 3], and due also to the rarity of multiplets, most hydrogen bonds
will be isomorphic to long, linear neighborhoods of contiguous positive labels
that are one pixel in width when the structure is embedded as an image-like
tensor [36, 9]. While large feature patches are not uncommon in many computer
vision tasks, image segmentation does not typically involve the detection of
wider, two-dimensional connected components characterized by such high spatial
frequency and fine detail. Unusually large convolutional kernels like those in
Kirigami may thus be more effective at delineating stems from nearby
unstructured regions of molecules than kernels with commoner [36, 32, 9, 10,
15, 20], smaller receptive fields that can only detect more localized
structure. For example, in Figure 2A-C, molecule bpRNA_RFAM_35590 has a stem
consisting of 6 bonds that, along with up to five bases of the surrounding
hairpin loop, can fit well within the boundaries of an 11-pixel kernel.
Kirigami is hence able to detect both the location and extent of the loop with
high accuracy, exhibiting low activations for the edges surrounding the stem
(Figure 2A). Similarly, pooling layers for down-sampling, in spite of their
utility for models like U-Net [29] designed for more standard image
segmentation tasks, may be less adept at precisely locating the narrow strips
of positive labels that are characteristic of embedded RNA secondary
structures.

Despite the success of Kirigami’s simple ResNet architecture and the recent
advances in convolutional neural networks for computer vision, transformer-
based models [44] have proliferated in many other fields of computational
biology [13]. Future directions for research into neural approaches for RNA
structure may ultimately include attention mechanisms, which have proven
highly performant in protein structure prediction tasks [13].

Still, the most challenging aspect of RNA computational biology is the paucity
of high-quality data [36]. As shown in the recent CASP15 challenge, deep
learning approaches are not yet up to the accuracy of statistical or empirical
methods for many structure prediction tasks. Recent improvements in RNA
experimental structure determination techniques such as cryogenic electron
microscopy [3] will eventually expand existing databases and enable the
development of more sophisticated machine learning models.

##  4 Materials and methods

###  4.1 Input features and output labels

More formally, we consider the structure of a molecule to be a weighted graph
in which edges represent base pairing probabilities (Figure 2D-F) and nodes,
embedded as feature vectors, represent nucleotide residues. Secondary
structure prediction thus amounts to inferring the corresponding adjacency
matrix, with intermediate, “hidden” representations of the molecule being
updated via permutation-invariant layers in the form of two-dimensional
convolutions. The topologies of the network and the tensors that it
manipulates, however, remain identical to those found in standard computer
vision models.

As input features, each $i$th node $\mathbf{h}_{i}$ in the molecular graph
consists of the nucleic acids adenine (A), cytosine (C), guanine (G), and
uracil (U) represented by the standard basis vectors $\mathbf{e}_{1}$,
$\mathbf{e}_{2}$, $\mathbf{e}_{3}$, and $\mathbf{e}_{4}$, respectively, while
degenerate bases [6] are mapped to the corresponding averages thereof
$\bigl{(}$e.g., ketones are represented by
$\frac{1}{2}\left(\mathbf{e}_{1}+\mathbf{e}_{4}\right)\bigr{)}$. The
relationship between each pair of nucleotides $\left(i,j\right)$, in order to
be iteratively refined by the network into a corresponding base pairing
probability $\hat{y}_{ij}$, is initially represented by simply concatenating
$\mathbf{h}_{i}$ and $\mathbf{h}_{j}$ to a form a vector
$\mathbf{x}_{ij}\in\mathbb{R}^{8}$. Therefore, the final input feature for all
$L^{2}$ ordered pairs of nucleotides is a tensor
$\mathbf{x}\in\mathbb{R}^{8\times L\times L}$, analogous to a square eight-
channel image.

As an output label, the RNA secondary structure can be straightforwardly
embedded as a binary adjacency matrix
$\mathbf{y}\in\left\\{0,1\right\\}^{L\times L}$, where $y_{ij}=1$ if and only
if the $i$th and $j$th nucleotides form a base pair.

###  4.2 Network architecture

The Kirigami pipeline consists of a 32-block residual [10] fully convolutional
network (FCN) [20], meaning that it is does not contain multilayer perceptron
[35] or recurrent neural network [11, 33] layers. The principal
hyperparameters of the network consist of the triple
$\left(N_{\text{channels}},\ N_{\text{blocks}},\ k\right)$, where
$N_{\text{channels}}$ denotes the number of intermediate input channels in the
network, $N_{\text{blocks}}$ the number of ResNet blocks [10], and $k$ the
size of the kernels of the convolutions in each block. Irrespective of the
hyperparameters, the first two layers consist of a convolution by a $1\times
1$ kernel followed by ReLU activation, which expand the input tensor
$\mathbf{x}\in\mathbb{R}^{8\times L\times L}$ to an intermediate feature map
$\mathbf{z}^{(1)}\in\mathbb{R}^{N_{\text{channels}}\times L\times L}$. Each
message-passing block resembles that of a typical ResNet insofar as each is
comprised of two 2D convolutions—both with $k\times k$ kernels and followed by
ReLU activation and 2D instance normalization [43]—with a single dropout layer
separating these two halves. Denoting the output of the $i$th block
$\mathbf{z}^{(i+1)}\in\mathbb{R}^{N_{\text{channels}}\times L\times L}$, a
residual skip connection adds the initial input to the block
$\mathbf{z}^{(i)}\in\mathbb{R}^{N_{\text{channels}}\times L\times L}$ back to
the output of the second normalization layer (Figure 1). The final layers of
the network consist of a $1\times 1$ convolution and sigmoid activation to
reduce the dimension and range of the output to the final label
$\mathbf{\hat{y}}_{0}\in\left[0,1\right]^{L\times L}$. All convolutions have
strides and dilations of 1 [15].

Grid search, as constrained by hardware limitations, revealed that optimal
performance was obtained by setting $N_{\text{channels}}$ to 32,
$N_{\text{blocks}}$ to 32, and $k$ to 11.

###  4.3 Post-processing

RNA contact maps [40, 37] bear a close resemblance to, but are ultimately
distinct from, secondary structure adjacency matrices. Whereas in the former,
any pair of nucleotides can be considered in contact so long as their
interatomic distance is within 8Å, two nucleotides are considered paired in
the latter if and only if they are paired by specific hydrogen bonding
interactions. Hence, predicted RNA secondary structures must obey certain
constraints [36], namely that, if nucleotides $i$ and $j$ are paired, they are
assumed to exhibit:

  1. (I)

Symmetry: $j$ is also paired to $i$;

  2. (II)

Minimal loop length: $\left|i-j\right|\geq 4$;

  3. (III)

Watson-Crick or Wobble pairing: the nucleotide pair must be A:U, C:G, or G:U;

  4. (IV)

Absence of multiplets: neither nor can be paired with any other base.

These collectively necessitate constraint-satisfaction [16] via post-
processing. To enforce Constraint (I), the output matrix
$\mathbf{\hat{y}}_{0}$ is simply averaged with its own transpose, yielding a
new matrix
$\mathbf{\hat{y}}_{1}:=\frac{1}{2}\left(\mathbf{\hat{y}}_{0}+\mathbf{\hat{y}}_{0}^{\intercal}\right)$
(Figure 2A), while satisfying Constraint (II) is straightforwardly
accomplished by zeroing out the entries of the upper and lower diagonals to
produce another intermediate matrix $\mathbf{\hat{y}}_{2}$ (Figure 2D displays
$\mathbf{\hat{y}}_{2}$ as a graph).

Canonicalization was enforced by encoding the bases A, C, G, and U as the
prime numbers 2, 3, 5, and 7, respectively. Prime number-encoding converts the
RNA sequence into a vector $\mathbf{v}\in\left\\{2,3,5,7\right\\}^{L}$ such
that the matrix $\mathbf{M}:=\mathbf{v}\otimes\mathbf{v}$ uniquely represents
each base pair permutation, with 14, 15, and 35 corresponding to A:U, C:G, and
G:U pairs, respectively. Enforcing (III) can thus be performed by applying the
differentiable entry-wise matrix multiplication
$\mathbf{\hat{y}}_{3}:=\mathbf{\hat{y}}_{2}\odot\mathds{1}_{\left\\{14,15,25\right\\}}\left(\mathbf{M}\right)$,
where $\mathds{1}_{\left\\{14,15,25\right\\}}$ denotes the indicator function
associated with the three canonical pairs in the prime number-encoding (Figure
2E).

Somewhat similar to the post-processing procedure used by SPOT-RNA [36],
Constraint (IV) is then enforced by a greedy algorithm that proceeds as
follows. All pairs are sorted in descending order by score in
$\mathbf{\hat{y}}_{3}$ and then iterated over. For each pair $(i,j)$, if there
exists a pair $(i,k)$ or $(j,k)$ already in the solution set $A$, $(i,j)$ is
discarded. After the conclusion of the loop, a new symmetric matrix
$\mathbf{\hat{y}}_{4}$ is initialized such that only the pairs of indices in
$A$ are set to 1. Notably, in contrast to dynamic programming [2], pseudoknots
are able to be returned.

As in both SPOT-RNA and UFold [9], insignificant pairs are excluded by
applying a threshold $T$ computed via grid search on VL0 to produce the final
output $\mathbf{\hat{y}}_{5}:=\mathds{1}_{y\geq
T}\left(\mathbf{\hat{y}}_{4}\right)$ (Figure 2CF). The final result was a
cutoff of $T=0.455$.

###  4.4 Training procedure

The network was implemented in the PyTorch library [26] with the overall
software package written according to the Lightning specification. Separate
loops for training, validation, and testing were thus automatically delineated
as separate class methods, with the validation loop called at the conclusion
of each fifth iteration of the training loop.

As to the latter, training of Kirigami was conducted for 100 epochs over the
10,814 molecules in TR0 using the Adam gradient descent algorithm [14] with a
learning rate of $1\times 10^{-4}$ for binary cross-entropy loss minimization
and a dropout rate [39] of 0.15. Due to the hardware consisting of a compute
node with a single NVIDIA GeForce RTX 3090, batch size was limited to 1;
gradient accumulation [30] was therefore conducted across every 64 batches. Of
Constraints (I-IV) described above, only (I) was enforced during training; no
thresholding was performed or metrics evaluated aside from cross-entropy loss.

Validation epochs consisted of iterating over VL0, enforcing Constraints
(I-IV), thresholding the output matrix across a grid of 100 evenly spaced
values along the interval $\left[0,1\right]$ (Figure 2), and evaluating the
classification metrics at each threshold. For each sample in VL0, the Matthews
Correlation Coefficient (MCC) [23] along with other statistics was computed by
the formula below at successive values of $T$, then cached in a table
initialized at the beginning of each validation epoch. Following the call to
the validation loop method, the maximizing argument of $T$ with respect to the
mean MCC across VL0 was computed via a final round of grid search and written
to the network as a parameter. Network weights were checkpointed if the
maximum mean MCC exceeded that of previous epochs.

The training procedure can be reproduced via scripts in the corresponding
GitHub repository (see above). More straightforwardly, the test loop, which
replicates the results below (Table 1), can be called using pretrained weights
made publicly available in the same codebase. In order to facilitate the use
of Kirigami on molecules outside of TS0, a prediction loop for reading in
FASTA files and outputting corresponding dot-bracket files was also developed,
the conversion between the output matrix and the dot-bracket string computed
via an application of Nussinov’s recurrence relation for identifying
pseudoknots as described by [38].

###  4.5 Evaluation metrics

Performance for each model was measured in terms of Matthews correlation
coefficient (MCC) as follows. Representing the ground truth and predicted
secondary structures of a molecule of length $L$, the upper triangles of the
symmetric binary matrices
$\mathbf{y},\mathbf{\hat{y}}\in\left\\{0,1\right\\}^{L\times L}$ were
flattened into feature vectors
$\mathbf{v},\mathbf{\hat{v}}\in\left\\{0,1\right\\}^{\frac{(L-1)L}{2}}$,
respectively. Critically, the upper triangles were offset by one element to
exclude the diagonal from the computation. The MCC between $\mathbf{v}$ and
$\mathbf{\hat{v}}$ was then computed via the formula

$MCC=\frac{TN\times TP-FN\times FP}{\sqrt{(TP+FP)(TP+FN)(TN+FP)(TN+FN)}},$ (1)

where $TP$ denotes the number of true positives, $FP$ false positives, $TN$
true negatives, and $FN$ false negatives [23]. To evaluate the MCC on
pseudoknots specifically, the indices $i_{1},i_{2},\ldots,i_{n}$ of
$\mathbf{v}$ that corresponded to non-nested pairs were used to create new
vectors $\mathbf{v}_{p}:=\left(v_{i_{1}},v_{i_{2}},\ldots,v_{i_{n}}\right)$
and
$\mathbf{\hat{v}}_{p}:=\left(\hat{v}_{i_{1}},\hat{v}_{i_{2}},\ldots,\hat{v}_{i_{n}}\right)$,
between which the MCC was then computed according to the formula above. The
percentage increase $\Delta$ in MCC of Kirigami over other models was simply
computed via

$\Delta=100\%\times\frac{MCC_{\text{Kirigami}}-MCC_{\text{model}}}{MCC_{\text{model}}}.$
(2)

## Acknowledgements

We acknowledge the invaluable support, guidance, and insight of Anna Marie
Pyle at Yale University in formulating our project.

## References

  * [1] M. AlQuraishi.  Machine learning in protein structure prediction.  Curr. Opin. Chem. Biol., 65:1–8, 2021.
  * [2] R. Bellman.  Dynamic programming.  Science, 153:34–37, 1966.
  * [3] H.M. Berman, J. Westbrook, Z. Feng, G. Gilliland, T.N. Bhat, H. Weissig, I.N. Shindyalov, and P.E. Bourne.  The protein data bank.  Nucleic Acids Res., 28:235–242, 2000.
  * [4] Y. Byun and K. Han.  Pseudoviewer: web application and web service for visualizing rna pseudoknots and secondary structures.  Nucleic Acids Res., 34:W416–W422, 2006.
  * [5] X. Chen, Y. Li, R. Umarov, X. Gao, and L. Song.  Rna secondary structure prediction by learning unrolled algorithms.  arXiv preprint arXiv:2002.05810, 2020.
  * [6] A. Cornish-Bowden.  Nomenclature for incompletely specified bases in nucleic acid sequences: recommendations 1984.  Nucleic Acids Res., 13:3021, 1985.
  * [7] P. Danaee, M. Rouches, M. Wiley, D. Deng, L. Huang, and D. Hendrix.  bprna: large-scale automated annotation and analysis of rna secondary structure.  Nucleic Acids Res., 46:5381–5394, 2018.
  * [8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, and S. Gelly.  An image is worth 16x16 words: Transformers for image recognition at scale.  ArXiv Prepr. ArXiv201011929, 2020.
  * [9] L. Fu, Y. Cao, J. Wu, Q. Peng, Q. Nie, and X. Xie.  Ufold: fast and accurate rna secondary structure prediction with deep learning.  Nucleic Acids Res., 50:e14–e14, 2022.
  * [10] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pages 770–778, 2016.
  * [11] S. Hochreiter and J. Schmidhuber.  Long short-term memory.  Neural Comput., 9:1735–1780, 1997.
  * [12] K. Hornik, M. Stinchcombe, and H. White.  Multilayer feedforward networks are universal approximators.  Neural Netw., 2:359–366, 1989.
  * [13] J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, and A. Potapenko.  Highly accurate protein structure prediction with alphafold.  Nature., 596:583–589, 2021.
  * [14] D.P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  2014\.
  * [15] A. Krizhevsky, I. Sutskever, and G.E. Hinton.  Imagenet classification with deep convolutional neural networks.  Commun. ACM, 60:84–90, 2017.
  * [16] V. Kumar.  Algorithms for constraint-satisfaction problems: A survey.  AI Mag., 13:32–32, 1992.
  * [17] W. Li and A. Godzik.  Cd-hit: a fast program for clustering and comparing large sets of protein or nucleotide sequences.  Bioinformatics, 22:1658–1659, 2006.
  * [18] Y. Li, C. Zhang, E.W. Bell, W. Zheng, X. Zhou, D.-J. Yu, and Y. Zhang.  Deducing high-accuracy protein contact-maps from a triplet of coevolutionary matrices through deep residual convolutional networks.  PLoS Comput. Biol., 17:e1008865, 2021.
  * [19] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie.  A convnet for the 2020s.  In Proc. IEEECVF Conf. Comput. Vis. Pattern Recognit., pages 11976–11986, 2022.
  * [20] J. Long, E. Shelhamer, and T. Darrell.  Fully convolutional networks for semantic segmentation.  In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pages 3431–3440, 2015.
  * [21] R. Lorenz, S.H. Bernhart, C. Höner zu Siederdissen, H. Tafer, C. Flamm, P.F. Stadler, and I.L. Hofacker.  Viennarna package 2.0.  Algorithms Mol. Biol., 6:1–14, 2011.
  * [22] R.B. Lyngsø and C.N. Pedersen.  Rna pseudoknot prediction in energy-based models.  J. Comput. Biol., 7:409–427, 2000.
  * [23] B.W. Matthews.  Comparison of the predicted and observed secondary structure of t4 phage lysozyme.  Biochim. Biophys. Acta BBA-Protein Struct., 405:442–451, 1975.
  * [24] R. Nussinov and A.B. Jacobson.  Fast algorithm for predicting the secondary structure of single-stranded rna.  Proc. Natl. Acad. Sci., 77:6309–6313, 1980.
  * [25] S.J. Pan and Q. Yang.  A survey on transfer learning.  IEEE Trans. Knowl. Data Eng., 22:1345–1359, 2009.
  * [26] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, and L. Antiga.  Pytorch: An imperative style, high-performance deep learning library.  In Advances in Neural Information Processing Systems, volume 32, 2019.
  * [27] J.S. Reuter and D.H. Mathews.  Rnastructure: software for rna secondary structure prediction and analysis.  BMC Bioinformatics, 11:1–9, 2010.
  * [28] E. Rivas, J. Clements, and S.R. Eddy.  A statistical test for conserved rna structure shows lack of evidence for structure in lncrnas.  Nat. Methods, 14:45–48, 2017.
  * [29] O. Ronneberger, P. Fischer, and T. Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Int. Conf. Med. Image Comput. Comput.-Assist. Interv., pages 234–241. Springer, 2015.
  * [30] S. Ruder.  An overview of gradient descent optimization algorithms.  In arXiv preprint arXiv:1609.04747, 2016.
  * [31] K. Sato, M. Akiyama, and Y. Sakakibara.  Rna secondary structure prediction using deep learning with thermodynamic integration.  Nat. Commun., 12:1–9, 2021.
  * [32] K. Sato, Y. Kato, M. Hamada, T. Akutsu, and K. Asai.  Ipknot: fast and accurate prediction of rna secondary structures with pseudoknots using integer programming.  Bioinformatics, 27:i85–i93, 2011.
  * [33] M. Schuster and K.K. Paliwal.  Bidirectional recurrent neural networks.  IEEE Trans. Signal Process., 45:2673–2681, 1997.
  * [34] S.E. Seemann, J. Gorodkin, and R. Backofen.  Unifying evolutionary and thermodynamic information for rna folding of multiple alignments.  Nucleic Acids Res., 36:6355–6362, 2008.
  * [35] A. Shrestha and A. Mahmood.  Review of deep learning algorithms and architectures.  IEEE Access, 7:53040–53065, 2019.
  * [36] J. Singh, J. Hanson, K. Paliwal, and Y. Zhou.  Rna secondary structure prediction using an ensemble of two-dimensional deep neural networks and transfer learning.  Nat. Commun., 10:1–13, 2019.
  * [37] J. Singh, K. Paliwal, T. Litfin, J. Singh, and Y. Zhou.  Predicting rna distance-based contact maps by integrated deep learning on physics-inferred secondary structure and evolutionary-derived mutational coupling.  Bioinformatics., 38:3900–3910, 2022.
  * [38] S. Smit, K. Rother, J. Heringa, and R. Knight.  From knotted to nested rna structures: a variety of computational methods for pseudoknot removal.  RNA., 14:410–416, 2008.
  * [39] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  J. Mach. Learn. Res., 15:1929–1958, 2014.
  * [40] S. Sun, W. Wang, Z. Peng, and J. Yang.  Rna inter-nucleotide 3d closeness prediction by deep residual neural networks.  Bioinformatics., 37:1093–1098, 2021.
  * [41] Z. Sükösd, B. Knudsen, J. Kjems, and C.N. Pedersen.  Ppfold 3.0: fast rna secondary structure prediction using phylogeny and auxiliary data.  Bioinformatics, 28:2691–2692, 2012.
  * [42] D.H. Turner and D.H. Mathews.  Nndb: the nearest neighbor parameter database for predicting stability of nucleic acid secondary structure.  Nucleic Acids Res., 38:D280–D282, 2010.
  * [43] D. Ulyanov, A. Vedaldi, and V. Lempitsky.  Instance normalization: The missing ingredient for fast stylization.  2016\.
  * [44] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, and I. Polosukhin.  Attention is all you need.  In Advances in Neural Information Processing Systems, volume 30, 2017.
  * [45] S. Wang, S. Sun, Z. Li, R. Zhang, and J. Xu.  Accurate de novo prediction of protein contact map by ultra-deep learning model.  PLoS Comput. Biol., 13:e1005324, 2017.
  * [46] C. Weinreb, A.J. Riesselman, J.B. Ingraham, T. Gross, C. Sander, and D.S. Marks.  3d rna and functional interactions from evolutionary couplings.  Cell, 165:963–975, 2016.
  * [47] S. Zakov, Y. Goldberg, M. Elhadad, and M. Ziv-Ukelson.  Rich parameterization improves rna structure prediction.  J. Comput. Biol., 18:1525–1542, 2011.
  * [48] D.-X. Zhou.  Universality of deep convolutional neural networks.  Appl. Comput. Harmon. Anal., 48:787–794, 2020.
  * [49] M. Zuker.  On finding all suboptimal foldings of an rna molecule.  Science, 244:48–52, 1989.


# Diversifying Knowledge Enhancement of Biomedical Language Models using
Adapter Modules and Knowledge Graphs

Juraj Vladika, Alexander Fichtl, Florian Matthes
Department of Computer Science, Technical University of Munich,
Boltzmannstraße 3, 85748 Garching bei München, Germany
{juraj.vladika, alexander.fichtl<EMAIL_ADDRESS>

###### Abstract

Recent advances in natural language processing (NLP) owe their success to pre-
training language models on large amounts of unstructured data. Still, there
is an increasing effort to combine the unstructured nature of LMs with
structured knowledge and reasoning. Particularly in the rapidly evolving field
of biomedical NLP, knowledge-enhanced language models (KELMs) have emerged as
promising tools to bridge the gap between large language models and domain-
specific knowledge, considering the available biomedical knowledge graphs
(KGs) curated by experts over the decades. In this paper, we develop an
approach that uses lightweight adapter modules to inject structured biomedical
knowledge into pre-trained language models (PLMs). We use two large KGs, the
biomedical knowledge system UMLS and the novel biochemical ontology OntoChem,
with two prominent biomedical PLMs, PubMedBERT and BioLinkBERT. The approach
includes partitioning knowledge graphs into smaller subgraphs, fine-tuning
adapter modules for each subgraph, and combining the knowledge in a fusion
layer. We test the performance on three downstream tasks: document
classification, question answering, and natural language inference. We show
that our methodology leads to performance improvements in several instances
while keeping requirements in computing power low. Finally, we provide a
detailed interpretation of the results and report valuable insights for future
work.

##  1 INTRODUCTION

The field of natural language processing (NLP) has been marked by impressive
advancements in recent years. The appearance of new model architectures,
including the emergence of generative transformers and pre-trained language
models (PLMs), has brought along with it widespread usage and attention.
Still, most of these models were trained on large amounts of web content, and
while they excel at tasks in a general-purpose setting, there is still a
performance gap when it comes to domain-specific challenges.

One of these challenging domains is bio-medicine, which centers around the
study of the human body, diseases, drugs, and treatments. Biomedical text is
often characterized as highly complex because of its advanced terminology,
which frequently includes names of chemical compounds, long-spanning
relations, and other jargon not commonly used in everyday language. For NLP
models trained on general corpora to work well in the biomedical domain,
researchers have turned to transfer learning methods and domain adaption. The
most common approach to domain adaptation is to continue the initial general
pre-training of language models with data from domain-specific medical
corpora. Examples of models adapted in this way are BioBERT Lee et al., (2019)
and SciBERT Beltagy et al., (2019), which drew the additional training data
from biomedical and computer science research abstracts. Dropping the mixed-
domain approach from previous frameworks, models like PubMedBERT Gu et al.,
(2020) and BioLinkBERT Yasunaga et al., (2022) were instead trained solely on
PubMed research articles, with BioLinkBERT even leveraging links (citations)
to other research articles.

While domain fine-tuning of whole PLMs has proven to increase the performance
on downstream biomedical NLP tasks, additional pre-training can often be
resource-intensive and infeasible for smaller research groups and situations
where computing power is limited. A promising research direction has emerged
in the form of knowledge-enhanced language models (KELMs) Hu et al., (2023).
It refers to any set of methods that try to incorporate external knowledge
into language models, usually by injecting it into the model’s input,
architecture, or output. In a sea of knowledge-enhancement methods, an
especially interesting one is the utilization of adapters.

Broadly speaking, adapters are small bottleneck feed-forward layers inserted
within each layer of a transformer-based language model Houlsby et al.,
(2019); Pfeiffer et al., 2020b . The small amount of additional parameters
allows for the injection of new data or knowledge without requiring the whole
model to be fine-tuned. Adapters plugged on top of large language models will
often only have around 1% of the number of training parameters compared to the
transformer. The transformer model’s learned parameters (weights) are frozen
and left unchanged, and only the adapter is fine-tuned. Other than being
lightweight on resources, this approach also helps avoid the problem of
catastrophic forgetting, where language models forget their existing knowledge
from the pre-training corpora when they are fine-tuned on a new, smaller
corpus Colon-Hernandez et al., (2021).

This paper specifically focuses on using adapters to inject structured
biomedical knowledge from large knowledge graphs into PLMs. We provide an
overview of existing adapter approaches for the biomedical domain, as well as
existing biomedical language models. We perform extensive experiments to test
the performance of knowledge-enhanced, adapter-based biomedical language
models on a number of representative biomedical classification tasks (document
classification, question answering, natural language inference). We show that
the model performance is improved in several instances on downstream tasks and
provide a deeper look into the resulting change in model predictions. Finally,
our experiments demonstrate that the OntoChem ontology Irmer et al., (2013),
which has not been used for knowledge enhancement yet, is a viable alternative
to other prominent knowledge sources.

##  2 RELATED WORK

###  2.1 Knowledge-Enhanced PLMs

PLMs are trained on enormous corpora of training data, ranging from 3.3
billion tokens in the case of the original BERT Devlin et al., (2019), all the
way to 3.5 trillion tokens in the case of the recent Falcon-180B model
Almazrouei et al., (2023). The power of the model architecture, combined with
transfer learning, has led to these models showing impressive capabilities on
most NLP tasks. While the textual data used for the model training is usually
completely unstructured in nature, research has shown that models like BERT do
encode, to some extent, syntactic structures, hierarchical concepts, and
certain semantic conceptual relations Rogers et al., (2021). Still, other
studies have shown weakness in modeling tasks dealing with structured
knowledge, such as hyponymy relations Ravichander et al., (2020) or preserving
the association between text and meaning Di Sciullo, (2018).

In most cases, the knowledge we find and gather, especially scientific
knowledge, can be represented in a structured manner. This is the underlying
idea of knowledge graphs (KGs), a data structure that models concepts
(entities) and relations between them in a graph-like format Ji et al.,
(2021). KGs have been used in the field of NLP to enhance the performance of
NLP models in many downstream NLP tasks Schneider et al., (2022). There are
multiple ways to combine KGs with PLMs. The knowledge triples from KGs can be
embedded as vector representations such as TransE Wang et al., (2014) or
TuckER Balazevic et al., (2019) and then combined with the vectors encoding
text. Alternatively, the triples from KGs can be converted to sentences, and,
in turn, these textual representations can then be used to fine-tune PLMs in
the same way as with any other text. This approach was followed by COMET
Bosselut et al., (2019), which utilized the knowledge graph ConceptNet Speer
et al., (2017) to enhance the performance on commonsense reasoning tasks.
Besides knowledge graphs, lexicons are sometimes used for knowledge
enhancement Hoang et al., (2022).

While there are numerous ways to inject structured knowledge into PLMs such as
adding it to the input and output of models Wei et al., (2021), an especially
promising approach is adding adapters to the architecture of the model Colon-
Hernandez et al., (2021). Adapters are small layers that are inserted within a
language model and are subsequently fine-tuned to a specific task. The major
benefit of adapters is that they add a minimal amount of additional
parameters, thus significantly reducing the needed training time. Combined
with freezing original model weights, adapters can avoid catastrophic
forgetting, where the PLM’s performance deteriorates when all of its weights
are fine-tuned with a new knowledge source. Adapters have been used for
numerous purposes such as learning hierarchical representation Chronopoulou et
al., (2022), transferring models trained on English to low-resource languages
Wang et al., (2021), and in the domain of efficient transformers as low-rank
adapters (LoRA) Hu et al., (2022). General knowledge-enhanced PLMs utilizing
adapters include, for example, KnowBERT Peters et al., (2019) and K-Adapter
Wang et al., (2020). A practical tool emerged that combines well-known adapter
architectures in one place, called AdapterHub Pfeiffer et al., 2020b .

###  2.2 Biomedical Knowledge-Enhanced PLMs

A major focus of knowledge enhancement in PLMs is in domain adaption to expert
domains such as the biomedical domain. So far, most of the advancements have
focused on utilizing the knowledge graph UMLS Bodenreider, (2004) for this
purpose. Examples include BERT-MK He et al., (2020) and KeBioLM Yuan et al.,
(2021), which both fine-tune the whole weights of the base language model by
using masked language modeling of triples from UMLS. Biomedical PLMs can then
be used for various NLP tasks, such as biomedical text summarization Abacha et
al., (2021), named entity recognition Sung et al., (2022), medical fact-
checking Vladika and Matthes, (2023), information retrieval Luo et al.,
(2022), or health question answering Vladika et al., (2023).

There are also existing approaches using adapters for biomedical knowledge
enhancement. Representative works are DAKI Lu et al., (2021), which fine-tunes
the adapters with entity prediction task, and KEBLM Lai et al., (2023), which
fine-tunes the adapters on three different knowledge types from UMLS and
PubChem Kim et al., (2019), namely entity descriptions, entity-entity
relations, and entity synonyms. The most similar approach to ours and a direct
inspiration was the Mixture-of-Partitions (MoP) approach Meng et al., (2021),
where the adapters were fine-tuned on smaller subgraphs of UMLS.

Figure 1: Triplet from the OntoChem Fact
Finder111https://sciwalker.com/analytics/factfinder

In essence, our work builds on the present foundations of adapter-based
biomedical models and uses the yet unexplored knowledge graph OntoChem, which
is rich with chemical knowledge. For our experiments, we use the well-known
biomedical PLM PubMedBERT as well as the yet unexplored but powerful
BioLinkBERT base model. Following the suggestions of Meng et al., (2021), we
use only the triplets corresponding to the 20 most frequent relations of
OntoChem for the knowledge injection. An example of an OntoChem triplet can be
seen in Figure 1. Finally, we provide a deeper qualitative analysis of learned
structured knowledge on a specific dataset. Notably, our work achieves the
SOTA (averaged) performance on the question-answering BioASQ-7b dataset.

##  3 METHODOLOGY

In this section, we will explain the training methodology we used for the
experiments in this paper. It is depicted in Figure 2.

Figure 2: Methodology used to construct the final model and run the
experiments

###  3.1 Knowledge Graph Representation

A central element of our method is the knowledge graph (KG). This KG is a
structured representation of information denoted as a collection of ordered
triples Ji et al., (2021). We denote these triples as $(s,r,o)$, where s is a
subject, r is a relation, and o is an object. Both s and o are entities that
come from an entity set E, while relations come from a relation set R. Each
entity and relation in the KG is associated with its corresponding textual
surface form. This form can take the shape of a single word or a compound term
(e.g., for names of chemicals) or even a concise phrase, especially in case of
relations. This textual association is critical as it bridges the gap between
the structured KG and natural language, allowing for easier injection of KG
knowledge into the language models and associated fine-tuning.

The primary objective is to enhance the capabilities of a pre-trained language
model, denoted as LM, by integrating the knowledge contained within the KG. To
achieve this, we need a training objective that effectively incorporates the
KG knowledge into the model. Most encoder-only PLMs based on the original BERT
use masked language modeling (MLM) as one of its pre-training objectives. This
task consists of masking a certain word in a given sentence and having the
model predict which word would fit the best in the place of the masked token.
We follow the established approach of using an entity prediction objective,
where we mask one of the entities and have the model predict which token would
best fit. In this way, the model incorporates the structured knowledge of
$(s,r,o)$ triples into its internal weights.

###  3.2 Graph Partitioning

During the prediction of the masked token, the model produces a probability
distribution (with a softmax function) over all of the entities from KG’s
entity set E. Considering the massive size of the biomedical KGs we use in the
paper, computing the softmax function over all its entities carries a lot of
computation complexity. This issue can significantly slow down model training
and inference. To bridge this challenge, some approaches have been suggested
in the literature. We opt for the approach of Meng et al., (2021), which
involves partitioning the KG into smaller subgraphs, which are then trained on
independently, and later, their knowledge combined to unified knowledge
representations.

The process of dividing a KG yields smaller subgraphs that we denote as
$G_{1},G_{2},...,G_{k}$. We set $k$ in final experiments to be $20$, following
empirical observations and previous literature, which balances efficiency and
graph coverage well. Ideally, these $20$ smaller subgraphs should be almost
equal-sized components, meaning nodes are balanced across partitions.
Additionally, the capacity of edges between different components should be
minimized to maximize the retention of factual knowledge. This is a problem
known as balanced graph partitioning and is known to be an NP-complete problem
Andreev and Räcke, (2004). Several good approximations have been developed to
determine the exact solution despite its computational complexity. We opt for
the METIS algorithm Karypis and Kumar, (1997), which was used in other works
dealing with large-scale KG partitioning Zheng et al., (2020).

###  3.3 Adapter Module Learning

Once the KG is appropriately partitioned, the process of fine-tuning the LM
can be started. We deploy adapter modules for this purpose. As mentioned
previously, adapters are newly initialized feed-forward networks inserted
between the transformer model’s layers. Notably, the training of adapter
modules does not require fine-tuning the existing parameters of the pre-
trained model. Instead, it focuses solely on updating the parameters within
the adapters. This strategy ensures that the pre-trained model’s core
knowledge remains intact while enabling the model to specialize in the
biomedical domain by adapting to the specific knowledge contained in the KG.

There are multiple adapter module configurations, such as Houlsby et al.,
(2019) and Bapna and Firat, (2019). The adapter module configuration used in
the paper is based on the one by Pfeiffer et al., 2020a , the so-called
Pfeiffer architecture. In this configuration, only one adapter module is added
as a down-projection and up-projection, unlike the Houlsby architecture, where
there are two projections. While the Houlsby architecture has more learning
capacity, it comes with training and inference speed costs. Previous studies
showed no significant difference in performance between the model
architectures, making Pfeiffer architecture a very lightweight choice that
brings powerful learning capabilities.

As already mentioned, masked language modeling is used to fine-tune the
adapter modules. More precisely, it is a task of entity prediction since a
missing entity from the graph triple is being predicted. Given a subgraph
$G_{k}$ and its triples $(s,r,o)$, each triple has a textual representation.
The object entity o is removed from each triple, and the remaining two
elements of the triple are transformed into a textual representation like:
”[CLS] s [SEP] r [SEP]”. The adapter module is then trained to predict the
missing object entity using the representation of the [CLS] token. The
parameters of the adapter module are optimized by minimizing the cross-entropy
loss.

###  3.4 Knowledge Fusion

Finally, with a set of knowledge-encapsulated adapter modules at hand, we need
to fuse their knowledge together into a final representation. For this, we use
the so-called AdapterFusion mixture layers Pfeiffer et al., 2020a . These
layers serve the purpose of combining knowledge from various adapters to
enhance the model’s performance on downstream tasks. It is a relatively recent
approach designed to effectively learn how to combine information from a set
of task-specific adapters. It does so by employing a softmax attention
mechanism that assigns contextual mixture weights over the adapters. These
weights are then used to predict task labels in the final layer. The
composition of these layers and their interactions ultimately contribute to
the model’s ability to generalize and perform well on a range of tasks.

##  4 EXPERIMENTS

In this section, we describe our approach to leveraging data from OntoChem’s
SciWalker platform together with adapters to improve existing approaches to
biomedical KELMs. For reproducibility, we made the code for the experiment
runs available on GitHub.222https://github.com/alexander-
fichtl/diversifying˙KELMs.git

###  4.1 Datasets

All of our datasets, with the exception of MedNLI, originated from a
collection of common biomedical NLP tasks known as BLURB – Biomedical Language
Understanding and Reasoning
Benchmark.333https://microsoft.github.io/BLURB/index.html Inspired by a
similar suite of tasks for general-purpose natural language understanding
(NLU) known as GLUE Wang et al., (2018), BLURB covers a wide-range of tasks
related to biomedical NLU. This means no tasks include text generation and are
all essentially classification tasks, which makes them convenient to evaluate
with common classification metrics such as precision, recall, accuracy, and F1
score. The four datasets are described in continuation.

MedNLI Romanov and Shivade, (2018) is a dataset for natural language inference
(NLI). It consists of 14,049 unique sentence pairs, where one sentence is a
hypothesis, and the other one is a premise. The task is to infer whether the
premise entails the hypothesis, contradicts it, or is in a neutral relation
with respect to it. The premises were collected from MIMIC-III Johnson et al.,
(2016), the largest repository of publicly available clinical data (patient
notes).

BioASQ-7b Nentidis et al., (2020) is a biomedical question answering (QA)
benchmark dataset containing questions in English, along with golden standard
(reference) answers and related material. It has been designed to reflect real
information needs of biomedical experts. Other than only exact answers, the
BioASQ dataset also includes ideal answers (summaries). Researchers working on
paraphrasing and textual entailment can also measure the degree to which their
methods improve the performance of biomedical QA systems. The dataset is a
part of the ongoing shared challenge with the same name Tsatsaronis et al.,
(2015), while our dataset (7b) is from the 2019 challenge.

PubMedQA Jin et al., (2019) is a different QA dataset collected from PubMed
abstracts, the largest collection of biomedical research papers White, (2020).
The task of PubMedQA is to answer research questions with yes/no/maybe using
the corresponding abstracts. The dataset has 1,000 expert-annotated instances
of question-answer pairs. Each PubMedQA instance is composed of a question, a
context (abstract without the conclusion), a long answer (conclusion of the
abstract), and a yes/no/maybe label that summarizes the conclusion.

The Hallmarks of Cancer (HOC) Corpus Baker et al., (2015) consists of 1852
PubMed publication abstracts manually annotated by experts according to a
taxonomy. The taxonomy consists of 37 classes in a hierarchy. Zero or more
class labels are assigned to each sentence in the corpus. These hallmarks
refer to the alterations in cell behavior that characterize the cancer cell.
Proposed as a strategy to capture the complexity of cancer in a few basic
principles, it provides an organized framework comprising of ten hallmarks
Baker et al., (2017).

UMLS20 | #Triples |  Onto20Fused | #Triples |  Onto20Type | #Triples
---|---|---|---|---|---
has finding site | 367,237 |  relates to | 708,076 |  [protein] relates to [disease] | 295,841
has method | 275,398 |  induces | 502,512 |  [substance] induces [physiology] | 282,721
has associated morphology | 269729 |  modulates | 326,534 |  [food] contains [compound] | 269,211
has procedure site | 129,686 |  treats | 225,279 |  [substance] treats [disease] | 247,348
possibly equivalent to | 91,446 |  inhibits | 219,720 |  [biomarker] of [disease] | 205,604
has causative agent | 86,780 |  is analyzed by | 195,291 |  [substance] is analyzed by [method] | 130,275
interprets | 84,533 |  produces | 173,979 |  [plant] produces [compound] | 102,270
has direct procedure site | 83,749 |  increases activity of | 148,673 |  [protein] induces [physiology] | 85,411
has active ingredient | 63,792 |  contains | 133,241 |  [compound] increases activity of [protein] | 85,196
has pathological process | 54,639 |  increases | 110,803 |  [compound] decreases activity of [protein] | 72,311
has occurrence | 40,154 |  detects | 93,373 |  [substance] inhibits [physiology] | 68,728
has dose form | 30,940 |  decreases activity of | 85,425 |  [protein] is a [biomarker] | 65,558
has direct morphology | 29,667 |  prevents | 82,574 |  [anatomy] produces [protein] | 64,206
has intent | 25,907 |  increases expression of | 80,771 |  [substance] prevents [disease] | 60,260
has interpretation | 24,624 |  expresses | 62,142 |  [protein] induces [disease] | 59,577
has direct substance | 23,042 |  attenuates | 54,865 |  [substance] modulates [protein] | 54,533
has direct device | 17,726 |  decreases expression of | 51,152 |  [protein] is analyzed by [method] | 54,250
moved to | 17,507 |  binds to | 49,206 |  [method] treats [disease] | 35,768
has temporal context | 17,195 |  is a | 47,435 |  [method] detects [physiology] | 33,504
has subject relationship context | 16,926 |  affects expression of | 37,399 |  [protein] modulates [physiology] | 24,332
Total | 1,750,677 |  | 3,388,450 |  | 2,296,904
|  |  |  |  |
Table 1: Twenty most common relations in each of the three KGs used in the
experiments

###  4.2 Knowledge Sources

The Unified Medical Language System (UMLS) is a set of resources and tools
developed by the US National Library of Medicine (NLM) to facilitate the
integration and retrieval of biomedical and clinical information from various
sources Bodenreider, (2004). Created in 1986 and continuously developed over
the decades, it can be viewed as a comprehensive thesaurus and ontology of
biomedical concepts, making it easier to connect and use medical terminology
in research, clinical practice, and healthcare information systems. We use the
most recent SNOMED CT, US Edition vocabulary from September
2023.444https://www.nlm.nih.gov/healthit/snomedct/us˙edition.html

The second knowledge graph, more precisely ontology, that we use, is the
OntoChem Ontology Irmer et al., (2013). The ontology contains more than 900
complex relationships between two or more named entities. Entities include
chemical compounds, diseases, drug combinations, chemical reactions,
biological activities, adverse reactions, etc. Relationships can be downloaded
as RDF files. The data originates from
MedLine,555https://www.nlm.nih.gov/medline/index.html a bibliographic database
from the US National Library of Medicine’s (NLM), that contains more than 30
million journal articles focusing on medicine and life sciences. The KG
triples can be interactively queried and also downloaded from the SciWalker
platform with the Fact Finder
tool.666https://sciwalker.com/analytics/factfinder

###  4.3 KG Subsets

The versions of the KGs from the two knowledge sources we use in this work are
subsets of their respective full KGs – we use versions including only the top
20 most common relations. This was done to increase the efficiency of training
but also because initial experiments showed this smaller version does not hurt
the performance on downstream tasks. For UMLS, the list of most common
relations was taken from MoP and SNOMED, a systematically organized collection
of medical terms providing codes, terms, synonyms and definitions used in
clinical documentation and reporting. We label this KG as UMLS20.

The relations provided by OntoChem are unique to the type of entities that the
relation connects, so there can be several types of the same relation. For
example, the relation ”induces” can have a ”substance” as a subject and a
”disease” as an object, so the full relation becomes ”[substance] induces
[disease]”, while another one is with a ”physiology” as a subject and a
”disease” as an object, producing ”[physiology] induces [disease]”. To test
the performance between these two types, we produce both a KG with top 20
fused relations (independent of entity types) and with top 20 typed relations
(dependent on entity types). We call these two KGs Onto20Fused and Onto20Type.

The top 20 relations in each of the three KGs is shown in Table 1. This also
gives a good insight into what kind of structured knowledge is actually
contained in these manually curated biomedical knowledge bases. While there
are certain overlaps between top relations UMLS and OntoChem, a lot of them
refer to different types of interactions between entities. Therefore, a
promising research avenue that we did not explore in this work would be to
merge these two knowledge bases into a unified KG and use both to fine-tune
the adapters.

###  4.4 Setup

Task-specific fine-tuning is carried out for the four chosen benchmark
downstream tasks. We aligned our hyperparameters with the settings recommended
by the BLURB creators Gu et al., (2020): We deploy the Adam optimizer Zhang,
(2018) alongside the typical slanted triangular learning rate schedule, with a
warm-up for the initial 10 percent of steps and a cool-down for the subsequent
90 percent, and set the dropout probability at 0.1. Furthermore, we followed
Pfeiffer et al., 2020a  and Meng et al., (2021) by introducing mixture layers
and AdapterFusion to route valuable knowledge from the adapters to downstream
tasks automatically. Given the random initialization of the task-specific
model and dropout, outcomes can fluctuate based on different random seeds,
particularly for the small PubMedQA and BioASQ7b datasets. For a more accurate
representation, we present average results from ten iterations for BioASQ7b
and PubMedQA, five iterations HoC, and three for MedNLI, as done in related
biomedical NLP papers benchmarking these tasks.

The training was carried out on Google Colab, with V100 and T4 GPUs provided
on the platform. Specific hyperparameters and settings used in our experiments
are shown in Table 2. Run seeds are reported on GitHub.

Setting/Task | HoC | PubMedQA | BioASQ7b | MedNLI
---|---|---|---|---
repeat runs | 5 | 10 | 10 | 3
epochs | 20 | 30 | 25 | 20
patience | 3 | 4 | 5 | 3
batch size | 16 | 4 | 4 | 8
learning rate | 1e-5 | 0.5e-5 | 0.5e-5 | 0.5e-5
max. seq. len. | 128 | 512 | 512 | 256
Table 2: Settings and hyperparameters used for training each of the datasets
of the downstream tasks

##  5 RESULTS

This section describes the detailed experiment results. We provide both a
numerical analysis and a qualitative analysis of the results.

$\downarrow$ model—dataset $\rightarrow$ | HoC | PubMedQA | BioASQ7b | MedNLI
---|---|---|---|---
SciBERT-base | 80.52±0.60 | 57.38±4.22 | 75.93±4.20 | 81.19±0.54
\+ MoP | 81.79${}^{\dagger}_{\pm 0.66}\uparrow$ | 54.66±3.10 | 78.50${}^{\dagger}_{\pm 4.06}\uparrow$ | 81.20±0.37
\+ KEBLM | / | 59.0 | / | 82.14
|  |  |  |
BioBERT-base | 81.41±0.59 | 60.24±2.32 | 77.50±2.92 | 82.42±0.59
\+ MoP | 82.53${}^{\dagger}_{\pm 1.08}\uparrow$ | 61.04${}_{\pm 4.81}\uparrow$ | 80.79${}^{\dagger}_{\pm 4.40}\uparrow$ | 82.93${}_{\pm 0.55}\uparrow$
\+ KEBLM | / |  68.00 $\uparrow$ | / |  84.24 $\uparrow$
\+ DAKI | / | / | / |  83.41 $\uparrow$
PubMedBERT-base | 82.25±0.46 | 55.84±1.78 | 87.71±4.25 | 84.18±0.19
\+ UMLS20 | 83.26${}^{\dagger}_{\pm 0.32}\uparrow$ | 62.84${}^{\dagger}_{\pm 2.71}\uparrow$ | 90.64${}^{\dagger}_{\pm 2.43}\uparrow$ | 84.70${}_{\pm 0.19}\uparrow$
\+ Onto20Type | 82.17±0.62 | 55.40±5.57 | 86.36±3.07 | 83.94±0.63
\+ Onto20Fused | 82.39${}_{\pm 0.65}\uparrow$ | 56.12${}_{\pm 2.91}\uparrow$ | 84.36±4.73 | 83.97±0.59
|  |  |  |
BioLinkBERT-base | 82.21±0.87 | 56.76±3.00 | 91.29±3.18 | 84.1±0.03
+UMLS20 | 82.36${}_{\pm 0.57}\uparrow$ | 63.62${}^{\dagger}_{\pm 5.31}\uparrow$ | 91.50${}_{\pm 2.25}\uparrow$ | 83.78±0.09
+Onto20Type | 82.37${}_{\pm 0.42}\uparrow$ | 60.46${}_{\pm 5.81}\uparrow$ | 92.14${}_{\pm 2.30}\uparrow$ | 82.84±0.34
+Onto20Fused | 82.24${}_{\pm 1.25}\uparrow$ | 63.28${}^{\dagger}_{\pm 4.46}\uparrow$ | 90.57±3.14 | 83.69±0.55
|  |  |  |
Table 3: Final results of the model experiments: The metric for HoC is Micro
F1, while for the other three it is accuracy. The best results for every task
are in bold. ”$\uparrow$” denotes that improvements are observed when compared
to the base model. “$\dagger$” denotes a statistically significant better
result over the base model (T-test, p $<$ 0.05). The results in italic are
taken from previous works, while the rest of results comes from our
experiments.

###  5.1 Numerical Analysis

Table 3 shows the final results of the experiments. Each section first shows
the performance of the base biomedical model on its own, namely SciBERT
Beltagy et al., (2019), BioBERT Lee et al., (2019), PubMedBERT Gu et al.,
(2020), and BioLinkBERT Yasunaga et al., (2022). Afterwards, indentended rows
show the performance of knowledge-enhanced versions of the models. For SciBERT
and BioBERT, we report on competing approaches that use structured knowledge
integration: MoP Meng et al., (2021), DAKI Lu et al., (2021), and KEBLM Lai et
al., (2023). For PubMedBERT and BioLinkBERT, we report on the knowledge-
enhanced versions as described in our paper, augmented with structured
knowledge from knowledge graphs UMLS20, Onto20Fused, and Onto20Type. It should
be noted that the BioLinkBERT results differ from the ones in the original
publication because we report on averaged experiment results over multiple
runs, unlike the best single run in the original paper.

The results demonstrate that our knowledge enhancement approach improved
PubMedBERT in six instances and the BioLinkBERT model in eight instances,
either with the UMLS data or the OntoChem data. Notably, there is a difference
in the margin of improvement between the datasets. For HoC, the improvement is
either negligible or 1% in the best case. This shows that the task of trying
to classify document abstracts according to cancer properties is mostly
dependent on the document context itself and does not noticeably benefit from
external knowledge. Similar is the case for MedNLI, which either deteriorates
or improves less than 1%, showing that entailment recognition is mostly tied
to the reasoning capabilities of a language model and not the deeper medical
knowledge.

On the other hand, the two question-answering datasets experience noticeable
improvements. This makes sense considering the knowledge-intensive nature of
QA, where factual knowledge is at its core. Especially for PubMedQA, both base
PLMs get a 7% jump in accuracy with different KGs. An impressive result is the
BioLinkBert-base + Onto20Type model achieving state-of-the-art performance on
the BioASQ7b dataset (when looking at the averaged performance over 10 runs).
When looking at the difference between the two styles of OntoChem relations,
the fused version was superior for PubMedQA (by 3%), while the more detailed,
typed version performed better for BioASQ (by 1.5%). We attribute this to the
slight difference in the domain of these two datasets – BioASQ contains more
questions relating to chemical knowledge, where specific types could come into
play, while PubMedQA covers diverse medical diagnoses and treatments.

An interesting result that we have to investigate further is the relatively
worse performance of our approach with OntoChem KGs on PubMedBERT compared to
BioLinkBERT, even when factoring in the stronger base performance of
BioLinkBERT. When the base models don’t match, it is hard to distinguish
whether performance gains or losses come from the difference in base models or
the difference in the adapter-based approaches. Here, the base models of
BioLinkBERT generally perform better than those of PubMedBERT or SciBERT over
a variety of tasks. Therefore, whenever we use BioLinkBERT, we cannot say how
much of the performance gains come from the superiority of our approach versus
the superiority of the base model.

###  5.2 Qualitative Analysis

To investigate the performance of our knowledge-enhanced models on a deeper
level, we decided to look at the classification performance on an instance
level and singled out some interesting examples. Table 4 shows two instances
from the BioASQ dataset where our knowledge-enhanced model predicted the
answer correctly, unlike the base model. Instances in BioASQ consist of a
question and context, and the goal is to answer the question with a yes/no
verdict.

Question |  Context |  Predictions |
---|---|---|---
Can Diazepam be beneficial in the treatment of traumatic brain injury? |  The present experiment examined the effects of diazepam, a positive modulator at the GABA(A) receptor, on survival and cognitive performance in traumatically brain-injured animals. |  BioLinkBERT:
BLBERT+Onto20Type:

Gold Label: |  no
yes

yes
Does axitinib prolong the survival of pancreatic cancer patients? |  Axitinib/gemcitabine, while tolerated, did not provide survival benefit over gemcitabine alone in patients with advanced pancreatic cancer from Japan or other regions […]. |  BioLinkBERT:
BLBERT+Onto20Type:

Gold Label: |  yes
no

no
|  |  |
Table 4:  Examples of two instances from the BioASQ dataset (with a question,
context, and verdict) where the knowledge-enhanced model performed correctly,
unlike its vanilla counterpart.

The first row contains a question on the relationship between Diazepam and
traumatic brain injury. While the vanilla BioLinkBERT answered the question
incorrectly, our knowledge-enhanced BioLinkBERT + Onto20Type model gave the
correct answer. Diazepam (first marketed as Valium) is listed as an entity in
the OntoChem KG, where it has a direct relation to brain injuries – the full
triple is ”diazepam [substance] treats [disease] brain injury” (see also
figure 1. It is likely that, thanks to the injection of this knowledge, the
enhanced model was able to deduce the answer, while the base model was not.

The second row shows a question about axitinib and its relation to pancreatic
cancer. Here, the base version of BioLinkBERT incorrectly predicted that
axitinib does prolong the survival of pancreatic cancer patients, while our
BioLinkBERT + Onto20Type model gave the correct negative answer. This time,
there is no relation between axitinib and any form of cancer listed in the KG.
Therefore, our enhanced model might have been able to rely on its injected
knowledge and deduce that there are no such connections between the entities
in question.

##  6 CONCLUSION

This paper investigated the performance of biomedical pre-trained language
models when enhanced with structured domain-specific biomedical knowledge. For
this purpose, we utilized two biomedical PLMs (PubMedBERT and BioLinkBERT) and
external knowledge from two large KGs, UMLS and OntoChem. The KGs were
partitioned into smaller subgraphs and later fused into a common knowledge
representation. The knowledge was injected into the PLMs by using lightweight
but powerful adapter modules. We tested the performance on four downstream
biomedical NLP tasks and showed that the knowledge-enhanced models
consistently improved the results, indicating a clear benefit of infusing
external structured knowledge into unstructured PLMs. By updating the adapter
weights, which are only about 1–2% amount of PLM weights, the performance (in
best setting) increased on HoC and MedNLI for 1%, on BioASQ 3%, and on
PubMedQA 7%. This demonstrates the power of using adapter modules to fine-tune
PLMs for domain-specific purposes. Moreover, we have demonstrated that
OntoChem is a viable alternative to UMLS and other knowledge sources in the
field of biomedical knowledge enhancement.

#### Future Research

In future work, we would like to further investigate the potential of the
OntoChem ontology. Besides entities and relations, every data triple comes
with the source sentence from which it was extracted. Drawing inspiration from
works like K-Adapter Wang et al., (2020), this linguistic knowledge could be
extracted and used in additional adapters to enhance the models. Moreover, the
idea of merging together the data from Ontochem with subgraphs from MSI Ruiz
et al., (2021), UMLS Bodenreider, (2004), or PubChemKim et al., (2020)
presents a promising direction. Finally, future work could be more human-
centric and have medical professionals curate the KGs. This way, the resulting
KELMs would be tailored directly by those who use them.

#### Limitations

Our research did not come without certain challenges and limitations. A
portion of the data from OntoChem was not usable due to incomplete ID
mappings. As a result, only a fraction of the available knowledge was
integrated into the experimental segment of this work, which has likely led to
less thoroughly connected KGs. Additionally, medical professionals often
indicate concerns regarding ethical questions and the development and use of
LLMs in bio-medicine. While our methodology and models will likely not be used
in practice without further research and improvements, we did not specifically
address the medical community’s concerns in our work. We tried to improve the
overall model performance and factual accuracy to reduce hallucinations, but
there is no way to entirely eliminate the risk of wrong predictions and other
critical issues. At the time of writing, we are conducting a survey involving
clinicians to address their concerns in our future work.

## REFERENCES

  * Abacha et al., (2021) Abacha, A. B., M’rabet, Y., Zhang, Y., Shivade, C., Langlotz, C., and Demner-Fushman, D. (2021).  Overview of the mediqa 2021 shared task on summarization in the medical domain.  In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 74–85.
  * Almazrouei et al., (2023) Almazrouei, E., Alobeidli, H., Alshamsi, A., Cappelli, A., Cojocaru, R., Debbah, M., Goffinet, E., Heslow, D., Launay, J., Malartic, Q., Noune, B., Pannier, B., and Penedo, G. (2023).  Falcon-40B: an open large language model with state-of-the-art performance.
  * Andreev and Räcke, (2004) Andreev, K. and Räcke, H. (2004).  Balanced graph partitioning.  In Proceedings of the Sixteenth Annual ACM Symposium on Parallelism in Algorithms and Architectures, SPAA ’04, page 120–124, New York, NY, USA. Association for Computing Machinery.
  * Baker et al., (2017) Baker, S., Ali, I., Silins, I., Pyysalo, S., Guo, Y., Högberg, J., Stenius, U., and Korhonen, A. (2017).  Cancer Hallmarks Analytics Tool (CHAT): a text mining approach to organize and evaluate scientific literature on cancer.  Bioinformatics, 33(24):3973–3981.
  * Baker et al., (2015) Baker, S., Silins, I., Guo, Y., Ali, I., Högberg, J., Stenius, U., and Korhonen, A. (2015).  Automatic semantic classification of scientific literature according to the hallmarks of cancer.  Bioinformatics, 32(3):432–440.
  * Balazevic et al., (2019) Balazevic, I., Allen, C., and Hospedales, T. (2019).  TuckER: Tensor factorization for knowledge graph completion.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5185–5194, Hong Kong, China. Association for Computational Linguistics.
  * Bapna and Firat, (2019) Bapna, A. and Firat, O. (2019).  Simple, scalable adaptation for neural machine translation.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1538–1548, Hong Kong, China. Association for Computational Linguistics.
  * Beltagy et al., (2019) Beltagy, I., Lo, K., and Cohan, A. (2019).  Scibert: A pretrained language model for scientific text.  In Conference on Empirical Methods in Natural Language Processing.
  * Bodenreider, (2004) Bodenreider, O. (2004).  The unified medical language system (umls): integrating biomedical terminology.  Nucleic acids research, 32(suppl_1):D267–D270.
  * Bosselut et al., (2019) Bosselut, A., Rashkin, H., Sap, M., Malaviya, C., Celikyilmaz, A., and Choi, Y. (2019).  COMET: Commonsense transformers for automatic knowledge graph construction.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4762–4779, Florence, Italy. Association for Computational Linguistics.
  * Chronopoulou et al., (2022) Chronopoulou, A., Peters, M., and Dodge, J. (2022).  Efficient hierarchical domain adaptation for pretrained language models.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1336–1351, Seattle, United States. Association for Computational Linguistics.
  * Colon-Hernandez et al., (2021) Colon-Hernandez, P., Havasi, C., Alonso, J. B., Huggins, M., and Breazeal, C. (2021).  Combining pre-trained language models and structured knowledge.  ArXiv, abs/2101.12294.
  * Devlin et al., (2019) Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019).  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Di Sciullo, (2018) Di Sciullo, A. M. (2018).  Knowledge of language and knowledge science.
  * Gu et al., (2020) Gu, Y., Tinn, R., Cheng, H., Lucas, M. R., Usuyama, N., Liu, X., Naumann, T., Gao, J., and Poon, H. (2020).  Domain-specific language model pretraining for biomedical natural language processing.  ACM Transactions on Computing for Healthcare (HEALTH), 3:1 – 23.
  * He et al., (2020) He, B., Zhou, D., Xiao, J., Jiang, X., Liu, Q., Yuan, N. J., and Xu, T. (2020).  BERT-MK: Integrating graph contextualized knowledge into pre-trained language models.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 2281–2290, Online. Association for Computational Linguistics.
  * Hoang et al., (2022) Hoang, S. N., Nguyen, B., Nguyen, N. P., Luu, S. T., Phan, H. T., and Nguyen, H. D. (2022).  Enhanced task-based knowledge for lexicon-based approach in vietnamese hate speech detection.  In 2022 14th International Conference on Knowledge and Systems Engineering (KSE), pages 1–5.
  * Houlsby et al., (2019) Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q., Gesmundo, A., Attariyan, M., and Gelly, S. (2019).  Parameter-efficient transfer learning for nlp.  In International Conference on Machine Learning, pages 2790–2799. PMLR.
  * Hu et al., (2022) Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. (2022).  LoRA: Low-rank adaptation of large language models.  In International Conference on Learning Representations.
  * Hu et al., (2023) Hu, L., Liu, Z., Zhao, Z., Hou, L., Nie, L., and Li, J. (2023).  A survey of knowledge enhanced pre-trained language models.  IEEE Transactions on Knowledge and Data Engineering.
  * Irmer et al., (2013) Irmer, M., Bobach, C., Böhme, T., Püschel, A., and Weber, L. (2013).  Using a chemical ontology for detecting and classifying chemical terms mentioned in texts.  Proceedings of Bio-Ontologies 2013.
  * Ji et al., (2021) Ji, S., Pan, S., Cambria, E., Marttinen, P., and Philip, S. Y. (2021).  A survey on knowledge graphs: Representation, acquisition, and applications.  IEEE transactions on neural networks and learning systems, 33(2):494–514.
  * Jin et al., (2019) Jin, Q., Dhingra, B., Liu, Z., Cohen, W., and Lu, X. (2019).  PubMedQA: A dataset for biomedical research question answering.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2567–2577, Hong Kong, China. Association for Computational Linguistics.
  * Johnson et al., (2016) Johnson, A. E., Pollard, T. J., Shen, L., Lehman, L.-w. H., Feng, M., Ghassemi, M., Moody, B., Szolovits, P., Anthony Celi, L., and Mark, R. G. (2016).  Mimic-iii, a freely accessible critical care database.  Scientific data, 3(1):1–9.
  * Karypis and Kumar, (1997) Karypis, G. and Kumar, V. (1997).  Metis: A software package for partitioning unstructured graphs, partitioning meshes, and computing fill-reducing orderings of sparse matrices.
  * Kim et al., (2020) Kim, S., Chen, J., Cheng, T., Gindulyte, A., He, J., He, S., Li, Q., Shoemaker, B., Thiessen, P., Yu, B., Zaslavsky, L., Zhang, J., and Bolton, E. (2020).  Pubchem in 2021: New data content and improved web interfaces.  Nucleic Acids Research, 49.
  * Kim et al., (2019) Kim, S., Chen, J., Cheng, T., Gindulyte, A., He, J., He, S., Li, Q., Shoemaker, B. A., Thiessen, P. A., Yu, B., et al. (2019).  Pubchem 2019 update: improved access to chemical data.  Nucleic acids research, 47(D1):D1102–D1109.
  * Lai et al., (2023) Lai, T. M., Zhai, C., and Ji, H. (2023).  Keblm: Knowledge-enhanced biomedical language models.  Journal of Biomedical Informatics, 143:104392.
  * Lee et al., (2019) Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C. H., and Kang, J. (2019).  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics, 36(4):1234–1240.
  * Lu et al., (2021) Lu, Q., Dou, D., and Nguyen, T. H. (2021).  Parameter-efficient domain knowledge integration from multiple sources for biomedical pre-trained language models.  In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3855–3865. Association for Computational Linguistics.
  * Luo et al., (2022) Luo, M., Mitra, A., Gokhale, T., and Baral, C. (2022).  Improving biomedical information retrieval with neural retrievers.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11038–11046.
  * Meng et al., (2021) Meng, Z., Liu, F., Clark, T., Shareghi, E., and Collier, N. (2021).  Mixture-of-partitions: Infusing large biomedical knowledge graphs into BERT.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4672–4681, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Nentidis et al., (2020) Nentidis, A., Bougiatiotis, K., Krithara, A., and Paliouras, G. (2020).  Results of the seventh edition of the bioasq challenge.  In Cellier, P. and Driessens, K., editors, Machine Learning and Knowledge Discovery in Databases, pages 553–568, Cham. Springer International Publishing.
  * Peters et al., (2019) Peters, M. E., Neumann, M., RobertL.Logan, I., Schwartz, R., Joshi, V., Singh, S., and Smith, N. A. (2019).  Knowledge enhanced contextual word representations.  In Conference on Empirical Methods in Natural Language Processing.
  * (35) Pfeiffer, J., Kamath, A., Rücklé, A., Cho, K., and Gurevych, I. (2020a).  Adapterfusion: Non-destructive task composition for transfer learning.  ArXiv, abs/2005.00247.
  * (36) Pfeiffer, J., Rücklé, A., Poth, C., Kamath, A., Vulić, I., Ruder, S., Cho, K., and Gurevych, I. (2020b).  AdapterHub: A framework for adapting transformers.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 46–54, Online. Association for Computational Linguistics.
  * Ravichander et al., (2020) Ravichander, A., Hovy, E., Suleman, K., Trischler, A., and Cheung, J. C. K. (2020).  On the systematicity of probing contextualized word representations: The case of hypernymy in BERT.  In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 88–102, Barcelona, Spain (Online). Association for Computational Linguistics.
  * Rogers et al., (2021) Rogers, A., Kovaleva, O., and Rumshisky, A. (2021).  A primer in bertology: What we know about how bert works.  Transactions of the Association for Computational Linguistics, 8:842–866.
  * Romanov and Shivade, (2018) Romanov, A. and Shivade, C. (2018).  Lessons from natural language inference in the clinical domain.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1586–1596, Brussels, Belgium. Association for Computational Linguistics.
  * Ruiz et al., (2021) Ruiz, C., Zitnik, M., and Leskovec, J. (2021).  Identification of disease treatment mechanisms through the multiscale interactome.  Nature Communications, 12:1–15.
  * Schneider et al., (2022) Schneider, P., Schopf, T., Vladika, J., Galkin, M., Simperl, E., and Matthes, F. (2022).  A decade of knowledge graphs in natural language processing: A survey.  In He, Y., Ji, H., Li, S., Liu, Y., and Chang, C.-H., editors, Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 601–614, Online only. Association for Computational Linguistics.
  * Speer et al., (2017) Speer, R., Chin, J., and Havasi, C. (2017).  Conceptnet 5.5: An open multilingual graph of general knowledge.  In Proceedings of the AAAI conference on artificial intelligence, volume 31.
  * Sung et al., (2022) Sung, M., Jeong, M., Choi, Y., Kim, D., Lee, J., and Kang, J. (2022).  Bern2: an advanced neural biomedical named entity recognition and normalization tool.  Bioinformatics, 38(20):4837–4839.
  * Tsatsaronis et al., (2015) Tsatsaronis, G., Balikas, G., Malakasiotis, P., Partalas, I., Zschunke, M., Alvers, M. R., Weissenborn, D., Krithara, A., Petridis, S., Polychronopoulos, D., et al. (2015).  An overview of the bioasq large-scale biomedical semantic indexing and question answering competition.  BMC bioinformatics, 16(1):1–28.
  * Vladika and Matthes, (2023) Vladika, J. and Matthes, F. (2023).  Scientific fact-checking: A survey of resources and approaches.  In Rogers, A., Boyd-Graber, J., and Okazaki, N., editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 6215–6230, Toronto, Canada. Association for Computational Linguistics.
  * Vladika et al., (2023) Vladika, J., Schneider, P., and Matthes, F. (2023).  Healthfc: A dataset of health claims for evidence-based medical fact-checking.
  * Wang et al., (2018) Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. (2018).  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al., (2020) Wang, R., Tang, D., Duan, N., Wei, Z., Huang, X., Ji, J., Cao, G., Jiang, D., and Zhou, M. (2020).  K-adapter: Infusing knowledge into pre-trained models with adapters.  In Findings.
  * Wang et al., (2021) Wang, X., Tsvetkov, Y., Ruder, S., and Neubig, G. (2021).  Efficient test time adapter ensembling for low-resource language varieties.  In Conference on Empirical Methods in Natural Language Processing.
  * Wang et al., (2014) Wang, Z., Zhang, J., Feng, J., and Chen, Z. (2014).  Knowledge graph embedding by translating on hyperplanes.  In Proceedings of the AAAI conference on artificial intelligence, volume 28.
  * Wei et al., (2021) Wei, X., Wang, S., Zhang, D., Bhatia, P., and Arnold, A. O. (2021).  Knowledge enhanced pretrained language models: A compreshensive survey.  ArXiv, abs/2110.08455.
  * White, (2020) White, J. (2020).  Pubmed 2.0.  Medical reference services quarterly, 39(4):382–387.
  * Yasunaga et al., (2022) Yasunaga, M., Leskovec, J., and Liang, P. (2022).  Linkbert: Pretraining language models with document links.  In Annual Meeting of the Association for Computational Linguistics.
  * Yuan et al., (2021) Yuan, Z., Liu, Y., Tan, C., Huang, S., and Huang, F. (2021).  Improving biomedical pretrained language models with knowledge.  In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 180–190, Online. Association for Computational Linguistics.
  * Zhang, (2018) Zhang, Z. (2018).  Improved adam optimizer for deep neural networks.  In 2018 IEEE/ACM 26th international symposium on quality of service (IWQoS), pages 1–2. Ieee.
  * Zheng et al., (2020) Zheng, D., Ma, C., Wang, M., Zhou, J., Su, Q., Song, X., Gan, Q., Zhang, Z., and Karypis, G. (2020).  Distdgl: Distributed graph neural network training for billion-scale graphs.  In 2020 IEEE/ACM 10th Workshop on Irregular Applications: Architectures and Algorithms (IA3), pages 36–44, Los Alamitos, CA, USA. IEEE Computer Society.