(INN) or the name of the ATC level. These names given to ATC codes as textual
information are used.

###  6.3 Learning Knowledge Graph Embeddings

####  6.3.1 Knowledge Graph Definition

A heterogeneous KG is treated as a directed graph whose nodes and edges have
semantic types. The semantic types are assigned to different types of nodes
(drug, protein, pathway, etc.) and relations (target, carrier, etc.) to
represent detailed information about nodes and relations. A KG is defined as a
directed graph $\mathcal{G}=(E,R,F)$, where the nodes $E$ denotes the set of
typed entities, $R$ refers to the set of typed relations and $F$ represents
the set of facts (i.e., directed edges). The nodes are often called entities.
The facts or directed edges are often called triplets and are represented as a
$(h,r,t)$ tuple, when $h$ is the head entity, $t$ is the tail entity and $r$
is the relation from the head entity to the tail entity.

####  6.3.2 Scoring Functions

The methods that represent the KG by using embeddings of entities and
relations can catch the structure information of the KG and provide structure-
based embeddings. Entities and relations are directly represented as real-
valued vectors, matrices or complex-valued vectors. Scoring function
$f(h,r,t)$ is defined on each triple $(h,r,t)$ to access the validity of
triples. Triples observed in the KG tend to have higher scores than those that
have not been observed. The following four scoring functions are employed.

##### TransE

TransE [78] is a representative translational distance model that represents
entities and relations as vectors in the same semantic space of dimension
$\mathbb{R}^{d}$ where $d$ is the dimension of the target space with reduced
dimension. A fact in the source space is represented as a triplet $(h,r,t)$.
The relationship is interpreted as a translation vector so that the embedded
entities are connected by relation $r$ and have a short distance. The norm is
set to $2$, and the scoring function is computed as:

$f(h,r,t)=-|\mathbf{h}+\mathbf{r}-\mathbf{t}|_{2}.$ (6.1)

##### DistMult

DistMult [87] is a method that speeds up the RESCAL model [68] by considering
only symmetric relations and restricting $M_{r}$ from a general asymmetric
$r\times r$ matrix to a diagonal square matrix, thus reducing the number of
parameters per relation to $O(d)$. DistMult scoring function is computed as:

$f(h,r,t)=\mathbf{h}^{\mathsf{T}}\mathrm{diag}(\mathbf{r})\mathbf{t}=\sum^{d-1}_{i=0}[\mathbf{h}]_{i}[\mathbf{r}]_{i}[\mathbf{t}]_{i}.$
(6.2)

##### ComplEx

ComplEx [91] uses complex vector operations to consider both symmetric and
asymmetric relation. The scoring function for complex entity and relation
vectors $\mathbf{h}$, $\mathbf{r}$, and $mathbf{t}\in\mathbb{C}^{d}$ is
computed as:

$f(h,r,t)=\mathrm{Real}\big{(}\mathbf{h}^{\mathsf{T}}\mathrm{diag}(\mathbf{r})\mathbf{t}\big{)},$
(6.3)

where $\mathrm{Real}$ extracts the real part of the complex vectors.

##### SimplE

SimplE [92] considers two vectors $\mathbf{h},\mathbf{t}\in\mathbb{R}^{d}$ as
the head and tail embeddings for each entity and two vectors
$\mathbf{v}_{r},\mathbf{v}_{r^{-1}}\in\mathbb{R}^{d}$ for each relation $r$.
The similarity function of SimplE for a triple $(h,r,t)$ is defined as:

$f(h,r,t)=\frac{1}{2}\big{(}\langle\mathbf{h}_{h},\mathbf{v}_{r},\mathbf{t}_{t}\rangle+\langle\mathbf{h}_{t},\mathbf{v}_{r^{-1}},\mathbf{t}_{h}\rangle\big{)}.$
(6.4)

The above four score functions are chosen because these are widely used and
cover the standard ideas for scoring relational triples: distance-based,
bilinear-based and complex number-based.

####  6.3.3 Negative Sampling and Loss Functions

Generally, to train a KG embedding, the models apply a variety of negative
sampling by corrupting triplets $(h,r,t)$. They corrupt either $h$, or $t$ by
sampling from the set of head or tail entities for heads and tails,
respectively. The corrupted triples can be either of $(h^{\prime},r,t)$ or
$(h,r,t^{\prime})$, where $h^{\prime}$ and $t^{\prime}$ are the negative
examples. The author acknowledges that due to the incompleteness of the
current KG, the unregistered and potentially positive relational triples can
be negative examples: this problem is common in most studies that tackle with
the link prediction task. To avoid easy negative examples and utilize the
entity type information, the node types of negative examples is restricted
depending on $r$. The logistic loss and the margin-based pairwise ranking loss
are commonly used for training. The logistic loss returns $-1$ for negative
examples and $+1$ for the positive examples. $\mathbb{D}^{+}$ and
$\mathbb{D}^{-}$ are negative and positive data, $y=\pm 1$ is the label for
positive and negative triples, and $f(\cdot)$ is the scoring function. Model
parameters are trained by minimizing the negative log-likelihood of the
logistic model with $L2$ regularization on the parameters $\Theta$ of the
model;

$L_{KG}\sum_{(h,r,t)\in\mathbb{D}^{+}\cup\mathbb{D}^{-}}\log\big{(}1+\exp(y\times
f(h,r,t))\big{)}+\lambda||\Theta||^{2}_{2}.$ (6.5)

The margin-based pairwise ranking loss minimizes the rank for positive
triples. Ranking loss is given by:

$L_{KG}=\sum_{(h,r,t)\in\mathbb{D}^{+}}\sum_{(h^{\prime},r^{\prime},t^{\prime})\in\mathbb{D}^{-}}\max(0,\gamma-f(h,r,t)+f(h^{\prime},r^{\prime},t^{\prime}))+\lambda||\Theta||^{2}_{2}.$
(6.6)

###  6.4 Methods

This section verifies the usefulness of using text information in KG embedding
training by three methods explained below. Figure 6.2 shows the overview of
the three methods that utilize text information for KG embedding
representation. BERT [4], which is an extremely high-performance contextual
language representation model, is employed in encoding text. BERT is pre-
trained with the masked language model objective and next sentence prediction
task objective on large unlabeled corpora, and fine-tuned BERT towards the
target task achieved the state-of-the-art performance.

Figure 6.2: Overview of methods: (A) Initializing node embeddings
(Initialization), (B) Aligning entity embeddings and textual embeddings
(Alignment), and (C) Augmenting KG embeddings (Augmentation)

####  6.4.1 Initializing Node Embeddings

Usually, the initial value of embedding for each node in the KG is given
randomly in the existing methods. As shown in Figure 6.2 (A), first, the type
of text to use, e.g., drug nodes have text types such as Name, Description and
Synonyms, is selected. Then, the selected text is taken as the input of the
text encoder model BERT and the <CLS> embeddings of the BERT as the initial
value of the node embeddings. For the methods that use two embeddings for an
entity, i.e., ComplEx (real and imaginary embeddings) and SimplE (head and
tail embeddings), both vectors with the <CLS> embeddings are initialized. When
multiple text items are registered (e.g., the drug Acetaminophen has multiple
synonyms, “Acenol”, “APAP”, “Paracetamol”), these terms are connected with a
comma and taken as an input for BERT. This method is called the Initialization
method. The motivation of the Initialization method is to help represent node
embeddings by using the BERT embeddings that pre-trained on a large amount of
biomedical literature. Correct relational triples are aimed to predict from
textual information by BERT even if the information of structures in the graph
is insufficient.

####  6.4.2 Aligning Entity Embeddings and Textual Embeddings

The aligning method aims to gradually project KG embeddings into textual
embeddings space by adding the regularization term to the loss function.

$L_{a}=\lambda_{a}||V_{KG}-V_{text}||_{2},$ (6.7) $L=L_{KG}+L_{a},$ (6.8)

where $\lambda_{a}$ is a regularization coefficient of alignment, $V_{KG}$ and
$V_{text}$ are vector lookup table matrices of KG and textual embeddings
respectively. Similar to the initialization method, the textual embeddings are
obtained from BERT and when there are two embeddings for an entity, both
vectors are regularized. This method is called the Alignment method. The
motivation of the Alignment method is that as the updating the node
representation progresses, the two spaces of the text embeddings and the graph
structure embeddings are projected into the same space, and finally more
suitable node representations are obtained.

####  6.4.3 Augmenting KG Embeddings

In this method, as shown in Figure 6.2 (C), the KG structure is augmented by
adding relation triples based on the text information of the node. The node’s
own embedding is initialized with textual embeddings of Name. The embedding
value of linked nodes is initialized with the BERT output. Moreover, since ATC
classification codes have a hierarchical structure as shown in Figure 6.2 (C),
after extending the link from the drug node to create new categorical nodes,
further linking is made between the categorical nodes. A graph that can
consider both text information and the hierarchical information is
constructed. This method is called the Augmentation method. The motivation of
the Augmentation method is to consider multiple text information of one entity
at once.

###  6.5 Experimental Settings

####  6.5.1 Constructing Heterogeneous KG with Textual Information

The overview of constructing a heterogeneous KG with textual information is
shown in Figure 6.1. Four publicly available databases, DrugBank, UniProt,
MeSH term descriptions and SMPDB are downloaded, and first DrugBank is
processed and relations between the drug and other heterogeneous items are
extracted. Here, the text information on each drug is also extracted and
associated with the entity ID in the KG. Next, for entities other than drugs,
the link ID of DrugBank is used to refer to other databases and associated the
text information with the entity ID in the KG. As a result, five types of
entities (i.e., drug, protein, pathway, category, and ATC) are included in the
constructed KG. Between entities, there are relation links: category, ATC,
pathway, interact, target, enzyme, carrier, and transporter. The total number
of relational triples is about 2.7M, and as shown in Table 6.2, the number of
drug-interact-drug triples is large and accounts for the majority of them.
Note that only the relation drug-interact-drug is symmetric, and the other
relations are asymmetric, that is, when there is a DrugA-interact-DrugB
relation triple in the KG, there is also a DrugB-interact-DrugA triple.

####  6.5.2 Encoding Text Information

PubMedBERT [93] is employed to encode textual information into fixed-length
real-valued embeddings. PubMedBERT is a model that uses 21B words of the
PubMed corpus for pre-training, and shows high performance in several NLP
tasks in the biomedical domain. In this paper, texts such as names and
descriptions are used as inputs for pre-trained PubMedBERT, and the output
<CLS> token embedding is used as a textual representation. The maximum length
of the input subword is set to 512.

####  6.5.3 KG Embedding Training Settings

Four KG embedding scoring functions are employed as explained in Section
6.3.2. For each of the scoring functions, three methods are applied to train
embeddings using textual information; the initialization, aligning and
augmenting methods.

| Name (%) | Description (%) | Synonyms (%)
---|---|---|---
drug | 100 | 53.72 | 48.50
protein | 100 | 96.17 | 100
category | 100 | 94.01 | 91.42
ATC | 100 | - | -
pathway | 100 | 100 | -
Table 6.3: The percentage of nodes that have each type of text. Nodes in all
databases have Name text information. While many proteins and categories have
Description information and Synonyms information, the percentage of drugs that
have this information is low.

The ratio of nodes that have each textual information is shown in Table 6.3.
The node has the text information of Name in any database. In UniProt, most
proteins have Description and Synonyms texts information, and many categorical
terms in MeSH also have Description and Synonyms. On the other hand, some
drugs in DrugBank do not have some text information. In the Initialization
method and Alignment method, one text type is selected and the embeddings of
textual information are used444the combination of different text information
in these methods is left for future work. When the node does not have text
information, the text of Name is used instead.

Drugs and proteins have textual information that other nodes do not have, and
their coverage is as follows: 32.61% of drugs have Indication information,
24.60% of drugs have Pharmacodynamics information, 18.40% of drugs have
Metabolism information, 30.52% of drugs have Mechanism-of-action information
and 96.05% of proteins have Gene-name. These text items are linked to the KG
nodes in the Augmentation method, so the Augmentation method can utilize all
text information.

The random initialization method is prepared without textual information (No
Text) as the baseline. In this setting, embeddings of entities and relations
are initialized with the random values drawn from a uniform distribution
between $\pm(\gamma+\frac{\epsilon}{d})$, where $\gamma=12$, $\epsilon=2$ and
$d$ is a dimension of KG embeddings.

####  6.5.4 Task Setting

The quality of node and edge embeddings are evaluated by the link prediction
task. Link prediction is a task to search for an entity that probably
constructs a new fact with another given entity and a specific relation. For
KGs are always imperfect, link prediction aims to discover and add missing
knowledge into it. With the existing relations and entity, candidate entities
are selected to form a new fact. The head or tail of the triples in the
validation or test data set are replaced with other entities that have the
same entity types and calculate the scores of all created negative triples in
the KG. The calculated positive triple score and the scores of all negative
triples are sorted and the rank of the positive triple score is evaluated.
Mean reciprocal rank (MRR) is used as an evaluation metric. When negative
example triples are created, if there are correct triples that exist in the
KG, such triples from the ranking are excluded. This evaluation setting has
been adopted in many existing studies as a filtered setting [78, 91, 92]. In
addition, similar to the negative sampling setting during training, given the
relational edge label, the node types of head or tail are trivial, so triples
with inappropriate combinations of edge and node types are also excluded.

The extracted approximately 2.7M relational triples are divided into 90:5:5 as
the train, valid and test data sets. In the augmentation method, relational
triples created from textual nodes are added to the train data set.

####  6.5.5 Hyper-parameter Settings

Hyper-parameters are tuned by evaluating the MRR score on the validation set
for each model. Hyper-parameters are chosen with following values:
regularization coefficient
$\lambda\in\\{10^{-3},10^{-6},10^{-9},10^{-12},0\\}$, alignment regularization
coefficient $\lambda_{a}\in\\{10^{-3},10^{-6},10^{-9},10^{-12}\\}$, initial
learning rate $\alpha_{0}\in\\{0.5,0.25,0.1,0.05,0.025,0.01\\}$, For the loss
function, the pair-wise hinge loss function is adopted for TransE and DistMult
and the logistic loss function for ComplEx and SimplE according to the setting
of the original papers. The KG embedding dimension is set to 768 in order to
match the dimension of the output of BERT embedding. For all models, the batch
size is set to 4096 and the number of epochs to 100.

####  6.5.6 Implementation Details

All the models are implemented by using the PyTorch library [94], the DGL-KE
library [95] for KG embeddings, and the transformers library [96] for BERT.
The original DGL-KE implementation is modified in the following point. While
DGL-KE samples negative examples from all combinations of entity pairs, the
proposed model excludes impossible negative instances by restricting the types
of entities by the relations (e.g., a drug-interact-category triple is not
created for negative examples) as explained in Section 6.3.3.

###  6.6 Results and Discussions

TransE
---
| No Text | Initialization | Alignment | Augmentation
|  | Name | Desc. | Syn. | Name | Desc. | Syn. |
category | 0.1978 | 0.2117 | 0.2120 | 0.2231 | 0.2136 | 0.2059 | 0.1913 | 0.2239
ATC | 0.2929 | 0.2695 | 0.2571 | 0.2495 | 0.3000 | 0.2973 | 0.2872 | 0.2571
pathway | 0.6741 | 0.5674 | 0.5792 | 0.5793 | 0.6694 | 0.6711 | 0.6713 | 0.5473
interact | 0.3109 | 0.2867 | 0.2845 | 0.2843 | 0.3103 | 0.3106 | 0.3108 | 0.2644
target | 0.0802 | 0.0748 | 0.0811 | 0.0808 | 0.0808 | 0.0780 | 0.0821 | 0.0889
enzyme | 0.3262 | 0.3067 | 0.3314 | 0.3090 | 0.3474 | 0.3564 | 0.3590 | 0.2926
carrier | 0.4155 | 0.3078 | 0.2843 | 0.3679 | 0.4037 | 0.4044 | 0.4010 | 0.3459
transporter | 0.3576 | 0.3194 | 0.3104 | 0.3182 | 0.3444 | 0.3308 | 0.3391 | 0.2866
Avg. (Macro) | 0.3319 | 0.2930 | 0.2950 | 0.3015 | 0.3337 | 0.3318 | 0.3302 | 0.2883
DistMult
| No Text | Initialization | Alignment | Augmentation
|  | Name | Desc. | Syn. | Name | Desc. | Syn. |
category | 0.2539 | 0.2876 | 0.2797 | 0.2753 | 0.2679 | 0.2661 | 0.2586 | 0.2649
ATC | 0.2428 | 0.2674 | 0.2899 | 0.2639 | 0.2612 | 0.2617 | 0.2698 | 0.2904
pathway | 0.6792 | 0.5424 | 0.5542 | 0.6002 | 0.6524 | 0.6711 | 0.6615 | 0.4997
interact | 0.7730 | 0.6338 | 0.5895 | 0.6302 | 0.7911 | 0.7868 | 0.7990 | 0.6113
target | 0.0738 | 0.0866 | 0.0864 | 0.0947 | 0.0778 | 0.0758 | 0.0734 | 0.1049
enzyme | 0.2501 | 0.2516 | 0.2358 | 0.2847 | 0.2247 | 0.2334 | 0.2140 | 0.2183
carrier | 0.2023 | 0.2254 | 0.1640 | 0.1622 | 0.1369 | 0.1311 | 0.1649 | 0.2134
transporter | 0.2293 | 0.2703 | 0.1840 | 0.2190 | 0.1969 | 0.1939 | 0.1708 | 0.2062
Avg. (Macro) | 0.3380 | 0.3206 | 0.2979 | 0.3162 | 0.3261 | 0.3274 | 0.3265 | 0.3011
ComplEx
| No Text | Initialization | Alignment | Augmentation
|  | Name | Desc. | Syn. | Name | Desc. | Syn. |
category | 0.0905 | 0.3455 | 0.3495 | 0.3386 | 0.3302 | 0.0577 | 0.0611 | 0.3420
ATC | 0.3326 | 0.3463 | 0.3623 | 0.3485 | 0.3271 | 0.3425 | 0.3407 | 0.3652
pathway | 0.6956 | 0.6856 | 0.7051 | 0.7157 | 0.7220 | 0.6963 | 0.7323 | 0.6820
interact | 0.8678 | 0.7632 | 0.7166 | 0.7802 | 0.8578 | 0.8189 | 0.8497 | 0.8230
target | 0.0496 | 0.1116 | 0.1093 | 0.1153 | 0.0859 | 0.0640 | 0.0740 | 0.1243
enzyme | 0.2103 | 0.2256 | 0.2512 | 0.2538 | 0.2245 | 0.1907 | 0.2097 | 0.2073
carrier | 0.1533 | 0.1557 | 0.1817 | 0.1423 | 0.0994 | 0.1750 | 0.1462 | 0.1934
transporter | 0.1942 | 0.3119 | 0.2667 | 0.2593 | 0.2151 | 0.2076 | 0.2362 | 0.2801
Avg. (Macro) | 0.3242 | 0.3681 | 0.3678 | 0.3692 | 0.3577 | 0.3190 | 0.3312 | 0.3771
SimplE
| No Text | Initialization | Alignment | Augmentation
|  | Name | Desc. | Syn. | Name | Desc. | Syn. |
category | 0.0461 | 0.3591 | 0.3536 | 0.3668 | 0.0520 | 0.3263 | 0.2619 | 0.3367
ATC | 0.3278 | 0.3820 | 0.3617 | 0.3732 | 0.3644 | 0.3410 | 0.3425 | 0.3475
pathway | 0.7513 | 0.7164 | 0.7299 | 0.7180 | 0.7336 | 0.7428 | 0.7448 | 0.7189
interact | 0.6215 | 0.7229 | 0.7253 | 0.7338 | 0.6488 | 0.6242 | 0.6602 | 0.7230
target | 0.0815 | 0.1128 | 0.1169 | 0.1171 | 0.0971 | 0.0918 | 0.0873 | 0.1163
enzyme | 0.1903 | 0.2442 | 0.2143 | 0.2555 | 0.2499 | 0.2031 | 0.1977 | 0.2304
carrier | 0.1358 | 0.2544 | 0.2441 | 0.2526 | 0.1881 | 0.1766 | 0.1266 | 0.1493
transporter | 0.2242 | 0.2718 | 0.2189 | 0.2543 | 0.2396 | 0.2042 | 0.2417 | 0.2173
Avg. (Macro) | 0.2973 | 0.3829 | 0.3705 | 0.3839 | 0.3216 | 0.3387 | 0.3328 | 0.3549
Table 6.4: Comparison of MRR performance for each method. The MRR for each
relational triple and calculated the macro-averaged MRR are summarized. The
highest score for each node row is shown in bold.

Table 6.4 shows the comparison of link prediction MRR for each relation edge
type, the macro-averaged MRR. While a micro-average MRR is calculated by
directly calculating the MRR for all instances in the KG without considering
the types, a macro-averaged MRR is calculated by first calculating the MRR for
each type and then taking the average of the MRR scores. Since the constructed
triples are highly imbalanced and the proportion of interact triples is large,
models with high prediction performance of relation interact can result in
high micro-averaged MRR. The macro-averaged MRR is reported to avoid the
effect of this imbalance. For each scoring function, the comparison of
performance between the models with and without text information is shown.

When the TransE algorithm is used, in the category types, the textual models
improved MRR but in other relation triple types, the MRR decreased and the
averaged MRR also decreased. Of the three methods that used text, the
Initialization by synonyms embeddings method showed the highest macro-averaged
MRR.

When the DistMult scoring function is used, the MRR decreased in interact and
pathway, but on the categorical relation category and ATC, the MRR was
improved when the Initialization method is adopted. Initialization methods
that use Name information improved the MRR of target, enzyme, carrier and
transporter, which are the relations between drugs and proteins. The averaged
MRR was lower than that of the models without textual information.

When the ComplEx scoring function is used, the MRR decreased in the interact
and pathway relation, while the MRR increased on the categorical relations and
relations between drugs and proteins, these are the same tendency as the
DistMult algorithm. Especially in the category relation, the ComplEx scoring
function model without text information has a much lower MRR than TransE or
DistMult-based models, but the performance was improved by using text
information. The Initialization and Augmentation methods show higher macro-
averaged MRR than the model without text information.

When the SimplE scoring function is used, the model without text information
showed the lowest macro-averaged MRR; however, the Initialization model that
used the Synonyms information showed a higher MRR than the model without text
information for all relation types except pathway, and showed the highest
macro-averaged MRR in all models. These results showed that it is effective to
utilize text information during updating KG embeddings under the SimplE
scoring function.

These results show that the utility of textual information for learning KG
embeddings depends on the scoring functions and relation types. The textual
information is always useful in predicting categorical relations such as
category and ATC, while the text information can be harmful for other
relations and the utility depends on the scoring functions. The best settings
for each relation type are summarized in Table 6.5. This shows that there is
no best single embedding method. The best method to incorporate text
information including No Text and the most useful text type also depends on
the relation types.

| MRR | Method | Text Information
---|---|---|---
category | 0.3668 | SimplE | Initialization+Synonyms
ATC | 0.3820 | SimplE | Initialization+Name
pathway | 0.7513 | SimplE | No Text
interact | 0.8678 | ComplEx | No Text
target | 0.1243 | ComplEx | Augmentation
enzyme | 0.3590 | TransE | Alignment+Synonyms
carrier | 0.4155 | TransE | No Text
transporter | 0.3576 | TransE | No Text
Avg. (Macro) | 0.3839 | SimplE | Initialization+Synonyms
Table 6.5: Summary of the best settings for each relation

####  6.6.1 Analysis of the Data Ombalance of the Constructed KG

Why some models that use text information show lower performance in interact
and pathway relation and show higher performance in categorical relation and
drug-protein relation? In order to analyze these tendencies, The frequency of
nodes in the constructed KG is investigated. Figure 6.3 shows the distribution
of the frequencies of category nodes that have the category link and drug
nodes that have the interact link in train triples. Compared with the
distribution of drug nodes frequency, the frequency distribution of category
nodes is extremely imbalanced. The distribution shows that a small part of
category nodes have a large number of triples between drugs, and many other
category nodes have few triples, it could be difficult to predict triples that
contain these nodes. Even if it is difficult to train the representation of
nodes from the information of KG structure, it may be possible to predict the
correct triples by utilizing the textual embeddings encoded by pre-trained
BERT.

Figure 6.3: The distribution of the frequency of nodes in the train data set.
The frequencies of category nodes linked by category relation are highly
imbalanced, while the frequencies of drug nodes linked by interact relation
are less imbalanced.

####  6.6.2 Ablation Study of Augmentation Method

In the Augmentation method, multiple text items can be considered at the same
time. Table 6.6 shows the results of removing each text item. Here,
description and synonyms are text items that heterogeneous entities have in
common, indication, pharmacodynamics, mechanism-of-action and metabolism are
text items that only drug entities have, and gene-name is that only protein
entities have. From Table 6.6, it can be seen that the averaged MRR becomes
lower regardless of which text items are removed, and these results show that
all text items are effective for the link prediction task. In addition, the
averaged MRR drops greatly when description or synonyms is excluded, these are
the text items that many entities have. The averaged MRR also drops greatly
when text information with high coverage is excluded, such as metabolism-of-
action.

| Averaged MRR
---|---
Full text nodes | 0.3771
\- description | 0.3626
\- synonyms | 0.3655
\- indication (drug) | 0.3761
\- pharmacodynamics (drug) | 0.3727
\- mechanism-of-action (drug) | 0.3646
\- metabolism (drug) | 0.3689
\- gene-name (protein) | 0.3754
Table 6.6: Ablation study of text information on Augmentation method (ComplEx
score function)

####  6.6.3 Effect of Node Type Filtering

As explained in Sections 6.3.3, 6.5.4, 6.5.6, the model filters out impossible
negative instances by restricting the types of entities in the relations.
Table 6.7 shows the effect of the entity type filtering. Overall, by
performing entity type filtering, the averaged MRR is improved. In particular,
in the Augmentation method, entity type filtering is very effective; this is
because the Augmentation method adds textual nodes to the graph and is more
likely to create an inappropriate negative example during negative sampling.

|  | No Text | Initialization | Alignment | Aug.
---|---|---|---|---|---
|  |  | Name | Desc. | Syn. | Name | Desc. | Syno. |
TransE | w/ type filtering | 0.3319 | 0.2930 | 0.2950 | 0.3015 | 0.3337 | 0.3318 | 0.3302 | 0.2883
| w/o type filtering | 0.2759 | 0.2373 | 0.2265 | 0.2429 | 0.2738 | 0.2722 | 0.2753 | 0.1932
DistMult | w/ type filtering | 0.3380 | 0.3206 | 0.2979 | 0.3162 | 0.3261 | 0.3274 | 0.3265 | 0.3011
| w/o type filtering | 0.2217 | 0.2285 | 0.2423 | 0.2547 | 0.2462 | 0.2219 | 0.2464 | 0.1449
ComplEx | w/ type filtering | 0.3242 | 0.3681 | 0.3678 | 0.3692 | 0.3577 | 0.3190 | 0.3312 | 0.3771
| w/o type filtering | 0.2848 | 0.2906 | 0.2887 | 0.2981 | 0.2931 | 0.3052 | 0.3095 | 0.2373
SimplE | w/ type filtering | 0.2973 | 0.3829 | 0.3705 | 0.3839 | 0.3216 | 0.3387 | 0.3328 | 0.3549
| w/o type filtering | 0.2848 | 0.2906 | 0.2887 | 0.2981 | 0.2931 | 0.3052 | 0.3095 | 0.2373
Table 6.7: Comparison of averaged MRR performance for w/ (with) and w/o
(without) entity type filtering Examples where textual information is helpful
---
(a) |  Relation: category, textual model rank:1, non-textual model rank:65
| Head | ID |  DB13746 (drug entity)
|  | Name |  Bioallethrin
|  | Desc. |  Bioallethrin refers to a mixture of two of the allethrin isomers (1R,trans;1R and 1R,trans;1S) in an approximate ratio of 1:1, where both isomers are active ingredients. A mixture of the two same stereoisomers, but in an approximate ratio of R:S in 1:3, is called esbiothrin.
|  | Syn. |  Depalléthrine
| Tail | ID |  D013237 (category entity)
|  | Name |  Stereoisomerism
|  | Desc. |  The phenomenon whereby compounds whose molecules have the same number and kind of atoms and the same atomic arrangement, but differ in their spatial relationships.
|  | Syn. |  Molecular Stereochemistry, Stereochemistry, Molecular, Stereoisomers, Stereoisomer
(b) |  Relation: ATC, textual model rank:1, non-textual model rank:25
| Head | ID |  DB00369 (drug entity)
|  | Name |  Cidofovir
|  | Desc. |  Cidofovir is an injectable antiviral medication employed in the treatment of cytomegalovirus (CMV) retinitis in patients diagnosed with AIDS.
|  | Syn. |  CDV, Cidofovir anhydrous, Cidofovirum
| Tail | ID |  J05A (ATC entity)
|  | Name |  DIRECT ACTING ANTIVIRALS
|  | Desc. |  ATC entity has no description
|  | Syn. |  ATC entity has no synonyms
Examples where textual information is harmful
(c) |  Relation: pathway, textual model rank:25, non-textual model rank:1
| Head | ID |  P51589 (protein entity)
|  | Name |  Cytochrome P450 2J2
|  | Desc. |  This enzyme metabolizes arachidonic acid predominantly via a NADPH-dependent olefin epoxidation to all four regioisomeric cis-epoxyeicosatrienoic acids.
|  | Syn. |  1.14.14.1, Arachidonic acid epoxygenase, CYPIIJ2
| Tail | ID |  SMP0000695 (pathway entity)
|  | Name |  Etoricoxib Action Pathway
|  | Desc. |  Etoricoxib (also named as Arcoxia) is a COX-2 selective inhibitor. It can be used to treat fever, pain, swelling, inflammation, and platelet aggregation.
|  | Syn. |  pathway entity has no synonyms
(d) |  Relation: interact, textual model rank:4,119, non-textual model rank:1
| Head | ID |  DB08893 (drug entity)
|  | Name |  Mirabegron
|  | Desc. |  Mirabegron is a beta-3 adrenergic receptor agonist for the management of overactive bladder. It is an alternative to antimuscarinic drugs for this indication.
|  | Syn. |  Mirabegron
| Tail | ID |  DB00937 (drug entity)
|  | Name |  Diethylpropion
|  | Desc. |  A appetite depressant considered to produce less central nervous system disturbance than most drugs in this therapeutic category. It is also considered to be among the safest for patients with hypertension.
|  | Syn. |  alpha-Benzoyltriethylamine, alpha-Diethylaminopropiophenone, Amfepramone
Table 6.8: The content of the text in the examples where the difference
between the rank of textual model and the rank of non-textual model is largest
for each relation type. The score function SimplE was used for category, ATC
and pathway relation and ComplEx was used for interact relation. The
Augmentation model was selected as the model with textual information. The
highlighted part is the mention common to head and tail entities. The
Description and Synonyms are partly excerpted due to space limitations.

####  6.6.4 Case Study

As can be seen from Table 6.5, textual information acts harmfully in some
relations. In this section, the content of the text is analyzed to investigate
when the text information is harmful or helpful. Table 6.8 shows examples of
improved or worsened score ranks on the link prediction task. The examples are
where the difference between the rank of the textual model and the rank of the
non-textual model is largest, that is, examples where textual information is
most useful or harmful for each relation type. In addition, the cases are
narrowed down where the better rank is 1. In example (a), the highlighted
“stereoisomers” in the description of the drug entity appears in the synonyms
of the category entity. Similarly, in example (b), “antiviral” in the
description of the drug entity appears in the name of the ATC entity. The
description of the drug entity directly mentions the category in which the
drug is included, which is thought to have helped to predict the link of the
categorical relation type.

On the other hand, for the examples where the textual information is most
harmful, in example (c), the description of protein “Cytochrome P450 2J2” does
not directly mention the “Etoricoxib Action Pathway” pathway. In example (d),
the description of each drug entity mainly describes the indication of the
drug, not the relationship to other drugs. It is difficult to tell the cause
of the poor rank because multiple factors may be involved, but the description
of the head entity mainly explains the function and role of the head entity
itself, and there is no description that mentions the relationship with the
tail entity. This point is considered to be one of the causes of the textual
information becoming noise.

###  6.7 Summary

A new heterogeneous knowledge graph containing textual information PharmaHKG
from several databases is constructed. The combinations of three methods to
use textual information and four scoring functions on the link prediction task
are compared. The utility of text information and the best combination for the
link prediction depend on the target relation types. In addition, when the
averaged MRR is focused on all relation types, a method that combines SimplE
and text information achieved the highest MRR, and this result showed the
usefulness of text information in the link prediction task in the drug domain.

##  Integrating Heterogeneous Domain Information for Relation Extraction

This chapter proposes a novel method that utilizes the heterogeneous KG
information for relation extraction. This chapter includes work from Asada et
al. (2022) [11].

###  7.1 Background

Chapter 6 integrated multiple databases into a heterogeneous KG and conducted
a link prediction task on the heterogeneous KG to obtain representation
vectors of the drugs. This chapter proposes a novel method that effectively
combines the input sentence information and the heterogeneous KG information
to extract DDIs from texts. The model on the data set of the
DDIExtraction-2013 task is evaluated to demonstrate the usefulness of
heterogeneous KG information.

In Chapter 5, the approach to combining the input sentence representation
vector by BERT with the description and molecular structure representation
vectors was simply concatenating the respective representation vectors. This
approach is considered insufficient to capture the correlation between the
context around the drug mentions in the input sentence and the external
knowledge information.

This chapter uses the idea of “entity marker” to devise a model that
incorporates the heterogeneous KG embeddings from the lowest layer of BERT and
integrally considers the correlation of input sentence information and
heterogeneous KG information. Many relation extraction methods based on the
entity marker idea have been studied [97, 98]. The levitated marker model [99]
that most inspired the proposed method is described. Figure 7.1 shows the
overview of solid marker [100, 101] and levitated marker [99]. Solid marker
explicitly inserts two solid markers before and after the span to highlight
the span in the input sentence. Here, [Md] and [TK] stand for Method and Task,
and these markers indicate the type of mentions in the sentence. The levitated
marker first sets the pair of markers to share the same position with the
target tokens and then ties a pair of markers by a directional attention. The
levitated marker can identify the mentions without collapsing the original
input sentence. The mention and the marker are tied by an attention mechanism
of the Transformer model. The levitated marker embeddings are replaced with
the KG embedding to which the target drug mention corresponds, and consider
the relationship between the word embeddings and the KG embeddings for DDI
extraction.

Figure 7.1: Overview of the levitated marker. Tokens of the same color share
the position IDs in the levitated marker.

###  7.2 Method

####  7.2.1 Obtaining Heterogeneous KG Embeddings

In constructing the heterogeneous KG, the augmentation method described in
Chapter 6 is used. In the Augmentation method, textual nodes as well as entity
nodes on the heterogeneous KG are placed, and the relational triples in the
train data set are augmented.

One extension is made from the heterogeneous KG of Chapter 6, that is, the
molecular structural nodes of the drugs are added to the heterogeneous KG. An
overview of the newly constructed KG is shown in Figure 7.2. Similar to
initializing textual nodes with embeddings by a pre-trained BERT model,
molecular structural nodes with a pre-trained model of the SMILES string
coding representation embeddings [102] are also initialized.

The constructed pharmaceutical heterogeneous KG enables us to obtain Drug
representation vectors that take into account various information such as
hierarchical categorical information, interacted protein information, related
pathway information, drug textual information, and drug molecular structural
information. In the next section, a novel method for DDI extraction from the
literature using the obtained heterogeneous KG representations of drugs is
described.

Figure 7.2: Heterogeneous KG with additional drug molecular structure
information

####  7.2.2 DDI Extraction from Texts with Heterogeneous KG Embeddings

An overview of the DDI extraction model is shown in Figure 7.3. the proposed
model adopts the idea of a levitated entity marker, and two drug mention
markers are placed at the end of a sentence.

##### Embedding Layer

The input sentence $S$ is tokenized to sub-word units by the BERT tokenizer
and converted into the format shown below:

$S=\\{\texttt{[CLS]},w_{1},w_{2},\cdots,w_{m_{1}},\cdots,w_{m_{2}},\cdots,\texttt{[SEP]},\texttt{[KG1]},\texttt{[KG2]}\\},$
(7.1)

where $w_{i}$ is the $i$-th sub-word, and [CLS], [SEP] are the special tokens
of BERT, $m_{1}$ is the drug mention 1 (DRUG1), and $m_{2}$ is the drug
mention 2 (DRUG2), and [KG1], [KG2] are markers for mapping mentions and KG
entries.

Then, in the lowest embedding layer of the BERT model, the sub-word $w_{i}$
and special tokens are looked up from the pre-trained BERT embedding table and
converted to embedding vectors. In addition, the marker embeddings are
replaced with the heterogeneous KG embeddings. All tokens are converted to
embedding vectors and the embedding matrix $\bm{W}^{0}$ of the input sentence
is shown as follows:

$\bm{W}^{0}=\\{\mathbf{w}_{\texttt{CLS}},\mathbf{w}_{1},\mathbf{w}_{2},\cdots,\mathbf{w}_{m_{1}},\cdots,\mathbf{w}_{m_{2}},\cdots,\mathbf{w}_{\texttt{SEP}},\mathbf{w}_{\texttt{KG1}},\mathbf{w}_{\texttt{KG2}}\\},$
(7.2)

Here, $\mathbf{w}_{i}$, $\mathbf{w}_{\texttt{CLS}}$ and
$\mathbf{w}_{\texttt{SEP}}$ are looked up from the BERT embedding table
$\bm{V}_{BERT}\in\mathbb{R}^{N_{v}\times d}$ and $\mathbf{w}_{\texttt{KG1}}$,
$\mathbf{w}_{\texttt{KG2}}$ are looked up from the heterogeneous KG embedding
table $\bm{V}_{KG}\in\mathbb{R}^{N_{e}\times d}$. $d$ is the dimension of the
embedding vector, and $N_{v}$ is the number of vocabularies of the BERT
tokenizer, and $N_{e}$ is the number of entities in the heterogeneous KG.

##### Self-Attention Layer

The embedding matrix $\bm{W}^{0}$ is the input to the $L$-layers of BERT self-
attention module:

$\bm{W}^{l+1}=\mathrm{SelfAttention}^{l}(\bm{W}^{l}),$ (7.3)

where $l=0,1,2,\cdots,L-1$. The output of the final attention layer
$\bm{W}^{L}$ is shown as follows:

$\bm{W}^{L}=\\{\textbf{h}_{\texttt{CLS}},\textbf{h}_{1},\textbf{h}_{2},\cdots,\textbf{h}_{m_{1}},\cdots,\textbf{h}_{m_{2}},\cdots,\textbf{h}_{\texttt{SEP}},\textbf{h}_{\texttt{KG1}},\textbf{h}_{\texttt{KG2}}\\},$
(7.4)

where $\textbf{h}_{i}\in\mathbb{R}^{d}$ is the hidden state vector of $i$-th
token.

As shown in Figure 7.3, mention 1 and its KG entity, and mention 2 and its KG
entity share the position ID, which ties mention and marker by directional
attention.

##### Prediction Layer

The loss function is calculated from the hidden representation vectors of the
final layer of BERT architecture. First, the hidden representation of the
[CLS] token and two drug mention tokens are concatenated as follows:

$\textbf{h}_{all}=[\textbf{h}_{\texttt{CLS}};\textbf{h}_{m_{1}};\textbf{h}_{m_{2}}].$
(7.5)

The concatenated representation vector $\textbf{h}_{all}$ is passed through a
dense layer and middle layer representation is obtained,

$\textbf{h}_{mid}=\bm{W}_{mid}\textbf{h}_{all}+\textbf{b}_{mid},$ (7.6)

where $\bm{W}_{mid}\in\mathbb{R}^{3d\times d_{m}}$,
$\textbf{b}_{mid}\in\mathbb{R}^{d_{m}}$ are the trainable weight and bias, and
$d_{m}$ is the dimension of middle layer vector. Then the middle layer
representation is converted into fully-connected representation as follows:

$\textbf{h}_{fc}=\bm{W}_{fc}\textbf{h}_{mid}+\textbf{b}_{fc},$ (7.7)

where $\bm{W}_{fc}\in\mathbb{R}^{d_{m}\times c}$,
$\textbf{b}_{fc}\in\mathbb{R}^{c}$ are the trainable weight and bias, and $c$
is the number of label types. The fully-connected representation vector
$\textbf{h}_{fc}$ is converted to probability form by the softmax function:

$\hat{\bm{y}}=\mathrm{softmax}(\textbf{h}_{fc}).$ (7.8)

The cross-entropy loss between the prediction probability $\hat{\bm{y}}$ and
the gold label $\bm{y}$ is employed,

$L=-\sum\bm{y}\log\hat{\bm{y}}.$ (7.9)

The model parameters are updated to minimize the loss $L$.

Figure 7.3: The DDI extraction model which utilize heterogeneous information
of drugs

###  7.3 Experimental Settings

####  7.3.1 Mention Linking

In contrast to the previous chapters, in this chapter, not only with DrugBank
drug entities but also with MeSH categorical terms, ATC code categorical
terms, and UniProt protein terms are linked. The DDIExtraction-2013 shared
task data set consists of four types of entities, DRUG, DRUG_N, BRAND, and
GROUP. The GROUP type drug mentions may be linked to categorical terms, so the
linking coverage is better than in the previous chapters.

As a result, 97.05% of the unique mentions in train data set were linked to
heterogeneous KG entries, and 97.89% of the unique mentions in test data set
were linked. As for the coverage on instances where mention 1 and mention 2
are both linked, the coverage of train data instances was 91.90% (25,540 /
27,792), and the coverage of test instances was 90.75% (5,187 / 5,716).

When the drug mention is not linked to the KG entries, the special tokens (KG1
or KG2) are masked and these embeddings are excluded from the attention
calculation. In this way, the proposed model takes into account KG embedding
information for linked examples, and for unlinked examples, the model behaves
like the baseline model.

####  7.3.2 Link Prediction Settings

The same train/validation/test split triples is used as the data sets created
in Chapter 6 and the link prediction task is conducted. Mini-batch training
using the Adagrad [103] optimizer is employed.

Hyper-parameter tuning is conducted on the validation data set. Hyper-
parameters include initial learning rate and mini-batch size. As in Chapter 6,
TransE, DistMult, ComplEx and SimplE were used for score functions. The
employed hyper-parameters is shown in Table 3.3.

SMILES strings are extracted from DrugBank database. The 9,859 drug entities
in the heterogeneous KG data set have SMILES strings. Relation triples (drug,
structure, SMILES) are added to the train data set and molecular structural
nodes (SMILES nodes) are initialized by the embedding vectors of the pre-
trained SMILES representation language model.

The [CLS] token representation of PubMedBERT [93] is used as the initial value
of the textual nodes, and ChemBERTa [102] was used as the initial value for
the molecular structural nodes. Chrithranada et al. pre-trained ChemBERTa on
77M unique SMILES from PubChem [104], the world’s largest open-source
collection of chemical structures. The SMILES were canonicalized and globally
shuffled to facilitate large-scale pre-training. ChemBERTa is based on the
RoBERTAa [105] transformer model. In pre-training, ChemBERTa model masks 15%
of the tokens in each SMILES string.

####  7.3.3 DDI Extraction Settings

The AdamW optimizer [58] is employed, and mixed-precision training [59] is
employed for memory efficiency. The weight averaging [106, 107] technique is
employed, where all model parameters are saved at each update and the model
predicts the DDI label from the average of all stored parameters.

PubMedBERT is employed as the textual representation model for the DDI
extraction task. The word embeddings of PubMedBERT and heterogeneous KG
embeddings are frozen during training.

Stratified 5-fold cross-validation is conducted on the DDIExtraction-2013
train data set for hyper-parameter tuning. Hyper-parameters include learning
rate, weight decay coefficient, dropout probability and mini-batch size. The
employed hyper-parameters are shown in Table 7.1. The significance results are
based on the Randomization test [51].

Parameter | Value
---|---
Link Predction
Entity embedding size | 768
Learning rate | 0.25
Regularization parameter | 9e-9
Mini-batch size | 8,192
DDI Extraction
Embedding size | 768
Middle layer size | 768
Maximum sequence length | 256
Learning rate | 5e-5
Regularization parameter | 8e-0
Dropout probability | 0.5
Mini-batch size | 128
Table 7.1: Hyper-parameters for link prediction and DDI extraction model

###  7.4 Results and Discussions

First, the results of link prediction task on the heterogeneous KG are shown
in Table 7.2. For each of the four score functions, TransE, DistMult, ComplEx
and SimplE, the four methods listed below are evaluated:

entity nodes only

This is a method that trains only from the nodes contained in the
heterogeneous KG. In Figure 7.2, only the actual nodes (the blue one) is
included in the train data set.

with textual nodes

In this method, in addition to the actual nodes, pseudo nodes that hold
textual information such as synonyms and descriptions of entity items are
added to the heterogeneous KG. In Figure 7.2, textual nodes (the red one) are
added to the actual nodes.

with molecular structural nodes

In this method, pseudo molecular structure modes are added to the
heterogeneous KG in addition to the actual nodes. As shown in Figure 7.2,
molecular structural nodes (the yellow ones) are added.

with textual nodes and molecular structural nodes

In this method, both textual nodes and molecular structural nodes are added.
This approach can consider a wide variety of heterogeneous information about
drugs.

Table 7.2 showed that the TransE model performs poorly for both MRR and
Hits@k. This should be due to the inability of the TransE model to capture the
symmetrical relational triples. The TransE model showed low performance of MRR
and Hits@k because the heterogeneous KG contained a large proportion of the
(drug, interact, drug) triples, which is a symmetric relationship.
Furthermore, the TransE model showed the highest MRR and Hits@k when using the
“entity nodes only” method, meaning that adding textual or molecular
structural nodes did not improve link prediction performance.

On the other hand, DistMult, ComplEx, and SimplE, which can consider symmetric
relationships, showed higher performance than TransE. These models
successfully improved the performance of link prediction task by adding
textual nodes and molecular structural nodes, respectively. And when both
textual nodes and molecular structural nodes are added, the further
performance improvement was achieved. As shown by the underlined values in
Table 7.2, the highest performance for all MRR and Hits@k metrics was achieved
by the method using both textual and molecular structural nodes. These results
show that rich embedding representations are obtained by considering various
heterogeneous domain information.

| TransE | DistMult | ComplEx | SimplE
---|---|---|---|---
entity nodes only
MRR | 0.3114 | 0.6732 | 0.6627 | 0.6228
Hits@1 | 0.0108 | 0.5416 | 0.5436 | 0.3954
Hits@3 | 0.5590 | 0.7680 | 0.7377 | 0.8287
Hits@10 | 0.7364 | 0.9138 | 0.8892 | 0.9481
with textual nodes
MRR | 0.2894 | 0.7702 | 0.7874 | 0.7175
Hits@1 | 0.0092 | 0.6199 | 0.6424 | 0.5019
Hits@3 | 0.5102 | 0.9104 | 0.9258 | 0.9303
Hits@10 | 0.6987 | 0.9703 | 0.9722 | 0.9744
with molecular structural nodes
MRR | 0.3003 | 0.7677 | 0.7313 | 0.7156
Hits@1 | 0.0094 | 0.6171 | 0.5383 | 0.4987
Hits@3 | 0.5352 | 0.9092 | 0.9180 | 0.9307
Hits@10 | 0.7195 | 0.9700 | 0.9717 | 0.9746
with textual nodes and molecular structural nodes
MRR | 0.2877 | 0.7933 | 0.7923 | 0.7235
Hits@1 | 0.0091 | 0.6610 | 0.6509 | 0.5086
Hits@3 | 0.5051 | 0.9166 | 0.9279 | 0.9386
Hits@10 | 0.6995 | 0.9711 | 0.9729 | 0.9753
Table 7.2: The comparison of link prediction performance on heterogeneous KG.
Figures marked in bold indicate the highest performance when limited to each
individual score function. The underlines indicate the highest performance for
all score functions.

Then, the performance of DDI extraction models that leverage these
heterogeneous KG embeddings is described. The score functions and hyper-
parameters are chosen by the results of 5-fold cross-validation, as described
in later sections. Table 7.3 shows the performances evaluated on the
DDIExtraction-2013 task test set. The proposed model PubMedBERT+HKG achieved
the micro-averaged F-score of 85.40%, showing the current state-of-the-art
performance. The proposed model achieved a significant F-score improvement of
1.70 percent points over the baseline model by using heterogeneous information
about drugs. Compared to other existing models, the PubMedBERT+HKG model
showed a higher F-score. The SciBERT+Mol.+Desc. model is an ensemble of
SciBERT+Mol. and SciBERT+Desc. The proposed model showed higher performance
than the ensemble of multiple models.

| Method | P | R | F (%)
---|---|---|---|---
Reported scores | CNN [26] | 75.29 | 60.37 | 67.01
| BiLSTM [31] | 67.77 | 66.80 | 67.28
| PubMedBERT [93] | - | - | 82.42
| SciFive-Large [108] | - | - | 83.67
The author’s implementation | CNN + Mol. [43] | 73.31 | 71.81 | 72.55
| SciBERT + Mol. [9] | 83.57 | 82.12 | 82.84
| SciBERT + Desc. [9] | 84.05 | 81.81 | 82.91
| SciBERT + Mol. + Desc. [9] | 85.36 | 82.83 | 84.08
| PubMedBERT (baseline) | 83.45 | 83.96 | 83.70
| PubMedBERT + HKG | 85.32 | 85.49 |  85.40*
Table 7.3: The comparison of DDI extraction performance on DDIExtraction-2013
test data set. * indicates performance improvement from PubMedBERT (baseline)
at a significance level of $p<0.001$.

Then, the F-scores for each of the four DDI labels of DDIExtraction-2013 task
data set are shown in Table 7.4. As shown in Table 7.4, the model with
heterogeneous KG information improves F-scores for all DDI types. In
particular, the F-score greatly improves for Mechanism relation. HKG model
improved F-score by 3.80 percent points from the baseline model.

Method |  | Mech. | Effect | Adv. | Int. | Avg. (%)
---|---|---|---|---|---|---
PubMedBERT (baseline) | P | 87.12 | 87.18 | 78.71 | 82.69 | 83.93
| R | 85.10 | 92.31 | 88.33 | 44.79 | 77.63
| F | 86.10 | 89.67 | 83.25 | 58.11 | 79.28
PubMedBERT + HKG | P | 88.96 | 88.84 | 81.06 | 81.13 | 84.99
| R | 88.08 | 93.67 | 89.17 | 44.79 | 78.92
| F | 88.52 | 91.19 | 84.92 | 57.72 | 80.58
Table 7.4: The comparison of F-scores for individual DDI types and macro-
averaged F-score on DDIExtraction-2013 test data set

####  7.4.1 Selecting Score Functions

Method | P | R | F (%)
---|---|---|---
baseline | 82.19 | 83.23 | 82.54
\+ HKG (TransE) | 83.46 | 84.37 | 83.82
\+ HKG (DistMult) | 83.68 | 85.42 | 84.48
\+ HKG (ComplEx) | 83.68 | 84.32 | 83.90
\+ HKG (SimplE) | 83.46 | 84.50 | 83.86
Table 7.5: The comparison of DDI extraction performances with different score
functions for training KG embeddings. the author performs 5-fold cross-
validation and show the average of F-scores over the five validation data
sets.

In this section, the author will discuss which score function was effective
for DDI extraction. Table 7.3 shows the average F-scores for the five
validation data sets for each score function. F-score is higher than the
baseline model when using heterogeneous KG embeddings trained by any of score
functions. The improvement of F-score points from the baseline model is 1.28
for the TransE model and 1.94 for the DistMult model and 1.36 for the ComplEx
and 1.32 for the SimplE model. From these results, the DistMult score function
is adopted for the DDI extraction model. The DistMult model preformed best on
the link prediction task on MRR, Hits@1, and also showed the best performance
on the DDI extraction task.

####  7.4.2 Ablation Study on Model Architecture

Method | P | R | F (%) |  $\Delta$ (pp)
---|---|---|---|---
Full model | 83.68 | 85.42 | 84.48 | -
w/o sharing position ids | 82.49 | 86.04 | 84.18 | 0.30
w/o freezing KG embeddings | 83.69 | 84.32 | 83.90 | 0.58
w/o CLS representation | 82.63 | 85.17 | 83.81 | 0.67
w/o mention representation | 82.07 | 85.69 | 83.78 | 0.70
w/o KG embeddings (baseline) | 82.19 | 83.23 | 82.54 | 1.94
Table 7.6: The ablation study on the model architecture. The performance with
5-fold cross-validation on the training set is showed.

This section provides ablation studies. Table 7.6 shows the ablation study
results on 5-fold cross-validation data sets.

##### w/o Sharing Position IDs

First, the case excluding the sharing of position IDs is discussed. The
sharing of position IDs has the effect of linking the mention embeddings and
KG embeddings. As shown in Figure 7.3, with position sharing, the position ID
of [KG1] is 1 of the drug mention 1 and the position ID of [KG2] is 6 of the
drug mention 2. When this sharing is disabled, the position IDs of [KG1] and
[KG2] are the values following from the IDs of the [SEP] token. From Table
7.6, when position sharing is excluded, the F-score is reduced by 0.30 percent
points from the full model.

##### w/o freezing KG embeddings

In the proposed model, the BERT embeddings the KG embeddings are frozen and
the attention weight is trainable. Table 7.6 shows that when embedding
freezing is disabled, 0.58 percent points of F-score is lower than the full
model. The author thinks the reason embedding freezing is effective is that if
the embedding is not frozen, there will be a gap between the KG embeddings
when drugs that appear on the train set and those that appear only on the test
set.

##### w/o CLS representation

When $\textbf{h}_{\texttt{CLS}}$ was excluded from the input vector of the
middle layer, the F-score decreased by 0.67 percent points. The CLS token
representation holds information of the entire sentence. It is effective to
use the representation of the [CLS] token.

##### w/o mention representation

When $\textbf{h}_{m_{1}}$ and $\textbf{h}_{m_{2}}$ were excluded from the
input vector of the middle layer, the F-score decreased by 0.70 percent
points. In addition to the [CLS] token representation, it is effective to use
the drug mention representation.

##### w/o KG embeddings

Finally, the case without KG embeddings is discussed. In the baseline model,
[KG1] and [KG2] tokens are not fed to the BERT architecture and KG embeddings
are not used. Except for this point, the model structure is the same as the
proposed model.

The use of heterogeneous KG information increased the F-score by 1.94 percent
points. This result showed that the drug heterogeneous information is useful
for DDI extraction from literature.

Method | P | R | F (%) |  $\Delta$ (pp)
---|---|---|---|---
PubMedBERT + HKG | 83.68 | 85.42 | 84.48 | -
w/o Protein nodes | 83.22 | 85.04 | 84.08 | 0.40
w/o MeSH category nodes | 83.28 | 83.75 | 83.38 | 1.10
w/o ATC nodes | 83.33 | 84.27 | 83.73 | 0.75
w/o Pathway nodes | 83.98 | 84.85 | 84.30 | 0.18
w/o Textual nodes | 83.75 | 84.20 | 83.86 | 0.62
w/o Molecular structure nodes | 83.36 | 84.30 | 83.71 | 0.77
Table 7.7: The ablation study on node types. The performance with 5-fold
cross-validation on the training set is showed.

####  7.4.3 Ablation Study on Heterogeneous KG Node Types

Table 7.7 shows the ablation study on the effect of individual KG node type.
As shown in Table 7.7, all types of nodes in the heterogeneous KG contribute
to the performance improvement of DDI extraction from the literature. The
results show the importance of simultaneously considering multiple pieces of
drug-related information. Among the types of nodes, the MeSH categorical
information contributed most to the performance improvement while the pathway
information contributed least.

####  7.4.4 Verification of DDI Label Leakage from KGs

The constructed heterogeneous KG contains triples of interact relations
between Drug nodes and Drug nodes. This section examines whether the interact
triples directly affects the performance improvement in DDI extraction from
the literature and whether there is a leakage of DDI labels. First, the
percentage of drug pairs in the DDIExtraction-2013 train data set that are
registered as interact relations in the KG is shown. 39.79% (11,060 / 27,792)
of the drugs pairs are included in the constructed KG and others are not
included in the KG. It should be noted that the DDI labels in the
DDIExtraction-2013 data set are Mechanism, Effect, Advice and Int., whereas
there is only on relation label interact in the KG. Table 7.8 shows a
comparison of DDI extraction performance for drug pairs that are included in
the KG and drug pairs that are not included in the KG. As shown in Table 7.8,
both for drug pairs that are registered in the KG and drug pairs that are not
registered in the KG, the proposed model (baseline + HKG) outperformed the
baseline model. When the pairs are included in the KG, the improvement of
F-score is 1.1 percent, 0.97 pp, whereas when the pairs are not included in
the KG, the improvement of F-score is 3.4 percent, 2.79 pp. Therefore, the
performance improvement of the proposed model is greater when the drug pairs
is not registered in the KG than when they are registered in the KG. The
author believes that these results indicate that the proposed model does not
improve DDI extraction performance because of the leakage of DDI labels from
the constructed KG.

Method | P | R | F (%)
---|---|---|---
All drug pairs |  |  |
baseline | 82.19 | 83.23 | 82.54
baseline + HKG | 83.68 | 85.42 | 84.48
Drug pairs that are included in KGs as interact triples |  |  |
baseline | 85.11 | 85.74 | 85.25
baseline + HKG | 85.94 | 86.73 | 86.22
Drug pairs that are NOT included in KGs as interact triples |  |  |
baseline | 79.83 | 80.94 | 80.23
baseline + HKG | 81.83 | 84.32 | 83.02
Table 7.8: The DDI extraction performance for drug pairs that are included in
the constructed heterogeneous KG and drug pairs that are not included in KG

####  7.4.5 Learning Curve

The learning curve of baseline model and proposed model is shown in Figure
7.4. Figure 7.4 plots the average of the F-scores of the five validation data
sets per epoch. Learning curves show that the proposed method always has a
higher F-score than the baseline model. From the 4th epoch, the model using KG
embeddings outperforms the baseline in F-score more than 2 percent points.

Figure 7.4: Learning curve of baseline model and the proposed model on 5-fold
cross-validation data sets. The average of F-scores over the five validation
data sets is reported.

####  7.4.6 Analysis of Prediction Results

The confusion matrices of the baseline and the proposed model are shown in
Figure 7.5. The numbers indicate the total count of five validation data sets.
The proposed method reduced the all patterns of errors (non-diagonal
components in the table) compared to the baseline model. In particular, errors
in which the model incorrectly predicts the Effect interaction as negative and
errors in which the model incorrectly predicts the negative as Effect are
greatly reduced. For Mechanism, Advice, and Int. relations, the use of
heterogeneous KG information also reduced the number of cases of false
negative or false positive relations. On the other hand, there were cases in
which the use of heterogeneous KG information slightly increased errors by
classifying relations into wrong types, e.g., incorrectly classifying a
Mechanism relation as Effect.

In addition, four examples of prediction results are shown. Example 1, 2, and
3 in Table 7.9 are cases correctly predicted by using heterogeneous KG
information but incorrectly predicted by the baseline model. In Examples 1 and
2, many drug entities appear, and DRUG1 and DRUG2 are included in parentheses.
As shown in Example 3, the baseline model tends to predict the cases where the
distance between DRUG1 and DRUG2 is extremely short as negative, but the
proposed model correctly predicts them. From these examples, heterogeneous KG
representation of the drug entities may be helpful to predict correct
relations when the prediction is difficult only from their surrounding
contexts in the sentences.

Example 4 is a case incorrectly predicted by using heterogeneous KG
information while correctly predicted by the baseline model. According to the
annotation guideline of DDIExtraction-2013 data set, an interaction should
only be annotated when it occurs in the text. Example 4 shows some studies
about given interactions were performed, however, the sentence does not
provide any evidence. In such a case, background knowledge of drugs may have
disturbed correct prediction.

Figure 7.5: The confusion matrices (left: baseline, right: proposed method).
The vertical axis shows the actual labels and the horizontal axis shows the
predicted labels. The numbers indicates the total count of five validation
data sets.  Example 1
---
Text: In patients receiving nonselective DRUGOTHER (DRUGOTHER) (e.g., DRUG1)
in combination with DRUGOTHER (e.g., DRUGOTHER, DRUGOTHER, DRUGOTHER,
DRUGOTHER, DRUG2), there have been reports of serious, sometimes fatal,
reactions.
DRUG1: selegiline hydrochloride, DRUG2: venlafaxine
Gold label: Effect Baseline: negative Proposed model Effect:
Example 2
Text: DRUGOTHER: In a study of 7 healthy male volunteers, DRUG1 treatment
potentiated the blood glucose lowering effect of DRUGOTHER (a DRUGOTHER
similar to DRUG2) in 3 of the 7 subjects.
DRUG1: acitretin, DRUG2: chlorpropamide
Gold label: negative Baseline: Effect Proposed model negative:
Example 3
Text: Caution should be exercised when considering the use of DRUG1 and DRUG2
in patients with depressed myocardial function.
DRUG1: BREVIBLOC, DRUG2: verapamil
Gold label: Advice Baseline: negative Proposed model Advice:
Example 4
Text: To determine whether DRUG1 has a direct effect on the distribution of
DRUG2, the elimination and distribution of DRUGOTHER was studied in six
patients, five lacking kidney function and one with a partially impaired renal
function, in the presence or absence of DRUGOTHER.
DRUG1: probenecid, DRUG2: cloxacillin
Gold label: negative Baseline: negative Proposed model Mechanism:
Table 7.9: Case studies of the proposed model

###  7.5 Summary

This chapter first added the molecular structure information of drugs to the
KG data set constructed in the previous chapter. The results on the link
prediction task showed that the MRR and Hits@k was improved by considering the
molecular structure information of drugs.

Then, a method to use the heterogeneous KG representations for the relation
extraction task was proposed. The proposed model incorporates heterogeneous KG
embeddings into the input sentence in the form of levitated markers and
considers the relationship between contexts and KG information through an
attention mechanism. In the experiment, a significant improvement of 1.70
percent points on the DDIExtraction-2013 data set was achieved by using
heterogeneous KG information.

##  Conclusions

###  8.1 Summary

This thesis proposed a novel approach that focuses on heterogeneous domain
information for relation extraction from the literature. In case studies, the
proposed method can perform DDI extraction integrally considering
heterogeneous information about drugs, and showed the usefulness of utilizing
various information about drugs for the DDI extraction task. After describing
an attention mechanism for relation extraction in Chapter 3 and reviewing an
approach to using a single kind of domain information in Capter 4, this thesis
devised neural relation extraction models that can consider heterogeneous
domain information.

Chapter 5 proposed a novel neural method for relation extraction from text
using large-scale raw text information and two kinds of domain information,
which are the drug descriptions and the drug molecular structure information.
The results show that the large-scale raw text information with SciBERT
greatly improves the performance of DDI extraction from the literature on the
DDIExtraction-2013 data set. In addition, either of the drug description and
the molecular structure information can further improve the performance for
specific DDI types, and their simultaneous use can improve the performance on
all the DDI types.

Chapter 6 constructed a new heterogeneous knowledge graph containing textual
information PharmaHKG from several databases. The combinations of three
methods to use textual information and four scoring functions on the link
prediction task are compared. The utility of text information and the best
combination for the link prediction depends on the target relation types were
found. In addition, when the averaged MRR for all relation types was in focus,
a method that combines SimplE and text information achieved the highest MRR,
and this result showed the usefulness of text information in the link
prediction task in the drug domain.

Chapter 7 proposes a neural architecture that integrates sentence information
and knowledge graph information constructed in Chapter 6. The molecular
structure information of drugs is added to the KG data set constructed in the
previous chapter. The results on the link prediction task showed that the MRR
and Hits@k were improved by considering the molecular structure information of
drugs. Then, a method to use the heterogeneous KG representations for the
relation extraction task was proposed. The proposed model incorporates
heterogeneous KG embeddings into the input sentence in the form of levitated
markers and considers the relationship between contexts and KG information. In
the experiment, an improvement of 1.70 percent points on the
DDIExtraction-2013 data set was achieved by adding heterogeneous KG
information.

###  8.2 Future Work

####  8.2.1 Employing the Deep Neural Entity Linking Method

As mentioned in Chapter 7, the coverage of linking drug mentions in the input
sentence to drug entries in the KG is only about 90%, which has become a
bottleneck in the overall DDI extraction model. In recent years, many deep
neural entity linking models [109] have been proposed, and these models can
classify the input mentions into KG entries even if the number of KG entries
is more than several hundred thousand. The deep neural linking model has been
reported to perform better than the model with simple string matching, and
higher coverage and more accurate linking are expected by employing the deep
neural linking model in the model.

In addition, when the issues about memory usage and training time are solved,
a method that jointly train entity linking and DDI extraction can be realized.
These ideas are left for future work.

####  8.2.2 Joint Learning of BERT and KG Embeddings

In the proposed method, the learning of KG embeddings is decoupled from the
learning of DDI extraction. If the KG embeddings and BERT model are jointly
updated, it is expected that DDI label information can be taken into account
when updating KG embeddings and the learning KG representation can be more
suitable for DDI extraction.

Many GNN-based text classification methods [110, 111] have been proposed that
perform the sentence classification task by placing words in the input
sentence on a graph space and learning the graph representation. A possible
extension of these models is to place the input sentences in a heterogeneous
KG and train the word/sentence entities and other heterogeneous entities
jointly to perform DDI extraction.

####  8.2.3 End-to-End DDI Extraction

DDI extraction from the literature consists of two parts: drug mention
recognition and DDI extraction between the mention pair, and in this thesis,
the author focuses on the DDI extraction part. Efficient simultaneous training
of the two tasks may improve the performance of both tasks. It is important to
analyze the impact of using heterogeneous information in the DDI extraction
part on the mention recognition part.

Furthermore, drug mention recognition is also closely correlated with drug
mention linking to KG entries. KG entries information can help to improve the
drug mention recognition performance. The construction of a framework of
combining the drug mention recognition part, drug mention linking part and DDI
extraction part is left for future work.

####  8.2.4 Extension to Other Tasks

This thesis limits the experiment to DDI extraction and drug-protein
interaction extraction. The main idea of the proposed model consists of two
parts: representation of heterogeneous items in the form of KG and integration
of representation embeddings by an attention mechanism. The proposed method
can be extended to other tasks, since the method is not domain specific and
domain information from many other fields can be applied to the input format
of the model. The author would like to apply the proposed model to other tasks
and test the versatility of the method.

## Publications

  * •

Using drug descriptions and molecular structures for drug-drug interaction
extraction from literature [9]
Masaki Asada, Makoto Miwa, and Yutaka Sasaki, Bioinformatics volume 37, issue
12, pages 1739-1746, 2021.

  * •

Representing a heterogeneous pharmaceutical knowledge-graph with textual
information [10]
Masaki Asada, Nallappan Gunasekaran, Makoto Miwa, and Yutaka Sasaki, Frontiers
in Research Mtrics and Analytics, volume 6, pages 1-13, 2021.

  * •

TTI-COIN at BioCreative VII Track 1 - Drug-protein interaction extraction with
external database information [52]
Naoki Iinuma, Masaki Asada, Makowo Miwa, and Yutaka Sasaki, In Proceedings of
BioCreative VII workshop, volume 1, pages 49-53, 2021.

  * •

Integrating heterogeneous knowledge graphs into drug-drug interaction
extraction from the literature [11]
Masaki Asada, Makoto Miwa, and Yutaka Sasaki, Bioinformatics, 2022.

## References

  * [1] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep Learning.  The MIT Press, 2016.
  * [2] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In Proceedings of NeurIPS 2013, pages 3111–3119, 2013.
  * [3] Aditya Grover and Jure Leskovec.  node2vec: Scalable feature learning for networks.  In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855–864, 2016.
  * [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of NAACL-HLT 2019, pages 4171–4186, Minneapolis, Minnesota, June 2019.
  * [5] Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, and Andrew Y. Ng.  Multimodal deep learning.  In Proceedings of ICML 2011, page 689–696, Madison, WI, USA, 2011\. Omnipress.
  * [6] Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun.  Multi-view learning overview.  Inf. Fusion, 38(C):43–54, nov 2017.
  * [7] Council on Family Health.  DRUG INTERACTIONS: WHAT YOU SHOULD KNOW.  Federal Citizen Information Center, 2001.
  * [8] David L Sackett.  Evidence-based medicine.  In Seminars in perinatology, volume 21, pages 3–5. Elsevier, 1997\.
  * [9] Masaki Asada, Makoto Miwa, and Yutaka Sasaki.  Using drug descriptions and molecular structures for drug–drug interaction extraction from literature.  Bioinformatics, 37(12):1739–1746, 10 2020.
  * [10] Masaki Asada, Nallappan Gunasekaran, Makoto Miwa, and Yutaka Sasaki.  Representing a heterogeneous pharmaceutical knowledge-graph with textual information.  Front. Res. Metr. Anal., 6:670206, July 2021.
  * [11] Masaki Asada, Makoto Miwa, and Yutaka Sasaki.  Integrating heterogeneous knowledge graphs into drug-drug interaction extraction from the literature.  Bioinformatics, 2022.
  * [12] Sachin Pawar, Girish K. Palshikar, and Pushpak Bhattacharyya.  Relation extraction : A survey.  CoRR, abs/1712.05191, 2017.
  * [13] George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie M Strassel, and Ralph M Weischedel.  The automatic content extraction (ace) program-tasks, data, and evaluation.  In Lrec, volume 2, pages 837–840. Lisbon, 2004.
  * [14] Xavier Carreras and Lluís Màrquez.  Introduction to the CoNLL-2004 shared task: Semantic role labeling.  In Proceedings of CoNLL 2004, pages 89–97, Boston, Massachusetts, USA, May 6 - May 7 2004.
  * [15] Iris Hendrickx, Su Nam Kim, Zornitsa Kozareva, Preslav Nakov, Diarmuid Ó Séaghdha, Sebastian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz.  SemEval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals.  In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 33–38, Uppsala, Sweden, July 2010.
  * [16] Razvan Bunescu, Ruifang Ge, Rohit J Kate, Edward M Marcotte, Raymond J Mooney, Arun K Ramani, and Yuk Wah Wong.  Comparative experiments on learning information extractors for proteins and their interactions.  Artificial intelligence in medicine, 33(2):139–155, 2005.
  * [17] Sampo Pyysalo, Antti Airola, Juho Heimonen, Jari Björne, Filip Ginter, and Tapio Salakoski.  Comparative analysis of five protein-protein interaction corpora.  In BMC bioinformatics, volume 9, pages 1–11. BioMed Central, 2008\.
  * [18] Isabel Segura-Bedmar, Paloma Martínez, and María Herrero-Zazo.  SemEval-2013 task 9 : Extraction of drug-drug interactions from biomedical texts (DDIExtraction 2013).  In Proceedings of SemEval 2013, pages 341–350, Atlanta, Georgia, USA, June 2013.
  * [19] Martin Krallinger, Obdulia Rabal, Saber Ahmad Akhondi, Martín Pérez Pérez, Jesus Santamaría, Gael Pérez Rodríguez, Georgios Tsatsaronis, Ander Intxaurrondo, J. A. Lopez, Umesh K. Nandal, Erin M. van Buel, Ambika Chandrasekhar, Marleen Rodenburg, Astrid Lægreid, Marius A. Doornenbal, Julen Oyarzábal, Anália Lourenço, and Alfonso Valencia.  Overview of the biocreative vi chemical-protein interaction track.  In Proceedings of the sixth BioCreative challenge evaluation workshop, volume 1, pages 141–146, 2017.
  * [20] Antonio Miranda, Farrokh Mehryary, Jouni Luoma, Sampo Pyysalo, Alfonso Valencia, and Martin Krallinger.  Overview of drugprot biocreative vii track: quality evaluation and large scale text mining of drug-gene/protein relations.  In Proceedings of the seventh BioCreative challenge evaluation workshop, 2021.
  * [21] Daojian Zeng, Kang Liu, Siwei Lai, Guangyou Zhou, and Jun Zhao.  Relation classification via convolutional deep neural network.  In Proceedings of COLING 2014, pages 2335–2344, Dublin, Ireland, August 2014.
  * [22] Bernhard E Boser, Isabelle M Guyon, and Vladimir N Vapnik.  A training algorithm for optimal margin classifiers.  In Proceedings of the fifth annual workshop on Computational learning theory, pages 144–152, 1992.
  * [23] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural Comput., 9(8):1735–1780, nov 1997.
  * [24] Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio.  On the properties of neural machine translation: Encoder–decoder approaches.  In Proceedings of SSST-8, pages 103–111, Doha, Qatar, October 2014.
  * [25] Richard Socher, Brody Huval, Christopher D. Manning, and Andrew Y. Ng.  Semantic compositionality through recursive matrix-vector spaces.  In Proceedings of EMNLP 2012, pages 1201–1211, Jeju Island, Korea, July 2012.
  * [26] Shengyu Liu, Buzhou Tang, Qingcai Chen, and Xiaolong Wang.  Drug-drug interaction extraction via convolutional neural networks.  Computational and mathematical methods in medicine, 2016.
  * [27] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Proceedings of NeurIPS 2017, page 6000–6010, Red Hook, NY, USA, 2017. Curran Associates Inc.
  * [28] Masaki Asada, Makoto Miwa, and Yutaka Sasaki.  Extracting drug-drug interactions with attention CNNs.  In BioNLP 2017, pages 9–18, Vancouver, Canada,, August 2017\.
  * [29] Md Faisal Mahbub Chowdhury and Alberto Lavelli.  FBK-irst: A multi-phase kernel based approach for drug-drug interaction detection and classification that exploits linguistic information.  Atlanta, Georgia, USA, 351:53, 2013.
  * [30] Zhehuan Zhao, Zhihao Yang, Ling Luo, Hongfei Lin, and Jian Wang.  Drug drug interaction extraction from biomedical literature using syntax convolutional neural network.  Bioinformatics, 32(22):3444–3453, 2016.
  * [31] Sunil Kumar Sahu and Ashish Anand.  Drug-drug interaction extraction from biomedical texts using long short-term memory network.  Journal of biomedical informatics, 86:15–24, 2018.
  * [32] Linlin Wang, Zhu Cao, Gerard de Melo, and Zhiyuan Liu.  Relation classification via multi-level attention CNNs.  In Proceedings of ACL 2016, pages 1298–1307, 2016.
  * [33] Sangrak Lim, Kyubum Lee, and Jaewoo Kang.  Drug drug interaction extraction from the literature using a recursive neural network.  PloS one, 13(1):e0190926, 2018.
  * [34] Cícero dos Santos, Bing Xiang, and Bowen Zhou.  Classifying relations by ranking with convolutional neural networks.  In Proceedings of ACL-IJCNLP 2015, pages 626–634, Beijing, China, July 2015.
  * [35] Y-Lan Boureau, Jean Ponce, and Yann LeCun.  A theoretical analysis of feature pooling in visual recognition.  In Proceedings of ICML 2020, pages 111–118, 2010.
  * [36] Arthur E Hoerl and Robert W Kennard.  Ridge regression: Biased estimation for nonorthogonal problems.  Technometrics, 12(1):55–67, 1970.
  * [37] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Proceedings of ICLR 2015, 2015.
  * [38] Isabel Segura-Bedmar, Paloma Martínez Fernández, and Daniel Sánchez Cisneros.  The 1st ddiextraction-2011 challenge task: Extraction of drug-drug interactions from biomedical texts.  In CEUR Workshop Proceedings, volume 461, pages 1–9. CEUR-WS.org, 2011.
  * [39] David S Wishart, Yannick D Feunang, An C Guo, Elvis J Lo, Ana Marcu, Jason R Grant, Tanvir Sajed, Daniel Johnson, Carin Li, Zinat Sayeeda, Nazanin Assempour, Ithayavani Iynkkaran, Yifeng Liu, Adam Maciejewski, Nicola Gale, Alex Wilson, Lucy Chin, Ryan Cummings, Diana Le, Allison Pon, Craig Knox, and Michael Wilson.  DrugBank 5.0: a major update to the DrugBank database for 2018.  Nucleic Acids Research, 46(D1):D1074–D1082, 11 2017.
  * [40] Yoon Kim.  Convolutional neural networks for sentence classification.  In Proceedings of EMNLP 2014, pages 1746–1751, Doha, Qatar, October 2014.
  * [41] Chanqin Quan, Lei Hua, Xiao Sun, and Wenjun Bai.  Multichannel convolutional neural network for biological relation extraction.  BioMed Research International, 2016:1850404, Dec 2016.
  * [42] Sun Kim, Haibin Liu, Lana Yeganova, and W John Wilbur.  Extracting drug–drug interactions from literature using a rich feature-based linear kernel approach.  Journal of biomedical informatics, 55:23–30, 2015.
  * [43] Masaki Asada, Makoto Miwa, and Yutaka Sasaki.  Enhancing drug-drug interaction extraction from texts by molecular structure information.  In Proceedings of ACL 2018, pages 680–685, Melbourne, Australia, July 2018.
  * [44] David K Duvenaud, Dougal Maclaurin, Jorge Iparraguirre, Rafael Bombarell, Timothy Hirzel, Alan Aspuru-Guzik, and Ryan P Adams.  Convolutional Networks on Graphs for Learning Molecular Fingerprints.  In Proceedings of NeurIPS 2015, pages 2224–2232, 2015.
  * [45] Yujia Li, Daniel Tarlow, Marc Brockschmidt, and Richard Zemel.  Gated graph sequence neural networks.  In Proceedings of ICLR 2016, 2016.
  * [46] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl.  Neural Message Passing for Quantum Chemistry.  In Proceedings of ICML 2017, pages 1263–1272, 2017.
  * [47] Yoshimasa Tsuruoka, Yuka Tateishi, Jin-Dong Kim, Tomoko Ohta, John McNaught, Sophia Ananiadou, and Jun’ichi Tsujii.  Developing a robust part-of-speech tagger for biomedical text.  In Panhellenic Conference on Informatics, pages 382–392, 2005.
  * [48] David Weininger.  SMILES, a chemical language and information system. 1. Introduction to methodology and encoding rules.  Journal of chemical information and computer sciences, 28(1):31–36, 1988.
  * [49] Greg Landrum.  RDKit: Open-source cheminformatics, 2016.
  * [50] Wei Zheng, Hongfei Lin, Ling Luo, Zhehuan Zhao, Zhengguang Li, Yijia Zhang, Zhihao Yang, and Jian Wang.  An attention-based effective neural model for drug-drug interactions extraction.  BMC Bioinformatics, 18(1):445, Oct 2017.
  * [51] Ronald Aylmer Fisher et al.  The design of experiments.  The design of experiments., 1937.
  * [52] Iinuma Naoki, Masaki Asada, Makoto Miwa, and Yutaka Sasaki.  TTI-COIN at BioCreativeVII Track 1.  In Proceedings of sixth BioCreative Challenge Evaluation Workshop, pages 49–53, Online, November 2021.
  * [53] Masashi Tsubaki, Kentaro Tomii, and Jun Sese.  Compound–protein interaction prediction with end-to-end learning of neural networks for graphs and sequences.  Bioinformatics, 35(2):309–318, 2019.
  * [54] Iz Beltagy, Kyle Lo, and Arman Cohan.  SciBERT: A pretrained language model for scientific text.  In Proceedings of EMNLP-IJCNLP 2019, pages 3615–3620, Hong Kong, China, November 2019.
  * [55] Taku Kudo and John Richardson.  SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.  In Proceedings of EMNLP 2018, pages 66–71, Brussels, Belgium, November 2018.
  * [56] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (GELUs).  CoRR, abs/1606.08415, 2016.
  * [57] Greg Landrum.  RDKit: Open-source cheminformatics software, 2016.
  * [58] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In Proceedings of ICLR 2019, 2019.
  * [59] Manuel Le Gallo, Abu Sebastian, Roland Mathis, Matteo Manica, Heiner Giefers, Tomas Tuma, Costas Bekas, Alessandro Curioni, and Evangelos Eleftheriou.  Mixed-precision in-memory computing.  Nature Electronics, 1(4):246–253, 2018.
  * [60] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  Journal of Machine Learning Research, 15(56):1929–1958, 2014.
  * [61] Yifan Peng, Shankai Yan, and Zhiyong Lu.  Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets.  In Proceedings of BioNLP 2019, pages 58–65, Florence, Italy, August 2019.
  * [62] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [63] Wei Wang, Xi Yang, Canqun Yang, Xiaowei Guo, Xiang Zhang, and Chengkun Wu.  Dependency-based long short term memory network for drug-drug interaction extraction.  BMC Bioinformatics, 18(16):578, Dec 2017.
  * [64] Naoki Iinuma, Makoto Miwa, and Yutaka Sasaki.  Improving supervised drug-protein relation extraction with distantly supervised models.  In Proceedings of BioNLP 2022, pages 161–170, Dublin, Ireland, May 2022.
  * [65] The UniProt Consortium.  UniProt: a worldwide hub of protein knowledge.  Nucleic Acids Research, 47(D1):D506–D515, 11 2018.
  * [66] Thien Huu Nguyen and Ralph Grishman.  Relation extraction: Perspective from convolutional neural networks.  In Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing, pages 39–48, Denver, Colorado, June 2015.
  * [67] Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  Bioinformatics, 36(4):1234–1240, 09 2019.
  * [68] Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel.  A three-way model for collective learning on multi-relational data.  In Proceedings of ICML 2011, page 809–816, Madison, WI, USA, 2011. Omnipress.
  * [69] Antoine Bordes, Jason Weston, Ronan Collobert, and Yoshua Bengio.  Learning structured embeddings of knowledge bases.  In Proceedings of AAAI 2011, page 301–306. AAAI Press, 2011.
  * [70] Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng.  Reasoning with neural tensor networks for knowledge base completion.  Advances in Neural Information Processing Systems, 26:926–934, 2013\.
  * [71] Maximilian Nickel, Lorenzo Rosasco, and Tomaso Poggio.  Holographic embeddings of knowledge graphs.  In Proceedings of AAAI 2016, volume 30, 2016.
  * [72] Duc-Hong Pham and Anh-Cuong Le.  Learning multiple layers of knowledge representation for aspect based sentiment analysis.  Data & Knowledge Engineering, 114:26–39, 2018.
  * [73] Ruobing Xie, Zhiyuan Liu, and Maosong Sun.  Representation learning of knowledge graphs with hierarchical types.  In IJCAI, pages 2965–2971, 2016.
  * [74] Yankai Lin, Zhiyuan Liu, and Maosong Sun.  Knowledge representation learning with entities, attributes and relations.  ethnicity, 1:41–52, 2016.
  * [75] Zizheng Ji, Zhengchao Lei, Tingting Shen, and Jing Zhang.  Joint representations of knowledge graphs and textual information via reference sentences.  IEICE Transactions on Information and Systems, 103(6):1362–1370, 2020.
  * [76] Ni Lao, Tom Mitchell, and William Cohen.  Random walk inference and learning in a large scale knowledge base.  In Proceedings of EMNLP 2011, pages 529–539, 2011.
  * [77] Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu.  Learning entity and relation embeddings for knowledge graph completion.  In Proceedings of AAAI 2015, page 2181–2187. AAAI Press, 2015\.
  * [78] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko.  Translating embeddings for modeling multi-relational data.  In Proceedings of NuerIPS 2013, pages 1–9, 2013.
  * [79] Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling.  Modeling relational data with graph convolutional networks.  In European semantic web conference, pages 593–607. Springer, 2018\.
  * [80] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu.  Heterogeneous graph attention network.  In The World Wide Web Conference, pages 2022–2032, 2019.
  * [81] Xu Han, Zhiyuan Liu, and Maosong Sun.  Joint representation learning of text and knowledge for knowledge graph completion.  arXiv preprint arXiv:1611.04125, 2016.
  * [82] Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, and Michael Gamon.  Representing text for joint embedding of text and knowledge bases.  In Proceedings of EMNLP 2015, pages 1499–1509, 2015.
  * [83] Yashen Wang, Huanhuan Zhang, Ge Shi, Zhirun Liu, and Qiang Zhou.  A model of text-enhanced knowledge graph representation learning with mutual attention.  IEEE Access, 8:52895–52905, 2020.
  * [84] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor.  Freebase: a collaboratively created graph database for structuring human knowledge.  In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247–1250, 2008.
  * [85] Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum.  Yago: a core of semantic knowledge.  In Proceedings of WWW 2017, pages 697–706, 2007.
  * [86] George A Miller.  Wordnet: a lexical database for english.  Communications of the ACM, 38(11):39–41, 1995.
  * [87] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng.  Embedding entities and relations for learning and inference in knowledge bases.  In Proceedings of ICLR 2014, 2014.
  * [88] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In Proceedings of ICLR 2017. OpenReview.net, 2017.
  * [89] Timothy Jewison, Yilu. Su, Fatemeh Miri Disfany, Yongjie Liang, Craig Knox, Adam Maciejewski, Jenna Poelzer, Jessica Huynh, You Zhou, David Arndt, Yannick Djoumbou, Yifeng Liu, Lu Deng, An Chi Guo, Beomsoo Han, Allison Pon, Michael Wilson, Shahrzad Rafatnia, Philip Liu, and David S Wishart.  SMPDB 2.0: big improvements to the Small Molecule Pathway Database.  Nucleic Acids Res, 42(Database issue):D478–484, Jan 2014.
  * [90] C. E. Lipscomb.  Medical Subject Headings (MeSH).  Bulletin of the Medical Library Association, 88(3):265–266, Jul 2000.
  * [91] Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard.  Complex embeddings for simple link prediction.  In International Conference on Machine Learning, pages 2071–2080. PMLR, 2016.
  * [92] Seyed Mehran Kazemi and David Poole.  Simple embedding for link prediction in knowledge graphs.  In Proceedings of NeurIPS 2018, pages 4289–4300, 2018.
  * [93] Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon.  Domain-specific language model pretraining for biomedical natural language processing.  ACM Transactions on Computing for Healthcare (HEALTH), 3(1):1–23, 2021.
  * [94] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035. Curran Associates, Inc., 2019.
  * [95] Da Zheng, Xiang Song, Chao Ma, Zeyuan Tan, Zihao Ye, Jin Dong, Hao Xiong, Zheng Zhang, and George Karypis.  DGL-KE: Training knowledge graph embeddings at scale.  In Proceedings of SIGIR 2020, page 739–748, New York, NY, USA, 2020.
  * [96] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush.  Transformers: State-of-the-art natural language processing.  In Proceedings of EMNLP 2020, pages 38–45, Online, October 2020\.
  * [97] Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu.  ERNIE: Enhanced language representation with informative entities.  In Proceedings of ACL 2019, pages 1441–1451, Florence, Italy, July 2019.
  * [98] Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith.  Knowledge enhanced contextual word representations.  In Proceedings of EMNLP-IJCNLP 2019, pages 43–54, Hong Kong, China, November 2019.
  * [99] Zexuan Zhong and Danqi Chen.  A frustratingly easy approach for entity and relation extraction.  In Proceedings of NAACL 2021, pages 50–61, Online, June 2021.
  * [100] Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski.  Matching the blanks: Distributional similarity for relation learning.  In Proceedings of ACL 2019, pages 2895–2905, Florence, Italy, July 2019.
  * [101] Chaojun Xiao, Yuan Yao, Ruobing Xie, Xu Han, Zhiyuan Liu, Maosong Sun, Fen Lin, and Leyu Lin.  Denoising relation extraction from document-level distant supervision.  In Proceedings of EMNLP 2020, pages 3683–3688, Online, November 2020.
  * [102] Seyone Chithrananda et al.  Chemberta: Large-scale self-supervised pretraining for molecular property prediction.  In Machine Learning for Molecules Workshop at NeurIPS 2020, 2020\.
  * [103] John Duchi, Elad Hazan, and Yoram Singer.  Adaptive subgradient methods for online learning and stochastic optimization.  J. Mach. Learn. Res., 12(null):2121–2159, jul 2011.
  * [104] Sunghwan Kim, Jie Chen, Tiejun Cheng, Asta Gindulyte, Jia He, Siqian He, Qingliang Li, Benjamin A Shoemaker, Paul A Thiessen, Bo Yu, Leonid Zaslavsky, Jian Zhang, and Evan E Bolton.  PubChem in 2021: new data content and improved web interfaces.  Nucleic Acids Research, 49(D1):D1388–D1395, 11 2020.
  * [105] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  RoBERTa: A robustly optimized BERT pretraining approach.  CoRR, abs/1907.11692, 2019.
  * [106] David Ruppert.  Efficient estimations from a slowly convergent robbins-monro process.  Technical report, Cornell University Operations Research and Industrial Engineering, 1988.
  * [107] Boris T Polyak and Anatoli Juditsky.  Acceleration of stochastic approximation by averaging.  SIAM Journal on Control and Optimization, 30(4):838–855, 1992.
  * [108] Long N Phan, James T Anibal, Hieu Tran, Shaurya Chanana, Erol Bahadroglu, Alec Peltekian, and Grégoire Altan-Bonnet.  Scifive: a text-to-text transformer model for biomedical literature.  CoRR, abs/2106.03598, 2021.
  * [109] Dongfang Xu and Timothy Miller.  A simple neural vector space model for medical concept normalization using concept embeddings.  Journal of Biomedical Informatics, 130:104080, 2022.
  * [110] Liang Yao, Chengsheng Mao, and Yuan Luo.  Graph convolutional networks for text classification.  In Proceedings of AAAI’19/IAAI’19/EAAI’19. AAAI Press, 2019.
  * [111] Yufeng Zhang, Xueli Yu, Zeyu Cui, Shu Wu, Zhongzhen Wen, and Liang Wang.  Every document owns its structure: Inductive text classification via graph neural networks.  In Proceedings of ACL 2020, pages 334–339, Online, July 2020.