#  Toyota Technological Institute Ph.D. Thesis
Integrating Heterogeneous Domain Information into Relation Extraction: A Case
Study on Drug-Drug Interaction Extraction

December 2022 Masaki Asada Computational Intelligence Laboratory

###### Abstract

Relation extraction from the literature is a crucial task in natural language
processing. It is very important to extract relationships between named
entities from news articles and academic papers in various fields and to
summarize these relations into structured data. However, the number of digital
documents on the Internet is increasing daily, and it is impossible to perform
relation extraction for all of the texts manually. Therefore, there is a need
to study on automatic relation extraction from the literature using machine
learning.

The development of deep neural networks has improved representation learning
in various domains, including textual, graph structural, and relational triple
representations. This development opened the door to new relation extraction
beyond the traditional text-oriented relation extraction. However, research on
the effectiveness of considering multiple heterogeneous domain information
simultaneously is still under exploration, and if a model can take an
advantage of integrating heterogeneous information, it is expected to exhibit
a significant contribution to many problems in the world.

This thesis works on Drug-Drug Interactions (DDIs) from the literature as a
case study and realizes relation extraction utilizing heterogeneous domain
information. Drugs are related to information about various aspects, including
textual information, information of molecular structures, categorical
information, and related protein information. Therefore, extracting DDIs from
the literature is a suitable target to verify the effectiveness of considering
heterogeneous domain information in the relation extraction task.

First, a deep neural relation extraction model is prepared and its attention
mechanism is analyzed. Next, a method to combine the drug molecular structure
information and drug description information to the input sentence information
is proposed, and the effectiveness of utilizing drug molecular structures and
drug descriptions for the relation extraction task is shown.

Then, in order to further exploit the heterogeneous information, drug-related
items, such as protein entries, medical terms and pathways are collected from
multiple existing databases and a new data set in the form of a knowledge
graph (KG) is constructed. A link prediction task on the constructed data set
is conducted to obtain embedding representations of drugs that contain the
heterogeneous domain information.

Finally, a method that integrates the input sentence information and the
heterogeneous KG information is proposed. The proposed model is trained and
evaluated on a widely used data set, and as a result, it is shown that
utilizing heterogeneous domain information significantly improves the
performance of relation extraction from the literature. Overall, this study
demonstrates the importance of considering heterogeneous domain items in the
information extraction task beyond the text-oriented information extraction.

##### Acknowledgments

First, and most of all, I would like to express my gratitude to my supervisor,
Prof. Yutaka Sasaki. I have been indebted to him for more than five years, and
it is impossible to get to this point without his continual support. He gave
me sincere guidance for my research plan, for writing papers, and for
preparing presentation slides.

I would also like to thank Assoc. Prof. Makoto Miwa, who has given me a lot of
extremely important advice in my research on neural relation extraction. He is
a great motivator in my decision to pursue a Ph.D. degree. During our daily
discussions, he gave me very thoughtful comments that made my Ph.D. research
even greater.

I would like to thank my co-supervisor, Prof. Kazuo Hotate for his great
advice in the introduction part of my research presentation. I would like also
to thank the members of my Ph.D. thesis review committee. I would like to
thank Prof. Yukihiro Motoyama for giving me insightful comments about drugs. I
would like to thank Prof. Norimichi Ukita for the great discussions about deep
learning methods that helps me to improve my Ph.D. thesis. I would like to
thank Prof. Yoshimasa Tsuruoka for his extremely insightful comments on
natural language processing.

Another big thank you goes to all of my colleagues at Toyota Technological
Institute, who have supported me. As a senior member of the Ph.D. program,
Tomoki Tsujimura provided me with a lot of advice on my research. We had a lot
of fun conversations outside of work.

My deepest gratitude goes to my family for their endless love and support
during all these years. I would like to thank my parents for supporting my
decisions with generosity.

###### Contents

  1. 1 Introduction
    1. 1.1 Motivations and Objectives
    2. 1.2 Contributions
    3. 1.3 Thesis Structure
  2. 2 New Approach to Relation Extraction: Unified Use of Heterogeneous Domain Information
    1. 2.1 Relation Extraction: An Overview
      1. 2.1.1 Task Definition
      2. 2.1.2 Data Sets and Corpora
      3. 2.1.3 Evaluation Metrics
      4. 2.1.4 Conventional Approaches Dealing with Relation Extraction from the Literature
    2. 2.2 Integrating Heterogeneous Domain Information into Relation Extraction
  3. 3 Relation Extraction from Texts with Attention CNNs
    1. 3.1 Background
    2. 3.2 Preprocessing
    3. 3.3 Methods
      1. 3.3.1 Base CNN Model
      2. 3.3.2 Embedding Layer
      3. 3.3.3 Convolution Layer
      4. 3.3.4 Pooling Layer
      5. 3.3.5 Prediction Layer
      6. 3.3.6 Attention Mechanism
      7. 3.3.7 Training Method
    4. 3.4 Experimental Settings
      1. 3.4.1 DDI Data Sets
      2. 3.4.2 Initializing Word Embeddings
      3. 3.4.3 Hyper-parameter Tuning
    5. 3.5 Results and Discussions
      1. 3.5.1 Performance Analysis
      2. 3.5.2 Comparison with Existing Models
      3. 3.5.3 Comparison of Attention Mechanisms
      4. 3.5.4 Visual Analysis
    6. 3.6 Summary
  4. 4 Relation Extraction with a Single Kind of Domain Information
    1. 4.1 Background
    2. 4.2 Methods
      1. 4.2.1 Text-based DDI Extraction
      2. 4.2.2 Molecular Structure-based DDI Classification
      3. 4.2.3 DDI Extraction from Texts Using Molecular Structures
    3. 4.3 Experimental Settings
      1. 4.3.1 Data and Task for Molecular Structures
      2. 4.3.2 Training Settings
    4. 4.4 Results and Discussions
    5. 4.5 Summary
  5. 5 Relation Extraction with Multiple Domain Information
    1. 5.1 Background
    2. 5.2 Methods
      1. 5.2.1 Input Sentence Representation
      2. 5.2.2 Drug Description Representation
      3. 5.2.3 Molecular Structure Representation
      4. 5.2.4 DDI Extraction Using Database Information
      5. 5.2.5 Training
      6. 5.2.6 Ensemble
    3. 5.3 Experimental Settings
      1. 5.3.1 DrugBank Preprocessing
      2. 5.3.2 Drug Mention Linking
      3. 5.3.3 Training Settings
    4. 5.4 Results and Discussions
      1. 5.4.1 Pre-training of GNNs and CNNs on DrugBank
      2. 5.4.2 Can DrugBank Information Alone Extract DDIs from Texts?
      3. 5.4.3 Error Analysis
    5. 5.5 BioCreativeVII Track-1 DrugProt
      1. 5.5.1 Introduction
      2. 5.5.2 Task Definition
      3. 5.5.3 Methods
      4. 5.5.4 Experiments
    6. 5.6 Summary
  6. 6 Representing Heterogeneous Knowledge Graph
    1. 6.1 Background: An Overview of Knowledge Graph Representation
    2. 6.2 Heterogeneous Pharmaceutical Knowledge Graph with Textual Information
      1. 6.2.1 Constructing Heterogeneous Pharmaceutical Knowledge Graph
      2. 6.2.2 Textual Information of Knowledge Graph
    3. 6.3 Learning Knowledge Graph Embeddings
      1. 6.3.1 Knowledge Graph Definition
      2. 6.3.2 Scoring Functions
      3. 6.3.3 Negative Sampling and Loss Functions
    4. 6.4 Methods
      1. 6.4.1 Initializing Node Embeddings
      2. 6.4.2 Aligning Entity Embeddings and Textual Embeddings
      3. 6.4.3 Augmenting KG Embeddings
    5. 6.5 Experimental Settings
      1. 6.5.1 Constructing Heterogeneous KG with Textual Information
      2. 6.5.2 Encoding Text Information
      3. 6.5.3 KG Embedding Training Settings
      4. 6.5.4 Task Setting
      5. 6.5.5 Hyper-parameter Settings
      6. 6.5.6 Implementation Details
    6. 6.6 Results and Discussions
      1. 6.6.1 Analysis of the Data Ombalance of the Constructed KG
      2. 6.6.2 Ablation Study of Augmentation Method
      3. 6.6.3 Effect of Node Type Filtering
      4. 6.6.4 Case Study
    7. 6.7 Summary
  7. 7 Integrating Heterogeneous Domain Information for Relation Extraction
    1. 7.1 Background
    2. 7.2 Method
      1. 7.2.1 Obtaining Heterogeneous KG Embeddings
      2. 7.2.2 DDI Extraction from Texts with Heterogeneous KG Embeddings
    3. 7.3 Experimental Settings
      1. 7.3.1 Mention Linking
      2. 7.3.2 Link Prediction Settings
      3. 7.3.3 DDI Extraction Settings
    4. 7.4 Results and Discussions
      1. 7.4.1 Selecting Score Functions
      2. 7.4.2 Ablation Study on Model Architecture
      3. 7.4.3 Ablation Study on Heterogeneous KG Node Types
      4. 7.4.4 Verification of DDI Label Leakage from KGs
      5. 7.4.5 Learning Curve
      6. 7.4.6 Analysis of Prediction Results
    5. 7.5 Summary
  8. 8 Conclusions
    1. 8.1 Summary
    2. 8.2 Future Work
      1. 8.2.1 Employing the Deep Neural Entity Linking Method
      2. 8.2.2 Joint Learning of BERT and KG Embeddings
      3. 8.2.3 End-to-End DDI Extraction
      4. 8.2.4 Extension to Other Tasks

###### List of Tables

  1. 2.1 Contingency table of true positives (TP), false positives (FP), true negatives (TN) and false negative (FN) for binary DDI classification
  2. 3.1 An example of the preprocessed input sentence
  3. 3.2 Statistics for the DDIExtraction-2013 shared task data set
  4. 3.3 Hyper-paramters
  5. 3.4 Statistics of the development data set
  6. 3.5 Performance of softmax/ranking CNN models with and without the attention mechanism
  7. 3.6 Comparison with existing models
  8. 3.7 Comparison of attention mechanisms on CNN models with ranking objective function
  9. 4.1 Evaluation on DDI extraction from texts
  10. 4.2 F-scores of each DDI type
  11. 4.3 Accuracy of binary classification on DrugBank pairs
  12. 4.4 Classification of DDIs in texts by molecular structure-based DDI classification model
  13. 5.1 Hyper-parameters for CNNs
  14. 5.2 Hyper-parameters for GNNs
  15. 5.3 Evaluation on DDI extraction from texts on the test set
  16. 5.4 Evaluation on DDI extraction from texts on the development set
  17. 5.5 Performance on individual DDI types in F-scores
  18. 5.6 Individual F-scores on 5-fold cross-validated training data set
  19. 5.7 Comparisons of F-scores on different parts of the test set
  20. 5.8 Accuracy of binary classification on the DrugBank pairs
  21. 5.9 Evaluation on DDI extraction from texts with or without pre-training of GNNs for the molecular structure and CNNs for the description
  22. 5.10 Micro-averaged F-scores on DrugProt development set and test set
  23. 5.11 F-scores per class on DrugProt development set
  24. 6.1 Statistics of heterogeneous pharmaceutical KG entities
  25. 6.2 Statistics of heterogeneous pharmaceutical KG edges for each relation type
  26. 6.3 The percentage of nodes that have each type of text
  27. 6.4 Comparison of MRR performance for each method
  28. 6.5 Summary of the best settings for each relation
  29. 6.6 Ablation study of text information on Augmentation method (ComplEx score function)
  30. 6.7 Comparison of averaged MRR performance for w/ (with) and w/o (without) entity type filtering
  31. 6.8 The content of the text in the examples where the difference between the rank of textual model and the rank of non-textual model is largest for each relation type
  32. 7.1 Hyper-parameters for link prediction and DDI extraction model
  33. 7.2 The comparison of link prediction performance on heterogeneous KG
  34. 7.3 The comparison of DDI extraction performance on DDIExtraction-2013 test data set
  35. 7.4 The comparison of F-scores for individual DDI types and macro-averaged F-score on DDIExtraction-2013 test data set
  36. 7.5 The comparison of DDI extraction performances with different score functions for training KG embeddings
  37. 7.6 The ablation study on model architecture
  38. 7.7 The ablation study on node types
  39. 7.8 The DDI extraction performance for drug pairs that are included in the constructed heterogeneous KG and drug pairs that are not included in KG
  40. 7.9 Case studies of the proposed model

###### List of Figures

  1. 1.1 Yearly number of papers with the MeSH category of ”Chemicals and Drugs” in PubMed
  2. 1.2 Heterogeneous domain information about a mention in the literature
  3. 2.1 The visualization of some instances in the DDIExtraction-2013 data set
  4. 2.2 The outline of integrating heterogeneous domain information into the relation extraction
  5. 3.1 The overview of CNN with attention mechanism
  6. 3.2 The visualization of attention weights
  7. 4.1 Overview of the CNN-based DDI extraction model that use molecular structure information
  8. 5.1 Overview of the DDI extraction model with drug descriptions and drug molecular structures
  9. 5.2 Illustration of molecular fingerprints. This figure shows the extraction of several fingerprint subgraphs from a molecular structure when radius is 2.
  10. 5.3 Linking between mentions and DrugBank entry
  11. 5.4 F-scores for different sentence lengths on the 5-fold cross-validated training data set
  12. 5.5 The model with description and structure information
  13. 6.1 Illustration of the Heterogeneous Pharmaceutical Knowledge Graph
  14. 6.2 Overview of methods: (A) Initializing node embeddings (Initialization), (B) Aligning entity embeddings and textual embeddings (Alignment), and (C) Augmenting KG embeddings (Augmentation)
  15. 6.3 The distribution of the frequency of nodes in the train data set
  16. 7.1 Overview of the levitated marker
  17. 7.2 Heterogeneous KG with additional drug molecular structure information
  18. 7.3 The DDI extraction model which utilize heterogeneous information of drugs
  19. 7.4 Learning curve of baseline model and the proposed model on 5-fold cross-validation data sets
  20. 7.5 The confusion matrices (left: baseline, right: proposed method)

##  Introduction

###  1.1 Motivations and Objectives

Relation extraction from the literature is a crucial task in natural language
processing. It is very important to extract relationships between named
entities from the literature and summarize these relations into structured
data. However, the number of digital documents on the Internet is increasing
daily, and it is impossible to conduct relation extraction for all of the
texts manually. Figure 1.1 shows the yearly number of papers in the medicine
and biomedical journal literature search engine PubMed. The number of papers
is generally increasing year by year, and there is a need to study on
automatic relation extraction from the literature using machine learning.

It was considered very difficult to utilize different kinds of information at
the same time. However, with the advent of deep learning [1], it has become
possible to represent each attribute as fixed-size numerical vectors using
methods such as word2vec [2] and node2vec [3]. As for text representation, the
advent of Bidirectional Encoder Representations from Transformer (BERT) [4]
has made it more powerful. Although these developments have opened the door to
new relation extraction beyond the traditional text-oriented relation
extraction, the research on simultaneously considering multiple heterogeneous
domain information is still under exploration. The author believes that the
new techniques will provide clues for the integrated use of heterogeneous
domain information. This thesis elaborates to establish deep learning-based
methods that can comprehensively handle heterogeneous information relevant to
mentions in relation extraction from the literature.

Figure 1.1: Yearly number of papers with the MeSH category of “Chemicals and
Drugs” in the medicine and biomedical journal literature search engine PubMed.
The number of papers is generally increasing year by year.

Similar to this motivation is multimodal learning [5] and multi-view learning
[6]. Multimodal learning is a method that simultaneously considers information
in two or three different modalities, such as text and image, or text and
audio. Multi-view learning is an emerging direction in machine learning which
considers learning with multiple views to improve the generalization
performance. Multi-view learning is also known as data fusion or data
integration from multiple feature sets. Figure 1.2 shows an example of the
heterogeneous domain information of an entity in the literature. The examples
of drug entities in the literature show multiple kinds of domain information
related to drugs. Drugs contain various kinds of information, such as
molecular structures, descriptions, and categorical hierarchies. This thesis
further develops multimodal learning and incorporates multi-view learning to
integrate a lot of domain information in the unified vector space, and
utilizes them to conduct the relation extraction task.

One thing which must be clear here is that the relation extraction task
consists of two parts: (i) recognition of named entities in input sentences
and (ii) extraction of the relation between identified named mentions, and
this thesis focuses only on the relation extraction part. This is because many
heterogeneous factors need to be considered in order to extract relations from
the literature.

Extracting relations between drugs from the literature is an appropriate
problem in verifying the effectiveness in considering heterogeneous domain
information. Medicines help us feel better and stay healthy. But sometimes
drug interactions can cause problems. Drug interactions can be classified into
the following three broad categories [7]:

Drug-drug interaction

A reaction between two (or more) drugs.

Drug-food interaction

A reaction between a drug and a food or beverage.

Drug-condition interaction

A reaction that occurs when taking a drug while having a certain medical
condition.

This thesis chooses the drug-drug interaction (DDI) extraction task as a case
study. When considering DDIs, it is necessary to refer to the descriptions in
drug articles for achieving evidence-based medicine [8]; therefore it is
important to extract DDIs from the literature. Hence, the author emphasizes
that the goal is to determine whether a sentence in the articles represents
that two drugs interact or not. It is not a goal to determine whether the two
drugs actually interact. For instance, when a sentence just mentions a pair of
drug entities without justifying their interaction, the model cannot extract
the DDI of the two drugs in the context of the sentence.

Figure 1.2: Heterogeneous domain information about a mention in the literature

###  1.2 Contributions

The contributions of this thesis are shown as follows:

  * •

To utilize heterogeneous domain information in relation extraction.

  * •

To construct a new benchmark data set in the form of knowledge graphs to use
heterogeneous information.

  * •

To propose a neural architecture that integrally uses sentence information and
domain information.

  * •

To propose methods to utilize the heterogeneous domain representation of drugs
trained from the constructed data set to extract DDIs from texts.

  * •

To evaluate the advantage of the proposed method using drug-drug interaction
extraction as a case study.

  * •

To show that several kinds of heterogeneous drug information are complementary
and their effective combinations can largely improve the DDI extraction
performance.

  * •

To show that the proposed neural architecture can effectively utilize
heterogeneous domain information for DDI extraction.

###  1.3 Thesis Structure

The remainder of this thesis is organized as follows.

Chapter 2 provides the novelties of the thesis based on an overview of
relation extraction from the literature. First, how the relation extraction
corpus was created and the relation extraction task settings, evaluation
metrics and data set statistics are described. Then, the history of how
machine learning-based automatic relation extraction from the literature is
described. Finally, a new approach that integrates heterogeneous domain
information into relation extraction tasks is introduced.

Chapter 3 provides a base neural model for relation extraction as a
preliminary, and reviews the effectiveness of the method of employing an
attention mechanism in the CNN-based model on relation extraction tasks. This
model is a text-oriented relation extraction model and is not capable of
domain information.

Chapter 4 reviews the study on relation extraction from texts considering
molecular structures of drugs in addition to the context of input sentences. A
baseline model with the word2vec algorithm and Convolutional Neural Networks
(CNNs) is addressed in this section. Since this model cannot deal with
heterogeneous domain information, the following chapters develop relation
extraction models that can consider heterogeneous domain information.

Chapter 5 proposes a relation extraction model that can consider two types of
domain information: the molecular structure information of the drug and the
description information of the drug in a case study. In addition, the relation
extraction model is applied to the drug-protein interaction (DPI) extraction
task which was held in the BioCreative VII shared task. This section and later
sections employ Bidirectional Encoder Representations from Transformer (BERT)
[4] as the baseline model. This chapter includes work from the published paper
[9].

Chapter 6 builds a novel data set in the form of a Knowledge Graph (KG) to
consider further heterogeneous information about drugs. The link prediction
task is conducted on the created heterogeneous KG and the embedding
representations of the drugs are obtained. This chapter is based on the
published work [10].

Chapter 7 utilizes the heterogeneous KG embeddings of the drugs for DDI
extraction task. The effectiveness of combining the input sentence
representation and KG representation is described. This chapter is based on
[11].

Chapter 8 summarizes the findings in this thesis and concludes the thesis.
Finally, the further developments of the model and the future plan are
discussed.

##  New Approach to Relation Extraction: Unified Use of Heterogeneous Domain
Information

This chapter firstly provides an overview of relation extraction tasks. Then a
new approach that integrates heterogeneous domain information into relation
extraction tasks is introduced.

###  2.1 Relation Extraction: An Overview

####  2.1.1 Task Definition

With the expanding use of the Internet, a large amount of the literature
becomes available every day in the form of news articles, research
publications, question answering forums, and social media. It is important to
develop techniques for extracting information automatically from these
documents and summarizing relations into structured data. The entity like the
person and the organization is the most basic unit of information. Occurrences
of entities in a sentence are often linked through well-defined relations;
e.g., occurrences of person and organization in a sentence may be linked
through relations such as employed_at. The task of relation extraction is to
identify such relations automatically. [12]

Figure 2.1 shows some examples of annotated sentences. These sentences are
included in the abstract of the PubMed article. The first sentence indicates
that there are three entities in the sentence, and the entity S-ketamine and
the entity ticlopidine have relation Effect. The aim of the relation
extraction task is to classify the given all possible entity pairs into well-
defined relations.

Figure 2.1: The visualization of some instances in the DDIExtraction-2013 data
set

####  2.1.2 Data Sets and Corpora

The construction of a common data set is essential for the development of
research of automatic relation extraction from the literature and several data
sets have been created to assist the development of relation extraction
methods. These data sets were constructed by trusted annotators who are
experts in their fields.

For the relation extraction task in the general field, Automatic Content
Extraction (ACE) [13] data set which contains named entities, relations, and
events in various languages, mostly from news articles was constructed.
Another widely used data set is CoNLL04 [14], which also contains entities and
relations for the general field. SemEval-2010 task 8 [15] data set was created
a few years later and used widely to develop relation extraction models.

For the biomedical field, protein-protein interaction was the first target for
relation extraction task and several protein-protein interaction extraction
data sets such as AIMed [16] and BioInfer [17] data sets were created. Drug-
drug interaction (DDI) extraction has also been studied. DDIExtraction-2013
[18] data set was constructed and it is used as a standard benchmark data set.
BioCreative VI ChemProt [19] data set targeted Chemical-Protein relations and
is a large relation extraction data set with multiple fine-grained relations.
Later, BioCreative VII DrugProt [20] task data set was constructed, which
contains 13 types of drug-protein interactions.

This thesis focuses on the DDIExtraction-2013 task and investigates the
effectiveness of heterogeneous domain information about drugs.

####  2.1.3 Evaluation Metrics

Evaluation is relation-oriented and based on the standard Precision, Recall,
and micro-averaged F-score metrics. Precision is also called Positive
Predictive Value (PPV) and Recall is also called Sensitivity.
$TP~{}(True~{}Positives)$ correspond to the number of pairs correctly
identified by the model as having a relation, $TN~{}(True~{}Negatives)$
correspond to the total number of pairs correctly identified as not having a
relation. $FP~{}(False~{}Positives)$ correspond to the total number of missed
pairs which have a relation and $FN~{}(False~{}Negatives)$ correspond to the
total number of missed pairs which do not have a relation. Table 2.1 shows the
contingency table of TP, FP, TN and FN for a binary classification task.

| Ground Truth
---|---
|  | Relation | No Relation
Prediction | Relation | TP | FP
No Relation | FN | TN
Table 2.1: Contingency table of true positives (TP), false positives (FP),
true negatives (TN) and false negative (FN) for binary DDI classification

Using these statistics, The evaluation metrics Precision (P), Recall (R) and
F1-score (F1) are calculated as follows:

$P=\frac{TP}{TP+FP}$ (2.1) $R=\frac{TP}{TP+FN}$ (2.2) $F1=\frac{2PR}{P+R}$
(2.3)

The above described statistics and metrics are typically used in the case of
binary classification. However, the DDIExtraction-2013 data set has four types
of positive labels. When the F-score is used as an evaluation metric for
multi-class classification, there are two types of F-scores, micro-averaged
and macro-averaged F-scores. While micro-averaged F-score is calculated by
constructing a global contingency table and then calculating precision and
recall, macro-averaged F-score is calculated by first calculating precision
and recall for each type and then taking the average of these results. The
micro- and macro-averaged metrics for Precision, Recall and F-score are
described in the following equations, where $c$ indicates the interaction
category.

$P_{micro}=\frac{\sum_{c}TP_{c}}{\sum_{c}TP_{c}+\sum_{c}FP_{c}},~{}~{}~{}~{}~{}R_{micro}=\frac{\sum_{c}TP_{c}}{\sum_{c}TP_{c}+\sum_{c}FN_{c}}$
(2.4) $F1_{micro}=\frac{2P_{micro}R_{micro}}{P_{micro}+R_{micro}}$ (2.5)
$P_{macro}=\frac{1}{|c|}\sum_{c}P_{c},~{}~{}~{}~{}~{}R_{macro}=\frac{1}{|c|}\sum_{c}R_{c}$
(2.6) $F1_{macro}=\frac{2P_{macro}R_{macro}}{P_{macro}+R_{macro}}$ (2.7)

In this task, the micro-averaged F1-score is used as the main evaluation
metric.

####  2.1.4 Conventional Approaches Dealing with Relation Extraction from the
Literature

##### CNN-based Relation Extraction

The first application of deep Convolutional Neural Networks (CNNs) to the
relation extraction task was the CNNs model proposed by Zeng et al. [21]. CNN
models take a candidate relation instance that is represented by the word
embeddings and position embeddings as input. Then, the input is fed to the
convolutional layer in order to extract features with filters of different
sizes. Subsequently, the pooling layer performs down-sampling on the feature
maps, which provides two advantages: firstly, it can identify the most
relevant and essential local features; and secondly, it reduces the
computational complexity which the reduced resolution. Finally, for
classifying the relation type, the feature vector generated by pooling is fed
into a fully connected softmax layer. The experiments showed that this CNN-
based method can capture the semantic information and relative position of
words better than conventional Support Vector Machines (SVMs) [22] methods.

##### RNNs for Relation Extraction

Recurrent Neural Networks (RNNs) mainly address sequential data. Each output
depends on the previous values in this structure, which is similar to the
human memory mechanism. Generally, these networks are composed of an input
layer, a hidden layer that is connected to itself, and an output layer. In the
standard formulation, RNNs suffer from exploding and vanishing gradient
problems. To address these problems, Long Short-Term Memory (LSTM) [23] units
and Gated Recurrent Units (GRU) [24] networks model the hidden layer with a
memory cell that controls the extent of what is to be forgotten (and to be
incorporated) given the input, the previous state, and the current state.
These networks can explicitly learn when to forget information and when to
store it, which is fit for an input of arbitrary length. Undoubtedly, these
variants are efficient in the relation extraction task. As for RNN-based
approaches, Socher et al. [25] presented a novel relation classification model
that learns vectors in the syntactic tree path that connects two nominals to
determine their semantic relationship.

##### Transformers and BERT Family for Relation Extraction

Many neural network-based methods have been proposed since Liu et al. [26]
first tacked the DDI extraction task with a neural network-based method. RNNs
and CNNs have been widely used for NLP tasks, but both methods have the
problem that it is difficult to capture relationships between distant words in
a sentence. Transformer [27], a model that utilizes the attention mechanism,
has been newly devised and has been believed to solve this problem. The
attention is a mechanism for calculating the importance of each word in a
sentence. Given a single input sentence, a contextualized word representation
can be obtained by a self-attention mechanism that calculates the importance
of each word in the sentence. While CNNs are unable to consider context
outside the window size and RNNs are thought to be difficult to capture
relationships between distant words, Transformers are thought to overcome
these shortcomings.

Bidirectional Encoder Representation from Transformers (BERT) [4] is a
Transformer model that is pre-trained on a large unlabeled corpus. The
approach of fine-tuning the pre-trained BERT shows high performance on various
downstream tasks, including the relation extraction task.

###  2.2 Integrating Heterogeneous Domain Information into Relation Extraction

This section outlines the novel approach to integrating heterogeneous domain
information into the relation extraction task. This thesis develops in three
steps.

##### Using individual domain information

Figure 2.2 (1) shows the first step of the proposed method. Here, the relation
extraction that utilizes the drug-related information from external databases
are addressed. The molecular structures and descriptions of drugs are
represented as vectors separately by a neural network model. Details based on
case studies are provided in Chapter 4 and Chapter 5.

Figure 2.2: The outline of integrating heterogeneous domain information into
the relation extraction

##### Integrating Heterogeneous domain information

Figure 2.2 (2) shows the second step of the development. Heterogeneous drug-
related knowledge, including the molecular structures, is represented in a
unified vector space. Diverse information is integrated and the vectors that
aggregate heterogeneous domain information are obtained.

##### Combining heterogeneous domain information and input sentences

Figure 2.2 (3) displays the main novelty of this thesis. In this figure, the
vectors of aggregated heterogeneous domain information are effectively
utilized for relation extraction from the literature. The heterogeneous domain
information is represented in a unified vector space. This approach enables to
integrate heterogeneous knowledge in relation extraction.

##  Relation Extraction from Texts with Attention CNNs

As a preliminary, this chapter introduces a baseline relation extraction model
with an attention mechanism for a CNN-based relation extraction model [28] and
reviews the usefulness of the attention mechanism. This verification is the
preliminary experiment to a method that utilizes heterogeneous information.
The technique of preprocessing data set and detailed statistics of the
DDIExtraction-2013 article, which are common in subsequent chapters in this
thesis, are discussed.

The overview of this chapter is as follows:

  * •

This chapter provides an attention mechanism that can boost the performance on
CNN-based DDI extraction.

  * •

The DDI extraction model with the attention mechanism achieves the performance
with an F-score of 69.12%, which is competitive with other state-of-the-art
DDI extraction models when the performance without negative instance filtering
[29] is compared.

###  3.1 Background

For the DDI extraction, deep neural network-based methods have recently drawn
a considerable attention [26, 30, 31]. Deep neural networks have been widely
used in the NLP field. They show high performance on several NLP tasks without
requiring manual feature engineering. CNNs and RNNs are often employed for
network architectures. Among these, CNNs have the advantage that they can be
easily parallelized and the calculation is thus fast with recent Graphical
Processing Units (GPUs).

Liu et al. [26] showed that a CNN-based model can achieve a high accuracy on
the task of DDI extraction. Sahu et al. [31] proposed an RNN-based model with
the attention mechanism to tackle the DDI extraction task and show state-of-
the-art performance. The integration of an attention mechanism into a CNN-
based relation extraction is proposed by Wang et al. [32]. This is applied to
a general domain relation extraction task SemEval-2010 Task 8 [15]. Their
model showed state-of-the-art performance on the task. CNNs with attention
mechanisms, however, are not evaluated on the task of DDI extraction.

This chapter reviews a attention mechanism that is integrated into a CNN-based
DDI extraction model). The attention mechanism extends the attention mechanism
by [32] in that it deals with anonymized entities separately from other words
and incorporates a smoothing parameter. A CNN-based relation extraction model
is implemented and the mechanism into the model is integrated. The model is
evaluated on the DDIExtraction-2013 data set [18].

###  3.2 Preprocessing

When three or more drug mentions appear in an input sentence, the sentence is
duplicated for each drug mention pair. Specifically, if an input sentence
contains $n$ drug mentions, $\binom{n}{2}$ input sentences with different drug
mention pairs are prepared.

According to the settings of many existing methods [26, 31, 33], before
tokenizing an input sentence, the mentions of the target drugs in the pair are
replaced with DRUG1 and DRUG2 according to their order of appearance. The
other mentions of drugs are replaced with DRUGOTHER.

Table 3.1 shows an example of preprocessing when the input sentence Exposure
to oral S-ketamine is unaffected by itraconazole but greatly increased by
ticlopidine is given with a target entity pair. By performing preprocessing,
it is possible to prevent the DDI extraction model to be specialized for the
surface forms of the drugs in a training data set and to perform DDI
extraction using the information of the whole context.

Entity1 | Entity2 |  Preprocessed input sentence
---|---|---
S-ketamine | itraconazole |  Exposure to oral DRUG1 is unaffected by DRUG2 but greatly increased by DRUGOTHER.
S-ketamine | ticlopidine |  Exposure to oral DRUG1 is unaffected by DRUGOTHER but greatly increased by DRUG2.
itraconazole | ticlopidine |  Exposure to oral DRUGOTHER is unaffected by DRUG1 but greatly increased by DRUG2.
Table 3.1: An example of preprocessing on the sentence Exposure to oral
S-ketamine is unaffected by itraconazole but greatly increased by ticlopidine
for each target pair.

###  3.3 Methods

An attention mechanism for a CNN-based DDI extraction model is described. The
overview of the DDI extraction model is illustrated in Figure 3.1. The model
extracts interactions from sentences with drugs are given. This section first
presents preprocessing of input sentences, then introduces the base CNN model
and explains the attention mechanism. Finally, the training method is
described.

Figure 3.1: The overview of CNN with attention mechanism

####  3.3.1 Base CNN Model

The CNN model for extracting DDIs is based on Zeng et al. [21]. In addition to
the original objective function, the ranking-based objective function by dos-
Santos et al. [34] is employed. The model consists of four layers: embedding,
convolution, pooling, and prediction layers. The CNN model is shown at the
lower half of Figure 3.1.

####  3.3.2 Embedding Layer

In the embedding layer, each word in the input sentence is mapped to a real-
valued vector representation using an embedding matrix that is initialized
with pre-trained embeddings. Given an input sentence $S=(w_{1},\cdots,w_{n})$
with drug entities $e_{1}$ and $e_{2}$, each word $w_{i}$ is first converted
into a real-valued vector $\bm{w}^{w}_{i}$ by an embedding matrix
$\bm{W}^{emb}\in\mathbb{R}^{d_{w}{\times}|V|}$ as follows:

$\displaystyle\bm{w}^{w}_{i}=\bm{W}^{emb}\bm{v}^{w}_{i},$ (3.1)

where $d_{w}$ is the number of dimensions of the word embeddings, $V$ is the
vocabulary in the training data set and the pre-trained word embeddings, and
$\bm{v}^{w}_{i}$ is a one-hot vector that represents the index of word
embedding in $\bm{W}^{emb}$. $\bm{v}^{w}_{i}$ thus extracts the corresponding
word embedding from $\bm{W}^{emb}$. The word embedding matrix $\bm{W}^{emb}$
is fine-tuned during training.

$d_{wp}$-dimensional word position embeddings $\bm{w}^{p}_{i,1}$ and
$\bm{w}^{p}_{i,2}$ that correspond to the relative positions from first and
second target entities are prepared, respectively. The word embedding
$\bm{w}^{w}_{i}$ and these word position embeddings $\bm{w}^{p}_{i,1}$ and
$\bm{w}^{p}_{i,2}$ as in the following Equation (3.2) are concatenated, and
the resulting vector is used as the input to the subsequent convolution layer:

$\displaystyle\bm{w}_{i}=[\bm{w}^{w}_{i};\bm{w}^{p}_{i,1};\bm{w}^{p}_{i,2}].$
(3.2)

####  3.3.3 Convolution Layer

A weight tensor for convolution is defined as
$\bm{W}_{k}^{conv}{\in}\mathbb{R}^{d_{c}\times(d_{w}+2d_{wp})\times k}$ and
represents the $j$-th column of $\bm{W}_{k}^{conv}$ is defined as
$\bm{W}^{conv}_{k,j}{\in}\mathbb{R}^{(d_{w}+2d_{wp})\times k}$. Here, $d_{c}$
denotes the number of filters for each window size, $k$ is a window size, and
$K$ is a set of the window sizes of the filters. $\bm{z}_{i,k}$ that is
concatenated $k$ word embeddings is also introduced:

$\bm{z}_{i,k}=[\bm{w}^{\mathrm{T}}_{\lfloor{i-(k-1)/2}\rfloor};\ldots;\bm{w}^{\mathrm{T}}_{\lfloor{i-(k+1)/2}\rfloor}]^{\mathrm{T}}.$
(3.3)

The convolution to the embedding matrix is applied as follows:

$m_{i,j,k}=f(\bm{W}^{conv}_{k,j}\odot\bm{z}_{i,k}+b),$ (3.4)

where $\odot$ is an element-wise product, $b$ is the bias term, and $f$ is the
ReLU function defined as:

$\displaystyle f(x)=\begin{cases}x,&\mbox{if}\ x>0\\\
0,&\mbox{otherwise}.\end{cases}$ (3.5)

####  3.3.4 Pooling Layer

The max pooling [35] is employed to convert the output of each filter in the
convolution layer into a fixed-size vector as follows:

$\bm{c}_{k}=[c_{1,k},\cdots,c_{d_{c},k}],\ c_{j,k}=\max_{i}m_{i,j,k}.$ (3.6)

Then the $d_{p}$-dimensional output of this pooling layer is obtained, where
$d_{p}$ equals to $d_{c}{\times}|K|$, by concatenating the obtained outputs
$\bm{c}_{k}$ for all the window sizes $k_{1},\cdots,k_{K}(\in K$):

$\displaystyle\bm{c}=[\bm{c}_{k_{1}};\ldots;\bm{c}_{k_{i}};\ldots;\bm{c}_{k_{K}}].$
(3.7)

####  3.3.5 Prediction Layer

The relation types are predicted using the output of the pooling layer. First
$\bm{c}$ is converted into scores using a weight matrix
$\bm{W}^{pred}\in\mathbb{R}^{o{\times}d_{p}}$:

$\bm{s}=\bm{W}^{pred}\bm{c},$ (3.8)

where $o$ is the total number of relationships to be classified and
$\bm{s}=[s_{1},\cdots,s_{o}]$. Then the following two different objective
functions are employed for prediction.

##### Softmax

$\bm{s}$ is converted into the probability of possible relations $\bm{p}$ by a
softmax function:

$\bm{p}=[p_{1},\cdots,p_{o}],\
p_{j}=\frac{\exp{(s_{j})}}{\sum^{o}_{l=1}\exp{(s_{l})}}.$ (3.9)

The cross-entropy loss function $L_{CE}$ is defined as in the Equation (3.10)
when the gold type distribution $\bm{y}$ is given. $\bm{y}$ is a one-hot
vector where the probability of the gold label is 1 and the others are 0.

$L_{CE}=-\sum\bm{y}\log{\bm{p}}$ (3.10)

##### Ranking

The ranking-based objective function is employed following [34]. Using the
scores $\bm{s}$ in Equation (3.8), the loss is calculated as follows:

$\displaystyle L_{ranking}=\log(1+\exp(\gamma(m^{+}-s_{y}))$
$\displaystyle+\log(1+\exp(\gamma(m^{-}+s_{c})),$ (3.11)

where $m^{+}$ and $m^{-}$ are margins, $\gamma$ is a scaling factor, $y$ is a
gold label, and $c$ $(\neq y)$ is a negative label with the highest score in
$\bm{s}$. $\gamma$ to 2, $m^{+}$ is set to 2.5 and $m^{-}$ is set to 0.5
following [34].

####  3.3.6 Attention Mechanism

The attention mechanism is based on the input attention by Wang et al.
[32]111The attention-based pooling in [32] is not incorporated. This is left
for future work.. The attention mechanism is different from the base one in
which separate attentions are prepared for entities and a bias term is
incorporated to adjust the smoothness of attentions. The attention mechanism
is illustrated at the upper half of Figure 3.1.

The word indexes of the first and second target drug entities in the sentence
are defined as $e_{1}$ and $e_{2}$, respectively. The set of indices are
denoted as $E=\\{e_{1},e_{2}\\}$ and the indexes of the entities as
$j\in\\{1,2\\}$. The attentions are calculated using:

$\displaystyle\beta_{i,j}$ $\displaystyle=$
$\displaystyle\bm{w}_{e_{j}}\cdot\bm{w}_{i}$ (3.12)
$\displaystyle\alpha_{i,j}$ $\displaystyle=$
$\displaystyle\begin{cases}\frac{\exp{(\beta_{i,j})}}{\sum_{1\leq l\leq
n,l\notin E}\exp{(\beta_{l,j})}},&\mbox{if}\ i\notin E\\\
a_{drug},&\mbox{otherwise}\end{cases}$ (3.13) $\displaystyle\alpha_{i}$
$\displaystyle=$
$\displaystyle\frac{\alpha_{i,1}+\alpha_{i,2}}{2}+b_{\alpha}.$ (3.14)

Here, $a_{drug}$ is an attention parameter for entities and $b_{\alpha}$ is
the bias term. $a_{drug}$ and $b_{\alpha}$ are tuned during training. If $E$
is set to empty and $b_{\alpha}$ is set to zero, the attention will be the
same as one by Wang et al. [32]. The attentions $\alpha_{i}$ are incorporated
into the CNN model by replacing Equation (3.4) with the following equation:

$\displaystyle m_{i,j,k}=f(\bm{W}^{conv}_{j}\odot\bm{z}_{i,k}\alpha_{i}+b).$
(3.15)

####  3.3.7 Training Method

L2 regularization [36] is used to avoid over-fitting. The following objective
functions $L^{\prime}_{*}$ ($L^{\prime}_{softmax}$ or $L^{\prime}_{ranking}$)
is used by incorporating the L2 regularization on weights to Equation (5.11).

$\displaystyle
L^{\prime}_{*}=L_{*}+\lambda(||\bm{W}^{emb}||^{2}_{F}+||\bm{W}^{conv}||^{2}_{F}$
(3.16) $\displaystyle+||\bm{W}^{pred}||^{2}_{F})$

Here, $\lambda$ is a regularization parameter and $||\cdot||_{F}$ denotes the
Frobenius norm. All the parameters including the weights $\bm{W}^{emb}$,
$\bm{W}^{conv}$, and $\bm{W}^{pred}$, biases $b$ and $b_{\alpha}$, and the
attention parameter $a_{drug}$ are updated to minimize $L^{\prime}_{*}$. The
adaptive moment estimation (Adam) [37] is used for the optimizer. Training
data set is randomly shuffled and divided into mini-batch samples in each
epoch.

###  3.4 Experimental Settings

####  3.4.1 DDI Data Sets

The DDIExtraction-2011 [38] workshop (First Challenge Task on Drug-Drug
Interaction Extraction) focuses on the extraction of DDIs from biomedical
texts and aims to promote the development of text mining and information
extraction systems applied to the pharmacological domain in order to reduce
time spent by the medical experts reviewing the literature for potential DDIs.
For performance comparisons between machine learning models, it is important
to evaluate models on the data set created by trusted annotators with shared
problem settings. The construction of a common data set is essential for the
development of research on automatic DDI extraction from the literature. The
main goal of this shared task is to have a benchmark for the comparison of
advanced techniques, rather than competitive aspects. The DDIExtraction-2013
[18] follows up the DDIExtraction-2011 whose main goal was the detection of
DDIs from texts. The DDIExtraction-2013 includes the classification of DDI
types in addition to DDI detection. Figure 2.1 shows some examples of labeled
sentences. In the DDIExtraction-2013 task, it is necessary not only to detect
DDIs but also to correctly classify DDIs into the type of the relations such
as “Effect” and “Mechanism”. Additionally, while the data set used for the
DDIExtraction-2011 task was composed by texts describing DDIs from the
DrugBank [39], the new data set for DDIExtraction-2013 also includes MEDLINE
abstracts in order to deal with different types of texts and language styles.

The aim of the DDIExtraction-2013 task is to classify a given pair of drugs
into the following four interaction types or no interaction:

Mechanism

: A sentence describes pharmacokinetic mechanisms of a DDI,
e.g., Grepafloxacine may inhibit the metabolism of theobromine.

Effect

: A sentence represents the effect of a DDI, e.g., Methionine may protect
against the ototoxic effects of gentamicin.

Advice

: A sentence represents a recommendation or advice on the concomitant use of
two drugs, e.g., Alpha-blockers should not be combined with uroxatral.

Int. (Interaction)

: A sentence simply represents the occurrence of a DDI without any information
about the DDI, e.g., The interaction of omeprazole and ketoconazole has
established.

The DDIExtraction-2013 task relies on the DDI corpus, which is a semantically
annotated corpus of documents describing DDIs from the DrugBank database and
MEDLINE abstracts on the subject of DDIs. The statistics of the
DDIExtraction-2013 task data set are shown in Table 3.2. As shown in this
table, the number of pairs that have no interaction (negative pairs) is larger
than that of pairs that have interactions (positive pairs).

| Train | Test
---|---|---
| DrugBank | MEDLINE | DrugBank | MEDLINE
#documents | 572 | 142 | 158 | 33
#sentences | 5,675 | 1,301 | 973 | 326
#pairs | 26,005 | 1,787 | 5,265 | 451
#positive DDIs | 3,789 | 232 | 884 | 95
#negative DDIs | 22,216 | 1,555 | 4,381 | 356
#Mechanism pairs | 1,257 | 62 | 278 | 24
#Effect pairs | 1,535 | 152 | 298 | 62
#Advice pairs | 818 | 8 | 214 | 7
#Int pairs | 179 | 10 | 94 | 2
Table 3.2: Statistics for the DDIExtraction-2013 shared task data set

####  3.4.2 Initializing Word Embeddings

Skip-gram [2] was employed for the pre-training of word embeddings. The 2014
MEDLINE/PubMed baseline distribution is used, and the size of vocabulary was
1,630,978. The embedding of the drugs, i.e., DRUG1, DRUG2 and DRUGOTHER are
initialized with the pre-trained embedding of the word drug. The embeddings of
training words that did not appear in the pre-trained embeddings, as well as
the word position embeddings, are initialized with the random values drawn
from a uniform distribution and normalized to unit vectors. Words whose
frequencies are one in the training data were replaced with an UNK word during
training, and the embedding of words in the test data set that did not appear
in both training and pre-trained embeddings were set to the embedding of the
UNK word.

####  3.4.3 Hyper-parameter Tuning

The official training data set is split into two parts: training and
development data sets. The hyper-parameters are tuned on the development data
set on the softmax model without attentions. Table 3.3 shows the best hyper-
parameters on the softmax model without attentions. The same hyper-parameters
are applied to the other models. The statistics of the development data set is
shown in Table 3.4. The sizes of the convolution windows is set to [3, 4, 5]
that are the same as in Kim et al. [40]. The word position embedding size is
chosen from {10, 20, 30, 40, 50}, the convolutional filter size from {10, 50,
100, 200}, the learning rate of Adam from {0.01, 0.001, 0.0001}, the mini-
batch size from {10, 20, 50, 100, 200}, and the L2 regularization parameter
$\lambda$ from {0.01, 0.001, 0.0001, 0.00001}.

Parameter | Value
---|---
Word embedding size | 200
Word position embeddings size | 20
Convolutional window size | [3, 4, 5]
Convolutional filter size | 100
Initial learning rate | 0.001
Mini-batch size | 100
L2 regularization parameter | 0.0001
Table 3.3: Hyper-paramters | Counts
---|---
Sentences | 1,404
Pairs | 4,998
Mechanism pairs | 232
Effect pairs | 339
Advice pairs | 132
Int pairs | 48
Table 3.4: Statistics of the development data set

###  3.5 Results and Discussions

This section first summarizes the performance of the models and compares the
performance with existing models. Then attention mechanisms are compared and
finally some results for the analysis of the attentions are illustrated.

Type |  $P$ (%) |  $R$ (%) |  $F$ (%)
---|---|---|---
| Softmax without attention
Mechanism | 76.24 ($\pm$4.48) | 57.58 ($\pm$4.41) | 65.31 ($\pm$1.76)
Effect | 67.84 ($\pm$3.56) | 63.61 ($\pm$4.95) | 65.39 ($\pm$1.38)
Advice | 82.26 ($\pm$7.04) | 66.65 ($\pm$9.07) | 72.75 ($\pm$2.72)
Int |  78.99 ($\pm$6.87) | 33.55 ($\pm$2.62) |  47.05 ($\pm$1.71)
All (micro) | 73.69 ($\pm$3.00) | 59.92 ($\pm$3.73) | 65.93 ($\pm$1.21)
| Softmax with attention
Mechanism | 76.34 ($\pm$4.20) |  64.43 ($\pm$5.72) | 67.86 ($\pm$4.10)
Effect | 66.84 ($\pm$3.12) | 65.98 ($\pm$2.63) | 65.58 ($\pm$2.09)
Advice | 80.98 ($\pm$6.14) | 70.83 ($\pm$2.72) | 76.28 ($\pm$1.40)
Int | 73.21 ($\pm$6.30) |  38.44 ($\pm$9.82) | 46.11 ($\pm$3.96)
All (micro) | 73.74 ($\pm$1.88) | 63.05 ($\pm$1.39) | 67.94 ($\pm$0.70)
| Ranking without attention
Mechanism | 78.41 ($\pm$3.99) | 58.17 ($\pm$5.10) | 66.51 ($\pm$2.61)
Effect | 68.16 ($\pm$3.30) | 65.75 ($\pm$3.22) | 66.80 ($\pm$1.46)
Advice |  84.49 ($\pm$3.55) | 67.14 ($\pm$4.68) | 74.61 ($\pm$1.82)
Int | 73.95 ($\pm$7.09) | 33.43 ($\pm$1.18) | 45.91 ($\pm$1.23)
All (micro) | 74.79 ($\pm$2.41) | 60.99 ($\pm$2.65) | 67.10 ($\pm$1.09)
| Ranking with attention
Mechanism |  80.75 ($\pm$2.76) | 61.09 ($\pm$3.03) |  69.45 ($\pm$1.45)
Effect |  69.73 ($\pm$2.64) |  66.63 ($\pm$2.93) |  68.05 ($\pm$1.29)
Advice | 83.86 ($\pm$2.29) |  71.81 ($\pm$2.61) |  77.30 ($\pm$1.13)
Int | 74.20 ($\pm$8.95) | 33.02 ($\pm$1.40) | 45.50 ($\pm$1.51)
All (micro) |  76.30 ($\pm$2.18) |  63.25 ($\pm$1.71) |  69.12 ($\pm$0.71)
Table 3.5: Performance of softmax/ranking CNN models with and without the attention mechanism. The highest scores are shown in bold. Methods |  $P$ (%) |  $R$ (%) |  $F$ (%)
---|---|---|---
No negative instance filtering
CNN [26] | 75.29 | 60.37 | 67.01
MCCNN [41] | - | - | 67.80
SCNN [30] | 68.5 | 61.0 | 64.5
Joint AB-LSTM [31] | 71.82 | 66.90 | 69.27
Attention model | 76.30 | 63.25 | 69.12
With negative instance filtering
FBK-irst [29] | 64.6 | 65.6 | 65.1
Kim et al. [42] | - | - | 67.0
CNN [26] | 75.72 | 64.66 | 69.75
MCCNN [41] | 75.99 | 65.25 | 70.21
SCNN [30] | 72.5 | 65.1 | 68.6
Joint AB-LSTM [31] | 73.41 | 69.66 | 71.48
Table 3.6: Comparison with existing models |  $P$ (%) |  $R$ (%) |  $F$ (%)
---|---|---|---
No attention | 74.79 ($\pm$2.41) | 60.99 ($\pm$2.65) | 67.10 ($\pm$1.09)
Input attention by Wang et al. [32] | 73.48 ($\pm$1.96) | 59.58 ($\pm$1.51) | 65.77 ($\pm$0.80)
Attention | 76.30 ($\pm$2.66) | 63.25 ($\pm$2.59) | 69.12 ($\pm$0.71)
Attention w/o separate attentions $a_{drug}$ | 74.03 ($\pm$2.11) | 63.30 ($\pm$2.41) | 68.17 ($\pm$0.71)
Attention w/o the bias term $b_{\alpha}$ | 71.56 ($\pm$2.18) | 64.19 ($\pm$2.21) | 67.62 ($\pm$0.96)
Table 3.7: Comparison of attention mechanisms on CNN models with ranking
objective function

####  3.5.1 Performance Analysis

The performance of the base CNN models with two objective functions, as well
as with or without the attention mechanism, is summarized in Table 3.5. The
incorporation of the attention mechanism improved the F-scores by about 2
percent points (pp) on models with both objective functions. Both improvements
were statistically significant (p $<$ 0.01) with $t$-test. This shows that the
attention mechanism is effective for both models. The improvement of F-scores
from the least performing model (softmax objective function without the
attention mechanism) to the best performing model (ranking objective function
with the attention mechanism) is 3.19 pp (69.12% versus 65.93%), and this
shows both objective function and attention mechanism are key to improve the
performance. When looking into the individual types, ranking function with the
attention mechanism archived the best F-scores on Mechanism, Effect, Advice,
while the base CNN model achieved the best F-score on Int.

####  3.5.2 Comparison with Existing Models

A comparison with the existing state-of-the-art models is shown in Table 3.6.
The performance is mainly compared without negative instance filtering, which
omits some apparent negative instance pairs with rules [29], since it is not
incorporated. The performance of the existing models with negative instance
filtering is also shown for reference.

In the comparison without negative instance filtering, the model outperformed
the existing CNN models [26, 41, 30]. The model was competitive with Joint AB-
LSTM model [31] that was composed of multiple RNN models.

####  3.5.3 Comparison of Attention Mechanisms

The attention mechanism is compared with the input attention of Wang et al.
[32] to show the effectiveness of the attention mechanism. Table 3.7 shows the
comparison of the attention mechanisms. The base CNN-based model with ranking
loss is also shown for reference, and the results of ablation tests. As is
shown in the table, the attention mechanism by Wang et al. [32] did not work
in DDI extraction. However, the attention improved the performance. This
result shows that the extensions are crucial for modeling attentions in DDI
extraction. The ablation test results show that both extensions to the
attention mechanism, i.e., separate attentions for entities and incorporation
of the bias term, are effective for the task.

####  3.5.4 Visual Analysis

Figure 3.2 shows a visualization of attentions on some sentences with DDI
pairs using the attention mechanism. In the first sentence, DRUG1 and DRUG2
have a Mechanism interaction. The attention mechanism successfully highlights
the keyword concentration. In the second sentence, which have an Effect
interaction, the attention mechanism put high weights on increase and effects.
The word necessary has a high weight on the third sentence with an Advice
interaction. For an Int interaction in the last sentence, the word interaction
is most highlighted.

Figure 3.2: The visualization of attention weights. The dark part indicates
that the attention value is large.

###  3.6 Summary

In this chapter, an attention mechanism for relation extraction is provided as
a preliminary. Base CNN-based DDI extraction models is built with two
different objective functions, softmax and ranking, and The attention
mechanism is incorporated into the models. The performance on the
DDIExtraction-2013 data set is evaluated, and it is shown that both the
attention mechanism and ranking-based objective function are effective for
extracting DDIs. The final model achieved an F-score of 69.12%. This model is
a baseline, text-oriented relation extraction model and is not capable of
domain information.

##  Relation Extraction with a Single Kind of Domain Information

This chapter reviews a study on dealing with a single kind of domain
information, which is the molecular structure of drugs as the preparatory step
in the case study of using heterogeneous information on drugs. A method that
combines the input sentence information and molecular structure information is
reviewed based on [43]. The summary of this chapter is three-fold:

  * •

A neural method to extract DDIs from texts with the related molecular
structure information is described.

  * •

GCNs is applied to pairwise drug molecules and it is shown that GCNs can
predict DDIs between drug molecular structures with high accuracy.

  * •

It is shown that the molecular information is useful in extracting DDIs from
texts.

###  4.1 Background

In parallel to the progress in DDI extraction from texts, Graph Convolutional
Networks (GCNs) have been proposed and applied to estimate physical and
chemical properties of molecular graphs such as solubility and toxicity [44,
45, 46].

It is a very challenging attempt to consider different kinds of items such as
text information and molecular structure information at the same time as
described in Chapters 5-7. However, both the input sentence vector encoded by
CNNs and the molecular structure vector encoded by GCNs are embedded in a low-
dimensional real-valued vector space, so both information is effectively
utilized for the DDI extraction task.

###  4.2 Methods

####  4.2.1 Text-based DDI Extraction

The proposed model for extracting DDIs from texts is based on the CNN model by
Zeng et al. [21]. When an input sentence $S=(w_{1},w_{2},\cdots,w_{N})$ is
given, word embedding $\bm{w}^{w}_{i}$ of $w_{i}$ and word position embeddings
$\bm{w}^{p}_{i,1}$ and $\bm{w}^{p}_{i,2}$ that correspond to the relative
positions from the first and second target entities are prepared,
respectively. These embeddings are concatenated as in Equation (4.1), and the
resulting vector are used as the input to the subsequent convolution layer:

$\displaystyle\bm{w}_{i}=[\bm{w}^{w}_{i};\bm{w}^{p}_{i,1};\bm{w}^{p}_{i,2}],$
(4.1)

where $[;]$ denotes the concatenation. The expression for each filter $j$ with
the window size $k_{l}$ is calculated as:

$\displaystyle\bm{z}_{i,l}$ $\displaystyle=$
$\displaystyle[\bm{w}_{i-(k_{l}-1)/2},\cdots,\bm{w}_{i-(k_{l}+1)/2}],$ (4.2)
$\displaystyle m_{i,j,l}$ $\displaystyle=$
$\displaystyle\mathrm{relu}(\bm{W}^{conv}_{j}\odot\bm{z}_{i,l}+b^{conv}),$
(4.3) $\displaystyle m_{j,l}$ $\displaystyle=$
$\displaystyle\max_{i}m_{i,j,l},$ (4.4)

where $L$ is the number of windows, $\bm{W}^{conv}_{j}$ and $b^{conv}$ are the
weight and bias of CNN, and $\max$ indicates max pooling [35].

The output of the convolution layer is converted into a fixed-size vector that
represents a textual pair as follows:

$\displaystyle\bm{m}_{l}$ $\displaystyle=$
$\displaystyle[m_{1,l},\cdots,m_{J,l}],$ (4.5) $\displaystyle\bm{h}_{t}$
$\displaystyle=$ $\displaystyle[\bm{m}_{1};\ldots;\bm{m}_{L}],$ (4.6)

where $J$ is the number of filters.

Prediction $\hat{\bm{y}_{t}}$ is obtained by the following fully connected
neural networks:

$\displaystyle\bm{h}_{t}^{(1)}$ $\displaystyle=$
$\displaystyle\mathrm{relu}(\bm{W}_{t}^{(1)}\bm{h}_{t}+\bm{b}_{t}^{(1)}),$
(4.7) $\displaystyle\hat{\bm{y}_{t}}$ $\displaystyle=$
$\displaystyle\mathrm{softmax}(\bm{W}_{t}^{(2)}\bm{h}^{(1)}_{t}+\bm{b}_{t}^{(2)}),$
(4.8)

where $\bm{W}_{t}^{(1)}$ and $\bm{W}_{t}^{(2)}$ are weights and
$\bm{b}_{t}^{(1)}$ and $\bm{b}_{t}^{(2)}$ are bias terms.

####  4.2.2 Molecular Structure-based DDI Classification

Drug pairs are represented in molecular graph structures using two GCN
methods: CNNs for fingerprints (NFP) [44] and Gated Graph Neural Networks
(GGNN) [45]. They both convert a drug molecule graph $G$ into a fixed size
vector $\bm{h}_{g}$ by aggregating the representation $\bm{h}^{T}_{v}$ of an
atom node $v$ in $G$. Atoms are represented as nodes and bonds are represented
as edges in the graph.

NFP first obtains the representation $\bm{h}^{t}_{v}$ by the following
equations [44].

$\displaystyle\bm{m}^{t+1}_{v}$ $\displaystyle=$
$\displaystyle\bm{h}^{t}_{v}+\sum_{w\in{N(v)}}\bm{h}^{t}_{w},$ (4.9)
$\displaystyle\bm{h}^{t+1}_{v}$ $\displaystyle=$
$\displaystyle\sigma(\bm{H}_{t}^{deg(v)}\bm{m}_{v}^{t+1}),$ (4.10)

where $\bm{h}^{t}_{v}$ is the representation of $v$ in the $t$-th step, $N(v)$
is the neighbors of $v$, and $\bm{H}_{t}^{deg(v)}$ is a weight parameter.
$\bm{h}^{0}_{v}$ is initialized by the atom features of $v$. $deg(v)$ is the
degree of a node $v$ and $\sigma$ is a sigmoid function. NFP then acquires the
representation of the graph structure

$\bm{h}_{g}=\sum_{v,t}\mathrm{softmax}(\bm{W}^{t}\bm{h}^{t}_{v}),$ (4.11)

where $\bm{W}^{t}$ is a weight matrix.

GGNN first obtains the representation $\bm{h}^{t}_{v}$ by using Gated
Recurrent Unit (GRU)-based recurrent neural networks [45] as follows:

$\displaystyle\bm{m}^{t+1}_{v}$ $\displaystyle=$
$\displaystyle\sum_{w\in{N(v)}}\bm{A}_{e_{vw}}\bm{h}^{t}_{w}$ (4.12)
$\displaystyle\bm{h}^{t+1}_{v}$ $\displaystyle=$
$\displaystyle\mathrm{GRU}([\bm{h}^{t}_{v};\bm{m}_{v}^{t+1}]),$ (4.13)

where $\bm{A}_{e_{vw}}$ is a weight for the bond type of each edge $e_{vw}$.
GGNN then acquires the representation of the graph structure.

$\bm{h}_{g}=\sum_{v}\sigma(i([\bm{h}^{T}_{v};\bm{h}^{0}_{v}]))\odot(j(\bm{h}^{T}_{v})),$
(4.14)

where $i$ and $j$ are linear layers and $\odot$ is the element-wise product.

The representation of a molecular pair is obtained by concatenating the
molecular graph representations of drugs $g_{1}$ and $g_{2}$, i.e.,
$\bm{h}_{m}=[\bm{h}_{g_{1}};\bm{h}_{g_{2}}]$.

Prediction $\hat{\bm{y}}_{m}$ is obtained as follows:

$\displaystyle\bm{h}^{(1)}_{m}$ $\displaystyle=$
$\displaystyle\mathrm{relu}(\bm{W}^{(1)}_{m}\bm{h}_{m}+\bm{b}^{(1)}_{m}),$
(4.15) $\displaystyle\hat{\bm{y}}_{m}$ $\displaystyle=$
$\displaystyle\mathrm{softmax}(\bm{W}^{(2)}_{m}\bm{h}^{(1)}_{m}+\bm{b}^{(2)}_{m}),$
(4.16)

where $\bm{W}^{(1)}_{m}$ and $\bm{W}^{(2)}_{m}$ are weights and
$\bm{b}^{(1)}_{m}$ and $\bm{b}^{(2)}_{m}$ are bias terms.

####  4.2.3 DDI Extraction from Texts Using Molecular Structures

Figure 4.1: Overview of the CNN-based DDI extraction model that use molecular
structure information

The simultaneous use of textual and molecular information is achieved by
concatenating the text-based and molecule-based vectors:
$\bm{h}_{all}=[\bm{h}_{t};\bm{h}_{m}]$. Molecule-based vectors are normalized.
Then $\bm{h}_{all}$ is used instead of $\bm{h}_{t}$ in Equation 4.7.

In training, the molecular-based DDI classification model is trained. The
molecular-based classification is conducted by minimizing the loss function
$L_{m}=-\sum{\bm{y}_{m}\log\hat{\bm{y}}_{m}}$. The parameters for GCNs are
fixed and the text-based DDI extraction model is trained by minimizing the
loss function. $L_{t}=-\sum{\bm{y}_{t}\log\hat{\bm{y}}_{t}}$.

###  4.3 Experimental Settings

As preprocessing, sentences are split into words using the GENIA tagger [47].
The drug mentions of the target pair are replaced with DRUG1 and DRUG2
according to their order of appearance. Other drug mentions are also replaced
with DRUGOTHER. Negative instance filtering is not employed unlike other
existing methods, e.g., Liu et al. [26], since the focus is to evaluate the
effect of the molecular information on texts.

Mentions in texts are linked to DrugBank entries by string matching. The
mentions and the names are lowercased in the entries and chose the entries
with the most overlaps. As a result, 92.15% and 93.09% of drug mentions in
train and test data set matched the DrugBank entries.

####  4.3.1 Data and Task for Molecular Structures

255,229 interacting (positive) pairs are extracted from DrugBank. Note that,
unlike text-based interactions, DrugBank only contains the information about
interacting pairs; there are no detailed labels and no information for non-
interacting (negative) pairs. Thus the same number of pseudo-negative pairs
are generated by randomly pairing drugs and removing those in positive pairs.
To avoid overestimation of the performance, drug pairs mentioned in the test
set of the text corpus are deleted. Positive and negative pairs are split into
4:1 for training and test data, and the molecular GCN-based model is evaluated
on the classification accuracy.

To obtain the graph of a drug molecule, the SMILES [48] string encoding of the
molecule is taken as input from DrugBank and then converted into the graph
using RDKit [49]. For the atom features, randomly embedded vectors are used
for each atom (i.e., C, O, N, …). 4 bond types, single, double, triple, and
aromatic, are used.

####  4.3.2 Training Settings

Mini-batch training is employed using the Adam optimizer [37]. L2
regularization is used to avoid over-fitting. The bias term $\bm{b}^{(2)}_{t}$
is tuned for negative examples in the final softmax layer. Pre-trained word
embeddings trained by using the word2vec tool [2] on the 2014 MEDLINE/PubMed
baseline distribution are employed. The vocabulary size was 215,840. The
embedding of the drugs, i.e., DRUG1 and DRUG2 were initialized with the pre-
trained embedding of the word drug. The embeddings of training words that did
not appear in the pre-trained embeddings were initialized with the average of
all pre-trained word embeddings. Words that appeared only once in the training
data were replaced with an UNK word during training, and the embedding of
words in the test data set that did not appear in both training and pre-
trained embeddings were set to the embedding of the UNK word. Word position
embeddings are initialized with random values drawn from a uniform
distribution. The molecule-based vectors of unmatched entities are set to zero
vectors.

###  4.4 Results and Discussions

Methods | P | R | F (%)
---|---|---|---
Liu et al. [26] | 75.29 | 60.37 | 67.01
Zheng et al. [50] | 75.9 | 68.7 | 71.5
Lim et al. [33] | 74.4 | 69.3 | 71.7
Text-only | 71.97 | 68.44 | 70.16
\+ NFP | 72.62 | 71.81 | 72.21
\+ GGNN | 73.31 | 71.81 | 72.55
Table 4.1: Evaluation on DDI extraction from texts DDI Type | Mech. | Effect | Adv. |  Int (%)
---|---|---|---|---
Text-only | 69.52 | 69.27 | 79.81 | 48.18
\+ NFP | 72.70 | 72.44 | 79.56 | 46.98
\+ GGNN | 73.83 | 71.03 | 81.62 | 45.83
Table 4.2: F-scores of each DDI type Methods | Accuracy (%)
---|---
NFP | 94.19
GGNN | 98.00
Table 4.3: Accuracy of binary classification on DrugBank pairs Methods | P | R | F | Acc. (%)
---|---|---|---|---
NFP | 15.56 | 48.93 | 23.61 | 45.78
GGNN | 15.11 | 57.10 | 23.90 | 37.72
Table 4.4: Classification of DDIs in texts by molecular structure-based DDI
classification model

Table 4.1 shows the performance of DDI extraction models. The performance
without negative instance filtering or ensemble is shown for a fair
comparison. The increase in recall and F-score are observed by using molecular
information, which results in the state-of-the-art performance with GGNN.

Both GCNs improvements were statistically significant ($p<0.05$ for NFP and
$p<0.005$ for GGNN) with randomized shuffled test [51].

Table 4.2 shows F-scores on individual DDI types. The molecular information
improves F-scores especially on type Mechanism and Effect.

The accuracy of binary classification on DrugBank pairs is also evaluated by
using only the molecular information in Table 4.3. The performance is high,
although the accuracy is evaluated on automatically generated negative
instances.

Finally, the molecular-based DDI classification model trained on DrugBank is
applied to the DDIExtraction 2013 task data set. Since the DrugBank has no
detailed labels, all four types of interactions are mapped to positive
interactions and evaluated the classification performance. The results in
Table 4.4 show that GCNs produce higher recall than precision and the overall
performance is low considering the high performance on DrugBank pairs. This
might be because the interactions of drugs are not always mentioned in texts
even if the drugs can interact with each other and because hedged DDI mentions
are annotated as DDIs in the text data set. The DDI extraction model is
trained only with molecular information by replacing $\bm{h}_{all}$ with
$\bm{h}_{m}$, but the F-scores were quite low ($<$ 5%). These results show
that textual relations cannot be predicted only with molecular information.

###  4.5 Summary

This chapter reviewed a neural method for relation extraction using both
textual information and molecular structures. The model was evaluated on the
DDI extraction task as a case study. The results show that DDIs can be
predicted with using molecular structure information. Since this model cannot
deal with heterogeneous domain information, the following chapters develop
relation extraction models that can consider heterogeneous domain information.

##  Relation Extraction with Multiple Domain Information

This chapter includes work from the published paper Asada et al. (2021a) [9]
and Iinuma et al. (2021) [52]. This chapter proposes a novel relation
extraction method that utilizes two kinds of domain information. The DDI
extraction task is chosen as a case study. A method to utilize the description
and the structure of the entity obtained from drug database DrugBank as well
as large-scale raw text information is proposed. DrugBank is in focus because
the DDIExtraction 2013 shared task data set is created based on the DrugBank
database. Other databases are left for future work. Specifically, the
description and molecular structure information of drugs in the database are
utilized. The information from large-scale raw texts is incorporated by using
a Bidirectional Encoder Representations from Transformers (BERT) model [4]
pre-trained on large-scale raw text.

Experimental results show that SciBERT boosts the performance of the baseline
model. As a result, the performance is already strong enough and better than
the previously reported performance. It is shown that the drug database
information is complementary to the large-scale pre-trained information, and
the simultaneous use of drug description and drug molecular structure
information can enhance the performance of DDI extraction from texts with
SciBERT.

###  5.1 Background

Since the annotation efforts are costly and time-consuming, it is unrealistic
to prepare a sufficient amount of annotated data. In addition, it is difficult
to learn how to extract DDIs from text only with the limited amount of
annotated text because a deep understanding of DDI interaction descriptions
often requires domain knowledge of drugs. Various drug information, such as
detailed descriptions and molecular structure information on drugs, are
registered in drug databases. Furthermore, models pre-trained on large-scale
raw text show significant improvements in various natural language processing
(NLP) tasks [4]. Effective use of such external information is necessary to
reduce the reliance on annotated text.

This chapter extends the previous section at the following points.

  * •

The token representation changes from word2vec to contextualized vectors
obtained by SciBERT. As a result, the performance of the baseline with the
state-of-the-art performance is remarkably improved.

  * •

The neural molecular GNN [53] that considers relatively large fragments of
atoms and better represents molecular structures is employed.

  * •

Drug descriptions registered in the drug database is used and it is shown that
drug description information is useful for extracting DDIs from the corpus for
some DDI types.

  * •

It is found that the large-scale pre-training information, drug description,
and drug molecular information are complementary and their effective
combination can largely improve the DDI extraction performance.

Figure 5.1: Overview of the DDI extraction model with drug descriptions and
drug molecular structures. (A) Illustrates how to encode input sentences, drug
descriptions and drug molecular structures. (B) and (C) show the prediction
layer when the drug description representation and the drug molecular
structure representation are used.

###  5.2 Methods

The overview of the proposed method is illustrated in Figure 5.1. For the
baseline model, the convolutional neural network (CNN)-based DDI extraction
model [43] that receives an input sentence with a target drug pair and
classifies the pair into a specific DDI type is employed. The input sentence
is enriched using SciBERT [54], which is a BERT model trained on large-scale
biomedical and computer science text. The drug description representation of
the target drugs is obtained using SciBERT and the molecular structure
representation of the target drugs using molecular graph neural network (GNN)
model proposed by Tsubaki et al. [53]. These drug description and molecular
structure representation are combined with the enriched input sentence
representation and classify the target drug pair into a specific DDI type.

####  5.2.1 Input Sentence Representation

This section follows the previous section to preprocess the input sentences.
Then a preprocessed input sentence is converted into a real-valued fixed size
vector by BERT and CNN-based model [4, 21] and shows the model in the left
part of Figure 5.1A. Given an input sentence $S=(w_{1},\cdots,w_{n})$ with
drug mentions $m_{1}$ and $m_{2}$, the sentence is first split into wordpieces
(a.k.a., subwords) by the WordPiece algorithm [55]. Each wordpiece $w_{i}$ is
converted into a real-valued pre-trained contextualized embedding
$\bm{w}^{w}_{i}\in\mathbb{R}^{d^{w}}$ by the BERT model (light blue vectors in
Figure 5.1A). $d^{p}$-dimensional position embeddings $\bm{w}^{p1}_{i}$ and
$\bm{w}^{p2}_{i}$ for each wordpiece, which correspond to the relative
positions from the first and second target mentions are prepared,
respectively. (green vectors in Figure 5.1A) The wordpiece embedding
$\bm{w}^{w}_{i}$ and the position embeddings $\bm{w}^{p1}_{i}$ and
$\bm{w}^{p2}_{i}$ are concatenated as in the following Equation (5.1):

$\bm{w}_{i}=[\bm{w}^{w}_{i};\bm{w}^{p1}_{i};\bm{w}^{p2}_{i}],$ (5.1)

where [;] denotes concatenation. the resulting embeddings are used to prepare
the input to the convolution layer.

$\bm{z}_{i}$ that is the concatenation of $k$ input embeddings222Multiple
windows can be employed instead of a single window with the size $k$, but
there is no significant difference in the performance in the preliminary
experiment. around $w_{i}$ is introduced:

$\bm{z}_{i}=[\bm{w}^{\mathrm{T}}_{\lfloor{i-(k-1)/2}\rfloor};\ldots;\bm{w}^{\mathrm{T}}_{\lfloor{i-(k+1)/2}\rfloor}]^{\mathrm{T}}.$
(5.2)

Convolution to the embeddings is applied as follows:

$m_{i,j}=f(\bm{W}^{sent}_{j}\odot\bm{z}_{i}+b^{sent}),$ (5.3)

where $\odot$ is an element-wise product, $b^{sent}$ is a bias term, and
$f(\cdot)$ is a GELU [56] function.333 The GELU activation function is chosen
from ReLU, eLU, SeLU and GELU based on the results in the preliminary
experiment. A weight tensor for convolution is defined as
$W^{sent}\in\mathbb{R}^{d^{c}\times(d^{w}+2d^{p})\times k}$. The $j$-th column
of $W^{sent}$ is represented as $W^{sent}_{j}$. $k$ is a window size. The
tensor $W^{sent}$ is depicted as a red box in the left part of Figure 5.1A.
Then, max-pooling to convert the output of each filter in the convolution
layer into a fixed-size vector is employed as follows:

$\bm{h}^{sent}=\max_{i}m_{i,j}.$ (5.4)

####  5.2.2 Drug Description Representation

Similarly to the input sentences, the description sentences of a drug mention
are converted to the real-valued fixed size vector by BERT and CNN. The
wordpiece embeddings by BERT are directly used without word position
embeddings to prepare the input to the convolution layer. A convolution weight
tensor $W^{desc}\in\mathbb{R}^{d^{c}\times(d^{w})\times k}$ and bias
$b^{desc}$ for description are defined. Convolution and max-pooling are
employed in the same way as the processing of the input sentences and the
description representations $\bm{h}^{desc1}$ and $\bm{h}^{desc2}$ of drug
mentions $m_{1}$ and $m_{2}$ are obtained respectively.

####  5.2.3 Molecular Structure Representation

Figure 5.2: Illustration of molecular fingerprints. This figure shows the
extraction of several fingerprint subgraphs from a molecular structure when
radius is 2.

The molecular graph structures of drugs are represented using GNNs. GNNs
convert a drug molecule graph $G$ into a fixed size vector $h^{g}$. Atoms as
represented as nodes and bonds as edges in the graph. The neural molecular GNN
method proposed by Tsubaki et al. [53] is employed. The molecular GNN method
uses relatively large fragments referred to as $r$-radius subgraphs or
molecular fingerprints to represent atoms with their contexts in the graph.
The molecular GNN adopts fingerprint vectors as atom vectors, initializes the
vectors randomly, and updates them considering the graph structure of a
molecule. The vector of the $i$-th atom in a drug molecule is defined as
$\bm{h}_{i}$ and the set of its neighboring atoms as $N_{i}$. The vector
$\bm{h}_{i}$ is updated in the $\ell$-th step as follows:

$\bm{h}^{\ell}_{i}=\bm{h}^{\ell-1}_{i}+\sum_{j\in
N_{i}}f(\bm{W}^{\ell-1}_{hidden}\bm{h}^{\ell-1}_{j}+\bm{b}_{hidden}^{l-1}),$
(5.5)

where $f(\cdot)$ denotes a ReLU function. The drug molecular vector is
obtained by summing up all the atom vectors and then the resulting vectors are
fed into a linear layer.

$\bm{h}^{mol}=f(\bm{W}_{output}\sum_{i}^{M}\bm{h}^{L}_{i}+\bm{b}_{output}),$
(5.6)

where $M$ is the number of fingerprints. Figure 5.2 shows how the molecular
GNN model extracts fingerprints including $\beta$-lactam ($\bm{h}_{1}$) from
penicillin drug ($r$=2) and update fingerprint vectors.

The molecular structure representations $\bm{h}^{mol1}$ and $\bm{h}^{mol2}$ of
drug mentions $m_{1}$ and $m_{2}$ are obtained, respectively.

####  5.2.4 DDI Extraction Using Database Information

When the drug description information for DDI extraction is used, the input
sentence representation and two description representations as in Equation
5.7:

$\bm{h}=[\bm{h}^{sent};\bm{h}^{desc1};\bm{h}^{desc2}].$ (5.7)

Similarly, two molecular structure representations are concatenated with the
input sentence representation as in Equation 5.8:

$\bm{h}=[\bm{h}^{sent};\bm{h}^{mol1};\bm{h}^{mol1}].$ (5.8)

The resulting vector is used as the input to the prediction layer. $\bm{h}$
into prediction scores is converted using a weight matrix
$\bm{W}^{pred}\in\mathbb{R}^{o{\times}d_{p}}$:

$\bm{s}=\bm{W}^{pred}\bm{h},$ (5.9)

where $\bm{s}=[s_{1},\cdots,s_{o}]$ and $o$ is the number of DDI types.
$\bm{s}$ is converted into the probability of possible interactions $\bm{p}$
by a softmax function:

$\bm{p}=[p_{1},\cdots,p_{o}],\
p_{j}=\frac{\exp{(s_{j})}}{\sum^{o}_{l=1}\exp{(s_{l})}}.$ (5.10)

The DDI extraction using drug description information and drug molecular
structure information in Figure 5.1B and C are illustrated, respectively.

####  5.2.5 Training

The loss function $L$ is defined as in the Equation (5.11) using $\bm{p}$ in
Equation (5.10) when the gold type distribution $\bm{y}$ is given. $\bm{y}$ is
a one-hot vector where the probability of the gold label is 1 and the other
probabilities are 0.

$L=-\sum\bm{y}\log{\bm{p}}$ (5.11)

####  5.2.6 Ensemble

An ensemble technique is employed to combine the prediction from different
models. Specifically, the prediction scores are simply summed up from
different models for the ensemble after each of the models is trained
separately. For instance, when the prediction of the model with the
description information and that with the molecular structure information are
combined, the prediction scores are summed up in Equation (5.9) as follows:

$\bm{s}=\bm{s}^{desc}+\bm{s}^{mol}.$ (5.12)

###  5.3 Experimental Settings

This section explains the DDI extraction task settings, drug database
preprocessing, drug mention linking, and hyper-parameter settings.

####  5.3.1 DrugBank Preprocessing

DrugBank is a freely available drug database containing more than 10,000
drugs. Each drug is given sentences describing its characteristics and
efficacy. The first sentence of the drug description of Salbutamo is shown as
an example: Salbutamol is a short-acting, selective beta2-adrenergic receptor
agonist used in the treatment of asthma and COPD. DrugBank also contains drug
molecular structure information. Structure information is registered in SMILES
string encoding.

To obtain the graph of a drug molecule, the SMILES string encoding of the
molecule is obtained as input from DrugBank and then converted it into the
graph structure using RDKit [57]. Fingerprints are extracted from the graph
using preprocessing scripts provided by Tsubaki et al. [53].

####  5.3.2 Drug Mention Linking

Mentions in the corpus are linked to DrugBank entries by relaxed string
matching. In particular, each mention and the following items in the DrugBank
entries are lowercased, and the entry that includes an item showing the most
overlap with the mention is chosen.

  * •

Name: Headword of the drug entry

  * •

Brand: Brand names from different manufactures

  * •

Product: The final commercial preparation of the drug

  * •

Synonym: Synonyms of the drug

  * •

ATC codes: Codes for hierarchical drug classification

For the ATC code, the same code can be assigned to multiple drugs, so only the
ATC codes that are assigned to single drugs is used for mention linking. Also,
for synonyms, mentions and synonyms are linked by exact string matching
instead of relaxed string matching to avoid the matching with very short
strings (e.g., abbreviations). With this linking, 90.50% and 91.10% of drug
mentions in DDIExtraction-2013 train and test data set matched the DrugBank
entries. Figure 5.3 shows how the linking is performed. The input sentence
contains two mentions “norgestrel” and “norethindrone”. String matching is
conducted to link these mentions to DrugBank entries. As a result, the mention
“noregestrel” matched the Name item and the mention “norethindrone” matched
the Product item.

Figure 5.3: Linking between mentions and DrugBank entry

####  5.3.3 Training Settings

This section follows the training settings for the fine-tuning of BERT on the
GLUE tasks [4] except for the following two points. First,the AdamW optimizer
[58] is employed instead of Adam optimizer. Second, mixed-precision training
[59] is employed for the memory efficiency.

Dropout [60] is employed to the input of the convolution layer for
regularization. Word position embeddings are initialized with random values
drawn from a uniform distribution between $-10^{-3}$ and $10^{-3}$. The
description and molecular structure vectors of unmatched entities are set to
zero vectors. Table 5.1 and 5.2 shows hyper-parameters for CNNs and GNNs. The
same hyper-parameters are used as the GLUE tasks in Devlin et al. [4] for the
BERT layer. In the DDIExtraction 2013 shared task, the official development
data set is not provided; thus a development data set is prepared from the
official training data set to choose the other hyper-parameters. In order to
train the model on the same setting as other existing models [26, 43, 61], the
development data set is included in the entire training data set for training
the model. The entire training data set for training the model is used to
evaluate the performance on the test set. For GNNs, the results with different
radii 0, 1, and 2 for molecular fingerprints is shown. Note that GNNs with a
radius of 0 mean no molecular fingerprints, which assign vectors to atoms.

Parameter | Value
---|---
Word embedding size $d^{w}$ | 768
Initial learning rate | 5e-5
Number of fine-tuning epochs | 3
L2 weight decay | 0.01
Dropout rate | 0.1
Mini-batch size | 32
Word position embedding size $d^{p}$ | 10
Convolution window size $k$ | 5
Convolution filter size $d^{c}$ | 768
Convolution window size for description | 3
Convolution filter size for description | 20
Table 5.1: Hyper-parameters for CNNs Parameter | Value
---|---
Molecular embedding size $d^{g}$ | 50
Number of hidden layer $L$ | 5
Radius | 1
Table 5.2: Hyper-parameters for GNNs

###  5.4 Results and Discussions

Table 5.3 shows the performance of DDI extraction models including the
proposed models with different settings and the state-of-the-art models. It
can be seen that the baseline text-only model (SciBERT CNN) using SciBERT is
powerful. SciBERT improved the performance of the model without SciBERT
(word2vec CNN) by 11.04% points in the micro F-score. With this improvement,
the model with SciBERT has achieved the state-of-the-art performance when it
is compared with the state-of-the-art models in the top rows of the table.
When the CNNs are ommited from the baseline model (SciBERT Linear), the first
special token [CLS] is used as the aggregated representation of the sentence
and fed the embedding of [CLS] into the linear classifier layer. The
performance slightly dropped with this omission but the difference is
negligible. This indicates the BERT model is powerful enough to capture the
similar information as CNNs.

Additional increase of the micro F-score is observed by using drug description
and molecular structure information as shown in the bottom part of the table.
This shows the large-scale raw text information from SciBERT and the database
information are complementary, and they are both useful for extracting DDIs
from text. For GNNs, GNNs with molecular fingerprints (radius=1 or 2) show
better performance than GNNs without them (radius=0), and GNNs with the radius
of 1 show the highest performance. When comparing the description and
molecular structure information, the micro F-score with molecular structure
information (radius=1) is slightly higher than one with the description
information (+Desc), but their difference is not significant and the
superiority depends on how to represent the molecular structure information,
i.e., molecular fingerprints. The author leaves the search of the better
representations for future work. The improvement by the ensemble model of
description and the molecular structure information is statistically
significant when compared with the baseline model ($p<0.005$, McNemar test).
The scikit-learn [62] Python library is used for evaluating the statistical
significance.

Table 5.4 shows the performance of DDI extraction models on the development
data set. Consistently with the results on the test set in Table 5.3, either
of the description information and molecular structure information improves
the performance and the combination of the two kinds of information showed the
highest F-scores on the development data set. However, there are some
inconsistencies in the results on development and test data sets; the model
with molecular structure information showed a higher F-score than the model
with description information on the development data set, while the model with
molecular structure information showed a lower F-score on the test data set.

Table 5.5 shows the F-scores on individual DDI types. The description
information improves F-scores for Mechanism, Effect, and Int. types, but it
degrades the F-scores for Advice. The molecular structure information improves
F-scores for Effect and Advice, but it degrades the F-scores for Mechanism and
Int. for some radii. This indicates the two information have different effects
on extracting DDIs, and each kind of information is not enough to improve the
entire DDI extraction performance. When both the description and molecular
structure information are used by the ensemble technique, the model shows
higher performance than the baseline model on all types. The training data set
is cross-validated using 5-fold cross-validation and further analyzed the
performance on individual DDI types. Table 5.6 shows the F-scores for folds of
cross-validated training data set. The micro-averaged F-score is used to
calculate the average of the folds. The models with individual information
source show higher performance than the baseline model on Mechanism and Int.,
while they show comparable or lower performance than the baseline model on
other labels. Although the changes in performance are inconsistent for the DDI
types and folds, the model with the ensemble technique shows higher
performance than the models with individual information source on average. As
a result, the model with the ensemble technique improves the F-scores on
average for all the types except for Int., where the model performs on par
with the baseline model. These results show that the performance on each label
is affected by data splitting, but overall, when both the description
information and molecular structure information are used by the ensemble
technique, the model is effective for improving the performance of DDI
extraction.

Table 5.7 shows the comparison of F-scores on the two different subsets of the
test set: MEDLINE and DrugBank. The model with the description and one with
molecular structure (radius=1) degrade the F-score for MEDLINE, whereas both
the description and molecular structure information improved the F-scores for
DrugBank. For both subsets, the ensemble model greatly improved the F-score.
These results also indicate the description and molecular structure
information are complementary.

Method | P | R | F (%)
---|---|---|---
Liu et al. [26] | 75.29 | 60.37 | 67.01
BioBERT [61] | - | - | 78.8
Text-only (word2vec CNN) |  |  |
[43] | 71.97 | 68.44 | 70.16
Text-only (SciBERT Linear) | 80.28 | 81.92 | 81.09
Text-only (SciBERT CNN) | 83.10 | 80.38 | 81.72
\+ Desc | 84.05 | 81.81 | 82.91
\+ Mol (radius=0) | 83.29 | 82.02 | 82.65
\+ Mol (radius=1) | 83.57 | 82.12 | 82.84
\+ Mol (radius=2) | 83.66 | 81.10 | 82.36
\+ Desc + Mol (radius=1) | 85.36 | 82.83 | 84.08
\+ Desc + Mol (radius=0,1,2) | 84.51 | 82.53 | 83.51
\+ Mol (radius=0,1,2) | 84.69 | 82.53 | 83.60
Table 5.3: Evaluation on DDI extraction from texts on the test set. Text only (SciBERT CNN) model is defined as the baseline model. Method | P | R | F (%)
---|---|---|---
Text-only (SciBERT CNN) | 83.55 | 80.19 | 81.84
\+ Desc | 83.19 | 82.31 | 82.75
\+ Mol (radius=0) | 83.73 | 81.25 | 82.47
\+ Mol (radius=1) | 82.85 | 83.90 | 83.37
\+ Mol (radius=2) | 82.88 | 83.58 | 83.23
\+ Desc + Mol (radius=1) | 84.59 | 84.32 | 84.46
Table 5.4: Evaluation on DDI extraction from texts on the development set | DDI Type
---|---
Method | Mech. | Effect | Adv. |  Int. (%)
Text-only | 86.18 | 79.12 | 88.34 | 55.94
\+ Desc | 87.62 | 81.08 | 87.05 | 60.27
\+ Mol (radius=0) | 84.65 | 81.20 | 90.67 | 55.71
\+ Mol (radius=1) | 86.33 | 80.48 | 92.07 | 49.25
\+ Mol (radius=2) | 84.02 | 82.24 | 88.58 | 57.34
\+ Desc + Mol (radius=1) | 87.61 | 82.05 | 90.79 | 58.74
Table 5.5: Performance on individual DDI types in F-scores. The best score for each type is shown in bold and the scores lower than the baseline model are shown with underlines. |  | DDI Type
---|---|---
| Method | Mech. | Effect | Adv. |  Int. (%)
Fold 1 | Text-only | 84.60 | 86.38 | 85.80 | 68.29
| \+ Desc | 82.55 | 81.82 | 85.23 | 64.37
| \+ Mol (radius=1) | 84.55 | 84.62 | 84.53 | 71.05
| \+ Desc + Mol (radius=1) | 86.13 | 85.46 | 86.69 | 67.47
Fold 2 | Text-only | 83.46 | 83.26 | 78.80 | 81.48
| \+ Desc | 84.15 | 82.52 | 81.99 | 79.01
| \+ Mol (radius=1) | 82.26 | 83.45 | 81.64 | 76.54
| \+ Desc + Mol (radius=1) | 84.29 | 83.38 | 82.64 | 79.01
Fold 3 | Text-only | 84.91 | 59.21 | 76.54 | 91.43
| \+ Desc | 83.40 | 88.31 | 73.53 | 91.67
| \+ Mol (radius=1) | 84.43 | 86.24 | 75.24 | 94.44
| \+ Desc + Mol (radius=1) | 86.09 | 87.25 | 76.22 | 92.96
Fold 4 | Text-only | 76.81 | 81.56 | 78.01 | 79.45
| \+ Desc | 77.54 | 82.47 | 79.65 | 81.16
| \+ Mol (radius=1) | 78.17 | 84.03 | 77.34 | 76.92
| \+ Desc + Mol (radius=1) | 77.35 | 85.15 | 79.40 | 83.33
Fold 5 | Text-only | 81.97 | 81.76 | 89.51 | 76.54
| \+ Desc | 84.95 | 83.02 | 87.73 | 81.48
| \+ Mol (radius=1) | 86.09 | 83.74 | 87.23 | 73.33
| \+ Desc + Mol (radius=1) | 86.26 | 84.91 | 88.34 | 75.00
Average | Text-only | 82.34 | 76.99 | 81.67 | 79.07
| \+ Desc | 83.09 | 84.39 | 81.27 | 78.09
| \+ Mol (radius=1) | 82.47 | 83.57 | 81.60 | 78.97
| \+ Desc + Mol (radius=1) | 84.01 | 85.20 | 82.70 | 78.99
Table 5.6: Individual F-scores on 5-fold cross-validated training data set. The micro-averaged F-score is used to calculate the average of the folds. The best score for each type is shown in bold and the scores lower than the baseline model are shown with underlines. Method | MEDLINE | DrugBank | Overall (%)
---|---|---|---
Text-only (SciBERT CNN) | 74.57 | 82.44 | 81.72
\+ Desc | 74.41 | 83.75 | 82.91
\+ Mol (radius=0) | 75.00 | 83.41 | 82.65
\+ Mol (radius=1) | 73.98 | 83.71 | 82.84
\+ Mol (radius=2) | 74.57 | 83.15 | 82.36
\+ Desc + Mol (radius=1) | 78.16 | 84.67 | 84.08
Table 5.7: Comparisons of F-scores on different parts of the test set

####  5.4.1 Pre-training of GNNs and CNNs on DrugBank

To investigate the further use of DrugBank information, it is verified if the
DrugBank DDI labels can improve the DDI extraction performance. Specifically,
GNNs are pretrained for molecular structure information and CNNs for
description information on DrugBank DDI labels. Many drug pairs have
information of interactions, so this pre-training needs no additional
annotations.

50,000 interacting (positive) pairs are extracted from DrugBank. Note that,
unlike the DDIExtraction 2013 shared task data set, DrugBank only contains the
information of interacting pairs; there are no detailed labels and no
information for non-interacting (negative) pairs. Thus, the same number of
pseudo negative pairs are generated by randomly pairing drugs and removing
those in positive pairs. To avoid overestimation of the performance, drug
pairs mentioned in the test set of the text corpus are deleted in preparing
the pairs. Positive and negative pairs are split into 4:1 for train and test
data, and evaluated the classification accuracy using only the molecular
information or only the description.

First, the performances of the accuracy of binary classification on DrugBank
DDI pairs are shown in Table 5.8. The performance is surprisingly high,
although the accuracy is evaluated on automatically generated negative
instances. Overall, both drug description and molecular structure information
can capture DDI information in DrugBank. In detail, the accuracy with drug
description information is higher than that with molecular structure
information. For molecular structure information, GNN with the radius of 2
shows the best performance. The difference in accuracy between radius 0 and 2
is 21.78% points, and this large difference shows the importance of capturing
molecular fingerprints for DDI.

|  | Accuracy (%)
---|---|---
Description | SciBERT | 91.05
Molecular Structure | GNN (radius=0) | 67.58
| GNN (radius=1) | 82.21
| GNN (radius=2) | 89.36
Table 5.8: Accuracy of binary classification on the DrugBank pairs

CNNs and GNNs are pre-trained using the DrugBank interaction labels including
the pseudo negative labels and fine-tuned on the DDIExtraction 2013 data set.
Table 5.9 shows the comparison of the F-scores with or without pre-training.
Unfortunately, for all the settings, the models with pre-training show lower
performance than those without pre-training. This may be because the labels in
the DDI extraction tasks are annotated depending on the context of the pairs
and the labels can be inconsistent with labels in DrugBank and because the
pseudo negative examples are used in training instead of the real negative
examples.

| Methods | P | R | F (%)
---|---|---|---|---
| SciBERT | 83.10 | 80.38 | 81.72
w/ pre-training | \+ Desc | 84.62 | 79.26 | 81.85
| \+ Mol (radius=0) | 82.69 | 81.00 | 81.83
| \+ Mol (radius=1) | 84.51 | 80.28 | 82.34
| \+ Mol (radius=2) | 82.36 | 80.28 | 81.74
w/o pre-training | \+ Desc | 84.05 | 81.81 | 82.91
| \+ Mol (radius=0) | 83.29 | 82.02 | 82.65
| \+ Mol (radius=1) | 83.57 | 82.12 | 82.84
| \+ Mol (radius=2) | 83.66 | 81.10 | 82.36
Table 5.9: Evaluation on DDI extraction from texts with or without pre-
training of GNNs for the molecular structure and CNNs for the description

####  5.4.2 Can DrugBank Information Alone Extract DDIs from Texts?

To further investigate how contextual information is important in the DDI
task, it is verified whether the textual DDI can be extracted only from the
drug information in DrugBank without using the input sentence. The input
sentence representation $\bm{h}^{sent}$ is simply omitted from Equation 5.7
and 5.8 and the DDI extraction models are trained, but the F-scores were quite
low (¡5%) for both models. This result shows that DDI relations cannot be
extracted from texts only with the description and molecular structure
information. This indicates that DDI extraction from text greatly depends on
the context information around drug mention pairs and the models on the
database information serve as a supplement to the textual CNN model.

Figure 5.4: F-scores for different sentence lengths on the 5-fold cross-
validated training data set. The micro-averaged F-score is used to calculate
the average of the folds.

####  5.4.3 Error Analysis

Figure 5.4 shows F-scores for different sentence lengths on the validation
data set. Since the instances with longer sentence lengths are relatively few,
5-fold cross-validation is used on the official training data set. Here, the
sentence length is defined to be the number of subwords divided by the SciBERT
vocabulary. In the previous work, Quan et al. [41] analyzed the F-scores for
the sentence length and pointed out that the performance is low for very long
sentences with 60 or more words. Wang et al. [63] also analyzed the F-scores
for the sentence length and showed that F-scores tend to drop when the lengths
of the instances are in the range from 71 to 100. The baseline model shows
lower performance for long sentences with 80 or more subwords, and this result
shows the same tendency as the previous analyses. The model shows higher
performance than the baseline model, especially for sentences with more than
100 subwords. This shows that the DrugBank information is helpful to predict
DDIs when the input sentences are long and complex and it is difficult to
consider the whole contexts.

###  5.5 BioCreativeVII Track-1 DrugProt

####  5.5.1 Introduction

The DrugProt task of the BioCreative VII Track 1 is tackled with neural models
that employ external knowledge. The models are based on BERT, which shows the
state-of-the-art performance on several NLP tasks and can be considered as
external knowledge from other texts. In addition, distant supervision data
[64] is utilized and information of structure of drugs and proteins is
utilized as external knowledge from knowledge bases.

####  5.5.2 Task Definition

The DrugProt data set is a corpus that is exhaustively annotated by domain
experts, and all drug and protein mentions in PubMed articles are labeled. In
addition, for all possible drug-protein pairs, binary relationships
corresponding to the 13 types of drug-protein interactions are annotated. In
other words, when some binary relations are true, the drug-protein pair has
multiple interactions and when all binary relations are false, it indicates
that there is no interaction between the pair. The goal of the task is to
correctly predict the interaction between drug-protein pairs given the input
sentence and the mentions of drug and protein.

####  5.5.3 Methods

The model that utilizes the description and structure of the protein and drug
entities is proposed.

##### Utilizing descriptions and structures of entities

The first model utilizes the descriptions and structures of the protein and
drug entities. This model is based on the drug-drug interaction extraction
method of Asada et al. [9] and the model for drug-protein interaction
extraction has been extended. The drug and protein mentions in an input
sentence are linked to the databases DrugBank [39] and Uniprot [65],
respectively. The textual information and information of structure registered
in the database are then used for relation extraction. Figure 5.5 shows the
overview of the model.

Figure 5.5: The model with description and structure information

The mentions in the sentence and database entries are linked by relaxed string
matching. For each drug, the “description” item registered in DrugBank is used
as the description information and the “SMILES” [48] item is used as
information of structures. For each protein, the “function” item registered in
Uniprot is used as the description information and the “sequence” item is used
as the information of structures.

##### Encoding input sentences

Each preprocessed input sentence is fed into a BERT encoder and the embedding
of [CLS] token is used as the input sentence representation vector
$\textbf{h}_{input}$. $\textbf{h}_{input}$ is then taken as the input of the
fully connected layer and obtains the $d_{r}$-dimensional vector
$\textbf{h}^{fc}_{input}$, where $d_{r}$ is the number of relation labels
including the negative label. It should be noted here that although DrugProt
data set contains multiple labels for an instance, the approach cannot predict
the multiple labels correctly. However, since it is known that the number of
instances of multiple labels is extremely small in the exploratory experiment,
such a standard classification approach is conducted.

$\textbf{h}^{fc}_{input}$ is converted into the probability of possible
relations by a softmax function
$\textbf{p}_{input}=\textrm{softmax}(\textbf{h}^{fc}_{input})$. The cross-
entropy loss $L=-\sum\textbf{y}\textrm{log}\textbf{p}_{input}$ is used as the
loss function, where y is the gold type distribution. y is a one-hot vector
where the probability is 1 for the correct label and 0 otherwise.

##### Encoding description information

The descriptions registered in the database are also encoded by BERT in the
same way as the input sentence of the corpus. A separate BERT is prepared for
database descriptions. The vector $\textbf{h}_{desc\\_CLS}$ of the BERT [CLS]
token is converted into a $d_{d}$-dimensional vector $\textbf{h}_{desc}$ as
follows:

$\textbf{h}_{desc}=\textrm{GELU}(\textbf{Wh}_{desc\\_CLS}+\textbf{b}),$ (5.13)

where the GELU is an activation function and W and b are weights and bias of
the linear layer, respectively. The representations of the entity 1
description $\textbf{h}_{desc1}$ and the entity 2 description
$\textbf{h}_{desc2}$ and the input sentence $\textbf{h}_{input}$ are
concatenated. Then, the resulting vector is used as the input to the fully
connected layer:

$\textbf{h}^{fc}_{desc}=\textrm{FC}([\textbf{h}_{input};\textbf{h}_{desc1};\textbf{h}_{desc2}]),$
(5.14)

where FC is a fully-connected layer and $[;]$ denotes the vector
concatenation. $\textbf{h}^{fc}_{desc}$ is converted into the probability
$\textbf{p}_{desc}$ by a softmax function, and the model parameters are
updated by minimizing the loss function
$L=-\sum\textbf{y}\textrm{log}\textbf{p}_{desc}$.

##### Encoding information of structures

For drugs, the SMILES strings are used as the information of structures. For
proteins, the amino acid sequences are used. Both SMILES strings and amino
acid sequences are encoded by character-based CNNs.

First, the $d_{c}$-dimensional character embedding is assigned to each
character of the sequence; specifically, atoms of drugs such as ‘C’ and ‘N’,
or bonds of drugs such as ‘=’ and ‘#’, amino acid symbols of proteins such as
‘A’, ‘R’, and ‘N’. After each character of the sequence is converted to the
corresponding embedding, all character embeddings are encoded as the inputs to
CNNs with multiple convolutional window sizes [66], and max pooling is
employed to obtain the whole sequence representation. The representation of
the entity 1 structure representation $\textbf{h}_{struct1}$, the entity 2
structure representation $\textbf{h}_{struct2}$, and the input sentence
representation $\textbf{h}_{input}$ are concatenated to make the input of the
fully connected layer:

$\textbf{h}^{fc}_{struct}=\textrm{FC}([\textbf{h}_{input};\textbf{h}_{struct1};\textbf{h}_{struct2}]),$
(5.15)

$\textbf{h}^{fc}_{struct}$ is converted into the probability
$\textbf{p}_{struct}$ by a softmax function, and update the model parameters
by minimizing the cross-entropy loss.

##### Inference

Finally, description and structure information are confined using an ensemble
technique when predicting the drug-protein relation label. The final
prediction is obtained by averaging the prediction probabilities of the three
models described in the previous section as follows:

$\textbf{p}_{all}=\frac{1}{3}(\textbf{p}_{input}+\textbf{p}_{desc}+\textbf{p}_{struct}).$
(5.16)

The relation label prediction is calculated as
$\textrm{argmax}\textbf{p}_{all}$.

Method type | Method | Development | Test
---|---|---|---
|  | P | R | F1 | P | R | F1
with database information | BioBERT-Large | 0.788 | 0.746 | 0.766 | - | - | -
| +desc | 0.770 | 0.773 | 0.772 | - | - | -
| +struct | 0.759 | 0.784 | 0.771 | - | - | -
| 1-desc_struct | 0.772 | 0.778 | 0.775 | 0.749 | 0.777 | 0.763
with distant supervised data | PubMedBERT | 0.776 | 0.751 | 0.763 | - | - | -
| 4-ds_pretrain | 0.766 | 0.726 | 0.746 | 0.752 | 0.739 | 0.746
| 5-ds_pretrain_init | 0.789 | 0.739 | 0.763 | 0.720 | 0.721 | 0.721
ensemble | 2-ds_desc_struct | 0.791 | 0.761 | 0.776 | 0.767 | 0.755 | 0.761
| 3-ds_init_desc_struct | 0.780 | 0.752 | 0.766 | 0.765 | 0.746 | 0.756
Table 5.10: Micro-averaged F-scores on DrugProt development set and test set. The results on the test set are shown only for the five submitted models. Bold is the best F-score. Development | 1-de._st. | 2-ds_de._st. | 3-in._de._st. | 4-ds_pr. | 5-ds_pr._in.
---|---|---|---|---|---
ACTIVATOR | 0.754 | 0.766 | 0.748 | 0.728 | 0.748
AGONIST | 0.770 | 0.783 | 0.785 | 0.769 | 0.780
AGONIST-A. | 0.000 | 0.000 | 0.000 | 0.571 | 0.000
AGONIST-I. | 0.000 | 0.000 | 0.000 | 0.667 | 0.000
ANTAGONIST | 0.915 | 0.931 | 0.916 | 0.916 | 0.925
DIRECT-REGULATOR | 0.658 | 0.638 | 0.613 | 0.583 | 0.620
INDIRECT-D. | 0.758 | 0.772 | 0.742 | 0.747 | 0.778
INDIRECT-U. | 0.775 | 0.780 | 0.741 | 0.691 | 0.761
INHIBITOR | 0.859 | 0.850 | 0.854 | 0.844 | 0.843
PART-OF | 0.733 | 0.730 | 0.748 | 0.681 | 0.703
PRODUCT-OF | 0.602 | 0.637 | 0.603 | 0.549 | 0.611
SUBSTRATE | 0.728 | 0.726 | 0.713 | 0.690 | 0.703
SUBSTRATE_P. | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
macro-average | 0.580 | 0.585 | 0.574 | 0.649 | 0.575
micro-average | 0.775 | 0.776 | 0.766 | 0.746 | 0.763
Table 5.11: F-scores per class on DrugProt development set. AGONIST-A.,
AGONIST-I., INDIRECT-D., INDIRECT-U. and SUBSTRATE_P. stand for AGONIST-
ACTIVATOR, AGONIST-INHIBITOR, INDIRECT-DOWNREGULATOR, INDIRECT-UPREGULATOR and
SUBSTRATE_PRODUCT-OF, respectively.

####  5.5.4 Experiments

##### Models

The five models have been submitted for BioCreative VII competition:

1-desc_struct

A model using the description and structure information of protein/drug
entity.

2-ds_desc_struct

The ensemble of model 1 and 4

3-ds_init_desc_struct

The ensemble of model 1 and 5

4-ds_pretrain

A model using distant supervised data. All parameters are pre-trained on
distant supervised data.

5-ds_pretrain_init

A model using distant supervised data. Layers other than the fully connected
layer are initialized with parameters pre-trained on distant supervised data.

##### Experimental settings

For the model 1-desc_struct, BioBERT-Large [67] is used as the text encoder.
The BioBERT-Large was prepared separately for the input sentence and the
entity description, and they are fine-tuned during training. The maximum
sentence length was set to 128 for both the input sentence and the entity
description. The dimension size $d_{d}$ was set to 32.

For drugs, string matching was performed for the entry names, synonyms,
product names and brand names in DrugBank. For proteins, string matching was
performed for the entry names, recommended names, alternative names, and gene
names in Uniprot. As a result, 94% and 99% of drug and protein mentions in the
train data set matched the DrugBank and Uniprot entries. When the entity could
not be linked to database entries or the description and structure information
is not registered in the database, an empty string is used as the input of
BERT and CNNs, that is, all tokens are replaced with the padding token.

In the information of structures encoding using character CNNs, the maximum
sequence length of SMILES was set to 200 and that of amino acid sequences was
set to 1,500. For both drugs and proteins, the character embedding dimension
size $d_{c}$ was set to 100, the convolution output vector dimension size was
set to 16, and the convolution window size was set to [3,5,7]. Since three
convolution windows are used, the dimension sizes of the structure vectors
$\textbf{h}_{struct1}$ and $\textbf{h}_{struct2}$ are both $16\times 3=48$.

##### Results

The performance of proposed models is evaluated on the development set and
test set in Table 5.10. Regarding the method using the description and
structure information of the database, the F-score is improved in the
development data set in both the cases where the description information and
the structure information is used individually, compared with the baseline
BioBERT-Large model. The model 1-desc_struct, which uses both description and
structure information, further improved the F-score from baseline model.

Table 5.11 shows the F-score for each interaction type. For most of the
classes of F-scores and a micro average, 5-ds_pretrain_init showed better
performance than 4-ds_pretrain. On the other hand, for AGONIST-ACTIVATOR and
AGONIST-INHIBITOR with less training data, 4-ds_pretrain, which initializes
all weights with weights pre-trained on distantly supervised data, showed
higher performance.

###  5.6 Summary

A novel neural method for relation extraction from text using large-scale raw
text information and drug database information, especially the drug
descriptions and the drug molecular structure information, is proposed. The
results show that the large-scale raw text information with SciBERT greatly
improves the performance of DDI extraction from texts on the data set of the
DDIExtraction-2013 shared task. In addition, either the drug descriptions and
the molecular structures can further improve the performance for specific DDI
types, and their simultaneous use can improve the performance on all the DDI
types.

##  Representing Heterogeneous Knowledge
Graph

This chapter includes work from the published paper Asada et al. (2021) [10].

###  6.1 Background: An Overview of Knowledge Graph Representation

Recently, obtaining the representation of Knowledge Graph (KG) elements in a
dense vector space has attracted a lot of research attention. Major advances
in the KG representation learning model, which expresses entities and
relationships as elements of a continuous vector space, are witnessed. The
vector space embedding of all elements in KGs has received considerable
attention because it is used to create a statistical model of the whole KGs,
i.e., to easily calculate the semantic distance between all elements and to
predict the probabilities of possible relational events (i.e., edges) on the
graph. Such models can be used to infer new knowledge from known facts (i.e.,
link prediction), to clarify entities (i.e., entity resolution), to classify
triples (i.e., triple classification), and to answer the probability question
and answering [68, 69, 70, 71]. They can enhance knowledge learning
capabilities from the perspectives of knowledge reasoning, knowledge fusion,
and knowledge completion [72, 73, 74, 75].

Applications of the KG are often severely affected by data sparseness;
however, a typical large-scale KG is usually far from perfect. The task of
completing the KG aims to enrich the KGs with new facts. Many graph-based
methods have been proposed to find new facts between entities based on the
network structure of KG [76]. Much effort has also been put into extracting
relevant facts from plain text [21]. However, these approaches do not utilize
KG information. Neural-based knowledge representations have been proposed to
encode both entities and relationships in a low-dimensional space where new
facts can be found in [77, 78]. While traditional methods often deal with KGs
without node types, in many real world data, entities have different semantic
types. Recent methods deal with heterogeneous KGs with different types of
nodes [79, 80]. More importantly, neural models can be used to perform
learning of text and knowledge within a unified semantic space to more
accurately complete the KG [81].

Nowadays, there has been a lot of interest in jointly learning KG and
embedding textual information. However, traditional KG models based on
representation learning only use the information of molecular structures
embedded in a particular KG. Plain text textual information, on the other
hand, provides a wealth of semantic and contextual information that can
contribute to the clarity and completion of entity representations and
relationship representations of a given KG. Therefore, textual information can
be seen as an effective supplement to the completed task of the KG. To explore
the informative semantic signals of plain text, there has recently been a
great deal of interest in learning together the embeddings of KG and text
information in [82]. Moreover, the researchers provided a text-enhanced KG
representation model that utilized textual information to enhance the
knowledge representations [83].

KGs have attracted great attention from both academia and industry as a means
of representing structured human knowledge. Various kinds of KGs have been
proposed such as Freebase [84], YAGO [85], and WordNet [86]. A KG is a
structured representation of facts that consists of entities, relations, and
semantic descriptions. Entities are real-world objects and abstract concepts,
relations represent relationships between entities, and semantic descriptions
of entities, and these relationships include types and properties that have
well-defined meanings. The KG usually consists of a set of triples
$\\{(h,r,t)\\}$, where $h,r$, and $t$ represent the head entity, relationship,
and tail entity, respectively.

Based on the above motivation, this chapter investigates a heterogeneous
pharmaceutical knowledge graph containing textual information constructed from
several databases. The heterogeneous entity items consisting of drug, protein,
category, pathway, and Anatomical Therapeutic Chemical (ATC) code, and
relations among them, which include category, ATC, pathway, interact, target,
enzyme, carrier, and transporter, are constructed. Three methods are compared
to incorporate text information in KG embedding training with representing
text with BERT. The resulting node and edge embeddings are evaluated by the
link prediction task and the usefulness of using text information in KG
embedding training is verified. The study of KG completion is roughly divided
into two types: a study in which the link prediction task is performed by
using score functions such as TransE [78], DistMult [87], and a study in which
Graph Convolutional Networks [88] etc, are applied to the whole KG, and the
node classification task is performed. In this study, the link prediction task
is in focus and the usefulness of text information in scoring function-based
link prediction tasks is investigated.

The contributions are summarized as follows:

  * •

A heterogeneous KG with textual information (called PharmaHKG) in the drug
domain is proposed. This can be used to develop and evaluate heterogeneous
knowledge embedding methods.

  * •

Three methods to incorporate text information into KG embedding models are
proposed.

  * •

The combinations of four KG embeddings models and three methods are evaluated
and compared on the link prediction task in the proposed KG, and it is shown
that there is no single method that can perform best for different relations
and the best combination depends on the relation type.

###  6.2 Heterogeneous Pharmaceutical Knowledge Graph with Textual Information

In this section, a heterogeneous pharmaceutical KG PharmaHKG that is
constructed in this paper is first introduced. Then, the definition of KG and
the learning method of embeddings in the KG are explained. Finally, the
proposed method that effectively uses text information for KG representation
learning is explained.

Figure 6.1: Illustration of the Heterogeneous Pharmaceutical Knowledge Graph

A heterogeneous pharmaceutical KG with textual information is constructed from
DrugBank [39] and its related data sources. DrugBank is one of the rich drug
databases. It contains several different types of nodes, which can be a good
source for a heterogeneous knowledge graph. The nodes are related to several
textual information in DrugBank and their linked entries in several other data
sources such as UniProtKG [65], Small Molecule Pathway Database (SMPDB) [89]
and medical thesaurus Medical Subject Headings (MeSH). The existence of such
textual information fits the objective to evaluate the utility of textual
information in KG representation. The KG and the related data sources are
illustrated in Figure 6.1. This section first explains the nodes and relations
in the KG and then explains the textual information.

####  6.2.1 Constructing Heterogeneous Pharmaceutical Knowledge Graph

A KG consisting of five different types of heterogeneous items, i.e., drug,
protein, pathway, category, and ATC code, is constructed from different
databases and thesaurus. The number of nodes is shown in Table 6.1.

Entity type | #
---|---
Drug (DrugBank-ID) | 11,516
Protein (Uniprot-ID) | 5,339
Pathway (SMPDB-ID) | 874
Category (MESH-ID) | 2,166
ATC (ATC-code) | 1,093
Total | 20,988
Table 6.1: Statistics of heterogeneous pharmaceutical KG entities

  * •

Drug: information of drugs is extracted from DrugBank [39]. More than 10,000
drugs are registered in DrugBank, and various types of information such as
drug names, descriptions, molecular structures and experimental properties are
registered.

  * •

Protein: The information of proteins is extracted from UniProtKG [65].
UniProtKG consists of the Swiss-Prot which is manually annotated and reviewed
and TrEMBL which is automatically annotated and not reviewed, and the Swiss-
Prot knowledge base is used.

  * •

Pathway: Information of pathways from Small Molecule Pathway Database (SMPDB)
[89] is extracted. SMPDB is an interactive, visual database containing more
than 30,000 small molecule pathways found in humans.

  * •

Category: Information of drug categories is extracted from medical thesaurus
Medical Subject Headings (MeSH) [90]. Each drug recorded in DrugBank has
several hypernymy categorical classes and these classes have MeSH term ID. As
an example, a drug Morphine has categories such as Alkaloids (MeSH
ID:D000470), Anesthetics (MeSH ID:D018681), and the detailed information can
be obtained by referring to MeSH.

  * •

ATC: Anatomical Therapeutic Chemical (ATC) classification system also has
categorical information of drugs. In the ATC classification system, drugs are
divided into different groups according to the organ or system on which they
act and their therapeutic, pharmacological, and chemical properties. Drugs are
classified in groups at five different levels. The drugs are divided into
fourteen main groups (first level), with pharmacological or therapeutic
subgroups (second level). The third and fourth levels are
chemical/pharmacological/theraperutic subgroups and the fifth level is the
chemical substance. For example, the drug “Metformin” is classified into “A:
Alimentary tract and metabolism” (first level), “A10: Drugs used in diabetes”
(second level), “A10B: Blood glucose lowering drugs, excl. insulins” (third
level), “A10BA: Biguanides” (fourth level) and “A10BA02: metformin” (fifth
level).

Relation type | ALL | train | valid | test
---|---|---|---|---
category | 60,459 | 54,419 | 3,020 | 3,020
ATC | 16,341 | 14,711 | 815 | 815
pathway | 18,707 | 16,847 | 930 | 930
interact | 2,682,142 | 2,413,932 | 134,105 | 134,105
target | 18,467 | 16,627 | 920 | 920
enzyme | 5,206 | 4,686 | 260 | 260
carrier | 815 | 735 | 40 | 40
transporter | 3,093 | 2,793 | 150 | 150
Total | 2,750,228 | 2,525,829 | 140,240 | 140,240
Table 6.2: Statistics of heterogeneous pharmaceutical KG edges for each
relation type

Five different types of nodes are connected by the following eight types of
relations: category, ATC, pathway, interact, target, enzyme, carrier, and
transporter. The statistics of the KG edges for each relation type are shown
in Table 6.2. The relation triples from DrugBank are extracted.

Drug nodes and MeSH categorical terms are linked by category relation.

  * •

category: This relation type indicates the MeSH category of drugs. This
relationship indicates that the drug is classified into the therapeutic
category or the general category (anti-convulsant, antibacterial, etc.)
defined by MeSH. These relationships are registered by the manual search of
DrugBank developers.

Drug nodes and ATC classification system codes are linked by ATC relation. In
order to incorporate hierarchical information into the KG, ATC codes are
linked to ATC codes by ATChypernym relation. ATC codes are linked to the next
higher level codes with this relation. Relational triples such as A10BA -
ATChypernym - A10B, N02 - ATChypernym - N by linking the ATC code of the next
higher level are created. Since this relation is apparent from the surface
strings of ATC codes, this relation for link prediction is not considered.

  * •

ATC: Drugs are linked to any level of ATC codes with this relation. In
DrugBank, drug elements may have one or more ATC-code elements, e.g., drug
Morphine has four ATC codes (A07DA52, N02AA51, N02AA01 and N02AG01), and each
ATC-code element has child elements. All these child entities and the drug
entity are connected by the ATC relation.

Drug nodes and protein nodes are also connected with pathways.

  * •

pathway: This relation type indicates a drug or protein is included in a
pathway. When the drug is involved in metabolic, disease, and biological
pathways as identified by the SMPDB, the drug entity and the pathway entity is
connected by the pathway relation. Also, when the enzyme protein is involved
in the same pathways, the protein entity and the pathway entity are connected
by the pathway relation.

Drug nodes can be connected by a relation interact.

  * •

interact: A triple of this relation type indicates that the drug pair has a
DDI. When concomitant use of the pair of drugs will affect its activity or
result in adverse effects, these two drug entities are connected by interact
relation. These interactions may be synergistic or antagonistic depending on
the physiological effects and mechanism of action of each drug.

Drug nodes and protein nodes can be linked by target, enzyme, carrier, or
transporter relation [39].

  * •

target: A protein, macromolecule, nucleic acid, or small molecule to which a
given drug binds, resulting in an alteration of the normal function of the
bound molecule and a desirable therapeutic effect. Drug targets are most
commonly proteins such as enzymes, ion channels, and receptors.

  * •

enzyme: A protein that catalyzes chemical reactions involving a given drug
(substrate). Most drugs are metabolized by the Cytochrome P450 enzymes.

  * •

carrier: A secreted protein that binds to drugs, carrying them to cell
transporters, where they are moved into the cell. Drug carriers may be used in
drug design to increase the effectiveness of drug delivery to the target sites
of pharmacological actions.

  * •

transporter: A membrane bound protein that shuttles ions, small molecules, or
macromolecules across membranes, into cells or out of cells.

####  6.2.2 Textual Information of Knowledge Graph

Here, the text information relating to each type of node is explained.

  * •

Drug: Drugs are assigned a unique DrugBank-id. Various text information
contained in the DrugBank xml file is used. “Name”, to the heading of the drug
and standard name of the drug as provided by the drug manufacturer,
“Description”, which describes the general facts, composition and/or
preparation of the drug, “Indication” is a description or common names of
diseases that the drug is used to treat, “Pharmacodynamics” is a description
of how the drug works at a clinical or physiological level, “Mechanism of
Action” is a description of how the drug works or what it binds to at a
molecular level, “Metabolism” is a mechanism by which or organ location where
the drug is neutralized, and “Synonyms” indicates alternate drug names.

  * •

Protein: Protein targets of drug actions, enzymes that are inhibited/induced
or involved in metabolism, and carrier or transporter proteins involved in the
movement of the drug across biological membranes. Each of targets, enzymes,
carriers, transporters have unique UniProt-id. UniProt-id is referred to and
the following types of textual information, functions, miscellaneous
description, short name, alternative names, and gene names are obtained.

  * •

Pathway: Pathway relations are extracted from DrugBank. Each pathway has a
unique ID defined by The Small Molecule Pathway Database (SMPDB) [89]. “Name”
and “Description” of the pathway are registered in SMPDB.

  * •

Category: Drug categories are classified according to the medical thesaurus
MeSH. This textual information is registered in MeSH: “Name” is a definition
word, “ScopeNote” is a term description, “Entry terms” is a synonym.

  * •

ATC: Drugs are classified in a hierarchy with five different levels by WHO
drug classification system (ATC) identifiers. Each level of ATC classification
code has a name, which is defined as the international nonproprietary name