# Multiple Relations Classification using Imbalanced Predictions Adaptation

Sakher Khalil Alqaaidi School of Computing, University of Georgia. {sakher.a,
elika.bozorgi<EMAIL_ADDRESS>Elika Bozorgi11footnotemark: 1 Krzysztof J.
Kochut11footnotemark: 1

###### Abstract

The relation classification task assigns the proper semantic relation to a
pair of subject and object entities; the task plays a crucial role in various
text mining applications, such as knowledge graph construction and entities
interaction discovery in biomedical text. Current relation classification
models employ additional procedures to identify multiple relations in a single
sentence. Furthermore, they overlook the imbalanced predictions pattern. The
pattern arises from the presence of a few valid relations that need positive
labeling in a relatively large predefined relations set. We propose a multiple
relations classification model that tackles these issues through a customized
output architecture and by exploiting additional input features. Our findings
suggest that handling the imbalanced predictions leads to significant
improvements, even on a modest training design. The results demonstrate
superiority performance on benchmark datasets commonly used in relation
classification. To the best of our knowledge, this work is the first that
recognizes the imbalanced predictions within the relation classification task.

##  1 Introduction

The relation classification (RC) task aims to identify relations that capture
the dependency in every pair of entities within unstructured text. The task is
employed in several applications, such as knowledge graph construction and
completion [1] and entities interaction detection in biomedical text [2]. In
knowledge graphs, it is common to employ relational triples as the base
structure. A triple consists of a subject entity, an object entity, and a
semantic relation connecting them. For instance, Wikipedia articles rely on
Wikidata knowledge base to provide its content [3]; users can query Wikidata
in a structured format using SPARQL and retrieve the information as RDF
triples. In biomedical text, the RC task helps in discovering the interactions
between entities such as proteins, drugs, chemicals and diseases in medical
corpora.

In the supervised RC task, the objective is to learn a function that takes a
sentence and its tagged entities as input, then assigns a binary class to each
relation from a predefined set. A positive label indicates that the relation
is valid for an entity pair. Thus, the output consists of the positive
relations. We use this formal notation for the task:

(1.1) $f(W,E,P)=\left\\{\begin{array}[]{ll}R,&\textrm{Multiple relations}\\\
r,&\textrm{Single relation}\\\
\emptyset,&\textrm{otherwise}\end{array}\right.$

where $W$ is a sequence of words $[w_{1},\ w_{2}\ ...\ w_{n}]$, $E$ is the set
of one or more entity pairs. Each entity pair consists of a subject entity and
an object entity, where an entity is a sub-sequence of $W$. $P$ is the
predefined relations set. $R$ is a set of multiple relations found for $E$.
$r$ is a single relation. $\emptyset$ indicates that no relation exists
connecting any of the entities. In an example from the Nyt dataset [4] with
the sentence “Johnnie Bryan Hunt was born on Feb. 28 , 1927 , in rural Heber
Springs , in north-central Arkansas.”, the valid relations are “contains” and
“place lived” for the entity pairs (“Arkansas”, “Heber Springs”) and (“Johnnie
Bryan Hunt”, “Heber Springs”), respectively.

Usually, a sentence incorporates multiple relations. Table 1 shows the average
number of relations in two well used benchmarks [4, 5]. Therefore, a single RC
approach is only valid for limited cases. However, majority of the literature
work follow the single relation approach. Single RC models require additional
preprocessing procedure to be able to identify multiple relations [6], that is
by replicating the sentence $W$ in equation 1.1, then assigning an entity pair
and a single relation $r$ to each copy. Such approach does not only incur
additional steps but also an added training load. An additional downside is
losing the contextual information due to splitting the entities data in the
input [7, 8], which would result missed accuracy enhancements. Besides that,
several single RC models evaluate their work on highly class-imbalanced
benchmarks, such as Tacred [9] or datasets with a few predefined relations.
For instance, SemEval [10] has only six relations. Such performance
measurements make it hard to generalize to real-world scenarios. Additionally,
these models employ complicated approaches, such as attention mechanisms,
additional training and tuning efforts [11, 12]. Furthermore, most approaches
neglect the imbalanced prediction pattern in the predefined relations set,
when the model learns to predict only one relation out of many others in the
predefined set.

The multiple RC approach tackles the previously mentioned problems. However,
regular methods still unable to achieve competitive results, mainly affected
by the need to adapt to the imbalanced prediction. Despite the ability to
predict several relations, their number is relatively smaller than the
predefined relations set. This gap is shown in Table 1 when comparing the
average number of relations with the predefined set size, which indicates high
imbalanced distribution of positive and negative labels in each sentence.
Furthermore, the table shows the percentage of sentences of three or more
prediction relations, reflecting the importance of the multiple RC task.

Table 1: The number of predefined relations in the Nyt and Webnlg datasets, the average number of positive relations in each sentence, the standard deviation, and the percentage of sentences with 3 or more positive relations. Dataset | Relations | Avg. | Stdev. | 3+ Rels.
---|---|---|---|---
Nyt | 24 | 2.00 | 2.88 | 18.48%
Webnlg | 216 | 2.74 | 2.23 | 41.72%

In this paper, we propose a Multiple Relations Classification model using
Imbalanced Predictions Adaptation (MRCA). Our approach adapts to the
imbalanced predictions issue through adjusting both the output activation
function and the loss function. The utilized loss function has proved its
efficiency in several imbalanced tasks. However, our customization shows
additional enhancements within the RC task. Furthermore, we utilize the entity
features through concatenating an additional vector to the word embeddings in
the text encoder level.

The evaluation shows that our approach outperforms other models that reported
their multiple RC performances in the relation extraction task on two popular
benchmarks. To the best of our knowledge, this is the first work that
addresses the imbalanced predictions within the RC task. The ablation study
demonstrates the efficacy of our approach components in adapting to the
imbalanced predictions, and in utilizing the text and the entity features.
Furthermore, the architecture of our model has a light design that yields
astonishing performance. We make our code available
online111https://github.com/sa5r/MRCA.

##  2 Related Work

###  2.1 Single Relation Classification

Generally, RC models pursued efficient text representation to identify
relations. Early supervised approaches [13, 14] employed natural language
processing (NLP) tools to extract text features, such as word lexical
features, using dependency tree parsers [15], part-of-speech (POS) taggers and
named entity recognition. Relex [14] generated dependency parse trees and
transformed them into features for a rule-based method.

With the achievements of neural network methods, deep learning models utilized
a combination of text lexical features and word embeddings for the input [16,
17] while other approaches [12, 18, 19, 20] depended on those embeddings
solely to avoid NLP tools error propagation to later stages [18]. Neural
network-based models employed word embeddings in different ways. First,
embeddings generated from algorithms such as Word2Vec [21] using custom
training data such as in [16, 18]. Second, embeddings from pre-trained
language models (PLMs), such as Glove [22]. These PLMs were utilized in the
works including [12, 17, 19, 20]. In [12], authors presented a neural
attention mechanism with bidirectional LSTM layers without any external NLP
tools. In C-GCN [17], the dependency parser features were embedded into a
graph convolution neural network for RC. TANL [23] is a framework to solve
several structure prediction tasks in a unified way, including RC. The authors
showed that classifiers cannot benefit from extra latent knowledge in PLMs,
and run their experiments on the T5 language model.

Bert [24] is a contextualized PLM that has presented significant results in
various NLP tasks and several RC models employed it [25, 26, 27, 28]. The
earliest was R-Bert [25], where authors customized Bert for the RC task by
adding special tokens for the entity pairs. Later, Bert’s output was used as
an input for a multi-layer neural network. In [27], the traditional
classification was replaced with a span prediction approach, adopted from the
question-answering task. In [28], the model combined short dependency path
representation generated from dependency parsers with R-Bert generated
embeddings.

Figure 1: The main architecture of our model. The adaptation approach uses a
linear activation function in the output and the Dice loss extension.
Furthermore, we enhance the embeddings by adding two vectors, a character case
vector and an entity type vector denoted by the orange and blue squares.

###  2.2 Multiple Relations Classification

Methods that classify multiple relations in a single input pass vary based on
the usage of NLP tools, neural networks and PLM models. Senti-LSSVM [7] is an
SVM-based model that explained the consequences on the performance when
handling multi-relational sentences using a single relation approach.

CopyRE [5] is an end2end entity tagging and RC model that leveraged the copy
mechanism [29] and did not use a PLM. Instead the model used the training
platform’s layer to generate word embeddings. In the RC part of the model, the
authors used a single layer to make predictions over the softmax function.
Inspired by CopyRE, CopyMTL [30] is a joint entity and relation extraction
model with a seq2seq architecture. The model followed CopyRE’s approach in
representing text.

Several models employed Bert in the RC task [6, 31]. The work in [6]
elaborated on the flaws of the single relation prediction in multi-relational
sentences and presented a model that is based on customizing Bert.
Specifically, the model employed an additional prediction layer and considered
the positions of the entities in the input. In [31], authors showed that RC is
not one of the training objectives in the popular PLMs. Therefore, they
leveraged Bert and used a product matrix to relate the identified relations to
the sentence entities.

GAME model [32] used the NLP tool Spacy [33] to generate word embeddings. The
model is based on graph convolution networks for global sentence dependency
and entities interaction features. ZSLRC [34] is a zero-shot learning model
that used Glove PLM. We mention this work because it reports the supervised
learning performance in RC task.

##  3 Methodology

Our model incorporates two components, an output adaptation module and an
input utilization techniques. Between the two implementations, we employ a
light design to achieve low training parameters and better performance. We use
an average pooling layer to reduce the dimensionality of the network before
the output layer. The dropout layer is used to tackle training overfitting.
Finally, in the output layer, each unit represents a relation. Figure 1 shows
the main architecture of our model.

###  3.1 Text Encoder

We utilize Glove [22] pre-computed word embeddings to encode the input
sentences. Glove embeddings are retrieved from a key-value store where words
in lowercase are the keys for a float vectors matrix $R^{s\times d}$, where
$s$ is the vocabulary size and $d$ is the embedding dimensions. We find Glove
more convenient for the task to tackle the out-of-vocabulary (OOV) [35]
problem. Specifically, Glove’s most used
variant222https://nlp.stanford.edu/projects/glove/ has relatively large
dictionary of 400,000 words. However, the embeddings are context-free and the
keys are case insensitive. Other popular PLMs have much smaller vocabularies
but support Glove’s missed features. For instance, Bert [24] generates
contextual embeddings and has character case support. Nevertheless, the
commonly used Bert
variant333https://tfhub.dev/tensorflow/bert_en_uncased_L-12_H-768_A-12/4 has
28,997 vocabulary entries only. Thus, OOV words will get its representation
based on the latent training parameters [36]. At the same time, several
studies showed that RC is not one of the training objectives in Bert [31, 37].
Thus, we adjust Glove to provide the missed features as the following.

First, having case sensitive embeddings is essential to denote entity words in
the sentence. Realizing entities in the RC task is crucial to detect the
proper relation. Generally, a word with an uppercase first character is an
entity word. Thus, we add an additional vector to the word embeddings to
denote the first character case. For uppercase first character words we use
the value of ceiling the largest vector value in Glove. Formally, the vector
value is computed as the following:

(3.2) $v=\lceil\max_{1\leq i\leq s}{(\max_{1\leq j\leq d}{(R[i][j])})}\rceil$

where $R$ is the vectors matrix in Glove, $s$ is the vocabulary size, and $d$
is the embedding dimensions. For lowercase first character words, we use the
negative value of $v$. We employ the maximum and minimum values in the PLM to
boost distinguishing entity words from non-entity words. The orange square in
Figure 1 denotes the first character case vector.

Second, to provide contextual sentence representation, we make us of a
bidirectional long short-term memory (LSTM) as our first layer in the model
architecture.

Although we employ large vocabulary in encoding the sentence, a few words are
still not matched. Thus, generate their embeddings by combining the character
level embeddings.

Entity Features We show in equation 1.1, that the task input consists of
subject and object entities in addition to the sentence. We attempt to enrich
the input with these details by following a similar approach of appending an
additional vector from section 3.1. Specifically, we append a vector of the
value $v$ from equation 3.2 to the word representation when the input
indicates that the word is a subject entity or part of it, the negative value
of $v$ for object entity words, and 0 for non-entity words. The dense blue
square in Figure 1 denotes this additional vector. Formally the vector is
given by the function $f_{entVec}$ as the following:

(3.3) $f_{entVec}(w)=\left\\{\begin{array}[]{lll}v&,&w\in E_{sub}\\\ -1\times
v&,&w\in E_{obj}\\\ 0&,&w\notin\\{E_{sub}\cup E_{obj}\\}\end{array}\right.$

where $w$ is a word in the sentence, $E_{sub}$ is the subject entities set and
$E_{obj}$ is the object entities set. We use the negative value in the object
entity to emphasize the difference between entity types and make the relation
direction between entity pairs recognizable while training.

Figure 2: Comparison between prediction ranges in the sigmoid function and our
implementation

###  3.2 Imbalanced Predictions Adaptation

In real-world scenarios, the number of predefined relations is usually greater
than the number of positive relations in a single sentence by a big ratio.
Consider the gap in Table 1 between Webnlg relations and the average number of
valid relations in each sentence. We see that it is impractical to employ
traditional probability activation functions in neural networks (NN) for this
case. For instance, sigmoid and softmax are commonly used functions in NNs
[38]. Our claim is supported by the fact that these functions treat positive
and negative predictions equally. In other words, all probability predictions
of 0.5 or greater are considered as positive label predictions in the
mentioned functions. Thus, we improve the model’s ability to predict negative
labels by devoting 75% of the prediction range for the negative labels. We
implement this step by restricting the model’s layers output to a value within
the range of -1 and 1. We perform that through applying tanh activation
function to the first layer, then using a linear activation function in the
output layer. As a result, three quarters of the range are used for the
negative labels, i.e., all predictions between -1 and 0.5 indicate a negative
label. Figure 2 compares the prediction ranges in a probability activation
function (sigmoid) and the output of the tanh activation function.

Dice Loss Extension Traditionally, straightforward classification models
employ the cross-entropy loss functions [38], that are used to improve the
accuracy, whereas the RC task objective is to reduce the false positive and
false negative predictions. Thus, we seek improving the precision and recall
performances, i.e., enhancing the model’s f1 score. Dice Loss has shown
significant results in several domains, such as computer vision [39] and other
NLP tasks that have imbalanced data [40]. The function was designed with
inspiration of the f1 metric as the following:

(3.4)
$DiceLoss(y_{i},p_{i})=1-\frac{2p_{i}y_{i}+\gamma}{p_{i}^{2}+y_{i}^{2}+\gamma}$

where $y_{i}$ is the ground-truth label for relation $i$, $p_{i}$ is the
prediction value, and $\gamma$ is added to the nominator and the denominator
for smoothing, which has a small value of 1e-6 in our implementation.

Table 2: Loss calculations for ground truth $y$ and the prediction value $p$ in Dice loss and in our implementation. The underlined numbers are the unconventional values in Dice loss  $y$ | $p$ | Expected loss | Dice loss | RC Dice loss
---|---|---|---|---
0 | 1 |  $\geq$ 1 | 0.9 | 0.9
0 | 0.1 |  $\approx$ 0 | 0.9 | 9e-13
0 | -0.1 |  $\approx$ 0 | 0.9 | 9e-13
0 | -1 | 0 | 0.9 | 9e-13
1 | 1 | 0 | 0 | 0
1 | 0 |  $\geq$ 1 | 0.9 | 0.9
1 | -1 | >1 | 1.9 | 1.9

Utilizing Dice Loss in our adapted predictions may incur unconventional
behaviour. Specifically, when having negative ground truth labels and negative
value predictions at the same time. Such case would result high loss when
using Dice Loss, whereas low loss is the natural result. Our analysis in Table
2 shows the invalid loss values and the expected ones. Therefore, we expand
our adaptation by implementing an extension for Dice Loss. Specifically, we
address the negative prediction case by computing the loss from a division
operation; the nominator is the squared smoothing value; the denominator is
the regular Dice loss denominator. Raising the smoothing value to the second
power is necessary to present a small loss value. Our corrected loss value
examples can be observed in Table 2. We call this extension, RC_DiceLoss and
formally define as the following:

(3.5) $\begin{split}&RC\\_DiceLoss(y_{i},p_{i})=\\\
&\left\\{\begin{array}[]{lll}\frac{\gamma^{2}}{p_{i}^{2}+y_{i}^{2}+\gamma}&,&y_{i}=0\>and\>p_{i}\textless
0.5\\\ \\\
1-\frac{2p_{i}y_{i}+\gamma}{p_{i}^{2}+y_{i}^{2}+\gamma}&,&otherwise\\\
\end{array}\right.\end{split}$

##  4 Experiments

###  4.1 Datasets and Experimental Setup

To demonstrate the generalization and the applicability of our model, we
evaluated it on diverse and widely used datasets. The Nyt dataset [4] was
generated from a large New York Times articles corpus, where each input item
consisted of a sentence and a set of triples, each triple is composed of
subject and object entities, and a relation. Webnlg dataset was originally
generated for the Natural Language Generation (NLG) task, CopyRE [5]
customized the dataset for the triples and relations extraction tasks. Table 3
shows the statistics and the splits of the datasets.

Our model achieved the best results using Glove PLM. The language model has
been trained on 6 Billion tokens with a 400,000 words vocabulary and 300
dimensional word embeddings. Nevertheless, the experiments demonstrated that
our model can adopt other PLMs and still provide competitive results. We
performed the experiments using TensorFlow. Our model’s hyper-parameters and
training settings are the unified for both experimental datasets, which
confirms the applicability of our approach to real-world data. Table 4 shows
the training settings and the model hyper-parameters. We used Adam optimizer
for stochastic gradient descent, and performed the training for five times on
every dataset with different random seed and reported the mean performance and
the standard deviation. Although we implement the training for 50 epochs, the
mean convergence epoch for the Nyt dataset was 21.4. The hyper-parameters were
chosen based on tuning the model for best performance. We ran the experiments
on a server with NVIDIA A100-SXM-80GB GPU device and AMD EPYC MILAN (3rd gen)
processor, but using only 8 cores. We used only 20GB of the available main
memory for the Webnlg dataset experiments and 100GB for the Nyt dataset due to
its size. We conducted an ablation study to test our model’s components using
different variants as shown in Section 4.4.

Table 3: Statistics of the evaluation datasets. Dataset |  | Nyt | Webnlg
---|---|---|---
Relations |  | 24 | 216
Samples | Training | 56,196 | 5,019
Validation | 5,000 | 500
Testing | 5,000 | 703
Total | 66,196 | 6,222
Table 4: Model hyperparameters and training settings. Parameter |  | Value
---|---|---
Average Pooling | Pool Size | 80
Strides | 2
Learning | Rate | 0.0015
Decay | 3e-5
Bi-LSTM units |  | $2\times 500$
Dropout rate |  | 0.15
Sequence padding |  | 100
Epochs |  | 50
Early stopping patience |  | 5
Batch size |  | 32
Generated parameters |  | 13M
Average epoch time |  | 2355ms

###  4.2 Comparison Baselines

We compare our results with the following supervised models. We refer to the
main characteristics of each one in section 2. CopyRE [5] and CopyMTL [30] are
based on the copy mechanism and used the same approach to generate word
embeddings. Both evaluated their work on the Nyt and Webnlg datasets. GAME
model [32] used Spacy to generated word embeddings and reported their results
on the Nyt dataset.

Other multiple relations classification models were not considered in the
comparison due to their utilization of a different release of the Nyt dataset,
such as [31] and ZSLRC [34]. We found that the used release is not commonly
used in the literature.

Table 5: Our models F1 score on the Nyt dataset compared with the baseline models. Model | GAME | CopyRE | CopyMTL | MRCA
---|---|---|---|---
F1 | 77.1 | 87.0 | 86.9 | 96.650.17
Table 6: Our models F1 score on the Webnlg dataset compared with the baseline models. Model | CopyRE | CopyMTL | MRCA
---|---|---|---
F1 | 75.1 | 79.7 | 93.350.29
Table 7: The performance of our model’s variants on the Webnlg dataset. Model | Precision | Recall | F1
---|---|---|---
MRCA | 95.40.25 | 91.30.48 | 93.350.29
MRCA-Sigmoid-BCE | 93.350.31 | 88.730.55 | 90.880.3
MRCA-Bert | 94.50.2 | 89.90.49 | 92.150.26
MRCA-Bert-noLSTM | 55.182.21 | 53.71.1 | 54.41.16

###  4.3 Main Results and Analysis

We report our average F1 scores in Table 5 and Table 6 for the Nyt and Webnlg
datasets, respectively. Additionally, we visualize the training performance in
Figure 3. The results show superiority among the baseline models. We report
the precision and recall scores in Table 7. We highlight our results in the
Webnlg dataset, as we find that relation predictions in that dataset is highly
imbalanced due to the large number of predefined relations. Furthermore, the
dataset has smaller training data. Nevertheless, the Webnlg’s F1 score is
close to the Nyt’s score. Knowing that, the Nyt dataset has much smaller
predefined relations and more training data, which indicates that our
adaptation method supported achieving better predictions despite the
imbalanced distribution of the binary labels.

###  4.4 Ablation Study

To examine the effectiveness of our model’s components, we evaluate the
imbalanced predictions adaptation approach, and the text encoder adjustments.
We design different variants of our model and perform training using the same
main evaluation settings in Table 4. Moreover, We report the average score of
five runs and the standard deviation. We use the Webnlg dataset for the
ablation study experiments. We report the performances in Table 7, then we
present the following analysis.

Imbalanced Predictions Adaptation Effectiveness To evaluate the contribution
of our imbalanced predictions adaptation approach, we assess our model using
different activation and loss functions. Specifically, we use the traditional
sigmoid activation function and the binary cross entropy loss function. We
report this variant’s performance in Table 7 with the name MRCA-Sigmoid-BCE.
The variant’s F1 score is approximately 3% less than our model’s score, which
is an average value between the precision scores difference and the recall
scores difference. Noting that the recall gap is larger, which presents the
first indication that the adaptation approach improved predicting negative
labels.

Encoder effectiveness To evaluate our text encoder adjustments, we need to
consider two sub-components in the assessments, that are the usage of Glove
language model and the addition of the entity type vector to the embeddings.
Thus, we test the following variants of our model. MRCA-Bert is a variant the
uses Bert PLM instead of Glove and MRCA-Bert-noLSTM is a variant that uses
Bert but with no LSTM layers. We use Bert’s
release444https://tfhub.dev/tensorflow/bert_en_cased_L-12_H-768_A-12/4 with
character case support since we added the same case feature in our
implementation. In the former variant, there is a slight difference between
the reported F1 score and our model’ score, which demonstrates less
contribution of the Glove employment in our overall performance. However,
using Glove, our model still outperforms the Bert variant due to the better
OOV terms support. Noting that Bert is known as a language model with
contextual text representation support. Thus, the assumption is that, the LSTM
layers would not affect Bert’s performance. Nonetheless, in the second variant
MRCA-Bert-noLSTM, the performance is way worst. This result supports our claim
that RC is not one of Bert’s training objectives in section 3.1 because of the
abstract usage of Bert. Furthermore, with a weak contextual representation in
Bert, OOV words will split into non-meaningful tokens as described in the
tokenization algorithm that is used in Bert [41]. This concludes the
importance of using a language model with larger vocabulary.

##  5 Conclusion

We propose MRCA, a multiple relations classification model that aims at
improving the imbalanced predictions. Our light-design implementation
leverages wider prediction range for negative labels and customize a
remarkable loss function for the same purpose. Furthermore, text and entity
features are utilized efficiently to improve the relations prediction. The
experiments presented superiority among state-of-the-art models that reported
the relation classification performance. Assessing our model’s components
showed that addressing the imbalanced predictions yields significant
improvement in the relation classification task. Furthermore, representing
sentences using language models with rich vocabularies provides performance
enhancements in the relation classification task.

Figure 3: The validation F1 score during training for the evaluation datasets.
(a) indicates the Nyt training performance. (b) indicates the Webnlg training
performance.

##  6 Future Work and Limitations

Although the relation classification task has limited applications as a single
module, it has wider usages in the relation extraction task. Therefore, we see
that our approach can be adopted to achieve new scores in several applications
that utilize the relation classification task. Further improvements can be
achieved when using NLP tools for lexical and syntactic text features.
Additionally, it would be typical to advance our model to assign the predicted
relation to the corresponding entities pair in the input. However, this
approach cannot be considered as an ideal way for the relation or triple
extraction task because errors in the entities tagging step would propagate to
the relation classification task. Finally, our imbalanced predictions
adaptation promises enhancements if used by similar tasks of imbalanced
classes.

Our evaluation was limited by the small number of models that reported the
relation classification performance. However, the results proved our model’s
superiority, denoted by the gap between our F1 score and the closest model.

## References

  * [1] Zhe Chen et al.  “Knowledge graph completion: A review”  In _Ieee Access_ 8 IEEE, 2020, pp. 192435–192456
  * [2] Markus Bundschus et al.  “Extraction of semantic biomedical relations from text using conditional random fields”  In _BMC bioinformatics_ 9.1 BioMed Central, 2008, pp. 1–14
  * [3] Denny Vrandečić and Markus Krötzsch  “Wikidata: a free collaborative knowledgebase”  In _Communications of the ACM_ 57.10 ACM New York, NY, USA, 2014, pp. 78–85
  * [4] Sebastian Riedel, Limin Yao and Andrew McCallum  “Modeling relations and their mentions without labeled text”  In _ECML PKDD_ , 2010, pp. 148–163  Springer
  * [5] Xiangrong Zeng et al.  “Extracting relational facts by an end-to-end neural model with copy mechanism”  In _ACL_ , 2018, pp. 506–514
  * [6] Haoyu Wang et al.  “Extracting Multiple-Relations in One-Pass with Pre-Trained Transformers”  In _ACL_ Association for Computational Linguistics, 2019, pp. 1371–1377
  * [7] Lizhen Qu et al.  “Senti-lssvm: Sentiment-oriented multi-relation extraction with latent structural svm”  In _Transactions of the Association for Computational Linguistics_ 2 MIT Press One Rogers Street, Cambridge, MA 02142-1209, USA journals-info …, 2014, pp. 155–168
  * [8] Xiaoxin Yin, Jiawei Han, Jiong Yang and Philip S Yu  “Efficient classification across multiple database relations: A crossmine approach”  In _IEEE Transactions on Knowledge and Data Engineering_ 18.6 IEEE, 2006, pp. 770–783
  * [9] Yuhao Zhang et al.  “Position-aware Attention and Supervised Data Improve Slot Filling”  In _EMNLP_ , 2017, pp. 35–45
  * [10] Iris Hendrickx et al.  “SemEval-2010 Task 8: Multi-Way Classification of Semantic Relations between Pairs of Nominals”  In _Proceedings of the 5th International Workshop on Semantic Evaluation_ Association for Computational Linguistics, 2010, pp. 33–38
  * [11] Linlin Wang, Zhu Cao, Gerard De Melo and Zhiyuan Liu  “Relation classification via multi-level attention cnns”  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , 2016, pp. 1298–1307
  * [12] Peng Zhou et al.  “Attention-based bidirectional long short-term memory networks for relation classification”  In _Proceedings of the 54th annual meeting of the association for computational linguistics (volume 2: Short papers)_ , 2016, pp. 207–212
  * [13] Mengqiu Wang  “A re-examination of dependency path kernels for relation extraction”  In _Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-II_ , 2008
  * [14] Katrin Fundel, Robert Küffner and Ralf Zimmer  “RelEx—Relation extraction using dependency parse trees”  In _Bioinformatics_ 23.3 Oxford University Press, 2007, pp. 365–371
  * [15] Dan Klein and Christopher D Manning  “Fast exact inference with a factored model for natural language parsing”  In _Advances in neural information processing systems_ 15, 2002
  * [16] Matthew R Gormley, Mo Yu and Mark Dredze  “Improved relation extraction with feature-rich compositional embedding models”  In _arXiv preprint arXiv:1505.02419_ , 2015
  * [17] Yuhao Zhang, Peng Qi and Christopher D Manning  “Graph convolution over pruned dependency trees improves relation extraction”  In _arXiv preprint arXiv:1809.10185_ , 2018
  * [18] Daojian Zeng et al.  “Relation classification via convolutional deep neural network”  In _COLING_ , 2014, pp. 2335–2344
  * [19] Joohong Lee, Sangwoo Seo and Yong Suk Choi  “Semantic relation classification via bidirectional lstm networks with entity-aware attention using latent entity typing”  In _Symmetry_ 11.6 MDPI, 2019, pp. 785
  * [20] Haijie Ding and Xiaolong Xu  “Relation Classification based on Selective Entity-Aware Attention”  In _CSCWD_ , 2022, pp. 177–182  IEEE
  * [21] Tomas Mikolov, Kai Chen, Greg Corrado and Jeffrey Dean  “Efficient estimation of word representations in vector space”  In _arXiv preprint arXiv:1301.3781_ , 2013
  * [22] Jeffrey Pennington, Richard Socher and Christopher D Manning  “Glove: Global vectors for word representation”  In _EMNLP_ , 2014, pp. 1532–1543
  * [23] Giovanni Paolini et al.  “Structured prediction as translation between augmented natural languages”  In _arXiv preprint arXiv:2101.05779_ , 2021
  * [24] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “Bert: Pre-training of deep bidirectional transformers for language understanding”  In _arXiv preprint arXiv:1810.04805_ , 2018
  * [25] Shanchan Wu and Yifan He  “Enriching pre-trained language model with entity information for relation classification”  In _CIKM_ , 2019, pp. 2361–2364
  * [26] Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling and Tom Kwiatkowski  “Matching the Blanks: Distributional Similarity for Relation Learning”  In _ACL_ Association for Computational Linguistics, 2019, pp. 2895–2905
  * [27] Amir DN Cohen, Shachar Rosenman and Yoav Goldberg  “Relation classification as two-way span-prediction”  In _arXiv preprint arXiv:2010.04829_ , 2020
  * [28] Haluk Alper Karaevli and Tunga Güngör  “Enhancing Relation Extraction by Using Shortest Dependency Paths Between Entities with Pre-trained Language Models”  In _INISTA_ , 2022, pp. 1–7
  * [29] Jiatao Gu, Zhengdong Lu, Hang Li and Victor OK Li  “Incorporating copying mechanism in sequence-to-sequence learning”  In _arXiv preprint arXiv:1603.06393_ , 2016
  * [30] Daojian Zeng, Haoran Zhang and Qianying Liu  “Copymtl: Copy mechanism for joint extraction of entities and relations with multi-task learning”  In _AAAI_ 34.05, 2020, pp. 9507–9514
  * [31] Cheng Li and Ye Tian  “Downstream model design of pre-trained language model for relation extraction task”  In _arXiv preprint arXiv:2004.03786_ , 2020
  * [32] Harry Cheng, Lizi Liao, Linmei Hu and Liqiang Nie  “Multi-Relation Extraction via A Global-Local Graph Convolutional Network”  In _IEEE Transactions on Big Data_ 8.6 IEEE, 2022, pp. 1716–1728
  * [33] Matthew Honnibal and Ines Montani  “spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing”  In _To appear_ 7.1, 2017, pp. 411–420
  * [34] Jiaying Gong and Hoda Eldardiry  “Zero-shot relation classification from side information”  In _CIKM_ , 2021, pp. 576–585
  * [35] Philip C Woodland, Sue E Johnson, Pierre Jourlin and K Spärck Jones  “Effects of out of vocabulary words in spoken document retrieval”  In _Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval_ , 2000, pp. 372–374
  * [36] Anmol Nayak, Hariprasad Timmapathini, Karthikeyan Ponnalagu and Vijendran Gopalan Venkoparao  “Domain adaptation challenges of BERT in tokenization and sub-word representations of out-of-vocabulary words”  In _Proceedings of the First Workshop on Insights from Negative Results in NLP_ , 2020, pp. 1–5
  * [37] Yinhan Liu et al.  “Roberta: A robustly optimized bert pretraining approach”  In _arXiv preprint arXiv:1907.11692_ , 2019
  * [38] Francois Chollet  “Deep learning with Python”  SimonSchuster, 2021, pp. 11460
  * [39] Qing Huang et al.  “Robust liver vessel extraction using 3D U-Net with variant dice loss function”  In _Computers in biology and medicine_ 101 Elsevier, 2018, pp. 153–162
  * [40] Xiaoya Li et al.  “Dice Loss for Data-imbalanced NLP Tasks”  In _ACL_ Association for Computational Linguistics, 2020, pp. 465–476
  * [41] Xinying Song et al.  “Fast WordPiece Tokenization”  In _EMNLP_ , 2021, pp. 2089–2103