# Analysis of skin lesion images with deep learning

Josef Steppan, Sten Hanke J. Steppan was with the Department of eHealth at FH
Joanneum University of Applied Sciences, Alte Poststrasse 149, 8020 Graz,
AUSTRIA. e-mail<EMAIL_ADDRESS>Hanke is with the
Department of eHealth at FH Joanneum University of Applied Sciences, Alte
Poststrasse 149, 8020 Graz. e-mail<EMAIL_ADDRESS>
###### Abstract

Skin cancer is the most common cancer worldwide, with melanoma being the
deadliest form. Dermoscopy is a skin imaging modality that has shown an
improvement in the diagnosis of skin cancer compared to visual examination
without support. We evaluate the current state of the art in the
classification of dermoscopic images based on the ISIC-2019 Challenge for the
classification of skin lesions and current literature. Various deep neural
network architectures pre-trained on the ImageNet data set are adapted to a
combined training data set comprised of publicly available dermoscopic and
clinical images of skin lesions using transfer learning and model fine-tuning.
The performance and applicability of these models for the detection of eight
classes of skin lesions are examined. Real-time data augmentation, which uses
random rotation, translation, shear, and zoom within specified bounds is used
to increase the number of available training samples. Model predictions are
multiplied by inverse class frequencies and normalized to better approximate
actual probability distributions. Overall prediction accuracy is further
increased by using the arithmetic mean of the predictions of several
independently trained models. The best single model has been published as a
web service. The source code is publicly available at
http://github.com/j05t/lesion-analysis

###### Index Terms:

Lesion, Skin, Melanoma, Deep Learning

##  I Introduction

Skin cancer is the most common cancer worldwide, with melanoma being the
deadliest form. A later stage in the diagnosis of melanoma is associated with
a strong influence on melanoma mortality within 5 years of diagnosis [1].
Early detection of melanoma can significantly reduce both morbidity and
mortality [2]. The risk of dying from the disease is directly related to the
depth of the cancer, which is directly related to the time it has been
growing. Self-examination of the skin by patients, full-body skin examinations
by a doctor, and patient education are the keys to early detection. Self-
examiners are generally diagnosed with thinner melanomas than non-self-
examiners (0.77 mm versus 0.95 mm) [3].

This paper evaluates the current state of the art in the classification of
dermoscopic images based on the ISIC-2019 Challenge for the classification of
skin lesions and current literature. Since medical image data sets often show
a class imbalance, several approaches for the training of deep neural networks
on imbalanced data sets have been reviewed. Because the training of deep
neural networks requires a large amount of training data, further publicly
available dermoscopic as well as clinical image data sets of skin lesions have
been evaluated for expanding the ISIC-2019 training data set. Since the
heterogeneity of the image data of the ISIC data set requires preprocessing, a
suitable approach towards preprocessing, as well as the effects of
preprocessing on the achieved accuracy of trained networks have been
investigated. Furthermore, the potential of real-time data augmentation to
increase the number of available training patterns during training and to
improve the prediction accuracy at inference time has been investigated.
Current ensembling strategies and an overview of current architectures of deep
neural networks for the classification of image content have been reviewed.

##  II Image Classification

Convolutional Neural Networks (CNNs) [4] are currently state of the art in
image classification and have been exceeding the recognition rate of human
experts in the ImageNet Large Scale Visual Recognition
Challenge111http://image-net.org/challenges/LSVRC/2017 (ILSVRC) [5] since 2015
[6]. The ILSVRC evaluates algorithms for object recognition and image
classification on a large scale. An important motivation is to enable
researchers to compare progress in recognition for a wider variety of objects.
Another motivation is to measure the progress of computer vision algorithms
for classifying images on a large scale. The ImageNet training data set
contains 1.000 categories and 1,2 million images. Image classification
algorithms are compared using a test data set of 150.000 images in 1.000
categories. Highest accuracy rates are currently achieved with the
architectures SENet [7] 154 (81.3% top-1 accuracy), PNASNet-5 Large [8]
(82.9%), AmoebaNet-C [9, 10] (83.9%) and EfficientNet-B7 [11] (84.4%) [12].
Algorithms for classifying image content are constantly being improved. Deep
learning has shown enormous potential in this area due to the constantly
increasing amounts of data [13, 14]. Some deep learning approaches outperform
teams of certified dermatologists in the detection of melanoma in dermoscopic
images [15, 16, 17] or achieve equivalent detection rates [18, 19].

##  III Skin Lesion Datasets

###  III-A ISIC-2019

To make specialist knowledge more widely available, the International Skin
Imaging Collaboration developed the ISIC archive, an international repository
for dermoscopic images, both for clinical training purposes and to support
technical research on automated algorithmic analysis by hosting the ISIC
Challenges. The training data set of the ISIC-2019 Challenge consists of
several dermoscopic image databases: BCN_20000 [20] with dermoscopic images of
the most common classes of skin lesions: actinic keratosis, squamous cell
carcinoma, basal cell carcinoma, seborrheic keratosis, solar lentigo, and
dermatological lesions. The HAM10000 dataset [21], with 600x450 images
centered and cropped on lesions. The MSK data set [22] with images of
different resolutions. A total of 25,331 images are available for training in
8 different categories. The test data set consists of 8,238 images whose
labels are not publicly available. Also, the test data set contains an
additional outlier class that is not contained in the training data and must
be identified by developed systems. Predictions on the ISIC-2019 test data set
are assessed by an automatic evaluation system. The goal of the ISIC-2019
Challenge222https://challenge2019.isic-archive.com/ is to classify dermoscopic
images among nine different diagnostic categories:

  1. 1.

Melanoma (MEL)

  2. 2.

Melanocytic nevus (NV)

  3. 3.

Basal cell carcinoma (BCC)

  4. 4.

Actinic Keratosis (AK)

  5. 5.

Benign keratosis (solar lentigo / seborrheic keratosis / lichen planus-like
keratosis) (BKL)

  6. 6.

Dermatofibroma (DF)

  7. 7.

Vascular Lesion (VASC)

  8. 8.

Squamous cell carcinoma (SCC)

  9. 9.

None of the others (UNK)

###  III-B PH2 database

The PH2 database [23] includes manual segmentation, clinical diagnosis, and
the identification of multiple dermoscopic structures performed by experienced
dermatologists in a set of 200 dermoscopic images. The images were obtained in
the dermatology department of the Pedro Hispano hospital (Matosinhos,
Portugal) under the same conditions by the Tuebinger Mole Analyzer System
using 20-fold magnification. These are 8-bit RGB color images with a
resolution of 768x560 pixels. The image database contains a total of 200
dermoscopic images of melanocytic lesions, including 80 common nevi, 80
atypical nevi, and 40 melanomas. The PH2 database contains a medical
annotation of all images, namely a medical segmentation of the lesion, a
clinical and histological diagnosis as well as the evaluation of several
dermoscopic criteria (colors; pigment network; dots/spheres; stripes;
regression areas; blue-whitish haze). The database was made freely available
for research and benchmarking
purposes333https://www.fc.up.pt/addi/ph2%20database.html.

###  III-C Light Field Image Dataset of Skin Lesions

Faria et al. [24] present a contribution to the research community in the form
of the publicly available data set of skin lesions, the ”Light Field Image
Dataset of Skin Lesions” (SKINL2)444https://www.it.pt/AutomaticPage?id=3459.
The dataset contains 250 light fields [25], which were recorded with a focused
plenoptic camera and divided into eight clinical categories depending on the
type of lesion. Each light field consists of 81 different views of the same
lesion. The database also contains the dermoscopic image of each lesion. The
data set offers great potential the further development of medical imaging
research and the development of new classification algorithms based on light
fields as well as for clinically oriented dermatological studies; however,
only dermoscopic images contained in the data set are taken into account for
this work.

###  III-D SD-198

In contrast to dermoscopic images with largely constant lighting and low image
disturbances, clinical images are often created with a large number of
different image recording devices, such as digital cameras or smartphones. The
SD-198 data set [26] contains 6,584 clinical images from 198 classes, which
vary according to scale, color, shape, and structure. The SD-198 benchmark
data set is intended to stimulate further research into the visual
classification of skin diseases. The authors also carry out an extensive
analysis of this data set using modern methods including CNNs. The ground
truth labels of the images were created via
DermQuest555https://www.dermquest.com, with each image being examined by
qualified experts and labeled with the name of its class. To ensure the
quality of the labels, two experts were also invited to check the data set.

###  III-E 7-point criteria evaluation database

Kawahara et al. [27] provide a database for evaluating the computerized image-
based prediction of the 7-point checklist for malignant skin
lesions666https://derm.cs.sfu.ca/Welcome.html. The seven-point checklist,
published in 1998, is one of the best-validated dermoscopic algorithms due to
its high sensitivity and specificity, even when used by non-specialists. The
seven criteria were originally applied to 342 melanocytic lesions (117
melanomas and 225 atypical nevi) tested and selected for their frequent
association with melanoma [28]. Three of them were defined as the main
criteria (atypical network, blue-white haze, and atypical vascular pattern),
while the remaining four were considered minor (irregular stripes, irregular
spots or globules, irregular spots, and regression structures) [29]. The data
set contains over 2000 clinical and dermoscopic color images as well as
corresponding structured metadata that are tailored to the training and
evaluation of CAD (Computer Aided Diagnostic) systems.

###  III-F MED-NODE

The MED-NODE data set [30] consists of 70 melanoma and 100 nevus images from
the digital image archive of the Department of Dermatology at the University
Hospital Groningen (UMCG), which is used for the development and testing of
the MED-NODE Decision Support System for the detection of Skin cancer using
macroscopic images. The system proposed by the authors achieves results with a
diagnostic accuracy of 81%. The final classification was achieved by a
majority vote of the predictions of several models. The dataset is publicly
available777http://www.cs.rug.nl/~imaging/databases/melanoma\\_naevi/.

##  IV Combined training dataset

A combined training data set has been created from all the data sets described
in section III. 32,748 images are available for training in total. Images from
SD-198 were used exclusively for the creation of training data for the ”UNK”
class, after prior removal of image data from the eight categories of the
ISIC-2019 training data set. The combined data set is still heavily imbalanced
(Figure 1).

Figure 1: Combined training data set from the data sets ISIC-2019, PH2, Light
Field Image Dataset of Skin Lesions, SD-198, the 7-point criteria evaluation
database, and MED-NODE. The ”UNK” category is mainly formed from data from the
SD-198 dataset. The combined data set is divided into a training (90%) and
validation data set (10%), so 29.469 images are available for training and
3.279 images for assessing the generalizability of the predictions and for
adapting hyperparameters in the validation data set. The ISIC-2019 test data
set consists of 8.238 images whose labels are not publicly available. The test
data set is not used for training or parameter adjustment.

##  V Methodology

###  V-A Preprocessing

Training and test data of the ISIC-2019 dataset have been preprocessed to
remove black areas surrounding dermoscopic images, and subsequently rescaled
maintaining aspect ratio (Figure 2). Descriptive text appended to images in
the SD-198 dataset has been removed.

Figure 2: Preprocessing of the ISIC 2019 dataset. Black image borders are
detected and removed. The top row shows images of the original training data
set, shown below are preprocessed images

###  V-B Data Augmentation

To avoid overfitting [31] in neural networks, dropout [32] is often used.
Another simple method for regularization (and expansion of the number of
different training samples) of CNNs is data augmentation. During training,
input data is changed randomly according to certain criteria (translation,
rotation, scaling, etc.). Additionally, Cutout [33] has been used for
regularization. Figure 3 shows the applied augmentations.

Figure 3: Applied augmentations for a single training image. Random rotation,
translation in the x and y directions as well as scaling within defined limits
avoid overfitting on the training data and enable a better generalization of
the model. Used augmentation parameters are: max_rotate=45, p_affine=0.5,
do_flip=True, flip_vert=True, max_zoom=1.05, max_lighting=0.2,
crop_pad(input_size), cutout(n_holes=(1,1), length=(16,16), p=.5).

###  V-C Out of Distribution Detection

Neural networks offer little or no guarantee of reliable prediction when
applied to data that was not generated through the same process that was used
to create the network’s training data. With such Out-of-Distribution (OOD)
inputs, the prediction may not only be incorrect but also associated with a
high level of confidence [34, 35] of the network, which restricts the
reliability of deep learning classifiers in real world applications. Often the
predictions of (ensembles of) classifiers that have been trained on data
within the distribution are examined for the presence of OOD inputs using
statistical methods [36, 37]. Alternatively, the input distribution can be
modeled directly by using generative models that do not require the presence
of class labels. However, it has been shown that this method can also output
higher probabilities on OOD inputs than on inputs within the distribution
[38]. In the ISIC 2019 Challenge, classes that are not included in the
training data set should be detected as OOD and recognized as class ”UNK”. In
this work, a data-driven approach to the recognition of OOD inputs is pursued
by using images (mostly from SD-198, see subsection III-D) as training data
for the ”UNK” class that are not labeled as one of the classes of the
ISIC-2019 training data set. However, this approach is far from optimal, and
OOD detection in deep learning classifiers remaining an unsolved problem.
Further work is needed to improve classifier performance regarding OOD
detection.

###  V-D Dataset Imbalance

A common problem with deep learning-based applications is the fact that some
classes have a significantly higher number of samples in the training set than
other classes. This difference is known as class imbalance. There are many
examples in areas such as computer vision [39, 40, 41, 42, 43], medical
diagnosis [44, 45], fraud detection [46], and others [47, 48, 49] where this
problem is highly significant and the incidence of one class (e.g. cancer) can
be 1000 times less than another class (e.g. healthy patient) [50]. It has been
shown that a class imbalance in training data sets can have a significant
adverse effect on the training of traditional classifiers [51], including
classic neural networks or multilayer perceptrons [52]. The class imbalance
influences both the convergence of neural networks during the training phase
and the generalization of a model to real or test data [50].

####  V-D1 Undersampling / Oversampling

Undersampling and oversampling in data analysis are techniques to adjust the
class distribution of a data set (i.e. the relationship between the different
classes/categories represented). These terms are used in statistical sampling,
survey design methodology, and machine learning. The goal of undersampling and
oversampling is to create a balanced data set. Many machine learning
techniques, such as neural networks create more reliable predictions when
trained on balanced data. Oversampling is generally used more often than
undersampling. The reasons for using undersampling are mainly practical and
often resource-dependent. With random oversampling, the training data is
supplemented by multiple copies of samples from minority classes. This is one
of the earliest methods proposed that has also proven robust [53]. Instead of
duplicating minority class samples, some of them can be chosen at random by
substitution. Other methods of handling unbalanced data sets such as synthetic
oversampling [54] are more suitable for traditional machine learning tasks
[55] and were therefore not considered any further in this work.

####  V-D2 Weighted Cross-Entropy Loss

Weighted cross-entropy [56] is useful for training neural networks on
unbalanced data sets. [57] suggest adding a margin-based loss value to the
cross-entropy on in-distribution training patterns in order to ensure a
minimum difference in average entropy between in-distribution and out-of-
distribution data. This ensemble-based method is intended to surpass previous
methods of recognizing out-of-distribution inputs such as ODIN [58]. Cross
entropy can be described as

$L(x,y)=-log\left(\frac{\exp(x[y])}{\sum_{j}\exp(x[j])}\right)=-x[y]+log\left(\underset{j}{\sum}\exp(x[j])\right)$

or, by using class weights:

$L(x,y)=W[y]\left(-x[y]+log\left(\underset{j}{\sum}\exp(x[j])\right)\right)$

The arithmetic mean of the loss values achieved is calculated for each mini-
batch. A weight vector can be calculated using effective class weights [59]
with the simple formula $(1-\beta^{n})/(1-\beta)$, with the hyperparameter
beta equal to 0.999 (a choice of the parameter beta equal to zero would not
apply any weighting and a choice of beta equal to 1 would correspond to
weighting by the inverse class frequency). In the simplest case, loss values
can be weighted by multiplying by inverse class frequencies.

####  V-D3 Thresholding

Also referred to as threshold shifting or rescaling, thresholding adapts the
decision threshold of a classifier. This method is used at inference time and
involves changing the output class probabilities. There are several ways in
which the network outputs can be rescaled. In general, an optimization
algorithm can be used to configure the network to minimize any criteria [60].
The simplest method only compensates for a priori class probabilities [61]. It
has been shown that neural networks estimate Bayesian a posteriori
probabilities [61]. That is, for a given data point x, the output for class c
is implicitly $y_{i}(x)=p(c|x)=\frac{p(c)p(x|c)}{p(x)}$. The actual
probabilities of class membership can therefore be calculated by dividing the
output of the network by the estimated a priori probability
$p(c)=\frac{|c|}{\sum_{k}|k|}$, where $|c|$ is the number of samples of class
$c$ [50]. The resulting class probabilities are normalized after thresholding
is applied. This simple method of handling an existing class imbalance can
significantly increase the class probability distribution approximation made
by classifiers.

###  V-E Transfer Learning

Transfer learning in the context of machine learning is a technique that uses
information obtained from solving a problem and applies it to a similar
problem. When using transfer learning, a model that has already been trained
on another data set is adapted to custom data. Ideally, the pre-trained model
has been trained on similar data, but this is not strictly necessary. The
final layers of the network are removed and replaced by output layers
featuring appropriate dimensions. The model is then trained on custom data. By
using transfer learning, the time required for training a network can be
greatly reduced [62, 63, 64]. The existing pre-trained model thus serves as a
feature extractor, which forwards features such as edges, texture, position of
recognized objects, etc. to the last layer for classification. A softmax
function (normalized exponential function) transforms the network output into
a vector of numbers between zero and one which sum up to one which allows
interpreting the output of the network as a probability distribution.

###  V-F Test Time Augmentation

Data augmentation is a technique widely used to improve neural network
training performance and reduce generalization errors. The same image data
augmentation technique can also be used at inference time to allow the model
to make predictions for several different versions of each image in the test
data. Test Time Augmentation (TTA) predictions are formed by calculating the
average of the regular predictions (with a weighting of beta=0.4) with the
average of the predictions obtained by predicting on augmented versions of the
image data (with a weighting of 1-beta). The transformations specified for the
training set are applied with the following changes: Scaling with a factor of
1.05 controls the scaling for the zoom (which is not random for TTA).
Furthermore, the cropping is not random to ensure that the four corners of the
picture are used. Reflection is not random but is applied once to each of
these corner images (so that a total of 8 augmented versions are created).

###  V-G Ensembling

Ensembling is the use of several independently trained models to form an
overall prediction. The basic idea of ensembling is that individual models
have weaknesses in different areas, which are compensated by the combination
with predictions of other independently trained models. Possible ensembling
strategies are e.g. majority voting, the use of a weighted average based on
classifier confidences, or simply using the arithmetic mean of several
predictions of different models and model architectures [65].

##  VI Experiments

The CNN architectures Inception-ResNet-v2 [66], SE-ResNeXt-101 (32x4d) [7],
NASNet-A-Large [8], EfficientNet-B4 and EfficientNet-B5 [11] pre-trained on
the ImageNet data set were adapted for the task of classifying the nine
classes of the ISIC-2019 Challenge by replacing final layers with a custom
linear layer to output nine class probabilities. Real-time data augmentation
has been used to improve the generalizability of the resulting models. Models
have been trained on an NVIDIA GTX 1070 GPU. Batch sizes (number of training
samples that are used for a single forward pass) were adapted to individual
architectures and input sizes to achieve optimal utilization of the available
video memory. Images have been resized to fit model input sizes prior
training.

Models have been trained via transfer learning over 32 epochs followed by
model fine-tuning using differential learning rates until convergence using
One Cycle Policy [67], allowing very rapid convergence rates of trained
networks [68]. Appropriate learning rates were determined manually at regular
intervals. The use of a weighted loss function has, contrary to expectations,
only proven to be advantageous for training the NASNet-A-Large architecture,
which has been unable to converge without applying weighted loss. Other
architectures could not benefit from training using a weighted loss function.
Early stopping has been applied to avoid model overfitting. Best models have
been selected based on their performance on the validation data. Out-of-
distribution detection using thresholding proved to provide inferior results
to using a data-driven approach as described in V-C.

The unsatisfactory balanced multiclass accuracy of the NASNet model may be
caused by the relatively small batch size, which was limited to four due to
the size of the model. As expected, improved performance of deep neural
networks in the classification of ImageNet data can be directly translated to
models trained on custom data sets. Improved CNN architectures, which achieve
higher accuracy in the classification of the ImageNet data set, thus also
provide better results in the classification of dermoscopic images.

A rescaling of the outputs of the models by multiplying the output
probabilities by inverse class frequency have proven to be advantageous for
the balanced multiclass accuracy of the network predictions in all cases where
no weighted loss function has been used. Applying rescaling on models trained
using a weighted loss function did not improve balanced multiclass prediction
accuracy. The outputs of several independently trained models were combined
into an overall prediction using the arithmetic mean of all model predictions
and transmitted to the automated evaluation system of the ISIC-2019 Challenge.

Table I shows results for individual models. Best performing models were used
to form ensemble predictions. NASNet-A-Large was not included in the ensemble
due to the unsatisfactory overall accuracy achieved. Although EfficientNet
shows the best results of all trained network architectures, the combination
with predictions from SE-ResNeXt-101 (32x4d) and Inception-ResNet-v2 models
still lead to higher average accuracy than any single model could achieve
independently.

TABLE I: Single Model, Ensemble Balanced Accuracy Architecture | Accuracy
---|---
EfficientNet-B5 | 0.600
SE-ResNeXt-101(32x4d) | 0.582
EfficientNet-B4 | 0.577
Inception-ResNet-v2 | 0.569
NASNet-A-Large | 0.504
Ensemble (excluding NasNet) | 0.634
TABLE II: Metrics (Ensemble) Category | Mean |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---
Metrics | Value | MEL | NV | BCC | AK | BKL | DF | VASC | SCC | UNK
AUC | .902 | .924 | .957 | .942 | .917 | .893 | .977 | .932 | .936 | .638
AUC, Sens>80% | .813 | .853 | .926 | .883 | .829 | .776 | .966 | .868 | .876 | .336
Avg. Precision | .561 | .766 | .923 | .719 | .366 | .572 | .586 | .502 | .326 | .285
Accuracy | .923 | .899 | .894 | .908 | .933 | .933 | .983 | .978 | .969 | .808
Sensitivity | .525 | .581 | .752 | .666 | .580 | .384 | .744 | .614 | .408 | .00
Specificity | .973 | .963 | .962 | .944 | .952 | .985 | .986 | .983 | .982 | 1.00
Dice Coeff | .491 | .659 | .821 | .654 | .468 | .499 | .523 | .434 | .364 |  .00
PPV | .609 | .760 | .905 | .642 | .392 | .713 | .404 | .335 | .328 | 1.00
NPV | .941 | .919 | .890 | .950 | .977 | .944 | .997 | .995 | .987 | .808
Figure 4: ROC curve for the 0.634 balanced multiclass accuracy ensemble. The
ROC curve shows the diagnostic capability of a binary classifier as its
decision threshold varies. The ROC curve is constructed by plotting the true
positive rate (TPR) against the false positive rate (FPR) at various threshold
settings. The true positive rate is also referred to as sensitivity, recall or
detection probability, whereas FPR corresponds to the false positive rate (1 -
specificity).

Table II shows metrics for the ensemble with 0.634 balanced multiclass
accuracy, as computed by the ISIC challenge website. AUC: Area under the
receiver operating characteristic (ROC) curve. AUC, Sens >80%: area under the
ROC curve, evaluated exclusively for the region in which the sensitivity is
greater than 80%. Average precision (precision is also called Positive
Predictive Value - PPV) measures the area under the interpolated precision-
recall curve (recall = sensitivity). Accuracy measures the overall accuracy of
the classifier, i.e.
$Accuracy=sensitivity*prevalence+specificity*(1-prevalence)$. Sensitivity
measures true-positive predictions, specificity (recall) measures true-
negative predictions of the classifier. The F1 score (Dice Coefficient) is the
harmonious mean of precision and recall, with an F1 score reaching its best
value at 1 (perfect precision and recall). F1 score is also known as the
Sørensen-Dice coefficient or Dice similarity coefficient (DSC). A positive
predictive value (PPV) is the likelihood that subjects who test positive will
actually have the disease. A negative predictive value (NPV) is the likelihood
that subjects who test negative really do not have the disease. Figure 4 shows
the receiver operating characteristic curve for the ensemble.

##  VII Conclusion

Deep learning has become a mature technology for the classification of image
content and can achieve similar or superior accuracy as human experts in the
classification of skin lesions. The use of deep learning applications that
automatically evaluate clinical and dermoscopic images and classify skin
lesions offer great potential for improving and implementing prevention and
screening measures and increasing their efficiency. One of the main criticisms
of deep learning applications, that these networks have to be treated as a
black box and that there is no easy explanation of how they form their
decisions remain unchanged despite some progress in the visualization of
network activations. Careful validation of trained models using real-world
data sets before and also during use is essential. Progress in the development
of more efficient architectures of deep neural networks and improved accuracy
in the classification of images with high image quality does not automatically
mean that results can be transferred to real-world applications. For instance,
[69] examined the use of a classification system created by Google researchers
to detect diabetic retinopathy in 11 clinics in Thailand and found that this
technology does not yet work well in practice despite all the research
advances. Advantages of deep learning applications in the medical field are
the rapid availability of diagnosis compared to analysis by human specialists
and cost-effective provisioning of models for large numbers of simultaneous
users. Central provisioning of deep learning models allows uncomplicated and
transparent delivery of improved models without having to make changes to
client software. Cloud applications can serve current deep learning models
cost-effectively through automatic horizontal scaling of active services and
flexible price calculations. Also, deep learning applications can help nursing
staff to better argument their own assessments to specialists and to
prioritize urgent cases accordingly. Even if decisions made by deep learning
models still have to be manually verified by human experts, automated image
classifiers can support these human experts and reduce the workload by
accelerating decision making processes, therefore contributing to more
efficient utilization of the resources of health systems.

## References

  * [1] K. J. Wernli, N. B. Henrikson, C. C. Morrison, M. Nguyen, G. Pocobelli, and P. R. Blasi, “Screening for skin cancer in adults: updated evidence report and systematic review for the us preventive services task force,” _Jama_ , vol. 316, no. 4, pp. 436–447, 2016.
  * [2] L. F. di Ruffano, Y. Takwoingi, J. Dinnes, N. Chuchu, S. E. Bayliss, C. Davenport, R. N. Matin, K. Godfrey, C. O’Sullivan, A. Gulati _et al._ , “Computer-assisted diagnosis techniques (dermoscopy and spectroscopy-based) for diagnosing skin cancer in adults,” _Cochrane Database of Systematic Reviews_ , no. 12, 2018.
  * [3] P. Carli, V. De Giorgi, D. Palli, A. Maurichi, P. Mulas, C. Orlandi, G. L. Imberti, I. Stanganelli, P. Soma, D. Dioguardi _et al._ , “Dermatologist detection and skin self-examination are associated with thinner melanomas: results from a survey of the italian multidisciplinary group on melanoma,” _Archives of dermatology_ , vol. 139, no. 5, pp. 607–612, 2003.
  * [4] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _Advances in neural information processing systems_ , 2012, pp. 1097–1105.
  * [5] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein _et al._ , “Imagenet large scale visual recognition challenge,” _International journal of computer vision_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [6] C. Langlotz, B. Allen, B. Erickson, J. Kalpathy-Cramer, K. Bigelow, T. Cook, A. Flanders, M. Lungren, D. Mendelson, J. Rudie, G. Wang, and K. Kandarpa, “A roadmap for foundational research on artificial intelligence in medical imaging: From the 2018 nih/rsna/acr/the academy workshop,” _Radiology_ , vol. 291, p. 190613, 04 2019.
  * [7] J. Hu, L. Shen, and G. Sun, “Squeeze-and-excitation networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 7132–7141.
  * [8] C. Liu, B. Zoph, J. Shlens, W. Hua, L. Li, L. Fei-Fei, A. L. Yuille, J. Huang, and K. Murphy, “Progressive neural architecture search,” _CoRR_ , vol. abs/1712.00559, 2017. [Online]. Available: http://arxiv.org/abs/1712.00559
  * [9] C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua, L.-J. Li, L. Fei-Fei, A. Yuille, J. Huang, and K. Murphy, “Progressive neural architecture search,” in _The European Conference on Computer Vision (ECCV)_ , September 2018.
  * [10] E. Real, A. Aggarwal, Y. Huang, and Q. V. Le, “Regularized evolution for image classifier architecture search,” in _Proceedings of the aaai conference on artificial intelligence_ , vol. 33, 2019, pp. 4780–4789.
  * [11] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” _CoRR_ , vol. abs/1905.11946, 2019. [Online]. Available: http://arxiv.org/abs/1905.11946
  * [12] S. Bianco, R. Cadene, L. Celona, and P. Napoletano, “Benchmark analysis of representative deep neural network architectures,” _IEEE Access_ , vol. 6, pp. 64 270–64 277, 2018.
  * [13] X. Cui, R. Wei, L. Gong, R. Qi, Z. Zhao, H. Chen, K. Song, A. A. Abdulrahman, Y. Wang, J. Z. Chen _et al._ , “Assessing the effectiveness of artificial intelligence methods for melanoma: A retrospective review,” _Journal of the American Academy of Dermatology_ , vol. 81, no. 5, pp. 1176–1180, 2019.
  * [14] Y. Fujisawa, S. Inoue, and Y. Nakamura, “The possibility of deep learning-based, computer-aided skin tumor classifiers,” _Frontiers in Medicine_ , vol. 6, p. 191, 2019.
  * [15] A. Hekler, J. S. Utikal, A. H. Enk, A. Hauschild, M. Weichenthal, R. C. Maron, C. Berking, S. Haferkamp, J. Klode, D. Schadendorf _et al._ , “Superior skin cancer classification by the combination of human and artificial intelligence,” _European Journal of Cancer_ , vol. 120, pp. 114–121, 2019\.
  * [16] R. C. Maron, M. Weichenthal, J. S. Utikal, A. Hekler, C. Berking, A. Hauschild, A. H. Enk, S. Haferkamp, J. Klode, D. Schadendorf _et al._ , “Systematic outperformance of 112 dermatologists in multiclass skin cancer image classification by convolutional neural networks,” _European Journal of Cancer_ , vol. 119, pp. 57–65, 2019.
  * [17] T. J. Brinker, A. Hekler, A. H. Enk, J. Klode, A. Hauschild, C. Berking, B. Schilling, S. Haferkamp, D. Schadendorf, T. Holland-Letz _et al._ , “Deep learning outperformed 136 of 157 dermatologists in a head-to-head dermoscopic melanoma image classification task,” _European Journal of Cancer_ , vol. 113, pp. 47–54, 2019.
  * [18] A. Blum, H. Luedtke, U. Ellwanger, R. Schwabe, G. Rassner, and C. Garbe, “Digital image analysis for diagnosis of cutaneous melanoma. development of a highly effective computer algorithm based on analysis of 837 melanocytic lesions,” _British Journal of Dermatology_ , vol. 151, no. 5, pp. 1029–1038, 2004.
  * [19] M. Zortea, T. R. Schopf, K. Thon, M. Geilhufe, K. Hindberg, H. Kirchesch, K. Møllersen, J. Schulz, S. O. Skrøvseth, and F. Godtliebsen, “Performance of a dermoscopy-based computer vision system for the diagnosis of pigmented skin lesions compared with visual evaluation by experienced dermatologists,” _Artificial intelligence in medicine_ , vol. 60, no. 1, pp. 13–26, 2014.
  * [20] M. Combalia, N. C. Codella, V. Rotemberg, B. Helba, V. Vilaplana, O. Reiter, A. C. Halpern, S. Puig, and J. Malvehy, “Bcn20000: Dermoscopic lesions in the wild,” _arXiv preprint arXiv:1908.02288_ , 2019.
  * [21] P. Tschandl, C. Rosendahl, and H. Kittler, “The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions,” _Scientific data_ , vol. 5, p. 180161, 2018.
  * [22] N. C. Codella, D. Gutman, M. E. Celebi, B. Helba, M. A. Marchetti, S. W. Dusza, A. Kalloo, K. Liopyris, N. Mishra, H. Kittler _et al._ , “Skin lesion analysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic),” in _2018 IEEE 15th International Symposium on Biomedical Imaging (ISBI 2018)_. IEEE, 2018, pp. 168–172.
  * [23] T. Mendonça, P. M. Ferreira, J. S. Marques, A. R. Marcal, and J. Rozeira, “Ph 2-a dermoscopic image database for research and benchmarking,” in _2013 35th annual international conference of the IEEE engineering in medicine and biology society (EMBC)_. IEEE, 2013, pp. 5437–5440.
  * [24] S. M. de Faria, J. N. Filipe, P. M. Pereira, L. M. Tavora, P. A. Assuncao, M. O. Santos, R. Fonseca-Pinto, F. Santiago, V. Dominguez, and M. Henrique, “Light field image dataset of skin lesions,” in _2019 41st Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_. IEEE, 2019, pp. 3905–3908.
  * [25] G. Wu, B. Masia, A. Jarabo, Y. Zhang, L. Wang, Q. Dai, T. Chai, and Y. Liu, “Light field image processing: An overview,” _IEEE Journal of Selected Topics in Signal Processing_ , vol. 11, no. 7, pp. 926–954, 2017.
  * [26] X. Sun, J. Yang, M. Sun, and K. Wang, “A benchmark for automatic visual classification of clinical skin disease images,” in _European Conference on Computer Vision_. Springer, 2016, pp. 206–222.
  * [27] J. Kawahara, S. Daneshvar, G. Argenziano, and G. Hamarneh, “Seven-point checklist and skin lesion classification using multitask multimodal neural nets,” _IEEE Journal of Biomedical and Health Informatics_ , vol. 23, no. 2, pp. 538–546, 2019.
  * [28] G. Argenziano, G. Fabbrocini, P. Carli, V. De Giorgi, E. Sammarco, and M. Delfino, “Epiluminescence microscopy for the diagnosis of doubtful melanocytic skin lesions: comparison of the abcd rule of dermatoscopy and a new 7-point checklist based on pattern analysis,” _Archives of dermatology_ , vol. 134, no. 12, pp. 1563–1570, 1998.
  * [29] H. Kittler, A. A. Marghoob, G. Argenziano, C. Carrera, C. Curiel-Lewandrowski, R. Hofmann-Wellenhof, J. Malvehy, S. Menzies, S. Puig, H. Rabinovitz _et al._ , “Standardization of terminology in dermoscopy/dermatoscopy: results of the third consensus conference of the international society of dermoscopy,” _Journal of the American Academy of Dermatology_ , vol. 74, no. 6, pp. 1093–1106, 2016.
  * [30] I. Giotis, N. Molders, S. Land, M. Biehl, M. F. Jonkman, and N. Petkov, “Med-node: a computer-assisted melanoma diagnosis system using non-dermoscopic images,” _Expert systems with applications_ , vol. 42, no. 19, pp. 6578–6585, 2015.
  * [31] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov, “Improving neural networks by preventing co-adaptation of feature detectors,” _arXiv preprint arXiv:1207.0580_ , 2012. [Online]. Available: http://arxiv.org/abs/1207.0580
  * [32] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting.” _Journal of machine learning research_ , vol. 15, no. 1, pp. 1929–1958, 2014\.
  * [33] T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” _arXiv preprint arXiv:1708.04552_ , 2017.
  * [34] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” _arXiv preprint arXiv:1412.6572_ , 2014.
  * [35] A. Nguyen, J. Yosinski, and J. Clune, “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 427–436.
  * [36] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” _arXiv preprint arXiv:1610.02136_ , 2016.
  * [37] B. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in _Advances in neural information processing systems_ , 2017, pp. 6402–6413.
  * [38] J. Ren, P. J. Liu, E. Fertig, J. Snoek, R. Poplin, M. Depristo, J. Dillon, and B. Lakshminarayanan, “Likelihood ratios for out-of-distribution detection,” in _Advances in Neural Information Processing Systems_ , 2019, pp. 14 680–14 691.
  * [39] G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie, “The inaturalist species classification and detection dataset,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 8769–8778.
  * [40] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, and A. Torralba, “Sun database: Large-scale scene recognition from abbey to zoo,” in _2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition_. IEEE, 2010, pp. 3485–3492.
  * [41] B. A. Johnson, R. Tateishi, and N. T. Hoan, “A hybrid pansharpening approach and multiscale object-based image analysis for mapping diseased pine and oak trees,” _International journal of remote sensing_ , vol. 34, no. 20, pp. 6969–6982, 2013.
  * [42] M. Kubat, R. C. Holte, and S. Matwin, “Machine learning for the detection of oil spills in satellite radar images,” _Machine learning_ , vol. 30, no. 2-3, pp. 195–215, 1998.
  * [43] O. Beijbom, P. J. Edmunds, D. I. Kline, B. G. Mitchell, and D. Kriegman, “Automated annotation of coral reef survey images,” in _2012 IEEE Conference on Computer Vision and Pattern Recognition_. IEEE, 2012, pp. 1170–1177.
  * [44] J. W. Grzymala-Busse, L. K. Goodwin, W. J. Grzymala-Busse, and X. Zheng, “An approach to imbalanced data sets based on changing rule strength,” in _Rough-neural computing_. Springer, 2004, pp. 543–553.
  * [45] B. Mac Namee, P. Cunningham, S. Byrne, and O. I. Corrigan, “The problem of bias in training data in regression problems in medical decision support,” _Artificial intelligence in medicine_ , vol. 24, no. 1, pp. 51–70, 2002.
  * [46] K. Philip and S. Chan, “Toward scalable learning with non-uniform class and cost distributions: A case study in credit card fraud detection,” in _Proceeding of the Fourth International Conference on Knowledge Discovery and Data Mining_ , 1998, pp. 164–168.
  * [47] P. Radivojac, N. V. Chawla, A. K. Dunker, and Z. Obradovic, “Classification and knowledge discovery in protein databases,” _Journal of Biomedical Informatics_ , vol. 37, no. 4, pp. 224–239, 2004.
  * [48] C. Cardie and N. Nowe, “Improving minority class prediction using case-specific feature weights,” in _Proceedings of the Fourteenth International Conference on Machine Learning_ , ser. ICML ’97. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 1997, p. 57–65.
  * [49] G. Haixiang, L. Yijing, J. Shang, G. Mingyun, H. Yuanyue, and G. Bing, “Learning from class-imbalanced data: Review of methods and applications,” _Expert Systems with Applications_ , vol. 73, pp. 220–239, 2017.
  * [50] M. Buda, A. Maki, and M. A. Mazurowski, “A systematic study of the class imbalance problem in convolutional neural networks,” _Neural Networks_ , vol. 106, pp. 249–259, 2018.
  * [51] N. Japkowicz and S. Stephen, “The class imbalance problem: A systematic study,” _Intelligent data analysis_ , vol. 6, no. 5, pp. 429–449, 2002.
  * [52] M. A. Mazurowski, P. A. Habas, J. M. Zurada, J. Y. Lo, J. A. Baker, and G. D. Tourassi, “Training neural network classifiers for medical decision making: The effects of imbalanced datasets on classification performance,” _Neural networks_ , vol. 21, no. 2-3, pp. 427–436, 2008.
  * [53] C. X. Ling and C. Li, “Data mining for direct marketing: Problems and solutions.” in _Kdd_ , vol. 98, 1998, pp. 73–79.
  * [54] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer, “Smote: synthetic minority over-sampling technique,” _Journal of artificial intelligence research_ , vol. 16, pp. 321–357, 2002.
  * [55] A. Fernández, S. Garcia, F. Herrera, and N. V. Chawla, “Smote for learning from imbalanced data: progress and challenges, marking the 15-year anniversary,” _Journal of artificial intelligence research_ , vol. 61, pp. 863–905, 2018.
  * [56] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [57] A. Vyas, N. Jammalamadaka, X. Zhu, D. Das, B. Kaul, and T. L. Willke, “Out-of-distribution detection using an ensemble of self supervised leave-out classifiers,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 550–564.
  * [58] S. Liang, Y. Li, and R. Srikant, “Enhancing the reliability of out-of-distribution image detection in neural networks,” _arXiv preprint arXiv:1706.02690_ , 2017.
  * [59] Y. Cui, M. Jia, T.-Y. Lin, Y. Song, and S. Belongie, “Class-balanced loss based on effective number of samples,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 9268–9277.
  * [60] S. Lawrence, I. Burns, A. Back, A. C. Tsoi, and C. L. Giles, “Neural network classification and prior class probabilities,” in _Neural networks: tricks of the trade_. Springer, 1998, pp. 299–313.
  * [61] M. D. Richard and R. P. Lippmann, “Neural network classifiers estimate bayesian a posteriori probabilities,” _Neural computation_ , vol. 3, no. 4, pp. 461–483, 1991.
  * [62] S. J. Pan, J. T. Kwok, and Q. Yang, “Transfer learning via dimensionality reduction.” in _AAAI_ , vol. 8, 2008, pp. 677–682.
  * [63] S. J. Pan, Q. Yang _et al._ , “A survey on transfer learning,” _IEEE Transactions on knowledge and data engineering_ , vol. 22, no. 10, pp. 1345–1359, 2010.
  * [64] S. Hoo-Chang, H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, J. Yao, D. Mollura, and R. M. Summers, “Deep convolutional neural networks for computer-aided detection: Cnn architectures, dataset characteristics and transfer learning,” _IEEE transactions on medical imaging_ , vol. 35, no. 5, p. 1285, 2016.
  * [65] K. Kowsari, M. Heidarysafa, D. E. Brown, K. J. Meimandi, and L. E. Barnes, “Rmdl: Random multimodel deep learning for classification,” in _Proceedings of the 2nd International Conference on Information System and Data Mining_ , 2018, pp. 19–28.
  * [66] C. Szegedy, S. Ioffe, and V. Vanhoucke, “Inception-v4, inception-resnet and the impact of residual connections on learning,” _CoRR_ , vol. abs/1602.07261, 2016. [Online]. Available: http://arxiv.org/abs/1602.07261
  * [67] L. N. Smith, “A disciplined approach to neural network hyper-parameters: Part 1–learning rate, batch size, momentum, and weight decay,” _arXiv preprint arXiv:1803.09820_ , 2018.
  * [68] L. N. Smith and N. Topin, “Super-convergence: Very fast training of neural networks using large learning rates,” in _Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications_ , vol. 11006. International Society for Optics and Photonics, 2019, p. 1100612.
  * [69] E. Beede, E. Baylor, F. Hersch, A. Iurchenko, L. Wilcox, P. Ruamviboonsuk, and L. M. Vardoulakis, “A human-centered evaluation of a deep learning system deployed in clinics for the detection of diabetic retinopathy,” in _Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems_ , 2020, pp. 1–12.


# BioLORD: Learning Ontological Representations from Definitions
for Biomedical Concepts and their Textual Descriptions

François Remy  Kris Demuynck  Thomas Demeester
The Internet and Data Science Lab (IDLab)
Ghent University (UGent) - Imec Belgium
<EMAIL_ADDRESS>

###### Abstract

This work introduces BioLORD, a new pre- training strategy for producing
meaningful rep- resentations for clinical sentences and bio- medical concepts.
State-of-the-art methodolo- gies operate by maximizing the similarity in
representation of names referring to the same concept, and preventing collapse
through con- trastive learning. However, because biomedical names are not
always self-explanatory, it some- times results in non-semantic
representations. BioLORD overcomes this issue by ground- ing concept
representations using definitions, as well as short descriptions derived from
a multi-relational knowledge graph consisting of biomedical ontologies. Thanks
to this ground- ing, our model produces more semantic concept representations
that match more closely the hi- erarchical structure of ontologies. BioLORD
establishes a new state of the art for text simi- larity on both clinical
sentences (MedSTS) and biomedical concepts (MayoSRS).

Figure 1: BioLORD aims to bring the representation of biomedical concept names
($\odot$) and their definitions ($\otimes$) closer to each other, to ground
the name representations with knowledge from the definitions. This is
illustrated for the Ranitidine and Aspirin concepts from UMLS. Knowledge from
the ontology’s relational knowledge graph is injected by extending the set of
known definitions with automatically generated descriptions ($\oplus$). Each
such description pairs a more generic concept with one relationship (of the
described concept) and its related concept, thereby setting the described
concept apart from the more generic one. Contrastive learning is applied to
attract the representations of compatible pairs ($\odot$, $\otimes$ or
$\oplus$) and repel incompatible ones (obtained as in-batch negatives).

##  1 Introduction

Natural language processing models are well positioned to support healthcare
providers by automatically extracting and synthesizing relevant information
from clinical notes. For this, we introduce BioLORD, a pre-training strategy
for end-to-end biomedical information extraction, capable of producing
meaningful representations for biomedical terms and clinical sentences
simultaneously.

This is achieved through the continued pre-training of an existing sentence
embedding model, using contrastive learning and pairs consisting of the names
and definitions of a given biomedical concept (see Fig. 1). This design choice
proved crucial for the effectiveness of BioLORD, as it enables the transfer of
knowledge from the definitions to the representation of biomedical names,
thereby overcoming limitations of existing works (see §2.3) through a more
effective usage of the knowledge contained in biomedical ontologies (see
§2.1).

Indeed, to improve coverage and diversity, we supplemented definitions with
textual descriptions generated from the numerous concept-to-concept
relationships contained in biomedical ontologies.

Our key contributions are 1 a versatile training strategy using dictionaries
and knowledge graphs to create highly semantic representations for the key
phrases of a domain, 2 an associated BioLORD model trained on the biomedical
domain, 3 an extensive evaluation (§4) demonstrating its ability to provide
semantic representations usable in a broad range of information extraction
scenarios, including a new state of the art for Biomedical Concept
Representation and Clinical Sentence Similarity, and 4 an in-depth analysis of
the strengths and weaknesses of our proposed approach (§5).

##  2 Related Work

Let us first consider how prior works attempted to address the biomedical
domain’s usage of a large, specialized, and often opaque vocabulary (e.g.,
PAPA syndrome111a hereditary inflammatory disorder affecting the skin or cat
scratch disease222a bacterial skin infection caused by Bartonella Henselae).

###  2.1 Biomedical ontologies

To condense this lexical knowledge in digital form, medical practitioners
developed semi-structured concept hierarchies called biomedical ontologies,
merging a dictionary and a knowledge graph.

SnomedCT (Systematized Nomenclature of Medicine and Clinical Terms) is one
such ontology covering around 700k medical concepts in total and a small set
of important relationships between these concepts (Schulz and Klein, 2008).

UMLS (Unified Medical Language System) bridges several biomedical ontologies
to cover more than 4 million concepts, each with on average 4 listed names
(Bodenreider, 2004). UMLS also contains around 90 million labeled concept-to-
concept relationships of 900 different types.

###  2.2 Contrastive Learning Strategies

On the machine learning side, efforts in the tasks of named entity recognition
(NER) and normalization (NEL) are strongly influenced by the challenges posed
by such a large and specialized vocabulary. In recent years, approaches using
ontologies through string-based pattern matching, such as MetaMap Aronson
(2001), have been consistently outperformed by newer works relying on
constrative learning with Transformers.

BioSyn (Sung et al., 2020) was the first model to introduce the idea of
contrastive learning to produce embeddings of biomedical concepts. It takes
existing NEL benchmarks and proposes to use their training sets in a
contrastive manner. An encoder model initialized with BioBERT (Lee et al.,
2020) is trained to produce embeddings for batches of concept names (grouped
by pairs referring to the same concept). A contrastive loss is then applied to
ensure that the embeddings of synonyms are significantly closer to each other
than they are to the other names in the batch, which refer to other concepts.
After pre-training, the model can be finetuned for the end task of NEL using
cross-entropy training.

SapBERT (Liu et al., 2021) was the first large-scale contrastive model to
leverage UMLS. Just like BioSyn, it produces embeddings for biomedical concept
names, without considering the context they are used in. But, unlike BioSyn,
it is based on PubMedBERT (Gu et al., 2020) and uses the synonyms defined for
concepts in UMLS to form the training pairs. This enables the model to
contrast millions of entries, many more than BioSyn.

BIOCOM (Ujiie et al., 2021) and KRISSBERT (Zhang et al., 2021) independently
extended this approach in a similar way, by noting the need for context-based
disambiguation for some entities. For each UMLS concept, sentences mentioning
the concept are collected from PubMed articles. These sentences are used as
context during training.

###  2.3 Challenges with existing models

BIOCOM and KRISSBERT propose to disambiguate mentions of biomedical concepts
using contextual information. Ambiguous notations requiring context to
disambiguate can indeed be found in clinical notes. However, using these
contextual models for inference is only possible after identifying text spans
denoting such concepts in the input text. This requires introducing a mention
detection model, which comes with its own challenges and errors. Worse,
reducing mentions to text spans is not always possible, as concepts are
sometimes alluded to in a diffused way (see Fig. 2).

Figure 2: Concept mapping sometimes requires considering the entire sentence,
rather than mentions.  Figure 3: In SapBERT’s latent space, none of the
nearest neighbors of "apyrexial" (i.e. fever-free) happen to share the word’s
meaning. Instead, the alpha-privative was over-indexed by the model, among
other biases.

However, models which do not use in-context mentions usually learn
representations of lower quality than in-context models. By pairing synonyms
with a significant word or token overlap with each other, these models isolate
concepts containing rare words or tokens early in the training, in a way that
is rarely semantic (see Fig. 3). Indeed, the training loss of contrastive
models only requires placing all mentions of a particular concept close to
each other, but it does not provide strong guarantees about the relative
location of different but similar concepts in the latent space.

While hierarchical relationships from medical ontologies have sometimes been
used to produce more meaningful concept embeddings (Zhang et al., 2021), this
is however not sufficient to overcome the issues stated above, because
relatedness is not always possible to encode hierarchically.

##  3 Pre-training methodology

To produce representations of biomedical concepts that overcome the
limitations described above, we modified the way the positive pairs are
constructed. Like the prior works cited in §2.2, we start by establishing a
list of names for each UMLS concept. However, unlike previous works, we do not
use these names directly to form positive pairs. Instead, we construct pairs
formed with, on the one side, a randomly selected name for a given concept
and, on the other side, a definition or description for that concept (see Fig.
1).

We hypothesize that a definition or description of a given concept provides a
more robust semantic anchor for this concept than another of its names. As
mentioned before, names in the medical domain can be quite opaque, and do not
always offer useful insights into what exactly is being referred to. By
inducing representational similarity between a concept name and its known
definitions, we aim to distill their respective knowledge into the
representations of the concept names themselves. This key idea influenced some
design choices for our experimental setup, including the choice of the data
curation process, model initialization, and training procedure (as described
in this section).

###  3.1 Curating definitions and descriptions

Around 5% of the concepts found in UMLS are clarified by one or more
definitions. These definitions aim to provide the most relevant pieces of
information about a given concept to the practitioners reading them, and we
can therefore include them directly in our training set (see Fig. 1).

This is however insufficient, since most concepts have no matching definition
in UMLS. Additionally, definitions might not always cover all the relevant
aspects of a given concept, and the particular aspects they cover vary from
one concept to another. Consequently, pairing concept names and their
definitions, alone, cannot be expected to produce satisfactory results for all
UMLS concepts.

We therefore supplement the definitions already available in UMLS with
automatically generated textual descriptions, based on the structured
information contained in the ontology and its 90M concept-to-concept
relationships.

These concept descriptions are constructed using the following template:
“[more-generic-concept] which [has-relationship-with] [related-concept]” (e.g.
"drug which may treat headache").

The replacement for “[more-generic-concept]” is randomly sampled among the
known names of the ancestors and/or semantic types of that concept, or left
blank (i.e. replaced by “something”). Most UMLS relationships are already
expressed using a verbal form which can be used as-is in the template, but a
set of rules was crafted to convert the relationships which were not (usually
by prepending “is” or “has” before their name). Finally, a known name of the
related concept is randomly selected to finish forming the description.

The descriptions constructed that way do not always unambiguously refer to a
unique concept, but we consider this to be a desirable property because it
tends to pull closer to each other the concepts which share characteristics
that practitioners found useful to encode as relationships.

###  3.2 Pre-training setup

To maximally leverage the meaning of definitions in contrastive pairs, a
sentence embedding model trained on 1B positive pairs (STAMB2) was used as
initialization (Reimers et al., 2021). As a result, the representations
produced for opaque concept names are likely to improve quickly, by drawing
insights from the definitions. It is worth noting that the STAMB2 model has
seen PubMed titles and abstracts as part of its pre-training; therefore, it
already possesses some general understanding of the biomedical domain, albeit
a partial one as our evaluation demonstrates.

To continue training this model, we constructed a dataset containing 100
million pairs of concept names and corresponding definitions or descriptions.
From all training pairs, 85% contain a textual description generated using the
concept-to-concept relationships from UMLS, and 15% contain an actual
definition (to achieve this, each definition is sampled 50 times, with
different concept names whenever this is possible). Our analysis found that
concepts having multiple definitions are also more likely to appear in
clinical notes: oversampling them therefore appears beneficial. We release the
dataset333huggingface.co/datasets/FremyCompany/BioLORD-DS to make it easier to
reproduce our results.

Our model is trained over this large dataset for one epoch, in batches of 64
pairs, using the InfoNCE loss (Oord et al., 2018). In Appendix A, additional
details are presented for readers interested in faithfully reproducing our
experiments.

##  4 Experimental evaluation

To demonstrate the effectiveness of our definition-based pre-training
strategy, we subsequently train and evaluate our model on multiple Semantic
Text Similarity (STS) tasks. For each benchmark, we compare our results with
the state of the art. We also report results for the models BioSyn (Sung et
al., 2020) and SapBERT (Liu et al., 2021), since these share their encoder
architecture (BERT-base) and parameter count (110M) with our model, enabling
fair comparison. We also provide results for the base models of BioSyn,
SapBERT, and BioLORD: respectively BioBERT (Lee et al., 2020), PubMedBERT (Gu
et al., 2020) and STAMB2 (Reimers et al., 2021). When applicable, we use the
same finetuning strategy for all six models.

Our key evaluation tasks are biomedical concept similarity, biomedical concept
normalization, and sentence similarity (both in and out of the biomedical
domain). In the following paragraphs, we describe the datasets and the
experimental setup for these tasks, and already refer to the respective
results tables. We however reserve our thoughts and insights for §5 to enable
cross-task comparisons.

###  4.1 Biomedical concept similarity

Given their pre-training strategy, BioLORD models would be expected to produce
particularly strong semantic representations of biomedical concepts. To
confirm this, we use the cosine distance between their representations to
evaluate the degree of similarity between pairs of biomedical concepts. We
compare this similarity measure against similarities derived from human
judgment. Given the limited size of those datasets, no finetuning is
performed.

Following the approach of Kalyan and Sangeetha (2021), we evaluate our model
using four benchmarks: MayoSRS, UMNSRS-Similarity, UMNSRS-Relatedness, and
EHR-RelB.

MayoSRS (Pakhomov et al., 2011) is a dataset formed by 101 clinical term pairs
whose relatedness was reported on a 4-point scale by nine medical coders and
three physicians.

UMNSRS (Pakhomov et al., 2010) is a pair of datasets, consisting of 725
clinical term pairs whose semantic similarity and relatedness were determined
on a continuous scale by 4 clinicians.

EHR-RelB (Schulz et al., 2020) is a dataset containing 3630 concept pairs
sampled from electronic health records, rated for relatedness by 3 doctors.

In Table 1, we report the Spearman correlation between the similarity scores
attributed by a model and the scores attributed by the medical practitioners.
Achieving a high correlation on this benchmark indicates that the embedding
space defined by the model possesses a latent structure which corresponds well
with the human perception of similarity between concepts.

|  BioBERT |  BioSyn |  PubMedB. |  SapBERT |  STAMB2 |  BioLORD
---|---|---|---|---|---|---
MayoSRS |  28.2 |  45.1 |  42.5 |  62.5 |  47.1 |  74.7†
Umnsrs-R |  25.3 |  39.1 |  25.3 |  47.5 |  43.9 |  54.4†
Umnsrs-S |  33.0 |  43.6 |  25.2 |  53.0 |  46.7 |  56.0†
Ehr-RelB |  33.8 |  42.5 |  31.7 |  51.7 |  54.1 |  57.5†
Table 1:  Spearman scores obtained on the Biomedical concept similarity
benchmarks. Given the small size of these datasets, no finetuning is
performed. Cosine similarity between the representations of the concepts
performed best in all cases, but Euclidian and Manhattan distances were tested
as well. Higher is better.

†: We also trained a BioLORD model based on PubMedBERT, whose results (69.0,
52.7, 56.7, 51.1) were superior to SapBERT but inferior to the BioLORD model
based on STAMB2 which we present in this article.

Other state-of-the-art models for the tasks: While BioLORD performs best among comparable models, two systems combining text embeddings with graph-based enhancements perform better for some tasks. See works by Kalyan and Sangeetha (2021) and Mao and Fung (2020). These techniques could be applied to BioLORD, but this is left as future work.  Sct-L2P |  BioBERT |  BioSyn |  PubMedB. |  SapBERT |  STAMB2 |  BioLORD
---|---|---|---|---|---|---
\- Mrr |  28.1 |  35.4 |  24.8 |  40.6 |  41.6 |  49.9
\- Acc@1 |  20.4 |  25.9 |  17.9 |  29.1 |  31.1 |  37.0
Table 2:  Mean-reciprocal rank (MRR) and Top 1 Accuracy (Acc@1) of the similarity mapping of leaf node concepts of Snomed-CT on their parent concepts, after leaving only non-leaf nodes of the Snomed-CT ontology as candidates.  |  BioBERT |  BioSyn |  PubMedB. |  SapBERT |  STAMB2 |  BioLORD
---|---|---|---|---|---|---
MedSts |  83.7 |  84.0 |  85.8 |  86.0 |  85.9 |  86.3
Biosses |  88.1 |  92.1 |  91.5 |  89.3 |  90.7 |  84.0
Sick |  86.8 |  86.7 |  86.3 |  80.3 |  90.7 |  89.3
Sts |  79.8 |  79.4 |  82.5 |  81.9 |  88.0 |  86.5
Table 3:  Pearson scores obtained by our model on the various Semantic Text
Similarity tasks on which it was finetuned then tested on. Higher is better.
We include two SOTA numbers: one for models of equivalent size and one for
models of any size.

Other state-of-the-art models for the tasks: No comparable model surpasses BioLORD in biomedical sentence similarity tasks but, in the the general-purpose sentence similarity benchmark (STS), the DistillBERT model by Sanh et al. (2019) performs better (90.7) than BioLORD. Models performing better than BioLORD for the biomedical tasks exists, but they are significantly larger models: GatorTron by Yang et al. (2022) for MedSTS (89.0), LLM4Biomedical by Tinn et al. (2021) for BIOSSES (93.6), and SMART by Jiang et al. (2020) for STS (92.9). Models which BioLORD suprasses but remain worth mentioning include TLBNLP by Peng et al. (2019) for MedSTS (84.8), BioNSTS by Blagec et al. (2019) for BIOSSES (81.9), and SiameseBERT by Reimers and Gurevych (2019) for SICK (74.5).  |  BioBERT |  BioSyn |  PubMedB. |  SapBERT |  STAMB2 |  BioLORD
---|---|---|---|---|---|---
MedNli-S |  89.0 |  89.5 |  90.1 |  90.5 |  89.4 |  89.9
Table 4:  Accuracy on text similarity benchmark inspired from MedNLI. Higher is better.  |  SapBERT |  BioLORD |  KRISSBERT
---|---|---|---
NCBI-Diseases |  63.0* |  68.4 |  89.9
MedMentions |  37.6* |  34.1 |  70.7
Table 5:  Accuracy on named entity linking on MedMentions and NCBI. Higher is
better. (*): Because we share the concerns raised by Zhang et al. (2021), and
because we evaluate our model with the same methodology as theirs, we report
in this table the results as presented in their paper, even for the SapBERT
model.

(Bold results indicate superiority amongst comparable models while
underlined results indicate the overall state of the art)

###  4.2 Biomedical Entity Linking

Another potential application of our model is Entity Linking. In the context
of the biomedical domain, it consists in assigning to a given textual mention
the biomedical concept which most faithfully represents it, among those
defined in a target ontology.

NCBI-Diseases (Doğan et al., 2014) is an annotated corpus containing 6892
disease mentions, mapped to 790 unique diseases.

MedMentions (Mohan and Li, 2019) is an annotated corpus aimed at the
recognition of biomedical concepts in biomedical documents. Over 4,000
abstracts were annotated manually, for a total of over 350k linked mentions.
These mentions are linked to their best match among the 3M+ concepts which
were already referenced by UMLS in 2017.

We finetune our model to make use of the context using the training set of
MedMentions444We do so by forming positive pairs using the templates
“[mention] [SEP] (context: [sentence])” and “[canonical-concept-name]”. At
inference time, we compute the representation for a mention with its context
sentence between parentheses, then we identify the concept in the target
ontology whose representation is the closest to that of the mention., and
report our mapping accuracy in Table 5.

A variation of this task where our approach should particularly shine is
entity normalization in non-exhaustive ontologies. Ontologies cannot possibly
cover all concepts, but concepts absent from the ontology should ideally be
normalized to broader, but semantically-compatible concepts.

SCT-L2P (our contribution; short for SnomedCT Leaf-to-Parent) is a benchmark
in which concept names from leaf terms of the SnomedCT ontology are mapped
within a reduced subset of the SnomedCT ontology, which is obtained by pruning
all the leaf nodes from the hierarchy (to prevent self-mapping, see §A.3).

In Table 2, we report how often these leaf SnomedCT concepts were mapped to
one of their parents, as opposed to a concept which is not one of their
parents. This benchmark therefore evaluates whether the representations
produced for biomedical concepts have a good hierarchical structure.

###  4.3 Semantic Text Similarity

Thanks to our initialization with a sentence embedding model, semantic text-
similarity inference in the biomedical domain remains possible. We evaluate
our performance on this task using 4 text-similarity benchmarks, after
finetuning (see §A.4).

Text similarity benchmarks aim to evaluate how accurately models estimate the
similarity in meaning between two pieces of text, usually sentences. As for
the biomedical concept similarity task, human experts assign a similarity
score to these pairs, and a set of such scores is averaged across experts to
form a golden standard. We trained and evaluated all models on four such
datasets.

MedSTS (Wang et al., 2020) was developed for evaluating clinical semantic
textual similarity. It contains 1,068 sentence pairs which were annotated by
two medical experts with semantic similarity scores of 0-5 (low to high
similarity).

BIOSSES (Sogancioglu et al., 2017) considers small paragraphs rather than
sentences, and focuses on scientific articles in the biomedical domain, rather
than clinical notes. It is a challenging dataset because of the length of its
entries.

SICK (Marelli et al., 2014) consists of about 10k English sentence pairs,
designed to be rich in lexical, syntactic, and semantic phenomena. Pairs have
been annotated for relatedness on a 0-5 scale.

STS Benchmark (Cer et al., 2017) regroups several other general-purpose text
similarity datasets (and contains 8628 sentence pairs).

Table 3 reports the Pearson correlation between the similarity scores
attributed to sentence pairs by the model under evaluation, and the gold
standard for that sentence pair.

###  4.4 Natural Language Inference

Natural Language Inference (NLI) is the task of deciding whether a piece of
evidence (usually a paragraph of text) can be used to support a given
conclusion (usually a sentence about the same topic). The training and test
data of NLI models usually contain pairs of sentences and a judgement on
whether the first one, called the premise, entails (supports), contradicts
(anti-supports), or is neutral towards a second sentence called the
hypothesis.

MedNLI (Romanov and Shivade, 2018) is the dataset we used to perform this
analysis. It was curated by doctors asked to provide three statements (one
entailed, one contradicted, and one neutral) grounded in the medical history
of a given patient.

BioLORD is inherently not suited to capture the relation between two
sentences, since it would encode them separately, unlike cross-encoder models
that are typically trained for NLI tasks.

Yet, we can hypothesize that semantically strong sentence-encoders should
yield representations that are more similar for entailed sentence pairs than
contradicted ones. We therefore propose the following evaluation strategy: all
premises of the input dataset for which both an entailed and a contradicted
hypothesis have been curated are retained for usage as an input in triplet
form. For these triplets, we report in Table 4 whether the hypothesis most
similar with the premise (in terms of cosine similarity) is indeed the
entailed hypothesis, on average for the dataset.

##  5 Discussion

Thanks to the experiments described in the previous section, the suitability
of definition-based contrastive learning can now be demonstrated, which we
will do in this section. We also discuss patterns appearing in our results,
which might be useful in designing further improvements to our technique.

Our pre-training strategy significantly improves the alignment with human
judgement of biomedical entity representations (+27.6pts on MayoSRS compared
to STAMB2, our initialization model; see Table 1). This improvement is more
significant than those elicited by either SapBERT (+20.0pts) or BioSyn
(+16.9pts). Since our model also performs better in absolute terms, this
confirms that our pre-training strategy is the most effective.

Unlike both SapBERT and BioSyn, BioLORD’s performance on general-purpose
sentence representations remains competitive. It is however worth noting that
the aforementioned pre-training strategies seem to impair general-purpose
sentence representation, even after fine-tuning (see Table 3). Despite making
use of descriptions and definitions in the training data, this remains true
for BioLORD (-1.5pts on STS Benchmark). We dig deeper into these results in
§5.2.

In the specific case of biomedical sentences, the BioLORD pre-training
nonetheless remains a net-positive. Thanks to this, we achieve state-of-the-
art results for bi-encoders on MedSTS (86.3pts, +0.4pts over initialization),
and we can report improvements over initialization on MedNLI-S as well
(89.9pts, +0.5pts).

###  5.1 Biomedical concept representations

When it comes to the concept embedding task, BioLORD again outperforms its
peers, achieving significantly higher Spearman correlation than SapBERT on all
available benchmarks (+12.2pts for MayoSRS, +6.9pts for UMNSRS-R, +3.0pts for
UMNSRS-S, +5.8pts for EHR-RelB; see Table 1).

BioLORD performs particularly well w.r.t. its peers on the relatedness
benchmarks. We attribute this to the fact our model is trained on all the
relationship types described in UMLS: its representations might therefore
encompass more ways for concepts to be related to each other, compared to
models trained on the hierarchical relationships only. This matters less for
similarity benchmarks, because similar entities are more likely to be close in
the hierarchical graph, too.

Our improved representations provide additional benefits, as shown by our
SCT-L2P evaluation task. When matching leaf-concepts against a leaves-removed
ontology, BioLORD was able to match leaf-concepts to their parent concepts
more often than SapBERT (+7.9pts on SCT-L2P Accuracy@1, see Table 2). BioLORD
also assigns significantly fewer concepts to a representation so disconnected
from those of its parents that none appear in the 1000 nearest neighbors (3.8%
vs 8.6%). BioLORD therefore achieves a much higher mean-reciprocal rank than
SapBERT (+9.3pts, see Table 2).

It is worth noting that combining (Mao and Fung, 2020) or enhancing (Kalyan
and Sangeetha, 2021) concept-name representations with graph embeddings
remains the state of the art in some benchmarks even after our contributions,
which means that further improvements might still be achievable by merging
BioLORD representations with graph-based representations. We leave this
investigation for a future article.

For completeness, we also evaluated our model on the more traditional
benchmarks of biomedical entity linking. As our results on MedMentions and
NCBI-Diseases show, BioLORD remains competitive with other alternatives on
this task as well (see Table 5).

###  5.2 Biomedical sentence representations

Unlike previous models, BioLORD also aims to provide good representations for
biomedical sentences from clinical notes, and indeed achieves state-of-the-art
results for MedSTS among bi-encoder models (see Table 3).

While cross-encoder models are known to achieve better results, bi-encoder
models remain important for search and retrieval scenarios as they support the
independent treatment of documents, enabling parallel workflows. Given that
most hospitals accumulate millions of clinical notes, this is a highly
desirable feature.

What makes our model particularly suitable for clinical note understanding is
that it shows solid performance on general-purpose text as well, strongly
outperforming all the other state-of-the-art biomedical models in this setup.
This is true on both the SICK and STS Benchmark datasets (+9pts over SapBert
on SICK, +5.5pts on STS Benchmark). Because clinical notes often contain a mix
of text in medical jargon and more mundane discourse, we believe using models
generalizing well to multiple domains is valuable, as both types of discourse
are likely to carry useful information.

Benefit of the descriptions: As we expected, the performance degradation on
SICK (-1.4pts) remained smaller than on STS Benchmark (-3.5pts) in the case of
BioLORD, while the SapBERT pre-training caused significant performance
reduction on SICK (-6pts). We can explain this by the way the SICK dataset is
constructed. Unlike most STS benchmarks, SICK tries to require as little as
possible the understanding of a domain language (e.g. idiomatic multiword
expressions, named entities, etc…) while putting more attention on lexical,
syntactic and semantic phenomena that generalize well across domains (for
instance, by replacing nouns or adjectives by equivalent relative clauses).
This mixes well with our pre-training approach which combines nouns and
descriptions.

In the case of BioLORD, a large part of the performance degradation on
general-purpose semantic text similarity performance (the STS Benchmark) is
therefore likely caused by the loss of knowledge about concepts unrelated to
the biomedical domain. This is reassuring, because that type of knowledge is
less likely to be useful in biomedical text understanding than the general
understanding of sentence structures which is tested by SICK.

##  6 Limitations

While BioLORD models perform better on general-purpose text than other state-
of-the-art biomedical models, we would like to recognize that no guarantee can
be made regarding their ability to make sense of non-ontological modalities of
languages in the biomedical domain (spoken language, social media, online
forums, etc…). This might cause issues in circumstances where the content
being processed is not originating from clinicians.

BioLORD models are not well suited to tasks requiring the latest scientific
knowledge, for example found in PubMed papers, because the origin of most of
their biomedical knowledge is ontologies. This type of knowledge might be
better encoded in PubMedBERT-based models. Hybrid approaches might therefore
be worthwhile for natural language inference and scientific paper analysis
scenarios.

Another limitation worth mentioning is that, unlike Graph Neural Networks,
BioLORD models do not provide a single and unique embedding for each concept
of an ontology, because concepts might have more than one known name. As
representations are produced for names rather than concepts, each name will
produce a different (but hopefully close) representation for a concept. This
might amper rule-based decision processes.

Despite these limitations, we envision that most use cases traditionally
considered in the biomedical domain are suitable for the usage of BioLORD
models, in an end-to-end setup.

##  7 Conclusion

The results detailed in this paper demonstrate that our name-to-definition
contrastive pre-training strategy (BioLORD) is able to produce bi-encoder
models with a state-of-the-art performance for both biomedical concept
similarity and clinical sentence similarity, using one set of weights.

As part of our investigation, we were also able to show that appropriate
finetuning strategies can enable general-purpose models (like STAMB2) to fully
overcome their initial disadvantage at bio-medical text understanding compared
to models of the same size trained on biomedical data only (such as
PubMedBERT). This can be achieved while preserving most of their pre-existing
general-purpose text understanding capabilities.

By releasing a bi-encoder model able to embed both clinical sentences and
biomedical concepts in the same latent space, we immensely simplify some
clinical language processing scenarios, like clinical note retrieval and
diffuse information extraction. For this reason, we hope that future works in
the biomedical domain will continue to combine clinical sentences and
biomedical concept benchmarks to produce directly usable end-to-end models.

##  8 Ethical considerations

Healthcare is a very sensitive domain, and we could not simply conclude this
article without first taking some time to reflect on the ethical implications
of the usage in the real world of models trained using our BioLORD
methodology.

Using machine learning models to affect a patient’s treatment is never a
decision to be taken lightly, and we are conscious of the fact medical
practitioners should always be kept in the loop in such a process. Machine
learning models might help surface information more easily, but should never
replace medical practitioners entirely, given the need for verification of the
information extracted by models, and for guiding the usage that is made of the
information once extracted.

Because of our intentional focus on publicly-available ontologies and
annotated benchmarks, only a limited set of sources were used to train our
models. However, we are aware that knowledge originating only from ontologies
might need to be supplemented by other data sources to provide trustworthy
results in real-word settings. We would therefore like to urge machine
learning specialists to consider the issue of human-machine alignment before
using models built using our approach.

In-house data and annotations will be crucial to achieve good outcomes in the
real world. We hope hospitals will continue to invest in the annotation of
data to fully realize the potential of machine learning models, but we
recognize this is a time-consuming process. Better tools would probably help
medical experts to spend less time on data annotation than they need to spend
today.

We also want to point out that multi-linguality is important for hospitals
taking care of patients coming from regions where different languages are
spoken. End-to-end models are very suitable for cross-lingual distillation,
which require less data than training from scratch (Reimers and Gurevych,
2020). This justifies our decision to train an end-to-end model, and we aim to
talk more about multi-linguality in the near future.

Taking the previous points in consideration leads us to believe that the
publication of this model and of our results can have a positive impact on
society in addition to the machine learning field, but we remain available to
discuss further ethical concerns.

## References

  * Aronson (2001) Alan R. Aronson. 2001.  Effective mapping of biomedical text to the umls metathesaurus: The metamap program.  In _AMIA Symposium_ , pages 17–21.
  * Blagec et al. (2019) Kathrin Blagec, Hong Xu, Asan Agibetov, and Matthias Samwald. 2019.  Neural sentence embedding models for semantic similarity estimation in the biomedical domain.  _BMC Bioinformatics_ , 20(1):178.
  * Bodenreider (2004) Olivier Bodenreider. 2004.  The unified medical language system (UMLS): integrating biomedical terminology.  _Nucleic Acids Res_ , 32(Database issue):D267–70.
  * Cer et al. (2017) Daniel Cer, Mona Diab, Eneko Agirre, Iñigo Lopez-Gazpio, and Lucia Specia. 2017\.  SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation.  In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_ , pages 1–14, Vancouver, Canada. Association for Computational Linguistics.
  * Doğan et al. (2014) Rezarta Islamaj Doğan, Robert Leaman, and Zhiyong Lu. 2014.  NCBI disease corpus: a resource for disease name recognition and concept normalization.  _J Biomed Inform_ , 47:1–10.
  * Gu et al. (2020) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2020.  Domain-specific language model pretraining for biomedical natural language processing.  _arXiv preprint_ , abs/2007.15779.
  * Jiang et al. (2020) Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Tuo Zhao. 2020.  SMART: Robust and efficient fine-tuning for pre-trained natural language models through principled regularized optimization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2177–2190, Online. Association for Computational Linguistics.
  * Kalyan and Sangeetha (2021) Katikapalli Subramanyam Kalyan and Sivanesan Sangeetha. 2021.  A hybrid approach to measure semantic relatedness in biomedical concepts.  abs/2101.10196.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Liu et al. (2021) Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, and Nigel Collier. 2021\.  Self-alignment pretraining for biomedical entity representations.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4228–4238, Online. Association for Computational Linguistics.
  * Mao and Fung (2020) Yuqing Mao and Kin Wah Fung. 2020.  Use of word and graph embedding to measure semantic relatedness between unified medical language system concepts.  _J Am Med Inform Assoc_ , 27(10):1538–1546.
  * Marelli et al. (2014) Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014.  A SICK cure for the evaluation of compositional distributional semantic models.  In _Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC’14)_ , pages 216–223, Reykjavik, Iceland. European Language Resources Association (ELRA).
  * Mohan and Li (2019) Sunil Mohan and Donghui Li. 2019.  Medmentions: A large biomedical corpus annotated with umls concepts.  _Proceedings of the 2019 Conference on Automated Knowledge Base Construction (AKBC 2019)_.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint_ , abs/1807.03748.
  * Pakhomov et al. (2010) Serguei Pakhomov, Bridget McInnes, Terrence Adam, Ying Liu, Ted Pedersen, and Genevieve B Melton. 2010.  Semantic similarity and relatedness between clinical terms: An experimental study.  _AMIA Annu Symp Proc_ , 2010:572–576.
  * Pakhomov et al. (2011) Serguei Pakhomov, Ted Pedersen, Bridget Mcinnes, Genevieve Melton, Alexander Ruggieri, and Christopher Chute. 2011.  Towards a framework for developing semantic relatedness reference standards.  _Journal of biomedical informatics_ , 44:251–65.
  * Peng et al. (2019) Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019.  Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets.  In _Proceedings of the 18th BioNLP Workshop and Shared Task_ , pages 58–65, Florence, Italy. Association for Computational Linguistics.
  * Reimers et al. (2021) Niels Reimers et al. 2021.  Train the best sentence embedding model ever with 1b training pairs.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Reimers and Gurevych (2020) Nils Reimers and Iryna Gurevych. 2020.  Making monolingual sentence embeddings multilingual using knowledge distillation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4512–4525, Online. Association for Computational Linguistics.
  * Romanov and Shivade (2018) Alexey Romanov and Chaitanya Shivade. 2018.  Lessons from natural language inference in the clinical domain.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1586–1596, Brussels, Belgium. Association for Computational Linguistics.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint_ , abs/1910.01108.
  * Schulz et al. (2020) Claudia Schulz, Josh Levy-Kramer, Camille Van Assel, Miklos Kepes, and Nils Hammerla. 2020.  Biomedical concept relatedness – a large EHR-based benchmark.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6565–6575, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Schulz and Klein (2008) Stefan Schulz and Gunnar O. Klein. 2008.  Snomed ct – advances in concept mapping, retrieval, and ontological foundations. selected contributions to the semantic mining conference on snomed ct (smcs 2006).  _BMC Medical Informatics and Decision Making_ , 8(1):S1.
  * Sogancioglu et al. (2017) Gizem Sogancioglu, Hakime Öztürk, and Arzucan Özgür. 2017.  BIOSSES: a semantic sentence similarity estimation system for the biomedical domain.  _Bioinformatics_ , 33(14):i49–i58.
  * Sung et al. (2020) Mujeen Sung, Hwisang Jeon, Jinhyuk Lee, and Jaewoo Kang. 2020.  Biomedical entity representations with synonym marginalization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3641–3650, Online. Association for Computational Linguistics.
  * Tinn et al. (2021) Robert Tinn, Hao Cheng, Yu Gu, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Fine-tuning large neural language models for biomedical natural language processing.  _arXiv preprint_ , abs/2112.07869.
  * Ujiie et al. (2021) Shogo Ujiie, Hayate Iso, and Eiji Aramaki. 2021.  Biomedical entity linking with contrastive context matching.  _arXiv preprint_ , abs/2106.07583.
  * Wang et al. (2020) Yanshan Wang, Naveed Afzal, Sunyang Fu, Liwei Wang, Feichen Shen, Majid Rastegar-Mojarad, and Hongfang Liu. 2020.  Medsts: a resource for clinical semantic textual similarity.  _Language Resources and Evaluation_ , 54(1):57–72.
  * Yang et al. (2022) Xi Yang, Nima PourNejatian, Hoo Chang Shin, Kaleb Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Mona Flores, Ying Zhang, Tanja Magoc, Christopher Harle, Gloria Lipori, Duane Mitchell, William Hogan, Elizabeth Shenkman, Jiang Bian, and Yonghui Wu. 2022.  Gatortron: A large language model for clinical natural language processing.  _medRxiv preprint_ , 2022.02.27.22271257.
  * Zhang et al. (2021) Sheng Zhang, Hao Cheng, Shikhar Vashishth, Cliff Wong, Jinfeng Xiao, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Knowledge-rich self-supervision for biomedical entity linking.  _arXiv preprint_ , abs/2112.07887.

##  Appendix A Replication details

Given this article covers many benchmarks and datasets, it is not practical to
provide all the details required for reproduction in the main text. Details
which we do not believe contribute to the understanding of our paper are
listed here.

###  A.1 BioLORD Dataset

The dataset contains 100M pairs (86M with descriptions, 14M with definitions).
Another set of 20M descriptions based on the same knowledge graph serves as a
development set (86M generations certainly do not exhaust the graph). However,
this would not be a suitable test set. Instead, a test of time consisting of
new concepts currently absent from UMLS would make more sense, but this will
have to wait until enough new concepts have been added to UMLS. Our
supplementary materials contain a sample of our train set, and we uploaded our
full train data on huggingface (24Gb file).

###  A.2 BioLORD Pre-training

Given the time required for training BioLORD models with the hardware
available to us, only cursory hyperparameter tuning was performed. Our
investigation focused on the type of loss (InfoNCE vs MegaBatchMarginLoss),
the ratio of definition and description, the number of training examples (20M
vs 100M), and the batch size (64 vs 96). We kept all other hyperparameters at
their default value (AdamW for the optimizer, WarmupLinear for the scheduler,
2e-5 for the learning rate, 5% of the data for the warmup window, 0.01 for the
weight decay, 1 for the number of epochs, PyTorch 1.7.1 AMP for the mixed-
precision training).

Rather than using the downstream task as evaluation, a manual investigation
was carried out by inspecting the latent space neighborhood of a set of
biomedical concepts555cat / cat scan / cat scratch / cat scratch disease / cat
scratch disease antigen / nephrectomy / partial nephrectomy / lobotomy / hand
/ left hand / cleft hand / apyrexial / afebrile / ranitidine / ranitidine
allergy / falcons / belgium / apyrexial / afebrile / hepatic arteriogram /
bowling / bowling alley / bowling alley manager / endometriosis / heparin
sodium. Due to the subjective nature of this test, we cannot discard the
chance that another set of hyperparameters might have yielded to a better
model.

The model whose results are described in this paper (100M pairs (15% of
definitions and 85% of descriptions) in batches of 64 pairs) took 7 days of
training on a NVIDIA V100 GPU.

###  A.3 SnomedCT-L2P Dataset

SnomedCT contains about 700k concept names, among which about 550k concern
leaf concepts, and about 150k concern non-leaf concepts. These concepts have
hierarchical relationships defined between each other, and the goal of this
dataset is to detect whether leaf concepts have a representation which is
closer to the representation of their hierarchical parents than to the
representation of other concepts in the ontology. We do not define a train /
dev split, and use the entire dataset as a test set, because we intend to
evaluate model performance without further finetuning. SnomedCT is an ever-
evolving dataset, so our supplementary materials contain the files we used for
evaluation, so that our results can be replicated.

###  A.4 STS Finetuning

For all the finetuning experiments, the following hyperparameters have been
selected: AdamW for the optimizer, WarmupLinear for the scheduler, 6e-6 for
the learning rate, 5% of the data for the warmup window, 0.01 for the weight
decay, 10 for the number of epochs (the weights are saved after every epoch,
and the best weights for the dev set are retained for the final evaluation),
64 for the batch size, PyTorch 1.7.1 AMP for the mixed-precision training. The
finetuning time varies per dataset, but at most reaches a couple of minutes
(same hardware as before).

###  A.5 MedMentions Finetuning

For all the finetuning experiments, the following hyperparameters have been
selected: AdamW for the optimizer, WarmupLinear for the scheduler, 2e-5 for
the learning rate, 5% of the data for the warmup window, 0.01 for the weight
decay, 1 for the number of epochs, 64 for the batch size, PyTorch 1.7.1 AMP
for the mixed-precision training. The finetuning requires about 4.5 hours to
finish (same hardware as before).

###  A.6 Downloading our model

Our biomedical model is hosted on huggingface.co/ FremyCompany/BioLORD-
STAMB2-v1.

Its dataset is hosted on: huggingface.co/datasets/ FremyCompany/BioLORD-DS.

Our supplementary materials include our training code and a sample of our
dataset.

##  Appendix B Qualitative analysis

In this appendix, we compare the similarity matrix of a few biomedical
concepts, as generated by BioSyn, SapBERT, and BioLORD. This highlights the
improvements of the latter over its peers.

Within each group of three, concepts are significantly more related to each
other than the rest.

Figure B.1: Concept Similarity matrix for BioSyn Figure B.2: Concept
Similarity matrix for SapBERT Figure B.3: Concept Similarity matrix for
BioLORD

As can be seen above, BioLORD does a much better job clustering the related
concepts together, without mixing them with the other groups. It even clusters
"cat scratch disease" with other types of Bartonella infections.

##  Appendix C Acknowledgments

This work would not have been possible without the joint financial support of
the Vlaams Agentschap Innoveren & Ondernemen (VLAIO) and the RADar innovation
center of the AZ Delta hospital group.

I am especially indebted to Ir. Peter De Jeager, Chief Innovation Officer of
RADar, for his role in setting up and keeping on track the Advanced Data-Aided
Medicine project (ADAM), and ensuring prompt and smooth communication between
me and the other shareholders working at the hospital.

A particular thanks is also extended to the team maintaining and developing
GPULab, the machine learning infrastructure for AI computing built in
collaboration between UGent, UAntwerpen and the Imec research and development
center.

Finally, I would also like to thank my co-supervisors, Kris Demuynck and
Thomas Demeester, for their support and constructive advice during the
ideation process, and all along the development of this project up to this
very article.