# LSTM-based Deep Neural Network With A Focus on Sentence Representation for
Sequential Sentence Classification
in Medical Scientific Abstracts

Phat Lam1∗, Lam Pham2∗, Tin Nguyen3, Hieu Tang4, Seidl Michael5, Alexander
Schindler6 L. Pham, S. Michael, and A. Schindler are with Austrian Institute
of Technology, Austria.H. Tang is with FPT University, Vietnam.P. Lam and T.
Nguyen are with HCM University of Technology, Vietnam.(*) Main and equal
contribution into the paper.

###### Abstract

The Sequential Sentence Classification task within the domain of medical
abstracts, termed as SSC, involves the categorization of sentences into pre-
defined headings based on their roles in conveying critical information in the
abstract. In the SSC task, sentences are often sequentially related to each
other. For this reason, the role of sentence embedding is crucial for
capturing both the semantic information between words in the sentence and the
contextual relationship of sentences within the abstract to provide a
comprehensive representation for better classification. In this paper, we
present a hierarchical deep learning model for the SSC task. First, we propose
a LSTM-based network with multiple feature branches to create well-presented
sentence embeddings at the sentence level. To perform the sequence of
sentences, a convolutional-recurrent neural network (C-RNN) at the abstract
level and a multi-layer perception network (MLP) at the segment level are
developed that further enhance the model performance. Additionally, an
ablation study is also conducted to evaluate the contribution of individual
component in the entire network to the model performance at different levels.
Our proposed system is very competitive to the state-of-the-art systems and
further improve F1 scores of the baseline by 1.0%, 2.8%, and 2.6% on the
benchmark datasets PudMed 200K RCT, PudMed 20K RCT and NICTA-PIBOSO,
respectively.

Keywords— Sentence embeddings, bidirectional long short-term memory network,
convolutional neural network, attention, multiple feature branches.

##  I INTRODUCTION

When searching for a large-scale source of scientific papers, people have the
tendency to skim through abstracts to quickly identify whether papers align
with their research interest. This process becomes more straightforward when
abstracts are organized with semantic headings such as ”background”,
”objective”, ”methods”, ”results”, and ”conclusion”. Therefore, automatically
categorizing each sentence in a scientific abstract into a relevant heading,
which is known as the task of Sequential Sentence Classification (SSC),
significantly facilitates the information retrieval process within large-scale
data. In medical domain, research abstracts present a large volume and have
grown exponentially. Manually sorting through these documents to find relevant
insights can be a time-consuming and labor-intensive task. Therefore, the
result of the SSC tasks enables researchers and learners to catch up and
categorize research abstracts very quickly. In the other words, the SSC task
significantly facilitates learners and researchers by accelerating their
educational processes of literature review, information extraction, evidence-
based decision-making, etc. Recently, the SSC task in medical scientific
abstracts has drawn attention from NLP research community. Indeed, some large
and benchmark datasets such as PubMed RCT [1] and NICTA-PIBOSO [2] were
published. Additionally, a wide range of machine learning based and deep
learning based models have been proposed for this task. Traditional machine
learning methods utilized hand-crafted feature extraction for individual
sentences. These extracted features are related to lexical, semantic and
structural information of an individual sentence such as lemma, synonyms, bag-
of-words, part of speech, etc. Then, sentences are classified by Hidden Markov
Model (HMM) [3], Naive Bayes [4] or CRF [5]. While the traditional machine
learning based models present a limitation of exploring the relation among the
sentences as using the hand-crafted features, leveraging deep neural networks
in deep learning based models allow to capture the patterns on contextual
relationship among sentences in the same abstract that leads a breakthrough on
model performance. For examples, Dernoncourt et al. [6] introduced ANN model
which used a CRF layer to optimize the predicted label sequence, where
adjacent sentences have impact on the prediction of each other. Jin and
Szolovits [7] proposed a hierarchical sequential labeling network to further
improve the semantic information within surrounding sentences for
classification. Recently, Yamada et al. [8] or Shang et al. [9] introduced
some methodologies to assign labels at abstract level and span level rather
than single sentences, However, these systems leverage sentence embeddings
extracted from the pre-trained BERT model [10] without improving the quality
of the sentence embeddings which plays an important role in achieving better
model performance for the SSC task.

In this paper, we therefore present a deep neural network with a focus on
extracting well-presented sentence embeddings for the SSC task in medical
scientific abstracts. In particular, we explore the independent features of
sentence, word sequence, character sequence, and statistic information of
sentences in one abstract. We then develop a LSTM-based deep neural network
with multiple-feature branches for classifying individual sentences. The
network is then used to extract the well-presented sentence embeddings. Given
these sentence embeddings, a Convolutional-RNN based network (C-RNN) at the
abstract level and a Multi-layer Perception network (MLP) at the segment level
are introduced to learn the contextual patterns of sentences in the same
abstract. Finally, the results of C-RNN and MLP models are fused to achieve
the final predicted sentences in an abstract. We evaluate our proposed models
on two benchmark datasets, PubMed RCT [1] and NICTA-PIBOSO [2]. The results on
three above levels indicate that exploring multiple features extracted from
sentence, word sequence, character sequence, and statistic information of
sentences in the abstract helps to generate well-presented sentence embeddings
at the sentence level. Both models of C-RNN and MLP at the abstract level and
the segment level respectively further improve the performance when leveraging
these well-presented sentence embeddings.

##  II The Overall Proposed System

The proposed system in this paper for the task of sequential sentence
classification in medical scientific abstracts is generally presented in Fig.
1. As Fig. 1 shows, the proposed network comprises of three main sub-networks,
referred to as the classification model (Sen-Model) at the sentence level, the
regression model at the abstract level (Abs-Model) and the classification
model at the segment level (Seg-Model). At the sentence level, we establish
the task sentence classification for individual sentences. The classification
model at the sentence level (Sen-Model) presents 4 branches, each of which
explores the distinct feature from the full sentence, the words in the
sentence, the character in the sentence, and the statistic information of the
sentence. Given the classification model at the sentence level, we extract the
sentence embeddings
$\mathbf{SE}=[\mathbf{se_{1}},\mathbf{se_{2}},...,\mathbf{se_{S}}]$, where
$\mathbf{se_{m}}$ represents for an individual sentence. The sentence
embeddings are then fed into the regression model at the abstract level (Abs-
Model) and the classification model at the segment level (Seg-Model) to
improve the tasks of sentence classification by exploring the contextual
relationship among sentences and structural information of segments in the
same abstract.

Figure 1: The overall architecture of the proposed system

All three mentioned models at different levels leverage the LSTM based
architecture, the attention mechanism, and the multi-layer perception (MLP)
architecture which are comprehensively presented in next subsections.

###  II-A The classification model at the sentence level (Sen-Model)

The classification model at the sentence level (Sen-Model) is comprehensively
presented in Fig. 2. Given a sentence, we have $W$ words
$[w_{1},w_{2},...w_{W}]$ and $C$ characters $[c_{1},c_{2},...c_{C}]$. We make
use of the pre-trained model Glove [11] to extract a sequence of word
embeddings $\mathbf{WE}=[\mathbf{we_{1}},\mathbf{we_{1}},...\mathbf{we_{W}}]$,
where $\mathbf{we_{w}}\in R^{d^{w}}$ presents a word embedding and $d^{w}$ is
the dimension of a word embedding. Regarding the sequence of characters in one
sentence, the character embedding is randomly initialized in the uniform
distribution to extract the character embeddings
$\mathbf{CE}=[\mathbf{ce_{1}},\mathbf{ce_{2}},...\mathbf{ce_{C}}]$, where
$\mathbf{ce_{c}}\in R^{d^{c}}$ presents a character embedding and $d^{c}$ is
the dimension of a character embedding.

Figure 2: The Sen-Model architecture for classification at the sentence level

The sequence of word embeddings $\mathbf{WE}$ and the sequence of character
embeddings $\mathbf{CE}$ are fed into stacked Bi-LSTM-Attention encoder
blocks, referred as SBA blocks, to generate the encoded word embeddings
$\mathbf{WX}=[\mathbf{wx_{1}},\mathbf{wx_{2}},...\mathbf{wx_{K}}]$ and the
encoded character embeddings
$\mathbf{CX}=[\mathbf{cx_{1}},\mathbf{cx_{2}},...\mathbf{cx_{K}}]$,
respectively. The SBA block includes a Bi-LSTM network which comprises of two
stacked Bidirectional LSTM layers, followed by a Scaled Dot-Product Attention
layer [12]. In the Bi-LSTM network, each layer takes the output sequence of
the previous layer as input, which allows the capture of more complex lexical,
syntactic, and semantic information between words and characters in an
individual sentence. Given the sequential word representation and character
representation extracted from the Bi-LSTM layer, we applies linear transform
to create query, key and value matrix $\mathbf{Q}\in R^{N_{q}\times d_{m}}$,
$\mathbf{K}\in R^{N_{k}\times d_{m}}$, $\mathbf{V}\in R^{N_{v}\times d_{v}}$,
where $N_{q}$, $N_{k}$, $N_{v}$ are the number of queries, keys and values,
$d_{m}$ and $d_{v}$ are the dimension of query and key, the dimension of
value, respectively. The output matrix of Scaled Dot-Product attention layer
is computed as:

$Attention(\mathbf{Q,K,V})=\text{Softmax}\left(\frac{\mathbf{QK}^{T}}{\sqrt{d_{m}}}\right)\mathbf{V}$

These two encoded embeddings extracted from SBA blocks of words and characters
are then concatenated to generated the word-char embedding
$\mathbf{WC}=[\mathbf{wc_{1}},\mathbf{wc_{2}},...\mathbf{wc_{K}}]$ where
$\mathbf{wc_{K}}\in R^{d^{w}+d^{c}}$. The word-char embedding $\mathbf{WC}$ is
then fed into the word-char encoder block to generate the encoded word-char
embeddings
$\mathbf{WC\mbox{-}X}=[\mathbf{wc\mbox{-}x_{1}},\mathbf{wc\mbox{-}x_{2}},...\mathbf{wc\mbox{-}x_{K}}]$.
The word-char encoder block reuse the two stacked Bidirectional LSTM layers
from the SBA block without using the attention layer.

Besides the lexical, syntactic and semantic information for each sentence
extracted from word and character branches, we consider the statistics
information of individual sentence: the number of sentences in the same
abstract, the index of sentence in the abstract, and the number of words in
the sentence, which are represented by one-hot vectors
$\mathbf{t_{1},t_{2},t_{3}}$. These statistic vectors are fed into a Multi-
layer perception (MLP) block to generate encoded statistic embeddings
$\mathbf{TE}=[\mathbf{te_{1}},\mathbf{te_{2}},...\mathbf{te_{K}}]$ with
$\mathbf{te_{k}}\in R^{d^{t}}$. The encoded statistic embeddings $\mathbf{TE}$
and then concatenated with the encoded word-char embedding
$\mathbf{WC\mbox{-}X}$ to generate the word-char-stat embedding
$\mathbf{WCT}=[\mathbf{wct_{1}},\mathbf{wct_{2}},...\mathbf{wct_{K}}]$ where
$\mathbf{wct_{k}}\in R^{d^{w}+d^{c}+d^{t}}$. Again, one Bi-LSTM layer is used
to learn the sequence of word-char-stat embeddings, which combines of both
semantic, syntactic information of word-char encoded embedding and statistical
information of statistic embedding, to generate the encoded word-char-stat
embedding.

To further improve the presentation of a sentence, we leverage the BiomedBERT
[13], which are pre-trained on PudMed corpus, to extract the PubMed-based
embedding. The PubMed-based embedding is concatenated with the encoded word-
char-stat embedding before feeding into the multi-layer perception block for
classification. After training the Sen-Model at the sentence level, we extract
the output of the Dense layer before the final Softmax layer and consider it
as the sentence-level embedding $\mathbf{se_{i}}$, where $\mathbf{se_{i}}\in
R^{d^{c}}$ presents a sentence-level embedding and $d^{c}$ is the dimension of
a sentence-level embedding (i.e. the Softmax layer presents $C$ outputs which
matches $C$ labels of sentences in an abstract).

TABLE I: MLP based network for classification at the segment level

Blocks | Layers | Output Shape
---|---|---
F1 | Dense (512) - Elu - BN -Dr(0.5) | 512
F2 | Dense (256) - Elu - BN - Dr(0.5) | 256
F3 | Dense (128) - Elu - BN - Dr(0.5) | 128
F4 | Dense (64) - Elu - BN - Dr(0.5) | 64
F5 | Dense (C) - Softmax | C

###  II-B The regression model at the abstract level (Abs-Model)

Given the original dataset comprising of $S$ sentences
$[s_{1},s_{2},...s_{S}]$, each sentence is now represented by a sentence-level
embedding $\mathbf{se_{i}}$ extracted from the Sen-Model at the sentence
level. To explore the properties of sentences in an abstract, we group
sentence embeddings in the same abstract to create the abstract representation
$\mathbf{AB}=[\mathbf{se_{1}},\mathbf{se_{2}},...\mathbf{se_{I}}]$ where
$\mathbf{se_{i}}\in R^{d^{c}}$. The abstract representation $\mathbf{AB}$ is a
sequence of sentence-level embeddings which is fed into the regression model
at the abstract level (Abs-Model). The regression model at the abstract level
(Abs-Model) is comprehensively presented at the left corner of the upper part
of Fig. 1. The network includes two parts: Convolutional-RNN network and
Logistic Regression classifier.

Each abstract representation $\mathbf{AB}$ is now considered as a two-
dimensional tensor which is fed into the convolution layers to extract
essential features among sentences in an abstract. The two 2D-convolution
layers in the C-RNN network present the same kernel size $(8,3)$ and the same
padding. The number of filters in each layer are set to $16$. Next, the Bi-RNN
decoder is used for better learning sequential relationship feature maps
extracted from the convolution block. Finally, the Logistic Regression
classifier receives the feature maps from the Bi-RNN decoder as input and
generate the sequence of predicted values
$\mathbf{\hat{Y}}_{\mathbf{{ab}}}=[\mathbf{\hat{y_{1}}},\mathbf{\hat{y_{2}}},...\mathbf{\hat{y_{I}}}]$,
where $\mathbf{\hat{y_{i}}}\in R^{d^{c}}$. The ground truth
$\mathbf{Y}=[\mathbf{y_{1}},\mathbf{y_{2}},...\mathbf{y_{I}}]$,
$\mathbf{y_{i}}\in R^{d^{c}}$, are then compared with the predicted values
using the Binary Cross Entropy loss.

###  II-C The classification model at the segment level (Seg-Model)

Given the sentence embeddings, we create fixed-length segments with the size
of $m$ by grouping every $m$ consecutive sentences in the abstract. One
segment representation is described as
$\mathbf{SM}=[\mathbf{se_{1}},\mathbf{se_{2}},...\mathbf{se_{M}}]$. The
corresponding label $\mathbf{y_{sm}}$ of one segment $\mathbf{sm}$ is defined
as:

$\mathbf{y_{sm}}=\frac{\displaystyle{\sum^{M}_{m=1}\mathbf{y_{m}}}}{\displaystyle{\sum^{M}_{m=1}\sum^{\text{C}}_{c=1}y_{mc}}}$

where $\displaystyle{\sum^{\text{C}}_{c=1}y_{mc}}$ is the sum of elements in
the label vector $\mathbf{y_{i}}$ of the sentence $\mathbf{se_{i}}$,
respectively. In our experiments, the fixed-length $m$ is set to 3. The Seg-
model at the segment level uses the same labels as that of the sentence level.

To classify segments, we use the MLP network which is shown at table I. The
network consists of five fully-connected blocks. The first four blocks has the
number of same layers, which are Dense layer, Elu activation, Batch
Normalization and Dropout, respectively. The output of the last block is used
for segment-based classification task. Since the labels of segment embeddings
are no longer one-hot encoded, we use the Kullback-Leibler (KL) divergence
loss for the segment-based classification task, which is defined as:

$L_{KL}(\theta)=\sum^{N}_{n=1}\mathbf{y_{sm}}_{(n)}\log{\frac{\mathbf{y_{sm}}_{(n)}}{\hat{\mathbf{y}}_{\mathbf{sm}(n)}}}+\frac{\lambda}{2}||\theta||^{2}_{2}$

where $\theta$ is the trainable parameters of the network, $\lambda$ denote
the $l_{2}$ regularization coefficient experimentally set to 0.0001, $N$ is
the batch number, $\mathbf{y_{sm}}_{(n)}$ and
$\hat{\mathbf{y}}_{\mathbf{sm}_{(n)}}$ denote the ground-truth and the network
output in a batch, respectively.

###  II-D Inference with the entire system

Given the predicted labels of Abs-Model at the abstract level and Seg-Model at
the segment level, referred to as $\hat{\mathbf{Y}}_{\mathbf{ab}}$ and
$\hat{\mathbf{Y}}_{\mathbf{sm}}$, the final predicted labels of our proposed
system is defined as:

$\hat{\mathbf{Y}}_{\textbf{final}}=\lambda_{ab}\hat{\mathbf{Y}}_{\mathbf{ab}}+\lambda_{sm}\hat{\mathbf{Y}}_{\mathbf{sm}}$

where $\lambda_{ab}$ and $\lambda_{sm}$ are the hyperparameters to control the
predicted labels at the abstract level and the segment level.

##  III Experiment And Results

###  III-A Datasets

In this paper, we evaluate our proposed deep neural networks on two benchmark
datasets: PubMed RCT [1] and NICTA-PIBOSO [2].

PudMed RCT: This dataset presents the largest and published dataset of text-
based medical scientific abstracts. In particular, the PubMed dataset presents
approximately 200,000 abstracts of randomized controlled trials. Each sentence
of each abstract is labeled with ‘BACKGROUND’, ‘OBJECTIVE’, ‘METHOD’,
‘RESULT’, or ‘CONCLUSION’ which matches its role in the abstract. The PubMed
dataset proposed two sets of PubMed 20K and PubMed 200K, each of which
presents three subsets of Training, Validation, and Test for training,
validation and test processes, respectively.

NICTA-PIBOSO: This dataset is the official dataset of the ALTA 2012 Shared
Task. The task was to build classifiers which automatically divide sentences
to a pre-defined set of categories in the domain of Evidence Based Medicine
(EBM), which are ’BACKGROUND’, ’INTERVENTION’, ’OUTCOME’, ’POPULATION’, ’STUDY
DESIGN’, ’OTHER’.

###  III-B Evaluation metric

TABLE II: Compare our proposed systems with the baseline on the Test set (F1
score/Presision/Recall)

Systems | PubMed 20K | NICTA-PIBOSO
---|---|---
bi-ANN [6] (baseline) | 90.0/-/- | 82.7/-/-
Sen-Model w/ word only | 84.0/84.2/83.9 | 69.9/70.3/69.8
Sen-Model w/ word & char | 84.2/84.2/84.2 | 70.0/70.3/69.8
Sen-Model w/ word & char & stat | 89.5/89.7/89.3 | 76.9/76.9/76.9
Sen-Model w/ pre-trained sentence only | 87.0/87.1/87.0 | 78.5/78.8/78.5
Sen-Model w/ sentence & word & char & stat | 91.1/91.9/90.9 | 81.8/81.8/81.8
Abs-Model w/ word only | 90.6/91.2/90.4 | 81.5/83.4/80.3
Abs-Model w/ word & char | 90.7/91.3/90.5 | 81.4/83.1/80.5
Abs-Model w/ word & char & stat | 91.5/91.8/91.2 | 81.2/82.6/80.1
Abs-Model w/ pre-trained sentence only | 91.9/92.1/91.7 | 82.5/84.0/81.5
Abs-Model w/ sentence & word & char & stat | 92.7/93.2/92.6 | 84.6/85.5/84.1

TABLE III: Compare Our Proposed Systems on Different Levels with the Baseline
on The Test set of Two Benchmark Datasets(F1 score/Presision/Recall)

Systems | PubMed 20K | NICTA-PIBOSO
---|---|---
bi-ANN [6] (baseline) | 90.0/-/- | 82.7/-/-
Sen-Model (Sentence) | 91.1/91.9/90.9 | 81.8/81.8/81.8
Abs-Model (Abstract) | 92.7/93.2/92.6 | 84.6/85.5/84.1
Seg-Model (Segment) | 91.0/92.5/89.6 | 79.5/80.7/78.5
Combine-Model | 92.8/93.4/92.7 | 85.3/86.5/84.5

In this paper, we follow the original paper [1, 2] which proposed the PubMed
RCT and NICTA-PIBOSO datasets. We then use Precision, Recall, and F1 scores as
the evaluation metrics.

###  III-C Experimental settings

We construct our proposed deep neural networks with the Tensorflow framework.
While the deep neural network used for Sen-Model is trained for 30 epochs, we
train Abs-Model and Seg-Model with 60 epochs. All deep neural networks in this
paper are trained with the Titan RTX 24GB GPU. We use the Adam [14] method for
the optimization. The learning rate for Sen-Model, Abs-Model and Seg-Model are
0.001, 0.003 and 0.001, respectively. A reduce learning rate scheme by a
factor of 0.1 is set during training. The Bi-RNN decoder at Abs-Model uses Bi-
LSTM for PudMed dataset and Bi-GRU for NICTA-PIBOSO dataset, respectively. The
hyperparameters $\lambda_{ab}$ and $\lambda_{sm}$ are set to $1$ and $0.2$,
respectively.

###  III-D Experimental results

TABLE IV: Compare our best model with the state-of-the-art systems On Test set
of PudMed 20k dataset

Authors | Systems | F1-score
---|---|---
Yamada et al. [8] | Semi-Markov CRFs | 93.1
Athur Brack et al. [15] | Transfer /Multi-task learning | 93.0
Cohan et al. [16] | Pretrained BERT | 92.9
Xichen Shang et al. [9] | SDLA | 92.8
Jin and Szolovits [7] | HSLN | 92.6
Gaihong Yu et al. [17] | MSM | 91.2
Gonçalves et al. [18] | CNN-GRU | 91.0
Dernoncourt et al. [6] | bi-ANN | 90.0
Agibetov et al. [19] | fastText | 89.6
Our proposed model | Stacked-BiLSTM-CRNN | 92.8

TABLE V: Compare our best model with the state-of-the-art systems on the Test
set of NICTA-PIBOSO dataset

Authors | Systems | F1-score
---|---|---
Xichen Shang et al. [9] | SDLA | 86.8
Athur Brack et al. [15] | Transfer /Multi-task learning | 86.0
Yamada et al. [8] | Semi-Markov CRFs | 84.4
Jin and Szolovits [7] | HSLN | 84.3
Sarker et al. [20] | SVM | 84.1
Cohan et al. [16] | Pretrained BERT | 83.0
Dernoncourt et al. [6] | bi-ANN | 82.7
M Lui [21], [22] | Feature stacking + Metalearner | 82.0
Our proposed model | Stacked-BiLSTM-CRNN | 85.3

We first evaluate our proposed models at the sentence level with different
input features: using only word sequence (Sen-Model w/ word only); using both
word and character sequences (Sen-Model w/ word & char); using word,
character, and statistics (Sen-Model w/ word & char & stat); using only
sentence embeddings extracted from the pre-trained PudMed model (Sen-Model w/
sentence only); using all input features of word, character, statistics,
sentence embeddings (Sen-Model w/ sentence & word & char & stat). The
experimental results shown in the middle part of Table II prove that each
input feature helps to further improve the performance at the sentence level.
The best performance at the sentence level is from a combination of all input
features of word, character, statistics, sentence embeddings (Sen-Model w/
sentence & word & char & stat), presenting the F1/Precision/Recall scores of
91.1/91.9/90.9 and 81.8/81.8/81.8 on PubMed 20K and NICTA-PIBOSO datasets,
respectively. Leveraging sentence embeddings from Sen-model, both models at
the abstract level (Abs-Model) and segment level (Seg-Model) further enhance
the system performance. At the abstract level, we achieve the
F1/Precision/Recall scores of 92.7/93.2/92.6 on PubMed 20K and 84.6/85.5/84.1
on NICTA-PIBOSO datasets. The best system (Combine-Model), which combines of
Abs-Model and Seg-Model, achieves the best result of 92.8/93.4/92.7 on PudMed
20K and 85.3/86.5/84.5 on NICTA-PIBOSO. This model also outperforms the
baseline [6] by 1.0%, 2.8$\%$, and 2.6$\%$ on PubMed 200K, PubMed 20K, and
NICTA-PIBOSO datasets in term of F1 Score, respectively.

Compare with the state-of-the-art systems as shown in Table IV and Table V,
although our best model presents a simple network architecture at the abstract
level and the segment level, we achieve very competitive results (top-4 on
PubMed 20K and top-3 on NICTA-PIBOSO). This indicates that the role of the
Sen-Model at the sentence level is important to achieve the well-represented
sentence embeddings, which can be leveraged in higher levels of segment level
and abstract level. Therefore, our future work is to investigate more
architectures at the abstract level and the segment level for further
improving the model performance.

##  IV Conclusion

This paper has presented a hierarchical deep learning model for the Sequential
Sentence Classfication (SSC) in medical scientific abstract. By conducting
extensive experiments, we achieve the best system which outperforms the
baseline by 1.0%, 2.8%, and 2.6% on the benchmark datasets of PudMed 200K RCT,
PudMed 20K RCT, and NICTA-PIBOSO in term of F1 scores, respectively. The
results are also very competitive to the state-of-the-art systems on these two
datasets. Particularly, our proposed LSTM-based network at the sentence level
plays a vital role in creating a more comprehensive sentence representation
compared with that extracted from other pre-trained language models, which
provides a strong foundation for further explore and improve performance of
the SSC task on higher abstract or segment levels.

## ACKNOWLEDGMENTS

The work described in this paper is performed in the H2020 project STARLIGHT
(“Sustainable Autonomy and Resilience for LEAs using AI against High priority
Threats”). This project has received funding from the European Union’s Horizon
2020 research and innovation program under grant agreement No 101021797.

## References

  * [1] Franck Dernoncourt and Ji Young Lee,  “PubMed 200k RCT: a dataset for sequential sentence classification in medical abstracts,”  in Proceedings of the Eighth International Joint Conference on Natural Language Processing, Nov. 2017, pp. 308–313.
  * [2] Iman Amini, David Martinez, and Diego Molla,  “Overview of the ALTA shared task,”  in Proceedings of the Australasian Language Technology Association Workshop 2012, 2012, pp. 124–129.
  * [3] Jimmy Lin, Damianos Karakos, Dina Demner-Fushman, and Sanjeev Khudanpur,  “Generative content models for structural analysis of medical abstracts,”  in Proceedings of the HLT-NAACL BioNLP Workshop on Linking Natural Language and Biology, 2006, pp. 65–72.
  * [4] Patrick Ruch et al.,  “Using argumentation to extract key sentences from biomedical abstracts,”  International Journal of Medical Informatics, vol. 76, no. 2, pp. 195–200, 2007.
  * [5] Su Nam Kim, David Martinez, Lawrence Cavedon, and Lars Yencken,  “Automatic classification of sentences to support evidence based medicine,”  in BMC bioinformatics, 2011, vol. 12, pp. 1–10.
  * [6] Franck Dernoncourt, Ji Young Lee, and Peter Szolovits,  “Neural networks for joint sentence classification in medical paper abstracts,”  in Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics, 2017, pp. 694–700.
  * [7] Di Jin and Peter Szolovits,  “Hierarchical neural networks for sequential sentence classification in medical scientific abstracts,”  in Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2018, pp. 3100–3109.
  * [8] Kosuke Yamada, Tsutomu Hirao, Ryohei Sasano, Koichi Takeda, and Masaaki Nagata,  “Sequential span classification with neural semi-Markov CRFs for biomedical abstracts,”  in Findings of the Association for Computational Linguistics: EMNLP, 2020, pp. 871–877.
  * [9] Xichen Shang, Qianli Ma, Zhenxi Lin, Jiangyue Yan, and Zipeng Chen,  “A span-based dynamic local attention model for sequential sentence classification,”  in Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, 2021, pp. 198–203.
  * [10] Yifan Peng, Shankai Yan, and Zhiyong Lu,  “Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets,”  in Proceedings of the 18th BioNLP Workshop and Shared Task, 2019, pp. 58–65.
  * [11] Jeffrey Pennington, Richard Socher, and Christopher Manning,  “GloVe: Global vectors for word representation,”  in in Proc. EMNLP, 2014, pp. 1532–1543.
  * [12] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin,  “Attention is all you need,” 2023.
  * [13] Yu Gu et al.,  “Domain-specific language model pretraining for biomedical natural language processing,”  ACM Transactions on Computing for Healthcare (HEALTH), vol. 3, no. 1, pp. 1–23, 2021.
  * [14] P. K. Diederik and B. Jimmy,  “Adam: A method for stochastic optimization,”  CoRR, vol. abs/1412.6980, 2015.
  * [15] Arthur Brack et al.,  “Sequential sentence classification in research papers using cross-domain multi-task learning,”  International Journal on Digital Libraries, pp. 1–24, 2024.
  * [16] Arman Cohan, Iz Beltagy, Daniel King, Bhavana Dalvi, and Dan Weld,  “Pretrained language models for sequential sentence classification,”  in Proceedings of the Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 2019, pp. 3693–3699.
  * [17] Gaihong Yu, Zhixiong Zhang, Huan Liu, and Liangping Ding,  “Masked sentence model based on bert for move recognition in medical scientific abstracts,”  Journal of Data and Information Science, vol. 4, no. 4, pp. 42–55, 2019.
  * [18] Sérgio Gonçalves, Paulo Cortez, and Sérgio Moro,  “A deep learning classifier for sentence classification in biomedical and computer science abstracts,”  Neural Comput. Appl., vol. 32, no. 11, pp. 6793–6807, 2020.
  * [19] Asan Agibetov, Kathrin Blagec, Hong Xu, and Matthias Samwald,  “Fast and scalable neural embedding models for biomedical sentence classification,”  BMC bioinformatics, vol. 19, pp. 1–9, 2018.
  * [20] Abeed Sarker, Diego Mollá, and Cécile Paris,  “An approach for automatic multi-label classification of medical sentences,”  in Proceedings of the 4th International Louhi Workshop on Health Document Text Mining and Information Analysis. Sydney, NSW, Australia, 2013.
  * [21] Marco Lui,  “Feature stacking for sentence classification in evidence-based medicine,”  in Proceedings of the Australasian Language Technology Association Workshop, 2012, pp. 134–138.
  * [22] Diego Mollá,  “Overview of the alta shared task: Piboso sentence classification, 10 years later,”  in Proceedings of the The 20th Annual Workshop of the Australasian Language Technology Association, 2022, pp. 178–182.