# ATSumm: Auxiliary information enhanced approach for abstractive disaster
Tweet Summarization with sparse training data

Piyush Kumar Garg  Roshni Chakraborty  Sourav Kumar Dandapat

###### Abstract

The abundance of situational information on Twitter poses a challenge for
users to manually discern vital and relevant information during disasters. A
concise and human-interpretable overview of this information helps decision-
makers in implementing efficient and quick disaster response. Existing
abstractive summarization approaches can be categorized as sentence-based or
key-phrase-based approaches. This paper focuses on sentence-based approach,
which is typically implemented as a dual-phase procedure in literature. The
initial phase, known as the extractive phase, involves identifying the most
relevant tweets. The subsequent phase, referred to as the abstractive phase,
entails generating a more human-interpretable summary. In this study, we adopt
the methodology from prior research for the extractive phase. For the
abstractive phase of summarization, most existing approaches employ deep
learning-based frameworks, which can either be pre-trained or require training
from scratch. However, to achieve the appropriate level of performance, it is
imperative to have substantial training data for both methods, which is not
readily available. This work presents an Abstractive Tweet Summarizer (ATSumm)
that effectively addresses the issue of data sparsity by using auxiliary
information. We introduced the Auxiliary Pointer Generator Network (AuxPGN)
model, which utilizes a unique attention mechanism called Key-phrase
attention. This attention mechanism incorporates auxiliary information in the
form of key-phrases and their corresponding importance scores from the input
tweets. We evaluate the proposed approach by comparing it with $10$ state-of-
the-art approaches across $13$ disaster datasets. The evaluation results
indicate that ATSumm achieves superior performance compared to state-of-the-
art approaches, with improvement of $4-80\%$ in ROUGE-N F1-score.

###### keywords:

Disasters , Abstractive summarization , Social media , Crisis scenario ,
Pointer generator network model , Deep learning , Data sparsity

††journal: Arxiv

[label1]organization=Computer Science & Engineering Department, Indian
Institute of Technology Patna,state=Bihar, country=India

[label2]organization=Institute of Computer Science, University of Tartu,
country=Estonia

##  1 Introduction

During a disaster, emergency responders, government agencies, and non-
governmental organizations (NGOs) require event information to ensure quick
and effective support. Social media platforms, like Twitter, play an important
role in providing information during disaster events. However, there are
several challenges in identifying relevant information from these tweets
manually, such as the huge number of tweets and the informal language.
Therefore, existing research works [1, 2, 3, 4] have proposed disaster tweet
summarization approaches to automatically process these huge number of tweets
to generate a concise summary.

Existing disaster tweet summarization approaches are either extractive [4, 5,
3, 6] or abstractive approaches [7, 8, 9, 10]. Extractive approaches aim to
identify the most relevant tweets from a set of input tweets, ensuring
comprehensive information coverage while minimizing redundancy. On the other
hand, abstractive summarization approaches strive to generate a more human-
interpretable summary while still maintaining comprehensive information
coverage and minimizing redundancy. Existing abstractive summarization
approaches on the disaster can be categorized based on the presentation of the
final summary into two categories: 1) Sentence-based presentation [8, 9, 7,
10, 11] where the final summary consists of a number of full-length sentences;
and 2) Key-phrase-based presentation [12, 13] where the final summary consists
of key-phrases. In this paper, we focus on generating a sentence-based
abstractive summary.

Table 1: Table shows F1-score of ROUGE-1, 2, and L of the summaries generated
by various baselines on CNN/Daily Mail dataset. [Note: “AuxInfo” represents
the auxiliary information of tweets in terms of tweets key-phrases and their
importance scores.]

Approach | ROUGE-1 | ROUGE-2 | ROUGE-L
---|---|---|---
| F1-score | F1-score | F1-score
PGN [14] \+ AuxInfo (100% training data) | 0.39 | 0.19 | 0.42
PGN [14] \+ AuxInfo (50% training data) | 0.38 | 0.18 | 0.41
PGN [14] \+ AuxInfo (25% training data) | 0.39 | 0.18 | 0.41
PGN [14] \+ AuxInfo (10% training data) | 0.38 | 0.17 | 0.40
PGN [14] \+ AuxInfo (1% training data) | 0.37 | 0.17 | 0.40
PGN [14] (100% training data) | 0.31 | 0.10 | 0.34
PGN [14] (50% training data) | 0.31 | 0.11 | 0.35
PGN [14] (25% training data) | 0.30 | 0.09 | 0.36
PGN [14] (10% training data) | 0.28 | 0.08 | 0.32
PGN [14] (1% training data) | 0.27 | 0.07 | 0.30
PGN [14] | 0.31 | 0.10 | 0.34
PTESumm [11] | 0.28 | 0.08 | 0.32
PEGASUS [15] | 0.28 | 0.08 | 0.29
Longformer [16] | 0.35 | 0.14 | 0.39
ProphetNet [17] | 0.25 | 0.09 | 0.27
T5 [18] | 0.36 | 0.15 | 0.38
BART [19] | 0.37 | 0.16 | 0.40

Existing sentence-based abstractive disaster tweet summarization approaches
[8, 9, 7, 10, 11] generate the final summary in two subsequent phases: 1)
Phase-I, which is an extractive phase; and 2) Phase-II, which is an
abstractive phase. Existing unsupervised approaches [8, 9, 7] used for Phase-I
could not cluster tweets properly due to the overlapping keywords between
clusters, while the existing supervised approach [10] used for Phase-I
requires a huge amount of labelled data, which incurs cost as well as time.
However, a supervised method by Garg et al. [20] handles the sparsity of the
training data by augmenting auxiliary information in terms of key-phrases of
tweets. So for the extractive phase of the proposed approach, we directly
adopt the methodology proposed by Garg et al. [20]. For Phase-II, i.e.,
abstractive-phase, in literature, there are either graph-based approach [10]
or deep learning-based approaches [18, 14]. For the graph-based approach, we
find the work proposed by Rudra et al. [10]. The advantages of the graph-based
approach is that it does not depend on the availability of training dataset
and effectively handles disaster-specific information. For example, Rudra et
al. [10] generate the summary sentences by traversing the constructed word
graph, where nodes are bigrams (adjacent words), and edges are bigrams
relationships between two nodes. However, this approach does not consider
semantic similarity among words while creating the word graph, which can
inherently lead to redundancy in the summary. On the other hand, existing
deep-learning-based approaches can be further categorised into: 1) Pre-trained
model-based approaches and 2) Approaches where the models are trained from
scratch. The problem with using a pre-trained model for abstractive
summarization is that the model is trained with a very generic corpus, which
fails to handle domain-specific challenges for summary creation. For example,
Vitiugin et al. [8] utilizes a pre-trained T5 model, which fails to ensure
high performance as they can not handle disaster-specific challenges. On the
other hand, the only methodology found in literature used for abstractive
summarization by training the model from scratch is the sequence-to-sequence
(Seq2Seq) model. To train an existing Seq2Seq model from scratch requires a
good amount of training data, which is not readily available. To handle the
unavailability of the disaster-specific training data, Lin et al. [9] trains a
Seq2Seq model, i.e., PGN [14] model using available CNN/Daily Mail news
dataset. However, due to the training on non-disaster-specific data, it does
not give the desired performance in result. Therefore, our main aim of this
work is to modify the existing PGN architecture so that it can handle the
sparse data problem by utilizing the domain knowledge.

Deep learning-based architecture learns the feature weight through the
repeated feedback mechanism. Given a sufficient size of training data, a deep
learning-based framework can automatically identify and learn the importance
of features. However, in the absence of sufficient training data, it may be
difficult for the model to identify and learn the features importance.
Therefore, in this paper, we try to address the above limitation by computing
features importance (key-phrase and its importance) separately using ontology
and providing it as the input to the deep learning model. There exists deep-
learning-based architecture, which uses this kind of hybrid model where
auxiliary information is provided by handcrafted features [21, 22], we are
adopting the same to solve the sparse data problem in this paper. The
effectiveness of auxiliary information is verified using a small experiment,
where we use a fraction of CNN/Daily Mail dataset [23] along with auxiliary
information to summarize documents related to a news domain. As shown in Table
1, experiment results clearly indicate that it can produce a comparable
summary even when only $1$% of training data is used with auxiliary
information.

In this paper, we propose a Two-Phase approach, ATSumm (Abstractive Tweet
Summarizer), specifically designed to create a summary of a disaster event,
where in Phase-I, we follow Garg et al. [20] to identify the most relevant
tweets automatically by effectively utilizing augmenting auxiliary information
as domain knowledge from an existing disaster ontology. For Phase-II, we
propose an extended version of the PGN model, which we refer to as a AuxPGN
(Auxiliary Pointer Generator Network) model that utilizes auxiliary
information in the form of key-phrases from the extracted tweets along with
their importance score to ensure the AuxPGN model effectively work with the
sparse training data. The novelty of AuxPGN is that it utilizes a novel
attention mechanism referred to as Key-phrase attention guided by source
tweets as well as its key-phrases and their importance scores as auxiliary
information generated by DRAKE [20]. The key-phrase generated by DRAKE
consists of the most relevant phrase (key-phrase) related to a disaster event.
ATSumm can ensure better performance compared to existing state-of-the-art
abstractive summarization approaches by $4-61$%, $7-80$%, and $4-64$%, in
terms of ROUGE-1, 2, and L, respectively. We also perform ablation experiments
where we analyze the performance of each phase of ATSumm with $10$ different
variants of ATSumm to justify the model of each phase in ATSumm. We provide a
human evaluation of the effectiveness of ATSumm in the abstractive summary
generation based on five different quality metrics, i.e., Fluency,
Readability, Conciseness, Relevance, and Non-redundancy. Finally, through
experimental analysis, we show that ATSumm outperforms the five existing
transformer models fine-tuned by $3.92-33.33$% in terms of ROUGE-N F1-score on
the ARIES dataset, which we discussed in the Subsection 5.8.

Our major contributions can be summarized as follows:

  1. 1.

To the best of our knowledge, our proposed ATSumm is the first model on
abstractive disaster tweet summarization, which utilizes auxiliary information
in a deep-learning framework to handle the problem of sparse training data.

  2. 2.

We propose a novel attention mechanism, Key-phrase attention that incorporates
the tweet’s key-phrases and their importance scores as auxiliary information
into the proposed AuxPGN model.

  3. 3.

We prepare a new ARIES (AbstRactive dIsaster twEet Summ-arization) dataset. To
the best of our knowledge, ARIES is the first disaster domain training dataset
for the AuxPGN/PGN model, containing $7500$ training samples where each
training sample comprises $400$ input words and corresponding $200$ words
output summary.

  4. 4.

Additionally, we prepare and release the ground-truth summaries of $13$
different disaster events belonging to the different types and locations (used
for evaluation).

  5. 5.

Evaluation results clearly indicate that the proposed approach, ATSumm,
outperforms existing state-of-the-art abstractive summarization approaches for
all $13$ disaster events.

Rest of the paper is organized as follows. In Section 2, we discuss the
existing research works of abstractive tweet summarization. Section 3
discusses the ARIES training dataset and evaluation datasets in detail. In
Section 4, we discuss our proposed approach in detail, followed by experiments
and comparison results in Section 5. We discuss the ablation experiments in
Subsection 5.4. Finally, we conclude our paper and provide future works in
Section 6.

##  2 Related Works

Summarization refers to creating a short and concise overview of an input text
while preserving all its important key information [24, 25, 26, 27, 28].
Several researchers have proposed automatic text summarization approaches for
various domains, including legal documents summarization [29, 30], news
documents summarization [31, 32, 33], scientific documents summarization [34,
35], tweets summarization [36, 37], product reviews summarization [38, 39]
etc. Social media platforms, such as Twitter, have gained prominence as a
means for users to share personal updates, opinions, and information about a
real-time event. For real-time updates, Twitter is considered one of the best
social media platforms [40]. However, the huge amount of information posted on
Twitter makes it very difficult for any human being to get relevant
information. Therefore, the research community devoted a lot of attention to
automatic tweet summarization.

While all tweet summarization approaches share basic objectives, such as
maximizing coverage and decreasing redundancy, the specific knowledge required
may vary depending on the domain of the summarizer. Domain expertise enables
the summarizer to comprehend the significance of a certain keyword or phrase.
The research community proposed automatic summarization approaches for
different domains, such as sports game tweets summarization [41, 42, 43], news
tweets summarization [44, 45, 37], disaster tweet summarization [4, 46, 20],
political event tweets summarization [47, 48], etc. Existing disaster tweet
summarization approaches can be classified into extractive and abstractive.
Existing abstractive summarization approaches on the disaster can be further
categorized based on the generation of the final summary into two classes: 1)
Sentence-based [8, 9, 7, 10, 11] where the final summary consists of a number
of full-length sentences; and 2) Key-phrase-based [12, 13] where the final
summary consists of key-phrases. In this work, we only focus on an abstractive
sentence-based approach, and therefore, we discuss only the related work of
abstractive sentence-based disaster tweet summarization approaches in detail.
Existing approaches for abstractive sentence-based disaster tweet
summarization are either unsupervised or supervised. We discuss each of them
next.

###  2.1 Unsupervised sentence-based approaches

Existing unsupervised abstractive sentence-based tweet summarization
approaches are graph-based [10]. These approaches initially construct a word
graph in which nodes are words and edges are between consecutive words and
then traverse the word graph to generate new sentences. In the final summary,
it selects the generated new sentences that provide the most information
coverage and diversity. For example, Rudra et al. [10] propose an extractive-
abstractive tweet summarization framework for disaster events where they
initially identify the set of important tweets based on the importance of
their content words (i.e., nouns, verbs, adjectives, and numerals). Further,
from important tweets, they created a word-graph where nodes are bigrams
(adjacent words), and edges are between consecutive words. Finally, they
generate sentences by word-graph traversal and create a summary by selecting
those sentences into the summary, which provides maximum information coverage
and diversity in the final summary using the Integer Linear Programming (ILP)
objective function. However, this approach does not consider semantical
similarity among words while creating the word graph, which can inherently
lead to redundancy in the summary.

###  2.2 Supervised sentence-based approaches

Recent abstractive supervised tweet summarization approaches are based on
sequence-to-sequence problems. These abstractive models/approaches are
classified into two categories based on their training strategy: 1) model
training from scratch and 2) pre-trained model. We next discuss each of these
categories in detail.

####  2.2.1 Abstractive models used as a pre-trained model

The introduction of transformer architectures has revolutionized the field of
automatic document summarization, as they have been shown to be highly
effective in various natural language Seq2Seq tasks. Microsoft’s Unified
Pretrained Language Model (UniLM) [49] stands out as one of the leading
Transformer-based models. UniLM, designed by Microsoft, adopts a model
architecture similar to BERT (large) [50] and is pre-trained using English
Wikipedia and BookCorpus [51]. Another important model is ProphetNet [17],
which addresses the traditional Seq2Seq optimization problem by predicting the
n-next tokens instead of a single token at a time. The next model is Pegasus
[15], which uses a novel summarization-specific pre-training objective called
Gap Sentence Generation (GSG) that involves masking out important sentences
from input sentences and then generating those sentences from the remaining
sentences. The next model is the Text-to-Text Transfer Transformer (T5) model
introduced by Google [18], which utilizes transfer learning on the basic
transformer architecture proposed by Vaswani et al. [52]. Further, the
Bidirectional Auto-Regressive Transformers (BART) model introduced by Facebook
[19] utilizes a denoising autoencoder that is trained on corrupted text with
an arbitrary noising function to learn the reconstruction of the input text.
The final model is Longformer-Encoder-Decoder (LED) [16], which utilizes a
local windowed attention mechanism to only attend to a small window of tokens
at a time and compute the attention weights.

Existing abstractive tweet summarization work [8] follows a two-phase
architecture to generate a meaningful summary, where the first phase is
extractive, and the second is abstractive. In the extractive phase, it
initially identifies different clusters using a K-means clustering-based
approach and then selects the representative tweets from each cluster. In the
abstractive phase, it utilizes diversification operations to create a human-
interpretable summary of the tweets related to a disaster event by using a
pre-trained T5 model. However, the pre-trained model is trained with a very
generic corpus, which fails to handle domain-specific challenges for summary
creation.

####  2.2.2 Abstractive models training from scratch

Nallapati et al. [53] proposed an Attention-based Encoder-Decoder model (AED),
one of the first deep learning architectures to demonstrate efficacy in
abstractive summarization. The attention mechanism relies upon the concept
that only specific input portions are relevant during each generation step.
Therefore, to take advantage of the AED architecture, See et al. [14] propose
a Pointer Generator Network (PGN) model, which solves the Out-Of-Vocabulary
(OOV) word problem by allowing the summarizer to copy words from the input
text and generate new text.

Recent existing sentence-based abstractive tweet summarization approaches [9,
11] follow a two-phase architecture to create a more human interpretable
summary where the first phase is extractive, and the second is abstractive. In
the extractive phase, they intend to extract a set of important tweets from
the input tweets using either a clustering-based approach [9] or a ranking-
based approach [11] that ensures maximum information coverage without
redundancy. However, the clustering-based approach could not cluster tweets
properly due to the overlapping keywords between clusters, while the ranking
method is very generic, which fails to compute the proper importance of a
tweet with respect to a disaster. In the abstractive phase, they take the
output of the extractive phase as an input and then create a concise and more
human-interpretable summary of a predefined length using the existing PGN
model. They utilize the available CNN/Daily Mail dataset [23, 53] to train the
PGN model. However, the PGN model requires huge amounts of training data,
which is difficult to obtain in disaster domains where we have sparse data for
training. Therefore, to address this limitation, we proposed an
extended/modified version of the PGN model as Auxiliary Pointer Generator
Network (AuxPGN) model that utilizes the auxiliary information (i.e., key-
phrases and their importance scores) extracted from the input text and
provides this auxiliary information through novel key-phrase attention with
input text while training the AuxPGN model. As a result, the AuxPGN model does
not require extensive training data and performs better with various start-of-
the-art approaches for abstractive summary generation.

##  3 Datasets

In this Section, we initially discuss the AbstRactive dIsaster twEet
Summarization (ARIES) dataset used for training. Then we discuss $13$
evaluation datasets belonging to different disaster types and locations.

###  3.1 Training Dataset Details

Although the existing research works [10, 8] have provided abstractive ground-
truth summaries for six disaster events. All these works consider all the
tweets for a disaster event as input tweets, and the corresponding summary
consists of $250$ words. However, the PGN/AuxPGN model expects an input of
size $400$ words and produces a summary output of $200$ words. Hence, we can
not use the existing datasets with their ground-truth summaries for training
in our proposed AuxPGN model. Therefore, in this paper, we manually create a
training dataset to train the AuxPGN model. To prepare a training dataset, we
found $30$ available disaster datasets [54, 55, 56], which do not have any
available ground-truth summaries. For each disaster dataset, we follow the
method proposed by Nallapati et al. [53] to create a training dataset. We use
all the tweets of a dataset as input and generate several chunks (samples) of
$400$ words. Finally, we got a total of $7500$ samples from all the $30$
disaster events. We ask two human annotators to write a summary of each sample
manually based on their understanding and wisdom. We select these annotators
through the Quality Assessment Evaluation proposed by Garg et al. [46]. The
annotators are undergraduate students between the ages of $20-30$ who are
fluent in English, are regular Twitter111https://twitter.com users, and are
not part of this project. To ensure a high-quality ground-truth summary, we
follow a two-step annotation procedure, wherein in the first step, an
annotator goes through all the given words of a sample and then writes a
summary of approximately but not more than $200$ words in length that includes
all the relevant information related to that event with minimum repetition.
Then, in the second step, another annotator who did not participate in the
first step reads all the given words of a sample and the first annotator’s
annotated summary and then corrects the already annotated summary to create a
final summary if it contains irrelevant or incorrect information.

Table 2: Table shows the details of $D_{1}$-$D_{13}$ datasets.

Num | Dataset Name | Year | # Tweets | Continent | Disaster type
---|---|---|---|---|---
$D_{1}$ | Sandy Hook Elementary School Shooting | 2012 | 2080 | USA | Man-made
$D_{2}$ | Uttrakhand Flood | 2013 | 2069 | Asia | Natural
$D_{3}$ | Hagupit Typhoon | 2014 | 1461 | Asia | Natural
$D_{4}$ | Hyderabad Blast | 2013 | 1413 | Asia | Man-made
$D_{5}$ | Harda Twin Train Derailment | 2015 | 4171 | Asia | Man-made
$D_{6}$ | Los Angeles Airport Shooting | 2013 | 1409 | USA | Man-made
$D_{7}$ | Matthew Hurricane | 2016 | 1654 | USA | Natural
$D_{8}$ | Puebla Mexico Earthquake | 2017 | 2015 | USA | Natural
$D_{9}$ | Pakistan Earthquake | 2019 | 1958 | Asia | Natural
$D_{10}$ | Midwestern U.S. Floods | 2019 | 1880 | USA | Natural
$D_{11}$ | Kaikoura Earthquake | 2016 | 2195 | Oceania | Natural
$D_{12}$ | Cyclone Pam | 2015 | 1508 | Oceania | Natural
$D_{13}$ | Greece Wildfire | 2018 | 1282 | Europe | Natural

###  3.2 Evaluation Dataset and Pre-processing:

We evaluate the performance of ATSumm on $13$ different disaster datasets,
which as are follows:

  1. 1.

$D_{1}$: This dataset is prepared by Dutta et al. [6] on the basis of the
Sandy Hook Elementary School
Shooting222https://en.wikipedia.org/wiki/Sandy_Hook_Elementary_School_shooting
on December, $2012$ in which around $20$ people were killed by a $20$-year-old
shooter.

  2. 2.

$D_{2}$: This dataset is also prepared by [6] on the basis of the dreadful
floods from the Uttarakhand
Flood333https://en.wikipedia.org/wiki/2013_North_India_floods on June, $2013$
in which $5,700$ people were died and around $128$ houses were damaged.

  3. 3.

$D_{3}$: This dataset is also prepared by [6] on the basis of the Hagupit
Typhoon444https://en.wikipedia.org/wiki/Typhoon_Hagupit_(2014) on December,
$2014$ in which $18$ people were died and caused an estimated damage were
$$114$ million USD.

  4. 4.

$D_{4}$: This dataset is also prepared by [6] on the basis of the two
consecutive bomb blast from the Hyderabad
Blast555https://en.wikipedia.org/wiki/2013_Hyderabad_blasts on February,
$2013$ in which $18$ people were killed and more than $131$ people were
injured.

  5. 5.

$D_{5}$: This dataset is prepared by [57] on the basis of a massive train
incident from the Harda Twin Train
Derailment666https://en.wikipedia.org/wiki/Harda_twin_train_derailment on
August, $2015$ in which $31$ people were died and more than $100$ people were
injured.

  6. 6.

$D_{6}$: This dataset is prepared by [58] on the basis of a terrorist attack
from the Los Angeles Airport
Shooting777https://en.wikipedia.org/wiki/2013_Los_Angeles_International
_Airport_shooting on November $2013$, in which one person was killed and more
than $15$ people were injured.

  7. 7.

$D_{7}$: This dataset is prepared by [54] on the basis of a devastating impact
of a Hurricane Matthew888https://en.wikipedia.org/wiki/Hurricane_Matthew on
October, $2016$ in which $603$ people were died and caused an estimated damage
were $$2.8$ billion USD.

  8. 8.

$D_{8}$: This dataset is also prepared by [54] on the basis of a massive
earthquake from the Puebla Mexico
Earthquake999https://en.wikipedia.org/wiki/2017_Puebla_earthquake on
September, $2017$ in which $370$ people was died and more than $6000$ people
were injured.

  9. 9.

$D_{9}$: This dataset is also prepared by [54] on the basis of a massive
earthquake from the Pakistan
Earthquake101010https://en.wikipedia.org/wiki/2019_Kashmir_earthquake on
September, $2019$ in which $40$ people were died and around $319$ houses were
damaged.

  10. 10.

$D_{10}$: This dataset is also prepared by [54] on the basis of a dreadful
floods from the Midwestern U.S.
Floods111111https://en.wikipedia.org/wiki/2019_Midwestern_U.S._floods in which
around $14$ million people were affected and caused approximately damage were
$$2.9$ billion USD.

  11. 11.

$D_{11}$: This dataset is also prepared by [54] on the basis of a massive
earthquake from the Kaikoura
Earthquake121212https://en.wikipedia.org/wiki/2016_Kaikoura_earthquake on
November, $2016$ in which $2$ people were died and more than $57$ people were
injured.

  12. 12.

$D_{12}$: This dataset is prepared by [59] on the basis of a devastating
impact of the Cyclone Pam131313https://en.wikipedia.org/wiki/Cyclone_Pam on
March, $2015$ in which $15$ people were died and around $3300$ people were
displaced.

  13. 13.

$D_{13}$: This dataset is also prepared by [54] on the basis of a mass-fire
from the Greece
Wildfire141414https://en.wikipedia.org/wiki/2018_Attica_wildfires on July,
$2018$ in which $4$ people were died and around 4000 homes/buildings were
damaged.

Table 3: Table lists key notations and their descriptions used in ATSumm.

Notation | Description
---|---
$D$ | Disaster event dataset
$T$ | A set of tweets in $D$
$m$ | Desired length summary (i.e., $200$ words)
$T^{{}^{\prime}}$ | Set of tweets extracted from Phase-I
$N$ | Number of the words in $T^{{}^{\prime}}$
$h_{i}$ | Encoder hidden state
$x_{t}$ | Decoder input
$s_{t}$ | Decoder state
$V$ | Fixed length vocabulary
$KP_{i}$ | Key-phrase of $i^{th}$ indexed tweet in $T^{{}^{\prime}}$
$a^{t}$ | Key-phrase attention distribution
$\gamma^{t}$ | Key-phrase word vector
$e^{t}$ | Precursor attention vector
$W_{i}$ | Word probability vector of $KP_{i}$
$h^{*}_{t}$ | A context vector
${P_{V}}$ | Output word probability distribution of all $W\in V$
$P_{gen}$ | Generation probability
$c^{t}$ | A coverage vector
$L_{t}$ | A loss function
$DRAKE_{Score}(KP_{i})/T$ | DRAKE score of $i^{th}$ key-phrase in $T$

Dataset Pre-processing: For pre-processing, we remove URLs, emoticons,
punctuation marks, and stop words. Further, we remove hashtags and usernames
as we only consider the Twitter text. We also observe that lengths less than
$3$ characters words do not provide any relevant information specific to
disasters [60], so we also remove these words. We also perform lowercase
conversion of the dataset text. We prepare the ground-truth summary of $200$
words in length of these $13$ datasets with the help of the two-step
annotation procedure discussed in Subsection 3.1 and show an overview of these
datasets in Table 2.

##  4 Proposed Approach

In this Section, we elaborate proposed approach in detail.

###  4.1 Model Overview

The proposed approach, ATSumm, comprises of two phases. In Phase-I, we
identify the most relevant tweets on the basis of disaster domain knowledge as
auxiliary information to ensure maximum information coverage and diversity. In
Phase-II, we utilize the extracted tweets from Phase-I as input to generate
new sentences in a more human-interpretable manner, while maintaining the
information coverage and minimizing redundancy. We propose an Auxiliary
Pointer Generator Network (AuxPGN) model that utilizes the tweets’ key-phrases
and their importance scores through our proposed Key-phrase attention
mechanism, and therefore, can ensure maximum disaster relevant information
coverage in summary. Therefore, our proposed AuxPGN requires less data for
training as it learns the most important information related to a disaster
event through the importance of key-phrases. We show the overview of ATSumm in
Figure 1 and show all the used notations and corresponding descriptions in
Table 3.

Figure 1: Figure shows an overview of ATSumm. Figure 2: Figure shows the
proposed architecture of AuxPGN model.

###  4.2 Phase-I

In this phase, we follow Garg et al. [20] to automatically identify the most
relevant tweets related to a disaster event that can captures the maximum
information coverage and ensures diversity in summary. Although there are
several approaches available [4, 5, 61, 6], which identify the most relevant
tweets, we follow Garg et al. [20] for its inclusion of auxiliary information
as domain-specific knowledge. Additionally, Garg et al. [20] can ensure the
best performance, i.e., $0.47-0.61$ ROUGE-1 F1-score, without any manual
intervention. We got a ranked list of tweets as an output of [20] approach on
the basis of their importance and diversity scores, and then we select the
top-ranked tweets until the word count reached $400$ [14]. We now provide
these tweets as input to Phase-II for abstractive summary generation.

###  4.3 Phase-II

In Phase-II, we aim to generate new sentences as a summary, which is more
human-interpretable from the tweets extracted from Phase-I. Existing
abstractive tweet summarization approaches [9, 11] have proposed PGN [62]
model to generate a human-interpretable summary. The PGN-based Seq2Seq model
outperforms traditional Seq2Seq models for summarization by handling the out-
of-vocabulary word problem. Hence, we chose the PGN-based model over other
Seq2Seq models. However, the PGN requires a huge amount of training data,
which is very difficult to obtain for disaster datasets. To handle this
challenge, we propose integration of the most relevant auxiliary information
with respect to each disaster dataset to PGN, namely the proposed AuxPGN
Model. Therefore, we ensure significant reduction in the training dataset
size. Additionally, by integration of disaster-specific most relevant
auxiliary information, we can ensure a high increase in performance specific
to each disaster event without incurring any human intervention. In AuxPGN, we
introduce the tweet’s key-phrases and their importance scores as auxiliary
information generated by the DRAKE [20]. Additionally, we introduce a novel
Key-phrase attention, which utilizes this auxiliary information into the
AuxPGN, which helps reduce the training data size required for the AuxPGN
model. We show an overview of the AuxPGN model in Figure 2 and describe the
architecture next by expanding on details regarding the constituent
components, Encoder and Decoder (Subsection 4.3.1), Key-Phrase Attention
(Subsection 4.3.2), Pointer Generator (Subsection 4.3.3), Coverage Mechanism
(Subsection 4.3.4), and Decoding (Subsection 4.3.5).

####  4.3.1 Encoder and Decoder

For pre-processing, we use Stanford
CoreNLP151515https://stanfordnlp.github.io/CoreNLP/ tokenizer for
tokenization. We feed these tweet tokens to the encoder layer one-by-one.
Although there are several encoders available, such as RNN [63], LSTM [64],
and GRU [65], we choose a single-layer Bi-directional Long Short-Term Memory
(BiLSTM) [66] model. BiLSTM is able to capture long-term dependencies and
handles sequential data to generate concise and contextually relevant
summaries [67]. The encoder output is a sequence of encoder hidden states
$h_{i}$, and each hidden state is a concatenation of both the forward and
backward hidden states, as in [68]. For decoder, we employ a single-layer Long
Short-Term Memory (LSTM) [64] model, which receives the word embedding of the
preceding word to decide whether to generate a word from the vocabulary or
copy a word from the input sequence to create a summary.

####  4.3.2 Key-Phrase Attention

We propose a novel key-phrase attention distribution $a^{t}$, which is a
combination of the usual attention weights as in [68] and a key-phrase word
vector derived from a key-phrase model. We follow Garg et al. [20] for key-
phrase identification as it does not require any training data. It utilizes
the domain knowledge from existing ontology, Empathi [69] to identify the key-
phrase $KP$ of each tweet in $T^{{}^{\prime}}$, where $T^{{}^{\prime}}$ is a
set of tweets extracted from Phase-I. We calculate a key-phrase word vector,
$\gamma^{t}$ of $|V|$-dimension (where $V$ is a fixed length vocabulary)
considering all the words in $T^{{}^{\prime}}$ as:

$\gamma^{t}=\begin{cases}\sum\limits_{i}(DRAKE_{Score}(KP_{i})/T^{{}^{\prime}}).W_{i},\quad
If\>W_{i}\in KP_{i}\\\ 0,\>elsewhere.\end{cases}\quad$ (1)

where $DRAKE_{Score}(KP_{i})/T^{{}^{\prime}}$ is the score of $i^{th}$ key-
phrase in $T^{{}^{\prime}}$ by [20] and $W_{i}$ is a $|V|$ dimensional vector
represents word probabilities of $KP_{i}$ calculated as $W_{i,j\in
KP_{i}}=P(word_{j}/KP_{i})$ in vocabulary $|V|$. For each decoding step $t$,
we compute a final attention distribution, i.e., key-phrase attention
distribution $a^{t}\in R^{N}$ ($N$ is the number of words in
$T^{{}^{\prime}}$), which is the combination of both precursor attention
vector, $e^{t}\in R^{N}$ and key-phrase word vector, $\gamma^{t}$ as:

$e^{t}={v^{T^{{}^{\prime}}}}{tanh({W_{h}}{h_{k}}+{W_{s}}{s_{t}}+b_{att})}$ (2)

$a^{t}=f(e^{t},{\gamma}^{t})$ (3)

where $h_{k}\in R^{n}$ represents $k^{th}$ encoder hidden state, $s_{t}\in
R^{l}$ represents decoder state, and $v$, $W_{h}$, $W_{s}$, $b_{att}$ are the
learnable parameters. Further, we use function $f$ to combine both the
precursor attention vector and the key-phrase word vector as:

$f=w_{1}*softmax(e^{t})+w_{2}*softmax(\bar{\gamma}^{t})$ (4)
$\textrm{s.t.}\quad w_{1}+w_{2}=1\quad and\quad w_{1},w_{2}\in(0,1)$ (5)

where $w_{1}$ and $w_{2}$ are the weights of $e^{t}$ and $\bar{\gamma}^{t}$,
respectively, and we decide these weights through an experiment analysis
discussed in Subsection 5.7. $\bar{\gamma}^{t}\in R^{N}$ is a reduced key-
phrase word vector of size $N$, created by selecting the $N$ terms of
${\gamma}^{t}$. This attention distribution instructs the decoder about where
to concentrate while creating the next word. Further, we derive a context
vector, $h^{*}_{t}$, which is a weighted sum of the encoder hidden states and
$a^{t}$ calculated as:

$h^{*}_{t}=\sum\limits_{k}a^{t}_{k}.h_{k}$ (6)

Finally, we generate an output distribution probability ${P_{V}}$ over all
words in $V$ by concatenating $h^{*}_{t}$ with $s_{t}$ and then linearly
transforming it and pass it through a softmax activation function as:

$P_{V}=softmax(U[s_{t},h^{*}_{t}]+b)$ (7)

where $U$ and b are the learnable parameters.

####  4.3.3 Pointer Generator

The subsequent component of AuxPGN is a copy mechanism [70]. The idea behind
the pointer generator is to circumvent the limitations of pure abstraction
when dealing with factual information that necessitates copying from the input
to generate a correct summary. The basic encoder-decoder model [53] does not
capture factual information accurately while generating the summary.
Therefore, Vinyals et al. [62] introduce a pointer network in the machine
translation domain. We utilize the concept of pointer generators in our
AuxPGN, in order to provide the choice of selection to the model whether to
generate a word from a fixed vocabulary or to copy it from the input source
directly when needed.

We define a generation probability, $P_{gen}$, such that $P_{gen}\in[0,1]$.
For time step $t$, we calculate $P_{gen}$ from the context vector $h^{*}_{t}$,
the decoder state $s_{t}$ and the decoder input $x_{t}$ as:

$P_{gen}=\sigma(w^{T}_{h^{*}}h^{*}_{t}+w^{T}_{s}s_{t}+w^{T}_{x}x_{t}+b_{p})$
(8)

where, the vectors $w^{T}_{h^{*}}$, $w^{T}_{s}$, $w^{T}_{x}$ and scalar
$b_{p}$ are the learnable parameters and $\sigma$ is a sigmiod activation
function. Further, we define ‘extended vocabulary’ as a union of all the input
words with $|V|$ and then calculate the final probability distribution over
extended vocabulary as:

$P(w)=P_{gen}*P_{V}(w)+(1-P_{gen})*\sum\limits_{\forall i:w_{i}=w}a^{t}_{i}$
(9)

We observe from Equation 9 that if a word $w$ is an out-of-vocabulary (OOV)
word, then $P_{V}(w)$ is $0$. Similarly, if $w$ does not exist in the input,
$\sum\limits_{\forall i:w_{i}=w}a^{t}_{i}$ equals $0$. Therefore, the pointer
generator model efficiently solves the problem of factual errors in the output
summary by including the ability to switch between OOV words and words from
$|V|$.

####  4.3.4 Coverage Mechanism

Existing attention-based recurrent neural networks model [53] suffer from the
text repetition problem while generating the summary. To solve this, in
AuxPGN, we adopt a coverage mechanism proposed by Tu et al. [71] that utilizes
a coverage vector $c^{t}$, which is a sum of attention vectors at previous
time steps as:

$c^{t}=\sum\limits_{i=0}^{t-1}a^{i}$ (10)

At time step $t=1$, when no input has been covered, $c^{0}$ represents the
zero vector. Then, first, we utilize $c^{t}$ as an extra input to the
attention mechanism (discussed in 4.3.2) and then modify Equation 2 as:

$e^{t}={v^{T}}{tanh({W_{h}}{h_{k}}+{W_{s}}{s_{t}}+{W_{c}}{c_{k}^{t}}+b_{att})}$
(11)

where $W_{c}$ is a learnable parameter. Second, to calculate the final loss
$L_{t}$, we follow See et al. [14] to add negative likelihood loss with
$c^{t}$ (as an additional loss) by a hyperparameter $\lambda$ as:

$L_{t}=-logP(w_{t}|w_{<t})+\lambda\sum_{i=0}^{k}min(a_{i}^{t}.c_{i}^{t})$ (12)

By incorporating $c^{t}$, the attention module is encouraged to distribute
attention weights such that the input words with significant attention through
previous decoding steps are assigned lower weights. This encourages the model
to explore and focus on different aspects of the input sequence during the
generation process.

####  4.3.5 Decoding

We use a Beam search algorithm [72] to create the output summary. While
evaluating AuxPGN using test data, we do not provide AuxPGN with any key-
phrase information from the key-phrase model. As a result, the right side of
Equation 4 simplifies to the softmax ($e^{t}$) only. We believe that AuxPGN
parameters are optimized during training to maximize the effectiveness of the
provided key-phrase attention distribution. This implicit learning enables
AuxPGN to detect patterns in the weights of key-related terms, allowing it to
utilize the key-related information during summary creation more effectively.
Now, we give a set of input tweets and corresponding key-phrases with scores
into the AuxPGN encoder model, and then, the decoder model of AuxPGN generates
the more human-interpretable summary efficiently.

##  5 Experiments

In this Section, we initially discuss the different existing state-of-the-art
abstractive summarization approaches that we use as baselines, followed by the
experimental setting, comparison results, ablation experiments, comparison
with fine-tuned transformer models, and finally, human evaluation experiments.

###  5.1 Baselines

In this Subsection, we introduce a number of state-of-the-art abstractive
summarization approaches, which we use as baselines. We segregate these
approaches into disaster-specific and non-disaster-specific approaches. We
select a few prominent tweet summarization approaches from each type, which we
discuss next.

  1. 1.

Disaster-specific Approaches: Disaster-specific approaches are ones that are
specifically developed to summarize disaster-related tweets. From this
category, we use the following approaches as baselines to compare with our
proposed ATSumm.

    1. (a)

CowABS: Rudra et al. [10] propose an extractive-abstractive framework in which
they extract the most important tweets by maximizing the disaster information
content of the tweets during the extractive phase. They then form new
sentences by traversing the nodes of a bigram-based word graph created from
these important tweets. Finally, they incorporate the new sentences into the
summary to ensure maximum disaster information coverage and diversity.

    2. (b)

CLiQS_CM: Vitiugin et al. [8] propose an approach for disaster event
summarization, where in extractive phase, they utilize a K-mean clustering
algorithm to select most relevant tweets and then in abstractive phase, they
utilize a transformer-based language model (i.e., T5 [18] model) to create the
final summary.

    3. (c)

PTESumm: Rudrapal et al. [11] propose an abstractive disaster summarization
approach that initially identifies the most relevant and informative tweets
using a ranking-based approach and then utilizes a PGN model with coverage
technique for abstractive summary generation.

  2. 2.

Non-disaster-specific Approaches: Non-disaster-specific approaches are ones
that are not specifically developed to summarize disaster events but are
developed for other summarization domains. From this category, we use the
following approaches as baselines to compare with our proposed ATSumm.

    1. (a)

PGN: See et al. [14] propose a pointer generator model that solves the out-of-
vocabulary problem by allowing the summarizer to both copy words from the
source text and to generate new text.

    2. (b)

PSumm: Zogan et al. [73] propose an abstractive summarization approach that
uses a k-mean algorithm to identify the different clusters and then selects
cluster centre as a representative tweet from each cluster. They then
construct a concise summary using a pre-trained BART model fine-tuned on the
CNN/Daily Mail dataset.

    3. (c)

T5: Raffel et al. [18] propose a pre-trained Seq2Seq transformer transformer-
based architecture, T5, for abstractive summarization. For training, they use
the Colossal Clean Crawled Corpus.

    4. (d)

BART: Lewis et al. [19] propose a pre-trained Seq2Seq encoder-decoder
transformer-based architecture, BART, for abstractive summarization. BART’s
concept is to use the denoising objective during the pre-training stage.

    5. (e)

PEGASUS: Zhang et al. [15] utilizes a novel summarization-specific pre-
training objective known as Gap Sentence Generation (GSG), which involves
masking out important sentences from a document and then creating those
sentences from the remaining sentences. This training objective helps the
model learn how to identify and summarize the most important information in a
document.

    6. (f)

Longformer: Beltagy et al. [16] propose a modified transformer-based Seq2Seq
architecture that utilizes a local windowed attention mechanism to address the
long input sequence issue. The advantage of this mechanism is it only attends
to a small window of tokens at a time, significantly reducing the time it
takes to compute the attention weights.

    7. (g)

ProphetNet: Qi et al. [17] propose PropheNet to address the problem of
existing Seq2Seq models failing to capture long-term dependencies between
tokens while creating the summary. PropheNet utilizes a novel self-supervised
objective known as future n-gram prediction, which encourages the model to
learn long-range dependencies between tokens and generate more fluent and
grammatically correct summaries.

Table 4: Table shows F1-score (F1) of ROUGE-1, 2, and L, as R-1, R-2, and R-L,
respectively, for ATSumm and baselines on $D_{1}-D_{6}$ disaster datasets.
[Note: we mark the best-performing baseline in blue colour.]

Dataset | Approach | R-1 | R-2 | R-L | Dataset | Approach | R-1 | R-2 | R-L | Dataset | Approach | R-1 | R-2 | R-L
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
|  | F1 | F1 | F1 |  |  | F1 | F1 | F1 |  |  | F1 | F1 | F1
| $ATSumm$ | 0.58 | 0.30 | 0.41 |  | $ATSumm$ | 0.54 | 0.25 | 0.33 |  | $ATSumm$ | 0.42 | 0.23 | 0.29
| $ExAbsSumm$ | 0.38 | 0.16 | 0.34 |  | $ExAbsSumm$ | 0.35 | 0.13 | 0.24 |  | $ExAbsSumm$ | 0.31 | 0.06 | 0.24
| $PGN$ | 0.34 | 0.15 | 0.31 |  | $PGN$ | 0.26 | 0.08 | 0.12 |  | $PGN$ | 0.24 | 0.04 | 0.18
| $PTESumm$ | 0.42 | 0.17 | 0.32 |  | $PTESumm$ | 0.35 | 0.12 | 0.23 |  | $PTESumm$ | 0.28 | 0.07 | 0.19
| $PSumm$ | 0.52 | 0.20 | 0.31 |  | $PSumm$ | 0.46 | 0.22 | 0.29 |  | $PSumm$ | 0.37 | 0.19 | 0.26
| $CLiQS\\_CM$ | 0.53 | 0.23 | 0.32 |  | $CLiQS\\_CM$ | 0.46 | 0.19 | 0.28 |  | $CLiQS\\_CM$ | 0.35 | 0.18 | 0.25
${D_{1}}$ | $T5$ | 0.54 | 0.24 | 0.36 | ${D_{5}}$ | $T5$ | 0.48 | 0.23 | 0.31 | ${D_{9}}$ | $T5$ | 0.37 | 0.20 | 0.26
| $BART$ | 0.53 | 0.21 | 0.32 |  | $BART$ | 0.47 | 0.22 | 0.26 |  | $BART$ | 0.36 | 0.18 | 0.26
| $PEGASUS$ | 0.39 | 0.12 | 0.29 |  | $PEGASUS$ | 0.45 | 0.21 | 0.30 |  | $PEGASUS$ | 0.31 | 0.06 | 0.20
| $Longformer$ | 0.50 | 0.22 | 0.36 |  | $Longformer$ | 0.39 | 0.18 | 0.25 |  | $Longformer$ | 0.34 | 0.09 | 0.25
| $ProphetNet$ | 0.51 | 0.23 | 0.34 |  | $ProphetNet$ | 0.42 | 0.13 | 0.22 |  | $ProphetNet$ | 0.32 | 0.04 | 0.18
| $ATSumm$ | 0.44 | 0.20 | 0.32 |  | $ATSumm$ | 0.50 | 0.20 | 0.29 |  | $ATSumm$ | 0.46 | 0.26 | 0.36
| $ExAbsSumm$ | 0.17 | 0.06 | 0.16 |  | $ExAbsSumm$ | 0.33 | 0.07 | 0.19 |  | $ExAbsSumm$ | 0.40 | 0.09 | 0.23
| $PGN$ | 0.25 | 0.05 | 0.15 |  | $PGN$ | 0.28 | 0.09 | 0.21 |  | $PGN$ | 0.36 | 0.07 | 0.21
| $PTESumm$ | 0.32 | 0.09 | 0.18 |  | $PTESumm$ | 0.38 | 0.10 | 0.22 |  | $PTESumm$ | 0.40 | 0.15 | 0.31
| $PSumm$ | 0.34 | 0.11 | 0.22 |  | $PSumm$ | 0.43 | 0.11 | 0.27 |  | $PSumm$ | 0.41 | 0.15 | 0.32
| $CLiQS\\_CM$ | 0.32 | 0.11 | 0.25 |  | $CLiQS\\_CM$ | 0.41 | 0.11 | 0.25 |  | $CLiQS\\_CM$ | 0.42 | 0.18 | 0.30
${D_{2}}$ | $T5$ | 0.40 | 0.13 | 0.25 | ${D_{6}}$ | $T5$ | 0.44 | 0.12 | 0.27 | ${D_{10}}$ | $T5$ | 0.43 | 0.22 | 0.33
| $BART$ | 0.31 | 0.04 | 0.19 |  | $BART$ | 0.42 | 0.13 | 0.25 |  | $BART$ | 0.40 | 0.16 | 0.31
| $PEGASUS$ | 0.32 | 0.06 | 0.19 |  | $PEGASUS$ | 0.43 | 0.12 | 0.24 |  | $PEGASUS$ | 0.26 | 0.07 | 0.18
| $Longformer$ | 0.40 | 0.14 | 0.26 |  | $Longformer$ | 0.39 | 0.10 | 0.21 |  | $Longformer$ | 0.42 | 0.21 | 0.30
| $ProphetNet$ | 0.36 | 0.05 | 0.18 |  | $ProphetNet$ | 0.39 | 0.11 | 0.18 |  | $ProphetNet$ | 0.41 | 0.21 | 0.31
| $ATSumm$ | 0.48 | 0.24 | 0.30 |  | $ATSumm$ | 0.48 | 0.21 | 0.29 |  | $ATSumm$ | 0.55 | 0.31 | 0.33
| $ExAbsSumm$ | 0.24 | 0.06 | 0.22 |  | $ExAbsSumm$ | 0.35 | 0.07 | 0.23 |  | $ExAbsSumm$ | 0.44 | 0.12 | 0.26
| $PGN$ | 0.23 | 0.06 | 0.21 |  | $PGN$ | 0.31 | 0.09 | 0.24 |  | $PGN$ | 0.39 | 0.07 | 0.23
| $PTESumm$ | 0.35 | 0.08 | 0.22 |  | $PTESumm$ | 0.40 | 0.11 | 0.25 |  | $PTESumm$ | 0.42 | 0.11 | 0.24
| $PSumm$ | 0.41 | 0.10 | 0.26 |  | $PSumm$ | 0.46 | 0.15 | 0.27 |  | $PSumm$ | 0.50 | 0.14 | 0.26
| $CLiQS\\_CM$ | 0.41 | 0.15 | 0.26 |  | $CLiQS\\_CM$ | 0.43 | 0.12 | 0.25 |  | $CLiQS\\_CM$ | 0.51 | 0.20 | 0.26
${D_{3}}$ | $T5$ | 0.40 | 0.14 | 0.25 | ${D_{7}}$ | $T5$ | 0.40 | 0.09 | 0.23 | ${D_{11}}$ | $T5$ | 0.52 | 0.22 | 0.25
| $BART$ | 0.40 | 0.09 | 0.25 |  | $BART$ | 0.46 | 0.16 | 0.28 |  | $BART$ | 0.53 | 0.23 | 0.28
| $PEGASUS$ | 0.39 | 0.07 | 0.23 |  | $PEGASUS$ | 0.44 | 0.14 | 0.25 |  | $PEGASUS$ | 0.43 | 0.11 | 0.26
| $Longformer$ | 0.39 | 0.15 | 0.27 |  | $Longformer$ | 0.46 | 0.16 | 0.26 |  | $Longformer$ | 0.41 | 0.15 | 0.26
| $ProphetNet$ | 0.42 | 0.15 | 0.28 |  | $ProphetNet$ | 0.29 | 0.11 | 0.24 |  | $ProphetNet$ | 0.37 | 0.06 | 0.18
| $ATSumm$ | 0.52 | 0.28 | 0.34 |  | $ATSumm$ | 0.51 | 0.26 | 0.30 |  | $ATSumm$ | 0.49 | 0.24 | 0.34
| $ExAbsSumm$ | 0.28 | 0.10 | 0.23 |  | $ExAbsSumm$ | 0.40 | 0.10 | 0.25 |  | $ExAbsSumm$ | 0.34 | 0.10 | 0.23
| $PGN$ | 0.31 | 0.09 | 0.26 |  | $PGN$ | 0.33 | 0.06 | 0.22 |  | $PGN$ | 0.37 | 0.14 | 0.27
| $PTESumm$ | 0.38 | 0.15 | 0.21 |  | $PTESumm$ | 0.38 | 0.11 | 0.21 |  | $PTESumm$ | 0.39 | 0.10 | 0.22
| $PSumm$ | 0.44 | 0.25 | 0.32 |  | $PSumm$ | 0.45 | 0.17 | 0.25 |  | $PSumm$ | 0.44 | 0.17 | 0.29
| $CLiQS\\_CM$ | 0.41 | 0.24 | 0.30 |  | $CLiQS\\_CM$ | 0.41 | 0.15 | 0.24 |  | $CLiQS\\_CM$ | 0.39 | 0.15 | 0.27
${D_{4}}$ | $T5$ | 0.43 | 0.24 | 0.31 | ${D_{8}}$ | $T5$ | 0.39 | 0.12 | 0.23 | ${D_{12}}$ | $T5$ | 0.40 | 0.17 | 0.29
| $BART$ | 0.44 | 0.25 | 0.31 |  | $BART$ | 0.47 | 0.17 | 0.28 |  | $BART$ | 0.40 | 0.16 | 0.27
| $PEGASUS$ | 0.45 | 0.26 | 0.32 |  | $PEGASUS$ | 0.37 | 0.08 | 0.23 |  | $PEGASUS$ | 0.36 | 0.08 | 0.21
| $Longformer$ | 0.39 | 0.20 | 0.24 |  | $Longformer$ | 0.45 | 0.16 | 0.25 |  | $Longformer$ | 0.44 | 0.16 | 0.26
| $ProphetNet$ | 0.41 | 0.18 | 0.26 |  | $ProphetNet$ | 0.39 | 0.10 | 0.24 |  | $ProphetNet$ | 0.40 | 0.09 | 0.22
| $ATSumm$ | 0.52 | 0.25 | 0.35 |  |  |  |  |  |
| $ExAbsSumm$ | 0.43 | 0.15 | 0.30 |  |  |  |  |  |
| $PGN$ | 0.37 | 0.07 | 0.20 |  |  |  |  |  |
| $PTESumm$ | 0.40 | 0.09 | 0.23 |  |  |  |  |  |
| $PSumm$ | 0.44 | 0.17 | 0.30 |  |  |  |  |  |
| $CLiQS\\_CM$ | 0.45 | 0.20 | 0.29 |  |  |  |  |  |
${D_{13}}$ | $T5$ | 0.47 | 0.22 | 0.31 |  |  |  |  |  |
| $BART$ | 0.42 | 0.14 | 0.25 |  |  |  |  |  |
| $PEGASUS$ | 0.43 | 0.14 | 0.29 |  |  |  |  |  |
| $Longformer$ | 0.41 | 0.18 | 0.25 |  |  |  |  |  |
| $ProphetNet$ | 0.39 | 0.08 | 0.22 |  |  |  |  |  |

###  5.2 Experimental Setting

For our hyperparameters, we use the hidden state dimensions of the LSTMs to
$256$, the word-embedding dimensions at $128$, and the min-batch size to $16$.
For AuxPGN model, we have set the input tweet length to $400$ words while the
output summary length to $200$ words. For training, we use Adagard [74] with a
learning rate of $0.15$ and an initial accumulator value of $0.1$.
Furthermore, when decoding the summaries for the test dataset, we use the beam
size to $5$ and the minimum generating summary length to $35$ tokens. We also
restricted the vocabulary of our model to $50,000$ tokens. We run these
experiments on a Google Collab GPU environment using Keras and TensorFlow with
a Nvidia K80/T4 GPU with $25$ GB RAM. We set these hyperparameters through the
various experiments discussed in Subsection 5.5.

We follow the same procedure as Poddar et al. [75] to construct an abstractive
summary using pre-trained transformer models. We initially split the large-
sized input tweets into equal-sized chunks/samples so that the length of each
sample did not exceed the maximum permissible length. Then, we obtain the
summary for each sample and combine these sampled summaries to form the
overall summary. Further, we create a final summary of $200$ words in length
by selecting top-ranked sentences (from the combined summary) based on their
TF-IDF scores.

###  5.3 Comparison with Existing Research Works

To evaluate the summary generated by ATSumm and different baselines with the
gold standard summaries, we use the ROUGE-N score [76]. ROUGE-N calculates a
score based on the number of words that overlap between the model-generated
summary and the gold standard summary. We use the F1-score for three variants
of the ROUGE-N scores, where N takes 1, 2, and L values, respectively. The
higher ROUGE-N score indicates the higher the quality of the resulting
summary. Our observations, as shown in Table 4 indicate that the ATSumm
outperforms over all the baselines by $3.63-61.36$%, $7.14-80$%, and
$3.45-63.63$% in terms of ROUGE-1, 2, and L F1-score, respectively and it
above over all the baselines. We found that the highest and lowest
improvements are inconsistent with the baselines. For example, the improvement
is the highest over PGN for $D_{2}$, $D_{3}$, $D_{5}$, $D_{6}$, $D_{8}$,
$D_{9}$, $D_{11}$, and $D_{13}$, CowABS for $D_{4}$, and PEGASUS for $D_{10}$,
and $D_{12}$ and lowest over T5 for $D_{1}$, $D_{5}$, $D_{6}$, $D_{9}$,
$D_{10}$, and $D_{13}$ BART for $D_{7}$, $D_{8}$, and $D_{11}$, Longformer for
$D_{2}$, ProphetNet for $D_{3}$, PSumm for $D_{12}$, and PEGASUS for $D_{4}$.
The improvement of ATSumm is the best for $D_{1}$ with $0.58-0.30$ and worst
for $D_{9}$ with $0.42-0.23$ in terms of ROUGE-N F1-score. We also observe
that the improvement compared to the best-performing baseline (marked in blue
colour in the tables) in each scenario remains in the range of $0.02-0.07$ in
terms of ROUGE-1 F1-score. However, the best-performing baseline is not
constant across the datasets. Therefore, there is always a significant margin
of good results for ATSumm with the best-performing baseline.

###  5.4 Ablation Experiments

To understand the effectiveness of each phase of ATSumm, we compare the
performance of ATSumm with its different variants and study the role of
auxiliary information in the proposed AuxPGN model.

####  5.4.1 Phase-I

In this experiment, we compare Phase-I of ATSumm with its different variants,
which are as follows:

  * •

In ATSumm-1A161616The first number in the name of each of the variants of
ATSumm, i.e., ATSumm-1A represents the phase, and the second number denotes
the sequence of the ablation experiment in that phase., we extract the top-
ranked tweets based on an ontology-based tweet summarization approach proposed
by Garg et al. [4].

  * •

In ATSumm-1B, we extract the top-ranked tweets based on an entropy-based tweet
summarization approach proposed by Garg et al. [5].

  * •

In ATSumm-1C, we extract the top-ranked tweets based on the importance of
their content words (i.e., nouns, verbs, numerals, and adjectives) using TF-
IDF, proposed by Rudra et al. [61].

Table 5: Table shows F1-score (F1) of ROUGE-1, 2, and L, as R-1, R-2, and R-L,
respectively, for Phase-I and Phase-II variants of ATSumm, i.e., ATSumm-1A,
ATSumm-1B, ATSumm-1C, ATSumm-2A, ATSumm-2B, ATSumm-2C, ATSumm-2D, ATSumm-2E,
ATSumm-2F, and ATSumm-2G with ATSumm on six datasets: $D_{1}$, $D_{3}$,
$D_{7}$, $D_{8}$, $D_{10}$, and $D_{12}$.

Dataset | Approach | R-1 | R-2 | R-L | Dataset | Approach | R-1 | R-2 | R-L | Dataset | Approach | R-1 | R-2 | R-L
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
|  | F1 | F1 | F1 |  |  | F1 | F1 | F1 |  |  | F1 | F1 | F1
| $ATSumm$ | 0.58 | 0.30 | 0.41 |  | $ATSumm$ | 0.48 | 0.21 | 0.29 |  | $ATSumm$ | 0.46 | 0.26 | 0.36
| $ATSumm-1A$ | 0.50 | 0.27 | 0.33 |  | $ATSumm-1A$ | 0.43 | 0.15 | 0.27 |  | $ATSumm-1A$ | 0.43 | 0.24 | 0.35
| $ATSumm-1B$ | 0.42 | 0.28 | 0.35 |  | $ATSumm-1B$ | 0.35 | 0.07 | 0.21 |  | $ATSumm-1B$ | 0.42 | 0.21 | 0.31
| $ATSumm-1C$ | 0.38 | 0.11 | 0.26 |  | $ATSumm-1C$ | 0.32 | 0.11 | 0.22 |  | $ATSumm-1C$ | 0.37 | 0.14 | 0.29
| $ATSumm-2A$ | 0.33 | 0.18 | 0.27 |  | $ATSumm-2A$ | 0.43 | 0.13 | 0.22 |  | $ATSumm-2A$ | 0.43 | 0.21 | 0.32
${D_{1}}$ | $ATSumm-2B$ | 0.47 | 0.18 | 0.30 | ${D_{7}}$ | $ATSumm-2B$ | 0.44 | 0.16 | 0.23 | ${D_{10}}$ | $ATSumm-2B$ | 0.42 | 0.23 | 0.31
| $ATSumm-2C$ | 0.34 | 0.07 | 0.23 |  | $ATSumm-2C$ | 0.26 | 0.05 | 0.19 |  | $ATSumm-2C$ | 0.32 | 0.12 | 0.22
| $ATSumm-2D$ | 0.44 | 0.18 | 0.35 |  | $ATSumm-2D$ | 0.45 | 0.15 | 0.26 |  | $ATSumm-2D$ | 0.36 | 0.19 | 0.27
| $ATSumm-2E$ | 0.28 | 0.09 | 0.23 |  | $ATSumm-2E$ | 0.36 | 0.09 | 0.25 |  | $ATSumm-2E$ | 0.40 | 0.18 | 0.25
| $ATSumm-2F$ | 0.31 | 0.13 | 0.27 |  | $ATSumm-2F$ | 0.34 | 0.13 | 0.24 |  | $ATSumm-2F$ | 0.39 | 0.20 | 0.21
| $ATSumm-2G$ | 0.56 | 0.28 | 0.39 |  | $ATSumm-2G$ | 0.46 | 0.19 | 0.27 |  | $ATSumm-2G$ | 0.44 | 0.23 | 0.34
| $ATSumm$ | 0.48 | 0.24 | 0.30 |  | $ATSumm$ | 0.51 | 0.26 | 0.30 |  | $ATSumm$ | 0.49 | 0.24 | 0.34
| $ATSumm-1A$ | 0.42 | 0.14 | 0.27 |  | $ATSumm-1A$ | 0.49 | 0.22 | 0.24 |  | $ATSumm-1A$ | 0.44 | 0.20 | 0.28
| $ATSumm-1B$ | 0.37 | 0.19 | 0.29 |  | $ATSumm-1B$ | 0.42 | 0.16 | 0.26 |  | $ATSumm-1B$ | 0.38 | 0.12 | 0.28
| $ATSumm-1C$ | 0.35 | 0.07 | 0.23 |  | $ATSumm-1C$ | 0.40 | 0.16 | 0.27 |  | $ATSumm-1C$ | 0.37 | 0.14 | 0.28
| $ATSumm-2A$ | 0.31 | 0.17 | 0.27 |  | $ATSumm-2A$ | 0.41 | 0.17 | 0.24 |  | $ATSumm-2A$ | 0.41 | 0.13 | 0.25
${D_{3}}$ | $ATSumm-2B$ | 0.40 | 0.15 | 0.26 | ${D_{8}}$ | $ATSumm-2B$ | 0.47 | 0.18 | 0.28 | ${D_{12}}$ | $ATSumm-2B$ | 0.43 | 0.15 | 0.27
| $ATSumm-2C$ | 0.30 | 0.04 | 0.22 |  | $ATSumm-2C$ | 0.19 | 0.05 | 0.18 |  | $ATSumm-2C$ | 0.21 | 0.05 | 0.16
| $ATSumm-2D$ | 0.35 | 0.16 | 0.27 |  | $ATSumm-2D$ | 0.45 | 0.14 | 0.25 |  | $ATSumm-2D$ | 0.44 | 0.17 | 0.29
| $ATSumm-2E$ | 0.26 | 0.03 | 0.18 |  | $ATSumm-2E$ | 0.38 | 0.13 | 0.27 |  | $ATSumm-2E$ | 0.25 | 0.10 | 0.23
| $ATSumm-2F$ | 0.28 | 0.14 | 0.17 |  | $ATSumm-2F$ | 0.36 | 0.13 | 0.25 |  | $ATSumm-2F$ | 0.34 | 0.15 | 0.22
| $ATSumm-2G$ | 0.45 | 0.21 | 0.27 |  | $ATSumm-2G$ | 0.48 | 0.24 | 0.27 |  | $ATSumm-2G$ | 0.47 | 0.22 | 0.31

####  5.4.2 Phase-II

To understand the role and effectiveness of the AuxPGN model in Phase-II of
ATSumm, we compare the performance of ATSumm with its different variants,
which are as follows:

  * •

In ATSumm-2A, we use the T5-small model proposed by Raffel et al. [18] to
create an abstractive summary of the input tweets.

  * •

In ATSumm-2B, we use the BART model proposed by Lewis et al. [19] to create an
abstractive summary of the input tweets.

  * •

In ATSumm-2C, we use the PEGASUS model proposed by Zhang et al. [15] to create
an abstractive summary of the input tweets.

  * •

In ATSumm-2D, we use the Longformer model proposed by Beltagy et al. [16] to
create an abstractive summary of the input tweets.

  * •

In ATSumm-2E, we use the ProphetNet model proposed by Qi et al. [17] to create
an abstractive summary of the input tweets.

  * •

In ATSumm-2F, we use the PGN model proposed by See et al. [14] to create an
abstractive summary of the input tweets.

  * •

In ATSumm-2G, we use the proposed AuxPGN model without considering the key-
phrase scores to create an abstractive summary of the given input tweets.

####  5.4.3 Results and Discussions

Now, we compare and evaluate the performance of the discussed different
variants of ATSumm on six different disaster event datasets, such as $D_{1}$,
$D_{3}$, $D_{7}$, $D_{8}$, $D_{10}$, and $D_{12}$ on ROUGE-1, 2, and L
F1-score. Our observations in Table 5 indicate that the ATSumm outperforms all
its variants by $3.44-62.74\%$, $6.67-87.50\%$, and $3.33-78.26\%$, ROUGE-1,
2, and L F1-score, respectively, across disasters.

  1. 1.

Phase-I:  Our observations indicate that ATSumm outperforms ATSumm-1C, which
extracts top-ranked tweets based on the importance of their content words
using TF-IDF by $3.92-34.48\%$, $6.67-70.83$%, $3.33-36.58$% in ROUGE-1, 2,
and L F1-score. These results highlight the effectiveness of incorporating
key-phrases with the BERT model to extract the tweets in the extractive phase.

  2. 2.

Phase-II:  Our observations indicate that the inclusion of AUxPGN in ATSumm
outperforms of $3.44-62.74$%, $6.67-87.50$%, and $4.87-78.26$% in ROUGE-1, 2,
and L F1-score, respectively, with its different variants, such as ATSumm-2A,
ATSumm-2B, ATSumm-2C, ATSumm-2D, ATSumm-2E, ATSumm-2F, and ATSumm-2G. This
justifies that incorporating the auxiliary information (key-phrases and their
importance scores) in the PGN helps to create a more human-interpretable
summary even if we provide less training data to train it. We further observe
that the information gain of ATSumm is highest over ATSumm-2C as it employs
the PEGASUS model to generate the summary, while it is lowest over ATSumm-2G,
which combines only key-phrases in AuxPGN for summary creation. We also
observe that the inclusion of novel AuxPGN in Phase-II provides maximum
performance of ATSumm.

(a)

(b)

(c)

Figure 3: Figure shows the comparison results of ROUGE-1 F1-score of ATSumm
with the different hyperparameters, i.e., min batch size, learning rate, and
number of training iterations on five datasets, such as $D_{1}$, $D_{6}$,
$D_{9}$, $D_{10}$, and $D_{11}$ in Figure 3(a), Figure 3(b) and 3(c),
respectively.

###  5.5 Hyperparameter Analysis

In this Subsection, we discuss how we set the values of hyperparameters for
ATSumm, i.e., min batch size, learning rate, and number of training
iterations. For each parameter or hyperparameter, we vary its values and
calculate the ROUGE-1 F1-score for each value, and then we select that value
for a hyperparameter that gives the best ROUGE-1 F1-score. We randomly select
five different datasets for our experiment on different hyperparameters:
$D_{1}$, $D_{6}$, $D_{9}$, $D_{10}$, and $D_{11}$ and the experimental results
shown in Figure 3. Our observations from Figure 3(a) indicate that by varying
the min batch size from $8$ to $128$, the min batch size of $16$ provides the
maximum ROUGE-1 F1-score irrespective of the datasets. We also observe that
there is no significant improvement when we increase the min batch size from
$32$ to $128$. Therefore, we select $16$ as a min batch size in ATSumm.
Similarly, we observe that $1,20,000$ number of iterations provides the
maximum ROUGE-1 F1-score for ATSumm across datasets, as shown in Figure 3(b).
Similarly, to decide the learning rate of Adagrad [74] optimizer, we vary the
learning rate from $0.05$ to $0.30$ and measure ROUGE-1 F1-score for ATSumm
across five datasets. Our observations from Figure 3(c) indicate that the
learning rate of $0.15$ provides the maximum ROUGE-1 F1-score and no
significant improvement when we vary the learning rate from $0.20$ to $0.30$.
Therefore, in ATSumm, we select $0.15$ as a min learning rate.

###  5.6 Scalability Analysis

In this Subsection, we investigate the scalability of AuxPGN in ATSumm by
comparing its performance when the training dataset size increases gradually
from $10$% to $100$%. We select five different datasets: $D_{1}$, $D_{6}$,
$D_{9}$, $D_{10}$, and $D_{11}$ for our experiment on different training
dataset sizes and follow the same procedure for implementation as discussed in
Section 4 and then calculate the ROUGE-1 F1-score. Our results, as shown in
Figure 4, indicate that AuxPGN is scalable with increasing training dataset
size. We also observe that the ATSumm ensure significantly higher performance
in terms of ROUGE-1 F1-score for $100$% training dataset size than other
training sizes. Subsequently, ATSumm ensures the lowest ROUGE-1 F1-score with
$10$% training dataset size. Additionally, we observe that after the $90$%
training dataset, the F1-score is almost constant, and there is not much
further improvement. Hence, we train the AuxPGN model in ATSumm on $100$%
training samples, which are $7500$ samples of the ARIES dataset.

Figure 4: Figure shows the comparison results of ROUGE-1 F1-score of ATSumm
with the different sizes of the ARIES training dataset in percentage.

###  5.7 Weight Analysis

In this Subsection, we experimentally choose the weight values of $w_{1}$ and
$w_{2}$ used in Equation 4. For our experiment, we randomly select two
disaster datasets: $D_{2}$ and $D_{6}$ and vary the range of $w_{1}$ from $0$
to $1$ and $w_{2}$ from $1$ to $0$ such that $w_{1}+w_{2}=1$. For each value
of pair of $w_{1}$ and $w_{2}$, we train the AuxPGN model and then generate
the summary using ATSumm. Further, we evaluate the performance of these
generated summaries using ATSumm with the gold standard summaries using
ROUGE-1 F1-score. We show the ROUGE-1 F1-score of each value of pair of
$w_{1}$ and $w_{2}$ for $D_{2}$ and $D_{6}$ in Figure 5. Our observations
indicate that the ROUGE-1 F1-score is highest when $w_{1}=w_{2}=0.5$ compared
to other combination values of $w_{1}$ and $w_{2}$. Therefore, based on our
observations, we select $0.5$ as the weights of $w_{1}$ and $w_{2}$ in ATSumm.

Table 6: Table shows the comparison results of five existing state-of-the-art
transformer models, i.e., T5, BART, PEGASUS, Longformer, and ProphetNet which
are fine-tuned on our proposed ARIES dataset with ATSumm model for F1-score
(F1) of ROUGE-1, 2, and L, as R-1, R-2, and R-L, respectively, on six
datasets: $D_{1}$, $D_{3}$, $D_{7}$, $D_{8}$, $D_{10}$, and $D_{12}$.

Dataset | Approach | R-1 | R-2 | R-L | Dataset | Approach | R-1 | R-2 | R-L
---|---|---|---|---|---|---|---|---|---
|  | F1 | F1 | F1 |  |  | F1 | F1 | F1
| $ATSumm$ | 0.58 | 0.30 | 0.41 |  | $ATSumm$ | 0.51 | 0.26 | 0.30
| $T5-Fine\\_tuning$ | 0.55 | 0.26 | 0.38 |  | $T5-Fine\\_tuning$ | 0.47 | 0.21 | 0.26
${D_{1}}$ | $BART-Fine\\_tuning$ | 0.55 | 0.26 | 0.37 | ${D_{8}}$ | $BART-Fine\\_tuning$ | 0.49 | 0.22 | 0.28
| $PEGASUS-Fine\\_tuning$ | 0.48 | 0.24 | 0.33 |  | $PEGASUS-Fine\\_tuning$ | 0.44 | 0.19 | 0.27
| $Longformer-Fine\\_tuning$ | 0.53 | 0.25 | 0.37 |  | $Longformer-Fine\\_tuning$ | 0.47 | 0.21 | 0.27
| $ProphetNet-Fine\\_tuning$ | 0.54 | 0.25 | 0.37 |  | $ProphetNet-Fine\\_tuning$ | 0.46 | 0.21 | 0.26
| $ATSumm$ | 0.48 | 0.24 | 0.30 |  | $ATSumm$ | 0.46 | 0.26 | 0.36
| $T5-Fine\\_tuning$ | 0.45 | 0.17 | 0.27 |  | $T5-Fine\\_tuning$ | 0.44 | 0.23 | 0.32
${D_{3}}$ | $BART-Fine\\_tuning$ | 0.43 | 0.16 | 0.25 | ${D_{10}}$ | $BART-Fine\\_tuning$ | 0.42 | 0.21 | 0.33
| $PEGASUS-Fine\\_tuning$ | 0.44 | 0.17 | 0.26 |  | $PEGASUS-Fine\\_tuning$ | 0.39 | 0.18 | 0.29
| $Longformer-Fine\\_tuning$ | 0.45 | 0.19 | 0.28 |  | $Longformer-Fine\\_tuning$ | 0.44 | 0.23 | 0.32
| $ProphetNet-Fine\\_tuning$ | 0.45 | 0.20 | 0.28 |  | $ProphetNet-Fine\\_tuning$ | 0.42 | 0.23 | 0.33
| $ATSumm$ | 0.48 | 0.21 | 0.29 |  | $ATSumm$ | 0.49 | 0.24 | 0.34
| $T5-Fine\\_tuning$ | 0.45 | 0.17 | 0.26 |  | $T5-Fine\\_tuning$ | 0.43 | 0.20 | 0.30
${D_{7}}$ | $BART-Fine\\_tuning$ | 0.46 | 0.19 | 0.27 | ${D_{12}}$ | $BART-Fine\\_tuning$ | 0.45 | 0.19 | 0.31
| $PEGASUS-Fine\\_tuning$ | 0.45 | 0.17 | 0.27 |  | $PEGASUS-Fine\\_tuning$ | 0.43 | 0.18 | 0.32
| $Longformer-Fine\\_tuning$ | 0.46 | 0.19 | 0.27 |  | $Longformer-Fine\\_tuning$ | 0.46 | 0.20 | 0.31
| $ProphetNet-Fine\\_tuning$ | 0.44 | 0.18 | 0.26 |  | $ProphetNet-Fine\\_tuning$ | 0.44 | 0.19 | 0.29

Figure 5: Figure shows the ROUGE-1 F1-score of each value of pair of $w_{1}$
and $w_{2}$ for $D_{2}$ and $D_{6}$ datasets.

###  5.8 Comparison with Fine-tuned Transformer models

In this Subsection, we compare and validate the effectiveness of ATSumm with
the existing state-of-the-art transformer models fined-tuned on our proposed
ARIES dataset. For this, we consider $5$ different transformer models, i.e.,
T5 [18], BART [19], PEGASUS [15], Longformer [16], and ProphNet [17]. We then
fine-tune these transformer models on the ARIES dataset using the following
parameters: batch size is $16$, maximum output summary length is $200$ tokens,
and others are default parameters. We decide these parameter values using an
experiment on randomly selected two datasets by varying the batch size value
from $4$ to $32$ for fine-tuning. We found the batch size $16$ gives high
efficiency compared to other batch sizes and then chose this value for this
experiment. For our experiment, we randomly select six different disaster
datasets: $D_{1}$, $D_{3}$, $D_{7}$, $D_{8}$, $D_{10}$, and $D_{12}$. Our
results, as shown in Table 6, indicate that the ATSumm outperforms with the
fine-tuned transformer models by $3.92-17.24$%, $9.52-33.33$%, and
$5.88-19.51$% in terms of ROUGE-1, ROUGE-2, and ROUGE-L F1-score,
respectively. Furthermore, we observe that the performance increase of ATSumm
is highest when compared to the fine-tuned PEGASUS model and lowest when
compared to the fine-tuned Longformer model. Therefore, based on the above
experiment, we conclude that ATSumm ensures high performance with the existing
transformer models even after being fined-tuned on disaster events.

###  5.9 Human Evaluation

As the abstractive summary is more closer to the human written summaries, in
this Subsection, we perform a human evaluation to evaluate the quality of the
summary generated by ATSumm. Following [77, 78, 79], we compare the summary
generated by ATSumm with the baseline summaries based on five different
quality metrics:

  * •

Fluency- The summary must follow the grammatical rules without any formatting
problems in order to read well to the readers.

  * •

Readability- The summary should be easy to read and understandable to the
readers.

  * •

Conciseness- The summary should contain no unnecessary words.

  * •

Relevance- The summary should only contain relevant information from the given
input tweets.

  * •

Non-redundancy- The summary must be unique, with no two tweets containing the
same information in the summary.

Table 7: Table shows the Aggregated (average) Fluency Score (AFS), Readability
Score (AReS), Conciseness Score (ACS), Relevance Score (ARS), and Non-
redundancy Score (ANS) of the summary generated by ATSumm and different
baselines on four randomly selected datasets: $D_{1}$, $D_{4}$, $D_{9}$, and
$D_{12}$.

Dataset | Approach | AFS | AReS | ACS | ARS | ANS | Dataset | Approach | AFS | AReS | ACS | ARS | ANS
---|---|---|---|---|---|---|---|---|---|---|---|---|---
| $ATSumm$ | 4.25 | 4.59 | 4.62 | 4.42 | 4.33 |  | $ATSumm$ | 4.61 | 4.75 | 4.50 | 4.75 | 4.69
| $CowABS$ | 3.08 | 3.41 | 3.52 | 3.33 | 3.58 |  | $CowABS$ | 3.41 | 3.50 | 3.25 | 3.33 | 3.50
| $PGN$ | 2.75 | 2.82 | 2.91 | 2.66 | 2.91 |  | $PGN$ | 3.16 | 2.98 | 2.80 | 3.08 | 3.33
| $PTESumm$ | 2.91 | 3.08 | 3.12 | 2.96 | 3.00 |  | $PTESumm$ | 3.69 | 3.56 | 3.72 | 3.50 | 3.75
| $PSumm$ | 3.33 | 3.75 | 3.85 | 3.91 | 3.85 |  | $PSumm$ | 3.92 | 3.96 | 3.83 | 3.91 | 3.50
| $CLiQS\\_CM$ | 3.98 | 4.05 | 3.83 | 4.00 | 3.83 |  | $CLiQS\\_CM$ | 4.05 | 4.16 | 4.00 | 4.25 | 4.25
| $T5$ | 4.05 | 4.16 | 4.00 | 4.25 | 4.16 |  | $T5$ | 4.25 | 4.50 | 4.25 | 4.25 | 4.50
${D_{1}}$ | $BART$ | 3.85 | 3.69 | 3.91 | 3.83 | 3.75 | ${D_{9}}$ | $BART$ | 4.14 | 4.25 | 4.25 | 4.16 | 4.32
| $PEGASUS$ | 3.91 | 3.83 | 3.75 | 3.66 | 3.83 |  | $PEGASUS$ | 4.14 | 4.00 | 4.05 | 4.00 | 4.16
| $Longformer$ | 4.00 | 4.25 | 4.05 | 4.16 | 3.75 |  | $Longformer$ | 3.91 | 4.00 | 3.83 | 3.83 | 4.05
| $ProphetNet$ | 3.91 | 4.00 | 3.83 | 4.16 | 3.91 |  | $ProphetNet$ | 3.83 | 3.98 | 3.91 | 3.91 | 4.00
| $ATSumm$ | 4.41 | 4.50 | 4.75 | 4.65 | 4.50 |  | $ATSumm$ | 4.75 | 4.69 | 4.75 | 4.50 | 4.61
| $CowABS$ | 2.91 | 2.66 | 2.82 | 2.82 | 2.75 |  | $CowABS$ | 3.58 | 3.50 | 3.33 | 3.41 | 3.66
| $PGN$ | 2.83 | 2.66 | 2.82 | 2.75 | 2.91 |  | $PGN$ | 3.33 | 3.12 | 3.41 | 3.41 | 3.58
| $PTESumm$ | 3.03 | 2.96 | 2.91 | 3.00 | 3.12 |  | $PTESumm$ | 3.56 | 3.75 | 3.50 | 3.96 | 3.90
| $PSumm$ | 3.69 | 3.75 | 3.56 | 3.42 | 3.12 |  | $PSumm$ | 4.25 | 4.25 | 4.41 | 4.25 | 4.16
| $CLiQS\\_CM$ | 3.91 | 4.05 | 4.14 | 4.05 | 4.25 |  | $CLiQS\\_CM$ | 4.00 | 4.14 | 4.05 | 3.98 | 4.00
| $T5$ | 4.05 | 4.16 | 4.25 | 4.00 | 4.14 |  | $T5$ | 3.91 | 4.00 | 4.16 | 4.00 | 4.05
${D_{4}}$ | $BART$ | 3.91 | 3.99 | 4.00 | 3.83 | 3.75 | ${D_{12}}$ | $BART$ | 3.96 | 4.00 | 4.12 | 3.92 | 3.96
| $PEGASUS$ | 4.25 | 4.25 | 4.32 | 4.25 | 4.36 |  | $PEGASUS$ | 4.00 | 3.98 | 4.05 | 3.91 | 3.91
| $Longformer$ | 4.05 | 4.00 | 3.99 | 4.16 | 3.83 |  | $Longformer$ | 3.98 | 4.00 | 3.91 | 3.83 | 3.98
| $ProphetNet$ | 4.14 | 3.83 | 4.00 | 4.05 | 3.91 |  | $ProphetNet$ | 4.05 | 3.98 | 4.00 | 4.14 | 4.05

We employ three meta-annotators171717We select these meta-annotators through
the Quality Assessment Evaluation proposed by [46]. for evaluating the
summaries, taking care not to select the same annotators who generated the
summaries. The meta-annotators are post-graduate students between the age
group of $25-30$ who speak English fluently. We follow existing research works
[75, 46], where each annotator individually reads the summaries generated by
ATSumm and the different baselines and then assigns a score to each of the
summaries in the range of $1$ (lowest score) - $10$ (highest score) based on
metrics defined above. We get all of the above-mentioned scores from all three
meta-annotators for each summary. To merge scores from different meta-
annotators, we aggregate all of them and average them. For example, if the
Fluency Score of a summary from three meta-annotators is x, y, and z, then the
Aggregate Fluency Score is (x+y+z)/3. Similarly, we calculated the Aggregate
Readability Score, Aggregate Conciseness Score, Aggregate Relevance Score and
Aggregate Non-redundancy Score. We show the aggregated score over all three
meta-annotators for four randomly selected datasets: $D_{1}$, $D_{4}$,
$D_{9}$, and $D_{12}$ in Table 7. Our observations indicate that the quality
of the ATSumm summary is better than existing summarization approaches by
$3.11-40.88$%. We also observe that the aggregated score of ATSumm is above
$4.25$ for all five metrics across the datasets. The above results show that
the summaries generated by ATSumm are of higher quality than the other
baseline approaches.

###  5.10 Human A/B Testing

In this Subsection, we design a human A/B testing [78, 80] to ensure the
effectiveness of our proposed model against the existing state-of-the-art
models. In this experiment, we consider two models, A and B, where A could be
our model (ATSumm), and B could be the considered baselines (discussed in
Subsection 5.1) or the other way round. In any case, the annotators will not
know which model is A or B. We employ three annotators who are students in the
age group of $20-30$ and take care not to select the same annotators who
generated the summaries. Each annotator goes through all the summaries
generated by all these models of a dataset and picks the best summary from
them. An annotator may select a Tie if he/she finds the summary of both models
containing the same information. In the case of Tie, we bring in a $4^{th}$
annotator and perform the same experiment. We repeat this procedure for four
datasets: $D_{1}$, $D_{4}$, $D_{9}$, and $D_{12}$, and each dataset is then
passed to various models for recording the response of each model. Once we
have the ratings for each model’s comparison with ours, we maintain records of
how often our model wins, loses, or ties with the other model out of the four
datasets. Our results, as shown in Table 8, indicate that the summary
generated by ATSumm has a higher win percentage than the loss or tie
percentage, showing how well-appreciated our model is. Therefore, based on the
above experiment, we can say that the generated summary quality by ATSumm is
high and is highly preferred over the other various models.

Table 8: Table shows the results of human A/B testing for assessing the
summary generated by ATSumm and different baseline models. We show the average
win, loss, and tie percentage of three annotators scores on four different
disaster datasets.

Model | ATSumm Vs Model
---|---
| Win% | Loss% | Tie%
$CowABS$ | 100 | - | -
$PGN$ | 100 | - | -
$PTESumm$ | 91.67 | - | 8.33
$PSumm$ | 75.00 | 8.33 | 16.67
$CLiQS\\_CM$ | 83.34 | 8.33 | 8.33
$T5$ | 66.66 | 16.67 | 16.67
$BART$ | 75.00 | 16.67 | 8.33
$PEGASUS$ | 75.00 | - | 25.00
$Longformer$ | 91.67 | 8.33 | -
$ProphetNet$ | 83.34 | - | 16.67

##  6 Conclusions and Future works

In this paper, we propose a deep learning-based framework ATSumm, which
generates a human-readable summary of a disaster with sparse training data.
The main challenge of any deep learning-based approach is that it requires
huge training data. The sparsity of the training data is handled using domain
knowledge in the form of auxiliary information. The first phase of ATSumm
(extractive phase) ranks the tweets using domain knowledge from an existing
ontology; while we provide key-pharse and their importance score as auxiliary
information in the second phase of ATSumm (abstractive phase). To the best of
our knowledge, our proposed ATSumm is the first model on abstractive disaster
tweet summarization, which utilizes auxiliary information in a deep-learning
framework to handle the problem of sparse training data. Both quantitative and
qualitative experiments confirm the superiority of ATSumm over existing state-
of-the-art models. Our quantitative experimental analysis shows that ATSumm
can ensure $4-80\%$ higher ROUGE-N F1-score than existing state-of-the-art
baseline approaches. Additionally, our qualitative experiments analysis shows
that ATSumm can ensure $3.11-40.88\%$ better performance than existing
summarization approaches for all five qualitative metrics across the datasets.
Apart from ATSumm, in this paper, we generate and publicly provide a novel
ARIES dataset consisting of $7500$ training samples and $13$ different
disaster events with corresponding ground-truth summaries. As future work, to
handle the data sparsity problem, we can look further into more auxiliary
information, which we can utilize along with key-phrases and their importance
scores. Additionally, we wish to apply a similar model for analyzing domains
where such data sparsity is generic, like insurance fraud detection, patient
record summarization, etc.

## References

  * [1] S. Dutta, A. K. Das, A. Bhattacharya, G. Dutta, K. K. Parikh, A. Das, D. Ganguly, Community detection based tweet summarization, Emerging Technologies in Data Mining and Information Security (2019) 797–808.
  * [2] A. Sharma, K. Rudra, N. Ganguly, Going beyond content richness: Verified information aware summarization of crisis-related microblogs, in: Proceedings of the 28th ACM International Conference on Information and Knowledge Management, ACM, New York, NY, USA, 2019, pp. 921–930.
  * [3] K. Rudra, P. Goyal, N. Ganguly, P. Mitra, M. Imran, Identifying sub-events and summarizing disaster-related information from microblogs, in: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, ACM, New York, NY, USA, 2018, pp. 265–274.
  * [4] P. K. Garg, R. Chakraborty, S. K. Dandapat, Ontodsumm: Ontology-based tweet summarization for disaster events, IEEE Transactions on Computational Social Systems (2023).
  * [5] P. K. Garg, R. Chakraborty, S. K. Dandapat, Endsum: Entropy and diversity based disaster tweet summarization, in: Proceedings of Text2Story - Fifth Workshop on Narrative Extraction From Texts held in conjunction with the 44th European Conference on Information Retrieval (ECIR 2022), Stavanger, Norway, April 10, 2022, Vol. 3117, 2022, pp. 91–96.
  * [6] S. Dutta, V. Chandra, K. Mehra, A. K. Das, T. Chakraborty, S. Ghosh, Ensemble algorithms for microblog summarization, IEEE Intelligent Systems 33 (3) (2018) 4–14.
  * [7] H. R. Faghihi, B. Alhafni, K. Zhang, S. Ran, J. Tetreault, A. Jaimes, Crisisltlsum: A benchmark for local crisis event timeline extraction and summarization, in: Findings of the Association for Computational Linguistics: EMNLP 2022, 2022, pp. 5455–5477.
  * [8] F. Vitiugin, C. Castillo, Cross-lingual query-based summarization of crisis-related social media: An abstractive approach using transformers, in: Proceedings of the 33rd ACM Conference on Hypertext and Social Media, 2022, pp. 21–31.
  * [9] C. Lin, Z. Ouyang, X. Wang, H. Li, Z. Huang, Preserve integrity in realtime event summarization, ACM Transactions on Knowledge Discovery from Data (TKDD) 15 (3) (2021) 1–29.
  * [10] K. Rudra, P. Goyal, N. Ganguly, M. Imran, P. Mitra, Summarizing situational tweets in crisis scenarios: An extractive-abstractive approach, IEEE Transactions on Computational Social Systems 6 (5) (2019) 981–993.
  * [11] D. Rudrapal, A. Das, B. Bhattacharya, A new approach for twitter event summarization based on sentence identification and partial textual entailment, Computación y Sistemas 23 (3) (2019) 1065–1078.
  * [12] T. H. Nguyen, K. Rudra, Rationale aware contrastive learning based approach to classify and summarize crisis-related microblogs, in: Proceedings of the 31st ACM International Conference on Information and Knowledge Management, CIKM ’22, Association for Computing Machinery, New York, NY, USA, 2022, p. 1552–1562.  doi:10.1145/3511808.3557426.
URL https://doi.org/10.1145/3511808.3557426

  * [13] T. H. Nguyen, K. Rudra, Towards an interpretable approach to classify and summarize crisis events from microblogs, in: Proceedings of the ACM Web Conference 2022, 2022, pp. 3641–3650.
  * [14] A. See, P. J. Liu, C. D. Manning, Get to the point: Summarization with pointer-generator networks, arXiv preprint arXiv:1704.04368 (2017).
  * [15] J. Zhang, Y. Zhao, M. Saleh, P. Liu, Pegasus: Pre-training with extracted gap-sentences for abstractive summarization, in: International Conference on Machine Learning, PMLR, 2020, pp. 11328–11339.
  * [16] I. Beltagy, M. E. Peters, A. Cohan, Longformer: The long-document transformer, arXiv preprint arXiv:2004.05150 (2020).
  * [17] W. Qi, Y. Yan, Y. Gong, D. Liu, N. Duan, J. Chen, R. Zhang, M. Zhou, Prophetnet: Predicting future n-gram for sequence-to-sequencepre-training, in: Findings of the Association for Computational Linguistics: EMNLP 2020, 2020, pp. 2401–2410.
  * [18] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, Exploring the limits of transfer learning with a unified text-to-text transformer, The Journal of Machine Learning Research 21 (1) (2020) 5485–5551.
  * [19] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer, Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7871–7880.
  * [20] P. K. Garg, R. Chakraborty, S. Gupta, S. K. Dandapat, Ikdsumm: Incorporating key-phrases into bert for extractive disaster tweet summarization, Computer Speech & Language 87 (2024) 101649.  doi:https://doi.org/10.1016/j.csl.2024.101649.
  * [21] Z. Wen, L. Gui, Q. Wang, M. Guo, X. Yu, J. Du, R. Xu, Sememe knowledge and auxiliary information enhanced approach for sarcasm detection, Information Processing & Management 59 (3) (2022) 102883.
  * [22] J. Ni, Z. Huang, Y. Hu, C. Lin, A two-stage embedding model for recommendation with multimodal auxiliary information, Information Sciences 582 (2022) 22–37.
  * [23] K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, P. Blunsom, Teaching machines to read and comprehend, Advances in neural information processing systems 28 (2015).
  * [24] W. Liu, Y. Sun, B. Yu, H. Wang, Q. Peng, M. Hou, H. Guo, H. Wang, C. Liu, Automatic text summarization method based on improved textrank algorithm and k-means clustering, Knowledge-Based Systems 287 (2024) 111447.
  * [25] Y. Qu, W. E. Zhang, J. Yang, L. Wu, J. Wu, Knowledge-aware document summarization: A survey of knowledge, embedding methods and architectures, Knowledge-Based Systems 257 (2022) 109882.
  * [26] R. Srivastava, P. Singh, K. Rana, V. Kumar, A topic modeled unsupervised approach to single document extractive text summarization, Knowledge-Based Systems 246 (2022) 108636.
  * [27] Q. Zhao, J. Niu, X. Liu, Als-mrs: Incorporating aspect-level sentiment for abstractive multi-review summarization, Knowledge-Based Systems 258 (2022) 109942.
  * [28] L. Xiao, H. He, Y. Jin, Fusionsum: Abstractive summarization with sentence fusion and cooperative reinforcement learning, Knowledge-Based Systems 243 (2022) 108483.
  * [29] D. Jain, M. D. Borah, A. Biswas, Bayesian optimization based score fusion of linguistic approaches for improving legal document summarization, Knowledge-Based Systems 264 (2023) 110336.
  * [30] P. Bhattacharya, S. Poddar, K. Rudra, K. Ghosh, S. Ghosh, Incorporating domain knowledge for extractive summarization of legal case documents, in: Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, 2021, pp. 22–31.
  * [31] O. Ahuja, J. Xu, A. Gupta, K. Horecka, G. Durrett, Aspectnews: Aspect-oriented summarization of news documents, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 6494–6506.
  * [32] Á. Hernández-Castañeda, R. A. García-Hernández, Y. Ledeneva, C. E. Millán-Hernández, Language-independent extractive automatic text summarization based on automatic keyword extraction, Computer Speech & Language 71 (2022) 101267.
  * [33] A. Curiel, C. Gutiérrez-Soto, J.-R. Rojano-Cáceres, An online multi-source summarization algorithm for text readability in topic-based search, Computer Speech & Language 66 (2021) 101143.
  * [34] N. Saini, S. M. Reddy, S. Saha, J. G. Moreno, A. Doucet, Multi-view multi-objective clustering-based framework for scientific document summarization using citation context, Applied Intelligence (2023) 1–25.
  * [35] S. K. Mishra, N. Saini, S. Saha, P. Bhattacharyya, Scientific document summarization in multi-objective clustering framework, Applied Intelligence 52 (2) (2022) 1520–1543.
  * [36] R. Chakraborty, M. Bhavsar, S. K. Dandapat, J. Chandra, Tweet summarization of news articles: An objective ordering-based perspective, IEEE Transactions on Computational Social Systems 6 (4) (2019) 761–777.
  * [37] R. Chakraborty, M. Bhavsar, S. Dandapat, J. Chandra, A network based stratification approach for summarizing relevant comment tweets of news articles, in: International Conference on Web Information Systems Engineering, 2017, pp. 33–48.
  * [38] R. Boorugu, G. Ramesh, A survey on nlp based text summarization for summarizing product reviews, in: 2020 Second International Conference on Inventive Research in Computing Applications (ICIRCA), IEEE, 2020, pp. 352–356.
  * [39] N. Komwad, P. Tiwari, B. Praveen, C. R. Chowdary, A survey on review summarization and sentiment classification, Knowledge and Information Systems (2022) 1–39.
  * [40] S. Priya, R. Sequeira, J. Chandra, S. K. Dandapat, Where should one get news updates: Twitter or reddit, Online Social Networks and Media 9 (2019) 17–29.
  * [41] P. Goyal, P. Kaushik, P. Gupta, D. Vashisth, S. Agarwal, N. Goyal, Multilevel event detection, storyline generation, and summarization for tweet streams, IEEE Transactions on Computational Social Systems 7 (1) (2019) 8–23.
  * [42] Y. Huang, C. Shen, T. Li, Event summarization for sports games using twitter streams, World Wide Web 21 (3) (2018) 609–627.
  * [43] M. Gillani, M. U. Ilyas, S. Saleh, J. S. Alowibdi, N. Aljohani, F. S. Alotaibi, Post summarization of microblogs of sporting events, in: Proceedings of the 26th International Conference on World Wide Web Companion, ACM, New York, NY, USA, 2017, pp. 59–68.
  * [44] X. Zheng, A. Sun, K. Muthuswamy, Tweet-aware news summarization with dual-attention mechanism, in: Companion Proceedings of the Web Conference 2021, ACM, New York, NY, USA, 2021, pp. 473–480.
  * [45] Y. Duan, A. Jatowt, Across-time comparative summarization of news articles, in: Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, ACM, New York, NY, USA, 2019, pp. 735–743.
  * [46] P. K. Garg, R. Chakraborty, S. K. Dandapat, Portrait: a hybrid approach to create extractive ground-truth summary for disaster event (2023).  arXiv:2305.11536.
  * [47] R. Panchendrarajan, W. Hsu, M. Li Lee, Emotion-aware event summarization in microblogs, in: Companion Proceedings of the Web Conference 2021, ACM, New York, NY, USA, 2021, pp. 486–494.
  * [48] T.-Y. Kim, J. Kim, J. Lee, J.-H. Lee, A tweet summarization method based on a keyword graph, in: Proceedings of the 8th International Conference on Ubiquitous Information Management and Communication, ACM, New York, NY, USA, 2014, pp. 1–8.
  * [49] L. Dong, N. Yang, W. Wang, F. Wei, X. Liu, Y. Wang, J. Gao, M. Zhou, H.-W. Hon, Unified language model pre-training for natural language understanding and generation, Advances in neural information processing systems 32 (2019).
  * [50] J. D. M.-W. C. Kenton, L. K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, in: Proceedings of naacL-HLT, Vol. 1, 2019, p. 2.
  * [51] Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, S. Fidler, Aligning books and movies: Towards story-like visual explanations by watching movies and reading books, in: Proceedings of the IEEE international conference on computer vision, 2015, pp. 19–27.
  * [52] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, Advances in neural information processing systems 30 (2017).
  * [53] R. Nallapati, B. Zhou, C. Gulcehre, B. Xiang, et al., Abstractive text summarization using sequence-to-sequence rnns and beyond, in: Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, 2016, pp. 280–290.
  * [54] F. Alam, U. Qazi, M. Imran, F. Ofli, Humaid: Human-annotated disaster incidents data from twitter with deep learning benchmarks, in: Proceedings of the Fifteenth International AAAI Conference on Web and Social Media, ICWSM, AAAI Press, 2021, pp. 933–942.
  * [55] A. Olteanu, C. Castillo, F. Diaz, S. Vieweg, Crisislex: A lexicon for collecting and filtering microblogged communications in crises, in: Proceedings of the international AAAI conference on web and social media, Vol. 8, 2014, pp. 376–385.
  * [56] S. Priya, M. Bhanu, S. K. Dandapat, K. Ghosh, J. Chandra, Taqe: tweet retrieval-based infrastructure damage assessment during disasters, IEEE transactions on computational social systems 7 (2) (2020) 389–403.
  * [57] K. Rudra, N. Ganguly, P. Goyal, S. Ghosh, Extracting and summarizing situational information from the twitter social media during disasters, ACM Transactions on the Web (TWEB) 12 (3) (2018) 1–35.
  * [58] A. Olteanu, S. Vieweg, C. Castillo, What to expect when the unexpected happens: Social media communications across crises, in: Proceedings of the 18th ACM conference on computer supported cooperative work & social computing, ACM, New York, NY, USA, 2015, pp. 994–1009.
  * [59] M. Imran, P. Mitra, C. Castillo, Twitter as a lifeline: Human-annotated twitter corpora for nlp of crisis-related messages, in: Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC), 2016, pp. 1638–1643.
  * [60] F. Alam, F. Ofli, M. Imran, Crisismmd: Multimodal twitter datasets from natural disasters, in: Twelfth international AAAI conference on web and social media, 2018.
  * [61] K. Rudra, S. Ghosh, N. Ganguly, P. Goyal, S. Ghosh, Extracting situational information from microblogs during disaster events: a classification-summarization approach, in: Proceedings of the 24th ACM international on conference on information and knowledge management, 2015, pp. 583–592.
  * [62] O. Vinyals, M. Fortunato, N. Jaitly, Pointer networks, Advances in neural information processing systems 28 (2015).
  * [63] K. Cho, B. Merrienboer, C. Gulcehre, F. Bougares, H. Schwenk, Y. Bengio, Learning phrase representations using rnn encoder-decoder for statistical machine translation, in: EMNLP, 2014.
  * [64] K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, J. Schmidhuber, Lstm: A search space odyssey, IEEE transactions on neural networks and learning systems 28 (10) (2016) 2222–2232.
  * [65] J. Chung, C. Gulcehre, K. Cho, Y. Bengio, Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint arXiv:1412.3555 (2014).
  * [66] A. Graves, J. Schmidhuber, Framewise phoneme classification with bidirectional lstm and other neural network architectures, Neural networks 18 (5-6) (2005) 602–610.
  * [67] G. Liu, J. Guo, Bidirectional lstm with attention mechanism and convolutional layer for text classification, Neurocomputing 337 (2019) 325–338.
  * [68] D. Bahdanau, K. Cho, Y. Bengio, Neural machine translation by jointly learning to align and translate, arXiv preprint arXiv:1409.0473 (2014).
  * [69] M. Gaur, S. Shekarpour, A. Gyrard, A. Sheth, empathi: An ontology for emergency managing and planning about hazard crisis, in: 2019 IEEE 13th International Conference on Semantic Computing (ICSC), IEEE, 2019, pp. 396–403.
  * [70] J. Gu, Z. Lu, H. Li, V. O. Li, Incorporating copying mechanism in sequence-to-sequence learning, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016, pp. 1631–1640.
  * [71] Z. Tu, Z. Lu, Y. Liu, X. Liu, H. Li, Modeling coverage for neural machine translation, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2016, pp. 76–85.
  * [72] C. Tillmann, H. Ney, Word reordering and a dynamic programming beam search algorithm for statistical machine translation, Computational linguistics 29 (1) (2003) 97–133.
  * [73] H. Zogan, I. Razzak, S. Jameel, G. Xu, Depressionnet: learning multi-modalities with user post summarization for depression detection on social media, in: proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, 2021, pp. 133–142.
  * [74] J. Duchi, E. Hazan, Y. Singer, Adaptive subgradient methods for online learning and stochastic optimization., Journal of machine learning research 12 (7) (2011).
  * [75] S. Poddar, A. M. Samad, R. Mukherjee, N. Ganguly, S. Ghosh, Caves: A dataset to facilitate explainable classification and summarization of concerns towards covid vaccines, in: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2022, pp. 3154–3164.
  * [76] C.-Y. Lin, Rouge: A package for automatic evaluation of summaries, in: Text summarization branches out, 2004, pp. 74–81.
  * [77] R. Vadlapudi, R. Katragadda, On automated evaluation of readability of summaries: Capturing grammaticality, focus, structure and coherence, in: Proceedings of the NAACL HLT 2010 student research workshop, 2010, pp. 7–12.
  * [78] N. Iskender, T. Polzehl, S. Möller, Reliability of human evaluation for text summarization: Lessons learned and challenges ahead, in: Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval), 2021, pp. 86–96.
  * [79] A. R. Fabbri, W. Kryściński, B. McCann, C. Xiong, R. Socher, D. Radev, Summeval: Re-evaluating summarization evaluation, Transactions of the Association for Computational Linguistics 9 (2021) 391–409.
  * [80] T. Goyal, J. J. Li, G. Durrett, News summarization and evaluation in the era of gpt-3, arXiv preprint arXiv:2209.12356 (2022).