# WikiDes: A Wikipedia-Based Dataset for Generating Short Descriptions from
Paragraphs

Hoang Thang Ta<EMAIL_ADDRESS>Abu Bakar Siddiqur Rahman
<EMAIL_ADDRESS>Navonil Majumder<EMAIL_ADDRESS>Amir Hussain<EMAIL_ADDRESS>Lotfollah Najjar<EMAIL_ADDRESS>Newton Howard<EMAIL_ADDRESS>Soujanya Poria<EMAIL_ADDRESS>Alexander Gelbukh<EMAIL_ADDRESS>Centro de Investigación en Computación
(CIC), Instituto Politécnico Nacional (IPN), Mexico  College of Information
Science and Technology, University of Nebraska Omaha, USA  ISTD, Singapore
University of Technology and Design, Singapore  Edinburgh Napier University,
UK  University of Oxford, UK

###### Abstract

As free online encyclopedias with massive volumes of content, Wikipedia and
Wikidata are key to many Natural Language Processing (NLP) tasks, such as
information retrieval, knowledge base building, machine translation, text
classification, and text summarization. In this paper, we introduce WikiDes, a
novel dataset to generate short descriptions of Wikipedia articles for the
problem of text summarization. The dataset consists of over 80k English
samples on 6987 topics. We set up a two-phase summarization method —
description generation (Phase I) and candidate ranking (Phase II) — as a
strong approach that relies on transfer and contrastive learning. For
description generation, T5 and BART show their superiority compared to other
small-scale pre-trained models. By applying contrastive learning with the
diverse input from beam search, the metric fusion-based ranking models
outperform the direct description generation models significantly up to
$\approx$ 22 ROUGE in topic-exclusive split and topic-independent split.
Furthermore, the outcome descriptions in Phase II are supported by human
evaluation in over 45.33% chosen compared to 23.66% in Phase I against the
gold descriptions. In the aspect of sentiment analysis, the generated
descriptions cannot effectively capture all sentiment polarities from
paragraphs while doing this task better from the gold descriptions. The
automatic generation of new descriptions reduces the human efforts in creating
them and enriches Wikidata-based knowledge graphs. Our paper shows a practical
impact on Wikipedia and Wikidata since there are thousands of missing
descriptions. Finally, we expect WikiDes to be a useful dataset for related
works in capturing salient information from short paragraphs. The curated
dataset is publicly available at: https://github.com/declare-lab/WikiDes.

###### keywords:

Text summarization , contrastive learning , sentiment analysis , metric fusion
, Wikipedia , Wikidata

††journal: Information Fusion

##  1 Introduction

Text summarization is the task of producing a summary that keeps salient
information of a certain document. The task can be monolingual or cross-
lingual [1]. The monolingual task has been addressed for languages other than
English [2]. In dataset building, the document and the summary are aligned as
a pair, having various lengths depending on the purpose of use. Free online
platforms such as Wikipedia and Wikidata provide massive-scale and diverse
content to assemble information for summarization tasks. A Wikidata item
contains a short descriptive phrase to distinguish between items with the same
or similar labels. In the mobile version, these descriptions appear on the top
of Wikipedia articles, helping users to perceive the topic of articles they
want to read [3]. Since being sister projects, Wikidata and Wikipedia have
connected each other by interwiki links which store on Wikidata. We observe a
high correlation between Wikidata descriptions and Wikipedia articles,
especially in the first paragraphs. Therefore, our objective is to construct a
novel dataset named WikiDes for generating short descriptions as summaries
from the first paragraphs as documents. WikiDes is a monolingual dataset with
over 80k English samples with 6987 instances as topics, extracted data from
both Wikipedia and Wikidata.

With the rapid development of Wikipedia and Wikidata in recent years, the
editor community has been overloaded with contributing new information
adapting to user requirements, and patrolling the massive content daily.
Hence, the application of NLP and deep learning is key to solving these
problems effectively. In this paper, we propose a summarization approach
trained on WikiDes that generates missing descriptions in thousands of
Wikidata items, which reduces human efforts and boosts content development
faster. The summarizer is responsible for creating descriptions while humans
toward a role in patrolling the text quality instead of starting everything
from the beginning. Our work can be scalable to multilingualism, which takes a
more positive impact on user experiences in searching for articles by short
descriptions in many Wikimedia projects.

Some authors mentioned different types of summarization, which depend on user
requirements in their survey works about text summarization systems [4, 5]. If
classifying by summary content, there are indicative summarization and
informative summarization. Indicative summarization systems retrieve about 5
to 10 percent of content as the main idea from the document. This kind of
summarization has the role of encouraging users to continue reading the
document because the summary only provides brief information about the subject
of the document. Meanwhile, informative summarization systems offer a brief
version, which can replace the main document [4]. In our view, description
generation is an indicative summarization task where descriptions are short
text pieces and just enough to let users know about the article topic. For
example, "family" is the description of article "The House of FitzJames".

Sakota et al. [3] conducted similar work to our paper on generating short
descriptions from paragraphs and considered it a type of extreme summarization
[6]. They used Wikipedia’s first paragraphs, Wikidata descriptions, and
Wikidata instances as the input for the summarization. Their multilingual
dataset is massive-scale when capturing content from Wikipedia and Wikidata in
over 25 languages, which makes the model architecture bulky with a high
training cost. Therefore, they chose to turn the custom attention mechanism
instead fine-tune the encoder over languages in the training process. In
contrast, we offer an available English dataset that uses Wikipedia’s first
paragraphs as the input and Wikidata descriptions as the output. Our work
relies on the correlation between Wikipedia and Wikidata which the input is
independent of the output. The main task is to generate summaries for a
project from the content of another project. We apply two-phase summarization
— description generation and candidate ranking — to improve the generation
performance supported by contrastive learning and beam search. Thus, the
generated descriptions capture more salient information from paragraphs,
making our summarization task less "extreme". For example, "noble house
founded by James FitzJames, 1st Duke of Berwick" is the generated description
of the article "The House of FitzJames" instead of the gold description
"family".

In short, we introduce WikiDes, a novel dataset for description summarization,
which creates short descriptions in Wikidata style from given paragraphs. The
main contributions are as follows:

  * 1.

Dataset creation: We provide an available dataset on
GitHub111https://github.com/declare-lab/WikiDes for research purposes related
to Wikipedia summarization.

  * 2.

Setting up a trending approach: We apply two-phase summarization – description
generation and candidate ranking [7, 8] – to improve the quality of the
generated descriptions. In more detail, we deploy transfer learning from
small-scale pre-trained models such as T5 and BART for description generation
and contrastive learning for candidate ranking by fusing metrics, such as
cosine similarity and ROUGE.

  * 3.

Sentiment correlation: We measure the correlations of the generated
descriptions versus the paragraphs and the gold descriptions by comparing
their sentiment polarities on cumulative distribution and the Kolmogorov-
Smirnov test.

The remainder of this paper is organized as follows: Section 2 describes
related works to the summarization tasks, especially ones related to Wikipedia
and Wikidata. Section 3 and Section 4 introduce how to create WikiDes and
perform several data analyses on it. We present our methods of description
generation and sentiment measurement in Section 5 and use them for the
experiments in Section 6 with human evaluation and error analysis. Finally, we
give our conclusion and outline the future work in Section 7.

##  2 Related Works

In this section, we present existing datasets and deep learning approaches for
text summarization not only in wiki text but also in other domains. The works
on integrating sentiment analysis into text summarization are also outlined by
some prominent approaches. Besides, the evaluation metrics such as ROUGE which
evaluate the quality of generated summaries over the overlap of semantic units
or embeddings similarity are mentioned in Section 6.1.

###  2.1 Datasets Extracted from Wikipedia and Wikidata

Monolingual and multilingual are two types of datasets that collect content as
articles and knowledge graphs as RDF triples from Wikipedia and Wikidata.
Monolingual datasets are usually in English because the most coverage and
massive-scale content appear in English Wikipedia in a great collaborative
effort from millions of editors. On another point, multilingual datasets
capture the multilingualism strength of Wikipedia when this encyclopedia
supports up to 327 language
editions222https://meta.wikimedia.org/wiki/List_of_Wikipedias.

Monolingual datasets are built from several popular languages, of which
English and German are two typical examples. Anjalie et al. [9] built a large
dataset in English, including 14.4M articles with section titles and their
content. The summarization task is to create a title from a given section’s
content. Frefel [10] collected a large corpus of 240,000 texts in German
Wikipedia. For each article, they considered the first paragraph as a summary
and the rest of the content as the document. Haichao et al. [11] introduced
WIKIREF, a large query-focused summarization dataset collected from Wikipedia
articles with more than 280,000 examples. This research benefits from the
information synthesis process of Wikipedia editors in writing articles.

Multi-document summarization (MDS) exists in some monolingual datasets. Zopf
et al. [12] proposed hMDS, a new, heterogeneous, and multi-genre corpus for
MDS. Later, the same first author upgraded this work to auto$-$hMDS, a
multilingual multi-document summarization (MMS) dataset [13]. Diego and
Faltings [14] built GameWikiSum, an MDS dataset in the game domain with 14,652
samples. The dataset has video game reviews taken from online platforms such
as Play Station or Xbox as documents and gameplay sections of Wikipedia
articles as summaries. Another MDS dataset extracted content from Wikipedia
Current Events Portal (WCEP) was collected by Demian Gholipour et al. [15] to
provide summaries for news events.

To convert a dataset from monolingual to multilingual, we can use machine
translation systems to create pseudo-cross-lingual summarization data.
However, this method may contain noises from the translation process so there
is better to create cross-lingual datasets without translation. Mehwish and
Strube [16] collected a cross-lingual dataset from Spektrum der Wissenschaft
and Wikipedia’s Science portals (WSP) with 51,312 English and German
scientific articles. Laura and Lapata [17] selected Wikipedia body paragraphs
and leading paragraphs as document-summary pairs and their cross-lingual
dataset XWikis contains 12 languages. Later, Pavel and Malykh [18] built
WikiMulti from Wikipedia good articles. For each article, the first paragraph
is a summary, and the remaining content is a document. WikiMulti contains
22,061 unique English articles and 9,639 articles on average in other
languages that align with English articles.

###  2.2 Other Summarization Datasets

Some early notable datasets are Document Understanding Conference (DUC) and
Text Analysis Conference (TAC), sponsored by NIST for the summarization task
on a small set of documents. DUC had a series from 2001-2007, then became a
Summarization track of TAC in 2008. They both focused on generic and question
summaries of English newspapers and newswire articles. DUC contains two
evaluation methods: a baseline by an automatic system in NIST and human
evaluation by the linguistic quality and conciseness [19]. As a pioneer of the
guided summarization task, TAC 2010 required generating a 100-word summary for
a given topic from a set of 10 Newswire articles [20].

In the domain of news content, Gigaword, New York Times (NYT), CNN/DailyMail
newspapers (CNNDM or CNN/DM), and Newsroom are several examples of popular
datasets, created by collecting hundreds to millions of news articles from
publishers. Gigaword is an available large-scale corpus of English news with
nearly 10 million documents from seven news outlets. NYT has over 65,0000
articles-summary pairs [21] while CNNDM is more popular and contains 93K
articles from CNN (Cable News Network) and 220K articles from DailyMail [22].
Newsroom is another large-scale dataset in the news domain, with 1.3 million
summaries collected from 38 news publishers. The content has various writing
styles due to extracting the source texts from diverse sources such as social
media and articles on the Internet [23]. Different from mentioned datasets,
Multi-News is the first news dataset that relied on the MDS of news articles.
[24]. From a set of new events, the main task is to generate a well-organized
summary, which can cover all events comprehensively and simultaneously avoid
redundancy. XSum is similar to our dataset, WikiDes in terms of extreme
summarization when generating a short, one-sentence news summary that just
answers the question “What is the article about?” [6]. In our case, we
generate a short description that can represent a given paragraph by taking
the salient information.

News datasets usually contain typical writing styles of journalists with
importance in the first paragraph. However, Wiki datasets bring a new wind of
diversity with writing styles from ordinary people. WikiHow is a large corpus
of more than 230,000 article and summary pairs. Only some common features of
n-grams exist between the source articles and the reference summaries in
WikiHow. The more unique n-grams between the source articles and the reference
summaries, the more it is challenging to generate a quality summary compared
to the reference summary [25]. WikiSum has the same knowledge base as WikiHow
but uses simple English in documents. Summaries are coherent paragraphs as
tips written by the document authors in a friendly manner. Therefore, its
content is highly readable and easily comprehensive for readers [26].

In the academic domain, Arxiv and PubMed are summarization datasets with
coherent paragraph summaries in scientific writing styles [27], and the
document length is significantly longer than the one in the news domain. The
content of arXiv and PubMed was retrieved from scientific papers on arXiv.org
and PubMed.com. Another dataset, BIGPATENT contains 1.3 million records of
U.S. patent documents and human-written abstractive summaries [28]. Besides
Multi-News and WikiSum, Multi-XScience is also a large-scale MDS dataset
collected from scientific articles [29]. It brings a challenge for the MDS
task: given a paper, generate a related-work section from the paper abstract
and the abstracts of reference pieces.

###  2.3 Deep Learning Approaches for Text Summarization

As a traditional task in NLP, text summarization has a long development
history with many methods or approaches invented to address the summarization
issues. Some prominent approaches can be named as statistical-based
approaches, topic-based approaches (LSA [30], topic themes [31]), and graph-
based approaches (LexRank [32], TextRank [33], Opinosis [34], GraphSum [35]),
and approaches based on machine learning [36]. In this section, we focus on
deep learning approaches due to their effectiveness recently in generating
quality summaries.

Many summarization tasks apply encoder-decoder architectures by different deep
learning methods. Alexander M. et al. [37] and Ramesh et al. [38] used an
attention-based encoder-decoder model to create summaries in abstractive
summarization. A neural sequence-to-sequence model sometimes reproduces
incorrect factual details and repeats text pieces. To solve these problems,
See et al. [39] proposed a pointer-generator network (PGN) to point to copy
words from the source documents for supporting correct information
reproduction and avoid repetition when tracking the sentences covered in the
document. In another work, Ramesh et al. [40] proposed a two-layer
bidirectional Gated Recurrent Unit (GRU) recurrent neural network-based (RNN)
classifier to generate the extractive summary based on the content richness of
each sentence and saliency based on the overall document. Narayan et al. [6]
built a topic-conditioned neural model based on convolutional neural networks
(CNN). Compared to RNNs, convolution layers hold long-range dependencies
between words better and allow doing the inference, abstraction, and
paraphrasing at a document level.

Recently, transformers has emerged as a library to construct high-capacity
models with transformer architecture and apply pretraining effectively for
diverse NLP tasks, including text summarization [41]. This library allows
researchers can inherit many pre-trained models such as BERT [42, 43], BART
[44], T5 [45], and PEGASUS [46] to extend training on text summarization
datasets not only for academics but also for industrial sectors. Instead of
taking time to train datasets in the beginning, some works used transfer
learning from these pre-trained models to their downstream tasks [47, 48].

Using multilingual BART, Sakota et al. [3] did a similar task to our work on
generating short descriptions from the first paragraphs of Wikipedia articles,
providing a quick insight into a Wikipedia article. They deployed Descartes
(Description of articles), a model built based on a pre-trained multilingual
BART model with data in 25 languages. In multilingual content generation, a
created summary is usually in the same language as the document . In this
case, helpful texts in other languages can not utilize. Authors thus leveraged
the multilingualism of Wikipedia so that any summarizer can generate a
description in a language without requiring the source document in that
language [3]. However, this method creates a bulky generation architecture and
takes more natural noises from Wikipedia and Wikidata. Due to the openness of
Wikimedia projects [49], the community sometimes can not prevent vandalism
types on contribution content in many language editions. Hence, the more
content used in more languages, the more noise the dataset may have.

Contrastive learning is a popular technique to maximize similarities of
feature representation of the same images and minimize those in the different
images. Based on a similar sense, Shusheng et al. [50] used contrastive
learning in summarization tasks by maximizing the similarities of the same
semantic meaning articles. Liu and Liu [7] also used BART and RoBERTa in two-
phase summarization to generate and score candidate summaries which were
generated from diverse beam search [51] to improve the qualities of output
texts. The benefit of two-phase summarization mainly lies in the decoding
strategy by beam search and other superior methods such as nucleus sampling
[52], where diverse data creates more opportunities to search for an "ideal"
candidate. In one of the most recent works, Liu et al. [53] followed a new
training paradigm that assigns probabilities of candidate summaries concerning
to their quality in contrastive learning. So far, some other summarization
works have been done with contrastive learning in different methods [54, 55,
56, 57, 58]. In addition, there are different approaches, including few-shot
and zero-shot learning [59], reinforcement learning [48], prompting [60],
prefix-tuning on massive-scale models (GTP-2) [61], hybrid approaches [62],
and others for various types of text summarization.

###  2.4 Sentiment Analysis in Text Summarization

An ideal and typical summary is able to capture the salient information from a
given document [63, 64]. However, it may lack sentiment information of the
document, which are key in datasets like IMDB movie reviews. For capturing
sentiment, sentiment analysis is incorporated into text summarization models.

One of the integration approaches is to extract sentiment polarity (negative,
neutral, and positive scores) in texts [65], which allows us to turn sentiment
analysis into a problem of text sentiment classification [66, 67, 68]. Beineke
et al. [69] introduced a method for sentiment summary generation, which
discovers a key aspect of the author’s opinion in movie reviews on the Rotten
Tomatoes website and applied Naive Bayes and regularized logistic regression
to fit features extracted from good summaries in text classification. Another
approach to combining sentiment analysis in text summarization is aspect-based
sentiment summarization (ABSA), which contains two tasks, aspect
identification with mention extraction and sentiment classification. Wu et al.
[70] mentioned a definition named Aspect-based Opinion Summarization (AOS)
with the same tasks as ASBA. Titov and McDonald [71] presented a joint model
of text and aspect ratings, which used a Multi-Aspect Sentiment model (MAS) to
form representative aspects as topics and a set of sentiment predictors to
demonstrate the correlations between a topic and a particular aspect. In
another work, Dhanush et al. [72] designed an RNN for extracting aspects with
contexts in a sentence and a CNN for sentiment classification at the sentence
level. Several works exploited other sentiment information in texts. For
example, Nishikawa et al. [73] addressed informativeness and readability in
restaurant reviews. They set up an algorithm to create a summary by choosing
and arranging sentences by informativeness and readability scores. Lerman et
al. [74] utilized the benefit of user preferences to construct a new
summarizer on sentiment-based models.

In WikiDes, a large number of texts are neutral due to the neutrality policy
of Wikipedia, so we prefer to determine the uniformity of the sentiment
polarity distribution between descriptions and paragraphs by cumulative
distribution and the Kolmogorov-Smirnov test [75] more than performing
sentiment summarization approaches to capture other sentiment polarities.
Besides the Kolmogorov-Smirnov test, some proper methods for measuring the
distribution among multiple sets are the chi-squared test [76], the
Mann–Whitney U test [77], and the Fisher’s z-test [78].

##  3 Data Creation

We rely on the APIs of Wikidata333https://www.wikidata.org/w/api.php and
English Wikipedia444https://en.wikipedia.org/w/api.php to collect the data in
JSON format. Algorithm 1 is a simple algorithm for selecting samples randomly
to avoid the bias of topic distribution in WikiDes. The is-a or instance-of
attribute of the Wikidata items is considered as the topic of the items. We
eliminate the Wikidata items with following the topics because they do not
have any corresponding articles in Wikipedia:

  * 1.

Scholarly article

  * 2.

Wikimedia disambiguation page

  * 3.

Wikinews article

Given the number of samples $N$ needed to collect, a $while$ loop is used to
extract output samples $S$. For each loop, an $id$ is randomly generated from
1 to 99 million555 https://www.wikidata.org/wiki/Wikidata:Statistics. If the
$id$ does not appear in the scanned list $K$, then continue to extract
information from Wikidata and Wikipedia to build a sample. Next, we validate
the sample, e.g., lengths of paragraphs and descriptions. If it is a good
sample, then save it to $S$. Otherwise, the algorithm continues the $while$
loop until the number of samples $S$ equals the desired given number $N$.

Figure 1: A random sample in WikiDes: The first paragraph (shown in a green
bar) of the Wikipedia article is used to infer the description (shown in an
orange bar) of the corresponding item Q1497 in Wikidata. Algorithm 1 Collect
samples from Wikipedia and Wikidata

Input: N

Output: S

$S,K\leftarrow[],[]$

while $len(S)\leq N$ do

$id\leftarrow random(1,99000000)$

if $id\notin K$ then

$description,instances,...,en\\_site\\_link\leftarrow extractWikidata(id)$

$title,first\\_para,first\\_sen\leftarrow extractWikipedia(en\\_site\\_link)$

$description\leftarrow preProcessing(description)$

$first\\_para\leftarrow preProcessing(first\\_para)$

$first\\_sen\leftarrow extractFirstSentence(first\\_para)$

$sample\leftarrow concateTuple(description,instances,...,first\\_sen)$

if ($sample$ is good) then

add $sample$ to S

end if

add $id$ to K

end if

end while

For each item in Wikidata, the crawler extracts a label, a short description,
instances (P31), and a site link or interwiki link. This link leads to an
article in English Wikipedia, where the crawler can gather the first
paragraph. We apply a few pre-processing steps to paragraphs and descriptions
to remove special symbols and redundant spaces, which are assumed not to
contribute effectively to the model performance in the description generation
process. Furthermore, we discard samples with empty descriptions or paragraphs
having fewer than 10 tokens.

Redirect articles are also discarded because they link to their representative
articles, whose first paragraphs may not necessarily stand for the subjects of
the redirect articles. For example, Dong Nguyen, a game developer links to the
main article named .Gears, which is his company and more famous. In this case,
we can not use the first paragraph of article .Gears representing for Dong
Nguyen. In other technical details, we use the package NLTK punkt to extract
the first sentences from the first paragraphs. Due to using online APIs, we
applied multithreading to our crawler with max_workers=8 to speed up the data
collection process. In less than 72 hours, 81,418 English samples on 6,987
topics were collected to build the dataset for this paper.

Figure 1 describes a typical sample, which contains two essential texts: the
Wikidata description "river system in North America" and the first paragraph
of the corresponding Wikipedia article "The Mississippi River is the second-
longest river and chief river of the second-largest drainage system in North
America…". The task is to generate this description from the given paragraph.
The first element of the instance list ("river") is the baseline candidate in
the experiment.

##  4 Dataset Analyses

In this section, we perform some analyses on the collected dataset, which are
text length, instance distribution, and token position distribution. Later, we
compare WikiDes to existing popular datasets by several features in the
summarization task.

###  4.1 Data Distribution

We count the lengths of paragraphs and descriptions as the number of words,
without considering punctuations. Figure 2 shows the distribution of texts by
lengths in the first 500 lengths. The average length of descriptions is 4.5
while the average length of paragraphs is 82.24. The descriptions are short
with the majority of parts less than 100 words. Meanwhile, the paragraph
distribution by length is more evenly compared to the descriptions.

Figure 2: The distribution of paragraphs (blue) and descriptions (orange) by
their lengths, limited to 500 first lengths. Figure 3: The distribution of
instances in the dataset.

Next, we check the instance distribution by samples in WikiDes with 6987
instances. Wikidata instances (P31) belong to a complex hierarchy system of
hypernyms and hyponyms, defined and built by its user community. In WikiDes,
some most popular instances by descending order are human (30%), taxon (8%),
film (3%), album (2%), village (2%), and human settlement (2%), shown in
Figure 3. The dataset also contains 3905 rare instances (55.88%), appear in
only 1 sample, such as miniature war gaming or mask stone.

###  4.2 Correlation Between Paragraphs and Descriptions

Currently, most pre-trained models for the summarization task such as BART and
T5 support the maximum length of 1024 tokens. As shown in Figure 2, the
paragraphs are relatively short. Before the training process, we check the
overlap rate between paragraphs and descriptions to know a proper maximum
length of paragraphs for the training and reduce the capacity of input data.
In the training set, we chop descriptions into text chunks with different
token sizes (32, 64, 128, 256, 512, and 1024). The proper metric for this
correlation is ROUGE-N-precision, counted by the number of overlap grams
between descriptions and paragraphs over the number of grams in descriptions.
Table 1 shows ROUGE-1, ROUGE-2, and ROUGE-L (or ROUGE-LCS, LCS refers to Long
Common Subsequence) values between paragraphs and descriptions. Hence, 256 is
an optimized length of paragraphs for the data training. ROUGE scores after
256 tokens have a little increment, which is less than 0.2 ROUGE while we have
to feed into the model more than twice or fourth times of tokens.

Table 1: ROUGE scores between paragraphs and descriptions of the training set by different numbers of first tokens. The scores were calculated on 5000 random samples because it takes so much time to perform on the whole training set. number of tokens | R-1-precision | R-2-precision | R-L-precision
---|---|---|---
32 | 51.39 | 27.89 | 48.51
64 | 60.99 | 33.31 | 57.24
128 | 63.13 | 34.09 | 59.13
256 | 63.72 | 34.24 | 59.70
512 | 63.89 | 34.24 | 59.83
1024 | 63.91 | 34.25 | 59.84
Figure 4: The distribution of token positions of descriptions in paragraphs.
Positions are converted to relative values, and their volumes are represented
as ratios. The maximum position is 1024.

To support the optimized length of 256, we take another analysis on the token
positions in paragraphs, displayed in Figure 4. We encode paragraphs and
descriptions by pre-trained model facebook/bart-base with the maximum length
of 1024. Stop-words and punctuations are removed in descriptions; however, not
from paragraphs. We see that most tokens appear 20% of first positions,
roughly 204.8 tokens. Therefore, this confirms the rationality of taking the
maximum length of 256 tokens for the training process.

###  4.3 Data Split

In Figure 3, the instance distribution is used to split the collected dataset
into training ($\approx$ 80%), validation ($\approx$ 10%), and test ($\approx$
10%) sets. Here, instances are topics, and we apply two split methods:

  * 1.

Topic-exclusive split: We create a dictionary of topics by the number of
samples and rank these topics from the most popular to rare. For example, we
have human with 29494 samples, taxon with 7686 samples, film with 2316
samples, etc in Figure 3. The training set contains data with the top popular
topics, while the other topics are allocated randomly to validation and test
sets.

  * 2.

Topic-independent split: In this split, we randomly put data into training,
validation, and test sets without caring for topics.

Table 2: Topics in sets by topic-exclusive split and topic-independent split. Data split | Training set | Validation and Test sets
---|---|---
Topic-exclusive |  6cmhuman, taxon, film, album, village, human settlement, family name, river, business, mountain, video game, chemical compound, organization, gene, radio station, road, high school, building, town, tennis event, fossil taxon, civil parish, airport, lake, city, island, language, military unit, school, political party, metro station, …, reservoir, academic journal. |
There are 156 topics in total. |  6cmsoftware, cemetery, skyscraper, shopping center, free software, music genre, airbase, glacier, television channel, airline, sculpture, aircraft, valley, bridge, county seat, award ceremony, mosque, formation, manuscript, conflict, destroyer, publisher, poem, …, college, protein, monastery, filmography, election, submarine, concept, trademark, toponym, atmosphere. |
There are 6831 topics in total. These topics are allocated randomly into validation and test sets. |  |
Topic-independent |  6cmTopics are taken randomly from 6987 topics in every run time. |  6cmTopics are taken randomly from 6987 topics in every run time.

Topic-exclusive split provides data on different topics to see how well the
model can infer data on unseen topics. Otherwise, topic-independent split
offers the random distribution of data in sets, reflects the similarity of
randomness in data creation, and avoids data bias. In specific, Table 2 shows
the topics in training, validation, and test sets by topic-exclusive split and
topic-independent split. Meanwhile, Section 4.4 represents the number of
samples in different sets by these splits.

Due to a large number of instances with only one sample, we could not be able
to split data by the same topics. These rare instances can be merged into
their parents when they belong to a hierarchy system of hypernyms and hyponyms
in Wikidata. However, this system is complex and overlaps due to a trade-off
to user community [79, 80] with a freedom contribution in Wikidata. Therefore,
it is sometimes difficult to choose the proper parents.

###  4.4 Comparison with Other Summarization Datasets

In this section, we compare our dataset with existing summarization datasets,
especially those related to Wikipedia and Wikidata. Document length (doc.
len.) and summary length (summ. len.) are counted by the average number of
words appearing in texts. In our dataset, the number of words in paragraphs
and descriptions is considered. We apply compression ratio (comp. ratio) [25,
81], calculated by the ratio between the average length of paragraphs and the
average length of descriptions. This value offers a measurement of the task
difficulty. The summarization task is more difficult with the higher value of
compression ratio when the model has to deal with higher levels of abstraction
and semantics. We counted only unique words without punctuations to build the
vocabulary set in WikiDes. The statistics of other datasets were extracted
from their original papers or related papers in our best effort of searching.
However, we self-calculated the compression ratio of these datasets based on
the document length and the summary length that we got in hand.

Table 3: The comparison of our dataset with other existing datasets in the
summarization task.

dataset |  train/val/test | doc. len. | summ. len. | comp. ratio | vocab. size
---|---|---|---|---|---
arXivβ [27] |  215K | 4938 | 220 | 22.44 | -
CNNDMα [26] |  287,113/13,368/11,490 | 789.9 | 55.6 | 14.20 | 717,951
NEWSROOMα [23] |  1,321,995 | 658.6 | 26.7 | 24.66 | 6,925,712
NYTα [26] |  - | 795.9 | 44.9 | 17.72 | -
PubMedβ [27] |  133K | 3016 | 203 | 14.85 | -
Multi-Newsα [24] |  44,972/5,622/5,622 | 2,103.49 | 263.66 | 7.97 | 666,515
Multi-XScienceβ [29] |  30,369/5,066/5,093 | 778.08 | 116.44 | 6.68 | -
WikiDesω |  65,772/7,820/7,827#
68,296/8,540/8,542${}^{\epsdice{6}}$ | 82.24 | 4.50 | 18.27 | 354,946
WikiHowω [25] |  230,843 | 579.8 | 62.1 | 9.33 | 556,461
WikiSumω [29] |  15m/38k/38k | 1,334.2 | 139.4 | 9.57 | -
XSumα [6] |  204,045/11,332/11,334 | 431.07 | 23.26 | 18.53 | -
$\alpha$: news, $\beta$: scientific document, $\omega$: wiki, #: topic-
exclusive, 6: topic-independent

Section 4.4 shows statistics of WikiDes against other existing datasets, where
WikiDes has the lowest value of vocabulary size. In Wikimedia projects
(Wikipedia, Wikidata, Wikibooks, etc.), content is freely contributed by
millions of users from everywhere and complies with formal writing standards
that offer easy readability for readers based on using popular words.
Therefore, any data taken from Wikipedia and Wikidata is normally less diverse
in terms of word usage. WikiDes has a high compression ratio of 18.27,
indicating that the summarization task on it is more difficult than some other
datasets. There is a mismatch between set sizes of topic-exclusive split and
topic-independent split in WikiDes. This is because we keep samples with empty
instances in topic-independent split while removing them in topic-exclusive
split.

##  5 Tasks and Models

###  5.1 Task Description

There are 2 main tasks of this paper:

  * 1.

Two-phase summarization: From a given paragraph $X$ as the input, the task is
to generate the output as a set of candidate descriptions $\tilde{Y}_{1}$,
$\tilde{Y}_{2}$, …, $\tilde{Y}_{n}$, which is ranked by metric values in
descending order. The list also includes the best description $\tilde{Y}_{b}$,
which obtains the highest metric value.

  * 2.

Sentiment correlations: Given two cumulative distributions $F_{1,N}(x)$ and
$F_{2,M}(x)$ of two sets with the sample sizes $N$ and $M$ of the first set
and second set correspondingly. Our task is to calculate a maximum distance
$D$ between two sets and check this value against the rejection thresholds by
some $\alpha$ levels of significance before concluding to accept or reject the
null hypothesis, which is to answer the question: "Do two sets have the same
distribution or not?".

###  5.2 Method

Previous approaches in text summarization use only one-phase summarization, in
which the model learns how to generate summaries based on the probably next
tokens in a sequence-to-sequence architecture. However, as a trending
approach, two-phase summarization shows the effectiveness in improving the
quality of output summaries in many works [7, 8, 82, 83]. The first phase is
to train the summarization model before using it to infer a set of candidate
summaries based on various decoding strategies [52] such as beam search. The
second phase applies contrastive learning [7, 8, 50] to train a ranking model,
which depends on the combination of semantic and lexical similarity of each
element of the candidate set and the gold summary against the document to rank
the output candidates by several popular metrics.

Figure 5: The diagram of two-phase summarization used to generate the quality
descriptions.

Figure 5 presents the diagram of two-phase summarization — description
generation and candidate ranking, which is used as the main method of this
paper. In Phase I, the encoder transforms the given paragraph $X$ to the
representation vector $Z$, then the decoder uses $Z$ to generate a set of
candidate description $\tilde{Y}_{1}$, $\tilde{Y}_{2}$, …, $\tilde{Y}_{n}$
based on beam search configurations. In Phase II, the ranker takes the
candidate list and the gold (target) description $\hat{Y}$ for measuring the
fusion of semantic and lexical similarity with the paragraph $X$. The output
is a candidate list sorted descending by fused-metric values and the best
candidate $\tilde{Y}_{b}$ will have the highest value. Finally, we investigate
the uniformity of sentiment distribution on the output descriptions against
the paragraphs and the gold descriptions by the cumulative test and the
Kolmogorov-Smirnov test.

####  5.2.1 Baseline Method for Description Generation

For each sample, there is a list of instances (P31) ranked by an order in
Wikidata which expresses the is-a relations with the item. In that meaning,
they are considered as topics of a certain description. We consider the first
element of this list as the baseline description. For example, item "Bugema
University" (Q4986155) contains two instances, "university" and "church
college". So we choose "university" as the baseline description.

####  5.2.2 Description Generation Models

The description generation is trained on small-scale pre-trained models with a
Transformer architecture. Invented by Vaswani et al. [84], a Transformer
consists of three components – an encoder, a decoder, and an attention
mechanism – mixed with recurrence and convolutions and applies a sequence-to-
sequence structure for the conditional generation task.

Let define a source paragraph $X={x_{1},x_{2},x_{3},...,x_{n}}$ is a sequence
of $n$ tokens and a target description $Y={y_{1},y_{2},y_{3},...,y_{m}}$ is a
sequence of $m$ tokens. The encoder simply converts $X$ into a representation
sequence $Z={z_{1},z_{2},z_{3},...,z_{n}}$. The number of elements in $Z$
equals to the number of elements in $X$.

$\displaystyle
Z={z_{1},z_{2},z_{3},...,z_{n}}=Encoder(x_{1},x_{2},x_{3},...,x_{n})$ (1)

Next, the decoder is responsible for generating the target description $Y$
from $Z$ in the equation $Y=Decoder(Z)$. Following the chain rule, the
probability $p(Y|Z)$ that the decoder generates $Y$ from $Z$ is:

$\displaystyle p(Y|Z)$ $\displaystyle=\prod_{i}^{m}p(y_{i}|Y_{<i},Z)$ (2)

where $y_{0}$ is the "beginning" token (<bos>) of sentence and $Y_{<i}$ is a
sequence of previous tokens of the token $y_{i}$. At an inference time, the
decoder generates only one output token and concatenates it with previously
generated tokens as additional input to produce a new token. The decoder stops
the inference process when it generates the end token (<eos>) or meets the
maximum length.

The model loss $L_{entropy}$ is a cross-entropy loss that minimizes the sum of
negative loglikelihoods of the tokens:

$\displaystyle
L_{entropy}=-\sum^{m}_{j=1}\sum_{w}p_{true}(w|Y_{<j},Z)log(p(w|Y_{<j},Z))$ (3)

where $p_{true}$ is a one-hot distribution which follows:

$\displaystyle
p_{true}(w|Y_{<j},Z)=\bigg{\\{}\begin{matrix}1\;\;\;\;\;w=y_{j}\\\
0\;\;\;\;\;w\neq y_{j}\end{matrix}$ (4)

In the encoder and the decoder, there are six identical layers, each of which
has two sub-layers containing a multi-head self-attention mechanism and a
simple, position-wise fully connected feed-forward network. Multi-head
attention over the encoder’s output is used as the third sub-layer of each
decoder layer [84].

Scaled Dot-Product Attention and Multi-Head Attention are two attention
functions in Transformers. Let’s define $Q$ as a query matrix, $K$ as the key
matrix, and $Q$ is the value matrix. Let $d_{k}$, $d_{k}$ be the dimensions of
queries and keys and $d_{v}$ be the dimension of values. The attention
function of Scaled Dot-Product Attention can be calculated as the formula
[84]:

$\displaystyle Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V$ (5)

where the scaling factor $\frac{1}{\sqrt{d_{k}}}$ is used to avoid extremely
small gradient of the softmax function when $d_{k}$ has a large value.

To allow the model to learn other information from subspace representation in
various positions, the Multi-Head Attention is performed with
$d_{model}$-dimensional keys, values, and queries [84]:

$\displaystyle MultiHead(Q,K,V)$
$\displaystyle=Concat(head_{1},head_{2},...,head_{h})W^{O}$ (6)
$\displaystyle\text{where }head_{i}$
$\displaystyle=Attention(QW_{i}^{Q},KW_{i}^{K},VW_{i}^{V})$

where $h$ is the number of heads while
$W^{Q}_{i}\in\mathbb{R}^{d_{model}\times d_{k}}$,
$W^{K}_{i}\in\mathbb{R}^{d_{model}\times d_{k}}$,
$W^{V}_{i}\in\mathbb{R}^{d_{model}\times d_{v}}$ are the parameter matrices of
head $i$. When concatenating these heads, we need to multiply with $W^{O}$, a
parameter matrix belonging to $\mathbb{R}^{hd_{v}\times d_{model}}$ to get the
final multi-head matrix.

####  5.2.3 Ranking Models

Inspired by the works of Zhong et al. [8] and Liu and Liu [7], we create a
model for ranking the generated candidates from the best performing models in
Phase I. Given a paragraph $X$ and a reference description $\hat{Y}$, the
model $\mathcal{G}$ will generate a candidate description Y = $\mathcal{G}$
(X), which is compared to the reference description $\hat{Y}$ by a score
$s=M(Y,\hat{Y})$ produced by a metric $M$. We apply an evaluation function
$f(.)$, composed of multiple metrics, to produce different similarity scores
$s_{1},s_{2},...,s_{n}$ between the candidate descriptions ($Y_{1}$, $Y_{2}$,
…, $Y_{n}$) and the source paragraph $X$ based on the given metrics. The
similarity score $s_{i}$ between the candidate description $Y_{i}$ and the
source paragraph $X$ is calculated by:

$\displaystyle s_{i}$ $\displaystyle=f({Y_{i}},X)$ (7)
$\displaystyle=\Bigg{\\{}\begin{matrix}M_{CosineSim}(Y_{i},X)&\text{if using
cosine similarity}\\\ M_{ROUGE}(Y_{i},X)&\text{if using ROUGE}\\\
HM(Y_{i},X)&\text{if using harmonic mean}\end{matrix}$

$\displaystyle HM(Y_{i},X)$
$\displaystyle=HM(M_{ROUGE}(Y_{i},X),M_{CosineSim}(Y_{i},X))$ (8)
$\displaystyle=\frac{2*M_{ROUGE}(Y_{i},X)*M_{CosineSim}(Y_{i},X)}{M_{ROUGE}(Y_{i},X)+M_{CosineSim}(Y_{i},X)}$

$\displaystyle M_{CosineSim}(Y_{i},X)=Cosine(BERT(Y_{i}),BERT(X))$ (9)

In Equation 7, we calculate the score $s_{i}$ between $Y_{i}$ and $X$ by
different given metrics. If using cosine similarity and ROUGE, $f(.)$ provides
cosine similarity and ROUGE values between $Y_{i}$ and $X$ by corresponding
metrics. If using harmonic mean, $f(.)$ fuses cosine similarity
$(M_{CosineSim}$) and ROUGE ($M_{ROUGE}$) using harmonic mean ($HM$) in
Equation 8. For measuring cosine similarity values, we use the embedding
vectors produced by the last hidden layer of BERT as in Equation 9. In
contrast, we use raw texts of $Y_{i}$ and $X$ for measuring ROUGE values.

Supposedly, cosine similarity and ROUGE scores represent two distinct views of
similarity, namely, semantic and lexical, respectively. Thus, fusing them
would likely give us a balanced sense of the quality of the candidates. Our
intuition shows that a good candidate description should take high and
positive cosine similarity values when comparing it to the paragraph.
Therefore, for any description with negative values, we set these values to 0
to lower the description in the ranking. In the output, we can have the list
of candidates ranked by metrics including the best candidate $\tilde{Y}_{b}$
which has the highest score:

$\displaystyle\tilde{Y}_{b}=argmax_{Y_{i}}(f(Y_{i},X))$ (10)

As in some works [56, 85, 86, 87], ranking models were trained on the
difference between positive and negative examples in terms of some metric. The
candidate descriptions generated from Phase I can provide a diverse spectrum
of data; however, they do not provide negative examples. In this case,
contrastive learning reflects the diverse qualities of candidates [7] more
than a contrast between negative and positive examples. Since there are no
negative examples, we use a ranking loss based on margin ranking loss to
$f(.)$:

$\displaystyle L_{ranking}=L_{gold}+L_{candidate}$ (11)

$\displaystyle
L_{gold}=\sum_{i}max\Bigl{(}0,f(X,\tilde{Y}_{i})-f(X,\hat{Y})+\lambda_{gold}\Bigl{)}$
(12)

$\displaystyle
L_{candidate}=\sum_{i}\sum_{j>i}max\Bigl{(}0,f(X,\tilde{Y}_{j})-f(X,\tilde{Y}_{i})+\lambda_{ij}\Bigl{)}$
(13)

The ranking loss in Equation 11 is a sum loss of the gold loss and the
candidate loss. In the gold loss in Equation 12, we compare the difference
between the candidate $\tilde{Y}_{i}$ and the gold description $\hat{Y}$
against the paragraph $X$. We use hyperparameter $\lambda$gold as a gold
margin. The candidate loss in Equation 13 is a difference between each
candidate with the other candidates in their list. At first, the candidate
list is sorted in descending order by scores. Then, $\lambda$ij =
(j-i)*$\lambda$candidate is a margin of each candidate compared to the others
[7]. In the experiment, $\lambda$ij = $\lambda$i = i*$\lambda$candidate with
$i$ is the ranking position, from 1 to list size $n$. The higher ranking a
candidate has, the less margin it takes. A default value of 0.01 is set to
both $\lambda$gold and $\lambda$candidate in the experiment.

Instead of using the ranking loss $L_{ranking}$, we use validation loss
$L_{val}$ to check the model performance.

$\displaystyle L_{val}=1-\frac{1}{N}\sum_{i}^{N}f(\tilde{Y}_{b},\hat{Y}_{i})$
(14)

In Equation 14, the best candidate $\tilde{Y}_{b}$ is the closest one to the
paragraph by metrics or has the highest value $f(\tilde{Y}_{b},\hat{Y}_{i})$.
We count the average value of the best candidates compared to their gold
description in the validation set and subtract it from 1 (the gold value) to
calculate the validation loss.

###  5.3 Sentiment Consistency

Wikimedia stresses the importance of keeping neutrality in texts across its
projects. Thus, when generating descriptions for Wikidata, we must comply with
this principle by guaranting that each machine-generated description will have
the same sentiment polarities, especially neutrality as in its paragraph. The
sentiment consistency helps to evaluate the quality of machine-generated
descriptions in the aspect of sentiment beside the capture of salient
information from the paragraph.

To measure the overall sentiment consistency between the generated description
and the input paragraph, we employ Kolmogorov-Smirnov (K-S) test. The
Kolmogorov-Smirnov (K-S) test is a test to calculate a distance between two
distributions [75, 88]. Let define $F_{1,N}(x)$ and $F_{2,M}(x)$ are the
cumulative distributions with the sample sizes $N$ and $M$ of the first set
and second set correspondingly. The distance $D$ between two sets can be
calculated by Equation 15, where $sup_{x}$ is the supremum function.

$\displaystyle D=sup_{x}|F_{1,N}(x)-F_{2,M}(x)|$ (15)

$\displaystyle c_{\alpha}=\sqrt{-ln(\frac{\alpha}{2}).\frac{1}{2}}$ (16)

$\displaystyle D>c_{\alpha}\sqrt{\frac{N+M}{N\times M}}$ (17)

The critical value $c_{\alpha}$ at the $\alpha$ level of significance is
defined by Equation 16. The null hypothesis (two sets have the same
distribution) is rejected at level $\alpha$ if satisfying the condition of the
inequality in Equation 17 for the large numbers of $N$ and $M$.

##  6 Experiments

###  6.1 Summarization Evaluation Metrics

Currently, there are many metrics used to evaluate the model performance not
only for summarization but also for text generation and machine translation
tasks. In this section, we focus on several metrics that are generally used
for summarization tasks and are broadly mentioned in many scientific papers.

_ROUGE (Recall-Oriented Understudy for Gisting Evaluation)_ is one of the most
popular and conventional metrics [89]. It automatically defines the quality of
a summary by measuring an overlap rate between it and the gold summary created
by humans. This value is reflected by the number of semantic units (n-gram,
word sequences, or word pairs) appearing in both the generated summary and the
gold summary.

_ROUGE-WE_ is an extension of ROUGE that uses Word2Vec embeddings of words
taken from summaries to compute the semantic similarity. Therefore, it is more
proper for abstractive summarization or substantial paraphrasing on summaries.
ROUGE-WE also obtains better correlations with human judgments by Spearman and
Kendall coefficients [90].

_BLEU_ is measured by the number of position-independent and overlapped
n-grams between generated texts and references with a brevity penalty on the
text length. It has several advantages that are execution speed, cheap cost,
language independence, and high correlations with human evaluation [91].

_METEOR_ depends on the matching of unigrams between a hypothesis and a given
reference to compute a score for the hypothesis quality by three word-mapping
modules: exact, stem, and synonymy [92]. F-mean is calculated as a
parametrized harmonic mean of precision P and recall R over single-word
matches. METEOR addresses BLEU’s weakness when applied to low-resource
languages and has a better correlation with human judgment at the
sentence/segment level than BLEU [93].

_MoverScore_ depends on Word Mover’s Distance on contextualized embeddings to
compute a semantic distance between summaries and references instead of using
traditional semantic units such as words or n-grams for the measures. This
metric shows strong generalization capability over many summarization tasks
[94].

_BertScore_ uses token alignments between generated summaries and references
to provide similarity scores. It maximizes cosine similarity between BERT’s
token embeddings by using a greedy matching [95].

_InfoLM_ is a recently proposed metric for evaluating summarization and
data2text generation tasks. In the family of string-based metrics, ROUGE and
BLUE depend on exact matches of semantic units such as n-grams. However, they
can not compare two strings based on synonyms. InfoLM overcomes this drawback
by using a pre-trained masked language model but does not require training to
compute similarity scores between summaries and references over the vocabulary
by discrete probability distributions [96].

As most of the reference descriptions in WikiDes are naturally short and have
a high correlation with the source paragraphs, using ROUGE may be enough for
evaluating the description quality. However, we still use BertScore, METEOR,
and BLEU to have more viewpoints in the result evaluation. Especially,
BertScore brings an ability to match paragraphs and descriptions on semantic
similarity. ROUGE-N-F-measure, BertScore, METEOR, and BLEU are used to
estimate the model performance in two-phase summarization, description
generation, and candidate ranking. In another use, ROUGE-1, ROUGE-2, and
ROUGE-L precision values are for checking the correlation between paragraphs
and descriptions in Section 4.2.

###  6.2 Phase I: Description Generation

####  6.2.1 Models

In Phase I, we use these models:

  * 1.

Baseline model: We apply the same baseline for topic-exclusive split and
topic-independent split on the validation and test sets. As mentioned in
Section 5.2.1, the baseline model considers the first element of the instance
list as the gold description.

  * 2.

Pre-trained models: There are four small-scale pre-trained models, BART-base,
T5-small, T5-base, and SSR-base are downloaded from Huggingface to use in the
training process.

####  6.2.2 Experimental Details and Results

As the trending of summarization tasks, we follow a sequence-to-sequence
fashion and apply transfer learning from small-scale pre-trained models to
train the data with batch_size=8 and epochs=3. The maximum length of the
encoder is 256, while we apply a length of 32 to the decoder to support the
content generation of longer descriptions, though their average gold length is
$\approx$ 4.5. The function Seq2SeqTrainer() of package Transformers is used
for the data training and evaluates validation and test sets by
ROUGE-1-F-measure (R-1), ROUGE-2-F-measure (R-2), and ROUGE-L-F-measure (R-L).
We let the models automatically adjust the learning rate after each epoch, and
save the best model state with the highest metrics on the validation sets.

Table 4: ROUGE, BertScore (BS), METEOR (ME), and BLEU scores between the generated descriptions and the gold descriptions on the validation set in Phase I. All models use greedy decoding. Model |  Topic | Validation set
---|---|---
R-1 | R-2 | R-L | BS | ME | BLEU
BART-base |  exclusive | 44.04 | 25.57 | 43.22 | 89.49 | 36.87 | 7.85
T5-small |  exclusive | 47.06 | 26.89 | 46.09 | 89.41 | 38.94 | 8.35
T5-base |  exclusive | 39.52 | 20.68 | 38.78 | 87.44 | 30.83 | 4.46
SSR-base |  exclusive | 27.58 | 14.92 | 25.30 | 86.11 | 35.36 | 3.79
Baseline |  exclusive | 36.25 | 16.87 | 35.74 | 87.03 | 25.41 | 2.12
BART-base |  independent | 68.79 | 53.72 | 68.34 | 93.99 | 62.10 | 16.54
T5-small |  independent | 64.82 | 48.26 | 64.26 | 93.16 | 58.07 | 14.47
T5-base |  independent | 65.74 | 49.22 | 65.22 | 93.27 | 58.94 | 14.54
SSR-base |  independent | 27.56 | 16.25 | 26.42 | 85.97 | 37.80 | 3.71
Baseline |  independent | 20.72 | 8.43 | 20.54 | 83.51 | 13.69 | 0.46
* R-1, R-2, and R-L are measured by F-measure values. |
Table 5: ROUGE, BertScore (BS), METEOR (ME), and BLEU scores between the generated descriptions and the gold descriptions on the test set in Phase I. All models use greedy decoding. Model |  Topic | Test set
---|---|---
R-1 | R-2 | R-L | BS | ME | BLEU
BART-base |  exclusive | 44.41 | 26.10 | 43.63 | 89.58 | 37.20 | 7.99
T5-small |  exclusive | 46.49 | 26.20 | 45.59 | 89.41 | 38.30 | 8.03
T5-base |  exclusive | 39.60 | 20.59 | 38.98 | 87.66 | 30.82 | 4.76
SSR-base |  exclusive | 27.43 | 14.76 | 25.23 | 86.09 | 35.12 | 3.67
Baseline |  exclusive | 36.25 | 16.91 | 35.73 | 87.09 | 25.44 | 2.21
BART-base |  independent | 69.59 | 54.59 | 69.12 | 94.06 | 62.82 | 17.47
T5-small |  independent | 65.57 | 48.97 | 64.93 | 93.26 | 59.13 | 14.79
T5-base |  independent | 66.39 | 49.64 | 65.84 | 93.33 | 59.56 | 14.68
SSR-base |  independent | 27.92 | 16.54 | 26.75 | 86.04 | 38.70 | 3.72
Baseline |  independent | 20.99 | 8.42 | 20.77 | 83.45 | 13.93 | 0.41
* R-1, R-2, and R-L are measured by F-measure values. |

Table 4 and Table 5 show the results of validation and tests of models with
two types of data split: topic-exclusive split and topic-independent split.
The baseline method considers the first item of each instance’s list as the
gold description. In topic-exclusive split, T5-small outperformed the baseline
and obtained the best performance. In topic-independent split, BART-base is
the winner. SSR-base models are the worst when they even have lower scores
than the baseline in the topic-exclusive split.

We take T5-small and BART-base as the best models in both data splits to
generate candidate descriptions for Phase II, candidate ranking. To improve
the diversity of the generated descriptions, we set num_beams, the number of
beams of beam search from 1 to 25. When num_beams equals 1, it means no beam
search used.

###  6.3 Phase II: Candidate Ranking

####  6.3.1 New datasets

The purpose of ranking models is to learn the order of candidates by pre-
defined metrics. Therefore, we don’t need so many samples for the training
process. We extract subsets from training, validation, and test sets of Phase
I for creating new sets for the training in Phase II. From a given paragraph,
five candidates are generated from the best models (T5-small and BART-base) in
Phase I.

The new sets have three components: paragraphs, gold descriptions, and lists
of candidates. There are two groups of sets, one for topic-exclusive split and
another for topic-independent split, which have the same set distribution: the
training set (6000 samples, 75%), the validation set (1000 samples, 12.5%),
and the test set (1000 samples, 12.5%).

####  6.3.2 Models

In Phase II, we use some models on new sets which are mentioned in Section
6.3.1. Note that the results of T5-small and BART-base models in here (Table 6
and Table 7) are different from those in Phase I (Table 4 and Table 5).

  * 1.

T5-small (topic-exclusive split): the best model from Phase I by topic-
exclusive split on new sets.

  * 2.

BART-base (topic-independent split): the best model from Phase I by topic-
independent split on new sets.

  * 3.

BERT + sim: the ranking model based on BERT by cosine similarity.

  * 4.

BERT + R-1-F: the ranking model based on BERT by ROUGE-1-F-measure.

  * 5.

BERT + sim + R-1-F: the ranking model based on BERT by fusing cosine
similarity and ROUGE-1-F-measure in the form of harmonic mean.

  * 6.

Gold des. vs. Para.: The comparison between gold descriptions and paragraphs.

####  6.3.3 Experiment Details and Results

Table 6: ROUGE, BertScore (BS), METEOR (ME), and BLEU scores between the
generated descriptions and the paragraphs on the validation set in Phase II.
T5-small-greedy and BART-base-greedy are the best models in Phase I, shown in
Table 4.

Phase I: Model | Phase II: Model + Metric | Topic | Validation set |
---|---|---|---|---
R-1 | R-2 | R-L | BS | ME | BLEU
T5-small-greedy | – | exclusive | 12.71 | 7.13 | 12.27 | 82.18 | 5.19 | 0.55
T5-small-beam | BERT + sim | exclusive | 24.17 | 19.06 | 23.64 | 84.84 | 13.35 | 5.21
T5-small-beam | BERT + R-1-F | exclusive | 25.53 | 20.29 | 24.96 | 85.35 | 14.17 | 5.47
T5-small-beam | BERT + sim + R-1-F | exclusive | 25.53 | 20.29 | 24.96 | 85.36 | 14.13 | 5.46
– |  Gold des. vs Para.+ | exclusive | 13.55 | 6.85 | 12.58 | 82.60 | 5.57 | 0.43
BART-base-greedy | – | independent | 13.67 | 6.60 | 12.74 | 81.90 | 5.13 | 0.29
BART-base-beam | BERT + sim | independent | 13.97 | 7.20 | 12.88 | 81.81 | 5.33 | 0.61
BART-base-beam | BERT + R-1-F | independent | 15.61 | 8.29 | 14.45 | 82.38 | 6.09 | 0.63
BART-base-beam | BERT + sim + R-1-F | independent | 15.61 | 8.29 | 14.45 | 82.39 | 6.11 | 0.64
– |  Gold des. vs Para.+ | independent | 13.03 | 5.61 | 12.03 | 81.69 | 4.78 | 0.19
+ Gold descriptions against paragraphs. |

Table 7: ROUGE, BertScore (BS), METEOR (ME), and BLEU scores between the
generated descriptions and the paragraphs on the test set in Phase II.
T5-small-greedy and BART-base-greedy are the best models in Phase I, shown in
Table 5.

Phase I: Model | Phase II: Model + Metric | Topic | Test set |
---|---|---|---|---
R-1 | R-2 | R-L | BS | ME | BLEU
T5-small-greedy | – | exclusive | 13.16 | 7.42 | 12.66 | 82.12 | 5.34 | 0.59
T5-small-beam | BERT + sim | exclusive | 23.93 | 18.50 | 23.30 | 84.71 | 12.71 | 4.56
T5-small-beam | BERT + R-1-F | exclusive | 25.36 | 19.61 | 24.64 | 85.26 | 13.43 | 4.77
T5-small-beam | BERT + sim + R-1-F | exclusive | 25.36 | 19.69 | 24.67 | 85.26 | 13.43 | 4.77
– |  Gold des. vs Para.+ | exclusive | 13.65 | 6.81 | 12.59 | 82.54 | 5.57 | 0.37
BART-base-greedy | – | independent | 14.06 | 6.74 | 13.04 | 82.09 | 5.38 | 0.16
BART-base-beam | BERT + sim | independent | 14.47 | 7.41 | 13.18 | 82.11 | 5.69 | 0.48
BART-base-beam | BERT + R-1-F | independent | 16.35 | 8.74 | 15.06 | 82.67 | 6.63 | 0.51
BART-base-beam | BERT + sim + R-1-F | independent | 16.35 | 8.72 | 15.01 | 82.67 | 6.63 | 0.51
– |  Gold des. vs Para.+ | independent | 14.28 | 6.67 | 13.09 | 82.08 | 5.70 | 0.28
+ Gold descriptions against paragraphs. |

Table 8: ROUGE, BertScore (BS), METEOR (ME), and BLEU scores between the
generated descriptions and the gold descriptions on the validation set in
Phase II. T5-small-greedy and BART-base-greedy are the best models in Phase I,
shown in Table 4.

Phase I: Model | Phase II: Model + Metric | Topic | Validation set |
---|---|---|---|---
R-1 | R-2 | R-L | BS | ME | BLEU
T5-small-greedy | – | exclusive | 40.68 | 21.73 | 39.53 | 89.53 | 33.84 | 6.69
T5-small-beam | BERT + sim | exclusive | 42.42 | 24.55 | 41.09 | 89.48 | 39.16 | 9.05
T5-small-beam | BERT + R-1-F | exclusive | 43.84 | 26.34 | 42.48 | 89.41 | 41.34 | 9.78
T5-small-beam | BERT + sim + R-1-F | exclusive | 43.63 | 26.02 | 42.27 | 89.39 | 41.07 | 9.75
BART-base-greedy | – | independent | 43.96 | 29.23 | 43.70 | 90.19 | 37.23 | 10.48
BART-base-beam | BERT + sim | independent | 60.83 | 45.15 | 60.52 | 93.48 | 54.66 | 16.16
BART-base-beam | BERT + R-1-F | independent | 66.56 | 51.44 | 66.14 | 92.66 | 61.54 | 17.74
BART-base-beam | BERT + sim + R-1-F | independent | 66.44 | 51.25 | 66.02 | 92.65 | 61.35 | 17.54

Table 9: ROUGE, BertScore (BS), METEOR (ME), and BLEU scores between the
generated descriptions and the gold descriptions on the test set in Phase II.
T5-small-greedy and BART-base-greedy are the best models in Phase I, shown in
Table 5.

Phase I: Model | Phase II: Model + Metric | Topic | Test set |
---|---|---|---|---
R-1 | R-2 | R-L | BS | ME | BLEU
T5-small-greedy | – | exclusive | 38.26 | 19.94 | 37.27 | 88.97 | 31.77 | 4.70
T5-small-beam | BERT + sim | exclusive | 42.65 | 25.20 | 41.43 | 89.54 | 39.98 | 8.81
T5-small-beam | BERT + R-1-F | exclusive | 44.48 | 27.11 | 43.29 | 89.77 | 42.59 | 10.07
T5-small-beam | BERT + sim + R-1-F | exclusive | 44.37 | 26.88 | 43.18 | 89.77 | 42.51 | 9.92
BART-base-greedy | – | independent | 55.44 | 40.14 | 55.03 | 92.33 | 47.83 | 10.61
BART-base-beam | BERT + sim | independent | 59.72 | 45.41 | 59.16 | 93.28 | 53.78 | 16.57
BART-base-beam | BERT + R-1-F | independent | 67.79 | 54.41 | 67.35 | 94.46 | 62.96 | 19.64
BART-base-beam | BERT + sim + R-1-F | independent | 67.73 | 54.37 | 67.28 | 94.43 | 62.89 | 19.73

The best ranking models are saved within 3 epochs by transfer learning from
the pre-trained model bert-base-cased. Table 6 and Table 7 show several
metrics of the ranking models (Phase II) compared to the best generative
models (Phase I) between the generated descriptions and the paragraphs on the
validation and test sets. We also compare the gold descriptions versus their
paragraphs (Gold des. vs. Para.). T5-small-greedy and BART-base-greedy as the
best models in Phase I are not better than Gold des. vs. Para. when they learn
to generate new descriptions. However, their performance is only less than
$\approx$ 1 ROUGE in topic-exclusive split and even surpasses Gold des. vs.
Para. at least $\approx$ 0.6 ROUGE in the validation set of topic-independent
split. These shreds of evidence indicate that the generative models were well-
trained.

Table 8 and Table 9 show several metrics of the ranking models (Phase II)
compared to the best generative models (Phase I) between the generated
descriptions and the gold descriptions on the validation and test sets. All
the ranking models (Phase II) help to boost the quality of the generated
descriptions with the better metric values. In this experiment, the BERT +
R-1-F model obtains the best performance while the BERT + sim + R-1-F model is
the very close follower.

In general, when applying contrastive learning, the ranking models outperform
significantly the generative models in both topic-exclusive split and topic-
independent split. Between the generated descriptions and the paragraphs, the
ranking models obtain from $\approx$ 11 to 13 ROUGE better than the generative
models in topic-exclusive split and from $\approx$ 0.5 to 2 ROUGE in topic-
independent split. Between the generated descriptions and the gold
description, the ranking models gain from $\approx$ 1 to 6 ROUGE better than
the generative models in topic-exclusive split and from $\approx$ 4 to 22
ROUGE in topic-independent split. The best ranking models come from R-1 and
the fuse of R-1 and cosine similarity for calculating the rank of the
candidates.

###  6.4 Sentiment Correlations

We use a pre-trained model from
Huggingface666https://huggingface.co/cardiffnlp/twitter-xlm-roberta-base-
sentiment to extract the sentiment polarities — negative, neutral, and
positive — of the generated descriptions, the paragraphs, and the gold
descriptions. The goal here is to measure the sentiment correlations between
the generated descriptions and the gold descriptions and between the generated
descriptions and the paragraphs. Table 10 presents the average values of
sentiment polarities over the test sets of the two best methods of Phase II.
For each text, we have the distribution formula
$P(negative)+P(neutral)+P(positive)=1$.

In WikiDes, most texts are non-opinionated i.e., neutral because they were
extracted from Wikimedia projects, which portray neutrality as one of five
fundamental principles777https://en.wikipedia.org/wiki/Wikipedia:Five_pillars.
This principle represents the stance of Wikimedia in avoiding any bias of
content to which millions of users with different viewpoints have contributed.
Furthermore, it is also easier for administrators to minimize the harsh
arguments on a certain article when forcing the stakeholders toward a neutral
point. In this sense, a description should keep the sentiment consistent with
the Wikipedia paragraph beside the salient information to reflect the original
sentiment, especially the neutrality of that paragraph. This creates a
sentiment uniformity of content across Wikimedia projects, including Wikipedia
and Wikidata.

Table 10: The average values of sentiment polarities in texts on the test sets of two methods, BERT + sim + R-1-F  and BERT + R-1-F. | Polarity | BERT + sim + R-1-F | BERT + R-1-F
---|---|---|---
| (topic-exclusive) | (topic-independent)
Paragraphs | Negative | 19.53 | 17.30
Neutral | 63.25 | 64.45
Positive | 17.20 | 18.23
Best descriptions | Negative | 17.91 | 20.13
Neutral | 70.05 | 67.12
Positive | 12.03 | 12.73
Gold descriptions | Negative | 20.45 | 21.42
Neutral | 67.28 | 65.09
Positive | 12.26 | 13.47
Figure 6: The sentiment cumulative distribution by polarities on the test sets produced by two methods, BERT + sim + R-1-F (first row) and BERT + R-1-F (second row). There are 3 sets, the paragraphs (source), the generated descriptions (candidate), and the gold descriptions (target).  Table 11: The critical values ($c_{\alpha}$) and the rejection thresholds ($c_{\alpha}\sqrt{\frac{N+M}{N\times M}}$) by some $\alpha$ levels of significance. Our test sets have the same size of 1000, therefore $N=M=1000$. $\alpha$ | 0.20 | 0.15 | 0.10 | 0.05 | 0.01
---|---|---|---|---|---
$c_{\alpha}$ | 1.0729 | 1.1380 | 1.2238 | 1.3581 | 1.6276
$c_{\alpha}\sqrt{\frac{N+M}{N.M}}$ | 0.0479 | 0.0508 | 0.0547 | 0.0607 | 0.0727
Figure 7: The Kolmogorov-Smirnov test on the test sets produced by the method
BERT + sim + R-1-F with topic-exclusive split. The test statistic value is the
distance $D$. There are 3 sets, the paragraphs (source), the generated
descriptions (candidate), and the gold descriptions (target).  Figure 8: The
Kolmogorov-Smirnov test on the test sets produced by the method BERT + R-1-F
with topic-independent split. The test statistic value is the distance $D$.
There are 3 sets, the paragraphs (source), the generated descriptions
(candidate), and the gold descriptions (target).

The cumulative distributions of semantic polarities by texts are demonstrated
in Figure 6 to provide a general vision of how similar these distributions
are. There is no doubt that the distributions of the generated descriptions
and the gold descriptions are somewhat analogous in all plots when generation
models must learn gold descriptions to create new descriptions. However, the
distributions of paragraphs are still independent in some parts, especially in
neutral and positive polarities, compared to those of descriptions. This hints
that even gold descriptions can not capture all sentiment information in
paragraphs.

Although cumulative distributions bring visually a helpful analysis, it is
better to obtain a qualitative measure to determine the uniformity of
distributions. We perform the Kolmogorov-Smirnov (K-S) test [75, 88] to
compare the distributions of paragraphs vs. generated descriptions and
paragraphs vs. gold descriptions. The test output is a statistic value that
allows concluding whether two input sets have the same distribution or not.
Table 11 lists several critical values by $\alpha$ levels of significance,
which are used to validate the condition of Equation 17 to reject or accept
the null hypothesis.

Figure 7 and Figure 8 show the K-S statistic values on various sets by
sentiment polarities. We reject the null hypothesis in all levels of
significance between the generated descriptions and the paragraphs. In other
say, the distributions of the generated descriptions and the paragraphs are
different. Here, the K-S highest distance of positive polarity is up to 0.3850
which indicates that there is a big difference in sentiment distribution
between the generated descriptions and the paragraphs. In this case, the
generated descriptions can not be considered good descriptions in capturing
sentiment information.

In another aspect, the null hypothesis is accepted in some levels of
significance between the generated descriptions and the gold descriptions. At
the one percent level of significance (0.01), the positive polarity of BERT +
sim + R-1-F (Figure 7) and the negative, neutral, and positive polarities of
BERT + R-1-F (Figure 8) accept the null hypothesis. At the five percent level
of significance (0.05), the null hypothesis is true only with the negative and
positive polarities of BERT + R-1-F (Figure 8). Overall, we conclude that
training models on topic-independent split can capture sentiment polarities
better than on topic-exclusive split. Furthermore, the sentiment distributions
between the generated description and the gold descriptions are more identical
than between the generated descriptions and the paragraphs.

###  6.5 Correlation with Human Evaluation

In this section, we randomly take 100 samples – 50 samples from the test set
from topic-exclusive split and 50 samples from the test set from topic-
independent split – for human evaluation. Table 12 shows some samples in the
human evaluation process, where each sample contains a paragraph, a gold
description, and a machine-generated candidate. We label the machine-generated
description and the gold description as "Summary 1" and "Summary 2" to avoid
bias in the evaluation process. We also include some empty summaries to check
the seriousness of evaluators.

Table 12: Some typical samples are used for human evaluation. "Machine-
generated description" and "Gold description" are the best machine-generated
description and the gold description. Descriptions with $\checkmark$ were
selected by all evaluators.

# Samples
---
Paragraph 1: The Berliner Verkehrsbetriebe (German: Berlin Transport Company)
is the main public transport company of Berlin, the capital city of Germany.
It manages the city’s U-Bahn underground railway, tram, bus, replacement
services (Ersatzverkehr, EV), and ferry networks, but not the S-Bahn urban
rail system. The generally used abbreviation, BVG, has been retained from the
company’s original name, Berliner Verkehrs-Aktiengesellschaft (Berlin
Transportation Stock Company). Subsequently, the company was renamed Berliner
Verkehrs-Betriebe. During the division of Berlin, the BVG was split between
BVG…
Machine-generated description: public transport company of Berlin, Germany
$\checkmark$
Gold description: public transport agency in Berlin
Paragraph 2: The Harbour Grace CeeBee Stars, (also commonly known as the
Harbour Grace Ocean Enterprises CeeBee Stars due to a sponsorship deal that
began October 23, 2015) are a senior ice hockey team based in Harbour Grace,
Newfoundland and Labrador and part of the Avalon East Senior Hockey League.
The CeeBees are eight-time winners of the Herder Memorial Trophy as provincial
champions.
Machine-generated description: ice hockey team
Gold description: ice hockey team in Harbour Grace, Newfoundland and Labrador
$\checkmark$
Paragraph 3: The Albanian Urban Lyric Song is a musical tradition of Albania
that started in the 18th century and culminated in the 1930s.
Machine-generated description: Albanian musical tradition $\checkmark$
Gold description: music genre
Paragraph 4: Fuidio is a hamlet and minor local entity located in the
municipality of Condado de Treviño, in Burgos province, Castile and León,
Spain. As of 2020, it has a population of 26.
Machine-generated description: human settlement in Burgos Province, Castile
and León, Spain $\checkmark$
Gold description: human settlement in Spain
Paragraph 5: On 4 September 2014, 82-year-old Palmira Silva was beheaded in
her back garden in Edmonton, London, by 25-year-old Nicholas Salvador, who was
on a rampage… Psychiatrists found evidence that Salvador had paranoid
schizophrenia. On 23 June 2015, he was found not guilty of murder on the basis
of insanity and was detained indefinitely in a psychiatric hospital.
Machine-generated description: Psychiatrists found evidence that Nicolas
Salvador had paranoid schizophrenia, not guilty of murder on the basis of
insanity, and detained indefinitely in a psychiatric hospital
Gold description: 2014 beheading in Edmonton, London $\checkmark$
Paragraph 6: Planar Handbook is an optional supplemental source book for the
3.5 edition of the Dungeons & Dragons fantasy roleplaying game.
Machine-generated description: supplemental book for the 3.5 edition of the
Dungeons & Dragons fantasy roleplaying game $\checkmark$
Gold description: tabletop role-playing game supplement
Paragraph 7: U.S. Games Systems, Inc. (USGS) is a publisher of playing cards,
tarot cards, and games located in Stamford, Connecticut. Founded in 1968 by
Stuart R. Kaplan, it has published hundreds of different card sets, and about
20 new titles are released annually. The company’s product line includes
children’s card games, museum products, educational cards, motivational cards,
tarot cards, and other fortune-telling card decks…
Machine-generated description: American publisher of playing cards, tarot
cards, and games $\checkmark$
Gold description: card game publishing company

Table 13: Human agreement by evaluation coefficients over 4 criteria. The
machine-generated descriptions were obtained from Phase I.

| Phase I generated. vs gold. | adequacy | relevance | correctness | concise
---|---|---|---|---|---
$\alpha_{n}$ | 0.7609 | -0.0461 | 0.0793 | 0.2307 | -0.0863
$\alpha_{i}$ | 0.7609 | -0.0398 | 0.1753 | 0.3175 | -0.0070
$\kappa_{c}$ | 0.7533 | 0.1123 | 0.1204 | 0.2341 | 0.0627
$\kappa_{f}$ | 0.7615 | 0.0715 | 0.1137 | 0.2303 | 0.0326
S | 0.8266 | 0.0666 | 0.3911 | 0.6088 | 0.0133
$\pi$ | 0.7601 | -0.0496 | 0.0762 | 0.2281 | -0.0899
average score | - | 3.91/5 | 4.57/5 | 4.7/5 | 3.93/5
distribution | 23.66% vs 76.33% | - | - | - | -
$\alpha_{n}$: alpha nominal, $\alpha_{i}$: alpha interval

Table 14: Human agreement by evaluation coefficients over 4 criteria. The
machine-generated descriptions were obtained from Phase II.

| Phase II selected. vs gold. | adequacy | relevance | correctness | concise
---|---|---|---|---|---
$\alpha_{n}$ | 0.3833 | -0.0152 | -0.1742 | -0.0880 | -0.0979
$\alpha_{i}$ | 0.3833 | 0.0423 | 0.1388 | -0.1191 | -0.2012
$\kappa_{c}$ | 0.3814 | 0.0370 | -0.0198 | -0.0073 | -0.0063
$\kappa_{f}$ | 0.3816 | 0.0299 | -0.0190 | -0.0088 | -0.0051
S | 0.3866 | 0.0541 | 0.1000 | 0.3458 | 0.1833
$\pi$ | 0.3812 | -0.0186 | -0.1781 | -0.0916 | -0.1015
average score | - | 3.62/5 | 4.31/5 | 4.52/5 | 4.23/5
distribution | 45.33% vs 55.66% | - | - | - | -
$\alpha_{n}$: alpha nominal, $\alpha_{i}$: alpha interval

We selected three postgraduate students as evaluators. First, we ask them to
choose whether the gold description or the best candidate is the accurate
description of a given paragraph. Then, for the description they choose, rate
several criteria by scores from 1 to 5: bad and can not use = 1, not
recommended for use = 2, fair but need to consider = 3, good = 4, and perfect
= 5.

van der Lee et al. [97] listed at least 17 different criteria gathered from
many scientific papers for evaluating the generated texts. Though the authors
did not design these criteria in mind for summarization tasks, this work is
helpful enough for us to refer to. Following that, we consider using 4
evaluation criteria that are appropriate for short generated descriptions:

  * 1.

adequacy or informativeness [98]: This criterion is to check whether a
generated description captures enough salient information from the paragraph
or not.

  * 2.

relevance or related: Does a generated description have an appropriate level
such as a topic or theme to the paragraph?

  * 3.

correctness: This criterion validates the correctness of grammar and facts
from a generated description compared to the paragraph.

  * 4.

concise or brief: A generated description must be concise enough but not too
short, and it still holds the important information from the paragraph.

After the annotation process, we measure the agreement of evaluators, using
Krippendorff’s alpha coefficient ($\alpha$) [99], Cohen’s kappa ($\kappa_{c}$)
[100], Fleiss’ kappa or multi-kappa ($\kappa_{f}$) [101], Bennett, Alpert and
Goldstein’s S (S) [102], and Scott’s Pi ($\pi$) [103]. They are common
statistics used to measure inter-rater reliability between a number of
evaluators over categories. Here are the scales of each coefficient:

  * 1.

$\alpha$ and $\kappa_{f}$: The scale range is from $-1$ to $1$, in which $1$
is a perfect agreement, $0$ shows no agreement beyond chance, and negative
values indicate disagreement [104].

  * 2.

$\kappa_{c}$: The original author suggested values $\leq$ 0 for no agreement,
0.01–0.20 for none to slight, 0.21–0.40 for fair, 0.41–0.60 for moderate,
0.61–0.80 for substantial, and 0.81–1.00 for almost perfect agreement [105].
$\kappa_{c}$ is formally identical to $\pi$ but has a different calculation of
the expected agreement [106].

  * 3.

S: Scale is a range from $-1$ to $1$. Between the two raters, $1$ indicates
perfect agreement, and $-1/(n-1)$ if the proportion of observed agreement
$P_{o}$ equals $0$. The minimum value $-1$ with the number of categories $n$
equals $2$, and it will go toward to $0$ when $n$ increases [107].

  * 4.

$\pi$: The value is calculated by formula $\pi=P_{o}-P_{e}/(1-P_{e})$. $P_{o}$
is the observed proportion of agreement and $P_{e}$ is the expected proportion
by chance. Scale is a range from a minimum value $-P_{e}/(1-P_{e})$ (when
$P_{o}$ equals $0$) to a maximum value $1$ (when $P_{o}$ equals $1$) [106].

To evaluate the descriptions produced by Phase I and Phase II, we consider the
following two scenarios:

  * 1.

Phase I generated vs. gold (Table 13): The generated descriptions from Phase I
versus the gold descriptions.

  * 2.

Phase II selected vs. gold (Table 14): The selected descriptions from Phase II
versus the gold descriptions.

Table 13 and Table 14 presents the results of coefficients over criteria with
their average values and data distribution on Phase I generated vs. gold and
Phase II selected vs. gold correspondingly. In Table 13, we have a slight
disagreement on criteria adequacy and concise while we have from a slight to a
fair agreement on criteria relevance and correctness. In Table 14, we have a
slight disagreement on the criteria adequacy, relevance, correctness, and
concise.

Meanwhile, there is a high consensus among evaluators in choosing Phase I
generated vs. gold and Phase II selected vs. gold. Especially, there are
45.33% descriptions chosen for Phase II compared to 23.66% descriptions chosen
for Phase I against the gold descriptions. Therefore, we can conclude that the
quality of descriptions in Phase II is better than those in Phase I in human
evaluation. Furthermore, it can infer the descriptions taken from Wikidata are
not likely qualified as the gold descriptions under the eyes of evaluators.

When applying contrastive learning, the generated descriptions are closer to
the paragraphs because they have a higher semantic similarity. By this, they
tend to be longer in length and capture more important information, which
impacts the evaluator’s decision. In addition, the average scores over the
5-scale of all criteria are generally high, at least 4. Having the lowest
value, criterion adequacy indicates the missing importance in descriptions. It
may be from the difference in expectations for Wikidata descriptions compared
to ordinary summaries in abstractive summarization. Wikidata descriptions are
generally short and reflect the most prominent information of a Wikipedia
paragraph, while an ordinary summary needs to capture all important
information, even in the short length.

###  6.6 Error Analysis

The combination of beam search and contrastive learning is a tasty recipe for
text summarization, however, there remain two problems in producing quality
descriptions. First, we observe that some descriptions contain repetitive
texts from setting various beam search configurations in the inference time in
Phase I. This makes a false description look closer to its paragraph when
measuring the semantic similarity between it and its paragraph in Phase II.
Second, it may occur in a case where a few descriptions contain incorrect
factual information. The sequence-two-sequence models learn to generate the
next tokens with the highest probabilities based on the token frequencies in
the dataset. Therefore, a generated description is a sequence of the possible
occurrences of tokens instead one with accurate facts.

Table 15: A few false descriptions in the output.

# Errors
---
Paragraph 1: The Order of Merit of the Federal Republic of Germany (German:
Verdienstorden der Bundesrepublik Deutschland, or Bundesverdienstorden, BVO)
is the only federal decoration of Germany. It is awarded for special
achievements in political, economic, cultural, intellectual or honorary
fields…
Description: Order of Merit of the Federal Republic of Germany, awarded for
special achievements in political, economic, cultural, intellectual or
honorary fields; awarded for special achievements in political, economic,
cultural, intellectual or honorary fields… – _The description contains
repetitive texts._
Paragraph 2: Malaysian Standard Time (MST; Malay: Waktu Piawai Malaysia, WPM)
or Malaysian Time (MYT) is the standard time used in Malaysia. It is 8 hours
ahead of Coordinated Universal Time (UTC). The local mean time in Kuala Lumpur
was originally GMT+06:46:46…
Description: Malaysia clock clock clock clock clock clock… – _The description
contains repetitive texts._
Paragraph 3: Transition metal dinitrogen complexes are coordination compounds
that contain transition metals as ion centers the dinitrogen molecules (N2) as
ligands.
Description: complexes of protein-coding protein-coding protein-coding
protein-coding protein-coding protein-coding protein-coding protein-coding
protein-coding… – _The description contains repetitive texts._
Paragraph 4: A shape-up, also called a line-up or an edge-up, is a hairstyle
that involves cutting along the natural hairline to straighten it. Shape-ups
or edge-ups are the fundamental outline for haircuts today. Edge-ups are
typically found among men and short-haired women…
Description: a hairstyle that involves cutting along the natural hairline to
straighten it. a line-up or an edge-up is a hairstyle that involves cutting
along the natural hairline to straighten it. – _Repeat the first sentence._
Paragraph 5: The National Office of Statistics is the Algerian ministry
charged with the collection and publication of statistics related to the
economy, population, and society of Algeria at national and local levels. Its
head office is in Algiers.
Description: statistical service – _The subject is a ministry, not a
statistical service._
Paragraph 6: The Bergen Region is a statistical metropolitan region in the
county of Hordaland in Norway. It is centered on the city of Bergen.
Description: geographical feature – _The subject is a statistical metropolitan
region, not a geographical feature. The description does not contain any token
or lemma from the paragraph._
Paragraph 7: Gargoyle is a free OpenWrt-based Linux distribution for a range
of wireless routers based on Broadcom, Atheros, MediaTek and others chipsets,
Asus Routers, Netgear, Linksys and TP-Link routers. Among notable features is
the ability to limit and monitor bandwidth and set bandwidth caps per specific
IP address.
Description: embedded operating system – _The subject is a free OpenWrt-based
Linux distribution, not a embedded operating system. The description does not
contain any token or lemma from the paragraph._

Table 15 shows a few false descriptions and their errors, which are
highlighted in orange with corresponding explanations. Although our research
did not design any mechanism to control the repetitive texts and factual
information, these problems were addressed by some approaches such as Pointer-
Generator Networks [39], Global Encoding [108], reinforcement learning [109],
rule-based/heuristic transformations [110, 111], and graph attention [112].

##  7 Conclusion

In this paper, we introduced WikiDes, a novel summarization dataset with over
80k samples on 6987 topics created by collecting data from Wikipedia and
Wikidata. The dataset aims to produce short descriptions from the given
paragraphs. In this work, the two-phase summarization — description generation
and candidate ranking — proved its advantage in boosting the quality of the
produced descriptions. The human evaluation confirmed this statement with
45.33% of descriptions in Phase II chosen compared to 23.66% of those in Phase
I when comparing with the gold descriptions. Furthermore, the difference in
sentiment distribution between the descriptions and paragraphs suggests a
potential work on integrating sentiment analysis into text summarization.

We performed some analyses on WikiDes to understand the data distribution,
correlations between the paragraphs and the descriptions, and comparison with
other existing datasets. In description generation, small-scale pre-trained
models were applied to introduce the description diversity by beam search
decoding. In candidate ranking, we trained the BERT models to rank the
candidate descriptions by various metrics in a contrastive learning setup.
Later, we checked the uniformity of sentiment polarity on the descriptions
versus the paragraphs to trigger the need of integrating sentiment analysis
into text summarization. In addition, we measured the level of evaluators’
agreement by several evaluation criteria for the generated descriptions with
some popular correlation coefficients.

Though our work has a limited scope to generate short descriptions from
paragraphs, we think it could be extended to other short text generation
tasks, e.g. title generation. In the future, we will expand WikiDes to a
multilingual scale and apply other deep-learning methods to eliminate
repetitive texts and capture more factual information. Last but not least, the
joint modeling of sentiment analysis and text summarization is an interesting
direction to improve the quality of summaries. In particular, one could use
multitask learning to train the model on two tasks – text summarization and
sentiment analysis – at the same time. Another potential approach could be to
disentangle the sentiment information from the text and propose a distance or
similarity measurement loss function to measure sentiment consistency.

## Data Availability Statement

The curated dataset is publicly available at:

  * 1.

https://github.com/declare-lab/WikiDes

## Compliance with Ethical Standards

  * 1.

This article does not contain any studies with human participants or animals
performed by any of the authors.

  * 2.

All authors certify that they have no affiliations with or involvement in any
organization or entity with any financial interest or non-financial interest
in the subject matter or materials discussed in this manuscript.

## Acknowledgements

This research is supported by the Ministry of Education, Singapore, under its
AcRF Tier-2 grant (Project no. T2MOE2008, and Grantor reference no.
MOE-T2EP20220-0017). Any opinions, findings and conclusions or recommendations
expressed in this material are those of the author(s) and do not reflect the
views of the Ministry of Education, Singapore.

## References

  * Boudin et al. [2011] F. Boudin, S. Huet, J.-M. Torres-Moreno,  A graph-based approach to cross-language multi-document summarization,  Polibits 43 (2011) 113–118.
  * Zulkhazhav et al. [2019] A. Zulkhazhav, Z. Kozhirbayev, Z. Yessenbayev, A. Sharipbay,  Kazakh text summarization using fuzzy logic,  Computacion y Sistemas 23 (2019). doi:10.13053/cys-23-3-3239.
  * Sakota et al. [2022] M. Sakota, M. Peyrard, R. West,  Descartes: Generating Short Descriptions of Wikipedia Articles,  arXiv preprint arXiv:2205.10012 (2022).
  * Gholamrezazadeh et al. [2009] S. Gholamrezazadeh, M. A. Salehi, B. Gholamzadeh,  A comprehensive survey on text summarization systems,  in: 2009 2nd International Conference on Computer Science and its Applications, IEEE, 2009, pp. 1–6.
  * El-Kassas et al. [2021] W. S. El-Kassas, C. R. Salama, A. A. Rafea, H. K. Mohamed,  Automatic text summarization: A comprehensive survey,  Expert Systems with Applications 165 (2021) 113679.
  * Narayan et al. [2018] S. Narayan, S. B. Cohen, M. Lapata,  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization,  arXiv preprint arXiv:1808.08745 (2018).
  * Liu and Liu [2021] Y. Liu, P. Liu,  SimCLS: A simple framework for contrastive learning of abstractive summarization,  arXiv preprint arXiv:2106.01890 (2021).
  * Zhong et al. [2020] M. Zhong, P. Liu, Y. Chen, D. Wang, X. Qiu, X. Huang,  Extractive summarization as text matching,  arXiv preprint arXiv:2004.08795 (2020).
  * Anjalie et al. [2020] F. Anjalie, S. Rothe, S. Baumgartner, C. Yu, A. Ittycheriah,  A Generative Approach to Titling and Clustering Wikipedia Sections,  arXiv preprint arXiv:2005.11216 (2020).
  * Frefel [2020] D. Frefel,  Summarization corpora of Wikipedia articles,  In Proceedings of the 12th Language Resources and Evaluation Conference (2020) 6651–6655.
  * Haichao et al. [2022] Z. Haichao, L. Dong, F. Wei, B. Qin, T. Liu,  Transforming wikipedia into augmented data for query-focused summarization,  IEEE/ACM Transactions on Audio, Speech, and Language Processing (2022).
  * Zopf et al. [2016] M. Zopf, M. Peyrard, J. Eckle-Kohler,  The next step for multi-document summarization: A heterogeneous multi-genre corpus built with a novel construction approach,  in: Proceedings of the 26th International Conference on Computational Linguistics (COLING 2016), Association for Computational Linguistics, Osaka, Japan, 2016, pp. 1535–1545. URL: https://www.ukp.tu-darmstadt.de/fileadmin/user_upload/Group_AIPHES/publications/2016/2016_COLING_hMDS_cameraReady.pdf.
  * Zopf [2018] M. Zopf,  Auto-hMDS: Automatic construction of a large heterogeneous multilingual multi-document summarization corpus,  in: Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), 2018\.
  * Diego and Faltings [2020] A. Diego, B. Faltings,  GameWikiSum: a novel large multi-document summarization dataset,  arXiv preprint arXiv:2002.06851 (2020).
  * Demian Gholipour et al. [2020] G. Demian Gholipour, C. Hokamp, N. The Pham, J. Glover, G. Ifrim,  A Large-Scale Multi-Document Summarization Dataset from the Wikipedia Current Events Portal,  arXiv preprint arXiv:2005.10070 (2020).
  * Mehwish and Strube [2021] F. Mehwish, M. Strube,  A novel Wikipedia based dataset for monolingual and cross-lingual summarization.,  In Proceedings of the Third Workshop on New Frontiers in Summarization. Online and in Dominican Republic. Association for Computational Linguistics (2021) 39–50.
  * Laura and Lapata [2021] P.-B. Laura, M. Lapata,  Models and datasets for cross-lingual summarisation,  Proceedings of The 2021 Conference on Empirical Methods in Natural Language Processing, Punta Cana, Dominican Republic (2021).
  * Pavel and Malykh [2022] T. Pavel, V. Malykh,  WikiMulti: a Corpus for Cross-Lingual Summarization,  arXiv preprint arXiv:2204.11104 (2022).
  * Paul et al. [2007] O. Paul, H. Dang, D. Harman,  DUC in context,  Information Processing & Management 43 (2007) 1506–1520.
  * Pierre-Etienne and Lapalme [2012] G. Pierre-Etienne, G. Lapalme,  Fully abstractive approach to guided summarization,  In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics 2 (2012) 354–358.
  * Sandhaus [2008] E. Sandhaus,  The new york times annotated corpus,  Linguistic Data Consortium, Philadelphia 6 (2008) e26752.
  * Hermann et al. [2015] K. M. Hermann, T. Kocisky, E. Grefenstette, L. Espeholt, W. Kay, M. Suleyman, P. Blunsom,  Teaching machines to read and comprehend,  Advances in neural information processing systems 28 (2015).
  * Grusky et al. [2018] M. Grusky, M. Naaman, Y. Artzi,  Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies,  arXiv preprint arXiv:1804.11283 (2018).
  * Fabbri et al. [2019] A. R. Fabbri, I. Li, T. She, S. Li, D. R. Radev,  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model,  arXiv preprint arXiv:1906.01749 (2019).
  * Koupaee and Wang [2018] M. Koupaee, W. Y. Wang,  Wikihow: A large scale text summarization dataset,  arXiv preprint arXiv:1810.09305 (2018).
  * Cohen et al. [2021] N. Cohen, O. Kalinsky, Y. Ziser, A. Moschitti,  WikiSum: Coherent Summarization Dataset for Efficient Human-Evaluation,  in: Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 2021, pp. 212–219.
  * Cohan et al. [2018] A. Cohan, F. Dernoncourt, D. S. Kim, T. Bui, S. Kim, W. Chang, N. Goharian,  A discourse-aware attention model for abstractive summarization of long documents,  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA 2 (2018) 615–621.
  * Sharma et al. [2019] E. Sharma, C. Li, L. Wang,  BIGPATENT: A large-scale dataset for abstractive and coherent summarization,  In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy 1 (2019) 2204– 2213.
  * Lu et al. [2020] Y. Lu, Y. Dong, L. Charlin,  Multi-XScience: A large-scale dataset for extreme multi-document summarization of scientific articles,  arXiv preprint arXiv:2010.14235 (2020).
  * Steinberger et al. [2004] J. Steinberger, K. Jezek, et al.,  Using latent semantic analysis in text summarization and summary evaluation,  Proc. ISIM 4 (2004) 8.
  * Harabagiu and Lacatusu [2005] S. Harabagiu, F. Lacatusu,  Topic themes for multi-document summarization,  in: Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval, 2005, pp. 202–209.
  * Erkan and Radev [2004] G. Erkan, D. R. Radev,  Lexrank: Graph-based lexical centrality as salience in text summarization,  Journal of artificial intelligence research 22 (2004) 457–479.
  * Mihalcea and Tarau [2004] R. Mihalcea, P. Tarau,  Textrank: Bringing order into text,  in: Proceedings of the 2004 conference on empirical methods in natural language processing, 2004, pp. 404–411.
  * Ganesan et al. [2010] K. Ganesan, C. Zhai, J. Han,  Opinosis: A graph based approach to abstractive summarization of highly redundant opinions,  in: Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), Coling 2010 Organizing Committee, Beijing, China, 2010, pp. 340–348. URL: https://aclanthology.org/C10-1039.
  * Baralis et al. [2013] E. Baralis, L. Cagliero, N. Mahoto, A. Fiori,  GraphSum: Discovering correlations among multiple terms for graph-based summarization,  Information Sciences 249 (2013) 96–109.
  * Gambhir and Gupta [2017] M. Gambhir, V. Gupta,  Recent automatic text summarization techniques: a survey,  Artificial Intelligence Review 47 (2017) 1–66.
  * Alexander M. et al. [2015] R. Alexander M., S. Chopra, J. Weston,  A neural attention model for abstractive sentence summarization,  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal (2015) 379–389.
  * Ramesh et al. [2016] N. Ramesh, B. Zhou, C. Gulcehre, B. Xiang,  Abstractive text summarization using sequence-to-sequence rnns and beyond,  In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, Berlin, Germany. (2016) 280–290.
  * See et al. [2017] A. See, P. J. Liu, C. D. Manning,  Get to the point: Summarization with pointer generator networks,  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada (2017) 1073–1083.
  * Ramesh et al. [2017] N. Ramesh, F. Zhai, B. Zhou,  Summarunner: A recurrent neural network based sequence model for extractive summarization of documents,  In Thirty-first AAAI conference on artificial intelligence (2017).
  * Wolf et al. [2020] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al.,  Transformers: State-of-the-art natural language processing,  in: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, 2020, pp. 38–45.
  * Liu [2019] Y. Liu,  Fine-tune BERT for extractive summarization,  arXiv preprint arXiv:1903.10318 (2019).
  * Tinghuai et al. [2022] M. Tinghuai, Q. Pan, H. Rong, Y. Qian, Y. Tian, N. Al-Nabhan,  T-bertsum: Topic-aware text summarization based on bert,  IEEE Transactions on Computational Social Systems 9 (2022) 879–890.
  * Lewis et al. [2019] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer,  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension,  arXiv preprint arXiv:1910.13461 (2019).
  * Raffel et al. [2020] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, P. J. Liu, et al.,  Exploring the limits of transfer learning with a unified text-to-text transformer,  J. Mach. Learn. Res. 21 (2020) 1–67.
  * Jingqing et al. [2020] Z. Jingqing, Y. Zhao, M. Saleh, P. Liu,  Pegasus: Pre-training with extracted gap-sentences for abstractive summarization,  In International Conference on Machine Learning (2020) 11328–11339.
  * Laskar et al. [2020] M. T. R. Laskar, E. Hoque, J. Huang,  Query focused abstractive summarization via incorporating query relevance and transfer learning with transformer models,  in: Canadian conference on artificial intelligence, Springer, 2020, pp. 342–348.
  * Alomari et al. [2022] A. Alomari, N. Idris, A. Q. M. Sabri, I. Alsmadi,  Deep reinforcement and transfer learning for abstractive text summarization: A review,  Computer Speech & Language 71 (2022) 101276.
  * Javanmardi et al. [2009] S. Javanmardi, Y. Ganjisaffar, C. Lopes, P. Baldi,  User contribution and trust in Wikipedia,  in: 2009 5th International Conference on Collaborative Computing: Networking, Applications and Worksharing, IEEE, 2009, pp. 1–6.
  * Shusheng et al. [2022] X. Shusheng, X. Zhang, Y. Wu, F. Wei,  Sequence level contrastive learning for text summarization,  In Proceedings of the AAAI Conference on Artificial Intelligence 36 (2022) 11556–11565.
  * Vijayakumar et al. [2016] A. K. Vijayakumar, M. Cogswell, R. R. Selvaraju, Q. Sun, S. Lee, D. Crandall, D. Batra,  Diverse beam search: Decoding diverse solutions from neural sequence models,  arXiv preprint arXiv:1610.02424 (2016).
  * Holtzman et al. [2019] A. Holtzman, J. Buys, L. Du, M. Forbes, Y. Choi,  The curious case of neural text degeneration,  arXiv preprint arXiv:1904.09751 (2019).
  * Liu et al. [2022] Y. Liu, P. Liu, D. Radev, G. Neubig,  Brio: Bringing order to abstractive summarization,  in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 2890–2903.
  * Jiaxin et al. [2019] S. Jiaxin, C. Liang, L. Hou, J. Li, Z. Liu, H. Zhang,  Deepchannel: Salience estimation by contrastive learning for extractive document summarization,  arXiv preprint arXiv:2106.01890 33 (2019) 6999–7006.
  * Junpeng et al. [2021] L. Junpeng, Y. Zou, H. Zhang, H. Chen, Z. Ding, C. Yuan, X. Wang,  Topic-aware contrastive learning for abstractive dialogue summarization,  arXiv preprint arXiv:2109.04994 (2021).
  * Shuyang and Wang [2021] C. Shuyang, L. Wang,  CLIFF: Contrastive learning for improving faithfulness and factuality in abstractive summarization,  arXiv preprint arXiv:2109.09209 (2021).
  * Danqing et al. [2021] W. Danqing, J. Chen, H. Zhou, X. Qiu, L. Li,  Contrastive aligned joint learning for multilingual summarization,  In Findings of the Association for Computational Linguistics: ACL-IJCNLP (2021) 2739–2750.
  * Wei et al. [2021] L. Wei, H. Wu, W. Mu, Z. Li, T. Chen, D. Nie,  CO2Sum: Contrastive Learning for Factual-Consistent Abstractive Summarization,  arXiv preprint arXiv:2112.01147 (2021).
  * Goodwin et al. [2020] T. R. Goodwin, M. E. Savery, D. Demner-Fushman,  Flight of the PEGASUS? Comparing transformers on few-shot and zero-shot multi-document abstractive summarization,  in: Proceedings of COLING. International Conference on Computational Linguistics, volume 2020, NIH Public Access, 2020, p. 5640\.
  * Aghajanyan et al. [2021] A. Aghajanyan, D. Okhonko, M. Lewis, M. Joshi, H. Xu, G. Ghosh, L. Zettlemoyer,  Htlm: Hyper-text pre-training and prompting of language models,  arXiv preprint arXiv:2107.06955 (2021).
  * Li and Liang [2021] X. L. Li, P. Liang,  Prefix-tuning: Optimizing continuous prompts for generation,  arXiv preprint arXiv:2101.00190 (2021).
  * Zhang et al. [2019] X. Zhang, K. Meng, G. Liu,  Hie-transformer: a hierarchical hybrid transformer for abstractive article summarization,  in: International Conference on Neural Information Processing, Springer, 2019, pp. 248–258.
  * Ghosal et al. [2022] D. Ghosal, S. Shen, N. Majumder, R. Mihalcea, S. Poria,  Cicero: A dataset for contextualized commonsense inference in dialogues,  in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. 5010–5028.
  * Hazarika et al. [2021] D. Hazarika, S. Poria, R. Zimmermann, R. Mihalcea,  Conversational transfer learning for emotion recognition,  Information Fusion 65 (2021) 1–12.
  * Yadav and Chatterjee [2016] N. Yadav, N. Chatterjee,  Text summarization using sentiment analysis for DUC data,  in: 2016 International Conference on Information Technology (ICIT), IEEE, 2016, pp. 229–234.
  * Ohana and Tierney [2009] B. Ohana, B. Tierney,  Sentiment classification of reviews using SentiWordNet,  Proceedings of IT&T 8 (2009).
  * Doǧan and Kaya [2019] E. Doǧan, B. Kaya,  Deep learning based sentiment analysis and text summarization in social networks,  in: 2019 International Artificial Intelligence and Data Processing Symposium (IDAP), IEEE, 2019, pp. 1–6.
  * Shetty and Bajaj [2015] A. Shetty, R. Bajaj,  Auto text summarization with categorization and sentiment analysis,  International Journal of Computer Applications 130 (2015) 57–60.
  * Beineke et al. [2004] P. Beineke, T. Hastie, C. Manning, S. Vaithyanathan,  Exploring sentiment summarization,  in: Proceedings of the AAAI spring symposium on exploring attitude and affect in text: theories and applications, volume 39, The AAAI Press Palo Alto, CA, 2004.
  * Wu et al. [2016] H. Wu, Y. Gu, S. Sun, X. Gu,  Aspect-based opinion summarization with convolutional neural networks,  in: 2016 International Joint Conference on Neural Networks (IJCNN), IEEE, 2016, pp. 3157–3163.
  * Titov and McDonald [2008] I. Titov, R. McDonald,  A joint model of text and aspect ratings for sentiment summarization,  in: proceedings of ACL-08: HLT, 2008, pp. 308–316.
  * Dhanush et al. [2016] D. Dhanush, A. K. Thakur, N. P. Diwakar,  Aspect-based sentiment summarization with deep neural networks,  International Journal of Engineering Research & Technology (IJERT) 5 (2016).
  * Nishikawa et al. [2010] H. Nishikawa, T. Hasegawa, Y. Matsuo, G. Kikui,  Optimizing informativeness and readability for sentiment summarization,  in: Proceedings of the ACL 2010 Conference Short Papers, 2010, pp. 325–330.
  * Lerman et al. [2009] K. Lerman, S. Blair-Goldensohn, R. McDonald,  Sentiment summarization: evaluating and learning user preferences (2009).
  * Massey Jr [1951] F. J. Massey Jr,  The Kolmogorov-Smirnov test for goodness of fit,  Journal of the American statistical Association 46 (1951) 68–78.
  * Plackett [1983] R. L. Plackett,  Karl Pearson and the chi-squared test,  International statistical review/revue internationale de statistique (1983) 59–72.
  * McKnight and Najab [2010] P. E. McKnight, J. Najab,  Mann-Whitney U Test,  The Corsini encyclopedia of psychology (2010) 1–1.
  * Lawley [1938] D. Lawley,  A generalization of Fisher’s z test,  Biometrika 30 (1938) 180–187.
  * Pellissier Tanon et al. [2020] T. Pellissier Tanon, G. Weikum, F. Suchanek,  Yago 4: A reason-able knowledge base,  in: European Semantic Web Conference, Springer, 2020, pp. 583–596.
  * Shenoy et al. [2022] K. Shenoy, F. Ilievski, D. Garijo, D. Schwabe, P. Szekely,  A Study of the Quality of Wikidata,  Journal of Web Semantics 72 (2022) 100679.
  * Wu et al. [2017] Z. Wu, L. Lei, G. Li, H. Huang, C. Zheng, E. Chen, G. Xu,  A topic modeling based approach to novel document automatic summarization,  Expert Systems with Applications 84 (2017) 12–23.
  * Liu et al. [2021] Y. Liu, Z.-Y. Dou, P. Liu,  Refsum: Refactoring neural summarization,  arXiv preprint arXiv:2104.07210 (2021).
  * Zhang et al. [2019] H. Zhang, J. Xu, J. Wang,  Pretraining-based natural language generation for text summarization,  arXiv preprint arXiv:1902.09243 (2019).
  * Vaswani et al. [2017] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin,  Attention is all you need,  Advances in neural information processing systems 30 (2017).
  * Chen et al. [2020] T. Chen, S. Kornblith, M. Norouzi, G. Hinton,  A simple framework for contrastive learning of visual representations,  in: International conference on machine learning, PMLR, 2020, pp. 1597–1607.
  * Wu et al. [2020] H. Wu, T. Ma, L. Wu, T. Manyumwa, S. Ji,  Unsupervised reference-free summary quality evaluation via contrastive learning,  arXiv preprint arXiv:2010.01781 (2020).
  * Lee et al. [2020] S. Lee, D. B. Lee, S. J. Hwang,  Contrastive learning with adversarial perturbations for conditional text generation,  arXiv preprint arXiv:2012.07280 (2020).
  * Justel et al. [1997] A. Justel, D. Peña, R. Zamar,  A multivariate Kolmogorov-Smirnov test of goodness of fit,  Statistics & probability letters 35 (1997) 251–259.
  * Lin [2004] C.-Y. Lin,  Rouge: A package for automatic evaluation of summaries,  in: Text summarization branches out, 2004, pp. 74–81.
  * Ng and Abrecht [2015] J.-P. Ng, V. Abrecht,  Better summarization evaluation with word embeddings for ROUGE,  arXiv preprint arXiv:1508.06034 (2015).
  * Papineni et al. [2002] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu,  Bleu: a method for automatic evaluation of machine translation,  in: Proceedings of the 40th annual meeting of the Association for Computational Linguistics, 2002, pp. 311–318.
  * Banerjee and Lavie [2005] S. Banerjee, A. Lavie,  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,  in: Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, 2005, pp. 65–72.
  * Lavie and Denkowski [2009] A. Lavie, M. J. Denkowski,  The METEOR metric for automatic evaluation of machine translation,  Machine translation 23 (2009) 105–115.
  * Zhao et al. [2019] W. Zhao, M. Peyrard, F. Liu, Y. Gao, C. M. Meyer, S. Eger,  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance,  arXiv preprint arXiv:1909.02622 (2019).
  * Zhang et al. [2019] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi,  Bertscore: Evaluating text generation with bert,  arXiv preprint arXiv:1904.09675 (2019).
  * Colombo et al. [2022] P. J. A. Colombo, C. Clavel, P. Piantanida,  Infolm: A new metric to evaluate summarization & data2text generation,  in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2022, pp. 10554–10562.
  * van der Lee et al. [2021] C. van der Lee, A. Gatt, E. van Miltenburg, E. Krahmer,  Human evaluation of automatically generated text: Current trends and best practice guidelines,  Computer Speech & Language 67 (2021) 101151.
  * Novikova et al. [2018] J. Novikova, O. Dušek, V. Rieser,  RankME: Reliable human ratings for natural language generation,  arXiv preprint arXiv:1803.05928 (2018).
  * Klaus [1980] K. Klaus, Content analysis: An introduction to its methodology, 1980.
  * Cohen [1960] J. Cohen,  A coefficient of agreement for nominal scales,  Educational and psychological measurement 20 (1960) 37–46.
  * Davies and Fleiss [1982] M. Davies, J. L. Fleiss,  Measuring agreement for multinomial data,  Biometrics (1982) 1047–1051.
  * Bennett et al. [1954] E. M. Bennett, R. Alpert, A. Goldstein,  Communications through limited-response questioning,  Public Opinion Quarterly 18 (1954) 303–308.
  * Scott [1955] W. A. Scott,  Reliability of content analysis: The case of nominal scale coding,  Public opinion quarterly (1955) 321–325.
  * Zapf et al. [2016] A. Zapf, S. Castell, L. Morawietz, A. Karch,  Measuring inter-rater reliability for nominal data–which coefficients and confidence intervals are appropriate?,  BMC medical research methodology 16 (2016) 1–10.
  * McHugh [2012] M. L. McHugh,  Interrater reliability: the kappa statistic,  Biochemia medica 22 (2012) 276–282.
  * Craig [1981] R. T. Craig,  Generalization of Scott’s index of intercoder agreement,  Public Opinion Quarterly 45 (1981) 260–264.
  * Warrens [2012] M. J. Warrens,  The effect of combining categories on Bennett, Alpert and Goldstein’s S,  Statistical Methodology 9 (2012) 341–352.
  * Lin et al. [2018] J. Lin, X. Sun, S. Ma, Q. Su,  Global encoding for abstractive summarization,  arXiv preprint arXiv:1805.03989 (2018).
  * Zhang et al. [2021] M. Zhang, G. Zhou, W. Yu, W. Liu,  FAR-ASS: Fact-aware reinforced abstractive sentence summarization,  Information Processing & Management 58 (2021) 102478.
  * Kryściński et al. [2019] W. Kryściński, B. McCann, C. Xiong, R. Socher,  Evaluating the factual consistency of abstractive text summarization,  arXiv preprint arXiv:1910.12840 (2019).
  * Cao et al. [2020] M. Cao, Y. Dong, J. Wu, J. C. K. Cheung,  Factual error correction for abstractive summarization models,  arXiv preprint arXiv:2010.08712 (2020).
  * Zhu et al. [2020] C. Zhu, W. Hinthorn, R. Xu, Q. Zeng, M. Zeng, X. Huang, M. Jiang,  Enhancing factual consistency of abstractive summarization,  arXiv preprint arXiv:2003.08612 (2020).