https://www.nltk.org/_modules/nltk/tokenize/punkt.html

# Keyphrase Generation Beyond the Boundaries of Title and Abstract

Krishna Garg  Jishnu Ray Chowdhury  Cornelia Caragea
Computer Science
University of Illinois Chicago
<EMAIL_ADDRESS>
###### Abstract

Keyphrase generation aims at generating important phrases (keyphrases) that
best describe a given document. In scholarly domains, current approaches have
largely used only the title and abstract of the articles to generate
keyphrases. In this paper, we comprehensively explore whether the integration
of additional information from the full text of a given article or from
semantically similar articles can be helpful for a neural keyphrase generation
model or not. We discover that adding sentences from the full text,
particularly in the form of the extractive summary of the article can
significantly improve the generation of both types of keyphrases that are
either present or absent from the text. Experimental results with three widely
used models for keyphrase generation along with one of the latest transformer
models suitable for longer documents, Longformer Encoder-Decoder (LED)
validate the observation. We also present a new large-scale scholarly dataset
FullTextKP for keyphrase generation. Unlike prior large-scale datasets,
FullTextKP includes the full text of the articles along with the title and
abstract. We release the source code at https://github.com/kgarg8/FullTextKP.

##  1 Introduction

Keyphrases of scientific papers provide important topical information about
the papers in a highly concise form and are crucial for understanding the
evolution of ideas in a scientific field (Hall et al., 2008; Augenstein et
al., 2017). Keyphrases of scientific papers can be also useful for mining and
analyzing the literature (Augenstein et al., 2017) and for multiple downstream
tasks such as index construction (Ritchie et al., 2006), summarization
(Qazvinian et al., 2010), query formulation (Song et al., 2006),
recommendation (Augenstein et al., 2017), reviewer matching for paper
submissions (Augenstein et al., 2017) and for clustering papers for fast
retrieval (Hammouda et al., 2005).

Identifying keyphrases from the scientific documents has been popularly
studied under the paradigm of Keyphrase Extraction for over a decade Mihalcea
and Tarau (2004); Wan and Xiao (2008a); Florescu and Caragea (2017); Liu et
al. (2009, 2010); Sterckx et al. (2015, 2016); Caragea et al. (2014); Das
Gollapalli and Caragea (2014); Al-Zaidy et al. (2019); Patel and Caragea
(2021). However, one of the major limitations is that these approaches are not
able to capture the semantic information in the document, particularly the
absent keyphrases (keyphrases that are not present in the document).

In this paper, we therefore focus on a newer paradigm of keyphrase generation
introduced by Meng et al. (2017). Instead of extracting keyphrases from the
document, we model the problem as a neural sequence-to-sequence (seq2seq)
approach where we learn to generate keyphrases in an autoregressive way. This
approach can not only predict the exact keyphrases that are present in a
document (present keyphrases) but also those that are semantically relevant
but absent from the document (absent keyphrases). While many works (Meng et
al., 2017; Chen et al., 2019b, 2020; Yuan et al., 2020; Ye et al., 2021b)
adopt this approach, their focus is on architectural innovations to improve
the generation of both present and absent keyphrases. In contrast, our focus
is different from that of the prior works in keyphrase generation: we propose
to extend the source of information of the input sequence, which is thus far
limited to only title and abstract.

Method | Snippets from title and abstracts and beyond
---|---
Title |  Performance of Power Detector Sensors of DTV Signals in IEEE 802.22 WRANs
Abstract |  Sensing is the most important component in any cognitive radio system. The IEEE 802.22 Working Group (WG) is formulating the first worldwide standard for cognitive radios to operate in the television (TV) bands…
Citations |  The IEEE 802.22 functional requirements document [2] states that spectrum sensing is required …
Non-Citations |  The keep-out region is a region around the primary user (e.g. DTV transmitter) where… Given these simulation scenarios we can evaluate various spectrum sensing techniques.
Summary |  Of course, identification of which TV channels… This paper will describe … in the IEEE 802.22 WG to evaluate spectrum sensing techniques… Calculating the base station keep-out region …
Ret-Aug |  The Bernoulli nonuniform sampling is further extended to matrix formulation, which allows the application of spectrum sensing for cognitive radio signal detection. \- From Paper: ’Noise Enhancement and SNR Equivalence in Bernoulli Nonuniform Sampling’
Keyphrases |  Cognitive Radio, IEEE 802.22, Spectrum Sensing, Power Detector, Keep-out Region, False alarm probability, Misdetection probability

Table 1: A sample from FulltextKP dataset showing Present and Absent gold
keyphrases in the different semantic spaces of the article. Refer to the
Methods section for the description of the methods.

To this end, we curate a large-scale dataset of papers published by
ACM111https://dl.acm.org/ which contains not only the title and abstract of
the documents but also the full text from the documents. We call this dataset
FullTextKP. Owing to the scarcity of such large-scale datasets with full text
and the difficulty of parsing and understanding the entire text of the
documents, the integration of additional information that goes beyond the
title and abstract for keyphrase generation has been largely ignored.

In this paper, we provide innovative ways of using certain parts of the
documents that could be rich source of information, e.g., using citation
sentences from the content of the document or an extractive summary of the
document (as illustrated in Table 1). Interestingly, through comprehensive
experiments, we find that the citation contexts or provenance that had been a
rich source of information for the keyphrase extraction task Caragea et al.
(2014); Das Gollapalli and Caragea (2014) are not the richest sources of
information in the keyphrase generation task. In contrast, we observe that the
semantic information from the document could be best assimilated if we
summarize the document and use the summary of the document instead of the
document itself. To summarize each document, we use an unsupervised
summarization approach and show that the extractive summmary not only contains
the highest topical information useful for keyphrase generation but can also
fit the computational budget. Remarkably, the approach based on the
incorporation of summary outperforms methods that use just title and abstract
by a wide margin, oftentimes yielding 2-3 times improvements in performance in
both present and absent keyphrases.

Our experiments are performed with three widely used models for keyphrase
generation as well as, to our knowledge, for the first time in keyphrase
generation, with a transformer model suited for long documents, Longformer
Encoder-Decoder (LED) Beltagy et al. (2020).

Overall, we make the following contributions:

  1. 1.

We explore the benefit of integrating additional information from different
data sources (not just the title and abstract) into neural seq2seq models for
keyphrase generation (i.e., predicting both present and absent keyphrases).
The different data sources include random sentences from the article
body,222Body includes all the sections of a scientific article besides the
title and abstract sentences from the summary of the body, citation sentences,
non-citation sentences, and sentences from other related documents in the
training set.

  2. 2.

We show that the sentences from the body of the article that form an
extractive summary are great sources of finding good content for keyphrase
generation.

  3. 3.

We present a new dataset, FullTextKP, of $\sim$140,000 articles, each with its
full text.

##  2 Related Work

Keyphrase Generation. Recent approaches to keyphrase generation have been
dominated by neural seq2seq models because they provide a mechanism to also
generate absent keyphrases. Meng et al. Meng et al. (2017) originally proposed
an RNN-based seq2seq model along with CopyNet (CopyRNN) for keyphrase
generation. Chen et al. Chen et al. (2018) improved CopyRNN by taking
correlations between the predicted keyphrases into account (CorrRNN). Yuan et
al. (2020) extended the CopyRNN model to dynamically generate a variable
number of diverse keyphrases. Chen et al. (2020) proposed an exclusive
hierarchical decoding framework to capture the hierarchical compositionality
of keyphrases (ExHiRD). Swaminathan et al. (2020) explored Generative
Adversarial Networks for keyphrase generation. Ye et al. (2021b) modeled the
target keyphrases as a set instead of a predefined sequence to improve the
biased matching of the generated and target keyphrases. These methods use only
title and abstract as the input sequence. Our work can aid all of these
methods to further enhance the performance of keyphrase generation by
augmenting the source texts with information from the body of the documents.

External Information. Some of the previous methods try to incorporate external
information to enhance the performance of keyphrase extraction. For example,
MAUI Medelyan et al. (2009) uses semantic information from Wikipedia, CeKE
Caragea et al. (2014) and CiteTextRank Gollapalli and Caragea (2014) use
information from citation networks, and SingleRank and ExpandRank Wan and Xiao
(2008a) use information from local textual neighborhoods. However, these
methods are limited to extraction-based methods whereas we focus on generating
keyphrases to have a holistic or semantic understanding of the document when
the natural language text is fed to the neural model.

Several works in keyphrase generation started to use external information to
generate keyphrases. For example, Chen et al. (2019a), Santosh et al. (2021)
proposed methods where they augmented a set of additional keyphrases from
semantically similar documents and use hidden state representations built from
these keyphrases, Diao et al. (2020) used cross-document attention networks,
Ye et al. (2021a) used information from retrieved document-keyphrase pairs
that are similar to the source document, Shen et al. (2021) used a phrase bank
by pooling all keyphrases from all the articles. Similar to these works, one
of our methods (Retrieval-Augmentation) retrieves relevant sentences from
semantically similar documents and directly appends them to the title and
abstract.

Datasets. There have been numerous efforts in the direction of curating large-
scale datasets for either keyphrase extraction or keyphrase generation. In the
non-scholarly domain, the datasets such as KPTimes (Gallina et al., 2019),
JPTimes (Gallina et al., 2019), KPCrowd (Marujo et al., 2013), DUC-2001 (Wan
and Xiao, 2008b), were curated from news articles, OpenKP (Xiong et al., 2019)
was mined from webpages from a search engine. In the scholarly domain, there
has been ongoing research with the datasets such as Krapivin (Krapivin et al.,
2008), Inspec (Hulth, 2003), SemEval (Kim et al., 2010), NUS (Nguyen and Kan,
2007), KP20k (Meng et al., 2017) and OAGK (Cano and Bojar, 2018). The
limitation of these datasets is that they are either very small (of the
magnitude of a few hundreds or thousands) or they are bounded to title and
abstract. For instance, the widely used KP20k dataset (Meng et al., 2017)
consists of only the title and abstract of the research articles and the
average words per document is as low as 176. To leverage and understand the
benefit of additional data, we mined a new dataset based on the articles
primarily from the ACM database which contains not just the titles and
abstracts but also the full text of the scholarly documents. In our dataset,
the typical length of full papers is 5000-10000 tokens.

##  3 Methods

We use four different models, three of which are popular models in the
Keyphrase Generation task, viz. catSeq Yuan et al. (2020), One2Set Ye et al.
(2021b), ExHiRD Chen et al. (2020); and the fourth model is based on one of
the latest Transformer models, i.e., Longformer Encoder-Decoder (LED) Beltagy
et al. (2020), suitable for long documents. The baseline in each model takes
the title and abstract (T+A) as input and predicts a sequence of keyphrases.
We explore different extensions (also illustrated in Table 1) for each model
by concatenating different types of data to the title and abstract (T+A).
During the concatenation of additional information, we use a delimiter
$<$sep$>$ between the title and abstract and between all the additional
sentences as follows: Title $<$sep$>$ Abstract $<$sep$>$ Sent1 $<$sep$>$ Sent2
$<$sep$>$ … Sentk. We describe different types of data (input sequence) below:

T+A+Random: For this setup, we concatenate $k$ randomly chosen sentences (from
the body) to the title and abstract in the original order of their occurrence
in the body.

T+A+Citations: For this setup, we collect all the sentences (from the body)
that have cited some other article. We call such sentences as citation
sentences. We randomly select $k$ sentences from the entire pool and
concatenate them to the title and abstract in the order of their occurrence in
the body of the given article. The incorporation of citation sentences is
inspired by Naïve-Bayes or TextRank-based approaches Gollapalli and Caragea
(2014); Caragea et al. (2014) or construction of CitationGraph Viswanathan et
al. (2021) which enhanced their keyphrase extraction performance through the
integration of citation information.

T+A+Non-Citations: For this setup, we collect all the sentences (from the
body) that do not cite any other article. We randomly select $k$ sentences
from the entire pool and concatenate them to the title and abstract in the
order of their occurrence in the body of the given article.

T+A+Summary: In this method, we summarize the body of the article using a
state-of-the-art unsupervised summarization algorithm, PacSum (Zheng and
Lapata, 2019) and append $k$ sentences from the summary to T+A of the article.
Zheng and Lapata showed that TF-IDF-based PacSum is better than TextRank and
other baselines, and is also highly competitive with even BERT-based PacSum.
PacSum can be replaced with other summarization algorithms but we choose TF-
IDF-based PacSum simply as a representative of a powerful and efficient model
for unsupervised summarization to test the efficacy of summarizing the body.

The steps to obtain the extractive summary of the documents using PacSum are
as follows. Consider a document as a list of sentences and each sentence as a
list of words. The entire document can be visualized as a directed graph in
which the individual sentences are the nodes and the edges are weighted by
similarity. We first calculate the tf-idf to understand the relevance of a
word to a sentence in the collection of sentences (document), quite analogous
to computing tf-idf of a word of a document in a corpus of documents. We then
compute the similarity between every pair of sentences. Next, we compute a
threshold based on which we normalize the similarity scores. Subsequently, we
compute forward and backward edge scores for the directed edges. Afterward, we
compute the degree centrality of a sentence (node) in the document (graph) as
the weighted average of forward and backward edge scores. Finally, we rank the
sentences based on their centrality values, select the $k$ highest ranked
sentences, and concatenate them to the title and abstract. We describe the
algorithm in Appendix C.

T+A+Retrieval-Augmentation: In this method, we retrieve and augment $k$
semantically similar sentences from the training corpus to the T+A of each
article. To this end, we first create a set of all the sentences from the
titles and abstracts of the articles in the entire training dataset. Next, we
embed each sentence in the set using SPECTER (Cohan et al., 2020). We treat
these embeddings as key embeddings representing the corresponding sentences
(values). Given a target article (query), we then embed its title and abstract
using SPECTER. This embedding serves as a query embedding to search related
sentences from other articles. Subsequently, we compute the dot-product
similarity of the query embedding with all the key embeddings using FAISS
(Johnson et al., 2019). Last, we select $k$ sentences (values) corresponding
to the top $k$ most similar key embeddings and concatenate the sentences to
the title and abstract of the query article. We provide the algorithm in
Appendix D.

##  4 Experimental Setup

###  4.1  FullTextKP dataset

To evaluate the performance of models leveraging different types of
information, we construct a new dataset, which we call FullTextKP. Our
FullTextKP dataset consists of research papers that are published by ACM333The
full ACM dataset is available from the ACM Digital Library by request. and are
available in the ACM digital library in its International Conference
Proceedings Series (ICPS).

We used only the articles which have at least the five fields, viz., title,
abstract, keywords, full text and references. We lowercased the text and
constructed numerous regexes to remove the escape sequences, html tags, urls,
emails, etc. We also replaced all the numbers and roman numerals with a
<digit> token. Further, we used PunktSentenceTokenizer to segment the document
into sentences and NLTK’s word_tokenizer to tokenize the sentences into
tokens. We constructed suitable regex to extract the citation sentences and we
considered all other sentences as non-citation sentences. The articles without
any citation sentences and duplicates were further removed from the
collection. Thus, we were able to collect 142,844 articles. We split the
FullTextKP dataset into 80/10/10 for train, test and validation sets.
Statistical information about the dataset is shown in Table 2.

The construction of this dataset addresses the sparsity of large-scale
datasets for keyphrase extraction/generation from scientific papers and is
aimed at enabling deep learning modeling. Currently, there exist only a couple
of large-scale datasets for this task, e.g., KP20k Meng et al. (2017), and
OAGK Cano and Bojar (2018). Unlike these datasets, which contain only the
title and abstract of each paper, our dataset provides access to the full text
of each paper.

Total papers in the corpus | $142,844$
---|---
Training set size | $114,271$
Validation set size | $14,287$
Test set size | $14,286$
% of present keyphrases | $55.8$
% of absent keyphrases | $44.2$
Average keyphrases per paper | $4.3$
|
Table 2: FulltextKP dataset statistics Model | Method | P@5 | R@5 | F1@5 | P@M | R@M | F1@M
---|---|---|---|---|---|---|---
catSeq | Title+Abstract | 0.1672 | 0.3433 | 0.2242 | 0.3333 | 0.3434 | 0.3381
\+ Ret-Aug | 0.1660 | 0.3421 | 0.2240 | 0.3383 | 0.3431 | 0.3401
\+ Citations | 0.1701 | 0.3513 | 0.2292 | 0.3602 | 0.3513 | 0.3551
\+ Non-Citations | 0.1781 | 0.3653 | 0.2402 | 0.3653 | 0.3663 | 0.3662
\+ Random | 0.1752 | 0.3595 | 0.2363 | 0.3704 | 0.3595 | 0.3644
\+ Summary | 0.1871 | 0.3802 | 0.2502 | 0.3794 | 0.3802 | 0.3802
One2Set | Title+Abstract | 0.2033 | 0.4184 | 0.2733 | 0.3006 | 0.4256 | 0.3512
\+ Ret-Aug | 0.1784 | 0.3689 | 0.2415 | 0.3135 | 0.3709 | 0.3393
\+ Citations | 0.1921 | 0.3944 | 0.2582 | 0.3165 | 0.3963 | 0.3511
\+ Non-Citations | 0.1926 | 0.39513 | 0.2599 | 0.3166 | 0.39714 | 0.3522
\+ Random | 0.1914 | 0.3917 | 0.2574 | 0.3149 | 0.3948 | 0.3493
\+ Summary | 0.1895 | 0.38513 | 0.2539 | 0.3257 | 0.38714 | 0.3532
ExHiRD | Title+Abstract | 0.1842 | 0.3813 | 0.2362 | 0.3203 | 0.3823 | 0.3251
\+ Ret-Aug | 0.1870 | 0.3902 | 0.2411 | 0.3172 | 0.3912 | 0.3252
\+ Citations | 0.1940 | 0.4021 | 0.2490 | 0.3361 | 0.4031 | 0.3400
\+ Non-Citations | 0.1971 | 0.4106 | 0.2578 | 0.3532 | 0.4071 | 0.3541
\+ Random | 0.1950 | 0.4031 | 0.2490 | 0.3522 | 0.4041 | 0.3511
\+ Summary | 0.2022 | 0.4163 | 0.2582 | 0.3691 | 0.4164 | 0.3660
LED  | Title+Abstract | 0.19311 | 0.38921 | 0.25815 | 0.3331 | 0.39222 | 0.3609
\+ Ret-Aug | 0.1936 | 0.39110 | 0.2587 | 0.3194 | 0.39411 | 0.3532
\+ Citations | 0.2015 | 0.4039 | 0.2686 | 0.3384 | 0.40710 | 0.3692
\+ Non-Citations | 0.2093 | 0.4216 | 0.2804 | 0.3489 | 0.4257 | 0.3823
\+ Random | 0.2130 | 0.4281 | 0.2840 | 0.3546 | 0.4310 | 0.3893
\+ Summary | 0.2174 | 0.4318 | 0.2895 | 0.3652 | 0.4358 | 0.3973
|  |  |  |  |  |  |
Table 3: Results comparing present keyphrase performance with different methods of using additional data with the FullTextKP dataset. Subscripts denote the standard deviation as multiples of $\pm$ 0.001. Model | Method | P@5 | R@5 | F1@5 | P@M | R@M | F1@M
---|---|---|---|---|---|---|---
catSeq | Title+Abstract | 0.0070 | 0.0171 | 0.0101 | 0.0302 | 0.0171 | 0.0211
\+ Ret-Aug | 0.0090 | 0.0211 | 0.0121 | 0.0371 | 0.0211 | 0.0271
\+ Citations | 0.0131 | 0.0312 | 0.0192 | 0.0523 | 0.0312 | 0.0393
\+ Non-Citations | 0.0210 | 0.0461 | 0.0290 | 0.0681 | 0.0461 | 0.0550
\+ Random | 0.0221 | 0.0492 | 0.0301 | 0.0733 | 0.0492 | 0.0593
\+ Summary | 0.0331 | 0.0692 | 0.0451 | 0.0932 | 0.0692 | 0.0792
One2Set | Title+Abstract | 0.0181 | 0.0422 | 0.0261 | 0.0431 | 0.0422 | 0.0430
\+ Ret-Aug | 0.0170 | 0.0392 | 0.0231 | 0.0442 | 0.0392 | 0.0411
\+ Citations | 0.0211 | 0.0472 | 0.0291 | 0.0531 | 0.0472 | 0.0491
\+ Non-Citations | 0.0231 | 0.0532 | 0.0321 | 0.0541 | 0.0532 | 0.0532
\+ Random | 0.0240 | 0.0532 | 0.0331 | 0.0561 | 0.0532 | 0.0542
\+ Summary | 0.0233 | 0.0506 | 0.0324 | 0.0544 | 0.0516 | 0.0525
ExHiRD | Title+Abstract | 0.0120 | 0.0272 | 0.0161 | 0.0421 | 0.0260 | 0.0300
\+ Ret-Aug | 0.0100 | 0.0220 | 0.0130 | 0.0351 | 0.0220 | 0.0250
\+ Citations | 0.0190 | 0.0410 | 0.0240 | 0.0601 | 0.0410 | 0.0451
\+ Non-Citations | 0.0240 | 0.0510 | 0.0310 | 0.0711 | 0.0510 | 0.0550
\+ Random | 0.0260 | 0.0541 | 0.0320 | 0.0761 | 0.0541 | 0.0591
\+ Summary | 0.0371 | 0.0743 | 0.0461 | 0.1002 | 0.0743 | 0.0802
LED  | Title+Abstract | 0.0270 | 0.0631 | 0.0380 | 0.06111 | 0.0631 | 0.0615
\+ Ret-Aug | 0.0141 | 0.0342 | 0.0201 | 0.0531 | 0.0342 | 0.0411
\+ Citations | 0.0252 | 0.0584 | 0.0353 | 0.0732 | 0.0584 | 0.0653
\+ Non-Citations | 0.0331 | 0.0712 | 0.0451 | 0.0881 | 0.0712 | 0.0791
\+ Random | 0.0362 | 0.0784 | 0.0492 | 0.0931 | 0.0784 | 0.0852
\+ Summary | 0.0471 | 0.1001 | 0.0641 | 0.1124 | 0.1001 | 0.1062
|  |  |  |  |  |  |
Table 4: Results comparing absent keyphrase performance with different methods
of using additional data with the FullTextKP dataset. Subscripts denote the
standard deviation as multiples of $\pm$ 0.001.

###  4.2 Evaluation

We use the same evaluation method as Chan et al. (2019); Chen et al. (2020)
and report the macro-averaged Precision (P), Recall (R) and F1-scores ($\bf
F_{1}$) in two different evaluation settings: @5 (dummy keyphrases are
appended to the predicted keyphrases to make the total count as 5) and @M
(directly compare the generated keyphrases against the gold keyphrases). All
the keyphrases are stemmed using
PorterStemmer444https://www.nltk.org/_modules/nltk/stem/porter.html before
comparison. As in prior works, we treat a keyphrase as absent if it is absent
from the title and abstract, which helps us compare the performance of
generating absent and present keyphrases for different models in a consistent
way.

For the entire data including Title, Abstract and the additional data using
different methods, we restrict the maximum sequence length to $800$. This
requires us to use an adaptive $k$ for the additional number of sentences that
can fit in the given maximum sequence length. The selection of the sequence
length was based on the tradeoff between the performance and computational
budget of the models. We also experiment with longer sequence lengths for the
LED model in §5.3.

##  5 Results

In this section, we first present the results for present and absent keyphrase
generation performance in §5.1 for the different methods described in §3. We
further explore our best performing method, i.e., T+A+summary into two sub-
categories: extractive and abstractive summarization methods in §5.2. Finally,
in §5.3 we explore the performance of the best performing model, i.e.,
Longformer Encoder-Decoder with sequence lengths much higher than 800. We run
all the experiments three times and report the average.

###  5.1 Present & Absent Keyphrase Generation

Tables 3 and 4 present the results for the present and absent keyphrase
performances, respectively, for our different experiments. The results
demonstrate that adding extra information from the body of the given article
to the baseline (T+A) is beneficial for both present and absent keyphrase
generation. Augmenting the summary sentences provides the most substantial
boost compared to any other method for both present and absent keyphrase
performance. For instance, improvements in present keyphrase performance on
F1@M metric are: catSeq (0.338 $\rightarrow$ 0.380), ExHiRD (0.325
$\rightarrow$ 0.366), LED (0.360 $\rightarrow 0.397$). The absent keyphrase
performance using the summary method improves up to four times the baseline
performance; for example, catSeq (0.021 $\rightarrow$ 0.079), ExHiRD (0.030
$\rightarrow$ 0.080), LED (0.061 $\rightarrow$ 0.106). Interestingly, with
One2Set, although F1@M for present keyphrase generation does not improve with
Summary and Random methods, the trend for absent keyphrase performance is very
similar to that of the other models. The discrepancy in the present keyphrase
generation performance could be because One2Set might potentially be biased
towards identifying the keyphrases from the earlier portions of the documents.
For our setting of long documents, it may require better heuristics to
initialize the control codes so as to attend to the later portions of the
document. Still, in three out of four models, adding summary information gives
us the best results. This seems to follow the natural intuition that the
summary of the article contains the most topical information useful for the
generation the keyphrases.

Figure 1 further validates the above intuition. Summary method is the richest
source of present keyphrases. The total number of present keyphrases in the
texts of all Summary samples is about $\sim$88,000 higher than just
Title+Abstract samples. So, the model learns to generalize better when trained
on such highly topical input texts of Summary method.

Figure 1: #Present and #Absent Keyphrases in terms of the whole input text
(for various methods) in FulltextKP dataset. $+$ indicates concatenation of
title and abstract with the specified information.

Interestingly, we find that using only the citation sentences results in worse
performance compared to using non-citation sentences or just random sentences.
This observation is in contrast to the observation of the use of citation
sentences for the keyphrase extraction methods. We hypothesize the reason for
such an observation is that keyphrase extraction methods focus mainly on
identifying keyphrases by capturing the statistical information such as tf-idf
or word co-occurrences that appear in the target document and the cited ones
(through the citation sentences), whereas the deep learning methods try to get
a semantic or holistic understanding of the document itself (i.e., from all
sentences).

Incorporating either non-citation sentences or random sentences gives very
similar results because the entire text contains majority of non-citation
sentences. Unfortunately, appending the sentences retrieved from the
semantically similar documents (using SPECTER) performs worse than the
baseline. We hypothesize that even though the sentences have semantic
similarity on the embedding level, they could still be significantly different
in terms of their meaning and may not contain exactly the same keyphrases.
Rather, a lot of such sentences may confuse and deviate the model from
predicting the gold keyphrases.

Among the four different models, Longformer Encoder-Decoder (LED) performs the
best. We hypothesize that this happens because it is based on the more
sophisticated Transformer model architecture and also because it is suited for
reading long documents. However, LED training takes at least 5-10 times more
than the catSeq model. We provide more details about the compute power and
time in Appendix B.

We performed a 2-tailed statistical significance test with alpha555Probability
of rejecting the null hypothesis when it was actually true value as 0.05 and
the p-values666Probability of getting a result that is as extreme or more
extreme when the null hypothesis is true less than 0.05 were considered
statistically different. We observed that the results were statistically
different for all models except One2Set. With One2Set, the additional
information to the T+A baseline does not yield statistically different
results.

Method | P@5 | R@5 | F1@5 | P@M | R@M | F1@M
---|---|---|---|---|---|---
Abs_Sum | 0.2011 | 0.4052 | 0.2681 | 0.3264 | 0.4092 | 0.3633
Ext_Sum | 0.2174 | 0.4318 | 0.2895 | 0.3652 | 0.4358 | 0.3973
Abs_Sum | 0.0201 | 0.0482 | 0.0291 | 0.0621 | 0.0482 | 0.0541
Ext_Sum | 0.0471 | 0.1001 | 0.0641 | 0.1124 | 0.1001 | 0.1062
|  |  |  |  |  |
Table 5:  Results comparing the performance of Abstractive Summarization vs. Extractive Summarization methods. Subscripts denote the standard deviation as multiples of $\pm$ 0.001. Type | Method | Seq-Len | P@5 | R@5 | F@5 | P@M | R@M | F1@M
---|---|---|---|---|---|---|---|---
Present Keyphrases | Summary | 800 | 0.2174 | 0.4318 | 0.2895 | 0.3652 | 0.4358 | 0.3973
Random | 800 | 0.2130 | 0.4281 | 0.2840 | 0.3546 | 0.4310 | 0.3893
1500 | 0.2112 | 0.4243 | 0.2822 | 0.3703 | 0.4273 | 0.3971
2000 | 0.2152 | 0.4283 | 0.2862 | 0.3724 | 0.4313 | 0.3994
2500 | 0.2124 | 0.4256 | 0.2835 | 0.3891 | 0.4277 | 0.4073
Absent Keyphrases | Summary | 800 | 0.0471 | 0.1001 | 0.0641 | 0.1124 | 0.1001 | 0.1062
Random | 800 | 0.0362 | 0.0784 | 0.0492 | 0.0931 | 0.0784 | 0.0852
1500 | 0.0440 | 0.0971 | 0.0611 | 0.1102 | 0.0971 | 0.1031
2000 | 0.0472 | 0.1003 | 0.0632 | 0.1142 | 0.1003 | 0.1073
2500 | 0.0492 | 0.1084 | 0.0672 | 0.1262 | 0.1084 | 0.1163
|  |  |  |  |  |  |  |
Table 6: Results comparing the performance of Longformer Encoder-Decoder Model
with source texts of different sequence lengths. Subscripts denote the
standard deviation as multiples of $\pm$ 0.001.

###  5.2 Analysis on Summarization Methods

In this section, we further investigate the capability of our best performing
method, i.e., extractive summarization, for keyphrase generation in comparison
with an abstractive summarization approach, abbreviated as T+A+Abs_Sum or
simply Abs_Sum. For the abstractive summarization, we use BigBirdPegasus
Zaheer et al. (2020) model, pretrained on ArXiv dataset Cohan et al. (2018) to
first generate the abstractive summaries. Next, we augment the sentences from
abstractive summaries to the title and abstract of the articles. Further, we
train our best performing model (i.e., LED) using T+A+Abs_Sum method. In Table
5, we compare the results of T+A+Abs_Sum with the T+A+Summary (extractive)
method. Note that, by default, we use Summary for the Extractive Summarization
method Zheng and Lapata (2019), also abbreviated as Ext_Sum.

From Table 5, we observe that the extractive summary added to T+A performs
better than the abstractive summary (added to T+A) by 0.034 (F1@M) and 0.052
(F1@M) for present and absent keyphrase generation, respectively. The behavior
could be explained as follows: (1) Extractive summaries contain sentences that
are more central to the paper Zheng and Lapata (2019). Thus, they tend to
bring complementary information from all parts of the paper that improves the
performance of keyphrase generation models; and (2) We manually inspected a
small subset of the dataset and observed that about two-thirds of the
sentences in the abstractive summaries were from the Introduction section of
the papers, which although contain extended but potentially overlapping or
paraphrased description of the Abstract section. Further, we observed that
there were about 10-15% repetitive statements in the abstractive summaries. We
conclude that augmenting repetitive statements or statements with redundant
information as the abstract do not aid the keyphrase generation performance.

Snippets from title and abstracts and summary with model predictions | Gold Keyphrases
---|---
Segmentation of brain MR images using intuitionistic fuzzy clustering algorithm $<$sep$>$ A new algorithm … using intuitionistic fuzzy clustering (IFCM), is proposed in this paper. |  segmentation, brain mr image, intuitionistic fuzzy set
GPU-based simulation of wireless body area network $<$sep$>$ … in a local workstation to perform model simulations of a … the goal of this project is to gain an understanding of the gpu-based performance of wban model simulation … |  gpu, wireless body area network, model, matlab
Spectrum sharing for directional systems $<$sep$>$ Dynamic spectrum access systems will need to … An example involving a satellite downlink antenna and a broadband wireless access system using directional antennas is presented … |  dynamic spectrum access, directionality
Automatic classification of anuran sounds using convolutional neural networks $<$sep$>$ … networks with mel-frequency cepstral coefficients (mfccs) as input for the task of … $<$sep$>$ Different machine learning approaches for anuran classification … |  anuran, convolutional neural networks, wireless sensor networks, mfcc, machine learning

Table 7: Predicted keyphrases by Longformer Encoder-Decoder model are
highlighted in cyan. The source text is in the format: Title $<$sep$>$
Abstract $<$sep$>$ Summary-Sent_1 … $<$sep$>$ Summary-Sent_k

###  5.3 Exploring longer sequence lengths with Longformer Encoder-Decoder

Scientific documents are often very long sequences of text, ranging from a few
thousands to tens of thousands of tokens. Longformer Encoder-Decoder, proposed
by Beltagy et al. (2020), has the capability to deal with long sequences of
text. We therefore explore the effect of sequence lengths longer than 800 on
the present and absent keyphrase performance. We prepare different versions of
the T+A+Random with the maximum sequence lengths of 800, 1500, 2000 and 2500.
We chose the Random version instead of directly using the entire source
document since Random would be a better representative of the long document if
we need to truncate the sequence length to a fixed budget (i.e., rather than
truncating and completely removing a part of the document, we ensure coverage
of sentences from the entire document). Note that these sequence length values
are based on the word-level tokenization based on NLTK’s word-tokenizer. The
sequence length value based on subword tokenization (used by Transformers) is
actually much longer, but fortunately, LED is able to handle sequence length
up to 16000 tokens.

The results in Table 6 show that both present and absent keyphrase performance
generally increases as we increase the source text length. In the table, we
also compare the performance of these different versions of T+A+Random with
the T+A+Summary with a much smaller sequence length of 800, so as to justify
the purpose of smaller intended summary. We observe that the performance of
Summary-800 is quite comparable to Random-2000 for both present and absent
keyphrase performance, particularly for the most important metric F1@M. This
further validates our hypothesis that summary indeed gives us a richer source
of information. Just the sequence length of 800 can condense the information
that could otherwise be extracted using sequence length of 2000. Furthermore,
the summary has the benefit of fitting a smaller memory and computational
resources budget. We observed that with the increasing sequence lengths, the
training time increased at least at a linear rate and used considerable amount
of GPU and RAM memory. In the case we add more sentences to the summary, we
expect the quality to degrade and converge to that of the Random versions and
furthermore, it will no more stand out as a summary of the document.

In Table 7, we show sample predictions for the best performing method T+A+
(Extractive) Summary using LED model. The gold keyphrases and the predicted
keyphrases are tokenized and stemmed before comparison. We make the following
observations from the table. First, the model can predict accurately both the
present and absent keyphrases. Second, some predicted keyphrases are near-
matches of the gold keyphrases, and despite their quality as keyphrases, due
to the limitation of the metric (exact match), they do not contribute to the
model performance.

##  6 Conclusion

In this paper, we explore numerous ways of incorporating additional data from
the body of the scholarly documents for improving the performance of the
keyphrase generation task. We conclude from the results that the extractive
summary sentences, that are more central to the paper, provide the most
topical information for boosting both present and absent keyphrase generation
performance. The citation, non-citation, and random sentences also bring
complementary information to improve the performance modestly. Some other ways
such as augmenting semantically similar sentences from other papers or
augmenting abstractive summary sentences, that bring repetitive or redundant
information to titles and abstracts, are not effective. We present a
comprehensive analysis with four models including LED, yet unexplored for the
task. Our work aims at breaking apart the barriers of using only titles and
abstracts, and presents a large-scale dataset with full texts for keyphrase
generation.

##  7 Limitations

One of the limitations of the proposed methods is the increased (up to 2-3
times) compute time and memory for the training of the models compared to the
conventional training of the models using only T+A of the articles. We provide
more details in Appendix B. Further, our best performing method T+A+Summary
requires an additional modest overhead of pre-computing summaries for all the
articles.

Another potential limitation of our work is that we can not directly compare
our models on the widely used datasets, e.g., KP20k, Inspec, Krapivin, NUS
since these datasets do not have full texts of the papers. To be
comprehensive, we considered the performance of four models on the new
dataset.

We encourage future work in the direction of better ways of integrating
external information including more sophisticated approaches for the
summarization of scientific documents.

##  8 Ethical Considerations

Since keyphrase generation finds direct application in many downstream tasks
such as recommender systems, reviewer matching, articles clustering for fast
retrieval, etc., it is important to consider the ethical implications of these
models. The models are not perfect and might make misleading predictions at
times. The users must make their own discrimination in deploying the current
models for decision-making systems.

## Acknowledgements

This research is supported in part by NSF CAREER award #1802358, NSF CRI award
#1823292, and UIC Discovery Partners Institute (DPI) award. Any opinions,
findings, and conclusions expressed here are those of the authors and do not
necessarily reflect the views of NSF or DPI. We thank AWS for computational
resources used in this study. We also thank our anonymous reviewers for their
constructive feedback and suggestions.

## References

  * Al-Zaidy et al. (2019) Rabah Al-Zaidy, Cornelia Caragea, and C. Lee Giles. 2019.  Bi-lstm-crf sequence labeling for keyphrase extraction from scholarly documents.  In _Proceedings of The Web Conference (WWW 2019), San Francisco, California, USA_.
  * Augenstein et al. (2017) Isabelle Augenstein, Mrinal Das, Sebastian Riedel, Lakshmi Vikraman, and Andrew McCallum. 2017.  SemEval 2017 task 10: ScienceIE - extracting keyphrases and relations from scientific publications.  In _SemEval (2017)_ , pages 546–555. ACL.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv:2004.05150_.
  * Cano and Bojar (2018) Erion Cano and Ondřej Bojar. 2018.  Keyphrase generation: A text summarization struggle.  _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Long Papers), Melbourne, Australia_ , page 1383–1392.
  * Caragea et al. (2014) Cornelia Caragea, Florin Adrian Bulgarov, Andreea Godea, and Sujatha Das Gollapalli. 2014.  Citation-enhanced keyphrase extraction from research papers: A supervised approach.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1435–1446, Doha, Qatar. Association for Computational Linguistics.
  * Chan et al. (2019) Hou Pong Chan, Wang Chen, Lu Wang, and Irwin King. 2019.  Neural keyphrase generation via reinforcement learning with adaptive rewards.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2163–2174, Florence, Italy. Association for Computational Linguistics.
  * Chen et al. (2018) Jun Chen, Xiaoming Zhang, Yu Wu, Zhao Yan, and Zhoujun Li. 2018.  Keyphrase generation with correlation constraints.  In _EMNLP_ , pages 4057–4066.
  * Chen et al. (2019a) Wang Chen, Hou Pong Chan, Piji Li, Lidong Bing, and Irwin King. 2019a.  An integrated approach for keyphrase generation via exploring the power of retrieval and extraction.  In _Proceedings of the 2019 Conference of the NAACL, Volume 1_ , pages 2846–2856. ACL.
  * Chen et al. (2020) Wang Chen, Hou Pong Chan, Piji Li, and Irwin King. 2020.  Exclusive hierarchical decoding for deep keyphrase generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1095–1105, Online. Association for Computational Linguistics.
  * Chen et al. (2019b) Wang Chen, Yifan Gao, Jiani Zhang, Irwin King, and Michael R Lyu. 2019b.  Title-guided encoding for keyphrase generation.  In _AAAI_ , pages 6268–6275.
  * Cohan et al. (2018) Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018.  A discourse-aware attention model for abstractive summarization of long documents.  _In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2:615–621.
  * Cohan et al. (2020) Arman Cohan, Sergey Feldman, Iz Beltagy, Doug Downey, and Daniel S Weld. 2020.  Specter: Document-level representation learning using citation-informed transformers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2270–2282.
  * Das Gollapalli and Caragea (2014) Sujatha Das Gollapalli and Cornelia Caragea. 2014.  Extracting keyphrases from research papers using citation networks.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 28(1).
  * Diao et al. (2020) Shizhe Diao, Yan Song, and Tong Zhang. 2020.  Keyphrase generation with cross-document attention.  _arXiv preprint arXiv:2004.09800_.
  * Florescu and Caragea (2017) Corina Florescu and Cornelia Caragea. 2017.  PositionRank: An unsupervised approach to keyphrase extraction from scholarly documents.  In _Proceedings of the 55th ACL (Vol 1)_ , pages 1105–1115. ACL.
  * Gallina et al. (2019) Ygor Gallina, Florian Boudin, and Beatrice Daille. 2019.  KPTimes: A large-scale dataset for keyphrase generation on news documents.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 130–135, Tokyo, Japan. Association for Computational Linguistics.
  * Gollapalli and Caragea (2014) Sujatha Das Gollapalli and Cornelia Caragea. 2014.  Extracting keyphrases from research papers using citation networks.  In _AAAI_ , pages 1629–1635.
  * Hall et al. (2008) David Hall, Daniel Jurafsky, and Christopher D. Manning. 2008.  Studying the history of ideas using topic models.  In _EMNLP_ , pages 363–371.
  * Hammouda et al. (2005) Khaled M. Hammouda, Diego N. Matute, and Mohamed S. Kamel. 2005.  Corephrase: Keyphrase extraction for document clustering.  In _Proceedings of the 4th International Conference on Machine Learning and Data Mining in Pattern Recognition_ , MLDM’05, pages 265–274, Berlin, Heidelberg. Springer-Verlag.
  * Hulth (2003) Anette Hulth. 2003.  Improved automatic keyword extraction given more linguistic knowledge.  In _Proceedings of the 2003 conference on Empirical methods in natural language processing_ , pages 216–223.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with gpus.  _IEEE Transactions on Big Data_.
  * Kim et al. (2010) Su Nam Kim, Olena Medelyan, Min-Yen Kan, and Timothy Baldwin. 2010.  Semeval-2010 task 5: Automatic keyphrase extraction from scientific articles.  In _SemEval ’10_ , pages 21–26. ACL.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  _International Conference on Learning Representations (ICLR)_.
  * Krapivin et al. (2008) Mikalai Krapivin, Aliaksandr Autayeu, and Maurizio Marchese. 2008.  Large dataset for keyphrases extraction.  Technical Report DISI-09-055.
  * Liu et al. (2010) Zhiyuan Liu, Wenyi Huang, Yabin Zheng, and Maosong Sun. 2010.  Automatic keyphrase extraction via topic decomposition.  In _EMNLP ’10_ , pages 366–376. ACL.
  * Liu et al. (2009) Zhiyuan Liu, Peng Li, Yabin Zheng, and Maosong Sun. 2009.  Clustering to find exemplar terms for keyphrase extraction.  In _2009 EMNLP_.
  * Marujo et al. (2013) Luís Marujo, Anatole Gershman, Jaime Carbonell, Robert Frederking, and João P Neto. 2013.  Supervised topical key phrase extraction of news stories using crowdsourcing, light filtering and co-reference normalization.  _arXiv preprint arXiv:1306.4886_.
  * Medelyan et al. (2009) Olena Medelyan, Eibe Frank, and Ian H. Witten. 2009.  Human-competitive tagging using automatic keyphrase extraction.  In _EMNLP ’09_ , pages 1318–1327. ACL.
  * Meng et al. (2017) Rui Meng, Sanqiang Zhao, Shuguang Han, Daqing He, Peter Brusilovsky, and Yu Chi. 2017.  Deep keyphrase generation.  In _ACL_ , pages 582–592. ACL.
  * Mihalcea and Tarau (2004) Rada Mihalcea and Paul Tarau. 2004.  Textrank: Bringing order into text.  In _2004 EMNLP_.
  * Nguyen and Kan (2007) Thuy Dung Nguyen and Min-Yen Kan. 2007.  Keyphrase extraction in scientific publications.  In _International conference on Asian digital libraries_ , pages 317–326. Springer.
  * Patel and Caragea (2021) Krutarth Patel and Cornelia Caragea. 2021.  Exploiting position and contextual word embeddings for keyphrase extraction from scientific papers.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1585–1591, Online. Association for Computational Linguistics.
  * Qazvinian et al. (2010) Vahed Qazvinian, Dragomir R. Radev, and Arzucan Özgür. 2010.  Citation summarization through keyphrase extraction.  In _Proceedings of the 23rd International Conference on Computational Linguistics_ , COLING ’10, pages 895–903, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Ritchie et al. (2006) Anna Ritchie, Simone Teufel, and Stephen Robertson. 2006.  How to find better index terms through citations.  In _Proceedings of the Workshop on How Can Computational Linguistics Improve Information Retrieval?_ , CLIIR ’06, pages 25–32, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Santosh et al. (2021) TYSS Santosh, Debarshi Kumar Sanyal, Plaban Kumar Bhowmick, and Partha Pratim Das. 2021.  Gazetteer-guided keyphrase generation from research papers.  In _Pacific-Asia Conference on Knowledge Discovery and Data Mining_ , pages 655–667. Springer.
  * Shen et al. (2021) Xianjie Shen, Yinghan Wang, Rui Meng, and Jingbo Shang. 2021.  Unsupervised deep keyphrase generation.  _arXiv preprint arXiv:2104.08729_.
  * Song et al. (2006) I. Y. Song, R. B. Allen, Z. Obradovic, and M. Song. 2006.  Keyphrase extraction-based query expansion in digital libraries.  In _Proceedings of the 6th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL ’06)_ , pages 202–209.
  * Sterckx et al. (2016) Lucas Sterckx, Cornelia Caragea, Thomas Demeester, and Chris Develder. 2016.  Supervised keyphrase extraction as positive unlabeled learning.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1924–1929, Austin, Texas. Association for Computational Linguistics.
  * Sterckx et al. (2015) Lucas Sterckx, Thomas Demeester, Johannes Deleu, and Chris Develder. 2015.  Topical word importance for fast keyphrase extraction.  In _WWW’15_ , pages 121–122. ACM.
  * Swaminathan et al. (2020) Avinash Swaminathan, Haimin Zhang, Debanjan Mahata, Rakesh Gosangi, Rajiv Ratn Shah, and Amanda Stent. 2020.  A preliminary exploration of GANs for keyphrase generation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8021–8030, Online. Association for Computational Linguistics.
  * Viswanathan et al. (2021) Vijay Viswanathan, Graham Neubig, and Pengfei Liu. 2021.  Citationie: Leveraging the citation graph for scientific information extraction.  In _Joint Conference of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP)_ , Virtual.
  * Wan and Xiao (2008a) Xiaojun Wan and Jianguo Xiao. 2008a.  Single document keyphrase extraction using neighborhood knowledge.  In _23rd National Conference on AI_ , AAAI’08, pages 855–860. AAAI Press.
  * Wan and Xiao (2008b) Xiaojun Wan and Jianguo Xiao. 2008b.  Single document keyphrase extraction using neighborhood knowledge.  In _AAAI_ , volume 8, pages 855–860.
  * Xiong et al. (2019) Lee Xiong, Chuan Hu, Chenyan Xiong, Daniel Campos, and Arnold Overwijk. 2019.  Open domain web keyphrase extraction beyond language modeling.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5174–5183, Hong Kong, China. Association for Computational Linguistics.
  * Ye et al. (2021a) Jiacheng Ye, Ruijian Cai, Tao Gui, and Qi Zhang. 2021a.  Heterogeneous graph neural networks for keyphrase generation.  _arXiv preprint arXiv:2109.04703_.
  * Ye et al. (2021b) Jiacheng Ye, Tao Gui, Yichao Luo, Yige Xu, and Qi Zhang. 2021b.  One2set: Generating diverse keyphrases as a set.  In _Proceedings of ACL_.
  * Yuan et al. (2020) Xingdi Yuan, Tong Wang, Rui Meng, Khushboo Thaker, Peter Brusilovsky, Daqing He, and Adam Trischler. 2020.  One size does not fit all: Generating and evaluating variable number of keyphrases.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7961–7975, Online. Association for Computational Linguistics.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020.  Big bird: Transformers for longer sequences.  _Advances in Neural Information Processing Systems_ , 33:17283–17297.
  * Zheng and Lapata (2019) Hao Zheng and Mirella Lapata. 2019.  Sentence centrality revisited for unsupervised summarization.  _In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6236––6247.

##  Appendix A Hyperparamters

For the catSeq model, we use the same hyperparameters as Chan et al. (2019).
For ExHiRD and One2Set, we use the respective repositories with their
respective hyperparameters. For the LED model, we did a hyperparameter sweep
over the different learning rates $\in$ {1e-3, 1e-4, 1e-5, 2e-5, 5e-5, 8e-5}
and finally chose the learning rate as 5e-5, optimizer as Adam Kingma and Ba
(2015), train batch size as 12. For preprocessing the articles to generate
summary using PACSUM (TF-IDF), we use the default hyperparameters $\beta=0$,
$\lambda_{1}=0$, $\lambda_{2}=1$. We ran each experiment thrice with different
random seeds using V100 (16GB) and A6000 (48GB) GPU machines.

##  Appendix B Time and Memory Consumption

In Table 8, we provide the comparison of time and memory utilization between
the conventional (i.e., Title+Abstract) and new methods (i.e.,
Title+Abstract+Body) in reference to our best performing model (i.e., LED).
Body refers to sentences retrieved using any of the methods described in §3,
i.e., Random, Citation, Non-Citation, Summary, Retrieval-Augmentation.

The conventional method takes about 3.6 hours/ epoch and 10 GB of CUDA memory
with A6000 GPU for training. Whereas the newer methods take 5.5 hours/ epoch
and about 30 GB of CUDA memory for training. The inference time is less than 1
GPU hour for both types of methods.

##  Appendix C PacSum Algorithm

Methods | Time | CUDA Memory
---|---|---
Title+Abstract | 3.6 hrs/ epoch | 10 GB
+Body | 5.5 hrs/ epoch | 30 GB
|  |
Table 8:  Time and Memory consumption using the conventional method (T+A) and
the new methods (T+A+Body) where Body could be sentences retrieved using any
of the methods like Summary, Citation, Non-Citation, Random, Ret-Aug, with
total sequence length 800. Algorithm 1 Summarization: PacSum (TF-IDF)

Input $\Phi$: Document split into a list of sentences $(s_{i})_{i\in[1,N]}$
(tokenized into list of words); hyperparameters: $\beta$; $\lambda_{1}$;
$\lambda_{2}$
Output $\mathbb{S}$: Summary of document $\Phi$

1:$N\leftarrow\text{Number of sentences in document}$

2:$\text{IDF}\leftarrow\Big{[}\log(N-\text{count}(w,\Phi)+0.5)-\log(\text{count}(w,\Phi)+0.5)\textbar
w\in s_{i};s_{i}\in\Phi\Big{]}$

3:$\text{TF}\leftarrow\Big{[}\big{[}\text{count}(w,s_{i})\textbar w\in
s_{i}\big{]}\textbar s_{i}\in\Phi\Big{]}$

4:$\forall i,j\in[1,N],\text{sim}(i,j)\leftarrow\newline
\left\\{\begin{array}[]{cl}\mathbf{1}&\mbox{i == j}\\\ \underset{w\in
s_{i}\&s_{j}}{\sum}\Big{(}\text{TF}[s_{i}][w]*\text{TF}[s_{j}][w]*\text{IDF}[w]\Big{)}^{2}&\mbox{i
!= j}\end{array}\right.$

5:$\tau=\min(\text{sim})+\beta*(\max(\text{sim})-\text{min}(\text{sim}))$

6:$\forall i,j\in[1,N],\text{sim}(i,j)=\text{sim}(i,j)-\tau$

7:$\underset{\forall
i\in[1,N]}{\text{FS}(i)}\leftarrow\left\\{\begin{array}[]{cl}\underset{\forall
j,j<i}{-\sum}\text{sim}(i,j)&\text{sim}(i,j)>\tau\\\
0&\text{otherwise}\end{array}\right.$

8:$\underset{\forall
i\in[1,N]}{\text{BS}(i)}\leftarrow\left\\{\begin{array}[]{cl}\underset{\forall
j,j>i}{\sum}\text{sim}(i,j)&\text{sim}(i,j)>\tau\\\
0&\text{otherwise}\end{array}\right.$

9:$\text{centrality}(i)\leftarrow\lambda_{1}*\text{FS}(i)+\lambda_{2}*\text{BS}(i),\forall
i\in[1,N]$

10:$\text{indices}\leftarrow\text{argsort}(\text{centrality, desc=True})$

11:$\mathbb{S}\leftarrow\Phi[\text{indices}]$

12:$\text{Return }\mathbb{S}$

##  Appendix D Retrieval-Augmentation Algorithm

Algorithm 2 Retrieval-Augmentation

Input Collection of scientific articles $\mathbb{C}$
Output Collection of scientific articles $\mathbb{D}$ with Title $\oplus$
Abstract $\oplus$ Semantically similar sentences

1:$\Phi\leftarrow\underset{a\in\mathbb{C}_{\text{train}}}{\bigcup}\\{\underset{s_{j}\in(T_{a}\oplus
A_{a})}{\bigcup}s_{j}\\}$ $\triangleright$ $T_{a}$: Title, $A_{a}$: Abstract,
$\Phi$: Set of sentences from titles and abstracts of all articles in training
dataset

2:$\mathbb{D}\leftarrow\varnothing$

3:for all $a\in\mathbb{C}$ do

4: $\Phi_{q}\leftarrow\underset{s_{j}\in(T_{a}\oplus A_{a})}{\bigcup}s_{j}$

5: $\Phi_{\text{key}}\leftarrow\Phi-\Phi_{q}$

6:
$E_{\text{key}}\leftarrow\big{(}\text{embed}(s_{i})\big{)}_{s_{i}\in\Phi_{\text{key}}}$

7: $\vec{e}_{q}\leftarrow\text{embed}(T_{a}\oplus A_{a})$

8:
$\text{similarity}\leftarrow\big{(}\vec{e}_{\text{key}}\cdot\vec{e}_{q}\big{)}_{\vec{e}_{\text{key}}\in
E_{\text{key}}}$

9: $\text{indices}\leftarrow\mathrm{argmax_{k}}(\text{similarity})$

10:
$\text{extra\\_info}\leftarrow\big{(}\Phi_{\text{key}}[i]\big{)}_{i\in\text{indices}}$

11: $a\leftarrow T_{a}\oplus A_{a}\oplus\text{extra\\_info}$

12: $\mathbb{D}\leftarrow\mathbb{D}\cup\\{a\\}$

13:end for

14:$\text{Return }\mathbb{D}$