# A study of audio mixing methods for piano transcription in violin-piano
ensembles

###### Abstract

While piano music transcription models have shown high performance for solo
piano recordings, their performance degrades when applied to ensemble
recordings. This study aims to analyze the impact of different data
augmentation methods on piano transcription performance, specifically focusing
on mixing techniques applied to violin-piano ensembles. We apply mixing
methods that consider both harmonic and temporal characteristics of the audio.
To create datasets for this study, we generated the PFVN-synth dataset, which
contains 7 hours of violin-piano ensemble audio by rendering MIDI files and
corresponding labels, and also collected unaccompanied violin recordings and
mixed them with the MAESTRO dataset. We evaluated the transcription results on
both synthesized and real audio recordings datasets.

Index Terms—  Piano transcription, dataset, mixing audios, violin-piano
ensemble

##  1 Introduction

Automatic music transcription models have shown significant improvement in
performance for single-instrument recordings, especially for piano, thanks to
the development of carefully designed neural networks [1, 2, 3, 4] and the
availability of large-scale datasets such as the MAESTRO dataset [5]. However,
in multi-instrument settings, such as when the piano accompanies melodic
instruments like the violin, cello, or flute, note detection for the piano
becomes more challenging due to other instrumental sounds causing noise
interference [6, 7]. This challenge has been addressed using multi-instrument
datasets, such as MusicNet [8] or synthesized Lakh dataset (Slakh) [9].
However, obtaining precise MIDI labels from mixed audio is an extremely
difficult task. To address this issue, researchers have either aligned music
scores to audio as noisy labels [8] or synthesized the audio from MIDI files
[9].

Music source separation also faces a similar challenge where expensive multi-
track audio recordings are required to train models. To overcome this problem,
researchers have used various audio mixing methods as data augmentation during
training. Among these methods, random mixing of audio segments has been the
most commonly used. It involves incoherent mixing of audio segments from
different songs at different positions [9] or using two segments from the same
source as an augmented target source [10, 11]. Some studies have introduced
more organized random mixing based on acoustic knowledge, such as matching
audio segments with chromagrams, assuming that mixing two audio segments with
similar pitch content simulates real ensembles better than random mixing [12,
13]. However, these audio mixing methods have not been extensively studied for
automatic music transcription.

This paper presents the development of a piano transcription model for violin-
piano ensembles. The model transcribes the piano part only, as an extension of
the transcription model previously developed for solo piano recordings [1]. To
achieve this goal, we analyze various audio mixing methods for the two
instrumental sounds in violin-piano ensembles. Unlike sound source separation,
which mainly uses frame-level metrics, note onsets are particularly important
in piano music transcription. Additionally, notes of the two instruments in
ensembles are arranged tonally in Western classical music, which we use in
this work. Based on these concepts, we propose an audio mixing approach that
considers harmonic and temporal characteristics and compare it with random
mixing. We also created a new dataset called PFVN-synth, containing 7 hours of
violin-piano ensemble audio annotated with piano and violin MIDI labels. In
addition, we collected unaccompanied violin recordings and mixed them with the
MAESTRO piano dataset. As far as we know, this study is the first to explore
data augmentation for piano transcription in ensemble music using harmonically
and temporally controlled mixing methods.

Fig. 1: An example of a major scale and a harmonic minor scale for the key-
based mixing method. When the piano excerpt is in E key (red boxes), the
violin excerpt in B, A, or Am (blue boxes) including E is selected.

##  2 Method for audio mixing

###  2.1 PFVN-synth Dataset

The PFVN-synth dataset was created to contain realistic instrumental triplet
audio recordings of the piano, violin, and their mixture111PFVN-synth is
available at https://zenodo.org/record/7703620. This was achieved by rendering
MIDI files with virtual instruments using musical scores of 45 different
pieces by 23 classical composers, with a total duration of 7 hours222We used
commercial virtual instruments which are ‘Bösendorfer Grand Piano’ in Apple
Logic Pro and ‘SWAM Violin V3’ by Audio Modeling.. All MIDI files were
collected from the MuseScore website, which provides not only sheet music but
also corresponding MIDI files with dynamics and sustain pedals. All tracks
were rendered into monaural audio files with the standard CD quality: 44.1kHz,
16-bit. The dataset includes recordings ranging from 58 seconds to almost 40
minutes in length. To ensure that composers are not biased to the split sets,
it was divided into a train set containing 32 pieces with a duration of 5.8
hours, a validation set containing 3 pieces with a duration of 30 minutes, and
a test set containing 10 pieces with a duration of 50 minutes.

###  2.2 Real Performance Datasets

We also trained the model with real performance recordings by artificially
mixing violin and piano recordings from different sources. We collected
unaccompanied violin pieces from the solo violin
list333https://en.wikipedia.org/wiki/List_of_solo_violin_pieces. To ensure
diversity across eras, we selected recordings from 7 composers and downloaded
them from YouTube. The total duration of the collected recordings is 4.6
hours. We then mixed the solo violin audio with piano audio from the MAESTRO
dataset using various mixing methods, which are introduced in the following
subsections. As a test set, we used a subset of the MusicNetEM dataset [14] by
selecting violin-piano ensemble pieces. This dataset has refined labels of the
original MusicNet dataset [8]. We excluded 7 noisy pieces and used a total of
14 pieces for testing, out of the 21 accompanied violin pieces in the dataset.

###  2.3 Loudness Control

The envelope of piano sounds exhibits a sudden increase of energy followed by
a gradual decrease [15], whereas the envelope of violin sounds features a
relatively soft increase followed by sustained energy. To mix the two
instrument audios, we set the volume ratio based on the amplitude envelope.
Specifically, we set the ratio of the RMS values of the piano and violin to
0.5 for the test and between 0.3 and 1.2 during the training. This ensured
that the violin sound was not too loud and that the piano sound was not much
louder than the violin. We also set the peak value of the mixed audio not to
exceed 0.99.

Fig. 2: An illustrated diagram of the onset-based mixing method. Since the
MIDI file is not always available, the onsets are estimated and matched from
the audio.

###  2.4 Musically Controlled Mixing

We used the random mixing technique as our baseline audio mixing method. In
addition, we suggest two musically controlled mixing by matching tonal and
temporal characteristics between the piano and violin audio excerpts.

| Frame | Note with onset | Note with offset | Note w. offset & vel
---|---|---|---|---
Mixing method | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Lower bound | 82.0 | 51.7 | 62.7 | 78.3 | 62.2 | 68.5 | 51.9 | 41.0 | 45.3 | 40.8 | 32.6 | 35.7
Original pair | 90.8 | 81.8 | 85.7 | 99.1 | 96.8 | 97.9 | 72.6 | 71.0 | 71.8 | 71.4 | 69.8 | 70.6
Random mixing | 95.4 | 86.6 | 90.6 | 99.2 | 95.5 | 97.2 | 80.2 | 77.2 | 78.6 | 78.9 | 75.9 | 77.3
Onset-based mixing | 96.0 | 88.9 | 92.1 | 99.3 | 97.1 | 98.2 | 82.3 | 80.5 | 81.3 | 80.8 | 79.0 | 79.9
Key-based mixing | 96.2 | 88.1 | 91.8 | 99.4 | 97.2 | 98.2 | 81.7 | 79.9 | 80.7 | 80.2 | 78.5 | 79.3
Key and onset-based mixing | 95.5 | 87.9 | 91.4 | 99.3 | 96.1 | 97.6 | 80.4 | 77.8 | 79.1 | 79.2 | 76.7 | 77.9
Upper bound | 97.5 | 90.9 | 94.0 | 99.2 | 97.8 | 98.5 | 86.3 | 85.1 | 85.7 | 85.3 | 84.2 | 84.7

Table 1: Transcription results from training on the PFVN-synth train split and
evaluating on the test split.

####  2.4.1 Key-based Mixing

To ensure that the harmonic characteristics of the mixed piano and violin
audios were similar, we mixed two audio excerpts that had similar tonality. We
estimated the tonal key using the madmom library [16]444We used
madmom.features.key to estimate the key.. To achieve this, we filtered the
violin excerpts to ensure that they were in the same key as a given piano
excerpt. We expanded the range of valid key combinations by adding the
dominant key and subdominant key as harmonically-related keys. For example, in
the case of a piano excerpt in the E key, violin excerpts in the E, B, A, and
Am keys were sampled, as shown in Figure 1. In the minor mode, the subdominant
key of Em is also Am, so excerpts in keys such as Am, E, Dm, and Em were
selected to mix with the target in Am. By using these methods, we ensured that
the randomly-selected two excerpts had similar tonality, which helped to
create more realistic mixtures of piano and violin audios.

####  2.4.2 Onset-based Mixing

We also mixed the piano and violin audio excerpts in a way that aligned their
note onsets as closely as possible. To achieve this, we shifted the entire
piano clip by adjusting the global offset in the unit of time frame, as shown
in Figure 2. We aimed to find the optimal offset value that resulted in the
highest degree of onset overlap between the two recordings by shifting the
piano audio by one frame at a time, regardless of the tempo in the two audio
excerpts. Then, we mixed the two audio excerpts at the optimal position. For
synthesized audio, we used the corresponding MIDI files to find the note onset
positions. For real audio, on the other hand, we estimated onsets using
_librosa_ [17]555We used librosa.onset.onset_detect to detect onsets.. By
aligning the note onsets in this way, we improved the realism and accuracy of
our mixed audio excerpts.

##  3 Experiments

###  3.1 Experiment Settings

We conducted experiments to compare the performance of various audio mixing
methods using _Onsets and Frames_ , a state-of-the-art piano transcription
model [1]. We based our implementation and hyperparameters on the source code
available at this link666https://github.com/jongwook/onsets-and-frames. We
used a batch size of 16, and the maximum number of training iterations was set
to 150,000 for the PFVN-synth train split and 500,000 for the real audio
datasets. During validation, we selected the model with the highest note
F1-score for the test. We used the standard frame-based and note-based metrics
provided by mir_eval [18]. To apply the onset-based mixing method, we set the
length of the piano excerpt to about 20 seconds, and the violin excerpt was
5.5 seconds longer than the piano audio.

In our experiments, we compared up to five audio mixing methods: 1) random
mixing, 2) onset-based mixing, 3) key-based mixing, 4) key and onset-based
mixing, and 5) the original mixing (only for PFVN-synth). We also compared the
results to the lower-bound case, where we evaluated violin-piano ensembles
using the model trained with piano only, and the upper-bound case, where we
evaluated piano audio alone using the same model. By comparing the performance
of these methods, we were able to identify the most effective approaches for
piano transcription in ensemble music.

##  4 Results and Discussion

###  4.1 Models Trained with the PFVN-synth Dataset

Table 1 presents the piano transcription results using five different audio
mixing methods. The lower-bound case, where the model is trained with piano
only and tested with violin-piano ensembles, achieved a note F1-score of
68.5%. After incorporating additional violin audio, note precision and recall
increased significantly to around 99% and 95%, respectively. Onset-based
mixing and key-based mixing produced the highest note F1-score of 98.2%, which
is close to the upper bound score of 98.5%. In particular, onset-based mixing
outperformed all four metrics of frame-level, note onset, note with offset,
and note with offset and velocity, exhibiting the best overall performance.
Key-based mixing, key and onset-based mixing, and random mixing followed
onset-based mixing in descending order of performance. Data augmentation with
random mixing failed to improve note onset F1-score, although it improved the
scores of six metrics with note offset when compared to mixing with the
original pair.

| Frame | Note with onset | Note with offset | Note w. offset & vel
---|---|---|---|---
Mixing method | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
PFVN-synth
Lower bound | 71.8 | 62.5 | 66.2 | 63.5 | 75.0 | 68.4 | 36.4 | 43.1 | 39.2 | 26.3 | 31.1 | 28.3
Random mixing | 82.3 | 75.4 | 78.4 | 92.5 | 81.5 | 86.4 | 57.6 | 50.7 | 53.9 | 45.6 | 40.2 | 42.6
Onset-based mixing | 83.4 | 73.9 | 78.1 | 89.8 | 81.1 | 85.0 | 57.6 | 52.0 | 54.5 | 45.1 | 40.9 | 42.7
Key-based mixing | 85.2 | 75.1 | 79.6 | 92.3 | 83.1 | 87.3 | 59.0 | 53.0 | 55.7 | 46.5 | 41.9 | 44.0
Key and Onset-based mixing | 82.9 | 74.5 | 78.2 | 90.2 | 81.0 | 85.1 | 57.0 | 51.1 | 53.8 | 44.6 | 40.2 | 42.2
Upper bound | 78.7 | 87.7 | 82.7 | 89.3 | 93.0 | 91.0 | 56.1 | 58.6 | 57.3 | 43.9 | 45.9 | 44.8
MusicNetEM
Lower bound | 75.1 | 50.3 | 59.7 | 71.8 | 80.8 | 75.9 | 28.9 | 32.2 | 30.4 | - | - | -
Random mixing | 78.0 | 58.2 | 65.8 | 87.7 | 78.4 | 82.7 | 36.7 | 32.9 | 34.7 | - | - | -
Onset-based mixing | 78.9 | 57.4 | 65.6 | 87.4 | 78.4 | 82.4 | 37.1 | 33.4 | 35.0 | - | - | -
Key-based mixing | 79.2 | 56.8 | 65.5 | 87.7 | 79.4 | 83.2 | 37.1 | 33.7 | 35.3 | - | - | -
Key and Onset-based mixing | 79.2 | 56.6 | 65.1 | 87.1 | 79.1 | 82.8 | 37.0 | 33.8 | 35.3 | - | - | -

Table 2: Transcription results from training on the artificially mixed real
performance dataset and evaluating on PFVN-synth and MusicNetEM.

###  4.2 Models Trained with the Real Performance Dataset

Table 2 presents the piano transcription results on four different audio
mixing methods using two test datasets. The upper part of the table shows the
results on the entire PFVN-synth dataset, including the training, validation,
and test sets. After training with additional unaccompanied violin recordings,
note precision significantly increases from 63.5% to about 90%, while note
recall mildly increases from 75.0% to over 81%. The lower bound of the note
F1-score is 68.4%, and the upper bound is 91.0%. All four mixing methods
achieve more than 85% of note F1-score. In this real performance case, the
key-based mixing method achieves the highest F1-score of frame-level, note
onset, note with offset, and note with offset and velocity. Onset-based mixing
methods have relatively lower performance.

The lower part of Table 2 shows the piano transcription results on the violin-
piano ensemble pieces from MusicNetEM. After training with additional
unaccompanied violin recordings, note precision greatly increases from 71.8%
to over 87%, while note recall slightly decreases. The key-based mixing method
achieves the highest note F1-score of 83.2% and the note with offset F1-score
of 35.3%. The key and onset-based mixing method achieves the best note with
offset F1-score along with the key-based mixing method.

###  4.3 Analysis

Based on the results of Tables 1 and 2, we observe that the performance of
different audio mixing methods depends on the dataset used. The onset-based
mixing method performs best on the PFVN-synth dataset while the key-based
mixing method performs best on the real performance dataset. To investigate
this difference, we counted the number of overlapping note onsets in the two
datasets. Figure 3 shows that there are significantly more onset matches in
the MAESTRO and unaccompanied violin recordings datasets compared to PFVN-
synth, with two and a half to four times more matches. This can be attributed
to the fact that the MAESTRO dataset was recorded at a piano competition with
a higher number of onsets, and that unaccompanied violin recordings have more
onset frames than violin onsets in PFVN-synth. We hypothesize that when the
number of onsets is already high, the process of matching onsets does not
significantly improve performance.

Fig. 3:  The number of overlapping note onsets between piano and violin audios
in the two datasets.

##  5 Conclusions

We conducted experiments to evaluate the performance of various audio mixing
methods on piano transcription in violin-piano ensembles. We created a new
synthesized dataset and collected unaccompanied violin recordings along with
the MAESTRO dataset. The results indicate that the onset-based mixing method
performed the best in our synthesized dataset, while the key-based mixing
method was the most effective when the model was trained on real performance
datasets. This suggests that musically-informed mixing is more effective than
random mixing. In the future, we plan to collect real violin-piano ensemble
recordings and further investigate mixing methods.

## References

  * [1] C. Hawthorne, E. Elsen, J. Song, A. Roberts, I. Simon, C. Raffel, J. Engel, S. Oore, and D. Eck,  “Onsets and frames: Dual-objective piano transcription,”  in Proceedings of the 19th International Society for Music Information Retrieval Conference, Paris, France, 2018, pp. 50–57.
  * [2] Q. Kong, B. Li, X. Song, Y. Wan, and Y. Wang,  “High-resolution piano transcription with pedals by regressing onset and offset times,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3707–3717, 2021.
  * [3] J.W. Kim and J. P. Bello,  “Adversarial learning for improved onsets and frames music transcription,”  in Proceedings of the 21st International Society for Music Information Retrieval Conference, 2019, pp. 670–677.
  * [4] T. Kwon, D. Jeong, and J. Nam,  “Polyphonic piano transcription using autoregressive multi-state note model,”  in Proceedings of the 21st International Society for Music Information Retrieval Conference, 2020, pp. 454–461.
  * [5] C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C. Huang, S. Dieleman, E. Elsen, J. Engel, and D. Eck,  “Enabling factorized piano music modeling and generation with the maestro dataset,”  in Proceedings of the 7th International Conference on Learning Representations, 2019.
  * [6] J. Gardner, I. Simon, E. Manilow, C. Hawthorne, and J. Engel,  “MT3: Multi-task multitrack music transcription,”  in International Conference on Learning Representations, 2022.
  * [7] Y.-T Wu, B. Chen, and L. Su,  “Multi-instrument automatic music transcription with self-attention-based instance segmentation,”  IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 28, pp. 2796–2809, 2020.
  * [8] J. Thickstun, Z. Harchaoui, and S. Kakade,  “Learning features of music from scratch,”  in International Conference on Learning Representations, 2017.
  * [9] E. Manilow, G. Wichern, P. Seetharaman, and J. Roux,  “Cutting music source separation some Slakh: A dataset to study the impact of training data quality and quantity,”  in Proceedings of the IEEE Workshop on Applications of Signal Processing to Audio and Acoustics (WASPAA). IEEE, 2019.
  * [10] Q. Kong, Y. Cao, H. Liu, K. Choi, and Y. Wang,  “Decoupling magnitude and phase estimation with deep resunet for music source separation,”  in Proceedings of the 22nd International Society for Music Information Retrieval Conference, 2021, pp. 342–349.
  * [11] X. Song, Q. Kong, X. Du, and Y. Wang,  “Catnet: Music source separation system with mix-audio augmentation,”  arXiv preprint arXiv:2102.09966, 2021.
  * [12] C. Y. Chiu, W. Y. Hsiao, Y. C. Yeh, Y. H. Yang, and A. W. Y. Su,  “Mixing-specific data augmentation techniques for improved blind violin/piano source separation,”  in IEEE 22nd International Workshop on Multimedia Signal Processing, 2020, pp. 1–6.
  * [13] S. Yuan, Z. Wang, U. Isik, R. Giri, J.-M. Valin, M. M Goodwin, and A. Krishnaswamy,  “Improved singing voice separation with chromagram-based pitch-aware remixing,”  in Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. IEEE, 2022, pp. 111–115.
  * [14] B. Maman and A. H Bermano,  “Unaligned supervision for automatic music transcription in the wild,”  in Proceedings of the 39th International Conference on Machine Learning, 2022, pp. 14918–14934.
  * [15] M. Müller,  Fundamentals of Music Processing Audio, Analysis, Algorithms, Applications,  Springer, 2015.
  * [16] S. Böck, F. Korzeniowski, J. Schlüter, F. Krebs, and G. Widmer,  “madmom: a new python audio and music signal processing library,”  in Proceedings of the 24th ACM International Conference on Multimedia, Amsterdam, Netherlands, 2016, pp. 1174–1178.
  * [17] B. McFee, C. Raffel, D. Liang, D. P Ellis, M. McVicar, E. Battenberg, and O. Nieto,  “librosa: Audio and music signal analysis in python,”  in Proceedings of the 14th python in science conference, 2015, pp. 18–25.
  * [18] C. Raffel, B. McFee, E. J. Humphrey, J. Salamon, O. Nieto, D. Liang, and D. PW Ellis,  “mir_eval: A transparent implementation of common MIR metrics,”  in Proceedings of the 15th International Society for Music Information Retrieval Conference, 2014, pp. 367–372.


11institutetext:  University of Tyumen, Tyumen, Russia
11email<EMAIL_ADDRESS>
22institutetext: Novosibirsk State University, Novosibirsk, Russia
22email<EMAIL_ADDRESS>
# Applying Transformer-based Text Summarization for Keyphrase
Generation††thanks: Supported by the grant of the President of the Russian
Federation no. MK-3118.2022.4.

Anna Glazkova 11 0000-0001-8409-6457 Dmitry Morozov 22 0000-0003-4464-1355

###### Abstract

Keyphrases are crucial for searching and systematizing scholarly documents.
Most current methods for keyphrase extraction are aimed at the extraction of
the most significant words in the text. But in practice, the list of
keyphrases often includes words that do not appear in the text explicitly. In
this case, the list of keyphrases represents an abstractive summary of the
source text. In this paper, we experiment with popular transformer-based
models for abstractive text summarization using four benchmark datasets for
keyphrase extraction. We compare the results obtained with the results of
common unsupervised and supervised methods for keyphrase extraction. Our
evaluation shows that summarization models are quite effective in generating
keyphrases in the terms of the full-match F1-score and BERTScore. However,
they produce a lot of words that are absent in the author’s list of
keyphrases, which makes summarization models ineffective in terms of ROUGE-1.
We also investigate several ordering strategies to concatenate target
keyphrases. The results showed that the choice of strategy affects the
performance of keyphrase generation.

###### Keywords:

Keyphrase extraction Scholarly document Natural language processing Text
summarization Transformer BART T5.

##  1 Introduction

A list of keyphrases is an important attribute of a scientific text.
Keyphrases contain a brief representation of the contents of a text. They help
search engines find and systematize papers. A qualitative selection of
keyphrases positively affects a paper’s visibility and its number of citations
[2, 17].

Many of the current approaches to keyphrase extraction involve selecting words
from the source text, ranking the candidates, and choosing the top $N$. The
value of $N$ is determined by the user. The methods of directly extracting
keyphrases from the text will produce only those phrases that are explicitly
contained in the text. But in practice, keyphrases often represent an
abstractive summary of the text. This summary can include hypernyms and
paraphrased sentences from the source text. Therefore, studying the
applicability of abstractive text summarization methods is a major area of
interest within the field of generating multiple keyphrases as a sequence.

Compared with traditional methods for keyphrase extraction, the approaches
based on abstractive text summarization have the following properties: a) $N$
is a value determined by the model and is not an input parameter; b) the model
takes into account both semantic and syntactic components of the source text;
c) not only those words or phrases that are found in the source text can be
proposed as keyphrases. To date, few studies have investigated generating
keywords using text summarization [9, 35]. However, the performance of state-
of-the-art models based on the transformer architecture has not been closely
examined for the task of keyphrase extraction. Moreover, there have been no
studies that compare the effectiveness of different ordering strategies for
concatenating target keyphrases.

In this paper, we aim to fill this research gap by systematically evaluating
transformer-based abstractive text summarization models on several keyphrase
extraction benchmarks. We seek to answer the following research questions:

  * •

RQ1: Do transformer-based models for abstractive summarization outperform
other baselines?

  * •

RQ2: What is the effect of different ordering strategies for concatenating
target keyphrases?

The paper is organized as follows. Section 2 contains a brief review of
related works. Next, we describe our methods and experiments in Section 3. In
Section 4, we discuss the results. Finally, Section 5 concludes this paper.

##  2 Related Work

The aim of keyphrase extraction is to define a set of phrases that are related
to the main topics discussed in a given document [15]. Up to now, there have
been a large volume of published studies presenting unsupervised and
supervised approaches to keyphrase extraction.

Unsupervised approaches basically consist of the following stages: selecting
candidate words or n-grams in accordance with some characteristics; ranking
the candidate words; formatting the keyphrases by selecting the top-ranked
words [29]. Unsupervised keyphrase extraction is mainly performed by
statistical and graph-based methods. Statistical methods, such as TFIDF,
KPMiner [12], and YAKE! [8], utilize textual statistical features to define
the most important words in the text. The general idea of graph-based methods
is to create a document graph consisting of candidate phrases as nodes and
their relations as edges. Well-known examples of graph-based methods are
TextRank [27], TopicRank [6], and PositionRank [13].

Supervised methods for keyphrase extraction include methods based on
traditional supervised algorithms as well as deep learning methods. In terms
of machine learning, the words in a document are “examples” and the objective
of the keyphrase extraction system is to divide the examples into “keyphrases”
and “not-keyphrases” [1]. One of the common keyphrase extraction systems is
KEA [38] which identifies candidate keyphrases using lexical methods,
calculates feature values for each candidate, and uses a Naïve Bayes
classifier to predict the most probable keyphrases. Binary classification
models for keyphrase extraction were also proposed in [24, 28, 36]. Zhang et
al. [40] proposed a deep recurrent neural network model to combine keyphrases
and context information that jointly process the keyphrase ranking and
keyphrase generation task. Meng et al. [26] presented a generative model for
keyphrase prediction with an encoder-decoder framework (CopyRNN). Wang et al.
[37] proposed a topic-based adversarial neural network (TANN), which uses the
idea of transfer learning.

To date, the models based on transformer-based models, such as Bidirectional
Encoder Representations from Transformers (BERT) [11], show state-of-the-art
results in many natural language processing tasks. Several studies have
investigated the use of BERT-based language models for keyphrase extraction.
Thus, KeyBERT [14] calculates cosine similarity between BERT-embeddings to
find the sub-phrases more fully reflecting the content of the document. Some
researchers attempted to fine-tune BERT-based models for keyphrase extraction
as a classification or sequence labelling task [21, 32].

A number of studies have examined neural models to generate multiple
keyphrases as a sequence [9, 35]. Chowdhury et al. [10] showed that fine-tuned
BART [20] can show competitive results in keyphrase generation compared with
the existing neural models for keyphrase extraction. The authors ranked the
produced keyphrases, selected a fixed number of generated keyphrases per
source text, and separately calculated an F1-score for keyphrases present in
the source text and Recall for keyphrases absent in the text. Meng et al. [25]
explored the effect of different strategies for concatenating target
keyphrases with the example of the One2Seq model. They showed that the
ordering of concatenating target phrases matters. The best results were
achieved where target keyphrases were sorted by their first occurrences in the
source text.

##  3 Experimental Setup

###  3.1 Datasets

Characteristic | Krapivin-A | Krapivin-T | KP20K | Inspec | SemEval2017
---|---|---|---|---|---
Size | 2294 | 2293 | 20000 | 2000 | 500
Domains | CS | CS | CS | CS |  | CS,
---
material science,
physics
Type of texts | abstracts | texts | abstracts | abstracts | paragraphs
Avg symbols per text | 1001.74 | 43807.85 | 995.85 | 777.25 | 1113.66
STD | 381.37 | 12565.47 | 451.32 | 392.69 | 310.45
Avg tokens per text | 169.06 | 8597.63 | 165.72 | 127.35 | 194.99
STD | 68.58 | 2411.77 | 76.89 | 65.03 | 58.14
Avg keyphrases per text | 5.34 | 5.34 | 5.28 | 14.11 | 17.3
STD | 2.77 | 2.77 | 3.77 | 6.41 | 7
Absent keyphrases, % | 51.3 | 18.04 | 43.67 | 43.8 | 0
STD | 25.99 | 19.69 | 28.38 | 17.83 | 0
Table 1:  Data statistics. CS - Computer Science. The number of tokens is
obtained using NLTK [4]. STD means the standard deviation for the
correspondence characteristic.

We use several English corpora of scientific text for evaluating automatic
keyphrase extraction methods. The main characteristics of the datasets are
demonstrated in Table 1. The percentage of absent keyphrases means the
proportion of keyphrases from the list of keyphrases that do not appear in the
text.

  * •

Krapivin2009 [19], a dataset that contains full papers divided into title,
abstract, and text. In this work, we extract keyphrases using abstracts and
texts separately (Krapivin-A and Krapivin-T respectively).

  * •

Inspec [16], a dataset for keyphrase extraction from scientific abstracts.

  * •

KP20K [26], a large-scale scholarly abstracts corpus with 528K abstracts for
training, 20K abstracts for validation and 20K abstracts for testing. In this
work, we limit ourselves to utilizing only the test set.

  * •

SemEval2017 [3], a corpus that consists of the paragraphs manually selected
from scientific papers among several domains.

###  3.2 Text Summarization Models

For each dataset, we fine-tune two pretrained language models for abstractive
text summarization (Figure 1):

  * •

BART [20], a transformer-based denoising autoencoder for pre-training a
seq2seq model. We use BART-base, which has 139 M parameters, 12 layers, and
the hidden size of 768;

  * •

T5 [30], an encoder-decoder model pre-trained on a multi-task mixture of
unsupervised and supervised tasks and for which each task is converted into a
text-to-text format. We utilize T5-small, which has 60M parameters, 12 layers,
and the hidden size of 512.

We do not fix the number of keyphrases to be generated and thereby this value
is determined by the model. This approach can be useful when the number of
keyphrases in the training set differs significantly depending on the topic
and type of the source text. The other advantage is that the model can
independently choose the optimal number of keyphrases based on the training
set. However, the model trained in such a manner is not able to generate the
exact number of keywords specified during the evaluation stage.

Figure 1: Keyphrase generation with pretrained language models.

###  3.3 Ordering for Concatenating Keyphrases

We define eight strategies for concatenating target keyphrases, the first six
of which were presented in [25]: a) No-Sort, i.e. keyphrases in their original
order; b) Random, in random order; c) Length, keyphrases sorted by their
length in terms of symbols; d) Alpha, in alphabetical order; e) Appear-Pre,
sorted by their first occurrences, absent keyphrases are randomly shuffled and
added at the beginning; f) Appear-Post, same as the previous one, but absent
keyphrases are added at the end.

In addition to these strategies, we consider the following ways to concatenate
keyphrases:

  * •

Appear-Pre-CC, same as Appear-Pre but absent and present keyphrases are marked
with control codes <absent> and <present> respectively, for example:
$\textbf{<absent>},keyphrase_{1},keyphrase_{2},...,keyphrase_{i-1},\textbf{<present>},\\\
keyphrase_{i},...,keyphrase_{N}$, where $N$ is a number of keyphrases. The
idea of the use of control codes was introduced in [18]. Control codes
represent certain words, sentences, or links which are added to the input of
the text generation model to generate coherent outputs. For example, control
codes were used for the joined generating titles and short summaries (TLDRs)
for scientific papers in [7]. The authors appended each source with control
codes for short summary and title generation, respectively. This allowed the
parameters of the model to learn to generate TLDR or title depending on the
control code. In our work, we utilize control codes to distinguish between
absent and present keyphrases.

  * •

Appear-Post-CC, same as Appear-Post but using control codes.

The presented strategies were used during the training phase, i.e. before
starting the model training.

###  3.4 Baselines

To evaluate the performance of transformer-based text summarization models for
keyphrase generation, we compare them with the following baselines:

  * •

TFIDF, a method based on statistical frequencies that expresses the importance
of a word in a particular document of a corpus.

  * •

TopicRank [6], an unsupervised extractive method that extracts the noun
phrases and creates top-ranked topic clusters and uses the extracted noun
phrases as vertices in a complete graph.

  * •

YAKE! [8], an unsupervised multilingual method that utilizes various features,
such as term position, term frequency, and others.

  * •

KEA [38], a supervised method that extracts candidate keyphrases on the basis
of several linguistic features and uses the Naïve Bayes algorithm to classify
candidate phrases into keyphrases and not.

  * •

KeyBERT [14], a method that utilizes document and word embeddings from BERT
and cosine similarity to find the sub-phrases in a document that are the most
similar to the document itself.

###  3.5 Evaluation Metrics

Since previous studies highlighted that different metrics are necessary to
capture different aspects of text generation [33, 34], we adopt three diverse
metrics to evaluate the performance of the selected transformer-based text
summarization models and baselines: ROUGE-1 [22], full-match F1-score, and
BERTScore [41]. For those methods that return the predefined number of
keyphrases, we computed metrics on the top 5, top 10, and top 15 returned
keyphrases.

The ROUGE-1 score calculates the number of matching unigrams between the
model-generated text and the reference. To measure ROUGE-1, the keyphrases for
each text were combined into a string with a comma as a separator.

The full-match F1-score evaluates the number of exact matches between the
original and generated sets of keyphrases. It is calculated as a harmonic mean
of precision and recall.

BERTScore utilizes the pre-trained contextual embeddings from BERT-based
models and matches words in the source and generated texts using cosine
similarity. It has been shown that human judgment correlates with this metric
on sentence-level and system-level evaluation. To calculate BERTScore, we use
contextual embeddings from RoBERTa-large [23], a modification of BERT that is
pretrained using dynamic masking.

###  3.6 Implementation Details

We utilize PKE [5], a state-of-the-art open-source Python-based keyphrase
extraction toolkit, which makes it possible to implement TFIDF, TopicRank,
YAKE!, and KEA. We use the n-gram range of (1,3), which means that produced
keyphrases could be unigrams, bigrams, or trigrams. To implement KeyBERT, we
use the BERT embeddings produced by the all-MiniLM-L6-v2 model [31] that maps
texts to a 384-dimensional dense vector space.

We conduct our experiments on text summarization models using the Transformers
library [39]. We use 3 epochs to fine-tune models on each dataset, the batch
size of 8, the maximum sequence length of 256, and the learning rate of 4e-5.
All summarization models are evaluated by three-fold cross-validation. We
compute ROUGE-1, F1-score, and BERTScore for each fold separately and then
determine the mean value.

##  4 Results and Discussion

To answer RQ1, we compared the results of all considered models in Tables 2,
3, and 4 in terms of F1-score, ROUGE-1, and BERTScore, respectively. For
baselines, we compute metrics on the top 5, 10, and 15 returned keyphrases.
The best result for each dataset is highlighted. The results for Appear-Post,
Appear-Pre-CC, and Appear-Post-CC are missed for SemEval2017 because all
keyphrases from this dataset are present in the corresponding texts.
Therefore, Appear-Pre just produces the list of keyphrases sorted by their
first occurrences.

Full-match results on all corpora are reported in Table 2. Among unsupervised
methods, TFIDF shows the highest results on Krapivin-A, Krapivin-T, and KP20K.
TopicRank demonstrates the best performance on Inspec and SemEval2017. KEA
outperforms other baselines on Krapivin-A (in terms of F1@5), Krapivin-T
(F1@10), and Inspec (F1@15). KeyBERT does not show the highest performance on
any of the datasets and demonstrates the worst F1-score on Inspec and
SemEval2017. Overall, BART achieves the best full-match results on Krapivin-A
and Krapivin-T. T5 outperforms other methods on Inspec, KP20K, and
SemEval2017.

Table 2 indicates the superiority of abstractive summarization models on the
datasets which have a large proportion of absent keyphrases. In particular,
the number of summarization models that outperform the best baseline result is
higher for the datasets consisting of abstracts, i.e., KP20K (43.67% of absent
keyphrases) – 9 models, Inspec (43.8%) – 8 models, Krapivin-A (51.3%) – 3
models, Krapivin-T (18.04%) – 1 model, SemEval2017 (0%) – 1 model.

Data | Krapivin-A | Krapivin-T | Inspec | KP20K | SemEval2017
---|---|---|---|---|---
F1@5 |  F1@10 |  F1@15 |  F1@5 |  F1@10 |  F1@15 |  F1@5 |  F1@10 |  F1@15 |  F1@5 |  F1@10 |  F1@15 |  F1@5 |  F1@10 |  F1@15
TFIDF | 10.8 | 10.5 | 9.48 | 7.85 | 8.59 | 8.45 | 9.76 | 13.27 | 14.81 | 11.12 | 10.69 | 9.66 | 14.86 | 19.33 | 21.96
TopicRank | 7.61 | 7.73 | 7.38 | 5.4 | 5.72 | 5.44 | 12.01 | 14.91 | 16 | 8.51 | 8.13 | 7.54 | 17.31 | 22.83 | 24.93
YAKE! | 6.93 | 8.33 | 8.1 | 7.68 | 8.34 | 7.79 | 10.53 | 13.46 | 13.96 | 7.71 | 8.65 | 8.58 | 13.03 | 18.52 | 21.04
KEA | 10.88 | 10.47 | 9.53 | 8.2 | 9.19 | 9.1 | 9.76 | 13.17 | 16.61 | 7.85 | 7.49 | 6.84 | 14.85 | 19.42 | 21.72
KeyBERT | 9.46 | 9.35 | 8.63 | 5.54 | 5.29 | 4.76 | 8.62 | 10.75 | 11.6 | 8.27 | 8.32 | 7.81 | 9.47 | 12.59 | 14.29
BART
No-Sort | 9.19 | 5.55 | 14.45 | 10.68 | 10.23
Random | 9.29 | 5.95 | 12.14 | 11.14 | 12.8
Length | 9.09 | 5.43 | 11.77 | 10.38 | 13.58
Alpha | 8.71 | 5.74 | 9.64 | 10.72 | 10.74
Appear-Pre | 10.1 | 5.17 | 14.55 | 11.72 | 18.22
Appear-Post | 7.59 | 4.57 | 13.33 | 11.36 | -
Appear-Pre-CC | 11.24 | 6.38 | 15.11 | 11.76 | -
Appear-Post-CC | 9.6 | 9.65 | 15.58 | 11.31 | -
T5
No-Sort | 10.2 | 6.05 | 22.29 | 11.65 | 16.62
Random | 11 | 5.9 | 17.49 | 11.94 | 18.62
Length | 10.69 | 5.31 | 17.3 | 10.96 | 17.51
Alpha | 9.76 | 6.34 | 17.43 | 11.18 | 18.3
Appear-Pre | 10.93 | 6.09 | 19.53 | 11.43 | 25.39
Appear-Post | 10.02 | 6.18 | 19.04 | 10.92 | -
Appear-Pre-CC | 8.88 | 4.61 | 21.24 | 9.61 | -
Appear-Post-CC | 7.38 | 4.78 | 18.18 | 7.53 | -
Table 2:  Results (F1-score, %). The best result for each dataset is
highlighted.

ROUGE-1 scores are reported in Table 3. TopicRank is the best on Inspec and
SemEval2017, KEA demonstrates the highest performance on Krapivin-T, and
KeyBERT achieves the best results on Krapivin-A and KP20K. The scores of BART
and T5 are quite low on each dataset. To sum up, the results obtained in terms
of F1-score and ROUGE-1 show that abstractive text summarization models are
relatively successful in predicting full-match keyphrases, but the generated
sequence of keyphrases contains a small number of words from the original list
of keyphrases.

Data | Krapivin-A | Krapivin-T | Inspec | KP20K | SemEval2017
---|---|---|---|---|---
R1@5 |  R1@10 |  R1@15 |  R1@5 |  R1@10 |  R1@15 |  R1@5 |  R1@10 |  R1@15 |  R1@5 |  R1@10 |  R1@15 |  R1@5 |  R1@10 |  R1@15
TFIDF | 27.66 | 29.91 | 29.14 | 21.04 | 24.84 | 25.21 | 26.55 | 36.45 | 41.45 | 27.9 | 30.11 | 29.3 | 23.67 | 34.92 | 41.72
TopicRank | 24.91 | 25.36 | 24.19 | 19.66 | 21.92 | 21.62 | 31.77 | 40.49 | 44.09 | 25.74 | 25.77 | 24.31 | 29.99 | 41.67 | 47.78
YAKE! | 24.63 | 27.7 | 28.25 | 22.85 | 25.98 | 26.36 | 30.29 | 37.44 | 40.9 | 25.14 | 27.82 | 28.21 | 26.43 | 35.65 | 41.39
KEA | 28.09 | 30.05 | 29.37 | 21.88 | 26.54 | 27.26 | 26.49 | 36.03 | 40.88 | 19.86 | 21.36 | 20.89 | 23.51 | 34.4 | 40.83
KeyBERT | 30.22 | 31.11 | 30.49 | 24.78 | 25.16 | 24.01 | 31.01 | 38.33 | 41.93 | 29.68 | 30.41 | 29.46 | 26.69 | 36.91 | 42.77
BART
No-Sort | 22.69 | 16.29 | 36.77 | 22.9 | 22.27
Random | 22.88 | 16.48 | 31.17 | 22.97 | 26.61
Length | 22.28 | 15.38 | 30.4 | 21.6 | 27.2
Alpha | 21.51 | 15.72 | 27.51 | 22.91 | 20.23
Appear-Pre | 22.53 | 16.09 | 34.38 | 23.57 | 38.54
Appear-Post | 21.71 | 15.27 | 34.44 | 17 | -
Appear-Pre-CC | 22.22 | 15.98 | 34.25 | 23.52 | -
Appear-Post-CC | 21.46 | 16.47 | 35.08 | 17.19 | -
T5
No-Sort | 21.68 | 14.37 | 41.51 | 21.76 | 27.98
Random | 22.37 | 14.04 | 33.52 | 21.9 | 33.12
Length | 22.73 | 13.62 | 34.27 | 20.84 | 30.52
Alpha | 21.66 | 15.38 | 32.67 | 22.02 | 29.11
Appear-Pre | 22.45 | 14.94 | 36.2 | 21.04 | 40.34
Appear-Post | 21.07 | 14.89 | 36.7 | 22.13 | -
Appear-Pre-CC | 19.1 | 12.8 | 39.5 | 19.48 | -
Appear-Post-CC | 18.13 | 12.47 | 35.02 | 18.09 | -
Table 3:  Results (ROUGE-1, %). The best result for each dataset is
highlighted.

The results in terms of BERTScore are shown in Table 4. Among unsupervised
methods, the best scores for all datasets are obtained by TopicRank. KEA and
KeyBERT perform worse on all considered text corpora. The results indicate the
sustained superiority of BART, which achieves the highest scores for all
datasets. The results of T5 differ depending on the data. For instance, T5
shows rather high scores on Inspec but never exceeds the best result obtained
by traditional methods on Krapivin-T. In addition, BERTScore also shows that
abstractive summarization models mostly outperform traditional methods on the
datasets with a large proportion of absent keyphrases, as is the case with
F1-score. For example, 15 models of 16 are superior to TopicRank on Inspec and
eleven models beat TopicRank on Krapivin-A. On the other hand, only two
summarization models of ten outperform the best baseline result on
SemEval2017, the proportion of absent keyphrases of which is 0%. Overall,
BERTScore indicates that abstractive summarization models can produce
keyphrases that are close in meaning to original keyphrases in terms of token
similarity calculated with contextual embeddings.

Data | Krapivin-A | Krapivin-T | Inspec | KP20K | SemEval2017
---|---|---|---|---|---
BS@5 |  BS@10 |  BS@15 |  BS@5 |  BS@10 |  BS@15 |  BS@5 |  BS@10 |  BS@15 |  BS@5 |  BS@10 |  BS@15 |  BS@5 |  BS@10 |  BS@15
TFIDF | 85.91 | 85.12 | 84.24 | 85.32 | 85.12 | 84.5 | 83.45 | 84.15 | 84.28 | 85.6 | 84.7 | 83.86 | 83.64 | 84.61 | 84.73
TopicRank | 86.53 | 86.19 | 85.7 | 85.95 | 85.74 | 85.16 | 84.25 | 85.17 | 85.45 | 86.21 | 85.75 | 85.23 | 84.29 | 85.29 | 85.59
YAKE! | 85.28 | 84.74 | 84.14 | 84.98 | 84.73 | 84.19 | 83.64 | 84.1 | 84.21 | 84.88 | 84.24 | 83.72 | 83.5 | 84.2 | 84.49
KEA | 85.9 | 85.06 | 84.2 | 85.49 | 85.41 | 84.83 | 83.44 | 84.1 | 84.21 | 85.56 | 84.66 | 83.82 | 83.59 | 84.51 | 84.61
KeyBERT | 86.43 | 85.46 | 84.72 | 85.45 | 84.21 | 83.33 | 84.39 | 84.64 | 84.56 | 85.72 | 84.71 | 84 | 84.3 | 84.82 | 84.91
BART
No-Sort | 87.9 | 86.59 | 87.72 | 86.97 | 84.42
Random | 88 | 87.01 | 86.81 | 86.98 | 84.91
Length | 87.85 | 86.97 | 86.94 | 87.13 | 84.95
Alpha | 87.72 | 86.68 | 86.45 | 87.08 | 83.91
Appear-Pre | 88.09 | 86.66 | 87.36 | 87.1 | 86.8
Appear-Post | 87.59 | 86.64 | 87.53 | 85.75 | -
Appear-Pre-CC | 86.31 | 85.43 | 86.39 | 86.86 | -
Appear-Post-CC | 86.8 | 86.21 | 86.73 | 86.47 | -
T5
No-Sort | 87.3 | 85.03 | 86.74 | 84.84 | 84.21
Random | 87.02 | 85.07 | 85.3 | 86.79 | 83.98
Length | 87.29 | 84.23 | 85.52 | 84.11 | 84.1
Alpha | 86.35 | 84.51 | 86.1 | 85.75 | 83
Appear-Pre | 86.59 | 83.57 | 86.03 | 86.05 | 86.43
Appear-Post | 85.32 | 84.39 | 85.67 | 84 | -
Appear-Pre-CC | 86.06 | 83.72 | 86.31 | 85.07 | -
Appear-Post-CC | 85.05 | 84.28 | 85.77 | 84.25 | -
Table 4:  Results (BERTScore, %). The best result for each dataset is
highlighted. Figure 2: An example of the differences between the metrics.
Original keyphrases (reference): e-books, library journal, library automation,
electronic books, electronic publishing. T5 (candidate): e-book revolution,
consumer market, reading devices, electronic publishing. F1-score: 22.22%,
ROUGE-1: 26.67%, BERTScore: 88.86%

Table 5 shows two examples of generating keyphrases. The first row contains
the original list of keyphrases provided by the authors. The rest of the table
illustrates keyphrases derived through various methods. For BART and T5, we
provide the list of keyphrases produced by the best models. Full matches for
the keyphrases from the original list are shown in bold. Exact word matches
for the words from the authors’ list of keyphrases are underlined. Such
examples demonstrate that text summarization models generate more abstractive
keyphrases and use fewer repeated words.

Our study demonstrates that different evaluation metrics estimate different
aspects of keyphrase generation. For instance, Figure 2 visualizes matching
scores between the tokens from original and generated keyphrases obtained by
BERTScore. This original example was taken from Inspec and the generated list
of keyphrases was produced by T5 (No-Sort). It has an F1-score, which is close
to the mean F1-score for this dataset, while the ROUGE-1 score is much lower
and BERTScore is higher compared to the mean value. Therefore, the choice of
metric is an important step of the research that directly affects the
conclusions. In this work, ROUGE-1 indicates the superiority of traditional
methods for keyphrase extraction. By contrast, BERTScore defines BART as the
most effective model.

Source | Example 1 | Example 2
---|---|---
Original |  female computer science doctorates, survey of earned doctorates, information science, computer science education, gender issues |  OS porting, application development, consumer operating system, hardware design, operating systems (computers), software portability
TFIDF |  doctorates, women, completing doctorates, science, computer science, 1997, academic year, sed, females, academic |  operating system, porting, deliver improved, deliver improved usability, improved usability, high-end portable, high-end portable consumer, portable consumer, portable consumer products, consumer products
TopicRank |  doctorates, computer, women, academic year, science, percentages, degrees, sed, prior research, females |  major complexity, device developer, appropriate consumer operating system, validation, porting, problem, support, system implementation, trend, real-time OS
YAKE! |  education statistics, national center, computer science, computer, science, doctorates, degrees, statistics, national, center |  portable consumer products, high-end portable consumer, deliver improved usability, appropriate consumer operating, consumer products, portable consumer, appropriate consumer, consumer operating system, deliver improved, improved usability
KEA |  doctorates, women, science, computer science, completing doctorates, 1997, academic year, academic, earned, education statistics |  operating system, deliver improved, deliver improved usability, improved usability, high-end portable, high-end portable consumer, portable consumer, portable consumer products, consumer products, appropriate consumer
KeyBERT |  doctorates computer, doctorate computer, include doctorates, earned doctorates, science doctorates, computer science, doctorates, doctorates sed, doctorates degrees, doctorates include |  OS porting, platform OS, supported OS, operating OS, OS, cation device, portable consumer, implementation porting, device developer, platform
BART |  academic year, computer science education, women education, national center for education statistics |  portable computing, software development, portable consumer products, software, hardware design, real-time OS, porting, validation, software aspects, portability, performance, user interfaces, user experience, portable products, consumer operating system, operating system
T5 |  computer science education, women, computer science doctorates, academic levels, gender issues, education |  porting, virtual reality, portable consumer products, consumer operating system, OS, commercially supported OS, complete operating system, real-time OS, complex platform OS, real-time operating systems, asynchronous operating systems, portable devices, mobile computing
Table 5:  Keyphrases extracted by different models. Full matches are bolted,
and exact matches are underline.

To answer RQ2, we estimate the performance growth of the keyphrase generation
in comparison with the No-Sort strategy for all other concatenating
strategies. The performance growth is calculated as follows:

$\frac{R_{i}-R_{NoSort}}{R_{NoSort}},$ (1)

where $R_{NoSort}$ is a performance (F1-score or BERTScore) for the No-Sort
strategy, $i$ is a strategy from the set of strategies $I=\\{$Random, Length,
Alpha, Appear-Pre, Appear-Post, Appear-Pre-CC, Appear-Post-CC$\\}$.

a) BART (F1-score)

b) T5 (F1-score)

a) BART (BERTScore)

b) T5 (BERTScore)

Figure 3: Comparing strategies for summarization models.

As shown in Figure 3, Random can increase the results, but we have not found
stable improvement when using this strategy. Length, Alpha, and Appear-Post
demonstrate low performance in most models. As can be seen on all charts,
Appear-Pre improves the performance on SemEval2017. Therefore, the ordering is
effective for those datasets that does not include absent keyphrases. The
results of Appear-Pre-CC and Appear-Post-CC are opposite for BART (F1-Score)
and other cases. The use of control codes improves the full-match results of
BART on all corpora. In all other considered cases, control codes appeared to
be ineffective.

To illustrate the resource usage for keyphrase generation by different models,
we show the time and memory consumption in Figure 4 on the example of
SemEval2017. The time and memory usage for training was not included in the
calculation. In other words, we loaded the trained model, generated keyphrases
for all texts in the dataset, and measured the indicators using the Python and
Google Colaboratory tools. The figure shows that memory usage is expectedly
higher for transformer-based models than for traditional methods. The time
consumption during running on CPU is the highest for TFIDF, KEA, and BART.
However, the use of GPU for transformers allows a reduction of the generation
time to less than one minute for KeyBERT and approximately two minutes for
BART and T5.

Figure 4: Resource usage on the example of SemEval2017.

##  5 Conclusion

In this paper, we explored the effectiveness of abstractive summarization
models based on transformer architecture for the task of predicting keyphrases
for scientific texts. We performed an extensive evaluation of unsupervised and
supervised models for keyphrase extraction and compared several ordering
strategies for concatenating keyphrases on several datasets. Our results
showed some pros and cons of the use of transformer-based summarization models
for keyphrase extraction. First, we obtained promising results in terms of the
full-match F1-score and BERTScore, but ROUGE-1 indicates the superiority of
traditional methods for keyphrase extraction. Second, we indicated that
summarization models are more competitive in generating keyphrases that are
not explicitly presented in the source text. Finally, we demonstrated that
some ordering strategies provide better results in keyphrase generation, while
others decrease the performance.

## References

  * [1] Alami Merrouni, Z., Frikh, B., Ouhbi, B.: Automatic keyphrase extraction: a survey and trends. Journal of Intelligent Information Systems 54(2), 391–424 (2020). 10.1007/s10844-019-00558-9
  * [2] Ale Ebrahim, N., Salehi, H., Embi, M. A., Habibi, F., Gholizadeh, H., Motahar, S. M., Ordi, A.: Effective strategies for increasing citation frequency. International Education Studies 6(11), 93–99 (2013). 10.5539/ies.v6n11p93
  * [3] Augenstein, I., Das, M., Riedel, S., Vikraman, L., McCallum, A.: SemEval 2017 Task 10: ScienceIE-Extracting Keyphrases and Relations from Scientific Publications. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), 546–555 (2017). 10.18653/v1/s17-2091
  * [4] Bird, S.: NLTK: The Natural Language Toolkit. In: Proceedings of the COLING/ACL 2006 Interactive Presentation Sessions, 69–72 (2006). 10.3115/1225403.1225421
  * [5] Boudin, F.: PKE: an open source python-based keyphrase extraction toolkit. In: Proceedings of COLING 2016, the 26th international conference on computational linguistics: system demonstrations, 69–73 (2016).
  * [6] Bougouin, A., Boudin, F., Daille, B.: TopicRank: Graph-based topic ranking for keyphrase extraction. In: International joint conference on natural language processing (IJCNLP), 543–551 (2013).
  * [7] Cachola, I., Lo, K., Cohan, A., Weld, D. S.: TLDR: Extreme Summarization of Scientific Documents. In: Findings of the Association for Computational Linguistics: EMNLP 2020, 4766–4777 (2020).
  * [8] Campos, R., Mangaravite, V., Pasquali, A., Jorge, A., Nunes, C., Jatowt, A.: YAKE! Keyword extraction from single documents using multiple local features. Information Sciences 509, 257–289 (2020). 10.1016/j.ins.2019.09.013
  * [9] Çano, E., Bojar, O.: Keyphrase Generation: A Text Summarization Struggle. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 666–672 (2019). 10.18653/v1/N19-1070
  * [10] Chowdhury, M. F. M., Rossiello, G., Glass, M., Mihindukulasooriya, N., Gliozzo, A. Applying a Generic Sequence-to-Sequence Model for Simple and Effective Keyphrase Generation. arXiv (2022). 10.48550/ARXIV.2201.05302
  * [11] Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186 (2019). 10.18653/v1/N19-1423
  * [12] El-Beltagy, S. R., Rafea, A.: KP-Miner: A keyphrase extraction system for English and Arabic documents. Information systems 34(1), 132–144 (2009). 10.1016/j.is.2008.05.002
  * [13] Florescu, C., Caragea, C.: PositionRank: An unsupervised approach to keyphrase extraction from scholarly documents. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1105–1115 (2017). 10.18653/v1/P17-1102
  * [14] Grootendorst, M.: KeyBERT: Minimal keyword extraction with BERT. Zenodo (2020). 10.5281/zenodo.4461265
  * [15] Hasan, K. S., Ng, V.: Automatic keyphrase extraction: A survey of the state of the art. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 1262–1273 (2014). 10.3115/v1/P14-1119
  * [16] Hulth, A.: Improved automatic keyword extraction given more linguistic knowledge. In: Proceedings of the 2003 conference on Empirical methods in natural language processing, 216–223 (2003). 10.3115/1119355.1119383
  * [17] Ilango, D. V., Kumar, D. S. M.: Factors For Improving The Research Publications And Quality Metrics. International Journal of Civil Engineering & Technology 8(4), 477–496 (2017).
  * [18] Keskar, N. S., McCann, B., Varshney, L. R., Xiong, C., Socher, R.: CTRL: A conditional transformer language model for controllable generation. arXiv (2019). 10.48550/arxiv.1909.05858.
  * [19] Krapivin, M., Autaeu, A., Marchese, M.: Large dataset for keyphrases extraction (2009).
  * [20] Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., Zettlemoyer, L.: BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 7871–7880 (2020). 10.18653/v1/2020.acl-main.703
  * [21] Lim, Y., Seo, D., Jung, Y.: Fine-tuning BERT Models for Keyphrase Extraction in Scientific Articles. Journal of advanced information technology and convergence 10(1), 45–56 (2020). 10.14801/jaitc.2020.10.1.45
  * [22] Lin, C. Y. Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out, pp. 74-81 (2004).
  * [23] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V. RoBERTa: A robustly optimized BERT pretraining approach. arXiv (2019). 10.48550/arXiv.1907.11692
  * [24] Medelyan, O., Frank, E., Witten, I. H.: Human-competitive tagging using automatic keyphrase extraction. In: Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, 1318–1327 (2009). 10.3115/1699648.1699678
  * [25] Meng, R., Yuan, X., Wang, T., Brusilovsky, P., Trischler, A., He, D.: Does order matter? an empirical study on generating multiple keyphrases as a sequence. arXiv (2019). 10.48550/ARXIV.1909.03590
  * [26] Meng, R., Zhao, S., Han, S., He, D., Brusilovsky, P., Chi, Y.: Deep Keyphrase Generation. In: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 582–592 (2017). 10.18653/v1/P17-1054
  * [27] Mihalcea, R., Tarau, P.: TextRank: Bringing order into text. In: Proceedings of the 2004 conference on empirical methods in natural language processing, 404-411 (2004).
  * [28] Nguyen, T. D., Luong, M. T.: WINGNUS: Keyphrase extraction utilizing document logical structure. In: Proceedings of the 5th international workshop on semantic evaluation, 166–169 (2010).
  * [29] Papagiannopoulou, E., Tsoumakas, G.: A review of keyphrase extraction. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 10(2), e1339 (2020). 10.1002/widm.1339
  * [30] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P. J.: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21, 1–67 (2020).
  * [31] Reimers, N., Gurevych, I.: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), 3982–3992 (2019). 10.18653/v1/D19-1410
  * [32] Sahrawat, D., Mahata, D., Zhang, R., Kulkarni, M., Sharma, A., Gosangi, R., Stent, A., Kumar, Y., Shah, R. R., Zimmermann, R.: Keyphrase Extraction as Sequence Labeling Using Contextualized Embeddings. Lecture Notes in Computer Science 12036, 328–335 (2020). 10.1007/978-3-030-45442-5_41
  * [33] Shen, L., Jiang, H., Liu, L., Shi, S.: Revisiting the Evaluation Metrics of Paraphrase Generation. arXiv (2022). 10.48550/arXiv.2202.08479
  * [34] Stowe, K., Beck, N., Gurevych, I.: Exploring Metaphoric Paraphrase Generation. In: Proceedings of the 25th Conference on Computational Natural Language Learning, 323–336 (2021). 10.18653/v1/2021.conll-1.26
  * [35] Swaminathan, A., Zhang, H., Mahata, D., Gosangi, R., Shah, R., Stent, A.: A preliminary exploration of GANs for keyphrase generation. In: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), 8021–8030 (2020). 10.18653/v1/2020.emnlp-main.645
  * [36] Wang, L., Li, S.: PKU_ICL at SemEval-2017 task 10: Keyphrase extraction with model ensemble and external knowledge. In: Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), 934–937 (2017). 10.18653/v1/S17-2161
  * [37] Wang, Y., Liu, Q., Qin, C., Xu, T., Wang, Y., Chen, E., Xiong, H.: Exploiting topic-based adversarial neural network for cross-domain keyphrase extraction. In: 2018 IEEE International Conference on Data Mining (ICDM), 597–606 (2018). 10.1109/icdm.2018.00075
  * [38] Witten, I. H., Paynter, G. W., Frank, E., Gutwin, C., Nevill-Manning, C. G.: Kea: Practical automated keyphrase extraction. Design and Usability of Digital Libraries: Case Studies in the Asia Pacific. IGI global, 129–152 (2005). 10.4018/978-1-59140-441-5.ch008
  * [39] Wolf, T, Debut, L., Sanh, V, Chaumond, J., Delangue, C, Moi, A., Cistac, P., Rault, T, Louf, R., Funtowicz, M., Davison, J., Shleifer, S., von Platen, P., Ma, C, Jernite, Y, Plu, J., Xu, C, Scao, T.L., Gugger, S., Drame, M., Lhoest, Q., Rush, A.M.: Transformers: State-of-the-art natural language processing. In: Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, 38–45 (2020). 10.18653/v1/2020.emnlp-demos.6
  * [40] Zhang, Q., Wang, Y., Gong, Y., Huang, X. J.: Keyphrase extraction using deep recurrent neural networks on Twitter. In: Proceedings of the 2016 conference on empirical methods in natural language processing, 836–845 (2016). 10.18653/v1/D16-1080
  * [41] Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., Artzi, Y.: BERTScore: Evaluating Text Generation with BERT. In: International Conference on Learning Representations (2019).