1 B. König-Ries et al. Datenbanksysteme für Business, Technologie und Web (BTW
2023) 2023

# On the State of German (Abstractive) Text Summarization

Dennis Aumiller111Heidelberg University, Institute of Computer Science, Im
Neuenheimer Feld 205, 69120 Heidelberg, Germany<EMAIL_ADDRESS>heidelberg.de Jing Fan222Heidelberg University, Institute of Computer Science,
Im Neuenheimer Feld 205, 69120 Heidelberg, Germany<EMAIL_ADDRESS>heidelberg.de Michael Gertz333Heidelberg University, Institute of Computer
Science, Im Neuenheimer Feld 205, 69120 Heidelberg, Germany
<EMAIL_ADDRESS>
###### Abstract

With recent advancements in the area of Natural Language Processing, the focus
is slowly shifting from a purely English-centric view towards more language-
specific solutions, including German. Especially practical for businesses to
analyze their growing amount of textual data are text summarization systems,
which transform long input documents into compressed and more digestible
summary texts. In this work, we assess the particular landscape of German
abstractive text summarization and investigate the reasons why practically
useful solutions for abstractive text summarization are still absent in
industry.
Our focus is two-fold, analyzing a) training resources, and b) publicly
available summarization systems. We are able to show that popular existing
datasets exhibit crucial flaws in their assumptions about the original
sources, which frequently leads to detrimental effects on system
generalization and evaluation biases. We confirm that for the most popular
training dataset, MLSUM, over 50% of the training set is unsuitable for
abstractive summarization purposes. Furthermore, available systems frequently
fail to compare to simple baselines, and ignore more effective and efficient
extractive summarization approaches. We attribute poor evaluation quality to a
variety of different factors, which are investigated in more detail in this
work: A lack of qualitative (and diverse) gold data considered for training,
understudied (and untreated) positional biases in some of the existing
datasets, and the lack of easily accessible and streamlined pre-processing
strategies or analysis tools. We therefore provide a comprehensive assessment
of available models on the cleaned versions of datasets, and find that this
can lead to a reduction of more than 20 ROUGE-1 points during evaluation. As a
cautious reminder for future work, we also highlight the problems of solely
relying on $n$-gram based scoring methods by presenting particularly
problematic failure cases. The code for dataset filtering and reproducing
results can be found online: https://github.com/dennlinger/summaries

###### keywords:

Abstractive Text Summarization Natural Language Generation German Evaluation

##  1 Introduction

Libraries simplifying the access to pre-trained neural models have greatly
pushed the recent advancement of state-of-the-art performance in many tasks
[wolf-etal-2020-transformers]. However, with the general absence of non-
English resources, one of the prevalent challenges in the Natural Language
Processing (NLP) community is the extension of approaches to other languages
beyond English. Subsequently, evaluation quality and consistency is even
harder to maintain in setups, where high-quality gold data is scarce. This can
lead to unintended consequences during the interpretation of model performance
and generalization capabilities beyond narrow domain-specific use cases.

A sub-task of the NLP community that deserves particular attention is text
summarization. The focus here is to produce an abridged version of an input
text that accurately _summarizes_ the key points of the original text. Such
systems offer an immediate benefit in times with ever-increasing amounts of
textual information, and allow users to quickly grasp the contents of even
complex documents. In particular, we differentiate between various sub-tasks
of text summarization: _extractive_ systems provide summaries by simply
copying text snippets from the original input, which is efficient to compute,
but comes at the cost of lower textual fluency. On the other hand,
_abstractive_ summarization systems may introduce new phrases, or even full
sentences, which are not present in the original document. This potentially
increases a summary’s fluency and conciseness over extractive methods.
Abstractive text summarization systems are generally built upon the more
recent development of sequence-to-sequence neural models [sutskever-
etal-2014-sequence, bahdanau-etal-2015-neural], which come with an exploding
computational cost.

Particularly for (abstractive) summarization, the previously mentioned issues
of data scarcity for non-English methods are further worsened by a lack of
diverse (and readily available) evaluation metrics. Most works rely entirely
on $n$-gram-based analysis of system summaries, such as ROUGE
[lin-2004-rouge], which cannot accurately judge the truthfulness of a
generated summary, i.e., how accurately the original text’s factual statements
are represented in the generated summary. Only few works extend their
evaluation to human result inspections, given its higher cost. However, there
are several critical assumptions that –even under basic premises– are exposed
as oftentimes insufficient for a comprehensive analysis [sparck-
jones-1997-summarising, ter-hoeve-etal-2020-what]. Examples are the focus on
singular target summaries, ignoring the subjective nature of differing
viewpoints of annotators, as well as the focus on particularly prominent
sentences in the first few paragraphs of reference articles [zhu-
etal-2021-leveraging].

In this work, we focus on German abstractive summarization systems and set out
to investigate, reproduce, and evaluate summarization systems. In conjunction
to a model-centric view of summarization, we further review the existing
training resources for German, including their particular domain and data
curation processes.

  1. 1.

We find that in particular automatically created and multilingual resources
suffer from insufficient pre-processing, potentially due to the absence of a
native speaker during the curation process.

  2. 2.

News documents seem to be overly represented in trained systems, potentially
due to a popularity bias in English summarization datasets for news resources.

  3. 3.

Baseline scores are heavily affected by data biases in test sets of prominent
datasets.

Upon conducting a qualitative analysis of outputs from publicly available
models, we further find that most systems fall severely short of the expected
quality in at least one of the following areas:

  1. 1.

Due to positional biases, text snippets may be directly copied from the
beginning of the input text, constituting an _extractive_ instead of an
_abstractive_ summary. Especially considering the computational requirements
of neural systems being orders of magnitudes greater than simple extractive
summarizers, this undermines the quality of neural text generations.

  2. 2.

Generated outputs may contain (severe) syntactic errors, to the point of
becoming illegible or hard to interpret.

  3. 3.

Semantic mistakes introduce factual errors, leading to incorrect conclusions
from the summary alone. This problem is exacerbated for longer input
documents, where a structured content understanding is necessary to maintain
factual consistency.

For data-centric issues, current pipelines are not taking user-specified
filtering steps into account; oftentimes, datasets are directly used “out of
the box”, without any further data verification step involved. For this
purpose, we extend available summarization-specific filtering steps and
provide a simple-to-use and language-agnostic processing library.
For model-centric problems, it is near impossible to identify failure cases
with existing metrics; costly manual inspection of individual samples would be
required. Simultaneously, we work towards expanding the available scores to
help facilitate a better understanding of current expectations towards
summarization systems. In the following, we will briefly mention work on
automated evaluation of summarization systems, including a comprehensive look
at the current landscape of German abstractive summarization; we follow with a
formal introduction of our proposed filtering methods for summarization
datasets, as well as a list of model-centric checks to consider. We discuss
exhibited quality issues in existing datasets and systems for German
summarization, and conclude with a brief outlook for future work.

##  2 Related Work

We establish an extensive overview of currently available training resources
for German summarization systems and survey the landscape of trained models,
with a particular focus on publicly available methods.
Aside from this, we further reiterate some of the common pitfalls in
evaluating summarization systems, which will become particularly relevant
during the experiments in this work.

###  2.1 German Data Sources for Summarization

In our experiments, we focus on seven different datasets across a variety of
domains. To our knowledge, these cover all of the publicly available sources
used for training German systems.

##### MLSUM [scialom-etal-2020-mlsum]

This multilingual dataset was presented as one of the first efforts in making
larger-scale training sets available for multiple languages that also include
German as a language. MLSUM is constructed by extracting news articles and
associated summary sections as generation targets. We use the German subset in
this work, which is by far the most popular dataset used for training and
evaluating resources in German, based on our survey. Despite its popularity,
issues in the quality of samples have gone unnoticed until early 2022, when
Philip May [may-2022-anomalies] was the first to report on problems with fully
extractive summaries, an aspect we will analyze in more detail later.

##### MassiveSumm [varab-schluter-2021-massivesumm]

The construction of this particular dataset is similar to MLSUM and focuses on
a large number of automatically extracted summaries from news articles in
multiple languages. The authors perform some rudimentary filtering with
respect to empty samples and even go as far as avoiding similar issues to
MLSUM by removing what they call “ellipsoid summaries”, i.e., fully extractive
summaries that appear at the beginning of the reference text. While the
quality of the samples is comparatively low due to the automated extraction
process, this corpus is by far the largest considered, with around 480,000
samples, and has the potential to improve existing training setups with its
sheer number of samples.

##### Swisstext [frefel-etal-2020-german, frefel-2020-summarization]

In contrast to the –generally shorter– news articles available in MLSUM, the
Swisstext dataset provides longer-form summaries based on German Wikipedia
pages, which has been later extended to the GeWiki corpus
[frefel-2020-summarization]. For the construction, the central argument is
that the introductionary paragraph serves as a “summary” of the remaining
article text. The provided dataset comes with a training portion and a private
test set, meaning no ground truth summaries are available for the test
samples. A multilingual variant of this idea, the XWikis corpus, was
introduced shortly after [perez-beltrachini-lapata-2021-models]. While the
XWikis corpus contains more samples per language, including for German,
monolingual data is not readily available for download. Adding the fact that
German summarization works primarily deal with the Swisstext dataset, we
choose the latter for our experiments.

##### Klexikon [aumiller-gertz-2022-klexikon]

Another Wikipedia-related resource, but with different target summaries.
Instead of utilizing a page’s introductionary paragraph, the authors align
articles from a simplified children’s encyclopedia (Klexikon) on the same
topic. Consequently, this dataset has much longer summary lengths but covers a
much smaller subset of only around 3,000 samples. Given the secondary focus on
simplification in the target summaries, this corpus requires a considerably
higher level of abstractive reformulations during the generation.

##### WikiLingua [ladhak-etal-2020-wikilingua]

As the third multilingual resource, summaries in this corpus are extracted
from the WikiHow platform. Here, Ladhak et al. [ladhak-etal-2020-wikilingua]
consider short instruction summaries of individual steps in WikiHow guides and
align those with the referenced paragraphs. The general tone of the dataset is
rather informal and is in a more imperative style in comparison to other data
sources. To align non-English samples, associated images are used to identify
paragraphs occurring in different languages. Importantly, this means that for
German articles, frequently only some of the article’s paragraphs are actually
contained in the dataset.

##### LegalSum [glaser-etal-2021-summarization]

Another area benefiting enormously from high-quality summaries is the legal
domain. LegalSum is the first German resource providing summaries of around
100,000 court rulings. On average, these samples require the highest amount of
compression across evaluated datasets.

##### EUR-Lex-Sum [aumiller-etal-2022-eur]

As a secondary resource for legal texts, Aumiller et al. present a
multilingual corpus based on EU-level legal acts, semi-aligned across
languages. The corpus is considerably smaller than LegalSum, with only about
1,900 German documents available. While the EUR-Lex-Sum corpus has extremely
long documents, it also presents a more challenging summary generation with
the longest average summary length across all considered corpora (generally
between 600-800 words). Importantly, summaries are also written by human
expert annotators and therefore present a much higher-quality standard for
summaries compared to some of the other datasets.

##### Further Resources

In addition to these datasets, we are aware of at least two more news-related
resources. One is used in experiments by Nitsche [nitsche-2019-towards], where
data was supplied by the German Press Agency, but no public record of it
exists. The second corpus is hinted at online by users on Huggingface’s
platform444A news corpus with ca. 400,000 articles is indicated here:
https://huggingface.co/Einmalumdiewelt/PegasusXSUM_GNAD/discussions/1#6308eb5037556c4ab03258df,
last accessed: 2023-01-14.
For clinical summarization, Liang et al. [liang-etal-2022-fine] present a
resource of about 11,000 radiology reports; given the sensitive nature of the
data, no publicly available version exists as of now. We are also aware of a
secondary source of the WikiLingua dataset by GEM555https://gem-
benchmark.com/data_cards/wiki_lingua, last accessed: 2023-01-14, which
provides additional samples, as well as a pre-split validation and test
section not provided in the original German subset. In preliminary
experiments, we found that $>99.89\%$ of the data were valid samples for the
GEM source. Most problematic is the automatic combination of paragraphs into
one summary, which can cause disjoint reference texts or summaries.
Finally, all of the discussed corpora so far are types of single document
summarization resources, where a summary is extracted from a _singular text
only_. Datasets for training summarization systems that consider multiple
source texts exist at smaller scales, but require further manual adjustment
for acquisition [benikova-etal-2016-bridging, zopf-2018-auto]. More recent
experiments with neural models utilizing the latter corpus have been conducted
by Johner et al. [johner-etal-2021-error].

###  2.2 German Summarization Systems

Model | Training data | Test Set | Evaluation | Filtering | Public | Reprod.
---|---|---|---|---|---|---
mrm8488/bert2bert666https://hf.co/mrm8488/bert2bert_shared-german-finetuned-summarization, last accessed: 2022-10-06 | MLSUM | MLSUM | ROUGE | None | ✓ | ✓
ml6team/mt5-small777https://huggingface.co/ml6team/mt5-small-german-finetune-mlsum, last accessed: 2022-10-06 | MLSUM | MLSUM | ROUGE | Length | ✓ | ✗
T-Systems/mt5-small888https://huggingface.co/T-Systems-onsite/mt5-small-sum-de-en-v2, last accessed: 2022-10-06 |  CNN/DailyMail, MLSUM, XSum, Swisstext | MLSUM | ROUGE |  Length & Overlap | ✓ | ✗
Shahm/t5-small999https://huggingface.co/Shahm/t5-small-german, last accessed: 2022-10-06 | MLSUM | MLSUM | ROUGE | None | ✓ | ✗
T5-base101010https://huggingface.co/Einmalumdiewelt/T5-Base_GNAD, last accessed: 2022-10-06 | ? | ? | ROUGE | ? | ✓ | ✗
german-t5111111https://github.com/GermanT5/german-t5-eval, last accessed: 2022-10-06 | Swisstext | MLSUM | ROUGE | ? | ✗ | ✗
BERT-Copy [aksenov-etal-2020-abstractive] | Swisstext | Swisstext |  ROUGE & manual | ? | ✓ | ?
Transformer [parida-motlicek-2019-swisstext] |  Swisstext & CommonCrawl | Swisstext |  ROUGE & manual | None | ✗ | ✗
Fact-Encoder [venzin-etal-2019-swisstext] | Swisstext | Swisstext |  ROUGE & manual | None | ✗ | ✗
Pointer-Gen [fecht-etal-2019-swisstext] | Swisstext | Swisstext |  ROUGE & manual | ? | ✗ | ✗
Enc-Dec [glaser-etal-2021-summarization] | LegalSum | LegalSum | ROUGE | ? | ✓ | ?
bert2bert [liang-etal-2022-fine] | Radiology | Radiology |  ROUGE & manual | ? | ✗ | ✗
Table 1: List of German abstractive summarization systems. We detail their
known properties from training recipes or papers. If we have access to models,
we denote whether public scores are reproducible within $\pm$0.5 ROUGE points
(“ _Reprod._ ”); ? in the reproducibility column indicates models that are
available, however, we were unable to successfully run locally.

While we are slowly starting to see a greater diversity in the available
training resources for German text summarization, it comes as a small surprise
that the availability of trained system is much less diverse. As will become
more apparent in later sections, the primary focus for training systems is a
combination of a pre-trained checkpoint and one predominant training resource
(“MLSUM”, particularly the German subset). Below, we elaborate on considered
model properties, differentiating between the availability levels of related
works and their backgrounds. A summary of known properties can be seen in
Table 1.

####  2.2.1 Publicly Available Systems

The primary source for available models is the Huggingface
Hub121212https://huggingface.co/models, last accessed: 2023-01-14, which
allows filtering by supported language and appropriate task (in our case
summarization). We note that some of the available models are not properly
tagged, but spent considerable time to ensure no other models were
accidentally ignored. For users who have uploaded several different versions,
we selected the model with the highest self-reported evaluation scores.
Given that users on the platform are likely familiar with other services of
Huggingface (including their datasets browser), it comes as no surprise that
the diversity between models and training setups is low. The primary choice
falls on either mT5 [xue-etal-2021-mt5] or variants of T5 [raffel-
etal-2020-exploring], with some alternatives based on (m)BART [lewis-
etal-2020-bart, liu-etal-2020-multilingual] being consistently outperformed
according to self-reported metrics by authors. In order to train effectively
on large quantities on data, most approaches use one of the smaller
checkpoints, referring to model variants with fewer parameters. Outside of the
model hub, code repositories exist for the BERT-Copy architecture by Aksenov
et al. [aksenov-etal-2020-abstractive] and Encoder-Decoder models used by
Glaser et al. [glaser-etal-2021-summarization]. However, we were unable to set
up inference for custom datasets based on the respective code bases.

####  2.2.2 Private Models

A further selection of models has been published in response to the Swisstext
2019 summarization challenge [parida-motlicek-2019-swisstext, venzin-
etal-2019-swisstext, fecht-etal-2019-swisstext]. However, neither team has
published any associated public repository. Similarly, no models are available
from Liang et al.’s work on radiology reports [liang-etal-2022-fine]. As the
only one of the major cloud providers, Microsoft offers an extractive
summarization service through Azure that supports
German.131313https://learn.microsoft.com/en-us/azure/cognitive-
services/language-service/summarization/language-support, last accessed:
2022-10-06 Otherwise, the only commercial solution providing a platform for
abstractive summarization also supporting German texts is currently Aleph
Alpha.141414https://www.aleph-alpha.com/use-cases/conversion#trilingual-
summary, last accessed: 2022-10-06

###  2.3 Evaluation Metrics for Summarization

As previously mentioned, the de-facto gold standard for evaluating
summarization systems is the usage of ROUGE [lin-2004-rouge]. The authors
introduce unigram overlap (ROUGE-1), bigram overlap (ROUGE-2) and the longest
common subsequence (ROUGE-L) between system and gold predictions. The
underlying core assumption is based on $n$-gram co-occurrences in the
generated text with respect to one or more gold summaries. The fact that ROUGE
can handle several reference samples at the same time is crucial for
understanding some of the implications in the later parts of this work: with
several references, variation in wording, e.g., particular expressions, are
much easier to compare against than in a single reference summary. However,
despite the theoretical support for multi-labels, few datasets ever provide
such costly annotations.
In turn, more recently proposed alternatives to ROUGE rely on score
computation from a single gold summary only [ermakova-eta-2019-survey].
Examples include primarily neural similarity scoring between a generated
summary and a gold reference [sellam-etal-2020-bleurt, zhang-
etal-2020-bertscore]. Ultimately, neural methods are also incredibly expensive
to employ for evaluation settings, potentially taking several days to evaluate
a single test set [nan-etal-2021-improving]. Besides the cost factor, the main
issue with such alternative scores is two-fold: On the one hand, a distinct
advantage of simple co-occurrence-based metrics such as ROUGE is the
simplicity in transferring it to another language. Even basic extensions, such
as stemmers, are readily available in a multitude of languages other than
English. Trained metrics, such as BERTScore [zhang-etal-2020-bertscore] or
QAEval [deutsch-etal-2021-towards], however, are severely limited in their
transferability to other languages, and would require dedicated efforts to
port them to German, for example. On the other hand, recent statistical
analyses have shown that when accounting for annotator expertise, correlation
can vary significantly [fabbri-etal-2021-summeval]. When additionally
controlling for variance and confidence intervals, correlation with human
judgments over ROUGE correlation is only statistically significant in rare
cases [deutsch-etal-2021-statistical]. A particular investigation on metrics
for German summarization was conducted during the second Swisstext challenge
[frefel-etal-2020-german]. However, submitted resources were only marginally
better than ROUGE baselines for judging system quality [paraschiv-
cercel-2020-upb, biesner-etal-2020-hybrid]. For crowd-sourced evaluation
approaches, Iskender et al. [iskender-etal-2020-best] further elaborate on the
importance of survey setups and considerations for expert annotators to ground
evaluation results.

##  3 Assessing the Quality of Summarization Systems

Issue | Reference | Summary
---|---|---
Short text |  Wir verwenden Cookies, um unser Angebot für Sie zu verbessern. Mehr Informationen […]. | –
Duplicates | ‘Virtuelles Bergsteigen mit dem Project360 […] | Leben und Kultur in Europa
Historische Dokumente: Bilder der Wende […] | Leben und Kultur in Europa
Relative Length |  Chef-Sprüche: “Ich sehe meine Kinder auch nur im Urlaub.” |  Die besten Chef-Sprüche zum Thema Überstunden.
Figure 1: List of faulty summarization samples in the MassiveSumm dataset
uncovered by various data checks. Despite checking for unrelated issues, we
notice a trend where filtered samples are of especially low semantic quality,
too.

When using existing models for abstractive text summarization, the expectation
is that they should work “as expected”, meaning that a model provides
appropriate and correct summaries. However, in practice, the automated
collection of samples may lead to insufficient sample quality or systematic
biases in the data. This has further detrimental consequences for models
trained on those datasets.
In this section, we lay out a series of very basic sanity checks for both data
and models, which help to ensure a minimal level of generalization from
experimental results. As we will later find, even such basic data assurances
lead to a significant reduction of valid samples in available German
summarization data.

###  3.1 Data-centric Sanity Checks

The best strategy to achieve decent experimental results is ensuring high
quality in the training data – in line with the popular saying “garbage in,
garbage out”. We present a list of minimal quality checks for individual
samples, as well as dataset-wide assurances of data quality. Most of these
measures are fully automated and at most require single hyperparameter
settings to filter datasets.
Further, suggested data checks are language-independent at their core and can
therefore be applied in basic form to any dataset, even beyond German. This
also implies that no further existing tools or libraries for tokenization,
etc., are required.

##### Empty Samples

The most trivial sanity check is verifying that _both_ the reference text and
summary are present for all samples. This is simultaneously the most prevalent
check implemented by authors of resource papers in our experience. Even so,
several issues can arise for this criterion, primarily revolving around
varying definitions of “emptiness”. For example, one could also consider a
sample as empty if only whitespaces (or whitespace-like symbols, such as
$\backslash$t) are present. Extensions are, for example, faulty encodings or
only special characters in a text (cf., data audit insights by Kreutzer et al.
[kreutzer-etal-2022-quality]).

##### Minimum Text Length

A superset of “empty samples”, imposing a required minimum text length
presents a stricter filtering criterion for sample validity. Where empty texts
are universally to be avoided, hard length requirements are harder to impose,
since the appropriate cutoff depends strongly on the dataset domain. For
domain-specific datasets, e.g., the instruction-like texts in the WikiLingua
dataset [ladhak-etal-2020-wikilingua], having extremely short summaries with
only a few characters (and comparatively short references) may make sense. For
summaries stemming from news articles, however, length requirements imposed on
the reference might ensure a longer minimum text length for quality control.

##### Compression Ratio Filtering

Another key metric used in summarization research is the _Compression Ratio
(CR)_ , defined as the relation between reference text length and summary
length. We follow the definition by Grusky et al. [grusky-etal-2018-newsroom]:
$CR(\text{ref},\text{summ})=\frac{\text{len}(\text{ref})}{\text{len}(\text{summ})}$.
For filtering by compression ratio, a significant difference should be ensured
by establishing a minimum compression ratio. For our purposes, we argue that a
reduction of at least 20% in the summary length is required, which equals
$CR\geq 1.25$. We note that this is not a strict requirement per se and may
depend on domain-specific factors. It can be argued, however, that samples
with summaries longer (or equal) than their respective references (i.e.,
$CR\leq 1.0$) always pose an inadequate sample and must be filtered.
Related work sometimes takes a more drastic approach to compression ratio
filtering, arguing that extreme content reduction may result in a lossy
summary and should therefore also be avoided [urlana-etal-2022-tesum].

##### Duplicate Removal

Some lesser-checked property seems to be the existence of duplicates in
training data, which is also applicable in more general machine learning
settings. However, given that each sample for summarization comes with _two_
distinct texts (the reference and the summary), we can further distinguish
between different instances of duplication. Trivial to consider are instances
of what we call exact duplicates, i.e., samples that have the exact same
combination of reference and summary appearing as another tuple in the
dataset.
We can further expand this idea by three more considerations, which we call
partial duplicates. These are instances where we find either the reference or
summary in other dataset instances. Finally, it could also occur that both
summary and reference are duplicated, but across different samples; such
instances are also considered partial duplicates and are relatively rare.
To understand why duplicates, including partial ones, can be considered
harmful as a training resource, we need only look at the potential effects
during training or evaluation. For exact duplicates, no real gain is achieved
by including one sample several times in the training data. Worse yet, if we
encounter exact duplicates _across different splits_ , this can cause active
falsification of evaluation results (train-test leakage). While partial
duplicates are less severe, we still encourage removal, as they can cause
confusion during the learning process: cases where different input texts
should generate the same summary hamper generalization of models, and the
reverse case of similar input texts generating different summaries conveys
unclear learning signals during training. Finally, we also want to note that
partial duplicates can uncover incorrectly aligned samples (cf. Figure 1).
While _spotting_ duplicates is fairly straightforward, removing duplicate
content is often non-trivial, as there exist several valid strategies for
deduplication, leading to differing results. In an attempt to reduce impact on
smaller test and validation sets, we adopt a “additive” strategy for the
remainder of this work. We start with an empty dataset, and iteratively add
new samples if and only if neither the reference nor the summary have been
previously included.

##### Sample Inspection

Even with all of the proposed automated measures, nothing can ensure data
quality quite as well as manually inspecting data. All of the previous
measures can point to systemic failures in the data collection process, but
may ignore more localized quality issues for particular samples. While a
manual analysis step is not feasible at scale, often enough reviewing few
samples will already reveal tendencies about the underlying data quality. We
generally differentiate between the following strategies to inspect data
samples and their respective up- and downsides:

  1. 1.

Reviewing samples in order: A linear sequence of samples may reveal particular
issues in the consistency of samples, which can be linked to the crawling
process. We emphasize that “linearity” can follow many particular axes, not
just the order in which data is stored. Further possible orderings can be
based on available metadata descriptors, such as sortings by timestamps,
source or length. In-order samples are most likely to uncover systematic
issues, such as incorrect alignment settings that span several samples.

  2. 2.

Reviewing random samples: Another popular approach is to shuffle data and
randomly select instances for review. This is fairly easy to implement and
does not require iterating over the full dataset or sorting operations.
Advantages of random reviews are a more holistic coverage of the data
distribution, but requires potentially more manual reviews to find systemic
failures.

  3. 3.

Outliers and representative samples: If data statistics are already known or
easy to compute, a more targeted approach is to look for distributional
outliers. There are again a variety of metrics that can be considered, with
the most obvious being text length and compression ratio of individual
samples. Manually reviewing outliers can also sharpen the requirements of
expected outputs, e.g., the minimum/maximum length of a summary in relation to
the input text. Related are _representative samples_ , which constitute
instances close to the mean or median of a distribution.

###  3.2 Model-centric Checks

While we have compiled a detailed list of what can be done about checking the
data used for summarization systems, it is significantly harder to judge a
trained system, especially given that many neural methods can only be treated
as black box systems. But even with a lack of clarity around the original
training procedures and model learnings, we can use several probing techniques
to estimate the robustness and performance of systems.

##### Evaluation on Cleaned Test Sets

The standard procedure to evaluate on withheld (but still in-domain) test
sets. While these evaluation approaches may give insights on the overfitting
of trained models, such experiments tend to fall short of giving more concrete
evidence on the pattern of how summaries are generated. This is especially
crucial if no further manual evaluation is performed. Testing models on
modified or generalized test data can serve as a partial remedy to this, by
probing the generalization ability of particular systems. In combination with
the proposed filtering techniques, we suggest the evaluation on cleaned test
sets for models that were trained on the unfiltered training set. The main
advantage is that no additional re-training with altered training sets is
required, and insights can be acquired from a generally much smaller
evaluation set through inference alone. Further, we hypothesize that intrinsic
summarization metrics [narayan-etal-2018-dont, grusky-etal-2018-newsroom,
zhong-etal-2019-closer, bommasani-cardie-2020-intrinsic] applied to _system
summaries_ can be used as a preliminary gauge for text quality in comparison
to the original input. Especially _abstractiveness_ of generated outputs,
essentially constituting the number of novel $n$-grams in summaries, could
indicate changes in the vocabulary.

##### Domain Generalization

An extreme case of the previous point is testing on completely out-of-domain
data, which usually means taking test splits of a different dataset. While
this approach can be useful to evaluate general purpose summarization systems,
the evaluated models in this work all present rather focused domain-specific
summarization systems. For this reason, we refrain from evaluating performance
based on out-of-domain abilities.

##### Factual Consistency

A rather important argument for summarization especially: facts that are
stated in the original reference text must be maintained in the respective
summary. Therefore, models should be measured with respect to their
truthfulness, which has been previously attempted with automatic metrics for
English summarization systems [kryscinski-etal-2020-evaluating], or even
implemented as an optimization target for more truthful summaries [zhu-
etal-2021-enhancing].

###  3.3 Extractive Models and Baseline Systems

Given the relatively one-dimensional approach to evaluation, we should at
least expect additional context for better interpretability of model scores.
In practice though, we rarely find a consistent reporting of baseline scores,
if any comparisons are reported at all. To this end, we strive to provide
consistent baselines and reporting of such in the context of German
abstractive summarization. In addition to the scores, baselines also serve an
important purpose by providing a sensible complexity trade-off: Unlike most
neural methods, they should be able to generate summaries faster and with
fewer parameters than heavy-weight state-of-the-art approaches. Similar to
English works, we therefore fall back on extractive summarization systems,
which – as the name indicates – simply copy text snippets from the reference
to generate a summary.
To our knowledge, the only work that has explicitly worked on extractive
summarization for German is over 20 years old [reithinger-
etal-2000-summarizing]. This does not imply, however, that there is no
dedicated extractive system available. Especially for unsupervised methods,
such as TextRank [mihalcea-tarau-2004-textrank] or LexRank [erkan-
radev-2004-lexrank], language-specific taggers or lemmatizers can easily be
replaced in existing libraries to enable application on German texts as well.
For our experiments, we rely on three variants of baselines, which extract a
specified number of sentences from the input text to generate a summary.
Overall, extractive summaries are guaranteed to ensure a more factually
consistent summary, and have high intra-sentence coherence. On the other hand,
these methods cannot be fine-tuned and rely on singular hyperparameters – the
length of the generated summary. This can still significantly impact the
evaluation performance, but does not factor in domain-specific variance in
text distribution. For all systems, we rely on the sentence splitting module
by spaCy151515We use the model de_core_news_sm in our experiments., unless
datasets provide a pre-split sentence format.

##### Lead-3

The simplest possible baseline system is _lead-3_ , popularized by Nallapati
et al. [nallapati-etal-2017-summarunner] as a simple but strong baseline for
news article summarization. Here, the summary is equal to the first three
sentences of an input text. The method works particularly well for news texts,
where key information has to be conveyed early on to both inform and catch the
interest of a potential reader. The prevalence of this so-called “lead bias”
differs significantly across different domains.

##### Lead-$k$

For other domains, three sentences may underestimate the actual summary
length. For this purpose, Aumiller and Gertz [aumiller-gertz-2022-klexikon]
introduce a variant that extends the lead baseline to the $k$ leading
sentences, in their particular context the full first paragraph of a Wikipedia
page. Given that in general, datasets do not contain paragraph-level
information, the authors later extended this baseline and instead consider an
approximate $\hat{k}$ for each sample by using the average compression ratio
[aumiller-etal-2022-eur]:

$\displaystyle\hat{k}=\frac{\text{len(reference)}}{CR_{\text{avg}}},$ (1)

where len(reference) is the number of sentences in the summary, and
$CR_{\text{avg}}$ denotes the average compression ratio across the _training_
split of a dataset.

##### Modified LexRank (LexRank-ST)

A more complex baseline that also considers sentences at other positions of
the article is a modification of LexRank [erkan-radev-2004-lexrank], similarly
used by Aumiller et al. [aumiller-gertz-2022-klexikon, aumiller-
etal-2022-eur]. The key modification lies in exchanging the centrality
computation – which is originally based on pure occurrence counts – with dense
sentence embeddings obtained through sentence-transformers [reimers-
gurevych-2019-sentence, reimers-gurevych-2020-making]. While the underlying
neural model can be of arbitrary complexity, it does not need to be trained
further to work in the summarization application. After scoring individual
sentences, the highest-ranking $k$ sentences are selected as the summary; we
use the same method for estimating an optimal length $\hat{k}$ as for the
lead-$k$ baseline.

Finally, we also point towards oracle extractive summaries as a form of upper-
bound for extractive summarization, which can be computed from greedy ROUGE-2
alignments [nallapati-etal-2017-summarunner, glaser-etal-2021-summarization].
Given that we focus on _abstractive_ results in this work, we omit the
computation of extractive oracle summaries.

##  4 A Sober Look at State-of-the-Art Results

Given the presented set of tools, we now set out to put current models’
capabilities into a better context. To this end, we conduct a set of four
experiments: We start by applying the filters introduced in Section 3.1 to
available German summarization datasets, noting varying size reductions as a
result. To remedy the changes introduced by our filtering, we re-compute a set
of strong baselines as updated results for datasets with available validation
and test sets. Further, given the previously uncovered discrepancies in some
datasets, we repeat more comprehensive experiments on MLSUM and MassiveSumm
across the pre- and post-filtered dataset to highlight the effect of filtering
on ROUGE scores. We are able to show that this change in data quality also
significantly impacts the reproducibility of results. Finally, we provide a
small case study in which we examine a subset of generated samples that
highlight some of the particular model-centric issues.

###  4.1 Filtering Datasets

> Key Finding 1: German subsets of two popular multilingual resources (MLSUM
> and MassiveSumm) have extreme data quality issues, affecting more than 25%
> of samples across all splits.

Table 2 presents our findings for filtering the available German summarization
datasets; hyperparameters for filters are specified in the table caption. We
refrain from imposing any particularly strict filtering metrics, particularly
for the length of texts. Most concerning is the fraction of affected samples
in MLSUM, given its popularity as a training resource for many public models.
While a strong lead bias is to be expected due to the domain of these samples
being exclusively news articles, the eventual performance of models trained on
the unfiltered dataset is severely impacted; a finding that we confirm in
subsequent experiments. Primarily, it indicates that for fully extractive
samples, summaries can be generated by directly running an extractive
summarization system, and thus obtain similar (or better) quality at a much
lower cost. For MassiveSumm, a large fraction of invalid samples can be
attributed to duplicate content; manual inspection reveals that there are
frequent generic references or summary texts, such as “ _Read more after
logging in!_ ”. We assume the reason to be a faulty extraction of HTML
elements for particular websites.
The remaining inspected datasets were affected at a much lower rate; we see
several subsets that have only a handful of faulty instances. Depending on the
overall size of the dataset, this implies that evaluation scores will differ
less between unfiltered and filtered splits of largely unaffected datasets.

|  |  | Min Length |  | Min | Fully | Duplicates |  |
---|---|---|---|---|---|---|---|---|---
Dataset | Split | Samples | Ref | Summ | Id | CR | Extr | Exact | Ref | Summ | Valid Samples
| Train | $220{,}887$ | $0$ | $0$ | $39$ | $30$ | $\mathbf{126{,}204}$ | $31$ | $45$ | $105$ | $94{,}433$ | $(42.75\%)$
MLSUM | Val | $11{,}394$ | $0$ | $0$ | $0$ | $0$ | $\mathbf{3{,}285}$ | $1$ | $1$ | $5$ | $8{,}102$ | $(71.11\%)$
| Test | $10{,}701$ | $0$ | $0$ | $0$ | $0$ | $\mathbf{3{,}306}$ | $1$ | $5$ | $2$ | $7{,}387$ | $(69.03\%)$
MassiveS | Train | $478{,}143$ | $253$ | $\mathbf{16{,}294}$ | $0$ | $\mathbf{33{,}959}$ | $0$ | $805$ | $\mathbf{73{,}886}$ | $4{,}882$ | $348{,}064$ | $(72.79\%)$
Swisstext | Train | $100{,}000$ | $0$ | $0$ | $0$ | $0$ | $3$ | $0$ | $0$ | $2$ | $99{,}995$ | $(100.00\%)$
WikiLing | Train | $58{,}341$ | $11$ | $0$ | $0$ | $\mathbf{1{,}435}$ | $0$ | $4$ | $2$ | $52$ | $56{,}837$ | $(97.42\%)$
| Train | $2{,}346$ | $0$ | $0$ | $0$ | $10$ | $0$ | $0$ | $2$ | $0$ | $2{,}334$ | $(99.49\%)$
Klexikon | Val | $273$ | $0$ | $0$ | $0$ | $1$ | $0$ | $0$ | $0$ | $0$ | $272$ | $(99.63\%)$
| Test | $274$ | $0$ | $0$ | $0$ | $1$ | $0$ | $0$ | $0$ | $0$ | $273$ | $(99.64\%)$
| Train | $1{,}115$ | $0$ | $0$ | $0$ | $18$ | $0$ | $0$ | $0$ | $0$ | $1{,}097$ | $(98.39\%)$
EUR-Lex | Val | $187$ | $0$ | $0$ | $0$ | $0$ | $0$ | $0$ | $0$ | $0$ | $187$ | $(100.00\%)$
| Test | $188$ | $0$ | $0$ | $0$ | $0$ | $0$ | $0$ | $0$ | $0$ | $188$ | $(100.00\%)$
| Train | $79{,}937$ | $0$ | $2$ | $0$ | $12$ | $326$ | $233$ | $95$ | $\mathbf{3{,}106}$ | $76{,}163$ | $(95.28\%)$
LegalSum | Val | $9{,}992$ | $0$ | $0$ | $0$ | $4$ | $32$ | $14$ | $2$ | $157$ | $9{,}783$ | $(97.91\%)$
| Test | $9{,}993$ | $0$ | $0$ | $0$ | $7$ | $33$ | $8$ | $1$ | $59$ | $9{,}885$ | $(98.92\%)$
Table 2: German text summarization datasets in numbers. Given are the original
sample count and breakdown of filtered samples by automated assessment (cf.,
Section 3.1) for all provided splits. We set the _Minimum Length_ to 20
characters for summaries and 50 for references, except for WikiLingua, which
has limits of 8 and 20 characters, respectively, due to a different domain.
_Id_ refers to samples with same reference and summary text, _Min CR_ ensures
references are at least 25% longer than summaries, and _Fully Extr_ identifies
consecutive segments that are used as fully extractive summaries. For
duplicates, we differentiate between both reference and summary appearing in
the corpus (_Exact_), versus partial duplicates where only one of reference
(_Ref_) _or_ summary (_Summ_) are appearing elsewhere. Numbers in bold
highlight issues affecting more than 2% of the split data.

###  4.2 Consistent Results and Baseline Runs

> Key Finding 2: Existing evaluation scores are hard (if not outright
> impossible) to reproduce, even with model weights publicly available.
> Key Finding 3: Authors frequently fail to put scores into context, not
> comparing their own results against baseline methods for further scrutiny.

Another worrying trend we observe in the “reproducibility” column of Table 1,
is the consistent inability to even approximately reproduce self-reported
scores for any of the evaluated models. In our reproduction attempts, we
employed no particular further filtering, and observed scores that were
anywhere from 5 points worse to 3 points better than self-reported ones on the
test set. Only a singular result was reproducible within 0.5 ROUGE points of
the expected results. In particular, we find that implementation details on
filtering steps and other subselection criteria are rarely (if ever) included
in the documentation of training procedures. While the usage of so-called
“model cards” [mitchell-etal-2019-model], i.e., dedicated documentation pages
for particular training results, has improved the availability of at least
_some form_ of documentation, these descriptions are still insufficient to
fully reproduce results. As a side note, it should also be mentioned that
multiple implementations for the ROUGE evaluation metric exist161616e.g.,
rouge-score (https://pypi.org/project/rouge-score/, last accessed: 2022-10-06)
or pyrouge (https://pypi.org/project/pyrouge/, last accessed: 2022-10-06),
which may result in scoring differences by utilizing different text processing
tools or implementations. To ensure reproducibility of our own scores, we
mention that scores were computed with help of the rouge-score package,
version 0.1.2. We further replaced the default stemming algorithm with the
German Cistem stemmer [weissweiler-fraser-2017-cistem] to provide a reasonable
upper-bound of scores and use the provided bootstrap sampler with $n=2000$.

Aside from the lack of reproducible results, we also noted that only few
public models report against a set of (consistent) baselines, with the most
commonly compared approach being lead-3. Given that we have also presented a
cleaned portion of popular evaluation models, we strive for a more
comprehensive comparison of actual results, and investigate resulting
implications that were omitted in the original evaluation settings.
In our particular setup, we compare against the three mentioned extractive
baselines mentioned in Section 3.3 and report scores in Table 3. Depending on
the dataset, the choice of a baseline can heavily skew the interpretation
compared to neural methods. For example, on the Klexikon dataset, using
lead-$3$ can lead to a roughly 12-13 point drop in ROUGE-1 scores compared to
scores by the lead-$k$ or LexRank-ST baseline. On the other hand, for lead-
heavy and short texts in MLSUM, lead-3 serves as the best baseline method. Our
recommendation is therefore to similarly use multiple (different) baseline
approaches, resulting in a more defined context for evaluation based on ROUGE
scores. While it may be easier to simply copy results from prior work, we
highly recommend the reproduction of these results first, as scores may
ultimately vary between different experimental setups.

|  | Validation Set | Test Set
---|---|---|---
Dataset | Method | R-1 | R-2 | R-L | R-1 | R-2 | R-L
| lead-3 | $19.06$ | $5.58$ | $13.21$ | $18.90$ | $5.47$ | $13.04$
MLSUM | lead-$k$ | $14.93$ | $4.12$ | $11.31$ | $15.08$ | $4.17$ | $11.45$
| LexRank-ST | $15.78$ | $3.36$ | $11.52$ | $16.04$ | $3.30$ | $11.55$
| lead-3 | $15.19$ | $3.46$ | $9.10$ | $15.87$ | $3.64$ | $9.35$
Klexikon | lead-$k$ | $28.11$ | $5.51$ | $12.43$ | $28.34$ | $5.50$ | $12.50$
| LexRank-ST | $27.23$ | $4.63$ | $11.48$ | $27.42$ | $4.58$ | $11.55$
| lead-3 | $16.72$ | $2.80$ | $10.51$ | $16.74$ | $2.86$ | $10.53$
LegalSum | lead-$k$ | $14.34$ | $2.27$ | $8.78$ | $14.36$ | $2.34$ | $8.78$
| LexRank-ST | $21.54$ | $6.22$ | $12.97$ | $21.35$ | $5.99$ | $12.74$
| lead-3 | $3.31$ | $2.25$ | $2.72$ | $3.31$ | $2.19$ | $2.67$
EUR-Lex-Sum | lead-$k$ | $41.74$ | $17.77$ | $16.04$ | $39.42$ | $17.08$ | $15.52$
| LexRank-ST | $39.37$ | $15.13$ | $15.26$ | $38.48$ | $15.18$ | $15.19$
Table 3: Baseline results for all datasets with available validation and/or
test splits. We report ROUGE F1 scores on the filtered datasets.

###  4.3 Impact of Data Filtering

> Key Finding 4: After filtering, scores can drop by more than 20 ROUGE-1
> points on the MLSUM test set.

To illustrate the effect of dataset filtering on downstream performance, we
further compare results on the two most-affected datasets (MLSUM and
MassiveSumm). Without any additional training, we run all available public
models on the validation and test portion of MLSUM, for which we also obtain
scores on the original unfiltered sets. Our findings can be seen in Table 4,
where one can observe a performance drop in _every model_ , even those that
were not originally trained on the MLSUM dataset itself (_t5-base_). By far
the worst affected are the two baselines constructed from leading sentences,
as well as the mT5-small models by users _mrm8488_ and _Shahm_. These four
models all achieve unreasonably high ROUGE-2 scores before filtering and see a
reduction to about one fifth of the original scores after filtering. Upon
inspection, we similarly found that these models were ultimately simply re-
generating the first tokens from the input article. These findings are
concerning, as they ultimately question the current state-of-the-art on the
MLSUM dataset. It further validates the necessity of filtering, given that we
can ultimately change the course of evaluation and interpretation of models.
For MLSUM, per our results, the _t5-base_ model, trained on a related news
dataset and utilizing the largest underlying neural model, seems to perform
best on filtered datasets while originally lagging behind even a simple lead-3
baseline. This is particularly interesting, because the underlying model
checkpoint used is primarily trained on English texts.

(a) Reference Text Lengths

(b) Summary Text Lengths

Figure 2: Violin plots illustrating the distributional shift on the MLSUM and
MassiveSumm training splits through filtering. Black dashed lines indicate
mean and quartiles of the distribution.

Figure 2 further visualizes the impact of filtering on the length
distributions of the two heavily affected datasets, MLSUM and MassiveSumm.
Analyzing the resulting changes in more detail, we can observe a more strictly
enforced minimum length for both references and summaries in the MLSUM dataset
even before filtering. In stark contrast, MassiveSumm is shrunk considerably
by the minimum length filter, which in turn shifts the samples towards
generally longer reference texts. Since MLSUM is affected more by the
extractiveness filter, one can observe a noticeable change in the mean of the
distribution of summary texts, particularly longer ones.
Changes in the length distribution, however, do not explain any of the
deterioration in raw ROUGE scores; a further indicator that several different
evaluation methods need to be combined in order to paint a more complete
picture for the realistic performance of models. We particularly recommend the
utilization of violin plots also for the evaluation of system outputs, as they
allow the comparison of length estimates by the system in comparison to ground
truth data.

###  4.4 Qualitative Analysis of Generated Summaries

> Key Finding 5: With the exception of one work [aksenov-
> etal-2020-abstractive], no _publicly_ available system performs experiments
> beyond simple ROUGE score computation.
> Key Finding 6: Despite high reported scores, catastrophic failures can be
> observed in some systems.
> Key Finding 7: All utilized architectures only work with a relatively
> limited context, proving to be incapable of dealing with long-form
> summarization.

The first criterion we were looking at when checking for existing systems is
the evaluation setting that was used in the respective work. The findings,
reported in Table 1, point towards a more rigorous evaluation setting for
models backed by a scientific publication, which comes as no surprise.
However, we also note that these systems are also more likely to withhold
their respective models from public access. This ultimately means that those
models can only be judged based on the reported evaluation and no further
black-box model checks can be performed on them. To aggregate the insights
gained across these works, most frequently mentioned is the issue of factual
consistency [venzin-etal-2019-swisstext, fecht-etal-2019-swisstext], which
does not bode well for the practical suitability of such systems beyond simple
settings. Secondly, several works also investigate system outputs’ fluency
[fecht-etal-2019-swisstext, aksenov-etal-2020-abstractive], where abstractive
models could provide sensible improvements over extractive systems. However,
especially for earlier works, consistent generations from language models
still prove to be difficult.

| MLSUM Validation Split | MLSUM Test Split
---|---|---
| Unfiltered | Filtered | Unfiltered | Filtered
Model | R-1 | R-2 | R-L | R-1 | R-2 | R-L | R-1 | R-2 | R-L | R-1 | R-2 | R-L
Lead-3 | $36.22$ | $26.24$ | $31.89$ | $19.06$ | $5.58$ | $13.21$ | $37.15$ | $27.48$ | $32.94$ | $18.90$ | $5.47$ | $13.04$
Lead-$k$ | $29.25$ | $20.92$ | $26.51$ | $14.93$ | $4.12$ | $11.31$ | $31.35$ | $22.86$ | $28.58$ | $15.08$ | $4.17$ | $11.45$
LexRank-ST | $18.62$ | $6.46$ | $14.26$ | $15.78$ | $3.36$ | $11.52$ | $18.83$ | $6.45$ | $14.36$ | $16.04$ | $3.30$ | $11.55$
mrm8488 | $\mathbf{42.77}$ | $31.89$ | $\mathbf{38.93}$ | $21.63$ | $6.64$ | $16.32$ | $\mathbf{44.05}$ | $33.44$ | $\mathbf{40.36}$ | $21.31$ | $6.36$ | $16.09$
ml6team | $28.17$ | $18.81$ | $26.05$ | $17.08$ | $5.03$ | $14.18$ | $28.51$ | $19.52$ | $26.53$ | $16.56$ | $4.80$ | $13.78$
T-Systems | $23.74$ | $11.08$ | $20.34$ | $19.87$ | $6.49$ | $16.40$ | $23.67$ | $11.21$ | $20.36$ | $19.20$ | $6.11$ | $15.84$
Shahm | $42.59$ | $\mathbf{31.96}$ | $38.70$ | $21.50$ | $6.87$ | $16.15$ | $43.92$ | $\mathbf{33.62}$ | $40.09$ | $21.20$ | $6.62$ | $15.79$
t5-base | $27.54$ | $11.31$ | $20.88$ | $\mathbf{23.31}$ | $\mathbf{7.19}$ | $\mathbf{16.99}$ | $27.99$ | $11.65$ | $21.20$ | $\mathbf{23.40}$ | $\mathbf{7.20}$ | $\mathbf{16.91}$
Table 4: ROUGE F1 scores on the MLSUM validation and test splits, comparing
results with and without data filtering. Across all tested models, a stark
drop in performance can be observed. We highlight the highest score for each
split in bold.

To follow our own advice, we manually investigated instances of generated
outputs from systems in Table 4. In addition to samples from the MLSUM
dataset, we further tested with instances from the Klexikon and WikiLingua
datasets to check for domain generalization. As others have noted, the factual
consistency of abstractive systems is questionable at best, but understated
just how badly summaries can deviate from the original. Several times a
reversed order of aggressors and victims (respectively, winners or losers in
sports game) was generated, and in one particular instance the context was
altered from “live-saving” to “drowning (someone)” by the summarization
system. This happened on “in-domain samples” from the MLSUM test set.
A similar observation can be made for the syntactic quality of generations,
where overfitting of systems becomes particularly apparent during the zero-
shot evaluation on other datasets. While it can be expected that the quality
of a generated summary may lack in content accuracy or truthfulness,
oftentimes no coherent sentence was provided. Less tragic, but difficult for
system comparison, is the multitude of parameters for generation functions.
While self-reported scores of public models generally rely on greedily decoded
summaries, one model frequently started repeating short sequences of about
three words indefinitely until the maximum generation length was reached.
Importantly, such repetitions are not obvious from looking at a ROUGE-based
evaluation of model outputs alone, but could be easily suppressed by enabling
$n$-gram-based filtering during the generation.
We were also able to verify that the highly-scoring models by users _mrm8488_
and _Shahm_ indeed only copy the leading tokens from the input samples, likely
due to training on unfiltered MLSUM splits. This spells further trouble for
“state-of-the-art” models, as it requires a deeper examination for determining
which summaries are actually better than simple string selection approaches,
such as lead-3. We hypothesize that the same concept used in our
_extractiveness_ filter can also be applied to generated outputs; with a
slightly altered similarity scoring mechanism, e.g., the longest common
subsequence algorithm, even near duplicates could be detected and flagged for
manual review.

Most prominently though, due to architectural constraints of the underlying
neural models, none of the currently public systems is able to capture an
input context beyond 512 subword tokens, the default length limit for the
Transformer architectures [vaswani-etal-2017-attention]. In the instance of
domain-specific datasets, such as EUR-Lex-Sum, this means that even the length
of summary texts exceeds the limitation of models, effectively rendering them
useless in this particular context.

##  5 Conclusion and Future Work

Studying the current landscape of German abstractive summarization initially
paints a grim picture: While the general willingness and ease of sharing
systems has greatly increased over the past years, around half of the
currently known German summarization systems still remain inaccessible to the
public. Of those that _are_ available for public scrutiny, a prominent focus
on news summarization is still persisting, preventing more broader
applications. Even worse, the most prominent dataset contains severe flaws in
the sample quality, leading to models whose generalization capabilities, even
in-domain, are severely hampered by the unfiltered data. This also hints at
the general level of care practitioners take with respect to exploratory data
analysis, given that several issues can be spotted by simply inspecting just a
few samples. And finally, even models that take care of filtering some of
these issues, a qualitative analysis of generations can still reveal
catastrophic problems that prevent an ethically responsible deployment of the
solution in practice.

However, there are some silver linings at the horizon. Many of the major data-
centric issues can be easily fixed with the introduced quality checks, which
can be applied cost-effectively across multiple datasets, as we have
demonstrated in this work. Through publishing our pre-processing pipeline, we
hope to encourage others in taking a more data-centric exploration before
starting with the ultimate model training.
Within just two years, we have also seen an unbelievable influx of available
summarization datasets for German, importantly extending past the narrow
domains into application-specific fields, such as law and medicine, and
totaling more than 700.000 samples across publicly available resources. This
hopefully paves the way towards a more consistent and generalized approach in
German abstractive summarization research; should the efforts of the community
keep at the current rate, we will likely see meaningful progress within the
next year. The latest trends in the English summarization community also
indicate a shift towards greater awareness of long-form summarization [phang-
etal-2022-investigating]; while dedicated long context German (or
multilingual) model checkpoints are still absent, we estimate that such
systems will become available shortly, serving as a compute-intensive way to
escape the current restrictions on input length.

As for our own efforts, we are currently investigating how systems can be
designed to work well across multiple domains at once, without the need for
several distinct models. This requires careful analysis of the underlying
data, as well as a more agnostic training framework to prevent overfitting
towards a particular style.


# Emotion-driven Piano Music Generation via Two-stage Disentanglement and
Functional Representation

###### Abstract

Managing the emotional aspect remains a challenge in automatic music
generation. Prior works aim to learn various emotions at once, leading to
inadequate modeling. This paper explores the disentanglement of emotions in
piano performance generation through a two-stage framework. The first stage
focuses on valence modeling of lead sheet, and the second stage addresses
arousal modeling by introducing performance-level attributes. To further
capture features that shape valence, an aspect less explored by previous
approaches, we introduce a novel functional representation of symbolic music.
This representation aims to capture the emotional impact of major-minor
tonality, as well as the interactions among notes, chords, and key signatures.
Objective and subjective experiments validate the effectiveness of our
framework in both emotional valence and arousal modeling. We further leverage
our framework in a novel application of emotional controls, showing a broad
potential in emotion-driven music generation.

##  1 Introduction

With the recent advancements in symbolic music generation [1, 2, 3, 4, 5, 6],
there has been a growing interest in controlling high-level musical features
throughout the generation process. Among these features, _emotion-driven music
generation_ [7, 8, 9, 10, 11, 12, 13] aims to generate music that conveys
specific emotions, representing a crucial aspect for music appreciation and
analysis. The downstream applications of such models have also been explored,
such as music therapy for healthcare and educational purposes [14] and
soundtrack generation for videos and movies [15].

Emotion could be represented in two dimensions from the literature [16]:
_valence_ and _arousal_. Valence refers to the positiveness of an emotion and
arousal refers to energy or activation [17, 18, 19]. These two dimensions can
be further divided into four quadrants (4Q), namely high valence high arousal
(Q1), low valence high arousal (Q2), low valence low arousal (Q3), and high
valence low arousal (Q4). In this paper, we focus on the emotion-driven _piano
performance generation_ of these four quadrants. Throughout prior works, we
observe crucial challenges from the perspectives of both model design and
musical inductive bias.

First, previous emotion-driven piano performance generation models [7, 8]
attempt to learn emotion quadrants and expressions in an _end-to-end_
paradigm. In terms of model design, this approach poses training difficulty on
the generation model, leading to the instability in achieving results of
desired emotions. For example, many existing works [7, 9, 11] could
effectively control the arousal levels of music, while their performance of
_valence modeling_ , especially in generating low valence (i.e., negative)
music, is still poor. In terms of music, the creation process of music
typically involves multiple stages, such as the _lead sheet composition_ for
melodies and chord progressions, and _performance generation_ for textures and
expressiveness. Consistently, emotion can be evoked through a combination of
musical elements (e.g., melody, chord, texture). For example, major/minor
chords have been found to seize different valence trends in psychological
studies [20] and performance-level attributes like articulation, tempo, and
velocity are more related to arousal [21, 22]. It is worth to explore the
potential relation between the _disentanglement_ of the generation process and
the emotion expression.

Second, previous emotion-driven generation models have received limited
attention regarding the influence of _tonality_ on emotion modeling. It has
been widely shown that major-minor tonality in composition is highly related
to valence perception [23, 22, 24, 25]. For example, as depicted in Figure 1,
the histogram of musical keys derived from the emotion-labeled music dataset
EMOPIA [7] supports the distribution skews to major keys for high valence
clips and opposite trend for low valence ones. Furthermore, different
tonalities may reveal similar patterns in the relative relationships between
melodies and chords, while the distribution of melodies, chords, and
tonalities can exhibit distinct shapes across different emotions. Current
representations of symbolic music, such as REMI [2] and CP-Word [26], do not
explicitly incorporate such interactions nor address its connection to emotion
adequately.

Figure 1: Key histogram of high/low valence clips from the emotion-labeled
piano music dataset EMOPIA [7].

Therefore, it is necessary to consider a functional format of symbolic music
representation considering the relationships between notes, chords and key
signatures to better model the tonality in the emotion-driven music generation
process.

In this paper, we contribute to combat above challenges:

  * •

We employ a two-stage Transformer-based model on emotion-driven piano
performance generation. The first stage focuses on valence modeling via lead
sheet composition, while the second stage addresses arousal modeling by
introducing performance-level attributes.

  * •

We propose a novel functional representation for symbolic music, encoding both
melody and chords with Roman numerals relative to musical keys, to consider
the interactions among notes, chords and tonalities [27].

  * •

Experiments demonstrate the effectiveness of our framework and representation
on emotion modeling. Additionally, our method enables new capabilities to
control the arousal levels of generation under the same lead sheet, leading to
more flexible emotion controls.

As a minor contribution, we also refine key signature labels and extract lead
sheet annotations for the EMOPIA dataset [7] to ensure the correct training of
the two-stage framework. We share the data, open source our
code111https://github.com/Yuer867/EMO-Disentanger and present generation
samples in the demo page.222https://emo-disentanger.github.io/

##  2 Related Work

###  2.1 Emotion-driven Piano Performance Generation

Prior works apply emotion conditions on deep-learning models to guide the
generation of piano performance [11, 7, 8], or develop searching methods to
generate music of desired emotions [28, 29]. Musical elements via feature
disentanglement [9] or supervised clustering [10] can further be regarded as a
bridge between emotion labels and performances for generation. In contrast,
our framework employs a two-stage generation approach to reduce the
complexities of one-stage generation, fostering a more nature process of music
creation as well as a better incorporation between emotion labels and
generation results.

###  2.2 Tonality, Functional Harmony, and Emotion

Musical keys and functional harmony have been explored in the field of roman
numeral analysis [30, 31, 32]. The analysis of how modes and tonalities relate
to mid-level perceptual features (e.g., dissonance, tonal stability,
minorness) and affect the emotional perception of music pieces has also been
discovered [22, 24].

While some music generation works attempted to combine key information into
data representation [33], loss function [34] and text conditions [6], none of
them explore the relation between musical keys and emotional perception. In
this paper, we leverage both functional harmony knowledge and class-octave
based pitch representation [35] to design a new data representation,
incorporating the relationships between notes, chords and keys for emotion-
driven music generation.

##  3 Method

In this section, we will first introduce the functional representation of
symbolic music as the main generation unit. Then we introduce the two-stage
model as the main component of the emotion disentanglement and generation.

###  3.1 Functional Representation

Figure 2 illustrates our proposed functional representation. Its design is
initially based on REMI [2], a widely used event-based representation for
symbolic music. We incorporate different note and chord events assisting to
better learn the joint information of emotion and key signature.

Figure 2: Illustration of (a) REMI [2], (b) the proposed functional
representation, and their differences.

####  3.1.1 Emotion and Key Events

We follow CTRL [36] to set up the condition within the autoregressive
generation process in Transformer architecture. To denote distinct emotions
and affect overall properties, we begin the event sequence with <Emotion_*>
event to indicate the emotion label of music clips. The <Key_*> event is
appended after <Emotion_*> to provide the musical key property, with the total
of 24 keys (12 tonic notes with two modes in EMOPIA [7]).

####  3.1.2 Bar, Sub-Beat, Tempo and EOS Events

Similar to REMI, a <Bar> event denotes the new start of a bar; a <Sub-Beat_*>
event denotes one of 16 possible discrete beat locations within a bar; a
<Tempo_*> event denotes local tempo changes every four beats; and an <EOS>
event denotes the end of sequence.

####  3.1.3 Chord Events

A musical chord name typically consists of root note and chord quality. For
example, Fmaj represents the chord F-A-C with root F and major quality. Such
symbols describe correct note information in chord within the tonality, but
they overlook the variations in _chord functions_ of the same chord across
different tonalities. For example, while Fmaj serves the tonic function in F
major scale, it serves the subdominant function in C major scale. Moreover,
the chord progression follows these functional harmony rules to establish
tonality and convey musical emotion [27].

To introduce chord functions in the emotion modeling, we adopt Roman numerals
from Roman Numeral Analysis [31] to notate chord roots in Figure 3. Given the
<Key_*> event, root notes in the absolute pitch are directly converted into
Roman numerals based on their scale degrees relative to the key (i.e.,
relative pitch). For roots outside the scale, we employ a direct conversion
for I#, II#, IV#, V# and VI# appearing in major keys, but randomly assign III#
and VII#, which only appear in minor keys, as one of their neighboring degrees
during the encoding and decoding process. This design ensures the notation to
be key-independent and make every conversion of notes reasonable to the music
theory. The notations of chord qualities remain unchanged, and the chord event
<Chord_*> appears every four beats.

Figure 3: The conversion between letters and Roman numerals in the cases of C
major and c minor scales. Solid arrows denote strict one-to-one conversions,
and dotted arrows denote optional one-to-either conversions.

####  3.1.4 Note-related Events

A note is denoted by <Pitch_*>, <Duration_*> and <Velocity_*> events, where
<Pitch_*> event indicates the onset of pitches from A0 to C8. Inspired by [35,
37], we decompose <Pitch_*> into <Octave_*> and <Degree_*> events according to
the note octave and degree in the certain key scale. The conversion rule from
<Pitch_*> to <Degree_*> is the same as that of chord roots in Figure 3. For
example, pitch D#4 is decomposed into <Octave_4> and <Degree_III> in c minor
scale, but <Degree_I> in D# major scale. Such degree-octave pitch
representation narrows the difference between melodies, thus improves the
learning of connections between emotions, chords, and melodies, as
demonstrated in Figure 4.

###  3.2 Two-stage Emotion Disentanglement

We use the idea of Compose & Embellish [38] to generate music in two stages:
lead sheet first, and then piano performance. While Compose & Embellish is
emotion-agnostic, we extend it so that the lead sheet model involves _valence
modeling_ and the performance model _arousal modeling_.

Figure 4: Two lead sheet examples from different songs in EMOPIA. In our
functional representation, they have the same melody events (green), but
different chord events (yellow) by different emotions (Positive and Negative
by pink) and keys (D major or c minor by purple). Figure 5: The two-stage
framework of emotion-driven piano performance generation. Squares with
transparent background denote the tokens that are not included in the loss
computation during the training phase.

####  3.2.1 Valence Modeling

The top left section of Figure 5 denotes the first stage, where only emotion
events <Emotion_Positive> and <Emotion_Negative> are considered as conditions.
The former includes music pieces of Q1 and Q4 (high valence) and the latter
includes those of Q2 and Q3 (low valence). The lead sheet model first predicts
a key event $k$ conditioned on the given emotion event $e$, and then generates
the lead sheet sequence $M=\\{m_{1},\cdots m_{T}\\}$ of length $T$, as melody
and chord progression, conditioned on previous tokens step-by-step:

$p(k,M|e)=p(k|e)\prod_{t=1}^{T}p(m_{t}|e,k,M_{<t})\,,$ (1)

where $p(k|e)$ and $p(m_{t}|e,k,M_{<t})$ are jointly learned through the
Transformer-based generation model [26, 38]. Performance-related events
<Velocity_*> and <Tempo_*> are removed in the first stage (i.e., lead sheet
generation), as we mainly focus on the contributions of key, pitch and chord
for valence perception.

####  3.2.2 Arousal Modeling

The top right section of Figure 5 denotes the second stage. Given the lead
sheet $M$, the performance model generates performance $X$ conditioned on the
true emotion label (Q1 to Q4). As the valence aspect has already been modeling
in the first stage, this stage focuses on the generation of musical textures
for the lead sheet, and more importantly, on how to perform it through
variations of tempo, velocity, articulation, and other performance-level
attributes that largely influence perceived arousal [22, 21]. During the
training and inference phases, with the positions of <Bar> events, $M$ and $X$
are further segmented into $\\{M_{1},\cdots,M_{b}\\}$ and
$\\{X_{1},\cdots,X_{b}\\}$, where $b$ is the number of bars. The segmented
sequences are “interleaved” in the form of $\\{\cdots$<Track_M>, $M_{i}$,
<Track_X>, $X_{i}\cdots\\}$ with additional <Track_*> events to distinguish
$M$ and $X$ tracks. In that, the target performance bar $X_{i}$ is appended to
its corresponding conditions $M_{i}$, as mapping each lead sheet segment to
its corresponding performance segment [26]. With the emotion condition and key
event from lead sheet as prefix tokens, the performance model is summarized as

$p(X|e,k,M)=\prod_{i=1}^{b}p(X_{i}|e,k,M_{\leq i},X_{<i})\,.$ (2)

####  3.2.3 Training Objectives

Lead sheet and performance models are trained separately by both optimizing
the negative log-likelihood loss of the sequence. Since existing emotion-
labeled music datasets are not large, we leverage large-scale music datasets
without emotion annotations to _pretrain_ both models for better music
understanding. During pretraining, the emotion event is marked as
<Emotion_None>. We then finetune two models on the emotion-labeled dataset
(detail in Section 4) to learn composition and performance styles specific to
different emotion contexts.

####  3.2.4 Two-stage Inference

The left bottom section of Figure 5 denotes the inference process of both
models. In the first stage, the lead sheet model predicts the key event and
generates the lead sheet sequence step-by-step given <Emotion_Positive> or
<Emotion_Negative> event, creating a musical motif for the specific valence
preference. Even though our framework has the capability to generate any-key
music of specific emotions, we observe that some generation results, such as a
high-valence and high-arousal song with a minor key scale, may go beyond the
current definition of emotion in [16], where the valence naturally has a
strong correlation to the major-minor tonality (Figure 1). Therefore, we limit
major keys to <Emotion_Positive> and minor keys to <Emotion_Negative> _during
the inference stage_. We acknowledge that this can be overly simplifying.
Since this paper focuses mainly on the valence-arousal disentanglement during
the generation process, we leave this exploration of generating any emotion
within any key as an advanced topic for future research.

In the second stage, the performance model generates piano performance with
desired valence and arousal combination given the lead sheet from the first
stage. For example, to generate a music piece of Q3, a “Negative” lead sheet
and a “Q3” emotion event are selected as conditions. Additionally, this two-
stage framework enables the flexibility to generate different arousal levels
of piano performance under the same lead sheet, delivering some scenarios when
the music need to shift quickly to complement the scenes in movies or daily
videos (detail in Section 4 and the demo page).

##  4 Experiments

###  4.1 Datasets and Preprocessing

As presented in Table 1, we collect different datasets for pretraining and
finetuning phases as mentioned in Section 3.2.3. For pretraining the lead
sheet model, we use the HookTheory dataset [39, 40], where we choose 18,206
lead sheets with high-quality and human-transcribed melody, chord and key
annotations in 4/4 time signature. We simplify 249 chord quality classes into
11 types333Major, minor, augment, diminish, suspend2, suspend4, major7,
minor7, dominant7, diminish7, half-diminish7 as the same set in the other
datasets below. For pretraining the performance model, we use the Pop1k7
dataset [26], consisting of 1747 transcribed pop piano performances. Since
Pop1k7 does not contain lead sheet annotations, we refer [38] to extract
melodies using the skyline algorithm [41], recognize chords using the chorder
library [42], and detect key signatures using [43] in MIDI Toolbox[44].

For finetuning the models with emotion conditions, we use the EMOPIA dataset
[7], consisting of 1,071 music clips with human-annotated emotion labels.
Similar to Pop1k7, we obtain the lead sheets of EMOPIA by extracting melodies
using the algorithm in [45] and recognizing chords using the algorithm in[46].
Empirically, we obverse that specifically in the EMOPIA dataset, melodies and
chords extracted by these alternative algorithms are more correct compared to
the skyline algorithm and the chorder library. Additionally, we found the key
signature labels in EMOPIA are not fully correct since they are also obtained
by the detection algorithm with error rates. Since the valence modeling is
strongly related to the musical keys and modes, we manually correct the key
annotations of 367 clips in EMOPIA to ensure a high quality of lead sheets.

All datasets are randomly divided into respective training and validation sets
at the ratio of 9:1. As a result in our functional representation, the
vocabulary size of events is 215 for lead sheet and 324 for piano performance.

Dataset | # clips (major) | # bars | # events
---|---|---|---
HookTheory [40] | 18,206 (9,737) | 10.84 | 282.81
Pop1k7 [26] | 1,747 (1,264) | 104.82 | 6794.86
EMOPIA(L) [7] | 1,071 (618) | 16.94 | 435.22
EMOPIA [7] | 1,071 (618) | 17.09 | 1311.47

Table 1: The datasets. (major) denotes the number of clips in major key (and
the left is in minor key). The #bars and #events are average numbers across a
dataset. EMOPIA(L) refers to EMOPIA lead sheets.

###  4.2 Model Settings

The lead sheet model is a 12-layer Transformer Decoder [47] with 8 heads, 512
hidden dimensions and relative positional encoding [48]. The performance model
is similar to the lead sheet model except with Performer attention [49]. The
total parameter sizes are 41 million and 38 million respectively.

Both models are trained with the batch size of 4, the maximum sequence length
of 512 (lead sheet model) or 3072 (performance model), and the Adam optimizer
with $\beta=(0.99,0.9)$. We adopt a 200-step warm-up to achieve the maximum
learning rate of 1e-4 for pretraining and 1e-5 for finetune. All models are
implemented by PyTorch and trained on one NVIDIA Tesla V100 GPU. The lead
sheet model took around 180,000 steps to converge and the performance model
took around 200,000 steps. Nucleus sampling [50] is employed in the inference
phase. We referred [51, 38] to choose the sampling hyperparameters $\tau=1.2$,
$p=0.97$ for the lead sheet model and $\tau=1.1$, $p=0.99$ for the performance
model.

###  4.3 Baseline and Ablations

We consider the emotion-driven piano performance generation model in EMOPIA
[7] as our baseline, which generates music in an end-to-end paradigm instead
of two stages. To ensure the fair comparison of generation performance, we
trained the baseline model under ths same datasets in both pretraining and
finetune phases, and replaced the original CP-Word representation with REMI as
the former usually yields better generation performance and more comparable to
our proposed functional representation. Two other related works [9, 8] are not
included in comparison due to the main reason that we focus more on the
evaluation of the two-stage framework in valence and arousal modeling; and the
partial reason that they are not open-source or releasing the reproducible
model weights.

We conduct a comprehensive ablation study to evaluate if each proposed design
benefits the emotion modeling of music generation. Specially, these designs
include: 1) the two-stage generation, 2) the functional representation, and 3)
the dataset pretraining. In the following sections, models are denoted as
<representation(stage)>. For example, REMI(one) denotes the one-stage
generation model with REMI representation as the baseline, and REMI(two)
denotes the two-stage generation as one variant.

| M | C | M+C | P
---|---|---|---|---
REMI+key (two) | 0.465 | 0.065 | 0.075 | 0.418
–w/o. pretrain | 0.350 | 0.105 | 0.130 | 0.343
functional (two) | 0.505 | 0.700 | 0.735 | 0.548
–w/o. pretrain | 0.400 | 0.570 | 0.625 | 0.430
Real data | 0.578 | 0.695 | 0.746 | 0.812

Table 2: Key consistency calculated across all components, including melody
(M), chord (C), lead sheet (M+C) and performance (P).

###  4.4 Objective Evaluation and Results

Even though previous studies [7, 8, 9] employ metrics, such as Pitch Range
(PR) and Number of Pitch Classes (NPC), to evaluate the generation
performance, they do not provide any evidences on the superiority of melody
development, chord progression, and texture arrangement of music. Therefore, a
model with more similar PR and NPC values to those of the target dataset does
not necessarily promise a better generation quality than others.

Instead of using such metrics, we wish to evaluate the consistency between the
input musical conditions and the generation results. We introduce key
consistency to assess if a model can generate music pieces that adhere to the
desired input key signatures, which is highly correlated to the lead sheet
development and valence modeling. Specifically, key consistency measures the
match between the key condition <Key_*> and the actual key detected in the
generation using the algorithm [44] with an 81% accuracy rate. We compare REMI
and our functional representation to determine if the functional
representation can improve the key consistency via more close and interactive
designs on key, melody, and chord. Since this metric requires the key as
conditions, we add the <Key_*> event in REMI after <Emotion_*> (as REMI+key in
Table 2) when training the model. The non-pretrained versions are also
included for comparison. Each model generates 200 lead sheets (100 high and
100 low valence) and 400 performance samples (100 per emotion quadrant) for
evaluation.

From Table 2, the functional representation outperforms REMI (i.e., REMI+key)
across all components and even achieves compatible accuracy to real data over
the lead sheet component. This demonstrates the effectiveness of the
functional representation, by representing notes and chord roots relative to
key events for key modeling. In contrast, REMI struggles with associating
chord events with keys due to the ignorance of chord labels serving different
functions in different key scales. Moreover, pretraining process introduces
musical priors to enhance the learning of key relationships with other musical
elements, improving key consistency for both representations.

###  4.5 Subjective Evaluation and Results

We leverage an online listening test to assess the emotion modeling ability of
models. The test was conducted to collect user responses on three parts: 1)
valence modeling, 2) arousal modeling, and 3) 4Q emotion modeling. During this
test, the quality of the generated music has also been assessed implicitly as
it is a prerequisite to the emotion expression in the music. 22 participants
were engaging in this test, 5 with less than 2 years of musical training, 8
with 2-5 years, 3 with 5-10 years, and 6 with more than 10 years.

####  4.5.1 Valence Modeling

In this part, each participant listened to 16 generated tracks of piano
performance from four models [four tracks (two high valence and two low
valence) per model]: 1) Real data; 2) REMI (one); 3) REMI (two); 4) Functional
(two). For each track, participants rated its positiveness from –2 (low
valence) to 2 (high valence) with the step size 1.

The left of Figure 6 (‘(a)’) presents the mean opinion scores for the valence-
oriented test, where the Functional (two) model significantly outperforms both
REMI (two) and REMI (one) models. The REMI (two) model shows a slight
improvement over REMI (one) due to its two-stage design. Our proposed
Functional (two) model even marginally exceeds real data in low valence
scores, which could be due to the potential subjective biases in the negative
emotion as discussed in [10]. And our model achieves both great performance in
high valence and low valence results, demonstrating a good balance in valence
modeling.

Figure 6: The mean opinion score performance on the valence-oriented and
arousal-oriented listening tests. For (a-1) and (b-1), the higher score the
better performance; for (a-2) and (b-2), the lower score the better
performance.

####  4.5.2 Arousal Modeling

In the second part, the functional (two) and REMI (two) models are chosen to
compare their arousal modeling performance. Specifically, we wish to explore
whether they can generate piano performance with either high or low arousal
under the same lead sheet based on the given conditions (Q1 and Q4 for
positive lead sheets, Q2 and Q3 for negative ones). Two pairs of generated
tracks are randomly drawn for each model and each valence level, where every
pair includes two tracks of different arousal conditions. For each track,
participants rated its arousal level from –2 (low arousal) to 2 (high arousal)
with the step size 1.

The right of Figure 6 (‘(b)’) presents the results. The Functional (two) model
surpasses REMI (two) by an average of 0.5 point, highlighting its superior
ability to differentiate between the musical features of the two arousal
levels through performance. Additionally, it is rare for Functional (two) to
be incorrectly identified as high arousal under low arousal conditions (Figure
6 (b-2)).

####  4.5.3 4Q Emotion Judgement

In the last part, participates needed to choose the best option from four
options (4Q) for each track, with 8 tracks in total for the two models the
last section (4 tracks per model and 1 track per emotion).

Figure 7 presents the confusion matrices of two models. The Functional (two)
model achieves the higher overall accuracy than that of REMI (two) (71.5% vs.
31.0%). When examining each emotion category, Functional(two) demonstrates
superior performance in Q3 and Q4 than Q1 and Q2. Furthermore, music pieces
generated from it with high valence conditions are misidentified almost based
on their arousal levels; for instance, pieces intended for Q1 are almost
mistaken for Q4 and vice versa. In contrast, for REMI(two), the
misclassifications are across all categories, demonstrating its limitations in
modeling the four emotion classes although through two-stage generation.

Figure 7: The confusion matrices on the 4Q listening tests.

All above evaluations support that the combination of two-stage framework and
functional representation is effective in controlling the emotion of the music
it generates to a certain extent.

##  5 Conclusion and Future Work

In this paper, we first explore emotion disentanglement through a two-stage
Transformer-based framework for emotion-driven piano performance generation.
Then we propose a novel functional representation for symbolic music to
capture the interactions among musical keys, modes, chords, and melodies in
relation to the emotion contexts. An objective metric is designed to qualify
the key modeling of the proposed method, and subjective evaluations further
confirm its ability to convey desired emotional perception. In the future, we
wish to focus on enhancing the flexibility of emotional music generation
across all musical keys and investigating new applications fostered by our
framework, such as the controls of valence and arousal attributes under
similar music motifs.

##  6 Acknowledgment

The work is supported by a grant from the National Science and Technology
Council of Taiwan (NSTC 112-2222-E-002-005-MY2). This work also benefits from
the review comments of an unpublished work [52], which is an early version of
the present study. While the concept of functional representation has been
mentioned in that previous work [52], its focus was on emotion-driven melody
harmonization and the effectiveness of the functional representation in that
context was unclear, as the melody was pre-given and fixed. In the current
work, melody and chords are generated together, allowing more flexibility for
emotion control with the proposed functional representation. The source code
of that previous work can be found at the GitHub repo:
https://github.com/Yuer867/EMO_Harmonizer.

## References

  * [1] C.-Z. A. Huang, A. Vaswani, J. Uszkoreit, I. Simon, C. Hawthorne, N. Shazeer, A. M. Dai, M. D. Hoffman, M. Dinculescu, and D. Eck, “Music Transformer: Generating music with long-term structure,” in _Proc. ICLR_ , 2019.
  * [2] Y.-S. Huang and Y.-H. Yang, “Pop Music Transformer: Beat-based modeling and generation of expressive pop piano compositions,” in _Proc. ACM Multimed._ , 2020.
  * [3] D. von Rütte, L. Biggio, Y. Kilcher, and T. Hofmann, “FIGARO: Generating symbolic music with fine-grained artistic control,” in _Proc. ICLR_ , 2023.
  * [4] K. Chen, C. Wang, T. Berg-Kirkpatrick, and S. Dubnov, “Music sketchnet: Controllable music generation via factorized representations of pitch and rhythm,” in _Proc. ISMIR_ , 2020.
  * [5] S.-L. Wu and Y.-H. Yang, “MuseMorphose: Full-song and fine-grained piano music style transfer with one Transformer VAE,” _IEEE Trans. Audio, Speech, Lang. Process._ , vol. 31, pp. 1953–1967, 2023.
  * [6] P. Lu, X. Xu, C. Kang, B. Yu, C. Xing, X. Tan, and J. Bian, “MuseCoco: Generating symbolic music from text,” _CoRR_ , vol. abs/2306.00110, 2023.
  * [7] H. Hung, J. Ching, S. Doh, N. Kim, J. Nam, and Y.-H. Yang, “EMOPIA: A multi-modal pop piano dataset for emotion recognition and emotion-based music generation,” in _Proc. ISMIR_ , 2021.
  * [8] P. L. T. Neves, J. Fornari, and J. B. Florindo, “Generating music with sentiment using Transformer-GANs,” in _Proc. ISMIR_ , 2022.
  * [9] S. Ji and X. Yang, “MusER: Musical element-based regularization for generating symbolic music with emotion,” in _Proc. AAAI_ , 2024.
  * [10] C. Kang, P. Lu, B. Yu, X. Tan, W. Ye, S. Zhang, and J. Bian, “EmoGen: Eliminating subjective bias in emotional music generation,” _CoRR_ , vol. abs/2307.01229, 2023.
  * [11] L. Ferreira and J. Whitehead, “Learning to generate music with sentiment,” in _Proc. ISMIR_ , 2019.
  * [12] E. Choi, Y. Chung, S. Lee, J. Jeon, T. Kwon, and J. Nam, “YM2413-MDB: A multi-instrumental FM video game music dataset with emotion annotations,” in _Proc. ISMIR_ , 2022.
  * [13] W. Cui, P. Sarmento, and M. Barthet, “MoodLoopGP: Generating emotion-conditioned loop tablature music with multi-granular features,” in _Proc. EvoMUSART_ , 2024.
  * [14] K. Zheng, R. Meng, C. Zheng, X. Li, J. Sang, J. Cai, J. Wang, and X. Wang, “EmotionBox: A music-element-driven emotional music generation system based on music psychology,” _Frontiers in Psychology_ , vol. 13, 2022.
  * [15] M. T. Haseeb, A. Hammoudeh, and G. Xia, “GPT-4 driven cinematic music generation through text processing,” in _Proc. ICASSP_ , 2024.
  * [16] J. A. Russell, “A circumplex model of affect,” _Journal of Personality and Social Psychology_ , 1980.
  * [17] Y.-H. Yang, Y.-C. Lin, Y.-F. Su, and H. H. Chen, “A regression approach to music emotion recognition,” _IEEE Trans. Audio, Speech, Lang. Process._ , vol. 16, no. 2, pp. 448–457, 2008.
  * [18] ——, “Machine recognition of music emotion: A review,” _ACM Trans. Intelligent Systems and Technology_ , vol. 3, no. 3, 2012.
  * [19] J. S. G. Cañón, E. Cano, T. Eerola, P. Herrera, X. Hu, Y.-H. Yang, and E. Gómez, “Music emotion recognition: Toward new, robust standards in personalized and context-sensitive applications,” _IEEE Signal Process. Magzine_ , vol. 38, no. 6, pp. 106–114, 2021.
  * [20] D. R. Bakker and F. H. Martin, “Musical chords and emotion: major and minor triads are processed for emotion,” _Cognitive, Affective, & Behavioral Neuroscience_, 2015.
  * [21] Y.-C. Wu and H. H. Chen, “Generation of affective accompaniment in accordance with emotion flow,” _IEEE Trans. Audio, Speech, Lang. Process._ , 2016.
  * [22] S. Chowdhury and G. Widmer, “On perceived emotion in expressive piano performance: Further experimental evidence for the relevance of mid-level perceptual features,” in _Proc. ISMIR_ , 2021.
  * [23] R. Panda, R. Malheiro, and R. P. Paiva, “Audio features for music emotion recognition: A survey,” _IEEE Trans. Affective Computing_ , 2020.
  * [24] A. Aljanaki and M. Soleymani, “A data-driven approach to mid-level perceptual musical feature modeling,” in _Proc. ISMIR_ , 2018.
  * [25] Y. Hong, R. K. Mo, and A. Horner, “The effects of mode, pitch, and dynamics on valence in piano scales and chord progressions,” in _Proc. ICMC_ , 2018.
  * [26] W. Hsiao, J. Liu, Y. Yeh, and Y.-H. Yang, “Compound Word Transformer: Learning to compose full-song music over dynamic directed hypergraphs,” in _Proc. AAAI_ , 2021.
  * [27] T. Chen and L. Su, “Functional harmony recognition of symbolic music data with multi-task recurrent neural networks,” in _Proc. ISMIR_ , 2018.
  * [28] L. N. Ferreira, L. Mou, J. Whitehead, and L. H. S. Lelis, “Controlling perceived emotion in symbolic music generation with monte carlo tree search,” in _Proc. of AAAI (AIIDE Workshop)_ , 2022.
  * [29] L. N. Ferreira, L. H. S. Lelis, and J. Whitehead, “Computer-generated music for tabletop role-playing games,” in _Proc. of AAAI (AIIDE Workshop_ , 2020.
  * [30] N. N. López, M. Gotham, and I. Fujinaga, “Augmentednet: A roman numeral analysis network with synthetic training examples and additional tonal tasks,” in _Proc. ISMIR_ , 2021.
  * [31] G. Micchi, M. Gotham, and M. Giraud, “Not all roads lead to Rome: Pitch representation and model architecture for automatic harmonic analysis,” _TISMIR_ , 2020.
  * [32] E. Karystinaios and G. Widmer, “Roman numeral analysis with graph neural networks: Onset-wise predictions from note-wise features,” in _Proc. ISMIR_ , 2023.
  * [33] B. L. Sturm, J. F. Santos, O. Ben-Tal, and I. Korshunova, “Music transcription modelling and composition using deep learning,” _CoRR_ , vol. abs/1604.08723, 2016.
  * [34] Y. Yeh, W. Hsiao, S. Fukayama, T. Kitahara, B. Genchel, H. Liu, H. Dong, Y. Chen, T. Leong, and Y.-H. Yang, “Automatic melody harmonization with triad chords: A comparative study,” _CoRR_ , vol. abs/2001.02360, 2020.
  * [35] Y. Li, S. Li, and G. Fazekas, “An comparative analysis of different pitch and metrical grid encoding methods in the task of sequential music generation,” _CoRR_ , vol. abs/2301.13383, 2023.
  * [36] N. S. Keskar, B. McCann, L. R. Varshney, C. Xiong, and R. Socher, “CTRL: A conditional transformer language model for controllable generation,” _CoRR_ , vol. abs/1909.05858, 2019.
  * [37] M. Mongeau and D. Sankoff, “Comparison of musical sequences,” _Computers and the Humanities_ , vol. 24, no. 3, pp. 161–175, 1990.
  * [38] S.-L. Wu and Y.-H. Yang, “Compose & Embellish: Well-structured piano performance generation via a two-stage approach,” in _Proc. ICASSP_ , 2023.
  * [39] “HookTheory,” https://www.hooktheory.com/ [Accessed: (September 1, 2023)].
  * [40] C. Donahue, J. Thickstun, and P. Liang, “Melody transcription via generative pre-training,” in _Proc. ISMIR_ , 2022.
  * [41] A. L. Uitdenbogerd and J. Zobel, “Manipulation of music for melody matching,” in _Proc. ACM Multimed._ , 1998.
  * [42] J. Chang, “Chorders,” https://github.com/joshuachang2311/chorder.
  * [43] C. L. Krumhansl, “Cognitive foundations of musical pitch,” _Oxford University Press_ , 2001.
  * [44] P. Toiviainen and T. Eerola, “MIDI toolbox 1.1,” https://github.com/miditoolbox/, 2016.
  * [45] “Midi_Toolkit,” https://github.com/RetroCirce/Midi_Toolkit [Accessed: (September 1, 2023)].
  * [46] Z. Wang, K. Chen, J. Jiang, Y. Zhang, M. Xu, S. Dai, and G. Xia, “POP909: A pop-song dataset for music arrangement generation,” in _Proc. ISMIR_ , 2020.
  * [47] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _Proc. NeurIPS_ , 2017.
  * [48] Z. Dai, Z. Yang, Y. Yang, J. G. Carbonell, Q. V. Le, and R. Salakhutdinov, “Transformer-XL: Attentive language models beyond a fixed-length context,” in _Proc. ACL_ , 2019.
  * [49] K. M. Choromanski, V. Likhosherstov, D. Dohan, X. Song, A. Gane, T. Sarlós, P. Hawkins, J. Q. Davis, A. Mohiuddin, L. Kaiser, D. B. Belanger, L. J. Colwell, and A. Weller, “Rethinking attention with Performers,” in _Proc. ICLR_ , 2021.
  * [50] A. Holtzman, J. Buys, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” in _Proc. ICLR_ , 2019.
  * [51] Z. Fu, W. Lam, A. M. So, and B. Shi, “A theoretical analysis of the repetition problem in text generation,” in _Proc. AAAI_ , 2021.
  * [52] J. Huang and Y.-H. Yang, “Emotion-driven melody harmonization via melodic variation and functional representation,” _CoRR_ , vol. abs/2407.20176, 2024.