# Controllable Text Summarization: Unraveling Challenges, Approaches, and
Prospects - A Survey

Ashok Urlana1 Pruthwik Mishra2 Tathagato Roy2 Rahul Mishra2
TCS Research, Hyderabad, India1 IIIT Hyderabad2
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Generic text summarization approaches often fail to address the specific
intent and needs of individual users. Recently, scholarly attention has turned
to the development of summarization methods that are more closely tailored and
controlled to align with specific objectives and user needs. While a growing
corpus of research is devoted towards a more controllable summarization, there
is no comprehensive survey available that thoroughly explores the diverse
controllable aspects or attributes employed in this context, delves into the
associated challenges, and investigates the existing solutions. In this
survey, we formalize the Controllable Text Summarization (CTS) task,
categorize controllable aspects according to their shared characteristics and
objectives, and present a thorough examination of existing methods and
datasets within each category. Moreover, based on our findings, we uncover
limitations and research gaps, while also delving into potential solutions and
future directions for CTS.

##  1 Introduction

Despite the significant advancements in automatic text summarization, its one-
size-fits-all approach falls short in meeting the varied needs of different
segments of users and application scenarios. For example, generic automatic
summarization may struggle to produce easily understandable summaries of
scientific documents for non-expert users or create extremely brief summaries
of news stories for online feeds. Lately, a myriad of works have emerged aimed
at generating more controlled Maddela et al. (2022) and tailored text
summaries that meet a wide range of user needs.

Figure 1: Number of controllable text summarization publications for various
attributes

Controllable Text Summarization (CTS) is centered around creating summaries of
source documents that adhere to specific criteria. These criteria are managed
through various controllable attributes (CA) or aspects like summary length
Kwon et al. (2023), writing style Goyal et al. (2022), coverage of key
information Li et al. (2018); Jin et al. (2020b), content diversity Narayan et
al. (2022), and more. These criteria vary depending on the task, user needs,
and specific application context. For example, length-controlled summaries
Hitomi et al. (2019) are particularly useful in situations where brevity is
crucial, like in social media posts, headlines, and abstracts. In areas such
as marketing, academic writing, or professional communication, using a style-
controlled summary Chawla et al. (2019) is essential to ensure that the
information aligns with the intended tone and messaging strategy. On the other
hand, topic-controlled summaries Bahrainian et al. (2021) are commonly used in
research papers, reports, and content curation, providing a concentrated focus
on a specific topic to enhance clarity and coherence in the presented
information.

There is an uneven distribution of attention within the research community
towards various CAs as depicted in Fig 1. The majority of CTS works
concentrate on managing length, topic, and style. This could be attributed to
two main factors. First, it is comparatively simpler to develop datasets for
evaluating length, topic, and style compared to aspects like structure and
diversity. Second, there is a plethora of application scenarios for length or
topic-oriented summaries, such as generating concise news feeds or crafting
focused legal reports.

In this survey, we collect and analyze 61 research papers pertaining to
various possible CAs. Subsequently, we classify these CAs into 10 categories,
grouping similar ones based on shared characteristics and objectives.
Moreover, we delve into the existing datasets, evaluating their creation
methods and appropriateness for the respective task in each CA category.
Furthermore, we scrutinize the current CTS methodologies for each CA category,
drawing comparisons between their overarching frameworks and discussing
relevant limitations. Subsequently, we discuss in detail the generic and
specific evaluation strategies for CAs utilized by various works. Finally, we
attempt to critique the current approaches in addressing Controllable Text
Synthesis (CTS) and unravel potential future research trajectories.

The key contributions of this survey include 1) To the best of our knowledge,
it is the first comprehensive survey on CTS. 2) We formalize and categorize
the various possible CAs. 3) We describe current datasets utilized in CTS and
conduct a comprehensive examination of proposed CTS methods and evaluation
techniques, emphasizing their limitations. 4) We delve into current challenges
and promising avenues in research, with the aim of facilitating future
inquiries.

##  2 Task Formulation

This section introduces the Controllable Text Summarization (CTS) task by
outlining its definition and offering a categorized breakdown of the different
controllable attributes along with concise descriptions for each.

Given a set of source documents $D=\\{d_{1},d_{2},\ldots,d_{k}\\}$. Each
document, $d_{i}$, consists of a sequence of $n$ tokens:
$\\{x_{i,1},x_{i,2},\ldots,x_{i,n}\\}$. $S_{i}$ is the target summary of
document $d_{i}$, which comprises of a sequence of $m$ tokens:
$\\{s_{i,1},s_{i,2},\ldots,s_{i,m}\\}$, where $m\ll n$. The user wants to
control a set of controllable attributes $C$. The task can be framed as a
conditional generative problem: $P(S|D,C)=\prod_{i}^{k}P(S_{i}|d_{i},C)$

Attribute | Definition
---|---
Length |  Controlling the length of the summary
Style |  Controlling the readability levels, politeness, humor, and emotion
Coverage |  Controlling the salient information in summary
Entity |  Summary specific to pre-defined entities
Structure |  Create summaries with predefined structure or order
Abstractivity |  Controlling the novelty in sentence formation
Salience |  Adjusting the presence of prominent information
Role |  Providing role-specific summaries
Diversity |  Generating semantically diverse summaries
Topic |  Controlling topic-focused summary generation
Table 1: Controllable attributes definitions.

###  2.1 Controllable Attributes

A Controllable Attribute or Aspect (CA) refers to a user or application-driven
trait of the summary designed to meet specific criteria or conditions, such as
Length, Style, Coverage, Role, etc. In the literature, it is evident that
various authors use different terms to describe the same CAs, which exhibit
similar characteristics and objectives (such as "Salience: Key information",
and "Coverage: Granularity"). Additionally, numerous attributes can be
encapsulated by a representative class; for instance, "Style" may serve as a
class encompassing Tone, Readability, Humor, Romance, and similar aspects,
facilitating their classification within the same category. Based on these
observations, we group the CAs into ten categories as listed in Table 1.

##  3 Related Surveys

In the literature, a multitude of surveys center around conventional text
summarization methods (El-Kassas et al., 2021; Nazari and Mahdavi, 2019;
Allahyari et al., 2017; Gambhir and Gupta, 2017), including task-specific
surveys such as multi-document summarization (Sekine and Nobata, 2003), cross-
lingual summarization Wang et al. (2022), dialogue-based Tuggener et al.
(2021) and text generation Zhang et al. (2023a). On the contrary, this survey
focuses on user-centric summarization approaches by offering a thorough
analysis of controllable text summarization methods, challenges, and
prospects.

Dataset | Controllable attribute | Human-annotated | Size | Domain | Dataset URL
---|---|---|---|---|---
Multi-LexSum (Shen et al., 2022c) | Coverage | Yes | 9280 | Legal | https://tinyurl.com/22ksfase
GranDUC (Zhong et al., 2022) | Coverage | Yes | 50 | News | https://tinyurl.com/2x72ubrw
TS and PLS (Luo et al., 2022) | Style | No | 28124 | Biomedical | https://tinyurl.com/yck3v9px
MACSUM (Zhang et al., 2023b) | Style | Yes | 9686 | News, meetings | https://tinyurl.com/3d2dsc7u
NEWTS (Bahrainian et al., 2022) | Topic | Yes | 6000 | News | https://tinyurl.com/36hzk3ew
WikiAsp (Hayashi et al., 2021) | Topic | No | 320272 | Encyclopedia | https://tinyurl.com/3u45hfbn
ASPECTNEWS (Ahuja et al., 2022) | Topic | No | 2000 | News | https://tinyurl.com/bdzxs8ej
Tourism ASPECTS (Mukherjee et al., 2020) | Topic | No | 7000 | Reviews | https://tinyurl.com/ypjhhrxv
EntSUM (Maddela et al., 2022) | Entity | Yes | 2788 | News | https://tinyurl.com/2pz9vzyw
JAMUL (Hitomi et al., 2019) | Length | No | 1932398 | News | https://tinyurl.com/3s3ecua9
CSDS (Lin et al., 2021) | Role | Yes | 10700 | Dialogues | https://tinyurl.com/adk7zc7u
MReD (Shen et al., 2022b) | Structure | Yes | 7089 | Meta reviews | https://tinyurl.com/4nn87fd6
Table 2: List of controllable summarization datasets.

##  4 Datasets

This section provides a broad overview of the controllable summarization
datasets and corresponding creation/acquisition strategies. The controllable
summarization methods have been applied in several ways: 1) by utilizing
publicly available summarization datasets, 2) by employing data augmentation
or synthetic methods on generic datasets, 3) by creating human-annotated
datasets.

###  4.1 Generic Datasets

Controllable summarization research predominantly leverages widely used news
summarization datasets. Notably, about 57% of CTS studies utilize either CNN-
DailyMail Nallapati et al. (2016) or DUC Over and Yen (2004); Dang (2005).
Other popular datasets, include Gigaword Napoles et al. (2012), XSum Narayan
et al. (2018), NYTimes Sandhaus (2008), NEWSROOM Narayan et al. (2018), and
dialogue-based SAMSUM Gliwa et al. (2019), along with opinion-based datasets
Angelidis and Lapata (2018); Angelidis et al. (2021), are employed for
controllable summarization. However, these generic datasets lack explicit
guidance on emphasizing summary sentences. Moreover, as detailed in Table 2,
controllable summarization requires specialized datasets to provide evaluation
opportunities for specific aspects like length, topic, style, etc,.

###  4.2 Synthetic Datasets

The synthetic datasets can be derived by applying the aspect-specific
heuristics to the widely used generic datasets. In this section, we list out
few synthetic datasets and their creation strategies.
JAMUL. Hitomi et al. (2019) collect the length-sensitive headlines for the
Japanese language. Each article consists of three headlines with varying
lengths of 10, 13, and 26 characters respectively. All three headlines are
written by professional editors of the Asahi Shima newspaper company. These
datasets act as a benchmark to build length-controllable summarization
systems. TS and PLS. In order to enhance the readability of biomedical
documents, Luo et al. (2022) introduce two types of summaries. The Technical
Summary is an abstract of a peer-reviewed bio-medical research paper and the
Plain Language Summary (PLS) is the authors submitted summary as part of the
journal submission process. Wikiasp. In order to construct the multi-domain
aspect-based summarization corpus, Hayashi et al. (2021) utilized the
Wikipedia articles from 20 domains. Moreover, utilized the section titles and
paragraph boundaries of each article as a proxy of aspect annotation.
ASPECTNEWS. The study by Ahuja et al. (2022) aim to create the ASPECTNEWS
dataset for aspect-oriented summarization. They achieved it by utilizing
articles from the CNN/DailyMail dataset and identifying documents related to
‘earthquakes’ and ‘fraud investigations’ by using the universal sentence
encoder Cer et al. (2018). Tourism ASPECTS. Mukherjee et al. (2020) collected
a controllable aspect-based opinion summarization dataset of tourist reviews.
The reviews were obtained from TripAdvisor website and identified the relevant
aspects using the unsupervised attention-based aspect extraction technique He
et al. (2017).

###  4.3 Human annotated

Human-annotated datasets are crucial to building robust models and evaluating
the effectiveness of the CTS models. This section provides the details on
human-annotated controllable summarization datasets.
Multi-LexSum. Shen et al. (2022c) is a human-annotated corpus of 9,280 civil
rights lawsuits and corresponding summaries with different degrees of
granularity (level of details). The summaries are created by well-trained
lawyers and law students with carefully crafted guidelines. The target summary
length ranges from one-sentence to multi-paragraph level. EntSUM. Maddela et
al. (2022) is a human-annotated entity-specific controllable summarization
dataset. It utilizes the articles from The New York Times Annotated Corpus
(NYT) Sandhaus (2008) and includes annotated summaries for PERSON and
ORGANISATION tags. NEWTS. Bahrainian et al. (2022) introduce the topically
focused summarization corpus. It was created by leveraging documents from
CNN/DailyMail and employing crowd-sourcing to generate two distinct summaries
for each document, each focusing on a different thematic aspect. CSDS. To
obtain the role-oriented summaries for Chinese dialogues, Lin et al. (2021)
introduce the Chinese Customer Service Dialogue Summarization (CSDS) dataset.
This dataset is meticulously annotated, segmenting the dialogues based on
their topics and summarizing each segment as a QA pair. MReD. To tackle the
task of structure-controllable summarization, Shen et al. (2022b) introduce
the Meta-Review Dataset (MReD). It was created by gathering meta-reviews from
the open review system and categorizing each sentence into one of nine
predefined intent categories (abstract, strength, weakness, etc.,). MACSUM.
Zhang et al. (2023b) a human-annotated corpus to control the mix of attributes
(Topic, Speaker, Length, Extractiveness, and Specificity) together. MACSUM
covers source articles from CNN/DailyMail and QMSUM Zhong et al. (2021)
datasets. GranDUC. By re-annotating the DUC-2004 Dang (2005), Zhong et al.
(2022) released novel benchmark dataset for the granularity control.
Annotators were instructed to create summaries of multiple documents with
coarse, medium, and fine granularity levels.

##  5 Approaches to Controlled Summarization

Various CAs have been investigated in controllable summary generation tasks,
including style (e.g., politeness, humor, formality), content (e.g., length,
entities, keywords), structure, etc. In this section, we describe various
approaches to achieve CTS for the attributes mentioned in Table 1.

Length. Early methods controlled the length of summaries in an ad-hoc manner
either by stopping the generation after a fixed number of tokens or ignoring
the end-of-sequence symbol until a fixed number of tokens are generated during
the decoding stage. These shortcomings are handled using four different
approaches to integrate length as a learnable parameter.
Adding length in input: Fan et al. (2018a) proposes a convolutional encoder-
decoder-based summarization system, where they quantize summary lengths into
discrete bins of different size ranges. During training, the input data is
prepended with the gold summary length in terms of bin lengths. Due to a fixed
number of length bins, the system fails to generate summaries of arbitrary
lengths. CTRLSUM He et al. (2022) presents a generic framework to generate
controlled summaries using keywords specific to a control parameter identified
using the ROUGE score. Instead of controlling a single attribute, Zhang et al.
(2023b) allows values of the length (normal, short, long) attribute to be used
as inputs along with the source text for hard prompt tuning.

Adding length in encoder: Yu et al. (2021) proposes a length context vector
concatenated with the decoder hidden state and other attention vectors. But
the main limitation of the system is the generation of incomplete summaries
for short desired length. Liu et al. (2022b) proposes a length-aware attention
model that adapts the source encodings based on the desired length by
pretraining the model. Zhang et al. (2023b) adds a hyperparameter for learning
the prefix embeddings for different attributes at each layer of the encoder
for soft prefix tuning.

Adding length in decoder: Kikuchi et al. (2016) proposed the first method to
control length using a BiLSTM encoder-decoder architecture with attention
Luong et al. (2015) for sentence compression. In each step of the decoding
process, an additional input for the remaining length is provided as an
embedding. Instead of pre-defined length ranges, Liu et al. (2018) adds a
desired length parameter at the decoding step to each convolutional block of
the initial layer of the convolutional encoder-decoder model. Févry and Phang
(2018) designs an unsupervised denoising auto-encoder for sentence
compression, where the decoder has an additional input of the remaining
summary length at each time step. Although it generates grammatical summaries,
in some cases, the generated summaries are semantically different from the
source text.

Hitomi et al. (2019) uses a similar architecture to that of Liu et al. (2018),
where the desired number of tokens are replaced by characters on a Japanese
headline generation task with headlines of different lengths. To handle length
constraint, Takase and Okazaki (2019) proposes two modifications to the
sinusoidal positional embeddings on the decoder side: length-difference
positional encoding and length-ratio positional encoding. Sarkhel et al.
(2020) proposes a multi-level summarizer that models a multi-headed attention
mechanism using a series of timestep-independent interpretable semantic
kernels to control lengths reducing the trainable parameters significantly.
The model does not encode the length attribute directly. Song et al. (2021)
proposes a confidence-driven generator that is trained on a denoising
objective with a decoder-only architecture, where the source and summary
tokens are masked with position-aware beam search. Goyal et al. (2022) uses a
mixture-of-experts model with multiple transformer-based decoders for
identifying different styles or features of summaries. Kwon et al. (2023)
introduces the summary length prediction task on the encoder side and this
predicted summary length is inserted with a length-fusion positional encoding
layer. Zhang et al. (2023b) also adds a hyperparameter for learning the prefix
embeddings for different attributes at each layer of the decoder for soft
prefix tuning.

Adding length in loss/reward function: Makino et al. (2019) proposes a global
minimum risk training optimization method under length constraint for the
neural summarization tasks which is faster and generates five times fewer
over-length summaries on an average than others. Chan et al. (2021) uses an
RL-based Constrained Markov Decision process with a mix of attributes. Hyun et
al. (2022) devises an RL-based framework that incorporates both the length and
quality constraints in the reward function with the generation of multiple
summaries of different lengths. This model is computationally expensive.
Style. The generation of user-specific summaries has gained significant
interest, but achieving distinct styles has posed an enduring challenge. These
stylistic variations may encompass tone, readability control, or the
modulation of user emotions.

Style control aims to generate source-specific summaries Fan et al. (2018b) by
utilizing the convolutional encoder-decoder network. In their work, Chawla et
al. (2019) obtains the formality-tailored summaries by utilizing the input-
dependent reward function. They use the pointer-generator See et al. (2017)
network as the under-laying architecture, and modify the loss function with an
addition of a formality-based-reward function. In another study, Jin et al.
(2020a) attempts to control the humor, romance, and clickbait in headlines. A
multitask learning framework is adopted by utilizing a sequence-to-sequence
architecture and denoising auto-encoder to generate stylistically different
headlines.

By employing an inference style classifier, Cao and Wang (2021) adjusts the
decoder final states to obtain stylistic summaries. Moreover, obtain the
lexical control by utilizing the word unit prediction that can directly
constrain the output vocabulary. Similarly, Goyal et al. (2022) extends the
decoder architecture to a mixture-of-experts version by using multiple
decoders. The gating mechanism helps to obtain the multiple summaries for the
single source. However, the major limitation in this model is its manual
gating mechanism. To control various fine-grained reading grade levels,
Ribeiro et al. (2023) present three methods: instruction-prompting,
reinforcement learning based reward model, and lookahed readability decoding
approach.
Coverage. Managing the information granularity is essential to measure the
semantic coverage between the source document and the summary. To regulate the
granularity, Wu et al. (2021) introduces a two-stage approach where the
controllable model incorporates a summary sketch, that encompasses user
intentions and key phrases, serving as a form of weak supervision.
Furthermore, they leverage a text-span-based conditional generation method to
govern the level of detail in the generated dialogue summaries. Zhong et al.
(2022) proposes a multi-granular event-aware summarization method composed of
4 stages: event identification, unsupervised event-based summarizer
pretraining, event ranking, and summary generation by adding events as hints.
Extraction of events from source text may lower the abstractiveness. Zhang et
al. (2023b) uses the hard and soft-prompting strategies to control the amount
of extracted text from the source in the summary. Additionally, Huang et al.
(2023) utilizes the natural language inference models to improve the coverage.

Entity. Entity-centric summarization concentrates on producing a summary of a
document that is specific to a given target entity Hofmann-Coyle et al.
(2022). Zheng et al. (2020) extracts the named entities using a pre-trained
BERT Devlin et al. (2019) based model and feeds both the article and the
selected entities to a bidirectional LSTM Hochreiter and Schmidhuber (1997)
encoder-decoder model. In another study, Liu and Chen (2021) extracts the
entities (speakers and non-speaker entities) from a dialogue to form a
planning sequence. The entities extracted from the intersection of the gold
summary and the source document are concatenated to the source dialogue for
training the conditional BART-based model. This model introduces factual
inconsistency due to paraphrasing from a personal perspective.

Maddela et al. (2022) extends the GSum Dou et al. (2021) for entity-centric
summarization by feeding it either sentences or strings, which mentions
extracted entities as guidance. It also adapts BERTSum Liu and Lapata (2019)
by only feeding it all the sentences containing the entity string mention and
its coreferent mentions, instead of all the source documents. They show that
the GSum-based method outperforms all other abstractive methods while their
BERTSum extension is the best-in-class extractive model for entity-centric
summarization in the EntSUM dataset. Hofmann-Coyle et al. (2022) models
entity-centric extractive summarization as a sentence selection task. Building
upon BERTSum Liu and Lapata (2019), they use a BERT Devlin et al. (2019) based
bi-encoders to represent the sentence and target entity pair and train with
contrastive loss objective.

Structure. Generic datasets lack key elements for emphasizing specific aspects
in the corresponding ground truth summaries. To address this limitation and
emphasize summary structure, Shen et al. (2022b) introduces the MRed dataset
(see Section 4). They achieve structure-controllable text generation by adding
a control sequence at the beginning of the input text and treating summary
generation as a standalone process. However, this approach has two main
limitations 1) generated tokens are solely based on logits predictions without
ensuring that the sequence satisfies the control signal, 2) Auto-regressive
models face error propagation in generation due to self-attention, causing
subsequent generations to deviate from the desired output.

To overcome these challenges, the sentence beam-search (SentBS) Shen et al.
(2022a) approach produces multiple sentence options for each sentence and
selects the best sentence based on both the control structure and the model’s
likelihood probabilities. In a related study, Zhong and Litman (2023) utilizes
predicted argument role information to control the structure in legal opinion
documents. Additionally, in the work of Zhang et al. (2023b), the prompt of
entity chains, representing an ordered sequence of entities, is used for pre-
training and fine-tuning with a planning objective to control the summary
structure.

Abstractivity. Abstractivity measures the degree of textual novelty between
the source text and summary. In their work, See et al. (2017) introduces a
pointer-generator network to control the source copying and generate novel
word/sentence generation. The pointing mechanism aims to copy the source
information, whereas the generator component will generate the novel
information. However, the pointer-generator mechanism fails to generate higher
abstraction levels, Kryściński et al. (2018) tackles this problem in two ways:
1) decompose the decoder into a contextual network to retrieve the relevant
parts of the source text and generate the summary by utilizing a pretrained
model, 2) a mixed RL-based objective jointly optimizes the n-gram overlap with
the ground truth summary. Similarly, Song et al. (2020) controls the copying
behavior by using a mix-and-match strategy to generate summaries with varying
n-gram copy rates. Based on the seen, unseen words from the source text, the
system controls the copying percentage by acting as a language modeling task.

Moreover, token-based controllable approaches such as ControlSum Fan et al.
(2018b) allow the users to explicitly specify the control attribute to
facilitate better control. However, controlSum does not provide any
supervision on violating the controllability. To alleviate this issue, Chan et
al. (2021) proposes an RL-based framework on the constrained Markov decision
process and introduced a reward to penalize the violation of attribute
requirement.

Salience.  Salience captures the most important information or sentences in a
document. In SummaRuNNer Nallapati et al. (2017), the salience is modeled as a
feature in a classification objective. It models the task of summarization as
a GRU Cho et al. (2014) based text sequence to binary sequence learning
Bahdanau et al. (2014) task at a sentence level for single documents. The
binary score indicates the membership of the sentence in the summary. The
system performs poorly on out-of-domain datasets. To retain key content from
the source in the summary, Li et al. (2018) introduces a Key Information Guide
Network, where keywords are identified by the TextRank algorithm with a
modified attention mechanism that accommodates the key information as an
additional input. However, it focuses on informativeness ignoring coherence
and readability features.

Deutsch and Roth (2023) models salience in terms of noun phrases using
question and answering (QA) signals. The QA signal is integrated into a two-
stage summarization model where the first stage identifies the salient noun
phrases in a document as a classification task and then the second stage
generates the summary conditioned on these identified noun phrases. The
summaries are generated using a BART Lewis et al. (2019) model on CNN-
Dailymail and DUC-2004 datasets. This approach is not applicable to languages
for which question generation and question answering models are not available.
In long document CLS tasks, summarization systems often fail to respond to
user queries. To resolve this issue, Pagnoni et al. (2023) proposes a pre-
training approach that involves two tasks of salient information
identification from sentences having the highest self ROUGE score and a
question generation system to generate questions whose answers are the salient
sentences.
Role. Role-oriented dialogue summarization generates summaries for different
roles in dialogue (e.g. doctor and patient) Liang et al. (2022). In dialogue
summarization, the summary is specific for each agent present in the dataset.
Lin et al. (2021) proposed the CSDS dataset (see in the dataset section) and
benchmarked a variety of existing state-of-the-art summarization models for
the task of generating agent and user surveys. They find that agent summaries
generated by the existing method lack key information needed to be extracted
from dialogues of the other role. To bridge this gap, Lin et al. (2022) builds
a role-aware summarization model for two users (agent and user) present in the
dataset. They use two separate decoders for generating the user and agent
summaries by utilizing user and agent masks to split the context information.
A role attention mechanism is introduced to each decoder so that it can
leverage the overall context by attending to the hidden states of the other
role. Liang et al. (2022) uses a role-aware centrality scoring model that
computes role-aware centrality scores for each utterance, which measures the
relevance between the utterance and the role prompts (signaling whether the
summary is for the user or agent). This is then used to reweight the attention
scores for each utterance, which is then used by the decoder to generate the
summary.
Diversity. Traditional decoding strategies, like beam search, excel at
generating single summaries but often struggle to produce diverse ones. On the
other hand, techniques such as top-k and nucleus sampling are effective in
generating diverse outputs but may sacrifice faithfulness. In response to
these challenges, Narayan et al. (2022) introduces compositional sampling, a
decoding method to obtain diverse and faithful summaries. This approach begins
by planning Narayan et al. (2021) a semantic composition of the target in the
form of entity chains, and then leverages beam search to generate diverse
summaries.
Topic. Long documents often cover multiple topics, a brief summary may not
capture the complete range. The usefulness of the summary is dependent on the
point of interest of the end user, making the topic control an active area of
research. Krishna and Srinivasan (2018) trains a topic-conditioned pointer-
generator network See et al. (2017) by concatenating one hot encoding
representation of the topic with the embedding of each token in the input
document. However, news categories are used as the predefined topics, that
limits the generalization to other tasks. Tan et al. (2020) extends the same
for any arbitrary topics by leveraging external knowledge sources like
Wikipedia and ConceptNet to develop a weakly supervised framework for
summarization, which can work with any encoder-decoder architecture.

Suhara et al. (2020) proposes an unsupervised method that can generate
customized summaries of review opinions without needing any gold summaries.
Aspect-specific opinions are extracted from a set of reviews by a pre-trained
opinion extractor, and then merged, filtered, and ranked. Using these
extracted and processed opinions, a summary of the opinion is generated by a
generator model trained to reconstruct the reviews from the opinions.

Mukherjee et al. (2020) proposes an integer linear programming-based
extractive framework for aspect-based opinion summarization. Given a set of
reviews for a product (Eg., Hotels), Amplayo et al. (2021) trains a Multiple
Instance Learning (MIL) model to extract the predictions for aspect (like
cleanliness) codes at document, sentence token level. These predicted aspects
are used to transform the input such that relevant sentences and keywords
along with aspect tokens are fed into the pre-trained T5 Raffel et al. (2020)
model.

Hsu and Tan (2021) introduces the task of decision-focused summarization to
generate summaries that are most helpful in making decisions. The focus is on
the task of predicting future Yelp ratings from the set of reviews using a
Longformer-based Beltagy et al. (2020) regression model. They propose an
iterative algorithm that selects the sentence for summary from the set of
current unselected sentences which can minimize this difference between the
rating prediction model when given the whole input (set of reviews) versus
when only the summary is provided. Hayashi et al. (2021) combines a RoBERTa
Liu et al. (2019) based aspect discovery model and pretrained transformer-
based summarization model for aspect summarization on the wikiasp dataset.
Mukherjee et al. (2022) extends topic-focused summarization for multimodal
documents by leveraging on-topic images to generate a topic-focused summary by
creating a joint image-text context vector. Bahrainian et al. (2021)
introduces LDA Blei et al. (2003) based topic modeling with a novel topical
attention mechanism for Pointer Generator Networks.

Automatic Evaluation
---
Type of metric | Attribute | Papers | Metrics | Short description
Embedding-based (Language Model) | General |  Lin et al. (2022), Liang et al. (2022) | MoverScore |  Computed using pretrained language models, either by computing similarity scores between reference and generated text embeddings or through likelihood computation of the generated text.
|  Song et al. (2020), Shen et al. (2022a),
---
Cao and Wang (2021) , Deutsch and Roth (2023),
Chan et al. (2021), Pagnoni et al. (2023),
Lin et al. (2022), Liang et al. (2022),
Narayan et al. (2022), Zhong and Litman (2023),
Ribeiro et al. (2023), Shen et al. (2022c),
Maddela et al. (2022), Lin et al. (2021)
BERTScore
Huang et al. (2023) | BartScore
Zheng et al. (2020) | Bert-Reo
Readability | Luo et al. (2022) |  Masked Noun Phrase-based Text Complexity, Ranked NP Based Text Complexity, Masked Random Token-Based Text Complexity
Ngram Based | General |  |  Lin et al. (2022), Liang et al. (2022),
---
Jin et al. (2020a), Narayan et al. (2022)
BLEU |  These metrics are based on matching ngram tokens between reference and generated summaries
|  Except* Zhang et al. (2023b), Goldsack et al. (2023),
---
Cao and Wang (2021), Hsu and Tan (2021),
Hofmann-Coyle et al. (2022)
ROUGE
Jin et al. (2020a), Sarkhel et al. (2020) | METEOR
Jin et al. (2020b) | Word Mover’s Distance
Miscellaneous | Length |  Goyal et al. (2022), Kwon et al. (2023), Liu et al. (2018), Chan et al. (2021) |  Absolute Length, Compression Ratio, Length Variance, Var, Bin Percentage |  Non-normative metrics proposed by authors to evaluate specific controlled aspect
Entity | Chan et al. (2021) | QA-F1
Narayan et al. (2021) | Entity Planning, Entity Specificity
| Topic, Speaker,
---
Length, Extractiveness,
Specificity
Zhang et al. (2023b) |  | Control Correlation,
---
Control Error Rate
Abstractiveness, Degree of Specificity | Goyal et al. (2022) | abstractiveness, degree of specificity
Readability |  Goyal et al. (2022), Cao and Wang (2021) | Dale-Chall
Ribeiro et al. (2023) |  Flesch Reading Ease, Gunning Fog Index, Coleman Liau Index

Table 3: Automatic evaluation metrics for controllable summarization,
“General” refers to all controllable attributes

##  6 Evaluation Strategies

This section catalogs and briefly describes the variety of automatic and human
evaluation metrics that are being used to evaluate the summaries generated by
the different methods studied in this paper.

###  6.1 Automatic Evaluation

The automatic evaluation metrics can be categorized based on how they are
defined. We categorize the metrics into n-gram-based, language-model-based,
and aspect-specific.

N-gram based evaluation metrics like ROUGE Lin (2004), BLEU Papineni et al.
(2002), METEOR Banerjee and Lavie (2005) are based on matching n-grams from
candidate summaries to a set of reference summaries. ROUGE is a widely used
metric for controllable summarization literature.

Language-model based metrics are computed using pre-trained language models
like BERT Devlin et al. (2019) or BART Lewis et al. (2019). One approach is to
evaluate the generated text, use the language model to generate embeddings for
the generated and reference text and compute the cosine similarity as done by
BERTScore Zhang* et al. (2020). Another approach is based on computing the log
probability of the generated text conditioned on input text as demonstrated in
BARTScore Yuan et al. (2021).

Aspect specific metrics do not fall cleanly into either of the above-mentioned
categories. These metrics focused on evaluating specific controllable aspects
such as Flesh Reading Ease Flesch (1948), Gunning Fog Index, and Coleman Liau
Index for readability, control correlation, and error rate Zhang et al.
(2023b) for topic, abstractivity and role attributes as mentioned in Table 3.

We find that there are limited set of metrics that can evaluate the degree and
efficacy of control that is obtained for a specific aspect. This is due to a
both, lack of aspect-specific ground truth summaries and standard metrics. We
find that aspect-specific metrics are not adaptable, across different
frameworks or approaches. This prevents a clear picture from emerging,
regarding which approaches work best in controlling a specific attribute.

###  6.2 Human Evaluation

Human evaluation is an indicator of the robustness and effectiveness of
different summarization systems on specific aspects that cannot be captured by
automatic evaluation metrics. These aspects include generic properties of a
summary such as truthfulness Song et al. (2020); Hyun et al. (2022), relevance
Goyal et al. (2022); He et al. (2022); Shen et al. (2022b), fluency Narayan et
al. (2022); Suhara et al. (2020), and readability Cao and Wang (2021);
Kryściński et al. (2018) or specific properties such as completeness Yu et al.
(2021); Liu et al. (2022a) for length-controlled summaries, coverage Mukherjee
et al. (2020, 2022) for the entity, and topic-controlled summary generation.

Broadly two kinds of scoring mechanisms are used for human evaluation: binary
and rank-based. The rank-based scores usually range from 1 to 5. The details
about different evaluation metrics detailed in table 4.

Human Evaluation
---
Aspect(s) | Papers | Metrics | Short description
| Abstractivity, Length, Style,
---
Topic, Coverage
Role, Entity
|  Sarkhel et al. (2020), Song et al. (2020)
---
Liu et al. (2022b), Cao and Wang (2021)
Amplayo et al. (2021), Wu et al. (2021),
Jin et al. (2020b), Kwon et al. (2023),
Lin et al. (2022), Liu and Chen (2021),
Zheng et al. (2020), Bahrainian et al. (2021),
Suhara et al. (2020), Lin et al. (2021)
Informativeness | Has the summary covered key content of the input text?
| Structure, Length, Entity,
---
Salience, Topic
Kwon et al. (2023) | Conciseness/Granularity | Is the key information presented in a crisp way?
| Structure, Style, Topic
---
Length, Entity, Abstractivity,
Coverage, Role, Diversity
|  Goyal et al. (2022), Tan et al. (2020)
---
Yu et al. (2021), Shen et al. (2022b),
Song et al. (2020), Liu et al. (2022b),
Févry and Phang (2018), Zheng et al. (2020),
Shen et al. (2022a), Cao and Wang (2021),
Amplayo et al. (2021), Hyun et al. (2022),
Chan et al. (2021), Jin et al. (2020b),
Liu et al. (2022a), Lin et al. (2022),
Zhong et al. (2022), Jin et al. (2020a),
Lin et al. (2021)
Fluency/Grammaticality | Are the sentences in a summary grammatically correct?
Role, Topic, Diversity |  |  Narayan et al. (2022), Suhara et al. (2020),
---
Lin et al. (2022), Lin et al. (2021),
Mukherjee et al. (2020)
Non-redundancy/Diversity | Is the summary conveying diverse information?
Topic | Krishna et al. (2018) | Contextual Appropriateness | Is the substituted word more readable in the summary?
Style, Diversity |  |  Goyal et al. (2022), Narayan et al. (2022),
---
Chan et al. (2021), Jin et al. (2020b),
Zhong et al. (2022), Huang et al. (2023)
Faithfulness/Factuality |  | Does the summary present factually correct content
---
with respect to the source?
| Style, Topic.
---
Entity, Structure
Goyal et al. (2022), Zhong and Litman (2023) | Coherence | Is the summary composed of correlated sentences?
| Style, Structure, Length,
---
Entity, Abstractivity,
Coverage, Topic, Diversity
|  Goyal et al. (2022), He et al. (2022),
---
Shen et al. (2022b), Chan et al. (2021),
Krishna and Srinivasan (2018), Shen et al. (2022a),
Cao and Wang (2021), Zhong et al. (2022),
Jin et al. (2020a), Kryściński et al. (2018),
Luo et al. (2022), Huang et al. (2023)
Relevance |  | Does the summary contain relevant information
---
regarding the user provided attribute (topic/entity)?
Abstractivity, Length |  |  Song et al. (2020), Hyun et al. (2022),
---
Févry and Phang (2018), Huang et al. (2023)
Truthfulness/Fidelity |  | Has the summary successfully preserved the meaning
---
of the original text?
Length, Entity |  He et al. (2022), Yu et al. (2021) | Accuracy/Correctness | Is the information in the summary accurate?
Style |  |  Kryściński et al. (2018), Ribeiro et al. (2023),
---
Cao and Wang (2021)
Readability | Is the text inside the summary readable?
Length |  Yu et al. (2021), Liu et al. (2022a) | Completeness | Does the summary contain incomplete text?
Topic, Structure |  |  Zhong and Litman (2023), Mukherjee et al. (2020),
---
Mukherjee et al. (2022)
Coverage |  | Does the summary include all the topics or aspects
---
defined in the source?

Table 4: Human evaluation metrics for controllable text summarization

##  7 Challenges and Future Prospects

Generic vs specialized benchmarks. We observe that more than 75% of CTS works
either utilize or alter the generic news summarization datasets to evaluate
the controllable summarization. As shown in Table 1, out of the 10 categories,
we could find one or more CA-specific datasets for only seven categories. We
envisage that conducting evaluations with specialized datasets that align
closely with real-world application scenarios or user requirements will help
better in assessing the practical utility, robustness, and performance of CTS.
It is worth noting from our survey of CTS systems that evaluations are often
confined to specific domains, like news, possibly due to the abundance of
available datasets in that domain. However, this narrow focus limits the
evaluation of the CTS model’s robustness. Creating benchmark datasets that
span multiple domains relevant to CA will undoubtedly enhance the progress of
CTS research.

Standardization of metrics. The CTS task aims to generate attribute-specific
summaries and requires aspect-specific evaluation methods. The current
literature is limited to utilizing the popular N-gram-based or embedding-based
metrics as listed in Table 3. Future work toward developing aspect-specific
evaluation metrics is a promising direction.

Explainability. For effectively controlling user or application-specific
attributes, it is imperative to leverage the understanding of the decision-
making process within CTS systems. Also, this comprehension is essential for
users or stakeholders, enabling them to discern how the system generates
summaries from source text. This holds particular significance in applications
where human decision-making or interpretation plays a pivotal role, such as in
legal, medical, or financial domains.The existing CTS efforts lack proper
emphasis on the explainability aspects, which can be readily addressed through
the incorporation of suitable explanation methodologies, such as attention
flow Abnar and Zuidema (2020), Integrated gradients Sundararajan et al. (2017)
, and Shapley Lundberg and Lee (2017), etc.

Multi-lingual, multi-model, and code-mixed CTS. The existing literature on CTS
predominantly focuses on works in English, with only one study addressing the
topic in a Japanese context. We could not find any studies and datasets
related to multilingual and code-mixed CTS approaches. Moreover, the task of
controllable summarization in multi-modal and multi-document settings remains
largely unexplored, presenting unique challenges for models to address and
offering avenues for intriguing research problems.

Multi-CA control. Even though, few of the works attempt to perform multi-
attribute controllable summarization Goyal et al. (2022); Narayan et al.
(2022); Zhang et al. (2023b), either they are computationally expensive or the
performance highly depends on the quality of the prompts (can be keywords,
entities, structure, etc,.)
Fixed Length Beam Search. Majority of the CTS systems generate the most
likelihood sequences with a fixed length beam search, which subdues their
capability of generating a diverse set of summaries Ippolito et al. (2019).
The current systems ignore the content-level diversity and rather focus on
generating summaries differing in either punctuations or word forms with the
same word roots. For generating diverse outputs, many extensions to beam
search such as Noisy Parallel Approximate Decoding Cho (2016), Top-g Capping
Li and Jurafsky (2016); Li et al. (2016), iterative and clustering beam search
Kulikov et al. (2019); Tam (2020) have been proposed.
Reproducibility. In the detailed analysis outlined in Table 6, we note that
35% of research studies do not share their code publicly. Furthermore, 25% of
the papers did not carry out any human evaluation, and among the remaining
studies, 79% did not conduct Inter Annotator Agreement (IAA) assessments. The
lack of reproducibility Ito et al. (2023); Gao et al. (2023); Iskender et al.
(2021) measures hinders the scientific community’s ability to validate and
build upon existing work. On the other hand, the human study component should
be a must for a text summarization evaluation scheme, otherwise, we are
potentially overlooking essential aspects of real-world applicability.
Standing on the shoulders of LLMs. The rise and success of large language
models (LLMs) have opened up unparalleled possibilities for leveraging their
capabilities across diverse stages of the Natural Language Processing (NLP)
pipeline. In the context of CTS, LLMs can be fine-tuned to grasp context-
specific nuances pertaining to CAs without the need for a dedicated training
set. Additionally, when it comes to evaluating CTS models, LLMs can serve as
effective substitutes for human experts or judges (similar to Liu et al.
(2023)), offering a streamlined and efficient method for assessing
performance.

##  8 Conclusions

In this paper, we review 61 papers on controllable text summarization (CTS),
offering a detailed analysis, from formalizing various controllable
attributes, classifying them based on shared characteristics and delving into
existing datasets, proposed models, associated limitations, and evaluation
strategies for CTS tasks. Moreover, we discuss the challenges and future
prospects, making it a helpful guide for researchers interested in CTS. We
plan to keep the
GitHub111https://github.com/ashokurlana/controllable_text_summarization_survey
repository regularly updated with the latest papers in the field.

##  9 Limitations

Even though we attempt to conduct a rigorous analysis of existing literature
on controllable summarization, some works may have been possibly left out due
to variations in keywords like “text generation" vs “text summarization".
Additionally, due to space constraints, our survey focuses exclusively on the
theoretical aspects of the literature and does not include any experimental
comparisons.

## References

  * Abnar and Zuidema (2020) Samira Abnar and Willem Zuidema. 2020.  Quantifying attention flow in transformers.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4190–4197, Online. Association for Computational Linguistics.
  * Ahuja et al. (2022) Ojas Ahuja, Jiacheng Xu, Akshay Gupta, Kevin Horecka, and Greg Durrett. 2022.  ASPECTNEWS: Aspect-oriented summarization of news documents.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6494–6506, Dublin, Ireland. Association for Computational Linguistics.
  * Allahyari et al. (2017) Mehdi Allahyari, Seyedamin Pouriyeh, Mehdi Assefi, Saeid Safaei, Elizabeth D Trippe, Juan B Gutierrez, and Krys Kochut. 2017.  Text summarization techniques: a brief survey.  _arXiv preprint arXiv:1707.02268_.
  * Amplayo et al. (2021) Reinald Kim Amplayo, Stefanos Angelidis, and Mirella Lapata. 2021.  Aspect-controllable opinion summarization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6578–6593, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Angelidis et al. (2021) Stefanos Angelidis, Reinald Kim Amplayo, Yoshihiko Suhara, Xiaolan Wang, and Mirella Lapata. 2021.  Extractive opinion summarization in quantized transformer spaces.  _Transactions of the Association for Computational Linguistics_ , 9:277–293.
  * Angelidis and Lapata (2018) Stefanos Angelidis and Mirella Lapata. 2018.  Summarizing opinions: Aspect extraction meets sentiment prediction and they are both weakly supervised.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3675–3686, Brussels, Belgium. Association for Computational Linguistics.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_.
  * Bahrainian et al. (2022) Seyed Ali Bahrainian, Sheridan Feucht, and Carsten Eickhoff. 2022.  NEWTS: A corpus for news topic-focused summarization.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 493–503, Dublin, Ireland. Association for Computational Linguistics.
  * Bahrainian et al. (2021) Seyed Ali Bahrainian, George Zerveas, Fabio Crestani, and Carsten Eickhoff. 2021.  Cats: Customizable abstractive topic-based summarization.  _ACM Transactions on Information Systems (TOIS)_ , 40(1):1–24.
  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv:2004.05150_.
  * Blei et al. (2003) David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003.  Latent dirichlet allocation.  _J. Mach. Learn. Res._ , 3(null):993–1022.
  * Cao and Wang (2021) Shuyang Cao and Lu Wang. 2021.  Inference time style control for summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5942–5953, Online. Association for Computational Linguistics.
  * Cer et al. (2018) Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018.  Universal sentence encoder.  _arXiv preprint arXiv:1803.11175_.
  * Chan et al. (2021) Hou Pong Chan, Lu Wang, and Irwin King. 2021.  Controllable summarization with constrained markov decision process.  _Transactions of the Association for Computational Linguistics_ , 9:1213–1232.
  * Chawla et al. (2019) Kushal Chawla, Balaji Vasan Srinivasan, and Niyati Chhaya. 2019.  Generating formality-tuned summaries using input-dependent rewards.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 833–842, Hong Kong, China. Association for Computational Linguistics.
  * Cho (2016) Kyunghyun Cho. 2016.  Noisy parallel approximate decoding for conditional recurrent language model.  _arXiv preprint arXiv:1605.03835_.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.
  * Dang (2005) Hoa Trang Dang. 2005.  Overview of duc 2005.  In _Proceedings of the document understanding conference_ , volume 2005, pages 1–12. Citeseer.
  * Deutsch and Roth (2023) Daniel Deutsch and Dan Roth. 2023.  Incorporating question answering-based signals into abstractive summarization via salient span selection.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 575–588.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dou et al. (2021) Zi-Yi Dou, Pengfei Liu, Hiroaki Hayashi, Zhengbao Jiang, and Graham Neubig. 2021.  GSum: A general framework for guided neural abstractive summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4830–4842, Online. Association for Computational Linguistics.
  * El-Kassas et al. (2021) Wafaa S El-Kassas, Cherif R Salama, Ahmed A Rafea, and Hoda K Mohamed. 2021.  Automatic text summarization: A comprehensive survey.  _Expert systems with applications_ , 165:113679.
  * Fan et al. (2018a) Angela Fan, David Grangier, and Michael Auli. 2018a.  Controllable abstractive summarization.  _ACL 2018_ , page 45.
  * Fan et al. (2018b) Angela Fan, David Grangier, and Michael Auli. 2018b.  Controllable abstractive summarization.  In _Proceedings of the 2nd Workshop on Neural Machine Translation and Generation_ , pages 45–54, Melbourne, Australia. Association for Computational Linguistics.
  * Févry and Phang (2018) Thibault Févry and Jason Phang. 2018.  Unsupervised sentence compression using denoising auto-encoders.  In _Proceedings of the 22nd Conference on Computational Natural Language Learning_ , pages 413–422, Brussels, Belgium. Association for Computational Linguistics.
  * Flesch (1948) Rudolf Flesch. 1948.  A new readability yardstick.  _Journal of Applied Psychology_ , 32:221–233.
  * Gambhir and Gupta (2017) Mahak Gambhir and Vishal Gupta. 2017.  Recent automatic text summarization techniques: a survey.  _Artificial Intelligence Review_ , 47:1–66.
  * Gao et al. (2023) Mingqi Gao, Jie Ruan, and Xiaojun Wan. 2023.  A reproduction study of the human evaluation of role-oriented dialogue summarization models.  In _Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems_ , Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.
  * Gliwa et al. (2019) Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019.  SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization.  In _Proceedings of the 2nd Workshop on New Frontiers in Summarization_ , pages 70–79, Hong Kong, China. Association for Computational Linguistics.
  * Goldsack et al. (2023) Tomas Goldsack, Zheheng Luo, Qianqian Xie, Carolina Scarton, Matthew Shardlow, Sophia Ananiadou, and Chenghua Lin. 2023.  Overview of the biolaysumm 2023 shared task on lay summarization of biomedical research articles.  In _The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks_ , pages 468–477, Toronto, Canada. Association for Computational Linguistics.
  * Goyal et al. (2022) Tanya Goyal, Nazneen Rajani, Wenhao Liu, and Wojciech Kryscinski. 2022.  HydraSum: Disentangling style features in text summarization with multi-decoder models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 464–479, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Hayashi et al. (2021) Hiroaki Hayashi, Prashant Budania, Peng Wang, Chris Ackerson, Raj Neervannan, and Graham Neubig. 2021.  WikiAsp: A dataset for multi-domain aspect-based summarization.  _Transactions of the Association for Computational Linguistics_ , 9:211–225.
  * He et al. (2022) Junxian He, Wojciech Kryscinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. 2022.  CTRLsum: Towards generic controllable text summarization.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5879–5915, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * He et al. (2017) Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2017.  An unsupervised neural attention model for aspect extraction.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 388–397, Vancouver, Canada. Association for Computational Linguistics.
  * Hitomi et al. (2019) Yuta Hitomi, Yuya Taguchi, Hideaki Tamori, Ko Kikuta, Jiro Nishitoba, Naoaki Okazaki, Kentaro Inui, and Manabu Okumura. 2019.  A large-scale multi-length headline corpus for analyzing length-constrained headline generation model evaluation.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 333–343, Tokyo, Japan. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Hofmann-Coyle et al. (2022) Ella Hofmann-Coyle, Mayank Kulkarni, Lingjue Xie, Mounica Maddela, and Daniel Preotiuc-Pietro. 2022.  Extractive entity-centric summarization as sentence selection using bi-encoders.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 326–333, Online only. Association for Computational Linguistics.
  * Hsu and Tan (2021) Chao-Chun Hsu and Chenhao Tan. 2021.  Decision-focused summarization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 117–132, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Huang et al. (2023) Kung-Hsiang Huang, Siffi Singh, Xiaofei Ma, Wei Xiao, Feng Nan, Nicholas Dingwall, William Yang Wang, and Kathleen McKeown. 2023.  SWING: Balancing coverage and faithfulness for dialogue summarization.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , Dubrovnik, Croatia. Association for Computational Linguistics.
  * Hyun et al. (2022) Dongmin Hyun, Xiting Wang, Chayoung Park, Xing Xie, and Hwanjo Yu. 2022.  Generating multiple-length summaries via reinforcement learning for unsupervised sentence summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 2939–2951, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Ippolito et al. (2019) Daphne Ippolito, Reno Kriz, João Sedoc, Maria Kustikova, and Chris Callison-Burch. 2019.  Comparison of diverse decoding methods from conditional language models.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3752–3762.
  * Iskender et al. (2021) Neslihan Iskender, Tim Polzehl, and Sebastian Möller. 2021.  Reliability of human evaluation for text summarization: Lessons learned and challenges ahead.  In _Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)_ , Online. Association for Computational Linguistics.
  * Ito et al. (2023) Takumi Ito, Qixiang Fang, Pablo Mosteiro, Albert Gatt, and Kees van Deemter. 2023.  Challenges in reproducing human evaluation results for role-oriented dialogue summarization.  In _Proceedings of the 3rd Workshop on Human Evaluation of NLP Systems_ , Varna, Bulgaria. INCOMA Ltd., Shoumen, Bulgaria.
  * Jin et al. (2020a) Di Jin, Zhijing Jin, Joey Tianyi Zhou, Lisa Orii, and Peter Szolovits. 2020a.  Hooks in the headline: Learning to generate headlines with controlled styles.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5082–5093, Online. Association for Computational Linguistics.
  * Jin et al. (2020b) Hanqi Jin, Tianming Wang, and Xiaojun Wan. 2020b.  Semsum: Semantic dependency guided neural abstractive summarization.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 8026–8033.
  * Kikuchi et al. (2016) Yuta Kikuchi, Graham Neubig, Ryohei Sasano, Hiroya Takamura, and Manabu Okumura. 2016.  Controlling output length in neural encoder-decoders.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1328–1338.
  * Krishna et al. (2018) Kundan Krishna, Aniket Murhekar, Saumitra Sharma, and Balaji Vasan Srinivasan. 2018.  Vocabulary tailored summary generation.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 795–805.
  * Krishna and Srinivasan (2018) Kundan Krishna and Balaji Vasan Srinivasan. 2018.  Generating topic-oriented summaries using neural attention.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1697–1705, New Orleans, Louisiana. Association for Computational Linguistics.
  * Kryściński et al. (2018) Wojciech Kryściński, Romain Paulus, Caiming Xiong, and Richard Socher. 2018.  Improving abstraction in text summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1808–1817, Brussels, Belgium. Association for Computational Linguistics.
  * Kulikov et al. (2019) Ilia Kulikov, Alexander Miller, Kyunghyun Cho, and Jason Weston. 2019.  Importance of search and evaluation strategies in neural dialogue modeling.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , Tokyo, Japan. Association for Computational Linguistics.
  * Kwon et al. (2023) Jingun Kwon, Hidetaka Kamigaito, and Manabu Okumura. 2023.  Abstractive document summarization with summary-length prediction.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 606–612.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdel rahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Annual Meeting of the Association for Computational Linguistics_.
  * Li et al. (2018) Chenliang Li, Weiran Xu, Si Li, and Sheng Gao. 2018.  Guiding generation for abstractive text summarization based on key information guide network.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 55–60, New Orleans, Louisiana. Association for Computational Linguistics.
  * Li and Jurafsky (2016) Jiwei Li and Dan Jurafsky. 2016.  Mutual information and diverse decoding improve neural machine translation.  _arXiv preprint arXiv:1601.00372_.
  * Li et al. (2016) Jiwei Li, Will Monroe, and Dan Jurafsky. 2016.  A simple, fast diverse decoding algorithm for neural generation.  _arXiv preprint arXiv:1611.08562_.
  * Liang et al. (2022) Xinnian Liang, Chao Bian, Shuangzhi Wu, and Zhoujun Li. 2022.  Towards modeling role-aware centrality for dialogue summarization.  In _Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 43–50, Online only. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lin et al. (2021) Haitao Lin, Liqun Ma, Junnan Zhu, Lu Xiang, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2021.  CSDS: A fine-grained Chinese dataset for customer service dialogue summarization.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4436–4451, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Lin et al. (2022) Haitao Lin, Junnan Zhu, Lu Xiang, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2022.  Other roles matter! enhancing role-oriented dialogue summarization via role interactions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2545–2558, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2022a) Puyuan Liu, Xiang Zhang, and Lili Mou. 2022a.  A character-level length-control algorithm for non-autoregressive sentence summarization.  _Advances in Neural Information Processing Systems_ , 35:29101–29112.
  * Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023.  G-eval: Nlg evaluation using gpt-4 with better human alignment, may 2023.  _arXiv preprint arXiv:2303.16634_.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Text summarization with pretrained encoders.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3730–3740, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu et al. (2022b) Yizhu Liu, Qi Jia, and Kenny Zhu. 2022b.  Length control in abstractive summarization by pretraining information selection.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 6885–6895, Dublin, Ireland. Association for Computational Linguistics.
  * Liu et al. (2018) Yizhu Liu, Zhiyi Luo, and Kenny Zhu. 2018.  Controlling length in abstractive summarization using a convolutional neural network.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4110–4119.
  * Liu and Chen (2021) Zhengyuan Liu and Nancy F. Chen. 2021.  Controllable neural dialogue summarization with personal named entity planning.  In _Conference on Empirical Methods in Natural Language Processing_.
  * Lundberg and Lee (2017) Scott M Lundberg and Su-In Lee. 2017.  A unified approach to interpreting model predictions.  _Advances in neural information processing systems_ , 30.
  * Luo et al. (2022) Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. 2022.  Readability controllable biomedical document summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 4667–4680, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Luong et al. (2015) Minh-Thang Luong, Hieu Pham, and Christopher D Manning. 2015.  Effective approaches to attention-based neural machine translation.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1412–1421.
  * Maddela et al. (2022) Mounica Maddela, Mayank Kulkarni, and Daniel Preotiuc-Pietro. 2022.  EntSUM: A data set for entity-centric extractive summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3355–3366, Dublin, Ireland. Association for Computational Linguistics.
  * Makino et al. (2019) Takuya Makino, Tomoya Iwakura, Hiroya Takamura, and Manabu Okumura. 2019.  Global optimization under length constraint for neural text summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1039–1048, Florence, Italy. Association for Computational Linguistics.
  * Mukherjee et al. (2020) Rajdeep Mukherjee, Hari Chandana Peruri, Uppada Vishnu, Pawan Goyal, Sourangshu Bhattacharya, and Niloy Ganguly. 2020.  Read what you need: Controllable aspect-based opinion summarization of tourist reviews.  In _Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval_ , pages 1825–1828.
  * Mukherjee et al. (2022) Sourajit Mukherjee, Anubhav Jangra, Sriparna Saha, and Adam Jatowt. 2022.  Topic-aware multimodal summarization.  In _Findings of the Association for Computational Linguistics: AACL-IJCNLP 2022_ , pages 387–398, Online only. Association for Computational Linguistics.
  * Nallapati et al. (2017) Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017.  Summarunner: A recurrent neural network based sequence model for extractive summarization of documents.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 31.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gülçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence RNNs and beyond.  In _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning_ , Berlin, Germany. Association for Computational Linguistics.
  * Napoles et al. (2012) Courtney Napoles, Matthew R Gormley, and Benjamin Van Durme. 2012.  Annotated gigaword.  In _Proceedings of the joint workshop on automatic knowledge base construction and web-scale knowledge extraction (AKBC-WEKEX)_ , pages 95–100.
  * Narayan et al. (2018) Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807, Brussels, Belgium. Association for Computational Linguistics.
  * Narayan et al. (2022) Shashi Narayan, Gonçalo Simões, Yao Zhao, Joshua Maynez, Dipanjan Das, Michael Collins, and Mirella Lapata. 2022.  A well-composed text is half done! composition sampling for diverse conditional generation.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1319–1339, Dublin, Ireland. Association for Computational Linguistics.
  * Narayan et al. (2021) Shashi Narayan, Yao Zhao, Joshua Maynez, Gonçalo Simões, Vitaly Nikolaev, and Ryan McDonald. 2021.  Planning with learned entity prompts for abstractive summarization.  _Transactions of the Association for Computational Linguistics_ , 9:1475–1492.
  * Nazari and Mahdavi (2019) Narges Nazari and MA Mahdavi. 2019.  A survey on automatic text summarization.  _Journal of AI and Data Mining_ , 7(1):121–135.
  * Over and Yen (2004) Paul Over and James Yen. 2004.  An introduction to duc-2004.  _National Institute of Standards and Technology_.
  * Pagnoni et al. (2023) Artidoro Pagnoni, Alex Fabbri, Wojciech Kryscinski, and Chien-Sheng Wu. 2023.  Socratic pretraining: Question-driven pretraining for controllable summarization.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12737–12755, Toronto, Canada. Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  Bleu: a method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics_ , pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21(140):1–67.
  * Ribeiro et al. (2023) Leonardo FR Ribeiro, Mohit Bansal, and Markus Dreyer. 2023.  Generating summaries with controllable readability levels.  _arXiv preprint arXiv:2310.10623_.
  * Rush et al. (2015) Alexander M Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 379–389.
  * Sandhaus (2008) Evan Sandhaus. 2008.  The new york times annotated corpus.  _Linguistic Data Consortium, Philadelphia_ , 6(12):e26752.
  * Sarkhel et al. (2020) Ritesh Sarkhel, Moniba Keymanesh, Arnab Nandi, and Srinivasan Parthasarathy. 2020.  Interpretable multi-headed attention for abstractive summarization at controllable lengths.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6871–6882, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1073–1083, Vancouver, Canada. Association for Computational Linguistics.
  * Sekine and Nobata (2003) Satoshi Sekine and Chikashi Nobata. 2003.  A survey for multi-document summarization.  In _Proceedings of the HLT-NAACL 03 Text Summarization Workshop_ , pages 65–72.
  * Shen et al. (2022a) Chenhui Shen, Liying Cheng, Lidong Bing, Yang You, and Luo Si. 2022a.  SentBS: Sentence-level beam search for controllable summarization.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10256–10265, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Shen et al. (2022b) Chenhui Shen, Liying Cheng, Ran Zhou, Lidong Bing, Yang You, and Luo Si. 2022b.  MReD: A meta-review dataset for structure-controllable text generation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2521–2535, Dublin, Ireland. Association for Computational Linguistics.
  * Shen et al. (2022c) Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey. 2022c.  Multi-lexsum: Real-world summaries of civil rights lawsuits at multiple granularities.  _Advances in Neural Information Processing Systems_ , 35:13158–13173.
  * Song et al. (2021) Kaiqiang Song, Bingqing Wang, Zhe Feng, and Fei Liu. 2021.  A new approach to overgenerating and scoring abstractive summaries.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1392–1404, Online. Association for Computational Linguistics.
  * Song et al. (2020) Kaiqiang Song, Bingqing Wang, Zhe Feng, Ren Liu, and Fei Liu. 2020.  Controlling the amount of verbatim copying in abstractive summarization.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 8902–8909.
  * Suhara et al. (2020) Yoshihiko Suhara, Xiaolan Wang, Stefanos Angelidis, and Wang-Chiew Tan. 2020.  OpinionDigest: A simple framework for opinion summarization.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5789–5798, Online. Association for Computational Linguistics.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _International conference on machine learning_ , pages 3319–3328. PMLR.
  * Takase and Okazaki (2019) Sho Takase and Naoaki Okazaki. 2019.  Positional encoding to control output sequence length.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 3999–4004, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Tam (2020) Yik-Cheung Tam. 2020.  Cluster-based beam search for pointer-generator chatbot grounded by knowledge.  _Computer Speech & Language_, 64:101094.
  * Tan et al. (2020) Bowen Tan, Lianhui Qin, Eric Xing, and Zhiting Hu. 2020.  Summarizing text on any aspects: A knowledge-informed weakly-supervised approach.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6301–6309, Online. Association for Computational Linguistics.
  * Tuggener et al. (2021) Don Tuggener, Margot Mieskes, Jan Deriu, and Mark Cieliebak. 2021.  Are we summarizing the right way? a survey of dialogue summarization data sets.  In _Proceedings of the Third Workshop on New Frontiers in Summarization_ , pages 107–118, Online and in Dominican Republic. Association for Computational Linguistics.
  * Wang et al. (2022) Jiaan Wang, Fandong Meng, Duo Zheng, Yunlong Liang, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2022.  A survey on cross-lingual summarization.  _Transactions of the Association for Computational Linguistics_ , 10:1304–1323.
  * Wu et al. (2021) Chien-Sheng Wu, Linqing Liu, Wenhao Liu, Pontus Stenetorp, and Caiming Xiong. 2021.  Controllable abstractive dialogue summarization with sketch supervision.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 5108–5122, Online. Association for Computational Linguistics.
  * Yu et al. (2021) Zhongyi Yu, Zhenghao Wu, Hao Zheng, Zhe XuanYuan, Jefferson Fong, and Weifeng Su. 2021.  LenAtten: An effective length controlling unit for text summarization.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 363–370, Online. Association for Computational Linguistics.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 27263–27277. Curran Associates, Inc.
  * Zhang et al. (2023a) Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2023a.  A survey of controllable text generation using transformer-based pre-trained language models.  _ACM Computing Surveys_ , 56(3):1–37.
  * Zhang* et al. (2020) Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhang et al. (2023b) Yusen Zhang, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng, and Rui Zhang. 2023b.  Macsum: Controllable summarization with mixed attributes.  _Transactions of the Association for Computational Linguistics_ , 11:787–803.
  * Zheng et al. (2020) Changmeng Zheng, Yi Cai, Guanjie Zhang, and Qing Li. 2020.  Controllable abstractive sentence summarization with guiding entities.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5668–5678, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Zhong et al. (2022) Ming Zhong, Yang Liu, Suyu Ge, Yuning Mao, Yizhu Jiao, Xingxing Zhang, Yichong Xu, Chenguang Zhu, Michael Zeng, and Jiawei Han. 2022.  Unsupervised multi-granularity summarization.  In _Findings of the Association for Computational Linguistics: EMNLP 2022_ , pages 4980–4995, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Zhong et al. (2021) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021.  QMSum: A new benchmark for query-based multi-domain meeting summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5905–5921, Online. Association for Computational Linguistics.
  * Zhong and Litman (2023) Yang Zhong and Diane Litman. 2023.  Strong–structure controllable legal opinion summary generation.  _arXiv preprint arXiv:2309.17280_.

##  Appendix A Appendix

Aspect | Paper | Novel contribution | Dataset(s) | Limitations
---|---|---|---|---
Structure | Shen et al. (2022b) | Prepend structure prompt to the input | MRed | Subsequent generations deviate from the desired output
Shen et al. (2022a) | Sentence-beam approach | MRed | Decoding methods significantly impact performance
Zhong and Litman (2023) | Utilize predicted-role argument to control the structure | CanLII | Computationally expensive
Zhang et al. (2023b) | Prepend entity chains as planning objective to the input | MACSUM | -
Abstractivity | See et al. (2017) | Pointer-generator network | CNNDM |  | Failed to achieve higher abstraction and
---
ineffective in core text selection
Kryściński et al. (2018) |  | Decouples the decoder into a contextual network and
---
mixed RL objective to encourage abstraction
CNNDM | -
Song et al. (2020) |  | Mix-and-match strategy to generate summaries with
---
various degree of copying levels
Gigaword, NEWSROOM | -
Chan et al. (2021) |  | RL-based framework on constrained markov decision
---
process to penalize the violation of control requirement
CNNDM, NEWSROOM | -
Diversity | Narayan et al. (2022) | Compositional sampling decoding method | CNNDM, XSUM | -
Style | Fan et al. (2018a) |  | Convolutional encoder-decoder to generate stylistic
---
summaries by adding the source prompt to the input
CNNDM, DUC2004 | -
Chawla et al. (2019) | RL-based method to generate formality-tailored summaries | CNNDM, Webis-TLDR-17 | -
Jin et al. (2020a) |  | Multi-task learning framework with style-dependent
---
layer normalization and style-guided encoder attention
| NYT, CNN, Humor,
---
Romance, Clickbait corpus
-
Cao and Wang (2021) |  | Novel decoding methods: decode state adjustment,
---
word unit prediction based
| Hyperpartisan News
---
detection dataset
-
Goyal et al. (2022) | Mixture of experts strategy | CNNDM, XSUM, NEWSROOM | Manual gating mechanism
Luo et al. (2022) | Readability control of bio-medical documents | LS, PLS | Fail to handle fine-grained readability control
Ribeiro et al. (2023) | Fine-grained readability control | CNNDM |  | Style insights may not generalize beyond English
---
newswire datasets
Coverage | Wu et al. (2021) | A two-stage control generation strategy | SAMSUM | -
Zhong et al. (2022) |  | Unsupervised framework to multi-granularity summary
---
generation
| Multi-NEWS, arXiv,
---
DUC2004
| Events extraction from source may effect the
---
abstractiveness
Zhang et al. (2023b) | Soft and hard prompting to control the granularity | MACSUM | -
Huang et al. (2023) | Utilize the NLI models to improve the coverage | DIALOGSUM, SAMSUM | Partially addressing the factuality problem
Role | Lin et al. (2022) |  | Decoders for user and agent summaries and
---
attention divergence loss for the same topic
CSDS, MC | -
Liang et al. (2022) |  | Role aware centrality scores to reweight
---
context representations for decoding
CSDS,MC | -
Entity | Zheng et al. (2020) | Controllable neural network with guiding entities | GigaWord, DUC 2004 | -
Liu and Chen (2021) |  | Graph convolutional network based coreference fusion
---
layer and entity conditioned Summary Generation
SAMSUM |  | Paraphrasing introduces factual inconsistencies
---
in person-specific summaries
Hofmann-Coyle et al. (2022) |  | Model as a sentence selection task using transformer
---
based biencoder with a cosine similarity based loss
and adapting contrastive loss
EntSUM | -
Salience | Nallapati et al. (2017) |  | Summarization as a sentence selection task with salience
---
as a feature using sequence-to-sequence model
CCNDM | Poor performance on out-of-domain datasets
Li et al. (2018) | Key information guided network with modified attention | CCNDM | Coverage mechanism not implemented
Deutsch and Roth (2023) |  | Model salience in terms of noun phrases by
---
incorporating QA signals
CCNDM, DUC-2004 |  | Performance relies on question generation
---
and answering models
Pagnoni et al. (2023) |  | Unsupervised pretraining involving
---
salient sentence selection
QMSum, SQuALITY | Computationally expensive
Length | Kikuchi et al. (2016) | Remaining words provided as additional input to decoder | Gigaword | Poor performance on DUC-2004
Fan et al. (2018a) |  | Convolutional encoder-decoder, summary length
---
grouping into bins and the source document
prepend with length bin’s value
CNNDM | Fails to generate summaries of arbitrary lengths
Liu et al. (2018) |  | Remaining number of tokens replaced by characters at
---
the decoder
CNNDM, DMQA | Fails to generalize to new control aspects at test time
Févry and Phang (2018) |  | Unsupervised denoising auto-encoder for the task of
---
sentence compression and the decoder provided with
an additional input of the remaining summary length
at each time step
Gigaword | Unfaithful summary generation in some cases
Hitomi et al. (2019) |  | Desired number of tokens were replaced by desired
---
number characters at the decoder
JAMUS corpus (Japanese) | -
Makino et al. (2019) |  | Global minimum risk training optimization method
---
under length constraint
CNNDM, Mainichi | Fails to control length
Sarkhel et al. (2020) |  | Multi-level summarizer with a multi-headed attention
---
mechanism using a series of timestep
independent semantic kernels
| MSR Narratives and
---
Thinking-Machines
Fail to encode desired length
Takase and Okazaki (2019) |  | Extension to the sinusoidal positional embeddings to
---
preserve the length constraint with length-difference
positional encoding and length-ratio positional encoding
JAMUS corpus (Japanese) | Poor performance when desired target length is unseen
Yu et al. (2021) |  | Concatenate the length context vector with the decoder
---
hidden state and other attention vectors
CNNDM | Incomplete shorter summary generation
Song et al. (2021) |  | Confidence driven generator trained on a denoising
---
objective with a decoder only architecture with masked
source and summary tokens
Gigaword, NEWSROOM | Poor performance on large datasets
Chan et al. (2021) |  | Used a reinforcement learning based Constrained
---
Markov Decision Process to control length along
with constraints on a mix of attributes such
as abstractiveness and covered entity
| CNNDM, NEWSROOM
---
DUC-2002
Length control only at word level
Liu et al. (2022a) |  | Dynamic programming algorithm based on the
---
Connectionist Temporal Classification model
Gigaword, DUC2004 |  | Poor performance compared to
---
autoregressive models
Goyal et al. (2022) | Mixture-of-expert model with multiple decoders |  | CNNDM, XSUM,
---
NEWSROOM
| No insights about style diversity in
---
non-English and non-newswire datasets
He et al. (2022) | A generic framework using keywords | CNNDM, arXiv, BIGPATENT | High reliance on the quality of extracted keywords
Liu et al. (2022b) |  | Length aware attention model adapting the source
---
encodings
CNNDM, XSUM |  | Performance directly proportional to the
---
summary length
Zhong et al. (2022) |  | Events identification with unsupervised summary
---
generation
| GranuDUC, MultiNews,
---
DUC2004, Arxiv
Fails to capture abstractness due to event extraction
Hyun et al. (2022) |  | RL based framework incorporating both the length
---
and quality constraints in the reward function
DUC2004 | Computationally expensive
Kwon et al. (2023) |  | Summary length prediction task on the encoder side
---
and encoded this information inserting a
length-fusion positional encoding layer
CNNDM, NYT, WikiHow |  | Performance decreases with increase in summary
---
length variance
Zhang et al. (2023b) | Hard prompt tuning and soft prefix tuning | CNNDM, QMSum | Low specificity in long generated summaries
Topic | Krishna and Srinivasan (2018) |  | RNN based attention model to generate multiple topic
---
conditioned summaries
CNNDM |  | News categories provide predefined topics,
---
limiting generalization to other tasks.
Tan et al. (2020) |  | Extends topic based summarization to arbitrary topics,
---
integrating external knowledge from ConceptNet and Wikipedia
| CNNDM, MA News,
---
All the News
-
Suhara et al. (2020) | Framework for opinion summarization | HOTEL, Yelp | -
Amplayo et al. (2021) |  | Multi-Instance Learning and a document preprocessing
---
mechanism
SPACE, OPOSUM+ | Incapable of handling unseen aspects
Mukherjee et al. (2020) | Iterative sentence extraction algorithm | YELP | -
Mukherjee et al. (2022) | Topic-aware multimodal summarization system | MSMO | Output quality relies on data size

Table 5: CTS models descriptions and corresponding limitations.

| Paper | Publication info | Controllable aspect | Model details | Dataset details | Automatic evaluation | Human Evaluation | Limitations | Reproducible
---|---|---|---|---|---|---|---|---|---
Year | Venue |  | Controllable
---
attribute
| Controlling
---
more than
one aspect
Model type* | Training strategy | Approach | Code access | Code link | Dataset used | Source | Nature of the data | Released data? | Domain | Data link | Metric name | Proposed New Metric | Human Evaluation | Metrics names | IAA Yes or No | Mentioned? | High/Medium/Not sure*
1 | Chawla et al. (2019) | 2019 | CoNLL | Style | No | encoder-decoder | ante-hoc | supervised | no | - |  | CNN Daily Mail +
---
Webis-TLDR-17 corpus
document | generic | no | news, social media | no | ROUGE | no | no | no | no | no | not sure
2 | Goyal et al. (2022) | 2022 | EMNLP | Style | Yes | encoder-decoder | ante-hoc | supervised | yes |  Link |  | CNN Daily Mail,
---
XSUM, Newsroom
document | generic | no | news | no | ROUGE |  | abstractiveness,
---
degree of specificity,
length, readability
yes |  | relevance, coherence,
---
grammaticality,
factuality
no | yes | high
3 | Song et al. (2021) | 2021 | NAACL | Length | No | decoder | ante-hoc | supervised | yes |  Link | Gigaword, Newsroom | document | generic | yes | news |  Link | ROUGE | no | yes | yes-no type questions | no | no | high
4 | Song et al. (2020) | 2020 | AAAI | Abstractivity | No | decoder | ante-hoc | supervised | yes |  Link | Gigaword, Newsroom | document | generic | no | news | no | ROUGE, BERTScore | no | yes |  | informativeness,
---
grammaticality,
truthfulness
no | no | high
5 | Fan et al. (2018a) | 2018 | ACL-NMT(W) |  | Length, Style
---
Entity
Yes | encoder-decoder | pre-hoc | supervised | no | - | CNN-DailyMail | document | generic | no | news | no | ROUGE | no | yes |  | rater asked to preferred summary,
---
binary evaluation
no | no | medium
6 | Liu et al. (2022b) | 2022 | ACL | Length | No | encoder-decoder | ante-hoc | supervised | yes |  Link | CNN-DailyMail, XSUM | document | generic | no | news | no | ROUGE | no | yes |  | grammatically correct,
---
informativeness, overall
no | no | high
7 | Zhang et al. (2023b) | 2023 | TACL |  | Length, Topic
---
Coverage
Yes | encoder-decoder | pre-hoc | supervised | yes |  Link | CNN Daily Mail, QMSum | document, dialogues | human annotated | yes | news, social media |  Link |  | Control Error Rate,
---
Control Correlation
| control error rate,
---
control correlation
yes | yes or no questions | yes | no | high
8 | Takase and Okazaki (2019) | 2019 | NAACL | Length | No | encoder-decoder | ante-hoc | supervised | yes |  Link | JAMUS | document | generic | no | news | no | ROUGE | no | no | no | no | no | medium
9 | Shen et al. (2022a) | 2022 | EMNLP | Structure | No | encoder-decoder | post-hoc | supervised | yes |  Link | Meta Review Dataset (MReD) | reviews | generic | no | scientific | no | ROUGE,BERTSCORE | no | yes |  | fluency, content relevance,
---
structure similarity,
decision correctness
no | yes | high
10 | Goldsack et al. (2023) | 2023 | ACL-BIoNLP(W) | Style | No | - | - | - | - | - | PLOS and eLife | - | - | - | - | - |  | Relevance, Readability,
---
Factuality
- | - | - | no | - | -
11 | Cao and Wang (2021) | 2021 | NAACL | Style | No | encoder-decoder | ante-hoc | supervised | yes |  Link | CNNDM | document | generic | no | news | no |  | Readability, Perplexity,
---
BERTScore
| readability,
---
perplexity
yes |  | informativeness,
---
fluency, relevance
no | no | high
12 | Narayan et al. (2021) | 2021 | TACL | Structure | No | encoder-decoder | ante-hoc | supervised | no | - |  | CNN/DailyMail, XSum,
---
SAMSum, and BillSum
document, dialogues | generic | no | news, legal | no | ROUGE |  | entity planning,
---
specificity
yes |  | relevant, accurate,
---
concise, fluent
no | no | medium
13 | Amplayo et al. (2021) | 2021 | EMNLP | Topic | No | encoder-decoder | ante-hoc | supervised | yes |  Link | SPACE, OPOSUM+ | reviews | generic | no | social media | no | ROUGE | no | yes |  | informativeness,coherence,
---
conciseness, fluency
no | no | high
14 | Deutsch and Roth (2023) | 2023 | EACL | Saliency | No | encoder-decoder | ante-hoc | supervised | no | - | CNNDM, XSUM, NYTimes | document | generic | no | news | no |  | ROUGE, BERTScore,
---
QAEval
no | yes |  | selection of best summary
---
(binary evaluation)
no | no | not sure
15 | Wu et al. (2021) | 2021 | ALC-IJCNLP | Coverage | No | encoder-decoder | ante-hoc | supervised | yes |  Link | SAMSum | dialogues | generic | no | social media | no | ROUGE | no | yes |  | factual consistency,
---
informativeness
no | no | high
16 | Hsu and Tan (2021) | 2021 | EMNLP | Topic | No | encoder-decoder | ante-hoc | supervised | yes |  Link |  | Yelp’s businesses, reviews,
---
and user data
reviews | generic | no | reastaurant reviews |  Link | MSE | no | yes | - | no | no | high
17 | Hyun et al. (2022) | 2022 | EMNLP | Length | Yes | RL | ante-hoc | unsupervised | yes |  Link | DUC2004 | document | generic | no | news | no | ROUGE | no | yes | fluency, fidelity | yes | yes | high
18 | Chan et al. (2021) | 2021 | TACL |  | Length, Entity,
---
Abstractivity
Yes | RL | ante-hoc | supervised | yes |  Link | CNNDM, Newsroom, DUC-2002 | document | generic | no | news | no |  | ROUGE, BERTSCORE,
---
MoveScore, QA-F
yes | yes |  | fluency, entity relevance,
---
faithfulness
yes | no | high
19 | Krishna and Srinivasan (2018) | 2018 | NAACL-HLT | Topic | No | encoder-decoder | pre-hoc | supervised | no | - | CNNDM | document | generic | no | news | no | ROUGE | no | yes | topic relevance | no | no | not sure
20 | Krishna et al. (2018) | 2018 | ACL | Topic | No | encoder-decoder | ante-hoc | supervised | no | - | CNNDM | document | generic | no | news | no | ROUGE | no | yes | quality | yes | no | not sure
21 | Rush et al. (2015) | 2015 | EMNLP | Length | No | encoder-decoder | ante-hoc | supervised | no | - | NYT, DUC2004 | document | generic | no | news | no | ROUGE | no | no | no | no | no | high
22 | Févry and Phang (2018) | 2018 | CoNLL | Length | No | encoder-decoder | ante-hoc | unsupervised | yes |  Link | Gigaword | document | generic | no | news | no | ROUGE | no | yes |  | grammaticality,
---
preserving original information
no | no | high
23 | Li et al. (2018) | 2018 | NAACL-HLT | Saliency | No | encoder-decoder | ante-hoc | supervised | no | - | CNNDM | document | generic | no | news | no | ROUGE | no | no | no | no | no | not sure
24 | Jin et al. (2020b) | 2020 | AAAI | Coverage | No | encoder-decoder | ante-hoc | supervised | no | - |  | Gigaword, DUC2004,
---
MSR dataset
document | generic | no | news |  Link | ROUGE |  | Word Mover’s Distance,
---
Bert Score
yes |  | faithfulness, informativeness,
---
fluency
no | no | not sure
25 | Liu et al. (2022a) | 2022 | Neurips | Length | No |  | Non Auto Regressive Model
---
ante-hoc | supervised | yes |  Link | Gigaword, DUC2004 | document | generic | no | news | no | ROUGE | no | yes |  | overall quality,
---
completeness/fluency
no | yes | medium
26 | Kwon et al. (2023) | 2023 | EACL | Length | No | encoder-decoder | ante-hoc | supervised | no | - | CNNDM, NYT, WikiHow | document | generic | no | news | no | ROUGE, Length Variance | no | yes |  | informativeness,
---
conciseness
no | yes | not sure
27 | Pagnoni et al. (2023) | 2023 | ACL | Entity, Saliency | yes | encoder-decoder | ante-hoc | supervised | yes |  Link | QMSum and SQuALITY | document | generic | no | news | no |  | ROUGE, BERTSCORE,
---
levenshtein distance
no | yes | unambiguity, conciseness | no | yes | high
28 | Hofmann-Coyle et al. (2022) | 2022 | AACL | Entity | No | encoder-decoder | ante-hoc | supervised | no | - | EntSum | document | generic | no | news | no | F1 | no | no | no | no | no | not sure
29 | Lin et al. (2022) | 2022 | ACL | Role | No | encoder-decoder | ante-hoc | supervised | yes |  Link | CSDS, MC | dialogues | generic | no |  | customer service data,
---
medical inquiries
Link |  | ROUGE, BertScore,
---
BLEU, MoverScore
no | yes |  | informativeness,
---
non-redundancy, fluency
yes | no | high
30 | Zhong et al. (2022) | 2022 | EMNLP | Coverage | Yes | encoder-decoder | ante-hoc | unsupervised | no | - |  | GranuDUC, MultiNews,
---
DUC2004, Arxiv
document | generic | yes | news, scientific papers |  Link | ROUGE | BERTScore | yes |  | fluency,
---
relevance, faithfulness
no | yes | medium
31 | Liang et al. (2022) | 2022 | AACL | Role | Yes | encoder-decoder | ante-hoc | supervised | no | - | CSDS, MC | dialogues | generic | no |  | customer service data,
---
medical inquiries
Link |  | ROUGE, BLEU,
---
BERTScore, MoverScore
no | no | no | no | no | not sure
32 | Liu and Chen (2021) | 2021 | EMNLP | Entity | no | encoder-decoder | ante-hoc | supervised | no |  Link | SAMSum | dialogues | generic | no | social media | no | ROUGE | no | yes |  | factual consistency,
---
informativeness
no | no | medium
33 | Zheng et al. (2020) | 2020 | COLING | Entity | no | encoder-decoder | ante-hoc | supervised | yes |  Link | Gigaword, DUC2004 | document | generic | no | news | no | BERT-REO, ROUGE | BERT-REO | yes |  | informativeness,
---
grammaticality,
coherence
no | no | high
34 | Huang et al. (2023) | 2023 | EACL | Coverage | no | encoder-decoder | ante-hoc | supervised | yes |  Link | DIALOG-SUM, SAMSUM | dialogues | generic | no | social media | no |  | BARTScore, ROUGE,
---
FactCC
no | yes |  | recall, precision,
---
faithfulness
yes | yes | high
35 | Mukherjee et al. (2022) | 2022 | AACL | Topic | no | encoder-decoder | ante-hoc | supervised | yes |  Link | MSMO | document | semi-automatic | yes | news | Link | ROUGE | no | yes |  | Coverage, Grammar,
---
Topic-Aware-Text
no | yes | high
36 | Jin et al. (2020a) | 2020 | ACL | Style | yes | encoder-decoder | ante-hoc | supervised | yes |  Link | NYT, CNN | document | generic | no | news | no |  | BLEU, ROUGE,
---
METEOR, CIDEr
no | yes |  | relevance, attractiveness,
---
language fluency,
style strength
no | no | high
37 | See et al. (2017) | 2017 | ACL | Abstractivity, Coverage | yes | encoder-decoder | ante-hoc | supervised | yes |  Link | CNNDM | document | generic | no | news | no | ROUGE | no | no | no | no | no | high
38 | Kryściński et al. (2018) | 2018 | EMNLP | Abstractivity | no | encoder-decoder | ante-hoc | supervised | no | - | CNNDM | document | generic | no | news | no | ROUGE | no | yes | readibility, relevance | no | no | not sure
39 | Bahrainian et al. (2021) | 2021 | ACM | Topic | no | encoder-decoder | ante-hoc | supervised | yes |  Link | CNNDM, AMI , ICSI, ADSE | document | generic | no | news., meetings | no | ROUGE | no | yes | informativeness, readability | yes | no | high
40 | Kikuchi et al. (2016) | 2016 | EMNLP | Length | no | encoder-decoder | ante-hoc, post-hoc | supervised | yes |  Link | DUC2004, Gigaword | document | generic | no | news | no | ROUGE | no | no | no | no | no | high
41 | Liu et al. (2018) | 2018 | EMNLP | Length | no | encoder-decoder | ante-hoc | supervised | yes |  Link | CNNDM, DMQA | document | generic | no | news | no |  | ROUGE,
---
VAR, Similarity
no | no | no | no | no | high
42 | Makino et al. (2019) | 2019 | ACL | Length | no | encoder-decoder | ante-hoc | supervised | no | - | CNNDM, Mainichi | document | generic | no | news | no | ROUGE | no | yes | post-editing-time | no | no | not sure
43 | Yu et al. (2021) | 2021 | ACL | Length | no | encoder-decoder | ante-hoc | supervised | yes |  Link | CNNDM | document | generic | no | news | no | ROUGE | no | yes |  | Correctness, completeness
---
fluency
no | no | high
44 | Sarkhel et al. (2020) | 2020 | COLING | Length | no | encoder-decoder | ante-hoc | supervised | no | - |  | MSR Narratives,
---
Thinking-Machines
document | generic | no |  | social media,
---
science/education
no | ROUGE, METEOR | no | yes | presence of key facts | no | no | not sure
45 | He et al. (2022) | 2022 | EMNLP | Length, Entity | yes | encoder-decoder | ante-hoc | supervised | yes |  Link | CNNDM, arXiv, BIGPATENT | document | generic | no | news, scientific data | no | ROUGE | no | yes |  | Control Accuracy,
---
Control Relevance
no | yes | high
46 | Tan et al. (2020) | 2020 | EMNLP | Topic | no | encoder-decoder | ante-hoc | supervised | yes |  Link | CNN -Dailymail, MA News | document | generic | no | news | no | ROUGE | no | yes |  | Accuracy , Informativeness,
---
Fluency
yes | no | high
47 | Narayan et al. (2022) | 2022 | ACL | Diversity | no | encoder-decoder | post-hoc | supervised | yes(partial) |  Link |  | CNN/DailyMail, XSum,
---
SQuAD)
document | generic | no | news | no |  | ROUGE,BertScore,
---
SELF-BLEU
no | yes | faithfulness and diversity | no | no | medium
48 | Zhong and Litman (2023) | 2023 | IJCNLP-AACL | Structure | no | encoder-decoder | ante-hoc | supervised | no | - | CanLII | document | generic | no | legal | no |  | ROUGE, BERTScore,
---
structure similarity.
no | yes | coherence, coverage | no | yes | not sure
49 | Suhara et al. (2020) | 2020 | ACL | Topic | yes | extractor + generator(decoder) | ante-hoc | unsupervised | yes |  Link | Hotel, Yelp | reviews | generic | no | reviews | no | ROUGE | no | yes |  | informativeness, coherence,
---
non-redundancy,
content-support
no | no | high
50 | Ribeiro et al. (2023) | 2023 | EMNLP | Style | no | encoder-decoder | pre-hoc, ante-hoc | supervised | yes |  Link | CNNDM | document | generic | no | news | no |  | ROUGE, BERTScore,
---
FRE, GFI, CLI
yes | yes | most readable and least readable | no | yes | high
51 | Nallapati et al. (2017) | 2017 | AAAI | Abstractivity, Saliency | yes | GRU-RNN | ante-hoc | supervised | yes |  Link | CNN/DM, DUC2002 | document | generic | no | news | no | ROUGE | no | no | no | no | no | high
52 | Hitomi et al. (2019) | 2019 | INLG | Length | no | encoder-decoder | pre-hoc, ante-hoc | supervised | no | - | JAMUS | document | generic | yes | news |  Link | ROUGE | no | no | no | no | no | medium
53 | Shen et al. (2022c) | 2022 | NIPS | Coverage | no | encoder-decoder | pre-hoc | supervised | yes |  Link | Multi-LexSum | document | Human-annotated | yes | Legal |  Link | ROUGE,BERTSCORE | no | yes | Rate the generation (0-3 scale) | no | yes | high
54 | Maddela et al. (2022) | 2022 | ACL | Entity | no | encoder, encoder-decoder | pre-hoc, ante-hoc | supervised | yes |  Link | CNNDM, NYT | document | generic | yes | news |  Link | ROUGE, BERTScore | no | no | no | no | no | high
55 | Bahrainian et al. (2022) | 2022 | ACL | Topic | no | encoder-decoder, decoder | - | supervised | no | - | NEWTS | document | Human-annotated | yes | news |  Link | ROUGE | no | yes | Binay evaluation | yes | no | high
56 | Hayashi et al. (2021) | 2021 | TACL | Topic | no | encoder, textRank | pre-hoc | supervised | yes |  Link | WikiAsp | Wikipedia | Automatic | yes | Wikipedia |  Link | ROUGE | no | yes | no | no | no | high
57 | Luo et al. (2022) | 2022 | EMNLP | Style | no | encoder-decoder | pre-hoc | supervised | no | - | TS and PLS | document | Automatic | yes | Scientific |  Link |  | ROUGE, FKG,
---
CLI,ARI, RNTPC
yes | yes |  | Relevance, Grammar,
---
Coherence
no | yes | high
58 | Ahuja et al. (2022) | 2022 | ACL | Topic | no | encoder, encoder-decoder | pre-hoc | supervised | yes |  Link | ASPECTNEWS | document | Automatic | yes | news |  Link | ROUGE | no | no | no | no | no | Medium
59 | Mukherjee et al. (2020) | 2020 | SIGIR | Topic | no | Graph-based | ante-hoc | Unsupervised | yes |  Link | Tourism Reviews | document | Automatic | yes | Tourism |  Link | ROUGE | no | yes |  | aspect-coverage, readability,
---
diversity
no | no | high
60 | Lin et al. (2021) | 2021 | EMNLP | Role | no | all | pre-hoc | supervised | yes |  Link | CSDS | reviews | Human-annotated | yes | reviews |  Link | ROUGE, BERTScore | no | yes |  | Informativeness,
---
non-redundancy,
fluency, matching rate
yes | no | high
61 | Shen et al. (2022b) | 2022 | ACL | Structure | no | encoder-decoder | pre-hoc | supervised | yes |  Link | MReD | reviews | Human-annotated | yes | Scientific |  Link | ROUGE | no | yes |  | Fluency, content relevance,
---
structure, decision correctness
no | no | high

Table 6: Survey paper checklist details (*marked fields are filled to best of
our understanding based on the available information).

This is a section in the appendix.