# A Survey of Controllable Text Generation using Transformer-based Pre-trained
Language Models

Hanqing Zhang<EMAIL_ADDRESS>Beijing Institute of
TechnologyBeijingChina100089 ,  HaoLin Song<EMAIL_ADDRESS>Beijing
Institute of TechnologyBeijingChina100089 ,  Shaoyu Li<EMAIL_ADDRESS>Beijing Institute of TechnologyBeijingChina100089 ,  Ming Zhou
<EMAIL_ADDRESS>Langboat TechnologyBeijingChina100089 and  Dawei Song
Beijing Institute of TechnologyBeijingChina100089

(2022)

###### Abstract.

Controllable Text Generation (CTG) is emerging area in the field of natural
language generation (NLG). It is regarded as crucial for the development of
advanced text generation technologies that are more natural and better meet
the specific constraints in practical applications. In recent years, methods
using large-scale pre-trained language models (PLMs), in particular the widely
used transformer-based PLMs, have become a new paradigm of NLG, allowing
generation of more diverse and fluent text. However, due to the lower level of
interpretability of deep neural networks, the controllability of these methods
need to be guaranteed. To this end, controllable text generation using
transformer-based PLMs has become a rapidly growing yet challenging new
research hotspot. A diverse range of approaches have emerged in the recent 3-4
years, targeting different CTG tasks which may require different types of
controlled constraints. In this paper, we present a systematic critical review
on the common tasks, main approaches and evaluation methods in this area.
Finally, we discuss the challenges that the field is facing, and put forward
various promising future directions. To the best of our knowledge, this is the
first survey paper to summarize CTG techniques from the perspective of
Transformer-based PLMs. We hope it can help researchers in related fields to
quickly track the academic frontier, providing them with a landscape of the
area and a roadmap for future research.

controllable text generation, pre-trained language models, transformer,
controllability, systematic review

††copyright: acmcopyright††journalyear: 2022††doi: xxx††journal:
JACM††journalvolume: 37††journalnumber: 4††article: 111††publicationmonth:
8††ccs: Computer systems organization Embedded systems††ccs: Computer systems
organization Redundancy††ccs: Computer systems organization Robotics††ccs:
Networks Network reliability

##  1\. Introduction

Natural language generation (NLG) is regarded as complementary to natural-
language understanding (NLU), an essential branch of natural language
processing (NLP). Contrary to the task of NLU, which aims to disambiguate an
input text to produce a single normalized representation of the idea expressed
in the text, NLG mainly focuses on transforming the potential representations
into specific, self-consistent natural language text (Horacek, 2001). In other
words, NLU aims to develop an intelligent machine that can read and understand
human language, while NLG enables computers to write like humans. As an
embodiment of advanced artificial intelligence, NLG technologies play a
crucial role in a range of applications, such as dialogue systems,
advertising, marketing, story generation and data augmentation.

Figure 1. Toy examples for controllable text generation. The left hand side
shows an application of story generation, which needs to ensure that the
generated story matches the key elements provided by the story line and the
order in which they appear. The right hand side shows an application of
dialogue text generation. One important controlled requirement is to avoid
generating toxic responses, such as the harmful introductory advice about
”smoking” and gibberish as shown in the figure.

Making text generation controllable is an important and fundamental issue in
NLG. Some concrete examples are shown in Figure 1. Generally speaking, a NLG
system should be able to reliably generate texts that meet certain
controllable constraints as humans wish. In general, these constraints are
task-specific. For example, the task of story generation always needs to
control the storyline and the ending. In the task of dialogue response
generation, controlling emotion (Li et al., 2020a), persona (Zhang et al.,
2018a), politeness, etc., is required. For generation-based data augmentation
(Grover et al., 2019), it is necessary to ensure the data distribution balance
in different domains. Moreover, for ethical development (Bender et al., 2021)
of AI applications, it is crucial to avoid generating mindless and offensive
content such as gender bias, racial discrimination, and toxic words.
Therefore, if the controllability of an NLG system can not be guaranteed, it
becomes hard to generate significant practical value in real applications.

In recent years, the development of deep learning (DL) has given rise to a
series of studies on DL-driven controllable text generation (CTG), which has
brought genuine breakthroughs in this field. Early approaches are based on
sequential models and style embedding (Ficler and Goldberg, 2017; Li et al.,
2016b), and achieved some promising progress. After that, there is a surge of
methods based on deep generative models, such as Variational Autoencoders
(VAEs) (Yang et al., 2018; Hu et al., 2017; Sohn et al., 2015; Vechtomova et
al., 2018; Xu et al., 2019; Wang et al., 2019), Generative Adversarial Nets
(GANs) (Wang and Wan, 2018; Scialom et al., 2020), and Energy-based Models
(Deng et al., 2020; Zhao et al., 2017; Bhattacharyya et al., 2021; Tu et al.,
2020). Deep-learning based methods are capable of an end-to-end learning in a
data-driven way to learn low-dimensional dense vectors that implicitly
represent the linguistic features of text. Such representation is also useful
to avoid the bias of hand-crafted features, and has shown great potential in
text generation.

However, the success of the above DL-based methods relies heavily on large-
scale datasets, posing a challenge for supervised and cross-domain text
generation tasks. Since 2018, large-scale pre-trained Language models (PLMs)
such as BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019b), GPT (Radford
et al., 2019), T5 (Raffel et al., 2020) and mBART (Liu et al., 2020a), have
gradually become a new paradigm of NLP. Owing to its use of large corpora and
unsupervised learning based on the Transformer structure, PLMs are believed to
have learned a great deal of semantic and syntactic knowledge from the data,
and only fine-tuning is required for downstream tasks to get the state-of-the-
art (SOTA) performance. In terms of NLG, PLMs have learned from a large number
of corpus materials to model the distribution of natural language to a large
extent, so that they are able to generate texts of unprecedented quality (Deng
et al., 2020). Moreover, a large-scale PLM itself can be viewed as a well-
informed knowledge base, making it possible to generate text with specific
constraints without the need for external domain knowledge. Nevertheless, PLMs
are neural network based, which essentially are still black boxes, lacking a
good level of interpretability. Those models always generate texts according
to the latent representation of the context, resulting in the difficulty of
controlling them to generate content as what the human want (i.e.,
controllability issues). How to improve the interpretability and
controllability of the PLM-based models for generating text has become a hot
research topic.

In the above application and research contexts, PLMs-based methods are
becoming the mainstream of controllable text generation (CTG) research and are
expected to bring milestone progress. As a rapidly growing yet challenging
research field, there is an urgent need for a comprehensive critical review of
the current literature to draw a landscape of the area and set out a roadmap
for promising future directions. There are some existing surveys on CTG
(Prabhumoye et al., 2020), but it lacks (1) a systematic review of
representative application tasks, main approaches, and evaluation
methodologies of CTG; (2) a tracking of the latest large-scale PLM-based CTG
approaches. In this paper, we provide an introduction to the main tasks and
evaluation metrics related to CTG, a dedicated and comprehensive literature
review on CTG approaches using PLMs, and finally, an outlook on the possible
future research directions. We hope that this survey paper will help the
researchers to quickly capture the overall picture as well as detailed
cutting-edge methods in PLM-based CTG, and promote the further development of
this promising area.

The remainder of the paper is organized as follows: Section 2 gives a brief
introduction to the two key aspects of the area, i.e., the concept of
controllable text generation and pre-trained language models. Then, we divide
the main approaches to PLM-based controllable text generation into three
categories and discuss them in more detail in Section 3. Section 4 summarizes
the relevant evaluation methodologies and metrics for CTG. In Section 5, we
discuss the challenges that the field is facing and put forward promising
future directions. Finally, we conclude the paper in Section 6. All the
literature appearing in this paper is filtered following two rules, one is
that we tend to select the latest papers that are published within 3-4 years,
and the second is to select papers that have been published in the leading and
influential conferences or journals in the NLP field such as ACL, EMNLP,
NAACL, TACL, etc.

##  2\. An Introduction to Controllable Text Generation and Pre-trained
Language Models

This paper is closely related to two key aspects: controllable text generation
and pre-trained language models, which will be briefly introduced in this
section.

###  2.1. Controllable Text Generation

Controllable text generation (CTG) refers to the task of generating text
according to the given controlled element (Prabhumoye et al., 2020). As shown
in Figure 2, a typical CTG system consists of three components: the controlled
element, including a controlled condition (e.g., a positive sentiment) and a
source text (which can be vacant or just a text prompt in some applications)
as input (I); the generative model (e.g., a PLM-based model) as the process
(P), and the generated text satisfying the input control condition, as output
(O). Take sentiment control as an example. If we want to generate a sentence
with positive emotion, then the condition ”positive sentiment” and
corresponding prompt ”I am always” are taken as the control element and input
to a PLM-based generative model. The output sentence’s sentiment disposition
would be what we want, such as ”I am always happy to see you”.

Figure 2. The IPO of controlled text generation. A typical CTG system consist
of three components: the controlled element (controlled condition and source
text) as input (I), the generative model as process (P), and the generated
text satisfying the input control condition as output (O).

Depending on different applications, the attributes of control conditions can
be in different forms and connotations. They could range from text attribution
(such as sentiment, topic, and keywords); author style and speaker identity of
the person writing the text (such as gender and age); text genre and formats
(such as poems, couplets); ordering of events (such as storylines); to
structured data description (such as table-to-text and Knowledge Graph(KG)-to-
text generation). All the above task types can be formalized mathematically in
a unified form as follows.

Given a vocabulary $\mathscr{V}$, the goal of CTG is to generate a target text
$Y=\\{y_{1},y_{2},\dots,y_{n}\\}$, where $y_{n}\in\mathscr{V}$, with respect
to a control element denoted as $C$. Then CTG can be formally described as:

(1) $P(Y|C)=p(y_{1},y_{2},\dots,y_{n}|C)$

The specific expression of $C$ may vary according to different tasks, and we
divide those control conditions into three categories (i.e., semantic,
structural, and lexical constraints); the detailed content can be seen in
Figure 3. As for the sentence $Y$ generated by the model, it is also expected
to satisfy the constraint conditions while conforming to the general natural
language characteristics, such as fluency, rationality, and readability, to
the greatest extent.

Figure 3. The taxonomy of control condition. We generally divide the control
conditions into three different types: (1) Semantic: it generally refers to
content control at the semantic level, which is the reverse process of some
text understanding tasks, such as emotion/topic classification and toxic
detection. (2)Structural: it is opposed to the semantic, represents control
over the structure level of the generated text, it is always related to the
text’s structure analysis tasks such as information extraction, sentence
parsing, etc. (3)Lexical: it represents a type of controllable text generation
task at the vocabulary level, such as making the generated text contain
certain keywords.

###  2.2. Relevant Tasks Involving CTG

We now introduce typical tasks related to controlled text generation.
Controllability is the fundamental problem of text generation, which is indeed
required by almost all text generation tasks. Here, we only focus on tasks
with explicitly controlled conditions and goals. Table 1 summarizes some
typical tasks involving CTG, with a description of the input/output,
controlled aspects, and representative references for each task. They are
explained in more detail below:

  * •

Attribute-based Generation: Attribute-based CTG aims to generate natural
language sentences that satisfy specific attributes such as topic, emotion,
and keywords. Precisely controlling the various attributes of sentences is an
essential requirement of intelligent writing. By combining multiple control
attributes, the system can, in theory, create interpretable and controllable
paragraphs or articles. Thus, attribute-controlled text generation has always
been the focus of attention in the field of text generation.

  * •

Dialogue Generation: The goal of dialogue systems is to build an agent who can
mimic human conversations using natural language. Generative dialogue models
always have a higher requirement in consistency, semantics, and
interactiveness (Huang et al., 2020b). Therefore, the constraint on emotion,
the speaker’s personal style, dialogue intent/action, etc., are used to
control the dialogue response and improve the interactivity of dialogue
systems.

  * •

Storytelling: Storytelling requires the model to generate texts with complete
narrative logic, which needs a higher level of control on long text
generation. Storylines and story endings are often regarded as controlled
conditions, and the model needs to produce stories with fluent text and sound
plots according to the given controlled conditions.

  * •

Data to Text: The main goal of data-to-text generation is to convert non-
linguistic structured data (e.g., a table or a graph) into natural language
text, which can be applied in tasks like weather forecast, healthcare (Castro
Ferreira et al., 2019), and so on. The controllability of data-to-text tasks
is to ensure that the generated text contains information manifested in the
original table or graph structure data.

  * •

Data Augmentation: Neural networks heavily rely on a large amount of labeled
data. Nowadays, the importance of data augmentation is becoming more and more
conspicuous, as a result of the significant cost of data collection and
cleaning. Since recent neural network models are capable of generating near-
realistic text, it is possible to utilize them to expand existing datasets and
even create new data. Identifying and replacing some entities in the given
text or generating new sentences according to given attributes through CTG has
become an efficient way for data augmentation.

  * •

Debiasing: Biased training data may cause a model to learn incorrect knowledge
and thus output the results that are biased. Therefore, text debiasing has
attracted increasing attention. Rewriting biased text to unbiased text or
changing the data distribution of the CTG-generated text has been shown
feasible. The major controllable aspects in this task include gender, race,
and toxicity.

  * •

Format Control: There are also text generation tasks that need to control the
format of the generated text, such as text length and rhythm. For example,
traditional Chinese poetry and couplet generation has strict requirements in
format, including the number of words, structure, etc.

Table 1. A general overview of the tasks involving CTG. We enumerate 7
categories of tasks involving CTG, and briefly describe the input and output
of each task, and list relevant representative works based on controllable
aspects.

Task | Input & Output | Controllable Aspects
---|---|---
Attribute-based Generation |  | Input: Keywords, discrete attributes
---
Output: Attributes-specific sentence
| Topic (Wang et al., 2019; Tang et al., 2019; Khalifa et al., 2021; Dathathri
et al., 2020),
---
tense (Logeswaran et al., 2018),politeness (Sennrich et al., 2016; Saha et
al., 2022),
sentiment (Khalifa et al., 2021; Dathathri et al., 2020; Logeswaran et al.,
2018)
(Zhang et al., 2019b; Chen et al., 2019b; Samanta et al., 2020; Zhang and
song, 2022),
keywords (Wang et al., 2021b; Zhang et al., 2020d; He, 2021; Carlsson et al.,
2022a)
Dialogue Generation |  | Input: dialogue content, additional structural
---
information(eg: persona, emotion, intent, template, etc.)
Output: Dialogue Response
| Persona (Mazaré et al., 2018; Wolf et al., 2019; Zhang et al., 2018b; Song
et al., 2020b)
---
(Zhong et al., 2020; Song et al., 2021; Zheng et al., 2020; Zeng and Nie,
2021),
Politeness(Niu and Bansal, 2018; Firdaus et al., 2022b),
Sentiment (Zhang et al., 2017; Zhou et al., 2018; Liu et al., 2019a; Lubis et
al., 2018)
(Ruan and Ling, 2021; Song et al., 2019; Wei et al., 2019; Firdaus et al.,
2022a),
Template(Kale and Rastogi, 2020a; Wang et al., 2021a),
Ground-truth (Ghazvininejad et al., 2018; Dinan et al., 2019; Qin et al.,
2019)
(Wu et al., 2021; Peng et al., 2020; Rashkin et al., 2021)
Storytelling |  | Input: Story elements
---
Output: Story paragraph
| Story structure (Fan et al., 2018; Goldfarb-Tarrant et al., 2020; Fang et
al., 2021),
---
story ending (Peng et al., 2018; Tambwekar et al., 2019; Luo et al., 2019),
topic (Feng et al., 2018; Wang et al., 2020; Yang et al., 2019b)
(Lin and Riedl, 2021; Chang et al., 2021),
persona (Chandu et al., 2019; Liu et al., 2020b)
Data to Text |  | Input: Table/graph data
---
Output: The texts describing the data information
| Structural information (Puduppully et al., 2019; Song et al., 2020a; Ribeiro
et al., 2021a)
---
(Zhao et al., 2020; Ribeiro et al., 2021b; Su et al., 2021)
Data Augmentation |  | Input: Original text, predefined slot values
---
Output: Text that specific features are replaced
Pre-defined slot values (Malandrakis et al., 2019; Amin-Nejad et al., 2020;
Liu et al., 2020c)
Debiasing |  | Input: Biased text/biased model
---
Output: Unbiased text/unbiased model
| Predefined bias types(eg: political bias (Liu et al., 2021a)
---
gender bias (Dinan et al., 2020; Qian et al., 2019),
subjective bias (Pryzant et al., 2020), social bias (Sheng et al., 2020;
Barikeri et al., 2021),
sentiment bias (Huang et al., 2020a)), toxicity (Krause et al., 2021; Liu et
al., 2021c))
Format Control |  | Input: Desired Format, prompt text
---
Output: Text in predefined formation
| Format (Liao et al., 2019; Li et al., 2020b; Sheng et al., 2021; Shao et
al., 2021)
---

###  2.3. Transformer-based Pre-trained Language Models

Table 2. An overview of the characteristics of typical PLMs. ”MLM” means ”Mask
Language Model”; ”NSP” means ”Next Sentence Prediction”; ”SLM” means ”Standard
Language Mode”; ”CTR” means ”Corrupted Text Reconstruction”; ”FTR” means ”Full
Text Reconstruction”; ”PLM” means ”Permutation Language Modeling”; and ”TLM”
means ”Translation Language Modeling”. The details of pre-trained task’s
definition could be seen in literatrue (Liu et al., 2021d).

Name | Model Type | Infrastructures | Pre-trained Task | Main Application
---|---|---|---|---
BERT (Devlin et al., 2019) | AE | Encoder | MLM+NSP | NLU
XLNET (Yang et al., 2019a) | AR | Transformer-XL | PLM | NLU
GPT2 (Radford et al., 2019),GPT3 (Brown et al., 2020) | AR | Decoder | SLM | NLG
T5 (Raffel et al., 2020) | Seq2Seq | Encoder+ Decoder | CTR | NLG+NLU
mBART (Liu et al., 2020a) | Seq2Seq | Encoder+ Decoder | FTR | NLG
UniLM (Dong et al., 2019) |  | AE+AR+Seq2Seq
---
Encoder+Decoder | SLM+CTR+NSP | NLG+NLU
ERNIE-T (Zhang et al., 2019a) | AE | Encoder | CTR+NSP | NLU
XLM(Conneau and Lample, 2019) | Seq2Seq | Encoder+ Decoder | TLM | NLG
CPM(Zhang et al., 2020a) | AR | Decoder | SLM | NLG

Recent years have witnessed the emergence and successful applications of a
large number of pre-trained language models. They are regarded as a
revolutionary breakthrough in deep learning and NLP. During the pre-training
stage, the use of large-scale unlabeled data can provide strong support for an
increased model scale, and make the system better grasp the diverse knowledge
(e.g., linguistic knowledge, commonsense, facts, expertise, etc.) in the data.
State-of-the-art (SOTA) performance has been achieved in downstream tasks by
fine-tuning the PLMs based on only a small amount of supervised data.

The early work related to the idea of the pre-trained language model can be
traced back to NNLM (Bengio et al., 2003),word2Vector (Mikolov et al., 2013),
and ELMo (Peters et al., 2018). More recently, the pre-trained model based on
Transformer (Vaswani et al., 2017) has greatly improved the performance in
various NLP tasks and become mainstream. Thus this paper is focused on
Transformer-based PLMs. The representative PLM infrastructures mainly include
Transformer (Vaswani et al., 2017) or its variants such as Transformer-XL (Dai
et al., 2019), Longformer (Beltagy et al., 2020), and Reformer (Kitaev et al.,
2020), etc. The objectives of model learning mainly include masked language
modeling (MLM) and corrupted text reconstruction (CTR), etc. In order for a
good understanding of them, we give a summary of representative PLMs in Table
2 according to their different data construction modes, model infrastructures,
and pre-trained tasks, etc. Here, we roughly divide the existing PLMs into the
following three categories and provide a brief introduction to each of them.

Auto-Encoding (AE) Models: This type of PLMs is constructed based on
destroying the input text in some way, such as masking some words of a
sentence and then trying to reconstruct the original text. Typical examples of
this type include BERT, ROBERTA, and ERNIE. Because these models aim to build
bidirectional encoding representations of the entire sentences, their
infrastructures often correspond to the encoder part of the Transformer, which
does not require any masking mechanism, and all input can be accessed at each
location. They can then be fine-tuned in downstream tasks and have achieved
excellent results. The natural applications are sentence classification and
sequence labeling, etc., which are more inclined to natural language
understanding (NLU) tasks.

Auto-Regressive (AR) Models: The same as the classical language modeling
approach, the main task of AR models is to predict the next word based on what
has been read in the text. A representative of this type of model is the GPT
family. Unlike the aforementioned AE language models, the infrastructures of
AR are composed of the Transformer’s decoder part, and they use a masking
mechanism in the training phase so that the attention calculations can only
see the content before a word, but not the content after it. While it is
possible to fine-tune such a PLM and achieve excellent results on many
downstream tasks, it is most natural application is NLG tasks.

Seq2seq Models: The seq2seq models use both the encoder and decoder of the
Transformer for better model flexibility. Currently, the most representative
models of this type include T5(Raffel et al., 2020) and mBART(Liu et al.,
2020a). In principle, almost all pre-trained tasks used in AE and AR models
can be adapted to the seq2seq models. Relevant research (Raffel et al., 2020)
has found that seq2seq models can achieve better performance. Moreover, a
seq2seq model unifies the NLU and NLG tasks so that they can be solved under
the same framework. It can be fine-tuned on a variety of NLG tasks such as
translation and summarization, as well as NLU tasks that can be converted into
a text2text form (Raffel et al., 2020), including sentence classification,
semantic similarity matching, etc.

When it comes to controlled text generation using PLMs, most of the methods
exploit the generative model, including AR and Seq2seq models as a basis, and
then guide them to generate the desired text. Generally, CTG tasks always
treat the PLMs as a conditional generation model, and its formulation is
consistent with the standard language model:

(2) $P(x_{n}|X_{1:n-1})=p(x_{n}|x_{1},x_{2},\dots,x_{n-1}).$

Based on the pre-trained language model manifested above, the goal of
conditional text generation can be formulated as:

(3) $P(X|C)=\displaystyle\prod_{i=1}^{n}p(x_{i}|x_{<i},C),$

where $C$ denotes the controlled conditions, which will be integrated into the
PLM in a specific form, and $X$ is the generated text that incorporates the
knowledge encoded in PLM and complies with the control conditions.

In the next section, we will review the main approaches to CTG using
Transformer-based PLMs.

##  3\. Main Approaches to PLM-based CTG

From a generative point of view, PLMs have learned a variety of knowledge from
a large-scale corpus that can help can produce a more fluent and richer
variety of text. This provides an effective way for natural language
generation. However, the existing PLMs are essentially still black-box models
like other deep neural networks, making it difficult to interpret the
generated text and lacking controllability. How to make good use of PLMs in
text generation while realizing the controllability of the generative model
has recently become a hot research topic. In this section, we provide a
comprehensive review of the main approaches in this area from the unified view
of how to use the Transformer-based PLM.

Figure 4. The Overview of the approaches of CTG based on PLM. According to the
way how the control signal work with the pre-trained language model, we have
roughly divided the existing methods into three categories, each of which is
further divided into several subclasses.

###  3.1. Overview

The core idea of PLM-based CTG is to give the model a control signal in an
explicit or implicit way to drive the generation of text satisfying the
control conditions. According to the way how the control signal works, we have
roughly divided the existing methods into three categories, each of which is
further divided into several subclasses. An overview is given in Figure 4. The
most direct way is to fine-tune the PLMs, which can perform the controllable
generation task at a lower cost. The second way is to retrain or refactor the
PLMs for controlled text generation. This method could produce better results
in principle but may consume a lot of computing resources and also face the
problem of lacking labeled data. As the parameter size of PLMs increase
rapidly, even fine-tuning has become resource-intensive. To tackle the
problems, text generation methods that work on the decoder, named post-
processing, have emerged, where PLMs are always fixed, and the control signal
works on the decoding-stage. Such methods not only require fewer computation
resources for training, but also can guarantee a better quality of the
generated text to some extent. As a consequence, increasing attention from the
academic community has been paid to this direction in recent years. In the
following sections, we will review the related literature that appeared in
recent years and discuss these three types of methods in more detail.

###  3.2. Fine-tuning

This type of method aims to fine-tune part or all of the PLMs’ parameters to
produce text that satisfies the specific constraints. As discussed in Section
2.3, ”PLM + fine-tuning” has become a new paradigm in the general field of
NLP. First, a large amount of training data (usually unlabelled samples) and
model parameters are used to learn general knowledge from the text into a PLM.
Then a domain/task-adapted model will be obtained to achieve competitive
performance by fine-tuning the PLM based on a small amount of labeled data for
the specific downstream task.

This paradigm is also applicable to controllable text generation, and a large
number of related studies have been carried out. Recent work has found that
fine-tuning PLMs on target data such as AMR-to-text (Nan et al., 2021; Ribeiro
et al., 2021a; Kale and Rastogi, 2020b) for dialogue generation can establish
a new level of performance. While the conventional fine-tuning method is
relatively concise and easy to understand, we focus on more advanced methods
below.

Adapted Module: This method constructs a task-related adapted network module
around a PLM, and then it is trained with the PLM on the target dataset just
like usual fine-tuning. Auxiliary Tuning (Zeldes et al., 2020) introduces an
extra condition modeling module based on the original PLM, which takes
$X(x_{n};C)$, the concatenation of control condition $C$ and training text
$x_{n}$, as input and outputs logits in the vocabulary space. The auxiliary
model is trained by adding its logits to the PLM’s logits and maximizing the
likelihood of the target task output. (Ribeiro et al., 2021b) add an adapter
module after the feed-forward sub-layer of each layer on both the encoder and
decoder of the PLM, which can encode the graph structure into the PLMs without
contaminating its original distributional knowledge. During the training
stage, the PLM’s parameters are frozen, and only the injected adapter is
trainable. Avoiding catastrophic forgetting while maintaining the topological
structure of the graph, the model achieves the SOTA performance on two AMR-to-
text benchmarks. On the controlled dialogue generation task, the idea of an
adapted module is also applied. Lin et al. propose an adapter-bot for dialogue
generation. The model builds a series of lightweight adapters on top of a PLM
for dialogue generation, namely, DialGPT (Zhang et al., 2020c). The model
allows for a high-level control and continuous integration of various control
conditions for different conversational requirements (e.g., emotions,
personas, text styles, etc.).

In summary, the adaptive modules essentially aim to bridge the gap between the
controlled attributes and the PLMs, while guiding the language model to
generate text that meets the corresponding control conditions.

Figure 5. An illustration of prompt-learning. The top one shows an example of
sentiment classification based on prompt learning, while the bottom one shows
an example of sentiment-controlled text generation. The text with red color is
the templates, which could be either manually constructed by humans, or
automated search for discrete/continuous tokens.

Prompt: A more effective way for the use PLMs is to keep the fine-tuning
phase’s training objective consistent with the original task from where the
PLMs are derived (Gao et al., 2021). This idea gives rise to the so-called
prompt-based approaches. Take a sentiment classification task for example.
Suppose we need to recognize the sentiment of a sentence, e.g., ”I am always
happy to see you”. Different from the traditional approaches that encode the
sentence into a set of vectors and then classify their sentiment through a
fully connected layer, the prompt-based method will construct a set of
templates, for example: (”I am always happy to see you, the sentiment of the
sentence is [MASK]”), and then ask the model to predict the token [mask]
according to the original training task for the PLM. On the side of the
sentiment-control text generation, the template could be regarded as the
control prompts to instruct the PLM to generate desired texts. A specific
illustration of prompt learning can be seen in Figure5. This approach has gone
through various stages, from manual template construction (Jiang et al.,
2020), to automated search for discrete tokens (Shin et al., 2020), to
continuous virtual Tokon representations (Lester et al., 2021; Li and Liang,
2021). Since those methods keep most parameters of the PLM fixed in most
cases, remaining generative capacity of the original PLMs, thereby it has
achieved great success in few-shot scenarios.

From the CTG point of view, the prompt-based approach still applies. Li and
Liang (Li and Liang, 2021) propose a method named “prefix tuning”, which
freezes the PLM’s parameters and back-propagates the error to optimize a small
continuous task-specific vector called ”prefix”. The learned prefix, also
called ”prompt”, can guide the PLM to generate the required text, thus
enhancing the controllability to a certain extent. The approach achieves
impressive results on some generative tasks such as data-to-text. An extension
of the model, namely P-tuning (Lester et al., 2021), serves a similar purpose.
Different from prefix-tuning (Li and Liang, 2021), p-tuning does not place a
prompt with the ”prefix” in the input, but constructs a suitable template to
prompt the PLM, and the template is composed of the continuous virtual token,
which is obtained through gradient descent. Based on prefix-tuning, Qian et
al. (2022) leverage contrastive learning to train attribute-specific vectors.
Different from vanilla prefix tuning, where each prefix is trained
independently under the attribute-specific corpus, they take the relationship
among attribute prefixes into consideration and train multiple prefixes
simultaneously, thereby boosting the control performance. DisCup (Zhang and
song, 2022) provides a new prompt-based alternative for attribute-controllable
generation, which uses the attribute-discriminator to assist prompt-tuning.
This allows the learned control-prompt could absorb the information of inter-
attribute knowledge, achieving the state-of-the-art attribute control
performance so far. In view of the promising effect and the parameter-
efficient structure features of the prompt-tuning in CTG, some researchers
have started to propose prompt-based multi-attribute control approaches.
Tailor (Yang et al., 2022) explores two strategies, including a non-training
method based on the concatenation of single-attribute prompts and a training
method using attribute-connector, to achieve the pure prompt-based multi-
attribute CTG task.

In order to relieve the problems that the distance from the prompt and the
next predicted token correlates negatively with the prompt’s influence, Zou et
al. (2021) propose a method called Inverse Prompt, the main idea of which is
to use generated text candidates of the PLM to inversely predict the prompt
(topic, Poetry name, etc.) during beam search so as to enhance the relevance
between the prompt and the generated text and provide better controllability.
However, the generation process requires the reverse prediction for each
candidate token, leading to an increased computation cost. According to our
actual tests based on the provided source code, it takes up to around 10
minutes to generate a seven-word rhyming poem, making it difficult to be
applied in real application scenarios.

More recently, in order to address the challenge of the fine-grained
controllable text generation for the PLM, an encoder-decoder architecture
based on a pair of GPT2 models is introduced (Carlsson et al., 2022b), namely
non-residual prompting, which enables intermediate text prompts at arbitrary
time steps of the generative PLM. Specifically, the proposed method uses an
auxiliary prompt encoder, composed of a trainable GPT2, to guide another
generative language models towards certain constraints, including themes,
sentiment, keywords, etc. The generative language model (i.e., GPT2 model) is
always fixed during generation, and the different prompt instructions can be
used at different time steps, enabling fine-grained controllable text
generation. Non-residual prompt trends to be versatile and shows promise
towards the unified controllable text generation. However, we hold the
critical opinion that it lacks the systematic comparison and analysis between
the PLM with natural encoder-encoder architecture such as T5 (Raffel et al.,
2020) and Bart(Liu et al., 2020a), and the training process is complex, the
CTG model is not parameter-efficient.

To summarize, most of the prompt-based methods show a certain degree of
versatility. From the CTG perspective, this kind of method essentially uses
the characteristics of PLM in its pre-training stage to guide the PLM to
generate restricted text by selecting an appropriate prompt in the fine-tuning
stage, so as to achieve the purpose of controllability.

Reinforcement Learning (RL) inspired Approaches: The core motivation of this
type of method is to feed back whether or how the control conditions are
achieved as a reward to the fine-tuning of the PLM. Ziegler et al. (2019) use
reinforcement learning to fine-tune the PLMs, with a reward model trained from
human preferences. First, it initializes a policy $\pi=\rho$, where $\rho$
denotes a PLM such as GPT2. Given a dataset $\mathcal{D}\in(X,Y)$, the goal is
to fine-tune $\pi$ so that it can approximate the distribution of the data
$\mathcal{D}$. This is done using RL by optimizing the expectation of the
reward:

(4) $\mathbb{E}_{\pi}[r]=\mathbb{E}_{x\sim\mathcal{D},y\sim\pi(\cdot\mid
x)}[r(\pi(x),y)].$

Then the reward model $r$ is trained based on the sample
$(x,y_{0},y_{1},y_{2},y_{3})$ via $x\in\mathcal{D}$, and $y_{i}$ is generated
from $p(y_{i}|x)$. Human labelers are required to choose the human preferred
sentence from $(y_{0},y_{1},y_{2},y_{3})$. To prevent $\pi$ from moving too
far away from the original PLM $p$ for ensuring the fluency of the generated
text to the greatest extent, a penalty item is added to the reward function
during the actual process of fine-tuning $\pi$:

(5) $R(x,y)=r(x,y)-\beta KL(\pi,p),$

where $R(x,y)$ is the re-defined reward function, $\beta$ is the regular
coefficient, and $KL(\pi,p)$ aims to ensure that the two distributions are as
close as possible.

Liu et al. (2020c) propose a data augmentation approach, which uses
reinforcement learning to guide the GPT2 model to generate texts towards a
specified conditional direction (i.e., target class). Specifically, an
additional RL stage is added between the softmax and argmax functions of GPT2,
and then the parameter of the PLM’s hidden states $\theta$ is updated towards
the target label according to the signal of the RL reward. The generated texts
are regarded as augmentation data to help improve the classification
performance. Moreover, Stiennon et al. (2020) use an RL-based approach on the
task of English summarization, which fine-tunes the PLM by combining with
human feedbacks.

The reinforcement learning is also applied for controllable story generation.
Tambwekar et al. (2019) designs a reward-shaping technique that produces
intermediate rewards at all different time-steps, which are then back-
propagated into a language model in order to guide the generation of plot
points towards a given goal. It should be noted that the above work is carried
out on a language model based on LSTM, but its principles are applicable to
the subject described in this article, so that we have included it here.

In summary, the idea of applying reinforcement learning to PLM-based
controlled text generation is natural. The central challenge is to ensure that
the PLM is optimized towards the RL’s rewards while maintaining the fluency of
generated text. To address this challenge, the key is to achieve a better
balance between these two aspects.

###  3.3. Retraining/Refactoring

According to the characteristics of a specific downstream task, it is also
feasible to change the original architecture of PLMs or retrain a large
conditional language model from scratch. This kind of approach is promising to
substantially improve the quality and controllability of text generation, but
is limited by the insufficient labeled data and large computing resource
consumption.

CTRL (Keskar et al., 2019) is an early attempt in this direction. It trains a
language model conditioned on a variety of control codes. The network model
used in this approach is also a commonly used Transformer, and a piece of
control code (domain, style, topics, dates, entities, relationships between
entities, etc.) is added in front of the text corpus. That is, it lets the
original language model $p\left(x_{i}\mid x_{<i}\right)$ transform into
$p\left(x_{i}\mid x_{<i},c\right)$. A language model with 1.63 billion
parameters is retrained on a 140Gb corpus. Another contribution of this work
is to propose a new top-k sampling algorithm:

(6) $p_{i}=\frac{\exp\left(x_{i}/(T\cdot I(i\in
g))\right.}{\sum_{j}\exp\left(x_{j}/(T\cdot I(j\in g))\right.}\quad
I(c)=\theta\text{ if }\mathrm{c}\text{ is True else 1},$

where $g$ is a list of generated tokens, $p_{i}$ is the probability
distribution for the next token, and the introduction of $I(c)$ reduces the
probability of words that have already appeared.

Zhang et al. (2020d) propose POINTER, an insertion-based method for hard-
constrained (keep the specific words that appeared in generated text) text
generation. Different from the auto-regressive method, such as GPT2, this
method modifies the structure of the Transformer so that it can generate text
in a progressive manner. Specifically, given certain lexical constraints,
POINTER first generates the constrained words to satisfy the control
condition, then more detailed words are inserted at a finer granularity
between those words. The above process iterates until the entire sentence is
completed. This kind of method can ensure the generated sentences meet the
lexical constraints. However, the model needs to be trained from scratch on
the large-scale corpus, and the fluency of the generated sentences is not as
good as the auto-regressive model in most cases.

Similar to the idea of insertion-based method, like POINTER (Zhang et al.,
2020d), a lexically constrained text generation framework called Constrained
BART (CBART) is proposed (He, 2021). This approach also adopts the way of
progressive insertion/replacement for text generation, yet without the
modification of the Transformer’s architecture. Concretely, based on the pre-
trained model BART, it divides the generation process into two steps. First, a
token-level classifier is added on BART’s encoder to predict where to replace
and insert. Then, the predicted results are regarded as signals to guide the
decoder to refine multiple tokens of the input in one step by inserting or
replacing tokens before specific positions. Different from the normal way of
generating texts step by step, the decoder predicts all tokens in parallel so
as to accelerate the inference. Although CBART does not need to reconstruct
the architecture of PLMs, the training and inference processes are different
from the original pre-training tasks, which can lead to a negative impact on
the quality of text generation.

CoCon (Content-Conditioner) (Chan et al., 2021) introduces a conditional
control module in addition to the original pre-trained language model, which
can realize the precise control of the generated text at the word- and phrase-
levels. In terms of model architecture, the approach injects a control block
into the GPT model and provides the control code as a separate input. In order
to tackle the problem of lacking labeled data, it adopts self-supervised
learning and constructs four different loss functions, including Self
Reconstruction Loss, Null Content Loss, Cycle Reconstruction Loss, and
Adversarial Loss. The core of these self-supervised losses is to use one part
in a piece of text as a control condition, leaving the rest for the model to
refactor, so that the model can learn to generate specific text conditioned on
the control code. The experimental results show that CoCon can incorporate
content inputs into the generated texts and control the high-level text
attributes in a more flexible way. Similar to CoCon’s idea of injecting an
additional controlled module to an existing PLM, Wang et al. propose a Mention
Flags (MF) module, which is injected into the decoder of Transformer, to
achieve a higher level of constraint satisfaction. The MF is designed to trace
whether a lexical constraint has been realized in the decoder’s output, and it
is formally represented as mentioned status embeddings injected into the
Transformer’s decoder, to provide a signal to encourage the generative model
to satisfy all constraints before generation.

This type of approach is also used for the task of controlled dialogue
generation. Zheng et al. propose a pre-training-based method to build a
personalized dialogue agent. The whole framework is in an encoder-encoder
(Transformer) fashion, and its initial parameters inherit from an existing PLM
model. The personalized information is represented as attribute embeddings,
which are added into the encoder to capture rich persona-related features when
modeling dialogue histories. Further, an attention routing network is added to
the decoder to incorporate the target persona in the decoding process while
maintaining the trade-off of the historical dialogue information dynamically.
To solve the problem of the lack of labeled data in conditional dialogue
generation, a multi-task learning framework is proposed (Zeng and Nie, 2021),
which utilizes both conditional labeled dialogue data and non-dialogue text
data. Based on a condition-aware transformer block (reconstructed from the
original Transformer), three sub-tasks are designed based on the existing PLM,
namely, conditional text generation based on labeled dialogue data,
conditional conversation encoder, and conditional dialogue generation task
based on non-dialogue text to optimize the model simultaneously. Persona and
topic-controlled experiments are conducted under the scenario of dialogue
generation, and the results show that this approach has achieved state-of-the-
art performance so far.

In summary, this type of approach refactor or retrain the PLMs to achieve
controlled text generation. They are more flexible and convenient to use, but
may make the PLM lose its versatility to some extent. As for the methods that
need retraining, they may face the dual challenges of increased computation
cost and the lack of large-scale labeled data.

###  3.4. Post-Processing

When the number of parameters of a PLM increases, the model has memorized more
and more knowledge and patterns, allowing it to achieve competitive results
even without fine-tuning in many NLP tasks (Brown et al., 2020). In the realm
of controlled text generation, the idea of fixing the PLM’s parameters first
and re-ranking the generated text in a post-processing manner becomes
achievable and promising. As shown in Figure 6, we could use a post-process
module to achieve sentiment-control text generation.

Figure 6. The scheme of the post-process approaches for CTG. During the
generation, the parameters of the PLM are fixed. Given the input prompts ”I am
always”, the PLM will produce the original next-token distribution; the post-
process module aims to re-rank the token distribution, allowing the PLM to
choose the sentiment-desired tokens (i.e., significantly increasing the
probability of the ”happy”), so as to achieve the positive sentiment control
text generation.

The most realistic idea about the post-processing method is to use some
standard decoding algorithms in text generation, e.g., the Greedy search,
constraint beam search (Anderson et al., 2017), Top-k sampling (Fan et al.,
2018), Nucleus sample (Holtzman et al., 2020), etc. The approaches discussed
below can be seen as an extension of them for CTG tasks. They are grouped into
two categories: guided strategies and trainable strategies.

Guided Strategies: This type of method decouple the PLMs for text generation
and the post-processing module, and the post-processing module guides the PLM
to generate conditioned text only in the inference stage.

A representative method of this type is PPLM (Dathathri et al., 2020). It
first trains an attribute discriminant model and then uses it to guide PLM to
generate the corresponding text. In this work, the attribute model is a simple
classifier, consisting of a user-specified bag of words or a single learning
layer whose parameters are 100,000 times less than PLM. During the text
sampling process, it requires a forward and backward process in which the
gradient from the attribute model drives the hidden activation of the PLM to
guide the target text generation. PPLM does not need to change the structure
or retrain the PLM, and it is able to achieve a significant improvement in
attribute alignment. However, it has a slight deficiency in text fluency
measured with the metric of PPL (Perplexity (Liu et al., 2021c; Krause et al.,
2021)).

MEGATRON-CNTR (Xu et al., 2020) is a controllable story generation framework
that combines external knowledge and PLM. Given a story context, a predictor
is used to get a set of keywords for the next sentence. Then, a knowledge
retriever is introduced to get external knowledge-enhanced sentences from an
external knowledge base according to the keywords. Next, a ranker is trained
to choose the most relevant knowledge-enhanced sentences, which are later fed
into the generator of PLM (GPT2) with the story context to get the next
sentence. The entire process is repeated. Human evaluation results show that
up to 91.5% of the generated stories are successfully controlled by the new
keywords. In this framework, GPT2 is independent of the other modules and
generates context-relevant sentences using introductory text provided by
MEGATRON-CNTR’s component as input, without any adaptation in training.

Hua and Wang (2020) propose a content-controlled text generation framework,
namely FAIR. It uses the BERT (Devlin et al., 2019) model to automatically
construct a content plan, including keyword assignments and their
corresponding sentence-level positions. After that, the BART (Liu et al.,
2020a) without structure modification is applied to fill the masked tokens
appearing in the generated text template. Finally, an iterative refinement
algorithm that works within the sequence-to-sequence (seq2seq) models is
designed to improve generation quality with flexible editing. The reported
experimental results show that FAIR can significantly improve the relevance
and coherence between the key phrases and the generated texts.

Additionally, a series of discriminator-guided ways have been developed, which
train an attribute discriminator to help PLM select text for the specific
attributes when decoding. Adversarial Search (Scialom et al., 2020), inspired
by GAN (Generative Adversarial Network), trains the discriminator to
distinguish human-created text from the machine-generated text. The
discriminator predicts a label for each token instead of for the entire
sequence. Its logit probability is added to the score to guide sampling toward
the human-written style. For the tokenizer with 10 thousand words, decoding
using a discriminator to classify each token is time-consuming. Aimed at
solving this problem, GeDi (Krause et al., 2021) trains a small class-
conditional language model (CC-LM) as generative discriminators to guide the
generation from large PLM (GPT2, GPT3). Specifically, a CC-LM is trained and
formulated as the following equation:

(7) $P_{\theta}\left(x_{1:T}\mid
c\right)=\prod_{t=1}^{T}P_{\theta}\left(x_{t}\mid x_{<t},c\right),$

where $c$ is the control code, $T$ is the length of generated text. GeDi
assumes that there is a CC-LM with the desired control code $c$ and an
undesired or anti-control code $\bar{c}$, and then uses the contrast between
$P_{\theta}\left(x_{1:T}\mid c\right)$ and
$P_{\theta}\left(x_{1:T}\mid\bar{c}\right)$ to guide sampling from an original
PLM. A contrast mechanism is designed to compute the probability that every
candidate token $x_{t}$ belongs to the desired class, given by
$P_{\theta}(C\mid x_{t},x<t)$:

(8) $P_{\theta}\left(c\mid
x_{1:t}\right)=\frac{P(c)\prod_{j=1}^{t}P_{\theta}\left(x_{j}\mid
x_{<j},c\right)}{\sum_{c^{\prime}\in\\{c,\bar{c}\\}}\prod_{j=1}^{t}P\left(c^{\prime}\right)P_{\theta}\left(x_{j}\mid
x_{<j},c^{\prime}\right)},$

where $P(c)$ and $P(c^{\prime})$ are biased parameters which could be learnt
or set manually as a hyper-parameter. During the generation step for token
$x_{t}$, Equation 8 is mulplied with the conditional probability
$P_{LM}\left(x_{t}\mid x_{<t}\right)$ of the original PLM via the Bayes rule:

(9) $P_{w}\left(x_{t}\mid x_{<t},c\right)\propto P_{LM}\left(x_{t}\mid
x_{<t}\right)P_{\theta}\left(c\mid x_{t},x_{<t}\right),$

where $P_{w}\left(x_{t}\mid x_{<t},c\right)$ is regarded as the final
probability for text generation. Since the calculation of the Equation 8 only
needs two parallel forward passes of CC-LM, the generation efficiency is
greatly improved.

Inspired by the GEDI (Krause et al., 2021), a batch of similar work has
emerged. DEXPERTS (Liu et al., 2021c) re-ranks the predictions of the PLM
based on expert (and anti-expert) opinions during the decoding stage to steer
the language model towards the desired generation. FUDGE (Yang and Klein,
2021) learns an attribute predictor operating on a partial sequence to adjust
the original PLM’s probabilities, and obtain an improved performance on the
tasks of couplet completion in poetry, topic control in language generation,
and formality change in machine translation. Plug-and-Blend (Lin and Riedl,
2021) extends the GEDI model to controlled story generation by introducing a
planner module.

More recently, Pascual et al. (2021) propose a simple yet efficient plug-and-
play decoding method, namely K2T, which even does not need a discriminator.
Specifically, given a topic or keyword which is considered a hard constraint,
K2T adds a shift to the probability distribution over the vocabulary towards
the words which are semantically similar to the target constraint word. The
shift is calculated based on word embedding. Although the K2T is intuitive,
the shift added to the probability distribution of vocabulary may be too rough
and cause the generated texts to fall short in fluency.

Mireshghallah et al. (2022) propose a scored-based controllable text
generation framework, namely Mix and Match, which regards the task of
generating attribute-specific text as a Metropolis-Hastings sampling process.
Specifically, the proposal samples are first produced by the MLM model(i.e.,
BERT). And then, an energy-based model, which is a linear combination of
scores conditioning context from the different black-box experts representing
fluency, the control attribute, and faithfulness, respectively, is used to
accept/reject the proposed sample. Those two steps are iterated until
obtaining the desired texts. The framework is training-free, without any fine-
tuning or structural assumptions. Nevertheless, Mix and Match is an iteration-
base method; it takes almost 11 seconds to generate a sequence of length 20,
which reduces the practical value. Similarly, COLD Decoding (Qin et al.,
2022), an energy-based and iterative CTG approach, formulates the controlled
text generation task as sampling from an energy-based model using Langevin
Dynamics, without the need for any task-specific fine-tuning. Like Mix and
Match mentioned earlier, it also suffers from the problems of lower efficiency
in text generation.

To sum up, the idea of ”guided strategies” is simple and flexible. The main
advantage of this approach lies in the separation of the post-processing
module from the model. When the number of parameters of the PLM increases,
such advantage becomes more apparent.

Trainable Strategies: Different from Guided Strategies, although the methods
using the trainable strategy also work in the inference phase, the extra
processing module needs to be trained jointly with PLM whose parameters are
fixed.

Energy Based Model (EBM) (Deng et al., 2020) is proposed to guide the PLM to
generate desired text. The generative model is formalised as follows:

(10) $P_{\theta}(x)\propto P_{LM}(x)\exp\left(-E_{\theta}(x)\right),$

where $P_{LM}(x)$ is a local normalized language model whose parameters are
frozen during training, and $E_{\theta}(x)$ is an energy function that is
aimed to steer the joint model $P_{\theta}(x)$ to get close to the desired
data distribution. The Noise Contrastive Estimation (NCE) algorithm is used to
train the model to cope with the intractability issue of the energy model.
Experiments show that the proposed method yields a lower perplexity compared
with locally normalized baselines on the task of generating human-like text.
The use of large-scale PLM makes this method possible, because the quality of
the generated text from the joint model relies heavily on the quality of the
underlying language model.

Furthermore, since RL-based methods may lead to the problem of ”degeneration”
in the sense of producing poor examples that improve the average reward but
forgo the coherence and fluency, a distributional approach for controlled text
generation (Khalifa et al., 2021) was proposed to solve the problem. It uses
the Energy-based Model (EBM) to represent the point-wise and distributional
constraints in one go:

(11) $p(x)\doteq\frac{P(x)}{Z},$

(12) $Z\doteq\sum_{x}P(x),$

(13) $P(x)=a(x)e^{\sum_{i}\lambda_{i}\phi_{i}(x)},$

where $p(x)$ is the desired normalized distribution, $Z$ is the normalized
term, $\phi_{i}(x)$ represents the constraint judgment function (point-wise it
means 0 or 1, distributional-wise it may be a continuous value between 0 and
1), and $\lambda_{i}$ is the corresponding coefficient estimated by using Self
Normalized Importance Sampling (SNIS), $a(x)$ is the original PLM. As it is
hard to calculate $p(x)$ directly, the approach adopts a method similar to
variational inference to approximate the distribution, by first initializing a
policy $\pi=a(x)$, and then minimizing the cross entropy between the policy
$\pi$ and the desired normalized distribution $p(x)$:

(14) $CE\left(p,\pi_{\theta}\right)=-\sum_{x}p(x)\log\pi_{(}x).$

The optimization process adopts the KL-Adaptive distributional policy gradient
(DPG) algorithm to make $\pi$ approximate the model $p(x)$ that satisfies the
constraints. The method unifies the point-wise and distributional-wise
constraints in a single framework, and the experimental results show
superiority in satisfying the constraints while avoiding degeneration.
However, this approach suffers from a high computational cost. This challenge
needs to be addressed in the future.

Generally speaking, Trainable Strategies require the post-process module to be
jointly trained on the basis of PLM, and realize controllable text generation
by adjusting the original probability distribution of PLM to the desired data
distribution using the trained post-process module. This type of method build
upon the probability modeling theory and thus has a good theoretical basis,
yet they are still at the early stage and need to improve the issues related
to computational efficiency and text quality.

Table 3.  A summary of each CTG method. We list the main characteristics of
three categories of CTG methods from a macro perspective, and list the typical
literatures in seven sub-categories.

Method | Main characteristics | Subcategory | Typical References
---|---|---|---
Fine-tuning |  standard training; efficient inference; higher text quality; weaker controllability | Adapted Module |  (Zeldes et al., 2020), (Ribeiro et al., 2021b),(Lin et al., 2021)
Prompt |  |  (Jiang et al., 2020),(Shin et al., 2020),(Lester et al., 2021), (Qian et al., 2022)
---
(Li and Liang, 2021) (Zhang and song, 2022),(Carlsson et al., 2022b),(Yang et
al., 2022)
Reinforce Learning |  |  (Ziegler et al., 2019),(Stiennon et al., 2020),(Liu et al., 2020c),
---
(Tambwekar et al., 2019)
Refact/Retrain |  computationally expensive training; higher text quality; better controllability | Retrain |  (Keskar et al., 2019),(Zhang et al., 2020d)
Refact |  |  (Zhang et al., 2020d),(Chan et al., 2021),(Wang et al., 2021b),(He, 2021)
---
(Zheng et al., 2020),(Zeng and Nie, 2021)
Post-Process |  efficient training; computationally expensive inference; lower text quality; better controllability | Guided Strategy |  |  (Fan et al., 2018),(Holtzman et al., 2020), (Dathathri et al., 2020),
---
(Xu et al., 2020), (Scialom et al., 2020),(Krause et al., 2021),
(Liu et al., 2021c),(Yang and Klein, 2021),(Lin and Riedl, 2021)
(Pascual et al., 2021), (Mireshghallah et al., 2022),(Qin et al., 2022)
Trainbale Strategy |  (Deng et al., 2020),(Khalifa et al., 2021)

###  3.5. Summary

In this section, we divide the current PLM-based CTG approaches into three
categories according to the way how PLMs are used. For each category, we
analyze its main principles, process, and methods. A summary of the works is
shown in Table 3 . ”Fine-tuning” is a more general method that has been widely
used in both NLU and NLG tasks. How to make full use of the power of PLMs in
specific tasks is still a hot research topic for future research. The
retraining or refactoring approaches typically involve high training costs and
the lack of large-scale labeled data. To overcome the limitations, combining
general pre-trained models and the use of semi-supervised or self-supervised
learning to build a pre-trained model dedicated to CTG, would be a feasible
way in the future.

The emergence of post-processing methods is rooted in the powerful text-
generation capabilities of pre-trained language models. This kind of method
generally assumes that the PLMs can produce high-quality text, and then use a
post-processing module as a filter to screen out the desired type of text.
Most current decoding-time approaches (post-process) are still computationally
expensive, and the quality of generated text can be low. However, it has some
promising advantages, because the parameters of the PLMs do not need to be
retrained, thus greatly saving computing resources for the model training
stage. In recent years, the scale of pre-trained models has been getting
larger, and their mastery of language knowledge is getting more comprehensive.
At the same time, the sheer size of parameters makes the PLMs resource-
intensive to fine-tune and retrain. The above-mentioned trends coincide
perfectly with the advantages of the ”post-process” methods. Thus it has a
great potential for future research and development.

##  4\. Evaluation Methods

After building an NLG model, we need to assess its performance, for which
various evaluation methods can be used. The performance of a model is
reflected by suitable evaluation metrics. CTG is slightly different from the
general NLG tasks due to the need to fulfilling the controlled elements.
Therefore, CTG is concerned about not only the quality of the generated text
but also the satisfaction with the controlled elements. As a consequence, we
usually use both general and CTG-specific metrics to evaluate a CTG model.

###  4.1. General NLG Evaluation Metrics

For any CTG model, it is essential to evaluate the quality of generated text
in general aspects such as: 1) fluency: how fluent the language in the output
text is (Celikyilmaz et al., 2018; Du et al., 2017), 2) factuality: to what
extent the generated text reflects the facts described in the context.
(Holtzman et al., 2019; Welleck et al., 2019), 3) grammar: how grammatically
correct the generated text is, 4) diversity: whether the generated text is of
different types or styles. The ways of measuring these general evaluation
aspects can be divided into three categories based on who makes the judgments
on them: human beings or the machine (as shown in Figure 7).

Figure 7. An overview of general evaluation metrics divided by the
participants.

####  4.1.1. Human-Centric Evaluation Metrics

Human beings create Natural Language as a crucial form of human communication.
So humans are the best evaluators of the natural language texts generated by
NLG systems. We call the evaluation metrics that only human beings involve as
human-centric evaluation metrics, and they can be roughly divided into two
types:

##### Direct evaluation

In this type, human assessors judge the quality of the generated texts
directly. A simple way is to make a binary decision, i.e., good or bad, and a
more complex way is to use finer-grained decisions: e.g., Likert scale as
shown in Figure 8(a), RankME in Figure 8(b), etc. (Novikova et al., 2018;
Celikyilmaz et al., 2018; Holtzman et al., 2019).

(a) An example of Likert scale (Celikyilmaz et al., 2020). Human annotators
need to rate the informativeness (an index of measuring whether generated text
provides all the useful information from the meaning representation) from 1 to
5 in this example based on the meaning representation.

(b) An example of RankME (Celikyilmaz et al., 2020). Human annotators need to
give a score of the informativeness in a given range for each utterance in
this example based on the meaning representation.

Figure 8. Examples of human-centric evaluation metrics include Likert scale
and RankME.

##### Indirect evaluation

Different from direct evaluation, indirect evaluation is done by measuring the
effect of the generated text on downstream tasks, from either a user’s
perspective (such as whether or not it leads to an improved decision-making or
text comprehension accuracy (Gkatzia and Mahamood, 2015) which are usually
measured by the User Task Success evaluation), or from a system’s perspective
(Aziz et al., 2012; Denkowski et al., 2014), such as the performance of a
dialogue system which is usually measured by the System Purpose Success
evaluation.

####  4.1.2. Automatic Evaluation Metrics

Automatic evaluation metrics for NLG usually compare the similarity of the NLG
model generated texts $G$ to the corresponding reference texts $R$ (i.e.,
human-written texts) in benchmarking datasets. We divide the similarity
measures into three categories: lexical-based, syntactic-based, and semantic-
based. For the convenience of explanation, we use a pair of sentences “the man
was found on the chair” (a generated text) and “the man was on the chair” (the
corresponding reference text), as an example to illustrate the evaluation
metrics in the rest of this section.

##### Lexical-based Metrics

The lexical-based metrics measure the similarities between basic lexical units
(e.g., words or phrases) across the pair of sentences, which are then
aggregated into the overall sentence-level similarity.

BLEU (Papineni et al., 2002) is a commonly used metric in natural language
processing tasks to evaluate the difference between generated texts of an NLG
model and reference texts. Its value falls into the range between 0.0 and 1.0.
If two sentences match perfectly, the value of BLEU is $1.0$. If there is no
overlap between them at all, the BLEU value is $0.0$.

Specifically, BLEU counts the n-gram matches in the generated text with the
reference text. For convenience, we use BLEU-n to represent BLEU with respect
to n-grams, as follows.

(15) $\text{BLEU-n}=\frac{\sum_{t\in G}\sum_{n-gram\in
t}Count_{match}(n-gram)}{\sum_{t\in G}\sum_{n-gram\in t}Count(n-gram)},$

where $t$ is the generated text to be compared, and match means that an n-gram
appears in both the generated and reference texts. The larger BLEU-n, the
better quality of the generated text.

Take BLEU-2 as an example. The set of 2-grams in the above example generated
text, i.e., $G$, is {the man, man was, was found, found on, on the, the
chair}, and in the reference text, i.e., $R$, is {the man, man was, was on, on
the, the chair}. According to Equation 15, BLEU-2 in this example is
$4/6=2/3$.

The n-grams in BLEU are simple to use and fast to calculate, but it ignores
the structure and semantic information of the text. Furthermore, as an
extension for BLEU, Self-BLEU (Zhu et al., 2018) is proposed as a metric to
measure the diversity of the generated text. Since BLEU can measure the
similarity of two different texts, self-BLEU calculates the BLEU score between
each generated text, and takes the average BLEU score to represent the
diversity of all generated texts. The generated text with a higher diversity
has a lower self-BLEU score.

ROUGE (Lin, 2004) is the abbreviation of Recall-Oriented Understanding for
Gisting Evaluation. It is an automatic summary evaluation method, a set of
indicators to evaluate the quality of the generated texts automatically. It
compares the generated texts with a group of reference texts, counts the
number of overlapping basic units (n-gram), and obtains the corresponding
score to measure the similarity between the automatically generated texts and
the reference texts. The formula of $ROUGE$ is as follows:

(16) $\text{ROUGE-n}=\frac{\sum_{t\in G}\sum_{n-gram\in
t}Count_{match}(n-gram)}{\sum_{t\in R}\sum_{n-gram\in t}Count(n-gram)},$

Comparing the Equations 15 and 16, the only difference between BLEU-n and
ROUGE-n lies in the denominator. BLEU-n is focused on precision and the
denominator is the total number of n-grams in the generated texts. While
ROUGE-n is recall-oriented, so the denominator is the total number of n-grams
in the reference texts. A larger ROUGE-n indicates better recall-oriented
quality. For the example above, Equation 16 gives the ROUGE-2 of 4/5.

Perplexity (PPL) is a metric to measure how well the language probability
model predicts a sample. A Language probability model is a probability
distribution on a given text, i.e., probability of the n+1-th word given the
first n words of the text. For the task of NLG, when a language probability
model is trained on the training set with reference texts, and is used to
predict the generated text. The higher prediction probability indicates the
better quality of the generated text. In practice, PPL takes the reciprocal
form as follows:

(17) $PPL=\sqrt[n]{\prod_{i=1}^{n}\frac{1}{p(w_{i}\mid
w_{1}w_{2}...w_{i-1})}},$

where $n$ is the number of words in the generated text, $w_{i}$ is the i-th
word in it, and $p(w_{i})$ is the probability of the i-th word in a language
model trained with reference texts. The smaller the PPL value, the better
fluency of the generated text.

In addition, a variant of PPL called Reverse PPL (Shi et al., 2020) mainly
focuses on the diversity of generated text by training a language probability
model on the generated texts and calculating the PPL score on the reference
texts. A smaller Reverse PPL means that the generated texts are quite
different from the reference texts, indicating a higher diversity.

Distinct-n (Li et al., 2016a) is an n-gram-based metric and applies to some
scenes that pursue the diversity of generated texts (such as dialogue and
advertisement generation). The greater the value, the higher the diversity. It
is formulated as follows:

(18) $\text{Distinct-n}=\frac{Count(unique\ n-gram)}{Count(n-gram)},$

where the numerator is the number of n-grams that appear once in the generated
texts and the denominator is the total number of n-grams in the generated
texts. In the above example, since there is no duplicated n-gram in the
generated text, i.e., the count of unique n-grams equals the total number of
n-grams, the value of Distinct-2 is 1.

##### Syntactic-based Metrics

The syntax is related to the grammatical arrangement of words in a sentence.
Common syntactic-based metrics are as follows: TESLA (Liu et al., 2010) is the
abbreviation for Translation Evaluation of Sentences with Linear-programming-
based Analysis. The goal of it is to output a score to measure the quality of
the generated texts. TESLA-M (Dahlmeier et al., 2011), a lighter version of
TESLA, is the arithmetic average of F-measures between bags of n-grams (BNGs).
It first matches BNGs with two different similarity functions: maxSim and POS
(part-of-speech) tag. Then, we combine different n-grams and similarity
functions and get a score based on them to measure the quality of the
generated texts. TESLA is an advanced version that adopts a general linear
combination of three types of features, including TESLA-M. There are also
various other syntactic-based metrics, for example, Liu and Gildea (2005)
adopts the constituent labels and head-modifier dependencies in the
evaluation.

##### Semantic-based Metrics

Semantic-based metrics aim to handle the evaluation of texts that are
lexically different but have a similar semantic meaning. Compared with
lexical-based and syntactic similarity, semantic similarity requires more
information to be considered and is more difficult to measure. In recent
years, PLM-based evaluation methods have emerged, which aim to evaluate the
text using the trained PLMs inversely. BERTscores (Zhang et al., 2020b)
replaces the n-gram overlaps defined in BLEU, with embeddings from BERT, to
learn a semantic-awareness metric. Similarly, Bertr and YiSi (Mathur et al.,
2019) take advantage of BERT embeddings to capture the semantic similarity
between the generated text and its reference. Some approaches also try to
fine-tune PLMs for text quality estimation (Reimers and Gurevych, 2019; Zhou
and Xu, 2020; Liu et al., 2021b). Particularly, Sellam et al. (2020) propose a
task-specific pre-trained model, namely BLEURT, for text assessment. BLEURT
first adds random perturbations to Wikipedia sentences to construct millions
of synthetic examples. Then, a BERT-based pre-trained model is trained on
several lexical- and semantic-level supervision signals with a multitask loss.
The experiments show that BLEURT benefits from pre-training and is robust to
both domain and quality drifts. MAUVE (Pillutla et al., 2021) is an automatic
metric to measure human-written and machine-generated texts. Specifically,
MAUVE compares the machine text distribution to that of human text, which is
embedded by a PLM, using divergence frontiers, to reveal the two different
errors of the NLG model, that is the model assigns high probability to
sequences that do not resemble human-written text, and generative model fails
to yield diverse samples as human-written texts.

####  4.1.3. Semi-automatic Evaluation Metrics

When faced with more diverse tasks such as story generation and open-domain
dialogue generation, the automatic evaluation methods turn out less ideal,
because they can mainly evaluate the surface and simple similarity between the
reference and target sentences. Humans can better distinguish a more diverse
range of features, such as fluency and grammar, so that semi-automatic
evaluation combines automatic and human-centric evaluation methods to make
better use of the benchmarking datasets for these harder tasks in order to get
more reliable evaluation results.

Lowe et al. (2017) encode the context, the generated text, and a reference
text into vectors (c, g, and r, respectively) using a hierarchical RNN
encoder. Then the dot-product operation is adopted to transfer the vectors
into a score
($score(\textbf{c},\textbf{r},\textbf{g})=(\textbf{c}^{T}M\textbf{g}+\textbf{r}^{T}N\textbf{g}-\alpha)/\beta$).
Finally this score is made as close as possible to the human judged score. A
model that uses human judgments as labels can ensure good consistency with
human judgments, but sometimes it may be too conservative and lacks diversity
due to the quality of human evaluators.

Without using a human-judged score as the label, we can simply calculate the
perplexity of a probabilistic model first and then let humans evaluate the
beam-searched outputs. Specifically, Hashimoto et al. (2019) encode the human
judgments and model outputs into the same space by using the same encoder.
Then a discriminator is trained to distinguish whether the text is generated
by a model or by a human evaluator. Finally, the leave-one-out error of the
discriminator is computed. By doing so, we can preserve the diversity and
quality of the generated text at the same time.

###  4.2. CTG-specific Evaluation

In addition to the above three categories of general NLG evaluation metrics,
the CTG task demands additional evaluation metrics that take into account the
controlled elements. In most cases, the control conditions are specific and
relatively easy to evaluate whether a CTG model has achieved the desired
constraints. According to the definition of the controlled elements in
section2.1, we can divide CTG-specific evaluation methods into two kinds of
automatic evaluation methods. Additionally, human evaluation metrics for CTG
are also discussed.

##### Semantic consistency metric.

This kind of evaluation method generally corresponds to the semantic control
conditions. With this type of method, we first need to construct a training
set with positive (samples satisfying the control conditions) and negative
(samples not satisfying the control conditions) samples. Then a classifier is
trained to identify whether the model generates the controlled text, e.g., the
sentiment, topic(Dathathri et al., 2020; Qian et al., 2022), style(Fu et al.,
2018; Gao et al., 2019; Chen et al., 2019a) and toxic (Krause et al., 2021;
Liu et al., 2021c; Zhang and song, 2022) attributes and contextual
consistency. Accuracy, equal to the number of test samples divided by the
number of test samples correctly accord with controlled elements, is always
used to measure the performance of the CTG model . It can be applied to a
well-labeled dataset but will become less reliable when facing unbalanced
data. Recently, CTRLEVAL (Ke et al., 2022), a PLM-based and reference-free
method, is proposed to conduct the training-free controllable text generation
evaluation. Specifically, CTRLEVAL formulates three aspects, including
coherence, consistency, and attribute relevance, to evaluate a generated text.
Each aspect is defined as multiple text-infilling tasks with the PLM, from
which the ensemble of generation probabilities forms the final evaluation
results. This approach does not need any labeled data and achieves higher
correlations with human judgments.

##### Rule-based metric.

When it comes to the structural and lexical-based controlled text generation
tasks, we usually use some rules to judge when the generated text conforms to
the pre-defined controlled element and then count the number of test samples
that satisfy the control conditions as the final evaluation results. For
example, we always use Coverage to measure effectiveness on the lexical
constraint text generation task (Wang et al., 2021b; Zhang et al., 2020d; He,
2021; Carlsson et al., 2022a), that is, to calculate the average percentage of
input keywords that are present in the generated text of the CTG model.
Success Rate is used to measure the matching degree between the generated text
and the given structural control elements (i.e., Parts-of-Speech, Syntax
Spans, and Syntax Tree), during which an external extraction tool for sentence
structure such as oracle POS tagger and off-the-shelf syntax parser is applied
to the generated texts (Li et al., 2022). Moreover, some general text
evaluation metrics are also appropriate to be used as CTG-specific evaluation.
For instance, the lexical-level overlap between the reference sentences
containing control elements and generated texts could reflect the completion
of control conditions on the task of table/graph to text (Puduppully et al.,
2019; Song et al., 2020a; Ribeiro et al., 2021a; Zhao et al., 2020; Ribeiro et
al., 2021b; Su et al., 2021) and story generation (Goldfarb-Tarrant et al.,
2020; Fang et al., 2021), etc. Thereby some metrics such as BLEU-n and ROUGE-n
are also could regarded as the CTG-specific evaluation metrics to indicate the
satisfaction of the desired constraints.

##### Human evaluation metric.

Most of the CTG-specific metrics for the evaluation of text semantics and
structures mentioned above require the introduction of an additional semantic-
relevant classifier or structure-relevant parse tools to help with
measurement, which may not be as reliable as the well-educated human.
Therefore, human evaluation is also essential to test a CTG model at the
aspect of the satisfaction of control elements. Similar to the Human-Centric
method described in the general NLG evaluation Metrics section, human
assessors are always directly asked to score the relevance between the control
attributes (e.g., sentiment, topic, toxic, contextual relevancy, common sense
(Chan et al., 2021; Dathathri et al., 2020; Liu et al., 2021c; Yang and Klein,
2021; Zhang and song, 2022; Carlsson et al., 2022a) etc.) and the given
generated texts, the average score obtained is used as the final evaluation
result.

###  4.3. Summary

The evaluation metrics discussed above need to be flexibly combined according
to different tasks. They have their advantages and disadvantages. Sometimes,
multiple evaluation methods can be used together to determine the performance
of an NLG model.

Human-centric evaluation is the most precise method for evaluating the quality
of system-generated texts. It would be ideal when human resources permit,
which unfortunately is not always the case. Thus this type of evaluation
suffers from various weaknesses: 1) Expensive and time-consuming: the
recruitment of evaluators, selection of personnel, cost of the evaluation, and
other steps all require time and manpower, especially the evaluation tasks
that can only be handled by domain experts; 2) Maintaining quality control
(Ipeirotis et al., 2010; Mitra et al., 2015): although the emergence of online
crowd-sourcing platforms has eased the cost problem to a certain extent, we
cannot guarantee the quality of personnel conducting online evaluations; 3)
Lack of consistency (van der Lee et al., 2021): due to the change of
evaluation personnel, the reproducibility of the evaluation results may not be
high, which leads to the inconsistency problem.

Compared with human-centric evaluation metrics, automatic evaluation metrics
are easy to use, fast to obtain results, and low-cost. However, the evaluation
of this type is less precise than human assessments. It is vital to develop
automated evaluation methods comparable to the human level in the future.
Semi-automatic evaluation metric combines the advantages of human-centric
methods and automatic methods, but still require a lot of human judgments or
labeling, which is also expensive and time-consuming. One key future research
direction is to develop a better way to augment human judgments with automatic
evaluation, or vice versa, to obtain improved evaluation quality and
diversity.

When it comes to the CTG tasks, we not only need to use the above general
evaluation metrics to evaluate the generated texts but also need specific
metrics to evaluate whether the generated texts are consistent with the
controlled elements. Generally speaking, the consistency of controlled
elements and generated texts are relatively easy to measure in most cases; the
challenges of text quality evaluation still lie in the general evaluation
methods.

##  5\. Challenges and Future Directions

###  5.1. Challenges

The birth of large-scale pre-trained language models brings the research of
text generation to a new stage. The PLMs have mastered a remarkable level of
linguistic knowledge (semantic, syntax, etc.) from large-scale corpus,
enabling the production of more fluent and diverse text naturally. However,
due to the black box characteristics of neural networks, the general PLMs are
still not sufficiently controllable during the text generation process. How to
make full use of the powerful PLMs to generate the desired and controllable
text, has become a promising yet challenging field in both academia and
industry. Based on the systematic review of the key concepts, methods and
findings in the latest development of PLM-based controllable text generation,
we think this promising and fast-growing area is still facing a number of
challenges in the following aspects.

First, pre-trained language models have learned rich knowledge from large-
scale corpus used for pre-training. However, an NLG model needs to learn
control constraints on its own training corpus. It is often difficult for the
existing PLM-based models to ensure the domain diversity of the generated text
while pursuing controllability. This is indeed the well-known catastrophic
forgetting problem in PLM. In the field of text generation, it is still a
challenge to overcome this problem and make the PLM-based NLG model generate
multi-domain text that satisfies specific control conditions with few or zero
domain-specific samples.

Second, controlling the generation of text in the decoding stage of a
generative model is a low-cost method in terms of model training. It can
maintain the characteristics of the original language model to the greatest
extent. However, in most cases, the existing methods are relatively
rudimentary, and only use the external decoupled attribute discriminator to
control the attributes. There is a distribution gap between the discriminator
and the generator, leading to a coarser granularity in the guidance process
and decreased quality of the generated text. In addition, decode-time
approaches are hard to be directly applied to fine-grained control scenarios
such as date-to-text or multi-attribute control tasks.

Third, from the perspective of probability theory, a generative Pre-trained
language model (referring specifically to the GPT-like models) is essentially
an enhanced version of dense conditional probability $p\left(x_{n}\mid
x_{1},x_{2},\ldots,x_{n-1}\right)$ to describe the probability distribution of
natural language. However, this local normalization format has certain
limitations in paragraph/document-level modeling. For example, it is hard to
keep long-range coherency in both semantic logic and controlled condition. It
calls for further research to establish global normalization based on PLMs to
ensure that text generation can be controlled locally and globally at the same
time.

Fourth, the construction of large-scale pre-trained language models is
typically data-driven, which allows the models to learn the primary logic and
common sense contained in the training corpus. However, the knowledge captured
in those models is rather superficial. The PLMs will lose generalization
ability when the training data does not contain relevant common sense and
domain-specific knowledge. Therefore, purely relying on PLMs could be
difficult to control the generated texts faithfully to common sense and rich
knowledge specific to the target domain.

Fifth, a reasonable and reliable evaluation has always been a bottleneck
restricting the development of more advanced text generation technologies.
This is also the case for controllable text generation. Generally speaking,
the satisfaction of controlled conditions is relatively easy to evaluate.
However, there is still a lack of an objective, accurate and comprehensive
evaluation mechanism that is fully compatible with human judgment. For
controllable text generation, in addition to the control conditions, the
quality of the text itself is equally important. If the quality of the
generated text of an NLG model cannot be accurately evaluated, it is hard to
think of a way to control them.

Finally, we believe that the research on controlled text generation is still
in the early stages. In Section 2.2 of this paper, we have summarized a range
of tasks involving CTG. However, few of them are actually dedicated CTG tasks.
With the rapid development of text generation, there is a need to come up with
dedicated benchmarking tasks for CTG with diverse control requirements.

###  5.2. Future Directions

Based on the summary of current work and the challenges mentioned above, we
suggest the following promising future directions for PLM-based controllable
text generation.

Prompt-based Learning: Prompt-based learning has become a new way for fine-
tuning PLMs. Based on the well-designed prompting function, a PLM is able to
perform few-shot or even zero-shot learning, adapting to new scenarios with
few or no labeled data, thus overcoming the problem of catastrophic
forgetting. The above features are also attractive for controlled text
generation, since the prompt-based methods are able to generate more diverse
text fields and increase the distribution space of the text to be filtered, so
that it is theoretically more possible to produce text that meets the specific
control conditions. At present, most of the Prompt-based methods are applied
to NLU tasks, but there is relatively little work on NLG tasks. Therefore, we
believe there is an urgent need to explore prompt-based learning methods
specifically for CTG.

Fine-grained Decoding Control: More fine-grained decoding control methods need
to be explored. On the one hand, the decoding-time methods can be improved to
achieve more effective control. For example, co-training between the guided
model and the generative model ensures finer-grained text generation and
higher text quality. On the other hand, the existing single-attribute (e.g.,
emotion, topic, etc.) controlled tasks can also be extended to multi-attribute
controlled tasks in a unified framework, achieving the simultaneous control of
multiple aspects for a generated sentence.

Integration with Classic Generative Theory and Linguistic Knowledge: The
controllable text generation task can be regarded as obtaining, from a natural
language distribution constructed by a PLM, the corresponding distribution
that satisfies certain specified constraints. This process is intrinsically
related to classic generative models such as Generative Adversarial Networks
(GANs) (Goodfellow et al., 2020), Variational Autoencoders (VAEs) (Kingma and
Welling, 2013), Energy-based Model (Deng et al., 2020) and their variants. It
is known that auto-regressive PLMs, i.e., the GPT family models, can not model
global information of the generated texts naturally, making it difficult to
control the generated text’s distribution at the paragraph/doc level. We
expect that combining probability theory to bridge the gap between PLMs and
traditional generative models, will help solve the problem at the theoretical
level.

In addition, auto-regressive PLM is essentially a locally normalized fashion,
allowing it to produce fluent text in short text-generation scenarios. From a
linguistic point of view, it is believed that more linguistic knowledge, such
as paragraph/chapter structures and logic, are needed in long text generation,
which PLMs can not provide directly. A promising solution is to combine
linguistic knowledge with PLMs to overcome the inherent problems of auto-
regressive models in long text modeling, so as to better ensure quality and
controllability.

Incorporation of Knowledge Graphs: A knowledge graph is a natural carrier of
explicit common sense and domain-specific knowledge and also provides
effective reasoning mechanisms. Therefore it can be a good complement to the
use of PLMs, which lack domain-specific knowledge and logical reasoning
capabilities. A straightforward way is to convert the knowledge graphs into
texts first and then treat them as a training corpus for the pre-training
process of PLMs, so that the PLM models would have learned the corresponding
domain knowledge. In addition, knowledge graphs can also be introduced in the
downstream tasks. We can regard knowledge graphs as constraint conditions, and
filter out the texts that do not meet the constraint conditions during the
decoding phase of PLMs, ensuring the controllability of the generated text in
terms of common sense and logic.

Novel Evaluation Metrics and Methods: Developing innovative evaluation metrics
for text generation is still an important topic that needs to be further
studied from both the general text generation perspective (such as fluency,
diversity, and coherence) and the CTG-specific perspective (such as fidelity).
Since the pre-trained language models have mastered a great deal of semantic
and grammatical knowledge, applying them in reverse to assess the text quality
of an NLG model would be an interesting and fascinating area for further
investigation.

New CTG tasks: Controllable text generation is a relatively broad concept. In
addition to the existing common controlled element of generated text, such as
content, attributes, and format, the connotation and the form of control
elements can be re-defined, and new tasks can be proposed according to new
application scenarios in the future. For instance, we could construct
corresponding syntax templates to make the text’s syntax manageable, or
combine with knowledge graphs, including logical rules to make the generated
text better conform to the prescribed logic, so as to achieve good logical
control. Furthermore, single-attribute control could be extended to any other
aspects, such as humor text generation.

##  6\. Conclusions

In this paper, we have comprehensively summarized the typical applications,
main approaches, and evaluation methodologies of controllable text generation
based on large-scale pre-trained language models. Based on the critical
analysis of the existing methods, we have identified a series of key
challenges in this field and highlighted several promising future directions.
Large-scale pre-trained language models have brought unprecedented
opportunities for the development of controllable text generation
technologies, calling for more researchers to join the field and create a new
era of it. We are hopeful that this literature survey is able to provide a
clear picture of the field and set a roadmap for researchers to move forward.

## References

  * (1)
  * Amin-Nejad et al. (2020) Ali Amin-Nejad, Julia Ive, and Sumithra Velupillai. 2020\.  Exploring Transformer Text Generation for Medical Dataset Augmentation. In _Proceedings of the Twelfth Language Resources and Evaluation Conference_. European Language Resources Association, Marseille, France, 4699–4708.  https://aclanthology.org/2020.lrec-1.578
  * Anderson et al. (2017) Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2017.  Guided Open Vocabulary Image Captioning with Constrained Beam Search. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017_ , Martha Palmer, Rebecca Hwa, and Sebastian Riedel (Eds.). Association for Computational Linguistics, 936–945.  https://doi.org/10.18653/v1/d17-1098
  * Aziz et al. (2012) Wilker Aziz, Sheila Castilho, and Lucia Specia. 2012\.  PET: a Tool for Post-editing and Assessing Machine Translation. In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_. European Language Resources Association (ELRA), Istanbul, Turkey, 3982–3987.  http://www.lrec-conf.org/proceedings/lrec2012/pdf/985_Paper.pdf
  * Barikeri et al. (2021) Soumya Barikeri, Anne Lauscher, Ivan Vulić, and Goran Glavaš. 2021.  RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 1941–1955.  https://doi.org/10.18653/v1/2021.acl-long.151
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020\.  Longformer: The Long-Document Transformer.  _CoRR_ abs/2004.05150 (2020).  arXiv:2004.05150 https://arxiv.org/abs/2004.05150
  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?. In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ (Virtual Event, Canada) _(FAccT ’21)_. Association for Computing Machinery, New York, NY, USA, 610–623.  https://doi.org/10.1145/3442188.3445922
  * Bengio et al. (2003) Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003.  A Neural Probabilistic Language Model.  _J. Mach. Learn. Res._ 3, null (mar 2003), 1137–1155.  https://dl.acm.org/doi/pdf/10.5555/944919.944966
  * Bhattacharyya et al. (2021) Sumanta Bhattacharyya, Amirmohammad Rooshenas, Subhajit Naskar, Simeng Sun, Mohit Iyyer, and Andrew McCallum. 2021.  Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 4528–4537.  https://doi.org/10.18653/v1/2021.acl-long.349
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _Advances in Neural Information Processing Systems_ , H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (Eds.), Vol. 33. Curran Associates, Inc., 1877–1901.  https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf
  * Carlsson et al. (2022a) Fredrik Carlsson, Joey Öhman, Fangyu Liu, Severine Verlinden, Joakim Nivre, and Magnus Sahlgren. 2022a.  Fine-Grained Controllable Text Generation Using Non-Residual Prompting. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Dublin, Ireland, 6837–6857.  https://doi.org/10.18653/v1/2022.acl-long.471
  * Carlsson et al. (2022b) Fredrik Carlsson, Joey Öhman, Fangyu Liu, Severine Verlinden, Joakim Nivre, and Magnus Sahlgren. 2022b.  Fine-Grained Controllable Text Generation Using Non-Residual Prompting. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Dublin, Ireland, 6837–6857.  https://doi.org/10.18653/v1/2022.acl-long.471
  * Castro Ferreira et al. (2019) Thiago Castro Ferreira, Chris van der Lee, Emiel van Miltenburg, and Emiel Krahmer. 2019.  Neural data-to-text generation: A comparison between pipeline and end-to-end architectures. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 552–562.  https://doi.org/10.18653/v1/D19-1052
  * Celikyilmaz et al. (2018) Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018.  Deep Communicating Agents for Abstractive Summarization. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. Association for Computational Linguistics, New Orleans, Louisiana, 1662–1675.  https://doi.org/10.18653/v1/N18-1150
  * Celikyilmaz et al. (2020) Asli Celikyilmaz, Elizabeth Clark, and Jianfeng Gao. 2020\.  Evaluation of Text Generation: A Survey.  _CoRR_ abs/2006.14799 (2020).  arXiv:2006.14799 https://arxiv.org/abs/2006.14799
  * Chan et al. (2021) Alvin Chan, Yew-Soon Ong, Bill Pung, Aston Zhang, and Jie Fu. 2021.  CoCon: A Self-Supervised Approach for Controlled Text Generation. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=VD_ozqvBy4W
  * Chandu et al. (2019) Khyathi Chandu, Shrimai Prabhumoye, Ruslan Salakhutdinov, and Alan W Black. 2019.  “My Way of Telling a Story”: Persona based Grounded Story Generation. In _Proceedings of the Second Workshop on Storytelling_. Association for Computational Linguistics, Florence, Italy, 11–21.  https://doi.org/10.18653/v1/W19-3402
  * Chang et al. (2021) Haw-Shiuan Chang, Jiaming Yuan, Mohit Iyyer, and Andrew McCallum. 2021.  Changing the Mind of Transformers for Topically-Controllable Language Generation. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_. Association for Computational Linguistics, Online, 2601–2611.  https://doi.org/10.18653/v1/2021.eacl-main.223
  * Chen et al. (2019a) Cheng-Kuan Chen, Zhufeng Pan, Ming-Yu Liu, and Min Sun. 2019a.  Unsupervised stylish image description generation via domain layer norm. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 8151–8158.
  * Chen et al. (2019b) Huimin Chen, Xiaoyuan Yi, Maosong Sun, Wenhao Li, Cheng Yang, and Zhipeng Guo. 2019b.  Sentiment-Controllable Chinese Poetry Generation. In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_. International Joint Conferences on Artificial Intelligence Organization, 4925–4931.  https://doi.org/10.24963/ijcai.2019/684
  * Conneau and Lample (2019) Alexis Conneau and Guillaume Lample. 2019.  _Cross-Lingual Language Model Pretraining_.  Curran Associates Inc., Red Hook, NY, USA.  https://proceedings.neurips.cc/paper/2019/file/c04c19c2c2474dbf5f7ac4372c5b9af1-Paper.pdf
  * Dahlmeier et al. (2011) Daniel Dahlmeier, Chang Liu, and Hwee Tou Ng. 2011.  TESLA at WMT 2011: Translation Evaluation and Tunable Metric. In _Proceedings of the Sixth Workshop on Statistical Machine Translation_. Association for Computational Linguistics, Edinburgh, Scotland, 78–84.  https://aclanthology.org/W11-2106
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019\.  Transformer-XL: Attentive Language Models beyond a Fixed-Length Context. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 2978–2988.  https://doi.org/10.18653/v1/P19-1285
  * Dathathri et al. (2020) Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2020\.  Plug and Play Language Models: A Simple Approach to Controlled Text Generation. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=H1edEyBKDS
  * Deng et al. (2020) Yuntian Deng, Anton Bakhtin, Myle Ott, Arthur Szlam, and Marc’Aurelio Ranzato. 2020.  Residual Energy-Based Models for Text Generation. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=B1l4SgHKDH
  * Denkowski et al. (2014) Michael Denkowski, Chris Dyer, and Alon Lavie. 2014.  Learning from Post-Editing: Online Model Adaptation for Statistical Machine Translation. In _Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics_. Association for Computational Linguistics, Gothenburg, Sweden, 395–404.  https://doi.org/10.3115/v1/E14-1042
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Dinan et al. (2020) Emily Dinan, Angela Fan, Adina Williams, Jack Urbanek, Douwe Kiela, and Jason Weston. 2020\.  Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 8173–8188.  https://doi.org/10.18653/v1/2020.emnlp-main.656
  * Dinan et al. (2019) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2019\.  Wizard of Wikipedia: Knowledge-Powered Conversational Agents. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=r1l73iRqKm
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  _Unified Language Model Pre-Training for Natural Language Understanding and Generation_.  Curran Associates Inc., Red Hook, NY, USA.  https://dl.acm.org/doi/pdf/10.5555/3454287.3455457
  * Du et al. (2017) Xinya Du, Junru Shao, and Claire Cardie. 2017.  Learning to Ask: Neural Question Generation for Reading Comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Vancouver, Canada, 1342–1352.  https://doi.org/10.18653/v1/P17-1123
  * Fan et al. (2018) Angela Fan, Mike Lewis, and Yann Dauphin. 2018.  Hierarchical Neural Story Generation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Melbourne, Australia, 889–898.  https://doi.org/10.18653/v1/P18-1082
  * Fang et al. (2021) Le Fang, Tao Zeng, Chaochun Liu, Liefeng Bo, Wen Dong, and Changyou Chen. 2021\.  Outline to Story: Fine-grained Controllable Story Generation from Cascaded Events.  _arXiv preprint arXiv:2101.00822_ (2021).  https://arxiv.org/pdf/2101.00822v1.pdf
  * Feng et al. (2018) Xiaocheng Feng, Ming Liu, Jiahao Liu, Bing Qin, Yibo Sun, and Ting Liu. 2018\.  Topic-to-Essay Generation with Neural Networks. In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_. International Joint Conferences on Artificial Intelligence Organization, 4078–4084.  https://doi.org/10.24963/ijcai.2018/567
  * Ficler and Goldberg (2017) Jessica Ficler and Yoav Goldberg. 2017.  Controlling Linguistic Style Aspects in Neural Language Generation. In _Proceedings of the Workshop on Stylistic Variation_. Association for Computational Linguistics, Copenhagen, Denmark, 94–104.  https://doi.org/10.18653/v1/W17-4912
  * Firdaus et al. (2022a) Mauajama Firdaus, Hardik Chauhan, Asif Ekbal, and Pushpak Bhattacharyya. 2022a.  EmoSen: Generating Sentiment and Emotion Controlled Responses in a Multimodal Dialogue System.  _IEEE Transactions on Affective Computing_ 13, 3 (2022), 1555–1566.  https://doi.org/10.1109/TAFFC.2020.3015491
  * Firdaus et al. (2022b) Mauajama Firdaus, Arunav Shandilya, Asif Ekbal, and Pushpak Bhattacharyya. 2022b.  Being Polite: Modeling Politeness Variation in a Personalized Dialog Agent.  _IEEE Transactions on Computational Social Systems_ (2022), 1–10.  https://doi.org/10.1109/TCSS.2022.3182986
  * Fu et al. (2018) Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, and Rui Yan. 2018.  Style transfer in text: Exploration and evaluation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.  https://ojs.aaai.org/index.php/AAAI/article/view/11330
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making Pre-trained Language Models Better Few-shot Learners. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 3816–3830.  https://doi.org/10.18653/v1/2021.acl-long.295
  * Gao et al. (2019) Xiang Gao, Yizhe Zhang, Sungjin Lee, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2019.  Structuring Latent Spaces for Stylized Response Generation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. Association for Computational Linguistics, Hong Kong, China, 1814–1823.  https://doi.org/10.18653/v1/D19-1190
  * Ghazvininejad et al. (2018) Marjan Ghazvininejad, Chris Brockett, Ming-Wei Chang, Bill Dolan, Jianfeng Gao, Wen-tau Yih, and Michel Galley. 2018\.  A knowledge-grounded neural conversation model. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.  https://ojs.aaai.org/index.php/AAAI/article/view/11977
  * Gkatzia and Mahamood (2015) Dimitra Gkatzia and Saad Mahamood. 2015.  A Snapshot of NLG Evaluation Practices 2005 - 2014\. In _Proceedings of the 15th European Workshop on Natural Language Generation (ENLG)_. Association for Computational Linguistics, Brighton, UK, 57–60.  https://doi.org/10.18653/v1/W15-4708
  * Goldfarb-Tarrant et al. (2020) Seraphina Goldfarb-Tarrant, Tuhin Chakrabarty, Ralph Weischedel, and Nanyun Peng. 2020.  Content Planning for Neural Story Generation with Aristotelian Rescoring. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 4319–4338.  https://doi.org/10.18653/v1/2020.emnlp-main.351
  * Goodfellow et al. (2020) Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2020\.  Generative adversarial networks.  _Commun. ACM_ 63, 11 (2020), 139–144.
  * Grover et al. (2019) Aditya Grover, Jiaming Song, Alekh Agarwal, Kenneth Tran, Ashish Kapoor, Eric Horvitz, and Stefano Ermon. 2019.  _Bias Correction of Learned Generative Models Using Likelihood-Free Importance Weighting_.  Curran Associates Inc., Red Hook, NY, USA.  https://papers.nips.cc/paper/2019/file/d76d8deea9c19cc9aaf2237d2bf2f785-Paper.pdf
  * Hashimoto et al. (2019) Tatsunori B. Hashimoto, Hugh Zhang, and Percy Liang. 2019\.  Unifying Human and Statistical Evaluation for Natural Language Generation. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 1689–1701.  https://doi.org/10.18653/v1/N19-1169
  * He (2021) Xingwei He. 2021\.  Parallel Refinements for Lexically Constrained Text Generation with BART. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 8653–8666.  https://doi.org/10.18653/v1/2021.emnlp-main.681
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019.  The Curious Case of Neural Text Degeneration. In _International Conference on Learning Representations_.  https://arxiv.org/abs/1904.09751
  * Holtzman et al. (2020) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020.  The Curious Case of Neural Text Degeneration. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=rygGQyrFvH
  * Horacek (2001) Helmut Horacek. 2001\.  Building Natural Language Generation Systems - Ehud Reiter and Robert Dale (Eds.), University of Aberdeen and Macquarie University, Cambridge University Press, 2000, ISBN 0-521-62036-8.  _Artif. Intell. Medicine_ 22, 3 (2001), 277–280.  https://doi.org/10.1016/S0933-3657(00)00114-7
  * Hu et al. (2017) Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017.  Controllable Text Generation.  _CoRR_ abs/1703.00955 (2017).  arXiv:1703.00955 http://arxiv.org/abs/1703.00955
  * Hua and Wang (2020) Xinyu Hua and Lu Wang. 2020\.  PAIR: Planning and Iterative Refinement in Pre-trained Transformers for Long Text Generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 781–793.  https://doi.org/10.18653/v1/2020.emnlp-main.57
  * Huang et al. (2020b) Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020b.  Challenges in building intelligent open-domain dialog systems.  _ACM Transactions on Information Systems (TOIS)_ 38, 3 (2020), 1–32.
  * Huang et al. (2020a) Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack Rae, Vishal Maini, Dani Yogatama, and Pushmeet Kohli. 2020a.  Reducing Sentiment Bias in Language Models via Counterfactual Evaluation. In _Findings of the Association for Computational Linguistics: EMNLP 2020_. Association for Computational Linguistics, Online, 65–83.  https://doi.org/10.18653/v1/2020.findings-emnlp.7
  * Ipeirotis et al. (2010) Panagiotis G. Ipeirotis, Foster Provost, and Jing Wang. 2010\.  Quality Management on Amazon Mechanical Turk. In _Proceedings of the ACM SIGKDD Workshop on Human Computation_ (Washington DC) _(HCOMP ’10)_. Association for Computing Machinery, New York, NY, USA, 64–67.  https://doi.org/10.1145/1837885.1837906
  * Jiang et al. (2020) Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020\.  How Can We Know What Language Models Know?  _Transactions of the Association for Computational Linguistics_ 8 (2020), 423–438.  https://doi.org/10.1162/tacl_a_00324
  * Kale and Rastogi (2020a) Mihir Kale and Abhinav Rastogi. 2020a.  Template Guided Text Generation for Task-Oriented Dialogue. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Association for Computational Linguistics, Online, 6505–6520.  https://doi.org/10.18653/v1/2020.emnlp-main.527
  * Kale and Rastogi (2020b) Mihir Kale and Abhinav Rastogi. 2020b.  Text-to-Text Pre-Training for Data-to-Text Tasks. In _Proceedings of the 13th International Conference on Natural Language Generation_. Association for Computational Linguistics, Dublin, Ireland, 97–102.  https://aclanthology.org/2020.inlg-1.14
  * Ke et al. (2022) Pei Ke, Hao Zhou, Yankai Lin, Peng Li, Jie Zhou, Xiaoyan Zhu, and Minlie Huang. 2022\.  CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generation. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Dublin, Ireland, 2306–2319.  https://doi.org/10.18653/v1/2022.acl-long.164
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R. Varshney, Caiming Xiong, and Richard Socher. 2019\.  CTRL: A Conditional Transformer Language Model for Controllable Generation.  _CoRR_ abs/1909.05858 (2019).  arXiv:1909.05858 http://arxiv.org/abs/1909.05858
  * Khalifa et al. (2021) Muhammad Khalifa, Hady Elsahar, and Marc Dymetman. 2021\.  A Distributional Approach to Controlled Text Generation. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=jWkw45-9AbL
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_ (2013).
  * Kitaev et al. (2020) Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020\.  Reformer: The Efficient Transformer. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=rkgNKkHtvB
  * Krause et al. (2021) Ben Krause, Akhilesh Deepak Gotmare, Bryan McCann, Nitish Shirish Keskar, Shafiq Joty, Richard Socher, and Nazneen Fatema Rajani. 2021.  GeDi: Generative Discriminator Guided Sequence Generation. In _Findings of the Association for Computational Linguistics: EMNLP 2021_. Association for Computational Linguistics, Punta Cana, Dominican Republic, 4929–4952.  https://doi.org/10.18653/v1/2021.findings-emnlp.424
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021\.  The Power of Scale for Parameter-Efficient Prompt Tuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 3045–3059.  https://doi.org/10.18653/v1/2021.emnlp-main.243
  * Li et al. (2016a) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016a.  A Diversity-Promoting Objective Function for Neural Conversation Models. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. Association for Computational Linguistics, San Diego, California, 110–119.  https://doi.org/10.18653/v1/N16-1014
  * Li et al. (2016b) Jiwei Li, Michel Galley, Chris Brockett, Georgios Spithourakis, Jianfeng Gao, and Bill Dolan. 2016b.  A Persona-Based Neural Conversation Model. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Berlin, Germany, 994–1003.  https://doi.org/10.18653/v1/P16-1094
  * Li et al. (2020b) Piji Li, Haisong Zhang, Xiaojiang Liu, and Shuming Shi. 2020b.  Rigid Formats Controlled Text Generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (Eds.). Association for Computational Linguistics, 742–751.  https://doi.org/10.18653/v1/2020.acl-main.68
  * Li et al. (2020a) Shifeng Li, Shi Feng, Daling Wang, Kaisong Song, Yifei Zhang, and Weichao Wang. 2020a.  EmoElicitor: An Open Domain Response Generation Model with User Emotional Reaction Awareness. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_ , Christian Bessiere (Ed.). International Joint Conferences on Artificial Intelligence Organization, 3637–3643.  https://doi.org/10.24963/ijcai.2020/503 Main track.
  * Li and Liang (2021) Xiang Lisa Li and Percy Liang. 2021.  Prefix-Tuning: Optimizing Continuous Prompts for Generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 4582–4597.  https://doi.org/10.18653/v1/2021.acl-long.353
  * Li et al. (2022) Xiang Lisa Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto. 2022\.  Diffusion-LM Improves Controllable Text Generation.  _ArXiv_ abs/2205.14217 (2022).  https://arxiv.org/abs/2205.14217
  * Liao et al. (2019) Yi Liao, Yasheng Wang, Qun Liu, and Xin Jiang. 2019\.  Gpt-based generation for classical chinese poetry.  _arXiv preprint arXiv:1907.00151_ (2019).  https://arxiv.org/abs/1907.00151
  * Lin (2004) Chin-Yew Lin. 2004\.  ROUGE: A Package for Automatic Evaluation of Summaries. In _Text Summarization Branches Out_. Association for Computational Linguistics, Barcelona, Spain, 74–81.  https://aclanthology.org/W04-1013
  * Lin et al. (2021) Zhaojiang Lin, Andrea Madotto, Yejin Bang, and Pascale Fung. 2021.  The Adapter-Bot: All-In-One Controllable Conversational Model. In _Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021_. AAAI Press, 16081–16083.  https://ojs.aaai.org/index.php/AAAI/article/view/18018
  * Lin and Riedl (2021) Zhiyu Lin and Mark O Riedl. 2021.  Plug-and-Blend: A Framework for Plug-and-Play Controllable Story Generation with Sketches. In _Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment_ , Vol. 17. 58–65.  https://arxiv.org/abs/2104.04039
  * Liu et al. (2021c) Alisa Liu, Maarten Sap, Ximing Lu, Swabha Swayamdipta, Chandra Bhagavatula, Noah A. Smith, and Yejin Choi. 2021c.  DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Experts. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 6691–6706.  https://doi.org/10.18653/v1/2021.acl-long.522
  * Liu et al. (2010) Chang Liu, Daniel Dahlmeier, and Hwee Tou Ng. 2010\.  TESLA: Translation Evaluation of Sentences with Linear-Programming-Based Analysis. In _Proceedings of the Joint Fifth Workshop on Statistical Machine Translation and MetricsMATR_. Association for Computational Linguistics, Uppsala, Sweden, 354–359.  https://aclanthology.org/W10-1754
  * Liu and Gildea (2005) Ding Liu and Daniel Gildea. 2005.  Syntactic features for evaluation of machine translation. In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_. 25–32.
  * Liu et al. (2020b) Danyang Liu, Juntao Li, Meng-Hsuan Yu, Ziming Huang, Gongshen Liu, Dongyan Zhao, and Rui Yan. 2020b.  A character-centric neural model for automated story generation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 1725–1732.  https://ojs.aaai.org//index.php/AAAI/article/view/5536
  * Liu et al. (2019a) Feng Liu, Qirong Mao, Liangjun Wang, Nelson Ruwa, Jianping Gou, and Yongzhao Zhan. 2019a.  An emotion-based responding model for natural language conversation.  _World Wide Web_ 22, 2 (2019), 843–861.  https://doi.org/10.1007/s11280-018-0601-2
  * Liu et al. (2021d) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021d.  Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.  _CoRR_ abs/2107.13586 (2021).  arXiv:2107.13586 https://arxiv.org/abs/2107.13586
  * Liu et al. (2021a) Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. 2021a.  Mitigating political bias in language models through reinforced calibration. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 35. 14857–14866.  https://arxiv.org/abs/2104.14795
  * Liu et al. (2020c) Ruibo Liu, Guangxuan Xu, Chenyan Jia, Weicheng Ma, Lili Wang, and Soroush Vosoughi. 2020c.  Data Boost: Text Data Augmentation Through Reinforcement Learning Guided Conditional Generation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020_ , Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 9031–9041.  https://doi.org/10.18653/v1/2020.emnlp-main.726
  * Liu et al. (2020a) Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020a.  Multilingual Denoising Pre-training for Neural Machine Translation.  _Transactions of the Association for Computational Linguistics_ 8 (2020), 726–742.  https://doi.org/10.1162/tacl_a_00343
  * Liu et al. (2021b) Ye Liu, Wolfgang Maier, Wolfgang Minker, and Stefan Ultes. 2021b.  Naturalness Evaluation of Natural Language Generation in Task-oriented Dialogues using BERT.  _arXiv preprint arXiv:2109.02938_ (2021).
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_ abs/1907.11692 (2019).  arXiv:1907.11692 http://arxiv.org/abs/1907.11692
  * Logeswaran et al. (2018) Lajanugen Logeswaran, Honglak Lee, and Samy Bengio. 2018\.  Content preserving text generation with attribute controls. In _Advances in Neural Information Processing Systems_ , S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.), Vol. 31. Curran Associates, Inc.  https://proceedings.neurips.cc/paper/2018/file/7cf64379eb6f29a4d25c4b6a2df713e4-Paper.pdf
  * Lowe et al. (2017) Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017.  Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics, Vancouver, Canada, 1116–1126.  https://doi.org/10.18653/v1/P17-1103
  * Lubis et al. (2018) Nurul Lubis, Sakriani Sakti, Koichiro Yoshino, and Satoshi Nakamura. 2018.  Eliciting Positive Emotion through Affect-Sensitive Dialogue Response Generation: A Neural Network Approach. In _Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018_ , Sheila A. McIlraith and Kilian Q. Weinberger (Eds.). AAAI Press, 5293–5300.  https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/16317
  * Luo et al. (2019) Fuli Luo, Damai Dai, Pengcheng Yang, Tianyu Liu, Baobao Chang, Zhifang Sui, and Xu Sun. 2019.  Learning to Control the Fine-grained Sentiment for Story Ending Generation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 6020–6026.  https://doi.org/10.18653/v1/P19-1603
  * Malandrakis et al. (2019) Nikolaos Malandrakis, Minmin Shen, Anuj Goyal, Shuyang Gao, Abhishek Sethi, and Angeliki Metallinou. 2019.  Controlled Text Generation for Data Augmentation in Intelligent Artificial Agents. In _Proceedings of the 3rd Workshop on Neural Generation and Translation_. Association for Computational Linguistics, Hong Kong, 90–98.  https://doi.org/10.18653/v1/D19-5609
  * Mathur et al. (2019) Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019\.  Putting Evaluation in Context: Contextual Embeddings Improve Machine Translation Evaluation. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. Association for Computational Linguistics, Florence, Italy, 2799–2808.  https://doi.org/10.18653/v1/P19-1269
  * Mazaré et al. (2018) Pierre-Emmanuel Mazaré, Samuel Humeau, Martin Raison, and Antoine Bordes. 2018.  Training Millions of Personalized Dialogue Agents. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun’ichi Tsujii (Eds.). Association for Computational Linguistics, 2775–2779.  https://doi.org/10.18653/v1/d18-1298
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013\.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_ (2013).