# Neural Conversation Models and How to Rein Them in:
A Survey of Failures and Fixes

Fabian Galetzka<EMAIL_ADDRESS>
Computational Linguistics
University of Potsdam
Anne Beyer<EMAIL_ADDRESS>
Computational Linguistics
University of Potsdam
David Schlangen<EMAIL_ADDRESS>
Computational Linguistics
University of Potsdam

###### Abstract

Recent conditional language models are able to continue any kind of text
source in an often seemingly fluent way. This fact encouraged research in the
area of “open-domain” conversational systems that are based on powerful
language models and aim to imitate an interlocutor by generating appropriate
contributions to a written dialogue. From a linguistic perspective, however,
the complexity of _contributing to a conversation_ is high. In this survey, we
interpret Grice’s maxims of cooperative conversation from the perspective of
this specific research area and systematize the literature under the aspect of
what makes a contribution _appropriate_ : A neural conversation model has to
be fluent, informative, consistent, coherent, and follow social norms. In
order to ensure these qualities, recent approaches try to “tame” the
underlying language models at various _intervention points_ , such as data,
training regime or decoding. Sorted by these categories and intervention
points, we discuss promising attempts and suggest novel ways for future
research.

This review was started in 2021 and was largely finalised in 2022. The present
version represents a lightly edited variant of a chapter from the first
author’s PhD thesis (?). Note that the article hence does not cover any
material newer than 2022. We believe, however, that at least the systematic
presentation of the challenges still has value, even if in the current state
of the art, not all of the intervention points are still equally accessible.

##  1 Introduction

With the advent of powerful conditional language models, the idea of building
“open-domain” conversational systems that can simulate casual conversation
about anything and everything has seen a resurgence as a respectable research
goal (rather than being left to the pursuit of hobbyists).

Two recent developments have contributed to this. First, with the increasing
popularity and use of social media, ever larger amounts of conversational data
(e.g., posts and questions and answer threads) have become easily available. ?
(?) was an early example of how these data can be made useful for the task of
modelling conversations; in this case using techniques from statistical
machine translation to “translate” utterances into responses to them. In
parallel to this, statistical language models (and therefore conditional
language generators) have become more powerful due to new architectures like
Self-Attention (?), an increase in raw computational power, and, with this, an
increase in the size of the trainable models.

Early intuitions suggested that these ingredients might be enough to build
open domain conversational systems. Even though first experiments with such
models yielded surprisingly fluent and seemingly fitting responses given short
contexts, these hopes were quickly dashed with the subsequent observation of
many problems, such as an overuse of uninformative answers (“I don’t know”),
inconsistent personality projections (?), or generally missing context
awareness (?). After the early enthusiasm, to a certain degree a realisation
of the complexity of the task set in, answering, as it does, to a variety of
constraints from both outside the discourse (accuracy of factual claims) and
inside of it (needing to fit to what has previously been said).

In this paper, we attempt to systematise the literature about the attested
problems of neural conversation models (conditional language models realised
with neural networks) used as chat-partner simulators, and approaches to
addressing these. We show that these approaches can be framed as attempts to
“tame” the underlying model, in the sense that they impose further constraints
on it. For this, we identify different targets or intervention points, such as
data, training regime, decoding, etc. We show that the constraints can be
related to observations from the field of pragmatics, and more specifically to
Grice’s maxims of cooperative conversation (?).111? (?) similarly take
inspiration from Grice’s cooperativity principle in their taxonomy of chat-bot
error types, but do not relate these specifically to neural models and
attempts at improving those.

Finally, we show that this relation can be made productive, by suggesting, for
future work, novel ways to address problems.

This paper aims to provide both a historical overview of developments in this
field as well as serve as a catalogue of the various ways of improving models
that have been explored, and through this, to be of use to researchers looking
for ways to improve their models.

##  2 Contributing to a Conversation

A first step towards understanding what properties a neural conversation model
needs to have to carry on any conversation appropriately, we start with
analyzing what a contribution to a conversation is first.

###  2.1 The Task, Formalized

The task of participating in a conversation can be seen as identifying the
contextually most appropriate contribution $c_{t}^{*}$ at turn $t$ out of all
possible contributions $c$:

$c_{t}^{*}=\mbox{most\\_appropriate}_{c}\;R(c|C_{t}).$ (1)

with $R$ being a conversation model that generates contributions $c$ given a
context $C_{t}$. In order to determine a contextually appropriate
contribution, $C_{t}$ needs to include at least the observable public context,
which is the dialogue history $D_{t}=(o_{1},c_{1},...o_{t})$. It consists of
observations $o_{i}$ (contributions by the dialogue partner) and own
(previous) contributions $c_{i}$ until the current speaker turn $t$.

However, the observable context is not sufficient. Additionally, a non-
observable internal state is needed, otherwise any latent variables that
control the internal consistency within $c_{i}$ must be fully expressed in the
observable information. This dimension of consistency can be achieved by
providing additional private information $K$ to the model. $K$ can contain
factual information (about the world in which the conversation is taking
place) or personal information, for example stylistic parameters (make
utterances conform in style to the personality ”40-year-old male accountant”)
or proffered information (make self-claims consistent with ”40-year-old male
accountant”). The full formulation of the context $C_{t}$ then is:

$C_{t}=(D_{t},K)\mbox{ with }K=\\{p_{1},\dots,p_{n},f_{1},\dots,f_{m}\\},$ (2)

where the $p_{i}$ are the persona facts and the $f_{i}$ the general facts.

We further need to clarify the exact meaning of “appropriateness” to the
public and private context $C_{t}$ at turn $t$. For this, we introduce Grice’s
model of cooperative conversation in the following Subsection 2.2 and adapt it
to the research area of neural generation models in Subsection 2.3.

###  2.2 The Cooperative Principle

The philosopher H.P. Grice proposed in 1975 in his seminal paper “Logic and
Conversation” the adherence to the _Coorperative Principle_ as a fundamental
feature of contribution to discourse:

> Make your conversational contribution such as is required, at the stage at
> which it occurs, by the accepted purpose or direction of the talk exchange
> in which you are engaged. (?, p. 45)

This theoretical proposal attempts to explain certain aspects of discourse
meaning, where meaning can also come from apparently violating these maxims.
The proposal has been enormously influential in linguistic pragmatics and has
also been successfully used in Natural Language Processing (e.g., ? (?)),
which motivates us to take it as our starting point as well. He lists nine
conversational maxims, sorted into four categories, that need to be followed
to make a contribution “such as is required”.

Figure 1: Operationalising the Conversational Maxims (?) as Demands on
Conversation Models.

The category Manner includes the supermaxim ‘Be perspicuous’ and relates not
to what is said, but how it is said. Grice listed the maxims:

  1. 1.

Avoid obscurity of expression.

  2. 2.

Avoid ambiguity.

  3. 3.

Be brief.

  4. 4.

Be orderly.

The category Quantity relates to the right amount of information that is
required in a conversation. Grice formulated the following maxims:

  1. 5.

Make your contribution as informative as is required (for the current purposes
of the exchange).

  2. 6.

Do not make your contribution more informative than is required.

The category Quality includes the supermaxim ‘Try to make your contribution
one that is true’ with the two maxims:

  1. 7.

Do not say what you believe to be false.

  2. 8.

Do not say that for which you lack adequate evidence.

The category Relation is only defined by the maxim:

  1. 9.

Be relevant.

The formulation of what this entails is not straightforward for Grice, but he
relates it to the focus on topics within a conversation, and how they can be
reasonably shifted.

Figure 1 on the left shows the Cooperative Principle summarized. Grice notes
that this list is not necessarily complete, and he proposes “Be polite” as a
possible addition. We follow up on this as a fifth category in the next
Subsection 2, where we try to operationalize these maxims as measurable
features for neural generation models.

###  2.3 Appropriateness as Conversational Cooperativity

A neural conversation model should produce grammatical correct and not
semantically malformed contributions. In the literature this is often referred
to as _fluent_. In the category Manner Grice proposed that a conversation has
to be clear and understandable to allow for a meaningful exchange. We
introduce Fluency as our first neural conversation model related category.
Fluency of a model can be approximated conveniently by calculating perplexity
on a test set. Perplexity measures the probability with which a model would
have generated the test data (normalized for length). Under the assumption
that the training data are formally (or grammatically) fluent, low perplexity
should be a good indicator for the model’s ability to account for the
syntactic structure of the dialogues and generated utterances (?). Therefore,
we can assume that fluency is achieved, if the unconditional probability
assigned to the contribution is high. With that definition, fluency is a
property that can be approximated without taking the context $C_{t}$ into
account:

$m_{\textrm{fluency}}(c_{t}|C_{t})\sim f(c_{t}),$ (3)

with $f$ being the probability function realized by the model proper. We
discuss fluency problems and survey interventions in Subsection 5.1.

According to Grice’s category Quantity a contribution should neither be more
nor less informative than necessary. We operationalize that as Informativity
for neural conversation models and separate that into two dimensions:
Genericity, and Repetitiveness. We define a contribution $c_{t}$ as generic,
if it fits equally well to many contexts $C_{t}^{\prime}$, which should be
avoided:

$m_{\textrm{inf,gen}}(c_{t}|C_{t})\sim\frac{1}{|\mathcal{C}|}\sum_{C_{t}^{\prime}\in\mathcal{C}}\frac{f(c_{t},C_{t})}{f(c_{t},C_{t}^{\prime})},$
(4)

where $f$ is the model proper, and $\mathcal{C}$ is a set of randomly sampled
other contexts. A constructed example would be the following contribution from
$P_{B}$:

. $P_{a}$: Do you like movies?
$P_{b}$: # I don’t know!

It is a fluent reply, but not informative, as it is, presumably, an equally
good reply in many contexts. There are not well established informativity
metrics for neural conversation models in the literature. ? (?) and ? (?) used
token distributions and rare word counts as metrics in their experiments.
Under the assumption that generic contributions contain generic (high
frequent) tokens, this might be a way of quantifying informativity for
conversation models. Informativity could also be measured by comparing the
conditional and the unconditional contribution probability, as specific
context should not make generic contributions more probable.
Repetitiveness, the second dimension of informativity, measures whether
$c_{t}$ repeats information from $C_{t}$:

$m_{\textrm{inf,2}}(c_{t}|C_{t})\sim f(c_{t},C_{t})$ (5)

where $f$ can be simple (sub)string match. If $c_{t}$ repeats content from
$C_{t}$ no new information is contributed to the conversation. For example:

. $P_{a}$: Pulp Fiction was released in 1994.
$P_{b}$: # Did you know that the movie was released in 1994?

Counting repeating tokens or $n$-grams is a way to detect repetition. However,
one can repeat content without using the same tokens. Therefore, word
embedding comparison metrics might be more robust. We discuss informativity
problems and survey interventions in Subsection 5.2.

The Gricean category of Quality suggests contributing with true statements.
Therefore, models need to have some representation of what is true. We
operationalize this as Consistency towards additional knowledge. Recent
approaches can be categorized into two main directions: adding additional
facts by providing a knowledge base and giving the model an implicit
personality by adding descriptive persona information. We define $c_{t}$ as
consistent with a knowledge base if it represents one or more facts $f_{i}$
correctly:

$m_{\textrm{consistency,facts}}(c_{t}|C_{t})\sim f(c_{t},f_{i}),$ (6)

and consistent with a given personality if $c_{t}$ is consistent with every
$p_{i}$ of the personality description:

$m_{\textrm{consistency,persona}}(c_{t}|C_{t})\sim f(c_{t},p_{i}).$ (7)

The function $f$ can be a neural model that predicts fact and opinion
entailments, similar to natural language inference approaches. There are no
precise and established metrics in the literature for consistency. Evaluation
has to be done with humans, which makes it difficult to compare the different
approaches. Additionally, human evaluations are expensive and time-consuming,
limiting the number of experiments that researchers can do, such as
hyperparameter searches. Natural language inference models fine-tuned on
persona or fact and utterance pairs might be a solution, but need to be
explored.

In the category Relation, Grice proposes that a dialogue contribution should
“be relevant” and therefore follow some vaguely defined relevance criteria
related to the subject of a conversation. We operationalize that as Coherence.
We define $c_{t}$ as coherent if it has a pragmatic relation to $C_{t}$,
specifically, to the dialogue history $D_{t}$:

$m_{\textrm{coherence}}(c_{t}|C)\sim
f(c_{t}|c_{t-1},...c_{1},o_{t},...o_{1}).$ (8)

A fluent, informative and consistent contribution that is not coherent given
the dialogue context could be:

. $f_{1}$: Uma Thurman is an actress.
$P_{a}$: When was Pulp Fiction released?
$P_{b}$: # Uma Thurman is a very popular actress.

No automated metrics have yet been proposed that cover these complex notions
of dialogue (?). Again, evaluation dialogue coherence requires manual
evaluation, which causes the same problems mentioned previously. We discuss
coherence problems and survey interventions in Subsection 5.4.

Not derived from the four proposed categories from Grice, but perhaps on the
statement: “Be polite!”, a neural conversation model has to follow several
rules (the social norm) to not be offensive to the user of that system or any
other third person in general. We propose that a contribution $c_{t}$ follows
the social norm, if $c_{t}$ can be seen as non-offensive by the dialogue
partner or by any other third person who knows $C_{t}$ and listens to $c_{t}$.
Therefore, in addition to the full context $C_{t}$ a set of social rules $R$
is needed to determine good social norm:

$m_{\textrm{social norm}}(c_{t}|C_{t})\sim f(c_{t},C_{t},K,R).$ (9)

? (?) categorized harmful contributions of neural conversation models into
three different types: Generating $c_{t}$ that are harmful by themselves
because they contain offensive language:

. $P_{b}$: # I hate you!

Generating $c_{t}$ as agreements to previous contributions $o_{i}$ that were
harmful: . $P_{a}$: I hate him!
$P_{b}$: # I agree with you!

And lastly, generating $c_{t}$ that contain inappropriate advices in safety-
critical situations:

. $P_{a}$: Should I jump down this cliff?
$P_{b}$: # I think this is a good idea!

We discuss social norm problems and survey interventions in Subsection 5.5. We
sum up the operationalization of Grice’s categories in Figure 1 on the right-
hand side.

##  3 Research History and Problems

Before we discuss the possible intervention points for neural conversation
models in Section 4 and survey the literature that addressed the above demands
in Section 5 we briefly go over the history and encountered problems in that
area first.

? (?) were the first who transferred the task of phrase-based machine
translation to the field of dialogue generation. Instead of translating, they
reformulated the task to transform an utterance into an appropriate reply.
They identified two important challenges: First, unlike in bilingual machine
translation, a reply is not semantically equivalent to the utterance. This
leads to a wider range of possible responses, and being fluent is more
challenging. Secondly, a reply has more unaligned words compared to bilingual
pairs and often parts of the stimulus utterance are not referenced in the
reply at all. To be fluent the model has to learn to reference correctly and
to use different tokens in a clear and understandable way. They used the
messaging platform Twitter 222https://twitter.com to get utterance-reply pairs
by scraping. They collected a total of 1.3 million pairs, such as:

. $P_{a}$ (utterance): I’m slowly making this soup ….. and it smells gorgeous!
$P_{b}$ (reply): I’ll bet it looks delicious too! Haha
(Data was taken from ? (?))

The authors observed that in a natural conversation utterance-reply pairs have
strong structural relationships, as shown in the example above. Their task can
be expressed in our formulation as:

$c_{t}=\textrm{most\\_appropriate}_{c}(R(c|o_{t})).$ (10)

At this point, the goal was to find an appropriate reply (given an utterance)
and not to carry on a conversation. In their evaluation, replies generated by
the statistical machine translation approach were preferred by humans over
those yielded by information retrieval approaches; in 15% of the cases,
participants even preferred them over gold answers from the dataset.

From here many improvements were made regarding data sources and model
capacities, and research was motivated by the success of being able to
generate contributions that seem to be fluent in themselves (or at least
grammatically appropriate). ? (?) argued that having one utterance (in our
formalization, just $o_{t}$) is not sufficient for generating an appropriate
response ($c_{t}$) in a longer exchange, as relevant information can be placed
earlier in the context. Therefore, they created utterance triples by scraping
Twitter again, for example:
. $P_{a}$ (context): because of your game ?
$P_{b}$ (message): yeah I’m on my way now
$P_{a}$ (reply): ok good luck !
(Data was taken from (?))

Their approach can be expressed as:

$c_{t}=\textrm{most\\_appropriate}_{c}(R(c|c_{t-1},o_{t})).$ (11)

Being closer to a real conversation, but still having the property of a static
conversation history. The new utterance $c_{t-1}$ in the dialogue history led
them to develop a new model architecture, since the machine translation
sequence-to-sequence models don’t allow a differentiation between speakers.
Their new recurrent language model consists of two separate Recurrent Neural
Network (RNN) cells to encode $c_{t-1}$ and $o_{t}$. They could show in a
human evaluation that participants preferred replies generated with this
approach over replies from the previous machine translation approach.

The first step towards variable sized dialogue histories was done by ? (?)
with the introduction of a new hierarchical RNN structure. One RNN cell is
supposed to encode an utterance into a fixed-sized utterance representation,
and another RNN cell encodes these representations into the final hidden
(fixed-sized) state. This way the architecture allows variable dialogue
history lengths, which was an important step towards modern neural
conversation models. However, in their experiments they still used triples for
training by processing the Movie-DIC dataset (?) into 245k three turn dialogue
samples.

? (?) also showed that pre-training greatly reduces model perplexity (a
measure of quality of the model, where lower numbers are better; see below):
They used 5.5 million utterance-reply pairs from the SubTle corpus (?) before
fine-tuning and were able to improve perplexity from 35.63 to 27.09 on the
static RNN model and to 26.81 on the hierarchical RNN architecture.

Later, ? (?) showed that architectures solely based on Self-Attention layers
(proposed as the Transformer model) outperform recurrent neural networks in
machine translation, and ? (?) introduced the Generative Pre-trained
Transformer (GPT) model (and their successor GPT-2). Most recent approaches
make use of these pre-trained Transformer-based models. In the ConvAI2
challenge (?), researchers were tasked to build the best conversation model on
the PersonaChat dataset (?). The winning model by ? (?) was a GPT model fine-
tuned on the dialogue dataset. They trained and added additional embeddings to
differentiate between the persona profiles and the dialogue history. The model
reached a perplexity of 16.28 on the unseen test set, compared to 29.01
perplexity for sequence-to-sequence models based on recurrent neural networks.
Transformer-based models have high capacity, but do not work well without the
generative pre-training. However, with unsupervised pre-training they appear
to learn the structure of language and word embeddings well, which makes them
superior compared to the previous approaches.

The described improvements of language generation had an important impact on
neural conversation models, since they appear to be fluent (or at least
grammatically appropriate), which is important. But that alone is not
sufficient for a good conversation, as discussed above. Lately, a large body
of research has also accumulated investigating how to improve informativity,
consistency, coherence and adherence to social norm, and not only by improving
model architectures, but through interventions at a whole range of entry
points around the base language model and along the processing pipeline. In
the following section we go over these, as we call them, intervention points.

##  4 Intervention Points

Figure 2: Intervention points for developing current non-goal driven neural
conversation models.

Figure 2 illustrates the schematic organization of the neural conversation
models that are our topic here, in the context of both their training and
their application. All the elements listed in the figure (and further
described below) have been investigated as dimensions of change for improving
the appropriateness of the productions of the model as a whole, as we will see
in the next sections.

The core is formed by what we label here the model proper, i.e., the
conditional language model $P(c|C)$. These sequence-to-sequence architectures
encode textual information (here dialogue history $D_{t}$ and optional private
information $K$) into a hidden representation, which is transformed by a
decoder into a new contribution $c_{t}$. They are either based on Self-
Attention layers (?) in various architectures or on RNNs (?, ?). They are
designed to transform a sequence of fixed-sized input representations (here
word embeddings) into a fixed-sized latent representation (encoder) or to
generate fixed-sized output representations (here probability distributions
over the vocabulary) based on a latent representation (decoder).

Although their task is to produce dialogue contributions, these models are
typically pre-trained on (monological) text material, here labelled pre-
training data. Transformer-based architectures require pre-training but RNN-
based approaches also benefit from it, as shown by ? (?). Pre-training is
supposed to train word embeddings and general language modeling abilities.
This has been proven to be very effective for many NLP tasks (?). Thereafter,
models were fine-tuned on dialogue data $D_{t}$ and (optionally) context $K$,
often called domain data in the literature. Given the way the required data is
collected, namely automatic or crowdsourced and often without good curation
practices, rule-based filtering can be applied to remove material from which a
model should not induce patterns.

The common machine learning approach is to optimize model parameter $\Theta$
on a tokenized corpus
$\mathbf{X}=\\{\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{n}\\}$ by
minimizing the _loss function_ , usually negative log likelihood. Using
reinforcement learning is a rather new approach in neural dialogue modelling
and makes possible full sequence (contributions) sampling and the use of non-
differentiable loss functions, the _reinforcement learning objectives_.

Decoding itself is an intervention point, as it has a high impact on the
generated contribution. This is particularly the case for models trained with
the language modelling loss, as this training regime cannot account for error
accumulation when generating sequences. Decoding is not only about finding the
best beams but also to control the generated sequence with different rules, as
we will show in the next section.

Finally, the last intervention point we are discussing here is to apply a
post-filter. Based on specific rules, fully generated candidate contributions
can be rejected or ranked to improve the conversation model.

##  5 Survey of Proposed Interventions

Now, in this section, we survey approaches from the literature that tried to
improve neural generation models regarding our proposed operationalized
categories, sorted by the intervention points. For each category, we start
with examples of problematic model output from the literature and then discuss
how these problems have been addressed.

###  5.1 Manner (Clarity) // Fluency

Language models can produce quite fluent utterances and _fluency_ improved
over the years significantly, as described in our historical overview in
Section 3, through improvements in model size and architecture as well as the
availability of larger training datasets. One additional issue that causes low
fluency is that decoding strategies that optimize for sequences with high
probability tokens (such as the original beam search algorithm) produce
generic and repetitive output (?). Language models tend to compute the highest
probabilities to these sequences. They found that the probability for
repeating phrases increases with each repetition, which creates a positive
feedback loop and that text generated by humans has much lower probabilities,
compared to text generated by beam search. This variance appears to be
important in order to produce fluent output, and therefore different decoding
strategies and various rules of thump were explored and are used in modern
neural conversation models to reduce repetition and try to control sequence
lengths.

#### Pre-training Data

Model size and pre-training data are important factors to decrease perplexity
and therefore increase the fluency. We already listed Important pre-training
datasets in our historical overview in Section 3.

#### Decoding

Different decoding strategies prevent repetition by blocking repeating tokens
or by adding randomness to that process. Randomness reduces the overall
likelihood of the generated sequences and therefore repetition and since this
is closely related to _informativity_ , we will discuss that in the next
section. To maintain fluency it is important to block token-based repetition,
as loops make utterances unfluent. ? (?) evaluated decoding strategies in the
context of neural conversation generation. To avoid loops they explicitly
block tokens or n-grams that already occurred in the generated sequence. That
means at every decoding step, regardless of the estimated token probabilities
by the model, tokens that would cause a repeated 3-gram either in $o_{t}$ or
$c_{t}$ are ignored. The idea of beam blocking originally came from ? (?) and
can be applied to any decoding algorithm.

#### Objective Function

The root cause of repetition might be the training objective itself, since
assigning high probabilities for frequently occurring tokens is done by the
model itself. Unlikelihood training (?) is a way of improving the objective
function in several ways. It reduces the gap between training (next-token
prediction only) and inference (generating full sequences) through an
additional inverse loss $\lambda_{\textsc{UL}}$, that is added to the usual
maximum likelihood estimation loss $\lambda_{\textsc{MLE}}$:

$\lambda_{\textsc{ULE}}=\lambda_{\textsc{MLE}}+\lambda_{\textsc{UL}},$ (12)

and is called the unlikelihood loss for step $t$ with a set of negative
candidate tokens $C^{t}$:

$\lambda_{\textsc{UL}}(p_{\theta}(\cdot|x_{<t}),C^{t})=-\sum_{c\epsilon
C^{t}}{\textrm{log}(1-p_{\theta}}(c|x_{<t})).$ (13)

The negative candidates $C^{t}$ can be set to anything that causes non-
appropriate contributions. We will follow up on this approach in later
sections as well. For _fluency_ it is important to avoid repetition. The
authors therefore added negative candidates with repeating n-grams according
to the following scheme:

$C_{t}^{\mathrm{contextcopy}}=\begin{cases}\\{y_{t}\\}&y_{t}\epsilon\mbox{repeating
context}\\\ \emptyset&\mbox{otherwise,}\end{cases}$ (14)

where $y_{t}$ is a token in a repeating context n-gram. This should reduce the
probability of tokens that form an n-gram that is already been decoded at
inference.

#### Reinforcement Learning

Reinforcement learning allows the use of non-differentiable functions and some
are supposed to tackle repetition. ? (?) introduced a low repetition reward by
counting occurrences of 1-grams to compute the following token ratio:

$\textrm{R}_{\textrm{rep}}=1-\frac{\textrm{\\# repeated-tokens-in-
response}}{\textrm{\\# tokens-in-response}}$ (15)

The reward function measures repeated tokens and since the reward improves
over the training, their approach seems to work. Unfortunately, they don’t
compare repetition between both models explicitly. The human evaluation
focused on semantic plausibility and factual consistency, which we will
discuss in future sections, and also didn’t cover repetition.

###  5.2 Quantity // Informativity

Neural conversation models trained on large dialogue corpora with standard
language modelling loss tend to generate “simple, short and sometimes
unsatisfying answers” and predicting the next word might be an
oversimplification (?). Similarly, ? (?) observed that responses generated by
generative conversation models with the highest average probability log-
likelihoods tend to be trivial or generic, like I don’t know! or I’m OK. The
authors stated that models assign high probability to generic responses, if
they are optimized for the likelihood, as they are suitable for many contexts.
? (?) observed the same behavior and offered several explanations: Data
scarcity could cause the models to only predict the most frequent utterances.
The high number of punctuation mark tokens in the data set makes it hard for
the model to learn and predict diverse utterances, since the error signal is
dominated by these frequent tokens. And the dialogue context could be too
short, and more dialogue history is needed to diversify generated content.

More recently, ? (?) observed that per-token probability of text created by
humans is on average lower than text generated by language models, meaning
that human-generated text is more surprising and models trained on minimizing
negative log-likelihood on next-token prediction don’t or cannot learn the
real token distribution of human text. This problem is more general and not
only tied to neural conversation models. The objective function, which only
takes one true next-token into account, cannot capture the diversity of
natural language or, specifically for this research area, the diversity of
natural conversations. Prioritizing high probability tokens makes text less
surprising and therefore less informative, as such probability distributions
have lower entropy on average. Minimizing the cross-entropy loss causes the
model to avoid making mistakes and because bland utterances fit to many
contexts, it is less likely that a bland utterance is inappropriate. Current
research focus on improving the objective function, better models and a lot on
decoding algorithms.

#### Objective Function

To improve informativity better objective functions are needed. Unlikelihood
training (?, ?) also contributes to informativity: the authors introduced a
loss based on the frequency of occurring tokens:

$\beta(y_{c})=p_{\mathrm{model}}(y_{c})\mbox{log}(\frac{p\mathrm{model}(y_{c})}{p_{*}(y_{c})}),$
(16)

with $p_{\mathrm{model}}(y_{c})$ being the model’s unigram distribution of the
candidate $y_{c}$ and ${p_{*}(y_{c}})$ the pre-estimated human distribution.
For these candidates, they use earlier contributions to explicitly punish
repetition and tokens with large distribution difference to punish frequent
(and therefore generic) tokens. This causes the negative candidate token
probabilities to decrease and therefore reduces the probability of sequences
with these candidates. ? (?) reported a higher rare words cumulative mass with
increasing unlikelihood objective scaling in their experiments using that
approach compared to the usual language modelling approach. They further
showed that this metric also correlates with human judgement (participants had
to choose between two models through the question: ”Which response answers the
question better?”). Because of this, we see an indicator for token
distribution characteristics being an important part of quantifying
informativity. ? (?) observed similar success: They showed that using this
loss in order to decrease probabilities of inappropriate tokens, it increases
the number of unique tokens generated by the model on the test set from 11.8k
to 13.8k. Compared to a human reference of 19.8k unique tokens, this is still
significantly worse, but a promising step towards more informative
contributions.

#### Model

The SpaceFusion model by ? (?) tries to increase relevance and
interestingness. The latter correlates with informativity, since bland and
generic content should be less interesting. Their idea is to spread the
response distribution across a latent space to cluster them along diversity
directions. In each direction, the distance from the true response
additionally identifies the relevance. Two model parts are trained
simultaneously: a sequence-to-sequence encoder (generating response
representations $z_{\mathrm{s2s}}$) and an auto-encoder (generating target
response representations $z_{\mathrm{AE}}$). Both share the same decoder. The
loss is based on the interpolation
$z_{\mathrm{interp}}=uz_{\mathrm{s2s}}+(1-u)z_{\mathrm{AE}}$ with $u\sim
U(0,1)$:

$\lambda_{\mathrm{interp}}(y)=-\frac{1}{|y|}\mbox{log}p(y|z_{\mathrm{interp}}).$
(17)

That leads to semantically different responses being pushed into different
directions in the latent space by using the shared decoder. The authors claim
that their Space Fusion model was ranked highest in terms of relevance and
interest compared to several baselines by a human evaluation.

#### Decoding

Most research in terms of improving informativity is based on decoding, even
though the decoding itself is not the core problem of it, as discussed
earlier. Better decoding algorithms, however, that try to _cover_ the original
problems by having specific rules or by adding randomness show improvements in
this dimension.

Repetition does not only hurt fluency, but also informativity. ? (?)
experimented with a weighted decoding to tackle word and phrase repetition to
reduce uninformative repeating utterances. They first identify repeating
bigrams in the currently generated utterance and the previous utterances from
the history, as well as repeating content words. Then, they apply a negative
weight to these tokens. The authors used that repetition control in all
experiments and highlighted the importance of it. However, this highlights the
problematic model output (that seem to be repetitive) and does not address the
real problem. If the weight is $-\infty$, that method is similar to the n-gram
blocking explained in the previous section. This n-gram blocking also helps
with informativity, as blocking n-grams from previous utterances helps to
increase informativity through restricting the algorithm to repeat what was
already said.

Another way of reducing repetition is to add randomness to the decoding
algorithms. ? (?) showed that sampling one of the top k tokens (top-k-
sampling) or setting the sampling threshold to a cumulative probability (top-
p-sampling) is beneficial. Because of that, they come up with Nucleus
sampling, a combination of both: From a token distribution $P(x|x_{1:i-1})$
the top-$p$ vocabulary $V^{(p)}$ (the set with the least number of tokens with
a cumulative probability $>=p$) is used to reshape that distribution to:

$P^{\prime}(x|x_{1:i-1})=\begin{cases}P(x|x_{1:i-1})/P^{\prime}&\mbox{if
}x\epsilon V^{(p)}\\\ 0&\mbox{otherwise.}\end{cases}$ (18)

They stated that this approach gives more flexibility, as the sampling process
is dynamic. However, ? (?) and ? (?) showed in their experiments that
increasing randomness reduces consistency and coherence, since low probability
tokens might cause contradictions when sampled.

Very short utterances are generally uninformative. Forcing decoding algorithms
to create a minimum number of tokens by restricting the end-of-sequence
sampling until a minimum sequence length is reached helps to create longer
utterances, but is static. Sometimes short utterances are more suitable. ? (?)
expanded on this idea and proposed a model that predicts the sequence lengths
of next utterances given the dialogue history. At inference, they use this
estimation to set the minimum length for the decoding algorithm. While this
helps to generate longer sequences when it is required to increase
informativity, it does not solve the core issue, like the previous approaches
for decoding.

###  5.3 Quality // Consistency

Offering factual information is part of a natural conversation and (according
to Grice) they should be correct. Neural conversation models solely trained on
dialogue data have to have this explicit information encoded in its weights.
Gathering such information from pure dialogues is a huge generalization
challenge, and ? (?) observed that this information is indeed often wrongly
remembered and used by the models. A common way is to induce knowledge into a
model in addition to the dialogue history. This causes two problems: The train
part now also includes the knowledge processing, which makes the training
objective harder and the model architecture needs changes or gathering
augmentations, as usually these models have a limited amount of input space
for the additional knowledge.

Even if supported with context $K$, neural conversation models tend to
generate contributions that do not entail with $K$, which is called
_hallucination_. A recent survey (?) covered hallucination in the context of
natural language generation in general. ? (?) also investigated these
phenomena and showed that there is a positive correlation with hallucination
and predictive uncertainty. The predictive uncertainty for a token $x_{i}$ is
measured by their entropy:

$H(x_{i}|C_{i})=-\sum_{v\epsilon V}P(x_{i}=v|C_{i})\mbox{log }p(x_{i}|C_{i})$
(19)

with $C_{i}$ being the current context for $x_{i}$ and $V$ the set of all
possible tokens (vocabulary).

Similar to the problem with factual consistency a neural conversation model
only trained on dialogue also has an inconsistent personality: The datasets
often contain many different personalities and as long as they are not
explicitly labelled, this data is inconsistent by itself for the model (?).
Additionally, due to simple training objectives that do not cover personality
explicitly, the model cannot generalize good enough to learn different and
consistent behaviors (?). Again, this is tackled by augmenting datasets with
additional personality information along with new model architectures.

#### Domain Data

Being consistent towards $K$ requires augmented dialogues for training. To our
best knowledge, there are no large pre-training datasets of this type
available, as they require numerous resources to make. Therefore, many fine-
tuning datasets were built to learn consistent behavior.

? (?) introduced the knowledge augmented conversational dataset “Wizard of
Wikipedia”. In the collection process, one crowd-worker had access to
Wikipedia excerpts that the was tasked to use while the other crowd-worker was
tasked to “play the role of a curious learner, eager to chat”. They collected
22 thousand dialogues (over 200 thousand speaker turns) on a knowledge
database with 93 million sentences across 1.4 thousand different topics.

? (?) proposed the personality-focused dataset PersonaChat. It consists of
dialogues based on sets of 4–5 sentences (also crowdsourced) that describe a
personality. Participants were tasked to get to know each other while
pretending to be their given personality. The authors further asked another
set of crowd-workers to rewrite the persona descriptions afterwards, since
they noted a significant amount of word overlap between utterances and persona
descriptions. This should emphasise the model to understand the personas
rather than copying parts of it into the conversation.

Two additional important datasets are created by ? (?). The authors proposed
dialogue datasets to address inconsistency towards model personality. The
Twitter Persona dataset was extracted from Twitter in 2012 and only
interactions where both participants had engaged in at least 60 3-turn
conversations during the collection period were collected. This resulted in a
dataset with labelled personalities (with the assumption that each participant
has one consistent personality) with many data points per personality. They
collected 74 thousand different writer sources with at least 60 different
contributions to 3-turn conversations. Validation and test data were collected
later in 2012 to increase the likelihood of uncorrelated data. The second
dataset, Television Series Transcripts, had a focus on longer and more
dialogues from the same personality but with a greatly reduced number of
speakers. For that, they processed scripts from the television series Friends
and The Big Bang Theory, which resulted in 69 thousand speaker turns across 13
different personalities (characters).

? (?) introduced the KOMODIS (Knowledgeable and Opinionated Movie Discussions)
dataset with a focus on combining knowledge and personality, where crowd-
workers were tasked to talk about different movies. They were provided with
shared and private knowledge, as well as opinions (for personality) about the
movies, persons and facts. The final dataset contains 7.5 thousand dialogues
(100 thousand utterances) about 500 different movies.

? (?) also had the idea to combine knowledge and personality and created the
Blended Skill Talk dataset. They used parts from the previously discussed
datasets Wizard of Wikipedia, PersonaChat and Empathetic Dialogues (?) to
crowdsource new dialogues based on different contexts. Dialogues are seeded
with a pair of utterances from one of the datasets (and some context) and two
profile sentences from PersonaChat. Crowd worker can choose to freely talk or
use a generated next utterance to prevent them from getting stuck in a
specific skill. The generated utterances come from models fine-tuned on these
datasets beforehand. They collected 5 thousand conversations with an average
of 11.2 utterances.

A different dataset is the BeliefsBank dataset (?) that consists of 2.6
thousand statements either labelled as false or true in the form of triples.
Each triple consists of a sentence, a label and a strength value, for example:
(“A poodle is a dog.”, True, 0.9) or (”An eagle is a mammal“, False, 1.0).
This does not augment dialogues explicitly, but can be used independently to
control or learn beliefs about the world.

#### Models

Another large intervention point for consistency is the model itself. ? (?)
proposed a simple but effective way of using a GPT model without architecture
changes: They augment the dialogue history with the context from PersonaChat
and separated the persona profiles from the dialogue history by a segments
layer. That layer is added to the normal word tokens and differentiates
between the source using two new segment tokens. This way, the model learns to
differentiate between persona description token embeddings and token
embeddings from the dialogue history. The amount of additional context
information in this approach is limited by the input sequence length of the
Transformer models.

The following contributions focus on adding additional layers or memory
modules to pre-trained language models.

? (?) proposed a way to select a subset of relevant knowledge based on the
dialogue history from a large knowledge base by a knowledge selection module
utilizing a Bidirectional Encoder Representations from Transformers (BERT)
model. Experiments are done with the Wizard of Wikipedia dataset. And since
there are no knowledge subset labels available, they constructed weak labels
automatically by assuming that gold answers have higher similarity with the
correct context. These weak labels are only used to bootstrap the knowledge
selection model, since random knowledge estimation at the beginning leads to
error accumulation (feeding entirely wrong knowledge to the GPT). After
bootstrapping, the approach is unsupervised. A human evaluation indicates that
the model is superior regarding context consistency and relevance compared to
the proposed memory model by the authors of the dataset.

? (?) experimented with a knowledge copy mechanism. They use Long Short-Term
Memory (LSTM) cells for encoding the dialogue history, a Transformer to encode
the knowledge and then another LSTM cell and a pointer network with a
differentiable soft switch to either decode based on the history or based on
the knowledge. Therefore, their model is able to either create text with
standard language modelling (generating) or by explicitly copying tokens from
the context. They also evaluate their approach on the Wizard of Wikipedia
dataset. According to their human evaluation, this approach creates fluent and
consistent dialogues, but they do not compare their approach to others.

? (?) introduced a knowledge fetching based on K-Nearest Neighbours (KNN)
algorithms. Given a dialogue history
$\mathbf{X}={\\{x_{1},x_{2},...,x_{m}\\}}$ and context
$\mathbf{E}={\\{e_{1},e_{2},...,e_{m}\\}}$, the KNN algorithm identifies the
closest information in $\mathbf{E}$ that is relevant to $x_{i}$. And since KNN
is fully differentiable, they were able to incorporate this algorithm into a
deep learning setup with a Transformer-based language model. A human
evaluation showed that this architecture produces more consistent knowledge
compared to retrieval-based approaches. Participants had to choose between
both models, and around 60 % preferred the author’s model.

? (?) introduced a search engine augmented generation, an algorithm that
estimates a search query for a black-box search engine and uses the results
for generation. Their method is based on two components: A search query
generator (an encoder-decoder Transformer) that uses the dialogue context to
generate a search query and another encoder-decoder model that encodes the
search results, concatenates them to the dialogue context and then generates
the next response. Their human evaluation proofed the feasibility of this
approach. Human annotators evaluated 750 responses across 100 model
conversations and compared the search engine-based method with a no-knowledge
baseline. The new approach outperformed the baseline in all metrics: more
knowledgeable (46.5% of the time; compared to 38.4%), more consistent (76.1%
compared to 66.5%) and more engaging (81.4% compared to 69.9%).

#### Objective Function

Unlikelihood training (see Section 5.1) from ? (?) also provides possibilities
to explicitly train a model to be consistent: The unlikelihood loss
$\lambda_{\textrm{UL}}$ penalises tokens that are part of inconsistent
content. Here the challenge is to create inconsistent candidate pairs
$(\mathbf{x},\mathbf{y}^{-})$. The authors refer to approaches from natural
language inference tasks. They use an existing corpus of utterance pairs as
well as utterance-persona pairs, semi-automatically annotated for entailment,
contradiction or neutral relations. This approach is also suitable for
tackling incoherence, which we discuss in Section 5.4.

Good knowledge selection mechanisms are important to create consistent
content, as choosing the wrong knowledge in the first place eliminates the
ability to create appropriate contributions at all. ? (?) introduced a new
distilled distant supervision loss to bootstrap correct labels for knowledge
selection. The loss is a combination of a confidence score estimated by word
overlap between knowledge candidates $k^{i}_{t}$ and gold response $y_{t}$:

$W_{k^{i}_{t}}=\mbox{softmax}(\frac{2\cdot|\mbox{set}(k^{i}_{t})\cap\mbox{set}(y_{t})|}{|\mbox{set}(k^{i}_{t})|+|\mbox{set}(y_{t})|+\epsilon})$
(20)

and a teacher module (that has access to the gold response) that is used to
transfer the knowledge to the student (the real) module, which aims to reduce
the noise in the labels, which they call knowledge distillation
$\lambda_{\mathrm{KD}}$. The final loss is:

$\lambda_{\mathrm{DDSL}}=\lambda_{\mathrm{CE}}(S,k^{{}^{\prime}}_{t})\cdot
W_{k^{i}_{t}}+\lambda_{\mathrm{KD}}$ (21)

with $\lambda_{\mathrm{CE}}(S,k^{{}^{\prime}}_{t})$ being the cross entropy
loss between a knowledge selection distribution $S$ and estimated knowledge
$k^{{}^{\prime}}_{t}$. They reported improvements in knowledge selection for
their unsupervised setup over previous supervised setups, including the
baseline for the Wizard of Wikipedia dataset.

####

Another intervention point to tackle inconsistency is reinforcement learning
(?) (see Section 5.1): Similar to unlikelihood training, the authors use an
Natural Language Inference (NLI) model that can determine entailment between a
dialogue utterance and context. For that, they fine-tuned a BERT model with
the Dialogue NLI dataset (?) (an adaption of the PersonChat dataset) to
predict whether a generated utterance entails or contradicts a personality
trait. The classification probabilities of the model are then used to compute
a persona consistency reward:

$R=\frac{1}{|p|}\sum_{f_{i}\epsilon
p}P^{\mathrm{NLI}}_{e}(f_{i},r)-\frac{\beta}{|p|}\sum_{f_{i}\epsilon
p}P^{\mathrm{NLI}}_{c}(f_{i},r)$ (22)

used to evaluate fully sampled sequences $r$. The reward sums up the
entailment probabilities $P_{e}^{\mathrm{NLI}}$ and the contradiction
probabilities $P_{c}^{\mathrm{NLI}}$ for all personality traits $f_{i}$
compared to $r$. $\beta\geq 1.0$ as a hyperparameter is used to slightly
penalize contradictions more over entailments.

###  5.4 Relation // Coherence

The discrepancy between standard language modelling loss and the real
communicative intent of a human conversation (mentioned in Section 5.2) also
leads to incoherent responses (?). ? (?) stated further that simple sequence-
to-sequence models cannot acquire the ability to generalize over situational
context without explicit guidance. This is further hindered by having many
good contributions to a dialogue state, but only learning a specific one (the
gold next utterance). Another problem is that models do not make use of the
full observable dialogue history $C_{t}$: ? (?) studied the sensitivity of
recurrent and Transformer-based dialogue models to synthetic perturbations
$D^{P}_{t}$ of $D_{t}$ (order of words and utterances, dropping certain words
or utterances) using four different dialogue datasets. They showed that the
perplexity difference between $D_{t}$ and $D^{P}_{t}$ is small, concluding
that these models are insensitive to order, which is problematic for capturing
conversational dynamics. Many approaches across different intervention points
have been proposed recently.

#### Pre-training

? (?) did not introduce new pre-training data, but several pre-training tasks
to create strong dialogue representations for down stream tasks like next
utterance generation or dialogue act prediction. To model coherence, they
proposed the task of Inconsistency Identification (InI): the objective is to
determine a wrong (randomly chosen) utterance, which replaced a correct
utterance in the dialogue history. They reported that this task improves on F1
and BiLingual Evaluation Understudy (BLEU) on their test set.

#### Objective Function

? (?) integrated the perturbations $D^{P}_{t}$ into the train process by
defining a reward function based on whether $D^{P}_{t}$ has an effect on the
likelihood of the next utterance or not:

$R(Y|X,X^{\prime})=NLL_{adv}-NLL_{orig}$

Additionally, they defined a penalty function using a (not further specified)
margin $\mathcal{M}$:

$P(Y|X,X^{\prime})=min(0,NLL_{adv}-NLL_{orig}-\mathcal{M}).$

They pre-trained their model with the standard language modelling objective
until perplexity on validation converges, and then continued the training with
their inverse adversarial method, where the next utterance is alternately
taken from the training set or generated by the model itself. A human
evaluation in terms of fluency, consistency (not our definition, but asking:
“how likely the generated text is related to the input dialogue history”), and
diversity showed that this approach improves greatly over a simple baseline
and also outperformed other approaches in all three categories.

#### Model

The model itself is the most researched intervention point for maintaining
coherence. (?) introduced approaches based on Conditional Variational Auto
Encoder (CVAE) for dialogue generation to account for the complexity of
several possible continuations for a given context. The model encodes the
dialogue context (together with meta information on the topic) as a
distribution vector to allow for variation in the decoding step, where the
original input is reconstructed from the latent representation. The
distribution of the latent representation is learned by minimizing the
Kullback Leibler (KL) Divergence to a recognition network that has access to
the target response as well. The model is trained on the SwitchBoard corpus
(?). They reported best results in terms of automatic metrics (BLEU and GloVe-
based embedding distance) and a manual analysis of model outputs. ? (?) built
upon this further by adding the calculated coherence score between the context
and the gold response as an additional latent variable instead of dialogue act
labels. Their results on the OpenSubtitles corpus suggest that this approach
yields improvements, even when trained on the unfiltered data and evaluated on
the filtered test set. ? (?) also adapted the CVAE idea by changing the
training objective to a generative adversarial network discriminator to allow
for more complex distributions in the latent space. This architecture enables
the model to capture the multimodal distribution of possible responses and
shows improvements in terms of the automatic metrics proposed by ? (?).
Further, ? (?) showed that moving from euclidean to hyperbolic space for the
latent representation yields further improvements in the direct comparison
with ? (?).

Increasing informativeness can sacrifice dialogue coherence (?). Both can be
optimized jointly by using the already proposed SpaceFusion model. This
approach improves over baselines and the CVAE model from ? (?) in BLEU and
human rated relevance and interest.

? (?) proposed another model architecture based on conditional training, which
they call PLATO. They jointly trained a latent speech act classifier and a
response generator. Their model is trained unsupervised and also includes the
bag of words loss introduced by ? (?) to learn to reconstruct the response
words from the latent speech act’s hidden state $z$. During inference, one
response is generated for all values of $z$ and a response selection component
is used to find the highest scoring (i.e., most coherent) response. Their
human evaluation on 100 random samples, where judges were asked to rate
coherence as “whether the generated response is relevant with the dialogue
context and consistent with the expressed information or background knowledge”
on a three-point Likert scale, showed improvements across all categories.
Later, they proposed PLATO-2 ? (?) which is pre-trained on a cleaned Reddit
corpus. It consists of a diverse response generation model based on their
baseline approach and a separate response coherence estimator which combines
the response selection loss from above with a masked language modelling
objective. During inference, as above, the first model will generate a
response for every latent speech act and the second one will select the
highest scoring one. PLATO-2 models outperformed their counterparts (PLATO
(?), DialoGPT (?), Blender (?) and Microsoft XiaoIce (?)) in multiple human
evaluations.

Referenced-based word overlap metrics such as BLEU do not correlate well with
coherence (?). To create appropriate training signals, one line of research
focuses on models that are able to predict coherence scores. ? (?) collected a
dataset with human judgements on next utterance appropriateness to train a
model with it. However, generic (and therefore uninformative) responses were
often rated as highly appropriate, which again shows a negative correlation
between informativity and coherence. ? (?) proposed a Referenced metric and
Unreferenced metric Blended Evaluation Routine (RUBER): The referenced metric
$s_{R}$ calculates the similarity of a generated response and the ground truth
with a cosine distance, and the unreferenced metric $s_{U}$ scores the
appropriateness of a reply for a given context. This is done by training a
neural network to distinguish between the gold next response $r$ and randomly
sampled $r^{-}$ by minimizing:

$\mathcal{L}=max(0,\Delta-s_{U}(q,r)+s_{U}(q,r^{-}))$

where $\Delta$ is a constant based on the development set. ? (?) extended this
measure by using BERT embeddings instead of word-to-vec and some model
modifications. A direct comparison with the RUBER metric showed that this
measure can profit from the richer representations provided by BERT. ? (?)
noted that the above metrics only focus on the utterance level without taking
topic transitions into account. To address this problem, they proposed to use
Graph-enhanced Representations for Automatic Dialogue Evaluation (GRADE),
which is based on sampled responses that are similar to the ground truth
rather than sampled randomly.

#### Decoding

? (?) suggested to use weighted decoding to control for different response
attributes by changing the score of a possible next word according to specific
weighted features $w_{i}$:

$\begin{split}score(w,y_{<t};x)=score(y_{<t};x)+logP_{CM}(w|y_{<t},x)+\sum_{i}w_{i}*f_{i}(w;y_{<t},x)\end{split}$

For response relatedness, they defined $f$ to be a cosine-based similarity
measure of GloVe-based embeddings of the candidate next word and the last
observed utterance $o_{t}$:

$\begin{split}resp\\_rel(w;y_{<t},x)=cos\\_sim(word\\_emb(w),sent\\_emb(o_{t}))\end{split}$

This seemed to improve the topic relatedness of the response in a manual
evaluation by the authors. However, this effect was not reproducible in their
crowdsourced human evaluation, where workers were asked to judge the dialogue
quality in terms of listening and making sense (among other qualities) after
six turns of interactive chat. The authors attributed this discrepancy to the
problem that deviations from the learned distribution like this can also lead
to uncontrolled nonsensical output by the model. This confirms the findings of
? (?), who used a similar approach that combines response relatedness with an
additional topic similarity feature for weighted decoding. They extracted the
topic distributions for the context and the next utterance from a pre-trained
HMM-LDA topic model and computed their similarity. In their human evaluation,
they showed that an increase in content richness (similar to our definition of
Informativity) comes with a decrease in plausibility (i.e., coherence).
However, a combination of their approach with MMI re-ranking and beam search
led to the best results in both categories (though still leaving quite some
room for improvement).

###  5.5 Social Norm

The problem of generating contributions that do not follow the social norm is
similar to the coherence problem from the previous section. While directly
insulting contributions (for example: “I hate you!”) are often caused by the
observed language from the pre-training and domain data, the problem of
agreeing to unsocial utterances and the problem of giving unsafe advice
requires semantic understanding of the full context $C_{t}$ and the social
norm $R$ (?). Since $R$ is not defined explicitly, the models also have to
learn a social behavior or ethical understanding implicitly.

#### Pre-Training Data

? (?) further stated that harmful contributions like in Example 2.3 can be
avoided by cleaning up the train data, since datasets scraped from the web
usually have a biased hegemonic world-view and tend to misrepresent social
movements ? (?). Similarly, (?) showed that models trained on data from Reddit
with more unsafe contexts generate more unsafe utterances in a dialogue than
models trained on the less toxic BlendedSkillTask (?) dataset. They measured
toxicity as frequency of occurring toxic tokens in the generated text.

#### Domain Data

? (?) experiment with smaller datasets and fine-tuned a language model
according to different behaviour types. They showed that fine-tuning with a
set of handcrafted question answering pairs (“What makes a person beautiful?”,
“The attractiveness of a person is a highly subjective measure. […]”) reduced
toxicity measured in a human evaluation, which matches the statement from ?
(?). However, all these approaches do not help with indirect harmful content
like in Example 2.3 or unethical contributions like in Example 2.3.

#### Decoding

Manipulating the decoding algorithms is a common intervention point to reduce
toxicity, however, most of these approaches are in the context of general
language generation. We still review two of them here, as these strategies
could be adapted to a dialogue context as well. ? (?) introduced a self-
diagnosis approach: They created sentence patterns like: “Does the above text
contain toxicity?” and added them at the end of a text they want to analyse.
Measuring the probability of ‘yes’ and ‘no’ tokens as a next token, seems to
give a good toxicity estimation for bigger models. They report an accuracy of
up to 87.3% on the largest T5 model. With this self-diagnosis capacity they
detect inappropriate tokens at decoding to avoid them as a next token. ? (?)
experimented with non-toxic expert and (toxic) anti-expert language models. In
order to create these models they fine-tuned a GPT-2 model on the dataset from
the ‘Jigsaw Unintended Bias in Toxicity Classification’
challenge.333https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-
classification The toxic model is only trained on toxic data, while the expert
model is only trained on clean data. At decoding they look at tokens with a
large probability difference (assuming that they are toxic) and modify the
token probability distribution according to the measured toxicity while
decoding.

#### Post Filtering

Instead of encouraging a neural conversation model to stick to the social
norm, this intervention point prevents the model from inappropriate
contributions by denying already generated contributions. ? (?) introduced a
four-step approach to train a robust detection system: Build an initial model
on the Wikipedia Toxic Comments dataset that detects single-turn utterances.
Then ask crowd workers to find contributions that are not covered by the
model. Retrain the classifier and repeat the crowd-working task again. Since
single-turn analysis is not sufficient, they also experimented with
crowdsourced multi-turn samples and reached an F1 score of 66.4 on their best
multi-turn classifier. ? (?) followed up on this work, but focused on
eliciting offensive messages from the neural conversation model in the break-
it phases. For that, they formulated four different strategies: training
classifiers for detecting unsafe messages, making unsafe content explicitly
unlikely, avoiding topics like politics or religion and forcing the model to
respond gender-neutral. They reported an F1 of 85.9 on the same multi-turn
data, and humans rated their best model as safe in 96,6% of the time.

##  6 Conclusion

In this paper, we surveyed current problems and shortcomings with neural
generation models for non-goal oriented dialogue. For that we took inspiration
from Grice’s Cooperative Principle and introduced five categories (fluency,
informativity, consistency, coherence and social norm) that need to be
followed to achieve a good conversation model. We analysed current solution
proposals, sorted them according to our categories and possible intervention
points, and highlighted their potentials and limitations.

Pre-trained language models are strong tools in the field of neural generation
models, but unsupervised training on pure dialogues can only solve one aspect
of a good artificial dialogue partner: they are able to make dialogues fluent.
All other aspects require labelled high quality datasets and strong automated
metrics for training and evaluation. We noticed an emerging trend away from
pure end-to-end training (i.e., building chatbots) to more specific solutions
for different dialogue aspects, such as diversification of generated words,
learning the utilisation of general knowledge or using consistent personality
traits. We support that trend, as human language and specifically dialogue,
has a high complexity, and therefore the generalisation effort that neural
networks have to make is also high.

We want to stress one particular issue in the field, namely the lack of
comparability: The aforementioned trend to very specific solutions leads to
many different datasets that are used to train models and that are not
comparable. Additionally, the lack of available metrics enforces the use of
human evaluations or self-defined metrics, which are also not comparable among
each other. That makes it hard to quantify the reining effect of different
approaches, therefore comparability should receive more attention in future
work.

## References

  * Ameixa et al. Ameixa, D., Coheur, L., Fialho, P., and Quaresma, P. (2014).  Luke, i am your father: Dealing with out-of-domain requests by using movies subtitles.  In IVA.
  * Baheti et al. Baheti, A., Ritter, A., Li, J., and Dolan, B. (2018).  Generating more interesting responses in neural conversation models with distributional constraints.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3970–3980, Brussels, Belgium. Association for Computational Linguistics.
  * Banchs Banchs, R. E. (2012).  Movie-DiC: a movie dialogue corpus for research and development.  In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 203–207, Jeju Island, Korea. Association for Computational Linguistics.
  * Bao et al. Bao, S., He, H., Wang, F., Wu, H., and Wang, H. (2020).  PLATO: Pre-trained dialogue generation model with discrete latent variable.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 85–96, Online. Association for Computational Linguistics.
  * Bao et al. Bao, S., He, H., Wang, F., Wu, H., Wang, H., Wu, W., Guo, Z., Liu, Z., and Xu, X. (2021).  PLATO-2: Towards building an open-domain chatbot via curriculum learning.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 2513–2525, Online. Association for Computational Linguistics.
  * Bender et al. Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021).  On the dangers of stochastic parrots: Can language models be too big?.  In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, p. 610–623, New York, NY, USA. Association for Computing Machinery.
  * Chen et al. Chen, X., Chen, F., Meng, F., Li, P., and Zhou, J. (2021).  Unsupervised knowledge selection for dialogue generation.  In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 1230–1244, Online. Association for Computational Linguistics.
  * Cho et al. Cho, K., van Merriënboer, B., Bahdanau, D., and Bengio, Y. (2014).  On the properties of neural machine translation: Encoder–decoder approaches.  In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, pp. 103–111, Doha, Qatar. Association for Computational Linguistics.
  * Dai et al. Dai, S., Gan, Z., Cheng, Y., Tao, C., Carin, L., and Liu, J. (2021).  APo-VAE: Text generation in hyperbolic space.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 416–431, Online. Association for Computational Linguistics.
  * Deriu et al. Deriu, J., Rodrigo, A., Otegi, A., Echegoyen, G., Rosset, S., Agirre, E., and Cieliebak, M. (2021).  Survey on evaluation methods for dialogue systems.  Artificial Intelligence Review, 54(1), 755–810.
  * Devlin et al. Devlin, J., Chang, M., Lee, K., and Toutanova, K. (2018).  BERT: pre-training of deep bidirectional transformers for language understanding.  CoRR, abs/1810.04805.
  * Dinan et al. Dinan, E., Abercrombie, G., Bergman, A. S., Spruit, S., Hovy, D., Boureau, Y.-L., and Rieser, V. (2021).  Anticipating safety issues in e2e conversational ai: Framework and tooling..
  * Dinan et al. Dinan, E., Humeau, S., Chintagunta, B., and Weston, J. (2019a).  Build it break it fix it for dialogue safety: Robustness from adversarial human attack.  In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4537–4546, Hong Kong, China. Association for Computational Linguistics.
  * Dinan et al. Dinan, E., Logacheva, V., Malykh, V., Miller, A. H., Shuster, K., Urbanek, J., Kiela, D., Szlam, A., Serban, I., Lowe, R., Prabhumoye, S., Black, A. W., Rudnicky, A. I., Williams, J., Pineau, J., Burtsev, M. S., and Weston, J. (2019b).  The second conversational intelligence challenge (convai2).  CoRR, abs/1902.00098.
  * Dinan et al. Dinan, E., Roller, S., Shuster, K., Fan, A., Auli, M., and Weston, J. (2019c).  Wizard of wikipedia: Knowledge-powered conversational agents.  ArXiv, abs/1811.01241.
  * Fan et al. Fan, A., Gardent, C., Braud, C., and Bordes, A. (2021).  Augmenting transformers with KNN-based composite memory for dialog.  Transactions of the Association for Computational Linguistics, 9, 82–99.
  * Galetzka Galetzka, F. (2022).  Investigating and improving background context consistency in neural conversation models.  doctoralthesis, Universität Potsdam.
  * Galetzka et al. Galetzka, F., Eneh, C. U., and Schlangen, D. (2020).  A corpus of controlled opinionated and knowledgeable movie discussions for training neural conversation models.  In Proceedings of the 12th Language Resources and Evaluation Conference, pp. 565–573, Marseille, France. European Language Resources Association.
  * Galetzka et al. Galetzka, F., Rose, J., Schlangen, D., and Lehmann, J. (2021).  Space efficient context encoding for non-task-oriented dialogue generation with graph attention transformer.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7028–7041, Online. Association for Computational Linguistics.
  * Gao et al. Gao, X., Lee, S., Zhang, Y., Brockett, C., Galley, M., Gao, J., and Dolan, B. (2019).  Jointly optimizing diversity and relevance in neural response generation.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1229–1238, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ghazarian et al. Ghazarian, S., Wei, J., Galstyan, A., and Peng, N. (2019).  Better automatic evaluation of open-domain dialogue systems with contextualized embeddings.  In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pp. 82–89, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Godfrey and Holliman Godfrey, J. J., and Holliman, E. (1997).  Switchboard-1 Release 2..
  * Grice Grice, H. P. (1975).  Logic and conversation.  In Cole, P., and Morgan, J. L. (Eds.), Syntax and Semantics: Vol. 3: Speech Acts, pp. 41 – 58. Brill, Leiden, The Netherlands.
  * Gu et al. Gu, X., Cho, K., Ha, J.-W., and Kim, S. (2019).  DialogWAE: Multimodal response generation with conditional wasserstein auto-encoder.  In International Conference on Learning Representations.
  * Heeman and Hirst Heeman, P. A., and Hirst, G. (1995).  Collaborating on referring expressions.  Computational Linguistics, 21(3), 351–382.
  * Higashinaka et al. Higashinaka, R., Funakoshi, K., Araki, M., Tsukahara, H., Kobayashi, Y., and Mizukami, M. (2015).  Towards taxonomy of errors in chat-oriented dialogue systems.  In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, pp. 87–95, Prague, Czech Republic. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber Hochreiter, S., and Schmidhuber, J. (1997).  Long short-term memory.  Neural Comput., 9(8), 1735–1780.
  * Holtzman et al. Holtzman, A., Buys, J., Du, L., Forbes, M., and Choi, Y. (2020).  The curious case of neural text degeneration..
  * Huang et al. Huang, L., Ye, Z., Qin, J., Lin, L., and Liang, X. (2020).  GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systems.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 9230–9240, Online. Association for Computational Linguistics.
  * Ji et al. Ji, Z., Lee, N., Frieske, R., Yu, T., Su, D., Xu, Y., Ishii, E., Bang, Y., Madotto, A., and Fung, P. (2022).  Survey of hallucination in natural language generation..
  * Kassner et al. Kassner, N., Tafjord, O., Schütze, H., and Clark, P. (2021).  BeliefBank: Adding memory to a pre-trained language model for a systematic notion of belief.  In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 8849–8861, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Komeili et al. Komeili, M., Shuster, K., and Weston, J. (2021).  Internet-augmented dialogue generation.  CoRR, abs/2107.07566.
  * Li et al. Li, J., Galley, M., Brockett, C., Gao, J., and Dolan, B. (2016a).  A diversity-promoting objective function for neural conversation models.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 110–119, San Diego, California. Association for Computational Linguistics.
  * Li et al. Li, J., Galley, M., Brockett, C., Spithourakis, G., Gao, J., and Dolan, B. (2016b).  A persona-based neural conversation model.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 994–1003, Berlin, Germany. Association for Computational Linguistics.
  * Li et al. Li, M., Roller, S., Kulikov, I., Welleck, S., Boureau, Y.-L., Cho, K., and Weston, J. (2020).  Don’t say that! making inconsistent dialogue unlikely with unlikelihood training.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4715–4728, Online. Association for Computational Linguistics.
  * Lin et al. Lin, X., Jian, W., He, J., Wang, T., and Chu, W. (2020).  Generating informative conversational response using recurrent knowledge-interaction and knowledge-copy.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 41–52, Online. Association for Computational Linguistics.
  * Liu et al. Liu, A., Sap, M., Lu, X., Swayamdipta, S., Bhagavatula, C., Smith, N. A., and Choi, Y. (2021).  DExperts: Decoding-time controlled text generation with experts and anti-experts.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 6691–6706, Online. Association for Computational Linguistics.
  * Lowe et al. Lowe, R., Noseworthy, M., Serban, I. V., Angelard-Gontier, N., Bengio, Y., and Pineau, J. (2017).  Towards an automatic Turing test: Learning to evaluate dialogue responses.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1116–1126, Vancouver, Canada. Association for Computational Linguistics.
  * Mehri et al. Mehri, S., Razumovskaia, E., Zhao, T., and Eskenazi, M. (2019).  Pretraining methods for dialog context representation learning.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3836–3845, Florence, Italy. Association for Computational Linguistics.
  * Mesgar et al. Mesgar, M., Simpson, E., and Gurevych, I. (2021).  Improving factual consistency between a response and persona facts.  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 549–562, Online. Association for Computational Linguistics.
  * Paulus et al. Paulus, R., Xiong, C., and Socher, R. (2018).  A deep reinforced model for abstractive summarization.  In International Conference on Learning Representations.
  * Radford et al. Radford, A., Narasimhan, K., Salimans, T., and Sutskever, I. (2018).  Improving language understanding by generative pre-training..
  * Rashkin et al. Rashkin, H., Smith, E. M., Li, M., and Boureau, Y.-L. (2019).  Towards empathetic open-domain conversation models: A new benchmark and dataset.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5370–5381, Florence, Italy. Association for Computational Linguistics.
  * Ritter et al. Ritter, A., Cherry, C., and Dolan, W. B. (2011).  Data-driven response generation in social media.  In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pp. 583–593.
  * Roller et al. Roller, S., Dinan, E., Goyal, N., Ju, D., Williamson, M., Liu, Y., Xu, J., Ott, M., Smith, E. M., Boureau, Y.-L., and Weston, J. (2021).  Recipes for building an open-domain chatbot.  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 300–325, Online. Association for Computational Linguistics.
  * Sankar et al. Sankar, C., Subramanian, S., Pal, C., Chandar, S., and Bengio, Y. (2019).  Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Study.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 32–37, Florence, Italy. Association for Computational Linguistics.
  * Schick et al. Schick, T., Udupa, S., and Schütze, H. (2021).  Self-diagnosis and self-debiasing: A proposal for reducing corpus-based bias in nlp.  Computing Research Repository, arXiv:2103.00453.
  * See et al. See, A., Roller, S., Kiela, D., and Weston, J. (2019).  What makes a good conversation? how controllable attributes affect human judgments.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 1702–1723, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Serban et al. Serban, I., Sordoni, A., Bengio, Y., Courville, A. C., and Pineau, J. (2016).  Building end-to-end dialogue systems using generative hierarchical neural network models.  In AAAI.
  * Smith et al. Smith, E. M., Williamson, M., Shuster, K., Weston, J., and Boureau, Y.-L. (2020).  Can you put it all together: Evaluating conversational agents’ ability to blend skills.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 2021–2030, Online. Association for Computational Linguistics.
  * Solaiman and Dennison Solaiman, I., and Dennison, C. (2021).  Process for adapting language models to society (PALMS) with values-targeted datasets.  CoRR, abs/2106.10328.
  * Sordoni et al. Sordoni, A., Galley, M., Auli, M., Brockett, C., Ji, Y., Mitchell, M., Nie, J.-Y., Gao, J., and Dolan, B. (2015).  A neural network approach to context-sensitive generation of conversational responses.  In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 196–205, Denver, Colorado. Association for Computational Linguistics.
  * Tao et al. Tao, C., Mou, L., Zhao, D., and Yan, R. (2018).  RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems.  In McIlraith, S. A., and Weinberger, K. Q. (Eds.), Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pp. 722–729. AAAI Press.
  * Vaswani et al. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. (2017).  Attention is all you need.  In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (Eds.), Advances in Neural Information Processing Systems, Vol. 30. Curran Associates, Inc.
  * Vinyals and Le Vinyals, O., and Le, Q. (2015).  A Neural Conversational Model.  In International Conferenceon Machine Learning: Deep Learning Workshop.
  * Welleck et al. Welleck, S., Kulikov, I., Roller, S., Dinan, E., Cho, K., and Weston, J. (2020).  Neural text generation with unlikelihood training.  In International Conference on Learning Representations.
  * Welleck et al. Welleck, S., Weston, J., Szlam, A., and Cho, K. (2019).  Dialogue natural language inference.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3731–3741, Florence, Italy. Association for Computational Linguistics.
  * Wolf et al. Wolf, T., Sanh, V., Chaumond, J., and Delangue, C. (2019).  Transfertransfo: A transfer learning approach for neural network based conversational agents..
  * Xiao and Wang Xiao, Y., and Wang, W. Y. (2021).  On hallucination and predictive uncertainty in conditional language generation.  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2734–2744, Online. Association for Computational Linguistics.
  * Xu et al. Xu, J., Ju, D., Li, M., Boureau, Y., Weston, J., and Dinan, E. (2020).  Recipes for safety in open-domain chatbots.  CoRR, abs/2010.07079.
  * Xu et al. Xu, X., Dušek, O., Konstas, I., and Rieser, V. (2018).  Better Conversations by Modeling, Filtering, and Optimizing for Coherence and Diversity.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3981–3991, Brussels, Belgium. Association for Computational Linguistics.
  * Zhang et al. Zhang, S., Dinan, E., Urbanek, J., Szlam, A., Kiela, D., and Weston, J. (2018).  Personalizing dialogue agents: I have a dog, do you have pets too?..
  * Zhang et al. Zhang, Y., Sun, S., Galley, M., Chen, Y.-C., Brockett, C., Gao, X., Gao, J., Liu, J., and Dolan, B. (2020).  DIALOGPT : Large-scale generative pre-training for conversational response generation.  In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 270–278, Online. Association for Computational Linguistics.
  * Zhao et al. Zhao, T., Zhao, R., and Eskenazi, M. (2017).  Learning discourse-level diversity for neural dialog models using conditional variational autoencoders.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 654–664, Vancouver, Canada. Association for Computational Linguistics.
  * Zhao et al. Zhao, X., Wu, W., Xu, C., Tao, C., Zhao, D., and Yan, R. (2020).  Knowledge-grounded dialogue generation with pre-trained language models.  In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3377–3390, Online. Association for Computational Linguistics.
  * Zhou et al. Zhou, L., Gao, J., Li, D., and Shum, H.-Y. (2020).  The design and implementation of XiaoIce, an empathetic social chatbot.  Computational Linguistics, 46(1), 53–93.
  * Zhou et al. Zhou, W., Li, Q., and Li, C. (2021).  Learning from perturbations: Diverse and informative dialogue generation with inverse adversarial training.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 694–703, Online. Association for Computational Linguistics.

RNN

    Recurrent Neural Network
NLI

    Natural Language Inference
BERT

    Bidirectional Encoder Representations from Transformers
GPT

    Generative Pre-trained Transformer
LSTM

    Long Short-Term Memory
KNN

    K-Nearest Neighbours
InI

    Inconsistency Identification
BLEU

    BiLingual Evaluation Understudy
CVAE

    Conditional Variational Auto Encoder
KL

    Kullback Leibler
RUBER

    Referenced metric and Unreferenced metric Blended Evaluation Routine
GRADE

    Graph-enhanced Representations for Automatic Dialogue Evaluation
  *[RNN]: Recurrent Neural Network
  *[GPT]: Generative Pre-trained Transformer
  *[BERT]: Bidirectional Encoder Representations from Transformers
  *[LSTM]: Long Short-Term Memory
  *[KNN]: K-Nearest Neighbours
  *[NLI]: Natural Language Inference
  *[InI]: Inconsistency Identification
  *[BLEU]: BiLingual Evaluation Understudy
  *[CVAE]: Conditional Variational Auto Encoder
  *[KL]: Kullback Leibler
  *[RUBER]: Referenced metric and Unreferenced metric Blended Evaluation Routine
  *[GRADE]: Graph-enhanced Representations for Automatic Dialogue Evaluation