capbtabboxtable[][]

# On the Use of Linguistic Features for the Evaluation of Generative Dialogue
Systems

Ian Berlot-Attwell 1,2, Frank Rudzicz 1,2,3
1 Department of Computr Science, University of Toronto;
2 Vector Institute for Artificial Intelligence;
3 Unity Health Toronto
{ianberlot<EMAIL_ADDRESS>

###### Abstract

Automatically evaluating text-based, non-task-oriented dialogue systems (i.e.,
‘chatbots’) remains an open problem. Previous approaches have suffered
challenges ranging from poor correlation with human judgment to poor
generalization and have often required a gold standard reference for
comparison or human-annotated data.

Extending existing evaluation methods, we propose that a metric based on
linguistic features may be able to maintain good correlation with human
judgment and be interpretable, without requiring a gold-standard reference or
human-annotated data. To support this proposition, we measure and analyze
various linguistic features on dialogues produced by multiple dialogue models.
We find that the features’ behaviour is consistent with the known properties
of the models tested, and is similar across domains. We also demonstrate that
this approach exhibits promising properties such as zero-shot generalization
to new domains on the related task of evaluating response relevance.

##  1 Introduction

Automatic evaluation of generative non-task oriented dialogue systems remains
an open problem. Given a context $c$ (e.g., “Speaker 1: Hello, how are you?
Speaker 2: I’m fine, yourself?") these systems produce a response $r$ (e.g.,
“Speaker 1: Pretty good, all in all"). Two of the major difficulties in
evaluating such a system are the inherently qualitative nature of a response’s
quality, and the vast space of acceptable responses. A variety of approaches
have been tried, but all suffer from problems such as dependence on human-
annotated data, poor generalization between domains, requiring human gold
references, overfitting to a specific dialogue model, and so on (a full
breakdown can be found in Section 2).

In this work, we propose that an approach based on linguistic features can 1)
be implemented without any human-annotated data or test-time gold references,
2) be easily interpreted, and 3) exhibit zero-shot generalization to new
domains. To support our claims, we evaluate and analyze a set of linguistic
features on dialogues from different domains produced by a variety of dialogue
systems. Furthermore, we train an unsupervised feature-based model for the
related task of relevance-prediction and demonstrate superior performance to
our unsupervised baselines as well as zero-short generalization to other
domains.

##  2 Related Work

Historically, non-task-oriented dialogue system evaluation has been approached
by viewing dialogue as a machine-translation problem with the context as the
source language and the response as the translation. Under this formulation,
machine translation metrics based on $n$-gram overlap (e.g., BLEU, METEOR) or
word embedding similarity were used to evaluate system responses, based on
human references. However, these approaches correlated very poorly with human
judgment Liu et al. (2016).

This incongruity can be explained by several observations. First, the majority
of these translation metrics were designed to have multiple gold-standard
translations, whereas dialogue corpora typically have only one human response
per context. Second, these metrics only compare the system’s response against
the gold-standard response – they fail to consider the context that generated
these responses.

To some extent, $n$-gram models are disadvantaged in that many valid dialogue
responses lack word overlap of any kind. Similarly, approaches based on word
embeddings are disadvantaged in that semantically similar words can still
create very inappropriate responses Liu et al. (2016).

###  2.1 BLEU-Inspired Approaches

Despite these problems, there have been attempts to modify BLEU for dialogue.
One such attempt is $\Delta$BLEU Galley et al. (2015); in this approach, a
subset of a pre-existing dialogue corpus is extended with additional possible
responses by automatically finding $\langle context,response\rangle$ pairs on
the full corpus similar to those in the subset, and adding the responses to
the subset. Each reference is scored by humans on a 5-point Likert scale, and
then re-scaled to $\left[-1,1\right]$. These reference scores are then used in
a modified BLEU – specifically the brevity penalty is unchanged, and the
corpus-level $n$-gram precision is re-defined as:

$\frac{\displaystyle\sum_{i}\sum_{g\in ngrams(h_{i})}\max_{j:g\in
r_{ij}}\\{w_{ij}\min(\\#_{g}(h_{i}),\\#_{g}(r_{ij}))\\}}{\displaystyle\sum_{i}\sum_{g\in
ngrams(h_{i})}\max_{j}\\{w_{ij}\\#_{g}(h_{i})\\}},$

where $h_{i}$ is the hypothesis for some context $c_{i}$, $r_{ij}$ and
$w_{ij}$ are a reference and its weight for context $c_{i}$, and $\\#_{g}(x)$
is the count of the $n$-gram $g$ in the text $x$ Galley et al. (2015). Despite
achieving a Spearman’s correlation of $0.485$ on groups of 100 responses
(compared with BLEU’s performance with a single response of $0.26$), the
sentence-level correlation was still poor (less than $0.1$). Another work
loosely based on this approach achieved a sentence-wise Pearson’s correlation
of $0.514$ Galley et al. (2015). The approach first hired humans to create new
responses for every context under zero or one constraints (e.g., the response
must be less than $X$ characters, or only $X\%$ of the context is visible).
Next, these responses were human-scored based on pairwise-win rate (i.e., the
percentage of comparisons with which it is preferred over another possible
response). Finally, they were sorted into decreasing order of win rate, and
used to train an SVR to predict the pairwise-win rate of a new response from
the $\left[-1,1\right]$-normalized word error rate (WER) of the new response
with the top $M$ human-created responses Galley et al. (2015). However, both
of these approaches have the problem that, for each context, a set of human-
scored possible responses is required. As such, these approaches become
impractical as the set of contexts expands.

###  2.2 Surrogate Tasks

Another approach for dialogue evaluation has been to evaluate performance on
the surrogate task of ‘next utterance classification’ (NUC) Lowe et al.
(2016). NUC is the task of selecting the most appropriate response from a list
of multiple possible responses, and is evaluated using Recall@$k$ (i.e., the
percentage with which the correct response is among the top $k$ responses
suggested by the system). The performance of dialogue systems on this task is
still inferior to human experts, and thus it is a useful but insufficient
metric for dialogue system evaluation Lowe et al. (2016). A drawback of NUC is
that it may be impossible or difficult to compute Recall@$k$ depending on the
model. For instance, TF-IDF retrieval models (see Section 3.1.2) cannot be
evaluated with NUC unless all possible responses are in its dataset, and it is
often intractable to calculate the probability of a response under a latent
variable model such as VHRED, (see Section 3.1.3.3).

###  2.3 Regression-Based Approaches

Lowe et al. (2017) worked to solve this problem by instead formulating the
dialogue evaluation task as a regression problem. Their approach was to
construct a dialogue corpus where each response was annotated with an average
human response on a rating scale of the perceived quality. This corpus was
then used as training data for a neural net. The net (ADEM) functioned by
first encoding the context, system response, and gold standard response as
vectors ($\mathbf{c}$, $\mathbf{\hat{r}}$, and $\mathbf{r}$, respectively)
using a hierarchical RNN. The score was then computed using learned matrices
$M$ and $N$ as:
$(\mathbf{c}^{T}M\mathbf{\hat{r}}+\mathbf{r}^{T}M\mathbf{\hat{r}}-\alpha)/\beta$
(the values $\alpha,\beta\in\mathbb{R}$ being constants chosen at the start of
training so that initially all predictions are in the range $[1,5]$). This
approach yielded superior results than re-purposed MT methods, achieving an
utterance-level Pearson’s correlation of $0.436$ on the test data (for
comparison, the best of the other methods was ROUGE, with a Pearson’s
correlation of $0.147$).

However, this approach is not without its own flaws. ADEM has had difficulty
transferring to other datasets – a behaviour that is currently believed to
originate in insufficient training data possibly combined with an inherent
bias in the dataset Lowe (2018). This is a problem inherent to all trained
black-box evaluators, which have the potential to overfit to the training
data, the domain of the data, or even the methods used to generate the subpar
examples in the training data. This vulnerability to overfitting was shown by
Sai et al. (2019) who demonstrated that ADEM can be tricked by methods as
simple as reversing or jumbling word order.

More recently, Deriu and Cieliebak (2019) developed AutoJudge, a spiritual
successor to ADEM that does not require a gold-standard response at evaluation
time. Although it achieved strong correlation of $0.577$, it is once again
dependent on a dataset of human-annotated dialogues. Furthermore, they found
that their trained metric suffered from instability due to the relatively
small size of their created dataset and, more importantly, they could not use
AutoJudge as a reward signal for reinforcement learning. The authors concluded
that although AutoJudge can distinguish the good and bad utterances of trained
systems, there are classes of bad responses that trick AutoJudge. They
suggested that modifying the training procedure would be required for
AutoJudge to be able to identify these, leading us to conclude that AutoJudge
is overfitting to its training data. We therefore also speculate that
AutoJudge may share ADEM’s tendency to overfit to its domain.

###  2.4 Adversarial Approaches

Adversarial evaluation also suffers from the same potential overfitting
problems, but does not require human-annotated quality data Kannan and Vinyals
(2017). In this approach, a neural net is trained as a binary classifier that
discriminates between human and machine-generated responses given the context.
In an experiment to discriminate between human and RNN-generated dialogue, the
experimenters achieved an accuracy of 62.5%, and found the discriminator
correlated strongly with length and weakly with the generator likelihood (it
has been long observed that generated responses tend to be shorter, and that
highly likely responses tend to be generic) Kannan and Vinyals (2017).
However, as noted by the authors, there is no evidence that a dialogue system
that fools the discriminator would necessarily have superior human evaluation
Kannan and Vinyals (2017).

##  3 Methods

We propose that, through linguistic features, a metric for generative dialogue
systems can be created that is interpretable, does not require gold-standard
responses, does not require human annotated data, and generalizes to other
dialogue domains. As a proof-of-concept, we perform two experiments to
demonstrate that a feature-based approach can display these properties.

First, we generate responses to the same contexts using the dialogue models
described in this section and analyze the distribution of features seen over
different sources. We expect different sources to exhibit different
distributions based on the properties of the generative model.

Second, we empirically demonstrate that a feature-based approach can
demonstrate domain generalization, and be produced and evaluated in an
unsupervised manner. Ideally, we would want to measure the correlation of our
metric with human ratings for the quality of the response. Unfortunately, to
our knowledge, there is at present no publicly available dataset of dialogues
annotated directly with response quality. Instead, we use the HUMOD dataset
Merdivan et al. (2020) and infer the relevance of responses to contexts rather
than the quality. We then compare performance to supervised baselines and
measure model performance when fitted on one domain, and tested on another.
Response relevance is clearly not a perfect proxy for response quality, but
our goal is to show the feasibility of the approach rather than to present a
final solution, and it is self-evident that response relevance is a major
component of overall response quality. To demonstrate that a metric can be
produced from only sample conversations, we train an unsupervised logistic
regression model to predict response relevance and measure correlation to
human raters on a different domain.

###  3.1 Generative dialogue models

####  3.1.1 Simple baselines

We have three simple baselines. The simplest, Collapsed, returns “I don’t
know" to all inputs; this is representative of a model that has undergone mode
collapse, giving a highly generic response to all contexts Li et al. (2015);
Serban et al. (2015). The second baseline, Random, returns a random response
from the training set, representing a model that yields very natural text but
which is independent of the context. The final baseline, Gold Reference, is
the true human response from the corpus.

####  3.1.2 TF-IDF retrieval-based model

As described by Lowe et al. (2015), this retrieval model works by computing
the TF-IDF vector for each context in the training data. When a new context is
provided, its TF-IDF vector is computed, the closest context in the training
data is found using cosine similarity and, finally, the response corresponding
to the closest context is returned. Note that we used TF-IDF vectors as we
wanted to include a non-neural dialogue model in our analysis.

####  3.1.3 Generative models

#####  3.1.3.1 LSTM RNNLM

The simplest generative dialogue model used in our analysis is a RNNLM using
2000 LSTM hidden units which was pretrained by Serban et al. (2016).

#####  3.1.3.2 HRED

The second model is a hierarchical recurrent encoder-decoder. The HRED views
dialogue as being divided into utterances
$\mathbf{U_{1}},\mathbf{U_{2}},\dots,\mathbf{U_{M}}$ alternating between two
interlocutors. Each utterance decomposes into tokens such that
$\mathbf{U_{i}}=(w^{(i)}_{1},w^{(i)}_{2},\dots,w^{(i)}_{N_{i}})$. The HRED
uses three RNNs to generate the dialogue. The first RNN independently encodes
utterances to a vector representation. The second (the context RNN) encodes
the sequence of utterance vectors into the context representation. The final
RNN is the decoder, which is conditioned at all time steps on the context
vector Serban et al. (2015).

The pretrained model contains 500 GRU hidden units in the encoder RNN, 1000
GRU hidden units in the context RNN, and 500 GRU hidden units in the decoder
RNN Serban et al. (2016).

#####  3.1.3.3 VHRED

The final generative architecture is a variational hierarchical encoder-
decoder (VHRED) Serban et al. (2016). The VHRED maintains the overall design
of the HRED, except that the decoder is conditioned on the concatenation of
the context vector with a latent variable that is sampled once per decoded
utterance. The latent variable follows a multivariate normal distribution with
diagonal covariance, where the exact parameters are a learned function of the
context vector.

###  3.2 Relevance prediction model

To observe the properties of a feature-based approach, we train a model to
predict response relevance given the context. Specifically, we perform
unsupervised training of a logistic regression model, where $0$ corresponds to
maximum relevance and $1$ to minimal relevance. For a context $c$ and response
$r$, the model is defined below where $\mathbf{f}(c,r)$ is a vector of
computed linguistic features, $\mathbf{w}$ are the learned feature weights,
$b$ is a learned bias, $\sigma$ is the sigmoid function, and $y(c,r)$ is the
predicted relevance:

$y(c,r)=\sigma\left(\mathbf{w}^{T}\mathbf{f}(c,r)+b\right)$

To train the model, we developed a loss function inspired by triplet loss
(Schroff et al., 2015). Specifically, for context $c$, true human response
$r$, randomly sampled human response $r^{\prime}$, and margin $m$ (a
hyperparameter) we use the loss in Equation 2. Note that Equation 1 is triplet
loss modified to use the model prediction as a measure of distance as opposed
to the norm of two embeddings, and Equation 2 is a function of the triplet
loss specifically for logistic regression to prevent the sigmoid in the model
from driving the gradient to zero.

$\mathcal{L}_{t}(c,r)=\max\left(y(c,r)-y(c,r^{\prime})+m,0\right)$ (1)

$\mathcal{L}(c,r)=-\log\left(1+m-\mathcal{L}_{t}(c,r)\right)$ (2)

Note that as we trained this model to predict response relevance instead of
overall quality we only use a subset of our linguistic features in our
experiments.

##  4 Linguistic features

We evaluate the following linguistic features on the responses generated by
the models outlined in Section 3.1.3.

###  4.1 Acknowledgement (Ack)

To measure the degree to which the response addresses the context, the
contentful words of the response are extracted using SpaCy (i.e., nouns,
verbs, adverbs and adjectives that are not stop words) and we use WordNet
Miller (1995) to determine the percentage of these contentful words which have
a synonym within the context. If the response contains no contentful parts of
speech, then NaN is returned. When training or evaluating our unsupervised
model, we replace NaN with $0$.

###  4.2 Relatedness (Rel)

To measure the semantic relatedness of the contentful words that do not have a
synonym in the context, we use GloVe vectors pretrained on the Twitter Corpus
Pennington et al. (2014) and determine the average cosine distance between the
contentful word and the most similar word in the context. Words for which we
do not have a word vector are ignored. We empirically experiment with 25- and
200-dimensional GloVe vectors.

The motivation for looking at contentful tokens which do not have a synonym in
the context is that these tokens provide new information to the response.
Meaningful dialogue responses should add new related information. For example,
given the context “So, any hobbies? It’s soccer and badminton with me.", the
answer “My hobbies are my hobbies" fails to add information, the answer
“Asphalt" fails to add relevant information, whereas the response “Yodelling"
adds new relevant information. Note that in the context, the only token that
is truly related to “Yodelling" is “hobbies"; soccer and badminton and very
different concepts to yodelling but become part of a coherent whole through
the link of hobbies. To allow for examples such as this, in computing rel we
only consider the most similar word in the context. Thus, this metric would
ideally measure the relatedness of the new information in the response with
the context. Consequently, Rel is defined to be zero if there are no
contentful words without a synonym in the context.

###  4.3 N-gram precision

As a baseline measure of coherence and similarity, BLEU’s clipped $n$-gram
precision Papineni et al. (2002) is calculated between the stemmed response
and the stemmed context for 2-, 3-, and 4-grams. Stemming is performed using
NLTK Loper et al. (2009).

Note that this feature is very different to BLEU. First, BLEU is calculated
between the generated response and the “true" response (i.e., the gold
reference) whereas this feature is calculated between the generated response
and the context. This makes the feature context-dependent unlike BLEU, and
removes the need for a “true" human response which may not be available.
Second, unlike BLEU we stem the context and response before calculating the
precision, this helps to account for changes in verb tense or number between
the response and context. Finally, unlike BLEU we do not apply the brevity
penalty, nor average the $n$-gram precision over several values of $n$.

###  4.4 Normalized language tool (LTNorm)

To measure grammaticality, we calculate the number of LanguageTool errors
detected, normalized by response length, specifically
$1-\frac{\\#errors}{\\#tokens}$ as described by Napoles et al. (2016). To
ensure the focus on grammaticality only Grammar, Collocation, and
Capitalization errors are considered. This also reduces spurious errors such
as false positive spelling errors for technical terms in the Ubuntu corpus.

###  4.5 Neural acceptability (nn_acc)

Warstadt et al. (2018) presented a measure of acceptability using a neural
classifier to label sentences as acceptable or unacceptable, trained on a
developed corpus. The acceptability of each response is calculated using the
pretrained model from Warstadt et al. (2018).

Note that LTNorm measures (syntactic) grammaticality whereas nn_acc is meant
to measure acceptability, according to a native speaker. This distinction can
be seen in garden-path sentences that are considered grammatical, but
unacceptable. For instance, “the old man the boat" actually means that “the
old [people] staff the boat".

Note that unlike the other features, nn_acc is not interpretable – we do not
understand how the feature is calculated. However, the ensemble of features
can still be interpretable as we know the aspect it should be capturing. For
an ensemble such as logistic regression we can trace its decisions to a set of
features that are abnormally low or high. If, for instance, nn_acc is
abnormally low then we know that our model is scoring poorly as the metric
believes that the model has difficulties producing acceptable responses. From
here it is a straightforwards matter to determine if the problem lies with the
model, or the metric.

##  5 Data

We use two corpora for analysis and unsupervised training – the subsets of the
Ubuntu Corpus Lowe et al. (2015) and Twitter Dialogue Corpus (Ritter et al.,
2011) used by Serban et al. (2016) to train their VHRED, HRED, and LSTM
generative dialogue models. Dialogue is divided into contexts and responses,
where the contexts contain multiple turns (i.e., continuous sections of text
by a single person) and the response is a single turn. Turns are subdivided
into utterances – complete sentences or consecutive posts by the same speaker.
The Twitter Dialogue corpus consists of dialogues from Twitter, and the Ubuntu
Corpus from conversations from the Ubuntu technical support chat.

For evaluation we use the HUMOD Dataset (Merdivan et al., 2020) consisting of
dialogues extracted from the Cornell movie dialogues dataset (Danescu-
Niculescu-Mizil and Lee, 2011). Each dialogue is annotated with three human
ratings for the relevance of the true response, and three human ratings for
the relevance of a response selected from the dataset at random. Ratings are
on a Likert scale from $1$ to $5$, with $5$ being the most relevant. Following
(Merdivan et al., 2020), we split off a test set of $1000$ dialogues. As their
exact train/test split is not provided, we take the first $7500$ dialogues as
training data for supervised models, the following $1000$ dialogues as a
validation set, and the last $1000$ dialogues as a test set.

###  5.1 Pre- and post-processing

The raw subset of the Ubuntu Corpus comes with modest preprocessing (e.g.,
simple word tokenization, end-of-utterance and -turn tags, and the replacement
of unknown words in the response with **unknown**). We performed preprocessing
on the Twitter Corpus to replace URLs and references with fixed symbols, to
remove emoticons, and to apply NLTK’s TweetTokenizer (Loper et al., 2009). We
perform no further preprocessing for the retrieval models.

All of the contexts and responses produced by the various systems are then
further postprocessed before the linguistic features are calculated.
Specifically, we remove the end-of-utterance tags and detokenize grammatical
symbols and possesive “’s" (e.g., ‘‘Bob ’s ten . " $\rightarrow$ “Bob’s ten.")
to reduce noise in the grammaticality metrics. For the Twitter Corpus, all
responses are also converted to lower case.

As for the HUMOD dataset, it comes with no preprocessing and we performed
none.

##  6 Results

###  6.1 Details by feature

We begin with our analysis of the features. Our primary observation is
qualitative – the distribution of features across the Ubuntu and Twitter
corpora tend to be similar, indicating that a feature-based approach may
generalize well. In the following sections, we go into more detail and note
where each feature did or did not meet our expectations based on the
properties of the generative models used.

Specifically, we produce boxplots of the various distributions and perform the
paired sign test between the features of each dialogue model and the gold
standard (the true human response to the context). Pairing is by context; the
null hypothesis of the test is that the median of the difference between the
values is zero. We use a significance level of $0.05$; as we perform 60 tests
(tests for 6 dialogue models and 5 features over 2 domains) we apply
Bonferroni correction and round down our significance threshold to 8.3e-4.

####  6.1.1 Acknowledgement

Figure 1: Boxplot of Acknowledgement on the subset of data where defined. Results for the Ubuntu Corpus are plotted in white and results for the Twitter Corpus are plotted in grey. Model | $p_{\text{Ub}}$ | $\mu_{\text{Ub}}$ | $p_{\text{Tw}}$ | $\mu_{\text{Tw}}$
---|---|---|---|---
Gold Ref | - | 0.3280 | - | 0.3029
Collapsed | 4e-14* | 0.3077 | 2e-16* | 0.1922
Random | 6e-16* | 0.1506 | 2e-16* | 0.2009
TF-IDF | 6e-16* | 0.2348 | 4e-16* | 0.2341
LSTM | 4e-16* | 0.3490 | 5e-05* | 0.3327
HRED | 1.6e-03 | 0.3283 | 5.6e-02 | 0.3057
VHRED | 3.4e-02 | 0.3324 | 7.4e-01 | 0.3175
Table 1: Paired sign test on Ack values between the Gold Reference and all
other categories. Data points paired by the context that elicited the
response. The p-values on the Ubuntu and Twitter datasets are under reported
under the columns $p_{\text{Ub}}$ and $p_{\text{Tw}}$ respectively. Stars
indicate statistical significance at the Bonferroni-corrected $0.05$ level.
Mean Ack for the Ubuntu and Twitter corpora are also reported under
$\mu_{\text{Ub}}$ and $\mu_{\text{Tw}}$ respectively.

As Ack may be NaN, we only analyze Ack on a subset of the data where Ack is
defined. For the paired sign test, we only considered pairs where Ack was non-
NaN for both models. We found that Ack was defined on $100\%$ of the Twitter
data, and $88.86\%$ of the Ubuntu data. We name this subset ‘the contentful
set’. The results on the contentful set are plotted in Figure 1, the results
of the sign-test and the mean for each response source are in Table 1. As
expected, the Collapsed and Random baselines have low medians. Note that VHRED
and HRED have medians that are not significantly different from that of the
Gold Reference; as expected, the most advanced generative models are the most
similar. We also see that, with the exception of LSTM, all sources have
similar distributions across the two different domains.

####  6.1.2 Relatedness

Figure 2: Boxplot of relatedness of contentful words that do not have synonyms
in the context, specifically using 25D GLoVe vectors. White boxplots are of
the Ubuntu dataset, and grey boxplots are of the Twitter dataset. The
distribution of relatedness of the gold standard is very similar to that of
the Random Human, suggesting that we are failing to measure the semantic
relatedness of the new information in the response with the context. The
Collapsed distribution also highly corroborates this conclusion.

Unfortunately, this feature did not perform as well. From Figure 2, we can see
that the distribution of relatedness of the gold standard is very similar to
that of the Random baseline on both the Twitter and Ubuntu corpora, suggesting
that we are failing to measure the semantic relatedness of the new information
in the response to the context. We attempted to use higher dimensional word
embeddings (200D) as well as training GloVe embeddings on Ubuntu data, but the
undesired behaviour persisted. Interestingly enough, this feature does seem to
contain some useful information, as its inclusion improves performance in the
relevance prediction task.

####  6.1.3 N-gram reuse

Figure 3: Boxplot of clipped bigram reuse between stemmed response and stemmed context. White boxplots are of the Ubuntu dataset, grey boxplots are of the Twitter dataset. Model | $p_{\text{Ub}}$ | $\mu_{\text{Ub}}$ | $p_{\text{Tw}}$ | $\mu_{\text{Tw}}$
---|---|---|---|---
Gold Ref | - | 0.0554 | - | 0.0741
Collapsed | 2e-16* | 0.0697 | 4e-16* | 0.0329
Random | 2e-16* | 0.0125 | 4e-16* | 0.0319
TF-IDF | < 1e-16* | 0.0217 | 2e-16* | 0.0390
LSTM | 2e-16* | 0.0409 | 2.1e-04* | 0.0778
HRED | 7.1e-01 | 0.0650 | 1e-12* | 0.0691
VHRED | 2e-16* | 0.0564 | 4e-16* | 0.0613
Table 2: Paired sign test on 2gram values between the Gold Reference and all
other categories.

Figure 3 shows that the gold standard, HRED, and VHRED are the only models
with considerable bigram reuse. As expected, the Collapsed and Random
baselines have negligible $n$-gram reuse. In fact, Random has the lowest
average bigram reuse of all the sources. Interestingly, on the Ubuntu data,
the LSTM and TF-IDF models also have quite low bigram reuse. For TF-IDF, this
may be caused by a combination of TF-IDF vectors ignoring word order and data
sparsity. For the LSTM, the most likely explanation is the well documented
tendency of the model to produce short and generic responses. In the Ubuntu
data, approximately 64% of the LSTM responses are just 1- or 2-token
responses. As expected, Random has the lowest bigram precision.

Note that, for the Twitter dataset, there is some inflation of bigram reuse as
all tweets start with a symbol referring to a speaker and the @ sign.

Unfortunately, trigram and four-gram reuse are less useful, as over 75% of all
samples have zero reuse. We also observe that the VHRED model consistently has
a lower average $n$-gram reuse than the HRED model. This is to be expected, as
VHRED has additional stochasticity in the form of a latent variable. As such,
it is reasonable to believe that VHRED would be more likely to attempt
trajectories in the response that do not reuse $n$-grams from the context.
Unexpectedly, as seen in Table 2, HRED is the only model with a median that is
not significantly different from the gold reference, occurring only on the
Ubuntu data (on the Twitter data it has the largest $p$-value, but is still
significantly different).

####  6.1.4 Language tool errors & neural acceptability

Figure 4: Boxplots of Neural Acceptability. White boxes are boxplot over
Ubuntu dataset, and grey boxes are over the Twitter dataset.

As expected for normalized language tool errors, the median of the random
human responses is not significantly different from that of gold reference,
and the TF-IDF model is only just significantly different. Similarly, neither
TF-IDF nor Random have significantly different medians for Neural
Acceptability. We can see similar distribution in neural acceptability
regardless of domain in Figure 4.

Unsupervised Model | r | p-value
---|---|---
BLEU-4 | 0.055 | 0.08
ROUGE |  $-$0.035 | 0.26
METEOR |  $-$0.017 | 0.59
ULRoF-1 on HUMOD | 0.312 | 6.27e-24
ULRoF-1 on Twitter | 0.283 | 9.63e-20
ULRoF-1 on Ubuntu | 0.292 | 7.58e-21
ULRoF-2 on HUMOD | 0.323 | 2.58e-24
ULRoF-2 on Twitter | 0.259 | 1.67e-14
ULRoF-2 on Ubuntu | 0.303 | 6.41e-21
Table 3: Comparison of our model’s Pearson’s correlation with human raters to
that of all unsupervised baselines reported by Merdivan et al. (2020). Observe
that we have significant correlation even when the model has not seen the
domain before test-time, and has been trained on the considerably different
domain of tech-support chat logs. Note that as human raters give higher scores
for more relevant responses and ours does the opposite, we negate our model’s
relevance score before calculating the correlation.

###  6.2 Unsupervised relevance prediction

For our unsupervised relevance prediction experiments, we train our model from
Section 3.2 over 20 epochs using ADAM (Kingma and Ba, 2015) and a learning
rate of $0.1$. The performance of our unsupervised model, as compared to all
unsupervised baselines reported by Merdivan et al. (2020), is in Table 3. Note
that we tried two variants of our Unsupervised Logistic Regression of Features
(ULRoF) model, ULRoF-1, which only uses Ack as well as the n-gram precisions,
and ULRoF-2, which also uses Rel (note that we computed two Rel values, with
the 25D and 200D vectors respectively, as we found it improved performance on
the validation set). We find that our trained models generalize well, even
when they are trained on Twitter or Ubuntu data and thus have absolutely no
knowledge of the HUMOD domain. This is particularly impressive as in addition
to the difference in domain there are also differences in pre-processing
between the Twitter, Ubuntu, and HUMOD datasets.

Compared to the supervised baseline metrics reported by Merdivan et al.
(2020), we perform significantly better than the second-best supervised
baseline, HAN-R(MSE) with a reported correlation of 0.128, but worse than the
supervised BERT baseline that achieved a reported correlation of $0.602$.
However, the BERT metric was both fine-tuned to HUMOD and the predictions were
trained in a supervised manner. Given that our model is more interpretable,
trained without any human annotated data, and performs domain generalization,
these results nevertheless demonstrate that using classic techniques in this
novel way holds promise. Future work in this direction is merited.

As a followup experiment, we train ULRoF on BERT features, dubbed ULRoF-BERT.
To ensure a fair comparison with the Merdivan et al. (2020) baseline we re-
implemented their supervised BERT baseline – a single-layer 5-way classifier
on BERT features. We use ADAM with a batch size of $6$, and a learning rate of
$0.00005$ when fine-tuning ($0.001$ otherwise). We use bert-base-uncased from
HuggingFace Wolf et al. (2019) as our pretrained BERT. We train all models for
20 epochs; we checkpoint after each epoch and test the checkpoint with the
highest performance on our HUMOD validation set.

We train the baseline and ULRoF-BERT architectures both with and without fine
tuning of the BERT model (i.e., updating the pretrained weights). To determine
the generalization of the unsupervised models we train on HUMOD, Ubuntu, and
Twitter data before determining the model correlation with the human
annotations on the HUMOD test set (see Table 4). We found poor performance
using randomly samples as negative examples, and instead used “I don’t know”
as our negative examples.

We immediately note that allowing the unsupervised model to fine-tune improves
performance when training on HUMOD data, but causes catastrophic drops in
performance when training on data from another domain. As expected, the full-
capacity model overfits to the train domain. It is clear that we are
overfitting to the training domain and not the specifics of the training set
as we achieve excellent performance when the train and test data are both from
the HUMOD domain. Next, we note ULRoF-BERT achieves good performance and much
better generalization between domains. This demonstrates that even a
simplistic model such as ULRoF can achieve good performance given the proper
features. It also shows that our hand-engineered features vary less by domain
than BERT features as ULRoF-BERT’s drop in performance is two to five times
worse than in either ULRoF-1 or ULRoF-2. Finally, the fact that BERT features
are intended to be very general and that we achieve good performance without
tuning them lends support to our proposal for the use of general linguistic
features as input to a low-capacity model. What remains an open question is
how scalably a hand-engineered set of features can be developed. Even if the
hand-engineered approach proves to be impractical, our work suggests that
less-interpretable but highly generic text features combined with unsupervised
learning is another promising path to a dialogue evaluation metric that
maintains the desiderata of not requiring human annotated data while avoiding
overfitting to domain; albeit now without the benefit of interpretability.

| Training Data Domain
---|---
Model | HU. | Ub. | Tw.
Sup. BERT + FT | 0.563 | - | -
Sup. BERT | 0.627 | - | -
Unsup. BERT + FT | 0.651 | 0.030 | 0.240
ULRoF-BERT | 0.591 | 0.487 | 0.492

Table 4: Pearson’s correlation between average human rating and model
predictions on the HUMOD test set for relevance BERT models. Row is the
relevance model with +FT used to indicate whether the BERT model was fine-
tuned. Column is the domain of the training data. The architectures and loss
of the last two rows are the same, the difference is whether BERT is fine-
tuned. Note that by not fine-tuning BERT we achieve far better domain
generalization.

##  7 Conclusions

In this work, we outline known shortcomings affecting existing dialogue
metrics, and propose a feature-based approach to circumvent them. We
demonstrate that feature-based approaches can be well-suited for zero-shot
generalization between domains, both by an analysis of the features and by
training a model on the related task of relevance prediction. By using a low-
capacity model with well-chosen features, we can generalize well to unseen
domains. Furthermore, this approach has the desirable property of being easily
interpreted. At test time, the exact value, meaning, and definition of each
feature is known, and with a model such as ULRoF we also know the relative
importance of each feature, as encoded by regression weights.

Our ULRoF experiments also show that training and evaluation can be done
without gold-standard responses at test time, and without expensive human-
annotated data. Although our ULRoF experiment is for relevance prediction, the
triplet loss can be modified for dialogue evaluation by introducing additional
types of negative responses, such as repeating the context in response, or
jumbling the word order. We believe that the low capacity of the model,
combined with generic features, should allow such an approach to avoid the
overfitting problems that would impact a deep model trained in this manner.

Now that we have demonstrated that this approach is sound in principle, future
work will include the determination of a full set of linguistic features
appropriate for dialogue evaluation (particularly features regarding dialogue
acts and dialogue breakdowns Higashinaka et al. (2015); Chinaei et al.
(2017)), to extend our existing training procedure to train for overall
response quality instead of relevance, and to obtain human response quality
scores for dialogues from a variety of domains so that the final set of
features can be fully evaluated and compared to existing methods.

## References

  * Chinaei et al. (2017) H. Chinaei, L.C. Currie, A. Danks, H. Lin, T. Mehta, and F. Rudzicz. 2017.  Identifying and avoiding confusion in dialogue with people with Alzheimer’s disease.  _Computational Linguistics_ , 43(2):377–406.
  * Danescu-Niculescu-Mizil and Lee (2011) Cristian Danescu-Niculescu-Mizil and Lillian Lee. 2011.  Chameleons in imagined conversations: A new approach to understanding coordination of linguistic style in dialogs.  In _Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, ACL 2011_.
  * Deriu and Cieliebak (2019) Jan Milan Deriu and Mark Cieliebak. 2019.  Towards a metric for automated conversational dialogue system evaluation and improvement.  In _Proceedings of the 12th International Conference on Natural Language Generation_ , pages 432–437, Tokyo, Japan. Association for Computational Linguistics.
  * Galley et al. (2015) Michel Galley, Chris Brockett, Alessandro Sordoni, Yangfeng Ji, Michael Auli, Chris Quirk, Margaret Mitchell, Jianfeng Gao, and Bill Dolan. 2015.  deltaBLEU: A discriminative metric for generation tasks with intrinsically diverse targets.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing of the Asian Federation of Natural Language Processing, ACL 2015, July 26-31, 2015, Beijing, China, Volume 2: Short Papers_ , pages 445–450. The Association for Computer Linguistics.
  * Higashinaka et al. (2015) Ryuichiro Higashinaka, Masahiro Mizukami, Kotaro Funakoshi, Masahiro Araki, Hiroshi Tsukahara, and Yuka Kobayashi. 2015.  Fatal or not: Finding errors that lead to dialogue breakdowns in chat-oriented dialogue systems.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 2243–2248. Association for Computational Linguistics.
  * Kannan and Vinyals (2017) Anjuli Kannan and Oriol Vinyals. 2017.  Adversarial evaluation of dialogue models.  _CoRR_ , abs/1701.08198.
  * Kingma and Ba (2015) Diederik P. Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_.
  * Li et al. (2015) Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015.  A diversity-promoting objective function for neural conversation models.  _CoRR_ , abs/1510.03055.
  * Liu et al. (2016) Chia-Wei Liu, Ryan Lowe, Iulian Vlad Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016.  How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation.  _CoRR_ , abs/1603.08023.
  * Loper et al. (2009) Edward Loper, Ewan Klein, and Steven Bird. 2009.  _Natural Language Processing with Python_.  O’Reilly Media, Inc.
  * Lowe (2018) Ryan Lowe. 2018.  Personal communication.
  * Lowe et al. (2017) Ryan Lowe, Michael Noseworthy, Iulian Vlad Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017.  Towards an automatic turing test: Learning to evaluate dialogue responses.  _CoRR_ , abs/1708.07149.
  * Lowe et al. (2015) Ryan Lowe, Nissan Pow, Iulian Serban, and Joelle Pineau. 2015.  The Ubuntu dialogue corpus: A large dataset for research in unstructured multi-turn dialogue systems.  _CoRR_ , abs/1506.08909.
  * Lowe et al. (2016) Ryan Lowe, Iulian Vlad Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016.  On the evaluation of dialogue systems with next utterance classification.  _CoRR_ , abs/1605.05414.
  * Merdivan et al. (2020) Erinc Merdivan, Deepika Singh, Sten Hanke, Johannes Kropf, Andreas Holzinger, and Matthieu Geist. 2020.  Human annotated dialogues dataset for natural conversational agents.  _Applied Sciences_ , 10(3):762.
  * Miller (1995) George A. Miller. 1995.  Wordnet: A lexical database for english.  _Commun. ACM_ , 38(11):39–41.
  * Napoles et al. (2016) Courtney Napoles, Keisuke Sakaguchi, and Joel R. Tetreault. 2016.  There’s no comparison: Reference-less evaluation metrics in grammatical error correction.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 2109–2115. The Association for Computational Linguistics.
  * Papineni et al. (2002) Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002.  BLEU: A method for automatic evaluation of machine translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ACL ’02, pages 311–318, Stroudsburg, PA, USA. Association for Computational Linguistics.
  * Pennington et al. (2014) Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014.  GloVe: Global vectors for word representation.  In _Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543.
  * Ritter et al. (2011) Alan Ritter, Colin Cherry, and William B. Dolan. 2011.  Data-driven response generation in social media.  In _Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing_ , pages 583–593, Edinburgh, Scotland, UK. Association for Computational Linguistics.
  * Sai et al. (2019) Ananya B. Sai, Mithun Das Gupta, Mitesh M. Khapra, and Mukundhan Srinivasan. 2019\.  Re-evaluating ADEM: A deeper look at scoring dialogue responses.  In _The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019_ , pages 6220–6227. AAAI Press.
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015.  Facenet: A unified embedding for face recognition and clustering.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 815–823.
  * Serban et al. (2015) Iulian Vlad Serban, Alessandro Sordoni, Yoshua Bengio, Aaron C. Courville, and Joelle Pineau. 2015.  Hierarchical neural network generative models for movie dialogues.  _CoRR_ , abs/1507.04808.
  * Serban et al. (2016) Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2016.  A hierarchical latent variable encoder-decoder model for generating dialogues.  _CoRR_ , abs/1605.06069.
  * Warstadt et al. (2018) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2018.  Neural network acceptability judgments.  _CoRR_ , abs/1805.12471.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.