# Actionable Conversational Quality Indicators for Improving Task-Oriented
Dialog Systems

Michael Higgins  Dominic Widdows  Chris Brew  Gwen Christian  Andrew Maurer  Matthew Dunn  Sujit Mathi  Akshay Hazare  George Bonev  Beth Ann Hockey  Kristen Howell  Joe Bradley  mhiggins | dwiddows | cbrew | amaurer | gchristian | mdunn | smathi | ahazare | gbonev | bhockey | khowell | jbradley @liveperson.com LivePerson Inc.

###### Abstract

Automatic dialog systems have become a mainstream part of online customer
service. Many such systems are built, maintained, and improved by customer
service specialists, rather than dialog systems engineers and computer
programmers. As conversations between people and machines become commonplace,
it is critical to understand what is working, what is not, and what actions
can be taken to reduce the frequency of inappropriate system responses. These
analyses and recommendations need to be presented in terms that directly
reflect the user experience rather than the internal dialog processing.

This paper introduces and explains the use of Actionable Conversational
Quality Indicators (ACQIs), which are used both to recognize parts of dialogs
that can be improved, and to recommend how to improve them. This combines
benefits of previous approaches, some of which have focused on producing
dialog quality scoring while others have sought to categorize the types of
errors the dialog system is making.

We demonstrate the effectiveness of using ACQIs on LivePerson internal dialog
systems used in commercial customer service applications, and on the publicly
available CMU LEGOv2 conversational dataset (Raux et al.,, 2005). We report on
the annotation and analysis of conversational datasets showing which ACQIs are
important to fix in various situations.

The annotated datasets are then used to build a predictive model which uses a
turn-based vector embedding of the message texts and achieves a 79% weighted
average f1-measure at the task of finding the correct ACQI for a given
conversation. We predict that if such a model worked perfectly, the range of
potential improvement actions a bot-builder must consider at each turn could
be reduced by an average of 81%.

##  1 Introduction and Outline

Customer service dialog systems have become widely-used in many everyday
settings, but still make frustrating errors that are sometimes obvious to a
human — for example, failing to understand a customer’s message and asking an
irrelevant question as a result. In practice, tuning these systems to limit
these behaviors is an expensive and time-consuming art. This paper describes
the design, implementation and early results of an approach to improving
overall dialog system quality by recognizing and addressing such individual
failures. This is done by combining individual Actionable Conversational
Quality Indicators (ACQIs) with a running Interaction Quality score (IQ), to
show which problems were identified, what steps were taken to fix them, and
how these changes affected an overall assessment of the user experience. IQ is
a standard conversational quality measure developed by Schmitt and Ultes,
(2015). ACQIs are designed so that each conversational quality indicator has
associated recommended actions, and are introduced in this paper.

The paper starts by explaining some of the background on how task-oriented
dialog systems are built and maintained (Section 2): while there are several
online tools that support this, many readers may be unfamiliar with their use.
Section 3 introduces the datasets that are used for examples and experiments
throughout the rest of the paper.

Previous works on evaluating dialog system performance (discussed in Section
4) have investigated the use of conversation-level quality metrics, and
individual turn-level assessments of good and bad interactions. In particular,
the Interaction Quality (IQ) score of Schmitt and Ultes, (2015) is explained
in this section, and the combined use of ACQIs and IQ score plays a major role
throughout the rest of the paper.

Section 5 explains the heart of this paper: the design of the ACQI
taxonomy.111The term ‘taxonomy’ is used here to mean an agreed categorization,
and is used for classifications of dialog systems, intents, and failure
states, in addition to the quality indicators and actions of the ACQI
taxonomy. There are not always tree-like ‘taxonomic’ relationships between
these categories, though sometimes there are. This explains the motivation and
decisions behind ACQIs, including how they are made to be actionable,
explainable, and to give feedback that is specifically tailored to the dialog
system in question. Section 6 describes the annotation work for ACQI datasets,
analysis of the ACQI distributions, and the work on combining ACQI and IQ
scoring to distinguish those parts of the dialog that need particular
improvement.

Section 7 describes experiments on automatically predicting ACQIs based on the
annotated datasets and features extracted from the dialog text. We demonstrate
that correct ACQI labels can be predicted with a weighted average f1-score of
79%, and demonstrate the effectiveness of textual features to predict labeled
IQ scores (1 to 5) with an average accuracy of 60%. Analyzing the distribution
of ACQIs and suggested actions when IQ drops shows that the number of
potential improvement strategies to evaluate could be reduced by up to 81%,
depending on the accuracy of the ACQI and IQ classifiers. We argue that tools
built using these approaches could improve the effectiveness and reduce
cognitive burden on bot-builders.

##  2 Conversational AI Systems and Challenges for Bot-Builders

Since the year 2000, dialog systems (often called chatbots) have gone from
mainly research demonstration systems to include various user-facing
commercial offerings. Dialogue systems fall into three broad categories (Deriu
et al.,, 2020): conversational agents, question answering systems, and task-
oriented systems (which are the topic of this paper). Each category has
corresponding dialog quality measurement strategies. Conversational agents,
which often receive the most attention in news articles when released, are
typically unstructured and open-domain, with no particular objective other
than an engaging conversation. Question answering systems can be evaluated by
considering the accuracy of answers given. Task-oriented systems, which are
the focus of this article, typically have a rigid structure and a limited
scope. They are built to resolve the consumer’s issue, answer questions, route
to an appropriate representative, or guide the user through a task as
efficiently as possible. Like question answering systems, task-oriented
systems have clear ‘failure’ cases: just as a question answering system can
fail to answer a question, a task-oriented system can fail to complete a task.

Task-oriented systems are prevalent in customer service: ideally, they can
automate simple tasks like routing requests to the right agent, freeing up
human customer service specialists to handle more demanding situations.
Several technology companies offer dialog system services to support such
chatbots. These include offerings from large and general technology companies
such as Microsoft LUIS, IBM Watson, Google DialogFlow, and from more
specialist providers such as Salesforce, Intercom, and LivePerson.

While there are several differences between these platforms, there are some
typical themes:

  * •

The platforms provide various tools and widgets for customers to build and
deploy their own dialog systems.

  * •

The dialogs built in this way are ‘designed’ or ‘scripted’. The process of
building a dialog involves declaring various steps, inputs the user can be
encouraged to make at those steps, and what action the system should take in
response.

  * •

This process admits the possibility of error or failure states, when then
input from a user is something the system (knowingly or unknowingly) cannot
process successfully.

A simple example might be that the dialog is at a stage where the user is
asked to pick from a list of available options, such as “Enter 1 for English,
Ingrese 2 para Español”. In this case, any entry other than the numbers ‘1’ or
‘2’ would be a problem. This problem can be addressed in a few ways. A
traditional keyboard interface might say “Option unrecognized, please type 1
or 2.” A form-filling approach might be to use a “Select from Menu” element
instead of a “Prompted Text Entry” element, so that the customer can only give
input that corresponds to the actions the dialog system can take right now. An
example from the LivePerson Conversation Builder is shown in Figure 1. The
user is about to add an new interaction to the conversation so far, has a
choice of widgets including those for text entry, scheduling, payments, and
various formats for asking questions and tracking the responses. There are
analogous features in other bot-building platforms.

Figure 1: A bot-building interface where the bot-builder is about to add a
multiple choice question

It is crucial to note that dialog system platforms are typically used by
customer support specialists, not the customers themselves. In the rest of
this paper, these users will be described using the colloquial but industry-
wide term ‘bot-builders’. This is emphasized because one of the ways to
improve these dialog systems is to provide tools that make bot-builders more
effective. The work described in this paper shows that two established
approaches to measuring conversations, scoring individual actions and
assessing the conversation quality as a whole, can be combined to provide such
a tool that helps bot-builders to identify and fix particular pain-points in a
dialog system. The intuitively appealing assumption that useful actionable
feedback about dialog systems can be produced through (semi-) automated
methods is supported by Hockey et al., (2003), which shows that users of a
task-oriented dialog system who received actionable feedback in failure cases
outperformed the control group that did not.

##  3 Datasets Used in This Work

The datasets used in this paper are introduced here and referred to throughout
this paper. The following two datasets were annotated and used in the
prediction experiments in subsequent sections. Statistics about these datasets
are summarized in Table 1.

###  3.1 LEGOv2

LEGOv2 represents a portion of the CMU Let’s Go Public DataSet (Raux et al.,,
2005, 2006), which was instrumental in the development of the Interaction
Quality (IQ) score of Schmitt and Ultes, (2015), explained in Section 4.

Let’s Go Public is a record of phone-mediated customer-service interaction
between an automated dialogue system and callers drawn from the general
population in the vicinity of Pittsburgh, Pennsylvania. It has been studied
intensively since its initial creation, and subsets have been used, in
particular, to support analysis and prediction of IQ (Stoyanchev et al.,,
2017).

The LEGOv2 dataset still represents many of the challenges of deploying task-
oriented spoken language systems ‘in the wild’:

  * •

The callers are drawn from the general population.

  * •

The task-at-hand is authentic: callers have a presumed need to access bus
information.

  * •

The callers are using standard personal or public telephones in real-world
settings that include such challenges as third party speech, television
programs in the background, very variable audio quality, and irritated and or
amused speech.

Things that have changed since the creating of LEGOv2 include:

  * •

Speech recognition technology is far more capable now than it was when Let’s
Go began.

  * •

The public is much more familiar with conversational AI systems.

  * •

Both speech-mediated and text-mediated dialogue systems are commercially
important and widely deployed, so lessons learned from Let’s Go have greater
potential impact.

Previous approaches for modeling quality in LEGOv2 have included features
automatically extracted from the speech recognition and dialog system
(Stoyanchev et al.,, 2017). We have chosen to use only features derived from
the text itself from a transcribed version of LEGOv2, as this approach is
applicable to text-based dialog systems including those developed by
LivePerson, and avoids using system implementation decisions as features from
the dataset itself.

###  3.2 Datasets from LivePerson dialog systems

An approved222The approval and annotation process includes checking all
aspects of legal compliance and scrubbing personally identifiable information
(PII) from messages. collection of transcripts of customer-service
conversations with LivePerson dialog systems was also extracted and annotated.

We have intentionally chosen a set of bots that serve different functions,
come from different industries, and have different overall quality (as
measured by final IQ score). To this end, 130 conversations were chosen from
each of 4 dialog systems, giving a total of 520 conversations, a comparable
number of conversations to those used from the LEGOv2 dataset, though the
LivePerson conversations themselves on average have fewer turns (Table 1).

These dialog systems often respond with structured content, such as embedded
HTML with buttons, toggles, or drop-down menus. In these cases we represent
the text of the options separated by ‘***’. For example: “BUTTON OPTIONS ***
Main Menu *** Pick a Color *** Pick a different item”.

Table 1: Summary description and statistics on parts of CMU LEGOv2 and LivePerson datasets annotated and used in this work Bot | Vertical | Bot Functions | Avg # Turns | # Conversations
---|---|---|---|---
LEGOv2 | Travel | Transactional | $13.67\pm 11.25$ | $541$
Junior Sales Assistant | Retail | Data Gathering, Routing | $4.95\pm 2.13$ | 130
Help and Route | Travel | Routing, FAQ | $5.89\pm 3.71$ | 130
Router | Tech | Routing | $2.25\pm 1.45$ | 130
Food Expert | Food Industry | Transactional (Ordering) | $5.44\pm 3.31$ | 130

##  4 Previous Approaches to Evaluating Dialogue Systems, Including
Interaction Quality (IQ)

Meaningful evaluation of automated dialogue system performance has long been
recognized as crucial to progress in dialogue system research Deriu et al.,
(2020) and is an active area of development, including the recent introduction
of the ‘sensibleness and specificity’ metric of Adiwardana et al., (2020).
Since the introduction of bot-building platforms as described in Section 2, it
has also become a daily concern for customer service bot-builders.

One of the most common industry measures for dialog system effectiveness is
the automation rate or containment rate. Containment rate directly affects the
cost savings that a business may make through automation, for example “Over
three years and a conservative 25% containment rate, the cost savings is worth
more than $13.0 million to the organization (Forrester Research,, 2020).”
However, analysis of containment or automation rates is still relatively rare
in the research literature, and is mainly found in evaluation of speech /
voice driven dialog systems (Pieraccini et al.,, 2009). The tradeoff between
the automation rate and the number of setbacks a user can encounter is
analysed in Witt, (2011) — also in speech. The relative lack of research
literature on containment rates can be easily attributed to different settings
and incentives. The case study presented in Forrester Research, (2020) notes
that “Over three years and a conservative 25% containment rate, the cost
savings is worth more than $13.0 million to the organization.” By contrast,
most dialog systems in research settings do not have human agents to respond
to escalations, so the systems cannot escalate, and containment cannot be
measured.

For the task-oriented dialogue systems that are our primary concern, there is
an underlying concept of giving a ‘right’ response. Early evaluation work
based on this idea compared the actual responses to a predefined key of
reference answers (Hirschman et al.,, 1990). The portion of matches to the key
gave the measure of performance. Well known weaknesses of this approach
include being specific to particular systems, domains and dialogue strategies.
More portable strategies that measure inappropriate utterance ratio, turn
correction ratio, or implicit recovery (Danieli and Gerbino,, 1995; Hirschman
and Pao,, 1993; Polifroni et al.,, 1992; Shriberg et al.,, 1992) are
intellectual ancestors of the ACQI part of our approach, in that they identify
events that are indicators of the quality of the conversation. Both of these
early approaches share the limitation of being unable to model or compare the
contribution that the various factors have on performance.

The PARADISE approach (Walker et al.,, 1997) overcomes this limitation by
using a decision-theoretic framework to specify the relative contribution of
various factors to a dialogue system’s overall performance. This and other
ideas introduced by PARADISE, such as separating the accomplishment of a task
from how the system does it, support evaluation that is portable across
different systems, domains and dialogue strategies. We have tried to emulate
some of these best practices by defining ACQIs that avoid being too
implementation-specific, though without insisting that all ACQIs must be
relevant to all dialog systems (see Section 5). For example, in contrast with
Stoyanchev et al., (2017), we have chosen not to utilize any features
extracted from the dialog systems themselves (e.g. failed intent match from
the natural language understanding system) and rely solely on the text of the
dialog.

###  4.1 Interaction Quality (IQ) Score

More recent work has developed several dialog quality measurement strategies
which are categorized by Bodigutla et al., (2019) as: sentiment detection; per
turn dialog quality (e.g. Interaction Quality (Schmitt et al.,, 2012) and
Response Quality (RQ)); explicitly soliciting feedback from the user; task
success; and dialog-level satisfaction ratings as in PARADISE (Walker et al.,,
1997). These methods are useful but have well-known limitations: for example,
sentiment analysis on messages misses many problems, there is response bias in
user polling, and negative outcomes weigh more in the consumer’s mind than
positive ones (Han and Anderson,, 2020).

The method in this paper most directly builds on the IQ method of Schmitt et
al., (2012); Stoyanchev et al., (2019). The methods involve annotating a
conversation, typically adding one point for a good interaction and
subtracting a point for a bad interaction, with some exceptional cases, for
example when the dialog ‘obviously collapses’. A benefit of these methods is
that they can help to identify where there are problems in a dialog system:

While the intention of PARADISE is to evaluate and compare SDS or different
system versions among each other, it is not suited to evaluate a spoken
dialogue at arbitrary points during an interaction. (Schmitt et al.,, 2012)

Our work can be seen as an extension of the IQ approach (Schmitt et al.,,
2011) in our use of a running dialogue quality measure. ACQI improves over IQ
by recommending how to resolve a problem instead of only identifying where it
exists.

##  5 The Design of the ACQI Taxonomy

This central section describes the design of the ACQI taxonomy and how it
improves over the IQ system introduced above.

A running score like IQ allows bot-builders to identify dialog system
responses where there is a meaningful decrease in score, but does not provide
direct diagnosis of the problems that may be there. If nothing other than the
running score is available, bot-builders have little option other than to
manually review, form their own taxonomy of failure causes, and come up with
appropriate fixes. This process in practice typically takes days or weeks to
complete, and is prioritized largely by operator intuition.

To make the problems explicit and to suggest solutions, we introduce
Actionable Conversation Quality Indicators or ACQIs. ACQIs highlight moments
in chatbot conversations that impact customer experience. Our ACQI taxonomy
can be found in Table 2. Note that for each ACQI, there is an associated set
of an actions a bot-builder can take to mitigate the issue. The ACQIs and the
taxonomy dimensions are all derived from analyzing the user experiences
directly. As well as making them actionable, this is motivated by wanting to
make the issues aggregable, so that bot-builders can analyze aggregated
statistics about conversations and prioritize fixing the most prevalent issues
accordingly. By exposing predicted ACQIs in an appropriately aggregated
format, we empower bot-builders to make more data-driven decisions when
improving their dialog systems. Not all ACQIs are relevant to all dialog
systems: for example, the CMU LEGOv2 system does not support transfer to a
human agent, so ‘Bad Transfer’ cannot occur (though ‘Set Transfer
Expectations’ can still potentially be useful, even to say that there are no
human agents available).

Table 2: ACQIs with their associated actions in example text and spoken dialog
systems

ACQI Description LivePerson Action CMU LEGOv2 Action Does Not Understand The
bot says that it does not understand the consumer’s response and says
something like ‘I didn’t get that’ or ‘I don’t understand’. This also includes
when the bot’s response is incorrect and it is clear that the bot has
misunderstood what the consumer said. Add/Remove Features, Update NLU
Taxonomy, Update NLU Training Data, Add/Remove/Update Confirm Consumer
Statement(s), Add FAQs, Add/Update Bot Statements, Add/Update Bot Prompts,
Add/Update Max No Match Behavior Add/Remove Features, Update ASR Training
Data, Update NLU Taxonomy, Update NLU Training Data, Add/Remove/Update Confirm
Consumer Statement(s), Add/Update Consumer-Requested Repeat, Add FAQs,
Add/Update Bot Statements, Add/Update Bot Prompts, Add/Update Max No Match
Behavior Input Rejected The bot does not accept a consumer response to menu
options, forms, or other structured content, such as ‘1, 2, 3, 4’ , ‘a, b, c,
d’, or ‘yes, no‘. Improve Response Flexibility, Add/Update Bot Statements,
Add/Update Bot Prompts, Add/Update Max No Match Behavior Improve Response
Flexibility, Add/Update Bot Statements, Add/Update Bot Prompts, Add/Update Max
No Match Behavior Ignored Consumer The dialog system does not recognize a free
text consumer response and asks an unrelated question. Set Delay Expectations,
Enable/Disable Context Switching, Add FaaS code, Confirm Successful Bot
Handoff, Handle Backend Errors Set Delay Expectations, Enable/Disable Context
Switching, Add/Update Turn-Taking Signals, Add FaaS code, Add/Update Max No
Input Behavior, Handle Backend Errors Restart When explicitly asked for by the
consumer and only used when the conversation starts over within the same
engagement with the bot. Includes when the bot goes back to the main menu.
Add/Remove/Update Confirm Consumer Statement(s), Add/Update Conversational
Navigation, Add/Update Help, Add/Update Max Restart Behavior Add/Remove/Update
Confirm Consumer Statement(s), Add/Update Conversational Navigation,
Add/Update Help, Add/Update Max Restart Behavior Bad Transfer The dialog
system attempts to transfer the consumer to an agent, but either leaves them
hanging or abruptly ends the chat. It might also fail to tell them EARLY
enough in the conversation that there are no agents available at that hour.
Set Transfer Expectations, Confirm Successful Transfer Transfer is not
possible in the CMU system, since its only use is out of hours. Set Transfer
Expectations makes sense, but the issue never arose, because Bad Transfer was
never used. Unable to Resolve The bot explicitly states that it cannot provide
the information the consumer requested and does not offer to transfer to an
agent. Add/Remove Features, Offer External Solutions, Set Transfer
Expectations Add/Remove Features, Offer External Solutions, Set Transfer
Expectations Ask for Information The bot asks the consumer for information.
Remove Unnecessary Questions, Set Delay Expectations, Enable/Disable Context
Switching, Add FaaS code, Confirm Successful Bot Handoff, Handle Backend
Errors Remove Unnecessary Questions, Set Delay Expectations, Enable/Disable
Context Switching, Add/Update Turn-Taking Signals, Add FaaS code, Add/Update
Max No Input Behavior, Handle Backend Errors Provide Assistance Bot responds
to consumers query, attempting to fulfill their intent. Add/Remove/Update
Validate Bot Resolution, Add/Remove Features, Update NLU Taxonomy, Update NLU
Training Data, Add/Remove/Update Confirm Consumer Statement(s), Add FAQs,
Add/Update Bot Statements, Add/Update Bot Prompts Add/Remove/Update Validate
Bot Resolution, Add/Remove Features, Update ASR Training Data, Update NLU
Taxonomy, Update NLU Training Data, Add/Remove/Update Confirm Consumer
Statement(s), Add FAQs, Add/Update Bot Statements, Add/Update Bot Prompts Ask
for Confirmation Bot asks for user confirmation of input. Includes when the
bot asks if the information provided is correct. Update NLU Taxonomy, Update
NLU Training Data, Add/Remove/Update Confirm Consumer Statement(s), Add/Update
Bot Statements, Add/Update Bot Prompts Update ASR Training Data, Update NLU
Taxonomy, Update NLU Training Data, Add/Remove/Update Confirm Consumer
Statement(s), Add/Update Bot Statements, Add/Update Bot Prompts

The ACQI taxonomy is inspired from a variety of sources, including literature
on the (human) evaluation of open-domain dialog systems, consultation with
experts, and ongoing feedback from expert users involved in bot-building.

We carried out a series of interviews with bot-builders and assembled a
repertoire of tuning actions based on their practices. We identified 28
distinct actions bot-builders take at LivePerson. In the case of LEGOv2, we
consulted with 2 domain experts and identified 31 actions. For each of our
ACQIs we assigned a set of actions that the bot-builder may make (Table 2).
For each of our ACQIs we assigned a set of actions that the bot builder may
take (Table 2). The mapping showed that our ACQIs could guide bot builders to
take 23 of 28 possible actions for LivePerson systems, and 25 of 31 possible
actions for CMU.

The mapping from ACQIs to actions was created by a LivePerson expert in
improving dialog systems, and later refined with further feedback from
LivePerson bot-builders. Because IQ+ACQI will be aggregated and used to
improve consumer experience (CX) we put an explicit emphasis on CX and
actionability.

Finch and Choi, (2020) list 21 dimensions across 20 publications that human
evaluators have used to measure the quality of open-domain dialog systems.
While these dimensions are not used directly, they partially inspire our
taxonomy of failure states (Table 5, leftmost column). For instance the
‘Doesn’t Understand’ failure state is inspired by Coherence (Luo et al.,,
2018; Wu et al.,, 2019), Correctness (Liu et al.,, 2018; Wang et al.,, 2020),
Relevance (Moghe et al.,, 2018; Lin et al.,, 2019; Qiu et al.,, 2019), Logic
(Li and Sun,, 2018) and Sensibleness (Adiwardana et al.,, 2020). Other
elements of our taxonomy are informed by Jain et al., (2018) which provides a
set of best practices when developing dialog systems for messaging based bots.
We separated ‘misunderstanding’ into ‘Input Rejected’, ‘Ignores Consumer’ and
‘Does Not Understand’ categories, because each of these require different
actions that can mitigate the understanding issue. Such separation of failure
states highlights the design-for-actionability of the ACQI taxonomy.

###  5.1 Desirable Properties of a ACQI Taxonomy

A good ACQI taxonomy should be actionable, easy to understand, have highly
bot-dependent ACQI incidence rates, and have a significant impact on consumer
experience. In the following sections we will justify these properties, and
provide measurement strategies where appropriate. For some ACQIs and dialog
systems it may be appropriate to bypass the need for annotation by
automatically extracting ACQIs from the system logs. An example for this would
be an ACQI that indicates that the NLU returned a confidence score less than a
predefined threshold and responded with a request for the user to rephrase
their intent.

####  5.1.1 Actionable

Without actionable ACQIs, bot-builders are left to the time consuming process
of reviewing large amounts of transcripts and/or a careful analysis of the
model’s feature space, from which they can attempt to deduce what mitigation
strategies are appropriate. Unfortunately many features are not actionable
from the bot-builder’s perspective. For instance, conversation length can be
highly predictive of conversation quality, but from their perspective, there
are no clear steps to uniformly reduce conversation length. The problem is
that conversation length is not the cause of a bad conversation, it is a
consequence of it.

For example, for the dialog system ‘Food Expert’, the initial turn was
predicted to have ignored a consumer’s initial intent. This can lead to longer
conversations, as can an order with complicated modifications, but the
associated dialog quality improvement strategies for the operator are quite
different. ACQI allows for separation and sizing of these situations: the
second is varied and complicated, whereas the first is a single problem with a
negative affect on IQ. From the bot-builder’s perspective this one is a
relatively easy fix: they just need to make sure that intent recognition is
applied to the first customer utterance.

####  5.1.2 Easy to understand

For bot-builders, understanding is a necessary condition for fixing an issue.
The terms used in the ACQI taxonomy are deliberately chosen to be familiar to
bot-builders, and have been refined with use to add clarity where requested.
The relative success of this effort is reflected in the encouraging inter-
annotator agreements reported in Table 4 below. However, this finding is
potentially influenced partly by the use of experts in bot-building and
conversation modeling as annotators. This has yet to be corroborated with less
experienced bot-builders.

####  5.1.3 Bot Dependent ACQI Rates

As the purpose of the ACQI taxonomy is to guide bot-builders to appropriate
fixes it is imperative that the rates at which ACQIs are present are bot-
dependent. From Figure 3 we can see this is indeed the case. As our ACQI
taxonomy is meant to fit a wide breadth of task-driven dialog systems, teams
working on very specific dialog systems may wish to use a subset, or create
their own smaller taxonomy. For instance, if the dialog system does not allow
for transfers (this is the case for LEGOv2) ‘Bad Transfer’ should be excluded
from the taxonomy.

####  5.1.4 Impact on Customer Experience

Each ACQI should have an actionable relevance to CX. Figure 4 illustrates this
property for our ACQI taxonomy as each score change distribution differs from
the overall distribution (topmost bar). Several elements of the taxonomy only
indicate a poor customer experience in particular circumstances. For example,
the ‘Ask for Confirmation’ is more likely to indicate a negative CX when it
occurs multiple times within a given dialog (Figure 5).

This concludes the summary of the ACQI taxonomy itself. In practice we found
that ACQIs were most reliably predicted and effectively used in combination
with a running quality score. This work is described next.

##  6 Combining ACQIs with IQ in Conversational Datasets

The section describes the work done on annotating ACQIs along with IQ, which
turned out to be necessary for distinguishing those ACQIs that warrant action.
This is due to the fact that dialog context matters. For instance, a system
attempting to correct a misunderstanding of a malformed user statements is
quite different than the system failing to understand an unambiguous answer to
a direct question.

###  6.1 Observations from Annotating ACQIs

Our initial approach to building a model for finding ACQIs and making
associated recommendations to bot-builders was based on the assumption that
particular ACQIs are bad for the user experience and should be avoided. This
turned out not to be the case. As the conversations from the LivePerson
datasets were annotated, we observed that ACQIs alone may be bad or good or
neither. For example, ‘Ask for Confirmation’ is good when the user input is
genuinely vague, but is bad when the user input was clear and the system
should have understood. For example, asking the user to confirm that ‘next
Wednesday’ refers to a particular calendar date is sometimes helpful
(especially if today is Tuesday!). Asking if the user’s response was ‘3’ if
the user just selected ‘3’ can by contrast be obvious and irritating. If a
consumer uses ‘Restart’ one time, it can be seen as a good signal that the
consumer requested to go back and the system responded appropriately, but if
it is used more frequently it can be a signal that their request is not being
properly handled. Based on ACQI designations alone, bot-builders cannot always
be sure whether a corrective action is needed. Essentially, ACQIs are context
dependent and combine (with context and themselves) nonlinearly. In this work
we explore the context dependency via the relationship with IQ. We leave a
more structured statistical modeling framework for future work.

###  6.2 Annotating ACQIs and IQ Together

To mitigate the issue of ACQI instances not being universally good, bad, or
neutral, it was decided to combine ACQIs with an overall quality score such as
Response Quality (Bodigutla et al.,, 2020) or Interaction Quality (Schmitt and
Ultes,, 2015), discussed in Section 4. For these purposes, IQ was chosen,
partly because of the availability of the LEGOv2 dataset and comparable prior
work.

There are several differences between the guidelines given in Schmitt et al.,
(2011) and our own. Most importantly, the removal of all guidelines relating
to how much the score can be increased/decreased. Due to their restrictive
guidelines the change in IQ is almost entirely (99.7%) in increments of 0,1,
or -1. From observations in negativity bias (Rozin and Royzman,, 2001) we know
this does not reflect how consumers feel when encountering undesirable
behaviours. The impact of the removal of these guidelines can be seen in
figure 2. In spite of these changes our annotator agreement has slightly
increased with $\rho=.69,.72$ for them and $\rho=.76$ for us.

We also altered the ‘dissatisfaction scale’ used by Schmitt et al., (2011)
(5-satisfactory, 4-slightly unsatisfactory, 3-unsatisfactory, 2-very
unsatisfactory, 1-extremely unsatisfactory ) to 5-good, 4-satisfactory, 3-bad,
2-very bad, 1-terrible. We added the ‘good‘ category as we want our IQ score
to show that there is room for improvement even if the recommended improvement
is not currently technically feasible. The guideline dictating that each
conversation should start with a satisfactory rating was removed as our dialog
systems the consumer is often the party that initiates the dialog (in LEGOv2
the system always opens the dialog with a greeting), so the dialog system can
make a mistake with interpreting the very first message in the conversation.

Figure 2: Change in IQ score distribution between IQ annotations in Ultes et
al., (2015) and the work reported in this paper.

###  6.3 Annotation and Inter-Annotator Agreement

Annotation of the 531 LEGOv2 conversations and 520 LivePerson conversations
was carried out by three annotators employed by LivePerson who are experienced
live customer service agents. Following Schmitt et al., (2011) we took the
median score of our 3 annotators as ground truth. For 3 way ties for ACQI
(6.4% of turns) we chose to use the most common label for that particular
dialog system out of the labels the annotators have chosen. So as in Schmitt
et al., (2011) and Stoyanchev et al., (2019) we use third parties rather than
the users of the dialog system to judge the turn by turn quality, and like
Stoyanchev et al., (2019) our annotators are expert rather than crowdsourced.
The biggest differences between their annotation work and ours is that the
guidelines for the running IQ score are simplified, and we include an
additional annotation task for ACQI to recommend an appropriate fix. The
averages of the minimum and final IQ scores are shows in Table 3, which shows
that the CMU LEGOv2 has the lowest dips in IQ score during a conversation, but
by the end of the conversations, the IQ for the various bots is quite similar.

Table 3: Average Minimum and Final IQ Scores from Annotation Bot | Ave Final IQ Score | Ave Min IQ Score
---|---|---
LEGOv2 | $3.78$ | $2.44$
Junior Sales Assistant | 3.43 | 3.15
Help and Route | 3.79 | 3.44
Router | 3.74 | 3.61
Food Expert | 4.08 | 3.48

In spite of having similar overall IQ outcomes, the CMU LEGOv2 dialog system
has fewer neutral neutral steps, and more positive and negative turns, whereas
LivePerson dialog systems have more neutral turns.

Figure 3: Distribution of negative/neutral/positive score changes grouped by
CMU and LivePerson dialog systems.

We measured inter-annotator agreement to measure the clarity of our ACQI
taxonomy and IQ rules when applied to real dialog systems. The results have a
Cohen’s Kappa (CK) of .68, which is substantial agreement according to Landis
and Koch, (1977). See Table 4 for more details.

Table 4: Annotator Agreement for annotating IQ and ACQI, showing linear weighted cohen kappa (LWCK), unweighted average recall (UAR), spearman rank correlation ($\rho$) for IQ and cohen kappa (CK) for ACQI. Following Schmitt and Ultes, (2015) we take the average agreement across each pair of annotators. | IQ | ACQIs
---|---|---
Bot | UAR | LWCK | $\rho$ | CK
CMU | .61 | .65 | .76 | .71
Junior Sales Assistant | .46 | .34 | .21 | .64
Help and Route | .55 | .50 | .53 | .67
Router | .39 | .49 | .64 | .78
Food Expert | .60 | .63 | .71 | .60
All (macro-averaged) | .52 | .52 | .57 | .68

###  6.4 Analysis of Combined ACQI and IQ Annotations

We are now in a position to analyze the correspondence between individual
ACQIs and changes in the overall IQ score. By looking at the turn to turn
change in IQ given the presence of a particular ACQI we gain a more nuanced
understanding of how a dialog system is performing. In previous attempts to
measure dialog quality a lack of understanding and an inability to resolve a
consumers issue (task completion) are taken to be unequivocally bad. This is
not always the case.

From Figure 4 we see that with the exception of ‘Bad Transfer’ (which was not
annotated as negative) all of our ACQIs can be positive, negative or neutral.
‘Does Not Understand’, ‘Ignored Consumer Statement’ and ‘Input Rejected’ are
usually negative, but can be reasonable responses if the consumer is typing
things that are out of scope or incomprehensible. However, they are negative
when the consumers utterance should have been understood by the system.
‘Unable to Resolve’ is positive when the system correctly identifies that the
consumer’s request is out of scope. ‘Provides Assistance’ is positive if the
assistance was requested and appropriate, but negative if it was not, or if
the assistance has already been provided. Similarly, ‘Ask for Information’ can
be positive or negative, depending on the relevance of the requested
information.

Figure 4: ACQIs from Table 2 along with the proportions of each that were
aligned with positive, negative, and neutral changes in IQ. Note that for the
above graphic we excluded any turn whose preceding score IQ was a 1 or 5.

Analyzing the annotated datasets by ACQI and dialog system is also instructive
(Table 5). For most of the dialog systems, ‘Does Not Understand’ is the
largest category of ACQI associated to a decrease in score. The Junior Sales
Assistant is an exception, and for this, 54% of the ACQIs are marked as
‘Provides Assistance’, indicating that this bot is making too many improper
transfers or inappropriate suggestions to the customer. While having the score
alone would be somewhat valuable in locating this issue, the ACQI gives
additional guidance on what steps can be taken to mitigate the undesirable
behavior.

Table 5: Distribution of ACQIs given a decrease in IQ score. Dialog System |  CMU LEGOv2 |  Junior Sales Assistant |  Help and Route |  Router |  Food Expert
---|---|---|---|---|---
Does Not Understand |  0.655 |  0.196 |  0.631 |  0.667 |  0.557
Ignored Consumer Statement |  0.093 |  0.009 |  0.046 |  0.000 |  0.076
Ask for Information |  0.078 |  0.140 |  0.077 |  0.000 |  0.165
Input Rejected |  0.057 |  0.009 |  0.000 |  0.000 |  0.013
Unable to Resolve |  0.042 |  0.000 |  0.000 |  0.000 |  0.051
Ask for Confirmation |  0.034 |  0.000 |  0.000 |  0.000 |  0.089
Provides Assistance |  0.024 |  0.542 |  0.246 |  0.333 |  0.051
Restart |  0.017 |  0.000 |  0.000 |  0.000 |  0.000
Bad Transfer |  0.000 |  0.103 |  0.000 |  0.000 |  0.000

We can also analyze combinations of ACQIs and their affect on a conversation.
For example, Figure 5 shows that asking for confirmation once normally leads
to an increase in IQ score, but after this the effectiveness decreases and
asking for confirmation more that 6 times can be actively harmful. It is
important to realize that the ACQIs combine nonlinearly. That is, a single
instance of an ACQI in a conversation may be healthy, but multiple copies of
it may be a negative indicator. The extent to which the ACQIs have known
meaning is the extent to which structural statistical models can be built,
allowing bot creators the ability to test and refine explicit hypotheses about
how the conversational events actually aggregate to the dialog system user
experience. The careful definition and annotation work described so far
enables us to start quantifying such effects in ways that were not hitherto
available. Analyzing such combinations of causes and effects in dialog systems
will be extended in future work.

Figure 5: Dependence of score change on number of confirmations.

##  7 Experimental Validation of the Predictive Ability of the ACQI and IQ
Model

While the annotation and analysis so far was already able to provide some
useful insights, the larger goal of these efforts is to build systems that can
automatically highlight crucial ACQIs in a dialog system so that they can be
fixed.

There is prior work in predicting IQ from labelled conversations, using a
variety of methods. The original work of Schmitt et al., (2011) used Support
Vector Machines, and more recent work has used stateful neural network models
including LSTMs (Rach et al.,, 2017) and biLSTMs (Bodigutla et al.,, 2020).

###  7.1 Vectorized Features from Conversations

In common with these approaches, we adopted a vector representation for
features. However, we deliberately restricted our model to use only features
extracted directly from the conversation text and the annotation, so that the
method could be more universally applicable, and in particular, to be able to
use the same annotation setup and featurization processes for data from the
LEGOv2 and LivePerson dialog systems.

For our ‘text’ features (Table 6), we use the pretrained contextual sentence
embeddings of BERT (Reimers and Gurevych,, 2019). Our embedding dimension is
768 for each speaker’s response. When predicting ACQI and IQ we concatenate
the embeddings for both consumer and dialog system for this turn and the
previous. This results in a $4\times 768=3072$ dimensional vector. When the
previous utterance is unavailable, we use the zero vector of the appropriate
dimension. This feature vector was used to represent a system where the model
uses only surface textual features for the previous two turns per user.

To compare with a system that also uses its observations and predictions of
the rest of the conversation so far, we experimented with adding features
derived from the ACQI labels. For the features ‘only-gold-acqi’ and ‘only-
pred-acqi’ we use cumulative counts on the one-hot encoding for the presence
of ACQIs.

These feature sets were also combined as ‘gold-acqi+text’ and ‘pred-
acqi+text’, for which we concatenated the cumulative counts to the contextual
sentence embeddings with our cumulative ACQI counts.

###  7.2 Model Training and Prediction

To get a good representation of many ACQIs across the training and test sets,
we used the multi-label Scikit Learn python package and methodology of
Szymański and Kajdanowicz, (2017), which supports balanced multi-label
train/test splits which we adapted for nested cross-validation (Krstajic et
al.,, 2014). Utilizing this package we implemented nested cross-validation
using 5 cross-validated folds for the inner and outer loop. For predicting
ACQI we tested logistic regression, random forest, and xgboost. For IQ we
tested the above and a linear regressor. We found that the best performing
text-based model for predicting both ACQIs and IQ was logistic regression with
C (inverse regularization strength) set to 0.01 and with the ‘balanced’ class
weight setting in Scikit Learn.

###  7.3 Predicting IQ: Results and Discussion

For the IQ prediction experiment, results for each of the dialog systems using
the features above are presented in Table 6. The results found can also be
compared with the annotator agreement findings of Table 4.

Points to highlight include:

  * •

The text-only prediction, which is the easiest to implement, achieves linear
weighted Cohen Kappa performance slightly lower but comparable to annotator
agreement (average 0.49 vs 0.52).

  * •

Using only ACQI information (either gold or predicted) leads to a loss in
recall in all but one dialog system, and on average reduces recall by around
10% using only gold labels and 20% using only predicted labels.

  * •

Though the use of gold labels along with text features leads to improved
correlation with annotators (for example, an average 6% improvement in kappa
score), these expectations do not transfer reliably to the more realistic case
of using predicted ACQI labels as features.

  * •

The increase in average recall from using gold labels is small (only 2%), and
the use of predicted labels causes average recall to drop by 7%.

These experiments leave much room for optimization and improvement of various
kinds, including trying different text featurizers, and the number of turns
and relative weights of messages used in the vector encoding. The important
findings are that we can predict the exact IQ score approximately 60% of the
time, and that the use of vector embeddings derived directly from the message
texts is the most reliable practical method tested for building features.

Table 6: IQ Model Performance: linear weighted Cohen kappa (LWCK), unweighted average recall (UAR), Spearman rank correlation ($\rho$) for IQ. Model Selection and Hyper-parameter selection was accomplished by nested cross-validation (5 folds). Dialog System | Features Used | $\rho$ | UAR | LWCK
---|---|---|---|---
CMU | text | 0.572 | 0.585 | 0.47
only-gold-acqi | 0.469 | 0.499 | 0.404
only-pred-acqi | 0.296 | 0.396 | 0.242
gold-acqi+text | 0.622 | 0.597 | 0.529
pred-acqi+text | 0.593 | 0.516 | 0.475
Junior Sales | text | 0.721 | 0.567 | 0.678
only-gold-acqi | 0.147 | 0.429 | 0.349
Assistant | only-pred-acqi | 0.529 | 0.399 | 0.494
gold-acqi+text | 0.84 | 0.599 | 0.797
pred-acqi+text | 0.831 | 0.573 | 0.798
Help | text | 0.569 | 0.503 | 0.495
only-gold-acqi | 0.25 | 0.363 | 0.285
and Route | only-pred-acqi | 0.288 | 0.384 | 0.256
gold-acqi+text | 0.673 | 0.553 | 0.597
pred-acqi+text | 0.682 | 0.434 | 0.583
Router | text | 0.621 | 0.346 | 0.585
only-gold-acqi | 0.608 | 0.482 | 0.521
only-pred-acqi | 0.636 | 0.341 | 0.592
gold-acqi+text | 0.765 | 0.635 | 0.679
pred-acqi+text | 0.841 | 0.374 | 0.791
Food Expert | text | 0.698 | 0.603 | 0.609
only-gold-acqi | 0.574 | 0.469 | 0.493
only-pred-acqi | 0.514 | 0.435 | 0.448
gold-acqi+text | 0.765 | 0.635 | 0.679
pred-acqi+text | 0.787 | 0.55 | 0.691
All Bots | text | 0.595 | 0.599 | 0.499
only-gold-acqi | 0.462 | 0.511 | 0.417
only-pred-acqi | 0.326 | 0.408 | 0.274
gold-acqi+text | 0.651 | 0.612 | 0.561
pred-acqi+text | 0.624 | 0.531 | 0.514

###  7.4 Predicting ACQI: Results and Discussion

For the IQ prediction experiment, results for each of the dialog systems using
only text-derived features and logistic regression as a classifier (as
described above) are presented in Table 6.

Points to note include:

  * •

The overall weighted average f1-score is 0.790. We are predicting the correct
ACQI nearly 80% of the time overall.

  * •

The accuracy of the classification depends significantly on the support of the
class: common ACQIs are predicted much more accurately that rare ones.

  * •

Because of this, the macro average performance (not weighted by support) is
worse, with an f1-score of just 0.574.

  * •

The score can be cross-referenced against Figure 4 and Table 5 to guide
improvement efforts. For example, ‘Does Not Understand’ occurs relatively
frequently and with overwhelmingly negative impact on IQ score, but the
F1-score for predicting this class is only 0.509. Improving ACQI
classification performance for this class would therefore be especially
impactful.

Table 7: ACQI Model Performance ACQI | Precision | Recall | F1-score | Support
---|---|---|---|---
Ask for Information | 0.948 | 0.896 | 0.921 | 4144
Ask for Confirmation | 0.777 | 0.781 | 0.779 | 2069
Does Not Understand | 0.524 | 0.495 | 0.509 | 1417
Provides Assistance | 0.884 | 0.902 | 0.893 | 1010
Ignored Consumer Statement | 0.196 | 0.270 | 0.227 | 226
Unable to Resolve | 0.599 | 0.807 | 0.688 | 176
Input Rejected | 0.236 | 0.392 | 0.295 | 120
Restart | 0.484 | 0.762 | 0.592 | 80
Bad Transfer | 0.250 | 0.286 | 0.267 | 14
macro avg | 0.544 | 0.621 | 0.574 | 9256
weighted avg | 0.799 | 0.784 | 0.790 | 9256

###  7.5 Using ACQIs to Simplify Bot-Tuning

As a final result, we estimate the extent to which predicting the correct ACQI
could help bot-builders involved in bot-tuning. Referring back to the ACQI
taxonomy in Table 2, without any extra contextual guidance, bot-builders have
28 possible action strategies with LivePerson dialog systems and 31 with
LEGOv2 dialog systems.

This was compared with the number of options that would be available using the
predict ACQI labeling. For this simple simulation, we made the following
assumptions:

  1. 1.

Each appropriate action is equally likely in the absence of IQ/ACQI.

  2. 2.

If IQ is available, tuning is only required when the score decrements. If IQ
is unavailable then all actions related to the system are relevant.

  3. 3.

Given the presence of a decremented IQ score, each action is equally likely.

  4. 4.

If an ACQI is available, all actions that are not assigned to at least one
ACQI are included in our list of options that the bot-builder can make.

  5. 5.

There is always a special action, $\langle$No Action$\rangle$, that may
applicable for the bot-builder.

The results of this exercise are presented in Table 8. We found that the
largest single simplification comes from the use of IQ — if IQ can be modelled
accurately, then the average number of recommended options is reduced from
28.6 to 9.73 (about 34%). Adding ACQI classifications as well reduces the
average down to 5.4 (about 19% of the original number). This makes a
hypothetical but strong case: if IQ and ACQI can be accurately predicted on a
turn-by-turn level, the amount of effort it takes a bot-builder to diagnose
problems and suggest possible solutions could be reduced by an estimated 81%.
While this is an optimistic hypothesis, the potential reward is large enough
to encourage more development in this area.

Table 8: Average number of recommended actions per dialog system when there is no measurement strategy (None), IQ is available (assuming no actions required when score does not decrement), ACQI alone is available, and IQ+ACQI. 95% confidence intervals were calculated taking 1000 bootstrapped samples (at turn level) per dialog system. Dialog System | None | IQ | ACQI | IQ + ACQI
---|---|---|---|---
CMU | 31 |  7.61 $\pm$ .28 |  7.25 $\pm$ .04 |  4.2 $\pm$ .14
Junior Sales Assistant | 28 |  10.95 $\pm$ .99 |  14.29 $\pm$ .11 |  5.99 $\pm$ .50
Help and Route | 28 |  7.87 $\pm$ .83 |  14.49 $\pm$ .07 |  4.57 $\pm$ .40
Router | 28 |  13.26 $\pm$ 1.43 |  14.77 $\pm$ .16 |  7.33 $\pm$ .80
Food Expert | 28 |  8.98 $\pm$ .86 |  14.39 $\pm$ .09 |  4.94 $\pm$ .47
All DS | 28.6 |  9.73 $\pm$ .88 |  13.03 $\pm$ .09 |  5.40 $\pm$ .46

##  8 Conclusion

Actionable Conversation Quality Indicators (ACQIs) are designed to to provide
bot-builders with actionable explanations of why their dialog systems fail. We
have explored the key desirable properties for building an ACQI taxonomy,
based on recommendations from the literature, interviews and collaboration
with dialog system experts. Based on an annotated dataset of just over 1000
conversations, we have shown that ACQIs are particularly useful when combined
with Interaction Quality (IQ), in particular so that the decision of whether
to take a recommended action can be focused on places in the dialog where
quality decreases.

The annotated datasets were used to train predictive models, which achieved a
weighted average f1-score of 79% using features based just on vectorized
embeddings of recent messages in order and logistic regression for
classification. While these results are preliminary, such a classification
model could be used to reduce the number of options for a bot-builder to
consider by as much as 81%. Results like this should influence UI design for
bot-builders directly: if the ACQI-based suggestions show up as a tooltip
(similar to refactoring tips in software integrated development environments),
they may be useful in the majority of cases, while being easy to ignore in the
remainder.

The prioritization of the bot-builder as a key user persona is the driving
principle for much of this work. We hope that more research focused on making
bot-builders more effective is encouraged and highlighted in the dialog system
community, as a crucial route to optimizing the experience of dialog system
users overall.

## Competing Interests

All authors are employed at LivePerson, Inc. The authors declare no other
competing interest.

## References

  * Adiwardana et al., (2020) Adiwardana, D., Luong, M.-T., So, D. R., Hall, J., Fiedel, N., Thoppilan, R., Yang, Z., Kulshreshtha, A., Nemade, G., Lu, Y., and others 2020\.  Towards a human-like open-domain chatbot.  arXiv preprint arXiv:2001.09977.
  * Bodigutla et al., (2019) Bodigutla, P. K., Polymenakos, L., and Matsoukas, S. 2019\.  Multi-domain conversation quality evaluation via user satisfaction estimation.  arXiv preprint arXiv:1911.08567.
  * Bodigutla et al., (2020) Bodigutla, P. K., Tiwari, A., Vargas, J. V., Polymenakos, L., and Matsoukas, S. 2020\.  Joint turn and dialogue level user satisfaction estimation on multi-domain conversations.  arXiv preprint arXiv:2010.02495.
  * Danieli and Gerbino, (1995) Danieli, M. and Gerbino, E. 1995\.  Metrics for evaluating dialogue strategies in a spoken language system.  In Proceedings of the 1995 AAAI Spring Symposium on Empirical Methods in Discourse Interpretation and Generation, pp. 34–39.
  * Deriu et al., (2020) Deriu, J., Rodrigo, A., Otegi, A., Echegoyen, G., Rosset, S., Agirre, E., and Cieliebak, M. 2020\.  Survey on evaluation methods for dialogue systems.  Artificial Intelligence Review, pp. 1–56.
  * Finch and Choi, (2020) Finch, S. E. and Choi, J. D. 2020\.  Towards unified dialogue system evaluation: A comprehensive analysis of current evaluation protocols.  arXiv preprint arXiv:2006.06110.
  * Forrester Research, (2020) Forrester Research 2020\.  The total economic impact of IBM Watson assistant.  Technical report, Forrester, commissioned by IBM.
  * Han and Anderson, (2020) Han, S. and Anderson, C. K. 2020\.  Customer motivation and response bias in online reviews.  Cornell Hospitality Quarterly, 61(2):142–153.
  * Hirschman et al., (1990) Hirschman, L., Dahl, D. A., McKay, D. P., Norton, L. M., and Linebarger, M. C. 1990\.  Beyond class a: A proposal for automatic evaluation of discourse.  In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990.
  * Hirschman and Pao, (1993) Hirschman, L. and Pao, C. 1993\.  The cost of errors in a spoken language system.  In Proceedings of the Third European Conference on Speech Communication and Technology, pp. 1419–1422.
  * Hockey et al., (2003) Hockey, B. A., Lemon, O., Campana, E., Hiatt, L., Aist, G., Hieronymus, J., Gruenstein, A., and Dowding, J. 2003\.  Targeted help for spoken dialogue systems.  In 10th Conference of the European Chapter of the Association for Computational Linguistics, Budapest, Hungary. Association for Computational Linguistics.
  * Jain et al., (2018) Jain, M., Kumar, P., Kota, R., and Patel, S. N. 2018\.  Evaluating and informing the design of chatbots.  In Proceedings of the 2018 Designing Interactive Systems Conference, pp. 895–906.
  * Krstajic et al., (2014) Krstajic, D., Buturovic, L. J., Leahy, D. E., and Thomas, S. 2014\.  Cross-validation pitfalls when selecting and assessing regression and classification models.  Journal of cheminformatics, 6(1):1–15.
  * Landis and Koch, (1977) Landis, J. R. and Koch, G. G. 1977\.  The measurement of observer agreement for categorical data.  biometrics, pp. 159–174.
  * Li and Sun, (2018) Li, J. and Sun, X. 2018\.  A syntactically constrained bidirectional-asynchronous approach for emotional conversation generation.  arXiv preprint arXiv:1806.07000.
  * Lin et al., (2019) Lin, Z., Madotto, A., Shin, J., Xu, P., and Fung, P. 2019\.  Moel: Mixture of empathetic listeners.  arXiv preprint arXiv:1908.07687.
  * Liu et al., (2018) Liu, S., Chen, H., Ren, Z., Feng, Y., Liu, Q., and Yin, D. 2018\.  Knowledge diffusion for neural dialogue generation.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1489–1498.
  * Luo et al., (2018) Luo, L., Xu, J., Lin, J., Zeng, Q., and Sun, X. 2018\.  An auto-encoder matching model for learning utterance-level semantic dependency in dialogue generation.  arXiv preprint arXiv:1808.08795.
  * Moghe et al., (2018) Moghe, N., Arora, S., Banerjee, S., and Khapra, M. M. 2018\.  Towards exploiting background knowledge for building conversation systems.  arXiv preprint arXiv:1809.08205.
  * Pieraccini et al., (2009) Pieraccini, R., Suendermann, D., Dayanidhi, K., and Liscombe, J. 2009\.  Are we there yet? research in commercial spoken dialog systems.  In International Conference on Text, Speech and Dialogue, pp. 3–13. Springer.
  * Polifroni et al., (1992) Polifroni, J., Hirschman, L., Seneff, S., and Zue, V. 1992\.  Experiments in evaluating interactive spoken language systems.  In Proceedings of the DARPA Speech and NL Workshop, pp. 28–33.
  * Qiu et al., (2019) Qiu, L., Li, J., Bi, W., Zhao, D., and Yan, R. 2019\.  Are training samples correlated? learning to generate dialogue responses with multiple references.  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3826–3835.
  * Rach et al., (2017) Rach, N., Minker, W., and Ultes, S. 2017\.  Interaction quality estimation using long short-term memories.  In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pp. 164–169.
  * Raux et al., (2006) Raux, A., Bohus, D., Langner, B., Black, A., and Eskénazi, M. 2006\.  Doing research on a deployed spoken dialogue system: one year of let’s go! experience.  In INTERSPEECH.
  * Raux et al., (2005) Raux, A., Langner, B., Bohus, D., Black, A., and Eskénazi, M. 2005\.  Let’s go public! taking a spoken dialog system to the real world.  In INTERSPEECH.
  * Reimers and Gurevych, (2019) Reimers, N. and Gurevych, I. 2019\.  Sentence-bert: Sentence embeddings using siamese bert-networks.  arXiv preprint arXiv:1908.10084.
  * Rozin and Royzman, (2001) Rozin, P. and Royzman, E. B. 2001\.  Negativity bias, negativity dominance, and contagion.  Personality and social psychology review, 5(4):296–320.
  * Schmitt et al., (2011) Schmitt, A., Schatz, B., and Minker, W. 2011\.  Modeling and predicting quality in spoken human-computer interaction.  In Proceedings of the SIGDIAL 2011 Conference, pp. 173–184.
  * Schmitt and Ultes, (2015) Schmitt, A. and Ultes, S. 2015\.  Interaction quality: assessing the quality of ongoing spoken dialog interaction by experts—and how it relates to user satisfaction.  Speech Communication, 74:12–36.
  * Schmitt et al., (2012) Schmitt, A., Ultes, S., and Minker, W. 2012\.  A parameterized and annotated spoken dialog corpus of the CMU let’s go bus information system.  In LREC, pp. 3369–3373.
  * Shriberg et al., (1992) Shriberg, E., Wade, E., and Price, P. 1992\.  Human-machine problem solving using spoken language systems (sls): Factors affecting performance and user satisfaction.  In Proceedings of the DARPA Speech and NL Workshop, pp. 49–54.
  * Stoyanchev et al., (2017) Stoyanchev, S., Maiti, S., and Bangalore, S. 2017\.  Predicting interaction quality in customer service dialogs.  In IWSDS.
  * Stoyanchev et al., (2019) Stoyanchev, S., Maiti, S., and Bangalore, S. 2019\.  Predicting interaction quality in customer service dialogs.  In Advanced Social Interaction with Agents, pp. 149–159. Springer.
  * Szymański and Kajdanowicz, (2017) Szymański, P. and Kajdanowicz, T. 2017\.  A network perspective on stratification of multi-label data.  In First International Workshop on Learning with Imbalanced Domains: Theory and Applications, pp. 22–35. PMLR.
  * Ultes et al., (2015) Ultes, S., Sánchez, M. J. P., Schmitt, A., and Minker, W. 2015\.  Analysis of an extended interaction quality corpus.  In Natural Language Dialog Systems and Intelligent Assistants, pp. 41–52. Springer.
  * Walker et al., (1997) Walker, M. A., Litman, D. J., Kamm, C. A., and Abella, A. 1997\.  PARADISE: A framework for evaluating spoken dialogue agents.  In 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics, pp. 271–280, Madrid, Spain. Association for Computational Linguistics.
  * Wang et al., (2020) Wang, J., Liu, J., Bi, W., Liu, X., He, K., Xu, R., and Yang, M. 2020\.  Improving knowledge-aware dialogue generation via knowledge base question answering.  In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 9169–9176.
  * Witt, (2011) Witt, S. 2011\.  A global experience metric for dialog management in spoken dialog systems.  In Proceedings of SemDial, pp. 158–166.
  * Wu et al., (2019) Wu, W., Guo, Z., Zhou, X., Wu, H., Zhang, X., Lian, R., and Wang, H. 2019\.  Proactive human-machine conversation with explicit conversation goals.  arXiv preprint arXiv:1906.05572.


# S3PRL-VC: OPEN-SOURCE VOICE CONVERSION FRAMEWORK WITH SELF-SUPERVISED SPEECH
REPRESENTATIONS

###### Abstract

This paper introduces S3PRL-VC, an open-source voice conversion (VC) framework
based on the S3PRL toolkit. In the context of recognition-synthesis VC, self-
supervised speech representation (S3R) is valuable in its potential to replace
the expensive supervised representation adopted by state-of-the-art VC
systems. Moreover, we claim that VC is a good probing task for S3R analysis.
In this work, we provide a series of in-depth analyses by benchmarking on the
two tasks in VCC2020, namely intra-/cross-lingual any-to-one (A2O) VC, as well
as an any-to-any (A2A) setting. We also provide comparisons between not only
different S3Rs but also top systems in VCC2020 with supervised
representations. Systematic objective and subjective evaluation were
conducted, and we show that S3R is comparable with VCC2020 top systems in the
A2O setting in terms of similarity, and achieves state-of-the-art in S3R-based
A2A VC. We believe the extensive analysis, as well as the toolkit itself,
contribute to not only the S3R community but also the VC community. The
codebase is now open-sourced111
https://github.com/s3prl/s3prl/tree/master/s3prl/downstream/a2o-vc-vcc2020.

Index Terms—  voice conversion, open-source, self-supervised learning, self-
supervised speech representation

##  1 Introduction

Voice conversion (VC) refers to a technique that converts a certain aspect of
speech from a source to that of a target without changing the linguistic
content [1, 2]. In this work, we focus on speaker conversion, which is the
most widely investigated type of VC. From an information perspective, VC can
be performed by first extracting the spoken contents from the source speech,
and then synthesizing the converted speech from the extracted contents with
the identity of the target speaker. Such a paradigm is sometimes referred to
as recognition-synthesis (rec-syn) based VC, as depicted in Figure 1.
Formally, starting from the source speech $\mathbf{X}$, a recognizer first
extracts the spoken contents, $\mathbf{H}$, which is then consumed by the
synthesizer to generate the converted speech, $\mathbf{Y}$:

${\mathbf{Y}}={\text{Synth}}({\mathbf{H}}),{\mathbf{H}}={\text{Recog}}({\mathbf{X}}).$
(1)

In the latest voice conversion challenge 2020 (VCC2020) [3], one of the
baselines directly concatenated an automatic speech recognition (ASR) model
and a text-to-speech (TTS) model [4]. In addition, several top performing
systems also implemented such a framework [5], showing state-of-the-art
performance in terms of both naturalness and similarity.

In rec-syn based VC, an ASR model trained on a labeled dataset is often used
to extract the supervised spoken content representation, such as text [4] or
phonetic posteriorgram (PPG) [6]. The collection of labeled datasets is often
costly, especially in a low-resource setting, such as the cross-lingual VC
scenario [3]. Therefore, researchers have resorted to unsupervised or the so-
called self-supervised speech representation (S3R) learning paradigm, where a
large-scale unlabeled data is used to learn rich, compact speech
representations. S3Rs have been applied to any-to-one VC [7], many-to-many VC
[8], any-to-any VC [9, 10] and cross-lingual VC [11].

In addition to its label-free property, S3R based VC is also attractive in it
being a good probing task for S3R analysis. A recently published SUPERB
benchmark [12] dedicates to compare different S3Rs across a range of
discriminative speech processing tasks, while it remains unclear what
representations are optimal for generation tasks like VC. For instance,
wav2vec 2.0 [13] has been shown to be powerful in not only ASR but also
speaker and language recognition [14], implying that it encodes rich content,
speaker and language information. Based on the discussion on the information
perspective of VC, we may hypothesize that a good ${\mathbf{H}}$ in Eq. 1
should be compact in content but contains little to none speaker information.
Based on such an assumption, wav2vec 2.0 may not be an optimal representation
for VC.

Fig. 1: The training and conversion procedures in any-to-one recognition-
synthesis based VC.

In this paper, we describe S3PRL-VC, an extension of the S3PRL toolkit and
SUPERB. Our main focus was any-to-one (A2O) VC, where the synthesizer is
trained in a target-speaker-dependent fashion. We used the VCC2020 dataset,
which allows us to test intra-lingual and cross-lingual settings. We also
provide an any-to-any (A2A) extension by using an off-the-shelf d-vector [15]
model to encode the unseen speaker information. We implemented models
resembling the top systems in VCC2018 [16] and VCC2020 [17], which allows us
to focus on the comparison. We conducted a large-scale evaluation, both
objectively and subjectively, to compare the performance between not only
different S3Rs but also state-of-the-art systems. S3PRL-VC is a competitive
system by yielding (1) a comparable performance with VCC2020 top systems in
the A2O setting in terms of similarity, and (2) state-of-the-art performance
in S3R-based A2A VC. Our main contributions are:

  * •

Inheriting the property of SUPERB, our S3PRL-VC implementation ensures fast
benchmarking but also state-of-the-art performance. Such a fast, easy-to-use
property benefits not only S3R researchers but also the VC community.

  * •

We present a large-scale comparison of S3Rs from the VC point-of-view,
providing new insights and perspectives to analyze the representations. We
also compared with top systems in VCC2020 that used PPGs, showing the
limitation and competitiveness of S3Rs.

##  2 Tasks

###  2.1 General description of VCC2020

All experiments in this work are benchmarked on the VCC2020 dataset [3]. There
are two tasks in VCC2020, with intra-lingual VC being task 1 and cross-lingual
VC being task 2. The two tasks share the same two English male and female
source speakers. The target speakers include two male and two female English
speakers for task 1, and one male and one female speaker each of Finnish,
German, and Mandarin for task 2. For each speaker, 70 utterances (roughly five
minutes) in their respective languages and contents are provided, and there
are 25 test sentences for evaluation. During conversion, ${\mathbf{X}}$ (which
is in English) is converted as if it was uttered by the target speaker while
keeping the linguistic contents unchanged.

###  2.2 Intra-lingual and cross-lingual any-to-one VC

We first consider the two tasks in VCC2020 under the A2O setting. Any-to-one
VC aims to convert from any arbitrary speech into that of a predefined target
speaker. The training and conversion processes are depicted in Figure 1. The
ability to encode ${\mathbf{H}}$ from any unseen speaker is ensured by the
common practice of training S3Rs on a multi-speaker dataset. Using the target
speaker dataset, ${\mathbf{D}_{\text{trg}}}$, the synthesizer is trained to
reconstruct the acoustic feature from ${\mathbf{H}}$. In the conversion phase,
the converted features, ${\mathbf{Y}}$, are generated following Eq. 1.
Finally, a waveform synthesizer (ex. neural vocoder) generates the converted
waveform.

Any-to-one VC is a good probing task to investigate several characteristics of
an upstream S3R model. First, a fundamental requirement of VC is the
linguistic consistency, so there is a positive correlation between the VC
performance of an S3R model and its ability to faithfully encode
${\mathbf{H}}$. Second, if an S3R model encodes rich speaker information, then
the source speaker information in ${\mathbf{X}}$ will conflict with the target
speaker attributes injected by the synthesizer, which hurts the VC
performance. Finally, during the synthesizer training in cross-lingual VC, the
S3R model may fail to generalize to ${\mathbf{X}}$ from a non-English target
speaker since most existing S3R models are trained with English datasets only.
It is worthwhile to examine the ability of mono-lingual S3R models to transfer
to different languages.

###  2.3 Intra-lingual any-to-any VC

We then provide an extension for the A2A scenario, also known as zero-shot VC.
A2A VC attempts to convert to a target speaker where
${\mathbf{D}_{\text{trg}}}$ is so limited (less than one minute) such that
fine-tuning in infeasible. A2A VC models are usually trained on a multi-
speaker dataset. Instead of recovering the target speaker information by the
synthesizer as in A2O VC, we use speaker embeddings, $s$, extracted by an off-
the-shelf speaker encoder, which is pretrained on an automatic speaker
verification (ASV) dataset and objective. Such a paradigm is also used in
zero-shot TTS [18]. In training, the speaker embedding extracted from the
target waveform is used. During conversion, given ${\mathbf{D}_{\text{trg}}}$,
${\mathbf{s}}$ is formed as an average of each embedding from each utterance.
We may then rewrite Eq. 1 as:

$\vspace{-0.1cm}{\mathbf{Y}}={\text{Synth}}({\mathbf{H}},{\mathbf{s}}),{\mathbf{H}}={\text{Recog}}({\mathbf{X}}),{\mathbf{s}}={\text{SpkEnc}}({\mathbf{D}_{\text{trg}}}).$
(2)

Fig. 2: The models implemented in this work. Left: the simple model. Middle: the simple model with an AR loop. Right: the Tacotron2 model, with extension to an any-to-any model by accepting a d-vector as the speaker embedding.  Table 1: Objective evaluation results on different VC settings over various S3Rs. For MCD and WER, the smaller the better; for ASV, the higher the better. Upstream | Intra-lingual A2O | Cross-lingual A2O | Intra-lingual A2A
---|---|---|---
Simple | Simple-AR | Taco2-AR | Taco2-AR | Taco2-AR
MCD | WER | ASV | MCD | WER | ASV | MCD | WER | ASV | WER | ASV | MCD | WER | ASV
mel | 8.41 | 48.5 | 59.00 | 8.92 | 22.7 | 49.75 | 8.47 | 38.3 | 77.25 | 39.0 | 46.67 | 9.49 | 4.2 | 19.50
PPG (TIMIT) | 7.78 | 69.0 | 85.50 | 7.83 | 58.9 | 95.25 | 7.18 | 33.6 | 99.75 | 51.0 | 84.67 | 8.31 | 12.9 | 83.50
PASE+ | 9.29 | 5.0 | 26.75 | 9.52 | 5.7 | 26.00 | 8.66 | 30.6 | 63.20 | 36.3 | 34.67 | 9.85 | 4.2 | 8.00
APC | 8.67 | 8.6 | 48.00 | 8.73 | 7.1 | 41.75 | 8.05 | 27.2 | 87.25 | 33.9 | 52.33 | 9.57 | 3.5 | 23.25
VQ-APC | 8.12 | 10.8 | 81.25 | 8.37 | 7.4 | 60.50 | 7.84 | 22.4 | 94.25 | 28.4 | 68.00 | 9.43 | 4.0 | 22.00
NPC | 7.74 | 39.0 | 92.75 | 8.15 | 21.1 | 76.75 | 7.86 | 30.4 | 94.75 | 37.6 | 59.00 | 9.39 | 4.4 | 21.00
Mockingjay | 8.58 | 31.3 | 51.00 | 8.74 | 9.5 | 47.00 | 8.29 | 35.1 | 79.75 | 39.2 | 46.00 | 9.43 | 5.0 | 25.00
TERA | 8.60 | 11.4 | 46.50 | 8.67 | 6.0 | 42.50 | 8.21 | 25.1 | 83.75 | 29.2 | 49.33 | 9.31 | 5.2 | 18.75
Modified CPC | 8.71 | 9.4 | 40.00 | 8.87 | 7.0 | 30.00 | 8.41 | 26.2 | 71.00 | 35.3 | 32.83 | 9.61 | 4.1 | 10.75
DeCoAR 2.0 | 8.31 | 7.4 | 54.75 | 8.33 | 6.4 | 53.00 | 7.83 | 17.1 | 90.75 | 26.8 | 59.33 | 9.28 | 4.0 | 27.00
wav2vec | 7.45 | 14.0 | 95.50 | 7.64 | 4.9 | 90.50 | 7.45 | 10.1 | 98.25 | 13.9 | 75.83 | 8.77 | 3.5 | 40.00
vq-wav2vec | 7.41 | 13.4 | 91.00 | 7.24 | 11.6 | 98.75 | 7.08 | 13.4 | 100.00 | 21.0 | 88.83 | 8.47 | 4.2 | 73.25
wav2vec 2.0 Base | 7.80 | 24.7 | 92.75 | 7.77 | 5.0 | 86.50 | 7.50 | 10.5 | 98.00 | 14.9 | 82.17 | 9.03 | 3.2 | 27.00
wav2vec 2.0 Large | 7.64 | 12.5 | 81.75 | 7.67 | 9.0 | 82.75 | 7.63 | 15.8 | 97.25 | 22.7 | 78.00 | 8.99 | 4.1 | 22.25
HuBERT Base | 7.70 | 5.5 | 89.25 | 7.79 | 4.7 | 84.25 | 7.47 | 8.0 | 98.50 | 13.5 | 82.33 | 9.19 | 3.4 | 23.25
HuBERT Large | 7.54 | 5.6 | 95.00 | 7.54 | 5.6 | 93.00 | 7.22 | 9.0 | 99.25 | 15.9 | 86.50 | 9.13 | 3.0 | 27.75

##  3 Implementation

###  3.1 Recognizer (upstream models)

Table 1 depicts the list of S3Rs we compared in this work, which are the
upstream models supported in S3PRL at the date of publication. For a complete
list of information (architecture, objective, etc.), refer to [12]. All
upstreams are trained with English data (mostly Librispeech). In addition to
the S3Rs, two extra upstreams were included: (1) mel-spectrogram, “mel”, and
(2) “PPG (TIMIT)”, which is trained supervisedly on the TIMIT dataset.

###  3.2 Synthesizer model design

Mel-spectrogram was selected as the target acoustic feature. We implemented
several models to resemble top systems of past VCCs, as illustrated in Figure
2. We avoid expensive model components like attention [19] for fast
benchmarking.

Simple: We start from the model used by the top system in VCC2018 [16]. The
simple model consists of a single layer feed-forward network (FFN), two long
short-term memory layers with projection (LSTMP), and a linear projection
layer.

Simple-AR: As autoregressive (AR) modeling has been shown to be effective in
speech synthesis [20], we added an AR loop to the simple model. At each time
step, the previous output is consumed by the first LSTMP layer. Dropout is
essential in the AR loop to avoid exposure bias brought by teacher-forcing
[21, 22].

Taco2-AR: We increase the model complexity by using a model architecture
similar to that of Tacotron 2 [23], which resembles the model used by the top
system in VCC2020 [17]. Different from Tacotron 2, the attention module was
not used as it was reported to be useless in [17].

###  3.3 Other setups

Any-to-any settings. The dataset used to train the A2A VC model is the VCTK
dataset [24]. For the speaker encoder, we used the d-vector model [15] trained
on a mix of datasets, including LibriSpeech, VoxCeleb 1 and 2.

Waveform synthesizer. We used the HiFi-GAN [25], a state-of-the-art parallel
real-time neural vocoder. For the A2O setup, we mixed the data of all 14
speakers in VCC2020 with the VCTK dataset, while for the A2A setup we used
only the VCTK dataset.

##  4 Evaluation metrics and protocols

###  4.1 Objective evaluation

We chose three objective evaluation metrics, all of which measure different
aspects of a VC system. Mel cepstrum distortion (MCD) is an intrusive, L2-norm
based metric which measures the general performance. Word error rate (WER)
measures the intelligibility and the linguistic consistency, and in this work
we used a pretrained wav2vec 2.0 model. The accept rate from a pretrained ASV
model measures the speaker similarity by calculating the cosine similarity
using speaker embeddings. For scenarios like the cross-lingual A2O task where
the reference speech is not accessible, we report WER and ASV only since they
are non-intrusive.

###  4.2 Subjective evaluation

For the subjective test, we asked listening participants to evaluate two
common aspects in VC: naturalness and similarity. Listeners were asked to
evaluate the naturalness on a five-point scale. For conversion similarity, a
natural target speech and a converted speech were presented, and listeners
were asked to judge whether the two samples were produced by the same speaker
on a four-point scale.

For each system, a total of 80 utterances (5 random $\times$ 16 conversion
pairs) were evaluated. Recordings of the target speakers were also included in
the naturalness test and served as the upper bound. We used an open-source
toolkit [26] that implemented the ITU-T Recommendation P.808 [27] to screen
unreliable ratings obtained through the Amazon Mechanical Turk (Mturk). We
recruited more than 280 listeners from the United States and had each sample
rated by five different participants on average. Audio samples are available
online222https://bit.ly/3oydaY2.

##  5 Evaluation results and discussions

###  5.1 Comparison of different models

We first investigate the impact of using different synthesizer models
described in Section 3.2 in the intra-lingual A2O setting, as shown in Table
1. First, only by adding the AR loop to the Simple model, most S3Rs benefit
from large improvements in WER. With Taco-AR, all S3Rs except PASE+ and
modified CPC achieved an ASV accept rate higher 80%, while all S3Rs suffered
from a degradation in WER. This shows that increasing the model capacity can
significantly improve the speaker similarity, while sacrificing the
intelligibility. However, we would like to emphasize that WER is a strict
measurement of intelligibility, and human can actually recognize better than
machine. On the other hand, the Taco2-AR model yields the best MCD scores,
which, as we will show later, correlates better with subjective naturalness
and similarity. Also, we empirically found the training time of the three
models similar. Based on these reasons, we decided to use the taco2-AR model
for the succeeding tasks and comparisons.

###  5.2 Results on different tasks

Next, we compare the results of using S3Rs for different tasks. Looking again
at Table 1, we first find S3Rs trained on a mono-lingual corpus can still work
well in the cross-lingual setting, demonstrating the ability to transfer
across languages. However, compared with the intra-lingual A2O task, it could
be clearly observed that all S3Rs degraded in terms of both the WER and ASV
accept rate, which is similar to the findings in [28]. Finally, in the intra-
lingual A2A setting, all S3Rs yielded WERs much lower than those in the A2O
setting, while the MCD values and ASV accept rates were significantly worse.
Even the best upstream, vq-wav2vec, yielded only an accept rate of $73.25$.
One possible reason is that in the A2A VC setting, modern S3Rs still fail to
disentangle content, such that the synthesizer preserves too much speaker
information. Another reason may be that a jointly trained speaker encoder [10]
is essential for S3R-based VC.

Table 2: Comparison with state-of-the-art systems. All upstreams use the Taco2-AR model. System | MCD | WER | ASV | Naturalness | Similarity
---|---|---|---|---|---
Intra-lingual A2O
mel | 8.47 | 38.3 | 77.25 |  2.61 $\pm$ 0.11 |  35$\%$ $\pm$ 3$\%$
PPG (TIMIT) | 7.18 | 33.6 | 99.75 |  3.32 $\pm$ 0.10 |  58$\%$ $\pm$ 4$\%$
PASE+ | 8.66 | 30.6 | 63.20 |  2.58 $\pm$ 0.12 |  31$\%$ $\pm$ 3$\%$
APC | 8.05 | 27.2 | 87.25 |  2.92 $\pm$ 0.11 |  43$\%$ $\pm$ 4$\%$
VQ-APC | 7.84 | 22.4 | 94.25 |  3.08 $\pm$ 0.10 |  40$\%$ $\pm$ 4$\%$
NPC | 7.86 | 30.4 | 94.75 |  2.98 $\pm$ 0.11 |  46$\%$ $\pm$ 3$\%$
Mockingjay | 8.29 | 35.1 | 79.75 |  2.81 $\pm$ 0.12 |  42$\%$ $\pm$ 4$\%$
TERA | 8.21 | 25.1 | 83.75 |  2.91 $\pm$ 0.12 |  37$\%$ $\pm$ 4$\%$
Modified CPC | 8.41 | 26.2 | 71.00 |  2.74 $\pm$ 0.11 |  33$\%$ $\pm$ 3$\%$
DeCoAR 2.0 | 7.83 | 17.1 | 90.75 |  3.04 $\pm$ 0.11 |  43$\%$ $\pm$ 4$\%$
wav2vec | 7.45 | 10.1 | 98.25 |  3.40 $\pm$ 0.05 |  52$\%$ $\pm$ 2$\%$
vq-wav2vec | 7.08 | 13.4 | 100.00 |  3.59 $\pm$ 0.10 |  59$\%$ $\pm$ 4$\%$
wav2vec 2.0 B. | 7.50 | 10.5 | 98.00 |  3.36 $\pm$ 0.06 |  51$\%$ $\pm$ 2$\%$
wav2vec 2.0 L. | 7.63 | 15.8 | 97.25 |  3.26 $\pm$ 0.10 |  50$\%$ $\pm$ 4$\%$
HuBERT B. | 7.47 | 8.0 | 98.50 |  3.48 $\pm$ 0.10 |  55$\%$ $\pm$ 4$\%$
HuBERT L. | 7.22 | 9.0 | 99.25 |  3.47 $\pm$ 0.10 |  54$\%$ $\pm$ 4$\%$
USTC-2018$\dagger$ | – | 6.5 | 99.00 |  4.20 $\pm$ 0.08 |  55$\%$ $\pm$ 4$\%$
USTC-2020 | 6.98 | 5.4 | 100.00 |  4.41 $\pm$ 0.07 |  82$\%$ $\pm$ 3$\%$
SRCB | 8.90 | 11.5 | 92.00 |  4.16 $\pm$ 0.08 |  68$\%$ $\pm$ 3$\%$
CASIA | 7.13 | 11.0 | 98.25 |  4.25 $\pm$ 0.08 |  61$\%$ $\pm$ 4$\%$
ASR+TTS | 6.48 | 8.2 | 100.00 |  3.84 $\pm$ 0.09 |  75$\%$ $\pm$ 3$\%$
Target | – | 0.7 | – |  4.57 $\pm$ 0.14 | –
Cross-lingual A2O
PPG (TIMIT) | – | 51.0 | 84.67 |  2.79 $\pm$ 0.08 |  43$\%$ $\pm$ 3$\%$
vq-wav2vec | – | 21.0 | 88.83 |  3.28 $\pm$ 0.08 |  44$\%$ $\pm$ 3$\%$
HuBERT L. | – | 15.9 | 86.50 |  3.13 $\pm$ 0.08 |  41$\%$ $\pm$ 3$\%$
USTC-2018 | – | 5.6 | 97.67 |  4.17 $\pm$ 0.06 |  34$\%$ $\pm$ 3$\%$
USTC-2020 | – | 7.6 | 96.00 |  4.27 $\pm$ 0.07 |  43$\%$ $\pm$ 3$\%$
SRCB | – | 8.6 | 78.67 |  4.34 $\pm$ 0.07 |  34$\%$ $\pm$ 3$\%$
CASIA | – | 10.5 | 91.67 |  4.11 $\pm$ 0.07 |  45$\%$ $\pm$ 3$\%$
ASR+TTS | – | 34.5 | 67.83 |  2.51 $\pm$ 0.08 |  39$\%$ $\pm$ 3$\%$
Target | – | – | – |  4.48 $\pm$ 0.12 | –
Intra-lingual A2A
PPG (TIMIT) | 8.32 | 12.7 | 84.25 |  3.41 $\pm$ 0.08 |  34$\%$ $\pm$ 4$\%$
vq-wav2vec | 8.47 | 4.2 | 73.25 |  3.58 $\pm$ 0.09 |  28$\%$ $\pm$ 3$\%$
S2VC$\dagger$ | – | 12.4 | 71.50 |  2.90 $\pm$ 0.09 |  29$\%$ $\pm$ 3$\%$
$\dagger$: Systems generate 16kHz, so MCD is not calculable and direct score
comparison should be made with caution.
Table 3: Linear correlation coefficients between different metrics. Metric | MCD | WER | ASV | Nat. | Sim.
---|---|---|---|---|---
MCD | – | 0.678 | -0.934 | -0.968 | -0.961
WER | – | – | -0.640 | -0.808 | -0.587
ASV | – | – | – | 0.910 | 0.911
Nat. | – | – | – | – | 0.932
Sim. | – | – | – | – | –

###  5.3 Comparing with top systems using subjective evaluation

We then compared S3R-based VC models with state-of-the-art systems. USTC-2018
[16], USTC-2020 [5, 17]333USTC’s systems used text and PPG for the intra-
lingual and cross-lingual tasks, respectively., SRCB [29], CASIA [30] were top
systems in VCC2020, all of which adopted PPGs, synthesizer pretraining on a
multi-speaker dataset, and AR vocoders. Notably, they used thousands of hours
of internal data for training. ASR+TTS [4] was the seq2seq+non-AR vocoder
baseline in VCC2020. S2VC [10] is the STOA system for A2A VC. The results are
shown in Table 2. We summarize our observations as follows:

  * •

vq-wav2vec outperformed all other upstreams in the subjective test, with a
3.59 naturalness and 59$\%$ similarity in the intra-lingual A2O setting.

  * •

In the A2O settings, there was still a naturalness gap between vq-wav2vec and
other VCC2020 top systems (3.59 v.s. 4.16-4.25, 3.28 v.s. 4.11-4.34). As for
similarity, vq-wav2vec was on par with USTC-2018 and CASIA in the intra-
lingual A2O setting, and achieved top in the cross-lingual setting.

  * •

In the A2A setting, vq-wav2vec was on par with S2VC in similarity, while being
significantly better in naturalness. Our system is therefore the new state-of-
the-art in S3R-based A2A VC.

###  5.4 Impact of supervision

Although top systems using PPG greatly outperformed vq-wav2vec in naturalness,
they used AR vocoders and the system was trained on large internal datasets,
so the impact of supervision is not yet clear. To this end, we compared vq-
wav2vec result with “PPG (TIMIT)” and the same vocoder. The high WERs and low
naturalness scores showed that the PPG was indeed of low quality. Nonetheless,
in all three settings, “PPG (TIMIT)” can achieve similar or higher similarity
scores than vq-wav2vec. This shows that supervision greatly contributes to
similarity, especially in difficult settings like A2A VC. This also shows that
the ability of current S3Rs to disentangle speaker information is still
limited when compared to PPG, and can be further improved in the future.

###  5.5 Justify the objective metrics with correlation analysis

Conducting a subjective test whenever a new S3R is developed cannot meet the
fast benchmark requirement of SUPERB. Therefore, we examine if the objective
measures align well with human perception. Using the intra-lingual A2O results
over different upstreams, we calculated pairwise linear correlation
coefficients. Results in Table 3 suggested that MCD best aligned with both
naturalness and similarity. Note that in this correlation analysis, we
considered systems that used the same decoder and neural vocoder. Since the
correlation result is strongly affected by the pool of methods evaluated in a
listening test, this good correlation could be observed only in such a
homogeneous condition. Nonetheless, this result is still very useful for the
benchmarking requirement of SUPERB.

##  6 Conclusions and future work

We presented S3PRL-VC, an extension of the S3PRL toolkit that applied S3R to
VC. We described the model design choice, and covered a variety of tasks.
Extensive experiments, both objective and subjective, evaluated the capability
of various S3Rs when applied to different VC scenarios. By comparing S3Rs with
supervised presentations like PPG, we showed the competitiveness of S3Rs in
certain settings, meanwhile shedding light on improving directions.

We suggest different future directions for readers from different communities.
From the VC perspective, it is worthwhile to continue investigating better
downstream model design. For instance, in A2A VC, a proper speaker encoder
should be used instead of fixed d-vector. Meanwhile, we encourage to use VC as
a probing task when designing a new S3R model, considering the challenges to
overcome brought by all aspects required in VC.

Acknowledgements We would like to thank the S3PRL/SUPERB team for the fruitful
discussions. This work was partly supported by JSPS KAKENHI Grant Number
21J20920, JST CREST Grant Number JPMJCR19A3, and a project, JPNP20006,
commissioned by NEDO, Japan.

## References

  * [1] Y. Stylianou, O. Cappe, and E. Moulines,  “Continuous probabilistic transform for voice conversion,”  IEEE TSAP, vol. 6, no. 2, pp. 131–142, 1998.
  * [2] T. Toda, A. W. Black, and K. Tokuda,  “Voice Conversion Based on Maximum-Likelihood Estimation of Spectral Parameter Trajectory,”  IEEE TASLP, vol. 15, no. 8, pp. 2222–2235, 2007.
  * [3] Y. Zhao, W.-C. Huang, X. Tian, J. Yamagishi, R. K. Das, T. Kinnunen, Z. Ling, and T. Toda,  “Voice Conversion Challenge 2020 - Intra-lingual semi-parallel and cross-lingual voice conversion -,”  in Proc. Joint Workshop for the BC and VCC 2020, 2020, pp. 80–98.
  * [4] W.-C. Huang, T. Hayashi, S. Watanabe, and T. Toda,  “The Sequence-to-Sequence Baseline for the Voice Conversion Challenge 2020: Cascading ASR and TTS,”  in Proc. Joint Workshop for the BC and VCC 2020, 2020, pp. 160–164.
  * [5] J.-X. Zhang, L.-J. Liu, Y.-N. Chen, Y.-J. Hu, Y. J., Z.-H. Ling, and L.-R. Dai,  “Voice Conversion by Cascading Automatic Speech Recognition and Text-to-Speech Synthesis with Prosody Transfer,”  in Proc. Joint Workshop for the BC and VCC 2020, 2020, pp. 121–125.
  * [6] L. Sun, K. Li, H. Wang, S. Kang, and H. Meng,  “Phonetic posteriorgrams for many-to-one voice conversion without parallel data training,”  in Proc. ICME, 2016, pp. 1–6.
  * [7] W.-C. Huang, Y.-C. Wu, T. Hayashi, and T. Toda,  “Any-to-One Sequence-to-Sequence Voice Conversion using Self-Supervised Discrete Speech Representations,”  in Proc. ICASSP, 2021, pp. 5944–5948.
  * [8] A. Polyak, Y. Adi, J. Copet, E. Kharitonov, K. Lakhotia, W.-N. Hsu, A. Mohamed, and E. Dupoux,  “Speech Resynthesis from Discrete Disentangled Self-Supervised Representations,”  in Proc. Interspeech, 2021, pp. 3615–3619.
  * [9] Y. Y Lin, C.-M. Chien, J.-H. Lin, H.-Y. Lee, and L.-S. Lee,  “FragmentVC: Any-to-Any Voice Conversion by End-to-End Extracting and Fusing Fine-Grained Voice Fragments With Attention,”  in Proc. ICASSP, 2021, pp. 5939–5943.
  * [10] J.-H. Lin, Y. Y Lin, C.-M. Chien, and H.-Y. Lee,  “S2VC: A Framework for Any-to-Any Voice Conversion with Self-Supervised Pretrained Representations,”  in Proc. Interspeech, 2021, pp. 836–840.
  * [11] W.-C. Huang, T. Hayashi, X. Li, S. Watanabe, and T. Toda,  “On Prosody Modeling for ASR+ TTS based Voice Conversion,”  in Proc. ASRU, 2021.
  * [12] S.-W. Yang, P.-H. Chi, Y.-S. Chuang, C.-I J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin, T.-H. Huang, W.-C. Tseng, K. t. Lee, D.-R. Liu, Z. Huang, S. Dong, S.-W. Li, S. Watanabe, A. Mohamed, and H.-Yi Lee,  “SUPERB: Speech processing Universal PERformance Benchmark,”  in Proc. Interspeech, 2021, pp. 1194–1198.
  * [13] A. Baevski, H. Zhou, A. Mohamed, and M. Auli,  “wav2vec 2.0: A framework for self-supervised learning of speech representations,”  in Proc. NeruIPS, 2020.
  * [14] Z. Fan, M. Li, S. Zhou, and B. Xu,  “Exploring wav2vec 2.0 on speaker verification and language identification,”  in Proc. Interspeech, 2021, pp. 1509–1513.
  * [15] E. Variani, X. Lei, E. McDermott, I. L. Moreno, and J. Gonzalez-Dominguez,  “Deep neural networks for small footprint text-dependent speaker verification,”  in Proc. ICASSP, 2014, pp. 4052–4056.
  * [16] L.-J. Liu, Z.-H. Ling, Y. Jiang, M. Zhou, and L.-R. Dai,  “WaveNet Vocoder with Limited Training Data for Voice Conversion,”  in Proc. Interspeech, 2018, pp. 1983–1987.
  * [17] L.-J. Liu, Y.-N. Chen, J.-X. Zhang, Y. Jiang, Y.-J. Hu, Z.-H. Ling, and L.-R. Dai,  “Non-Parallel Voice Conversion with Autoregressive Conversion Model and Duration Adjustment,”  in Proc. Joint Workshop for the BC and VCC 2020, 2020, pp. 126–130.
  * [18] Y. Jia, Y. Zhang, R. Weiss, Q. Wang, J. Shen, F. Ren, Z. Chen, P. Nguyen, R. Pang, I. Lopez Moreno, and Y. Wu,  “Transfer Learning from Speaker Verification to Multispeaker Text-To-Speech Synthesis,”  in Proc. NIPS, 2018, pp. 4480–4490.
  * [19] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N Gomez, L. Kaiser, and I. Polosukhin,  “Attention is All you Need,”  in Proc. NIPS, 2017, pp. 5998–6008.
  * [20] X. Wang, S. Takaki, and J. Yamagishi,  “An autoregressive recurrent mixture density network for parametric speech synthesis,”  in Proc. ICASSP, 2017, pp. 4895–4899.
  * [21] X. Wang, S. Takaki, and J. Yamagishi,  “Autoregressive Neural F0 Model for Statistical Parametric Speech Synthesis,”  IEEE/ACM TASLP, vol. 26, no. 8, pp. 1406–1419, 2018.
  * [22] Y. Wang, R.J. Skerry-Ryan, D. Stanton, Y. Wu, R. J. Weiss, N. Jaitly, Z. Yang, Y. Xiao, Z. Chen, S. Bengio, Q. Le, Y. Agiomyrgiannakis, R. Clark, and R. A. Saurous,  “Tacotron: Towards end-to-end speech synthesis,”  in Proc. Interspeech, 2017, pp. 4006–4010.
  * [23] J. Shen, R. Pang, R. J. Weiss, M. Schuster, N. Jaitly, Z. Yang, Z. Chen, Y. Zhang, Y. Wang, R. Skerry-Ryan, R. A. Saurous, Y. Agiomyrgiannakis, and Y. Wu,  “Natural TTS Synthesis by Conditioning WaveNet on MEL Spectrogram Predictions,”  in Proc. ICASSP, 2018, pp. 4779–4783.
  * [24] C. Veaux, J. Yamagishi, and Kirsten MacDonald,  “CSTR VCTK Corpus: English Multi-speaker Corpus for CSTR Voice Cloning Toolkit,” 2017.
  * [25] J. Kong, J. Kim, and J. Bae,  “HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis,”  in Proc. NeurIPS, 2020, vol. 33, pp. 17022–17033.
  * [26] B. Naderi and R. Cutler,  “An Open Source Implementation of ITU-T Recommendation P.808 with Validation,”  in Proc. Interspeech, 2020, pp. 2862–2866.
  * [27] ITU-T Recommendation P.808,  “Subjective evaluation of speech quality with a crowdsourcing approach,” 2018.
  * [28] R. Kumar Das, T. Kinnunen, W.-C. Huang, Z.-H. Ling, J. Yamagishi, Z. Yi, X. Tian, and T. Toda,  “Predictions of Subjective Ratings and Spoofing Assessments of Voice Conversion Challenge 2020 Submissions,”  in Proc. Joint Workshop for the BC and VCC 2020, 2020, pp. 99–120.
  * [29] Q. Ma, R. Liu, X. Wen, C. Lu, and X. Chen,  “Submission from SRCB for Voice Conversion Challenge 2020,”  in Proc. Joint Workshop for the BC and VCC 2020, 2020, pp. 131–135.
  * [30] L. Zheng, J. Tao, Z. Wen, and R. Zhong,  “CASIA Voice Conversion System for the Voice Conversion Challenge 2020,”  in Proc. Joint Workshop for the BC and VCC 2020, 2020, pp. 136–139.