# Improved Goal Oriented Dialogue via Utterance Generation and Look Ahead

Eyal Ben-David 1 , Boaz Carmeli* 2, Ateret Anaby-Tavor2
1Faculty of Industrial Engineering and Management, Technion, IIT
2IBM Research
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Both authors equally contributed to this work. Work done during an internship
at IBM Research.

###### Abstract

Goal oriented dialogue systems have become a prominent customer-care
interaction channel for most businesses. However, not all interactions are
smooth, and customer intent misunderstanding is a major cause of dialogue
failure. We show that intent prediction can be improved by training a deep
text-to-text neural model to generate successive user utterances from
unlabeled dialogue data. For that, we define a multi-task training regime that
utilizes successive user-utterance generation to improve the intent
prediction. Our approach achieves the reported improvement due to two
complementary factors: First, it uses a large amount of unlabeled dialogue
data for an auxiliary generation task. Second, it uses the generated user
utterance as an additional signal for the intent prediction model. Lastly, we
present a novel look-ahead approach that uses user utterance generation to
improve intent prediction in inference time. Specifically, we generate
counterfactual successive user utterances for conversations with ambiguous
predicted intents, and disambiguate the prediction by reassessing the
concatenated sequence of available and generated utterances.

##  1 Introduction

Dialogue systems have emerged as a prevalent method for humans to interact
with their digital surroundings (Chen et al., 2017; Ralston et al., 2019;
Bocklisch et al., 2017). Consequently, goal-oriented dialogue systems have
become a prominent customer-service interaction-channel for most businesses
(Yan et al., 2017). However, there is still a long way to go until we reach a
pervasively-integrated, fully-automated, and reliable dialogue system to serve
as the preferred customer-care go-to channel (Gao et al., 2018).

Figure 1: Using dialogue context for improved intent prediction. Left box:
Successfully completed dialogue. Right box: Bot misunderstood user intent thus
provided an incorrect response. Error is attributed to the incorrect attention
that the dialogue system gives to the misleading adjective ’in the list’
instead of the indicative verb ’took’.

A key component of these systems are information-completion processes that are
mutually carried out by two actors: the user and the bot (Luo et al., 2014).
The user knows her goal but is usually unaware of the various options and the
optimal way to achieve it. The bot, on the other hand, has extensive knowledge
of the possible options and the various ways to achieve them, but is unaware
of the user’s exact goal. Via a dialogue, the two actors cooperate to achieve
the user’s goal in the most effective way.

Typically, users express their intent at the beginning of the conversation,
and the bot reacts with its most suitable response. Bot responses are
dependent on user intents. Ultimately, an accurate bot response depends on its
ability to correctly understand the intent. Bot misunderstanding of the user
intent will most probably lead to an unsatisfactory response, which in turn
will cause the user to react accordingly (see Figure 1). For example, the user
might ask for a human agent, express negative sentiment, or rephrase her
request. Therefore, the likelihood of a dialogue system to retrospectively
identifying errors in its initial predicted user intent increases as the
conversation progresses.

Given this observation, we hypothesise that successive user utterances provide
an invaluable, yet overlooked, indication for the language understanding level
expressed by the previous bot response. Thus, training a model to predict
successive user utterances can help predict the user’s intent (§ 2.2, 3.2).

To validate our hypothesis, we experiment with training a text-to-text
transformer model using multi-task regime that simultaneously i) generates
predicted user utterances from a vast amount of unsupervised dialogue data;
and ii) predicts the intent from dialogue context, enriched with successive,
generated, user utterances. While earlier deep neural network models require a
dedicated classification layer and parameters (Devlin et al., 2019), a text-
to-text model, such as T5 (Raffel et al., 2019), solves the classification
task in a generative manner. This approach allows the model to share all its
parameters across the prediction and generation tasks and thus, potentially
achieve better results (§ 2.3).

Having a model that is trained to generate user utterances suggests an
innovative way to further improve the intent prediction by generating
successive user utterances and using them as a look-ahead signal during
inference (§ 3.3). Furthermore, such a model can also assist in solving the
specific situation of conflicting intents. For that, one may generate
counterfactual successive user utterances and use them as a look-ahead signal
to disambiguate the conflicting intents (§ 6).

Our contribution is thus, two-fold. We show that i) training a text-to-text
model using a multi-task regime, in which the secondary task is to predict the
succeeding user utterance, further improves the main task of intent
prediction, and ii) generating a successive user utterance and using it as a
look-ahead signal improves prediction in inference time.

##  2 Related Work

###  2.1 Contextual Intent Prediction

Intent prediction is usually formulated as a sentence classification task.
Given an utterance (e.g., “what are the details of this flight?”), a system
needs to predict its intent (e.g., “flight details”). Modern approaches use
neural networks to model intent prediction, using many different
classification architectures such as RNNs (Ravuri and Stolcke, 2015), CNNs
(Zhang et al., 2015), attention-based CNN (Zhao and Wu, 2016), and
transformer-based architectures (Devlin et al., 2019). In many cases, intent
prediction is modeled jointly with slot filling (Xu and Sarikaya, 2013; Chen
et al., 2016; Liu and Lane, 2016; Goo et al., 2018; Chen et al., 2019).

Another way to approach this prediction task is to use more contextual
knowledge, such as more dialogue turns (Feng et al., 2020). This approach was
made possible thanks to the creation of public datasets for task-oriented
dialogue, which contain utterance level annotations for full conversations
(Budzianowski et al., 2018; Rastogi et al., 2020). While most studies in this
direction seek to build a dialogue state tracking model (Liu and Lane, 2017;
Nouri and Hosseini-Asl, 2018; Cheng et al., 2020), several works model it
together with intent prediction (Ma et al., 2019).

In this paper, we use the dialogue context for intent prediction. Unlike
previous work, we face a setup in which there are few conversations with
intent-annotation, as we consider this more realistic for a system’s
development phase. Furthermore, we use only dialogue utterances for
prediction, without any other information (e.g., internal dialogue state
tracking variables).

###  2.2 Modeling User-side Utterances

Modeling the user side in dialogue systems has been researched in several
tasks and applications. One approach is to develop a user simulator, which is
essential e.g., for training dialogue models based on reinforcement learning
(RL). Naturally, the performance of the simulator directly impacts the RL
policy. Within this line of work, two methods have proven particularly useful:
1) Agenda-based simulators, using rule-based software to generate a simulation
(Schatzmann et al., 2006, 2007; Li et al., 2016b), and 2) data-driven
simulators that learn user responses directly from a corpus (Asri et al.,
2016; He et al., 2018; Kreyssig et al., 2018).

The modeling of users’ written and spoken utterances is another direction that
has been extensively studied within the standard dialogue modeling framework
(Kobsa and Carberry, 1989; Kobsa, 1990; Lin and Walker, 2011; Li et al.,
2016a). Existing methods strive to improve users’ language consistency in
conversations and generate meaningful responses in an open-domain dialog.

In this work, we use user-side modeling to improve the performance of goal-
oriented dialogue systems. We rely on the hypothesis that the user conveys her
intent to the bot and then implicitly reflects, in successive utterances,
whether the bot correctly understood it. Our model, thus, incorporates the
generation of successive user-utterances, aiming to learn from this weak-
supervision signal.

###  2.3 Transformer-based Language Modeling

Previous research works consider two main approaches when training a language
model (LM) that is based on the Transformer architecture (Vaswani et al.,
2017). The first treats the model as a contextualized word embedding (CWE)
(Devlin et al., 2019; Yang et al., 2019; Liu et al., 2019; Sanh et al., 2019;
Martin et al., 2020). As such, it employs the model as an encoder. When fine-
tuning the model, additional architecture is placed on top of the model,
adjusting model’s CWE representation to a specific downstream discriminative
task.

The second approach handles the task in a sequence-to-sequence manner. As
such, it uses an encoder-decoder architecture (or just a decoder (Radford et
al., 2019)), such that when given a context, it generates the missing tokens.
Because of their generative nature, these models are commonly integrated into
text generation downstream tasks (Radford et al., 2019; Anaby-Tavor et al.,
2020; Rothe et al., 2020). Recently, Raffel et al. (2019) presented T5, a
text-to-text transfer transformer, and showed that encoder-decoder LMs are
efficient in both discriminative and generative tasks. T5 demonstrated its
superiority in many tasks, eliminating the need to add task-related
architecture.

In this work we follow the second approach and take advantage of the T5
unified framework, which converts all text-based language problems into a
text-in text-out format. We apply T5 to the task of intent prediction under
the low-supervision scenarios we encounter by enriching our supervised task
with semi-supervised signals from unlabeled data.

###  2.4 Utterance Generation for Look Ahead

Although there is line of work on learning from user feedback (Hancock et al.,
2019; Shin et al., 2019), and specifically from generating user-side
utterances (Weston, 2016), ours is the first work we know of that uses
generated user utterance as a look-ahead signal to improve dialogue
understanding during inference.

##  3 Model and Methods

Figure 2: T5 model trained on three tasks: 1) predict intent, 2) reorder
utterances and 3) generate third utterance (3UG).

Dialogue representation relies on a complex data structure, composed of
multiple utterances, generated alternately by two actors: user and bot. The
number of utterances in a dialogue varies dramatically and so does the length
of each utterance. To cope with this complexity we based our experiments on a
flexible text-to-text model (T5) that supports multi-task training by design
via encoder-decoder architecture and language-modeling training. In T5, all
tasks are mapped into text-in text-out format. Specifically, the model
resolves classification tasks by generating the predicted class’ label. The
model is thus able to use the same loss function and parameter weights across
different tasks, data sets, and training regimes.

Next, we explain our text-in text-out experimental method in more details.

###  3.1 Contextual Intent Prediction

We start by training two T5-based variants: The first, referred to as semantic
utterance classification (_SUC_) (Tur et al., 2012) is trained on just the
first user-utterance from each dialogue. This classifier serves as our
baseline. The second, referred to as semantic dialogue classification (_SDC_),
is trained on, at least, three successive dialogue utterances.

For low data regimes, we further enriched the supervised dialogue data with a
weak supervision signal. To that end, we trained RoBERTa (Liu et al., 2019)
and BERT (Devlin et al., 2019), two state-of-the-art classifiers, on the
available supervised dataset and used them to predict the intent for unlabeled
conversation. We added samples for which the two classifiers agreed on the
predicted intent to the supervised train set.

To measure the model’s ability to correctly predict the intent, we constructed
a full-dialogue test set. Each dialogue in this set consisted of at least
three successive utterances that pertain to the same user intent. We measured
the model’s accuracy by classifying one, two, and three initial utterances
from each dialogue in the dataset.

###  3.2 Utterance Generation

To evaluate the hypothesis that utterance generation improves intent
prediction, we defined a generative task to predict (i.e., by generating) the
third user utterance from a given first user utterance and successive bot
response. We refer to this task as _3UG_. To train the T5 model on this task,
we used a vast amount of unlabeled data that was continuously logged by the
dialogue system during interaction with users. Alternatively, dialogue data
from user to human-agent interactions can be used in situations where data
from dialogue systems does not exist.

We test results by comparing the _SDC_ performance after training on the _3UG_
task to the _SUC_ intent prediction performance. Importantly, one may notice
that the _SDC_ model takes advantage of the additional supervision signal
available within the successive user utterance during training, although this
utterance is not available during inference. Thus, for comparing classifier
performance on real-time conversations, we experiment with complementing the
first available utterance with two additional utterances as follows. First, we
used the dialogue system to get the bot response i.e., the second utterance.
Second, we used the T5 model already-trained on the _3UG_ task to generate a
successive user utterance, conditioned upon the first user utterance and the
bot response (see Figure 3).

We report results for these two complementary experiments in section 5.

###  3.3 Disambiguate Conflicting Intents

Conflicting-intents refers to situations where the classifier prediction
doesn’t clearly distinguish between small set of competing intents. To solve
this specific problem we experimented with a novel counterfactual approach
that uses the 3UG model to generate a counterfactual user utterance for each
of the conflicting intents as described in § 3.2. We then evaluated the
performance by predicting the intent based on the ensemble of predictions for
all look-ahead conversations, where each conversation now has three successive
utterances.

###  3.4 Multi-task Training Regimes

We wanted to evaluate the level of intrinsic synergy between intent
classification and utterance generation tasks. To accomplish this, we compared
the intent prediction accuracy achieved by T5 trained on the supervised full-
dialogue dataset alone, (thus performing only a classification task), to the
results when the model was asked to first perform the successive utterance
generation task (trained on the unsupervised dataset) and then predict the
intent (trained on the supervised dataset). We also extracted an utterance
reordering task out of unsupervised data, in which we shuffled dialogue
utterances and train the model to correctly reorder them. We elaborate more on
dialogue-related auxiliary tasks in the supplemental material.

Figure 3: Using third utterance generation to improve intent prediction via
look ahead. $a_{1}>a_{2}$ are _SUC_ confidence scores when using just the
first utterance. $b_{1}>b_{2}$ are _SDC_ confidence scores when using three
utterances and assuming that search for new activity is the intent.
$c_{1}>c_{2}$ are _SDC_ confidence scores when using three utterances and
assuming missing completion is the intent. In this example, the highest score
is $c_{1}$, thus the predicted intent is missing completion.

##  4 Experiments

We conducted most of the experiments by fine-tuning a pre-trained T5-base
model, which shares its parameters across tasks. Specifically, we fine-tuned
the T5 model repeatedly using unsupervised, semi-supervised, and supervised
datasets in various orders across three different datasets.

###  4.1 Datasets

We used the following datasets (see Table 1):
MultiWOZ \- a large-scale multi-domain Wizard-Of-Oz dataset for task-oriented
dialogue modelling, containing over 8400 dialogues, and spanning seven domains
(Budzianowski et al., 2018). This dataset contains human to human
conversations, and thus does not obey any well-defined bot-side dialogue rules
or learning algorithm.
SGD \- a schema-guided dialogue dataset, containing over 16K conversations,
spanning 86 intents, and covering 16 different domains (Rastogi et al., 2019).
This dataset was created by asking crowd-sourced workers to rephrase dialogue
utterances that were created by a rule-based simulator. Thus, this dataset
obeys predefined dialogue rules but still reflects the language diversity and
variations one expects to find in goal-oriented dialogues.
EDU \- an unpublished dataset extracted from an operational goal-oriented
dialogue system. It contains over 4K dialogues across 115 intents pertained to
the learning-and-education domain; out of these 4K dialogues, only 350 have
intent labels. This dataset was extracted from a real customer-care dialogue
system that uses an SVM-based intent classifier and a rule-based dialogue-flow
software. Intent identification in this dataset depends solely on prediction
from the SVM-based single-utterance classifier; the dialogue system uses
additional rule-based logic to return a concrete bot response.

From the original datasets, we extracted conversations with at least three
successive utterances pertaining to the same intent. We further split
dialogues that expressed multiple intents into different parts, keeping those
that have at least three successive utterances pertaining to a single intent
as separate dialogues. This left us with 3994, 34056, and 2063 conversations
pertaining to 11, 86, and 115 intents for MultiWOZ, SGD, and EDU,
respectively. From these we used 1012, 1000 and 100 conversations for
supervised training and 2,538, 23,128, and 1,200 conversations for
unsupervised training, for the three datasets, respectively. For MultiWOZ and
SGD, we mimicked the situation of unsupervised data by ignoring the available
label. For EDU, we had only 350 labeled conversations, which we split into
100, 50, and 200 for the train, validation and test sets, respectively. To
increase the very low amount of supervised EDU data, we further applied the
weak-labeling algorithm described in § 3.1, which added an additional 513
semi-supervised conversations.

| MultiWOZ | SGD | EDU
---|---|---|---
#Intents | $11$ | $86$ | $115$
Unsupervised | $2,538$ | $23,128$ | $1,200$
Supervised | $1,012$ | $1,000$ | $100$
Weakly labeled | $0$ | $0$ | $513$
Dev | $211$ | $4933$ | $50$
Test | $233$ | $4995$ | $200$
Table 1: Intents and data-splits for the three datasets

###  4.2 Models and Baselines

We used _SDC_ as our main model for researching the performance of contextual
intent prediction. This involved training _SDC_ by fine-tuning a pre-trained
T5 model on the supervised data.

We then used _SUC_ as a baseline model for the contextual intent prediction
experiment. To train _SUC_ , we fine-tuned a pre-trained T5 model with
supervised data containing just the first utterance from each dialogue and the
associated intent label.

To better understand the effect of the multi-task training, we further
examined three _SDC_ model variants that differ in their fine-tuning regimes:
_ALL_ , which we fine-tuned on all available datasets: the unsupervised,
supervised, and semi-supervised, as described in § 3.4.
_PART-SDC_ , which we fine-tuned in two steps: first on all except for the
supervised intent prediction dataset, and then on the supervised dataset.
_ALL-SDC_ , which we also fine-tuned in two steps: first on all available
datasets and then, again, on the supervised intent prediction dataset.

###  4.3 Experimental Setup

Our experiments assume at least three consecutive dialogue utterances of user-
bot-user relating to the same intent. We tested model’s performance in
predicting the user intent as the dialogue progressed by letting it classify
one, two, and three consecutive utterances from each dialogue available in the
test set. We refer to these as _1-u_ , _2-u_ and _3-u_ test scenarios,
respectively, as can be seen in Table 2. To evaluate performance, we measured
intent accuracy, namely $t/d$ where $t$ is the number of true predictions and
$d$ is the number of samples, i.e., predictions, in the test data. In the EDU
experiments, we report a model’s improvement in accuracy with respect to _SUC_
, to avoid revealing sensitive information regarding an inner-system’s
performance.

To evaluate the hypothesis that a third generated utterance serves as a good
look-ahead signal during run-time, we sampled such utterances from the best-
performing T5 model and used them instead of the actual utterance available in
the test set. This experiment is referred to as _3-gen_ in Table 4. To better
understand the power of the generative model to produce qualitative
utterances, we experimented with the ’five times third-user-utterance’ method,
referred as _3-5xg_. This method generates five third-user-utterance
alternatives for each two successive user-bot utterances. It then predicts the
intent when each one of them is used as a third utterance. Last, it uses
majority voting to select from the predicted intents. We complement these two
generation methods with a baseline method, referred as _3-rnd_ , which
generated a random utterance and used it to predict the intent.

##  5 Results

We report the key results of our experiments in Table 2. As shown, the
performance of intent prediction increases significantly with dialogue
progression along all datasets, even if the model was trained on just the
supervised data (_SDC_). Specifically, MultiWOZ: $93.1\%\rightarrow 96.6\%$;
SGD: $68.5\%\rightarrow 73.6\%$; EDU: $+1.8\%\rightarrow+6.3\%$. Moreover,
training the model on all available data and all tasks (_ALL-SDC_) improved
the results across all datasets and all test scenarios: _1-u_ , _2-u_ , and
_3-u_. These results clearly prove our hypothesis that generating a third
utterance as a secondary task in a multi-task training regime improves intent
prediction performance.

Interestingly, we observed that both _SDC_ and _ALL-SDC_ models outperform the
baseline _SUC_ model even when tested on the _1-u_ scenario, namely when
tested to predict the intent from only the first utterance. This phenomenon
occurs across all datasets (MultiWOZ: $90.5\%\rightarrow 93.1\%\rightarrow
94.4\%$; SGD: $68.2\%\rightarrow 68.5\%\rightarrow 71.1\%$; EDU:
$0.0\%\rightarrow+1.8\%\rightarrow+5.5\%$). This important finding clearly
indicates the benefit of training text-to-text models over full conversations,
even if the model is used to predict intents from just the first utterance
during inference.

MultiWOZ | _1-u_ | _2-u_ | _3-u_ | _3-5xg_
---|---|---|---|---
_SUC_ | 90.5 | - | - | -
_SDC_ | 93.1 | 95.5 | 96.6 | 96.6
_ALL-SDC_ | 94.4 | 97.2 | 98.7 | 98.3
SGD | _1-u_ | _2-u_ | _3-u_ | _3-5xg_
_SUC_ | 68.2 | - | - | -
_SDC_ | 68.5 | 73.2 | 73.6 | 74.0
_ALL-SDC_ | 71.1 | 74.2 | 74.6 | 75.8
EDU | _1-u_ | _2-u_ | _3-u_ | _3-5xg_
_SUC_ | $0.0$ | - | - | -
_SDC_ | $+1.8$ | $+1.8$ | $+6.3$ | $+5.5$
_ALL-SDC_ | $+5.5$ | $+8.2$ | +9.1 | $+8.2$
Table 2: Results for three different models (_SUC_ , _SDC_ , _ALL-SDC_) on
four different test scenarios (_1-u_ , _2-u_ , _3-u_ , _3-5xg_) across the
three datasets.

#### Improved prediction via look-ahead

The rightmost column (_3-5xg_) in Table 2 reports results of the look-ahead
experiment. Here, we generated five user utterances and used majority voting
to choose among them. As can be observed, the generation of a third utterance
improves intent prediction compared to using just the two utterances available
during inference. Notably, these results hold for all datasets and all models
except EDU:_ALL-SDC_. Interestingly, intent classification performance with
the generated utterance (_3-5xg_ column) is comparable to the one using real
utterances (_3-u_ column). More specifically, MultiWOZ: $98.7\%$ vs $98.3\%$;
SGD: $74.6\%$ vs $75.8\%$; EDU: $+9.1\%$ vs $+8.2\%$.

##  6 Ablation Analysis

We conducted several ablation experiments to shed more light on our key
results. First, we investigated the impact of various training regimes on
model performance. Next, we examined the effect of conversation reordering as
an auxiliary task in our multi-task training regime. Then, we assess the
quality of utterances sampled from our generative models. Finally, we
demonstrate the ability of our model to generate counterfactual utterances
which may resolve situations of conflicting intents. To ensure compatibility
with previous results, we kept all experiment details similar to what
described in § 4.3.

MultiWOZ | _1-u_ | _2-u_ | _3-u_
---|---|---|---
_SDC_ | 93.1 | 95.5 | 96.6
_PART-SDC_ | 94.0 | 95.7 | 97.0
_ALL_ | 94.8 | 97.0 | 98.2
_ALL-SDC_ | 94.4 | 97.2 | 98.7
SGD | _1-u_ | _2-u_ | _3-u_
_SDC_ | 68.5 | 73.2 | 73.6
_PART-SDC_ | 68.7 | 73.3 | 74.0
_ALL_ | 71.6 | 76.1 | 76.6
_ALL-SDC_ | 71.1 | 74.2 | 74.6
EDU | _1-u_ | _2-u_ | _3-u_
_SDC_ | $+1.8$ | $+1.8$ | $+6.3$
_PART-SDC_ | $+0.9$ | $+2.7$ | $+5.4$
_ALL_ | $-1.8$ | $+2.7$ | $+6.3$
_ALL-SDC_ | $+5.4$ | $+8.2$ | $+9.1$
Table 3: Results from four different training regimes, tested for accuracy on
one, two and three dialogue utterances across the three datasets.

#### Training regimes effect

Table 3 presents results from the three different training regimes _PART-SDC,
ALL, ALL-SDC_ , compared to the _SDC_ baseline-model trained solely on the
supervised data. Adding unsupervised and semi-supervised data was shown to
improve the _SDC_ model performance across all training regimes. Further
analysis shows that fine-tuning the model in two steps, first on all but the
supervised data and then on the supervised data (_PART-SDC_), is inferior to
fine-tuning the model on all available data in a single step (_ALL_). Lastly,
we observe that as in _ALL-SDC_ , the two-step training regime further
improves results over the _ALL_ model when tested on three utterances, in two
out of the three datasets (MultiWOZ: $98.2\%\rightarrow 98.7\%$; EDU:
$+6.3\%\rightarrow+9.1\%$)

Figure 4: The data proportion for reordering conversations according to intent
accuracy. Data is divided between utterance reordering and third utterance
generation.

#### Sentence reordering importance

We studied the impact of the additional semi-supervised task (see § 3.4) on
intent prediction. Notably, creating a semi-supervised dataset for the
utterance reordering task is straightforward. Thus, this task can be added to
any model, using any dialogue dataset.

Figure 4 shows intent-classification accuracy when training with five
utterance reordering ratio alternatives ($0.0,0.1,0.3,0.5,1.0$) on the
MulitWOZ and SGD datasets. Per the $0.0$ alternative, we used all data for the
utterance generation task, while per the $1.0$ alternative, we used all data
for the utterance reordering task. The results indicate similar trends in both
datasets, showing that using a small portion ($0.1$) of utterance reordering
data is most beneficial. The proportion parameter affects model performance,
with a gap higher than $1.1\%$ between best- and worst-performing model in
both datasets.

| _3-gen_ | _3-5xg_ | _3-rnd_
---|---|---|---
MultiWOZ | _PART_ | _ALL_ | _PART_ | _ALL_ | –
_SDC_ | 94.4 | 95.5 | 95.7 | 96.6 | 92.7
_PART-SDC_ | 95.3 | 95.3 | 96.6 | 97.0 | 89.7
_ALL_ | 96.6 | 97.9 | 97.4 | 97.9 | 92.3
_ALL-SDC_ | 96.6 | 96.6 | 98.3 | 98.3 | 94.8
SGD | _PART_ | _ALL_ | _PART_ | _ALL_ | –
_SDC_ | 72.3 | 72.6 | 73.4 | 74.0 | 62.3
_PART-SDC_ | 73.1 | 73.1 | 73.7 | 74.3 | 64.7
_ALL_ | 75.9 | 75.3 | 76.7 | 77.2 | 65.8
_ALL-SDC_ | 73.7 | 72.6 | 75.6 | 75.8 | 64.3
EDU | _PART_ | _ALL_ | _PART_ | _ALL_ | –
_SDC_ | $+2.7$ | $+4.6$ | $+3.7$ | +5.5 | $+2.7$
_PART-SDC_ | $+1.8$ | $+4.6$ | +5.5 | +5.5 | $-2.7$
_ALL_ | $+3.7$ | +5.5 | $+3.7$ | $+3.7$ | $+1.8$
_ALL-SDC_ | $+7.2$ | +9.0 | $+7.2$ | $+7.2$ | $+6.3$
Table 4: Comparison of quality of sentences generated by _PART_ and _ALL_
models, and sampled with three different approaches: once (_3-gen_); five-
times with majority voting (_3-5xg_); and without context (_3-rnd_)

#### Look-ahead for intent prediction

Even though our main table reports results from majority voting, this was not
our first approach to generation-for-look-ahead. Naively, we started our
experiments with generating just a single look-ahead utterance. We report the
results from this experiment in the _3-gen_ columns of Table 4. When this
approach failed to meet the expected performance, we turned to sample several
(e.g., five) times from the generative model and used majority voting to elect
the predicted intent. Results of this experiment appear in the _3-5xg_ columns
of Table 4. Notably, majority voting improves performance across most models
and datasets. As a last baseline-step of our analysis, we used random
generated utterances for look-ahead. The results appear in the _3-rnd_ column
of Table 4. Clearly, the results degrade significantly across all datasets and
all models, reconfirming the importance of qualitative third utterances for
accurate intent prediction.

#### Qualitative look-ahead utterances

To analyse the ability of the model to generate qualitative look-ahead
utterances, we considered two model alternatives: PART, which was trained on
all tasks but not on the supervised intent prediction, and _ALL_ , which was
trained on all tasks, intent-prediction included. We used all the above-
mentioned _SDC_ model variants to evaluate utterance quality, as indicated by
the rows of Table 4. Each of these models was tested with utterances generated
by both generation model alternatives (_PART_ and _ALL_ columns of Table 4)
and under both look-ahead techniques (_3-gen_ and _3-5xg_).

Table 4 shows that _ALL_ is a statistically stronger generative-model,
outperforing PART in 15 out of 24 look-ahead experiments. In comparison, only
in 2 out of the 24 experiments did the models under evaluation achieve higher
accuracy when using utterances generated by the PART model. For both
generative models, generating five user utterances and using majority voting
to choose among them outperforms the single-generation alternative. This
result is consistent with results presented in § 5, and confirms that the
quality of generated utterances does not depend on the quality of the
discriminative model used for intent prediction.

#### Solving intent conflicts with counterfactual user utterances

Most goal oriented dialogue systems use specific approaches for solving
ambiguity in intent prediction. For example, the bot can present the user with
the conflicting intents and let her choose the right one, or it can ask a
clarification question and wait for user response. Here, we suggest a novel
counterfactual approach for solving intent ambiguity. In contrast to other
approaches, ours doesn’t require any additional input from the user. To
achieve that we use the 3UG model to generate counterfactual user utterance
for each of the conflicting intents (see Figure 5). We test our approach with
the following setup: (1) Fit an ALL-SDC model with an intent classification
head to the SGD training set and use it to predict intent of test examples
with single user-utterance; (2) Identify conflicting intents by setting a
threshold bound on the last softmax layer of the ALL-SDC classifier; (3) For
each conflicting intent, generate the bot’s response (second utterance). Since
we do not able to generate bot responses in our experimental setup , we mimic
the generation process by choosing a response utterance that is associated
with the predicted intent and has the most overlapping slots with the gold
response (gold slots are provided in the SGD dataset); (4) Generate the third
utterance by applying the 3UG model to the first two utterances; (5) Predict
the intent based on the ensemble of predictions for all look-ahead
conversations, where each conversation now has three successive utterances.

While, ideally, one may detect conflicting intents by applying a threshold
bound (Threshold-based conflicts), we also examine our counterfactual
algorithm when given a hindsight regards conflicting intents. We examine two
types of hindsight: (1) Identifying all mistaken examples and count the two
top predicted intents as the conflict intents (Mistake Oracle); and (2)
Identifying mistaken examples in which the second top predicted intent is the
gold intent (Conflict-oracle).

In Table 5, we report experiment results. As seen, with the Conflict-oracle
hindsight the algorithm fixes more than $30\%$ of the mistakes examples. This
illustrates the potential of our algorithm. To fully realize this potential
one must have an intent classifier with high accuracy/confidence correlation.
However, when using our classifier’s confidence, algorithm achieves modest
improvement. This can be attributed to the un-calibrated nature of the used
classifier and its failure, in most cases, to predict the gold intent as the
second- or third–top prediction.

Figure 5: Using ALL-SDC for solving intent conflicts with counterfactual based algorithm. The model assigns a probability for each of the intents given an input, i.e. a 1-turn conversation (costumer request). Then, if more than one intent surpasses a pre-chosen threshold bound, we choose the top two intents (orange and blue arrows) and generate a counterfactual conversation for each. Finally, the model predicts the intent given each of the conversation and ensembles both predictions to choose a final intent. Conflicts Selection Method | Error reduction
---|---
Conflict-oracle | $31\%$
Mistake-oracle | $8\%$
Threshold-based conflicts | $2\%$
Table 5: Error reduction for our counterfactual-based conflict solving
algorithm.

##  7 Conclusions

In this work we show how recent text-to-text models can benefit from abundant
past conversation data to improve the intent prediction along the course of
the dialogue. Specifically, we show that training text-to-text models to
generate successive user utterances as a secondary task in a multi-task
training regime improves the performance of the main intent prediction task.
We further show that the real power of these generative text-to-text models,
trained on vast amounts of unlabeled data, lies in their ability to
synthetically sample possible user responses and use them as a look ahead
signal to improve intent prediction in inference time.

## References

  * Anaby-Tavor et al. (2020) Ateret Anaby-Tavor, Boaz Carmeli, Esther Goldbraich, Amir Kantor, George Kour, Segev Shlomov, Naama Tepper, and Naama Zwerdling. 2020.  Do not have enough data? deep learning to the rescue! In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 7383–7390. AAAI Press.
  * Asri et al. (2016) Layla El Asri, Jing He, and Kaheer Suleman. 2016.  A sequence-to-sequence model for user simulation in spoken dialogue systems.  In _Interspeech 2016, 17th Annual Conference of the International Speech Communication Association, San Francisco, CA, USA, September 8-12, 2016_ , pages 1151–1155. ISCA.
  * Bocklisch et al. (2017) Tom Bocklisch, Joey Faulkner, Nick Pawlowski, and Alan Nichol. 2017.  Rasa: Open source language understanding and dialogue management.  _arXiv preprint arXiv:1712.05181_.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  Multiwoz-a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  _arXiv preprint arXiv:1810.00278_.
  * Chen et al. (2017) Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017.  A survey on dialogue systems: Recent advances and new frontiers.  _Acm Sigkdd Explorations Newsletter_ , 19(2):25–35.
  * Chen et al. (2019) Qian Chen, Zhu Zhuo, and Wen Wang. 2019.  BERT for joint intent classification and slot filling.  _CoRR_ , abs/1902.10909.
  * Chen et al. (2016) Yun-Nung Chen, Dilek Hakkani-Tür, Gökhan Tür, Jianfeng Gao, and Li Deng. 2016.  End-to-end memory networks with knowledge carryover for multi-turn spoken language understanding.  In _Interspeech 2016, 17th Annual Conference of the International Speech Communication Association, San Francisco, CA, USA, September 8-12, 2016_ , pages 3245–3249. ISCA.
  * Cheng et al. (2020) Jianpeng Cheng, Devang Agrawal, Hector Martinez Alonso, Shruti Bhargava, Joris Driesen, Federico Flego, Dain Kaplan, Dimitri Kartsaklis, Lin Li, Dhivya Piraviperumal, et al. 2020.  Conversational semantic parsing for dialog state tracking.  _arXiv preprint arXiv:2010.12770_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Feng et al. (2020) Shaoxiong Feng, Xuancheng Ren, Hongshen Chen, Bin Sun, Kan Li, and Xu Sun. 2020\.  Regularizing dialogue generation by imitating implicit scenarios.  _arXiv preprint arXiv:2010.01893_.
  * Gao et al. (2018) Jianfeng Gao, Michel Galley, and Lihong Li. 2018.  Neural approaches to conversational ai.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pages 1371–1374.
  * Goo et al. (2018) Chih-Wen Goo, Guang Gao, Yun-Kai Hsu, Chih-Li Huo, Tsung-Chieh Chen, Keng-Wei Hsu, and Yun-Nung Chen. 2018.  Slot-gated modeling for joint slot filling and intent prediction.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers)_ , pages 753–757. Association for Computational Linguistics.
  * Hancock et al. (2019) Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. 2019.  Learning from dialogue after deployment: Feed yourself, chatbot!  _arXiv preprint arXiv:1901.05415_.
  * He et al. (2018) He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018.  Decoupling strategy and generation in negotiation dialogues.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , pages 2333–2343. Association for Computational Linguistics.
  * Kobsa (1990) Alfred Kobsa. 1990.  User modeling in dialog systems: Potentials and hazards.  _AI Soc._ , 4(3):214–231.
  * Kobsa and Carberry (1989) Alfred Kobsa and Sandra Carberry. 1989.  _User models in dialog systems_.  Symbolic computation : Artificial intelligence. Springer.
  * Kreyssig et al. (2018) Florian Kreyssig, Iñigo Casanueva, Pawel Budzianowski, and Milica Gasic. 2018\.  Neural user simulation for corpus-based policy optimisation of spoken dialogue systems.  In _Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue, Melbourne, Australia, July 12-14, 2018_ , pages 60–69. Association for Computational Linguistics.
  * Li et al. (2016a) Jiwei Li, Michel Galley, Chris Brockett, Georgios P. Spithourakis, Jianfeng Gao, and William B. Dolan. 2016a.  A persona-based neural conversation model.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers_. The Association for Computer Linguistics.
  * Li et al. (2016b) Xiujun Li, Zachary C. Lipton, Bhuwan Dhingra, Lihong Li, Jianfeng Gao, and Yun-Nung Chen. 2016b.  A user simulator for task-completion dialogues.  _CoRR_ , abs/1612.05688.
  * Lin and Walker (2011) Grace I. Lin and Marilyn A. Walker. 2011.  All the world’s a stage: Learning character models from film.  In _Proceedings of the Seventh AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, AIIDE 2011, October 10-14, 2011, Stanford, California, USA_. The AAAI Press.
  * Liu and Lane (2016) Bing Liu and Ian Lane. 2016.  Attention-based recurrent neural network models for joint intent detection and slot filling.  In _Interspeech 2016, 17th Annual Conference of the International Speech Communication Association, San Francisco, CA, USA, September 8-12, 2016_ , pages 685–689. ISCA.
  * Liu and Lane (2017) Bing Liu and Ian Lane. 2017.  An end-to-end trainable neural network model with belief tracking for task-oriented dialog.  In _Interspeech 2017, 18th Annual Conference of the International Speech Communication Association, Stockholm, Sweden, August 20-24, 2017_ , pages 2506–2510. ISCA.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Luo et al. (2014) Jiyun Luo, Sicong Zhang, and Hui Yang. 2014.  Win-win search: Dual-agent stochastic game in session search.  In _Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval_, pages 587–596.
  * Ma et al. (2019) Yue Ma, Zengfeng Zeng, Dawei Zhu, Xuan Li, Yiying Yang, Xiaoyuan Yao, Kaijie Zhou, and Jianping Shen. 2019.  An end-to-end dialogue state tracking system with machine reading comprehension and wide & deep classification.  _CoRR_ , abs/1912.09297.
  * Martin et al. (2020) Louis Martin, Benjamin Müller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  Camembert: a tasty french language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 7203–7219. Association for Computational Linguistics.
  * Nouri and Hosseini-Asl (2018) Elnaz Nouri and Ehsan Hosseini-Asl. 2018.  Toward scalable neural dialogue state tracking model.  _CoRR_ , abs/1812.00899.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1(8).
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _CoRR_ , abs/1910.10683.
  * Ralston et al. (2019) Kennedy Ralston, Yuhao Chen, Haruna Isah, and Farhana Zulkernine. 2019.  A voice interactive multilingual student support system using ibm watson.  In _2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)_ , pages 1924–1929. IEEE.
  * Rastogi et al. (2019) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2019.  Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.  _arXiv preprint arXiv:1909.05855_.
  * Rastogi et al. (2020) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020.  Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 8689–8696. AAAI Press.
  * Ravuri and Stolcke (2015) Suman V. Ravuri and Andreas Stolcke. 2015.  Recurrent neural network and LSTM models for lexical utterance classification.  In _INTERSPEECH 2015, 16th Annual Conference of the International Speech Communication Association, Dresden, Germany, September 6-10, 2015_ , pages 135–139. ISCA.
  * Rothe et al. (2020) Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020.  Leveraging pre-trained checkpoints for sequence generation tasks.  _Trans. Assoc. Comput. Linguistics_ , 8:264–280.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of BERT: smaller, faster, cheaper and lighter.  _CoRR_ , abs/1910.01108.
  * Schatzmann et al. (2007) Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, and Steve J. Young. 2007\.  Agenda-based user simulation for bootstrapping a POMDP dialogue system.  In _Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics, Proceedings, April 22-27, 2007, Rochester, New York, USA_ , pages 149–152. The Association for Computational Linguistics.
  * Schatzmann et al. (2006) Jost Schatzmann, Karl Weilhammer, Matthew N. Stuttle, and Steve J. Young. 2006.  A survey of statistical user simulation techniques for reinforcement-learning of dialogue management strategies.  _Knowl. Eng. Rev._ , 21(2):97–126.
  * Shin et al. (2019) Jamin Shin, Peng Xu, Andrea Madotto, and Pascale Fung. 2019.  Happybot: Generating empathetic dialogue responses by improving user experience look-ahead.  _arXiv preprint arXiv:1906.08487_.
  * Tur et al. (2012) Gokhan Tur, Li Deng, Dilek Hakkani-Tür, and Xiaodong He. 2012.  Towards deeper understanding: Deep convex networks for semantic utterance classification.  In _2012 IEEE international conference on acoustics, speech and signal processing (ICASSP)_ , pages 5045–5048. IEEE.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA_ , pages 5998–6008.
  * Weston (2016) Jason E Weston. 2016.  Dialog-based language learning.  In _Advances in Neural Information Processing Systems_ , pages 829–837.
  * Xu and Sarikaya (2013) Puyang Xu and Ruhi Sarikaya. 2013.  Convolutional neural network based triangular CRF for joint intent detection and slot filling.  In _2013 IEEE Workshop on Automatic Speech Recognition and Understanding, Olomouc, Czech Republic, December 8-12, 2013_ , pages 78–83. IEEE.
  * Yan et al. (2017) Zhao Yan, Nan Duan, Peng Chen, Ming Zhou, Jianshe Zhou, and Zhoujun Li. 2017.  Building task-oriented dialogue systems for online shopping.  In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ , pages 4618–4625.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada_ , pages 5754–5764.
  * Zhang et al. (2015) Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 649–657.
  * Zhao and Wu (2016) Zhiwei Zhao and Youzheng Wu. 2016.  Attention-based convolutional neural networks for sentence classification.  In _Interspeech 2016, 17th Annual Conference of the International Speech Communication Association, San Francisco, CA, USA, September 8-12, 2016_ , pages 705–709. ISCA.

##  Appendix A Auxiliary Tasks

We took further advantage of the T5 model’s ability to share parameters across
tasks and defined several auxiliary dialogue-related tasks:
Utterance reordering: shuffle dialogue utterances and train the model to
correctly reorder them.
Conversation escalation: predict false if the bot contains the entire
conversation and true if it was eventually handled by a human agent.
Utterance repetition: predict true for repetitive bot utterances or rephrased
user utterances, and false otherwise.
We used various weak labeling techniques to add labels to the raw dialogue
data for each of these tasks. We then used these semi-supervised datasets
together with the supervised full-dialogue dataset and the unsupervised
dialogue data to train a T5 model to predict the user intent. We evaluated the
performance gained from each of these datasets by introducing them to the T5
model in different orders and amount ratios, and measured the intent
prediction accuracy.

We performed an analysis by examining the performance of additional auxiliary
tasks. Table 6 presents results for the EDU and SGD datasets. For the EDU
dataset, we considered all auxiliary tasks presented in § A. For SGD, we did
not include utterance-repetition and conversation-escalation as this dataset
does not contain these phenomena. We report intent prediction results achieved
with a one-step training regime, similar to the training regime that we use to
train the _ALL_ model.

Clearly, combining all tasks together performs better than using each task
separately. The gap between the _ALL_ model and the best-performing single-
task model is $1.7\%$ for EDU and $0.4\%$ for SGD. Furthermore, when used
separately, the escalation and repetitious tasks perform sub-optimally, with
measured accuracy of $+1.8\%$ and $0.0\%$ respectively, compared to $+4.6\%$
and $+3.6\%$ achieved by the utterance generation and reordering tasks,
respectively. This did not come as surprise, since these signals correlate
with dialogue outcome more than with a specific user intent.

| EDU | SGD
---|---|---
| _3-u_ | _3-u_
Utterance Generation | $+4.6$ | 76.2
Reordering | $+3.6$ | 75.8
Escalation | $+1.8$ | -
Repetitious | $+0.0$ | -
_ALL_ | $+6.3$ | 76.6
Table 6: Model performance using various auxiliary tasks.

##  Appendix B Alternative Classification Models

In this analysis, we focus on a potential alternative modeling solution. As
described in the main paper, we direct our modeling efforts at T5 variants
(Raffel et al., 2019): through baselines and our own contributions. However,
another possible modeling direction for the task could use a state-of-the-art
discriminative classifier (Devlin et al., 2019; Yang et al., 2019; Liu et al.,
2019).

Thus, as another baseline, we used a BERT classifier (Devlin et al., 2019),
which we trained to predict the intent from three successive utterances,
denoted as _SDC-BERT_. In Table 7, we report models’ accuracy across the three
different datasets: Multiwoz, SGD, and EDU. We further tested this model under
our look-ahead scenario, _3-gen_ , while using our trained T5 _ALL_ model to
generate the look-ahead utterances. For a complete picture, we present the
results of our three T5 variants, _SDC_ , _ALL_ , and _ALL-SDC_ , even though
these results have already been presented in the main paper.

Our unsupervised and semi-supervised data-integration methods improved SDC
performance across most training regimes; _ALL_ and _ALL-SDC_ outperform _SDC-
BERT_ in five out of six different setups. The performance of the T5 _SDC_ and
_SDC-BERT_ models that use only supervised data, are on a par. In three out of
six setups, the T5-based classifier outperforms BERT, and vice versa. However,
looking deeper, _SDC-BERT_ has the largest degradation when evaluated on the
look-ahead scenario, with an average of $2.4\%$ accuracy drop across all
setups, compared to $1.3\%$, $0.8\%$, and $1.0\%$ achieved by _SDC_ , _ALL_ ,
and _ALL-SDC_ respectively.

Look-ahead is an important setup for an intent classifier because it supports
intent prediction improvement while the dialogue is on-going. More research is
needed to determine the best way to integrate the unsupervised and semi-
supervised tasks described here with the supervised data, to improve a
discriminative model’s performance. We consider _SDC-BERT_ ’s large
degradation in this setup to be another incentive towards applying a unified
test-to-test model, such as _T5_ , to the task.

| MultiWOZ | SGD | EDU
---|---|---|---
| _3-u_ | _3-gen_ | _3-u_ | _3-gen_ | _3-u_ | _3-gen_
_SDC-BERT_ | 98.2 | 97.0 | 74.2 | 71.0 | +5.4 | +2.7
_SDC_ | 96.6 | 95.5 | 73.6 | 72.6 | +6.3 | +4.6
_ALL_ | 98.2 | 97.9 | 76.6 | 75.3 | +6.3 | +5.5
_ALL-SDC_ | 98.7 | 96.6 | 74.6 | 72.6 | +9.1 | +10.0
Table 7: Results on four different intent classification data-sets.

##  Appendix C Hyper-Parameters, Configurations, and Experimental Details

We use a single V100 GPU with 64 GB RAM in all experiments. All T5-based
models, namely _SUC_ , _SDC_ , _ALL_ , and _ALL-SDC_ share the same hyper-
parameters with respect to their shared architecture and to the optimization
process. They all use an initialized T5-Base (Raffel et al., 2019). _SDC-BERT_
use an initialized BERT-Base encoder (Devlin et al., 2019) and share the same
optimization process as the T5-based models. Configuration details and the
hyper-parameters of the training process are provided in Table 8.

All hyper-parameters were tuned on the development set. We tuned the maximal
sequence lengths {128, 256, 512} and the batch size {32, 64}. We then chose
the best performing set of hyperparameter according to the higher accuracy
score on the appropriate development set. For all models we set the amount of
training epochs to be 50, with an early stopping criteria of 7.

T5
---
hidden size | 512
number of attention heads | 8
number of hidden layers | 6
vocab size | 32128
max sequence length | 256
number of parameters | 220 M
Encoder - BERT
hidden size | 768
number of attention heads | 12
number of hidden layers | 12
vocab size | 30522
hidden activation | _’gelu’_
number of parameters | 108 M
Classifiers - Pooler
input dim | 768
first hidden dim | 768
second hidden dim | 256
output dim | #Intents
Optimization
optimizer | _AdamOptimizer_
beta1 | $0.9$
beta2 | $0.997$
epsilon | $1e-9$
batch size | 100
Table 8: The hyper-parameters of the _SUC_ , _SDC_ , _ALL_ , _ALL-SDC_ , and
_SDC-BERT_ models.

##  Appendix D URLs of Code and Data

We provide here the URLs for the datasets and code we have used:

  * •

We use code and pre-trained weights of the pre-trained _BERT-Base, Uncased_
(Devlin et al., 2019) and the pre-trained _T5_ (Raffel et al., 2019) from
huggingface: https://github.com/huggingface/transformers

  * •

_MultiWOZ_ (Budzianowski et al., 2018) conversations with intent labels are
extracted from the official paper’s repository:
https://github.com/budzianowski/multiwoz

  * •

_SGD_ (Rastogi et al., 2019) conversations with intent labels are extracted
from the official paper’s repository: https://github.com/google-research-
datasets/dstc8-schema-guided-dialogue