# Taming Continuous Posteriors for Latent Variational Dialogue Policies

Marin Vlastelica1, 2, Patrick Ernst2 György Szarvas2
1Autonomous Learning Group, Max Planck Institute for Intelligent Systems,
Tübingen, Germany
2Amazon Development Center Germany GmbH, Berlin, Germany
<EMAIL_ADDRESS>2{peernst<EMAIL_ADDRESS>
###### Abstract

Utilizing amortized variational inference for latent-action reinforcement
learning (RL) has been shown to be an effective approach in Task-oriented
Dialogue (ToD) systems for optimizing dialogue success. Until now, categorical
posteriors have been argued to be one of the main drivers of performance.

In this work we revisit Gaussian variational posteriors for latent-action RL
and show that they can yield even better performance than categoricals. We
achieve this by simplifying the training procedure and propose ways to
regularize the latent dialogue policy to retain good response coherence. Using
continuous latent representations our model achieves state of the art dialogue
success rate on the MultiWOZ benchmark, and also compares well to categorical
latent methods in response coherence.

Taming Continuous Posteriors for Latent Variational Dialogue Policies

Marin Vlastelica1, 2, Patrick Ernst2 and György Szarvas2 1Autonomous Learning
Group, Max Planck Institute for Intelligent Systems, Tübingen, Germany 2Amazon
Development Center Germany GmbH, Berlin, Germany
<EMAIL_ADDRESS>2{peernst<EMAIL_ADDRESS>

##  1 Introduction

Task-oriented Dialogue (ToD) systems have reached a degree of maturity, which
enables them to engage with human users and assist them in various tasks. They
are able to steer natural-language conversations in order to complete users’
goals, such as booking restaurants, querying weather forecasts and resolving
customer service issues. At their core, the behavior of these systems is
controlled by a dialogue policy, which receives user inputs in the form of
utterances and additional features or states.

Template-based methods Walker et al. (2007); Inaba and Takahashi (2016)
leverage ranking or classification approaches to select the most fitting
response from a pre-defined set of responses, i.e. templates. While template-
based methods offer better control over the dialogue policy behavior, they are
less versatile due to their dependency on template sets. Moreover,
constructing comprehensive template sets is a challenge in itself Gao et al.
(2019). In retrieval-based approaches Yan et al. (2016); Henderson et al.
(2019); Tao et al. (2019) candidate responses are not predefined, but are
retrieved from massive dialogue corpora, e.g. by executing ad-hoc search
queries a priori.

Generative models do not require such additional inputs as prior knowledge.
They enable end-to-end (E2E) learning of dialogue policies with the potential
of extrapolation to diverse responses Serban et al. (2017); Zhao et al.
(2017); Gu et al. (2018). Methods based on amortized variational inference
achieve this by inferring low-dimensional variational posteriors mapped from
dialogue contexts, similar to variational autoencoders (VAEs) (Kingma and
Welling, 2013). Even though such fully data-driven approaches offer great
versatility and a faster adoption, they may exhibit degenerate behavior by
generating incomprehensible utterances. This is apparent in multi-turn
dialogues, which span hundreds of words, while the success signal is only
observed at the end of dialogues. Reinforcement learning (RL) based approaches
are able to optimize for such long-term, sparse rewards and have been applied
in this setup. Previous approaches prominently applied word-level RL Lewis et
al. (2017); Kottur et al. (2017), where the action space is defined over the
entire vocabulary. The response utterances are then generated auto-
regressively by consecutive next-word predictions. Unfortunately, the use of
large action spaces often impedes the convergence of policy learning
algorithms, which makes it hard to ensure coherent responses. Prior work makes
use of _latent-action RL_ to address the dimensionality problem by utilizing
variational inference approaches (Zhao et al., 2019; Lubis et al., 2020).
These methods rely on a supervised learning stage followed by fine-tuning via
reinforcement learning in the latent space.

In this paper we follow this paradigm, but extend prior work substantially by
introducing the TCUP approach, which aims to _T_ ame _C_ ontin _U_ ous _P_
osteriors for latent variational dialogue policies. TCUP makes the following
contributions: (i) A new formulation of the variational inference objective
for learning continuous latent response representations without auxiliary
learning objectives. (ii) A more robust approach for learning from offline ToD
data in a RL setup, which utilizes the fact that we are dealing with expert
dialogue trajectories.

We compare the performance of our proposed method to competing approaches on
the MultiWOZ benchmark (Budzianowski et al., 2018). Our experimental results
show that we are able to improve the state-of-the-art performance across
different benchmark metrics. Apart from MultiWOZ’s context-to-text metrics, we
demonstrate the benefits of TCUP’s learned latent representations
quantitatively using a clustering analysis following Lubis et al. (2020).

##  2 Preliminaries

Usually, latent-action reinforcement learning methods are trained in two
stages: (i) In the first stage an encoder-decoder architecture is applied to
learn latent representations from a supervised signal, i.e. latent actions,
over dialogue responses. (ii) In the consecutive stage a RL-based policy
predicts the best latent action given a particular context in order to
optimize for long-term dialogue success. The trained decoder stays fixed and
receives the output of the RL policy to generate the final reponse utterance.

##### Learning Latent Response Representations

We denote the dialogue context as c, which contains the user input utterance
and dialogue state, and the response utterance generated by the system as x.
Provided a dataset of context and optimal response pairs
${({\textnormal{c}},{\textnormal{x}})}$, we want to extract a latent
representation z, representing the dialogue responses given context.
Approaches based on variational inference have shown to be beneficial for
learning such latent representations. This is done by optimizing the evidence
lower bound (ELBO):

$\begin{split}{\mathcal{L}}(\phi,\theta)=\mathbb{E}_{q_{\theta}({\textnormal{z}}|{\textnormal{x}},{\textnormal{c}})}[-\log
p_{\phi}({\textnormal{x}}|{\textnormal{z}})]\\\
+D_{\mathrm{KL}}[q_{\theta}({\textnormal{z}}|{\textnormal{x}},{\textnormal{c}})||p({\textnormal{z}}|{\textnormal{c}})]\end{split}$
(1)

Where $q_{\theta}$ denotes the variational posterior parametrized by $\theta$
and $p_{\phi}$ the decoder parametrized by $\phi$. Arguing that the full ELBO
formulation suffers from “explaining away”, Zhao et al. (2019) introduced the
“lite” ELBO to mitigate this issue:

$\begin{split}{\mathcal{L}}(\phi,\theta)=\mathbb{E}_{q_{\theta}({\textnormal{z}}|{\textnormal{c}})}[-\log
p_{\phi}({\textnormal{x}}|{\textnormal{z}})]\\\
+D_{\mathrm{KL}}[q_{\theta}({\textnormal{z}}|{\textnormal{c}})||p({\textnormal{z}})]\end{split}$
(2)

The goal is to be closer to the information available during testing time,
where the decoder only sees $z$ conditionally sampled on the context
$q_{\theta}({\textnormal{z}}|{\textnormal{c}})$.

##### Learning Latent Dialogue Policies

After learning to extract a compressed representation z in the supervised
learning stage, the encoder $q_{\phi}({\textnormal{z}}|{\textnormal{c}})$ is
fine-tuned via reinforcement learning to optimize the dialogue reward, where
mostly directly the success metric is used. A Markov Decision Process
(${\mathcal{M}\mathcal{D}\mathcal{P}}$) is defined as a tuple
($\mathcal{S},\mathcal{A},r,p$) of state space $\mathcal{S}$, action space
$\mathcal{A}$, reward function $r$, and transition density $p$. The general
goal of reinforcement learning is to optimize the expected return of policy
$\pi$, denoted as $\mathbb{E}[J(\pi)]$. Many works utilize a Monte-Carlo
estimate of the policy gradient:

$\nabla_{\phi}\mathbb{E}\Big{[}J(\pi_{\phi})\Big{]}=\mathbb{E}\Big{[}\nabla_{\phi}\sum_{({\textnormal{s}},{\textnormal{a}})\in\tau}\log\pi_{\phi}({\textnormal{a}}|{\textnormal{s}})r({\textnormal{s}},{\textnormal{a}})\Big{]}$
(3)

Where the expectation is taken over the trajectory distribution
$\eta^{\pi}(\tau)$, i.e. distribution of sequences of
$({\textnormal{s}},{\textnormal{a}})$. In the context of ToD, we may cast the
state s as being the underlying dialogue state which may be unobserved, in
which case the problem becomes partially observable. In our setting, the
latent dialogue policy is warm-started by the parameters of the variational
posterior from the first stage of training.

##  3 Method

Figure 1:  High-level schema of the proposed supervised training stage. The
encoder receives either context or both the context and the response that is
to be decoded from latent z. In a sense, when samples from
$q({\textnormal{z}}|{\textnormal{c}},{\textnormal{x}})$ are used for decoding,
the architecture acts as a proper conditional autoencoder. We use the normal
distribution family for modelling the distribution of z.

In line with the general latent action RL paradigm outlined in Sec. 2, TCUP is
also based on a two-stage approach. In Sec. 3.1 we describe the reformulation
of the ELBO used in the supervised learning stage, which is vital to reach
state-of-the performance with continuous variational posteriors. In Sec. 3.2
we discuss our reinforcement learning setup and introduce methods to address
the deterioration of _latent action_ based dialogue policies by
regularization.

###  3.1 Revisiting the Full ELBO

As aforementioned prior work by Zhao et al. (2019) has introduced the “lite”
ELBO (described in equation 2) to alleviate the overexposure bias that emerges
when they incorporate response information in the optimization objective.
Indeed, if we only optimize for samples of
$q({\textnormal{z}}|{\textnormal{c}},{\textnormal{x}})$, the most information
about the responses (x) is found within the responses themselves. However, not
conditioning on x will reduce the expressiveness of the variational posterior,
especially if responses and contexts share indicative patterns.

In this work we rely on a full ELBO and introduce a conditional prior
${q^{p}_{\zeta}}({\textnormal{z}}|{\textnormal{c}})$ indicated by the
superscript $p$ which is constrained by the free prior $p(z)$ and also
parametrized. This leads us to the following objective:

$\begin{split}{\mathcal{L}}(\phi,\theta,\zeta)&=D_{\mathrm{KL}}[q_{\theta}||{q^{p}_{\zeta}}]+D_{\mathrm{KL}}[{q^{p}_{\zeta}}||p({\textnormal{z}})]-\\\
&\mathbb{E}_{q_{\theta}}[\log{p_{\phi}}({\textnormal{x}}|{\textnormal{z}})]-\mathbb{E}_{q^{p}_{\zeta}}[\log{p_{\phi}}({\textnormal{x}}|{\textnormal{z}})]\end{split}$
(4)

We defer the full derivation of this objective to the Appendix Sec. A.

During training time, we leverage samples from both the prior and the
variational posterior for reconstruction, which makes sure that the decoder
doesn’t overly rely on information from x. Apart from achieving better
performance as outlined in Sec. 4, this also simplifies the training procedure
considerably.

###  3.2 Regularizing for Success and Coherence

Reinforcement learning algorithms have shown to be able to exploit weaknesses
in simulators of game environments (Mnih et al., 2013) by finding high reward
states which don’t align with solving the underlying task. Similarly, in the
context of latent action RL for ToD, the decoder can be seen as a weak
simulator that is prone to exploitation due to the nature of the success rate
metric. By checking for relevant slots to be present in the response, success
rate permits that long, possibly incoherent responses more easily achieve
success due to higher chance of containing the correct slot tokens. Anecdotal
evidence for this is presented in Table 2. When using continuous latent
actions, this issue is even more severe, since the model is able to sample out
of distribution (OOD) utterances that provide success. For measuring response
coherence, we rely on the BLEU metric as specified in the MultiWOZ benchmark.

#### Penalizing Out of Distribution Samples

In the supervised training stage we apply variational inference and implicitly
maximize the BLEU score through maximizing the ELBO(see equation 3). We denote
this BLEU-maximizing policy with $\pi^{VI}$. Since we constrain our policy
with an isotropic Gaussian prior in the first training stage, we can leverage
this information to prevent the policy from deviating from this prior in the
form of a divergence cost which can be efficiently computed. Concretely, the
regularized reward function is defined as follows:

$\begin{split}r({\textnormal{x}},{\textnormal{c}})=succ({\textnormal{x}},{\textnormal{c}})-\beta
D_{\mathrm{KL}}[\pi({\textnormal{z}}|{\textnormal{c}})\,||\,p({\textnormal{z}})]\end{split}$
(5)

#### ToD as Offline RL

We are dealing with an offline reinforcement learning problem, since we have a
dataset of optimal responses without the possiblity of obtaining more samples
via a simulator or users. By shifting the reinforcement learning problem to
the latent space, we are implicitly creating a surrogate online problem, where
we need to obtain samples form $\pi({\textnormal{z}}|{\textnormal{c}})$ and
evaluate them.

We argue that one of the reasons why Gausssian latent spaces have been
reported as under-performing in comparison to categoricals is the biased and
noisy gradient estimate based on samples from a single dialogue. Contrary to
prior work Lubis et al. (2020); Zhao et al. (2019), which estimates the policy
gradient over the responses from a single dialogue sample, we take advantage
of a Monte Carlo policy gradient estimate across _multiple dialogues_.

#### Replaying Succesful Samples

Re-using encountered experience by storing it in memory (also called a replay
buffer) has proved to be beneficial for sample-efficiency in reinforcement
learning. However, a naive usage of the buffer has multiple caveats in the
MultiWOZ setting. Firstly, since the success signal is calculated on the
dialogue level, some responses that might be actually successful conditioned
on the dialogue state and context, might be labelled as negative. Intuitively,
the policy can start off the dialogue correctly, but fail to successfully
complete it. This can lead to conflicting examples in the replay buffer, which
destabilises training. Secondly, we have a many-to-one mapping from responses
to success, which leads to multimodality, but also modes that might be
incoherent.

If we make the observation that the response at time step $t$ is conditionally
independent of the dialogue history given the dialogue state and input
utterance, we notice that we can attribute success directly to the response
independent of past utterances. This motivates the storage of only successful
responses in the replay buffer, which we sample as a fraction of the training
batch. That way it mitigates the problem of false negative responses and
ensures fewer conflicting examples in the batch. Replaying past experience
also ensures that certain difficult samples are not forgotten, which both
increases training stability and ensures that the current policy $\pi$ stays
close to $\pi^{VI}$.

In practice, we exchange a certain sample in the batch generated by the
current policy with a sample from the replay buffer with probability
$\lambda$. As $\lambda\rightarrow 0$ we arrive to the simple REINFORCE update,
$\lambda=1$ means that we only use replayed samples for updates.

Table 1: Results comparison to competing methods on MultiWOZ. We have marked
methods that utilize RL with $\dagger$, transformer architecture with $\star$.

Model | BLEU | Inform | Success | Av. len. | CBE | # unigrams | # trigrams
---|---|---|---|---|---|---|---
MarCo⋆ | 17.3 | 94.5 | 87.2 | 16.01 | 1.94 | 319 | 3002
HDSA | 20.7 | 87.9 | 79.4 | 14.42 | 1.64 | 259 | 2019
HDNO† | 17.8 | 93.3 | 83.4 | 14.96 | 0.84 | 103 | 315
SFN† | 14.1 | 93.4 | 82.3 | 14.93 | 1.63 | 188 | 1218
UniConv | 18.1 | 66.7 | 58.7 | 14.17 | 1.79 | 338 | 2932
LAVA†111Results taken for best runs, mean performance is actually lower, additional commentary available in Appendix Sec. C | 16.2 | 89.7 | 77.6 | 14.41 | 1.96 | 272 | 2365
LAVA†-Cat | 10.8 | 95.9 | 93.5 | 13.28 | 1.27 | 176 | 708
TCUP† | 10.3 | 96.3 | 95.9 | 15.14 | 1.44 | 210 | 1838
TCUP†-Cat | 14.3 | 96.1 | 92.1 | 14.33 | 1.61 | 230 | 1490

Table 2: Example of successfull responses where the policy was able to “game” the metric, leading to a successful dialogue but resulting in incoherent utterances and hence low BLEU score. USER |  we are staying [value_count] people for [value_count] nights starting from [value_day] . i need the reference number?
---|---
SYSTEM |  [hotel_name] [hotel_name] [hotel_name] [hotel_reference] [hotel_name] [hotel_reference] [hotel_name] [hotel_reference] [hotel_name] [hotel_phone] [hotel_phone] [hotel_phone] [hotel_phone] [hotel_phone] [hotel_phone] [hotel_phone] [hotel_phone] [hotel_phone] [hotel_name] [hotel_phone] [hotel_name] -ly [hotel_phone] [hotel_name] -ly [hotel_name] -ly [hotel_name] -ly [hotel_name] -ly [hotel_name] -ly [hotel_name] …
USER |  just the address please
SYSTEM |  [attraction_name] is located at [attraction_address], postcode [attraction_postcode]. the phone number is [attraction_phone]. [attraction_name] is located at [attraction_address], [attraction_postcode]. phone number is [attraction_phone] [attraction_name] is located at [attraction_address], [attraction_postcode]. phone number is [attraction_phone]. [attraction_name] is located at [attraction_address], [attraction_postcode]. phone

##  4 Experiments

We provide a detailed evaluation of TCUP’s dialogue policy in Sec. 4.1. This
includes performance on the MultiWOZ benchmark; an ablation study to assess
the importance of our technical contributions proposed in Sec. 3; and a
qualitative analysis of response coherence. Finally, we analyze the quality of
the latent representations in Sec. 4.2 and provide evidence that TCUP is able
to learn representations using continuous latent variables which yield good
separation and clustering of domains and actions present in MultiWOZ. We make
use of a recurrent encoder-decoder architecture with dot-product attention,
further details on the architecture and training procedure can be found in
Appendix Sec. E.

##### Supervised Learning Stage (SL)

Here, we learn the mapping from context to response using equation 1. In
practice, we optimize the $p_{\phi}({\textnormal{x}}|{\textnormal{z}})$ part
with a weighted cross-entropy loss that puts higher weights on slot
placeholders in the response. The best model is selected based on its BLEU
score.

##### Reinforcement Learning Stage (RL)

In this stage we fix the parameters of the decoder and train only the encoder
parameters via policy gradient with two important modifications: (i) we use a
batched version of the policy gradient which contains samples from multiple
dialogues and hence reduces the variance of the gradient, and (ii) in each
batch we sample a mix of newly generated and old experience (with which we
have obtained a success signal earlier). This implicitly keeps us close to the
starting policy which results in more stable training.

We have found that it is also beneficial to replace the standard
$D_{\mathrm{KL}}[q||p]$ term of the variational objective with a symmetric
version of it $\frac{1}{2}(D_{\mathrm{KL}}[q||p]+D_{\mathrm{KL}}[p||q])$. This
ensures that regions where the densities of $p$ and $q$ behave differently are
treated equally irrespective of the ordering.

###  4.1 Context-to-Response Generation

We validate the proposed method on the MultiWOZ benchmark (Wang et al., 2020)
in the policy learning task with ground-truth dialoge states and use the same
delexicalization approach as in Lubis et al. (2020). Table 1 shows that TCUP
improves the state-of-the-art inform- and success rate metric across all
competitors. Also, it is competitive in terms of language diversity metrics.
Compared to the currently best performing latent action reinforcement learning
approach (LAVA), we increase all metrics except for minor decreases in the
BLEU metric. The response coherence score is further discussed in Sec. 4.1.

##### Latent Representations

While prior latent-action reinforcement learning approaches(Lubis et al.,
2020; Zhao et al., 2019) favor categorical latent distributions with modified
attention mechanisms in the decoder, our results demonstrate that relying on
isotropic Gaussian latents is advantageous, even if we only use simple dot-
product attention in the decoder. In particular, if we base TCUP on
categorical latent distributions (coined TCUP-Categorical in Table 1) we
observe competitive results in terms of inform and success rate compared to
prior work, but inferior performance compared to TCUP with continuous latents.

Using categoricals limits us in the sense that we don’t make full use of the
nonlinearity in the decoder to decode diverse responses, which manifests
itself with poor results in the diversity metrics (Table 1). By using a
continuous latent distribution, we are able to improve the diversity of the
generated responses compared to those two competitors.

##### Ablation Study

Fig. 2 presents an ablation study of different parts of TCUP. In general,
optimal replay (_rep_) and the constrained rewards with the additional KL
penalty (_KL_) lead to higher success rate and BLEU, with optimal replay being
the superior choice between the two, especially in the BLEU metric. Further
gains can be achieved by utilizing both. Not using optimal replay introduced
higher variance across runs. During the first training stage, utilizing prior
and posterior shuffling (_sl shuff_) enables a better fit in terms of BLEU
scores. Histograms of further metrics used in the MultiWOZ benchmark can be
found in Appendix Sec. B. In summary, these results (_shuff kl rep_) validate
the benefits of the proposed variational inference objective and
regularization.

Figure 2: Ablation study. All TCUP configurations are averaged over three
random seeds. _sl_ denotes supervised training, _sl+rl_ supervised training
followed by reinforcement learning, _ctxp_ usage of contextual prior, but
without any prior on it, _kl_ the addition of KL penalty term to identity
covariance Gaussian, _larl_ the approach proposed by Zhao et al. (2019),
_shuff_ prior and posterior shuffling in the supervised stage, _pp_ identity
covariance Gaussian prior on the contextual prior, _rep_ usage of replay
buffer

#### Response Coherence

We have observed that it is possible to maximize the dialogue success rate at
the cost of lower coherence in terms of BLEU scores. In Fig. 4 we show that we
achieve state-of-the-art success rates, whilst achieving a BLEU score of
effectively 0. This is an artifact of the success rate metric in that it
disregards coherence of responses and only checks if the correct slot values
have been addressed by the dialogue policy.

In Fig. 3 we see different runs of our method with different strength of
regularization in terms of KL penalty and optimal replay fraction. Depending
on the strength of regularization they form a Pareto front. This further shows
the multi-objective trade-off between success rates and BLEU scores.

Figure 3: Each point depicts a training run and the line is a 2nd degree
polynomial fit of the runs illustrating the Pareto front.

The BLEU collapse is to be expected when using latent-action reinforcement
learning, since longer and more diverse answers have positive impact on the
dialogue success, which leads to the policy selecting degenerate responses.
Furthermore, if no additional regularization is used, the latent policy learns
to sample outliers in terms of the prior over z. We argue that it is not
realistic to expect the policy resulting from the reinforcement learning stage
to outperform the response coherence of the supervised learning stage in terms
of BLEU (as long as the primary purpose of the RL stage is to improve dialogue
success metrics). Instead, we show that through regularization and optimal
replay, the BLEU score can be kept from deterioration through the RL stage
while we optimize dialogue level metrics. An alternative approach to
alleviating degenerate policies would be to simply make the BLEU score or
other coherence metrics part of the reward function. For example, the final
comparison of models in (Budzianowski et al., 2018) is done by comparing
$\frac{success+inform}{2}+BLEU$. However, we suggest that there are multiple
problems with the approach of maximizing such a hybrid metric directly. First
of all, constructing reliable metrics is challenging Jiang et al. (2021);
Mehri and Eskenazi (2020) in itself. More importantly, as soon as the
coherence metric is part of the reward, we encounter the problem of adequate
scaling in comparison to the success rate.

---
Figure 4: Influence of optimal replay on success rate and BLEU score over
multiple epochs

##### Impact of Regularization

When introducing regularization in the form of optimal replay sampling and the
KL penalty term, we are able to achieve state-of-the-art performance in terms
of success rate without lowering the BLEU score significantly. As depicted in
Fig. 4 our method using optimal replay buffer sampling is more stable during
training in comparison to the naive application of the policy gradient. Also,
we observe that the BLEU deterioration is kept at bay and roughly deteriorates
linearly over time. By increasing the replay fraction $\lambda$ too much we
are over-constraining the latent dialogue policy to the initial experience,
which leads to biased updates and hurts exploration. Nevertheless, as shown in
the ablation study (Fig. 2) optimal replay and the KL penalty term have shown
to be essential for preventing the BLEU score from deteriorating too rapidly.
In this work we aimed to retain a BLEU score that is competitive to that
reported by LAVA (Lubis et al., 2020), while improving overall dialogue
success. A sensitivity analysis over the weights of the penalty term and the
replay fraction $\lambda$ is described in Appendix Fig. 8.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 5: UMAP embeddings of latent representations. Figures a-c show domain
labeled embeddings for the SL, SL+RL and SL+RL with replay sampling,
respectively. In the bottom row we find the same embeddings labeled by
response type. Naively executing the RL stage of the training results in
representations that are difficult to separate (b and e). When applying replay
sampling we obtain higher specialization in the clusters (c and f).

###  4.2 Latent Space Analysis

Fig. 5 depicts a UMAP(McInnes et al., 2018) projection of the learned latent
samples z for the Gaussian case. We observe similar behavior to Lubis et al.
(2020). In the supervised learning stage there is apparent clustering in terms
of domain labels (Fig. 5(a)). The reinforcement learning stage with
regularization leads to specialization of the clusters (Fig. 5(c)). In
comparison, a good cluster separation is lost (Fig. 5(b)) without applying
regularization in the RL stage. This can be explained by the fact that the z
samples are degenerate samples that lie in low-support regions of
${q^{p}_{\zeta}}({\textnormal{z}}|{\textnormal{c}})$.

We have calculated the Caliński-Harabasz index (Caliński and Harabasz, 1974),
otherwise known as Variance Ratio Criterion, to evaluate the clusterings in
the latent space with respect to domain and action type labelings taken from
DAMD (Zhang et al., 2020). In comparison to the results reported by Lubis et
al. (2020), our model is able to obtain high scores in the supervised stage of
training already. The scores drop slightly after RL fine-tuning, but generally
remain at a much higher level than those reported for categorical latents.
It’s important to note that the scores for LaRL∗ and LAVA∗ were computed with
categorical latent variables, whereas our scores are based on Gaussian
latents, which are continuous and unbounded.

Consequently, the purpose of Table 3 is to demonstrate the value in using
continuous latent representations, rather than to make direct numeric
comparisons.

Table 3: Caliński-Harabasz scores (higher is better). Model | SL | RL
---|---|---
Domain | Action | Domain | Action
LaRL∗ | 93.19 | 23.20 | 121.15 | 17.5
LAVA∗ | 104.92 | 25.28 | 158.00 | 41.75
TCUP | 345.48 | 80.76 | 329.39 | 78.95
11footnotetext: Here we take the best scores of each method obtained by Lubis
et al. (2020)

##  5 Related Work

Supervised Learning The majority of prior work applies some form of Supervised
Learning. For an extensive overview we refer to Gao et al. (2018) and focus on
the state-of-the-art competitors HDSA Chen et al. (2019) as well as UniConv Le
et al. (2020). Both approaches demonstrate the benefits of jointly training
multiple dialogue tasks at once, such as predicting dialogue acts and states.

Variational Inference Several works employ variational inference for learning
conditional response distributions. VHRED Serban et al. (2017) is a
variational hierarchical RNN architecture for modelling the dependencies
between words and utterances. Stochastic latent variables are introduced to
generate the next utterance in a conversation. To avoid the problem of
collapsing posteriors Zhao et al. (2017) aim to learn better response
representation using an auxiliary task introducing a bag of words prediction
loss. Shen et al. (2018) improves the stability of VHRED by splitting the
training process into two parts, where first text is auto-encoded into
continuous embeddings, which serve as starting point for learning general
latent representations by reconstructing the encoded embedding. To address the
degeneration problem, Variational Hierarchical Conversation RNNs (VHCR) (Park
et al., 2018) exploits an utterance drop regularization. DialogWAE (Gu et al.,
2018) represents the prior distribution as Gaussian mixture and adapts WGAN
(Arjovsky et al., 2017) for training.

Combined Supervised- & Reinforcement Learning Henderson et al. (2008) was the
first work to combine Reinforcement and Supervised Learning introducing a
value function, which relies on SL for predicting the expected future reward
of states not covered by the data directly. Williams et al. (2017) prevent
high variance by performing a SL gradient update, if the RL policy output
deviates from the training data. Fatemi et al. (2016); Su et al. (2017) apply
two-stage approaches, where they pretrain a dialogue policy supervised, which
is then further optimized by RL. HappyBot (Shin et al., 2019) relies on a
weighted combination of a maximum likelihood and a REINFORCE objective. Saleh
et al. (2020) uses REINFORCE-based policy gradients to update the prior
probability distribution of the latent variational model trained using
supervised learning. Structured Fusion Networks (SFNs) (Mehri et al., 2019)
apply RL to fuse dialogue modules, where each module serves a different
purpose (NLU, NLG, etc.) and is pretrained in individual supervised stages.
Apart from task-oriented dialogues a combination of SL and RL has also been
applied for generating responses for open-domain dialogues (Xu et al., 2018).
The aforementioned approaches are based on word-level RL, suffering from huge
action-spaces covering the entire input vocabulary. Due to this, ensuring
coherent responses is challenging (Lewis et al., 2017; Kottur et al., 2017),
especially in multi-turn dialogs spanning hundreds of words. LaRL (Zhao et
al., 2019) and LAVA (Lubis et al., 2020) address this problem by learning a
low-dimensional latent representation with amortized variational inference
followed by a RL fine-tuning stage where the decoder parameters are frozen and
the encoder is fine-tuned.

##  6 Conclusion

In this paper, we showed that with appropriate modifications to the
reinforcement learning procedure and the latent space structure, it is
possible to obtain state-of-the-art results in task-oriented dialogue with
simple Gaussian latent distributions. The problem of coherence deterioration
when optimizing for success rate points to the fact that better metrics are
needed for developing efficient dialogue policies - we were able to alleviate
this with optimal replay and KL regularization.

Although TCUP has shown promising results in the policy learning setting with
access to ground-truth dialogue state, it would be of interest to see how TCUP
compares in the end-to-end learning setting. Furthermore, the deterioration of
the BLEU score is tightly connected to the capacity of the model to achieve
good likelihood fits in the supervised learning stage. Our experiments also
hint that this variational formulation can benefit from higher-capacity
models. Along these lines, increasing capacity and making use of the
transformer architecture is a promising future research direction.

## References

  * Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017.  Wasserstein generative adversarial networks.  In _International conference on machine learning_ , pages 214–223. PMLR.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  _arXiv preprint arXiv:1810.00278_.
  * Caliński and Harabasz (1974) Tadeusz Caliński and Jerzy Harabasz. 1974.  A dendrite method for cluster analysis.  _Communications in Statistics-theory and Methods_ , 3(1):1–27.
  * Chen et al. (2019) Wenhu Chen, Jianshu Chen, Pengda Qin, Xifeng Yan, and William Yang Wang. 2019.  Semantically conditioned dialog response generation via hierarchical disentangled self-attention.  _arXiv preprint arXiv:1905.12866_.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using rnn encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734.
  * Fatemi et al. (2016) Mehdi Fatemi, Layla El Asri, Hannes Schulz, Jing He, and Kaheer Suleman. 2016.  Policy networks with two-stage training for dialogue systems.  _arXiv preprint arXiv:1606.03152_.
  * Gao et al. (2018) Jianfeng Gao, Michel Galley, and Lihong Li. 2018.  Neural approaches to conversational ai.  In _The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval_, pages 1371–1374.
  * Gao et al. (2019) Jianfeng Gao, Michel Galley, and Lihong Li. 2019.  now.  [link].
  * Gu et al. (2018) Xiaodong Gu, Kyunghyun Cho, Jung-Woo Ha, and Sunghun Kim. 2018.  Dialogwae: Multimodal response generation with conditional wasserstein auto-encoder.  _arXiv preprint arXiv:1805.12352_.
  * Henderson et al. (2008) James Henderson, Oliver Lemon, and Kallirroi Georgila. 2008.  Hybrid reinforcement/supervised learning of dialogue policies from fixed data sets.  _Computational Linguistics_ , 34(4):487–511.
  * Henderson et al. (2019) Matthew Henderson, Ivan Vulić, Daniela Gerz, Iñigo Casanueva, Paweł Budzianowski, Sam Coope, Georgios Spithourakis, Tsung-Hsien Wen, Nikola Mrkšić, and Pei-Hao Su. 2019.  Training neural response selection for task-oriented dialogue systems.  _arXiv preprint arXiv:1906.01543_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Inaba and Takahashi (2016) Michimasa Inaba and Kenichi Takahashi. 2016.  Neural utterance ranking model for conversational dialogue systems.  In _Proceedings of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 393–403.
  * Jiang et al. (2021) Haoming Jiang, Bo Dai, Mengjiao Yang, Tuo Zhao, and Wei Wei. 2021.  Towards automatic evaluation of dialog systems: A model-free off-policy evaluation approach.  _arXiv preprint arXiv:2102.10242_.
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013.  Auto-encoding variational bayes.  _arXiv preprint arXiv:1312.6114_.
  * Kottur et al. (2017) Satwik Kottur, José MF Moura, Stefan Lee, and Dhruv Batra. 2017.  Natural language does not emerge’naturally’in multi-agent dialog.  _arXiv preprint arXiv:1706.08502_.
  * Le et al. (2020) Hung Le, Doyen Sahoo, Chenghao Liu, Nancy F Chen, and Steven CH Hoi. 2020.  Uniconv: A unified conversational neural architecture for multi-domain task-oriented dialogues.  _arXiv preprint arXiv:2004.14307_.
  * Lee et al. (2021) Hwaran Lee, Seokhwan Jo, Hyungjun Kim, Sangkeun Jung, and Tae-Yoon Kim. 2021.  Sumbt+ larl: Effective multi-domain end-to-end neural task-oriented dialog system.  _IEEE Access_ , 9:116133–116146.
  * Lewis et al. (2017) Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. 2017.  Deal or no deal? end-to-end learning for negotiation dialogues.  _arXiv preprint arXiv:1706.05125_.
  * LI et al. (2020) SHIYANG LI, Semih Yavuz, Kazuma Hashimoto, Jia Li, Tong Niu, Nazneen Rajani, Xifeng Yan, Yingbo Zhou, and Caiming Xiong. 2020.  Coco: Controllable counterfactuals for evaluating dialogue state trackers.  In _International Conference on Learning Representations_.
  * Lubis et al. (2020) Nurul Lubis, Christian Geishauser, Michael Heck, Hsien-Chin Lin, Marco Moresi, Carel van Niekerk, and Milica Gasic. 2020.  LAVA: latent action spaces via variational auto-encoding for dialogue policy optimization.  In _Proceedings of the 28th International Conference on Computational Linguistics, COLING 2020, Barcelona, Spain (Online), December 8-13, 2020_ , pages 465–479. International Committee on Computational Linguistics.
  * McInnes et al. (2018) Leland McInnes, John Healy, and James Melville. 2018.  Umap: Uniform manifold approximation and projection for dimension reduction.  _arXiv preprint arXiv:1802.03426_.
  * Mehri and Eskenazi (2020) Shikib Mehri and Maxine Eskenazi. 2020.  Unsupervised evaluation of interactive dialog with dialogpt.  _arXiv preprint arXiv:2006.12719_.
  * Mehri et al. (2019) Shikib Mehri, Tejas Srinivasan, and Maxine Eskenazi. 2019.  Structured fusion networks for dialog.  _arXiv preprint arXiv:1907.10016_.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_.
  * Park et al. (2018) Yookoon Park, Jaemin Cho, and Gunhee Kim. 2018.  A hierarchical latent structure for variational conversation modeling.  _arXiv preprint arXiv:1804.03424_.
  * Saleh et al. (2020) Abdelrhman Saleh, Natasha Jaques, Asma Ghandeharioun, Judy Shen, and Rosalind Picard. 2020.  Hierarchical reinforcement learning for open-domain dialog.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8741–8748.
  * Serban et al. (2017) Iulian Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2017.  A hierarchical latent variable encoder-decoder model for generating dialogues.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31.
  * Shen et al. (2018) Xiaoyu Shen, Hui Su, Shuzi Niu, and Vera Demberg. 2018.  Improving variational encoder-decoders in dialogue generation.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32.
  * Shin et al. (2019) Jamin Shin, Peng Xu, Andrea Madotto, and Pascale Fung. 2019.  Happybot: Generating empathetic dialogue responses by improving user experience look-ahead.  _arXiv preprint arXiv:1906.08487_.
  * Su et al. (2017) Pei-Hao Su, Pawel Budzianowski, Stefan Ultes, Milica Gasic, and Steve Young. 2017\.  Sample-efficient actor-critic reinforcement learning with supervised data for dialogue management.  _arXiv preprint arXiv:1707.00130_.
  * Tao et al. (2019) Chongyang Tao, Wei Wu, Can Xu, Wenpeng Hu, Dongyan Zhao, and Rui Yan. 2019.  Multi-representation fusion network for multi-turn response selection in retrieval-based chatbots.  In _Proceedings of the twelfth ACM international conference on web search and data mining_ , pages 267–275.
  * Walker et al. (2007) Marilyn A Walker, Amanda Stent, François Mairesse, and Rashmi Prasad. 2007\.  Individual and domain adaptation in sentence planning for dialogue.  _Journal of Artificial Intelligence Research_ , 30:413–456.
  * Wang et al. (2020) Kai Wang, Junfeng Tian, Rui Wang, Xiaojun Quan, and Jianxing Yu. 2020.  Multi-domain dialogue acts and response co-generation.  _arXiv preprint arXiv:2004.12363_.
  * Williams et al. (2017) Jason D Williams, Kavosh Asadi, and Geoffrey Zweig. 2017.  Hybrid code networks: practical and efficient end-to-end dialog control with supervised and reinforcement learning.  _arXiv preprint arXiv:1702.03274_.
  * Xu et al. (2018) Can Xu, Wei Wu, and Yu Wu. 2018.  Towards explainable and controllable open domain dialogue generation with dialogue acts.  _arXiv preprint arXiv:1807.07255_.
  * Yan et al. (2016) Rui Yan, Yiping Song, and Hua Wu. 2016.  Learning to respond with deep neural networks for retrieval-based human-computer conversation system.  In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_ , pages 55–64.
  * Zhang et al. (2020) Yichi Zhang, Zhijian Ou, and Zhou Yu. 2020.  Task-oriented dialog systems that consider multiple appropriate responses under the same context.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 9604–9611.
  * Zhao et al. (2019) Tiancheng Zhao, Kaige Xie, and Maxine Eskenazi. 2019.  Rethinking action spaces for reinforcement learning in end-to-end dialog agents with latent variable models.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1208–1218, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Zhao et al. (2017) Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017.  Learning discourse-level diversity for neural dialog models using conditional variational autoencoders.  _arXiv preprint arXiv:1703.10960_.

##  Appendix A Revisiting the Full ELBO

Further we use shorthand $q$ for
$q({\textnormal{z}}|{\textnormal{x}},{\textnormal{c}})$ Firstly we define our
optimization problem:

$\displaystyle\min_{q}D_{\mathrm{KL}}[q||p({\textnormal{z}}|{\textnormal{x}},{\textnormal{c}})]$

czx Figure 6: Probabilistic model of dependencies between c, z and x

######  Lemma 1.

Given the probabilistic model diagram 6:

$\displaystyle
D_{\mathrm{KL}}[q||p({\textnormal{z}}|{\textnormal{x}},{\textnormal{c}})]\leq\mathbb{E}_{q}[\log
q]-$ $\displaystyle\mathbb{E}_{q}[\log
p({\textnormal{x}}|{\textnormal{z}})]-\mathbb{E}_{q}[\log
p({\textnormal{z}}|{\textnormal{c}})]$

###### Proof.

We expand the KL divergence of equation 6.

$\displaystyle
D_{\mathrm{KL}}[q||p({\textnormal{z}}|{\textnormal{x}},{\textnormal{c}})]=\mathbb{E}_{q}[\log
q-\log p({\textnormal{z}}|{\textnormal{x}},{\textnormal{c}})]$ (6)

We first expand the $p({\textnormal{z}}|{\textnormal{x}},{\textnormal{c}})$ by
Bayes rule and make use of independence between $x$ and $c$:

$\displaystyle
p({\textnormal{z}}|{\textnormal{x}},{\textnormal{c}})=\frac{p({\textnormal{x}},{\textnormal{c}}|{\textnormal{z}})p({\textnormal{z}})}{p({\textnormal{x}},{\textnormal{c}})}$
$\displaystyle=\frac{p({\textnormal{x}}|{\textnormal{z}})p({\textnormal{c}}|{\textnormal{z}})p({\textnormal{z}})}{p({\textnormal{x}},{\textnormal{c}})},\;{\textnormal{x}}\perp\\!\\!\\!\\!\perp{\textnormal{c}}|\,{\textnormal{z}}$
$\displaystyle=\frac{p({\textnormal{x}}|{\textnormal{z}})p({\textnormal{z}}|{\textnormal{c}})p({\textnormal{c}})}{p({\textnormal{x}},{\textnormal{c}})}$

Putting back these terms into equation 6:

$\displaystyle
D_{\mathrm{KL}}[q||p({\textnormal{z}}|{\textnormal{x}},{\textnormal{c}})]=$
$\displaystyle\mathbb{E}_{q}[\log q]-\mathbb{E}_{q}[\log
p({\textnormal{x}}|{\textnormal{z}})]-\mathbb{E}_{q}[\log
p({\textnormal{z}}|{\textnormal{c}})]$ $\displaystyle-\mathbb{E}_{q}[\log
p({\textnormal{c}})]+\mathbb{E}[\log p({\textnormal{x}},{\textnormal{c}})]$
$\displaystyle=\mathbb{E}_{q}[\log q]-\mathbb{E}_{q}[\log
p({\textnormal{x}}|{\textnormal{z}})]-\mathbb{E}_{q}[\log
p({\textnormal{z}}|{\textnormal{c}})]$ $\displaystyle-\log
p({\textnormal{c}})+\log p({\textnormal{x}},{\textnormal{c}})$
$\displaystyle\leq\mathbb{E}_{q}[\log q]-\mathbb{E}_{q}[\log
p({\textnormal{x}}|{\textnormal{z}})]$ $\displaystyle-\mathbb{E}_{q}[\log
p({\textnormal{z}}|{\textnormal{c}})]\,p({\textnormal{c}})\geq
p({\textnormal{x}},{\textnormal{c}})$

∎

In practice, we don’t have access to $p({\textnormal{z}}|{\textnormal{c}})$
and hence we estimate it with
$q^{p}_{\theta}({\textnormal{z}}|{\textnormal{c}})$, which results in an
$\operatorname*{arg\,min}$ operation within the expectation.

$\displaystyle\mathbb{E}_{q}[\log q]-\mathbb{E}_{q}[\log
p({\textnormal{x}}|{\textnormal{z}})]-\mathbb{E}_{q}[\log
p({\textnormal{z}}|{\textnormal{c}})]=$ $\displaystyle\mathbb{E}_{q}[\log
q]-\mathbb{E}_{q}[\log\operatorname*{arg\,min}_{q^{p}}D_{\mathrm{KL}}[q^{p}||p({\textnormal{z}}|{\textnormal{c}})]]$
$\displaystyle-\mathbb{E}_{q}[\log p({\textnormal{x}}|{\textnormal{z}})]$

The new KL term within the expectation can be expanded to obtain the
traditional evidence lower-bound, ie. bound on the objective:

$\displaystyle
D_{\mathrm{KL}}[q^{p}||p({\textnormal{z}}|{\textnormal{c}})]\leq
D_{\mathrm{KL}}[{q^{p}_{\zeta}}||p({\textnormal{z}})]$
$\displaystyle+\mathbb{E}_{q^{p}_{\zeta}}[p({\textnormal{c}}|{\textnormal{z}})]$

In the end, this is our objective function, but we want to get rid of
${p_{\phi}}({\textnormal{c}}|{\textnormal{z}})$.

$\displaystyle{\mathcal{L}}(\phi,\theta,\zeta)$
$\displaystyle=D_{\mathrm{KL}}[q_{\theta}||{q^{p}_{\zeta}}]+D_{\mathrm{KL}}[{q^{p}_{\zeta}}||p({\textnormal{z}})]-$
$\displaystyle\mathbb{E}_{q_{\theta}}[\log{p_{\phi}}({\textnormal{x}}|{\textnormal{z}})]-\mathbb{E}_{q^{p}_{\zeta}}[\log
p({\textnormal{c}}|{\textnormal{z}})]$

The current formulation would require us to approximate
$p({\textnormal{c}}|{\textnormal{z}})$.

######  Remark.

The KL term
$D_{\mathrm{KL}}[p({\textnormal{z}}|{\textnormal{c}})||p({\textnormal{z}}|{\textnormal{x}},{\textnormal{c}})]$
is zero iff x doesn’t provide any additional information about z.

We further simplify the problem by replacing
$p({\textnormal{c}}|{\textnormal{z}})$ with
${p_{\phi}}({\textnormal{x}}|{\textnormal{z}})$. By applying Bayes rule, we
arrive to the following:

$\displaystyle{\mathcal{L}}(\phi,\theta,\zeta)$
$\displaystyle=D_{\mathrm{KL}}[q_{\theta}||{q^{p}_{\zeta}}]+D_{\mathrm{KL}}[{q^{p}_{\zeta}}||p({\textnormal{z}})]-$
$\displaystyle\mathbb{E}_{q_{\theta}}[\log{p_{\phi}}({\textnormal{x}}|{\textnormal{z}})]-\mathbb{E}_{q^{p}_{\zeta}}[\log{p_{\phi}}({\textnormal{x}}|{\textnormal{z}})]$
$\displaystyle-\mathbb{E}_{q^{p}_{\zeta}}[\log\frac{p({\textnormal{c}}|{\textnormal{z}})}{p_{\theta}({\textnormal{x}}|{\textnormal{z}})}]$

######  Lemma 2.

$p(z|x,c)\approx E_{p(c)}[p(z|x,c)]$ implies that
$p(c|z)\approx\frac{p(c)}{p(x)}p(x|z)$

###### Proof.

$\displaystyle p(z|x,c)\approx E_{p(c)}[p(z|x,c)]$ $\displaystyle
p(c|z)=p(z|c)p(c)/p(z)$ $\displaystyle p(x|z)=p(z|x)p(x)/p(z)$
$\displaystyle\frac{p(c|z)}{p(x|z)}\approx\frac{p(c)p(z|c,x)}{p(x)\mathbb{E}_{p(c)}[p(z|x,c)]}$
$\displaystyle\frac{p(c|z)}{p(x|z)}\approx\frac{p(c)}{p(x)}$

∎

######  Proposition 1.

From the above it follows that for the optimization problem
$\min_{q}D_{\mathrm{KL}}[q||p({\textnormal{z}}|{\textnormal{x}},{\textnormal{c}})]$
it suffices to optimize the following objective:

$\displaystyle{\mathcal{L}}(\phi,\theta,\zeta)$
$\displaystyle=D_{\mathrm{KL}}[q_{\theta}||{q^{p}_{\zeta}}]+D_{\mathrm{KL}}[{q^{p}_{\zeta}}||p({\textnormal{z}})]-$
$\displaystyle\mathbb{E}_{q_{\theta}}[\log{p_{\phi}}({\textnormal{x}}|{\textnormal{z}})]-\mathbb{E}_{q^{p}_{\zeta}}[\log{p_{\phi}}({\textnormal{x}}|{\textnormal{z}})]+C$

How accurate the approximation is depends highly on on the relationship
between $p({\textnormal{z}}|{\textnormal{x}})$ and
$p({\textnormal{z}}|{\textnormal{c}})$.

######  Lemma 3.

In the case where
$p({\textnormal{z}}|{\textnormal{x}})=p({\textnormal{z}}|{\textnormal{c}})$,
the approximation from 2 holds with equality.

###### Proof.

This can be easily seen by replacing the
$p({\textnormal{z}}|{\textnormal{x}})$ and
$p({\textnormal{z}}|{\textnormal{c}})$ in the proof of Lemma 2, we arrive to
the same result without making the approximation step. ∎

Intuitively, $p({\textnormal{z}}|{\textnormal{x}})$ and
$p({\textnormal{z}}|{\textnormal{c}})$ in the ideal case would be equal, i.e.
knowing either x or c would be enough to obtain all the information about z,
which allows for good information compression.

In our approach, we optimize
$\mathbb{E}_{q^{p}_{\zeta}}[{p_{\phi}}({\textnormal{x}}|{\textnormal{z}})]$
and $\mathbb{E}_{q_{\theta}}[{p_{\phi}}({\textnormal{x}}|{\textnormal{z}})]$
by interchangebly sampling from ${p_{\phi}}$ and ${q^{p}_{\zeta}}$ based on a
fair coin flip, ie Bernoulli distribution and a assume an identity covariance
Gaussian prior $p({\textnormal{z}})$.

##  Appendix B Evaluation Histograms

In Fig. 7 we show histograms of different evaluation metrics for the MultiWOZ
2.1 dataset and different approaches. We ran 3 seeds for each of the methods.
Most notably, we notice that we are able to obtain the highest BLEU score in
the supervised learning stage by utilizing prior and posterior shuffling,
denoted as _sl shuff_ , with the result slightly increasing when using an
additonal identity covariance Gaussian prior on the prior (_sl pp shuff_).

(a)  (b)  (c)  (d)  (e)  (f)

Figure 7:  Here we show results for different metrics on MultiWOZ 2.1 for different approaches denoted on the x-axis. _sl_ denotes supervised trainig, _sl+rl_ denotes supervised training followed by reinforcement learning, _ctxp_ usage of contextual priror, but without any prior on it, _kl_ the addition of KL penalty to identity covariance Gaussian, _larl_ the approach proposed by Zhao et al. (2019), _shuff_ prior and posterior shuffling in the supervised stage, _pp_ identity covariance Gaussian prior on the contextual prior, _rep_ usage of replay buffer for the reinforcement learning fine-tuning stage of training.  USER |  hi . i was wondering if there are any eastern [value_food] type restaurant -s in the [value_area] part of the city?
---|---
SYSTEM |  [restaurant_name] is a [value_food] restaurant in the [value_area] of town . it is [value_pricerange] -ly priced and located at [restaurant_address]. would you like to book a table?
USER |  i am looking for an [value_pricerange] eastern [value_food] place in the [value_area]
SYSTEM |  [restaurant_name] is a [value_pricerange] [value_food] restaurant in the [value_area] of town . would you like me to book a table for you?
USER |  how about a [value_food] restaurant?
SYSTEM |  i have [value_count] [value_pricerange] [value_food] restaurant -s in the [value_area] [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name], [restaurant_name]
USER |  i would like a reservation for [value_count] to the [restaurant_name] restaurant.
SYSTEM |  i have booked you a table for [value_count] at [restaurant_name] on [value_day] at [value_time]. your reference number is [restaurant_reference].
USER |  i would like to make a reservation for [value_day] at [value_time] . and there has been a change in plans , i will be dining alone.
SYSTEM |  your booking was successful . the table will be reserved for [value_count] minutes . reference number is [restaurant_reference].
USER |  can i get the reference number for that reservation ?
SYSTEM |  your booking was successful . the table will be reserved for [value_count] minutes . reference number is [restaurant_reference].
USER |  excellent . that is everything!
SYSTEM |  you are welcome . have a great day!
Table 4: Example of resulting delexicalized dialogue outputted by learned
dialogue policy learned with optimal replay.

##  Appendix C LAVA Gauss Results

To obtain scores with the newest evaluation method for MultiWOZ, we ran the
code available at https://gitlab.cs.uni-duesseldorf.de/general/dsml/lava-
public. We didn’t make any modification to the hyperparameters and ran the
reinforcement learning stage for 10 epochs. Notably, the LAVA method with
continuous latent exibits high variance in the success scores, with a standard
deviation of $3.32$ and mean success rate $73.33$ across 3 runs, it is able to
keep the BLEU score high since it’s under-optimizing the success metric and is
trained for few epochs which prevents divergence. In fact, TCUP can obtain the
same BLEU score by sacrificing a bit on the success rate side, see Fig. 3.

##  Appendix D Regularization Sensitivity

In Fig. 8 we do a regularization sensitivity analysis for different scales of
KL regularization and fraction of optimal replay in the reinforcement learning
training stage. We do the same for the categorical latent variable case in
Fig. 9.

(a)  (b)  (c)

Figure 8: Regularization sensitivity 400 epochs.

##  Appendix E Model and Training Setup

We make use of the Long-Short Term Memory (Hochreiter and Schmidhuber, 1997)
with a dot-product attention mechanism for the decoder architecture and Gated
Recurrent Unit (Cho et al., 2014) for the encoder architecture. In both cases,
we use a hidden size of $300$ and a latent embedding vector of size $200$.
Note that in the case of prior-posterior shuffling, there are shared
parameters for ${p_{\phi}}$ and ${q^{p}_{\zeta}}$ through the encoder, the
hidden state is mapped to the appropriate mean and variance by independent
neural networks. Under the context c we assume access to the ground-truth
dialogue state, this can be easily replaced by the predictions of a dialogue
state-tracker (LI et al., 2020; Lee et al., 2021).

During training we choose the best models by keeping track of validation
success rate and BLEU score and evaluate the best performing checkpoint on the
test set. As discussed, the training of TCUP is separated into two stages.

(a)  (b)  (c)  (d)  (e)  (f)

Figure 9: Sensitivity analysis for the categorical latent case with 10
30-dimensional categoricals in the latent.