# Interacting with Non-Cooperative User:
A New Paradigm for Proactive Dialogue Policy

Wenqiang Lei∗1, Yao Zhang∗2, Feifan Song3, Hongru Liang1, Jiaxin Mao4
Jiancheng Lv1, Zhenglu Yang2, Tat-Seng Chua5 1College of Computer Science,
Sichuan University, China 2TKLNDST, CS, Nankai University, China3MOE Key Lab
of Computational Linguistics, Peking University, China4GSAI, Renmin University
of China, China 5National University of Singapore, Singapore
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
(2022)

###### Abstract.

Proactive dialogue system is able to lead the conversation to a goal topic and
has advantaged potential in bargain, persuasion and negotiation. Current
corpus-based learning manner limits its practical application in real-world
scenarios. To this end, we contribute to advance the study of the proactive
dialogue policy to a more natural and challenging setting, i.e., interacting
dynamically with users. Further, we call attention to the non-cooperative user
behavior — the user talks about off-path topics when he/she is not satisfied
with the previous topics introduced by the agent. We argue that the targets of
reaching the goal topic quickly and maintaining a high user satisfaction are
not always converge, because the topics close to the goal and the topics user
preferred may not be the same. Towards this issue, we propose a new solution
named I-Pro that can learn Proactive policy in the Interactive setting.
Specifically, we learn the trade-off via a learned goal weight, which consists
of four factors (dialogue turn, goal completion difficulty, user satisfaction
estimation, and cooperative degree). The experimental results demonstrate
I-Pro significantly outperforms baselines in terms of effectiveness and
interpretability.

Proactive Dialogue Policy, Dynamic Interaction, Non-cooperative User Behavior

∗Both authors contributed equally to this research.

Wenqiang Lei is the Corresponding Author.

††copyright: acmcopyright††journalyear: 2022††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; July 11–15, 2022; Madrid,
Spain††booktitle: SIGIR’22: July 11–15, 2022, Madrid, Spain††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Natural
language processing††ccs: Computing methodologies Discourse, dialogue and
pragmatics††ccs: Computing methodologies Interactive simulation

##  1\. Introduction

Figure 1.  Illustration of proactive dialogue: the agent targets at leading
the conversation from the start topic “Andy Lau” to the goal topic “Raging
Fire”, based on (a) a movie-related KG fragment. (b) A successful
conversation, in which the agent leads the conversation to the goal topic.
This scenario is ideal because the user always behaves cooperatively. (c) A
realistic conversation fragment. The non-cooperative user behavior takes the
conversation out of the agent’s control, so the agent needs to reconsider
which topic (in the dotted box) to introduce to the user at the $3^{rd}$ turn.
Since the KG used in this work is in Chinese, both Chinese and its
corresponding English translation are shown here. Best viewed in color.

Proactive dialogue agent aims to lead the conversation with a user from the
start topic (“Andy Lau”) to the goal topic (“Raging Fire”) through chatting
with the user (Wu et al., 2019), as shown in Figure 1. This task has great
potential in scenarios like bargain (He et al., 2018), persuasion (Wang et
al., 2019; Dutt et al., 2021) and negotiation (Lewis et al., 2017; Yang et
al., 2021). Current solutions (Wu et al., 2019; Yuan and An, 2020; Bai et al.,
2021; Wu and Zhou, 2021; Zhu et al., 2021) follow the corpus-based learning
setting — given a knowledge graph (KG), a goal topic and a dialogue context
between two human (e.g., the leader and the follower in the DuConv corpus (Wu
et al., 2019)), the agent is required to predict a topic of the next turn and
generate a response based on this topic. However, turn-level policy might not
align to the conversation-level policy well (Ghandeharioun et al., 2019; Zhang
et al., 2021). Thus, we argue that the corpus-based learning setting is
insufficient to meet the ultimate end that the agent is capable to chat with
the user dynamically. In this work, we take one step further to scrutinize
proactive dialogue policy in the interactive setting. This is a more natural
but challenging setting where the agent is required to optimize long-term goal
during the dynamic interaction.

Intuitively, the most straightforward solution for the policy would be a
shortest path from the start to the goal, as presented in Figure 1 (b).
However, the users may non-cooperatively introduce off-path topics. It usually
happens when they are not satisfied with the current topic (Su et al., 2018).
Take Figure 1 (c), as an example, when the user is unsatisfied with the
previous topics introduced by the agent, he/she talks about off-path topics
(“Eighteen Springs” and “Leon Lai”) in the 2nd turn. Such non-cooperative user
behavior has been seldom studied in prior efforts, but is crucial in the
dynamic interaction. First, it can make the conversation out of the agent’s
control — the agent is unable to introduce the next topic (“Benny Chan”) in
the currently planned shortest path and needs to find a new path to the goal
topic. Second, the low satisfaction may hurt the user’s engagedness and may
even lead the user to terminate the interaction session. This motivates us to
pay attention to the non-cooperative user behavior and carefully manage user
satisfaction during the conversation.

In this work, we aim to learn proactive dialogue policy in the interactive
setting where we call attention to the non-cooperative user behavior. To
advance this, we believe the agent should accomplish two targets — I) reaching
the goal topic quickly to make the conversation short and II) maintaining a
high user satisfaction to keep the user engaged during the conversation (Wu et
al., 2019). However, these targets cannot always converge. For example, in the
$3^{rd}$ turn of Figure 1 (c), “Jacklyn Wu” is closer to the goal but “Almost
A Love Story” may be more preferred by the user. Hence, the agent’s choice
needs to take into account long-term issues — introducing the topic close to
the goal is risky to trigger non-cooperative user behaviors, or, conversely,
introducing the topic user preferred is likely to make the agent take longer
turns to reach the goal. A good proactive agent is expected to achieve the
balance between these targets.

Inspired by the aforementioned analyses, we build I-Pro, a novel proactive
agent that can learn Proactive policy in the Interactive setting.
Specifically, it employs the deep Q-learning algorithm (Mnih et al., 2015) to
train the dialogue policy function, optimizing the reward of faster goal
arrival and higher user satisfaction. Towards the desired trade-off, we design
a learned goal weight. It is derived from dialogue turn, goal completion
difficulty, user satisfaction estimation, and cooperative degree. These four
factors comprehensively decide which topic to be introduced in the next turn.
In the experiment, we recur to user simulators to automatically interact with
the agent because involving real users is laborious. Mimicking the user
behavior, the simulator decides whether to be cooperative based on the
quantitative satisfaction or to choose new topics based on their preference.
We further give the simulator a personality character “tolerance” describing
the variation of non-cooperative user behaviors. A high-tolerance simulator is
less likely to behave non-cooperatively, while a low-tolerance simulator does
the opposite. The experiments demonstrate the effectiveness and
interpretability of the proposed solution over baselines. In addition, we
observe two interesting insights. First, the agent distinctly tends to
prioritize the target “reaching the goal topic quickly” when the number of
dialogue turns is large. Second, users with different tolerance levels require
different dialogue policies. In the face of low-tolerance users, the agent
prefers to improve user satisfaction to keep he/she more engaged.

In summary, the main contributions of this work are as follows:

  * •

We take the first step to scrutinize proactive dialogue policy in the natural
interaction setting, where we call attention to cope with non-cooperative user
behavior.

  * •

We propose a novel model I-Pro that enables a trade-off between the targets of
reaching the goal topic quickly and maintaining a high user satisfaction.

  * •

Our grounded work can serve as a preliminary baseline, and the insightful
analysis provides the potential of benefiting further research.

##  2\. Preliminary

To investigate the proposed paradigm, we formalize a simple and practical
interactive setting. We hope that this setting will serve as a good foundation
for the current exploration and inspire more complex and realistic settings in
the future.

Our setting refers to the scenario proposed by (Wu et al., 2019) where agent
and user engage in a conversation based on topics in KG. Starting from a given
topic, based on the KG, the agent introduces new topics to the user one by
one, and then the user responses to the topic introduced by the agent in the
current turn. The conversation ends when the goal topic is reached.
Specifically, at the current turn, the agent will introduce a new topic
adjacent to the one previously talked about to the user and the user will give
comments on the topic. If the user is satisfied with the current conversation,
he/she will behave cooperatively, giving positive comments on the topic
introduced by the agent in the immediate previous turn, e.g., sounds great.
However, if the user is not satisfied, then he/she will behave non-
cooperatively, proactively mentioning new topics that the user prefers to
increase his/her satisfaction (Su et al., 2018), e.g., I recently watched a
good movie, “Eighteen Springs” starring Leon Lai.

Figure 2.  The paradigm for proactive dialogue policy: we reuse the $2^{nd}$
turn dialogue in Figure 1 (c) as an example. The dialogue is abstracted from
natural language level (a) to topic level (b). At each turn, the agent
introduces one topic to the user. Then the user responds with a list of topics
and their corresponding preferences.

As this paper focuses on the dialogue policy for the goal leading and user
satisfaction, we abstract the whole dialogue process as a topic-level
interaction to simplify the problem, as shown in Figure 2. Specifically, we
assume a third-party natural language understanding techniquecan perfectly
detect the topics and user’s preference score corresponding to this topic
111The details of extracting the topic and preference values are not discussed
in this paper as it is not the focus of this paper.. For example, in the
$1^{st}$ turn of the dialogue in Figure 2, the cooperative comments indeed he
is in response to the topic “Andy Lau” can be extracted as
$\left\\{\right.$(Andy Lau, $p_{1}$)$\left.\right\\}$. In the $2^{nd}$ turn,
the user utterance I recently watched a good movie, “Eighteen Springs”
starring Leon Lai can be extracted as $\left\\{\right.$(Eighteen Springs,
$p_{2}$), (Leon Lai, $p_{3}$)$\left.\right\\}$. $p_{1},p_{2},p_{3}$ are
preference scores corresponds to “Andy Lau”, “Eighteen Springs”, and “Leon
Lai”, respectively. In a similar way, we also assume a natural language
generation component can perfectly generate an utterance given a chosen topic.
For example, the topic “A Moment of Romance” can be generated as He has a
movie called “A Moment of Romance” is very good. Therefore, the conversational
policy only needs to focus on topic choosing. This perfectly isolates the
dialogue policy from natural language understanding and generation, making the
conversational agent is dedicated to policy learning.

Formally, the agent starts the conversation with topic $e_{s}$, and leads the
conversation to its goal topic $e_{g}$. The background KG
$\mathcal{G}=\left\\{(e,r,e^{\prime})|e,e^{\prime}\in\mathcal{E},r\in\mathcal{R}\right\\}$
is defined as a set of triples with a topic (entity) set $\mathcal{E}$ and
relation set $\mathcal{R}$. Given $T$ dialogue turns, the dialogue history is
defined as
$H_{T}=\left\\{(e_{1}^{a},u_{1}),(e_{2}^{a},u_{2}),...,(e_{T}^{a},u_{T})\right\\}$,
where $e_{i}^{a}$ is the topic chosen by the agent and $u_{i}$ refers to the
user utterance at the $i^{th}$ turn. At the $t^{th}$ turn, given the goal
topic $e_{g}$ and the dialogue history $H_{t-1}$, the agent chooses a new
topic $e_{t}^{a}$ from the candidate topic set ${C}_{t}$. Following prior
works (He et al., 2017; Moon et al., 2019; Tuan et al., 2019), to ensure the
coherence of the dialogue, the candidate topic set contains only the topics
within one hop centered on the latest mentioned topic $e_{c}$, i.e.,
${C}_{t}=\left\\{e|(e,r,e_{c})\in\mathcal{G}\right\\}$. The user utterance
$u_{t}=\left\\{(e_{t,1},p_{e_{t,1}}),...,(e_{t,\left|u_{t}\right|},p_{e_{t,\left|u_{t}\right|}})\right\\}$
is the set of topic and preference pairs.

Figure 3. Soft distance estimation method. If the distance between two topics
is less than the pre-set distance limit, we perform accurate distance
estimation (a); otherwise, we perform fuzzy distance estimation (b).

##  3\. Proactive Dialogue Policy

With the proactive dialogue scenario setting, we propose a proactive dialogue
policy I-Pro, which employs a learned goal weight to achieve the desired
trade-off between the targets of reaching the goal topic quickly, and
maintaining a high user satisfaction. At the $t$-th turn, with the goal topic
$e_{g}$ and the dialogue history $H_{t-1}$ as input, I-Pro chooses a topic
$e_{t}^{a}$ from the candidate topic set ${C}_{t}$ as output. During the topic
choosing process, I-Pro mainly tackles three issues: i) which topic is closest
to the goal, ii) which topic the user prefers, and iii) how to make a trade-
off between the two targets. Accordingly, I-Pro consists of a distance
estimation module, a preference estimation module and a goal weight learning
module.

###  3.1. Distance Estimation

In order to lead the conversation to the goal, it is inevitable to clarify
which topic is closest to the goal. The most straightforward method is to
traverse the entire KG in advance to obtain the shortest distance between
topics. However, KGs are usually large-scale and continuously evolving,
leading to high computational costs for global searches. We propose a soft
distance estimation method. Specifically, if the distance between two topics
is less than a pre-set distance limit $D$, we perform accurate distance
estimation; otherwise, we perform fuzzy distance estimation.

We take the distance estimation between topics $e_{i}$ and $e_{j}$ as an
example to detail the soft distance estimation method, as shown in Figure 3.
First, for the topic $e_{i}$ ($e_{j}$), we search with it as the center within
radius $r$, where $r=\tfrac{1}{2}D$, and collect the topics covered during the
search as the set $S_{i}$ ($S_{j}$). Then, if the two sets, $S_{i}$ and
$S_{j}$, intersect, we perform accurate distance estimation: using the minimum
value of the sum of the distances from the overlapping topics to the two
centers. If the two sets are disjoint, we use a maximal value $d_{MAX}$ as the
fuzzy distance estimate. The process can be formulated as:

(1) $ed_{i,j}=\left\\{\begin{matrix}\min_{e_{k}\in S_{i}\cap
S_{j}}(d_{i,k}+d_{j,k}),&S_{i}\cap S_{j}\neq\varnothing\\\ d_{MAX},&S_{i}\cap
S_{j}=\varnothing\end{matrix}\right.$

where $ed_{i,j}$ is the estimated distance between $e_{i}$ and $e_{j}$,
$e_{k}$ belongs to the intersection of two sets $S_{i}$ and $S_{j}$, $d_{i,k}$
and $d_{j,k}$ are accuracy distances from the center topics, $e_{i}$ and
$e_{j}$, to topic $e_{k}$.

###  3.2. Preference Estimation

Which topic the user prefers is an inevitable consideration for agents working
to improve user satisfaction. However, the agent can only obtain partially
accurate preference information from the user through interaction. Therefore,
we propose a preference estimation method to estimate unknown preferences
based on the known ones.

According to the core assumptions in recommender systems (Su and Khoshgoftaar,
2009; Shi et al., 2014; Liang et al., 2016), user preferences
$P\in\mathbb{R}^{\left|\mathcal{E}\right|}$ can be viewed as the product of
topic embeddings $E\in\mathbb{R}^{\left|\mathcal{E}\right|\times d}$ and the
$d$-dimensional user vector $u\in\mathbb{R}^{d}$, i.e., $P=Eu$. We can obtain
the user vector by solving this equation using the known preference
information and its corresponding topics. At the $t$-th turn, we estimate the
user vector by using the normal equation:

Figure 4. Preference estimation method. We first obtain the estimated user
vector $\hat{u}_{t}$ based on the known preference information and its
corresponding topics. Then we use $\hat{u}_{t}$ to estimate the unknown user
preferences and obtain the user preferences $\hat{P}_{t}$.

(2) $\hat{u}_{t}=({E}_{t-1}^{T}{E}_{t-1}+\beta I)^{-1}{E}_{t-1}^{T}{P}_{t-1},$

where $\hat{u}_{t}$ is the estimated user vector 222Note that the estimated
user vector calculated here will be used at the $t$-th turn, so we use
$\hat{u}_{t}$ instead of $\hat{u}_{t-1}$., $E_{t-1}$ and $P_{t-1}$ are the
topic embedding matrix and their corresponding user preference matrix for all
topics that appear in the dialogue history $H_{t-1}$. The topic embeddings are
learned by the node embedding learning model
333https://github.com/thunlp/OpenNE. Considering the computational complexity
for real-time interaction, we aware that Eq. (2) is suitable for the case when
the number of topics is small. When the number of topics is too large , we can
use gradient descent to solve $\hat{u}_{t}$.

Finally, we use the estimated user vector to estimate the unknown user
preferences and obtain the user preferences:

(3) $\hat{P}_{t}=P_{t-1}\oplus(E\setminus E_{t-1})~{}\hat{u}_{t},$

where the whole user preferences set $\hat{P}_{t}$ contains the accurate user
preferences and estimated user preferences.

###  3.3. Goal Weight Learning

This module is proposed to achieve the trade-off between the target I)
reaching the goal topic quickly and target II) maintaining a high user
satisfaction to keep the user engaged during the conversation. At the $t$-th
turn, given the dialogue history $H_{t-1}$ and the goal topic $e_{g}$, the
agent learns a goal weight $gw_{t}$ to denote the importance of the target I
and $1-{gw}_{t}$ to denote the importance of the target II. Then, the weighted
score of each topic in the candidate topic set ${C}_{t}$ is calculated:

(4) $Score(e_{t,i})=gw_{t}\times Rank_{d}(ed_{i,g})+(1-gw_{t})\times
Rank(ep_{t,i}),$

where $e_{t,i}$ is the $i$-th topic in ${C}_{t}$, $ed_{i,g}$ is the estimated
distance between $e_{t,i}$ and the goal topic $e_{g}$,
$ep_{t,i}\in\hat{P}_{t}$ is the estimated user preference of $e_{t,i}$. To
eliminate the difference of magnitude between $ed_{i,g}$ and $ep_{t,i}$, we
transform the raw $ed_{i,g}$ value into its descending ranking and the raw
$ep_{t,i}$ value into its ascending ranking by the rank functions
$Rank_{d}(\cdot)$ and $Rank(\cdot)$, respectively. The topic with the highest
score in ${C}_{t}$ will be chosen as $e_{t}^{a}$.

We abstract four factors from the dialogue history to blend into the goal
weight $gw_{t}$. The specific factors are: i) dialogue turn and ii) goal
completion difficulty, related to the target I; iii) user satisfaction
estimation and iv) cooperative degree, related to the target II. Following are
the details:

  * i)

Dialogue turn $\bm{t}$ We expect the agent to lead the conversation to the
goal as soon as possible. In that case, as the number of dialogue turns
increases, so does the importance of achieving the goal. Thus, we introduce
the dialogue turn $t$ vector in the goal weight.

  * ii)

Goal completion difficulty $\bm{gcd_{t}}$ Another factor related to the target
I is the difficulty of completing it. We approximate this difficulty by
calculating the distance between the current talked topic $e_{c}$ and the goal
topic $e_{g}$, i.e., $gcd_{t}=ed_{c,g}$. A high $gcd_{t}$ indicates that the
current conversation is still far from the goal and that it will take a lot of
effort to lead the conversation to the target. Conversely, the agent can
easily complete the target I.

  * iii)

User satisfaction estimation $\bm{eus_{t}}$ This factor directly portrays the
time-realistic state of the target II. We estimate it by using information
about user preferences that appears in user utterances. Specifically, we
calculate the average preference value of all user utterances in the previous
$t-1$ turns to yield the estimated user satisfaction, i.e.,
$eus_{t}=\textstyle\frac{1}{t-1}{\textstyle\sum_{i=1}^{t-1}\frac{1}{\left|u_{i}\right|}{\textstyle\sum_{j=1}^{\left|u_{i}\right|}ep_{e_{i,j}}}}$,
where $e_{i,j}\in u_{i}$ denotes the $j$-th topic mentioned by user at the
$i$-th turn, and $ep_{e_{i,j}}\in\hat{P}_{t}$ denotes the preference of
$e_{i,j}$.

  * iv)

Cooperative degree $\bm{cd_{t}}$ This factor is learned from the historical
user behaviors. At the $t$-th turn, we generate a one-hot sequence based on
the user behavior in the previous $t-1$ turns, where 0 indicates that the user
cooperate in the current turn, 1 indicates the user did not cooperate. Then we
use GRU(Cho et al., 2014) to encode this one-hot sequence to yield the
cooperative degree $cd_{t}$. When $cd_{t}$ is low, the user is easy to behave
uncooperatively, so the agent should be careful to avoid triggering
uncooperative user behavior.

Finally, we concatenate four factors and adopt a 2-layer policy MLP to learn
the goal weight:

(5) $gw_{t}=MLP(t\oplus gcd_{t}\oplus eus_{t}\oplus cd_{t}).$

###  3.4. Deep Q-Network Learning

We model the process by which an agent chooses a new topic based on a dialogue
policy as a Markov Decision Process (MDP). In details, as a RL problem, at the
$t$-th turn, $\left\langle s_{t},a_{t},p_{t},r_{t}\right\rangle$ in MDP are
defined as: 1) State $s_{t}$ is equal to the dialogue history $H_{t-1}$ with
the goal topic $e_{g}$; 2) Action $a_{t}$ is equal to the candidate topic set
${C}_{t}$; 3) Transition $p_{t}$ is the transition function with
$p(s_{t+1}|s_{t},e_{t}^{a})$ being the probability of seeing state $s_{t+1}$
after taking action $e_{t}^{a}$ based on $s_{t}$; and 4) Reward $r_{t}$ is the
reward that the agent obtains from the environment, including rewards
reflecting changes in goal completion and user satisfaction. Our goal is to
learn a dialogue policy which can maximize the cumulative reward over the
whole conversation process as:

(6)
$\theta^{*}=\arg\max_{\pi\in\prod}\mathbb{E}\left[\sum_{t=1}^{T}r(s_{t},e_{t}^{a})\right],$

where $\theta^{*}$ is the optimal parameter set of I-Pro and
$r_{t}=r_{us}+r_{quit}+r_{goal}+r_{suc}+r_{fail}$. Specifically, $r_{us}$ is
the change of the user satisfaction, i.e., $\alpha(US_{t}-US_{t-1})$, where
$\alpha$ is the magnification factor; $r_{quit}$ is a strongly negative reward
if the user quits the conversation; $r_{goal}$ is the change of the goal
completion degree, i.e., $e^{-\lambda t}(d_{t}-d_{t-1})$, where $\lambda$ is
the time decay coefficient, $d_{t}$ denotes the distance between the current
topic at the $t$-th turn and the goal topic; $r_{suc}$ is a strongly positive
reward if the conversation reach the goal topic; and $r_{fail}$ is a strongly
negative reward if the conversation do not reach the goal topic before the
maximum turn limit $T$.

We adopt Q-learning algorithm to train I-Pro. We use the score calculated by
Eq. 4 as the Q-value $Q_{\theta}(s_{t},e_{t}^{a})$. Based on the optimal
Bellman Equation (Sutton and Barto, 1998), we can represent the optimal
Q-value with the the maximum expected reward achievable is:

(7)
$Q_{\theta}^{\ast}(s_{t},e_{t}^{a})=\mathbb{E}\left[r_{t}+\gamma\operatornamewithlimits{max}_{e_{t+1}^{a}\in
a_{t+1}}Q_{\theta}(s_{t+1},e_{t+1}^{a})\right].$

We improve the value function $Q_{\theta}(s_{t},e_{t}^{a})$ by adjusting
$\theta$ to minimize the mean-square loss function, defined as follows:

(8)
$l(\theta)=\mathbb{E}_{(s_{t},a_{t},s_{t+1},r_{t})\sim{M}}[(Q_{\theta}^{\ast}(s_{t},e_{t}^{a})-Q_{\theta}(s_{t},a_{t}))^{2}].$

##  4\. Experiments

The key contributions of this work are on studying the proactive dialogue
policy interactively and the design of a proactive dialogue policy I-Pro. We
first conduct a comparison experiment to evaluate the effectiveness of I-Pro
interactively. Then we explore the impact of different types of users on the
proactive dialogue policy. In addition, we perform ablation study to
investigate the effect of each factor in the goal weight learning. Finally, we
conduct a case study to demonstrate the superiority of I-Pro. Specifically, we
have the following research questions (RQs) to guide experiments:

  * RQ1: How is the overall performance of I-Pro comparing with existing dialogue policies?

  * RQ2: How do different types of users affect the proactive dialogue policy?

  * RQ3: How do different factors affect the goal weight learning?

Table 1.  GCR(%) and US(%) performance ($\pm$ standard deviation) of compared
dialogue policies (pairwise t-test at 5% significance level). All dialogue
policies are evaluated on three types of user simulators with different
tolerance values ($k=0.8$, $1.0$ and $1.2$) as well as on a mixed-user
simulator.

| Mixed User Simulator |  User Simulator ($k=0.8$) |  User Simulator ($k=1.0$) |  User Simulator ($k=1.2$)
---|---|---|---|---
| GCR | US | GCR | US | GCR | US | GCR | US
Random |  1.29 $\pm$0.01 |  52.86 $\pm$0.01 |  1.67 $\pm$0.01 |  55.29 $\pm$0.01 |  1.18 $\pm$0.01 |  52.65 $\pm$0.01 |  1.02 $\pm$0.01 |  50.65 $\pm$0.01
Pop (GCR) |  57.00 $\pm$0.03 |  53.52 $\pm$0.00 |  48.18 $\pm$0.03 |  56.05 $\pm$0.00 |  59.18 $\pm$0.03 |  53.24 $\pm$0.00 |  63.63 $\pm$0.02 |  51.27 $\pm$0.00
Pop (US) |  1.16 $\pm$0.01 |  62.14 $\pm$0.00 |  1.39 $\pm$0.01 |  64.47 $\pm$0.00 |  1.07 $\pm$0.01 |  61.80 $\pm$0.00 |  1.03 $\pm$0.00 |  60.15 $\pm$0.00
NICF |  3.46 $\pm$0.01 |  54.48 $\pm$0.00 |  3.55 $\pm$0.01 |  57.76 $\pm$0.00 |  3.30 $\pm$0.01 |  54.10 $\pm$0.00 |  3.52 $\pm$0.01 |  51.58 $\pm$0.01
DeepPath |  3.84 $\pm$0.01 |  54.41 $\pm$0.00 |  4.32 $\pm$0.01 |  57.94 $\pm$0.00 |  3.81 $\pm$0.01 |  54.06 $\pm$0.00 |  3.38 $\pm$0.01 |  51.24 $\pm$0.00
NKD |  3.15 $\pm$0.01 |  52.96 $\pm$0.00 |  4.19 $\pm$0.02 |  56.45 $\pm$0.01 |  2.97 $\pm$0.01 |  52.31 $\pm$0.00 |  2.28 $\pm$0.01 |  50.13 $\pm$0.00
I-Pro |  29.74 $\pm$0.07 |  59.28 $\pm$0.01 |  22.47 $\pm$0.06 |  61.91 $\pm$0.01 |  32.20 $\pm$0.07 |  58.64 $\pm$0.01 |  34.54 $\pm$0.08 |  57.28 $\pm$0.01

###  4.1. User Simulation

Considering the huge costs associated with involving humans in the
interaction, we design user simulators to complete the process of interacting
with the agents when conducting the experiments. The design of the simulator
is also at the topic-level. When interacting with the agent, at the $t$-th
turn, the simulator accepts the dialogue history and the topic introduced by
the agent at the current turn, i.e., $H_{t-1}\cup\left\\{e_{t}^{a}\right\\}$.
Then, it responds $u_{t}$ to the agent or quits the conversation.

Here, we describe in detail how to generate $u_{t}$. The user utterance is
based on satisfaction with the current conversation. We argue that
satisfaction can be formalized as the cumulative average of users’ preferences
for the topics covered by the conversation:

(9)
$US_{t}\triangleq{\textstyle\frac{1}{t}\sum_{i=1}^{t}{\textstyle\frac{1}{\left|u_{i}+1\right|}(\sum_{j=1}^{\left|u_{i}\right|}p_{e_{i,j}}+p_{e_{i}^{a}})}},$

where $US_{t}$ denotes the user satisfaction at the $t$-th turn, $e_{i,j}\in
u_{i}$ denotes the $j$-th topic mentioned by user at the $i$-th turn and
$p_{e_{i,j}}$ denotes the user preference to the topic $e_{i,j}$.
${\left|u_{i}\right|}$ is the total number of topic mentioned in the $t$ turn
utterance (without removing duplicates). Note that when $e_{i,j}=e_{i}^{a}$,
there is no need to double count the latter $e_{i}^{a}$. User preferences are
personalized. For one user simulator, we first sample a user vector
$u\in\mathbb{R}^{d}$. Each element in the user vector is sampled from the
Gaussian distribution $N(0,2)$. Next, following the core assumptions in
recommendation techniques (Su and Khoshgoftaar, 2009; Shi et al., 2014; Liang
et al., 2016), we map the user vector to the topic embeddings
$E\in\mathbb{R}^{\left|\mathcal{E}\right|\times d}$ to calculate user
preferences, i.e., $P=Eu$, where $P=\left\\{p_{e}|e\in\mathcal{E}\right\\}$.

At the $t$-th turn, based on the calculated user satisfaction, the user
utterance can be generated in cases:

(10) $u_{t}=\begin{cases}\left\\{(e_{t}^{a},p_{e_{t}^{a}})\right\\}&\text{ if
}US_{t}>Q_{c},\\\ \left\\{(e,p_{e})\mid e\in\mathcal{E}\wedge
Sel(e)=1\right\\}&\text{ if }Q_{q}<US_{t}\leq Q_{c},\\\ \varnothing&\text{ if
}US_{t}\leq Q_{q},\end{cases}$

specifically, the user behavior can be deconstructed into three types:
cooperative, non-cooperative and quit. If $US_{t}$ is above the cooperative
threshold $Q_{c}$, the simulator will only respond the topic introduced by
agent at this turn and the corresponding preference. Whereas if $US_{t}$ is
below the cooperative threshold $Q_{c}$, the simulator will respond new topics
with their preferences. The new topic in $u_{t}$ is chosen with its user
preference as probability. We design the function $Sel(e)$ to calculate
whether the topic $e$ is chosen or not:

(11) $Sel(e)\sim Bernoulli(p_{e}),$

where the $Sel(e)$ is sampled from the Bernoulli distribution: 1 indicates the
topic is chosen, while 0 indicates the opposite. If $US_{t}$ fall below an
even lower quit threshold $Q_{q}$, the simulator will quit the conversation in
advance.

Finally, to further imitate humans, we personalize the thresholds $Q_{c}$ and
$Q_{q}$ as well. We design a new characteristic “tolerance”. A high tolerance
simulator is less likely to behave non-cooperatively or to quit a conversation
outright, while a low tolerance simulator does the opposite. Concretely, the
tolerance character $k$ is used as a shrinker on the thresholds. That is, we
use $\tfrac{1}{k}Q_{c}^{*}$ and $\tfrac{1}{k}Q_{q}^{*}$ as the personalized
cooperative and quit thresholds, where $Q_{c}^{*}$ and $Q_{q}^{*}$ are pre-set
constants.

In the evaluation process, for one start-goal topic pair, we design three user
simulators with the same user vector and different tolerance characteristics,
low ($k=0.8$), medium ($k=1.0$) and high ($k=1.2$), respectively. We choose
0.5 as $Q_{c}^{*}$ and 0.4 as $Q_{q}^{*}$, experimentally. Considering the
computational consumption caused by large-scale KG, we set the topic selection
scope of the simulator to all neighbors within 3 hops of the current topic,
while $\left|u_{t}\right|$ is limited to maximum 3.

###  4.2. Settings

####  4.2.1. Dataset

The dataset we used is collected from KdConv (Zhou et al., 2020), which
contains a film-related KG and a large amount of multi-turn conversation data.
First, we continue to use the KG of KdConv, which has 7477 entities, 4939
relations and 89618 triples. Then, because this work focuses on studying the
dialogue policy in an interactive manner, we abandon the static conversation
data, and extract only the start-goal topic pairs from the multi-turn
conversation data. Specifically, the start topic is extracted from the
$1^{st}$ turn and the goal topic is extracted from the last turn of the
conversation. Statistically, the KG has 7477 film-related topics. The training
and testing sets contain 353 and 89 start-goal topic pairs, respectively.

####  4.2.2. Evaluation Metrics

With the targets I and II, we use two metrics to automatically evaluate the
all-turn performance of the dialogue policy:

  * Goal Completion Rate (GCR) We use GCR to evaluate the ability of the policy to lead the conversation successfully. GCR is defined as:

(12) $\mathrm{GCR}=\textstyle\frac{1}{N}\textstyle\sum_{k=1}^{N}e^{-\lambda
T_{k}}D_{k}$

where $N$ is the total number of dialogues, $T_{k}$ is the total number of
turns of the $k$-th dialogue, $\lambda$ is the time decay coefficient. $D_{k}$
is set to 1 if the $k$-th dialogue succeeded in achieving the agent’s goal,
otherwise it is 0.

  * User Satisfaction (US) We use US to evaluate the ability of the policy to maintain the high user satisfaction. We use the average preference of all the topics involved in the whole conversation to represent US:

(13)
$\mathrm{US}=\textstyle\frac{1}{N}\sum_{k=1}^{N}{\textstyle\frac{1}{T_{k}}\sum_{i=1}^{T_{k}}\frac{1}{\left|u_{i}\right|+1}({\textstyle\sum_{j=1}^{\left|u_{k,i}\right|}p_{e_{k,i,j}}}+p_{e_{k,i}^{a}}}),$

where $p_{e_{k,i,j}}\in u_{k,i}$ is the preference of the $j$-th topic
mentioned by user at the $i$-th turn in the $k$-th dialogue, and
$p_{e_{k,i}^{a}}$ is the preference of the topic introduced by the agent at
the $i$-th turn in the $k$-th dialogue. ${T_{k}}$ is the total number of
dialogue turns for the $k$-th conversation. Note that when
$e_{k,i,j}=e_{k,i}^{a}$, there is no need to double count the latter
$e_{k,i}^{a}$.

####  4.2.3. Training Details

The dimensions of the user vector and topic embedding are all set to $50$. The
node embedding learning model is DeepWalk (Perozzi et al., 2014). The maximum
dialogue turn $T$ is set to $20$. The distance limit $D$ used in the distance
estimation is set to 6. The optimizer chosen is the Adam and we empirically
set the learning rate as $1e{-7}$. The maximum training epoch is set to 100.
The RL algorithm used here is DQN(Mnih et al., 2015), in which the memory
size, batch size, decay factor $\gamma$, exploration factor $\varepsilon$ for
the $\varepsilon$-greedy policy, time decay coefficient $\lambda$, and the
magnification factor $\alpha$ are $2000$, $100$, $0.9$, $0.2$, $0.02$ and
$100$, respectively. Rewards $r_{quit},r_{suc},r_{fail}$ are $-10$, $20$ and
$-10$ if triggered, respectively; otherwise they are all $0$.

For an in-depth and extensive evaluation, we test all comparison dialogue
policies for 100 rounds. Due to a certain randomness of user behavior, there
is a slight variation in the results of 100 rounds. We expect to select the
best result for reporting. However, under the multi-metric evaluation setting,
the two metrics we use (GCR and US) are equally important, so the best results
are often a set. In this set, no one result can outperform the other results
on both metrics at the same time. Therefore, we report the average of all
results in the optimal results set as the final performance for each policy.

####  4.2.4. Baselines

This work is the first attempt to study a proactive dialogue policy which has
the ability to quickly achieve the goal topic as well as maintain high user
satisfaction. We first present three baselines to perceive the difficulty of
this task:

  * Random At each turn, the agent randomly selects the topic.

  * Pop(GCR) At each turn, the agent chooses the topic closest to the goal, i.e., with the smallest estimated distance, in order to get the highest GCR.

  * Pop(US) At each turn, the agent chooses the user’s favorite topic, i.e., with the highest estimated preference, in order to get the highest US.

Figure 5. Comparison performance of I-Pro and the Variant I-Pro-degrade. The
optimal results for I-Pro and I-Pro-degrade are depicted. Apparently, the
optimal results of I-Pro completely surround the optimal results of I-Pro-
degrade.  Figure 6. Performance w.r.t. different user simulator types.
Performance on three user simulator types with different tolerance: low
($k=0.8$), medium ($k=1.0$) and high ($k=1.2$) are visualized. GCR increases
and US decreases as the tolerance level increases.

Considering that there are currently no dialogue policies that address both
targets, we chose several policies that are committed to one target alone as
the baselines.

  * NICF (Zou et al., 2020) This model is originally designed to recommend items for user in an interactive setting. In our proactive dialogue scenario, it can be used to enhance US during multi-turn interactions.

  * DeepPath (Xiong et al., 2017) This model is focus on searching a path in KG through multi-hop reasoning from the start entity to the goal entity. Here, it can work on leading the conversation to the goal topic.

  * NKD (Liu et al., 2018a) This model works on multi-turn dialogue generation based on the dialogue history and KG. We degrade it to the topic level dialogue generation.

All the above baselines and our I-Pro are trained and tested in an interactive
manner with our designed user simulators.

###  4.3. Overall Performance (RQ1)

I-Pro vs. Baselines Table 1 demonstrates the performance of I-Pro and
baselines. As can be clearly seen, I-Pro significantly outperforms existing
dialogue policies both on GCR and US. We also make the following observations:
1) All policies show a decrease in the US metric as tolerance increases,
because users with low tolerance are more likely to be triggered to quit the
conversation at a higher satisfaction level. 2) The Random baseline shows a
decreasing trend under the GCR metric as user tolerance increases. When the
user tolerance is low, there is more non-cooperative behavior from the user.
And the perturbation caused by the user’s non-cooperative behavior on the
conversation forward direction makes the conversation have a chance to reach
the goal topic. 3) Two “Pop” baselines of our own design show the upper bounds
of the effects currently achievable on both GCR and US metrics. These two
upper bounds can be further improved by improving the estimation accuracy of
user preferences and the distance. Interestingly, under the GCR metric, the
Pop (GCR) tends to increase as tolerance increases, while the Pop (US) does
the opposite. We speculate that this is because the Pop (GCR) aims to improve
GCR, so the lower the user tolerance, the more likely it is to trigger non-
cooperative behavior. While, Pop (US), same as Random, has a higher
probability of the agent walking to the goal by chance at low user tolerance.
4) Three existing works, NICF (Zou et al., 2020), DeepPath (Xiong et al.,
2017) and NKD (Liu et al., 2018a), all underperform, exposing the challenges
in proactive dialogue policy study. We suspect that their drawbacks are the
inaccurate estimation of user preferences and distance to the goal, as well as
a lack of ability to make the trade-off between target I and II.

Table 2. Performance (GCR and US) and the learned goal weight value ($\pm$
standard deviation) at different tolerance values.

k | 0.4 | 0.6 | 0.8 | 1.0 | 1.2 | 1.4 | 1.6
---|---|---|---|---|---|---|---
GCR | 15.84 | 17.04 | 22.47 | 32.20 | 34.54 | 34.1 | 34.75
US | 62.37 | 62.71 | 61.91 | 58.64 | 57.28 | 56.24 | 55.99
$\overline{gw}$ | 0.37 | 0.41 | 0.45 | 0.49 | 0.50 | 0.51 | 0.51
$\pm$ sd | 0.17 | 0.17 | 0.17 | 0.13 | 0.11 | 0.10 | 0.09

I-Pro vs. I-Pro-degrade We propose one variant of I-Pro, namely I-Pro-degrade,
to further explore the utility of our proposed goal weight. In this variant,
the goal weight is degraded to one trainable parameter, i.e.,
$Score(e_{t,i})=\beta\times Rank_{d}(ed_{i,g})$ $+(1-\beta)\times
Rank(ep_{t,i})$. We intuitively present the performance comparison in Figure
5. Clearly, the optimal results of I-Pro completely surround the optimal
results of I-Pro-degrade. This exhibits the shallow capability of one
parameter to make the trade-off between two targets. We also observe that the
effect of I-Pro-degrade has many repeated values, which shows the inflexible
ability of only using one parameter. Apparently, it is intractable for one
parameter to perceive the progress of the conversation and the personality of
the user. This suggests that our integration of multiple factors on the goal
weight learning is necessary.

###  4.4. Performance w.r.t. Different User Types (RQ2)

To investigate the differences in the effects of I-Pro on different types of
users, we evaluate I-Pro on three user simulator types with different
tolerance: low ($k=0.8$), medium ($k=1.0$) and high ($k=1.2$), and visualize
the results in Figure 6. Obviously, the difference in tolerance leads to a
significant difference in performance: GCR increases and US decreases as the
tolerance level increases. Users with low tolerance are more likely to behave
non-cooperatively when their satisfaction is low, making it difficult for the
agent to achieve the goal topic. In addition, agents chooses users’ preferred
topics in order to try to avoid non-cooperative user behaviors. Therefore, low
tolerance users will lead to a low GCR and a high US. High tolerance users are
just the opposite.

We further explore more diverse tolerance values and the correlation between
the tolerance values and the learned goal weight. Here, for the goal weights,
we use their averages over all dialogue turns. Results are reported in Table
2. We observe that the goal weight is perceptible to changes in tolerance and
there is a positive correlation between the two terms. When the goal weight
perceives that the user has a high tolerance, it will spontaneously be more
biased towards improving GCR. We also observe that the standard deviation of
the goal weight is negatively correlated with the tolerance. This is because
the less tolerance the user has, the more drastic the change in user behavior
(cooperative or non-cooperative) will be. The goal weight reacts to the
perceived change as a larger standard deviation.

Figure 7. Ablation study I: performance on deactivating “approach the goal”
related factors and “satisfy the user” related factors, respectively.

###  4.5. Ablation Study on Goal Weight (RQ3)

We abstract four factors from the dialogue history to learn the goal weight.
Here, we first analyze the effectiveness of different factors in goal weight
learning. As shown in Figure 7, we evaluate the performance on our two
variants: 1) the goal weight learning without two target I-related factors:
dialogue turn factor and goal completion degree factor, and 2) the goal weight
learning without two target II-related factors: user satisfaction estimation
factor and cooperative degree factor. We observe that, when deactivating the
target I-related factors, the GCR score decreases significantly and the US
score increases slightly. This confirms that when the agent is unable to
perceive the progress of the conversation, its control over the conversation
decreases, and the conversation is easily led to the user’s preferred topic by
the user’s non-cooperative behaviors. This eventually manifests as an increase
in user satisfaction, but the conversation fails to achieve the goal topic.
When deactivating the target II-related factors, both the US and GCR scores
decrease. This suggests that the conduct of proactive conversations is
sensitive to user satisfaction and behavior. So, it is critical to estimate
the satisfaction of current users as well as to predict the cooperative degree
of users. When deactivating the user satisfaction estimation factor, the agent
cannot perceive user dissatisfaction in time, leading to a decrease in US. In
addition, when the agent cannot perceive the cooperative degree of users, more
non-cooperative user behaviors are triggered, leading to a decrease in GCR.

Figure 8. Ablation study II: correlations between different factors and the
goal weight. The horizontal axis indicates the individual factors. The
vertical axis indicates the corresponding goal weight values under the
particular factor.

We secondly analyze the specific correlation between each factor and the goal
weight. The correlations are pulled from the dialogue data collected during
the evaluation, which are visualized in Figure 8. We make the following
observations:

Figure 9. Case study of four dialogue policies: Pop (GCR), Pop (US), I-Pro-
degrade and I-Pro. We choose the same start and goal topics, and record the
interaction process between each dialogue policy and the user simulator whose
tolerance is 1. We report partial transcripts of the conversations, including
turns 1, 2, 14, and 15. Specifically, at each turn, we report the conversation
detail, the current user satisfaction ($US_{t}$) and the distance to goal
($d_{t}$). The finally results also are reported. Specially, we also report
the goal weight values of I-Pro-degrade and I-Pro, i.e., $\beta$ and $gw_{t}$.
Due to the space limitation, we show here only the English translation of the
cases.

  * i)

The dialogue turn factor correlates positively with the goal weight. The agent
aims to achieve the goal topic as soon as possible, thus, the goal weight
increases with the number of dialogue turns. Interestingly, we observe that
the weight values are particularly low at the beginning of the conversation,
which means that the agent is more willing to satisfy the user at the early
stage.

  * ii)

The goal completion difficulty factor correlates negatively with the goal
weight. Note that larger factor value indicates greater distance from the
goal, i.e. higher goal completion difficulty. This is reasonable because when
the agent perceives that the current conversation is close to the goal topic,
the potential gain obtained by choosing a topic close to the goal is higher
than that preferred by the user.

  * iii)

The user satisfaction estimation factor correlates positively with the goal
weight. This factor directly reflects user satisfaction, so when the agent
estimates that user satisfaction is low, it will be more inclined to choose
topics that users prefer.

  * iv)

The cooperative degree factor correlates positively with the goal weight. This
factor models the tolerance character of users implicitly. Higher factor
values indicate less risk of triggering non-cooperative user behavior when
agents lead the conversation. Thus, when this factor is high, the agent
intelligently increases the goal weight to accelerate reaching the goal topic.
Conversely, the agent decreases the goal weight to avoid triggering non-
cooperative user behavior.

###  4.6. Case Study

In this section, a case study was conducted to better demonstrate the
superiority between our proactive dialogue policy and other compared dialogue
policies. Figure 9 shows four dialogue fragments of I-Pro and three baselines:
Pop (GCR), Pop (US) and I-Pro-degrade, interacting with a user simulator whose
tolerance is 1.0, as well as the final results containing the GCR and US
scores. We obtain the following observations:

  * i)

Pop (GCR) baseline works to improve the GCR score solely, so it chooses the
topic closest to the goal in each turn. However, due to the agent ignores the
user satisfaction, the user behaves non-cooperatively in almost every turn.
ach non-cooperative behavior again takes the conversation off-goal again,
resulting in the agent’s efforts being wasted.

  * ii)

Pop (US) baseline works to improve the US score and therefore introduces the
user’s favorite topics in each turn. High user satisfaction is maintained
during the conversation, resulting in few non-cooperative user behaviors. Due
to the neglect of the target I, the GCR score is low at the end of the
conversation, despite maintaining a high user satisfaction.

  * iii)

In I-Pro-degrade, the goal weight is a fixed value (after training), i.e.,
$\beta=0.54$. The $\beta$ value greater than 0.5 means that the agent will
always be partial to the goal achievement in the conversation. As we can see,
in the first turn, when the user satisfaction is significantly low, the agent
still chooses one topic that is close to the goal rather than preferred by the
user. However, this decision triggers the non-cooperative user behavior in the
next turn. Obviously, it is difficult to intelligently balance user
satisfaction and goal completion using only one parameter.

  * iv)

I-Pro is significantly more flexible and strategic than the three baselines.
It is able to choose to favor US when user satisfaction is low (e.g.,
$gw_{2}=0.07$), while favoring GCR at a high US and later stages of the
conversation (e.g., $gw_{15}=0.55$).

##  5\. Related Work

Conventional conversational systems are dedicated to assisting users to
accomplish goals or achieve users’ satisfactions. For example, task-oriented
systems, always as virtual personal assistants, are designed to help users
with daily tasks, such as booking accommodation and restaurant (Yan et al.,
2017; Liu et al., 2018b; Këpuska and Bohouta, 2018) conversational
recommendation systems (Kang et al., 2019; Lei et al., 2020) can recommend
products to users that they want; and human-like chatbots can interact with
users to provide reasonable responses for entertainment (Bordes et al., 2017;
Adiwardana et al., 2020).

In recent years, however, there has been growing interest in exploring on
another feature of dialogue systems: proactivity. Proactive dialogue systems
aims to lead the conversation to their own goals, which may not perfectly
align well with the user’s goal or satisfaction. Proactive dialogue systems
have great potential in various scenarios. For example, persuasion dialogue
system attempts to convince the user to change his/her attitude, opinion, or
behavior. Wang et al. (2019) collect a large dialogue dataset in which a
participant persuaded another participant to make a donation. Tian et al.
(2020) build a classifier to predict users’ policies for resisting donations.
Negotiation dialogue system interacts with a user strategically to reach an
agreement. He et al. (2018) propose a modular approach based on coarse
dialogue acts that decouples policy and generation to make the policy
controllable. Zhou et al. (2019) model both semantic and policy history to
improve both dialogue policy planning and generation performance. However, the
proactive policy studied in the aforementioned works lack explainability hence
helping less on revealing the rationale of proactive dialogues.

To gain better explainability and empower strategic reasoning for proactive
dialogue, Wu et al. (2019) introduce a knowledge graph and formalize the
proactive policy as a path reasoning problem over the KG. They contribute a
corpus through crowdsourcing where each dialogue history is composed by two
workers, one worker acting as a conversation leader while the other acting as
the follower. Under this setting and the corpus, Yuan and An (2020) propose an
End-to-End dialogue model based on Memory network to study natural language
response generation; Bai et al. (2021) focus on the issue of knowledge
coherence in proactive dialogues; Zhu et al. (2021) use a retrieval-based
approach for knowledge prediction in proactive dialogues. However, these
efforts all follow a corpus-based learning manner. Fundamentally different
from existing efforts, we take one step further to scrutinize proactive
dialogue policy with interactive learning in this paper.

##  6\. Conclusion

In this work, we study the proactive dialogue policy in an interactive manner
and call attention to the non-cooperative user behavior during the
conversation. We argue that the interactive proactive dialogue policy learning
has two targets: leading the conversation to the goal quickly and maintaining
a high user satisfaction. To advance the two targets, we propose I-Pro which
employs a learned goal weight to achieve a trade-off between them. We design
user simulators to interact with the agents during training and evaluation.
The experimental results demonstrate that I-Pro opens the performance gap for
interactive proactive dialogue policy learning.

Our work takes the first step to advance the interactive proactive dialogue
policy learning, and can serve as a preliminary baseline to benefit further
research. Naturally, there are thus a few loose ends for further
investigation, especially with respect to more diverse user behavior and
richer user personalities. We will also try to enhance the goal weight by
considering more influencing factors. Lastly, we will deploy I-Pro to online
applications that interact with real users to gain more insights for further
improvements.

## Acknowledgments

This work was supported in part by the scholarship from China Scholarship
Council (CSC) under Grant No.202006200134; in part by the China Postdoctoral
Science Foundation under Grant No.2021TQ0222 and No.2021M700094; in part by
the Fundamental Research Funds for the Central Universities of China; and in
part by the Beijing Academy of Artificial Intelligence (BAAI).

## References

  * (1)
  * Adiwardana et al. (2020) Daniel Adiwardana, Minh-Thang Luong, David R So, Jamie Hall, Noah Fiedel, Romal Thoppilan, Zi Yang, Apoorv Kulshreshtha, Gaurav Nemade, Yifeng Lu, and Quoc V. Le. 2020\.  Towards a Human-like Open-Domain Chatbot.  _CoRR_ abs/2001.09977 (2020).
  * Bai et al. (2021) Jiaqi Bai, Ze Yang, Xinnian Liang, Wei Wang, and Zhoujun Li. 2021.  Learning to Copy Coherent Knowledge for Response Generation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ _(AAAI ’21, 14)_. 12535–12543.
  * Bordes et al. (2017) Antoine Bordes, Y-Lan Boureau, and Jason Weston. 2017\.  Learning End-to-End Goal-Oriented Dialog. In _Proceedings of the 5th International Conference on Learning Representations_ _(ICLR ’17)_.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014.  On the Properties of Neural Machine Translation: Encoder–Decoder Approaches. In _Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation_. 103–111.
  * Dutt et al. (2021) Ritam Dutt, Sayan Sinha, Rishabh Joshi, Surya Shekhar Chakraborty, Meredith Riggs, Xinru Yan, Haogang Bao, and Carolyn Rose. 2021\.  ResPer: Computationally Modelling Resisting Strategies in Persuasive Conversations. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics_ _(EACL ’21)_. 78–90.
  * Ghandeharioun et al. (2019) Asma Ghandeharioun, Judy Hanwen Shen, Natasha Jaques, Craig Ferguson, Noah Jones, Agata Lapedriza, and Rosalind Picard. 2019\.  Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems. In _Advances in Neural Information Processing Systems_ _(NeurIPS ’19)_. Curran Associates, Inc.  https://proceedings.neurips.cc/paper/2019/file/fc9812127bf09c7bd29ad6723c683fb5-Paper.pdf
  * He et al. (2017) He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017.  Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics_ _(ACL ’17)_. 1766–1776.
  * He et al. (2018) He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018\.  Decoupling Strategy and Generation in Negotiation Dialogues. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ _(EMNLP ’18)_. 2333–2343.
  * Kang et al. (2019) Dongyeop Kang, Anusha Balakrishnan, Pararth Shah, Paul A Crook, Y-Lan Boureau, and Jason Weston. 2019.  Recommendation as a Communication Game: Self-Supervised Bot-Play for Goal-oriented Dialogue. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_ _(EMNLP-IJCNLP ’19)_. 1951–1961.
  * Këpuska and Bohouta (2018) Veton Këpuska and Gamal Bohouta. 2018.  Next-generation of virtual personal assistants (microsoft cortana, apple siri, amazon alexa and google home). In _2018 IEEE 8th annual computing and communication workshop and conference (CCWC)_. IEEE, 99–103.
  * Lei et al. (2020) Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, and Tat-Seng Chua. 2020.  Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems. In _Proceedings of the 13th International Conference on Web Search and Data Mining_ _(WSDM ’20)_. 304–312.
  * Lewis et al. (2017) Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017.  Deal or No Deal? End-to-End Learning of Negotiation Dialogues. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ _(EMNLP ’17)_. 2443–2453.
  * Liang et al. (2016) Dawen Liang, Jaan Altosaar, Laurent Charlin, and David M. Blei. 2016.  Factorization Meets the Item Embedding: Regularizing Matrix Factorization with Item Co-Occurrence. In _Proceedings of the 10th ACM Conference on Recommender Systems_ _(RecSys ’16)_. 59––66.
  * Liu et al. (2018b) Bing Liu, Gokhan Tur, Dilek Hakkani-Tur, Pararth Shah, and Larry Heck. 2018b.  Dialogue Learning with Human Teaching and Feedback in End-to-End Trainable Task-Oriented Dialogue Systems. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ _(NAACL ’18)_. 2060–2069.
  * Liu et al. (2018a) Shuman Liu, Hongshen Chen, Zhaochun Ren, Yang Feng, Qun Liu, and Dawei Yin. 2018a.  Knowledge diffusion for neural dialogue generation. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_ _(ACL ’18)_. 1489–1498.
  * Mnih et al. (2015) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. 2015\.  Human-level control through deep reinforcement learning.  _Nature_ 518, 7540 (2015), 529–533.
  * Moon et al. (2019) Seungwhan Moon, Pararth Shah, Anuj Kumar, and Rajen Subba. 2019\.  Opendialkg: Explainable conversational reasoning with attention-based walks over knowledge graphs. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ _(ACL ’19)_. 845–854.
  * Perozzi et al. (2014) Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. 2014\.  Deepwalk: Online learning of social representations. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_ _(KDD ’14)_. 701–710.
  * Shi et al. (2014) Yue Shi, Martha Larson, and Alan Hanjalic. 2014.  Collaborative Filtering beyond the User-Item Matrix: A Survey of the State of the Art and Future Challenges.  _ACM Comput. Surv._ 47, 1 (2014), 45 pages.
  * Su et al. (2018) Ning Su, Jiyin He, Yiqun Liu, Min Zhang, and Shaoping Ma. 2018.  User intent, behaviour, and perceived satisfaction in product search. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_ _(WSDM ’18)_. 547–555.
  * Su and Khoshgoftaar (2009) Xiaoyuan Su and Taghi M Khoshgoftaar. 2009.  A survey of collaborative filtering techniques.  _Advances in artificial intelligence_ 2009 (2009).
  * Sutton and Barto (1998) R Sutton and A Barto. 1998\.  _Reinforcement Learning:An Introduction_.  MIT Press.
  * Tian et al. (2020) Youzhi Tian, Weiyan Shi, Chen Li, and Zhou Yu. 2020\.  Understanding User Resistance Strategies in Persuasive Conversations. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ _(EMNLP ’20)_. 4794–4798.
  * Tuan et al. (2019) Yi-Lin Tuan, Yun-Nung Chen, and Hung-yi Lee. 2019.  DyKgChat: Benchmarking Dialogue Generation Grounding on Dynamic Knowledge Graphs. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing_ _(EMNLP-IJCNLP ’19)_. 1855–1865.
  * Wang et al. (2019) Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019.  Persuasion for Good: Towards a Personalized Persuasive Dialogue System for Social Good. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ _(ACL ’19)_. 5635–5649.
  * Wu and Zhou (2021) Junjie Wu and Hao Zhou. 2021.  Augmenting Topic Aware Knowledge-Grounded Conversations with Dynamic Built Knowledge Graphs. In _Proceedings of Deep Learning Inside Out (DeeLIO): The 2nd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_. 31–39.
  * Wu et al. (2019) Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019.  Proactive Human-Machine Conversation with Explicit Conversation Goal. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ _(ACL ’19)_. 3794–3804.
  * Xiong et al. (2017) Wenhan Xiong, Thien Hoang, and William Yang Wang. 2017\.  DeepPath: A Reinforcement Learning Method for Knowledge Graph Reasoning. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ _(EMNLP ’17)_. 564–573.
  * Yan et al. (2017) Zhao Yan, Nan Duan, Peng Chen, Ming Zhou, Jianshe Zhou, and Zhoujun Li. 2017\.  Building Task-Oriented Dialogue Systems for Online Shopping. In _Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence_ _(AAAI’17)_. AAAI Press, 4618–4625.
  * Yang et al. (2021) Runzhe Yang, Jingxiao Chen, and Karthik Narasimhan. 2021\.  Improving Dialog Systems for Negotiation with Personality Modeling. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_ _(ACL-IJCNLP ’21)_. 681–693.
  * Yuan and An (2020) Hao Yuan and Jinqi An. 2020\.  Multi-Hop Memory Network with Graph Neural Networks Encoding for Proactive Dialogue. In _Proceedings of the 2020 6th International Conference on Computing and Artificial Intelligence_ _(ICCAI ’20)_. 24–29.
  * Zhang et al. (2021) Chen Zhang, Yiming Chen, Luis Fernando D’Haro, Yan Zhang, Thomas Friedrichs, Grandee Lee, and Haizhou Li. 2021.  DynaEval: Unifying Turn and Dialogue Level Evaluation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_ _(ACL-IJCNLP ’21)_. Association for Computational Linguistics, 5676–5689.
  * Zhou et al. (2020) Hao Zhou, Chujie Zheng, Kaili Huang, Minlie Huang, and Xiaoyan Zhu. 2020.  KdConv: A Chinese Multi-domain Dialogue Dataset Towards Multi-turn Knowledge-driven Conversation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ _(ACL ’20)_. 7098–7108.
  * Zhou et al. (2019) Yiheng Zhou, Yulia Tsvetkov, Alan W Black, and Zhou Yu. 2019\.  Augmenting Non-Collaborative Dialog Systems with Explicit Semantic and Strategic Dialog History. In _Proceedings of the 7th International Conference on Learning Representations_ _(ICLR ’19)_.
  * Zhu et al. (2021) Yutao Zhu, Jian-Yun Nie, Kun Zhou, Pan Du, Hao Jiang, and Zhicheng Dou. 2021\.  Proactive Retrieval-Based Chatbots Based on Relevant Knowledge and Goals. In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’21)_. 2000–2004.
  * Zou et al. (2020) Lixin Zou, Long Xia, Yulong Gu, Xiangyu Zhao, Weidong Liu, Jimmy Xiangji Huang, and Dawei Yin. 2020.  Neural Interactive Collaborative Filtering. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ _(SIGIR ’20)_. 749–758.