# Towards a Neural Era in Dialogue Management for Collaboration: A Literature
Survey

Amogh Mannekote
University of Florida
<EMAIL_ADDRESS>
###### Abstract

Dialogue-based human-AI collaboration can revolutionize collaborative problem-
solving, creative exploration, and social support. To realize this goal, the
development of automated agents proficient in skills such as negotiating,
following instructions, establishing common ground, and progressing shared
tasks is essential. This survey begins by reviewing the evolution of dialogue
management paradigms in collaborative dialogue systems, from traditional
handcrafted and information-state based methods to AI planning-inspired
approaches. It then shifts focus to contemporary data-driven dialogue
management techniques, which seek to transfer deep learning successes from
form-filling and open-domain settings to collaborative contexts. The paper
proceeds to analyze a selected set of recent works that apply neural
approaches to collaborative dialogue management, spotlighting prevailing
trends in the field. This survey hopes to provide foundational background for
future advancements in collaborative dialogue management, particularly as the
dialogue systems community continues to embrace the potential of large
language models.

##  1 Introduction

Human collaboration through dialogue is a fundamental aspect of our everyday
lives, as we often engage in conversations to work together, solve problems,
and achieve shared goals Streeck et al. (2011). However, effective
collaboration through dialogue is a complex process, as it requires the
participants to take turns, negotiate meaning Clark and Brennan (1991),
coordinate actions Clark and Wilkes-Gibbs (1986), and manage conflicts Traum
et al. (2008), all while maintaining a shared understanding of the task at
hand. These factors contribute to the complexity of building an automated
dialogue system that can act as a collaborative partner to one or more humans.

It has been a longstanding goal in dialogue systems research to develop
domain-specific systems, general-purpose frameworks, and algorithms to build
automated agents that can act as collaborative partners to human users Rich et
al. (2001); Bohus and Rudnicky (2009); Allen et al. (2002); Galescu et al.
(2018). However, most approaches that predate the use of neural network models
require some form of manual labor to extend their functionality to a new
domain. Recently, the advent of deep learning based approaches ushered in a
new era of data-driven methods for carrying out various tasks within a
dialogue system, including natural language understanding Louvan and Magnini
(2020); Shah et al. (2019), dialogue management Mrkšić et al. (2016); Ren et
al. (2018), and natural language generation Wen et al. (2015); Santhanam and
Shaikh (2019). In fact, a long line of work from the mid-2010s has produced
success stories in the realms of form-filling agents (e.g., used to carry out
tasks such as booking a restaurant table on behalf of the user) and open-
domain agents (also called chatbots, which engage in open-ended conversation
with the user with no particular end-goal). Nevertheless, compared to form-
filling and open-domain agents, there has been a paucity in both theoretical
advances and practical implementations that use neural-network based
approaches to build agents for collaborative dialogue.

In this survey, I first explore traditional dialogue management paradigms used
for over three decades in developing automated agents for collaborative
dialogue. Dialogue management paradigms are abstract computational models
offering guidelines for processing inputs, guiding conversations, and
maintaining context. Concrete data structures and algorithms translate these
paradigms into functioning systems. I provide a concise description of each
paradigm, emphasizing its applications in collaborative contexts, in Section
4.

The survey then narrows its focus to neural network-based approaches for
collaborative dialogue management. Since the mid-2010s, various methodologies
have sought to combine traditional paradigm strengths with the
representational flexibility of neural networks. This integration involves
modifying standard end-to-end neural architectures, typically used for form-
filling and open-domain settings, which is explored in Section 5.

This literature survey systematically explores these themes through the
following two research questions (RQs):

  * •

RQ 1: What dialogue management approaches have been utilized to develop
collaborative dialogue agents?

  * •

RQ 2: What techniques have been applied to adapt neural network based dialogue
management approaches to collaborative settings?

The remainder of this paper is organized as follows. Section 2 provides
background on collaborative dialogue systems and the dialogue management
problem. In Section 3, I list the research questions and discuss the
literature search process. Section 4 examines various design paradigms for
building collaborative dialogue managers. Before answering RQ 2, Section 4.5.1
outlines challenges faced by neural approaches to collaborative dialogue
management. In Section 5, I focus on adaptations made to neural-network-based
dialogue management approaches for collaborative settings. Section 6 discusses
survey findings and future research directions, while 7 concludes the survey
with a summary.

##  2 Background

This section begins by providing a working definition of collaborative
dialogue systems. Then, it highlights the role of the dialogue manager within
a dialogue system, which is the module responsible for guiding the
conversation.

###  Defining Collaborative Dialogue Systems

Collaborative dialogue systems are designed to collaborate with human users to
achieve one or more shared goals, such as co-creating an artifact or
navigating a physical environment Streeck et al. (2011). These systems must
support functions such as negotiating shared goals and developing a mental
model of the human user. To better understand collaborative dialogue systems,
it is helpful to first introduce two other types of dialogue agents: form-
filling assistants and chatbots, and then contrast them with collaborative
dialogue agents.

Form-filling assistants, also known as task-oriented dialogue systems Williams
et al. (2014a), perform tasks on the user’s behalf, such as reserving a table
or booking a flight. However, they do not possess their own goals or agency.
Chatbots, on the other hand, engage users in open-ended conversations without
specific goals, making them non-collaborative as well Huang et al. (2020). In
contrast to these two types of dialogue agents, collaborative dialogue agents
play a variety of roles that involve working together with the human user.

Some examples of roles that collaborative dialogue agents may assume include
that of an information giver (providing step-by-step instructions to the human
user to follow), a collaborative partner (planning and working together with
the human to complete a task), a tutor (helping the human user learn a concept
or a skill), and even an opponent (in a negotiation setting or within the
context of a game). These agents have their own goals that may be slightly or
even greatly different from that of the human user (e.g., an agent designed to
bargain with a human).

###  Role of the Dialogue Manager

Computational approaches to dialogue serves two distinct purposes: dialogue
modeling and dialogue management Traum (2017). Dialogue modeling focuses on
understanding and explaining conversation dynamics without actively
participating. In contrast, dialogue management involves deciding what to say
next based on the dialogue context, thereby enabling the agent to actively
participate in the conversation. This survey is specifically concerned with
dialogue management.

The dialogue manager guides the conversation within a dialogue system.
Conceptually, it can be divided into representation and decision-making.
Figure 1 shows a standard pipeline architecture of a dialogue system. Note
that while most dialogue systems can be factored into these components, it may
not always be the case. In this pipeline architecture, dialogue management is
carried out jointly by the state tracker and the dialogue policy. The
representation, or dialogue state, provides an internal summary of the current
dialogue status, which needs to be continuously updated throughout the
conversation. The decision-making aspect, or dialogue policy, determines the
most suitable action based on the dialogue state and other relevant factors,
such as the state of a shared task. In some approaches, such as rudimentary
script-based approaches, these two functions may be inseparably coupled into a
single functional module.

Figure 1: A pipeline-based architecture of a dialogue system, comprising
modules for natural language understanding (NLU), state tracking, dialogue
policy, and natural language generation (NLG). The state tracking and dialogue
policy are jointly responsible for carrying out the function of dialogue
management.

##  3 Methods

This section outlines the search methodology, including the inclusion and
exclusion criteria employed to select relevant works for the literature
survey. Then, it details the approach taken to identify the modeling themes
for answering RQ 2.

###  Search Strategies

The search strategy for RQ 1 and RQ 2 were conducted independently.

####  3.1.1 Search Strategy for RQ 1

To address RQ 1, the search began with a curated “seed” collection of
influential works presenting foundational dialogue management paradigms for
collaborative dialogue. Additional works were identified through the reference
lists of these highly cited publications (snowballing). Specifically, the
inclusion and exclusion criteria for this RQ were as follows:

  * •

IC1: The work should present either a novel paradigm of dialogue management or
a domain-independent framework that can be leveraged to develop a wide range
of domain-specific dialogue systems.

  * •

IC2:  A sizeable number of practical dialogue system implementations should be
implemented using the paradigm.

  * •

IC3:  The dialogue management paradigm should have been used to develop one or
more of the following types of collaborative agents: collaborative problem-
solving, negotiation, or tutoring.

  * •

EC1:  The system or framework should not specific to one particular domain, in
which case it cannot be considered to be a “paradigm.”

From reviewing these combined set of works, I identified five distinct
paradigms of collaborative dialogue management: script-based, plan-based,
information-state based, those based on Allen et al. (2002)’s collaborative
problem solving (CPS) framework, and neural-network based approaches.

####  3.1.2 Search Strategy for RQ 2

For RQ 2, a separate and distinct search was conducted with the goal of
discovering neural network-based approaches to developing collaborative
dialogue managers. A targeted search was performed using terms such as
"collaborative dialogue systems," "collaborative dialogue state," and
"collaborative dialogue" across Association for Computational Linguistics
(ACL) and search engines such as Zeta Alpha111zeta-alpha.com, Semantic
Scholar222semanticscholar.org, and Google Scholar333scholar.google.com. The
inclusion and exclusion criteria for selecting papers were applied
specifically for RQ 2, as follows:

  * •

IC1: Inclusion of neural network-based approaches to developing collaborative
dialogue managers.

  * •

IC2: Consideration of works primarily published post-2016, reflecting the
emergence of neural network approaches during this period.

  * •

IC3:  Negotiation dialogue systems; although commonly characterized as non-
collaborative dialogue, these systems were included as negotiation frequently
features in collaborative dialogue during planning and shared goal setting.

  * •

EC1: Exclusion of works that lack a data-driven, neural network-based dialogue
manager implementation.

  * •

EC2: Exclusion of studies focusing on multi-party collaborative dialogue
systems, with the scope limited to two-party systems.

The final set of papers was derived from a combination of this search strategy
for RQ 2 and additional publications discovered through the citation networks
of these articles that also fit these criteria.

##  4 RQ 1: What dialogue management approaches have been utilized to develop
collaborative dialogue agents?

In this section, I describe script-based, plan-based, information-state, and
neural-network based paradigms to collaborative dialogue management. In
addition, I also describe the Collaborative Problem Solving (CPS) framework,
which was designed specially with collaborative dialogue management in mind.
Except for the CPS framework, I begin with a general description of each
paradigm and then touch upon how they have been used specifically for
collaborative settings.

###  Script-based Approaches

Script-based approaches to dialogue management are based on a predetermined,
“ideal” flow of dialogue and are handcrafted to each specific domain Traum
(2017). Specifically, it involves specifying a handcrafted ”script” that both
the dialogue system and the user must follow. A script usually consists of
fixed patterns and pre-written responses, which can be matched to user input
through pattern matching, keyword extraction, or rule-based methods. For
instance, a script can encode an adjacency-pair rule stating that a question
needs to be followed by an answer Dählback and Jonsson (1989). More complex
scripts are also equipped to handle a rudimentary level of deviations, such as
clarification questions in the dialogue. Examples of popular, early
implementations of script-based dialogue systems include ELIZA Weizenbaum
(1966), LINLIN Dahlbäck et al. (1999), and RailTel Bennacef et al. (1996).

Within collaborative contexts, one popular type of dialogue systems where
script-based dialogue management is extensively employed is dialogue-based
intelligent tutoring systems (ITSs) Webb and Farivar (1999); Strijbos (2004).
ITSs are adaptive educational systems that provide personalized instruction
and feedback to learners, employing techniques from artificial intelligence,
cognitive psychology, and education research to optimize the learning process
Wollny et al. (2021). Many ITSs such as CIRCSIM-Tutor Evens et al. (1997),
AutoTutor Graesser et al. (2004) and Why2-Atlas VanLehn et al. (2002) make use
of dialogue-grammar based (also called script-based) dialogue managers. These
grammars (or scripts) are designed in such a way so as to help the user
achieve a goal.

For example, the grammar in Why2-Atlas VanLehn et al. (2002) enforces the
student to provide a comprehensive explanation of a physics phenomenon. It
works by mapping utterances to an explanation or a misconception using pattern
matching techniques. Once the agent believes it has identified the right
explanation or misconception, it then responds with a pre-defined response
corresponding to it. The dialogue manager continues to cycle through the list
of all pieces of the unmentioned explanations until it is satisfied that the
student understands the concept.

While handcrafted, script-based approaches can be useful for creating simple,
controlled dialogue systems with predictable conversation flows, they tend to
lack flexibility and adaptability. As a result, they might be limited in their
ability to handle diverse, complex, or unanticipated user inputs, which can
limit their effectiveness in more dynamic and interactive scenarios. Despite
these limitations, handcrafted dialogue management has laid the foundation for
more advanced techniques, and it continues to be employed in specific contexts
where a well-defined, structured conversation is required.

###  Plan-Based Approaches

Humans naturally ascribe intentions and goals to their conversational
counterparts, expecting rational behavior in line with these objectives. Plan-
based methodologies for dialogue management encapsulate this collaborative
mindset Papaioannou et al. (2018). One notable implication of this viewpoint
is that natural language expressions are perceived as actions (dialogue acts
Searle (1969)) undertaken by rational agents to achieve goals Allen (1983);
Pollack (1992). Several systems also make use of plan recognition to deduce
user’s objective based on their actions. The goal-oriented nature of this
paradigm offers a degree of coherence and consistency in a dialogue agent’s
behavior that is challenging to attain through alternative dialogue management
paradigms. As such, this approach has been adopted in high-stakes application
domains such as healthcare Vannaprathip et al. (2022); Kane et al. (2022) and
education Freedman (2000).

Plan-based dialogue management originates from AI planning Ghallab et al.
(2004), which can be described as a search problem aimed at identifying a
sequence of actions that, upon execution, accomplishes a goal. At any given
moment, the agent resides in one of several potential states. Actions
transition the system from one state to another. Each action possesses a set
of preconditions, which must be satisfied for the action to be deemed eligible
by the planning algorithm. The outcome of performing an action is defined as a
series of effects on the state. Ultimately, a planning problem is initiated
with both an initial and a final goal state.

Over the past three decades, numerous practical implementations of plan-based
dialogue management in collaborative dialogues have been proposed. Two notable
implementations of plan-based dialogue managers in collaborative problem-
solving settings are the TRAINS Allen et al. (1995) and TRIPS Allen et al.
(2005) systems by James Allen’s group. More recent works integrate mixed-
initiative capabilities, such as MAPGEN Ai-Chang et al. (2004), which
implemented a plan-based system for ground operations of a NASA Mars Rover and
SIADEX de la Asunción et al. (2005), which was deployed to help coordinate
forest-fire operations. Various domain-independent frameworks such as COLLAGEN
Rich et al. (2001), PASSAT Myers et al. (2002), and RavenClaw Bohus and
Rudnicky (2009) have been developed to allow domain-specific agents to be
built.

###  Information-State Based Approaches

In the information-state based approach to dialogue management, as proposed by
Larsson and Traum (2000), the dialogue state is defined by a collection of
beliefs, desires, and intentions (BDI) Rao and Georgeff (1991). As the
dialogue unfolds, each utterance leads to an update in the dialogue state. The
precise nature of these updates relies on a series of update rules and
strategies. This approach allows for a high level of adaptability due to its
minimal constraints on the dialogue manager, making it suitable for various
dialogue situations, including collaboration.

When examining the information-state approach more closely, it can be
characterized by three primary components: informational components, dialogue
acts, and update rules Larsson and Traum (2000).

  1. 1.

Informational Components. In collaborative settings, information states are
categorized into private aspects (individual agent’s beliefs, desires, and
intentions) and shared aspects (mutually agreed-upon beliefs, desires, and
intentions). These components can be either static, such as agent persona and
background knowledge, or dynamic, such as evolving goals and dialogue history.

  2. 2.

Formal Representations. Various representations may be used for informational
components, affecting their accessibility, ease of updating, and overall
efficiency. These representations can include basic data structures, composite
data types, logical forms, or neural embeddings.

  3. 3.

Dialogue Moves. Dialogue moves represent the "illocutionary force" of an
utterance as defined by Searle (1969). They are identified using domain-
specific coding schemes.

  4. 4.

Update Rules. Drawing inspiration from plan-based dialogue management (see
Section 4.2), update rules consist of preconditions and effects Traum et al.
(1999). A rule can only be applied if its preconditions are met; once they
are, the effects are used to update the information state.

  5. 5.

Control Strategy. Control strategies dictate the selection and sequence of
applied update rules. These strategies can involve: 1) applying the first rule
that matches, 2) sequentially applying all matching rules, 3) employing a
probabilistic rule for selection, or 4) allowing the user to make the choice.

TrindiKit is a domain-independent framework for collaborative dialogue
management Larsson and Traum (2000). Basilica is another example of a domain-
independent dialogue management framework proposed by Kumar and Rosé (2011).
It uses a unified architecture for authoring tutoring and collaborative
learning agents. Basilica works on a decentralized network of object-oriented
modules, each one programmed and functioning independently through common
interfaces. Basilica has been used to develop several collaborative learning
agents, including CycleTalk tutor Roman et al. (2020) and TuTalk Jordan et al.
(2006). To share information and events across modules, Basilica uses a
shared, global state. This state corresponds to the information state of the
agent. Modules can perform a wide variety of functions, including recognition
of student affect, monitoring of task progress, and dialogue management.

Figure 2: Collaborative problem solving model Allen et al. (2002)

###  Collaborative-Problem Solving (CPS) Framework

Allen et al. (2002) introduced the domain-independent CPS framework for
dialogue management collaborative agents. The CPS model is designed from
ground up to handle more complex tasks than state-based or script-based models
(such as those in Sections 4.1 and 4.3). The model views a collaborative
dialogue as operating over three levels of interaction (see Figure 2):

  1. 1.

Problem Solving Level: This level describes how a single agent solves
problems. It includes processes such as planning, decision-making, and
execution.

  2. 2.

Interaction Level: This level involves the interaction between two or more
agents. It includes processes such as communication, negotiation, and
coordination.

  3. 3.

Coordination Level: This level involves the coordination of multiple
interactions between agents to achieve a common goal. It includes processes
such as resource allocation, task assignment, and conflict resolution.

The CPS model involves multiple agents working together to solve a problem,
with each agent contributing its own expertise and knowledge to the process.
While the authors are primarily focused on human-machine collaboration, they
believe that the model will equally well apply to interactions between
sophisticated software agents that need to coordinate their activities.

Based on the CPS model, Galescu et al. (2018) introduce the Cogent dialogue
shell as a practical, extensible, domain-independent “shell” on top of which
domain-specific agents can be built. It achieves this by providing many of the
domain-independent capabilities out-of-the-box, including general-purpose
natural language parsing and collaborative dialogue management capabilities.

###  Neural-Network Based Approaches

In their most general form, neural-network based dialogue managers use
sequence-to-sequence models Sutskever et al. (2014) to generate the dialogue
system’s next response directly from the natural language dialogue history.
The advantage of building a dialogue manager this way is that it circumvents
labor-intensive handcrafting of dialogue management strategies by learning it
from a corpus of training dialogues. These corpora are typically collected
from human-human interactions in the same or similar domain. This paradigm of
training a dialogue agent was originally popularized in the context of open-
domain chatbots Shum et al. (2018); Wolf et al. (2019); Serban et al. (2016a).
However, several works soon followed it up by extending the paradigm to task-
oriented dialogue systems that performed slot-filling Bordes et al. (2017);
Wen et al. (2017).

####  4.5.1 Challenges in Neural Collaborative Dialogue Management

Before diving deep into the second research question, I enumerate some of the
key challenges associated with neural network-based approaches to building
collaborative dialogue systems as background.

  1. 1.

Lack of interpretability and explainability: The absence of explicit internal
structure and intermediate representations in end-to-end models can make it
difficult to interpret and understand the decision-making process, which is
particularly important in strategic and collaborative settings where trust and
explainability play a critical role.

  2. 2.

Difficulty in grounding to real-world entities: Collaborative dialogue often
takes place within the context of a common artifact such as a co-observed
image Shinagawa et al. (2020); Kim et al. (2019) or an environment Roman et
al. (2020); Chi et al. (2020). However, vanilla neural dialogue models444The
term ”vanilla” is used to describe unmodified or basic versions of
architectures, without any special enhancements or adaptations. It refers to
the fundamental, original form of a model, serving as a foundation for more
advanced or customized variations. often struggle with grounding, as they lack
explicit mechanisms to represent or reason about real-world entities, leading
to contextually inappropriate or inconsistent responses.

  3. 3.

Large amounts of training data required: These models often need extensive
training data to achieve good performance, which may not be readily available
for specialized or niche domains Sutskever et al. (2014); Li et al. (2016);
Serban et al. (2016b).

  4. 4.

Hallucinations and challenges in generating accurate responses: In the context
of generative language models, a hallucination refers to a situation where the
AI model generates output that is not grounded in the input data, factual
information, or the intended context Ji et al. (2023); Cao et al. (2022). This
can happen due to several reasons, such as overfitting, biases in the training
data, or insufficient constraints on the generative process. For example, if a
language model is asked to generate an answer to a historical question and it
produces a response with incorrect dates or events, that would be considered a
hallucination. Hallucinations are, therefore, a challenge to be dealt with
when using neural models for generating dialogue responses, decreasing the
overall quality and reliability of the conversation.

  5. 5.

Difficulty incorporating external knowledge and adapting to changes: Such
models have trouble incorporating external knowledge or adapting to changes in
the environment during a conversation, which is a common requirement in
strategic and collaborative applications Sutskever et al. (2014); Weston et
al. (2018).

##  5 RQ 2: What techniques have been applied to adapt neural network based
dialogue management approaches to collaborative settings?

To systematically analyze the 17 selected works for addressing RQ 2, I
employed a three-step approach to investigate each work, extract relevant
architectural modifications, and classify them into overarching themes. The
process unfolded as follows:

  1. 1.

In-depth examination: I analyzed each work, delving into the proposed
architectural alterations and evaluating their significance for neural
dialogue management.

  2. 2.

Extraction and preliminary grouping: While assessing each work, I extracted
and recorded all architectural changes, observing their distinct attributes
and qualities. This process led to an initial collection of categories that
encompassed the diverse facets of the modifications.

  3. 3.

Thematic integration: After completing the extraction and preliminary
categorization, I executed an extensive synthesis of the findings. Through a
recursive process of analysis and comparison, I discerned connections and
patterns among the various categories.

Table 1: Overview of different papers and their corresponding modeling themes. Columns with boolean values are marked with a check symbol for true and left empty for false. The columns represent the following: (1) Paper, (2) Participant Roles (* indicates the participant being modeled), (3) Incorporating Shared Artifacts or Environments, (4) Decoupling Semantics and Surface Realization, (5) Graph-Based Representations of Dialogue State, (6) Incorporating Domain-Specific or Expert Knowledge, (7) Theory-of-Mind Modeling, (8) Modality Encoding/Fusion |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---
Paper |  Participant Roles |  Modalities or Task | Decoupling | Graph | Domain | ToM | Modality |
Bara et al. (2021) |  Player 1*,
Player 2* |  theory-of-mind questionnaires,
Minecraft 3D game state |  |  |  | ✓ |  |
Chi et al. (2020) |  Navigator*,
Instructor |  3D room environment |  | ✓ |  |  | ✓ |
de Vries et al. (2018) |  Guide*,
Tourist |  Map of a City |  | ✓ |  |  |  |
He et al. (2017) |  Friend 1*,
Friend 2* |  Knowledge Bases of Friends |  | ✓ |  |  |  |
He et al. (2018) |  Buyer,
Seller |  Negotiation Scenario + Item Details | ✓ |  |  |  |  |
Jayannavar et al. (2020) |  Builder*,
Architect |  Blocks World |  |  |  |  | ✓ |
Li et al. (2019) |  User*,
Attacker |  Private Profile Information | ✓ |  | ✓ |  |  |
Kim et al. (2019) |  Teller*,
Drawer |  Drawing Canvas with Objects |  |  |  |  | ✓ |
Narayan-Chen et al. (2017) |  Builder*,
Architect |  2D Blocks World |  |  |  |  | ✓ |
Narayan-Chen et al. (2019) |  Builder*,
Architect |  3D Minecraft World |  |  |  |  | ✓ |
Qiu et al. (2022b) |  Player*,
Other Players* |  Game State (including Persona Descriptions) |  | ✓ | ✓ | ✓ |  |
Santhanam et al. (2020) |  Persuader*,
Persuadee |  Charity Donation | ✓ |  |  |  |  |
Shi et al. (2022) |  Architect,
Builder* |  3D Minecraft World |  |  |  |  | ✓ |
Yarats and Lewis (2017) |  Negotiator 1*,
Negotiator 2* |  Private Negotiation Goals + Item Pool | ✓ |  |  |  |  |
Zhou et al. (2019) |  Buyer,
Seller,
Coach* |  Product Listing |  |  | ✓ |  |  |
Zhou et al. (2020) |  Negotiatior 1*,
Negotiator 2* |  Product Listing (for CraigslistBargain) |  |  | ✓ |  |  |
Zhou et al. (2022) |  DM*,
Players |  Game Background Story + Game State |  |  |  | ✓ |  |
Table 1: Overview of different papers and their corresponding features
(Continued)

The above process led to the consolidation of five overarching modeling
themes. In this section, I examine these main modeling themes, discussing the
details and features of neural dialogue management models. Table
LABEL:tab:overview maps each work to one or more modeling themes that are
identified.

###  Decoupling Semantics and Surface Realizations

In this section, I review approaches that address an important limitations of
vanilla end-to-end trained models of dialogue: separation of semantics and
surface realization. In vanilla sequence-to-sequence modeling of dialogue, a
trained neural network processes raw dialogue history as input and directly
generates the subsequent natural language response without distinct
intermediate steps for natural language understanding, state tracking, and
generation. Crucially, traditional end-to-end training approaches fail to
distinguish between "what to say" (semantics) and "how to say it" (surface
realization), leading to complications as a single concept can be expressed in
numerous ways.

####  5.1.1 Conditioning Response Generation on Intermediate Symbolic
Representations

He et al. (2018) propose an approach that separates a dialogue system’s
strategy from the natural language surface realization in a negotiation
setting. They employ two distinct sequence-to-sequence models to represent the
dialogue (see Figure 3). The first model operates directly on the tokens in
natural language utterances, while the second model works with coarse dialogue
acts corresponding to the utterances. First, a parser is trained to map an
utterance, $x_{t-1}$, to a coarse dialogue act $z_{t-1}$, which captures the
high-level dialogue flow. A dialogue act consists of an intent and optional
arguments. For instance, the utterance "I would like to pay $125 for it. How
does that sound?" is mapped to the dialogue act propose(price=125). The
dialogue act ontology includes inform, inquire, propose, counter, and agree.
Both sequences are modeled using LSTMs Hochreiter and Schmidhuber (1997).
Given $z_{t-1}$ and the dialogue history, $x_{<t}$, the dialogue manager
predicts the coarse dialogue act of its response, $z_{t}$. Finally, a natural
language response is generated based on $z_{t}$.

Figure 3: In He et al. (2018), the natural language parser generates dialogue
acts, $z_{<t}$, corresponding to the utterances $x_{<t}$ respectively. The
next response is generated in two steps: (1) generating the next dialogue act,
$z_{t}$ based on the previous dialogue acts, $z_{<t}$ and 2) generating the
natural language response based on both the dialogue history, $x_{<t}$ and the
preceding dialogue act, $z_{t}$.

Santhanam et al. (2020) present an approach that generates an intermediate
"plan" to condition the natural language response. They demonstrate this
method on the Persuasion For Good dialogue corpus Wang et al. (2019), in which
the agent persuades a listener by offering compelling incentives. The process
consists of two steps: first, generating a pseudo-natural language plan, and
second, conditioning the model on this plan to produce the final response.
Figure 4 illustrates an example dialogue context along with a system-generated
intermediate plan. In response to B’s final utterance, the model first
generates a plan with an ask and framing, such as PERFORM [provides [relief]].
It then creates a natural language response, such as "Save the Children is an
international non-governmental organization that promotes children’s rights,
provides relief, and supports children in developing countries" (surface
realization).

Figure 4: Separation of NLG into planning and surface realization Santhanam et
al. (2020).

####  5.1.2 Conditioning Response Generation on Latent Representations

In contrast to approaches that employ symbolic intermediate representations
like dialogue acts and plans, some methods utilize latent variables instead.

Yarats and Lewis (2017) propose generating a short-term plan using latent
sentence representations. The natural language response is then generated by
conditioning the response generation step on this plan. To learn these latent
sentence representations from data, the authors deviate from the traditional
paradigm of maximizing the likelihood of the current utterance $x_{t}$ given
the latent representation $z_{t}$. Rather, they train $z_{t}$ to maximize the
likelihood of the response at the next turn, $x_{t+1}$. This approach is
motivated by the observation that while the traditional method effectively
models semantic similarity, it fails to capture the influence of an utterance
on the dialogue partner’s response. This distinction can be crucial in
negotiation settings, where even subtle lexical substitutions such as using
"two" instead of "one," can significantly affect the final outcomes of the
conversation.

###  Incorporating Shared Artifacts, Environments and Knowledge

Collaborative dialogues often take place when participants are working on a
shared artifact or within a co-observed environment (e.g., an architect and a
builder collaborating to build a Minecraft structure). Collaboration can also
involve an asymmetry in the knowledge that the participants are privy to
(e.g., the architect may have a bird’s eye view of the world, while the
builder only has a first-person view). In this section, I provide a high-level
overview of the approaches taken by collaborative dialogue systems to
incorporate three common modalities: navigable environments, co-created
artifacts, and text-based background knowledge555Note that I do not delve into
fine-grained details of multimodal encoding and fusion methods. Several
existing surveys on multimodal dialogue systems cover those topics, and I
refer readers to works such as Liu et al. (2022) for a comprehensive overview
of such techniques..

####  5.2.1 Navigable Environments

Visual-language navigation (VLN) Park and Kim (2023); Lukin et al. (2018) is a
broad research area that explores the interplay between natural language
instructions and visual perception to enable autonomous agents or robots to
navigate complex environments. Typically, a leader (e.g., a human user)
provides natural language directions to a follower (e.g., a robot) navigating
the environment. The follower may ask clarification questions to resolve
ambiguities in either the environment or the instructions. In this section, I
discuss two VLN systems, Chi et al. (2020) and de Vries et al. (2018),
focusing on their world and dialogue state representations, how these
representations are used in dialogue management, and their approaches to
handling clarification questions.

In Chi et al. (2020), a human user instructs a navigating robot situated in a
3D environment with commands such as, “Walk straight, right before you reach
the bed”. The authors propose a navigating agent modeled as a Markov Decision
Process (MDP) with states representing the agent’s position and orientation.
The agent’s action space consists of environment-defined "navigable locations"
and an additional clarification question action. Visual perception, position
information, and user instructions serve as multimodal input to an LSTM model
Hochreiter and Schmidhuber (1997), outputting a softmax distribution, $P^{t}$,
over the action space. The authors explore both supervised and reinforcement
learning methods for training.

In supervised learning, the environment calculates the shortest path
$R_{t}=\\{v_{t},v_{t+1},\dots,v_{n}\\}$ from the current viewpoint $v_{t}$ to
the target location $v_{n}$ at each time step $t$. Training loss is determined
by the cross-entropy loss between the predicted action and the next action in
the shortest path. For reinforcement learning, the agent selects its action by
sampling from the action distribution $P^{t}$. A reward of +2 is granted if
the final location is within 3 meters of $v_{n}$, and -2 otherwise. The
Advantage Actor Critic (A2C) algorithm Konda and Tsitsiklis (1999) is utilized
for RL training. The reward mechanism is further adjusted to encourage the
agent to disambiguate its future trajectory by minimizing clarification
questions.

A closely related work is that of de Vries et al. (2018), where a Tourist
agent and a Guide agent collaborate using natural language dialogue to reach a
target location that is known only to the Guide. While the Guide has access to
a 2D city map, the Tourist only has a first-person view of the local
surroundings.

####  5.2.2 Co-Created Artifacts

Many collaborative activities involve co-creating some kind of artifact.
Artifacts can range from block-based structures in a Minecraft game Shi et al.
(2022); Narayan-Chen et al. (2019) to drawings Kim et al. (2019) to source
code Griffith et al. (2022). The two participants may be set up in either an
asymmetric leader-follower setup or a symmetric one where both of them possess
equal knowledge and skills.

Minecraft is a widely-used platform for studying collaborative dialogue in
"Architect-Builder" scenarios, where the Architect agent instructs the Builder
agent on constructing a target structure in the game environment. Narayan-Chen
et al. (2019) introduce a dialogue model for the Architect that utilizes two
types of counters—global and local block counters—to track the building
progress. Global counters monitor overall block placements, upcoming
placements, and removals, while local counters focus on a 3x3x3 region
surrounding the Builder’s last action. These block counter vectors, capturing
attributes such as position and color, are processed through a fully-connected
layer and combined with the word embedding vector for the decoder. This
integration of dialogue history and world state representations enables the
generation of contextually accurate instructions and a mental model of the
target structure.

Shi et al. (2022) explore generating clarification questions from the
Builder’s perspective. Unlike previous works, which prioritize Architect
communication and Builder actions, their study addresses the generation of
Builder utterances. The dialogue state consists of dialogue history, world
state (with 3D voxels represented as 8-dimensional vectors), and the last
action (encoded as a vector detailing action type, block color, and location).
These three elements are transformed into vector representations through a
convolutional neural network (CNN) and merged using a neural fusion module,
resulting in the final dialogue state representation.

Figure 5: Architecture Diagram from Narayan-Chen et al. (2019).

Kim et al. (2019) present a goal-driven collaborative task very similar to the
Minecraft setting, involving two participants—a Teller and a Drawer—where the
Teller describes a drawing to the Drawer, who then recreates it based on the
instructions. The authors propose initial models for automating both agents,
assuming the Drawer does not ask clarifying questions. The Teller’s role
involves understanding the drawing and generating natural language
descriptions, while the Drawer is expected to comprehend the Teller’s
instructions and replicate the drawing. Neural models are proposed for both
agents, with the Teller’s model using a conditional language model with an
LSTM and two attention modules that attend to individual clip art pieces in
the drawing. The Teller model also incorporates an auxiliary training
objective to track which parts of the scene have been communicated, using the
LSTM’s output at each utterance separator token to classify whether each clip
art has been described or not.

####  5.2.3 Textual Background Knowledge

In many instances, it is common for the background knowledge pertaining to the
collaborative task to be in the form of text. For example, in Qiu et al.
(2022b), a game agent is provided background and situational knowledge of the
world using three pieces of text: the setting (e.g., The main foyer is
massive. A grand staircase sits to the back of the foyer leading to the
upstairs …), the agent’s self-persona (e.g., Servant. I come from the lower
class. I do what I am told without question. I can not read. I have not seen
my family in a long time. I carry a duster, a small bucket, a rag …), the
partner’s persona (e.g., I am a king of the whole empire. I give rules and
pursuit them. I am brave and fearless. I am carrying a crown and a scepter).

The proposed model in Qiu et al. (2022b) encodes this text-based
representation by feeding it into GPT-3 Brown et al. (2020), a large language
model (LLM) as part of its prompt. For example, the above description is fed
into the model as follows: “The following is a conversation that happened in a
game of Dungeons and Dragons: [Context] [DM Text] [Player Name]:[Player
Ability Check] Question: What do you think that the DM is trying to guide the
player to do by mentioning [Extracted Guiding Sentence]? Answer”.

###  Graph-Based Representations of Dialogue State

Graph-based dialogue state representations strive to strike a balance between
rigid data structures, such as slot-value pairs, and completely unstructured
continuous vector representations. By employing graphs, these representations
closely mirror the inherent structure of real-world tasks or domains. This
approach offers semi-interpretable and robust neural representations, blending
the advantages of visualizations with the potential for adaptable
representation. Such representations learn entity representations from data,
and their popularity has surged in recent years for task-oriented dialogue
applications Walker et al. (2022); Andreas et al. (2020). In this section, I
discuss the components (nodes and edges) of a graph and the methods employed
to update the graph as the dialogue unfolds.

Figure 6: Architecture of symmetric collaborative dialogue state from He et
al. (2017).

####  5.3.1 Graph Representation

Among the reviewed works, He et al. (2017) uses a graph-based dialogue state
representation for a task that involves identifying a mutual friend from a
knowledge base through text-based dialogue. Their proposed approach consists
of a knowledge graph, graph embeddings, and an utterance generator. The
knowledge graph comprises the following three types of nodes, with edges
between them representing their relations. When an utterance $t$ mentions a
new entity, it is added as a new node to the knowledge graph.

  1. 1.

entity nodes representing people, universities, and companies,

  2. 2.

item nodes corresponding to a single row-entry in the agent’s knowledge base
(e.g., a friend), and

  3. 3.

attribute nodes that represent relationships between entities, such as a
person’s connection to a university

In Qiu et al. (2022b), a knowledge graph is used to represent the agent’s
mental state at a given point in a game. The graph is initialized by parsing a
text-based description of the agent’s persona into constituent entities and
relations using a rule-based parser. The nodes in the graph correspond to the
agents involved in the game, persona descriptions, objects, descriptions of
the objects, and descriptions of the environment. The edges describe the
relationships between the agents and the objects as present in the agent’s
mental state. Figure 8 shows an example “mental state graph.”

####  5.3.2 Graph Encoding and Updating

In He et al. (2017), the final dialogue state representation is obtained from
the dialogue state graph’s node embeddings (Figure 6 provides a sample
snapshot of the dialogue state used in this system). Given a dialogue of $T$
utterances, $T$ graphs $(G_{t})_{t=1}^{T}$ are constructed for each agent $A$.
The graph $G_{t}$ is constructed by updating $G_{t-1}$ with any new entities
not previously mentioned. To obtain a latent representation of the graph, an
embedding is associated with each node in the graph, $v\in G_{t}$. A node
embedding $V_{t}(v)$ is built from structural properties of an entity in the
knowledge base, embeddings of the utterances in the text-based dialogue
history, and the message passing between the graph nodes. Each of these is
elaborated below.

  1. 1.

Structural properties: The structural properties of a node $v$ are encoded
using a featurization function $F_{t}(v)$, which encodes various pieces of
information such as the degree, type and number of mentions of the node.

  2. 2.

Mention vectors: A mention vector of a node $v$ contains an aggregate
representation of the mentions of that entity so far in the dialogue history.
At every dialogue turn $t$, the mention vector $M_{t}(v)$ is set to
$M_{t-1}(v)$ if utterance $t$ does not contain mention of the entity and a
soft weighted sum of $M_{t-1}(v)$ and $\tilde{u}$ if the utterance does
contain mention of the entity. $\tilde{u}$ is obtained by using an LSTM to
encode the utterance.

  3. 3.

Recursive node embeddings: It is important to take into account information
from neighboring nodes in the knowledge graph. For example, to answer the
question “anyone went to colombia?” as in Figure 6, the model should take
cognizance of the entity nodes of type person who went to colombia university
(in this case, jessica and josh). This is achieved through a mechanism known
as belief propagation.

In Qiu et al. (2022b), the graph update step is broken down into two update
types: discrete and continuous. The actions in the game environment, which can
be converted into simple, text-based templates are modeled as discrete graph
updates, while the unconstrained natural language utterances, which are harder
to model, are incorporated to the representation through a continuous graph
update instead.

  1. 1.

Discrete update: In the discrete update step, $\delta g_{t}$ is a set of ADD
and DEL operations. Each action in the game environment is parsed into a set
of these atomic operations over the graph. For example, the action “give
object to agent” is parsed into DEL($actor,object,carrying$) and
ADD($agent,object,carrying$).

  2. 2.

Continuous update: The continuous update step is motivated by the reasoning
that an utterance can also have implicit effects on the agent’s mental state.
This step is handled by a recurrent neural network in the following steps:

$\displaystyle\delta g_{t}=f_{\delta}(h_{G_{t-1}},h_{O_{t}})$ $\displaystyle
h_{t}=\text{RNN}(\delta g_{t},h_{t-1})$ $\displaystyle
G_{t}=\text{MLP}(h_{t})$

Here, $f_{\delta}$ is a bi-directional attention layer Yu et al. (2018) that
aggregates information from the previous graph state and the latest set of
observations to generate the graph update, $g_{\delta}$. A graph convolutional
network (GCN) Schlichtkrull et al. (2018) and a BERT-based text encoder is
used to encode $G_{t-1}$ and the text-based dialogue history into their latent
representations $h_{G_{t-1}}$ and $h_{t-1}$ respectively. The RNN is modeled
by an LSTM Hochreiter and Schmidhuber (1997), which models an intermediate
hidden state, which is finally used by a multi-layer perceptron (MLP) to
generate the updated graph.

###  Incorporating Domain-Specific or Expert Knowledge

Neural dialogue management methods aim to be data-driven without manual
feature engineering. Nonetheless, encoding domain-specific or expert knowledge
as inductive biases can be advantageous, as it may accelerate learning or
reduce data dependency. In this section, I examine various approaches taken in
the selected works to incorporate domain-specific or expert knowledge into
neural dialogue managers.

####  5.4.1 Utterance-Level Labels

One straightforward method to encode domain knowledge into neural dialogue
managers is through the use of handcrafted taxonomies for utterance
classification. For instance, in a negotiation context, Zhou et al. (2019)
initially classify each dialogue utterance into one or more tactics from a
predefined set. This set of tactics, derived manually from behavioral science
research, includes tactics such as communicate politely, use hedge words, and
show dominance. A sequence model is trained on the sequence of these tactic
labels to determine the subsequent dialogue move. Similarly, Zhou et al.
(2020) transform utterances into a sequence of tactic and strategy labels for
another negotiation scenario.

####  5.4.2 Vector Featurization

Domain knowledge can also come into play while designing one-hot
representations preceding an embedding layer. Kim et al. (2019) develop a
neural dialogue model to act as a Teller agent that describes a clip-art
drawing to a Drawer through a natural language dialogue. From human-human
dialogues in the same environment, they observed that humans use compositional
language to describe clip-art configurations and attributes (e.g., "where is
trunk of second tree, low or high"). To help understand such utterances, each
clip-art piece in the scene is represented using a vector that is a sum of
learned embeddings for attributes such as type=Mike, size=small, etc.

Another example of domain-specific vector featurization is in developing an
Architect for the visual-language navigation setting described by Shi et al.
(2022). Each voxel in the 3D Minecraft environment is first encoded using a
one-hot vector that captures the voxel’s 3D position, its colors, and its
neighboring voxels. Along with that, their representation also includes the
location at which the Builder last made a move. This is motivated by the
observation that Builders often refer to their recent actions in their
utterances.

####  5.4.3 Response Ranking

Qiu et al. (2022b) train a dialogue agent to behave in a “socially
intelligent” way. Their agent is situated within a large-scale, crowd-sourced
fantasy text-adventure game called LIGHT Urbanek et al. (2019). In their work
on intercultural research, Schwartz (1992) identify a set of universal basic
human values that transcends cultures (see Figure 7), which Qiu et al. (2022b)
adopt in the response generation step of their model. Specifically, a learned
value function $f_{v}(\cdot)$ ($v\in\\{achievement,power,security,\dots\\})$
takes in an action (or utterance) and outputs a scalar value. The value
function is learned from a knowledge base of human values from the ValueNet
dataset Qiu et al. (2022a). During response generation, the agent first
narrows down the action space based on feasibility, and then ranks the
filtered actions based on the equation below (here, $p$ is a text-based
description of the agent’s persona that the agent is seeded with (elaborated
in Section 5.2.3)). The main purpose of doing the second step is to ensure
that the agent’s responses are consistent with its own value system.

$u(a_{i})=\sqrt{\sum_{v\in V}{(f_{v}(p)-f_{v}(a_{i}))^{2}}}$

Figure 7: Theory of basic human values used in Qiu et al. (2022b)’s socially
intelligent agent.

####  5.4.4 Reward Functions for RL

Another way to incorporate domain-specific knowledge is through the use of
handcrafted reward functions for reinforcement learning. In Cuayahuitl et al.
(2015), a dialogue agent is trained to engage in conversations within the
context of the Settlers of Catan game, focusing on resource trading. An
example trading offer might be, "I will give anyone sheep for clay." The
reward function is manually defined within the environment through a user
simulator. Specifically, the reward, $r$, is constructed as a piecewise
function that depends on the number of points obtained in the game. This
information is used to train a deep reinforcement learning model.

$r=\begin{cases}GainedPts\times w_{gp},&\text{if }GainedPts>0\\\
TotalPts\times w_{tp},&\text{otherwise}\end{cases}$

In the reward function above, $w_{gp}$ and $w_{tp}$ represent weighted
hyperparameters that are manually set by the authors. The function assigns a
reward based on the number of points gained during a specific action
(GainedPts) or the total points accumulated in the game (TotalPts), depending
on whether any points were gained during that action.

###  Theory-of-Mind Modeling

Theory-of-Mind (ToM) Premack and Woodruff (1978) is a concept in cognitive
science that refers to the ability to attribute mental states such as beliefs,
desires, intentions, and emotions to oneself and others, understanding that
these mental states may differ among individuals. The incorporation of a
Theory-of-Mind model into a collaborative dialogue manager is closely related
to the BDI (beliefs, desires, and intentions) framework Gratch (2000) used in
plan-based approaches.

This section examines the chosen works to detail how they incorporate ToM in
their dialogue management. Specifically, it discusses two things: 1) the
aspects of the mental state captured and 2) the methods used to collect labels
for training machine learning models that can predict theory-of-mind variables
from the dialogue context.

####  5.5.1 Dimensions of Mental State

Among the reviewed works, the aspects of mental states that are modeled
include both task-related (e.g., what a participant is currently working on or
believes the task state to be) as well as aspects related to a participant’s
personality and emotion Qiu et al. (2022b).

In Bara et al. (2021), the authors investigate a setting involving two
partners (Player A and Player B) collaborating to create a target material in
a Minecraft environment by mining and combining blocks. Owing to the asymmetry
in both their skills and knowledge, the participants engage in rich natural
language dialogue to develop a strong mental model of the task requirements,
their own task-state, their partner’s knowledge, and their partner’s task-
state. Specifically, the proposed ToM model in their paper seeks to capture 1)
a player’s own task status (whether a specific material has already been
created or not), 2) a player’s self-knowledge (whether a player knows how to
create a material or not), and 3) their partner’s current task (what a player
thinks their partner is working on at a given point in time).

The setting of dialogue in Qiu et al. (2022b) is a fantasy text-adventure
game, LIGHT Urbanek et al. (2019), in which agents can talk to other agents in
free-form text, take a closed set of actions, or express certain emotions. In
this setting, Urbanek et al. (2019) present a “socially-aware” response
generation model that explicitly models the mental state of the agent as a
time-varying graph. The graph is constructed (and updated) based on three
sources: 1) an initial persona description associated with the agent, 2) the
dialogue history, and 3) the game state. A sample graph from their model is
shown in Figure 8. The nodes correspond to natural language descriptions of
agents and objects in the world.

Figure 8: Graph-based mental state representation of the Servant agent in the
LIGHT fantasy text-adventure game (from Qiu et al. (2022b)).

Zhou et al. (2022) develop an agent that can play the role of a Dungeon Master
(DM) in a Dungeons-and-Dragons (D&D) game. The agent’s goal is to produce
utterances such that the remaining players are incentivized to perform a
specific action that the DM has in mind. This action, in turn, is designed to
make progress in the story that the DM has constructed. Therefore, the
specific aspect of the player’s mental model that is modeled by the DM agent
in this work is the following: How will the player react to my utterance?

####  5.5.2 Label Collection Methods for ToM Modeling

Collecting training labels is a crucial challenge when training machine
learning models for ToM modeling. In this section, I discuss three approaches
identified from the reviewed works.

##### Simulated Environment.

In the agent proposed by Qiu et al. (2022b), situated within a simulated game,
the dialogue agent has full visibility into the game state. This visibility
enables the construction and updating of the mental state representation
directly from the available information. The agent’s mental state is
represented as a graph, initialized by parsing the "seed" natural language
description of the agent’s persona. As the conversation progresses, the
dialogue history and game state are used to incrementally update this graph
representation.

##### Human-in-the-Loop.

In the setting proposed by Bara et al. (2021), dialogue agents operate within
a Minecraft world and collaborate with a partner to combine primitive blocks
into a target block. To collect labels for ToM modeling, they employ an online
human-in-the-loop data collection methodology, prompting users to reveal their
mental models during the interaction. A machine learning model is then trained
to predict the mental models of both partners based on dialogue and task
context. The proposed model utilizes either an LSTM or a Transformer to
process inputs from three different modalities: a video feed encoded with a
convolutional neural network, knowledge in the form of a graph encoded using a
gated recurrent unit (GRU) network Chung et al. (2014), and dialogue
utterances encoded with a BERT model. User-prompted questions at 90-second
intervals are also encoded and combined with the Transformer representation,
which is then fed into a feed-forward network predicting the partner’s ToM
attributes. This architecture is depicted in Figure 9.

Figure 9: Architecture of the ToM-prediction model proposed in Bara et al.
(2021).

##### Leveraging LLMs to Generate Labels.

Zhou et al. (2022) propose a method designed to circumvent the expensive
nature of human-in-the-loop approaches. The dialogue agent plays the role of a
Dungeon Master (DM) in a Dungeons and Dragons game, with the objective of
generating utterances so as to generate utterances that maximize the
likelihood of players making a specific move intended by the DM. The agent
predicts the players’ next moves to a given utterance using a set of language
models working together to generate affordable proxies for user responses. The
process consists of the following steps:

  1. 1.

Annotators label a small number of Dungeon Master’s (DM’s) utterances in a
collected corpus of interactions, identifying the specific span that
influenced the players’ actions.

  2. 2.

An "inverse dynamics model (IDM)" is trained over these labels to predict the
specific span that influenced the player’s actions, given their next action
(from the future).

  3. 3.

An intent generator is trained by first mining intents from the DM’s
utterances and then training another model over the mined intents.

  4. 4.

A player model is trained to predict the player’s next move based on the
dialogue history.

  5. 5.

Lastly, the DM model is trained using reinforcement learning (RL) to generate
utterances that lead to the player performing an action aligned with the DM’s
intent.

Both the IDM and the intent generator are trained by prompting GPT-3 Brown et
al. (2020) to 1) output excerpts from the DM’s utterance and 2) generate
natural language descriptions of intents, respectively.

##  6 Discussion

This section summarizes the findings from the literature review and discusses
open challenges and future research directions for neural collaborative
dialogue management.

###  General Trends

I begin by outlining some general trends uncovered through this literature
review.

##### Historical Trends.

Most of the works that laid the foundation for collaborative dialogue systems
(outlined in Section 4) came about in the period from the late-1980s to the
mid-2000s. Throughout the late-2000s and the 2010s, dialogue systems research
became predominantly focused on slot-filling assistants and open-domain
chatbots, resulting in limited advancements in both theoretical and practical
frameworks for collaborative agents. Similar observations about the research
trends have been echoed by notable researchers in the field Cohen (2019);
Papaioannou et al. (2018).

##### Amount of Research Activity.

The search for works proposing neural network-based techniques in constructing
collaborative agents yielded only a small number of results, despite the
flurry in research activity for neural network based techniques for task-
oriented and open-domain dialogue systems during the same time period. I
hypothesize three reasons for this disparity:

  1. 1.

The lack of straightforward automated evaluation methods in complex
collaborative dialogue settings (elaborated below in Section 6.2)

  2. 2.

Immediate commercial impetus for task-oriented and open-domain dialogue
provided by the proliferation of “virtual assistants” such as Amazon Alexa,
Apple Siri, and Google Home, and

  3. 3.

The lack of collaborative analogues of large benchmark corpora such as
MultiWOZ Budzianowski et al. (2018) and Schema-Guided Dialog Rastogi et al.
(2020); Mehri and Eskenazi (2021) as well as shared community challenges such
as DSTC Williams et al. (2014b), which were instrumental in catapulting
research activity in task-oriented dialogue systems.

###  Challenges to Holistic Evaluation

We see that in all the works identified under RQ 2, the tasks, domains,
environments, and objectives are carefully curated so as to facilitate
automated, quantitative evaluation. However, in this process, the authors are
often forced to make simplifying assumptions that might not be reflective of
the full complexity of the real-world. For example, a common simplifying
assumption made in Leader-Follower setups such as Kim et al. (2019); Chi et
al. (2020); Narayan-Chen et al. (2019, 2017) is that the Follower cannot
initiate dialogue, but can instead only receive instructions, or at best, ask
clarification questions.

As another example, Sanders et al. (2022) observe, in the context of a
persuasion dialogue for charity donation, that a strict definition of success
such as the amount of money donated by the latter may not accurately capture
the performance of the Persuader agent since the Persuadee’s decision can be
influenced by factors out of the Persuader’s control, such as the Persuadee
not having money to donate. In such a situation, holistic, automated
evaluation can be challenging. Such difficulties in evaluation is a general
trend among most collaborative dialogue settings excluding a few well-defined
cases such as certain types of negotiations, where the outcomes can be
captured quantitatively by the end result of the dialogue.

In general, the roadblock for holistic evaluation is that on one hand,
automated measures can only evaluate individual modules of a dialogue system
or individual aspects of what makes a successful collaborative dialogue. On
the other hand, complete end-to-end human evaluation is expensive. Existing
works mostly take the former, piecemeal approach to evaluating a dialogue
system, which, albeit useful, may not be fully capture the features that
stakeholders ultimately care about.

###  Combining the Strengths of Neural and Plan-Based Approaches

The works identified for RQ 2 reveal a clear trend of incorporating features
typically found in plan-based approaches to dialogue management. Specifically,
the modeling approaches proposed in these works, such as decoupling semantics
from surface-level representations (Section 5.1), moving towards human-
interpretable representations of dialogue state (Section 5.3), and
incorporating Theory-of-Mind models (Section 5.5), can all be mapped one-to-
one with constructs that are “natively available” in plan-based dialogue
managers (Section 4.2). Plan-based dialogue management techniques (Section
4.2) employ discrete action-spaces, can explain their decisions Shams et al.
(2016); Nothdurft et al. (2015a); Honold et al. (2014); Nothdurft et al.
(2015b), and maintain explicit notions of their partners’ beliefs through the
use of belief-desire-intention (BDI) models in their architectures Cohen and
Galescu (2023). Given this inherent applicability of plan-based dialogue
management for collaborative dialogue, exploring ways to combine the strengths
of both approaches presents a promising area for further research. Neural
approaches can learn directly from data and provide high representational
power and flexibility, while plan-based approaches are recognized for their
reliability, consistency, and dependability.

###  LLMs for Data Generation

Figure 10: Gilardi et al. (2023) shows that ChatGPT outperforms M-Turk
crowdworkers when it comes to inter-annotator agreement against a trained
annotator.

Deep learning has achieved remarkable success in various NLP tasks, including
question-answering, text classification, and natural language generation.
Nevertheless, dialogue management remains an unresolved problem, as collecting
extensive data for new domains is often exceptionally difficult and
impractical. Although reinforcement learning-based approaches can learn from
interactions without a large corpus to begin with, deploying sub-optimal
dialogue policies in sensitive areas such as medicine, aviation, and education
may lead to detrimental or even fatal consequences. However, the potential of
large language models (LLMs) could transform this situation.

Traditionally, gathering training data for neural dialogue models in specific
domains involves either using a small number of in-house researchers as
annotators (e.g., graduate students) or enlisting larger groups of individuals
through crowdsourcing platforms like Amazon M-Turk. Both methods demand
substantial effort and incur significant costs. This issue becomes even more
pronounced when deep domain expertise is required for annotation, as
crowdsourced annotations from the general populace may not be reliable.
Recently, Gilardi et al. (2023) demonstrated that instruction-tuned LLMs, such
as ChatGPT, outperform crowdworkers in annotation quality for several NLP
tasks. Figure 10 compares ChatGPT and M-Turk crowdworkers in terms of inter-
annotator agreement with trained annotators. Advancements in harnessing large
language models for annotating data, creating new datasets, and augmenting
existing ones are promising directions to help address this challenge.

###  Absence of a Software Ecosystem

Finally, despite over four decades of research, the software landscape for
practical authoring tools and development frameworks for developing
collaborative dialogue systems is still largely limited to research prototypes
and narrow domain-specific techniques. Developing general-purpose tools for
system designers to quickly prototype an agent in a completely new domain
continues to be non-trivial even today. This situation is in striking contrast
to the current landscape for form-filling and open-domain assistants, where
widely accessible commercial tools such as Google
DialogFlow666https://cloud.google.com/dialogflow, Amazon
Lex777https://aws.amazon.com/lex/, Rasa888https://rasa.com/,
LUIS999https://www.luis.ai/, and ParlAI101010https://parl.ai/ empower
individuals without expertise in dialogue systems or natural language
processing to create agents suitable for a vast range of scenarios.

Schema-driven dialogue system design Kane et al. (2022); Mehri and Eskenazi
(2021); Rastogi et al. (2020) and prompt-based techniques Kaplan et al.
(2020); Wei et al. (2022) represent two promising classes of interfaces for
dialogue system authoring. However, to determine the optimal type of interface
for a specific user and domain, further human-centered research is needed,
including user studies that encompass diverse user groups and various dialogue
systems.

##  7 Summary

This paper begins by reviewing foundational paradigms for developing
collaborative dialogue agents that have emerged over the past four decades of
dialogue systems research. Each dialogue paradigm is described, and the
applications in handling collaborative dialogues are illustrated. The focus
then shifts to data-driven, neural approaches for collaborative dialogue
management that require minimal handcrafting. The limitations of "vanilla"
end-to-end neural networks and slot-based methods for dialogue state
representation are discussed, serving as motivation for reviewing neural
modeling approaches for collaborative dialogue. From a selection of 17 works,
this review identifies five core modeling themes and examines the
implementation of each theme by the respective studies in specific domains.
Lastly, the overarching trends are analyzed, and the paper concludes by
presenting a set of open challenges and future research directions for the
field of collaborative dialogue management.

## Acknowledgement

I would like to thank Dr. Kristy Elizabeth Boyer and Dr. Bonnie Dorr for their
invaluable advice. I would like to also thank Dr. Anthony Botelho for his
helpful inputs during the initial stages of organizing this literature review.

## References

  * Ai-Chang et al. (2004) Mitchell Ai-Chang, John Bresina, Len Charest, Adam Chase, JC-J Hsu, Ari Jonsson, Bob Kanefsky, Paul Morris, Kanna Rajan, Jeffrey Yglesias, et al. 2004\.  Mapgen: mixed-initiative planning and scheduling for the mars exploration rover mission.  _IEEE Intelligent Systems_ , 19(1):8–12.
  * Allen (1983) James Allen. 1983.  Recognizing intention from natural language utterances.  _Computational Model of Discourse_ , pages 107–166.  Publisher: MIT press.
  * Allen et al. (2002) James Allen, Nate Blaylock, and George Ferguson. 2002.  A problem solving model for collaborative agents.  In _Proceedings of the first international joint conference on autonomous agents and multiagent systems: Part 2_ , AAMAS ’02, pages 774–781, New York, NY, USA. Association for Computing Machinery.  Number of pages: 8 Place: Bologna, Italy.
  * Allen et al. (2005) James Allen, George Ferguson, Amanda Stent, Scott C Stoness, Mary Swift, Lucian Galescu, Nathanael Chambers, Ellen Campana, et al. 2005.  Two diverse systems built using generic components for spoken dialogue (recent progress on trips).  In _Proceedings of the ACL Interactive Poster and Demonstration Sessions_ , pages 85–88.
  * Allen et al. (1995) James F Allen, Lenhart K Schubert, George Ferguson, Peter Heeman, Chung Hee Hwang, Tsuneaki Kato, Marc Light, Nathaniel Martin, Bradford Miller, Massimo Poesio, et al. 1995.  The trains project: A case study in building a conversational planning agent.  _Journal of Experimental & Theoretical Artificial Intelligence_, 7(1):7–48.
  * Andreas et al. (2020) Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020.  Task-oriented dialogue as dataflow synthesis.  _Transactions of the Association for Computational Linguistics_ , 8:556–571.  Place: Cambridge, MA Publisher: MIT Press.
  * Bara et al. (2021) Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai. 2021.  MindCraft: Theory of Mind Modeling for Situated Dialogue in Collaborative Tasks.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1112–1125, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Bennacef et al. (1996) Samir Bennacef, Laurence Devillers, Sophie Rosset, and Lori Lamel. 1996.  Dialog in the RAILTEL telephone-based system.  In _Proceeding of fourth international conference on spoken language processing. ICSLP’96_ , volume 1, pages 550–553.  Tex.organization: IEEE.
  * Bohus and Rudnicky (2009) Dan Bohus and Alexander I. Rudnicky. 2009.  The RavenClaw dialog management framework: Architecture and systems.  _Computer Speech & Language_, 23(3):332–361.
  * Bordes et al. (2017) Antoine Bordes, Y.-Lan Boureau, and Jason Weston. 2017.  Learning End-to-End Goal-Oriented Dialog.  ArXiv:1605.07683 [cs].
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and others. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Inigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  Multiwoz–a large-scale multi-domain wizard-of-oz dataset for task-oriented dialogue modelling.  _arXiv preprint arXiv:1810.00278_.
  * Cao et al. (2022) Meng Cao, Yue Dong, and Jackie Chi Kit Cheung. 2022.  Hallucinated but factual! inspecting the factuality of hallucinations in abstractive summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3340–3354.
  * Chi et al. (2020) Ta-Chung Chi, Minmin Shen, Mihail Eric, Seokhwan Kim, and Dilek Hakkani-tur. 2020\.  Just Ask: An Interactive Learning Framework for Vision and Language Navigation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(03):2459–2466.  Number: 03.
  * Chung et al. (2014) Junyoung Chung, Çaglar Gülçehre, KyungHyun Cho, and Yoshua Bengio. 2014.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  _CoRR_ , abs/1412.3555.  ArXiv: 1412.3555 tex.bibsource: dblp computer science bibliography, https://dblp.org tex.biburl: https://dblp.org/rec/journals/corr/ChungGCB14.bib tex.timestamp: Mon, 13 Aug 2018 16:47:38 +0200.
  * Clark and Brennan (1991) Herbert H. Clark and Susan Brennan. 1991.  Grounding in communication.  In _Perspectives on socially shared cognition_.
  * Clark and Wilkes-Gibbs (1986) Herbert H Clark and Deanna Wilkes-Gibbs. 1986.  Referring as a collaborative process.  _Cognition_ , 22(1):1–39.  Publisher: Elsevier.
  * Cohen (2019) Philip Cohen. 2019.  Foundations of Collaborative Task-Oriented Dialogue: What’s in a Slot? In _Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 198–209, Stockholm, Sweden. Association for Computational Linguistics.
  * Cohen and Galescu (2023) Philip R. Cohen and Lucian Galescu. 2023.  A Planning-Based Explainable Collaborative Dialogue System.  ArXiv:2302.09646 [cs].
  * Dahlbäck et al. (1999) Nils Dahlbäck, Annika Flycht-Eriksson, Arne Jönsson, and Pernilla Qvarfordt. 1999\.  An architechture for multi-modal natural dialogue systems.  In _ESCA tutorial and research workshop (ETRW) on interactive dialogue in multi-modal systems_.
  * de la Asunción et al. (2005) Marc de la Asunción, Luis Castillo, Juan Fdez-Olivares, Óscar García-Pérez, Antonio González, and Francisco Palao. 2005.  Siadex: An interactive knowledge-based planner for decision support in forest fire fighting.  _Ai Communications_ , 18(4):257–268.
  * de Vries et al. (2018) Harm de Vries, Kurt Shuster, Dhruv Batra, Devi Parikh, Jason Weston, and Douwe Kiela. 2018.  Talk the Walk: Navigating New York City through Grounded Dialogue.  ArXiv:1807.03367 [cs].
  * Dählback and Jonsson (1989) Nils Dählback and Arne Jonsson. 1989.  Empirical studies of discourse representations for natural language interfaces.  In _Fourth conference of the European chapter of the association for computational linguistics_ , Manchester, England. Association for Computational Linguistics.
  * Evens et al. (1997) Martha W. Evens, Ru-Charn Chang, Yoon Hee Lee, Leem Seop Shim, Chong Woo Woo, and Yuemei Zbang. 1997.  CIRCSIM-Tutor: An Intelligent Tutoring System Using Natural Language Dialogue.  In _Fifth Conference on Applied Natural Language Processing: Descriptions of System Demonstrations and Videos_ , pages 13–14, Washington, DC, USA. Association for Computational Linguistics.
  * Freedman (2000) Reva Freedman. 2000.  Plan-Based Dialogue Management in a Physics Tutor.  In _Sixth Applied Natural Language Processing Conference_ , pages 52–59, Seattle, Washington, USA. Association for Computational Linguistics.
  * Galescu et al. (2018) Lucian Galescu, Choh Man Teng, James Allen, and Ian Perera. 2018.  Cogent: A Generic Dialogue System Shell Based on a Collaborative Problem Solving Model.  In _Proceedings of the 19th Annual SIGdial Meeting on Discourse and Dialogue_ , pages 400–409, Melbourne, Australia. Association for Computational Linguistics.
  * Ghallab et al. (2004) Malik Ghallab, Dana Nau, and Paolo Traverso. 2004.  _Automated Planning: theory and practice_.  Elsevier.
  * Gilardi et al. (2023) Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023.  Chatgpt outperforms crowd-workers for text-annotation tasks.  _arXiv preprint arXiv:2303.15056_.
  * Graesser et al. (2004) Arthur C. Graesser, Shulan Lu, George Tanner Jackson, Heather Hite Mitchell, Mathew Ventura, Andrew Olney, and Max M. Louwerse. 2004.  AutoTutor: A tutor with dialogue in natural language.  _Behavior Research Methods, Instruments, and Computers_ , 36(2):180–192.
  * Gratch (2000) Jonathan Gratch. 2000.  Emile: Marshalling passions in training and education.  In _Proceedings of the fourth international conference on Autonomous agents_ , pages 325–332.
  * Griffith et al. (2022) Amanda E. Griffith, Gloria Ashiya Katuka, Joseph B. Wiggins, Kristy Elizabeth Boyer, Jason Freeman, Brian Magerko, and Tom McKlin. 2022.  Investigating the Relationship Between Dialogue States and Partner Satisfaction During Co-Creative Learning Tasks.  _International Journal of Artificial Intelligence in Education_.
  * He et al. (2017) He He, Anusha Balakrishnan, Mihail Eric, and Percy Liang. 2017.  Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings.  ArXiv:1704.07130 [cs].
  * He et al. (2018) He He, Derek Chen, Anusha Balakrishnan, and Percy Liang. 2018.  Decoupling strategy and generation in negotiation dialogues.  In _Proceedings of the 2018 conference on empirical methods in natural language processing_ , pages 2333–2343, Brussels, Belgium. Association for Computational Linguistics.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Honold et al. (2014) Frank Honold, Pascal Bercher, Felix Richter, Florian Nothdurft, Thomas Geier, Roland Barth, Thilo Hörnle, Felix Schüssel, Stephan Reuter, Matthias Rau, Gregor Bertrand, Bastian Seegebarth, Peter Kurzok, Bernd Schattenberg, Wolfgang Minker, Michael Weber, and Susanne Biundo. 2014.  Companion-technology: Towards user- and situation-adaptive functionality of technical systems.  In _Proceedings of the 2014 International Conference on Intelligent Environments_ , IE ’14, page 378–381, USA. IEEE Computer Society.
  * Huang et al. (2020) Minlie Huang, Xiaoyan Zhu, and Jianfeng Gao. 2020.  Challenges in building intelligent open-domain dialog systems.  _ACM Transactions on Information Systems (TOIS)_ , 38(3):1–32.
  * Jayannavar et al. (2020) Prashant Jayannavar, Anjali Narayan-Chen, and Julia Hockenmaier. 2020.  Learning to execute instructions in a Minecraft dialogue.  In _Proceedings of the 58th annual meeting of the association for computational linguistics_ , pages 2589–2602, Online. Association for Computational Linguistics.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.  Publisher: ACM New York, NY.
  * Jordan et al. (2006) Pamela Jordan, Michael Ringenberg, and Brian Hall. 2006.  Rapidly developing dialogue systems that support learning studies.  In _Proceedings of ITS06 Workshop on Teaching with Robots, Agents, and NLP_ , pages 1–8.
  * Kane et al. (2022) Benjamin Kane, Catherine Giugno, Lenhart Schubert, Kurtis Haut, Caleb Wohn, and Ehsan Hoque. 2022.  A Flexible Schema-Guided Dialogue Management Framework: From Friendly Peer to Virtual Standardized Cancer Patient.  ArXiv:2207.07276 [cs].
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.  Scaling laws for neural language models.  _CoRR_ , abs/2001.08361.
  * Kim et al. (2019) Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh. 2019.  CoDraw: Collaborative drawing as a testbed for grounded goal-driven communication.  In _Proceedings of the 57th annual meeting of the association for computational linguistics_ , pages 6495–6513, Florence, Italy. Association for Computational Linguistics.
  * Konda and Tsitsiklis (1999) Vijay Konda and John Tsitsiklis. 1999.  Actor-Critic Algorithms.  In _Advances in Neural Information Processing Systems_ , volume 12. MIT Press.
  * Kumar and Rosé (2011) Rohit Kumar and Carolyn P. Rosé. 2011.  Architecture for Building Conversational Agents that Support Collaborative Learning.  _IEEE Transactions on Learning Technologies_ , 4(1):21–34.  Conference Name: IEEE Transactions on Learning Technologies.
  * Larsson and Traum (2000) Staffan Larsson and David R Traum. 2000.  Information state and dialogue management in the TRINDI dialogue move engine toolkit.  _Natural language engineering_ , 6(3-4):323–340.  Publisher: Cambridge University Press.
  * Li et al. (2016) Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016.  Deep Reinforcement Learning for Dialogue Generation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1192–1202, Austin, Texas. Association for Computational Linguistics.
  * Li et al. (2019) Yu Li, Kun Qian, Weiyan Shi, and Zhou Yu. 2019.  End-to-End Trainable Non-Collaborative Dialog System.  ArXiv:1911.10742 [cs].
  * Liu et al. (2022) Guangya Liu, Shiqi Wang, Jianxing Yu, and Jian Yin. 2022.  A survey on multimodal dialogue systems: Recent advances and new frontiers.  In _2022 5th international conference on advanced electronic materials, computers and software engineering (AEMCSE)_ , pages 845–853.  Tex.organization: IEEE.
  * Louvan and Magnini (2020) Samuel Louvan and Bernardo Magnini. 2020.  Recent neural methods on slot filling and intent classification for task-oriented dialogue systems: A survey.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 480–496, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Lukin et al. (2018) Stephanie M. Lukin, Felix Gervits, Cory J. Hayes, Pooja Moolchandani, Anton Leuski, John G. Rogers III, Carlos Sanchez Amaro, Matthew Marge, Clare R. Voss, and David Traum. 2018.  ScoutBot: A dialogue system for collaborative navigation.  In _Proceedings of ACL 2018, system demonstrations_ , pages 93–98, Melbourne, Australia. Association for Computational Linguistics.
  * Mehri and Eskenazi (2021) Shikib Mehri and Maxine Eskenazi. 2021.  Schema-Guided Paradigm for Zero-Shot Dialog.  In _Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 499–508, Singapore and Online. Association for Computational Linguistics.
  * Mrkšić et al. (2016) Nikola Mrkšić, Diarmuid O Séaghdha, Tsung-Hsien Wen, Blaise Thomson, and Steve Young. 2016.  Neural belief tracker: Data-driven dialogue state tracking.  _arXiv preprint arXiv:1606.03777_.
  * Myers et al. (2002) Karen L Myers, W Mabry Tyson, Michael J Wolverton, Peter A Jarvis, Thomas J Lee, and Marie desJardins. 2002.  Passat: A user-centric planning framework.  In _Proceedings of the 3rd International NASA Workshop on Planning and Scheduling for Space_ , pages 1–10. AAAI Press Stanford, California.
  * Narayan-Chen et al. (2017) Anjali Narayan-Chen, Colin Graber, Mayukh Das, Md Rakibul Islam, Soham Dan, Sriraam Natarajan, Janardhan Rao Doppa, Julia Hockenmaier, Martha Palmer, and Dan Roth. 2017.  Towards Problem Solving Agents that Communicate and Learn.  In _Proceedings of the First Workshop on Language Grounding for Robotics_ , pages 95–103, Vancouver, Canada. Association for Computational Linguistics.
  * Narayan-Chen et al. (2019) Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. 2019.  Collaborative Dialogue in Minecraft.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5405–5415, Florence, Italy. Association for Computational Linguistics.
  * Nothdurft et al. (2015a) Flo rian Nothdurft, Stefan Ultes, and Wolfgang Minker. 2015a.  Finding appropriate interaction strategies for proactive dialogue systems—an open quest.  pages 73–80.
  * Nothdurft et al. (2015b) Florian Nothdurft, Gregor Behnke, Pascal Bercher, Susanne Biundo, and Wolfgang Minker. 2015b.  The interplay of user-centered dialog systems and AI planning.  In _Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 344–353, Prague, Czech Republic. Association for Computational Linguistics.
  * Papaioannou et al. (2018) Ioannis Papaioannou, Christian Dondrup, and Oliver Lemon. 2018.  Human-robot interaction requires more than slot filling-multi-threaded dialogue for collaborative tasks and social conversation.  In _FAIM/ISCA Workshop on Artificial Intelligence for Multimodal Human Robot Interaction_ , pages 61–64.
  * Park and Kim (2023) Sang-Min Park and Young-Gab Kim. 2023.  Visual language navigation: A survey and open challenges.  _Artificial Intelligence Review_ , 56(1):365–427.  Publisher: Springer.
  * Pollack (1992) Martha E Pollack. 1992.  The uses of plans.  _Artificial Intelligence_ , 57(1):43–68.  Publisher: Citeseer.
  * Premack and Woodruff (1978) David Premack and Guy Woodruff. 1978.  Does the chimpanzee have a theory of mind?  _Behavioral and brain sciences_ , 1(4):515–526.  Publisher: Cambridge University Press.
  * Qiu et al. (2022a) Liang Qiu, Yizhou Zhao, Jinchao Li, Pan Lu, Baolin Peng, Jianfeng Gao, and Song-Chun Zhu. 2022a.  Valuenet: A new dataset for human value driven dialogue system.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, pages 11183–11191.
  * Qiu et al. (2022b) Liang Qiu, Yizhou Zhao, Yuan Liang, Pan Lu, Weiyan Shi, Zhou Yu, and Song-Chun Zhu. 2022b.  Towards Socially Intelligent Agents with Mental State Transition and Human Value.  In _Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 146–158, Edinburgh, UK. Association for Computational Linguistics.
  * Rao and Georgeff (1991) Anand S. Rao and Michael P. Georgeff. 1991.  Modeling rational agents within a bdi-architecture.  In _Proceedings of the Second International Conference on Principles of Knowledge Representation and Reasoning_ , KR’91, page 473–484, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
  * Rastogi et al. (2020) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020.  Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):8689–8696.  Number: 05.
  * Ren et al. (2018) Liliang Ren, Kaige Xie, Lu Chen, and Kai Yu. 2018.  Towards universal dialogue state tracking.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2780–2786, Brussels, Belgium. Association for Computational Linguistics.
  * Rich et al. (2001) Charles Rich, Candace L. Sidner, and Neal Lesh. 2001.  Collagen: Applying Collaborative Discourse Theory to Human-Computer Interaction.  _AI Magazine_ , 22(4):15–15.  Number: 4.
  * Roman et al. (2020) Homero Roman Roman, Yonatan Bisk, Jesse Thomason, Asli Celikyilmaz, and Jianfeng Gao. 2020.  RMM: A Recursive Mental Model for Dialog Navigation.  ArXiv:2005.00728 [cs].
  * Sanders et al. (2022) Abraham Sanders, Tomek Strzalkowski, Mei Si, Albert Chang, Deepanshu Dey, Jonas Braasch, and Dakuo Wang. 2022.  Towards a Progression-Aware Autonomous Dialogue Agent.  ArXiv:2205.03692 [cs].
  * Santhanam et al. (2020) Sashank Santhanam, Zhuo Cheng, Brodie Mather, Bonnie Dorr, Archna Bhatia, Bryanna Hebenstreit, Alan Zemel, Adam Dalton, Tomek Strzalkowski, and Samira Shaikh. 2020.  Learning to Plan and Realize Separately for Open-Ended Dialogue Systems.  _arXiv:2009.12506 [cs]_.  ArXiv: 2009.12506.
  * Santhanam and Shaikh (2019) Sashank Santhanam and Samira Shaikh. 2019.  A survey of natural language generation techniques with a focus on dialogue systems-past, present and future directions.  _arXiv preprint arXiv:1906.00500_.
  * Schlichtkrull et al. (2018) Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018.  Modeling relational data with graph convolutional networks.  In _The Semantic Web: 15th International Conference, ESWC 2018, Heraklion, Crete, Greece, June 3–7, 2018, Proceedings 15_ , pages 593–607. Springer.
  * Schwartz (1992) Shalom H Schwartz. 1992.  Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries.  In _Advances in experimental social psychology_ , volume 25, pages 1–65. Elsevier.
  * Searle (1969) John R. Searle. 1969.  _Speech acts: An essay in the philosophy of language_.  Cambridge University Press, Cambridge.
  * Serban et al. (2016a) Iulian Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2016a.  Building end-to-end dialogue systems using generative hierarchical neural network models.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 30.  Number: 1.
  * Serban et al. (2016b) Iulian Vlad Serban, Alessandro Sordoni, Ryan Lowe, Laurent Charlin, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. 2016b.  A hierarchical latent variable encoder-decoder model for generating dialogues.  _CoRR_ , abs/1605.06069.
  * Shah et al. (2019) Darsh J Shah, Raghav Gupta, Amir A Fayazi, and Dilek Hakkani-Tur. 2019.  Robust zero-shot cross-domain slot filling with example values.  _arXiv preprint arXiv:1906.06870_.
  * Shams et al. (2016) Zohreh Shams, Marina De Vos, Nir Oren, and Julian Padget. 2016.  Normative practical reasoning via argumentation and dialogue.  In _Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence_ , IJCAI’16, page 1244–1250. AAAI Press.
  * Shi et al. (2022) Zhengxiang Shi, Yue Feng, and Aldo Lipani. 2022.  Learning to Execute Actions or Ask Clarification Questions.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 2060–2070, Seattle, United States. Association for Computational Linguistics.
  * Shinagawa et al. (2020) Seitaro Shinagawa, Koichiro Yoshino, Seyed Hossein Alavi, Kallirroi Georgila, David Traum, Sakriani Sakti, and Satoshi Nakamura. 2020.  An Interactive Image Editing System Using an Uncertainty-Based Confirmation Strategy.  _IEEE Access_ , 8:98471–98480.  Conference Name: IEEE Access.
  * Shum et al. (2018) Heung-Yeung Shum, Xiao-dong He, and Di Li. 2018.  From Eliza to XiaoIce: challenges and opportunities with social chatbots.  _Frontiers of Information Technology & Electronic Engineering_, 19:10–26.  Publisher: Springer.
  * Streeck et al. (2011) Jürgen Streeck, Charles Goodwin, and Curtis LeBaron. 2011.  _Embodied interaction: Language and body in the material world_.  Cambridge University Press.
  * Strijbos (2004) Jan-Willem Strijbos. 2004.  The effect of roles on computer-supported collaborative learning.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  _Advances in neural information processing systems_ , 27.
  * Traum (2017) David Traum. 2017.  Computational Approaches to Dialogue.  In Weigand Edda, editor, _The Routledge Handbook of Language and Dialogue_ , 1 edition, pages 143–161. Routledge, New York, NY : Routledge, [2017] | Series: Routledge Handbooks in Linguistics.
  * Traum et al. (1999) David Traum, Johan Bos, Robin Cooper, Staffan Larsson, Ian Lewin, Colin Matheson, and Massimo Poesio. 1999.  A model of dialogue moves and information state revision.
  * Traum et al. (2008) David Traum, Stacy C. Marsella, Jonathan Gratch, Jina Lee, and Arno Hartholt. 2008\.  Multi-party, Multi-issue, Multi-strategy Negotiation for Multi-modal Virtual Agents.  In _Intelligent Virtual Agents_ , Lecture Notes in Computer Science, pages 117–130, Berlin, Heidelberg. Springer.
  * Urbanek et al. (2019) Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktäschel, Douwe Kiela, Arthur Szlam, and Jason Weston. 2019\.  Learning to speak and act in a fantasy text adventure game.  In _Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP)_ , pages 673–683, Hong Kong, China. Association for Computational Linguistics.
  * VanLehn et al. (2002) Kurt VanLehn, Pamela W. Jordan, Carolyn P. Rosé, Dumisizwe Bhembe, Michael Böttner, Andy Gaydos, Maxim Makatchev, Umarani Pappuswamy, Michael Ringenberg, Antonio Roque, Stephanie Siler, and Ramesh Srivastava. 2002.  The Architecture of Why2-Atlas: A Coach for Qualitative Physics Essay Writing.  In _Intelligent Tutoring Systems_ , Lecture Notes in Computer Science, pages 158–167, Berlin, Heidelberg. Springer.
  * Vannaprathip et al. (2022) Narumol Vannaprathip, Peter Haddawy, Holger Schultheis, and Siriwan Suebnukarn. 2022\.  Intelligent Tutoring for Surgical Decision Making: a Planning-Based Approach.  _International Journal of Artificial Intelligence in Education_ , 32(2):350–381.
  * Walker et al. (2022) Nicholas Thomas Walker, Stefan Ultes, and Pierre Lison. 2022.  GraphWOZ: Dialogue Management with Conversational Knowledge Graphs.  ArXiv:2211.12852 [cs].
  * Wang et al. (2019) Xuewei Wang, Weiyan Shi, Richard Kim, Yoojung Oh, Sijia Yang, Jingwen Zhang, and Zhou Yu. 2019.  Persuasion for good: Towards a personalized persuasive dialogue system for social good.  In _Proceedings of the 57th annual meeting of the association for computational linguistics_ , pages 5635–5649, Florence, Italy. Association for Computational Linguistics.
  * Webb and Farivar (1999) Noreen M. Webb and Sydney Farivar. 1999.  Developing productive group interaction in middle school mathematics.  In _Cognitive perspectives on peer learning_ , The Rutgers Invitational Symposium On Education Series, pages 117–149. Lawrence Erlbaum Associates Publishers, Mahwah, NJ, US.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022\.  Emergent abilities of large language models.  _arXiv preprint arXiv:2206.07682_.
  * Weizenbaum (1966) Joseph Weizenbaum. 1966.  ELIZA—a computer program for the study of natural language communication between man and machine.  _Communications of the ACM_ , 9(1):36–45.  Publisher: ACM New York, NY, USA.
  * Wen et al. (2015) Tsung-Hsien Wen, Milica Gasic, Nikola Mrksic, Pei-Hao Su, David Vandyke, and Steve Young. 2015.  Semantically conditioned lstm-based natural language generation for spoken dialogue systems.  _arXiv preprint arXiv:1508.01745_.
  * Wen et al. (2017) Tsung-Hsien Wen, David Vandyke, Nikola Mrksic, Milica Gasic, Lina M. Rojas-Barahona, Pei-Hao Su, Stefan Ultes, and Steve Young. 2017.  A Network-based End-to-End Trainable Task-oriented Dialogue System.  ArXiv:1604.04562 [cs, stat].
  * Weston et al. (2018) Jason Weston, Emily Dinan, and Alexander Miller. 2018.  Retrieve and refine: Improved sequence generation models for dialogue.  In _Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI_ , pages 87–92, Brussels, Belgium. Association for Computational Linguistics.
  * Williams et al. (2014a) Jason D Williams, Matthew Henderson, Antoine Raux, Blaise Thomson, Alan Black, and Deepak Ramachandran. 2014a.  The dialog state tracking challenge series.  _AI Magazine_ , 35(4):121–124.
  * Williams et al. (2014b) Jason D. Williams, Matthew Henderson, Antoine Raux, Blaise Thomson, Alan Black, and Deepak Ramachandran. 2014b.  The Dialog State Tracking Challenge Series.  _AI Magazine_ , 35(4):121–124.  Number: 4.
  * Wolf et al. (2019) Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. 2019.  TransferTransfo: A transfer learning approach for neural network based conversational agents.
  * Wollny et al. (2021) Sebastian Wollny, Jan Schneider, Daniele Di Mitri, Joshua Weidlich, Marc Rittberger, and Hendrik Drachsler. 2021.  Are We There Yet? - A Systematic Literature Review on Chatbots in Education.  _Frontiers in Artificial Intelligence_ , 4:654924.
  * Yarats and Lewis (2017) Denis Yarats and Mike Lewis. 2017.  Hierarchical text generation and planning for strategic dialogue.  In _International conference on machine learning_.
  * Yu et al. (2018) Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. 2018.  Qanet: Combining local convolution with global self-attention for reading comprehension.  _arXiv preprint arXiv:1804.09541_.
  * Zhou et al. (2022) Pei Zhou, Andrew Zhu, Jennifer Hu, Jay Pujara, Xiang Ren, Chris Callison-Burch, Yejin Choi, and Prithviraj Ammanabrolu. 2022.  An AI Dungeon Master’s Guide: Learning to Converse and Guide with Intents and Theory-of-Mind in Dungeons and Dragons.  ArXiv:2212.10060 [cs].
  * Zhou et al. (2019) Yiheng Zhou, He He, Alan W Black, and Yulia Tsvetkov. 2019.  A dynamic strategy coach for effective negotiation.  In _Proceedings of the 20th annual SIGdial meeting on discourse and dialogue_ , pages 367–378, Stockholm, Sweden. Association for Computational Linguistics.
  * Zhou et al. (2020) Yiheng Zhou, Yulia Tsvetkov, Alan W. Black, and Zhou Yu. 2020.  Augmenting Non-Collaborative Dialog Systems with Explicit Semantic and Strategic Dialog History.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.