# LLM-Based Open-Domain Integrated Task and Knowledge Assistants
with Programmable Policies

Harshit Joshi  Shicheng Liu  James Chen  Robert Weigle  Monica S. Lam
Computer Science Department, Stanford University
Stanford CA
{harshitj, shicheng<EMAIL_ADDRESS>
###### Abstract

Programming LLM-based knowledge and task assistants that faithfully conform to
developer-provided policies is challenging. These agents must retrieve and
provide consistent, accurate, and relevant information to address user’s
queries and needs. Yet such agents generate unfounded responses
("hallucinate"). Traditional dialogue trees can only handle a limited number
of conversation flows, making them inherently brittle. To this end, we present
KITA – a programmable framework for creating task-oriented conversational
agents that are designed to handle complex user interactions. Unlike LLMs,
KITA provides reliable grounded responses, with controllable agent policies
through its expressive specification, KITA Worksheet. In contrast to dialog
trees, it is resilient to diverse user queries, helpful with knowledge
sources, and offers ease of programming policies through its declarative
paradigm. Through a real-user study involving 62 participants, we show that
KITA beats the GPT-4 with function calling baseline by 26.1, 22.5, and 52.4
points on execution accuracy, dialogue act accuracy, and goal completion rate,
respectively. We also release 22 real-user conversations with KITA manually
corrected to ensure accuracy.

LLM-Based Open-Domain Integrated Task and Knowledge Assistants
with Programmable Policies

Harshit Joshi Shicheng Liu James Chen Robert Weigle Monica S. Lam Computer
Science Department, Stanford University Stanford CA {harshitj, shicheng,
<EMAIL_ADDRESS>

Figure 1: Overview of KITA. The user’s utterance contains tasks and knowledge-
related queries, requiring one to fill in the parameters of BookRestaurant API
and make a query to the database. KITA’s semantic parser translates the user
utterance into a composition of task API call and database query, and
generates a new dialogue state. Using the programmable policies, the agent
policy finally proposes dialogue acts, which form the response to the user.

##  1 Introduction

Task-oriented dialog (TOD) agents that can interact with users to achieve some
goal have many diverse applications, including customer support, financial
services, and medical form-filling, and are of great interest to researchers
and industry practitioners Budzianowski et al. (2018); Amazon (2023); Google
(2024). Helpful TOD agents must understand the user’s requests, guide the
conversations, and provide relevant information to users, with the help of
different data sources or APIs.

Recent research has successfully leveraged the In-Context Learning (ICL)
ability of Large Language Models (LLMs) to develop zero-shot and few-shot
task-oriented dialogue agents, achieving superior results to fine-tuning
pretrained models Hudeček and Dušek (2023); Zhang et al. (2023a). Furthermore,
the function-calling capability of LLMs now allows them to invoke provided
functions and use their outputs to assist in task completion Patil et al.
(2023); Kim et al. (2023). However, despite these strengths, LLM’s
capabilities present critical limitations that hinder their application in
task-oriented systems. Notably, they are unreliable and prone to generating
misleading information, or “hallucinations”, often using convincing and
confident language Bang et al. (2023). Moreover, even with function-calling,
LLMs offer minimal control to TOD agent developers over the dialogue flow,
making it challenging to manage interactions with users. Although complex
instructions can be added to the prompts, as instructions accumulate, LLMs
struggle to maintain consistent attention across all directives, often
neglecting some Liu et al. (2024a), resulting in unreliable behavior.

While dialogue trees can offer explicit control and reliability over the
conversations for task-oriented tasks, they are inherently brittle. Dialogue
flows must be explicitly hard-coded, making it infeasible to cover the all the
exponentially many possible dialogues. Furthermore, dialogue trees are very
limited in addressing subsequent user inquiries, as their rigid framework
impedes the integration of diverse data sources and the utilization of
knowledge assistants. This makes using dialogue trees for unified tasks and
knowledge assistants exceptionally challenging.

To this end, we present KITA, an open-domain LLM-based Knowledge Integrated
with Tasks Assistant. We propose a novel specification, KITA Worksheet that
provides explicit control to the TOD agent developer through programmable
policies to manage dialogue flow and deliver high-level support for integrated
knowledge assistants. KITA Worksheet enables programming policies in the
declarative paradigm, in contrast to the much more complex imperative approach
employed by dialogue trees. KITA Worksheets can express a variety of real-
world tasks, ranging from restaurant booking assistants that require hybrid
knowledge retrieval capabilities to handling complex workflows as encapsulated
in web pages.

Moreover, KITA can facilitate the composition of tasks and knowledge queries.
As illustrated in Figure 1, the user asks to book an Italian restaurant in NYC
for two on Valentine’s Day. Our agent uses a few-shot LLM-based semantic
parser to understand users’ queries and translate them into a composition of
API invocation (BookRestaurant) and knowledge query (Answer) with a succinct
Python-like syntax. The agent policy maintains the dialogue state, executes
actions, and proposes the subsequent dialog acts. Finally, the response
generator uses the dialogue state and dialog acts to generate a response to
the user’s initial query.

We present the following contributions:

  1. 1.

We introduce KITA, an open-domain integrated task and knowledge assistant that
automatically follows high-level developer-provided agent policies and answers
questions by grounding in hybrid knowledge sources. KITA supports full
compositionality of tasks and knowledge queries.

  2. 2.

We propose a novel specification for TOD agents, called KITA Worksheet. KITA
executes the Worksheet by tracking the state of the conversations and
providing turn-by-turn instructions to the LLM.

  3. 3.

We conduct real-user studies involving 62 participants across three different
task-oriented dialog agents created using KITA. Our results demonstrate an
impressive 91.5% execution accuracy, 91.6% dialog acts accuracy, and 74.2%
goal completion rate, achieving 26.1, 22.5, and 52.4 points over the GPT-4
function-calling baseline, respectively.

  4. 4.

Finally, we release a dataset consisting of 180 dialogue turns from 22 real-
user conversations with KITA, manually corrected to ensure accuracy.

##  2 Related Works

###  2.1 Dialogue Tree-based frameworks

A wide variety of commercial and open-sourced products Amazon (2023); Press
(2024); Xie et al. (2022); Google (2024); Microsoft (2022a, b); Reach (2022);
Bocklisch et al. (2017); Watson (2022) exist that help developers program a
conversational virtual assistant using dialogue trees, with slight differences
in design. For instance, Press (2024); Google (2024); Microsoft (2022a); Reach
(2022); Watson (2022) allow users to interactively program a dialogue tree
with a graphical user interface. Developers can manipulate various building
blocks that classify user intents, sometimes with the help of built-in NLU
(natural language understanding) modules. RASA allows developers to declare a
list of global intents and program linear conversational paths accordingly
Bocklisch et al. (2017). Converse allows developers to program a special kind
of “and-or” trees Xie et al. (2022).

In dialogue trees, developers explicitly program all possible dialogue turns
and responses. The resulting dialogues are very structured and easily
controllable. However, This leads to exponentially many possible dialogue
paths, almost impossible to exhaust in real-life scenarios.

###  2.2 LLM and task-oriented dialogues

A series of recent works attempts to integrate LLMs with TOD, showing its
capability in dialogue state tracking Hu et al. (2022); Feng et al. (2023);
Hudeček and Dusek (2023); Li et al. (2024); Zhang et al. (2023b). In
particular, Li et al. (2024) uses GPT’s function calling capability for state
tracking, which we compare against in Section 6.

When building end-to-end dialogue agents with LLMs, existing works typically
feed the entire set of instructions to the model for response generation
(Zhang et al., 2023b). While this works well on smaller domains, LLMs struggle
to follow all instructions in real-life situations Liu et al. (2024a), as we
will demonstrate in Section 6. Recently, a popular framework Nemo-Guardrails
(Rebedea et al., 2023), which allows programs to add “guardrails” to LLM-based
conversational systems, has started to gain community attention. Operating
under a simple trigger-action model, Nemo-Guardrails cannot handle multi-turn
conversations where the dialogue state needs to be tracked, e.g., slot-
fillings across multiple turns.

Figure 2: Programmable policies for ticket submission assistant using KITA
Worksheets. The KITA Worksheets allow expressing different APIs and databases
in a single specification as a spreadsheet. The system developer can write
predicates and actions for tasksheets and fields. Moreover, the programming
paradigm is declarative in contrast to the convoluted imperative paradigm of
traditional dialogue tree-based TOD systems.

##  3 Design of the KITA Worksheet

Traditionally, task-oriented agents in NLP literature were designed as slot-
filling frameworks. The users were prompted for specific values to fill
predefined slots based on their utterances. These slots provide values to the
parameters of API calls. However, slots are insufficient for capturing
knowledge-oriented queries. Instead, formal representations or query languages
should be used Andreas et al. (2020); Lam et al. (2022); Liu et al. (2024b).

Contrary to dialog trees, we design KITA Worksheet as a high-level declarative
specification, not too different from the information we would give to a human
operator. It is designed to be read by our KITA agent to perform the given
task. We describe the design goals along with the design below.

###  3.1 Design Objectives

Conditional inputs. Online tasks today are primarily implemented as webpages,
with each website containing numerous pages. Users navigate the website by (1)
choosing possible pages from a menu, which often opens up to more websites or
(2) filling in requested fields iteratively. Suppose a user wants to report a
lost coat. After they navigate the page for lost items, they need to report
the flight information if it was lost on a plane and the terminal if it was
lost at the terminal, etc. It is not a single API call with a fixed set of
parameters.

Programmable Agent Policy. Task-oriented agents not only just passively
execute user commands but have their own objectives. For example, after a user
books a flight, the agent may wish to offer the chance to book a rental car at
a discount. Or, if the user is checking into a flight that is known to be
delayed, the agent should inform the user. In other words, developers need to
have the ability to code the agent policy in terms of actions to take and
information to be supplied to the user. To ensure control, ease of
development, and quick updates, it is important to let developers specify the
policy rather than training a neural policy via dialogue training samples.

Composition of APIs and queries. We observe that a single user utterance can
combine parameters for an API with queries, suggesting that we need to support
full compositionality between APIs and queries. Consider the following
example:

##### Example 1:

A restaurant reservation agent may start by asking the user:

Agent: “Hi, I am the restaurant reservation assistant. Which restaurant would
you like to book?”.

A traditional TOD system would expect the user’s response to directly provide
a restaurant name to fill the corresponding slot. However, users often do not
know which restaurant they want, but may however offer characteristics of the
restaurants they want to book as well as details for the reservation. For
example:

User:“I want to book an Italian restaurant in NYC for two on Valentine’s day”.

Here, the result of the query (Restaurant) is supplied as an argument to a
task (Book Restaurant). Moreover, agents must record all the relevant
parameters in the dialogue state. This example highlights how users can
respond to a system’s question with another query while simultaneously
providing information about other relevant task parameters.

###  3.2 The KITA Worksheet

The KITA Worksheet is a high-level declarative specification of a TOD agent
task designed to achieve the above design objectives. A task specification
consists of developer-defined types, knowledge corpus schemas, API calls, and
a set of worksheets. A worksheet, inspired by the versatility of webforms,
consists of a set of (potentially optional or conditional) typed inputs of
interests and the arbitrary actions to take.

A worksheet, as illustrated in Figure 2, has a name (WS Name), a predicate (WS
Predicate) indicating when it is activated, and a set of fields. The top-level
worksheet is unconditionally executed, and the rest of the worksheets are
activated if their corresponding predicates are true. Each field has these
attributes:

  * •

Predicate: whether the field is active

  * •

Input: whether it is an input or an internal value, the latter is computed
rather than solicited from the user

  * •

Type: if the type is “Enum”, the possible values are specified in the “Enum
Values” field

  * •

Name: the field name

  * •

Description: a natural language description

  * •

Don’t Ask: if true, the agent saves the information if offered by the user,
but does not solicit it. An example of such a field could be: “Is the user
annoyed?”. In this case, the system won’t explicitly ask the user if they are
annoyed; however, if the user mentions that they are, the value can be set to
True.

  * •

Required: if true, solicits the user for a value

  * •

Confirmation: if true, confirms the value with the user, which is useful if an
undesirable side effect can result from a mistaken value.

  * •

Actions: code (in Python) to be executed whenever a value is assigned to the
variable.

Finally, each worksheet also has a WS actions field, where the developer can
provide Python code to be executed when all the required fields are filled.
Several built-in actions are provided to the developer: (1) say (str) responds
to the user with the given string str. (2) propose (ws, [fld,val]*)
instantiates a new worksheet ws with the given field value pairs.

Worksheets do not just specify the information to be gathered, they are also
used to keep track of the user supplied values in the dialogue so far. For
example, if the users book two restaurants, then the dialogue history will be
represented by 2 different instances of the BookRestaurant worksheet.

In Figure 2, we illustrate a KITA Worksheet created for the ticket submission
agent, with two worksheets and schema for free-text corpus111The actual KITA
Worksheet contains seven worksheets (please refer to Appendix C) (in a darker
shade). The first worksheet is the main worksheet, which needs to be submitted
using submit_ticket API. It is denoted by WS under the Type column. This
worksheet has no predicate, meaning it can be updated by the user or the agent
at any time.

###  3.3 Query Handling

As discussed above, the task includes the schemas of all knowledge bases to be
used in the task. To keep track of the dialogues involving databases, we
introduce a degenerate worksheet, the KB-Worksheet. Each query issued by the
user will instantiate a KB-Worksheet automatically derived from the KB schema.
The worksheet has the following fields: NL query: a string of the form
answer(s), where s is the de-contextualized (i.e. self-contained) query in
natural language. This is expected to be produced by the agent’s semantic
parser. SUQL query: the SUQL query associated with the NL query. KB result:
the result of executing the SUQL query.

The action associated with the NL query field is to invoke the SUQL parser to
convert the NL query into SUQL. The action with the SUQL query is to execute
the query if all the required parameters are available.

###  3.4 Composition of Worksheets

Compositions of queries and APIs are supported by passing the result of an
instance of a worksheet in as a field of another worksheet. For instance, the
semantic parser translates “I want to book an Italian restaurant in NYC for
two on Valentine’s day” into 2 partially filled worksheets:

  1. 1.

A KB-worksheet $w_{1}$ instantiated from the restaurant schema, where NL query
has value

        Answer(SELECT * FROM restaurants
           WHERE ’italian’ = ANY (cuisines)
           AND location = ’NYC’;")

  2. 2.

A worksheet $w_{2}$ instantiated from BookRestaurant where restaurant =
$w_{1}$, date = "02/14/2024", and num_people=2.

Figure 3: KITA provides the latest set of worksheets and only one previous
conversation turn to the semantic parser. The parser outputs the current
worksheets, which are used by the Agent Policy to generate the Agent Dialogue
Acts. These Acts along with the latest worksheet value and the latest user
utterance are used to generate the response.

##  4 The KITA Agent Design

Despite the impressive natural understanding capabilities of LLMs, they often
struggle with maintaining relevant contextual information across longer
conversations. This leads to hallucinated or fabricated values and repeating
questions. To mitigate this, we use the KITA Worksheet to track the dialog
state. Specifically, at each turn, the current state of the KITA Worksheet and
the previous conversational turn are provided as input to both the LLM-based
semantic parser and the response generator. This prevents the system from
forgetting critical field values and inadvertently generating fictitious ones.
Furthermore, to address the challenge of LLMs failing to solicit required
parameters, we utilize a programmable agent policy that checks all the fields
that need to be captured and suggests the next appropriate action.

The KITA agent has these 3 components: a contextual semantic parser that
translates user utterance in the context of the dialogue, the agent policy,
and the response generation as in Figure 3.

Contextual Semantic Parsing The contextual semantic parser translates the user
utterance given the current dialogue state and generates the new dialogue
state. Our active and completed worksheets are a complete encapsulation of
information relevant for understanding the next user statement. Thus, they are
used as formal context rather than the complete history of utterances.

The output of the parser is an updated set of worksheets. The user may (1)
supply values to fields that are anticipated in existing worksheets; (2)
modify a previously filled field or even remove its value; (3) initiate new
tasks or queries. In the last case, new worksheets will be created.

Agent Policy The agent policy module interprets the state of the worksheet to
determine its actions. The agent can communicate with the user through a set
of 5 agent dialogue acts (Figure 4). The policy execution consists of the
following steps:

  1. 1.

If there are new database queries, the agent policy follows the actions on the
instantiated KB-worksheets. A new NL query is translated into an SUQL query
using the SUQL parser; if required parameters are missing, the agent policy
uses the AskAct policy to ask users for the missing values. If the SUQL query
is ready to be executed, it saves the returned result in the KB-worksheet.

  2. 2.

Results of these databases are reported using the ReportAct.

  3. 3.

It examines all the newly populated fields in the order in the worksheet. If
the field is indicated as requiring confirmation, it issues the
ConfirmationAct action. Otherwise, execute any associated developer-defined
actions that need no confirmation, or for which confirmation has been
received.

  4. 4.

After all the newly filled fields are processed, the agent policy finds the
first unfilled field and invokes the AskAct on it.

  5. 5.

If all the required fields are filled and all their corresponding actions are
completed, then execute the developer-defined action associated with the
entire worksheet.

  6. 6.

If the worksheet’s result is used as an input field in other worksheets,
assign its value to such fields.

  * •

ReportAct: Report the result of any API or Database call.

  * •

ConfirmationAct: Confirm the value of a parsed field.

  * •

SayAct: Explicitly say the defined utterance.

  * •

ProposeAct: Propose to the user a new worksheet with possibly partially pre-
filled values.

  * •

AskAct: Ask for the value of the next field.

Figure 4: Agent Dialogue Acts

##### Response Generator

All the dialogue acts produced by the agent policy are supplied to the
generator, along with new KITA Worksheet state to generate a helpful response
to the user.

##  5 Experiment: A Slot-Filling Task

Before we launch into the discussion of our real user study on 3 real-life
applications, we perform a small comparison study on one of the more complex
examples of a slot-based benchmark. We evaluate KITA on the banking domain of
the StarV2 dataset Zhao et al. (2023). We chose this domain because its
program logic is relatively more complex than others, and existing agents
perform poorly on it. AnyTOD Zhao et al. (2023) has the SOTA result on StarV2.
With KITA, the agent policy for the banking domain can be specified in 9 lines
of code, as opposed to 31 lines in AnyTOD. We only provide three examples to
the LLM-based semantic parser222We use OpenAI’s gpt-4-turbo-2024-04-09 for all
our experiments. All experiments performed before June 15 2024.; AnyTOD
finetunes T5 XXL (13B) model on all domains except the one tested. ToD
achieves a weighted System action F1 (SaF1) of 65; KITA achieves an impressive
weighted SaF1 of 83 on a randomly sampled subset of 100 conversations from the
banking domain. Furthermore, our analysis indicates that most of our errors
are caused by inconsistent annotations in the dataset. (Worksheet, error
description, and other experimental details are present in Appendix C).

Applications | WS | Fields | DB | Pred | At.
---|---|---|---|---|---
Restaurant Reservation | 2 | 8 | 2 | 2 | 3
Course Assistant | 4 | 20 | 4 | 3 | 1
Ticket Submission | 7 | 28 | 1 | 18 | 2
Table 1: Implementation statistics for three distinct applications with the
total number of worksheets (WS), fields across all worksheets, databases (DB)
provided, predicates (Pred) and actions (At.) defined in the complete KITA
Worksheet.

##  6 Evaluation with Real Users

We evaluate KITA across three distinct applications with varying requirements
to demonstrate the expressiveness and context-handling capabilities.

###  6.1 Baseline

We compare our system against OpenAI’s GPT-4-turbo, leveraging its functional
calling abilities, which we call GPT-4 (FC). This baseline closely follows Li
et al. (2024). Each worksheet is defined as a function, and the worksheet-
level instructions are provided in the function description. The high-level
policies are provided in the system prompt. Moreover, we provide the baseline
system with the ability to use the Answer() function for external knowledge
access.

###  6.2 Applications

We build two visually identical systems for all the applications, one that
uses KITA and the other uses GPT-4 (FC). All the KITA Worksheet are provided
in Appendix C.

Restaurant Reservation Making a restaurant reservation requires finding a
suitable restaurant and providing booking information to complete a
transaction. We use the real-life dataset containing restaurants from Yelp.com
from Liu et al. (2024b).

Ticket Submission In this study, we aim to replicate a subset of tasks found
within a university’s student services portal. University student services
portals typically contain various tasks categorized under different sections
and subsections, posing a navigational challenge for students seeking to
locate the appropriate link. Moreover, these portals often contain a vast
corpus of free-text data, which students must peruse before submitting a
ticket. We evaluated agents’ capability to handle nested webpages with
predicates and subsequent actions.

Course Enrollment Finally, we evaluate the performance of KITA as a course
enrollment assistant, which combines hybrid data sources to search for course
details and fill out complicated nested forms. The assistants allow students
to ask questions about course requirements, student reviews, and ratings while
filling out their enrollment forms. We collect a real-life dataset containing
courses from the Computer Science program, with 4 tables (courses, offerings,
ratings, and programs).

Study Design: We use Prolific to recruit 22 participants for Restaurant
Reservations and 20 users who identified as students for Ticket Submission. We
recruited 20 university students to evaluate the Course Enrollment assistant.
We instructed them to attempt to book a reservation, submit an issue ticket,
and enroll in two courses, respectively. We randomly assigned users to one of
the two systems. In total, we collected 99, 81, and 127 turns with KITA, and
90, 70, and 144 turns with With GPT 4 (FC), respectively. More details on
instructions provided to the users are in Appendix B.

###  6.3 Evaluation Metrics

Semantic Parsing Accuracy For each user turn, we manually inspect the code
generated by the semantic parser for correct APIs and Databases and filled
fields. We define the gold Semantic Parsing output ($SP$) as the set of
correct API calls ($A$), Database calls ($D$), and fields to fill ($F$). Then
for each user utterance, $SP=\\{s_{1},s_{2},...,s_{m}\\}$, where
$s_{i}\in\\{A\cup D\cup F\\}$ and $m$ is the total number of choices, such
that $m=|A\cup D\cup F|$. The Semantic Parsing Accuracy (SP Acc) for a system
is defined as the number of correct choices in the semantic parser’s output
divided by $m$.

Execution Accuracy We manually inspect each turn to check whether the agent
executes the correct API and databases. For each agent turn, let the
executions be $E=\\{e_{1},e_{2},...,e_{m}\\}$ where $e_{i}$ is an API or
database call. We evaluate whether each $e_{i}$ is a true positive or false
positive. We calculate the Execution (Ex Acc) as the number of true positives
divided by the number of true and false positives for all the execution calls
in a conversation.

Agent Dialogue Act Accuracy For each turn, we manually inspect whether the
agent’s dialogue act follows the policies provided by the developer .
Formally, for each turn, we define a list of gold acts $a_{1},a_{2},...,a_{m}$
where $a_{i}\in A$, all the possible agent dialogue acts. The Agent Dialogue
Act Accuracy (DA Acc) for a system is defined as the number of correctly
predicted actions divided by $m$. For GPT-4 (FC), we map its responses to the
equivalent elements in the power set of all the possible actions.

Goal Completion Rate We define Goal Completion Rate (Goal CR) as the user’s
ability to successfully complete the task with the system’s assistance. with
the appropriate parameter values, then the goal is considered completed. Goal
Completion Rate is a binary metric for each conversation, where 1 denotes that
the goal was completed, and 0 indicates that the goal was not achieved.

##  7 Evaluation Results

###  7.1 User Studies

Table 2 compares KITA against GPT 4 (FC) across three metrics and find that
KITA performs significantly better than GPT-4 (FC) on all three domains.

We observe that KITA consistently demonstrates a high semantic parsing rate
(exceeding $85\%$), indicating that KITA Worksheet is simple to understand for
LLMs with few shot examples. The marginally lower SP Acc ($85.8\%$) observed
in the Ticket Submission application can be attributed to the application’s
complexity, featuring several worksheets ($=8$) and API fields ($=28$), as
shown in Table 1.

Additionally, the higher Ex Acc can be attributed to providing compressed
context as a worksheet state, enabling the LLM to invoke the correct API and
execute the suitable database query. The superior performance on DA Acc can be
ascribed to our agent’s capability to provide turn-by-turn instructions rather
than presenting all the instructions at once, as is the case with GPT-4 (FC).
The results validate the benefit of programmable policies in delivering a
reliable assistant.

In terms of the final Goal Completion Rate, we observe that GPT-4 (FC) scores
relatively higher in restaurant reservations, the domain it is most familiar
with and well represented in existing dialogue datasets (Ye et al., 2022;
Rastogi et al., 2020). Similar results are also observed by Zhang et al.
(2023a) and Hudeček and Dusek (2023). However, GPT 4 (FC) struggles with less
familiar domains, like ticket submission and course enrollment. We observe
that the higher number of predicates, necessitating several instructions in
the Ticket Submission application, makes it extremely challenging for GPT-4
(FC) to assist users in completing their tasks effectively. In most cases,
GPT-4 (FC) does not elicit all the required fields from the user, hence making
incomplete API calls. Moreover, despite having access to the Answer()
function, GPT-4 (FC) often hallucinates non-existent courses and fails to
enroll students.

| SP Acc | Ex Acc | DA Acc | Goal CR
---|---|---|---|---
System | All Domains
GPT 4 (FC) | - | 65.4 | 69.1 | 21.8
KITA (ours) | 91.7 | 91.5* | 91.6* | 74.2*
| Restaurant Reservation
GPT 4 (FC) | - | 50.0 | 57.7 | 54.54
KITA (ours) | 93.8 | 88.0* | 92.5* | 91.6
| Ticket Submission
GPT 4 (FC) | - | 58.6 | 67.6 | 0.0
KITA (ours) | 85.8 | 80.0* |  82.5* | 80.0*
| Course Enrollment
GPT 4 (FC) | - | 79.2 | 77.8 | 10.0
KITA (ours) | 94.5 | 98.6* |  98.1* | 50.0
Table 2: We perform a real user study with 62 users across three domains. We
find that KITA performs significantly better than GPT-4 with Function Calling
ability on the three applicable metrics. We perform t-test and mark the
results with $p<0.05$ with *. Bold represents the better result.

On the contrary, we observe that KITA facilitates accurately completing and
achieving their goals. We note that KITA effectively aids the users in
completing their tasks ($\geq 80\%$) in both the Restaurant Reservation and
Ticket Submission applications. For the Course Enrollment assistant, we
observe that users who could not complete the task abandoned the conversation
prematurely. In most cases, this is due to the database not having enough
information to answer the query. Nonetheless, KITA still drastically
outperforms GPT-4 (FC), which uses the same database.

Figure 5: Analysis of semantic parsing errors with KITA for all the
assistants. Most of the execution and dialogue act errors occur due to
incorrect semantic parsing.

###  7.2 Error Analysis

We analyze all the error cases for our system and GPT-4 (FC) and provide
examples in the Appendix.

##### GPT-4 (Function Calling)

GPT 4 (FC) fails to use the provided Answer() function and responds directly
to user queries, leading to fictitious responses. We note that it hallucinates
in (7/10), (4/11), and (5/10) conversations for course enrollment, ticket
submission, and restaurant reservation domains. Moreover, it prematurely
invokes the submission API in all three domains and disregards instructions
such as failing to seek confirmation before invoking API’s. More error
analysis is Appendix E.

##### KITA

We analyze errors in the Ticket Submission application. In 21% of the cases,
the agent fails to handle cases where the user refuses to provide information
for a required field, resulting in an unassigned value. Another 21% of errors
stem from the semantic parser’s inability to call all the necessary worksheets
or populate the correct parameter values. 43% of errors were due to incorrect
usage of Answer() function, either by using it incorrectly to query knowledge
sources or by neglecting to use it altogether. In another 14% of errors, the
LLM-based parser failed to initiate a new worksheet when the user attempted to
start a new transaction and instead updated an existing worksheet.

We also observe that 33% and 66% errors in semantic parsing are due to
creating erroneous worksheets or incorrect fields for Course Enrollment and
Restaurant Reservation assistant. On the other hand, there are 66% and 33%
errors due to incorrect usage of Answer().

##  8 Conclusion

We introduce KITA, a novel framework for building knowledge-integrated task
assistants with LLMs, with KITA Worksheet, a high-level specification that
allows for detailed control over conversation flows. Our real user studies
with 62 participants demonstrate KITA drastically outperforms traditional LLM-
based systems and dialogue trees.The accompanying dataset with 180 dialogue
turns provides a resource for further research and underscores the practical
applicability of our approach.

## Limitation

As an LLM-based system, KITA’s performance is directly tied to the capability
of the underlying LLM. The non-deterministic nature of Language Models can
also lead to performance variations across different runs.

Being a programmable framework, the performance of KITA is influenced by the
specific policies implemented by developers for new domains. The selection of
few-shot examples is also expected to impact the agent’s performance. To
address this, we will provide comprehensive guidelines and tools for
developers to create effective policies and select optimal few-shot examples
in our release.

We acknowledge that the additional intermediate LLM calls introduced by KITA
will lead to higher costs and increased latency. However, we believe these
trade-offs are necessary to deliver a reliable and accurate integrated task
and knowledge assistant.

## Ethical Considerations

Large Language Models have been increasingly utilized to build various task-
oriented agents. We propose a novel method to enhance their accuracy and
reliability. We do not anticipate any harm resulting from this work.

For the user evaluation of the Course Enrollment application, we recruited
university students who voluntarily participated in the study, awarding each
participant an Amazon gift card worth $10 per 15 minutes. For the Restaurant
Reservation and Ticket Submission applications, we used the Prolific platform
to recruit participants and compensated them fairly, beyond the minimum wage.
Our procedure has been approved by the Institutional Review Board (IRB) at our
institution. All collected information is anonymous. We also remove any
personal identifiable information from the collected dialogues.

Our code will be released publicly under the Apache License, Version 2.0, and
the collected data will be made available to the community.

## Acknowledgement

This work is supported in part by the National Science Foundation, the Alfred
P. Sloan Foundation, the Verdant Foundation, Microsoft Azure AI credit, KDDI,
JPMorgan Chase, and the Stanford Human-Centered Artificial Intelligence (HAI)
Institute. We also thank Adit Negi, Aryaman Arora, Chenglei Si, Jordan
Juravsky, Lucia Zheng, Liza Pertseva, Sagnik Bhattacharya, Sina Semnani,
Tristan Thrush, Yijia Shao, Yanzhe Zhang and the members of the Stanford OVAL
(Open Virtual Assistant Lab) for helpful discussion about the project and
comments on the manuscript.

## References

  * Amazon (2023) Amazon. 2023.  Amazon lex.
  * Andreas et al. (2020) Jacob Andreas, John Bufe, David Burkett, Charles Chen, Josh Clausman, Jean Crawford, Kate Crim, Jordan DeLoach, Leah Dorner, Jason Eisner, Hao Fang, Alan Guo, David Hall, Kristin Hayes, Kellie Hill, Diana Ho, Wendy Iwaszuk, Smriti Jha, Dan Klein, Jayant Krishnamurthy, Theo Lanman, Percy Liang, Christopher H. Lin, Ilya Lintsbakh, Andy McGovern, Aleksandr Nisnevich, Adam Pauls, Dmitrij Petters, Brent Read, Dan Roth, Subhro Roy, Jesse Rusak, Beth Short, Div Slomin, Ben Snyder, Stephon Striplin, Yu Su, Zachary Tellman, Sam Thomson, Andrei Vorobev, Izabela Witoszko, Jason Wolfe, Abby Wray, Yuchen Zhang, and Alexander Zotov. 2020.  Task-oriented dialogue as dataflow synthesis.  _Transactions of the Association for Computational Linguistics_ , 8:556–571.
  * Bang et al. (2023) Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023.  A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity.  In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 675–718, Nusa Dua, Bali. Association for Computational Linguistics.
  * Bocklisch et al. (2017) Tom Bocklisch, Joey Faulkner, Nick Pawlowski, and Alan Nichol. 2017.  Rasa: Open source language understanding and dialogue management.  _arXiv preprint_.
  * Budzianowski et al. (2018) Paweł Budzianowski, Tsung-Hsien Wen, Bo-Hsiang Tseng, Iñigo Casanueva, Stefan Ultes, Osman Ramadan, and Milica Gašić. 2018.  MultiWOZ - a large-scale multi-domain Wizard-of-Oz dataset for task-oriented dialogue modelling.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 5016–5026, Brussels, Belgium. Association for Computational Linguistics.
  * Feng et al. (2023) Yujie Feng, Zexin Lu, Bo Liu, Liming Zhan, and Xiao-Ming Wu. 2023.  Towards llm-driven dialogue state tracking.  _arXiv preprint arXiv:2310.14970_.
  * Google (2024) Google. 2024.  Google dialogueflow.
  * Hu et al. (2022) Yushi Hu, Chia-Hsuan Lee, Tianbao Xie, Tao Yu, Noah A Smith, and Mari Ostendorf. 2022.  In-context learning for few-shot dialogue state tracking.  _arXiv preprint arXiv:2203.08568_.
  * Hudeček and Dusek (2023) Vojtěch Hudeček and Ondrej Dusek. 2023.  Are large language models all you need for task-oriented dialogue? In _Proceedings of the 24th Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 216–228, Prague, Czechia. Association for Computational Linguistics.
  * Hudeček and Dušek (2023) Vojtěch Hudeček and Ondřej Dušek. 2023.  Are llms all you need for task-oriented dialogue?  _arXiv preprint arXiv:2304.06556_.
  * Kim et al. (2023) Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. 2023.  An llm compiler for parallel function calling.  _arXiv preprint arXiv:2312.04511_.
  * Lam et al. (2022) Monica S. Lam, Giovanni Campagna, Mehrad Moradshahi, Sina J. Semnani, and Silei Xu. 2022.  Thingtalk: An extensible, executable representation language for task-oriented dialogues.  _Preprint_ , arXiv:2203.12751.
  * Li et al. (2024) Zekun Li, Zhiyu Zoey Chen, Mike Ross, Patrick Huber, Seungwhan Moon, Zhaojiang Lin, Xin Luna Dong, Adithya Sagar, Xifeng Yan, and Paul A. Crook. 2024.  Large language models as zero-shot dialogue state tracker through function calling.  _Preprint_ , arXiv:2402.10466.
  * Liu et al. (2024a) Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024a.  Lost in the middle: How language models use long contexts.  _Transactions of the Association for Computational Linguistics_ , 12:157–173.
  * Liu et al. (2024b) Shicheng Liu, Jialiang Xu, Wesley Tjangnaka, Sina J. Semnani, Chen Jie Yu, and Monica S. Lam. 2024b.  Suql: Conversational search over structured and unstructured data with large language models.  In _Findings of the Association for Computational Linguistics: NAACL 2024_.
  * Microsoft (2022a) Microsoft. 2022a.  Microsoft bot framework composer.
  * Microsoft (2022b) Microsoft. 2022b.  Microsoft power virtual assistants.
  * Patil et al. (2023) Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023.  Gorilla: Large language model connected with massive apis.  _Preprint_ , arXiv:2305.15334.
  * Press (2024) Bot Press. 2024.  Bot press.
  * Rastogi et al. (2020) Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020.  Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 8689–8696.
  * Reach (2022) One Reach. 2022.  One reach.
  * Rebedea et al. (2023) Traian Rebedea, Razvan Dinu, Makesh Narsimhan Sreedhar, Christopher Parisien, and Jonathan Cohen. 2023.  NeMo guardrails: A toolkit for controllable and safe LLM applications with programmable rails.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 431–445, Singapore. Association for Computational Linguistics.
  * Watson (2022) Watson. 2022.  Ibm watson assistants.
  * Xie et al. (2022) Tian Xie, Xinyi Yang, Angela S. Lin, Feihong Wu, Kazuma Hashimoto, Jin Qu, Young Mo Kang, Wenpeng Yin, Huan Wang, Semih Yavuz, Gang Wu, Michael Jones, Richard Socher, Yingbo Zhou, Wenhao Liu, and Caiming Xiong. 2022.  Converse: A tree-based modular task-oriented dialogue system.  _arXiv preprint_.
  * Ye et al. (2022) Fanghua Ye, Jarana Manotumruksa, and Emine Yilmaz. 2022.  MultiWOZ 2.4: A multi-domain task-oriented dialogue dataset with essential annotation corrections to improve state tracking evaluation.  In _Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue_ , pages 351–360, Edinburgh, UK. Association for Computational Linguistics.
  * Zhang et al. (2023a) Xiaoying Zhang, Baolin Peng, Kun Li, Jingyan Zhou, and Helen Meng. 2023a.  Sgp-tod: Building task bots effortlessly via schema-guided llm prompting.  _arXiv preprint arXiv:2305.09067_.
  * Zhang et al. (2023b) Xiaoying Zhang, Baolin Peng, Kun Li, Jingyan Zhou, and Helen Meng. 2023b.  SGP-TOD: Building task bots effortlessly via schema-guided LLM prompting.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 13348–13369, Singapore. Association for Computational Linguistics.
  * Zhao et al. (2023) Jeffrey Zhao, Yuan Cao, Raghav Gupta, Harrison Lee, Abhinav Rastogi, Mingqiu Wang, Hagen Soltau, Izhak Shafran, and Yonghui Wu. 2023.  AnyTOD: A programmable task-oriented dialog system.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 16189–16204, Singapore. Association for Computational Linguistics.

##  Appendix A Experimental Settings

We used OpenAI’s GPT 4-turbo for semantic parsing and response generation. The
underlying SUQL system uses GPT 3-turbo. All the experiments were performed
before June 15, 2024. The semantic parser uses a temperature of 0.0, and the
response generation module uses a temperature of 0.7.

##  Appendix B User Study

We use the same user interface for KITA and GPT 4 (FC) as shown in Figure 6

Figure 6: User study webapp for Course enrollment assistant, similar interface
was used for other studies Figure 7: user study survey form Figure 8: user
study survey form Figure 9: user study survey form Figure 10: user study
survey form questionnaire

###  B.1 User instruction for Restuarant Reservation

The users were instructed to search for restaurants in one of four available
location and then attempt to book the restaurant.

###  B.2 User instruction for Course enrollment

These users were instructed to request course recommendations from the
assistant and subsequently attempt to complete the course enrollment process
using the assistant. The users interacted with KITA for 127 turns and 146
turns with GPT-4 (FC).

###  B.3 User instruction for Ticket Submission

  * •

You applied for a leave of absence but cannot check your form status.

  * •

You are having trouble with joining the waitlist for a course.

  * •

You cannot find your AP credits in your transcript.

###  B.4 Example Conversations

Here are examples of good and bad conversations for the KITA system and the
GPT system across all three applications. University names, services, and some
classes have been obfuscated.

####  B.4.1 KITA Course Enroll Good Example

User: Find me programming courses that will teach me object oriented
programming
Bot: I searched for programming courses that specifically teach object-
oriented programming, but unfortunately, I couldn’t find any available
courses. Could you please provide me with the details of the courses you want
to take?
User: Provide me details about CS108
Bot: I searched for details about CS108. Here is the information: \- **Course
Title**: Object-Oriented Systems Design \- **Course Codes**: CS 108 \-
**Description**: Software design and construction in the context of large OOP
libraries. Taught in Java. Topics include OOP design, design patterns,
testing, graphical user interface (GUI) OOP libraries, software engineering
strategies, and approaches to programming in teams. Prerequisite: CS 107. \-
**Average Hours Spent**: 16.29 hours per week.

Could you please provide me with the details of the courses you want to take?
User: How does this class compare to CS124?
Bot: I compared CS108 and CS124 based on their average ratings. Here’s what I
found: \- **CS108 - Object-Oriented Systems Design** has an average rating of
4.13. \- **CS124 - From Languages to Information** has an average rating of
4.58.

It appears that CS124 has a higher average rating compared to CS108.

Could you please provide me with the details of the courses you want to take?
User: Am I well prepared for "CS Course" after taking CS124?
Bot: I searched to see if taking CS 124 prepares you for CS 224N, but I
couldn’t find any direct information indicating that CS 124 is a prerequisite
for CS 224N. Could you please provide me with the details of the courses you
want to take?
User: I would like to take classes related to the movie industry and movie
making, such as graphics and computer vision. Can you give me some
recommendations?
Bot: I searched for courses related to the movie industry and movie making,
focusing on graphics and computer, and found a couple of interesting options:
1\. **Topics in Computer Graphics: Agile Hardware Design (CS 448H)** - This
course covers various topics which change each quarter, including
computational photography, data visualization, character animation, virtual
worlds, graphics architectures, and advanced rendering. More details can be
found at [University’s graphics course
offerings](http://graphics.university.edu/courses). This course can be
repeated for credit. 2\. **Topics in Computer Graphics (CS 448)** - Similar to
CS 448H, this course also explores topics like computational photography, data
visualization, character animation, and more, with details available on the
[course website](http://graphics.university.edu/courses). This course is also
repeatable for credit.

Could you please provide me with the details of the courses you want to take?
User: How many units is CS 448? And when is it offered?
Bot: I looked up the course CS 448, specifically ’Data Visualization’, and
found that it is offered as a 4-unit course. The class sessions are scheduled
on Mondays and Wednesdays from 11:30 AM to 12:50 PM. Could you please provide
me with the details of the courses you want to take?
User: What quarter is it offered?
Bot: I checked the quarter in which CS 448B ’Data Visualization’ is offered.
It is offered in the autumn quarter. Could you please provide me with the
details of the courses you want to take?
User: Let’s enroll in CS448
Bot: Could you please specify the desired grading basis for CS 448B ’Data
Visualization’? The options are: Credit/No Credit, Letter.
User: Letter
Bot: Thank you for confirming the grading basis. Could you please specify the
number of units you want to take for the course CS 448B ’Data Visualization’?
User: 3 units
Bot: Thank you! Now, could you please provide the details for the second
course you would like to take? This includes the course name, grade type, and
number of units.
User: Let’s also take CS147 for 5 units in the fall
Bot: Thank you for the information on the second course, CS147. Do you wish to
take more courses?
User: No, that is all. Can you please tell me what I am currently enrolled in?
Bot: I have the details for the courses you’ve selected so far. Could you
please confirm all the course details before we proceed?

\- **First Course:** \- Name: CS 448B \- Grade Type: Letter \- Number of
Units: 3 \- **Second Course:** \- Name: CS147 \- Grade Type: Letter \- Number
of Units: 5

Can you confirm all the course details before saving?
User: Yes that’s correct!
Bot: Thank you for confirming the course details. Could you please provide me
with some additional details about the student, such as the name, student ID,
and student email address?
User: My name is Roger Corman, ID rogerc, and email<EMAIL_ADDRESS>
Bot: Thank you for providing your details, Roger. Could you please confirm if
all the information you want to submit is correct?
User: Yes that is correct
Bot: Thank you for confirming. Your course enrollment has been successfully
processed. Your transaction ID is 4b087961-b779-4958-a205-9a0938e4cbd0. If you
have any other questions or need further assistance, feel free to ask!

####  B.4.2 KITA Course Enroll Bad Example

User: Compile a list of computer vision courses available
Bot: I searched for computer vision courses available and found the following:

1\. **Computer Vision: From 3D Reconstruction to Recognition** (CS 231A): An
introduction to the concepts and applications in computer vision. Topics
include cameras and projection models, low-level image processing methods such
as filtering and edge detection; mid-level vision topics such as segmentation
and clustering; shape reconstruction from stereo, as well as high-level vision
tasks such as object recognition, scene recognition, face detection, and human
motion categorization.

2\. **Deep Learning for Computer Vision** (CS 231N): This course is a deep
dive into details of the neural-network-based deep learning methods for
computer vision. It covers learning algorithms, neural network architectures,
and practical engineering tricks for training and fine-tuning networks for
visual recognition tasks.

3\. **Computer Vision and Image Analysis of Art** (CS 231C): Presents the
application of rigorous image processing, computer vision, machine learning,
computer graphics, and artificial intelligence techniques to problems in the
history and interpretation of fine art paintings, drawings, murals, and other
two-dimensional works, including abstract art.

4\. **Generative Adversarial Networks** (CS 236G): Generative Adversarial
Networks (GANs) have rapidly emerged as the state-of-the-art technique in
realistic image generation. This course presents theoretical intuition and
practical knowledge on GANs, from their simplest to their state-of-the-art
forms.

5\. **AI for Social Good** (CS 21SI): Students will learn about and apply
cutting-edge artificial intelligence techniques to real-world social good
spaces. The class will focus on techniques from machine learning and deep
learning, including regression, neural networks, convolutional neural networks
(CNNs), and recurrent neural networks (RNNs).

Could you please provide me with the details of the courses you want to take?
User: What is available in summer quarter?
Bot: I searched for computer vision courses available in the summer quarter,
but unfortunately, I couldn’t find any information on computer vision courses
being offered during that time. Could you please provide me with the details
of any other courses you are interested in taking this summer?
User: which ones do not involve UNet?
Bot: I searched for summer quarter courses that do not involve UNet, but I
couldn’t find any information on such courses. Could you please provide me
with the details of any other courses you are interested in taking this
summer?
User: dont restrict yourself to summer
Bot: Understood, without restricting to a specific quarter, let’s look into
the courses again. Could you please provide me with the details of the courses
you are interested in taking?
User: i am interested in computer vision courses which do not involve UNet
Bot: I searched for computer vision courses that do not involve UNet, but
unfortunately, I couldn’t find any information. Could you provide me with more
details or any specific requirements for the computer vision courses you are
interested in taking?
User: which ones of the above involve less than 15 hours effort per week
Bot: I found several computer vision courses that involve less than 15 hours
of effort per week. Here they are:

1\. **Computer Vision and Image Analysis of Art (CS 231C):** This course
focuses on applying image processing, computer vision, machine learning, and
other techniques to problems in the history and interpretation of fine art. It
covers the analysis of brushstrokes, medium, artists’ working methods, and
more. Recommended prerequisites include CS 131 or EE 168.

2\. **Generative Adversarial Networks (CS 236G):** This course offers
theoretical and practical knowledge on Generative Adversarial Networks (GANs),
covering their applications in realistic image editing, medicine, and
visualizing scenarios of climate change. Prerequisites include linear algebra,
statistics, CS106B, plus a graduate-level AI course such as CS230, CS229, or
CS221.

3\. **Computer Vision: Foundations and Applications (CS 131):** This course
introduces fundamental concepts in image processing and various applications
of Computer Vision in automotive, healthcare, and other sectors. It requires
familiarity with Python, Calculus, & Linear Algebra.

4\. **AI-Assisted Care (CS 337, MED 277):** Focused on applying computer
vision and machine learning technologies in healthcare, this course offers
research projects on care for seniors, surgical quality analysis, and more. It
is open to students in both the school of medicine and the school of
engineering.

5\. **Computer Vision for Education and Social Science Research (CS 432, EDUC
463):** This course aims to provide education and social science researchers
with the knowledge to apply computer vision algorithms in their work. It
requires familiarity with Python and some experience with data.

Could you please provide me with the details of the courses you are interested
in taking?
User: Which of the above has more grad students than undergrads?
Bot: I searched for courses with a minimum of 100 units, as a possible
indicator of graduate-level courses, but couldn’t find any information on
which of the courses has more grad students than undergrads. Could you please
provide me with the details of the courses you are interested in taking?
User: which of the computer vision courses you mentioned has no final exam and
one project?
Bot: I searched for computer vision courses that have no final exam, one
project, are graded as ’Letter or Credit/No Credit’, and have exactly one
prerequisite course, but I couldn’t find any information. Could you please
provide me with the details of the courses you want to take?
User: I will take CS131. Summarize the course content
Bot: The course CS131 focuses on Computer Vision technologies, which are
revolutionizing sectors such as automotive, healthcare, manufacturing, and
agriculture. It delves into how technologies enable robots to navigate and
perform tasks, search engines to index vast amounts of images and videos,
algorithms to diagnose diseases from medical images, and smart cars to drive
safely. The course covers fundamental principles and applications of Computer
Vision, introducing concepts in image processing and leading students through
projects to implement advanced algorithms. Optional discussion sections are
available on Fridays. Prerequisites include familiarity with Python, Calculus,
and Linear Algebra. Could you please provide me with the details of any other
courses you would like to take?
User: Yes, I am also thinking about taking CS432. How is the content different
from CS131?
Bot: I’m sorry, but there seems to be a misunderstanding. The course CS432 was
not listed in the results I provided earlier. However, I can share details
about other courses that were mentioned:

1\. **Hap.py Code: The Python Programming Language (CS 41)** - This course
covers the fundamentals and contemporary usage of the Python programming
language, focusing on developing best practices in writing Python and
exploring the extensible and unique parts of the Python language. Topics
include Pythonic conventions, data structures, powerful built-ins, and Python
libraries. Towards the end, students will work on developing a significant
Python project.

2\. **CS + Social Good Studio: Implementing Social Good Projects (CS 52)** -
This course is a continuation of CS51, focusing on applying scalable technical
frameworks, methods to measure social impact, tools for deployment, user
acquisition techniques, and growth/exit strategies to build a sustainable
infrastructure around a product idea.

3\. **Dissecting The Modern Computer (CS 80E)** - Offers a high-level,
accessible introduction to computer architecture through the RISC-V ISA,
covering topics from simple digital circuits, assembly, processors, memory
systems, and bonus topics like GPUs.

4\. **How to Make VR: Introduction to Virtual Reality Design and Development
(CS 11SI)** - A hands-on course where students will design and develop virtual
reality applications using the Unity game engine, focusing on design best
practices and the creation pipeline for VR applications.

5\. **Topics in Computer Networks (CS 344)** - Often referred to as "Build an
Internet Router," this course provides hands-on experience in building the
hardware and software of a high-performance network system, including
designing a control plane in C and the data plane in the P4 language.

Could you please provide me with the details of any other courses you would
like to take?
User: Show me courses for spring quarter which require calculus knowledge
Bot: I found some courses for the spring quarter that require calculus
knowledge: 1\. **Introduction to Probability for Computer Scientists (CS
109)** - Offered on Monday, Wednesday, and Friday from 3:00 PM to 4:20 PM.
Instructors include Chris Piech and Jerry Cain. 2\. **Natural Language
Processing with Deep Learning (CS 224N, LINGUIST 284, SYMSYS 195N)** - Offered
on Tuesday and Thursday from 4:30 PM to 5:50 PM. Instructors are Diyi Yang and
Tatsunori Hashimoto. Another offering by Christopher Manning does not have
specified days or times.

Could you please provide me with the details of the courses you would like to
take?
User: I’m taking "CS Course"
Bot: Great choice! For "CS Course", could you please specify the desired
grading basis? Options are: Credit/No Credit, Letter.
User: Letter
Bot: Could you also tell me how many units you want to take for this course?
User: what is the maximum credits for this?
Bot: The maximum number of units you can take for the course CS 224N is 4. How
many units would you like to take for this course?
User: 3
Bot: Could you please provide the details for the second course you would like
to take, such as the name, grade type, and units?
User: I’m taking CS131
Bot: What is the desired grading basis for CS131? The options are: Credit/No
Credit, Letter.
User: Same as the previous one
Bot: Could you please specify the number of units you want to take for CS131?
User: same as that for "CS Course"
Bot: Could you please confirm if you would like to take more courses?
User: no
Bot: Great, let’s review and confirm the details of the courses you’ve
selected: \- **First Course:** \- Name: CS 224N \- Grade Type: Letter \-
Number of Units: 3 \- **Second Course:** \- Name: CS 131 \- Grade Type: Letter
\- Number of Units: 3

Can you confirm all the course details before we proceed?
User: change the grade type for "CS Course" to credit/no credit
Bot: Certainly! I will update the grade type for CS 224N to Credit/No Credit.
Here are the updated course details: \- **First Course:** \- Name: CS 224N \-
Grade Type: Credit/No Credit \- Number of Units: 3 \- **Second Course:** \-
Name: CS 131 \- Grade Type: Letter \- Number of Units: 3

Can you confirm all the course details before we proceed?
User: yes
Bot: Great! Now, could you please provide me with the details about yourself
such as your name, student ID, and student email address?
User: Tony, 6894536<EMAIL_ADDRESS>
Bot: Thank you, Tony. Could you also tell me if you are an international
student?
User: no
Bot: Great, thank you for confirming that. Before we proceed, could you please
confirm all the fields you want to submit for your course enrollment?
User: what are the fields?
Bot: The fields you need to confirm for your course enrollment are as follows:
\- **First Course:** \- Name: CS 224N \- Grade Type: Credit/No Credit \-
Number of Units: 3 \- **Second Course:** \- Name: CS131 \- Grade Type: Letter
\- Number of Units: 3 \- **Student Information:** \- Name: Tony \- Student ID:
6894536 \- Email Address<EMAIL_ADDRESS>\- International Student: No

Could you please confirm if all these details are correct?
User: yes
Bot: Thank you for confirming the details. Your course enrollment has been
successfully processed. Your transaction ID is
6ccfa962-ed48-416e-a3bd-017281733e4c. If you have any more questions or need
further assistance, feel free to ask!

####  B.4.3 KITA Ticket Submission Good Example

User: II cannot find my AP credits in my transcript
Bot: Could you please specify when the test scores were submitted?
User: May 2024
Bot: Thank you for providing the submission date. Could you please provide any
additional details or specifics about the issue that you would like to add?
User: They were for AP psychology
Bot: Thank you for providing the details. Could I also get your full name to
proceed with addressing your issue regarding the AP psychology credits?
User: Jane Doe
Bot: Great! Before I submit the ticket, can you confirm the details? \-
Student Task: External Test Credits \- Specific Issue: Missing or Incorrect
Record \- Test Issues: Credit not posted \- Test Type: Advanced Placement (AP)
Scores \- Time of Test Score Submission: May 2024 \- Extra Details: They were
for AP psychology \- Full Name: Jane Doe
User: Correct
Bot: Your request has been successfully submitted. Here are the details: \-
**Student Task:** External Test Credits \- **Specific Issue:** Missing or
Incorrect Record \- **Test Issues:** Credit not posted \- **Test Type:**
Advanced Placement (AP) Scores \- **Time of Test Score Submission:** May 2024
\- **Extra Details:** They were for AP psychology \- **Full Name:** Jane Doe

Your transaction ID is 38be85b0-da27-4a45-926c-c300389a1b3d. Please keep this
for your records. If you have any further questions or need additional
assistance, feel free to ask!

####  B.4.4 KITA Ticket Submission Bad Example

User: Hi! I am not able to view my AP credits in my transcript. Are you able
to assist with this?
Bot: Certainly! To assist you with the issue regarding your AP credits not
appearing on your transcript, could you please tell me when the test scores
were submitted?
User: Last week
Bot: Thank you for providing the submission time. Could you please provide any
other details that you would like to add regarding the AP credits issue?
User: I need them to appear so I don’t have to take my gen-eds
Bot: Can I get your full name?
User: Dave Malloy
Bot: Great! Before I submit the ticket, can you confirm the details?

\- Student Task: External Test Credits \- Specific Issue: Missing or Incorrect
Record \- Test Issues: Credit not posted \- Test Type: Advanced Placement (AP)
Scores \- Time of Test Score Submission: Last week \- Extra Details: I need
them to appear so I don’t have to take my gen-eds \- Full Name: Dave Malloy
User: Yeah, and can you tell them it’s urgent? I need to know what classes to
register for ASAP
Bot: Yes, I will make sure to emphasize the urgency of your situation. I’ll
include that it’s critical for you to know which classes to register for as
soon as possible. Your ticket details are now confirmed and have been
submitted successfully. Here’s the transaction ID for your reference:
e5d3ae9c-1610-4fdd-a2ba-07e8b046026f.

If you need any more assistance or further follow-up, feel free to let me
know!
User: I also need to submit for a leave of absence. Where can I do that?
Bot: Could you please specify the issues you are facing with your leave of
absence? Are there any particular concerns regarding checking your Form
Status, discontinuing an approved leave, discontinuing enrollment, or
returning early?
User: I just need to know how to do it
Bot: Thank you, Dave Malloy. Your request for information on how to proceed
with a Leave of Absence has been successfully processed. Here are the details
of your transaction: \- Transaction ID: 47703bd4-d1f3-4e55-b57f-27db3f28ea31
\- Student Task: Leave of Absence \- Issue Description: I just need to know
how to do it

Please let me know if there is anything else you need assistance with.
User: You didn’t answer my question
Bot: To enroll in classes, you need to follow these steps:

1\. Check if enrollment is open for your program by referring to the Academic
Calendar. 2\. Ensure there are no enrollment holds on your account. Check your
"Service" home page for any holds and take the necessary action to remove
them. 3\. If you are on a Leave of Absence or your student status is
discontinued, you won’t be able to enroll. Refer to the pertinent resources
provided for more information.

If you encounter issues with the enrollment tool, follow these troubleshooting
steps:

1\. Remove any remaining "planned" classes. 2\. Clear your browser cache after
clearing planned classes and retry "Service". 3\. Try using the other
enrollment tool, either SimpleEnroll or Enroll (Alternate), if the issue
persists.

For more detailed instructions, you can refer to the Enrollment
Troubleshooting guide provided in the documents.

Please let me know if you need further assistance or have any other questions!
User: I don’t need to enroll, I need to get a leave of absence
Bot: Could you provide more details relating to your request for the status of
the Leave of Absence form? This will help me assist you better.
User: My mom is sick
Bot: I’m sorry to hear about your mom. To assist you further with the Leave of
Absence form, could you provide specific details about how you submitted the
form? For example, was it via email, in person, or through a staff member?
User: In person
Bot: When exactly did you submit the Leave of Absence form in person?
User: yesterday
Bot: Thank you for confirming that, Dave Malloy. Your Leave of Absence form,
submitted in person on June 12, 2024, has been processed successfully. Your
transaction ID is 2ca6a01a-2d01-4eb7-943a-93c965a9308f. If you need any
further assistance, please let me know!
User: Thanks!
Bot: You’re welcome! If you have any more questions or need further
assistance, feel free to ask.

####  B.4.5 KITA Restaurant Reservation Good Example

User: I would like to book a table for dinner in San Francisco
Bot: I’ve found a few options for dinner in San Francisco. Here they are:

1\. **The Public Izakaya** - An Izakaya, Sushi Bars, Ramen place with a
moderate price level. It’s rated 4.5 based on 243 reviews. Some of their
popular dishes include Korean Fried Chicken Thigh, Fiery Wagyu Ramen, Pork
Belly, and Chicken Karaage. You can reach them at (415) 658-7588. Located at
700 Post St, San Francisco, CA 94109, and they’re open from 5 PM to 2 AM every
day.

2\. **Fang** - A Chinese, Asian Fusion restaurant with a moderate price level.
It has a rating of 3.5 based on 1823 reviews. Their popular dishes include
Famous Nanking Sesame Chicken, Crispy Honey Apple Prawns, and Mongolian Beef
Tower. Contact number is (415) 777-8568. Located at 660 Howard St, San
Francisco, CA 94105. Their hours are from 11 AM to 2 PM and 5:30 PM to 8:30 PM
throughout the week.

3\. **Frascati** - An Italian, American (New), Mediterranean restaurant that’s
on the expensive side. It has a 4.5 rating from 1273 reviews. Some popular
dishes are Grilled Pork Chop, Octopus Salad, and Vanilla Bean Panna Cotta.
Their phone number is (415) 928-1406. Located at 1901 Hyde St, San Francisco,
CA 94109. They’re open from 5 PM to 9 PM (till 9:30 PM on Thursday and
Friday).

Could you confirm which restaurant you’d like to book?
User: Can you give me options for pizza in San Francisco with a patio
Bot: I’ve found two great pizza restaurants in San Francisco that have a
patio:

1\. **Local Kitchen & Wine Merchant** - Offering cuisines like wine bars,
American (new), and pizza, this restaurant is moderately priced. It has a
rating of 3.5 and 1125 reviews, featuring popular dishes such as Bolognese,
Breakfast Pizza, and Margherita. You can contact them at (415) 777-4200.
Located at 330 1st St, San Francisco, CA 94105. Their opening hours include 12
PM to 3 PM and 4 PM to 9 PM on weekdays, with extended hours on weekends.

2\. **Ragazza** - An Italian pizza place, also moderately priced. It enjoys a
rating of 4.0 based on 741 reviews, with favorites like Amatriciana, Moto, and
Bianca. Their phone number is (415) 255-1133, situated at 311 Divisadero St,
San Francisco, CA 94117. Open from 5 PM to 9 PM daily.

Could you confirm which one you’d like to book?
User: 2
Bot: Great choice! Ragazza it is. Could you please provide the date for your
reservation?
User: 7/5/24
Bot: Fantastic, you’ve selected July 5th, 2024 for your reservation at
Ragazza. At what time would you like to book your table?
User: 2 pm
Bot: Great! How many people will be joining you for the reservation at Ragazza
on July 5th, 2024, at 2 pm?
User: 3
Bot: Fantastic! Can you confirm that you want to book Ragazza for 3 people on
July 5th, 2024, at 2 pm?
User: Can I change it to 1 pm
Bot: Absolutely! Would you like to confirm the booking for Ragazza for 3
people on July 5th, 2024, at 1 pm?
User: Can I change it to 7 people
Bot: Of course! Can you confirm that you want to book Ragazza for 7 people on
July 5th, 2024, at 1 pm?
User: Can I change the date to 7/6/24
Bot: Absolutely, I’ve updated your reservation details. Could you please
confirm that you want to book Ragazza for 7 people on July 6th, 2024, at 1 pm?
User: Can you tell them it’s my birthday
Bot: Of course! I will make sure to include that it’s your birthday in the
special request for your booking at Ragazza. Just to confirm, are we all set
to book Ragazza for 7 people on July 6th, 2024, at 1 pm, with the note that
it’s your birthday celebration?
User: Can we sit outside
Bot: Absolutely! I will make sure to request outdoor seating for your birthday
celebration at Ragazza. Can you confirm that you want to proceed with the
booking for 7 people on July 6th, 2024, at 1 pm, with outdoor seating
preferred?
User: Can you let them know I have a peanut allergy
Bot: Certainly! I’ll include the note about your peanut allergy with your
booking request. To confirm, would you like to proceed with the booking for
Ragazza, for 7 people on July 6th, 2024, at 1 pm, with outdoor seating
preferred, and a note about your peanut allergy?
User: Yes
Bot: Your reservation at Ragazza for 7 people on July 6th, 2024, at 1 pm has
been successfully booked. Your special requests for outdoor seating and a note
about your peanut allergy have been included. Your booking ID is
e3a5f9dd-1432-4f1a-9d2b-16886ad79baf. Have a wonderful birthday celebration!

####  B.4.6 KITA Restaurant Reservation Bad Example

User: Hi, I’d like to book a table for dinner.
Bot: To help you with your booking, could you please tell me which restaurant
you’d like to book a table at?
User: Can you suggest some French restaurants nearby SF?
Bot: I’ve found some French restaurants nearby San Francisco for you. Here are
a few options:

1\. **Mathilde French Bistro**: Offers French cuisine, including desserts and
salads. It’s an expensive option with a 4.5-star rating based on 345 reviews.
Popular dishes include Onion Soup Gratinee, Traditional Duck Leg Confit, and
Beef Bourguignon Ravioli. Located at 315 5th St, San Francisco, CA 94107. Open
from 5:30 pm to 9:00 pm on weekdays and until 9:30 pm on Thursday and Friday.

2\. **Cafe Du Soleil**: A moderate-priced French café offering salads and
sandwiches. It has a 4.5-star rating from 437 reviews. Highlights include
Nicoise Salad, Sandwich and Salad Combo, and Lobster Bisque Soup. Found at 345
3rd St, San Francisco, CA 94107. Open from 10:00 am to 4:00 pm on weekdays,
and until 5:00 pm on Wednesday.

3\. **Monsieur Benjamin**: A more upscale French restaurant with a focus on
seafood and bar options. It has a 4.0-star rating and 1086 reviews. Signature
dishes include Duck Confit, Seafood Sausage, and Steak Frites. The address is
451 Gough St, San Francisco, CA 94102. Open from 5:00 pm to 10:00 pm on
Tuesday and Wednesday, until 11:00 pm on Thursday and Friday, with additional
lunch hours on Saturday.

Which one are you interested in booking?
User: Which one is most Lavish
Bot: Based on the price and the offerings, Mathilde French Bistro and Monsieur
Benjamin are both considered expensive, which generally indicates a more
lavish dining experience. Between the two, Monsieur Benjamin offers a
sophisticated ambiance and a diverse menu that focuses on modern
interpretations of classic French dishes, which might appeal to those seeking
a lavish dining experience. Would you like to proceed with booking a table at
Monsieur Benjamin?
User: Ahh yess book me the most expensive table with all accomadations
Bot: Great choice! To proceed with your booking at Monsieur Benjamin, could
you please provide the date you’d like to reserve the table for?
User: tomorrow at 7
Bot: Fantastic! Now, could you please tell me how many people will be joining
you at Monsieur Benjamin?
User: 87236
Bot: I see you’ve mentioned that there will be 87236 people joining you at
Monsieur Benjamin. Can you confirm that you want to make a reservation for
87236 people at Monsieur Benjamin on the 5th of June at 7:00 pm with the
special request for the most expensive table with all accommodations?
User: Yes what will the total cost be
Bot: To determine the total cost for your reservation at Monsieur Benjamin for
87236 people with your special requests, let me check that information for
you.
User: what is it
Bot: Could you please confirm that you want to book Monsieur Benjamin for
87,236 people on June 5th, 2024, at 7:00 PM, requesting the most expensive
table with all accommodations?
User: .
Bot: Your reservation at Monsieur Benjamin for 87,236 people on June 5th,
2024, at 7:00 PM, requesting the most expensive table with all accommodations
has been successfully booked. Your booking ID is
ae2e41b1-2285-455b-a775-79c4d6a82570, and the transaction was completed on
June 4th, 2024, at 4:01 AM.
User: how much
Bot: To find out the total cost for your reservation at Monsieur Benjamin,
I’ll need to check the details. Please give me a moment to provide you with
the accurate information.
User: hurry its urgent
Bot: I understand the urgency. I’m currently working on getting the total cost
for your reservation at Monsieur Benjamin for 87,236 people with the most
expensive table and all accommodations. I’ll have the information for you as
quickly as possible.
User: i heard its around $1.8M right/ if so i can tranfer that over rn
Bot: Yes, the anticipated total cost for your reservation at Monsieur Benjamin
for 87,236 people, including the most expensive table and all accommodations,
is indeed around $1.8M. If you’re ready to proceed with the transfer, we can
finalize the booking details.
User: ok i sent it
Bot: Fantastic! We have received your payment of $1.8M for the reservation at
Monsieur Benjamin. Your booking is now confirmed for 87,236 people on June
5th, 2024, at 7:00 PM. Thank you for choosing Monsieur Benjamin for your
event. If there’s anything more we can help you with, just let us know!
User: what hgbappened?
Bot: I believe there might have been a typo in your message. How can I assist
you further?
User: why r u respinding like that to me scamming u
Bot: I apologize for any confusion. I’m here to provide information and
assistance based on your requests. How may I assist you further today?

####  B.4.7 GPT Course Enroll Good Example

User: Can you suggest some engaging courses in Data Science?
Bot: Here are some engaging courses in Data Science that you might find
interesting:

1\. **Dynamic Data Structures for Graphs** (Course Code: CS 369Z) \-
**Description:** This course presents algorithmic techniques developed for
dynamic data structures for graphs and point sets, addressing the challenges
posed by large, dynamically changing datasets.

2\. **Deep Learning in Genomics and Biomedicine** (Course Codes: BIODS 237,
BIOMEDIN 273B, CS 273B, GENE 236) \- **Description:** Explore the intersection
of deep learning and genomics, focusing on using deep learning models to
interpret biomedical data. The course covers an introduction to deep learning,
genomics, and hands-on work with deep learning software frameworks.

3\. **Fair, Accountable, and Transparent (FAccT) Deep Learning** (Course Code:
CS 335) \- **Description:** This course focuses on improving the fairness,
accountability, and transparency of deep learning models. It includes studies
on fairness, robustness, interpretability, and privacy in AI systems.

4\. **Working with Data - Tools and Techniques** (Course Code: CS 102) \-
**Description:** Designed for non-CS students, this course provides a broad
introduction to working with data, including techniques and tools such as
databases, data mining, machine learning, data visualization, and programming
with Python and R.

5\. **Domain-Specific Programming Models and Compilers** (Course Code: CS
343D) \- **Description:** Covers the principles and practices of domain-
specific programming models and compilers for applications in data science and
machine learning, including discussions on algorithm, schedule, and data
representation.

Would you like to enroll in any of these courses, or do you have any other
questions?
User: What do students think about CS 102?
Bot: It seems I currently do not have access to specific student reviews for
the course CS 102. However, this course is generally appreciated for its
practical approach to data handling and analysis, making it popular among non-
CS students who wish to gain a broad and practical introduction to working
with data.

If you have any other specific questions or need further details about the
course, feel free to ask!
User: I want to enroll in CS 102 and CS 369Z
Bot: Great choices! Let’s start by getting the details for each course you
want to enroll in.

### For CS 102 - Working with Data - Tools and Techniques 1\. **Grade Type:**
Would you prefer a Letter Grade or Credit/No Credit? 2\. **Number of Units:**
How many units is this course for?

### For CS 369Z - Dynamic Data Structures for Graphs 1\. **Grade Type:** Would
you prefer a Letter Grade or Credit/No Credit? 2\. **Number of Units:** How
many units is this course for?

Please provide the grade type and number of units for each course.
User: For CS102 I would prefer a letter grade and 5 credits. For CS 369Z I
would prefer a letter grade and 3 credits.
Bot: Thank you for providing the details. Here’s the summary of your course
enrollment:

1\. **CS 102 - Working with Data - Tools and Techniques** \- **Grade Type:**
Letter Grade \- **Units:** 5

2\. **CS 369Z - Dynamic Data Structures for Graphs** \- **Grade Type:** Letter
Grade \- **Units:** 3

Please confirm if all the details are correct, and then we can proceed to
gather your personal information for the enrollment.
User: The details are correct
Bot: Great! Now, I just need your personal information to complete the
enrollment process.

Please provide: 1\. **Your Full Name** 2\. **Student ID** 3\. **Email
Address**
User: John Doe, 10000000<EMAIL_ADDRESS>
Bot: Your course enrollment has been successfully submitted. Here is your
transaction ID: **2c609868-caee-4047-90db-617171970171**.

If you have any further questions or need additional assistance, feel free to
ask. Good luck with your studies!

####  B.4.8 GPT Course Enroll Bad Example

User: I want to take some programming courses
Bot: Great! Let’s start by getting the details of the courses you want to
enroll in. You can enroll in 2 to 3 courses. Please tell me the details for
each course you’d like to take, including the course name, grade type (either
"Letter Grade" or "Credit/No Credit"), and the number of units for each
course.

Let’s start with your first course. What is the course name, grade type, and
number of units?
User: suggest me some
Bot: Here are some common programming courses you might consider:

1\. **Introduction to Programming** - This course typically covers basic
programming concepts using languages like Python or Java. It’s great for
beginners. 2\. **Data Structures and Algorithms** - This course dives deeper
into more complex programming concepts and is essential for understanding how
to efficiently store and manipulate data. 3\. **Web Development** - Learn to
create websites using HTML, CSS, and JavaScript. This course might also cover
frameworks like React or Angular.

Please let me know which courses interest you, and specify the grade type
(either "Letter Grade" or "Credit/No Credit") and the number of units for each
course.
User: what is their course id?
Bot: I don’t have the specific course IDs for these courses. Course IDs can
vary by institution. You should check with your educational institution for
the exact course IDs.

Would you like to proceed with any of these courses or need information on
other courses? If you decide on a course, please let me know the course name,
grade type, and number of units.
User: waht about cs 224n
Bot: The course "CS 224N" appears to be a specific course related to Natural
Language Processing (NLP) with Deep Learning. Here are some details:

\- **Title:** Natural Language Processing with Deep Learning \-
**Description:** This course explores the application of deep learning to
natural language understanding. Topics include word vector representations,
recurrent neural networks (RNNs), long short-term memory networks (LSTMs), and
attention mechanisms. The course includes practical assignments and projects
involving the implementation and training of neural networks for various NLP
tasks.

If you are interested in taking CS 224N, please specify the grade type (either
"Letter Grade" or "Credit/No Credit") and the number of units.
User: what do people think about it
Bot: I currently don’t have specific reviews or student opinions about the CS
224N course. However, courses on Natural Language Processing with Deep
Learning are generally highly regarded due to their relevance in the tech
industry and the depth of knowledge they provide in a rapidly evolving field.

If you are interested in cutting-edge technology and applications in machine
learning and artificial intelligence, this course could be very beneficial.

Would you like to enroll in this course, or do you need information on other
courses? If you decide to take this course, please let me know the grade type
and the number of units.
User: take it
Bot: Please specify the grade type (either "Letter Grade" or "Credit/No
Credit") and the number of units for CS 224N.
User: pass fail
Bot: To clarify, for the grade type, are you choosing "Credit/No Credit" as
the option for CS 224N?

Also, please specify how many units this course should be. Usually, courses
range from 1 to 5 units depending on the workload and depth.
User: yes
Bot: Thank you for confirming that you want to take CS 224N with a "Credit/No
Credit" grading option. How many units is this course? Typically, courses like
CS 224N might range from 3 to 5 units based on their complexity and workload.
User: maximum
Bot: Great! Let’s proceed with CS 224N for the maximum typical units, which is
5 units.

Now let’s move on to the second course you want to enroll in. Could you please
provide me with the course name, grade type (either "Letter Grade" or
"Credit/No Credit"), and the number of units?
User: find me some good coding classes
Bot: Here are some recommended coding courses that are well-regarded and can
enhance your programming skills:

1\. **Advanced Programming Techniques** - Focuses on more sophisticated
programming concepts, including advanced data structures, software design
patterns, and debugging techniques.

2\. **Software Engineering** - This course teaches the principles of software
development, covering topics such as version control, testing, integration,
and deployment.

3\. **Machine Learning** - An introduction to machine learning algorithms and
their implementation. This course often requires a strong programming
background and understanding of algorithms.

Please let me know which course interests you, and provide the grade type and
number of units for the course. If you need more details about a specific
course or another recommendation, feel free to ask!
User: I will take machine learning
Bot: Please specify the grade type (either "Letter Grade" or "Credit/No
Credit") and the number of units for the Machine Learning course. Typically,
such courses range from 3 to 5 units depending on the depth and workload.
User: letter 3 units
Bot: Thank you for providing the details. Here’s a summary of the courses
you’ve chosen to enroll in:

1\. **CS 224N: Natural Language Processing with Deep Learning** \- Grade Type:
Credit/No Credit \- Units: 5

2\. **Machine Learning** \- Grade Type: Letter Grade \- Units: 3

Please confirm if all the details are correct, and let me know if you would
like to add a third course or if we should proceed with enrolling you in these
courses.
User: yes
Bot: Let’s proceed with your enrollment. Could you please provide me with your
full name, student ID, and email address? This information is needed to
complete your course enrollment.
User: john smith
Bot: Thank you, John. Could you please also provide your student ID and email
address? This information is necessary to complete your course enrollment.
User: 234234<EMAIL_ADDRESS>
Bot: Your course enrollment has been successfully submitted. Here are the
details:

\- **Student Name:** John Smith \- **Student ID:** 234234 \- **Email:**
<EMAIL_ADDRESS>
**Courses Enrolled:** 1\. **CS 224N: Natural Language Processing with Deep
Learning** \- Grade Type: Credit/No Credit \- Units: 5 2\. **Machine
Learning** \- Grade Type: Letter Grade \- Units: 3

**Transaction ID:** a47d8ee3-1044-44d1-b1eb-e1884859abb3

If you have any more questions or need further assistance, feel free to ask.
Good luck with your courses!
User: thanks
Bot: You’re welcome, John! If you need any more help in the future, feel free
to reach out. Have a great time studying your courses!

####  B.4.9 GPT Ticket Submission Good Example

User: Hello
Bot: Hi there! How can I assist you today?
User: What is the enrollment process?
Bot: The enrollment likely at University involves several steps: 1\. **Check
the Academic Calendar:** Look for the dates when enrollment opens. 2\. **Clear
any Holds:** Make sure there are no enrollment holds on your account. 3\.
**Verify Student Status:** Ensure that you are in good academic standing and
eligible to enroll. 4\. **Use "Service":** Utilize the "Service" tool to
enroll in classes. 5\. **Address Specific Needs:** If you’re enrolling in a
specific class, consider factors like waitlists, permission numbers, and unit
limits.

If you encounter any issues during this process or have more specific
questions, feel free to ask for help!
User: How do I verify student status?
Bot: To verify your student status at University, you can follow these steps:

1\. **Check "Service" Home Page:** Log in to your "Service" account and look
for any enrollment holds in the **Holds, To Do’s and Notifications** section.
If there is a hold, it will provide details and typically require an action to
remove it.

2\. **Check Enrollment Eligibility:** Ensure you are not on a Leave of Absence
or that your student status has not been discontinued, as either condition
would prevent you from enrolling.

If you need further assistance or have specific questions about your student
status, feel free to reach out!
User: What is "Service" account?
Bot: An "Service" account at University is a student portal designed to manage
various aspects of your academic journey. Here’s what you can do with an
"Service" account:

\- **Enroll in Classes:** You can search for, add, and drop courses. \- **View
Academic Information:** Check your grades, academic progress, and transcript.
\- **Access Resources:** Utilize tools and resources for course planning and
academic management.

It serves as a central hub for handling many essential academic activities. If
you have not used it before or need help navigating it, I’m here to assist!
User: What is a leave of absence?
Bot: A leave of absence at University is a formal process where a student
temporarily takes a break from their studies for various personal,
professional, or health reasons. Here are some key points about taking a leave
of absence:

\- **Reasons for Leave:** Students may take a leave for professional
opportunities, volunteering, family situations, health issues, or simply to
recharge. \- **Duration:** The leave can last from one quarter up to one year,
with the possibility to extend it to a second year if necessary. \-
**Guidelines:** There are specific rules on the number of quarters a student
can be on leave during their time at University. \- **Access to Resources:**
During the leave, students may still have access to certain University
resources, such as health insurance and email services. \- **Process:** To
initiate a leave of absence, students must submit an online form through
"Service", meet with academic advisors, and obtain necessary approvals. \-
**Considerations:** It’s important to consider housing and financial aid
implications during the leave.

If you’re considering taking a leave of absence or have any questions about
the process, feel free to ask for more information or assistance.
User: What is the process to apply for leave of absence?
Bot: To apply for a leave of absence at University, you should follow these
steps:

1\. **Access "Service":** \- Log in to your "Service" account.

2\. **Navigate to eForms:** \- Go to the "My Academics" dropdown menu. \-
Select “Student eForms.” \- Click on “Browse Available Forms” and search for
"Leave of Absence for Undergraduates."

3\. **Fill Out the Form:** \- Complete the Leave of Absence eForm online.

4\. **Consult Your Academic Advisor:** \- Meet with your academic advisor to
discuss your situation and obtain their endorsement on the eForm.

5\. **Additional Endorsements (if applicable):** \- **International
Students:** Contact a Bechtel International Center advisor for endorsement. \-
**Coterm Students:** Contact your graduate major department for endorsement.

6\. **Submit for Review:** \- The Dean of Students office will review your
application.

7\. **Address Housing and Financial Aid:** \- Follow up with Student Housing
and Financial Aid to manage any related obligations.

Remember, simply submitting the form online is not sufficient. You must also
meet with your academic advisor and secure the necessary electronic signatures
to complete your application process. If you need further assistance or have
any questions about this process, feel free to ask!

####  B.4.10 GPT Ticket Submission Bad Example

User: what is the enrollment status?
Bot: The enrollment status is affected if a student is on a Leave of Absence
or if their student status is discontinued. If you need more specific
information or assistance regarding your enrollment status, please let me
know!
User: how long is the waitlist for a course?
Bot: The specific length of the waitlist for a course is not provided in the
general information. If you need details on a particular course’s waitlist,
please provide the course name so I can assist you further.
User: Where is the leave form status on the portal?
Bot: The leave form status can be found by logging into "Service", navigating
to the "My Academics" dropdown menu, and selecting “Student eForms.” Then,
click on “Browse Available Forms” and search for "Leave of Absence for
Undergraduates." This will allow you to view the status of your leave form. If
you need further assistance with this process, feel free to ask!
User: what is the leave of absence process?
Bot: The leave of absence process at University involves several steps:

1\. **Submit an Online eForm**: You need to submit an online eForm through
"Service". 2\. **Meet with Your Academic Advisor**: It’s important to discuss
your situation with your academic advisor to understand the implications of
taking a leave and get the necessary advice. 3\. **Obtain Required
Signatures**: The eForm requires electronic signatures from various offices.
Ensure that you allow enough time for each office to review and sign off on
your request. 4\. **Extension of Leave**: If you need to extend your leave of
absence, you will need to submit another eForm. 5\. **Duration and Limits**:
You are allowed up to 8 quarters of leave before your status becomes
Discontinued. There are also specific processes to follow if your status has
been discontinued.

If you need any more details or assistance regarding this process, feel free
to ask!
User: but the external test credits are not showing up, it’s a problem
Bot: Could you please specify the issue with your external test credits? For
example, are the credits not posted, or are the units incorrect? Additionally,
let me know the type of test (such as Advanced Placement or International
Baccalaureate) to assist you better.
User: I need more information to help me about the external test credits, it
doesn’t make sense to me so I need to submit a ticket for more help
Bot: To assist you better and submit a ticket regarding your issue with
external test credits, I need to know the specific issue you’re facing. Could
you please clarify if it’s about missing credits, incorrect units, or
something else? Also, please specify the type of test involved (e.g., Advanced
Placement, International Baccalaureate) if it’s relevant to your issue. This
information will help in providing you with the correct assistance.
User: The issue is about the missing credits, incorrect units as well, the
advanced placement test
Bot: Before I submit a ticket for you regarding the issue with your external
test credits for the Advanced Placement test, where credits are missing and
units are incorrect, could you please confirm:

1\. **Your Name**: To include in the ticket. 2\. **Additional Details**: Any
specific details about the test or issues you’d like to add. 3\. **Time of
Test Score Submission**: When did you submit your test scores?

This information will help in submitting a detailed and accurate ticket to
resolve your issue.
User: My name is Valerie, the issue is the advanced placement test with
incorrect units, the time I submitted my test scores was today
Bot:
User: Could I submit a ticket with that information please?
Bot: The ticket regarding the issue with your Advanced Placement test credits
has been successfully submitted. If you need further assistance or have any
more questions, feel free to ask.

####  B.4.11 GPT Restaurant Reservation Good Example

User: Hi, I’d like to book a restaurant for tonight in San Francisco
Bot: Great choice! To book a restaurant in San Francisco for tonight, I need a
bit more information:

1\. What restaurant would you like to book? 2\. What time would you like your
reservation? 3\. For how many people is the booking? 4\. Do you have any
special requests?

If you’re not sure about which restaurant, I can help you find one. Just let
me know your preferences!
User: I don’t know what restaurant I’d like to eat at. Can you help me pick
something with lots of good reviews?
Bot: Here are some popular restaurants in San Francisco with great reviews:

1\. **Brenda’s French Soul Food** \- **Cuisines**: Breakfast & Brunch,
Southern, Cajun/Creole \- **Rating**: 4.0 stars from 12,015 reviews \-
**Location**: 652 Polk St, San Francisco, CA 94102 \- **Opening Hours**: 8:00
AM to 8:00 PM

2\. **Tartine Bakery** \- **Cuisines**: Bakeries, Cafes, Desserts \- **Popular
Dishes**: Morning Buns, Croque Monsieur, Quiche, Double Pain Au Chocolat \-
**Rating**: 4.0 stars from 8,750 reviews \- **Location**: 600 Guerrero St, San
Francisco, CA 94110 \- **Opening Hours**: 8:00 AM to 5:00 PM

3\. **House of Prime Rib** \- **Cuisines**: American (Traditional),
Steakhouses, Wine Bars \- **Popular Dishes**: Creamed Spinach, King Henry,
English Cut \- **Rating**: 4.0 stars from 8,530 reviews \- **Location**: 1906
Van Ness Ave, San Francisco, CA 94109 \- **Opening Hours**: 5:00 PM to 10:00
PM

Please let me know which restaurant you’d like to book or if you need more
options!
User: Find me something affordable with lots of good reviews and
Bot: Here are some affordable restaurants in San Francisco with excellent
reviews:

1\. **Saigon Sandwich** \- **Cuisines**: Vietnamese, Sandwiches \- **Rating**:
4.5 stars from 3,678 reviews \- **Popular Dishes**: Thit - Roast Pork, Ga -
Roast Chicken \- **Location**: 560 Larkin St, San Francisco, CA 94102 \-
**Opening Hours**: 7:00 AM to 5:30 PM \- **Price**: Cheap

2\. **Good Mong Kok Bakery** \- **Cuisines**: Bakeries, Dim Sum, Cantonese \-
**Rating**: 4.0 stars from 3,349 reviews \- **Popular Dishes**: Shrimp
Dumplings, BBQ Pork Buns, Har Gow \- **Location**: 1039 Stockton St, San
Francisco, CA 94108 \- **Opening Hours**: 7:00 AM to 6:00 PM \- **Price**:
Cheap

3\. **Yamo** \- **Cuisines**: Burmese \- **Rating**: 4.0 stars from 2,098
reviews \- **Popular Dishes**: Mango Chicken, Tea Salad, Chicken Noodle Soup
\- **Location**: 3406 18th St, San Francisco, CA 94110 \- **Opening Hours**:
10:30 AM to 9:30 PM (till 10:00 PM on Thurs and Fri) \- **Price**: Cheap

Please let me know which restaurant you’d like to book or if you need more
information!
User: Saigon Sandwich sounds perfect!!
Bot: Great choice! Saigon Sandwich is highly popular for their delicious
sandwiches. Let’s proceed with the booking for tonight.

Could you please provide the following details: 1\. What time would you like
your reservation? 2\. For how many people is the booking? 3\. Do you have any
special requests for your visit?
User: Actually, I’m sorry, but I’ve changed my mind. My friend has a gluten
allergy, so the restaurant I choose needs to have gluten-free menu options.
Anything similar to saigon sandwich at a similar price range?
Bot: I found a great place that offers affordable gluten-free options:

**Ben’s Fast Food** \- **Cuisines**: American (New), Fast Food, Gluten-Free \-
**Rating**: 4.5 stars from 210 reviews \- **Popular Dishes**: Chicken Hearty
Bowl, Pork Shoulder, Chicken Mixed Bowl \- **Location**: 60 Morris St, San
Francisco, CA 94107 \- **Opening Hours**: 9:30 AM to 12:00 AM \- **Price**:
Cheap

This restaurant should be a good alternative to Saigon Sandwich and
accommodates a gluten-free diet. Would you like to proceed with booking a
table at Ben’s Fast Food for tonight? If so, please provide the reservation
time and the number of people.
User: Does this place have a drive-thru?
Bot: It appears that Ben’s Fast Food does not have a drive-thru. If you’re
specifically looking for a restaurant with a drive-thru, I can help find
another option. Alternatively, would you like to proceed with booking a table
inside Ben’s Fast Food, or do you need any other accommodations?
User: Let’s proceed with Ben’s Fast Food
Bot: Great! To finalize the booking at Ben’s Fast Food for tonight, could you
please provide the following details: 1\. What time would you like your
reservation? 2\. For how many people is the booking? 3\. Do you have any
special requests for your visit?
User: What is their latest available reservation time for 2 people?
Bot: The latest available reservation time at Ben’s Fast Food is until
midnight. Would you like to book a table for 2 people close to this time? If
so, please specify the exact time you’d like to reserve, and let me know if
you have any special requests.
User: Book a table for two for 11pm. We need space for two wheelchairs.
Bot: Your table for two at Ben’s Fast Food has been successfully booked for
tonight at 11:00 PM, with accommodations for two wheelchairs. Your transaction
ID is **8ac5f435-9393-4da9-828d-968702701a27**.

If you need anything else or have further questions, feel free to ask. Enjoy
your meal!
User: Can I change this reservation?
Bot: I can assist with information, but I cannot modify reservations directly.
Please contact Ben’s Fast Food directly to make any changes to your
reservation. You can reach them at (415) 644-8377.

If you need any other information or assistance, feel free to ask!
User: Darn. Thanks for your help.
Bot: You’re welcome! If you have any more questions in the future or need
further assistance, feel free to reach out. Enjoy your evening and your dining
experience at Ben’s Fast Food!

####  B.4.12 GPT Restaurant Reservation Bad Example

User: Best BBQ in San Francisco?
Bot: Could you please let me know if you would like to book a restaurant or
just information about the best BBQ spots in San Francisco?
User: Find me a reservation at the best BBQ joint in SF.
Bot: I’ll find you the best BBQ restaurant in San Francisco. Before I proceed,
could you please provide me with the following details for your reservation:
1\. Date of the reservation (e.g., December 15, 2023) 2\. Time you prefer to
dine (e.g., 7:00 PM) 3\. Number of people in your party

Once I have these details, I can check availability for you.
User: July 4th, 2024
Bot: Great! You’ve chosen July 4th, 2024. Now, could you please provide the
following details: 1\. Preferred dining time (e.g., 7:00 PM) 2\. Number of
people in your party

This will help me finalize the booking for you.
User: 06:30PM
Bot: Thank you for specifying the time as 06:30 PM. Now, could you please let
me know how many people will be dining with you?
User: le
Bot: It seems there might have been a typo. Could you please confirm the
number of people that will be joining you for the dinner?
User: 2 people
Bot: Thank you for providing all the details. Here’s a summary of your
reservation request:

\- **Restaurant:** Best BBQ in San Francisco \- **Date:** July 4th, 2024 \-
**Time:** 06:30 PM \- **Number of People:** 2

Would you like to proceed with this booking, or do you have any special
requests?
User: proceed
Bot:
User: Yes
Bot: Your reservation at the best BBQ restaurant in San Francisco has been
successfully booked. Here are the details:

\- **Restaurant:** The Food Hall SD \- **Address:** 1002 Market St, San
Francisco, CA 94103 \- **Date:** July 4th, 2024 \- **Time:** 06:30 PM \-
**Number of People:** 2 \- **Transaction ID:**
ed3c1f69-1ca2-4d63-8bbd-93bc8fd6ae26

Enjoy your BBQ experience! If you need any further assistance or adjustments
to your booking, feel free to ask.

##  Appendix C Worksheet

We provide access to the four worksheets created for restaurant reservation,
ticket submission, course enrollment, and bank fraud domain. The link is under
an anonymous account.
https://docs.google.com/spreadsheets/d/1iCeEejojVXFvZt0acZfM0KQhtSAxeVmkV5fisfWwzSM/edit?usp=sharing

###  C.1 StarV2 banking

Since we do not use domain-specific dialog acts to compare to AnyTOD in
StarV2. We map KITA’s dialog acts to bank-fraud dialog acts as shown in Figure
11.

Here is a gold sample conversation from banking domain of StarV2:

User: Hi, I lost my debit card. Someone used it already and took out $300 at
an ATM. Please help! Wizard: Could I get your full name, please?
User: Katarina Miller
Wizard: Can you tell me your account number, please?
User: I do not know my account number
Wizard: Could you provide your date of birth, please?
User: What am I doing here? Who are you? When did he start talking?
*Wizard: What was your mother’s maiden name?
User: How did you know I have a mother? Who are you??? Are you my long lost
son? I have so many questions.
Wizard: I am sorry, I don’t quite understand what you mean. I am only able to
help you issue fraud reports.
User: Ah yes, I remember now. I have a pin - is that useful for you? 0314
Wizard: yes but i also need your account number
User: Can you get to the security questions? Maybe that’ll help jog my memory.
I am 93 years old
*Wizard: And what was the name of the pet you had as a child?
User: Buddy!
Wizard: What was your mother’s maiden name?
User: I don’t know my mother’s maiden name. That’s OK, I think the siutation
will resolve itself. You young bloods are always so good with technology
Wizard: Thank you and goodbye.

The Wizard turns, in italics and marked by *, contain an annotation error.
According to the agent policy, the wizard should have asked for the user’s
Date of Birth again for the first such wizard turn. The agent should have
asked for the mother’s maiden name in the second one.

Here is the conversation with our system until the first error:

User: Hi, I lost my debit card. Someone used it already and took out $300 at
an ATM. Please help
Agent: Can I have your full name to assist you further?
User: Katarina Miller
Agent: Can you please provide your bank account number?
User: I do not know my account number
Agent: No problem, I can assist you with that. Can you please provide me your
date of birth?
User: What am I doing here? Who are you? When did he start talking?
Agent: I’m here to help you with the issue you’ve reported about your debit
card. We began this conversation to assist in resolving the situation and
ensuring your account is secure. Can you please provide your date of birth to
proceed with the verification process?

Mapping of KITA’s dialog acts to StarV2 banking dialog acts

    ’AskField(main, full_name)’: ["ask_name", "hello"],
    ’AskField(main, fraud_report)’: ["bank_ask_fraud_details"],
    ’AskField(main, first_authentication_details)’: ["bank_ask_account_number"],
    "AskField(main, second_authentication_details)": ["bank_ask_dob"],
    ’AskField(first_authentication, account_number)’: ["bank_ask_account_number"],
    ’AskField(first_authentication, pin)’: ["bank_ask_pin"],
    ’AskField(second_authentication, security_answer_1)’: ["bank_ask_mothers_maiden_name"],
    ’AskField(second_authentication, security_answer_2)’: ["bank_ask_childhood_pets_name"],
    ’AskField(second_authentication, date_of_birth, )’: ["bank_ask_dob"],
    ’Report(bank_fraud_report(), main.result)’: ["bank_inform_fraud_report_submitted", "bank_inform_cannot_authenticate"]

Figure 11: Mapping of KITA’s dialog acts

###  C.2 Full Worksheets

Figure 12: Here we define the main worksheet, which has no predicate meaning
it can be updated at any point of time. More over once all the fields are
updated we call the developer defined submit_enrollment_form api. The fields
are courses_to_take, a CoursesToTake worksheet (Figure 11) that details what
courses the student wants to take, student_info_details, a StudentInfo
worksheet (Figure 10), another form which contains information about the
student, and confirm_submission, which allows the agent to explicitly confirm
the value of the field with the user.

Figure 13: Here we define the StudentInfo worksheet, which contains a set of
information about the student. It doesn’t have a predicate, so it can be
updated at any point. It has fields for the student_name, their student_id
number, their student_email_address, and a boolean value for
is_international_student.

Figure 14: Here we define the CoursesToTake worksheet, which details the set
of courses the student wants to take. The form has no predicates so it an be
updated at any time. It has fields for course0_details and course_1_details,
which take the form of Course worksheets (Figure 12) and represent the details
for two classes the student wants to take. In addition, it has a boolean value
more_course_2 for if there is a third course the student wants to take. The
next field, course_2_details, has this boolean as a predicate so it will only
be filled if more_course_2 is true. If it is, then course_2_details will be
filled with the details for the third course. Additionally, in this case there
will be another field, more_courses_3, which will be available to be filled,
and depending on this value course_3_details can be filled with a fourth
Course worksheet. Finally, there is a confirm field to allow the agent to
explicitly confirm the value of the field with the user.

Figure 15: Here, we define the Course worksheet. This contains the details
about a single course. The form doesn’t have any predicates, so it can be
filled at any time. It contains fields for the course_name, the grade_type
which can be either Credit/No Credit or Letter, and course_num_units which
indicates how many units the course is being taken for.

Figure 16: Course database worksheet

Figure 17: Available course offerings database worksheet

Figure 18: Course ratings database worksheet Figure 19: Program requirements
database worksheet

Figure 20: Here we define the main worksheet, which has no predicate meaning
it can be updated at any point of time. More over once all the fields are
updated we call the developer defined submit_api api. The first field is
student_task, which is one of "Troubleshoot Student Enrollment", "Leave of
Absence", or "External Test Credits". After that, there are three different
fields which have predicates that make it so they are only filled
conditionally upon the value for student_task. troubleshoot_student_enrollment
is a TroubleShootStudentEnrollment worksheet (Figure 18) which maintains the
required information for that particular student_task. Similarly, we have
leave_of_absence which is a LeaveOfAbsence worksheet (Figure 21) and
external_test_credits which is an ExternalTestCredits worksheet (Figure 23).
Additionally, after one of these fields has been filled out the predicate for
extra_details is fulfilled and it can be filled with any extra information the
user wants to add. There are also fields for the student’s full_name, a
confirm field for the agent to explicitly confirm the fields of the worksheet,
and an exit field.

Figure 21: Here, we define the TroubleShootStudentEnrollment worksheet. It has
a form predicate of student_task being equal to "Troubleshoot Student
Enrollment" which means that it can only be filled if the student is inquiring
about this task. The worksheet’s first field is the
trouble_shoot_specific_issue, which takes the value "Change Course" or "Join
Waitlist". The next field, change_course_issue, has a predicate of
trouble_shoot_specific_issue being "Change Course", and is of type
ChangeCourse worksheet (Figure 21). Similarly, if the predicate of
trouble_shoot_specific_issue being "Join Waitlist" is met, then join_waitlist
can be filled with a JoinWaitlist worksheet

Figure 22: Here, we define the ChangeCourse form. It has a predicate of
student_task being "Troubleshoot Student Enrollment" and
trouble_shoot_specific_issue being "Change Course", so it will only be filled
if these criteria are met. This worksheet has fields for change_type, which
can be one of "Add a Course", "Drop a Course", "Change Units", or "Other
Enrollment Issues", course_code which indicates the class that has the issue,
and an issue_description.

Figure 23: Here, we define the JoinWaitlist form. It has a predicate of
student_task being "Troubleshoot Student Enrollment" and
trouble_shoot_specific_issue being "Join Waitlist", so it will only be filled
if these criteria are met. This worksheet has fields for the course_name, an
issue_description, a waitlist_confirmation that the waitlist in question
exists, and a field to confirm that there is no schedule_conflict.

Figure 24: Here we define the LeaveOfAbsence form. It has a predicate of
student_task being "Leave of Absence", so it will only be filled if this is
true. It’s first field is the leave_of_absence_specific_issue, which is either
"Status of Leave of Absence form" or "Others". The next field has a predicate
of this leave_of_absence_specific_issue being "Status of Leave of Absence
form", if so form_status is filled out with a FormStatus worksheet (Figure 25)
type. Additionally, there is a final field for a specific issue_description.

Figure 25: Here, we define the FormStatus form. It has a predicate of
student_task being "Leave of Absence" and leave_of_absence_specific_issue
being "Status of Leave of Absence form", so it will only be filled if these
criteria are met. This worksheet has a field for the submission_method, which
can take the value of "Email", "In Person", or "Through Staff". it also has a
field for the submission_date.

Figure 26: Here we define the ExternalTestCredits form. It has a predicate of
student_task being "Leave of Absence", so it will only be filled if this is
true. It’s first field is the leave_of_absence_specific_issue, which is either
"Status of Leave of Absence form" or "Others". The next field has a predicate
of this leave_of_absence_specific_issue being "Status of Leave of Absence
form", if so form_status is filled out with a FormStatus worksheet (Figure 25)
type. Additionally, there is a final field for a specific issue_description.

Figure 27: General ServiceNow information database

Figure 28: Restaurant Reservation: User Info form

Figure 29: Restaurant Reservation: Book Restaurant form

Figure 30: Restaurant Reservation: Restaurant Database form

Figure 31: Restaurant Reservation: General Info Database form

Figure 32: Restaurant Reservation: Restaurant Type form

Figure 33: Bank Fraud: Main form

Figure 34: Bank Fraud: First Authentication form

Figure 35: Bank Fraud: Second Authentication form

##  Appendix D Prompts

We use a common system and user prompt for all the applications in jinja
template. For example applation few shot examples need to be provided. The
system prompt is Figure LABEL: and the user prompt is in Figure LABEL:

System prompt for semantic parsing KITA

    You are a semantic parser. Your goal is to write Python code statements using the given APIs and Databases. Plan your response
    first, then write the code.

    Today’s date is {{ date }} and the day is {{ day }}.

    These are the APIs available to you:
    {{ worksheets }}

    DO NOT ASSUME ANY FIELD. Always use the new information provided by the user.
    DO NOT create lists for creating multiple answer instances. Write them in separate lines.

    Here are some examples:
    {{ few shot examples }}

Figure 36: System prompt for semantic parsing KITA  Prompt for semantic
parsing KITA

    State:
    ‘‘‘
    {{ state }}
    ‘‘‘
    Agent Action:
    ‘‘‘
    {{ agent_actions }}
    ‘‘‘

    Last-turn Conversation:
    Agent: {{ agent_utterance }}
    User: {{ user_utterance }}

    User Target:

Figure 37: Prompt for KITA  Some few-shot examples for Course Enrollment
Assistant’s semantic parsing

    Example: To update any field of the APIs, you can use the following syntax: ‘api_name.field_name = value‘
    State:
    ‘‘‘
    course = Course(course_name = ’CS 231’)
    courses_to_take = CoursesToTake(course_0_details = course)
    main = Main(courses_to_take = courses_to_take)

    ‘‘‘
    Agent Action:
    ‘‘‘
    ["AskField(course, grade_type, The desired grading bases of the student Options are: Credit/No Credit, Letter)"]
    ‘‘‘

    Last-turn Conversation:
    Agent: What is the desired grading basis for CS 231? Your options are: Credit/No Credit, Letter.
    User: I will take the letter grade for 4 units

    User Target:
    ‘‘‘
    course.grade_type = ’Letter’
    course.course_num_units = 4
    ‘‘‘
    --
    Example: To get the answer to a question, you can use the following syntax: ‘answer("Your question here")‘.
    Always do what the user says.
    State:
    ‘‘‘
    main = Main(courses_to_take = courses_to_take)
    answer = answer("What courses are required for logic automata complexity?")
    answer.result = "CS 103"
    courses_to_take = CoursesToTake()
    course = Course()
    ‘‘‘
    Agent Action:
    ‘‘‘
    [Report(answer, answer.result), AskField(course, course_name, "What is the name of the course?")]
    ‘‘‘

    Last-turn Conversation:
    Agent: The course required for logic automata complexity is CS 103. Do you want to enroll in this course?
    User: Are there any prerequisites for this course?
    User Target:
    ‘‘‘
    answer("What are the prerequisites for the course CS 103?")
    ‘‘‘
    --
    Example: Write functional code when you are creating new instances of the APIs. Do not use this to update
    the existing ones in the state.
    State:
    ‘‘‘
    main = Main()
    answer = answer(’What are the ratings for the Artificial Intelligence: Principles and Techniques (CS 221) course?’)
    answer.result = [(’Artificial Intelligence: Principles and Techniques’, Decimal(’4.16’))]
    ‘‘‘
    Agent Action:
    ‘‘‘
    ["Report(answer, answer.result)", "AskField(main, courses_to_take, The courses that the student wants to take)"]
    ‘‘‘

    Last-turn Conversation:
    Agent: The course "Artificial Intelligence: Principles and Techniques" (CS 221) has a rating of 4.16.
    Which courses are you interested in enrolling in?
    User: I will take cs 221
    User Target:
    ‘‘‘
    main.courses_to_take = CoursesToTake(course_0_details=Course(course_name=’CS 221’))
    ‘‘‘
    --
    Example: If the user explicitly refuses to provide any information, you should update the value with "NA".
    State:
    ‘‘‘
    answer = answer(’Is SYMSYS 195N offered in the spring quarter?’)
    answer.result = [{’title’: ’Natural Language Processing with Deep Learning’, ’course_codes’: [’LINGUIST 284’, ’SYMSYS 195N’]}]
    main = Main(courses_to_take = courses_to_take)
    course = Course(course_name = ’SYMSYS 195N’, grade_type = ’Letter’, course_num_units = 3)
    course_1 = Course(course_name = ’CS 347’, grade_type = ’Letter’, course_num_units = 3)
    courses_to_take = CoursesToTake(course_0_details = course, course_1_details = course_1)
    ‘‘‘
    Agent Action:
    ‘‘‘
    ["AskField(course_2_deatils, course_name, None)"]
    ‘‘‘

    Last-turn Conversation:
    Agent: Could you please provide the details for another course you would like to take?
    User: No thats all for now
    User Target:
    ‘‘‘
    course_2_details.course_name = "NA"
    ‘‘‘

Figure 38: Subset of few shot examples given to LLM for semantic parsing KITA
(Course Enrollment Assistant)

##  Appendix E Error Analysis

The majority of agent action errors in the course enrollment task stem from
assistant’s failure to solicit details regarding the second mandatory course.
Instead, the assistant asks for student information, such as their name. This
misdirects the GPT-4 (FC) to prematurely invoke submit_form(), subsequently
encountering an error. However, upon receiving this error, GPT-4 (FC)
demonstrates the capability to recover by requesting additional information
about the second course.

In the ticket submission domain, rather than utilizing the submit_ticket() API
to submit a ticket, the assistant incorrectly states that it has submitted the
ticket. We also observe that once the GPT4 (FC) hallucinates a response, it
enters a spiral of producing increasingly unfounded responses.

Regarding restaurant reservations, the assistant disregards the instructions.
Instead of using the Answer() function to recommend restaurants, it asks for
additional details about the type of restaurant the user prefers.
Additionally, we note that rather than consulting the provided hybrid data
sources, the assistant attempts to answer user queries directly based on the
information retrieved from previous turns. This leads to incomplete and, at
times, fabricated responses.

One common issue across all domains is GPT-4 (FC)’s failure to seek
confirmation before submitting the final API call.

##  Appendix F User Feedback

We solicited feedback from our participants with three free-form questions.
Overall, the feedback was positive: “always factually correct regarding all
the information it provides”, “very natural and smooth, and I could see myself
using for next term.”, “Friendly, made sure I confirmed my selection and
verified that it understood what I was saying. Natural conversational flow.”,
“got quick answers to restaurant info that would have taken longer to figure
out.”

The negative feedback included: “It was a bit slow. I wish it started the
convo, like what would you like to eat?” “I had to restructure my query for it
to give results.” “I disliked the wait time for a response.”, “The
conversation felt a little forced.”