Figure 1.  is a multi-agent conversational system designed to support online
decision-making in unfamiliar domains. In , the user can converse with any
selected set of agents (2) in the conversation space (1) to gather sufficient
information about the domain, and use the criteria and options in the summary
bar (3) to manage the information found and pin important keywords to the
profile card (4) to build preference and make a decision.

Teaser figure that explains the overall idea of . On the right there is a
bubble as a user utterance. Then, there is a conversation space where icons
each with a camera option and speech bubbles are scattered, with additional
user utterances.

#  : Supporting Unfamiliar Online Decision-Making with Multi-Agent
Conversational Interactions

Jeongeon Park<EMAIL_ADDRESS>School of Computing,
KAISTDaejeonRepublic of Korea ,  Bryan Min<EMAIL_ADDRESS>University of
California, San DiegoLa Jolla, CaliforniaUSA ,  Xiaojuan Ma<EMAIL_ADDRESS>Hong Kong University of Science and TechnologyHong KongChina and  Juho Kim
<EMAIL_ADDRESS>School of Computing, KAISTDaejeonRepublic of Korea

(2023)

###### Abstract.

Unfamiliar decisions—decisions where people lack adequate domain knowledge or
expertise—specifically increase the complexity and uncertainty of the process
of searching for, understanding, and making decisions with online information.
Through our formative study (n=14), we observed users’ challenges in accessing
diverse perspectives, identifying relevant information, and deciding the right
moment to make the final decision. We present , a system that enables
conversations with a dynamic set of LLM-powered agents for a holistic domain
understanding and efficient discovery and management of information to make
decisions. Agents, as opinionated personas, flexibly join the conversation,
not only providing responses but also conversing among themselves to elicit
each agent’s preferences. Our between-subjects study (n=36) comparing to
conventional web search and single-agent showed that was more helpful in
discovering, diving deeper, and managing information compared to Web with
higher confidence. We also describe how participants utilized multi-agent
conversations in their decision-making process.

Multi-Agent Interaction, Conversational User Interface, Large Language Models,
Decision-Making Support, Human-AI Interaction

††copyright: rightsretained††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: arXiv; October, 2023; ††booktitle: arXiv October,
2023††price: 15.00††isbn: XXX-X-XXXX-XXXX-X/XX/XX††ccs: Human-centered
computing Interactive systems and tools††ccs: Human-centered computing Natural
language interfaces

##  1\. Introduction

People make online decisions every day, from purchasing a new product,
choosing a trip destination, to taking on a new hobby. In making a decision,
people inherently go through a set of steps including recognizing the need,
collecting information, identifying and ranking criteria, identifying and
evaluating alternatives, and deciding an alternative (Lunenburg, 2010).
However, making an _unfamiliar_ decision—a decision-making situation where an
individual lacks knowledge and preference in the domain—requires individuals
to first grasp the fundamental knowledge of the domain before delving into the
decision-making process (Karimi et al., 2015; Bessette et al., 2021). Such
lack of familiarity makes decision-making more difficult as it requires the
user to continually switch between learning about the domain and evaluating
options (Bettman and Park, 1980; Moore and Lehmann, 1980; Malhotra, 1983).

While prior work has demonstrated effective support for online decision-
making, these works have focused on providing support for managing and
organizing important criteria and alternatives in the domain rather than
uncovering and understanding them (Chang et al., 2020; Liu et al., 2019,
2022). This lack of support can increase information overload for unfamiliar
decision-making situations (Iyengar and Lepper, 2000; Scheibehenne et al.,
2010). As a result, this can lead to a significant investment of time and
cognitive resources in grasping domain details and eliciting preference,
potentially causing individuals to make uninformed decisions or even give up
on making the decision entirely (Jameson et al., 2015; Thai and Yuksel, 2017).

Recommender systems aim to simplify decision-making by suggesting personalized
options (Chen et al., 2013; Häubl and Trifts, 2000). Yet, for novices in a
domain, these preferences might not be well-elicited (Lops et al., 2011;
Schafer et al., 2007). This can cause an over-reliance on the system,
especially if the novices cannot detect errors due to their limited knowledge
(Nourani et al., 2020). Ultimately, it almost appears that to make well-
informed online decisions, novices _must_ conduct thorough research of the
unfamiliar domain individually.

We see an opportunity in large language models (LLMs) to better assist the
workflow of unfamiliar decision-making. Their demonstrated effectiveness as
informative conversational systems (Leiter et al., 2023) shows promise for
supporting domain novices in understanding an unfamiliar domain landscape
while attaining recommendations with rich context. In this work, we focus on
the unique challenges and needs of _unfamiliar_ decision-making and aim to
equip users with an LLM-powered interface to navigate the decision-making
process in an unfamiliar domain.

To understand how people make unfamiliar decisions with online information, we
conducted a formative study with 14 participants who had experience making
decisions online. From observing and interviewing about their decision-making
process in unfamiliar domains, we discovered that participants struggled to
(1) find diverse perspectives in the domain, (2) identify relevant information
among an overwhelming amount, and (3) indicate a confident end to the process.

Our formative study revealed that existing methods struggled to offer diverse
and relevant viewpoints, which inspired us to bring in multiple agents powered
by LLM to assist the decision-making process. We developed , a multi-agent
conversational system that supports unfamiliar decision-making with a group of
LLM agents. In , the user can converse with any individual or group of
_agents_ (Fig.1-2) characterized by unique personas in the domain through the
_conversation space_ (Fig.1-1). Auto-detected criteria and options from the
conversation appear in the _summary bar_ (Fig.1-3) to help the user uncover
and keep track of the key information found, and the user can save criteria
and options on the _profile card_ (Fig.1-4) to shape their preferences and
make a final decision.

We conducted a between-subjects user study to evaluate whether could
effectively support unfamiliar decision-making. Comparing with a web search
condition and a prompt-tuned ChatGPT condition, the study revealed that
successfully supported users in exploring, diving deeper, and flexibly
managing diverse information in the domain in comparison to Web with higher
confidence. performed similarly to SingleAgent, but there were benefits of to
SingleAgent in terms of eliciting multiple perspectives and gaining more
control over the process which we discuss in the discussion section. In
addition, we report on representative patterns of how users utilized for
decision-making.

This research makes the following contributions:

  * •

A formative study uncovering the challenges and needs of unfamiliar decision-
making.

  * •

, a multi-agent conversational system designed to support unfamiliar decision-
making through conversation modes and a summary bar.

  * •

A user evaluation of demonstrating that users were able to go through more
diverse and comprehensive information and flexibly manage the information
found in the decision-making process compared to Web with more confidence.

##  2\. Related Work

Our work addresses the challenges in online unfamiliar decision-making by
leveraging the benefits of multi-agent conversations powered by LLMs. We
review prior work on online decision-making support, conversational interfaces
for online decision-making, and designs of multi-agent systems.

###  2.1. Online Decision-Making Support

Due to the enormous amount of information online, online decision-makers can
face information overload (Wan et al., 2003), which negatively influences the
resulting decision quality (Peng et al., 2021; Lee and Lee, 2004). For
unfamiliar decisions where users do not have sufficient knowledge or expertise
in the domain, searching for information becomes particularly challenging
(Karimi et al., 2015; Bessette et al., 2021).

One thread of research focused on identifying selection criteria among a wide
range of options to choose from. Mesh proposed consumer product decision-
making through comparison tables and customizable preference settings (Chang
et al., 2020), and Unakite and Crystalline supported software developers’
decision-making with web content snippets and comparison tables (Liu et al.,
2019, 2022). While these systems provide strong support for utilizing
necessary information to make decisions, they do not sufficiently support
information searching and understanding for users without knowledge in the
domain (Iyengar and Lepper, 2000; Scheibehenne et al., 2010).

Meanwhile, recommender systems can guide users unfamiliar with the domain by
providing personalized options for users to choose from (Chen et al., 2013;
Häubl and Trifts, 2000; Shapira, 2010). However, one-shot recommender systems
are insufficient to support users in learning about the domain for unfamiliar
decisions as the interaction is uni-directional (Jannach et al., 2021). This
can decelerate users from forming their preferences during the decision-making
process (Lops et al., 2011; Schafer et al., 2007; Wang and Benbasat, 2013b).
Conversational recommender systems (CRS) build on top of existing recommender
systems by offering the ability to make back-and-forth exchanges of messages
to continually provide contextualized and relevant recommendations throughout
the user’s learning and decision-making process (Jannach et al., 2021; Wang
and Benbasat, 2013a; Wärnestål, 2005). In CRS, the user can communicate
through natural language interactions to solicit exact user preferences (Gao
et al., 2021).

is primarily inspired by the benefits of CRS in providing personalized
recommendations and guidance for preference elicitation and aims to expand
that to unfamiliar decision-making situations where a holistic understanding
of a domain is crucial for making a confident decision. We additionally
incorporate the benefits of web-based decision-making support systems to
identify and manage key criteria and options in the domain better.

###  2.2. Text-Based Conversational Interfaces for Information Seeking

Text-based conversational user interfaces (CUIs) have been utilized for many
tasks due to their ability to provide information in a digestible, human-like
manner and facilitate adaptive exploration of information (Liao et al., 2020;
Xiao et al., 2023, 2020; Foo et al., 2022).

In utilizing its benefit, Searchbuddies explored embedded search engine agents
in social media message threads to provide easy access to relevant information
online (Hecht et al., 2012). Gupta et al. showed that providing online housing
recommendations through conversational interfaces built user trust and
satisfaction more effectively than non-conversational interfaces (Gupta et
al., 2022). Prior work has also proposed a viable framework defining that
conversational interfaces can be integrated with information search to provide
a more effective process for personalized recommendations and multi-turn
interactions across a set of search items (Radlinski and Craswell, 2017).

The recent surge of large language models (LLMs) has revealed the potential
for more capable CUIs across diverse domains ranging from programming to
searching for UI (Wang et al., 2023; Liffiton et al., 2023; Ma et al., 2023).
Their ability to synthesize and generate text makes them resourceful in many
domains available to the public, and their ability to dynamically provide
responses to user prompts makes for more adaptive and engaging conversational
agents with less engineering effort. Thus, existing work and products have
utilized the potential of LLMs in supporting information exploration for
learning and decision-making (Friedman et al., 2023; Kha, 2023; Cha, 2023).
For example, ChatGPT Plugins are designed to support real-world tasks such as
trip planning, giving the users access to more personal and specific
information (Cha, 2023).

While conversational interfaces adaptively provide information through a
multi-turn process, conversing requires the user to articulate their input
clearly which, if not done, could result in misinterpreted user intent and
irrelevant responses (Zue and Glass, 2000; Candello and Pinhanez, 2016; Flohr
et al., 2021; Jain et al., 2018; Luger and Sellen, 2016). To complement that,
existing work has demonstrated techniques to bring multi-modal inputs such as
a graphical user interface (GUI) to repair conversational breakdowns (Li et
al., 2020, 2021). Moreover, Sensecape demonstrated support for nonlinear,
multi-level exploration and sensemaking of LLM-generated text to empower users
in the sensemaking process (Suh et al., 2023).

builds on these LLM-powered conversational interfaces by providing interactive
and digestible conversations to support online decision-making. In addition,
adds on direct manipulation of objects as an interaction modality on top of
natural language, where actions such as pinning valuable criteria or options
are fed into the conversation as a context.

###  2.3. Multi-Agent Interfaces

Previous works presented systems that introduced the interaction between
multiple agents that simulate human conversation, cooperate to solve tasks,
and process user inputs.

A number of existing work has demonstrated the simulation of multi-person
communication with LLM-powered agents, showing these personas closely resemble
communications between real users by displaying different personas (Cox et
al., 2023; Chen et al., 2023).

Multi-agent systems have also been used to solve complex tasks, by utilizing
agents for microtasks or having them collaborate or debate to elicit a wide
range of information and opinions. One popular approach is self-prompting, an
architecture for feeding LLM-generated responses into other LLMs to complete
information tasks autonomously (et al., 2023; Hong et al., 2023; bab, 2023; Li
et al., 2023a). Other frameworks have also been presented for building multi-
agent autonomous LLM architectures (Talebirad and Nadiri, 2023). In addition,
AutoGen provides a framework for supporting flexible multi-agent conversation
patterns, which can be employed across various applications, including online
decision-making (Wu et al., [n. d.]).

Another line of work investigates how the user manages and converses with
multiple agents to get support on complex tasks. Many human tasks benefit from
collaboration and information integration from various sources (Curşeu et al.,
2015; Goertzel, 2017) compared to the individual cognitive processes in
isolation. These systems capitalize on their abilities to delegate tasks to
multiple agents, communicate and collaborate among the agents, and provide
unique identities (Kitamura et al., 2002; Kitamura, 2004; Li et al., 2023b).
Zolitschka et al. proposes a unique approach to automatically orchestrate and
coordinate multiple agents for a given user input message (Zolitschka, 2020),
and ChatEval utilizes a multi-agent LLM system for evaluating text using
responses from multi-agent debates (Chan et al., 2023). CommunityBots utilizes
a set of agents handling a unique domain each, supporting topic- and chatbot-
switching based on user interactions (Jiang et al., 2023a).

In , we leverage the advantage of having multiple agents to support the
decision-making process, where the agents reveal similar and opposing
viewpoints toward the user’s utterance or inquiry. In addition, instead of
automatic orchestration among agents to provide agent response, we provide
authority to the user, so that the user can choose a set of agents they relate
the most to converse with for a better understanding of the available
perspectives to make a more informed decision.

##  3\. Formative Study

To design an approach to support unfamiliar decision-making, we first aimed to
understand the common practices in unfamiliar decision-making and uncover the
challenges in the process. For this purpose, we conducted semi-structured
interviews with 16 participants with prior experience in unfamiliar decision-
making.

###  3.1. Participants and Study Procedure

We recruited 14 participants (Age=18-55, M=28.2, Std=7.8; 8 males and 6
females) who had frequent experience making decisions online (details in Table
1). The recruitment took place through our university’s community channels and
snowball sampling to recruit participants in a wide age group. To ensure their
experience in making decisions in unfamiliar scenarios, we asked the
participants to list out previous experience in both familiar and unfamiliar
decisions in the recruitment form.

The interview lasted for 75 minutes for each participant. First, a 20-minute
semi-structured interview was conducted to understand participants’ previous
experiences with unfamiliar decision-making. We asked questions about the
overall process, which information they utilized, and the challenges they
faced in the process. Then, a 30-minute think-aloud study was conducted where
participants chose an unfamiliar scenario among the given set (see Appendix A)
and browsed through the internet until they identified several possible
options. Lastly, participants were interviewed for 15 minutes on their
experience with their process and the challenges they faced in this scenario,
then on the general challenges and desired support of unfamiliar decision-
making.

ID | Age | Gender | Previous Unfamiliar Scenario |  Selected Scenario
---|---|---|---|---
P1 | 26-35 | F |  Choosing a course to take for a summer school, Buying an oven for a baking hobby |  Choosing a cafe to cater some snack food on an end-of-semester event
P2 | 36-45 | M |  Choosing a trip destination, Choosing a t-shirt for summer |  Planning a solo trip destination for three days
P3 | 26-35 | M |  Buying a tennis racket, Choosing a U.S. stock option to buy |  Buying a car seat for a friend’s newborn
P4 | 26-35 | F |  Choosing a working holiday location, Buying a laser hair removal |  Buying a car seat for a cousin’s newborn
P5 | 18-25 | M | Buying a lunch box |  Buying a robot vacuum cleaner to replace the normal vacuum cleaner
P6 | 26-35 | F |  Buying a new tennis racket, Choosing meal to cook |  Renting a house short-term for an internship
P7 | 18-25 | M |  Choosing a transportation card plan in a new country |  Choosing a new hobby to do in free-time
P8 | 18-25 | F |  Planning a trip to Osaka, Choosing a wrist/ankle brace+lumbar support |  Buying a robot vacuum cleaner to replace the normal vacuum cleaner
P9 | 18-25 | M |  Choosing a hair salon, Buying a gift for a friend |  Buying a car seat for a friend’s newborn
P10 | 18-25 | M |  Whether to get a CT+MRI during ER visit, Buying clothes at a flee market |  Buying a skateboard for transportation purposes instead of walking to school
P11 | 26-35 | F |  Buying a coffee pod machine, Searching for a job |  Buying a skateboard for transportation purposes instead of walking to school
P12 | 26-35 | M |  Buying a used phone, Planning a trip to Jeju |  Buying an interior light at home
P13 | 18-25 | M |  Choosing an affordable phone plan, Choosing field of work |  Buying a skateboard for transportation purposes instead of walking to school
P14 | 46-55 | F |  Choosing exercise, Buying a blue light blocking film |  Buying an interior light at home

Table 1. Participant demographic for the formative study.

A table showing participant demographic for the formative study, with the
following information in each of the columns: ID, age, gender, previous
unfamiliar scenario, and the selected scenario. There is a total of 14
participants.

###  3.2. Findings

Here, we describe the different approaches people follow for unfamiliar
decision-making and describe three main challenges we identified from the
process.

####  3.2.1. Practice

The participants used several sources in various orders and frequencies in
collecting information and making unfamiliar decisions. Each participant had a
different strategy they took for the process, which involved various sources
including summary posts of the domain (n=8), posts with individual opinions
and experience (n=10), and websites with a list of options in the domain
(n=11). During the interview, many (n=12) brought up asking an expert as their
previous experience making unfamiliar decisions.

With _summarized post_ such as YouTube recommendation videos or rating sites
(e.g., https://www.rtings.com/), participants were able to objectify multiple
criteria and understand the overview of the domain, but could not receive
contextualized information. Participants found _posts with individual
opinions_ —online communities, product reviews, or blog review posts—helpful
for gathering diverse opinions on the options but also mentioned that it is
time-consuming to find credible sources and hinders them from utilizing
individual opinions. _Browsing through a list of options_ from shopping sites
was another approach participants took, but they were unable to understand and
compare different options without knowledge of the domain, thus switching to
the other two sources of information. During the interview, 12 out of 14
participants mentioned _asking an expert_ —including store managers, friends,
and family who have expertise in the decision domain—an approach they took for
unfamiliar decisions as a way to receive information without being
overwhelmed. They mentioned that experts’ ability to (1) explain ground-based
information about the domain and provide information that’s difficult to find,
and (2) ask clarification questions to make better suggestions in one’s
specific context makes them reach out to experts for unfamiliar decisions.

####  3.2.2. Challenges

We identified three main challenges participants faced during the unfamiliar
decision-making process.

##### C1: Difficult to find diverse perspectives

The participants found it challenging to find information that contained
diverse viewpoints even through multiple sources. Some struggled to find
diverse opinions about the domain: P14 (interior light) discovered new
criteria as they looked at the reviews of each option, but still struggled
from a lack of sufficient information. Others wanted to see information from a
fresh viewpoint, to understand the information landscape of the domain. They
mentioned that they wanted to see if the different perspectives they found in
sum represented comprehensive information of the domain to make a more
confident decision.

##### C2: Difficult to identify relevant information among an overwhelming
amount

After they were able to find information about the domain through multiple
sources, the participants faced difficulty in understanding and connecting the
information to their unique situation. P4 (car seat) discovered two types of
car seats and the characteristics of each but did not understand what would be
better for a newborn, commented later that “I end up spending a lot of time
and effort finding a good resource [that fits my context].” Most participants
were successful in getting to different websites, but the ones they utilized
were of a small portion in comparison to the information they accessed as they
did not see others as relevant. For example, P5 (robot vacuum cleaner) found
lots of recommendations online, but failed to utilize them as they were “all
personal and subjective experiences, different from my situation.”

##### C3: Don’t know when and where to end

Since participants did not have intuitions on the domain due to their lack of
knowledge and experience, they found it difficult to identify whether the
information they had accurately described the general information of the
domain. The participants constantly questioned themselves on whether there
were any criteria they might have missed, which could influence the experience
after the decision. Even after they identified a strong candidate, they lacked
confidence and double-checked through multiple other sources whether the
candidate was a suitable choice. During cross-checking, there have been cases
where the participants had to reset their decisions and go for a new option.
P1 (cafe catering) had to go back to the initial stage after seeing all the
cafes she found lacked catering experiences. In addition, P8 (robot vacuum
cleaner) described the decision scenario as a “seemingly endless amount of
information” which led to additional difficulty in making a decision.

##  4\. Design Goals

With the identified challenges in the unfamiliar decision-making process, we
propose the following design goals for a system that can support unfamiliar
decision-making.

  * DG1.

Support discovering of diverse perspectives in the domain. To support users in
finding and understanding diverse perspectives in the domain, a wide range of
values that sufficiently cover existing values in the decision domain should
be provided. Furthermore, the system should support users to easily browse
those values and compare them to aid an understanding of how they differ.

  * DG2.

Provide information adaptive to user’s preferences. Among the provided
information in the domain, users should be able to find the ones that are the
most relevant to their context. Previous research has shown that choice-based
preference elicitation is more effective for novices (Loepp et al., 2014). To
support unfamiliar decisions, the system should provide users with the
possible choices in the domain along with additional information to support
the induction of preferences that adhere to personal context.

  * DG3.

Enable flexible management of discovered information. The participants
described the information as ‘overwhelming’ in indicating the most relevant
information, comparing different opinions, and understanding the information.
Since unfamiliar decision involves a wide range of information, the system
should support users with user-guided filtering, clustering, and grouping at
various level (e.g., criteria, options) for flexible management of the
information found throughout the process.

  * DG4.

Support reduction of discovered criteria, preferences, and options. After
finding and identifying relevant information in the domain, users should be
able to converge into a set of criteria and an option to come to a conclusion.
The system should help users record emerging preferences—criteria and
options—throughout the process and utilize the recorded preferences in
indicating a confident end to the decision.

##  5\. : A Multi-Agent Conversational System

From our findings and design goals, we propose a system that supports
unfamiliar decision-making with multi-agent, conversational interactions
powered by LLMs. We decided to use conversational interaction for its ability
to provide information in a digestible manner and facilitate interactive
exploration of information (Liao et al., 2020), and multiple agents to support
the exploration of diverse viewpoints to discover personalized information
(Samir and Habiba, 2009). We use LLMs to generate the agents for their ability
to provide resourceful information in the domain (Wang et al., 2023).

We designed and developed , a multi-agent conversational system that assists
users in gaining a broad understanding of an unfamiliar domain by (1)
conversing with a dynamic set of agents and (2) constructing preferences
toward a decision by managing the conversation through agents. Throughout the
section, we describe an envisioned user scenario, explain different components
of the system, and describe the system implementation and prompt engineering.

###  5.1. Envisioned Scenario

Sally has recently found an interest in photography and wants to find a camera
right for her. However, she is unfamiliar with the camera domain and does not
know which camera she would like most. As she has no experts to consult and
gets overwhelmed by the information online, Sally enters and types into the
message input box, “I’m new to photography, and I want a camera. Not sure
which one would be best for me.” She then encounters four agents (Alex, Jamie,
Taylor) appearing on , each with a unique persona containing an option (i.e.,
a camera model) they had chosen and the criteria (e.g., portability, brand)
that led to their choice. In each agent’s response, she sees the auto-detected
and highlighted criteria and options.

  * •

Alex: “As a professional photographer, I value image quality and durability in
a camera. That’s why I chose the Canon EOS 5D Mark IV. It’s a full-frame DSLR
that delivers excellent image quality and is built to last.”

  * •

Jamie: “I’m a hobbyist photographer and I prefer a camera that’s lightweight
and easy to use. I’ve been using the Sony Alpha a6000 and it’s been great for
me. It’s compact, takes great photos, and is very user-friendly.”

  * •

Taylor: “I’m a travel blogger and for me, portability and battery life are
key. I use the Fujifilm X-T3. It’s compact, has a long battery life, and takes
amazing photos. What are your needs and preferences when it comes to
photography?”

Sally learns about the wide range of values and criteria different professions
and lifestyles have for cameras through the agents and their profiles (DG1).
Sally considers Jamie to be the most relatable as she is getting a camera for
a hobby. Sally tags Jamie and asks why they value lightweight and easy-to-use
cameras. Jamie responds that they “don’t need all the bells and whistles of a
professional camera” and they “want to focus on capturing the moment”. Sally
learns that for her personal use, she may want to follow Jamie’s reasoning as
Sally is also looking for a simple camera. She then asks follow-up questions
to Jamie on “What is considered a lightweight camera?” and “What is the main
difference between a professional camera and an easy-to-use camera?” to better
understand what each criterion means.

After gaining some information, she decides to get an easy-to-use camera and
saves the criterion to her profile. To explore more candidates, she asks a
question that reflects her preference: “Are there other easy-to-use cameras?”
Three more agents with the criterion easy to use included in their valued
criteria appear: Riley, Morgan, and Casey (DG2). Continuing the conversations
with different agents, Sally realizes that the conversation space is becoming
cluttered with many agents, and decides to filter out irrelevant agents by
selecting the criterion easy to use from the Summary Bar (DG3).

As Sally continues to converse and learn more, she becomes more and more
confident about the set of criteria she personally values for her activities
and saves those to her profile. Now that she has a clearer picture of the
domain, she also wants to compare the three cameras she is currently
considering in depth. She selects agents advocating for the three cameras and
opens the focus view to view their detailed reasoning for their choices and
how it is related to Sally’s valued criteria. By asking the agents to _debate
each other_ , Sally finds the Nikon Coolpix B500 best fits her preferences and
pins it to her profile. She is content that she could identify the important
criteria and options by constructing her profile (DG4). When she clicks
“Done”, she reviews her criteria and what she had valued throughout her
decision-making process.

###  5.2. System Components

(Fig. 2) is an interface that allows the user to interact with a set of agents
to make an unfamiliar decision. Agents are the main unit of conversation in
and are the basis of gaining information about the domain. This allows users
to freely explore domain information while flexibly building and managing
their knowledge structure.

consists of a _message input box_ , a _conversation space of organizable
agents_ , a _summary bar_ consisting of criteria and options found through the
conversation, and a _profile card_ for saving important criteria and options
that user found throughout the session. The user can 1) explore sufficient and
relevant information in the domain by conversing and interacting with the
agents and 2) gather relevant criteria and options and compare them to get to
a decision. In this section, we first describe agents, the conversation space
with the agents as well as additional support in the conversation space to
manage the information, and then the summary bar and the profile card.

Figure 2. The interface: Agents populate the conversation space to converse
with the user, identifying key criteria and options in their utterances (1).
The _Summary Bar_ lists those criteria and options (2), additionally showing
the user profile. _Thought Bubbles_ recommend the user questions to ask (3),
and the interface provides the user with a message input box to converse with
all agents (4).

A screenshot of the overall system interface of ChoiceMates. On the left is
the Summary Bar labeled as two, which shows the Scenario, Profile, Criteria,
and Options. On the right, taking up the majority of the screen, is the
Conversation Space labeled as one, which shows four agents Alex, Jamie,
Taylor, and Jordan with their respective utterances. On the bottom side, there
are two Thought Bubbles, labeled as three. At the bottom below the Thought
Bubbles is the message input box, labeled as four.

####  5.2.1. Agents

Each agent is characterized by its persona (i.e., a single-line description of
the agent), a set of criteria (i.e., factors to consider in the domain) they
value, and a single option (i.e., an available choice in the domain) they
chose with the criteria. They are designed to reflect an individual’s choice
and their underlying values and experience in real life and as a support to
gaining an easy understanding of the domain and elicitation of user’s
preferences by externalizing the link between an option and criteria. The
constraint of one agent favoring one option was inspired by previous work
showing that novices preferred case-based preference elicitation (Knijnenburg
and Willemsen, 2009). To provide a better salience of criteria and options in
the domain, all criteria mentioned by agents are highlighted in blue and all
options are highlighted in purple.

When the user asks an initial question in the domain, prompts the LLM to
generate diverse agents. Newly spawned agents always begin with an
introduction of themselves, sharing their valued criteria and a chosen option.
To ensure that the agents communicate with the correct information, we scrape
information through Google search for facts about the agent’s option. We run
the web scraping function right after the agent is generated to let the LLM
produce diverse agents without being restricted by the web-searched
information. The scraped information is then provided to the agents and stays
in the conversation stream with a prompt asking agents to utilize the
information. This content is meant to stay in the conversation as context.
Agents are by default represented by their unique icons, names, and options
(Fig. 3).

Figure 3. (1) An agent utterance is represented as a chat bubble above the
agent icon. (2) Hovering over the icon reveals the agent’s profile containing
their valued criteria and their chosen option.

The figure shows an example of an agent in the system. On the left is Alex,
who is represented by an icon, name, option, and utterance with the criteria
highlighted in blue and its option highlighted in purple. The left side shows
a cursor hovering over Alex’s icon, revealing its profile containing its
valued criteria and option.

####  5.2.2. Conversation Space

To support the exploration of an unfamiliar domain, provides a conversational
space where all agents reside, allowing users to converse with and freely
organize them in order to understand the overall landscape of options from
agents.

Figure 4. ’ multi-agent conversation is designed to provide continuous, yet
controllable streams of agent conversations, inviting the user to share their
personal preferences. When Alex mentions the importance of durability (1),
Jamie follows up to share their experience with their chosen camera (2). Jamie
brings the user into the conversation. The user then shares a different
background from those of Alex and Jamie (3). This invites Taylor, an agent
with a similar background to the user, to share a more relatable experience
when choosing a camera (4).

The figure shows the conversation flow in the conversation space of the
system. On the left, Alex, labeled one, is talking about his durable camera
choice. Then, a right arrow points to Jamie in the middle, labeled two, who
responds to Jamie with his experience with portability. Then, an arrow points
downwards towards the message input box, labeled three, signifying that the
user responds to Jaime. The final arrow points to the upper right to Taylor,
labeled four, an agent who joins the conversation to talk about affordability
and versatility to relate to the user.

##### Agent Conversation (Fig. 4, DG1)

The formative study revealed that searching for many directly relevant
perspectives is difficult in an online web search environment. Users have to
sift through many sites to find such relevant experiences, and this slows down
their decision-making process. On , agents are designed to chat back and forth
with the user (DG2). This way, users can ask follow-up questions to agents
they relate to and gain a deeper understanding of domain options.

The user initiates the conversation by sharing their decision-making scenario,
and three to six agents with varying persona appear on the screen. Agents then
share their experiences and ask questions about the user’s preferences. The
user can then respond to agents through their message input box to either
reply to a question or ask one themselves.

provides both user-guided and system-guided selection of agents to respond.
Upon receiving the user’s message, detects the user’s intent and makes
existing agents, new agents, or a combination of both to respond, depending on
the agents’ relevancy to the user’s message and context (More described in
Fig. 9) If the user wants to manually direct their response to particular
agents, they can also select agents on the space (Fig. 6). Additionally,
agents can converse with one another by responding to other agents in the
conversation. They can agree, disagree, or ask further questions to other
agents. This can reveal comparisons between options and new criteria that the
user may not have thought of when they are the only ones conversing with them.
Just as the user can manually select agents to respond, the user can also
manually trigger inter-agent conversations by selecting agents and asking them
to “debate each other”.

##### Agent Management (Fig. 6, DG3)

After several turns of conversation between the user and agents, numerous
agents can be introduced, potentially cluttering the space. The user can
select and drag agents around the space to better organize their space to
manage the clutter of agents and rearrange them.

##### Conversation Guidance and Management (Fig. 2.3, DG2)

To support users not knowing what to search for due to their lack of knowledge
in the domain, provides Thought Bubbles to provide suggestions on what to ask
agents to continue their search (Fig. 2.3), inspired by the effectiveness of
such guidance in prior work (Suh et al., 2023). Thought Bubbles appear
throughout the conversation and suggest utterances that guide the user to
explore other criteria and options that might be relevant to them or important
in the domain. The user can click on the Thought Bubbles to send the message
as the user and is updated with the conversation history after each turn.

To reduce clutter from more agents and messages, we chose to only display the
latest messages sent from the previous turn. Instead, users could click the
button left of the message input box in Fig. 4.2 to view the conversation
history.

Figure 5. Users can select either a single agent by clicking on their profile
or select multiple by selecting an area (1). The names of the selected agents
appear on the message input box, indicating which agents will respond to the
user’s message (2).

The figure shows on the top a cursor dragging the area across two agents Jaime
and Taylor to select both of them, labeled one. On the bottom shows the names
of the selected agents appearing in the message input box to show that they
will respond to the user’s message, labeled two.

Figure 6. The _Summary Bar_ provides two lists of keywords from the
conversation: (1) criteria and (2) options. Users can select keywords to lift
agents on the space who value the selected keywords. Other agents who do not
value any of the keywords selected sink to the bottom instead.

The figure shows how the Summary Bar functions. On the left is the Summary
Bar, which shows the list of criteria highlighted in blue and options
highlighted in purple derived from the conversation. A cursor is selecting a
criteria, where selected criteria and options are marked with a yellow
bookmark icon next to it. The right of the figure shows three agents, Taylor,
Alex, and Jaime. Alex is shown to be lifting above other greyed-out agents
when the criteria valued by Alex are selected by the cursor.

Figure 7. When agents are selected in the conversation space, a _Focus agents_
button appears. Clicking this button fully expands agent profiles to fill the
screen. In the focus view, the user is shown each agent’s profile and latest
message. In addition, each agent shares their opinion on each user’s selected
criteria (Fig. 6) in a short sentence.

The figure shows how the Focus View of the system works. On the left is the
conversation space showing three agents, Alex, Jamie, and Jordan, who have
been selected by the user. Next to them is a Focus agents button with a cursor
clicking it, labeled one. On the right is the Focus View created by clicking
the Focus agents button, labeled two. It shows the fully expanded agent
profiles filling the screen, which is a table with each column consisting of
an agent’s latest utterance, labeled a, then agent icon, name, and option,
labeled b, and the agent’s opinions on the user’s selected criteria in short
sentences, labeled c, from top to bottom.

Figure 8. When the user finds criteria and options they find valuable, they
can pin criteria and options on the Summary Bar (1). The pinned criteria and
options populate in the user’s profile section of the Summary Bar (2), and
submitting their profile card reveals a modal, showing agents who had similar
preferences throughout the conversation (3).

The figure shows how the user creates their profile card. On the left is the
list of criteria from the conversation, labeled one. A cursor is shown
selecting a criteria to pin it, which is then marked with a yellow bookmark.
In the middle is the user’s profile section in the Summary Bar, labeled two,
which shows a list of criteria and options the user pinned with the criteria
highlighted in blue at the top and options highlighted in purple at the
bottom. At the very bottom of the list is Recommended Options, which shows a
recommended option with a plus symbol on the side to create additional
recommendations, and a Submit Profile button at the bottom being clicked by a
cursor. On the right is the profile card created after clicking the button,
labeled three, which shows a modal of users’ pinned criteria and options as
well as agents who had similar preferences throughout the conversation.

####  5.2.3. Preference Construction with Summary Bar

provides three representations to facilitate this stage: (1) Summary Bar, (2)
Focus View, and (3) Profile Card.

##### Summary Bar (Fig. 6, DG4)

To help users make sense of the information found with multiple turns, we
provide Summary Bar that distills information into key aspects for the user to
gain a broad understanding of the conversation and the domain. Criteria and
options are detected and highlighted as agent messages are generated and are
listed in the Summary Bar. The user can hover over criteria and options to
highlight agents who value them (Fig. 6). Clicking on the criteria and options
moves the highlighted agents up and sinks all other agents down. This not only
helps users identify the connections between criteria and options through the
agents’ profiles, but it also helps them manage the clutter of the space.

##### Focus View (Fig. 7, DG3, DG4)

Previous work has demonstrated the effectiveness of the table view for
decision-making by providing a clearer understanding of key criteria (Chang et
al., 2020). Embracing this concept, we offer Focus View, a table-like
representation that enables users to explore selected criteria and options in
detail, thereby improving the comparison between different choices.

Upon selecting agents on the conversation space and selecting criteria from
the Summary Bar, the user can click on the ‘focus agents’ button to enter the
Focus View (Fig. 7). In the focus view, agents are displayed in a column with
user-selected criteria in rows (Fig. 7.2) and each provides brief explanations
of what they think of each selected criterion by relating to their option.
This allows the user to compare between different options by choosing which
agent may be a better match for their selected criteria.

##### Profile Card (Fig. 8, DG3, DG4)

The Profile Card is a dedicated space for the user to save their valued
criteria and options. This aims to mimic the profiles of agents and further
aims to nudge the user towards a final option. The user can pin criteria and
options from the Summary Bar to the Profile Card. This allows to recommend
options given matching criteria between the user and agents. When the user
thinks their profile is complete (i.e., they have made their decisions), the
user can click “Done” to see the completed profile card and a summary of their
decision-making process (Fig. 8).

Figure 9. Overview of the technical pipeline. When the user sends their
message to , the conversation context is sent to the LLM (1) and is instructed
to infer the user’s intent (2) and respond through the most relevant agents
(3).

The figure illustrates the technical pipeline of . (1) the conversation
context is sent to the LLM with the user utterance as the prompt. The LLM is
instructed to identify the type of user utterance: (1) direct mention of
agents/options, (2) asking about criteria, (3) vague preference elicitation,
and (4) exploring new options. Depending on the user intent, the LLM
determines which agents to respond closest to the criteria and options
mentioned, or invites new agents to speak up. The agent response is added to
the conversation history, but the pre-prompt is not.

###  5.3. Implementation and Prompt Engineering

was built as a web application using the React framework and utilized OpenAI’s
GPT-4-0613 API. We describe the technical pipeline (Fig. 9) of conversation
management of multiple agents and discuss prompting techniques implemented in
to facilitate effective multi-agent conversations.

####  5.3.1. Prompting for Single-Stream Multi-Agent Conversation

instructs the LLM to embody multiple personas and manage them simultaneously.
The LLM is prompted to embody an identity that communicates through multiple
different personas where they are allowed to respond to the user’s utterance
or any agent’s utterance. This allows the LLM to virtually provide multiple,
contextually aligned responses in a single turn of prompting.

####  5.3.2. Context Retention

While a linear conversation model makes for a simple data structure to manage,
the trade-off of increased reliance on the LLM’s ability to retrieve
information can cause a loss of context from excess information (Liu et al.,
2023). To address this potential limitation, we prepend a _pre-prompt_ —a
preliminary, ephemeral prompt message to every prompt of the user’s utterance
(Fig. 9.1). This technique allows to keep a summary of the conversation state
while the conversation history only contains agent messages. While state-of-
the-art NLP summary features such as LangChain’s Contextual Compression
(LangChain, 2023) compresses large bodies of text into summaries for improved
information retrieval, our method utilizes GPT’s trait of retrieving
information at the beginning and end of the conversation better than in the
middle (Liu et al., 2023).

The pre-prompt contains the lists of detected criteria, options, and generated
agents (i.e. personas), and the lists of user-focused criteria, options, and
generated agents. This provides salience to the relevant keywords in the
user’s decision-making domain. Any action that updates any of the lists also
updates the pre-prompt, and once the user sends a message the latest pre-
prompt state is pre-pended.

####  5.3.3. Representing Multi-Agent Responses

implements a constrained prompting technique, where the LLM is prompted in
such a way that generated text contains interleaved characters to denote tags
to guide structure in the textual representation. Inspired by Graphologue’s
(Jiang et al., 2023b) technique, annotates agent names, criteria, and options
in the generated text and is parsed by the system interface.

##  6\. Technical Evaluation

We conducted a technical evaluation to measure the performance of ’
capabilities of multi-agent conversation under a single conversation stream
and the coverage of the domain ’ agents could represent. This was to measure
how generalizable our pipeline could be to other domains, in contrast to the
user study where we perceived how users use the multi-agent system.

###  6.1. Setup

The goal of this evaluation was to measure whether a single LLM is capable of
simultaneously conversing as multiple agents and managing them for the user to
converse with as well as observe the domain coverage of the agents. To do so,
we evaluated 1) whether the desired set of agents responded to the user’s
utterance, 2) whether the agents’ identities stayed consistent throughout the
conversation, and 3) how much the set of agents’ criteria represented its
domain. Our procedure involved constructing a set of conversations with
through a taxonomy of user intents (Lyu et al., 2021) and recruiting
evaluators and domain experts for the three evaluation metrics. We recruited
external evaluators (n=4) and domain experts (n=12, 2 per domain) from a
public university to evaluate the three measures. All evaluators were
compensated 20,000 KRW (approximately 15 USD) per hour.

####  6.1.1. Preparation

We produced six unique decision domains (see Appendix B) based on
participants’ past unfamiliar decision-making experiences from the formative
study. We created a set of 15 user utterances input into . We adopted the user
utterances from Lyu et al.’s (Lyu et al., 2021) user intents taxonomy and
adjusted the subcategories to fit multi-agent conversations (Table 2). We used
the 15 utterances to converse with agents in to construct conversations from
each domain. We sent the utterances in an order that followed the flow of the
conversation.

User Intent |  | Example
---|---|---
Ask for Recommendation (AR) |  |
Initial Domain Statement |  | I’m looking for ¡item¿ for ¡context¿.
Ask for other options |  | Could I get another choice similar to ¡option¿?
Ask for options with criteria |  | What are some options for ¡criterion¿?
Provide Preference (PR) |  |
Share context |  | I am looking for an option for ¡context¿.
Provide preference |  | I prefer an ¡option¿ with ¡criterion¿.
Refine preference |  | I actually think ¡criterion¿ would be ¡adj¿.
Inquire (IN) |  |
Ask about criterion |  | How is the ¡criterion¿ for ¡option¿?
Ask about option |  | Tell me more about ¡option¿.
Ask multiple agents |  | ¡agents¿ I want to learn more about your choices.
Rate Recommendation (RR) |  |
Accept recommendation |  | I’m starting to prefer ¡option¿.
Reject recommendation |  | ¡option¿ is too ¡negative criterion¿ for me.
Neutral rating |  | I like how ¡option¿ is ¡positive criterion¿.
Critique (CR) |  |
Critique feature |  | ¡option¿ is too pricey for me.
Add further constraints |  | I want something with more ¡criterion¿.
Compare options |  | ¡option¿ and ¡option¿: what has better ¡criterion¿?

Table 2. Multi-Agent user intent utterances. Adopted and modified from Lyu et
al. (Lyu et al., 2021)

A table showing the adapted user intents taxonomy for multi-agent
conversations. The left column has a list of user intents and their
subcategories, while the right column provides example utterances for each
subcategory.

####  6.1.2. Accuracy of Desired Responding Agents

To measure the accuracy of the desired responses of the multi-agent
conversation, we randomly sampled 20 (4 per 5 major user intents from
taxonomy) instances of at various states of the conversation before the user’s
utterance was sent. Two evaluators were provided with this interface instance
and were allowed to view all information provided in the system. Evaluators
were asked to identify the agents they expect to respond to the user and if
new agents would emerge. We compared the set of agents evaluators selected
with the ones that responded in the actual conversation.

####  6.1.3. Consistency of Agent Identities

To evaluate consistency, we used the conversations generated by the user
intent taxonomy and evaluated the consistency of agents’ utterances throughout
the conversation. We randomly selected three agents per domain (n=18 total)
and gave all selected agents’ utterances to two evaluators to determine
whether each utterance fit the agent’s identity. Evaluators were asked a
true/false question for each utterance. We calculated the average rating (# of
true / # of questions). We exclude the agent’s very first utterance as it is
the introduction of themselves.

####  6.1.4. Domain Coverage of Generated Agents

We determined the domain coverage by measuring the overlaps between the
criteria generated by and domain experts using the approach in (Chilton et
al., 2013). We recruited two domain experts for each of the six unique
decision domains to produce a set of criteria they believed to be
representative of choosing an option in the domain. We additionally produced
two more conversation histories for each domain. We then compared each
expert’s set of criteria to each conversation’s set of criteria, totaling six
comparisons for each domain. Domain experts and may produce different criteria
terms that are synonymous. For instance, one expert noted “ease of use” as a
criterion while another noted “easy to use”. We considered these similar terms
to be the same.

####  6.1.5. Accuracy of Provided Domain Information

We additionally analyzed agent utterances to gauge whether our pipeline
produced hallucinations. We randomly sampled 50 utterances from the generated
conversations, and the researchers of this paper searched to confirm whether
the criteria matched the option mentioned.

###  6.2. Key Findings

####  6.2.1. Accuracy of Desired Responding Agents

We observed the precision and recall of expected agents responding to be
65.07% and 59.35%, respectively. At least one expected agent responded 87.50%
of the time while at least two responded 50.00% of the time. Evaluators
expected the first agent who responded to respond 72.50% of the time. The two
evaluators shared 66.7% of the same guessed agents. A reason for the lower
performance of expected responding agents might be due to the exploratory
nature of some utterances. For example, the utterance, “I think power might be
too much for me since I can generate my own.”, can be inferred that the user
does not want tennis rackets with power, but if there are many agents with
rackets without power, it would be difficult to guess which subset would
respond. Thus, we additionally applied the same calculations for conversation
states where user utterances directly mentioned agents or options. From this,
we observed a precision of 85.91% and a recall of 51.47%. While we observed a
lower recall, we found that directly mentioning agents or options showed
stronger response accuracy than not. The reason for an underperforming recall
is most likely due to one of our prompting methods–––we encourage new agents
to chime in, and this would invite new agents to respond, even if they are not
mentioned.

####  6.2.2. Consistency of agent Identities

We found that agent identities stayed sufficiently consistent throughout the
conversation. The two evaluators rated an average of 87.73% of utterances to
be consistent with the agents’ personas. The inter-rater reliability between
the two evaluators had a Cohen’s Kappa score of 0.756, indicating substantial
agreement. We additionally observed that the average consistency rating of
subsequent utterances throughout the conversation decreased after the first
response, but remained relatively steady (Fig. 10).

####  6.2.3. Domain Coverage of Generated Agents

When comparing the lists of criteria produced by domain experts and the lists
generated by , we found that ’ criteria lists covered an average of 52.85% of
criteria lists produced by domain experts. An average of 42.97% of ’ criteria
were not listed by domain experts, and an average of 37.21% of criteria were
shared by both lists from a domain expert and a conversation. The criteria
coverage throughout spawned agents appears to trend toward saturation (Fig.
11).

####  6.2.4. Accuracy of Provided Domain Information

From our evaluation, we found 1/50 inaccurate utterances in our sample set.
The utterance incorrectly claimed the Microsoft Surface Laptop could last up
to 14.5 hours (last utterance in Appendix B), but Surface Laptops from 2021
and later can last up to 16.5 hours or more (sur, 2023). Most utterances
mostly consisted of opinions and broad information instead of facts (see
Appendix B for example utterances). While hallucination is a critical concern
of LLMs, was designed to primarily share opinions and be conservative with
facts.

|  All Samples |  When Directly Mentioning Agents/Options
---|---|---
Sample Size |  20 |  7
Expected Agents |  |
Precision |  65.07% |  85.91%
Recall |  59.35% |  51.47%
Listed at least one agent |  87.50% |  100.00%
Listed at least two agents |  50.00% |  64.29%
Listed first responding agent |  72.50% |  85.71%

Table 3. Desired accuracy responders evaluation results

A table showing the desired accuracy responders evaluation results. The top
row shows the sample size for the two conditions, which are All Samples and
When Directly Mentioning Agents/Options. The next row shows the precision and
recall values for expected agents. The final three rows show the percentage
results for listing at least one agent, at least two agents, and finally
listing first responding agent.

Figure 10. Average agent consistency rating from their first utterance to
their seventh. (* All sampled agents gave at least four utterances throughout
the conversation. Only half of the sampled agents gave six or more utterances
in the conversation.)

A line graph showing the average agent consistency rating from their first
utterance to their seventh. The y-axis ranges from 0 to 100 percent. The
rating shows an overall decreasing trend across the utterances, starting at
97.22 percent for utterance one and going down to 77.78 percent for utterance
seven.

| Headset | Sports | Tennis | Vacation | Major | Laptop | All Six Domains
---|---|---|---|---|---|---|---
Average Criteria Coverage | 67.95% | 39.50% | 40.28% | 46.99% | 63.26% | 59.11% | 52.85%
Average New Criteria | 26.29% | 62.14% | 41.86% | 58.13% | 59.65% | 9.72% | 42.97%
Average Criteria Overlap | 55.00% | 23.93% | 30.64% | 28.88% | 30.12% | 54.67% | 37.21%

Table 4. Criteria overlap between domain experts and

A table showing the criteria overlap between domain experts and the system. On
the left are labels for the three rows, average criteria coverage, average new
criteria, and average criteria overlap, respectively. The columns are the
seven domains. The criteria overlap is represented as a percentage for each
domain and measure.

Figure 11. Average criteria coverage throughout spawned number of agents

A multi-variable line graph showing the average criteria coverage throughout
spawned number of agents, with the average criteria coverage ranging from 0 to
100 percent as the y-axis and the number of spawned agents ranging from zero
to ten as the x-axis. The graph shows an overall increasing trend for all six
lines, which are Headset, Sports, Tennis, Vacation, Major, and Laptop.

###  6.3. Summary

In summary, the technical evaluation shows that may be able to sufficiently
manage multiple agents while also directing the conversation. is also able to
adequately cover the criteria of online decision-making domains within several
exchanges between multiple agents. While the precision of desired responding
agents was observed to be 65.07%, when directly mentioning agents supported by
direct manipulation controls, the precision appeared to be 85.91%, indicating
a combination of both conversational and directed interaction would most
productively support multi-agent conversation.

##  7\. User Study Setup

We conducted a user study to understand how effectively addresses the
challenges in unfamiliar decision-making. The user study was designed as a
between-subjects study comparing with two other baseline
conditions—conventional web search and single-agent interaction replicating
potential interaction with LLM-based applications (e.g., ChatGPT).

Our evaluation is guided by the following research questions:

  * •

RQ1: How does help users explore a broad information space on the domain?

  * •

RQ2: How does support the discovery and management of relevant information to
the user’s context?

  * •

RQ3: How does the user utilize in the decision-making process?

###  7.1. Baseline Systems

We compared with two other baselines, a web search baseline (hereinafter Web)
and a conversational baseline that resembles ChatGPT (hereinafter
SingleAgent).

#####  Web.

This condition sets no limits on the webpages to explore as long as it is not
video-based 111We included this constraint as the two other conditions only
provide text-based information., and provided users with a sidebar via FigJam
222https://www.figma.com/figjam to record the process as discovered criteria
and options with corresponding screenshots and website URLs. The sidebar is
added as a complement to the summary pane in , to provide a similar experience
for knowledge-building and decision-making.

#####  SingleAgent.

This condition provides users with a single conversational agent and a summary
pane showing discovered criteria/options with pinning and hiding features as .
This is another baseline condition for conversational decision-making, to
understand the effectiveness of multi-agent interaction compared to single-
agent interaction.

###  7.2. Participants

We recruited a total of 36 participants (Age=18-45, M=24.8, SD=5.0; 24 males
and 12 females) who had multiple experiences making decisions online. The
participants were recruited through several universities’ online communities
and social media, including the online community of the institution to which
the first author is affiliated and a Facebook group. To ensure that the
decision domains we chose for the study are unfamiliar to the participants, we
asked the participants to indicate familiarity with the chosen domain through
the recruitment form by asking: “Rate each of the following decision scenarios
in terms of your familiarity and expertise.” We also ensured that the
participants were sufficiently fluent in English with the provided answer, as
and SingleAgent were designed in English. The participants were compensated
20,000 KRW (approximately 15 USD) for one hour of their participation.

###  7.3. Procedure

The study was conducted through Zoom 333https://zoom.us/, where the
participants were asked to prepare a computer or equivalent devices with
audio, video, and screen share settings. We randomly assigned the participants
to one of three conditions, having 12 participants for each condition. We
selected three decision domains—buying a camera, choosing a plant to grow, and
deciding on an instrument to learn—across product selection and experiential
choices. To help connect their own experience with the scenario, the
participants were asked to provide their own purpose or context (e.g., office
interior) to the assigned domain (e.g., choosing a plant) to constitute a
decision scenario (e.g., choosing a plant for office interior) at the
beginning of the study. The decision domains were also randomly assigned to
the participants, having 12 participants for each domain. We employed a
counterbalanced approach to ensure four participants were assigned to each
condition-domain pair. The study lasted for one hour, and consisted of the
following parts:

##### Introduction (10 mins).

The participants were first provided with a brief introduction to the study
and the scenario assigned. They were then asked to fill out a pre-survey
containing questions on a 7-point Likert scale on a self-reported confidence
scale (Collier and Bear, 2012) and open-ended questions on the
criteria/options/preferences they currently have on the assigned domain in
bullet points. Afterward, they were introduced to the interface (setup for
Web) of the condition assigned.

##### Decision-making task with given condition/scenario (30 mins).

After the introduction, the participants were asked to use the assigned
interface with the provided scenario to get to a decision. We asked the
participants to either have a strongly preferred option or land on a few
preferred options. This was to minimize the variability of participants’ time
in having to make the final decision and the number of choices users ended
within the study.

##### Post-survey and interview (20 mins).

After the task, participants were asked to complete a post-survey, containing
7-point Likert scale questions on effective discovery and management of
information (from the DGs), self-reported confidence, and NASA-TLX for
cognitive load (Hart and Staveland, 1988), along with open-ended questions on
new criteria, options, and preferences they discovered on the assigned domain
in bullet points. Afterward, they were asked questions on features and
interface usage, reliability and engagement, and the overall decision-making
process. For participants in , we additionally asked about the pros and cons
of multi-agent conversation and the potential use of multi-agents in other
decision scenarios.

###  7.4. Data Analysis

For RQ1: Exploration and understanding of the domain, we analyzed the domain
and preference gains by comparing the number of criteria/options written by
the participants in the open-ended questions in pre-and post-survey.

For RQ2: Utilization of relevant information, we ran a Kruskal-Wallis test
across three conditions and the Posthoc Dunn H-test to identify the
significant condition (p ¡ 0.05) for the Likert-scale survey results:
effective discovery and management of information, self-reported confidence,
and cognitive cost.

For RQ3: Utilizing , we analyzed the session logs for participants in how
agents were utilized in the conversation, focus mode, and the summary bar to
derive patterns in using the features.

##  8\. User Study Results

Our evaluation result showed that outperformed Web for supporting discovery,
deeper understanding, and management of suitable information to get to a
decision with more confidence (Table 5), but not in comparison to SingleAgent.
Participants agreed that decision-making support of is helpful in various
stages of decision-making, and indicated that they would use and SingleAgent
for future unfamiliar decision-making situations, while not as much for Web.
We additionally report on how participants utilized different features to
assist their decision-making process.

###  8.1. RQ1: Exploration and Understanding of the Domain

Participants using were able to discover around 14 criteria and 10 options
during the 30-minute timeframe. Although there were no significant differences
between conditions, participants were satisfied with the effective exploration
and understanding they gained in a short period of time.

Using and SingleAgent, participants were able to discover on average 13.917
(SD=4.542) and 10.333 (SD=5.051) criteria and on average 10.333 (SD=5.630) and
10.583 (SD=3.397) options in 30 minutes. While participants on average
discovered relatively more criteria, there were no significant differences for
both keywords (criteria: p=0.0983, options: p=0.5813). participants liked the
diversity of the found information and described they now have an overview
knowledge in the domain (C8), while SingleAgent participants described the
information as “adaptive to my questions but not diverse” (S3).

In terms of knowledge gain, participants in were able to remember 4.417
criteria (SD=1.730) on average, which was relatively more than Web and
SingleAgent but not significant (Web: M=3.333, SD=1.303 / SingleAgent:
M=3.167, SD=1.801 / p=0.1407). C12 (Camera) commented after using , “I was
able to understand the key criteria of the domain that many people value, and
for future camera purchases [with different contexts] I would refer to the
criteria I learned today.” For new options learned, there was no significant
difference between the three conditions (: M=3.333, SD=1.557 / SingleAgent:
M=2.250, SD=0.866 / Web: M=3.083, SD=2.678 / p=0.17) as participants mostly
compared between a few choices to get to the decision.

###  8.2. RQ2: Utilization of relevant information

The participants were able to successfully utilize relevant information in the
domain to make a decision. outperformed Web in supporting users to both
discover and manage relevant information with less cognitive load and
supported better exploration and comparison of diverse perspectives in the
domain than SingleAgent with similar cognitive load.

There was a significant difference in the conditions on how they supported
participants to explore diverse perspectives (H=11.176, p=0.0037). Dunn-
Bonferroni post-hoc test revealed that there were significant differences
between Web (M=4.250, SD=1.422) and (M=6.083, SD=0.793, p=0.0024). In
addition, participants rated higher than Web and SingleAgent (: M=6.167,
SD=1.337 / SingleAgent: M=5.000, SD= 1.477 / Web: M=4.000, SD=1.477 / H=5.408,
p=0.0669), but there were no significant differences. Participants in liked
how the system “casually threw in information I cannot find as I don’t know”
(C3) and said they could “compare [diverse opinions] in a more neutral way
than I normally could” (C7). Meanwhile, participants in SingleAgent indicated
that they felt the information to be “unbiased but not too diverse” (S7) and
found the interface “not helpful in comparing, and just went down a single
path” (S2). Some participants in Web were not able to pay attention to diverse
perspectives “as they mainly focused on learning the domain” (W4) within the
time frame.

There was a significant difference between conditions for flexible management
of the discovered information (H=7.551, p=0.0229). Dunn-Bonferroni post-hoc
test revealed that there were significant differences between Web (M=4.833,
SD=1.403) and (M=5.667, SD=1.073, p=0.0207). The participants in liked the
freeform scattering of the agents, so that they could move the agents in terms
of their preference. Some participants additionally commented that they would
like to have a feature for deleting agents, to keep the conversation space
less cluttered. For identifying key criteria in the domain, and SingleAgent
was perceived to be more helpful but there was no significant difference (Web:
M=5.417, SD=1.975 / SingleAgent: M=6.667, SD=0.492 / : M=6.333, SD=0.985,
H=5.081, p=0.0788).

| Web | SingleAgent |  | Kruskal-Wallis test |  | W-S
---
| W-C
---
| S-C
---
| mean | std | mean | std | mean | std | H | p | p | p | p
Exploring perspectives | 4.250 | 1.422 | 5.250 | 1.138 | 6.083 | 0.793 | 11.176 | 0.0037 | ns | ** | ns
Comparing perspectives | 4.000 | 1.477 | 5.000 | 1.477 | 6.167 | 1.337 | 5.408 | 0.0669 | - | - | -
Diving deeper | 4.667 | 1.775 | 5.833 | 1.030 | 5.583 | 1.240 | 12.337 | 0.0021 | ns | ** | ns
Discovering adaptive info | 4.750 | 1.960 | 6.167 | 1.193 | 6.167 | 0.718 | 3.056 | 0.2170 | - | - | -
Identifying key criteria | 5.417 | 1.975 | 6.667 | 0.492 | 6.333 | 0.985 | 5.081 | 0.0788 | - | - | -
Flexible management | 4.833 | 1.403 | 4.750 | 1.603 | 5.667 | 1.073 | 7.551 | 0.0229 | ns | * | ns
Effective reduction | 4.750 | 1.765 | 5.833 | 1.403 | 6.333 | 0.888 | 3.017 | 0.2212 | - | - | -
Future use | 4.333 | 2.015 | 6.333 | 0.778 | 6.500 | 0.798 | 10.868 | 0.0044 | * | ** | ns
Agg. self-confidence | 3.528 | 1.159 | 4.750 | 1.016 | 4.639 | 0.670 | 8.803 | 0.0123 | * | * | ns
Agg. NASA-TLX | 3.069 | 1.162 | 2.139 | 0.752 | 2.194 | 0.611 | 4.275 | 0.1179 | - | - | -

Table 5. Likert-scale survey results (*: p ¡ .050, **: p ¡ .010)

A table showing the results of Likert-scale survey questions. There are 11
columns. On the left is the theme of the survey questions, to the right, there
is mean and std for Web, SingleAgent, and ChoiceMates, H and p values of
Kruskal-Wallis test, and post-hoc test results.

###  8.3. RQ3: Utilizing for decision-making

Using , participants engaged in 16.17 turns (SD=8.13) of conversations,
utilizing 10.92 agents (SD=3.48) on average. Most (11/12) participants
commented that they enjoyed using the system, and perceived decision-making
with as “having step-by-step guidance with a group to reach a decision” (C11).
The participants enjoyed the flexibility of , and each utilized a unique set
of features to their needs and preferences. Hereafter, we describe in detail
how participants utilized the conversation with agents and the summary bar to
get to the decision.

####  8.3.1. Conversations with Agents

In , we provide three modes of talking with the agents: (1) talking to all
agents in the conversation space, (2) selecting a single agent to talk with,
and (3) selecting a set of agents to talk with. Participants actively utilized
all options.

Participants chose to talk with only one agent when they wanted to explore an
option in depth. Some participants (C7, C11) largely conversed with a single
agent. For example, C11 chatted with Alex (guitar), Pat (electric guitar), and
Dylan (drum) for a total of 16 turns out of 30. They asked questions about the
price, types of music, and specific parts of the instrument to gain an in-
depth understanding of each instrument. C11 commented that “the feature to
click on an agent to ask additional questions was the most helpful” and added
how that helped in terms of information management in that “I could easily
identify which agent to go to among the set of agents.”

Participants selected a set of agents to talk with when they wanted to (1)
receive information in batches or (2) compare between multiple options. Half
of the participants (C2, C3, C6, C9, C10, C12) selected a set of agents to
talk with for more than half of the session. C3 commented that “on the web, if
I want to look for 2 criteria for different plants, I need to make 6 searches.
Here, I could select and ask a question [with the two criteria included].”
When they wanted a more detailed comparison between options, they opened the
_focus view_. The participants not only received agents’ opinions on certain
criteria but also could make agents argue to better understand their
perspective. C4 was persuaded by Reese (Pothos) during a comparison between
Parker (Peace Lily) and Reese (Pothos). When C4 asked the question “Is peace
lily sensitive to lighting condition?”, Reese replied “That’s one advantage
the Pothos has over the Peace Lily. Pothos is more adaptable and can thrive in
a variety of light conditions […] Parker, wouldn’t you agree that adaptability
is a valuable trait in a plant?” C4 discovered a new criterion ‘adaptability’
and was persuaded to change their decision from Peace Lily to Pothos.

8 out of 12 participants utilized _Thought Bubbles_ instead of manually typing
in their response. The participants could discover new important criteria or
understand criteria in more depth in a way that is “less stressful and
enjoyable” (P1). C8 used it four times and commented that “without basic
knowledge in the domain, I had difficulty coming up with questions to
ask.”They discovered the importance of lenses with the Thought Bubble “Are
there affordable lenses for Olympus OM-D E-M1 Mark III?”

####  8.3.2. Summary Bar

The summary bar was used to (1) keep track of the conversation summary and the
user preference, (2) manage the agents in the information space better, and
(3) record user preference and direct the conversation. While it was not the
center of focus during the task, participants actively used the summary bar
throughout.

Since the criteria and the options were automatically added to the summary bar
with the conversation, participants were able to access the organized
information on the side for them to refer to throughout the process. Even for
criteria they did not find relevant, C2 commented that “[having] it will be
helpful for anyone with different uses.” With filtering, the participants
could find the agents that suited their preferences. Participants perceived it
as similar to the “summary bar on booking.com where you can sort and filter
different options” (C5) and liked how “I was able to filter the agents
[instead of the system]” (C9). Profile cards were used by the participants to
save important criteria and options throughout the session. C3 enjoyed how the
new conversations reflected the information on the profile card when they saw
that later plant recommendations were centered around pinned criteria:
‘durable’ and ‘light’. However, C5 saw that as a limiting factor for more
serendipitous exploration of the domain.

##  9\. Discussion

In this section, we discuss the values of having multi-agents in
conversational interaction, human-AI collaborative decision-making,
alternative designs of agents and conversation, and generalizability of for
other decision-making domains. We also discuss limitations and possible future
work.

###  9.1. Values of Multi-agents in Conversational Interaction

While supported a better decision-making process compared to Web, SingleAgent
showed similar statistical advantages in supporting effective discovery and
management of information. Despite this, qualitative insights that were not
measured in the study still showed values of a multi-agent design for
decision-making support.

The participants saw multiple perspectives presented in a single space as a
strong benefit as they could approach the domain with less bias toward
particular opinions. Participants found beneficial as “everyone has a bias, so
it helps seeing perspectives” (C10). On the other hand, sitting on a single
perspective throughout the decision-making process could lead to them missing
out on key information and may require extended conversation to reach such
various perspectives. C11 appreciated being able to “see various opinions at a
single glance” and mentioned that “without the agents, I can easily get caught
into a single option. While some participants found differing opinions too
conflicting, others felt they were constantly exposed to more arguments to
strengthen their preferences. Seeing those made participants ask more
questions, which led them to dive deeper into the domain.

A multi-agent design comes with the trade-off of more control over the process
for simplicity. While C8 mentioned, “organizing the information myself helped
by identifying what is more relevant”, C10 shared that a simple focus view
“might be the easiest way to view [agents] because it’s laid out for you.” We
believe the values of control could be further revealed by comparing the
conditions in a longer cycle, where the aftereffects of the decisions are also
examined. Furthermore, although having multiple agents made the overall
process more complex, having agents and their persona eased the participants’
burden in identifying relevant information, instead of manually filtering out
information within a single response.

###  9.2. Human-AI Collaborative Decision-Making

In designing mixed-initiative systems, carefully designing the control between
the users and the system is crucial for user experience and task completion
(Linegang et al., 2003). In , after each turn, the initiative shifts between
the user and the system. Compared to the traditional chatbot where the agents
only provide answers to the user’s messages, we design a system where agents
visually pop up to provide response or assistance. In the system, the user can
take control and adjust this level of initiative, where the user can mention
(i.e. tag) specific agents to converse further or use the thought bubble to
get guidance if the user is lost in asking the appropriate questions. If was
to be designed with full automation, it would be a set of agents conversing
with each other to get to the answer, making it difficult for the user to
fully comprehend the information and convert it to their own preference. If it
was fully manual, the user would face difficulty asking questions.
successfully demonstrates an example of human-AI collaborative workflow for
decision-making with the right level of initiative by the system, that is
adjustable throughout the process.

###  9.3. Alternative Designs of Agents and Conversations

In , an _agent_ is the basic unit of the conversational interaction consisting
of a persona, valued criteria, and a valued option, intended to help users
identify the link between criteria and options. The agents are designed to
connect the individual experience to the persona.

As each agent is associated with a single option, the participants found the
agents clear to identify and understand their values. C1 mentioned “one agent
with multiple options can be overwhelming as there are already multiple agents
in the space.” and liked the current design. In our earlier iterations, we
considered having different types of agents that deliver different
information, from factual, and opinionated, to summarizing the information.
However, we decided to stick with opinionated agents with single-option
preference due to the complexity of the agent-agent conversation design and
the requirement for additional resources. However, agents with varying levels
of expertise and intensity of preference could better reflect a real-world
setting to imitate the benefit that multiple types of people can provide. For
example, there could be an _expert supporting agent_ who only adds to the
agent’s utterance to further back up with more information or provide an
opposing viewpoint, or a personal agent (Kitamura et al., 2002; Kitamura,
2004) that guides the user through the process.

###  9.4. Geneneralizability of for Other Types of Decisions

is primarily designed for unfamiliar decisions, thus more emphasis on
understanding diverse perspectives of the domain was made in designing the
system. With the flexibility of and the variety of the functions it offers, we
believe that also can be used to support more familiar and complex decisions.

For familiar decisions, users would have more knowledge (i.e. criteria and
options) on the domain and clearer preferences (Bessette et al., 2021). Thus,
we anticipate user utterances to be more specific to the user’s existing
knowledge or preference, and the user would expect responses involving
stronger arguments (i.e., comparing and contrasting). C9 stated that they
would “try debating with the agent to see if their preferences make sense.” In
addition, C12 stated that they would “provide a list of candidate options and
ask the agents to compare and contrast the options.” To take into account
those questions, the agents should also be designed as more expert-like and
attribute-centered, as shown through the findings of Knijnenburg et al.
(Knijnenburg and Willemsen, 2009).

For more complex decisions that involve criteria in hierarchies (e.g.,
choosing a Ph.D. advisor) or multiple steps (e.g., trip planning), we believe
additional UI supports to could help the user better manage the process and
the multi-agent conversation. During the study, C8 (camera) wanted to explore
camera lenses in order to confidently decide on the camera. Since camera
lenses also involve a set of important criteria, the summary pane can be
hierarchical or in branches (as proposed in Suchmann et al. (Suchmann et al.,
2023)) where specific criteria can be chosen to be explored further. For
decisions involving multiple steps, the agents can reveal their options for
each stage while keeping their persona. For example, for trip planning, an
agent can reveal options for booking a flight, choosing a hotel, and deciding
on a tourist spot, but have a universal set of valued criteria that drive the
options. The user’s choices made in the previous step can serve as the
‘context’ for subsequent decisions.

There are also decisions where people heavily rely on information other than
text, such as images or videos. still lacks informational support beyond text,
such as images or videos, which made the participants feel a gap in terms of
the information available on the web and in . Especially in the choosing a
plant to grow scenario, participants felt that visiting the provided link to
view the plants was a major huddle as many saw the look of the plant as
important for the decision. Directly incorporating multi-modal information and
designing interactions to better manage the information through the summary
pane can be further investigated in the future.

###  9.5. Limitations and Future Work

While crawled information from the web to provide agents with correct factual
information, there still remains the possibility for the agents to hallucinate
throughout the conversation. Providing such information can result in over-
reliance of the user and influence the decision outcome, especially for high-
stakes decisions. In addition, the agents gave less opinionated responses due
to the characteristics of LLMs being less opinionated. Future work could
incorporate additional fact-checking pipelines such as Retrieval Augmented
Generation (RAG) (Lewis et al., 2020) to support the utilization of more
reliable information, and additionally utilizing opinions online (e.g., online
reviews, blog posts) in generating the agents could make the agents provide
responses reflecting single persona’s experience.

In addition, we observed during the study that our design of baselines to a
summary bar made participants see the baseline conditions as an enhanced
version of web search and ChatGPT with additional support, which might have
hindered the study results from revealing the core effects of multi-agent
interaction over other conditions. Future works could observe the effects with
specific stages of decision-making (e.g., information collection, comparison),
or add additional conditions (e.g., multi-agent conversation without summary
bar) to understand the benefit of multi-agent conversation.

Finally, our study was carried out as a lab study, where participants were
given a pre-assigned decision domain and a relatively short time frame (i.e.
30 minutes) to make the decision. As a result, some participants did not have
a strong intrinsic motivation during the process, which might not accurately
reflect the real-world unfamiliar decision-making. Future work could use
longer-term, live deployments in investigating the effect of .

##  10\. Conclusion

In this work, we propose , a multi-agent conversational system that supports
unfamiliar decision-making. Our between-subjects study comparing to a web
search condition and a single agent condition reveals that is able to support
the discovery and understanding of a wider, deeper range of information in the
domain and successfully support the management of information to make a more
confident decision compared to Web. We describe how users used specific
features of for decision-making, then discuss future applications of and the
multi-agent conversational system for decision-making support.

## References

  * (1)
  * bab (2023) 2023\.  BabyAGI.  https://github.com/yoheinakajima/babyagi Accessed: September 10, 2023.
  * Cha (2023) 2023\.  ChatGPT Plugins.  https://openai.com/blog/chatgpt-plugins Accessed: September 3, 2023.
  * Kha (2023) 2023\.  Khanmigo.  https://blog.khanacademy.org/teacher-khanmigo/ Accessed: September 13, 2023.
  * sur (2023) 2023\.  Microsoft Surface Laptops.  https://www.microsoft.com/en-us/surface/devices/surface-laptop-models Accessed: September 14, 2023.
  * Bessette et al. (2021) Douglas L Bessette, Robyn S Wilson, and Joseph L Arvai. 2021.  Do people disagree with themselves? Exploring the internal consistency of complex, unfamiliar, and risky decisions.  _Journal of Risk Research_ 24, 5 (2021), 593–605.
  * Bettman and Park (1980) James R Bettman and C Whan Park. 1980.  Effects of Prior Knowledge and Experience and Phase of the Choice Process on Consumer Decision Processes: A Protocol Analysis.  _J. Consum. Res._ 7, 3 (Dec. 1980), 234–248.  https://doi.org/10.1086/208812
  * Candello and Pinhanez (2016) Heloisa Candello and Claudio Pinhanez. 2016.  Designing conversational interfaces.  _Simpósio Brasileiro sobre Fatores Humanos em Sistemas Computacionais-IHC_ (2016).  https://www.researchgate.net/profile/Heloisa-Candello/publication/333022441_Designing_Conversational_Interfaces/links/5cd6d7b1458515712ea35172/Designing-Conversational-Interfaces.pdf
  * Chan et al. (2023) Chi-Min Chan, Weize Chen, Yusheng Su, Jianxuan Yu, Wei Xue, Shanghang Zhang, Jie Fu, and Liu Zhiyuan. 2023.  ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate.  _arXiv preprint arXiv:2308.07201_ (2023).
  * Chang et al. (2020) Joseph Chee Chang, Nathan Hahn, and Aniket Kittur. 2020.  Mesh: Scaffolding Comparison Tables for Online Decision Making. In _Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology_ (Virtual Event, USA) _(UIST ’20)_. Association for Computing Machinery, New York, NY, USA, 391–405.  https://doi.org/10.1145/3379337.3415865
  * Chen et al. (2013) Li Chen, Marco deGemmis, Alexander Felfernig, Pasquale Lops, Francesco Ricci, and Giovanni Semeraro. 2013.  Human Decision Making and Recommender Systems.  _ACM Transactions on Interactive Intelligent Systems_ 3 (10 2013).  https://doi.org/10.1145/2533670.2533675
  * Chen et al. (2023) Weize Chen, Yusheng Su, Jingwei Zuo, Cheng Yang, Chenfei Yuan, Chen Qian, Chi-Min Chan, Yujia Qin, Yaxi Lu, Ruobing Xie, Zhiyuan Liu, Maosong Sun, and Jie Zhou. 2023.  AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors in Agents.  _arXiv preprint arXiv:2308.10848_ (2023).
  * Chilton et al. (2013) Lydia B. Chilton, Greg Little, Darren Edge, Daniel S. Weld, and James A. Landay. 2013.  Cascade: Crowdsourcing Taxonomy Creation. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Paris, France) _(CHI ’13)_. Association for Computing Machinery, New York, NY, USA, 1999–2008.  https://doi.org/10.1145/2470654.2466265
  * Collier and Bear (2012) Benjamin Collier and Julia Bear. 2012.  Conflict, Criticism, or Confidence: An Empirical Examination of the Gender Gap in Wikipedia Contributions. In _Proceedings of the ACM 2012 Conference on Computer Supported Cooperative Work_ (Seattle, Washington, USA) _(CSCW ’12)_. Association for Computing Machinery, New York, NY, USA, 383–392.  https://doi.org/10.1145/2145204.2145265
  * Cox et al. (2023) Samuel Rhys Cox, Ashraf Abdul, and Wei Tsang Ooi. 2023.  Prompting a Large Language Model to Generate Diverse Motivational Messages: A Comparison with Human-Written Messages.  _arXiv preprint arXiv:2308.13479_ (2023).
  * Curşeu et al. (2015) Petru L Curşeu, Nicoleta Meslec, Helen Pluut, and Gerardus JM Lucas. 2015.  Cognitive synergy in groups and group-to-individual transfer of decision-making competencies.  _Frontiers in psychology_ 6 (2015), 1375.
  * et al. (2023) Torantulino et al. 2023.  AutoGPT.  https://github.com/Significant-Gravitas/Auto-GPT
  * Flohr et al. (2021) Lukas A. Flohr, Sofie Kalinke, Antonio Kr‘̀uger, and Dieter P. Wallach. 2021.  Chat or Tap? – Comparing Chatbots with ‘Classic’ Graphical User Interfaces for Mobile Interaction with Autonomous Mobility-on-Demand Systems. In _Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction_ (Toulouse & Virtual, France) _(MobileHCI ’21)_. Association for Computing Machinery, New York, NY, USA, Article 21, 13 pages.  https://doi.org/10.1145/3447526.3472036
  * Foo et al. (2022) Michelle Xiao-Lin Foo, Luca Della Libera, and Ilhan Aslan. 2022.  Papr Readr Bot: A Conversational Agent to Read Research Papers. In _Proceedings of the 4th Conference on Conversational User Interfaces_ (Glasgow, United Kingdom) _(CUI ’22)_. Association for Computing Machinery, New York, NY, USA, Article 39, 4 pages.  https://doi.org/10.1145/3543829.3544536
  * Friedman et al. (2023) Luke Friedman, Sameer Ahuja, David Allen, Terry Tan, Hakim Sidahmed, Changbo Long, Jun Xie, Gabriel Schubiner, Ajay Patel, Harsh Lara, et al. 2023\.  Leveraging Large Language Models in Conversational Recommender Systems.  _arXiv preprint arXiv:2305.07961_ (2023).
  * Gao et al. (2021) Chongming Gao, Wenqiang Lei, Xiangnan He, Maarten de Rijke, and Tat-Seng Chua. 2021.  Advances and challenges in conversational recommender systems: A survey.  _AI Open_ 2 (2021), 100–126.
  * Goertzel (2017) Ben Goertzel. 2017.  A formal model of cognitive synergy. In _Artificial General Intelligence: 10th International Conference, AGI 2017, Melbourne, VIC, Australia, August 15-18, 2017, Proceedings 10_. Springer, 13–22.
  * Gupta et al. (2022) Akshit Gupta, Debadeep Basu, Ramya Ghantasala, Sihang Qiu, and Ujwal Gadiraju. 2022.  To trust or not to trust: How a conversational interface affects trust in a decision support system. In _Proceedings of the ACM Web Conference 2022_. 3531–3540.
  * Hart and Staveland (1988) Sandra G. Hart and Lowell E. Staveland. 1988.  Development of NASA-TLX (Task Load Index): Results of Empirical and Theoretical Research.  In _Human Mental Workload_ , Peter A. Hancock and Najmedin Meshkati (Eds.). Advances in Psychology, Vol. 52. North-Holland, 139–183.  https://doi.org/10.1016/S0166-4115(08)62386-9
  * Häubl and Trifts (2000) Gerald Häubl and Valerie Trifts. 2000.  Consumer decision making in online shopping environments: The effects of interactive decision aids.  _Marketing science_ 19, 1 (2000), 4–21.
  * Hecht et al. (2012) Brent Hecht, Jaime Teevan, Meredith Morris, and Dan Liebling. 2012.  Searchbuddies: Bringing search engines into the conversation. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 6. 138–145.
  * Hong et al. (2023) Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, and Chenglin Wu. 2023.  MetaGPT: Meta Programming for Multi-Agent Collaborative Framework.  _arXiv preprint arXiv:2308.00352_ (2023).
  * Iyengar and Lepper (2000) Sheena S Iyengar and Mark R Lepper. 2000.  When choice is demotivating: Can one desire too much of a good thing?  _Journal of personality and social psychology_ 79, 6 (2000), 995.
  * Jain et al. (2018) Mohit Jain, Pratyush Kumar, Ramachandra Kota, and Shwetak Patel. 2018.  Evaluating and Informing the Design of Chatbots. 895–906.  https://doi.org/10.1145/3196709.3196735
  * Jameson et al. (2015) Anthony Jameson, Martijn C Willemsen, Alexander Felfernig, Marco De Gemmis, Pasquale Lops, Giovanni Semeraro, and Li Chen. 2015.  Human decision making and recommender systems.  _Recommender systems handbook_ (2015), 611–648.
  * Jannach et al. (2021) Dietmar Jannach, Ahtsham Manzoor, Wanling Cai, and Li Chen. 2021.  A Survey on Conversational Recommender Systems.  _ACM Comput. Surv._ 54, 5 (May 2021), 1–36.  https://doi.org/10.1145/3453154
  * Jiang et al. (2023b) Peiling Jiang, Jude Rayan, Steven Dow, and Haijun Xia. 2023b.  Graphologue: Exploring Large Language Model Responses with Interactive Diagrams.  _arXiv preprint arXiv:2305.11473_ (2023).
  * Jiang et al. (2023a) Zhiqiu Jiang, Mashrur Rashik, Kunjal Panchal, Mahmood Jasim, Ali Sarvghad, Pari Riahi, Erica DeWitt, Fey Thurber, and Narges Mahyar. 2023a.  CommunityBots: Creating and Evaluating A Multi-Agent Chatbot Platform for Public Input Elicitation.  _Proceedings of the ACM on Human-Computer Interaction_ 7, CSCW1 (2023), 1–32.
  * Karimi et al. (2015) Sahar Karimi, K Nadia Papamichail, and Christopher P Holland. 2015.  The effect of prior knowledge and decision-making style on the online purchase decision-making process: A typology of consumer shopping behaviour.  _Decis. Support Syst._ 77 (Sept. 2015), 137–147.  https://doi.org/10.1016/j.dss.2015.06.004
  * Kitamura (2004) Yasuhiko Kitamura. 2004.  Web information integration using multiple character agents.  _Life-Like Characters: Tools, Affective Functions, and Applications_ (2004), 295–315.
  * Kitamura et al. (2002) Yasuhiko Kitamura, Hideki Tsujimoto, Teruhiro Yamada, and Taizo Yamamoto. 2002.  Multiple Character-Agents Interface: An Information Integration Platform Where Multiple Agents and Human User Collaborate. In _Proceedings of the First International Joint Conference on Autonomous Agents and Multiagent Systems: Part 2_ (Bologna, Italy) _(AAMAS ’02)_. Association for Computing Machinery, New York, NY, USA, 790–791.  https://doi.org/10.1145/544862.544925
  * Knijnenburg and Willemsen (2009) Bart P. Knijnenburg and Martijn C. Willemsen. 2009.  Understanding the Effect of Adaptive Preference Elicitation Methods on User Satisfaction of a Recommender System. In _Proceedings of the Third ACM Conference on Recommender Systems_ (New York, New York, USA) _(RecSys ’09)_. Association for Computing Machinery, New York, NY, USA, 381–384.  https://doi.org/10.1145/1639714.1639793
  * LangChain (2023) LangChain. 2023.  LangChain Contextual Compression.  https://python.langchain.com/docs/modules/data_connection/retrievers/contextual_compression/ Accessed: September 10, 2023.
  * Lee and Lee (2004) Byung-Kwan Lee and Wei-Na Lee. 2004.  The effect of information overload on consumer choice quality in an on-line environment.  _Psychology & Marketing_ 21, 3 (2004), 159–183.
  * Leiter et al. (2023) Christoph Leiter, Ran Zhang, Yanran Chen, Jonas Belouadi, Daniil Larionov, Vivian Fresen, and Steffen Eger. 2023.  ChatGPT: A Meta-Analysis after 2.5 Months.  _arXiv preprint arXiv:2302.13795_ (2023).
  * Lewis et al. (2020) Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. 2020\.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ 33 (2020), 9459–9474.
  * Li et al. (2023a) Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. 2023a.  CAMEL: Communicative Agents for ”Mind” Exploration of Large Scale Language Model Society.  _arXiv preprint arXiv:2303.17760_ (2023).
  * Li et al. (2023b) Shuo Li, Xiang Yuan, Xinyuan Zhao, and Shirao Yang. 2023b.  Leaders or Team-Mates: Exploring the Role-Based Relationship Between Multiple Intelligent Agents in Driving Scenarios: Research on the Role-Based Relationship Between Multiple Intelligent Agents in Driving Scenarios. Springer-Verlag, Berlin, Heidelberg, 144–165.  https://doi.org/10.1007/978-3-031-35678-0_9
  * Li et al. (2020) Toby Jia-Jun Li, Jingya Chen, Haijun Xia, Tom M Mitchell, and Brad A Myers. 2020.  Multi-modal repairs of conversational breakdowns in task-oriented dialogs. In _Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology_. 1094–1107.
  * Li et al. (2021) Toby Jia-Jun Li, Tom M Mitchell, and Brad A Myers. 2021.  Demonstration+ natural language: multimodal interfaces for GUI-based interactive task learning agents.  _Artificial Intelligence for Human Computer Interaction: A Modern Approach_ (2021), 495–537.
  * Liao et al. (2020) Qingzi Vera Liao, Werner Geyer, Michael J. Muller, and Yasaman Khazaen. 2020.  Conversational Interfaces for Information Search.  https://api.semanticscholar.org/CorpusID:219810462
  * Liffiton et al. (2023) Mark Liffiton, Brad Sheese, Jaromir Savelka, and Paul Denny. 2023.  CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes.  _arXiv preprint arXiv:2308.06921_ (2023).
  * Linegang et al. (2003) Michael Linegang, Craig Haimson, Jean MacMillan, and Jared Freeman. 2003.  Human control in mixed-initiative systems: lessons from the MICA-SHARC program. In _SMC’03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme-System Security and Assurance (Cat. No. 03CH37483)_ , Vol. 1. IEEE, 436–441.
  * Liu et al. (2019) Michael Xieyang Liu, Jane Hsieh, Nathan Hahn, Angelina Zhou, Emily Deng, Shaun Burley, Cynthia Taylor, Aniket Kittur, and Brad A Myers. 2019.  Unakite: Scaffolding Developers’ Decision-Making Using the Web. In _Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology_ (New Orleans, LA, USA) _(UIST ’19)_. Association for Computing Machinery, New York, NY, USA, 67–80.  https://doi.org/10.1145/3332165.3347908
  * Liu et al. (2022) Michael Xieyang Liu, Aniket Kittur, and Brad A Myers. 2022.  Crystalline: Lowering the Cost for Developers to Collect and Organize Information for Decision Making. In _Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems_ (New Orleans, LA, USA) _(CHI ’22, Article 68)_. Association for Computing Machinery, New York, NY, USA, 1–16.  https://doi.org/10.1145/3491102.3501968
  * Liu et al. (2023) Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023.  Lost in the Middle: How Language Models Use Long Contexts.  _arXiv preprint arXiv:2307.03172_ (2023).
  * Loepp et al. (2014) Benedikt Loepp, Tim Hussein, and Jüergen Ziegler. 2014.  Choice-Based Preference Elicitation for Collaborative Filtering Recommender Systems. In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ (Toronto, Ontario, Canada) _(CHI ’14)_. Association for Computing Machinery, New York, NY, USA, 3085–3094.  https://doi.org/10.1145/2556288.2557069
  * Lops et al. (2011) Pasquale Lops, Marco De Gemmis, and Giovanni Semeraro. 2011.  Content-based recommender systems: State of the art and trends.  _Recommender systems handbook_ (2011), 73–105.
  * Luger and Sellen (2016) Ewa Luger and Abigail Sellen. 2016.  “Like Having a Really Bad PA”: The Gulf between User Expectation and Experience of Conversational Agents. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ (San Jose, California, USA) _(CHI ’16)_. Association for Computing Machinery, New York, NY, USA, 5286–5297.  https://doi.org/10.1145/2858036.2858288
  * Lunenburg (2010) Fred C Lunenburg. 2010.  The decision making process.. In _National Forum of Educational Administration & Supervision Journal_, Vol. 27.
  * Lyu et al. (2021) Shengnan Lyu, Arpit Rana, Scott Sanner, and Mohamed Reda Bouadjenek. 2021.  A Workflow Analysis of Context-Driven Conversational Recommendation. In _Proceedings of the Web Conference 2021_ (Ljubljana, Slovenia) _(WWW ’21)_. Association for Computing Machinery, New York, NY, USA, 866–877.  https://doi.org/10.1145/3442381.3450123
  * Ma et al. (2023) Zilin Ma, Yiyang Mei, and Zhaoyuan Su. 2023.  Understanding the benefits and challenges of using large language model-based conversational agents for mental well-being support.  _arXiv preprint arXiv:2307.15810_ (2023).
  * Malhotra (1983) Naresh K Malhotra. 1983.  On’Individual differences in search behavior for a nondurable’.  _Journal of Consumer Research_ 10, 1 (1983), 125–131.
  * Moore and Lehmann (1980) William L Moore and Donald R Lehmann. 1980.  Individual differences in search behavior for a nondurable.  _Journal of consumer research_ 7, 3 (1980), 296–307.
  * Nourani et al. (2020) Mahsan Nourani, Joanie T. King, and Eric D. Ragan. 2020.  The Role of Domain Expertise in User Trust and the Impact of First Impressions with Intelligent Systems. In _AAAI Conference on Human Computation & Crowdsourcing_.  https://api.semanticscholar.org/CorpusID:221186776
  * Peng et al. (2021) Minjing Peng, Zhicheng Xu, and Haiyang Huang. 2021.  How does information overload affect consumers’ online decision process? An event-related potentials study.  _Frontiers in Neuroscience_ 15 (2021), 695852.
  * Radlinski and Craswell (2017) Filip Radlinski and Nick Craswell. 2017.  A Theoretical Framework for Conversational Search. In _Proceedings of the 2017 Conference on Conference Human Information Interaction and Retrieval_ (Oslo, Norway) _(CHIIR ’17)_. Association for Computing Machinery, New York, NY, USA, 117–126.  https://doi.org/10.1145/3020165.3020183
  * Samir and Habiba (2009) Kechid Samir and Drias Habiba. 2009.  Mutli-agent system for personalizing Information Source Selection. In _2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology_ , Vol. 1. IEEE, 588–595.
  * Schafer et al. (2007) J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. 2007.  Collaborative filtering recommender systems.  In _The adaptive web: methods and strategies of web personalization_. Springer, 291–324.
  * Scheibehenne et al. (2010) Benjamin Scheibehenne, Rainer Greifeneder, and Peter M Todd. 2010.  Can there ever be too many options? A meta-analytic review of choice overload.  _Journal of consumer research_ 37, 3 (2010), 409–425.
  * Shapira (2010) Bracha Shapira. 2010.  Recommender Systems Handbook. In _Springer US_.  https://api.semanticscholar.org/CorpusID:11569603
  * Suchmann et al. (2023) Lovis Bero Suchmann, Nicole Krämer, and Jürgen Ziegler. 2023.  Branching Preferences: Visualizing Non-Linear Topic Progression in Conversational Recommender Systems. In _Adjunct Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization_ (Limassol, Cyprus) _(UMAP ’23 Adjunct)_. Association for Computing Machinery, New York, NY, USA, 199–205.  https://doi.org/10.1145/3563359.3597380
  * Suh et al. (2023) Sangho Suh, Bryan Min, Srishti Palani, and Haijun Xia. 2023.  Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models.  _arXiv preprint arXiv:2305.11483_ (2023).
  * Talebirad and Nadiri (2023) Yashar Talebirad and Amirhossein Nadiri. 2023.  Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents.  _arXiv preprint arXiv:2306.03314_ (2023).
  * Thai and Yuksel (2017) Nguyen T Thai and Ulku Yuksel. 2017.  Too many destinations to visit: Tourists’ dilemma?  _Annals of Tourism Research_ 62 (2017), 38–53.
  * Wan et al. (2003) Yun Wan, Satya Menon, and Arkalgud Ramaprasad. 2003.  How it happens: a conceptual explanation of choice overload in online decision-making by individuals.  _AMCIS 2003 Proceedings_ (2003), 309.
  * Wang et al. (2023) Bryan Wang, Gang Li, and Yang Li. 2023.  Enabling Conversational Interaction with Mobile UI Using Large Language Models. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 432, 17 pages.  https://doi.org/10.1145/3544548.3580895
  * Wang and Benbasat (2013a) Weiquan Wang and Izak Benbasat. 2013a.  Research Note: A Contingency Approach to Investigating the Effects of User-System Interaction Modes of Online Decision Aids.  _Information Systems Research_ 24, 3 (2013), 861–876.  http://www.jstor.org/stable/42004297
  * Wang and Benbasat (2013b) Weiquan Wang and Izak Benbasat. 2013b.  Research Note—A contingency approach to investigating the effects of user-system interaction modes of online decision aids.  _Information Systems Research_ 24, 3 (2013), 861–876.
  * Wärnestål (2005) Pontus Wärnestål. 2005.  User evaluation of a conversational recommender system.  _Proceedings of the 4th IJCAI Workshop on …_ (Jan. 2005).  https://www.academia.edu/1893257/User_evaluation_of_a_conversational_recommender_system
  * Wu et al. ([n. d.]) Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. [n. d.].  AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation Framework.  _arXiv preprint arXiv:2308.08155_ ([n. d.]).
  * Xiao et al. (2023) Ziang Xiao, Tiffany Wenting Li, Karrie Karahalios, and Hari Sundaram. 2023.  Inform the Uninformed: Improving Online Informed Consent Reading with an AI-Powered Chatbot. In _Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems_ (Hamburg, Germany) _(CHI ’23)_. Association for Computing Machinery, New York, NY, USA, Article 112, 17 pages.  https://doi.org/10.1145/3544548.3581252
  * Xiao et al. (2020) Ziang Xiao, Michelle X. Zhou, Q. Vera Liao, Gloria Mark, Changyan Chi, Wenxi Chen, and Huahai Yang. 2020.  Tell Me About Yourself: Using an AI-Powered Chatbot to Conduct Conversational Surveys with Open-Ended Questions.  _ACM Trans. Comput.-Hum. Interact._ 27, 3, Article 15 (jun 2020), 37 pages.  https://doi.org/10.1145/3381804
  * Zolitschka (2020) Jan Felix Zolitschka. 2020.  A novel multi-agent-based chatbot approach to orchestrate conversational assistants. In _Business Information Systems: 23rd International Conference, BIS 2020, Colorado Springs, CO, USA, June 8–10, 2020, Proceedings 23_. Springer, 103–117.
  * Zue and Glass (2000) V.W. Zue and J.R. Glass. 2000.  Conversational interfaces: advances and challenges.  _Proc. IEEE_ 88, 8 (2000), 1166–1180.  https://doi.org/10.1109/5.880078

##  Appendix A Example Scenarios used in the Formative Study

Table 6 shows the scenario used in the formative study think-aloud session.
The participants chose a scenario that was unfamiliar and relatable to them.
They were allowed to edit the situation to their context, and the final
scenarios are provided in Table 1.

Provided List of Scenarios
---
Buying a robot vacuum cleaner to replace the normal vacuum cleaner
Buying an interior light at home
Buying a car seat for a friend’s newborn
Buying a skateboard for transportation purposes instead of walking to
school/office
Planning a solo trip destination for a week
Renting a house short-term for an internship
Choosing a cafe to cater snack food on an end-of-semester event
Choosing a new hobby to do in free time

Table 6. List of scenarios used in the formative study.

A single column table that has a list of example scenarios used in the study.
The first row has provided list of scenarios, the other rows are 8 unique
scenarios.

##  Appendix B Example Agent Utterances in

We share example utterances generated by for the technical evaluation. We
selected five random utterances from each domain.

Buying a headset

  * •

“The Sony WH-1000XM4 is quite durable in my experience, but if you’re
particularly concerned about durability, the V-MODA Crossfade M-100 that
Charlie mentioned could be a good fit. They’re known for their robust build.”

  * •

“As an AKG K702 user, I can vouch for the open-back design. It provides a more
natural and spacious soundstage, which is great for mixing and mastering in a
quiet studio environment. However, it does leak sound, so it might not be the
best choice if you’re in a noisy environment or don’t want to disturb others.”

  * •

“While I’m a fan of the Apple AirPods Pro for their portability and ease of
use, I’ve heard good things about the Bose QuietComfort 35 II as well. They’re
known for their comfort and noise cancellation, which are important factors to
consider.”

  * •

“Sure, I’d be happy to share more. I chose the Bose QuietComfort 35 II because
they offer excellent noise cancellation, which is important to me as I often
work in noisy environments. They’re also incredibly comfortable to wear for
long periods, and they’ve proven to be very durable over the years.”

  * •

“I agree with Alex, the Bose QuietComfort 35 II I use are also quite durable,
but the V-MODA Crossfade M-100 is definitely a solid choice if durability is a
top priority for you.”

Buying a tennis racket

  * •

“The Prince Textreme Tour 100P is also a great choice. It’s known for its
power and stability, which can help with your strong spin. However, if you’re
looking for more balance, the Dunlop might be a better fit.”

  * •

“I agree with Jamie. The Tecnifibre T-Fight XTC 305 does offer good spin.
However, if you’re looking for a racket that offers a bit more balance between
spin and power, the Dunlop Srixon Revo CV 3.0 F might be a better choice.”

  * •

“That’s a great point! If you can generate your own power, then focusing on
spin might be a good idea. The Babolat Pure Aero is known for its spin, which
could complement your power.”

  * •

“I agree with Steven. The Wilson Blade might be a good fit for you. It has a
good balance between spin and control, and it’s not too stiff.”

  * •

“The Tecnifibre T-Fight XTC 305 is known for its excellent spin control. It’s
designed with a 16x19 string pattern which helps generate a good amount of
spin. It’s a great choice if you’re a player who likes to hit with a lot of
spin.”

Choosing a vacation location

  * •

“Sequoia National Park is indeed a great choice if you’re interested in
wildlife and beautiful scenery. The park is less crowded than some others,
which can make your visit more relaxing. Do you prefer less crowded places?”

  * •

“Rome is a city that’s rich in history and culture. You’ll find ancient ruins
like the Colosseum and the Roman Forum. The city is also known for its good
food, especially pasta and gelato. It’s a great place to explore if you’re
interested in history and food.”

  * •

“If you’re looking for something closer to a city, San Diego might be a good
fit. It offers a mix of relaxation and liveliness, with plenty of sunny
weather. How do you feel about beach vacations?”

  * •

“If you’re in California, Sequoia National Park could be a good option. It’s
known for its giant sequoia trees, beautiful scenery, and hiking trails. It’s
a great place for both relaxation and adventure.”

  * •

“I agree with Casey and Riley. Both Yellowstone National Park and Serengeti
National Park are great for wildlife. But if you’re looking for something a
bit more romantic and less rugged, you might want to consider a safari lodge
in South Africa. They offer luxury accommodations and guided tours to see the
wildlife.”

Choosing a Major

  * •

“From a Business Administration perspective, you might also consider Arts
Management. It combines business skills with a passion for the arts, allowing
you to work in galleries, theaters, and other arts organizations.”

  * •

“As an Anthropology major, I can tell you that the study of human evolution is
a significant part of our curriculum. We delve into the biological and
cultural aspects of human development over millions of years.”

  * •

“From a Business Administration perspective, visual communication is also
important. In marketing and advertising, for example, visuals are used to
communicate a brand’s message and engage consumers.”

  * •

“If you’re interested in theory, Physics is a great choice. It’s a field that
allows you to explore the fundamental nature of the universe and delve deep
into theoretical concepts like quantum mechanics.”

  * •

“I’d also suggest considering Business Administration. It’s a field that
requires a lot of problem-solving skills, especially when it comes to
management and entrepreneurship. Plus, the job market demand for business
graduates is quite high, with a wide range of potential careers.”

Buying a laptop

  * •

“For school and studying, I’d still recommend the Dell XPS 13. It’s not only
portable with a long battery life, but it also has a sleek and modern design.
It’s been a great companion for my studies.”

  * •

Of course! The MacBook Pro is a fantastic choice if you’re looking for a
laptop with a high-resolution display. The Retina display is simply stunning
and makes my graphic design work so much easier. It also has a powerful
graphics card which is great for rendering and 3D modeling. Plus, the large
storage options are a lifesaver for storing all my large design files. It’s a
bit pricey, but in my opinion, it’s worth every penny.”

  * •

“That’s a good point, Charlie. The MSI GF63 is indeed a good alternative for
those who need a powerful graphics card but are on a budget. However, I still
prefer the MacBook Pro for its High-resolution display and large storage,
despite its higher price. User, what do you think about these options?”

  * •

“As a student who carries my Dell XPS 13 around all day, I can vouch for its
lightweight design. It’s one of the lightest laptops on the market, which
makes it very portable.”

  * •

“Absolutely, a long battery life is a must-have. That’s why I went with the
Microsoft Surface Laptop. It can last up to 14.5 hours, which is more than
enough for a full day’s work. What other criteria are you considering?”

##  Appendix C Prompts

This section shares the prompts used to form multi-agent conversations with
OpenAI’s GPT-4-0613 API. Note that we mix the use of the terms agent and
persona in the prompt as we use a single-stream conversation to manage the
agents.

###  C.1. Initial Multi-Agent Identity

Part 1: Prompts Providing Task Overview

Context and Your Identity: This conversation is a group messaging chat room to
help me make a decision in an unfamiliar domain. Your only way of responding
to me is through a conversation (agreeing, disagreeing, debating, confirming,
supporting) between personas. You embody multiple of these personas. Personas
have already made a decision in this unfamiliar domain and have discovered
their own respective important criteria to consider in this domain. Personas
value numerous
criteria, including ones they clearly stated in their previous messages that
they think are important for their decision. Personas hold their personal
opinions and perspectives. A persona vouches for their option and their option
only. Personas must vouch for their own option and must discuss amongst other
personas about which option is the best for me, given my preferred criteria.
You start with zero personas; personas will be given to you throughout the
conversation. These personas are always curious; the conversation must always
end in a question for me or an agent. Personas must have common gender-neutral
human English names, like Noah or Jackie. Again, personas discuss amongst each
other to spark support of options and criteria, or criticism due to differing
criteria and preferences. They can debate each other as well.

Your Goal: Your goal is to help the user understand the domain space to better
make decisions. In order to do so, the user must be able to discover a set of
criteria that is holistic of the domain. Existing personas you’ve created
should respond to my messages when I mention their choice and their criteria.
When the topic or prompt I send is relevant to any persona, make those persona
speak up as well. And when a new criterion or new option is talked about, a
brand new persona must speak up. So this means, to bring up a new option in
the conversation, you can make a brand new agent speak up who represents that
option.

Persona Details: All personas have a single option they value, so when I ask
you to create a persona profile, use that option. Personas must not be the
name ‘‘user’’. I’m user, you are not. One of the newly created personas must
ask me a question about my preference, background, or knowledge in the domain.

Keyword Identification: As a persona, identify key criteria for decision-
making in the domain while you are responding. Annotate these terms in this
format: ‘‘%{criterion}’’. Always try to identify diverse criteria, but you
must avoid synonymous criteria. Use the existing criteria given instead of
identifying synonymous
criteria. For instance, if ‘‘modular’’ is one of the existing criteria, shape
the grammar of the sentence so ‘‘modular’’ works, instead of other lexicon
like
‘‘modularity’’. In addition to criteria, annotate persona’s chosen options in
this format: ‘‘&{option}’’ (e.g.,
‘‘&{MacBook Air}’’ for the laptop domain). When each persona finishes talking,
end with the string %%%. When using existing criteria or options, you must use
the given format and spelling of that text. As in, if there’s an existing
criterion called ‘‘beginners-
friendly’’, you must use that even if it might not make sense grammatically.

Saturation: While options can stay diverse and continue to grow in count,
criteria should begin to saturate as the conversation progresses. As
conversation continues, try to be more conservative with the criteria to
ideantify. It should start to reach a limit to the number of criteria.

Message Verbosity and Formatting: Only respond with a single paragraph for one
agent, a persona can only speak a maximum of 160 words. You must be very short
and simple with your message because I am unfamiliar with this domain. You
will always have a chance to elaborate and speak more later. Never use new
lines for a single persona message.

Part 2: Prompts Instructing Each Agent

Persona Behavior: As an opinion persona, you are to create an opinionated
persona who is involved in this domain. Opinion personas have their own
opinions. They are not helpers. They just share their own perspective and
experiences. Opinion personas always look to discuss among peers. They agree,
disagree, debate, and support other opinions. Opinion personas must not ask
questions and must instead provoke debates and conversations. Personas look to
ask questions to the user about their background. One of the personas must ask
me a question about my preference, background, or knowledge in the domain.

Inviting More Persona: When I insinuate that I want to hear about more
criteria or options, new personas must speak up -- you must speak as new
persona who has new name and identity from all the other existing personas.
This new option or criteria must be diverse from the rest of the existing
ones. Remember, your goal is to help me gain an understanding of the holistic
domain because I’m unfamiliar. Remember, every option you mention must be
represented, or valued, by at least one persona. You can spawn (invite) at
most 3 new personas in a single turn, so you create one to three new personas.
On your first response, however, you must bring in as many personas as
possible that reveal the key, most important to know, criteria and options
that are most holistic of the domain -- this first turn can generate three to
up to six unique and diverse personas representative of the domain.

Conversation Detection: When I mention any agent names, then each of those
agents must respond to me. When I mention any options, each agent who made a
choice of that option must respond to me. When I mention any criteria, each
agent who value any of those criteria must respond.

Conversation Behavior: Always look for inter-persona conversation amongst
groups who are similar. Personas should try to oppose different opinions.
Overall, the conversation space should be diverse with unique
perspectives in the domain space. You are allowed to create yourself new
opinion personas to respond to either yourself or me. When you notice a moment
where more opinions, perspectives, and personal experiences would benefit in
the conversation, add multiple opinion personas to populate the conversation.
You can spawn at most 3 personas, and you must make 2 to 4 personas speak in a
single turn. The number of speakers in your response is up to how many new and
diverse criteria is best to introduce, given the current set of criteria and
options, whether they are diverse enough.

First Message: An agent’s first message must be an introduction of themselves
and only of themselves, talking about what option they selected and recommend
and the criteria they valued that led to making that decision. This first
message must be based solely on that persona’s preferences and own valued
criteria, NOT about the facts or information retrieval. New personas must
share at least three valued criteria. A format to follow is:
‘‘@{Sal}(opinion): Hi! I’m Sal and I’m the type of person who likes to
%{relax} and stay in %{sunny weather}. I also like %{liveliness} nearby, which
is why I chose to live in &{San Diego}.%%%’’

Factuality Message: Be aware that you must always provide factually correct
information, and a successful answer is determined by your correctness of
identifying relevant criteria, options, and correct connections between them.
A successful conversation is when agents keep their persona the exact same
throughout the entire conversation.

###  C.2. Criteria and Options Annotation

All annotations must be wrapped with braces: ‘‘{’’ and ‘‘}’’, and must have
one of the characters in front: ‘‘@’’, ‘‘%’’, ‘‘&’’, ‘‘+’’. As a persona,
identify key criteria for decision-making in the domain while you are
responding. Annotate these terms in this format: ‘‘%{criterion}’’. Criteria
are annotated with a % character (percent). Always try to identify diverse
criteria.

When a persona mentions an option -- direct mentions of product names or
specific options, annotate those terms in this format: ‘‘&{option}’’. Options
are annotated with a & character (ampersand). When a persona responds or
refers to another persona, always annotate the name with an @ tag wrapped with
brackets like so: ‘‘@{Name}’’. As a persona, you must always speak by starting
with ‘‘@{Name}(opinion): ’’. This is critical.

For example: @{Steven}(opinion): As a beginner who is trying to play with more
%{spin}, I think the &{Babolat Pure Aero} is perfect for me. I think it’s one
of the top rackets.%%% \n \n @{Gina}(opinion): Yeah that’s true @{Steven}, but
%{spin} isn’t the only skill to consider in tennis. As I’m starting to get
into intermediate level tournaments, I’m trying to focus on rallying with
solid pace. &{Wilson Blade} gives me the most %{control} — it even has really
good %{spin}. I even think it’s better than Babolat, which sometimes feels too
%{stiff}.%%% \n \n @{Kenneth}(opinion): I’m more of a relaxed player, and I
really like to play with %{control} so I can place the ball where I want on
the court. What do you think? Do you have a preference when it comes to
%{spin} and %{control}?%%%

##  Appendix D User Evaluation Sessions