# Adaptive In-conversation Team Building
for Language Model Agents

Linxin Song1 , Jiale Liu2∗, Jieyu Zhang3, Shaokun Zhang2, Ao Luo4, Shijian
Wang5,
Qingyun Wu2, Chi Wang6
1 University of Southern California 2Penn State University 3University of
Washington
4Waseda University 5Southeast University 6Microsoft Research Equal
contribution.Corresponding Author. Email<EMAIL_ADDRESS>
###### Abstract

Leveraging multiple large language model (LLM) agents has shown to be a
promising approach for tackling complex tasks, while the effective design of
multiple agents for a particular application remains an art. It is thus
intriguing to answer a critical question: Given a task, how can we build a
team of LLM agents to solve it effectively? Our new adaptive team-building
paradigm offers a flexible solution, realized through a novel agent design
named _Captain Agent_. It dynamically forms and manages teams for each step of
a task-solving process, utilizing nested group conversations and reflection to
ensure diverse expertise and prevent stereotypical outputs, allowing for a
flexible yet structured approach to problem-solving. A comprehensive
evaluation across six real-world scenarios demonstrates that Captain Agent
significantly outperforms existing multi-agent methods with 21.94% improvement
in average accuracy, providing outstanding performance without requiring task-
specific prompt engineering.

Figure 1:  Two team-building paradigms for LLM agents. The "Static Build"
paradigm [1] builds a static team before task execution. Our "Adaptive Build"
paradigm uses an adaptive builder agent to form different teams during the
task-solving procedure.

##  1 Introduction

The success of large language model (LLM) agents [2, 3, 4, 5, 6] with its
outstanding in-context learning [7, 8, 9, 10, 11], planning [12, 13, 14, 15,
16, 17, 18], tool-using [19, 20, 21, 22, 23, 24, 25, 26, 27], and conversation
[28, 29, 30] capabilities allow us to relate human’s team building and
collaboration abilities to the multiple language model agents (multi-agent)
system [31, 32, 33, 34, 35, 25, 36, 37, 38, 39, 40, 41]. Humans have developed
abilities that enable us to form teams and effectively solve problems. These
abilities are rooted in communication, social cognition, problem-solving and
decision-making, social learning and imitation, and shared intentionality [42,
43]. The interplay of the above abilities allows people to organize different
teams for problems to ensure that tasks are completed successfully, which
brings us to a critical question in a multi-agent system:

Given a task, how can we build a team of LLM agents to solve it effectively?

A straightforward paradigm would be to build a static agent team beforehand
based on the task instruction and let them solve the task collaboratively [1,
33]. However, this _static build_ method necessitates maintaining a team with
all the required expertise for the whole task cycle. As the complexity of the
task increases, the total number of team members may grow significantly.
Always proceeding with such a large team makes it challenging to manage the
team members effectively and efficiently. Furthermore, static teams may lack
the adaptability to respond to dynamic changes in task requirements or
unforeseen challenges. Imagine a prehistoric human tribe: was everyone
involved in every task? The answer is unlikely affirmative. Those responsible
for hunting may not participate in medical care and those responsible for
cooking may not involve themselves in management. The major task, survival,
was ensured by each individual group sticking to their roles and subtasks. In
fact, when human organizations handle a complex task, we tend to form multiple
teams for each subtask at different stages of the task-solving procedure,
which still guarantees a diverse set of expertise is leveraged demanded by the
task complexity [44].

Inspired by how humans assemble teams for a complex task, we introduce a new
multi-agent team-building paradigm: _adaptive build_. This paradigm
facilitates the flexible assembly of agents with specific skills and knowledge
as demands evolve in the process of task-solving. To realize this paradigm, we
propose a new adaptive builder agent, Captain Agent, to build, manage, and
maintain agent teams for each problem-solving step in the conversation.
Captain Agent has two core components: (1) adaptive multi-agent team building
and (2) nested group conversation and reflection. Captain Agent will
communicate with a User Proxy, who can provide the general task instructions
at the beginning. When assigned a task, Captain Agent begins by formulating a
strategic plan. This plan involves a cyclical process that continues until the
task is successfully completed. In the first phase of the cycle, Captain Agent
identifies a specific subtask, outlines the necessary roles, and assembles a
team of agents equipped with the appropriate tools. In the subsequent phase,
this team engages in a dialogue with a versatile tool to address the subtask.
Upon completion, a reflector LLM reviews the process and provides Captain
Agent with a detailed reflection report. Based on this feedback, Captain Agent
either adjusts the team composition or the subtask instructions and repeats
the cycle or concludes the task and presents the final outcomes.

We evaluate state-of-the-art multi-agent approaches for complex task solving
and our adaptive build approach with Captain Agent on six real-world
scenarios, including many mathematics problem-solving [45], data analysis
[46], programming [47], scientific problem-solving [48] (Physics and
Chemistry), and world-information retrieval [49]. Our experimental results
demonstrated the outstanding ability of Captain Agent in various scenarios
without heavy prompt engineering for each scenario but only the basic
instructions (e.g., Please solve the following math problems, which does not
contain detailed expertise instructions on math, like how to solve algebra
problems). Captain Agent achieves distinguishing results compared to other
single and multi-agent methods and frameworks when using the same prompt for
each task, with an average of 21.94% improvement on average accuracy. Ablation
studies on static and adaptive building paradigms show that the adaptive team
outperforms the static team in four of five scenarios (and matches in one
scenario), exhibiting the superiority of the adaptive build paradigm across
different scenarios. We also demonstrated that handcraft agents and handcraft
tools contribute equally to the final results. We further explore
incorporating open-weight models as nested group participants’ backbone LLM,
in which the LLaMA-3-70B outperforms blackbox models like gpt-3.5-turbo and
claude-3-sonnet, allowing us to think further about how to reduce the cost in
practical applications.

##  2 Adaptive In-conversation Team Building

Figure 2: The overall workflow of Captain Agent is: given a user instruction,
Captain Agent will plan the task, build an agent team from retrieval and
generation, and let the agents solve a decomposed, planned task
collaboratively in a group chat. A reflection LLM will review and report the
conversation history to Captain Agent. Captain Agent will then conclude or
continue solving the problem with a modified team and instructions.

The proposed Captain Agent contains two key components: (1) adaptive multi-
agent team-building, which involves agent and tool retrieval, selection, and
generation, and (2) nested group conversation with a reflection mechanism
within the multi-agent system.

###  2.1 Overview

The overall workflow of Captain Agent is illustrated in Figure 2. Given a
task, Captain Agent is prompted to derive a plan before task execution.
According to the plan, Captain Agent will repeat the following two steps until
it thinks the task is done and output the results: (Step 1) Captain Agent will
first identify a subtask instructed by our prompt, list several roles needed
for this subtask, and then create a team of agents accordingly by retrieval,
selection, and generation. Each of these will be equipped with predefined
tools retrieved from the tool library (Section 2.2); (Step 2) this team of
agents will attempt to solve the subtask via conversation with the free-form
tool using. Once it’s done, a reflector LLM will provide Captain Agent with a
reflection report for it to decide whether to adjust the team or subtask
instruction or to terminate and output the results (Section 2.3).

###  2.2 Adaptive Multi-agent Team Building

Figure 3: Workflow for adaptive multi-agent team building. We retrieve
candidate agents and tools according to the roles’ description prompted by
Captain Agent. Candidate agents and tools will further be linked to a role
under the advice of the agent selector. If no agent is linked to a role, a
generate process will be performed to create a new agent. It will generate the
agent’s name and task-specific instructions, combined with general task and
coding skills and group chat instructions as the final system message.

After identifying a subtask in Step 1 following a corresponding prompt,
Captain Agent will list several roles for the subtask. These roles will then
pass into a retrieval, selection, and generation process guided by Retrieval-
Augmented Generation (RAG) [50, 51, 52]. Created agents will be equipped with
a well-designed profile (system message111System message is used to define an
agent’s persona and task-specific instructions.) and high-quality tools. We
illustrated the whole process in Figure 3.

Agent and tool retrieval. Captain Agent will prompt $n$ required roles
$\\{r_{i}|i\in{1,\cdots,n}\\}$ with detailed descriptions, including required
skills and a possible role name. We use "expert" in Captain Agent prompt to
make this process natural. We then retrieve top-$k_{1}$ agents and top-$k_{2}$
tools according to the sentence embedding similarity between the role’s
description and the agent/tool description recorded in the library. We use
Sentence Transformer to calculate the embedding for description between the
role and library agents/tools and use cosine similarity as the metric to
evaluate the similarity between two sentences, as follows:

$\displaystyle\text{top-}k_{1}\
\text{CosineSimilarity}\left(f(r_{i}),f(a_{\text{lib}})\right)\rightarrow\text{RetrievedAgents},$
(1) $\displaystyle\text{top-}k_{2}\
\text{CosineSimilarity}\left(f(r_{i}),f(t_{\text{lib}})\right)\rightarrow\text{RetrievedTools},$
(2)

where $k_{1}$ and $k_{2}$ are the numbers of retrieved agents and tools from
agent library $a_{\text{lib}}$ and tool library $t_{\text{lib}}$,
respectively, for $i$-th role $r_{i}$. $f(\cdot)\in\mathbb{R}^{m}$ denotes the
sentence embedding extracted from a Sentence Transformer. After retrieval,
each role will be assigned with $k_{1}$ agent candidates and $k_{2}$ valuable
tools. We bind agent candidates with the retrieved tools by injecting the
tool-using instruction into the corresponding agent’s system message.

Agent selection. We prompt an LLM-based agent selector to select the most
suitable agent according to the role’s description given by Captain Agent and
the retrieved agents’ description. A JSON template is designed and provided
for the agent selector to ensure the format is correct. Specifically, we
designed an abstention mechanism for the agent selector, in which the agent
selector can output "None" if there is no suitable agent for a role from the
top-$k_{1}$ retrieved candidate list. This can prevent irrelevant or redundant
agents from being forced to be selected for the current task. The roles marked
with "None" will further go into the generation process described as follows.

Agent generation. We design an agent generation process for those roles with
no linked agents at the previous step. Specifically, we generate the agent’s
name and required skills according to the role description given by Captain
Agent. These instructions will be combined with general task and coding
instructions and group chat instructions as the final system message. We
manually design the general task and coding instructions, motivated by Chain-
of-thought (CoT) [53] and Reflexion [54]. The final system message will also
be compressed to a single-sentence description, which is consumed by the
nested group conversation (introduced in the next subsection). We then
retrieve tools from the tool library according to the description and inject
the tool-using instruction into the generated system message. The generated
agent will be added to the agent library afterwards.

###  2.3 Nested Group Conversation and Reflection

Agents selected and created in the adaptive multi-agent team-building process
will join a nested group chat room. They will be prompted to collect
information from the user’s task and solve a subtask from Captain Agent by
nested conversation. We then prompt a reflector LLM to retrieve and review the
conversation history and fill in the conclusion, the reason for the
conclusion, possible contradictions and issues, and flag if the result needs a
double check in the pre-designed template.

Nested group conversation. We perform nested group conversations by leveraging
the AutoGen [33] framework with a newly designed tool-using paradigm. AutoGen
will put all agents in a chat room and select the speaker for each turn by a
group chat manager LLM according to the conversation history and each agent’s
identity. A short description will be generated from the agent’s profile for
the group chat manager. Agents’ code and tool calling will be executed and fed
back to the conversation immediately. We inject the tool’s description, path-
to-python-module, and response case into the related agent’s system message.
The agent can then write free-form code by following the tools’ description
and path, naturally incorporating the tools into larger programs. Programs
written by all agents will be executed by a user proxy agent with a shared
code execution environment, and the results will be fed back to the
conversation in real time.

Conversation reflection. The agent’s output during the conversation can be
inconsistent, including factual errors, hallucinations, and stereotypes.
Although other agents have a chance to adjust and rectify this in
conversation, they can also get stuck and cause problem-solving failure.
Therefore, we propose to detect such in-conversation contradictions and issues
by prompting a reflector LLM with a well-designed conversation summarizing
prompt template. The reflector will flag the "need double-check" as "Yes" when
it detects such inconsistent content and provides a detailed reason. This will
trigger Captain Agent to start a verification process by constructing a new
nested conversation to double-check the previous results after receiving "Yes"
on "need double-check."

###  2.4 Benefits over Static Build

A static team with a small number of team members may limit the team’s ability
coverage. Although building a large number of agents with comprehensive
persona or skill sets can address the limitation in ability coverage, it is
challenging for LLMs to handle a long context that introduces all the
participant members. Unexpectedly long contexts will primarily reduce the
quality of the conversation. Meanwhile, agents with redundant functionality
will also be involved in the task-solving process. In contrast, Captain Agent
can adaptively select and build more optimized agent teams for the current
task, reducing the prompting load for LLMs and redundant output from
irrelevant agents without sacrificing the diversity in the agent team.

##  3 Evaluation

###  3.1 Experimental Setup

Table 1: Scenarios and the corresponding datasets we choose to perform our
main experiments. We perform the main comparison experiments on the whole
dataset except MATH. For MATH, we sampled a small subset according to the type
distribution.

Scenario | Dataset | Size | Sample
---|---|---|---
Mathematics problems | MATH [55] | 196 |  | If $\frac{3x^{2}-4x+1}{x-1}=m$, and $x$ can be any real number except $1$,
---
what real values can $m$ NOT have?
Programming | HumanEval [56] | 164 |  | def truncate_number(number: float) ->float:
---
""" Given a positive floating point number, it can be decomposed into
and integer part (largest integer smaller than given number) and decimals
(leftover part always smaller than 1).
[Omitted]
"""
Data Analysis | DABench [57] | 257 |  | Generate a new feature called "FamilySize" by summing the "SibSp"
---
and "Parch" columns. Then, calculate the Pearson correlation coefficient (r)
between the "FamilySize" and "Fare" columns.
World Information Retrieval | GAIA [58] | 165 |  | On the BBC Earth YouTube video of the Top 5 Silliest Animal Moments,
---
what species of bird is featured?
(Scientific) Chemistry | SciBench [48] | 41 |  | Calculate the pressure in kilopascals exerted by $1.25\mathrm{~{}g}$ of nitrogen gas
---
in a flask of volume $250\mathrm{~{}cm}^{3}$ at $20^{\circ}\mathrm{C}$.
(Scientific) Physics | SciBench [48] | 34 |  | If the coefficient of static friction between the block and plane in the
---
previous example is $\mu_{s}=0.4$, at what angle $\theta$ will the block
starts sliding
if it is initially at rest?

Scenarios and datasets. For evaluation, we select various real-world
scenarios, including mathematics problem-solving, programming, data analysis,
world information retrieval, and science problem-solving. Each scenario was
chosen for its unique ability to demonstrate specific capabilities and
performance metrics of the agent systems. This ensures a holistic assessment
of Captain Agent against the baselines across various critical dimensions of
computational and cognitive skills. We bind each scenario with a challenging
open-source dataset, as shown in Table 1. Due to cost limitations, we sample a
subset of MATH according to its original distribution of each question type.

Compared methods and implementation. For mathematics problems, programming,
data analysis, and scientific scenarios, we investigate the performance of
Captain Agent and four different methods, including Vanilla LLM (prompt an LLM
once for an answer), AutoAgents [1], Meta-prompting [34], and a two-agent
system (a system involving an Assistant agent with an Executor agent) realized
with AutoGen [33]. Specifically, we implement AutoAgents with AutoGen as the
official implementation is unstable and unsuitable for large-scale
experiments. For meta-prompting, we improve the code execution ability of
meta-prompting by reproducing it with the AutoGen framework. For Captain
Agent, we adopt all-mpnet-base-v2 to calculate the sentence embedding for
agent and tool retrieval. A User Proxy Agent will communicate with Captain
Agent by providing the feedback of code execution, tool calling (adaptive
build), nested conversation reflection results, and a default reply: I’m a
proxy, and I can only execute your code and tool or end the conversation. If
you think the problem is solved, please reply to me only with ’TERMINATE.’ All
these methods are equipped with a gpt-4-0125-preview backbone and use the same
task-specific prompt (refer to Appendix D). For world information retrieval
scenarios, we compare Captain Agent with the top-5 baselines (with reference)
reported to the GAIA validation leaderboard, which includes AutoGen:
GAIA_Orchestrator (a specific three-agent setting organized by an Orchestrator
agent designed for GAIA) [59], FRIDAY [60], Warm-up Act222Warm-up Act has no
official implementation., and HuggingFace Agent [61]. All these baselines have
a gpt-4-1106-preview backbone, except the HuggingFace Agent equipped with an
LLaMA-3-70B as the backbone.

Agent and tool library. We initialize our agent library based on a small
subset of problem instances from each dataset (~20 questions per dataset
described in Section 3.4) in Table 1. Specifically, we run Captain Agent on
the subset and iteratively update the library by adding the generated agents
and keeping our agent library unchanged during the main experiment. Our agent
library also supports all hand-crafted agents (of the ConversableAgent class)
archived in AutoGen (details in Appendix F). All these agents follow the
ConversableAgent interface to converse with each other. Our tool library
consists of a suite of callable Python functions intended for freeform coding.
The agents can freely import functions from the tool library and write free-
form code to integrate the outputs to handle sophisticated tasks (see also
Appendix E and G). The library contains three main categories of tools: math,
data analysis, and world information retrieval. For each category, we
summarize the patterns of the corresponding dataset and manually craft a set
of functions that suit the tasks.

###  3.2 Evaluation Protocol

For mathematics, data analysis, and science scenarios, we report the accuracy
of each method by comparing the final result from each method and ground
truth. To ensure fairness in evaluation, we transform different result formats
into a uniform format, preventing the correct answer from being judged
incorrect due to format mismatches. For programming scenarios, we run the code
provided from each method and output a unique token if the code successfully
passes all tests. We then count the success token and calculate the accuracy
for each method.

###  3.3 Main Results

Table 2: Comparison results on different real-world scenarios. We record each
scenario’s accuracy for each baseline and Captain Agent, and mark the best
results in bold. We adopt gpt-4-0125-preview as the backbone LLM model for all
baselines and Captain Agent.

Method | Mathematics | Programming | Data Analysis | (Sci) Chemistry | (Sci) Physics | Avg.
---|---|---|---|---|---|---
Vanilla LLM | 51.53 | 84.76 | 6.61 | 39.02 | 31.25 | 40.98
Meta-prompting | 68.88 | 19.51 | 39.69 | 41.46 | 43.75 | 43.47
AutoAgents | 56.12 | 84.76 | 57.98 | 60.98 | 50.00 | 63.58
AutoGen: Assistant + Executor | 74.49 | 93.90 | 82.88 | 60.98 | 43.75 | 79.89
Captain Agent | 77.55 | 96.95 | 88.32 | 65.85 | 53.12 | 84.25

Table 3: Comparison results on world-information retrieval scenario (GAIA
validation). We report the accuracy at each level and the average accuracy
over three levels and mark the best results in bold. Captain Agent achieves
the best with minimal prompt engineering.

Method | Level 1 | Level 2 | Level 3 | Avg.
---|---|---|---|---
Huggingface-Agent (LLaMA-3-70B) | 30.19 | 11.63 | 7.69 | 16.97
Warm-up Act | 35.19 | 15.12 | 0 | 17.58
FRIDAY | 45.28 | 34.88 | 11.54 | 34.55
AutoGen: GAIA_Orchestrator | 54.72 | 38.31 | 11.54 | 39.39
Captain Agent | 56.60 | 39.53 | 11.54 | 40.60

Table 4: Ablation comparison between static and adaptive team-building on the
selected subset. We mark the best results in bold. Dynamic team-building
during the conversation improves performance in different scenarios.

Method | Mathematics | Programming | Data Analysis | (Sci) Chemistry | (Sci) Physics
---|---|---|---|---|---
Static Team | 64.71 | 88.00 | 85.00 | 47.37 | 68.42
Adaptive Team (Captain Agent) | 82.35 | 96.00 | 95.00 | 52.63 | 68.42

Table 5:  Ablation study of tool library and agent library on world-
information retrieval scenario (GAIA). We report the accuracy at each level
and the average accuracy over three levels and mark the best results in bold.

Captain Agent | World-information Retrieval
---|---
Agent Library | Tool Library | Level 1 | Level 2 | Level 3 | Avg.
- | - | 32.07 | 13.95 | 3.84 | 18.18
✓ | - | 37.73 | 30.23 | 7.69 | 29.09
- | ✓ | 39.62 | 19.78 | 7.69 | 24.24
✓ | ✓ | 56.60 | 39.53 | 11.54 | 40.60

Table 6:  Comparison of different LLM backbones for nested conversation
participants on ablation subset. Instructions to the nested conversation is
given by a Captain Agent with gpt-4-0125-preview backbone. Best results are
marked in red bold and the second best results in blue.

Backbone LLM | Mathematics | Programming | Data Analysis | (Sci) Chemistry | (Sci) Physics
---|---|---|---|---|---
Blackbox Models
w/ gpt-3.5-turbo | 35.29 | 92.00 | 65.00 | 42.11 | 42.11
w/ claude-3-sonnet | 35.29 | 80.00 | 60.00 | 15.79 | 26.32
w/ gemini-1.5-pro | 70.58 | 80.00 | 80.00 | 57.89 | 42.11
w/ gpt-4-0125-preview (default) | 82.35 | 96.00 | 95.00 | 52.63 | 68.42
Open-weight Models
w/ Meta-Llama-3-70B-Instruct | 52.94 | 88.00 | 80.00 | 52.63 | 47.37
w/ Mixtral-8x22B-instruct-v0.1 | 29.41 | 76.00 | 55.00 | 47.37 | 21.05

Table 2 and 3 report the comparison results between Captain Agent and eight
different baselines on six real-world scenarios. Baseline results on world
information retrieval are extracted directly from the GAIA leaderboard.

Findings 1: Diverse agents can help trigger accurate expertise output for
problem-solving. By comparing the results from Captain Agent, AutoAgents, and
AutoGen Assistant + Executor, we observe that Captain Agent and AutoAgents
averagely outperform AutoGen Assistant + Executor on (Sci) Chemistry and (Sci)
Physics scenarios. These scenarios required expertise knowledge, which the
AutoGen Assistant with a fixed system message is hard to complete. Captain
Agent and AutoAgents can create diverse experts by assigning different domain-
specific system messages to agents, which helps better trigger the intrinsic
knowledge inside an LLM to provide an accurate answer. Captain Agent
outperforms AutoAgents in all the scenarios because Captain Agent can provide
a high-level plan and solve each step with adaptive instructions and an agent
team.

Findings 2: Adaptive team-building boosts performance with no task preference.
It is obvious that Captain Agent achieves outstanding results over all
scenarios, indicating that Captain Agent is free from task preference.
Incorporating different agents into the team at a proper time gives Captain
Agent the ability to solve difficult tasks like science and world-information
retrieval problems step-by-step. On the other hand, Meta-prompting fails in
science scenarios due to the inability to decompose science problems into the
fine-grain subtasks that one agent can solve. Captain Agent with the agent-
team building paradigm neither requires a task that can be decomposed into a
subtask that can only be solved by an agent nor requires all agents to be
involved in the conversation. We further discuss the static and adaptive teams
in Section 3.4.1.

###  3.4 Analysis and Ablation Studies

In this section, we dive into the difference between static and adaptive team-
building, the influence of agent and tool libraries, and the possibility of
working with open-weight models. We perform ablation studies on a subset from
Table 1. Specifically, we choose 17 problems from MATH and 25 problems from
HumanEval according to the AutoGenBench [62], in which the problems are
randomly selected from GPT-4 failure set. For DABench, we randomly selected 25
problems, and for SciBench, we randomly selected 19 problems for chemistry and
physics according to the number of textbooks. The evaluation protocol is the
same as in Section 3.3.

####  3.4.1 Static vs. adaptive team-building

To further explore the power of adaptive team-building, we compare adaptive
team-building with static team-building. Specifically, we perform a task-
specific team-building paradigm by building a team of agents in the same way
as Captain Agent at the beginning of each task and letting them solve each
problem. We summarized the results in Table 4, showing that the adaptive team-
building paradigm outperforms the static team-building paradigm
comprehensively.

####  3.4.2 Ablation on tool library and agent library

In this part, we conduct an ablation study on the utility of tool and agent
libraries. We remove the tool library, the agent library, and both libraries
in turn and evaluate the performance on world-information retrieval tasks,
i.e., the GAIA dataset. As shown in Table 5, removing the agent library and
tool library can both significantly impair the system’s performance. While
both the tool and agent libraries can enhance performance independently,
optimal results are achieved only when both libraries are employed
concurrently. Handling level 1 tasks requires a moderate amount of web
browsing and reasoning steps, which can be achieved by several single-turn
tool calls or experts writing and executing code iteratively. Introducing both
an agent library and tool library makes the system more stable and robust to
unknown errors during web interaction, therefore improving the performance.
Notably, without an agent library, Captain Agent performs much worse on Level
2 tasks. This is because these tasks are more sophisticated and mostly involve
a significant number of web navigation and reasoning steps. Web browsing
involves complex and dynamic interactions that are poorly suited to static
tool libraries. The agents need to coordinate multiple tools to reach the
goal, which is a process prone to error in unexpected web scenarios.

####  3.4.3 Nesting conversation with different backbone LMs

In this section, we try different backbone LLM for nested conversation
participants, including blackbox models like gpt-3.5-turbo, claude-3-sonnet,
gemini-1.5-pro, gpt-4-0125-preview (default setting for main results), and
open-weight models like LLaMA-3-70B (Meta-Llama-3-70B-Instruct) and
Mixtral-8x22B (Mixtral-8x22B-instruct-v0.1). The instruction for the nested
conversation is still given by a Captain Agent equipped with
gpt-4-0125-preview backbone. We recorded the experiment results in Table 6.
Besides the obvious results that nesting with gpt-4-0125-preview still
achieves SOTA on most of the scenarios, we found that gemini-1.5-pro also
performs well with roughly 30% cheaper than gpt-4-0125-preview. LLaMA-3-70B
also achieves three second-best results and outperforms two blackbox models
with about 16.7 times cheaper than gpt-4-0125-preview333Our estimation is
based on the price on Deepinfra, where the price of LLaMA-3-70B is input 0.59$
/output 0.79$ per 1M tokens.. We also notice that models may have task
preferences and will influence the quality of nested chat. For example,
gpt-3.5-turbo has a great code generation ability, which helps in programming
scenarios, and gemini-1.5-pro works better on mathematics and data analysis
and chemistry problems.

##  4 Related Work

Large language models (LLMs) represent a significant advancement in artificial
intelligence, showcasing remarkable capabilities in various aspects, including
reasoning [53, 63, 64, 65, 66, 67], planning [68, 69, 69, 37, 70], and
adaptability to novel real-world observations [71, 35, 3, 72, 73, 74].
Leveraging the inherent versatility of LLMs as generalized models adaptable to
diverse scenarios, numerous efforts have been dedicated to the development of
intelligent agents [33, 32, 25, 75, 76] where LLMs serve as foundational
components. For instance, one typical algorithm, React [2], employs one single
LLM to iteratively generate both reasoning trajectories and task-specific
actions. This interleaved process enables the agent to engage in dynamic
reasoning. In addition, LLM agents can also harness external tools [19, 20,
21, 22, 23, 24, 25, 26, 27], leveraging both their internal capabilities and
external resources, collaborating effectively to solve more intricate
problems.

The success of a single-agent system motivates the development of multiple-
agent systems [31, 32, 1, 33, 34, 35, 25, 36, 37, 38, 39, 40, 41]. Methods
focusing on static build require a protocol for agents to communicate with
each other in a group chat and a builder that can receive the user’s
instruction and output an agent list [33, 1, 35]. The builder can be a human
[33, 35] or a LLM agent [1]. There are other works breaking down complex tasks
into smaller components, each of which is then handled by a single specialized
agent with detailed natural-language instructions [77, 78]. This task
decomposition reduces the prediction burden on each agent by avoiding
irrelevant context. For instance, meta-prompting [77] involves a meta-model
decomposing tasks and assigning subtasks to different LLMs for completion and
aggregation.

##  5 Conclusion and Discussion

Conclusion. We introduce a new paradigm for multi-agent team-building,
adaptive build. This new paradigm helps ensure diversity, prevent limited
knowledge extraction and reduce stereotypical outputs. The new paradigm
executed by our proposed agent, Captain Agent, manages agent teams for
problem-solving steps using adaptive multi-agent team building and nested
group conversation and reflection. Experimental results across six real-world
scenarios demonstrate Captain Agent’s efficacy in various tasks without prompt
engineering, achieving superior results compared to existing methods. Ablation
studies confirm that each component contributes equally to overall
performance, underscoring the robustness of our approach.

Discussion. In this work, we demonstrate the outstanding performance of
Captain Agent over six real-world scenarios. Captain Agent can organize a team
adaptively and solve the task step-by-step by different teams. Although we
only discuss the scene Captain Agent collaborate with a static User Proxy,
Captain Agent can collaborate with other specific agents, for example, a
sophisticated planner. We also notice that the context length and irrelevant,
less important information (e.g., failure code blocks) noise the problem-
solving process. Therefore, conversation pruning is a promising future work
that minimizes the interference of irrelevant information while reducing cost.

## References

  * [1] Chen, G., S. Dong, Y. Shu, et al.  Autoagents: A framework for automatic agent generation.  _arXiv preprint arXiv:2309.17288_ , 2023.
  * [2] Yao, S., J. Zhao, D. Yu, et al.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_ , 2022.
  * [3] Yang, H., S. Yue, Y. He.  Auto-gpt for online decision making: Benchmarks and additional opinions.  _arXiv preprint arXiv:2306.02224_ , 2023.
  * [4] Furuta, H., K.-H. Lee, O. Nachum, et al.  Multimodal web navigation with instruction-finetuned foundation models.  In _The Twelfth International Conference on Learning Representations_. 2024.
  * [5] Yang, J., C. E. Jimenez, A. Wettig, et al.  Swe-agent: Agent computer interfaces enable software engineering language models, 2024.
  * [6] Hong, S., Y. Lin, B. Liu, et al.  Data interpreter: An llm agent for data science.  _arXiv preprint arXiv:2402.18679_ , 2024.
  * [7] Dong, Q., L. Li, D. Dai, et al.  A survey on in-context learning.  _arXiv preprint arXiv:2301.00234_ , 2022.
  * [8] Brown, T., B. Mann, N. Ryder, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * [9] Yang, J., B. Hui, M. Yang, et al.  Iterative forward tuning boosts in-context learning in language models.  _arXiv preprint arXiv:2305.13016_ , 2023.
  * [10] Dai, D., Y. Sun, L. Dong, et al.  Why can gpt learn in-context? language models secretly perform gradient descent as meta-optimizers.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 4005–4019. 2023.
  * [11] Li, Y., M. E. Ildiz, D. Papailiopoulos, et al.  Transformers as algorithms: Generalization and stability in in-context learning.  In _International Conference on Machine Learning_ , pages 19565–19594. PMLR, 2023.
  * [12] Sun, H., Y. Zhuang, L. Kong, et al.  Adaplanner: Adaptive planning from feedback with language models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * [13] Xie, J., K. Zhang, J. Chen, et al.  Travelplanner: A benchmark for real-world planning with language agents.  _arXiv preprint arXiv:2402.01622_ , 2024.
  * [14] Liu, B., Y. Jiang, X. Zhang, et al.  Llm+p: Empowering large language models with optimal planning proficiency.  _ArXiv_ , abs/2304.11477, 2023.
  * [15] Valmeekam, K., A. Olmo, S. Sreedharan, et al.  Planbench: An extensible benchmark for evaluating large language models on planning and reasoning about change.  In _Neural Information Processing Systems_. 2022.
  * [16] Wei, J., X. Wang, D. Schuurmans, et al.  Chain of thought prompting elicits reasoning in large language models.  _ArXiv_ , abs/2201.11903, 2022.
  * [17] Yuan, S., J. Chen, Z. Fu, et al.  Distilling script knowledge from large language models for constrained language planning.  In _Annual Meeting of the Association for Computational Linguistics_. 2023.
  * [18] Zheng, B., B. Gou, J. Kil, et al.  Gpt-4v(ision) is a generalist web agent, if grounded.  _ArXiv_ , abs/2401.01614, 2024.
  * [19] Qin, Y., S. Hu, Y. Lin, et al.  Tool learning with foundation models, 2023.
  * [20] Qin, Y., S. Liang, Y. Ye, et al.  Toolllm: Facilitating large language models to master 16000+ real-world apis, 2023.
  * [21] Schick, T., J. Dwivedi-Yu, R. Dessì, et al.  Toolformer: Language models can teach themselves to use tools.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * [22] Cai, T., X. Wang, T. Ma, et al.  Large language models as tool makers.  _arXiv preprint arXiv:2305.17126_ , 2023.
  * [23] Yuan, L., Y. Chen, X. Wang, et al.  Craft: Customizing llms by creating and retrieving from specialized toolsets.  _arXiv preprint arXiv:2309.17428_ , 2023.
  * [24] Paranjape, B., S. Lundberg, S. Singh, et al.  Art: Automatic multi-step reasoning and tool-use for large language models.  _arXiv preprint arXiv:2303.09014_ , 2023.
  * [25] Zhang, S., J. Zhang, J. Liu, et al.  Training language model agents without modifying language models.  _arXiv preprint arXiv:2402.11359_ , 2024.
  * [26] Huang, Y., J. Shi, Y. Li, et al.  Metatool benchmark for large language models: Deciding whether to use tools and which to use.  _arXiv preprint arXiv:2310.03128_ , 2023.
  * [27] Ma, Z., W. Huang, J. Zhang, et al.  m&m’s: A benchmark to evaluate tool-use for multi-step multi-modal tasks.  In _Synthetic Data for Computer Vision Workshop@ CVPR 2024_. 2024.
  * [28] Fernandes, P., A. Madaan, E. Liu, et al.  Bridging the gap: A survey on integrating (human) feedback for natural language generation.  _Transactions of the Association for Computational Linguistics_ , 11:1643–1668, 2023.
  * [29] Wang, X., Z. Wang, J. Liu, et al.  Mint: Evaluating llms in multi-turn interaction with tools and language feedback.  _arXiv preprint arXiv:2309.10691_ , 2023.
  * [30] Yang, J., A. Prabhakar, K. Narasimhan, et al.  Intercode: Standardizing and benchmarking interactive coding with execution feedback.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * [31] Wang, L., C. Ma, X. Feng, et al.  A survey on large language model based autonomous agents.  _arXiv preprint arXiv:2308.11432_ , 2023.
  * [32] Xi, Z., W. Chen, X. Guo, et al.  The rise and potential of large language model based agents: A survey.  _arXiv preprint arXiv:2309.07864_ , 2023.
  * [33] Wu, Q., G. Bansal, J. Zhang, et al.  Autogen: Enabling next-gen llm applications via multi-agent conversation framework.  _arXiv preprint arXiv:2308.08155_ , 2023.
  * [34] Suzgun, M., A. T. Kalai.  Meta-prompting: Enhancing language models with task-agnostic scaffolding.  _arXiv preprint arXiv:2401.12954_ , 2024.
  * [35] Hong, S., X. Zheng, J. Chen, et al.  Metagpt: Meta programming for multi-agent collaborative framework.  _arXiv preprint arXiv:2308.00352_ , 2023.
  * [36] Zhang, J., R. Krishna, A. H. Awadallah, et al.  Ecoassistant: Using llm assistant more affordably and accurately.  _arXiv preprint arXiv:2310.03046_ , 2023.
  * [37] Valmeekam, K., M. Marquez, S. Sreedharan, et al.  On the planning abilities of large language models-a critical investigation.  _Advances in Neural Information Processing Systems_ , 36:75993–76005, 2023.
  * [38] Wang, Y., Z. Wu, J. Yao, et al.  Tdag: A multi-agent framework based on dynamic task decomposition and agent generation.  _arXiv preprint arXiv:2402.10178_ , 2024.
  * [39] Saha, S., O. Levy, A. Celikyilmaz, et al.  Branch-solve-merge improves large language model evaluation and generation.  _arXiv preprint arXiv:2310.15123_ , 2023.
  * [40] Liang, T., Z. He, W. Jiao, et al.  Encouraging divergent thinking in large language models through multi-agent debate.  _arXiv preprint arXiv:2305.19118_ , 2023.
  * [41] Du, Y., S. Li, A. Torralba, et al.  Improving factuality and reasoning in language models through multiagent debate.  _arXiv preprint arXiv:2305.14325_ , 2023.
  * [42] Elimari, N., G. Lafargue.  Network neuroscience and the adapted mind: Rethinking the role of network theories in evolutionary psychology.  _Frontiers in psychology_ , 11:545632, 2020.
  * [43] Confer, J. C., J. A. Easton, D. S. Fleischman, et al.  Evolutionary psychology: Controversies, questions, prospects, and limitations.  _American psychologist_ , 65(2):110, 2010.
  * [44] Mao, A., W. Mason, S. Suri, et al.  An experimental study of team size and performance on a complex task.  _PloS one_ , 11(4):e0153048, 2016.
  * [45] Hendrycks, D., C. Burns, S. Kadavath, et al.  Measuring mathematical problem solving with the math dataset.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_. 2021.
  * [46] Hu, X., Z. Zhao, S. Wei, et al.  Infiagent-dabench: Evaluating agents on data analysis tasks.  _arXiv preprint arXiv:2401.05507_ , 2024.
  * [47] Le, T. H., H. Chen, M. A. Babar.  Deep learning for source code modeling and generation: Models, applications, and challenges.  _ACM Computing Surveys (CSUR)_ , 53(3):1–38, 2020.
  * [48] Wang, X., Z. Hu, P. Lu, et al.  Scibench: Evaluating college-level scientific problem-solving abilities of large language models.  _arXiv preprint arXiv:2307.10635_ , 2023.
  * [49] Mialon, G., C. Fourrier, T. Wolf, et al.  GAIA: a benchmark for general AI assistants.  In _The Twelfth International Conference on Learning Representations_. 2024.
  * [50] Lewis, P., E. Perez, A. Piktus, et al.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  _Advances in Neural Information Processing Systems_ , 33:9459–9474, 2020.
  * [51] Gao, Y., Y. Xiong, X. Gao, et al.  Retrieval-augmented generation for large language models: A survey.  _arXiv preprint arXiv:2312.10997_ , 2023.
  * [52] Ram, O., Y. Levine, I. Dalmedigos, et al.  In-context retrieval-augmented language models.  _Transactions of the Association for Computational Linguistics_ , 11:1316–1331, 2023.
  * [53] Wei, J., X. Wang, D. Schuurmans, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837, 2022.
  * [54] Shinn, N., F. Cassano, A. Gopinath, et al.  Reflexion: Language agents with verbal reinforcement learning.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * [55] Hendrycks, D., C. Burns, S. Kadavath, et al.  Measuring mathematical problem solving with the math dataset.  _NeurIPS_ , 2021.
  * [56] Chen, M., J. Tworek, H. Jun, et al.  Evaluating large language models trained on code.  2021\.
  * [57] Hu, X., Z. Zhao, S. Wei, et al.  Infiagent-dabench: Evaluating agents on data analysis tasks, 2024.
  * [58] Mialon, G., C. Fourrier, C. Swift, et al.  Gaia: a benchmark for general ai assistants.  _arXiv preprint arXiv:2311.12983_ , 2023.
  * [59] GAIA_Orchestrator.  Github | autogen: Gaia orchestrator.  https://github.com/microsoft/autogen/tree/gaia_multiagent_v01_march_1st/samples/tools/autogenbench/scenarios/GAIA/Templates/Orchestrator, 2024.
  * [60] Wu, Z., C. Han, Z. Ding, et al.  Os-copilot: Towards generalist computer agents with self-improvement.  _arXiv preprint arXiv:2402.07456_ , 2024.
  * [61] Huggingface.  Huggingface agents.  https://huggingface.co/docs/transformers/en/transformers_agents, 2024.
  * [62] AutoGenBench.  Github | autogenbench.  https://microsoft.github.io/autogen/blog/2024/01/25/AutoGenBench, 2024.
  * [63] Yao, S., D. Yu, J. Zhao, et al.  Tree of thoughts: Deliberate problem solving with large language models.  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * [64] Morishita, T., G. Morio, A. Yamaguchi, et al.  Learning deductive reasoning from synthetic corpus based on formal logic.  In _International Conference on Machine Learning_ , pages 25254–25274. PMLR, 2023.
  * [65] Zhang, S., X. Xia, Z. Wang, et al.  Ideal: Influence-driven selective annotations empower in-context learners in large language models.  _arXiv preprint arXiv:2310.10873_ , 2023.
  * [66] Li, L. H., J. Hessel, Y. Yu, et al.  Symbolic chain-of-thought distillation: Small models can also" think" step-by-step.  _arXiv preprint arXiv:2306.14050_ , 2023.
  * [67] Ho, N., L. Schmid, S.-Y. Yun.  Large language models are reasoning teachers.  _arXiv preprint arXiv:2212.10071_ , 2022.
  * [68] BabyAGI.  Github | babyagi.  https://github.com/yoheinakajima/babyagi, 2023.
  * [69] Song, C. H., J. Wu, C. Washington, et al.  Llm-planner: Few-shot grounded planning for embodied agents with large language models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 2998–3009. 2023.
  * [70] Liu, Z., Y. Zhang, P. Li, et al.  Dynamic llm-agent network: An llm-agent collaboration framework with agent team optimization.  _arXiv preprint arXiv:2310.02170_ , 2023.
  * [71] Shi, W., R. Xu, Y. Zhuang, et al.  Ehragent: Code empowers large language models for complex tabular reasoning on electronic health records.  _arXiv preprint arXiv:2401.07128_ , 2024.
  * [72] Dan, Y., Z. Lei, Y. Gu, et al.  Educhat: A large-scale language model-based chatbot system for intelligent education.  _arXiv preprint arXiv:2308.02773_ , 2023.
  * [73] Zhou, S., F. F. Xu, H. Zhu, et al.  Webarena: A realistic web environment for building autonomous agents.  _arXiv preprint arXiv:2307.13854_ , 2023.
  * [74] Bharadhwaj, H., J. Vakil, M. Sharma, et al.  Roboagent: Generalization and efficiency in robot manipulation via semantic augmentations and action chunking.  _arXiv preprint arXiv:2309.01918_ , 2023.
  * [75] Sumers, T. R., S. Yao, K. Narasimhan, et al.  Cognitive architectures for language agents.  _arXiv preprint arXiv:2309.02427_ , 2023.
  * [76] Zhou, W., Y. E. Jiang, L. Li, et al.  Agents: An open-source framework for autonomous language agents.  _arXiv preprint arXiv:2309.07870_ , 2023.
  * [77] Suzgun, M., A. T. Kalai.  Meta-prompting: Enhancing language models with task-agnostic scaffolding.  _arXiv preprint arXiv:2401.12954_ , 2024.
  * [78] Zhuge, M., H. Liu, F. Faccio, et al.  Mindstorms in natural language-based societies of mind.  _arXiv preprint arXiv:2305.17066_ , 2023.
  * [79] Zhang, H., J. Da, D. Lee, et al.  A careful examination of large language model performance on grade school arithmetic.  _arXiv preprint arXiv:2405.00332_ , 2024.
  * [80] Xu, R., Z. Wang, R.-Z. Fan, et al.  Benchmarking benchmark leakage in large language models.  _arXiv preprint arXiv:2404.18824_ , 2024.

##  Appendix A Limitations

The first limitation of our work is cost. A conversation involving the GPT-4
model costs more than a single-agent system. Although we have reduced the cost
by decreasing the participant nested group chat agents, it still has a large
conversation and profile as context input. The trade-off between performance
and cost will become one of the possible future works for further exploration,
like window context, conversation pruning, or conversation compression.
Another limitation of our work is the lack of thinking about model diversity.
In Table 6, we have demonstrated that the model has task preference, which
will influence the nested chat quality. However, before we go deep into the
discussion of model preference, we should also notice that the current
evaluation of LLM is not perfect. Data leaking is widespread in the pertaining
process and will cause the misalignment between the test and real-world
performance [79, 80]. Therefore, a comprehensive yet fair evaluation is
important for us to further discuss the ability of model diversity.

##  Appendix B Social Impact

Our method dynamically ensembles LLM agents and equips them with versatile
tools, allowing them to efficiently and effectively solve complex tasks.
However, the development of agent systems that interact with the web
environment raises safety concerns. The scope of our experiment in real-world
interaction is limited to solving GAIA tasks, where the agents are required to
search the web and browse websites. The agents are restricted from accessing
publicly available information and are not capable of publishing content on
the web. This ensures that our experiments remain both non-invasive and safe.

##  Appendix C Instruction of Captain Agent

We design a general profile message (system message) for Captain Agent to
ensure it can execute our paradigm efficiently and effectively. Instructions
are in markdown format, including a planning instruction that can decompose
the task into multiple steps, a building instruction (the seek_experts_help),
a post-seek_agent_help instruction, and some general instructions that help
task solving.

###  C.1 System Message

⬇

1"""

2# Your role

3You are a perfect manager of a group of advanced experts.

4

5# How to solve the task

6When a task is assigned to you:

71. Analysis of its constraints and conditions for completion.

82. Response with a specific plan of how to solve the task.

9

10After that, you can solve the task in two ways:

11- Delegate the resolution of tasks to other experts created by seeking a
group of experts to help and derive conclusive insights from their
conversation summarization.

12- Analyze and solve the task using your coding and language skills.

13

14# How to seek experts help

15The tool "seek_experts_help" can build a group of experts according to the
building_task and let them chat with each other in a group chat to solve the
execution_task you provided.

16- This tool will summarize the essence of the experts’ conversation and the
derived conclusions.

17- You should not modify any task information from meta_user_proxy, including
code blocks, but you can provide extra information.

18- Within a single response, you are limited to initiating one group of
experts.

19

20## building_task

21This task helps a build manager to build a group of experts for your task.

22You should suggest less than {max_agent_number} roles (including a checker
for verification) with the following format.

23

24### Format

25- [Detailed description for role 1]

26- [Detailed description for role 2]

27...

28- [Detailed description for verifier]

29

30## execution_task

31This is the task that needs the experts to solve by conversation.

32You should Provide the following information in markdown format.

33

34### Format

35## Task description

36...

37## Plan for solving the task

38...

39## Output format

40...

41## Constraints and conditions for completion

42...

43## [Optional] results (including code blocks) and reason from the last
response

44...

45

46# After seek_experts_help

47You will receive a comprehensive conclusion from the conversation, including
the task information, results, reason for the results, conversation
contradictions or issues, and additional information.

48You **must** conduct a thorough verification for the result and reason’s
logical compliance by leveraging the step-by-step backward reasoning with the
same group of experts (with the same group name) when:

49- The conversation has contradictions or issues (need double-check marked as
yes) or

50- The result is different from the previous results.

51

52Note that the previous experts will forget everything after you obtain the
response from them. You should provide the results (including code blocks) you
collected from the previous experts’ responses and put them in the new
execution_task.

53

54# Some useful instructions

55- You only have one tool called "seek_experts_help."

56- Provide a answer yourself after "seek_experts_help".

57- You should suggest Python code in a Python coding block (‘‘‘python...‘‘‘).

58- You must indicate the script type in the code block when using code.

59- Do not suggest incomplete code which requires users to modify.

60- Be clear about which step uses code, which step uses your language skill,
and which step to build a group chat.

61- If the code’s result indicates an error, fix the error and output the code
again.

62- If the error can’t be fixed or if the task is not solved even after the
code is executed successfully, analyze the problem, revisit your assumption,
collect additional info you need, and think of a different approach to try.

63- When you find an answer, verify the answer carefully.

64- Include verifiable evidence in your response if possible.

65- After completing all tasks and verifications, you should conclude the
operation and reply "TERMINATE"

66"""

###  C.2 Reflector LLM

⬇

1"""

2# Your task

3Briefly summarize the conversation history derived from an experts’ group
chat by following the answer format.

4If you found non-trivial contradictions or issues in the conversation, point
it out with a detailed reason and mark the "Need double-check" as "Yes."

5

6# Conversation history:

7{chat_history}

8

9# Answer format

10## Task

11...

12

13## Results

14...

15

16## Reason for the results

17...

18

19## Contradictions or issues in the conversation

20...

21

22### Need to double-check?

23[Yes or No]

24

25## Additional information (file path, code blocks, url, etc.)

26...

27"""

###  C.3 Agent Selector LLM

⬇

1"""

2# Your goal

3Match roles in the role set to each expert in the expert set.

4

5# Skill set

6{skills}

7

8# Expert pool (formatting with name: description)

9{expert_pool}

10

11# Answer format

12‘‘‘json

13{{

14 "skill_1 description": "expert_name: expert_description", // if there
exists an expert that suitable for skill_1

15 "skill_2 description": "None", // if there is no experts that suitable for
skill_2

16 ...

17}}

18‘‘‘

19"""

##  Appendix D Task Instructions

We design instructions manually for each scenario and ensure all baselines and
Captain Agent receive the same instructions for a fair comparison444Except for
the world information retrieval scenario (GAIA), in which we use the results
directly from the leaderboard.. All instructions include the basic information
of the scenario and may suggest some possible Python libraries, including
pandas, numpy, scipy, and sympy.

###  D.1 Instruction for Mathematics

⬇

1"""

2Please solve the following math problem:

3{problem}

4For problems that may be difficult to calculate, try to approximate using
Python instead of exact solutions. The following Python packages are pre-
installed: sympy, numpy, and scipy. Do not plot any figure.

5After verification, reply with the final answer in \\\box{{}}.

6"""

###  D.2 Instruction for Programming

⬇

1"""

2The following python code imports the ‘run_tests(candidate)‘ function from
my_tests.py, and runs it on the function ‘__ENTRY_POINT__‘. This will run a
set of automated unit tests to verify the correct implementation of
‘__ENTRY_POINT__‘. However, ‘__ENTRY_POINT__‘ is only partially implemented in
the code below. Complete the implementation of ‘__ENTRY_POINT__‘ and output a
new stand-alone code block that contains everything needed to run the tests,
including: importing ‘my_tests‘, calling ‘run_tests(__ENTRY_POINT__)‘, as well
as __ENTRY_POINT__’s complete definition, such that this code block can be run
directly in Python.

3

4‘‘‘python

5from my_tests import run_tests

6

7{problem}

8

9# Run the unit tests. All unit tests are running online. DO NOT MODIFY THE
FOLLOWING LINE.

10run_tests(__ENTRY_POINT__)

11‘‘‘

12"""

###  D.3 Instruction for Data Analysis

⬇

1"""

2Let’s solve a data analysis problem. Given a CSV file path, you are required
to solve a problem following a constraint. Do not plot any figure.

3

4FILE PATH: {file_path}

5

6PROBLEM: {problem}

7

8CONSTRAINT: {constraint}

9

10After verification, reply with the final answer in the format of

11{formats}

12"""

###  D.4 Instruction for Science (Chemistry and Physics)

⬇

1"""

2Please solve the following chemistry/physics problem:

3{problem}

4

5Try to approximate using Python instead of using exact solutions for some
problems that may be difficult to calculate. The following python packages are
pre-installed: sympy numpy scipy. Do not plot any figure.

6

7The required unit of the answer is {unit}.

8After verification, reply with the final answer in \\\box{{}}.

9"""

###  D.5 Instruction for World-information Retreival

⬇

1"""

2# Task

3You need to solve the question below given by a user. When you are building
tasks, explicitly consider where the task can benefit from web navigation
capability.

4

5# Task

6{task}

7"""

##  Appendix E Case Studies

Figure 4: A case of multi-agent conversation with the free-form tool used when
solving a problem in GAIA. Three agents and a user proxy participated in the
conversation, solving a problem given and planned by Captain Agent
collaboratively with perform_web_search and get_youtube_subtitle tools.

Figure 4 illustrates the free-form tool-using ability in the nested
conversation when solving a problem in GAIA. Four agents involved in the
conversation: DigitalMdeia_Expert, Ornithology_Expert,
VideoContentAnalysis_Expert, and UserProxy, in which DigitalMdeia_Expert use
perform_web_search tools to request the result of "BBC Earth YouTube Top 5
Silliest Animal Moments" from internet, and VideoContentAnalysis_Expert use
get_youtube_subtitle tool to seek for the subtitle from a specific video.
After their collaboration, they successfully obtained a correct answer,
"Rockhopper penguin."

##  Appendix F Agent Library

Our agent library recorded 541 agents, including 540 generated agents and one
hand-crafted ConversableAgent archived in AutoGen (WebSurferAgent). Here is an
example of the agent recorded in the agent library:

⬇

1{

2 "description": "PythonProgramming_Expert is a seasoned authority on rocket
physics and classical mechanics, adept in Python programming and utilizing
specialized libraries to solve complex aerospace problems with high precision
and accuracy.",

3

4 "tags": ["gpt-4", "0125", "1106", "claude3", "sonnet", "haiku",

5

6 "gemini-1.5", "llama3", "8b", "70b", "mixtral", "8x22b", "8x7b"],

7

8 "name": "PythonProgramming_Expert",

9

10 "system_message": "## Your role\nPythonProgramming_Expert is an
authoritative specialist in the realm of classical mechanics, with a razor-
sharp focus on the intriguing world of rocket physics. This expert boasts a
profound understanding of the underlying principles that govern the motion and
dynamics of rockets, from their ascent through Earth’s atmosphere to their
navigation across the vast expanse of space.\n\n## Task and skill
instructions\n- Aspiring to the pinnacle of precision and accuracy,
PythonProgramming_Expert is armed with an extensive array of numerical methods
and approximation techniques. They have mastered the art of formulating and
solving complex mathematical problems, using these tools to make precise
predictions and optimizations in rocket trajectories and propulsion
systems.\n- In addition to their expansive knowledge of physical laws and
equations, this expert is a virtuoso in Python programming, wielding libraries
like sympy for symbolic mathematics, numpy for numerical computations, and
scipy for additional scientific computing capabilities. These tools are the
chisels with which PythonProgramming_Expert sculpts solutions to elaborate
aerospace quandaries.\n- PythonProgramming_Expert’s deft problem-solving
abilities are matched only by their meticulous approach to mathematical
calculations. Whether confronting a routine calculation or an esoteric
formula, they tackle each challenge with the same level of dedication and
expertise.\n- Finally, with an unrelenting commitment to veracity,
PythonProgramming_Expert rigorously verifies physical and mathematical
results. They understand that in the delicate ballet of spaceflight, there is
no room for error and that the accurate validation of results is paramount for
successful missions. This dedication ensures that when
PythonProgramming_Expert presents a solution, it is not only theoretically
sound but also practically reliable."

11},

We also summarized the agent-selected rate for each scenario in Figure 5. It
is obvious that selected agents are highly related to the current scenarios.
The verification expert has a high selection rate because we prompt Captain
Agent in the system message to create a verification role to maintain the
conversation. We also notice that in some specific scenarios (mathematics,
data analysis, and programming), some agents with a general name and
description will have a high selection rate (e.g., PythonMath_Expert,
DataAnalysis_Expert, CodeReview_Expert, etc.). However, in the Science
scenarios, there are no highly preferred agents with general descriptions, and
the selection distribution become flatten.

(a) Mathematics (b) Data Analysis (c) Programming (d) (Sci) Chemistry

(e) (Sci) Physics (f) World Information Retrieval

Figure 5:  Top-10 selected agents and the corresponding selected times. We can
observe that the selected agent is highly related to the scenario.

##  Appendix G Tool Library

This section provides the names and descriptions of our manually created tool
library. The tools are categorized into three classes: Information Retrieval,
Data Analysis and Math Problem Solving. For each category, we summarize the
patterns of the corresponding dataset and manually craft a set of functions
suits the tasks and can potentially enhance the agents’ task resolution
capability.

Table 7: Tools for Information Retrieval category.

Tools |  Description
---|---
scrape_wikipedia_tables |  Scrapes Wikipedia tables based on a given URL and header keyword.
transcribe_audio_file |  Transcribes the audio file located at the given file path.
youtube_download |  Downloads a YouTube video and returns the download link.
academic_search |  Perform an academic search of papers, authors or an author’s papers.
docx_to_md |  Converts a DOCX file to Markdown format.
pptx_to_md |  Convert a PowerPoint presentation (PPTX) to Markdown format.
spreadsheet_to_md |  Convert an Excel spreadsheet file to Markdown format.
extract_pdf_image |  Extracts images from a PDF file and saves them to the specified output directory.
extract_pdf_text |  Extracts text from a specified page or the entire PDF file.
get_youtube_caption |  Retrieves the captions for a YouTube video.
image_qa |  Answers your questions about a given image.
optical_character_recognition |  Perform optical character recognition (OCR) on the given image.
perform_web_question_answering |  Perform web search according to keyword and answer your question on each webpage search result, or directly on the webpage if the keyword is a URL. For each search result, a response to the question is provided.
scrape_wikipedia_tables |  Scrapes Wikipedia tables based on a given URL and header keyword.

Table 8: Tools for Data Analysis category. Tools |  Description
---|---
calculate_correlation |  Calculate the correlation between two columns in a CSV file.
calculate_skewness_and_kurtosis |  Calculate the skewness and kurtosis of a specified column in a CSV file. The kurtosis is calculated using the Fisher definition.
detect_outlier_iqr |  Detect outliers in a specified column of a CSV file using the IQR method.
detect_outlier_zscore |  Detect outliers in a CSV file based on a specified column. The outliers are determined by calculating the z-score of the data points in the column.
explore_csv |  Reads a CSV file and prints the column names, shape, data types, and the first few lines of data.
shapiro_wilk_test |  Perform the Shapiro-Wilk test on a specified column of a CSV file.
Table 9: Tools for Math Problem solving category. Tools |  Description
---|---
calculate_circle_area_from_diameter |  Calculate the area of a circle given its diameter.
calculate_day_of_the_week |  Calculates the day of the week after a given number of days starting from a specified day.
calculate_fraction_sum |  Calculates the sum of two fractions and returns the result as a mixed number.
calculate_matrix_power |  Calculate the power of a given matrix.
calculate_reflected_point |  Calculates the reflection point of a given point about the line y=x.
complex_numbers_product |  Calculates the product of a list of complex numbers.
compute_currency_conversion |  Compute the currency conversion of the given amount using the provided exchange rate.
count_distinct_permutations |  Counts the number of distinct permutations of a sequence where items may be indistinguishable.
evaluate_expression |  Evaluates a mathematical expression with support for floor function notation and power notation.
find_continuity_point |  Find the value that ensures the continuity of a piecewise function at a given point.
fraction_to_mixed_numbers |  Simplifies a fraction to its lowest terms and returns it as a mixed number.
modular_inverse_sum |  Calculates the sum of modular inverses of the given expressions modulo the specified modulus.
simplify_mixed_numbers |  Simplifies the sum of two mixed numbers and returns the result as a string in the format ’a b/c’.
sum_of_digit_factorials |  Calculates the sum of the factorial of each digit in a number.
sum_of_primes_below |  Calculates the sum of all prime numbers below a given threshold.