# Auto-GPT for Online Decision Making: Benchmarks and Additional Opinions

Hui Yang  AmazonUnited States<EMAIL_ADDRESS>,  Sifu Yue
AmazonUnited States<EMAIL_ADDRESS>and  Yunzhong He  Kaito.aiUnited
States<EMAIL_ADDRESS>
(2023)

###### Abstract.

Auto-GPT is an autonomous agent that leverages recent advancements in adapting
Large Language Models (LLMs) for decision-making tasks. While there has been a
growing interest in Auto-GPT stypled agents, questions remain regarding the
effectiveness and flexibility of Auto-GPT in solving real-world decision-
making tasks. Its limited capability for real-world engagement and the absence
of benchmarks contribute to these uncertainties. In this paper, we present a
comprehensive benchmark study of Auto-GPT styled agents in decision-making
tasks that simulate real-world scenarios. Our aim is to gain deeper insights
into this problem and understand the adaptability of GPT-based agents. We
compare the performance of popular LLMs such as GPT-4, GPT-3.5, Claude, and
Vicuna in Auto-GPT styled decision-making tasks. Furthermore, we introduce the
Additional Opinions algorithm, an easy and effective method that incorporates
supervised/imitation-based learners into the Auto-GPT scheme. This approach
enables lightweight supervised learning without requiring fine-tuning of the
foundational LLMs. We demonstrate through careful baseline comparisons and
ablation studies that the Additional Opinions algorithm significantly enhances
performance in online decision-making benchmarks, including WebShop and
ALFWorld.

††copyright: acmcopyright††journalyear: 2023††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Adapting Large Language Models (LLM) to autonomous agents have recently
achieved great success in various decision-making(Chowdhery et al., 2022),
virtual character simulation(Park et al., 2023) and tool manipulation
tasks(Liang et al., 2023). While there are evidence suggesting that scaling up
LLM can lead to certain level of general intelligence(Bubeck et al., 2023),
there are still limitations in leveraging LLM as autonomous agents directly.
The lack of long-term memory, limited token length, and the lack of
deterministic control over its behaviors, etc. are among them. Various
techniques in prompting(Wei et al., 2022), planning and memory retrieval(Wei
et al., 2022; Liang et al., 2023; Park et al., 2023) to overcome the
limitations are proposed recently and worked well. Among all of the agents,
there is Auto-GPT111https://github.com/Significant-Gravitas/Auto-GPT, a GPT-
based autonomous agent that connects to the internet and attempts to
accomplish any task. Despite its sudden surge in attention, the extent of its
effectiveness in accomplishing tasks remains uncertain due to its limited
capacity for action.

We characterize Auto-GPT styled agent with the following properties (1) it
receives only high-level goals and instructions at the beginning of complex
multi-step tasks, without requiring step-by-step guidance from humans (2) it
engages in self-monologue by generating ’Thoughts,’ ’Reasoning,’ ’Plan,’ and
’Criticism’ for each individual step of action (effectively, CoT and Reflexion
(Wei et al., 2022; Shinn et al., 2023)) (3) it possesses the capability to
integrate various tools through simple tool instructions and a few examples
(4) it incorporates long-term self-memory and memory retrieval mechanisms (5)
tasks-specific adaption should only require minimal efforts like providing
goal definitions and tool descriptions.

To gain deeper insights into the performance and limitations of Auto-GPT
styled agents, we conduct experiments by adapting Auto-GPT for online
decision-making tasks that involve responding to unknown external
environments. We evaluate various popular LLMs across multiple online learning
tasks and present our findings and insights. Additionally, we propose a novel
approach to demonstrate how external models can be leveraged as providers for
additional opinions. Despite the recent advancements in LLM research, such as
self-consistency and group voting techniques (Wang et al., 2023; Fu et al.,
2023), as well as incorporating external expert models and APIs to enhance
LLMs (Shen et al., 2023; Liang et al., 2023), it has not been reported that
LLMs like GPT-4 can benefit from additional opinions in the same way humans do
(Yaniv, 2004). This intriguing discovery opens up possibilities for a new
paradigm where smaller expert models can collaborate with LLMs.

In this work we offer the following contributions (1) to the best of our
knowledge, we are the first to show Auto-GPT can be easily adapted to online
decision-making tasks that closely resemble real-world scenarios. (2) We
provide comprehensive benchmark comparisons between popular LLMs, including
GPT-4 (OpenAI, 2023), GPT-3.5, Claude (Bai et al., 2022), and Vicuna (Peng et
al., 2022). We present our findings regarding the adaptation of these models
for autonomous agents. (3) We demonstrate that incorporating a second opinion
from a supervised learner can significantly enhance task performance,
providing a low-cost way of introducing supervision signals into Auto-GPT
styled autonomous agent without model fine-tuning.

##  2\. Methodology

###  2.1. Tasks and baseline models

####  2.1.1. WebShop

Webshop (Yao et al., 2022a) is a simulated environment that replicates a web
shopping experience by scraping 1,181,436 products from Amazon.com and hosting
them on an isolated server. The environment offers agents a realistic action
space, including options such as performing product searches, clicking on
items, navigating back to previous pages, and making purchases. Equipped with
an integrated search engine, the environment provides the shopping agent with
real-time observations that mimic those from a web browser. The evaluation
process involves determining whether the agent successfully purchases the
intended product based on its description, where a success requires all
matches on the product itself, attributes, options and price together. We use
the IL (Imitation Learning) method with a fine-tuned action policy component
as the baseline model, and compare it with popular generative LLMs with Auto-
GPT styled adaption towards this web shopping task.

####  2.1.2. ALFWorld

ALFWorld (Shridhar et al., 2021) is a ground-breaking research environment
that harmonizes the sophisticated, task-oriented language understanding of the
ALFRED (Shridhar et al., 2020) dataset with the immersive interactive fiction
of TextWorld (Côté et al., 2018). The ALFRED (Action Learning From Realistic
Environments and Directives) benchmark offers a robust testing ground for
models to learn to parse and carry out intricate tasks from language
directives within a detailed, interactive 3D environment. Meanwhile, TextWorld
serves as a dynamic learning playground for training and evaluating
reinforcement learning agents on text-based games. By interweaving these two
platforms, ALFWorld brings together the linguistic comprehension and decision-
making challenges of text-based games with the physical interactions in a 3D
environment, embodying a critical step towards melding natural language
instructions with real-world physical interactions. The environment contains
over 25,000 unique, procedurally-generated tasks across photorealistic
settings in various areas such as kitchens, living rooms, and bedrooms. These
tasks require complex problem-solving skills and a thorough understanding of
both language and environment, creating an elevated benchmark for AI
performance. As much as ALFWorld presents a challenging yet fertile testbed
for reinforcement learning, natural language understanding, and interactive
decision-making research, we also start the evaluation process with the DAgger
(Ross et al., 2011) IL (Imitation Learning) agent against the unseen dataset
as the baseline. Then we benchmark it against prevailing generative language
learning models that utilize an Auto-GPT style approach, with these models
only being adjusted for the ALFWorld task with tool demonstrations.

###  2.2. Prompt design

We adapt Auto-GPT for both tasks without extensive tuning, simply by providing
the task requirements or questions directly as Auto-GPT’s goal. For instance,
we input sentences such as ”I want to purchase a folding storage box that is
easy to install, made of faux leather, and has dimensions of 60x40x40cm”. To
facilitate Auto-GPT’s understanding of available actions, we represent each
action as a tool. It is worth noting that we observed poor performance when
using tool instructions without examples in a sermon-style manner. However,
with just a few examples, the performance improved significantly. Therefore,
we include one to three few-shot examples for tool demonstrations, to harness
the in-context learning abilities of LLMs.

###  2.3. Considering additional opinions

We further engineer changes to the Auto-GPT workflow to take additional
opinions from external expert models into consideration. Specifically, we
sample top k opinions from an expert model in Auto-GPT’s decision phase, and
present those opinions into the context section of the prompt towards more
informed decisions. Details of the modified Auto-GPT workflow is outlined in
Algorithm 1. In this work we simply use readily available IL models for both
tasks as the external expert. The prompt to suggest additional opinions to LLM
follows the template as ’Here’s one(a few) suggestion(s) for the command:
¡action with parameters¿ Please use this suggestion as a reference and make
your own judgement. ’

Figure 1. One step of Auto-GPT with Additional Opinions. Here Additional
Opinions are generated by other expert models - IL models here, but extensible
to any other models like Rule or other LLMs. Algorithm 1 Additional Opinion
Auto-GPT Algorithm.

0: $o_{i}$: additional opinion sampled from expert model. $P_{o}$($o_{i}$): a
prompt template wrapping top k $o_{i}$ as suggestions to LLM. $P_{h}$: the
regular prompt as a human to trigger LLM response. Add(x): Add x into Auto-GPT
context.

1: Initialize Auto-GPT

2: for each Auto-GPT step do

3: Add(Initial Goal and Instruction Prompt)

4: if sampled $o_{i}$ from expert models exists then

5: Add($P_{o}$($o_{i}$)) for i ¡ k

6: else

7: Add($P_{h}$)

8: end if

9: Auto-GPT runs with the prompt added.

10: end for

11: return result

##  3\. Experiments

###  3.1. Experimental setup

Table 1. Updated Webshop Model Performance Metrics Model | Success Rate | Reward | Precision | Purchase Rate
---|---|---|---|---
Base Models |  |  |  |
Rule | 0.060 | 44.589 | 0.060 | 1.000
IL w/o. Image | 0.213 | 56.056 | 0.213 | 1.000
IL | 0.227 | 57.689 | 0.227 | 1.000
Auto-GPT(Claude) Variants |  |  |  |
Auto-GPT(Claude) | 0.140 | 47.617 | 0.146 | 0.960
Auto-GPT(Claude) + IL | 0.240 | 48.600 | 0.270 | 0.890
Auto-GPT(Claude) + IL(top5) | 0.220 | 52.010 | 0.229 | 0.960
Auto-GPT(GPT3.5) Variants |  |  |  |
Auto-GPT(GPT3.5) | 0.120 | 43.833 | 0.140 | 0.860
Auto-GPT(GPT3.5) + IL | 0.200 | 47.717 | 0.241 | 0.830
Auto-GPT(GPT3.5) + IL(top5) | 0.230 | 52.827 | 0.279 | 0.820
AutoGPT(GPT3.5) + Random | 0.060 | 22.333 | 0.136 | 0.440
Auto-GPT(GPT4) Variants |  |  |  |
Auto-GPT(GPT4) | 0.240 | 46.133 | 0.353 | 0.680
Auto-GPT(GPT4) + IL | 0.300 | 56.233 | 0.361 | 0.830
Auto-GPT(GPT4) + IL(top5) | 0.320 | 61.550 | 0.372 | 0.860
Auto-GPT(Vicuna) |  |  |  |
Auto-GPT(Vicuna) | 0.000 | 0.000 | 0.000 | 0.000

####  3.1.1. Webshop

We utilized the original WebShop server setup from the GitHub Repository
provided in the original paper(Yao et al., 2022a). For testing, we adhered to
a fixed order of iteration, selecting the first 50 instructions. This limited
test set is a trade-off due to cost and computational efficiency concerns,
especially running GPT4. Imitation Learning (IL) models, both with and without
image embeddings, were used to ensure a fair comparison with large language
models, given that the latter lack image access. The ’additional opinions’
provided to Auto-GPT consistently used the superior IL model with image
access. The temperature was set to 0.01 across all models to reduce
randomness. Evaluation of IL models and Auto-GPT + IL variants followed a
rigorous protocol to mitigate sampling randomness and assess small observed
variations, respectively. In the case of Auto-GPT alone, we performed a single
run due to the minimal variations observed as a result of the low temperature
setting. Nonetheless, small variations were noticed in the Auto-GPT + IL
variants, prompting us to conduct two runs and use their average for analysis.

In the original research paper, a supervised Imitation Learning (IL) model was
trained to aid the agent in making optimal decisions at each stage, with the
ultimate goal of executing the correct purchase based on a given human
instruction. The system was structured around four principal tasks: (1)
generating a quality search query based on the original instruction, (2)
selecting an item to browse from a variety of items and their titles, (3)
deciding whether to check ’Description’, ’Features’ or ’Reviews’ in the
product detail page while also making the correct product option choices such
as size, color etc., and (4) finalizing the purchase.

Two IL models were utilized: a BART model for generating the most effective
search query (task type 1) and a BERT model to make the correct choices for
task type 2 to 4. Rule based model always directs to purcahse the first item
after search.

####  3.1.2. ALFWorld

Adopting a similar approach to the Webshop experiment, we leverage the
existing ALFWorld task simulation environment, including the unseen set of 134
games that was utilized for benchmarking in the original research paper. From
the original study, we specifically incorporate the task agent (BUTLER::BRAIN)
from the Imitation Learning (IL) model, excluding the visual input agent
(BUTLER::VISION). The training of the text agent is performed using the DAgger
(Dataset Aggregation) approach within an imitation learning context using
expert demonstrations. To manage task execution failures, Beam Search is
deployed to generate alternate action sentences, typically opting for the best
sequence of words greedily for efficiency. The application of Beam Search
primarily aims at enhancing the action sentence generation during failures
rather than optimizing over embodied interactions. Echoing the Webshop
experiment, to ensure a fair comparison with large language models (LLMs), we
furnish the ’additional opinions’ from the IL model to Auto-GPT. To control
randomness, we maintain a temperature setting of 0.01 for all LLMs to minimize
noise and strictly adhere to the original evaluation protocol to further
mitigate randomness.

Table 2. Updated ALFWorld Model Performance Metrics Model | Success Rate | Reward | Precision | Completion Rate
---|---|---|---|---
Base Models, average of three |  |  |  |
IL w/o. Beam Search | 0.179 | 24 | 1.000 | 0.179
IL | 0.306 | 41 | 1.000 | 0.441
Auto-GPT(Claude) Variants |  |  |  |
Auto-GPT(Claude) | 0.082 | 11 | 0.104 | 0.791
Auto-GPT(Claude) + IL | 0.090 | 12 | 0.130 | 0.687
Auto-GPT(GPT3.5) Variants |  |  |  |
Auto-GPT(GPT3.5) | 0.075 | 10 | 0.078 | 0.866
Auto-GPT(GPT3.5) + IL | 0.030 | 4 | 0.048 | 0.470
Auto-GPT(GPT4) Variants |  |  |  |
Auto-GPT(GPT4) | 0.485 | 65 | 0.628 | 0.582
Auto-GPT(GPT4) + IL | 0.515 | 69 | 0.789 | 0.530
Auto-GPT(Vicuna) |  |  |  |
Auto-GPT(Vicuna) | 0.000 | 0.000 | 0.000 | 0.000

###  3.2. Baseline comparison

####  3.2.1. Webshop

Table 1 illustrates the results of running the first 50 test cases across the
original IL models and Auto-GPT agent using different large language models
(LLMs). The original IL model, lacking image input, only achieved a modest
success rate, illustrating the complexity of the task. IL models incorporating
image input as embeddings performed more favorably. Auto-GPT agents utilizing
GPT3.5 or Claude alone performed worse than the original IL models, with or
without images. However, GPT4 by itself exhibited superior performance
compared to both IL models.

A noteworthy point is that IL models display better rewards than Auto-GPT
baselines without IL due to a higher purchase rate, and an allowance for more
steps (100 vs. 20). However, the reward metric may not necessarily serve as
the best end measurement, particularly considering real-world shopping
scenarios where an agent refraining from making a purchase could be preferable
to it making a purchase that doesn’t entirely meet the requirements. In cases
where a purchase is made, if we calculate precision solely based on these
instances, Auto-GPT (GPT4), with or without IL, demonstrates significantly
higher precision compared to any other variants. Further details on GPT4
considering the additional opinion can be found in Appendix 1.

####  3.2.2. ALFWorld

Table 2 presents the results of our ALFWorld experiment involving the
Imitation Learning (IL) model and Large Language Models (LLMs) in an AutoGPT
configuration, evaluated across an unseen set of 134 data points from
ALFWorld’s IL Model. Notably, the IL model with Beam Search significantly
outperformed the version without Beam Search, as indicated by a considerable
decrease in the success rate from 0.306 to 0.179 when Beam Search was omitted.

While Claude and GPT3.5 operating in the AutoGPT setting fell short of
surpassing the IL model, GPT4 markedly exceeded the IL model’s performance,
irrespective of the use of Beam Search, despite the disatvantage of steps
allowed (35 vs. 50). We hypothesize that the comparatively lower performance
of Claude and GPT3.5 can be attributed to their lack of full episode
demonstrations, and we posit that the introduction of more examples might
enhance their performance.

This is particularly pertinent in tasks where the description carries
implications, such as ”heat the mug and put the mug on the countertop,” which
presupposes the agent’s understanding of how to heat the mug. Both Claude and
GPT3.5 struggled with deriving such implicit knowledge. However, we anticipate
that supplementing the context with more examples would likely enhance their
performance in such complex tasks.

###  3.3. LLM comparison

####  3.3.1. Webshop

Across all LLMs, Claude and GPT3.5 alone (0.140 vs. 0.120 success rate)
compare similarly as each other in the AutoGPT setting. GPT4 alone performs
the best (0.24 success rate) and Vicuna is tested not able to generate
formatted responses thus reported 0 here. Another perspective to consider here
is that the speed of calling these LLM APIs - Claude is faster than GPT3.5 and
much faster than GPT4. We recommend here that Claude could be a great solution
considering the performance latency tradeoff for real world problems.

####  3.3.2. ALFWorld

In line with our observations from the Webshop experiment, the success rates
for Claude and GPT3.5, when applied in the AutoGPT setting, were relatively
low, success rate at 0.075 and 0.082, respectively. Among the models, GPT4
demonstrated superior performance, achieving the highest success rate of 0.485
and a precision as high as 0.628, surpassing all other models including the
Imitation Learning model. Despite Claude’s advantage in speed over GPT3.5 and
GPT4, its performance was markedly inferior to GPT4. Taking into account these
observations, we recommend the use of GPT4 given its performance supremacy
over the other models under consideration.

###  3.4. Additional opinions

####  3.4.1. Webshop

A novel paradigm emerged from our study, amalgamating Large Language Models
(LLMs) with Expert models. Rather than solely relying on Expert models and
their generated results, we propose an integrated approach. Firstly, the top k
Additional Opinions are sampled from Expert models. Subsequently, these
opinions are presented to the LLMs, prompting them to consider these views and
make the final decision. This methodology particularly resonates with GPT4.
Even though the underlying mechanism is still elusive and it is not yet clear
if it mimics human decision-making processes(Yaniv, 2004), the effectiveness
of this approach is tangible in the experimental outcomes. Our working
hypothesis postulates that GPT4 exhibits inherent biases when making
autonomous decisions. However, by introducing opinions from various weak
learners, GPT4 can enhance its performance. Considering these diverse
viewpoints may allow GPT4 to mitigate its own biases and overcome inherent
limitations.

Interestingly, the inclusion of a single IL choice as an additional opinion in
the context resulted in improved performance for all LLMs. This performance
boost is particularly noteworthy for GPT4, because GPT4 by itself outperforms
the additional opinions provided by IL models, while still benefiting from
this method.

To explore the impact of single vs. multiple additional opinions, we also
tested sampled top five additional opinions vs. one (see Table 1), and
observed GPT4 with top 5 additional opinions reached the best Success Rate,
Rewards and Precision across all groups (Table 1). From an intelligent agent
perspective, awareness of an additional opinion, or even multiple distinct
opinions, can be beneficial, as argued in (Yaniv, 2004). This suggests that
providing LLMs with one or a few additional opinions of reasonable quality can
serve as a reference, resulting in a more informed decision.

Out of curiosity, we also conducted one ablation study by providing random one
additional opinion to GPT3.5. We observed the worst performance - the lowest
reward (22.333) and an equivalently low success rate (0.060) as Rule based
model.

In Figure 2, we observe that Language Learning Models (LLMs) predominantly
take in the additional opinion suggested by expert models, with GPT4
exhibiting the highest standard and the greatest proportion of disagreements.
We consider any match among the top 5 additional opinions as being taken into
account by the LLMs. Intriguingly, for GPT4, the ratio of considered opinions
escalates from 0.549 to 0.602 as the number of opinions increases from 1 to 5.
This trend could partially elucidate the disparity in the final outcomes of
success rates.

Figure 2. For Webshop, the ratios of LLMs considering or disagreeing with
additional opinion provided by expert models. For the top 5 scenario, we
consider it as an agreement if any additional opinion matches.

####  3.4.2. ALFWorld

Taking inspiration from our previous Webshop experiment, we employed a similar
paradigm to combine Large Language Models (LLMs) with Expert models in the
context of the ALFWorld dataset. The fundamental premise remained analogous,
but due to resource constraints and the extensive size of ALFWorld (134 unseen
games), we confined our initial trial to a top 1 Imitation Learning (IL)
opinion.

Our hypothesis received empirical confirmation from the data observed for GPT4
and Claude, as detailed in Table 2. The success rate showed slight
improvements, although the original DAgger agent provided by ALFWorld was less
effective for general tasks compared to specific ones, as demonstrated by the
task-specific performance rates listed in Table 4 from the ALFWorld paper
(Shridhar et al., 2021), i.e in a disparate performance range, the DAgger
agent excelled in the ’Cool and Place’ task, achieving a remarkable 1.00
success rate. However, for the ’Pick Two and Place’ task, its proficiency was
considerably lower, with a success rate of only 0.24. Our investigation
further revealed a substantial variance in efficiency concerning the number of
steps taken by the ALFWorld IL model to complete tasks. This observation
highlights an additional layer of complexity, suggesting that the IL’s
performance is not only variable in terms of task success rates, but also in
the practical efficiency of task execution. This large variability in step
numbers indicates that while the IL model may excel in some tasks, it can be
markedly inefficient in others, further reinforcing the inherent limitations
and variability of IL models in complex task simulations like ALFWorld. In
terms of task completion, the IL’s outcomes are governed by real-time rewards.
As such, all completed tasks have to be executed correctly, as evidenced by
1.00 precision for the both IL with and without Beam Search. Conversely, the
tasks left incomplete were typically ones where the agent had exhausted the
available steps, with most of these marked by repeated and meaningless actions
as presented in Appendix 2.

One of the standout observations from our study was GPT4’s ability to
integrate past memories, task descriptions, and environmental data to discern
the pertinence of suggestions from the IL model. Even amidst noise, GPT4
demonstrated a robust capacity to differentiate beneficial from irrelevant
advice, often confidently disregarding suggestions that were not beneficial,
as illustrated in Appendix 3. Moreover, GPT4 was even able to extract values
from the initial part of a repetitive action pattern suggested by the IL,
underscoring its exceptional ability to distill useful information.
Contrastingly, GPT3.5 was easily misled by irrelevant suggestions and
frequently became entangled in the repetitive advice offered by the IL.
Indeed, such confusion even compromised GPT3.5’s capability to perform tasks
that it could otherwise successfully accomplish independently, as detailed in
Appendix 4. This highlights a stark divergence from the patterns observed with
GPT4 under ALFWorld Context.

In a compelling revelation, this study demonstrated a marked difference
between the contexts of the Webshop and ALFWorld experiments. The beneficial
guidance provided by the Webshop’s IL model effectively condensed the choice
spaces for LLMs, contrasted with the repetitive and misleading advice offered
by the IL in the ALFWorld context. Interesting discrepancies were also
observed in how the LLMs disagreed with the IL’s recommendations: Claude
registered a disagreement rate of 0.814, GPT3.5 of 0.769, and GPT4, leading
the pack, registered a rate of 0.854. This suggests an inherent capability
within LLMs to filter out misleading suggestions. However, the extent to which
this disagreement improved or impeded performance appeared to be context-
dependent, highlighting the importance of discernment in processing the IL’s
advice. Claude and specially GPT4 showed remarkable adeptness at avoiding the
pitfalls of misleading and repetitive advice. By contrast, GPT3.5 exhibited a
clear shortfall in this respect, a performance echoed by its pairing with a
random action in our Webshop ablation study. This underscores the importance
of context when integrating IL models with LLMs and signals the need for
careful evaluation when dealing with potentially misleading input, especially
on an LLM like GPT3.5 which can get easily confused.

Figure 3. For ALFWorld, the ratios of LLMs considering or disagreeing with
additional opinion provided by expert models. For the top 1 case, we only
redeem it as an agreement if the exact opinion matches.

###  3.5. Discussions

Initially, Auto-GPT was conceptualized as an experimental idea rather than a
robust workflow suitable for real-world applications. However, our research
demonstrates otherwise. Auto-GPT not only proves its potential for practical
use but also outperforms supervised state-of-the-art IL models with GPT4,
signifying a shift in perspective towards this innovative approach.

In the current discourse, we posit that this additional opinion approach can
readily find widespread adoption across diverse industries, given the existing
prevalence of expert models such as recommendation systems and traditional
natural language processing (NLP) services, inclusive of text classification
models and the like. An immediate application for this methodology can be
envisaged in leveraging LLMs for making definitive determinations and give
explanations regarding the prioritization of items, such as movies or songs,
to be displayed to the user. This is achievable by providing the selected
top-k outputs derived from a supervised recommendation model to LLMs as
Additional Opinions.

It is crucial to note, however, that the two tasks we have chosen to benchmark
in this research do not fully encapsulate the vast array of potential real-
world scenarios. They serve merely as a starting point for the exploration of
this idea. This is the inaugural instance where the concept of adapting Auto-
GPT to handle complex tasks by introducing Additional Opinions has been
proposed. This innovative approach opens new avenues for further research and
development, potentially expanding the realm of practical applications for AI
models and significantly impacting our understanding of complex decision-
making mechanisms.

##  4\. Related Work

Foundation models trained on self-supervision tasks have shown significant
success in downstream tasks, particularly in few-shot and zero-shot settings
(Devlin et al., 2018; Liu et al., 2019; Lan et al., 2019). More recently,
generative pre-trained foundation models have demonstrated impressive in-
context learning abilities, allowing them to tackle decision-making tasks that
require logic reasoning (Li et al., 2018) and/or interactions with external
APIs (Yang et al., 2023; Liang et al., 2023; Patil et al., 2023). However,
adapting LLMs for decision-making tasks often involves non-trivial prompt
design, memory retrieval mechanisms to dynamically construct the agent’s
context (Yao et al., 2022b; Shinn et al., 2023; Park et al., 2023), and
sometimes model fine-tuning (Liang et al., 2023) to enhance its decision-
making abilities.

Several techniques have been proposed to adapt Large Language Models (LLMs)
for improved planning and reasoning. These include methods for enabling
explicit Chain of Thought (CoT) thinking processes (Wei et al., 2022), as well
as prompting and decoding techniques aimed at enhancing the self-consistency
of LLMs (Fu et al., 2023; Wang et al., 2023). However, most of these
techniques primarily focus on offline reasoning tasks that can be planned
ahead, while their implications in online decision-making scenarios are rarely
discussed.

##  5\. Conclusion

Our experimental results highlight the successful adaptation of the Auto-GPT
styled agent to complex online decision-making tasks through straightforward
prompt design, surpassing IL-based baseline models specifically designed for
these tasks. Among the foundational LLMs powering Auto-GPT, GPT-4 demonstrates
superior performance. Additionally, we introduce an innovative strategy of
incorporating additional opinions from external expert models, further
enhancing the decision-making capabilities of Auto-GPT styled agents,
particularly benefiting GPT-4. Our Additional Opinions algorithm provides a
lightweight supervised training approach for Auto-GPT styled agents, enabling
improved performance without requiring extensive fine-tuning of the LLMs. We
demonstrate the effectiveness and adaptability of this approach, especially
for tasks with easily collectible training data for action policy. The code of
this work is shared in: https://github.com/younghuman/LLMAgent

## References

  * (1)
  * Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, and Cameron McKinnon. 2022.  Constitutional AI: Harmlessness from AI Feedback. In _arXiv:2212.08073_. arXiv.  https://ar5iv.org/abs/2212.08073
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023\.  Sparks of Artificial General Intelligence: Early experiments with GPT-4. (March 2023).  https://www.microsoft.com/en-us/research/publication/sparks-of-artificial-general-intelligence-early-experiments-with-gpt-4/
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022\.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ (2022).
  * Côté et al. (2018) Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, et al. 2018\.  Textworld: A learning environment for text-based games.  (2018), 41–75.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Fu et al. (2023) Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. 2023.  Complexity-Based Prompting for Multi-Step Reasoning.  arXiv:2210.00720 [cs.CL]
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019\.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_ (2019).
  * Li et al. (2018) Yang Li, Zhi-Ping Cai, and Hong Xu. 2018.  LLMP: exploiting LLDP for latency measurement in software-defined data center networks.  _Journal of Computer Science and Technology_ 33 (2018), 277–285.
  * Liang et al. (2023) Yaobo Liang, Chenfei Wu, Ting Song, Wenshan Wu, Yan Xia, Yu Liu, Yang Ou, Shuai Lu, Lei Ji, Shaoguang Mao, et al. 2023\.  Taskmatrix. ai: Completing tasks by connecting foundation models with millions of apis.  _arXiv preprint arXiv:2303.16434_ (2023).
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * OpenAI (2023) OpenAI. 2023.  GPT-4 Technical Report.  arXiv:2303.08774 [cs.CL]
  * Park et al. (2023) Joon Sung Park, Joseph C. O’Brien, Carrie Cai, Meredith Ringel Morris, Percy Liang, and Michael Bernstein. 2023.  Generative Agents: Interactive Simulacra of Human Behavior.  https://arxiv.org/pdf/2304.03442.pdf
  * Patil et al. (2023) Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023.  Gorilla: Large Language Model Connected with Massive APIs.  arXiv:2305.15334 [cs.CL]
  * Peng et al. (2022) Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, and Jianfeng Gao. 2022.  _Instruction Tuning with GPT-4_.  https://ar5iv.org/abs/2304.03277
  * Ross et al. (2011) Stephane Ross, Geoffrey J. Gordon, and J. Andrew Bagnell. 2011\.  A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning.  _arXiv preprint arXiv:1011.0686_ (2011).  arXiv:1011.0686
  * Shen et al. (2023) Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023\.  HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace.  arXiv:2303.17580 [cs.CL]
  * Shinn et al. (2023) Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023.  Reflexion: an autonomous agent with dynamic memory and self-reflection.  _arXiv preprint arXiv:2303.11366_ (2023).
  * Shridhar et al. (2020) Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020\.  ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks.  _arXiv preprint arXiv:1912.01734_ (2020).
  * Shridhar et al. (2021) Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. 2021.  AlfWorld: Aligning Text and Embodied Environments for Interactive Learning.  _arXiv preprint arXiv:2010.03768_ (2021).
  * Wang et al. (2023) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023\.  Self-Consistency Improves Chain of Thought Reasoning in Language Models.  arXiv:2203.11171 [cs.CL]
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ (2022).
  * Yang et al. (2023) Sherry Yang, Ofir Nachum, Yilun Du, Jason Wei, Pieter Abbeel, and Dale Schuurmans. 2023\.  Foundation Models for Decision Making: Problems, Methods, and Opportunities.  _arXiv e-prints_ (2023), arXiv–2303.
  * Yaniv (2004) Ilan Yaniv. 2004\.  The benefit of additional opinions.  _Current directions in psychological science_ 13, 2 (2004), 75–78.
  * Yao et al. (2022a) Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. 2022a.  Webshop: Towards scalable real-world web interaction with grounded language agents.  _arXiv preprint arXiv:2207.01206_ (2022).
  * Yao et al. (2022b) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2022b.  React: Synergizing reasoning and acting in language models.  _arXiv preprint arXiv:2210.03629_ (2022).

##  Appendix A Appendix

###  A.1. APPENDIX 1. Webshop: Evidence of GPT-4 considering additional
opinion.

####  A.1.1. Example 1. Webshop: AutoGPT(GPT-4) disagreed with the Additional
Opinion

    \footnotesize[breaklines]
    Command search returned: =Observation=
    Instruction:
    i want a pink niuta 2 pack hair towel wrap for dry hair, and price lower than 20.00 dollars
    [button] Back to Search [button_]
    Page 1 (Total results: 50)
    [button] Next > [button_]

    [button] B08G14B779 [button_]
    NIUTA 2 Pack Hair Towel Wrap, Microfiber Quick Drying Hair Towels, Super Absorbent Quick Dry Hair Towel,
    Wrapped Bath Cap (Pink+Light Blue)
    $4.99

    [button] B092QTHX2Z [button_]
    Microfiber Hair Towel Wrap POPCHOSE Ultra Absorbent, Fast Drying Hair Turban Soft, No Frizz Hair Wrap Towels for Women Wet Hair,
    Curly, Longer, Thicker Hair…
    $8.99

    [button] B08NX5S6S2 [button_]
    Microfiber Hair Towel Wrap for Women Super Absorbent Quick Dry Hair Towels with Button Design for Drying Curly&Thick Hair 2
    Pack(Purple+Camel)
    $8.99

    [button] B09KKCQ7FK [button_]
    VOLO Hero Microfiber Hair Towel, Super Absorbent, Ultra-Soft, Fast Drying | Reduce Dry Time by 50%, Large, Premium Wrap Towel,
    Sustainable Packaging, Anti-Frizz | Luna Gray (2 Pack)
    $70.0

    [button] B083FPQ77W [button_]
    Microfiber Hair Towels Super Absorbent Hair Drying Towel Turban for Women and Girls Quick Magic Hair Dry Hat Hair Towel Wrap Bathing
    Wrapped Cap 2 Packs (Pink+Grey)
    $13.99

    [button] B09KK4GZW1 [button_]
    VOLO Hero Microfiber Hair Towel | Super Absorbent, Ultra-Soft, Fast Drying | Reduce Dry Time by 50% | Large, Premium Wrap Towel for
    All Hair Types | Anti-Frizz, Anti-Breakage | Luna Gray (2 Pack)
    $78.0

    [button] B074T227KV [button_]
    Microfiber Hair Towel, 2 Pack Dry Hair Towel Twist Wrap Absorbent Quickly Dry Hair Towel for Kids and Women (Pink+Purple)
    $10.97

    [button] B08CZBMHDR [button_]
    Hair Towel Wrap, [2 Pack] Microfiber Quick Dry Hair Turban Wrap - Super Absorbent,Microfiber Hair Towel with Button,Quick Dry
    Hair Towel for Women,Hair Turbans for Wet Hair,Wrapped Bath Cap(Pink Blue)
    $10.99

    [button] B09FB25N5V [button_]
    SYMGILA Microfiber Hair Towel Wrap, Hair Turbans for Wet Hair, Drying Hair Wrap Towels for Curly Hair Women Anti Frizz 2 Pack
    (Blue/Beige)
    $100.0

    [button] B08YMYXBK8 [button_]
    VOLO Hero Microfiber Hair Towel, Super Absorbent, Ultra-Soft, Fast Drying | Reduce Dry Time by 50%, Large, Premium Wrap Towel,
    Sustainable Packaging, Anti-Frizz, Anti-Breakage, Hands-Free (Luna Gray)
    $34.99

    loop number:2
    Here’s one suggestion for the command: click with ’back to search’.
    Please use this suggestion as a reference and make your own judgement. Determine which next command to use, and respond
    using the JSON format specified above:
    {
        "thoughts": {
            "text": "I found the matching item with a price lower than $20.",
            "reasoning": "The first item (B08G14B779) matches the description and has a price of $4.99, which is lower than $20.",
            "plan": "- click on the matching item (B08G14B779)\n- review the item details\n- make the purchase",
            "criticism": "Clicking ’back to search’ is not necessary since I already found the matching item."
        },
        "command": {
            "name": "click",
            "args": {
                "tool_input": "B08G14B779"
            }
        }
    }

####  A.1.2. Example 2. Webshop: AutoGPT(GPT-4) agreed with the Additional
Opinion

    \footnotesize[breaklines,fontsize=\footnotesize]{text}
    Command search returned: =Observation=
    Instruction:
    i want to buy a folding storage box ottoman which i can easily install and has faux leather, the size of it should be 60x40x40cm,
    and price lower than 160.00 dollars
    [button] Back to Search [button_]
    Page 1 (Total results: 50)
    [button] Next > [button_]

    [button] B09LYXGGDD [button_]
    Storage Stool Faux Leather Pouffe Folding Storage Ottoman Footstool Box Toy Chest with Lid, Foldable Foot Stool Seat for Living Room,
    Bedroom (Pink,15.7x15.7x15.7in) Footstool Sofa Stool
    $274.66

    [button] B09LYZ12DH [button_]
    Storage stool Folding Ottoman Storage Bench Cube Hold up 881lbs Faux Leather Long Chest with Memory Foam Seat Footrest Padded
    Upholstered Stool for Bedroom Box Bed Coffee Table Rectangular Purple Foot
    $344.02

    [button] B09NXSRVSK [button_]
    Storage stool Folding Ottoman Storage Bench Cube Hold up 881lbs Faux Leather Long Chest with Memory Foam Seat Footrest Padded
    Upholstered Stool for Bedroom Box Bed Coffee Table Rectangular Brown (Size
    $408.25

    [button] B09NXP95GC [button_]
    Storage stool Storage Ottoman Bench, Foldable Footrest Shoe Bench with Large Space, End of Bed Storage Seat, Storage Stool Box Toy
    Chest Support 881lbs, Faux Leather fabric (Colourl:Orange,Size:40x40x
    $139.42

    [button] B09NTCCVGX [button_]
    Ottoman Footstools Folding Storage Box with Lid, Large Storage Seat Faux Leather Footstool Toy Storage Box, Yellow, 40x40x40cm (Color :
    Yellow, Size : 40X40X40cm)
    $149.97

    [button] B09NXRK5MX [button_]
    Storage stool Faux Leather Folding Organizer Storage Ottoman Bench Cube Foot Stool, Footrest Step Stool for Living Room, Bedroom,
    Office, Garden, Traveling, Fishing, Camping 15.7"x15.7"x15.7" (Purple)
    $150.01

    [button] B09NTFLKXR [button_]
    Folding Storage Ottoman Bench,Storage Chest Footrest Padded Seat,Faux Leather Ottoman Foot Rest Stool for Bedroom(Colour: Black,Size:
    40x40x40cm/15.7x15.7x15.7in) (Color : Black, Size : 60X40X40cm)
    $181.34

    [button] B09MYV58YY [button_]
    QTQHOME Rectangular Faux Leather Storage Ottoman Bench,Modern Tufted Storage Bench with Metal Leg,Upholstered Shoe Bench for Living
    Room Entryway Bedroom-Orange 60x40x40cm(24x16x16inch)
    $100.0

    [button] B08F3LZKSB [button_]
    YZZSJC Storage Stool Foot Stool Rest Ottoman Storage Boxes, Cloth Shoe Changing Stool Storage Box Minimalist Clothing Store Sofa Stool
    (Color : Light Blue, Size : 60x40x40cm)
    $264.04

    [button] B089D5K8LY [button_]
    XMZDDZ Ottoman Storage Seat Bench,Faux Leather Tufted Storage Bench Footrest Stool with Padded Seat,Rectangular Bed Bench for Entryway
    Bedroom Pink 60x40x40cm(24x16x16in)
    $100.0

    loop number:2
    Here’s one suggestion for the command: click with ’b09nxp95gc’.
    Please use this suggestion as a reference and make your own judgement. Determine which next command to use, and respond using the JSON
    format specified above:
    {
        "thoughts": {
            "text": "I need to click on the item B09NXP95GC.",
            "reasoning": "The item B09NXP95GC matches the size, material, and price requirements.",
            "plan": "- click on the item\n- review the item details\n- make the purchase",
            "criticism": "N/A"
        },
        "command": {
            "name": "click",
            "args": {
                "tool_input": "B09NXP95GC"
            }
        }
    }

###  A.2. APPENDIX 2. ALFWorld: Evidence of ALFWorld DAgger Agent

####  A.2.1. Example 1. ALFWorld: DAgger Agent repeated actions for ”pick and
place” task

    \footnotesize[breaklines,fontsize=\footnotesize]{text}

    pick_and_place_simple-PepperShaker-None-Drawer-10
    look | go to cabinet 4 | go to cabinet 4 | go to cabinet 4 | go to cabinet 1 | go to cabinet 4 | go to cabinet 4 | go to cabinet 4 | go to
    cabinet 4 | go to cabinet 4 | go to cabinet 3 | take saltshaker 1 from cabinet 3 | go to drawer 3 | open drawer 3 | put saltshaker 1 in/on drawer
    3 | close drawer 3 | go to cabinet 2 | open cabinet 2 | close cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 |
    go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 |
    go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 |
    go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 | go to cabinet 2 |
    go to cabinet 2 | go to cabinet 2 | go to cabinet 2

    pick_and_place_simple-Watch-None-Safe-219
    look | go to shelf 2 | go to shelf 1 | go to shelf 1 | go to shelf 1 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to
    shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12
    | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to
    shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12
    | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to
    shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12 | go to shelf 12

    pick_and_place_simple-Mug-None-Desk-308
    look | go to shelf 5 | go to shelf 2 | go to shelf 2 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to
    shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to
    shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to
    shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to
    shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to
    shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5 | go to shelf 5

####  A.2.2. Example 2. ALFWorld: DAgger Agent effective action for ”pick
clean then place” task

    \footnotesize[breaklines,fontsize=\footnotesize]{text}

    pick_clean_then_place_in_recep-Cloth-None-CounterTop-424
    look | go to cabinet 1 | open cabinet 1 | close cabinet 1 | go to countertop 1 | take cloth 3 from countertop 1 | go to sinkbasin 1 | clean cloth
    3 with sinkbasin 1 | go to countertop 1 | put cloth 3 in/on countertop 1 | --

    pick_clean_then_place_in_recep-Pan-None-CounterTop-1
    look | go to countertop 1 | take pan 1 from countertop 1 | go to sinkbasin 1 | clean pan 1 with sinkbasin 1 | go to countertop 3 | put pan 1
    in/on countertop 3 | --

    pick_clean_then_place_in_recep-Spatula-None-Drawer-10
    look | go to countertop 3 | take spatula 1 from countertop 3 | go to sinkbasin 1 | clean spatula 1 with sinkbasin 1 | go to drawer 2 | open
    drawer 2 | put spatula 1 in/on drawer 2 | --

###  A.3. APPENDIX 3. ALFWorld: Evidence of GPT-4 considering additional
opinion.

####  A.3.1. Example 1. ALFWorld: AutoGPT(GPT-4) disagreed with the Additional
Opinion

    \footnotesize[breaklines,fontsize=\footnotesize]{text}

    Instruction:
    You are in the middle of a room. Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a
    garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1.Your
    task is to: clean some soapbar and put it in countertop.
    After you go to countertop 1, current observation is: On the countertop 1, you see a soapbar 1, and a spraybottle 1.
    Available Actions are:
    action] examine countertop 1 [action_]
    [action] go to cabinet 1 [action_]
    [action] go to cabinet 2 [action_]
    [action] go to cabinet 3 [action_]
    [action] go to cabinet 4 [action_]
    [action] go to garbagecan 1 [action_]
    [action] go to handtowelholder 1 [action_]
    [action] go to handtowelholder 2 [action_]
    [action] go to sinkbasin 1 [action_]
    [action] go to sinkbasin 2 [action_]
    [action] go to toilet 1 [action_]
    [action] go to toiletpaperhanger 1 [action_]
    [action] go to towelholder 1 [action_]
    [action] inventory [action_]
    [action] look [action_]
    [action] take soapbar 1 from countertop 1 [action_]
    [action] take spraybottle 1 from countertop 1 [action_]

    HumanMessage(content=’
    Here’s one suggestion for the command: [’put handtowel 1 in/on countertop 1’].
    Please use this suggestion as a reference and make your own judgement. Determine which next command to use, and respond using the JSON format
    specified above:’)

    AIMessage(content=’
    {
        "thoughts": {
            "text": "...",
            "reasoning": "The suggestion to put handtowel 1 in/on countertop 1 is not relevant to the task. I found the soapbar on countertop 1,
            so I should take it and proceed with cleaning it.",
            "plan": "- Take the soapbar\\n- Clean the soapbar\\n- Put the soapbar on the countertop",
            "criticism": "N/A."
        },
        "command": {
            "name": "alfworld_action",
            "args": {
                "tool_input": "take soapbar 1 from countertop 1"
            }
        }
    }’)

####  A.3.2. Example 2. ALFWorld: AutoGPT(GPT-4) agreed with the Additional
Opinion

    \footnotesize[breaklines,fontsize=\footnotesize]{text}
    Instruction:
    You are in the middle of a room. Looking quickly around you, you see a cabinet 4, a cabinet 3, a cabinet 2, a cabinet 1, a countertop 1, a
    garbagecan 1, a handtowelholder 2, a handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, toiletpaperhanger 1, and a towelholder 1. Your
    task is to: clean some soapbar and put it in countertop.
    After you open cabinet 4, current observation is: Nothing happens.
    Available Actions are:
    [action] close cabinet 3 [action_]
    [action] examine cabinet 3 [action_]
    [action] go to cabinet 1 [action_]
    [action] go to cabinet 2 [action_]
    [action] go to cabinet 4 [action_]
    [action] go to countertop 1 [action_]
    [action] go to garbagecan 1 [action_]
    [action] go to handtowelholder 1 [action_]
    [action] go to handtowelholder 2 [action_]
    [action] go to sinkbasin 1 [action_]
    [action] go to sinkbasin 2 [action_]
    [action] go to toilet 1 [action_]
    [action] go to toiletpaperhanger 1 [action_]
    [action] go to towelholder 1 [action_]
    [action] inventory [action_]
    [action] look [action_]
    [action] take candle 1 from cabinet 3 [action_]
    [action] take toiletpaper 3 from cabinet 3 [action_]

    HumanMessage(content="
    Here’s one suggestion for the command: [’go to countertop 1’].
    Please use this suggestion as a reference and make your own judgement. Determine which next command to use, and respond using the JSON format
    specified above:")

    AIMessage(content=’{
        "thoughts": {
            "text": "...",
            "reasoning": "I have checked all the cabinets and didn\’t find a soapbar. I will now go to countertop 1 to check if there\’s a soapbar
            there.",
            "plan": "- Find and take the soapbar\\n- Clean the soapbar\\n- Put the soapbar on the countertop",
            "criticism": "N/A."
        },
        "command": {
            "name": "alfworld_action",
            "args": {
                "tool_input": "go to countertop 1"
            }
        }
    }’)

###  A.4. APPENDIX 4. ALFWorld: Evidence of GPT-3.5 confused by additional
opinion.

####  A.4.1. Example 1. ALFWorld: AutoGPT(GPT-3.5) performed the task by it’s
own

    \footnotesize[breaklines,fontsize=\footnotesize]{text}

    Instruction:
    You are in the middle of a room. Looking quickly around you, you see a bed 1, a desk 2, a desk 1, a drawer 6, a drawer 5, a drawer 4, a drawer 3,
    a drawer 2, a drawer 1, a garbagecan 1, a laundryhamper 1, a safe 1, a shelf 6, a shelf 5, a shelf 4, a shelf 3, a shelf 2, and a shelf 1.Your
    task is to: examine the mug with the desklamp.\nAfter you go to shelf 2, current observation is: On the shelf 2, you see a creditcard 2, and a
    pen 2.
    Available Actions are:
    [action] examine shelf 2 [action_]
    [action] go to bed 1 [action_]
    [action] go to desk 1 [action_]
    [action] go to desk 2 [action_]
    [action] go to drawer 1 [action_]
    [action] go to drawer 2 [action_]
    [action] go to drawer 3 [action_]
    [action] go to drawer 4 [action_]
    [action] go to drawer 5 [action_]
    [action] go to drawer 6 [action_]
    [action] go to garbagecan 1 [action_]
    [action] go to laundryhamper 1 [action_]
    [action] go to safe 1 [action_]
    [action] go to shelf 1 [action_]
    [action] go to shelf 3 [action_]
    [action] go to shelf 4 [action_]
    [action] go to shelf 5 [action_]
    [action] go to shelf 6 [action_]
    [action] inventory [action_]
    [action] look [action_]
    [action] take creditcard 2 from shelf 2 [action_]
    [action] take pen 2 from shelf 2 [action_]

    HumanMessage(content=’Determine which next command to use, and respond using the JSON format specified above:’)
    AIMessage(content=’
    {
        "thoughts": {
            "text": "...",
            "reasoning": "I need to find a mug and a desklamp. A mug is more likely to appear in shelf 1-6, desk 1-2, drawer 1-6. A desklamp is more
            likely to appear in desk 1-2, shelf 1-6. I can check one by one, starting with shelf 3.",
            "plan": "- Find a mug and a desklamp\\n- Examine the mug with the desklamp",
            "criticism": "N/A."
        },
        "command": {
            "name": "alfworld_action",
            "args": {
                "tool_input": "go to shelf 3"
            }
        }
    }’)

####  A.4.2. Example 2. ALFWorld: AutoGPT(GPT-3.5) confused by IL repeated
opinion

    \footnotesize[breaklines,fontsize=\footnotesize]{text}
    Instruction:
    You are in the middle of a room. Looking quickly around you, you see a bed 1, a desk 2, a desk 1, a drawer 6, a drawer 5, a drawer 4, a drawer 3,
    a drawer 2, a drawer 1, a garbagecan 1, a laundryhamper 1, a safe 1, a shelf 6, a shelf 5, a shelf 4, a shelf 3, a shelf 2, and a shelf 1.Your
    task is to: examine the mug with the desklamp.
    After you go to shelf 2, current observation is: Nothing happens.
    Available Actions are:
    [action] examine shelf 2 [action_]
    [action] go to bed 1 [action_]
    [action] go to desk 1 [action_]
    [action] go to desk 2 [action_]
    [action] go to drawer 1 [action_]
    [action] go to drawer 2 [action_]
    [action] go to drawer 3 [action_]
    [action] go to drawer 4 [action_]
    [action] go to drawer 5 [action_]
    [action] go to drawer 6 [action_]
    [action] go to garbagecan 1 [action_]
    [action] go to laundryhamper 1 [action_]
    [action] go to safe 1 [action_]
    [action] go to shelf 1 [action_]
    [action] go to shelf 3 [action_]
    [action] go to shelf 4 [action_]
    [action] go to shelf 5 [action_]
    [action] go to shelf 6 [action_]
    [action] inventory [action_]
    [action] look [action_]
    [action] take creditcard 2 from shelf 2 [action_]
    [action] take pen 2 from shelf 2 [action_]

    HumanMessage(content="Here’s one suggestion for the command: [’go to shelf 2’].
    Please use this suggestion as a reference and make your own judgement. Determine which next command to use, and respond using the JSON format
    specified above:")

    AIMessage(content=’{
        "thoughts": {
            "text": "I will use the \’alfworld_action\’ command to examine shelf 2 again.",
            "reasoning": "Since I did not find the desklamp on shelf 2, I need to examine it again to make sure I didn\’t miss               anything.",
            "plan": "- Use the \’alfworld_action\’ command to go to shelf 2.\\\\n- Examine the items on the shelf again to  find the desklamp.
            \\\\n- Once I find the desklamp, I can use it to examine the mug.",
            "criticism": "I should have been more thorough in my examination of shelf 2 the first time."
            },
        "command": {
            "name": "alfworld_action",
            "args": {
                "tool_input": "go to shelf 2"
            }
        }
    })