# Describe, Explain, Plan and Select: Interactive Planning with Large Language
Models Enables Open-World Multi-Task Agents

Zihao Wang  Shaofei Cai  Anji Liu  Xiaojian Ma  Yitao Liang

###### Abstract

In this paper, we study the problem of planning in Minecraft, a popular,
democratized yet challenging open-ended environment for developing multi-task
embodied agents. We’ve found two primary challenges of empowering such agents
with planning: 1) planning in an open-ended world like Minecraft requires
precise and multi-step reasoning due to the long-term nature of the tasks, and
2) as vanilla planners do not consider the proximity to the current agent when
ordering parallel sub-goals within a complicated plan, the resulting plan
could be inefficient. To this end, we propose “Describe, Explain, Plan and
Select” (DEPS), an interactive planning approach based on Large Language
Models (LLMs). Our approach helps with better error correction from the
feedback during the long-haul planning, while also bringing the sense of
proximity via goal Selector, a learnable module that ranks parallel sub-goals
based on the estimated steps of completion and improves the original plan
accordingly. Our experiments mark the milestone of the first multi-task agent
that can robustly accomplish 70+ Minecraft tasks and nearly doubles the
overall performances. Finally, the ablation and exploratory studies detail how
our design beats the counterparts and provide a promising update on the
ObtainDiamond grand challenge with our approach. The code is released at
https://github.com/CraftJarvis/MC-Planner.

Machine Learning, ICML

##  1 Introduction

Developing multi-task agents that can accomplish a vast and diverse suite of
tasks in an open-ended world has been viewed as one of the key milestones
towards generally capable artificial intelligence (Reed et al., 2022; Alayrac
et al., 2022; Brown et al., 2020; Fan et al., 2022). To enable such
capabilities, earlier works have suggested employing a hierarchical goal
execution architecture (Bacon et al., 2017; Brohan et al., 2022), where a
planner generates action plans that would then be executed by low-level goal-
conditioned controllers. This architecture has been delivering promising
progress in many robotics domains, including table-top and mobile manipulation
(Zeng et al., 2022; Brohan et al., 2022), 2D shape drawing (Liang et al.,
2022) and table rearrangement (Huang et al., 2022b). However, whether the
success can be transferred to a more open-ended world like Minecraft remains
open (Guss et al., 2019b; Fan et al., 2022; Guss et al., 2019a, 2021;
Kanervisto et al., 2022). To understand the gap, we’ve identified two primary
challenges of planning in these environments.

Figure 1:  Planning success rates of tasks that require a different number of
reasoning steps. Complex tasks in open-ended environments usually comprise
more sub-goals to be completed, which places a higher demand on the multi-step
reasoning of the planner. The planning success rate is computed considering an
oracle and perfect controller if provided.

First of all, many tasks in Minecraft can be rather complex, as they usually
comprise multiple sub-goals to be completed, e.g. the task build a bed
includes 7 sub-goals and therefore demand significantly longer reasoning steps
of the planner. Meanwhile, many of these sub-goals have to be planned
precisely with the exact object name and quantities, e.g. mine 3 woods , kill
3 sheep ; otherwise, the subsequent sub-goals won’t be executed due to failed
preconditions. As demonstrated in Figure 1, accurate multi-step reasoning
indeed imposes a great challenge for the planners and we can observe a
significant regression in tasks requiring longer steps to be completed.

Another challenge brought by the complicated tasks in an open-ended world is
the efficiency of the produced plans, which is illustrated in Figure 2. A
complicated plan in an open-world environment usually includes multiple
parallel sub-goals that can be simultaneously executed. However, some
orderings of these sub-goals are better than others, e.g. kill sheep
$\rightarrow$ mine wood in Figure 2 is more efficient than other sequences
given the agent situation. Unfortunately, canonical planners do not consider
the current proximity of the sub-goals to the agent when devising the plans,
thereby producing inefficient plans and ultimately leading to a lower success
rate.

Figure 2:  Comparison of plans produced by the planner with or without the
“Select” component. Given parallel sub-goals, i.e. mine wood and kill sheep ,
our Selector will determine the sequence in which to carry out these sub-goals
based on their current proximity to the agent  and modify the original plan
produced by the planner accordingly (black $\rightarrow$ red plan). The
resulting plan is more efficient and therefore yields a higher success rate.

To this end, we propose “Describe, Explain, Plan and Select” (DEPS), an
interactive planning approach based on Large Language Models (LLMs) to
alleviate the aforementioned issues in most open-world environments. Whenever
a failure happens when executing the current plan, a descriptor will summarize
the current situation as text and send it back to the LLM-based planner. The
LLM-based planner will then be prompted as explainer to locate the errors in
the previous plan. Finally, the planner will re-plan the task to obtain a
correct plan. This allows the feedback from the agents to be better handled by
the planner and increases the overall success rate on Minecraft tasks by
52.74%. Additionally, in the case when there are parallel sub-goals lining up,
a goal-selector will be invoked to select the most accessible sub-task based
on the proximity to the agent and edits the original plan accordingly. The
complete DEPS system ultimately yields an 88.44% gain in average success rate.

Our experiments are conducted on 71 tasks in Minecraft. We find DEPS
outperforms all baselines by nearly doubling the overall success rate. Our
ablation and exploratory studies then explain how our approach beats the
counterparts and becomes the first planning-based agent that accomplishes the
challenging ObtainDiamond task.

To summarize, our contributions are two fold:

  * •

We propose “Describe, Explain, Plan and Select” (DEPS), an interactive
planning method based on Large Language Models to address the main issues in
open-world planning, including multi-step reasoning and the lack of sense of
agent’s proximity to the sub-goals.

  * •

We conduct comprehensive studies on the popular yet challenging Minecraft
domain with many strong baselines and our approach. Overall the results
indicate the significant advantages of DEPS over its counterparts in terms of
success rate. Notably, DEPS is the first agent with planning that ever
achieves an above-zero success rate on the challenging ObtainDiamond task.

##  2 Background

###  2.1 Problem Formulation

We aim to learn an agent that can solve arbitrary long-horizon goal-reaching
tasks with image observation and language goals. To achieve this, we consider
planning, which combines goal-conditioned policies trained to accomplish sub-
goals with a planner that decomposes a long-horizon task into $K$ short-
horizon sub-goals. Specifically, the planner chooses $K$ sub-goals,
$g_{1},\ldots,g_{K}$, and attempts to accomplish them one by one. At every
time step $t$, for different sub-goal $g_{k}$, a goal-conditioned policy
$\pi(a_{t}\mid s_{t},g_{k})$ samples an action $a_{t}$ considering the current
state $s_{t}$. A key benefit of this planning-based approach is that the same
set of sub-goals can be reused and composed flexibly to carry out various
tasks. In this paper, we employ an LLM-based planner to leverage its encoded
abundant knowledge.

###  2.2 Challenges of Planning in Open Worlds

Compared to canonical domains such as robotics table rearrangement (Huang et
al., 2022b), planning in open-world embodied environments poses the following
additional challenges. 1) long-term planning. Due to the richness of terrains,
objects, and action spaces, tasks in open-world domains require exceptionally
long planning horizons, while the plan still has to be precise and executable.
As a result, the success rate degrades significantly as the number of
reasoning steps required by the task increases (see Figure 1); 2) planning
efficiency. Complicated plans in open-world environments usually comprise sub-
goals that can be executed in parallel, but canonical planners do not take
into consideration the agent’s proximity to every sub-goal, measured by the
time steps required to accomplish it. Therefore, these planners tend to
generate inefficient plans (an example can be found in Figure 2). In this
paper, we hope to address these two challenges that bottleneck planning-based
agents in open-world environments.

Figure 3:  Overview of our proposed dynamic planner architecture. Given a task
$T$, our “Describe, Explain, and Plan with LLM by interactive prompting”
pipeline generates executable and explainable plans $P_{t}$ iteratively. The
horizon-predictive Selector selects as state-aware goal as the current goal
$g_{t}$. The symbol $\land$ and $\vee$ in plan $P$ represent the relation AND
(all goals need to be finished, with no particular requirement on the order)
and OR (either needs to be finished), respectively.

###  2.3 Planning with Large Language Models

Previous works have shown that large language models (LLMs) such as
InstructGPT (Ouyang et al., 2022) and Codex (Chen et al., 2021) can be used as
zero-shot planners to generate sub-goal sequences for various tasks in
embodied environments (Huang et al., 2022a; Singh et al., 2022). Formally,
given the task description $T$ as prompt $p$, LLM acts as a planner to decode
$T$ into $K$ sub-goals, $g_{1},\ldots,g_{K}$, which are then executed one by
one by the low-level controller $\pi(a_{t}\mid s_{t},g_{k})$ to accomplish the
task.

One potential deficit of the above pipeline is that the LLM-based planner
cannot receive feedback from the environment, which could lead to non-
executable and redundant plans. Prior works enabled feedback to the LLM
through external components such as affordance functions (Brohan et al.,
2022), success detector (Liang et al., 2022) or scene descriptor (Huang et
al., 2022b) to ground the LLM plans executable by robots. As discussed in
Section 2.2, the long-horizon tasks in partially observed open-world domains
make it more challenging for LLM-based planners.

##  3 Towards Reliable Planning in Embodied Open-World Environments

In this section, we first give an overview of our proposed interactive
planning framework “Descibe, Explain, Plan, and Select” (DEPS) for solving
complex and long-horizon tasks in open-world environments. Next, in Section
3.2, we will walk through how our framework iterates its plan with feedback
from the agent. In Section 3.3, we take a closer look at how our “Selector”
optimizes the original plan to be more efficient by leveraging the agent’s
proximity to each sub-goal. Additional implementation details of DEPS are
provided in Appendix C.

###  3.1 DEPS Architecture

As demonstrated in Figure 3, our agent (DEPS) consists of an event-triggered
Descriptor, a Large Language Model (LLM) as Explainer and Planner, a goal
Selector based on horizon prediction and a goal-conditioned controller.

We take a large language model (LLM) as zero-shot Planner of the agent to
complete tasks. Given an instruction command (e.g., ObtainDiamond) as task
$T$, the LLM-based planner extracts the high-level task into a sequence of
sub-goals $\\{g_{1},\ldots,g_{K}\\}$, as the initial plan $P_{0}$. The goals
are instructions in natural language, e.g., mine oak wood . We provide a list
of all possible goals in Appendix B.

Though many goals are sequential in nature, i.e., one is pre-conditoned on the
completion of an other, some are connected through AND$\land$ and OR$\vee$.
When goals are put together with AND$\land$, all of them need to be completed,
yet the order to carry out them does not matter as there is no interdependency
among these goals. When goals are put together with OR$\vee$, we can simply
choose any of them to execute. For example, mine coal $\land$ mine iron ore ,
and mine oak wood $\vee$ mine birch wood in Figure 3. Given in either case,
any completion order is the same, these goals are referred to as parallel
goals. The Selector is trained to predict the number of time steps remaining
to achieve every goal $g_{k}$ in a set of parallel goals given the current
state $s_{t}$. Using this information, it then selects the suitable goal as
the current goal $g_{t}$. For example, the Selector will choose  as current
goal $g_{t}$ instead of  as iron ore is closer to the agent. More details on
the Selector can be found in Section 3.3.

Finally, the selected goal $g_{t}$ will be executed by the low-level multi-
task controller. The controller is denoted as a goal-conditioned policy
$\pi(a|s,g)$ for decision-making, which can map the current state $s_{t}$ to
action $a_{t}$ conditioned on goal $g_{t}$. The policy $\pi$ can be trained by
reinforcement learning (Schulman et al., 2017) or imitation learning (Baker et
al., 2022) methods.

However, the initial plan $P_{0}$ is difficult to execute successfully in one
shot, because it often contains errors. For example, the goal  can not be
finished only with a wooden pickaxe . When failure pops up, Descriptor will
summarize the current state $s_{t}$ and execution outcome of the most recent
goal into text $d_{t}$ and send it to the LLM. The LLM will first try to
locate the errors in the previous plan $P_{t-1}$ by self-explanation, e.g.,
the goal  need to be executed with a stone pickaxe . Then it will re-plan the
current task $T$ and generate a revised plan $P_{t}$ according to the
explanation. In this process, the LLM is also treated as an Explainer in
addition to the Planner role. The Descriptor, Explainer, and Planner will be
detailed in Section 3.2.

Description $\displaystyle:d_{t}=f_{\text{DESC}}(s_{t-1}),$ (1) Prompt
$\displaystyle:p_{t}=\text{CONCAT}(p_{t-1},d_{t}),$ Plan
$\displaystyle:P_{t}=f_{\text{LM}}(p_{t}),$ Goal $\displaystyle:g_{t}\sim
f_{\text{S}}(\text{P}_{t},s_{t-1}),$ Action
$\displaystyle:a_{t}\sim\pi(a_{t}\mid s_{t-1},g_{t})$

As shown in Equation (1), DEPS will iteratively update the plan $P_{t}$ until
the task is finished, where $f_{\text{DESC}}$ is the descriptor model,
$f_{\text{LM}}$ denotes the language model as explainer and planner,
$f_{\text{S}}$ is the selector model, $\pi$ is goal-conditioned policies from
the controller.

###  3.2 Describe, Explain and Plan with LLM Generates Executable Plans

Figure 4:  A demonstration of the process of “describe, explain, plan and
select” with a LLM-based planner. The context in the figure is modified for
better visualization. The full interaction dialogue of our agent is listed in
Appendix H.

Current LLM-based planners usually query the LLM once at the beginning of
every episode and use the output plan throughout the episode (Huang et al.,
2022a; Singh et al., 2022). However, as demonstrated by Figure 1, such one-
shot planning methods often fail on long-horizon tasks that require many sub-
goals. This is caused by two major issues. First, since the correct plan for
long-horizon tasks needs to respect various complex preconditions, it is
extremely hard for the LLM to generate a flawless plan directly from the task
instructions, resulting in failure when simply following the initial plan.
Additionally, due to the unpredictable transition dynamics, some incidents may
happen during the execution and makes the initial plan non-executable. To
remedy these problems, existing methods introduce feedback (e.g., from success
detector or scene descriptor) to reflect on the results of previous executions
(Huang et al., 2022b; Liang et al., 2022; Brohan et al., 2022). However,
merely informing the LLM whether a sub-goal is completed is often insufficient
to correct the planning error.

To remedy this, we propose “describe, explain and plan”, a new interactive
planning method to generate more executable and explainable plans. We start
with rewriting the prompt into an interactive dialogue format as in ChatGPT
(Ouyang et al., 2022) so that subsequent feedback can be passed to the LLM
effectively. The produced plan is also augmented with the preconditions and
effects of each goal. The structured prompt improves the readability and
interpretability of the plan and facilitates error-locating when the execution
fails later. For example, we modify the original goal description craft wooden
pickaxe into:

⬇

craft({wooden_pickaxe:1}, {planks:3, stick:2}, crafting_table); # craft 1
wooden_pickaxe from 3 planks and 2 stick, on crafting_table

The Descriptor will then collect the feedback generated by the agent during
the execution of the task. The feedback can be practically obtained either by
a person (human feedback (Brohan et al., 2022)), or by a pre-trained vision-
language model CLIP (Radford et al., 2021). While the previous type of
feedback needs intensive human involvement, the latter from the pre-trained
model needs to be fine-tuned for the specific domain, which decreases the
automation and generalization of the agent. On the contrary, Minecraft returns
the ‘info’ and other high-level observations (such as biome, GPS, and
compass), we can easily translate the unstructured information into structured
language. Therefore we take the symbolic information available in the game and
translate it into feedback description $d_{t}$ in this work. To avoid carrying
unrelated information in the prompt, we further distill plan-related messages
(e.g., inventory information, biome, action results, etc.) as final event-
level description $d_{t}$ as demonstrated in Figure 4.

Notably, we also treat the LLM as an Explainer to explain why the previous
plans $P_{t-1}$ failed. Specifically, by analyzing the current state from
description $d_{t}$ and precondition of current goal $g_{t}$, the explainer
can identify the reason why the current goal cannot be executed successfully.
As shown in Figure 4, the reason may be the current goal requires the use of
an iron pickaxe, but the tool is not prepared in advance, or the current goal
requires the use of 3 planks, but the currently available planks are not
enough. To implement this, we provide few-shot demonstrations to the LLM as in
chain-of-thoughts prompting (Wei et al., 2022). Finally, the LLM goes back to
its role as a Planner and re-plans the task with the explicit explanation of
existing bugs in previous plan $P_{t-1}$, ultimately generating an updated
plan $P_{t}$ according to the explanation.

###  3.3 Horizon-Predictive Selector Yields Efficient Plans

Due to the abundance of objects and the compositional nature of their
functionalities, there often exist multiple feasible plans to complete a task,
i.e., there are usually multiple paths for the completion of a particular
goal. However, despite the feasibility of all such plans, most of them are
highly inefficient to execute in the current episode. For example, as shown in
Figure 3, obtaining a wood can be done by chopping oak trees , birch trees ,
or acacia trees . But only oak trees are available in the plains biome. So the
planner needs to choose oak trees since it is more efficient, as the agent
does not need to travel to another biome.

On the other hand, there is no strict sequential requirement for some goals in
the plan $P_{t}$, i.e., $g_{i},g_{j}\sim P_{t}$ enjoy the same precondition,
which means $g_{i}$ and $g_{j}$ can be executed in any order. As shown in
Figure 2, the choice of different paths (sequences) may affect the execution
efficiency of the plan $P_{t}$ as one goal might be closer to the agent.
Always choosing the closer goal to execute first could yield more efficient
plans and improve the final success rate under a limited episode length.
Moreover, the dynamic nature of open-world environments further amplifies the
impact of more efficient plans on the success rate. For example, in Minecraft,
if the agent chooses to execute a further goal like collect wood first, the
much closer target sheep may have disappeared and hard to be found again.

In order to improve the efficiency of our plans, we propose to use a Selector
that selects the most efficient path with the highest execution success rate
as the final plan. Specifically, we design a state-aware selector to choose
the nearest goal under state $s_{t}$ as the current goal $g_{t}$ from the
candidate goal sets in plan $P_{t}$. It predicts the goal distribution
$p(g_{t}|s_{t},P_{t})$ under the current state $s_{t}$ and plan $P_{t}$, where
$g_{t}\in G_{t}$, $G_{t}$ describes all current executable goals in $P_{t}$. A
straight way to implement the selector is to leverage the semantic similarity
between the current state and the goal text using a vision-language model
(VLM) such as CLIP (Radford et al., 2021). Nevertheless, this may not exactly
reflect the difficulty of completing the goal since VLM lacks practical
experience. For example, an “oak tree” in front of the agent could lead to
high semantic similarity for the “chopping tree” goal, but it may be
inefficient to achieve this goal if a canyon is in the middle between the
agent and the oak tree.

To mitigate this, we implement a horizon-predictive selector that embeds
practical task experience to accurately rank the goal based on their
efficiency and feasibility. Here, we define the horizon of a goal
$h_{t}(g):=T_{g}-t$ as the remaining time steps to complete the given goal,
where $T_{g}$ is the time of completing goal $g$. This metric accurately
reflects how quickly we can achieve the given goal from the current state. To
estimate the horizon, we learn a neural network $\mu$ to fit the offline
trajectories by minimizing the entropy loss $-\text{log}\
\mu(h_{t}(g)|s_{t},g)$, where $h_{t}$ is the ground-truth horizon in
trajectories of completing goal $g$. Therefore, the goal distribution can be
formulated as follows:

$f(g_{t}\mid s_{t},P_{t})=\frac{exp(-\mu(g_{t},s_{t}))}{\sum_{g\in
G_{t}}exp(-\mu(g,s_{t}))}.$ (2)

We set goal-sensitive Impala CNN (Cai et al., 2023) as the backbone of the
selector. In practice, the horizon predictive selector can be jointly trained
with the controller policies and share the backbone parameters (Cai et al.,
2023).

##  4 Experiments

This section analyzes and evaluates our proposed “describe, explain, plan, and
select” (DEPS) method. To minimize performance variation caused by the low-
level controller, we standardize all experiments with one controller learned
by behavior cloning. We refer to the details of this controller in Appendix B.
In Section 4.1, we introduce our testing environments and our proposed
Minecraft Task101, consisting of the hardest 71 tasks distilled from Minedojo
(Fan et al., 2022). In Section 4.2, we report our performance in the context
of existing LLM-based planners. Ablation studies are conducted in Section 4.3.
Finally, we pay close attention to the hardest among Task101, ObtainDiamond,
which is long-hailed as a major challenge in the community.

###  4.1 Experimental Setup

Table 1: Attributes of 8 meta tasks covering Task101: We evaluate the
algorithm on Minecraft Task101. We group the consisted 71 task into 8
different meta groups, with each focusing on testing a different aspect of our
proposed method.

Meta | Name | Num. | Example Task | Max. Steps
---|---|---|---|---
MT1 | Basic | 14 | Make a crafting table. | 3000
MT2 | Tool(Simple) | 12 | Make a stone pickaxe. | 3000
MT3 | Hunt&Food | 7 | Cook the beef. | 6000
MT4 | Dig-Down | 6 | Mine coal. | 3000
MT5 | Equipment | 9 | Equip the leather helmet. | 6000
MT6 | Tool(Complex) | 7 | Make shears and bucket. | 6000
MT7 | IronStage | 13 | Craft minecart. | 6000
MT8 | Challenge | 1 | Mine diamond! | 12000

We evaluate our proposed method in Minecraft, a popular open-world environment
with both challenges discussed in 2.2. We distill the 1523 MineDojo (Fan et
al., 2022) programmatic tasks and select the hardest setting (i.e., no task-
specific tools are given for agents in advance) to form our proposed Minecraft
Task101. To better present the results, we divide the 71 Task101 tasks into 8
meta groups according to the ingredients and function of the tasks, i.e.,
MT1-MT8 as illustrated in Table 1. Considering how long it typically takes
human players to complete each task as a ballpark (Guss et al., 2019b), we set
different maximum episode steps for different meta tasks from 3000 (for
easiest Basic tasks) to 12000 (for the hardest Challenge tasks). The names,
number of required reasoning steps, and functions of all tasks are listed in
Appendix A. Note that our agent will be summoned in different environments
randomly for each evaluation. Biomes and initial positions are also different
each time. Following the previous work (Johnson et al., 2016), we take the
success rate as the evaluation metric.

Table 2: Success rates of DEPS and existing LLM planners on Minecraft Task101.
The full list for each task is in Appendix G.

Methods | MT1 | MT2 | MT3 | MT4 | MT5 | MT6 | MT7 | MT8 | AVG
---|---|---|---|---|---|---|---|---|---
Zero-shot Planner | 25.85$\pm$24.8 | 47.88$\pm$31.5 | 10.78$\pm$14.6 | 7.14$\pm$9.0 | 1.98$\pm$5.9 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 15.42
ProgPrompt | 30.61$\pm$23.6 | 40.09$\pm$30.6 | 17.13$\pm$19.1 | 16.00$\pm$17.3 | 3.21$\pm$4.9 | 0.47$\pm$1.3 | 0.60$\pm$2.2 | 0.0$\pm$0.0 | 16.88
Chain of Thought | 40.24$\pm$30.8 | 55.21$\pm$26.8 | 6.82$\pm$11.6 | 4.76$\pm$8.2 | 1.73$\pm$5.2 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 0.0$\pm$0.0 | 18.89
Inner Monologue | 46.89$\pm$31.4 | 53.73$\pm$20.8 | 3.64$\pm$6.9 | 18.41$\pm$17.4 | 4.57$\pm$7.4 | 0.64$\pm$1.7 | 1.02$\pm$2.5 | 0.0$\pm$0.0 | 21.64
Code as Policies | 60.08$\pm$17.3 | 60.11$\pm$20.24 | 8.72$\pm$9.7 | 20.33$\pm$ 21.0 | 2.84$\pm$4.6 | 0.63$\pm$1.3 | 0.60$\pm$2.2 | 0.0$\pm$0.0 | 25.77
DEP (w/o Selector) | 75.70$\pm$10.4 | 66.13$\pm$13.4 | 45.69$\pm$16.2 | 43.35$\pm$20.2 | 15.93$\pm$13.9 | 5.71$\pm$3.7 | 4.60$\pm$7.1 | 0.50$\pm$0.5 | 39.36
DEPS | 79.77$\pm$8.5 | 79.46$\pm$10.6 | 62.40$\pm$17.9 | 53.32$\pm$29.3 | 29.24$\pm$27.3 | 13.80$\pm$8.0 | 12.56$\pm$13.3 | 0.59$\pm$0.5 | 48.56

###  4.2 Main Results

We compare our methods with other LLM-based planners, including Zero-shot
Planner (Huang et al., 2022a), ProgPrompt (Singh et al., 2022), Chain of
Thought (Wei et al., 2022), Inner Monologue (Huang et al., 2022b), Code as
Policies (Liang et al., 2022). Since these methods are not originally
experimented in Minecraft, we reproduce them to conform to the Minecraft
specification based on prompt and feedback template design. All planner
methods access the LLM model through OpenAI API (text-davinci-03 model (Ouyang
et al., 2022) for Zero-shot Planner, Chain of Thought, and Inner Monologue,
and code-davinci-02 model (Chen et al., 2021) for Code as Policies,
ProgPrompt, and Ours). All hyper-parameters of LLM are kept as default.

Every task is executed 30 times and the average results for every meta task
are listed in Table 2. Our approach achieves the best performance with all
meta tasks. As the complexity of the task increases from MT1-MT8, the planner
usually needs to give more accurate task steps (i.e., longer goal sequence) to
achieve the final task. Therefore the success rate of all agents decreases
with the reasoning steps increasing. Starting from MT6, almost all existing
LLM-based planners fail (nearly 0 success rate). DEP (w/o Selector) already
consistently beats existing LLM-based planners in all meta tasks with a
significant margin. This validates that “describe, explain and plan” can
estimate the reason for current plan failure and correct the original flawed
plans. Due to the limited maximum episode length and restricted control
success rate for a hard goal (e.g., Mine diamond with iron_pickaxe), the final
success rate is still capped.

In addition, Selector also greatly improves the final task success rate of the
agent (from DEP w/o Selector to DEPS). Hard meta tasks usually require the
completion of multiple sub-goals (up to dozens of goals), thus bringing more
flexibility and providing more candidate goals for the Selector. At the same
time, as the agent conducts experiments with limited episode length, it also
places high demands on the efficiency of the plan. Therefore, the Selector
brings a significant improvement on efficiency-sensitive tasks such as MT7 (up
to +2.7 times success rate).

###  4.3 Ablation Study

We conduct ablation experiments to investigate the specific impact of the
rounds of DEPS and the number of candidate executable goals for different
Selector models.

####  4.3.1 Ablation on DEPS rounds

We evaluate our agent on all tasks with increasing maximum rounds of DEPS. The
round is defined as a cycle of interactive LLM-based planning with
description, explanation, and planning and selecting, i.e., an updated plan.
All tasks for every maximum round are executed 30 times and the average
success rate is reported in Table 3. We take the agent without DEPS (i.e., 0
round) as the baseline model. Our results in the previous subsection utilize
the maximum rounds possible under 8000 tokens capped by OpenAI, We also report
the success rate increment of every meta task in column $\Delta$ in Table 3.
This set of experiments demonstrates that DEPS can iteratively improve its
plan in open-world environments. More DEPS rounds produce better results,
especially for hard tasks.

Table 3: Success rates of DEPS under different maximum rounds of interactive
feedback.

Rounds | 0 | 1 | 3 | 5 | $\infty$ | $\Delta(0\rightarrow\infty)$
---|---|---|---|---|---|---
MT1 | 28.61 | 50.61 | 68.08 | 80.23 | 79.77 | +51.16
MT2 | 37.09 | 71.25 | 71.38 | 79.22 | 79.46 | +42.37
MT3 | 15.13 | 20.10 | 40.33 | 40.87 | 62.4 | +47.27
MT4 | 15.88 | 17.42 | 48.33 | 50.70 | 53.32 | +37.44
MT5 | 3.16 | 3.17 | 2.84 | 15.23 | 29.24 | +26.08
MT6 | 0.48 | 0.51 | 1.11 | 1.94 | 13.8 | +13.32
MT7 | 0.58 | 2.25 | 2.85 | 2.87 | 12.56 | +11.98
MT8 | 0.00 | 0.00 | 0.00 | 0.00 | 0.59 | +0.59

####  4.3.2 Ablation on Selector

We verify the robustness of our proposed Selector under different parallel
goals. The agent is asked to complete 2, 3, and 4 candidate goals (the
precondition is consistent for all goals), respectively. The goals of the task
correspond to different kinds of mobs or materials.

We report the final success rate of our method (DEP) with different selector
implementations, including using a fixed sequence of goals, a random sequence
of goals, and selecting a goal based on MineCLIP (Fan et al., 2022), CLIP
(Radford et al., 2021), and our horizon-predictive Selector (HPS). As Figure 5
shows, in one round of parallel candidate goals, an improvement of
$\Delta$=+22.3%, +29.2%, +32.6% is obtained using our horizon-predictive
Selector compared to not any selector (i.e., fixed plan), respectively.

At a limited episode length, e.g., 1000 steps, goal-model shows a greater
advantage, which proves that goal-model can improve the execution efficiency
of the plan in embodied environments. In addition, compared to using vision-
language models such as CLIP (Radford et al., 2021) and MineCLIP (Fan et al.,
2022) as a goal model, horizon-predictive has the best performance due to
better estimation of the horizon information. The curve trend also
demonstrates that agents with Selector scale up under large amounts of goals
in an open-world environment.

Figure 5:  Ablation experiments on different Selector implementations. The
success rates of different selectors under varying numbers of parallel goals
and maximum episode lengths.

###  4.4 ObtainDiamond Challenge

Mining diamonds in the open-world game Minecraft, i.e. MT8 in Table 2, has
been a long-standing challenge for the community (Guss et al., 2019b). It is
challenging because mining diamonds from scratch in Minecraft involves
acquiring a sequence of difficult-to-obtain items that require complex
planning on goals like mining, inventory management, crafting with and without
a crafting table, tool use, smelting iron ingot in a furnace, and mining at
the lowest depths. We take the ObtainDiamond task as a bonus experiment to
show the capabilities of our zero-shot planner on complex tasks in embodied
environments. Previous methods’ success rates on this challenge further vouch
for its difficulty. Skrynnik et al. (2021); Patil et al. (2020) leverages
domain-secific reward functions and RL fine-tuning to achieve $\backsim$0.1%
success rate in 15 minutes of game play. VPT further boosts the success rate
to $2.5\%$ within 20 minutes of play through pre-training on collects
$\backsim$70k hours human demonstrations (Baker et al., 2022). DreamerV3 is
the first to collect diamonds by training from scratch with a success rate of
2%(Hafner et al., 2023).

Our DEPS manages to achieve on-par performance in this grand challenge; our
agent achieves a 0.59% success rate within 10 minutes of gameplay. Note our
method does not specifically fine-tune for this challenge. It is designed to
be multi-task in its nature. Furthermore, considering our planner operates
with demonstration prompts on a fixed Large Language Model, it can be
straightforwardly adapted to other open-ended environments with modifications.

##  5 Related Works

##### Task planning with LLMs

There have been some methods leveraging the large language model to generate
action plans for high-level tasks in embodied environments (Zeng et al., 2022;
Dasgupta et al., ). Huang et al. (2022a) decompose natural language commands
into sequences of executable actions by text completion and semantic
translation, while SayCan generates feasible plans for robots by jointly
decoding an LLM weighted by skill affordances from value functions (Brohan et
al., 2022). For better executing the plan in embodied environments, some
methods use an object detector describing the initial environment into the
language prompt to produce environment-suitable plans and adopt success
detectors to check that each step is executed successfully (Huang et al.,
2022b; Liang et al., 2022). (Singh et al., 2022) and (Liang et al., 2022) use
the pythonic-style prompt to produce more executable plans. However, all of
the above methods assume that the initial plan from the LLM is correct. When
there are bugs in the initial plan, it’s difficult for the agent to finish the
task successfully.

Interactive Planning with LLMs: Inner Monologue (Huang et al., 2022b) pilots
the front of interactive planning with LLMs, which introduces the feedback
(including success detection and scene description) to the planner. However,
we found it could still suffer from accumulative planning error, especially in
long-horizon open-world tasks. Rather, our “Describe, Explain, Plan and
Select” (DEPS) method can produce more reliable plans by leveraging chain-of-
thought thinking and explanation to locate the errors in previous plans.
Moreover, we also propose a goal Selector to further improve the efficiency of
the plan, thereby yielding much better performances. Readers are encouraged to
refer to the comparative results in Section 4.2 between DEPS and these prior
arts.

##### Agents in Minecraft

Some previous works have employed the hierarchical architecture to solve long-
horizon tasks in Minecraft (Oh et al., 2017; Mao et al., 2022; Lin et al.,
2021). Recently, based on the internet-scale corpus, Fan et al. (2022) pre-
trains a language-conditioned reward function and learns multi-task MineAgent.
Baker et al. (2022) collects a vast amount of human demonstrations to train a
behavior cloning agent. More recently, Hafner et al. (2023) utilized a learned
world model to distill a policy that can efficiently explore in Minecraft.
These efforts all focus on improving the low-level controller. Rather, the
planner in our architecture emphasizes applying domain knowledge to propose
and arrange the sub-goals. It significantly influences the complexity and
breadth of tasks that the agent can handle. Moreover, our planner is zero-
shot, making it possible to generalize to other long-horizon open worlds.

##  6 Conclusion

We investigate the problem of planning in open worlds. We identify two major
challenges unique to these environments: 1) long-term planning requires
precise and multi-step reasoning, and 2) planning efficiency could be
compromised since canonical planners do not take the agent’s proximity to
parallel goals/subtasks into consideration. We propose “Describe, Explain,
Plan and Select” (DEPS), an interactive approach based on Large Language
Models (LLMs) to tackle them both. Our experiments in the challenging
Minecraft domain verify the advantages of our approach over counterparts by
marking the milestone of robustly accomplishing 70+ Minecraft tasks and nearly
doubling the overall performances. DEPS also becomes the first planning-based
agent that can reach the diamond in this game.

## References

  * Alayrac et al. (2022) Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.  Flamingo: a visual language model for few-shot learning.  _arXiv preprint arXiv:2204.14198_ , 2022.
  * Bacon et al. (2017) Bacon, P.-L., Harb, J., and Precup, D.  The option-critic architecture.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 31, 2017.
  * Baker et al. (2022) Baker, B., Akkaya, I., Zhokhov, P., Huizinga, J., Tang, J., Ecoffet, A., Houghton, B., Sampedro, R., and Clune, J.  Video pretraining (vpt): Learning to act by watching unlabeled online videos.  _arXiv preprint arXiv:2206.11795_ , 2022.
  * Brohan et al. (2022) Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., et al.  Do as i can, not as i say: Grounding language in robotic affordances.  In _6th Annual Conference on Robot Learning_ , 2022.
  * Brown et al. (2020) Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Cai et al. (2023) Cai, S., Wang, Z., Ma, X., Liu, A., and Liang, Y.  Open-world multi-task control through goal-aware representation learning and adaptive horizon prediction.  _arXiv preprint arXiv:2301.10034_ , 2023.
  * Chen et al. (2021) Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. d. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ , 2021.
  * (8) Dasgupta, I., Kaeser-Chen, C., Marino, K., Ahuja, A., Babayan, S., Hill, F., and Fergus, R.  Collaborating with language models for embodied reasoning.  In _NeurIPS 2022 Foundation Models for Decision Making Workshop_.
  * Fan et al. (2022) Fan, L., Wang, G., Jiang, Y., Mandlekar, A., Yang, Y., Zhu, H., Tang, A., Huang, D.-A., Zhu, Y., and Anandkumar, A.  Minedojo: Building open-ended embodied agents with internet-scale knowledge.  _Advances in Neural Information Processing Systems Datasets and Benchmarks_ , 2022.
  * Guss et al. (2019a) Guss, W. H., Codel, C., Hofmann, K., Houghton, B., Kuno, N., Milani, S., Mohanty, S., Liebana, D. P., Salakhutdinov, R., Topin, N., et al.  Neurips 2019 competition: the minerl competition on sample efficient reinforcement learning using human priors.  _arXiv preprint arXiv:1904.10079_ , 2019a.
  * Guss et al. (2019b) Guss, W. H., Houghton, B., Topin, N., Wang, P., Codel, C., Veloso, M., and Salakhutdinov, R.  Minerl: A large-scale dataset of minecraft demonstrations.  _arXiv preprint arXiv:1907.13440_ , 2019b.
  * Guss et al. (2021) Guss, W. H., Castro, M. Y., Devlin, S., Houghton, B., Kuno, N. S., Loomis, C., Milani, S., Mohanty, S. P., Nakata, K., Salakhutdinov, R., Schulman, J., Shiroshita, S., Topin, N., Ummadisingu, A., and Vinyals, O.  The minerl 2020 competition on sample efficient reinforcement learning using human priors.  _arXiv: Learning_ , 2021.
  * Hafner et al. (2023) Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T.  Mastering diverse domains through world models.  _arXiv preprint arXiv:2301.04104_ , 2023.
  * Huang et al. (2022a) Huang, W., Abbeel, P., Pathak, D., and Mordatch, I.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  _ICML_ , 2022a.
  * Huang et al. (2022b) Huang, W., Xia, F., Xiao, T., Chan, H., Liang, J., Florence, P., Zeng, A., Tompson, J., Mordatch, I., Chebotar, Y., et al.  Inner monologue: Embodied reasoning through planning with language models.  _arXiv preprint arXiv:2207.05608_ , 2022b.
  * Johnson et al. (2016) Johnson, M., Hofmann, K., Hutton, T., and Bignell, D.  The malmo platform for artificial intelligence experimentation.  In _Ijcai_ , pp. 4246–4247. Citeseer, 2016.
  * Kanervisto et al. (2022) Kanervisto, A., Milani, S., Ramanauskas, K., Topin, N., Lin, Z., Li, J., Shi, J., Ye, D., Fu, Q., Yang, W., Hong, W., Huang, Z., Chen, H., Zeng, G., Lin, Y., Micheli, V., Alonso, E., Fleuret, F., Nikulin, A., Belousov, Y., Svidchenko, O., and Shpilman, A.  Minerl diamond 2021 competition: Overview, results, and lessons learned.  _neural information processing systems_ , 2022.
  * Liang et al. (2022) Liang, J., Huang, W., Xia, F., Xu, P., Hausman, K., Ichter, B., Florence, P., and Zeng, A.  Code as policies: Language model programs for embodied control.  _arXiv preprint arXiv:2209.07753_ , 2022.
  * Lin et al. (2021) Lin, Z., Li, J., Shi, J., Ye, D., Fu, Q., and Yang, W.  Juewu-mc: Playing minecraft with sample-efficient hierarchical reinforcement learning.  _arXiv preprint arXiv:2112.04907_ , 2021.
  * Mao et al. (2022) Mao, H., Wang, C., Hao, X., Mao, Y., Lu, Y., Wu, C., Hao, J., Li, D., and Tang, P.  Seihai: A sample-efficient hierarchical ai for the minerl competition.  In _Distributed Artificial Intelligence: Third International Conference, DAI 2021, Shanghai, China, December 17–18, 2021, Proceedings 3_ , pp. 38–51. Springer, 2022.
  * Min et al. (2022) Min, S., Lyu, X., Holtzman, A., Artetxe, M., Lewis, M., Hajishirzi, H., and Zettlemoyer, L.  Rethinking the role of demonstrations: What makes in-context learning work?  _arXiv preprint arXiv:2202.12837_ , 2022.
  * Oh et al. (2017) Oh, J., Singh, S., Lee, H., and Kohli, P.  Zero-shot task generalization with multi-task deep reinforcement learning.  In _International Conference on Machine Learning_ , pp. 2661–2670. PMLR, 2017.
  * Ouyang et al. (2022) Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al.  Training language models to follow instructions with human feedback.  _arXiv preprint arXiv:2203.02155_ , 2022.
  * Patil et al. (2020) Patil, V. P., Hofmarcher, M., Dinu, M.-C., Dorfer, M., Blies, P. M., Brandstetter, J., Arjona-Medina, J. A., and Hochreiter, S.  Align-rudder: Learning from few demonstrations by reward redistribution.  _ICML_ , 2020.
  * Radford et al. (2021) Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pp. 8748–8763. PMLR, 2021.
  * Reed et al. (2022) Reed, S., Zolna, K., Parisotto, E., Colmenarejo, S. G., Novikov, A., Barth-Maron, G., Gimenez, M., Sulsky, Y., Kay, J., Springenberg, J. T., et al.  A generalist agent.  _arXiv preprint arXiv:2205.06175_ , 2022.
  * Reimers & Gurevych (2019) Reimers, N. and Gurevych, I.  Sentence-bert: Sentence embeddings using siamese bert-networks.  _arXiv preprint arXiv:1908.10084_ , 2019.
  * Scao et al. (2022) Scao, T. L., Fan, A., Akiki, C., Pavlick, E., Ilić, S., Hesslow, D., Castagné, R., Luccioni, A. S., Yvon, F., Gallé, M., et al.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_ , 2022.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Singh et al. (2022) Singh, I., Blukis, V., Mousavian, A., Goyal, A., Xu, D., Tremblay, J., Fox, D., Thomason, J., and Garg, A.  Progprompt: Generating situated robot task plans using large language models.  _arXiv preprint arXiv:2209.11302_ , 2022.
  * Skrynnik et al. (2021) Skrynnik, A., Staroverov, A., Aitygulov, E., Aksenov, K., Davydov, V., and Panov, A. I.  Forgetful experience replay in hierarchical reinforcement learning from expert demonstrations.  _Knowledge-Based Systems_ , 218:106844, 2021.
  * Wei et al. (2022) Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D.  Chain of thought prompting elicits reasoning in large language models.  _36th Conference on Neural Information Processing Systems (NeurIPS 2022)_ , 2022.
  * Zeng et al. (2022) Zeng, A., Wong, A., Welker, S., Choromanski, K., Tombari, F., Purohit, A., Ryoo, M., Sindhwani, V., Lee, J., Vanhoucke, V., et al.  Socratic models: Composing zero-shot multimodal reasoning with language.  _arXiv preprint arXiv:2204.00598_ , 2022.
  * Zhang et al. (2022) Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al.  Opt: Open pre-trained transformer language models.  _arXiv preprint arXiv:2205.01068_ , 2022.

##  Appendix A Task Details

To fully validate the multitask planning and execution capability of our
agent, we distill 71 tasks from the 1572 programmatic tasks of Minedojo (Fan
et al., 2022) and select them as the set of evaluation tasks. Minedojo exists
some programmatic tasks sharing the same object item given different
conditions (e.g., obtain wool given shear or obtain wool given nothing).
Minedojo expands the richness of the same tasks (sharing the same Minecraft
item as an object) by giving different initial conditions (e.g., obtain wool
given shears or obtain wool given nothing). We keep only the 71 hardest
conditions (i.e. given nothing) as tasks, and we call the set of tasks
TASK101.

We list all task names, objects, and their reasoning steps for planning in
Table 4. Object item is used as the basis for the successful completion of the
task. These objects cannot be obtained directly from the environment, and
usually require multiple goals (i.e., reasoning steps) to be constructed. Here
we only consider the number of required goal types, and multiple identical
goals are unified into 1 reasoning step. Note that the reasoning steps for
each task are not fixed, and as the initial state of the agent and the biome
is in change, more reasoning steps may be required to complete it, we only
report the most basic case here.

Table 4: Full task list of every meta task in Minecraft TASK101.

Meta-Task | Task |  | Reasoning
---
Steps
Object | Task |  | Reasoning
---
Steps
Object
MT1 Basic | CraftPlanks | 2 | planks | CraftOakStairs | 4 | oak_stairs
CraftSticks | 3 | stick | CraftSign | 5 | sign
CraftWoodenSlab | 4 | wooden_slab | CraftFence | 5 | fence
CraftWoodenPressurePlate | 3 | wooden_pressure_plate | CraftFenceGate | 5 | fence_gate
CraftBowl | 4 | bowl | CraftBoat | 4 | boat
CraftWoodenButton | 3 | wooden_button | CraftTrapdoor | 4 | trapdoor
CraftChest | 4 | chest | CraftWoodenDoor | 4 | wooden_door
MT2 Tool (Simple) | CraftCraftingTable | 3 | crafting_table | CraftFurnace | 7 | furnace
CraftWoodenPickaxe | 5 | wooden_pickaxe | CraftStonePickaxe | 7 | stone_pickaxe
CraftWoodenAxe | 5 | wooden_axe | CraftStoneAxe | 7 | stone_axe
CraftWoodenHoe | 5 | wooden_hoe | CraftStoneHoe | 7 | stone_hoe
CraftWoodenSword | 5 | wooden_sword | CraftStoneShovel | 7 | stone_shovel
CraftWoodenShovel | 5 | wooden_shovel | CraftStoneSword | 7 | stone_sword
MT3 Hunt & Food | CraftBed | 5 | bed | CookPorkchop | 9 | cooked_porkchop
CraftPainting | 6 | painting | CookBeef | 9 | cooked_beef
CraftCarpet | 5 | carpet | CookMutton | 9 | cooked_mutton
CraftItemFrame | 6 | item_frame |  |  |
MT4 Dig-Down | CraftStoneStairs | 7 | stone_stairs | CraftStoneBrick | 9 | stone_brick
CraftStoneSlab | 7 | stone_slab | CraftTorch | 9 | torch
CraftArmorStand | 10 | armor_stand | ObtainCoal | 8 | coal
CraftCobblestoneWall | 7 | cobblestone_wall | CraftStoneBrickStairs | 10 | stonebrick_stairs
CraftQuartzBlock | 10 | quartz_block | CraftStonePressurePlate | 9 | stone_pressure_plate
CraftStoneBrick | 9 | stonebrick | CraftStoneButton | 7 | stone_button
SmeltStone | 9 | stone | CraftLever | 7 | level
MT5 Equipment | EquipLeatherBoots | 5 | leather_boots | EquipIronChestplate | 11 | iron_chestplate
EquipLeatherChestplate | 5 | leather_chestplate | EquipIronLeggings | 11 | iron_leggings
EquipLeatherHelmet | 5 | leather_helmet | EquipIronHelmet | 11 | iron_helmet
EquipLeatherLeggings | 5 | leather_leggings | EquipIronBoots | 11 | iron_boots
EquipShield | 11 | shield |  |  |
MT6 Tool (Complex) | CraftBucket | 11 | bucket | CraftIronHoe | 11 | iron_hoe
CraftShears | 11 | shears | CraftIronShovel | 11 | iron_shovel
CraftIronPickaxe | 11 | iron_pickaxe | CraftIronSword | 11 | iron_sword
CraftIronAxe | 11 | iron_axe |  |  |
MT7 Iron-Stage | CraftIronBars | 11 | iron_bars | CraftChestMinecart | 13 | chest_minecart
CraftIronNugget | 11 | iron_nugget | CraftIronDoor | 11 | iron_door
CraftMinecart | 11 | minecart | CraftIronTrapdoor | 11 | iron _trapdoor
CraftHopper | 12 | hopper | CraftTripwireHook | 11 | tripwire_hook
CraftHopperMinecart | 14 | hopper_minecart | CraftHeavyWeightedPressurePlate | 11 | heavy_weighted_pressure_plate
CraftFurnaceMinecart | 12 | furnace_minecart | CraftRail | 11 | rail
CraftCauldron | 11 | cauldron |  |  |
| Challenge
---
MT8
MineDiamond | 12 | diamond |  |  |

As shown in Figure 5, for each task, a relaxed (longer) maximum episode steps
will increase the success rate of the task. To fully test the efficiency of
our method, we set an upper limit on the episode length for each task. Since
different tasks have different difficulty levels, we double the average
completion time of human players for different meta-tasks as the upper limit
of the episode. The play time are computed as corresponding maximum steps
(i.e., Max. Steps in Table 1) of episode length at 20Hz.

##  Appendix B Goal Setting and Controller Details

As the name implies, tasks in Minecraft are usually related to mine and craft
goals. Mine goals require the agent to collect raw materials from the
environment using the appropriate tools. Craft goals ask the agent to
synthesize using existing materials. Any raw material used requires the agent
to collect through suitable tools (e.g., diamonds can only be collected by an
iron pickaxe or a better pickaxe). So a task usually requires dozens of step-
by-step mine and craft goals, as the Reasoning Steps in Table 4. Note that the
successful execution of a task need to satisfy certain exact numerical
constraints due to the presence of strict generation recipes in the
environment (e.g., a log can craft 4 planks, so harvesting 6 planks require at
least 2 logs). When the number of materials collected is not enough, the goal
cannot be completed successfully. When more materials are collected than
actually needed, the execution success rate of the task could also be reduced
because the plan can not be finished under the maximum action steps.

We designed the agent’s skill space based on these goals, as shown in Table 5,
with a total of 262 goals. Every goal is designed with an objective item
(e.g., 1 minecraft:cobblestone for skill “Mine 1 cobblestone with pickaxe”),
which is used to evaluate the achievement of the goal. The skill, as a goal-
conditioned policy $\pi(a|s,g)$ for decision-making, maps the current state
$s$ and goal $g$ to action $a$. The goal is specified as natural language
instructions here, which is similar to (Brohan et al., 2022).

When training the controller, we adopt the observation space provided by
MineDoJo (Fan et al., 2022), which includes an RGB camera view, yaw/pitch
angle, GPS location, and the type of $3\times 3$ blocks surrounding the agent.
We discretize the original multi-discrete action space provided by MineDojo
into 42 discrete actions. We use the proposed imitation learning method
proposed by Cai et al. (2023) in training. To be specific, a modified goal-
sensitive Impala CNN is used as the backbone network. The success rate under a
fixed episode length of every skill is listed in Table 5.

Table 5: The success rate of different skill/goal with imitation learning controller. ID | Skill Description | Success Rate | Episode Length
---|---|---|---
0 | Mine 1 oak wood | 0.39 | 600
1 | Mine birch wood | 0.29 | 600
2 | Mine 1 cobblestone with pickaxe | 0.95 | 600
3 | Mine 1 stone with pickaxe | 0.70 | 600
4 | Mine 1 seed | 0.18 | 600
5 | Mine 1 leaves with shears | 0.68 | 600
6 | Mine 1 dirt | 0.54 | 600
7 | Mine 1 iron ore with stone pickaxe | 0.40 | 3000
8 | Mine 3 iron ore with stone pickaxe | 0.16 | 3000
9 | Mine 1 diamond with iron pickaxe | 0.35 | 12000
10 | Mine 1 diamond with stone pickaxe | 0.00 | 12000
11 | Kill 1 sheep with axe | 0.44 | 600
12 | Kill 1 cow with axe | 0.60 | 600
13 | Kill 1 chicken with axe | 0.46 | 600
14 | Kill 1 pig with axe | 0.49 | 600
15 | Kill 1 llama | 0.50 | 600
16 | Equip tool on mainhand | 1.00 | 600
17-261 | Craft w/o crafting_table | 1.00 | 600
Craft w/ crafting_table | 0.90 | 600
Smelt w/ furnace | 0.80 | 600

##  Appendix C Other Key Components in our Agent

##### Goal Mapping

We need to map the plan expressed in free-form language to the pre-defined
controller skills set. We first use regular expression to parse the language
plan. For the goals not following pre-defined code expression, we calculate
its semantic distance to the skills by cosine similarity with pre-trained
Sentence-Bert model (Reimers & Gurevych, 2019) and select the most similar
skill as the corresponding goal. All executable goals are listed in Appendix
B.

##### Prompt Generator

The generalization of the LLM to different tasks relies on well-designed
prompts and related demonstrations (Min et al., 2022). Given an instruction
command (e.g., ObtainDiamond) as task $T$, a prompt generator (ProG) will
translate $T$ into prompt text. We also added two DEP examples in the prompt
as demonstrations to make the LLM output familiar to the chain-of-thought
thinking and structural output. We also design a chain-of-thought code-
comments-type planning prompt to better demonstrate the capabilities of LLM.
All messages are modified to suitable prompts through the prompt-generator
before being input to LLM, including task $T$ and description $d_{t}$. The
full prompt sentences and interaction logs are listed in Appendix H.

##  Appendix D Comparison to other LLM-based Planner

The architectures of the different LLM-based planners are illustrated in
Figure 6. Where (a) is Zero-shot planner (Huang et al., 2022a), which directly
executes the plan from LLM sequentially, without considering the state of the
environment with the feedback from the agent performing the task. (b)
describes the information in the environment into LLM via scene descriptor and
success detector, and re-plan according to the feedback. We further rethink
and explain the feedback of previous plans explicitly with the descriptor and
explainer. The LLM-based planner will re-plan the task according to the
explanation, as demonstrated in Figure 6(c). In addition, the goal Selector
further improves the executability of the LLM plan.

Figure 6:  Comparison of LLM-based planner architecture. (a), (b), (c)
represents planner of (Huang et al., 2022a), (Huang et al., 2022b) and ours,
respectively.

##  Appendix E Limitations

Albeit the impressive results of our approach, we believe there are at least
two major limitations within our approach. First of all, our framework relies
on privately-held LLMs like GPT-3 and ChatGPT, which makes it less accessible
to those who cannot afford or access the service. However, we’re fully
committed to ensuring a more democratized method and will explore using open-
sourced models including OPT (Zhang et al., 2022) and BLOOM (Scao et al.,
2022). Another issue is the explicit planning in our system. Although it
brings us superior performances over the baselines, the planning bottleneck
can also prevent our model from being further scaled up. A more appealing
approach will be amortizing the planning within an end-to-end trainable goal-
conditioned policy, which is worth exploring next.

##  Appendix F ObtainDiamond Experiment

We run ObtainDiamond experiment 671 times and report the success rate of
milestone items in Figure 7.

The milestone items in the order are:

  1. 1.

minecraft:log

  2. 2.

minecraft:planks

  3. 3.

minecraft:crafting_table

  4. 4.

minecraft:stick

  5. 5.

minecraft:wooden_pickaxe

  6. 6.

minecraft:cobblestone

  7. 7.

minecraft:stone_pickaxe

  8. 8.

minecraft:furnace

  9. 9.

minecraft:iron_ore

  10. 10.

minecraft:iron_ingot

  11. 11.

minecraft:iron_pickaxe

  12. 12.

minecraft:diamond

Figure 7:  Success rate of milestone items for mining diamond.

##  Appendix G Success Rates of All 71 Tasks in Task101

We report the complete and detailed success rate table of all tasks for
different methods in Table 6, including Zero-shot Planner (Huang et al.,
2022a), ProgPrompt (Singh et al., 2022), Chain-of-Thought (Wei et al., 2022),
Inner Monologue (Huang et al., 2022b), Code as Policies (Liang et al., 2022),
and proposed methods (i.e., DEP w/o Selector, and DEPS).

All tasks are executed for 30 times across different world seeds, given the
same initial conditions. The birth positions of the world are random according
to the seed. The average success rates are listed in Table 6. Our approach is
state-of-the-art on almost all tasks, especially on difficult tasks that
require long reasoning steps.

Table 6: Success rate comparison of various methods on Minecraft TASK101.

Meta-Task | Task Object | Zero-shot Planner | ProgPrompt | Chain of Thought | Inner Monologue | Code as Policies | DEP w/o Selector | DEPS
---|---|---|---|---|---|---|---|---
| planks | 55.56 | 56.39 | 83.30 | 83.30 | 83.30 | 83.30 | 83.30
| stick | 0.00 | 56.39 | 83.30 | 83.30 | 83.30 | 83.30 | 86.70
| wooden_slab | 27.74 | 27.82 | 55.56 | 83.30 | 83.30 | 83.30 | 76.70
| wooden_button | 24.41 | 49.62 | 73.30 | 73.30 | 73.30 | 73.30 | 96.70
| wooden_pressure_plate | 80.00 | 80.00 | 53.36 | 80.00 | 80.00 | 80.00 | 86.70
| chest | 0.00 | 25.54 | 0.00 | 0.00 | 51.16 | 76.70 | 76.70
| oak_stairs | 18.88 | 38.39 | 37.82 | 18.88 | 37.82 | 56.70 | 60.00
| sign | 21.08 | 0.00 | 42.22 | 0.00 | 42.22 | 63.30 | 86.70
| fence | 21.08 | 21.08 | 0.00 | 21.08 | 42.22 | 63.30 | 80.00
| fence_gate | 63.16 | 0.00 | 62.23 | 62.23 | 62.23 | 93.30 | 73.30
| boat | 0.00 | 0.00 | 0.00 | 27.74 | 55.56 | 83.30 | 73.30
| trapdoor | 27.74 | 27.74 | 27.74 | 55.56 | 55.56 | 83.30 | 76.70
| bowl | 0.00 | 23.31 | 0.00 | 23.31 | 46.69 | 70.00 | 80.00
Basic MT1 | wooden_door | 22.21 | 22.21 | 44.49 | 44.49 | 44.49 | 66.70 | 80.00
| crafting_table | 70.00 | 23.31 | 70.00 | 70.00 | 70.00 | 70.00 | 90.00
| wooden_pickaxe | 80.00 | 80.00 | 80.00 | 80.00 | 80.00 | 80.00 | 80.00
| wooden_axe | 46.69 | 46.69 | 70.00 | 70.00 | 70.00 | 70.00 | 96.70
| wooden_hoe | 86.70 | 57.83 | 86.70 | 28.87 | 86.70 | 86.70 | 86.70
| wooden_sword | 83.30 | 83.30 | 83.30 | 83.30 | 83.30 | 83.30 | 86.70
| wooden_shovel | 76.70 | 76.70 | 76.70 | 76.70 | 76.70 | 76.70 | 90.00
| furnace | 19.98 | 19.98 | 0.00 | 40.02 | 40.02 | 60.00 | 66.70
| stone_pickaxe | 17.75 | 17.75 | 35.55 | 35.55 | 53.30 | 53.30 | 73.30
| stone_axe | 0.00 | 0.00 | 31.15 | 31.15 | 31.15 | 46.70 | 70.00
| stone_hoe | 18.88 | 18.88 | 37.82 | 37.82 | 56.70 | 56.70 | 66.70
| stone_shovel | 56.70 | 56.70 | 37.82 | 37.82 | 37.82 | 56.70 | 66.70
Tool(Simple) MT2 | stone_sword | 17.82 | 0.00 | 53.50 | 53.50 | 35.68 | 53.50 | 80.00
| bed | 15.54 | 23.30 | 23.30 | 7.76 | 7.76 | 23.30 | 43.40
| painting | 35.55 | 0.00 | 0.00 | 17.75 | 17.75 | 53.30 | 86.70
| carpet | 0.00 | 33.30 | 0.00 | 0.00 | 11.09 | 33.30 | 43.40
| item_frame | 24.41 | 48.89 | 24.41 | 0.00 | 24.41 | 73.30 | 83.30
| cooked_porkchop | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 40.00 | 50.00
| cooked_beef | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 53.30 | 63.30
Hunt and Food MT3 | cooked_mutton | 0.00 | 14.42 | 0.00 | 0.00 | 0.00 | 43.30 | 66.70
| stone_stairs | 18.88 | 23.11 | 18.88 | 37.82 | 18.88 | 56.70 | 66.70
| stone_slab | 16.65 | 50.00 | 0.00 | 16.65 | 33.35 | 50.00 | 73.30
| cobblestone_wall | 14.45 | 14.45 | 14.45 | 14.45 | 43.40 | 43.40 | 63.30
| lever | 0.00 | 0.00 | 0.00 | 46.69 | 46.69 | 70.00 | 83.30
| coal | 0.00 | 16.70 | 0.00 | 5.56 | 0.00 | 16.70 | 20.00
Dig-down MT4 | torch | 0.00 | 7.76 | 0.00 | 7.76 | 0.00 | 23.30 | 13.30
| leather_boots | 0.00 | 12.22 | 0.00 | 12.22 | 12.22 | 36.70 | 60.00
| leather_chestplate | 0.00 | 7.76 | 15.54 | 0.00 | 7.76 | 23.30 | 36.60
| leather_helmet | 17.81 | 8.89 | 0.00 | 8.89 | 0.00 | 26.70 | 70.00
| leather_leggings | 0.00 | 0.00 | 0.00 | 20.01 | 0.00 | 30.00 | 56.70
| iron_chestplate | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
| iron_leggings | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 3.30 | 3.30
| iron_helmet | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 3.30
| iron_boots | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 6.70 | 20.00
Equipment MT5 | shield | 0.00 | 0.00 | 0.00 | 0.00 | 5.56 | 16.70 | 13.30
| bucket | 0.00 | 3.33 | 0.00 | 0.00 | 3.33 | 10.00 | 6.70
| shears | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 10.00 | 30.00
| iron_pickaxe | 0.00 | 0.00 | 0.00 | 4.47 | 0.00 | 6.70 | 10.00
| iron_axe | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 16.60
| iron_hoe | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 3.30 | 13.30
| iron_shovel | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 6.70 | 13.30
Tool Complex MT6 | iron_sword | 0.00 | 0.00 | 0.00 | 0.00 | 1.10 | 3.30 | 6.70
| iron_bars | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 6.70
| iron_nugget | 0.00 | 7.76 | 0.00 | 7.76 | 7.76 | 23.30 | 40.00
| minecart | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 3.30 | 10.00
| hopper | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 3.30 | 6.70
| hopper_minecart | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00
| furnace_minecart | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 3.30 | 3.30
| chest_minecart | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 3.30 | 3.30
| iron_door | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 3.30
| iron_trapdoor | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 3.30 | 16.70
| tripwire_hook | 0.00 | 0.00 | 0.00 | 5.56 | 0.00 | 16.70 | 30.00
| heavy_weighted_pressure_plate | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 3.30 | 33.30
| rail | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 6.70
Iron-Stage MT7 | cauldron | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 3.30
| MT8
---
Challenge
diamond | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.50 | 0.59

##  Appendix H Full interactions with LLM

Here we list the full interactive prompting process with LLM of our agent to
finish the task “ObtainDiamond”. The text about “CraftStoneSword” is the
demonstration in the prompt. The outputs of the Explainer and Planner are from
the language model. The outputs from Descriptor and Replanner are captured
from Descriptor and Prompt Generator, respectively.

Listing 1: Full interaction process of task ObtainDiamond

⬇

Agent: You are a helper agent in Minecraft. You need to generate the sequences
of goals (actions) for a certain task in Minecraft.

Planner: OK.

======================================

Agent: How to craft stone_sword?

Planner: The code for crafting stone_sword is as bellows:

def craft_stone_sword(inventory = {}):

mine({’log’:3}, null); # action 1: mine 3 log without tool

craft({’planks’:12}, {’log’:3}, null); # action 2: craft 12 planks from 3 log

craft({’stick’:4}, {’planks’:2}, null); # action 3: craft 4 stick from 2
planks

mine({’cobblestone’:2}, null); # action 4: mine 2 cobblestone without tool

craft({’stone_sword’:1}, {’cobblestone’:2, ’stick’:1}, ’crafting_table’); #
action 5: craft 1 stone_sword from 2 cobblestone and 1 stick, on
crafting_table

return ’stone_sword’

Descriptor: I succeed on step 1, 2, 3.

Descriptor: I locate in plains biome. My inventory now has 10 planks, 4 stick.

Descriptor: I fail on step 4 "mine({’cobblestone’:2}, null);".

Explainer: Because mine cobblestone need to use the tool wooden_pickaxe.

Replanner: Please fix above errors and replan the task "How to craft 1
stone_sword".

Planner: The code for crafting stone_sword is as bellows:

def craft_stone_sword(inventory = {}):

mine({’log’:3}, null); # action 1: mine 3 log without tool

craft({’planks’:12}, {’log’:3}, null); # action 2: craft 12 planks from 3 log
without tool

craft({’stick’:4}, {’planks’:2}, null); # action 3: craft 4 stick from 2
planks without tool

craft({’wooden_pickaxe’:1}, {’planks’:3, ’stick’:2}, ’crafting_table’); #
action 4: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table

mine({’cobblestone’:2}, ’wooden_pickaxe’); # action 5: mine 2 cobblestone with
wooden_pickaxe

craft({’stone_sword’:1}, {’cobblestone’:2, ’stick’:1}, ’crafting_table’); #
action 6: craft 1 stone_sword from 2 cobblestone and 1 stick, on
crafting_table

return ’stone_sword’

Descriptor: I locate in plains biome. My inventory now has 10 planks, 4 stick.

Descriptor: I fail on step 4 "craft({’wooden_pickaxe’:1}, {’planks’:3,
’stick’:2}, ’crafting_table’);"

Explainer: because the action needs to use the tool crafting_table, but I do
not have it.

Replanner: Please fix above errors and replan the task "How to craft 1
stone_sword".

Planner: The code for crafting stone_sword is as bellows:

def craft_1_stone_sword(inventory = {}):

mine({’log’:3}, null); # action 1: mine 3 log without tool

craft({’planks’:12}, {’log’:3}, null); # action 2: craft 12 planks from 3 log

craft({’stick’:4}, {’planks’:2}, null); # action 3: craft 4 stick from 2
planks

craft({’crafting_table’:1}, {’planks’:4}, null); # action 4: craft 1
crafting_table from 4 planks

craft({’wooden_pickaxe’:1}, {’planks’:3, ’stick’:2}, ’crafting_table’); #
action 5: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table

mine({’cobblestone’:2}, ’wooden_pickaxe’); # action 6: mine 2 cobblestone with
wooden_pickaxe

craft({’stone_sword’:1}, {’cobblestone’:2, ’stick’:2}, ’crafting_table’); #
action 7: craft 1 stone_sword from 2 cobblestone and 1 stick, on
crafting_table

return ’stone_sword’

Descriptor: I succeed on step 4, 5, 6, 7.

Descriptor: Good. I finish the task.

Planner: OK.

======================================

Agent: How to mine 1 diamond?

Planner: The code for mining diamond is as bellows:

def mine_1_diamond(inventory = {}):

mine({’log’:3}, null); # action 1: mine 3 log without tool

craft({’planks’:12}, {’log’:3}, null); # action 2: craft 12 planks from 3 log

craft({’stick’:4}, {’planks’:2}, null); # action 3: craft 4 stick from 2
planks

craft({’crafting_table’:1}, {’planks’:4}, null); # action 4: craft 1
crafting_table from 4 planks

craft({’wooden_pickaxe’:1}, {’planks’:3, ’stick’:2}, ’crafting_table’); #
action 5: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table

mine({’cobblestone’:2}, ’wooden_pickaxe’); # action 6: mine 2 cobblestone with
wooden_pickaxe

craft({’stone_pickaxe’:1}, {’cobblestone’:3, ’stick’:2}, ’crafting_table’); #
action 7: craft 1 stone_pickaxe from 3 cobblestone and 2 stick, on
crafting_table

mine({’diamond’:1}, ’stone_pickaxe’); # action 8: mine 1 diamond with
stone_pickaxe

return ’diamond’

Descriptor: I succeed on step 1, 2, 3, 4, 5, 6.

Descriptor: I locate in plains biome. My inventory now has 7 planks, 2 stick,
1 crafting_table, 1 wooden_pickaxe, 2 cobblestone, 5 dirt.

Descriptor: I fail on step 7 "craft({’stone_pickaxe’:1}, {’cobblestone’:3,
’stick’:2}, ’crafting_table’);"

Explainer: because the action needs 3 cobblestone, but I only have 2
cobblestone.

Replanner: Please fix above errors and replan the task "How to mine 1
diamond".

Planner: The code for mining diamond is as bellows:

def mine_1_diamond(inventory = {}):

mine({’log’:3}, null); # action 1: mine 3 log without tool

craft({’planks’:12}, {’log’:3}, null); # action 2: craft 12 planks from 3 log

craft({’stick’:4}, {’planks’:2}, null); # action 3: craft 4 stick from 2
planks

craft({’crafting_table’:1}, {’planks’:4}, null); # action 4: craft 1
crafting_table from 4 planks

craft({’wooden_pickaxe’:1}, {’planks’:3, ’stick’:2}, ’crafting_table’); #
action 5: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table

mine({’cobblestone’:2}, ’wooden_pickaxe’); # action 6: mine 2 cobblestone with
wooden_pickaxe

mine({’cobblestone’:1}, ’wooden_pickaxe’); # action 7: mine 1 cobblestone with
wooden_pickaxe

craft({’stone_pickaxe’:1}, {’cobblestone’:3, ’stick’:2}, ’crafting_table’); #
action 8: craft 1 stone_pickaxe from 3 cobblestone and 2 stick, on
crafting_table

mine({’diamond’:1}, ’stone_pickaxe’); # action 9: mine 1 diamond with
stone_pickaxe

return ’diamond’

Descriptor: I succeed on step 7, 8.

Descriptor: I locate in plains biome. My inventory now has 7 planks, 1
crafting_table, 1 wooden_pickaxe, 1 cobblestone, 5 dirt, 1 stone_pickaxe.

Descriptor: I fail on step 9 "mine({’diamond’:1}, ’stone_pickaxe’);"

Explainer: because mine diamond need to use the tool iron_pickaxe.

Replanner: Please fix above errors and replan the task "How to mine 1
diamond".

Planner: The code for mining diamond is as bellows:

def mine_1_diamond(inventory = {}):

mine({’log’:3}, null); # action 1: mine 3 log without tool

craft({’planks’:12}, {’log’:3}, null); # action 2: craft 12 planks from 3 log

craft({’stick’:4}, {’planks’:2}, null); # action 3: craft 4 stick from 2
planks

craft({’crafting_table’:1}, {’planks’:4}, null); # action 4: craft 1
crafting_table from 4 planks

craft({’wooden_pickaxe’:1}, {’planks’:3, ’stick’:2}, ’crafting_table’); #
action 5: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table

mine({’cobblestone’:2}, ’wooden_pickaxe’); # action 6: mine 2 cobblestone with
wooden_pickaxe

mine({’cobblestone’:1}, ’wooden_pickaxe’); # action 7: mine 1 cobblestone with
wooden_pickaxe

craft({’stone_pickaxe’:1}, {’cobblestone’:3, ’stick’:2}, ’crafting_table’); #
action 8: craft 1 stone_pickaxe from 3 cobblestone and 2 stick, on
crafting_table

mine({’iron_ore’:1}, ’stone_pickaxe’); # action 9: mine 1 iron_ore with
stone_pickaxe

craft({’iron_ingot’:1}, {’iron_ore’:1}, ’furnace’); # action 10: craft 1
iron_ingot from 1 iron_ore, on furnace

craft({’iron_pickaxe’:1}, {’iron_ingot’:3, ’stick’:2}, ’crafting_table’); #
action 11: craft 1 iron_pickaxe from 3 iron_ingot and 2 stick, on
crafting_table

mine({’diamond’:1}, ’iron_pickaxe’); # action 12: mine 1 diamond with
iron_pickaxe

return ’diamond’

Descriptor: I success on step 9.

Descriptor: I locate in plains biome. My inventory now has 7 planks, 1
crafting_table, 1 wooden_pickaxe, 6 cobblestone, 1 iron_ore, 20 dirt, 64
stone.

Descriptor: I fail on step 10 "craft({’iron_ingot’:1}, {’iron_ore’:1},
’furnace’);"

Explainer: because the action needs to use the tool furnace, but I do not have
it.

Replanner: Please fix above errors and replan the task "How to mine 1
diamond".

Planner: The code for mining diamond is as bellows:

def mine_1_diamond(inventory = {}):

mine({’log’:3}, null); # action 1: mine 3 log without tool

craft({’planks’:12}, {’log’:3}, null); # action 2: craft 12 planks from 3 log

craft({’stick’:4}, {’planks’:2}, null); # action 3: craft 4 stick from 2
planks

craft({’crafting_table’:1}, {’planks’:4}, null); # action 4: craft 1
crafting_table from 4 planks

craft({’wooden_pickaxe’:1}, {’planks’:3, ’stick’:2}, ’crafting_table’); #
action 5: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table

mine({’cobblestone’:2}, ’wooden_pickaxe’); # action 6: mine 2 cobblestone with
wooden_pickaxe

mine({’cobblestone’:1}, ’wooden_pickaxe’); # action 7: mine 1 cobblestone with
wooden_pickaxe

craft({’stone_pickaxe’:1}, {’cobblestone’:3, ’stick’:2}, ’crafting_table’); #
action 8: craft 1 stone_pickaxe from 3 cobblestone and 2 stick, on
crafting_table

mine({’iron_ore’:1}, ’stone_pickaxe’); # action 9: mine 1 iron_ore with
stone_pickaxe

craft({’furnace’:1}, {’cobblestone’:8}, null); # action 10: craft 1 furnace
from 8 cobblestone

craft({’iron_ingot’:1}, {’iron_ore’:1}, ’furnace’); # action 11: craft 1
iron_ingot from 1 iron_ore, on furnace

craft({’iron_pickaxe’:1}, {’iron_ingot’:3, ’stick’:2}, ’crafting_table’); #
action 12: craft 1 iron_pickaxe from 3 iron_ingot and 2 stick, on
crafting_table

mine({’diamond’:1}, ’iron_pickaxe’); # action 13: mine 1 diamond with
iron_pickaxe

return ’diamond’

Descriptor: I locate in plains biome. My inventory now has 7 planks, 1
crafting_table, 1 wooden_pickaxe, 6 cobblestone, 1 iron_ore, 20 dirt, 64
stone.

Descriptor: I fail on step 10 "craft({’furnace’:1}, {’cobblestone’:8}, null);"

Explainer: because I do not have 8 cobblestone.

Replanner: Please fix above errors and replan the task "How to mine 1
diamond".

Planner: The code for mining diamond is as bellows:

def mine_1_diamond(inventory = {}):

mine({’log’:3}, null); # action 1: mine 3 log without tool

craft({’planks’:12}, {’log’:3}, null); # action 2: craft 12 planks from 3 log

craft({’stick’:4}, {’planks’:2}, null); # action 3: craft 4 stick from 2
planks

craft({’crafting_table’:1}, {’planks’:4}, null); # action 4: craft 1
crafting_table from 4 planks

craft({’wooden_pickaxe’:1}, {’planks’:3, ’stick’:2}, ’crafting_table’); #
action 5: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table

mine({’cobblestone’:2}, ’wooden_pickaxe’); # action 6: mine 2 cobblestone with
wooden_pickaxe

mine({’cobblestone’:1}, ’wooden_pickaxe’); # action 7: mine 1 cobblestone with
wooden_pickaxe

craft({’stone_pickaxe’:1}, {’cobblestone’:3, ’stick’:2}, ’crafting_table’); #
action 8: craft 1 stone_pickaxe from 3 cobblestone and 2 stick, on
crafting_table

mine({’iron_ore’:1}, ’stone_pickaxe’); # action 9: mine 1 iron_ore with
stone_pickaxe

mine({’cobblestone’:2}, ’stone_pickaxe’); # action 10: mine 2 cobblestone with
stone_pickaxe

craft({’furnace’:1}, {’cobblestone’:8}, null); # action 11: craft 1 furnace
from 8 cobblestone

craft({’iron_ingot’:1}, {’iron_ore’:1}, ’furnace’); # action 12: craft 1
iron_ingot from 1 iron_ore, on furnace

craft({’iron_pickaxe’:1}, {’iron_ingot’:3, ’stick’:2}, ’crafting_table’); #
action 13: craft 1 iron_pickaxe from 3 iron_ingot and 2 stick, on
crafting_table

mine({’diamond’:1}, ’iron_pickaxe’); # action 14: mine 1 diamond with
iron_pickaxe

return ’diamond’

Descriptor: I succeed on step 10.

Descriptor: I locate in plains biome. My inventory now has 7 planks, 1
crafting_table, 1 wooden_pickaxe, 1 stone_pickaxe, 8 cobblestone, 1 iron_ore,
20 dirt, 64 stone.

Descriptor: I fail on step 11 "craft({’furnace’:1}, {’cobblestone’:8}, null);"

Explainer: because craft({’furnace’:1}, {’cobblestone’:8}, null) needs to use
the tool crafting_table, but I do not have it.}

Replanner: Please fix above errors and replan the task "How to mine 1
diamond".

Planner: The code for mining diamond is as bellows:

def mine_1_diamond(inventory = {}):

mine({’log’:3}, null); # action 1: mine 3 log without tool

craft({’planks’:12}, {’log’:3}, null); # action 2: craft 12 planks from 3 log

craft({’stick’:4}, {’planks’:2}, null); # action 3: craft 4 stick from 2
planks

craft({’crafting_table’:1}, {’planks’:4}, null); # action 4: craft 1
crafting_table from 4 planks

craft({’wooden_pickaxe’:1}, {’planks’:3, ’stick’:2}, ’crafting_table’); #
action 5: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table

mine({’cobblestone’:2}, ’wooden_pickaxe’); # action 6: mine 2 cobblestone with
wooden_pickaxe

mine({’cobblestone’:1}, ’wooden_pickaxe’); # action 7: mine 1 cobblestone with
wooden_pickaxe

craft({’stone_pickaxe’:1}, {’cobblestone’:3, ’stick’:2}, ’crafting_table’); #
action 8: craft 1 stone_pickaxe from 3 cobblestone and 2 stick, on
crafting_table

mine({’iron_ore’:1}, ’stone_pickaxe’); # action 9: mine 1 iron_ore with
stone_pickaxe

mine({’cobblestone’:2}, ’stone_pickaxe’); # action 10: mine 2 cobblestone with
stone_pickaxe

craft({’furnace’:1}, {’cobblestone’:8}, ’crafting_table’); # action 11: craft
1 furnace from 8 cobblestone, on crafting_table

craft({’iron_ingot’:1}, {’iron_ore’:1}, ’furnace’); # action 12: craft 1
iron_ingot from 1 iron_ore, on furnace

craft({’iron_pickaxe’:1}, {’iron_ingot’:3, ’stick’:2}, ’crafting_table’); #
action 13: craft 1 iron_pickaxe from 3 iron_ingot and 2 stick, on
crafting_table

mine({’diamond’:1}, ’iron_pickaxe’); # action 14: mine 1 diamond with
iron_pickaxe

return ’diamond’

Descriptor: I succeed on step 11, 12.

Descriptor: I locate in plains biome. My inventory now has 6 planks, 1
crafting_table, 1 wooden_pickaxe, 1 stone_pickaxe, 1 furnace, 1 iron_ingot, 20
dirt, 64 stone.

Descriptor: I fail on step 13 "craft({’iron_pickaxe’:1}, {’iron_ingot’:3,
’stick’:2}, ’crafting_table’);"

Explainer: because ecraft({’iron_pickaxe’:1}, {’iron_ingot’:3, ’stick’:2},
’crafting_table’) needs 3 iron_ingot, but I only have 1 iron_ingot.

Replanner: Please fix above errors and replan the task "How to mine 1
diamond".

Planner: The code for mining diamond is as bellows:

def mine_1_diamond(inventory = {}):

mine({’log’:3}, null); # action 1: mine 3 log without tool

craft({’planks’:12}, {’log’:3}, null); # action 2: craft 12 planks from 3 log

craft({’stick’:4}, {’planks’:2}, null); # action 3: craft 4 stick from 2
planks

craft({’crafting_table’:1}, {’planks’:4}, null); # action 4: craft 1
crafting_table from 4 planks

craft({’wooden_pickaxe’:1}, {’planks’:3, ’stick’:2}, ’crafting_table’); #
action 5: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table

mine({’cobblestone’:2}, ’wooden_pickaxe’); # action 6: mine 2 cobblestone with
wooden_pickaxe

mine({’cobblestone’:1}, ’wooden_pickaxe’); # action 7: mine 1 cobblestone with
wooden_pickaxe

craft({’stone_pickaxe’:1}, {’cobblestone’:3, ’stick’:2}, ’crafting_table’); #
action 8: craft 1 stone_pickaxe from 3 cobblestone and 2 stick, on
crafting_table

mine({’iron_ore’:1}, ’stone_pickaxe’); # action 9: mine 1 iron_ore with
stone_pickaxe

mine({’cobblestone’:8}, ’stone_pickaxe’); # action 10: mine 6 cobblestone with
stone_pickaxe

craft({’furnace’:1}, {’cobblestone’:8}, ’crafting_table’); # action 11: craft
1 furnace from 8 cobblestone, on crafting_table

craft({’iron_ingot’:1}, {’iron_ore’:1}, ’furnace’); # action 12: craft 1
iron_ingot from 1 iron_ore, on furnace

mine({’iron_ore’:2}, ’stone_pickaxe’); # action 13: mine 2 iron_ore with
stone_pickaxe

craft({’iron_ingot’:2}, {’iron_ore’:2}, ’furnace’); # action 14: craft 2
iron_ingot from 2 iron_ore, on furnace

craft({’iron_pickaxe’:1}, {’iron_ingot’:3, ’stick’:2}, ’crafting_table’); #
action 15: craft 1 iron_pickaxe from 3 iron_ingot and 2 stick, on
crafting_table

mine({’diamond’:1}, ’iron_pickaxe’); # action 16: mine 1 diamond with
iron_pickaxe

return ’diamond’

Descriptor: I succeed on step 13, 14.

Descriptor: I locate in plains biome. My inventory now has 5 planks, 1
crafting_table, 1 wooden_pickaxe, 1 stone_pickaxe, 1 furnace, 3 iron_ingot, 20
dirt, 64 stone.

Descriptor: I fail on step 15 "craft({’iron_pickaxe’:1}, {’iron_ingot’:3,
’stick’:2}, ’crafting_table’);"

Explainer: because craft({’iron_pickaxe’:1}, {’iron_ingot’:3, ’stick’:2},
’crafting_table’) needs 2 stick, but I only have 1 stick.

Replanner: Please fix above errors and replan the task "How to mine 1
diamond".

Planner: The code for mining diamond is as bellows:

def mine_1_diamond(inventory = {}):

mine({’log’:3}, null); # action 1: mine 3 log without tool

craft({’planks’:12}, {’log’:3}, null); # action 2: craft 12 planks from 3 log

craft({’stick’:4}, {’planks’:2}, null); # action 3: craft 4 stick from 2
planks

craft({’crafting_table’:1}, {’planks’:4}, null); # action 4: craft 1
crafting_table from 4 planks

craft({’wooden_pickaxe’:1}, {’planks’:3, ’stick’:2}, ’crafting_table’); #
action 5: craft 1 wooden_pickaxe from 3 planks and 2 stick, on crafting_table

mine({’cobblestone’:2}, ’wooden_pickaxe’); # action 6: mine 2 cobblestone with
wooden_pickaxe

mine({’cobblestone’:1}, ’wooden_pickaxe’); # action 7: mine 1 cobblestone with
wooden_pickaxe

craft({’stone_pickaxe’:1}, {’cobblestone’:3, ’stick’:2}, ’crafting_table’); #
action 8: craft 1 stone_pickaxe from 3 cobblestone and 2 stick, on
crafting_table

mine({’iron_ore’:1}, ’stone_pickaxe’); # action 9: mine 1 iron_ore with
stone_pickaxe

mine({’cobblestone’:8}, ’stone_pickaxe’); # action 10: mine 6 cobblestone with
stone_pickaxe

craft({’furnace’:1}, {’cobblestone’:8}, ’crafting_table’); # action 11: craft
1 furnace from 8 cobblestone, on crafting_table

craft({’iron_ingot’:1}, {’iron_ore’:1}, ’furnace’); # action 12: craft 1
iron_ingot from 1 iron_ore, on furnace

mine({’iron_ore’:2}, ’stone_pickaxe’); # action 13: mine 2 iron_ore with
stone_pickaxe

craft({’iron_ingot’:2}, {’iron_ore’:2}, ’furnace’); # action 14: craft 2
iron_ingot from 2 iron_ore, on furnace

craft({’stick’:2}, {’planks’:1}, null); # action 15: craft 2 stick from 1
planks

craft({’iron_pickaxe’:1}, {’iron_ingot’:3, ’stick’:2}, ’crafting_table’); #
action 16: craft 1 iron_pickaxe from 3 iron_ingot and 2 stick, on
crafting_table

mine({’diamond’:1}, ’iron_pickaxe’); # action 17: mine 1 diamond with
iron_pickaxe

return ’diamond’

Descriptor: I succeed on step 15, 16, 17.

Descriptor: Good. I finish the task.

Planner: OK.


* Gulavani and Gulwani (2008) Bhargav S. Gulavani and Sumit Gulwani. 2008.  A Numerical Abstract Domain Based on Expression Abstraction and Max Operator with Application in Timing Analysis. In _Proceedings of the 20th International Conference on Computer Aided Verification_ (Princeton, NJ, USA) _(CAV ’08)_. Springer-Verlag, Berlin, Heidelberg, 370–384.  https://doi.org/10.1007/978-3-540-70545-1_35
  * Isermann (1982) H. Isermann. 1982\.  Linear lexicographic optimization.  _Operations-Research-Spektrum_ 4 (1982).  Issue 4.
  * Kincaid et al. (2018) Z. Kincaid, J. Cyphert, J. Breck, and T.W. Reps. 2018\.  Non-Linear Reasoning for Invariant Synthesis.  _PACMPL_ 2(POPL) (2018), 54:1–54:33.
  * Komuravelli et al. (2016) A. Komuravelli, A. Gurfinkel, and S. Chaki. 2016\.  SMT-Based Model Checking for Recursive Programs. In _Formal Methods in System Design_.  https://doi.org/10.1007/s10703-016-0249-4
  * Li et al. (2014) Yi Li, Aws Albarghouthi, Zachary Kincaid, Arie Gurfinkel, and Marsha Chechik. 2014.  Symbolic Optimization with SMT Solvers.  _SIGPLAN Not._ 49, 1 (jan 2014), 607–618.  https://doi.org/10.1145/2578855.2535857
  * Loos and Weispfenning (1993) R. Loos and V. Weispfenning. 1993.  Applying Linear Quantifier Elimination.  _Comput. J._ 36 (1993).
  * Putinar (1993) Mihai Putinar. 1993\.  Positive Polynomials on Compact Semi-Algebraic Sets.  _Indiana Univ. Math. J._ 42 (1993), 969–984.  Issue 3.
  * Schmüdgen (2017) Konrad Schmüdgen. 2017\.  _The Moment Problem on Compact Semi-Algebraic Sets_.  Springer International Publishing, 283–313.  https://doi.org/10.1007/978-3-319-64546-9_12
  * Sebastiani and Tomasi (2012) Roberto Sebastiani and Silvia Tomasi. 2012.  Optimization in SMT with $\mathcal{LA}$($\mathbb{Q}$) Cost Functions. In _Automated Reasoning_ , Bernhard Gramlich, Dale Miller, and Uli Sattler (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 484–498.