#  RePLan: Robotic Replanning with Perception and Language Models

Marta Skreta1, Zihan Zhou1, Jia Lin Yuan1, Kourosh Darvish, Alán Aspuru-Guzik,
and Animesh Garg  University of Toronto, Vector Institute for Artificial
Intelligence, Georgia Institute of Technology, NVIDIA
{martaskreta, footoredo, jly<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
1These authors contributed equally to this work.

###### Abstract

Advancements in large language models (LLMs) have demonstrated their potential
in facilitating high-level reasoning, logical reasoning and robotics planning.
Recently, LLMs have also been able to generate reward functions for low-level
robot actions, effectively bridging the interface between high-level planning
and low-level robot control. However, the challenge remains that even with
syntactically correct plans, robots can still fail to achieve their intended
goals due to imperfect plans or unexpected environmental issues. To overcome
this, Vision Language Models (VLMs) have shown remarkable success in tasks
such as visual question answering. Leveraging the capabilities of VLMs, we
present a novel framework called Robotic Replanning with Perception and
Language Models (RePLan) that enables online replanning capabilities for long-
horizon tasks. This framework utilizes the physical grounding provided by a
VLM’s understanding of the world’s state to adapt robot actions when the
initial plan fails to achieve the desired goal. We developed a Reasoning and
Control (RC) benchmark with eight long-horizon tasks to test our approach. We
find that RePLan enables a robot to successfully adapt to unforeseen obstacles
while accomplishing open-ended, long-horizon goals, where baseline models
cannot, and can be readily applied to real robots. Find more information at
https://replan-lm.github.io/replan.github.io/

##  I Introduction

Designing embodied agents to execute multi-stage, long-horizon tasks is
challenging. Firstly, agents need manipulation skills for physical engagement
with their environment. They also need to be adept at perceiving their
surrounding environment and reasoning on cause-and-effect relationships of
their actions on the environment. Moreover, these agents should be able to
plan and carry out a series of actions that are in line with the main goals
they are tasked to accomplish [54], with minimal human intervention.

Methods based on rule-driven frameworks like Task and Motion Planning (TAMP)
[15] and learning approaches, such as Hierarchical Reinforcement Learning
(HRL) and Imitation Learning (IL), have advanced the field of long-horizon
planning. Yet, these methods often require extensive domain knowledge,
intricate reward engineering, and time-consuming dataset creation efforts [24,
6].

In contrast, the rise of Large Language Models (LLMs) has shown considerable
promise in robot planning [12, 7]. The integration of LLMs into complex, long-
horizon tasks in robot planning still poses substantial challenges. We
identify four primary areas where these challenges are most pronounced, which
we expand on below: a) long-horizon plan generation, b) verifying plan
correctness, c) obtaining feedback to adjust plans online, and d) open-ended
task execution.

Long-horizon, multi-stage task planning requires reasoning over extended
periods, which is a challenge for LLMs [52]. In the context of LLMs, tackling
large-scale problems often leads to issues like hallucination or failing to
consider important details, rendering their plans ineffective or error-ridden
[4, 51]. To address this, prompting schemes like ReAct and Chain-of-Thought
distill complex problems into intermediate reasoning steps with the aid of
exemplars, facilitating effective reasoning [57, 53]. However, these efforts
are still constrained by the number of stages the robot can handle.

Another challenge in using LLMs for task planning is verifying that their
outputs are correct, such as syntax checking [48, 50, 11] and semantics [45]
verification, as well as simulating task execution [34] to provide
success/failure feedback to the LLM. Verification enables the ability to plan
over multiple steps in complex domains.

Working with multi-step tasks in robotics also involves dealing with
uncertainties and changes in the environment. Effectively handling these tasks
requires combining task instructions with sensory data, which helps the robot
adapt to the changing surroundings. Recent research has shown that Vision-
Language Models (VLMs) can guide robots in interpreting their environment more
accurately and generalizing to unseen scenarios. We find that VLMs are able to
provide more accurate feedback about object states compared with LLMs. When
converting object states to pose estimations and asking an LLM to reason about
obstacles, we find that it is uncapable of inferring spatial relations between
objects (Appendix C-A). By integrating visual cues with linguistic context,
VLMs enable robots to better interpret their surrounding environment [7] and
can even provide rewards for training RL agents [5]. However, prompting VLMs
to directly generate out-of-the-box plans for multi-stage tasks may not work
as task complexity increases. For example, in Figure LABEL:fig:overview and
Appendix D-B, we show that prompting GPT-4V [41](state-of-the-art VLM with
public access) does not take into account environment obstacles, and therefore
generates an unfeasible plan.

Table I: Comparison of RePLan with other LLM-based multi-stage planners.
Autonomous online replanning refers to replanning without human intervention.
If a feature is unclear from the paper description, it is denoted by a
question mark.

Paper |  | No human
---
assistance
| Open-ended
---
problem-solving
| Verifier
---
| Sensory
---
feedback
| Reward
---
generator
| Unstructured state
---
representation
| No
---
fine-tuning
| Autonomous online
---
replanning
Text2Reward [56] | $\times$ | $\times$ | ✓ | ✓ | ✓ | $\times$ | $\times$ | $\times$
|  Language to Rewards [59]
---
$\times$ | $\times$ | $\times$ | ✓ | ✓ | ✓ | ✓ | $\times$
REFLECT [36] | ✓ | ✓ | ✓ | ✓ | $\times$ | $\times$ | ✓ | ✓
DoReMi [18] | ✓ | $\times$ | ✓ | ✓ | $\times$ | ✓ | ✓ | ✓
DROC [60] | $\times$ | $\times$ | ✓ | $\times$ | $\times$ | ✓ | ✓ | $\times$
SayCan [8] | ✓ | ✓ | $\times$ | ✓ | $\times$ | ✓ | ✓ | $\times$
Inner Monologue [23] | $\times$ | ✓ | ✓ | ✓ | $\times$ | ✓ | ? | ✓
RePLan | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓

In order for robots to generalize across diverse environments, the acquisition
of rich low-level skills is essential [50, 61]. Many works have utilized pre-
trained and encoded skills, supplying a skill list to LLMs for skill selection
[32]. It should also be noted that merely relying on user-provided skills is
insufficient for enabling robots to operate effectively in an open-ended
world. Overcoming this limitation necessitates a focus on reward generation
strategies. Recently, there has been a shift towards directly setting up
rewards generated by LLMs [59, 28, 56]. This approach, employed in both RL for
policy learning [28, 56] and Model Predictive Control (MPC) [13, 46] to
enhance data efficiency [40, 59], enables users to more easily guide robot
behavior by creating and combining rewards.

In this paper, we present a framework called Robotic Replanning with
Perception and Language Models (RePLan) that enables multi-stage, long-horizon
task execution without human intervention. Specifically, our contributions are
the following:

  1. 1.

RePLan: an end-to-end framework that does multi-level replanning with
verification through zero-shot models, incorporating high-level planning and
low-level reward generation (Figure LABEL:fig:overview)

  2. 2.

We use vision-language signals for grounding and closing the RePLan feedback
loop as an error correction system

  3. 3.

We demonstrate that RePLan can solve tasks that are (a) ambiguous (i.e., the
correct solution to the problem is not known at the beginning) and (b) adapt
its plan online as challenges are encountered during task execution without
human intervention

  4. 4.

We present a Reasoning and Control (RC) benchmark of eight tasks that combine
both of these aspects. We show that RePLan is able to succeed almost $4\times$
as often compared to baselines and can perform real-world experiments .

##  II Related work

Long-horizon Robot Planning. Addressing long-horizon planning in robotics has
been a persistent challenge. Rule-based methods [39, 3], such as Planning
Domain Definition Language (PDDL) [1], attempted to solve task and motion
planning sequentially, however, the planned task may not feasible when the
domain knowledge is incomplete. The task and motion planning (TAMP) approach
[15], addresses this by simultaneously determining symbolic actions and low-
level motions. While these methods excel in verifying task and motion plans
during planning, their generalization to new environments and tasks as well as
their efficiency on replanning are constrained. In addressing multi-stage
planning challenges, many works focus on learning task plans from input task
specifications, leveraging reinforcement learning (RL) and imitation learning
(IL) techniques. For example, Behavior-1K [29] employs RL to acquire semantics
and physical manipulation skills, often benefiting from classical motion
planners and simplifying assumptions.

However, it’s important to note that these learning-based techniques demand
significant domain expertise for reward engineering and rely on large datasets
for task learning [20]. While they adeptly react to environmental
uncertainties by iteratively updating policies based on observations, their
zero-shot generalization across multi-stage tasks remains a persistent
challenge.

Robot Control with Physically Grounded Language Models. Recent advancements in
LLMs have resulted in their adoption in robot planning, leveraging their
natural language capabilities and common-sense reasoning for generating robot
task and motion plans[51, 55]. Notably, LLMs have been applied to planning
multi-stage tasks [47, 12], by utilizing LLMs to improve sample efficiency in
reinforcement learning. ProgPrompt and Code-As-Policies are among the early
approaches using code-writing LLMs to generate code for robot policies [47,
30]. However, generated codes are prone to compilation errors.

Language models with a verifier have been used for generating long-horizon
tasks in an iterative prompting technique with lower error rates [58];
however, there is no guarantee that the output task plan can be executed. To
overcome that shortcoming, SayPlan used LLMs to reason over scene graphs and
generate plans across large environments, using iterative replanning to ensure
scene graph constraints were not violated [45].

Toward grounding the language belief with visual and motor control feedback,
several works have employed a variation of vision-language models with agent
embodiment for solving planning problems [19, 7, 8, 23]. Table I provides a
detailed comparison of RePLan with most related works.

Language to Reward Shaping. In contrast to approaches that map natural task
descriptions to robot actions and subsequently to rewards, an alternative
approach seeks to directly infer rewards from natural language inputs,
addressing the challenge of reward engineering [31]. This language-driven
reward-shaping approach has demonstrated utility in various domains, including
negotiation [28] and gaming [17], facilitating desired behavior learning
through RL. [37] introduce a visuo-language model that generates robot motion
policy reward on goal text and raw pixel observations, in a manner similar to
[43], enabling zero-shot prompting for unseen scenarios. [59] employs an
iterative prompting method using an LLM to link user task specifications and
robot motion through reward functions. While excelling in motion generation
with minimal data, their approach falls short in handling long-horizon
multistage tasks and lacks real-time environment feedback, necessitating user
intervention for adaptation. [56] extended the previous work for robot reward
policy refinement by requiring substantial human involvement and Pythonic
observation from the environment. Both of these methods struggle with open-
ended problems and multi-stage tasks. To mitigate these limitations, our work
autonomously performs long-horizon tasks and adapts to execution outcomes by
leveraging motor control and raw visual feedback.

Online Replanning Based on Sensory Feedback. Recent works have explored online
robotic adaption for recovering from errors during task execution. CAPE
queries an LLM for a possible corrective action if a precondition is not met
[44]. DROC introduces a framework that can provide a robot with low-level and
high-level corrections, which are then stored in a knowledge base that can be
retrieved later [60]. However, the corrections must be provided by humans,
whereas our framework does not require any human intervention. DoReMi uses an
LLM to generate constraints that must be met during a given subtask that the
robot is executing, and then queries a VLM to ensure those preconditions are
met [18]. However, this framework has to rely on constraints that an LLM
generates and only probes the VLM for a yes-or-no answer, which may miss
unexpected obstacles, and does not allow for tasks which have ambiguous
solutions. The framework also queries a VLM every $t$ seconds, which may
increase latency depending on the VLM used. REFLECT uses audio and video
sensors to hierarchically generate captions for object states, and then uses
an LLM to reason upon possible failure reasons if a subtask is not
accomplished [36]. However, this method requires predefining caption states
and sensor labels. Furthermore, none of these methods allow for task
completion where the task is ambiguous (except for CAPE and REFLECT), and none
output unconstrained robot rewards directly.

Figure 1: RePLan model architecture for hierarchical reasoning and control. In
the Task Control level, RePLan first generates high-level subtasks conditioned
on the user’s goal and a scene image. Next, the Subtask Control level
determines what action types need to be taken to execute the subtask. If the
action requires obtaining information about object attributes or states, the
VLM Perceiver may be called to answer any questions. VLM answers are reasoned
upon and used to update the world knowledge of the Planners. Otherwise, if the
subtasks require low-level actions, a low level motion plan is generated and
passed to the Low-Level Planner at the Skill Controls level. The Planner
generates robot skill-level rewards for execution via MPC. If failures occur,
feedback is provided to subtask and task controls for replanning. If the goal
is not reached even after replanning and completing the subtasks, the High-
Level Planner reflects on past experiences and proposes a new plan. A more
detailed version is shown in Appendix Figure B.1.

##  III RePLan: Model Architecture

We present an overview of our method in Figure 1. The input to our system is a
goal described in natural language. The goal can be specific (e.g. place
kettle on stove) or open-ended (e.g. search for the banana). RePLan has five
modules, which are described below. All prompts used for the modules can be
found in Appendix B.

  1. 1.

a High-Level LLM Planner for planning, replanning, reflecting, and reasoning

  2. 2.

a VLM Perceiver for physically-grounded reasoning

  3. 3.

a Low-Level LLM Planner for converting high-level tasks to low-level rewards

  4. 4.

a Motion Controller for low-level robot actions

  5. 5.

a LLM Verifier to check that the Planner/Perceiver is correct and fixes them
if applicable

###  III-A High-Level LLM Planner

Inspired by the ability of LLMs to generate actionable steps from high-level
tasks [22], we employ a High-Level Planner to take in as input a user-
specified task and return a list of subtasks on how a robot should accomplish
the task. We use a prompting scheme similar to ReAct [57] for generating
subtasks. The benefit of using a High-Level Planner is that there are no
restrictions on the abstraction level of the user input. The user input can be
a specific task (e.g. “Place my keys on the counter”), or an open-ended task
where the procedure requires exploration (e.g. “Find my keys”). This is
because the Planner can propose a procedure, even if it doesn’t know the exact
answer a priori. If the robot completes the sequence of actions proposed by
the Planner and the overall goal is still not accomplished, the High-Level
Planner can propose a new procedure. The High-Level Planner can also utilize
the past recommended procedures to prevent the redundancy of having the agent
perform the same tasks repetitively [48]. The High-Level Planner is also used
to incorporate feedback from perception models when generating high-level
plans. This is important because the Planner should not generate plans that
are not aligned with the physical state of the world, or should be able to
replan if there are any obstacles.

###  III-B VLM Perceiver

While LLMs have demonstrated powerful reasoning skills over text, they lack
grounding in the physical world [34]. Consequently, although LLMs can generate
plans that sound reasonable, they may fail to account for any obstacles or
uncertainties that are present because they cannot perceive the environment.
In constrast, while VLMs offer physical grounding to text queries, they are
much weaker in language-only reasoning than LLMs. Hence, we use the High-Level
Planner to decide what it wants to query from the Perceiver, and then the
Planner incorporates feedback from the Perceiver when it needs to know about
the object states or replan because the robot failed to do an action. The
High-Level LLM-based Planner decides on simple, yet specific, questions to ask
the Perceiver (Figures B.15-LABEL:fig:vlm_object_state) and samples multiple
answers before consolidating them into a summary observation that is
consistent with the environment state (Figures B.10-B.13).

###  III-C Low-Level LLM Planner

Recently, it has been demonstrated that LLMs are capable of producing low-
level plans that enable robot motion control [59, 56]. This is exciting
because it bridges the gap between high-level, human-specified goals and robot
actions in a zero-shot manner (without the need for extensive training
datasets). However, while previous works are able to generate low-level robot
actions for a concrete task (e.g. “Open the drawer"), we find that they fail
when asked to generate plans for long-horizon, open-ended tasks. Thus, we
utilize the High-Level Planner to generate concrete subtasks from a high-level
goal, which are then passed to the Low-Level Planner to generate the
corresponding low-level actions. Our Low-Level Planner uses the same Reward
Translator as in [59] which we have found works well. The Low-Level Planner
works in two stages. First, it generates a motion plan from a user-specified
input. The motion plan is a natural language description of the actions a
robot should do to achieve the goal. Then, the motion plan is then translated
to reward functions, which serve as a representation of the desired robot
motion. These reward functions are then passed to the Motion Controller.

###  III-D Motion Controller

Table II: Different features in RePLan benchmark tasks.

Feature |  | Cabinet-
---
Open
| Cabinet-
---
Closed
| Cabinet-
---
Blocked
| Cabinet-
---
Locked
| Cubes-
---
Color
| Cubes-
---
Blocked
| Kitchen-
---
Explore
| Composite-
---
Explore
Multi-step planning | $\times$ | ✓ | ✓ | ✓ | $\times$ | ✓ | ✓ | ✓
Visual feedback | $\times$ | $\times$ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓
Causal reasoning | $\times$ | $\times$ | ✓ | ✓ | $\times$ | ✓ | ✓ | ✓
Exploration | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | ✓ | ✓

The Motion Controller receives reward functions and instructs the robot on
what actions to take in order to satisfy those functions. For motion control,
we use MuJoCo MPC (MJPC), an open-source real-time predictive controller,
implemented on MuJoCo physics [21]. Given the initial condition $x_{0}$, the
control problem is defined as:

$\underset{x_{1:{T}},u_{1:T}}{\text{minimize
}}\quad\sum\limits_{t=0}^{T}c(x_{t},u_{t}),\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \text{subject to}\quad
x_{t+1}=f(x_{t},u_{t}),$

where $x_{t}$ and $u_{t}$ are the state and control signals at time step $t$,
and the mapping $f$ is the transition dynamics. The goal is to minimize the
cost function $c(.)$ along the trajectory from the current time step to the
horizon $T$. We define the $M$ output rewards provided by the low-level
planner as the negative of the cost function, i.e.,
$c(x_{t},u_{t})=-\sum\limits_{i=1}^{M}w_{i}\leavevmode\nobreak\
r_{i}(x_{t},u_{t},\phi_{i})$, where $\phi_{i}$ and $w_{i}$ are the $i$’th
reward parameters and weight. To solve the optimization problem in our work,
the predictive sampling implementation is used [21].

A subtask can have more than one reward function, the Low-Level Planner also
reasons about which reward function actually determines the success state of
the action. For example, for the subtask: Move the kettle away from the
microwave, the Planner generates:

⬇

minimize_l2_distance_reward("palm", "kettle")

maximize_l2_distance_reward("kettle", "microwave_handle")

The Planner is able to reason that once the function
maximize_l2_distance_reward ("kettle", "microwave_handle") has been satisfied,
the kettle has been moved from the microwave. Thus, the Planner correctly
labels this as the primary reward function.

###  III-E LLM & VLM Verifier

LLM outputs can contain errors and are often non-deterministic. One way to
increase consistency and reliability is to verify the outputs [48]. We do that
by taking plans produced by both the High-Level and Low-Level Planners and
asking the Verifier to check that every step proposed is necessary to achieve
the goal. For the Low-Level Planner, the Verifier is used to determine whether
each step in the generated motion plan is useful for completing the subtask.
This eliminates any unrelated actions that the Low-Level Planner. Motion
control is a hard problem, eliminating any unnecessary actions increases the
chances of the subtask being completed.

Since the VLMs are prone to hallucination [10], they are used in a very
constrained manner. Thus, the Verifier also corrects any observations made by
the Perceiver based on objects that it knows exist in the environment. For
instance, VLMs can identify an object using different synonyms. While it may
be easy for humans to disambiguate, a robot could require instructions that
adhere to strict syntax rules. Thus, the Verifier ensures that plans generated
using perceived objects conform to the robot-interpretable entity names.

##  IV Reasoning & Control (RC) Benchmark

In order to assess the long-term planning, as well as the logical and low-
level planning capabilities of our system, we devised a new benchmark of eight
long-horizon tasks that demand causal reasoning and exploration (Table II) for
evaluation. The readers can refer to Figure A.2 or our website for
visualizations and Appendix A for detailed descriptions.
Cabinet-Open A scene with an opened cabinet and a cube. The robot is required
to put the cube inside the cabinet.
Cabinet-Closed Same as Cabinet-Open except that the cabinet is closed.
Cabinet-Blocked Same as Cabinet-Closed except that there is a bar blocking the
cabinet handles.
Cabinet-Locked The robot is tasked to retrieve a cube from a cabinet locked by
a lever.
Cubes-Color There are two cubes with different colors and one crate. The task
is to move the cube that matches the color of the crate into the crate.
Cubes-Blocked Cubes-Color, however, a cube with the wrong color is in the
crate. It needs to be moved first.
Kitchen-Explore Explore a complex kitchen scene, in search of a green apple
while dealing with obstacles.
Composite-Explore The task is to open a sliding cabinet, which is locked by a
weight sensor. The robot has to explore the scene to locate the weight.

These environments were implemented in MuJoCo [49]. We used
furniture_sim111https://github.com/vikashplus/furniture_sim/ for the assets.
We used the MuJoCo MPC [21] to generate the motion control.

##  V Experimental Evaluation

###  V-A Experiment Setup

We evaluate our framework using a Franka Emika Panda arm robot equipped with a
Robotiq 2F-85 gripper and a ZED Mini camera mounted on the gripper. We use
MuJoCo MPC for physics simulation and real-time predictive control [21].

For the LLM modules in RePLan, we use OpenAI GPT GPT-4 [42]. For the VLM
Perceiver, we used Qwen-VL-Chat-7B [2] (except for Cubes-Blocked, where we
used GPT-4V due to hardware constraints). We show the performance of state-of-
the-art VLMs on Perceiver tasks in Appendix D-A. We found that from the open-
source models we tested, Qwen-VL-Chat-7B has the best object reasoning skills;
however, its object recognition capabilities improve when we first segment the
scene using a segmentation model (we use Segment Anything [26]). GPT-4V had
the best performance overall across object recognition and object reasoning,
but there is a restrictive rate limit on its API.

###  V-B Baselines and Ablations

We compare our method to the Language to Rewards [59] framework, a one-shot,
in-context learning agent. Language to Rewards uses a Reward Translator to
translate a high-level goal (such as “open the drawer”) to low-level reward
functions that are used by a Motion Controller to instruct a robot on what to
do. While Language to Rewards does not utilize a VLM to perceive the scene, we
give it access to the same objects as our model identifies at the start of the
scene. We also show sample plans using PDDL and PDDLStream in Appendix D-C for
Cabinet-{Open, Closed, Blocked} and GPT-4V for Cabinet-Blocked and Cubes-Color
in Appendix D-B. Notably, since our focus is on zero-shot online (re)planning,
we do not use RL based baselines, including recent LLMs to reward models.
However, we note that if pretrained multi-task controllers were available,
they could be utilized in low-level motion generation.

Finally, to demonstrate the importance of all the modules in our pipeline, we
do an ablation study on how well the robot can perform each task without each
module. We systematically remove the following modules: VLM Perceiver, LLM
Verifier, and replanning of High-Level Planner. We also provide additional
ablations for choice of VLM (Appendix D-A) as well as comparisons to naive
GPT-4V (Appendix D-B) and TAMP (Appendix D-C).

Table III: Completion rates in all 8 tasks (measured with 10 runs each).
Average completion rates across 8 tasks are listed in the last column.

Task | RePLan |  $-$verifier |  $-$perceiver |  $-$replan | L2R
---|---|---|---|---|---
CabinetOpen | 100% | 80% | 90% | 80% | 90%
CabinetClosed | 100% | 80% | 90% | 70% | 20%
CabinetBlocked | 60% | 20% | 30% | 0% | 0%
CabinetLocked | 100% | 100% | 50% | 0% | 0%
CubesColor | 90% | 100% | 20% | 80% | 50%
CubesBlocked | 90% | 80% | 0% | 10% | 10%
KitchenExplore | 80% | 60% | 20% | 0% | 0%
CompositeExplore | 70% | 20% | 0% | 20% | 0%
Average | 86.25% | 67.05% | 37.50% | 32.50% | 21.25%

###  V-C Experiment 1: Motion Controller

First, we run experiments to evaluate the consistency of our motion
controller. Specifically, we pick _4_ important motions in our main
experiments and run the motion controller _20_ times each. The picked motions
are:

  1. 1.

_Opening door._ A common motion that opens the cabinet door, specifically from
Cabinet-Closed.

  2. 2.

_Removing bar._ A task where a bar should be removed from handles, unblocking
the door. This task is from Cabinet-Blocked,

  3. 3.

_Pulling lever._ This task is from Cabinet-Locked. The robot needs to pull the
lever which unlocks a door.

  4. 4.

_Removing kettle._ This task is from Kitchen-Explore, where the robot needs to
remove the kettle that is obstructing the microwave door.

_Motion_ | _Success rate_
---|---
Opening door | 100%
Removing bar | 100%
Pulling lever | 100%
Removing kettle | 80%

It is worth noting that these tests are carried out in an ideal setting, hence
the success rates are overly optimistic. During actual planning, multiple
facts can affect the motion success rate. For example, the motion planner
could generate unnecessary reward functions; and previous steps could change
the locations or poses of the objects to interact.

###  V-D Experiment 2: Long-horizon task completion

We report the success rates of all models on their ability to complete our
benchmark tasks in Table III. We report the percentage of successful runs (ten
runs were done for each task). The number of actions performed by the robot in
each task is shown in Figure 2. On average, most tasks require 7 to 11 steps,
with some runs using up to 17 (all without human intervention).

Overall, RePLan achieves a 4$\times$ improvement over Language to Rewards.
From the eight tasks, Language to Rewards only shows non-negligible success
rates in Cabinet-Open, which is a single-motion task, and Cubes-Color, where
the algorithm can guess the correct cube to move, resulting in a 50% success
rate. Meanwhile, RePLan achieves at least 60% completion rate in all tested
tasks, with a close to 90% average completion rate. RePLan performs the worst
in Cabinet-Blocked, which we conjecture to be a result of both the requirement
of precise bar manipulation in tight spaces and the fact that the target block
can get stuck in irrecoverable positions. Furthermore, we find that PDDL and
naive GPT-4V are unable to solve these tasks out-of-the-box. PDDL requires
human-provided ground truth information (for example, that the door is
blocked, see Appendix D-C). PDDLStream is able to solve the problem
eventually, but in general, it requires a much longer time. When we prompt
GPT-4V naively to create a plan for solving some tasks, it is unable to
naturally identify obstacles or object states in the scene (Appendix D-B).
This indicates that constrained prompting is required for VLMs to provide
useful feedback.

Figure 2: Number of actions the robot executed in each task averaged over ten
runs. Actions requiring the Perceiver are shown in pink while those executed
using MPC are shown in purple. Standard deviations are shown using gray bars
while the minimum and maximum number of actions are shown using gray dots.

Figure 3: Rollout of robot solvingKitchen-Explore. The high-level plan is
shown in the top row. The second row shows each subtask and the corresponding
reward functions generated by the Low-Level Planner, as well as Perceiver
feedback. If the subtask fails, its box is colored in red. If the plan is
completed and the goal is achieved, its box is green.

To evaluate the significance of the Verifier, Perceiver, and Replan modules,
we notice that RePLan achieves +18.8%, +48.8%, and +53.8% improvement compared
with removing the three modules, respectively. We also notice that removing
each of the modules impacts the performance more in more complex tasks. For
example, in Kitchen-Explore, without the Perceiver, the agent is unable to
figure out that a kettle is blocking the microwave and is, therefore, unable
to open the microwave to complete the task. In Cubes-Blocked, the agent is
unable to see that a cube is already blocking the crate if it does not have
access to a Perceiver, so it cannot successfully place the cube within the
crate.

Noticeably, even in simpler tasks such as Cabinet-{Open, Closed}, RePLan still
demonstrates superior performances compared to other tested variants even
though replanning is not necessary for the ideal scenario. This shows RePLan’s
ability to act as a safe-net mechanism for unexpected errors through using a
Verifier, improving the overall consistency.

We show an example rollout of the robot completing Kitchen-Explore in Figure
3. For Kitchen-Explore, the High-Level Planner first instructs the robot to
look inside the cabinet. The Perceiver informs the Planner that the apple is
not in the cabinet. The Planner then instructs the robots to look inside the
microwave. However, the motion control fails. The Perceiver is again called
and informs the Planner that there is a kettle in front of the microwave. The
Planner replans, instructing the robot to first move the kettle out of the
way. The robot then resumes with the remaining subtasks. It opens the
microwave and the Perceiver informs the Planner that the green apple is there
and the task is complete. For another example rollout of Cabinet-Blocked, the
reader can refer to Appendix Figure B.0.

###  V-E RePLan with Real-Robot Environment

For real robot evaluation, we adapt Cubes-Blocked by replacing the colored
cubes with a lemon and an apple (Figure 4).

#### Pose estimation

In a real robot setting, we use GroundingDINO [35] to detect object bounding
boxes and Non-Maximum Suppression [62] to remove redundant ones. We then use
the Segment Anything Model (SAM) [27] to generate object segmentations. Using
depth information from a ZED
camera222https://www.stereolabs.com/products/zed-2, a 3D point cloud for each
segmented object is generated. Object poses are estimated by fitting the
smallest bounding box to each point cloud. This perception setup builds on a
recent Language guided model used in ORGANA [11].

#### Planning and replanning

For long-horizon planning, we follow the same RePLan’s pipeline as in
simulated environments, with the exception that the real-world images are used
for the Perceiver module. During the MPC low-level planning stage, we first
run the simulation with the estimated object poses from our perception module.
If the simulation ends in a failure state, we return to RePLan to replan with
the updated real-world images for diagnosis; if the simulation succeeds, we
then execute the simulated trajectory on our real robot. Additionally, we have
found that MPC often generates actions that would not be deemed safe in a lab.
To compensate this, we have incorporated supplementary terms into our
residuals to comply with safety constraints.

Figure 4: Real-world experiment. The robot is tasked with placing an apple
inside the bowl, but it has to figure out that the lemon must first be removed
in order to complete the task. The full experiment trajectory can be seen in
the video in Supplementary Material.

##  VI Limitations

We note one limitation of our method is its reliance on VLM’s understanding
and interpretation of spatial states. If the VLM cannot accurately recognize
an object or interpret the reason why a task is incomplete, it may lead to the
inaccurate processing of the robotic task. This will likely be improved as
VLMs become trained to directly predict spatial states, such as SpatialVLM
[9]. In the meantime, we find that by using an LLM to probe specific
questions, sampling multiple answers, and generating a summary consistent with
the LLM’s knowledge of the scene, RePLan is able to utilize the VLM to
accurately reason about the scene state. We have also identified some failure
points: Case 1: LLM-MPC communication failure (Figure 5(a)) In our Cabinet-
Close environment robot tried to open the cabinet door but failed. The
Perceiver gave a correct diagnosis to remove the bar from the handle. However,
when generating the reward functions to remove the bar, the LLM selected the
wrong primary reward function, as demonstrated below:

⬇

reset_reward()

minimize_l2_distance_reward("palm", "red_block_right_side",
primary_reward=True)

maximize_l2_distance_reward("red_block_right_side",
"target_position_in_cabinet")

execute_plan()

maximize_l2_distance_reward should be the primary function. As a result, MPC
ends prematurely. The robot will not be able to remove the bar. Another case
of this is failure attributed to the controller could be incorrectly
interpreted as the task being undoable.This misinterpretation can lead to
unnecessary task abandonment or repeated attempts, resulting in a waste of
time and resources. Case 2: LLM consistency failure (Figure 5(b)) The robot is
blocked by a kettle obstructing the path. The Perceiver gives five diagnoses,
of which three claimed that the kettle was blocking the way, one claimed the
cabinet door was blocking the way, and one did not give any conclusive
diagnosis. The summary LLM concludes that the cabinet door blocked the action.
The robot interacts with the cabinet and never removes the kettle. Case 3:
LLM-VLM communication failure (Figure 5(c)) The high-level planner proposed a
plan where the first step is _“Identify the cube with the same colour as the
crate”_ using the VLM. However, the LLM queries the VLM with _“Which cube has
the same colour?”_ , which is too vague. This results in the VLM answering
_“The same color cube is the yellow cube and the yellow cube in the middle of
the blue cube group.”_. This answer did not provide the necessary information
to solve the task. The robot put the wrong cube on the crate. Case 4:
preciever failure (Figure 5(d)) After the robot’s failure to execute the task
“Place the red cube on the crate", the Perceiver was called to help identify
any issues. The Perceiver’s diagnoses all mentioned that the robot was holding
the red cube but did not identify the yellow cube as blocking the crate, and
so the Planner’s summary of the VLMs diagnoses was: “Based on the given
information, the most probable reason why the robot cannot place the red_cube
on the crate is that it is currently holding the red cube." However, it’s also
important to note that Cubes-Blocked used GPT-4V which severely limits the
number of ouput tokens from the model, and so a lot of explanations were cut
off (for example: “In the image provided, the robot is holding the red cube,
which is currently"). Additional details in Appendix C.

Figure 5:  Examples of error cases that occurred in RePLan include a) failure
to remove the bar due to incorrect reward generation (CabinetBlocked); b)
failure to identify the reason for the inability to open the microwave door
(KitchenExplore); c) vague VLM question about object attributes resulting in
incorrect cube selection (CubesColor); d) failure to identify why the robot
could not place the red cube in the crate (CubesBlocked).

##  VII Discussion and Conclusions

In Section C-A, we demonstrate that only providing coordinates and bounding
boxes is insufficient for the LLM to correctly reason about the error. We also
find that out-of-the-box VLMs are unable to diagnose errors in one shot
either, as shown by our experiments in Appendix D-B. The perceiver is
important for diagnosing errors. However, we also found that we were able to
make MPC reward generation more robust by using a Verifier to eliminate any
reward functions that were not essential to completing the task. In essence,
we found that using a Verifier at multiple stages in our workflow was
essential in improving the success rate of long-horizon task execution.

In summary, our paper introduces RePLan, a robust solution for multi-stage
planning, utilizing the power of LLMs for plan generation and VLMs for
insightful feedback. Our multi-level planning approach, coupled with step-wise
verification and replanning, demonstrates promising results in addressing
multi-stage tasks.

##  VIII Acknowledgements

We would like to thank Miroslav Bogdanovic for insightful discussions and
Yuchi Zhao for assistance with real robot setup. This research was undertaken
thanks in part to funding provided to the University of Toronto’s Acceleration
Consortium from the Canada First Research Excellence Fund Grant number -
CFREF-2022-00042.

## References

  * Aeronautiques et al. [1998] Constructions Aeronautiques, Adele Howe, Craig Knoblock, ISI Drew McDermott, Ashwin Ram, Manuela Veloso, Daniel Weld, David Wilkins SRI, Anthony Barrett, Dave Christianson, et al.  PDDL - the planning domain definition language.  _Tech. Rep._ , 1998.
  * Bai et al. [2023] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou.  Qwen-vl: A frontier large vision-language model with versatile abilities.  _arXiv preprint arXiv:2308.12966_ , 2023.
  * Baier et al. [2009] Jorge A Baier, Fahiem Bacchus, and Sheila A McIlraith.  A heuristic search approach to planning with temporally extended preferences.  _Artif. Intell._ , 2009.
  * Bang et al. [2023] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, et al.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.  _arXiv preprint arXiv:2302.04023_ , 2023.
  * Baumli et al. [2023] Kate Baumli, Satinder Baveja, Feryal Behbahani, Harris Chan, Gheorghe Comanici, Sebastian Flennerhag, Maxime Gazeau, Kristian Holsheimer, Dan Horgan, Michael Laskin, et al.  Vision-language models as a source of rewards.  _arXiv preprint arXiv:2312.09187_ , 2023.
  * Brohan et al. [2022] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Jasmine Hsu, et al.  Rt-1: Robotics transformer for real-world control at scale.  _arXiv preprint arXiv:2212.06817_ , 2022.
  * Brohan et al. [2023a] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al.  Rt-2: Vision-language-action models transfer web knowledge to robotic control.  _arXiv preprint arXiv:2307.15818_ , 2023a.
  * Brohan et al. [2023b] Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al.  Do as i can, not as i say: Grounding language in robotic affordances.  In _Conference on Robot Learning_ , pages 287–318. PMLR, 2023b.
  * Chen et al. [2024] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brian Ichter, Danny Driess, Pete Florence, Dorsa Sadigh, Leonidas Guibas, and Fei Xia.  Spatialvlm: Endowing vision-language models with spatial reasoning capabilities.  _arXiv preprint arXiv:2401.12168_ , 2024.
  * Dai et al. [2023] Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung.  Plausible may not be faithful: Probing object hallucination in vision-language pre-training.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 2136–2148, Dubrovnik, Croatia, May 2023. Association for Computational Linguistics.  doi: 10.18653/v1/2023.eacl-main.156.  URL https://aclanthology.org/2023.eacl-main.156.
  * Darvish et al. [2024] Kourosh Darvish, Marta Skreta, Yuchi Zhao, Naruki Yoshikawa, Sagnik Som, Miroslav Bogdanovic, Yang Cao, Han Hao, Haoping Xu, Alán Aspuru-Guzik, Animesh Garg, and Florian Shkurti.  Organa: A robotic assistant for automated chemistry experimentation and characterization.  _arXiv preprint arXiv:2401.06949_ , 2024.
  * Driess et al. [2023] Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_ , 2023.
  * Garcia et al. [1989] Carlos E Garcia, David M Prett, and Manfred Morari.  Model predictive control: Theory and practice—a survey.  _Automatica_ , 25(3):335–348, 1989.
  * Garrett et al. [2020] Caelan Reed Garrett, Tomás Lozano-Pérez, and Leslie Pack Kaelbling.  PDDLStream: Integrating symbolic planners and blackbox samplers via optimistic adaptive planning.  In _Proceedings of the 30th Int. Conf. on Automated Planning and Scheduling (ICAPS)_ , pages 440–448. AAAI Press, 2020.
  * Garrett et al. [2021] Caelan Reed Garrett, Rohan Chitnis, Rachel Holladay, Beomjoon Kim, Tom Silver, Leslie Pack Kaelbling, and Tomás Lozano-Pérez.  Integrated task and motion planning.  _Annual review of control, robotics, and autonomous systems_ , 4:265–293, 2021.
  * Gerevini et al. [2006] Alfonso Gerevini, Alessandro Saetti, and Ivan Serina.  An approach to temporal planning and scheduling in domains with predictable exogenous events.  _Journal of Artificial Intelligence Research_ , 25:187–231, 2006.
  * Goyal et al. [2019] Prasoon Goyal, Scott Niekum, and Raymond J Mooney.  Using natural language for reward shaping in reinforcement learning.  _arXiv preprint arXiv:1903.02020_ , 2019.
  * Guo et al. [2023] Yanjiang Guo, Yen-Jen Wang, Lihan Zha, Zheyuan Jiang, and Jianyu Chen.  Doremi: Grounding language model by detecting and recovering from plan-execution misalignment.  _arXiv preprint arXiv:2307.00329_ , 2023.
  * Ha et al. [2023] Huy Ha, Pete Florence, and Shuran Song.  Scaling up and distilling down: Language-guided robot skill acquisition.  _arXiv preprint arXiv:2307.14535_ , 2023.
  * Heo et al. [2023] Minho Heo, Youngwoon Lee, Doohyun Lee, and Joseph J Lim.  Furniturebench: Reproducible real-world benchmark for long-horizon complex manipulation.  _arXiv preprint arXiv:2305.12821_ , 2023.
  * Howell et al. [2022] Taylor Howell, Nimrod Gileadi, Saran Tunyasuvunakool, Kevin Zakka, Tom Erez, and Yuval Tassa.  Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo.  dec 2022.  doi: 10.48550/arXiv.2212.00541.  URL https://arxiv.org/abs/2212.00541.
  * Huang et al. [2022a] Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  _arXiv preprint arXiv:2201.07207_ , 2022a.
  * Huang et al. [2022b] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al.  Inner monologue: Embodied reasoning through planning with language models.  _arXiv preprint arXiv:2207.05608_ , 2022b.
  * Hussein et al. [2017] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne.  Imitation learning: A survey of learning methods.  _ACM Computing Surveys (CSUR)_ , 50(2):1–35, 2017.
  * Khodeir et al. [2023] Mohamed Khodeir, Ben Agro, and Florian Shkurti.  Learning to search in task and motion planning with streams.  _IEEE Robotics and Automation Letters_ , 8(4):1983–1990, 2023.
  * Kirillov et al. [2023a] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything.  _arXiv:2304.02643_ , 2023a.
  * Kirillov et al. [2023b] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  _arXiv preprint arXiv:2304.02643_ , 2023b.
  * Kwon et al. [2023] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh.  Reward design with language models.  _arXiv preprint arXiv:2303.00001_ , 2023.
  * Li et al. [2023] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Martín-Martín, Chen Wang, Gabrael Levine, Michael Lingelbach, Jiankai Sun, et al.  Behavior-1k: A benchmark for embodied ai with 1,000 everyday activities and realistic simulation.  In _Conference on Robot Learning_ , pages 80–93. PMLR, 2023.
  * Liang et al. [2022] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng.  Code as policies: Language model programs for embodied control.  _arXiv preprint_ , 2022.  doi: 10.48550/arXiv.2209.07753.
  * Lin et al. [2022] Jessy Lin, Daniel Fried, Dan Klein, and Anca Dragan.  Inferring rewards from language in context.  _arXiv preprint arXiv:2204.02515_ , 2022.
  * Lin et al. [2023] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg.  Text2motion: From natural language instructions to feasible plans.  _arXiv preprint arXiv:2303.12153_ , 2023.
  * Liu et al. [2023a] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.  Improved baselines with visual instruction tuning, 2023a.
  * Liu et al. [2022] Ruibo Liu, Jason Wei, Shixiang Shane Gu, Te-Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M Dai.  Mind’s eye: Grounded language model reasoning through simulation.  _arXiv preprint arXiv:2210.05359_ , 2022.
  * Liu et al. [2023b] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al.  Grounding dino: Marrying dino with grounded pre-training for open-set object detection.  _arXiv preprint arXiv:2303.05499_ , 2023b.
  * Liu et al. [2023c] Zeyi Liu, Arpit Bahety, and Shuran Song.  Reflect: Summarizing robot experiences for failure explanation and correction.  _arXiv preprint arXiv:2306.15724_ , 2023c.
  * Mahmoudieh et al. [2022] Parsa Mahmoudieh, Deepak Pathak, and Trevor Darrell.  Zero-shot reward specification via grounded natural language.  In _International Conference on Machine Learning_ , pages 14743–14752. PMLR, 2022.
  * McDermott et al. [1998] Drew McDermott, Malik Ghallab, Adele Howe, Craig Knoblock, Ashwin Ram, Manuela Veloso, Daniel Weld, and David Wilkins.  Pddl-the planning domain definition language.  1998\.
  * Mehr et al. [2020] S Hessam M Mehr, Matthew Craven, Artem I Leonov, Graham Keenan, and Leroy Cronin.  A universal system for digitization and automatic execution of the chemical synthesis literature.  _Science_ , 370(6512):101–108, 2020.
  * Miyaoka et al. [2023] Yuya Miyaoka, Masaki Inoue, and Tomotaka Nii.  Chatmpc: Natural language based mpc personalization.  _arXiv preprint arXiv:2309.05952_ , 2023.
  * OpenAI [2023] OpenAI.  GPT-4V(ision) system card, 2023.  URL https://openai.com/research/gpt-4v-system-card.
  * OpenAI et al. [2023] OpenAI et al.  Gpt-4 technical report, 2023.
  * Radford et al. [2021] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Raman et al. [2023] Shreyas Sundara Raman, Vanya Cohen, David Paulius, Ifrah Idrees, Eric Rosen, Ray Mooney, and Stefanie Tellex.  Cape: Corrective actions from precondition errors using large language models.  In _2nd Workshop on Language and Robot Learning: Language as Grounding_ , 2023.
  * Rana et al. [2023] Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf.  Sayplan: Grounding large language models using 3d scene graphs for scalable task planning.  In _7th Annual Conference on Robot Learning_ , 2023.  URL https://openreview.net/forum?id=wMpOMO0Ss7a.
  * Rawlings [2000] James B Rawlings.  Tutorial overview of model predictive control.  _IEEE control systems magazine_ , 20(3):38–52, 2000.
  * Singh et al. [2022] Ishika Singh, Valts Blukis, Arsalan Mousavian, Ankit Goyal, Danfei Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg.  Progprompt: Generating situated robot task plans using large language models.  _arXiv preprint arXiv:2209.11302_ , 2022.
  * Skreta et al. [2023] Marta Skreta, Naruki Yoshikawa, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Kourosh Darvish, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg.  Errors are useful prompts: Instruction guided task programming with verifier-assisted iterative prompting.  _arXiv preprint arXiv:2303.14100_ , 2023.
  * Todorov et al. [2012] Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 5026–5033. IEEE, 2012.  doi: 10.1109/IROS.2012.6386109.
  * Wang et al. [2023a] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models.  _arXiv preprint arXiv: Arxiv-2305.16291_ , 2023a.
  * Wang et al. [2023b] Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, et al.  A survey on large language model based autonomous agents.  _arXiv preprint arXiv:2308.11432_ , 2023b.
  * Wang et al. [2023c] Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang.  Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents.  _arXiv preprint arXiv:2302.01560_ , 2023c.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837, 2022.
  * Wu et al. [2023] Bohan Wu, Roberto Martin-Martin, and Li Fei-Fei.  M-ember: Tackling long-horizon mobile manipulation via factorized domain transfer.  _arXiv preprint arXiv:2305.13567_ , 2023.
  * Xi et al. [2023] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al.  The rise and potential of large language model based agents: A survey.  _arXiv preprint arXiv:2309.07864_ , 2023.
  * Xie et al. [2023] Tianbao Xie, Siheng Zhao, Chen Henry Wu, Yitao Liu, Qian Luo, Victor Zhong, Yanchao Yang, and Tao Yu.  Text2reward: Automated dense reward function generation for reinforcement learning.  _arXiv preprint arXiv:2309.11489_ , 2023.
  * Yao et al. [2023] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.  React: Synergizing reasoning and acting in language models.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=WE_vluYUL-X.
  * Yoshikawa et al. [2023] Naruki Yoshikawa, Marta Skreta, Kourosh Darvish, Sebastian Arellano-Rubach, Zhi Ji, Lasse Bjørn Kristensen, Andrew Zou Li, Yuchi Zhao, Haoping Xu, Artur Kuramshin, Alán Aspuru-Guzik, Florian Shkurti, and Animesh Garg.  Large language models for chemistry robotics.  _Autonomous Robots_ , 47(8):1057–1086, 2023.
  * Yu et al. [2023] Wenhao Yu, Nimrod Gileadi, Chuyuan Fu, Sean Kirmani, Kuang-Huei Lee, Montse Gonzalez Arenas, Hao-Tien Lewis Chiang, Tom Erez, Leonard Hasenclever, Jan Humplik, et al.  Language to rewards for robotic skill synthesis.  _arXiv preprint arXiv:2306.08647_ , 2023.
  * Zha et al. [2023] Lihan Zha, Yuchen Cui, Li-Heng Lin, Minae Kwon, Montserrat Gonzalez Arenas, Andy Zeng, Fei Xia, and Dorsa Sadigh.  Distilling and retrieving generalizable knowledge for robot manipulation via language corrections.  _arXiv preprint arXiv:2311.10678_ , 2023.
  * Zhou and Garg [2023] Zihan Zhou and Animesh Garg.  Learning achievement structure for structured exploration in domains with sparse reward, 2023.
  * Zou et al. [2023] Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye.  Object detection in 20 years: A survey.  _Proceedings of the IEEE_ , 2023.

Appendices

##  Appendix A Environment Details

###  A-A Wooden cabinet scene

A room where there is a yellow cube placed on the floor beside a wooden
cabinet. There is a red bar holding the handles of the wooden cabinet closed.
The doors of the cabinet cannot be opened without removing the bar. We
implemented three tasks in this scene:

####  CabinetOpen: Place the yellow cube in the wooden cabinet (easy mode).

The robot must pick up the yellow cube and place it in the cabinet, which is
open. This is a simple 1-step task that evaluates the Low-Level Planner for
single motion planning.

####  CabinetClosed: Place the yellow cube in the wooden cabinet (hard mode).

This task is the same as the previous task, but now the wooden cabinet doors
are closed. This task requires two steps: 1) opening the door and 2) placing
the cube inside the cabinet. The High-Level Planner is assessed to plan a
sequence of actions and pass them to the Low-Level Planner for motion
generation.

####  CabinetBlocked: Place the yellow cube inside the wooden cabinet (expert
mode).

The challenge with this task is that the robot must identify that it cannot
open the wooden cabinet because there is a bar across the handles of the door.
After removing the bar, the robot can open the cabinet door and finish the
task. This task is challenging because it requires vision to identify that the
door cannot be opened, followed by replanning to remove the item blocking the
door.

###  A-B Kitchen environment scene

A kitchen that contains a cabinet with two doors, a microwave, a kettle, and a
green apple.

####  KitchenExplore: Find the green apple.

A green apple was hidden in the microwave is not visible to the robot at the
start of the scene. The robot must search for the apple in the kitchen. There
is an additional challenge where the kettle is blocking the microwave door
from being opened, and so to open the door, the robot must first remove the
kettle. Same as CabinetBlocked, Kitchen-Explore also requires both vision and
replanning to solve the task, but it has an additional challenge because the
goal requires open-ended exploration (it is unclear where the apple is), which
requires replanning at a high level.

###  A-C Wooden cabinet and lever scene

A room containing a wooden cabinet, a blue block, and a lever that controls
whether the cabinet door is locked or unlocked.

####  CabinetLocked: Remove the blue cube from the cabinet.

Just as with tasks 1-3, this task requires the robot to open a cabinet. There
is no physical obstruction preventing the cabinet from being opened; however,
the cabinet is locked. The cabinet becomes unlocked once a lever close to the
cabinet is pulled. Thus, after (unsuccessfully) trying to open the cabinet
door, the robot must reason that it should pull the lever first and then open
the door.

###  A-D Coloured cubes scene

A room containing a small red crate and two cubes (one is yellow and the other
is red).

####  CubesColor: Place the cube with the same colour as the crate on the
crate.

In this task, the robot has to identify the cube with the same colour as the
crate and place it on the crate.

####  CubesBlocked: Blocking cube.

The robot is given the colour of a cube it must put on the crate. However,
there is already a cube on the crate with a different colour and the crate can
only hold one cube at a time. The robot must remove the cube that is already
on the crate before placing the target one.

####  CompositeExplore: Weight sensor

The robot is asked to open a sliding cabinet. The cabinet is locked by a
weight sensor, which can be activated by putting a weight on top. To do this,
the robot needs first to explore the scene to find the weight.

The instructions we use for each task are listed below:

_Environment_ |  _Instruction_
---|---
Cabinet-{Open, Closed, Blocked} |  move the yellow_cube to target_position inside the wooden_cabinet
Kitchen-Explore |  find the green_cube
Cabinet-Locked |  find the blue_cube
TwoCube-Color |  place the cube with the same color as the crate on the crate
TwoCube-Blocked |  place the red cube on the crate
Composite-Explore |  open the stone_cabinet. The weight sensor lock can be unlocked by putting the red_cube on it.

(a) Cabinet-Open \- initial

(b) Cabinet-Open \- complete

(c) Cabinet-Closed \- initial

(d) Cabinet-Closed \- complete

(e) Cabinet-Blocked \- initial

(f) Cabinet-Blocked \- complete

(g) Kitchen-Explore \- initial

(h) Kitchen-Explore \- complete

Figure A.1: Initial and final scenes of the tested environments.

(a) Cabinet-Locked \- initial

(b) Cabinet-Locked \- complete

(c) TwoCube-Color \- initial

(d) TwoCube-Color \- complete

(e) TwoCube-Blocked \- initial

(f) TwoCube-Blocked \- complete

(g) Composite-Explore \- initial

(h) Composite-Explore \- complete

Figure A.2: Initial and final scenes of the tested environments.

##  Appendix B Prompts

In Figure B.0, we show a example rollout of RePLan solving the cabinet-closed
task In Figure B.1, we also show a prompt roadmap. Furthermore, in Figures
B.2-B.17 we show step-by-step all the prompts we use for the Planners,
Perceiver, and Verifier. We show the prompts as the robot would receive them
while excuting a task. The prompts are coloured according to the module it
comes from – LLM Planners: blue, VLM Perceiver: pink, Verifier: gray.

Figure B.0: Rollout of robot solving cabinet-closed. The high-level plan is
shown in the top row. The second row shows each subtask and the corresponding
reward functions generated by the Low-Level Planner, as well as Perceiver
feedback. If the subtask fails, its box is colored in red. If the plan is
completed and the goal is achieved, its box is green.  Figure B.1: RePLan
prompt roadmap. We show key modules in the pipeline and indicate under each
one what prompts are utilized. This is an expanded version of the Figure 1

⬇

Do you see a(n) {0}?

Figure B.2: VLM prompt for perceiving objects in the environment. {0} is an
object that the robot knows how to interact with. The VLM is prompted with the
list of objects the robot knows how to interact with. If the VLM replies with
"yes", that object is added to a list of observed objects.

⬇

A stationary robot arm is in a location where it sees the following list of
objects:

{0}

The robot has the following goal: {1}

Propose high-level, abstract subtasks of what the robot needs to do to {1}.
The plan can only use one object.

For example, if the goal is to find a fork, one plan might be:

<thought>To find the fork, I will start by looking inside the
drawer.</thought>

[start plan]

>Open the drawer

>Look inside the drawer

>Grab the fork

[end plan]

Rules:

1. You have access to the following objects: {0}. Do not create new objects.

2. Generate a plan that interacts with only one object from the list at a time. Keep it as short as possible. Most plans should be under 5 steps.

3. Assume that every action is completed successfully.

4. Assume the first thing you try works.

5. Your plan should only propose one way of accomplishing the task.

6. The robot only has one arm and it cannot hold two things at a time. Remember that when you are deciding on the order of actions.

7. Enclose your thought process with a single pair of tag <thought> and </thought>

8. Enclose your plan with the a single pair of tag [start plan] and [end plan]

{2}

Figure B.3: LLM prompt for generating high-level task plans. {0} is the list
of objects the robot can see (for example: [cabinet, blue kettle, microwave],
{1} is the overall task goal (for example: find the green apple), {2} are
previous plans that were attempted but failed (see B.17).

⬇

A robot was asked to do this action:

> {0}

If the central verb is related to vision, answer yes.

Figure B.4: LLM prompt to determine whether the action that the Planner asked
the robot to do involves vision or not. If no, then the Planner is called to
generate MPC reward functions (see B.5-B.14). If yes, the Perceiver is called
(see B.15-LABEL:fig:vlm_completion). Examples for {0}: “Compare the color of
the left cube with the crate", “Open the microwave".

⬇

We have a stationary robot arm and we want you to help plan how it should move
to perform tasks using the following template:

[start of description]

The manipulator’s palm should move close to {{CHOICE: {0}}}.{1}{2}

[end of description]

Rules:

0. You cannot use one line twice!!!!

1. If you see phrases like [NUM: default_value], replace the entire phrase with a numerical value.

2. If you see phrases like {{CHOICE: choice1, choice2, ...}}, it means you should replace the entire phrase with one of the choices listed.

3. If you see [optional], it means you only add that line if necessary for the task, otherwise remove that line.

4. The environment contains {0}. Do not invent new objects not listed here.

5. I will tell you a behavior/skill/task that I want the manipulator to perform and you will provide the full plan, even if you may only need to change a few lines. Always start the description with [start of description] and end it with [end of description].

6. You can assume that the robot is capable of doing anything, even for the most challenging task.

7. Your plan should be as close to the provided template as possible. Do not include additional details.

8. Your plan should be as concise as possible. Do not include or make up unncessary tasks.

9. Each object can only be close to or far from one thing.

This is the entire procedure:

{4}

These are the observations we have made so far:

{5}

Create a plan for the following action:

> {6}

Figure B.5: LLM prompt to determine the low-level motion plan for the robot.
{0} is the list of objects the robot can interact with. {1} and {2} are
modifiers, depending on what type of motion is involved. The Planner is asked
to determine whether motion is involved (see B.6. If yes, then {1} becomes:
object1={{CHOICE: {0}}} should be {{CHOICE: close to, far from}}
object2={{CHOICE: {0}}}. If no, then {1} becomes [optional] object1={{CHOICE:
{0}}} should be close to object2={{CHOICE: {0}}}. [optional] object1={{CHOICE:
{0}}} should be far from object2={{CHOICE: {0}}}. The modifier {2} is added if
there are any joints in the scene that are involved with objects the robot can
interact with: [optional] joint={{CHOICE: {3}}} needs to be {{CHOICE: open,
closed}}. (where {3} is the list of joints) (adapted from [59]) {4} is the
entire plan the robot was given to execute the goal. {5} includes any
observations made by the Perceiver (see LABEL:fig:vlm_object_state) using the
following format: Q: <question to Perceiver>, A: <answer from Perceiver>. {6}
is the action the motion plan should be made for.

⬇

A robot arm has to do this action:

> {0}

Does this action necessarily involve relocating an object to a different
location that does not involve the robot arm? Answer with yes or no.

Figure B.6: LLM prompt to determine if relocation is needed in order to
determine the motion plan modifier (see B.5.

⬇

We have a plan of a robot arm with palm to manipulate objects and we want you
to turn that into the corresponding program with following functions:

def minimize_l2_distance_reward(name_obj_A, name_obj_B)

where name_obj_A and name_obj_B are selected from {0}. This term sets a reward
for minimizing l2 distance between name_obj_A and name_obj_B so they get
closer to each other. rest_position is the default position for the palm when
it’s holding in the air.

def maximize_l2_distance_reward(name_obj_A, name_obj_B, distance=0.5)

This term encourages the orientation of name_obj to be close to the target
(specified by x_axis_rotation_radians).

def execute_plan(duration=2)

This function sends the parameters to the robot and execute the plan for
"duration" seconds, default to be 2.

def set_joint_fraction_reward(name_joint, fraction)

This function sets the joint to a certain value between 0 and 1. 0 means close
and 1 means open. name_joint needs to be select from {1}.

def reset_reward()

This function resets the reward to default values.

Example plan: To perform this task, the manipulator’s palm should move close
to object1=faucet_handle. object1 needs to be lifted to a height of 1.0.

This is the first plan for a new task.

Example answer code:

import numpy as np

reset_reward()

# This is a new task so reset reward; otherwise we don’t need it

minimize_l2_distance_reward("palm", "faucet_handle")

set_joint_fraction_reward("faucet", 1.0)

execute_plan(4)

Remember:

1. Always format the code in code blocks. In your response execute_plan should be called exactly once at the end.

2. Do not invent new functions or classes. The only allowed functions you can call are the ones listed above. Do not leave unimplemented code blocks in your response.

3. The only allowed library is numpy. Do not import or use any other library.

4. If you are not sure what value to use, just use your best judge. Do not use None for anything.

5. Do not calculate the position or direction of any object (except for the ones provided above). Just use a number directly based on your best guess.

6. You do not need to make the robot do extra things not mentioned in the plan such as stopping the robot.

The action to perform is {2} and the plan is:

{3}

Figure B.7: LLM prompt to generate MPC reward functions. {0} is the list of
objects the robot can interact with, to which we also append the word “palm"
to represent the robot hand. {1} is the list of object joints. {2} is the
high-level action the robot needs to perform and {3} is the motion plan
generated from B.5. Adapted from [59].

⬇

This is a motion plan generated for a robot:

{0}

This is a reward function generated to complete one step in the motion plan:

{1}

The function minimize_l2_distance_reward() refers to bringing two objects
close together.

The function maximize_l2_distance_reward() refers to moving two objects
further apart.

The function set_joint_fraction_reward() refers to opening or closing an
object (0 for closed, 1 for open)

The function set_obj_z_position_reward() specifies the target height of an
object.

The function set_obj_orientation_reward() specifies the target rotation of an
object.

Which step in the motion plan is the function referring to? Return the step
using <step></step> tags. If it does not refer to any of them, return
<step>-1</step>

Figure B.8: The Verifier checks that every reward function generated
corresponds to a step in the motion plan. If it does not, the function is
removed. {0} is the motion plan generated from Figure B.5 and {1} is one of
the generated reward functions (they are looped over individually).

⬇

A stationary robot arm was asked to do the following motion plan to complete
the task ’{0}’:

{1}

After which step in the motion plan will the task ’{0}’ be satisfied? First,
explain your thought then answer the step number enclosed with the tag <step>
and </step>. Opening a joint can also mean activating it depending on the
context. You must select one. If you think none of the steps does, select the
closest one.

Figure B.9: The verifier selects which step in the motion plan is considered
to be the most important. The reward function generated for that step in the
motion plan becomes labelled as the primary reward function. {0} is the action
the robot is currently doing and {1} is the motion plan.

⬇

A robot is in a simulation environment where it can interact with any object
like in the real world. The robot would like to {0} but it cannot. Is there
something in this scene preventing that, other than the robot? Assume the
robot can interact with anything. These are the names of the objects in our
scene: {1}

In a simulation, a robot wants to {0} but can’t. Is anything else, besides the
robot, blocking it? Check the objects in the scene: {1}.

Robot in a simulation wants to {0}, can’t. Something else stopping it? Objects
in scene: {1}.

A robot can engage with any item. It wants to {0} but can’t. Is an object in
this scene, apart from the robot, hindering it? Objects present: {1}

I would like to {0} but I cannot. Is there something in this scene preventing
that, other than the robot? These are the objects in the scene: {1}

I would like to {0} but I am unable to. Is there something in this scene
preventing me from doing that? Ignore the robot. These are the names of the
objects: {1}

Figure B.10: If the robot is unable to satisfy the primary reward function,
the Perceiver is queried on whether there are any obstacles in the scene. The
Perceiver is called once for each question (total of 6). Questions 2-4 and 6
were variations generated for Q1 and Q5, respectively, using ChatGPT.

⬇

We have access to the following objects in our scene: {0}

You are given a sentence describing an image of the scene, but it may have
gotten the names of the objects wrong. Does this sentence contain objects that
are not in our scene or get the names of the objects wrong? Start your answer
with yes or no.

{1}

Figure B.11: For every explanation from the Perceiver, the Verifier is called
to determine whether the explanation lists objects that do not exist in the
scene. {0} is the list of objects in the scene and {1} is the explanation from
the Perceiver. If the Verifier answers with ’yes’, the explanation is passed
to B.12 for object remapping.

⬇

We have access to the following objects in our scene: {0}

You are given a sentence describing an image of the scene, but got the names
of the objects wrong. Rewrite this sentence using the closest object(s) in our
environment:

{1}

Rules:

You can only use objects in the scene. Use your best judgement.

Figure B.12: If the Verifier identifies that the Perceiver explanation
contains objects that are not listed in the scene, the Verifier rewrites the
explanation ({1}) using the closest objects in our scene ({0}).

⬇

The stationary robot arm would like to {0} but it cannot. Here are possible
reasons why based on images of the scene:

{1}

Based on the above explanations, what are the top reason(s) why the robot
cannot {0}? List each reason on a separate line, enclosed with the tag
<reason> </reason>. Provide up to two reasons. Be as succinct as possible. You
must not include any reasons related to the robot, only reasons related to
objects in the scene.

Figure B.13: The Planner receives all explanations from the Perceiver ({1},
see B.10 -B.12) and summarizes them into key reasons explaining why the robot
could not do the action {0}.

⬇

<Prompt from B.3>

One or more previous attempts failed. Below are the details.

\---------------------------------- attempt #1
\----------------------------------

This attempt failed when executing ’\\{0\\}’.The plan failed because the robot
was not able to execute this action: ’\\{1\\}’. This was identified as a
possible reason the action failed: ’{2}’.

...

\---------------------------------- attempt #_R_
\----------------------------------

...

\---------------------------- end of failed attempts
\----------------------------

Reminder to propose a different plan than the above failed attempts.

Figure B.14: If the robot does not succeed in performing an action, the
Planner is able to replan how the robot does the action by providing the
failure reason(s) from the Perceiver from the _R_ failures reasons. Example
for {0}: ’Place the red_cube on the crate’. Example for {1}: ’Place the
red_cube on the crate (incidentally the action is the same as the overall
goal, but it doesn’t have to be)’. Example for {2}: ’The most probable reason
why the robot cannot place the red_cube on top of the crate is that the yellow
cube is currently on top of the crate, which would prevent the robot from
doing so.’

⬇

You are a robot in the process of executing this plan, with the overall goal
to ’{0}’:

{1}

You are currently performing this action: ’{2}’. You have access to a
perception model that can answer your questions related to vision.

{3}

What question do you want to ask the perception model in order to get the
answer to ’{2}’? You can ask up to two questions. You don’t have to ask if the
information is already sufficient. Avoid asking the vision model to compare
things. Enclose each of your questions with the tag <question> </question>.

Figure B.15: If the action requires calling the Perceiver, this prompt is used
to determine what questions the Planner wants to ask the Perceiver. {0} is the
overall goal, {1} is the high-level plan, {2} is the action the robot is
currecntly performing, {3} are the observed objects in the scene.

⬇

What type of question is this asking perception model: ’{0}’? Choose your
answer from [OBJECT_PRESENCE, OBJECT_ATTRIBUTE, NEITHER]

Figure B.16: Prompt to field what category of question the Planner wants to
ask the Perceiver model. {0} is the output from B.15.

⬇

<Output from B.15. The names of the objects in our scene are: {0}. {1}
\end{lstlisting}

\caption{Perceiver query on information about the state of objects in the
scene from \ref{fig:llm_vlm_question}. States are related to object presence
or object attributes. Examples of queries: ‘Look for the apple in the
cabinet’, ‘Check the color of the crate’.}

\label{fig:vlm_object_state}

\end{figure}

\FloatBarrier

\begin{figure}[!htb]

\begin{lstlisting}[backgroundcolor=\color{lightblue}, language=TeX]

A robot was tasked to do this plan:

{0}

The robot is currently doing this action: ’{1}’.

To do the action, the robot asked a perception model the following questions
(Q) and got the answers (A):

{2}

After receiving this answer, has the robot completed the action ’{1}’? Begin
your answer with yes or no. If your answer begins with no, write the remaining
action that needs to be completed using <Action></Action>
tags.\end{lstlisting}

\caption{After the Perceiver has provided information, the Planner is asked to
determine whether the action is completed. If not, it generates MPC reward
functions to finish the action (see
\ref{fig:llm_motion_plan}-\ref{fig:llm_new_plan}).}

\label{fig:vlm_completion}

\end{figure}

\FloatBarrier

\begin{figure}[!htb]

\begin{lstlisting}[backgroundcolor=\color{lightblue}, language=TeX]

<Prompt from B.3>

One or more previous attempts failed. Below are the details.

\---------------------------------- attempt #1
\----------------------------------

The proposed plan was:

<thought>{0}</thought>

[start plan]

{1}

[end plan]

The plan failed because {2}.

...

\---------------------------------- attempt #_P_
\----------------------------------

...

\---------------------------- end of failed attempts
\----------------------------

Reminder to propose a different plan than the above failed attempts.

Figure B.17: If by executing the plan or the replan and the goal is still not
accomplished, the Planner is prompted to generate a new plan using the prompt
in Figure B.3. The Planner is allowed to generate a new plan _P_ times. before
the task is considered undoable.

##  Appendix C Error Evaluation

Here we provide three error cases of RePLan and their analyses.

#### Case 1 (from Cabinet-Blocked, Figure 5(a))

The robot tried to open the cabinet door but failed and the Perceiver gave a
correct diagnosis to remove the bar from the handle. However, when generating
the reward functions to remove the bar, the LLM selected the wrong primary
reward function, as demonstrated below:

⬇

reset_reward()

minimize_l2_distance_reward("palm", "red_block_right_side",
primary_reward=True)

maximize_l2_distance_reward("red_block_right_side",
"target_position_in_cabinet")

execute_plan()

The correct primary function should be the second one. As a result, MPC ended
prematurely before the robot could remove the bar. The robot was not able to
remove the bar in the following steps.

#### Case 2 (from Kitchen-Explore, Figure 5(b)))

The robot tried to open the microwave door but failed due to a kettle
obstructing the path. The Perceiver gave five diagnoses, of which three
claimed that the kettle was blocking the way, one claimed the cabinet door was
blocking the way, and one did not give any conclusive diagnosis. The summary
LLM concluded that it was the cabinet door that blocked the action. The robot
went on to interact with the cabinet and never removed the kettle.

#### Case 3 (from TwoCube-Color, Figure 5(c)))

The high-level planner proposed a plan where the first step was _“Identify the
cube with the same colour as the crate”_. This was a VLM action. However, the
LLM proposed to ask the VLM _“Which cube has the same colour?”_ , which was a
bit vague, resulting in the VLM answering _“The same color cube is the yellow
cube and the yellow cube in the middle of the blue cube group.”_. This answer
did not provide the necessary information to solve the task. Eventually, the
robot put the wrong cube on the crate.

#### Case 4 (from TwoCube-Blocked, Figure 5(d)))

After the robot was not able to execute the task “Place the red cube on the
crate", the Perceiver was called to help identify any issues. The Perceiver’s
diagnoses all mentioned that the robot was holding the red cube but did not
identify the yellow cube as blocking the crate, and so the Planner’s summary
of the VLMs diagnoses was: “Based on the given information, the most probable
reason why the robot cannot place the red_cube on the crate is because it is
currently holding the red cube." However, it’s also important to note that
TwoCube-Blocked used GPT-4V which severly limits the number of ouput tokens
from the model, and so a lot of explanations were cut off (for example: “In
the image provided, the robot is holding the red cube, which is currently").

###  C-A LLM Diagnosis with Ground-truth Data

An alternative way to diagnose errors is to input the simulator ground-truth
state of the objects to an LLM and ask the LLM to infer a possible reason. We
show a scenario here where the cabinet door is locked by a red bar (from
Cabinet-Blocked) and test the capability of LLM in this regard. The prompt we
use is:

⬇

A robot is in a simulation environment where it can interact with any object
like in the real world. The robot would like to open the wooden cabinet but it
cannot. Is there something in this scene preventing that, other than the
robot? Assume the robot can interact with anything. These are the names of the
objects in our scene: wooden cabinet, red block, yellow cube.

The data of the objects are:

{

"joints": {

"wooden_cabinet": 0.16235716765387215

},

"objects": {

"wooden_cabinet_handle": [0.08292539, 0.50163332, 0.21],

"target_position_in_wooden_cabinet": [0.2, 0.85, 0.0775],

"red_block_right_side": [0.2158331, 0.55110136, 0.15143364],

"yellow_cube": [-0.24545329, 0.01631876, 0.03599184],

"palm": [0.2231031, -0.00875834, 0.76900653]

}

}

And the response from GPT-4 is:

⬇

The data does not indicate any object that would prevent the robot from
opening the cabinet. There’s no information about any object blocking the
cabinet or its handle. The problem could be due to the robot itself as it
might not possess the necessary functionality to interact with the objects as
intended. The cabinet’s joint also has a non-zero value, which might mean it’s
not locked or stuck. Hence, there is likely a mechanical limitation with the
robot or an issue with its programming.

This example demonstrates that LLM is not capable of directly inferring
spatial relations of objects through ground-truth data.

##  Appendix D Additional Experiments

###  D-A VLM Ablation

We conduct a VLM ablation study for Tasks 3-4 using two open-source models
(Qwen-VL-Chat7B [2] and Llava-1.5-7B [33]) and
GPT-4V333https://openai.com/research/gpt-4. The results are shown in Table IV.
We measure (a) their ability to recognize objects, reported as a percentage of
the number of object successfully detected in the scene (column 2), (b) their
ability to reason about obstacles in the scene, reported as a percentage of
times the correct reason was identified out of all the prompts it was given in
Figure B.10, and (c) the ability for the LLM to consilidate the outputs in (b)
into a single summary reason of why the robot cannot perform an action given
its knowledge about the scene. We found that Qwen somtimes struggled with
object detection of smaller objects, and so we coupled it with Segment
Anything Model (SAM) [26] to first segment the objects in the scene. We found
that all models did well with object recognition (except for Qwen when not
used with SAM). For object reasoning, Qwen + SAM was able to get the correct
scene error in 50-67% of the prompts it was given, and then the LLM was able
to summarize the prompts to generate the correct error reason overall. The
reason the LLM was able to do this despite the VLM not giving perfect answers
was that the remaining VLM answers pertained where the robot was located or a
general comment about the objects in the scene. Llava tended to reply that it
was unable to reason because the scene was a simulation and not real life.
GPT-4V had the best overall performance in all categories, but API calls to it
are still restricted. All ablations were repeated over 5 runs.

| Models
---|---
Scenarios | Qwen + SAM | Qwen | Llava | GPT-4V
VLM object detection | $100\%$ | $66\%$ | $100\%$ | $100\%$
VLM Reasoning | $67\%$ | $0\%$ | $23\%$ | $100\%$
LLM summarization and consistency step | $100\%$ | $0\%$ | $100\%$ | $100\%$
(a) Cabinet-Blocked

| Models
---|---
Scenarios | Qwen + SAM | Qwen | Llava | GPT-4V
VLM object detection | $100\%$ | $100\%$ | $100\%$ | $100\%$
VLM Reasoning | $50\%$ | $66\%$ | $40\%$ | $83\%$
LLM summarization and consistency step | $100\%$ | $100\%$ | $20\%$ | $100\%$
(b) Kitchen-Explore

Table IV: VLM ablation study.

###  D-B GPT-4V Experiments

We run the initial High-Level Planner prompt (Prompt B.3) using
GPT-4V444https://openai.com/research/gpt-4v-system-card on initial task scenes
to investigate the ability of GPT-4V to find the correct solution in a single
step.

(a) Prompt B.3 to GPT-4V.

(b) GPT-4V output.

Figure D.1: GPT-4V high-level plan for moving the yellow cube inside the
wooden cabinet (Cabinet-Blocked in Section A-A).

(a) Prompt B.3 to GPT-4V.

(b) GPT-4V output.

Figure D.2: GPT-4V high-level plan for placing the cube with the same color as
the crate on the crate (TwoCube-Color in Section A-D).

###  D-C Comparison with TAMP Experiments

To compare the RePLan with a TAMP framework, we use Planning Domain Definition
Language (PDDL) [38] to define the domain of Cabinet Tasks 1-3 in Table A-D in
Section A as follows:

⬇

1(define (domain pick-place-domain)

2 (:requirements :strips :typing :negative-preconditions :conditional-effects)

3

4 ;; Define the object and its possible locations

5 (:types

6 object

7 location

8 conf

9 robot

10 door cabinet cube block - object

11 area

12 remove_area cabinet_area - area

13

14 )

15 ;; define constants

16 (:constants

17 cube_loc cabinet_loc block_loc remove_loc init_loc door_loc open_door_loc -
location

18 init_conf robot_conf_cube robot_conf_block robot_conf_cabinet
robot_conf_remove robot_conf_door open_door_conf - conf

19 robot - robot

20 door - door

21 cabinet - cabinet

22 cube - cube

23 block - block

24 remove_area - remove_area

25 cabinet_area - cabinet_area

26 )

27

28 ;; Define predicates

29 (:predicates

30 (at ?obj - object ?loc - location) ; the Object at location loc

31 (grasped ?obj -object) ; the object is grasped

32 (at_conf ?conf -conf) ; the robot is at conf configuration

33 (rob_at_loc ?loc -location) ; the robot is location loc

34 (is_free ?rob - robot) ; the robot hand is free

35 (in ?loc -location ?area -area) ; sampling and certifying the loc location
is inside an area type

36 (is_closed ?door - door) ; the door is closed

37 (is_blocked ?door) ; the door is blocked

38 (is_moveable ?init_conf -conf ?final_loc -location ?final_conf - conf)

39 ; certifying the robot can move from the initial condition to the goal
location/pose with the sampled final configuration

40

41 )

42

43 ;; Define actions

44 (:action remove

45 :parameters (?block - block ?door - door ?rob - robot ?init_conf
?final_conf - conf ?init_loc ?final_loc -location)

46 :precondition (and

47 (is_blocked ?door)

48 (grasped ?block)

49 (not (is_free ?rob))

50 (at_conf ?init_conf )

51 (at ?block ?init_loc)

52 (rob_at_loc ?init_loc)

53 (in ?final_loc remove_area)

54 (is_moveable ?init_conf ?final_loc ?final_conf )

55 )

56 :effect (and

57 (rob_at_loc ?final_loc )

58 (not (rob_at_loc ?init_loc))

59 (at ?block ?final_loc)

60 (at_conf ?final_conf )

61 (is_free ?rob)

62 (not (grasped ?block))

63 (not (is_blocked ?door))

64 (not (at_conf ?init_conf ))

65 )

66 )

67

68 (:action place

69 :parameters (?cube - cube ?door - door ?rob - robot ?init_conf ?final_conf
- conf ?init_loc ?final_loc -location)

70 :precondition (and

71 (not (is_closed ?door))

72 (not (is_blocked ?door))

73 (grasped ?cube)

74 (not (is_free ?rob))

75 (at_conf ?init_conf )

76 (at ?cube ?init_loc)

77 (rob_at_loc ?init_loc)

78 (in ?final_loc cabinet_area)

79 (is_moveable ?init_conf ?final_loc ?final_conf )

80 )

81 :effect (and

82 (rob_at_loc ?final_loc )

83 (at ?cube ?final_loc)

84 (at_conf ?final_conf )

85 (not (at_conf ?init_conf ))

86 (is_free ?rob)

87 (not (grasped ?cube))

88 )

89 )

90

91

92 (:action pick

93 :parameters (?init_conf ?final_conf -conf ?obj - object ?loc - location
?rob -robot)

94 :precondition (and

95 (at_conf ?init_conf )

96 (at ?obj ?loc)

97 (is_free ?rob )

98 (is_moveable ?init_conf ?loc ?final_conf )

99 )

100 :effect (and

101 (at_conf ?final_conf)

102 (not (at_conf ?init_conf))

103 (rob_at_loc ?loc)

104 (not (is_free ?rob))

105 (grasped ?obj)

106 )

107 )

108

109 (:action open

110 :parameters (?init_conf -conf ?door -door ?rob -robot )

111 :precondition (and

112 (is_closed ?door)

113 (not (is_blocked ?door))

114 (grasped ?door)

115 (not (is_free ?rob))

116 (at_conf ?init_conf )

117 (is_moveable ?init_conf open_door_loc open_door_conf )

118 )

119 :effect (and

120 (not (is_closed ?door))

121 (not (grasped ?door))

122 (is_free ?rob)

123 (not (at_conf ?init_conf ))

124 (at_conf open_door_conf)

125 (rob_at_loc open_door_loc)

126 (at door open_door_loc)

127 )

128 )

129

130 (:action is_moveable_cube

131 :parameters ()

132 :precondition (and

133 (not (is_closed door))

134 (not (is_blocked door))

135 )

136 :effect (is_moveable robot_conf_cube cabinet_loc robot_conf_cabinet)

137 )

138

139)

Snippet 1: PDDL domain definition for Tasks 1-3.

Tasks 1-3 problem definition are written as follows:

⬇

1

2(define (problem pick-place-problem)

3 (:domain pick-place-domain)

4

5 ;; Define objects

6 (:objects

7

8 )

9

10 ;; Define initial state

11 (:init

12 (at cube cube_loc)

13 (at door door_loc)

14 (at block block_loc)

15 (at_conf init_conf)

16 (rob_at_loc init_loc)

17 (in remove_loc remove_area)

18 (in cabinet_loc cabinet_area)

19 (is_free robot)

20 (is_moveable init_conf cube_loc robot_conf_cube)

21 (is_moveable open_door_conf cube_loc robot_conf_cube)

22 (is_moveable robot_conf_block remove_loc robot_conf_remove)

23 (is_moveable robot_conf_remove door_loc robot_conf_door)

24 (is_moveable init_conf block_loc robot_conf_block)

25 (is_moveable init_conf door_loc robot_conf_door)

26 (is_moveable robot_conf_door open_door_loc open_door_conf)

27

28 ;=====================================================================

29 ;; commenting the following two initial conditions can change the robot
behavior greatly in terms of task plan

30 ;; easy mode (Task 1): comment both of the following lines [comment
(is_closed door) and (is_blocked door)]

31 ;; hard mode (Task 2): comment the second condition [(is_blocked door)]

32 ;; expert mode (Task 3): keep both of the following conditions uncommented.

33 ;=====================================================================

34

35 ; door is closed at the begining

36 (is_closed door)

37 ; door is blocked at the begining

38 (is_blocked door)

39

40

41 )

42

43 ;; Define goal

44 (:goal

45 (and

46 ;; only picking the cube

47 ; (grasped block)

48

49 ;; picking and placing the cube inside the cabinet

50 (at cube cabinet_loc)

51 ; (at block remove_loc)

52 ; (grasped door)

53

54

55 )

56 )

57)

Snippet 2: PDDL problem definition for Cabinet Tasks 1-3.

The cabinet task problem is solved using the lpg-td [16] solver from the
planutils library555https://github.com/AI-Planning/planutils.

As seen in Snippets LABEL:lst:pddldomain and LABEL:lst:pddlproblem, even for
three cabinet tasks 1-3, domain and problem definitions require careful and
laborious attention. Task-solving details are outlined in Snippets with blue
and olive colours. While our method discovers information through interaction
and reasoning over perceiver’s feedback, the PDDL solver relies on ground
truth (highlighted in blue) and rules (example in olive) provided by the user
for problem resolution.

For the same task, PDDLStream [14] offers an alternative using a Task and
Motion Planning (TAMP) framework. Rather than human-grounded truth
information, a motion planning framework certifies predicates through streams.
However, this requires user-defined rules for success or failure and a motion
planner. Solving long-horizon problems with PDDLStream may become
computationally expensive [25].

RePLan can robustly solve long-horizon multi-stage problems through
interaction with the environment and reasoning based on the perceiver’s
feedback. This capability enables RePLan to uncover underlying rules without
the need for an additional domain description and ground truth information.