# Language Models as
Zero-Shot Trajectory Generators

Teyun Kwon, Norman Di Palo, Edward Johns
The Robot Learning Lab
Department of Computing
Imperial College London
{john.kwon20, n.di-palo20<EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) have recently shown promise as high-level
planners for robots when given access to a selection of low-level skills.
However, it is often assumed that LLMs do not possess sufficient knowledge to
be used for the low-level trajectories themselves. In this work, we address
this assumption thoroughly, and investigate if an LLM (GPT-4) can directly
predict a dense sequence of end-effector poses for manipulation skills, when
given access to only object detection and segmentation vision models. We study
how well a single task-agnostic prompt, without any in-context examples,
motion primitives, or external trajectory optimisers, can perform across 26
real-world language-based tasks, such as “open the bottle cap” and “wipe the
plate with the sponge”, and we investigate which design choices in this prompt
are the most effective. Our conclusions raise the assumed limit of LLMs for
robotics, and we reveal for the first time that LLMs do indeed possess an
understanding of low-level robot control sufficient for a range of common
tasks, and that they can additionally detect failures and then re-plan
trajectories accordingly. Videos, code, and prompts are available at:
https://www.robot-learning.uk/language-models-trajectory-generators.

Figure 1: A selection of the tasks we use to study if a single, task-agnostic
LLM prompt can generate a dense sequence of end-effector poses, when given
only object detection and segmentation models, and no in-context examples,
motion primitives, pre-trained skills, or external trajectory optimisers.

##  1 Introduction

In recent years, Large Language Models (LLMs) have attracted significant
attention and acclaim for their remarkable capabilities in reasoning about
common, everyday tasks [1, 2, 3, 4]. This widespread recognition has since led
to efforts in the robotics community to adopt LLMs for high-level task
planning [5]. However, for low-level control, existing proposals have relied
on auxiliary components beyond the LLM, such as pre-trained skills, motion
primitives, trajectory optimisers, and numerous language-based in-context
examples (Fig. 2). Given the lack of exposure of LLMs to physical interaction
data, it is often assumed that LLMs are incapable of low-level control [6, 7,
8].

However, until now, this assumption has not been thoroughly examined. In this
paper, we now investigate if LLMs have sufficient understanding of low-level
control to be adopted for zero-shot dense trajectory generation for robot
manipulators, without the need for the aforementioned auxiliary components. We
provide an LLM (GPT-4 [2]) with access to off-the-shelf object detection and
segmentation models, and then require all remaining reasoning to predict a
dense sequence of end-effector poses to be performed by the LLM itself. We
also require that the same task-agnostic prompt is used for all tasks, without
any in-context examples.

Given these requirements, we studied if a single prompt could be designed to
solve a range of tasks taken from the recent literature, such as “open the
bottle cap” and “wipe the plate with the sponge”. And through this
investigation, we uncovered the underlying principles and strategies that
empower LLMs to navigate the complexities of robot manipulation.

Consequently, our contributions are threefold: (1) We demonstrate, for the
first time, that a pre-trained LLM, when provided with only an off-the-shelf
object detection and segmentation model, can guide zero-shot a robot
manipulator by outputting a dense sequence of end-effector poses, without the
need for pre-trained skills, motion primitives, trajectory optimisers, or in-
context examples. (2) We present several ablation studies which shed light on
what techniques and prompts lead to the emergence of these capabilities. (3)
We study how, by analysing the trajectory of objects across an image, LLMs can
also detect if a task has failed and subsequently re-plan an alternative
trajectory.

##  2 Related Work

While previous works have made significant strides in leveraging LLMs for
various aspects of robotic control [5], several limitations and dependencies
on external modules persist. The core motivation of our work is to investigate
whether these limitations are inherent, or if LLMs can be deployed for low-
level control, going from language to a dense sequence of end-effector poses.
In this section, we provide an overview of the relevant literature and
highlight key distinctions between prior approaches and our research focus.

Figure 2: A taxonomy of requirements of LLM-based zero-shot methods from the
recent literature. We investigate whether LLMs can be used to generate
trajectories for a range of tasks, assuming access to only vision models.

Predefined Motion Primitives: A subset of prior works, including Code as
Policies [9] and ChatGPT for Robotics [10], have predominantly employed LLMs
to address the high-level planning aspect of robotic control. These approaches
often rely on predefined movement primitives or pre-trained skills (such as
SayCan [8]) to execute lower-level actions, thereby only partially solving the
control stack. In contrast, our investigation aims to push these boundaries by
demonstrating that LLMs can delve deeper into the control stack, predicting
all lower-level actions for the robot autonomously, in the form of a dense
sequence of poses for the robot end-effector to follow to complete a given
task.

External Trajectory Optimisers: VoxPoser [6] and Language to Rewards [7] have
explored the use of LLMs to generate high-reward regions for robot movement,
significantly contributing to trajectory planning. However, these methods
still necessitate external trajectory optimisers to compute a trajectory, such
as cost and reward functions used to evaluate randomly sampled trajectories
along with Model Predictive Control (MPC) [6]. Our research deviates from this
paradigm by showcasing that LLMs are capable of autonomously shaping and
generating their own trajectories, either as lists of end-effector positions
and orientations predicted as language tokens, or as the prediction of code
snippets that can then generate these trajectories, both of which remove the
reliance on external trajectory optimisers.

Use of In-Context Examples: Previous approaches such as VoxPoser [6], Code as
Policies [9], and SayCan [8] have relied heavily on providing in-context
examples to the LLM input. However, these methods can encounter challenges
when extrapolating beyond the demonstrated tasks. In contrast, our research
illustrates that, even when relying on their internal understanding alone,
LLMs exhibit the capacity to comprehend and solve a diverse range of
manipulation tasks, thus broadening the scope of applicability and
adaptability in the real world and reducing the reliance on human expertise.

Robotics-Specific Pre-Training and Fine-Tuning: Recently, Brohan et al. [11]
and Driess et al. [12] demonstrated that a Vision Language Model (VLM) [13]
can be combined with a large robotics-related dataset of actions to enable
zero-shot language-conditioned control. However, both the VLM weights and the
compute capacity to fine-tune them are unavailable to most research groups:
therefore, we focus our investigation on widely available LLMs and vision
models [2], and tackle many tasks from the recent literature that require
similar or better dexterity than the ones included in the work by Brohan et
al. [11].

In summary, while prior research has made notable strides in harnessing LLMs
for robotics, often focusing on specific components of the control stack or
relying on external modules, our investigation represents a departure from
these paradigms. We explore the potential of LLMs to provide end-to-end
solutions, encompassing the entire control stack from language comprehension
to the prediction of dense sequences of end-effector poses. Our investigation
expands the known capabilities of LLM-guided robotics, offering a promising
avenue for enhancing human-robot interaction and task execution.

##  3 Problem Formulation

We investigate if an LLM (GPT-4 [2]) can predict a dense sequence of end-
effector poses to solve a range of manipulation tasks. We now explain what the
assumptions and constraints are in our investigation, followed by details of
our real-world experimental setup, and the tasks used for evaluation. Given
this background, we then present our investigation and its results in Sec. 4.

Figure 3: Example wrist-camera observations received by the robot at the start
of each task, and their corresponding task instructions.

Assumptions and Constraints: We design a task-agnostic prompt to study the
zero-shot control capabilities of LLMs, with the following assumptions: (1) no
pre-existing motion primitives, policies or trajectory optimisers: the LLM
should output the full sequence of end-effector poses itself; (2) no in-
context examples: we study the ability of LLMs to reason about tasks given
their internal knowledge alone, and no part of any task is explicitly
mentioned in the prompt, either in the form of examples or instructions; (3)
the LLM can query a pre-trained vision model to obtain information about the
scene, but should autonomously generate, parse and interpret the inputs and
outputs; (4) no additional pre-training or fine-tuning on robotics-specific
data: we focus our research on pre-trained and widely available models, so
that our work can easily be reproduced even with limited compute budget.

Real-World Experimental Setup: We run our experiments on a Sawyer robot
equipped with a 2F-85 Robotiq gripper. We use two Intel RealSense D435 RGB-D
cameras, one mounted on the wrist of the robot, and the other fixed on a
tripod, to observe the environment. The wrist-mounted camera captures a top-
down view of the environment at the beginning of an episode (Fig. 3), which is
used by a vision model if queried by the LLM. We utilise a pre-trained object
detection model, LangSAM [14] (based on Grounding DINO [15] and Segment
Anything [16]), and whenever the LLM calls detect_object, we automatically
calculate 3-D bounding boxes of the queried objects from the segmentation maps
returned by LangSAM using the camera calibration, and provide the bounding
boxes to the LLM. The LLM then leverages this visual understanding of the
environment to predict a sequence of 4-D end-effector poses (3 dimensions for
position, 1 dimension for rotation about the vertical axis), as well as either
open_gripper or close_gripper commands. This is then executed by the robot in
an open loop, using a position controller to move sequentially between each
pose, hence producing a full trajectory. During this execution, we use XMem
[17] to track the segmentation maps over the entire duration of the task,
which is then later used for detecting if the task was successful or not.

Task Selection: In pursuit of objectivity, we opt to benchmark our zero-shot
LLM-guided robotic control against a challenging repertoire of everyday
manipulation tasks. We recreated 26 everyday manipulation tasks from recent
robotics papers published at leading conferences [8, 18, 19, 20], often
tackled by relying on hundreds of manual demonstrations. This serves as a
representative benchmark of real-world challenges, mirroring the complexity
and diversity of the tasks encountered in contemporary robotics research. We
choose tasks which semantically cover the most representative tabletop robot
behaviours expressed in these papers, and success criteria are human-evaluated
and designed to mirror those proposed in the original papers. For each
combination of task and method in the following experimental sections, we
calculate the success rate over 5 randomised positions and orientations of the
objects. The task description is provided in natural language to the LLM,
after which no additional human feedback or intervention is allowed. The full
list of tasks is shown in Fig. 6, and videos are available at
https://www.robot-learning.uk/language-models-trajectory-generators.

Figure 4: An overview of the pipeline. (1) The main prompt along with the task
instruction is provided to the LLM, from which it (2) generates high-level
natural language reasoning steps before outputting Python code (3) to
interface with a pre-trained object detection model and execute the generated
trajectories on the robot. (4) After task execution, an off-the-shelf object
tracking model is used to obtain 3-D bounding boxes of the previously detected
objects over the duration of the task, which are then provided to the LLM as
numerical values to detect whether the task was executed successfully or not.

##  4 Prompt Development

Full Prompt: The core motivation of our work is to investigate whether LLMs
can inherently guide robots with minimal dependence on specialised external
models and components, in order to provide effective and useful insights for
the robotics community. Through this investigation, we designed a single task-
agnostic prompt for a range of everyday manipulation tasks, which does not
require any in-context examples or task-specific guidance. Fig. 4 illustrates
the main information flow in our framework, showing how the task-agnostic
prompt interfaces with the vision models and the robot.

Through our experiments outlined in this section, our final prompt formulation
instructs the LLM to self-summarise and decompose the predicted plan into
steps, before generating Python code which, when run by a standard Python
interpreter, outputs a dense sequence of poses for the end-effector to follow;
this pipeline resulted in the best performance across those we experimented
with. We include details fundamental to all tasks, such as coordinate
definitions, as well as functions available for the LLM to call, such as
detect_object, which returns the calculated 3-D bounding boxes of the queried
objects directly to the LLM. We also include instructions which aim to improve
the correctness and reliability of the generated trajectories, such as
guidance on step-by-step reasoning, code generation, and collision avoidance.
The full prompt is shown in Appendix A.

Prompt Ablations: During the design of this full prompt, we identified several
challenges when using LLMs for low-level control, without access to other
external dependencies. In this section, we now outline these challenges which
motivated the final design of the prompt, and accompany them with results from
ablation studies conducted across a diverse set of tasks (Fig. 5), where
certain parts of the full prompt were removed. We choose a subset of the 26
original tasks for the ablation studies, which we list in Appendix E, that
still capture the various manipulation challenges in the full set. The ablated
components of the full prompt are shown in Appendix B.

Figure 5: We investigate the effect of removing parts of the main prompt on
task success rates.

(1) LLMs often require step-by-step reasoning to solve tasks. Prior work has
shown that the reasoning capabilities of LLMs can be improved by asking them
to break down the task in a step-by-step manner [21, 22], and adopting this
strategy, we prompt the LLM (1) to break down the trajectory into a sequence
of sub-trajectory steps, and (2) to include in the plan when to lower the
gripper to make contact with an object. We find that, without including these
step-by-step reasoning prompts, the LLM often omits key trajectory steps
required to execute the task successfully, such as opening or closing the
gripper, or aligning the gripper to be parallel to the graspable side of the
object, which are not stated explicitly in the prompt. Indeed, the first three
columns in Fig. 5 show that prompting the LLM to think step by step resulted
in the highest performance increase.

(2) LLMs can be prone to write code which results in errors, both
syntactically and semantically. While much improvement has been made in the
domain of code generation by LLMs [23, 2], their outputs can still throw
errors, as well as produce undesirable results when executed. In order to
mitigate this, and inspired again by the power of LLMs performing an internal
monologue with natural language reasoning, we prompt the LLM to document any
functions it defines, with their expected inputs and outputs, and their data
types. In addition, we include a prompt instructing the LLM to define reusable
functions for common motions (for example, linear trajectory from one point to
another), to prevent instances where, as a notable example, it would hard-code
the height of the gripper inside a function definition, and reuse that
function for another sub-trajectory step which should have been executed at a
different height. Similarly, we prompt the LLM to name each sub-trajectory
step variable with a number to relate it to each of the steps in the high-
level trajectory plan, and to minimise the chance of omitting a sub-trajectory
step. The effects of removing these prompt components are, again, noticeable
(fourth and fifth columns in Fig. 5).

(3) LLMs are trained on limited grounded physical interaction data. Due to the
scarcity of grounded physical interaction data in their training corpora [24],
LLMs often fail to take into account possible collisions between the objects
being manipulated. We therefore prompt the LLM to pay attention to the
dimensions of the objects and to generate additional waypoints and sub-
trajectories, which could help with avoiding collisions. We also include in
the prompt a specific phrase which we noticed during our investigation was
being used frequently by the LLM for its internal reasoning (“clear objects
and the tabletop”). Our experiments show that, while removing this particular
phrase from the collision avoidance prompt lowered performance (sixth column
in Fig. 5), LLMs do possess some inherent understanding of possible collisions
between different objects, as they performed well even after removing the
entire collision avoidance prompt (tenth column in Fig. 5).

(4) LLMs often fail to reason about complex trajectory shapes. In a manner
similar to the first challenge, we employ a two-step strategy, where
initially, we explicitly ask the LLM to generate a textual description of the
shape of the motion trajectory as internal reasoning (for example, shaking
involves a sinusoidal motion), before outputting the actual sequence of poses
required to execute this trajectory (in contrast to Challenge (1), where we
prompted the LLM to output a more detailed step-by-step trajectory plan). This
has been shown to be beneficial in prior work [7], and indeed this result is
also reflected in the eighth column in Fig. 5.

(5) LLMs often fail to reason about how to interact with objects. In our
experiments, we found that LLMs often simplified and failed to reason about
more intricate details of object interaction, such as realising that some
objects require interaction with a specific part (for example, the rim of a
bowl, or the handle of a pan). In order to enable the LLM to detect the most
suitable object part to interact with, we prompt it to describe the object
part in high-level natural language, and we see in the ninth column in Fig. 5
that this results in more tasks being executed successfully.

Figure 6: Success rates of the best-performing prompt in our investigation on
26 manipulation tasks.

Full Prompt Evaluation: Here, we now investigate the LLM’s ability to solve
zero-shot a range of manipulation tasks, by evaluating the full prompt on the
full set of tasks taken from the recent literature. These tasks and their
success rates are presented in Fig. 6. Remarkably, our experiments reveal that
LLMs, when equipped with an off-the-shelf vision model and no external motion
primitives, policies, or trajectory optimisers, do indeed exhibit notable
proficiency in executing the majority of these tasks, by directly predicting a
dense sequence of end-effector poses. In the original papers from which these
tasks are selected [19, 18, 20], solving these tasks required numerous human
demonstrations. As such, these findings underscore the potential of LLMs as
intuitive and versatile guides for robotic manipulation that minimise the need
for human time and supervision.

##  5 Further Investigations

Figure 7: Given the full main prompt and the user input command, the LLM first
outputs a high-level natural language self-summarisation of the trajectory
plan, before generating either code which computes and executes the
trajectory, or the final trajectory directly as a list of numerical values.

In this section, we detail further ablation studies conducted regarding the
modality of the trajectory generation, the extent to which each output
modality is executable by the robot, and the ability of LLMs to detect whether
a task was executed successfully or not and subsequently re-plan the
trajectory. We list the subset of the original tasks used for this set of
ablation studies in Appendix E. The prompts for these ablation studies are
shown in Appendix C.

Figure 8: (A) We compare different modes for the trajectory output. (B) We
measure the percentage of control outputs from the LLM that are directly
executable by the robot. (C) We compare different modes for controlling the
gripper. (D) We demonstrate the ability of LLMs to detect failures and re-plan
autonomously.

(1) How should the final trajectory be represented? In this set of
experiments, we explore the optimal way for the LLM to output the sequence of
end-effector poses. Specifically, we conduct ablation studies to evaluate
whether this should be represented as a list of numerical values or as code
for trajectory generation. Fig. 7 shows the distinction between these two
different output modes.

The results, summarised in Fig. 8 A, offer valuable insights. Notably, our
investigation shows that outputting code that generates the trajectory
outperforms predicting the trajectory directly as an explicit list of
numerical poses for the end-effector to follow, represented as language tokens
(Fig. 7). In particular, we observe that representing trajectories as
numerical values or as code yields similar performance for most tasks, with
distinctions emerging in cases involving more intricate trajectories (for
example, drawing a circle or a five-pointed star), where outputting code that
generates such trajectories prevails (60% success rates for code output
compared to 10% for numerical output). This suggests a fundamental property of
LLMs for control: while not trained on physical interactions and trajectories,
their understanding of both code and mathematical / geometrical structures [2,
25] can bridge these two modes of thinking. Once the overall trajectory shape
has been identified by the LLM, while it can be challenging to follow it
directly in numbers, it is proficient at generating code that itself can
follow complex paths.

Additionally, we study whether directly generating a list of numerical poses,
or code that then generates the poses itself, leads to executable outputs more
often. Giving low-level control to the LLM poses the risk of the robot
receiving wrongly formatted outputs that cannot be executed by the robot.
Therefore, in this ablation, we investigate how often the output of the LLM is
formatted such that it is executable by the robot. We include prompts
instructing the LLM to follow a specific format for the trajectory generation
(for the former, we require a list between the ⟨trajectory⟩ and ⟨/trajectory⟩
tags without any Python functions, and for the latter, we require any Python
code to be between the ```python and ``` tags). Given the output of the LLM,
if an error is thrown during automatic parsing according to this format, we
provide the LLM with the error message and ask it to correct the output, for
up to three times. Measuring the percentage of executable outputs (Fig. 8 B)
demonstrates that outputting code results in 100% of executable trajectories,
while direct numerical values cannot be parsed even after three self-
corrections for some episodes.

(2) How should the LLM output the gripper action? We also investigate the
optimal way of letting the LLM control the gripper open or close action: we
compare using a binary variable $a\in\\{0,1\\}$ or explicit functions
open_gripper, close_gripper. Our results, in Fig. 8 C, demonstrate that the
LLM achieves better performance when using explicit functions, while using a
binary variable leads to more errors. A notable failure case stemmed from the
LLM hard-coding the gripper state to be open in one of the functions it
defined for itself, such that when the same function was then used to generate
the object approach-and-lift sub-trajectory steps, the gripper failed to close
and grasp the object. Having explicit functions to open and close the gripper,
on the other hand, allowed a decoupling of these fundamental actions and
enabled the correct functions to be called at any time during the overall
trajectory generation plan.

Figure 9: Our experiments demonstrate that LLMs can interpret the trajectories
of objects to detect successful and unsuccessful episodes. Figure 10: (1) The
LLM attempts to grasp the bowl at its centroid, recognises failure, and (2)
proposes a new trajectory. (3) On its third attempt after re-planning again,
it successfully grasps the bowl.

(3) Can LLMs recognise unsuccessful trajectories, and adapt their plan?
Finally, we delve into the ability of LLMs to recognise and respond to
failures during task execution, as shown in Fig. 9. Our experiments
demonstrate that, by analysing the numerical trajectories of objects, LLMs can
autonomously detect failure outcomes and initiate re-planning to rectify them.
We therefore demonstrate that LLMs possess not only the ability to generate
trajectories, but also to discern whether they represent successful or
unsuccessful episodes, given the tasks requested by the user.

For each of the 5 trials of a task, when a failure is identified, the LLM
modifies the original plan to tackle the possible issue. In Fig. 8 D, we
demonstrate that this leads to a small improvement in performance, without the
need for any human intervention. As a notable example, the LLM always fails at
grasping a bowl on its first try (Fig. 6), attempting to grasp it by the
centroid (Fig. 10). Through a sequence of two re-planning iterations, however,
the LLM adapts its trajectory and then successfully grasps the bowl by its
rim, leading to an increase from $0\%$ to $20\%$ in the overall task execution
success rate. The prompts for success detection and re-planning are shown in
Appendix D.

##  6 Conclusions

Can a Large Language Model be employed for zero-shot control of a robot
manipulator, going from a task description to a dense sequence of end-effector
poses directly, without the use of in-context prompts, predefined skills, or
external trajectory optimisers? Answering this question guided our work, and
led us to design a large set of experiments to explore the abilities and
design choices that enable LLMs to take full control of a manipulator. Our
experiments encompassed 26 diverse tasks drawn from the recent literature to
provide a comprehensive benchmark.

We have demonstrated that, when provided with the right prompt, LLMs can
successfully predict dense sequences of end-effector poses for a range of
real-world manipulation tasks. This is achieved under the constraints that the
LLM must use a single task-agnostic prompt without any in-context examples,
and has access to only off-the-shelf object detection and segmentation vision
models, with no other auxiliary components. This raises the assumed limit of
the utility of LLMs for robotics, and we hope that our investigations into how
to write an LLM prompt for robots will act as a helpful guide for the
community wishing to raise this limit further.

Additional details on the prompt and the code used in these experiments are
available in the Appendix and on our website at https://www.robot-
learning.uk/language-models-trajectory-generators.

##  7 Acknowledgements

The authors wish to thank Kamil Dreczkowski, Georgios Papagiannis and Pietro
Vitiello for their valuable discussion and feedback during the writing of the
paper.

## References

  * Brown et al. [2020] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf.
  * OpenAI [2023] OpenAI.  GPT-4 Technical Report.  _arXiv e-prints_ , art. arXiv:2303.08774, Mar. 2023.  doi:10.48550/arXiv.2303.08774.
  * Touvron et al. [2023] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al.  Llama 2: Open Foundation and Fine-Tuned Chat Models.  _arXiv e-prints_ , art. arXiv:2307.09288, July 2023.  doi:10.48550/arXiv.2307.09288.
  * Anil et al. [2023] R. Anil, A. M. Dai, O. Firat, M. Johnson, D. Lepikhin, A. Passos, S. Shakeri, E. Taropa, P. Bailey, Z. Chen, et al.  PaLM 2 Technical Report.  _arXiv e-prints_ , art. arXiv:2305.10403, May 2023.  doi:10.48550/arXiv.2305.10403.
  * Wang et al. [2023] L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al.  A Survey on Large Language Model based Autonomous Agents.  _arXiv e-prints_ , art. arXiv:2308.11432, Aug. 2023.  doi:10.48550/arXiv.2308.11432.
  * Huang et al. [2023] W. Huang, C. Wang, R. Zhang, Y. Li, J. Wu, and L. Fei-Fei.  VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models.  _arXiv e-prints_ , art. arXiv:2307.05973, July 2023.  doi:10.48550/arXiv.2307.05973.
  * Yu et al. [2023] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. Gonzalez Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik, et al.  Language to Rewards for Robotic Skill Synthesis.  _arXiv e-prints_ , art. arXiv:2306.08647, June 2023.  doi:10.48550/arXiv.2306.08647.
  * Ahn et al. [2022] M. Ahn, A. Brohan, N. Brown, Y. Chebotar, O. Cortes, B. David, C. Finn, C. Fu, K. Gopalakrishnan, K. Hausman, et al.  Do As I Can, Not As I Say: Grounding Language in Robotic Affordances.  _arXiv e-prints_ , art. arXiv:2204.01691, Apr. 2022.  doi:10.48550/arXiv.2204.01691.
  * Liang et al. [2023] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng.  Code as policies: Language model programs for embodied control.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 9493–9500, May 2023.  doi:10.1109/ICRA48891.2023.10160591.
  * Vemprala et al. [2023] S. Vemprala, R. Bonatti, A. Bucker, and A. Kapoor.  Chatgpt for robotics: Design principles and model abilities.  Technical Report MSR-TR-2023-8, Microsoft, February 2023.  URL https://www.microsoft.com/en-us/research/publication/chatgpt-for-robotics-design-principles-and-model-abilities/.
  * Brohan et al. [2023] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, X. Chen, K. Choromanski, T. Ding, D. Driess, A. Dubey, C. Finn, et al.  RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control.  _arXiv e-prints_ , art. arXiv:2307.15818, July 2023.  doi:10.48550/arXiv.2307.15818.
  * Driess et al. [2023] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, et al.  PaLM-e: An embodied multimodal language model.  In A. Krause, E. Brunskill, K. Cho, B. Engelhardt, S. Sabato, and J. Scarlett, editors, _Proceedings of the 40th International Conference on Machine Learning_ , volume 202 of _Proceedings of Machine Learning Research_ , pages 8469–8488. PMLR, 23–29 Jul 2023.  URL https://proceedings.mlr.press/v202/driess23a.html.
  * Alayrac et al. [2022] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al.  Flamingo: a visual language model for few-shot learning.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 23716–23736. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/960a172bc7fbf0177ccccbb411a7d800-Paper-Conference.pdf.
  * [14] L. Medeiros.  Langsam: Language segment-anything.  https://github.com/luca-medeiros/lang-segment-anything.  Accessed: 2023-10-01.
  * Liu et al. [2023] S. Liu, Z. Zeng, T. Ren, F. Li, H. Zhang, J. Yang, C. Li, J. Yang, H. Su, J. Zhu, and L. Zhang.  Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection.  _arXiv e-prints_ , art. arXiv:2303.05499, Mar. 2023.  doi:10.48550/arXiv.2303.05499.
  * Kirillov et al. [2023] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, et al.  Segment Anything.  _arXiv e-prints_ , art. arXiv:2304.02643, Apr. 2023.  doi:10.48550/arXiv.2304.02643.
  * Cheng and Schwing [2022] H. K. Cheng and A. G. Schwing.  Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model.  In S. Avidan, G. Brostow, M. Cissé, G. M. Farinella, and T. Hassner, editors, _Computer Vision – ECCV 2022_ , pages 640–658, Cham, 2022. Springer Nature Switzerland.  ISBN 978-3-031-19815-1.
  * Xiao et al. [2022] T. Xiao, H. Chan, P. Sermanet, A. Wahid, A. Brohan, K. Hausman, S. Levine, and J. Tompson.  Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models.  _arXiv e-prints_ , art. arXiv:2211.11736, Nov. 2022.  doi:10.48550/arXiv.2211.11736.
  * Brohan et al. [2022] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al.  RT-1: Robotics Transformer for Real-World Control at Scale.  _arXiv e-prints_ , art. arXiv:2212.06817, Dec. 2022.  doi:10.48550/arXiv.2212.06817.
  * Yu et al. [2023] T. Yu, T. Xiao, A. Stone, J. Tompson, A. Brohan, S. Wang, J. Singh, C. Tan, D. M, J. Peralta, et al.  Scaling Robot Learning with Semantically Imagined Experience.  _arXiv e-prints_ , art. arXiv:2302.11550, Feb. 2023.  doi:10.48550/arXiv.2302.11550.
  * Wei et al. [2022] J. Wei, X. Wang, D. Schuurmans, M. Bosma, b. ichter, F. Xia, E. Chi, Q. V. Le, and D. Zhou.  Chain-of-thought prompting elicits reasoning in large language models.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 24824–24837. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf.
  * Kojima et al. [2022] T. Kojima, S. S. Gu, M. Reid, Y. Matsuo, and Y. Iwasawa.  Large language models are zero-shot reasoners.  In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, _Advances in Neural Information Processing Systems_ , volume 35, pages 22199–22213. Curran Associates, Inc., 2022.  URL https://proceedings.neurips.cc/paper_files/paper/2022/file/8bb0d291acd4acf06ef112099c16f326-Paper-Conference.pdf.
  * Chen et al. [2021] M. Chen, J. Tworek, H. Jun, Q. Yuan, H. Ponde de Oliveira Pinto, J. Kaplan, H. Edwards, Y. Burda, N. Joseph, G. Brockman, et al.  Evaluating Large Language Models Trained on Code.  _arXiv e-prints_ , art. arXiv:2107.03374, July 2021.  doi:10.48550/arXiv.2107.03374.
  * Hoffmann et al. [2022] J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. de Las Casas, L. A. Hendricks, J. Welbl, A. Clark, et al.  Training Compute-Optimal Large Language Models.  _arXiv e-prints_ , art. arXiv:2203.15556, Mar. 2022.  doi:10.48550/arXiv.2203.15556.
  * Luo et al. [2023] H. Luo, Q. Sun, C. Xu, P. Zhao, J. Lou, C. Tao, X. Geng, Q. Lin, S. Chen, and D. Zhang.  WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct.  _arXiv e-prints_ , art. arXiv:2308.09583, Aug. 2023.  doi:10.48550/arXiv.2308.09583.

##  Appendix A Main Prompt

Figure 11: The full main prompt.

##  Appendix B Main Prompt Ablations

Figure 12: The full prompt with the highlighted sections removed for the
ablation studies on the main prompt. Figure 13: The full prompt with the
highlighted sections removed for the ablation studies on the main prompt
(continued). Figure 14: The full prompt with the highlighted sections removed
for the ablation studies on the main prompt (continued).

##  Appendix C Action Output Prompt Ablations

Figure 15: The full main prompt modified for evaluating the LLM’s ability to
generate trajectories directly in numbers as language tokens. Figure 16: The
full main prompt modified for ablation studies on the gripper action output.

##  Appendix D Task Success Detection and Re-Planning Prompts

Figure 17: Task success detection prompt. Figure 18: Task re-planning prompt,
to be appended to the main prompt if the LLM detects that the task has failed
and needs to be re-planned and retried. Figure 19: Task summary prompt, as
part of task re-planning.

##  Appendix E Tasks Selected for Ablation Studies

Figure 20: List of tasks selected for the main prompt ablation studies in Sec.
4. Figure 21: List of tasks selected for the action output ablation studies in
Sec. 5.