#  RoboScript: Code Generation for Free-Form Manipulation Tasks across Real
and Simulation

Junting Chen∗12, Yao Mu∗13, Qiaojun Yu4, Tianming Wei4, Silang Wu5, Zhecheng
Yuan5, Zhixuan Liang3,
Chao Yang1, Kaipeng Zhang1, Wenqi Shao1, Yu Qiao1, Huazhe Xu5, Mingyu Ding†6,
Ping Luo†13
OpenGVLab, Shanghai AI Laboratory1 ETH Zurich2 The University of Hong Kong3
Shanghai Jiao Tong University4 Tsinghua University5 UC Berkeley6

This work was completed during the internship at the OpenGV Lab, Shanghai AI
Laboratory by Yao Mu and Junting Chen. Mingyu Ding and Ping Luo are the co-
corresponding authors.* Equal contribution. $\dagger$ Corresponding Authors.

###### Abstract

Rapid progress in high-level task planning and code generation for open-world
robot manipulation has been witnessed in Embodied AI. However, previous
studies put much effort into general common sense reasoning and task planning
capabilities of large-scale language or multi-modal models, relatively little
effort on ensuring the deployability of generated code on real robots, and
other fundamental components of autonomous robot systems including robot
perception, motion planning, and control. To bridge this “ideal-to-real” gap,
this paper presents RobotScript, a platform for 1) a deployable robot
manipulation pipeline powered by code generation; and 2) a code generation
benchmark for robot manipulation tasks in free-form natural language. The
RobotScript platform addresses this gap by emphasizing the unified interface
with both simulation and real robots, based on abstraction from the Robot
Operating System (ROS), ensuring syntax compliance and simulation validation
with Gazebo. We demonstrate the adaptability of our code generation framework
across multiple robot embodiments, including the Franka and UR5 robot arms,
and multiple grippers. Additionally, our benchmark assesses reasoning
abilities for physical space and constraints, highlighting the differences
between GPT-3.5, GPT-4, and Gemini in handling complex physical interactions.
Finally, we present a thorough evaluation on the whole system, exploring how
each module in the pipeline: code generation, perception, motion planning, and
even object geometric properties, impact the overall performance of the
system.

##  I Introduction

Figure 1: Framework of RoboScript . The input layer contains sensor input,
human instruction, and robotic URDF (Unified Robot Description Format) data.
The system utilizes various perception tools, such as grasp detection, 2D
grounding, 3D perception, and joint prediction, to interpret the input data.
These tools are integrated with motion planning tools that include arm
planning, gripper force control, and solving place pose with inverse
kinematics (IK). The Robot Operating System (ROS) serves as the middleware to
provide abstraction to sensor drivers, controllers, and robot definitions
across real robots and the simulation. The framework controls multiple real
robots and their counterparts in the simulation with a unified code generation
pipeline. This modular approach enables flexibility in robotic applications
and adaptability to new code generation methods or robot architectures, from
simple tasks to complex manipulations.

Leveraging pre-trained language models for robotic applications is an active
research area, with many works focusing on planning and reasoning [24, 72, 8,
49, 25, 51, 23, 47, 52, 57, 37, 10, 26, 13, 71, 64, 40, 45, 28, 58, 65, 59].
To enable language models to perceive physical environments, textual scene
descriptions [25, 72, 51] or perception tools [36] are provided. Integrating
large language models (LLMs) with robots has significantly advanced robotics,
enhancing decision-making and control through improved language understanding
and task execution. However, current benchmarks focus more on high-level
semantic understanding, often overlooking the nuances of lower-level control
and physical constraints in robotic manipulation.

To address this gap, we introduce the RobotScript Benchmark, which maps human
language instructions to robot motion planning and provides evaluation within
physically realistic contexts. Going beyond traditional semantic parsing, it
incorporates nuances of physical interactions, workspace constraints, and
object properties that are critical for real-world robotic applications. By
assessing both conceptual and embodied intelligence, RobotScript combines the
rich semantic and pragmatic understanding of large language models with the
fine-grained motor control needed in dynamic, real-world settings. The
designed system provides a complete pipeline from 2D image detection to 3D
scene modeling, grasp pose prediction, and finally, motion planning. This
enables the robot to both understand high-level natural language commands and
autonomously leverage various perception tools and planning algorithms. As a
result, it can generate low-level motion control, without the need for
supervised information.

The RobotScript Benchmark revolves around three core components: 1) ROS-based
Code Generation Testing: The proposed benchmark facilitates testing and
validation of generated code within the Robot Operating System (ROS)
framework, enabling direct deployment on physical robots. It supports
connectivity with various sensors, actuators, and large neural network models,
ensuring the syntactic validity of generated code and simulation-based testing
in Gazebo. To demonstrate the versatility across robotic platforms, we
evaluated our framework on different robot embodiments, assessing the impact
of mechanical designs and gripper types on task performance. Historically, the
incompatibility between the ROS and conda environments has impeded the
deployment and testing of AI models on robotic hardware. By leveraging recent
advancements in Robostack [17], which is a conda-compatible ROS
infrastructure, our benchmark enables seamless model implementation within any
conda environment. 2) Perception-in-the-loop Benchmark Pipeline: In order to
make our tests more closely resemble real robot scenarios, in our benchmark,
the inputs of both the robot planner and controller are based on the results
from robot perception. We provide multiple tools, such as AnyGrasp [16], to
predict a grasp pose from a segmented object point cloud which is constructed
by multi-view 3D fusion, and then used as the input to the grasp motion
planning API. Basing the motion planning on the perceived grasp pose, rather
than ground truth, introduces realism and potential errors that would occur in
a live system. However, we also supply the original ground truth RGBD images
and 3D bounding box data to mitigate excessive noise from potential perception
failures. This pipeline is directly deployable to real robots by using sensor
RGBD images and predicted bounding boxes. 3) Physical Space and Constraint
Reasoning: Our benchmark introduces a comprehensive testbed that evaluates
reasoning capabilities regarding physical space and constraints. These tests
involve understanding complex interactions among various objects, such as
calculating feasible transitional positions without causing interference with
other objects. The benchmark underscores the challenges in constraint
satisfaction by highlighting the performance differences between GPT-3.5 and
GPT-4 in these scenarios.

To summarize, the contributions of this paper include:

  * •

Comprehensive Integration of LLMs with Robotics: RoboScript effectively
bridges the gap between high-level semantic understanding, as well as the
practical nuances of lower-level control and physical constraints in robotic
manipulation. It provides an autonomous manipulation pipeline covering task
interpretation, object detection, pose estimation, grasp planning, and motion
planning.

  * •

In-depth Ablation Study on System Modules: We provide an ablation study on
each system module, deeply analyzing the impact of individual module errors.
This enhances the benchmark’s relevance and practicality in real-world robotic
applications.

  * •

Assessment of LLM Reasoning for Physical Interactions: Our benchmark assesses
reasoning abilities for physical space and constraints, highlighting the
differences between GPT-3.5, GPT-4, and Gemini in handling complex physical
interactions.

##  II Related Work

###  II-A Large Language Models for Robotics

Recent research has delved into the integration of large language models
(LLMs) with embodied AI tasks, focusing on planning, reasoning, and control
[46, 5]. A common approach involves feeding LLMs with perceptual inputs, such
as scene descriptions [25, 72] or visual information [13, 43], and
implementing action capabilities through libraries of motion primitives [36].
Despite these promising developments, effectively representing intricate
spatial relationships continues to be a formidable challenge for LLMs. Several
works have demonstrated the potential of LLMs in composing sequential policies
for embodied domains including VoxPoser [27] for robotic manipulation, SayCan
[1] for instruction following, Palm-E [14] for dexterous manipulation, RT-2
[4] for mobile manipulation, EmbodiedGPT [41] for motion planning, Voyager
[58] and Smallville [44] for embodied gaming, VisProg [20] for program
synthesis, TAPA [62] and SayPlan [48] for task planning. While significant
strides have been taken, fully closing the perception-action loop in embodied
agents remains a dynamic and ongoing area of research. Key challenges
encompass representing complex spatial relationships and long-term planning,
enhancing sample efficiency in learning, and seamlessly integrating
perception, reasoning, and control within a unified framework. Further
exploration of benchmark tasks, and physical knowledge representation will
propel LLMs towards achieving generalized embodied intelligence.

###  II-B Benchmarks for Robotic Code Generation

Ravens [73] was originally proposed as a simulation benchmark for
reinforcement learning of robotic manipulation skills. It was later extended
by CLIPort [50] to include language conditioning, allowing for the study of
language-conditioned policy learning and generalization to new instructions.
Building on the Ravens benchmark, RoboCodeGen [36] was developed to test
language models to test language models on robotics programming problems.
RoboGen [60] further expanded this by encouraging the use of libraries like
NumPy and allowing for the definition of helper functions. These benchmarks
have provided valuable platforms for evaluating the performance of various
Large Language Models (LLMs) (_e.g._ , RoboCodeGen supports evaluation on
GPT-3, InstructGPT and Codex) in generating code for robotic tasks.

###  II-C Motion Planning

In the realm of robot motion planning, the Open Motion Planning Library (OMPL)
[53] is a key open-source framework that offers a range of advanced algorithms
for navigating complex environments. MoveIt [9] was developed to integrate
OMPL’s capabilities with the ROS ecosystem, providing a comprehensive platform
that includes SOTA motion planning, manipulation, 3D perception, kinematics,
and control. This integration greatly improves the applicability and
efficiency of robot motion planning within the ROS ecosystem.

###  II-D Grasp Pose Detection

Vision-guided grasp pose detection is a crucial area in robotics research,
transitioning from 4 degrees of freedom (DOF) top-down grasping methods [19,
70, 74] to more complex 6 DOF grasping methods [3, 31]. The state-of-the-art 6
DOF grasp pose detection model, AnyGrasp [16], achieves a success rate in
object grasping comparable to human performance by studying the geometric
features of objects. This has paved the way for further work on specified
object grasping [38, 32] or articulated object manipulation [69].

##  III RoboScript Pipeline

In this section, we introduce the RoboScript pipeline, which is designed to
deploy and evaluate Large Language Models (LLMs) in generating code to control
robotic arms for completing embodied AI tasks as instructed by humans. Given
RGBD images from the sensor and human instructions as input, the pipeline
generates Python code that directly manipulates a robotic arm. By leveraging
the capabilities of infrastructure including 1) the highly versatile
communication protocols and message interface provided by the Robot Operating
System (ROS), 2) the ROS-integrated Gazebo simulation suite, and 3) the
versatile motion planning framework MoveIt, the pipeline creates a unified
interface to both the simulation and real robots. This abstraction allows our
pipeline to have a snippet of generated code tested safely in a simulated
environment and then executed in reality.

###  III-A Pipeline Overview

Figure 2: Pipeline of RoboScript. RoboScript uses Perception and Motion
Planning Tools, activated by a task query, to generate a Python script with
the LLM. This script includes comments, processes images into 3D models, and
plans safe robot movements. Perception Tools identify objects and spatial
details for planning. Motion Planning Tools then create a collision-free path
for actions.

As described in the introduction, our framework provides a set of Perception
Tools and Motion Planning Tools. Each time a task query is sent to the system,
the LLM generates a Python script by using In-context Learning (ICL) [11].
This means that the few-shot example codes are integrated into the prompt as
demonstrations. Then, the code is executed to finish the task, as shown in
Fig. 2.

A generated Python script mainly consists of three parts:

  1. 1.

Chain-of-Thought (CoT) [61] Comments help LLMs decompose long-time-horizon and
complex tasks.

  2. 2.

Perception Tools process the raw sensor input of multi-view RGB-Depth images
to 3D representations and spatial arguments. When the “detect_obejects”
function is called, the system perception pipeline is triggered, building 3D
representations and implicitly sending them to the Moveit planning scene. This
behavior is designed to support fundamental functionalities of robotic system
like obstacle-aware path planning. Other perception utils are then called to
get spatial arguments like positions and orientations for motion planning.

  3. 3.

Motion Planning Tools take the output of perception utils as input arguments,
generate a collision-free path plan, and further call low-level controllers to
control the motion. For example, when opening a drawer attached to a cabinet,
the robot needs to pull the drawer handle with a Cartesian path outward. The
path should be parallel to the prismatic joint between the drawer and the
cabinet. The motion plan generates the path based on the predicted prismatic
joint direction from the perception tool “get_object_joint_info”.

You can find the details of all our tools API, their explanation, their
arguments, and returns definitions in Appendix VI-B.

It is also worth noting that there could be multiple rounds of perception
$\rightarrow$ motion planning in the code script, especially when the task
requires multiple rounds of robot-environment interaction. For instance,
consider a fundamental task for (future) household robots: storing table
containers in a cabinet. The robot might first detect the environment and find
a closed cabinet, with no direct motion plan to place a container into the
drawer. In this case, the robot needs to first interact with the environment
by opening the drawer and retracting its arm. It then needs to re-detect the
scene to update its plan.

###  III-B Code Generation

The full prompt for code generation involves three sections:

  1. 1.

System Prompt: A system prompt is a pre-defined and query-irrelevant input
that sets the context and general instructions for the LLM to respond. It
helps structure the output of long texts. GPTs and Llama-series models are
trained with special data to increase the model’s attention on system messages
wrapped in special tokens [42, 56]. We provide all format-related
instructions, e.g., “You should only generate Python code and comments.” in
this section. Since LLMs generally pay better attention to the system prompt,
especially when reading long text, we also include descriptions/doc-strings of
perception and motion planning tools into the system prompt.

  2. 2.

Few-shot Examples: To limit the LLM output format and help reduce
hallucination, we provide 10 rounds of task query: master script code
dialogues so that LLMs can learn from demonstrations how to respond to a task
query input.

  3. 3.

Task Query: The task query consists of two sentences. The first sentence
describes what objects are present in the current environment. The second
sentence provides the task instruction. For example, "objects=[apple, bowl].
Please place the apple and place it into the bowl" is a valid task query. For
LLMs, the user needs to input the scene description. For Large Multimodal
Models (LMMs), the scene description could also be generated by querying the
LMM with an RGB image and another pre-defined query prompt.

Similar to Liang et al. [36], RoboScript also employs a hierarchical agent
architecture to facilitate modular code generation. At the top is the main
agent responsible for analyzing the problem and producing the overall
workflow. As the main agent generates code, a syntax parser operates in
parallel, continuously verifying the syntactic correctness of each function
call. When the parser identifies a function that is not in loaded variables,
it triggers a specialized sub-agent to generate the required function from the
function name and description. The sub-agent then produces Python function
definitions as requested, using a similar three-section prompt as model input.

However, even for the state-of-the-art GPT-4 model, this hierarchical
decomposition is not sufficient when generating code for tasks with long time-
horizons or tasks involving complex reasoning processes. Chain-of-Thought [61]
(CoT), a prompting method, is adopted to teach the LLM to explicitly print its
thinking process step by step, as shown in the “CoT Comments” part of Fig. 2.
By leveraging the power of contextual attention in auto-regressive models, CoT
significantly enhances the reasoning capability of LLMs. Many following works
and variants [66, 67] have once again validated its advantages. Viewing our
pipeline as a baseline for future embodied control code generation
architectures, we accept the original CoT prompting and find that CoT already
performs well on the benchmark tasks.

###  III-C Perception Pipeline

In this section, we introduce the perception pipeline, a key component of the
Roboscript system. As mentioned in the overview, “detect_objects” function,
which serves as the foundation and prerequisite of all the other perception
tools, triggers the perception pipeline of the system, conducts 2D visual
grounding, builds 3D representations, and updates the Moveit planning space
with reconstructed meshes. The LLM can then choose the perception tools API to
retrieve the result of 3D representations or further process it for more
advanced information. For example, “get_3d_bbox” simply returns the stored
object instance 3D bounding box, while “get_plane_normal” returns the surface
normal vector of the plane closest to the given position. This design is
motivated by the observation that 1) 3D reconstruction and object 3D bounding
boxes, with relatively fixed algorithms to calculate, are shared as the
foundation data of most other perception tools, and 2) Moveit’s planning
scene, which is the data infrastructure of motion planning tools, is organized
based on objects’ meshes and an Octomap [22] for collision-free planning.
Therefore, we encapsulate these low-level perception processes in a high-level
API, This approach effectively minimizes code length and reduces redundancy,
also relieving the difficulties encountered when generating lengthy texts with
language models.

For clarity, we define all the mathematical notations in the perception
pipeline as follows: Generally, we assume the robot system is equipped with
$k\geq 1$ RGB-Depth cameras, generating images
$\mathbf{I}^{0},\dots,\mathbf{I}^{k-1}$. $\mathbf{I}^{i}$ denotes the input
RGBD image from camera $i$, $\mathbf{I}^{i}_{rgb}\in\mathbb{R}^{H\times
W\times 3}$ denotes the input RGB image of height $H$ and width $W$, and
$\mathbf{I}^{i}_{depth}\in\mathbb{R}^{H\times W}$ denotes the depth image. For
camera $i$, denote the camera intrinsic matrix as
$\mathbf{K_{i}}\in\mathbb{R}^{3\times 3}$. The rotation and translation of the
camera in the world frame are represented as
$\mathbf{R_{i}}\in\mathbb{R}^{3\times 3}$ and
$\mathbf{t_{i}}\in\mathbb{R}^{3}$, respectively. A 2D bounding box on $i$-th
RGB image $\mathbf{I}^{i}_{rgb}$ with label $o_{j}$ is denoted as
$\mathbf{b}^{ij}_{2D}=\left[\bm{x}_{min},\bm{y}_{min},\bm{x}_{max},\bm{y}_{max}\right]$.
And a 3D bounding box of object $o_{j}$ is denoted as
$\mathbf{b}^{j}_{3D}=\left[\bm{x}_{min},\bm{y}_{min},\bm{z}_{min},\bm{x}_{max},\bm{y}_{max},\bm{z}_{max}\right]$.
A 3D point cloud is denoted as
$\mathbf{P}=\\{\mathbf{p}_{0},\dots,\mathbf{p}_{n}\\}$, where
$\mathbf{p}_{i}=[\bm{x}_{i},\bm{y}_{i},\bm{z}_{i}]\in\mathbb{R}^{3}$
represents a point in space. A grasp pose is a 3D-homogeneous (_i.e._ SE(3))
transformation denoted as $\mathbf{T}\in\mathbb{R}^{4\times 4}$. Since we
build the whole system on top of Sucan et al. [53], we have chosen to omit the
details of joint motion planning and focus solely on the end-effector pose in
code generation. Thus, in the rest of this paper, we use the term “motion
trajectory” to indicate a sequence of waypoints of end-effector poses. We
denote a motion trajectory as
$\mathbf{M}=\\{\mathbf{T}_{0},\dots,\mathbf{T}_{n}\\}$, where
$\mathbf{T}_{i}\in\mathbb{R}^{4\times 4}$ represents an end-effector pose.

####  III-C1 2D Grounding

By leveraging the capability of the open-set text-to-image grounding model
GLIP [35], the pipeline first detects a list of task-relevant objects,
reasoned by the LLMs with scene description and task instruction, on each RGB
image $\mathbf{I}^{i}_{rgb}$. GLIP generates a list of 2D bounding boxes
$\mathbf{b}^{ij}_{2D}$ on each RGB image $\mathbf{I}^{i}_{rgb}$ of object
label $o_{j}$, given its text description:

$\mathbf{b}^{ij}_{2D}=\text{GLIP}(\mathbf{I}^{i}_{rgb},\text{Description}(o_{j})),i\in\\{0,\dots,k-1\\}.$
(1)

The open-set reasoning capabilities of LLMs, combined with the open-set
grounding capabilities of large 2D vision models, empower our system with
open-set intelligence capability. Limited by data and training paradigms, 3D
models are relatively more constrained to a specific domain or even a dataset.
This is the main reason we choose to use a 2D grounding model and manually
propagate semantic information to 3D representations, instead of using a 3D
grounding model.

####  III-C2 3D Reconstruction

In an indoor environment with a lot of objects and clutter, varying viewpoints
are helpful to circumvent the obstruction issues that can occur with a single
view. The multi-view RGBD images $\mathbf{I}^{0},\dots,\mathbf{I}^{k-1}$ are
firstly integrated into a Truncated Signed Distance Function (TSDF) volume,
using a TSDF fusion pipeline proposed by Zhou and Koltun [75] and implemented
in Open3D [76]. TSDF fusion is often preferred over cloud fusion from depth
maps in real-time reconstruction pipelines due to its simplicity, faster
speed, and ease of parallelization, as discussed in [54]. This process can be
formulated as:

$\vspace{-4pt}TSDF(\mathbf{x})=\bigoplus_{i=0}^{k-1}F(\mathbf{I}^{i},\mathbf{x}),$
(2)

where $F(\mathbf{I}^{i},\mathbf{x})$ denotes the fusion process of image
$\mathbf{I}^{i}$ at position $\mathbf{x}$, and $\bigoplus$ is the integration
operation over all views.

The volumetric representation is further converted to different
representations at request: (a) point cloud for grasp detection model AnyGrasp
[16], joint prediction model GAMMA, and plane detection model [2]; (b) mesh
for Moveit planning scene; (c) uniform voxel grids for GIGA [31]. The marching
Cube [39] algorithm is used in the surface extraction process to compute point
cloud and mesh.

####  III-C3 Cross-view Bounding Box Matching

With the TSDF volume and 2D grounding results, which are independent 2D object
bounding boxes across images of different camera views, the perception
pipeline continues to get a list of 3D object instances by (a) matching 2D
bounding boxes across different views and (b) filtering the point cloud within
matched 2D bounding boxes.

To match the 2D bounding boxes $\mathbf{b}^{ij}_{2D}$ on each RGB image
$\mathbf{I}^{i}_{rgb}$ of object label $\bm{o}_{j}$, we propose a simple
heuristic algorithm based on view frustum filtering and greedy matching.
Firstly, the algorithm takes a list of 2D bounding boxes for each camera view,
as well as camera intrinsic and extrinsic parameters, as input. It checks
which points are in the 2D bounding box with a frustum filter. Then, a 2D
bounding box $\mathbf{b}^{ij}_{2D}$ can be represented by the binary vector
$\mathbf{v}^{ij}_{2D}\in\\{0,1\\}^{N}$, indicating the projection of which
points out of all $N$ points are within the bounding box. We use the one-
dimension Intersection Over Union (IOU) of two binary vectors as the matching
score $s$ between two bounding boxes:

$\displaystyle s(\mathbf{b}^{ij}_{2D},\mathbf{b}^{kl}_{2D})$
$\displaystyle=\text{IOU}(\mathbf{v}^{ij}_{2D},\mathbf{v}^{kl}_{2D})$ (3)
$\displaystyle=\frac{\sum_{n=1}^{N}\min(\mathbf{v}^{ij}_{2D}[n],\mathbf{v}^{kl}_{2D}[n])}{\sum_{n=1}^{N}\max(\mathbf{v}^{ij}_{2D}[n],\mathbf{v}^{kl}_{2D}[n])},$

where $\mathbf{v}^{ij}_{2D}$ and $\mathbf{v}^{kl}_{2D}$ represent the binary
vectors of the bounding boxes $\mathbf{b}^{ij}_{2D},\mathbf{b}^{kl}_{2D}$ from
views $i$ and $k$. This heuristic is predicated on the premise that the
greater the number of points shared by two bounding boxes across views, the
higher the likelihood of these bounding boxes pertaining to the same object.
An demonstration of cross-view bounding box matching score computation is
shown in Fig. 3.

Figure 3: Cross-view bounding box matching score computation. The matching
score between two bounding boxes is calculated by the Intersection over Union
(IOU) of points presence binary vector.

With cross-view pairwise matching scores, the problem can be formulated as a
quadratic assignment problem (QAP), which is known to be an NP-hard problem
[29]. We use a greedy algorithm approach to match bounding boxes across
different views by maximizing the Intersection over Union (IoU) of these one-
hot vectors, subject to the constraint that each bounding box can be matched
at most once. The matched bounding boxes are then output as a list of tuples,
with each tuple containing the indices of matched bounding boxes across views
or "$-1$" if no match is found. It’s important to note that there could be
label discrepancies or failures across different camera views. We unify the
object label of all matched 2D bounding boxes by max voting.

####  III-C4 3D Instance Segmentation

With matched 2D bounding boxes for object
$\mathbf{B}^{*j}_{2D}=\\{\mathbf{b}^{ij}_{2D}\\},\ i\in\\{0,\cdots,k-1\\}$,
obtaining the objects’ 3D instance segmentations is straightforward. Simply
filter the point cloud within the matched bounding boxes across different
views, and we can get the 3D instance segmentation of the object. Note that
the shape of $\mathbf{B}^{*j}_{2D}$ is not fixed since there could be
detection failures in random camera views.

In this section, we have delved into the workings of the RoboScript pipeline
from receiving task instruction to execution, and explored its key components,
code generation and perception. Since the Motion Planning is mainly done by
Moveit, we do not repeat it here due to space constraints and will turn our
attention to the discussion of our benchmark from the perspective of language
model evaluation.

##  IV RoboScript Benchmark

Built on top of the infrastructure of Gazebo, MoveIt, and ROS, our pipeline
generates executable code both in simulation and real robots, which enables us
to evaluate the capability of language models to generate realistic deployable
code, as well as to study the limitations and strengths of different language
models in a real robotics system. For this purpose, we present our RoboScript
benchmark. Compared with other benchmarks or test environments for general
robot manipulation with natural language instructions [73, 36, 30], our
benchmark highlights realistic object manipulation, complex spatial reasoning
with articulated objects, and a smaller gap with the real robotic system.

###  IV-A Environment Setup

Figure 4: Benchmark tabletop environment and tasks.

The foundation of the RoboScript benchmark lies in its carefully designed base
environment, which includes the following key elements:

  * •

Robot Arm: Our platform supports testing for 2 different robot arms: the
Franka arm and the UR5 robot arm. The Franka arm is a lightweight and flexible
7-axis robot designed for research and manufacturing automation[21]. The UR5
is a versatile 6-axis collaborative industrial robot known for its
flexibility, precision, and safety [34].

  * •

Furniture and Objects: To simulate real-world scenarios, we have incorporated
10 cabinets with drawers and doors as parts from the storageFurniture category
in the PartNet-Mobility dataset [63]. Both the drawers and doors are
articulated objects that require a deep understanding of physical constraints
to manipulate successfully.

  * •

Diverse Containers: For object interaction and manipulation tasks, we populate
the environment with diverse containers. These containers are collected from
the Google Scanned Objects 1k dataset [12], representing some of the largest
indoor object meshes sourced from real scans. The inclusion of these
containers adds a layer of realism and complexity to the tasks.

  * •

Pickable Objects: The benchmark includes 57 pickable objects, carefully
selected from the YCB dataset [7]. This dataset encompasses a wide array of
objects, including both household and kitchen items, ensuring a diverse set of
objects for the robot to interact with.

###  IV-B Free-form Manipulation Tasks

As shown in Table I and Fig. 4, our benchmark comprises eight carefully
designed tasks featuring four distinct challenges that test the reasoning
capabilities of LLMs. The manipulation tasks outlined in the benchmark table
are cleverly designed to assess various levels of cognitive processing in
language models. The simplest task involves placing an apple into a bowl,
testing basic category understanding, object selection, and spatial
awareness—suitable for an ‘Easy’ difficulty level. As the tasks become more
nuanced, we see ‘Moderate’ challenges like nesting objects (a mug into a
drawer) or categorizing and repositioning items (fruits onto a plate, or
swapping an apple and a banana), which require a blend of categories and
spatial reasoning. A task such as placing a brown cup onto an empty plate
introduces property reasoning, adding a layer of complexity. The ‘Hard’ tasks
are the most intricate: clearing a table by moving all objects into a basket
demands advanced spatial and property reasoning while placing a white mug into
a container of the same color tests the model’s ability to comprehend and
apply abstract properties like color coordination. These tasks are
thoughtfully curated to evaluate the model’s understanding from simple object
recognition to complex, multi-step reasoning.

TABLE I: Free-form manipulation tasks in the RoboScript benchmark. The
benchmark comprises eight carefully designed tasks of three difficulty levels,
featuring four distinct challenges that test the reasoning capabilities of
LLMs.

Task ID | Difficulties in Language Model Reasoning | Task Instruction
---|---|---
| Category Understanding | Object Selection | Spatial Relationship | Property Reasoning |
Simple_0 |  |  |  |  | Place the apple into the footed bowl
Simple_1 |  |  | ✓ |  | Place the brown cup into an empty plate
Moderate_0 | ✓ | ✓ | ✓ |  | Open the second top cabinet drawer and place all boxes into it
Moderate_1 |  |  |  | ✓ | Move the white mug into a container with the same color
Moderate_2 |  |  | ✓ |  | Place the mug into the second top drawer
Hard_0 | ✓ | ✓ |  |  | Place all fruits on the table into the white plate.
Hard_1 |  |  | ✓ |  | Exchange the position of the apple and the banana on the table
Hard_2 |  | ✓ |  |  | Clear the table by moving all table objects into the basket

###  IV-C Perception and Motion Planning Tools

####  IV-C1 Grasp Pose Estimation

We provide 4 types of grasp pose estimation methods. a. Central Lift Grasp: As
a heuristic-based method provided in the HomeRobot project [68], the system
executes a top-down grasp by aligning the gripper directly above the object’s
geometric center. b. Horizontal Grasping: In this approach, the gripper is
oriented horizontally, making it suitable for objects aligned with a plane
perpendicular to the tabletop. c. GIGA Grasp Pose Prediction [31]: GIGA
presents a neural network-based approach for 6-DOF grasp pose prediction. This
method utilizes generative models and offers enhanced flexibility. However, it
may have limitations in constrained tasks. d. AnyGrasp [16] Grasp Pose
Prediction: AnyGrasp, a state-of-art neural network-based grasping model, is
adept at generating dense 6-DOF grasp poses. It is trained on the
comprehensive GraspNet dataset [15], which encompasses over 1 billion grasp
labels, ensuring a robust and versatile grasping capability.

####  IV-C2 Kinematic Modeling of Articulated Objects

We utilize the GAMMA[69] to predict articulated objects’ kinematic structures.
GAMMA effectively processes the object’s point clouds, segments them into
rigid parts, and accurately estimates the associated kinematic parameters.
More specifically, GAMMA predicts offsets for each point towards part
centroids, thereby grouping them into distinct articulated components.
Additionally, the model conducts regression on per-point projections onto
joint axes, facilitating a voting process for determining axis origins.
Simultaneously, it regresses per-point joint directions, aggregating votes to
establish precise axis directions. GAMMA demonstrates the capability to
generically model both revolute and prismatic joint parameters by clustering
adjusted and projected points, entirely independent of the articulated
object’s category.

####  IV-C3 Motion Planning Module

Our planning module is built upon the Open Motion Planning Library (OMPL)
platform [53] integrated within the MoveIt platform, which enables performance
benchmarking across 12 different motion planning algorithms including RRT*
[33], PRM* [18] and so on. For this work, we primarily utilized the RRT*
algorithm due to its efficient exploration and optimality guarantees. The
planning module samples configuration space to find a collision-free path to
connect the start and goal states. The output trajectory is checked for
dynamic feasibility before being passed to the control module for execution.

In the benchmark, we offer two modes for running the MoveIt planning scene,
which is the data structure used to store objects and obstacles for collision
checking during planning. Users can either 1) enable using ground truth
planning scenes, with perfect model meshes of Gazebo world loaded into the
MoveIt planning scene and updated automatically; or 2) enable the planning
scene sensor plugin to construct and update an Octomap from depth images in
the real-time. The meshes of objects to be contacted or taken as receptacles
are loaded at request by LLMs when calling “detect_objects” function. The
visualization of MoveIt planning scene under the two options is shown in Fig.
5.

(a) Run MoveIt with GT mesh

(b) Run MoveIt with sensor input

Figure 5: MoveIt planning scene. There are two modes for running MoveIt
planning scene in the benchmark. MoveIt can either load ground truth mesh data
from Gazebo, or build an Octomap from sensors in real time and load
reconstructed meshes from the perception pipeline. planning scene

##  V Experiments

In this section, we first present the LLMs evaluation results on the
RoboScript pipeline, discussing the strengths and limitations of LLMs for
realistic control code generation in subsection V-A. We then conduct an in-
depth study of how the physics properties of objects influence grasp and arm
motion, and subsequently, the final success rate of tasks in subsection V-B.
Lastly, in subsection V-C, we present an ablation study of the perception
pipeline, demonstrating how perception could potentially bottleneck the entire
model. We also present our findings on how language models and vision models
collaborate can make the pipeline work efficiently, and what happens
otherwise.

###  V-A Benchmarking LLMs on RoboScript

In this subsection, we present the evaluation results of three popular LLMs on
the RoboScript benchmark. Table II shows the evaluation result under the
ground truth perception settings, where the perception pipeline is turned off
and the ground truth 3D perception from the Gazebo simulation is always used.
The results are the average over 10 repeated trials. There are four metrics
presented in the results: 1) Grammar correctness, which indicates whether the
generated script has correct grammar; 2) Semantic correctness, which indicates
whether the generated script can faithfully complete the task as per the
language instruction without logical errors. This is determined by the success
of any of the 10 trials. However, we also manually check each code snippet
that fails all trials, to eliminate randomness in the reported results; 3)
Finished steps ratio, which indicates the average success rate of all pre-
defined steps of a task. For example, task hard_0 involves picking two fruits
on a plate, which comprises two steps: moving the first fruit and moving the
second fruit; 4) Finished whole task, which indicates the success rate of the
entire task. We further collect all failed snippets in Appendix VI-C and
analyze the failure reasons.

From the perspective of evaluating the code generation ability of LLMs, the
first two metrics are more meaningful, as the task execution success rate
depends more on other factors, including grasp pose detection, motion
planning, etc. From Table II, we can see that GPT-4 outperforms the other two
LLMs by a clear margin. GPT-4 reaches amazingly $100\%$ correctness on both
grammar and semantics. Surprisingly, despite our pre-defined prompt introduced
in section III-B, Gemini-pro performs even worse than GPT-3.5-turbo, in terms
of code generation quality. This result was observed even after we provided
Gemini-pro with an additional instruction prompt:

Please pay attention to the format of examples in chat history and imitate it.
Do NOT generate common texts or code to import any Python libraries. ONLY
generate comments or detailed planning starting with # and Python code.

Without this prompt, Gemini-pro fails almost all tasks due to the random
output format. Through the case study, we found that Gemini-pro exhibits more
hallucinations compared to GPT-3.5 or GPT-4, and pays less attention to
instructions in the system prompt. In the Gemini technical report [55], there
is no mention of techniques similar to system message tokens as used in
Llama-2 and GPTs. This could be related to the issue, but no definitive
conclusion can be drawn.

TABLE II: Evaluation result of LLMs on the RoboScript benchmark. There are
four metrics presented in the results: 1) grammar correctness, which indicates
whether the generated script has correct grammar; 2) semantic correctness,
which indicates whether the generated script can faithfully complete the task
as per the language instruction without logical errors. 3) finished steps
ratio, which indicates the average success rate of all steps of a task. 4)
finished whole task, which indicates the success rate of the entire task.

LLMs Metric Simple_0 Simple_1 Moderate_0 Moderate_1 Moderate_2 Hard_0 hard_1
Hard_2 Gemini-pro grammer correctness false false false false true true false
true semantic correctness false false false false true false false false
finished steps ratio 0 0 0 0 0.2 0.2 0 0.3 finished whole task 0 0 0 0 0.2 0 0
0 GPT-3.5-turbo grammer correctness true false true true true true true false
semantic correctness true false true true true false false false finished
steps ratio 0.8 0 0.3 0.35 0.7 0 0 0 finished whole task 0.8 0 0.3 0.1 0.7 0 0
0 GPT-4 grammer correctness true true true true true true true true semantic
correctness true true true true true true true true finished steps ratio 0.7
0.9 0.1 0.65 0.4 0.55 0.15 0.45 finished whole task 0.7 0.9 0.1 0.4 0.4 0.3 0
0

###  V-B Grasp and Motion Plan: Impact of Object Shapes

In this section, we shift our focus from the code generation and reasoning
capabilities of LLMs to lower-level factors: the environment objects and how
their shapes affect the manipulation task. Through the construction of the
benchmarking world and the selection of appropriate objects for different
tasks, we find that object shapes dramatically impact the success of object
manipulation for a gripper with certain mechanical structures. To better study
this impact, we isolate the errors in LLM reasoning and code generation by
selecting two fundamental manipulation tasks in daily scenarios: 1) object
pick and place on a table (Simple_0); 2) placing objects into a closed cabinet
drawer (Moderate_1). We execute verified scripts based on code generated by
GPT-4 and only change the drawers and objects.

First, we exhibit the results of table object pick and place in Fig. 6. Among
our object models, 21 come from YCB [6] and 25 come from Google Scanned
Objects [12].

(a) Google Scanned Objects

(b) YCB objects

Figure 6: Results of pick and place task. They present the success rates of
pick and place tasks for all object models in our model base, including 21 YCB
objects and 25 Google Scanned objects.

Secondly, we select the objects with the top success rates in tabletop pick
and place task (Fig. 6), and test their success rates in "open a drawer, pick
and place an object into the drawer" task. We present results in Table III and
IV.

A striking observation is that some round and cylindrical objects, like the
brown_medication_bottle and grey_medication_bottle from Google Scanned Objects
and orange and baseball from YCB dataset perform well in desktop pick and
place task but struggle when placed in the drawer. Through careful
investigation, we found that when using a thin two-fingered mechanical gripper
to grasp cylindrical and spherical objects, the grasp is always in an unstable
mechanical state. The end of the gripper could not always maintain a Cartesian
path when moving in a complex constrained space. And sudden accelerations and
rotations would cause this mechanical state to go out of control.

TABLE III: Success rate of task opening drawer_0 and placing Google Scanned
Objects. We observe two cylindrical objects have significant performance drop
compared to pick and place task.

object | open_drawer | place_object
---|---|---
wood_block | 1 | 0.8
grey_medication_bottle | 0.9 | 0.4
white_and_brown_box | 1 | 0.6
panda_toy_figure | 1 | 0.9
brown_medication_bottle | 1 | 0.2
blue_medication_bottle | 0.9 | 0.7
brown_ceramic_cup | 0.9 | 0.5
toy_bus | 1 | 0.8
Nintendo_Mario_toy_figure | 0.9 | 0.8
dog_figure | 1 | 0.5

TABLE IV: Success rate of task opening drawer_1 and placing YCB objects. We
observe that two sphere objects have a significant performance drop compared
to the pick and place task.

object | open_drawer | place_object
---|---|---
plum | 0.9 | 0.4
tomato_soup_can | 0.8 | 0.6
lemon | 1 | 0.3
orange | 1 | 0
mug | 0.9 | 0.7
baseball | 0.6 | 0.1
lemon | 1 | 0.3
tennis_ball | 1 | 0.4
banana | 1 | 0.3

###  V-C Perception Matters: Pipeline Ablation Study

In this section, we present the results of the pipeline ablation study, mostly
focusing on how different perception modules affect the overall instruction-
following manipulation pipeline. The perception pipeline generates 3D object
instances that can be utilized by perception or motion planning tools. Thus,
we study how 1) 3D object grounding and 2) input data of MoveIt planning scene
affect the pipeline. To minimize the influence of object shape and grasp
detection on the experiments, we selected the 10 objects from Google Scan
Objects that had the highest grasp success rate in the last experiment. For
each object model, each task repeats for 10 times and we report the success
rate out of these 10 trials.

Following the settings in last subsection, we test and report the system
performance on 1) Pick and Place in Fig. 7; 2) Open Drawer and Place Object
tasks in Fig. 8. By enabling the real perception pipeline, using 3D bounding
boxes from 2D grounding and cross-view matching rather than ground truth in
Gazebo, the success rate of pick and place dropped by $10.1\%$, and the
success rate of task open drawer and place object dropped by $6.0\%$. In
comparison, using 3D representations of the Octomap and reconstructed mesh
from perception makes the performance drop by $14.0\%$ and $15.0\%$ on the two
tasks separately. The result demonstrates that the noise in the perception
pipeline has a larger impact on motion planning than perception tools, mostly
the grasp pose detection models in our tasks. This could hint that the noise
in Octomap construction is large, affecting the motion planning accuracy.
However, we also find that MoveIt motion planning is sensitive to the error in
detections, as demonstrated in Fig. 9. In this case, the bounding box of the
target object is enlarged, resulting in an incorrect 3D instance bounding box
that encompasses the Franka robot. The robot arm’s mesh will cause unsolvable
self-collisions, rendering all states illegal for motion planning. Thus, any
motion request will be rejected.

Figure 7: Perception ablation study: pick and place. The figure presents the
success rate of task pick and place under three ablation settings: 1) GT
Detection & Planning, indicating the perception tools and motion planning
tools use ground truth 3D representations from Gazebo; 2) GT Planning Scene,
indicating the system only uses ground truth object meshes in motion planning
tools from MoveIt; 3) No GT, indicating the system does not use ground truth
data, all from perception pipeline. Figure 8: Perception ablation study: open
drawer and place objects. The figure presents the success rate of task pick
and place open the drawer and place objects under three ablation settings: 1)
GT Detection & Planning, indicating the perception tools and motion planning
tools use ground truth 3D representations from Gazebo; 2) GT Planning Scene,
indicating the system only uses ground truth object meshes in motion planning
tools from MoveIt; 3) GT Detection, indicating the system only uses ground
truth 3D object instances as the result of perception pipeline. Figure 9:
Wrong perception leads to motion planning failure. Incorrect 2D grounding can
result in an incorrect 3D instance bounding box that encompasses the Franka
robot. The robot arm’s mesh can cause unsolvable self-collisions, rendering
all states illegal for motion planning. As a result, the command execution is
refused.

###  V-D Real Robot Deployment

To further support our claim that RoboScript is a unified code generation
pipeline for both simulation testing and real robot deployment, we installed
RoboScript onto two sets of robot systems: the Franka Panda and the UR5 with
Robotiq 85 gripper. As shown in Fig. 10, this deployment demonstrated
RoboScript’s ability to seamlessly integrate with existing robotic frameworks,
ensuring that tasks such as object manipulation and precision placement are
executed with high fidelity across different hardware configurations. Our
findings indicate that RoboScript not only simplifies the transition from
simulation to real-world application but also enhances the robots’
adaptability to diverse tasks without requiring extensive programming effort.
This dual deployment validates RoboScript’s role as a pivotal technology in
bridging the gap between simulation-based development and practical, real-
world robotic automation.

Figure 10: Deployment on the real robot: open the drawer with the Franka
robot.

##  VI Conclusion

This paper presents RoboScript that facilitates the deployment and evaluation
of large language models for generating executable robot control code. By
integrating with simulation and real-world robotics frameworks like ROS,
Gazebo, and MoveIt, RoboScript bridges high-level reasoning and low-level
control. The comprehensive pipeline encompasses perception, planning, grasp
detection, and motion control achieving an end-to-end system from language
interpretation to plan execution. Experiments highlight the GPT model’s
reasoning in complex spatial, physical, and property reasoning manipulation
tasks. Ablation studies further illustrate the impact of factors like object
geometry, perception accuracy, and planning robustness on real-world
deployment. Overall, RoboScript advances the integration of language model
intelligence with robots for autonomous manipulation. Future work can focus on
enhancing open-world reliability, expanding the scope of
instructions/environments, and improving sim-to-real transfer sample
efficiency to further leverage the potential of AI-powered systems.

## Acknowledgments

We would like to express our sincere appreciation to Professor Yufeng Yue Mr.
Guangyan Chen from the Beijing Institute of Technology for their great help in
code deployment onto real robots, and their valuable feedback for the user-
friendly development of the project. Equally, our gratitude extends to
Professor Cewu Lu and Dr. Wenhai Liu from Shanghai Jiao Tong University, for
their great support in helping integrate the Anygrasp model into our system.

## References

  * Ahn et al. [2022] Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng.  Do as i can and not as i say: Grounding language in robotic affordances.  In _arXiv preprint arXiv:2204.01691_ , 2022.
  * Araújo and Oliveira [2020] Abner MC Araújo and Manuel M Oliveira.  A robust statistics approach for plane detection in unorganized point clouds.  _Pattern Recognition_ , 100:107115, 2020.
  * Breyer et al. [2021] Michel Breyer, Jen Jen Chung, Lionel Ott, Roland Siegwart, and Juan Nieto.  Volumetric grasping network: Real-time 6 dof grasp detection in clutter.  In _Conference on Robot Learning_ , pages 1602–1611. PMLR, 2021.
  * Brohan et al. [2023] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al.  Rt-2: Vision-language-action models transfer web knowledge to robotic control.  _arXiv preprint arXiv:2307.15818_ , 2023.
  * Brown et al. [2020] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Calli et al. [2015a] Berk Calli, Arjun Singh, Aaron Walsman, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar.  The ycb object and model set: Towards common benchmarks for manipulation research.  In _2015 international conference on advanced robotics (ICAR)_ , pages 510–517. IEEE, 2015a.
  * Calli et al. [2015b] Berk Calli, Aaron Walsman, Arjun Singh, Siddhartha Srinivasa, Pieter Abbeel, and Aaron M Dollar.  Benchmarking in manipulation research: The ycb object and model set and benchmarking protocols.  _arXiv preprint arXiv:1502.03143_ , 2015b.
  * Chen et al. [2022] Brian Chen, Fei Xia, Ben Ichter, Karthik Rao, Kaiyu Gopalakrishnan, Michael S Ryoo, Aviv Stone, and Dan Kappler.  Open-vocabulary queryable scene representations for real world planning.  _arXiv preprint arXiv:2209.09874_ , 2022.
  * Coleman et al. [2014] D.M. Coleman, Ioan Alexandru Sucan, Sachin Chitta, and Nikolaus Correll.  Reducing the barrier to entry of complex robotic software: a moveit! case study.  _ArXiv_ , abs/1404.3785, 2014.  URL https://api.semanticscholar.org/CorpusID:13939653.
  * Ding et al. [2023] Yiming Ding, Xuran Zhang, Chris Paxton, and Shiwen Zhang.  Task and motion planning with large language models for object rearrangement.  _arXiv preprint arXiv:2303.06247_ , 2023.
  * Dong et al. [2022] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui.  A survey for in-context learning.  _arXiv preprint arXiv:2301.00234_ , 2022.
  * Downs et al. [2022] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke.  Google scanned objects: A high-quality dataset of 3d scanned household items.  In _2022 International Conference on Robotics and Automation (ICRA)_ , pages 2553–2560. IEEE, 2022.
  * Driess et al. [2023a] Daniel Driess, Fei Xia, Mohammad Sadegh Sajjadi, Corey Lynch, Ankur Chowdhery, Ben Ichter, Azalia Mirza Wahid, Jonathan Tompson, Quoc Vuong, Tony Yu, et al.  Palm-e: An embodied multimodal language model.  _arXiv preprint arXiv:2303.03378_ , 2023a.
  * Driess et al. [2023b] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence.  Palm-e: An embodied multimodal language model.  In _arXiv preprint arXiv:2303.03378_ , 2023b.
  * Fang et al. [2020] Hao Fang, Chengkun Wang, Yifeng Gao, and Cewu Lu.  Graspnet-1billion: A large-scale benchmark for general object grasping.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11444–11453, 2020.
  * Fang et al. [2023] Hao-Shu Fang, Chenxi Wang, Hongjie Fang, Minghao Gou, Jirong Liu, Hengxu Yan, Wenhai Liu, Yichen Xie, and Cewu Lu.  Anygrasp: Robust and efficient grasp perception in spatial and temporal domains.  _IEEE Transactions on Robotics_ , 2023.
  * Fischer et al. [2021] Tobias Fischer, Wolf Vollprecht, Silvio Traversaro, Sean Yen, Carlos Herrero, and Michael Milford.  A robostack tutorial: Using the robot operating system alongside the conda and jupyter data science ecosystems.  _IEEE Robotics and Automation Magazine_ , 2021.  doi: 10.1109/MRA.2021.3128367.
  * Gang and Wang [2016] Li Gang and Jingfang Wang.  Prm path planning optimization algorithm research.  _Wseas Transactions on Systems and control_ , 11:81–86, 2016.
  * Guo et al. [2016] Di Guo, Tao Kong, Fuchun Sun, and Huaping Liu.  Object discovery and grasp detection with a shared convolutional neural network.  In _2016 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 2038–2043. IEEE, 2016.
  * Gupta and Kembhavi [2023] Tanmay Gupta and Aniruddha Kembhavi.  Visual programming: Compositional visual reasoning without training.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14953–14962, 2023.
  * Haddadin et al. [2022] Sami Haddadin, Sven Parusel, Lars Johannsmeier, Saskia Golz, Simon Gabl, Florian Walch, Mohamadreza Sabaghian, Christoph Jähne, Lukas Hausperger, and Simon Haddadin.  The franka emika robot: A reference platform for robotics research and education.  _IEEE Robotics & Automation Magazine_, 29(2):46–64, 2022.
  * Hornung et al. [2013] Armin Hornung, Kai M Wurm, Maren Bennewitz, Cyrill Stachniss, and Wolfram Burgard.  Octomap: An efficient probabilistic 3d mapping framework based on octrees.  _Autonomous robots_ , 34:189–206, 2013.
  * Huang et al. [2022a] Chih-Yao Huang, Oier Mees, Andrei Zeng, and Wolfram Burgard.  Visual language maps for robot navigation.  _arXiv preprint arXiv:2210.05714_ , 2022a.
  * Huang et al. [2022b] Wendy Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch.  Language models as zero-shot planners: Extracting actionable knowledge for embodied agents.  _International Conference on Machine Learning_ , pages 4558–4568, 2022b.
  * Huang et al. [2022c] Wendy Huang, Fei Xia, Tete Xiao, Hei Chan, Percy Liang, Pete Florence, Animesh Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al.  Inner monologue: Embodied reasoning through planning with language models.  In _arXiv preprint arXiv:2207.05608_ , 2022c.
  * Huang et al. [2023a] Wendy Huang, Fei Xia, Dhruv Shah, Daniel Driess, Andy Zeng, Yicheng Lu, Peter Florence, Igor Mordatch, Sergey Levine, Keno Hausman, et al.  Grounded decoding: Guiding text generation with grounded models for robot control.  _arXiv preprint arXiv:2303.00855_ , 2023a.
  * Huang et al. [2023b] Wenlong Huang, Chen Wang, Ruohan Zhang, Yunzhu Li, Jiajun Wu, and Li Fei-Fei.  Voxposer: Composable 3d value maps for robotic manipulation with language models.  _arXiv preprint arXiv:2307.05973_ , 2023b.
  * Jansen [2020] Peter Adriaan Jansen.  Visually-grounded planning without vision: Language models infer detailed plans from high-level instructions.  _arXiv preprint arXiv:2009.14259_ , 2020.
  * Jia et al. [2016] Kui Jia, Tsung-Han Chan, Zinan Zeng, Shenghua Gao, Gang Wang, Tianzhu Zhang, and Yi Ma.  Roml: A robust feature correspondence approach for matching objects in a set of images.  _International Journal of Computer Vision_ , 117:173–197, 2016.
  * Jiang et al. [2022] Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan.  Vima: General robot manipulation with multimodal prompts.  _arXiv_ , 2022.
  * Jiang et al. [2021] Zhenyu Jiang, Yifeng Zhu, Maxwell Svetlik, Kuan Fang, and Yuke Zhu.  Synergies between affordance and geometry: 6-dof grasp detection via implicit representations.  _arXiv preprint arXiv:2104.01542_ , 2021.
  * Ju et al. [2024] Yuanchen Ju, Kaizhe Hu, Guowei Zhang, Gu Zhang, Mingrun Jiang, and Huazhe Xu.  Robo-abc: Affordance generalization beyond categories via semantic correspondence for robot manipulation.  _arXiv preprint arXiv:2401.07487_ , 2024.
  * Karaman and Frazzoli [2011] Sertac Karaman and Emilio Frazzoli.  Sampling-based algorithms for optimal motion planning.  _International Journal of Robotics Research_ , 30(7):846–894, 2011.
  * Kebria et al. [2016] Parham M Kebria, Saba Al-Wais, Hamid Abdi, and Saeid Nahavandi.  Kinematic and dynamic modelling of ur5 manipulator.  In _2016 IEEE international conference on systems, man, and cybernetics (SMC)_ , pages 004229–004234. IEEE, 2016.
  * Li* et al. [2022] Liunian Harold Li*, Pengchuan Zhang*, Haotian Zhang*, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao.  Grounded language-image pre-training.  In _CVPR_ , 2022.
  * Liang et al. [2023] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng.  Code as policies: Language model programs for embodied control.  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 9493–9500. IEEE, 2023.
  * Lin et al. [2023] Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg.  Text2motion: From natural language instructions to feasible plans.  _arXiv preprint arXiv:2303.12153_ , 2023.  URL https://ar5iv.org/abs/2303.12153.
  * Liu et al. [2024] Peiqi Liu, Yaswanth Orru, Chris Paxton, Nur Muhammad Mahi Shafiullah, and Lerrel Pinto.  Ok-robot: What really matters in integrating open-knowledge models for robotics.  _arXiv preprint arXiv:2401.12202_ , 2024.
  * Lorensen and Cline [1998] William E Lorensen and Harvey E Cline.  Marching cubes: A high resolution 3d surface construction algorithm.  In _Seminal graphics: pioneering efforts that shaped the field_ , pages 347–353. 1998.
  * Lu et al. [2023] Yujia Lu, Peng Lu, Zhengyu Chen, Wenchao Zhu, Xi Eric Wang, and William Yang Wang.  Multimodal procedural planning via dual text-image prompting.  _arXiv preprint arXiv:2305.01795_ , 2023.
  * Mu et al. [2023] Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, and Ping Luo.  Embodiedgpt: Vision-language pre-training via embodied chain of thought.  _arXiv preprint arXiv:2305.15021_ , 2023.
  * OpenAI [2023a] OpenAI.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774, 2023a.
  * OpenAI [2023b] R OpenAI.  Gpt-4 technical report.  _arXiv_ , pages 2303–08774, 2023b.
  * Park et al. [2023] Joon Sung Park, Joseph C O’Brien, Carrie J Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein.  Generative agents: Interactive simulacra of human behavior.  _arXiv preprint arXiv:2304.03442_ , 2023.
  * Patel et al. [2023] Devansh Patel, Hamid Eghbalzadeh, Naman Kamra, Michael L Iuzzolino, Utkarsh Jain, and Raghavendra Desai.  Pretrained language models as visual planners for human assistance.  _arXiv preprint arXiv:2304.09179_ , 2023.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Raman et al. [2022] Shrayash Raman, Vinay Cohen, Edward Rosen, Isma Idrees, Dan Paulius, and Stefanie Tellex.  Planning with large language models via corrective re-prompting.  _arXiv preprint arXiv:2211.09935_ , 2022.
  * Rana et al. [2023] Krishan Rana, Jesse Haviland, Sourav Garg, Jad Abou-Chakra, Ian Reid, and Niko Suenderhauf.  Sayplan: Grounding large language models using 3d scene graphs for scalable task planning.  _arXiv preprint arXiv:2307.06135_ , 2023.
  * Shah et al. [2022] Devinik Shah, Barret Osinski, Brett Ichter, and Sergey Levine.  Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action.  _arXiv preprint arXiv:2207.04429_ , 2022.
  * Shridhar et al. [2022] Mohit Shridhar, Luca Manuelli, and Dieter Fox.  Cliport: What and where pathways for robotic manipulation.  In _Conference on Robot Learning_ , pages 894–906. PMLR, 2022.
  * Singh et al. [2022] Ishan Singh, Viktors Blukis, Arsalan Mousavian, Animesh Goyal, Denis Xu, Jonathan Tremblay, Dieter Fox, Jesse Thomason, and Animesh Garg.  Progprompt: Generating situated robot task plans using large language models.  _arXiv preprint arXiv:2209.11302_ , 2022.
  * Song et al. [2022] Chenghao Song, Jingyun Wu, Christopher Washington, Brent M. Sadler, Wen-lin Chao, and Yu Su.  Llm-planner: Fewshot grounded planning for embodied agents with large language models.  _arXiv preprint arXiv:2212.04088_ , 2022.
  * Sucan et al. [2012] Ioan A Sucan, Mark Moll, and Lydia E Kavraki.  The open motion planning library.  _IEEE Robotics & Automation Magazine_, 19(4):72–82, 2012.
  * Sun et al. [2021] Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao.  Neuralrecon: Real-time coherent 3d reconstruction from monocular video.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15598–15607, 2021.
  * Team et al. [2023] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Touvron et al. [2023] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_ , 2023.
  * Vemprala et al. [2023] Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor.  Chatgpt for robotics: Design principles and model abilities.  Technical Report MSR-TR-2023-8, Microsoft Research, February 2023.  URL https://www.microsoft.com/en-us/research/publication/chatgpt-for-robotics-design-principles-and-model-abilities/.
  * Wang et al. [2023a] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models.  _arXiv preprint arXiv:2305.16291_ , 2023a.
  * Wang et al. [2023b] Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang.  Gensim: Generating robotic simulation tasks via large language models.  _arXiv preprint arXiv:2310.01361_ , 2023b.
  * Wang et al. [2023c] Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang, Yian Wang, Katerina Fragkiadaki, Zackory Erickson, David Held, and Chuang Gan.  Robogen: Towards unleashing infinite data for automated robot learning via generative simulation.  _arXiv preprint arXiv:2311.01455_ , 2023c.
  * Wei et al. [2022] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in Neural Information Processing Systems_ , 35:24824–24837, 2022.
  * Wu et al. [2023] Zhenyu Wu, Ziwei Wang, Xiuwei Xu, Jiwen Lu, and Haibin Yan.  Embodied task planning with large language models.  _arXiv preprint arXiv:2307.01848_ , 2023.
  * Xiang et al. [2020] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, et al.  Sapien: A simulated part-based interactive environment.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11097–11107, 2020.
  * Xie et al. [2023] Yujing Xie, Chun yu, Tiancheng Zhu, Jiaze Bai, Zihan Gong, and Harold Soh.  Translating natural language to planning goals with large-language models.  _arXiv preprint arXiv:2302.05128_ , 2023.
  * Yang et al. [2023] Jiange Yang, Wenhui Tan, Chuhao Jin, Bei Liu, Jianlong Fu, Ruihua Song, and Limin Wang.  Pave the way to grasp anything: Transferring foundation models for universal pick-place robots.  _arXiv preprint arXiv:2306.05716_ , 2023.
  * Yao et al. [2023a] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan.  Tree of thoughts: Deliberate problem solving with large language models.  _arXiv preprint arXiv:2305.10601_ , 2023a.
  * Yao et al. [2023b] Yao Yao, Zuchao Li, and Hai Zhao.  Beyond chain-of-thought, effective graph-of-thought reasoning in large language models.  _arXiv preprint arXiv:2305.16582_ , 2023b.
  * Yenamandra et al. [2023] Sriram Yenamandra, Arun Ramachandran, Karmesh Yadav, Austin Wang, Mukul Khanna, Theophile Gervet, Tsung-Yen Yang, Vidhi Jain, Alex William Clegg, John Turner, Zsolt Kira, Manolis Savva, Angel Chang, Devendra Singh Chaplot, Dhruv Batra, Roozbeh Mottaghi, Yonatan Bisk, and Chris Paxton.  Homerobot: Open vocab mobile manipulation, 2023.  URL https://aihabitat.org/static/challenge/home_robot_ovmm_2023/OVMM.pdf.
  * Yu et al. [2023] Qiaojun Yu, Junbo Wang, Wenhai Liu, Ce Hao, Liu Liu, Lin Shao, Weiming Wang, and Cewu Lu.  Gamma: Generalizable articulation modeling and manipulation for articulated objects.  _arXiv preprint arXiv:2309.16264_ , 2023.
  * Yu et al. [2020] Yingying Yu, Zhiqiang Cao, Shuang Liang, Wenjie Geng, and Junzhi Yu.  A novel vision-based grasping method under occlusion for manipulating robotic system.  _IEEE Sensors Journal_ , 20(18):10996–11006, 2020.
  * Yuan et al. [2023] Haojian Yuan, Chenyi Zhang, Hong Wang, Fengda Xie, Peng Cai, Haoye Dong, and Zijian Lu.  Plan4mc: Skill reinforcement learning and planning for open-world minecraft tasks.  _arXiv preprint arXiv:2303.16563_ , 2023.
  * Zeng et al. [2022] Aishan Zeng, Alvin Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Ashish Purohit, Michael S Ryoo, Vikas Sindhwani, Joonseok Lee, Vincent Vanhoucke, et al.  Socratic models: Composing zero-shot multimodal reasoning with language.  _arXiv preprint arXiv:2204.00598_ , 2022.
  * Zeng et al. [2021] Andy Zeng, Pete Florence, Jonathan Tompson, Stefan Welker, Jonathan Chien, Maria Attarian, Travis Armstrong, Ivan Krasin, Dan Duong, Vikas Sindhwani, et al.  Transporter networks: Rearranging the visual world for robotic manipulation.  In _Conference on Robot Learning_ , pages 726–747. PMLR, 2021.
  * Zhang et al. [2019] Hanbo Zhang, Xuguang Lan, Site Bai, Lipeng Wan, Chenjie Yang, and Nanning Zheng.  A multi-task convolutional neural network for autonomous robotic grasping in object stacking scenes.  In _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 6435–6442. IEEE, 2019.
  * Zhou and Koltun [2013] Qian-Yi Zhou and Vladlen Koltun.  Dense scene reconstruction with points of interest.  _ACM Transactions on Graphics (ToG)_ , 32(4):1–8, 2013.
  * Zhou et al. [2018] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun.  Open3d: A modern library for 3d data processing.  _arXiv preprint arXiv:1801.09847_ , 2018.

## Appendix

###  VI-A Code generation algorithm

Data: Main system_prompt $S$, Main examples $E$, Sub system_prompt
$S^{\prime}$, Sub examples $E^{\prime}$, task_query $Q$

Result: Generated code following the hierarchical agent structure

1

2

3Class  _MainAgent(_$S$ , $E$_)_

4 Function  _process($Q$)_

5 code $C\leftarrow$ generate_code([S, E, Q]);

6 for  _function F in C_ do

7 if  _F is not defined_ then

8 $C^{\prime}\leftarrow$ SubAgent ($S^{\prime}$, $E^{\prime}$).process($F$,
context($F$));

9 $C\leftarrow$ append([$C^{\prime}$, $C$])

10

11 return $C$

12

13

14Class  _SubAgent(_$S^{\prime}$ , $E^{\prime}$_)_

15 Function  _process(function F, context CT)_

16 query $Q^{\prime}\leftarrow$ make_query(F, CT) ;

17 code $C\leftarrow$ generate_code([S’, E’, Q’]);

18 for  _function $F^{\prime}$ in C_ do

19 if  _$F^{\prime}$ is not defined_ then

20 $C^{\prime}\leftarrow$ SubAgent ($S^{\prime}$,
$E^{\prime}$).process($F^{\prime}$, context($F^{\prime}$));

21 $C\leftarrow$ append([$C^{\prime}$, $C$])

22

23 return $C$

24

25

26Code $\leftarrow$ MainAgent($S$, $E$).process(Q);

Algorithm 1 Hierarchical Agent Code Generation

###  VI-B Tool APIs design

This section gives a detailed description of all the tools API that language
models have access to.

The Perception Tools are shown in Code Block 11. The perception tools provide
geometric information as function arguments for the motion planning tools,
including object positions, object 3D bounding box, articulated joint
information (moving direction or rotational axis), and plane surface normals.
Perception tools are organized as object properties using object names as
arguments and function names to specify the property type. With this object-
centric design, LLMs only need to reason which object properties are involved
in a specific task. Through our observation, LLMs are good at handling high-
level task decomposition and context-consistent function calling rather than
low-level geometric understanding or mathematical-consistent algorithm
generation. This design makes the best use of LLM’s strengths and avoids its
weaknesses.

These perception tools provide spatial parameters for trajectory generation
and pick-and-place actions. For example, to generate a trajectory to open a
door, the LLM needs to call the "get_object_joint_info" function to get its
door joint axis, then to calculate the distance from the handle to the door
joint, and further generate an arc path around this joint, with radius same as
the distance from the door joint to handle, as demonstrated in
"generate_arc_path_around_joint." Then, the robot can call "follow_path" to
execute this trajectory.

The Motion Planning Tools are shown in Code Block 12. The motion planning
tools are mostly re-wrappers of the Moveit Python move group interface. They
hide system functional code under the hood, such as ROS synchronization and
failure recovery behavior. Two API tools worth mentioning are:

  * •

grasp. This tool executes a grasp motion with a given grasp pose: It first
moves to a pre-grasp pose, as did in [31][16], which is a pose retracted from
the grasp pose on the approaching direction of the gripper. Assume the target
grasp pose to be $\mathbf{P}=(\mathbf{R}|\mathbf{t})$, and the approaching
direction is $\mathbf{[0,0,1]}$ under identity rotation, then the pre-grasp
pose is
$\mathbf{P^{\prime}}=(\mathbf{R}|\mathbf{t}-\mathbf{R}\cdot\mathbf{[0,0,0.1]})$.
So this API plans and executes the path with two end-effector sub-goals
$\\{\mathbf{P^{\prime}},\mathbf{P}\\}$.

  * •

parse_place_pose. Although it may seem like a perception tool at first glance,
this tool relies on the object bounding boxes, object mesh, and Moveit Inverse
Kinematics solver to get the valid placing pose. Specifically, it first
predicts a candidate object place center position by a heuristic algorithm
given container bounding box and object bounding box and then solves a valid
orientation for the gripper with Moviet IK.

⬇

from perception_utils import (

get_object_center_position, # Returns the position of an object in the world
frame. Returns: position: np.array [x,y,z]

get_object_pose, # Returns the pose of an object in the world frame. Returns:
pose: Pose

get_3d_bbox, # Returns the 3D bounding box of an object in the world frame.
Args: object_name: str. Returns: bbox: np.array [x_min, y_min, z_min, x_max,
y_max, z_max]

get_object_name_list, # Returns a list of names of objects present in the
scene

parse_adaptive_shape_grasp_pose, # Parse adaptive grasp pose for objects.
Args: object_name: str, preferred_position: Optional(np.ndarray); preferred
gripper tip point position; preferred_approach_direction:
Optional(np.ndarray), preferred gripper approach direction;
preferred_plane_normal: Optional(np.ndarray), preferred gripper plane normal
direction. Returns: grasp_pose: Pose

parse_central_lift_grasp_pose, # This method involves a vertical lifting
action. The gripper closes at the center of the object and is not suitable for
elongated objects and is not suitable for the objects with openings, as the
gripper’s width is really small. It is optimal for handling spherical and
cuboid objects without any opening that are not ideal for off-center grasping.
Args: object_name: str, description: Optional(str) in [’top’, ’center’],
Returns: grasp_pose: Pose

detect_objects, # Detect and update task-relevant objects’ status in the
environment. Call this function before interaction with environment objects.
Args: object_list: Optional(List[str]), objects to detect.

get_object_joint_info, # Get the joint info of an object closest to a given
position. Args: obj_name: str, name of the object; position: np.ndarray,
select the joint closest to this position; type: str, allowed type of the
joint, choice in ["any", "revolute", "prismatic"]. Returns: joint_info: dict,
joint info of the object.
{"joint_position":[x,y,z],"joint_axis":[rx,ry,rz],"type":str}

get_plane_normal, # Get the plane normal of an object closest to the given
position. Args: obj_name: name of the object position: np.ndarray, select the
plane closest to this position. Returns: np.ndarray, normal vector [x,y,z]

)

Figure 11: Perception tools API in system prompt during code generation

⬇

from motion_utils import (

attach_object, # Attaches an object to the robot gripper in the planning
space. Call this function right after closing the gripper. Args: object_id:
str.

detach_object, # Detaches an object from the robot gripper in the planning
space. Call this function right after opening the gripper. Args: object_id:
str.

open_gripper, # Open the gripper. No args.

close_gripper, # Close the gripper. No args.

move_to_pose, # Move the gripper to pose. Args: pose: Pose

move_in_direction, # Move the gripper in the given direction in a straight
line by certain distance. Note that you can use the surface normal and the
joint axis. Args: axis: np.array, the move direction vector ; distance: float.

generate_arc_path_around_joint, # Generate a rotational gripper path of poses
around the revolute joint. Args: current_pose: Pose, current pose of the
gripper; joint_axis: np.array, the joint axis of the revolute joint;
joint_position: np.array, the joint position of the revolute joint; n: int,
number of waypoints; angle: float, angle of rotation in degree. Returns: path:
List[Pose]

follow_path, # Move the gripper to follow a path of poses. Args: path:
List[Pose]

get_gripper_pose, # Get the gripper pose. No args. Returns: pose: Pose

grasp, # Executes a grasp motion at the grasp_pose. Args: grasp_pose: Pose

parse_place_pose, # Predict the place pose for an object relative to a
receptacle. Args: object_name: str, receptacle_name: Optional(str), position:
Optional(np.array) [x,y,z], . Returns: place_pose: Pose

)

Figure 12: Motion Planning tools API in system prompt during code generation

###  VI-C Failed code case study

LLMs generate control code with a failure rate and all sorts of failure
patterns, including grammar failures and logical fallacies. The grammar
failures can be: 1) Hallucinated function named arguments, 2) wrong variable
type and operation, 3) garbage code (usually when code is too long or the
generation process goes out of control), etc. However, these are the basic
code-generation capabilities. They are well-studied in a lot of general LLM
code generation evaluation papers. Here, we discuss several failure cases due
to the limitation of LLMs’ reasoning capabilities on embodied tasks.

####  VI-C1 GPT-3.5-turbo

  * •

hard_3, place the brown cup into an empty plate Extra instruction in prompt:
you need to find empty plate by checking if there is any object in the plate’s
2D region on the table
This script has the wrong assumption that if the min and max values of a
plate’s bounding box are zero, it is empty, which is obviously not correct.

⬇

# Find an empty plate on the table

empty_plate = None

table_objects = get_object_name_list()

for object_name in table_objects:

if object_name.endswith(’_plate’):

bbox = get_3d_bbox(object_name)

if bbox[2] == 0 and bbox[5] == 0: # Check if the plate is empty by checking
its height (z-axis)

empty_plate = object_name

break

  * •

hard_4, clear the table by moving all table objects into the basket
Although it might look good initially, the script skips the table and basket.
However, the function did return cabinet, which the script does not need to
pick. The master code is to construct a list of objects, only including
objects that need to be picked and placed, rather than this exclusion logic.

⬇

# Iterate through each object on the table

for object_name in object_names:

# Skip the table and the basket

if object_name == ’table’ or object_name == ’basket_with_fabric_liner’:

continue

\par# Grasp the object

open_gripper()

grasp_pose = parse_adaptive_shape_grasp_pose(object_name)

…

####  VI-C2 Gemini

  * •

hard_3, Place the brown cup into an empty plate: Extra instruction in prompt:
you need to find empty plate by checking if there is any object in the plate’s
2D region on the table
This task requires LLM to understand what is an "empty plate" with spatial
reasoning process. However, Gemini generated a weird function to check if an
object is in a plate by assuming objects have a name format plate_apple, which
does not hold obviously. This failure can be attributed to Gemini’s lack of
attention to instruction and hallucination.

⬇

def get_object_names_in_region(plate, region=’table’):

"""

Get a list of object names in a given region.

\parArgs:

plate (str): The name of the plate.

region (str): The name of the region.

\parReturns:

list: A list of object names in the region.

"""

object_names = []

for obj in get_object_name_list():

# Check if the object is in the given region

if region == ’table’:

if is_object_on_table(obj, plate):

object_names.append(obj)

elif region == ’receptacle’:

if get_receptacle_name_from_object_name(obj) == plate:

object_names.append(obj)

return object_names

###  VI-D Task Data Generation and Dataset

One of the distinctive features of the RoboScript platform is the capability
of task data generation automatically. The platform can generate task data of
(task query, environment, generated code) to train custom embodied code
generation models. This generation involves three steps:

  1. 1.

Randomly select candidate containers and pickable objects, sample an valid
arrangement, and save to the environment definition file.

  2. 2.

Collect table top arrangement information, convert to text description ("Apple
is in the grey bowl. Banana is on the white plate."). Send the environment
description to LLM, to prompt the LLM to suggest what tasks can be done for
this environment.

  3. 3.

For each task query, generate a normal code snippet to execute the task.

This approach ensures task diversity and complexity. We have already generated
100 randomly arranged tabletop environments and 4500+ grammatically correct
code snippets and constructed into (task query, environment, generated code)
tuples for some private projects.

The models are selected from YCB or Google Scanned Objects. A collage of
pickable objects and container objects is given in Figure13 and Figure14.

Some of the auto-generated worlds are shown in Figure15. To better visualize
its heuristic arrangments, we show the top view.

Figure 13: Pickable objects selected from Google Scanned Objects. Figure 14:
Container objects selected from Google Scanned Objects.

Figure 15: Auto-generated Worlds. We show the top view of the table and saved
GT detection results of objects.

###  VI-E Implementation of RoboScript

LLMs and Prompting. We follow prompting structure by Liang et al. [36], which
recursively calls LLMs using their own generated code, where each language
model program (LMP) is responsible for a unique functionality (e.g.,
processing perception calls). We use GPT-4 [43] from OpenAI API. For each LMP,
we include 10 example queries and corresponding responses as part of the
prompt.

Motion Planning and control.  We use MoveIt-integrated Open Motion Planning
Library (OMPL), which is a collection of state-of-the-art sampling-based
motion planning algorithms. Specifically, we use RRT (Rapidly-exploring Random
Trees) and PRM (Probabilistic Roadmaps) by default for trajectory path
sampling.

For the robot control, by default, we use JointStateController in the ros
control package, taking as input the current joint states and joint goal
states. It runs a PID controller at the backend at 50 Hz. The PID parameters
are tuned and saved to individual Moveit robot configurations separately,
mostly provided by each robot manufacturer.

Dynamics Model. We use the known robot dynamics model in all tasks, where it
is used in motion planning for the end-effector to follow the waypoints. For
the majority of our considered tasks where the “entity of interest” is the
robot, no environment dynamics model is used (i.e., the scene is assumed to be
static), but we replan at every step to account for the latest observation.
For tasks in which the “entity of interest” is an object, we study only a
planar pushing model parametrized by contact point, push direction, and push
distance. We use a heuristic-based dynamics model that translates an input
point cloud along the push direction by the push distance.