# Predictive Coding for Animation-Based Video Compression

###### Abstract

We address the problem of efficiently compressing video for conferencing-type
applications. We build on recent approaches based on image animation, which
can achieve good reconstruction quality at very low bitrate by representing
face motions with a compact set of sparse keypoints. However, these methods
encode video in a frame-by-frame fashion, i.e., each frame is reconstructed
from a reference frame, which limits the reconstruction quality when the
bandwidth is larger. Instead, we propose a predictive coding scheme which uses
image animation as a predictor, and codes the residual with respect to the
actual target frame. The residuals can be in turn coded in a predictive
manner, thus removing efficiently temporal dependencies. Our experiments
indicate a significant bitrate gain, in excess of 70% compared to the HEVC
video standard and over 30% compared to VVC, on a dataset of talking-head
videos.

Index Terms—  Video compression, image animation, generative models, video
conferencing, predictive coding

##  1 Introduction

Recent work on learning-based video coding for videoconferencing applications
has shown that it is possible to compress videos of talking heads with
extremely low bitrate, without significant losses in visual quality [1, 2, 3,
4, 5, 6]. The basic tenet of these methods is that face motion can be
represented through a compact set of sparse keypoints [7], which can be
transmitted and used at the decoder side to animate a reference video frame.

However, despite the impressive coding performance of these methods at very
low bitrates, existing animation-based codecs for videoconferencing still have
several bottlenecks. Firstly, when the available bitrate increases, the
reconstruction quality quickly reaches saturation, and conventional coding
tools such as HEVC or VVC perform better. Secondly, bitrate variability in
current schemes is complex, unlike conventional coding methods where a simple
quantization parameter can be used to regulate bitrate. Finally, animation-
based codecs operate on a frame-by-frame basis, which is inefficient for
eliminating temporal redundancy in the video.

This paper addresses these limitations by proposing a predictive coding scheme
for videoconferencing applications. Specifically, we interpret the keypoint-
based image animation used in previous codecs [1] as a spatial predictor of
the current (target) frame, as depicted in Figure 1. The residual between the
animated and the target frame is then coded and used at the decoder side to
correct the animated target frame. Since animation residuals exhibit temporal
correlation, we also encode them in a predictive manner, i.e., we predict the
current animation residual based on the previously decoded residual and encode
the prediction difference. It is worth noting that this approach is similar in
principle to the classic video coding prediction loop, with the important
distinction that residual coding and animation are jointly learned in an end-
to-end fashion. We name our method RDAC, for Residual Deep Animation Codec.
Our results demonstrate significant rate-distortion improvements compared to
standard codecs such as HEVC and VVC, as measured by several classical and
learning-based perceptual quality metrics. Furthermore, the proposed technique
has the additional advantage of reducing temporal drift compared to previous
frame-by-frame approaches.

Fig. 1: Proposed RDAC framework. From a single reference frame, motion
keypoints and a compact residual layer, our framework reconstructs a video
with high perceptual and pixel fidelity. Note that when previously decoded
frame information is available, it is used to improve the prediction accuracy
of the next target residual through temporal correlation learning.

##  2 Related Work

Image animation models have been applied to compress talking head videos at
ultra-low bitrates in conferencing-type applications [1, 2, 3, 4, 5, 6].
Different from other learning-based compression frameworks [8, 9, 10, 11, 12,
13, 14], the animation-based codecs in [3] and [4] propose architectures that
use a variable number of motion keypoints to change the reconstruction quality
within a small range of low bitrates. The deep animation codec (DAC) in our
previous work [1] offers the possibility to vary the bitrate by creating a
list of reference frames from which the best reconstruction is computed.
Specifically, a new reference frame is added to the decoder buffer if all the
available frames give reconstruction below a predefined threshold. However,
this approach may introduce temporal jittering when adjacent animated frames
are predicted from different reference frames. Using second-order motion
coherence [6] introduces spatio-temporal stability in the decoded video, hence
reducing the jittering. However, this architecture is still limited in terms
of quality variability since it relies only on face animation. In our recent
work [2], we proposed a hybrid coding architecture (HDAC) that uses a low-
quality HEVC bitstream as side information to enhance the final result of the
animation codec. While improving on previous methods, the use of this low-
quality auxiliary stream limits in practice the possibility to reconstruct
high-frequency details.

In this work, we propose a residual deep animation codec (RDAC) that learns a
compact representation of the residual between a frame and its animation-based
prediction, and encodes this residual using temporal prediction.

##  3 PROPOSED METHOD

A general scheme of the proposed residual deep animation codec is depicted in
Fig. 1. The components of the proposed system are detailed as follows: Section
3.1 introduces the frame prediction and residual coding and Section 3.2
presents temporal learning in the residual space.

###  3.1 Deep Image Animation Prediction and Residual Coding

We leverage the principles developed in the First Order Model [7] for image
animation and our prior works [1, 2] for animation-based prediction. The image
animation process works by estimating a sparse set of motion landmarks using a
keypoint detector (KPD) which is a UNet-like architecture from [7]. The
keypoints are used by a motion transfer network (MTN) that generates the
optical flow between a decoded reference image $\mathbf{\tilde{X}}_{0}$ and
the desired target $\mathbf{X}_{t}$. Subsequently, the optical-flow map is
applied to the feature space representation of the reference frame derived by
the encoder of an autoencoder network. The deformed source features are
assumed to be a close approximation of the target frame’s feature
representation and are used by a decoder network to produce the final
animation $\mathbf{\hat{X}}_{t}$.

We build on this animation framework by including an encoder network that
learns a latent representation of
$\mathbf{R}_{t}=\mathbf{X}_{t}-\mathbf{\hat{X}}_{t}$ i.e. the residual after
animation as illustrated in Fig. 1. We start with the architecture of the
variational autoencoder network [9] used for learned image compression
frameworks. However, since the residual images have very sparse features we
mitigate the potential encoding of a noisy latent representation by increasing
the number of downsampling convolutional layers from 3 to 5 and symmetrically
increase the number of upsampling layers.

###  3.2 Using Temporal Correlation in the Residual Layer

For a sequence of target frames $\mathbf{X}_{1}\to\mathbf{X}_{T}$ animated
from a single reference frame, $\mathbf{X}_{0}$, we observe that the residual
differences $\mathbf{R}_{1}\to\mathbf{R}_{T}$ have a high temporal
correlation. In this paper, we use a simple differential coding scheme to
exploit this temporal correlation. Specifically, we compute the temporal
difference signal between consecutive frame residuals,
$\mathbf{D}_{t}=\mathbf{R}_{t}-\mathbf{\hat{R}}_{t-1}$, as shown in Fig. 1.
Note that, in general, more sophisticated prediction schemes are possible,
that could bring additional temporal decorrelation, e.g., any dense or block-
based motion compensated scheme. In this work, we demonstrated coding gains
even with a suboptimal zero-motion temporal predictor, leaving the study of
more advanced prediction schemes to future work.

The difference signal $\mathbf{D}_{t}$ is coded using an additional
autoencoder network, which is trained together with the animation-based
predictor and the reconstruction network. The decoding process consists in
reconstructing the residual
$\mathbf{\tilde{R}}_{t}=\mathbf{\tilde{D}}_{t}+\mathbf{\tilde{R}}_{t-1}$. The
reconstructed residual is then concatenated to the animation-based predictor
$\mathbf{\hat{X}}_{t}$ and passed as input to a reconstruction network that
produces the final decoded frame $\mathbf{\tilde{X}}_{t}$. The reconstruction
network consists of 2 convolution layers and 3 ResNet blocks.

###  3.3 Model Training

We initialize the animation module with pre-trained models from [1]. The loss
terms for image animation are the same as in [1, 7], while the rate-distortion
loss $\mathcal{L}_{RD}$ is derived as described in [9]:

$\mathcal{L}_{RD}=\lambda\cdot\text{MSE}(\mathbf{R_{t}},\mathbf{\hat{R}_{t}})+\text{Rate}$
(1)

where the bitrate cost in bits-per-pixel (bpp) is computed from the entropy
estimate of the residual latent representation.

Ground Truth | HEVC (8 kbps) | VVC (8kbps) | DAC (4.4 kbps) | HDAC (10kbps) | RDAC (Ours) (8 kbps)
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |
Fig. 2: Visual Comparison of coding results. A qualitative comparison of our
proposed coding framework shows significant quality improvement over HEVC and
VVC at low bitrates. We observe fewer smoothing and blocking artifacts as well
as better color and style preservation in the reconstructed frames with our
framework.

##  4 Experiments and Results

###  4.1 Evaluation Protocol

We randomly select 30 video sequences from the VoxCeleb test set with minimum
lengths of 128 frames. We note that chaning the GOP size affects the average
reconstruction quality of the video sequences. Therefore, we encode the
sequences with GOP sizes 16, 32, 64, and 128 and select the best
reconstruction point at each bitrate from a union of the computed metrics i.e.
the convex hull of all the GOP configurations. The reference frame is encoded
with QP 30 using the BPG codec (HEVC intra) and the motion keypoints as well
as the compressed residuals are entropy coded using a context-adaptive
arithmetic coder with a Prediction by Partial Match (PPM) model [15]. HEVC and
VVC (VTM-11) metrics are computed under low-delay configurations with high QP
values to minimize bitrate. We also compare against the LPIPS-VGG metrics
reported for BeyondKP [5] and FaceVid2Vid [3] since they use comparable test
conditions. Notice that for these last two methods, we only have a single
bitrate point, since they do not support bitrate variability beyond 10 kbps.
MSE loss is used at training time for residual learning. However, the other
loss terms used in training the network optimize for perceptual quality.
Therefore, we restrict our evaluation to use only perceptual metrics and
multi-scale pixel fidelity metrics.

###  4.2 RD Evaluation

In Tab. 1, we note over 70% bitrate savings for perceptual-based metrics i.e.
LPIPS [16], msVGG [17] and DISTS [18] as well as over 40% bitrate savings for
pixel-based metrics over HEVC. In Fig. 2 we make a visual comparison of our
proposed framework with HEVC and VVC in the low bitrate range.

Table 1: Bitrate savings (% BD-BR) computed over 30 video sequences with 128 frames from VoxCeleb test set. The bitrate savings are measured with HDAC [2], HEVC and VVC codec as anchors Metrics | HDAC | VVC | HEVC
---|---|---|---
msVGG | -55.10 | -66.65 | -74.99
DISTS | -48.46 | -63.01 | -82.62
LPIPS-VGG | -38.43 | -48.73 | -78.96
LPIPS | -6.45 | -33.11 | -75.96
FSIM | -42.89 | -16.02 | -63.10
MS-SSIM | -56.97 | -20.11 | -52.05

Fig. 3 illustrates the rate-distortion performance using the LPIPS metric.
RDAC significantly improves performance of conventional video codecs over a
wide range of bitrates, and it outperforms previous animation-based codecs
which do not employ predictive coding.

Fig. 3: RD Performance: Our method achieves considerable bitrate gains over a
wider range of bitrates relative to both state-of-the-art video compression
tools and the previously proposed frameworks.

###  4.3 Ablation study and temporal drift

An advantage of using a closed-loop prediction scheme for temporal coding of
residuals is that it avoids the temporal drifting affecting previous open-loop
schemes such as DAC. This is supported by Fig. 4, where we show the temporal
reconstruction quality (measured with MS-SSIM) of our framework and DAC.

We also investigate to which extent the temporal prediction contributes to the
RD gains, over a frame-by-frame scheme to code the prediction residuals
$\mathbf{R}_{t}$. To this end, we remove the temporal feedback loop in Fig. 1,
encoding the residuals as all Intra. Tab. 2 reports the gains of our proposed
RDAC (with temporal prediction) over this simpler solution, demonstrating that
reducing temporal correlation has a significant impact on coding performance.

Table 2: Bitrate Savings (% BD-BR) from our framework with temporal residual learning versus no temporal residual (10 sequences/64 frames) DISTS | LPIPS | msVGG | FSIM | MS-SSIM
---|---|---|---|---
-9.35 | -14.32 | -5.65 | -9.20 | -6.86
Fig. 4: Reconstruction quality as a function of time: RDAC temporal prediction
avoids temporal drift, in contrast with open-loop schemes such as DAC, where
quality degrades as the target frames get farther from the reference.

###  4.4 Computational complexity

In Tab. 3, we make a complexity evaluation by comparing the coding or decoding
time for a single interframe. The animation-based models DAC, HDAC, and our
framework are evaluated on a CPU and GPU while the HEVC and VVC codecs are
only evaluated on a CPU since they do not have GPU acceleration capability. We
note that our proposal adds only a moderate level of complexity relative to
HEVC. However since we achieve bitrate savings greater than VVC, we consider
this additional complexity as an acceptable tradeoff for the target
application.

Table 3: Computational Complexity: Time to encode/decode 1 frame (in seconds). The computation time is estimated for the highest RD point of our framework.  | CPU (Intel Corei7) | GPU (RTX 3090)
---|---|---
| Enc. | Dec. | Enc. | Dec.
HEVC | 0.09 | 0.005 | - | -
VVC | 13.5 | 0.01 | - | -
DAC | 0.04 | 0.35 | 0.03 | 0.02
HDAC | 0.10 | 0.35 | 0.12 | 0.02
RDAC (Ours) | 0.52 | 0.50 | 0.21 | 0.02

##  5 Conclusions

Animation-based compression offers the possibility to transmit videos with
very low bitrate. However, it is often limited to reconstructing the outputs
at a fixed quality level, cannot scale efficiently when higher bandwidth is
available, and does not compress efficiently temporal redundancies in the
signal. In this paper, we propose a coding scheme that integrates image
animation (re-interpreted as a frame predictor) with classical predictive
coding principles, where we exploit both spatial and temporal dependencies to
achieve a coding gain. Our RDAC codec outperforms previous methods and
standard codecs by a large margin on a dataset of talking head videos, despite
the very simple temporal prediction approach employed.

Acknowledgement: This work was funded by Labex DigiCosme - Université Paris-
Saclay. This work was performed using HPC resources from GENCI-IDRIS

## References

  * [1] Goluck Konuko, Giuseppe Valenzise, and Stéphane Lathuilière,  “Ultra-low bitrate video conferencing using deep image animation,”  in IEEE ICASSP, 2020.
  * [2] Goluck Konuko, Stéphane Lathuilière, and Giuseppe Valenzise,  “A hybrid animation codec for low bitrate video conferencing,”  in IEEE ICIP, 2022.
  * [3] Ting-Chun Wang, Arun Mallya, and Ming-Yu Liu,  “One-shot free-view neural talking-head synthesis for video conferencing,”  in CVPR, 2021.
  * [4] Maxime Oquab, Pierre Stock, Oran Gafni, Daniel Haziza, Tao Xu, Peizhao Zhang, Onur Celebi, Yana Hasson, Patrick Labatut, Bobo Bose-Kolanu, et al.,  “Low bandwidth video-chat compression using deep generative models,”  arXiv preprint arXiv:2012.00328, 2020.
  * [5] Bolin Chen, Zhao Wang, Binzhe Li, Rongqun Lin, Shiqi Wang, and Yan Ye,  “Beyond keypoint coding: Temporal evolution inference with compact feature representation for talking face video compression,”  in DCC, 2022.
  * [6] Zhehao Chen, Ming Lu, Hao Chen, and Zhan Ma,  “Robust ultralow bitrate video conferencing with second order motion coherency,”  in 2022 IEEE 24th International Workshop on Multimedia Signal Processing (MMSP), 2022, pp. 1–6.
  * [7] Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe,  “First order motion model for image animation,”  in Neurips, 2019.
  * [8] Oren Rippel and Lubomir Bourdev,  “Real-time adaptive image compression,”  arXiv preprint arXiv:1705.05823, 2017.
  * [9] Johannes Ballé, David Minnen, Saurabh Singh, Sung Jin Hwang, and Nick Johnston,  “Variational image compression with a scale hyperprior,”  arXiv preprint arXiv:1802.01436, 2018.
  * [10] Eirikur Agustsson, Michael Tschannen, Fabian Mentzer, Radu Timofte, and Luc Van Gool,  “Generative adversarial networks for extreme learned image compression,”  in IEEE ICCV, 2019.
  * [11] Zhihao Hu, Guo Lu, and Dong Xu,  “Fvc: A new frame- work towards deep video compression in feature space,”  in IEEE Conference on Computer Vision and Pattern Recognition(CVPR), 2021.
  * [12] Johannes Ballé, Valero Laparra, and Eero P Simoncelli,  “End-to-end optimized image compression,”  Neurips, 2016.
  * [13] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao,  “Dvc: An end-to-end deep video compression framework,”  in IEEE CVPR, 2019.
  * [14] Fabian Mentzer, George Toderici, David Minnen, Sung-Jin Hwang, Sergi Caelles, Mario Lucic, and Eirikur Agustsson,  “Vct: A video compression transformer,”  in NeurIPS, 2022.
  * [15] J. Cleary and I. Witten,  “Data compression using adaptive coding and partial string matching,”  IEEE Transactions on Communications, vol. 32, no. 4, pp. 396–402, 1984.
  * [16] Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang,  “The unreasonable effectiveness of deep features as a perceptual metric,”  in CVPR, 2018.
  * [17] Karen Simonyan and Andrew Zisserman,  “Very deep convolutional networks for large scale image recognition,”  in ICLR, 2015.
  * [18] Keyan Ding, Kede Ma, Shiqi Wang, and Eero P. Simoncelli,  “Image quality assessment: Unifying structure and texture similarity,”  in IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, vol. 44, pp. 2567–2581.


# Modeling infant object perception as program induction

J.-Philipp Fränken1∗, Christopher G. Lucas2, Neil R. Bramley2, Steven T.
Piantadosi3
1Stanford University, 2The University of Edinburgh, 3University of California,
Berkeley<EMAIL_ADDRESS>

## Abstract

Infants expect physical objects to be rigid and persist through space and time
and in spite of occlusion. Developmentists frequently attribute these
expectations to a “core system” for object recognition. However, it is unclear
if this move is necessary. If object representations emerge reliably from
general inductive learning mechanisms exposed to small amounts of environment
data, it could be that infants simply induce these assumptions very early.
Here, we demonstrate that a domain general learning system, previously used to
model concept learning and language learning, can also induce models of these
distinctive “core” properties of objects after exposure to a small number of
examples. Across eight micro-worlds inspired by experiments from the
developmental literature, our model generates concepts that capture core
object properties, including rigidity and object persistence. Our findings
suggest infant object perception may rely on a general cognitive process that
creates models to maximize the likelihood of observations.111Project page:
janphilippfranken.github.io/object-perception

> Keywords: core knowledge; perception; vector quantization; program
> induction; Bayes

## Introduction

Object representations serve as compositional building blocks for higher level
cognition in both humans and machines Xu Carey (1996); Schölkopf . (2021);
Chen . (2022). Developmental accounts suggest that infants rely on a “core
system” for object representations to perceive the boundaries of objects,
accurately represent their shapes even when they are partially or fully
occluded, and make predictions about object movements and their final
positions Spelke Kinzler (2007). Having a specific system for representing
objects from an early age can be beneficial because it allows for the
incorporation of prior knowledge and expectations about objects and their
physical regularities, such as the idea that objects usually maintain their
shape and size as they move (rigidity principle; spelke1990principles) and
continue to exist and retain their properties even when occluded (object
persistence principle; baillargeon1987object, baillargeon2008innate). Despite
converging evidence for the existence of a core object system in both human
infants (e.g., feigenson2003tracking, spelke2022babies) and non-human animals
(e.g., chiandetti2015inexperienced, hauser2003spontaneous), it is not clear if
a system specifically designed for this purpose is necessary or beneficial if
object representations can be learned effectively by a domain general
inductive system from only a small amount of data.

Figure 1: Illustration of inference pipeline. We take short videos (ten
frames) as input. These are preprocessed into a sequence of discrete feature
maps using vector quantization followed by one-hot encodings on each feature
map to obtain Boolean tensors (or “bitmasks”). Bitmasks are then processed by
a generic Bayesian concept learning algorithm to induce programs that
parsimoniously explain the underlying structure in the discretized data. For
example, evaluating the program $\mathcal{F}$ will move (“roll”) the upper
bitmask ($t$) by 1 on the x dimension predicting the bitmask shown below
($t_{+1}$).

## Infant object perception as program induction

We assume that object representations can be used to efficiently compress and
discretize perceptual input (e.g., visual). As such, object representations
might arise from reasoning about the physical regularities Spelke (1990) or
“invariants” Sloman Lagnado (2004) in one’s environment that facilitate
predictions about its future states. Inspired by recent advances in Bayesian
program learning Ellis . (2021); Tang Ellis (2022); Yang Piantadosi (2022) and
intuitive physics Piloto . (2022), we present an idealized model (Figure 1)
that discovers object representations and their physical regularities from
short sequences of 2D images (which should generalize to 3D scene
projections). Our model can be summarized in four steps: (1) Extract a
discrete ”codebook” representation $c$ for each image $x$ using a VQ-VAE
(van2017neural), a simple tool for efficient image encoding without relying on
semantic object assumptions $\rightarrow$ (2) Apply $n$ deterministic one-hot
encodings to each discrete feature map $c$ to generate Boolean tensor
representations (”bitmasks”), with $n$ representing the number of unique codes
in $c$ $\rightarrow$ (3) Use a Bayesian concept learning algorithm to process
the resulting bitmasks and generate programs that parsimoniously explain the
structure in the data $\rightarrow$ (4) Use discovered programs to improve the
representation by searching for structure in residuals or imputing missing
data to maximize likelihood. The final two steps are repeated until
convergence or until a time-out threshold is reached. To discover programs,
our model generates compositions of functions from the primitives listed in
Table 1 and computes posterior distributions over programs using Bayes’ rule:
$P(H\mid D)\propto P(H)P(D\mid H)$. The prior probability of a program $P(H)$
is determined by a probabilistic context-free grammar (PCFG) based on the
operations in Table 1. For likelihood $P(D\mid H)$ we assume a standard
exponential loss function. We use stochastic (MCMC) sampling as in
goodman2008rational to search for programs.

Table 1. Assumed primitive functions  Type Primitives Number functions (add
n), (sub n), (mult n), (div n), (mod n), (neg), (const) Set functions (union),
(intersection) Bitmask functions (move x n), (move y n), (complement), (const)

The space of programs consists of all compositions of these functions that
respect the input and output types.

a)b)c)d)e)f)g)h) Figure 2: Illustration of tested micro-worlds and learning
curves for the model. The $x$ axis corresponds to the number of samples (i.e.,
the length of the MCMC chain) and the $y$ axis corresponds to the log-
likelihood of the final program(s) from a given chain averaged across $100$
independent runs. Grey shadings correspond to standard error. Full sequence
for each micro-world and target program(s) are shown at the bottom of each
panel.

## Experiments

We evaluate our model’s ability to learn object representations and their
regularities across eight micro-worlds inspired by experiments from the
developmental literature. We use ten images for each probe. To demonstrate our
approach, we first examine a baseline probe including simple left-right
movement (Fig. 2a). To show that we can also handle natural categories that
violate standard object properties, we next consider a “melting” block (i.e.,
a block that is shrinking vertically; Fig. 2b). We then test the ability of
our approach to discover, from sparse input, principles that are often
considered as core knowledge, including the widely studied principles of
object persistence (baillargeon2008innate, piloto2022intuitive; Fig. 2c–d) and
rigidity (spelke1990principles, kemp2008ideal; Fig. 2e–g) . We additionally
include an example of unchangeableness following occlusion
(baillargeon2012core; Fig. 2h).

## Results

Panels a–b in Fig. 2 show that our model can find programs capturing simple
object regularities such as constant left-right movement, which can be
expressed as (@$\lambda$@ (move x n)) as well as “melting” which can be
expressed as (@$\lambda$@ (intersection (move y (neg n)) (const))). Figure 2c
shows that this ability still holds for an object that moves behind an
occluder. Figure 3a shows the probability of the occluded object for frame $5$
in Figure 2c. Consistent with a flattening learning curve at 1000 samples, the
model is learning representations at around 1000 samples. The representation
of the occluded object was obtained by imputing its representation using a
program such as (@$\lambda$@ (move x n)) which will have a maximum likelihood
if it keeps representing the object during occlusion. In line with this idea,
the example in Figure 2d has a weaker learning curve as the object does not
reappear, making it harder to find a physically plausible regularity of the
object. Overall, these findings are consistent with increased surprise in
infants when objects suddenly disappear or reappear after obstacles as well as
their tendency to keep representing objects during occlusion Baillargeon
(2008).

a)b) Figure 3: a) Probability of predicted object (greyscale) during occlusion
for different numbers of samples. b) Average object counts ($\pm$ SEM) for the
three example tests of rigidity.

Results in Figure 2e–g demonstrate our model’s ability to interpret ambiguous
scenes involving two blocks of different sizes similar to how infants do
Kestenbaum . (1987); Spelke . (1989). Specifically, our model provides a
single-object interpretation for the example shown in Figure 2e and a two-
object interpretation for the examples shown in Figure 2f-g, which require two
regularities (both (@$\lambda$@ (move x n)) and (@$\lambda$@ (const))).
Average object counts for different numbers of samples are shown in Figure 3b,
and stable performance is achieved around 1000 sample. Our final example
(Figure 2h) demonstrates the concept of unchangeableness where an object is
occluded by a plank moving across the scene (we do not model the plank’s
regularity). Following the same imputation approach as in Figure 2c, our model
can efficiently learn the objects regularity from a small amount of data.

## Discussion

Object representations form a fundamental aspect of human and machine
cognition. We proposed that these representations can be learned by domain
general learning system that aims to induce symbolic programs to maximize the
likelihood of observations. A limitation of the present proof-of-concept
results is that we did not jointly train the VQ-VAE and search for programs
but instead trained the VQ-VAE prior to search to obtain discrete codebooks.
Future work should thus explore joint end-to-end learning of both the VQ-VAE
and program search to test our model in more complex scenes (e.g.,
piloto2022intuitive, maoclevrer).

## References

  * Baillargeon (1987) baillargeon1987objectBaillargeon, R.  1987\.  Object permanence in 3$1/2$-and 4$1/2$-month-old infants. Object permanence in 3$1/2$-and 4$1/2$-month-old infants.  Developmental psychology235655.
  * Baillargeon (2008) baillargeon2008innateBaillargeon, R.  2008\.  Innate ideas revisited: For a principle of persistence in infants’ physical reasoning Innate ideas revisited: For a principle of persistence in infants’ physical reasoning.  Perspectives on Psychological Science312–13.
  * Baillargeon Carey (2012) baillargeon2012coreBaillargeon, R. Carey, S.  2012\.  Core cognition and beyond: The acquisition of physical and numerical knowledge. Core cognition and beyond: The acquisition of physical and numerical knowledge.
  * Chen . (2022) chen2022unsupervisedChen, H., Venkatesh, R., Friedman, Y., Wu, J., Tenenbaum, JB., Yamins, DL. Bear, DM.  2022\.  Unsupervised Segmentation in Real-World Images via Spelke Object Inference Unsupervised segmentation in real-world images via spelke object inference.  arXiv preprint arXiv:2205.08515.
  * Chiandetti . (2015) chiandetti2015inexperiencedChiandetti, C., Spelke, ES. Vallortigara, G.  2015\.  Inexperienced newborn chicks use geometry to spontaneously reorient to an artificial social partner Inexperienced newborn chicks use geometry to spontaneously reorient to an artificial social partner.  Developmental Science186972–978.
  * Ellis . (2021) ellis2021dreamcoderEllis, K., Wong, C., Nye, M., Sablé-Meyer, M., Morales, L., Hewitt, L.Tenenbaum, JB.  2021\.  Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning Dreamcoder: Bootstrapping inductive program synthesis with wake-sleep library learning.  Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation Proceedings of the 42nd acm sigplan international conference on programming language design and implementation ( 835–850).
  * Feigenson Carey (2003) feigenson2003trackingFeigenson, L. Carey, S.  2003\.  Tracking individuals via object-files: evidence from infants’ manual search Tracking individuals via object-files: evidence from infants’ manual search.  Developmental Science65568–584.
  * Goodman . (2008) goodman2008rationalGoodman, N., Tenenbaum, J., Feldman, J. Griffiths, T.  2008\.  A rational analysis of rule-based concept learning A rational analysis of rule-based concept learning.  Cognitive Science321108–154.
  * Hauser Carey (2003) hauser2003spontaneousHauser, MD. Carey, S.  2003\.  Spontaneous representations of small numbers of objects by rhesus macaques: Examinations of content and format Spontaneous representations of small numbers of objects by rhesus macaques: Examinations of content and format.  Cognitive Psychology474367–401.
  * Kemp Xu (2008) kemp2008idealKemp, C. Xu, F.  2008\.  An ideal observer model of infant object perception An ideal observer model of infant object perception.  Advances in neural information processing systems21.
  * Kestenbaum . (1987) kestenbaum1987perceptionKestenbaum, R., Termine, N. Spelke, ES.  1987\.  Perception of objects and object boundaries by 3-month-old infants Perception of objects and object boundaries by 3-month-old infants.  British journal of developmental psychology54367–383.
  * Mao . (2022) maoclevrerMao, J., Yang, X., Zhang, X., Goodman, N. Wu, J.  2022\.  CLEVRER-Humans: Describing Physical and Causal Events the Human Way Clevrer-humans: Describing physical and causal events the human way.  Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track. Thirty-sixth conference on neural information processing systems datasets and benchmarks track.
  * Piloto . (2022) piloto2022intuitivePiloto, LS., Weinstein, A., Battaglia, P. Botvinick, M.  2022\.  Intuitive physics learning in a deep-learning model inspired by developmental psychology Intuitive physics learning in a deep-learning model inspired by developmental psychology.  Nature human behaviour691257–1267.
  * Schölkopf . (2021) scholkopf2021towardSchölkopf, B., Locatello, F., Bauer, S., Ke, NR., Kalchbrenner, N., Goyal, A. Bengio, Y.  2021\.  Toward causal representation learning Toward causal representation learning.  Proceedings of the IEEE1095612–634.
  * Sloman Lagnado (2004) sloman2004causalSloman, S. Lagnado, DA.  2004\.  Causal invariance in reasoning and learning Causal invariance in reasoning and learning.  Psychology of learning and motivation44287–326.
  * Spelke (1990) spelke1990principlesSpelke, ES.  1990\.  Principles of object perception Principles of object perception.  Cognitive science14129–56.
  * Spelke (2022) spelke2022babiesSpelke, ES.  2022\.  What Babies Know: Core Knowledge and Composition Volume 1 What babies know: Core knowledge and composition volume 1 ( 1).  Oxford University Press.
  * Spelke Kinzler (2007) spelke2007coreSpelke, ES. Kinzler, KD.  2007\.  Core knowledge Core knowledge.  Developmental science10189–96.
  * Spelke . (1989) spelke1989objectSpelke, ES., von Hofsten, C. Kestenbaum, R.  1989\.  Object perception in infancy: Interaction of spatial and kinetic information for object boundaries. Object perception in infancy: Interaction of spatial and kinetic information for object boundaries.  Developmental Psychology252185.
  * Tang Ellis (2022) tang2022perceptionTang, H. Ellis, K.  2022\.  From perception to programs: regularize, overparameterize, and amortize From perception to programs: regularize, overparameterize, and amortize.  arXiv preprint arXiv:2206.05922.
  * Van Den Oord . (2017) van2017neuralVan Den Oord, A., Vinyals, O. .  2017\.  Neural discrete representation learning Neural discrete representation learning.  Advances in neural information processing systems30.
  * Xu Carey (1996) xu1996infantsXu, F. Carey, S.  1996\.  Infants’ metaphysics: The case of numerical identity Infants’ metaphysics: The case of numerical identity.  Cognitive psychology302111–153.
  * Yang Piantadosi (2022) yang2022oneYang, Y. Piantadosi, ST.  2022\.  One model for the learning of language One model for the learning of language.  Proceedings of the National Academy of Sciences1195e2021865119.


numbersnone#1none:
left:
right:

#  GenCHiP: Generating Robot Policy Code for
High-Precision and Contact-Rich Manipulation Tasks

Kaylee Burns∗1,2, Ajinkya Jain1, Keegan Go1, Fei Xia3, Michael Stark1, Stefan
Schaal1, Karol Hausman2,3 *Correspondence to<EMAIL_ADDRESS>Intrinsic 2Stanford University 3Google DeepMind

###### Abstract

Large Language Models (LLMs) have been successful at generating robot policy
code, but so far these results have been limited to high-level tasks that do
not require precise movement. It is an open question how well such approaches
work for tasks that require reasoning over contact forces and working within
tight success tolerances. We find that, with the right action space, LLMs are
capable of successfully generating policies for a variety of contact-rich and
high-precision manipulation tasks, even under noisy conditions, such as
perceptual errors or grasping inaccuracies. Specifically, we reparameterize
the action space to include compliance with constraints on the interaction
forces and stiffnesses involved in reaching a target pose. We validate this
approach on subtasks derived from the Functional Manipulation Benchmark (FMB)
and NIST Task Board Benchmarks. Exposing this action space alongside methods
for estimating object poses improves policy generation with an LLM by greater
than 3x and 4x when compared to non-compliant action spaces. More material is
available on our project webpage: https://dex-code-gen.github.io/dex-code-gen/

##  I Introduction

Many of the open problems in learning-based robotics revolve around the issue
of scaling: deep-learning methods require vast datasets that are not readily
available for robotics applications. One workaround for the data scarcity
problem is to retrofit deep learning models that have been trained on
internet-scale datasets from other modalities for robotics tasks. Recently,
large language models (LLMs) have emerged as a strong candidate for this
approach. LLMs are able to successfully generate code, complete numeric
sequences, and solve common-sense reasoning tasks [1, 2, 3, 4]. Because code
is one of the most popular interfaces for specifying robotic planning and
control commands, these capabilities hint at enormous potential when applied
to robotics [1].

Past work demonstrates that generating robot policy code from LLMs is
successful for high-level tasks such as navigation and open-vocabulary pick-
and-place [1, 2]. For example, a language model can compose high-level action
primitives like grab(chips) and move_to(human) to successfully generate a
policy conditioned on a natural language command such as “bring me the chips”
[1]. But at present, lower-level tasks and behaviors are generally considered
out of reach for LLMs because, to the best of our knowledge, there are no
compelling demonstrations of code generation that commands robots to perform
contact-rich tasks.

(a)

(b)

Figure 1: (a) We prompt an LLM to generate code for high-precision tasks. By
using an action space that paramterizes compliant behavior, the LLM is able to
generate action sequences for contact-rich tasks like peg insertion. (b)
Language models’ ability to reason about object geometry and make plans by
using world knowledge about different object enables zero-shot generalization
to new tasks.

While various robot learning approaches have been able to demonstrate
impressive generalization across different settings and target objects for
pick-and-place tasks [5, 6, 7, 8, 9, 10], such generalization is arguably more
difficult for dexterous tasks where a higher level of precision is required.
For example, for a peg-in-hole insertion task, surfaces with more friction or
tight insertion tolerances may require multiple insertion attempts or contact
force tuning to reach the insertion site [11]. Similarly, pegs with different
geometries may need different approach trajectories to achieve proper
alignment: a peg with a star-shaped cross-section may require an initial
rotation for insertion, whereas no such rotation is required for a circular
peg. In practice, the parameters of contact-rich insertion skills are mostly
tuned by experts to handle these differences and automating this process is
still an open problem. This presents a challenge for the approach taken in
past work [1], which directly provides a language model with a library of
high-level skills (such as insert(peg)).

We present GenCHiP: a promising alternative for automating the parameter-
tuning process within the control API by instead leveraging the world
knowledge inside language models (LMs) to compose lower-level control
primitives. Our goal is to understand if LLMs have the ability to reason about
motions and forces acting on objects so as to enable generalization across a
larger class of target objects and skills. To study this topic, we modify the
action space in which an LLM operates by exposing constraints on the contact
stiffness and forces observed during a high-precision task. With these
modifications we can study code generation for contact-rich manipulation
tasks, including industry relevant tasks such as high-precision insertion,
rigid body assembly, and deformable object manipulation (see Fig. 1).

The main contribution of this work is to demonstrate that LLMs, without any
specialized training, have the ability to perform contact-rich tasks when
given the appropriate action space. We develop a system for automatically
generating robot policy code for dexterous tasks by allowing LLMs to specify
constraints on the stiffnesses, forces, and trajectories required to perform
contact-rich manipulation tasks. We show that our method, GenCHiP, is able to
outperform a contact-unaware model by over 3x on average on subtasks developed
from two challenging contact-rich benchmarks. Specifically, our approach is
able to generate novel insertion patterns from high level descriptions of
object shape and texture on insertion tasks from the Functional Manipulation
Benchmark (FMB) [12], route and un-route cables in the style of the IROS 2020
Robotic Grasping and Manipulation Competition (IROS RGMC) [13], and complete a
waterproof connector insertion task selected from the NIST Assembly Task Board
#1 [14]. To the best of our knowledge, this is the first paper to provide a
strong proof-of-concept that LLMs are capable of parameterizing robotic
policies for precise, contact-rich manipulation tasks with code.

##  II Related Work

LLMs for robotics can successfully generate robot policy code for pick-and-
place style manipulation tasks [15, 16], compose mid-level plans for
navigation tasks [3], and compose multiple navigation and manipulation skills
for integrated household agents [17, 18, 19]. Many of these approaches rely on
filtering LLM-generated code based on what is executable [20] or on making
queries hierarchically [1, 10]. They also require a large number of prompt
examples: for example, [21] makes use of eight different prompts with
approximately nine subcommands per prompt. By contrast, in the Few-Shot
example settings we describe in Sec. IV-A, we use a maximum of four examples
in a given environment. Furthermore, none of these works establish whether or
not LLMs can generate robot policy code for performing high-precision,
contact-rich manipulation tasks, which we study in this work.

Contact-rich robot manipulation tasks are those that involve a robot making
controlled contact with its environment while performing them. These tasks
constitute the vast majority of manipulation tasks in daily life, including
household tasks such as wiping tables and sweeping dust into a dust-pan [22],
and industrial tasks such as high precision insertion [23, 24, 11] and
assembly [25, 26]. A robot needs to reason about the contact forces it will
impart on and sense from the environment while performing such tasks to
complete them successfully. Learning a general policy to perform a wide array
of contact-rich manipulation tasks has been studied in great detail in
robotics [27, 28, 29, 24, 30, 31, 23, 32], yet how to find a general approach
to these tasks remains an open question. Prior work directly learns policies
with imitation learning [33] or reinforcement learning [34, 25, 35], but these
require hundreds of human demonstrations, significant operator training,
dedicated simulators, or thousands of environment interactions to achieve a
performant policy. We step towards obtaining a general policy for contact-rich
manipulation tasks by leveraging the world knowledge inside LLMs and combining
it with the appropriate task action spaces. We choose robot impedance (or
equivalently, admittance) control as the action space for contact-rich
manipulation tasks as it can regulate the relationship between robot position
and contact forces effectively [36, 37].

##  III Preliminaries

⬇

# You’re a robot trying to insert a peg in a hole. Grab the circular peg.

pick_up(circular_peg)

Figure 2: We generate code by formatting natural language requests and
instructions as comments. Generations are highlighted in blue.

Our goal is to develop a system that can translate natural language
instructions into robotic actions by leveraging a sufficiently expressive API
for control. Past work [1] shows that off-the-shelf language models can be
adapted towards this goal with few-shot prompting. Concretely, pairs of
natural language requests with corresponding robot policy code are fed into a
language model. Then, the language model can output novel programs in response
to new commands as shown in Fig. 2. The success of this approach can be
attributed to the fact that during offline training on vast internet datasets,
language models absorb world knowledge about common-sense interactions and
learn mappings between natural language instructions and code. Strategies for
adapting this approach towards a contact-rich setting are discussed in the
next section.

##  IV GenCHiP

GenCHiP equips LLMs with a compliant action space. In this section, we first
describe the prompting strategies that enable a language model to successfully
paramterize and compose compliant move actions. Then, we discuss the different
choices of action spaces that can be made available to a language model
including our proposed action space.

###  IV-A Prompting for contact-rich control

We consider four prompting strategies when generating robot policy code from a
language model:

a. Task descriptions are high-level explanations of the scene and the task
goal written in natural language.

⬇

"""You’re a robot trying to undo cable routing. Unroute the cable from the
screws and brackets it is wrapped around."""

(a) Task Description

⬇

"""Use these methods:

- move: moves to specified offset

Args:

translation: (x, y, z) tuple

rotation: (x, y, z) tuple

..."""

(b) Control API Descriptions

⬇

"""Rules:

- Don’t define any new methods

- Don’t call any undefined methods

- Don’t add any if statements or while loops"""

(c) Hints

⬇

# Move the cable to the left until it snags

move((1, 0, 0),

constraint=(x.force>-1))

(d) Examples

Figure 3: We present information about the task and control API via prompting.
The API description is the same across all environments, the hints and
examples are the same within each environment, while the task description must
be modified to describe each task.

These can occur at both the beginning and end of a prompt and often include
important information about the task setup such as the peg shape or the
available objects. See Fig. 3a for an example.

b. Descriptions of available control APIs are formatted doc-strings that
describe code accessible to the LLM. These include lists of variables as well
as the expected range of values for floating point numbers. Fig. 3b shows an
example description for a simplified move function that controls end-effector
pose. We also include descriptions for the full library of available methods,
including a point-to-point move, a compliant move, conditions, gripper
movements, and methods or variables specifying the positions of relevant
objects.

c. Hints in our setting include rules, keywords that specify relevant control
primitives, and requests to have the model explain its reasoning in natural
language or in pseudocode. Phrases such as “perform a pattern search” guide
the model towards predicting behavior that better recovers from errors and
better handles imprecision in the position of target poses. Intuitively these
keywords help in reducing task ambiguity (e.g., by emphasizing that provided
locations are imprecise) and guide the model towards motion patterns that are
relevant to contact-rich tasks. Requests to explain in natural language can be
thought of as a variant of chain-of-thought prompting [38]. The specific
keywords and requests that are helpful in each task are described in the
experimental section.

d. Examples of the control APIs being used for basic movements, such as making
contact with a surface, are useful for tasks with ambiguity or where the
desired force constraints are difficult to infer from the given ranges.

The combination of the prompt strategies described above allows us to prompt a
language model with enough contextual information about the dexterous task at
hand. Next, we discuss how we can design the action space of the robot to be
able to perform such tasks in practice.

###  IV-B Action spaces for robot manipulation tasks

Past approaches assume access to a library of methods that exhaustively cover
all user-requested commands [1, 2]. Building such a library is challenging for
contact-rich tasks because in practice these policies are tuned by experts
across different object geometries, frictions, and scene layouts. This section
describes different approaches to parameterizing the control API and what the
right choice of a control API can achieve. Formally, we consider a contact-
rich robot manipulation task to be composed of a sequence of subtasks
$\tau=(t_{1},\ldots,t_{n})$. The specific definition of a subtask will change
based on the action space, as described below.

⬇

# Insert a peg into a hole

pick_up(peg)

# go down to make contact

move_point_to_point([0, 0, -1])

# wiggle to find opening

move_point_to_point([1, 0, 0])

move_point_to_point([-1, 0, 0])

# go down to insert

move_point_to_point([0, 0, -1])

Figure 4: Generating code with point-to-point moves limits policies to free-
space-motions. For this policy to run successfully, displacement along the
z-axis in the second and fifth actions must be exact.

Point-to-point moves. Past work [1, 2] makes use of an action space that
directly commands the robot to move to target poses in the Cartesian space,
$[\mathbf{x}_{target}]_{i}$ (See Fig. 4). In this setting, each sub-task
$t_{i}$ is simply defined as the next Cartesian pose (a.k.a. waypoint):
$t_{i}=([\mathbf{x}_{target}]_{i})$.

While this approach is successful for executing motions in free-space or for
simple pick-and-place tasks, it fails when the robot needs to explicitly make
a purposeful contact with its environment. Consider a robot trying to make
contact with a surface to perform a wiping motion. Successfully parameterizing
a policy in this action space would require predicting a precise Cartesian
pose with very little tolerance for error. Predicting a pose that is
millimeters short of the surface would fail to make a contact and predicting
millimeters too deep into the surface would cause the robot to exert high
forces on the surface, which can cause faults in the robot or even break it in
the worst case scenario.

Compliant moves. Addressing this shortcoming, we propose to parameterize the
action space for performing contact-rich manipulation tasks using a robot’s
compliance, realized in impedance control (or equivalently admittance control
which is an adequate action space for robot learning in [39]). An impedance
move action is parameterized by both a target Cartesian pose,
$[\mathbf{x}_{target}]_{i}$, and a vector that specifies stiffness along each
degree of freedom, $\sigma_{i}$:
$t_{i}=([\mathbf{x}_{target}]_{i},\sigma_{i})$, when the robot is in contact
with the environment. During execution, the stiffness vector for each subtask
can be used to define the parameters for a variable impedance controller [40]
of the form:

$\displaystyle
F_{external}=K_{p}(\mathbf{x}_{target}-\mathbf{x})+K_{d}(\dot{\mathbf{x}}_{target}-\dot{\mathbf{x}})+\Lambda(\ddot{\mathbf{x}}_{target}-\ddot{\mathbf{x}})$
(1)

where $\mathbf{x}_{target}$, $\mathbf{x}$, $\dot{\mathbf{x}}_{target}$,
$\dot{\mathbf{x}}$, $\ddot{\mathbf{x}}_{target}$, and $\ddot{\mathbf{x}}$
denote the target and current pose, twist, and accelerations, respectively.
$K_{p}$, $K_{d}$, and $\Lambda$ correspond to the stiffness, damping, and
task-space inertia matrices, respectively. The impedance controller realizes
that the end-effector in contact with the environment behaves like the linear
spring-damper-mass system above. $K_{p}$, $K_{d}$, and $\Lambda$ are computed
as a function of our specified stiffness vector $\sigma_{i}$ (explained below)
and robot specific parameters in order to achieve stable yet responsive
behavior.

⬇

# Insert a peg into a hole

pick_up(peg)

# go down to make contact

move_conditional_compliant([0, 0, -1],

constraint=(z.force>1))

# wiggle to find opening; stop when force lessens

move_conditional_compliant([1, 0, 0],

constraint=(z.force<1))

move_conditional_compliant([-1, 0, 0],

constraint=(z.force<1))

# go down to insert

move_conditional_compliant([0, 0, -1],

constraint=(z.force>2))

Figure 5: Compliance prevents the robot from faulting when in contact.
Conditional termination constraints enable the language model to reason about
contact forces. In this example, the robot moves a cable back and forth until
no more upward force is detected, which indicates that an opening has been
found.

Intuitively, the stiffness vector determines the interaction forces that the
robot will impart on its environment while performing the task. Low stiffness
coefficients in $\sigma$ regulate the robot’s compromise between contact
forces and the attempt to achieve position accuracy. In the example that we
discussed in the last paragraph, a low stiffness value would enable the robot
to maintain gentle contact with a surface that prevents the robot from
reaching a desired position. A higher stiffness value would create higher
contact forces, equivalent to a higher priority to reduce position error.

Conditional compliant moves. In addition to the impedance control
specification described above, we also allow the LLM to specify conditions
under which to terminate an impedance move. Specifically, these are thresholds
on force or position in a specified coordinate direction. Example pseudocode
is presented in Fig. 5. This is a powerful primitive as it enables the robot
to construct recipes for high-precision tasks without relying on fine-grained
perception. In the example of making contact with a surface, this may look
like moving a peg downwards with a termination constraint on upward force.

##  V Experiments

In this section, we evaluate the ability of GenCHiP to generate code for fine-
grained manipulation tasks that require high precision in a series of
experiments on a set of real robotic tasks. Specifically, we evaluate on a
subset of high-precision contact-rich manipulation tasks from the Functional
Manipulation Benchmark (FMB) [12] and a set of industrial manipulation tasks
selected from the NIST Assembly Task Board #1 [14] and the IROS 2020 Robotic
Grasping and Manipulation Competition (IROS RGMC) [13] (Sec. V-A). Later, we
ablate prompt hints to study the utility of incorporating additional hints in
generating relevant motion patterns for robot manipulation tasks (Sec. V-B).

###  V-A Contact-rich manipulation tasks

Sec. V-A1 details the task setup considered in our experiments, while
baselines and ablations are discussed in Sec. V-A3. We discuss experimental
results in Sec. V-A5.

####  V-A1 Task description

The Functional Manipulation Benchmark [12] studies robotic manipulation,
grasping, reorienting, and assembling of a set of dozens of 3D printed
objects. The benchmark emphasizes generalization across different object
shapes and positions. We evaluate our approach on a subset of peg insertion
tasks across three different object shapes: the circle, star, and half-pipe.
We use scripted motion to bring the pegs into a fixed position over the
insertion point and program a randomized rotation about the z-axis. There is
no rotation for the circular peg because it has a constant radius. Rotation of
the star is sampled uniformly between 0 and $\frac{\pi}{2}$. Rotation of the
half pipe is sampled uniformly from either 0 or $\pi$. Inserting these peg
shapes successfully requires generating search patterns contingent on object
geometry.

Figure 6: Left: The Functional Manipulation Benchmark (FMB) [12] and a
modified assembly board with tasks sourced from NIST Assembly Task Board #1
[14] and IROS RGMCS 2020 [13] we used for experimentation. Both environments
have relatively tight tolerances. Right: Example of the rollouts produced by
our method for two tasks: the waypoints from GenCHiP are able to successfully
complete the tasks.

Industrial Manipulation Tasks are adapted from the NIST Assembly Task Board #1
[14] and the IROS RGMC 2020 [13]. These benchmarks are designed to evaluate
proficiency in robotic assembly with an emphasis on small and medium sized
parts and deformable objects. We consider the wire routing and connector
insertion subtasks. Specifically, we study routing (insertion) and unrouting
(removal) of a wire through a plastic channel component and inserting a
waterproof connector plug into a socket. Task environments are visualized in
Fig. 6. Note there are multiple sources of noise across episodes. For example,
in the orientation of the cable within the grasp of the gripper, the tautness
of the cable, and the estimated poses of the waterproof connector.
Additionally, we evaluate on each side of the routing component to verify that
the pattern works across starting positions. An ideal method should result in
an execution policy that is robust to such sources of noise.

####  V-A2 Implementation details

We conduct our robot experiments on a Universal Robotics UR5e robot, which is
a position-controlled robot with ATI Axia80 force-torque sensor at the wrist.
To expose the compliant action space to the language model, we prompt it with
the doc-string for a Cartesian admittance move with parameters on stiffnesses,
impedances, and constraints in reaching a target pose. We add a suffix
describing the details of the given task, optionally including certain
keywords about relevant motion patterns when the environment setup is
ambiguous (e.g., we specify that the peg in the FMB insertion tasks is not
aligned, which requires the policy to search for the opening).

We give the LLM access to methods for computing transformations on reference
poses (i.e., pose_multiply) and for detecting objects in the environment
(i.e., estimate_and_update_pose). To support the latter, we train pose
estimators to do joint object detection and keypoint prediction using a
Faster-RCNN [41] architecture.

####  V-A3 Methods considered

We compare two classes of methods: a scripted baseline policy authored by an
expert and different variants of LLM-generated code using the prompting
strategies and control APIs outlined in Sec. IV-A and Sec. IV-B. For LLM-
generated code, we compare with and without code examples. For zero-shot
settings, the prompt includes includes Task Descriptions (Fig. 3a), Control
API Descriptions (Fig. 3b), and Hints (Fig. 3c). In few-shot settings,
Examples (Fig. 3d) are added to the prompt.

Scripted [Baseline]. We compare against a scripted pattern search insertion
move that is tuned by an expert on a single task setting. This baseline
reflects an alternative to our approach where a single skill is added to our
control library, but is not able to be tuned by an expert across different
task generalizations. On FMB tasks, we adapt a pattern search insertion skill
for peg insertion. The scripted move implements fixed get-in-contact, pattern
search, and insertion phases, with durations, motion patterns, and force
thresholds set by an expert on the circle setting.

Code-as-Policies (CaP) [1] [Baseline]. We compare with a baseline approach
akin to the prior work [1, 2] that uses the point-to-point action space for
performing robot manipulation tasks, i.e. to directly command the robot to
move to Cartesian target poses (Fig. 4).

GenCHiP, Fixed Compliance (FC). For each task, we compare against a baseline
where we do not expose the stiffness and impedance targets or the force
constraints to the LLM planner, but instead, use predefined compliance
parameters. This ablates the importance of force constraints in completing the
task, making the action space similar to prior work [1], but with compliant
motions. Concretely, we provide a modified prompt and access to a wrapper
around the Cartesian admittance move that provides fixed stiffness and
impedance targets and a fixed translation error constraint.

GenCHiP (Few-Shot). We expose force constraints to the language model and add
examples of calls to our control API, which includes conditional compliant
moves. This is similar to the GenCHiP, Fixed Compliance baselines, but all of
the force constraints and termination conditions are exposed to the language
model (Fig. 5). In these experiments, each “shot” is an example subcommand
that shows how to call the API. In the IROS RGMCS tasks, we include 3 example
subcommands: moving down until contact, moving up until a snag, and moving
right until a snag is detected.

GenCHiP (Zero-Shot). We follow the same approach of exposing force constraints
to the language model as GenCHiP (Few-Shot, Fig. 5), but do not include any
examples that call the control APIs. This is the most difficult generalization
setting because every command is an unseen command.

####  V-A4 Evaluation Protocol

Similar to [42], we take the best completion out of 5 calls to the underlying
language model and run 10 evaluations. All of our experiments use the OpenAI
ChatCompletions API [43] with the gpt-4-0613 endpoint and a temperature of
0.0. To make comparisons between different action spaces as fair as possible,
we take the most successful code generated from our method and overwrite the
control API to implement the relevant action space. For each environment, we
tune the insertion reference pose that appears in the prompt. Concretely, this
is the reference pose used in to make contact in the admittance move. This
hyperparameter is essential on the Point-to-Point baseline because insertion
reference poses that are too deep cause a fault.

| Circle | Star | Half-Pipe
---|---|---|---
Scripted | 100% | 10% | 0%
CaP [1] (Zero-Shot) | 70% | 0% | 0%
GenCHiP, FC (Zero-Shot) | 100% | 70% | 30%
GenCHiP (Zero-Shot) | 100% | 80% | 50%
TABLE I: Functional Manipulation Benchmark. [12] Compared to a scripted
policy, GenCHiP is better able to generalize an insertion pattern across
object geometries, including the challenging half-pipe task. We show average
performance across 10 evaluations.

####  V-A5 Results

Functional Manipulation Benchmark. Our first evaluation studies how well
different approaches generalize across unseen task settings, in particular,
the ability to modify insertion search patterns based on different object
geometries. Results on FMB are listed in Tab. I.

We find that GenCHiP outperforms the performance of other methods across
different peg shapes. The baseline scripted policy is successful on the star
only when the points are already in close-enough alignment with the hole and
fails on the half-pipe shape, which is the most difficult to align because
there is only one valid orientation for a successful insertion. In contrast,
our method is successful on half-pipe 50% of the time. Upon further analysis,
we notice that this is because GenCHiP generates a successful policy in only
one direction of rotation (i.e., 100% successful for one rotation and 0%
successful for the other randomized rotation).

When we inspect the code output from the LLM, we find that it generates
intuitive waypoints for the search that correspond to the object specified in
the prompt. For example, for the half-pipe, the output waypoints oscillate
between $0$ and $\frac{\pi}{2}$ while the star shape policy goes through
multiples of $\frac{\pi}{4}$. We also find that the zero-shot prompting setup
is sufficient for this application. Out of the box, the language model is able
to parameterize the conditional compliant move.

| Cable | Cable | Connector | Connector
---|---|---|---|---
| Unroute | Route | Insertion | Insertion
|  |  |  | (Perception)
CaP [1] (Few-Shot) | 40% | 0% | 0% | 0%
GenCHiP, FC (Few-Shot) | 80% | 30% | 20% | 0%
GenCHiP (Zero-Shot) | 60% | 0% | 0% | 0%
GenCHiP (Few-Shot) | 90% | 100% | 90% | 60%
TABLE II: RGMC [13] and NTB [14] Assembly Tasks. On challenging manufacturing
tasks, it’s critical to have compliance and conditional terminations in the
action space. GenCHiP outperforms other LLM-based code-generation approaches
for this reason.

Industrial Manipulation Tasks. To validate our method on more realistic force-
based manipulation environments, we turn to the IROS RGMC [13] and NIST
Assembly [14] experiments, which are arguably more directly targeted towards
force-based manipulation than the peg insertion task. Indeed, we find it
difficult to design an analogous scripted policy baseline for cable
(un-)routing that would perform well across both tasks, which is why we omit
it in this experiment. From the results in Table II, we observe that GenCHiP
(Few-Shot) again consistently outperforms the baselines. Fixed Compliance
(Few-Shot) is second-best, while Code-as-Policies [1] (Few-Shot) performs
worst, failing to complete the routing task even a single time. The Zero-Shot
version of our method also fails for these tasks, and for two reasons: (1) the
LLM tends to generate while loops that are incompatible with the way the API
is structured and (2) the program is successful within a narrow range of force
values that are difficult to infer without more information from examples.

In the connector insertion task, we consider two cases: with and without fixed
object placement. In the absence of fixed object placement, we estimate the
plug and socket reference poses using pose estimators, which can reduce their
accuracy by a small margin ($\epsilon\leq 4\mathrm{mm}$). Our method succeeded
in maintaining a reasonable success rate in this case as well.

Figure 7: We classify error types across three levels of hints. The severity
of errors decreases from left to right. After combining all hints, the most
common error is correct specification of force values.

###  V-B Ablating prompt hints

In the experiments underlying Sec. V-A5, we observed that incorporating
additional hints is critical in eliciting relevant motion patterns. To
understand this phenomenon in depth, we plot the distribution over error types
for increasing levels of hints on the cable un-routing task in the Zero-Shot
setting. We use three hint types: specifically asking for a pattern search,
adding extra rules about accessing undefined variables, and asking the model
to translate from pseudo-code, similar to Chain-of-Thought prompting [44].
Fig. 7 plots the distribution of failures for combinations of these hint
types. We perform rejection sampling to estimate the likelihood of each error
type given that the generated code fails. The errors are ordered from left to
right based on the level of intervention required by a human operator to make
the task succeed. For example, if the model outputs runnable code, but only
moves the cable up without any wiggling motion, the operator will likely need
to add new waypoints and tune the termination conditions and force values. The
most common example of invalid syntax or undefined variable accesses that we
observe is in attempts to access the force values directly to construct for
loops over waypoints. Explicitly adding rules against for loops helps
somewhat, but asking the model to translate from pseudocode is the most
helpful in minimizing these types of errors.

##  VI Conclusions

In this work, we study the capability of Large Language Models (LLMs) to
generate policies for a variety of high-precision, contact-rich manipulation
tasks. By allowing LLMs to place constraints on robot impedances and
interaction forces, GenCHiP improves success rates on subtasks derived from
the Functional Manipulation Benchmark (FMB) and NIST Task Boards by 3x and 4x,
respectively, when compared to code generation approaches that don’t allow for
compliance. This is the first work to show that robotic code generation using
language models can yield policies that are successful in completing contact-
rich manipulation tasks.

## References

  * [1] Jacky Liang et al.  “Code as Policies: Language Model Programs for Embodied Control”  In _arXiv preprint arXiv:2209.07753_ , 2022
  * [2] Suvir Mirchandani et al.  “Large Language Models as General Pattern Machines”  In _arXiv preprint_ , 2023
  * [3] Wenlong Huang, Pieter Abbeel, Deepak Pathak and Igor Mordatch  “Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents”  In _arXiv preprint arXiv:2201.07207_ , 2022
  * [4] Sai Vemprala, Rogerio Bonatti, Arthur Bucker and Ashish Kapoor  “Chatgpt for robotics: Design principles and model abilities”  In _arXiv preprint arXiv:2306.17582_ , 2023
  * [5] Yunfan Jiang et al.  “Vima: General robot manipulation with multimodal prompts”  In _arXiv preprint arXiv:2210.03094_ , 2022
  * [6] Mohit Shridhar, Lucas Manuelli and Dieter Fox  “Cliport: What and where pathways for robotic manipulation”  In _Conference on Robot Learning_ , 2022
  * [7] Anthony Brohan et al.  “Rt-1: Robotics transformer for real-world control at scale”  In _arXiv preprint arXiv:2212.06817_ , 2022
  * [8] Anthony Brohan et al.  “Rt-2: Vision-language-action models transfer web knowledge to robotic control”  In _arXiv preprint arXiv:2307.15818_ , 2023
  * [9] Mohit Shridhar, Lucas Manuelli and Dieter Fox  “Perceiver-actor: A multi-task transformer for robotic manipulation”  In _Conference on Robot Learning_ , 2023
  * [10] Suneel Belkhale et al.  “RT-H: Action Hierarchies Using Language”, 2024  arXiv:2403.01823 [cs.RO]
  * [11] Kevin Tracy et al.  “Efficient Online Learning of Contact Force Models for Connector Insertion”  In _arXiv preprint arXiv:2312.09190_ , 2023
  * [12] Jianlan Luo et al.  “FMB: A Functional Manipulation Benchmark for Generalizable Robotic Learning”, 2023
  * [13] Yu Sun, Joseph Falco, Máximo A. Roa and Berk Çalli  “Research Challenges and Progress in Robotic Grasping and Manipulation Competitions”  In _IEEE Robotics and Automation Letters_ 7, 2021, pp. 874–881
  * [14] Kenneth Kimble et al.  “Benchmarking Protocols for Evaluating Small Parts Robotic Assembly Systems”  In _IEEE Robotics and Automation Letters_ 5, 2020, pp. 883–889
  * [15] Wenlong Huang et al.  “Inner Monologue: Embodied Reasoning through Planning with Language Models”  In _arXiv preprint arXiv:2207.05608_ , 2022
  * [16] Kevin Lin et al.  “Text2motion: From natural language instructions to feasible plans”  In _Autonomous Robots_ Springer, 2023
  * [17] Ishika Singh et al.  “ProgPrompt: Generating Situated Robot Task Plans using Large Language Models”  In _2023 IEEE International Conference on Robotics and Automation (ICRA)_ , 2023
  * [18] Jimmy Wu et al.  “TidyBot: Personalized Robot Assistance with Large Language Models”  In _Autonomous Robots_ , 2023
  * [19] Amber Xie, Youngwoon Lee, Pieter Abbeel and Stephen James  “Language-conditioned path planning”  In _Conference on Robot Learning_ , 2023, pp. 3384–3396  PMLR
  * [20] Yu Gai et al.  “Grounded Graph Decoding Improves Compositional Generalization in Question Answering”  In _ArXiv_ abs/2111.03642, 2021
  * [21] Wenlong Huang et al.  “VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models”  In _arXiv preprint arXiv:2307.05973_ , 2023
  * [22] Youngsun Wi et al.  “CALAMARI: Contact-Aware and Language conditioned spatial Action MApping for contact-RIch manipulation”  In _7th Annual Conference on Robot Learning_ , 2023
  * [23] Jianlan Luo et al.  “Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assembly”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019  DOI: 10.1109/ICRA.2019.8793506
  * [24] Tony Z. Zhao et al.  “Offline Meta-Reinforcement Learning for Industrial Insertion”  In _2022 International Conference on Robotics and Automation (ICRA)_ , 2022
  * [25] Yashraj Narang et al.  “Factory: Fast contact for robotic assembly”  In _arXiv preprint arXiv:2205.03532_ , 2022
  * [26] Zhihao Liu et al.  “Robot learning towards smart robotic manufacturing: A review”  In _Robotics and Computer-Integrated Manufacturing_ 77 Elsevier, 2022, pp. 102360
  * [27] Oliver Kroemer, Scott Niekum and George Konidaris  “A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms”  In _J. Mach. Learn. Res._ 22.1 JMLR.org, 2021
  * [28] Markku Suomalainen, Yiannis Karayiannidis and Ville Kyrki  “A survey of robot manipulation in contact”  In _Robotics and Autonomous Systems_ 156 Elsevier, 2022, pp. 104224
  * [29] Íñigo Elguea-Aguinaco et al.  “A review on reinforcement learning for contact-rich robotic manipulation tasks”  In _Robotics and Computer-Integrated Manufacturing_ Elsevier, 2023
  * [30] Andrew S Morgan et al.  “Vision-driven compliant manipulation for reliable, high-precision assembly tasks”  In _arXiv preprint arXiv:2106.14070_ , 2021
  * [31] Todor Davchev et al.  “Residual Learning From Demonstration: Adapting DMPs for Contact-Rich Manipulation”  In _IEEE Robotics and Automation Letters_ , 2022
  * [32] Toki Migimatsu, Wenzhao Lian, Jeannette Bohg and Stefan Schaal  “Symbolic State Estimation with Predicates for Contact-Rich Manipulation Tasks”  In _2022 International Conference on Robotics and Automation (ICRA)_ , 2022
  * [33] Cheng Chi et al.  “Diffusion Policy: Visuomotor Policy Learning via Action Diffusion”  In _Proceedings of Robotics: Science and Systems (RSS)_ , 2023
  * [34] Gerrit Schoettler et al.  “Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards”  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2019, pp. 5548–5555
  * [35] Samarth Brahmbhatt, Ankur Deka, Andrew Spielberg and Matthias Müller  “Zero-Shot Transfer of Haptics-based Object Insertion Policies”  In _International Conference on Robotics and Automation (ICRA)_ , 2023
  * [36] Cristian C Beltran-Hernandez, Damien Petit, Ixchel G Ramirez-Alpizar and Kensuke Harada  “Variable compliance control for robotic peg-in-hole assembly: A deep-reinforcement-learning approach”  In _Applied Sciences_ 10.19 MDPI, 2020, pp. 6923
  * [37] Fares J Abu-Dakka and Matteo Saveriano  “Variable impedance control and learning—a review”  In _Frontiers in Robotics and AI_ 7 Frontiers Media SA, 2020, pp. 590681
  * [38] Takeshi Kojima et al.  “Large Language Models are Zero-Shot Reasoners”  In _ArXiv_ abs/2205.11916, 2022
  * [39] Roberto Martín-Martín et al.  “Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Tasks”  In _International Conference on Intelligent Robots and Systems (IROS)_ , 2019
  * [40] Jonas Buchli, Freek Stulp, Evangelos Theodorou and Stefan Schaal  “Learning variable impedance control”  In _The International Journal of Robotics Research_ SAGE Publications Sage UK: London, England, 2011
  * [41] Shaoqing Ren, Kaiming He, Ross Girshick and Jian Sun  “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks”  In _Advances in Neural Information Processing Systems (NIPS)_ , 2015
  * [42] Wenhao Yu et al.  “Language to Rewards for Robotic Skill Synthesis”  In _Arxiv preprint arXiv:2306.08647_ , 2023
  * [43] OpenAI  “GPT-4 Technical Report”, 2023
  * [44] Jason Wei et al.  “Chain of Thought Prompting Elicits Reasoning in Large Language Models”  In _ArXiv_ abs/2201.11903, 2022
  * [45] Tsung-Yi Lin et al.  “Feature Pyramid Networks for Object Detection”  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016
  * [46] Shivam Garg, Dimitris Tsipras, Percy Liang and Gregory Valiant  “What Can Transformers Learn In-Context? A Case Study of Simple Function Classes”  In _arXiv preprint_ , 2022
  * [47] Ashish Vaswani et al.  “Attention is All you Need”  In _Advances in Neural Information Processing Systems_ 30 Curran Associates, Inc., 2017  URL: https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf
  * [48] Rinon Gal et al.  “An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion”  In _The Eleventh International Conference on Learning Representations_ , 2023  URL: https://openreview.net/forum?id=NAQvF08TcyG
  * [49] Haotian Liu, Chunyuan Li, Qingyang Wu and Yong Jae Lee  “Visual Instruction Tuning”  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023  URL: https://openreview.net/forum?id=w0H2xGHlkw

## APPENDIX

We discuss implementation details in Sec. VII. We explore the fitness of
language models for high-precision prediction in Sec. VIII. Robustness
evaluations are in Sec. IX. Rollout visualizations are in Sec. X and example
prompts are in Sec. XI.

##  VII Implementation Details

###  VII-A Pose estimators

We use a Faster-RCNN [41] model that does joint object detection and keypoint
prediction with a ResNet-50 Feature Pyramid Network (FPN) [45] backbone. We
train with a bounding box and keypoint regression loss mixture.

###  VII-B Prompt Ablation Details

To generate Fig. 7, we take the first 15 prompts that generate an error
(skipping any successful prompts) and classify them based on failure type with
a human operator. Table II uses the best of five samples from the complete
prompt.

###  VII-C Compliant action space parameterizations

It is worth noting that different robotic platforms offer different interfaces
for control. Torque controlled robots, such as Franka-Emika Panda and Kuka LBR
iiwa robots, readily lend themselves to the impedance control law described in
Equation 1. On the other hand, position controlled robots, such as the UR5e,
require a target position. This will affect how the compliant action space is
parameterized.

To achieve soft interaction with the environment, we can implement an
admittance controller that makes use of a force-torque sensor attached to our
robot end-effector:

$\displaystyle\ddot{x}=M^{-1}(K_{p}(x_{d}-x)+K_{d}(\dot{x}_{d}-\dot{x})+(f_{d}-f_{s}))$

where $f_{d}$ is a desired force value, and $f_{s}$ is the current sensed
force value. The use of admittance control enables us to parameterize our
action space as $t_{i}=([\mathbf{x}_{target}]_{i},\sigma_{i})$, where
$[\mathbf{x}_{target}]_{i}$ denote target pose and $\sigma_{i}$ denote the
stiffness vector.

Even though the admittance controller differs from the impedance controller
described in Equation 1, the parameters, namely $K_{p}$, $K_{d}$, and $M$,
play a conceptually similar role to their corresponding variables in the
impedance control law, which implies that our method can generalize across
different robotic platforms.

##  VIII Probing language model fitness for precision tasks

###  VIII-A Reasoning Over Continuous Spaces

(a)

(b)

Figure 8: (a) Reconstruction error across different floating point number
resolutions and formatting styles. (b) Reconstructions of exemplary sinusoidal
sequences for different floating point number resolutions.

The goal of the main paper is to demonstrate that language models trained on
vast, offline, internet datasets can still be useful in generalizing robotic
policies. In Sec. V-A5, we show that this is because LLMs encode world
knowledge about contact forces and information about object geometry that can
be translated into motion patterns, such as the pattern search examples in
Tab. I. One reason this question is so pertinent is because of recent interest
in amortizing the cost of robot policy learning by adapting language models to
predict action tokens [8]. Code prediction is one way of using language models
out of the box to yield generalizable robotic policies. However, for language
models to be aqequate at action prediction for high-precision tasks, they also
need to be able to extrapolate floating point sequences with high precision
because contact-rich settings require millimeter-level precision.

While prior work demonstrates that LLMs are capable of serving as general
pattern machines [2], they have not been shown to be able to reason over
floating point numbers specifically. In this section, we thus examine two
arithmetic tasks that we believe to be precursors to reasoning successfully in
continuous action spaces: (1) a linear regression task, and (2) a continuous
sequence extrapolation task. In both cases, and in contrast to prior work [2],
we let the LLM operate on floating point numbers directly.

####  VIII-A1 Background

Arithmetic reasoning and pattern extrapolation abilities of LLMs [2]
demonstrated that models trained on internet-scale data are capable of doing
general spatial and sequential reasoning tasks. Similar to our work, their
analysis is inspired by robot control-relevant reasoning problems. However,
unlike them, we produce a proof of concept on real hardware and on a
challenging contact rich manipulation task. Our arithmetic reasoning tasks are
inspired by work from [46], which shows that the Transformers [47] can learn
simple function classes in-context after training on regression problems.
Unlike them, we show that this capability emerges in models trained only on
text generation.

####  VIII-A2 Results

Zero-Shot Regression In-Context: In Fig. 8(a), we show the result of prompting
a model with $x,y=f(x)$ from a 2-dimensional linear function $f$. We quantify
the ability of the model to implicitly learn the linear function by measuring
the $\hat{y}$ output by the model for a given $x$, and plotting the resulting
error for different floating point number resolutions. The x-axis denotes the
number of x, y pairs provided to the LLM and the y-axis is the average
prediction error of $\hat{y}$ for a given value $x$. We compare two different
formatting strategies. In the first, the $x,y$ pairs are written out as
f(x)=y. The second formatting strategy adds spaces between each digit so that
each digit is treated as a separate token. For example, f(1.393)=4.107 is
formatted as f(1.3 9 3)=4.1 0 7. As can be seen in Fig. 8(a), space-formating
with GPT-4 is able to successfully regress to the targets within an error of
$0.01$ when using at least $3$ decimal places.

Sequence Extrapolation: In Fig. 8(b), we illustrate the ability of an LLM to
extrapolate sequences of two-dimensional series of numbers (concretely,
sinusoids of different frequencies), as this might transfer to path-following
behaviors required in the robotic context. To generate these plots, we feed
samples of the sinusoidal function up to a certain value and then auto-
regressively sample from the LLM. The sampled points are shown in orange. For
visualization purposes, the true function is shown in blue. Qualitatively, we
see in the figure that the LLM is indeed capable of extrapolating the chosen
sinusoids. Unlike [2], we find that this ability also works for floating point
numbers. To realize these results, we modify the tokenization strategy by
placing a space in between each digit. The extrapolation ability begins to
break down as the precision and frequency of the sinusoid increase, which can
be seen on the right-most side of Fig. 8(b).

The takeaway from this set of experiments is that LLMs trained entirely on
offline language data can indeed achieve reasonable performance on arithmetic
tasks on continuous spaces. This result shows that large language models are
capable of performing least squares up to a small degree of error with in-
context learning.

###  VIII-B Reasoning over spatial patterns

To protoype alternative ways of providing the language model with perceptual
information for contact rich tasks, we provide schematics of an assembly board
in natural language. In Sec. V-A1, we allow the language model to reason over
object poses via calls to estimate_and_update_pose. Alternatively, we could
pass tokenized versions of the scene into the language model [48, 49]. We
explore this by prompting with spatial patterns.  ⬇ """ Here is the board
layout: B ccc c cscsc c ccc 1 x translations move left to right and y
translations moves bottom to top. The distance between screws is .03M. """
Spatial Patterns

Spatial patterns are symbolic summaries of a given scene and act as a
character-based representation of what is visible to the agent. An example of
a symbolic representation of an IROS RGMCS [13] board is summarized in Figure
VIII-B. The c symbol refers to the path of the cable, S refers to a screw, and
B refers to a plastic channel component through which the cable is routed.
These would also be defined in the prompt. Although these are specified by the
prompt designer in our examples, they could also be generated by a vision-
language model or other perception APIs [22].

Figure 9: We prototype an alternative way of passing perceptual information
into the language model on the cable unraveling task from IROS RGMC [13]. This
is a typical rollout.

In addition to adding the spatial pattern to the prompt, we append the
following task command: Explain how to unravel the cable from the screws and
then generate the code to unravel the cable if the robot is currently holding
the cable at position 1.Using GenCHiP, the best code generation is successful
in removing the cable from the first screw in 8 out of 10 evaluation runs but
only successful in removing the cable from both screws in 2 out of 10 cases.
We expect that these numbers could be further improved by allowing for
regrasps or by implementing more advanced and iterative scene tokenization
strategies, which we believe is a strong area for future work.

Figure 10: Example rollouts on all tasks.

##  IX Robustness

###  IX-A Cable un-routing

We study the robustness to initial position on the cable unrouting task. The
starting position on this task is parameterized by a script that localizes and
picks up the cable, which is assumed to be threaded through one of the RGMC
plastic components. To make the task challenging, the gripper starting
position is offset in the positive or negative x direction, which corresponds
to the left and right side of the tunnel. In Tab. III we show that GenCHiP is
able to successfully un-route the cable regardless of the displacement
direction.

| Left-side tunnel | Right-side tunnel
---|---|---
CaP [1] (Few-Shot) | 60% | 20%
GenCHiP, FC (Few-Shot) | 60% | 100%
GenCHiP (Few-Shot) | 100% | 80%
TABLE III: Cable unrouting [13] from different starting positions.

##  X Rollout visualization

We visualize rollouts from our method in Fig. 10.

##  XI Prompt and Output Code Example

In this section, we provide a complete prompt example for the cable un-routing
task (Listing 1) and the GPT-4-generated output (Listing 2). Importantly, this
example demonstrates how that the language model is capable of generating
novel combinations of moves and constraint conditions. In the prompt, there is
no constraint on right-ward snags for poses moving to the left in the prompt,
but the language model is able to synthesize these constraints for the task.
Please see our paper website for the remaining prompts.

Example 1: Complete prompt example for the cable un-routing task

⬇

prompt = """

You’re a robot holding a cable that’s threaded through a tunnel with a small
opening at the top.

You need to unroute the cable by removing it from the tunnel.

You will have access to the following methods, which are imported directly:

- cartesian_admittance_move: This moves the robot to a target_pose until a termination condition is reached.

Args:

max_cartesian_stiffness:

The maximum allowed stiffness along each cartesian dof (6d), expressed in

the robot base frame.

target_impedance:

(0,1] 6d-vector specifying the target impedance along each cartesian dof.

target_pose:

Target pose for the robot flange frame in the base frame.

termination_condition:

Termination condition.

virtual_cartesian_inertia:

The diagonal representation of the desired virtual Cartesian inertia

matrix, expressed in the robot base frame [kg, kg m^2]

execution_timeout_seconds:

Timeout for execution. Defaults to 30s if not specified

Default value: 10.0

tare_ft_sensor: False when in contact, True otherwise.

- types_pb2.Comparison: this specifies the termination condition above. It can’t be accessed directly, it can only be passed as an argument to the cartesian_admittance_move method.

Args:

operation: types_pb2.Comparison type. One of GREATER_THAN_OR_EQUAL,
LESS_THAN_OR_EQUAL, APPROX_EQUAL, or APPROX_NOT_EQUAL.

state_variable_name: Variable where condition is applied. One of
policy.status.{x_force, y_force, z_force, translation_error, rotation_error}.

double_value: [-0.4, 0.4] value of state variable on which operation is
applied

- types_pb2.Condition: Can’t be accessed directly, can only be passed as an argument to the cartesian_admittance_move method.

Args:

comparison: types_pb2.Comparison type.

- types_pb2.ConjunctionCondition: Wrapper to compose multiple conditions together.

Args:

operation: types_pb2.ConjunctionCondition type. One of ALL_OF or ANY_OF .

conditions: A list of Conditions over which the operation is applied

- pose3_rpy: returns a pose object that specifies a pose. Note: Poses can be composed using the multiply method. E.g., pose1.multiply(pose2) applies pose2 on top of pose1.

Args:

translation: x, y, z translation.

rotation: rotation along x, y, and z.

- move_gripper: opens and closes the gripper.

Args:

percent_open: Controls how open gripper is. 0 is fully closed, 100 is fully
open.

- grasp_cable: grab the free edge of the routed cable

You will also have access to the following fields:

- pose(number): the robot pose just above number. You can build more poses by using this as a reference with the multiply method.

"""

⬇

"""

Args:

number: the pose number for the layout below.

- grasp(pose): go to grasp pose and grasp the cable.

Args:

pose: the robot pose above the grasp position

Rules:

- Don’t define any new methods

- Don’t call any undefined methods

- Don’t add any if statements or while loops

Here are some examples using the cartesian_admittance_move method while
holding the cable:

‘‘‘python

position_condition = types_pb2.Comparison(

operation=types_pb2.Comparison.LESS_THAN_OR_EQUAL,

state_variable_name="policy.status.translation_error",

double_value=0.001,

)

# move down until contact is reached

upward_force_threshold = 0.4

contact_condition = types_pb2.Condition(

comparison=types_pb2.Comparison(

operation=types_pb2.Comparison.GREATER_THAN_OR_EQUAL,

state_variable_name="policy.status.z_force",

double_value=upward_force_threshold,

)

)

down_pose = pose(1).multiply(pose3_rpy(translation=[0, 0, -.01]))

cartesian_admittance_move(

target_pose=down_pose,

termination_condition=termination_condition=types_pb2.ConjunctionCondition(

operation=types_pb2.ConjunctionCondition.ANY_OF,

conditions=[contact_condition, position_condition]

))

# move up unless snag is detected

downward_force_threshold = -0.4

snag_condition = types_pb2.Condition(

comparison=types_pb2.Comparison(

operation=types_pb2.Comparison.LESS_THAN_OR_EQUAL,

state_variable_name="policy.status.z_force",

double_value=downward_force_threshold,

)

)

up_pose = pose(1).multiply(pose3_rpy(translation=[0, 0, .01]))

cartesian_admittance_move(

target_pose=up_pose,

termination_condition=termination_condition=types_pb2.ConjunctionCondition(

operation=types_pb2.ConjunctionCondition.ANY_OF,

conditions=[snag_condition, position_condition]

))

# move the cable to the right unless snag is detected

leftward_force_threshold = 0.4

leftwardsnag_condition = types_pb2.Condition(

comparison=types_pb2.Comparison(

operation=types_pb2.Comparison.GREATER_THAN_OR_EQUAL,

state_variable_name="policy.status.x_force",

double_value=leftward_force_threshold,

)

)

right_pose = pose(1).multiply(pose3_rpy(translation=[-0.01, 0, 0]))

cartesian_admittance_move(

target_pose=up_pose,

termination_condition=termination_condition=types_pb2.ConjunctionCondition(

operation=types_pb2.ConjunctionCondition.ANY_OF,

conditions=[leftwardsnag_condition, position_condition],

))

‘‘‘

Currently the robot is holding the cable at pose(1), but the cable is stuck in
a tunnel with a small opening at the top. Describe a strategy for unrouting
the cable using force information to search for the opening and then write a
program using the APIs above to unroute the cable and lift it out of the
tunnel."""

messages = [{"role": "user", "content":prompt}]

response = openai.ChatCompletion.create(

model="gpt-4",

messages = messages,

temperature=0.0,

max_tokens=512*3,

frequency_penalty=0.0

)

print(response["choices"][0]["message"]["content"])

⬇

Strategy:

1. First, we will move the cable upwards until we detect a snag. This will help us to find the top of the tunnel.

2. Once we detect a snag, we will move the cable to the right until we detect another snag. This will help us to find the right wall of the tunnel.

3. After finding the right wall, we will move the cable to the left until we no longer detect a snag. This will help us to find the opening of the tunnel.

4. Finally, we will move the cable upwards again to lift it out of the tunnel.

Program:

‘‘‘python

# Step 1: Move up until snag is detected

downward_force_threshold = -0.4

snag_condition = types_pb2.Condition(

comparison=types_pb2.Comparison(

operation=types_pb2.Comparison.LESS_THAN_OR_EQUAL,

state_variable_name="policy.status.z_force",

double_value=downward_force_threshold,

)

)

up_pose = pose(1).multiply(pose3_rpy(translation=[0, 0, .01]))

cartesian_admittance_move(

target_pose=up_pose,

termination_condition=types_pb2.ConjunctionCondition(

operation=types_pb2.ConjunctionCondition.ANY_OF,

conditions=[snag_condition, position_condition]

))

# Step 2: Move right until snag is detected

leftward_force_threshold = 0.4

leftwardsnag_condition = types_pb2.Condition(

comparison=types_pb2.Comparison(

operation=types_pb2.Comparison.GREATER_THAN_OR_EQUAL,

state_variable_name="policy.status.x_force",

double_value=leftward_force_threshold,

)

)

right_pose = pose(1).multiply(pose3_rpy(translation=[-0.01, 0, 0]))

cartesian_admittance_move(

target_pose=right_pose,

termination_condition=types_pb2.ConjunctionCondition(

operation=types_pb2.ConjunctionCondition.ANY_OF,

conditions=[leftwardsnag_condition, position_condition],

))

# Step 3: Move left until no snag is detected

rightward_force_threshold = -0.4

rightwardsnag_condition = types_pb2.Condition(

comparison=types_pb2.Comparison(

operation=types_pb2.Comparison.LESS_THAN_OR_EQUAL,

state_variable_name="policy.status.x_force",

double_value=rightward_force_threshold,

)

)

left_pose = pose(1).multiply(pose3_rpy(translation=[0.01, 0, 0]))

cartesian_admittance_move(

target_pose=left_pose,

termination_condition=types_pb2.ConjunctionCondition(

operation=types_pb2.ConjunctionCondition.ANY_OF,

conditions=[rightwardsnag_condition, position_condition],

))

# Step 4: Move up to lift the cable out of the tunnel

cartesian_admittance_move(

target_pose=up_pose,

termination_condition=types_pb2.ConjunctionCondition(

operation=types_pb2.ConjunctionCondition.ANY_OF,

conditions=[snag_condition, position_condition]

))

‘‘‘

Example 2: GPT-4-generated output for the cable-unrouting task