# A Survey on Robotic Manipulation of Deformable Objects: Recent Advances,
Open Challenges and New Frontiers

Feida Gu  Yanmin Zhou  Zhipeng Wang  Shuo Jiang  Bin He Research supported by
the National Natural Science Foundation of China under Grant 61825303, Grant
51975415, Grant 62088101; in part by the Shanghai Municipal Science and
Technology Major Project under Grant 2021SHZDZX0100; in part by the Shanghai
Municipal Commission of Science and Technology Project under Grant
22ZR1467100.(Corresponding author: Zhipeng Wang.)Feida Gu is with Shanghai
Research Institute for Intelligent Autonomous Systems, Tongji University,
Shanghai 201804, China, with National Key Laboratory of Autonomous Intelligent
Unmanned Systems, Shanghai 201210, China, and also with the Frontiers Science
Center for Intelligent Autonomous Systems, Shanghai 200120, China (e-mail:
feidagu@tongji.edu.cn).Yanmin Zhou, Zhipeng Wang, Shuo Jiang, Bin He are with
the Department of Control Science and Engineering, Tongji University, Shanghai
201804, China, with National Key Laboratory of Autonomous Intelligent Unmanned
Systems, Shanghai 201210, China, and also with the Frontiers Science Center
for Intelligent Autonomous Systems, Shanghai 200120, China (e-mail:
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>hebin@tongji.edu.cn).

###### Abstract

Deformable object manipulation (DOM) for robots has a wide range of
applications in various fields such as industrial, service and health care
sectors. However, compared to manipulation of rigid objects, DOM poses
significant challenges for robotic perception, modeling and manipulation, due
to the infinite dimensionality of the state space of deformable objects (DOs)
and the complexity of their dynamics. The development of computer graphics and
machine learning has enabled novel techniques for DOM. These techniques, based
on data-driven paradigms, can address some of the challenges that analytical
approaches of DOM face. However, some existing reviews do not include all
aspects of DOM, and some previous reviews do not summarize data-driven
approaches adequately. In this article, we survey more than 150 relevant
studies (data-driven approaches mainly) and summarize recent advances, open
challenges, and new frontiers for aspects of perception, modeling and
manipulation for DOs. Particularly, we summarize initial progress made by
Large Language Models (LLMs) in robotic manipulation, and indicates some
valuable directions for further research. We believe that integrating data-
driven approaches and analytical approaches can provide viable solutions to
open challenges of DOM.

The manipulation of deformable objects is very common in industry or daily
life. The applications of deformable object manipulation in industry can
create enormous economic benefits, and the applications in daily life can
improve quality of life, especially for people with disabilities and the
elderly. Robotic manipulation tasks such as food handling, assistive dressing
or clothing folding require open problems of perception, modeling and
manipulation of deformable objects to be solved. Recent advances in machine
learning have addressed some of the limitations of traditional approaches.
Combining analytical and data-driven approaches holds the potential to bring
the manipulation capabilities of robots close to that of humans. This paper
systematically reviews various approaches to robotic manipulation of
deformable objects, providing a quick reference for researchers and
practitioners in different fields. Additionally, the paper provides insights
into open research areas in the field and points out promising future research
such as LLMs for robotic manipulation.

Robotic manipulation, deformable objects, machine learning, perception,
modeling

##  1 Introduction

The ability to manipulate objects is an essential feature of robots, as the
definition of a robot requires it to exert influence on its surroundings. Over
the past decades, research on robot manipulation has progressed considerably,
aiming to utilize cost-effective robotic arms and end-effectors to directly
interact with the world and accomplish their objectives [1].

Object rigidity is a common assumption in robot manipulation [2]. However,
this assumption does not hold for all objects and scenarios, as any object can
deform under the application of external forces. The rigidity assumption is
only valid when the object deformation is negligible for the manipulation
task. However, many objects present non-negligible deformations during robotic
manipulation tasks.

Enabling robots to manipulate DOs opens up a wide range of potential
applications in various fields such as industrial manufacturing, medical
surgery, food processing and elderly care. These applications have the
potential to generate substantial economic benefits. Fig. 1. gives some
application examples in various sectors. In manufacturing plants, robots can
manipulate DOs to alleviate physical burden on workers [3, 4, 5, 6]. In the
medical field, robots can assist in various surgeries [7, 8, 9, 10]. The use
of robots in the food processing can reduce labor costs [11, 12, 13, 14].
Robots can assist the elderly and the disabled in their daily living
activities [15, 16, 17, 18].

Figure 1: Applications involving DOM. (a) Manufacturing industry [3, 4, 5, 6]
(b) Medical surgery [7, 8, 9, 10] (c) Food processing [11, 12, 13, 14] (d)
Daily living activities [15, 16, 17, 18]

Despite its importance, DOM has historically been less studied than
manipulation of rigid objects since its complexity of perception, modeling and
manipulation. When manipulating DOs, the planning strategy of a robotic system
(shown in Fig. 2.) should not only focus on the motion of a robot, but also
need to consider the DOs being manipulated. However, DOs have infinite
dimensionality and complex dynamics, making DOM extremely challenging. Recent
developments in computer graphics and machine learning provide useful
techniques for DOM. Data-driven methods can address some of the limitations of
traditional DOM approaches [19, 20].

Figure 2: A typical robotic system for handling DOs including robotic
hardware, perception hardware, robotic hands, tools used for DOM, algorithms
for various functions, etc.

The review surveyed by Kadi and Terzic [21] only focuses on manipulation of
clothing using data-driven approaches. Some past reviews [22, 23, 24] focus on
modeling DOs without considering perception and manipulation. The previous
reviews [25, 2] are comprehensive, but they mainly focus on analytical
approaches. The review surveyed by Yin et al. [20] concludes a comprehensive
review which includes both analytical and data-driven approaches. Our survey
mainly focuses on data-driven approaches for perception, modeling and
manipulation for DOs, although analytical approaches are also concisely
reviewed. In particular, progress in LLMs are discussed in our review. Below,
our contribution is concluded specifically. In the perception section, we
summarize the perception module in terms of visual, tactile and multimodal,
and regard multimodal manipulation datasets and tactile simulators as two key
elements for perception of DOM in the future. In the modeling section, we
focus on modeling DOs using graph neural networks (GNNs), which is not
thoroughly discussed in previous reviews. In the manipulation section, more
cutting-edge reinforcement learning (RL) and imitation learning (IL) methods
are reviewed. We also discuss current open challenges and point out future
research directions for perception, modeling and manipulation respectively.
Particularly, we surveyed research about robotic manipulation using LLMs
including task definition for manipulation, planning, reward function design
and uncertainty alignment, which is not mentioned in previous reviews.

The structure of this article is as follows. Section II provides an overview
of techniques for sensing DOs, including visual perception and tactile
perception. In section III, analytical modeling approaches and data-driven
modeling approaches are included. Section IV reviews manipulation methods,
including traditional planning and control methods as well as learning-based
methods. In Section V, we discuss how LLMs can impact DOM. Finally, Section VI
concludes the article.

##  2 Perception

Robots require fast, accurate and multi-modal perception abilities to sense
their surroundings, which is a prerequisite for accomplishing complex
manipulation tasks [26]. The aim of perception is to estimate the state of a
object $X^{\ast}$ by solving an optimization problem (Eq. 1) given the
observation O and the object representation $R(\cdot)$,

$X^{\ast}=\mathop{\arg\min}\limits_{X}\||O-R(X)||$ (1)

where $X$ is objects’ states.

However, finding accurate and efficient representation of a deformable object
(DO) is an open challenge, and solutions are application-specific.
Representing DOs as particles is one of commonly used approaches [27]. In the
perception section of this review, examples use particles to represent DOs,
unless otherwise specified.

Since DOs have an infinite-dimensional state space, the perception of
deformations becomes highly challenging. Moreover, occlusion and noise are
common in unstructured environments, which requires high robustness of state
estimation for DOs.

Visual perception and tactile perception are reviewed in this section. Table.
1. summarizes the primary advantages, limitations and representative
literature.

###  2.1 Visual perception

A complete vision-based pipeline for estimating the state of a DO consists of
three steps including segmentation, detection and tracking. Segmentation aims
to separate the DO from the background in an image. Detection refers to the
process of estimating the DO’s state from a single image. Tracking is tracking
the DO’s state across multiple frames. Fig. 3. shows a real-time tracking of
DOs.

Figure 3: Real-time tracking of DOs (Deformable linear objects (DLOs) as an
example [28]). The first row shows the overall point cloud. The second row
shows the segmented point cloud. The third row shows the real-time tracking
results.

####  2.1.1 Segmentation

Segmenting DO regions in an image is the first step in the whole processing
stream. It is aimed at isolating the DOs from their surroundings. A major
challenge of this task is to segment DO instances reliably, accurately and
quickly without prior knowledge of the background features and the object
count in a scene.

The identification of DOs in simple settings is a solved problem [29, 30]. In
case of complex backgrounds, some advanced DO-specific approaches based on
datasets have been proposed [31, 32, 33]. A major obstacle for applying data-
driven methods to DO segmentation is the lack of large-scale and high-quality
datasets that are publicly available and annotated with labels. To tackle the
problem, several methods [34, 35] that concentrate on synthetic data
generation processes have been proposed. With data generation processes, an
advanced method called FASTDLO has been introduced for segmenting DLOs in an
image [36]. This technique employs a deep convolutional neural network to
perform background segmentation, which produces a binary mask that separates
DLOs from the background pixels in an image. FASTDLO has achieved processing
speeds of more than 20 FPS in case of complex backgrounds. In addition to DLO
segmentation, segmenting fabrics is also studied. Qian et al. [37] proposed a
neural network that can segment the fabric regions that are suitable for
robotic grasping based on the depth image of the scene. The network assigns a
semantic label to each pixel, indicating whether it belongs to an outer edge,
an inner edge, a corner or none of these categories of the fabric. These
labels are then used to estimate the grasping point and direction based on the
uncertainty measurement for each outer edge point. The proposed method
demonstrated superior performance over the baseline methods on a real robot
system that can grasp the edges and corners of cloth in various crumpled
configurations. Other studies [38, 39] also focus on obtaining high-quality
pixel-level masks of DOs using data-driven methods.

####  2.1.2 Detection

Detection is the process of estimating the positions of the particles which
represent DOs in a single frame, using preprocessed sensory data as input.

Estimating the state of DOs from a single-frame point cloud with occlusion is
a challenging task. Existing methods, such as the one proposed by Wnun et al.
[40], can extract a skeleton curve and 3-D joint positions from the point
cloud, but they suffer from occlusion and DOs’ diversity. To address these
challenges, Lv et al. [41] proposed a novel dual-branch network architecture
that can effectively capture both global and local features of input point
cloud for DO estimation. Specifically, the network comprises a PointNet++
encoder, a regression branch for end-to-end estimation and a voting branch for
point-to-point refinement. The regression branch concentrates on the global
shape of DOs, while the voting branch attends to the local details of DOs. The
network also incorporates a fusion module that integrates the outputs of the
two branches to produce a smooth and precise estimation of DOs’ state.

####  2.1.3 Tracking

Tracking is the process of estimating the deformation of a DO across a
sequence of frames. The objective is to generate the state that is consistent
with the geometry of a DO at each frame .

Accurate tracking is challenging when objects are partially occluded. Recent
advances in computer vision have demonstrated effective methods for DO
reconstruction based on various techniques [42, 43]. With real-time
reconstruction, these methods can dynamically change the tracking model to
avoid the occlusion problem. However, they do not meet the criterion of model
consistency that is essential for visual-servoing algorithms applied to DOM.
DO tracking is also an active research area in surgical robotics. Some studies
[44, 8] have achieved remarkable results in tracking soft tissues in surgical
scenarios, but these methods are domain-specific and may not be applicable to
other types of objects such as ropes or fabrics. In the domain of robotics,
physics simulation is a common technique for tracking DOs that are partially
occluded by other objects. This approach has been adopted by several studies
[45, 46]. More recent approaches [47, 48] have also employed Gaussian Mixture
Modeling (GMM) and Coherent Point Drift (CPD [49]) algorithms to generate
inputs for their physics simulation engine. Nevertheless, the performance of
these algorithms relies on the availability of the physical model of DOs and
environmental geometry which are difficult to obtain in unstructured
scenarios. Using CPD, Chi and Berenson [50] developed a framework for tracking
DOs in RGBD sequences that can handle occlusion without relying on the physics
model of DOs and geometry of the environment. This framework demonstrates
higher accuracy than physics-based CPD methods even under occlusion, while
maintaining a reasonable run-time.

###  2.2 Tactile perception

The advancement of tactile sensing has stimulated a growing interest among
researchers in exploring the applications of tactile technology for DOM.
Vision is a sensory modality that mainly obtains global features such as shape
and color, whereas tactile sensing is a sensory modality that mainly obtains
local features such as friction and texture. Tactile sensing is a valuable
modality for manipulation tasks in occluded environments, as it can provide
local information about physical properties and interactions of objects
involved.

Figure 4: Real-time tactile perception while manipulating DOs (Cable following
using GelSight as an example [51]). The first row shows the gripper grasping a
cable. The second row presents the estimated cable poses. The white ellipse
represents the estimation of the contact region. The red and green lines
denote the first and second principal axes of the contact region, with their
lengths proportional to their corresponding eigenvalues. The third row shows a
top view of the pulled cable while the gripper registers marker displacements,
reflecting the magnitude and direction of the frictional forces.

####  2.2.1 Tactile devices

The BioTac tactile sensor is a remarkable device that can acquire sensory
modalities mimicking the full range of capabilities of human fingertip [52].
It has a rigid core that contains all of the sensory electronics and a
flexible skin made of low-cost silicone, which enhances the durability and
reparability of the design. However, a major limitation of BioTac is that its
output signal is subject to complex signal processing and data fusion,
increasing its computational effort and latency.

Vision-based tactile sensors are a class of sensors that transform tactile
information into visual information by capturing the deformation of a contact
surface with a camera. These sensors offer a distinctive benefit of high
spatial resolution and have been applied to various robotic manipulation
tasks. GelSight, a prominent vision-based tactile sensor, was proposed by Yuan
et al. in 2017 [53]. However, GelSight still has a major drawback of not being
able to emulate human tactile sensations such as temperature, humidity, pain,
etc.

Furthermore, Sundaram et al. [54] developed a tactile glove that enables
passive sensing of various tactile tasks. The glove consists of a knitted
glove made of a piezoresistive film with a sensor array of 548 sensors which
are connected by a network of conductive wire electrodes. The glove can employ
deep convolutional neural networks to process pressure signals for various
tactile tasks such as object recognition, weight estimation or tactile pattern
exploration.

####  2.2.2 DOM applications using tactile sensing

Although tactile sensing is valuable for manipulation especially when vision
is occluded, there are few examples of manipulation tasks that rely solely on
tactile sensing. One such task is cable-following or fabric-following. She et
al. [51] proposed a method to estimate the pose and the friction force of a
cable held by a robot using a tactile sensor based on GelSight technology. The
process is shown in the Fig. 4. Hellman et al. [55] introduced a real-time
tactile perception and decision-making method using BioTac for a tactile-
driven contour-tracking task, specifically, closing a sealed bag. Zheng et al.
[16] proposed a method to capture changes in tactile signals during page
flipping by Biotac. This study establishes the relationship between tactile
signals and a rational page flipping trajectory, enabling a robot to learn to
flip pages rationally. Moreover, Building upon the high-resolution tactile
glove presented in [54], a wide array of interactive tasks involving diverse
objects is performed in the system proposed by Zhang et al. [56]. The tactile
model combines a predictive model and a contrastive learning module to
estimate the 3-D coordinates of the hand and the object from only the touch
data.

###  2.3 Discussion of perception

By touching, humans can simultaneously feel, weigh and manipulate various
objects, as well as infer material properties of objects. This is a difficult
set of tasks for modern robots. In order to broaden the boundaries of robotic
manipulation, multimodal sensory inputs are essential. Vision and touch are
two vital senses for robots to perform dexterous manipulation. Relying solely
on a single sensory modality to monitor the dynamics of DOs is difficult. For
instance, vision can be obstructed, while tactile inputs lack the ability to
capture global information for tasks.

In order to realize DOM with multimodal perception, the development of usable
tactile perception devices is an important prerequisite. Several tactile
sensors have been developed [57, 58] in recent years. Although tactile sensors
have achieved good perception accuracy in reality, simulators for tactile
sensors are still rarely investigated and developed due to the difficulty of
reconstructing elastomer deformations in optical tactile stimulation. This
leads to difficulties in providing a high-fidelity simulation environment for
training robots. Multimodal perception holds great promise in the field of
robot manipulation, and the research of such tactile simulators is the key to
the application of multimodal perception in robot manipulation tasks.

Large-scale and high-quality manipulation datasets are crucial for training
robots that can collaborate and perform complex tasks in diverse and
unstructured environments. These datasets must span multiple sensing
modalities that are well-suited to complex and unstandardized environments.
Using the tactile glove developed in [54], as well as other wearable sensing
devices such as myoelectric sensors and eye-trackers, DelPreto et al. [59]
developed a multimodal manipulation dataset of a kitchen scene (including
DOs). However, this dataset can be further expanded and improved by
incorporating more scenarios or activities, as well as by exploring
alternative or novel wearable sensors or modalities that can overcome the
current limitations of data collection. Producing a dataset of multimodal-
aware robot manipulation tasks is another key to the use of multimodal
perception in robot manipulation applications.

Table 1: Overview of perception approaches of DOs. Categories | Advantages | Limitations | Representative literature
---|---|---|---
Visual perception | Acquire global information |  Not robust under occlusion, Affected by environment easily |  DLOs: Tang and Tomizuka [28] Lv et al. [41],
Cloth-like objects: Chi and Berenson [50] Qian et al. [37],
Play-dough: Shi et al. [60]
Tactile perception | Acquire local information |  DLOs: Hellman et al. [55] She et al. [51],
Tactile information is sparse, Requires contact for acquiring | Page: Zheng et al. [16],
| 3-D objects: Zaidi et al. [61]
Multimodel perception |  Acquire global and local information at the same time |  Hard to represent multimodal sensory inputs | DLOs: Pecyna et al. [62]
Deformable surface: Caccamo et al. [63],
Sponge: Rios et al. [64] anchez et al. [65]

##  3 Modeling

In order to perform DOM, a model that can predict the DO state in the
subsequent time steps based on the current state should be available. Such
models need to reason the dynamics of DOs. Analytical modeling approaches and
data-driven modeling approaches are reviewed in this section. Table. 2.
provides a glance of the main advantages, limitations and representative
literature.

###  3.1 Analytical modeling of DOs

Analytical modeling approaches are under Newton’s second law. In this section,
Mass-spring-damper (MSD) systems, Position-based dynamics (PBD) and Continuum
mechanics are reviewed.

Figure 5: Analytical modeling of DOs (DLOs as an example) (a) A mass-spring-
damper system (b) A position-based dynamics system with several constraints
(c) Continuum mechanics (FEM as an example)

####  3.1.1 Mass-spring-damper systems

As shown in Fig. 5 (a), MSD systems model deformable materials as network
structures. Each particle in the network has a mass attribute, and the
connections between particles are modeled by springs and dampers. For a single
particle $i$, the equation of motion is expressed as

$m_{i}a_{1}=f_{ext}(p_{i})+f_{d}(p_{i})+f_{s}(p_{i})$ (2)

where $m_{i}$ is the mass of particle $i$, $a_{i}$ is the acceleration of
particle $i$, $f_{ext}$ is the external force applied to particle $i$, $f_{d}$
is the spring damping force of particle $i$, $f_{s}$ is the spring force of
particle $i$.

MSD systems provide several advantages for simulating DOs, such as
intuitiveness, ease of implementation and computational efficiency, which
facilitate real-time animations. MSD models are suitable for applications that
require a compromise between physical accuracy and computational speed. They
have been employed to simulate various kinds of DOs, such as DLO simulation
[66] and clothing simulation [67].

One challenge with MSD systems is they are only suited for simulating small
deformations and not complex elastic effects. The system’s behavior does not
necessarily converge to the expected outcome by increasing the mesh resolution
[68]. The system is also sensitive to the network topology and the spring
parameters. Another challenge arises from lacking physical significance of the
spring parameters in relation to the material parameters, which implies that
numerous adjustments are necessary to achieve the desired dynamic
characteristics. A possible solution to this problem is the application of
learning methods and reference schemes [64]. MSD models face another challenge
in their inability to directly represent volume effects. Teschner et al. [69]
addressed this issue by introducing an additional energy equation.

####  3.1.2 Position-based dynamics

PBD shown in Fig. 5 (b). is a particle-based model that incorporates both
internal and collision constraints. The constraints include stretching,
bending, volumetric constraints, collision constraints, etc. The acceleration
of a object are determined by the sum of internal and external forces
according to Newton’s second law of motion. A time integration method is
employed to update the velocities and positions of the object. A set of
constraints is solved by an iterative solver loop which utilizes the Gauss-
Seidel method, until the convergence criterion is met or the maximum number of
iterations is exceeded.

The accelerations of the object are determined by the sum of internal and
external forces, according to Newton’s second law of motion. A time
integration method is employed to update the velocities and positions of the
object. A set of constraints is solved by an iterative solver loop which
utilizes the Gauss-Seidel method, until the convergence criterion is met or
the maximum number of iterations is exceeded.

For a single particle $i$, the equations of motion are expressed as

$v_{i}^{t+1}=v_{i}^{t}+\frac{\Delta
t}{m_{i}}(f_{i}^{ext}-\sum_{j}{k_{j}\nabla_{x_{i}^{t+1}}C_{j}}(x_{1}^{t+1},x_{2}^{t+1},...,x_{n}^{t+1}))$
(3) $x_{i}^{t+1}=x_{i}^{t}+\Delta tv_{i}^{t+1}$ (4)

where $k_{j}$ denotes the stiffness of the $j^{th}$ constraint
$C_{j}(x_{1},x_{2},\ldots,x_{n})=0,j=1,2,\ldots.$

PBD is a technique that allows for rapid and fully controllable simulation
with imporved stability. The method can easily incorporate various constraints
into a system and drive the system by setting boundary conditions. Several
popular physics engines, such as PhysX, Bullet and Havok, have implemented PBD
in their systems to simulate the dynamics of DOs.

One of the main difficulties in PBD is that the stiffness of constraints is
not only determined by the parameters specified by the user, but also by the
time step and the number of iterations used by the solver. As the number of
iterations increases or the time step decreases, the constraints become more
rigid. To address this issue, Macklin et al. [70] proposed a total Lagrange
multiplier method for PBD, which can decouple the constraint solving from
these factors.

####  3.1.3 Continuum mechanics

DOs are often modeled as continuous objects shown in Fig. 5 (c). Displacement,
strain and stress are the key quantities that significantly impact the
understanding of the behaviour characteristics of DOs. The equation of motion
for continuum mechanics is described as

$\rho\ddot{x}=f^{int}+f^{ext}=\nabla\bullet\sigma+f^{ext}$ (5)

where $\rho$ denotes density, and force terms are now body forces applied at a
unit volume. $\sigma$ represents a stress tensor that is a symmetrical matrix.
By applying the gradient operator to the stress tensor, the stress term
captures internal elastic effects.

Continuum materials are suitable for modeling the DOs that require high
precision in simulation, as they can capture deformation effects more
accurately. The model parameters possess a distinct physical significance and
can be effectively applied to diverse scenarios with the aid of flexible
computing frameworks like FEM. However, the computation and implementation of
continuum materials are more complex compared to PBD. Linear FEM finds common
usage in robotics situations where deformations remain within a small range.
However, unless employing an explicit integration scheme or a hardware-
accelerated implementation, nonlinear models typically lack the necessary
speed for achieving real-time performance.

###  3.2 Data-driven modeling of DOs

Data-driven models acquire extensive insights into complex dynamics through
direct analysis of data [71, 72]. Consequently, this method finds utility in
planning and control applications. Jacobian-matrix-based models and GNN-based
models are reviewed in this article.

####  3.2.1 Jacobian-matrix-based model

The state of DOs can be characterized by a set of features that describe their
shape and configuration. Modeling DOs can be regarded as studying the mapping
between the state of the end-effectors and the state of the DOs’ features. A
Jacobian matrix can be used to locally approximate this mapping. It is
described as

$\dot{x}=\ J(x)v$ (6)

where v is the velocity of end-effectors, $\dot{x}$ is the velocity of DOs’
features, $\ J(\cdot)$ is a Jacobian matrix which is fully determined by the
DOs’ current state x.

Figure 6: Architecture of a model to output the estimated Jacobian matrixes
[73]. A Jacobian matrix is fully determined by the DOs’ current state x.

These models have a linear structure and can be computed in real time with a
low data volume. Compared to directly modeling with deep neural networks
(DNNs), modeling approaches based on a Jacobian matrix can make more efficient
use of data.

Some researchers have employed exclusively online techniques to estimate the
Jacobian matrix to build a local linear deformation model of DOs [74, 75, 76,
77, 78, 79]. Due to their online execution, these models can be readily
applied to a wide range of new objects. However, the utilization of a limited
amount of local data in these estimated models compromises their accuracy,
limiting their effectiveness to local configurations. Consequently, this
method can only handle tasks with small and localized deformations. For
globally modeling and control of DOs, Yu et al. [73] developed a method to
efficiently learn the global deformation model of DOs by integrating offline
learning and online adaptation. They employed a radial basis function neural
network (RBFN) to approximate the nonlinear mapping from the current state of
the DO to the current linear deformation model, as illustrated in Fig. 6. The
RBFN was trained offline with randomly sampled data, then transferred to the
online phase as an initial estimate, and finally updated to adapt to the DOs
during real-world manipulation. Their framework achieves global modeling of
larger deformation using the Jacobian-matrix-based approach, although the term
“large” is relative to the existing Jacobian-matrix-based studies.

####  3.2.2 GNN-based model

In addition to Jacobian-matrix-based modeling approaches, global models of DOs
can be approximated with DNNs. DNN-based approaches have a stronger
representation power than Jacobian-matrix-based approaches, which enhances
their accuracy and robustness [80]. Furthermore, they can integrate physics
models and infer how objects interact [27]. These models can cope with very
complex systems and perform well in a broader range of situations, overcoming
(to some extent) the drawbacks of linear models that are constrained by local
conditions. However, such approach requires substantial datasets that may be
unavailable in certain scenarios.

Figure 7: One-step prediction through GNN-based Model (Message-passing GNN
type for manipulating a DLO as an example [81, 82]) (The model is IN, when k
equals 1.) (a) The GNN-based model predicts future states represented as
particles using its learned dynamics model. (b) It constructs a latent graph
from the input state. (c) It performs k rounds of learned message-passing over
the latent graphs. (d) It extracts the dynamics information from the final
latent graph.

Characteristics of DOs are complex, resulting in capturing the behavior of
various DOs is challenging for those generic network structure such as MLP,
CNN and LSTM. Research [83] on GNNs has brought a new structure for modeling
DOs. Battaglia et al. [27] proposed Interaction Network (IN), a general-
purpose, learnable physics engine that enables object- and relation-centered
reasoning about physics. IN represents the particles of an object as nodes of
a graph, and the interactions between pairs of nodes as edges of the graph. IN
uses an object function and a relation function to model objects and their
relations in a combinatorial manner. IN can learn pairwise local interactions
between different parts of a DO. The process of prediction for IN is shown in
Fig. 7.

Assuming two objects and one directed relationship, e.g., a fixed object
connected to a movable mass by a spring, the first object (the sender $o_{1}$)
affects the second (the receiver $o_{2}$) via their interaction. The effect of
this interaction $e_{t+1}$ can be estimated by a relation-centric function
$f_{R}$. The $f_{R}$ takes $o_{1}$, $o_{2}$ and attributes of their
relationship $r$, such as the spring constant, as input. The $f_{O}$ is
adapted to take both $e_{t+1}$ and the receiver’s current state $o_{1,t}$ as
input, allowing the interaction to influence its subsequent state $o_{2,t+1}$.
They are expressed as

$e_{t+1}=f_{R}(o_{1,t},o_{2,t},r)$ (7) $o_{2,t+1}=f_{O}(o_{2,t},e_{t+1})$ (8)

One of the limitations of IN is its failure to propagate the effect of the
interaction throughout the entire object. To address this problem, Some
studies [81, 84, 85] have proposed different methods. Li et al. [81]
introduced a propagation network called PropNet to address the drawback of
IN’s inability to propagate interactions across the entire object by doing
multi-step information transfer. Specifically, the shared information is first
encoded and the propagation steps are reused many times. Propagation networks
need to be repeated many times to deal with long distance dependencies, which
is both inefficient and difficult to train. Therefore, inspired by Mrowca et
al. [86], DPI-Net proposed by Li et al.  [84] adds an additional level to
efficiently propagate long-distance effects between particles. Yang et al.
[85] proposed BiINLSTM, which combines the IN dynamics learning method with a
recursive model (e.g., LSTM), to capture the interaction between neighboring
particles in a DLO. Furthermore, it utilizes the recursive model to propagate
the interaction effect along the entire length of the DLO. Unlike the original
IN approach, the BiINLSTM model enables the transmission of local interactions
to every particle in the DLO within one time step, eliminating the need for
repeated propagation as described in [81]. But this method is still only
verified to work in DLOs.

GNN-based approaches are a powerful way of representing structure information,
but they have some limitations. The expression of concepts like recursion,
control flow, and conditional iteration through graphs can pose challenges
unless specific assumptions are made, such as employing abstract syntax trees.
Programs and more “computer-like” processing can have more expressive power
and flexibility for these concepts, and some researchers think they are an
important part of human cognition (Tenenbaum et al. [87]; Lake et al. [88];
Goodman et al. [89]).

Data-driven approaches rely on simulators to generate data. As a result, the
gap between simulation and real-world scenarios remains a major obstacle for
applying the model to robotic manipulation tasks in the real world. Several
studies [90, 18, 60] have proposed different methods to address the issue of
sim-to-real gaps. Wang et al. [90] employed a GNN to learn a dynamics model
from synthetic data. Then, an online linear residual model is learned to
reduce the gap between simulation and reality. The work proposed by Zhang and
Demiris [18] compares real and simulated pairs of object observations to learn
their physical similarities, which allows the simulator to better adapt to
real-world objects’ physics. The study introduced by shi et al. [60] obtains
data directly from real observations and uses a GNN to learn particle-based
dynamics models to perform deformation operations on Play-Doh. All three
methods mentioned are effective (to some extent) for the problem of the sim-
real gap.

###  3.3 Discussion of modeling

Obtaining the exact deformation model of DOs is a challenging task since they
are difficult to calculate theoretically. Analytical modeling approaches are
not capable of accurately modeling DOs due to their infinite state dimensions
and the difficulty in acquiring DOs’ parameters in the real world. Data-driven
approaches can learn a fairly accurate model from data without understanding
the complex physical dynamics, given the availability of sufficient high-
quality data. The deformation models may differ considerably across different
DOs owing to their diverse lengths, thicknesses, materials, etc. It is
impractical to spend a large amount of time to collect new data for every DOM
task.

The predictive power of both analytical and learned models is limited. These
models are only effective for certain types of tasks. Therefore, researchers
have proposed evaluation methods [91, 92, 93] to assess the reliability of
models during a manipulation task. In situations where the model exhibits
unreliability, replanning or recovery strategies should be considered.

Table 2: Overview of modeling approaches of DOs. Categories | Advantages | Limitations |  Representative literature
---|---|---|---
Analytical approaches
MSD |  Fast, |  |  DLOs: Bergou et al. [94] Lv et al. [95],
Computationally efficient, |  Inaccurate for large deformation, |  Cloth-like objects: Kita et al. [67] Goldenthal et al. [96],
Easy to implement |  Require accurate physical parameters |  Tissue: Kuhnapfel et al. [97] Mollemans et al. [98]
PBD |  Fast, |  |
Fully controlled, | Visual fidelity only, |  Cloth-like objects: Macklin et al. [99] Macklin et al. [70],
Improved stability |  Lacking physical interpretability | Page: Zollhöfer et al. [100]
Continuum mechanics |  |  |  DLOs: Koessler et al. [101] Duenser et al. [102],
| Heavy computational | Cloth-like objects: Cusumano-Towner et al. [103],
Accurate, | requirements, | Muscles: Chen et al. [104],
Physical | Difficult to achieve | Ring-like objects: Yoshida et al. [105],
interpretation | real-time performance | Silicone: Kinio et al. [106]
Data-driven approaches
Jacobian-matrix-based |  |  |
Require a small amount of data, |  Only model small deformation |  DLOs: Lagneau et al. [78] Yu et al. [73],
computationally efficient |  |  3-D objects: Navarro-Alarcón et al. [74] Navarro-Alarcón et al. [75]
GNN-based |  |  |  DLOs: Yang et al. [85] Li et al. [84],
Model complex dynamics accurately, |  Require a large amount of data, |  Cloth-like objects: Li et al. [107] Huang et al. [108],
Flexible |  Lacking physical interpretability |  Dough-like objects: Shi et al. [60] Shi et al. [11]

##  4 Manipulation

The objective of manipulating DOs is to determine the optimal force or motion
at driving points to achieve the given task goal. We firstly review works on
analytical planning and control for DOM. Then, we mainly focus on reviewing
learning methods such as RL and IL. Table. 3. presents a comprehensive
overview of manipulation, and summarizes representative literature.

###  4.1 Planning for DOM

Planning is the process of finding an optimal sequence of configurations for a
robot or an object to achieve a desired goal. Let us consider a task such as
bending a rope into a certain shape. A common method to solve this problem is
to formulate it as an optimization problem

$x_{0:T}^{\ast},u_{0:T-1}^{\ast}=\mathop{\arg\min}\limits_{x_{0:T},u_{0:T-1}}{\mathcal{J}_{u_{0:T-1}}(x_{0:T})}$
(9) $x_{t+1}=SystemDynamics(x_{t},u_{t})$ (10)

where $x_{(}0:T)$ denotes the action of manipulating DOs, and
$\mathcal{J}(\cdot)$ denotes the total cost incurred in executing the planned
trajectory $x_{0:T}$, $u_{0:T-1}$ in the time range $T$.

Figure 8: Planning manipulation of DOs. (DLOs as an example) (a) Action
shooting: the decision variables are in the action and control space (dashed
rectangle). (b) State trajectory exploration: the decision variables are in
the state space (dashed rectangle), and actions are derived from state
trajectories and inverse dynamics.

####  4.1.1 Shooting in the action space

A possible solution to Eq. 9 is to apply a sample action sequence and then
update it based on the computed cost of the predicted trajectories. Fig 8 (a)
illustrates the process of shooting in the action space. In general, the
action space has a much lower dimensionality than the object state space,
which facilitates the search in the action space. Precise dynamics modeling
and efficient backward gradient evaluation are essential for this type of
planning method.

Huang et al. [109] proposed a method to perform shape control manipulation on
DLOs using a dual-arm robotic system by searching actions in the action space.
Shi et al. [60] proposed a method to deform Play-Doh into various letter
shapes with capabilities comparable to humans using such planning method.
Other studies [110, 111, 61] also focus on generating manipulation actions
relying on action-space shooting.

####  4.1.2 Searching state trajectories

One possible approach to plan manipulation actions is to explore in the object
state space, e.g., finding a low-cost path of $x_{t}$, and then generating the
corresponding actions for each temporal transition. The process of searching
state trajectories is illustrated in Fig. 8 (b). The challenge here is to
efficiently search or sample configurations that are dynamically feasible in a
large state space. Therefore, related research focuses on defining state
representation or transition models in an efficient way.

Sintov et al. [112] demonstrated that the configuration space of elastic rods,
which represents the set of all possible positions and orientations of the
rods, is a smooth manifold with finite dimension. They also proposed a graph-
based parameterization of this manifold and a sampling-based algorithm to
compute a collision-free and kinematically feasible path between any two
stable rod configurations. The research proposed by Lui and Saxena [113]
improves the efficiency of planning for DLO shaping tasks by using a simple
yet effective DLO energy model to calculate a coarse path that guaranteed the
achievability of the task. They also designed a local controller that follows
this path and adjusts it with closed-loop feedback to compensate for planning
errors and achieve task accuracy. Other studies [114, 115, 91] also focus on
defining state representation or a transition model efficiently to search
state trajectories.

###  4.2 Control strategies for DOM

Control aims at designing inputs for robots to execute desired motions.
Closed-loop control which utilizes sensory feedback can cope with various
sources of uncertainty in DOM. However, due to the complex dynamics of DOs, it
is not feasible to synthesize a global control strategy with theoretical
guarantees. Therefore, local controllers are employed to exert force or
velocity at designated operating points, typically following

$u_{t}=u(\emptyset(x_{t})-\emptyset(x_{g}),\theta)$ (11)

where $x_{t}$ is the current state, $x_{g}$ is the goal state and
$\phi(\cdot)$ is the feature space.

Figure 9: A simple closed-loop control framework based on visual features
(DLOs manipulation as an example).

Control minimizes the mismatch between the current state $x_{i}$ and the
target state $x_{g}$, possibly in a feature space $\phi(\cdot)$. Due to the
real-time constraint, $u(\cdot)$ is usually computed using simple methods,
such as a linear function between operating and feature points [116, 117, 118]
or a numerical optimization procedure with a single-step horizon [119, 120]. A
simple control framework for DOM is shown in Fig. 9.

Some examples use pure visual-servoing frameworks to regulate velocities,
e.g., arranging cloth [121, 119], manipulating DLOs [4, 114, 122, 123, 101].
On the other hand, tactile features are used to adjust the manipulation
actions, e.g., cable manipulation [51].

###  4.3 Learning-based approach

A variety of DOM tasks can be achieved by learning-based approaches. This
review focuses on RL and IL. Learning-based methods provide the advantage of
eliminating the need for DOs’ dynamical model which is difficult to be
acquired for many DOs.

####  4.3.1 Reinforcement learning for DOM

RL is a paradigm for learning optimal control policies from trial-and-error
interactions with an environment. The learning objective is to maximize the
cumulative rewards obtained during the interactions. Fig. 10. shows a pipeline
of RL.

RL aims to find optimal policy $\pi$ by maximizing the expected future
accumulative reward. It can be expressed as

$\pi^{\ast}=\mathop{\arg\max}\limits_{\pi\in\Pi}\mathbb{E}_{\tau\in
d^{\pi}}({\sum_{t=1}^{T}{\gamma^{t-1}r_{t}}})$ (12)

where $\tau$ denotes a trajectory from the distribution $d^{\pi}$ of
trajectories generated by the MDP or policy $\pi$. $r_{t}$ denotes the reward
collected at step $t$. $\gamma$ represents the discount factor ranging from 0
to 1 to regulate the significance of a reward with the increment of its
collected future step.

Figure 10: RL pipeline (Cloth folding as an example). First, the
configurations of the DO are encoded into a latent space. Then, the RL agent
outputs manipulation actions that are mapped from the current and goal latent
representation to interact with the environment continuously until the task is
completed.

Generally, an explicit dynamics model of DOs is not required in model-free RL.
Despite this, RL faces its own set of challenges. One such challenge is the
exploration process is often sample inefficient, especially for DOs with
infinite dimensions. Another challenge arises from the inherent danger of
conducting tens of thousands of trial-and-error interactions in real
environments. The development of simulators has facilitated training in
simulated environments. However, the primary challenge lies in transferring
learned strategies to real robots.

Embedding structures or reducing the dimensionality of features can be
performed in the strategy design to improve RL efficiency. Wu et al. [124]
suggested an iterative pick-and-place action space that takes into account the
conditional relationships between picking and placing actions on DOs. This
approach employs explicit structural encoding to facilitate learning an DOs’
organizing skill. Colomé et al. [125] investigated the simultaneous learning
of a DMP-characterized robot motion and its underlying joint couplings using
linear dimensionality reduction. This approach provides valuable qualitative
information, resulting in a concise and intuitive algebraic description of the
motion. Moreover, the proposed method efficiently reduces the number of
parameters to be explored for RL algorithms.

Model-free RL methods, which are widely used for deep RL algorithms, suffer
from high data demand that hinders their applicability to real-world robotics
problems. In contrast, model-based RL approaches are expected to achieve much
better data efficiency than model-free RL methods, provided that the model is
accessible [126]. A novel model-based RL system that incorporates sensing
information, named SAM-RL, was developed by Lv et al. [127]. SAM-RL utilizes
differential physics-based simulation and rendering to dynamically update the
model. This process involves comparing the rendered image with the real
original image, leading to the efficient generation of policies. The system
demonstrates remarkable performance in a needle-threading task, achieving a
substantial reduction in training time and a significant increase in success
rate. In addition, Yang et al. [128] employed a framework of model-based RL
that interleaves learning with exploration to solve a 3-D shape control task
of DLOs. The proposed method demonstrates superior cable shape control
performance while utilizing significantly fewer interaction samples than
state-of-the-art model-free RL methods.

In order to reduce the cost of strategy iteration, initializing RL by manual
demonstration has become a feasible way. Zheng et al. [16] initialized a page-
flipping trajectory manually in a real robot. The page-flipping trajectory is
then further learned by using feedback from tactile sensors as a reward,
allowing a real robot to learn how to turn a book in just 200 interactions of
trial and error. Moreover, the work proposed by Tsurumine et al. [129] has
demonstrated the capability to learn laundry folding with mere 80 samples for
strategy initialization and 100 samples for RL.

The emergence of simulators has enabled the execution of learning iterations
in simulated environments. However, when transferring the learned policies to
the real world, there is often a discrepancy between simulation and reality.
Domain randomization (DR) is a technique that aims to mitigate this
discrepancy between visual simulation and reality by randomly augmenting the
visual parameters of the simulation, such as texture and illumination. This
approach enables the strategy to learn generic and task-relevant visual
features. Despite the success of DR methods in achieving cross-domain
generalization [130], they still face challenges in terms of task-specificity
and adaptability. The work proposed by Matas et al. [131] uses a visual RL
pipeline based on pixel-level domain adaptation to bridge the gap between
simulation and reality, and demonstrates its effectiveness in a tissue
retraction medical manipulation task. Besides, learning a residual strategy to
bridge the gap between reality and simulation is another significant approach.
For instance, the work proposed by Lv et al. [127] successfully performs the
task of threading a needle in the real world by learning a residual strategy.
This shows that learning a residual strategy can help alleviate the sim-real
gap.

####  4.3.2 Imitation learning for DOM

IL is a control technique that leverages data from expert demonstrations to
learn a policy that maps raw inputs to actions. IL has been successfully
applied to various domains of manipulation [132, 133, 134]. Fig. 11. shows a
pipeline of IL. The objective of IL is to acquire a policy that can accurately
imitate the behavior of an expert demonstrator. This method can be
mathematically expressed as

$\pi^{\ast}=\mathop{\arg\min}\limits_{\pi\in\Pi}\mathbb{D}(p(\pi^{demo})||p(\pi))$
(13)

where $\mathbb{D}$ is the divergence metric that quantifies the dissimilarity
between the two distributions. The operands in the divergence can be the
distribution of state–action pairs, trajectory feature expectation or the
distribution of trajectory features.

Figure 11: IL Pipeline (Cloth folding as an example [17]). (a) Offline: Input
images are manually annotated with primitives and gripper poses. A neural
network is trained on the data and iteratively used for self-supervised data
collection. (b) Online: The neural network is used to predict a primitive and
a pair of poses given an input image, and execute it on the robot. If the
resulting garment configuration is classified as finished, the robot will
stop. Otherwise, it will repeat the process.

A key distinction between IL and RL is that IL does not require a reward
function, thus eliminating the need for reward engineering which is difficult
to achieve in many robotic applications. However, this method has some
limitations, such as the need for extensive demonstrations and poor
generalization.

Early work on IL focuses on extracting primitive motions and enabling robots
to learn primitive motions for DOM through kinesthetic teaching. For example,
a method for aerial knot tying by a two-armed, multi-fingered robot was
proposed by Kudoh et al. [135]. Knot tying tasks are decomposed into six
primitive motions, namely grasping, releasing, double grasping, wrapping,
twisting, and sliding. The robot learns these six primitive motions through
human demonstrations and combines them to perform various knot tying tasks. In
addition to aerial knotting, this method and its variants realize rope tying
[136], fabric folding [137] and stitching [138]. A further extension is to
incorporate task parameters to models for enhancing the generality of models.
For example, the dressing motion can be adapted to different arm postures by
using task-parameterized models [139]. Recent research has made it possible to
plan actions based on current observations. This is achieved by collecting and
annotating a large amount of demonstration data and extracting features [140,
141, 142, 143, 17]. For example, a series of two-arm action primitives for
folding clothes are first parameterized [17]. After learning from 4,300 human-
labeled actions, the system can predict pairs of jaw-clamping poses based on
visual information. It has been able to fold randomly placed garments with a
93$\%$ success rate in an average of 120 seconds.

IL methods require retraining to transfer learned skills from known instances
to similar/new instances. In the case of a hat-wearing task, the hat-wearing
step needs to be repeated thousands of times during the training phase in both
simulated and real-world environments. However, to process a new hat, we need
collect data new data and retrain the model for the new hat, due to the
variations in its shape and deformation. Consequently, these steps impose
significant computational and temporal costs, which hinders the feasibility of
this approach in real-world scenarios. Recently, several studies [144, 145]
have explored IL for category-level manipulation for rigid objects. However,
these methods cannot be applied to DOs. Unlike rigid objects whose poses can
be fully represented by low-dimensional vectors, DOs have an infinite
configuration space that is prone to severe self-occlusion. These features
pose a challenge for skill generalization across different DOs. Recently, Ren
et al. [146] proposed a novel framework for manipulating deformable 3-D
objects at the category level, which enables the transfer of learned skills to
new instances of similar objects with only a single demonstration.
Specifically, the framework consists of two modules. The first module, Nocs
State Transition (NST), converts the observed target point cloud to a
predefined uniform pose state (i.e., Nocs state), which serves as the
foundation for category-level skill learning. The second module, Neuro-Spatial
Encoding (NSE), adapts the learned skills to new instances by encoding
category-level spatial information to infer the optimal grasping point without
retraining.

###  4.4 Discussion of manipulation

Currently, significant progress in short-horizon tasks for DOM has been made.
However, real-world manipulation tasks often require a sequence of subtasks
that have distinct characteristics and objectives. These long-horizon and
complex tasks demand dexterous hands or multiple tools to perform different
subtasks. Compositional generalization for subskills is regarded as a crucial
challenge for long-horizon tasks.

Dexterous hands possess adaptability and versatility, enabling smooth
transitions between various functional modes without relying on external
tools. However, the high-dimensional action space of dexterous hands poses
challenges. Some research has shown dexterous hands can be utilized for
manipulation of rigid objects. Chen et al. [147] presented a comprehensive
system that leverages RL to chain multiple dexterous policies for achieving
complex task goals. However, their system only deals with rigid object
manipulation. The application of dexterous hands to long-horizon tasks for DOM
remains an area that requires further investigation.

In addition to dexterous hands, the usage of diverse tools is also a promising
approach to tackle long-horizon tasks. Human beings demonstrate remarkable
proficiency in performing complex and long-horizon manipulation tasks via
flexible tool use. However, robotic tool use is still constrained by the
difficulties in understanding tool-object interactions. RoboCook proposed by
Shi et al. [11], manipulates elasto-plastic objects such as dough for long-
horizon tasks using multiple tools. However, failures such as dough sticking
to the tool often occur. The utilization of multiple tools to accomplish long-
horizon tasks warrants further investigation.

Compositional generalization for subskills is a significant challenge for
long-horizon tasks. Planning skill sequences with a large search space is
inherently difficult, but the motion planning community has achieved
remarkable advances in producing a plan skeleton [148, 149]. For instance,
Garrett et al. [148] suggested two approaches. The first approach involves
alternating between searching for the skill sequence and performing lower-
level optimization. The second approach employs lazy placeholders for some
skills. Subsequent studies have also investigated the use of pre-trained LLMs
for discovering skill sequences [150, 151]. Compositional generalization for
subskills is a challenging problem that requires further study.

Table 3: Summary of main literature on manipulation planning, control, and learning. Year |  Representative literature | Type of objects | Peception | Modeling
---|---|---|---|---
|  |  1-D objects |  2-D objects |  3-D objects | Visual | Tactile |  Analytical approaches |  Data-driven approaches | None
Analytical planning and control
2014 | Bretl and McCarthy [114] | ✓ |  |  | ✓ |  | ✓ |  |
2015 | Li et al. [110] |  | ✓ |  | ✓ |  | ✓ |  |
2015 | Li et al. [111] |  |  | ✓ | ✓ |  | ✓ |  |
2017 | Zaidi et al. [61] |  |  | ✓ |  | ✓ | ✓ |  |
2018 | Navarro-Alarcon and Liu [118] |  | ✓ |  | ✓ |  | ✓ |  |
2018 | Li et al. [4] | ✓ |  |  | ✓ |  | ✓ |  |
2020 | Sintov et al. [112] | ✓ |  |  | ✓ |  | ✓ |  |
2020 | McConachie et al. [91] | ✓ |  |  | ✓ |  | ✓ |  |
2021 | Koessler et al. [101] | ✓ |  |  | ✓ |  | ✓ |  |
2021 | She et al. [51] | ✓ |  |  |  | ✓ | ✓ |  |
2022 | Shi et al. [60] |  |  | ✓ | ✓ |  |  | ✓ |
2022 | Lv et al. [127] | ✓ |  |  | ✓ |  | ✓ |  |
2023 | Huang et al. [109] | ✓ |  |  | ✓ |  |  | ✓ |
Reinforcement learning
2018 | Peng et al. [130] |  |  | ✓ | ✓ |  |  |  | ✓
2018 | Colomé and Torras [125] |  | ✓ |  | ✓ |  |  |  | ✓
2018 | Matas et al. [131] |  | ✓ |  | ✓ |  |  |  | ✓
2019 | Tsurumine et al. [129] |  | ✓ |  | ✓ |  |  |  | ✓
2020 | Wu et al. [124] |  | ✓ |  | ✓ |  |  |  | ✓
2022 | Zheng et al. [16] |  | ✓ |  |  | ✓ |  |  | ✓
2022 | Yang et al. [128] | ✓ |  |  | ✓ |  |  | ✓ |
2023 | Lv et al. [41] | ✓ |  |  | ✓ |  |  |  | ✓
Imitation learning
2015 | Kudoh et al. [135] | ✓ |  |  | ✓ |  |  |  | ✓
2017 | Yang et al. [140] |  | ✓ |  | ✓ |  |  |  | ✓
2017 | Pignat and Calinon [139] |  | ✓ |  | ✓ |  |  |  | ✓
2018 | Cherubini et al. [141] |  | ✓ |  | ✓ |  |  |  | ✓
2020 | Seita et al. [142] |  | ✓ |  | ✓ |  |  |  | ✓
2022 | Ha and Song [143] |  | ✓ |  | ✓ |  |  |  | ✓
2022 | Avigal et al. [17] |  | ✓ |  | ✓ |  |  |  | ✓
2022 | Lin et al. [152] |  |  | ✓ | ✓ |  |  |  | ✓
2022 | Lin et al. [153] |  |  | ✓ | ✓ |  |  |  | ✓
2023 | Ren et al. [146] |  |  | ✓ | ✓ |  |  |  | ✓

##  5 Discussion of Large Languages Models and Outlooks

LLMs recently have achieved impressive performance on various natural language
processing (NLP) tasks[154, 155]. Consequently, there has been a surge of
interest in their potential application to the field of robotic manipulation.
How can the rich internalized knowledge of the LLMs be leveraged in robotic
manipulation? Recent studies utilizing LLMs have made initial progress in task
definition for manipulation, planning, reward function design and uncertainty
alignment. LLMs are now mainly used for rigid object manipulation. For more
complex tasks involving DOM, further research is required.

###  5.1 Task definition

For accomplishing a robot manipulation task, the task first needs to be
defined at the semantic level. For example, what does it mean to fold a piece
of clothing? Existing approaches tend to design one configuration as the task
goal. However, it is impractical to manually specify all configurations of DOs
as the goal for such tasks. Instead, we need a method to acquire the semantic
knowledge of concepts, such as folding or wrapping, so that we can evaluate
the validity of a goal state of an object configuration. LLMs exhibit a
surprising ability to generalize, providing a possible way to achieve task
definitions for DOM at the semantic level.

Moreover, recent studies have made progress in learning human preferences
using LLMs. A system developed by Wu et al. [156] demonstrates the ability to
learn human preferences by engaging with an individual using a limited number
of examples. The finding indicates that robots can effectively leverage
language-based planning and perception, along with the summarization
capabilities of LLMs, to infer generic user preferences that can be widely
applied to future interactions. The generalized user preferences could
potentially serve as a priori knowledge for defining manipulation tasks.
Consequently, it is worthwhile to explore task definitions that incorporate
human preferences utilizing LLMs.

###  5.2 Planning

Combining abstract linguistic instructions (e.g., folding a shirt) with robot
actions is an important step in accomplishing robot manipulation tasks.
Previous studies have utilized lexical analysis to parse instructions [157,
158, 159]. More recent studies have utilized language models to decompose
instructions into a series of literal steps [151, 150, 160]. To achieve the
physical interaction between the robot and the environment according to the
processed instructions, existing methods typically depend on primitives (i.e.,
skills) that are either manually designed or pre-trained. For instance, a
series of primitive actions such as grasping, rolling, pushing, pulling,
tilting, closing, opening are pre-designed manually or trained. These
primitives can be invoked by LLMs or a planner to accomplish the corresponding
tasks. Liang et al. [161] showed that LLMs exhibit behavioral commonsense that
can be used for underlying control. Despite some promising signs, there is
still a need to manually design motion primitives. This reliance on individual
skill acquisition is often considered as a significant bottleneck for the
system due to the scarcity of large-scale robotic manipulation data. How can
the rich internalized knowledge of the LLMs be leveraged to provide robots
with a finer-grained level of action without the need for laborious data
collection or manual design for each individual primitive? This research
direction holds immense value.

###  5.3 Reward function design

Reward design in RL is challenging since specifying the notion of desired
human behavior through a reward function may be difficult or require much
expert argumentation. Consequently, an intriguing question arises: can rewards
be designed using a natural language interface? Kwon et al. [162] explored how
reward design can be simplified by prompting LLMs. In this approach, users
provide textual prompts that contains several examples or descriptions of
desired behaviors. The approach utilizes this agent reward functionality
within the RL framework. Specifically, users specify a cue once at the
beginning of training. During training, the LLM evaluates the behavior of the
RL agent based on the desired behavior described by the cue and outputs a
corresponding reward signal. Then, the RL agent updates its behavior using
that reward. The limitation of the framework is the LLMs can only specify
binary rewards. It is worth investigating how the likelihood values generated
by LLMs for each word can be used as non-binary reward signals in future
research. Similar works [163, 164] have also suggested the use of pre-trained
foundation models to generate reward functions for new robotic manipulation
tasks. However, further research should be conducted to explore for more
complex tasks involving DOM.

###  5.4 Uncertainty alignment for language-instructed robots

Large-scale language modeling has demonstrated a wide range of capabilities,
from step-by-step planning to commonsense reasoning, which can be useful for
robots. However, these models are still prone to confident illusory
predictions. In this regard, Yu et al. [165] proposed a framework for
measuring and adjusting uncertainty in language model-based planners. This
framework enables robots to recognize when they lack knowledge and seek
assistance when necessary. A key assumption of this framework’s task
completion guarantee is that the environment (objects) is solely based on the
text input to the language model. Additionally, it assumes that the actions
proposed by the language model planner can be executed successfully. Future
research needs to incorporate uncertainty in perceptual modules (e.g., visual
language models) and underlying action strategies (e.g., language-conditioned
affordance prediction).

##  6 Conclusion

In this review, we have presented a comprehensive review of recent advances,
open challenges and new frontiers of DOM. We mainly focus on data-driven
approaches in aspects of perception, modeling and manipulation, although
analytical approaches are also concisely reviewed.

For perception, visual perception and tactile perception methods are reviewed.
The robust sensing of DOs in the presence of complex backgrounds poses an open
challenge for visual perception. Tactile perception heavily depends on the use
of high-resolution tactile devices, both in real-world scenarios and
simulations. Additionally, we regard multimodal sensing as future research
direction for perception.

For modeling, analytical modeling approaches and data-driven modeling
approaches are reviewed. Analytical modeling approaches cannot accurately
represent DOs due to their infinitely-dimensional states and the challenges
associated with obtaining DOs’ parameters in real-world settings. In contrast,
data-driven approaches can generate a reasonably accurate model from data
without the need for understanding the physical dynamics of DOs, given an
abundance of high-quality data. Whether analytical or learned models are
employed, their predictive power is limited. When the model is unreliable, it
may be desirable to replan or recover.

For manipulation, analytical manipulation methods and learning-based methods
are reviewed. When the accurate dynamic model of DOs is available, analytical
manipulation methods are preferred. However, obtaining the explicit dynamic
model of DOs is challenging. Learning-based approaches are more versatile as
they do not require the explicit dynamic model of DOs. In the future, it is
crucial to explore long-horizon tasks that involve multiple tools, dexterous
hands and compositional generalization.

For LLMs, recent studies have made initial progress in task definition for
manipulation, planning, reward function design and uncertainty alignment.
However, leveraging the rich internalized knowledge of LLMs in robotic
manipulation especially for DOM still requires further investigation.

The survey provides insights into advanced research of DOM mainly focusing on
data-driven approaches, and points out promising future research such as LLMs
for DOM.

## References

  * [1] O. Kroemer, S. Niekum, and G. Konidaris, “A review of robot learning for manipulation: Challenges, representations, and algorithms,” _The Journal of Machine Learning Research_ , vol. 22, no. 1, pp. 1395–1476, 2021.
  * [2] J. Zhu, A. Cherubini, C. Dune, D. Navarro-Alarcon, F. Alambeigi, D. Berenson, F. Ficuciello, K. Harada, J. Kober, X. Li _et al._ , “Challenges and outlook in robotic manipulation of deformable objects,” _IEEE Robotics & Automation Magazine_, vol. 29, no. 3, pp. 67–77, 2022.
  * [3] Y. Gao, Z. Chen, Y. Ling, J. Yang, Y.-H. Liu, and X. Li, “A hierarchical manipulation scheme for robotic sorting of multiwire cables with hybrid vision,” _IEEE/ASME Transactions on Mechatronics_ , 2022.
  * [4] X. Li, X. Su, and Y.-H. Liu, “Vision-based robotic manipulation of flexible pcbs,” _IEEE/ASME Transactions on Mechatronics_ , vol. 23, no. 6, pp. 2739–2749, 2018.
  * [5] S. Jin, D. Romeres, A. Ragunathan, D. K. Jha, and M. Tomizuka, “Trajectory optimization for manipulation of deformable objects: Assembly of belt drive units,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 10 002–10 008.
  * [6] H. Zhou, S. Li, Q. Lu, and J. Qian, “A practical solution to deformable linear object manipulation: A case study on cable harness connection,” in _2020 5th International Conference on Advanced Robotics and Mechatronics (ICARM)_. IEEE, 2020, pp. 329–333.
  * [7] L. Cao, X. Li, P. T. Phan, A. M. H. Tiong, H. L. Kaan, J. Liu, W. Lai, Y. Huang, H. M. Le, M. Miyasaka _et al._ , “Sewing up the wounds: A robotic suturing system for flexible endoscopy,” _IEEE Robotics & Automation Magazine_, vol. 27, no. 3, pp. 45–54, 2020.
  * [8] N. Haouchine, J. Dequidt, I. Peterlik, E. Kerrien, M.-O. Berger, and S. Cotin, “Image-guided simulation of heterogeneous tissue deformation for augmented reality during hepatic surgery,” in _2013 IEEE international symposium on mixed and augmented reality (ISMAR)_. IEEE, 2013, pp. 199–208.
  * [9] X. Chen and J. Hu, “A review of haptic simulator for oral and maxillofacial surgery based on virtual reality,” _Expert Review of Medical Devices_ , vol. 15, no. 6, pp. 435–444, 2018.
  * [10] F. Liu, Z. Li, Y. Han, J. Lu, F. Richter, and M. C. Yip, “Real-to-sim registration of deformable soft tissue with position-based dynamics for surgical robot autonomy,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 12 328–12 334.
  * [11] H. Shi, H. Xu, S. Clarke, Y. Li, and J. Wu, “Robocook: Long-horizon elasto-plastic object manipulation with diverse tools,” _arXiv preprint arXiv:2306.14447_ , 2023.
  * [12] Z. Xu, Z. Xian, X. Lin, C. Chi, Z. Huang, C. Gan, and S. Song, “Roboninja: Learning an adaptive cutting policy for multi-material objects,” _arXiv preprint arXiv:2302.11553_ , 2023.
  * [13] M. C. Gemici and A. Saxena, “Learning haptic representation for manipulating deformable food objects,” in _2014 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2014, pp. 638–645.
  * [14] Z. Li, F. Miao, Z. Yang, P. Chai, and S. Yang, “Factors affecting human hand grasp type in tomato fruit-picking: A statistical investigation for ergonomic development of harvesting robot,” _Computers and electronics in agriculture_ , vol. 157, pp. 90–97, 2019.
  * [15] G. Narasimhan, K. Zhang, B. Eisner, X. Lin, and D. Held, “Self-supervised transparent liquid segmentation for robotic pouring,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 4555–4561.
  * [16] Y. Zheng, F. F. Veiga, J. Peters, and V. J. Santos, “Autonomous learning of page flipping movements via tactile feedback,” _IEEE Transactions on Robotics_ , vol. 38, no. 5, pp. 2734–2749, 2022.
  * [17] Y. Avigal, L. Berscheid, T. Asfour, T. Kröger, and K. Goldberg, “Speedfolding: Learning efficient bimanual folding of garments,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 1–8.
  * [18] F. Zhang and Y. Demiris, “Learning garment manipulation policies toward robot-assisted dressing,” _Science robotics_ , vol. 7, no. 65, p. eabm6010, 2022.
  * [19] A. Billard and D. Kragic, “Trends and challenges in robot manipulation,” _Science_ , vol. 364, no. 6446, p. eaat8414, 2019.
  * [20] H. Yin, A. Varava, and D. Kragic, “Modeling, learning, perception, and control methods for deformable object manipulation,” _Science Robotics_ , vol. 6, no. 54, p. eabd8803, 2021.
  * [21] H. A. Kadi and K. Terzić, “Data-driven robotic manipulation of cloth-like deformable objects: The present, challenges and future prospects,” _Sensors_ , vol. 23, no. 5, p. 2389, 2023.
  * [22] N. Lv, J. Liu, H. Xia, J. Ma, and X. Yang, “A review of techniques for modeling flexible cables,” _Computer-Aided Design_ , vol. 122, p. 102826, 2020.
  * [23] Y. C. Hou, K. S. M. Sahari, and D. N. T. How, “A review on modeling of flexible deformable object for dexterous robotic manipulation,” _International Journal of Advanced Robotic Systems_ , vol. 16, no. 3, p. 1729881419848894, 2019.
  * [24] V. E. Arriola-Rios, P. Guler, F. Ficuciello, D. Kragic, B. Siciliano, and J. L. Wyatt, “Modeling of deformable objects for robotic manipulation: A tutorial and review,” _Frontiers in Robotics and AI_ , vol. 7, p. 82, 2020.
  * [25] J. Sanchez, J.-A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar, “Robotic manipulation and sensing of deformable objects in domestic and industrial applications: a survey,” _The International Journal of Robotics Research_ , vol. 37, no. 7, pp. 688–716, 2018.
  * [26] M. A. Lee, Y. Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg, “Making sense of vision and touch: Learning multimodal representations for contact-rich tasks,” _IEEE Transactions on Robotics_ , vol. 36, no. 3, pp. 582–596, 2020.
  * [27] P. Battaglia, R. Pascanu, M. Lai, D. Jimenez Rezende _et al._ , “Interaction networks for learning about objects, relations and physics,” _Advances in neural information processing systems_ , vol. 29, 2016.
  * [28] T. Tang and M. Tomizuka, “Track deformable objects from point clouds with structure preserved registration,” _The International Journal of Robotics Research_ , vol. 41, no. 6, pp. 599–614, 2022.
  * [29] A. Keipour, M. Bandari, and S. Schaal, “Deformable one-dimensional object detection for routing and manipulation,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 4329–4336, 2022.
  * [30] M. Yan, Y. Zhu, N. Jin, and J. Bohg, “Self-supervised learning of state estimation for manipulating deformable linear objects,” _IEEE robotics and automation letters_ , vol. 5, no. 2, pp. 2372–2379, 2020.
  * [31] D. De Gregorio, G. Palli, and L. Di Stefano, “Let’s take a walk on superpixels graphs: Deformable linear objects segmentation and model estimation,” in _Asian Conference on Computer Vision_. Springer, 2018, pp. 662–677.
  * [32] J. Ma, S. Hu, J. Fu, and G. Chen, “A hierarchical attention detector for bearing surface defect detection,” _Expert Systems with Applications_ , p. 122365, 2023.
  * [33] A. Caporali, R. Zanella, D. De Greogrio, and G. Palli, “Ariadne+: Deep learning–based augmented framework for the instance segmentation of wires,” _IEEE Transactions on Industrial Informatics_ , vol. 18, no. 12, pp. 8607–8617, 2022.
  * [34] M. Denninger, M. Sundermeyer, D. Winkelbauer, D. Olefir, T. Hodan, Y. Zidan, M. Elbadrawy, M. Knauer, H. Katam, and A. Lodhi, “Blenderproc: Reducing the reality gap with photorealistic rendering,” in _International Conference on Robotics: Sciene and Systems, RSS 2020_ , 2020.
  * [35] W. Qiu and A. Yuille, “Unrealcv: Connecting computer vision to unreal engine,” in _Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III 14_. Springer, 2016, pp. 909–916.
  * [36] A. Caporali, K. Galassi, R. Zanella, and G. Palli, “Fastdlo: Fast deformable linear objects instance segmentation,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 4, pp. 9075–9082, 2022.
  * [37] J. Qian, T. Weng, L. Zhang, B. Okorn, and D. Held, “Cloth region segmentation for robust grasp selection,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 9553–9560.
  * [38] H. Dinkel, J. Xiang, H. Zhao, B. Coltin, T. Smith, and T. Bretl, “Wire point cloud instance segmentation from rgbd imagery with mask r-cnn,” in _IEEE Int. Conf. Robot. Autom.(ICRA) Workshop on Representing and Manipulating Deformable Objects_ , 2022.
  * [39] R. Zanella, A. Caporali, K. Tadaka, D. De Gregorio, and G. Palli, “Auto-generated wires dataset for semantic segmentation with domain-independence,” in _2021 International Conference on Computer, Control and Robotics (ICCCR)_. IEEE, 2021, pp. 292–298.
  * [40] M. Wnuk, C. Hinze, A. Lechler, and A. Verl, “Kinematic multibody model generation of deformable linear objects from point clouds,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 9545–9552.
  * [41] K. Lv, M. Yu, Y. Pu, X. Jiang, G. Huang, and X. Li, “Learning to estimate 3-d states of deformable linear objects from single-frame occluded point clouds,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 7119–7125.
  * [42] R. A. Newcombe, D. Fox, and S. M. Seitz, “Dynamicfusion: Reconstruction and tracking of non-rigid scenes in real-time,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 343–352.
  * [43] M. Dou, P. Davidson, S. R. Fanello, S. Khamis, A. Kowdle, C. Rhemann, V. Tankovich, and S. Izadi, “Motion2fusion: Real-time volumetric performance capture,” _ACM Transactions on Graphics (ToG)_ , vol. 36, no. 6, pp. 1–16, 2017.
  * [44] T. Collins, A. Bartoli, N. Bourdel, and M. Canis, “Robust, real-time, dense and deformable 3d organ tracking in laparoscopic videos,” in _International Conference on Medical Image Computing and Computer-Assisted Intervention_. Springer, 2016, pp. 404–412.
  * [45] J. Schulman, A. Lee, J. Ho, and P. Abbeel, “Tracking deformable objects with point clouds,” in _2013 IEEE International Conference on Robotics and Automation_. IEEE, 2013, pp. 1130–1137.
  * [46] A. Petit, V. Lippiello, G. A. Fontanelli, and B. Siciliano, “Tracking elastic deformable objects with an rgb-d sensor for a pizza chef robot,” _Robotics and Autonomous Systems_ , vol. 88, pp. 187–201, 2017.
  * [47] T. Tang, Y. Fan, H.-C. Lin, and M. Tomizuka, “State estimation for deformable objects by point registration and dynamic simulation,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 2427–2433.
  * [48] T. Tang, C. Wang, and M. Tomizuka, “A framework for manipulating deformable linear objects by coherent point drift,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 4, pp. 3426–3433, 2018.
  * [49] A. Myronenko and X. Song, “Point set registration: Coherent point drift,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 32, no. 12, pp. 2262–2275, 2010.
  * [50] C. Chi and D. Berenson, “Occlusion-robust deformable object tracking without physics simulation,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 6443–6450.
  * [51] Y. She, S. Wang, S. Dong, N. Sunil, A. Rodriguez, and E. Adelson, “Cable manipulation with a tactile-reactive gripper,” _The International Journal of Robotics Research_ , vol. 40, no. 12-14, pp. 1385–1401, 2021.
  * [52] Y. S. Narang, B. Sundaralingam, K. Van Wyk, A. Mousavian, and D. Fox, “Interpreting and predicting tactile signals for the syntouch biotac,” _The International Journal of Robotics Research_ , vol. 40, no. 12-14, pp. 1467–1487, 2021.
  * [53] W. Yuan, S. Dong, and E. H. Adelson, “Gelsight: High-resolution robot tactile sensors for estimating geometry and force,” _Sensors_ , vol. 17, no. 12, p. 2762, 2017.
  * [54] S. Sundaram, P. Kellnhofer, Y. Li, J.-Y. Zhu, A. Torralba, and W. Matusik, “Learning the signatures of the human grasp using a scalable tactile glove,” _Nature_ , vol. 569, no. 7758, pp. 698–702, 2019.
  * [55] R. B. Hellman, C. Tekin, M. van der Schaar, and V. J. Santos, “Functional contour-following via haptic perception and reinforcement learning,” _IEEE transactions on haptics_ , vol. 11, no. 1, pp. 61–72, 2017.
  * [56] Q. Zhang, Y. Li, Y. Luo, W. Shou, M. Foshey, J. Yan, J. B. Tenenbaum, W. Matusik, and A. Torralba, “Dynamic modeling of hand-object interactions via tactile sensing,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 2874–2881.
  * [57] J. A. Fishel and G. E. Loeb, “Sensing tactile microvibrations with the biotac—comparison with human sensitivity,” in _2012 4th IEEE RAS & EMBS international conference on biomedical robotics and biomechatronics (BioRob)_. IEEE, 2012, pp. 1122–1127.
  * [58] S. Zhang, Z. Chen, Y. Gao, W. Wan, J. Shan, H. Xue, F. Sun, Y. Yang, and B. Fang, “Hardware technology of vision-based tactile sensor: A review,” _IEEE Sensors Journal_ , 2022.
  * [59] J. DelPreto, C. Liu, Y. Luo, M. Foshey, Y. Li, A. Torralba, W. Matusik, and D. Rus, “Actionsense: A multimodal dataset and recording framework for human activities using wearable sensors in a kitchen environment,” _Advances in Neural Information Processing Systems_ , vol. 35, pp. 13 800–13 813, 2022.
  * [60] H. Shi, H. Xu, Z. Huang, Y. Li, and J. Wu, “Robocraft: Learning to see, simulate, and shape elasto-plastic objects with graph networks,” _arXiv preprint arXiv:2205.02909_ , 2022.
  * [61] L. Zaidi, J. A. Corrales, B. C. Bouzgarrou, Y. Mezouar, and L. Sabourin, “Model-based strategy for grasping 3d deformable objects using a multi-fingered robotic hand,” _Robotics and Autonomous Systems_ , vol. 95, pp. 196–206, 2017.
  * [62] L. Pecyna, S. Dong, and S. Luo, “Visual-tactile multimodality for following deformable linear objects using reinforcement learning,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 3987–3994.
  * [63] S. Caccamo, P. Güler, H. Kjellström, and D. Kragic, “Active perception and modeling of deformable surfaces using gaussian processes and position-based dynamics,” in _2016 IEEE-RAS 16th International Conference on Humanoid Robots (Humanoids)_. IEEE, 2016, pp. 530–537.
  * [64] V. E. Arriola-Rios and J. L. Wyatt, “A multimodal model of object deformation under robotic pushing,” _IEEE Transactions on Cognitive and Developmental Systems_ , vol. 9, no. 2, pp. 153–169, 2017.
  * [65] J. Sanchez, C. M. Mateo, J. A. Corrales, B.-C. Bouzgarrou, and Y. Mezouar, “Online shape estimation based on tactile sensing and deformation modeling for robot manipulation,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 504–511.
  * [66] M. Yu, K. Lv, C. Wang, M. Tomizuka, and X. Li, “A coarse-to-fine framework for dual-arm manipulation of deformable linear objects with whole-body obstacle avoidance,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 10 153–10 159.
  * [67] Y. Kita, F. Kanehiro, T. Ueshiba, and N. Kita, “Clothes handling based on recognition by strategic observation,” in _2011 11th IEEE-RAS International Conference on Humanoid Robots_. IEEE, 2011, pp. 53–58.
  * [68] M. Müller, J. Stam, D. James, and N. Thürey, “Real time physics: class notes,” in _ACM siggraph 2008 classes_ , 2008, pp. 1–90.
  * [69] M. Teschner, B. Heidelberger, M. Muller, and M. Gross, “A versatile and robust model for geometrically complex deformable solids,” in _Proceedings Computer Graphics International, 2004._ IEEE, 2004, pp. 312–319.
  * [70] M. Macklin, M. Müller, and N. Chentanez, “Xpbd: position-based simulation of compliant constrained dynamics,” in _Proceedings of the 9th International Conference on Motion in Games_ , 2016, pp. 49–54.
  * [71] Z. Hu, T. Han, P. Sun, J. Pan, and D. Manocha, “3-d deformable object manipulation using deep neural networks,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 4, pp. 4255–4261, 2019.
  * [72] W. Yan, A. Vangipuram, P. Abbeel, and L. Pinto, “Learning predictive representations for deformable objects using contrastive estimation,” in _Conference on Robot Learning_. PMLR, 2021, pp. 564–574.
  * [73] M. Yu, K. Lv, H. Zhong, S. Song, and X. Li, “Global model learning for large deformation control of elastic deformable linear objects: An efficient and adaptive approach,” _IEEE Transactions on Robotics_ , vol. 39, no. 1, pp. 417–436, 2022.
  * [74] D. Navarro-Alarcon, Y.-H. Liu, J. G. Romero, and P. Li, “Model-free visually servoed deformation control of elastic objects by robot manipulators,” _IEEE Transactions on Robotics_ , vol. 29, no. 6, pp. 1457–1468, 2013.
  * [75] D. Navarro-Alarcon, H. M. Yip, Z. Wang, Y.-H. Liu, F. Zhong, T. Zhang, and P. Li, “Automatic 3-d manipulation of soft objects by robotic arms with an adaptive deformation model,” _IEEE Transactions on Robotics_ , vol. 32, no. 2, pp. 429–441, 2016.
  * [76] J. Zhu, B. Navarro, P. Fraisse, A. Crosnier, and A. Cherubini, “Dual-arm robotic manipulation of flexible cables,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 479–484.
  * [77] S. Jin, C. Wang, and M. Tomizuka, “Robust deformation model approximation for robotic cable manipulation,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 6586–6593.
  * [78] R. Lagneau, A. Krupa, and M. Marchal, “Automatic shape control of deformable wires based on model-free visual servoing,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 5252–5259, 2020.
  * [79] J. Zhu, D. Navarro-Alarcon, R. Passama, and A. Cherubini, “Vision-based manipulation of deformable and rigid objects using subspace projections of 2d contours,” _Robotics and Autonomous Systems_ , vol. 142, p. 103798, 2021.
  * [80] A. J. Valencia and P. Payeur, “Combining self-organizing and graph neural networks for modeling deformable objects in robotic manipulation,” _Frontiers in Robotics and AI_ , vol. 7, p. 600584, 2020.
  * [81] Y. Li, J. Wu, J.-Y. Zhu, J. B. Tenenbaum, A. Torralba, and R. Tedrake, “Propagation networks for model-based control under partial observation,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 1205–1211.
  * [82] A. Sanchez-Gonzalez, J. Godwin, T. Pfaff, R. Ying, J. Leskovec, and P. Battaglia, “Learning to simulate complex physics with graph networks,” in _International conference on machine learning_. PMLR, 2020, pp. 8459–8468.
  * [83] F. Xia, K. Sun, S. Yu, A. Aziz, L. Wan, S. Pan, and H. Liu, “Graph learning: A survey,” _IEEE Transactions on Artificial Intelligence_ , vol. 2, no. 2, pp. 109–127, 2021.
  * [84] Y. Li, J. Wu, R. Tedrake, J. B. Tenenbaum, and A. Torralba, “Learning particle dynamics for manipulating rigid bodies, deformable objects, and fluids,” _arXiv preprint arXiv:1810.01566_ , 2018.
  * [85] Y. Yang, J. A. Stork, and T. Stoyanov, “Learning to propagate interaction effects for modeling deformable linear objects dynamics,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 1950–1957.
  * [86] D. Mrowca, C. Zhuang, E. Wang, N. Haber, L. F. Fei-Fei, J. Tenenbaum, and D. L. Yamins, “Flexible neural representation for physics prediction,” _Advances in neural information processing systems_ , vol. 31, 2018.
  * [87] J. B. Tenenbaum, C. Kemp, T. L. Griffiths, and N. D. Goodman, “How to grow a mind: Statistics, structure, and abstraction,” _science_ , vol. 331, no. 6022, pp. 1279–1285, 2011.
  * [88] B. M. Lake, R. Salakhutdinov, and J. B. Tenenbaum, “Human-level concept learning through probabilistic program induction,” _Science_ , vol. 350, no. 6266, pp. 1332–1338, 2015.
  * [89] N. D. Goodman, J. B. Tenenbaum, and T. Gerstenberg, “Concepts in a probabilistic language of thought,” Center for Brains, Minds and Machines (CBMM), Tech. Rep., 2014.
  * [90] C. Wang, Y. Zhang, X. Zhang, Z. Wu, X. Zhu, S. Jin, T. Tang, and M. Tomizuka, “Offline-online learning of deformation model for cable manipulation with graph neural networks,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 2, pp. 5544–5551, 2022.
  * [91] D. McConachie, T. Power, P. Mitrano, and D. Berenson, “Learning when to trust a dynamics model for planning in reduced state spaces,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 3540–3547, 2020.
  * [92] T. Power and D. Berenson, “Keep it simple: Data-efficient learning for controlling complex systems with simple models,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 1184–1191, 2021.
  * [93] P. Mitrano, D. McConachie, and D. Berenson, “Learning where to trust unreliable models in an unstructured world for deformable object manipulation,” _Science Robotics_ , vol. 6, no. 54, p. eabd8170, 2021.
  * [94] M. Bergou, M. Wardetzky, S. Robinson, B. Audoly, and E. Grinspun, “Discrete elastic rods,” in _ACM SIGGRAPH 2008 papers_ , 2008, pp. 1–12.
  * [95] N. Lv, J. Liu, X. Ding, J. Liu, H. Lin, and J. Ma, “Physically based real-time interactive assembly simulation of cable harness,” _Journal of Manufacturing Systems_ , vol. 43, pp. 385–399, 2017.
  * [96] R. Goldenthal, D. Harmon, R. Fattal, M. Bercovier, and E. Grinspun, “Efficient simulation of inextensible cloth,” in _ACM SIGGRAPH 2007 papers_ , 2007, pp. 49–es.
  * [97] U. Kühnapfel, H. K. Cakmak, and H. Maaß, “Endoscopic surgery training using virtual reality and deformable tissue simulation,” _Computers & graphics_, vol. 24, no. 5, pp. 671–682, 2000.
  * [98] W. Mollemans, F. Schutyser, J. Van Cleynenbreugel, and P. Suetens, “Tetrahedral mass spring model for fast soft tissue deformation,” in _International Symposium on Surgery Simulation and Soft Tissue Modeling_. Springer, 2003, pp. 145–154.
  * [99] M. Macklin, M. Müller, N. Chentanez, and T.-Y. Kim, “Unified particle physics for real-time applications,” _ACM Transactions on Graphics (TOG)_ , vol. 33, no. 4, pp. 1–12, 2014.
  * [100] M. Zollhöfer, M. Nießner, S. Izadi, C. Rehmann, C. Zach, M. Fisher, C. Wu, A. Fitzgibbon, C. Loop, C. Theobalt _et al._ , “Real-time non-rigid reconstruction using an rgb-d camera,” _ACM Transactions on Graphics (ToG)_ , vol. 33, no. 4, pp. 1–12, 2014.
  * [101] A. Koessler, N. R. Filella, B.-C. Bouzgarrou, L. Lequièvre, and J.-A. C. Ramon, “An efficient approach to closed-loop shape control of deformable objects using finite element models,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 1637–1643.
  * [102] S. Duenser, J. M. Bern, R. Poranne, and S. Coros, “Interactive robotic manipulation of elastic objects,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 3476–3481.
  * [103] M. Cusumano-Towner, A. Singh, S. Miller, J. F. O’Brien, and P. Abbeel, “Bringing clothing into desired configurations with limited perception,” in _2011 IEEE international conference on robotics and automation_. IEEE, 2011, pp. 3893–3900.
  * [104] D. T. Chen and D. Zeltzer, “Pump it up: Computer animation of a biomechanically based model of muscle using the finite element method,” in _Proceedings of the 19th annual conference on computer graphics and interactive techniques_ , 1992, pp. 89–98.
  * [105] E. Yoshida, K. Ayusawa, I. G. Ramirez-Alpizar, K. Harada, C. Duriez, and A. Kheddar, “Simulation-based optimal motion planning for deformable object,” in _2015 IEEE international workshop on advanced robotics and its social impacts (ARSO)_. IEEE, 2015, pp. 1–6.
  * [106] S. Kinio and A. Patriciu, “A comparative study of h∞ and pid control for indirect deformable object manipulation,” in _2012 IEEE International Conference on Robotics and Biomimetics (ROBIO)_. IEEE, 2012, pp. 414–420.
  * [107] Y. Li, A. Torralba, A. Anandkumar, D. Fox, and A. Garg, “Causal discovery in physical systems from videos,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 9180–9192, 2020.
  * [108] Z. Huang, X. Lin, and D. Held, “Mesh-based dynamics with occlusion reasoning for cloth manipulation,” _arXiv preprint arXiv:2206.02881_ , 2022.
  * [109] Y. Huang, C. Xia, X. Wang, and B. Liang, “Learning graph dynamics with external contact for deformable linear objects shape control,” _IEEE Robotics and Automation Letters_ , 2023.
  * [110] Y. Li, Y. Yue, D. Xu, E. Grinspun, and P. K. Allen, “Folding deformable objects using predictive simulation and trajectory optimization,” in _2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2015, pp. 6000–6006.
  * [111] H. Lin, F. Guo, F. Wang, and Y.-B. Jia, “Picking up a soft 3d object by “feeling” the grip,” _The International Journal of Robotics Research_ , vol. 34, no. 11, pp. 1361–1384, 2015.
  * [112] A. Sintov, S. Macenski, A. Borum, and T. Bretl, “Motion planning for dual-arm manipulation of elastic rods,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 6065–6072, 2020.
  * [113] W. H. Lui and A. Saxena, “Tangled: Learning to untangle ropes with rgb-d perception. in 2013 ieee/rsj int. conf. on intelligent robots and systems,” 2013.
  * [114] T. Bretl and Z. McCarthy, “Quasi-static manipulation of a kirchhoff elastic rod based on a geometric analysis of equilibrium configurations,” _The International Journal of Robotics Research_ , vol. 33, no. 1, pp. 48–68, 2014.
  * [115] A. J. Shah and J. A. Shah, “Towards manipulation planning for multiple interlinked deformable linear objects,” in _2016 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2016, pp. 3908–3915.
  * [116] D. Berenson, “Manipulation of deformable objects without modeling and simulating deformation,” in _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2013, pp. 4525–4532.
  * [117] J. Das and N. Sarkar, “Planning and control of an internal point of a deformable object,” in _2010 IEEE International Conference on Robotics and Automation_. IEEE, 2010, pp. 2877–2882.
  * [118] D. Navarro-Alarcon and Y.-H. Liu, “Fourier-based shape servoing: A new feedback method to actively deform soft objects into desired 2-d image contours,” _IEEE Transactions on Robotics_ , vol. 34, no. 1, pp. 272–279, 2017.
  * [119] M. Ruan, D. McConachie, and D. Berenson, “Accounting for directional rigidity and constraints in control for manipulation of deformable objects without physical simulation,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 512–519.
  * [120] F. Ficuciello, A. Migliozzi, E. Coevoet, A. Petit, and C. Duriez, “Fem-based deformation control for dexterous manipulation of 3d soft objects,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 4007–4013.
  * [121] D. Kruse, R. J. Radke, and J. T. Wen, “Collaborative human-robot manipulation of highly deformable materials,” in _2015 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2015, pp. 3782–3787.
  * [122] O. Roussel, A. Borum, M. Taix, and T. Bretl, “Manipulation planning with contacts for an extensible elastic rod by sampling on the submanifold of static equilibrium configurations,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. Ieee, 2015, pp. 3116–3121.
  * [123] N. Lv, J. Liu, and Y. Jia, “Dynamic modeling and control of deformable linear objects for single-arm and dual-arm robot manipulations,” _IEEE Transactions on Robotics_ , vol. 38, no. 4, pp. 2341–2353, 2022.
  * [124] Y. Wu, W. Yan, T. Kurutach, L. Pinto, and P. Abbeel, “Learning to manipulate deformable objects without demonstrations,” in _16th Robotics: Science and Systems, RSS 2020_. MIT Press Journals, 2020.
  * [125] A. Colomé and C. Torras, “Dimensionality reduction for dynamic movement primitives and application to bimanual manipulation of clothes,” _IEEE Transactions on Robotics_ , vol. 34, no. 3, pp. 602–615, 2018.
  * [126] T. Wang, X. Bao, I. Clavera, J. Hoang, Y. Wen, E. Langlois, S. Zhang, G. Zhang, P. Abbeel, and J. Ba, “Benchmarking model-based reinforcement learning,” _arXiv preprint arXiv:1907.02057_ , 2019.
  * [127] J. Lv, Y. Feng, C. Zhang, S. Zhao, L. Shao, and C. Lu, “Sam-rl: Sensing-aware model-based reinforcement learning via differentiable physics-based simulation and rendering,” _arXiv preprint arXiv:2210.15185_ , 2022.
  * [128] Y. Yang, J. A. Stork, and T. Stoyanov, “Online model learning for shape control of deformable linear objects,” in _2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2022, pp. 4056–4062.
  * [129] Y. Tsurumine, Y. Cui, E. Uchibe, and T. Matsubara, “Deep reinforcement learning with smooth policy update: Application to robotic cloth manipulation,” _Robotics and Autonomous Systems_ , vol. 112, pp. 72–83, 2019.
  * [130] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-real transfer of robotic control with dynamics randomization,” in _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2018, pp. 3803–3810.
  * [131] J. Matas, S. James, and A. J. Davison, “Sim-to-real reinforcement learning for deformable object manipulation,” in _Conference on Robot Learning_. PMLR, 2018, pp. 734–743.
  * [132] T. Osa, J. Pajarinen, G. Neumann, J. A. Bagnell, P. Abbeel, J. Peters _et al._ , “An algorithmic perspective on imitation learning,” _Foundations and Trends® in Robotics_ , vol. 7, no. 1-2, pp. 1–179, 2018.
  * [133] C. G. Rivera, D. A. Handelman, C. R. Ratto, D. Patrone, and B. L. Paulhamus, “Visual goal-directed meta-imitation learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 3767–3773.
  * [134] A. Wu, A. Piergiovanni, and M. S. Ryoo, “Action-conditioned convolutional future regression models for robot imitation learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , 2018, pp. 2035–2037.
  * [135] S. Kudoh, T. Gomi, R. Katano, T. Tomizawa, and T. Suehiro, “In-air knotting of rope by a dual-arm multi-finger robot,” in _2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2015, pp. 6202–6207.
  * [136] A. X. Lee, H. Lu, A. Gupta, S. Levine, and P. Abbeel, “Learning force-based manipulation of deformable objects from multiple demonstrations,” in _2015 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2015, pp. 177–184.
  * [137] J. Schulman, J. Ho, C. Lee, and P. Abbeel, “Learning from demonstrations through the use of non-rigid registration,” in _Robotics Research: The 16th International Symposium ISRR_. Springer, 2016, pp. 339–354.
  * [138] J. Schulman, A. Gupta, S. Venkatesan, M. Tayson-Frederick, and P. Abbeel, “A case study of trajectory transfer through non-rigid registration for a simplified suturing scenario,” in _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2013, pp. 4111–4117.
  * [139] E. Pignat and S. Calinon, “Learning adaptive dressing assistance from human demonstration,” _Robotics and Autonomous Systems_ , vol. 93, pp. 61–75, 2017.
  * [140] P.-C. Yang, K. Sasaki, K. Suzuki, K. Kase, S. Sugano, and T. Ogata, “Repeatable folding task by humanoid robot worker using deep learning,” _IEEE Robotics and Automation Letters_ , vol. 2, no. 2, pp. 397–403, 2016.
  * [141] A. Cherubini, J. Leitner, V. Ortenzi, and P. Corke, “Towards vision-based manipulation of plastic materials,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 485–490.
  * [142] D. Seita, A. Ganapathi, R. Hoque, M. Hwang, E. Cen, A. K. Tanwani, A. Balakrishna, B. Thananjeyan, J. Ichnowski, N. Jamali _et al._ , “Deep imitation learning of sequential fabric smoothing from an algorithmic supervisor,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 9651–9658.
  * [143] H. Ha and S. Song, “Flingbot: The unreasonable effectiveness of dynamic manipulation for cloth unfolding,” in _Conference on Robot Learning_. PMLR, 2022, pp. 24–33.
  * [144] A. Simeonov, Y. Du, A. Tagliasacchi, J. B. Tenenbaum, A. Rodriguez, P. Agrawal, and V. Sitzmann, “Neural descriptor fields: Se (3)-equivariant object representations for manipulation,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 6394–6400.
  * [145] B. Wen, W. Lian, K. Bekris, and S. Schaal, “You only demonstrate once: Category-level manipulation from single visual demonstration,” _arXiv preprint arXiv:2201.12716_ , 2022.
  * [146] Y. Ren, R. Chen, and Y. Cong, “Autonomous manipulation learning for similar deformable objects via only one demonstration,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2023, pp. 17 069–17 078.
  * [147] Y. Chen, C. Wang, L. Fei-Fei, and C. K. Liu, “Sequential dexterity: Chaining dexterous policies for long-horizon manipulation,” _arXiv preprint arXiv:2309.00987_ , 2023.
  * [148] C. Garrett, T. Lozano-Pérez, and L. Kaelbling, “Sample-based methods for factored task and motion planning,” 2017.
  * [149] B. Kim and L. Shimanuki, “Learning value functions with relational state representations for guiding task-and-motion planning,” in _Conference on Robot Learning_. PMLR, 2020, pp. 955–968.
  * [150] A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian _et al._ , “Do as i can, not as i say: Grounding language in robotic affordances,” in _Conference on Robot Learning_. PMLR, 2023, pp. 287–318.
  * [151] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch, “Language models as zero-shot planners: Extracting actionable knowledge for embodied agents,” in _International Conference on Machine Learning_. PMLR, 2022, pp. 9118–9147.
  * [152] X. Lin, Z. Huang, Y. Li, J. B. Tenenbaum, D. Held, and C. Gan, “Diffskill: Skill abstraction from differentiable physics for deformable object manipulations with tools,” in _International Conference on Learning Representations (ICLR)_ , 2022.
  * [153] X. Lin, C. Qi, Y. Zhang, Z. Huang, K. Fragkiadaki, Y. Li, C. Gan, and D. Held, “Planning with spatial-temporal abstraction from point clouds for deformable object manipulation,” in _Conference on Robot Learning (CoRL)_ , 2022.
  * [154] J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, B. Yin, and X. Hu, “Harnessing the power of llms in practice: A survey on chatgpt and beyond,” _arXiv preprint arXiv:2304.13712_ , 2023.
  * [155] W. X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong _et al._ , “A survey of large language models,” _arXiv preprint arXiv:2303.18223_ , 2023.
  * [156] J. Wu, R. Antonova, A. Kan, M. Lepert, A. Zeng, S. Song, J. Bohg, S. Rusinkiewicz, and T. Funkhouser, “Tidybot: Personalized robot assistance with large language models,” _arXiv preprint arXiv:2305.05658_ , 2023.
  * [157] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann _et al._ , “Palm: Scaling language modeling with pathways,” _arXiv preprint arXiv:2204.02311_ , 2022.
  * [158] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill _et al._ , “On the opportunities and risks of foundation models,” _arXiv preprint arXiv:2108.07258_ , 2021.
  * [159] S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, “Robots that use language,” _Annual Review of Control, Robotics, and Autonomous Systems_ , vol. 3, pp. 25–55, 2020.
  * [160] A. Zeng, M. Attarian, B. Ichter, K. Choromanski, A. Wong, S. Welker, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani _et al._ , “Socratic models: Composing zero-shot multimodal reasoning with language,” _arXiv preprint arXiv:2204.00598_ , 2022.
  * [161] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng, “Code as policies: Language model programs for embodied control,” in _2023 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2023, pp. 9493–9500.
  * [162] M. Kwon, S. M. Xie, K. Bullard, and D. Sadigh, “Reward design with language models,” _arXiv preprint arXiv:2303.00001_ , 2023.
  * [163] Y. Du, O. Watkins, Z. Wang, C. Colas, T. Darrell, P. Abbeel, A. Gupta, and J. Andreas, “Guiding pretraining in reinforcement learning with large language models,” _arXiv preprint arXiv:2302.06692_ , 2023.
  * [164] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and A. Anandkumar, “Eureka: Human-level reward design via coding large language models,” _arXiv preprint arXiv:2310.12931_ , 2023.
  * [165] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, J. Humplik _et al._ , “Language to rewards for robotic skill synthesis,” _arXiv preprint arXiv:2306.08647_ , 2023.