# Learning Dense Visual Correspondences
in Simulation to Smooth and Fold Real Fabrics

Aditya Ganapathi1, Priya Sundaresan1, Brijen Thananjeyan1, Ashwin
Balakrishna1,
Daniel Seita1, Jennifer Grannen1, Minho Hwang1, Ryan Hoque1,
Joseph E. Gonzalez1, Nawid Jamali2, Katsu Yamane2, Soshi Iba2, Ken Goldberg1
1AUTOLab at the University of California, Berkeley, USA2Honda Research
Institute, USACorrespondence to Aditya Ganapathi<EMAIL_ADDRESS>
(112001)

###### Abstract

Robotic fabric manipulation is challenging due to the infinite dimensional
configuration space, self-occlusion, and complex dynamics of fabrics. There
has been significant prior work on learning policies for specific deformable
manipulation tasks, but comparatively less focus on algorithms which can
efficiently learn many different tasks. In this paper, we learn visual
correspondences for deformable fabrics across different configurations in
simulation and show that this representation can be used to design policies
for a variety of tasks. Given a single demonstration of a new task from an
initial fabric configuration, the learned correspondences can be used to
compute geometrically equivalent actions in a new fabric configuration. This
makes it possible to robustly imitate a broad set of multi-step fabric
smoothing and folding tasks on multiple physical robotic systems. The
resulting policies achieve $80.3\%$ average task success rate across 10 fabric
manipulation tasks on two different robotic systems, the da Vinci surgical
robot and the ABB YuMi. Results also suggest robustness to fabrics of various
colors, sizes, and shapes. See https://tinyurl.com/fabric-descriptors for
supplementary material and videos.

##  I Introduction

Robot fabric manipulation has applications in folding laundry [46, 17, 4, 25],
bed making [37], surgery [43, 42, 38, 8], and manufacturing [27, 45]. However,
while robots are able to learn general purpose policies to manipulate a
variety of rigid objects with increasing reliability [22, 6, 28, 14, 20],
learning such policies for manipulating deformable objects remains an open
problem due to difficulties in sensing and control. While there is significant
prior work on geometric [35, 46, 1, 23] and learning based approaches [47, 36,
37] for fabric manipulation, these approaches often involve designing or
learning task-specific manipulation policies, making it difficult to
efficiently reuse information for different tasks.

In this work, we leverage recent advances in dense keypoint learning [6] to
learn visual point-pair correspondences across fabric in different
configurations. Then, given a single offline demonstration of a fabric
manipulation task from a given configuration, we utilize the learned
correspondences to compute geometrically equivalent actions to complete the
task on a similar fabric in a different configuration. For example, a human
might provide a sequence of actions that would fold a T-shirt when it is
placed neck up in a smoothed configuration. However, when a robot is operating
at test time, it will likely encounter a different T-shirt whose color, size
and pose may differ from the T-shirt used for the demonstration. We find that
learning visual correspondences that are invariant across these fabric
attributes provides a powerful representation for defining controllers that
can generalize to the above variations.

Figure 1:  We use learned visual correspondences across different fabric
configurations to perform a variety of fabric manipulation tasks on the ABB
YuMi (top) and the da Vinci Research Kit (bottom). Given a single
demonstration of smoothing or folding, the robot uses the learned
correspondences to compute geometrically equivalent actions for fabric of
different color and in different initial configurations. This enables robust
one-shot imitation learning of tasks that involve smoothing then folding.

We extend work by [41], which leverages dense object descriptors [6] to learn
visual correspondences for rope using synthetic depth data in simulation.
These correspondences are then used to learn new rope manipulation tasks such
as rearrangement or knot tying given a single task demonstration. We find that
similar visual correspondence learning methods are also effective for learning
correspondences between different fabric configurations using task-agnostic
RGB data collected entirely in simulation and can be used to perform fabric
manipulation tasks. Precisely, given a user demonstration of the task from a
given initial fabric configuration, we leverage the learned visual
correspondences to perform the same task from different initial configurations
by computing geometrically equivalent actions using the correspondences. This
approach has a number of appealing properties. First, visual correspondences
can be learned purely in simulation without task-specific data and widely
applied to a variety of real fabric manipulation tasks with no further
training. Second, training in simulation enables sufficient data variety
through domain randomization, making it possible to learn correspondences that
generalize to fabrics with different colors, shapes, and configurations.
Third, since perception and control are decoupled, the same perception module
can be used on different robots with no additional training.

We contribute (1) a framework for learning dense visual correspondences of
fabric in simulation using dense object descriptors from [6, 41] and applying
them to manipulation tasks on real fabrics with unseen colors, scales, and
textures, (2) a data generation pipeline for collecting images of fabrics and
clothing in Blender [3] and a testbed to experiment with different
manipulation policies on these fabrics in simulation and (3) physical
experiments on both the da Vinci Research Kit (dVRK) [15] and the ABB YuMi
suggesting that the learned descriptors transfer effectively on two different
robotic systems. We experimentally validate the method on 10 different tasks
involving 5 T-shirts and 5 square fabrics of varying dimensions and colors and
achieve an average task success rate of $80.3\%$.

##  II Related Work

Fabric manipulation is an active area of robotics research [2, 33, 18, 11].
Over the past decade, the research has primarily been focused on three
different categories: perception-based manipulation, learning-based algorithms
in the real world, and learning-based algorithms in simulation which are then
transferred to real robots.

Traditional Vision-Based Algorithms for Fabric Manipulation: Much of the prior
work on perception-based deformable object manipulation relies on traditional
image processing and vision techniques to estimate the state of the fabric.
This state estimation is then used to define geometric controllers which bring
the fabric into some desired configuration. However, due to limitations in
these traditional vision algorithms, most prior work makes specific
assumptions on the fabric’s initial configurations or requires more complex
robotic manipulators to bring the fabric into a desired starting
configuration. For example, Miller  _et al_. [25] demonstrate a robust folding
pipeline for clothing by fitting a polygonal contour to the fabric and
designing a geometric controller on top of it, but assume that the initial
state of the fabric is flat. Sun  _et al_. [39, 40] perform effective fabric
smoothing by estimating the wrinkles in the fabric, but condition on a near-
flat starting fabric. Other work relies on “vertically smoothing” fabrics
using gravity [26, 16, 17, 4, 23] to standardize the initial configuration and
to expose fabric corners before attempting the task, which is difficult for
large fabrics or single-armed robots.

Learning-Based Algorithms in the Real World: More recent approaches have
transitioned to end to end learning of fabric manipulation directly on a real
system, but these approaches have struggled to generalize to a variety of
fabrics and tasks due to the high volume of training data required. For
example, [5] use model-based reinforcement learning to learn fabric
manipulation policies which generalize to many tasks, but require several days
of continuous data collection on a real physical system and perform relatively
low precision tasks. Jia  _et al_. [12, 13] show impressive collaborative
human-robot cloth folding under the assumption that fabric has already been
grasped and is in a particular starting configuration, and [35] demonstrate
deformable object manipulation while requiring task-specific kinesthetic
demonstrations. In follow-up work, [19] consider many of the same tasks as in
this paper and demonstrate that policies can be learned to fold fabric using
reinforcement learning with only one hour of experience on a real robot. In
contrast, we learn entirely in simulation and decouple perception from
control, making it easier to generalize to different fabric colors and shapes
and flexibly deploy the learned policies on different robots.

Sim-to-Real Learning-Based Algorithms: Due to the recent success of sim-to-
real transfer [32, 44], many recent papers leverage simulation to collect
large amounts of training data which is used to learn fabric manipulation
policies. Seita  _et al_. [37, 36] and [47] followed-up on the smoothing task
from [39] by generalizing to a wider range of initial fabric states using
imitation learning (DAgger [31]), and reinforcement learning (Soft Actor-
Critic [9]) respectively, but still tailor policies specifically for
smoothing. Similarly, [24] learn fabric folding policies by using deep
reinforcement learning augmented with task-specific demonstrations. These
works use simulation to optimize fabric manipulation policies for specific
tasks. In follow-up and concurrent work, [10] and [48] use simulation to train
fabric manipulation policies using model-based reinforcement learning for
multiple tasks. In contrast, we leverage simulation to learn visual
representations of fabric to capture its geometric structure without task-
specific data or a model of the environment and then use this representation
to design intuitive controllers for several tasks from different starting
configurations.

Dense Object Descriptors: We learn visual representations for fabric by using
dense object descriptors [6, 34], which were shown to enable task oriented
manipulation of various rigid and slightly deformable objects [6]. This
approach uses a deep neural network to learn a representation which encourages
corresponding pixels in images of an object in different configurations to
have similar representations in embedding space. Such descriptors can be used
to design geometrically structured manipulation policies for grasping [6],
assembly [49], or for learning from demonstrations [7]. [41] extend this idea
to manipulation of ropes, and demonstrate that deformation-invariant dense
object descriptors can be learned for rope using synthetic depth data in
simulation and then transferred to a real physical system. [41] then use the
learned descriptors to imitate offline demonstrations of various rope
manipulation tasks. In this work, we apply the techniques from [41] to learn
descriptors which capture geometric correspondence across different fabric
configurations from synthetic RGB images and use them for 2D fabric
manipulation.

##  III Problem Definition

###  III-A Assumptions

We assume a deformable object is place on a planar workspace in an initial
configuration $\xi_{1}$ observed by an overhead camera corresponding RGB image
$I_{1}:=I_{1}(\xi_{1})\in\mathbb{R}^{W\times H\times 3}$. As in prior work
[36, 47], we assume that fabric manipulation tasks can be completed by a
sequence of actions where each includes grasping at a _pick point_ , pulling
to a _place point_ without changing the orientation of the end-effector, and
releasing at the place point. We additionally assume access to a single
demonstration of each task in the form of a sequence of pick and place actions
from some arbitrary initial fabric configuration $\xi_{1}$. These
demonstrations can be collected offline, such as through a GUI where a user
clicks on an image of fabric to indicate pick and place point pixels. However,
the fabric used to create the instruction does not have to be of the same
color, the same size or in the same initial configuration as the fabric the
robot sees at test time. The only requirement is that the fabric be of a
similar geometry. For example, T-shirts can be compared to other instances of
T-shirts, but not to pants or long-sleeved shirts.

###  III-B Task Definition

Define the action at step $j$ as

$\mathbf{a}_{j}=((x_{g},y_{g})_{j},(x_{p},y_{p})_{j})$ (III.1)

where $(x_{g},y_{g})_{j}$ and $(x_{p},y_{p})_{j}$ are the pixel coordinates of
a grasp point on the fabric and place point respectively in image $I_{j}$ at
time $j$. The robot grasps the world coordinate associated with the grasp
point and then moves to the world coordinate associated with the place point
without changing the end effector orientation. This causes the fabric located
at $(x_{g},y_{g})_{j}$ in the image to be placed on top of the world
coordinate associated with $(x_{p},y_{p})_{j}$ with the same surface normals
as before. In future work, we will investigate how to execute more complex
actions that result in reversed surface normals, which requires a rotation
motion during the action. We are given a sequence of actions
$\left(\mathbf{a}_{j}\right)_{j=1}^{n}$ executed on a fabric starting in
configuration $\xi_{1}$ and corresponding observations
$\left(I_{j}\right)_{j=1}^{n}$. Then at test-time, a similar object is dropped
onto the surface in a previously unseen configuration and the goal is to
generate a corresponding sequence of actions for a fabric in some previously
unseen configuration. Specifically, the robot generates a new sequence of
actions:

$\Big{(}\mathbf{a}_{j}^{\prime}\Big{)}_{j=1}^{n}=\Big{(}d_{I_{j}\rightarrow
I_{j}^{\prime}}(x_{g},y_{g})_{j},\;\;d_{I_{j}\rightarrow
I_{j}^{\prime}}(x_{p},y_{p})_{j}\Big{)}_{j=1}^{n}$ (III.2)

for $j\in\\{1,\ldots,n\\}$ where $d_{I_{j}\rightarrow
I_{j}^{\prime}}:\mathbb{R}^{2}\rightarrow\mathbb{R}^{2}$ is a function which
estimates the corresponding point $(x^{\prime},y^{\prime})_{j}$ in
$I_{j}^{\prime}$ given a point $(x,y)_{j}$ in $I_{j}$. This function is
difficult to compute directly from images in general, and even more so
difficult to compute for images of highly deformable objects due to their
infinite degrees of freedom and tendency to self-occlude. Thus, we leverage
dense object descriptors [6] to approximate $d_{I_{j}\rightarrow
I_{j}^{\prime}}$ for any $I_{j}$ and $I_{j}^{\prime}$, as described in
Sections V and VI.

##  IV Simulator

We use Blender 2.8, an open-source simulation and rendering engine [3]
released in mid-2019, to both create large synthetic RGB training datasets and
model the fabric dynamics for simulated experiments using its in-built fabric
solver based on [30, 29]. We simulate T-shirts and square fabrics, each of
which we model as a polygonal mesh made up of 729 vertices, a square number we
experimentally tuned to trade-off between fine-grained deformations and
reasonable simulation speed. These meshes can be easily constructed in Blender
by starting with a planar grid of vertices and then removing vertices and
edges to create desired shapes. See Figure 2 for an illustration. Each vertex
on the mesh has a global coordinate which we can query directly through
Blender’s API, allowing for easily available ground truth information about
various locations on the mesh and their pixel counterparts. See Figure 2 in
the supplement for examples of these meshes. We can also simulate finer-
grained manipulation of the mesh including grasps, pulls, and folds. To
implement the action space defined in Section III, we first deproject the
pixel corresponding to the pick point and map it to the vertex whose global
coordinates are closest in $\mathbb{R}^{3}$ to the pixel’s deprojected
coordinates. We then directly manipulate this vertex by pinning it and
translating it over a sequence of 30 frames. This generalized form of
manipulation allows us to easily execute experiments in simulation. See the
supplement for further details on how we pin vertices, the overall technique
and experiments in simulation.

##  V Dense Shape Descriptor Training

###  V-A Dense Object Descriptor Training Procedure

We consider an environment with a deformable fabric on a flat tabletop and
learn policies that perform smoothing and folding tasks. The policies we train
use point-pair correspondences that are generated between overhead images of
the fabric in different configurations. We generate deformation-invariant
correspondences by training dense object descriptors [41, 6] on synthetically
generated images of the fabric in different configurations.

Figure 2:  Fabric Meshes: Examples of the meshes generated in Blender for both
square cloth (left) and t-shirts (right). The ground-truth vertices are
highlighted in the second and fourth columns.

In [6], an input image $I$ is mapped to a descriptor volume
$Z=f_{\theta}(I)\in\mathbb{R}^{W\times H\times D}$, where each pixel $(i,j)$
has a corresponding descriptor vector $Z_{i,j}\in\mathbb{R}^{D}$. Descriptors
are generated by a Siamese network $f_{\theta}$ and are guided closer together
for corresponding pixels in images and pushed apart by at least some margin
$M$ for non-corresponding pairs by minimizing a pixel-wise contrastive loss
function during training [6]. Corresponding pairs of pixels represent the same
point on an object. In this work, we also train a Siamese network to output
descriptors that are close for corresponding pairs of pixels and separated for
non-corresponding pixel pairs for overhead images of fabric in different
configurations. Since ground-truth pixel correspondences are difficult to
obtain in images across deformations of a real fabric, we train the network on
synthetic RGB data from Blender (see Section IV), where perfect information
about the pixel correspondences is available through the global coordinates of
the fabric mesh’s vertices. Note that during training, the sampled image
inputs to the Siamese network are of the same fabric type to ensure valid
correspondences. That is, two different images of T-shirts can be passed into
the network, but not a T-shirt and square fabric. Figure 3 demonstrates the
pipeline for predicting descriptors for correspondence generation. The learned
descriptors can then be used to approximate the correspondence function
$d_{I\rightarrow I^{\prime}}$ described in Section III:

$\displaystyle\left((i^{\prime\prime}_{l},j^{\prime\prime}_{l})\right)_{l=1}^{k}$
$\displaystyle=\operatorname*{arg\,min}_{(i^{\prime}_{1},j^{\prime}_{1})...(i^{\prime}_{k},j^{\prime}_{k})}\sum_{l=1}^{k}\lVert
f_{\theta}(I)_{i,j}-f_{\theta}(I^{\prime})_{i^{\prime}_{l},j^{\prime}_{l}}\rVert_{2}$
$\displaystyle\text{s.t.
}(i^{\prime}_{n},j^{\prime}_{n})\neq(i^{\prime}_{m},j^{\prime}_{m})\;\forall
m,n\in[k]$ $\displaystyle d_{I\rightarrow I^{\prime}}(i,j)$
$\displaystyle=\operatorname*{arg\,min}_{(i^{\prime},j^{\prime})}\sum_{l=1}^{k}\lVert(i^{\prime},j^{\prime})-(i^{\prime\prime}_{l},j^{\prime\prime}_{l})\rVert_{2}$

The first equation computes the top $k$ pixel matches based on their distance
in descriptor space and the second equation computes the geometric median of
these matches in pixel space. In the physical setup, we experimentally found
$k=20$ to give us the most robust predictions.

Figure 3:  Learning Visual Correspondences: pipeline for training dense object
nets for robot fabric manipulation. Left: we train a dense correspondence
network on pairs of simulated fabric images to learn pixel-wise
correspondences using a pixel-wise contrastive loss. Right: we use the learned
descriptors for policy optimization. We can use correspondence to map a
reference action to a new fabric configuration. For example, we show an image
of a wrinkled fabric in “State 2,” and we can use descriptors to figure out
the action needed to smooth the fabric from “State 2” to “State 1.”

###  V-B Dataset Generation and Domain Randomization

To enable generalization of the learned descriptors to a range of fabric
manipulation tasks, we generate a diverse dataset of initial fabric
configurations. The first step simulates dropping the fabric onto the planar
workspace while executing similar pinning actions to those described in
Section IV on an arbitrary subset of vertices, causing some vertices to fall
due to gravity while others stay fixed. We then release the pinned vertices 30
frames later so that they collapse on top of the fabric. This allows us to
create realistic deformations in the mesh. We then export RGB images which
serve as inputs to the Siamese network, pixel-wise annotations which gives us
correspondences, and segmentation masks which allow us to sample matches on
the fabric.

Simulating soft-body animations is in general a computationally time-consuming
process which makes it difficult to render large datasets in short periods of
time. We take steps toward mitigating this issue by rendering 10 images per
drop, allowing us to collect 10x as much data in the same time period. In
simulation, we found that the test time pixel match error was unaffected when
including these unsettled images of the fabric in the dataset. We additionally
make use of domain randomization [32, 44] by rendering images of the scene
while randomizing parameters including mesh size, lighting, camera pose,
texture, color and specularity (see supplement for further details). We also
restrict the rotation about the z-axis to be between $(-\pi/4,\pi/4)$ radians
to reduce ambiguity during descriptor training due to the natural symmetry of
fabrics such as squares. To randomize the image background, we sample an image
from MSCOCO [21] and “paste” the rendered fabric mask on top. For our
experiments, we generated one (domain-randomized) dataset, including both
T-shirts and square fabric, and train a single model which we used for the
experiments in Section VII. For reference, generating a single dataset of
7,500 images, half T-shirts and half square cloth, with 729 annotations per
image takes approximately 2 hours on a 2.6GHz 6-core Intel Core i7 MacBook
Pro.

##  VI Descriptor-Parameterized Controller

As discussed in Section III-B, the robot receives a demonstration of the task
consisting of actions $\left(\mathbf{a}_{j}\right)_{j=1}^{n}$ and observations
$\left(I_{j}\right)_{j=1}^{n}$. At execution time, the robot starts with the
fabric in a different configuration, and the fabric itself may have a
different texture or color. At time $j\in[n]$, the robot observes
$I^{\prime}_{j}$ then executes
$\pi_{j}(I_{j}^{\prime})=\left(d_{I_{j}\rightarrow
I_{j}^{\prime}}(x_{g},y_{g})_{j},\;\;d_{I_{j}\rightarrow
I_{j}^{\prime}}(x_{p},y_{p})_{j}\right)$ where $d_{I_{j}\rightarrow
I_{j}^{\prime}}$ is defined in Section V-A. $\pi_{j}$ uses the geometric
structure learned by the descriptor network to identify semantically relevant
pixels in $I_{j}^{\prime}$ to generate actions that manipulate these
keypoints. Because the descriptor network learns about task-agnostic fabric
geometry, we train one network for a variety of tasks and use it across
different fabric configurations from the ones supplied in demonstrations.
Additionally, we do not require separate networks for each fabric type we wish
to train.

For example, one step of a task could involve grasping the top-right corner of
the fabric and taking an action to place it in alignment with the bottom-left
corner, thereby folding the fabric. The robot could receive an offline
demonstration of this task on an initially flat fabric, but then be asked to
perform the same task on a crumpled, rotated fabric. To do this, the robot
must be able to identify the corresponding points in the new fabric
configuration (top-right and bottom-left corners) and define a new action to
align them. $\pi_{j}$ computes correspondences for the grasp and place points
across the demonstration frame and the new observation to generate a
corresponding action for the new configuration.

###  VI-A Fabric Smoothing

In the square fabric smoothing task, the robot starts with a crumpled fabric
and spreads it into a smooth configuration on a planar workspace as in [36].
To complete this task, we use the approach from [36] and iterate over fabric
corners, pulling each one to their target locations on an underlying plane.
However, while [36] design a policy to do this using ground-truth knowledge of
the fabric in simulation, we alternatively locate corners on the crumpled
fabric using a learned descriptor network and a source image of a flat fabric
where the corners are labeled. For the T-shirt smoothing task, we apply a
similar method but instead iterate over the corners of the sleeves and the
base of the T-shirt.

###  VI-B Fabric Folding

The fabric folding task involves executing a sequence of folds on a fairly
smooth starting configuration. For each folding task, we use a single offline
demonstration containing up to 4 pick and place actions, each defined by a
pick and drop pixel location and collected by a human via a GUI. The
descriptor-parameterized controller is then executed in an open-loop manner.

Figure 4:  Fabric Specifications: Images and dimensions of the square fabrics
and shirts we use in experiments.

##  VII Experiments

We experimentally evaluate (1) the quality of the learned descriptors and
their sensitivity to training parameters and (2) the performance of the
descriptor-based policies from Section VI on two physical robotic systems, the
da Vinci Research Kit (dVRK) [15] and the ABB YuMi. Results suggest that the
learned descriptors and the resulting policies are robust to changes in fabric
configuration and color.

###  VII-A Tasks

We consider 10 fabric manipulation tasks executed on a set of 5 T-shirts and 5
square fabrics in the real world:

  1. 1.

Single Fold (SF): A single fold where one corner is pulled to its opposing
corner.

  2. 2.

Double Inward Fold (DIF): Two opposing corners are folded to the center of the
fabric.

  3. 3.

Double Triangle Fold (DTF): Two sets of opposing corners are aligned with each
other.

  4. 4.

Double Straight Fold (DSF): The square cloth is folded in half twice, first
along the horizontal bisector and then along the vertical bisector.

  5. 5.

Four Corners Inward Fold (FCIF): All four corners are sequentially folded to
the center of the cloth.

  6. 6.

T-Shirt Sleeves Fold (TSF): The two sleeves of a t-shirt are folded to the
center of the shirt.

  7. 7.

T-Shirt Sleeve to Sleeve Fold (TSTSF): The left sleeve of a T-shirt is folded
to the right sleeve of the T-shirt.

  8. 8.

Smoothing (S): Fabric is flattened from a crumpled state.

  9. 9.

Smoothing + Double Triangle Fold (SDTF): Fabric is smoothed then the DTF is
executed.

  10. 10.

Smoothing + Sleeve to Sleeve Fold (SSTSF): T-shirt is smoothed then TSTSF is
executed.

All fabrics are varied either in dimension or color according to Figure 4.
Additionally, we execute a subset of these tasks in simulation. A single
visual demonstration consisting of up to 4 actions is provided to generate a
policy which the robot then tries to emulate in the same number of actions.

Figure 5:  Policy Rollouts:  We visualize policy execution on the YuMi for
tasks 2, 3, 4, 5, 6 and 7 as described in Section VII-A. The first four
columns show the folding instructions on some initial fabric and the last four
columns show the corresponding folds executed on novel starting configurations
for a different fabric.  Figure 6:  Full Folding Sequence: The first and
second row is a time-lapse of a sequence of 6 actions taken by the YuMi and
dVRK respectively, and with actions overlaid by red arrows, to successively
smooth a wrinkled fabric and then fold it according to task 3 in Section
VII-A. The third row is a time-lapse of a sequence of 5 actions taken by the
YuMi to complete task 10 in Section VII-A. Here, robot actions are overlaid
with blue arrows.

###  VII-B Experimental Setup

We execute fabric folding and smoothing experiments on the dVRK [15] and ABB
YuMi robot. The dVRK is equipped with the Zivid OnePlus RGBD sensor that
outputs $1900\times 1200$ pixel images at 13 FPS at depth resolution $0.5$ mm.
The workspace of the dVRK is only $5"\times 5"$, so we use only square fabric
of the same dimension while varying the color according to Figure 4.
Manipulating small pieces of fabric into folds is challenging due to the
elasticity of the fabric, so we add weight to the fabric by dampening it with
water. Additionally, we place a layer of 1 inch foam rubber below the fabric
to avoid damaging the gripper. The YuMi has a $36"\times 24"$ workspace, and
since only one arm is utilized resulting in a more limited range of motion, we
only manipulate at most $12"\times 12"$ pieces of fabric which we do not
dampen. In this setup we use a 1080p Logitech webcam to collect overhead color
images. For the YuMi, we use both T-shirts and square fabric of varying
dimension and color but go no lower than $9"\times 9"$ fabrics due to its
larger gripper. Finally, for both robots, we use a standard pixel to world
calibration procedure to get the transformation from pixel coordinates to
planar workspace coordinates.

For both robots, we follow the same experimental protocol. We manually place
the fabric in configurations similar to those shown in Figure 2 and deform
them by pulling at multiple locations on the fabric. To obtain image input for
the descriptor networks, we crop and resize the overhead image to be
$485\times 485$ such that the fabric is completely contained within the image.
Although lighting conditions, camera pose and workspace dimensions are
significantly different between the two robotic systems, no manual changes are
made to the physical setup. We find that the learned descriptors are
sufficiently robust to handle this environmental variability.

We evaluate the smoothing task by computing the coverage of the cropped
workspace before and after execution. For the folding tasks, as in [19], we
consider an outcome a success if the final state is visually consistent with
the goal image. Conventional quantitative metrics such as intersection of
union between the final state and a target image provide limited diagnostic
information when starting configurations are significantly different as in the
presented experiments.

###  VII-C Results

We evaluate the smoothing and folding policies on both the YuMi and dVRK on
square fabrics and T-shirts. Table II shows the success rates of our method on
all proposed tasks in addition to a breakdown of the failure cases detailed in
Table III. We observe that the descriptor-parameterized controller is able to
successfully complete almost all folding tasks at least $75\%$ of the time,
and the smoothing policies are able to increase coverage of the cloth to over
$83\%$ (Table I). The execution of the smoothing policy followed by the double
triangle folding policy results in successful task completion $6/10$ and
$8/10$ times on the YuMi and dVRK respectively. We find that the most frequent
failure mode is an unsuccessful grasp of the fabric which is compounded for
tasks that require more actions. Though this is independent of the quality of
the learned descriptors, it highlights the need for more robust methods to
grasp highly deformable objects.

Task | Robot | Avg. Start Coverage | Avg. End Coverage
---|---|---|---
S | YuMi | $71.4\pm 6.2$ | $83.2\pm 8.1$
S | dVRK | $68.4\pm 4.4$ | $86.4\pm 5.2$

Table I: Physical Fabric Smoothing Experiments: We test the smoothing policies
designed in Section VI on the YuMi and the dVRK robots. Both robots are able
to increase coverage during the smoothing task by $11-22$ percent on average.

Task | Robot | # Actions | Success | Error A | Error B | Error C
---|---|---|---|---|---|---
SF | YuMi | 1 | 18/20 | 2 | 0 | 0
SF | dVRK | 1 | 20/20 | 0 | 0 | 0
DIF | YuMi | 2 | 16/20 | 3 | 0 | 1
DIF | dVRK | 2 | 20/20 | 0 | 0 | 0
DTF | YuMi | 2 | 14/20 | 3 | 2 | 1
DTF | dVRK | 2 | 18/20 | 0 | 2 | 0
TSF | YuMi | 2 | 15/20 | 3 | 0 | 2
SDTF | YuMi | 6 | 6/10 | 2 | 1 | 1
SDTF | dVRK | 6 | 8/10 | 0 | 2 | 0
DSF | YuMi | 3 | 15/20 | 1 | 1 | 3
DSF | dVRK | 3 | 17/20 | 1 | 0 | 2
FCIF | YuMi | 4 | 13/20 | 5 | 1 | 1
FCIF | dVRK | 4 | 18/20 | 0 | 1 | 1
TSTSF | YuMi | 1 | 17/20 | 2 | 0 | 1
SSTSF | YuMi | 5 | 6/10 | 2 | 0 | 2

Table II: Physical Fabric Folding Experiments: We test the folding policies
from Section VI on the YuMi and the dVRK. We observe both robots are able to
perform almost all folding tasks at least $75$ percent of the time. The YuMi
is able to perform the smoothing then folding task $6/10$ times and the dVRK
is able to do so $8/10$ times.

Error |  Description
---|---
A |  Gripper picks up more than one layer of fabric or fabric slips out of gripper due to inaccurate depth of grasp
B |  Pick or drop correspondence error greater than 30 pixels (10% of cloth width) or pick correspondence not on fabric mask
C |  Unintended physics: resulting fold does not hold due to variable stiffness of the fabric, friction of the fabric, or friction of the underlying plane

Table III: Failure Mode Categorization

##  VIII Discussion and Future Work

We present an approach for multi-task fabric manipulation by using visual
correspondences learned entirely in simulation. Experiments suggest that the
learned correspondences are robust to different fabric colors, shapes,
textures, and sizes and make it possible to efficiently learn 10 different
fabric smoothing and folding tasks on two different physical robotic systems
with no training in the real world. In future work, we plan to explore
hierarchical fabric manipulation policies, where visual correspondences can be
used to define coarse action plans while a closed loop controller can be
learned to realize these plans. We will also explore more complex fabric
manipulation tasks, such as wrapping rigid objects, in which reasoning about
fabric dynamics is critical or tasks involving manipulating multiple fabrics
simultaneously.

## References

  * [1] Benjamin Balaguer and Stefano Carpin  “Combining Imitation and Reinforcement Learning to Fold Deformable Planar Objects”  In _Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2011
  * [2] Julia Borras, Guillem Alenya and Carme Torras  “A Grasping-centered Analysis for Cloth Manipulation”  In _arXiv:1906.08202_ , 2019
  * [3] Blender Online Community  “Blender - a 3D modelling and rendering package”, 2018  Blender Foundation  URL: http://www.blender.org
  * [4] Andreas Doumanoglou, Andreas Kargakos, Tae-Kyun Kim and Sotiris Malassiotis  “Autonomous Active Recognition and Unfolding of Clothes Using Random Decision Forests and Probabilistic Planning”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2014
  * [5] Frederik Ebert et al.  “Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control”  In _arXiv:1812.00568_ , 2018
  * [6] Peter R. Florence, Lucas Manuelli and Russ Tedrake  “Dense Object Nets: Learning Dense Visual Object Descriptors By and For Robotic Manipulation”  In _Conf. on Robot Learning (CoRL)_ , 2018
  * [7] Peter Florence, Lucas Manuelli and Russ Tedrake  “Self-Supervised Correspondence in Visuomotor Policy Learning”  In _IEEE Robotics & Automation Letters_, 2020
  * [8] Jennifer Grannen* et al.  “Learning Robot Policies for Untangling Dense Knots in Linear Deformable Structures”, https://goldberg.berkeley.edu/pubs/CoRL-2020-Untangling-submitted.pdf, 2020
  * [9] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel and Sergey Levine  “Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor”  In _Proc. Int. Conf. on Machine Learning (ICML)_ , 2018
  * [10] Ryan Hoque et al.  “VisuoSpatial Foresight for Multi-Step, Multi-Task Fabric Manipulation”  In _Proc. Robotics: Science and Systems (RSS)_ , 2020
  * [11] Rishabh Jangir, Guillem Alenya and Carme Torras  “Dynamic Cloth Manipulation with Deep Reinforcement Learning”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2020
  * [12] Biao Jia, Zhe Hu, Jia Pan and Dinesh Manocha  “Manipulating Highly Deformable Materials Using a Visual Feedback Dictionary”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2018
  * [13] Biao Jia et al.  “Cloth Manipulation Using Random-Forest-Based Imitation Learning”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2019
  * [14] Dmitry Kalashnikov et al.  “QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation”  In _Conf. on Robot Learning (CoRL)_ , 2018
  * [15] P Kazanzides et al.  “An Open-Source Research Kit for the da Vinci Surgical System”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2014
  * [16] Yasuyo Kita, Toshio Ueshiba, Ee Sian Neo and Nobuyuki Kita  “A Method For Handling a Specific Part of Clothing by Dual Arms”  In _Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2009
  * [17] Yasuyo Kita, Toshio Ueshiba, Ee Sian Neo and Nobuyuki Kita  “Clothes State Recognition Using 3D Observed Data”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2009
  * [18] Oliver Kroemer, Scott Niekum and George Konidaris  “A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms”  In _arXiv:1907.03146_ , 2019
  * [19] Robert Lee et al.  “Learning Arbitrary-Goal Fabric Folding with One Hour of Real Robot Experience”  In _Conf. on Robot Learning (CoRL)_ , 2020
  * [20] Sergey Levine et al.  “Learning hand-eye coordination for robotic grasping with deep learning and large-scale data collection”  In _The International Journal of Robotics Research_ 37.4-5, 2018, pp. 421–436  DOI: 10.1177/0278364917710318
  * [21] Tsung-Yi Lin et al.  “Microsoft COCO: Common Objects in Context”  In _arXiv:1405.0312_ , 2014
  * [22] Jeffrey Mahler et al.  “Learning Ambidextrous Robot Grasping Policies”  In _Science Robotics_ , 2019
  * [23] Jeremy Maitin-Shepard, Marco Cusumano-Towner, Jinna Lei and Pieter Abbeel  “Cloth Grasp Point Detection Based on Multiple-View Geometric Cues with Application to Robotic Towel Folding”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2010
  * [24] Jan Matas, Stephen James and Andrew J. Davison  “Sim-to-Real Reinforcement Learning for Deformable Object Manipulation”  In _Conf. on Robot Learning (CoRL)_ , 2018
  * [25] Stephen Miller et al.  “A Geometric Approach to Robotic Laundry Folding”  In _Int. Journal of Robotics Research (IJRR)_ , 2012
  * [26] Fumiaki Osawa, Hiroaki Seki and Yoshitsugu Kamiya  “Unfolding of Massive Laundry and Classification Types by Dual Manipulator”  In _Journal of Advanced Computational Intelligence and Intelligent Informatics_ 11.5, 2007
  * [27] J.. Parker, R. Dubey, F.. Paul and R.. Becker  “Robotic Fabric Handling for Automating Garment Manufacturing”  In _Journal of Manufacturing Science and Engineering_ 105, 1983
  * [28] Lerrel Pinto and Abhinav Gupta  “Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hours”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2016
  * [29] Xavier Provot  “Collision and self-collision handling in cloth model dedicated to design garments”  In _Computer Animation and Simulation’97_ Springer, 1997, pp. 177–189
  * [30] Xavier Provot  “Deformation constraints in a mass-spring model to describe rigid cloth behaviour”  In _Graphics interface_ , 1995, pp. 147–147  Canadian Information Processing Society
  * [31] Stephane Ross, Geoffrey J Gordon and J Andrew Bagnell  “A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning”  In _International Conference on Artificial Intelligence and Statistics (AISTATS)_ , 2011
  * [32] Fereshteh Sadeghi and Sergey Levine  “CAD2RL: Real Single-Image Flight without a Single Real Image”  In _Proc. Robotics: Science and Systems (RSS)_ , 2017
  * [33] Jose Sanchez, Juan-Antonio Corrales, Belhassen-Chedli Bouzgarrou and Youcef Mezouar  “Robotic Manipulation and Sensing of Deformable Objects in Domestic and Industrial Applications: a Survey”  In _Int. Journal of Robotics Research (IJRR)_ , 2018
  * [34] Tanner Schmidt, Richard A. Newcombe and Dieter Fox  “Self-Supervised Visual Descriptor Learning for Dense Correspondence”  In _IEEE Robotics and Automation Letters_ 2, 2017, pp. 420–427
  * [35] John Schulman, Jonathan Ho, Cameron Lee and Pieter Abbeel  “Learning from Demonstrations Through the Use of Non-Rigid Registration”  In _Int. S. Robotics Research (ISRR)_ , 2013
  * [36] Daniel Seita et al.  “Deep Imitation Learning of Sequential Fabric Smoothing From an Algorithmic Supervisor”  In _Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2020
  * [37] Daniel Seita et al.  “Deep Transfer Learning of Pick Points on Fabric for Robot Bed-Making”  In _Int. S. Robotics Research (ISRR)_ , 2019
  * [38] Changyeob Shin et al.  “Autonomous Tissue Manipulation via Surgical Robot Using Learning Based Model Predictive Control”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2019
  * [39] Li Sun et al.  “A Heuristic-Based Approach for Flattening Wrinkled Clothes”  In _Towards Autonomous Robotic Systems. TAROS 2013. Lecture Notes in Computer Science, vol 8069_ , 2014
  * [40] Li Sun, Gerardo Aragon-Camarasa, Simon Rogers and J. Siebert  “Accurate Garment Surface Analysis using an Active Stereo Robot Head with Application to Dual-Arm Flattening”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2015
  * [41] Priya Sundaresan et al.  “Learning Rope Manipulation Policies using Dense Object Descriptors Trained on Synthetic Depth Data”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2020
  * [42] Brijen Thananjeyan et al.  “Safety Augmented Value Estimation from Demonstrations (SAVED): Safe Deep Model-Based RL for Sparse Cost Robotic Tasks”  In _IEEE Robotics & Automation Letters_, 2020
  * [43] Brijen Thananjeyan et al.  “Multilateral Surgical Pattern Cutting in 2D Orthotropic Gauze with Deep Reinforcement Learning Policies for Tensioning”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2017
  * [44] Josh Tobin et al.  “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World”  In _Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2017
  * [45] Eric Torgerson and Fanget Paul  “Vision Guided Robotic Fabric Manipulation for Apparel Manufacturing”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 1987
  * [46] Bryan Willimon, Stan Birchfield and Ian Walker  “Model for Unfolding Laundry using Interactive Perception”  In _Proc. IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS)_ , 2011
  * [47] Yilin Wu et al.  “Learning to Manipulate Deformable Objects without Demonstrations”  In _RSS_ , 2020
  * [48] Wilson Yan, Ashwin Vangipuram, Pieter Abbeel and Lerrel Pinto  “Learning Predictive Representations for Deformable Objects Using Contrastive Estimation”  In _Conf. on Robot Learning (CoRL)_ , 2020
  * [49] Kevin Zakka, Andy Zeng, Johnny Lee and Shuran Song  “Form2Fit: Learning Shape Priors for Generalizable Assembly from Disassembly”  In _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2020

##  IX Appendix

The appendix is organized as follows:

  * •

Appendix A contains additional details on the fabric simulator

  * •

Appendix B contains additional details on the experiments conducted in
simulation

  * •

Appendix C contains results for simulation experiments

  * •

Appendix D shows visualizations of the learned descriptor mappings.

  * •

Appendix E contains images from additional physical trials executed on the
dVRK and YuMi.

  * •

Appendix F conducts a detailed study on the effect of various hyperparameters
on the quality of the learned visual correspondences.

###  IX-A Fabric Simulator Details

We use Blender 2.8 to both create dynamic cloth simulations and to render
images of the fabric in different configurations. As can be seen in Figure 2,
we are able to retrieve the world coordinates of each vertex via Blender’s API
which we then use to find ground truth pixel correspondences through an
inverse camera to world transformation. This allows us to create dense pixel-
vertex annotations along the surface of the fabric which we feed to into the
descriptor training procedure. Figure 9 is a visualization of the learned
descriptors and Figure 7 contains examples of the domain randomized training
data we generate through Blender.

####  IX-A1 Fabric Model

To generate the square cloth in Blender, we first import a default square mesh
and subdivide it three times to create a grid of $27\times 27$ grid of
vertices. We found that this number of square vertices resulted in a visually
realistic animation in comparison to our real fabrics. We additionally add
0.02 meter thickness to the cloth to increase its weight which creates more
realistic collision physics. In order to apply Blender’s in-built cloth
physics to the mesh, we simply make use of the cloth physics modifier through
which we are able modify the parameters shown in Table IV. Internally, Blender
simulates fabric physics for polygonal meshes with gravitational forces,
damping, stiffness, and by interconnecting the mesh vertices with four types
of virtual springs: tension springs, compression springs, shear springs, and
angular bending springs. Each vertex also exerts repulsive forces within a
self-contained virtual sphere on vertices both within fabric and in
surrounding objects, to simulate self-collisions and collisions with other
objects. We visually tune the simulator by replaying a fabric folding action
while varying parameters, most notably the friction coefficients and spring
elasticity constants. From observing videos of the folding actions, we settle
on the parameter values specified in Table IV. A visualization of these steps
can be seen in the top row of Figure 8. To generate the t-shirt mesh, we
similarly import a default square mesh and subdivide it three times, but also
delete all vertices that do not lie in a predefined t-shirt cutout of the
square mesh which results in the bottom right image of Figure 2.

####  IX-A2 Manipulation with Hook Objects

We utilize hook objects to take actions in the Blender simulator. A hook
object attaches to a mesh vertex and exerts a proportional sphere of influence
over the selected vertex and those in its vicinity, pulling the fabric in the
direction of movement. We simulate a grasp, drag, and drop of the fabric by
assigning a hook object to a fabric vertex, moving this hook over a series of
frames to the deprojected pixel drop location, and removing the hook object
assignment to release the cloth.

Figure 7:  Examples of domain-randomized images of the starting fabric states
encountered in the dataset generation phase described in Section V-B. The
first two columns show examples of images with a square fabric, and the last
two columns show similar examples but with a t-shirt.  Figure 8:  The top row
illustrates the the process of creating cloth in Blender from a default square
mesh. The second row is an example of a starting configuration generated by
dropping the cloth from a fixed height and pinning a single arbitrary vertex.
The pinned vertex is labeled by the red circle. The third row illustrates
frames from a folding action in the simulator and the last row shows the
corresponding rendered images of the settled cloth before and after the
action.  Table IV: Blender Cloth Simulation Parameters

Parameter | Explanation | Value
---|---|---
Quality Steps | quality of cloth stability and collision response | 5.0
Speed Multiplier | how fast simulation progresses | 1.0
Cloth Mass (kg) | – | 0.3
Air Viscosity | air damping | 1.0
Tension Springs | tension damping/stretching | 5.0
Compression Springs | compression damping/stretching | 5.0
Shear Springs | damping of shear behavior | 5.0
Bending Springs | damping of bending behavior | 0.5
Friction | friction with self-contact | 5
Self-Collision Distance (m) | per-vertex spherical radius for repulsive forces | 0.015

Figure 9:  Visualization of the 3-dimensional descriptors learned via the
training procedure described in Section V by mapping each pixel’s descriptor
vector to an RGB vector. Thus, similar colors across the images of columns two
and four represent corresponding points on the square cloth.  Figure 10:
Additional rollouts of the smoothing task from randomly chosen starting
configurations. The learned descriptors are used to locate the corners of the
fabric and successively pull them to a reference location in an image of the
flat cloth.  Figure 11:  Additional rollouts of the smoothing task from
randomly chosen starting configurations. The learned descriptors are used to
locate the corners of the fabric and successively pull them to a reference
location in an image of the flat cloth.  Figure 12:  Additional rollouts of
the folding tasks described in Section VII-A from arbitrary starting
configurations. The left column, center column and right column contain
results for tasks 4, 3 and 2 respectively.

####  IX-A3 Starting Configurations and Actions

To generate varied starting configurations, we simulate dropping the fabric
from 0.2 meters above the workspace while pinning it an arbitrary subset of
the 729 vertices. After 30 frames in the animation, the pinned vertices are
released and are allowed to settle for another 30 frames. This creates natural
deformation in the cloth and introduces a wide range of starting
configurations to the training dataset. A sequence of these steps is shown in
the second row of Figure 8. When running simulated experiments, taking pick
and place actions requires manipulating the cloth via hook objects as defined
in Section IV. A sequence of frames throughout the course of an action using a
hook object as well as the corresponding rendered frames are shown in the last
two rows of Figure 8.

###  IX-B Simulation Experiment Details

In simulation, we conduct 50 trials of the first 4 folding tasks described in
VII-A on a domain randomized test set generated as described in V-B. We
consider an outcome a success if the final state is visually consistent with
the target image. We additionally declared a failure when the planned pick and
drop pixels were more than 50 pixels away from their correct ground truth
locations which we had access to in Blender. Note that this is neither a
sufficient nor necessary condition for a successful fold, but nevertheless
serves as a decent heuristic. While we considered more quantitative metrics
such as structural similarity between the target image and the final state and
summed distance between corresponding vertices on the mesh, these metrics are
insufficient when the test time starting configuration is significantly
different from the demonstration configuration.

###  IX-C Simulation Experiment Results

We evaluate the folding policies designed in Section VI in the simulated
fabric environment. The policies successfully complete the tasks $84$ to $96$
percent of the time (Table V).

Task | Success Rate
---|---
Single Fold | 46/50
Double Inward Fold | 48/50
Double Triangle Fold | 42/50
T-Shirt Sleeves Fold | 44/50

Table V: Simulated Fabric Folding Experiments: We observe that the system is
able to successfully complete the tasks $84$ to $96$ percent of the time in
simulation. Success is determined by visual inspection of the cloth after the
sequence of actions is executed. Example simulation rollouts are shown in
Figure 14.

###  IX-D Descriptor Mapping Visualizations

We present the descriptor volumes produced by a model trained to output
3-dimensional descriptors. We coarsely visualize the volumes by presenting
them as RGB images (Figure 9), and observe that corresponding pixels of the
cloth map to similar colors in the descriptor volumes across configurations.

###  IX-E Physical Trial Trajectories

In this section, we present additional trials of the physical experiments
conducted using the descriptor-based policies for smoothing (Figure 10, Figure
11) and folding (Figure 12).

###  IX-F Descriptor Quality Ablations

Figure 13:  Ablation studies:  We study the sensitivity of the learned dense
object descriptors as described in Sections V and IX-F to training parameters.
Starting from top left, and proceeding clockwise, we test the effect of
testing on RGB vs depth images, on the descriptor dimension (either 3, 9, or
16), on the number of ground truth annotations, and whether domain
randomization is used. All results are evaluated using pixel match error on a
held-out set of image pairs.  Figure 14:  Simulation Policy Visualization:
Visualization of the policy executed in simulation (with Blender) using
learned descriptors for folding tasks 2 and 4 described in VII-A. The first
two columns show the corresponding folding instructions from a web interface
(pick-and-place actions shown with red arrows) for tasks 2 and 4. The third
column shows images of the previously unseen initial configurations of fabrics
before the actions, while the last two columns show the result of executing
descriptor-parameterized actions. Results suggest that the learned descriptors
can be used to successfully perform a variety of folding tasks from varying
initial configurations.

To investigate the quality of the learned descriptors with training process
described in Section V, we perform four sets of ablation studies. We evaluate
the quality of learned descriptors in a manner similar to [41] by evaluating
the $\ell_{2}$ pixel distance of the pixel match error on a set of 100 pairs
of held-out validation set images, where for each we sample 100 pixel pairs.
We study the effect of training descriptors on (1) RGB or depth images, (2)
using descriptor dimension 3, 9, or 16, (3) using 200, 450, or 700 ground-
truth annotated images, and (4) whether domain randomization is used or not.
Results suggest that the learned descriptors are best with RGB data, with
descriptor dimension between 3 and 16 and with domain randomization, though
the performance is generally insensitive to the parameter choices, suggesting
a robust training procedure. Based on these results, we use RGB images with
domain randomization, and with descriptor dimension 3 for all simulated
experiments for both the t-shirt and square fabric. We use RGB, domain-
randomized, 9-dimensional descriptors for real fabric experiments. See Figure
13 for plots.


# Path integral control under McKean-Vlasov dynamics

Timothy Bennett111Corresponding author<EMAIL_ADDRESS>222Department of Mathematics and Statistics, University of South Alabama,
Mobile, AL, 36688, United States.

###### Abstract

We investigate the complexities of the McKean–Vlasov optimal control problem,
exploring its various formulations such as the strong and weak formulations,
as well as both Markovian and non-Markovian setups within financial markets.
Furthermore, we examine scenarios where the law governing the control process
impacts the dynamics of options. By conceptualizing controls as probability
measures on a fitting canonical space with filtrations, we unlock the
potential to devise classical measurable selection methods, conditioning
strategies, and concatenation arguments within this innovative framework.
These tools enable us to establish the dynamic programming principle under a
wide range of conditions.

## Introduction:

The Black-Scholes model stands as the quintessential example of financial
models, embodying key principles that have influenced subsequent developments.
Its widespread adoption reflects the common adjustments and trade-offs
necessary to translate theoretical concepts into practical tools for financial
analysis and decision-making. Let us dive deep into the core principles of
this model. The main aim of creating the Black-Scholes model is to evaluate
and manage options, operating on the premise that the volatility of the
underlying asset remains constant (Fournié et al.,, 1997). Building on this
assumption, it provides an analytical formula for calculating the value of a
European option, which relies on the volatility parameter of the system.
However, the model’s main drawback arises from the fact that in actuality,
volatility fluctuates, and the market does not necessarily conform to the
values predicted by this model (Pramanik,, 2020).

However, practitioners in financial markets have widely embraced the Black-
Scholes model, albeit with some modifications that deviate from the original
theoretical expectations. These adjustments include anticipating negotiations
for transactions in terms of implicit volatility rather than option prices,
implementing theoretical delta hedging as recommended by the model, and
recognizing that the implicit volatility tends to be higher for options
further from the money and closer to maturity (Pramanik, 2021c, ; Pramanik and
Polansky,, 2021, 2023). Consequently, the Black-Scholes model retains
significant practical utility, although its role differs from the primary
hypotheses outlined in the original theory. While one might assume that the
compromises and adaptations made in utilizing the Black-Scholes model are
unique to this particular context, we argue that they offer broader insights
applicable across various fields beyond finance.

So far in our initial discussion, we overlooked the inclusion of control
theory. Control theory encompasses two primary forms: open loop and closed
loop control. In open loop control, the decision of the policymaker remains
uninfluenced by the state variable’s information. Conversely, closed loop
control entails the policymaker’s decision being contingent upon all available
state variable information (Pramanik, 2024a, ). Integrating control variables
is crucial in understanding the dynamics of financial markets. For instance,
consider an entrepreneur aiming to maximize long-term profits, assuming
control over advertising expenditure. If they increase spending on commercials
presently, they may afford a celebrity endorsement, thereby boosting sales
probability in subsequent periods (Pramanik and Polansky,, 2020). Conversely,
heightened sales in the current period provide more funds for future
commercial investment. This exemplifies closed loop control within financial
markets. Furthermore, all of theses financial models are not always linear
stochastic differential equations (SDEs) in practice. Therefore, the
importance of non-linear SDEs come into place. One such SDE is McKean-Vlasov
(McKean Jr,, 1966) system. The next section gives an overview of it.

## Overview of McKean-Vlasov SDEs:

McKean-Vlasov type SDEs are commonly known as nonlinear SDEs, with _non-
linear_ highlighting the potential dependence of coefficients on the
solutions’ marginal distributions rather than the state variable itself. The
distinctive aspect of these coefficients, even when deterministic, leads to
non-local feedback effects, precluding the conventional Markov property
(Carmona and Delarue,, 2015). Incorporating the marginal distribution into the
system’s state could restore the Markov property but at the expense of
significantly increasing the complexity by introducing a probability measure,
rendering the system infinite-dimensional. While analyzing the infinitesimal
generator can utilize tools developed for infinite-dimensional differential
operators, conventional differential calculus, even in infinite dimensions,
struggles to ensure the second component of the state process matches the
statistical distribution of the first component (Carmona and Delarue,, 2015).
Nevertheless, pathwise analysis complexity remains manageable and can be
effectively addressed using standard stochastic calculus tools. This points
towards devising a wholly probabilistic approach to address optimal control in
these nonlinear systems.

The optimal control of systems governed by McKean-Vlasov SDEs appears to be a
relatively novel challenge, curiously overlooked in stochastic control
literature. Typically, solving a McKean-Vlasov SDE involves employing a fixed-
point argument: Initially, a set of potential marginal distribution candidates
is established, and then the resulting standard SDE is solved, with the fixed-
point argument entailing the requirement that the solution’s marginal
distributions match the initially chosen ones. Introducing stochastic control
introduces an additional layer of optimization atop this fixed-point process.
This formulation shares many similarities with the mean-field game (MFG)
problem, as initially proposed by Lasry and Lions, (2007). In Carmona et al.,
(2013), the similarities and disparities between these two problems are
identified and discussed. It is notably emphasized that optimizing first and
subsequently seeking a fixed point yields the solution to a mean-field game
problem, while first determining the fixed point and then optimizing leads to
the resolution of the optimal control for McKean-Vlasov SDEs (Polansky and
Pramanik,, 2021; Pramanik, 2024b, ). The solutions to both problems describe
equilibrium states of large populations whose interactions and objectives
follow mean-field patterns. The nuances between these equilibrium concepts are
subtle and hinge on the optimization component’s formulation within the
equilibrium model. Linear quadratic models are presented in Carmona et al.,
(2013) to offer illustrative examples of these distinctions.

As McKean-Vlasov SDEs exhibit non-Markovian dynamics, it is logical to tackle
optimization through a tailored version of the Pontryagin stochastic maximum
principle, rather than forcing fit with the Hamilton-Jacobi-Bellman (HJB)
framework. The stochastic Pontryagin method hinges on introducing and
manipulating adjoint processes, which are solutions to adjoint backward
stochastic differential equations (BSDEs). These equations involve partial
derivatives of the Hamiltonian function concerning the state variable (Carmona
and Delarue,, 2015). However, in the context of McKean-Vlasov SDEs, the
marginal distributions of solutions serve as full-fledged variables of the
Hamiltonian function, necessitating differentiation in the pursuit of critical
points. This discrepancy likely contributes to the impasse in existing
literature. Until now, only dynamics contingent on moments of the marginal
distributions have been explored, where differentiability with respect to the
measure can be achieved through conventional calculus chain rules (Carmona and
Delarue,, 2015). The right notation of differentiability was introduced in
Lasry and Lions, 2006a  and Lasry and Lions, 2006b .

## Pontryagin principle and Path integral control:

The stochastic Pontryagin principle stands as a potent tool, yet its
illuminating insights often necessitate restrictive assumptions on the models.
For instance, findings in Carmona and Delarue, (2015) lean on a set of
technical prerequisites that confine the scope of models to those
characterized by coefficients primarily linear in state, control, and measure
variables, alongside costs that exhibit convexity concerning the state and
control variables. While these conditions may seem limiting, they are typical
in applications of the stochastic Pontryagin principle to control problems.
Notably, assuming the convexity of the control space is done for simplicity’s
sake. More extensive spaces could be accommodated, albeit at the expense of
employing spike variation techniques and introducing an additional adjoint
equation. However, Carmona and Delarue, (2015) opted against pursuing this
broader generality, as it would entail added complexity in notation and
technicalities, potentially obscuring the core focus of their research,
especially without specific application-driven motivation (Pramanik, 2022a, ;
Pramanik, 2022b, ; Pramanik, 2023b, ; Pramanik, 2023c, ).

The essential aspect of the stochastic Pontryagin principle involves seeking
within the control set for a potential minimizer of the Hamiltonian (often
referred to as satisfying the Isaacs condition), while the complementary
aspect suggests incorporating the minimizer’s formula into both the forward
equation governing dynamics and the adjoint backward equation describing
adjoint processes (Hua et al.,, 2019; Pramanik,, 2016; Pramanik, 2021b, ). The
inclusion of the minimizer in these equations establishes a significant
interconnection between the forward and backward equations, effectively
reducing the control problem solution to that of a forward-backward stochastic
differential equation (FBSDE). Applying this approach to the current scenario
entails investigating FBSDEs where the marginal distributions of solutions
emerge in the coefficients of the equations. Carmona and Delarue, (2015) term
these equations “mean-field FBSDEs” or “FBSDEs of McKean-Vlasov type,”
asserting that their study had not been undertaken prior to their research.
They noted that while a general existence result was proposed in Carmona and
Delarue, (2013), one of the assumptions therein prevents its application to
the linear quadratic model.

A specific variant of the stochastic Pontryagin principle, such as the
Feynman-type path integral method introduced by Pramanik, (2020), can be
utilized and further developed by Pramanik and Polansky, 2024a  to obtain an
analytical solution for the system in question (Pramanik and Polansky, 2024b,
). When the state variable is high-dimensional and the system dynamics are
nonlinear, as seen in Merton-Garman-Hamiltonian SDEs, numerical construction
of an HJB equation becomes extremely difficult. The Feynman-type path integral
approach effectively overcomes this challenge of dimensionality and provides a
localized analytical solution. To employ this approach, we first establish a
stochastic Lagrangian for each continuous time point within the interval
$s\in[0,t]$, where $t>0$. Subsequently, we divide this entire time span into
$n$ equal-length subintervals and define a Riemann measure corresponding to
the state variable for each subinterval. After constructing a Euclidean action
function, we obtain a Schroedinger-type equation via Wick rotation. By
enforcing the first-order conditions with respect to both the state and
control variables, we determine the solution of the system. This methodology
holds potential application in cancer research (Dasgupta et al.,, 2023;
Hertweck et al.,, 2023; Kakkat et al.,, 2023; Khan et al.,, 2023; Vikramdeo et
al.,, 2023; Khan et al.,, 2024).

## Further impact:

We gain insight into the challenge of optimal control for mean-field
stochastic differential equations, also known as McKean–Vlasov stochastic
differential equations in academic literature. This problem presents a
stochastic control scenario where the state process is driven by SDEs with
coefficients influenced by the present time, the trajectories of the state
process, and its distribution (or conditional distribution in cases involving
shared noise). Likewise, the reward functionals can be influenced by the
distribution of the state process (Pramanik, 2023a, ).

A common limitation of the aforementioned results is their general requirement
for some Markovian property of the system or its distribution, along with
strong regularity assumptions on the coefficients and reward functions under
consideration. This might come as a surprise to those familiar with the
classical stochastic Pontryagin principle (Pramanik, 2021a, ). In stochastic
control problems, it is feasible to utilize measurable selection arguments to
derive the stochastic Pontryagin principle without necessitating more than
mild measurability assumptions (Pramanik and Maity,, 2024). Typically, two
crucial elements are needed to establish the dynamic programming principle:
first, ensuring the stability of controls concerning conditioning and
concatenation, and second, the measurability of the associated value function.
Measurable selection arguments offer a framework to validate the conditioning,
concatenation, and measurability requirements of the associated value function
without relying on strong assumptions. Through the concept of relaxed control,
interpreting a control as a probability measure on a canonical space, and
leveraging martingale problem theory, researchers have demonstrated a
stochastic Pontryagin principle using straightforward and transparent
reasoning (Djete et al.,, 2022). In the future, we aim to explore scenarios
where the drift and diffusion components, as well as the reward functions in
option markets, can depend on the joint conditional distribution of the state
process path and the closed-loop control.

## References

  * Carmona and Delarue, (2013) Carmona, R. and Delarue, F. (2013).  Mean field forward-backward stochastic differential equations.  Electronic Communications in Probability, 18:1–15.
  * Carmona and Delarue, (2015) Carmona, R. and Delarue, F. (2015).  Forward–backward stochastic differential equations and controlled mckean–vlasov dynamics.  The Annals of Probability, pages 2647–2700.
  * Carmona et al., (2013) Carmona, R., Delarue, F., and Lachapelle, A. (2013).  Control of mckean–vlasov dynamics versus mean field games.  Mathematics and Financial Economics, 7:131–166.
  * Dasgupta et al., (2023) Dasgupta, S., Acharya, S., Khan, M. A., Pramanik, P., Marbut, S. M., Yunus, F., Galeas, J. N., Singh, S., Singh, A. P., and Dasgupta, S. (2023).  Frequent loss of cacna1c, a calcium voltage-gated channel subunit is associated with lung adenocarcinoma progression and poor prognosis.  Cancer Research, 83(7_Supplement):3318–3318.
  * Djete et al., (2022) Djete, M. F., Possamaï, D., and Tan, X. (2022).  Mckean–vlasov optimal control: the dynamic programming principle.  The Annals of Probability, 50(2):791–833.
  * Fournié et al., (1997) Fournié, E., Lasry, J.-M., and Lions, P.-L. (1997).  Some nonlinear methods for studying far-from-the-money contingent claims.  Publ. Newton Inst, pages 115–45.
  * Hertweck et al., (2023) Hertweck, K. L., Vikramdeo, K. S., Galeas, J. N., Marbut, S. M., Pramanik, P., Yunus, F., Singh, S., Singh, A. P., and Dasgupta, S. (2023).  Clinicopathological significance of unraveling mitochondrial pathway alterations in non-small-cell lung cancer.  The FASEB Journal, 37(7):e23018.
  * Hua et al., (2019) Hua, L., Polansky, A., and Pramanik, P. (2019).  Assessing bivariate tail non-exchangeable dependence.  Statistics & Probability Letters, 155:108556.
  * Kakkat et al., (2023) Kakkat, S., Pramanik, P., Singh, S., Singh, A. P., Sarkar, C., and Chakroborty, D. (2023).  Cardiovascular complications in patients with prostate cancer: Potential molecular connections.  International Journal of Molecular Sciences, 24(8):6984.
  * Khan et al., (2023) Khan, M. A., Acharya, S., Anand, S., Sameeta, F., Pramanik, P., Keel, C., Singh, S., Carter, J. E., Dasgupta, S., and Singh, A. P. (2023).  Myb exhibits racially disparate expression, clinicopathologic association, and predictive potential for biochemical recurrence in prostate cancer.  Iscience, 26(12).
  * Khan et al., (2024) Khan, M. A., Acharya, S., Kreitz, N., Anand, S., Sameeta, F., Pramanik, P., Keel, C., Singh, S., Carter, J., Dasgupta, S., et al. (2024).  Mp60-05 myb exhibits racially disparate expression and clinicopathologic association and is a promising predictor of biochemical recurrence in prostate cancer.  The Journal of Urology, 211(5S):e1000.
  * (12) Lasry, J.-M. and Lions, P.-L. (2006a).  Jeux à champ moyen. i–le cas stationnaire.  Comptes Rendus Mathématique, 343(9):619–625.
  * (13) Lasry, J.-M. and Lions, P.-L. (2006b).  Jeux à champ moyen. ii–horizon fini et contrôle optimal.  Comptes Rendus. Mathématique, 343(10):679–684.
  * Lasry and Lions, (2007) Lasry, J.-M. and Lions, P.-L. (2007).  Mean field games.  Japanese journal of mathematics, 2(1):229–260.
  * McKean Jr, (1966) McKean Jr, H. P. (1966).  A class of markov processes associated with nonlinear parabolic equations.  Proceedings of the National Academy of Sciences, 56(6):1907–1911.
  * Polansky and Pramanik, (2021) Polansky, A. M. and Pramanik, P. (2021).  A motif building process for simulating random networks.  Computational Statistics & Data Analysis, 162:107263.
  * Pramanik, (2016) Pramanik, P. (2016).  Tail non-exchangeability.  Northern Illinois University.
  * Pramanik, (2020) Pramanik, P. (2020).  Optimization of market stochastic dynamics.  In SN Operations Research Forum, volume 1, page 31. Springer.
  * (19) Pramanik, P. (2021a).  Consensus as a nash equilibrium of a stochastic differential game.  arXiv preprint arXiv:2107.05183.
  * (20) Pramanik, P. (2021b).  Effects of water currents on fish migration through a feynman-type path integral approach under 8/3 liouville-like quantum gravity surfaces.  Theory in Biosciences, 140(2):205–223.
  * (21) Pramanik, P. (2021c).  Optimization of dynamic objective functions using path integrals.  PhD thesis, Northern Illinois University.
  * (22) Pramanik, P. (2022a).  On lock-down control of a pandemic model.  arXiv preprint arXiv:2206.04248.
  * (23) Pramanik, P. (2022b).  Stochastic control of a sir model with non-linear incidence rate through euclidean path integral.  arXiv preprint arXiv:2209.13733.
  * (24) Pramanik, P. (2023a).  Optimal lock-down intensity: A stochastic pandemic control approach of path integral.  Computational and Mathematical Biophysics, 11(1):20230110.
  * (25) Pramanik, P. (2023b).  Path integral control in infectious disease modeling.  arXiv preprint arXiv:2311.02113.
  * (26) Pramanik, P. (2023c).  Path integral control of a stochastic multi-risk sir pandemic model.  Theory in Biosciences, 142(2):107–142.
  * (27) Pramanik, P. (2024a).  Dependence on tail copula.  J, 7(2):127–152.
  * (28) Pramanik, P. (2024b).  Estimation of optimal lock-down and vaccination rate of a stochastic sir model: A mathematical approach.  European Journal of Statistics, 4:3–3.
  * Pramanik and Maity, (2024) Pramanik, P. and Maity, A. K. (2024).  Bayes factor of zero inflated models under jeffereys prior.  arXiv preprint arXiv:2401.03649.
  * Pramanik and Polansky, (2020) Pramanik, P. and Polansky, A. M. (2020).  Motivation to run in one-day cricket.  arXiv preprint arXiv:2001.11099.
  * Pramanik and Polansky, (2021) Pramanik, P. and Polansky, A. M. (2021).  Optimal estimation of brownian penalized regression coefficients.  arXiv preprint arXiv:2107.02291.
  * Pramanik and Polansky, (2023) Pramanik, P. and Polansky, A. M. (2023).  Scoring a goal optimally in a soccer game under liouville-like quantum gravity action.  In Operations Research Forum, volume 4, page 66. Springer.
  * (33) Pramanik, P. and Polansky, A. M. (2024a).  Optimization of a dynamic profit function using euclidean path integral.  SN Business & Economics, 4(1):8.
  * (34) Pramanik, P. and Polansky, A. M. (2024b).  Semicooperation under curved strategy spacetime.  The Journal of Mathematical Sociology, 48(2):172–206.
  * Vikramdeo et al., (2023) Vikramdeo, K. S., Anand, S., Sudan, S. K., Pramanik, P., Singh, S., Godwin, A. K., Singh, A. P., and Dasgupta, S. (2023).  Profiling mitochondrial dna mutations in tumors and circulating extracellular vesicles of triple-negative breast cancer patients for potential biomarker development.  FASEB BioAdvances, 5(10):412.