# DASH: Modularized Human Manipulation Simulation with Vision and Language for
Embodied AI

Yifeng Jiang  Stanford UniversityUnited States of America<EMAIL_ADDRESS>,  Michelle Guo  Stanford UniversityUnited States of America
<EMAIL_ADDRESS>,  Jiangshan Li  Stanford UniversityUnited States of
America<EMAIL_ADDRESS>,  Ioannis Exarchos  Stanford UniversityUnited
States of America<EMAIL_ADDRESS>,  Jiajun Wu  Stanford
UniversityUnited States of America<EMAIL_ADDRESS>and  C. Karen Liu
Stanford UniversityUnited States of America<EMAIL_ADDRESS>
(2021)

###### Abstract.

Creating virtual humans with embodied, human-like perceptual and actuation
constraints has the promise to provide an integrated simulation platform for
many scientific and engineering applications. We present Dynamic and
Autonomous Simulated Human (DASH), an embodied virtual human that, given
natural language commands, performs grasp-and-stack tasks in a physically-
simulated cluttered environment solely using its own visual perception,
proprioception, and touch, without requiring human motion data. By factoring
the DASH system into a vision module, a language module, and manipulation
modules of two skill categories, we can mix and match analytical and machine
learning techniques for different modules so that DASH is able to not only
perform randomly arranged tasks with a high success rate, but also do so under
anthropomorphic constraints and with fluid and diverse motions. The modular
design also favors analysis and extensibility to more complex manipulation
skills.

Virtual Human, Embodied AI System, Physics Simulation, Computer Vision,
Reinforcement Learning.

††journalyear: 2021††copyright: acmlicensed††conference: The ACM SIGGRAPH /
Eurographics Symposium on Computer Animation; September 6–9, 2021; Virtual
Event, USA††booktitle: The ACM SIGGRAPH / Eurographics Symposium on Computer
Animation (SCA ’21), September 6–9, 2021, Virtual Event, USA††price:
15.00††doi: 10.1145/3475946.3480950††isbn: 978-1-4503-8676-0/21/09††ccs:
Computing methodologies Modeling and simulation††ccs: Computing methodologies
Control methods††ccs: Computing methodologies Machine learning††ccs: Computing
methodologies Animation

##  1\. Introduction

Figure 1. Our system, dynamic and autonomous simulated human (DASH), is an
embodied virtual human modeled off of a child. DASH is able to manipulate
tabletop objects with a dexterous arm and hand in a human-like way, using its
own vision perception (white box), language understanding (user command in
black box), proprioception and touch. (Video 0m12s)

The recent breakthroughs in artificial intelligence (AI) have opened new
avenues for computer graphics research [Tewari et al., 2020; Peng et al.,
2018; Chaudhuri et al., 2020]. Systems originally built for graphics
applications are also playing an increasingly important role in the progress
of AI. For example, Embodied AI research, which centers around the idea that
intelligence emerges from the inseparable interplay of sensorimotor activities
and physical interactions with the environment [Smith and Gasser, 2005], has
utilized rendering systems and physics engines to provide inexpensive training
data and testbeds. An integrated simulation environment with both realistic
physics and visual appearances has the promise of creating diverse and safe
playgrounds for virtual embodied AI agents to learn through interaction [Savva
et al., 2019; Xia et al., 2020; Weihs et al., 2020].

In this paper, we argue for the crucial inclusion of simulated humans in
Embodied AI systems using integrated techniques and knowledge from Computer
Animation. Such virtual humans should not only manifest realistic physics and
appearance, but also function autonomously under embodied anthropomorphic
constraints, such as egocentric vision, and with human-like interaction, such
as natural language. Simulated humans that react to physics, perceive, and
interact as real humans can provide an effective and safe platform for
physical Human-Robot Interaction (pHRI) and assistive robotics, such as
modeling human behavior during robot-assisted eating and drinking [Erickson et
al., 2019]. Embodied virtual humans can also provide a new platform for
scientific studies such as the perception-action coupling of infants’
development: infants’ initial physical interactions with the environment,
though being imperfect with frequent failures, provide diverse experiences for
perception development, which in turn facilitates motor skill learning. Though
we envision most applications of our system to be in the virtual world,
knowledge obtained through computational modeling of virtual humans could as
well serve as a first step towards building physically embodied humanoid
companions.

Such realistic modeling of human perception, actuation, and interaction
constraints is in drastic contrast to existing virtual human systems, which
either focus on visual appearance and motion realism [Antoniades, 2016;
Shapiro, 2011], or on higher-level vision and language capabilities while
abstracting out actual motor skills [Thiebaux et al., 2008; Hartholt et al.,
2013; Puig et al., 2018]. In comparison with current cognitive modeling
systems, such as the ACT-R [Anderson et al., 1997] simulator, which abstracts
both perception and motor with symbolic modules, the physically embodied
visual, verbal, and manual modules we aim to build could eventually be used to
augment such systems. In contrast to existing character animation research,
which usually does not simulate human perception, embodied virtual humans base
their motor actions on noisy and partial perceptions of itself, rather than
ground-truth simulator states.

To this end, we propose and build a system to physically simulate manipulation
skills of a virtual human with egocentric vision, natural language command,
proprioception, and touch—a first one to the best of our knowledge (Figure 1).
We focus on object manipulation, which demands coordination between visual
perception and motor dexterity. This system presents a unique set of
requirements and challenges, as we summarize below:

  * •

Generality: A highly generalizable system is crucial for our envisioned
applications, because a simple verbal command alone could generate a wide
variety of scenarios—our system must be capable of manipulating objects of
different shapes and sizes arbitrarily placed in the workspace, while avoiding
an unknown number of surrounding, arbitrarily arranged objects. Our system
should demonstrate high success rates on a large number of such scenes.

  * •

Anthropomorphism: We extend the conventional notion of “being humanlike” in
character animation research to include both anthropomorphic constraints and
humanlike behaviors. Our system should operate under anthropomorphic
constraints such as delayed and noisy visual input. Our system must
demonstrate fluid motion on a high degrees-of-freedom (DoF) anthropomorphic
arm and hand, in contrast to slow, conservative motion optimized for
functionality that is typically seen on robotic manipulation systems. As we
opt to explore many humanlike constraints to be imposed on our system, this
work does not aim to create motions of better quality than current character
animation techniques using full simulator states.

  * •

Extensibility: Our system can be used as a self-contained black-box for
Embodied AI research or pHRI. At the same, it should have sufficient
modularity such that each module can be replaced or improved upon, so that
more complex manipulation skills can be augmented to our first system without
the need to redesign.

  * •

Interactivity: The system should run at an interactive rate, allowing users to
interact with the agent using online verbal commands.

Generality and anthropomorphism are often at odds with each other, because the
demand of high success rates often results in a single conservative
manipulation strategy with few natural variations. These important but
sometimes conflicting goals prompt us to prioritize modularization in our
system design. Specifically, we factor the problem into a vision module, a
language module and manipulation modules for different stages, so that we can
apply analytical methods to modules where correctness over a large range of
scenarios are most important, and apply machine learning approaches to places
where analytical methods are hard to solve and natural variations of motions
are desired. As we are building a first system of its kind, modularization
also facilitates analysis, interpretation, and extensibility.

We name our system Dynamic and Autonomous Simulated Human (DASH) and model
DASH’s visual appearance off a 5-year-old boy (Figure 1), as we believe
modeling and understanding of the simultaneous acquisition of perceptual and
motor skills are more important than demonstrating perfect skills themselves
for embodied intelligence research. We evaluate the scenario in which DASH
stands in front of a set of solid objects of various shapes and colors on a
tabletop. The user gives DASH a sequence of grasp-and-stack commands in a
domain-specific language to rearrange randomly scattered objects into a
desired configuration. We evaluate the success rate of DASH accomplishing
tasks correctly from randomized initial scenes, qualitatively demonstrate that
DASH can produce fluid motions with natural variations, and analyze learned
motor skills and various failure cases.

##  2\. Related Work

##### Virtual Human Systems.

Creating virtual humans is a multidisciplinary research challenge in Computer
Graphics, Artificial Intelligence, Cognitive Science, Human-Computer
Interaction, and so forth. Many existing virtual human systems rely on pre-
scripted, recorded or procedurally generated motion to animate the virtual
humans in response to a set of predefined events [Kalra et al., 1998; Caicedo
and Thalmann, 2000; Magnenat-Thalmann and Egges, 2006; Jung et al., 2011; Feng
et al., 2014]. Multimodal dialog systems have demonstrated the use of virtual
humans for verbal and nonverbal communication for training, education, and
virtual assistant applications [Rickel and Johnson, 1999; Cassell et al.,
2000; Chi et al., 2000; Lee and Marsella, 2006; Nguyen et al., 2015; Matsuyama
et al., 2016; Perera et al., 2018]. Computational modelings of human cognition
[Anderson et al., 1997; Funge et al., 1999] and perception mechanisms [Peters
and O’Sullivan, 2002; Rabie and Terzopoulos, 2000; Nakada et al., 2018; Eom et
al., 2019] have the promise of not only creating realistic motions but also
answering why humans show certain behaviors. Our work similarly models
realistic human perceptual and physical constraints, but further directly
utilizing RGB images and physics simulations as input. Our system generates
human motion from computational models rather than motion data, allowing
generalizable, unprescribed interactions with the world and potentially
answering scientific questions.

##### Robotic Systems with Vision and Language.

Systems that can perceive visual scenes and understand language instructions
in a humanlike way have been explored both in simulations and on real-world
robots. Virtual agents in [Puig et al., 2018; Shridhar et al., 2020] focus on
high-level reasoning and abstracts motor skills with animated motion
primitives, while our system focuses on fully physical motor skills. On real
robots, similar manipulation systems [Paxton et al., 2019; Shridhar and Hsu,
2018; Hatori et al., 2018; Liu and Zhang, 2019; Jiang et al., 2018; Mees et
al., 2020] usually employ robot grippers or suction cups where open-loop
control primitives are robust enough for picking up objects. In contrast, this
work explores the concatenation of multiple learned DRL controllers on a
simulated anthropomorphic hand which produces more fluid and diverse motion
while demonstrating robustness on precision tasks such as stacking of long and
narrow objects.

##### Dexterous Manipulation in Computer Graphics.

Dexterous manipulation is an integral component to creating interactive
virtual humans. In Computer Graphics, previous works have proposed to generate
physically plausible human grasps from recorded motion capture data [Pollard
and Zordan, 2005; Zhao et al., 2013], videos [Wang et al., 2013], or contact
forces [Kry and Pai, 2006], while our system aim at a more general grasp-and-
stack task in a cluttered environment. Compared with other works [Mordatch et
al., 2012; Liu, 2009; Ye and Liu, 2012; Andrews and Kry, 2012] that animate
dexterous manipulations, we rely on 2D images from the first-person view to
infer estimated object information, without accessing contact point locations
or forces from the ground-truth simulation states. A number of previous
animation works also investigate the whole body motion during manipulation
[Yamane et al., 2004; Jain and Liu, 2009; Ho et al., 2010; Lee et al., 2019;
Ye and Liu, 2012]. Merel et al. [Merel et al., 2019, 2020] recently developed
a hierarchical control framework that utilizes egocentric vision for
locomotion and whole-body manipulation. Our work focuses on complex dexterous
manipulation tasks in cluttered environments which requires coordinated
planning with vision and language inputs, without requiring motion data.

##### Dexterous Manipulation in Robotics.

In the Robotics community, manipulation primarily utilizes parallel jaw
grippers, but dexterous hands have also been of great interest to researchers
[Bicchi, 2000; Okamura et al., 2000; Ma and Dollar, 2011]. While we also solve
grasping with dexterous hands, our system lift the requirement of grasp
contact planning, facilitating the learned manipulation policies to zero-shot
transfer to novel geometries. In recent years, researchers have begun to
investigate DRL approaches for dexterous manipulation [Andrychowicz et al.,
2020; OpenAI et al., 2019; Nagabandi et al., 2019]. Our work is most closely
related to Rajeswaran et al. [2017]; Radosavovic et al. [2020], where they
used a combination of policy learning and imitation learning to solve a set of
dexterous manipulation problems in simulation, including pick-and-place tasks.
To reduce sample complexity and improve the quality of motion, Rajeswaran et
al. [2017] used a VR system and CyberGlove system to capture high fidelity
demonstrations from real humans. In contrast, our work does not require high-
quality human demonstrations acquired by specialized equipment. In addition,
our pick-and-place must be cognizant of other objects in the scene so the hand
can avoid collisions or interact with them.

##### Computer Vision and Natural Language Processing.

Recent development in deep learning has enabled perception systems that can
directly learn scene representations from raw visual input, especially in
constrained environments [Kulkarni et al., 2015; Wu et al., 2017; Nguyen-Phuoc
et al., 2019; Sitzmann et al., 2019; Mees et al., 2020]. Directly related to
our work, Yi et al. [2018] integrated an explicit visual representation with a
language understanding module, leads to high accuracy on visual reasoning.

Semantic parsing algorithms learn to map sentences to logical forms via a
knowledge base or a program [Berant et al., 2013; Liang et al., 2013; Guu et
al., 2017]. Recently, Rothe et al. [2017] showed that formal programs can be
effectively used to model human questions. For DASH, we choose to integrate an
off-the-shelf learning-based semantic parser [Honnibal and Montani, 2017] in
our language module.

##  3\. DASH: dynamic and autonomous simulated human

We mix and match analytical and learning approaches to balance between
generality and anthropomorphism requirements, and to favor extensibility. As
such, the DASH system consists of an ego-centric vision module, a language
parsing module and manipulation modules of two skill categories: transition
and interaction. We decompose the manipulation tasks in this work into four
self-contained stages: reaching, grasping, transporting, and placing/stacking
(Figure 2). We categorize reaching and transporting as transition stages, and
grasping and placing/stacking as interaction stages. Here we discuss some key
design decisions of our system.

Figure 2. We break down the general pick-and-place task into stages. (a)
initial rest pose; (b) end of reaching stage; (c) end of grasping stage; (d)
end of transporting stage; (e) end of placing/stacking stage; (f) retract to
task completion pose.

##### Combined or independent vision and manipulation modules?

We design a standalone vision module so we can reuse the same training
procedure for different purposes (e.g. initial planning, stacking). Such
modular design also allows us to activate or deactivate vision module
independent of the manipulation module. Compared with having one end-to-end
neural module, separating vision from motor training also significantly
reduces time necessary for training anthropomorphic arms and hands with high
degrees of freedom (DoFs). We also construct the vision architecture to be
object-centric, meaning that the same trained model should be able to estimate
properties of a varying number of objects in scene.

##### Integrated or skill-specific manipulation modules?

Another design decision we made is to mix and match model-based planning and
model-free reinforcement learning techniques for different manipulation
stages. For transition stages (reaching, transporting), model-based planning
has a probabilistic guarantee of a feasible solution if there exists one,
agnostic to scene complexity; in contrast, learning a motor policy would
require training to include all randomly arranged scenes, leading to a
combinatorial explosion of sample demand. On the other hand, using learned
controllers for interaction stages produces more fluid and diverse motions
compared to conservative open-loop grasping and placing primitives usually
seen for robotic grippers, which often manifest one single strategy with sole
finger motion and the arm fixed throughout. Concatenating multiple policies in
one manipulation system strikes a balance between functionality and
anthropomorphism, but it also poses additional challenges much unexplored in
previous works, which we discuss in Section 3.4.

##### With or without human motion data?

A final key design choice is avoiding the dependency on motion data. For our
envisioned applications where scientific questions, such as how impacted
perception affects motor skills, are asked, a system mimicking motion data
from healthy actors will not be sufficient. Besides, since we are interested
in a wide variety of highly constrained motion, such as stacking thin objects
in a cluttered environment, imitation learning alone might not be able to
produce feasible motions. On the other hand, we found that natural reaching
motion can be achieved using model-based path planning with re-timing based on
humanlike velocity profiles.

###  3.1. Vision Module

DASH includes a vision module that serves two main purposes: (i) to generate
the scene observation for initial task planning and (ii) to provide frequent
feedback to the manipulation module during object placing/stacking, both from
raw RGB images. Our vision module consists of an object segmentation model and
a feature extraction model, similar to [Yi et al., 2018]. Given a single scene
image, the vision module first predicts segments for all objects in the scene.
Each object segment is then concatenated with the original scene image and fed
into a feature extraction network to output attributes of this object, making
the model agnostic to number of objects in the scene. The model predicts
discrete attributes such as shape and color, as well as continuous attributes
including height, 3D position, and upright axis for each object (full
orientation is ambiguous for symmetrical objects). Gaze (head movement) is
animated to match the decided camera locations during stages (i) and (ii), and
interpolated between them for stages when vision module is turned off.
Replacing head movement with eye movement where possible could benefit overall
naturalness.

While our manipulation module can generalize to novel geometries unseen during
training (Video 3m14s), zero-shot transfer is much more challenging for
learning based vision algorithms. We therefore leave the generalization of
vision beyond primitive shapes for future work. The generation of training
data will be discussed in detail in Appendix A.1.

###  3.2. Language Module

The language module is invoked during initial task planning to infer the task
parameters from the natural language command. We do not assume the specific
form of the language command to allow natural variation in wording and
ordering, as long as the command semantically specifies a target object to
grasp, with reference object(s) and the desired final spatial relation to
them. We assume both the target and reference objects can be modified by a
shape, a color, and a modifier clause containing one secondary reference
object and one secondary spatial relation for disambiguation.

Our language module consists of two steps: dependency parsing and parse tree
search. We first use a robust neural-net-based dependency parser [Honnibal and
Montani, 2017] to obtain the semantic dependencies within our instruction
sentence in the form of a tree. This step is general to any sentence
structure. For the second step, with our assumptions, we perform a breadth-
first search on the parse tree to identify the target object to grasp, the
desired spatial relation defining the movement, and reference object(s).

The outputs of the vision and language modules are then cross-referenced.
There might exist multiple objects sharing the same shape and color in the
visual scene. A secondary search is performed in the clause subtree that
modifies the target or reference object. The system will halt if the
instruction still remains ambiguous after this disambiguation step.
Coordinates of the target object location and the placing destination location
are sent to the dexterous manipulation module.

###  3.3. Manipulation Module: Transition Stages

The transition stages (reaching and transporting) can be solved with model-
based motion planning, using a common procedure agnostic to scene arrangement:

  1. (1)

Compute the final arm pose that moves the hand close to target position for
next stage, using the task parameters extracted by the vision and language
modules;

  2. (2)

Plan a collision-free trajectory for the arm and hand (with the object in-hand
in case of transporting);

  3. (3)

Execute the planned trajectory with close-loop tracking control.

For our system, instead of finding a single destination pose, we produce a
distribution of destination poses, all of which have the hand being close to
the target position but with different palm facing orientations. Having
multiple choices for the final pose increases the chance for the motion
planner to yield a collision-free trajectory, alleviating failure cases where
the destination arm configuration is already in collision. Moreover, the
resultant motion exhibits richer diversity as the hand has multiple ways to
approach the object. The distribution of poses are solved with a parameterized
inverse kinematics (see Appendix A.2).

We solve for the collision-free trajectory with BiRRT algorithm [Kuffner and
LaValle, 2000], followed by shortening, smoothing [Hauser and Ng-Thow-Hing,
2010], and re-timing (Appendix A.2) to eliminate robot-like disjunct motions.
The re-timed trajectory is physically tracked with a PD control law and
inverse dynamics.

###  3.4. Manipulation Module: Interaction Stages

We want to develop generic grasping and placing/stacking controllers capable
of picking up objects of varying sizes, shapes, and locations, and placing
objects on other objects or surfaces. We apply model-free deep reinforcement
learning (DRL) to handle non-differentiable contacts involved, and fuse
multiple sensory modalities of vision, proprioception, and touch. Our
manipulation module can work without reasoning about grasping points, allowing
it to zero-shot generalize to irregular object shapes unseen during training
(Video 3m14s). We train one single grasping and one single placing/stacking
policy by randomizing target object shapes and locations during training. Such
randomization naturally leads the same policy to produce diverse behaviors.

Crucial to a successful concatenation of different stages, the initial
distribution of arm and finger pose during policy training should adequately
cover the space of possible scenarios that may result from its preceding
stage. For both grasping and stacking stages, the initial arm poses are
generated using the same procedure for generating final arm pose distributions
during reaching and transporting. However, there is an additional difficulty
in designing initial distribution for the stacking policy due to a unique
challenge of multi-finger hand manipulation. With a dexterous hand, the
relative transform between the hand and the object-in-hand has a considerable
variance among different grasps. Thus, to learn a robust placing/stacking
policy, we need to include such variance in the initial state distribution. As
such, we use the grasping policy to collect $10{,}000$ stable grasps, which we
use to randomly initialize the hand pose and the configuration of the object-
in-hand at the beginning of each training episode for the placing/stacking
policy. Small white noise is injected to capture likely object slips during
transportation.

Vision perception is crucial to placing/stacking, because this task requires
updated feedback about the object configuration in the context of the scene.
On the other hand, vision perception is less important for grasping as the
reaching stage already leads the hand close to the object using the vision-
estimated position from the initial scene. Thus, we use vision perception for
placing/stacking and depend solely on proprioception and tactile perception
during the grasping stage. Specifically, the observation space for the
placing/stacking policy additionally includes the current 3D position and
orientation of the involved objects as estimated by vision. For orientation,
we only require the direction of the axis that indicates the upright position
of the object (for spheres we simply use a unit-z vector). To mimic the
latency and low frequency of human visual perception system, we update the
vision input at $20$Hz and add $50$ milliseconds of delay in the vision input
to the policy. Since the training data of vision module is generated after the
stacking policy is trained, during training of stacking we simply feed the
policy with true object poses from simulation.

Technical details of the DRL modules are discussed in Appendix A.3.

##  4\. Experiments

In this section, we first evaluate whether the DASH system can balance between
robustness and anthropomorphism, for which we both ‘stress test’ our system on
$200$ randomly generated trials and evaluate the success rates of the full
system, and report qualitative evaluation on interactions with DASH. We also
report the performance statistics of DASH.

We then conduct experiments on DASH’s vision module and manipulation module to
evaluate our key system design decisions. We provide analysis on the accuracy
of vision module, showing that a standalone vision module facilitates analysis
and interpretation. We compare with the alternative design of a manipulation
module, showing that merging all stages in one policy would cause difficulty
in learning. We additionally show the zero-shot transfer capability of our
manipulation module to objects with unseen, irregular shapes, showing the
advantage of using close-loop, partially observable DRL policies which require
no contact point reasoning. We finish this section with qualitative examples
showing our manipulation module produces smooth motions with natural
variation, and can be easily extended to manipulation with the left arm and
hand.

###  4.1. Evaluations of the Whole System

We compare the system performance between using (i) ground truth object states
(upper bound) and (ii) vision-predicted object attributes as the input
observation to DASH using 200 randomly generated scenes (see Appendix B for
details of setup). For ground truth evaluation, DASH is provided with perfect
object attributes and 3D pose from the PyBullet simulator. For vision
evaluation, DASH is provided with imperfect estimations of the objects.

##### Quantitative results.

We show (Table 1) that DASH achieves high success rates on the large number of
random scenes. We report two metrics of success: (i) success when considering
all trials, and (ii) success when excluding trials that encountered a motion
planning failure. A trial is considered successful if the $(x,y)$ location of
the placed object is within 10 cm of the target $(x,y)$, and the $z$
coordinate of the center of mass is at least 5 cm above the target $z$
coordinate. Success rates of DASH when using vision-estimated observations are
within 5.8% of the success rates that use ground truth object states. Stacking
success rates are generally around 10 to 15% lower than placing success rates,
potentially due to higher sensitivity to observation accuracy.

We provide success rates excluding motion planning failures as additional
reference since our randomly generated scenes and tasks may sometimes lead to
motion planner unable to find a collision-free path during transition stages
(Figure 3). Potential improvements could be training DASH to grasp from top of
objects, and adding more Degrees of Freedoms (DoFs) around the shoulder
complex. In summary, motion plan failures and stacking policy failures are the
main sources of errors of our framework.

Table 1.  Success rates (%) of the full system on 100 placing and 100 stacking tasks. We report success rates when (i) including all trials and (ii) excluding trials that fail due to motion planning (Figure 3). Vision denotes using imperfect estimations of object attributes from the vision module as input to DASH. Ground truth denotes perfect observation of object attributes. (Video 1m10s-1m35s) | All Trials | Excluding Plan Failures
---|---|---
Observation | Overall | Place | Stack | Overall | Place | Stack
Vision | 76.0 | 82.0 | 70.0 | 85.8 | 89.1 | 82.4
Ground truth | 79.0 | 83.0 | 75.0 | 89.7 | 91.2 | 88.2
Figure 3. Motion planner during reaching stage unable to find a collision free
path to destination arm pose shown in transparent.

##### Qualitative results.

To evaluate DASH’s motion quality, the accompanying video shows four
successful examples of user-provided commands, mixing two-sentences and one-
sentence commands with different kinds of modifiers (0m12s-1m10s). We wrote
the instructions such that they cover the space specified by our assumptions
(target object, reference object(s), at most one modifier clause), with
variations in natural languages such as “stack on” vs “put on top of”. Example
failure cases are also provided in the accompanying video (2m35s-3m02s).

##### Performance.

Performance of the DASH system is measured on a personal desktop with Intel
Core i7-9700K processor and The NVIDIA GeForce RTX 2080 Ti graphics card. The
whole system runs at around 58 simulation steps per second. Rendering and
communications between PyBullet and Unity contribute to a significant portion
of the running time. Without Unity rendering, the manipulation module runs at
around 360 simulation steps per second.

###  4.2. Analyses of the Vision Module

Having a stand-alone vision module allows us to analyze its performance
independently. On the evaluation set ($20\%$ random split from the training
data), as shown in Table 2, the vision module errors during initial task
planning are much lower compared to the placing/stacking stage. This is likely
due to the fact that DASH uses a fixed camera pose during planning, but
varying camera pose to mimic human looking at destination during
placing/stacking. In addition, DASH’s arm is not in the camera view during the
planning stage. During the placing stage, DASH’s hand is in view, partially
occluding the object in hand as well as surrounding objects, making pose
estimation more challenging. Finally, during placing, the vision module and
the policy interact in a tight feedback loop, resulting in potential
compounding of errors over time. We also note that errors are generally lower
when considering only successful trials instead of all trials. This suggests
that the vision module could occasionally encounter states from failed placing
attempts that are outside of its training distribution.

Table 2. Vision module performance on the full system (test). We report classification accuracy on shape and color attributes, as well as the mean average error (MAE) on 3D position estimation (cm) across all predicted frames for the purpose of initial task planning, placing, or stacking. We show position prediction errors on all trials as well as when considering only successful trials. | Accuracy (%) | 3D Position MAE (cm)
---|---|---
Stage | Shape | Color | All Trials | Successful Trials
Task Plan | 100.0 | 100.0 | (0.8, 1.1, 0.4) | (0.8, 1.3, 0.4)
Place | 98.4 | 99.8 | (3.8, 3.2, 2.3) | (2.8, 2.7, 1.8)
Stack | 95.1 | 98.3 | (3.5, 2.9, 2.6) | (2.7, 2.4, 1.9)

###  4.3. Analyses of the Manipulation Module

##### Comparison with learning-only approach.

We show that stacking all four manipulation stages in one policy would cause
problems in learning (Video 4m01s). The learning-only approach formulates the
entire pick-and-place problem as a POMDP and solves for a single policy to
achieve all four stages of the task: reaching, grasping, transporting and
placing. In our attempt to train such a policy on a dexterous hand without any
motion demonstration, we found that the state-of-the-art learning method fails
at the very beginning of the task. It was unable to discover a collision-free
path to move the arm from a resting (straight down) position to one that is
above the tabletop. Subsequently, we significantly reduced the difficulty of
the problem to a manipulation task commonly addressed by prior art. The
simplified task only includes reaching and grasping, starting from a simple
initial state where a clear collision-free path is available. With such
simplification, we were able to successfully learn a policy using a learning-
only approach, similar to the findings from previous work. But on a similar
simplified task which includes reaching, grasping and stacking in place the
training still fails. These experiments confirm our hypothesis that a generic
pick-and-place task “in the wild” can pose great challenges to a DRL approach,
emphasizing the necessity of employing a hybrid approach of model-based
planning and model-free learning.

##### Generalization to novel, non-primitive shapes.

Without any re-training, we further demonstrate the grasp-and-stack task on
non-primitive objects including an hourglass, a cone, and a Stanford bunny
(Video 3m14s-3m40s). In these tasks we assume access to the object pose
information without use of the vision module. Results show that our policy
generalizes to novel, irregularly shaped geometries for the pick-and-place
task.

##### Emergent behaviors with natural variations.

Since both our grasping and placing policies are required to handle a large
range of task variation during training, multiple behaviors naturally emerge
from the same grasping or placing policy (Video 1m40s-2m16s). For instance,
the grasping policy occasionally rotates the target object slightly in order
to execute a more stable grasp. In other cases, if an object is unstable and
close to falling over during the grasping processes, the policy may switch to
grasping the bottom of the object as a rescue strategy. Similar variation of
behaviors can be observed for the placing/stacking policy as well. During
placing or stacking, if the object in hand is in an upright orientation, the
policy may choose to simply release the object from midair and allow the
object to drop onto the placing surface. In other scenarios, the policy may
choose to be conservative and wait for the object placing to finish before
releasing its hand. If a placed object appears to be on the verge of falling
over, the hand often maintains contact with object in an effort to restabilize
it.

##### Effect of reshaping and retiming.

We show in the accompanying video (2m22s) that our reshaping and retiming
technique (Figure 5) creates more smooth reaching motions, compared with
directly using the trajectory solved by the robotics package OpenRave. A
comparison of the palm velocity profiles is also given in Figure 4.

Figure 4. Velocity profiles of the palm during reaching to object, with our
retiming/reshaping technique, or using the OpenRave solved trajectory without
editing.

##### Extension: dual-arm manipulation.

In the accompanying video (3m45s), we show that the same policies trained for
the right arm can be directly mirrored and used for the left arm.

##  5\. Discussion and Conclusion

This paper introduces DASH, a first system for simulating human manipulation
skills with humanlike embodied constraints and language interaction. We show
that it is possible to build an embodied virtual human that solely uses its
own visual perception, natural language parsing, proprioception, and touch to
perform grasp-and-stack tasks in a physically-simulated, cluttered
environment. By factoring the DASH system into a vision module, a language
module and manipulation modules of two skill categories, we are able to mix
and match analytical and machine learning techniques for different modules, so
that DASH is able to not only perform randomly arranged tasks with a high
success rate, but also does so under incomplete information and with fluid and
diverse motions.

Our system does not depend on recorded human motion, hence having the
potential to be used for both engineering applications such as physical human-
robot interaction, and scientific applications such as the cognitive modeling
of how embodied constraints affect human motions. By carefully concatenating
DRL manipulation policies, our system can work without reasoning about
grasping points, allowing it to zero-shot generalize to irregular object
shapes unseen during training.

As the focus of the DASH system is to explore the inclusion of various
humanlike constraints, we are not able to use certain existing techniques to
improve motion quality, such as grasping algorithms that utilize explicit
geometry and contact point analysis. That said, the motion quality of the
current version of DASH can be much improved by allowing the torso to move and
adding more DoFs at the shoulder complex. This will also enlarge the reachable
area [Rodriguez et al., 2003], and could help the inverse kinematics solver to
avoid finding poses with large wrist bending.

There are many possible choices for the vision module design. We choose
egocentric RGB vision as it is closer to the human vision system. Alternative
designs such as third-person point clouds reconstructed from dual cameras
could have made the system easier to build. More scientific knowledge can be
injected to the vision module to make it more humanlike. For example, the
location uncertainty of human vision is not only distance dependent, but also
depends on nearby reference objects including the reaching hand.

DASH is currently limited to grasping and placing/stacking tasks, but the
modular design facilitates extensions such as concatenating additional DRL
policies for more complex skills. Just as humans simultaneously learn
perception and motor skills, interleaving stacking and vision training
(instead of only once in the current system) could further improve the
robustness of the system.

Even though the use of partially-observable DRL policies allows our
manipulation module to transfer easily to unseen shapes, achieving the same
transfer for vision module is much more challenging. One future direction is
to train the vision module with a large set of everyday objects, randomly
arranged on the tabletop. Finally, while our language module is robust to some
wording (e.g. “put on” versus “stack it on”) and natural ordering variations,
it will not handle ambiguous commands. This can be addressed by allowing DASH
to ask domain-specific questions when instructions are ambiguous.

###### Acknowledgements.

The authors would like to thank Michael Hayashi for mirroring the trained
policies for the dual-arm setting.

## References

  * [1]
  * Anderson et al. [1997] John R Anderson, Michael Matessa, and Christian Lebiere. 1997\.  ACT-R: A theory of higher level cognition and its relation to visual attention.  _Human–Computer Interaction_ 12, 4 (1997), 439–462.
  * Andrews and Kry [2012] Sheldon Andrews and Paul G. Kry. 2012.  Policies for Goal Directed Multi-Finger Manipulation. In _Workshop on Virtual Reality Interaction and Physical Simulation_ , Jan Bender, Arjan Kuijper, Dieter W. Fellner, and Eric Guerin (Eds.). The Eurographics Association.  https://doi.org/10.2312/PE/vriphys/vriphys12/137-145
  * Andrychowicz et al. [2020] Marcin Andrychowicz, Bowen Baker, Maciek Chociej, Rafal Jozefowicz, Bob McGrew, Jakub Pachocki, Arthur Petron, Matthias Plappert, Glenn Powell, Alex Ray, et al. 2020\.  Learning dexterous in-hand manipulation.  _The International Journal of Robotics Research_ 39, 1 (2020), 3–20.
  * Antoniades [2016] Tameem Antoniades. 2016\.  Creating a Live Real-Time Performance-Captured Digital Human. In _ACM SIGGRAPH 2016 Real-Time Live!_ (Anaheim, California) _(SIGGRAPH ’16)_. Association for Computing Machinery, New York, NY, USA, Article 36, 1 pages.  https://doi.org/10.1145/2933540.2933551
  * Berant et al. [2013] Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013\.  Semantic parsing on freebase from question-answer pairs. In _Proceedings of the 2013 conference on empirical methods in natural language processing (EMNLP)_. 1533–1544.
  * Bicchi [2000] Antonio Bicchi. 2000\.  Hands for dexterous manipulation and robust grasping: A difficult road toward simplicity.  _IEEE Transactions on robotics and automation_ 16, 6 (2000), 652–662.
  * Buss [2004] Samuel R Buss. 2004\.  Introduction to inverse kinematics with jacobian transpose, pseudoinverse and damped least squares methods.  _IEEE Journal of Robotics and Automation_ 17, 1-19 (2004), 16\.
  * Caicedo and Thalmann [2000] Angela Caicedo and Daniel Thalmann. 2000.  _Virtual Humanoids: Let them be autonomous without losing control_.  Technical Report. Swiss Federal Institute of Technology, EPFL.
  * Cassell et al. [2000] Justine Cassell, Joseph Sullivan, Elizabeth Churchill, and Scott Prevost. 2000.  _Embodied conversational agents_.  MIT press.
  * Chaudhuri et al. [2020] Siddhartha Chaudhuri, Daniel Ritchie, Jiajun Wu, Kai Xu, and Hao Zhang. 2020\.  Learning generative models of 3D structures. In _Computer Graphics Forum_ , Vol. 39. Wiley Online Library, 643–666.
  * Chi et al. [2000] Diane Chi, Monica Costa, Liwei Zhao, and Norman Badler. 2000\.  The EMOTE Model for Effort and Shape. In _Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques_ _(SIGGRAPH ’00)_. ACM Press/Addison-Wesley Publishing Co., USA, 173–182.
  * Coumans and Bai [2019] Erwin Coumans and Yunfei Bai. 2016–2019.  PyBullet, a Python module for physics simulation for games, robotics and machine learning.  http://pybullet.org.
  * Diankov [2010] Rosen Diankov. 2010\.  _Automated Construction of Robotic Manipulation Programs_.  Ph.D. Dissertation. Carnegie Mellon University, Robotics Institute.
  * Eom et al. [2019] Haegwang Eom, Daseong Han, Joseph S Shin, and Junyong Noh. 2019\.  Model Predictive Control with a Visuomotor System for Physics-based Character Animation.  _ACM Transactions on Graphics (TOG)_ 39, 1 (2019), 1–11.
  * Erickson et al. [2019] Zackory Erickson, Vamsee Gangaram, Ariel Kapusta, C Karen Liu, and Charles C Kemp. 2019.  Assistive Gym: A Physics Simulation Framework for Assistive Robotics.  _arXiv preprint arXiv:1910.04700_ (2019).
  * Feng et al. [2014] Andrew Feng, Yazhou Huang, Yuyu Xu, and Ari Shapiro. 2014\.  Fast, automatic character animation pipelines.  _Computer Animation and Virtual Worlds_ 25, 1 (2014), 3–16.
  * Funge et al. [1999] John Funge, Xiaoyuan Tu, and Demetri Terzopoulos. 1999\.  Cognitive modeling: knowledge, reasoning and planning for intelligent characters. In _Proceedings of the 26th annual conference on Computer graphics and interactive techniques_. 29–38.
  * Girshick et al. [2018] Ross Girshick, Ilija Radosavovic, Georgia Gkioxari, Piotr Dollár, and Kaiming He. 2018\.  Detectron.  https://github.com/facebookresearch/detectron
  * Guu et al. [2017] Kelvin Guu, Panupong Pasupat, Evan Liu, and Percy Liang. 2017\.  From Language to Programs: Bridging Reinforcement Learning and Maximum Marginal Likelihood. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL)_. 1051–1062.
  * Hartholt et al. [2013] Arno Hartholt, David Traum, Stacy C Marsella, Ari Shapiro, Giota Stratou, Anton Leuski, Louis-Philippe Morency, and Jonathan Gratch. 2013.  All together now. In _International Workshop on Intelligent Virtual Agents_. Springer, 368–381.
  * Hatori et al. [2018] Jun Hatori, Yuta Kikuchi, Sosuke Kobayashi, Kuniyuki Takahashi, Yuta Tsuboi, Yuya Unno, Wilson Ko, and Jethro Tan. 2018\.  Interactively picking real-world objects with unconstrained spoken language instructions. In _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 3774–3781.
  * Hauser and Ng-Thow-Hing [2010] Kris Hauser and Victor Ng-Thow-Hing. 2010.  Fast smoothing of manipulator trajectories using optimal bounded-acceleration shortcuts. In _2010 IEEE international conference on robotics and automation_. IEEE, 2493–2498.
  * He et al. [2017] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017.  Mask r-cnn. In _Proceedings of the IEEE international conference on computer vision_. 2961–2969.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Ho et al. [2010] Edmond S. L. Ho, Taku Komura, and Chiew-Lan Tai. 2010\.  Spatial Relationship Preserving Character Motion Adaptation.  _ACM Trans. Graph._ 29, 4, Article 33 (2010), 8 pages.
  * Honnibal and Montani [2017] Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. (2017).
  * Jain and Liu [2009] Sumit Jain and C. Karen Liu. 2009.  Interactive Synthesis of Human-Object Interaction. In _Proceedings of the 2009 ACM SIGGRAPH/Eurographics Symposium on Computer Animation_ (New Orleans, Louisiana) _(SCA ’09)_. Association for Computing Machinery, New York, NY, USA, 47–53.
  * Jiang et al. [2018] Yuqian Jiang, Shiqi Zhang, Piyush Khandelwal, and Peter Stone. 2018.  Task Planning in Robotics: an Empirical Comparison of PDDL-based and ASP-based Systems.  _arXiv preprint arXiv:1804.08229_ (2018).
  * Jung et al. [2011] Yvonne Jung, Arjan Kuijper, Dieter W. Fellner, Michael Kipp, Jan Miksatko, Jonathan Gratch, and Daniel Thalmann. 2011.  Believable Virtual Characters in Human-Computer Dialogs. In _Eurographics 2011 - State of the Art Reports_ , N. John and B. Wyvill (Eds.). The Eurographics Association.
  * Kalra et al. [1998] Prem Kalra, Nadia Magnenat-Thalmann, Laurent Moccozet, Gael Sannier, Amaury Aubel, and Daniel Thalmann. 1998.  Real-time animation of realistic virtual humans.  _IEEE Computer Graphics and Applications_ 18, 5 (1998), 42–56.
  * Kiperwasser and Goldberg [2016] Eliyahu Kiperwasser and Yoav Goldberg. 2016.  Simple and accurate dependency parsing using bidirectional LSTM feature representations.  _Transactions of the Association for Computational Linguistics_ 4 (2016), 313–327.
  * Krishnan-Barman et al. [2017] Sujatha Krishnan-Barman, Paul AG Forbes, and Antonia F de C Hamilton. 2017.  How can the study of action kinematics inform our understanding of human social interaction?  _Neuropsychologia_ 105 (2017), 101–110.
  * Kry and Pai [2006] Paul G. Kry and Dinesh K. Pai. 2006.  Interaction capture and synthesis.  _ACM Trans. Graph._ 25, 3 (2006), 872–880.
  * Kuffner and LaValle [2000] James J Kuffner and Steven M LaValle. 2000.  RRT-connect: An efficient approach to single-query path planning. In _Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065)_ , Vol. 2. IEEE, 995–1001.
  * Kuhn [1955] Harold W Kuhn. 1955\.  The Hungarian method for the assignment problem.  _Naval research logistics quarterly_ 2, 1-2 (1955), 83–97.
  * Kulkarni et al. [2015] Tejas D Kulkarni, William F Whitney, Pushmeet Kohli, and Joshua B Tenenbaum. 2015.  Deep convolutional inverse graphics network. In _Advances in neural information processing systems (NeurIPS)_. 2539–2547.
  * Lee and Marsella [2006] Jina Lee and Stacy Marsella. 2006.  Nonverbal Behavior Generator for Embodied Conversational Agents. In _Proceedings of the 6th International Conference on Intelligent Virtual Agents_ (Marina Del Rey, CA) _(IVA’06)_. Springer-Verlag, Berlin, Heidelberg, 243–255.
  * Lee et al. [2019] Seunghwan Lee, Moonseok Park, Kyoungmin Lee, and Jehee Lee. 2019\.  Scalable Muscle-Actuated Human Simulation and Control.  _ACM Trans. Graph._ 38, 4, Article 73 (July 2019), 13 pages.
  * Liang et al. [2013] Percy Liang, Michael I Jordan, and Dan Klein. 2013.  Learning dependency-based compositional semantics.  _Computational Linguistics_ 39, 2 (2013), 389–446.
  * Lin et al. [2017] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017\.  Feature pyramid networks for object detection. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2117–2125.
  * Liu [2009] C. Karen Liu. 2009\.  Dextrous Manipulation from a Grasping Pose. In _ACM SIGGRAPH 2009 Papers_ (New Orleans, Louisiana) _(SIGGRAPH ’09)_. Association for Computing Machinery, New York, NY, USA, Article 59, 6 pages.
  * Liu and Zhang [2019] Rui Liu and Xiaoli Zhang. 2019.  A review of methodologies for natural-language-facilitated human–robot cooperation.  _International Journal of Advanced Robotic Systems_ 16, 3 (2019), 1729881419851402\.
  * Ma and Dollar [2011] Raymond R Ma and Aaron M Dollar. 2011.  On dexterity and dexterous manipulation. In _2011 15th International Conference on Advanced Robotics (ICAR)_. IEEE, 1–7.
  * Magnenat-Thalmann and Egges [2006] Nadia Magnenat-Thalmann and Arjan Egges. 2006.  Interactive virtual humans in real-time virtual environments.  _IJVR_ 5, 2 (2006), 15–24.
  * Matsuyama et al. [2016] Yoichi Matsuyama, Arjun Bhardwaj, Ran Zhao, Oscar Romeo, Sushma Akoju, and Justine Cassell. 2016\.  Socially-aware animated intelligent personal assistant agent. In _Proceedings of the 17th annual meeting of the special interest group on discourse and dialogue_. 224–227.
  * Mees et al. [2020] Oier Mees, Alp Emek, Johan Vertens, and Wolfram Burgard. 2020\.  Learning Object Placements For Relational Instructions by Hallucinating Scene Representations.  _arXiv preprint arXiv:2001.08481_ (2020).
  * Merel et al. [2019] Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. 2019.  Reusable neural skill embeddings for vision-guided whole body movement and object manipulation.  _arXiv preprint arXiv:1911.06636_ (2019).
  * Merel et al. [2020] Josh Merel, Saran Tunyasuvunakool, Arun Ahuja, Yuval Tassa, Leonard Hasenclever, Vu Pham, Tom Erez, Greg Wayne, and Nicolas Heess. 2020.  Catch and Carry: Reusable Neural Controllers for Vision-guided Whole-Body Tasks.  _ACM Trans. Graph._ 39 (July 2020).
  * Merzić et al. [2019] Hamza Merzić, Miroslav Bogdanović, Daniel Kappler, Ludovic Righetti, and Jeannette Bohg. 2019\.  Leveraging contact forces for learning to grasp. In _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 3615–3621.
  * Mordatch et al. [2012] Igor Mordatch, Zoran Popoviundefined, and Emanuel Todorov. 2012\.  Contact-Invariant Optimization for Hand Manipulation. In _Proceedings of the ACM SIGGRAPH/Eurographics Symposium on Computer Animation_ (Lausanne, Switzerland) _(SCA ’12)_. Eurographics Association, Goslar, DEU, 137–144.
  * Nagabandi et al. [2019] Anusha Nagabandi, Kurt Konoglie, Sergey Levine, and Vikash Kumar. 2019.  Deep Dynamics Models for Learning Dexterous Manipulation.  _arXiv preprint arXiv:1909.11652_ (2019).
  * Nakada et al. [2018] Masaki Nakada, Tao Zhou, Honglin Chen, Tomer Weiss, and Demetri Terzopoulos. 2018.  Deep Learning of Biomimetic Sensorimotor Control for Biomechanical Human Animation.  _ACM Trans. Graph._ 37, 4, Article 56 (July 2018), 15 pages.
  * Nguyen et al. [2015] Truong-Huy D Nguyen, Elin Carstensdottir, Nhi Ngo, Magy Seif El-Nasr, Matt Gray, Derek Isaacowitz, and David Desteno. 2015.  Modeling warmth and competence in virtual characters. In _International Conference on Intelligent Virtual Agents_. Springer, 167–180.
  * Nguyen-Phuoc et al. [2019] Thu Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. 2019.  Hologan: Unsupervised learning of 3d representations from natural images. In _Proceedings of the IEEE International Conference on Computer Vision (ICCV)_. 7588–7597.
  * Okamura et al. [2000] Allison M Okamura, Niels Smaby, and Mark R Cutkosky. 2000\.  An overview of dexterous manipulation. In _Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065)_ , Vol. 1. IEEE, 255–262.
  * OpenAI et al. [2019] OpenAI, Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. 2019\.  Solving Rubik’s Cube with a Robot Hand.  _arXiv preprint arXiv:1910.07113_ (2019).
  * Paxton et al. [2019] Chris Paxton, Yonatan Bisk, Jesse Thomason, Arunkumar Byravan, and Dieter Foxl. 2019.  Prospection: Interpretable plans from language by predicting the future. In _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 6942–6948.
  * Peng et al. [2018] Xue Bin Peng, Angjoo Kanazawa, Jitendra Malik, Pieter Abbeel, and Sergey Levine. 2018.  SFV: Reinforcement Learning of Physical Skills from Videos.  _ACM Trans. Graph._ 37, 6, Article 178 (Nov. 2018), 14 pages.
  * Perera et al. [2018] Ian Perera, James Allen, Choh Man Teng, and Lucian Galescu. 2018\.  Building and learning structures in a situated blocks world through deep language understanding. In _Proceedings of the First International Workshop on Spatial Language Understanding_. 12–20.
  * Peters and O’Sullivan [2002] Christopher Peters and Carol O’Sullivan. 2002.  Synthetic vision and memory for autonomous virtual humans. In _Computer Graphics Forum_ , Vol. 21. Wiley Online Library, 743–752.
  * Pollard and Zordan [2005] Nancy S. Pollard and Victor Brian Zordan. 2005.  Physically Based Grasping Control from Example. In _Proceedings of the 2005 ACM SIGGRAPH/Eurographics Symposium on Computer Animation_ (Los Angeles, California) _(SCA ’05)_. Association for Computing Machinery, New York, NY, USA, 311–318.
  * Puig et al. [2018] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. 2018.  Virtualhome: Simulating household activities via programs. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_. 8494–8502.
  * Rabie and Terzopoulos [2000] Tamer F Rabie and Demetri Terzopoulos. 2000.  Active perception in virtual humans. In _Vision Interface_ , Vol. 2000.
  * Radosavovic et al. [2020] Ilija Radosavovic, Xiaolong Wang, Lerrel Pinto, and Jitendra Malik. 2020.  State-Only Imitation Learning for Dexterous Manipulation.  _arXiv preprint arXiv:2004.04650_ (2020).
  * Rajeswaran et al. [2017] Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. 2017.  Learning complex dexterous manipulation with deep reinforcement learning and demonstrations.  _arXiv preprint arXiv:1709.10087_ (2017).
  * Rickel and Johnson [1999] Jeff Rickel and W Lewis Johnson. 1999.  Animated agents for procedural training in virtual reality: Perception, cognition, and motor control.  _Applied artificial intelligence_ 13, 4-5 (1999), 343–382.
  * Rodriguez et al. [2003] I Rodriguez, Manuel Peinado, Ronan Boulic, and Daniel Meziat. 2003.  Bringing the human arm reachable space to a virtual environment for its analysis. In _2003 International Conference on Multimedia and Expo. ICME’03. Proceedings (Cat. No. 03TH8698)_ , Vol. 1. IEEE, I–229.
  * Rothe et al. [2017] Anselm Rothe, Brenden M Lake, and Todd Gureckis. 2017\.  Question asking as program generation. In _Advances in neural information processing systems (NeurIPS)_. 1046–1055.
  * Savva et al. [2019] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, et al. 2019\.  Habitat: A platform for embodied ai research. In _Proceedings of the IEEE International Conference on Computer Vision_. 9339–9347.
  * Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ (2017).
  * Shapiro [2011] Ari Shapiro. 2011\.  Building a character animation system. In _International conference on motion in games_. Springer, 98–109.
  * Shridhar and Hsu [2018] Mohit Shridhar and David Hsu. 2018.  Interactive visual grounding of referring expressions for human-robot interaction.  _arXiv preprint arXiv:1806.03831_ (2018).
  * Shridhar et al. [2020] Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. 2020\.  Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_. 10740–10749.
  * Sitzmann et al. [2019] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. 2019\.  Scene representation networks: Continuous 3D-structure-aware neural scene representations. In _Advances in Neural Information Processing Systems (NeurIPS)_. 1119–1130.
  * Smith and Gasser [2005] Linda Smith and Michael Gasser. 2005.  The development of embodied cognition: Six lessons from babies.  _Artificial life_ 11, 1-2 (2005), 13–29.
  * Tewari et al. [2020] Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, Rohit Pandey, Sean Fanello, Gordon Wetzstein, Jun-Yan Zhu, Christian Theobalt, Maneesh Agrawala, Eli Shechtman, Dan B. Goldman, and Michael Zollhöfer. 2020.  State of the Art on Neural Rendering.  _Computer Graphics Forum_ (2020).  https://doi.org/10.1111/cgf.14022
  * Thiebaux et al. [2008] Marcus Thiebaux, Stacy Marsella, Andrew N Marshall, and Marcelo Kallmann. 2008.  Smartbody: Behavior realization for embodied conversational agents. In _Proceedings of the 7th international joint conference on Autonomous agents and multiagent systems-Volume 1_. 151–158.
  * Wang et al. [2013] Yangang Wang, Jianyuan Min, Jianjie Zhang, Yebin Liu, Feng Xu, Qionghai Dai, and Jinxiang Chai. 2013.  Video-Based Hand Manipulation Capture through Composite Motion Control.  _ACM Trans. Graph._ 32, 4, Article 43 (July 2013), 14 pages.
  * Weihs et al. [2020] Luca Weihs, Jordi Salvador, Klemen Kotar, Unnat Jain, Kuo-Hao Zeng, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2020.  AllenAct: A Framework for Embodied AI Research.  _arXiv_ (2020).
  * Wu et al. [2017] Jiajun Wu, Joshua B Tenenbaum, and Pushmeet Kohli. 2017\.  Neural scene de-rendering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. 699–707.
  * Xia et al. [2020] Fei Xia, William B Shen, Chengshu Li, Priya Kasimbeg, Micael Edmond Tchapmi, Alexander Toshev, Roberto Martín-Martín, and Silvio Savarese. 2020.  Interactive Gibson Benchmark: A Benchmark for Interactive Navigation in Cluttered Environments.  _IEEE Robotics and Automation Letters_ 5, 2 (2020), 713–720.
  * Yamane et al. [2004] Katsu Yamane, James J. Kuffner, and Jessica K. Hodgins. 2004\.  Synthesizing Animations of Human Manipulation Tasks. In _ACM SIGGRAPH 2004 Papers_ (Los Angeles, California) _(SIGGRAPH ’04)_. Association for Computing Machinery, New York, NY, USA, 532–539.
  * Ye and Liu [2012] Yuting Ye and C. Karen Liu. 2012.  Synthesis of Detailed Hand Manipulations Using Contact Sampling.  _ACM Trans. Graph._ 31, 4, Article 41 (July 2012), 10 pages.
  * Yi et al. [2018] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenenbaum. 2018.  Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. In _Advances in Neural Information Processing Systems (NeurIPS)_. 1031–1042.
  * Zhao et al. [2013] Wenping Zhao, Jianjie Zhang, Jianyuan Min, and Jinxiang Chai. 2013.  Robust Realtime Physics-Based Motion Control for Human Grasping.  _ACM Trans. Graph._ 32, 6, Article 207 (Nov. 2013), 12 pages.

##  Appendix A System Implementations

In this section we present additional technical material that is important to
the success of the DASH system.

###  A.1. Vision and Language Modules

####  A.1.1. Vision module.

We use Mask R-CNN [He et al., 2017] with a ResNet-50 FPN [Lin et al., 2017]
backbone as our segmentation model, implemented using the “Detectron” library
[Girshick et al., 2018]. The model is trained for 100K iterations with a batch
size of two and learning rate of $2.5\times 10^{-4}$. For our feature
extraction model, we train a ResNet-18 [He et al., 2016] for each of the
planning, placing, and stacking stages for 60K iterations using a learning
rate of $0.002$ and batch size of 20. The model inputs are resized to 480 by
480.

The model is trained in camera coordinates; at test time, the predictions from
the vision module are transformed from the camera coordinate frame into the
world coordinate frame using the camera pose. We then use the Kuhn-Munkres
(Hungarian) algorithm [Kuhn, 1955] to associate objects predicted across time,
based on their predicted attributes.

We use the Unity 3D game engine111Unity game engine website:
https://unity3d.com/ to render training images and object segmentations. The
camera is placed near the eyes of DASH to provide vision observation. At the
initial task planning (before manipulation starts), the DASH camera points at
a fixed location such that the camera view covers the main region that
manipulable objects lie. The information predicted from this first image is
used to generate a task plan jointly with language module. During
placing/stacking stage, the camera points at the destination $x,y$ location
predicted during task planning. The camera target locations during training
are computed using ground truth object locations with 2 centimeter noise for
robustness.

The vision module is trained on frames from both the initial tabletop scene
and the placing stage to serve its two usages. We generate 20K training images
of initial tabletop scenes following the same procedure as that of our full
system evaluation pipeline (see Appendix B). To generate examples from the
placing stage, we collect 4,000 rollouts from a pretrained placing/stacking
policy. For this step of data generation, the policy utilizes perfect object
pose information from simulator, with an additional 2 centimeter noise on
object positions and 0.03 noise on up vectors. From all frames generated from
the policy rollouts, we uniformly downsample 40K frames for vision training.

####  A.1.2. Language module.

We used the neural network based dependency parser proposed by Kiperwasser and
Goldberg [2016] which is robust to noise in natural language, and its open
source implementation in spaCy [Honnibal and Montani, 2017]. For the breadth-
first search, we pre-define a dictionary of shape, color, and spatial relation
keywords. There might exist multiple objects sharing same shape and color in
the visual scene. To disambiguate, we then search in the subtrees of the
target object and of the primary reference object token for secondary spatial
relations and secondary reference object(s) that modify the target or
reference object.

###  A.2. Manipulation Module: Transition Stages

The manipulation controller operates on an anthropomorphic arm and hand
including four fingers, a thumb, and a palm, totaling $29$ degrees of freedom.
For clarity of exposition, we will refer the degrees of freedom (dofs) from
the shoulder to the palm as $\boldsymbol{q}^{A}\in\mathbb{R}^{7}$ and the dofs
for finger joints as $\boldsymbol{q}^{F}\in\mathbb{R}^{22}$. We only simulate
and control DASH’s right arm and right hand in main experiments, but also show
that the same manipulation controller can be easily mirrored to the left arm
and hand. We use the PyBullet physics engine [Coumans and Bai, 2019] as the
simulator.

####  A.2.1. Compute the final arm pose.

From the initial image of the scene and the language instruction, the vision
and language modules jointly infer the high-level pick-and-place task
parameters, including the index of the target object, its initial position
$\bar{\boldsymbol{p}}_{i}\in\mathbb{R}^{3}$, its desired position
$\bar{\boldsymbol{p}}_{d}\in\mathbb{R}^{3}$, and estimated initial locations
and shapes of all objects in the scene. Our first goal is to find the final
arm pose $\bar{\boldsymbol{q}}^{A}_{T}$ such that the palm is close to the
initial position $\bar{\boldsymbol{p}}_{i}$ at the end of the reaching stage,
and close to the desired position $\bar{\boldsymbol{p}}_{d}$ at the end of the
transporting stage.

To solve a distribution of final poses for the arm, we consider an Inverse
Kinematics (IK) problem that matches the position and orientation of the palm
in the world frame to a desired affine transformation $\boldsymbol{T}$,
parameterized by a random variable $\theta$,

(1)
$\boldsymbol{T}(\theta)=\begin{bmatrix}\boldsymbol{R}_{z}(\theta)&\boldsymbol{p}\\\
\boldsymbol{0}&1\end{bmatrix}\boldsymbol{T}_{0},$

where $\boldsymbol{p}$ is assigned to the initial or desired position
$\bar{\boldsymbol{p}}_{i}$ or $\bar{\boldsymbol{p}}_{d}$ depending on the
stage, $\boldsymbol{R}_{z}(\theta)$ is a rotation matrix about z-axis (upright
axis) by angle $\theta$, and $\boldsymbol{T}_{0}$ is a constant transformation
from the object frame to the palm frame, manually designed for each task. For
example, for the reaching stage, $\boldsymbol{T}_{0}$ positions the right palm
at $20$cm to the right of the object, with the palm facing the object. We
uniformly sample $\theta\in[0,2\pi]$ to create multiple candidates for the
final arm pose. The implication is that we can grasp and hold the object from
multiple orientations by its side, but not from its top. This multiple
candidate IK problem is iterated by candidate and each solved with [Buss,
2004]. The selected final arm pose $\bar{\boldsymbol{q}}^{A}_{T}$ is a
collision-free pose with the wrist closest to its rest angle.

Figure 5. Illustration: Reaching trajectory of the arm/palm solved from
OpenRave is reshaped and retimed to have non-zero velocity before the grasping
motion starts, making the stage switching more natural.

####  A.2.2. Plan a collision-free trajectory.

Once the final arm pose $\bar{\boldsymbol{q}}^{A}_{T}$ is determined, the next
task is to plan a smooth trajectory for the arm and the hand from the initial
pose given by the previous stage (for the first reaching stage we simply use
the rest pose) to $\bar{\boldsymbol{q}}^{A}_{T}$, while avoiding collision
with objects in the scene.

There exist many efficient and reliable motion planning algorithms from
robotics literature. We apply BiRRT [Kuffner and LaValle, 2000] to first
search for a collision-free path and then shorten and smoothen the path using
techniques proposed by [Hauser and Ng-Thow-Hing, 2010]. We use the open-source
implementation in OpenRave [Diankov, 2010] to obtain a collision-free
trajectory $\bar{\boldsymbol{q}}^{A}_{1:T}$.

The trajectory solved by OpenRave assumes zero terminal palm velocity since
this is the usual convention in robotics systems. While this is fine for the
transition between transporting and stacking, complete stop of palm at the
switching from reaching to grasping would result in unnatural, robot-like
motions. As OpenRave does not control the direction of the terminal tangent
(because of zero terminal speed), we first reshape the OpenRave trajectory by
replacing the final 10% with a Hermite curve, so that the terminal tangent
points from the palm towards the object; then a retiming is performed on the
edited trajectory to fit a quadratic velocity profile, mimicking that human
arm first accelerates then decelerates during reaching [Krishnan-Barman et
al., 2017]. We set the terminal palm speed to $0.4\textrm{m/s}$. Figure 5
shows an illustration of this procedure.

The finger joints trajectory $\bar{\boldsymbol{q}}^{F}_{1:T}$ holds the same
manually designed pose in Figure 2 (b) throughout reaching, and the same
grasping pose (see next paragraph) throughout transporting. During planning,
we replace the hand (and the object-in-hand) with a conservative bounding box
to speed up the computation.

####  A.2.3. Simulate the planned trajectory.

To track the planned trajectory $\bar{\boldsymbol{q}}^{A}_{1:T}$ in a physics
simulation, we first compute the desired velocity $\dot{\boldsymbol{q}}^{+}$
in the next time step using a PD control law,

(2)
$\dot{\boldsymbol{q}}^{+}=-K_{p}(\boldsymbol{q}_{t}-\bar{\boldsymbol{q}}_{t})-(K_{d}-1)\dot{\boldsymbol{q}}_{t},$

where $\boldsymbol{q}_{t}$ and $\dot{\boldsymbol{q}}_{t}$ are the joint state
at current time step. The desired velocity is realized by solving for the
appropriate control torque via Inverse Dynamics, and then sending solved
torques to joint motors for forward simulation in Bullet. Note that
$\dot{\boldsymbol{q}}^{+}$ is not always achieved due to joint limits, torque
limits, and the presence of environment contacts. The stability of simulation
is not sensitive to the gains $K_{p}$ and damping coefficients $K_{d}$, as a
result of solving torque from Inverse Dynamics.

During the transporting stage, we must ensure that the grasped object does not
fall out of the hand. Since the contact force applied to the object is due to
the tracking error of the hand pose, the target hand pose should be designed
in a way such that it induces effective forces to secure the object and is
unachievable during the transporting phase (so the hand will continuously
apply contact force to the object). Instead of manually designing such a pose,
we use the sign of the tracking error in the last time step during the
grasping stage,
$\boldsymbol{e}=\operatorname{sign}(\boldsymbol{\bar{q}}^{F}_{g}-\boldsymbol{q}^{F}_{g})$,
to define the target pose during transporting as
$\bar{\boldsymbol{q}}^{F}_{tran}=\boldsymbol{q}^{F}_{g}+0.1\boldsymbol{e}$.

###  A.3. Manipulation Module: Interaction Stages

We train a policy $\pi(\boldsymbol{a}_{t}|\boldsymbol{o}_{t})$ that maps the
agent’s current observation $\boldsymbol{o}_{t}$ to its action
$\boldsymbol{a}_{t}$ every $k=6$ simulation steps. We formulate the problem of
grasping and placing or stacking objects as a Partially-Observable Markov
Decision Process $(\mathcal{S},\mathcal{O},\mathcal{A},f,r,p_{0},g,\gamma)$,
where $\mathcal{S}$ is the state space, $\mathcal{O}$ is the observation
space, $\mathcal{A}$ is the action space, $f(\cdot)$ is the system dynamics,
$r(\cdot)$ is the reward function, $g(\cdot)$ is the observation function,
$p_{0}$ is the initial state distribution, and $\gamma$ is the discount
factor. We use a policy gradient method, PPO [Schulman et al., 2017], to solve
for a policy $\pi$ such that the accumulated reward is maximized:

(3)
$J(\pi)=\mathbb{E}_{\mathbf{s}_{0},\mathbf{a}_{0},\dots,\mathbf{s}_{T}}\sum_{t=0}^{T}\gamma^{t}r(\mathbf{s}_{t},\mathbf{a}_{t}),$

where $\mathbf{s}_{0}\sim p_{0}$,
$\mathbf{a}_{t}\sim\pi(\mathbf{a}_{t}|\mathbf{o}_{t})$,
$\boldsymbol{o}_{t}\sim g(\boldsymbol{s}_{t})$ and
$\mathbf{s}_{t+1}=f(\mathbf{s}_{t},\mathbf{a}_{t})$. Unlike model-based motion
planning, reinforcement learning requires offline training of the policy.
While the observations will be partial and noisy, we assume during training
the reward function additionally has access to the complete, accurate
simulator state $\boldsymbol{s}_{t}$ to better evaluate training progress.

####  A.3.1. Observation space and action space.

The observation space contains information from proprioception and tactile
sensors, as well as information about the task. The stacking policy
additionally uses frequent vision module estimates, as discussed in Section
3.4.

The proprioception consists of the current state and the target pose from the
previous time step: ($\boldsymbol{q}_{t}$, $\boldsymbol{\dot{q}}^{A}_{t}$,
$\boldsymbol{\bar{q}}_{t-1}$). Empirically, we found that finger velocity
$\boldsymbol{\dot{q}}^{F}_{t}$ does not improve performance, and its wide
range and sensitivity to contacts hurts generalization. The tactile perception
on the hand is represented as a vector of binary numbers $\boldsymbol{c}_{t}$,
each of which denotes whether the corresponding phalanx is in contact with an
object. The task information contains the location of interaction, i.e.,
$\bar{\boldsymbol{p}}_{i}$ for grasping and $\bar{\boldsymbol{p}}_{d}$ for
placing/stacking, and a one-hot vector representing the shape of the object
(e.g., cylinder, box, or sphere) to grasp, place, or stack.

The action space is defined as the change in desired joint angles
$\Delta\bar{\boldsymbol{q}}$ from the previous time step. Using the same
velocity-based control scheme described in Appendix A.2, we compute the
required torque so that the finger joints can closely track
$\bar{\boldsymbol{q}}_{t}=\bar{\boldsymbol{q}}_{t-1}+\Delta\bar{\boldsymbol{q}}$.

####  A.3.2. Initial state distribution.

At the beginning of each rollout during training, we sample from the initial
state distribution $p_{0}$ to reset the scene containing the arm, hand and
involved objects, from which the agent starts a new trial of grasping or
placing/stacking. Our goal is to design $p_{0}$ such that it covers the space
of possible scenarios that may result from its preceding stage. As such,
$p_{0}$ first provides randomization over the involved object’s shape, size,
mass, and friction coefficient, sampled from uniform distributions with ranges
listed in Appendix B. We also sample $\bar{\boldsymbol{p}}_{i}$ or
$\bar{\boldsymbol{p}}_{d}$ uniformly to cover the reachable range of the arm.
We apply white noise within $2$ cm to the sampled $\bar{\boldsymbol{p}}$ and
use it to reset the involved object’s initial position. The additional white
noise prepares the policy to handle vision inaccuracy of
$\bar{\boldsymbol{p}}$ from initial task planning. With the updated
$\bar{\boldsymbol{p}}_{i}$ or $\bar{\boldsymbol{p}}_{d}$, we follow the
procedure described in Section A.2 to generate a set of arm poses
parameterized by $\theta$.

####  A.3.3. Reward function design for grasping.

Empirically, we found that the more variation in the initial state
distribution that our policies are required to handle, the more carefully we
need to design the reward function in order to help policy training avoid
undesired local optima. The reward function of our grasping policy at each
step $r^{G}_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})$ consists of the
following terms (all norm operators $\left\lVert\cdot\right\rVert$ below
denote Euclidean distance):

(4) $r^{G}_{t}=10r^{o}_{t}+r^{p}_{t}+2r^{c}_{t}+1.5r^{s}_{t}+r^{d}_{t},$

each of which we will elaborate below.

The object pose term $r^{o}_{t}$ discourages objects from moving too much
horizontally, which might lead to knocking over surrounding objects during
test time:

(5)
$r^{o}_{t}=-\left\lVert(\boldsymbol{p}_{t}-\bar{\boldsymbol{p}}_{i})_{x,y}\right\rVert,$

where $\boldsymbol{p}_{t}$ is the object location provided by the simulator
and the subscript $(x,y)$ indicates that the height difference is ignored.

Let
$\boldsymbol{h}_{1}(\boldsymbol{q}^{F}),\cdots,\boldsymbol{h}_{5}(\boldsymbol{q}^{F})$
be the world locations of the finger tips from the thumb to the little finger,
and $\boldsymbol{h}_{p}(\boldsymbol{q}^{F})$ be the world location of the
palm. The form closure term $r^{p}_{t}$ encourages five finger tips and the
palm to stay close to the object:

(6)
$r^{p}_{t}=-5\left\lVert\boldsymbol{h}_{1,t}-\boldsymbol{p}_{t}\right\rVert-\sum_{i=2}^{5}\left\lVert\boldsymbol{h}_{i,t}-\boldsymbol{p}_{t}\right\rVert-2\left\lVert\boldsymbol{h}_{p,t}-\boldsymbol{p}_{t}\right\rVert.$

The contact term $r^{c}_{t}=|\boldsymbol{c}|$ encourages more phalanxes to be
in contact with the object. Similarly to Equation 6, we weigh the thumb
contact five times more than other fingers.

The finger shaping term $r^{s}_{t}$ encourages the four fingers to have a
similar pose:

(7)
$r^{s}_{t}=-\left\lVert\boldsymbol{q}^{F2}-\boldsymbol{q}^{F3}\right\rVert-\left\lVert\boldsymbol{q}^{F3}-\boldsymbol{q}^{F4}\right\rVert-\left\lVert\boldsymbol{q}^{F4}-\boldsymbol{q}^{F5}\right\rVert-\left\lVert\boldsymbol{q}^{F5}-\boldsymbol{q}^{F2}\right\rVert,$

where $\boldsymbol{q}^{F2},\cdots,\boldsymbol{q}^{F5}$ are the subsets of
$\boldsymbol{q}^{F}$ that correspond to the four finger DOFs.

Finally, similar to the training scheme used in Merzić et al. [2019], we add a
‘drop test’ around the end of each training episode to increase robustness of
the policy. During the test, we remove the table and apply downward external
forces on the object for around 0.4s. We add a penalty $r^{d}_{t}=-15$ for
each control step during this test if the object is dropped. Designing a
reward function that encourages lifting behaviors can be done by rewarding
force closure or terminal height of the object, but we found that directly
altering the physics environment results in more sample-efficient learning.

####  A.3.4. Reward function design for placing and stacking.

Designing the reward function for the placing and stacking policy is more
challenging as success states are sparse and the dynamics are much less
stable. At each step the reward
$r^{P}_{t}(\boldsymbol{s}_{t},\boldsymbol{a}_{t})$ consists of the following
terms:

(8) $r^{P}_{t}=5r^{o}_{t}+10r^{a}_{t}-0.5r^{c}_{t}+0.3r^{s}_{t}+r^{b}_{t}.$

With a slight abuse of notation, the object pose term $r^{o}_{t}$ for placing
penalizes both position and orientation deviations from a desired object final
pose $\bar{\boldsymbol{p}}_{f}$, where the object-in-hand is in an upright
pose above the bottom object or at the correct location on the tabletop:

(9)
$r^{o}_{t}=-\left\lVert\boldsymbol{p}_{t}-\bar{\boldsymbol{p}_{f}}\right\rVert+3\boldsymbol{z}^{T}\boldsymbol{R}_{t}\boldsymbol{z},$

where $\boldsymbol{p}_{t}$ is the position of the object being placed,
$\boldsymbol{R}_{t}$ is its rotation matrix, and $\boldsymbol{z}=[0,0,1]$, the
direction of gravity.

The action sparsity term $r^{a}_{t}$ encourages sparse hand force during
placing. Since in our PD-like control scheme, the source of hand force is the
tracking error $\boldsymbol{\bar{q}}-\boldsymbol{q}$, $r^{a}_{t}$ is defined
as:

(10)
$r^{a}_{t}=1/(\left\lVert\boldsymbol{\bar{q}}_{t}-\boldsymbol{q}_{t}\right\rVert+1).$

We use the same definition of contact term $r_{c}$ and the finger shaping term
$r_{s}$ as in those defined for grasping. Instead of encouraging contact with
the object ($2r_{c}$), we slightly discourage ($-0.5r_{c}$) it during
placing/stacking.

Finally, a bonus term $r^{b}_{t}=5$ is added for each step _if_ the object-in-
hand is sufficiently close to $\bar{\boldsymbol{p}}_{f}$, and an additional
bonus $r^{b}_{t}=20$ is added _if_ the hand releases object when the object is
close to $\bar{\boldsymbol{p}}_{f}$.

####  A.3.5. Training details.

We use the default learning parameters reported in the original PPO algorithm
[Schulman et al., 2017] to train both the grasping and placing/stacking
policies. We train the grasping policy for 12M simulation steps and
placing/stacking for 16M simulation steps. To train one single policy that
could both stack an object and place it on tabletop, we randomly split 70% of
the training episodes for stacking trials and 30% for placing. Since stacking
spheres is unlikely to be successfully trained, we trained a separate sphere
grasping policy and placing-on-table policy for qualitative demonstrations
only.

##  Appendix B Quantitative Evaluation Setup

We randomly generate $200$ scenes and tasks to systematically assess the
generality of our system. For each scene, the task is randomly selected from
either (i) place an object at a specified $(x,y,z)$ location, or (ii) stack an
object on top of another object. Language is excluded from this experiment, as
random commands generated by a program is effectively the same as randomly
sampled the parameters of pick-and-stack tasks. We leave user studies with
natural language commands provided by users to future work.

For each object in the scene, we randomly sample shape (box, cylinder, or
sphere), color (red, yellow, green, or blue), width ([6, 10] cm), height ([13,
18] cm), base position ($-0.1\leq x\leq 0.25$, $-0.1\leq y\leq 0.5$, $z=0$),
mass ([$1.0$, $5.0$] kg), and lateral (linear) contact friction coefficient
([0.8, 1.2]). Box widths are scaled by $0.8$ to roughly match the length of
its diagonal with the diameter of its cylindrical counterpart. We sample
sphere sizes according to the object height sampling range, downscaled by
$0.75$. The minimum distance between object centroids was set to $25$ cm. We
randomly generate between 4 and 6 objects per scene, subject to the
constraints listed above, and resample if generated objects fail to meet our
criteria. We limit the number of resampling trials to 50. As a result, some
scenes may have fewer than the desired object count.

For stacking, we set the minimum width of the bottom object in the stack to be
slightly thicker ($9$ cm) to avoid overly difficult tests of stacking onto a
narrower object than self. We additionally enforce shape and color uniqueness
between the task objects and surrounding objects to ensure that the task
specification is unambiguous without providing language commands.